Merge security-a: section A of the backlog (security)
CI / test (push) Failing after 17s

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 12:12:54 +02:00
commit 379074b6a5
149 files changed
+5998 -509

No files matched your search

+13 -6
View File
@@ -93,11 +93,10 @@ jobs:
./dist/llmgw-${version}-linux-amd64 --version
cat dist/checksums.txt
# Signed when the repository has a signing key, so a machine that trusts
# it installs nothing the server alone could have put there. Without the
# secret nothing is signed and releases are checked on their checksums,
# as before; a build that trusts a key refuses those, so the secret has
# to be in place before trusted.pub is filled in.
# Signed, so a machine that trusts the key built into it installs
# nothing the server alone could have put there. `release sign` checks
# the signature against that key before writing it, so a secret that
# does not match trusted.pub stops the release too.
- name: Sign the checksums
env:
RELEASE_SIGNING_KEY: ${{ secrets.RELEASE_SIGNING_KEY }}
@@ -105,7 +104,15 @@ jobs:
set -euo pipefail
version="${GITHUB_REF_NAME:-$(git describe --tags --always)}"
if [ -z "${RELEASE_SIGNING_KEY:-}" ]; then
echo "no RELEASE_SIGNING_KEY secret; releases are not signed"
# Builds that carry a trusted key install only signed releases:
# publishing an unsigned one would be a release no upgraded
# machine accepts, so it stops here instead.
if [ -s internal/release/trusted.pub ]; then
echo "internal/release/trusted.pub holds a key, and the RELEASE_SIGNING_KEY secret is not set:"
echo "add it under Settings -> Secrets, or this release could not be installed anywhere."
exit 1
fi
echo "no RELEASE_SIGNING_KEY secret and no trusted key; releases are not signed"
exit 0
fi
./dist/llmgw-${version}-linux-amd64 release sign dist/checksums.txt
+8
View File
@@ -19,6 +19,14 @@ Conventions this codebase keeps, which every item here must keep too:
## A. Security — do first
**Status (2026-10-04): done and merged.** A1–A9, A6b included — etcd and
Patroni's API on TLS, the PoC moved in maintenance mode without a failover.
Deliberately narrower than written: the admin API is a token on loopback
rather than a unix socket (A5); thread ids are bounded and documented as
project-wide rather than scoped per user (A8); Python engines resolve with an
explicit PyTorch index and a PEP 751 lock rather than locks shipped per engine
version (A8). All of A is live on the PoC.
### A1. Toolchain (S, hours)
`govulncheck` finds 11 standard-library vulnerabilities reachable from this
code at Go 1.26.3 (net/http, crypto/x509, os.Root, net/textproto). Raise the
+215
View File
@@ -23,6 +23,221 @@ APIs change between minors without a deprecation period.
## Unreleased
### Security
- **Built with Go 1.26.6.** Eleven standard-library vulnerabilities reachable
from this code at 1.26.3 (net/http, crypto/x509, os.Root, net/textproto)
are fixed by it. `make vuln` runs govulncheck; `make check` runs it with the
tests and vet.
- **Releases are signed, and only signed releases are installed.** The build
now carries the release signing key (`internal/release/trusted.pub`); the
control plane upgrading itself, an agent told to upgrade and a node being
added all refuse a release whose checksums are not signed with it. The CI
release job stops rather than publish an unsigned release. Adding a node
no longer has the node fetch the release and compare it with a checksum
from the same server: the control plane fetches it for the node's
architecture, checks the signature and the checksum, and hands it over.
Releases are fetched over https only (loopback excepted), and
`LLMGW_RELEASE_KEY` replaces the trusted key in development builds only.
- **The agent port is not opened without TLS by accident.** A control plane
with no `agent_tls` refused nothing: its agent port was plaintext, on every
interface, and any machine reaching it could register as any node — be
given replicas and so be sent prompts. It now refuses to start unless
`agent_insecure: true` is set, and then listens on loopback unless
`agent_listen` names an address. `server init` writes `agent_tls`, as
before.
- **A model definition cannot run code on its nodes.** A profile's `env` and
`args` reach a process the agent runs as root. The environment was checked
against a short denylist that missed NCCL's and CUDA's plugin and
injection libraries; arguments were not checked at all, so
`--trust-remote-code`, llama.cpp's `--log-file` or `--slot-save-path`, or
vLLM's `--tool-parser-plugin` were taken. `env` is now an allowlist of the
families engines are tuned with (CUDA_, NCCL_, VLLM_, OMP_, HF_…) less any
name for a path, plugin, library, address or credential, and llama.cpp's
`LLAMA_ARG_*` are checked as the flags they are; `args` refuse what reads,
writes or loads files, loads code, or moves what the agent sets (host,
port, key, model). Both are refused where a model is defined and again on
the node before the engine starts.
- **Engines no longer run as root.** The agent starts them as the system
user `llmgw-engine` (in the video and render groups for the GPUs), with a
home and caches of its own under the agent's work directory. `llmgw node
add` makes the user; a node without it keeps running engines as root and
the agent says so in its log. `supervisor.run_as` in agent.yaml names
another user, or `root`.
- **The admin API wants a token.** It listened on loopback with no
authentication, so any process or user on the control plane could add a
model, upgrade the fleet, read the engines' keys or dump the heap through
pprof. Its `/api` and `/debug` paths now demand the token the server
writes, 0600, to its state directory (`/var/lib/llmgw/admin-token`) on
first start; `llmgw ctl`, the console and `node add` read it, so they run
with sudo, and `LLMGW_ADMIN_TOKEN` or `ctl --token-file` give it from
elsewhere. pprof is off unless `debug_pprof: true`. `admin_listen` off
loopback is refused unless `admin_remote: true`. The read-only dashboard
pages, `/metrics` and `/health` are unchanged.
- **The replicated store asks for a password and speaks TLS.** Patroni's
REST API took changes from anyone who reached port 8008 — rewriting the
access rules to `trust`, or moving the leader — and PostgreSQL traffic
between the machines was plaintext (`sslmode=disable`). Patroni now asks
for a password before a change (kept with the cluster's others in
`/root/llmgw-pg`, and on the control plane in
`/etc/llmgw/secrets/patroni-rest` for `server add`); each member has a
certificate from the cluster's authority, PostgreSQL has `ssl` on, every
remote access rule is `hostssl`, and the control planes connect with
`sslmode=verify-full` against the authority's certificate. A new cluster is
set up this way; an existing one is changed in place, by reload, the
cluster serving throughout, by running **Replicate the store** in the
console again — or `sudo --preserve-env=SSH_AUTH_SOCK llmgw server init
--patroni <user>@<a> --patroni <user>@<b> --no-mirror --no-node
--no-model` — on the control plane that holds the authority, then on each
other control plane so it reconnects by TLS. etcd is not yet given TLS:
keep ports 2379 and 2380 to the cluster's machines.
- **A capacity fallback respects the key's allowed models, and is charged
at its own price.** When the model asked for had no capacity, the next in
its `fallbacks` served the request even if the key was not allowed it, and
the request was priced as the model asked for. Fallbacks a key may not use
are now skipped, and a request is charged what the model that answered
costs.
- **Suspending an account stops its keys.** Suspension closed the person's
sessions and left every API key they had made working. A key whose owner's
account is not active — suspended, pending, or gone — is now refused as a
revoked key is, within `auth.cache_ttl`, and works again when the account
is reinstated.
- **A join code is a credential only for its hour, and only on its
address.** The setup checked a code's expiry on the joining side alone, so
a code stayed good for as long as the setup ran; it now refuses an expired
one itself. Its listener binds the address the codes name rather than
every interface, and a joined machine is known by the address it connected
from, not the one it claimed.
- **The setup's sudo grant cannot outlive it.** Given a sudo password, the
setup writes a temporary `NOPASSWD` grant and removes it when it
finishes; a setup that crashed or was interrupted left it for good. The
grant now schedules its own removal within the hour as it is written, and
the password is dropped once the grant stands in for it. An address given
to the system ssh can no longer be read as one of its options.
- **Made-up keys cannot fill memory or the database.** Every well-formed key
presented, real or not, was cached for good and cost a database read the
first time. The cache is now bounded, and keys not in it are looked up at
most 20 a second per client address (50 at once); beyond that the request
is answered 429 without a read.
- **Slow or idle connections cannot pile up.** The servers bounded only how
long headers took. Every listener now also bounds how long a connection
may sit idle and how large its headers may be, and reading a request
(5 minutes on the client port, a minute or less elsewhere — never on the
agents' long-lived streams); the client port holds at most
`max_connections` (10,000) at once, and a message from an agent at most
16 MB.
- **The sign-in and signup forms resist scripts.** Ten wrong passwords from
anywhere locked an address's owner out for the window; attempts are now
counted per address and client, so somebody guessing elsewhere cannot lock
the owner out, with a looser count per address alone so guesses spread
over many clients are still bounded. The sign-in form carries a token its
cookie matches, so another site cannot sign a visitor into an account of
its choosing. Posting either form is rationed per client address (sign-in
30 a minute, signup 5), and a password hash that cannot get a turn within
ten seconds answers "busy" instead of holding the request. `server init`
now writes `signup_open: false`.
- **Forwarding headers are believed only from the proxies named.**
`X-Forwarded-Host` and `-Proto` were taken from anyone, and decide the
gateway address written into the `/setup` scripts clients pipe to a
shell; behind a caching proxy one request could poison them for others.
They — and `X-Forwarded-For`, now used for the per-address limits — are
believed only from `router.trusted_proxies` (addresses or CIDR prefixes).
**Behind a reverse proxy, set it**, or the pages and scripts name the
address the proxy reached and every client counts as the proxy. `/setup`
scripts are served `no-store`.
- **Removing a node revokes its certificate.** `ctl node remove` only
dropped the node from the inventory; its certificate, valid for a year,
still connected and was given work. Now the removal is recorded in the
store and the agent port refuses any certificate issued for that node
before it — onboarding the node again issues one that works. A mirror
serving its management API with `--tls-ca` now requires `--api-client`
(the control plane's name), since the same authority signs every node.
- **Engines listen on the management address only**, the one the control
plane reaches them on, instead of every address the node has — a public
one included. (When that address is a name, or not one of the node's,
they still listen on all.) **SGLang's key is off its command line**,
where every user on the node could read it: it reaches the engine in its
environment, and a short launcher hands it to SGLang inside the process.
Both take effect as engines restart under an upgraded agent.
- **Backups can be encrypted.** `backup_recipients` in server.yaml (age
recipients or SSH public keys) has `llmgw server backup` encrypt each
archive to them with age, as `.tar.gz.age`: an archive holds the CA's key
and every secret in the store, and is meant to be copied off the machine.
Open one with `age -d -i <key>`.
- **PyTorch's index gives torch and nothing else.** vLLM's CUDA 12.9 build
was resolved with PyTorch's index beside PyPI and `unsafe-best-match`, so
either index could supply any package — how torchtriton was hijacked in
2022. Now that index is explicit, named only for torch, torchvision and
torchaudio; every other package comes from PyPI alone, and the lock is
PEP 751's (`pylock.llmgw.toml`), pinning each file's address as well as
its hash. Checked against vLLM 0.30.0: the same 195 packages resolve.
- **The control plane's own fetches stay off link-local addresses.**
Checking a model's source and reading a GGUF's header went wherever the
URL said, cloud instance metadata (169.254.169.254) included. They now
refuse link-local, multicast and unspecified addresses, while still
reaching the cluster's network, where the mirror is. The mirror's fetches,
which refuse private addresses too, now also see through IPv6 addresses
carrying an IPv4 one (NAT64, 6to4, IPv4-compatible), and through a proxy:
the target's addresses are checked before the request is handed to it.
- **A CPU-only node reporting a problem no longer panics the agent
handler**, and **thread ids are bounded** to 128 letters, digits and
`- _ . :`. A thread is readable by every key of its project that knows
the id; the docs now say to use unguessable ids.
- **`/v1/models` lists only what the key may use**, rather than every
model the gateway serves to anyone.
- **Portal sessions end after a day unused** (still at most 14 days in
all), and expired or idle sessions and used or expired key claims are
deleted hourly — they were refused, but never removed.
- **Every page carries a Content-Security-Policy**: scripts only from the
gateway itself and the page's own inline ones, each named by its hash —
no inline handlers, no eval, nothing a model's reply or a model card
could slip into a page. The few `onchange`/`onsubmit` attributes moved
to `/_static/pages.js`, htmx runs with `allowEval: false`, and the live
pages' "only while visible" refresh no longer needs an evaluated filter.
Pages now carry their headers even when answered with an error status.
- **KaTeX 0.16.47** (from 0.16.11), past the advisories fixed in 0.16.21.
- **Smaller hardening.** The mirror no longer serves its bookkeeping files
(a file's `.source` URL and `.sha256`) on `:8081`. A GGUF header nesting
arrays more than 4 deep is refused rather than recursed into. Serve and
update jobs keep the last 100 finished, and the mirror forgets finished
downloads after a week, instead of remembering every one until restart.
- **A firewall for llmgw's internal ports.** `server init` (and the store
members it sets up with `--patroni`), `node add`, `server add` and
`mirror init` now keep the machine's internal ports — the agents' stream,
engines, the mirror, PostgreSQL, Patroni, etcd, node exporter — to the
cluster's own networks, in an nftables table of llmgw's own loaded by
`llmgw-firewall.service`. Only those ports are touched: the table accepts
everything else, so other services on the machine and any firewall it
already has are left as they were. Extra networks go in
`firewall_networks`; `--no-firewall` skips it; `llmgw doctor` says when a
node lacks it.
- **etcd and Patroni's API speak TLS.** etcd's peers and clients use
certificates from the cluster's authority, and a client must present one;
Patroni reaches it with its member's certificate and serves its API over
TLS. `server init --patroni`, run again, moves an existing cluster: with
Patroni in maintenance mode, so PostgreSQL serves on and nothing fails
over, etcd is rebuilt on TLS and Patroni's keys carried into it. The
control plane asks the API over HTTPS, verified, and over HTTP only where a
member has not moved yet.
- **A cluster keeping its access rules per member got no `hostssl`.** The
TLS step looked only at the shared configuration, and on a cluster set up
before the rules moved there said the rules were `hostssl` while
plaintext was still accepted. Each member's own rules are now made
`hostssl` too.
### Fixed
- **An engine that died after an agent restart went unnoticed.** The watcher
that reports a ready engine dying was tied to the stream it was started
on, so after a reconnect nothing reported it, and an engine adopted from
the previous agent had no watcher at all: its replica stayed "ready" and
was routed to. Every connection now watches every ready shard, started or
adopted. Found on the PoC: a killed engine on node06 stayed ready for
minutes.
- `server init` rewriting the store's address no longer leaves
`/etc/llmgw/secrets` unreadable to the service when it was made 0700 for
the service user, which kept the gateway from starting.
### Everything that can be changed on a running cluster, from the console
`sudo llmgw` only watched the cluster and edited the configuration files;
+12 -1
View File
@@ -2,7 +2,7 @@ MODULE := gitea.mynodes.xyz/ilker/llmgw
VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
LDFLAGS := -X '$(MODULE)/internal/version.Version=$(VERSION)'
.PHONY: build test race lint fmt clean rename proto proto-tools
.PHONY: build test race lint fmt vuln check clean rename proto proto-tools
build:
go build -ldflags "$(LDFLAGS)" -o bin/llmgw ./cmd/llmgw
@@ -20,6 +20,17 @@ fmt:
go fmt ./...
go vet ./...
# Known vulnerabilities reachable from this code, in the standard library and
# in dependencies. A toolchain left behind is the commonest cause: the go
# directive in go.mod is what fixes it.
vuln:
go run golang.org/x/vuln/cmd/govulncheck@latest ./...
# What has to pass before a change is merged.
check: test
go vet ./...
$(MAKE) vuln
# Regenerate the agent protocol. Generated code is committed, so this only
# needs running when api/proto changes.
proto: proto-tools
+4 -2
View File
@@ -17,8 +17,10 @@ agent_listen: ":7443"
# Mutual TLS on the agent port (plan §5.6). With it, a node's identity is taken
# from its certificate's common name and an agent claiming a different node id
# is refused. Without it the port is plaintext HTTP/2 and anyone who can reach
# it can claim to be any node, so it must stay on a trusted network.
# is refused. Without it the port would be plaintext HTTP/2 and anyone who can
# reach it could claim to be any node, so the control plane refuses to start
# unless `agent_insecure: true` is set — and then listens on loopback unless
# agent_listen names an address. That is for one machine or a test.
#
# These certificates come from llmgw's own authority, not a public one: public
# authorities stopped issuing the clientAuth usage, so no Let's Encrypt
+5 -1
View File
@@ -65,9 +65,13 @@ hand it certificates:
llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
--tls-ca /etc/llmgw/ca.pem \
--tls-cert /etc/llmgw/mirror.pem \
--tls-key /etc/llmgw/mirror-key.pem
--tls-key /etc/llmgw/mirror-key.pem \
--api-client control-plane
```
`--api-client` is required with `--tls-ca`: it names the one certificate the
API answers, because the CA signs every node's certificate as well.
That is the same CA the agents already use, so the certificates come from the
authority you have:
+87 -18
View File
@@ -78,7 +78,7 @@ llmgw --version
`CGO_ENABLED=0` deliberately, so a binary built today runs on a node whose libc
is older than the builder's.
To build instead of download, you need Go 1.26 or later and nothing else:
To build instead of download, you need Go 1.26.6 or later and nothing else (an older Go fetches it itself):
```
git clone https://gitea.mynodes.xyz/ilker/llmgw && cd llmgw
@@ -152,6 +152,15 @@ promoted within about thirty seconds of the primary being lost, and the
control plane follows it by itself. `llmgw ctl status` says who leads; see
[deploy/ha](../deploy/ha/README.md) for what it sets up.
Everything between the machines is TLS, with certificates from the cluster's
own authority: PostgreSQL (every remote rule is `hostssl`, and the control
plane verifies the member it reaches), etcd (peers and clients, and a client
must present a certificate), and Patroni's API, which also asks for a
password before a change. A cluster set up by an earlier llmgw is moved to
this by running the same command again, on the control plane that holds the
authority: etcd is rebuilt on TLS with Patroni in maintenance mode, so
PostgreSQL serves throughout and no failover happens.
Every step looks before it acts. Running it again finishes an install that
stopped halfway and changes nothing on one that did not; it never overwrites
`server.yaml`, the authority, or an existing account's password.
@@ -184,7 +193,7 @@ forwarded headers right; both are attached to every release.
```
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health # 200
llmgw ctl status
sudo llmgw ctl status
```
`ctl status` will report no nodes and no models, which is correct — nothing has
@@ -196,9 +205,15 @@ password, change it, and issue yourself a key from the profile page.
`/etc/llmgw/server.yaml` is yours from here on. Three things in it are worth
knowing before you edit it:
- `admin_listen` is on loopback on purpose. It serves pprof, which can dump the
process heap, and API keys sit in that heap in plaintext. To see the dashboard
from elsewhere use `/admin` on the client port — do not move this listener.
- `admin_listen` is on loopback on purpose, and the server refuses any other
address unless `admin_remote: true`. Its API under `/api` adds models and
upgrades nodes, so it also wants a token: the server writes one, 0600, to
`/var/lib/llmgw/admin-token` the first time it starts, and `llmgw ctl`
reads it from there — which is why `llmgw ctl` runs with sudo.
`LLMGW_ADMIN_TOKEN` gives it from elsewhere. pprof, whose heap holds the API
keys requests presented, is off unless `debug_pprof: true`, and then behind
the token too. To see the dashboard from elsewhere use `/admin` on the
client port — do not move this listener.
- `default_rpm`, `default_tpm` and `default_price` are what make limits and
budgets mean anything. Without a price every request costs zero and a budget
never binds; without the defaults, a project created without limits is
@@ -263,17 +278,20 @@ it can start downloads and delete models, and only the control plane needs it.
The split is what lets you lock the second one down without re-certifying every
agent that fetches from the first.
Give the management port the cluster's own CA and it takes client certificates:
Give the management port the cluster's own CA and it takes client certificates —
the control plane's, named with `--api-client`, and no other. The same CA signs
every node's certificate, so the name is required: without it any node, a
removed one's included, could delete models.
```
llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
--tls-ca /etc/llmgw/ca.pem \
--tls-cert /etc/llmgw/mirror.pem --tls-key /etc/llmgw/mirror-key.pem
--tls-cert /etc/llmgw/mirror.pem --tls-key /etc/llmgw/mirror-key.pem \
--api-client control-plane
```
Without those three flags the management port is open to anyone who can reach
it, and the mirror says so in its own log at every start rather than letting you
find out later.
Without those flags the management port answers this machine only, and the
addresses given with `--api-allow`.
### The layout
@@ -995,6 +1013,10 @@ curl $BASE/v1/chat/completions \
-d '{"model": "<model-name>", "messages": [{"role":"user","content":"and then?"}]}'
```
A thread belongs to the project, not to the key: **any key of the project that
knows the id reads the conversation.** Use ids nobody can guess — a UUID, not
`my-conversation` — of at most 128 letters, digits and `- _ . :`.
llmgw puts the stored history in front of that turn and sends it to whichever
model is serving — **including a different one from the one the conversation
started on**. That is the point: a thread begun on a large model can continue
@@ -1058,6 +1080,39 @@ thing to get right is the health check: point it at `/ready`, not `/health`.
so a load balancer using it would send half the traffic to something that
refuses every request.
## The firewall
`server init`, `node add`, `server add` and `mirror init` each keep that
machine's **internal** llmgw ports to the cluster's own networks, with an
nftables table of llmgw's own (`inet llmgw`, in `/etc/llmgw/firewall.nft`,
loaded at boot by `llmgw-firewall.service`):
| Machine | Ports, for the cluster only |
| --- | --- |
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
| node | engine ports (`control.port_range`, 20000–20999), 9100 |
| store member | 5432, 8008, 2379, 2380, 9100 |
| mirror | 8081, 8082, 9100 |
"The cluster's networks" are, on each machine, the network of the interface
it reaches the control planes (or the other store members) on — or just
their address, when the route goes through a gateway. Anything else that
must reach those ports, a Prometheus on another subnet or nodes behind a
router, goes in `server.yaml` before the machines are set up:
```yaml
firewall_networks: [10.0.5.0/24, 192.168.1.20]
```
Only those ports are touched. The table's policy is to accept, so whatever
else the machine serves — and whatever firewall it already has — is left as
it was; the client API (`:8080`) is not closed, being meant to be reached.
Running a command again adds to what is there rather than replacing it, so a
machine that is both a store member and a node keeps both sets. `llmgw
doctor` on a node says when the table is missing. `--no-firewall` on any of
the four commands leaves the firewall alone; `sudo systemctl disable --now
llmgw-firewall` removes it.
## Upgrading
On the control plane:
@@ -1178,6 +1233,20 @@ The control plane publishes when the newest archive was written as
`llmgw_backup_last_success_timestamp_seconds`, and **BackupTooOld** fires when
it is more than 26 hours old — including when there has never been one.
**Encrypt them.** Name who may open them in `/etc/llmgw/server.yaml`, and every
archive is encrypted to those keys with [age](https://age-encryption.org) and
named `.tar.gz.age`:
```yaml
backup_recipients:
- age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p # age-keygen's public key
- ssh-ed25519 AAAAC3Nza... ops@example # or an SSH public key
```
Keep the matching private keys **off** this machine — an archive whose key
sits beside it is not protected by it. Encrypted, a copy is safe wherever it
lands.
**Copy the archives off the machine.** They sit on the disk they back up and do
not survive losing it. Anything that copies files will do — rsync from the
mirror, a NAS mount, an offsite bucket — as long as it keeps the archives
@@ -1197,6 +1266,8 @@ them:
```
sudo systemctl stop llmgw-server
mkdir /tmp/restore && sudo tar -xzf /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz -C /tmp/restore
# an encrypted one: open it with the private key, from wherever it is kept
age -d -i key.txt llmgw-backup-<host>-<time>.tar.gz.age | sudo tar -xz -C /tmp/restore
cat /tmp/restore/MANIFEST # version and time it was taken
# /etc/llmgw: configuration, secrets, certificates and the CA
@@ -1318,11 +1389,9 @@ requests. On a slow model, set `timeouts.generation` higher.
has them. Single-node placement on real GPUs is the well-trodden path; if you
are the first to run a TP group across real cards, run `llmgw doctor --json`
on each node first and keep the reports.
- **Two engine setups are weaker than the rest.** An engine's key is passed
in its environment, where only root and its own user can read it, except
SGLang's, which takes it only as a flag: other users on the node can see it
in the process list. Mount `/proc` with `hidepid=2` on nodes where that
matters. vLLM's `launch_mode: ray` starts a Ray head without
authentication, so anyone who can reach its port can run code on the
node. The dashboard is off, but firewall the port to the other nodes in the
replica, or use the default `mp` launch mode, which needs no Ray at all.
- **One engine setup is weaker than the rest.** vLLM's `launch_mode: ray`
starts a Ray head without authentication, so anyone who can reach its
port can run code on the node, and Ray's ports are not among those
llmgw's firewall guards. The dashboard is off, but firewall the port to
the other nodes in the replica, or use the default `mp` launch mode, which
needs no Ray at all.
+1 -1
View File
@@ -26,7 +26,7 @@ make test
./bin/llmgw --version
```
`make build` writes `./bin/llmgw`. Go 1.26 or later is the only requirement.
`make build` writes `./bin/llmgw`. Go 1.26.6 or later is the only requirement (an older Go fetches it itself).
## 2. Check the path with a fake engine
+88
View File
@@ -152,6 +152,7 @@ rather than repeated.
| `inject_usage` | `bool` | [Router](#router) |
| `tenant_label` | `string` | [Router](#router) |
| `max_body_bytes` | `int64` | [Router](#router) |
| `trusted_proxies` | `[]string` | [Router](#router) |
| `timeouts` | `Timeouts` | [Router](#router) |
| `scoring` | `Scoring` | [Router](#router) |
| `admission` | `Admission` | [Router](#router) |
@@ -173,12 +174,19 @@ rather than repeated.
| `threads` | `Threads` | [Server](#server) |
| `landing` | `Landing` | [Server](#server) |
| `agent_tls` | `tlsconf.Config` | [Server](#server) |
| `max_connections` | `int` | [Server](#server) |
| `admin_token_file` | `string` | [Server](#server) |
| `admin_remote` | `bool` | [Server](#server) |
| `debug_pprof` | `bool` | [Server](#server) |
| `agent_insecure` | `bool` | [Server](#server) |
| `monitoring` | `Monitoring` | [Server](#server) |
| `tracing` | `Tracing` | [Server](#server) |
| `mirror` | `Mirror` | [Server](#server) |
| `engines` | `map[string]string` | [Server](#server) |
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
| `backup_dir` | `string` | [Server](#server) |
| `firewall_networks` | `[]string` | [Server](#server) |
| `backup_recipients` | `[]string` | [Server](#server) |
| `releases` | `string` | [Server](#server) |
| `router` | `Router` | [Server](#server) |
| `control` | `Control` | [Server](#server) |
@@ -244,6 +252,7 @@ rather than repeated.
| `health_interval` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
| `rendezvous_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
| `alive_grace` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
| `run_as` | `string` | [supervisor.Config](#supervisorconfig) |
| `ca` | `string` | [tlsconf.Config](#tlsconfconfig) |
| `cert` | `string` | [tlsconf.Config](#tlsconfconfig) |
| `key` | `string` | [tlsconf.Config](#tlsconfconfig) |
@@ -1220,6 +1229,15 @@ team is what an alert is usually about; "team" is the default.
MaxBodyBytes bounds the request body the router will read.
### `trusted_proxies`
`[]string`
TrustedProxies are the reverse proxies in front of the client port,
as addresses or CIDR prefixes. Their X-Forwarded-For, -Proto and -Host
are believed; anyone else's are not, since they decide the address
written into the setup scripts and whose attempts are counted where.
### `timeouts`
`Timeouts`
@@ -1376,6 +1394,49 @@ trusted certificate, while this one is mutual and private.
Takes a [tlsconf.Config](#tlsconfconfig) block.
### `max_connections`
`int`
MaxConnections bounds the client port's open connections; past it new
ones wait in the kernel's backlog. 0 is no bound.
### `admin_token_file`
`string`
AdminTokenFile holds the token the admin API's /api and /debug paths
demand. Empty is admin-token in the systemd unit's state directory
(/var/lib/llmgw), or secrets/admin-token beside this file for a server
run by hand. Made, 0600, when the server starts and it does not exist;
`sudo llmgw ctl` and the console read it from there.
### `admin_remote`
`bool`
AdminRemote allows admin_listen on an address other than loopback. The
token still guards the API, but it then crosses the network in clear.
### `debug_pprof`
`bool`
DebugPprof serves Go's profiler on the admin listener, behind the token.
Off by default: a heap profile holds the API keys requests presented.
### `agent_insecure`
`bool`
AgentInsecure lets a control plane run its agent port without
agent_tls: plaintext, and any machine that reaches it may register as
any node — be given replicas, and so be sent users' prompts — or, in
the other direction, a machine in the path may tell every agent what
to run. Refused unless said here, and then the port listens on
loopback unless agent_listen names an address. For one machine, a
test, or a network nothing untrusted reaches.
### `monitoring`
`Monitoring`
@@ -1433,6 +1494,26 @@ BackupDir is where `llmgw server backup` writes, and where the newest
archive's age is read for llmgw_backup_last_success_timestamp_seconds.
Empty means /var/backups/llmgw-server.
### `firewall_networks`
`[]string`
FirewallNetworks are networks, besides those each machine reaches the
cluster on, that llmgw's internal ports answer: a monitoring host on
another subnet, nodes behind a router. Written into every machine's
firewall by `server init`, `node add` and `server add`.
### `backup_recipients`
`[]string`
BackupRecipients encrypt every archive `llmgw server backup` writes, to
whoever holds one of their keys: age recipients (age1…) or SSH public
keys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds
the certificate authority's key and every secret in the store, and is
copied off the machine; encrypted, the copy is safe wherever it lands.
Empty writes archives unencrypted, readable by root only.
### `releases`
`string`
@@ -1882,6 +1963,13 @@ AliveGrace is how long a non-serving rank must stay up before it counts
as ready. There is no external signal for a worker, so staying alive is
the only evidence available.
### `run_as`
`string`
RunAs is the user engines run as when the agent is root; empty is
llmgw-engine, "root" keeps them root.
## tlsconf.Config
### `ca`
+5 -2
View File
@@ -1,13 +1,15 @@
module gitea.mynodes.xyz/ilker/llmgw
go 1.26.3
go 1.26.6
require (
connectrpc.com/connect v1.21.0
filippo.io/age v1.3.2
github.com/NVIDIA/go-nvml v0.13.4-0
github.com/cespare/xxhash/v2 v2.3.0
github.com/charmbracelet/bubbletea v1.3.10
github.com/charmbracelet/lipgloss v1.1.0
github.com/charmbracelet/x/ansi v0.11.6
github.com/jackc/pgx/v5 v5.11.0
github.com/knadh/koanf/parsers/yaml v1.1.1
github.com/knadh/koanf/providers/env/v2 v2.0.1
@@ -32,11 +34,12 @@ require (
)
require (
filippo.io/edwards25519 v1.2.0 // indirect
filippo.io/hpke v0.4.0 // indirect
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/charmbracelet/colorprofile v0.4.1 // indirect
github.com/charmbracelet/x/ansi v0.11.6 // indirect
github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
github.com/charmbracelet/x/term v0.2.2 // indirect
github.com/clipperhouse/displaywidth v0.9.0 // indirect
+8
View File
@@ -1,5 +1,13 @@
c2sp.org/CCTV/age v0.0.0-20260829155415-4448f2097b2d h1:Blprhc2SbChNZtWcU+BLTM4YdoqYAS9V7cJgOwJKyAs=
c2sp.org/CCTV/age v0.0.0-20260829155415-4448f2097b2d/go.mod h1:SrHC2C7r5GkDk8R+NFVzYy/sdj0Ypg9htaPXQq5Cqeo=
connectrpc.com/connect v1.21.0 h1:LhqSJt7jHf5NJBo9Jq/t/9FjcYAideif0mg+qe2jCUs=
connectrpc.com/connect v1.21.0/go.mod h1:A2ygJrukXwWy32vkCAAHNVguZrqZ+jeZ9rGRnGR4dN4=
filippo.io/age v1.3.2 h1:r6RSZLFSMm6rzKepZ7ZAYkKCu14f3/Me8c7uKYh7C8c=
filippo.io/age v1.3.2/go.mod h1:TH/Yr2sSRhCKbaH4XPxpUV0Us8Gv6txYUpiZQWz8Evk=
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
filippo.io/hpke v0.4.0 h1:p575VVQ6ted4pL+it6M00V/f2qTZITO0zgmdKCkd5+A=
filippo.io/hpke v0.4.0/go.mod h1:EmAN849/P3qdeK+PCMkDpDm83vRHM5cDipBJ8xbQLVY=
github.com/NVIDIA/go-nvml v0.13.4-0 h1:o3jp9u2x1R9ShFE3v+Aesp55XOSIQFMJz/VGNUcJaNE=
github.com/NVIDIA/go-nvml v0.13.4-0/go.mod h1:id63qwpoDWpFXwnwM6psDCSqW4BmNu6mWpr4YeQtPGo=
github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
+50
View File
@@ -398,3 +398,53 @@ func TestGuessingTheCurrentPasswordIsBounded(t *testing.T) {
t.Errorf("guessing was unbounded: %v", err)
}
}
func TestSomebodyGuessingElsewhereCannotLockTheOwnerOut(t *testing.T) {
svc, st := newStoreAndUser(t)
ctx := context.Background()
must(t, st.ApproveUser(ctx, "usr-1", "proj", "ilker", time.Now()))
cfg := DefaultConfig()
cfg.MaxAttempts = 3
svc = New(st, cfg, slog.New(slog.DiscardHandler))
attacker, owner := WithClient(ctx, "203.0.113.9"), WithClient(ctx, "198.51.100.7")
for range 5 {
_, _, _ = svc.SignIn(attacker, "ayse@example.com", "nope")
}
if _, _, err := svc.SignIn(attacker, "ayse@example.com", "nope"); !errors.Is(err, ErrTooManyTries) {
t.Errorf("the guessing client was not held back: %v", err)
}
if _, _, err := svc.SignIn(owner, "ayse@example.com", goodPassword); err != nil {
t.Errorf("the owner, elsewhere, was locked out: %v", err)
}
// Spread over many clients, guessing is still bounded per address.
held := false
for i := range 40 {
c := WithClient(ctx, "192.0.2."+strings.Repeat("1", 1+i%3)+string(rune('0'+i%10)))
if _, _, err := svc.SignIn(c, "ayse@example.com", "nope"); errors.Is(err, ErrTooManyTries) {
held = true
}
}
if !held {
t.Error("guesses from many clients were not bounded")
}
}
func TestAHashThatCannotGetATurnSaysTheServerIsBusy(t *testing.T) {
was := argonWait
argonWait = 10 * time.Millisecond
t.Cleanup(func() { argonWait = was })
// Every turn taken.
n := cap(argonSlots)
for range n {
argonSlots <- struct{}{}
}
t.Cleanup(func() {
for range n {
<-argonSlots
}
})
if _, err := HashPassword(goodPassword); !errors.Is(err, ErrBusy) {
t.Errorf("a hash with no turn: %v", err)
}
}
+23 -5
View File
@@ -17,6 +17,7 @@ import (
"fmt"
"runtime"
"strings"
"time"
"golang.org/x/crypto/argon2"
)
@@ -57,7 +58,10 @@ func HashPassword(password string) (string, error) {
if _, err := rand.Read(salt); err != nil {
return "", err
}
sum := idKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
sum, err := idKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
if err != nil {
return "", err
}
b64 := base64.RawStdEncoding.EncodeToString
return fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s",
@@ -94,7 +98,10 @@ func VerifyPassword(password, encoded string) (bool, error) {
return false, ErrBadHash
}
got := idKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
got, err := idKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
if err != nil {
return false, err
}
return subtle.ConstantTimeCompare(got, want) == 1, nil
}
@@ -119,8 +126,19 @@ func init() {
// Beyond the bound, attempts wait their turn instead.
var argonSlots = make(chan struct{}, max(2, runtime.NumCPU()/2))
func idKey(password, salt []byte, time, memory uint32, threads uint8, keyLen uint32) []byte {
argonSlots <- struct{}{}
// ErrBusy is a hash that waited its turn too long: the server is answering
// more sign-ins than it can hash, and says so rather than hold the request.
var ErrBusy = errors.New("too many sign-ins at once; try again in a moment")
// argonWait bounds the wait for a turn; a variable so a test can shorten it.
var argonWait = 10 * time.Second
func idKey(password, salt []byte, t, memory uint32, threads uint8, keyLen uint32) ([]byte, error) {
select {
case argonSlots <- struct{}{}:
case <-time.After(argonWait):
return nil, ErrBusy
}
defer func() { <-argonSlots }()
return argon2.IDKey(password, salt, time, memory, threads, keyLen)
return argon2.IDKey(password, salt, t, memory, threads, keyLen), nil
}
+30 -5
View File
@@ -138,7 +138,15 @@ func (s *Service) SignUp(ctx context.Context, email, name, password string) erro
// the CSRF token that must accompany the forms in it.
func (s *Service) SignIn(ctx context.Context, email, password string) (token, csrf string, err error) {
email = store.NormalizeEmail(email)
if !s.allowAttempt(email) {
// Counted per address and the client trying it, so somebody guessing
// from elsewhere cannot lock the owner out of their own account; and per
// address alone, more loosely, so guesses spread over many clients are
// still bounded.
byClient := email
if c := clientFrom(ctx); c != "" {
byClient = email + "|" + c
}
if !s.allowAttempt(byClient, s.cfg.MaxAttempts) || !s.allowAttempt("all|"+email, 10*s.cfg.MaxAttempts) {
return "", "", ErrTooManyTries
}
@@ -154,11 +162,15 @@ func (s *Service) SignIn(ctx context.Context, email, password string) (token, cs
}
ok, err := VerifyPassword(password, u.PasswordHash)
if errors.Is(err, ErrBusy) {
// Not an answer about the password: said as what it is.
return "", "", err
}
if err != nil || !ok {
return "", "", ErrBadCredentials
}
// The password was right, so this was not an attack.
s.clearAttempts(email)
s.clearAttempts(byClient)
// State is checked after the password, on purpose. Checking first would
// tell anyone who guesses an address whether it has a pending account.
@@ -224,7 +236,7 @@ func (s *Service) CheckCSRF(sess store.Session, submitted string) bool {
// Per address rather than per IP: an attacker has many addresses to come from
// and only one to guess against, and limiting by IP would lock out everyone
// behind one office NAT the moment somebody fat-fingers their password.
func (s *Service) allowAttempt(email string) bool {
func (s *Service) allowAttempt(email string, limit int) bool {
now := s.now()
s.mu.Lock()
defer s.mu.Unlock()
@@ -251,7 +263,20 @@ func (s *Service) allowAttempt(email string) bool {
return true
}
w.count++
return w.count <= s.cfg.MaxAttempts
return w.count <= limit
}
type clientKey struct{}
// WithClient says which client a request comes from — its address — for the
// sign-in throttle.
func WithClient(ctx context.Context, client string) context.Context {
return context.WithValue(ctx, clientKey{}, client)
}
func clientFrom(ctx context.Context) string {
c, _ := ctx.Value(clientKey{}).(string)
return c
}
// maxTrackedAddresses bounds the throttle's memory. Unbounded, it grew by one
@@ -307,7 +332,7 @@ type PasswordChanger interface {
func (s *Service) ChangePassword(ctx context.Context, pw PasswordChanger,
u store.User, current, next string,
) error {
if !s.allowAttempt("pw:" + u.ID) {
if !s.allowAttempt("pw:"+u.ID, s.cfg.MaxAttempts) {
return ErrTooManyTries
}
ok, err := VerifyPassword(current, u.PasswordHash)
+4 -4
View File
@@ -12,22 +12,22 @@ func TestSprayingAddressesCannotGrowTheThrottleOrEscapeIt(t *testing.T) {
// The target is being guessed at.
for i := 0; i < s.cfg.MaxAttempts; i++ {
s.allowAttempt("victim@example.com")
s.allowAttempt("victim@example.com", s.cfg.MaxAttempts)
}
// Then the table is flooded with made-up addresses.
for i := 0; i < maxTrackedAddresses+100; i++ {
s.allowAttempt(fmt.Sprintf("x%d@example.com", i))
s.allowAttempt(fmt.Sprintf("x%d@example.com", i), s.cfg.MaxAttempts)
}
if n := len(s.attempts); n > maxTrackedAddresses {
t.Errorf("the table grew to %d", n)
}
if s.allowAttempt("victim@example.com") {
if s.allowAttempt("victim@example.com", s.cfg.MaxAttempts) {
t.Error("flooding the table reset the target's throttle")
}
// Once the window passes, everything expires and sign-ins work again.
now = now.Add(s.cfg.AttemptWindow + time.Second)
if !s.allowAttempt("new@example.com") {
if !s.allowAttempt("new@example.com", s.cfg.MaxAttempts) {
t.Error("a new address was refused after the flood expired")
}
}
+7 -1
View File
@@ -306,7 +306,13 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
a.tokens.Charge(c, actual-res.EstimatedTokens)
}
u.CostMicros = res.Price.Cost(u.PromptTokens, u.CompletionTokens)
// Priced by the model that answered: a fallback served by another model
// costs what that one costs, not what the one asked for would have.
price := res.Price
if u.ServedModel != "" && u.ServedModel != u.Model && a.pricing != nil {
price = a.pricing.For(u.ServedModel)
}
u.CostMicros = price.Cost(u.PromptTokens, u.CompletionTokens)
for _, b := range res.Budgets {
a.budgets.Add(b.Counter, b.Period, u.CostMicros)
}
+19
View File
@@ -658,3 +658,22 @@ func TestHeadroomIsTheTightestLimitAfterThisRequest(t *testing.T) {
t.Errorf("a key with no limits advertised some: %+v", none)
}
}
func TestAFallbackIsChargedWhatTheModelThatAnsweredCosts(t *testing.T) {
a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
p := inTree("fb-k", teamBudget("fb", 1_000_000))
res, err := a.Check(p, "paid", 400)
if err != nil {
t.Fatal(err)
}
// Asked for "paid", answered by "free": nothing is owed.
a.Record(res, store.Usage{KeyID: "fb-k", Model: "paid", ServedModel: "free", PromptTokens: 200, CompletionTokens: 200})
if spent, _, _ := a.Status(p); spent != 0 {
t.Errorf("charged %d for a request the free model answered", spent)
}
res, _ = a.Check(p, "paid", 400)
a.Record(res, store.Usage{KeyID: "fb-k", Model: "paid", ServedModel: "paid", PromptTokens: 200, CompletionTokens: 200})
if spent, _, _ := a.Status(p); spent == 0 {
t.Error("the paid model answering cost nothing")
}
}
+87 -23
View File
@@ -14,10 +14,12 @@ import (
"math/rand/v2"
"net"
"net/http"
"net/netip"
"os"
"path/filepath"
"runtime"
"slices"
"strconv"
"sync"
"time"
@@ -310,7 +312,7 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
// The same two paths a freshly started shard takes: a rank that serves HTTP
// is health-checked, a worker is judged by staying alive.
if eng.ServesHTTP(o.Spec) {
base := fmt.Sprintf("http://127.0.0.1:%d", o.Spec.HTTPPort)
base := a.engineBase(o.Spec)
err = a.sup.WaitReady(ctx, sh, base, o.Spec.Profile.ServedModelName, a.cfg.AdoptReadyTimeout)
} else {
err = a.sup.WaitAlive(ctx, sh)
@@ -516,6 +518,7 @@ func (a *Agent) session(ctx context.Context, server string) error {
a.sendLoop(ctx, stream, out)
}()
defer wg.Wait()
go a.watchExits(ctx, out)
for {
msg, err := stream.Receive()
@@ -885,6 +888,42 @@ func (a *Agent) watchFetch(source string, fn func(weights.Progress)) (stop func(
}
}
// bindAddr is where engines listen: the management address the control
// plane reaches them on, not every address the node has — a public one
// included. Empty, on every address, when that is a name rather than an
// address this machine holds.
func (a *Agent) bindAddr() string {
ip, err := netip.ParseAddr(a.cfg.MgmtAddr)
if err != nil || ip.IsUnspecified() || !localAddr(ip) {
return ""
}
return ip.String()
}
// engineBase is where the agent asks an engine whether it is ready: where it
// listens, loopback when that is every address.
func (a *Agent) engineBase(s engines.ShardSpec) string {
host := s.BindAddr
if host == "" {
host = "127.0.0.1"
}
return "http://" + net.JoinHostPort(host, strconv.Itoa(s.HTTPPort))
}
// localAddr reports whether an address is one of this machine's.
func localAddr(ip netip.Addr) bool {
addrs, err := net.InterfaceAddrs()
if err != nil {
return false
}
for _, a := range addrs {
if p, err := netip.ParsePrefix(a.String()); err == nil && p.Addr().Unmap() == ip.Unmap() {
return true
}
}
return false
}
func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<- *llmgwv1.AgentMessage) {
shardID, replicaID := m.GetShardId(), m.GetReplicaId()
@@ -907,6 +946,7 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
spec, timeout := agentsvc.ShardSpecFromStart(m)
a.localizeSpec(&spec)
spec.BindAddr = a.bindAddr()
eng, err := a.reg.Get(spec.Profile.Engine)
if err != nil {
@@ -920,6 +960,12 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
// Validate before launching. An engine that refuses a profile does so after
// minutes of loading; catching it here costs nothing.
// Checked again here, whatever sent the profile: this is the process
// about to run as root.
if err := engines.UnsafeArgs(spec.Profile.Engine, spec.Profile.Args); err != nil {
report(supervisor.Failed, err.Error(), nil)
return
}
if err := eng.Validate(spec.Profile, gpuInfo(inv, spec.GPUIndices)); err != nil {
report(supervisor.Failed, err.Error(), nil)
return
@@ -987,7 +1033,7 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
// staying alive, and the replica's real readiness is the leader answering,
// which only happens once every worker has joined (plan §11, phase 4).
if eng.ServesHTTP(spec) {
base := fmt.Sprintf("http://127.0.0.1:%d", spec.HTTPPort)
base := a.engineBase(spec)
err = a.sup.WaitReady(ctx, sh, base, spec.Profile.ServedModelName, timeout)
} else {
err = a.sup.WaitAlive(ctx, sh)
@@ -1005,36 +1051,54 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
return
}
report(supervisor.Ready, "", nil)
// Watch for the engine dying later. Nothing else would notice: the router
// would simply see failing requests.
go a.watch(ctx, sh, replicaID, shardID, out)
// Its dying later is reported by the session's watchExits.
}
// watch reports a shard that exits after it was ready.
func (a *Agent) watch(ctx context.Context, sh *supervisor.Shard, replicaID, shardID string, out chan<- *llmgwv1.AgentMessage) {
// watchExits reports, for as long as this session lasts, every shard that
// was ready and has failed: the ones this agent started, and the ones it
// adopted from the agent before it.
//
// Per session rather than per shard. A watcher tied to the stream a shard
// was started on stopped with that stream, so after a reconnect nothing
// reported the engine dying; and an adopted engine had no watcher at all —
// on node06 one was killed and its replica stayed "ready" for the control
// plane, routed to and failing, indefinitely.
func (a *Agent) watchExits(ctx context.Context, out chan<- *llmgwv1.AgentMessage) {
t := time.NewTicker(time.Second)
defer t.Stop()
ready := map[string]bool{}
for {
for _, st := range a.sup.Statuses() {
switch st.State {
case supervisor.Ready:
ready[st.ID] = true
case supervisor.Failed:
if !ready[st.ID] {
continue // never ready here: its start reported it
}
delete(ready, st.ID)
sh, err := a.sup.Get(st.ID)
if err != nil {
continue
}
msg := &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Shard{Shard: &llmgwv1.ShardStatus{
NodeId: a.cfg.NodeID, ReplicaId: sh.Spec.ReplicaID, ShardId: st.ID,
State: agentsvc.StateToWire(st.State), Error: st.Err,
ExitCode: int32(st.ExitCode), LastOutput: sh.Logs(20),
}}}
select {
case out <- msg:
case <-ctx.Done():
return
}
default:
delete(ready, st.ID)
}
}
select {
case <-ctx.Done():
return
case <-t.C:
st := sh.Status()
if !st.State.Terminal() {
continue
}
msg := &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Shard{Shard: &llmgwv1.ShardStatus{
NodeId: a.cfg.NodeID, ReplicaId: replicaID, ShardId: shardID,
State: agentsvc.StateToWire(st.State), Error: st.Err,
ExitCode: int32(st.ExitCode), LastOutput: sh.Logs(20),
}}}
select {
case out <- msg:
case <-ctx.Done():
}
return
}
}
}
+30
View File
@@ -0,0 +1,30 @@
package agent
import (
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
)
func TestEnginesListenOnTheManagementAddressWhenItIsThisMachines(t *testing.T) {
cases := map[string]string{
"127.0.0.1": "127.0.0.1", // one of this machine's
"203.0.113.9": "", // not: every address rather than none
"gpu07.lan": "", // a name
"0.0.0.0": "",
"": "",
}
for mgmt, want := range cases {
a := &Agent{cfg: Config{MgmtAddr: mgmt}}
if got := a.bindAddr(); got != want {
t.Errorf("management address %q: listens on %q, expected %q", mgmt, got, want)
}
}
a := &Agent{}
if got := a.engineBase(engines.ShardSpec{HTTPPort: 8000}); got != "http://127.0.0.1:8000" {
t.Errorf("on every address, asked at %s", got)
}
if got := a.engineBase(engines.ShardSpec{HTTPPort: 8000, BindAddr: "10.0.0.147"}); got != "http://10.0.0.147:8000" {
t.Errorf("on one address, asked at %s", got)
}
}
+13 -1
View File
@@ -73,9 +73,13 @@ type ShardSpec struct {
HTTPPort int
Profile Profile
// BindAddr is the address the engine listens on: the node's management
// address, which is what the router reaches it on, set by the agent.
// Empty listens on every address.
BindAddr string `json:"bind_addr,omitempty"`
// APIKey is what the engine demands of every request. Engines listen on
// all of a node's addresses; without a key anybody who can reach the port
// the node's management address; without a key anybody who can reach it
// talks to the model directly, past the gateway's authentication, quotas
// and accounting. Given to the engine in its environment rather than on
// its command line, which every user on the node can read.
@@ -98,6 +102,14 @@ type ShardSpec struct {
ProfileFP string
}
// Host is where the engine listens.
func (s ShardSpec) Host() string {
if s.BindAddr == "" {
return "0.0.0.0"
}
return s.BindAddr
}
// Leader reports whether this shard serves HTTP for the replica.
func (s ShardSpec) Leader() bool { return s.Rank == 0 }
+34 -2
View File
@@ -323,8 +323,8 @@ func TestSGLangMultiNodeArgs(t *testing.T) {
if cmd.Path != "/opt/sglang/bin/python" {
t.Errorf("Path=%q", cmd.Path)
}
if cmd.Args[0] != "-m" || cmd.Args[1] != "sglang.launch_server" {
t.Errorf("not launched as a module: %v", cmd.Args[:2])
if cmd.Args[0] != "-c" || !strings.Contains(cmd.Args[1], `run_module("sglang.launch_server"`) {
t.Errorf("not launched as the module: %v", cmd.Args[:2])
}
if argValue(cmd.Args, "--nnodes") != "2" || argValue(cmd.Args, "--node-rank") != "1" {
t.Errorf("multi-node flags wrong: %v", cmd.Args)
@@ -334,6 +334,38 @@ func TestSGLangMultiNodeArgs(t *testing.T) {
}
}
func TestSGLangsKeyIsNotOnItsCommandLine(t *testing.T) {
s := singleNodeSpec()
s.APIKey = "k1"
cmd := buildOne(t, &SGLang{}, s)
if slices.Contains(cmd.Args, "k1") || slices.Contains(cmd.Args, "--api-key") {
t.Errorf("the key is in the arguments: %v", cmd.Args)
}
if !slices.Contains(cmd.Env, sglangKeyEnv+"=k1") {
t.Errorf("the key is not in the environment")
}
// A worker serves no HTTP, and is not given it.
w := multiNodeSpec(1)
w.APIKey = "k1"
if cmd := buildOne(t, &SGLang{}, w); slices.Contains(cmd.Env, sglangKeyEnv+"=k1") {
t.Error("a worker was given the key")
}
}
func TestEnginesListenOnTheAddressTheyAreGiven(t *testing.T) {
s := singleNodeSpec()
for _, e := range []Engine{&VLLM{}, &SGLang{}} {
s.BindAddr = ""
if got := argValue(buildOne(t, e, s).Args, "--host"); got != "0.0.0.0" {
t.Errorf("%s: without an address listens on %q", e.Name(), got)
}
s.BindAddr = "10.0.0.147"
if got := argValue(buildOne(t, e, s).Args, "--host"); got != "10.0.0.147" {
t.Errorf("%s: listens on %q, expected the management address", e.Name(), got)
}
}
}
func TestSGLangSingleNodeOmitsMultiNodeFlags(t *testing.T) {
cmd := buildOne(t, &SGLang{}, singleNodeSpec())
for _, flag := range []string{"--nnodes", "--node-rank", "--dist-init-addr"} {
+1 -1
View File
@@ -99,7 +99,7 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
}
args := []string{
"--model", model,
"--host", "0.0.0.0",
"--host", s.Host(),
"--port", strconv.Itoa(s.HTTPPort),
// Without this there is no /metrics, and an endpoint the router cannot
// measure is one it has to route to blind.
+19 -7
View File
@@ -79,9 +79,9 @@ func (g *SGLang) serverCommand(s ShardSpec) (Command, error) {
python := g.python()
p := s.Profile
args := []string{"-m", "sglang.launch_server",
args := []string{"-c", sglangLauncher,
"--model-path", p.ModelPath,
"--host", "0.0.0.0",
"--host", s.Host(),
"--port", strconv.Itoa(s.HTTPPort),
"--tp", strconv.Itoa(p.TP),
}
@@ -96,11 +96,23 @@ func (g *SGLang) serverCommand(s ShardSpec) (Command, error) {
)
}
args = append(args, p.Args...)
// SGLang reads its key only from the command line, so on a node with
// other users it is visible to them in the process list.
env := baseEnv(s)
if s.APIKey != "" && s.Leader() {
args = append(args, "--api-key", s.APIKey)
env[sglangKeyEnv] = s.APIKey
}
return Command{Name: "sglang", Path: python, Args: args, Env: materializeEnv(baseEnv(s))}, nil
return Command{Name: "sglang", Path: python, Args: args, Env: materializeEnv(env)}, nil
}
// SGLang reads its key only from the command line, where every user on the
// node can read it in the process list. So the key goes in the environment,
// which only the engine's own user and root can read, and this launcher —
// sglang.launch_server under another name — adds it to the arguments inside
// the process: sys.argv is Python's copy, not what /proc shows.
const (
sglangKeyEnv = "LLMGW_ENGINE_KEY"
sglangLauncher = `import os, runpy, sys
key = os.environ.pop("` + sglangKeyEnv + `", "")
sys.argv = ["sglang.launch_server"] + sys.argv[1:] + (["--api-key", key] if key else [])
runpy.run_module("sglang.launch_server", run_name="__main__", alter_sys=True)
`
)
+2 -1
View File
@@ -2,6 +2,7 @@ package engines
import (
"fmt"
"net"
"os"
"strconv"
)
@@ -63,7 +64,7 @@ func (s *Sim) Commands(spec ShardSpec) ([]Command, error) {
p := spec.Profile
args := []string{"sim-engine",
"--listen", "0.0.0.0:" + strconv.Itoa(spec.HTTPPort),
"--listen", net.JoinHostPort(spec.Host(), strconv.Itoa(spec.HTTPPort)),
"--model", p.ServedModelName,
}
if spec.NNodes > 1 {
+138 -21
View File
@@ -1,30 +1,147 @@
package engines
import "strings"
import (
"fmt"
"strings"
)
// A model profile's env and args are the operator's escape hatch for tuning
// an engine, and they reach a process the agent starts as root on every node
// the model is placed on. Some of what an engine accepts is not tuning but a
// way to run other code, or to read or write files, in that process: a
// loader preload, an NCCL network plugin, a CUDA injection library, vLLM's
// --trust-remote-code, llama.cpp's --log-file pointed at /etc/cron.d. Taken
// from a model definition, which the admin API or a config file supplies,
// they would turn "add a model" into "run this as root on every node".
//
// So env is an allowlist — the families of variables engines are tuned with
// — less any whose name says it names a path, a plugin or a library, and args
// are checked against what each engine takes that reads, writes or loads.
// Both are refused where a model is defined and again where an engine starts.
// envFamilies are the prefixes of variables engines are tuned with.
var envFamilies = []string{
"CUDA_", "NCCL_", "NVIDIA_", "NVTE_", "GLOO_", "UCX_", "TORCH_", "PYTORCH_",
"VLLM_", "SGLANG_", "GGML_", "LLAMA_", "OMP_", "MKL_", "OPENBLAS_", "KMP_",
"HF_", "TRANSFORMERS_", "TOKENIZERS_", "TRITON_", "FLASHINFER_",
}
// envExact are single variables that are tuning too.
var envExact = map[string]bool{
"PYTHONUNBUFFERED": true, "PYTHONHASHSEED": true, "TZ": true,
}
// envNever are names in an allowed family that load code or move where the
// engine reads and writes.
var envNever = []string{
"PLUGIN", "INJECTION", "PRELOAD", "LIBRARY", "_LIB",
"_PATH", "_DIR", "_HOME", "_FILE", "_ROOT", "_CACHE", "_CONFIG",
"_ADDRESS", "_ENDPOINT", "_TOKEN", "_URL",
}
// UnsafeEnv reports whether a profile may not set an environment variable,
// and why.
//
// A profile's env is the escape hatch for tuning no adapter can guess —
// CUDA_*, NCCL_*, VLLM_*, HF_* — and it reaches an engine the agent starts as
// root. Some variables are not tuning but a way to run other code in that
// process: the dynamic loader's LD_PRELOAD, Python's PYTHONSTARTUP, a shell's
// BASH_ENV. Set from a model definition, which an admin API or a config file
// supplies, they would turn "add a model" into "run this as root on every
// node". They are refused where a model is defined and again where an engine
// starts.
func UnsafeEnv(key string) (bool, string) {
k := strings.ToUpper(key)
switch {
case strings.HasPrefix(k, "LD_"), strings.HasPrefix(k, "DYLD_"):
return true, "it changes what the dynamic loader loads"
case k == "PATH":
return true, "it changes which binaries run"
case strings.HasPrefix(k, "PYTHON") && k != "PYTHONUNBUFFERED" && k != "PYTHONHASHSEED":
return true, "it changes what Python imports or runs at start"
case k == "BASH_ENV" || k == "ENV" || k == "PERL5OPT" || k == "PERL5LIB" ||
k == "NODE_OPTIONS" || k == "RUBYOPT" || k == "GCONV_PATH" || k == "OPENSSL_CONF":
return true, "it makes an interpreter or library load code"
k := strings.ToUpper(strings.TrimSpace(key))
if k == "" {
return true, "it has no name"
}
allowed := envExact[k]
for _, f := range envFamilies {
if strings.HasPrefix(k, f) {
allowed = true
break
}
}
if !allowed {
return true, "only the variables engines are tuned with are allowed (" +
strings.Join(envFamilies, "*, ") + "*)"
}
for _, n := range envNever {
if strings.Contains(k, n) && !envExact[k] {
return true, "it names a path, a plugin, a library or an address, which is a way to make the engine " +
"read, write or load something other than its model"
}
}
// llama.cpp reads every flag from LLAMA_ARG_<FLAG> too.
if rest, ok := strings.CutPrefix(k, "LLAMA_ARG_"); ok {
flag := "--" + strings.ToLower(strings.ReplaceAll(rest, "_", "-"))
if why := unsafeFlag("llamacpp", flag, ""); why != "" {
return true, "it is " + flag + ", which " + why
}
}
return false, ""
}
// flagsNever are flags no profile may give, by engine; "" is every engine.
// Each reads or writes files the agent did not choose, loads code, or moves
// what the agent itself sets: where the engine listens and the key it
// demands.
var flagsNever = map[string]map[string]string{
"": {
"--host": "is set by the agent", "--port": "is set by the agent",
"--api-key": "is set by the agent", "--api-key-file": "is set by the agent",
"--model": "is the model's source, set by the agent", "-m": "is the model's source, set by the agent",
"--ssl-keyfile": "reads a file", "--ssl-certfile": "reads a file",
"--ssl-key-file": "reads a file", "--ssl-cert-file": "reads a file",
"--trust-remote-code": "runs Python the model repository ships",
},
"llamacpp": {
"--log-file": "writes a file", "--slot-save-path": "writes files",
"--path": "serves a directory's files over HTTP", "--rpc": "sends the model's work to other machines",
"--chat-template-file": "reads a file", "--system-prompt-file": "reads a file",
"--grammar-file": "reads a file", "--json-schema-file": "reads a file",
"--lora": "loads a file; adapters are set in the model's adapters",
"--lora-scaled": "loads a file; adapters are set in the model's adapters",
"--control-vector": "loads a file", "--control-vector-scaled": "loads a file",
"--mmproj": "loads a file; the projector is fetched with the model",
"--model-url": "downloads a model", "-mu": "downloads a model",
"--hf-repo": "downloads a model", "-hf": "downloads a model", "--hf-file": "downloads a model",
"--hf-token": "is a credential", "--alias": "is set by the agent",
"--model-draft": "loads a file", "-md": "loads a file",
"--prompt-cache": "writes a file", "--logdir": "writes files",
},
"vllm": {
"--download-dir": "writes files", "--tokenizer": "reads another repository or path",
"--chat-template": "reads a file", "--allowed-local-media-path": "serves local files to requests",
"--lora-modules": "loads files; adapters are set in the model's adapters",
"--tool-parser-plugin": "runs a Python file", "--middleware": "imports Python",
"--worker-extension-cls": "imports Python", "--worker-cls": "imports Python",
"--load-format": "changes how weights are loaded, some ways running code",
"--otlp-traces-endpoint": "sends request traces elsewhere",
"--served-model-name": "is set by the agent",
"--enable-log-requests": "logs prompts",
},
"sglang": {
"--tokenizer-path": "reads another repository or path", "--chat-template": "reads a file",
"--file-storage-path": "writes files", "--log-requests": "logs prompts",
"--model-path": "is the model's source, set by the agent", "--served-model-name": "is set by the agent",
"--lora-paths": "loads files", "--download-dir": "writes files",
},
}
// unsafeFlag is why an engine may not be given a flag, or "".
func unsafeFlag(engine, flag, _ string) string {
name, _, _ := strings.Cut(flag, "=")
for _, set := range []string{"", engine} {
if why, ok := flagsNever[set][name]; ok && why != "" {
return why
}
}
return ""
}
// UnsafeArgs refuses the first of a profile's arguments an engine may not be
// given, saying why.
func UnsafeArgs(engine string, args []string) error {
for _, a := range args {
if !strings.HasPrefix(a, "-") {
continue // a value
}
if why := unsafeFlag(engine, a, ""); why != "" {
name, _, _ := strings.Cut(a, "=")
return fmt.Errorf("argument %s is not allowed: it %s", name, why)
}
}
return nil
}
+64
View File
@@ -0,0 +1,64 @@
package engines
import "testing"
func TestTuningVariablesAreAllowedAndWaysToRunCodeAreNot(t *testing.T) {
for _, k := range []string{
"CUDA_VISIBLE_DEVICES", "CUDA_MODULE_LOADING", "NCCL_IB_HCA", "NCCL_DEBUG", "NCCL_NET",
"VLLM_ATTENTION_BACKEND", "PYTORCH_CUDA_ALLOC_CONF", "OMP_NUM_THREADS", "HF_HUB_OFFLINE",
"GGML_VK_VISIBLE_DEVICES", "TOKENIZERS_PARALLELISM", "PYTHONUNBUFFERED", "LLAMA_ARG_CTX_SIZE",
} {
if bad, why := UnsafeEnv(k); bad {
t.Errorf("%s refused: %s", k, why)
}
}
for _, k := range []string{
"LD_PRELOAD", "PATH", "PYTHONPATH", "PYTHONSTARTUP", "BASH_ENV",
// What the denylist missed: each loads a library or moves where the
// engine reads and writes.
"NCCL_NET_PLUGIN", "NCCL_TUNER_PLUGIN", "NCCL_PROFILER_PLUGIN", "CUDA_INJECTION64_PATH",
"NVTX_INJECTION64_PATH", "TRITON_CACHE_DIR", "HF_HOME", "HF_TOKEN", "NCCL_DEBUG_FILE",
"VLLM_PLUGINS", "UCX_MODULE_DIR", "TORCH_EXTENSIONS_DIR",
// llama.cpp's flags as variables.
"LLAMA_ARG_LOG_FILE", "LLAMA_ARG_HOST", "LLAMA_ARG_SLOT_SAVE_PATH",
// Anything outside the families.
"SOMETHING_ELSE", "",
} {
if bad, _ := UnsafeEnv(k); !bad {
t.Errorf("%s allowed", k)
}
}
}
func TestArgumentsThatReadWriteOrLoadAreRefused(t *testing.T) {
ok := map[string][]string{
"llamacpp": {"--ctx-size", "16384", "--embedding", "--parallel", "4", "--jinja", "--n-gpu-layers=999"},
"vllm": {"--max-model-len", "8192", "--gpu-memory-utilization", "0.9", "--enable-auto-tool-choice"},
"sglang": {"--mem-fraction-static", "0.8", "--tool-call-parser", "qwen25"},
}
for eng, args := range ok {
if err := UnsafeArgs(eng, args); err != nil {
t.Errorf("%s %v: %v", eng, args, err)
}
}
for _, c := range []struct {
eng string
args []string
}{
{"vllm", []string{"--trust-remote-code"}},
{"sglang", []string{"--trust-remote-code"}},
{"vllm", []string{"--tool-parser-plugin", "/tmp/x.py"}},
{"vllm", []string{"--middleware=evil.mod"}},
{"vllm", []string{"--allowed-local-media-path", "/"}},
{"llamacpp", []string{"--log-file", "/etc/cron.d/x"}},
{"llamacpp", []string{"--slot-save-path=/root"}},
{"llamacpp", []string{"--path", "/"}},
{"llamacpp", []string{"--system-prompt-file", "/etc/shadow"}},
{"llamacpp", []string{"--host", "0.0.0.0"}},
{"vllm", []string{"--api-key", "x"}},
} {
if err := UnsafeArgs(c.eng, c.args); err == nil {
t.Errorf("%s %v allowed", c.eng, c.args)
}
}
}
+1 -1
View File
@@ -106,7 +106,7 @@ func (v *VLLM) serverCommand(s ShardSpec) (Command, error) {
p := s.Profile
args := []string{"serve", p.ModelPath,
"--host", "0.0.0.0",
"--host", s.Host(),
"--port", strconv.Itoa(s.HTTPPort),
"--tensor-parallel-size", strconv.Itoa(p.TP),
}
+68 -17
View File
@@ -8,6 +8,8 @@ import (
"net/http"
"os"
"path/filepath"
"slices"
"strconv"
"strings"
"syscall"
"time"
@@ -33,6 +35,10 @@ const pythonInstallTimeout = time.Hour
// lockName is the resolved requirements, kept in the environment.
const lockName = "llmgw-requirements.txt"
// pylockName is the lock of a build with a second index: PEP 751's, with
// every file's address.
const pylockName = "pylock.llmgw.toml"
func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
ctx, cancel := context.WithTimeout(ctx, pythonInstallTimeout)
defer cancel()
@@ -113,28 +119,16 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
b.Wheel.Tag + "/" + asset
reqs = append([]string{b.Wheel.Project + " @ " + url + "#sha256=" + sum}, reqs...)
}
// A second index, for the torch the engine was built on. Every package
// is taken from whichever of the two has the version resolved — both are
// the publishers' own — and each file is still pinned by its hash.
var index []string
if b.ExtraIndex != "" {
index = []string{"--extra-index-url", b.ExtraIndex, "--index-strategy", "unsafe-best-match"}
}
in.progress(st, "resolving %s against PyPI as of %s", strings.Join(reqs, " "), asOf[:10])
lock := filepath.Join(tmp, lockName)
args := append([]string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
"--generate-hashes", "--quiet", "--output-file", lock}, index...)
args = append(args, "-")
if out, err := in.runStdin(ctx, env, strings.Join(reqs, "\n")+"\n", uv.Binary, args...); err != nil {
return fmt.Errorf("resolving %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
lock, err := in.resolve(ctx, env, uv.Binary, python, tmp, asOf, reqs, b)
if err != nil {
return err
}
in.progress(st, "downloading and installing %s and its dependencies: several gigabytes, which takes minutes",
strings.Join(reqs, " "))
install := append([]string{"pip", "install", "--python", python,
"--require-hashes", "--no-deps", "-r", lock}, index...)
if out, err := in.run(ctx, env, uv.Binary, install...); err != nil {
if out, err := in.run(ctx, env, uv.Binary, "pip", "install", "--python", python,
"--require-hashes", "--no-deps", "-r", lock); err != nil {
return fmt.Errorf("installing %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
}
@@ -168,6 +162,63 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
return nil
}
// resolve pins every package the requirements need, each file by its hash,
// into a lock in the environment, and returns its path.
//
// With a second index — PyTorch's, for the torch a build was made against —
// the packages it is for come from it and nowhere else, and every other
// package from PyPI alone. Letting both indexes offer every package is how
// torchtriton was taken in 2022: a name one index had, registered on the
// other with a higher version. So the second index is explicit, named only
// for its packages, and the lock is PEP 751's, which records each file's
// address as well as its hash: installing it asks no index anything.
func (in *Installer) resolve(ctx context.Context, env []string, uv, python, tmp, asOf string,
reqs []string, b Build,
) (string, error) {
if b.ExtraIndex == "" {
lock := filepath.Join(tmp, lockName)
if out, err := in.runStdin(ctx, env, strings.Join(reqs, "\n")+"\n", uv, "pip", "compile",
"--python", python, "--exclude-newer", asOf, "--generate-hashes", "--quiet",
"--output-file", lock, "-"); err != nil {
return "", fmt.Errorf("resolving %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
}
return lock, nil
}
project := tmp + ".resolve"
if err := os.MkdirAll(project, 0o755); err != nil {
return "", err
}
defer os.RemoveAll(project) //nolint:errcheck // scratch
if err := os.WriteFile(filepath.Join(project, "pyproject.toml"), []byte(resolveProject(reqs, b)), 0o644); err != nil { //nolint:gosec // not secret
return "", err
}
lock := filepath.Join(tmp, pylockName)
if out, err := in.run(ctx, env, uv, "pip", "compile", "--python", python, "--exclude-newer", asOf,
"--format", "pylock.toml", "--quiet", "--output-file", lock,
filepath.Join(project, "pyproject.toml")); err != nil {
return "", fmt.Errorf("resolving %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
}
return lock, nil
}
// resolveProject is the pyproject.toml a build with a second index is
// resolved from. Its packages are direct requirements too: an index is
// given to a package only where the project names it.
func resolveProject(reqs []string, b Build) string {
var p strings.Builder
p.WriteString("[project]\nname = \"llmgw-engine\"\nversion = \"0\"\ndependencies = [\n")
for _, r := range append(slices.Clone(reqs), b.IndexPackages...) {
fmt.Fprintf(&p, " %s,\n", strconv.Quote(r))
}
p.WriteString("]\n\n[[tool.uv.index]]\nname = \"extra\"\n")
fmt.Fprintf(&p, "url = %s\nexplicit = true\n\n[tool.uv.sources]\n", strconv.Quote(b.ExtraIndex))
for _, name := range b.IndexPackages {
fmt.Fprintf(&p, "%s = { index = \"extra\" }\n", name)
}
return p.String()
}
// published is when PyPI received a release's last file.
func (in *Installer) published(ctx context.Context, project, version string) (time.Time, error) {
url := strings.TrimSuffix(in.pypiBase(), "/") + "/pypi/" + project + "/" + version + "/json"
+16 -4
View File
@@ -40,6 +40,14 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
return []byte("uv 0.12.20\n"), nil
case strings.HasSuffix(name, "/uv") && args[0] == "venv":
return nil, os.MkdirAll(filepath.Join(args[len(args)-1], "bin"), 0o755)
case strings.HasSuffix(name, "/uv") && args[0] == "pip" && args[1] == "compile":
// From a pyproject: kept in the record, as the request was.
project, _ := os.ReadFile(args[len(args)-1])
f.pmu.Lock()
f.commands = append(f.commands, "pyproject: "+string(project))
f.pmu.Unlock()
out := args[slices.Index(args, "--output-file")+1]
return nil, os.WriteFile(out, []byte("lock-version = \"1.0\"\n"), 0o644)
case strings.HasSuffix(name, "/uv") && args[0] == "pip" && args[1] == "install":
if f.failPip {
return []byte("error: Hash mismatch for torch\n"), errExit
@@ -220,8 +228,12 @@ func TestAnOlderDriverGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
// Pinned to the digest GitHub publishes for the wheel.
"vllm @ " + in.DownloadBase + "/vllm-project/vllm/releases/download/v0.30.0/" + wheel + "#sha256=" +
strings.TrimPrefix(gh.digests[wheel], "sha256:"),
"--extra-index-url https://download.pytorch.org/whl/cu129 --index-strategy unsafe-best-match -",
"--require-hashes --no-deps -r",
// The torch packages from PyTorch's index and only those; the lock
// records every file's address.
"--format pylock.toml",
`url = "https://download.pytorch.org/whl/cu129"` + "\nexplicit = true",
`torch = { index = "extra" }`, `torchvision = { index = "extra" }`,
"--require-hashes --no-deps -r " + filepath.Join(in.Root, "vllm", "0.30.0-cu129.partial", pylockName),
// Computing on the card, not only importing.
"/bin/python -c import torch",
} {
@@ -230,8 +242,8 @@ func TestAnOlderDriverGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
}
}
for _, c := range sys.commands {
if strings.Contains(c, "pip install") && !strings.Contains(c, "download.pytorch.org/whl/cu129") {
t.Errorf("installed without the index its torch comes from: %s", c)
if strings.Contains(c, "unsafe-best-match") || strings.Contains(c, "--extra-index-url") {
t.Errorf("every index offered every package: %s", c)
}
}
}
+11 -4
View File
@@ -50,10 +50,12 @@ type Build struct {
GPUBackend string
// Wheel, when set, is the engine itself as a wheel from a GitHub
// release rather than from PyPI, checked against the digest GitHub
// publishes for it. ExtraIndex is a package index besides PyPI that the
// rest resolves from too: the CUDA 12.9 torch the wheel was built on.
Wheel *Wheel
ExtraIndex string
// publishes for it. ExtraIndex is a package index besides PyPI, for
// IndexPackages only: the CUDA 12.9 torch the wheel was built on. Every
// other package comes from PyPI alone.
Wheel *Wheel
ExtraIndex string
IndexPackages []string
// GPUCheck, on a node with an NVIDIA GPU, has the installed environment
// compute something on one before the build counts as installed. A build
// with no kernels for the card — a GPU generation its torch dropped —
@@ -234,6 +236,10 @@ var VLLMVariants = []string{"cu13", "cu129"}
// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
// pytorchPackages are what is taken from PyTorch's index: the builds for one
// CUDA version, which PyPI does not have.
var pytorchPackages = []string{"torch", "torchvision", "torchaudio"}
// vllmCUDA13Since is the first vLLM whose PyPI build is CUDA 13, read from
// its releases: 0.19.1 publishes "+cu130" beside a CUDA 12 default, 0.20.0
// "+cu129" beside a CUDA 13 one.
@@ -332,6 +338,7 @@ func vllmCandidates(version, variant string, h discovery.Host) ([]Build, error)
Prefix: "vllm-" + v + "+cu129-", Suffix: "_" + map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch] + ".whl",
}
cu129.ExtraIndex = pytorchCU129
cu129.IndexPackages = pytorchPackages
return pick("vLLM "+v, variant, []candidate{
{pypi("cu13"), gpuNeeds{minDriverForCUDA13, minCC}, "vLLM " + v + " built for CUDA 13"},
{cu129, gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v + " built for CUDA 12.9"},
+92
View File
@@ -0,0 +1,92 @@
package supervisor
import (
"fmt"
"os"
"os/user"
"path/filepath"
"strconv"
"syscall"
)
// Engines run as their own user, not as the agent's root: an engine reads a
// model's files and answers requests, and a model's files are whatever a
// registry published. Root in that process would be root on the node for
// anything that can make an engine misbehave — a malformed file, a template,
// a flag nobody thought of.
//
// The user is DefaultRunAs, made by `llmgw node add`, in the video and render
// groups for the GPU devices. A node without it keeps running engines as
// root, and says so, so an existing fleet is not broken by an upgrade.
// DefaultRunAs is the user engines run as.
const DefaultRunAs = "llmgw-engine"
// runAs is how an engine process is started as another user: its
// credential, a working directory it owns, and the environment that points
// its caches there.
type runAs struct {
name string
cred *syscall.Credential
home string
}
// lookupUser is user.Lookup, replaced by tests.
var lookupUser = user.Lookup
// resolveRunAs finds the user engines run as. nil means as this process —
// when it is not root, there is nobody to drop to; when the user does not
// exist, why is returned.
func resolveRunAs(name, workDir string) (*runAs, string) {
if name == "" || name == "root" || os.Geteuid() != 0 {
return nil, ""
}
u, err := lookupUser(name)
if err != nil {
return nil, fmt.Sprintf("there is no user %s, so engines run as root; `llmgw node add` makes it, "+
"or: useradd --system --shell /usr/sbin/nologin %s && usermod -aG video,render %s", name, name, name)
}
r, err := credentialFor(u, workDir)
if err != nil {
return nil, fmt.Sprintf("engines run as root: %v", err)
}
return r, ""
}
func credentialFor(u *user.User, workDir string) (*runAs, error) {
uid, err1 := strconv.ParseUint(u.Uid, 10, 32)
gid, err2 := strconv.ParseUint(u.Gid, 10, 32)
if err1 != nil || err2 != nil {
return nil, fmt.Errorf("user %s has ids %s:%s", u.Username, u.Uid, u.Gid)
}
var groups []uint32
if ids, err := u.GroupIds(); err == nil {
for _, g := range ids {
if n, err := strconv.ParseUint(g, 10, 32); err == nil {
groups = append(groups, uint32(n))
}
}
}
// A home of its own under the work directory, which is root's: caches
// (Triton, torch, Hugging Face) are written there and nowhere else.
home := filepath.Join(workDir, "engine-home")
if err := os.MkdirAll(home, 0o700); err != nil {
return nil, fmt.Errorf("making %s: %w", home, err)
}
if err := os.Chown(home, int(uid), int(gid)); err != nil {
return nil, fmt.Errorf("giving %s to %s: %w", home, u.Username, err)
}
// The work directory is 0750 root's; the engine has to pass through it
// to its home.
if err := os.Chmod(workDir, 0o751); err != nil {
return nil, err
}
return &runAs{name: u.Username, home: home,
cred: &syscall.Credential{Uid: uint32(uid), Gid: uint32(gid), Groups: groups}}, nil
}
// env is what is added to an engine's environment when it runs as r: the
// last value of a key wins in exec.
func (r *runAs) env() []string {
return []string{"HOME=" + r.home, "XDG_CACHE_HOME=" + filepath.Join(r.home, ".cache"), "USER=" + r.name, "LOGNAME=" + r.name}
}
+51
View File
@@ -0,0 +1,51 @@
package supervisor
import (
"os"
"os/exec"
"os/user"
"path/filepath"
"slices"
"strconv"
"syscall"
"testing"
)
func TestAnEngineIsGivenItsUsersCredentialHomeAndCaches(t *testing.T) {
me, err := user.Current()
if err != nil {
t.Skip(err)
}
work := filepath.Join(t.TempDir(), "agent")
if err := os.MkdirAll(work, 0o750); err != nil {
t.Fatal(err)
}
r, err := credentialFor(me, work)
if err != nil {
t.Fatal(err)
}
uid, _ := strconv.Atoi(me.Uid)
if int(r.cred.Uid) != uid || r.home != filepath.Join(work, "engine-home") {
t.Fatalf("%+v", r)
}
if st, err := os.Stat(r.home); err != nil || st.Mode().Perm() != 0o700 {
t.Errorf("home: %v %v", st.Mode(), err)
}
s := &Supervisor{cfg: Config{WorkDir: work}, as: r}
c := exec.Command("true")
c.Env = []string{"HOME=/root", "PATH=/bin"}
c.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
s.dropTo(c)
if c.SysProcAttr.Credential == nil || c.Dir != r.home || !slices.Contains(c.Env, "HOME="+r.home) {
t.Errorf("dir %s env %v cred %v", c.Dir, c.Env, c.SysProcAttr.Credential)
}
}
func TestNotRootOrNoSuchUserLeavesEnginesAsTheAgent(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("run as root")
}
if r, why := resolveRunAs(DefaultRunAs, t.TempDir()); r != nil || why != "" {
t.Errorf("not root: %v %q", r, why)
}
}
+33 -1
View File
@@ -39,6 +39,9 @@ type Config struct {
// as ready. There is no external signal for a worker, so staying alive is
// the only evidence available.
AliveGrace time.Duration `koanf:"alive_grace"`
// RunAs is the user engines run as when the agent is root; empty is
// llmgw-engine, "root" keeps them root.
RunAs string `koanf:"run_as"`
}
// DefaultConfig returns sensible defaults.
@@ -50,6 +53,7 @@ func DefaultConfig() Config {
HealthInterval: 2 * time.Second,
RendezvousTimeout: 5 * time.Minute,
AliveGrace: 5 * time.Second,
RunAs: DefaultRunAs,
}
}
@@ -58,6 +62,9 @@ func (c *Config) normalize() {
if c.WorkDir == "" {
c.WorkDir = d.WorkDir
}
if c.RunAs == "" {
c.RunAs = d.RunAs
}
if c.LogLines <= 0 {
c.LogLines = d.LogLines
}
@@ -154,6 +161,9 @@ type Supervisor struct {
mu sync.Mutex
shards map[string]*Shard
// as is who engines run as; nil is as the agent.
as *runAs
}
// New builds a supervisor.
@@ -165,7 +175,16 @@ func New(cfg Config, health HealthChecker, log *slog.Logger) (*Supervisor, error
if err := os.MkdirAll(cfg.WorkDir, 0o750); err != nil {
return nil, fmt.Errorf("work dir: %w", err)
}
return &Supervisor{cfg: cfg, log: log, health: health, shards: map[string]*Shard{}}, nil
s := &Supervisor{cfg: cfg, log: log, health: health, shards: map[string]*Shard{}}
as, why := resolveRunAs(cfg.RunAs, cfg.WorkDir)
if why != "" {
log.Warn(why)
}
if as != nil {
log.Info("engines run as their own user", "user", as.name)
}
s.as = as
return s, nil
}
// ErrAlreadyRunning means a shard with that ID is already supervised.
@@ -261,6 +280,7 @@ func (s *Supervisor) launch(ctx context.Context, sh *Shard, cmds []engines.Comma
// Engines spawn workers; signalling only the parent leaves those workers
// holding GPU memory.
c.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
s.dropTo(c)
// CommandContext kills only the leader, so replace that with a group kill.
c.Cancel = func() error { return killGroup(c.Process.Pid, syscall.SIGKILL) }
@@ -298,6 +318,7 @@ func (s *Supervisor) spawn(ctx context.Context, sh *Shard, c engines.Command) (*
proc.Env = c.Env
proc.Dir = s.cfg.WorkDir
proc.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
s.dropTo(proc)
proc.Cancel = func() error { return killGroup(proc.Process.Pid, syscall.SIGKILL) }
stdout, err := proc.StdoutPipe()
@@ -779,3 +800,14 @@ func (s *Supervisor) KillOrphan(o Orphan) error {
}
return nil
}
// dropTo makes a command run as the engines' user, when there is one: its
// credential, its home as the working directory, and its caches there.
func (s *Supervisor) dropTo(c *exec.Cmd) {
if s.as == nil {
return
}
c.SysProcAttr.Credential = s.as.cred
c.Dir = s.as.home
c.Env = append(c.Env, s.as.env()...)
}
+11 -1
View File
@@ -19,10 +19,20 @@ func TestAnAgentUpgradesItselfAndSaysSoBeforeRestarting(t *testing.T) {
body := []byte("the new agent")
sum := sha256.Sum256(body)
asset := release.AssetName("v9.9.9", runtime.GOOS, runtime.GOARCH)
// Signed, as every release is: with a key this test's build trusts.
pub, priv, kerr := release.GenerateKey()
if kerr != nil {
t.Fatal(kerr)
}
t.Setenv("LLMGW_RELEASE_KEY", pub)
sums := hex.EncodeToString(sum[:]) + " " + asset + "\n"
sig, _ := release.Sign(priv, []byte(sums))
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/v9.9.9/checksums.txt":
_, _ = w.Write([]byte(hex.EncodeToString(sum[:]) + " " + asset + "\n"))
_, _ = w.Write([]byte(sums))
case "/v9.9.9/" + release.SignatureName:
_, _ = w.Write([]byte(sig))
case "/v9.9.9/" + asset:
_, _ = w.Write(body)
default:
+56
View File
@@ -0,0 +1,56 @@
package agent
import (
"context"
"log/slog"
"os/exec"
"testing"
"time"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
)
type alwaysReady struct{}
func (alwaysReady) Ready(context.Context, string, string, string) error { return nil }
func TestAnAdoptedEngineThatDiesIsReported(t *testing.T) {
sup, err := supervisor.New(supervisor.Config{WorkDir: t.TempDir(), RunAs: "-"}, alwaysReady{}, slog.New(slog.DiscardHandler))
if err != nil {
t.Fatal(err)
}
// An engine a previous agent started: not this process's child.
cmd := exec.Command("sleep", "60")
if err := cmd.Start(); err != nil {
t.Fatal(err)
}
go func() { _ = cmd.Wait() }()
sh, err := sup.Adopt(supervisor.Orphan{ShardID: "m-r0", ReplicaID: "m", PID: cmd.Process.Pid,
Spec: engines.ShardSpec{ReplicaID: "m", ShardID: "m-r0", HTTPPort: 1}, Alive: true})
if err != nil {
t.Fatal(err)
}
if err := sup.WaitReady(t.Context(), sh, "http://127.0.0.1:1", "m", 5*time.Second); err != nil {
t.Fatal(err)
}
a := &Agent{sup: sup, log: slog.New(slog.DiscardHandler)}
out := make(chan *llmgwv1.AgentMessage, 4)
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
go a.watchExits(ctx, out)
time.Sleep(1500 * time.Millisecond) // seen ready
_ = cmd.Process.Kill()
select {
case m := <-out:
s := m.GetShard()
if s.GetShardId() != "m-r0" || s.GetReplicaId() != "m" {
t.Errorf("reported %+v", s)
}
case <-time.After(10 * time.Second):
t.Fatal("the adopted engine died and nothing was reported")
}
}
+51 -3
View File
@@ -96,6 +96,9 @@ var (
ErrInvalid = errors.New("invalid api key")
ErrExpired = errors.New("api key expired")
ErrRevoked = errors.New("api key revoked")
// ErrThrottled is a client presenting more unknown keys than it is
// allowed to have looked up.
ErrThrottled = errors.New("too many unrecognised api keys from this address")
)
// Authenticator resolves keys and caches the result.
@@ -111,6 +114,10 @@ type Authenticator struct {
// and writing it per request would put a database write back on the path.
touchMu sync.Mutex
touch map[string]time.Time
// lookups rations the database reads of keys not in the cache, per
// client address.
lookups *buckets
}
type entry struct {
@@ -124,13 +131,22 @@ func New(s store.Store, cfg Config) *Authenticator {
cfg.normalize()
return &Authenticator{
store: s, cfg: cfg, now: time.Now,
cache: map[string]entry{},
touch: map[string]time.Time{},
cache: map[string]entry{},
touch: map[string]time.Time{},
lookups: newBuckets(20, 50),
}
}
// Authenticate resolves a presented credential.
func (a *Authenticator) Authenticate(ctx context.Context, presented string) (Principal, error) {
return a.AuthenticateFrom(ctx, presented, "")
}
// AuthenticateFrom resolves a credential presented by client, an address:
// a key not in the cache costs a database read, so those are rationed per
// client, and one that has made too many is told to slow down rather than
// read for. Empty client is not rationed.
func (a *Authenticator) AuthenticateFrom(ctx context.Context, presented, client string) (Principal, error) {
if presented == "" {
return Principal{}, ErrMissing
}
@@ -150,6 +166,9 @@ func (a *Authenticator) Authenticate(ctx context.Context, presented string) (Pri
return e.principal, e.err
}
if client != "" && !a.lookups.allow(client, a.now()) {
return Principal{}, ErrThrottled
}
p, err := a.resolve(ctx, hash)
if err != nil && !isRejection(err) {
// A database problem is not an authentication answer, so it must not be
@@ -185,6 +204,11 @@ func (a *Authenticator) resolve(ctx context.Context, hash []byte) (Principal, er
}
eff, err := a.store.Resolve(ctx, k.ID)
if errors.Is(err, store.ErrOwnerInactive) {
// Said as a revoked key is: whether the key exists is not the
// caller's to learn.
return Principal{}, ErrRevoked
}
if err != nil {
return Principal{}, err
}
@@ -206,10 +230,34 @@ func (a *Authenticator) lookup(id string) (entry, bool) {
return e, true
}
// maxCached bounds the cache. Every well-formed key anyone presents is an
// entry, a rejected one included, so without a bound a stream of made-up keys
// grows it until the process runs out of memory.
const maxCached = 100_000
func (a *Authenticator) store_(id string, e entry) {
a.mu.Lock()
defer a.mu.Unlock()
if len(a.cache) >= maxCached {
now := a.now()
for k, v := range a.cache {
if now.After(v.expires) {
delete(a.cache, k)
}
}
// Still full of live entries: half of them go, which costs only a
// lookup each for the keys in use.
if len(a.cache) >= maxCached {
n := 0
for k := range a.cache {
if n%2 == 0 {
delete(a.cache, k)
}
n++
}
}
}
a.cache[id] = e
a.mu.Unlock()
}
// Invalidate drops every cached decision.
+40
View File
@@ -438,3 +438,43 @@ func TestAlterAlwaysChangesTheKey(t *testing.T) {
t.Error("alter is a no-op when the key already ends in the replacement")
}
}
func TestAFloodOfMadeUpKeysFromOneAddressIsSlowedBeforeTheDatabase(t *testing.T) {
s, secret, _ := fixture(t)
a := New(s, DefaultConfig())
now := time.Now()
a.now = func() time.Time { return now }
ctx := context.Background()
throttled := 0
for i := 0; i < 200; i++ {
madeUp, _, _ := store.NewKey("x", "x", "proj1")
if _, err := a.AuthenticateFrom(ctx, madeUp, "203.0.113.9"); errors.Is(err, ErrThrottled) {
throttled++
}
}
if throttled < 100 {
t.Errorf("only %d of 200 made-up keys were held back", throttled)
}
// Another address, and the real key, are not caught by it.
if _, err := a.AuthenticateFrom(ctx, secret, "198.51.100.1"); err != nil {
t.Errorf("a real key from another address: %v", err)
}
// Time restores the allowance.
now = now.Add(5 * time.Second)
madeUp, _, _ := store.NewKey("y", "y", "proj1")
if _, err := a.AuthenticateFrom(ctx, madeUp, "203.0.113.9"); errors.Is(err, ErrThrottled) {
t.Error("still held back after waiting")
}
}
func TestTheCacheDoesNotGrowWithoutBound(t *testing.T) {
s, _, _ := fixture(t)
a := New(s, DefaultConfig())
for i := 0; i < maxCached+10; i++ {
a.store_(string(rune(i))+"-"+time.Now().String(), entry{expires: time.Now().Add(time.Hour)})
}
if n := len(a.cache); n > maxCached {
t.Errorf("%d entries", n)
}
}
+49
View File
@@ -0,0 +1,49 @@
package auth
import (
"sync"
"time"
)
// buckets are token buckets by key: rate a second, up to burst at once. Ones
// full and unused are dropped as more arrive, so a key is not kept for ever.
type buckets struct {
mu sync.Mutex
rate float64
burst float64
by map[string]*bucket
}
type bucket struct {
tokens float64
at time.Time
}
func newBuckets(rate, burst float64) *buckets {
return &buckets{rate: rate, burst: burst, by: map[string]*bucket{}}
}
// allow takes a token for key at now, reporting whether there was one.
func (b *buckets) allow(key string, now time.Time) bool {
b.mu.Lock()
defer b.mu.Unlock()
if len(b.by) > 50_000 {
for k, v := range b.by {
if now.Sub(v.at).Seconds()*b.rate >= b.burst {
delete(b.by, k)
}
}
}
k, ok := b.by[key]
if !ok {
k = &bucket{tokens: b.burst, at: now}
b.by[key] = k
}
k.tokens = min(b.burst, k.tokens+now.Sub(k.at).Seconds()*b.rate)
k.at = now
if k.tokens < 1 {
return false
}
k.tokens--
return true
}
+10 -1
View File
@@ -6,6 +6,8 @@ import (
"log/slog"
"net/http"
"strings"
"gitea.mynodes.xyz/ilker/llmgw/internal/clientip"
)
// Middleware authenticates every request before it reaches the router.
@@ -24,7 +26,7 @@ func Middleware(a *Authenticator, log *slog.Logger, exempt ...string) func(http.
return
}
p, err := a.Authenticate(r.Context(), BearerToken(r))
p, err := a.AuthenticateFrom(r.Context(), BearerToken(r), clientAddr(r))
if err != nil {
writeAuthError(w, err)
// The key prefix, never the key: a rejected credential in a log
@@ -66,6 +68,9 @@ func writeAuthError(w http.ResponseWriter, err error) {
switch {
case errors.Is(err, ErrMissing):
msg = "no api key was provided"
case errors.Is(err, ErrThrottled):
w.Header().Set("Retry-After", "1")
status, kind, msg = http.StatusTooManyRequests, "rate_limit_exceeded", "too many unrecognised api keys; slow down"
case errors.Is(err, ErrExpired):
msg = "this api key has expired"
case errors.Is(err, ErrRevoked):
@@ -104,3 +109,7 @@ func WriteForbiddenModel(w http.ResponseWriter, model string) {
"type": "invalid_request_error", "code": nil,
}})
}
// clientAddr is the address a request came from: through a trusted proxy,
// the client's.
func clientAddr(r *http.Request) string { return clientip.Addr(r) }
+100
View File
@@ -0,0 +1,100 @@
package cli
import (
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"io/fs"
"net"
"os"
"path/filepath"
"strings"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
)
// The admin token: what the admin API's /api and /debug paths demand, kept
// in a file only root and the server can read.
// defaultAdminTokenFile is where the server writes the token under its
// systemd unit — its state directory, the one place ProtectSystem=strict
// leaves it to write — and where clients look for it when told nothing.
const defaultAdminTokenFile = "/var/lib/llmgw/admin-token"
// adminTokenPath is the token's file for a configuration: as it says; under
// systemd, in the unit's state directory; else secrets/admin-token beside
// the configuration, for a server run by hand.
func adminTokenPath(cfgPath string, cfg config.Server) string {
if cfg.AdminTokenFile != "" {
return cfg.AdminTokenFile
}
if dir := os.Getenv("STATE_DIRECTORY"); dir != "" {
return filepath.Join(strings.Split(dir, ":")[0], "admin-token")
}
return filepath.Join(filepath.Dir(cfgPath), "secrets", "admin-token")
}
// ensureAdminToken reads the token, making it when there is none.
func ensureAdminToken(path string) (string, error) {
if b, err := os.ReadFile(path); err == nil { //nolint:gosec // the configured token file
if t := strings.TrimSpace(string(b)); len(t) >= 32 {
return t, nil
}
} else if !errors.Is(err, fs.ErrNotExist) {
return "", fmt.Errorf("reading the admin token %s: %w", path, err)
}
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
return "", fmt.Errorf("making %s: %w", filepath.Dir(path), err)
}
raw := make([]byte, 32)
if _, err := rand.Read(raw); err != nil {
return "", err
}
t := hex.EncodeToString(raw)
if err := os.WriteFile(path, []byte(t+"\n"), 0o600); err != nil {
return "", fmt.Errorf("writing the admin token %s: %w", path, err)
}
return t, nil
}
// readAdminToken is the token a client sends: LLMGW_ADMIN_TOKEN, else the
// first of these files it can read. Empty when none: the request then goes
// without, and the API says what is missing.
func readAdminToken(paths ...string) string {
if t := strings.TrimSpace(os.Getenv("LLMGW_ADMIN_TOKEN")); t != "" {
return t
}
for _, p := range append(paths, defaultAdminTokenFile) {
if p == "" {
continue
}
if b, err := os.ReadFile(p); err == nil { //nolint:gosec // the admin token file
if t := strings.TrimSpace(string(b)); t != "" {
return t
}
}
}
return ""
}
// checkAdminListen refuses an admin listener off loopback unless the
// configuration says it is meant.
func checkAdminListen(cfg config.Server) error {
host, _, err := net.SplitHostPort(cfg.AdminListen)
if err != nil {
return fmt.Errorf("admin_listen %q: %w", cfg.AdminListen, err)
}
if host == "localhost" {
return nil
}
if ip := net.ParseIP(host); ip != nil && ip.IsLoopback() {
return nil
}
if cfg.AdminRemote {
return nil
}
return fmt.Errorf("admin_listen %s is not loopback: the admin API adds models and upgrades nodes, and "+
"its token would cross the network in clear; keep it on 127.0.0.1 and reach it by ssh, or set "+
"admin_remote: true", cfg.AdminListen)
}
+52
View File
@@ -0,0 +1,52 @@
package cli
import (
"os"
"path/filepath"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
)
func TestTheAdminTokenIsMadeOnceReadableOnlyByItsOwner(t *testing.T) {
dir := t.TempDir()
cfgPath := filepath.Join(dir, "server.yaml")
p := adminTokenPath(cfgPath, config.Default())
if p != filepath.Join(dir, "secrets", "admin-token") {
t.Fatalf("path %s", p)
}
tok, err := ensureAdminToken(p)
if err != nil || len(tok) != 64 {
t.Fatalf("%q %v", tok, err)
}
if st, _ := os.Stat(p); st.Mode().Perm() != 0o600 {
t.Errorf("mode %v", st.Mode())
}
again, _ := ensureAdminToken(p)
if again != tok {
t.Error("a second start made a new token")
}
t.Setenv("LLMGW_ADMIN_TOKEN", "")
if got := readAdminToken(p); got != tok {
t.Errorf("read %q", got)
}
t.Setenv("LLMGW_ADMIN_TOKEN", "from-env")
if got := readAdminToken(p); got != "from-env" {
t.Errorf("env: %q", got)
}
}
func TestTheAdminListenerStaysOnLoopbackUnlessToldOtherwise(t *testing.T) {
cfg := config.Default()
for addr, ok := range map[string]bool{"127.0.0.1:9090": true, "localhost:9090": true, "[::1]:9090": true,
"0.0.0.0:9090": false, ":9090": false, "10.0.0.140:9090": false} {
cfg.AdminListen = addr
if err := checkAdminListen(cfg); (err == nil) != ok {
t.Errorf("%s: %v", addr, err)
}
}
cfg.AdminListen, cfg.AdminRemote = "10.0.0.140:9090", true
if err := checkAdminListen(cfg); err != nil {
t.Errorf("admin_remote: %v", err)
}
}
+37 -9
View File
@@ -15,6 +15,7 @@ import (
"strings"
"time"
agecrypt "filippo.io/age"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
@@ -44,9 +45,11 @@ func newBackupCmd() *cobra.Command {
"people, quotas, usage, the models added at runtime, the secret engine\n" +
"keys derive from — and /etc/llmgw, which holds the configuration, the\n" +
"store's address and the certificate authority's key.\n\n" +
"The archive holds that key, so it is written readable by root only. It\n" +
"is on the same disk as what it backs up: copy it elsewhere, or it does\n" +
"not survive the loss of this machine.\n\n" +
"The archive holds that key, so it is written readable by root only, and\n" +
"encrypted to backup_recipients in server.yaml when they are set (age\n" +
"recipients or SSH public keys; open it with `age -d -i <key>`). It is on\n" +
"the same disk as what it backs up: copy it elsewhere, or it does not\n" +
"survive the loss of this machine.\n\n" +
"Run daily by llmgw-backup.timer (`llmgw server backup`). The control plane publishes the time of\n" +
"the newest archive as llmgw_backup_last_success_timestamp_seconds, and\n" +
"BackupTooOld fires when it is more than a day old.",
@@ -55,7 +58,11 @@ func newBackupCmd() *cobra.Command {
if err != nil {
return err
}
path, err := writeBackup(cmd.Context(), cfg.Auth.Store, etcDir, dir)
to, err := config.ParseBackupRecipients(cfg.BackupRecipients)
if err != nil {
return err
}
path, err := writeBackup(cmd.Context(), cfg.Auth.Store, etcDir, dir, to)
if err != nil {
return err
}
@@ -87,7 +94,9 @@ const backupPrefix = "llmgw-backup-"
// writeBackup writes one archive and returns its path. Written beside its
// final name and renamed, so a backup interrupted halfway is never the newest
// one the control plane reports.
func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, error) {
//
// With recipients, the archive is encrypted to them, and named .tar.gz.age.
func writeBackup(ctx context.Context, storeAddr, etcDir, dir string, recipients []agecrypt.Recipient) (string, error) {
// Listable by anyone, so the control plane — not root — can see when the
// newest archive was written. The archives themselves are root's alone.
if err := os.MkdirAll(dir, 0o755); err != nil {
@@ -95,6 +104,9 @@ func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, er
}
host, _ := os.Hostname()
name := backupPrefix + host + "-" + time.Now().UTC().Format("20060102T150405Z") + ".tar.gz"
if len(recipients) > 0 {
name += ".age"
}
final := filepath.Join(dir, name)
part := final + ".part"
@@ -108,7 +120,13 @@ func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, er
_ = os.Remove(part)
}
}()
gz := gzip.NewWriter(f)
var sealed io.WriteCloser = nopCloser{f}
if len(recipients) > 0 {
if sealed, err = agecrypt.Encrypt(f, recipients...); err != nil {
return "", err
}
}
gz := gzip.NewWriter(sealed)
tw := tar.NewWriter(gz)
manifest := fmt.Sprintf("llmgw %s\nhost %s\ntime %s\n", version.Version, host,
@@ -124,7 +142,7 @@ func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, er
if err := addTree(tw, etcDir, "etc/llmgw"); err != nil {
return "", fmt.Errorf("%s: %w", etcDir, err)
}
if err := errors.Join(tw.Close(), gz.Close(), f.Sync(), f.Close()); err != nil {
if err := errors.Join(tw.Close(), gz.Close(), sealed.Close(), f.Sync(), f.Close()); err != nil {
return "", err
}
if err := os.Rename(part, final); err != nil {
@@ -134,6 +152,16 @@ func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, er
return final, nil
}
type nopCloser struct{ io.Writer }
func (nopCloser) Close() error { return nil }
// isBackup reports whether a file name is an archive this command wrote.
func isBackup(name string) bool {
return strings.HasPrefix(name, backupPrefix) &&
(strings.HasSuffix(name, ".tar.gz") || strings.HasSuffix(name, ".tar.gz.age"))
}
// addStore dumps the store into the archive.
func addStore(ctx context.Context, tw *tar.Writer, addr, scratch string) error {
addr, err := store.ResolveAddress(addr)
@@ -280,7 +308,7 @@ func pruneBackups(dir string, keep int) (int, error) {
}
var names []string
for _, e := range entries {
if n := e.Name(); strings.HasPrefix(n, backupPrefix) && strings.HasSuffix(n, ".tar.gz") {
if n := e.Name(); isBackup(n) {
names = append(names, n)
}
}
@@ -306,7 +334,7 @@ func NewestBackup(dir string) time.Time {
}
var newest time.Time
for _, e := range entries {
if n := e.Name(); !strings.HasPrefix(n, backupPrefix) || !strings.HasSuffix(n, ".tar.gz") {
if n := e.Name(); !isBackup(n) {
continue
}
if info, err := e.Info(); err == nil && info.ModTime().After(newest) {
+78 -2
View File
@@ -3,6 +3,7 @@ package cli
import (
"archive/tar"
"compress/gzip"
"crypto/ed25519"
"io"
"os"
"path/filepath"
@@ -10,9 +11,84 @@ import (
"testing"
"time"
agecrypt "filippo.io/age"
"golang.org/x/crypto/ssh"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
func TestABackupIsEncryptedToItsRecipients(t *testing.T) {
tmp := t.TempDir()
etc := filepath.Join(tmp, "etc")
_ = os.MkdirAll(filepath.Join(etc, "pki"), 0o700)
_ = os.WriteFile(filepath.Join(etc, "pki", "ca-key.pem"), []byte("THE-CA-KEY"), 0o600)
id, err := agecrypt.GenerateX25519Identity()
if err != nil {
t.Fatal(err)
}
to, err := config.ParseBackupRecipients([]string{id.Recipient().String()})
if err != nil {
t.Fatal(err)
}
out := filepath.Join(tmp, "backups")
path, err := writeBackup(t.Context(), "", etc, out, to)
if err != nil {
t.Fatal(err)
}
if !strings.HasSuffix(path, ".tar.gz.age") {
t.Errorf("named %s", filepath.Base(path))
}
raw, _ := os.ReadFile(path)
if strings.Contains(string(raw), "THE-CA-KEY") {
t.Fatal("the CA key is in the archive in the clear")
}
f, _ := os.Open(path)
defer f.Close()
plain, err := agecrypt.Decrypt(f, id)
if err != nil {
t.Fatalf("the recipient cannot open it: %v", err)
}
gz, err := gzip.NewReader(plain)
if err != nil {
t.Fatal(err)
}
tr := tar.NewReader(gz)
found := false
for {
h, err := tr.Next()
if err != nil {
break
}
found = found || h.Name == "etc/llmgw/pki/ca-key.pem"
}
if !found {
t.Error("the opened archive lacks the CA key")
}
// Seen and pruned like any other.
if NewestBackup(out).IsZero() {
t.Error("the control plane would not see an encrypted archive")
}
}
func TestBackupRecipientsMustBeKeys(t *testing.T) {
if _, err := config.ParseBackupRecipients([]string{"not a key"}); err == nil {
t.Error("nonsense was taken as a recipient")
}
pub, _, _ := ed25519.GenerateKey(nil)
sp, err := ssh.NewPublicKey(pub)
if err != nil {
t.Fatal(err)
}
if _, err := config.ParseBackupRecipients([]string{
strings.TrimSpace(string(ssh.MarshalAuthorizedKey(sp))) + " ops@example",
}); err != nil {
t.Errorf("an SSH key was refused: %v", err)
}
}
func TestABackupHoldsTheStoreAndTheConfiguration(t *testing.T) {
tmp := t.TempDir()
dbPath := filepath.Join(tmp, "store.db")
@@ -32,7 +108,7 @@ func TestABackupHoldsTheStoreAndTheConfiguration(t *testing.T) {
_ = os.WriteFile(filepath.Join(etc, "pki", "ca-key.pem"), []byte("KEY"), 0o600)
out := filepath.Join(tmp, "backups")
path, err := writeBackup(t.Context(), dbPath, etc, out)
path, err := writeBackup(t.Context(), dbPath, etc, out, nil)
if err != nil {
t.Fatal(err)
}
@@ -129,7 +205,7 @@ func TestNoScratchCopyOfTheStoreIsLeftBehind(t *testing.T) {
}
_ = db.Close()
out := filepath.Join(tmp, "backups")
if _, err := writeBackup(t.Context(), dbPath, t.TempDir(), out); err != nil {
if _, err := writeBackup(t.Context(), dbPath, t.TempDir(), out, nil); err != nil {
t.Fatal(err)
}
entries, _ := os.ReadDir(out)
+19 -8
View File
@@ -19,17 +19,22 @@ import (
// ctlClient talks to a server's admin API.
type ctlClient struct {
base string
http *http.Client
base string
http *http.Client
token string
}
func newCtlClient(addr string) *ctlClient {
// newCtlClient talks to the admin API at addr, with the admin token from
// LLMGW_ADMIN_TOKEN or the first of tokenFiles it can read, then the usual
// place.
func newCtlClient(addr string, tokenFiles ...string) *ctlClient {
if !strings.HasPrefix(addr, "http://") && !strings.HasPrefix(addr, "https://") {
addr = "http://" + addr
}
return &ctlClient{
base: strings.TrimRight(addr, "/"),
http: &http.Client{Timeout: 15 * time.Second},
base: strings.TrimRight(addr, "/"),
http: &http.Client{Timeout: 15 * time.Second},
token: readAdminToken(tokenFiles...),
}
}
@@ -75,6 +80,9 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
// Who made the change, kept with a stored model so a catalog that grew
// over months can say where each entry came from.
req.Header.Set("X-Llmgw-Actor", actor())
if c.token != "" {
req.Header.Set("Authorization", "Bearer "+c.token)
}
resp, err := c.http.Do(req)
if err != nil {
@@ -104,8 +112,9 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
func newCtlCmd() *cobra.Command {
var (
server string
asJSON bool
server string
asJSON bool
tokenFile string
)
cmd := &cobra.Command{
@@ -115,9 +124,11 @@ func newCtlCmd() *cobra.Command {
"default) rather than the client-facing port.",
}
cmd.PersistentFlags().StringVar(&server, "server", "127.0.0.1:9090", "admin API address")
cmd.PersistentFlags().StringVar(&tokenFile, "token-file", "",
"the admin token's file (default: LLMGW_ADMIN_TOKEN, else "+defaultAdminTokenFile+")")
cmd.PersistentFlags().BoolVar(&asJSON, "json", false, "emit JSON")
client := func() *ctlClient { return newCtlClient(server) }
client := func() *ctlClient { return newCtlClient(server, tokenFile) }
cmd.AddCommand(
newNodesCmd(client, &asJSON),
+27 -18
View File
@@ -281,6 +281,13 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
if len(clients) > 0 && !tlsc.Enabled() {
return fmt.Errorf("--api-client names a certificate, and needs --tls-ca, --tls-cert and --tls-key")
}
if tlsc.Enabled() && len(clients) == 0 {
// The same root signs every node's certificate, so without a
// name any node — a removed one's included — could delete
// models here.
return fmt.Errorf("with --tls-ca, name the control plane's certificate with --api-client: " +
"the authority signs every node's certificate too")
}
out := cmd.OutOrStdout()
log := slog.Default()
@@ -293,20 +300,20 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
// http.Dir refuses paths that escape the root, which is the whole
// traversal question answered by the standard library rather than
// by me.
files := &http.Server{
Addr: listen,
Handler: readOnly(withManifest(*dir, http.FileServer(http.Dir(*dir)))),
ReadHeaderTimeout: 10 * time.Second,
}
files := withLimits(&http.Server{
Addr: listen,
Handler: readOnly(withManifest(*dir, http.FileServer(http.Dir(*dir)))),
ReadTimeout: time.Minute,
})
allowed, err := parseAllow(allow)
if err != nil {
return err
}
admin := &http.Server{
Addr: apiListen,
Handler: onlyClients(clients, apiGate(tlsc.Enabled(), allowed, api.Handler())),
ReadHeaderTimeout: 10 * time.Second,
}
admin := withLimits(&http.Server{
Addr: apiListen,
Handler: onlyClients(clients, apiGate(tlsc.Enabled(), allowed, api.Handler())),
ReadTimeout: time.Minute,
})
errs := make(chan error, 2)
go func() { errs <- fmt.Errorf("serving weights: %w", files.ListenAndServe()) }()
@@ -315,13 +322,7 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
go func() { errs <- fmt.Errorf("serving the API: %w", serveAPI(admin, tlsc)) }()
switch {
case tlsc.Enabled():
fmt.Fprintf(out, "management API on %s, mutual TLS against %s\n", apiListen, tlsc.CA)
if len(clients) == 0 {
// The same root signs every node's certificate, so without a
// name any node could delete models here.
log.Warn("the management API takes any certificate from " + tlsc.CA +
", every node's included; name the control plane's with --api-client")
}
fmt.Fprintf(out, "management API on %s, mutual TLS for %s\n", apiListen, strings.Join(clients, ", "))
case len(allowed) > 0:
fmt.Fprintf(out, "management API on %s, for %s and this machine only\n", apiListen, strings.Join(allow, ", "))
default:
@@ -346,7 +347,7 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
c.Flags().StringArrayVar(&allow, "api-allow", nil,
"an address or network the management API answers, besides this machine; repeatable (default: this machine only)")
c.Flags().StringArrayVar(&clients, "api-client", nil,
"with mutual TLS, the common name of a certificate the management API answers; repeatable (default: any from --tls-ca)")
"with mutual TLS, the common name of a certificate the management API answers; repeatable; required with --tls-ca")
c.Flags().BoolVar(&allowPrivate, "allow-private-sources", false,
"let downloads reach loopback, private and link-local addresses")
c.Flags().StringVar(&tlsc.CA, "tls-ca", "", "CA bundle that management clients must present a certificate from")
@@ -387,6 +388,14 @@ func readOnly(h http.Handler) http.Handler {
http.Error(w, "the mirror is read-only", http.StatusMethodNotAllowed)
return
}
// The mirror's own bookkeeping — a file's checksum, the URL it came
// from — sits beside it under a dot, and is nobody's to download.
for _, seg := range strings.Split(r.URL.Path, "/") {
if strings.HasPrefix(seg, ".") {
http.NotFound(w, r)
return
}
}
h.ServeHTTP(w, r)
})
}
+21
View File
@@ -72,3 +72,24 @@ func TestAVersionAnswersHeadAsWellAsGet(t *testing.T) {
t.Errorf("expected a manifest, got %q", got)
}
}
func TestTheMirrorDoesNotServeItsBookkeeping(t *testing.T) {
root := t.TempDir()
dir := filepath.Join(root, "llama", "v1")
_ = os.MkdirAll(dir, 0o755)
_ = os.WriteFile(filepath.Join(dir, "m.gguf"), []byte("x"), 0o644)
_ = os.WriteFile(filepath.Join(dir, ".m.gguf.source"), []byte("https://example.com/m.gguf\n"), 0o644)
h := readOnly(withManifest(root, http.FileServer(http.Dir(root))))
for path, want := range map[string]int{
"/llama/v1/m.gguf": http.StatusOK,
"/llama/v1/.m.gguf.source": http.StatusNotFound,
"/llama/.hidden/x": http.StatusNotFound,
} {
w := httptest.NewRecorder()
h.ServeHTTP(w, httptest.NewRequest(http.MethodGet, path, nil))
if w.Code != want {
t.Errorf("%s: %d, expected %d", path, w.Code, want)
}
}
}
+44 -1
View File
@@ -6,6 +6,7 @@ import (
"fmt"
"io"
"net"
"net/netip"
"os"
"runtime"
"strings"
@@ -13,7 +14,9 @@ import (
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
)
@@ -26,6 +29,7 @@ func newMirrorInitCmd(dir *string) *cobra.Command {
var (
controlPlanes []string
base string
noFirewall bool
)
cmd := &cobra.Command{
Use: "init",
@@ -56,11 +60,19 @@ Safe to run again: it changes only what differs.`,
if _, err := parseAllow(controlPlanes); err != nil {
return err
}
return mirrorInit(cmd.Context(), cmd.OutOrStdout(), *dir, base, controlPlanes)
if err := mirrorInit(cmd.Context(), cmd.OutOrStdout(), *dir, base, controlPlanes); err != nil {
return err
}
if !noFirewall {
mirrorFirewall(cmd.Context(), cmd.OutOrStdout(), controlPlanes)
}
return nil
},
}
cmd.Flags().StringArrayVar(&controlPlanes, "control-plane", nil,
"address of the control plane that manages this mirror; repeatable")
cmd.Flags().BoolVar(&noFirewall, "no-firewall", false,
"leave the firewall alone, instead of keeping the mirror's ports to the cluster's networks")
cmd.Flags().StringVar(&base, "base", "",
"the address nodes fetch from, http://host:8081 (default: this machine's first address)")
return cmd
@@ -123,3 +135,34 @@ func mirrorInit(ctx context.Context, out io.Writer, dir, base string, controlPla
fmt.Fprintf(out, "\nThe mirror is up. On the control plane:\n\n sudo llmgw server init --mirror %s\n\n", api)
return nil
}
// mirrorFirewall keeps the mirror's ports to the networks it reaches the
// control planes on: its files are for the cluster's nodes, its API for the
// control planes, which the API checks again by address.
func mirrorFirewall(ctx context.Context, out io.Writer, controlPlanes []string) {
allowed, _ := parseAllow(controlPlanes)
var toward []string
var nets []netip.Prefix
for _, p := range allowed {
if p.IsSingleIP() {
toward = append(toward, p.Addr().String())
} else {
nets = append(nets, p)
}
}
found, err := firewall.Networks(ctx, pgha.Local{}, toward)
if err == nil {
var changed bool
changed, err = firewall.Install(ctx, pgha.Local{}, firewall.Spec{
Ports: []firewall.Port{firewall.MirrorFiles, firewall.MirrorAPI, firewall.NodeExporter},
Allow: append(found, nets...),
})
if err == nil {
if changed {
fmt.Fprintln(out, " ok firewall: the mirror's ports answer the cluster's networks only")
}
return
}
}
fmt.Fprintf(out, " !! firewall: %v; the mirror's ports answer every network\n", err)
}
+15 -2
View File
@@ -14,6 +14,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
@@ -122,6 +123,7 @@ engines are kept.`,
f.StringVar(&na.opts.Releases, "releases", "https://gitea.mynodes.xyz/ilker/llmgw/releases/download", "where release files are downloaded from")
f.BoolVar(&na.opts.IgnoreChecks, "ignore-checks", false, "add the node even when doctor finds a problem")
f.BoolVar(&na.opts.NoNodeExporter, "no-node-exporter", false, "leave Prometheus' node exporter off the machine")
f.BoolVar(&na.opts.NoFirewall, "no-firewall", false, "leave the machine's firewall alone, instead of keeping llmgw's ports to the cluster's networks")
f.StringArrayVar(&na.opts.DisableEngines, "disable-engine", nil, "an engine this node never installs, e.g. vllm on GPUs it does not run on; repeatable")
f.StringVar(&na.admin, "admin", "", "the control plane's admin API, to watch the node join (default: from server.yaml, else 127.0.0.1:9090)")
f.StringVar(&na.serverCfg, "config", "/etc/llmgw/server.yaml", "the control plane's configuration, for the admin API's address")
@@ -168,6 +170,12 @@ func (na *nodeAdd) prepare() error {
}
}
}
// The node's ports — its engines', its metrics — for the cluster only.
if cfg, err := config.Load(na.serverCfg); err == nil && !o.NoFirewall {
from, to := cfg.EnginePorts()
o.FirewallPorts = []firewall.Port{firewall.Engines(from, to), firewall.NodeExporter}
o.FirewallAllow, _ = cfg.FirewallAllow()
}
o.Version = version.Version
if o.Binary == "" {
exe, err := os.Executable()
@@ -301,16 +309,21 @@ func (na *nodeAdd) runPlain(ctx context.Context, o onboard.Options, title string
// this machine has it.
func (na *nodeAdd) adminAPI() *ctlClient {
addr := na.admin
token := ""
cfg, err := config.Load(na.serverCfg)
if err == nil {
token = adminTokenPath(na.serverCfg, cfg)
}
if addr == "" {
addr = "127.0.0.1:9090"
if cfg, err := config.Load(na.serverCfg); err == nil && cfg.AdminListen != "" {
if err == nil && cfg.AdminListen != "" {
addr = cfg.AdminListen
if strings.HasPrefix(addr, ":") {
addr = "127.0.0.1" + addr
}
}
}
return newCtlClient(addr)
return newCtlClient(addr, token)
}
// errNoAdmin is the watch giving up because the admin API is not here, which
+87 -13
View File
@@ -16,6 +16,7 @@ import (
"syscall"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/clientip"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
@@ -26,6 +27,7 @@ import (
"github.com/spf13/cobra"
"golang.org/x/net/http2"
"golang.org/x/net/http2/h2c"
"golang.org/x/net/netutil"
"golang.org/x/sync/errgroup"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
@@ -194,6 +196,12 @@ func newServerCmd() *cobra.Command {
// Patroni. One host is a single PostgreSQL, which has none.
if addr, err := storepkg.ResolveAddress(cfg.Auth.Store); err == nil && storepkg.IsPostgres(addr) {
if hosts, _, _, err := storepkg.SplitPostgres(strings.TrimSpace(addr)); err == nil && len(hosts) > 1 {
if cfg.AgentTLS.CA != "" {
// Members' APIs, once on TLS, verified against it.
if err := pgha.SetRESTCA(cfg.AgentTLS.CA); err != nil {
log.Warn("the store's API cannot be verified", "err", err)
}
}
storeHA = pgha.NewCollector(hosts)
if err := rt.RegisterMetrics(storeHA); err != nil {
return err
@@ -234,6 +242,10 @@ func newServerCmd() *cobra.Command {
}
adminAPI = api
api.SetReleases(cfg.Releases)
if st := rt.Store(); st != nil {
// Removing a node revokes its certificate.
api.SetRevoker(st.RevokeNode)
}
// Catalog changes need somewhere to persist, so this is wired
// only when there is a store. Without one, the endpoints say so
// rather than accepting a model that vanishes on restart.
@@ -348,21 +360,39 @@ func newServerCmd() *cobra.Command {
"api", cfg.Mirror.API, "mtls", cfg.Mirror.TLS.Enabled())
}
apiSrv := &http.Server{
Handler: rt.Handler(),
ReadHeaderTimeout: 10 * time.Second,
apiSrv := withLimits(&http.Server{
Handler: rt.Handler(),
// Reading the request: long enough for a body up to
// max_body_bytes on a slow link, short enough that a client
// trickling one cannot hold a connection and its buffer.
ReadTimeout: 5 * time.Minute,
// No write timeout: responses stream for minutes and the proxy
// enforces its own per-phase deadlines.
})
if err := checkAdminListen(cfg); err != nil {
return err
}
adminSrv := &http.Server{
Handler: router.AdminGuard(adminMux),
ReadHeaderTimeout: 10 * time.Second,
if bad := clientip.SetTrusted(cfg.Router.TrustedProxies); len(bad) > 0 {
return fmt.Errorf("router.trusted_proxies: %q is not an address or a CIDR prefix", bad[0])
}
adminToken, err := ensureAdminToken(adminTokenPath(configPath, cfg))
if err != nil {
return err
}
adminSrv := withLimits(&http.Server{
Handler: router.AdminGuard(router.AdminAuth(adminToken, cfg.DebugPprof, adminMux)),
ReadTimeout: time.Minute,
})
apiLn, err := net.Listen("tcp", cfg.Listen)
if err != nil {
return fmt.Errorf("listening on %s: %w", cfg.Listen, err)
}
if cfg.MaxConnections > 0 {
// Past this, new connections wait in the kernel's backlog
// rather than each taking a goroutine and its buffers.
apiLn = netutil.LimitListener(apiLn, cfg.MaxConnections)
}
adminLn, err := net.Listen("tcp", cfg.AdminListen)
if err != nil {
_ = apiLn.Close()
@@ -395,7 +425,7 @@ func newServerCmd() *cobra.Command {
mux.Handle("/", h)
h = mux
}
metricsSrv = &http.Server{Handler: h, ReadHeaderTimeout: 10 * time.Second}
metricsSrv = withLimits(&http.Server{Handler: h, ReadTimeout: 30 * time.Second})
go func() {
if err := metricsSrv.Serve(metricsLn); err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Error("metrics listener stopped", "err", err)
@@ -416,10 +446,9 @@ func newServerCmd() *cobra.Command {
if agentHandler != nil {
mux := http.NewServeMux()
mux.Handle(agentPath, agentHandler)
agentSrv = &http.Server{
Handler: mux,
ReadHeaderTimeout: 10 * time.Second,
}
// No read timeout: an agent's stream is one request that
// lasts as long as the agent does.
agentSrv = withLimits(&http.Server{Handler: mux})
if !cfg.AgentTLS.Enabled() {
// Without TLS, agents speak plaintext HTTP/2, which Go will
// not do unless asked. Nothing then stops a node from
@@ -427,10 +456,20 @@ func newServerCmd() *cobra.Command {
// network only.
agentSrv.Handler = h2c.NewHandler(mux, &http2.Server{})
}
if agentLn, err = net.Listen("tcp", cfg.AgentListen); err != nil {
agentAddr, aerr := agentListenAddr(cfg)
if aerr != nil {
_ = apiLn.Close()
_ = adminLn.Close()
return fmt.Errorf("listening on %s: %w", cfg.AgentListen, err)
return aerr
}
if !cfg.AgentTLS.Enabled() {
log.Warn("the agent port has no TLS (agent_insecure): agents are not authenticated and "+
"traffic is plaintext; use only on one machine or a trusted network", "listen", agentAddr)
}
if agentLn, err = net.Listen("tcp", agentAddr); err != nil {
_ = apiLn.Close()
_ = adminLn.Close()
return fmt.Errorf("listening on %s: %w", agentAddr, err)
}
if cfg.AgentTLS.Enabled() {
tc, terr := cfg.AgentTLS.ServerConfig()
@@ -586,6 +625,8 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
return urls, err
})
go svc.WatchControlPlanes(ctx, 30*time.Second)
// A removed node's certificate is refused.
svc.SetRevocations(st.NodeRevokedAt)
}
fwd.set(svc)
// Every InfiniBand fabric mapped every five minutes, through one node.
@@ -761,3 +802,36 @@ func servableCheck(llamaVersion string) func(context.Context, catalog.Model) err
return nil
}
}
// errAgentInsecure is a control plane asked to open its agent port with no
// TLS and not told that is meant.
var errAgentInsecure = errors.New("agent_tls is not set, so the agent port would be plaintext and anyone " +
"reaching it could register as a node: set agent_tls (llmgw server init writes it), or " +
"agent_insecure: true for one machine or a test")
// agentListenAddr is where the agent port listens: agent_listen with TLS;
// without it, refused unless agent_insecure says so, and then on loopback
// when agent_listen names only a port.
func agentListenAddr(cfg config.Server) (string, error) {
if cfg.AgentTLS.Enabled() {
return cfg.AgentListen, nil
}
// agent_tls.insecure says the same, in the older place.
if !cfg.AgentInsecure && !cfg.AgentTLS.Insecure {
return "", errAgentInsecure
}
if host, port, err := net.SplitHostPort(cfg.AgentListen); err == nil && host == "" {
return net.JoinHostPort("127.0.0.1", port), nil
}
return cfg.AgentListen, nil
}
// withLimits gives a server what every listener here wants: a bound on how
// long the headers may take, how long a connection may sit idle between
// requests, and how large the headers may be.
func withLimits(s *http.Server) *http.Server {
s.ReadHeaderTimeout = 10 * time.Second
s.IdleTimeout = 2 * time.Minute
s.MaxHeaderBytes = 1 << 20
return s
}
+33
View File
@@ -0,0 +1,33 @@
package cli
import (
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
)
func TestTheAgentPortOpensWithoutTLSOnlyWhenToldAndThenOnLoopback(t *testing.T) {
cfg := config.Default()
if _, err := agentListenAddr(cfg); err == nil {
t.Error("a control plane with no agent_tls opened its agent port")
}
cfg.AgentInsecure = true
if a, _ := agentListenAddr(cfg); a != "127.0.0.1:7443" {
t.Errorf("insecure, port only: %s", a)
}
cfg.AgentListen = "10.0.0.5:7443"
if a, _ := agentListenAddr(cfg); a != "10.0.0.5:7443" {
t.Errorf("insecure, an address named: %s", a)
}
cfg.AgentInsecure = false
cfg.AgentListen = ":7443"
cfg.AgentTLS = tlsconf.Config{Insecure: true}
if a, err := agentListenAddr(cfg); err != nil || a != "127.0.0.1:7443" {
t.Errorf("agent_tls.insecure: %s %v", a, err)
}
cfg.AgentTLS = tlsconf.Config{CA: "ca.pem", Cert: "s.pem", Key: "s-key.pem"}
if a, err := agentListenAddr(cfg); err != nil || a != ":7443" {
t.Errorf("with TLS: %s %v", a, err)
}
}
+47 -1
View File
@@ -20,6 +20,7 @@ import (
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
@@ -38,6 +39,8 @@ type serverAdd struct {
vip string
remote onboard.Remote
// noFirewall leaves the new control plane's firewall as it is.
noFirewall bool
// addr is the machine's address as given; ip what the others reach it at.
addr, ip, name string
out io.Writer
@@ -78,6 +81,8 @@ func newServerAddCmd() *cobra.Command {
f.StringArrayVar(&sa.hosts, "host", nil, "another name for the new control plane's certificate; repeatable")
f.StringVar(&sa.vip, "vip", "", "a floating address for whichever control plane leads, as 10.0.0.160/24")
f.StringVar(&sa.pkiDir, "pki", sa.pkiDir, "directory holding the certificate authority")
f.BoolVar(&sa.noFirewall, "no-firewall", false,
"leave its firewall alone, instead of keeping llmgw's internal ports to the cluster's networks")
return cmd
}
@@ -121,6 +126,12 @@ func (sa *serverAdd) run(ctx context.Context) error {
{"store", sa.storeAccess},
{"service", sa.service},
}
if !sa.noFirewall {
steps = append(steps, struct {
name string
fn func(context.Context) error
}{"firewall", sa.firewall})
}
if sa.vip != "" {
steps = append(steps, struct {
name string
@@ -136,6 +147,36 @@ func (sa *serverAdd) run(ctx context.Context) error {
return nil
}
// firewall keeps the new control plane's internal ports to the networks it
// reaches this one on. Not fatal: it serves without one.
func (sa *serverAdd) firewall(ctx context.Context) error {
cfg, err := config.Load(filepath.Join(sa.etcDir, "server.yaml"))
if err != nil {
return err
}
self, err := sourceIPToward(sa.ip)
if err != nil {
return err
}
nets, err := firewall.Networks(ctx, sa.remote, []string{self})
if err == nil {
extra, _ := cfg.FirewallAllow()
nets = append(nets, extra...)
var changed bool
if changed, err = firewall.Install(ctx, sa.remote,
firewall.Spec{Ports: controlPlanePorts(cfg), Allow: nets}); err == nil {
if changed {
sa.done("firewall: llmgw's internal ports answer the cluster's networks only")
} else {
sa.kept("firewall")
}
return nil
}
}
fmt.Fprintf(sa.out, " !! firewall: %v\n", err)
return nil
}
func (sa *serverAdd) connect(ctx context.Context) error {
out, err := sa.sh(ctx, "hostname; sudo -n true && echo sudo-ok", nil)
if err != nil {
@@ -307,7 +348,12 @@ func (sa *serverAdd) storeAccess(ctx context.Context) error {
case slices.Contains(hosts, sa.ip):
sa.kept("store: %s is one of its members", sa.ip)
case len(hosts) > 1:
changed, err := pgha.AllowClient(ctx, hosts, sa.ip)
rest := ""
if b, err := os.ReadFile(patroniRestFile(sa.etcDir)); err == nil { //nolint:gosec // our own secret
rest = strings.TrimSpace(string(b))
}
_ = pgha.SetRESTCA(filepath.Join(sa.etcDir, "ca.pem"))
changed, err := pgha.AllowClient(ctx, hosts, sa.ip, rest)
if err != nil {
return err
}
+14 -1
View File
@@ -81,6 +81,9 @@ type serverInit struct {
// make it serve something: a mirror, this machine as a node, and a small
// model on it.
noMirror, noNode, noModel bool
// noFirewall leaves this machine's firewall, and the store members',
// as they are.
noFirewall bool
// mirrorURL is a mirror on another machine to use instead of running one.
mirrorURL string
// pgSetup prepares PostgreSQL and returns the store address; a seam for
@@ -161,6 +164,8 @@ func newServerInitCmd() *cobra.Command {
"use the mirror on another machine, set up there by 'llmgw mirror init': its management API, http://host:8082")
f.BoolVar(&in.noNode, "no-node", false, "do not make this machine a node (and serve no model on it)")
f.BoolVar(&in.noModel, "no-model", false, "do not fetch and serve the starter model ("+starterName+")")
f.BoolVar(&in.noFirewall, "no-firewall", false,
"leave the firewall alone, instead of keeping llmgw's internal ports to the cluster's networks")
f.BoolVar(&in.postgres, "postgres", false,
"keep the store in PostgreSQL on this machine, installing it and moving an existing SQLite store into it")
f.StringArrayVar(&in.patroni, "patroni", nil,
@@ -221,6 +226,12 @@ func (in *serverInit) run(ctx context.Context) error {
return fmt.Errorf("starter model: %w", err)
}
}
if hasSystemd && !in.noFirewall {
if err := in.ensureFirewall(ctx); err != nil {
// The cluster works without it; said, not fatal.
fmt.Fprintf(in.out, " !! firewall: %v; llmgw's ports answer every network this machine is on\n", err)
}
}
in.nextSteps()
return nil
}
@@ -480,7 +491,9 @@ control:
auth:
enabled: true
store: "%s" # a SQLite path, or @file holding a PostgreSQL address
signup_open: true
# The signup form: accounts it makes wait for an administrator's approval,
# but anyone reaching port 8080 can make them. Off until you want it.
signup_open: false
# Applied to any key no scope gives a limit to. Without them, a project
# created without limits is metered by nothing at all.
default_rpm: 60
+7 -4
View File
@@ -552,11 +552,14 @@ func (in *serverInit) waitForStarter(ctx context.Context) error {
}
func (in *serverInit) adminAPI() *ctlClient {
addr := "127.0.0.1:9090"
if cfg, err := config.Load(in.configPath()); err == nil && cfg.AdminListen != "" {
addr = cfg.AdminListen
addr, token := "127.0.0.1:9090", ""
if cfg, err := config.Load(in.configPath()); err == nil {
token = adminTokenPath(in.configPath(), cfg)
if cfg.AdminListen != "" {
addr = cfg.AdminListen
}
}
return newCtlClient(addr)
return newCtlClient(addr, token)
}
// waitFor polls until done says so, fails, or the time runs out.
+124
View File
@@ -0,0 +1,124 @@
package cli
import (
"context"
"fmt"
"net"
"net/netip"
"os"
"strconv"
"strings"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
)
// ensureFirewall keeps this control plane's internal ports to the networks
// the cluster is on: the agents' stream, the metrics, the mirror's files,
// the engines and etcd when they are here. The client API is not among
// them; it is meant to be reached.
func (in *serverInit) ensureFirewall(ctx context.Context) error {
cfg, err := config.Load(in.configPath())
if err != nil {
return err
}
ports := controlPlanePorts(cfg)
if !in.noMirror && in.mirrorURL == "" {
ports = append(ports, firewall.MirrorFiles)
}
if !in.noNode {
from, to := cfg.EnginePorts()
ports = append(ports, firewall.Engines(from, to), firewall.NodeExporter)
}
if _, err := os.Stat("/etc/default/etcd"); err == nil {
// The store's third etcd vote.
ports = append(ports, firewall.EtcdClient, firewall.EtcdPeer)
}
self, err := ipv4Of(in.reachableAddr())
if err != nil {
return err
}
nets, err := firewall.Networks(ctx, pgha.Local{}, []string{self})
if err != nil {
return err
}
extra, _ := cfg.FirewallAllow()
nets = append(nets, extra...)
changed, err := firewall.Install(ctx, pgha.Local{}, firewall.Spec{Ports: ports, Allow: nets})
if err != nil {
return err
}
shown := make([]string, len(nets))
for i, n := range nets {
shown[i] = n.String()
}
if changed {
in.done("firewall: llmgw's internal ports answer %s only (%s)", strings.Join(shown, ", "), firewall.RulesFile)
} else {
in.kept("firewall: llmgw's internal ports answer %s only", strings.Join(shown, ", "))
}
return nil
}
// storeFirewall keeps each store member's database, Patroni and etcd ports to
// the networks it reaches this control plane and the other members on. A
// member it cannot be set on is said, and left as it was.
func (in *serverInit) storeFirewall(ctx context.Context, members []pgha.Node, self string, extra []netip.Prefix) {
ports := []firewall.Port{firewall.Postgres, firewall.PatroniAPI, firewall.EtcdClient, firewall.EtcdPeer,
firewall.NodeExporter}
for _, m := range members {
toward := []string{self}
for _, o := range members {
if o.IP != m.IP {
toward = append(toward, o.IP)
}
}
nets, err := firewall.Networks(ctx, m.Remote, toward)
if err == nil {
nets = append(nets, extra...)
var changed bool
if changed, err = firewall.Install(ctx, m.Remote, firewall.Spec{Ports: ports, Allow: nets}); err == nil {
if changed {
in.done("firewall on %s: the store's ports answer the cluster's networks only", m.Name)
} else {
in.kept("firewall on %s", m.Name)
}
continue
}
}
fmt.Fprintf(in.out, " !! firewall on %s: %v\n", m.Name, err)
}
}
// controlPlanePorts are a control plane's internal ports, from its
// configuration: the agents' stream, and the metrics when they listen
// beyond this machine.
func controlPlanePorts(cfg config.Server) []firewall.Port {
var ports []firewall.Port
if p := portOf(cfg.AgentListen); p > 0 {
ports = append(ports, firewall.Single(p, "control plane: the agents' stream"))
}
if p := portOf(cfg.MetricsListen); p > 0 && !loopbackListen(cfg.MetricsListen) {
ports = append(ports, firewall.Single(p, "control plane: metrics, for Prometheus"))
}
return ports
}
func portOf(listen string) int {
_, port, err := net.SplitHostPort(listen)
if err != nil {
return 0
}
p, _ := strconv.Atoi(port)
return p
}
func loopbackListen(listen string) bool {
host, _, err := net.SplitHostPort(listen)
if err != nil {
return false
}
ip := net.ParseIP(host)
return host == "localhost" || (ip != nil && ip.IsLoopback())
}
+39 -3
View File
@@ -6,6 +6,7 @@ import (
"net"
"os"
"os/exec"
"path/filepath"
"strings"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
@@ -58,15 +59,32 @@ func (in *serverInit) ensurePatroni(ctx context.Context) (err error) {
setup.Witness = &pgha.Node{Member: pgha.Member{Name: name, IP: self}, Remote: pgha.Local{}}
setup.Clients = []string{self}
cfg, err := config.Load(in.configPath())
if err != nil {
return err
}
// The cluster's authority signs each member's certificate, and the
// control plane verifies the member it reaches against it: where its
// key is on this machine, and the server has the certificate to read.
setup.CAFile = cfg.AgentTLS.CA
if ca, cerr := loadCA(in.pkiDir); cerr == nil && cfg.AgentTLS.CA != "" {
setup.CA = ca
} else {
in.kept("store: the authority's key is not here, so no member certificates are signed from this machine; " +
"run this on the control plane that holds it to turn TLS on")
}
dsn, err := setup.Run(ctx)
if err != nil {
return err
}
cfg, err := config.Load(in.configPath())
if err != nil {
if err := in.writeSecret(patroniRestFile(in.etcDir), setup.RestPassword); err != nil {
return err
}
if !in.noFirewall {
extra, _ := cfg.FirewallAllow()
in.storeFirewall(ctx, setup.DB, self, extra)
}
current := cfg.Auth.Store
if current == "" {
current = in.storeAddress()
@@ -180,3 +198,21 @@ func ipv4Of(host string) (string, error) {
}
return "", fmt.Errorf("%s has no IPv4 address", host)
}
// patroniRestFile holds what Patroni's API asks for before a change, on the
// control plane: for `server add`, which lets a second one in.
func patroniRestFile(etcDir string) string { return filepath.Join(etcDir, "secrets", "patroni-rest") }
// writeSecret writes a secret root alone reads, unless it is there already.
func (in *serverInit) writeSecret(path, value string) error {
if value == "" {
return nil
}
if b, err := os.ReadFile(path); err == nil && strings.TrimSpace(string(b)) == value { //nolint:gosec // our own secret
return nil
}
if err := os.MkdirAll(filepath.Dir(path), 0o750); err != nil {
return err
}
return os.WriteFile(path, []byte(value+"\n"), 0o600)
}
+9
View File
@@ -389,6 +389,15 @@ func (in *serverInit) writeDSN(dsn string) error {
if err := in.chown(dir, false); err != nil {
return err
}
// Root's, and the service's group may enter it. Its mode set too, not
// only on creation: a directory made 0700 for the service user (as
// `server add` made them) and handed to root here was one the service
// could no longer open, and the gateway did not start.
if in.user != "" {
if err := os.Chmod(dir, 0o750); err != nil {
return err
}
}
if err := os.WriteFile(in.dsnPath(), []byte(dsn+"\n"), 0o600); err != nil {
return err
}
+12 -4
View File
@@ -1370,10 +1370,18 @@ func filepathBase(p string) string {
// file is server.yaml on a control plane, agent.yaml on a node.
func runConsole(ctx context.Context, out io.Writer, file string) error {
addr := "127.0.0.1:9090"
if cfg, err := config.Load(file); err == nil && cfg.AdminListen != "" {
addr = cfg.AdminListen
if strings.HasPrefix(addr, ":") {
addr = "127.0.0.1" + addr
if cfg, err := config.Load(file); err == nil {
if cfg.AdminListen != "" {
addr = cfg.AdminListen
if strings.HasPrefix(addr, ":") {
addr = "127.0.0.1" + addr
}
}
// The admin token, for this screen and every command it runs in
// this process — `llmgw ctl` included — from this configuration's
// file rather than only the usual place.
if t := readAdminToken(adminTokenPath(file, cfg)); t != "" {
_ = os.Setenv("LLMGW_ADMIN_TOKEN", t)
}
}
menu := cOverview
+6 -2
View File
@@ -4,6 +4,7 @@ import (
"context"
"fmt"
"io"
"net"
"net/mail"
"os"
"strings"
@@ -448,8 +449,11 @@ func (m *setupModel) startJoins() (tea.Cmd, bool) {
continue
}
if m.listener == nil {
l, err := joinpkg.Listen(fmt.Sprintf(":%d", joinpkg.DefaultPort),
fmt.Sprintf("%s:%d", m.thisAddr(), joinpkg.DefaultPort), time.Hour)
// On the address the codes name, not every interface: a code is
// a credential, and the port it is spent on need not face more
// than the network the machines join from.
addr := net.JoinHostPort(m.thisAddr(), fmt.Sprint(joinpkg.DefaultPort))
l, err := joinpkg.Listen(addr, addr, time.Hour)
if err != nil {
m.formErr = err.Error()
return nil, false
+122
View File
@@ -0,0 +1,122 @@
// Package clientip says who sent a request and how, believing a proxy's
// X-Forwarded-* headers only when the request came from a proxy the
// configuration names.
//
// Believed from anyone, X-Forwarded-Host decides the address written into the
// setup scripts clients pipe to a shell, and X-Forwarded-For lets one client
// claim to be any number of others to whatever counts attempts per address.
// Believed from nobody, every client behind a proxy is one address, and a
// limit per address is a limit on all of them together.
package clientip
import (
"net"
"net/http"
"net/netip"
"strings"
"sync/atomic"
)
var trusted atomic.Pointer[[]netip.Prefix]
// SetTrusted names the proxies whose forwarding headers are believed:
// addresses or CIDR prefixes. Anything unparsable is skipped and returned.
func SetTrusted(list []string) (skipped []string) {
var out []netip.Prefix
for _, s := range list {
s = strings.TrimSpace(s)
if p, err := netip.ParsePrefix(s); err == nil {
out = append(out, p.Masked())
continue
}
if a, err := netip.ParseAddr(s); err == nil {
out = append(out, netip.PrefixFrom(a, a.BitLen()))
continue
}
skipped = append(skipped, s)
}
trusted.Store(&out)
return skipped
}
func isTrusted(addr string) bool {
a, err := netip.ParseAddr(addr)
if err != nil {
return false
}
a = a.Unmap()
list := trusted.Load()
if list == nil {
return false
}
for _, p := range *list {
if p.Contains(a) {
return true
}
}
return false
}
// peer is the address the connection came from.
func peer(r *http.Request) string {
host, _, err := net.SplitHostPort(r.RemoteAddr)
if err != nil {
return r.RemoteAddr
}
return host
}
// fromProxy reports whether the request came through a trusted proxy.
func fromProxy(r *http.Request) bool { return isTrusted(peer(r)) }
// Addr is the client's address: the connection's, or through trusted
// proxies the nearest address in X-Forwarded-For that is not one of them.
func Addr(r *http.Request) string {
p := peer(r)
if !isTrusted(p) {
return p
}
hops := strings.Split(r.Header.Get("X-Forwarded-For"), ",")
for i := len(hops) - 1; i >= 0; i-- {
h := strings.TrimSpace(hops[i])
if h == "" {
continue
}
if !isTrusted(h) {
return h
}
}
return p
}
// Proto is "https" when the client spoke TLS: to this server, or to a
// trusted proxy that says so.
func Proto(r *http.Request) string {
if r.TLS != nil {
return "https"
}
if fromProxy(r) {
if v := first(r.Header.Get("X-Forwarded-Proto")); v != "" {
return strings.ToLower(v)
}
}
return "http"
}
// Host is the host the client asked for: Host, or what a trusted proxy says
// it was asked for.
func Host(r *http.Request) string {
if fromProxy(r) {
if v := first(r.Header.Get("X-Forwarded-Host")); v != "" {
return v
}
}
return r.Host
}
func first(v string) string {
if i := strings.IndexByte(v, ','); i >= 0 {
v = v[:i]
}
return strings.TrimSpace(v)
}
+34
View File
@@ -0,0 +1,34 @@
package clientip
import (
"net/http/httptest"
"testing"
)
func TestForwardingHeadersAreBelievedOnlyFromTrustedProxies(t *testing.T) {
r := httptest.NewRequest("GET", "/setup/x", nil)
r.RemoteAddr = "203.0.113.9:4000"
r.Header.Set("X-Forwarded-For", "10.9.9.9")
r.Header.Set("X-Forwarded-Host", "evil.example")
r.Header.Set("X-Forwarded-Proto", "https")
r.Host = "gw.example:8080"
SetTrusted(nil)
if Addr(r) != "203.0.113.9" || Host(r) != "gw.example:8080" || Proto(r) != "http" {
t.Errorf("believed an untrusted client: %s %s %s", Addr(r), Host(r), Proto(r))
}
if bad := SetTrusted([]string{"203.0.113.0/24", "10.0.0.5", "nonsense"}); len(bad) != 1 {
t.Errorf("skipped %v", bad)
}
r.Header.Set("X-Forwarded-For", "198.51.100.7, 10.0.0.5")
if Addr(r) != "198.51.100.7" || Host(r) != "evil.example" || Proto(r) != "https" {
t.Errorf("through trusted proxies: %s %s %s", Addr(r), Host(r), Proto(r))
}
// A client prepending addresses gains nothing: the nearest untrusted one
// is the one the proxy saw.
r.Header.Set("X-Forwarded-For", "1.2.3.4, 198.51.100.7")
if Addr(r) != "198.51.100.7" {
t.Errorf("spoofed: %s", Addr(r))
}
}
+32
View File
@@ -0,0 +1,32 @@
package config
import (
"fmt"
"strings"
"filippo.io/age"
"filippo.io/age/agessh"
)
// ParseBackupRecipients reads backup_recipients: age recipients, or SSH
// public keys.
func ParseBackupRecipients(list []string) ([]age.Recipient, error) {
var out []age.Recipient
for i, s := range list {
s = strings.TrimSpace(s)
if strings.HasPrefix(s, "ssh-") {
r, err := agessh.ParseRecipient(s)
if err != nil {
return nil, fmt.Errorf("backup_recipients[%d]: %w", i, err)
}
out = append(out, r)
continue
}
rs, err := age.ParseRecipients(strings.NewReader(s))
if err != nil || len(rs) != 1 {
return nil, fmt.Errorf("backup_recipients[%d] is not an age recipient (age1…) or an SSH public key", i)
}
out = append(out, rs[0])
}
return out, nil
}
+55 -4
View File
@@ -65,6 +65,29 @@ type Server struct {
// cover the client-facing one: that surface is fronted by a publicly
// trusted certificate, while this one is mutual and private.
AgentTLS tlsconf.Config `koanf:"agent_tls"`
// MaxConnections bounds the client port's open connections; past it new
// ones wait in the kernel's backlog. 0 is no bound.
MaxConnections int `koanf:"max_connections"`
// AdminTokenFile holds the token the admin API's /api and /debug paths
// demand. Empty is admin-token in the systemd unit's state directory
// (/var/lib/llmgw), or secrets/admin-token beside this file for a server
// run by hand. Made, 0600, when the server starts and it does not exist;
// `sudo llmgw ctl` and the console read it from there.
AdminTokenFile string `koanf:"admin_token_file"`
// AdminRemote allows admin_listen on an address other than loopback. The
// token still guards the API, but it then crosses the network in clear.
AdminRemote bool `koanf:"admin_remote"`
// DebugPprof serves Go's profiler on the admin listener, behind the token.
// Off by default: a heap profile holds the API keys requests presented.
DebugPprof bool `koanf:"debug_pprof"`
// AgentInsecure lets a control plane run its agent port without
// agent_tls: plaintext, and any machine that reaches it may register as
// any node — be given replicas, and so be sent users' prompts — or, in
// the other direction, a machine in the path may tell every agent what
// to run. Refused unless said here, and then the port listens on
// loopback unless agent_listen names an address. For one machine, a
// test, or a network nothing untrusted reaches.
AgentInsecure bool `koanf:"agent_insecure"`
// Monitoring is where the dashboard reads its history from.
Monitoring Monitoring `koanf:"monitoring"`
@@ -98,6 +121,20 @@ type Server struct {
// Empty means /var/backups/llmgw-server.
BackupDir string `koanf:"backup_dir"`
// FirewallNetworks are networks, besides those each machine reaches the
// cluster on, that llmgw's internal ports answer: a monitoring host on
// another subnet, nodes behind a router. Written into every machine's
// firewall by `server init`, `node add` and `server add`.
FirewallNetworks []string `koanf:"firewall_networks"`
// BackupRecipients encrypt every archive `llmgw server backup` writes, to
// whoever holds one of their keys: age recipients (age1…) or SSH public
// keys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds
// the certificate authority's key and every secret in the store, and is
// copied off the machine; encrypted, the copy is safe wherever it lands.
// Empty writes archives unencrypted, readable by root only.
BackupRecipients []string `koanf:"backup_recipients"`
// Releases is where `llmgw ctl upgrade` fetches release binaries from,
// for this machine and — through their agents — every node. Empty means
// the project's own releases.
@@ -386,6 +423,12 @@ type Router struct {
// MaxBodyBytes bounds the request body the router will read.
MaxBodyBytes int64 `koanf:"max_body_bytes"`
// TrustedProxies are the reverse proxies in front of the client port,
// as addresses or CIDR prefixes. Their X-Forwarded-For, -Proto and -Host
// are believed; anyone else's are not, since they decide the address
// written into the setup scripts and whose attempts are counted where.
TrustedProxies []string `koanf:"trusted_proxies"`
Timeouts Timeouts `koanf:"timeouts"`
Scoring Scoring `koanf:"scoring"`
Admission Admission `koanf:"admission"`
@@ -483,10 +526,11 @@ type Endpoint struct {
// Default returns the configuration used when nothing is specified.
func Default() Server {
return Server{
Listen: ":8080",
AdminListen: "127.0.0.1:9090",
AgentListen: ":7443",
Mirror: Mirror{UpdateCheck: 6 * time.Hour},
Listen: ":8080",
AdminListen: "127.0.0.1:9090",
AgentListen: ":7443",
MaxConnections: 10_000,
Mirror: Mirror{UpdateCheck: 6 * time.Hour},
Control: Control{
Interval: 5 * time.Second,
BaseBackoff: 10 * time.Second,
@@ -589,9 +633,16 @@ func (s Server) Validate() error {
if s.Listen == "" {
return fmt.Errorf("listen is empty")
}
if _, err := ParseBackupRecipients(s.BackupRecipients); err != nil {
return err
}
if _, err := s.FirewallAllow(); err != nil {
return err
}
if err := s.AgentTLS.Validate(); err != nil {
return err
}
if s.Auth.Enabled && s.Auth.Store == "" {
return fmt.Errorf("auth.store is required when auth is enabled")
}
+34
View File
@@ -0,0 +1,34 @@
package config
import (
"fmt"
"net/netip"
"strings"
)
// FirewallAllow reads firewall_networks: networks, or single addresses.
func (s Server) FirewallAllow() ([]netip.Prefix, error) {
var out []netip.Prefix
for i, n := range s.FirewallNetworks {
n = strings.TrimSpace(n)
if p, err := netip.ParsePrefix(n); err == nil {
out = append(out, p.Masked())
continue
}
a, err := netip.ParseAddr(n)
if err != nil {
return nil, fmt.Errorf("firewall_networks[%d] is %q: a network (10.0.1.0/24) or an address", i, n)
}
out = append(out, netip.PrefixFrom(a, a.BitLen()))
}
return out, nil
}
// EnginePorts is the range engines listen in on every node.
func (s Server) EnginePorts() (int, int) {
if len(s.Control.PortRange) == 2 {
return s.Control.PortRange[0], s.Control.PortRange[1]
}
d := Default().Control.PortRange
return d[0], d[1]
}
+18 -2
View File
@@ -24,6 +24,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"gitea.mynodes.xyz/ilker/llmgw/internal/safedial"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
@@ -47,6 +48,8 @@ type API struct {
serve *modelserve.Server
// history is the Prometheus the console's charts read from.
history History
// revoke records a removed node, so its certificate is refused.
revoke func(ctx context.Context, nodeID string, at time.Time) error
models Models
capacity Capacity
@@ -58,6 +61,11 @@ type API struct {
// New builds the API over a controller.
func New(ctl *control.Controller) *API { return &API{ctl: ctl} }
// SetRevoker has removing a node revoke its certificate.
func (a *API) SetRevoker(f func(ctx context.Context, nodeID string, at time.Time) error) {
a.revoke = f
}
// Register mounts the endpoints on a mux under /api/v1.
func (a *API) Register(mux *http.ServeMux) {
mux.HandleFunc("GET /api/v1/summary", a.summary)
@@ -1098,6 +1106,14 @@ func (a *API) removeNode(w http.ResponseWriter, r *http.Request) {
"%s is %s: stop its agent (systemctl stop llmgw-agent) and remove it once it is lost, a minute later", id, n.State))
return
}
// Revoked first: a node removed but still holding a certificate the
// agent port accepts could connect again and be given work.
if a.revoke != nil {
if err := a.revoke(r.Context(), id, time.Now()); err != nil {
writeError(w, http.StatusInternalServerError, "revoking the node's certificate: "+err.Error())
return
}
}
inv.Remove(id)
a.ctl.Reconciler().Notify()
writeJSON(w, http.StatusOK, map[string]string{"id": id, "removed": "true"})
@@ -1700,7 +1716,7 @@ func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "no model "+name)
return
}
base, baseErr := gguf.KindFromURL(r.Context(), nil, m.Source)
base, baseErr := gguf.KindFromURL(r.Context(), safedial.Client(safedial.ControlPlane, 30*time.Second), m.Source)
have := map[string]bool{}
for _, ad := range m.Adapters {
have[ad.Source] = true
@@ -1709,7 +1725,7 @@ func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
if have[ad.Source] {
continue // already checked when it was attached
}
h, err := gguf.KindFromURL(r.Context(), nil, ad.Source)
h, err := gguf.KindFromURL(r.Context(), safedial.Client(safedial.ControlPlane, 30*time.Second), ad.Source)
switch {
case err != nil:
writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %v", ad.Name, err))
+28
View File
@@ -7,6 +7,7 @@ import (
"log/slog"
"net/http"
"net/http/httptest"
"slices"
"strings"
"testing"
"time"
@@ -707,3 +708,30 @@ func TestOnlyANodeThatIsGoneCanBeRemoved(t *testing.T) {
t.Errorf("removing it again gave %d, expected 404", code)
}
}
func TestRemovingANodeRevokesItsCertificate(t *testing.T) {
a, ctl := setup(t, testModels(), gpuNode("n1", 8), gpuNode("n2", 8))
var revoked []string
a.SetRevoker(func(_ context.Context, id string, _ time.Time) error {
revoked = append(revoked, id)
return nil
})
ctl.Inventory().SetState("n1", inventory.Lost)
if code := call(t, a, "DELETE", "/api/v1/nodes/n1", "", nil); code != 200 {
t.Fatalf("a lost node gave %d, expected 200", code)
}
if !slices.Equal(revoked, []string{"n1"}) {
t.Errorf("revoked %v, expected n1", revoked)
}
// Not revoked, not removed: a node left listed is one the operator can
// see and try again on.
a.SetRevoker(func(context.Context, string, time.Time) error { return errors.New("store down") })
ctl.Inventory().SetState("n2", inventory.Lost)
if code := call(t, a, "DELETE", "/api/v1/nodes/n2", "", nil); code != 500 {
t.Errorf("a failed revocation gave %d, expected 500", code)
}
if _, ok := ctl.Inventory().Get("n2"); !ok {
t.Error("the node was removed though its certificate was not revoked")
}
}
@@ -316,3 +316,12 @@ func TestAnotherTenantOnACardIsJudgedEveryHeartbeat(t *testing.T) {
t.Errorf("a heartbeat with no foreign reading cleared the card: %q", n.GPUs[0].Health.Problems)
}
}
func TestAProblemOnANodeWithoutGPUsDoesNotPanic(t *testing.T) {
s := newTestService()
nodeWith(s, inventory.Node{ID: "cpu1", State: inventory.Ready})
s.applyHeartbeat("cpu1", &llmgwv1.Heartbeat{NodeId: "cpu1", NewProblems: []string{"disk read errors"}})
if _, ok := s.inv.Get("cpu1"); !ok {
t.Fatal("the node is gone")
}
}
+7 -1
View File
@@ -4,6 +4,9 @@ import (
"context"
"crypto/x509"
"net/http"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
)
// peerKey carries the verified client identity through to the stream handler.
@@ -21,6 +24,9 @@ type PeerIdentity struct {
// Verified is false when the connection carried no client certificate,
// which only happens when TLS is switched off.
Verified bool
// IssuedAt is when the certificate was issued, for revocation: one issued
// for a node before it was removed is refused.
IssuedAt time.Time
}
// WithPeerIdentity records a verified identity on the context.
@@ -53,5 +59,5 @@ func AuthMiddleware(next http.Handler) http.Handler {
}
func identityOf(cert *x509.Certificate) PeerIdentity {
return PeerIdentity{CommonName: cert.Subject.CommonName, Verified: true}
return PeerIdentity{CommonName: cert.Subject.CommonName, Verified: true, IssuedAt: tlsconf.IssuedAt(cert)}
}
@@ -0,0 +1,57 @@
package agentsvc
import (
"context"
"errors"
"testing"
"time"
"connectrpc.com/connect"
)
func TestARemovedNodesCertificateIsRefused(t *testing.T) {
removed := time.Date(2026, 10, 4, 12, 0, 0, 0, time.UTC)
s := newTestService()
s.requireIdentity = true
s.SetRevocations(func(_ context.Context, id string) (time.Time, error) {
if id == "gpu01" {
return removed, nil
}
return time.Time{}, nil
})
peer := func(cn string, issued time.Time) context.Context {
return WithPeerIdentity(context.Background(), PeerIdentity{CommonName: cn, Verified: true, IssuedAt: issued})
}
cases := []struct {
name string
ctx context.Context
node string
refuse connect.Code
}{
{"issued before the removal", peer("gpu01", removed.Add(-time.Minute)), "gpu01", connect.CodePermissionDenied},
{"issued at the removal", peer("gpu01", removed), "gpu01", connect.CodePermissionDenied},
{"issued after, by onboarding again", peer("gpu01", removed.Add(time.Second)), "gpu01", 0},
{"another node", peer("gpu02", removed.Add(-time.Hour)), "gpu02", 0},
{"another node's name", peer("gpu02", removed.Add(time.Hour)), "gpu01", connect.CodePermissionDenied},
{"no certificate", context.Background(), "gpu02", connect.CodeUnauthenticated},
}
for _, c := range cases {
err := s.admit(c.ctx, c.node)
if c.refuse == 0 {
if err != nil {
t.Errorf("%s: refused: %v", c.name, err)
}
continue
}
if connect.CodeOf(err) != c.refuse {
t.Errorf("%s: got %v, expected %v", c.name, err, c.refuse)
}
}
// Not knowing is not "not revoked".
s.SetRevocations(func(context.Context, string) (time.Time, error) { return time.Time{}, errors.New("store down") })
if err := s.admit(peer("gpu02", removed), "gpu02"); connect.CodeOf(err) != connect.CodeUnavailable {
t.Errorf("with the store down: %v, expected unavailable", err)
}
}
+50 -18
View File
@@ -49,6 +49,9 @@ type Service struct {
// requireIdentity refuses agents that present no certificate. It is off
// only for the insecure development path.
requireIdentity bool
// revokedAt is when a node was removed, whose certificates issued before
// then are refused; nil checks nothing.
revokedAt func(ctx context.Context, nodeID string) (time.Time, error)
mu sync.RWMutex
conns map[string]*conn
@@ -137,12 +140,55 @@ func New(inv *inventory.Store, status StatusSink, requireIdentity bool, log *slo
}
}
// admit decides whether the peer on ctx may be nodeID.
func (s *Service) admit(ctx context.Context, nodeID string) error {
// The certificate decides who this is, not the message.
//
// Without this check an agent could register as any node in the fleet and
// receive that node's shards — which is the same exposure the stream had
// before TLS, just with encryption on top. mTLS is only worth having if the
// identity it proves is the one actually used.
if peer := peerFrom(ctx); peer.Verified {
if peer.CommonName != nodeID {
s.log.Warn("rejecting an agent claiming a node id its certificate does not authorize",
"certificate_cn", peer.CommonName, "claimed", nodeID)
return connect.NewError(connect.CodePermissionDenied,
fmt.Errorf("certificate authorizes node %q, not %q", peer.CommonName, nodeID))
}
if s.revokedAt != nil {
at, err := s.revokedAt(ctx, nodeID)
if err != nil {
// Unknown is not "not revoked": the agent retries.
return connect.NewError(connect.CodeUnavailable, fmt.Errorf("checking revocation: %w", err))
}
if !at.IsZero() && !peer.IssuedAt.After(at) {
s.log.Warn("rejecting a removed node's certificate", "node", nodeID,
"issued", peer.IssuedAt, "removed", at)
return connect.NewError(connect.CodePermissionDenied, fmt.Errorf(
"node %q was removed from the cluster, and its certificate with it; onboard it again", nodeID))
}
}
} else if s.requireIdentity {
return connect.NewError(connect.CodeUnauthenticated,
errors.New("no verified client certificate"))
}
return nil
}
// SetRevocations has agents refused whose certificate was issued before
// their node was removed.
func (s *Service) SetRevocations(f func(ctx context.Context, nodeID string) (time.Time, error)) {
s.revokedAt = f
}
// Handler returns the HTTP path and handler to mount.
//
// The handler is wrapped so the verified client identity reaches the stream:
// connect exposes the peer's address but not its certificate.
func (s *Service) Handler() (string, http.Handler) {
path, h := llmgwv1connect.NewAgentServiceHandler(s)
// A message is a heartbeat, a report or a log; nothing an agent says
// needs more, and a bound keeps one connection from taking the memory.
path, h := llmgwv1connect.NewAgentServiceHandler(s, connect.WithReadMaxBytes(16<<20))
return path, AuthMiddleware(h)
}
@@ -180,22 +226,8 @@ func (s *Service) Connect(
return connect.NewError(connect.CodeInvalidArgument, errors.New("node_id is required"))
}
// The certificate decides who this is, not the message.
//
// Without this check an agent could register as any node in the fleet and
// receive that node's shards — which is the same exposure the stream had
// before TLS, just with encryption on top. mTLS is only worth having if the
// identity it proves is the one actually used.
if peer := peerFrom(ctx); peer.Verified {
if peer.CommonName != nodeID {
s.log.Warn("rejecting an agent claiming a node id its certificate does not authorize",
"certificate_cn", peer.CommonName, "claimed", nodeID)
return connect.NewError(connect.CodePermissionDenied,
fmt.Errorf("certificate authorizes node %q, not %q", peer.CommonName, nodeID))
}
} else if s.requireIdentity {
return connect.NewError(connect.CodeUnauthenticated,
errors.New("no verified client certificate"))
if err := s.admit(ctx, nodeID); err != nil {
return err
}
node := NodeFromRegister(reg)
@@ -422,7 +454,7 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
n.GPUs[i].Health.Problems = ps
}
}
if len(problems) > 0 {
if len(problems) > 0 && len(n.GPUs) > 0 {
// Problems are reported per node, not per GPU, so attach them to the
// node's first GPU rather than guessing which one is affected. Any
// problem is enough to keep that GPU out of placement, which is the
+3
View File
@@ -440,6 +440,9 @@ func (m Model) validate() error {
m.Name, p.Name, k, why)
}
}
if err := engines.UnsafeArgs(p.Engine, p.Args); err != nil {
return fmt.Errorf("model %q profile %q: %w", m.Name, p.Name, err)
}
if p.GPUClass == "" && !cpuCapable(p.Engine) {
return fmt.Errorf("model %q profile %q: gpu_class is required for engine %q",
+18
View File
@@ -385,3 +385,21 @@ func TestARepositoryPageIsReadAsTheRepositoryForVLLM(t *testing.T) {
t.Errorf("a file under /resolve/ is a source: %v", err)
}
}
func TestAModelThatWouldRunCodeOnItsNodesIsRefusedWhereItIsDefined(t *testing.T) {
for _, p := range []Profile{
{Name: "p", Engine: "vllm", GPUClass: "h100-80g", TP: 1, Args: []string{"--trust-remote-code"}},
{Name: "p", Engine: "llamacpp", MinMemoryGB: 4, Args: []string{"--log-file", "/etc/cron.d/x"}},
{Name: "p", Engine: "vllm", GPUClass: "h100-80g", TP: 1, Env: map[string]string{"NCCL_NET_PLUGIN": "/tmp/x.so"}},
{Name: "p", Engine: "vllm", GPUClass: "h100-80g", TP: 1, Env: map[string]string{"CUDA_INJECTION64_PATH": "/tmp/x.so"}},
} {
if _, err := New([]Model{model("m", p)}); err == nil {
t.Errorf("accepted %v %v", p.Args, p.Env)
}
}
ok := Profile{Name: "p", Engine: "llamacpp", MinMemoryGB: 4, Args: []string{"--ctx-size", "16384"},
Env: map[string]string{"OMP_NUM_THREADS": "8"}}
if _, err := New([]Model{model("m", ok)}); err != nil {
t.Errorf("a tuned model was refused: %v", err)
}
}
+26 -2
View File
@@ -234,13 +234,37 @@ func (s *Server) Start(req Request) (Job, error) {
s.seq++
j := &Job{ID: fmt.Sprintf("serve-%d-%d", time.Now().Unix(), s.seq), Request: req,
State: Running, Started: time.Now()}
s.jobs[j.ID] = j
s.keepJob(j)
s.mu.Unlock()
go s.run(j)
return s.snapshot(j), nil
}
// Jobs is every job since the control plane started, the newest first.
// keptJobs is how many finished jobs are remembered: enough for the console's
// list, and not one more for every model ever served.
const keptJobs = 100
// keepJob records a job, and forgets the oldest finished ones beyond
// keptJobs. Called with s.mu held.
func (s *Server) keepJob(j *Job) {
s.jobs[j.ID] = j
var done []*Job
for _, o := range s.jobs {
if o.State != Running {
done = append(done, o)
}
}
if len(done) <= keptJobs {
return
}
slices.SortFunc(done, func(a, b *Job) int { return a.Started.Compare(b.Started) })
for _, o := range done[:len(done)-keptJobs] {
delete(s.jobs, o.ID)
}
}
// Jobs is the jobs since the control plane started — every one running, and
// the last keptJobs finished — the newest first.
func (s *Server) Jobs() []Job {
s.mu.Lock()
all := make([]*Job, 0, len(s.jobs))
@@ -4,6 +4,7 @@ import (
"bytes"
"context"
"encoding/binary"
"fmt"
"net/http"
"net/http/httptest"
"slices"
@@ -375,3 +376,24 @@ func TestAModelThatFitsBusyCardsIsAddedToWaitNotRefused(t *testing.T) {
t.Errorf("%+v", last)
}
}
func TestOnlyTheLastFinishedJobsAreRemembered(t *testing.T) {
s := New(Deps{})
start := time.Now().Add(-time.Hour)
s.mu.Lock()
running := &Job{ID: "running", State: Running, Started: start}
s.keepJob(running)
for i := range keptJobs + 5 {
s.keepJob(&Job{ID: fmt.Sprintf("done-%d", i), State: "done", Started: start.Add(time.Duration(i+1) * time.Second)})
}
s.mu.Unlock()
if n := len(s.Jobs()); n != keptJobs+1 {
t.Errorf("%d jobs remembered, expected %d finished and the running one", n, keptJobs)
}
if _, ok := s.Get("running"); !ok {
t.Error("the running job, the oldest, was forgotten")
}
if _, ok := s.Get("done-0"); ok {
t.Error("the oldest finished job is still remembered")
}
}
+1 -1
View File
@@ -436,7 +436,7 @@ func (s *Server) StartUpdate(name, by string) (Job, error) {
s.seq++
j := &Job{ID: fmt.Sprintf("update-%d-%d", time.Now().Unix(), s.seq), Kind: "update", Model: name,
Request: Request{Name: name, By: by}, State: Running, Started: time.Now()}
s.jobs[j.ID] = j
s.keepJob(j)
s.mu.Unlock()
go s.runUpdate(j)
return s.snapshot(j), nil
+9
View File
@@ -182,6 +182,7 @@ var Reference = []Block{
{Key: "inject_usage", Type: "bool", Doc: "InjectUsage asks engines for usage even when the client did not, so token\naccounting works regardless of client behaviour.", Nested: ""},
{Key: "tenant_label", Type: "string", Doc: "TenantLabel is the level of the tree the per-tenant request series are\nlabelled by — llmgw_tenant_requests_total and its siblings: \"org\",\n\"team\" or \"project\", or \"none\" for no such series. A label per project\non a cluster of thousands is more series than Prometheus wants, and a\nteam is what an alert is usually about; \"team\" is the default.", Nested: ""},
{Key: "max_body_bytes", Type: "int64", Doc: "MaxBodyBytes bounds the request body the router will read.", Nested: ""},
{Key: "trusted_proxies", Type: "[]string", Doc: "TrustedProxies are the reverse proxies in front of the client port,\nas addresses or CIDR prefixes. Their X-Forwarded-For, -Proto and -Host\nare believed; anyone else's are not, since they decide the address\nwritten into the setup scripts and whose attempts are counted where.", Nested: ""},
{Key: "timeouts", Type: "Timeouts", Doc: "", Nested: "config.Timeouts"},
{Key: "scoring", Type: "Scoring", Doc: "", Nested: "config.Scoring"},
{Key: "admission", Type: "Admission", Doc: "", Nested: "config.Admission"},
@@ -213,12 +214,19 @@ var Reference = []Block{
{Key: "threads", Type: "Threads", Doc: "Threads keeps conversations and agent memory without API keys, for a\ngateway on one trusted machine. With auth on they live in the auth store,\nper project, and this block is ignored.", Nested: "config.Threads"},
{Key: "landing", Type: "Landing", Doc: "Landing configures the page served at \"/\".", Nested: "config.Landing"},
{Key: "agent_tls", Type: "tlsconf.Config", Doc: "AgentTLS authenticates the fleet-facing port. It deliberately does not\ncover the client-facing one: that surface is fronted by a publicly\ntrusted certificate, while this one is mutual and private.", Nested: "tlsconf.Config"},
{Key: "max_connections", Type: "int", Doc: "MaxConnections bounds the client port's open connections; past it new\nones wait in the kernel's backlog. 0 is no bound.", Nested: ""},
{Key: "admin_token_file", Type: "string", Doc: "AdminTokenFile holds the token the admin API's /api and /debug paths\ndemand. Empty is admin-token in the systemd unit's state directory\n(/var/lib/llmgw), or secrets/admin-token beside this file for a server\nrun by hand. Made, 0600, when the server starts and it does not exist;\n`sudo llmgw ctl` and the console read it from there.", Nested: ""},
{Key: "admin_remote", Type: "bool", Doc: "AdminRemote allows admin_listen on an address other than loopback. The\ntoken still guards the API, but it then crosses the network in clear.", Nested: ""},
{Key: "debug_pprof", Type: "bool", Doc: "DebugPprof serves Go's profiler on the admin listener, behind the token.\nOff by default: a heap profile holds the API keys requests presented.", Nested: ""},
{Key: "agent_insecure", Type: "bool", Doc: "AgentInsecure lets a control plane run its agent port without\nagent_tls: plaintext, and any machine that reaches it may register as\nany node — be given replicas, and so be sent users' prompts — or, in\nthe other direction, a machine in the path may tell every agent what\nto run. Refused unless said here, and then the port listens on\nloopback unless agent_listen names an address. For one machine, a\ntest, or a network nothing untrusted reaches.", Nested: ""},
{Key: "monitoring", Type: "Monitoring", Doc: "Monitoring is where the dashboard reads its history from.", Nested: "config.Monitoring"},
{Key: "tracing", Type: "Tracing", Doc: "Tracing sends a trace of each inference request to an OpenTelemetry\ncollector. Off unless an endpoint is set here or in\nOTEL_EXPORTER_OTLP_ENDPOINT.", Nested: "config.Tracing"},
{Key: "mirror", Type: "Mirror", Doc: "Mirror points the control plane at a model mirror it can administer.\nEmpty means there is none, and /admin/mirror stays unmounted.", Nested: "config.Mirror"},
{Key: "engines", Type: "map[string]string", Doc: "Engines are the engines every node installs, and at which version. The\nagents install them themselves — the right build for their hardware,\nchecked against its publisher's checksum and seen to run — so a node is\nnever trusted to already have what somebody once put there.\n\nUnset keeps DefaultEngines. A version replaces the default; an empty\none leaves that engine to be installed by hand.", Nested: ""},
{Key: "engine_overrides", Type: "[]EngineOverride", Doc: "EngineOverrides give some nodes different engines from the rest: a\nvLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at\nall on nodes meant only for llama.cpp. Each picks nodes and lays its\nengines over the cluster's, in order, later ones last; an empty version\nremoves that engine from the nodes it picks.", Nested: ""},
{Key: "backup_dir", Type: "string", Doc: "BackupDir is where `llmgw server backup` writes, and where the newest\narchive's age is read for llmgw_backup_last_success_timestamp_seconds.\nEmpty means /var/backups/llmgw-server.", Nested: ""},
{Key: "firewall_networks", Type: "[]string", Doc: "FirewallNetworks are networks, besides those each machine reaches the\ncluster on, that llmgw's internal ports answer: a monitoring host on\nanother subnet, nodes behind a router. Written into every machine's\nfirewall by `server init`, `node add` and `server add`.", Nested: ""},
{Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""},
{Key: "releases", Type: "string", Doc: "Releases is where `llmgw ctl upgrade` fetches release binaries from,\nfor this machine and — through their agents — every node. Empty means\nthe project's own releases.", Nested: ""},
{Key: "router", Type: "Router", Doc: "", Nested: "config.Router"},
{Key: "control", Type: "Control", Doc: "", Nested: "config.Control"},
@@ -310,6 +318,7 @@ var Reference = []Block{
{Key: "health_interval", Type: "time.Duration", Doc: "HealthInterval is how often a starting shard is polled.", Nested: ""},
{Key: "rendezvous_timeout", Type: "time.Duration", Doc: "RendezvousTimeout bounds the wait for a multi-node rendezvous point to\nstart listening.", Nested: ""},
{Key: "alive_grace", Type: "time.Duration", Doc: "AliveGrace is how long a non-serving rank must stay up before it counts\nas ready. There is no external signal for a worker, so staying alive is\nthe only evidence available.", Nested: ""},
{Key: "run_as", Type: "string", Doc: "RunAs is the user engines run as when the agent is root; empty is\nllmgw-engine, \"root\" keeps them root.", Nested: ""},
}},
{Package: "tlsconf", Name: "Config", Title: "tlsconf.Config", Doc: "", Fields: []Field{
{Key: "ca", Type: "string", Doc: "CA is the bundle used to verify the other side. In a real deployment this\nis whichever authority issued the fleet's certificates.", Nested: ""},
+29
View File
@@ -93,6 +93,10 @@ type Facts struct {
EnginesDir string `json:"engines_dir,omitempty"`
EnginesFreeGB float64 `json:"engines_free_gb,omitempty"`
VLLMInstalled bool `json:"vllm_installed"`
// Firewall is whether llmgw's nftables table is loaded: "present",
// "absent", or "" when it could not be read (not root).
Firewall string `json:"firewall,omitempty"`
}
// Modules checked, and why each matters.
@@ -224,9 +228,22 @@ func Collect(ctx context.Context, o Options) Facts {
f.Engines[name] = firstLine(string(out))
}
}
f.Firewall = firewallState(ctx)
return f
}
// firewallState reads whether llmgw's table is loaded.
func firewallState(ctx context.Context) string {
out, err := exec.CommandContext(ctx, "nft", "list", "table", "inet", "llmgw").CombinedOutput()
switch {
case err == nil:
return "present"
case strings.Contains(string(out), "No such file or directory"):
return "absent"
}
return "" // no nft, or not root: unknown
}
func (f *Facts) readConfig(o Options) {
path := o.ConfigPath
if path == "" {
@@ -573,6 +590,18 @@ func Evaluate(f Facts) []Finding {
"LimitMEMLOCK=infinity in the unit")
}
// --- firewall ---
// Only on a node already set up: before that, onboarding writes it.
if f.ConfigPath != "" {
switch f.Firewall {
case "present":
add(OK, "firewall", "llmgw's ports answer the cluster's networks only", "")
case "absent":
add(Warn, "firewall", "no llmgw firewall: the engines' ports answer every network this machine is on",
"run node add for it again, or list extra networks in firewall_networks first")
}
}
sort.SliceStable(out, func(i, j int) bool { return rank(out[i].Level) < rank(out[j].Level) })
return out
}
+11
View File
@@ -248,3 +248,14 @@ func TestAnAdapterNoFabricUsesDoesNotKeepTheNodeOut(t *testing.T) {
has(t, fs, Fail, "mlx5_2/1 is RoCE with no RoCE v2 GID entry")
has(t, fs, Fail, "ib_uverbs is not loaded")
}
func TestAMissingFirewallIsSaidOnASetUpNode(t *testing.T) {
f := Facts{ConfigPath: "/etc/llmgw/agent.yaml", Firewall: "absent"}
has(t, Evaluate(f), Warn, "no llmgw firewall")
f.ConfigPath = "" // before onboarding has written one
for _, x := range Evaluate(f) {
if x.Area == "firewall" {
t.Errorf("said before the node is set up: %+v", x)
}
}
}
+319
View File
@@ -0,0 +1,319 @@
// Package firewall writes and applies the nftables rules that keep llmgw's
// internal ports to the cluster's own networks.
//
// Only llmgw's ports, in a table of its own. The machines llmgw is put on
// are often not llmgw's alone — a GPU node shared with a desktop's ComfyUI,
// a database host with other databases — so the policy is not "deny
// everything else": it is "these ports answer the cluster's networks and
// this machine, and nobody else". Whatever else the machine serves, and
// whatever firewall it already has, is left as it was. The client API
// (:8080) is not among them: it is meant to be reached.
package firewall
import (
"context"
"errors"
"fmt"
"net/netip"
"slices"
"strconv"
"strings"
)
// Runner runs a shell script on a machine, as onboarding's SSH and the
// local machine both do.
type Runner interface {
Run(ctx context.Context, script string, stdin []byte) ([]byte, error)
}
// Port is one port, or a range, and what answers there.
type Port struct {
From, To int
What string
}
// Single is one port.
func Single(port int, what string) Port { return Port{From: port, To: port, What: what} }
func (p Port) nft() string {
if p.To > p.From {
return strconv.Itoa(p.From) + "-" + strconv.Itoa(p.To)
}
return strconv.Itoa(p.From)
}
// The ports of each part of a cluster.
var (
AgentPort = Single(7443, "control plane: the agents' stream")
NodeExporter = Single(9100, "node exporter: the machine's metrics, for Prometheus")
MirrorFiles = Single(8081, "mirror: weights, for the nodes")
MirrorAPI = Single(8082, "mirror: its management API, for the control planes")
Postgres = Single(5432, "store: PostgreSQL")
PatroniAPI = Single(8008, "store: Patroni's API")
EtcdClient = Single(2379, "store: etcd, for Patroni")
EtcdPeer = Single(2380, "store: etcd, between its members")
)
// Engines is the engines' port range on a node: their HTTP, for the
// router, and their rendezvous, for the other nodes of a replica.
func Engines(from, to int) Port {
return Port{From: from, To: to, What: "engines: HTTP and rendezvous"}
}
// Spec is what a machine's ruleset holds: its llmgw ports, and the
// networks they answer besides the machine itself.
type Spec struct {
Ports []Port
Allow []netip.Prefix
}
// Table is the nftables table llmgw's rules live in, and nothing else does.
const Table = "llmgw"
// RulesFile is where the ruleset is kept, for the unit that loads it at boot.
const RulesFile = "/etc/llmgw/firewall.nft"
// Unit is the systemd unit that loads it.
const Unit = "llmgw-firewall.service"
// Script is the ruleset as nft reads it. It replaces llmgw's table whole,
// so applying it again, or a changed one, leaves no stale rule behind.
func Script(s Spec) string {
ports := slices.Clone(s.Ports)
slices.SortFunc(ports, func(a, b Port) int { return a.From - b.From })
ports = slices.CompactFunc(ports, func(a, b Port) bool { return a.From == b.From && a.To == b.To })
var v4, v6 []string
for _, p := range s.Allow {
p = p.Masked()
if p.Addr().Is4() {
v4 = append(v4, p.String())
} else {
v6 = append(v6, p.String())
}
}
slices.Sort(v4)
v4 = slices.Compact(v4)
slices.Sort(v6)
v6 = slices.Compact(v6)
var list []string
for _, p := range ports {
list = append(list, p.nft())
}
set := "{ " + strings.Join(list, ", ") + " }"
var b strings.Builder
b.WriteString("# Written by llmgw. llmgw's internal ports answer this machine and the\n")
b.WriteString("# cluster's networks only; nothing else on the machine is touched.\n")
for _, p := range ports {
fmt.Fprintf(&b, "# %-11s %s\n", p.nft(), p.What)
}
fmt.Fprintf(&b, "table inet %s\ndelete table inet %s\n", Table, Table)
fmt.Fprintf(&b, "table inet %s {\n", Table)
b.WriteString("\tchain input {\n\t\ttype filter hook input priority filter - 1; policy accept;\n")
b.WriteString("\t\tiif \"lo\" accept\n")
if len(ports) > 0 {
if len(v4) > 0 {
fmt.Fprintf(&b, "\t\tip saddr { %s } tcp dport %s accept\n", strings.Join(v4, ", "), set)
}
if len(v6) > 0 {
fmt.Fprintf(&b, "\t\tip6 saddr { %s } tcp dport %s accept\n", strings.Join(v6, ", "), set)
}
fmt.Fprintf(&b, "\t\ttcp dport %s drop\n", set)
}
b.WriteString("\t}\n}\n")
return b.String()
}
// unitFile loads the ruleset at boot, and again whenever nftables.service
// is (re)started — whose own configuration commonly begins by flushing every
// table, ours included.
const unitFile = `[Unit]
Description=llmgw firewall: llmgw's internal ports for the cluster's networks only
After=nftables.service
PartOf=nftables.service
Before=network-online.target llmgw-server.service llmgw-agent.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/nft -f ` + RulesFile + `
ExecReload=/usr/sbin/nft -f ` + RulesFile + `
ExecStop=/usr/sbin/nft delete table inet ` + Table + `
[Install]
WantedBy=multi-user.target nftables.service
`
// Install writes the ruleset and its unit on a machine and applies them,
// installing nftables first where it is missing. It reports whether
// anything changed. The ruleset is checked by nft before it is put in
// place, and a broken one changes nothing.
//
// What the machine's ruleset already holds is kept: a machine can be more
// than one thing — a store member and a node — and whichever of them is set
// up second must not take the first one's ports out of the rules.
func Install(ctx context.Context, r Runner, s Spec) (bool, error) {
if old, err := r.Run(ctx, "sudo cat "+RulesFile+" 2>/dev/null || true", nil); err == nil {
prev := parse(string(old))
s.Ports = append(prev.Ports, s.Ports...)
s.Allow = append(prev.Allow, s.Allow...)
}
if len(s.Allow) == 0 {
// Ports for this machine alone would cut every node and control
// plane off. Not a firewall anybody asked for.
return false, errors.New("no network to let in; the ports were left open rather than closed to every other machine")
}
rules := Script(s)
body := rules + "\x00" + unitFile
out, err := r.Run(ctx, `set -e
if ! command -v nft >/dev/null 2>&1 && [ ! -x /usr/sbin/nft ]; then
if command -v apt-get >/dev/null 2>&1; then
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq nftables >/dev/null
elif command -v dnf >/dev/null 2>&1; then
sudo dnf install -y -q nftables >/dev/null
else
echo "no nft, and no apt-get or dnf to install it with" >&2; exit 1
fi
fi
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
cat > "$t/all"
python3 - "$t" <<'PY'
import sys
d = sys.argv[1]
rules, unit = open(d + "/all", "rb").read().split(b"\x00")
open(d + "/rules", "wb").write(rules)
open(d + "/unit", "wb").write(unit)
PY
sudo /usr/sbin/nft -c -f "$t/rules"
changed=no
if ! sudo cmp -s "$t/rules" `+RulesFile+`; then
sudo install -d -m 755 /etc/llmgw
sudo install -m 644 "$t/rules" `+RulesFile+`
changed=yes
fi
if ! sudo cmp -s "$t/unit" /etc/systemd/system/`+Unit+`; then
sudo install -m 644 "$t/unit" /etc/systemd/system/`+Unit+`
sudo systemctl daemon-reload
changed=yes
fi
sudo systemctl enable `+Unit+` >/dev/null 2>&1
if [ "$changed" = yes ] || ! sudo /usr/sbin/nft list table inet `+Table+` >/dev/null 2>&1; then
sudo /usr/sbin/nft -f `+RulesFile+`
sudo systemctl start `+Unit+` >/dev/null 2>&1 || true
echo changed
else
echo same
fi`, []byte(body))
if err != nil {
return false, fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
}
return strings.HasSuffix(strings.TrimSpace(string(out)), "changed"), nil
}
// Networks are the networks a machine reaches others on: for each address,
// the prefix of the interface its route leaves by — or the address alone,
// when the route goes through a gateway and the network behind it is not
// the machine's to know.
func Networks(ctx context.Context, r Runner, toward []string) ([]netip.Prefix, error) {
var out []netip.Prefix
for _, t := range toward {
a, err := netip.ParseAddr(t)
if err != nil {
continue
}
b, err := r.Run(ctx, "ip -o route get "+a.String()+" && ip -o addr show", nil)
if err != nil {
return nil, fmt.Errorf("finding the network toward %s: %v: %s", a, err, strings.TrimSpace(string(b)))
}
p, ok := networkToward(string(b), a)
if !ok {
return nil, fmt.Errorf("cannot tell the network toward %s from: %s", a, strings.TrimSpace(string(b)))
}
out = append(out, p)
}
return out, nil
}
// networkToward reads `ip -o route get A` followed by `ip -o addr show`.
func networkToward(text string, a netip.Addr) (netip.Prefix, bool) {
lines := strings.Split(strings.TrimSpace(text), "\n")
if len(lines) == 0 {
return netip.Prefix{}, false
}
route := strings.Fields(lines[0])
field := func(name string) string {
for i := 0; i+1 < len(route); i++ {
if route[i] == name {
return route[i+1]
}
}
return ""
}
if field("via") != "" {
// Routed: the address alone.
return netip.PrefixFrom(a, a.BitLen()), true
}
dev, src := field("dev"), field("src")
if route[0] == "local" {
// This machine's own address: the network of the interface it is on.
dev, src = "", a.String()
}
for _, l := range lines[1:] {
f := strings.Fields(l)
// "2: eth0 inet 10.0.0.147/24 brd ..."
if len(f) < 4 || (dev != "" && strings.TrimSuffix(f[1], ":") != dev) || (f[2] != "inet" && f[2] != "inet6") {
continue
}
p, err := netip.ParsePrefix(f[3])
if err != nil || (src != "" && p.Addr().String() != src) {
continue
}
if p.Contains(a) {
return p.Masked(), true
}
}
return netip.PrefixFrom(a, a.BitLen()), true
}
// parse reads back a ruleset Script wrote: its ports from the comments
// naming them, its networks from the rules.
func parse(text string) Spec {
var s Spec
for _, l := range strings.Split(text, "\n") {
if rest, ok := strings.CutPrefix(l, "# "); ok {
f := strings.Fields(rest)
if len(f) == 0 {
continue
}
from, to, isRange := strings.Cut(f[0], "-")
a, err1 := strconv.Atoi(from)
b := a
var err2 error
if isRange {
b, err2 = strconv.Atoi(to)
}
if err1 == nil && err2 == nil {
s.Ports = append(s.Ports, Port{From: a, To: b, What: strings.Join(f[1:], " ")})
}
continue
}
l = strings.TrimSpace(l)
if !strings.HasPrefix(l, "ip saddr {") && !strings.HasPrefix(l, "ip6 saddr {") {
continue
}
inner := l[strings.Index(l, "{")+1 : strings.Index(l, "}")]
for _, n := range strings.Split(inner, ",") {
if p, err := netip.ParsePrefix(strings.TrimSpace(n)); err == nil {
s.Allow = append(s.Allow, p)
}
}
}
return s
}
// Present reports whether a machine has llmgw's table loaded.
func Present(ctx context.Context, r Runner) bool {
_, err := r.Run(ctx, "sudo -n /usr/sbin/nft list table inet "+Table+" >/dev/null 2>&1", nil)
return err == nil
}
+60
View File
@@ -0,0 +1,60 @@
package firewall
import (
"net/netip"
"strings"
"testing"
)
func TestTheRulesetGuardsOnlyLlmgwsPorts(t *testing.T) {
s := Script(Spec{
Ports: []Port{NodeExporter, Engines(20000, 20999), NodeExporter},
Allow: []netip.Prefix{netip.MustParsePrefix("10.0.0.147/24"), netip.MustParsePrefix("fd00::1/64"),
netip.MustParsePrefix("10.0.0.0/24")},
})
for _, want := range []string{
"table inet llmgw\ndelete table inet llmgw\ntable inet llmgw {",
"policy accept;",
`iif "lo" accept`,
"ip saddr { 10.0.0.0/24 } tcp dport { 9100, 20000-20999 } accept",
"ip6 saddr { fd00::/64 } tcp dport { 9100, 20000-20999 } accept",
"tcp dport { 9100, 20000-20999 } drop",
} {
if !strings.Contains(s, want) {
t.Errorf("no %q in:\n%s", want, s)
}
}
if strings.Contains(s, "policy drop") || strings.Contains(s, "flush ruleset") {
t.Errorf("the ruleset reaches beyond llmgw's ports:\n%s", s)
}
}
func TestTheNetworkIsTheInterfacesOrTheAddressAlone(t *testing.T) {
addrs := `1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
2: eth0 inet 10.0.0.147/24 brd 10.0.0.255 scope global eth0\ valid_lft forever preferred_lft forever
3: eth1 inet 192.168.5.7/16 brd 192.168.255.255 scope global eth1\ valid_lft forever preferred_lft forever`
cases := []struct {
route, addr, want string
}{
{"10.0.0.140 dev eth0 src 10.0.0.147 uid 1000 \\ cache", "10.0.0.140", "10.0.0.0/24"},
{"172.16.3.4 via 10.0.0.1 dev eth0 src 10.0.0.147 uid 1000 \\ cache", "172.16.3.4", "172.16.3.4/32"},
{"192.168.9.9 dev eth1 src 192.168.5.7 uid 1000 \\ cache", "192.168.9.9", "192.168.0.0/16"},
// Its own address: the network it is on.
{"local 10.0.0.147 dev lo src 10.0.0.147 uid 1000 \\ cache <local>", "10.0.0.147", "10.0.0.0/24"},
}
for _, c := range cases {
got, ok := networkToward(c.route+"\n"+addrs, netip.MustParseAddr(c.addr))
if !ok || got.String() != c.want {
t.Errorf("toward %s: %v %v, expected %s", c.addr, got, ok, c.want)
}
}
}
func TestARulesetReadsBackAsItsSpec(t *testing.T) {
in := Spec{Ports: []Port{Postgres, Engines(20000, 20999)},
Allow: []netip.Prefix{netip.MustParsePrefix("10.0.0.0/24"), netip.MustParsePrefix("fd00::/64")}}
got := parse(Script(in))
if Script(got) != Script(in) {
t.Errorf("read back as\n%s\nfrom\n%s", Script(got), Script(in))
}
}
+12
View File
@@ -163,6 +163,10 @@ func Read(r io.Reader) (Header, error) {
}
return nil
}
// depth is how many arrays the value being read is inside. Real files
// nest none; a crafted one nesting thousands deep made a recursion as
// deep, a few bytes per level.
depth := 0
var value func(t uint32) (string, error)
value = func(t uint32) (string, error) {
num = 0
@@ -195,6 +199,11 @@ func Read(r io.Reader) (Header, error) {
if n > math.MaxInt32 {
return "", fmt.Errorf("a %d-element array in a GGUF header", n)
}
if depth >= maxArrayDepth {
return "", fmt.Errorf("arrays nested more than %d deep in a GGUF header", maxArrayDepth)
}
depth++
defer func() { depth-- }()
for i := uint64(0); i < n; i++ {
if _, err := value(et); err != nil {
return "", err
@@ -296,6 +305,9 @@ func FromURL(ctx context.Context, client *http.Client, url string) (Header, erro
return Read(io.LimitReader(resp.Body, headerBytes))
}
// maxArrayDepth is how deep arrays may nest in a header.
const maxArrayDepth = 4
// IsAdapter reports whether the file is a LoRA adapter rather than a model.
func (h Header) IsAdapter() bool { return h.Type == "adapter" }
+20
View File
@@ -143,3 +143,23 @@ func TestTheKVCacheIsSizedFromTheModelsShape(t *testing.T) {
t.Error("a header with no shape produced a figure")
}
}
func TestArraysNestedDeepAreRefused(t *testing.T) {
var b bytes.Buffer
w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
b.WriteString("GGUF")
w(uint32(3))
w(uint64(0))
w(uint64(1))
w(uint64(len("deep")))
b.WriteString("deep")
w(uint32(tArray))
for range 10_000 {
w(uint32(tArray))
w(uint64(1))
}
_, err := Read(bytes.NewReader(b.Bytes()))
if err == nil || !strings.Contains(err.Error(), "nested") {
t.Errorf("ten thousand nested arrays: %v", err)
}
}
+40 -15
View File
@@ -131,13 +131,22 @@ type response struct {
// Listener waits for machines to join.
type Listener struct {
ln net.Listener
addr string
ln net.Listener
addr string
// now is time.Now, replaced by tests.
now func() time.Time
fingerprint [16]byte
ttl time.Duration
mu sync.Mutex
pending map[[16]byte]chan *Remote // by secret
pending map[[16]byte]waiting // by secret
}
// waiting is a code not yet used: where its machine is handed over, and
// until when it may come.
type waiting struct {
ch chan *Remote
expires time.Time
}
// Listen starts listening on bind (":7444") and advertises addr ("10.0.0.5:7444")
@@ -151,7 +160,7 @@ func Listen(bind, addr string, ttl time.Duration) (*Listener, error) {
if err != nil {
return nil, fmt.Errorf("listening for machines to join on %s: %w", bind, err)
}
l := &Listener{ln: ln, addr: addr, fingerprint: fp, ttl: ttl, pending: map[[16]byte]chan *Remote{}}
l := &Listener{ln: ln, addr: addr, fingerprint: fp, ttl: ttl, pending: map[[16]byte]waiting{}, now: time.Now}
go l.accept()
return l, nil
}
@@ -188,11 +197,12 @@ func (l *Listener) Code(role string) (Code, <-chan *Remote) {
var secret [16]byte
_, _ = rand.Read(secret[:])
ch := make(chan *Remote, 1)
expires := time.Now().Add(l.ttl)
l.mu.Lock()
l.pending[secret] = ch
l.pending[secret] = waiting{ch: ch, expires: expires}
l.mu.Unlock()
return Code{Addr: l.addr, Fingerprint: l.fingerprint, Secret: secret,
Expires: time.Now().Add(l.ttl).Unix(), Role: role}, ch
Expires: expires.Unix(), Role: role}, ch
}
// Close stops listening; machines already joined stay connected until their
@@ -220,16 +230,29 @@ func (l *Listener) admit(c net.Conn) {
}
l.mu.Lock()
var ch chan *Remote
expired := false
now := l.now()
for secret, p := range l.pending {
if subtle.ConstantTimeCompare(secret[:], h.Secret) == 1 {
ch = p
// One machine a code: whoever comes second is turned away.
// One machine a code: whoever comes second is turned away. And
// not after it expires, which the joining side checking was no
// guarantee of: a code is a credential for as long as this
// listener runs, unless this side looks too.
delete(l.pending, secret)
if now.After(p.expires) {
expired = true
} else {
ch = p.ch
}
}
}
l.mu.Unlock()
if ch == nil {
_, _ = c.Write([]byte(`{"error":"this code is not one the setup is waiting for, or it was used already"}` + "\n"))
msg := "this code is not one the setup is waiting for, or it was used already"
if expired {
msg = "this code expired; ask for a new one"
}
_, _ = c.Write([]byte(`{"error":"` + msg + `"}` + "\n"))
_ = c.Close()
return
}
@@ -238,17 +261,19 @@ func (l *Listener) admit(c net.Conn) {
_ = c.Close()
return
}
addr := h.Addr
if addr == "" {
addr, _, _ = net.SplitHostPort(c.RemoteAddr().String())
}
ch <- &Remote{conn: c, r: r, Hostname: h.Hostname, Addr: addr}
// The address it connected from, not the one it says it has: the setup
// lets that address into the store, and a claim is anyone's to make.
addr, _, _ := net.SplitHostPort(c.RemoteAddr().String())
ch <- &Remote{conn: c, r: r, Hostname: h.Hostname, Addr: addr, Claimed: h.Addr}
}
// Remote is a joined machine, as the setup runs things on it.
type Remote struct {
Hostname string
Addr string
// Addr is where it connected from; Claimed what it said its address is,
// for showing beside it when the two differ.
Addr string
Claimed string
mu sync.Mutex
conn net.Conn
+17
View File
@@ -92,3 +92,20 @@ func TestAnExpiredOrDamagedCodeIsRefusedBeforeDialling(t *testing.T) {
t.Errorf("%+v, %v", back, err)
}
}
func TestTheSetupRefusesAnExpiredCodeEvenFromAJoinerThatDidNotLook(t *testing.T) {
l := listen(t)
code, arrived := l.Code("node")
// Time moves on past the code's life on the setup's side; the joiner is
// handed the code as it was, without checking it again.
l.now = func() time.Time { return time.Now().Add(2 * time.Hour) }
err := Serve(context.Background(), code, func(string) {})
if err == nil || !strings.Contains(err.Error(), "expired") {
t.Fatalf("an expired code was taken: %v", err)
}
select {
case m := <-arrived:
t.Fatalf("a machine arrived on an expired code: %+v", m)
default:
}
}
+3 -1
View File
@@ -446,6 +446,8 @@ func TestTheAPIWillNotFetchFromThePrivateNetwork(t *testing.T) {
for _, src := range []string{
"http://127.0.0.1:1/x.gguf", "http://169.254.169.254/latest/meta-data",
"http://10.0.0.140:9090/api/v1/nodes", "http://[::1]:1/x",
// The metadata address inside NAT64 and 6to4 ones.
"http://[64:ff9b::a9fe:a9fe]/latest", "http://[2002:a9fe:a9fe::1]/latest",
} {
st, err := m.Start(src, Entry{Model: "m", Version: "v", File: "x.gguf"})
if err != nil {
@@ -456,7 +458,7 @@ func TestTheAPIWillNotFetchFromThePrivateNetwork(t *testing.T) {
time.Sleep(10 * time.Millisecond)
st, _ = m.Status("m", "v")
}
if st.State != Failed || !strings.Contains(st.Err, "refusing to fetch") {
if st.State != Failed || !strings.Contains(st.Err, "refusing to reach") {
t.Errorf("%s: %+v", src, st)
}
m.Forget("m", "v")
+14
View File
@@ -128,6 +128,7 @@ func (m *Manager) Start(source string, e Entry) (Status, error) {
State: Fetching, Source: source, StartedAt: time.Now(),
}
m.running[key] = st
m.forgetOld(time.Now())
out := *st
m.mu.Unlock()
@@ -308,6 +309,19 @@ func (m *Manager) All() ([]Status, error) {
return out, nil
}
// keptFinished is how long a finished download's status is remembered.
const keptFinished = 7 * 24 * time.Hour
// forgetOld drops downloads finished more than keptFinished ago. Called with
// m.mu held.
func (m *Manager) forgetOld(now time.Time) {
for key, st := range m.running {
if st.State != Fetching && !st.FinishedAt.IsZero() && now.Sub(st.FinishedAt) > keptFinished {
delete(m.running, key)
}
}
}
// Forget drops a remembered download, so a failed one can be retried.
//
// Without it a fetch that failed once would refuse every later attempt at the
+22
View File
@@ -0,0 +1,22 @@
package mirror
import (
"testing"
"time"
)
func TestAFinishedDownloadIsForgottenAfterAWeek(t *testing.T) {
m := NewManager(t.TempDir(), discardLog())
now := time.Now()
m.mu.Lock()
m.running["old"] = &Status{State: Present, FinishedAt: now.Add(-keptFinished - time.Hour)}
m.running["recent"] = &Status{State: Failed, FinishedAt: now.Add(-time.Hour)}
m.running["going"] = &Status{State: Fetching}
m.forgetOld(now)
_, old := m.running["old"]
n := len(m.running)
m.mu.Unlock()
if old || n != 2 {
t.Errorf("after a week: old remembered=%v, %d left", old, n)
}
}
+10 -55
View File
@@ -1,80 +1,35 @@
package mirror
import (
"errors"
"fmt"
"net"
"net/http"
"net/netip"
"net/url"
"syscall"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/safedial"
)
// errPrivateSource is a fetch refused because it would reach this machine or
// the network behind it rather than the internet.
var errPrivateSource = errors.New("refusing to fetch from a loopback, private or link-local address")
var errPrivateSource = safedial.ErrBlocked
// fetchClient downloads sources for the management API.
//
// The API fetches whatever URL it is given and the file server then hands the
// result to anyone. Without this, "fetch http://169.254.169.254/..." was a way
// to read a cloud's instance metadata — or anything else on the network the
// mirror sits in — through the mirror. The check is made on the address
// actually dialled, after resolution and on every redirect, so a name that
// resolves somewhere public and then somewhere private is caught too.
// mirror sits in — through the mirror. No overall timeout: a model is tens of
// gigabytes, and after the first byte a stall is the context's to end.
func fetchClient(allowPrivate bool) *http.Client {
d := &net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}
if !allowPrivate {
d.Control = func(_, address string, _ syscall.RawConn) error {
host, _, err := net.SplitHostPort(address)
if err != nil {
return err
}
a, err := netip.ParseAddr(host)
if err != nil {
return err
}
if blockedAddr(a.Unmap()) {
return fmt.Errorf("%w (%s)", errPrivateSource, a)
}
return nil
}
}
return &http.Client{
Transport: &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: d.DialContext,
TLSHandshakeTimeout: 30 * time.Second,
ResponseHeaderTimeout: 2 * time.Minute,
ForceAttemptHTTP2: true,
},
// No overall timeout: a model is tens of gigabytes. The dial, the
// handshake and the first byte are bounded above; after that a stall
// is the context's to end.
CheckRedirect: func(req *http.Request, via []*http.Request) error {
if len(via) >= 10 {
return errors.New("too many redirects")
}
return checkScheme(req.URL)
},
if allowPrivate {
return safedial.Client(safedial.Policy{Private: true, Loopback: true, LinkLocal: true}, 0)
}
return safedial.Client(safedial.Policy{}, 0)
}
func blockedAddr(a netip.Addr) bool {
cgnat := netip.MustParsePrefix("100.64.0.0/10")
return a.IsLoopback() || a.IsPrivate() || a.IsLinkLocalUnicast() ||
a.IsLinkLocalMulticast() || a.IsMulticast() || a.IsUnspecified() || cgnat.Contains(a)
}
func blockedAddr(a netip.Addr) bool { return safedial.Blocked(a, safedial.Policy{}) }
// checkScheme allows the two schemes a model is published over. file:// and
// the rest would read this machine's own disk.
func checkScheme(u *url.URL) error {
if u.Scheme != "http" && u.Scheme != "https" {
return fmt.Errorf("%q is not an http or https source", u.Redacted())
}
return nil
}
func checkScheme(u *url.URL) error { return safedial.CheckScheme(u) }
// validSource checks a source before a download is started for it.
func validSource(source string) error {
+24 -6
View File
@@ -227,6 +227,9 @@ func (g *GoSSH) hostKeyCallback() (ssh.HostKeyCallback, error) {
// sudoersDrop is the grant PrepareSudo writes and RevokeSudo removes.
const sudoersDrop = "/etc/sudoers.d/llmgw-setup"
// sudoGrantLife is how long the grant lasts if nothing removes it sooner.
const sudoGrantLife = time.Hour
// CanSudo reports whether the user can sudo without a password now.
func (g *GoSSH) CanSudo(ctx context.Context) bool {
_, err := g.Run(ctx, "sudo -n true", nil)
@@ -245,17 +248,14 @@ func (g *GoSSH) PrepareSudo(ctx context.Context) error {
if g.SudoPassword == "" {
return fmt.Errorf("%s cannot use sudo without a password, and none was given", g.User)
}
line := fmt.Sprintf("%s ALL=(ALL) NOPASSWD:ALL", g.User)
// Checked by visudo before it is put in place: a broken file in
// sudoers.d breaks sudo for everyone on the machine.
inner := fmt.Sprintf("umask 337; printf '%%s\\n' %s %s > %[3]s.new && visudo -cf %[3]s.new >/dev/null && mv -f %[3]s.new %[3]s",
shellQuote("# llmgw setup: removed when it finishes"), shellQuote(line), sudoersDrop)
script := "sudo -S -p '' sh -c " + shellQuote(inner)
script := "sudo -S -p '' sh -c " + shellQuote(sudoGrantScript(g.User))
if out, err := g.Run(ctx, script, []byte(g.SudoPassword+"\n")); err != nil {
return fmt.Errorf("sudo with the password given failed on %s: %v: %s", g.Addr, err, strings.TrimSpace(string(out)))
}
g.mu.Lock()
g.sudoGrant = true
// Not needed again: the grant stands in for it from here on.
g.SudoPassword = ""
g.mu.Unlock()
return nil
}
@@ -276,3 +276,21 @@ func (g *GoSSH) RevokeSudo(ctx context.Context) error {
g.mu.Unlock()
return nil
}
// sudoGrantScript writes the grant, run as root.
func sudoGrantScript(user string) string {
line := fmt.Sprintf("%s ALL=(ALL) NOPASSWD:ALL", user)
// Checked by visudo before it is put in place: a broken file in
// sudoers.d breaks sudo for everyone on the machine.
//
// And it removes itself an hour later whatever happens to this process:
// RevokeSudo is called when the setup finishes, and a setup that crashes
// or is interrupted never gets there — which would leave a permanent
// passwordless root behind.
inner := fmt.Sprintf("umask 337; printf '%%s\\n' %s %s > %[3]s.new && visudo -cf %[3]s.new >/dev/null && mv -f %[3]s.new %[3]s && "+
"{ systemd-run --quiet --on-active=%[4]d /bin/rm -f %[3]s >/dev/null 2>&1 || "+
"(setsid nohup sh -c 'sleep %[4]d; rm -f %[3]s' >/dev/null 2>&1 &) ; }",
shellQuote("# llmgw setup: removed when it finishes, or within the hour"), shellQuote(line), sudoersDrop,
int(sudoGrantLife.Seconds()))
return inner
}
+98 -19
View File
@@ -19,6 +19,9 @@ import (
"errors"
"fmt"
"net"
"net/http"
"net/netip"
"net/url"
"os"
"path/filepath"
"regexp"
@@ -30,6 +33,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/doctor"
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
)
@@ -62,6 +66,12 @@ func ParseFabric(s string) (Fabric, error) {
type Options struct {
// NoNodeExporter leaves Prometheus' node exporter off the machine.
NoNodeExporter bool
// NoFirewall leaves the machine's firewall as it is. Otherwise its llmgw
// ports — FirewallPorts — answer only the networks it reaches the
// control planes on, and FirewallAllow.
NoFirewall bool
FirewallPorts []firewall.Port
FirewallAllow []netip.Prefix
// Addr is how to reach the machine over SSH, and User who to log in as.
Addr, User string
@@ -88,6 +98,8 @@ type Options struct {
Binary, Version string
// Releases is where release files are downloaded from.
Releases string
// ReleaseClient fetches them; nil is a default client.
ReleaseClient *http.Client
// IgnoreChecks goes on when doctor finds a problem.
IgnoreChecks bool
// DisableEngines are engines this node never installs or advertises,
@@ -354,25 +366,25 @@ var releaseTag = release.Tag
func (r *Runner) binary(ctx context.Context, res *Result) error {
v := r.Opts.Version
if releaseTag.MatchString(v) {
r.emit(StepBinary, Running, "downloading llmgw "+v+" on the node", "")
name := "llmgw-" + v + "-linux-" + res.Arch
base := strings.TrimSuffix(r.Opts.Releases, "/") + "/" + v
// Fetched by the node, for its own architecture, and checked against
// the release's checksums, rather than handed whatever this machine
// happens to run.
script := fmt.Sprintf(`set -e
d=$(mktemp -d); cd "$d"
curl -fsSL -o %[1]s '%[2]s/%[1]s'
curl -fsSL -o checksums.txt '%[2]s/checksums.txt'
grep " %[1]s$" checksums.txt | sha256sum -c - >/dev/null
sudo install -m 0755 %[1]s /usr/local/bin/llmgw
cd /; rm -rf "$d"`, name, base)
out, err := r.Remote.Run(ctx, script, nil)
// Fetched here, for the node's architecture, and checked here —
// signature and checksum, by the same code that upgrades this
// machine — then handed to the node. The node checking a release
// itself could only compare it with a checksum from the same
// server, which proves nothing about who put both there.
r.emit(StepBinary, Running, "downloading llmgw "+v+" for "+res.Arch+" and checking it", "")
b, err := release.Fetch(ctx, r.Opts.ReleaseClient, r.Opts.Releases, v, "linux", goArch(res.Arch))
if err == nil {
r.emit(StepBinary, Done, "llmgw "+v+" from the release, checksum verified", "")
if err := r.installBinary(ctx, b); err != nil {
return err
}
what := "checksum verified"
if release.TrustedKey() != "" {
what = "signature and checksum verified"
}
r.emit(StepBinary, Done, "llmgw "+v+" from the release, "+what, "")
return nil
}
r.emit(StepBinary, Warn, "the node could not download the release: "+lastLine(out), "")
r.emit(StepBinary, Warn, "the release could not be fetched and checked here: "+err.Error(), "")
}
// A development build has no release to fetch, and a node that cannot
// reach the releases gets this machine's copy — when it can run it.
@@ -385,13 +397,32 @@ cd /; rm -rf "$d"`, name, base)
if err != nil {
return fmt.Errorf("reading %s: %w", r.Opts.Binary, err)
}
if out, err := r.Remote.Run(ctx, `set -e; t=$(mktemp); cat > "$t"; sudo install -m 0755 "$t" /usr/local/bin/llmgw; rm -f "$t"`, b); err != nil {
return fmt.Errorf("installing the binary: %v: %s", err, lastLine(out))
if err := r.installBinary(ctx, b); err != nil {
return err
}
r.emit(StepBinary, Done, "llmgw "+orUnknown(v)+", copied from this machine", "")
return nil
}
// installBinary puts bytes at /usr/local/bin/llmgw on the node.
func (r *Runner) installBinary(ctx context.Context, b []byte) error {
if out, err := r.Remote.Run(ctx, `set -e; t=$(mktemp); cat > "$t"; sudo install -m 0755 "$t" /usr/local/bin/llmgw; rm -f "$t"`, b); err != nil {
return fmt.Errorf("installing the binary: %v: %s", err, lastLine(out))
}
return nil
}
// goArch is Go's name for what uname -m says.
func goArch(uname string) string {
switch uname {
case "x86_64", "amd64":
return "amd64"
case "aarch64", "arm64":
return "arm64"
}
return uname
}
// --- check ---
func (r *Runner) check(ctx context.Context, res *Result) error {
@@ -426,6 +457,10 @@ func (r *Runner) check(ctx context.Context, res *Result) error {
}
return fmt.Errorf("llmgw doctor did not run on the node: %v: %s", err, lastLine(out))
}
if !r.Opts.NoFirewall && len(r.Opts.FirewallPorts) > 0 {
// A missing firewall is what this run is about to write.
rep.Findings = slices.DeleteFunc(rep.Findings, func(f doctor.Finding) bool { return f.Area == "firewall" })
}
res.Facts, res.Findings = rep.Facts, rep.Findings
fails := 0
@@ -665,7 +700,15 @@ func (r *Runner) checkEngines(ctx context.Context) []string {
// --- start ---
func (r *Runner) start(ctx context.Context, _ *Result) error {
func (r *Runner) start(ctx context.Context, res *Result) error {
// The user engines run as, so the agent drops root before starting one:
// in the groups that own the GPU devices.
if out, err := r.Remote.Run(ctx, `set -e
id llmgw-engine >/dev/null 2>&1 || sudo useradd --system --no-create-home --shell /usr/sbin/nologin llmgw-engine
for g in video render; do getent group "$g" >/dev/null && sudo usermod -aG "$g" llmgw-engine; done
true`, nil); err != nil {
r.emit(StepStart, Warn, "the user engines run as could not be made, so they run as root: "+lastLine(out), "")
}
r.emit(StepStart, Running, "starting the agent", "")
if out, err := r.Remote.Run(ctx, `set -e
sudo systemctl daemon-reload
@@ -679,9 +722,45 @@ systemctl is-active llmgw-agent`, nil); err != nil {
if !r.Opts.NoNodeExporter {
r.nodeExporter(ctx)
}
if !r.Opts.NoFirewall && len(r.Opts.FirewallPorts) > 0 {
r.firewall(ctx, res)
}
return nil
}
// firewall keeps the node's llmgw ports — its engines, its metrics — to the
// networks it reaches the control planes on. A warning when it cannot: the
// node serves either way, and says how to see why.
func (r *Runner) firewall(ctx context.Context, res *Result) {
var toward []string
for _, s := range res.Servers {
u, err := url.Parse(s)
if err != nil {
continue
}
if _, err := netip.ParseAddr(u.Hostname()); err == nil {
toward = append(toward, u.Hostname())
} else if ips, err := net.LookupIP(u.Hostname()); err == nil && len(ips) > 0 {
toward = append(toward, ips[0].String())
}
}
nets, err := firewall.Networks(ctx, r.Remote, toward)
if err == nil {
nets = append(nets, r.Opts.FirewallAllow...)
_, err = firewall.Install(ctx, r.Remote, firewall.Spec{Ports: r.Opts.FirewallPorts, Allow: nets})
}
if err != nil {
r.emit(StepStart, Warn, "the firewall was not set: "+err.Error()+
"; the node's engine ports answer every network it is on", "")
return
}
var shown []string
for _, n := range nets {
shown = append(shown, n.String())
}
r.emit(StepStart, Info, "llmgw's ports on the node answer "+strings.Join(shown, ", ")+" only ("+firewall.RulesFile+")", "")
}
// nodeExporter installs Prometheus' node exporter from the distribution, for
// what llmgw does not measure: network and disk traffic, filesystems. Best
// effort: a node without it serves as well, so a failure is a warning.
+83 -15
View File
@@ -2,8 +2,13 @@ package onboard
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"os/exec"
"regexp"
"runtime"
"strings"
@@ -12,6 +17,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/doctor"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
)
@@ -47,11 +53,6 @@ func (f *fakeMachine) Run(_ context.Context, script string, stdin []byte) ([]byt
return []byte("sudo: a password is required"), errors.New("exit 1")
}
return nil, nil
case strings.Contains(script, "checksums.txt"):
if f.download != nil {
return []byte("curl: (6) Could not resolve host"), f.download
}
return nil, nil
case strings.Contains(script, "install -m 0755") && stdin != nil:
f.files["/usr/local/bin/llmgw"] = stdin
return nil, nil
@@ -132,31 +133,85 @@ func TestANodeIsAddedFromItsAddressAlone(t *testing.T) {
}
}
func TestAReleaseIsFetchedByTheNodeAndCheckedThere(t *testing.T) {
// releaseFor serves one release for linux/<arch>, its checksums signed with
// a key the test then trusts; tamper changes the binary after signing.
func releaseFor(t *testing.T, version, arch string, body []byte, signed, tamper bool) string {
t.Helper()
pub, priv, err := release.GenerateKey()
if err != nil {
t.Fatal(err)
}
t.Setenv("LLMGW_RELEASE_KEY", pub)
asset := release.AssetName(version, "linux", arch)
sum := sha256.Sum256(body)
sums := hex.EncodeToString(sum[:]) + " " + asset + "\n"
sig, _ := release.Sign(priv, []byte(sums))
if tamper {
body = []byte("tampered")
}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/" + version + "/checksums.txt":
_, _ = w.Write([]byte(sums))
case "/" + version + "/" + release.SignatureName:
if !signed {
http.NotFound(w, r)
return
}
_, _ = w.Write([]byte(sig))
case "/" + version + "/" + asset:
_, _ = w.Write(body)
default:
http.NotFound(w, r)
}
}))
t.Cleanup(srv.Close)
return srv.URL
}
func TestAReleaseIsFetchedAndCheckedHereThenGivenToTheNode(t *testing.T) {
m := newFake()
r, _ := runner(t, m, Options{Version: "v1.2.3", Releases: "https://example.test/releases/download"})
base := releaseFor(t, "v1.2.3", hostArch(), []byte("the release"), true, false)
r, events := runner(t, m, Options{Version: "v1.2.3", Releases: base})
if _, err := r.Run(t.Context()); err != nil {
t.Fatal(err)
}
if !m.ran("sha256sum -c") || !m.ran("https://example.test/releases/download/v1.2.3") {
t.Error("the release was not fetched and verified on the node")
if got := string(m.files["/usr/local/bin/llmgw"]); got != "the release" {
t.Errorf("the node was given %q", got)
}
if _, copied := m.files["/usr/local/bin/llmgw"]; copied {
t.Error("this machine's binary was copied although the node fetched the release")
if m.ran("curl") {
t.Error("the node fetched something itself")
}
if !said(*events, Done, "signature and checksum verified") {
t.Errorf("the check was not reported: %v", *events)
}
}
func TestANodeThatCannotDownloadGetsThisMachinesCopy(t *testing.T) {
func TestAnUnsignedOrAlteredReleaseIsNotGivenToTheNode(t *testing.T) {
for name, c := range map[string][2]bool{"unsigned": {false, false}, "altered": {true, true}} {
m := newFake()
base := releaseFor(t, "v1.2.3", hostArch(), []byte("the release"), c[0], c[1])
r, events := runner(t, m, Options{Version: "v1.2.3", Releases: base})
_, _ = r.Run(t.Context())
if got := string(m.files["/usr/local/bin/llmgw"]); got == "the release" || got == "tampered" {
t.Errorf("%s: the node was given the release", name)
}
if !said(*events, Warn, "could not be fetched and checked") {
t.Errorf("%s: the refusal was not said", name)
}
}
}
func TestANodeGetsThisMachinesCopyWhenTheReleaseCannotBeReached(t *testing.T) {
m := newFake()
m.download = errors.New("exit 6")
r, events := runner(t, m, Options{Version: "v1.2.3", Releases: "https://example.test"})
r, events := runner(t, m, Options{Version: "v1.2.3", Releases: "https://127.0.0.1:1"})
if _, err := r.Run(t.Context()); err != nil {
t.Fatal(err)
}
if len(m.files["/usr/local/bin/llmgw"]) == 0 {
t.Error("the binary was not copied")
}
if !said(*events, Warn, "could not download") {
if !said(*events, Warn, "could not be fetched") {
t.Error("the failed download was not mentioned")
}
}
@@ -321,3 +376,16 @@ func TestTheFabricsGivenAreCheckedAsDeclared(t *testing.T) {
t.Errorf("doctor was not told the fabric: %v", m.scripts)
}
}
func TestTheSudoGrantRemovesItselfWithinTheHour(t *testing.T) {
inner := sudoGrantScript("ilker")
if !strings.Contains(inner, "--on-active=3600") || !strings.Contains(inner, "sleep 3600; rm -f "+sudoersDrop) {
t.Errorf("no removal scheduled: %s", inner)
}
if !strings.Contains(inner, "visudo -cf") {
t.Error("written without visudo's check")
}
if out, err := exec.Command("sh", "-n", "-c", inner).CombinedOutput(); err != nil {
t.Errorf("not shell: %v %s\n%s", err, out, inner)
}
}
+3 -1
View File
@@ -41,7 +41,9 @@ func (s *SSH) Run(ctx context.Context, script string, stdin []byte) ([]byte, err
if s.User != "" {
target = s.User + "@" + s.Addr
}
args := append(append([]string{}, sshOptions...), target, "sh", "-c", shellQuote(script))
// "--" before the target: an address beginning with "-" is not read as
// an option.
args := append(append([]string{}, sshOptions...), "--", target, "sh", "-c", shellQuote(script))
cmd := exec.CommandContext(ctx, "ssh", args...)
if stdin != nil {
cmd.Stdin = bytes.NewReader(stdin)
+109 -19
View File
@@ -37,28 +37,75 @@ type Member struct {
IP string
}
// Secrets are the cluster's three passwords.
// Secrets are the cluster's passwords: PostgreSQL's three, and the one
// Patroni's REST API asks for before a change.
type Secrets struct {
Superuser, Replicator, App string
Superuser, Replicator, App, RestAPI string
}
// Etcd is a member's /etc/default/etcd, for Ubuntu's etcd-server.
func Etcd(m Member, all []Member) string {
//
// With tls, members speak TLS to each other and to clients, with a
// certificate from the cluster's authority (EtcdTLSDir), and a client must
// present one too: without, anyone reaching :2379 could take the leader lock
// or rewrite Patroni's configuration.
func Etcd(m Member, all []Member, tls bool) string {
scheme := "http"
if tls {
scheme = "https"
}
var peers []string
for _, p := range all {
peers = append(peers, fmt.Sprintf("%s=http://%s:2380", p.Name, p.IP))
peers = append(peers, fmt.Sprintf("%s=%s://%s:2380", p.Name, scheme, p.IP))
}
return fmt.Sprintf(`# llmgw: one member of the etcd that holds Patroni's leader lock.
client := fmt.Sprintf("http://%s:2379,http://127.0.0.1:2379", m.IP)
if tls {
client = fmt.Sprintf("https://%s:2379,https://127.0.0.1:2379", m.IP)
}
conf := fmt.Sprintf(`# llmgw: one member of the etcd that holds Patroni's leader lock.
ETCD_NAME=%[1]s
ETCD_DATA_DIR=/var/lib/etcd/default
ETCD_LISTEN_PEER_URLS=http://%[2]s:2380
ETCD_LISTEN_CLIENT_URLS=http://%[2]s:2379,http://127.0.0.1:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://%[2]s:2380
ETCD_ADVERTISE_CLIENT_URLS=http://%[2]s:2379
ETCD_LISTEN_PEER_URLS=%[5]s://%[2]s:2380
ETCD_LISTEN_CLIENT_URLS=%[6]s
ETCD_INITIAL_ADVERTISE_PEER_URLS=%[5]s://%[2]s:2380
ETCD_ADVERTISE_CLIENT_URLS=%[5]s://%[2]s:2379
ETCD_INITIAL_CLUSTER=%[3]s
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=%[4]s
`, m.Name, m.IP, strings.Join(peers, ","), Scope+"-patroni")
`, m.Name, m.IP, strings.Join(peers, ","), Scope+"-patroni", scheme, client)
if tls {
conf += `ETCD_CERT_FILE=` + EtcdCert + `
ETCD_KEY_FILE=` + EtcdKey + `
ETCD_TRUSTED_CA_FILE=` + EtcdCA + `
ETCD_CLIENT_CERT_AUTH=true
ETCD_PEER_CERT_FILE=` + EtcdCert + `
ETCD_PEER_KEY_FILE=` + EtcdKey + `
ETCD_PEER_TRUSTED_CA_FILE=` + EtcdCA + `
ETCD_PEER_CLIENT_CERT_AUTH=true
`
}
return conf
}
// EtcdTLSDir holds an etcd member's certificate, which it presents both as
// a server and as a client of the other members, and the authority's.
const EtcdTLSDir = "/etc/etcd/tls"
// The files in EtcdTLSDir.
const (
EtcdCA = EtcdTLSDir + "/ca.crt"
EtcdCert = EtcdTLSDir + "/peer.crt"
EtcdKey = EtcdTLSDir + "/peer.key"
)
// etcdctl is the command that asks the cluster, over TLS with a member's
// certificate when tls; run on a member, as root, which reads it.
func etcdctl(tls bool, endpoints []string) string {
c := "sudo ETCDCTL_API=3 etcdctl --dial-timeout=3s --endpoints=" + strings.Join(endpoints, ",")
if tls {
c += " --cacert=" + EtcdCA + " --cert=" + EtcdCert + " --key=" + EtcdKey
}
return c
}
// Patroni is a database member's /etc/patroni/config.yml.
@@ -66,7 +113,10 @@ ETCD_INITIAL_CLUSTER_TOKEN=%[4]s
// clients are the addresses the control planes connect from; the members
// themselves are always allowed, since a second control plane usually runs on
// one of them.
func Patroni(m Member, db, etcd []Member, clients []string, version string, s Secrets) string {
//
// With tls, PostgreSQL is given the member's certificate (certs puts it in
// TLSDir) and every remote rule is hostssl.
func Patroni(m Member, db, etcd []Member, clients []string, version string, s Secrets, tls bool) string {
var hosts []string
for _, e := range etcd {
hosts = append(hosts, e.IP+":2379")
@@ -88,8 +138,17 @@ func Patroni(m Member, db, etcd []Member, clients []string, version string, s Se
}
var hbaLines string
for _, h := range hba {
if tls {
h = hostssl(h)
}
hbaLines += " - " + h + "\n"
}
var sslLines string
if tls {
for _, k := range []string{"ssl", "ssl_cert_file", "ssl_key_file", "ssl_ca_file"} {
sslLines += fmt.Sprintf(" %s: %q\n", k, sslParameters()[k])
}
}
var b strings.Builder
fmt.Fprintf(&b, `# llmgw: the control planes' store, kept by Patroni, its leader lock in etcd.
scope: %[1]s
@@ -99,10 +158,15 @@ name: %[2]s
restapi:
listen: %[3]s:%[4]d
connect_address: %[3]s:%[4]d
# Asked for before a change: anyone reaching this port could otherwise
# rewrite the access rules or move the leader.
authentication:
username: %[11]s
password: "%[10]s"
%[13]s
etcd3:
hosts: %[5]s
%[14]s
bootstrap:
dcs:
ttl: 30
@@ -120,7 +184,7 @@ bootstrap:
wal_keep_size: 1GB
wal_log_hints: "on"
password_encryption: scram-sha-256
# In the cluster's shared configuration rather than each member's, so a
%[12]s # In the cluster's shared configuration rather than each member's, so a
# control plane added later is let in by one change through Patroni's
# API (llmgw server add) instead of an edit on every member.
pg_hba:
@@ -146,7 +210,8 @@ postgresql:
password: "%[7]s"
parameters:
unix_socket_directories: /var/run/postgresql
`, Scope, m.Name, m.IP, PatroniPort, strings.Join(hosts, ","), version, s.Superuser, s.Replicator, hbaLines)
`, Scope, m.Name, m.IP, PatroniPort, strings.Join(hosts, ","), version, s.Superuser, s.Replicator, hbaLines,
s.RestAPI, RestUser, sslLines, restTLSLines(tls), etcdTLSLines(tls))
b.WriteString(`
tags:
nofailover: false
@@ -157,16 +222,41 @@ tags:
return b.String()
}
// restTLSLines put Patroni's API on TLS, with the member's certificate, and
// have patronictl verify it.
func restTLSLines(tls bool) string {
if !tls {
return ""
}
return " certfile: " + ServerCert + "\n keyfile: " + ServerKey + "\n cafile: " + CACert + "\n" +
"\nctl:\n cacert: " + CACert + "\n"
}
// etcdTLSLines have Patroni reach etcd over TLS, presenting the member's
// certificate.
func etcdTLSLines(tls bool) string {
if !tls {
return ""
}
return " protocol: https\n cacert: " + CACert + "\n cert: " + ServerCert + "\n key: " + ServerKey + "\n"
}
// DSN is the store address the control planes are given: every member, and
// the driver told to use whichever accepts writes, so a failover is followed
// without anything in front of the database.
func DSN(db []Member, password string) string {
// without anything in front of the database. With caFile — the cluster's
// authority, on the control plane — the connection is TLS and the member it
// reaches is verified against it.
func DSN(db []Member, password, caFile string) string {
var hosts []string
for _, d := range db {
hosts = append(hosts, d.IP+":5432")
}
return fmt.Sprintf("postgres://%s:%s@%s/%s?sslmode=disable&target_session_attrs=read-write",
Role, url.QueryEscape(password), strings.Join(hosts, ","), Database)
ssl := "sslmode=disable"
if caFile != "" {
ssl = "sslmode=verify-full&sslrootcert=" + url.QueryEscape(caFile)
}
return fmt.Sprintf("postgres://%s:%s@%s/%s?%s&target_session_attrs=read-write",
Role, url.QueryEscape(password), strings.Join(hosts, ","), Database, ssl)
}
func ips(ms []Member) []string {
+272
View File
@@ -0,0 +1,272 @@
package pgha
import (
"context"
"encoding/json"
"fmt"
"net/http"
"slices"
"strings"
"time"
)
// Hardening: what a cluster set up before this lacked, applied to it in
// place, and to a new one from the start.
//
// - Patroni's REST API asks for a password before a change. Without one,
// anyone who reached port 8008 could rewrite pg_hba through PATCH /config
// — `host all all 0.0.0.0/0 trust` — or switch the leader over.
// - PostgreSQL speaks TLS, with a certificate from the cluster's authority
// on each member, and every remote rule is hostssl: prompts kept in
// threads, key and session hashes, and the engines' secret no longer
// cross the network in clear. The control planes verify the member they
// reach (sslmode=verify-full against the cluster's CA).
//
// Each step looks first and changes only what differs, so running it on a
// hardened cluster does nothing, and each is applied by reload, not restart:
// the cluster keeps serving.
// RestUser is the user Patroni's REST API asks for before a change.
const RestUser = "patroni"
// TLSDir holds each member's certificate, key and the cluster's CA.
const TLSDir = "/etc/patroni/tls"
// ServerCert, ServerKey and CACert are the files in TLSDir.
const (
ServerCert = TLSDir + "/server.crt"
ServerKey = TLSDir + "/server.key"
CACert = TLSDir + "/ca.crt"
)
// sslParameters are what turn TLS on, in the cluster's shared parameters.
func sslParameters() map[string]any {
return map[string]any{"ssl": "on", "ssl_cert_file": ServerCert, "ssl_key_file": ServerKey, "ssl_ca_file": CACert}
}
// hostssl rewrites a remote rule to accept only TLS; local and loopback
// rules are left alone.
func hostssl(line string) string {
f := strings.Fields(line)
if len(f) < 4 || f[0] != "host" || strings.HasPrefix(f[3], "127.") || f[3] == "::1/128" {
return line
}
return "hostssl" + strings.TrimPrefix(line, "host")
}
// certs gives each database member a certificate from the cluster's
// authority, for its address and name — unless it has one for them already.
// It serves PostgreSQL and Patroni's API, and is what Patroni presents to
// etcd, so it identifies a client too. A member without PostgreSQL's user
// yet is skipped: the Patroni step calls this again once it is installed.
func (s *Setup) certs(ctx context.Context) error {
if s.CA == nil {
s.emit("tls", false, "no certificate authority on this machine: PostgreSQL stays without TLS")
return nil
}
for _, d := range s.DB {
have, _ := d.Remote.Run(ctx, "id postgres >/dev/null 2>&1 || echo no-postgres; "+
"sudo openssl x509 -in "+ServerCert+" -noout -ext subjectAltName,extendedKeyUsage 2>/dev/null", nil)
if strings.Contains(string(have), "no-postgres") {
continue
}
if strings.Contains(string(have), "IP Address:"+d.IP) && strings.Contains(string(have), "Client Authentication") {
continue
}
leaf, err := s.CA.IssuePeer(d.Name, []string{d.IP, d.Name, "127.0.0.1", "localhost"})
if err != nil {
return err
}
if err := putTLS(ctx, d, "postgres", TLSDir, CACert, ServerCert, ServerKey, s.CA.PEM, leaf.CertPEM, leaf.KeyPEM); err != nil {
return err
}
s.emit("tls", true, "certificate for %s (%s)", d.Name, d.IP)
}
return nil
}
// restAuth makes each member's Patroni ask for the password before a change,
// and reloads it where that was not so.
func (s *Setup) restAuth(ctx context.Context, sec Secrets) error {
for _, d := range s.DB {
out, err := d.Remote.Run(ctx, `set -e
test -f /etc/patroni/config.yml || { echo missing; exit 0; }
# The password from stdin into a file only this user reads: the program
# itself comes in on python's stdin.
f=$(mktemp); trap 'rm -f "$f"' EXIT
cat > "$f"
r=$(sudo python3 - "$f" <<'PY'
import sys, yaml
pw = open(sys.argv[1]).read().strip()
p = "/etc/patroni/config.yml"
c = yaml.safe_load(open(p))
want = {"username": "`+RestUser+`", "password": pw}
api = c.setdefault("restapi", {})
if api.get("authentication") == want:
print("same")
else:
api["authentication"] = want
open(p + ".new", "w").write("# llmgw: the control planes' store, kept by Patroni, its leader lock in etcd.\n" +
yaml.safe_dump(c, default_flow_style=False, sort_keys=False))
print("changed")
PY
)
if [ "$r" = changed ]; then
sudo chown root:postgres /etc/patroni/config.yml.new
sudo chmod 640 /etc/patroni/config.yml.new
sudo mv -f /etc/patroni/config.yml.new /etc/patroni/config.yml
sudo systemctl reload patroni
fi
echo "$r"`, []byte(sec.RestAPI+"\n"))
if err != nil {
return fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
}
switch strings.TrimSpace(lastLine(out)) {
case "changed":
s.emit("patroni api", true, "%s asks for a password before a change", d.Name)
case "missing":
return fmt.Errorf("%s has no /etc/patroni/config.yml", d.Name)
}
}
return nil
}
// tlsInShared turns TLS on in the cluster's shared configuration, and every
// remote access rule to hostssl, through Patroni's API — which reloads
// PostgreSQL on every member.
func (s *Setup) tlsInShared(ctx context.Context, sec Secrets) error {
if s.CA == nil {
return nil
}
hosts := make([]string, len(s.DB))
for i, d := range s.DB {
hosts[i] = d.IP
}
var lastErr error
for _, h := range hosts {
cfg, err := getConfig(ctx, h)
if err != nil {
lastErr = err
continue
}
pg, _ := cfg["postgresql"].(map[string]any)
params, _ := pg["parameters"].(map[string]any)
var hba, next []string
if list, ok := pg["pg_hba"].([]any); ok {
for _, l := range list {
if line, ok := l.(string); ok {
hba = append(hba, line)
next = append(next, hostssl(line))
}
}
}
if len(hba) == 0 {
// Kept in each member's own file, by a cluster set up before
// they moved here: made hostssl there.
if err := s.memberHostssl(ctx); err != nil {
return err
}
}
want := sslParameters()
same := slices.Equal(hba, next)
for k, v := range want {
if params[k] != v {
same = false
}
}
if same {
s.emit("tls", false, "PostgreSQL speaks TLS, and remote rules are hostssl")
return nil
}
patch := map[string]any{"parameters": want}
if len(hba) > 0 {
patch["pg_hba"] = next
}
if err := patchConfig(ctx, h, sec.RestAPI, map[string]any{"postgresql": patch}); err != nil {
return fmt.Errorf("Patroni on %s: %w", h, err)
}
s.emit("tls", true, "PostgreSQL speaks TLS on every member; remote rules are hostssl")
return nil
}
return fmt.Errorf("no member's Patroni answered: %w", lastErr)
}
// memberHostssl makes the remote rules in each member's own
// /etc/patroni/config.yml hostssl, and reloads Patroni there, which rewrites
// pg_hba.conf and reloads PostgreSQL without restarting it.
func (s *Setup) memberHostssl(ctx context.Context) error {
for _, d := range s.DB {
out, err := d.Remote.Run(ctx, `set -e
test -f /etc/patroni/config.yml || { echo missing; exit 0; }
r=$(sudo python3 - <<'PY'
import yaml
p = "/etc/patroni/config.yml"
c = yaml.safe_load(open(p))
rules = c.get("postgresql", {}).get("pg_hba") or []
def hostssl(line):
f = line.split()
if len(f) < 4 or f[0] != "host" or f[3].startswith("127.") or f[3] == "::1/128":
return line
return "hostssl" + line[len("host"):]
new = [hostssl(l) for l in rules]
if new == rules:
print("same")
else:
c["postgresql"]["pg_hba"] = new
open(p + ".new", "w").write("# llmgw: the control planes' store, kept by Patroni, its leader lock in etcd.\n" +
yaml.safe_dump(c, default_flow_style=False, sort_keys=False))
print("changed")
PY
)
if [ "$r" = changed ]; then
sudo chown root:postgres /etc/patroni/config.yml.new
sudo chmod 640 /etc/patroni/config.yml.new
sudo mv -f /etc/patroni/config.yml.new /etc/patroni/config.yml
sudo systemctl reload patroni
fi
echo "$r"`, nil)
if err != nil {
return fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
}
if strings.TrimSpace(lastLine(out)) == "changed" {
s.emit("tls", true, "%s's own access rules are hostssl", d.Name)
}
}
return nil
}
// patchConfig changes the cluster's shared configuration, with the REST
// password.
func patchConfig(ctx context.Context, host, password string, body map[string]any) error {
b, _ := json.Marshal(body)
resp, err := restRequest(ctx, host, http.MethodPatch, "/config", b, password, 15*time.Second)
if err != nil {
return err
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("refused the change: %s", resp.Status)
}
return nil
}
// sharedTLS reports whether the cluster's shared configuration has TLS on:
// for a control plane without the authority's key, which signs nothing but
// must still connect by TLS once the cluster asks for it.
func (s *Setup) sharedTLS(ctx context.Context) bool {
for _, d := range s.DB {
cfg, err := getConfig(ctx, d.IP)
if err != nil {
continue
}
pg, _ := cfg["postgresql"].(map[string]any)
params, _ := pg["parameters"].(map[string]any)
return params["ssl"] == "on"
}
return false
}
func lastLine(b []byte) string {
lines := strings.Split(strings.TrimSpace(string(b)), "\n")
return lines[len(lines)-1]
}
+16 -30
View File
@@ -1,14 +1,12 @@
package pgha
import (
"bytes"
"context"
"encoding/json"
"fmt"
"net"
"net/http"
"slices"
"strconv"
"strings"
"time"
)
@@ -24,12 +22,13 @@ func ClientLine(ip string) string {
// A cluster set up before the rules moved into the shared configuration
// keeps them in each member's own file, which this cannot reach; that is
// said, with what to add, rather than half done.
func AllowClient(ctx context.Context, hosts []string, ip string) (bool, error) {
client := &http.Client{Timeout: 5 * time.Second}
//
// password is what Patroni's API asks for before a change (RestUser); a
// cluster whose rules are hostssl lets the new control plane in by TLS too.
func AllowClient(ctx context.Context, hosts []string, ip, password string) (bool, error) {
var lastErr error
for _, h := range hosts {
base := "http://" + net.JoinHostPort(h, strconv.Itoa(restPort))
cfg, err := getConfig(ctx, client, base)
cfg, err := getConfig(ctx, h)
if err != nil {
lastErr = err
continue
@@ -50,42 +49,29 @@ func AllowClient(ctx context.Context, hosts []string, ip string) (bool, error) {
"on every member, and run `patronictl reload llmgw`", ClientLine(ip))
}
line := ClientLine(ip)
if slices.ContainsFunc(hba, func(l string) bool { return strings.HasPrefix(l, "hostssl ") }) {
line = hostssl(line)
}
if slices.Contains(hba, line) {
return false, nil
}
body, _ := json.Marshal(map[string]any{"postgresql": map[string]any{"pg_hba": append(hba, line)}})
req, err := http.NewRequestWithContext(ctx, http.MethodPatch, base+"/config", bytes.NewReader(body))
if err != nil {
return false, err
}
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
lastErr = err
continue
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return false, fmt.Errorf("Patroni on %s refused the change: %s", h, resp.Status)
if err := patchConfig(ctx, h, password,
map[string]any{"postgresql": map[string]any{"pg_hba": append(hba, line)}}); err != nil {
return false, fmt.Errorf("Patroni on %s: %w", h, err)
}
return true, nil
}
return false, fmt.Errorf("no member's Patroni answered: %w", lastErr)
}
func getConfig(ctx context.Context, client *http.Client, base string) (map[string]any, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, base+"/config", nil)
func getConfig(ctx context.Context, host string) (map[string]any, error) {
b, err := restJSON(ctx, host, http.MethodGet, "/config", nil, "", 10*time.Second)
if err != nil {
return nil, err
}
resp, err := client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
var cfg map[string]any
if err := json.NewDecoder(resp.Body).Decode(&cfg); err != nil {
return nil, fmt.Errorf("%s/config: %w", base, err)
if err := json.Unmarshal(b, &cfg); err != nil {
return nil, fmt.Errorf("%s /config: %w", host, err)
}
return cfg, nil
}
+190 -10
View File
@@ -12,6 +12,7 @@ import (
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
"github.com/prometheus/client_golang/prometheus/testutil"
"github.com/prometheus/common/expfmt"
"go.yaml.in/yaml/v3"
@@ -24,8 +25,8 @@ var (
)
func TestTheConfigurationsNameEveryMemberAndOnlyTheClientsGiven(t *testing.T) {
sec := Secrets{"su", "rep", "app"}
cfg := Patroni(pg01, []Member{pg01, ctl02}, []Member{pg01, ctl02, ctl}, []string{"10.0.0.140"}, "16", sec)
sec := Secrets{"su", "rep", "app", "api"}
cfg := Patroni(pg01, []Member{pg01, ctl02}, []Member{pg01, ctl02, ctl}, []string{"10.0.0.140"}, "16", sec, false)
for _, want := range []string{
"name: llmgw-pg01", "listen: 10.0.0.149:8008",
"hosts: 10.0.0.149:2379,10.0.0.150:2379,10.0.0.140:2379",
@@ -42,15 +43,34 @@ func TestTheConfigurationsNameEveryMemberAndOnlyTheClientsGiven(t *testing.T) {
if strings.Count(cfg, "host llmgw llmgw 10.0.0.149/32") != 1 {
t.Error("a member's address is allowed twice")
}
e := Etcd(ctl, []Member{pg01, ctl02, ctl})
e := Etcd(ctl, []Member{pg01, ctl02, ctl}, false)
if !strings.Contains(e, "ETCD_NAME=llmgw-ctl\n") ||
!strings.Contains(e, "llmgw-pg01=http://10.0.0.149:2380,llmgw-ctl02=http://10.0.0.150:2380,llmgw-ctl=http://10.0.0.140:2380") {
t.Errorf("etcd defaults:\n%s", e)
}
if got := DSN([]Member{pg01, ctl02}, "pw"); got !=
e = Etcd(ctl, []Member{pg01, ctl02, ctl}, true)
for _, want := range []string{
"llmgw-pg01=https://10.0.0.149:2380",
"ETCD_LISTEN_CLIENT_URLS=https://10.0.0.140:2379,https://127.0.0.1:2379\n",
"ETCD_ADVERTISE_CLIENT_URLS=https://10.0.0.140:2379\n",
"ETCD_CLIENT_CERT_AUTH=true\n", "ETCD_PEER_CLIENT_CERT_AUTH=true\n",
"ETCD_TRUSTED_CA_FILE=/etc/etcd/tls/ca.crt\n",
} {
if !strings.Contains(e, want) {
t.Errorf("etcd over TLS lacks %q:\n%s", want, e)
}
}
if strings.Contains(e, "http://") {
t.Errorf("etcd over TLS still listens in the clear:\n%s", e)
}
if got := DSN([]Member{pg01, ctl02}, "pw", ""); got !=
"postgres://llmgw:pw@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=disable&target_session_attrs=read-write" {
t.Errorf("dsn %s", got)
}
if got := DSN([]Member{pg01, ctl02}, "pw", "/etc/llmgw/ca.pem"); got !=
"postgres://llmgw:pw@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=verify-full&sslrootcert=%2Fetc%2Fllmgw%2Fca.pem&target_session_attrs=read-write" {
t.Errorf("dsn with tls %s", got)
}
}
func TestAClusterIsCompleteWithALeaderAndEveryOtherMemberStreaming(t *testing.T) {
@@ -102,7 +122,7 @@ func (m *machine) ranAny(frag string) bool {
return false
}
const pws = "superuser=a\nreplicator=b\nllmgw=c\n"
const pws = "superuser=a\nreplicator=b\nllmgw=c\nrestapi=d\n"
func healthy(n int) string {
return strings.Repeat("http://x:2379 is healthy: successfully committed proposal\n", n)
@@ -129,7 +149,8 @@ func TestARunningClusterIsLeftAsItIs(t *testing.T) {
t.Errorf("dsn %s", dsn)
}
for _, m := range []*machine{a, b, w} {
for _, frag := range []string{"apt-get", "systemctl restart", "/etc/default/etcd", "/etc/patroni/config.yml"} {
// Looked at, for the API's password, but not rewritten or restarted.
for _, frag := range []string{"apt-get", "systemctl restart", "/etc/default/etcd", "config.yml.new\nsudo systemctl"} {
if m.ranAny(frag) {
t.Errorf("%s was changed (%q) though the cluster is up", m.name, frag)
}
@@ -216,7 +237,7 @@ func TestAMemberThatStoppedIsReportedMissingNotForgotten(t *testing.T) {
func TestTheAccessRulesAreInTheClustersSharedConfiguration(t *testing.T) {
// Where `server add` can change them for every member at once.
cfg := Patroni(pg01, []Member{pg01, ctl02}, []Member{pg01, ctl02, ctl}, []string{"10.0.0.140"}, "16", Secrets{"a", "b", "c"})
cfg := Patroni(pg01, []Member{pg01, ctl02}, []Member{pg01, ctl02, ctl}, []string{"10.0.0.140"}, "16", Secrets{"a", "b", "c", "d"}, false)
var doc struct {
Bootstrap struct {
DCS struct {
@@ -262,7 +283,7 @@ func TestAControlPlaneIsLetInThroughTheSharedConfiguration(t *testing.T) {
restPort, _ = strconv.Atoi(u.Port())
defer func() { restPort = old }()
changed, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.150")
changed, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.150", "")
if err != nil || !changed {
t.Fatalf("%v %v", changed, err)
}
@@ -270,13 +291,172 @@ func TestAControlPlaneIsLetInThroughTheSharedConfiguration(t *testing.T) {
if len(patched) != 3 || patched[2] != ClientLine("10.0.0.150") || patched[1] != ClientLine("10.0.0.140") {
t.Errorf("patched %q", patched)
}
if changed, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.140"); err != nil || changed {
if changed, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.140", ""); err != nil || changed {
t.Errorf("one already let in was changed again: %v %v", changed, err)
}
hba = nil // the rules are in each member's file
if _, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.151"); err == nil ||
if _, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.151", ""); err == nil ||
!strings.Contains(err.Error(), "each member") {
t.Errorf("rules kept per member: %v", err)
}
}
func TestANewClusterAsksForAPasswordAndSpeaksTLS(t *testing.T) {
cfg := Patroni(pg01, []Member{pg01, ctl02}, []Member{pg01, ctl02, ctl}, []string{"10.0.0.140"}, "16",
Secrets{"a", "b", "c", "the-api-pw"}, true)
var doc struct {
RestAPI struct {
Authentication struct{ Username, Password string } `yaml:"authentication"`
CertFile string `yaml:"certfile"`
CAFile string `yaml:"cafile"`
} `yaml:"restapi"`
Ctl struct {
CACert string `yaml:"cacert"`
} `yaml:"ctl"`
Etcd3 struct {
Protocol, CACert, Cert, Key string
} `yaml:"etcd3"`
Bootstrap struct {
DCS struct {
PostgreSQL struct {
Parameters map[string]string `yaml:"parameters"`
PgHBA []string `yaml:"pg_hba"`
} `yaml:"postgresql"`
} `yaml:"dcs"`
} `yaml:"bootstrap"`
}
if err := yaml.Unmarshal([]byte(cfg), &doc); err != nil {
t.Fatalf("%v\n%s", err, cfg)
}
if a := doc.RestAPI.Authentication; a.Username != RestUser || a.Password != "the-api-pw" {
t.Errorf("rest api authentication %+v", a)
}
if doc.RestAPI.CertFile != ServerCert || doc.RestAPI.CAFile != CACert || doc.Ctl.CACert != CACert {
t.Errorf("the API is not served over TLS:\n%s", cfg)
}
if e := doc.Etcd3; e.Protocol != "https" || e.CACert != CACert || e.Cert != ServerCert || e.Key != ServerKey {
t.Errorf("etcd is not reached over TLS: %+v", e)
}
p := doc.Bootstrap.DCS.PostgreSQL.Parameters
if p["ssl"] != "on" || p["ssl_cert_file"] != ServerCert || p["ssl_key_file"] != ServerKey {
t.Errorf("parameters %v", p)
}
for _, l := range doc.Bootstrap.DCS.PostgreSQL.PgHBA {
if strings.HasPrefix(l, "host ") && !strings.Contains(l, " 127.0.0.1/32 ") {
t.Errorf("a remote rule without TLS: %q", l)
}
}
}
func TestOnlyRemoteRulesAreMadeHostssl(t *testing.T) {
for in, want := range map[string]string{
"host llmgw llmgw 10.0.0.140/32 scram-sha-256": "hostssl llmgw llmgw 10.0.0.140/32 scram-sha-256",
"host replication replicator 10.0.0.150/32 scram-sha-256": "hostssl replication replicator 10.0.0.150/32 scram-sha-256",
"host all all 127.0.0.1/32 scram-sha-256": "host all all 127.0.0.1/32 scram-sha-256",
"local all all peer": "local all all peer",
"hostssl llmgw llmgw 10.0.0.1/32 scram-sha-256": "hostssl llmgw llmgw 10.0.0.1/32 scram-sha-256",
} {
if got := hostssl(in); got != want {
t.Errorf("%q: %q", in, got)
}
}
}
func TestAClusterFromBeforeTheAPIPasswordIsGivenOne(t *testing.T) {
// The second member holds the three PostgreSQL passwords and no API one.
a := &machine{answers: map[string]string{"openssl rand": pws, "endpoint health": healthy(3)}}
b := &machine{answers: map[string]string{"sudo cat /root/llmgw-pg": "superuser=a\nreplicator=b\nllmgw=c\n"}}
up := Cluster{Members: []MemberState{{Name: "llmgw-pg01", Role: "leader", State: "running"},
{Name: "llmgw-ctl02", Role: "replica", State: "streaming"}}}
s := Setup{DB: []Node{{pg01, a}, {ctl02, b}}, Witness: &Node{ctl, &machine{}},
Status: func(context.Context, []string) (Cluster, error) { return up, nil }}
if _, err := s.Run(context.Background()); err != nil {
t.Fatal(err)
}
if !b.ranAny(`"restapi $api"`) || s.RestPassword != "d" {
t.Errorf("the API password was not carried: %q", s.RestPassword)
}
for _, m := range []*machine{a, b} {
if !m.ranAny(`"username": "patroni"`) {
t.Errorf("%s: the API was not given its password", m.name)
}
}
}
func TestTLSIsTurnedOnInTheSharedConfigurationWithThePassword(t *testing.T) {
var patched map[string]any
var user, pass string
hba := []any{"local all all peer", "host all all 127.0.0.1/32 scram-sha-256", ClientLine("10.0.0.140")}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodGet:
_ = json.NewEncoder(w).Encode(map[string]any{"postgresql": map[string]any{"pg_hba": hba, "parameters": map[string]any{}}})
case http.MethodPatch:
user, pass, _ = r.BasicAuth()
_ = json.NewDecoder(r.Body).Decode(&patched)
}
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
old := restPort
restPort, _ = strconv.Atoi(u.Port())
defer func() { restPort = old }()
ca, err := tlsconf.NewCA("test")
if err != nil {
t.Fatal(err)
}
s := Setup{DB: []Node{{Member{"m", "127.0.0.1"}, &machine{}}}, CA: ca}
if err := s.tlsInShared(context.Background(), Secrets{RestAPI: "pw"}); err != nil {
t.Fatal(err)
}
if user != RestUser || pass != "pw" {
t.Errorf("auth %q %q", user, pass)
}
pg := patched["postgresql"].(map[string]any)
if pg["parameters"].(map[string]any)["ssl"] != "on" {
t.Errorf("parameters %v", pg["parameters"])
}
got := pg["pg_hba"].([]any)
if got[1] != "host all all 127.0.0.1/32 scram-sha-256" || got[2] != hostssl(ClientLine("10.0.0.140")) {
t.Errorf("hba %v", got)
}
// A control plane let in afterwards is let in by TLS.
hba = got
if _, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.150", "pw"); err != nil {
t.Fatal(err)
}
last := patched["postgresql"].(map[string]any)["pg_hba"].([]any)
if last[len(last)-1] != hostssl(ClientLine("10.0.0.150")) {
t.Errorf("let in without TLS: %v", last)
}
}
func TestAControlPlaneWithoutTheAuthorityConnectsByTLSOnceTheClusterHasIt(t *testing.T) {
ssl := "on"
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{"postgresql": map[string]any{"parameters": map[string]any{"ssl": ssl}}})
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
old := restPort
restPort, _ = strconv.Atoi(u.Port())
defer func() { restPort = old }()
a := &machine{answers: map[string]string{"openssl rand": pws, "endpoint health": healthy(3)}}
b := &machine{answers: map[string]string{"sudo cat /root/llmgw-pg": pws}}
up := Cluster{Members: []MemberState{{Name: "llmgw-pg01", Role: "leader", State: "running"},
{Name: "llmgw-ctl02", Role: "replica", State: "streaming"}}}
s := Setup{DB: []Node{{Member{"llmgw-pg01", "127.0.0.1"}, a}, {Member{"llmgw-ctl02", "127.0.0.1"}, b}},
Witness: &Node{ctl, &machine{}}, CAFile: "/etc/llmgw/ca.pem",
Status: func(context.Context, []string) (Cluster, error) { return up, nil }}
dsn, err := s.Run(context.Background())
if err != nil || !strings.Contains(dsn, "sslmode=verify-full") {
t.Fatalf("%s %v", dsn, err)
}
ssl = "off"
if dsn, _ = s.Run(context.Background()); !strings.Contains(dsn, "sslmode=disable") {
t.Errorf("TLS asked of a cluster without it: %s", dsn)
}
}
+116
View File
@@ -0,0 +1,116 @@
package pgha
import (
"bytes"
"context"
"crypto/tls"
"crypto/x509"
"errors"
"fmt"
"io"
"net"
"net/http"
"os"
"strconv"
"strings"
"sync"
"time"
)
// Patroni's API answers over TLS once a member has its certificate, and
// over plain HTTP on a cluster set up before that, or while it is being
// moved. So every request is made over HTTPS first, verified against the
// cluster's authority, and over HTTP only when the member answers in plain
// HTTP — never when the certificate is wrong.
var (
restMu sync.RWMutex
restCA *x509.CertPool
)
// SetRESTCA is the cluster's authority, against which members' API
// certificates are verified: the control plane's ca.pem. Without one, the
// API is asked over plain HTTP.
func SetRESTCA(pemFile string) error {
b, err := os.ReadFile(pemFile) //nolint:gosec // the operator's own CA
if err != nil {
return err
}
return setRESTCAPEM(b)
}
func setRESTCAPEM(b []byte) error {
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(b) {
return errors.New("no certificate in the authority's file")
}
restMu.Lock()
restCA = pool
restMu.Unlock()
return nil
}
// restRequest asks one member's API.
func restRequest(ctx context.Context, host, method, path string, body []byte, password string,
timeout time.Duration,
) (*http.Response, error) {
restMu.RLock()
pool := restCA
restMu.RUnlock()
addr := net.JoinHostPort(host, strconv.Itoa(restPort))
try := func(scheme string, client *http.Client) (*http.Response, error) {
var r io.Reader
if body != nil {
r = bytes.NewReader(body)
}
req, err := http.NewRequestWithContext(ctx, method, scheme+"://"+addr+path, r)
if err != nil {
return nil, err
}
if body != nil {
req.Header.Set("Content-Type", "application/json")
}
if password != "" {
req.SetBasicAuth(RestUser, password)
}
return client.Do(req)
}
plain := &http.Client{Timeout: timeout}
if pool == nil {
return try("http", plain)
}
secure := &http.Client{Timeout: timeout, Transport: &http.Transport{
TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12},
}}
resp, err := try("https", secure)
if err != nil && speaksPlainHTTP(err) {
return try("http", plain)
}
return resp, err
}
// speaksPlainHTTP reports whether a TLS attempt failed because the other
// end answered in plain HTTP: a member whose API has no certificate yet.
func speaksPlainHTTP(err error) bool {
var rh tls.RecordHeaderError
return errors.As(err, &rh) || strings.Contains(err.Error(), "server gave HTTP response to HTTPS client")
}
// restJSON is restRequest for an answer that must be 200, read whole.
func restJSON(ctx context.Context, host, method, path string, body []byte, password string,
timeout time.Duration,
) ([]byte, error) {
resp, err := restRequest(ctx, host, method, path, body, password, timeout)
if err != nil {
return nil, err
}
defer resp.Body.Close()
b, err := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return b, fmt.Errorf("%s %s on %s: %s", method, path, host, resp.Status)
}
return b, nil
}
+65
View File
@@ -0,0 +1,65 @@
package pgha
import (
"crypto/tls"
"encoding/pem"
"net"
"net/http"
"net/http/httptest"
"strconv"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
)
func TestTheAPIIsAskedOverTLSAndOverHTTPOnlyWhenItSpeaksHTTP(t *testing.T) {
answer := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"scheme":"` + map[bool]string{true: "https", false: "http"}[r.TLS != nil] + `"}`))
})
secure := httptest.NewTLSServer(answer)
defer secure.Close()
plain := httptest.NewServer(answer)
defer plain.Close()
t.Cleanup(func() { restMu.Lock(); restCA = nil; restMu.Unlock(); restPort = PatroniPort })
if err := setRESTCAPEM(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: secure.Certificate().Raw})); err != nil {
t.Fatal(err)
}
ask := func(url string) (string, error) {
_, port, _ := net.SplitHostPort(url[len("https://"):])
if url[:5] == "http:" {
_, port, _ = net.SplitHostPort(url[len("http://"):])
}
restPort, _ = strconv.Atoi(port)
b, err := restJSON(t.Context(), "127.0.0.1", http.MethodGet, "/x", nil, "", 3*time.Second)
return string(b), err
}
if got, err := ask(secure.URL); err != nil || got != `{"scheme":"https"}` {
t.Errorf("a member on TLS: %s, %v", got, err)
}
if got, err := ask(plain.URL); err != nil || got != `{"scheme":"http"}` {
t.Errorf("a member not yet on TLS: %s, %v", got, err)
}
// A certificate from elsewhere is refused, not fallen back from.
ca, err := tlsconf.NewCA("elsewhere")
if err != nil {
t.Fatal(err)
}
leaf, err := ca.IssueServer("other", []string{"127.0.0.1"})
if err != nil {
t.Fatal(err)
}
cert, err := tls.X509KeyPair(leaf.CertPEM, leaf.KeyPEM)
if err != nil {
t.Fatal(err)
}
other := httptest.NewUnstartedServer(answer)
other.TLS = &tls.Config{Certificates: []tls.Certificate{cert}}
other.StartTLS()
defer other.Close()
if _, err := ask(other.URL); err == nil {
t.Error("a certificate the authority did not sign was accepted")
}
}
+211 -33
View File
@@ -10,6 +10,7 @@ import (
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
)
// Node is a machine the setup reaches, and how.
@@ -34,6 +35,15 @@ type Setup struct {
Emit func(step, detail string, changed bool)
// Wait bounds each wait for the cluster to come up.
Wait time.Duration
// CA signs each member's certificate; nil signs none and turns nothing
// on. CAFile is where this control plane finds the authority's
// certificate, for the DSN: used when TLS is on, whoever turned it on.
CA *tlsconf.CA
CAFile string
// RestPassword is what Patroni's API asks for before a change, once Run
// has found or made it: for `server add`, which changes the access rules.
RestPassword string
}
// Local is a Remote for the machine this runs on, which init runs as root.
@@ -83,16 +93,41 @@ func (s *Setup) Run(ctx context.Context) (string, error) {
if s.Status == nil {
s.Status = HTTPStatus
}
// Members' APIs are verified against the cluster's authority once they
// speak TLS.
switch {
case s.CA != nil:
if err := setRESTCAPEM(s.CA.PEM); err != nil {
return "", err
}
case s.CAFile != "":
_ = SetRESTCA(s.CAFile)
}
sec, err := s.secrets(ctx)
if err != nil {
return "", fmt.Errorf("passwords: %w", err)
}
if err := s.etcd(ctx); err != nil {
s.RestPassword = sec.RestAPI
if err := s.etcd(ctx, sec); err != nil {
return "", fmt.Errorf("etcd: %w", err)
}
if err := s.certs(ctx); err != nil {
return "", fmt.Errorf("certificates: %w", err)
}
if err := s.patroni(ctx, sec); err != nil {
return "", fmt.Errorf("patroni: %w", err)
}
// What a cluster set up before them lacks, applied in place; a new one
// has them already and these find nothing to change.
if err := s.restAuth(ctx, sec); err != nil {
return "", fmt.Errorf("patroni api: %w", err)
}
if err := s.tlsInShared(ctx, sec); err != nil {
return "", fmt.Errorf("tls: %w", err)
}
if err := s.ensurePatroniTLS(ctx, sec); err != nil {
return "", fmt.Errorf("patroni api: %w", err)
}
if err := s.database(ctx); err != nil {
return "", fmt.Errorf("database: %w", err)
}
@@ -100,52 +135,64 @@ func (s *Setup) Run(ctx context.Context) (string, error) {
for i, d := range s.DB {
members[i] = d.Member
}
return DSN(members, sec.App), nil
// TLS when this setup turned it on, or found it on: then the member
// reached is verified against the authority's certificate, which every
// control plane has even where its key is not.
caFile := ""
if s.CAFile != "" && (s.CA != nil || s.sharedTLS(ctx)) {
caFile = s.CAFile
}
return DSN(members, sec.App, caFile), nil
}
// secretDir holds the passwords on every database machine, readable by root
// alone. Generated on the first, carried to the others.
const secretDir = "/root/llmgw-pg"
var secretNames = []string{"superuser", "replicator", "llmgw"}
var secretNames = []string{"superuser", "replicator", "llmgw", "restapi"}
func (s *Setup) secrets(ctx context.Context) (Secrets, error) {
script := `set -e
sudo install -d -m 700 ` + secretDir + `
for f in superuser replicator llmgw; do
for f in superuser replicator llmgw restapi; do
sudo test -s ` + secretDir + `/$f.pw || sudo sh -c "umask 077; openssl rand -hex 24 > ` + secretDir + `/$f.pw"
done
for f in superuser replicator llmgw; do printf '%s=' $f; sudo cat ` + secretDir + `/$f.pw; done`
for f in superuser replicator llmgw restapi; do printf '%s=' $f; sudo cat ` + secretDir + `/$f.pw; done`
out, err := s.DB[0].Remote.Run(ctx, script, nil)
if err != nil {
return Secrets{}, fmt.Errorf("on %s: %v: %s", s.DB[0].Name, err, strings.TrimSpace(string(out)))
}
got := parseSecrets(out)
if got.Superuser == "" || got.Replicator == "" || got.App == "" {
if got.Superuser == "" || got.Replicator == "" || got.App == "" || got.RestAPI == "" {
return Secrets{}, fmt.Errorf("could not read them back from %s", s.DB[0].Name)
}
for _, d := range s.DB[1:] {
have, err := d.Remote.Run(ctx, `for f in superuser replicator llmgw; do
have, err := d.Remote.Run(ctx, `for f in superuser replicator llmgw restapi; do
if sudo test -s `+secretDir+`/$f.pw; then printf '%s=' $f; sudo cat `+secretDir+`/$f.pw; fi
done`, nil)
if err != nil {
return Secrets{}, fmt.Errorf("on %s: %v", d.Name, err)
}
if h := parseSecrets(have); h != (Secrets{}) {
if h != got {
h := parseSecrets(have)
if h.Superuser != "" || h.Replicator != "" || h.App != "" {
if h.Superuser != got.Superuser || h.Replicator != got.Replicator || h.App != got.App {
// Not overwritten: a member holding other passwords belongs
// to another cluster, or to an earlier one somebody meant to
// keep, and replacing them would lock it out of itself.
return Secrets{}, fmt.Errorf("%s holds passwords in %s that differ from %s's; "+
"remove them there if that machine is meant to join this cluster", d.Name, secretDir, s.DB[0].Name)
}
continue
if h.RestAPI == got.RestAPI {
continue
}
// A cluster from before the API asked for a password: the
// member is given the one the first holds.
}
body := fmt.Sprintf("%s\n%s\n%s\n", got.Superuser, got.Replicator, got.App)
body := fmt.Sprintf("%s\n%s\n%s\n%s\n", got.Superuser, got.Replicator, got.App, got.RestAPI)
if out, err := d.Remote.Run(ctx, `set -e
sudo install -d -m 700 `+secretDir+`
read su; read rep; read app
for kv in "superuser $su" "replicator $rep" "llmgw $app"; do
read su; read rep; read app; read api
for kv in "superuser $su" "replicator $rep" "llmgw $app" "restapi $api"; do
set -- $kv
printf '%s\n' "$2" | sudo sh -c "umask 077; cat > `+secretDir+`/$1.pw"
done`, []byte(body)); err != nil {
@@ -171,6 +218,8 @@ func parseSecrets(out []byte) Secrets {
s.Replicator = v
case "llmgw":
s.App = v
case "restapi":
s.RestAPI = v
}
}
return s
@@ -178,27 +227,117 @@ func parseSecrets(out []byte) Secrets {
// etcd brings up the voting members, unless every one of them is already
// healthy in one cluster — in which case nothing is touched, since rewriting
// a member's configuration means emptying its data.
func (s *Setup) etcd(ctx context.Context) error {
// a member's configuration means emptying its data. With the cluster's
// authority here, members speak TLS; a running cluster that does not yet is
// moved to it (etcdToTLS).
func (s *Setup) etcd(ctx context.Context, sec Secrets) error {
nodes := s.etcdMembers()
var members []Member
var endpoints []string
for _, n := range nodes {
members = append(members, n.Member)
endpoints = append(endpoints, "http://"+n.IP+":2379")
}
if s.etcdHealthy(ctx, nodes[0], endpoints, len(nodes)) {
s.emit("etcd", false, "%d members healthy; left as they are", len(nodes))
tls := s.CA != nil
if s.etcdHealthy(ctx, nodes[0], etcdEndpoints(nodes, tls), len(nodes), tls) {
s.emit("etcd", false, "%d members healthy%s; left as they are", len(nodes), map[bool]string{true: ", over TLS"}[tls])
return nil
}
if tls && s.etcdHealthy(ctx, nodes[0], etcdEndpoints(nodes, false), len(nodes), false) {
return s.etcdToTLS(ctx, sec)
}
return s.etcdBootstrap(ctx, tls)
}
changed := make([]bool, len(nodes))
func etcdEndpoints(nodes []Node, tls bool) []string {
scheme := "http"
if tls {
scheme = "https"
}
out := make([]string, len(nodes))
for i, n := range nodes {
out[i] = scheme + "://" + n.IP + ":2379"
}
return out
}
// etcdInstall puts etcd on every member, stopped if it was not running.
func (s *Setup) etcdInstall(ctx context.Context, nodes []Node) error {
for _, n := range nodes {
out, err := n.Remote.Run(ctx, `set -e
if ! command -v etcd >/dev/null 2>&1; then
sudo DEBIAN_FRONTEND=noninteractive apt-get -qq update >/dev/null
sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y etcd-server etcd-client >/dev/null
fi
fi`, nil)
if err != nil {
return fmt.Errorf("on %s: %v: %s", n.Name, err, strings.TrimSpace(string(out)))
}
}
return nil
}
// etcdCerts gives each etcd member a certificate from the cluster's
// authority, for its address, its name and loopback, unless it has one.
func (s *Setup) etcdCerts(ctx context.Context, nodes []Node) error {
for _, n := range nodes {
have, _ := n.Remote.Run(ctx, "sudo openssl x509 -in "+EtcdCert+" -noout -ext subjectAltName,extendedKeyUsage 2>/dev/null", nil)
if strings.Contains(string(have), "IP Address:"+n.IP) && strings.Contains(string(have), "Client Authentication") {
continue
}
leaf, err := s.CA.IssuePeer(n.Name, []string{n.IP, n.Name, "127.0.0.1", "localhost"})
if err != nil {
return err
}
if err := putTLS(ctx, n, "etcd", EtcdTLSDir, EtcdCA, EtcdCert, EtcdKey, s.CA.PEM, leaf.CertPEM, leaf.KeyPEM); err != nil {
return err
}
s.emit("etcd", true, "certificate for %s (%s)", n.Name, n.IP)
}
return nil
}
// putTLS installs an authority's certificate, a leaf and its key on a
// machine, owned by user, the key readable by it alone.
func putTLS(ctx context.Context, n Node, user, dir, caPath, certPath, keyPath string, ca, cert, key []byte) error {
var body bytes.Buffer
body.Write(ca)
body.WriteString("\x00")
body.Write(cert)
body.WriteString("\x00")
body.Write(key)
out, err := n.Remote.Run(ctx, `set -e
sudo install -d -o `+user+` -g `+user+` -m 750 `+dir+`
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
cat > "$t/all"
python3 - "$t" <<'PY'
import sys
d = sys.argv[1]
ca, crt, key = open(d + "/all", "rb").read().split(b"\x00")
for name, data in (("ca", ca), ("crt", crt), ("key", key)):
open(d + "/" + name, "wb").write(data)
PY
sudo install -o `+user+` -g `+user+` -m 644 "$t/ca" `+caPath+`
sudo install -o `+user+` -g `+user+` -m 644 "$t/crt" `+certPath+`
sudo install -o `+user+` -g `+user+` -m 600 "$t/key" `+keyPath, body.Bytes())
if err != nil {
return fmt.Errorf("on %s: %v: %s", n.Name, err, strings.TrimSpace(string(out)))
}
return nil
}
// etcdBootstrap writes every member's configuration and starts them
// together, emptying a member whose configuration changed.
func (s *Setup) etcdBootstrap(ctx context.Context, tls bool) error {
nodes := s.etcdMembers()
var members []Member
for _, n := range nodes {
members = append(members, n.Member)
}
if err := s.etcdInstall(ctx, nodes); err != nil {
return err
}
if tls {
if err := s.etcdCerts(ctx, nodes); err != nil {
return err
}
}
changed := make([]bool, len(nodes))
for i, n := range nodes {
out, err := n.Remote.Run(ctx, `set -e
new=$(cat)
if [ "$(sudo cat /etc/default/etcd 2>/dev/null)" != "$new" ]; then
sudo systemctl stop etcd >/dev/null 2>&1 || true
@@ -206,7 +345,7 @@ if [ "$(sudo cat /etc/default/etcd 2>/dev/null)" != "$new" ]; then
printf '%s\n' "$new" | sudo tee /etc/default/etcd >/dev/null
echo changed
fi
sudo systemctl enable etcd >/dev/null 2>&1`, []byte(Etcd(n.Member, members)))
sudo systemctl enable etcd >/dev/null 2>&1`, []byte(Etcd(n.Member, members, tls)))
if err != nil {
return fmt.Errorf("on %s: %v: %s", n.Name, err, strings.TrimSpace(string(out)))
}
@@ -237,7 +376,7 @@ sudo systemctl enable etcd >/dev/null 2>&1`, []byte(Etcd(n.Member, members)))
}
}
deadline := time.Now().Add(s.Wait)
for !s.etcdHealthy(ctx, nodes[0], endpoints, len(nodes)) {
for !s.etcdHealthy(ctx, nodes[0], etcdEndpoints(nodes, tls), len(nodes), tls) {
if time.Now().After(deadline) {
return fmt.Errorf("the %d members did not become healthy within %s; `journalctl -u etcd` on each says why", len(nodes), s.Wait)
}
@@ -245,16 +384,48 @@ sudo systemctl enable etcd >/dev/null 2>&1`, []byte(Etcd(n.Member, members)))
return err
}
}
s.emit("etcd", true, "%d members up", len(nodes))
s.emit("etcd", true, "%d members up%s", len(nodes), map[bool]string{true: ", over TLS"}[tls])
return nil
}
func (s *Setup) etcdHealthy(ctx context.Context, n Node, endpoints []string, want int) bool {
out, err := n.Remote.Run(ctx, "ETCDCTL_API=3 etcdctl --dial-timeout=3s --endpoints="+
strings.Join(endpoints, ",")+" endpoint health 2>&1", nil)
func (s *Setup) etcdHealthy(ctx context.Context, n Node, endpoints []string, want int, tls bool) bool {
out, err := n.Remote.Run(ctx, etcdctl(tls, endpoints)+" endpoint health 2>&1", nil)
return err == nil && strings.Count(string(out), "is healthy") == want
}
// ensurePatroniTLS puts a member whose Patroni does not yet reach etcd or
// serve its API over TLS on it, in maintenance mode: an etcd moved to TLS by
// a run that stopped before it got this far.
func (s *Setup) ensurePatroniTLS(ctx context.Context, sec Secrets) error {
if s.CA == nil {
return nil
}
need := false
for _, d := range s.DB {
out, err := d.Remote.Run(ctx, patroniTLSEdit("check"), nil)
if err != nil {
return fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
}
need = need || strings.TrimSpace(lastLine(out)) == "changed"
}
if !need {
return nil
}
if err := s.pause(ctx, sec, true); err != nil {
return err
}
if _, err := s.patroniOnTLS(ctx, true); err != nil {
return s.stuck(err)
}
if err := s.waitFor(ctx, s.dbHosts(), func(c Cluster) bool { return c.Leader() != "" }); err != nil {
return s.stuck(err)
}
if err := s.pause(ctx, sec, false); err != nil {
return s.stuck(err)
}
return s.waitFor(ctx, s.dbHosts(), func(c Cluster) bool { return c.Complete(len(s.DB)) })
}
// patroni installs PostgreSQL and Patroni on every database machine, unless
// the cluster already has a leader and every member streaming.
func (s *Setup) patroni(ctx context.Context, sec Secrets) error {
@@ -279,6 +450,10 @@ func (s *Setup) patroni(ctx context.Context, sec Secrets) error {
}
version = v
}
// Now that PostgreSQL's user exists, its certificates.
if err := s.certs(ctx); err != nil {
return err
}
var etcd, db []Member
for _, n := range s.etcdMembers() {
@@ -288,7 +463,7 @@ func (s *Setup) patroni(ctx context.Context, sec Secrets) error {
db = append(db, d.Member)
}
for i, d := range s.DB {
cfg := Patroni(d.Member, db, etcd, s.Clients, version, sec)
cfg := Patroni(d.Member, db, etcd, s.Clients, version, sec, s.CA != nil)
if out, err := d.Remote.Run(ctx, `set -e
sudo install -d -m 755 /etc/patroni
sudo install -d -o postgres -g postgres -m 700 /var/lib/postgresql/`+version+`/llmgw
@@ -373,6 +548,7 @@ func (s *Setup) database(ctx context.Context) error {
}
out, err := leader.Remote.Run(ctx, `set -e
pw=$(sudo cat `+secretDir+`/llmgw.pw)
sudo -u postgres psql -Atc "select 'had=' || (select count(*) from pg_roles where rolname = 'llmgw') + (select count(*) from pg_database where datname = 'llmgw')"
sudo -u postgres psql -q -v ON_ERROR_STOP=1 -v pw="$pw" <<'SQL'
SELECT format('CREATE ROLE %I LOGIN', 'llmgw') WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'llmgw') \gexec
ALTER ROLE llmgw PASSWORD :'pw';
@@ -381,7 +557,9 @@ SQL`, nil)
if err != nil {
return fmt.Errorf("on %s: %v: %s", leader.Name, err, strings.TrimSpace(string(out)))
}
s.emit("database", true, "role and database %s on %s", Database, leader.Name)
// Said as a change only when something was made: a rerun that finds
// both is not a reason to restart the gateway.
s.emit("database", !strings.Contains(string(out), "had=2"), "role and database %s on %s", Database, leader.Name)
return nil
}
+3 -13
View File
@@ -4,9 +4,7 @@ import (
"context"
"encoding/json"
"fmt"
"net"
"net/http"
"strconv"
"strings"
"time"
)
@@ -70,24 +68,16 @@ func (c Cluster) String() string {
// HTTPStatus asks each host's Patroni in turn and returns the first answer:
// any member describes the whole cluster.
func HTTPStatus(ctx context.Context, hosts []string) (Cluster, error) {
client := &http.Client{Timeout: 3 * time.Second}
var last error
for _, h := range hosts {
u := "http://" + net.JoinHostPort(h, strconv.Itoa(restPort)) + "/cluster"
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
if err != nil {
return Cluster{}, err
}
resp, err := client.Do(req)
b, err := restJSON(ctx, h, http.MethodGet, "/cluster", nil, "", 3*time.Second)
if err != nil {
last = err
continue
}
var c Cluster
err = json.NewDecoder(resp.Body).Decode(&c)
resp.Body.Close()
if err != nil {
last = fmt.Errorf("%s: %w", u, err)
if err := json.Unmarshal(b, &c); err != nil {
last = fmt.Errorf("%s /cluster: %w", h, err)
continue
}
for i := range c.Members {
+236
View File
@@ -0,0 +1,236 @@
package pgha
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"net/http"
"strings"
"time"
)
// Moving a running cluster's etcd to TLS cannot be done member by member:
// a member's peers reach it at the address the cluster recorded, http, and
// changing that one member at a time loses the majority half way. So it is
// done the way a first install is — every member's configuration rewritten
// and started together, empty — with what Patroni keeps there carried over.
//
// While etcd is down Patroni would demote the primary. So the cluster is in
// maintenance mode (pause) throughout: Patroni stops managing PostgreSQL,
// which goes on serving, and a lost leader lock means nothing. Patroni is
// then restarted against the new etcd, which in maintenance mode leaves
// PostgreSQL running, and maintenance ends.
// etcdToTLS moves a healthy plain-HTTP etcd, and Patroni with it, to TLS.
func (s *Setup) etcdToTLS(ctx context.Context, sec Secrets) error {
nodes := s.etcdMembers()
if err := s.etcdCerts(ctx, nodes); err != nil {
return err
}
if err := s.certs(ctx); err != nil {
return err
}
if err := s.pause(ctx, sec, true); err != nil {
return fmt.Errorf("maintenance mode: %w", err)
}
s.emit("etcd", true, "Patroni in maintenance mode: PostgreSQL serves on while etcd moves to TLS")
kept, err := s.etcdDump(ctx, nodes[0])
if err != nil {
return s.stuck(fmt.Errorf("reading what Patroni keeps in etcd: %w", err))
}
if err := s.etcdBootstrap(ctx, true); err != nil {
return s.stuck(err)
}
if err := s.etcdRestore(ctx, nodes[0], kept); err != nil {
return s.stuck(fmt.Errorf("putting Patroni's keys back: %w", err))
}
s.emit("etcd", true, "%d of Patroni's keys carried over", len(kept))
if _, err := s.patroniOnTLS(ctx, true); err != nil {
return s.stuck(err)
}
hosts := s.dbHosts()
if err := s.waitFor(ctx, hosts, func(c Cluster) bool { return c.Leader() != "" }); err != nil {
return s.stuck(fmt.Errorf("Patroni against the new etcd: %w", err))
}
if err := s.pause(ctx, sec, false); err != nil {
return s.stuck(fmt.Errorf("ending maintenance mode: %w", err))
}
if err := s.waitFor(ctx, hosts, func(c Cluster) bool { return c.Complete(len(s.DB)) }); err != nil {
return err
}
s.emit("etcd", true, "etcd and Patroni's API speak TLS; maintenance mode ended")
return nil
}
// stuck says where a move that failed half way leaves the cluster.
func (s *Setup) stuck(err error) error {
return fmt.Errorf("%w — the cluster is left in maintenance mode, PostgreSQL still serving; "+
"run this again, or end it with `patronictl -c /etc/patroni/config.yml resume` on a member", err)
}
func (s *Setup) dbHosts() []string {
hosts := make([]string, len(s.DB))
for i, d := range s.DB {
hosts[i] = d.IP
}
return hosts
}
// pause turns maintenance mode on or off, and waits until every member says
// so.
func (s *Setup) pause(ctx context.Context, sec Secrets, on bool) error {
var err error
for _, h := range s.dbHosts() {
if err = patchConfig(ctx, h, sec.RestAPI, map[string]any{"pause": on}); err == nil {
break
}
}
if err != nil {
return err
}
deadline := time.Now().Add(s.Wait)
for {
all := true
for _, h := range s.dbHosts() {
b, err := restJSON(ctx, h, http.MethodGet, "/patroni", nil, "", 5*time.Second)
var st struct {
Pause bool `json:"pause"`
}
if err != nil || json.Unmarshal(b, &st) != nil || st.Pause != on {
all = false
}
}
if all {
return nil
}
if time.Now().After(deadline) {
return fmt.Errorf("not every member was in maintenance mode=%v within %s", on, s.Wait)
}
if err := sleep(ctx, 2*time.Second); err != nil {
return err
}
}
}
// etcdKV is one key Patroni keeps, and whether it is tied to a lease: a
// leader lock or a member's entry, which Patroni writes again itself and
// which, put back without its lease, would never expire.
type etcdKV struct {
Key, Value []byte
Lease int64
}
func (s *Setup) etcdDump(ctx context.Context, n Node) ([]etcdKV, error) {
out, err := n.Remote.Run(ctx, etcdctl(false, []string{"http://" + n.IP + ":2379"})+
" get --prefix /service/ -w json", nil)
if err != nil {
return nil, fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
}
var got struct {
KVs []struct {
Key string `json:"key"`
Value string `json:"value"`
Lease int64 `json:"lease"`
} `json:"kvs"`
}
if err := json.Unmarshal(out, &got); err != nil {
return nil, err
}
var kvs []etcdKV
for _, kv := range got.KVs {
k, err1 := base64.StdEncoding.DecodeString(kv.Key)
v, err2 := base64.StdEncoding.DecodeString(kv.Value)
if err1 != nil || err2 != nil {
return nil, fmt.Errorf("an unreadable key in etcd's answer")
}
if kv.Lease == 0 {
kvs = append(kvs, etcdKV{Key: k, Value: v})
}
}
if len(kvs) == 0 {
return nil, fmt.Errorf("etcd holds nothing of Patroni's under /service/")
}
return kvs, nil
}
func (s *Setup) etcdRestore(ctx context.Context, n Node, kvs []etcdKV) error {
for _, kv := range kvs {
if strings.ContainsAny(string(kv.Key), "'\n") {
return fmt.Errorf("a key that cannot be written back: %q", kv.Key)
}
out, err := n.Remote.Run(ctx, etcdctl(true, []string{"https://" + n.IP + ":2379"})+
" put '"+string(kv.Key)+"'", kv.Value)
if err != nil {
return fmt.Errorf("%s: %v: %s", kv.Key, err, strings.TrimSpace(string(out)))
}
}
return nil
}
// patroniOnTLS has each member's Patroni reach etcd over TLS and serve its
// API over TLS, restarting those whose configuration changed. With paused
// false, maintenance mode is entered for the restart and left after.
func (s *Setup) patroniOnTLS(ctx context.Context, paused bool) (bool, error) {
var todo []Node
for _, d := range s.DB {
out, err := d.Remote.Run(ctx, patroniTLSEdit("check"), nil)
if err != nil {
return false, fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
}
switch strings.TrimSpace(lastLine(out)) {
case "changed":
todo = append(todo, d)
case "missing":
return false, fmt.Errorf("%s has no /etc/patroni/config.yml", d.Name)
}
}
if len(todo) == 0 {
return false, nil
}
if !paused {
return false, fmt.Errorf("Patroni on %d member(s) is not yet on TLS, which needs maintenance mode", len(todo))
}
for _, d := range todo {
if out, err := d.Remote.Run(ctx, patroniTLSEdit("apply")+`
sudo systemctl restart patroni`, nil); err != nil {
return false, fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
}
s.emit("patroni api", true, "%s reaches etcd over TLS and serves its API over TLS", d.Name)
}
return true, nil
}
// patroniTLSEdit edits /etc/patroni/config.yml: "check" says whether it
// would change, "apply" writes it.
func patroniTLSEdit(mode string) string {
return `set -e
test -f /etc/patroni/config.yml || { echo missing; exit 0; }
sudo python3 - ` + mode + ` <<'PY'
import sys, yaml
p = "/etc/patroni/config.yml"
c = yaml.safe_load(open(p))
want = {
"restapi": {"certfile": "` + ServerCert + `", "keyfile": "` + ServerKey + `", "cafile": "` + CACert + `"},
"ctl": {"cacert": "` + CACert + `"},
"etcd3": {"protocol": "https", "cacert": "` + CACert + `", "cert": "` + ServerCert + `", "key": "` + ServerKey + `"},
}
changed = False
for section, kv in want.items():
sec = c.setdefault(section, {})
for k, v in kv.items():
if sec.get(k) != v:
sec[k] = v
changed = True
if sys.argv[1] == "apply" and changed:
open(p + ".new", "w").write("# llmgw: the control planes' store, kept by Patroni, its leader lock in etcd.\n" +
yaml.safe_dump(c, default_flow_style=False, sort_keys=False))
import os, shutil
shutil.chown(p + ".new", "root", "postgres")
os.chmod(p + ".new", 0o640)
os.replace(p + ".new", p)
print("changed" if changed else "same")
PY`
}
Loaded 100 of 149 files, more files were not shown because too many files have changed in this diff. Show more