Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
379074b6a5
149 files changed
+5998
-509
No files matched your search
@@ -93,11 +93,10 @@ jobs:
|
||||
./dist/llmgw-${version}-linux-amd64 --version
|
||||
cat dist/checksums.txt
|
||||
|
||||
# Signed when the repository has a signing key, so a machine that trusts
|
||||
# it installs nothing the server alone could have put there. Without the
|
||||
# secret nothing is signed and releases are checked on their checksums,
|
||||
# as before; a build that trusts a key refuses those, so the secret has
|
||||
# to be in place before trusted.pub is filled in.
|
||||
# Signed, so a machine that trusts the key built into it installs
|
||||
# nothing the server alone could have put there. `release sign` checks
|
||||
# the signature against that key before writing it, so a secret that
|
||||
# does not match trusted.pub stops the release too.
|
||||
- name: Sign the checksums
|
||||
env:
|
||||
RELEASE_SIGNING_KEY: ${{ secrets.RELEASE_SIGNING_KEY }}
|
||||
@@ -105,7 +104,15 @@ jobs:
|
||||
set -euo pipefail
|
||||
version="${GITHUB_REF_NAME:-$(git describe --tags --always)}"
|
||||
if [ -z "${RELEASE_SIGNING_KEY:-}" ]; then
|
||||
echo "no RELEASE_SIGNING_KEY secret; releases are not signed"
|
||||
# Builds that carry a trusted key install only signed releases:
|
||||
# publishing an unsigned one would be a release no upgraded
|
||||
# machine accepts, so it stops here instead.
|
||||
if [ -s internal/release/trusted.pub ]; then
|
||||
echo "internal/release/trusted.pub holds a key, and the RELEASE_SIGNING_KEY secret is not set:"
|
||||
echo "add it under Settings -> Secrets, or this release could not be installed anywhere."
|
||||
exit 1
|
||||
fi
|
||||
echo "no RELEASE_SIGNING_KEY secret and no trusted key; releases are not signed"
|
||||
exit 0
|
||||
fi
|
||||
./dist/llmgw-${version}-linux-amd64 release sign dist/checksums.txt
|
||||
|
||||
@@ -19,6 +19,14 @@ Conventions this codebase keeps, which every item here must keep too:
|
||||
|
||||
## A. Security — do first
|
||||
|
||||
**Status (2026-10-04): done and merged.** A1–A9, A6b included — etcd and
|
||||
Patroni's API on TLS, the PoC moved in maintenance mode without a failover.
|
||||
Deliberately narrower than written: the admin API is a token on loopback
|
||||
rather than a unix socket (A5); thread ids are bounded and documented as
|
||||
project-wide rather than scoped per user (A8); Python engines resolve with an
|
||||
explicit PyTorch index and a PEP 751 lock rather than locks shipped per engine
|
||||
version (A8). All of A is live on the PoC.
|
||||
|
||||
### A1. Toolchain (S, hours)
|
||||
`govulncheck` finds 11 standard-library vulnerabilities reachable from this
|
||||
code at Go 1.26.3 (net/http, crypto/x509, os.Root, net/textproto). Raise the
|
||||
|
||||
+215
@@ -23,6 +23,221 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Security
|
||||
|
||||
- **Built with Go 1.26.6.** Eleven standard-library vulnerabilities reachable
|
||||
from this code at 1.26.3 (net/http, crypto/x509, os.Root, net/textproto)
|
||||
are fixed by it. `make vuln` runs govulncheck; `make check` runs it with the
|
||||
tests and vet.
|
||||
- **Releases are signed, and only signed releases are installed.** The build
|
||||
now carries the release signing key (`internal/release/trusted.pub`); the
|
||||
control plane upgrading itself, an agent told to upgrade and a node being
|
||||
added all refuse a release whose checksums are not signed with it. The CI
|
||||
release job stops rather than publish an unsigned release. Adding a node
|
||||
no longer has the node fetch the release and compare it with a checksum
|
||||
from the same server: the control plane fetches it for the node's
|
||||
architecture, checks the signature and the checksum, and hands it over.
|
||||
Releases are fetched over https only (loopback excepted), and
|
||||
`LLMGW_RELEASE_KEY` replaces the trusted key in development builds only.
|
||||
- **The agent port is not opened without TLS by accident.** A control plane
|
||||
with no `agent_tls` refused nothing: its agent port was plaintext, on every
|
||||
interface, and any machine reaching it could register as any node — be
|
||||
given replicas and so be sent prompts. It now refuses to start unless
|
||||
`agent_insecure: true` is set, and then listens on loopback unless
|
||||
`agent_listen` names an address. `server init` writes `agent_tls`, as
|
||||
before.
|
||||
- **A model definition cannot run code on its nodes.** A profile's `env` and
|
||||
`args` reach a process the agent runs as root. The environment was checked
|
||||
against a short denylist that missed NCCL's and CUDA's plugin and
|
||||
injection libraries; arguments were not checked at all, so
|
||||
`--trust-remote-code`, llama.cpp's `--log-file` or `--slot-save-path`, or
|
||||
vLLM's `--tool-parser-plugin` were taken. `env` is now an allowlist of the
|
||||
families engines are tuned with (CUDA_, NCCL_, VLLM_, OMP_, HF_…) less any
|
||||
name for a path, plugin, library, address or credential, and llama.cpp's
|
||||
`LLAMA_ARG_*` are checked as the flags they are; `args` refuse what reads,
|
||||
writes or loads files, loads code, or moves what the agent sets (host,
|
||||
port, key, model). Both are refused where a model is defined and again on
|
||||
the node before the engine starts.
|
||||
- **Engines no longer run as root.** The agent starts them as the system
|
||||
user `llmgw-engine` (in the video and render groups for the GPUs), with a
|
||||
home and caches of its own under the agent's work directory. `llmgw node
|
||||
add` makes the user; a node without it keeps running engines as root and
|
||||
the agent says so in its log. `supervisor.run_as` in agent.yaml names
|
||||
another user, or `root`.
|
||||
- **The admin API wants a token.** It listened on loopback with no
|
||||
authentication, so any process or user on the control plane could add a
|
||||
model, upgrade the fleet, read the engines' keys or dump the heap through
|
||||
pprof. Its `/api` and `/debug` paths now demand the token the server
|
||||
writes, 0600, to its state directory (`/var/lib/llmgw/admin-token`) on
|
||||
first start; `llmgw ctl`, the console and `node add` read it, so they run
|
||||
with sudo, and `LLMGW_ADMIN_TOKEN` or `ctl --token-file` give it from
|
||||
elsewhere. pprof is off unless `debug_pprof: true`. `admin_listen` off
|
||||
loopback is refused unless `admin_remote: true`. The read-only dashboard
|
||||
pages, `/metrics` and `/health` are unchanged.
|
||||
- **The replicated store asks for a password and speaks TLS.** Patroni's
|
||||
REST API took changes from anyone who reached port 8008 — rewriting the
|
||||
access rules to `trust`, or moving the leader — and PostgreSQL traffic
|
||||
between the machines was plaintext (`sslmode=disable`). Patroni now asks
|
||||
for a password before a change (kept with the cluster's others in
|
||||
`/root/llmgw-pg`, and on the control plane in
|
||||
`/etc/llmgw/secrets/patroni-rest` for `server add`); each member has a
|
||||
certificate from the cluster's authority, PostgreSQL has `ssl` on, every
|
||||
remote access rule is `hostssl`, and the control planes connect with
|
||||
`sslmode=verify-full` against the authority's certificate. A new cluster is
|
||||
set up this way; an existing one is changed in place, by reload, the
|
||||
cluster serving throughout, by running **Replicate the store** in the
|
||||
console again — or `sudo --preserve-env=SSH_AUTH_SOCK llmgw server init
|
||||
--patroni <user>@<a> --patroni <user>@<b> --no-mirror --no-node
|
||||
--no-model` — on the control plane that holds the authority, then on each
|
||||
other control plane so it reconnects by TLS. etcd is not yet given TLS:
|
||||
keep ports 2379 and 2380 to the cluster's machines.
|
||||
- **A capacity fallback respects the key's allowed models, and is charged
|
||||
at its own price.** When the model asked for had no capacity, the next in
|
||||
its `fallbacks` served the request even if the key was not allowed it, and
|
||||
the request was priced as the model asked for. Fallbacks a key may not use
|
||||
are now skipped, and a request is charged what the model that answered
|
||||
costs.
|
||||
- **Suspending an account stops its keys.** Suspension closed the person's
|
||||
sessions and left every API key they had made working. A key whose owner's
|
||||
account is not active — suspended, pending, or gone — is now refused as a
|
||||
revoked key is, within `auth.cache_ttl`, and works again when the account
|
||||
is reinstated.
|
||||
- **A join code is a credential only for its hour, and only on its
|
||||
address.** The setup checked a code's expiry on the joining side alone, so
|
||||
a code stayed good for as long as the setup ran; it now refuses an expired
|
||||
one itself. Its listener binds the address the codes name rather than
|
||||
every interface, and a joined machine is known by the address it connected
|
||||
from, not the one it claimed.
|
||||
- **The setup's sudo grant cannot outlive it.** Given a sudo password, the
|
||||
setup writes a temporary `NOPASSWD` grant and removes it when it
|
||||
finishes; a setup that crashed or was interrupted left it for good. The
|
||||
grant now schedules its own removal within the hour as it is written, and
|
||||
the password is dropped once the grant stands in for it. An address given
|
||||
to the system ssh can no longer be read as one of its options.
|
||||
- **Made-up keys cannot fill memory or the database.** Every well-formed key
|
||||
presented, real or not, was cached for good and cost a database read the
|
||||
first time. The cache is now bounded, and keys not in it are looked up at
|
||||
most 20 a second per client address (50 at once); beyond that the request
|
||||
is answered 429 without a read.
|
||||
- **Slow or idle connections cannot pile up.** The servers bounded only how
|
||||
long headers took. Every listener now also bounds how long a connection
|
||||
may sit idle and how large its headers may be, and reading a request
|
||||
(5 minutes on the client port, a minute or less elsewhere — never on the
|
||||
agents' long-lived streams); the client port holds at most
|
||||
`max_connections` (10,000) at once, and a message from an agent at most
|
||||
16 MB.
|
||||
- **The sign-in and signup forms resist scripts.** Ten wrong passwords from
|
||||
anywhere locked an address's owner out for the window; attempts are now
|
||||
counted per address and client, so somebody guessing elsewhere cannot lock
|
||||
the owner out, with a looser count per address alone so guesses spread
|
||||
over many clients are still bounded. The sign-in form carries a token its
|
||||
cookie matches, so another site cannot sign a visitor into an account of
|
||||
its choosing. Posting either form is rationed per client address (sign-in
|
||||
30 a minute, signup 5), and a password hash that cannot get a turn within
|
||||
ten seconds answers "busy" instead of holding the request. `server init`
|
||||
now writes `signup_open: false`.
|
||||
- **Forwarding headers are believed only from the proxies named.**
|
||||
`X-Forwarded-Host` and `-Proto` were taken from anyone, and decide the
|
||||
gateway address written into the `/setup` scripts clients pipe to a
|
||||
shell; behind a caching proxy one request could poison them for others.
|
||||
They — and `X-Forwarded-For`, now used for the per-address limits — are
|
||||
believed only from `router.trusted_proxies` (addresses or CIDR prefixes).
|
||||
**Behind a reverse proxy, set it**, or the pages and scripts name the
|
||||
address the proxy reached and every client counts as the proxy. `/setup`
|
||||
scripts are served `no-store`.
|
||||
- **Removing a node revokes its certificate.** `ctl node remove` only
|
||||
dropped the node from the inventory; its certificate, valid for a year,
|
||||
still connected and was given work. Now the removal is recorded in the
|
||||
store and the agent port refuses any certificate issued for that node
|
||||
before it — onboarding the node again issues one that works. A mirror
|
||||
serving its management API with `--tls-ca` now requires `--api-client`
|
||||
(the control plane's name), since the same authority signs every node.
|
||||
- **Engines listen on the management address only**, the one the control
|
||||
plane reaches them on, instead of every address the node has — a public
|
||||
one included. (When that address is a name, or not one of the node's,
|
||||
they still listen on all.) **SGLang's key is off its command line**,
|
||||
where every user on the node could read it: it reaches the engine in its
|
||||
environment, and a short launcher hands it to SGLang inside the process.
|
||||
Both take effect as engines restart under an upgraded agent.
|
||||
- **Backups can be encrypted.** `backup_recipients` in server.yaml (age
|
||||
recipients or SSH public keys) has `llmgw server backup` encrypt each
|
||||
archive to them with age, as `.tar.gz.age`: an archive holds the CA's key
|
||||
and every secret in the store, and is meant to be copied off the machine.
|
||||
Open one with `age -d -i <key>`.
|
||||
- **PyTorch's index gives torch and nothing else.** vLLM's CUDA 12.9 build
|
||||
was resolved with PyTorch's index beside PyPI and `unsafe-best-match`, so
|
||||
either index could supply any package — how torchtriton was hijacked in
|
||||
2022. Now that index is explicit, named only for torch, torchvision and
|
||||
torchaudio; every other package comes from PyPI alone, and the lock is
|
||||
PEP 751's (`pylock.llmgw.toml`), pinning each file's address as well as
|
||||
its hash. Checked against vLLM 0.30.0: the same 195 packages resolve.
|
||||
- **The control plane's own fetches stay off link-local addresses.**
|
||||
Checking a model's source and reading a GGUF's header went wherever the
|
||||
URL said, cloud instance metadata (169.254.169.254) included. They now
|
||||
refuse link-local, multicast and unspecified addresses, while still
|
||||
reaching the cluster's network, where the mirror is. The mirror's fetches,
|
||||
which refuse private addresses too, now also see through IPv6 addresses
|
||||
carrying an IPv4 one (NAT64, 6to4, IPv4-compatible), and through a proxy:
|
||||
the target's addresses are checked before the request is handed to it.
|
||||
- **A CPU-only node reporting a problem no longer panics the agent
|
||||
handler**, and **thread ids are bounded** to 128 letters, digits and
|
||||
`- _ . :`. A thread is readable by every key of its project that knows
|
||||
the id; the docs now say to use unguessable ids.
|
||||
- **`/v1/models` lists only what the key may use**, rather than every
|
||||
model the gateway serves to anyone.
|
||||
- **Portal sessions end after a day unused** (still at most 14 days in
|
||||
all), and expired or idle sessions and used or expired key claims are
|
||||
deleted hourly — they were refused, but never removed.
|
||||
- **Every page carries a Content-Security-Policy**: scripts only from the
|
||||
gateway itself and the page's own inline ones, each named by its hash —
|
||||
no inline handlers, no eval, nothing a model's reply or a model card
|
||||
could slip into a page. The few `onchange`/`onsubmit` attributes moved
|
||||
to `/_static/pages.js`, htmx runs with `allowEval: false`, and the live
|
||||
pages' "only while visible" refresh no longer needs an evaluated filter.
|
||||
Pages now carry their headers even when answered with an error status.
|
||||
- **KaTeX 0.16.47** (from 0.16.11), past the advisories fixed in 0.16.21.
|
||||
- **Smaller hardening.** The mirror no longer serves its bookkeeping files
|
||||
(a file's `.source` URL and `.sha256`) on `:8081`. A GGUF header nesting
|
||||
arrays more than 4 deep is refused rather than recursed into. Serve and
|
||||
update jobs keep the last 100 finished, and the mirror forgets finished
|
||||
downloads after a week, instead of remembering every one until restart.
|
||||
- **A firewall for llmgw's internal ports.** `server init` (and the store
|
||||
members it sets up with `--patroni`), `node add`, `server add` and
|
||||
`mirror init` now keep the machine's internal ports — the agents' stream,
|
||||
engines, the mirror, PostgreSQL, Patroni, etcd, node exporter — to the
|
||||
cluster's own networks, in an nftables table of llmgw's own loaded by
|
||||
`llmgw-firewall.service`. Only those ports are touched: the table accepts
|
||||
everything else, so other services on the machine and any firewall it
|
||||
already has are left as they were. Extra networks go in
|
||||
`firewall_networks`; `--no-firewall` skips it; `llmgw doctor` says when a
|
||||
node lacks it.
|
||||
- **etcd and Patroni's API speak TLS.** etcd's peers and clients use
|
||||
certificates from the cluster's authority, and a client must present one;
|
||||
Patroni reaches it with its member's certificate and serves its API over
|
||||
TLS. `server init --patroni`, run again, moves an existing cluster: with
|
||||
Patroni in maintenance mode, so PostgreSQL serves on and nothing fails
|
||||
over, etcd is rebuilt on TLS and Patroni's keys carried into it. The
|
||||
control plane asks the API over HTTPS, verified, and over HTTP only where a
|
||||
member has not moved yet.
|
||||
- **A cluster keeping its access rules per member got no `hostssl`.** The
|
||||
TLS step looked only at the shared configuration, and on a cluster set up
|
||||
before the rules moved there said the rules were `hostssl` while
|
||||
plaintext was still accepted. Each member's own rules are now made
|
||||
`hostssl` too.
|
||||
|
||||
|
||||
### Fixed
|
||||
|
||||
- **An engine that died after an agent restart went unnoticed.** The watcher
|
||||
that reports a ready engine dying was tied to the stream it was started
|
||||
on, so after a reconnect nothing reported it, and an engine adopted from
|
||||
the previous agent had no watcher at all: its replica stayed "ready" and
|
||||
was routed to. Every connection now watches every ready shard, started or
|
||||
adopted. Found on the PoC: a killed engine on node06 stayed ready for
|
||||
minutes.
|
||||
- `server init` rewriting the store's address no longer leaves
|
||||
`/etc/llmgw/secrets` unreadable to the service when it was made 0700 for
|
||||
the service user, which kept the gateway from starting.
|
||||
### Everything that can be changed on a running cluster, from the console
|
||||
|
||||
`sudo llmgw` only watched the cluster and edited the configuration files;
|
||||
|
||||
@@ -2,7 +2,7 @@ MODULE := gitea.mynodes.xyz/ilker/llmgw
|
||||
VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
|
||||
LDFLAGS := -X '$(MODULE)/internal/version.Version=$(VERSION)'
|
||||
|
||||
.PHONY: build test race lint fmt clean rename proto proto-tools
|
||||
.PHONY: build test race lint fmt vuln check clean rename proto proto-tools
|
||||
|
||||
build:
|
||||
go build -ldflags "$(LDFLAGS)" -o bin/llmgw ./cmd/llmgw
|
||||
@@ -20,6 +20,17 @@ fmt:
|
||||
go fmt ./...
|
||||
go vet ./...
|
||||
|
||||
# Known vulnerabilities reachable from this code, in the standard library and
|
||||
# in dependencies. A toolchain left behind is the commonest cause: the go
|
||||
# directive in go.mod is what fixes it.
|
||||
vuln:
|
||||
go run golang.org/x/vuln/cmd/govulncheck@latest ./...
|
||||
|
||||
# What has to pass before a change is merged.
|
||||
check: test
|
||||
go vet ./...
|
||||
$(MAKE) vuln
|
||||
|
||||
# Regenerate the agent protocol. Generated code is committed, so this only
|
||||
# needs running when api/proto changes.
|
||||
proto: proto-tools
|
||||
|
||||
@@ -17,8 +17,10 @@ agent_listen: ":7443"
|
||||
|
||||
# Mutual TLS on the agent port (plan §5.6). With it, a node's identity is taken
|
||||
# from its certificate's common name and an agent claiming a different node id
|
||||
# is refused. Without it the port is plaintext HTTP/2 and anyone who can reach
|
||||
# it can claim to be any node, so it must stay on a trusted network.
|
||||
# is refused. Without it the port would be plaintext HTTP/2 and anyone who can
|
||||
# reach it could claim to be any node, so the control plane refuses to start
|
||||
# unless `agent_insecure: true` is set — and then listens on loopback unless
|
||||
# agent_listen names an address. That is for one machine or a test.
|
||||
#
|
||||
# These certificates come from llmgw's own authority, not a public one: public
|
||||
# authorities stopped issuing the clientAuth usage, so no Let's Encrypt
|
||||
|
||||
@@ -65,9 +65,13 @@ hand it certificates:
|
||||
llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
|
||||
--tls-ca /etc/llmgw/ca.pem \
|
||||
--tls-cert /etc/llmgw/mirror.pem \
|
||||
--tls-key /etc/llmgw/mirror-key.pem
|
||||
--tls-key /etc/llmgw/mirror-key.pem \
|
||||
--api-client control-plane
|
||||
```
|
||||
|
||||
`--api-client` is required with `--tls-ca`: it names the one certificate the
|
||||
API answers, because the CA signs every node's certificate as well.
|
||||
|
||||
That is the same CA the agents already use, so the certificates come from the
|
||||
authority you have:
|
||||
|
||||
|
||||
+87
-18
@@ -78,7 +78,7 @@ llmgw --version
|
||||
`CGO_ENABLED=0` deliberately, so a binary built today runs on a node whose libc
|
||||
is older than the builder's.
|
||||
|
||||
To build instead of download, you need Go 1.26 or later and nothing else:
|
||||
To build instead of download, you need Go 1.26.6 or later and nothing else (an older Go fetches it itself):
|
||||
|
||||
```
|
||||
git clone https://gitea.mynodes.xyz/ilker/llmgw && cd llmgw
|
||||
@@ -152,6 +152,15 @@ promoted within about thirty seconds of the primary being lost, and the
|
||||
control plane follows it by itself. `llmgw ctl status` says who leads; see
|
||||
[deploy/ha](../deploy/ha/README.md) for what it sets up.
|
||||
|
||||
Everything between the machines is TLS, with certificates from the cluster's
|
||||
own authority: PostgreSQL (every remote rule is `hostssl`, and the control
|
||||
plane verifies the member it reaches), etcd (peers and clients, and a client
|
||||
must present a certificate), and Patroni's API, which also asks for a
|
||||
password before a change. A cluster set up by an earlier llmgw is moved to
|
||||
this by running the same command again, on the control plane that holds the
|
||||
authority: etcd is rebuilt on TLS with Patroni in maintenance mode, so
|
||||
PostgreSQL serves throughout and no failover happens.
|
||||
|
||||
Every step looks before it acts. Running it again finishes an install that
|
||||
stopped halfway and changes nothing on one that did not; it never overwrites
|
||||
`server.yaml`, the authority, or an existing account's password.
|
||||
@@ -184,7 +193,7 @@ forwarded headers right; both are attached to every release.
|
||||
|
||||
```
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health # 200
|
||||
llmgw ctl status
|
||||
sudo llmgw ctl status
|
||||
```
|
||||
|
||||
`ctl status` will report no nodes and no models, which is correct — nothing has
|
||||
@@ -196,9 +205,15 @@ password, change it, and issue yourself a key from the profile page.
|
||||
`/etc/llmgw/server.yaml` is yours from here on. Three things in it are worth
|
||||
knowing before you edit it:
|
||||
|
||||
- `admin_listen` is on loopback on purpose. It serves pprof, which can dump the
|
||||
process heap, and API keys sit in that heap in plaintext. To see the dashboard
|
||||
from elsewhere use `/admin` on the client port — do not move this listener.
|
||||
- `admin_listen` is on loopback on purpose, and the server refuses any other
|
||||
address unless `admin_remote: true`. Its API under `/api` adds models and
|
||||
upgrades nodes, so it also wants a token: the server writes one, 0600, to
|
||||
`/var/lib/llmgw/admin-token` the first time it starts, and `llmgw ctl`
|
||||
reads it from there — which is why `llmgw ctl` runs with sudo.
|
||||
`LLMGW_ADMIN_TOKEN` gives it from elsewhere. pprof, whose heap holds the API
|
||||
keys requests presented, is off unless `debug_pprof: true`, and then behind
|
||||
the token too. To see the dashboard from elsewhere use `/admin` on the
|
||||
client port — do not move this listener.
|
||||
- `default_rpm`, `default_tpm` and `default_price` are what make limits and
|
||||
budgets mean anything. Without a price every request costs zero and a budget
|
||||
never binds; without the defaults, a project created without limits is
|
||||
@@ -263,17 +278,20 @@ it can start downloads and delete models, and only the control plane needs it.
|
||||
The split is what lets you lock the second one down without re-certifying every
|
||||
agent that fetches from the first.
|
||||
|
||||
Give the management port the cluster's own CA and it takes client certificates:
|
||||
Give the management port the cluster's own CA and it takes client certificates —
|
||||
the control plane's, named with `--api-client`, and no other. The same CA signs
|
||||
every node's certificate, so the name is required: without it any node, a
|
||||
removed one's included, could delete models.
|
||||
|
||||
```
|
||||
llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
|
||||
--tls-ca /etc/llmgw/ca.pem \
|
||||
--tls-cert /etc/llmgw/mirror.pem --tls-key /etc/llmgw/mirror-key.pem
|
||||
--tls-cert /etc/llmgw/mirror.pem --tls-key /etc/llmgw/mirror-key.pem \
|
||||
--api-client control-plane
|
||||
```
|
||||
|
||||
Without those three flags the management port is open to anyone who can reach
|
||||
it, and the mirror says so in its own log at every start rather than letting you
|
||||
find out later.
|
||||
Without those flags the management port answers this machine only, and the
|
||||
addresses given with `--api-allow`.
|
||||
|
||||
### The layout
|
||||
|
||||
@@ -995,6 +1013,10 @@ curl $BASE/v1/chat/completions \
|
||||
-d '{"model": "<model-name>", "messages": [{"role":"user","content":"and then?"}]}'
|
||||
```
|
||||
|
||||
A thread belongs to the project, not to the key: **any key of the project that
|
||||
knows the id reads the conversation.** Use ids nobody can guess — a UUID, not
|
||||
`my-conversation` — of at most 128 letters, digits and `- _ . :`.
|
||||
|
||||
llmgw puts the stored history in front of that turn and sends it to whichever
|
||||
model is serving — **including a different one from the one the conversation
|
||||
started on**. That is the point: a thread begun on a large model can continue
|
||||
@@ -1058,6 +1080,39 @@ thing to get right is the health check: point it at `/ready`, not `/health`.
|
||||
so a load balancer using it would send half the traffic to something that
|
||||
refuses every request.
|
||||
|
||||
## The firewall
|
||||
|
||||
`server init`, `node add`, `server add` and `mirror init` each keep that
|
||||
machine's **internal** llmgw ports to the cluster's own networks, with an
|
||||
nftables table of llmgw's own (`inet llmgw`, in `/etc/llmgw/firewall.nft`,
|
||||
loaded at boot by `llmgw-firewall.service`):
|
||||
|
||||
| Machine | Ports, for the cluster only |
|
||||
| --- | --- |
|
||||
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
|
||||
| node | engine ports (`control.port_range`, 20000–20999), 9100 |
|
||||
| store member | 5432, 8008, 2379, 2380, 9100 |
|
||||
| mirror | 8081, 8082, 9100 |
|
||||
|
||||
"The cluster's networks" are, on each machine, the network of the interface
|
||||
it reaches the control planes (or the other store members) on — or just
|
||||
their address, when the route goes through a gateway. Anything else that
|
||||
must reach those ports, a Prometheus on another subnet or nodes behind a
|
||||
router, goes in `server.yaml` before the machines are set up:
|
||||
|
||||
```yaml
|
||||
firewall_networks: [10.0.5.0/24, 192.168.1.20]
|
||||
```
|
||||
|
||||
Only those ports are touched. The table's policy is to accept, so whatever
|
||||
else the machine serves — and whatever firewall it already has — is left as
|
||||
it was; the client API (`:8080`) is not closed, being meant to be reached.
|
||||
Running a command again adds to what is there rather than replacing it, so a
|
||||
machine that is both a store member and a node keeps both sets. `llmgw
|
||||
doctor` on a node says when the table is missing. `--no-firewall` on any of
|
||||
the four commands leaves the firewall alone; `sudo systemctl disable --now
|
||||
llmgw-firewall` removes it.
|
||||
|
||||
## Upgrading
|
||||
|
||||
On the control plane:
|
||||
@@ -1178,6 +1233,20 @@ The control plane publishes when the newest archive was written as
|
||||
`llmgw_backup_last_success_timestamp_seconds`, and **BackupTooOld** fires when
|
||||
it is more than 26 hours old — including when there has never been one.
|
||||
|
||||
**Encrypt them.** Name who may open them in `/etc/llmgw/server.yaml`, and every
|
||||
archive is encrypted to those keys with [age](https://age-encryption.org) and
|
||||
named `.tar.gz.age`:
|
||||
|
||||
```yaml
|
||||
backup_recipients:
|
||||
- age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p # age-keygen's public key
|
||||
- ssh-ed25519 AAAAC3Nza... ops@example # or an SSH public key
|
||||
```
|
||||
|
||||
Keep the matching private keys **off** this machine — an archive whose key
|
||||
sits beside it is not protected by it. Encrypted, a copy is safe wherever it
|
||||
lands.
|
||||
|
||||
**Copy the archives off the machine.** They sit on the disk they back up and do
|
||||
not survive losing it. Anything that copies files will do — rsync from the
|
||||
mirror, a NAS mount, an offsite bucket — as long as it keeps the archives
|
||||
@@ -1197,6 +1266,8 @@ them:
|
||||
```
|
||||
sudo systemctl stop llmgw-server
|
||||
mkdir /tmp/restore && sudo tar -xzf /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz -C /tmp/restore
|
||||
# an encrypted one: open it with the private key, from wherever it is kept
|
||||
age -d -i key.txt llmgw-backup-<host>-<time>.tar.gz.age | sudo tar -xz -C /tmp/restore
|
||||
cat /tmp/restore/MANIFEST # version and time it was taken
|
||||
|
||||
# /etc/llmgw: configuration, secrets, certificates and the CA
|
||||
@@ -1318,11 +1389,9 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
has them. Single-node placement on real GPUs is the well-trodden path; if you
|
||||
are the first to run a TP group across real cards, run `llmgw doctor --json`
|
||||
on each node first and keep the reports.
|
||||
- **Two engine setups are weaker than the rest.** An engine's key is passed
|
||||
in its environment, where only root and its own user can read it, except
|
||||
SGLang's, which takes it only as a flag: other users on the node can see it
|
||||
in the process list. Mount `/proc` with `hidepid=2` on nodes where that
|
||||
matters. vLLM's `launch_mode: ray` starts a Ray head without
|
||||
authentication, so anyone who can reach its port can run code on the
|
||||
node. The dashboard is off, but firewall the port to the other nodes in the
|
||||
replica, or use the default `mp` launch mode, which needs no Ray at all.
|
||||
- **One engine setup is weaker than the rest.** vLLM's `launch_mode: ray`
|
||||
starts a Ray head without authentication, so anyone who can reach its
|
||||
port can run code on the node, and Ray's ports are not among those
|
||||
llmgw's firewall guards. The dashboard is off, but firewall the port to
|
||||
the other nodes in the replica, or use the default `mp` launch mode, which
|
||||
needs no Ray at all.
|
||||
+1
-1
@@ -26,7 +26,7 @@ make test
|
||||
./bin/llmgw --version
|
||||
```
|
||||
|
||||
`make build` writes `./bin/llmgw`. Go 1.26 or later is the only requirement.
|
||||
`make build` writes `./bin/llmgw`. Go 1.26.6 or later is the only requirement (an older Go fetches it itself).
|
||||
|
||||
## 2. Check the path with a fake engine
|
||||
|
||||
|
||||
@@ -152,6 +152,7 @@ rather than repeated.
|
||||
| `inject_usage` | `bool` | [Router](#router) |
|
||||
| `tenant_label` | `string` | [Router](#router) |
|
||||
| `max_body_bytes` | `int64` | [Router](#router) |
|
||||
| `trusted_proxies` | `[]string` | [Router](#router) |
|
||||
| `timeouts` | `Timeouts` | [Router](#router) |
|
||||
| `scoring` | `Scoring` | [Router](#router) |
|
||||
| `admission` | `Admission` | [Router](#router) |
|
||||
@@ -173,12 +174,19 @@ rather than repeated.
|
||||
| `threads` | `Threads` | [Server](#server) |
|
||||
| `landing` | `Landing` | [Server](#server) |
|
||||
| `agent_tls` | `tlsconf.Config` | [Server](#server) |
|
||||
| `max_connections` | `int` | [Server](#server) |
|
||||
| `admin_token_file` | `string` | [Server](#server) |
|
||||
| `admin_remote` | `bool` | [Server](#server) |
|
||||
| `debug_pprof` | `bool` | [Server](#server) |
|
||||
| `agent_insecure` | `bool` | [Server](#server) |
|
||||
| `monitoring` | `Monitoring` | [Server](#server) |
|
||||
| `tracing` | `Tracing` | [Server](#server) |
|
||||
| `mirror` | `Mirror` | [Server](#server) |
|
||||
| `engines` | `map[string]string` | [Server](#server) |
|
||||
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
|
||||
| `backup_dir` | `string` | [Server](#server) |
|
||||
| `firewall_networks` | `[]string` | [Server](#server) |
|
||||
| `backup_recipients` | `[]string` | [Server](#server) |
|
||||
| `releases` | `string` | [Server](#server) |
|
||||
| `router` | `Router` | [Server](#server) |
|
||||
| `control` | `Control` | [Server](#server) |
|
||||
@@ -244,6 +252,7 @@ rather than repeated.
|
||||
| `health_interval` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
|
||||
| `rendezvous_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
|
||||
| `alive_grace` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
|
||||
| `run_as` | `string` | [supervisor.Config](#supervisorconfig) |
|
||||
| `ca` | `string` | [tlsconf.Config](#tlsconfconfig) |
|
||||
| `cert` | `string` | [tlsconf.Config](#tlsconfconfig) |
|
||||
| `key` | `string` | [tlsconf.Config](#tlsconfconfig) |
|
||||
@@ -1220,6 +1229,15 @@ team is what an alert is usually about; "team" is the default.
|
||||
|
||||
MaxBodyBytes bounds the request body the router will read.
|
||||
|
||||
### `trusted_proxies`
|
||||
|
||||
`[]string`
|
||||
|
||||
TrustedProxies are the reverse proxies in front of the client port,
|
||||
as addresses or CIDR prefixes. Their X-Forwarded-For, -Proto and -Host
|
||||
are believed; anyone else's are not, since they decide the address
|
||||
written into the setup scripts and whose attempts are counted where.
|
||||
|
||||
### `timeouts`
|
||||
|
||||
`Timeouts`
|
||||
@@ -1376,6 +1394,49 @@ trusted certificate, while this one is mutual and private.
|
||||
|
||||
Takes a [tlsconf.Config](#tlsconfconfig) block.
|
||||
|
||||
### `max_connections`
|
||||
|
||||
`int`
|
||||
|
||||
MaxConnections bounds the client port's open connections; past it new
|
||||
ones wait in the kernel's backlog. 0 is no bound.
|
||||
|
||||
### `admin_token_file`
|
||||
|
||||
`string`
|
||||
|
||||
AdminTokenFile holds the token the admin API's /api and /debug paths
|
||||
demand. Empty is admin-token in the systemd unit's state directory
|
||||
(/var/lib/llmgw), or secrets/admin-token beside this file for a server
|
||||
run by hand. Made, 0600, when the server starts and it does not exist;
|
||||
`sudo llmgw ctl` and the console read it from there.
|
||||
|
||||
### `admin_remote`
|
||||
|
||||
`bool`
|
||||
|
||||
AdminRemote allows admin_listen on an address other than loopback. The
|
||||
token still guards the API, but it then crosses the network in clear.
|
||||
|
||||
### `debug_pprof`
|
||||
|
||||
`bool`
|
||||
|
||||
DebugPprof serves Go's profiler on the admin listener, behind the token.
|
||||
Off by default: a heap profile holds the API keys requests presented.
|
||||
|
||||
### `agent_insecure`
|
||||
|
||||
`bool`
|
||||
|
||||
AgentInsecure lets a control plane run its agent port without
|
||||
agent_tls: plaintext, and any machine that reaches it may register as
|
||||
any node — be given replicas, and so be sent users' prompts — or, in
|
||||
the other direction, a machine in the path may tell every agent what
|
||||
to run. Refused unless said here, and then the port listens on
|
||||
loopback unless agent_listen names an address. For one machine, a
|
||||
test, or a network nothing untrusted reaches.
|
||||
|
||||
### `monitoring`
|
||||
|
||||
`Monitoring`
|
||||
@@ -1433,6 +1494,26 @@ BackupDir is where `llmgw server backup` writes, and where the newest
|
||||
archive's age is read for llmgw_backup_last_success_timestamp_seconds.
|
||||
Empty means /var/backups/llmgw-server.
|
||||
|
||||
### `firewall_networks`
|
||||
|
||||
`[]string`
|
||||
|
||||
FirewallNetworks are networks, besides those each machine reaches the
|
||||
cluster on, that llmgw's internal ports answer: a monitoring host on
|
||||
another subnet, nodes behind a router. Written into every machine's
|
||||
firewall by `server init`, `node add` and `server add`.
|
||||
|
||||
### `backup_recipients`
|
||||
|
||||
`[]string`
|
||||
|
||||
BackupRecipients encrypt every archive `llmgw server backup` writes, to
|
||||
whoever holds one of their keys: age recipients (age1…) or SSH public
|
||||
keys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds
|
||||
the certificate authority's key and every secret in the store, and is
|
||||
copied off the machine; encrypted, the copy is safe wherever it lands.
|
||||
Empty writes archives unencrypted, readable by root only.
|
||||
|
||||
### `releases`
|
||||
|
||||
`string`
|
||||
@@ -1882,6 +1963,13 @@ AliveGrace is how long a non-serving rank must stay up before it counts
|
||||
as ready. There is no external signal for a worker, so staying alive is
|
||||
the only evidence available.
|
||||
|
||||
### `run_as`
|
||||
|
||||
`string`
|
||||
|
||||
RunAs is the user engines run as when the agent is root; empty is
|
||||
llmgw-engine, "root" keeps them root.
|
||||
|
||||
## tlsconf.Config
|
||||
|
||||
### `ca`
|
||||
|
||||
@@ -1,13 +1,15 @@
|
||||
module gitea.mynodes.xyz/ilker/llmgw
|
||||
|
||||
go 1.26.3
|
||||
go 1.26.6
|
||||
|
||||
require (
|
||||
connectrpc.com/connect v1.21.0
|
||||
filippo.io/age v1.3.2
|
||||
github.com/NVIDIA/go-nvml v0.13.4-0
|
||||
github.com/cespare/xxhash/v2 v2.3.0
|
||||
github.com/charmbracelet/bubbletea v1.3.10
|
||||
github.com/charmbracelet/lipgloss v1.1.0
|
||||
github.com/charmbracelet/x/ansi v0.11.6
|
||||
github.com/jackc/pgx/v5 v5.11.0
|
||||
github.com/knadh/koanf/parsers/yaml v1.1.1
|
||||
github.com/knadh/koanf/providers/env/v2 v2.0.1
|
||||
@@ -32,11 +34,12 @@ require (
|
||||
)
|
||||
|
||||
require (
|
||||
filippo.io/edwards25519 v1.2.0 // indirect
|
||||
filippo.io/hpke v0.4.0 // indirect
|
||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
|
||||
github.com/beorn7/perks v1.0.1 // indirect
|
||||
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
|
||||
github.com/charmbracelet/colorprofile v0.4.1 // indirect
|
||||
github.com/charmbracelet/x/ansi v0.11.6 // indirect
|
||||
github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
|
||||
github.com/charmbracelet/x/term v0.2.2 // indirect
|
||||
github.com/clipperhouse/displaywidth v0.9.0 // indirect
|
||||
|
||||
@@ -1,5 +1,13 @@
|
||||
c2sp.org/CCTV/age v0.0.0-20260829155415-4448f2097b2d h1:Blprhc2SbChNZtWcU+BLTM4YdoqYAS9V7cJgOwJKyAs=
|
||||
c2sp.org/CCTV/age v0.0.0-20260829155415-4448f2097b2d/go.mod h1:SrHC2C7r5GkDk8R+NFVzYy/sdj0Ypg9htaPXQq5Cqeo=
|
||||
connectrpc.com/connect v1.21.0 h1:LhqSJt7jHf5NJBo9Jq/t/9FjcYAideif0mg+qe2jCUs=
|
||||
connectrpc.com/connect v1.21.0/go.mod h1:A2ygJrukXwWy32vkCAAHNVguZrqZ+jeZ9rGRnGR4dN4=
|
||||
filippo.io/age v1.3.2 h1:r6RSZLFSMm6rzKepZ7ZAYkKCu14f3/Me8c7uKYh7C8c=
|
||||
filippo.io/age v1.3.2/go.mod h1:TH/Yr2sSRhCKbaH4XPxpUV0Us8Gv6txYUpiZQWz8Evk=
|
||||
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
|
||||
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
|
||||
filippo.io/hpke v0.4.0 h1:p575VVQ6ted4pL+it6M00V/f2qTZITO0zgmdKCkd5+A=
|
||||
filippo.io/hpke v0.4.0/go.mod h1:EmAN849/P3qdeK+PCMkDpDm83vRHM5cDipBJ8xbQLVY=
|
||||
github.com/NVIDIA/go-nvml v0.13.4-0 h1:o3jp9u2x1R9ShFE3v+Aesp55XOSIQFMJz/VGNUcJaNE=
|
||||
github.com/NVIDIA/go-nvml v0.13.4-0/go.mod h1:id63qwpoDWpFXwnwM6psDCSqW4BmNu6mWpr4YeQtPGo=
|
||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
|
||||
|
||||
@@ -398,3 +398,53 @@ func TestGuessingTheCurrentPasswordIsBounded(t *testing.T) {
|
||||
t.Errorf("guessing was unbounded: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSomebodyGuessingElsewhereCannotLockTheOwnerOut(t *testing.T) {
|
||||
svc, st := newStoreAndUser(t)
|
||||
ctx := context.Background()
|
||||
must(t, st.ApproveUser(ctx, "usr-1", "proj", "ilker", time.Now()))
|
||||
cfg := DefaultConfig()
|
||||
cfg.MaxAttempts = 3
|
||||
svc = New(st, cfg, slog.New(slog.DiscardHandler))
|
||||
|
||||
attacker, owner := WithClient(ctx, "203.0.113.9"), WithClient(ctx, "198.51.100.7")
|
||||
for range 5 {
|
||||
_, _, _ = svc.SignIn(attacker, "ayse@example.com", "nope")
|
||||
}
|
||||
if _, _, err := svc.SignIn(attacker, "ayse@example.com", "nope"); !errors.Is(err, ErrTooManyTries) {
|
||||
t.Errorf("the guessing client was not held back: %v", err)
|
||||
}
|
||||
if _, _, err := svc.SignIn(owner, "ayse@example.com", goodPassword); err != nil {
|
||||
t.Errorf("the owner, elsewhere, was locked out: %v", err)
|
||||
}
|
||||
// Spread over many clients, guessing is still bounded per address.
|
||||
held := false
|
||||
for i := range 40 {
|
||||
c := WithClient(ctx, "192.0.2."+strings.Repeat("1", 1+i%3)+string(rune('0'+i%10)))
|
||||
if _, _, err := svc.SignIn(c, "ayse@example.com", "nope"); errors.Is(err, ErrTooManyTries) {
|
||||
held = true
|
||||
}
|
||||
}
|
||||
if !held {
|
||||
t.Error("guesses from many clients were not bounded")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAHashThatCannotGetATurnSaysTheServerIsBusy(t *testing.T) {
|
||||
was := argonWait
|
||||
argonWait = 10 * time.Millisecond
|
||||
t.Cleanup(func() { argonWait = was })
|
||||
// Every turn taken.
|
||||
n := cap(argonSlots)
|
||||
for range n {
|
||||
argonSlots <- struct{}{}
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
for range n {
|
||||
<-argonSlots
|
||||
}
|
||||
})
|
||||
if _, err := HashPassword(goodPassword); !errors.Is(err, ErrBusy) {
|
||||
t.Errorf("a hash with no turn: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -17,6 +17,7 @@ import (
|
||||
"fmt"
|
||||
"runtime"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"golang.org/x/crypto/argon2"
|
||||
)
|
||||
@@ -57,7 +58,10 @@ func HashPassword(password string) (string, error) {
|
||||
if _, err := rand.Read(salt); err != nil {
|
||||
return "", err
|
||||
}
|
||||
sum := idKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
|
||||
sum, err := idKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
b64 := base64.RawStdEncoding.EncodeToString
|
||||
return fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s",
|
||||
@@ -94,7 +98,10 @@ func VerifyPassword(password, encoded string) (bool, error) {
|
||||
return false, ErrBadHash
|
||||
}
|
||||
|
||||
got := idKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
|
||||
got, err := idKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return subtle.ConstantTimeCompare(got, want) == 1, nil
|
||||
}
|
||||
|
||||
@@ -119,8 +126,19 @@ func init() {
|
||||
// Beyond the bound, attempts wait their turn instead.
|
||||
var argonSlots = make(chan struct{}, max(2, runtime.NumCPU()/2))
|
||||
|
||||
func idKey(password, salt []byte, time, memory uint32, threads uint8, keyLen uint32) []byte {
|
||||
argonSlots <- struct{}{}
|
||||
// ErrBusy is a hash that waited its turn too long: the server is answering
|
||||
// more sign-ins than it can hash, and says so rather than hold the request.
|
||||
var ErrBusy = errors.New("too many sign-ins at once; try again in a moment")
|
||||
|
||||
// argonWait bounds the wait for a turn; a variable so a test can shorten it.
|
||||
var argonWait = 10 * time.Second
|
||||
|
||||
func idKey(password, salt []byte, t, memory uint32, threads uint8, keyLen uint32) ([]byte, error) {
|
||||
select {
|
||||
case argonSlots <- struct{}{}:
|
||||
case <-time.After(argonWait):
|
||||
return nil, ErrBusy
|
||||
}
|
||||
defer func() { <-argonSlots }()
|
||||
return argon2.IDKey(password, salt, time, memory, threads, keyLen)
|
||||
return argon2.IDKey(password, salt, t, memory, threads, keyLen), nil
|
||||
}
|
||||
@@ -138,7 +138,15 @@ func (s *Service) SignUp(ctx context.Context, email, name, password string) erro
|
||||
// the CSRF token that must accompany the forms in it.
|
||||
func (s *Service) SignIn(ctx context.Context, email, password string) (token, csrf string, err error) {
|
||||
email = store.NormalizeEmail(email)
|
||||
if !s.allowAttempt(email) {
|
||||
// Counted per address and the client trying it, so somebody guessing
|
||||
// from elsewhere cannot lock the owner out of their own account; and per
|
||||
// address alone, more loosely, so guesses spread over many clients are
|
||||
// still bounded.
|
||||
byClient := email
|
||||
if c := clientFrom(ctx); c != "" {
|
||||
byClient = email + "|" + c
|
||||
}
|
||||
if !s.allowAttempt(byClient, s.cfg.MaxAttempts) || !s.allowAttempt("all|"+email, 10*s.cfg.MaxAttempts) {
|
||||
return "", "", ErrTooManyTries
|
||||
}
|
||||
|
||||
@@ -154,11 +162,15 @@ func (s *Service) SignIn(ctx context.Context, email, password string) (token, cs
|
||||
}
|
||||
|
||||
ok, err := VerifyPassword(password, u.PasswordHash)
|
||||
if errors.Is(err, ErrBusy) {
|
||||
// Not an answer about the password: said as what it is.
|
||||
return "", "", err
|
||||
}
|
||||
if err != nil || !ok {
|
||||
return "", "", ErrBadCredentials
|
||||
}
|
||||
// The password was right, so this was not an attack.
|
||||
s.clearAttempts(email)
|
||||
s.clearAttempts(byClient)
|
||||
|
||||
// State is checked after the password, on purpose. Checking first would
|
||||
// tell anyone who guesses an address whether it has a pending account.
|
||||
@@ -224,7 +236,7 @@ func (s *Service) CheckCSRF(sess store.Session, submitted string) bool {
|
||||
// Per address rather than per IP: an attacker has many addresses to come from
|
||||
// and only one to guess against, and limiting by IP would lock out everyone
|
||||
// behind one office NAT the moment somebody fat-fingers their password.
|
||||
func (s *Service) allowAttempt(email string) bool {
|
||||
func (s *Service) allowAttempt(email string, limit int) bool {
|
||||
now := s.now()
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
@@ -251,7 +263,20 @@ func (s *Service) allowAttempt(email string) bool {
|
||||
return true
|
||||
}
|
||||
w.count++
|
||||
return w.count <= s.cfg.MaxAttempts
|
||||
return w.count <= limit
|
||||
}
|
||||
|
||||
type clientKey struct{}
|
||||
|
||||
// WithClient says which client a request comes from — its address — for the
|
||||
// sign-in throttle.
|
||||
func WithClient(ctx context.Context, client string) context.Context {
|
||||
return context.WithValue(ctx, clientKey{}, client)
|
||||
}
|
||||
|
||||
func clientFrom(ctx context.Context) string {
|
||||
c, _ := ctx.Value(clientKey{}).(string)
|
||||
return c
|
||||
}
|
||||
|
||||
// maxTrackedAddresses bounds the throttle's memory. Unbounded, it grew by one
|
||||
@@ -307,7 +332,7 @@ type PasswordChanger interface {
|
||||
func (s *Service) ChangePassword(ctx context.Context, pw PasswordChanger,
|
||||
u store.User, current, next string,
|
||||
) error {
|
||||
if !s.allowAttempt("pw:" + u.ID) {
|
||||
if !s.allowAttempt("pw:"+u.ID, s.cfg.MaxAttempts) {
|
||||
return ErrTooManyTries
|
||||
}
|
||||
ok, err := VerifyPassword(current, u.PasswordHash)
|
||||
|
||||
@@ -12,22 +12,22 @@ func TestSprayingAddressesCannotGrowTheThrottleOrEscapeIt(t *testing.T) {
|
||||
|
||||
// The target is being guessed at.
|
||||
for i := 0; i < s.cfg.MaxAttempts; i++ {
|
||||
s.allowAttempt("victim@example.com")
|
||||
s.allowAttempt("victim@example.com", s.cfg.MaxAttempts)
|
||||
}
|
||||
// Then the table is flooded with made-up addresses.
|
||||
for i := 0; i < maxTrackedAddresses+100; i++ {
|
||||
s.allowAttempt(fmt.Sprintf("x%d@example.com", i))
|
||||
s.allowAttempt(fmt.Sprintf("x%d@example.com", i), s.cfg.MaxAttempts)
|
||||
}
|
||||
if n := len(s.attempts); n > maxTrackedAddresses {
|
||||
t.Errorf("the table grew to %d", n)
|
||||
}
|
||||
if s.allowAttempt("victim@example.com") {
|
||||
if s.allowAttempt("victim@example.com", s.cfg.MaxAttempts) {
|
||||
t.Error("flooding the table reset the target's throttle")
|
||||
}
|
||||
|
||||
// Once the window passes, everything expires and sign-ins work again.
|
||||
now = now.Add(s.cfg.AttemptWindow + time.Second)
|
||||
if !s.allowAttempt("new@example.com") {
|
||||
if !s.allowAttempt("new@example.com", s.cfg.MaxAttempts) {
|
||||
t.Error("a new address was refused after the flood expired")
|
||||
}
|
||||
}
|
||||
@@ -306,7 +306,13 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
|
||||
a.tokens.Charge(c, actual-res.EstimatedTokens)
|
||||
}
|
||||
|
||||
u.CostMicros = res.Price.Cost(u.PromptTokens, u.CompletionTokens)
|
||||
// Priced by the model that answered: a fallback served by another model
|
||||
// costs what that one costs, not what the one asked for would have.
|
||||
price := res.Price
|
||||
if u.ServedModel != "" && u.ServedModel != u.Model && a.pricing != nil {
|
||||
price = a.pricing.For(u.ServedModel)
|
||||
}
|
||||
u.CostMicros = price.Cost(u.PromptTokens, u.CompletionTokens)
|
||||
for _, b := range res.Budgets {
|
||||
a.budgets.Add(b.Counter, b.Period, u.CostMicros)
|
||||
}
|
||||
|
||||
@@ -658,3 +658,22 @@ func TestHeadroomIsTheTightestLimitAfterThisRequest(t *testing.T) {
|
||||
t.Errorf("a key with no limits advertised some: %+v", none)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAFallbackIsChargedWhatTheModelThatAnsweredCosts(t *testing.T) {
|
||||
a := New(pricing(t), nil, slog.New(slog.DiscardHandler))
|
||||
p := inTree("fb-k", teamBudget("fb", 1_000_000))
|
||||
res, err := a.Check(p, "paid", 400)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Asked for "paid", answered by "free": nothing is owed.
|
||||
a.Record(res, store.Usage{KeyID: "fb-k", Model: "paid", ServedModel: "free", PromptTokens: 200, CompletionTokens: 200})
|
||||
if spent, _, _ := a.Status(p); spent != 0 {
|
||||
t.Errorf("charged %d for a request the free model answered", spent)
|
||||
}
|
||||
res, _ = a.Check(p, "paid", 400)
|
||||
a.Record(res, store.Usage{KeyID: "fb-k", Model: "paid", ServedModel: "paid", PromptTokens: 200, CompletionTokens: 200})
|
||||
if spent, _, _ := a.Status(p); spent == 0 {
|
||||
t.Error("the paid model answering cost nothing")
|
||||
}
|
||||
}
|
||||
+87
-23
@@ -14,10 +14,12 @@ import (
|
||||
"math/rand/v2"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"slices"
|
||||
"strconv"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@@ -310,7 +312,7 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
|
||||
// The same two paths a freshly started shard takes: a rank that serves HTTP
|
||||
// is health-checked, a worker is judged by staying alive.
|
||||
if eng.ServesHTTP(o.Spec) {
|
||||
base := fmt.Sprintf("http://127.0.0.1:%d", o.Spec.HTTPPort)
|
||||
base := a.engineBase(o.Spec)
|
||||
err = a.sup.WaitReady(ctx, sh, base, o.Spec.Profile.ServedModelName, a.cfg.AdoptReadyTimeout)
|
||||
} else {
|
||||
err = a.sup.WaitAlive(ctx, sh)
|
||||
@@ -516,6 +518,7 @@ func (a *Agent) session(ctx context.Context, server string) error {
|
||||
a.sendLoop(ctx, stream, out)
|
||||
}()
|
||||
defer wg.Wait()
|
||||
go a.watchExits(ctx, out)
|
||||
|
||||
for {
|
||||
msg, err := stream.Receive()
|
||||
@@ -885,6 +888,42 @@ func (a *Agent) watchFetch(source string, fn func(weights.Progress)) (stop func(
|
||||
}
|
||||
}
|
||||
|
||||
// bindAddr is where engines listen: the management address the control
|
||||
// plane reaches them on, not every address the node has — a public one
|
||||
// included. Empty, on every address, when that is a name rather than an
|
||||
// address this machine holds.
|
||||
func (a *Agent) bindAddr() string {
|
||||
ip, err := netip.ParseAddr(a.cfg.MgmtAddr)
|
||||
if err != nil || ip.IsUnspecified() || !localAddr(ip) {
|
||||
return ""
|
||||
}
|
||||
return ip.String()
|
||||
}
|
||||
|
||||
// engineBase is where the agent asks an engine whether it is ready: where it
|
||||
// listens, loopback when that is every address.
|
||||
func (a *Agent) engineBase(s engines.ShardSpec) string {
|
||||
host := s.BindAddr
|
||||
if host == "" {
|
||||
host = "127.0.0.1"
|
||||
}
|
||||
return "http://" + net.JoinHostPort(host, strconv.Itoa(s.HTTPPort))
|
||||
}
|
||||
|
||||
// localAddr reports whether an address is one of this machine's.
|
||||
func localAddr(ip netip.Addr) bool {
|
||||
addrs, err := net.InterfaceAddrs()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
for _, a := range addrs {
|
||||
if p, err := netip.ParsePrefix(a.String()); err == nil && p.Addr().Unmap() == ip.Unmap() {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<- *llmgwv1.AgentMessage) {
|
||||
shardID, replicaID := m.GetShardId(), m.GetReplicaId()
|
||||
|
||||
@@ -907,6 +946,7 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
|
||||
|
||||
spec, timeout := agentsvc.ShardSpecFromStart(m)
|
||||
a.localizeSpec(&spec)
|
||||
spec.BindAddr = a.bindAddr()
|
||||
|
||||
eng, err := a.reg.Get(spec.Profile.Engine)
|
||||
if err != nil {
|
||||
@@ -920,6 +960,12 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
|
||||
|
||||
// Validate before launching. An engine that refuses a profile does so after
|
||||
// minutes of loading; catching it here costs nothing.
|
||||
// Checked again here, whatever sent the profile: this is the process
|
||||
// about to run as root.
|
||||
if err := engines.UnsafeArgs(spec.Profile.Engine, spec.Profile.Args); err != nil {
|
||||
report(supervisor.Failed, err.Error(), nil)
|
||||
return
|
||||
}
|
||||
if err := eng.Validate(spec.Profile, gpuInfo(inv, spec.GPUIndices)); err != nil {
|
||||
report(supervisor.Failed, err.Error(), nil)
|
||||
return
|
||||
@@ -987,7 +1033,7 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
|
||||
// staying alive, and the replica's real readiness is the leader answering,
|
||||
// which only happens once every worker has joined (plan §11, phase 4).
|
||||
if eng.ServesHTTP(spec) {
|
||||
base := fmt.Sprintf("http://127.0.0.1:%d", spec.HTTPPort)
|
||||
base := a.engineBase(spec)
|
||||
err = a.sup.WaitReady(ctx, sh, base, spec.Profile.ServedModelName, timeout)
|
||||
} else {
|
||||
err = a.sup.WaitAlive(ctx, sh)
|
||||
@@ -1005,36 +1051,54 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
|
||||
return
|
||||
}
|
||||
report(supervisor.Ready, "", nil)
|
||||
|
||||
// Watch for the engine dying later. Nothing else would notice: the router
|
||||
// would simply see failing requests.
|
||||
go a.watch(ctx, sh, replicaID, shardID, out)
|
||||
// Its dying later is reported by the session's watchExits.
|
||||
}
|
||||
|
||||
// watch reports a shard that exits after it was ready.
|
||||
func (a *Agent) watch(ctx context.Context, sh *supervisor.Shard, replicaID, shardID string, out chan<- *llmgwv1.AgentMessage) {
|
||||
// watchExits reports, for as long as this session lasts, every shard that
|
||||
// was ready and has failed: the ones this agent started, and the ones it
|
||||
// adopted from the agent before it.
|
||||
//
|
||||
// Per session rather than per shard. A watcher tied to the stream a shard
|
||||
// was started on stopped with that stream, so after a reconnect nothing
|
||||
// reported the engine dying; and an adopted engine had no watcher at all —
|
||||
// on node06 one was killed and its replica stayed "ready" for the control
|
||||
// plane, routed to and failing, indefinitely.
|
||||
func (a *Agent) watchExits(ctx context.Context, out chan<- *llmgwv1.AgentMessage) {
|
||||
t := time.NewTicker(time.Second)
|
||||
defer t.Stop()
|
||||
|
||||
ready := map[string]bool{}
|
||||
for {
|
||||
for _, st := range a.sup.Statuses() {
|
||||
switch st.State {
|
||||
case supervisor.Ready:
|
||||
ready[st.ID] = true
|
||||
case supervisor.Failed:
|
||||
if !ready[st.ID] {
|
||||
continue // never ready here: its start reported it
|
||||
}
|
||||
delete(ready, st.ID)
|
||||
sh, err := a.sup.Get(st.ID)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
msg := &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Shard{Shard: &llmgwv1.ShardStatus{
|
||||
NodeId: a.cfg.NodeID, ReplicaId: sh.Spec.ReplicaID, ShardId: st.ID,
|
||||
State: agentsvc.StateToWire(st.State), Error: st.Err,
|
||||
ExitCode: int32(st.ExitCode), LastOutput: sh.Logs(20),
|
||||
}}}
|
||||
select {
|
||||
case out <- msg:
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
default:
|
||||
delete(ready, st.ID)
|
||||
}
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
st := sh.Status()
|
||||
if !st.State.Terminal() {
|
||||
continue
|
||||
}
|
||||
msg := &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Shard{Shard: &llmgwv1.ShardStatus{
|
||||
NodeId: a.cfg.NodeID, ReplicaId: replicaID, ShardId: shardID,
|
||||
State: agentsvc.StateToWire(st.State), Error: st.Err,
|
||||
ExitCode: int32(st.ExitCode), LastOutput: sh.Logs(20),
|
||||
}}}
|
||||
select {
|
||||
case out <- msg:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
)
|
||||
|
||||
func TestEnginesListenOnTheManagementAddressWhenItIsThisMachines(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
"127.0.0.1": "127.0.0.1", // one of this machine's
|
||||
"203.0.113.9": "", // not: every address rather than none
|
||||
"gpu07.lan": "", // a name
|
||||
"0.0.0.0": "",
|
||||
"": "",
|
||||
}
|
||||
for mgmt, want := range cases {
|
||||
a := &Agent{cfg: Config{MgmtAddr: mgmt}}
|
||||
if got := a.bindAddr(); got != want {
|
||||
t.Errorf("management address %q: listens on %q, expected %q", mgmt, got, want)
|
||||
}
|
||||
}
|
||||
a := &Agent{}
|
||||
if got := a.engineBase(engines.ShardSpec{HTTPPort: 8000}); got != "http://127.0.0.1:8000" {
|
||||
t.Errorf("on every address, asked at %s", got)
|
||||
}
|
||||
if got := a.engineBase(engines.ShardSpec{HTTPPort: 8000, BindAddr: "10.0.0.147"}); got != "http://10.0.0.147:8000" {
|
||||
t.Errorf("on one address, asked at %s", got)
|
||||
}
|
||||
}
|
||||
@@ -73,9 +73,13 @@ type ShardSpec struct {
|
||||
|
||||
HTTPPort int
|
||||
Profile Profile
|
||||
// BindAddr is the address the engine listens on: the node's management
|
||||
// address, which is what the router reaches it on, set by the agent.
|
||||
// Empty listens on every address.
|
||||
BindAddr string `json:"bind_addr,omitempty"`
|
||||
|
||||
// APIKey is what the engine demands of every request. Engines listen on
|
||||
// all of a node's addresses; without a key anybody who can reach the port
|
||||
// the node's management address; without a key anybody who can reach it
|
||||
// talks to the model directly, past the gateway's authentication, quotas
|
||||
// and accounting. Given to the engine in its environment rather than on
|
||||
// its command line, which every user on the node can read.
|
||||
@@ -98,6 +102,14 @@ type ShardSpec struct {
|
||||
ProfileFP string
|
||||
}
|
||||
|
||||
// Host is where the engine listens.
|
||||
func (s ShardSpec) Host() string {
|
||||
if s.BindAddr == "" {
|
||||
return "0.0.0.0"
|
||||
}
|
||||
return s.BindAddr
|
||||
}
|
||||
|
||||
// Leader reports whether this shard serves HTTP for the replica.
|
||||
func (s ShardSpec) Leader() bool { return s.Rank == 0 }
|
||||
|
||||
|
||||
@@ -323,8 +323,8 @@ func TestSGLangMultiNodeArgs(t *testing.T) {
|
||||
if cmd.Path != "/opt/sglang/bin/python" {
|
||||
t.Errorf("Path=%q", cmd.Path)
|
||||
}
|
||||
if cmd.Args[0] != "-m" || cmd.Args[1] != "sglang.launch_server" {
|
||||
t.Errorf("not launched as a module: %v", cmd.Args[:2])
|
||||
if cmd.Args[0] != "-c" || !strings.Contains(cmd.Args[1], `run_module("sglang.launch_server"`) {
|
||||
t.Errorf("not launched as the module: %v", cmd.Args[:2])
|
||||
}
|
||||
if argValue(cmd.Args, "--nnodes") != "2" || argValue(cmd.Args, "--node-rank") != "1" {
|
||||
t.Errorf("multi-node flags wrong: %v", cmd.Args)
|
||||
@@ -334,6 +334,38 @@ func TestSGLangMultiNodeArgs(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestSGLangsKeyIsNotOnItsCommandLine(t *testing.T) {
|
||||
s := singleNodeSpec()
|
||||
s.APIKey = "k1"
|
||||
cmd := buildOne(t, &SGLang{}, s)
|
||||
if slices.Contains(cmd.Args, "k1") || slices.Contains(cmd.Args, "--api-key") {
|
||||
t.Errorf("the key is in the arguments: %v", cmd.Args)
|
||||
}
|
||||
if !slices.Contains(cmd.Env, sglangKeyEnv+"=k1") {
|
||||
t.Errorf("the key is not in the environment")
|
||||
}
|
||||
// A worker serves no HTTP, and is not given it.
|
||||
w := multiNodeSpec(1)
|
||||
w.APIKey = "k1"
|
||||
if cmd := buildOne(t, &SGLang{}, w); slices.Contains(cmd.Env, sglangKeyEnv+"=k1") {
|
||||
t.Error("a worker was given the key")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnginesListenOnTheAddressTheyAreGiven(t *testing.T) {
|
||||
s := singleNodeSpec()
|
||||
for _, e := range []Engine{&VLLM{}, &SGLang{}} {
|
||||
s.BindAddr = ""
|
||||
if got := argValue(buildOne(t, e, s).Args, "--host"); got != "0.0.0.0" {
|
||||
t.Errorf("%s: without an address listens on %q", e.Name(), got)
|
||||
}
|
||||
s.BindAddr = "10.0.0.147"
|
||||
if got := argValue(buildOne(t, e, s).Args, "--host"); got != "10.0.0.147" {
|
||||
t.Errorf("%s: listens on %q, expected the management address", e.Name(), got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSGLangSingleNodeOmitsMultiNodeFlags(t *testing.T) {
|
||||
cmd := buildOne(t, &SGLang{}, singleNodeSpec())
|
||||
for _, flag := range []string{"--nnodes", "--node-rank", "--dist-init-addr"} {
|
||||
|
||||
@@ -99,7 +99,7 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
|
||||
}
|
||||
args := []string{
|
||||
"--model", model,
|
||||
"--host", "0.0.0.0",
|
||||
"--host", s.Host(),
|
||||
"--port", strconv.Itoa(s.HTTPPort),
|
||||
// Without this there is no /metrics, and an endpoint the router cannot
|
||||
// measure is one it has to route to blind.
|
||||
|
||||
@@ -79,9 +79,9 @@ func (g *SGLang) serverCommand(s ShardSpec) (Command, error) {
|
||||
python := g.python()
|
||||
p := s.Profile
|
||||
|
||||
args := []string{"-m", "sglang.launch_server",
|
||||
args := []string{"-c", sglangLauncher,
|
||||
"--model-path", p.ModelPath,
|
||||
"--host", "0.0.0.0",
|
||||
"--host", s.Host(),
|
||||
"--port", strconv.Itoa(s.HTTPPort),
|
||||
"--tp", strconv.Itoa(p.TP),
|
||||
}
|
||||
@@ -96,11 +96,23 @@ func (g *SGLang) serverCommand(s ShardSpec) (Command, error) {
|
||||
)
|
||||
}
|
||||
args = append(args, p.Args...)
|
||||
// SGLang reads its key only from the command line, so on a node with
|
||||
// other users it is visible to them in the process list.
|
||||
env := baseEnv(s)
|
||||
if s.APIKey != "" && s.Leader() {
|
||||
args = append(args, "--api-key", s.APIKey)
|
||||
env[sglangKeyEnv] = s.APIKey
|
||||
}
|
||||
|
||||
return Command{Name: "sglang", Path: python, Args: args, Env: materializeEnv(baseEnv(s))}, nil
|
||||
return Command{Name: "sglang", Path: python, Args: args, Env: materializeEnv(env)}, nil
|
||||
}
|
||||
|
||||
// SGLang reads its key only from the command line, where every user on the
|
||||
// node can read it in the process list. So the key goes in the environment,
|
||||
// which only the engine's own user and root can read, and this launcher —
|
||||
// sglang.launch_server under another name — adds it to the arguments inside
|
||||
// the process: sys.argv is Python's copy, not what /proc shows.
|
||||
const (
|
||||
sglangKeyEnv = "LLMGW_ENGINE_KEY"
|
||||
sglangLauncher = `import os, runpy, sys
|
||||
key = os.environ.pop("` + sglangKeyEnv + `", "")
|
||||
sys.argv = ["sglang.launch_server"] + sys.argv[1:] + (["--api-key", key] if key else [])
|
||||
runpy.run_module("sglang.launch_server", run_name="__main__", alter_sys=True)
|
||||
`
|
||||
)
|
||||
@@ -2,6 +2,7 @@ package engines
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"strconv"
|
||||
)
|
||||
@@ -63,7 +64,7 @@ func (s *Sim) Commands(spec ShardSpec) ([]Command, error) {
|
||||
p := spec.Profile
|
||||
|
||||
args := []string{"sim-engine",
|
||||
"--listen", "0.0.0.0:" + strconv.Itoa(spec.HTTPPort),
|
||||
"--listen", net.JoinHostPort(spec.Host(), strconv.Itoa(spec.HTTPPort)),
|
||||
"--model", p.ServedModelName,
|
||||
}
|
||||
if spec.NNodes > 1 {
|
||||
|
||||
@@ -1,30 +1,147 @@
|
||||
package engines
|
||||
|
||||
import "strings"
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// A model profile's env and args are the operator's escape hatch for tuning
|
||||
// an engine, and they reach a process the agent starts as root on every node
|
||||
// the model is placed on. Some of what an engine accepts is not tuning but a
|
||||
// way to run other code, or to read or write files, in that process: a
|
||||
// loader preload, an NCCL network plugin, a CUDA injection library, vLLM's
|
||||
// --trust-remote-code, llama.cpp's --log-file pointed at /etc/cron.d. Taken
|
||||
// from a model definition, which the admin API or a config file supplies,
|
||||
// they would turn "add a model" into "run this as root on every node".
|
||||
//
|
||||
// So env is an allowlist — the families of variables engines are tuned with
|
||||
// — less any whose name says it names a path, a plugin or a library, and args
|
||||
// are checked against what each engine takes that reads, writes or loads.
|
||||
// Both are refused where a model is defined and again where an engine starts.
|
||||
|
||||
// envFamilies are the prefixes of variables engines are tuned with.
|
||||
var envFamilies = []string{
|
||||
"CUDA_", "NCCL_", "NVIDIA_", "NVTE_", "GLOO_", "UCX_", "TORCH_", "PYTORCH_",
|
||||
"VLLM_", "SGLANG_", "GGML_", "LLAMA_", "OMP_", "MKL_", "OPENBLAS_", "KMP_",
|
||||
"HF_", "TRANSFORMERS_", "TOKENIZERS_", "TRITON_", "FLASHINFER_",
|
||||
}
|
||||
|
||||
// envExact are single variables that are tuning too.
|
||||
var envExact = map[string]bool{
|
||||
"PYTHONUNBUFFERED": true, "PYTHONHASHSEED": true, "TZ": true,
|
||||
}
|
||||
|
||||
// envNever are names in an allowed family that load code or move where the
|
||||
// engine reads and writes.
|
||||
var envNever = []string{
|
||||
"PLUGIN", "INJECTION", "PRELOAD", "LIBRARY", "_LIB",
|
||||
"_PATH", "_DIR", "_HOME", "_FILE", "_ROOT", "_CACHE", "_CONFIG",
|
||||
"_ADDRESS", "_ENDPOINT", "_TOKEN", "_URL",
|
||||
}
|
||||
|
||||
// UnsafeEnv reports whether a profile may not set an environment variable,
|
||||
// and why.
|
||||
//
|
||||
// A profile's env is the escape hatch for tuning no adapter can guess —
|
||||
// CUDA_*, NCCL_*, VLLM_*, HF_* — and it reaches an engine the agent starts as
|
||||
// root. Some variables are not tuning but a way to run other code in that
|
||||
// process: the dynamic loader's LD_PRELOAD, Python's PYTHONSTARTUP, a shell's
|
||||
// BASH_ENV. Set from a model definition, which an admin API or a config file
|
||||
// supplies, they would turn "add a model" into "run this as root on every
|
||||
// node". They are refused where a model is defined and again where an engine
|
||||
// starts.
|
||||
func UnsafeEnv(key string) (bool, string) {
|
||||
k := strings.ToUpper(key)
|
||||
switch {
|
||||
case strings.HasPrefix(k, "LD_"), strings.HasPrefix(k, "DYLD_"):
|
||||
return true, "it changes what the dynamic loader loads"
|
||||
case k == "PATH":
|
||||
return true, "it changes which binaries run"
|
||||
case strings.HasPrefix(k, "PYTHON") && k != "PYTHONUNBUFFERED" && k != "PYTHONHASHSEED":
|
||||
return true, "it changes what Python imports or runs at start"
|
||||
case k == "BASH_ENV" || k == "ENV" || k == "PERL5OPT" || k == "PERL5LIB" ||
|
||||
k == "NODE_OPTIONS" || k == "RUBYOPT" || k == "GCONV_PATH" || k == "OPENSSL_CONF":
|
||||
return true, "it makes an interpreter or library load code"
|
||||
k := strings.ToUpper(strings.TrimSpace(key))
|
||||
if k == "" {
|
||||
return true, "it has no name"
|
||||
}
|
||||
allowed := envExact[k]
|
||||
for _, f := range envFamilies {
|
||||
if strings.HasPrefix(k, f) {
|
||||
allowed = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !allowed {
|
||||
return true, "only the variables engines are tuned with are allowed (" +
|
||||
strings.Join(envFamilies, "*, ") + "*)"
|
||||
}
|
||||
for _, n := range envNever {
|
||||
if strings.Contains(k, n) && !envExact[k] {
|
||||
return true, "it names a path, a plugin, a library or an address, which is a way to make the engine " +
|
||||
"read, write or load something other than its model"
|
||||
}
|
||||
}
|
||||
// llama.cpp reads every flag from LLAMA_ARG_<FLAG> too.
|
||||
if rest, ok := strings.CutPrefix(k, "LLAMA_ARG_"); ok {
|
||||
flag := "--" + strings.ToLower(strings.ReplaceAll(rest, "_", "-"))
|
||||
if why := unsafeFlag("llamacpp", flag, ""); why != "" {
|
||||
return true, "it is " + flag + ", which " + why
|
||||
}
|
||||
}
|
||||
return false, ""
|
||||
}
|
||||
|
||||
// flagsNever are flags no profile may give, by engine; "" is every engine.
|
||||
// Each reads or writes files the agent did not choose, loads code, or moves
|
||||
// what the agent itself sets: where the engine listens and the key it
|
||||
// demands.
|
||||
var flagsNever = map[string]map[string]string{
|
||||
"": {
|
||||
"--host": "is set by the agent", "--port": "is set by the agent",
|
||||
"--api-key": "is set by the agent", "--api-key-file": "is set by the agent",
|
||||
"--model": "is the model's source, set by the agent", "-m": "is the model's source, set by the agent",
|
||||
"--ssl-keyfile": "reads a file", "--ssl-certfile": "reads a file",
|
||||
"--ssl-key-file": "reads a file", "--ssl-cert-file": "reads a file",
|
||||
"--trust-remote-code": "runs Python the model repository ships",
|
||||
},
|
||||
"llamacpp": {
|
||||
"--log-file": "writes a file", "--slot-save-path": "writes files",
|
||||
"--path": "serves a directory's files over HTTP", "--rpc": "sends the model's work to other machines",
|
||||
"--chat-template-file": "reads a file", "--system-prompt-file": "reads a file",
|
||||
"--grammar-file": "reads a file", "--json-schema-file": "reads a file",
|
||||
"--lora": "loads a file; adapters are set in the model's adapters",
|
||||
"--lora-scaled": "loads a file; adapters are set in the model's adapters",
|
||||
"--control-vector": "loads a file", "--control-vector-scaled": "loads a file",
|
||||
"--mmproj": "loads a file; the projector is fetched with the model",
|
||||
"--model-url": "downloads a model", "-mu": "downloads a model",
|
||||
"--hf-repo": "downloads a model", "-hf": "downloads a model", "--hf-file": "downloads a model",
|
||||
"--hf-token": "is a credential", "--alias": "is set by the agent",
|
||||
"--model-draft": "loads a file", "-md": "loads a file",
|
||||
"--prompt-cache": "writes a file", "--logdir": "writes files",
|
||||
},
|
||||
"vllm": {
|
||||
"--download-dir": "writes files", "--tokenizer": "reads another repository or path",
|
||||
"--chat-template": "reads a file", "--allowed-local-media-path": "serves local files to requests",
|
||||
"--lora-modules": "loads files; adapters are set in the model's adapters",
|
||||
"--tool-parser-plugin": "runs a Python file", "--middleware": "imports Python",
|
||||
"--worker-extension-cls": "imports Python", "--worker-cls": "imports Python",
|
||||
"--load-format": "changes how weights are loaded, some ways running code",
|
||||
"--otlp-traces-endpoint": "sends request traces elsewhere",
|
||||
"--served-model-name": "is set by the agent",
|
||||
"--enable-log-requests": "logs prompts",
|
||||
},
|
||||
"sglang": {
|
||||
"--tokenizer-path": "reads another repository or path", "--chat-template": "reads a file",
|
||||
"--file-storage-path": "writes files", "--log-requests": "logs prompts",
|
||||
"--model-path": "is the model's source, set by the agent", "--served-model-name": "is set by the agent",
|
||||
"--lora-paths": "loads files", "--download-dir": "writes files",
|
||||
},
|
||||
}
|
||||
|
||||
// unsafeFlag is why an engine may not be given a flag, or "".
|
||||
func unsafeFlag(engine, flag, _ string) string {
|
||||
name, _, _ := strings.Cut(flag, "=")
|
||||
for _, set := range []string{"", engine} {
|
||||
if why, ok := flagsNever[set][name]; ok && why != "" {
|
||||
return why
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// UnsafeArgs refuses the first of a profile's arguments an engine may not be
|
||||
// given, saying why.
|
||||
func UnsafeArgs(engine string, args []string) error {
|
||||
for _, a := range args {
|
||||
if !strings.HasPrefix(a, "-") {
|
||||
continue // a value
|
||||
}
|
||||
if why := unsafeFlag(engine, a, ""); why != "" {
|
||||
name, _, _ := strings.Cut(a, "=")
|
||||
return fmt.Errorf("argument %s is not allowed: it %s", name, why)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
package engines
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestTuningVariablesAreAllowedAndWaysToRunCodeAreNot(t *testing.T) {
|
||||
for _, k := range []string{
|
||||
"CUDA_VISIBLE_DEVICES", "CUDA_MODULE_LOADING", "NCCL_IB_HCA", "NCCL_DEBUG", "NCCL_NET",
|
||||
"VLLM_ATTENTION_BACKEND", "PYTORCH_CUDA_ALLOC_CONF", "OMP_NUM_THREADS", "HF_HUB_OFFLINE",
|
||||
"GGML_VK_VISIBLE_DEVICES", "TOKENIZERS_PARALLELISM", "PYTHONUNBUFFERED", "LLAMA_ARG_CTX_SIZE",
|
||||
} {
|
||||
if bad, why := UnsafeEnv(k); bad {
|
||||
t.Errorf("%s refused: %s", k, why)
|
||||
}
|
||||
}
|
||||
for _, k := range []string{
|
||||
"LD_PRELOAD", "PATH", "PYTHONPATH", "PYTHONSTARTUP", "BASH_ENV",
|
||||
// What the denylist missed: each loads a library or moves where the
|
||||
// engine reads and writes.
|
||||
"NCCL_NET_PLUGIN", "NCCL_TUNER_PLUGIN", "NCCL_PROFILER_PLUGIN", "CUDA_INJECTION64_PATH",
|
||||
"NVTX_INJECTION64_PATH", "TRITON_CACHE_DIR", "HF_HOME", "HF_TOKEN", "NCCL_DEBUG_FILE",
|
||||
"VLLM_PLUGINS", "UCX_MODULE_DIR", "TORCH_EXTENSIONS_DIR",
|
||||
// llama.cpp's flags as variables.
|
||||
"LLAMA_ARG_LOG_FILE", "LLAMA_ARG_HOST", "LLAMA_ARG_SLOT_SAVE_PATH",
|
||||
// Anything outside the families.
|
||||
"SOMETHING_ELSE", "",
|
||||
} {
|
||||
if bad, _ := UnsafeEnv(k); !bad {
|
||||
t.Errorf("%s allowed", k)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestArgumentsThatReadWriteOrLoadAreRefused(t *testing.T) {
|
||||
ok := map[string][]string{
|
||||
"llamacpp": {"--ctx-size", "16384", "--embedding", "--parallel", "4", "--jinja", "--n-gpu-layers=999"},
|
||||
"vllm": {"--max-model-len", "8192", "--gpu-memory-utilization", "0.9", "--enable-auto-tool-choice"},
|
||||
"sglang": {"--mem-fraction-static", "0.8", "--tool-call-parser", "qwen25"},
|
||||
}
|
||||
for eng, args := range ok {
|
||||
if err := UnsafeArgs(eng, args); err != nil {
|
||||
t.Errorf("%s %v: %v", eng, args, err)
|
||||
}
|
||||
}
|
||||
for _, c := range []struct {
|
||||
eng string
|
||||
args []string
|
||||
}{
|
||||
{"vllm", []string{"--trust-remote-code"}},
|
||||
{"sglang", []string{"--trust-remote-code"}},
|
||||
{"vllm", []string{"--tool-parser-plugin", "/tmp/x.py"}},
|
||||
{"vllm", []string{"--middleware=evil.mod"}},
|
||||
{"vllm", []string{"--allowed-local-media-path", "/"}},
|
||||
{"llamacpp", []string{"--log-file", "/etc/cron.d/x"}},
|
||||
{"llamacpp", []string{"--slot-save-path=/root"}},
|
||||
{"llamacpp", []string{"--path", "/"}},
|
||||
{"llamacpp", []string{"--system-prompt-file", "/etc/shadow"}},
|
||||
{"llamacpp", []string{"--host", "0.0.0.0"}},
|
||||
{"vllm", []string{"--api-key", "x"}},
|
||||
} {
|
||||
if err := UnsafeArgs(c.eng, c.args); err == nil {
|
||||
t.Errorf("%s %v allowed", c.eng, c.args)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -106,7 +106,7 @@ func (v *VLLM) serverCommand(s ShardSpec) (Command, error) {
|
||||
p := s.Profile
|
||||
|
||||
args := []string{"serve", p.ModelPath,
|
||||
"--host", "0.0.0.0",
|
||||
"--host", s.Host(),
|
||||
"--port", strconv.Itoa(s.HTTPPort),
|
||||
"--tensor-parallel-size", strconv.Itoa(p.TP),
|
||||
}
|
||||
|
||||
@@ -8,6 +8,8 @@ import (
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
@@ -33,6 +35,10 @@ const pythonInstallTimeout = time.Hour
|
||||
// lockName is the resolved requirements, kept in the environment.
|
||||
const lockName = "llmgw-requirements.txt"
|
||||
|
||||
// pylockName is the lock of a build with a second index: PEP 751's, with
|
||||
// every file's address.
|
||||
const pylockName = "pylock.llmgw.toml"
|
||||
|
||||
func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, pythonInstallTimeout)
|
||||
defer cancel()
|
||||
@@ -113,28 +119,16 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
b.Wheel.Tag + "/" + asset
|
||||
reqs = append([]string{b.Wheel.Project + " @ " + url + "#sha256=" + sum}, reqs...)
|
||||
}
|
||||
// A second index, for the torch the engine was built on. Every package
|
||||
// is taken from whichever of the two has the version resolved — both are
|
||||
// the publishers' own — and each file is still pinned by its hash.
|
||||
var index []string
|
||||
if b.ExtraIndex != "" {
|
||||
index = []string{"--extra-index-url", b.ExtraIndex, "--index-strategy", "unsafe-best-match"}
|
||||
}
|
||||
|
||||
in.progress(st, "resolving %s against PyPI as of %s", strings.Join(reqs, " "), asOf[:10])
|
||||
lock := filepath.Join(tmp, lockName)
|
||||
args := append([]string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
|
||||
"--generate-hashes", "--quiet", "--output-file", lock}, index...)
|
||||
args = append(args, "-")
|
||||
if out, err := in.runStdin(ctx, env, strings.Join(reqs, "\n")+"\n", uv.Binary, args...); err != nil {
|
||||
return fmt.Errorf("resolving %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
|
||||
lock, err := in.resolve(ctx, env, uv.Binary, python, tmp, asOf, reqs, b)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
in.progress(st, "downloading and installing %s and its dependencies: several gigabytes, which takes minutes",
|
||||
strings.Join(reqs, " "))
|
||||
install := append([]string{"pip", "install", "--python", python,
|
||||
"--require-hashes", "--no-deps", "-r", lock}, index...)
|
||||
if out, err := in.run(ctx, env, uv.Binary, install...); err != nil {
|
||||
if out, err := in.run(ctx, env, uv.Binary, "pip", "install", "--python", python,
|
||||
"--require-hashes", "--no-deps", "-r", lock); err != nil {
|
||||
return fmt.Errorf("installing %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
|
||||
}
|
||||
|
||||
@@ -168,6 +162,63 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
return nil
|
||||
}
|
||||
|
||||
// resolve pins every package the requirements need, each file by its hash,
|
||||
// into a lock in the environment, and returns its path.
|
||||
//
|
||||
// With a second index — PyTorch's, for the torch a build was made against —
|
||||
// the packages it is for come from it and nowhere else, and every other
|
||||
// package from PyPI alone. Letting both indexes offer every package is how
|
||||
// torchtriton was taken in 2022: a name one index had, registered on the
|
||||
// other with a higher version. So the second index is explicit, named only
|
||||
// for its packages, and the lock is PEP 751's, which records each file's
|
||||
// address as well as its hash: installing it asks no index anything.
|
||||
func (in *Installer) resolve(ctx context.Context, env []string, uv, python, tmp, asOf string,
|
||||
reqs []string, b Build,
|
||||
) (string, error) {
|
||||
if b.ExtraIndex == "" {
|
||||
lock := filepath.Join(tmp, lockName)
|
||||
if out, err := in.runStdin(ctx, env, strings.Join(reqs, "\n")+"\n", uv, "pip", "compile",
|
||||
"--python", python, "--exclude-newer", asOf, "--generate-hashes", "--quiet",
|
||||
"--output-file", lock, "-"); err != nil {
|
||||
return "", fmt.Errorf("resolving %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
|
||||
}
|
||||
return lock, nil
|
||||
}
|
||||
|
||||
project := tmp + ".resolve"
|
||||
if err := os.MkdirAll(project, 0o755); err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer os.RemoveAll(project) //nolint:errcheck // scratch
|
||||
if err := os.WriteFile(filepath.Join(project, "pyproject.toml"), []byte(resolveProject(reqs, b)), 0o644); err != nil { //nolint:gosec // not secret
|
||||
return "", err
|
||||
}
|
||||
lock := filepath.Join(tmp, pylockName)
|
||||
if out, err := in.run(ctx, env, uv, "pip", "compile", "--python", python, "--exclude-newer", asOf,
|
||||
"--format", "pylock.toml", "--quiet", "--output-file", lock,
|
||||
filepath.Join(project, "pyproject.toml")); err != nil {
|
||||
return "", fmt.Errorf("resolving %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
|
||||
}
|
||||
return lock, nil
|
||||
}
|
||||
|
||||
// resolveProject is the pyproject.toml a build with a second index is
|
||||
// resolved from. Its packages are direct requirements too: an index is
|
||||
// given to a package only where the project names it.
|
||||
func resolveProject(reqs []string, b Build) string {
|
||||
var p strings.Builder
|
||||
p.WriteString("[project]\nname = \"llmgw-engine\"\nversion = \"0\"\ndependencies = [\n")
|
||||
for _, r := range append(slices.Clone(reqs), b.IndexPackages...) {
|
||||
fmt.Fprintf(&p, " %s,\n", strconv.Quote(r))
|
||||
}
|
||||
p.WriteString("]\n\n[[tool.uv.index]]\nname = \"extra\"\n")
|
||||
fmt.Fprintf(&p, "url = %s\nexplicit = true\n\n[tool.uv.sources]\n", strconv.Quote(b.ExtraIndex))
|
||||
for _, name := range b.IndexPackages {
|
||||
fmt.Fprintf(&p, "%s = { index = \"extra\" }\n", name)
|
||||
}
|
||||
return p.String()
|
||||
}
|
||||
|
||||
// published is when PyPI received a release's last file.
|
||||
func (in *Installer) published(ctx context.Context, project, version string) (time.Time, error) {
|
||||
url := strings.TrimSuffix(in.pypiBase(), "/") + "/pypi/" + project + "/" + version + "/json"
|
||||
|
||||
@@ -40,6 +40,14 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
|
||||
return []byte("uv 0.12.20\n"), nil
|
||||
case strings.HasSuffix(name, "/uv") && args[0] == "venv":
|
||||
return nil, os.MkdirAll(filepath.Join(args[len(args)-1], "bin"), 0o755)
|
||||
case strings.HasSuffix(name, "/uv") && args[0] == "pip" && args[1] == "compile":
|
||||
// From a pyproject: kept in the record, as the request was.
|
||||
project, _ := os.ReadFile(args[len(args)-1])
|
||||
f.pmu.Lock()
|
||||
f.commands = append(f.commands, "pyproject: "+string(project))
|
||||
f.pmu.Unlock()
|
||||
out := args[slices.Index(args, "--output-file")+1]
|
||||
return nil, os.WriteFile(out, []byte("lock-version = \"1.0\"\n"), 0o644)
|
||||
case strings.HasSuffix(name, "/uv") && args[0] == "pip" && args[1] == "install":
|
||||
if f.failPip {
|
||||
return []byte("error: Hash mismatch for torch\n"), errExit
|
||||
@@ -220,8 +228,12 @@ func TestAnOlderDriverGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
|
||||
// Pinned to the digest GitHub publishes for the wheel.
|
||||
"vllm @ " + in.DownloadBase + "/vllm-project/vllm/releases/download/v0.30.0/" + wheel + "#sha256=" +
|
||||
strings.TrimPrefix(gh.digests[wheel], "sha256:"),
|
||||
"--extra-index-url https://download.pytorch.org/whl/cu129 --index-strategy unsafe-best-match -",
|
||||
"--require-hashes --no-deps -r",
|
||||
// The torch packages from PyTorch's index and only those; the lock
|
||||
// records every file's address.
|
||||
"--format pylock.toml",
|
||||
`url = "https://download.pytorch.org/whl/cu129"` + "\nexplicit = true",
|
||||
`torch = { index = "extra" }`, `torchvision = { index = "extra" }`,
|
||||
"--require-hashes --no-deps -r " + filepath.Join(in.Root, "vllm", "0.30.0-cu129.partial", pylockName),
|
||||
// Computing on the card, not only importing.
|
||||
"/bin/python -c import torch",
|
||||
} {
|
||||
@@ -230,8 +242,8 @@ func TestAnOlderDriverGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
|
||||
}
|
||||
}
|
||||
for _, c := range sys.commands {
|
||||
if strings.Contains(c, "pip install") && !strings.Contains(c, "download.pytorch.org/whl/cu129") {
|
||||
t.Errorf("installed without the index its torch comes from: %s", c)
|
||||
if strings.Contains(c, "unsafe-best-match") || strings.Contains(c, "--extra-index-url") {
|
||||
t.Errorf("every index offered every package: %s", c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -50,10 +50,12 @@ type Build struct {
|
||||
GPUBackend string
|
||||
// Wheel, when set, is the engine itself as a wheel from a GitHub
|
||||
// release rather than from PyPI, checked against the digest GitHub
|
||||
// publishes for it. ExtraIndex is a package index besides PyPI that the
|
||||
// rest resolves from too: the CUDA 12.9 torch the wheel was built on.
|
||||
Wheel *Wheel
|
||||
ExtraIndex string
|
||||
// publishes for it. ExtraIndex is a package index besides PyPI, for
|
||||
// IndexPackages only: the CUDA 12.9 torch the wheel was built on. Every
|
||||
// other package comes from PyPI alone.
|
||||
Wheel *Wheel
|
||||
ExtraIndex string
|
||||
IndexPackages []string
|
||||
// GPUCheck, on a node with an NVIDIA GPU, has the installed environment
|
||||
// compute something on one before the build counts as installed. A build
|
||||
// with no kernels for the card — a GPU generation its torch dropped —
|
||||
@@ -234,6 +236,10 @@ var VLLMVariants = []string{"cu13", "cu129"}
|
||||
// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
|
||||
const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
|
||||
|
||||
// pytorchPackages are what is taken from PyTorch's index: the builds for one
|
||||
// CUDA version, which PyPI does not have.
|
||||
var pytorchPackages = []string{"torch", "torchvision", "torchaudio"}
|
||||
|
||||
// vllmCUDA13Since is the first vLLM whose PyPI build is CUDA 13, read from
|
||||
// its releases: 0.19.1 publishes "+cu130" beside a CUDA 12 default, 0.20.0
|
||||
// "+cu129" beside a CUDA 13 one.
|
||||
@@ -332,6 +338,7 @@ func vllmCandidates(version, variant string, h discovery.Host) ([]Build, error)
|
||||
Prefix: "vllm-" + v + "+cu129-", Suffix: "_" + map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch] + ".whl",
|
||||
}
|
||||
cu129.ExtraIndex = pytorchCU129
|
||||
cu129.IndexPackages = pytorchPackages
|
||||
return pick("vLLM "+v, variant, []candidate{
|
||||
{pypi("cu13"), gpuNeeds{minDriverForCUDA13, minCC}, "vLLM " + v + " built for CUDA 13"},
|
||||
{cu129, gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v + " built for CUDA 12.9"},
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
package supervisor
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
// Engines run as their own user, not as the agent's root: an engine reads a
|
||||
// model's files and answers requests, and a model's files are whatever a
|
||||
// registry published. Root in that process would be root on the node for
|
||||
// anything that can make an engine misbehave — a malformed file, a template,
|
||||
// a flag nobody thought of.
|
||||
//
|
||||
// The user is DefaultRunAs, made by `llmgw node add`, in the video and render
|
||||
// groups for the GPU devices. A node without it keeps running engines as
|
||||
// root, and says so, so an existing fleet is not broken by an upgrade.
|
||||
|
||||
// DefaultRunAs is the user engines run as.
|
||||
const DefaultRunAs = "llmgw-engine"
|
||||
|
||||
// runAs is how an engine process is started as another user: its
|
||||
// credential, a working directory it owns, and the environment that points
|
||||
// its caches there.
|
||||
type runAs struct {
|
||||
name string
|
||||
cred *syscall.Credential
|
||||
home string
|
||||
}
|
||||
|
||||
// lookupUser is user.Lookup, replaced by tests.
|
||||
var lookupUser = user.Lookup
|
||||
|
||||
// resolveRunAs finds the user engines run as. nil means as this process —
|
||||
// when it is not root, there is nobody to drop to; when the user does not
|
||||
// exist, why is returned.
|
||||
func resolveRunAs(name, workDir string) (*runAs, string) {
|
||||
if name == "" || name == "root" || os.Geteuid() != 0 {
|
||||
return nil, ""
|
||||
}
|
||||
u, err := lookupUser(name)
|
||||
if err != nil {
|
||||
return nil, fmt.Sprintf("there is no user %s, so engines run as root; `llmgw node add` makes it, "+
|
||||
"or: useradd --system --shell /usr/sbin/nologin %s && usermod -aG video,render %s", name, name, name)
|
||||
}
|
||||
r, err := credentialFor(u, workDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Sprintf("engines run as root: %v", err)
|
||||
}
|
||||
return r, ""
|
||||
}
|
||||
|
||||
func credentialFor(u *user.User, workDir string) (*runAs, error) {
|
||||
uid, err1 := strconv.ParseUint(u.Uid, 10, 32)
|
||||
gid, err2 := strconv.ParseUint(u.Gid, 10, 32)
|
||||
if err1 != nil || err2 != nil {
|
||||
return nil, fmt.Errorf("user %s has ids %s:%s", u.Username, u.Uid, u.Gid)
|
||||
}
|
||||
var groups []uint32
|
||||
if ids, err := u.GroupIds(); err == nil {
|
||||
for _, g := range ids {
|
||||
if n, err := strconv.ParseUint(g, 10, 32); err == nil {
|
||||
groups = append(groups, uint32(n))
|
||||
}
|
||||
}
|
||||
}
|
||||
// A home of its own under the work directory, which is root's: caches
|
||||
// (Triton, torch, Hugging Face) are written there and nowhere else.
|
||||
home := filepath.Join(workDir, "engine-home")
|
||||
if err := os.MkdirAll(home, 0o700); err != nil {
|
||||
return nil, fmt.Errorf("making %s: %w", home, err)
|
||||
}
|
||||
if err := os.Chown(home, int(uid), int(gid)); err != nil {
|
||||
return nil, fmt.Errorf("giving %s to %s: %w", home, u.Username, err)
|
||||
}
|
||||
// The work directory is 0750 root's; the engine has to pass through it
|
||||
// to its home.
|
||||
if err := os.Chmod(workDir, 0o751); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &runAs{name: u.Username, home: home,
|
||||
cred: &syscall.Credential{Uid: uint32(uid), Gid: uint32(gid), Groups: groups}}, nil
|
||||
}
|
||||
|
||||
// env is what is added to an engine's environment when it runs as r: the
|
||||
// last value of a key wins in exec.
|
||||
func (r *runAs) env() []string {
|
||||
return []string{"HOME=" + r.home, "XDG_CACHE_HOME=" + filepath.Join(r.home, ".cache"), "USER=" + r.name, "LOGNAME=" + r.name}
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
package supervisor
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"syscall"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestAnEngineIsGivenItsUsersCredentialHomeAndCaches(t *testing.T) {
|
||||
me, err := user.Current()
|
||||
if err != nil {
|
||||
t.Skip(err)
|
||||
}
|
||||
work := filepath.Join(t.TempDir(), "agent")
|
||||
if err := os.MkdirAll(work, 0o750); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r, err := credentialFor(me, work)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
uid, _ := strconv.Atoi(me.Uid)
|
||||
if int(r.cred.Uid) != uid || r.home != filepath.Join(work, "engine-home") {
|
||||
t.Fatalf("%+v", r)
|
||||
}
|
||||
if st, err := os.Stat(r.home); err != nil || st.Mode().Perm() != 0o700 {
|
||||
t.Errorf("home: %v %v", st.Mode(), err)
|
||||
}
|
||||
s := &Supervisor{cfg: Config{WorkDir: work}, as: r}
|
||||
c := exec.Command("true")
|
||||
c.Env = []string{"HOME=/root", "PATH=/bin"}
|
||||
c.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
s.dropTo(c)
|
||||
if c.SysProcAttr.Credential == nil || c.Dir != r.home || !slices.Contains(c.Env, "HOME="+r.home) {
|
||||
t.Errorf("dir %s env %v cred %v", c.Dir, c.Env, c.SysProcAttr.Credential)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNotRootOrNoSuchUserLeavesEnginesAsTheAgent(t *testing.T) {
|
||||
if os.Geteuid() == 0 {
|
||||
t.Skip("run as root")
|
||||
}
|
||||
if r, why := resolveRunAs(DefaultRunAs, t.TempDir()); r != nil || why != "" {
|
||||
t.Errorf("not root: %v %q", r, why)
|
||||
}
|
||||
}
|
||||
@@ -39,6 +39,9 @@ type Config struct {
|
||||
// as ready. There is no external signal for a worker, so staying alive is
|
||||
// the only evidence available.
|
||||
AliveGrace time.Duration `koanf:"alive_grace"`
|
||||
// RunAs is the user engines run as when the agent is root; empty is
|
||||
// llmgw-engine, "root" keeps them root.
|
||||
RunAs string `koanf:"run_as"`
|
||||
}
|
||||
|
||||
// DefaultConfig returns sensible defaults.
|
||||
@@ -50,6 +53,7 @@ func DefaultConfig() Config {
|
||||
HealthInterval: 2 * time.Second,
|
||||
RendezvousTimeout: 5 * time.Minute,
|
||||
AliveGrace: 5 * time.Second,
|
||||
RunAs: DefaultRunAs,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -58,6 +62,9 @@ func (c *Config) normalize() {
|
||||
if c.WorkDir == "" {
|
||||
c.WorkDir = d.WorkDir
|
||||
}
|
||||
if c.RunAs == "" {
|
||||
c.RunAs = d.RunAs
|
||||
}
|
||||
if c.LogLines <= 0 {
|
||||
c.LogLines = d.LogLines
|
||||
}
|
||||
@@ -154,6 +161,9 @@ type Supervisor struct {
|
||||
|
||||
mu sync.Mutex
|
||||
shards map[string]*Shard
|
||||
|
||||
// as is who engines run as; nil is as the agent.
|
||||
as *runAs
|
||||
}
|
||||
|
||||
// New builds a supervisor.
|
||||
@@ -165,7 +175,16 @@ func New(cfg Config, health HealthChecker, log *slog.Logger) (*Supervisor, error
|
||||
if err := os.MkdirAll(cfg.WorkDir, 0o750); err != nil {
|
||||
return nil, fmt.Errorf("work dir: %w", err)
|
||||
}
|
||||
return &Supervisor{cfg: cfg, log: log, health: health, shards: map[string]*Shard{}}, nil
|
||||
s := &Supervisor{cfg: cfg, log: log, health: health, shards: map[string]*Shard{}}
|
||||
as, why := resolveRunAs(cfg.RunAs, cfg.WorkDir)
|
||||
if why != "" {
|
||||
log.Warn(why)
|
||||
}
|
||||
if as != nil {
|
||||
log.Info("engines run as their own user", "user", as.name)
|
||||
}
|
||||
s.as = as
|
||||
return s, nil
|
||||
}
|
||||
|
||||
// ErrAlreadyRunning means a shard with that ID is already supervised.
|
||||
@@ -261,6 +280,7 @@ func (s *Supervisor) launch(ctx context.Context, sh *Shard, cmds []engines.Comma
|
||||
// Engines spawn workers; signalling only the parent leaves those workers
|
||||
// holding GPU memory.
|
||||
c.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
s.dropTo(c)
|
||||
// CommandContext kills only the leader, so replace that with a group kill.
|
||||
c.Cancel = func() error { return killGroup(c.Process.Pid, syscall.SIGKILL) }
|
||||
|
||||
@@ -298,6 +318,7 @@ func (s *Supervisor) spawn(ctx context.Context, sh *Shard, c engines.Command) (*
|
||||
proc.Env = c.Env
|
||||
proc.Dir = s.cfg.WorkDir
|
||||
proc.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
s.dropTo(proc)
|
||||
proc.Cancel = func() error { return killGroup(proc.Process.Pid, syscall.SIGKILL) }
|
||||
|
||||
stdout, err := proc.StdoutPipe()
|
||||
@@ -779,3 +800,14 @@ func (s *Supervisor) KillOrphan(o Orphan) error {
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// dropTo makes a command run as the engines' user, when there is one: its
|
||||
// credential, its home as the working directory, and its caches there.
|
||||
func (s *Supervisor) dropTo(c *exec.Cmd) {
|
||||
if s.as == nil {
|
||||
return
|
||||
}
|
||||
c.SysProcAttr.Credential = s.as.cred
|
||||
c.Dir = s.as.home
|
||||
c.Env = append(c.Env, s.as.env()...)
|
||||
}
|
||||
@@ -19,10 +19,20 @@ func TestAnAgentUpgradesItselfAndSaysSoBeforeRestarting(t *testing.T) {
|
||||
body := []byte("the new agent")
|
||||
sum := sha256.Sum256(body)
|
||||
asset := release.AssetName("v9.9.9", runtime.GOOS, runtime.GOARCH)
|
||||
// Signed, as every release is: with a key this test's build trusts.
|
||||
pub, priv, kerr := release.GenerateKey()
|
||||
if kerr != nil {
|
||||
t.Fatal(kerr)
|
||||
}
|
||||
t.Setenv("LLMGW_RELEASE_KEY", pub)
|
||||
sums := hex.EncodeToString(sum[:]) + " " + asset + "\n"
|
||||
sig, _ := release.Sign(priv, []byte(sums))
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/v9.9.9/checksums.txt":
|
||||
_, _ = w.Write([]byte(hex.EncodeToString(sum[:]) + " " + asset + "\n"))
|
||||
_, _ = w.Write([]byte(sums))
|
||||
case "/v9.9.9/" + release.SignatureName:
|
||||
_, _ = w.Write([]byte(sig))
|
||||
case "/v9.9.9/" + asset:
|
||||
_, _ = w.Write(body)
|
||||
default:
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"os/exec"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
|
||||
)
|
||||
|
||||
type alwaysReady struct{}
|
||||
|
||||
func (alwaysReady) Ready(context.Context, string, string, string) error { return nil }
|
||||
|
||||
func TestAnAdoptedEngineThatDiesIsReported(t *testing.T) {
|
||||
sup, err := supervisor.New(supervisor.Config{WorkDir: t.TempDir(), RunAs: "-"}, alwaysReady{}, slog.New(slog.DiscardHandler))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// An engine a previous agent started: not this process's child.
|
||||
cmd := exec.Command("sleep", "60")
|
||||
if err := cmd.Start(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
go func() { _ = cmd.Wait() }()
|
||||
sh, err := sup.Adopt(supervisor.Orphan{ShardID: "m-r0", ReplicaID: "m", PID: cmd.Process.Pid,
|
||||
Spec: engines.ShardSpec{ReplicaID: "m", ShardID: "m-r0", HTTPPort: 1}, Alive: true})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := sup.WaitReady(t.Context(), sh, "http://127.0.0.1:1", "m", 5*time.Second); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
a := &Agent{sup: sup, log: slog.New(slog.DiscardHandler)}
|
||||
out := make(chan *llmgwv1.AgentMessage, 4)
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
defer cancel()
|
||||
go a.watchExits(ctx, out)
|
||||
time.Sleep(1500 * time.Millisecond) // seen ready
|
||||
_ = cmd.Process.Kill()
|
||||
|
||||
select {
|
||||
case m := <-out:
|
||||
s := m.GetShard()
|
||||
if s.GetShardId() != "m-r0" || s.GetReplicaId() != "m" {
|
||||
t.Errorf("reported %+v", s)
|
||||
}
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Fatal("the adopted engine died and nothing was reported")
|
||||
}
|
||||
}
|
||||
+51
-3
@@ -96,6 +96,9 @@ var (
|
||||
ErrInvalid = errors.New("invalid api key")
|
||||
ErrExpired = errors.New("api key expired")
|
||||
ErrRevoked = errors.New("api key revoked")
|
||||
// ErrThrottled is a client presenting more unknown keys than it is
|
||||
// allowed to have looked up.
|
||||
ErrThrottled = errors.New("too many unrecognised api keys from this address")
|
||||
)
|
||||
|
||||
// Authenticator resolves keys and caches the result.
|
||||
@@ -111,6 +114,10 @@ type Authenticator struct {
|
||||
// and writing it per request would put a database write back on the path.
|
||||
touchMu sync.Mutex
|
||||
touch map[string]time.Time
|
||||
|
||||
// lookups rations the database reads of keys not in the cache, per
|
||||
// client address.
|
||||
lookups *buckets
|
||||
}
|
||||
|
||||
type entry struct {
|
||||
@@ -124,13 +131,22 @@ func New(s store.Store, cfg Config) *Authenticator {
|
||||
cfg.normalize()
|
||||
return &Authenticator{
|
||||
store: s, cfg: cfg, now: time.Now,
|
||||
cache: map[string]entry{},
|
||||
touch: map[string]time.Time{},
|
||||
cache: map[string]entry{},
|
||||
touch: map[string]time.Time{},
|
||||
lookups: newBuckets(20, 50),
|
||||
}
|
||||
}
|
||||
|
||||
// Authenticate resolves a presented credential.
|
||||
func (a *Authenticator) Authenticate(ctx context.Context, presented string) (Principal, error) {
|
||||
return a.AuthenticateFrom(ctx, presented, "")
|
||||
}
|
||||
|
||||
// AuthenticateFrom resolves a credential presented by client, an address:
|
||||
// a key not in the cache costs a database read, so those are rationed per
|
||||
// client, and one that has made too many is told to slow down rather than
|
||||
// read for. Empty client is not rationed.
|
||||
func (a *Authenticator) AuthenticateFrom(ctx context.Context, presented, client string) (Principal, error) {
|
||||
if presented == "" {
|
||||
return Principal{}, ErrMissing
|
||||
}
|
||||
@@ -150,6 +166,9 @@ func (a *Authenticator) Authenticate(ctx context.Context, presented string) (Pri
|
||||
return e.principal, e.err
|
||||
}
|
||||
|
||||
if client != "" && !a.lookups.allow(client, a.now()) {
|
||||
return Principal{}, ErrThrottled
|
||||
}
|
||||
p, err := a.resolve(ctx, hash)
|
||||
if err != nil && !isRejection(err) {
|
||||
// A database problem is not an authentication answer, so it must not be
|
||||
@@ -185,6 +204,11 @@ func (a *Authenticator) resolve(ctx context.Context, hash []byte) (Principal, er
|
||||
}
|
||||
|
||||
eff, err := a.store.Resolve(ctx, k.ID)
|
||||
if errors.Is(err, store.ErrOwnerInactive) {
|
||||
// Said as a revoked key is: whether the key exists is not the
|
||||
// caller's to learn.
|
||||
return Principal{}, ErrRevoked
|
||||
}
|
||||
if err != nil {
|
||||
return Principal{}, err
|
||||
}
|
||||
@@ -206,10 +230,34 @@ func (a *Authenticator) lookup(id string) (entry, bool) {
|
||||
return e, true
|
||||
}
|
||||
|
||||
// maxCached bounds the cache. Every well-formed key anyone presents is an
|
||||
// entry, a rejected one included, so without a bound a stream of made-up keys
|
||||
// grows it until the process runs out of memory.
|
||||
const maxCached = 100_000
|
||||
|
||||
func (a *Authenticator) store_(id string, e entry) {
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
if len(a.cache) >= maxCached {
|
||||
now := a.now()
|
||||
for k, v := range a.cache {
|
||||
if now.After(v.expires) {
|
||||
delete(a.cache, k)
|
||||
}
|
||||
}
|
||||
// Still full of live entries: half of them go, which costs only a
|
||||
// lookup each for the keys in use.
|
||||
if len(a.cache) >= maxCached {
|
||||
n := 0
|
||||
for k := range a.cache {
|
||||
if n%2 == 0 {
|
||||
delete(a.cache, k)
|
||||
}
|
||||
n++
|
||||
}
|
||||
}
|
||||
}
|
||||
a.cache[id] = e
|
||||
a.mu.Unlock()
|
||||
}
|
||||
|
||||
// Invalidate drops every cached decision.
|
||||
|
||||
@@ -438,3 +438,43 @@ func TestAlterAlwaysChangesTheKey(t *testing.T) {
|
||||
t.Error("alter is a no-op when the key already ends in the replacement")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAFloodOfMadeUpKeysFromOneAddressIsSlowedBeforeTheDatabase(t *testing.T) {
|
||||
s, secret, _ := fixture(t)
|
||||
a := New(s, DefaultConfig())
|
||||
now := time.Now()
|
||||
a.now = func() time.Time { return now }
|
||||
ctx := context.Background()
|
||||
|
||||
throttled := 0
|
||||
for i := 0; i < 200; i++ {
|
||||
madeUp, _, _ := store.NewKey("x", "x", "proj1")
|
||||
if _, err := a.AuthenticateFrom(ctx, madeUp, "203.0.113.9"); errors.Is(err, ErrThrottled) {
|
||||
throttled++
|
||||
}
|
||||
}
|
||||
if throttled < 100 {
|
||||
t.Errorf("only %d of 200 made-up keys were held back", throttled)
|
||||
}
|
||||
// Another address, and the real key, are not caught by it.
|
||||
if _, err := a.AuthenticateFrom(ctx, secret, "198.51.100.1"); err != nil {
|
||||
t.Errorf("a real key from another address: %v", err)
|
||||
}
|
||||
// Time restores the allowance.
|
||||
now = now.Add(5 * time.Second)
|
||||
madeUp, _, _ := store.NewKey("y", "y", "proj1")
|
||||
if _, err := a.AuthenticateFrom(ctx, madeUp, "203.0.113.9"); errors.Is(err, ErrThrottled) {
|
||||
t.Error("still held back after waiting")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheCacheDoesNotGrowWithoutBound(t *testing.T) {
|
||||
s, _, _ := fixture(t)
|
||||
a := New(s, DefaultConfig())
|
||||
for i := 0; i < maxCached+10; i++ {
|
||||
a.store_(string(rune(i))+"-"+time.Now().String(), entry{expires: time.Now().Add(time.Hour)})
|
||||
}
|
||||
if n := len(a.cache); n > maxCached {
|
||||
t.Errorf("%d entries", n)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package auth
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// buckets are token buckets by key: rate a second, up to burst at once. Ones
|
||||
// full and unused are dropped as more arrive, so a key is not kept for ever.
|
||||
type buckets struct {
|
||||
mu sync.Mutex
|
||||
rate float64
|
||||
burst float64
|
||||
by map[string]*bucket
|
||||
}
|
||||
|
||||
type bucket struct {
|
||||
tokens float64
|
||||
at time.Time
|
||||
}
|
||||
|
||||
func newBuckets(rate, burst float64) *buckets {
|
||||
return &buckets{rate: rate, burst: burst, by: map[string]*bucket{}}
|
||||
}
|
||||
|
||||
// allow takes a token for key at now, reporting whether there was one.
|
||||
func (b *buckets) allow(key string, now time.Time) bool {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
if len(b.by) > 50_000 {
|
||||
for k, v := range b.by {
|
||||
if now.Sub(v.at).Seconds()*b.rate >= b.burst {
|
||||
delete(b.by, k)
|
||||
}
|
||||
}
|
||||
}
|
||||
k, ok := b.by[key]
|
||||
if !ok {
|
||||
k = &bucket{tokens: b.burst, at: now}
|
||||
b.by[key] = k
|
||||
}
|
||||
k.tokens = min(b.burst, k.tokens+now.Sub(k.at).Seconds()*b.rate)
|
||||
k.at = now
|
||||
if k.tokens < 1 {
|
||||
return false
|
||||
}
|
||||
k.tokens--
|
||||
return true
|
||||
}
|
||||
@@ -6,6 +6,8 @@ import (
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/clientip"
|
||||
)
|
||||
|
||||
// Middleware authenticates every request before it reaches the router.
|
||||
@@ -24,7 +26,7 @@ func Middleware(a *Authenticator, log *slog.Logger, exempt ...string) func(http.
|
||||
return
|
||||
}
|
||||
|
||||
p, err := a.Authenticate(r.Context(), BearerToken(r))
|
||||
p, err := a.AuthenticateFrom(r.Context(), BearerToken(r), clientAddr(r))
|
||||
if err != nil {
|
||||
writeAuthError(w, err)
|
||||
// The key prefix, never the key: a rejected credential in a log
|
||||
@@ -66,6 +68,9 @@ func writeAuthError(w http.ResponseWriter, err error) {
|
||||
switch {
|
||||
case errors.Is(err, ErrMissing):
|
||||
msg = "no api key was provided"
|
||||
case errors.Is(err, ErrThrottled):
|
||||
w.Header().Set("Retry-After", "1")
|
||||
status, kind, msg = http.StatusTooManyRequests, "rate_limit_exceeded", "too many unrecognised api keys; slow down"
|
||||
case errors.Is(err, ErrExpired):
|
||||
msg = "this api key has expired"
|
||||
case errors.Is(err, ErrRevoked):
|
||||
@@ -104,3 +109,7 @@ func WriteForbiddenModel(w http.ResponseWriter, model string) {
|
||||
"type": "invalid_request_error", "code": nil,
|
||||
}})
|
||||
}
|
||||
|
||||
// clientAddr is the address a request came from: through a trusted proxy,
|
||||
// the client's.
|
||||
func clientAddr(r *http.Request) string { return clientip.Addr(r) }
|
||||
@@ -0,0 +1,100 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
)
|
||||
|
||||
// The admin token: what the admin API's /api and /debug paths demand, kept
|
||||
// in a file only root and the server can read.
|
||||
|
||||
// defaultAdminTokenFile is where the server writes the token under its
|
||||
// systemd unit — its state directory, the one place ProtectSystem=strict
|
||||
// leaves it to write — and where clients look for it when told nothing.
|
||||
const defaultAdminTokenFile = "/var/lib/llmgw/admin-token"
|
||||
|
||||
// adminTokenPath is the token's file for a configuration: as it says; under
|
||||
// systemd, in the unit's state directory; else secrets/admin-token beside
|
||||
// the configuration, for a server run by hand.
|
||||
func adminTokenPath(cfgPath string, cfg config.Server) string {
|
||||
if cfg.AdminTokenFile != "" {
|
||||
return cfg.AdminTokenFile
|
||||
}
|
||||
if dir := os.Getenv("STATE_DIRECTORY"); dir != "" {
|
||||
return filepath.Join(strings.Split(dir, ":")[0], "admin-token")
|
||||
}
|
||||
return filepath.Join(filepath.Dir(cfgPath), "secrets", "admin-token")
|
||||
}
|
||||
|
||||
// ensureAdminToken reads the token, making it when there is none.
|
||||
func ensureAdminToken(path string) (string, error) {
|
||||
if b, err := os.ReadFile(path); err == nil { //nolint:gosec // the configured token file
|
||||
if t := strings.TrimSpace(string(b)); len(t) >= 32 {
|
||||
return t, nil
|
||||
}
|
||||
} else if !errors.Is(err, fs.ErrNotExist) {
|
||||
return "", fmt.Errorf("reading the admin token %s: %w", path, err)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
|
||||
return "", fmt.Errorf("making %s: %w", filepath.Dir(path), err)
|
||||
}
|
||||
raw := make([]byte, 32)
|
||||
if _, err := rand.Read(raw); err != nil {
|
||||
return "", err
|
||||
}
|
||||
t := hex.EncodeToString(raw)
|
||||
if err := os.WriteFile(path, []byte(t+"\n"), 0o600); err != nil {
|
||||
return "", fmt.Errorf("writing the admin token %s: %w", path, err)
|
||||
}
|
||||
return t, nil
|
||||
}
|
||||
|
||||
// readAdminToken is the token a client sends: LLMGW_ADMIN_TOKEN, else the
|
||||
// first of these files it can read. Empty when none: the request then goes
|
||||
// without, and the API says what is missing.
|
||||
func readAdminToken(paths ...string) string {
|
||||
if t := strings.TrimSpace(os.Getenv("LLMGW_ADMIN_TOKEN")); t != "" {
|
||||
return t
|
||||
}
|
||||
for _, p := range append(paths, defaultAdminTokenFile) {
|
||||
if p == "" {
|
||||
continue
|
||||
}
|
||||
if b, err := os.ReadFile(p); err == nil { //nolint:gosec // the admin token file
|
||||
if t := strings.TrimSpace(string(b)); t != "" {
|
||||
return t
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// checkAdminListen refuses an admin listener off loopback unless the
|
||||
// configuration says it is meant.
|
||||
func checkAdminListen(cfg config.Server) error {
|
||||
host, _, err := net.SplitHostPort(cfg.AdminListen)
|
||||
if err != nil {
|
||||
return fmt.Errorf("admin_listen %q: %w", cfg.AdminListen, err)
|
||||
}
|
||||
if host == "localhost" {
|
||||
return nil
|
||||
}
|
||||
if ip := net.ParseIP(host); ip != nil && ip.IsLoopback() {
|
||||
return nil
|
||||
}
|
||||
if cfg.AdminRemote {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("admin_listen %s is not loopback: the admin API adds models and upgrades nodes, and "+
|
||||
"its token would cross the network in clear; keep it on 127.0.0.1 and reach it by ssh, or set "+
|
||||
"admin_remote: true", cfg.AdminListen)
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
)
|
||||
|
||||
func TestTheAdminTokenIsMadeOnceReadableOnlyByItsOwner(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cfgPath := filepath.Join(dir, "server.yaml")
|
||||
p := adminTokenPath(cfgPath, config.Default())
|
||||
if p != filepath.Join(dir, "secrets", "admin-token") {
|
||||
t.Fatalf("path %s", p)
|
||||
}
|
||||
tok, err := ensureAdminToken(p)
|
||||
if err != nil || len(tok) != 64 {
|
||||
t.Fatalf("%q %v", tok, err)
|
||||
}
|
||||
if st, _ := os.Stat(p); st.Mode().Perm() != 0o600 {
|
||||
t.Errorf("mode %v", st.Mode())
|
||||
}
|
||||
again, _ := ensureAdminToken(p)
|
||||
if again != tok {
|
||||
t.Error("a second start made a new token")
|
||||
}
|
||||
t.Setenv("LLMGW_ADMIN_TOKEN", "")
|
||||
if got := readAdminToken(p); got != tok {
|
||||
t.Errorf("read %q", got)
|
||||
}
|
||||
t.Setenv("LLMGW_ADMIN_TOKEN", "from-env")
|
||||
if got := readAdminToken(p); got != "from-env" {
|
||||
t.Errorf("env: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheAdminListenerStaysOnLoopbackUnlessToldOtherwise(t *testing.T) {
|
||||
cfg := config.Default()
|
||||
for addr, ok := range map[string]bool{"127.0.0.1:9090": true, "localhost:9090": true, "[::1]:9090": true,
|
||||
"0.0.0.0:9090": false, ":9090": false, "10.0.0.140:9090": false} {
|
||||
cfg.AdminListen = addr
|
||||
if err := checkAdminListen(cfg); (err == nil) != ok {
|
||||
t.Errorf("%s: %v", addr, err)
|
||||
}
|
||||
}
|
||||
cfg.AdminListen, cfg.AdminRemote = "10.0.0.140:9090", true
|
||||
if err := checkAdminListen(cfg); err != nil {
|
||||
t.Errorf("admin_remote: %v", err)
|
||||
}
|
||||
}
|
||||
+37
-9
@@ -15,6 +15,7 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
agecrypt "filippo.io/age"
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
@@ -44,9 +45,11 @@ func newBackupCmd() *cobra.Command {
|
||||
"people, quotas, usage, the models added at runtime, the secret engine\n" +
|
||||
"keys derive from — and /etc/llmgw, which holds the configuration, the\n" +
|
||||
"store's address and the certificate authority's key.\n\n" +
|
||||
"The archive holds that key, so it is written readable by root only. It\n" +
|
||||
"is on the same disk as what it backs up: copy it elsewhere, or it does\n" +
|
||||
"not survive the loss of this machine.\n\n" +
|
||||
"The archive holds that key, so it is written readable by root only, and\n" +
|
||||
"encrypted to backup_recipients in server.yaml when they are set (age\n" +
|
||||
"recipients or SSH public keys; open it with `age -d -i <key>`). It is on\n" +
|
||||
"the same disk as what it backs up: copy it elsewhere, or it does not\n" +
|
||||
"survive the loss of this machine.\n\n" +
|
||||
"Run daily by llmgw-backup.timer (`llmgw server backup`). The control plane publishes the time of\n" +
|
||||
"the newest archive as llmgw_backup_last_success_timestamp_seconds, and\n" +
|
||||
"BackupTooOld fires when it is more than a day old.",
|
||||
@@ -55,7 +58,11 @@ func newBackupCmd() *cobra.Command {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
path, err := writeBackup(cmd.Context(), cfg.Auth.Store, etcDir, dir)
|
||||
to, err := config.ParseBackupRecipients(cfg.BackupRecipients)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
path, err := writeBackup(cmd.Context(), cfg.Auth.Store, etcDir, dir, to)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -87,7 +94,9 @@ const backupPrefix = "llmgw-backup-"
|
||||
// writeBackup writes one archive and returns its path. Written beside its
|
||||
// final name and renamed, so a backup interrupted halfway is never the newest
|
||||
// one the control plane reports.
|
||||
func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, error) {
|
||||
//
|
||||
// With recipients, the archive is encrypted to them, and named .tar.gz.age.
|
||||
func writeBackup(ctx context.Context, storeAddr, etcDir, dir string, recipients []agecrypt.Recipient) (string, error) {
|
||||
// Listable by anyone, so the control plane — not root — can see when the
|
||||
// newest archive was written. The archives themselves are root's alone.
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
@@ -95,6 +104,9 @@ func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, er
|
||||
}
|
||||
host, _ := os.Hostname()
|
||||
name := backupPrefix + host + "-" + time.Now().UTC().Format("20060102T150405Z") + ".tar.gz"
|
||||
if len(recipients) > 0 {
|
||||
name += ".age"
|
||||
}
|
||||
final := filepath.Join(dir, name)
|
||||
part := final + ".part"
|
||||
|
||||
@@ -108,7 +120,13 @@ func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, er
|
||||
_ = os.Remove(part)
|
||||
}
|
||||
}()
|
||||
gz := gzip.NewWriter(f)
|
||||
var sealed io.WriteCloser = nopCloser{f}
|
||||
if len(recipients) > 0 {
|
||||
if sealed, err = agecrypt.Encrypt(f, recipients...); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
gz := gzip.NewWriter(sealed)
|
||||
tw := tar.NewWriter(gz)
|
||||
|
||||
manifest := fmt.Sprintf("llmgw %s\nhost %s\ntime %s\n", version.Version, host,
|
||||
@@ -124,7 +142,7 @@ func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, er
|
||||
if err := addTree(tw, etcDir, "etc/llmgw"); err != nil {
|
||||
return "", fmt.Errorf("%s: %w", etcDir, err)
|
||||
}
|
||||
if err := errors.Join(tw.Close(), gz.Close(), f.Sync(), f.Close()); err != nil {
|
||||
if err := errors.Join(tw.Close(), gz.Close(), sealed.Close(), f.Sync(), f.Close()); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := os.Rename(part, final); err != nil {
|
||||
@@ -134,6 +152,16 @@ func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, er
|
||||
return final, nil
|
||||
}
|
||||
|
||||
type nopCloser struct{ io.Writer }
|
||||
|
||||
func (nopCloser) Close() error { return nil }
|
||||
|
||||
// isBackup reports whether a file name is an archive this command wrote.
|
||||
func isBackup(name string) bool {
|
||||
return strings.HasPrefix(name, backupPrefix) &&
|
||||
(strings.HasSuffix(name, ".tar.gz") || strings.HasSuffix(name, ".tar.gz.age"))
|
||||
}
|
||||
|
||||
// addStore dumps the store into the archive.
|
||||
func addStore(ctx context.Context, tw *tar.Writer, addr, scratch string) error {
|
||||
addr, err := store.ResolveAddress(addr)
|
||||
@@ -280,7 +308,7 @@ func pruneBackups(dir string, keep int) (int, error) {
|
||||
}
|
||||
var names []string
|
||||
for _, e := range entries {
|
||||
if n := e.Name(); strings.HasPrefix(n, backupPrefix) && strings.HasSuffix(n, ".tar.gz") {
|
||||
if n := e.Name(); isBackup(n) {
|
||||
names = append(names, n)
|
||||
}
|
||||
}
|
||||
@@ -306,7 +334,7 @@ func NewestBackup(dir string) time.Time {
|
||||
}
|
||||
var newest time.Time
|
||||
for _, e := range entries {
|
||||
if n := e.Name(); !strings.HasPrefix(n, backupPrefix) || !strings.HasSuffix(n, ".tar.gz") {
|
||||
if n := e.Name(); !isBackup(n) {
|
||||
continue
|
||||
}
|
||||
if info, err := e.Info(); err == nil && info.ModTime().After(newest) {
|
||||
|
||||
@@ -3,6 +3,7 @@ package cli
|
||||
import (
|
||||
"archive/tar"
|
||||
"compress/gzip"
|
||||
"crypto/ed25519"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -10,9 +11,84 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
agecrypt "filippo.io/age"
|
||||
"golang.org/x/crypto/ssh"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func TestABackupIsEncryptedToItsRecipients(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
etc := filepath.Join(tmp, "etc")
|
||||
_ = os.MkdirAll(filepath.Join(etc, "pki"), 0o700)
|
||||
_ = os.WriteFile(filepath.Join(etc, "pki", "ca-key.pem"), []byte("THE-CA-KEY"), 0o600)
|
||||
|
||||
id, err := agecrypt.GenerateX25519Identity()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
to, err := config.ParseBackupRecipients([]string{id.Recipient().String()})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
out := filepath.Join(tmp, "backups")
|
||||
path, err := writeBackup(t.Context(), "", etc, out, to)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.HasSuffix(path, ".tar.gz.age") {
|
||||
t.Errorf("named %s", filepath.Base(path))
|
||||
}
|
||||
raw, _ := os.ReadFile(path)
|
||||
if strings.Contains(string(raw), "THE-CA-KEY") {
|
||||
t.Fatal("the CA key is in the archive in the clear")
|
||||
}
|
||||
|
||||
f, _ := os.Open(path)
|
||||
defer f.Close()
|
||||
plain, err := agecrypt.Decrypt(f, id)
|
||||
if err != nil {
|
||||
t.Fatalf("the recipient cannot open it: %v", err)
|
||||
}
|
||||
gz, err := gzip.NewReader(plain)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tr := tar.NewReader(gz)
|
||||
found := false
|
||||
for {
|
||||
h, err := tr.Next()
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
found = found || h.Name == "etc/llmgw/pki/ca-key.pem"
|
||||
}
|
||||
if !found {
|
||||
t.Error("the opened archive lacks the CA key")
|
||||
}
|
||||
// Seen and pruned like any other.
|
||||
if NewestBackup(out).IsZero() {
|
||||
t.Error("the control plane would not see an encrypted archive")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupRecipientsMustBeKeys(t *testing.T) {
|
||||
if _, err := config.ParseBackupRecipients([]string{"not a key"}); err == nil {
|
||||
t.Error("nonsense was taken as a recipient")
|
||||
}
|
||||
pub, _, _ := ed25519.GenerateKey(nil)
|
||||
sp, err := ssh.NewPublicKey(pub)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := config.ParseBackupRecipients([]string{
|
||||
strings.TrimSpace(string(ssh.MarshalAuthorizedKey(sp))) + " ops@example",
|
||||
}); err != nil {
|
||||
t.Errorf("an SSH key was refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABackupHoldsTheStoreAndTheConfiguration(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
dbPath := filepath.Join(tmp, "store.db")
|
||||
@@ -32,7 +108,7 @@ func TestABackupHoldsTheStoreAndTheConfiguration(t *testing.T) {
|
||||
_ = os.WriteFile(filepath.Join(etc, "pki", "ca-key.pem"), []byte("KEY"), 0o600)
|
||||
|
||||
out := filepath.Join(tmp, "backups")
|
||||
path, err := writeBackup(t.Context(), dbPath, etc, out)
|
||||
path, err := writeBackup(t.Context(), dbPath, etc, out, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -129,7 +205,7 @@ func TestNoScratchCopyOfTheStoreIsLeftBehind(t *testing.T) {
|
||||
}
|
||||
_ = db.Close()
|
||||
out := filepath.Join(tmp, "backups")
|
||||
if _, err := writeBackup(t.Context(), dbPath, t.TempDir(), out); err != nil {
|
||||
if _, err := writeBackup(t.Context(), dbPath, t.TempDir(), out, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
entries, _ := os.ReadDir(out)
|
||||
|
||||
+19
-8
@@ -19,17 +19,22 @@ import (
|
||||
|
||||
// ctlClient talks to a server's admin API.
|
||||
type ctlClient struct {
|
||||
base string
|
||||
http *http.Client
|
||||
base string
|
||||
http *http.Client
|
||||
token string
|
||||
}
|
||||
|
||||
func newCtlClient(addr string) *ctlClient {
|
||||
// newCtlClient talks to the admin API at addr, with the admin token from
|
||||
// LLMGW_ADMIN_TOKEN or the first of tokenFiles it can read, then the usual
|
||||
// place.
|
||||
func newCtlClient(addr string, tokenFiles ...string) *ctlClient {
|
||||
if !strings.HasPrefix(addr, "http://") && !strings.HasPrefix(addr, "https://") {
|
||||
addr = "http://" + addr
|
||||
}
|
||||
return &ctlClient{
|
||||
base: strings.TrimRight(addr, "/"),
|
||||
http: &http.Client{Timeout: 15 * time.Second},
|
||||
base: strings.TrimRight(addr, "/"),
|
||||
http: &http.Client{Timeout: 15 * time.Second},
|
||||
token: readAdminToken(tokenFiles...),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -75,6 +80,9 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
|
||||
// Who made the change, kept with a stored model so a catalog that grew
|
||||
// over months can say where each entry came from.
|
||||
req.Header.Set("X-Llmgw-Actor", actor())
|
||||
if c.token != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+c.token)
|
||||
}
|
||||
|
||||
resp, err := c.http.Do(req)
|
||||
if err != nil {
|
||||
@@ -104,8 +112,9 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
|
||||
|
||||
func newCtlCmd() *cobra.Command {
|
||||
var (
|
||||
server string
|
||||
asJSON bool
|
||||
server string
|
||||
asJSON bool
|
||||
tokenFile string
|
||||
)
|
||||
|
||||
cmd := &cobra.Command{
|
||||
@@ -115,9 +124,11 @@ func newCtlCmd() *cobra.Command {
|
||||
"default) rather than the client-facing port.",
|
||||
}
|
||||
cmd.PersistentFlags().StringVar(&server, "server", "127.0.0.1:9090", "admin API address")
|
||||
cmd.PersistentFlags().StringVar(&tokenFile, "token-file", "",
|
||||
"the admin token's file (default: LLMGW_ADMIN_TOKEN, else "+defaultAdminTokenFile+")")
|
||||
cmd.PersistentFlags().BoolVar(&asJSON, "json", false, "emit JSON")
|
||||
|
||||
client := func() *ctlClient { return newCtlClient(server) }
|
||||
client := func() *ctlClient { return newCtlClient(server, tokenFile) }
|
||||
|
||||
cmd.AddCommand(
|
||||
newNodesCmd(client, &asJSON),
|
||||
|
||||
+27
-18
@@ -281,6 +281,13 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
|
||||
if len(clients) > 0 && !tlsc.Enabled() {
|
||||
return fmt.Errorf("--api-client names a certificate, and needs --tls-ca, --tls-cert and --tls-key")
|
||||
}
|
||||
if tlsc.Enabled() && len(clients) == 0 {
|
||||
// The same root signs every node's certificate, so without a
|
||||
// name any node — a removed one's included — could delete
|
||||
// models here.
|
||||
return fmt.Errorf("with --tls-ca, name the control plane's certificate with --api-client: " +
|
||||
"the authority signs every node's certificate too")
|
||||
}
|
||||
|
||||
out := cmd.OutOrStdout()
|
||||
log := slog.Default()
|
||||
@@ -293,20 +300,20 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
|
||||
// http.Dir refuses paths that escape the root, which is the whole
|
||||
// traversal question answered by the standard library rather than
|
||||
// by me.
|
||||
files := &http.Server{
|
||||
Addr: listen,
|
||||
Handler: readOnly(withManifest(*dir, http.FileServer(http.Dir(*dir)))),
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
}
|
||||
files := withLimits(&http.Server{
|
||||
Addr: listen,
|
||||
Handler: readOnly(withManifest(*dir, http.FileServer(http.Dir(*dir)))),
|
||||
ReadTimeout: time.Minute,
|
||||
})
|
||||
allowed, err := parseAllow(allow)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
admin := &http.Server{
|
||||
Addr: apiListen,
|
||||
Handler: onlyClients(clients, apiGate(tlsc.Enabled(), allowed, api.Handler())),
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
}
|
||||
admin := withLimits(&http.Server{
|
||||
Addr: apiListen,
|
||||
Handler: onlyClients(clients, apiGate(tlsc.Enabled(), allowed, api.Handler())),
|
||||
ReadTimeout: time.Minute,
|
||||
})
|
||||
|
||||
errs := make(chan error, 2)
|
||||
go func() { errs <- fmt.Errorf("serving weights: %w", files.ListenAndServe()) }()
|
||||
@@ -315,13 +322,7 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
|
||||
go func() { errs <- fmt.Errorf("serving the API: %w", serveAPI(admin, tlsc)) }()
|
||||
switch {
|
||||
case tlsc.Enabled():
|
||||
fmt.Fprintf(out, "management API on %s, mutual TLS against %s\n", apiListen, tlsc.CA)
|
||||
if len(clients) == 0 {
|
||||
// The same root signs every node's certificate, so without a
|
||||
// name any node could delete models here.
|
||||
log.Warn("the management API takes any certificate from " + tlsc.CA +
|
||||
", every node's included; name the control plane's with --api-client")
|
||||
}
|
||||
fmt.Fprintf(out, "management API on %s, mutual TLS for %s\n", apiListen, strings.Join(clients, ", "))
|
||||
case len(allowed) > 0:
|
||||
fmt.Fprintf(out, "management API on %s, for %s and this machine only\n", apiListen, strings.Join(allow, ", "))
|
||||
default:
|
||||
@@ -346,7 +347,7 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
|
||||
c.Flags().StringArrayVar(&allow, "api-allow", nil,
|
||||
"an address or network the management API answers, besides this machine; repeatable (default: this machine only)")
|
||||
c.Flags().StringArrayVar(&clients, "api-client", nil,
|
||||
"with mutual TLS, the common name of a certificate the management API answers; repeatable (default: any from --tls-ca)")
|
||||
"with mutual TLS, the common name of a certificate the management API answers; repeatable; required with --tls-ca")
|
||||
c.Flags().BoolVar(&allowPrivate, "allow-private-sources", false,
|
||||
"let downloads reach loopback, private and link-local addresses")
|
||||
c.Flags().StringVar(&tlsc.CA, "tls-ca", "", "CA bundle that management clients must present a certificate from")
|
||||
@@ -387,6 +388,14 @@ func readOnly(h http.Handler) http.Handler {
|
||||
http.Error(w, "the mirror is read-only", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
// The mirror's own bookkeeping — a file's checksum, the URL it came
|
||||
// from — sits beside it under a dot, and is nobody's to download.
|
||||
for _, seg := range strings.Split(r.URL.Path, "/") {
|
||||
if strings.HasPrefix(seg, ".") {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
}
|
||||
h.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
@@ -72,3 +72,24 @@ func TestAVersionAnswersHeadAsWellAsGet(t *testing.T) {
|
||||
t.Errorf("expected a manifest, got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheMirrorDoesNotServeItsBookkeeping(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
dir := filepath.Join(root, "llama", "v1")
|
||||
_ = os.MkdirAll(dir, 0o755)
|
||||
_ = os.WriteFile(filepath.Join(dir, "m.gguf"), []byte("x"), 0o644)
|
||||
_ = os.WriteFile(filepath.Join(dir, ".m.gguf.source"), []byte("https://example.com/m.gguf\n"), 0o644)
|
||||
|
||||
h := readOnly(withManifest(root, http.FileServer(http.Dir(root))))
|
||||
for path, want := range map[string]int{
|
||||
"/llama/v1/m.gguf": http.StatusOK,
|
||||
"/llama/v1/.m.gguf.source": http.StatusNotFound,
|
||||
"/llama/.hidden/x": http.StatusNotFound,
|
||||
} {
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, httptest.NewRequest(http.MethodGet, path, nil))
|
||||
if w.Code != want {
|
||||
t.Errorf("%s: %d, expected %d", path, w.Code, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/netip"
|
||||
"os"
|
||||
"runtime"
|
||||
"strings"
|
||||
@@ -13,7 +14,9 @@ import (
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
@@ -26,6 +29,7 @@ func newMirrorInitCmd(dir *string) *cobra.Command {
|
||||
var (
|
||||
controlPlanes []string
|
||||
base string
|
||||
noFirewall bool
|
||||
)
|
||||
cmd := &cobra.Command{
|
||||
Use: "init",
|
||||
@@ -56,11 +60,19 @@ Safe to run again: it changes only what differs.`,
|
||||
if _, err := parseAllow(controlPlanes); err != nil {
|
||||
return err
|
||||
}
|
||||
return mirrorInit(cmd.Context(), cmd.OutOrStdout(), *dir, base, controlPlanes)
|
||||
if err := mirrorInit(cmd.Context(), cmd.OutOrStdout(), *dir, base, controlPlanes); err != nil {
|
||||
return err
|
||||
}
|
||||
if !noFirewall {
|
||||
mirrorFirewall(cmd.Context(), cmd.OutOrStdout(), controlPlanes)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
cmd.Flags().StringArrayVar(&controlPlanes, "control-plane", nil,
|
||||
"address of the control plane that manages this mirror; repeatable")
|
||||
cmd.Flags().BoolVar(&noFirewall, "no-firewall", false,
|
||||
"leave the firewall alone, instead of keeping the mirror's ports to the cluster's networks")
|
||||
cmd.Flags().StringVar(&base, "base", "",
|
||||
"the address nodes fetch from, http://host:8081 (default: this machine's first address)")
|
||||
return cmd
|
||||
@@ -123,3 +135,34 @@ func mirrorInit(ctx context.Context, out io.Writer, dir, base string, controlPla
|
||||
fmt.Fprintf(out, "\nThe mirror is up. On the control plane:\n\n sudo llmgw server init --mirror %s\n\n", api)
|
||||
return nil
|
||||
}
|
||||
|
||||
// mirrorFirewall keeps the mirror's ports to the networks it reaches the
|
||||
// control planes on: its files are for the cluster's nodes, its API for the
|
||||
// control planes, which the API checks again by address.
|
||||
func mirrorFirewall(ctx context.Context, out io.Writer, controlPlanes []string) {
|
||||
allowed, _ := parseAllow(controlPlanes)
|
||||
var toward []string
|
||||
var nets []netip.Prefix
|
||||
for _, p := range allowed {
|
||||
if p.IsSingleIP() {
|
||||
toward = append(toward, p.Addr().String())
|
||||
} else {
|
||||
nets = append(nets, p)
|
||||
}
|
||||
}
|
||||
found, err := firewall.Networks(ctx, pgha.Local{}, toward)
|
||||
if err == nil {
|
||||
var changed bool
|
||||
changed, err = firewall.Install(ctx, pgha.Local{}, firewall.Spec{
|
||||
Ports: []firewall.Port{firewall.MirrorFiles, firewall.MirrorAPI, firewall.NodeExporter},
|
||||
Allow: append(found, nets...),
|
||||
})
|
||||
if err == nil {
|
||||
if changed {
|
||||
fmt.Fprintln(out, " ok firewall: the mirror's ports answer the cluster's networks only")
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(out, " !! firewall: %v; the mirror's ports answer every network\n", err)
|
||||
}
|
||||
+15
-2
@@ -14,6 +14,7 @@ import (
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
@@ -122,6 +123,7 @@ engines are kept.`,
|
||||
f.StringVar(&na.opts.Releases, "releases", "https://gitea.mynodes.xyz/ilker/llmgw/releases/download", "where release files are downloaded from")
|
||||
f.BoolVar(&na.opts.IgnoreChecks, "ignore-checks", false, "add the node even when doctor finds a problem")
|
||||
f.BoolVar(&na.opts.NoNodeExporter, "no-node-exporter", false, "leave Prometheus' node exporter off the machine")
|
||||
f.BoolVar(&na.opts.NoFirewall, "no-firewall", false, "leave the machine's firewall alone, instead of keeping llmgw's ports to the cluster's networks")
|
||||
f.StringArrayVar(&na.opts.DisableEngines, "disable-engine", nil, "an engine this node never installs, e.g. vllm on GPUs it does not run on; repeatable")
|
||||
f.StringVar(&na.admin, "admin", "", "the control plane's admin API, to watch the node join (default: from server.yaml, else 127.0.0.1:9090)")
|
||||
f.StringVar(&na.serverCfg, "config", "/etc/llmgw/server.yaml", "the control plane's configuration, for the admin API's address")
|
||||
@@ -168,6 +170,12 @@ func (na *nodeAdd) prepare() error {
|
||||
}
|
||||
}
|
||||
}
|
||||
// The node's ports — its engines', its metrics — for the cluster only.
|
||||
if cfg, err := config.Load(na.serverCfg); err == nil && !o.NoFirewall {
|
||||
from, to := cfg.EnginePorts()
|
||||
o.FirewallPorts = []firewall.Port{firewall.Engines(from, to), firewall.NodeExporter}
|
||||
o.FirewallAllow, _ = cfg.FirewallAllow()
|
||||
}
|
||||
o.Version = version.Version
|
||||
if o.Binary == "" {
|
||||
exe, err := os.Executable()
|
||||
@@ -301,16 +309,21 @@ func (na *nodeAdd) runPlain(ctx context.Context, o onboard.Options, title string
|
||||
// this machine has it.
|
||||
func (na *nodeAdd) adminAPI() *ctlClient {
|
||||
addr := na.admin
|
||||
token := ""
|
||||
cfg, err := config.Load(na.serverCfg)
|
||||
if err == nil {
|
||||
token = adminTokenPath(na.serverCfg, cfg)
|
||||
}
|
||||
if addr == "" {
|
||||
addr = "127.0.0.1:9090"
|
||||
if cfg, err := config.Load(na.serverCfg); err == nil && cfg.AdminListen != "" {
|
||||
if err == nil && cfg.AdminListen != "" {
|
||||
addr = cfg.AdminListen
|
||||
if strings.HasPrefix(addr, ":") {
|
||||
addr = "127.0.0.1" + addr
|
||||
}
|
||||
}
|
||||
}
|
||||
return newCtlClient(addr)
|
||||
return newCtlClient(addr, token)
|
||||
}
|
||||
|
||||
// errNoAdmin is the watch giving up because the admin API is not here, which
|
||||
|
||||
+87
-13
@@ -16,6 +16,7 @@ import (
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/clientip"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
@@ -26,6 +27,7 @@ import (
|
||||
"github.com/spf13/cobra"
|
||||
"golang.org/x/net/http2"
|
||||
"golang.org/x/net/http2/h2c"
|
||||
"golang.org/x/net/netutil"
|
||||
"golang.org/x/sync/errgroup"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
@@ -194,6 +196,12 @@ func newServerCmd() *cobra.Command {
|
||||
// Patroni. One host is a single PostgreSQL, which has none.
|
||||
if addr, err := storepkg.ResolveAddress(cfg.Auth.Store); err == nil && storepkg.IsPostgres(addr) {
|
||||
if hosts, _, _, err := storepkg.SplitPostgres(strings.TrimSpace(addr)); err == nil && len(hosts) > 1 {
|
||||
if cfg.AgentTLS.CA != "" {
|
||||
// Members' APIs, once on TLS, verified against it.
|
||||
if err := pgha.SetRESTCA(cfg.AgentTLS.CA); err != nil {
|
||||
log.Warn("the store's API cannot be verified", "err", err)
|
||||
}
|
||||
}
|
||||
storeHA = pgha.NewCollector(hosts)
|
||||
if err := rt.RegisterMetrics(storeHA); err != nil {
|
||||
return err
|
||||
@@ -234,6 +242,10 @@ func newServerCmd() *cobra.Command {
|
||||
}
|
||||
adminAPI = api
|
||||
api.SetReleases(cfg.Releases)
|
||||
if st := rt.Store(); st != nil {
|
||||
// Removing a node revokes its certificate.
|
||||
api.SetRevoker(st.RevokeNode)
|
||||
}
|
||||
// Catalog changes need somewhere to persist, so this is wired
|
||||
// only when there is a store. Without one, the endpoints say so
|
||||
// rather than accepting a model that vanishes on restart.
|
||||
@@ -348,21 +360,39 @@ func newServerCmd() *cobra.Command {
|
||||
"api", cfg.Mirror.API, "mtls", cfg.Mirror.TLS.Enabled())
|
||||
}
|
||||
|
||||
apiSrv := &http.Server{
|
||||
Handler: rt.Handler(),
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
apiSrv := withLimits(&http.Server{
|
||||
Handler: rt.Handler(),
|
||||
// Reading the request: long enough for a body up to
|
||||
// max_body_bytes on a slow link, short enough that a client
|
||||
// trickling one cannot hold a connection and its buffer.
|
||||
ReadTimeout: 5 * time.Minute,
|
||||
// No write timeout: responses stream for minutes and the proxy
|
||||
// enforces its own per-phase deadlines.
|
||||
})
|
||||
if err := checkAdminListen(cfg); err != nil {
|
||||
return err
|
||||
}
|
||||
adminSrv := &http.Server{
|
||||
Handler: router.AdminGuard(adminMux),
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
if bad := clientip.SetTrusted(cfg.Router.TrustedProxies); len(bad) > 0 {
|
||||
return fmt.Errorf("router.trusted_proxies: %q is not an address or a CIDR prefix", bad[0])
|
||||
}
|
||||
adminToken, err := ensureAdminToken(adminTokenPath(configPath, cfg))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
adminSrv := withLimits(&http.Server{
|
||||
Handler: router.AdminGuard(router.AdminAuth(adminToken, cfg.DebugPprof, adminMux)),
|
||||
ReadTimeout: time.Minute,
|
||||
})
|
||||
|
||||
apiLn, err := net.Listen("tcp", cfg.Listen)
|
||||
if err != nil {
|
||||
return fmt.Errorf("listening on %s: %w", cfg.Listen, err)
|
||||
}
|
||||
if cfg.MaxConnections > 0 {
|
||||
// Past this, new connections wait in the kernel's backlog
|
||||
// rather than each taking a goroutine and its buffers.
|
||||
apiLn = netutil.LimitListener(apiLn, cfg.MaxConnections)
|
||||
}
|
||||
adminLn, err := net.Listen("tcp", cfg.AdminListen)
|
||||
if err != nil {
|
||||
_ = apiLn.Close()
|
||||
@@ -395,7 +425,7 @@ func newServerCmd() *cobra.Command {
|
||||
mux.Handle("/", h)
|
||||
h = mux
|
||||
}
|
||||
metricsSrv = &http.Server{Handler: h, ReadHeaderTimeout: 10 * time.Second}
|
||||
metricsSrv = withLimits(&http.Server{Handler: h, ReadTimeout: 30 * time.Second})
|
||||
go func() {
|
||||
if err := metricsSrv.Serve(metricsLn); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
||||
log.Error("metrics listener stopped", "err", err)
|
||||
@@ -416,10 +446,9 @@ func newServerCmd() *cobra.Command {
|
||||
if agentHandler != nil {
|
||||
mux := http.NewServeMux()
|
||||
mux.Handle(agentPath, agentHandler)
|
||||
agentSrv = &http.Server{
|
||||
Handler: mux,
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
}
|
||||
// No read timeout: an agent's stream is one request that
|
||||
// lasts as long as the agent does.
|
||||
agentSrv = withLimits(&http.Server{Handler: mux})
|
||||
if !cfg.AgentTLS.Enabled() {
|
||||
// Without TLS, agents speak plaintext HTTP/2, which Go will
|
||||
// not do unless asked. Nothing then stops a node from
|
||||
@@ -427,10 +456,20 @@ func newServerCmd() *cobra.Command {
|
||||
// network only.
|
||||
agentSrv.Handler = h2c.NewHandler(mux, &http2.Server{})
|
||||
}
|
||||
if agentLn, err = net.Listen("tcp", cfg.AgentListen); err != nil {
|
||||
agentAddr, aerr := agentListenAddr(cfg)
|
||||
if aerr != nil {
|
||||
_ = apiLn.Close()
|
||||
_ = adminLn.Close()
|
||||
return fmt.Errorf("listening on %s: %w", cfg.AgentListen, err)
|
||||
return aerr
|
||||
}
|
||||
if !cfg.AgentTLS.Enabled() {
|
||||
log.Warn("the agent port has no TLS (agent_insecure): agents are not authenticated and "+
|
||||
"traffic is plaintext; use only on one machine or a trusted network", "listen", agentAddr)
|
||||
}
|
||||
if agentLn, err = net.Listen("tcp", agentAddr); err != nil {
|
||||
_ = apiLn.Close()
|
||||
_ = adminLn.Close()
|
||||
return fmt.Errorf("listening on %s: %w", agentAddr, err)
|
||||
}
|
||||
if cfg.AgentTLS.Enabled() {
|
||||
tc, terr := cfg.AgentTLS.ServerConfig()
|
||||
@@ -586,6 +625,8 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
|
||||
return urls, err
|
||||
})
|
||||
go svc.WatchControlPlanes(ctx, 30*time.Second)
|
||||
// A removed node's certificate is refused.
|
||||
svc.SetRevocations(st.NodeRevokedAt)
|
||||
}
|
||||
fwd.set(svc)
|
||||
// Every InfiniBand fabric mapped every five minutes, through one node.
|
||||
@@ -761,3 +802,36 @@ func servableCheck(llamaVersion string) func(context.Context, catalog.Model) err
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// errAgentInsecure is a control plane asked to open its agent port with no
|
||||
// TLS and not told that is meant.
|
||||
var errAgentInsecure = errors.New("agent_tls is not set, so the agent port would be plaintext and anyone " +
|
||||
"reaching it could register as a node: set agent_tls (llmgw server init writes it), or " +
|
||||
"agent_insecure: true for one machine or a test")
|
||||
|
||||
// agentListenAddr is where the agent port listens: agent_listen with TLS;
|
||||
// without it, refused unless agent_insecure says so, and then on loopback
|
||||
// when agent_listen names only a port.
|
||||
func agentListenAddr(cfg config.Server) (string, error) {
|
||||
if cfg.AgentTLS.Enabled() {
|
||||
return cfg.AgentListen, nil
|
||||
}
|
||||
// agent_tls.insecure says the same, in the older place.
|
||||
if !cfg.AgentInsecure && !cfg.AgentTLS.Insecure {
|
||||
return "", errAgentInsecure
|
||||
}
|
||||
if host, port, err := net.SplitHostPort(cfg.AgentListen); err == nil && host == "" {
|
||||
return net.JoinHostPort("127.0.0.1", port), nil
|
||||
}
|
||||
return cfg.AgentListen, nil
|
||||
}
|
||||
|
||||
// withLimits gives a server what every listener here wants: a bound on how
|
||||
// long the headers may take, how long a connection may sit idle between
|
||||
// requests, and how large the headers may be.
|
||||
func withLimits(s *http.Server) *http.Server {
|
||||
s.ReadHeaderTimeout = 10 * time.Second
|
||||
s.IdleTimeout = 2 * time.Minute
|
||||
s.MaxHeaderBytes = 1 << 20
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
func TestTheAgentPortOpensWithoutTLSOnlyWhenToldAndThenOnLoopback(t *testing.T) {
|
||||
cfg := config.Default()
|
||||
if _, err := agentListenAddr(cfg); err == nil {
|
||||
t.Error("a control plane with no agent_tls opened its agent port")
|
||||
}
|
||||
cfg.AgentInsecure = true
|
||||
if a, _ := agentListenAddr(cfg); a != "127.0.0.1:7443" {
|
||||
t.Errorf("insecure, port only: %s", a)
|
||||
}
|
||||
cfg.AgentListen = "10.0.0.5:7443"
|
||||
if a, _ := agentListenAddr(cfg); a != "10.0.0.5:7443" {
|
||||
t.Errorf("insecure, an address named: %s", a)
|
||||
}
|
||||
cfg.AgentInsecure = false
|
||||
cfg.AgentListen = ":7443"
|
||||
cfg.AgentTLS = tlsconf.Config{Insecure: true}
|
||||
if a, err := agentListenAddr(cfg); err != nil || a != "127.0.0.1:7443" {
|
||||
t.Errorf("agent_tls.insecure: %s %v", a, err)
|
||||
}
|
||||
cfg.AgentTLS = tlsconf.Config{CA: "ca.pem", Cert: "s.pem", Key: "s-key.pem"}
|
||||
if a, err := agentListenAddr(cfg); err != nil || a != ":7443" {
|
||||
t.Errorf("with TLS: %s %v", a, err)
|
||||
}
|
||||
}
|
||||
@@ -20,6 +20,7 @@ import (
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
@@ -38,6 +39,8 @@ type serverAdd struct {
|
||||
vip string
|
||||
|
||||
remote onboard.Remote
|
||||
// noFirewall leaves the new control plane's firewall as it is.
|
||||
noFirewall bool
|
||||
// addr is the machine's address as given; ip what the others reach it at.
|
||||
addr, ip, name string
|
||||
out io.Writer
|
||||
@@ -78,6 +81,8 @@ func newServerAddCmd() *cobra.Command {
|
||||
f.StringArrayVar(&sa.hosts, "host", nil, "another name for the new control plane's certificate; repeatable")
|
||||
f.StringVar(&sa.vip, "vip", "", "a floating address for whichever control plane leads, as 10.0.0.160/24")
|
||||
f.StringVar(&sa.pkiDir, "pki", sa.pkiDir, "directory holding the certificate authority")
|
||||
f.BoolVar(&sa.noFirewall, "no-firewall", false,
|
||||
"leave its firewall alone, instead of keeping llmgw's internal ports to the cluster's networks")
|
||||
return cmd
|
||||
}
|
||||
|
||||
@@ -121,6 +126,12 @@ func (sa *serverAdd) run(ctx context.Context) error {
|
||||
{"store", sa.storeAccess},
|
||||
{"service", sa.service},
|
||||
}
|
||||
if !sa.noFirewall {
|
||||
steps = append(steps, struct {
|
||||
name string
|
||||
fn func(context.Context) error
|
||||
}{"firewall", sa.firewall})
|
||||
}
|
||||
if sa.vip != "" {
|
||||
steps = append(steps, struct {
|
||||
name string
|
||||
@@ -136,6 +147,36 @@ func (sa *serverAdd) run(ctx context.Context) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// firewall keeps the new control plane's internal ports to the networks it
|
||||
// reaches this one on. Not fatal: it serves without one.
|
||||
func (sa *serverAdd) firewall(ctx context.Context) error {
|
||||
cfg, err := config.Load(filepath.Join(sa.etcDir, "server.yaml"))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
self, err := sourceIPToward(sa.ip)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
nets, err := firewall.Networks(ctx, sa.remote, []string{self})
|
||||
if err == nil {
|
||||
extra, _ := cfg.FirewallAllow()
|
||||
nets = append(nets, extra...)
|
||||
var changed bool
|
||||
if changed, err = firewall.Install(ctx, sa.remote,
|
||||
firewall.Spec{Ports: controlPlanePorts(cfg), Allow: nets}); err == nil {
|
||||
if changed {
|
||||
sa.done("firewall: llmgw's internal ports answer the cluster's networks only")
|
||||
} else {
|
||||
sa.kept("firewall")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(sa.out, " !! firewall: %v\n", err)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (sa *serverAdd) connect(ctx context.Context) error {
|
||||
out, err := sa.sh(ctx, "hostname; sudo -n true && echo sudo-ok", nil)
|
||||
if err != nil {
|
||||
@@ -307,7 +348,12 @@ func (sa *serverAdd) storeAccess(ctx context.Context) error {
|
||||
case slices.Contains(hosts, sa.ip):
|
||||
sa.kept("store: %s is one of its members", sa.ip)
|
||||
case len(hosts) > 1:
|
||||
changed, err := pgha.AllowClient(ctx, hosts, sa.ip)
|
||||
rest := ""
|
||||
if b, err := os.ReadFile(patroniRestFile(sa.etcDir)); err == nil { //nolint:gosec // our own secret
|
||||
rest = strings.TrimSpace(string(b))
|
||||
}
|
||||
_ = pgha.SetRESTCA(filepath.Join(sa.etcDir, "ca.pem"))
|
||||
changed, err := pgha.AllowClient(ctx, hosts, sa.ip, rest)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -81,6 +81,9 @@ type serverInit struct {
|
||||
// make it serve something: a mirror, this machine as a node, and a small
|
||||
// model on it.
|
||||
noMirror, noNode, noModel bool
|
||||
// noFirewall leaves this machine's firewall, and the store members',
|
||||
// as they are.
|
||||
noFirewall bool
|
||||
// mirrorURL is a mirror on another machine to use instead of running one.
|
||||
mirrorURL string
|
||||
// pgSetup prepares PostgreSQL and returns the store address; a seam for
|
||||
@@ -161,6 +164,8 @@ func newServerInitCmd() *cobra.Command {
|
||||
"use the mirror on another machine, set up there by 'llmgw mirror init': its management API, http://host:8082")
|
||||
f.BoolVar(&in.noNode, "no-node", false, "do not make this machine a node (and serve no model on it)")
|
||||
f.BoolVar(&in.noModel, "no-model", false, "do not fetch and serve the starter model ("+starterName+")")
|
||||
f.BoolVar(&in.noFirewall, "no-firewall", false,
|
||||
"leave the firewall alone, instead of keeping llmgw's internal ports to the cluster's networks")
|
||||
f.BoolVar(&in.postgres, "postgres", false,
|
||||
"keep the store in PostgreSQL on this machine, installing it and moving an existing SQLite store into it")
|
||||
f.StringArrayVar(&in.patroni, "patroni", nil,
|
||||
@@ -221,6 +226,12 @@ func (in *serverInit) run(ctx context.Context) error {
|
||||
return fmt.Errorf("starter model: %w", err)
|
||||
}
|
||||
}
|
||||
if hasSystemd && !in.noFirewall {
|
||||
if err := in.ensureFirewall(ctx); err != nil {
|
||||
// The cluster works without it; said, not fatal.
|
||||
fmt.Fprintf(in.out, " !! firewall: %v; llmgw's ports answer every network this machine is on\n", err)
|
||||
}
|
||||
}
|
||||
in.nextSteps()
|
||||
return nil
|
||||
}
|
||||
@@ -480,7 +491,9 @@ control:
|
||||
auth:
|
||||
enabled: true
|
||||
store: "%s" # a SQLite path, or @file holding a PostgreSQL address
|
||||
signup_open: true
|
||||
# The signup form: accounts it makes wait for an administrator's approval,
|
||||
# but anyone reaching port 8080 can make them. Off until you want it.
|
||||
signup_open: false
|
||||
# Applied to any key no scope gives a limit to. Without them, a project
|
||||
# created without limits is metered by nothing at all.
|
||||
default_rpm: 60
|
||||
|
||||
@@ -552,11 +552,14 @@ func (in *serverInit) waitForStarter(ctx context.Context) error {
|
||||
}
|
||||
|
||||
func (in *serverInit) adminAPI() *ctlClient {
|
||||
addr := "127.0.0.1:9090"
|
||||
if cfg, err := config.Load(in.configPath()); err == nil && cfg.AdminListen != "" {
|
||||
addr = cfg.AdminListen
|
||||
addr, token := "127.0.0.1:9090", ""
|
||||
if cfg, err := config.Load(in.configPath()); err == nil {
|
||||
token = adminTokenPath(in.configPath(), cfg)
|
||||
if cfg.AdminListen != "" {
|
||||
addr = cfg.AdminListen
|
||||
}
|
||||
}
|
||||
return newCtlClient(addr)
|
||||
return newCtlClient(addr, token)
|
||||
}
|
||||
|
||||
// waitFor polls until done says so, fails, or the time runs out.
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/netip"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
|
||||
)
|
||||
|
||||
// ensureFirewall keeps this control plane's internal ports to the networks
|
||||
// the cluster is on: the agents' stream, the metrics, the mirror's files,
|
||||
// the engines and etcd when they are here. The client API is not among
|
||||
// them; it is meant to be reached.
|
||||
func (in *serverInit) ensureFirewall(ctx context.Context) error {
|
||||
cfg, err := config.Load(in.configPath())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ports := controlPlanePorts(cfg)
|
||||
if !in.noMirror && in.mirrorURL == "" {
|
||||
ports = append(ports, firewall.MirrorFiles)
|
||||
}
|
||||
if !in.noNode {
|
||||
from, to := cfg.EnginePorts()
|
||||
ports = append(ports, firewall.Engines(from, to), firewall.NodeExporter)
|
||||
}
|
||||
if _, err := os.Stat("/etc/default/etcd"); err == nil {
|
||||
// The store's third etcd vote.
|
||||
ports = append(ports, firewall.EtcdClient, firewall.EtcdPeer)
|
||||
}
|
||||
self, err := ipv4Of(in.reachableAddr())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
nets, err := firewall.Networks(ctx, pgha.Local{}, []string{self})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
extra, _ := cfg.FirewallAllow()
|
||||
nets = append(nets, extra...)
|
||||
changed, err := firewall.Install(ctx, pgha.Local{}, firewall.Spec{Ports: ports, Allow: nets})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
shown := make([]string, len(nets))
|
||||
for i, n := range nets {
|
||||
shown[i] = n.String()
|
||||
}
|
||||
if changed {
|
||||
in.done("firewall: llmgw's internal ports answer %s only (%s)", strings.Join(shown, ", "), firewall.RulesFile)
|
||||
} else {
|
||||
in.kept("firewall: llmgw's internal ports answer %s only", strings.Join(shown, ", "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// storeFirewall keeps each store member's database, Patroni and etcd ports to
|
||||
// the networks it reaches this control plane and the other members on. A
|
||||
// member it cannot be set on is said, and left as it was.
|
||||
func (in *serverInit) storeFirewall(ctx context.Context, members []pgha.Node, self string, extra []netip.Prefix) {
|
||||
ports := []firewall.Port{firewall.Postgres, firewall.PatroniAPI, firewall.EtcdClient, firewall.EtcdPeer,
|
||||
firewall.NodeExporter}
|
||||
for _, m := range members {
|
||||
toward := []string{self}
|
||||
for _, o := range members {
|
||||
if o.IP != m.IP {
|
||||
toward = append(toward, o.IP)
|
||||
}
|
||||
}
|
||||
nets, err := firewall.Networks(ctx, m.Remote, toward)
|
||||
if err == nil {
|
||||
nets = append(nets, extra...)
|
||||
var changed bool
|
||||
if changed, err = firewall.Install(ctx, m.Remote, firewall.Spec{Ports: ports, Allow: nets}); err == nil {
|
||||
if changed {
|
||||
in.done("firewall on %s: the store's ports answer the cluster's networks only", m.Name)
|
||||
} else {
|
||||
in.kept("firewall on %s", m.Name)
|
||||
}
|
||||
continue
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(in.out, " !! firewall on %s: %v\n", m.Name, err)
|
||||
}
|
||||
}
|
||||
|
||||
// controlPlanePorts are a control plane's internal ports, from its
|
||||
// configuration: the agents' stream, and the metrics when they listen
|
||||
// beyond this machine.
|
||||
func controlPlanePorts(cfg config.Server) []firewall.Port {
|
||||
var ports []firewall.Port
|
||||
if p := portOf(cfg.AgentListen); p > 0 {
|
||||
ports = append(ports, firewall.Single(p, "control plane: the agents' stream"))
|
||||
}
|
||||
if p := portOf(cfg.MetricsListen); p > 0 && !loopbackListen(cfg.MetricsListen) {
|
||||
ports = append(ports, firewall.Single(p, "control plane: metrics, for Prometheus"))
|
||||
}
|
||||
return ports
|
||||
}
|
||||
|
||||
func portOf(listen string) int {
|
||||
_, port, err := net.SplitHostPort(listen)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
p, _ := strconv.Atoi(port)
|
||||
return p
|
||||
}
|
||||
|
||||
func loopbackListen(listen string) bool {
|
||||
host, _, err := net.SplitHostPort(listen)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
ip := net.ParseIP(host)
|
||||
return host == "localhost" || (ip != nil && ip.IsLoopback())
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
@@ -58,15 +59,32 @@ func (in *serverInit) ensurePatroni(ctx context.Context) (err error) {
|
||||
setup.Witness = &pgha.Node{Member: pgha.Member{Name: name, IP: self}, Remote: pgha.Local{}}
|
||||
setup.Clients = []string{self}
|
||||
|
||||
cfg, err := config.Load(in.configPath())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// The cluster's authority signs each member's certificate, and the
|
||||
// control plane verifies the member it reaches against it: where its
|
||||
// key is on this machine, and the server has the certificate to read.
|
||||
setup.CAFile = cfg.AgentTLS.CA
|
||||
if ca, cerr := loadCA(in.pkiDir); cerr == nil && cfg.AgentTLS.CA != "" {
|
||||
setup.CA = ca
|
||||
} else {
|
||||
in.kept("store: the authority's key is not here, so no member certificates are signed from this machine; " +
|
||||
"run this on the control plane that holds it to turn TLS on")
|
||||
}
|
||||
|
||||
dsn, err := setup.Run(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
cfg, err := config.Load(in.configPath())
|
||||
if err != nil {
|
||||
if err := in.writeSecret(patroniRestFile(in.etcDir), setup.RestPassword); err != nil {
|
||||
return err
|
||||
}
|
||||
if !in.noFirewall {
|
||||
extra, _ := cfg.FirewallAllow()
|
||||
in.storeFirewall(ctx, setup.DB, self, extra)
|
||||
}
|
||||
current := cfg.Auth.Store
|
||||
if current == "" {
|
||||
current = in.storeAddress()
|
||||
@@ -180,3 +198,21 @@ func ipv4Of(host string) (string, error) {
|
||||
}
|
||||
return "", fmt.Errorf("%s has no IPv4 address", host)
|
||||
}
|
||||
|
||||
// patroniRestFile holds what Patroni's API asks for before a change, on the
|
||||
// control plane: for `server add`, which lets a second one in.
|
||||
func patroniRestFile(etcDir string) string { return filepath.Join(etcDir, "secrets", "patroni-rest") }
|
||||
|
||||
// writeSecret writes a secret root alone reads, unless it is there already.
|
||||
func (in *serverInit) writeSecret(path, value string) error {
|
||||
if value == "" {
|
||||
return nil
|
||||
}
|
||||
if b, err := os.ReadFile(path); err == nil && strings.TrimSpace(string(b)) == value { //nolint:gosec // our own secret
|
||||
return nil
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o750); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.WriteFile(path, []byte(value+"\n"), 0o600)
|
||||
}
|
||||
@@ -389,6 +389,15 @@ func (in *serverInit) writeDSN(dsn string) error {
|
||||
if err := in.chown(dir, false); err != nil {
|
||||
return err
|
||||
}
|
||||
// Root's, and the service's group may enter it. Its mode set too, not
|
||||
// only on creation: a directory made 0700 for the service user (as
|
||||
// `server add` made them) and handed to root here was one the service
|
||||
// could no longer open, and the gateway did not start.
|
||||
if in.user != "" {
|
||||
if err := os.Chmod(dir, 0o750); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := os.WriteFile(in.dsnPath(), []byte(dsn+"\n"), 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -1370,10 +1370,18 @@ func filepathBase(p string) string {
|
||||
// file is server.yaml on a control plane, agent.yaml on a node.
|
||||
func runConsole(ctx context.Context, out io.Writer, file string) error {
|
||||
addr := "127.0.0.1:9090"
|
||||
if cfg, err := config.Load(file); err == nil && cfg.AdminListen != "" {
|
||||
addr = cfg.AdminListen
|
||||
if strings.HasPrefix(addr, ":") {
|
||||
addr = "127.0.0.1" + addr
|
||||
if cfg, err := config.Load(file); err == nil {
|
||||
if cfg.AdminListen != "" {
|
||||
addr = cfg.AdminListen
|
||||
if strings.HasPrefix(addr, ":") {
|
||||
addr = "127.0.0.1" + addr
|
||||
}
|
||||
}
|
||||
// The admin token, for this screen and every command it runs in
|
||||
// this process — `llmgw ctl` included — from this configuration's
|
||||
// file rather than only the usual place.
|
||||
if t := readAdminToken(adminTokenPath(file, cfg)); t != "" {
|
||||
_ = os.Setenv("LLMGW_ADMIN_TOKEN", t)
|
||||
}
|
||||
}
|
||||
menu := cOverview
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/mail"
|
||||
"os"
|
||||
"strings"
|
||||
@@ -448,8 +449,11 @@ func (m *setupModel) startJoins() (tea.Cmd, bool) {
|
||||
continue
|
||||
}
|
||||
if m.listener == nil {
|
||||
l, err := joinpkg.Listen(fmt.Sprintf(":%d", joinpkg.DefaultPort),
|
||||
fmt.Sprintf("%s:%d", m.thisAddr(), joinpkg.DefaultPort), time.Hour)
|
||||
// On the address the codes name, not every interface: a code is
|
||||
// a credential, and the port it is spent on need not face more
|
||||
// than the network the machines join from.
|
||||
addr := net.JoinHostPort(m.thisAddr(), fmt.Sprint(joinpkg.DefaultPort))
|
||||
l, err := joinpkg.Listen(addr, addr, time.Hour)
|
||||
if err != nil {
|
||||
m.formErr = err.Error()
|
||||
return nil, false
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
// Package clientip says who sent a request and how, believing a proxy's
|
||||
// X-Forwarded-* headers only when the request came from a proxy the
|
||||
// configuration names.
|
||||
//
|
||||
// Believed from anyone, X-Forwarded-Host decides the address written into the
|
||||
// setup scripts clients pipe to a shell, and X-Forwarded-For lets one client
|
||||
// claim to be any number of others to whatever counts attempts per address.
|
||||
// Believed from nobody, every client behind a proxy is one address, and a
|
||||
// limit per address is a limit on all of them together.
|
||||
package clientip
|
||||
|
||||
import (
|
||||
"net"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
)
|
||||
|
||||
var trusted atomic.Pointer[[]netip.Prefix]
|
||||
|
||||
// SetTrusted names the proxies whose forwarding headers are believed:
|
||||
// addresses or CIDR prefixes. Anything unparsable is skipped and returned.
|
||||
func SetTrusted(list []string) (skipped []string) {
|
||||
var out []netip.Prefix
|
||||
for _, s := range list {
|
||||
s = strings.TrimSpace(s)
|
||||
if p, err := netip.ParsePrefix(s); err == nil {
|
||||
out = append(out, p.Masked())
|
||||
continue
|
||||
}
|
||||
if a, err := netip.ParseAddr(s); err == nil {
|
||||
out = append(out, netip.PrefixFrom(a, a.BitLen()))
|
||||
continue
|
||||
}
|
||||
skipped = append(skipped, s)
|
||||
}
|
||||
trusted.Store(&out)
|
||||
return skipped
|
||||
}
|
||||
|
||||
func isTrusted(addr string) bool {
|
||||
a, err := netip.ParseAddr(addr)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
a = a.Unmap()
|
||||
list := trusted.Load()
|
||||
if list == nil {
|
||||
return false
|
||||
}
|
||||
for _, p := range *list {
|
||||
if p.Contains(a) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// peer is the address the connection came from.
|
||||
func peer(r *http.Request) string {
|
||||
host, _, err := net.SplitHostPort(r.RemoteAddr)
|
||||
if err != nil {
|
||||
return r.RemoteAddr
|
||||
}
|
||||
return host
|
||||
}
|
||||
|
||||
// fromProxy reports whether the request came through a trusted proxy.
|
||||
func fromProxy(r *http.Request) bool { return isTrusted(peer(r)) }
|
||||
|
||||
// Addr is the client's address: the connection's, or through trusted
|
||||
// proxies the nearest address in X-Forwarded-For that is not one of them.
|
||||
func Addr(r *http.Request) string {
|
||||
p := peer(r)
|
||||
if !isTrusted(p) {
|
||||
return p
|
||||
}
|
||||
hops := strings.Split(r.Header.Get("X-Forwarded-For"), ",")
|
||||
for i := len(hops) - 1; i >= 0; i-- {
|
||||
h := strings.TrimSpace(hops[i])
|
||||
if h == "" {
|
||||
continue
|
||||
}
|
||||
if !isTrusted(h) {
|
||||
return h
|
||||
}
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// Proto is "https" when the client spoke TLS: to this server, or to a
|
||||
// trusted proxy that says so.
|
||||
func Proto(r *http.Request) string {
|
||||
if r.TLS != nil {
|
||||
return "https"
|
||||
}
|
||||
if fromProxy(r) {
|
||||
if v := first(r.Header.Get("X-Forwarded-Proto")); v != "" {
|
||||
return strings.ToLower(v)
|
||||
}
|
||||
}
|
||||
return "http"
|
||||
}
|
||||
|
||||
// Host is the host the client asked for: Host, or what a trusted proxy says
|
||||
// it was asked for.
|
||||
func Host(r *http.Request) string {
|
||||
if fromProxy(r) {
|
||||
if v := first(r.Header.Get("X-Forwarded-Host")); v != "" {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return r.Host
|
||||
}
|
||||
|
||||
func first(v string) string {
|
||||
if i := strings.IndexByte(v, ','); i >= 0 {
|
||||
v = v[:i]
|
||||
}
|
||||
return strings.TrimSpace(v)
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
package clientip
|
||||
|
||||
import (
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestForwardingHeadersAreBelievedOnlyFromTrustedProxies(t *testing.T) {
|
||||
r := httptest.NewRequest("GET", "/setup/x", nil)
|
||||
r.RemoteAddr = "203.0.113.9:4000"
|
||||
r.Header.Set("X-Forwarded-For", "10.9.9.9")
|
||||
r.Header.Set("X-Forwarded-Host", "evil.example")
|
||||
r.Header.Set("X-Forwarded-Proto", "https")
|
||||
r.Host = "gw.example:8080"
|
||||
|
||||
SetTrusted(nil)
|
||||
if Addr(r) != "203.0.113.9" || Host(r) != "gw.example:8080" || Proto(r) != "http" {
|
||||
t.Errorf("believed an untrusted client: %s %s %s", Addr(r), Host(r), Proto(r))
|
||||
}
|
||||
|
||||
if bad := SetTrusted([]string{"203.0.113.0/24", "10.0.0.5", "nonsense"}); len(bad) != 1 {
|
||||
t.Errorf("skipped %v", bad)
|
||||
}
|
||||
r.Header.Set("X-Forwarded-For", "198.51.100.7, 10.0.0.5")
|
||||
if Addr(r) != "198.51.100.7" || Host(r) != "evil.example" || Proto(r) != "https" {
|
||||
t.Errorf("through trusted proxies: %s %s %s", Addr(r), Host(r), Proto(r))
|
||||
}
|
||||
// A client prepending addresses gains nothing: the nearest untrusted one
|
||||
// is the one the proxy saw.
|
||||
r.Header.Set("X-Forwarded-For", "1.2.3.4, 198.51.100.7")
|
||||
if Addr(r) != "198.51.100.7" {
|
||||
t.Errorf("spoofed: %s", Addr(r))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"filippo.io/age"
|
||||
"filippo.io/age/agessh"
|
||||
)
|
||||
|
||||
// ParseBackupRecipients reads backup_recipients: age recipients, or SSH
|
||||
// public keys.
|
||||
func ParseBackupRecipients(list []string) ([]age.Recipient, error) {
|
||||
var out []age.Recipient
|
||||
for i, s := range list {
|
||||
s = strings.TrimSpace(s)
|
||||
if strings.HasPrefix(s, "ssh-") {
|
||||
r, err := agessh.ParseRecipient(s)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("backup_recipients[%d]: %w", i, err)
|
||||
}
|
||||
out = append(out, r)
|
||||
continue
|
||||
}
|
||||
rs, err := age.ParseRecipients(strings.NewReader(s))
|
||||
if err != nil || len(rs) != 1 {
|
||||
return nil, fmt.Errorf("backup_recipients[%d] is not an age recipient (age1…) or an SSH public key", i)
|
||||
}
|
||||
out = append(out, rs[0])
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -65,6 +65,29 @@ type Server struct {
|
||||
// cover the client-facing one: that surface is fronted by a publicly
|
||||
// trusted certificate, while this one is mutual and private.
|
||||
AgentTLS tlsconf.Config `koanf:"agent_tls"`
|
||||
// MaxConnections bounds the client port's open connections; past it new
|
||||
// ones wait in the kernel's backlog. 0 is no bound.
|
||||
MaxConnections int `koanf:"max_connections"`
|
||||
// AdminTokenFile holds the token the admin API's /api and /debug paths
|
||||
// demand. Empty is admin-token in the systemd unit's state directory
|
||||
// (/var/lib/llmgw), or secrets/admin-token beside this file for a server
|
||||
// run by hand. Made, 0600, when the server starts and it does not exist;
|
||||
// `sudo llmgw ctl` and the console read it from there.
|
||||
AdminTokenFile string `koanf:"admin_token_file"`
|
||||
// AdminRemote allows admin_listen on an address other than loopback. The
|
||||
// token still guards the API, but it then crosses the network in clear.
|
||||
AdminRemote bool `koanf:"admin_remote"`
|
||||
// DebugPprof serves Go's profiler on the admin listener, behind the token.
|
||||
// Off by default: a heap profile holds the API keys requests presented.
|
||||
DebugPprof bool `koanf:"debug_pprof"`
|
||||
// AgentInsecure lets a control plane run its agent port without
|
||||
// agent_tls: plaintext, and any machine that reaches it may register as
|
||||
// any node — be given replicas, and so be sent users' prompts — or, in
|
||||
// the other direction, a machine in the path may tell every agent what
|
||||
// to run. Refused unless said here, and then the port listens on
|
||||
// loopback unless agent_listen names an address. For one machine, a
|
||||
// test, or a network nothing untrusted reaches.
|
||||
AgentInsecure bool `koanf:"agent_insecure"`
|
||||
|
||||
// Monitoring is where the dashboard reads its history from.
|
||||
Monitoring Monitoring `koanf:"monitoring"`
|
||||
@@ -98,6 +121,20 @@ type Server struct {
|
||||
// Empty means /var/backups/llmgw-server.
|
||||
BackupDir string `koanf:"backup_dir"`
|
||||
|
||||
// FirewallNetworks are networks, besides those each machine reaches the
|
||||
// cluster on, that llmgw's internal ports answer: a monitoring host on
|
||||
// another subnet, nodes behind a router. Written into every machine's
|
||||
// firewall by `server init`, `node add` and `server add`.
|
||||
FirewallNetworks []string `koanf:"firewall_networks"`
|
||||
|
||||
// BackupRecipients encrypt every archive `llmgw server backup` writes, to
|
||||
// whoever holds one of their keys: age recipients (age1…) or SSH public
|
||||
// keys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds
|
||||
// the certificate authority's key and every secret in the store, and is
|
||||
// copied off the machine; encrypted, the copy is safe wherever it lands.
|
||||
// Empty writes archives unencrypted, readable by root only.
|
||||
BackupRecipients []string `koanf:"backup_recipients"`
|
||||
|
||||
// Releases is where `llmgw ctl upgrade` fetches release binaries from,
|
||||
// for this machine and — through their agents — every node. Empty means
|
||||
// the project's own releases.
|
||||
@@ -386,6 +423,12 @@ type Router struct {
|
||||
// MaxBodyBytes bounds the request body the router will read.
|
||||
MaxBodyBytes int64 `koanf:"max_body_bytes"`
|
||||
|
||||
// TrustedProxies are the reverse proxies in front of the client port,
|
||||
// as addresses or CIDR prefixes. Their X-Forwarded-For, -Proto and -Host
|
||||
// are believed; anyone else's are not, since they decide the address
|
||||
// written into the setup scripts and whose attempts are counted where.
|
||||
TrustedProxies []string `koanf:"trusted_proxies"`
|
||||
|
||||
Timeouts Timeouts `koanf:"timeouts"`
|
||||
Scoring Scoring `koanf:"scoring"`
|
||||
Admission Admission `koanf:"admission"`
|
||||
@@ -483,10 +526,11 @@ type Endpoint struct {
|
||||
// Default returns the configuration used when nothing is specified.
|
||||
func Default() Server {
|
||||
return Server{
|
||||
Listen: ":8080",
|
||||
AdminListen: "127.0.0.1:9090",
|
||||
AgentListen: ":7443",
|
||||
Mirror: Mirror{UpdateCheck: 6 * time.Hour},
|
||||
Listen: ":8080",
|
||||
AdminListen: "127.0.0.1:9090",
|
||||
AgentListen: ":7443",
|
||||
MaxConnections: 10_000,
|
||||
Mirror: Mirror{UpdateCheck: 6 * time.Hour},
|
||||
Control: Control{
|
||||
Interval: 5 * time.Second,
|
||||
BaseBackoff: 10 * time.Second,
|
||||
@@ -589,9 +633,16 @@ func (s Server) Validate() error {
|
||||
if s.Listen == "" {
|
||||
return fmt.Errorf("listen is empty")
|
||||
}
|
||||
if _, err := ParseBackupRecipients(s.BackupRecipients); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := s.FirewallAllow(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.AgentTLS.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if s.Auth.Enabled && s.Auth.Store == "" {
|
||||
return fmt.Errorf("auth.store is required when auth is enabled")
|
||||
}
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/netip"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// FirewallAllow reads firewall_networks: networks, or single addresses.
|
||||
func (s Server) FirewallAllow() ([]netip.Prefix, error) {
|
||||
var out []netip.Prefix
|
||||
for i, n := range s.FirewallNetworks {
|
||||
n = strings.TrimSpace(n)
|
||||
if p, err := netip.ParsePrefix(n); err == nil {
|
||||
out = append(out, p.Masked())
|
||||
continue
|
||||
}
|
||||
a, err := netip.ParseAddr(n)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("firewall_networks[%d] is %q: a network (10.0.1.0/24) or an address", i, n)
|
||||
}
|
||||
out = append(out, netip.PrefixFrom(a, a.BitLen()))
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// EnginePorts is the range engines listen in on every node.
|
||||
func (s Server) EnginePorts() (int, int) {
|
||||
if len(s.Control.PortRange) == 2 {
|
||||
return s.Control.PortRange[0], s.Control.PortRange[1]
|
||||
}
|
||||
d := Default().Control.PortRange
|
||||
return d[0], d[1]
|
||||
}
|
||||
@@ -24,6 +24,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/safedial"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
|
||||
@@ -47,6 +48,8 @@ type API struct {
|
||||
serve *modelserve.Server
|
||||
// history is the Prometheus the console's charts read from.
|
||||
history History
|
||||
// revoke records a removed node, so its certificate is refused.
|
||||
revoke func(ctx context.Context, nodeID string, at time.Time) error
|
||||
|
||||
models Models
|
||||
capacity Capacity
|
||||
@@ -58,6 +61,11 @@ type API struct {
|
||||
// New builds the API over a controller.
|
||||
func New(ctl *control.Controller) *API { return &API{ctl: ctl} }
|
||||
|
||||
// SetRevoker has removing a node revoke its certificate.
|
||||
func (a *API) SetRevoker(f func(ctx context.Context, nodeID string, at time.Time) error) {
|
||||
a.revoke = f
|
||||
}
|
||||
|
||||
// Register mounts the endpoints on a mux under /api/v1.
|
||||
func (a *API) Register(mux *http.ServeMux) {
|
||||
mux.HandleFunc("GET /api/v1/summary", a.summary)
|
||||
@@ -1098,6 +1106,14 @@ func (a *API) removeNode(w http.ResponseWriter, r *http.Request) {
|
||||
"%s is %s: stop its agent (systemctl stop llmgw-agent) and remove it once it is lost, a minute later", id, n.State))
|
||||
return
|
||||
}
|
||||
// Revoked first: a node removed but still holding a certificate the
|
||||
// agent port accepts could connect again and be given work.
|
||||
if a.revoke != nil {
|
||||
if err := a.revoke(r.Context(), id, time.Now()); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "revoking the node's certificate: "+err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
inv.Remove(id)
|
||||
a.ctl.Reconciler().Notify()
|
||||
writeJSON(w, http.StatusOK, map[string]string{"id": id, "removed": "true"})
|
||||
@@ -1700,7 +1716,7 @@ func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, http.StatusBadRequest, "no model "+name)
|
||||
return
|
||||
}
|
||||
base, baseErr := gguf.KindFromURL(r.Context(), nil, m.Source)
|
||||
base, baseErr := gguf.KindFromURL(r.Context(), safedial.Client(safedial.ControlPlane, 30*time.Second), m.Source)
|
||||
have := map[string]bool{}
|
||||
for _, ad := range m.Adapters {
|
||||
have[ad.Source] = true
|
||||
@@ -1709,7 +1725,7 @@ func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
|
||||
if have[ad.Source] {
|
||||
continue // already checked when it was attached
|
||||
}
|
||||
h, err := gguf.KindFromURL(r.Context(), nil, ad.Source)
|
||||
h, err := gguf.KindFromURL(r.Context(), safedial.Client(safedial.ControlPlane, 30*time.Second), ad.Source)
|
||||
switch {
|
||||
case err != nil:
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %v", ad.Name, err))
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -707,3 +708,30 @@ func TestOnlyANodeThatIsGoneCanBeRemoved(t *testing.T) {
|
||||
t.Errorf("removing it again gave %d, expected 404", code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRemovingANodeRevokesItsCertificate(t *testing.T) {
|
||||
a, ctl := setup(t, testModels(), gpuNode("n1", 8), gpuNode("n2", 8))
|
||||
var revoked []string
|
||||
a.SetRevoker(func(_ context.Context, id string, _ time.Time) error {
|
||||
revoked = append(revoked, id)
|
||||
return nil
|
||||
})
|
||||
ctl.Inventory().SetState("n1", inventory.Lost)
|
||||
if code := call(t, a, "DELETE", "/api/v1/nodes/n1", "", nil); code != 200 {
|
||||
t.Fatalf("a lost node gave %d, expected 200", code)
|
||||
}
|
||||
if !slices.Equal(revoked, []string{"n1"}) {
|
||||
t.Errorf("revoked %v, expected n1", revoked)
|
||||
}
|
||||
|
||||
// Not revoked, not removed: a node left listed is one the operator can
|
||||
// see and try again on.
|
||||
a.SetRevoker(func(context.Context, string, time.Time) error { return errors.New("store down") })
|
||||
ctl.Inventory().SetState("n2", inventory.Lost)
|
||||
if code := call(t, a, "DELETE", "/api/v1/nodes/n2", "", nil); code != 500 {
|
||||
t.Errorf("a failed revocation gave %d, expected 500", code)
|
||||
}
|
||||
if _, ok := ctl.Inventory().Get("n2"); !ok {
|
||||
t.Error("the node was removed though its certificate was not revoked")
|
||||
}
|
||||
}
|
||||
@@ -316,3 +316,12 @@ func TestAnotherTenantOnACardIsJudgedEveryHeartbeat(t *testing.T) {
|
||||
t.Errorf("a heartbeat with no foreign reading cleared the card: %q", n.GPUs[0].Health.Problems)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAProblemOnANodeWithoutGPUsDoesNotPanic(t *testing.T) {
|
||||
s := newTestService()
|
||||
nodeWith(s, inventory.Node{ID: "cpu1", State: inventory.Ready})
|
||||
s.applyHeartbeat("cpu1", &llmgwv1.Heartbeat{NodeId: "cpu1", NewProblems: []string{"disk read errors"}})
|
||||
if _, ok := s.inv.Get("cpu1"); !ok {
|
||||
t.Fatal("the node is gone")
|
||||
}
|
||||
}
|
||||
@@ -4,6 +4,9 @@ import (
|
||||
"context"
|
||||
"crypto/x509"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
// peerKey carries the verified client identity through to the stream handler.
|
||||
@@ -21,6 +24,9 @@ type PeerIdentity struct {
|
||||
// Verified is false when the connection carried no client certificate,
|
||||
// which only happens when TLS is switched off.
|
||||
Verified bool
|
||||
// IssuedAt is when the certificate was issued, for revocation: one issued
|
||||
// for a node before it was removed is refused.
|
||||
IssuedAt time.Time
|
||||
}
|
||||
|
||||
// WithPeerIdentity records a verified identity on the context.
|
||||
@@ -53,5 +59,5 @@ func AuthMiddleware(next http.Handler) http.Handler {
|
||||
}
|
||||
|
||||
func identityOf(cert *x509.Certificate) PeerIdentity {
|
||||
return PeerIdentity{CommonName: cert.Subject.CommonName, Verified: true}
|
||||
return PeerIdentity{CommonName: cert.Subject.CommonName, Verified: true, IssuedAt: tlsconf.IssuedAt(cert)}
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
package agentsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"connectrpc.com/connect"
|
||||
)
|
||||
|
||||
func TestARemovedNodesCertificateIsRefused(t *testing.T) {
|
||||
removed := time.Date(2026, 10, 4, 12, 0, 0, 0, time.UTC)
|
||||
s := newTestService()
|
||||
s.requireIdentity = true
|
||||
s.SetRevocations(func(_ context.Context, id string) (time.Time, error) {
|
||||
if id == "gpu01" {
|
||||
return removed, nil
|
||||
}
|
||||
return time.Time{}, nil
|
||||
})
|
||||
peer := func(cn string, issued time.Time) context.Context {
|
||||
return WithPeerIdentity(context.Background(), PeerIdentity{CommonName: cn, Verified: true, IssuedAt: issued})
|
||||
}
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
ctx context.Context
|
||||
node string
|
||||
refuse connect.Code
|
||||
}{
|
||||
{"issued before the removal", peer("gpu01", removed.Add(-time.Minute)), "gpu01", connect.CodePermissionDenied},
|
||||
{"issued at the removal", peer("gpu01", removed), "gpu01", connect.CodePermissionDenied},
|
||||
{"issued after, by onboarding again", peer("gpu01", removed.Add(time.Second)), "gpu01", 0},
|
||||
{"another node", peer("gpu02", removed.Add(-time.Hour)), "gpu02", 0},
|
||||
{"another node's name", peer("gpu02", removed.Add(time.Hour)), "gpu01", connect.CodePermissionDenied},
|
||||
{"no certificate", context.Background(), "gpu02", connect.CodeUnauthenticated},
|
||||
}
|
||||
for _, c := range cases {
|
||||
err := s.admit(c.ctx, c.node)
|
||||
if c.refuse == 0 {
|
||||
if err != nil {
|
||||
t.Errorf("%s: refused: %v", c.name, err)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if connect.CodeOf(err) != c.refuse {
|
||||
t.Errorf("%s: got %v, expected %v", c.name, err, c.refuse)
|
||||
}
|
||||
}
|
||||
|
||||
// Not knowing is not "not revoked".
|
||||
s.SetRevocations(func(context.Context, string) (time.Time, error) { return time.Time{}, errors.New("store down") })
|
||||
if err := s.admit(peer("gpu02", removed), "gpu02"); connect.CodeOf(err) != connect.CodeUnavailable {
|
||||
t.Errorf("with the store down: %v, expected unavailable", err)
|
||||
}
|
||||
}
|
||||
@@ -49,6 +49,9 @@ type Service struct {
|
||||
// requireIdentity refuses agents that present no certificate. It is off
|
||||
// only for the insecure development path.
|
||||
requireIdentity bool
|
||||
// revokedAt is when a node was removed, whose certificates issued before
|
||||
// then are refused; nil checks nothing.
|
||||
revokedAt func(ctx context.Context, nodeID string) (time.Time, error)
|
||||
|
||||
mu sync.RWMutex
|
||||
conns map[string]*conn
|
||||
@@ -137,12 +140,55 @@ func New(inv *inventory.Store, status StatusSink, requireIdentity bool, log *slo
|
||||
}
|
||||
}
|
||||
|
||||
// admit decides whether the peer on ctx may be nodeID.
|
||||
func (s *Service) admit(ctx context.Context, nodeID string) error {
|
||||
// The certificate decides who this is, not the message.
|
||||
//
|
||||
// Without this check an agent could register as any node in the fleet and
|
||||
// receive that node's shards — which is the same exposure the stream had
|
||||
// before TLS, just with encryption on top. mTLS is only worth having if the
|
||||
// identity it proves is the one actually used.
|
||||
if peer := peerFrom(ctx); peer.Verified {
|
||||
if peer.CommonName != nodeID {
|
||||
s.log.Warn("rejecting an agent claiming a node id its certificate does not authorize",
|
||||
"certificate_cn", peer.CommonName, "claimed", nodeID)
|
||||
return connect.NewError(connect.CodePermissionDenied,
|
||||
fmt.Errorf("certificate authorizes node %q, not %q", peer.CommonName, nodeID))
|
||||
}
|
||||
if s.revokedAt != nil {
|
||||
at, err := s.revokedAt(ctx, nodeID)
|
||||
if err != nil {
|
||||
// Unknown is not "not revoked": the agent retries.
|
||||
return connect.NewError(connect.CodeUnavailable, fmt.Errorf("checking revocation: %w", err))
|
||||
}
|
||||
if !at.IsZero() && !peer.IssuedAt.After(at) {
|
||||
s.log.Warn("rejecting a removed node's certificate", "node", nodeID,
|
||||
"issued", peer.IssuedAt, "removed", at)
|
||||
return connect.NewError(connect.CodePermissionDenied, fmt.Errorf(
|
||||
"node %q was removed from the cluster, and its certificate with it; onboard it again", nodeID))
|
||||
}
|
||||
}
|
||||
} else if s.requireIdentity {
|
||||
return connect.NewError(connect.CodeUnauthenticated,
|
||||
errors.New("no verified client certificate"))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SetRevocations has agents refused whose certificate was issued before
|
||||
// their node was removed.
|
||||
func (s *Service) SetRevocations(f func(ctx context.Context, nodeID string) (time.Time, error)) {
|
||||
s.revokedAt = f
|
||||
}
|
||||
|
||||
// Handler returns the HTTP path and handler to mount.
|
||||
//
|
||||
// The handler is wrapped so the verified client identity reaches the stream:
|
||||
// connect exposes the peer's address but not its certificate.
|
||||
func (s *Service) Handler() (string, http.Handler) {
|
||||
path, h := llmgwv1connect.NewAgentServiceHandler(s)
|
||||
// A message is a heartbeat, a report or a log; nothing an agent says
|
||||
// needs more, and a bound keeps one connection from taking the memory.
|
||||
path, h := llmgwv1connect.NewAgentServiceHandler(s, connect.WithReadMaxBytes(16<<20))
|
||||
return path, AuthMiddleware(h)
|
||||
}
|
||||
|
||||
@@ -180,22 +226,8 @@ func (s *Service) Connect(
|
||||
return connect.NewError(connect.CodeInvalidArgument, errors.New("node_id is required"))
|
||||
}
|
||||
|
||||
// The certificate decides who this is, not the message.
|
||||
//
|
||||
// Without this check an agent could register as any node in the fleet and
|
||||
// receive that node's shards — which is the same exposure the stream had
|
||||
// before TLS, just with encryption on top. mTLS is only worth having if the
|
||||
// identity it proves is the one actually used.
|
||||
if peer := peerFrom(ctx); peer.Verified {
|
||||
if peer.CommonName != nodeID {
|
||||
s.log.Warn("rejecting an agent claiming a node id its certificate does not authorize",
|
||||
"certificate_cn", peer.CommonName, "claimed", nodeID)
|
||||
return connect.NewError(connect.CodePermissionDenied,
|
||||
fmt.Errorf("certificate authorizes node %q, not %q", peer.CommonName, nodeID))
|
||||
}
|
||||
} else if s.requireIdentity {
|
||||
return connect.NewError(connect.CodeUnauthenticated,
|
||||
errors.New("no verified client certificate"))
|
||||
if err := s.admit(ctx, nodeID); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
node := NodeFromRegister(reg)
|
||||
@@ -422,7 +454,7 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
|
||||
n.GPUs[i].Health.Problems = ps
|
||||
}
|
||||
}
|
||||
if len(problems) > 0 {
|
||||
if len(problems) > 0 && len(n.GPUs) > 0 {
|
||||
// Problems are reported per node, not per GPU, so attach them to the
|
||||
// node's first GPU rather than guessing which one is affected. Any
|
||||
// problem is enough to keep that GPU out of placement, which is the
|
||||
|
||||
@@ -440,6 +440,9 @@ func (m Model) validate() error {
|
||||
m.Name, p.Name, k, why)
|
||||
}
|
||||
}
|
||||
if err := engines.UnsafeArgs(p.Engine, p.Args); err != nil {
|
||||
return fmt.Errorf("model %q profile %q: %w", m.Name, p.Name, err)
|
||||
}
|
||||
|
||||
if p.GPUClass == "" && !cpuCapable(p.Engine) {
|
||||
return fmt.Errorf("model %q profile %q: gpu_class is required for engine %q",
|
||||
|
||||
@@ -385,3 +385,21 @@ func TestARepositoryPageIsReadAsTheRepositoryForVLLM(t *testing.T) {
|
||||
t.Errorf("a file under /resolve/ is a source: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAModelThatWouldRunCodeOnItsNodesIsRefusedWhereItIsDefined(t *testing.T) {
|
||||
for _, p := range []Profile{
|
||||
{Name: "p", Engine: "vllm", GPUClass: "h100-80g", TP: 1, Args: []string{"--trust-remote-code"}},
|
||||
{Name: "p", Engine: "llamacpp", MinMemoryGB: 4, Args: []string{"--log-file", "/etc/cron.d/x"}},
|
||||
{Name: "p", Engine: "vllm", GPUClass: "h100-80g", TP: 1, Env: map[string]string{"NCCL_NET_PLUGIN": "/tmp/x.so"}},
|
||||
{Name: "p", Engine: "vllm", GPUClass: "h100-80g", TP: 1, Env: map[string]string{"CUDA_INJECTION64_PATH": "/tmp/x.so"}},
|
||||
} {
|
||||
if _, err := New([]Model{model("m", p)}); err == nil {
|
||||
t.Errorf("accepted %v %v", p.Args, p.Env)
|
||||
}
|
||||
}
|
||||
ok := Profile{Name: "p", Engine: "llamacpp", MinMemoryGB: 4, Args: []string{"--ctx-size", "16384"},
|
||||
Env: map[string]string{"OMP_NUM_THREADS": "8"}}
|
||||
if _, err := New([]Model{model("m", ok)}); err != nil {
|
||||
t.Errorf("a tuned model was refused: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -234,13 +234,37 @@ func (s *Server) Start(req Request) (Job, error) {
|
||||
s.seq++
|
||||
j := &Job{ID: fmt.Sprintf("serve-%d-%d", time.Now().Unix(), s.seq), Request: req,
|
||||
State: Running, Started: time.Now()}
|
||||
s.jobs[j.ID] = j
|
||||
s.keepJob(j)
|
||||
s.mu.Unlock()
|
||||
go s.run(j)
|
||||
return s.snapshot(j), nil
|
||||
}
|
||||
|
||||
// Jobs is every job since the control plane started, the newest first.
|
||||
// keptJobs is how many finished jobs are remembered: enough for the console's
|
||||
// list, and not one more for every model ever served.
|
||||
const keptJobs = 100
|
||||
|
||||
// keepJob records a job, and forgets the oldest finished ones beyond
|
||||
// keptJobs. Called with s.mu held.
|
||||
func (s *Server) keepJob(j *Job) {
|
||||
s.jobs[j.ID] = j
|
||||
var done []*Job
|
||||
for _, o := range s.jobs {
|
||||
if o.State != Running {
|
||||
done = append(done, o)
|
||||
}
|
||||
}
|
||||
if len(done) <= keptJobs {
|
||||
return
|
||||
}
|
||||
slices.SortFunc(done, func(a, b *Job) int { return a.Started.Compare(b.Started) })
|
||||
for _, o := range done[:len(done)-keptJobs] {
|
||||
delete(s.jobs, o.ID)
|
||||
}
|
||||
}
|
||||
|
||||
// Jobs is the jobs since the control plane started — every one running, and
|
||||
// the last keptJobs finished — the newest first.
|
||||
func (s *Server) Jobs() []Job {
|
||||
s.mu.Lock()
|
||||
all := make([]*Job, 0, len(s.jobs))
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/binary"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"slices"
|
||||
@@ -375,3 +376,24 @@ func TestAModelThatFitsBusyCardsIsAddedToWaitNotRefused(t *testing.T) {
|
||||
t.Errorf("%+v", last)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyTheLastFinishedJobsAreRemembered(t *testing.T) {
|
||||
s := New(Deps{})
|
||||
start := time.Now().Add(-time.Hour)
|
||||
s.mu.Lock()
|
||||
running := &Job{ID: "running", State: Running, Started: start}
|
||||
s.keepJob(running)
|
||||
for i := range keptJobs + 5 {
|
||||
s.keepJob(&Job{ID: fmt.Sprintf("done-%d", i), State: "done", Started: start.Add(time.Duration(i+1) * time.Second)})
|
||||
}
|
||||
s.mu.Unlock()
|
||||
if n := len(s.Jobs()); n != keptJobs+1 {
|
||||
t.Errorf("%d jobs remembered, expected %d finished and the running one", n, keptJobs)
|
||||
}
|
||||
if _, ok := s.Get("running"); !ok {
|
||||
t.Error("the running job, the oldest, was forgotten")
|
||||
}
|
||||
if _, ok := s.Get("done-0"); ok {
|
||||
t.Error("the oldest finished job is still remembered")
|
||||
}
|
||||
}
|
||||
@@ -436,7 +436,7 @@ func (s *Server) StartUpdate(name, by string) (Job, error) {
|
||||
s.seq++
|
||||
j := &Job{ID: fmt.Sprintf("update-%d-%d", time.Now().Unix(), s.seq), Kind: "update", Model: name,
|
||||
Request: Request{Name: name, By: by}, State: Running, Started: time.Now()}
|
||||
s.jobs[j.ID] = j
|
||||
s.keepJob(j)
|
||||
s.mu.Unlock()
|
||||
go s.runUpdate(j)
|
||||
return s.snapshot(j), nil
|
||||
|
||||
@@ -182,6 +182,7 @@ var Reference = []Block{
|
||||
{Key: "inject_usage", Type: "bool", Doc: "InjectUsage asks engines for usage even when the client did not, so token\naccounting works regardless of client behaviour.", Nested: ""},
|
||||
{Key: "tenant_label", Type: "string", Doc: "TenantLabel is the level of the tree the per-tenant request series are\nlabelled by — llmgw_tenant_requests_total and its siblings: \"org\",\n\"team\" or \"project\", or \"none\" for no such series. A label per project\non a cluster of thousands is more series than Prometheus wants, and a\nteam is what an alert is usually about; \"team\" is the default.", Nested: ""},
|
||||
{Key: "max_body_bytes", Type: "int64", Doc: "MaxBodyBytes bounds the request body the router will read.", Nested: ""},
|
||||
{Key: "trusted_proxies", Type: "[]string", Doc: "TrustedProxies are the reverse proxies in front of the client port,\nas addresses or CIDR prefixes. Their X-Forwarded-For, -Proto and -Host\nare believed; anyone else's are not, since they decide the address\nwritten into the setup scripts and whose attempts are counted where.", Nested: ""},
|
||||
{Key: "timeouts", Type: "Timeouts", Doc: "", Nested: "config.Timeouts"},
|
||||
{Key: "scoring", Type: "Scoring", Doc: "", Nested: "config.Scoring"},
|
||||
{Key: "admission", Type: "Admission", Doc: "", Nested: "config.Admission"},
|
||||
@@ -213,12 +214,19 @@ var Reference = []Block{
|
||||
{Key: "threads", Type: "Threads", Doc: "Threads keeps conversations and agent memory without API keys, for a\ngateway on one trusted machine. With auth on they live in the auth store,\nper project, and this block is ignored.", Nested: "config.Threads"},
|
||||
{Key: "landing", Type: "Landing", Doc: "Landing configures the page served at \"/\".", Nested: "config.Landing"},
|
||||
{Key: "agent_tls", Type: "tlsconf.Config", Doc: "AgentTLS authenticates the fleet-facing port. It deliberately does not\ncover the client-facing one: that surface is fronted by a publicly\ntrusted certificate, while this one is mutual and private.", Nested: "tlsconf.Config"},
|
||||
{Key: "max_connections", Type: "int", Doc: "MaxConnections bounds the client port's open connections; past it new\nones wait in the kernel's backlog. 0 is no bound.", Nested: ""},
|
||||
{Key: "admin_token_file", Type: "string", Doc: "AdminTokenFile holds the token the admin API's /api and /debug paths\ndemand. Empty is admin-token in the systemd unit's state directory\n(/var/lib/llmgw), or secrets/admin-token beside this file for a server\nrun by hand. Made, 0600, when the server starts and it does not exist;\n`sudo llmgw ctl` and the console read it from there.", Nested: ""},
|
||||
{Key: "admin_remote", Type: "bool", Doc: "AdminRemote allows admin_listen on an address other than loopback. The\ntoken still guards the API, but it then crosses the network in clear.", Nested: ""},
|
||||
{Key: "debug_pprof", Type: "bool", Doc: "DebugPprof serves Go's profiler on the admin listener, behind the token.\nOff by default: a heap profile holds the API keys requests presented.", Nested: ""},
|
||||
{Key: "agent_insecure", Type: "bool", Doc: "AgentInsecure lets a control plane run its agent port without\nagent_tls: plaintext, and any machine that reaches it may register as\nany node — be given replicas, and so be sent users' prompts — or, in\nthe other direction, a machine in the path may tell every agent what\nto run. Refused unless said here, and then the port listens on\nloopback unless agent_listen names an address. For one machine, a\ntest, or a network nothing untrusted reaches.", Nested: ""},
|
||||
{Key: "monitoring", Type: "Monitoring", Doc: "Monitoring is where the dashboard reads its history from.", Nested: "config.Monitoring"},
|
||||
{Key: "tracing", Type: "Tracing", Doc: "Tracing sends a trace of each inference request to an OpenTelemetry\ncollector. Off unless an endpoint is set here or in\nOTEL_EXPORTER_OTLP_ENDPOINT.", Nested: "config.Tracing"},
|
||||
{Key: "mirror", Type: "Mirror", Doc: "Mirror points the control plane at a model mirror it can administer.\nEmpty means there is none, and /admin/mirror stays unmounted.", Nested: "config.Mirror"},
|
||||
{Key: "engines", Type: "map[string]string", Doc: "Engines are the engines every node installs, and at which version. The\nagents install them themselves — the right build for their hardware,\nchecked against its publisher's checksum and seen to run — so a node is\nnever trusted to already have what somebody once put there.\n\nUnset keeps DefaultEngines. A version replaces the default; an empty\none leaves that engine to be installed by hand.", Nested: ""},
|
||||
{Key: "engine_overrides", Type: "[]EngineOverride", Doc: "EngineOverrides give some nodes different engines from the rest: a\nvLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at\nall on nodes meant only for llama.cpp. Each picks nodes and lays its\nengines over the cluster's, in order, later ones last; an empty version\nremoves that engine from the nodes it picks.", Nested: ""},
|
||||
{Key: "backup_dir", Type: "string", Doc: "BackupDir is where `llmgw server backup` writes, and where the newest\narchive's age is read for llmgw_backup_last_success_timestamp_seconds.\nEmpty means /var/backups/llmgw-server.", Nested: ""},
|
||||
{Key: "firewall_networks", Type: "[]string", Doc: "FirewallNetworks are networks, besides those each machine reaches the\ncluster on, that llmgw's internal ports answer: a monitoring host on\nanother subnet, nodes behind a router. Written into every machine's\nfirewall by `server init`, `node add` and `server add`.", Nested: ""},
|
||||
{Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""},
|
||||
{Key: "releases", Type: "string", Doc: "Releases is where `llmgw ctl upgrade` fetches release binaries from,\nfor this machine and — through their agents — every node. Empty means\nthe project's own releases.", Nested: ""},
|
||||
{Key: "router", Type: "Router", Doc: "", Nested: "config.Router"},
|
||||
{Key: "control", Type: "Control", Doc: "", Nested: "config.Control"},
|
||||
@@ -310,6 +318,7 @@ var Reference = []Block{
|
||||
{Key: "health_interval", Type: "time.Duration", Doc: "HealthInterval is how often a starting shard is polled.", Nested: ""},
|
||||
{Key: "rendezvous_timeout", Type: "time.Duration", Doc: "RendezvousTimeout bounds the wait for a multi-node rendezvous point to\nstart listening.", Nested: ""},
|
||||
{Key: "alive_grace", Type: "time.Duration", Doc: "AliveGrace is how long a non-serving rank must stay up before it counts\nas ready. There is no external signal for a worker, so staying alive is\nthe only evidence available.", Nested: ""},
|
||||
{Key: "run_as", Type: "string", Doc: "RunAs is the user engines run as when the agent is root; empty is\nllmgw-engine, \"root\" keeps them root.", Nested: ""},
|
||||
}},
|
||||
{Package: "tlsconf", Name: "Config", Title: "tlsconf.Config", Doc: "", Fields: []Field{
|
||||
{Key: "ca", Type: "string", Doc: "CA is the bundle used to verify the other side. In a real deployment this\nis whichever authority issued the fleet's certificates.", Nested: ""},
|
||||
|
||||
@@ -93,6 +93,10 @@ type Facts struct {
|
||||
EnginesDir string `json:"engines_dir,omitempty"`
|
||||
EnginesFreeGB float64 `json:"engines_free_gb,omitempty"`
|
||||
VLLMInstalled bool `json:"vllm_installed"`
|
||||
|
||||
// Firewall is whether llmgw's nftables table is loaded: "present",
|
||||
// "absent", or "" when it could not be read (not root).
|
||||
Firewall string `json:"firewall,omitempty"`
|
||||
}
|
||||
|
||||
// Modules checked, and why each matters.
|
||||
@@ -224,9 +228,22 @@ func Collect(ctx context.Context, o Options) Facts {
|
||||
f.Engines[name] = firstLine(string(out))
|
||||
}
|
||||
}
|
||||
f.Firewall = firewallState(ctx)
|
||||
return f
|
||||
}
|
||||
|
||||
// firewallState reads whether llmgw's table is loaded.
|
||||
func firewallState(ctx context.Context) string {
|
||||
out, err := exec.CommandContext(ctx, "nft", "list", "table", "inet", "llmgw").CombinedOutput()
|
||||
switch {
|
||||
case err == nil:
|
||||
return "present"
|
||||
case strings.Contains(string(out), "No such file or directory"):
|
||||
return "absent"
|
||||
}
|
||||
return "" // no nft, or not root: unknown
|
||||
}
|
||||
|
||||
func (f *Facts) readConfig(o Options) {
|
||||
path := o.ConfigPath
|
||||
if path == "" {
|
||||
@@ -573,6 +590,18 @@ func Evaluate(f Facts) []Finding {
|
||||
"LimitMEMLOCK=infinity in the unit")
|
||||
}
|
||||
|
||||
// --- firewall ---
|
||||
// Only on a node already set up: before that, onboarding writes it.
|
||||
if f.ConfigPath != "" {
|
||||
switch f.Firewall {
|
||||
case "present":
|
||||
add(OK, "firewall", "llmgw's ports answer the cluster's networks only", "")
|
||||
case "absent":
|
||||
add(Warn, "firewall", "no llmgw firewall: the engines' ports answer every network this machine is on",
|
||||
"run node add for it again, or list extra networks in firewall_networks first")
|
||||
}
|
||||
}
|
||||
|
||||
sort.SliceStable(out, func(i, j int) bool { return rank(out[i].Level) < rank(out[j].Level) })
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -248,3 +248,14 @@ func TestAnAdapterNoFabricUsesDoesNotKeepTheNodeOut(t *testing.T) {
|
||||
has(t, fs, Fail, "mlx5_2/1 is RoCE with no RoCE v2 GID entry")
|
||||
has(t, fs, Fail, "ib_uverbs is not loaded")
|
||||
}
|
||||
|
||||
func TestAMissingFirewallIsSaidOnASetUpNode(t *testing.T) {
|
||||
f := Facts{ConfigPath: "/etc/llmgw/agent.yaml", Firewall: "absent"}
|
||||
has(t, Evaluate(f), Warn, "no llmgw firewall")
|
||||
f.ConfigPath = "" // before onboarding has written one
|
||||
for _, x := range Evaluate(f) {
|
||||
if x.Area == "firewall" {
|
||||
t.Errorf("said before the node is set up: %+v", x)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,319 @@
|
||||
// Package firewall writes and applies the nftables rules that keep llmgw's
|
||||
// internal ports to the cluster's own networks.
|
||||
//
|
||||
// Only llmgw's ports, in a table of its own. The machines llmgw is put on
|
||||
// are often not llmgw's alone — a GPU node shared with a desktop's ComfyUI,
|
||||
// a database host with other databases — so the policy is not "deny
|
||||
// everything else": it is "these ports answer the cluster's networks and
|
||||
// this machine, and nobody else". Whatever else the machine serves, and
|
||||
// whatever firewall it already has, is left as it was. The client API
|
||||
// (:8080) is not among them: it is meant to be reached.
|
||||
package firewall
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/netip"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Runner runs a shell script on a machine, as onboarding's SSH and the
|
||||
// local machine both do.
|
||||
type Runner interface {
|
||||
Run(ctx context.Context, script string, stdin []byte) ([]byte, error)
|
||||
}
|
||||
|
||||
// Port is one port, or a range, and what answers there.
|
||||
type Port struct {
|
||||
From, To int
|
||||
What string
|
||||
}
|
||||
|
||||
// Single is one port.
|
||||
func Single(port int, what string) Port { return Port{From: port, To: port, What: what} }
|
||||
|
||||
func (p Port) nft() string {
|
||||
if p.To > p.From {
|
||||
return strconv.Itoa(p.From) + "-" + strconv.Itoa(p.To)
|
||||
}
|
||||
return strconv.Itoa(p.From)
|
||||
}
|
||||
|
||||
// The ports of each part of a cluster.
|
||||
var (
|
||||
AgentPort = Single(7443, "control plane: the agents' stream")
|
||||
NodeExporter = Single(9100, "node exporter: the machine's metrics, for Prometheus")
|
||||
MirrorFiles = Single(8081, "mirror: weights, for the nodes")
|
||||
MirrorAPI = Single(8082, "mirror: its management API, for the control planes")
|
||||
Postgres = Single(5432, "store: PostgreSQL")
|
||||
PatroniAPI = Single(8008, "store: Patroni's API")
|
||||
EtcdClient = Single(2379, "store: etcd, for Patroni")
|
||||
EtcdPeer = Single(2380, "store: etcd, between its members")
|
||||
)
|
||||
|
||||
// Engines is the engines' port range on a node: their HTTP, for the
|
||||
// router, and their rendezvous, for the other nodes of a replica.
|
||||
func Engines(from, to int) Port {
|
||||
return Port{From: from, To: to, What: "engines: HTTP and rendezvous"}
|
||||
}
|
||||
|
||||
// Spec is what a machine's ruleset holds: its llmgw ports, and the
|
||||
// networks they answer besides the machine itself.
|
||||
type Spec struct {
|
||||
Ports []Port
|
||||
Allow []netip.Prefix
|
||||
}
|
||||
|
||||
// Table is the nftables table llmgw's rules live in, and nothing else does.
|
||||
const Table = "llmgw"
|
||||
|
||||
// RulesFile is where the ruleset is kept, for the unit that loads it at boot.
|
||||
const RulesFile = "/etc/llmgw/firewall.nft"
|
||||
|
||||
// Unit is the systemd unit that loads it.
|
||||
const Unit = "llmgw-firewall.service"
|
||||
|
||||
// Script is the ruleset as nft reads it. It replaces llmgw's table whole,
|
||||
// so applying it again, or a changed one, leaves no stale rule behind.
|
||||
func Script(s Spec) string {
|
||||
ports := slices.Clone(s.Ports)
|
||||
slices.SortFunc(ports, func(a, b Port) int { return a.From - b.From })
|
||||
ports = slices.CompactFunc(ports, func(a, b Port) bool { return a.From == b.From && a.To == b.To })
|
||||
var v4, v6 []string
|
||||
for _, p := range s.Allow {
|
||||
p = p.Masked()
|
||||
if p.Addr().Is4() {
|
||||
v4 = append(v4, p.String())
|
||||
} else {
|
||||
v6 = append(v6, p.String())
|
||||
}
|
||||
}
|
||||
slices.Sort(v4)
|
||||
v4 = slices.Compact(v4)
|
||||
slices.Sort(v6)
|
||||
v6 = slices.Compact(v6)
|
||||
|
||||
var list []string
|
||||
for _, p := range ports {
|
||||
list = append(list, p.nft())
|
||||
}
|
||||
set := "{ " + strings.Join(list, ", ") + " }"
|
||||
|
||||
var b strings.Builder
|
||||
b.WriteString("# Written by llmgw. llmgw's internal ports answer this machine and the\n")
|
||||
b.WriteString("# cluster's networks only; nothing else on the machine is touched.\n")
|
||||
for _, p := range ports {
|
||||
fmt.Fprintf(&b, "# %-11s %s\n", p.nft(), p.What)
|
||||
}
|
||||
fmt.Fprintf(&b, "table inet %s\ndelete table inet %s\n", Table, Table)
|
||||
fmt.Fprintf(&b, "table inet %s {\n", Table)
|
||||
b.WriteString("\tchain input {\n\t\ttype filter hook input priority filter - 1; policy accept;\n")
|
||||
b.WriteString("\t\tiif \"lo\" accept\n")
|
||||
if len(ports) > 0 {
|
||||
if len(v4) > 0 {
|
||||
fmt.Fprintf(&b, "\t\tip saddr { %s } tcp dport %s accept\n", strings.Join(v4, ", "), set)
|
||||
}
|
||||
if len(v6) > 0 {
|
||||
fmt.Fprintf(&b, "\t\tip6 saddr { %s } tcp dport %s accept\n", strings.Join(v6, ", "), set)
|
||||
}
|
||||
fmt.Fprintf(&b, "\t\ttcp dport %s drop\n", set)
|
||||
}
|
||||
b.WriteString("\t}\n}\n")
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// unitFile loads the ruleset at boot, and again whenever nftables.service
|
||||
// is (re)started — whose own configuration commonly begins by flushing every
|
||||
// table, ours included.
|
||||
const unitFile = `[Unit]
|
||||
Description=llmgw firewall: llmgw's internal ports for the cluster's networks only
|
||||
After=nftables.service
|
||||
PartOf=nftables.service
|
||||
Before=network-online.target llmgw-server.service llmgw-agent.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
RemainAfterExit=yes
|
||||
ExecStart=/usr/sbin/nft -f ` + RulesFile + `
|
||||
ExecReload=/usr/sbin/nft -f ` + RulesFile + `
|
||||
ExecStop=/usr/sbin/nft delete table inet ` + Table + `
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target nftables.service
|
||||
`
|
||||
|
||||
// Install writes the ruleset and its unit on a machine and applies them,
|
||||
// installing nftables first where it is missing. It reports whether
|
||||
// anything changed. The ruleset is checked by nft before it is put in
|
||||
// place, and a broken one changes nothing.
|
||||
//
|
||||
// What the machine's ruleset already holds is kept: a machine can be more
|
||||
// than one thing — a store member and a node — and whichever of them is set
|
||||
// up second must not take the first one's ports out of the rules.
|
||||
func Install(ctx context.Context, r Runner, s Spec) (bool, error) {
|
||||
if old, err := r.Run(ctx, "sudo cat "+RulesFile+" 2>/dev/null || true", nil); err == nil {
|
||||
prev := parse(string(old))
|
||||
s.Ports = append(prev.Ports, s.Ports...)
|
||||
s.Allow = append(prev.Allow, s.Allow...)
|
||||
}
|
||||
if len(s.Allow) == 0 {
|
||||
// Ports for this machine alone would cut every node and control
|
||||
// plane off. Not a firewall anybody asked for.
|
||||
return false, errors.New("no network to let in; the ports were left open rather than closed to every other machine")
|
||||
}
|
||||
rules := Script(s)
|
||||
body := rules + "\x00" + unitFile
|
||||
out, err := r.Run(ctx, `set -e
|
||||
if ! command -v nft >/dev/null 2>&1 && [ ! -x /usr/sbin/nft ]; then
|
||||
if command -v apt-get >/dev/null 2>&1; then
|
||||
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq nftables >/dev/null
|
||||
elif command -v dnf >/dev/null 2>&1; then
|
||||
sudo dnf install -y -q nftables >/dev/null
|
||||
else
|
||||
echo "no nft, and no apt-get or dnf to install it with" >&2; exit 1
|
||||
fi
|
||||
fi
|
||||
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
|
||||
cat > "$t/all"
|
||||
python3 - "$t" <<'PY'
|
||||
import sys
|
||||
d = sys.argv[1]
|
||||
rules, unit = open(d + "/all", "rb").read().split(b"\x00")
|
||||
open(d + "/rules", "wb").write(rules)
|
||||
open(d + "/unit", "wb").write(unit)
|
||||
PY
|
||||
sudo /usr/sbin/nft -c -f "$t/rules"
|
||||
changed=no
|
||||
if ! sudo cmp -s "$t/rules" `+RulesFile+`; then
|
||||
sudo install -d -m 755 /etc/llmgw
|
||||
sudo install -m 644 "$t/rules" `+RulesFile+`
|
||||
changed=yes
|
||||
fi
|
||||
if ! sudo cmp -s "$t/unit" /etc/systemd/system/`+Unit+`; then
|
||||
sudo install -m 644 "$t/unit" /etc/systemd/system/`+Unit+`
|
||||
sudo systemctl daemon-reload
|
||||
changed=yes
|
||||
fi
|
||||
sudo systemctl enable `+Unit+` >/dev/null 2>&1
|
||||
if [ "$changed" = yes ] || ! sudo /usr/sbin/nft list table inet `+Table+` >/dev/null 2>&1; then
|
||||
sudo /usr/sbin/nft -f `+RulesFile+`
|
||||
sudo systemctl start `+Unit+` >/dev/null 2>&1 || true
|
||||
echo changed
|
||||
else
|
||||
echo same
|
||||
fi`, []byte(body))
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
return strings.HasSuffix(strings.TrimSpace(string(out)), "changed"), nil
|
||||
}
|
||||
|
||||
// Networks are the networks a machine reaches others on: for each address,
|
||||
// the prefix of the interface its route leaves by — or the address alone,
|
||||
// when the route goes through a gateway and the network behind it is not
|
||||
// the machine's to know.
|
||||
func Networks(ctx context.Context, r Runner, toward []string) ([]netip.Prefix, error) {
|
||||
var out []netip.Prefix
|
||||
for _, t := range toward {
|
||||
a, err := netip.ParseAddr(t)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
b, err := r.Run(ctx, "ip -o route get "+a.String()+" && ip -o addr show", nil)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("finding the network toward %s: %v: %s", a, err, strings.TrimSpace(string(b)))
|
||||
}
|
||||
p, ok := networkToward(string(b), a)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("cannot tell the network toward %s from: %s", a, strings.TrimSpace(string(b)))
|
||||
}
|
||||
out = append(out, p)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// networkToward reads `ip -o route get A` followed by `ip -o addr show`.
|
||||
func networkToward(text string, a netip.Addr) (netip.Prefix, bool) {
|
||||
lines := strings.Split(strings.TrimSpace(text), "\n")
|
||||
if len(lines) == 0 {
|
||||
return netip.Prefix{}, false
|
||||
}
|
||||
route := strings.Fields(lines[0])
|
||||
field := func(name string) string {
|
||||
for i := 0; i+1 < len(route); i++ {
|
||||
if route[i] == name {
|
||||
return route[i+1]
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
if field("via") != "" {
|
||||
// Routed: the address alone.
|
||||
return netip.PrefixFrom(a, a.BitLen()), true
|
||||
}
|
||||
dev, src := field("dev"), field("src")
|
||||
if route[0] == "local" {
|
||||
// This machine's own address: the network of the interface it is on.
|
||||
dev, src = "", a.String()
|
||||
}
|
||||
for _, l := range lines[1:] {
|
||||
f := strings.Fields(l)
|
||||
// "2: eth0 inet 10.0.0.147/24 brd ..."
|
||||
if len(f) < 4 || (dev != "" && strings.TrimSuffix(f[1], ":") != dev) || (f[2] != "inet" && f[2] != "inet6") {
|
||||
continue
|
||||
}
|
||||
p, err := netip.ParsePrefix(f[3])
|
||||
if err != nil || (src != "" && p.Addr().String() != src) {
|
||||
continue
|
||||
}
|
||||
if p.Contains(a) {
|
||||
return p.Masked(), true
|
||||
}
|
||||
}
|
||||
return netip.PrefixFrom(a, a.BitLen()), true
|
||||
}
|
||||
|
||||
// parse reads back a ruleset Script wrote: its ports from the comments
|
||||
// naming them, its networks from the rules.
|
||||
func parse(text string) Spec {
|
||||
var s Spec
|
||||
for _, l := range strings.Split(text, "\n") {
|
||||
if rest, ok := strings.CutPrefix(l, "# "); ok {
|
||||
f := strings.Fields(rest)
|
||||
if len(f) == 0 {
|
||||
continue
|
||||
}
|
||||
from, to, isRange := strings.Cut(f[0], "-")
|
||||
a, err1 := strconv.Atoi(from)
|
||||
b := a
|
||||
var err2 error
|
||||
if isRange {
|
||||
b, err2 = strconv.Atoi(to)
|
||||
}
|
||||
if err1 == nil && err2 == nil {
|
||||
s.Ports = append(s.Ports, Port{From: a, To: b, What: strings.Join(f[1:], " ")})
|
||||
}
|
||||
continue
|
||||
}
|
||||
l = strings.TrimSpace(l)
|
||||
if !strings.HasPrefix(l, "ip saddr {") && !strings.HasPrefix(l, "ip6 saddr {") {
|
||||
continue
|
||||
}
|
||||
inner := l[strings.Index(l, "{")+1 : strings.Index(l, "}")]
|
||||
for _, n := range strings.Split(inner, ",") {
|
||||
if p, err := netip.ParsePrefix(strings.TrimSpace(n)); err == nil {
|
||||
s.Allow = append(s.Allow, p)
|
||||
}
|
||||
}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Present reports whether a machine has llmgw's table loaded.
|
||||
func Present(ctx context.Context, r Runner) bool {
|
||||
_, err := r.Run(ctx, "sudo -n /usr/sbin/nft list table inet "+Table+" >/dev/null 2>&1", nil)
|
||||
return err == nil
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
package firewall
|
||||
|
||||
import (
|
||||
"net/netip"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestTheRulesetGuardsOnlyLlmgwsPorts(t *testing.T) {
|
||||
s := Script(Spec{
|
||||
Ports: []Port{NodeExporter, Engines(20000, 20999), NodeExporter},
|
||||
Allow: []netip.Prefix{netip.MustParsePrefix("10.0.0.147/24"), netip.MustParsePrefix("fd00::1/64"),
|
||||
netip.MustParsePrefix("10.0.0.0/24")},
|
||||
})
|
||||
for _, want := range []string{
|
||||
"table inet llmgw\ndelete table inet llmgw\ntable inet llmgw {",
|
||||
"policy accept;",
|
||||
`iif "lo" accept`,
|
||||
"ip saddr { 10.0.0.0/24 } tcp dport { 9100, 20000-20999 } accept",
|
||||
"ip6 saddr { fd00::/64 } tcp dport { 9100, 20000-20999 } accept",
|
||||
"tcp dport { 9100, 20000-20999 } drop",
|
||||
} {
|
||||
if !strings.Contains(s, want) {
|
||||
t.Errorf("no %q in:\n%s", want, s)
|
||||
}
|
||||
}
|
||||
if strings.Contains(s, "policy drop") || strings.Contains(s, "flush ruleset") {
|
||||
t.Errorf("the ruleset reaches beyond llmgw's ports:\n%s", s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheNetworkIsTheInterfacesOrTheAddressAlone(t *testing.T) {
|
||||
addrs := `1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
|
||||
2: eth0 inet 10.0.0.147/24 brd 10.0.0.255 scope global eth0\ valid_lft forever preferred_lft forever
|
||||
3: eth1 inet 192.168.5.7/16 brd 192.168.255.255 scope global eth1\ valid_lft forever preferred_lft forever`
|
||||
cases := []struct {
|
||||
route, addr, want string
|
||||
}{
|
||||
{"10.0.0.140 dev eth0 src 10.0.0.147 uid 1000 \\ cache", "10.0.0.140", "10.0.0.0/24"},
|
||||
{"172.16.3.4 via 10.0.0.1 dev eth0 src 10.0.0.147 uid 1000 \\ cache", "172.16.3.4", "172.16.3.4/32"},
|
||||
{"192.168.9.9 dev eth1 src 192.168.5.7 uid 1000 \\ cache", "192.168.9.9", "192.168.0.0/16"},
|
||||
// Its own address: the network it is on.
|
||||
{"local 10.0.0.147 dev lo src 10.0.0.147 uid 1000 \\ cache <local>", "10.0.0.147", "10.0.0.0/24"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
got, ok := networkToward(c.route+"\n"+addrs, netip.MustParseAddr(c.addr))
|
||||
if !ok || got.String() != c.want {
|
||||
t.Errorf("toward %s: %v %v, expected %s", c.addr, got, ok, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestARulesetReadsBackAsItsSpec(t *testing.T) {
|
||||
in := Spec{Ports: []Port{Postgres, Engines(20000, 20999)},
|
||||
Allow: []netip.Prefix{netip.MustParsePrefix("10.0.0.0/24"), netip.MustParsePrefix("fd00::/64")}}
|
||||
got := parse(Script(in))
|
||||
if Script(got) != Script(in) {
|
||||
t.Errorf("read back as\n%s\nfrom\n%s", Script(got), Script(in))
|
||||
}
|
||||
}
|
||||
@@ -163,6 +163,10 @@ func Read(r io.Reader) (Header, error) {
|
||||
}
|
||||
return nil
|
||||
}
|
||||
// depth is how many arrays the value being read is inside. Real files
|
||||
// nest none; a crafted one nesting thousands deep made a recursion as
|
||||
// deep, a few bytes per level.
|
||||
depth := 0
|
||||
var value func(t uint32) (string, error)
|
||||
value = func(t uint32) (string, error) {
|
||||
num = 0
|
||||
@@ -195,6 +199,11 @@ func Read(r io.Reader) (Header, error) {
|
||||
if n > math.MaxInt32 {
|
||||
return "", fmt.Errorf("a %d-element array in a GGUF header", n)
|
||||
}
|
||||
if depth >= maxArrayDepth {
|
||||
return "", fmt.Errorf("arrays nested more than %d deep in a GGUF header", maxArrayDepth)
|
||||
}
|
||||
depth++
|
||||
defer func() { depth-- }()
|
||||
for i := uint64(0); i < n; i++ {
|
||||
if _, err := value(et); err != nil {
|
||||
return "", err
|
||||
@@ -296,6 +305,9 @@ func FromURL(ctx context.Context, client *http.Client, url string) (Header, erro
|
||||
return Read(io.LimitReader(resp.Body, headerBytes))
|
||||
}
|
||||
|
||||
// maxArrayDepth is how deep arrays may nest in a header.
|
||||
const maxArrayDepth = 4
|
||||
|
||||
// IsAdapter reports whether the file is a LoRA adapter rather than a model.
|
||||
func (h Header) IsAdapter() bool { return h.Type == "adapter" }
|
||||
|
||||
|
||||
@@ -143,3 +143,23 @@ func TestTheKVCacheIsSizedFromTheModelsShape(t *testing.T) {
|
||||
t.Error("a header with no shape produced a figure")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArraysNestedDeepAreRefused(t *testing.T) {
|
||||
var b bytes.Buffer
|
||||
w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
|
||||
b.WriteString("GGUF")
|
||||
w(uint32(3))
|
||||
w(uint64(0))
|
||||
w(uint64(1))
|
||||
w(uint64(len("deep")))
|
||||
b.WriteString("deep")
|
||||
w(uint32(tArray))
|
||||
for range 10_000 {
|
||||
w(uint32(tArray))
|
||||
w(uint64(1))
|
||||
}
|
||||
_, err := Read(bytes.NewReader(b.Bytes()))
|
||||
if err == nil || !strings.Contains(err.Error(), "nested") {
|
||||
t.Errorf("ten thousand nested arrays: %v", err)
|
||||
}
|
||||
}
|
||||
+40
-15
@@ -131,13 +131,22 @@ type response struct {
|
||||
|
||||
// Listener waits for machines to join.
|
||||
type Listener struct {
|
||||
ln net.Listener
|
||||
addr string
|
||||
ln net.Listener
|
||||
addr string
|
||||
// now is time.Now, replaced by tests.
|
||||
now func() time.Time
|
||||
fingerprint [16]byte
|
||||
ttl time.Duration
|
||||
|
||||
mu sync.Mutex
|
||||
pending map[[16]byte]chan *Remote // by secret
|
||||
pending map[[16]byte]waiting // by secret
|
||||
}
|
||||
|
||||
// waiting is a code not yet used: where its machine is handed over, and
|
||||
// until when it may come.
|
||||
type waiting struct {
|
||||
ch chan *Remote
|
||||
expires time.Time
|
||||
}
|
||||
|
||||
// Listen starts listening on bind (":7444") and advertises addr ("10.0.0.5:7444")
|
||||
@@ -151,7 +160,7 @@ func Listen(bind, addr string, ttl time.Duration) (*Listener, error) {
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listening for machines to join on %s: %w", bind, err)
|
||||
}
|
||||
l := &Listener{ln: ln, addr: addr, fingerprint: fp, ttl: ttl, pending: map[[16]byte]chan *Remote{}}
|
||||
l := &Listener{ln: ln, addr: addr, fingerprint: fp, ttl: ttl, pending: map[[16]byte]waiting{}, now: time.Now}
|
||||
go l.accept()
|
||||
return l, nil
|
||||
}
|
||||
@@ -188,11 +197,12 @@ func (l *Listener) Code(role string) (Code, <-chan *Remote) {
|
||||
var secret [16]byte
|
||||
_, _ = rand.Read(secret[:])
|
||||
ch := make(chan *Remote, 1)
|
||||
expires := time.Now().Add(l.ttl)
|
||||
l.mu.Lock()
|
||||
l.pending[secret] = ch
|
||||
l.pending[secret] = waiting{ch: ch, expires: expires}
|
||||
l.mu.Unlock()
|
||||
return Code{Addr: l.addr, Fingerprint: l.fingerprint, Secret: secret,
|
||||
Expires: time.Now().Add(l.ttl).Unix(), Role: role}, ch
|
||||
Expires: expires.Unix(), Role: role}, ch
|
||||
}
|
||||
|
||||
// Close stops listening; machines already joined stay connected until their
|
||||
@@ -220,16 +230,29 @@ func (l *Listener) admit(c net.Conn) {
|
||||
}
|
||||
l.mu.Lock()
|
||||
var ch chan *Remote
|
||||
expired := false
|
||||
now := l.now()
|
||||
for secret, p := range l.pending {
|
||||
if subtle.ConstantTimeCompare(secret[:], h.Secret) == 1 {
|
||||
ch = p
|
||||
// One machine a code: whoever comes second is turned away.
|
||||
// One machine a code: whoever comes second is turned away. And
|
||||
// not after it expires, which the joining side checking was no
|
||||
// guarantee of: a code is a credential for as long as this
|
||||
// listener runs, unless this side looks too.
|
||||
delete(l.pending, secret)
|
||||
if now.After(p.expires) {
|
||||
expired = true
|
||||
} else {
|
||||
ch = p.ch
|
||||
}
|
||||
}
|
||||
}
|
||||
l.mu.Unlock()
|
||||
if ch == nil {
|
||||
_, _ = c.Write([]byte(`{"error":"this code is not one the setup is waiting for, or it was used already"}` + "\n"))
|
||||
msg := "this code is not one the setup is waiting for, or it was used already"
|
||||
if expired {
|
||||
msg = "this code expired; ask for a new one"
|
||||
}
|
||||
_, _ = c.Write([]byte(`{"error":"` + msg + `"}` + "\n"))
|
||||
_ = c.Close()
|
||||
return
|
||||
}
|
||||
@@ -238,17 +261,19 @@ func (l *Listener) admit(c net.Conn) {
|
||||
_ = c.Close()
|
||||
return
|
||||
}
|
||||
addr := h.Addr
|
||||
if addr == "" {
|
||||
addr, _, _ = net.SplitHostPort(c.RemoteAddr().String())
|
||||
}
|
||||
ch <- &Remote{conn: c, r: r, Hostname: h.Hostname, Addr: addr}
|
||||
// The address it connected from, not the one it says it has: the setup
|
||||
// lets that address into the store, and a claim is anyone's to make.
|
||||
addr, _, _ := net.SplitHostPort(c.RemoteAddr().String())
|
||||
ch <- &Remote{conn: c, r: r, Hostname: h.Hostname, Addr: addr, Claimed: h.Addr}
|
||||
}
|
||||
|
||||
// Remote is a joined machine, as the setup runs things on it.
|
||||
type Remote struct {
|
||||
Hostname string
|
||||
Addr string
|
||||
// Addr is where it connected from; Claimed what it said its address is,
|
||||
// for showing beside it when the two differ.
|
||||
Addr string
|
||||
Claimed string
|
||||
|
||||
mu sync.Mutex
|
||||
conn net.Conn
|
||||
|
||||
@@ -92,3 +92,20 @@ func TestAnExpiredOrDamagedCodeIsRefusedBeforeDialling(t *testing.T) {
|
||||
t.Errorf("%+v, %v", back, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheSetupRefusesAnExpiredCodeEvenFromAJoinerThatDidNotLook(t *testing.T) {
|
||||
l := listen(t)
|
||||
code, arrived := l.Code("node")
|
||||
// Time moves on past the code's life on the setup's side; the joiner is
|
||||
// handed the code as it was, without checking it again.
|
||||
l.now = func() time.Time { return time.Now().Add(2 * time.Hour) }
|
||||
err := Serve(context.Background(), code, func(string) {})
|
||||
if err == nil || !strings.Contains(err.Error(), "expired") {
|
||||
t.Fatalf("an expired code was taken: %v", err)
|
||||
}
|
||||
select {
|
||||
case m := <-arrived:
|
||||
t.Fatalf("a machine arrived on an expired code: %+v", m)
|
||||
default:
|
||||
}
|
||||
}
|
||||
@@ -446,6 +446,8 @@ func TestTheAPIWillNotFetchFromThePrivateNetwork(t *testing.T) {
|
||||
for _, src := range []string{
|
||||
"http://127.0.0.1:1/x.gguf", "http://169.254.169.254/latest/meta-data",
|
||||
"http://10.0.0.140:9090/api/v1/nodes", "http://[::1]:1/x",
|
||||
// The metadata address inside NAT64 and 6to4 ones.
|
||||
"http://[64:ff9b::a9fe:a9fe]/latest", "http://[2002:a9fe:a9fe::1]/latest",
|
||||
} {
|
||||
st, err := m.Start(src, Entry{Model: "m", Version: "v", File: "x.gguf"})
|
||||
if err != nil {
|
||||
@@ -456,7 +458,7 @@ func TestTheAPIWillNotFetchFromThePrivateNetwork(t *testing.T) {
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
st, _ = m.Status("m", "v")
|
||||
}
|
||||
if st.State != Failed || !strings.Contains(st.Err, "refusing to fetch") {
|
||||
if st.State != Failed || !strings.Contains(st.Err, "refusing to reach") {
|
||||
t.Errorf("%s: %+v", src, st)
|
||||
}
|
||||
m.Forget("m", "v")
|
||||
|
||||
@@ -128,6 +128,7 @@ func (m *Manager) Start(source string, e Entry) (Status, error) {
|
||||
State: Fetching, Source: source, StartedAt: time.Now(),
|
||||
}
|
||||
m.running[key] = st
|
||||
m.forgetOld(time.Now())
|
||||
out := *st
|
||||
m.mu.Unlock()
|
||||
|
||||
@@ -308,6 +309,19 @@ func (m *Manager) All() ([]Status, error) {
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// keptFinished is how long a finished download's status is remembered.
|
||||
const keptFinished = 7 * 24 * time.Hour
|
||||
|
||||
// forgetOld drops downloads finished more than keptFinished ago. Called with
|
||||
// m.mu held.
|
||||
func (m *Manager) forgetOld(now time.Time) {
|
||||
for key, st := range m.running {
|
||||
if st.State != Fetching && !st.FinishedAt.IsZero() && now.Sub(st.FinishedAt) > keptFinished {
|
||||
delete(m.running, key)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Forget drops a remembered download, so a failed one can be retried.
|
||||
//
|
||||
// Without it a fetch that failed once would refuse every later attempt at the
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
package mirror
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestAFinishedDownloadIsForgottenAfterAWeek(t *testing.T) {
|
||||
m := NewManager(t.TempDir(), discardLog())
|
||||
now := time.Now()
|
||||
m.mu.Lock()
|
||||
m.running["old"] = &Status{State: Present, FinishedAt: now.Add(-keptFinished - time.Hour)}
|
||||
m.running["recent"] = &Status{State: Failed, FinishedAt: now.Add(-time.Hour)}
|
||||
m.running["going"] = &Status{State: Fetching}
|
||||
m.forgetOld(now)
|
||||
_, old := m.running["old"]
|
||||
n := len(m.running)
|
||||
m.mu.Unlock()
|
||||
if old || n != 2 {
|
||||
t.Errorf("after a week: old remembered=%v, %d left", old, n)
|
||||
}
|
||||
}
|
||||
@@ -1,80 +1,35 @@
|
||||
package mirror
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"net/url"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/safedial"
|
||||
)
|
||||
|
||||
// errPrivateSource is a fetch refused because it would reach this machine or
|
||||
// the network behind it rather than the internet.
|
||||
var errPrivateSource = errors.New("refusing to fetch from a loopback, private or link-local address")
|
||||
var errPrivateSource = safedial.ErrBlocked
|
||||
|
||||
// fetchClient downloads sources for the management API.
|
||||
//
|
||||
// The API fetches whatever URL it is given and the file server then hands the
|
||||
// result to anyone. Without this, "fetch http://169.254.169.254/..." was a way
|
||||
// to read a cloud's instance metadata — or anything else on the network the
|
||||
// mirror sits in — through the mirror. The check is made on the address
|
||||
// actually dialled, after resolution and on every redirect, so a name that
|
||||
// resolves somewhere public and then somewhere private is caught too.
|
||||
// mirror sits in — through the mirror. No overall timeout: a model is tens of
|
||||
// gigabytes, and after the first byte a stall is the context's to end.
|
||||
func fetchClient(allowPrivate bool) *http.Client {
|
||||
d := &net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}
|
||||
if !allowPrivate {
|
||||
d.Control = func(_, address string, _ syscall.RawConn) error {
|
||||
host, _, err := net.SplitHostPort(address)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
a, err := netip.ParseAddr(host)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if blockedAddr(a.Unmap()) {
|
||||
return fmt.Errorf("%w (%s)", errPrivateSource, a)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return &http.Client{
|
||||
Transport: &http.Transport{
|
||||
Proxy: http.ProxyFromEnvironment,
|
||||
DialContext: d.DialContext,
|
||||
TLSHandshakeTimeout: 30 * time.Second,
|
||||
ResponseHeaderTimeout: 2 * time.Minute,
|
||||
ForceAttemptHTTP2: true,
|
||||
},
|
||||
// No overall timeout: a model is tens of gigabytes. The dial, the
|
||||
// handshake and the first byte are bounded above; after that a stall
|
||||
// is the context's to end.
|
||||
CheckRedirect: func(req *http.Request, via []*http.Request) error {
|
||||
if len(via) >= 10 {
|
||||
return errors.New("too many redirects")
|
||||
}
|
||||
return checkScheme(req.URL)
|
||||
},
|
||||
if allowPrivate {
|
||||
return safedial.Client(safedial.Policy{Private: true, Loopback: true, LinkLocal: true}, 0)
|
||||
}
|
||||
return safedial.Client(safedial.Policy{}, 0)
|
||||
}
|
||||
|
||||
func blockedAddr(a netip.Addr) bool {
|
||||
cgnat := netip.MustParsePrefix("100.64.0.0/10")
|
||||
return a.IsLoopback() || a.IsPrivate() || a.IsLinkLocalUnicast() ||
|
||||
a.IsLinkLocalMulticast() || a.IsMulticast() || a.IsUnspecified() || cgnat.Contains(a)
|
||||
}
|
||||
func blockedAddr(a netip.Addr) bool { return safedial.Blocked(a, safedial.Policy{}) }
|
||||
|
||||
// checkScheme allows the two schemes a model is published over. file:// and
|
||||
// the rest would read this machine's own disk.
|
||||
func checkScheme(u *url.URL) error {
|
||||
if u.Scheme != "http" && u.Scheme != "https" {
|
||||
return fmt.Errorf("%q is not an http or https source", u.Redacted())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func checkScheme(u *url.URL) error { return safedial.CheckScheme(u) }
|
||||
|
||||
// validSource checks a source before a download is started for it.
|
||||
func validSource(source string) error {
|
||||
|
||||
@@ -227,6 +227,9 @@ func (g *GoSSH) hostKeyCallback() (ssh.HostKeyCallback, error) {
|
||||
// sudoersDrop is the grant PrepareSudo writes and RevokeSudo removes.
|
||||
const sudoersDrop = "/etc/sudoers.d/llmgw-setup"
|
||||
|
||||
// sudoGrantLife is how long the grant lasts if nothing removes it sooner.
|
||||
const sudoGrantLife = time.Hour
|
||||
|
||||
// CanSudo reports whether the user can sudo without a password now.
|
||||
func (g *GoSSH) CanSudo(ctx context.Context) bool {
|
||||
_, err := g.Run(ctx, "sudo -n true", nil)
|
||||
@@ -245,17 +248,14 @@ func (g *GoSSH) PrepareSudo(ctx context.Context) error {
|
||||
if g.SudoPassword == "" {
|
||||
return fmt.Errorf("%s cannot use sudo without a password, and none was given", g.User)
|
||||
}
|
||||
line := fmt.Sprintf("%s ALL=(ALL) NOPASSWD:ALL", g.User)
|
||||
// Checked by visudo before it is put in place: a broken file in
|
||||
// sudoers.d breaks sudo for everyone on the machine.
|
||||
inner := fmt.Sprintf("umask 337; printf '%%s\\n' %s %s > %[3]s.new && visudo -cf %[3]s.new >/dev/null && mv -f %[3]s.new %[3]s",
|
||||
shellQuote("# llmgw setup: removed when it finishes"), shellQuote(line), sudoersDrop)
|
||||
script := "sudo -S -p '' sh -c " + shellQuote(inner)
|
||||
script := "sudo -S -p '' sh -c " + shellQuote(sudoGrantScript(g.User))
|
||||
if out, err := g.Run(ctx, script, []byte(g.SudoPassword+"\n")); err != nil {
|
||||
return fmt.Errorf("sudo with the password given failed on %s: %v: %s", g.Addr, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
g.mu.Lock()
|
||||
g.sudoGrant = true
|
||||
// Not needed again: the grant stands in for it from here on.
|
||||
g.SudoPassword = ""
|
||||
g.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
@@ -276,3 +276,21 @@ func (g *GoSSH) RevokeSudo(ctx context.Context) error {
|
||||
g.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
|
||||
// sudoGrantScript writes the grant, run as root.
|
||||
func sudoGrantScript(user string) string {
|
||||
line := fmt.Sprintf("%s ALL=(ALL) NOPASSWD:ALL", user)
|
||||
// Checked by visudo before it is put in place: a broken file in
|
||||
// sudoers.d breaks sudo for everyone on the machine.
|
||||
//
|
||||
// And it removes itself an hour later whatever happens to this process:
|
||||
// RevokeSudo is called when the setup finishes, and a setup that crashes
|
||||
// or is interrupted never gets there — which would leave a permanent
|
||||
// passwordless root behind.
|
||||
inner := fmt.Sprintf("umask 337; printf '%%s\\n' %s %s > %[3]s.new && visudo -cf %[3]s.new >/dev/null && mv -f %[3]s.new %[3]s && "+
|
||||
"{ systemd-run --quiet --on-active=%[4]d /bin/rm -f %[3]s >/dev/null 2>&1 || "+
|
||||
"(setsid nohup sh -c 'sleep %[4]d; rm -f %[3]s' >/dev/null 2>&1 &) ; }",
|
||||
shellQuote("# llmgw setup: removed when it finishes, or within the hour"), shellQuote(line), sudoersDrop,
|
||||
int(sudoGrantLife.Seconds()))
|
||||
return inner
|
||||
}
|
||||
+98
-19
@@ -19,6 +19,9 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
@@ -30,6 +33,7 @@ import (
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/doctor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
@@ -62,6 +66,12 @@ func ParseFabric(s string) (Fabric, error) {
|
||||
type Options struct {
|
||||
// NoNodeExporter leaves Prometheus' node exporter off the machine.
|
||||
NoNodeExporter bool
|
||||
// NoFirewall leaves the machine's firewall as it is. Otherwise its llmgw
|
||||
// ports — FirewallPorts — answer only the networks it reaches the
|
||||
// control planes on, and FirewallAllow.
|
||||
NoFirewall bool
|
||||
FirewallPorts []firewall.Port
|
||||
FirewallAllow []netip.Prefix
|
||||
|
||||
// Addr is how to reach the machine over SSH, and User who to log in as.
|
||||
Addr, User string
|
||||
@@ -88,6 +98,8 @@ type Options struct {
|
||||
Binary, Version string
|
||||
// Releases is where release files are downloaded from.
|
||||
Releases string
|
||||
// ReleaseClient fetches them; nil is a default client.
|
||||
ReleaseClient *http.Client
|
||||
// IgnoreChecks goes on when doctor finds a problem.
|
||||
IgnoreChecks bool
|
||||
// DisableEngines are engines this node never installs or advertises,
|
||||
@@ -354,25 +366,25 @@ var releaseTag = release.Tag
|
||||
func (r *Runner) binary(ctx context.Context, res *Result) error {
|
||||
v := r.Opts.Version
|
||||
if releaseTag.MatchString(v) {
|
||||
r.emit(StepBinary, Running, "downloading llmgw "+v+" on the node", "")
|
||||
name := "llmgw-" + v + "-linux-" + res.Arch
|
||||
base := strings.TrimSuffix(r.Opts.Releases, "/") + "/" + v
|
||||
// Fetched by the node, for its own architecture, and checked against
|
||||
// the release's checksums, rather than handed whatever this machine
|
||||
// happens to run.
|
||||
script := fmt.Sprintf(`set -e
|
||||
d=$(mktemp -d); cd "$d"
|
||||
curl -fsSL -o %[1]s '%[2]s/%[1]s'
|
||||
curl -fsSL -o checksums.txt '%[2]s/checksums.txt'
|
||||
grep " %[1]s$" checksums.txt | sha256sum -c - >/dev/null
|
||||
sudo install -m 0755 %[1]s /usr/local/bin/llmgw
|
||||
cd /; rm -rf "$d"`, name, base)
|
||||
out, err := r.Remote.Run(ctx, script, nil)
|
||||
// Fetched here, for the node's architecture, and checked here —
|
||||
// signature and checksum, by the same code that upgrades this
|
||||
// machine — then handed to the node. The node checking a release
|
||||
// itself could only compare it with a checksum from the same
|
||||
// server, which proves nothing about who put both there.
|
||||
r.emit(StepBinary, Running, "downloading llmgw "+v+" for "+res.Arch+" and checking it", "")
|
||||
b, err := release.Fetch(ctx, r.Opts.ReleaseClient, r.Opts.Releases, v, "linux", goArch(res.Arch))
|
||||
if err == nil {
|
||||
r.emit(StepBinary, Done, "llmgw "+v+" from the release, checksum verified", "")
|
||||
if err := r.installBinary(ctx, b); err != nil {
|
||||
return err
|
||||
}
|
||||
what := "checksum verified"
|
||||
if release.TrustedKey() != "" {
|
||||
what = "signature and checksum verified"
|
||||
}
|
||||
r.emit(StepBinary, Done, "llmgw "+v+" from the release, "+what, "")
|
||||
return nil
|
||||
}
|
||||
r.emit(StepBinary, Warn, "the node could not download the release: "+lastLine(out), "")
|
||||
r.emit(StepBinary, Warn, "the release could not be fetched and checked here: "+err.Error(), "")
|
||||
}
|
||||
// A development build has no release to fetch, and a node that cannot
|
||||
// reach the releases gets this machine's copy — when it can run it.
|
||||
@@ -385,13 +397,32 @@ cd /; rm -rf "$d"`, name, base)
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading %s: %w", r.Opts.Binary, err)
|
||||
}
|
||||
if out, err := r.Remote.Run(ctx, `set -e; t=$(mktemp); cat > "$t"; sudo install -m 0755 "$t" /usr/local/bin/llmgw; rm -f "$t"`, b); err != nil {
|
||||
return fmt.Errorf("installing the binary: %v: %s", err, lastLine(out))
|
||||
if err := r.installBinary(ctx, b); err != nil {
|
||||
return err
|
||||
}
|
||||
r.emit(StepBinary, Done, "llmgw "+orUnknown(v)+", copied from this machine", "")
|
||||
return nil
|
||||
}
|
||||
|
||||
// installBinary puts bytes at /usr/local/bin/llmgw on the node.
|
||||
func (r *Runner) installBinary(ctx context.Context, b []byte) error {
|
||||
if out, err := r.Remote.Run(ctx, `set -e; t=$(mktemp); cat > "$t"; sudo install -m 0755 "$t" /usr/local/bin/llmgw; rm -f "$t"`, b); err != nil {
|
||||
return fmt.Errorf("installing the binary: %v: %s", err, lastLine(out))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// goArch is Go's name for what uname -m says.
|
||||
func goArch(uname string) string {
|
||||
switch uname {
|
||||
case "x86_64", "amd64":
|
||||
return "amd64"
|
||||
case "aarch64", "arm64":
|
||||
return "arm64"
|
||||
}
|
||||
return uname
|
||||
}
|
||||
|
||||
// --- check ---
|
||||
|
||||
func (r *Runner) check(ctx context.Context, res *Result) error {
|
||||
@@ -426,6 +457,10 @@ func (r *Runner) check(ctx context.Context, res *Result) error {
|
||||
}
|
||||
return fmt.Errorf("llmgw doctor did not run on the node: %v: %s", err, lastLine(out))
|
||||
}
|
||||
if !r.Opts.NoFirewall && len(r.Opts.FirewallPorts) > 0 {
|
||||
// A missing firewall is what this run is about to write.
|
||||
rep.Findings = slices.DeleteFunc(rep.Findings, func(f doctor.Finding) bool { return f.Area == "firewall" })
|
||||
}
|
||||
res.Facts, res.Findings = rep.Facts, rep.Findings
|
||||
|
||||
fails := 0
|
||||
@@ -665,7 +700,15 @@ func (r *Runner) checkEngines(ctx context.Context) []string {
|
||||
|
||||
// --- start ---
|
||||
|
||||
func (r *Runner) start(ctx context.Context, _ *Result) error {
|
||||
func (r *Runner) start(ctx context.Context, res *Result) error {
|
||||
// The user engines run as, so the agent drops root before starting one:
|
||||
// in the groups that own the GPU devices.
|
||||
if out, err := r.Remote.Run(ctx, `set -e
|
||||
id llmgw-engine >/dev/null 2>&1 || sudo useradd --system --no-create-home --shell /usr/sbin/nologin llmgw-engine
|
||||
for g in video render; do getent group "$g" >/dev/null && sudo usermod -aG "$g" llmgw-engine; done
|
||||
true`, nil); err != nil {
|
||||
r.emit(StepStart, Warn, "the user engines run as could not be made, so they run as root: "+lastLine(out), "")
|
||||
}
|
||||
r.emit(StepStart, Running, "starting the agent", "")
|
||||
if out, err := r.Remote.Run(ctx, `set -e
|
||||
sudo systemctl daemon-reload
|
||||
@@ -679,9 +722,45 @@ systemctl is-active llmgw-agent`, nil); err != nil {
|
||||
if !r.Opts.NoNodeExporter {
|
||||
r.nodeExporter(ctx)
|
||||
}
|
||||
if !r.Opts.NoFirewall && len(r.Opts.FirewallPorts) > 0 {
|
||||
r.firewall(ctx, res)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// firewall keeps the node's llmgw ports — its engines, its metrics — to the
|
||||
// networks it reaches the control planes on. A warning when it cannot: the
|
||||
// node serves either way, and says how to see why.
|
||||
func (r *Runner) firewall(ctx context.Context, res *Result) {
|
||||
var toward []string
|
||||
for _, s := range res.Servers {
|
||||
u, err := url.Parse(s)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if _, err := netip.ParseAddr(u.Hostname()); err == nil {
|
||||
toward = append(toward, u.Hostname())
|
||||
} else if ips, err := net.LookupIP(u.Hostname()); err == nil && len(ips) > 0 {
|
||||
toward = append(toward, ips[0].String())
|
||||
}
|
||||
}
|
||||
nets, err := firewall.Networks(ctx, r.Remote, toward)
|
||||
if err == nil {
|
||||
nets = append(nets, r.Opts.FirewallAllow...)
|
||||
_, err = firewall.Install(ctx, r.Remote, firewall.Spec{Ports: r.Opts.FirewallPorts, Allow: nets})
|
||||
}
|
||||
if err != nil {
|
||||
r.emit(StepStart, Warn, "the firewall was not set: "+err.Error()+
|
||||
"; the node's engine ports answer every network it is on", "")
|
||||
return
|
||||
}
|
||||
var shown []string
|
||||
for _, n := range nets {
|
||||
shown = append(shown, n.String())
|
||||
}
|
||||
r.emit(StepStart, Info, "llmgw's ports on the node answer "+strings.Join(shown, ", ")+" only ("+firewall.RulesFile+")", "")
|
||||
}
|
||||
|
||||
// nodeExporter installs Prometheus' node exporter from the distribution, for
|
||||
// what llmgw does not measure: network and disk traffic, filesystems. Best
|
||||
// effort: a node without it serves as well, so a failure is a warning.
|
||||
|
||||
@@ -2,8 +2,13 @@ package onboard
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"runtime"
|
||||
"strings"
|
||||
@@ -12,6 +17,7 @@ import (
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/doctor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
@@ -47,11 +53,6 @@ func (f *fakeMachine) Run(_ context.Context, script string, stdin []byte) ([]byt
|
||||
return []byte("sudo: a password is required"), errors.New("exit 1")
|
||||
}
|
||||
return nil, nil
|
||||
case strings.Contains(script, "checksums.txt"):
|
||||
if f.download != nil {
|
||||
return []byte("curl: (6) Could not resolve host"), f.download
|
||||
}
|
||||
return nil, nil
|
||||
case strings.Contains(script, "install -m 0755") && stdin != nil:
|
||||
f.files["/usr/local/bin/llmgw"] = stdin
|
||||
return nil, nil
|
||||
@@ -132,31 +133,85 @@ func TestANodeIsAddedFromItsAddressAlone(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReleaseIsFetchedByTheNodeAndCheckedThere(t *testing.T) {
|
||||
// releaseFor serves one release for linux/<arch>, its checksums signed with
|
||||
// a key the test then trusts; tamper changes the binary after signing.
|
||||
func releaseFor(t *testing.T, version, arch string, body []byte, signed, tamper bool) string {
|
||||
t.Helper()
|
||||
pub, priv, err := release.GenerateKey()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("LLMGW_RELEASE_KEY", pub)
|
||||
asset := release.AssetName(version, "linux", arch)
|
||||
sum := sha256.Sum256(body)
|
||||
sums := hex.EncodeToString(sum[:]) + " " + asset + "\n"
|
||||
sig, _ := release.Sign(priv, []byte(sums))
|
||||
if tamper {
|
||||
body = []byte("tampered")
|
||||
}
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/" + version + "/checksums.txt":
|
||||
_, _ = w.Write([]byte(sums))
|
||||
case "/" + version + "/" + release.SignatureName:
|
||||
if !signed {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte(sig))
|
||||
case "/" + version + "/" + asset:
|
||||
_, _ = w.Write(body)
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return srv.URL
|
||||
}
|
||||
|
||||
func TestAReleaseIsFetchedAndCheckedHereThenGivenToTheNode(t *testing.T) {
|
||||
m := newFake()
|
||||
r, _ := runner(t, m, Options{Version: "v1.2.3", Releases: "https://example.test/releases/download"})
|
||||
base := releaseFor(t, "v1.2.3", hostArch(), []byte("the release"), true, false)
|
||||
r, events := runner(t, m, Options{Version: "v1.2.3", Releases: base})
|
||||
if _, err := r.Run(t.Context()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !m.ran("sha256sum -c") || !m.ran("https://example.test/releases/download/v1.2.3") {
|
||||
t.Error("the release was not fetched and verified on the node")
|
||||
if got := string(m.files["/usr/local/bin/llmgw"]); got != "the release" {
|
||||
t.Errorf("the node was given %q", got)
|
||||
}
|
||||
if _, copied := m.files["/usr/local/bin/llmgw"]; copied {
|
||||
t.Error("this machine's binary was copied although the node fetched the release")
|
||||
if m.ran("curl") {
|
||||
t.Error("the node fetched something itself")
|
||||
}
|
||||
if !said(*events, Done, "signature and checksum verified") {
|
||||
t.Errorf("the check was not reported: %v", *events)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeThatCannotDownloadGetsThisMachinesCopy(t *testing.T) {
|
||||
func TestAnUnsignedOrAlteredReleaseIsNotGivenToTheNode(t *testing.T) {
|
||||
for name, c := range map[string][2]bool{"unsigned": {false, false}, "altered": {true, true}} {
|
||||
m := newFake()
|
||||
base := releaseFor(t, "v1.2.3", hostArch(), []byte("the release"), c[0], c[1])
|
||||
r, events := runner(t, m, Options{Version: "v1.2.3", Releases: base})
|
||||
_, _ = r.Run(t.Context())
|
||||
if got := string(m.files["/usr/local/bin/llmgw"]); got == "the release" || got == "tampered" {
|
||||
t.Errorf("%s: the node was given the release", name)
|
||||
}
|
||||
if !said(*events, Warn, "could not be fetched and checked") {
|
||||
t.Errorf("%s: the refusal was not said", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeGetsThisMachinesCopyWhenTheReleaseCannotBeReached(t *testing.T) {
|
||||
m := newFake()
|
||||
m.download = errors.New("exit 6")
|
||||
r, events := runner(t, m, Options{Version: "v1.2.3", Releases: "https://example.test"})
|
||||
r, events := runner(t, m, Options{Version: "v1.2.3", Releases: "https://127.0.0.1:1"})
|
||||
if _, err := r.Run(t.Context()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(m.files["/usr/local/bin/llmgw"]) == 0 {
|
||||
t.Error("the binary was not copied")
|
||||
}
|
||||
if !said(*events, Warn, "could not download") {
|
||||
if !said(*events, Warn, "could not be fetched") {
|
||||
t.Error("the failed download was not mentioned")
|
||||
}
|
||||
}
|
||||
@@ -321,3 +376,16 @@ func TestTheFabricsGivenAreCheckedAsDeclared(t *testing.T) {
|
||||
t.Errorf("doctor was not told the fabric: %v", m.scripts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheSudoGrantRemovesItselfWithinTheHour(t *testing.T) {
|
||||
inner := sudoGrantScript("ilker")
|
||||
if !strings.Contains(inner, "--on-active=3600") || !strings.Contains(inner, "sleep 3600; rm -f "+sudoersDrop) {
|
||||
t.Errorf("no removal scheduled: %s", inner)
|
||||
}
|
||||
if !strings.Contains(inner, "visudo -cf") {
|
||||
t.Error("written without visudo's check")
|
||||
}
|
||||
if out, err := exec.Command("sh", "-n", "-c", inner).CombinedOutput(); err != nil {
|
||||
t.Errorf("not shell: %v %s\n%s", err, out, inner)
|
||||
}
|
||||
}
|
||||
@@ -41,7 +41,9 @@ func (s *SSH) Run(ctx context.Context, script string, stdin []byte) ([]byte, err
|
||||
if s.User != "" {
|
||||
target = s.User + "@" + s.Addr
|
||||
}
|
||||
args := append(append([]string{}, sshOptions...), target, "sh", "-c", shellQuote(script))
|
||||
// "--" before the target: an address beginning with "-" is not read as
|
||||
// an option.
|
||||
args := append(append([]string{}, sshOptions...), "--", target, "sh", "-c", shellQuote(script))
|
||||
cmd := exec.CommandContext(ctx, "ssh", args...)
|
||||
if stdin != nil {
|
||||
cmd.Stdin = bytes.NewReader(stdin)
|
||||
|
||||
+109
-19
@@ -37,28 +37,75 @@ type Member struct {
|
||||
IP string
|
||||
}
|
||||
|
||||
// Secrets are the cluster's three passwords.
|
||||
// Secrets are the cluster's passwords: PostgreSQL's three, and the one
|
||||
// Patroni's REST API asks for before a change.
|
||||
type Secrets struct {
|
||||
Superuser, Replicator, App string
|
||||
Superuser, Replicator, App, RestAPI string
|
||||
}
|
||||
|
||||
// Etcd is a member's /etc/default/etcd, for Ubuntu's etcd-server.
|
||||
func Etcd(m Member, all []Member) string {
|
||||
//
|
||||
// With tls, members speak TLS to each other and to clients, with a
|
||||
// certificate from the cluster's authority (EtcdTLSDir), and a client must
|
||||
// present one too: without, anyone reaching :2379 could take the leader lock
|
||||
// or rewrite Patroni's configuration.
|
||||
func Etcd(m Member, all []Member, tls bool) string {
|
||||
scheme := "http"
|
||||
if tls {
|
||||
scheme = "https"
|
||||
}
|
||||
var peers []string
|
||||
for _, p := range all {
|
||||
peers = append(peers, fmt.Sprintf("%s=http://%s:2380", p.Name, p.IP))
|
||||
peers = append(peers, fmt.Sprintf("%s=%s://%s:2380", p.Name, scheme, p.IP))
|
||||
}
|
||||
return fmt.Sprintf(`# llmgw: one member of the etcd that holds Patroni's leader lock.
|
||||
client := fmt.Sprintf("http://%s:2379,http://127.0.0.1:2379", m.IP)
|
||||
if tls {
|
||||
client = fmt.Sprintf("https://%s:2379,https://127.0.0.1:2379", m.IP)
|
||||
}
|
||||
conf := fmt.Sprintf(`# llmgw: one member of the etcd that holds Patroni's leader lock.
|
||||
ETCD_NAME=%[1]s
|
||||
ETCD_DATA_DIR=/var/lib/etcd/default
|
||||
ETCD_LISTEN_PEER_URLS=http://%[2]s:2380
|
||||
ETCD_LISTEN_CLIENT_URLS=http://%[2]s:2379,http://127.0.0.1:2379
|
||||
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://%[2]s:2380
|
||||
ETCD_ADVERTISE_CLIENT_URLS=http://%[2]s:2379
|
||||
ETCD_LISTEN_PEER_URLS=%[5]s://%[2]s:2380
|
||||
ETCD_LISTEN_CLIENT_URLS=%[6]s
|
||||
ETCD_INITIAL_ADVERTISE_PEER_URLS=%[5]s://%[2]s:2380
|
||||
ETCD_ADVERTISE_CLIENT_URLS=%[5]s://%[2]s:2379
|
||||
ETCD_INITIAL_CLUSTER=%[3]s
|
||||
ETCD_INITIAL_CLUSTER_STATE=new
|
||||
ETCD_INITIAL_CLUSTER_TOKEN=%[4]s
|
||||
`, m.Name, m.IP, strings.Join(peers, ","), Scope+"-patroni")
|
||||
`, m.Name, m.IP, strings.Join(peers, ","), Scope+"-patroni", scheme, client)
|
||||
if tls {
|
||||
conf += `ETCD_CERT_FILE=` + EtcdCert + `
|
||||
ETCD_KEY_FILE=` + EtcdKey + `
|
||||
ETCD_TRUSTED_CA_FILE=` + EtcdCA + `
|
||||
ETCD_CLIENT_CERT_AUTH=true
|
||||
ETCD_PEER_CERT_FILE=` + EtcdCert + `
|
||||
ETCD_PEER_KEY_FILE=` + EtcdKey + `
|
||||
ETCD_PEER_TRUSTED_CA_FILE=` + EtcdCA + `
|
||||
ETCD_PEER_CLIENT_CERT_AUTH=true
|
||||
`
|
||||
}
|
||||
return conf
|
||||
}
|
||||
|
||||
// EtcdTLSDir holds an etcd member's certificate, which it presents both as
|
||||
// a server and as a client of the other members, and the authority's.
|
||||
const EtcdTLSDir = "/etc/etcd/tls"
|
||||
|
||||
// The files in EtcdTLSDir.
|
||||
const (
|
||||
EtcdCA = EtcdTLSDir + "/ca.crt"
|
||||
EtcdCert = EtcdTLSDir + "/peer.crt"
|
||||
EtcdKey = EtcdTLSDir + "/peer.key"
|
||||
)
|
||||
|
||||
// etcdctl is the command that asks the cluster, over TLS with a member's
|
||||
// certificate when tls; run on a member, as root, which reads it.
|
||||
func etcdctl(tls bool, endpoints []string) string {
|
||||
c := "sudo ETCDCTL_API=3 etcdctl --dial-timeout=3s --endpoints=" + strings.Join(endpoints, ",")
|
||||
if tls {
|
||||
c += " --cacert=" + EtcdCA + " --cert=" + EtcdCert + " --key=" + EtcdKey
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// Patroni is a database member's /etc/patroni/config.yml.
|
||||
@@ -66,7 +113,10 @@ ETCD_INITIAL_CLUSTER_TOKEN=%[4]s
|
||||
// clients are the addresses the control planes connect from; the members
|
||||
// themselves are always allowed, since a second control plane usually runs on
|
||||
// one of them.
|
||||
func Patroni(m Member, db, etcd []Member, clients []string, version string, s Secrets) string {
|
||||
//
|
||||
// With tls, PostgreSQL is given the member's certificate (certs puts it in
|
||||
// TLSDir) and every remote rule is hostssl.
|
||||
func Patroni(m Member, db, etcd []Member, clients []string, version string, s Secrets, tls bool) string {
|
||||
var hosts []string
|
||||
for _, e := range etcd {
|
||||
hosts = append(hosts, e.IP+":2379")
|
||||
@@ -88,8 +138,17 @@ func Patroni(m Member, db, etcd []Member, clients []string, version string, s Se
|
||||
}
|
||||
var hbaLines string
|
||||
for _, h := range hba {
|
||||
if tls {
|
||||
h = hostssl(h)
|
||||
}
|
||||
hbaLines += " - " + h + "\n"
|
||||
}
|
||||
var sslLines string
|
||||
if tls {
|
||||
for _, k := range []string{"ssl", "ssl_cert_file", "ssl_key_file", "ssl_ca_file"} {
|
||||
sslLines += fmt.Sprintf(" %s: %q\n", k, sslParameters()[k])
|
||||
}
|
||||
}
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, `# llmgw: the control planes' store, kept by Patroni, its leader lock in etcd.
|
||||
scope: %[1]s
|
||||
@@ -99,10 +158,15 @@ name: %[2]s
|
||||
restapi:
|
||||
listen: %[3]s:%[4]d
|
||||
connect_address: %[3]s:%[4]d
|
||||
|
||||
# Asked for before a change: anyone reaching this port could otherwise
|
||||
# rewrite the access rules or move the leader.
|
||||
authentication:
|
||||
username: %[11]s
|
||||
password: "%[10]s"
|
||||
%[13]s
|
||||
etcd3:
|
||||
hosts: %[5]s
|
||||
|
||||
%[14]s
|
||||
bootstrap:
|
||||
dcs:
|
||||
ttl: 30
|
||||
@@ -120,7 +184,7 @@ bootstrap:
|
||||
wal_keep_size: 1GB
|
||||
wal_log_hints: "on"
|
||||
password_encryption: scram-sha-256
|
||||
# In the cluster's shared configuration rather than each member's, so a
|
||||
%[12]s # In the cluster's shared configuration rather than each member's, so a
|
||||
# control plane added later is let in by one change through Patroni's
|
||||
# API (llmgw server add) instead of an edit on every member.
|
||||
pg_hba:
|
||||
@@ -146,7 +210,8 @@ postgresql:
|
||||
password: "%[7]s"
|
||||
parameters:
|
||||
unix_socket_directories: /var/run/postgresql
|
||||
`, Scope, m.Name, m.IP, PatroniPort, strings.Join(hosts, ","), version, s.Superuser, s.Replicator, hbaLines)
|
||||
`, Scope, m.Name, m.IP, PatroniPort, strings.Join(hosts, ","), version, s.Superuser, s.Replicator, hbaLines,
|
||||
s.RestAPI, RestUser, sslLines, restTLSLines(tls), etcdTLSLines(tls))
|
||||
b.WriteString(`
|
||||
tags:
|
||||
nofailover: false
|
||||
@@ -157,16 +222,41 @@ tags:
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// restTLSLines put Patroni's API on TLS, with the member's certificate, and
|
||||
// have patronictl verify it.
|
||||
func restTLSLines(tls bool) string {
|
||||
if !tls {
|
||||
return ""
|
||||
}
|
||||
return " certfile: " + ServerCert + "\n keyfile: " + ServerKey + "\n cafile: " + CACert + "\n" +
|
||||
"\nctl:\n cacert: " + CACert + "\n"
|
||||
}
|
||||
|
||||
// etcdTLSLines have Patroni reach etcd over TLS, presenting the member's
|
||||
// certificate.
|
||||
func etcdTLSLines(tls bool) string {
|
||||
if !tls {
|
||||
return ""
|
||||
}
|
||||
return " protocol: https\n cacert: " + CACert + "\n cert: " + ServerCert + "\n key: " + ServerKey + "\n"
|
||||
}
|
||||
|
||||
// DSN is the store address the control planes are given: every member, and
|
||||
// the driver told to use whichever accepts writes, so a failover is followed
|
||||
// without anything in front of the database.
|
||||
func DSN(db []Member, password string) string {
|
||||
// without anything in front of the database. With caFile — the cluster's
|
||||
// authority, on the control plane — the connection is TLS and the member it
|
||||
// reaches is verified against it.
|
||||
func DSN(db []Member, password, caFile string) string {
|
||||
var hosts []string
|
||||
for _, d := range db {
|
||||
hosts = append(hosts, d.IP+":5432")
|
||||
}
|
||||
return fmt.Sprintf("postgres://%s:%s@%s/%s?sslmode=disable&target_session_attrs=read-write",
|
||||
Role, url.QueryEscape(password), strings.Join(hosts, ","), Database)
|
||||
ssl := "sslmode=disable"
|
||||
if caFile != "" {
|
||||
ssl = "sslmode=verify-full&sslrootcert=" + url.QueryEscape(caFile)
|
||||
}
|
||||
return fmt.Sprintf("postgres://%s:%s@%s/%s?%s&target_session_attrs=read-write",
|
||||
Role, url.QueryEscape(password), strings.Join(hosts, ","), Database, ssl)
|
||||
}
|
||||
|
||||
func ips(ms []Member) []string {
|
||||
|
||||
@@ -0,0 +1,272 @@
|
||||
package pgha
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Hardening: what a cluster set up before this lacked, applied to it in
|
||||
// place, and to a new one from the start.
|
||||
//
|
||||
// - Patroni's REST API asks for a password before a change. Without one,
|
||||
// anyone who reached port 8008 could rewrite pg_hba through PATCH /config
|
||||
// — `host all all 0.0.0.0/0 trust` — or switch the leader over.
|
||||
// - PostgreSQL speaks TLS, with a certificate from the cluster's authority
|
||||
// on each member, and every remote rule is hostssl: prompts kept in
|
||||
// threads, key and session hashes, and the engines' secret no longer
|
||||
// cross the network in clear. The control planes verify the member they
|
||||
// reach (sslmode=verify-full against the cluster's CA).
|
||||
//
|
||||
// Each step looks first and changes only what differs, so running it on a
|
||||
// hardened cluster does nothing, and each is applied by reload, not restart:
|
||||
// the cluster keeps serving.
|
||||
|
||||
// RestUser is the user Patroni's REST API asks for before a change.
|
||||
const RestUser = "patroni"
|
||||
|
||||
// TLSDir holds each member's certificate, key and the cluster's CA.
|
||||
const TLSDir = "/etc/patroni/tls"
|
||||
|
||||
// ServerCert, ServerKey and CACert are the files in TLSDir.
|
||||
const (
|
||||
ServerCert = TLSDir + "/server.crt"
|
||||
ServerKey = TLSDir + "/server.key"
|
||||
CACert = TLSDir + "/ca.crt"
|
||||
)
|
||||
|
||||
// sslParameters are what turn TLS on, in the cluster's shared parameters.
|
||||
func sslParameters() map[string]any {
|
||||
return map[string]any{"ssl": "on", "ssl_cert_file": ServerCert, "ssl_key_file": ServerKey, "ssl_ca_file": CACert}
|
||||
}
|
||||
|
||||
// hostssl rewrites a remote rule to accept only TLS; local and loopback
|
||||
// rules are left alone.
|
||||
func hostssl(line string) string {
|
||||
f := strings.Fields(line)
|
||||
if len(f) < 4 || f[0] != "host" || strings.HasPrefix(f[3], "127.") || f[3] == "::1/128" {
|
||||
return line
|
||||
}
|
||||
return "hostssl" + strings.TrimPrefix(line, "host")
|
||||
}
|
||||
|
||||
// certs gives each database member a certificate from the cluster's
|
||||
// authority, for its address and name — unless it has one for them already.
|
||||
// It serves PostgreSQL and Patroni's API, and is what Patroni presents to
|
||||
// etcd, so it identifies a client too. A member without PostgreSQL's user
|
||||
// yet is skipped: the Patroni step calls this again once it is installed.
|
||||
func (s *Setup) certs(ctx context.Context) error {
|
||||
if s.CA == nil {
|
||||
s.emit("tls", false, "no certificate authority on this machine: PostgreSQL stays without TLS")
|
||||
return nil
|
||||
}
|
||||
for _, d := range s.DB {
|
||||
have, _ := d.Remote.Run(ctx, "id postgres >/dev/null 2>&1 || echo no-postgres; "+
|
||||
"sudo openssl x509 -in "+ServerCert+" -noout -ext subjectAltName,extendedKeyUsage 2>/dev/null", nil)
|
||||
if strings.Contains(string(have), "no-postgres") {
|
||||
continue
|
||||
}
|
||||
if strings.Contains(string(have), "IP Address:"+d.IP) && strings.Contains(string(have), "Client Authentication") {
|
||||
continue
|
||||
}
|
||||
leaf, err := s.CA.IssuePeer(d.Name, []string{d.IP, d.Name, "127.0.0.1", "localhost"})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := putTLS(ctx, d, "postgres", TLSDir, CACert, ServerCert, ServerKey, s.CA.PEM, leaf.CertPEM, leaf.KeyPEM); err != nil {
|
||||
return err
|
||||
}
|
||||
s.emit("tls", true, "certificate for %s (%s)", d.Name, d.IP)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// restAuth makes each member's Patroni ask for the password before a change,
|
||||
// and reloads it where that was not so.
|
||||
func (s *Setup) restAuth(ctx context.Context, sec Secrets) error {
|
||||
for _, d := range s.DB {
|
||||
out, err := d.Remote.Run(ctx, `set -e
|
||||
test -f /etc/patroni/config.yml || { echo missing; exit 0; }
|
||||
# The password from stdin into a file only this user reads: the program
|
||||
# itself comes in on python's stdin.
|
||||
f=$(mktemp); trap 'rm -f "$f"' EXIT
|
||||
cat > "$f"
|
||||
r=$(sudo python3 - "$f" <<'PY'
|
||||
import sys, yaml
|
||||
pw = open(sys.argv[1]).read().strip()
|
||||
p = "/etc/patroni/config.yml"
|
||||
c = yaml.safe_load(open(p))
|
||||
want = {"username": "`+RestUser+`", "password": pw}
|
||||
api = c.setdefault("restapi", {})
|
||||
if api.get("authentication") == want:
|
||||
print("same")
|
||||
else:
|
||||
api["authentication"] = want
|
||||
open(p + ".new", "w").write("# llmgw: the control planes' store, kept by Patroni, its leader lock in etcd.\n" +
|
||||
yaml.safe_dump(c, default_flow_style=False, sort_keys=False))
|
||||
print("changed")
|
||||
PY
|
||||
)
|
||||
if [ "$r" = changed ]; then
|
||||
sudo chown root:postgres /etc/patroni/config.yml.new
|
||||
sudo chmod 640 /etc/patroni/config.yml.new
|
||||
sudo mv -f /etc/patroni/config.yml.new /etc/patroni/config.yml
|
||||
sudo systemctl reload patroni
|
||||
fi
|
||||
echo "$r"`, []byte(sec.RestAPI+"\n"))
|
||||
if err != nil {
|
||||
return fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
switch strings.TrimSpace(lastLine(out)) {
|
||||
case "changed":
|
||||
s.emit("patroni api", true, "%s asks for a password before a change", d.Name)
|
||||
case "missing":
|
||||
return fmt.Errorf("%s has no /etc/patroni/config.yml", d.Name)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// tlsInShared turns TLS on in the cluster's shared configuration, and every
|
||||
// remote access rule to hostssl, through Patroni's API — which reloads
|
||||
// PostgreSQL on every member.
|
||||
func (s *Setup) tlsInShared(ctx context.Context, sec Secrets) error {
|
||||
if s.CA == nil {
|
||||
return nil
|
||||
}
|
||||
hosts := make([]string, len(s.DB))
|
||||
for i, d := range s.DB {
|
||||
hosts[i] = d.IP
|
||||
}
|
||||
var lastErr error
|
||||
for _, h := range hosts {
|
||||
cfg, err := getConfig(ctx, h)
|
||||
if err != nil {
|
||||
lastErr = err
|
||||
continue
|
||||
}
|
||||
pg, _ := cfg["postgresql"].(map[string]any)
|
||||
params, _ := pg["parameters"].(map[string]any)
|
||||
var hba, next []string
|
||||
if list, ok := pg["pg_hba"].([]any); ok {
|
||||
for _, l := range list {
|
||||
if line, ok := l.(string); ok {
|
||||
hba = append(hba, line)
|
||||
next = append(next, hostssl(line))
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(hba) == 0 {
|
||||
// Kept in each member's own file, by a cluster set up before
|
||||
// they moved here: made hostssl there.
|
||||
if err := s.memberHostssl(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
want := sslParameters()
|
||||
same := slices.Equal(hba, next)
|
||||
for k, v := range want {
|
||||
if params[k] != v {
|
||||
same = false
|
||||
}
|
||||
}
|
||||
if same {
|
||||
s.emit("tls", false, "PostgreSQL speaks TLS, and remote rules are hostssl")
|
||||
return nil
|
||||
}
|
||||
patch := map[string]any{"parameters": want}
|
||||
if len(hba) > 0 {
|
||||
patch["pg_hba"] = next
|
||||
}
|
||||
if err := patchConfig(ctx, h, sec.RestAPI, map[string]any{"postgresql": patch}); err != nil {
|
||||
return fmt.Errorf("Patroni on %s: %w", h, err)
|
||||
}
|
||||
s.emit("tls", true, "PostgreSQL speaks TLS on every member; remote rules are hostssl")
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("no member's Patroni answered: %w", lastErr)
|
||||
}
|
||||
|
||||
// memberHostssl makes the remote rules in each member's own
|
||||
// /etc/patroni/config.yml hostssl, and reloads Patroni there, which rewrites
|
||||
// pg_hba.conf and reloads PostgreSQL without restarting it.
|
||||
func (s *Setup) memberHostssl(ctx context.Context) error {
|
||||
for _, d := range s.DB {
|
||||
out, err := d.Remote.Run(ctx, `set -e
|
||||
test -f /etc/patroni/config.yml || { echo missing; exit 0; }
|
||||
r=$(sudo python3 - <<'PY'
|
||||
import yaml
|
||||
p = "/etc/patroni/config.yml"
|
||||
c = yaml.safe_load(open(p))
|
||||
rules = c.get("postgresql", {}).get("pg_hba") or []
|
||||
def hostssl(line):
|
||||
f = line.split()
|
||||
if len(f) < 4 or f[0] != "host" or f[3].startswith("127.") or f[3] == "::1/128":
|
||||
return line
|
||||
return "hostssl" + line[len("host"):]
|
||||
new = [hostssl(l) for l in rules]
|
||||
if new == rules:
|
||||
print("same")
|
||||
else:
|
||||
c["postgresql"]["pg_hba"] = new
|
||||
open(p + ".new", "w").write("# llmgw: the control planes' store, kept by Patroni, its leader lock in etcd.\n" +
|
||||
yaml.safe_dump(c, default_flow_style=False, sort_keys=False))
|
||||
print("changed")
|
||||
PY
|
||||
)
|
||||
if [ "$r" = changed ]; then
|
||||
sudo chown root:postgres /etc/patroni/config.yml.new
|
||||
sudo chmod 640 /etc/patroni/config.yml.new
|
||||
sudo mv -f /etc/patroni/config.yml.new /etc/patroni/config.yml
|
||||
sudo systemctl reload patroni
|
||||
fi
|
||||
echo "$r"`, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
if strings.TrimSpace(lastLine(out)) == "changed" {
|
||||
s.emit("tls", true, "%s's own access rules are hostssl", d.Name)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// patchConfig changes the cluster's shared configuration, with the REST
|
||||
// password.
|
||||
func patchConfig(ctx context.Context, host, password string, body map[string]any) error {
|
||||
b, _ := json.Marshal(body)
|
||||
resp, err := restRequest(ctx, host, http.MethodPatch, "/config", b, password, 15*time.Second)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("refused the change: %s", resp.Status)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// sharedTLS reports whether the cluster's shared configuration has TLS on:
|
||||
// for a control plane without the authority's key, which signs nothing but
|
||||
// must still connect by TLS once the cluster asks for it.
|
||||
func (s *Setup) sharedTLS(ctx context.Context) bool {
|
||||
for _, d := range s.DB {
|
||||
cfg, err := getConfig(ctx, d.IP)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
pg, _ := cfg["postgresql"].(map[string]any)
|
||||
params, _ := pg["parameters"].(map[string]any)
|
||||
return params["ssl"] == "on"
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func lastLine(b []byte) string {
|
||||
lines := strings.Split(strings.TrimSpace(string(b)), "\n")
|
||||
return lines[len(lines)-1]
|
||||
}
|
||||
+16
-30
@@ -1,14 +1,12 @@
|
||||
package pgha
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/http"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
@@ -24,12 +22,13 @@ func ClientLine(ip string) string {
|
||||
// A cluster set up before the rules moved into the shared configuration
|
||||
// keeps them in each member's own file, which this cannot reach; that is
|
||||
// said, with what to add, rather than half done.
|
||||
func AllowClient(ctx context.Context, hosts []string, ip string) (bool, error) {
|
||||
client := &http.Client{Timeout: 5 * time.Second}
|
||||
//
|
||||
// password is what Patroni's API asks for before a change (RestUser); a
|
||||
// cluster whose rules are hostssl lets the new control plane in by TLS too.
|
||||
func AllowClient(ctx context.Context, hosts []string, ip, password string) (bool, error) {
|
||||
var lastErr error
|
||||
for _, h := range hosts {
|
||||
base := "http://" + net.JoinHostPort(h, strconv.Itoa(restPort))
|
||||
cfg, err := getConfig(ctx, client, base)
|
||||
cfg, err := getConfig(ctx, h)
|
||||
if err != nil {
|
||||
lastErr = err
|
||||
continue
|
||||
@@ -50,42 +49,29 @@ func AllowClient(ctx context.Context, hosts []string, ip string) (bool, error) {
|
||||
"on every member, and run `patronictl reload llmgw`", ClientLine(ip))
|
||||
}
|
||||
line := ClientLine(ip)
|
||||
if slices.ContainsFunc(hba, func(l string) bool { return strings.HasPrefix(l, "hostssl ") }) {
|
||||
line = hostssl(line)
|
||||
}
|
||||
if slices.Contains(hba, line) {
|
||||
return false, nil
|
||||
}
|
||||
body, _ := json.Marshal(map[string]any{"postgresql": map[string]any{"pg_hba": append(hba, line)}})
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPatch, base+"/config", bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
lastErr = err
|
||||
continue
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return false, fmt.Errorf("Patroni on %s refused the change: %s", h, resp.Status)
|
||||
if err := patchConfig(ctx, h, password,
|
||||
map[string]any{"postgresql": map[string]any{"pg_hba": append(hba, line)}}); err != nil {
|
||||
return false, fmt.Errorf("Patroni on %s: %w", h, err)
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
return false, fmt.Errorf("no member's Patroni answered: %w", lastErr)
|
||||
}
|
||||
|
||||
func getConfig(ctx context.Context, client *http.Client, base string) (map[string]any, error) {
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, base+"/config", nil)
|
||||
func getConfig(ctx context.Context, host string) (map[string]any, error) {
|
||||
b, err := restJSON(ctx, host, http.MethodGet, "/config", nil, "", 10*time.Second)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
var cfg map[string]any
|
||||
if err := json.NewDecoder(resp.Body).Decode(&cfg); err != nil {
|
||||
return nil, fmt.Errorf("%s/config: %w", base, err)
|
||||
if err := json.Unmarshal(b, &cfg); err != nil {
|
||||
return nil, fmt.Errorf("%s /config: %w", host, err)
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
+190
-10
@@ -12,6 +12,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
"github.com/prometheus/client_golang/prometheus/testutil"
|
||||
"github.com/prometheus/common/expfmt"
|
||||
"go.yaml.in/yaml/v3"
|
||||
@@ -24,8 +25,8 @@ var (
|
||||
)
|
||||
|
||||
func TestTheConfigurationsNameEveryMemberAndOnlyTheClientsGiven(t *testing.T) {
|
||||
sec := Secrets{"su", "rep", "app"}
|
||||
cfg := Patroni(pg01, []Member{pg01, ctl02}, []Member{pg01, ctl02, ctl}, []string{"10.0.0.140"}, "16", sec)
|
||||
sec := Secrets{"su", "rep", "app", "api"}
|
||||
cfg := Patroni(pg01, []Member{pg01, ctl02}, []Member{pg01, ctl02, ctl}, []string{"10.0.0.140"}, "16", sec, false)
|
||||
for _, want := range []string{
|
||||
"name: llmgw-pg01", "listen: 10.0.0.149:8008",
|
||||
"hosts: 10.0.0.149:2379,10.0.0.150:2379,10.0.0.140:2379",
|
||||
@@ -42,15 +43,34 @@ func TestTheConfigurationsNameEveryMemberAndOnlyTheClientsGiven(t *testing.T) {
|
||||
if strings.Count(cfg, "host llmgw llmgw 10.0.0.149/32") != 1 {
|
||||
t.Error("a member's address is allowed twice")
|
||||
}
|
||||
e := Etcd(ctl, []Member{pg01, ctl02, ctl})
|
||||
e := Etcd(ctl, []Member{pg01, ctl02, ctl}, false)
|
||||
if !strings.Contains(e, "ETCD_NAME=llmgw-ctl\n") ||
|
||||
!strings.Contains(e, "llmgw-pg01=http://10.0.0.149:2380,llmgw-ctl02=http://10.0.0.150:2380,llmgw-ctl=http://10.0.0.140:2380") {
|
||||
t.Errorf("etcd defaults:\n%s", e)
|
||||
}
|
||||
if got := DSN([]Member{pg01, ctl02}, "pw"); got !=
|
||||
e = Etcd(ctl, []Member{pg01, ctl02, ctl}, true)
|
||||
for _, want := range []string{
|
||||
"llmgw-pg01=https://10.0.0.149:2380",
|
||||
"ETCD_LISTEN_CLIENT_URLS=https://10.0.0.140:2379,https://127.0.0.1:2379\n",
|
||||
"ETCD_ADVERTISE_CLIENT_URLS=https://10.0.0.140:2379\n",
|
||||
"ETCD_CLIENT_CERT_AUTH=true\n", "ETCD_PEER_CLIENT_CERT_AUTH=true\n",
|
||||
"ETCD_TRUSTED_CA_FILE=/etc/etcd/tls/ca.crt\n",
|
||||
} {
|
||||
if !strings.Contains(e, want) {
|
||||
t.Errorf("etcd over TLS lacks %q:\n%s", want, e)
|
||||
}
|
||||
}
|
||||
if strings.Contains(e, "http://") {
|
||||
t.Errorf("etcd over TLS still listens in the clear:\n%s", e)
|
||||
}
|
||||
if got := DSN([]Member{pg01, ctl02}, "pw", ""); got !=
|
||||
"postgres://llmgw:pw@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=disable&target_session_attrs=read-write" {
|
||||
t.Errorf("dsn %s", got)
|
||||
}
|
||||
if got := DSN([]Member{pg01, ctl02}, "pw", "/etc/llmgw/ca.pem"); got !=
|
||||
"postgres://llmgw:pw@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=verify-full&sslrootcert=%2Fetc%2Fllmgw%2Fca.pem&target_session_attrs=read-write" {
|
||||
t.Errorf("dsn with tls %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAClusterIsCompleteWithALeaderAndEveryOtherMemberStreaming(t *testing.T) {
|
||||
@@ -102,7 +122,7 @@ func (m *machine) ranAny(frag string) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
const pws = "superuser=a\nreplicator=b\nllmgw=c\n"
|
||||
const pws = "superuser=a\nreplicator=b\nllmgw=c\nrestapi=d\n"
|
||||
|
||||
func healthy(n int) string {
|
||||
return strings.Repeat("http://x:2379 is healthy: successfully committed proposal\n", n)
|
||||
@@ -129,7 +149,8 @@ func TestARunningClusterIsLeftAsItIs(t *testing.T) {
|
||||
t.Errorf("dsn %s", dsn)
|
||||
}
|
||||
for _, m := range []*machine{a, b, w} {
|
||||
for _, frag := range []string{"apt-get", "systemctl restart", "/etc/default/etcd", "/etc/patroni/config.yml"} {
|
||||
// Looked at, for the API's password, but not rewritten or restarted.
|
||||
for _, frag := range []string{"apt-get", "systemctl restart", "/etc/default/etcd", "config.yml.new\nsudo systemctl"} {
|
||||
if m.ranAny(frag) {
|
||||
t.Errorf("%s was changed (%q) though the cluster is up", m.name, frag)
|
||||
}
|
||||
@@ -216,7 +237,7 @@ func TestAMemberThatStoppedIsReportedMissingNotForgotten(t *testing.T) {
|
||||
|
||||
func TestTheAccessRulesAreInTheClustersSharedConfiguration(t *testing.T) {
|
||||
// Where `server add` can change them for every member at once.
|
||||
cfg := Patroni(pg01, []Member{pg01, ctl02}, []Member{pg01, ctl02, ctl}, []string{"10.0.0.140"}, "16", Secrets{"a", "b", "c"})
|
||||
cfg := Patroni(pg01, []Member{pg01, ctl02}, []Member{pg01, ctl02, ctl}, []string{"10.0.0.140"}, "16", Secrets{"a", "b", "c", "d"}, false)
|
||||
var doc struct {
|
||||
Bootstrap struct {
|
||||
DCS struct {
|
||||
@@ -262,7 +283,7 @@ func TestAControlPlaneIsLetInThroughTheSharedConfiguration(t *testing.T) {
|
||||
restPort, _ = strconv.Atoi(u.Port())
|
||||
defer func() { restPort = old }()
|
||||
|
||||
changed, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.150")
|
||||
changed, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.150", "")
|
||||
if err != nil || !changed {
|
||||
t.Fatalf("%v %v", changed, err)
|
||||
}
|
||||
@@ -270,13 +291,172 @@ func TestAControlPlaneIsLetInThroughTheSharedConfiguration(t *testing.T) {
|
||||
if len(patched) != 3 || patched[2] != ClientLine("10.0.0.150") || patched[1] != ClientLine("10.0.0.140") {
|
||||
t.Errorf("patched %q", patched)
|
||||
}
|
||||
if changed, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.140"); err != nil || changed {
|
||||
if changed, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.140", ""); err != nil || changed {
|
||||
t.Errorf("one already let in was changed again: %v %v", changed, err)
|
||||
}
|
||||
|
||||
hba = nil // the rules are in each member's file
|
||||
if _, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.151"); err == nil ||
|
||||
if _, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.151", ""); err == nil ||
|
||||
!strings.Contains(err.Error(), "each member") {
|
||||
t.Errorf("rules kept per member: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANewClusterAsksForAPasswordAndSpeaksTLS(t *testing.T) {
|
||||
cfg := Patroni(pg01, []Member{pg01, ctl02}, []Member{pg01, ctl02, ctl}, []string{"10.0.0.140"}, "16",
|
||||
Secrets{"a", "b", "c", "the-api-pw"}, true)
|
||||
var doc struct {
|
||||
RestAPI struct {
|
||||
Authentication struct{ Username, Password string } `yaml:"authentication"`
|
||||
CertFile string `yaml:"certfile"`
|
||||
CAFile string `yaml:"cafile"`
|
||||
} `yaml:"restapi"`
|
||||
Ctl struct {
|
||||
CACert string `yaml:"cacert"`
|
||||
} `yaml:"ctl"`
|
||||
Etcd3 struct {
|
||||
Protocol, CACert, Cert, Key string
|
||||
} `yaml:"etcd3"`
|
||||
Bootstrap struct {
|
||||
DCS struct {
|
||||
PostgreSQL struct {
|
||||
Parameters map[string]string `yaml:"parameters"`
|
||||
PgHBA []string `yaml:"pg_hba"`
|
||||
} `yaml:"postgresql"`
|
||||
} `yaml:"dcs"`
|
||||
} `yaml:"bootstrap"`
|
||||
}
|
||||
if err := yaml.Unmarshal([]byte(cfg), &doc); err != nil {
|
||||
t.Fatalf("%v\n%s", err, cfg)
|
||||
}
|
||||
if a := doc.RestAPI.Authentication; a.Username != RestUser || a.Password != "the-api-pw" {
|
||||
t.Errorf("rest api authentication %+v", a)
|
||||
}
|
||||
if doc.RestAPI.CertFile != ServerCert || doc.RestAPI.CAFile != CACert || doc.Ctl.CACert != CACert {
|
||||
t.Errorf("the API is not served over TLS:\n%s", cfg)
|
||||
}
|
||||
if e := doc.Etcd3; e.Protocol != "https" || e.CACert != CACert || e.Cert != ServerCert || e.Key != ServerKey {
|
||||
t.Errorf("etcd is not reached over TLS: %+v", e)
|
||||
}
|
||||
p := doc.Bootstrap.DCS.PostgreSQL.Parameters
|
||||
if p["ssl"] != "on" || p["ssl_cert_file"] != ServerCert || p["ssl_key_file"] != ServerKey {
|
||||
t.Errorf("parameters %v", p)
|
||||
}
|
||||
for _, l := range doc.Bootstrap.DCS.PostgreSQL.PgHBA {
|
||||
if strings.HasPrefix(l, "host ") && !strings.Contains(l, " 127.0.0.1/32 ") {
|
||||
t.Errorf("a remote rule without TLS: %q", l)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyRemoteRulesAreMadeHostssl(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
"host llmgw llmgw 10.0.0.140/32 scram-sha-256": "hostssl llmgw llmgw 10.0.0.140/32 scram-sha-256",
|
||||
"host replication replicator 10.0.0.150/32 scram-sha-256": "hostssl replication replicator 10.0.0.150/32 scram-sha-256",
|
||||
"host all all 127.0.0.1/32 scram-sha-256": "host all all 127.0.0.1/32 scram-sha-256",
|
||||
"local all all peer": "local all all peer",
|
||||
"hostssl llmgw llmgw 10.0.0.1/32 scram-sha-256": "hostssl llmgw llmgw 10.0.0.1/32 scram-sha-256",
|
||||
} {
|
||||
if got := hostssl(in); got != want {
|
||||
t.Errorf("%q: %q", in, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAClusterFromBeforeTheAPIPasswordIsGivenOne(t *testing.T) {
|
||||
// The second member holds the three PostgreSQL passwords and no API one.
|
||||
a := &machine{answers: map[string]string{"openssl rand": pws, "endpoint health": healthy(3)}}
|
||||
b := &machine{answers: map[string]string{"sudo cat /root/llmgw-pg": "superuser=a\nreplicator=b\nllmgw=c\n"}}
|
||||
up := Cluster{Members: []MemberState{{Name: "llmgw-pg01", Role: "leader", State: "running"},
|
||||
{Name: "llmgw-ctl02", Role: "replica", State: "streaming"}}}
|
||||
s := Setup{DB: []Node{{pg01, a}, {ctl02, b}}, Witness: &Node{ctl, &machine{}},
|
||||
Status: func(context.Context, []string) (Cluster, error) { return up, nil }}
|
||||
if _, err := s.Run(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !b.ranAny(`"restapi $api"`) || s.RestPassword != "d" {
|
||||
t.Errorf("the API password was not carried: %q", s.RestPassword)
|
||||
}
|
||||
for _, m := range []*machine{a, b} {
|
||||
if !m.ranAny(`"username": "patroni"`) {
|
||||
t.Errorf("%s: the API was not given its password", m.name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTLSIsTurnedOnInTheSharedConfigurationWithThePassword(t *testing.T) {
|
||||
var patched map[string]any
|
||||
var user, pass string
|
||||
hba := []any{"local all all peer", "host all all 127.0.0.1/32 scram-sha-256", ClientLine("10.0.0.140")}
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.Method {
|
||||
case http.MethodGet:
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"postgresql": map[string]any{"pg_hba": hba, "parameters": map[string]any{}}})
|
||||
case http.MethodPatch:
|
||||
user, pass, _ = r.BasicAuth()
|
||||
_ = json.NewDecoder(r.Body).Decode(&patched)
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
u, _ := url.Parse(srv.URL)
|
||||
old := restPort
|
||||
restPort, _ = strconv.Atoi(u.Port())
|
||||
defer func() { restPort = old }()
|
||||
|
||||
ca, err := tlsconf.NewCA("test")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := Setup{DB: []Node{{Member{"m", "127.0.0.1"}, &machine{}}}, CA: ca}
|
||||
if err := s.tlsInShared(context.Background(), Secrets{RestAPI: "pw"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if user != RestUser || pass != "pw" {
|
||||
t.Errorf("auth %q %q", user, pass)
|
||||
}
|
||||
pg := patched["postgresql"].(map[string]any)
|
||||
if pg["parameters"].(map[string]any)["ssl"] != "on" {
|
||||
t.Errorf("parameters %v", pg["parameters"])
|
||||
}
|
||||
got := pg["pg_hba"].([]any)
|
||||
if got[1] != "host all all 127.0.0.1/32 scram-sha-256" || got[2] != hostssl(ClientLine("10.0.0.140")) {
|
||||
t.Errorf("hba %v", got)
|
||||
}
|
||||
// A control plane let in afterwards is let in by TLS.
|
||||
hba = got
|
||||
if _, err := AllowClient(context.Background(), []string{"127.0.0.1"}, "10.0.0.150", "pw"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
last := patched["postgresql"].(map[string]any)["pg_hba"].([]any)
|
||||
if last[len(last)-1] != hostssl(ClientLine("10.0.0.150")) {
|
||||
t.Errorf("let in without TLS: %v", last)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAControlPlaneWithoutTheAuthorityConnectsByTLSOnceTheClusterHasIt(t *testing.T) {
|
||||
ssl := "on"
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"postgresql": map[string]any{"parameters": map[string]any{"ssl": ssl}}})
|
||||
}))
|
||||
defer srv.Close()
|
||||
u, _ := url.Parse(srv.URL)
|
||||
old := restPort
|
||||
restPort, _ = strconv.Atoi(u.Port())
|
||||
defer func() { restPort = old }()
|
||||
|
||||
a := &machine{answers: map[string]string{"openssl rand": pws, "endpoint health": healthy(3)}}
|
||||
b := &machine{answers: map[string]string{"sudo cat /root/llmgw-pg": pws}}
|
||||
up := Cluster{Members: []MemberState{{Name: "llmgw-pg01", Role: "leader", State: "running"},
|
||||
{Name: "llmgw-ctl02", Role: "replica", State: "streaming"}}}
|
||||
s := Setup{DB: []Node{{Member{"llmgw-pg01", "127.0.0.1"}, a}, {Member{"llmgw-ctl02", "127.0.0.1"}, b}},
|
||||
Witness: &Node{ctl, &machine{}}, CAFile: "/etc/llmgw/ca.pem",
|
||||
Status: func(context.Context, []string) (Cluster, error) { return up, nil }}
|
||||
dsn, err := s.Run(context.Background())
|
||||
if err != nil || !strings.Contains(dsn, "sslmode=verify-full") {
|
||||
t.Fatalf("%s %v", dsn, err)
|
||||
}
|
||||
ssl = "off"
|
||||
if dsn, _ = s.Run(context.Background()); !strings.Contains(dsn, "sslmode=disable") {
|
||||
t.Errorf("TLS asked of a cluster without it: %s", dsn)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
package pgha
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Patroni's API answers over TLS once a member has its certificate, and
|
||||
// over plain HTTP on a cluster set up before that, or while it is being
|
||||
// moved. So every request is made over HTTPS first, verified against the
|
||||
// cluster's authority, and over HTTP only when the member answers in plain
|
||||
// HTTP — never when the certificate is wrong.
|
||||
|
||||
var (
|
||||
restMu sync.RWMutex
|
||||
restCA *x509.CertPool
|
||||
)
|
||||
|
||||
// SetRESTCA is the cluster's authority, against which members' API
|
||||
// certificates are verified: the control plane's ca.pem. Without one, the
|
||||
// API is asked over plain HTTP.
|
||||
func SetRESTCA(pemFile string) error {
|
||||
b, err := os.ReadFile(pemFile) //nolint:gosec // the operator's own CA
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return setRESTCAPEM(b)
|
||||
}
|
||||
|
||||
func setRESTCAPEM(b []byte) error {
|
||||
pool := x509.NewCertPool()
|
||||
if !pool.AppendCertsFromPEM(b) {
|
||||
return errors.New("no certificate in the authority's file")
|
||||
}
|
||||
restMu.Lock()
|
||||
restCA = pool
|
||||
restMu.Unlock()
|
||||
return nil
|
||||
}
|
||||
|
||||
// restRequest asks one member's API.
|
||||
func restRequest(ctx context.Context, host, method, path string, body []byte, password string,
|
||||
timeout time.Duration,
|
||||
) (*http.Response, error) {
|
||||
restMu.RLock()
|
||||
pool := restCA
|
||||
restMu.RUnlock()
|
||||
addr := net.JoinHostPort(host, strconv.Itoa(restPort))
|
||||
try := func(scheme string, client *http.Client) (*http.Response, error) {
|
||||
var r io.Reader
|
||||
if body != nil {
|
||||
r = bytes.NewReader(body)
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, method, scheme+"://"+addr+path, r)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if body != nil {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
if password != "" {
|
||||
req.SetBasicAuth(RestUser, password)
|
||||
}
|
||||
return client.Do(req)
|
||||
}
|
||||
plain := &http.Client{Timeout: timeout}
|
||||
if pool == nil {
|
||||
return try("http", plain)
|
||||
}
|
||||
secure := &http.Client{Timeout: timeout, Transport: &http.Transport{
|
||||
TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12},
|
||||
}}
|
||||
resp, err := try("https", secure)
|
||||
if err != nil && speaksPlainHTTP(err) {
|
||||
return try("http", plain)
|
||||
}
|
||||
return resp, err
|
||||
}
|
||||
|
||||
// speaksPlainHTTP reports whether a TLS attempt failed because the other
|
||||
// end answered in plain HTTP: a member whose API has no certificate yet.
|
||||
func speaksPlainHTTP(err error) bool {
|
||||
var rh tls.RecordHeaderError
|
||||
return errors.As(err, &rh) || strings.Contains(err.Error(), "server gave HTTP response to HTTPS client")
|
||||
}
|
||||
|
||||
// restJSON is restRequest for an answer that must be 200, read whole.
|
||||
func restJSON(ctx context.Context, host, method, path string, body []byte, password string,
|
||||
timeout time.Duration,
|
||||
) ([]byte, error) {
|
||||
resp, err := restRequest(ctx, host, method, path, body, password, timeout)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return b, fmt.Errorf("%s %s on %s: %s", method, path, host, resp.Status)
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
package pgha
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"encoding/pem"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strconv"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
func TestTheAPIIsAskedOverTLSAndOverHTTPOnlyWhenItSpeaksHTTP(t *testing.T) {
|
||||
answer := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"scheme":"` + map[bool]string{true: "https", false: "http"}[r.TLS != nil] + `"}`))
|
||||
})
|
||||
secure := httptest.NewTLSServer(answer)
|
||||
defer secure.Close()
|
||||
plain := httptest.NewServer(answer)
|
||||
defer plain.Close()
|
||||
t.Cleanup(func() { restMu.Lock(); restCA = nil; restMu.Unlock(); restPort = PatroniPort })
|
||||
|
||||
if err := setRESTCAPEM(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: secure.Certificate().Raw})); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ask := func(url string) (string, error) {
|
||||
_, port, _ := net.SplitHostPort(url[len("https://"):])
|
||||
if url[:5] == "http:" {
|
||||
_, port, _ = net.SplitHostPort(url[len("http://"):])
|
||||
}
|
||||
restPort, _ = strconv.Atoi(port)
|
||||
b, err := restJSON(t.Context(), "127.0.0.1", http.MethodGet, "/x", nil, "", 3*time.Second)
|
||||
return string(b), err
|
||||
}
|
||||
if got, err := ask(secure.URL); err != nil || got != `{"scheme":"https"}` {
|
||||
t.Errorf("a member on TLS: %s, %v", got, err)
|
||||
}
|
||||
if got, err := ask(plain.URL); err != nil || got != `{"scheme":"http"}` {
|
||||
t.Errorf("a member not yet on TLS: %s, %v", got, err)
|
||||
}
|
||||
|
||||
// A certificate from elsewhere is refused, not fallen back from.
|
||||
ca, err := tlsconf.NewCA("elsewhere")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
leaf, err := ca.IssueServer("other", []string{"127.0.0.1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cert, err := tls.X509KeyPair(leaf.CertPEM, leaf.KeyPEM)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
other := httptest.NewUnstartedServer(answer)
|
||||
other.TLS = &tls.Config{Certificates: []tls.Certificate{cert}}
|
||||
other.StartTLS()
|
||||
defer other.Close()
|
||||
if _, err := ask(other.URL); err == nil {
|
||||
t.Error("a certificate the authority did not sign was accepted")
|
||||
}
|
||||
}
|
||||
+211
-33
@@ -10,6 +10,7 @@ import (
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
// Node is a machine the setup reaches, and how.
|
||||
@@ -34,6 +35,15 @@ type Setup struct {
|
||||
Emit func(step, detail string, changed bool)
|
||||
// Wait bounds each wait for the cluster to come up.
|
||||
Wait time.Duration
|
||||
// CA signs each member's certificate; nil signs none and turns nothing
|
||||
// on. CAFile is where this control plane finds the authority's
|
||||
// certificate, for the DSN: used when TLS is on, whoever turned it on.
|
||||
CA *tlsconf.CA
|
||||
CAFile string
|
||||
|
||||
// RestPassword is what Patroni's API asks for before a change, once Run
|
||||
// has found or made it: for `server add`, which changes the access rules.
|
||||
RestPassword string
|
||||
}
|
||||
|
||||
// Local is a Remote for the machine this runs on, which init runs as root.
|
||||
@@ -83,16 +93,41 @@ func (s *Setup) Run(ctx context.Context) (string, error) {
|
||||
if s.Status == nil {
|
||||
s.Status = HTTPStatus
|
||||
}
|
||||
// Members' APIs are verified against the cluster's authority once they
|
||||
// speak TLS.
|
||||
switch {
|
||||
case s.CA != nil:
|
||||
if err := setRESTCAPEM(s.CA.PEM); err != nil {
|
||||
return "", err
|
||||
}
|
||||
case s.CAFile != "":
|
||||
_ = SetRESTCA(s.CAFile)
|
||||
}
|
||||
sec, err := s.secrets(ctx)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("passwords: %w", err)
|
||||
}
|
||||
if err := s.etcd(ctx); err != nil {
|
||||
s.RestPassword = sec.RestAPI
|
||||
if err := s.etcd(ctx, sec); err != nil {
|
||||
return "", fmt.Errorf("etcd: %w", err)
|
||||
}
|
||||
if err := s.certs(ctx); err != nil {
|
||||
return "", fmt.Errorf("certificates: %w", err)
|
||||
}
|
||||
if err := s.patroni(ctx, sec); err != nil {
|
||||
return "", fmt.Errorf("patroni: %w", err)
|
||||
}
|
||||
// What a cluster set up before them lacks, applied in place; a new one
|
||||
// has them already and these find nothing to change.
|
||||
if err := s.restAuth(ctx, sec); err != nil {
|
||||
return "", fmt.Errorf("patroni api: %w", err)
|
||||
}
|
||||
if err := s.tlsInShared(ctx, sec); err != nil {
|
||||
return "", fmt.Errorf("tls: %w", err)
|
||||
}
|
||||
if err := s.ensurePatroniTLS(ctx, sec); err != nil {
|
||||
return "", fmt.Errorf("patroni api: %w", err)
|
||||
}
|
||||
if err := s.database(ctx); err != nil {
|
||||
return "", fmt.Errorf("database: %w", err)
|
||||
}
|
||||
@@ -100,52 +135,64 @@ func (s *Setup) Run(ctx context.Context) (string, error) {
|
||||
for i, d := range s.DB {
|
||||
members[i] = d.Member
|
||||
}
|
||||
return DSN(members, sec.App), nil
|
||||
// TLS when this setup turned it on, or found it on: then the member
|
||||
// reached is verified against the authority's certificate, which every
|
||||
// control plane has even where its key is not.
|
||||
caFile := ""
|
||||
if s.CAFile != "" && (s.CA != nil || s.sharedTLS(ctx)) {
|
||||
caFile = s.CAFile
|
||||
}
|
||||
return DSN(members, sec.App, caFile), nil
|
||||
}
|
||||
|
||||
// secretDir holds the passwords on every database machine, readable by root
|
||||
// alone. Generated on the first, carried to the others.
|
||||
const secretDir = "/root/llmgw-pg"
|
||||
|
||||
var secretNames = []string{"superuser", "replicator", "llmgw"}
|
||||
var secretNames = []string{"superuser", "replicator", "llmgw", "restapi"}
|
||||
|
||||
func (s *Setup) secrets(ctx context.Context) (Secrets, error) {
|
||||
script := `set -e
|
||||
sudo install -d -m 700 ` + secretDir + `
|
||||
for f in superuser replicator llmgw; do
|
||||
for f in superuser replicator llmgw restapi; do
|
||||
sudo test -s ` + secretDir + `/$f.pw || sudo sh -c "umask 077; openssl rand -hex 24 > ` + secretDir + `/$f.pw"
|
||||
done
|
||||
for f in superuser replicator llmgw; do printf '%s=' $f; sudo cat ` + secretDir + `/$f.pw; done`
|
||||
for f in superuser replicator llmgw restapi; do printf '%s=' $f; sudo cat ` + secretDir + `/$f.pw; done`
|
||||
out, err := s.DB[0].Remote.Run(ctx, script, nil)
|
||||
if err != nil {
|
||||
return Secrets{}, fmt.Errorf("on %s: %v: %s", s.DB[0].Name, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
got := parseSecrets(out)
|
||||
if got.Superuser == "" || got.Replicator == "" || got.App == "" {
|
||||
if got.Superuser == "" || got.Replicator == "" || got.App == "" || got.RestAPI == "" {
|
||||
return Secrets{}, fmt.Errorf("could not read them back from %s", s.DB[0].Name)
|
||||
}
|
||||
for _, d := range s.DB[1:] {
|
||||
have, err := d.Remote.Run(ctx, `for f in superuser replicator llmgw; do
|
||||
have, err := d.Remote.Run(ctx, `for f in superuser replicator llmgw restapi; do
|
||||
if sudo test -s `+secretDir+`/$f.pw; then printf '%s=' $f; sudo cat `+secretDir+`/$f.pw; fi
|
||||
done`, nil)
|
||||
if err != nil {
|
||||
return Secrets{}, fmt.Errorf("on %s: %v", d.Name, err)
|
||||
}
|
||||
if h := parseSecrets(have); h != (Secrets{}) {
|
||||
if h != got {
|
||||
h := parseSecrets(have)
|
||||
if h.Superuser != "" || h.Replicator != "" || h.App != "" {
|
||||
if h.Superuser != got.Superuser || h.Replicator != got.Replicator || h.App != got.App {
|
||||
// Not overwritten: a member holding other passwords belongs
|
||||
// to another cluster, or to an earlier one somebody meant to
|
||||
// keep, and replacing them would lock it out of itself.
|
||||
return Secrets{}, fmt.Errorf("%s holds passwords in %s that differ from %s's; "+
|
||||
"remove them there if that machine is meant to join this cluster", d.Name, secretDir, s.DB[0].Name)
|
||||
}
|
||||
continue
|
||||
if h.RestAPI == got.RestAPI {
|
||||
continue
|
||||
}
|
||||
// A cluster from before the API asked for a password: the
|
||||
// member is given the one the first holds.
|
||||
}
|
||||
body := fmt.Sprintf("%s\n%s\n%s\n", got.Superuser, got.Replicator, got.App)
|
||||
body := fmt.Sprintf("%s\n%s\n%s\n%s\n", got.Superuser, got.Replicator, got.App, got.RestAPI)
|
||||
if out, err := d.Remote.Run(ctx, `set -e
|
||||
sudo install -d -m 700 `+secretDir+`
|
||||
read su; read rep; read app
|
||||
for kv in "superuser $su" "replicator $rep" "llmgw $app"; do
|
||||
read su; read rep; read app; read api
|
||||
for kv in "superuser $su" "replicator $rep" "llmgw $app" "restapi $api"; do
|
||||
set -- $kv
|
||||
printf '%s\n' "$2" | sudo sh -c "umask 077; cat > `+secretDir+`/$1.pw"
|
||||
done`, []byte(body)); err != nil {
|
||||
@@ -171,6 +218,8 @@ func parseSecrets(out []byte) Secrets {
|
||||
s.Replicator = v
|
||||
case "llmgw":
|
||||
s.App = v
|
||||
case "restapi":
|
||||
s.RestAPI = v
|
||||
}
|
||||
}
|
||||
return s
|
||||
@@ -178,27 +227,117 @@ func parseSecrets(out []byte) Secrets {
|
||||
|
||||
// etcd brings up the voting members, unless every one of them is already
|
||||
// healthy in one cluster — in which case nothing is touched, since rewriting
|
||||
// a member's configuration means emptying its data.
|
||||
func (s *Setup) etcd(ctx context.Context) error {
|
||||
// a member's configuration means emptying its data. With the cluster's
|
||||
// authority here, members speak TLS; a running cluster that does not yet is
|
||||
// moved to it (etcdToTLS).
|
||||
func (s *Setup) etcd(ctx context.Context, sec Secrets) error {
|
||||
nodes := s.etcdMembers()
|
||||
var members []Member
|
||||
var endpoints []string
|
||||
for _, n := range nodes {
|
||||
members = append(members, n.Member)
|
||||
endpoints = append(endpoints, "http://"+n.IP+":2379")
|
||||
}
|
||||
if s.etcdHealthy(ctx, nodes[0], endpoints, len(nodes)) {
|
||||
s.emit("etcd", false, "%d members healthy; left as they are", len(nodes))
|
||||
tls := s.CA != nil
|
||||
if s.etcdHealthy(ctx, nodes[0], etcdEndpoints(nodes, tls), len(nodes), tls) {
|
||||
s.emit("etcd", false, "%d members healthy%s; left as they are", len(nodes), map[bool]string{true: ", over TLS"}[tls])
|
||||
return nil
|
||||
}
|
||||
if tls && s.etcdHealthy(ctx, nodes[0], etcdEndpoints(nodes, false), len(nodes), false) {
|
||||
return s.etcdToTLS(ctx, sec)
|
||||
}
|
||||
return s.etcdBootstrap(ctx, tls)
|
||||
}
|
||||
|
||||
changed := make([]bool, len(nodes))
|
||||
func etcdEndpoints(nodes []Node, tls bool) []string {
|
||||
scheme := "http"
|
||||
if tls {
|
||||
scheme = "https"
|
||||
}
|
||||
out := make([]string, len(nodes))
|
||||
for i, n := range nodes {
|
||||
out[i] = scheme + "://" + n.IP + ":2379"
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// etcdInstall puts etcd on every member, stopped if it was not running.
|
||||
func (s *Setup) etcdInstall(ctx context.Context, nodes []Node) error {
|
||||
for _, n := range nodes {
|
||||
out, err := n.Remote.Run(ctx, `set -e
|
||||
if ! command -v etcd >/dev/null 2>&1; then
|
||||
sudo DEBIAN_FRONTEND=noninteractive apt-get -qq update >/dev/null
|
||||
sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y etcd-server etcd-client >/dev/null
|
||||
fi
|
||||
fi`, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("on %s: %v: %s", n.Name, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// etcdCerts gives each etcd member a certificate from the cluster's
|
||||
// authority, for its address, its name and loopback, unless it has one.
|
||||
func (s *Setup) etcdCerts(ctx context.Context, nodes []Node) error {
|
||||
for _, n := range nodes {
|
||||
have, _ := n.Remote.Run(ctx, "sudo openssl x509 -in "+EtcdCert+" -noout -ext subjectAltName,extendedKeyUsage 2>/dev/null", nil)
|
||||
if strings.Contains(string(have), "IP Address:"+n.IP) && strings.Contains(string(have), "Client Authentication") {
|
||||
continue
|
||||
}
|
||||
leaf, err := s.CA.IssuePeer(n.Name, []string{n.IP, n.Name, "127.0.0.1", "localhost"})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := putTLS(ctx, n, "etcd", EtcdTLSDir, EtcdCA, EtcdCert, EtcdKey, s.CA.PEM, leaf.CertPEM, leaf.KeyPEM); err != nil {
|
||||
return err
|
||||
}
|
||||
s.emit("etcd", true, "certificate for %s (%s)", n.Name, n.IP)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// putTLS installs an authority's certificate, a leaf and its key on a
|
||||
// machine, owned by user, the key readable by it alone.
|
||||
func putTLS(ctx context.Context, n Node, user, dir, caPath, certPath, keyPath string, ca, cert, key []byte) error {
|
||||
var body bytes.Buffer
|
||||
body.Write(ca)
|
||||
body.WriteString("\x00")
|
||||
body.Write(cert)
|
||||
body.WriteString("\x00")
|
||||
body.Write(key)
|
||||
out, err := n.Remote.Run(ctx, `set -e
|
||||
sudo install -d -o `+user+` -g `+user+` -m 750 `+dir+`
|
||||
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
|
||||
cat > "$t/all"
|
||||
python3 - "$t" <<'PY'
|
||||
import sys
|
||||
d = sys.argv[1]
|
||||
ca, crt, key = open(d + "/all", "rb").read().split(b"\x00")
|
||||
for name, data in (("ca", ca), ("crt", crt), ("key", key)):
|
||||
open(d + "/" + name, "wb").write(data)
|
||||
PY
|
||||
sudo install -o `+user+` -g `+user+` -m 644 "$t/ca" `+caPath+`
|
||||
sudo install -o `+user+` -g `+user+` -m 644 "$t/crt" `+certPath+`
|
||||
sudo install -o `+user+` -g `+user+` -m 600 "$t/key" `+keyPath, body.Bytes())
|
||||
if err != nil {
|
||||
return fmt.Errorf("on %s: %v: %s", n.Name, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// etcdBootstrap writes every member's configuration and starts them
|
||||
// together, emptying a member whose configuration changed.
|
||||
func (s *Setup) etcdBootstrap(ctx context.Context, tls bool) error {
|
||||
nodes := s.etcdMembers()
|
||||
var members []Member
|
||||
for _, n := range nodes {
|
||||
members = append(members, n.Member)
|
||||
}
|
||||
if err := s.etcdInstall(ctx, nodes); err != nil {
|
||||
return err
|
||||
}
|
||||
if tls {
|
||||
if err := s.etcdCerts(ctx, nodes); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
changed := make([]bool, len(nodes))
|
||||
for i, n := range nodes {
|
||||
out, err := n.Remote.Run(ctx, `set -e
|
||||
new=$(cat)
|
||||
if [ "$(sudo cat /etc/default/etcd 2>/dev/null)" != "$new" ]; then
|
||||
sudo systemctl stop etcd >/dev/null 2>&1 || true
|
||||
@@ -206,7 +345,7 @@ if [ "$(sudo cat /etc/default/etcd 2>/dev/null)" != "$new" ]; then
|
||||
printf '%s\n' "$new" | sudo tee /etc/default/etcd >/dev/null
|
||||
echo changed
|
||||
fi
|
||||
sudo systemctl enable etcd >/dev/null 2>&1`, []byte(Etcd(n.Member, members)))
|
||||
sudo systemctl enable etcd >/dev/null 2>&1`, []byte(Etcd(n.Member, members, tls)))
|
||||
if err != nil {
|
||||
return fmt.Errorf("on %s: %v: %s", n.Name, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
@@ -237,7 +376,7 @@ sudo systemctl enable etcd >/dev/null 2>&1`, []byte(Etcd(n.Member, members)))
|
||||
}
|
||||
}
|
||||
deadline := time.Now().Add(s.Wait)
|
||||
for !s.etcdHealthy(ctx, nodes[0], endpoints, len(nodes)) {
|
||||
for !s.etcdHealthy(ctx, nodes[0], etcdEndpoints(nodes, tls), len(nodes), tls) {
|
||||
if time.Now().After(deadline) {
|
||||
return fmt.Errorf("the %d members did not become healthy within %s; `journalctl -u etcd` on each says why", len(nodes), s.Wait)
|
||||
}
|
||||
@@ -245,16 +384,48 @@ sudo systemctl enable etcd >/dev/null 2>&1`, []byte(Etcd(n.Member, members)))
|
||||
return err
|
||||
}
|
||||
}
|
||||
s.emit("etcd", true, "%d members up", len(nodes))
|
||||
s.emit("etcd", true, "%d members up%s", len(nodes), map[bool]string{true: ", over TLS"}[tls])
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Setup) etcdHealthy(ctx context.Context, n Node, endpoints []string, want int) bool {
|
||||
out, err := n.Remote.Run(ctx, "ETCDCTL_API=3 etcdctl --dial-timeout=3s --endpoints="+
|
||||
strings.Join(endpoints, ",")+" endpoint health 2>&1", nil)
|
||||
func (s *Setup) etcdHealthy(ctx context.Context, n Node, endpoints []string, want int, tls bool) bool {
|
||||
out, err := n.Remote.Run(ctx, etcdctl(tls, endpoints)+" endpoint health 2>&1", nil)
|
||||
return err == nil && strings.Count(string(out), "is healthy") == want
|
||||
}
|
||||
|
||||
// ensurePatroniTLS puts a member whose Patroni does not yet reach etcd or
|
||||
// serve its API over TLS on it, in maintenance mode: an etcd moved to TLS by
|
||||
// a run that stopped before it got this far.
|
||||
func (s *Setup) ensurePatroniTLS(ctx context.Context, sec Secrets) error {
|
||||
if s.CA == nil {
|
||||
return nil
|
||||
}
|
||||
need := false
|
||||
for _, d := range s.DB {
|
||||
out, err := d.Remote.Run(ctx, patroniTLSEdit("check"), nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
need = need || strings.TrimSpace(lastLine(out)) == "changed"
|
||||
}
|
||||
if !need {
|
||||
return nil
|
||||
}
|
||||
if err := s.pause(ctx, sec, true); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := s.patroniOnTLS(ctx, true); err != nil {
|
||||
return s.stuck(err)
|
||||
}
|
||||
if err := s.waitFor(ctx, s.dbHosts(), func(c Cluster) bool { return c.Leader() != "" }); err != nil {
|
||||
return s.stuck(err)
|
||||
}
|
||||
if err := s.pause(ctx, sec, false); err != nil {
|
||||
return s.stuck(err)
|
||||
}
|
||||
return s.waitFor(ctx, s.dbHosts(), func(c Cluster) bool { return c.Complete(len(s.DB)) })
|
||||
}
|
||||
|
||||
// patroni installs PostgreSQL and Patroni on every database machine, unless
|
||||
// the cluster already has a leader and every member streaming.
|
||||
func (s *Setup) patroni(ctx context.Context, sec Secrets) error {
|
||||
@@ -279,6 +450,10 @@ func (s *Setup) patroni(ctx context.Context, sec Secrets) error {
|
||||
}
|
||||
version = v
|
||||
}
|
||||
// Now that PostgreSQL's user exists, its certificates.
|
||||
if err := s.certs(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var etcd, db []Member
|
||||
for _, n := range s.etcdMembers() {
|
||||
@@ -288,7 +463,7 @@ func (s *Setup) patroni(ctx context.Context, sec Secrets) error {
|
||||
db = append(db, d.Member)
|
||||
}
|
||||
for i, d := range s.DB {
|
||||
cfg := Patroni(d.Member, db, etcd, s.Clients, version, sec)
|
||||
cfg := Patroni(d.Member, db, etcd, s.Clients, version, sec, s.CA != nil)
|
||||
if out, err := d.Remote.Run(ctx, `set -e
|
||||
sudo install -d -m 755 /etc/patroni
|
||||
sudo install -d -o postgres -g postgres -m 700 /var/lib/postgresql/`+version+`/llmgw
|
||||
@@ -373,6 +548,7 @@ func (s *Setup) database(ctx context.Context) error {
|
||||
}
|
||||
out, err := leader.Remote.Run(ctx, `set -e
|
||||
pw=$(sudo cat `+secretDir+`/llmgw.pw)
|
||||
sudo -u postgres psql -Atc "select 'had=' || (select count(*) from pg_roles where rolname = 'llmgw') + (select count(*) from pg_database where datname = 'llmgw')"
|
||||
sudo -u postgres psql -q -v ON_ERROR_STOP=1 -v pw="$pw" <<'SQL'
|
||||
SELECT format('CREATE ROLE %I LOGIN', 'llmgw') WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'llmgw') \gexec
|
||||
ALTER ROLE llmgw PASSWORD :'pw';
|
||||
@@ -381,7 +557,9 @@ SQL`, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("on %s: %v: %s", leader.Name, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
s.emit("database", true, "role and database %s on %s", Database, leader.Name)
|
||||
// Said as a change only when something was made: a rerun that finds
|
||||
// both is not a reason to restart the gateway.
|
||||
s.emit("database", !strings.Contains(string(out), "had=2"), "role and database %s on %s", Database, leader.Name)
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
+3
-13
@@ -4,9 +4,7 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
@@ -70,24 +68,16 @@ func (c Cluster) String() string {
|
||||
// HTTPStatus asks each host's Patroni in turn and returns the first answer:
|
||||
// any member describes the whole cluster.
|
||||
func HTTPStatus(ctx context.Context, hosts []string) (Cluster, error) {
|
||||
client := &http.Client{Timeout: 3 * time.Second}
|
||||
var last error
|
||||
for _, h := range hosts {
|
||||
u := "http://" + net.JoinHostPort(h, strconv.Itoa(restPort)) + "/cluster"
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
|
||||
if err != nil {
|
||||
return Cluster{}, err
|
||||
}
|
||||
resp, err := client.Do(req)
|
||||
b, err := restJSON(ctx, h, http.MethodGet, "/cluster", nil, "", 3*time.Second)
|
||||
if err != nil {
|
||||
last = err
|
||||
continue
|
||||
}
|
||||
var c Cluster
|
||||
err = json.NewDecoder(resp.Body).Decode(&c)
|
||||
resp.Body.Close()
|
||||
if err != nil {
|
||||
last = fmt.Errorf("%s: %w", u, err)
|
||||
if err := json.Unmarshal(b, &c); err != nil {
|
||||
last = fmt.Errorf("%s /cluster: %w", h, err)
|
||||
continue
|
||||
}
|
||||
for i := range c.Members {
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
package pgha
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Moving a running cluster's etcd to TLS cannot be done member by member:
|
||||
// a member's peers reach it at the address the cluster recorded, http, and
|
||||
// changing that one member at a time loses the majority half way. So it is
|
||||
// done the way a first install is — every member's configuration rewritten
|
||||
// and started together, empty — with what Patroni keeps there carried over.
|
||||
//
|
||||
// While etcd is down Patroni would demote the primary. So the cluster is in
|
||||
// maintenance mode (pause) throughout: Patroni stops managing PostgreSQL,
|
||||
// which goes on serving, and a lost leader lock means nothing. Patroni is
|
||||
// then restarted against the new etcd, which in maintenance mode leaves
|
||||
// PostgreSQL running, and maintenance ends.
|
||||
|
||||
// etcdToTLS moves a healthy plain-HTTP etcd, and Patroni with it, to TLS.
|
||||
func (s *Setup) etcdToTLS(ctx context.Context, sec Secrets) error {
|
||||
nodes := s.etcdMembers()
|
||||
if err := s.etcdCerts(ctx, nodes); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.certs(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.pause(ctx, sec, true); err != nil {
|
||||
return fmt.Errorf("maintenance mode: %w", err)
|
||||
}
|
||||
s.emit("etcd", true, "Patroni in maintenance mode: PostgreSQL serves on while etcd moves to TLS")
|
||||
|
||||
kept, err := s.etcdDump(ctx, nodes[0])
|
||||
if err != nil {
|
||||
return s.stuck(fmt.Errorf("reading what Patroni keeps in etcd: %w", err))
|
||||
}
|
||||
if err := s.etcdBootstrap(ctx, true); err != nil {
|
||||
return s.stuck(err)
|
||||
}
|
||||
if err := s.etcdRestore(ctx, nodes[0], kept); err != nil {
|
||||
return s.stuck(fmt.Errorf("putting Patroni's keys back: %w", err))
|
||||
}
|
||||
s.emit("etcd", true, "%d of Patroni's keys carried over", len(kept))
|
||||
|
||||
if _, err := s.patroniOnTLS(ctx, true); err != nil {
|
||||
return s.stuck(err)
|
||||
}
|
||||
hosts := s.dbHosts()
|
||||
if err := s.waitFor(ctx, hosts, func(c Cluster) bool { return c.Leader() != "" }); err != nil {
|
||||
return s.stuck(fmt.Errorf("Patroni against the new etcd: %w", err))
|
||||
}
|
||||
if err := s.pause(ctx, sec, false); err != nil {
|
||||
return s.stuck(fmt.Errorf("ending maintenance mode: %w", err))
|
||||
}
|
||||
if err := s.waitFor(ctx, hosts, func(c Cluster) bool { return c.Complete(len(s.DB)) }); err != nil {
|
||||
return err
|
||||
}
|
||||
s.emit("etcd", true, "etcd and Patroni's API speak TLS; maintenance mode ended")
|
||||
return nil
|
||||
}
|
||||
|
||||
// stuck says where a move that failed half way leaves the cluster.
|
||||
func (s *Setup) stuck(err error) error {
|
||||
return fmt.Errorf("%w — the cluster is left in maintenance mode, PostgreSQL still serving; "+
|
||||
"run this again, or end it with `patronictl -c /etc/patroni/config.yml resume` on a member", err)
|
||||
}
|
||||
|
||||
func (s *Setup) dbHosts() []string {
|
||||
hosts := make([]string, len(s.DB))
|
||||
for i, d := range s.DB {
|
||||
hosts[i] = d.IP
|
||||
}
|
||||
return hosts
|
||||
}
|
||||
|
||||
// pause turns maintenance mode on or off, and waits until every member says
|
||||
// so.
|
||||
func (s *Setup) pause(ctx context.Context, sec Secrets, on bool) error {
|
||||
var err error
|
||||
for _, h := range s.dbHosts() {
|
||||
if err = patchConfig(ctx, h, sec.RestAPI, map[string]any{"pause": on}); err == nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
deadline := time.Now().Add(s.Wait)
|
||||
for {
|
||||
all := true
|
||||
for _, h := range s.dbHosts() {
|
||||
b, err := restJSON(ctx, h, http.MethodGet, "/patroni", nil, "", 5*time.Second)
|
||||
var st struct {
|
||||
Pause bool `json:"pause"`
|
||||
}
|
||||
if err != nil || json.Unmarshal(b, &st) != nil || st.Pause != on {
|
||||
all = false
|
||||
}
|
||||
}
|
||||
if all {
|
||||
return nil
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
return fmt.Errorf("not every member was in maintenance mode=%v within %s", on, s.Wait)
|
||||
}
|
||||
if err := sleep(ctx, 2*time.Second); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// etcdKV is one key Patroni keeps, and whether it is tied to a lease: a
|
||||
// leader lock or a member's entry, which Patroni writes again itself and
|
||||
// which, put back without its lease, would never expire.
|
||||
type etcdKV struct {
|
||||
Key, Value []byte
|
||||
Lease int64
|
||||
}
|
||||
|
||||
func (s *Setup) etcdDump(ctx context.Context, n Node) ([]etcdKV, error) {
|
||||
out, err := n.Remote.Run(ctx, etcdctl(false, []string{"http://" + n.IP + ":2379"})+
|
||||
" get --prefix /service/ -w json", nil)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
var got struct {
|
||||
KVs []struct {
|
||||
Key string `json:"key"`
|
||||
Value string `json:"value"`
|
||||
Lease int64 `json:"lease"`
|
||||
} `json:"kvs"`
|
||||
}
|
||||
if err := json.Unmarshal(out, &got); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var kvs []etcdKV
|
||||
for _, kv := range got.KVs {
|
||||
k, err1 := base64.StdEncoding.DecodeString(kv.Key)
|
||||
v, err2 := base64.StdEncoding.DecodeString(kv.Value)
|
||||
if err1 != nil || err2 != nil {
|
||||
return nil, fmt.Errorf("an unreadable key in etcd's answer")
|
||||
}
|
||||
if kv.Lease == 0 {
|
||||
kvs = append(kvs, etcdKV{Key: k, Value: v})
|
||||
}
|
||||
}
|
||||
if len(kvs) == 0 {
|
||||
return nil, fmt.Errorf("etcd holds nothing of Patroni's under /service/")
|
||||
}
|
||||
return kvs, nil
|
||||
}
|
||||
|
||||
func (s *Setup) etcdRestore(ctx context.Context, n Node, kvs []etcdKV) error {
|
||||
for _, kv := range kvs {
|
||||
if strings.ContainsAny(string(kv.Key), "'\n") {
|
||||
return fmt.Errorf("a key that cannot be written back: %q", kv.Key)
|
||||
}
|
||||
out, err := n.Remote.Run(ctx, etcdctl(true, []string{"https://" + n.IP + ":2379"})+
|
||||
" put '"+string(kv.Key)+"'", kv.Value)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s: %v: %s", kv.Key, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// patroniOnTLS has each member's Patroni reach etcd over TLS and serve its
|
||||
// API over TLS, restarting those whose configuration changed. With paused
|
||||
// false, maintenance mode is entered for the restart and left after.
|
||||
func (s *Setup) patroniOnTLS(ctx context.Context, paused bool) (bool, error) {
|
||||
var todo []Node
|
||||
for _, d := range s.DB {
|
||||
out, err := d.Remote.Run(ctx, patroniTLSEdit("check"), nil)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
switch strings.TrimSpace(lastLine(out)) {
|
||||
case "changed":
|
||||
todo = append(todo, d)
|
||||
case "missing":
|
||||
return false, fmt.Errorf("%s has no /etc/patroni/config.yml", d.Name)
|
||||
}
|
||||
}
|
||||
if len(todo) == 0 {
|
||||
return false, nil
|
||||
}
|
||||
if !paused {
|
||||
return false, fmt.Errorf("Patroni on %d member(s) is not yet on TLS, which needs maintenance mode", len(todo))
|
||||
}
|
||||
for _, d := range todo {
|
||||
if out, err := d.Remote.Run(ctx, patroniTLSEdit("apply")+`
|
||||
sudo systemctl restart patroni`, nil); err != nil {
|
||||
return false, fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
s.emit("patroni api", true, "%s reaches etcd over TLS and serves its API over TLS", d.Name)
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// patroniTLSEdit edits /etc/patroni/config.yml: "check" says whether it
|
||||
// would change, "apply" writes it.
|
||||
func patroniTLSEdit(mode string) string {
|
||||
return `set -e
|
||||
test -f /etc/patroni/config.yml || { echo missing; exit 0; }
|
||||
sudo python3 - ` + mode + ` <<'PY'
|
||||
import sys, yaml
|
||||
p = "/etc/patroni/config.yml"
|
||||
c = yaml.safe_load(open(p))
|
||||
want = {
|
||||
"restapi": {"certfile": "` + ServerCert + `", "keyfile": "` + ServerKey + `", "cafile": "` + CACert + `"},
|
||||
"ctl": {"cacert": "` + CACert + `"},
|
||||
"etcd3": {"protocol": "https", "cacert": "` + CACert + `", "cert": "` + ServerCert + `", "key": "` + ServerKey + `"},
|
||||
}
|
||||
changed = False
|
||||
for section, kv in want.items():
|
||||
sec = c.setdefault(section, {})
|
||||
for k, v in kv.items():
|
||||
if sec.get(k) != v:
|
||||
sec[k] = v
|
||||
changed = True
|
||||
if sys.argv[1] == "apply" and changed:
|
||||
open(p + ".new", "w").write("# llmgw: the control planes' store, kept by Patroni, its leader lock in etcd.\n" +
|
||||
yaml.safe_dump(c, default_flow_style=False, sort_keys=False))
|
||||
import os, shutil
|
||||
shutil.chown(p + ".new", "root", "postgres")
|
||||
os.chmod(p + ".new", 0o640)
|
||||
os.replace(p + ".new", p)
|
||||
print("changed" if changed else "same")
|
||||
PY`
|
||||
}
|
||||
Loaded 100 of 149 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user