Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
91fe24a105 | ||
|
|
9bf8256577 | ||
|
|
1da4a2e1ef | ||
|
|
b661bfead8 |
No files matched your search
@@ -66,8 +66,44 @@ ControlPlaneLeadership, CertificateExpiresSoon and CertificateExpiresVerySoon.
|
||||
than its minimum, so an on-demand model that was asked for and fitted nowhere
|
||||
no longer reads as satisfied.
|
||||
|
||||
### Backups
|
||||
|
||||
`llmgw server backup` writes the store and `/etc/llmgw` — configuration,
|
||||
secrets, and the certificate authority's key — to one archive readable by root
|
||||
only, keeping the newest 14. `deploy/systemd/llmgw-backup.timer` runs it
|
||||
nightly. The control plane publishes the newest archive's time as
|
||||
`llmgw_backup_last_success_timestamp_seconds`, and BackupTooOld fires when it is
|
||||
more than 26 hours old or there has never been one. docs/install.md now says how
|
||||
to restore, including why running engines survive it: the engine-key secret
|
||||
comes back with the store.
|
||||
|
||||
### Security
|
||||
|
||||
- **Alertmanager answered anyone on the network**, including requests to
|
||||
silence alerts. The monitoring README now binds it to loopback, where
|
||||
Prometheus beside it is its only client.
|
||||
- **Login and signup could be made to use gigabytes.** Each attempt hashes with
|
||||
64 MiB of argon2, with no bound on how many ran at once, and the per-address
|
||||
throttle grew by one entry per address tried. Hashes now run a few at a time
|
||||
and the rest wait; the throttle expires old entries, and when it is still
|
||||
full it refuses new addresses rather than dropping live ones — which would
|
||||
have let a flood of made-up addresses reset the throttle on a real one.
|
||||
- **Any key holder could grow Prometheus without bound** by asking for made-up
|
||||
model names, each of which became a metric label. Requests are counted under
|
||||
the catalog's name for their model, or "unknown".
|
||||
- **A SQLite backup left a full copy of the store readable** by local users
|
||||
while it ran. The snapshot is now made in a private directory, and
|
||||
llmgw-backup.service sets `UMask=0077`.
|
||||
- **An engine archive could write outside its directory through chained
|
||||
links.** Archives are unpacked through `os.Root`, and a link may only point
|
||||
at a file beside it. Every llama.cpp b10991 archive, CPU and CUDA, already
|
||||
follows that rule.
|
||||
|
||||
### Fixes
|
||||
|
||||
- Tests that slept a fixed time and then asserted now wait for their
|
||||
condition. They are the likeliest cause of CI's intermittent failures on a
|
||||
loaded runner.
|
||||
- `ctl` reported a change the API had queued (202 Accepted) as an error, as
|
||||
`ctl restart` did on its first real use.
|
||||
|
||||
|
||||
@@ -46,6 +46,12 @@ apt-get install prometheus prometheus-alertmanager
|
||||
install -m 0644 prometheus.yml /etc/prometheus/prometheus.yml
|
||||
install -m 0644 llmgw-rules.yml /etc/prometheus/rules/llmgw.yml
|
||||
install -m 0644 alertmanager.yml /etc/prometheus/alertmanager.yml
|
||||
# Alertmanager answers on loopback only. Its API takes silences from anyone who
|
||||
# can reach it, with no login, and silencing BackupTooOld or ModelParked is how
|
||||
# something that should be noticed is not. Prometheus beside it is its only
|
||||
# client; one Alertmanager needs no cluster port.
|
||||
echo 'ARGS="--web.listen-address=127.0.0.1:9093 --cluster.listen-address="' \
|
||||
> /etc/default/prometheus-alertmanager
|
||||
promtool check config /etc/prometheus/prometheus.yml
|
||||
systemctl restart prometheus prometheus-alertmanager
|
||||
|
||||
@@ -75,7 +81,8 @@ Alertmanager groups and holds alerts and has no receiver.
|
||||
Where an alert should go — an address, a Slack workspace, a webhook — is a
|
||||
decision with credentials attached, and a monitoring system that quietly starts
|
||||
mailing people is worse than one that is visibly not wired up. Alerts are
|
||||
visible at `:9093` and in Prometheus; add a receiver to `alertmanager.yml` to
|
||||
visible in Prometheus, and at `:9093` on the monitoring host itself (`ssh -L
|
||||
9093:127.0.0.1:9093 <host>` to open its page); add a receiver to `alertmanager.yml` to
|
||||
make them travel. `critical` already routes separately so it can be pointed
|
||||
somewhere else without touching the rest.
|
||||
|
||||
@@ -86,7 +93,7 @@ An alerting system nobody has seen fire is an assumption.
|
||||
```
|
||||
llmgw ctl cordon <node> # work already there keeps running
|
||||
# ~2 minutes later
|
||||
curl -s http://<host>:9093/api/v2/alerts | jq '.[].labels'
|
||||
ssh <host> curl -s http://127.0.0.1:9093/api/v2/alerts | jq '.[].labels'
|
||||
llmgw ctl uncordon <node> # the alert resolves on its own
|
||||
```
|
||||
|
||||
|
||||
@@ -123,6 +123,17 @@ groups:
|
||||
summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
|
||||
description: See CertificateExpiresSoon; a week is not long.
|
||||
|
||||
- alert: BackupTooOld
|
||||
expr: time() - llmgw_backup_last_success_timestamp_seconds > 26 * 3600
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "The newest llmgw backup is {{ $value | humanizeDuration }} old"
|
||||
description: >-
|
||||
llmgw-backup.timer runs nightly. Also fires when there has never
|
||||
been a backup (the series is 0 then). `journalctl -u
|
||||
llmgw-backup.service` on the control plane says why the last one
|
||||
failed; docs/install.md, "Backing up", says how to set it up.
|
||||
|
||||
- alert: PoolSaturated
|
||||
expr: llmgw_pool_saturation > 1
|
||||
for: 5m
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
[Unit]
|
||||
Description=llmgw backup: the store and /etc/llmgw, to /var/backups/llmgw-server
|
||||
After=llmgw-server.service postgresql.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
# Nothing it writes is for anyone else: the archive holds the CA's key and a
|
||||
# copy of the store.
|
||||
UMask=0077
|
||||
# Root, because the archive includes /etc/llmgw/secrets and the certificate
|
||||
# authority's key, which the llmgw user deliberately cannot read all of.
|
||||
ExecStart=/usr/local/bin/llmgw server backup --config /etc/llmgw/server.yaml --keep 14
|
||||
Nice=10
|
||||
IOSchedulingClass=idle
|
||||
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=Back up llmgw every night
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 03:15:00
|
||||
# A machine that was off at 03:15 backs up when it next starts, rather than
|
||||
# waiting another day with a gap nobody notices until the alert.
|
||||
Persistent=true
|
||||
RandomizedDelaySec=10m
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
+56
-18
@@ -961,36 +961,74 @@ the choice is yours rather than silent.
|
||||
|
||||
## Backing up
|
||||
|
||||
Three things, in descending order of how badly you want them.
|
||||
`llmgw server backup` writes one archive holding everything a control plane
|
||||
cannot be rebuilt without:
|
||||
|
||||
**The certificate authority.** `ca-key.pem`, `/etc/llmgw/pki` unless you kept it elsewhere (step 2).
|
||||
Losing it does not stop the cluster, and it means every node identity has to be
|
||||
reissued the next time you touch one. Nothing in llmgw can reconstruct it.
|
||||
- **the store** — users, API keys, the tenancy tree, limits and budgets, usage
|
||||
history, models added at runtime, measured capacity, and the secret every
|
||||
engine key is derived from (`pg_dump` in its custom format for PostgreSQL, a
|
||||
consistent `VACUUM INTO` snapshot for SQLite; both are safe against a running
|
||||
gateway);
|
||||
- **`/etc/llmgw`** — the configuration, the store's address, and the
|
||||
certificate authority's key (`pki/`). Nothing in llmgw can reconstruct that
|
||||
key; losing it means reissuing every node's identity.
|
||||
|
||||
**The store.** Users, API keys, organisations, teams, projects, every limit and
|
||||
budget, all usage history, models added at runtime, and measured capacity.
|
||||
Archives go to `/var/backups/llmgw-server`, readable by root only because of the CA
|
||||
key, and the newest 14 are kept. Run it nightly with the timer:
|
||||
|
||||
```
|
||||
# SQLite
|
||||
sqlite3 /var/lib/llmgw/llmgw.db ".backup '/backup/llmgw-$(date +%F).db'"
|
||||
|
||||
# PostgreSQL
|
||||
pg_dump --format=custom llmgw > /backup/llmgw-$(date +%F).dump
|
||||
sudo install -m 0644 deploy/systemd/llmgw-backup.service deploy/systemd/llmgw-backup.timer /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now llmgw-backup.timer
|
||||
sudo systemctl start llmgw-backup.service # one now, to see it work
|
||||
ls -l /var/backups/llmgw-server
|
||||
```
|
||||
|
||||
`.backup` and `pg_dump` are both safe against a running gateway. Copying the
|
||||
SQLite file with `cp` while it is being written is not.
|
||||
The control plane publishes when the newest archive was written as
|
||||
`llmgw_backup_last_success_timestamp_seconds`, and **BackupTooOld** fires when
|
||||
it is more than 26 hours old — including when there has never been one.
|
||||
|
||||
**Copy the archives off the machine.** They sit on the disk they back up and do
|
||||
not survive losing it. Anything that copies files will do — rsync from the
|
||||
mirror, a NAS mount, an offsite bucket — as long as it keeps the archives
|
||||
readable by root only.
|
||||
|
||||
**Keys are stored hashed and cannot be recovered from a backup or from
|
||||
anywhere else.** Restoring an old store restores which keys exist and who owns
|
||||
them, not their secrets. A key created after the backup is gone and has to be
|
||||
reissued — which is why the store is worth backing up on a schedule rather than
|
||||
before upgrades.
|
||||
reissued.
|
||||
|
||||
### Restoring
|
||||
|
||||
On a machine with llmgw installed (the same version or newer) and, for
|
||||
PostgreSQL, an empty database and role as `server init --postgres` would create
|
||||
them:
|
||||
|
||||
```
|
||||
sudo systemctl stop llmgw-server
|
||||
mkdir /tmp/restore && sudo tar -xzf /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz -C /tmp/restore
|
||||
cat /tmp/restore/MANIFEST # version and time it was taken
|
||||
|
||||
# /etc/llmgw: configuration, secrets, certificates and the CA
|
||||
sudo cp -a /tmp/restore/etc/llmgw/. /etc/llmgw/
|
||||
|
||||
# the store — PostgreSQL
|
||||
sudo -u postgres pg_restore --clean --if-exists --no-owner --role=llmgw -d llmgw /tmp/restore/store.pgdump
|
||||
# the store — SQLite
|
||||
sudo install -o llmgw -g llmgw -m 0600 /tmp/restore/store.sqlite /var/lib/llmgw/llmgw.db
|
||||
|
||||
sudo systemctl start llmgw-server
|
||||
sudo rm -rf /tmp/restore
|
||||
```
|
||||
|
||||
Running engines keep serving throughout: agents reconnect to the restored
|
||||
control plane and their replicas are adopted, and because the engine-key secret
|
||||
came back with the store, the router can still reach them.
|
||||
|
||||
**The mirror's directory** is the one thing you can rebuild, by downloading it
|
||||
all again. Worth remembering that a pinned version may no longer be upstream to
|
||||
download: that is exactly what pinning it protected you from, and a backup is
|
||||
what makes the protection outlive the disk.
|
||||
all again. A pinned version may no longer be upstream to download: that is
|
||||
exactly what pinning it protected you from, and a copy of the directory is what
|
||||
makes the protection outlive the disk.
|
||||
|
||||
## Reclaiming disk
|
||||
|
||||
|
||||
@@ -169,6 +169,7 @@ rather than repeated.
|
||||
| `monitoring` | `Monitoring` | [Server](#server) |
|
||||
| `mirror` | `Mirror` | [Server](#server) |
|
||||
| `engines` | `map[string]string` | [Server](#server) |
|
||||
| `backup_dir` | `string` | [Server](#server) |
|
||||
| `router` | `Router` | [Server](#server) |
|
||||
| `control` | `Control` | [Server](#server) |
|
||||
| `models` | `[]catalog.Model` | [Server](#server) |
|
||||
@@ -1265,6 +1266,14 @@ never trusted to already have what somebody once put there.
|
||||
Unset keeps DefaultEngines. A version replaces the default; an empty
|
||||
one leaves that engine to be installed by hand.
|
||||
|
||||
### `backup_dir`
|
||||
|
||||
`string`
|
||||
|
||||
BackupDir is where `llmgw server backup` writes, and where the newest
|
||||
archive's age is read for llmgw_backup_last_success_timestamp_seconds.
|
||||
Empty means /var/backups/llmgw-server.
|
||||
|
||||
### `router`
|
||||
|
||||
`Router`
|
||||
|
||||
@@ -15,6 +15,7 @@ import (
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"fmt"
|
||||
"runtime"
|
||||
"strings"
|
||||
|
||||
"golang.org/x/crypto/argon2"
|
||||
@@ -56,7 +57,7 @@ func HashPassword(password string) (string, error) {
|
||||
if _, err := rand.Read(salt); err != nil {
|
||||
return "", err
|
||||
}
|
||||
sum := argon2.IDKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
|
||||
sum := idKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
|
||||
|
||||
b64 := base64.RawStdEncoding.EncodeToString
|
||||
return fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s",
|
||||
@@ -93,7 +94,7 @@ func VerifyPassword(password, encoded string) (bool, error) {
|
||||
return false, ErrBadHash
|
||||
}
|
||||
|
||||
got := argon2.IDKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
|
||||
got := idKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
|
||||
return subtle.ConstantTimeCompare(got, want) == 1, nil
|
||||
}
|
||||
|
||||
@@ -110,3 +111,16 @@ func init() {
|
||||
}
|
||||
dummyHash = h
|
||||
}
|
||||
|
||||
// argonSlots bounds how many hashes run at once. Each takes 64 MiB, and the
|
||||
// login and signup forms are on the client-facing port: without a bound, a
|
||||
// few hundred parallel attempts at random addresses asked for gigabytes at
|
||||
// once, and the per-address throttle does not see them — each address is new.
|
||||
// Beyond the bound, attempts wait their turn instead.
|
||||
var argonSlots = make(chan struct{}, max(2, runtime.NumCPU()/2))
|
||||
|
||||
func idKey(password, salt []byte, time, memory uint32, threads uint8, keyLen uint32) []byte {
|
||||
argonSlots <- struct{}{}
|
||||
defer func() { <-argonSlots }()
|
||||
return argon2.IDKey(password, salt, time, memory, threads, keyLen)
|
||||
}
|
||||
@@ -230,6 +230,22 @@ func (s *Service) allowAttempt(email string) bool {
|
||||
defer s.mu.Unlock()
|
||||
|
||||
w, ok := s.attempts[email]
|
||||
if !ok && len(s.attempts) >= maxTrackedAddresses {
|
||||
// Full of addresses nobody has signed in as for a while: forget the
|
||||
// ones whose window has passed. Only expiring, never evicting early —
|
||||
// dropping a live window is how an attacker spraying random addresses
|
||||
// would buy unlimited guesses at the one they care about.
|
||||
for k, v := range s.attempts {
|
||||
if now.Sub(v.since) > s.cfg.AttemptWindow {
|
||||
delete(s.attempts, k)
|
||||
}
|
||||
}
|
||||
if len(s.attempts) >= maxTrackedAddresses {
|
||||
// Still full, so somebody is spraying addresses faster than they
|
||||
// expire. New ones wait; those already tracked are unaffected.
|
||||
return false
|
||||
}
|
||||
}
|
||||
if !ok || now.Sub(w.since) > s.cfg.AttemptWindow {
|
||||
s.attempts[email] = &attemptWindow{count: 1, since: now}
|
||||
return true
|
||||
@@ -238,6 +254,10 @@ func (s *Service) allowAttempt(email string) bool {
|
||||
return w.count <= s.cfg.MaxAttempts
|
||||
}
|
||||
|
||||
// maxTrackedAddresses bounds the throttle's memory. Unbounded, it grew by one
|
||||
// entry per address tried and shrank only on a successful sign-in.
|
||||
const maxTrackedAddresses = 50000
|
||||
|
||||
func (s *Service) clearAttempts(email string) {
|
||||
s.mu.Lock()
|
||||
delete(s.attempts, email)
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
package account
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestSprayingAddressesCannotGrowTheThrottleOrEscapeIt(t *testing.T) {
|
||||
now := time.Unix(0, 0)
|
||||
s := &Service{cfg: DefaultConfig(), attempts: map[string]*attemptWindow{}, now: func() time.Time { return now }}
|
||||
|
||||
// The target is being guessed at.
|
||||
for i := 0; i < s.cfg.MaxAttempts; i++ {
|
||||
s.allowAttempt("victim@example.com")
|
||||
}
|
||||
// Then the table is flooded with made-up addresses.
|
||||
for i := 0; i < maxTrackedAddresses+100; i++ {
|
||||
s.allowAttempt(fmt.Sprintf("x%d@example.com", i))
|
||||
}
|
||||
if n := len(s.attempts); n > maxTrackedAddresses {
|
||||
t.Errorf("the table grew to %d", n)
|
||||
}
|
||||
if s.allowAttempt("victim@example.com") {
|
||||
t.Error("flooding the table reset the target's throttle")
|
||||
}
|
||||
|
||||
// Once the window passes, everything expires and sign-ins work again.
|
||||
now = now.Add(s.cfg.AttemptWindow + time.Second)
|
||||
if !s.allowAttempt("new@example.com") {
|
||||
t.Error("a new address was refused after the flood expired")
|
||||
}
|
||||
}
|
||||
@@ -400,11 +400,16 @@ func TestHeartbeatKeepsNodeAlive(t *testing.T) {
|
||||
n, _ := h.inv.Get("n1")
|
||||
first := n.LastSeen
|
||||
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
n, _ = h.inv.Get("n1")
|
||||
if !n.LastSeen.After(first) {
|
||||
t.Error("heartbeats are not refreshing LastSeen; the node would be declared lost")
|
||||
// Waited for rather than slept through: four heartbeat intervals was not
|
||||
// always enough on a loaded runner for one to have landed.
|
||||
deadline := time.Now().Add(3 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
if n, _ = h.inv.Get("n1"); n.LastSeen.After(first) {
|
||||
return
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
t.Error("heartbeats are not refreshing LastSeen; the node would be declared lost")
|
||||
}
|
||||
|
||||
func TestStartAndStopShardEndToEnd(t *testing.T) {
|
||||
|
||||
@@ -523,6 +523,11 @@ func (in *Installer) download(ctx context.Context, repo, tag, asset, dst, want s
|
||||
// untarGz unpacks an archive under dir, refusing anything that would land
|
||||
// outside it. A release archive is somebody else's file, and a path of
|
||||
// "../../etc/cron.d/x" in it would otherwise be written as root.
|
||||
//
|
||||
// Every write goes through an os.Root, which refuses to leave dir however the
|
||||
// path gets there. Checking names alone was not enough: "a -> ." followed by
|
||||
// "a/a/b -> ../.." reads as inside and resolves two levels above it, and a
|
||||
// file written through that lands wherever the links point.
|
||||
func untarGz(archive, dir string) error {
|
||||
f, err := os.Open(archive) //nolint:gosec // inside the install root
|
||||
if err != nil {
|
||||
@@ -533,6 +538,11 @@ func untarGz(archive, dir string) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
root, err := os.OpenRoot(dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer root.Close()
|
||||
tr := tar.NewReader(gz)
|
||||
for {
|
||||
h, err := tr.Next()
|
||||
@@ -542,22 +552,22 @@ func untarGz(archive, dir string) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
target, err := within(dir, h.Name)
|
||||
if err != nil {
|
||||
name := filepath.Clean(h.Name)
|
||||
if _, err := within(dir, name); err != nil {
|
||||
return err
|
||||
}
|
||||
switch h.Typeflag {
|
||||
case tar.TypeDir:
|
||||
if err := os.MkdirAll(target, 0o755); err != nil {
|
||||
if err := root.MkdirAll(name, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
case tar.TypeReg:
|
||||
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
|
||||
if err := root.MkdirAll(filepath.Dir(name), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
out, err := os.OpenFile(target, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, fs.FileMode(h.Mode)&0o755) //nolint:gosec // checked above
|
||||
out, err := root.OpenFile(name, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, fs.FileMode(h.Mode)&0o755)
|
||||
if err != nil {
|
||||
return err
|
||||
return fmt.Errorf("%s: %w", h.Name, err)
|
||||
}
|
||||
if _, err := io.Copy(out, tr); err != nil { //nolint:gosec // a release archive, size bounded by its checksum
|
||||
out.Close()
|
||||
@@ -567,19 +577,19 @@ func untarGz(archive, dir string) error {
|
||||
return err
|
||||
}
|
||||
case tar.TypeSymlink:
|
||||
// Libraries are versioned through symlinks, and those are
|
||||
// relative. One pointing anywhere else is refused.
|
||||
if filepath.IsAbs(h.Linkname) {
|
||||
return fmt.Errorf("%s links outside the archive", h.Name)
|
||||
// Libraries are versioned through symlinks to a file beside them —
|
||||
// libllama.so -> libllama.so.0 — and that is all a release needs.
|
||||
// A target with a separator in it could point anywhere once other
|
||||
// links are followed, and a binary found through one would be run
|
||||
// as root, so it is refused rather than reasoned about.
|
||||
if h.Linkname == "" || strings.ContainsAny(h.Linkname, `/\`) || h.Linkname == ".." || h.Linkname == "." {
|
||||
return fmt.Errorf("%s links to %q; only a file beside it is allowed", h.Name, h.Linkname)
|
||||
}
|
||||
if _, err := within(dir, filepath.Join(filepath.Dir(h.Name), h.Linkname)); err != nil {
|
||||
return fmt.Errorf("%s links outside the archive", h.Name)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
|
||||
if err := root.MkdirAll(filepath.Dir(name), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
_ = os.Remove(target)
|
||||
if err := os.Symlink(h.Linkname, target); err != nil {
|
||||
_ = root.Remove(name)
|
||||
if err := root.Symlink(h.Linkname, name); err != nil {
|
||||
return err
|
||||
}
|
||||
default:
|
||||
|
||||
@@ -342,3 +342,49 @@ func TestAVersionThatIsAPathIsRefused(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestChainedLinksCannotCarryAFileOutOfTheInstall(t *testing.T) {
|
||||
// Each link reads as inside the directory; together they point above it.
|
||||
var buf bytes.Buffer
|
||||
gz := gzip.NewWriter(&buf)
|
||||
tw := tar.NewWriter(gz)
|
||||
_ = tw.WriteHeader(&tar.Header{Name: "a", Linkname: ".", Typeflag: tar.TypeSymlink})
|
||||
_ = tw.WriteHeader(&tar.Header{Name: "a/a/b", Linkname: "../..", Typeflag: tar.TypeSymlink})
|
||||
_ = tw.WriteHeader(&tar.Header{Name: "a/a/b/escaped", Mode: 0o644, Size: 1, Typeflag: tar.TypeReg})
|
||||
_, _ = tw.Write([]byte("x"))
|
||||
_ = tw.Close()
|
||||
_ = gz.Close()
|
||||
|
||||
parent := t.TempDir()
|
||||
dir := filepath.Join(parent, "install")
|
||||
_ = os.MkdirAll(dir, 0o755)
|
||||
archive := filepath.Join(parent, "evil.tar.gz")
|
||||
_ = os.WriteFile(archive, buf.Bytes(), 0o644)
|
||||
|
||||
if err := untarGz(archive, dir); err == nil {
|
||||
t.Error("an archive of chained links was unpacked")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(parent, "escaped")); err == nil {
|
||||
t.Fatal("a file was written outside the install directory")
|
||||
}
|
||||
}
|
||||
|
||||
func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
gz := gzip.NewWriter(&buf)
|
||||
tw := tar.NewWriter(gz)
|
||||
_ = tw.WriteHeader(&tar.Header{Name: "lib/libllama.so.0", Mode: 0o755, Size: 1, Typeflag: tar.TypeReg})
|
||||
_, _ = tw.Write([]byte("x"))
|
||||
_ = tw.WriteHeader(&tar.Header{Name: "lib/libllama.so", Linkname: "libllama.so.0", Typeflag: tar.TypeSymlink})
|
||||
_ = tw.Close()
|
||||
_ = gz.Close()
|
||||
dir := t.TempDir()
|
||||
archive := filepath.Join(t.TempDir(), "ok.tar.gz")
|
||||
_ = os.WriteFile(archive, buf.Bytes(), 0o644)
|
||||
if err := untarGz(archive, dir); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if target, err := os.Readlink(filepath.Join(dir, "lib", "libllama.so")); err != nil || target != "libllama.so.0" {
|
||||
t.Errorf("link %q %v", target, err)
|
||||
}
|
||||
}
|
||||
@@ -243,7 +243,9 @@ func TestWaitReadyTimeoutIncludesLogs(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
time.Sleep(100 * time.Millisecond) // let the line be captured
|
||||
eventually(t, 3*time.Second, "the line was never captured", func() bool {
|
||||
return strings.Contains(strings.Join(sh.Logs(10), "\n"), "CUDA out of memory")
|
||||
})
|
||||
|
||||
err = s.WaitReady(t.Context(), sh, "http://x", "m", 200*time.Millisecond)
|
||||
if err == nil {
|
||||
@@ -295,7 +297,10 @@ func TestLogsAreCaptured(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
time.Sleep(400 * time.Millisecond)
|
||||
eventually(t, 3*time.Second, "not every line arrived", func() bool {
|
||||
l := strings.Join(sh.Logs(10), "\n")
|
||||
return strings.Contains(l, "line1") && strings.Contains(l, "line2") && strings.Contains(l, "line3")
|
||||
})
|
||||
|
||||
logs := strings.Join(sh.Logs(10), "\n")
|
||||
for _, want := range []string{"line1", "line2", "line3"} {
|
||||
@@ -372,11 +377,11 @@ func TestStateFileRemovedOnExit(t *testing.T) {
|
||||
|
||||
sh, _ := s.Start(t.Context(), "a", engines.ShardSpec{}, shellCommand("exit 0"))
|
||||
waitState(t, sh, Failed, 3*time.Second)
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
|
||||
if _, err := os.Stat(s.statePath("a")); !os.IsNotExist(err) {
|
||||
t.Error("the state file outlived the process; a later scan would find a dead orphan")
|
||||
}
|
||||
eventually(t, 3*time.Second,
|
||||
"the state file outlived the process; a later scan would find a dead orphan", func() bool {
|
||||
_, err := os.Stat(s.statePath("a"))
|
||||
return os.IsNotExist(err)
|
||||
})
|
||||
}
|
||||
|
||||
// --- state machine and ring buffer ---
|
||||
@@ -743,7 +748,9 @@ func TestWaitAliveFailsOnEarlyExit(t *testing.T) {
|
||||
if err := s.WaitAlive(t.Context(), sh); err == nil {
|
||||
t.Fatal("a worker that died immediately was reported ready")
|
||||
}
|
||||
if d := time.Since(start); d > 1500*time.Millisecond {
|
||||
// Below AliveGrace (5s), which is what waiting it out would take; the
|
||||
// headroom is for a loaded runner, where 1.5s was sometimes not enough.
|
||||
if d := time.Since(start); d > 4*time.Second {
|
||||
t.Errorf("waited %v for a process that had already exited", d)
|
||||
}
|
||||
}
|
||||
@@ -827,3 +834,17 @@ func TestAWorkersOnlyProcessWaitsForTheLeader(t *testing.T) {
|
||||
t.Errorf("the worker started after %v, before the leader was listening", d)
|
||||
}
|
||||
}
|
||||
|
||||
// eventually waits for cond, failing with msg if it does not hold within
|
||||
// limit. Sleeping a fixed time and then checking measured the machine the
|
||||
// test ran on, and a loaded CI runner under the race detector lost that race.
|
||||
func eventually(t *testing.T, limit time.Duration, msg string, cond func() bool) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(limit)
|
||||
for !cond() {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal(msg)
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,312 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
)
|
||||
|
||||
// DefaultBackupDir is where backups are written and where the control plane
|
||||
// looks for the newest one.
|
||||
//
|
||||
// Its own directory rather than /var/backups/llmgw, which a PostgreSQL move
|
||||
// had already made for its own dumps, owned by postgres and closed to
|
||||
// everyone else — so the control plane could not see a backup landing in it.
|
||||
const DefaultBackupDir = "/var/backups/llmgw-server"
|
||||
|
||||
func newBackupCmd() *cobra.Command {
|
||||
var (
|
||||
configPath string
|
||||
etcDir string
|
||||
dir string
|
||||
keep int
|
||||
)
|
||||
c := &cobra.Command{
|
||||
Use: "backup",
|
||||
Short: "Write the store and /etc/llmgw to one archive, and keep the last few",
|
||||
Long: "Everything a control plane cannot be rebuilt without: the store — keys,\n" +
|
||||
"people, quotas, usage, the models added at runtime, the secret engine\n" +
|
||||
"keys derive from — and /etc/llmgw, which holds the configuration, the\n" +
|
||||
"store's address and the certificate authority's key.\n\n" +
|
||||
"The archive holds that key, so it is written readable by root only. It\n" +
|
||||
"is on the same disk as what it backs up: copy it elsewhere, or it does\n" +
|
||||
"not survive the loss of this machine.\n\n" +
|
||||
"Run daily by llmgw-backup.timer (`llmgw server backup`). The control plane publishes the time of\n" +
|
||||
"the newest archive as llmgw_backup_last_success_timestamp_seconds, and\n" +
|
||||
"BackupTooOld fires when it is more than a day old.",
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
cfg, err := config.Load(configPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
path, err := writeBackup(cmd.Context(), cfg.Auth.Store, etcDir, dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if why := unlistable(dir); why != "" {
|
||||
fmt.Fprintf(cmd.ErrOrStderr(), "warning: %s; the control plane will report no backup "+
|
||||
"and BackupTooOld will fire\n", why)
|
||||
}
|
||||
removed, err := pruneBackups(dir, keep)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "wrote %s", path)
|
||||
if removed > 0 {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "; removed %d older", removed)
|
||||
}
|
||||
fmt.Fprintln(cmd.OutOrStdout())
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(&configPath, "config", "/etc/llmgw/server.yaml", "the control plane's configuration")
|
||||
c.Flags().StringVar(&etcDir, "etc", "/etc/llmgw", "configuration directory to include")
|
||||
c.Flags().StringVar(&dir, "dir", DefaultBackupDir, "where archives are written")
|
||||
c.Flags().IntVar(&keep, "keep", 14, "how many archives to keep")
|
||||
return c
|
||||
}
|
||||
|
||||
const backupPrefix = "llmgw-backup-"
|
||||
|
||||
// writeBackup writes one archive and returns its path. Written beside its
|
||||
// final name and renamed, so a backup interrupted halfway is never the newest
|
||||
// one the control plane reports.
|
||||
func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, error) {
|
||||
// Listable by anyone, so the control plane — not root — can see when the
|
||||
// newest archive was written. The archives themselves are root's alone.
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return "", err
|
||||
}
|
||||
host, _ := os.Hostname()
|
||||
name := backupPrefix + host + "-" + time.Now().UTC().Format("20060102T150405Z") + ".tar.gz"
|
||||
final := filepath.Join(dir, name)
|
||||
part := final + ".part"
|
||||
|
||||
f, err := os.OpenFile(part, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o600)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
ok := false
|
||||
defer func() {
|
||||
if !ok {
|
||||
_ = os.Remove(part)
|
||||
}
|
||||
}()
|
||||
gz := gzip.NewWriter(f)
|
||||
tw := tar.NewWriter(gz)
|
||||
|
||||
manifest := fmt.Sprintf("llmgw %s\nhost %s\ntime %s\n", version.Version, host,
|
||||
time.Now().UTC().Format(time.RFC3339))
|
||||
if err := addBytes(tw, "MANIFEST", []byte(manifest)); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if storeAddr != "" {
|
||||
if err := addStore(ctx, tw, storeAddr, dir); err != nil {
|
||||
return "", fmt.Errorf("the store: %w", err)
|
||||
}
|
||||
}
|
||||
if err := addTree(tw, etcDir, "etc/llmgw"); err != nil {
|
||||
return "", fmt.Errorf("%s: %w", etcDir, err)
|
||||
}
|
||||
if err := errors.Join(tw.Close(), gz.Close(), f.Sync(), f.Close()); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := os.Rename(part, final); err != nil {
|
||||
return "", err
|
||||
}
|
||||
ok = true
|
||||
return final, nil
|
||||
}
|
||||
|
||||
// addStore dumps the store into the archive.
|
||||
func addStore(ctx context.Context, tw *tar.Writer, addr, scratch string) error {
|
||||
addr, err := store.ResolveAddress(addr)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Inside a directory only its owner can enter. The archive directory is
|
||||
// listable on purpose, and VACUUM INTO creates its file with the default
|
||||
// mode: made there directly, a full copy of the store — password hashes,
|
||||
// the engine-key secret — sat readable by any local user for as long as
|
||||
// the backup took.
|
||||
tmpDir, err := os.MkdirTemp(scratch, ".store-")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer os.RemoveAll(tmpDir)
|
||||
tmpPath := filepath.Join(tmpDir, "store")
|
||||
|
||||
name := "store.sqlite"
|
||||
if store.IsPostgres(addr) {
|
||||
name = "store.pgdump"
|
||||
if err := pgDump(ctx, addr, tmpPath); err != nil {
|
||||
return err
|
||||
}
|
||||
} else {
|
||||
db, err := store.Open(ctx, addr)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
err = db.SnapshotTo(ctx, tmpPath)
|
||||
_ = db.Close()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return addFile(tw, tmpPath, name)
|
||||
}
|
||||
|
||||
// pgDump runs pg_dump in its custom format, which pg_restore can restore
|
||||
// selectively. The connection is given through the environment, not the
|
||||
// command line: a DSN carries the password, and every user on the machine
|
||||
// can read another process's arguments.
|
||||
func pgDump(ctx context.Context, dsn, out string) error {
|
||||
u, err := url.Parse(dsn)
|
||||
if err != nil {
|
||||
return fmt.Errorf("the store address is not a URL")
|
||||
}
|
||||
env := os.Environ()
|
||||
set := func(k, v string) {
|
||||
if v != "" {
|
||||
env = append(env, k+"="+v)
|
||||
}
|
||||
}
|
||||
set("PGHOST", u.Hostname())
|
||||
set("PGPORT", u.Port())
|
||||
set("PGUSER", u.User.Username())
|
||||
if pw, ok := u.User.Password(); ok {
|
||||
set("PGPASSWORD", pw)
|
||||
}
|
||||
set("PGDATABASE", strings.TrimPrefix(u.Path, "/"))
|
||||
set("PGSSLMODE", u.Query().Get("sslmode"))
|
||||
cmd := exec.CommandContext(ctx, "pg_dump", "--format=custom", "--no-owner", "--file="+out) //nolint:gosec // fixed binary, our own file
|
||||
cmd.Env = env
|
||||
if b, err := cmd.CombinedOutput(); err != nil {
|
||||
return fmt.Errorf("pg_dump: %v: %s", err, strings.TrimSpace(string(b)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func addTree(tw *tar.Writer, root, as string) error {
|
||||
return filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
rel, _ := filepath.Rel(root, p)
|
||||
name := filepath.ToSlash(filepath.Join(as, rel))
|
||||
info, err := d.Info()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch {
|
||||
case d.IsDir():
|
||||
return tw.WriteHeader(&tar.Header{Name: name + "/", Mode: int64(info.Mode().Perm()),
|
||||
Typeflag: tar.TypeDir, ModTime: info.ModTime()})
|
||||
case info.Mode().IsRegular():
|
||||
return addFile(tw, p, name)
|
||||
}
|
||||
return nil // sockets, links: not configuration
|
||||
})
|
||||
}
|
||||
|
||||
func addFile(tw *tar.Writer, path, name string) error {
|
||||
f, err := os.Open(path) //nolint:gosec // files we chose to back up
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
info, err := f.Stat()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: int64(info.Mode().Perm()),
|
||||
Size: info.Size(), ModTime: info.ModTime(), Typeflag: tar.TypeReg}); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = io.Copy(tw, f)
|
||||
return err
|
||||
}
|
||||
|
||||
func addBytes(tw *tar.Writer, name string, b []byte) error {
|
||||
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o600, Size: int64(len(b)),
|
||||
ModTime: time.Now(), Typeflag: tar.TypeReg}); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err := tw.Write(b)
|
||||
return err
|
||||
}
|
||||
|
||||
// pruneBackups keeps the newest keep archives. Only files this command
|
||||
// wrote are considered, so nothing else in the directory is ever removed.
|
||||
func pruneBackups(dir string, keep int) (int, error) {
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
var names []string
|
||||
for _, e := range entries {
|
||||
if n := e.Name(); strings.HasPrefix(n, backupPrefix) && strings.HasSuffix(n, ".tar.gz") {
|
||||
names = append(names, n)
|
||||
}
|
||||
}
|
||||
// The name carries a UTC timestamp, so name order is age order.
|
||||
slices.Sort(names)
|
||||
removed := 0
|
||||
for len(names) > max(keep, 1) {
|
||||
if err := os.Remove(filepath.Join(dir, names[0])); err != nil {
|
||||
return removed, err
|
||||
}
|
||||
names = names[1:]
|
||||
removed++
|
||||
}
|
||||
return removed, nil
|
||||
}
|
||||
|
||||
// NewestBackup is the modification time of the newest complete archive in
|
||||
// dir, or zero when there is none.
|
||||
func NewestBackup(dir string) time.Time {
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return time.Time{}
|
||||
}
|
||||
var newest time.Time
|
||||
for _, e := range entries {
|
||||
if n := e.Name(); !strings.HasPrefix(n, backupPrefix) || !strings.HasSuffix(n, ".tar.gz") {
|
||||
continue
|
||||
}
|
||||
if info, err := e.Info(); err == nil && info.ModTime().After(newest) {
|
||||
newest = info.ModTime()
|
||||
}
|
||||
}
|
||||
return newest
|
||||
}
|
||||
|
||||
// unlistable says why the control plane, which does not run as root, could
|
||||
// not see the archives in dir, or "" when it can.
|
||||
func unlistable(dir string) string {
|
||||
info, err := os.Stat(dir)
|
||||
if err != nil {
|
||||
return err.Error()
|
||||
}
|
||||
if info.Mode().Perm()&0o005 != 0o005 {
|
||||
return fmt.Sprintf("%s is %v, so only its owner can list it", dir, info.Mode().Perm())
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"compress/gzip"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func TestABackupHoldsTheStoreAndTheConfiguration(t *testing.T) {
|
||||
tmp := t.TempDir()
|
||||
dbPath := filepath.Join(tmp, "store.db")
|
||||
db, err := store.Open(t.Context(), dbPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
secret, err := db.ClusterSecret(t.Context(), "engine-key")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = db.Close()
|
||||
|
||||
etc := filepath.Join(tmp, "etc")
|
||||
_ = os.MkdirAll(filepath.Join(etc, "pki"), 0o700)
|
||||
_ = os.WriteFile(filepath.Join(etc, "server.yaml"), []byte("listen: :8080\n"), 0o644)
|
||||
_ = os.WriteFile(filepath.Join(etc, "pki", "ca-key.pem"), []byte("KEY"), 0o600)
|
||||
|
||||
out := filepath.Join(tmp, "backups")
|
||||
path, err := writeBackup(t.Context(), dbPath, etc, out)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
info, _ := os.Stat(path)
|
||||
if info.Mode().Perm() != 0o600 {
|
||||
t.Errorf("the archive holds the CA's key and is %v", info.Mode().Perm())
|
||||
}
|
||||
|
||||
files := map[string][]byte{}
|
||||
f, _ := os.Open(path)
|
||||
defer f.Close()
|
||||
gz, err := gzip.NewReader(f)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tr := tar.NewReader(gz)
|
||||
for {
|
||||
h, err := tr.Next()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := io.ReadAll(tr)
|
||||
files[h.Name] = b
|
||||
}
|
||||
for _, want := range []string{"MANIFEST", "store.sqlite", "etc/llmgw/server.yaml", "etc/llmgw/pki/ca-key.pem"} {
|
||||
if _, ok := files[want]; !ok {
|
||||
t.Errorf("%s is not in the archive", want)
|
||||
}
|
||||
}
|
||||
|
||||
// And the store in it is a working store, with what was in it.
|
||||
restored := filepath.Join(tmp, "restored.db")
|
||||
_ = os.WriteFile(restored, files["store.sqlite"], 0o600)
|
||||
rdb, err := store.Open(t.Context(), restored)
|
||||
if err != nil {
|
||||
t.Fatalf("the backed-up store does not open: %v", err)
|
||||
}
|
||||
defer rdb.Close()
|
||||
if got, _ := rdb.ClusterSecret(t.Context(), "engine-key"); got != secret {
|
||||
t.Error("the restored store lost the engine key secret; every engine would be locked out")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOldBackupsArePrunedAndTheNewestIsFound(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
for i, stamp := range []string{"20260101T000000Z", "20260102T000000Z", "20260103T000000Z"} {
|
||||
p := filepath.Join(dir, backupPrefix+"h-"+stamp+".tar.gz")
|
||||
_ = os.WriteFile(p, []byte("x"), 0o600)
|
||||
mt := time.Date(2026, 1, 1+i, 0, 0, 0, 0, time.UTC)
|
||||
_ = os.Chtimes(p, mt, mt)
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(dir, "someone-elses-file"), []byte("x"), 0o600)
|
||||
_ = os.WriteFile(filepath.Join(dir, backupPrefix+"h-20260104T000000Z.tar.gz.part"), []byte("x"), 0o600)
|
||||
|
||||
removed, err := pruneBackups(dir, 2)
|
||||
if err != nil || removed != 1 {
|
||||
t.Fatalf("removed %d, %v", removed, err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "someone-elses-file")); err != nil {
|
||||
t.Error("a file the backup did not write was removed")
|
||||
}
|
||||
newest := NewestBackup(dir)
|
||||
if !newest.Equal(time.Date(2026, 1, 3, 0, 0, 0, 0, time.UTC)) {
|
||||
t.Errorf("newest %v; an interrupted .part must not count", newest)
|
||||
}
|
||||
if !NewestBackup(filepath.Join(dir, "none")).IsZero() {
|
||||
t.Error("a missing directory reported a backup")
|
||||
}
|
||||
}
|
||||
|
||||
func TestABackupDirectoryTheControlPlaneCannotListIsSaid(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
_ = os.Chmod(dir, 0o700)
|
||||
if unlistable(dir) == "" {
|
||||
t.Error("a directory only its owner can list was taken as visible")
|
||||
}
|
||||
_ = os.Chmod(dir, 0o755)
|
||||
if why := unlistable(dir); why != "" {
|
||||
t.Errorf("a listable directory: %s", why)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoScratchCopyOfTheStoreIsLeftBehind(t *testing.T) {
|
||||
// The snapshot is made before it is archived. Made straight into the
|
||||
// listable archive directory, it was readable by anyone for that time.
|
||||
tmp := t.TempDir()
|
||||
dbPath := filepath.Join(tmp, "store.db")
|
||||
db, err := store.Open(t.Context(), dbPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = db.Close()
|
||||
out := filepath.Join(tmp, "backups")
|
||||
if _, err := writeBackup(t.Context(), dbPath, t.TempDir(), out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
entries, _ := os.ReadDir(out)
|
||||
for _, e := range entries {
|
||||
if strings.HasPrefix(e.Name(), ".store") {
|
||||
t.Errorf("a scratch copy of the store was left behind: %s", e.Name())
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"fmt"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
@@ -132,6 +133,23 @@ func newServerCmd() *cobra.Command {
|
||||
// On the same /metrics as the traffic figures: correlating
|
||||
// "the cluster was full" with "requests were queueing" should
|
||||
// not need two systems.
|
||||
// When the newest backup was written, so its absence alerts
|
||||
// rather than being found on the day it is needed.
|
||||
backupDir := cfg.BackupDir
|
||||
if backupDir == "" {
|
||||
backupDir = DefaultBackupDir
|
||||
}
|
||||
if err := rt.RegisterMetrics(prometheus.NewGaugeFunc(prometheus.GaugeOpts{
|
||||
Name: "llmgw_backup_last_success_timestamp_seconds",
|
||||
Help: "When the newest backup archive was written, as a Unix time; 0 when there is none.",
|
||||
}, func() float64 {
|
||||
if t := NewestBackup(backupDir); !t.IsZero() {
|
||||
return float64(t.Unix())
|
||||
}
|
||||
return 0
|
||||
})); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := rt.RegisterMetrics(tlsconf.ExpiryCollector(map[string]string{
|
||||
"agent-server": cfg.AgentTLS.Cert, "agent-ca": cfg.AgentTLS.CA,
|
||||
})); err != nil {
|
||||
@@ -367,6 +385,7 @@ func newServerCmd() *cobra.Command {
|
||||
f.StringArrayVar(&endpoints, "endpoint", nil, "static endpoint: id=URL=model[,model] (repeatable)")
|
||||
|
||||
cmd.AddCommand(newServerInitCmd())
|
||||
cmd.AddCommand(newBackupCmd())
|
||||
return cmd
|
||||
}
|
||||
|
||||
|
||||
@@ -81,6 +81,11 @@ type Server struct {
|
||||
// one leaves that engine to be installed by hand.
|
||||
Engines map[string]string `koanf:"engines"`
|
||||
|
||||
// BackupDir is where `llmgw server backup` writes, and where the newest
|
||||
// archive's age is read for llmgw_backup_last_success_timestamp_seconds.
|
||||
// Empty means /var/backups/llmgw-server.
|
||||
BackupDir string `koanf:"backup_dir"`
|
||||
|
||||
Router Router `koanf:"router"`
|
||||
Control Control `koanf:"control"`
|
||||
Models []catalog.Model `koanf:"models"`
|
||||
|
||||
@@ -42,14 +42,20 @@ func newController(t *testing.T, cfg Config, sat SaturationFunc) *Controller {
|
||||
|
||||
func TestAdmitsImmediatelyWhenIdle(t *testing.T) {
|
||||
s := &fixedSat{}
|
||||
c := newController(t, testConfig(), s.fn)
|
||||
// A queue that is looked at every ten seconds, so a request that went
|
||||
// through it rather than the fast path would take that long. A bound of
|
||||
// 50ms against a 1ms tick measured the machine the test ran on, and a
|
||||
// loaded CI runner under the race detector was slower than that.
|
||||
cfg := testConfig()
|
||||
cfg.Interval = 10 * time.Second
|
||||
c := newController(t, cfg, s.fn)
|
||||
|
||||
start := time.Now()
|
||||
lease, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if d := time.Since(start); d > 50*time.Millisecond {
|
||||
if d := time.Since(start); d > 2*time.Second {
|
||||
t.Errorf("idle admission took %v; the fast path is not being used", d)
|
||||
}
|
||||
lease.Release()
|
||||
@@ -131,8 +137,13 @@ func TestQueuedRequestAdmittedWhenSaturationDrops(t *testing.T) {
|
||||
done <- err
|
||||
}()
|
||||
|
||||
// Give it time to actually queue, then open the gate.
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
// Wait for it to actually queue, then open the gate. Waited for, not
|
||||
// slept through: on a loaded runner 50ms was sometimes not long enough
|
||||
// for the goroutine to have been scheduled at all.
|
||||
deadline := time.Now().Add(2 * time.Second)
|
||||
for c.Depths()[3].Depth != 1 && time.Now().Before(deadline) {
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
if d := c.Depths(); d[3].Depth != 1 {
|
||||
t.Fatalf("expected 1 queued bronze request, got %+v", d)
|
||||
}
|
||||
|
||||
@@ -51,7 +51,7 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
|
||||
case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded):
|
||||
// The client gave up while queued; nobody is listening.
|
||||
r.m.observeAdmissionRejected(req.Band, "client_gone")
|
||||
r.m.observeRequest(req.Model, "", 499, time.Since(start), nil, 0)
|
||||
r.m.observeRequest(r.metricModel(req.Model), "", 499, time.Since(start), nil, 0)
|
||||
return
|
||||
case errors.Is(err, admission.ErrQueueTimeout):
|
||||
status = http.StatusGatewayTimeout
|
||||
@@ -67,7 +67,7 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
|
||||
}
|
||||
r.m.observeAdmissionRejected(req.Band, reason)
|
||||
writeError(w, status, "server_error", err.Error())
|
||||
r.m.observeRequest(req.Model, "", status, time.Since(start), nil, 0)
|
||||
r.m.observeRequest(r.metricModel(req.Model), "", status, time.Since(start), nil, 0)
|
||||
}
|
||||
|
||||
// prefixBytes is how much of the prompt is kept for prefix-aware routing.
|
||||
@@ -289,7 +289,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
|
||||
if _, entry := r.resolve(parsed.Model); entry.ContextLength > 0 {
|
||||
if why := checkContextFits(entry, parsed.PromptTokens, parsed.MaxTokens); why != "" {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", why)
|
||||
r.m.observeRequest(parsed.Model, "", http.StatusBadRequest, time.Since(start), nil, 0)
|
||||
r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
|
||||
return
|
||||
}
|
||||
}
|
||||
@@ -297,7 +297,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
|
||||
if p, ok := auth.FromContext(req.Context()); ok {
|
||||
if !r.permits(p, parsed.Model) {
|
||||
auth.WriteForbiddenModel(w, parsed.Model)
|
||||
r.m.observeRequest(parsed.Model, "", http.StatusForbidden, time.Since(start), nil, 0)
|
||||
r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusForbidden, time.Since(start), nil, 0)
|
||||
return
|
||||
}
|
||||
// Limits are checked before admission, not after: a request that is
|
||||
@@ -339,7 +339,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
|
||||
}
|
||||
defer lease.Release()
|
||||
queued := time.Since(start)
|
||||
r.m.observeAdmission(lease.Band(), parsed.Model, queued)
|
||||
r.m.observeAdmission(lease.Band(), r.metricModel(parsed.Model), queued)
|
||||
if queued > time.Millisecond {
|
||||
w.Header().Set("x-llmgw-queue-ms", strconv.FormatInt(queued.Milliseconds(), 10))
|
||||
}
|
||||
@@ -349,7 +349,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
|
||||
// model that will actually serve, not the one that was asked for.
|
||||
if dropped, err := r.loadThread(req.Context(), parsed); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", err.Error())
|
||||
r.m.observeRequest(parsed.Model, "", http.StatusBadRequest, time.Since(start), nil, 0)
|
||||
r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
|
||||
return
|
||||
} else if dropped > 0 {
|
||||
// Dropping turns is the right answer and must not be a silent one.
|
||||
@@ -420,7 +420,7 @@ func (r *Router) finish(w http.ResponseWriter, req *parsedRequest, res dispatchR
|
||||
writeError(w, status, "server_error", res.err.Error())
|
||||
}
|
||||
|
||||
r.m.observeRequest(req.Model, res.pool, status, time.Since(start), res.outcome.Usage, res.outcome.TTFT)
|
||||
r.m.observeRequest(r.metricModel(req.Model), res.pool, status, time.Since(start), res.outcome.Usage, res.outcome.TTFT)
|
||||
r.account(req, res, status, time.Since(start))
|
||||
}
|
||||
|
||||
@@ -467,7 +467,7 @@ func (r *Router) rejectLimit(w http.ResponseWriter, req *parsedRequest, err erro
|
||||
// already know how to surface that code. What differs is Retry-After,
|
||||
// which is absent when waiting does not help.
|
||||
writeError(w, http.StatusTooManyRequests, "rate_limit_exceeded", le.Error())
|
||||
r.m.observeRequest(req.Model, "", http.StatusTooManyRequests, time.Since(start), nil, 0)
|
||||
r.m.observeRequest(r.metricModel(req.Model), "", http.StatusTooManyRequests, time.Since(start), nil, 0)
|
||||
}
|
||||
|
||||
// parse reads the body and extracts the routing-relevant fields.
|
||||
|
||||
@@ -151,3 +151,17 @@ func (m *metrics) observeScores(d scheduler.Decision) {
|
||||
}
|
||||
m.considered.Observe(float64(d.Considered))
|
||||
}
|
||||
|
||||
// metricModel is the model label a request is counted under: the catalog's
|
||||
// name for it, or "unknown". Taken as sent, any key holder could mint a new
|
||||
// series per request — a refused request counted its made-up model name —
|
||||
// and grow Prometheus without bound.
|
||||
func (r *Router) metricModel(name string) string {
|
||||
if name == "" {
|
||||
return ""
|
||||
}
|
||||
if m, ok := r.modelMap()[name]; ok {
|
||||
return m.Name
|
||||
}
|
||||
return "unknown"
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
@@ -655,3 +656,32 @@ func TestAModelThatCannotComeUpIsSaidSoAtOnce(t *testing.T) {
|
||||
t.Errorf("the reason was not given: %s", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMadeUpModelNameDoesNotBecomeAMetricLabel(t *testing.T) {
|
||||
// Counted as sent, every request for a nonsense model minted a new series.
|
||||
e := newFakeEngine(t, "m")
|
||||
cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}})
|
||||
cfg.Models = []catalog.Model{{Name: "m", Aliases: []string{"mm"}}}
|
||||
rt := newTestRouter(t, cfg)
|
||||
|
||||
if got := rt.metricModel("mm"); got != "m" {
|
||||
t.Errorf("an alias is counted as %q, not the model it names", got)
|
||||
}
|
||||
for i := 0; i < 20; i++ {
|
||||
post(t, rt.Handler(), map[string]any{"model": fmt.Sprintf("junk-%d", i), "messages": []any{}}, nil)
|
||||
}
|
||||
// Twenty names, one series: all of them "unknown".
|
||||
reg := prometheus.NewRegistry()
|
||||
reg.MustRegister(rt.m.requests)
|
||||
mfs, err := reg.Gather()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
series := 0
|
||||
for _, mf := range mfs {
|
||||
series += len(mf.GetMetric())
|
||||
}
|
||||
if series != 1 {
|
||||
t.Errorf("%d series after twenty made-up names", series)
|
||||
}
|
||||
}
|
||||
@@ -175,7 +175,10 @@ func TestABrokenContractIsReportedOnceAndLoudly(t *testing.T) {
|
||||
in := endpoint.Info{ID: "e1", URL: srv.URL, Engine: "vllm", Models: []string{"m"}}
|
||||
reg.Replace([]endpoint.Info{in})
|
||||
|
||||
p := NewPoller(reg, Config{Interval: 20 * time.Millisecond, Timeout: 10 * time.Millisecond},
|
||||
// A generous timeout: a scrape that timed out would be reported as
|
||||
// unreachable, and the violation reported again once it answered — two
|
||||
// warnings where the test counts one, on a slow runner only.
|
||||
p := NewPoller(reg, Config{Interval: 20 * time.Millisecond, Timeout: 2 * time.Second},
|
||||
slog.New(cap))
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
@@ -59,3 +59,9 @@ func resolveAddress(address string) (string, error) {
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// ResolveAddress expands an @file store address, as Open does.
|
||||
func ResolveAddress(address string) (string, error) { return resolveAddress(address) }
|
||||
|
||||
// IsPostgres reports whether a resolved address names a PostgreSQL server.
|
||||
func IsPostgres(address string) bool { return isPostgres(address) }
|
||||
@@ -817,3 +817,14 @@ func (s *DB) UsageByModel(ctx context.Context, since time.Time) ([]ModelUsage, e
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// SnapshotTo writes a consistent copy of a SQLite store to path, while it is
|
||||
// in use. VACUUM INTO takes a read transaction, so the copy is of one moment
|
||||
// even with writes going on, which copying the file is not.
|
||||
func (s *DB) SnapshotTo(ctx context.Context, path string) error {
|
||||
if s.dialect != SQLiteDialect {
|
||||
return fmt.Errorf("a snapshot is for SQLite; back up PostgreSQL with pg_dump")
|
||||
}
|
||||
_, err := s.raw.ExecContext(ctx, `VACUUM INTO ?`, path)
|
||||
return err
|
||||
}
|
||||
Reference in new issue
Block a user