Compare commits

...
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 91fe24a105 Close what a second security review found still open
Release / build (push) Successful in 1m8s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m50s
A review of v0.10.4-rc.4, and a look at the live network, found:

- Alertmanager listening on every interface: from any node its API took
  silences, with no login. Bound to loopback on the monitoring host, and the
  monitoring README now installs it that way; Prometheus beside it is its only
  client, and one Alertmanager needs no cluster port.
- Login and signup unbounded in memory. Each attempt runs argon2 with 64 MiB,
  and nothing limited how many ran at once; the per-address throttle kept one
  entry per address ever tried. Hashing now takes one of a few slots and
  waits for one otherwise. The throttle expires entries whose window has
  passed when it fills, and if it is still full it refuses new addresses:
  evicting live ones instead would let a flood of made-up addresses reset the
  throttle on the address actually being guessed.
- Metric labels taken from the request. A refused request counted the model
  name it asked for, so any key holder could mint a series per request.
  Requests are counted under the catalog's name, or "unknown".
- Two in code written this week. A SQLite backup made its snapshot with the
  default mode in the listable backup directory, a full copy of the store
  readable while the backup ran; it is now made in a private directory, and
  the unit sets UMask=0077. And archive links were checked by name only, so
  "a -> ." then "a/a/b -> ../.." passed and a file written through them landed
  above the install directory, as root. Archives now unpack through os.Root,
  and a link may only point at a file beside it — which every llama.cpp
  b10991 archive, CPU and CUDA, was checked to follow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 02:25:07 +02:00
Ilker ManapandClaude Opus 5.5 9bf8256577 Keep backups where the control plane can see them
Release / build (push) Successful in 1m1s
Release / notes (push) Successful in 2s
CI / test (push) Successful in 3m35s
The first backup on the live control plane was written, and the control
plane reported none: /var/backups/llmgw already existed, made by an earlier
move to PostgreSQL for its own dumps, owned by postgres and 0700. MkdirAll
leaves an existing directory as it is, so the archive landed where the
llmgw user could not list it, and llmgw_backup_last_success_timestamp_seconds
said 0.

Opening that directory up would have exposed the dumps already in it, so
backups get their own, /var/backups/llmgw-server. And `llmgw server backup`
now warns when the directory it wrote to is one the control plane cannot
list, rather than leaving that to be found from an alert.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:50:33 +02:00
Ilker ManapandClaude Opus 5.5 1da4a2e1ef Back up the store and the certificate authority, and alert when it stops
Release / build (push) Successful in 1m1s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m32s
There were no backups: docs/install.md gave a pg_dump command, and nothing
ran it. The store holds every key, person, quota and usage record, the models
added at runtime, and now the secret each engine's key is derived from; the
certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk
lost all of it.

`llmgw server backup` writes both to one archive: pg_dump's custom format, or
a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's
key, so it is 0600; the directory is listable, so the control plane — not
root — can see when the newest was written. pg_dump is given the connection
through its environment, since a DSN carries the password and a process's
arguments are readable by every user. Written beside its name and renamed,
so an interrupted backup is never the newest; the newest 14 are kept, and
nothing the command did not write is ever removed.

llmgw-backup.timer runs it nightly and catches up after downtime. The control
plane publishes the newest archive's time, and BackupTooOld fires past 26
hours — or when there has never been one. The install guide now covers
restoring, and says to copy archives off the machine, since they share its
disk.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:44:32 +02:00
Ilker ManapandClaude Opus 5.5 b661bfead8 Wait for what tests check instead of sleeping and hoping
CI / test (push) Successful in 3m23s
CI's test job has failed intermittently — two of three runs on some days —
while every run here passed, including six repetitions on one CPU under the
race detector with eight packages at once. The difference is a shared,
loaded Linux runner, and the tests most exposed to it are the ones that
sleep a fixed time and then assert:

- admission: idle admission bounded at 50ms of wall clock against a 1ms
  tick, and a queued request checked for after a 50ms sleep;
- scrape: a 10ms scrape timeout, which on a slow runner reports the engine
  unreachable and then the violation a second time, in a test that counts
  one;
- supervisor: log lines, three lines of output and a removed state file
  each checked after a fixed sleep, and an exited worker bounded at 1.5s;
- agent: a heartbeat checked for after four intervals.

Each now waits for its condition up to a deadline of seconds, and the two
wall-clock bounds are set against what they exist to distinguish — the
admission queue's tick raised to ten seconds, the exited worker bounded
below the five-second grace it must not wait out — rather than against how
fast this laptop is.

Without the CI logs this cannot be proven to be the failure; these are the
tests that fail when the machine is slow, and they no longer do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:36:55 +02:00
25 changed files with 891 additions and 63 deletions

No files matched your search

+36
View File
@@ -66,8 +66,44 @@ ControlPlaneLeadership, CertificateExpiresSoon and CertificateExpiresVerySoon.
than its minimum, so an on-demand model that was asked for and fitted nowhere
no longer reads as satisfied.
### Backups
`llmgw server backup` writes the store and `/etc/llmgw` — configuration,
secrets, and the certificate authority's key — to one archive readable by root
only, keeping the newest 14. `deploy/systemd/llmgw-backup.timer` runs it
nightly. The control plane publishes the newest archive's time as
`llmgw_backup_last_success_timestamp_seconds`, and BackupTooOld fires when it is
more than 26 hours old or there has never been one. docs/install.md now says how
to restore, including why running engines survive it: the engine-key secret
comes back with the store.
### Security
- **Alertmanager answered anyone on the network**, including requests to
silence alerts. The monitoring README now binds it to loopback, where
Prometheus beside it is its only client.
- **Login and signup could be made to use gigabytes.** Each attempt hashes with
64 MiB of argon2, with no bound on how many ran at once, and the per-address
throttle grew by one entry per address tried. Hashes now run a few at a time
and the rest wait; the throttle expires old entries, and when it is still
full it refuses new addresses rather than dropping live ones — which would
have let a flood of made-up addresses reset the throttle on a real one.
- **Any key holder could grow Prometheus without bound** by asking for made-up
model names, each of which became a metric label. Requests are counted under
the catalog's name for their model, or "unknown".
- **A SQLite backup left a full copy of the store readable** by local users
while it ran. The snapshot is now made in a private directory, and
llmgw-backup.service sets `UMask=0077`.
- **An engine archive could write outside its directory through chained
links.** Archives are unpacked through `os.Root`, and a link may only point
at a file beside it. Every llama.cpp b10991 archive, CPU and CUDA, already
follows that rule.
### Fixes
- Tests that slept a fixed time and then asserted now wait for their
condition. They are the likeliest cause of CI's intermittent failures on a
loaded runner.
- `ctl` reported a change the API had queued (202 Accepted) as an error, as
`ctl restart` did on its first real use.
+9 -2
View File
@@ -46,6 +46,12 @@ apt-get install prometheus prometheus-alertmanager
install -m 0644 prometheus.yml /etc/prometheus/prometheus.yml
install -m 0644 llmgw-rules.yml /etc/prometheus/rules/llmgw.yml
install -m 0644 alertmanager.yml /etc/prometheus/alertmanager.yml
# Alertmanager answers on loopback only. Its API takes silences from anyone who
# can reach it, with no login, and silencing BackupTooOld or ModelParked is how
# something that should be noticed is not. Prometheus beside it is its only
# client; one Alertmanager needs no cluster port.
echo 'ARGS="--web.listen-address=127.0.0.1:9093 --cluster.listen-address="' \
> /etc/default/prometheus-alertmanager
promtool check config /etc/prometheus/prometheus.yml
systemctl restart prometheus prometheus-alertmanager
@@ -75,7 +81,8 @@ Alertmanager groups and holds alerts and has no receiver.
Where an alert should go — an address, a Slack workspace, a webhook — is a
decision with credentials attached, and a monitoring system that quietly starts
mailing people is worse than one that is visibly not wired up. Alerts are
visible at `:9093` and in Prometheus; add a receiver to `alertmanager.yml` to
visible in Prometheus, and at `:9093` on the monitoring host itself (`ssh -L
9093:127.0.0.1:9093 <host>` to open its page); add a receiver to `alertmanager.yml` to
make them travel. `critical` already routes separately so it can be pointed
somewhere else without touching the rest.
@@ -86,7 +93,7 @@ An alerting system nobody has seen fire is an assumption.
```
llmgw ctl cordon <node> # work already there keeps running
# ~2 minutes later
curl -s http://<host>:9093/api/v2/alerts | jq '.[].labels'
ssh <host> curl -s http://127.0.0.1:9093/api/v2/alerts | jq '.[].labels'
llmgw ctl uncordon <node> # the alert resolves on its own
```
+11
View File
@@ -123,6 +123,17 @@ groups:
summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
description: See CertificateExpiresSoon; a week is not long.
- alert: BackupTooOld
expr: time() - llmgw_backup_last_success_timestamp_seconds > 26 * 3600
labels: { severity: warning }
annotations:
summary: "The newest llmgw backup is {{ $value | humanizeDuration }} old"
description: >-
llmgw-backup.timer runs nightly. Also fires when there has never
been a backup (the series is 0 then). `journalctl -u
llmgw-backup.service` on the control plane says why the last one
failed; docs/install.md, "Backing up", says how to set it up.
- alert: PoolSaturated
expr: llmgw_pool_saturation > 1
for: 5m
+14
View File
@@ -0,0 +1,14 @@
[Unit]
Description=llmgw backup: the store and /etc/llmgw, to /var/backups/llmgw-server
After=llmgw-server.service postgresql.service
[Service]
Type=oneshot
# Nothing it writes is for anyone else: the archive holds the CA's key and a
# copy of the store.
UMask=0077
# Root, because the archive includes /etc/llmgw/secrets and the certificate
# authority's key, which the llmgw user deliberately cannot read all of.
ExecStart=/usr/local/bin/llmgw server backup --config /etc/llmgw/server.yaml --keep 14
Nice=10
IOSchedulingClass=idle
+12
View File
@@ -0,0 +1,12 @@
[Unit]
Description=Back up llmgw every night
[Timer]
OnCalendar=*-*-* 03:15:00
# A machine that was off at 03:15 backs up when it next starts, rather than
# waiting another day with a gap nobody notices until the alert.
Persistent=true
RandomizedDelaySec=10m
[Install]
WantedBy=timers.target
+56 -18
View File
@@ -961,36 +961,74 @@ the choice is yours rather than silent.
## Backing up
Three things, in descending order of how badly you want them.
`llmgw server backup` writes one archive holding everything a control plane
cannot be rebuilt without:
**The certificate authority.** `ca-key.pem`, `/etc/llmgw/pki` unless you kept it elsewhere (step 2).
Losing it does not stop the cluster, and it means every node identity has to be
reissued the next time you touch one. Nothing in llmgw can reconstruct it.
- **the store** — users, API keys, the tenancy tree, limits and budgets, usage
history, models added at runtime, measured capacity, and the secret every
engine key is derived from (`pg_dump` in its custom format for PostgreSQL, a
consistent `VACUUM INTO` snapshot for SQLite; both are safe against a running
gateway);
- **`/etc/llmgw`** — the configuration, the store's address, and the
certificate authority's key (`pki/`). Nothing in llmgw can reconstruct that
key; losing it means reissuing every node's identity.
**The store.** Users, API keys, organisations, teams, projects, every limit and
budget, all usage history, models added at runtime, and measured capacity.
Archives go to `/var/backups/llmgw-server`, readable by root only because of the CA
key, and the newest 14 are kept. Run it nightly with the timer:
```
# SQLite
sqlite3 /var/lib/llmgw/llmgw.db ".backup '/backup/llmgw-$(date +%F).db'"
# PostgreSQL
pg_dump --format=custom llmgw > /backup/llmgw-$(date +%F).dump
sudo install -m 0644 deploy/systemd/llmgw-backup.service deploy/systemd/llmgw-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now llmgw-backup.timer
sudo systemctl start llmgw-backup.service # one now, to see it work
ls -l /var/backups/llmgw-server
```
`.backup` and `pg_dump` are both safe against a running gateway. Copying the
SQLite file with `cp` while it is being written is not.
The control plane publishes when the newest archive was written as
`llmgw_backup_last_success_timestamp_seconds`, and **BackupTooOld** fires when
it is more than 26 hours old — including when there has never been one.
**Copy the archives off the machine.** They sit on the disk they back up and do
not survive losing it. Anything that copies files will do — rsync from the
mirror, a NAS mount, an offsite bucket — as long as it keeps the archives
readable by root only.
**Keys are stored hashed and cannot be recovered from a backup or from
anywhere else.** Restoring an old store restores which keys exist and who owns
them, not their secrets. A key created after the backup is gone and has to be
reissued — which is why the store is worth backing up on a schedule rather than
before upgrades.
reissued.
### Restoring
On a machine with llmgw installed (the same version or newer) and, for
PostgreSQL, an empty database and role as `server init --postgres` would create
them:
```
sudo systemctl stop llmgw-server
mkdir /tmp/restore && sudo tar -xzf /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz -C /tmp/restore
cat /tmp/restore/MANIFEST # version and time it was taken
# /etc/llmgw: configuration, secrets, certificates and the CA
sudo cp -a /tmp/restore/etc/llmgw/. /etc/llmgw/
# the store — PostgreSQL
sudo -u postgres pg_restore --clean --if-exists --no-owner --role=llmgw -d llmgw /tmp/restore/store.pgdump
# the store — SQLite
sudo install -o llmgw -g llmgw -m 0600 /tmp/restore/store.sqlite /var/lib/llmgw/llmgw.db
sudo systemctl start llmgw-server
sudo rm -rf /tmp/restore
```
Running engines keep serving throughout: agents reconnect to the restored
control plane and their replicas are adopted, and because the engine-key secret
came back with the store, the router can still reach them.
**The mirror's directory** is the one thing you can rebuild, by downloading it
all again. Worth remembering that a pinned version may no longer be upstream to
download: that is exactly what pinning it protected you from, and a backup is
what makes the protection outlive the disk.
all again. A pinned version may no longer be upstream to download: that is
exactly what pinning it protected you from, and a copy of the directory is what
makes the protection outlive the disk.
## Reclaiming disk
+9
View File
@@ -169,6 +169,7 @@ rather than repeated.
| `monitoring` | `Monitoring` | [Server](#server) |
| `mirror` | `Mirror` | [Server](#server) |
| `engines` | `map[string]string` | [Server](#server) |
| `backup_dir` | `string` | [Server](#server) |
| `router` | `Router` | [Server](#server) |
| `control` | `Control` | [Server](#server) |
| `models` | `[]catalog.Model` | [Server](#server) |
@@ -1265,6 +1266,14 @@ never trusted to already have what somebody once put there.
Unset keeps DefaultEngines. A version replaces the default; an empty
one leaves that engine to be installed by hand.
### `backup_dir`
`string`
BackupDir is where `llmgw server backup` writes, and where the newest
archive's age is read for llmgw_backup_last_success_timestamp_seconds.
Empty means /var/backups/llmgw-server.
### `router`
`Router`
+16 -2
View File
@@ -15,6 +15,7 @@ import (
"encoding/base64"
"errors"
"fmt"
"runtime"
"strings"
"golang.org/x/crypto/argon2"
@@ -56,7 +57,7 @@ func HashPassword(password string) (string, error) {
if _, err := rand.Read(salt); err != nil {
return "", err
}
sum := argon2.IDKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
sum := idKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
b64 := base64.RawStdEncoding.EncodeToString
return fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s",
@@ -93,7 +94,7 @@ func VerifyPassword(password, encoded string) (bool, error) {
return false, ErrBadHash
}
got := argon2.IDKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
got := idKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
return subtle.ConstantTimeCompare(got, want) == 1, nil
}
@@ -110,3 +111,16 @@ func init() {
}
dummyHash = h
}
// argonSlots bounds how many hashes run at once. Each takes 64 MiB, and the
// login and signup forms are on the client-facing port: without a bound, a
// few hundred parallel attempts at random addresses asked for gigabytes at
// once, and the per-address throttle does not see them — each address is new.
// Beyond the bound, attempts wait their turn instead.
var argonSlots = make(chan struct{}, max(2, runtime.NumCPU()/2))
func idKey(password, salt []byte, time, memory uint32, threads uint8, keyLen uint32) []byte {
argonSlots <- struct{}{}
defer func() { <-argonSlots }()
return argon2.IDKey(password, salt, time, memory, threads, keyLen)
}
+20
View File
@@ -230,6 +230,22 @@ func (s *Service) allowAttempt(email string) bool {
defer s.mu.Unlock()
w, ok := s.attempts[email]
if !ok && len(s.attempts) >= maxTrackedAddresses {
// Full of addresses nobody has signed in as for a while: forget the
// ones whose window has passed. Only expiring, never evicting early —
// dropping a live window is how an attacker spraying random addresses
// would buy unlimited guesses at the one they care about.
for k, v := range s.attempts {
if now.Sub(v.since) > s.cfg.AttemptWindow {
delete(s.attempts, k)
}
}
if len(s.attempts) >= maxTrackedAddresses {
// Still full, so somebody is spraying addresses faster than they
// expire. New ones wait; those already tracked are unaffected.
return false
}
}
if !ok || now.Sub(w.since) > s.cfg.AttemptWindow {
s.attempts[email] = &attemptWindow{count: 1, since: now}
return true
@@ -238,6 +254,10 @@ func (s *Service) allowAttempt(email string) bool {
return w.count <= s.cfg.MaxAttempts
}
// maxTrackedAddresses bounds the throttle's memory. Unbounded, it grew by one
// entry per address tried and shrank only on a successful sign-in.
const maxTrackedAddresses = 50000
func (s *Service) clearAttempts(email string) {
s.mu.Lock()
delete(s.attempts, email)
+33
View File
@@ -0,0 +1,33 @@
package account
import (
"fmt"
"testing"
"time"
)
func TestSprayingAddressesCannotGrowTheThrottleOrEscapeIt(t *testing.T) {
now := time.Unix(0, 0)
s := &Service{cfg: DefaultConfig(), attempts: map[string]*attemptWindow{}, now: func() time.Time { return now }}
// The target is being guessed at.
for i := 0; i < s.cfg.MaxAttempts; i++ {
s.allowAttempt("victim@example.com")
}
// Then the table is flooded with made-up addresses.
for i := 0; i < maxTrackedAddresses+100; i++ {
s.allowAttempt(fmt.Sprintf("x%d@example.com", i))
}
if n := len(s.attempts); n > maxTrackedAddresses {
t.Errorf("the table grew to %d", n)
}
if s.allowAttempt("victim@example.com") {
t.Error("flooding the table reset the target's throttle")
}
// Once the window passes, everything expires and sign-ins work again.
now = now.Add(s.cfg.AttemptWindow + time.Second)
if !s.allowAttempt("new@example.com") {
t.Error("a new address was refused after the flood expired")
}
}
+9 -4
View File
@@ -400,11 +400,16 @@ func TestHeartbeatKeepsNodeAlive(t *testing.T) {
n, _ := h.inv.Get("n1")
first := n.LastSeen
time.Sleep(200 * time.Millisecond)
n, _ = h.inv.Get("n1")
if !n.LastSeen.After(first) {
t.Error("heartbeats are not refreshing LastSeen; the node would be declared lost")
// Waited for rather than slept through: four heartbeat intervals was not
// always enough on a loaded runner for one to have landed.
deadline := time.Now().Add(3 * time.Second)
for time.Now().Before(deadline) {
if n, _ = h.inv.Get("n1"); n.LastSeen.After(first) {
return
}
time.Sleep(20 * time.Millisecond)
}
t.Error("heartbeats are not refreshing LastSeen; the node would be declared lost")
}
func TestStartAndStopShardEndToEnd(t *testing.T) {
+26 -16
View File
@@ -523,6 +523,11 @@ func (in *Installer) download(ctx context.Context, repo, tag, asset, dst, want s
// untarGz unpacks an archive under dir, refusing anything that would land
// outside it. A release archive is somebody else's file, and a path of
// "../../etc/cron.d/x" in it would otherwise be written as root.
//
// Every write goes through an os.Root, which refuses to leave dir however the
// path gets there. Checking names alone was not enough: "a -> ." followed by
// "a/a/b -> ../.." reads as inside and resolves two levels above it, and a
// file written through that lands wherever the links point.
func untarGz(archive, dir string) error {
f, err := os.Open(archive) //nolint:gosec // inside the install root
if err != nil {
@@ -533,6 +538,11 @@ func untarGz(archive, dir string) error {
if err != nil {
return err
}
root, err := os.OpenRoot(dir)
if err != nil {
return err
}
defer root.Close()
tr := tar.NewReader(gz)
for {
h, err := tr.Next()
@@ -542,22 +552,22 @@ func untarGz(archive, dir string) error {
if err != nil {
return err
}
target, err := within(dir, h.Name)
if err != nil {
name := filepath.Clean(h.Name)
if _, err := within(dir, name); err != nil {
return err
}
switch h.Typeflag {
case tar.TypeDir:
if err := os.MkdirAll(target, 0o755); err != nil {
if err := root.MkdirAll(name, 0o755); err != nil {
return err
}
case tar.TypeReg:
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
if err := root.MkdirAll(filepath.Dir(name), 0o755); err != nil {
return err
}
out, err := os.OpenFile(target, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, fs.FileMode(h.Mode)&0o755) //nolint:gosec // checked above
out, err := root.OpenFile(name, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, fs.FileMode(h.Mode)&0o755)
if err != nil {
return err
return fmt.Errorf("%s: %w", h.Name, err)
}
if _, err := io.Copy(out, tr); err != nil { //nolint:gosec // a release archive, size bounded by its checksum
out.Close()
@@ -567,19 +577,19 @@ func untarGz(archive, dir string) error {
return err
}
case tar.TypeSymlink:
// Libraries are versioned through symlinks, and those are
// relative. One pointing anywhere else is refused.
if filepath.IsAbs(h.Linkname) {
return fmt.Errorf("%s links outside the archive", h.Name)
// Libraries are versioned through symlinks to a file beside them —
// libllama.so -> libllama.so.0 — and that is all a release needs.
// A target with a separator in it could point anywhere once other
// links are followed, and a binary found through one would be run
// as root, so it is refused rather than reasoned about.
if h.Linkname == "" || strings.ContainsAny(h.Linkname, `/\`) || h.Linkname == ".." || h.Linkname == "." {
return fmt.Errorf("%s links to %q; only a file beside it is allowed", h.Name, h.Linkname)
}
if _, err := within(dir, filepath.Join(filepath.Dir(h.Name), h.Linkname)); err != nil {
return fmt.Errorf("%s links outside the archive", h.Name)
}
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
if err := root.MkdirAll(filepath.Dir(name), 0o755); err != nil {
return err
}
_ = os.Remove(target)
if err := os.Symlink(h.Linkname, target); err != nil {
_ = root.Remove(name)
if err := root.Symlink(h.Linkname, name); err != nil {
return err
}
default:
+46
View File
@@ -342,3 +342,49 @@ func TestAVersionThatIsAPathIsRefused(t *testing.T) {
}
}
}
func TestChainedLinksCannotCarryAFileOutOfTheInstall(t *testing.T) {
// Each link reads as inside the directory; together they point above it.
var buf bytes.Buffer
gz := gzip.NewWriter(&buf)
tw := tar.NewWriter(gz)
_ = tw.WriteHeader(&tar.Header{Name: "a", Linkname: ".", Typeflag: tar.TypeSymlink})
_ = tw.WriteHeader(&tar.Header{Name: "a/a/b", Linkname: "../..", Typeflag: tar.TypeSymlink})
_ = tw.WriteHeader(&tar.Header{Name: "a/a/b/escaped", Mode: 0o644, Size: 1, Typeflag: tar.TypeReg})
_, _ = tw.Write([]byte("x"))
_ = tw.Close()
_ = gz.Close()
parent := t.TempDir()
dir := filepath.Join(parent, "install")
_ = os.MkdirAll(dir, 0o755)
archive := filepath.Join(parent, "evil.tar.gz")
_ = os.WriteFile(archive, buf.Bytes(), 0o644)
if err := untarGz(archive, dir); err == nil {
t.Error("an archive of chained links was unpacked")
}
if _, err := os.Stat(filepath.Join(parent, "escaped")); err == nil {
t.Fatal("a file was written outside the install directory")
}
}
func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
var buf bytes.Buffer
gz := gzip.NewWriter(&buf)
tw := tar.NewWriter(gz)
_ = tw.WriteHeader(&tar.Header{Name: "lib/libllama.so.0", Mode: 0o755, Size: 1, Typeflag: tar.TypeReg})
_, _ = tw.Write([]byte("x"))
_ = tw.WriteHeader(&tar.Header{Name: "lib/libllama.so", Linkname: "libllama.so.0", Typeflag: tar.TypeSymlink})
_ = tw.Close()
_ = gz.Close()
dir := t.TempDir()
archive := filepath.Join(t.TempDir(), "ok.tar.gz")
_ = os.WriteFile(archive, buf.Bytes(), 0o644)
if err := untarGz(archive, dir); err != nil {
t.Fatal(err)
}
if target, err := os.Readlink(filepath.Join(dir, "lib", "libllama.so")); err != nil || target != "libllama.so.0" {
t.Errorf("link %q %v", target, err)
}
}
+29 -8
View File
@@ -243,7 +243,9 @@ func TestWaitReadyTimeoutIncludesLogs(t *testing.T) {
if err != nil {
t.Fatal(err)
}
time.Sleep(100 * time.Millisecond) // let the line be captured
eventually(t, 3*time.Second, "the line was never captured", func() bool {
return strings.Contains(strings.Join(sh.Logs(10), "\n"), "CUDA out of memory")
})
err = s.WaitReady(t.Context(), sh, "http://x", "m", 200*time.Millisecond)
if err == nil {
@@ -295,7 +297,10 @@ func TestLogsAreCaptured(t *testing.T) {
if err != nil {
t.Fatal(err)
}
time.Sleep(400 * time.Millisecond)
eventually(t, 3*time.Second, "not every line arrived", func() bool {
l := strings.Join(sh.Logs(10), "\n")
return strings.Contains(l, "line1") && strings.Contains(l, "line2") && strings.Contains(l, "line3")
})
logs := strings.Join(sh.Logs(10), "\n")
for _, want := range []string{"line1", "line2", "line3"} {
@@ -372,11 +377,11 @@ func TestStateFileRemovedOnExit(t *testing.T) {
sh, _ := s.Start(t.Context(), "a", engines.ShardSpec{}, shellCommand("exit 0"))
waitState(t, sh, Failed, 3*time.Second)
time.Sleep(50 * time.Millisecond)
if _, err := os.Stat(s.statePath("a")); !os.IsNotExist(err) {
t.Error("the state file outlived the process; a later scan would find a dead orphan")
}
eventually(t, 3*time.Second,
"the state file outlived the process; a later scan would find a dead orphan", func() bool {
_, err := os.Stat(s.statePath("a"))
return os.IsNotExist(err)
})
}
// --- state machine and ring buffer ---
@@ -743,7 +748,9 @@ func TestWaitAliveFailsOnEarlyExit(t *testing.T) {
if err := s.WaitAlive(t.Context(), sh); err == nil {
t.Fatal("a worker that died immediately was reported ready")
}
if d := time.Since(start); d > 1500*time.Millisecond {
// Below AliveGrace (5s), which is what waiting it out would take; the
// headroom is for a loaded runner, where 1.5s was sometimes not enough.
if d := time.Since(start); d > 4*time.Second {
t.Errorf("waited %v for a process that had already exited", d)
}
}
@@ -827,3 +834,17 @@ func TestAWorkersOnlyProcessWaitsForTheLeader(t *testing.T) {
t.Errorf("the worker started after %v, before the leader was listening", d)
}
}
// eventually waits for cond, failing with msg if it does not hold within
// limit. Sleeping a fixed time and then checking measured the machine the
// test ran on, and a loaded CI runner under the race detector lost that race.
func eventually(t *testing.T, limit time.Duration, msg string, cond func() bool) {
t.Helper()
deadline := time.Now().Add(limit)
for !cond() {
if time.Now().After(deadline) {
t.Fatal(msg)
}
time.Sleep(10 * time.Millisecond)
}
}
+312
View File
@@ -0,0 +1,312 @@
package cli
import (
"archive/tar"
"compress/gzip"
"context"
"errors"
"fmt"
"io"
"io/fs"
"net/url"
"os"
"os/exec"
"path/filepath"
"slices"
"strings"
"time"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
)
// DefaultBackupDir is where backups are written and where the control plane
// looks for the newest one.
//
// Its own directory rather than /var/backups/llmgw, which a PostgreSQL move
// had already made for its own dumps, owned by postgres and closed to
// everyone else — so the control plane could not see a backup landing in it.
const DefaultBackupDir = "/var/backups/llmgw-server"
func newBackupCmd() *cobra.Command {
var (
configPath string
etcDir string
dir string
keep int
)
c := &cobra.Command{
Use: "backup",
Short: "Write the store and /etc/llmgw to one archive, and keep the last few",
Long: "Everything a control plane cannot be rebuilt without: the store — keys,\n" +
"people, quotas, usage, the models added at runtime, the secret engine\n" +
"keys derive from — and /etc/llmgw, which holds the configuration, the\n" +
"store's address and the certificate authority's key.\n\n" +
"The archive holds that key, so it is written readable by root only. It\n" +
"is on the same disk as what it backs up: copy it elsewhere, or it does\n" +
"not survive the loss of this machine.\n\n" +
"Run daily by llmgw-backup.timer (`llmgw server backup`). The control plane publishes the time of\n" +
"the newest archive as llmgw_backup_last_success_timestamp_seconds, and\n" +
"BackupTooOld fires when it is more than a day old.",
RunE: func(cmd *cobra.Command, _ []string) error {
cfg, err := config.Load(configPath)
if err != nil {
return err
}
path, err := writeBackup(cmd.Context(), cfg.Auth.Store, etcDir, dir)
if err != nil {
return err
}
if why := unlistable(dir); why != "" {
fmt.Fprintf(cmd.ErrOrStderr(), "warning: %s; the control plane will report no backup "+
"and BackupTooOld will fire\n", why)
}
removed, err := pruneBackups(dir, keep)
if err != nil {
return err
}
fmt.Fprintf(cmd.OutOrStdout(), "wrote %s", path)
if removed > 0 {
fmt.Fprintf(cmd.OutOrStdout(), "; removed %d older", removed)
}
fmt.Fprintln(cmd.OutOrStdout())
return nil
},
}
c.Flags().StringVar(&configPath, "config", "/etc/llmgw/server.yaml", "the control plane's configuration")
c.Flags().StringVar(&etcDir, "etc", "/etc/llmgw", "configuration directory to include")
c.Flags().StringVar(&dir, "dir", DefaultBackupDir, "where archives are written")
c.Flags().IntVar(&keep, "keep", 14, "how many archives to keep")
return c
}
const backupPrefix = "llmgw-backup-"
// writeBackup writes one archive and returns its path. Written beside its
// final name and renamed, so a backup interrupted halfway is never the newest
// one the control plane reports.
func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, error) {
// Listable by anyone, so the control plane — not root — can see when the
// newest archive was written. The archives themselves are root's alone.
if err := os.MkdirAll(dir, 0o755); err != nil {
return "", err
}
host, _ := os.Hostname()
name := backupPrefix + host + "-" + time.Now().UTC().Format("20060102T150405Z") + ".tar.gz"
final := filepath.Join(dir, name)
part := final + ".part"
f, err := os.OpenFile(part, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o600)
if err != nil {
return "", err
}
ok := false
defer func() {
if !ok {
_ = os.Remove(part)
}
}()
gz := gzip.NewWriter(f)
tw := tar.NewWriter(gz)
manifest := fmt.Sprintf("llmgw %s\nhost %s\ntime %s\n", version.Version, host,
time.Now().UTC().Format(time.RFC3339))
if err := addBytes(tw, "MANIFEST", []byte(manifest)); err != nil {
return "", err
}
if storeAddr != "" {
if err := addStore(ctx, tw, storeAddr, dir); err != nil {
return "", fmt.Errorf("the store: %w", err)
}
}
if err := addTree(tw, etcDir, "etc/llmgw"); err != nil {
return "", fmt.Errorf("%s: %w", etcDir, err)
}
if err := errors.Join(tw.Close(), gz.Close(), f.Sync(), f.Close()); err != nil {
return "", err
}
if err := os.Rename(part, final); err != nil {
return "", err
}
ok = true
return final, nil
}
// addStore dumps the store into the archive.
func addStore(ctx context.Context, tw *tar.Writer, addr, scratch string) error {
addr, err := store.ResolveAddress(addr)
if err != nil {
return err
}
// Inside a directory only its owner can enter. The archive directory is
// listable on purpose, and VACUUM INTO creates its file with the default
// mode: made there directly, a full copy of the store — password hashes,
// the engine-key secret — sat readable by any local user for as long as
// the backup took.
tmpDir, err := os.MkdirTemp(scratch, ".store-")
if err != nil {
return err
}
defer os.RemoveAll(tmpDir)
tmpPath := filepath.Join(tmpDir, "store")
name := "store.sqlite"
if store.IsPostgres(addr) {
name = "store.pgdump"
if err := pgDump(ctx, addr, tmpPath); err != nil {
return err
}
} else {
db, err := store.Open(ctx, addr)
if err != nil {
return err
}
err = db.SnapshotTo(ctx, tmpPath)
_ = db.Close()
if err != nil {
return err
}
}
return addFile(tw, tmpPath, name)
}
// pgDump runs pg_dump in its custom format, which pg_restore can restore
// selectively. The connection is given through the environment, not the
// command line: a DSN carries the password, and every user on the machine
// can read another process's arguments.
func pgDump(ctx context.Context, dsn, out string) error {
u, err := url.Parse(dsn)
if err != nil {
return fmt.Errorf("the store address is not a URL")
}
env := os.Environ()
set := func(k, v string) {
if v != "" {
env = append(env, k+"="+v)
}
}
set("PGHOST", u.Hostname())
set("PGPORT", u.Port())
set("PGUSER", u.User.Username())
if pw, ok := u.User.Password(); ok {
set("PGPASSWORD", pw)
}
set("PGDATABASE", strings.TrimPrefix(u.Path, "/"))
set("PGSSLMODE", u.Query().Get("sslmode"))
cmd := exec.CommandContext(ctx, "pg_dump", "--format=custom", "--no-owner", "--file="+out) //nolint:gosec // fixed binary, our own file
cmd.Env = env
if b, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("pg_dump: %v: %s", err, strings.TrimSpace(string(b)))
}
return nil
}
func addTree(tw *tar.Writer, root, as string) error {
return filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
rel, _ := filepath.Rel(root, p)
name := filepath.ToSlash(filepath.Join(as, rel))
info, err := d.Info()
if err != nil {
return err
}
switch {
case d.IsDir():
return tw.WriteHeader(&tar.Header{Name: name + "/", Mode: int64(info.Mode().Perm()),
Typeflag: tar.TypeDir, ModTime: info.ModTime()})
case info.Mode().IsRegular():
return addFile(tw, p, name)
}
return nil // sockets, links: not configuration
})
}
func addFile(tw *tar.Writer, path, name string) error {
f, err := os.Open(path) //nolint:gosec // files we chose to back up
if err != nil {
return err
}
defer f.Close()
info, err := f.Stat()
if err != nil {
return err
}
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: int64(info.Mode().Perm()),
Size: info.Size(), ModTime: info.ModTime(), Typeflag: tar.TypeReg}); err != nil {
return err
}
_, err = io.Copy(tw, f)
return err
}
func addBytes(tw *tar.Writer, name string, b []byte) error {
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o600, Size: int64(len(b)),
ModTime: time.Now(), Typeflag: tar.TypeReg}); err != nil {
return err
}
_, err := tw.Write(b)
return err
}
// pruneBackups keeps the newest keep archives. Only files this command
// wrote are considered, so nothing else in the directory is ever removed.
func pruneBackups(dir string, keep int) (int, error) {
entries, err := os.ReadDir(dir)
if err != nil {
return 0, err
}
var names []string
for _, e := range entries {
if n := e.Name(); strings.HasPrefix(n, backupPrefix) && strings.HasSuffix(n, ".tar.gz") {
names = append(names, n)
}
}
// The name carries a UTC timestamp, so name order is age order.
slices.Sort(names)
removed := 0
for len(names) > max(keep, 1) {
if err := os.Remove(filepath.Join(dir, names[0])); err != nil {
return removed, err
}
names = names[1:]
removed++
}
return removed, nil
}
// NewestBackup is the modification time of the newest complete archive in
// dir, or zero when there is none.
func NewestBackup(dir string) time.Time {
entries, err := os.ReadDir(dir)
if err != nil {
return time.Time{}
}
var newest time.Time
for _, e := range entries {
if n := e.Name(); !strings.HasPrefix(n, backupPrefix) || !strings.HasSuffix(n, ".tar.gz") {
continue
}
if info, err := e.Info(); err == nil && info.ModTime().After(newest) {
newest = info.ModTime()
}
}
return newest
}
// unlistable says why the control plane, which does not run as root, could
// not see the archives in dir, or "" when it can.
func unlistable(dir string) string {
info, err := os.Stat(dir)
if err != nil {
return err.Error()
}
if info.Mode().Perm()&0o005 != 0o005 {
return fmt.Sprintf("%s is %v, so only its owner can list it", dir, info.Mode().Perm())
}
return ""
}
+141
View File
@@ -0,0 +1,141 @@
package cli
import (
"archive/tar"
"compress/gzip"
"io"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
func TestABackupHoldsTheStoreAndTheConfiguration(t *testing.T) {
tmp := t.TempDir()
dbPath := filepath.Join(tmp, "store.db")
db, err := store.Open(t.Context(), dbPath)
if err != nil {
t.Fatal(err)
}
secret, err := db.ClusterSecret(t.Context(), "engine-key")
if err != nil {
t.Fatal(err)
}
_ = db.Close()
etc := filepath.Join(tmp, "etc")
_ = os.MkdirAll(filepath.Join(etc, "pki"), 0o700)
_ = os.WriteFile(filepath.Join(etc, "server.yaml"), []byte("listen: :8080\n"), 0o644)
_ = os.WriteFile(filepath.Join(etc, "pki", "ca-key.pem"), []byte("KEY"), 0o600)
out := filepath.Join(tmp, "backups")
path, err := writeBackup(t.Context(), dbPath, etc, out)
if err != nil {
t.Fatal(err)
}
info, _ := os.Stat(path)
if info.Mode().Perm() != 0o600 {
t.Errorf("the archive holds the CA's key and is %v", info.Mode().Perm())
}
files := map[string][]byte{}
f, _ := os.Open(path)
defer f.Close()
gz, err := gzip.NewReader(f)
if err != nil {
t.Fatal(err)
}
tr := tar.NewReader(gz)
for {
h, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
t.Fatal(err)
}
b, _ := io.ReadAll(tr)
files[h.Name] = b
}
for _, want := range []string{"MANIFEST", "store.sqlite", "etc/llmgw/server.yaml", "etc/llmgw/pki/ca-key.pem"} {
if _, ok := files[want]; !ok {
t.Errorf("%s is not in the archive", want)
}
}
// And the store in it is a working store, with what was in it.
restored := filepath.Join(tmp, "restored.db")
_ = os.WriteFile(restored, files["store.sqlite"], 0o600)
rdb, err := store.Open(t.Context(), restored)
if err != nil {
t.Fatalf("the backed-up store does not open: %v", err)
}
defer rdb.Close()
if got, _ := rdb.ClusterSecret(t.Context(), "engine-key"); got != secret {
t.Error("the restored store lost the engine key secret; every engine would be locked out")
}
}
func TestOldBackupsArePrunedAndTheNewestIsFound(t *testing.T) {
dir := t.TempDir()
for i, stamp := range []string{"20260101T000000Z", "20260102T000000Z", "20260103T000000Z"} {
p := filepath.Join(dir, backupPrefix+"h-"+stamp+".tar.gz")
_ = os.WriteFile(p, []byte("x"), 0o600)
mt := time.Date(2026, 1, 1+i, 0, 0, 0, 0, time.UTC)
_ = os.Chtimes(p, mt, mt)
}
_ = os.WriteFile(filepath.Join(dir, "someone-elses-file"), []byte("x"), 0o600)
_ = os.WriteFile(filepath.Join(dir, backupPrefix+"h-20260104T000000Z.tar.gz.part"), []byte("x"), 0o600)
removed, err := pruneBackups(dir, 2)
if err != nil || removed != 1 {
t.Fatalf("removed %d, %v", removed, err)
}
if _, err := os.Stat(filepath.Join(dir, "someone-elses-file")); err != nil {
t.Error("a file the backup did not write was removed")
}
newest := NewestBackup(dir)
if !newest.Equal(time.Date(2026, 1, 3, 0, 0, 0, 0, time.UTC)) {
t.Errorf("newest %v; an interrupted .part must not count", newest)
}
if !NewestBackup(filepath.Join(dir, "none")).IsZero() {
t.Error("a missing directory reported a backup")
}
}
func TestABackupDirectoryTheControlPlaneCannotListIsSaid(t *testing.T) {
dir := t.TempDir()
_ = os.Chmod(dir, 0o700)
if unlistable(dir) == "" {
t.Error("a directory only its owner can list was taken as visible")
}
_ = os.Chmod(dir, 0o755)
if why := unlistable(dir); why != "" {
t.Errorf("a listable directory: %s", why)
}
}
func TestNoScratchCopyOfTheStoreIsLeftBehind(t *testing.T) {
// The snapshot is made before it is archived. Made straight into the
// listable archive directory, it was readable by anyone for that time.
tmp := t.TempDir()
dbPath := filepath.Join(tmp, "store.db")
db, err := store.Open(t.Context(), dbPath)
if err != nil {
t.Fatal(err)
}
_ = db.Close()
out := filepath.Join(tmp, "backups")
if _, err := writeBackup(t.Context(), dbPath, t.TempDir(), out); err != nil {
t.Fatal(err)
}
entries, _ := os.ReadDir(out)
for _, e := range entries {
if strings.HasPrefix(e.Name(), ".store") {
t.Errorf("a scratch copy of the store was left behind: %s", e.Name())
}
}
}
+19
View File
@@ -7,6 +7,7 @@ import (
"fmt"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
"github.com/prometheus/client_golang/prometheus"
"log/slog"
"net"
"net/http"
@@ -132,6 +133,23 @@ func newServerCmd() *cobra.Command {
// On the same /metrics as the traffic figures: correlating
// "the cluster was full" with "requests were queueing" should
// not need two systems.
// When the newest backup was written, so its absence alerts
// rather than being found on the day it is needed.
backupDir := cfg.BackupDir
if backupDir == "" {
backupDir = DefaultBackupDir
}
if err := rt.RegisterMetrics(prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "llmgw_backup_last_success_timestamp_seconds",
Help: "When the newest backup archive was written, as a Unix time; 0 when there is none.",
}, func() float64 {
if t := NewestBackup(backupDir); !t.IsZero() {
return float64(t.Unix())
}
return 0
})); err != nil {
return err
}
if err := rt.RegisterMetrics(tlsconf.ExpiryCollector(map[string]string{
"agent-server": cfg.AgentTLS.Cert, "agent-ca": cfg.AgentTLS.CA,
})); err != nil {
@@ -367,6 +385,7 @@ func newServerCmd() *cobra.Command {
f.StringArrayVar(&endpoints, "endpoint", nil, "static endpoint: id=URL=model[,model] (repeatable)")
cmd.AddCommand(newServerInitCmd())
cmd.AddCommand(newBackupCmd())
return cmd
}
+5
View File
@@ -81,6 +81,11 @@ type Server struct {
// one leaves that engine to be installed by hand.
Engines map[string]string `koanf:"engines"`
// BackupDir is where `llmgw server backup` writes, and where the newest
// archive's age is read for llmgw_backup_last_success_timestamp_seconds.
// Empty means /var/backups/llmgw-server.
BackupDir string `koanf:"backup_dir"`
Router Router `koanf:"router"`
Control Control `koanf:"control"`
Models []catalog.Model `koanf:"models"`
+15 -4
View File
@@ -42,14 +42,20 @@ func newController(t *testing.T, cfg Config, sat SaturationFunc) *Controller {
func TestAdmitsImmediatelyWhenIdle(t *testing.T) {
s := &fixedSat{}
c := newController(t, testConfig(), s.fn)
// A queue that is looked at every ten seconds, so a request that went
// through it rather than the fast path would take that long. A bound of
// 50ms against a 1ms tick measured the machine the test ran on, and a
// loaded CI runner under the race detector was slower than that.
cfg := testConfig()
cfg.Interval = 10 * time.Second
c := newController(t, cfg, s.fn)
start := time.Now()
lease, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
if err != nil {
t.Fatal(err)
}
if d := time.Since(start); d > 50*time.Millisecond {
if d := time.Since(start); d > 2*time.Second {
t.Errorf("idle admission took %v; the fast path is not being used", d)
}
lease.Release()
@@ -131,8 +137,13 @@ func TestQueuedRequestAdmittedWhenSaturationDrops(t *testing.T) {
done <- err
}()
// Give it time to actually queue, then open the gate.
time.Sleep(50 * time.Millisecond)
// Wait for it to actually queue, then open the gate. Waited for, not
// slept through: on a loaded runner 50ms was sometimes not long enough
// for the goroutine to have been scheduled at all.
deadline := time.Now().Add(2 * time.Second)
for c.Depths()[3].Depth != 1 && time.Now().Before(deadline) {
time.Sleep(5 * time.Millisecond)
}
if d := c.Depths(); d[3].Depth != 1 {
t.Fatalf("expected 1 queued bronze request, got %+v", d)
}
+8 -8
View File
@@ -51,7 +51,7 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded):
// The client gave up while queued; nobody is listening.
r.m.observeAdmissionRejected(req.Band, "client_gone")
r.m.observeRequest(req.Model, "", 499, time.Since(start), nil, 0)
r.m.observeRequest(r.metricModel(req.Model), "", 499, time.Since(start), nil, 0)
return
case errors.Is(err, admission.ErrQueueTimeout):
status = http.StatusGatewayTimeout
@@ -67,7 +67,7 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
}
r.m.observeAdmissionRejected(req.Band, reason)
writeError(w, status, "server_error", err.Error())
r.m.observeRequest(req.Model, "", status, time.Since(start), nil, 0)
r.m.observeRequest(r.metricModel(req.Model), "", status, time.Since(start), nil, 0)
}
// prefixBytes is how much of the prompt is kept for prefix-aware routing.
@@ -289,7 +289,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
if _, entry := r.resolve(parsed.Model); entry.ContextLength > 0 {
if why := checkContextFits(entry, parsed.PromptTokens, parsed.MaxTokens); why != "" {
writeError(w, http.StatusBadRequest, "invalid_request_error", why)
r.m.observeRequest(parsed.Model, "", http.StatusBadRequest, time.Since(start), nil, 0)
r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
return
}
}
@@ -297,7 +297,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
if p, ok := auth.FromContext(req.Context()); ok {
if !r.permits(p, parsed.Model) {
auth.WriteForbiddenModel(w, parsed.Model)
r.m.observeRequest(parsed.Model, "", http.StatusForbidden, time.Since(start), nil, 0)
r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusForbidden, time.Since(start), nil, 0)
return
}
// Limits are checked before admission, not after: a request that is
@@ -339,7 +339,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
}
defer lease.Release()
queued := time.Since(start)
r.m.observeAdmission(lease.Band(), parsed.Model, queued)
r.m.observeAdmission(lease.Band(), r.metricModel(parsed.Model), queued)
if queued > time.Millisecond {
w.Header().Set("x-llmgw-queue-ms", strconv.FormatInt(queued.Milliseconds(), 10))
}
@@ -349,7 +349,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
// model that will actually serve, not the one that was asked for.
if dropped, err := r.loadThread(req.Context(), parsed); err != nil {
writeError(w, http.StatusBadRequest, "invalid_request_error", err.Error())
r.m.observeRequest(parsed.Model, "", http.StatusBadRequest, time.Since(start), nil, 0)
r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
return
} else if dropped > 0 {
// Dropping turns is the right answer and must not be a silent one.
@@ -420,7 +420,7 @@ func (r *Router) finish(w http.ResponseWriter, req *parsedRequest, res dispatchR
writeError(w, status, "server_error", res.err.Error())
}
r.m.observeRequest(req.Model, res.pool, status, time.Since(start), res.outcome.Usage, res.outcome.TTFT)
r.m.observeRequest(r.metricModel(req.Model), res.pool, status, time.Since(start), res.outcome.Usage, res.outcome.TTFT)
r.account(req, res, status, time.Since(start))
}
@@ -467,7 +467,7 @@ func (r *Router) rejectLimit(w http.ResponseWriter, req *parsedRequest, err erro
// already know how to surface that code. What differs is Retry-After,
// which is absent when waiting does not help.
writeError(w, http.StatusTooManyRequests, "rate_limit_exceeded", le.Error())
r.m.observeRequest(req.Model, "", http.StatusTooManyRequests, time.Since(start), nil, 0)
r.m.observeRequest(r.metricModel(req.Model), "", http.StatusTooManyRequests, time.Since(start), nil, 0)
}
// parse reads the body and extracts the routing-relevant fields.
+14
View File
@@ -151,3 +151,17 @@ func (m *metrics) observeScores(d scheduler.Decision) {
}
m.considered.Observe(float64(d.Considered))
}
// metricModel is the model label a request is counted under: the catalog's
// name for it, or "unknown". Taken as sent, any key holder could mint a new
// series per request — a refused request counted its made-up model name —
// and grow Prometheus without bound.
func (r *Router) metricModel(name string) string {
if name == "" {
return ""
}
if m, ok := r.modelMap()[name]; ok {
return m.Name
}
return "unknown"
}
+30
View File
@@ -4,6 +4,7 @@ import (
"bytes"
"encoding/json"
"fmt"
"github.com/prometheus/client_golang/prometheus"
"io"
"log/slog"
"net/http"
@@ -655,3 +656,32 @@ func TestAModelThatCannotComeUpIsSaidSoAtOnce(t *testing.T) {
t.Errorf("the reason was not given: %s", rec.Body.String())
}
}
func TestAMadeUpModelNameDoesNotBecomeAMetricLabel(t *testing.T) {
// Counted as sent, every request for a nonsense model minted a new series.
e := newFakeEngine(t, "m")
cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}})
cfg.Models = []catalog.Model{{Name: "m", Aliases: []string{"mm"}}}
rt := newTestRouter(t, cfg)
if got := rt.metricModel("mm"); got != "m" {
t.Errorf("an alias is counted as %q, not the model it names", got)
}
for i := 0; i < 20; i++ {
post(t, rt.Handler(), map[string]any{"model": fmt.Sprintf("junk-%d", i), "messages": []any{}}, nil)
}
// Twenty names, one series: all of them "unknown".
reg := prometheus.NewRegistry()
reg.MustRegister(rt.m.requests)
mfs, err := reg.Gather()
if err != nil {
t.Fatal(err)
}
series := 0
for _, mf := range mfs {
series += len(mf.GetMetric())
}
if series != 1 {
t.Errorf("%d series after twenty made-up names", series)
}
}
+4 -1
View File
@@ -175,7 +175,10 @@ func TestABrokenContractIsReportedOnceAndLoudly(t *testing.T) {
in := endpoint.Info{ID: "e1", URL: srv.URL, Engine: "vllm", Models: []string{"m"}}
reg.Replace([]endpoint.Info{in})
p := NewPoller(reg, Config{Interval: 20 * time.Millisecond, Timeout: 10 * time.Millisecond},
// A generous timeout: a scrape that timed out would be reported as
// unreachable, and the violation reported again once it answered — two
// warnings where the test counts one, on a slow runner only.
p := NewPoller(reg, Config{Interval: 20 * time.Millisecond, Timeout: 2 * time.Second},
slog.New(cap))
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
+6
View File
@@ -59,3 +59,9 @@ func resolveAddress(address string) (string, error) {
}
return out, nil
}
// ResolveAddress expands an @file store address, as Open does.
func ResolveAddress(address string) (string, error) { return resolveAddress(address) }
// IsPostgres reports whether a resolved address names a PostgreSQL server.
func IsPostgres(address string) bool { return isPostgres(address) }
+11
View File
@@ -817,3 +817,14 @@ func (s *DB) UsageByModel(ctx context.Context, since time.Time) ([]ModelUsage, e
}
return out, rows.Err()
}
// SnapshotTo writes a consistent copy of a SQLite store to path, while it is
// in use. VACUUM INTO takes a read transaction, so the copy is of one moment
// even with writes going on, which copying the file is not.
func (s *DB) SnapshotTo(ctx context.Context, path string) error {
if s.dialect != SQLiteDialect {
return fmt.Errorf("a snapshot is for SQLite; back up PostgreSQL with pg_dump")
}
_, err := s.raw.ExecContext(ctx, `VACUUM INTO ?`, path)
return err
}