Compare commits

...
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 9bf8256577 Keep backups where the control plane can see them
Release / build (push) Successful in 1m1s
Release / notes (push) Successful in 2s
CI / test (push) Successful in 3m35s
The first backup on the live control plane was written, and the control
plane reported none: /var/backups/llmgw already existed, made by an earlier
move to PostgreSQL for its own dumps, owned by postgres and 0700. MkdirAll
leaves an existing directory as it is, so the archive landed where the
llmgw user could not list it, and llmgw_backup_last_success_timestamp_seconds
said 0.

Opening that directory up would have exposed the dumps already in it, so
backups get their own, /var/backups/llmgw-server. And `llmgw server backup`
now warns when the directory it wrote to is one the control plane cannot
list, rather than leaving that to be found from an alert.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:50:33 +02:00
Ilker ManapandClaude Opus 5.5 1da4a2e1ef Back up the store and the certificate authority, and alert when it stops
Release / build (push) Successful in 1m1s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m32s
There were no backups: docs/install.md gave a pg_dump command, and nothing
ran it. The store holds every key, person, quota and usage record, the models
added at runtime, and now the secret each engine's key is derived from; the
certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk
lost all of it.

`llmgw server backup` writes both to one archive: pg_dump's custom format, or
a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's
key, so it is 0600; the directory is listable, so the control plane — not
root — can see when the newest was written. pg_dump is given the connection
through its environment, since a DSN carries the password and a process's
arguments are readable by every user. Written beside its name and renamed,
so an interrupted backup is never the newest; the newest 14 are kept, and
nothing the command did not write is ever removed.

llmgw-backup.timer runs it nightly and catches up after downtime. The control
plane publishes the newest archive's time, and BackupTooOld fires past 26
hours — or when there has never been one. The install guide now covers
restoring, and says to copy archives off the machine, since they share its
disk.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:44:32 +02:00
Ilker ManapandClaude Opus 5.5 b661bfead8 Wait for what tests check instead of sleeping and hoping
CI / test (push) Successful in 3m23s
CI's test job has failed intermittently — two of three runs on some days —
while every run here passed, including six repetitions on one CPU under the
race detector with eight packages at once. The difference is a shared,
loaded Linux runner, and the tests most exposed to it are the ones that
sleep a fixed time and then assert:

- admission: idle admission bounded at 50ms of wall clock against a 1ms
  tick, and a queued request checked for after a 50ms sleep;
- scrape: a 10ms scrape timeout, which on a slow runner reports the engine
  unreachable and then the violation a second time, in a test that counts
  one;
- supervisor: log lines, three lines of output and a removed state file
  each checked after a fixed sleep, and an exited worker bounded at 1.5s;
- agent: a heartbeat checked for after four intervals.

Each now waits for its condition up to a deadline of seconds, and the two
wall-clock bounds are set against what they exist to distinguish — the
admission queue's tick raised to ten seconds, the exited worker bounded
below the five-second grace it must not wait out — rather than against how
fast this laptop is.

Without the CI logs this cannot be proven to be the failure; these are the
tests that fail when the machine is slow, and they no longer do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:36:55 +02:00
Ilker ManapandClaude Opus 5.5 37a80a9b44 Park a model that fails every time it starts, and say so everywhere
Release / build (push) Successful in 1m3s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m34s
On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not
know — was started 155 times in eight hours, failing the same way each time,
and phi-4, which fitted on no node, logged "no placement" 250 times in two
days. Neither showed up anywhere an operator looked: the backoff never gave
up, no metric said a model was failing or unplaceable, and the one alert for
a model with nothing serving could not fire, because its series was published
only while a replica was ready.

After five failures in a row a model is parked: not placed again until its
definition changes — a new source or profile — or `llmgw ctl retry` is run.
Status, the model list and the admin API show it with its last error. The
placement warning is logged when its reason changes or every ten minutes.

A request for a parked or unplaceable model is answered at once with why,
rather than "loading; retry shortly", which a client retries forever. Each
request still clears the previous placement result and triggers a fresh
attempt, so a model that fits once room is made is not refused on a stale
answer; the router waits for that attempt, not for its full activation
window.

Metrics now cover model health (desired, ready at 0 for what should run,
consecutive start failures, parked, unplaceable), engine installs per node,
leadership, and certificate expiry read from disk at each scrape, with alerts
for each. `ctl status` counts a model against what it should have now, so an
on-demand model that was asked for and fitted nowhere no longer reads as
satisfied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:21:57 +02:00
Ilker ManapandClaude Opus 5.5 5e1b6ecea0 Read a queued change as a success in ctl
ctl accepted only 200, so the first real `ctl restart --all` printed
"Error: 202 Accepted" over a request that had queued nine restarts, all of
which then happened. Any 2xx is now a success, and its body is read.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:14:12 +02:00
31 changed files with 1412 additions and 60 deletions

No files matched your search

+50
View File
@@ -35,6 +35,56 @@ grace period and the new one starts in its place. This is how running engines
pick up the keys introduced in v0.10.3, an edited profile, or a new engine
version.
### A model that keeps failing is parked, and everybody is told
A model whose replicas failed at every start was retried forever: kev-0.5b,
whose architecture the nodes' llama.cpp did not know, was started 155 times in
eight hours. After five failures in a row a model is now parked — not placed
again until its definition changes or `llmgw ctl retry MODEL` is run — and
`ctl status`, `ctl models` and the admin API say so with the last error.
A request for a model that will not come up — parked, or fitting on no node —
is answered at once with the reason, instead of "the model is loading; retry
shortly", which a client would retry forever. Each request still makes the
control plane try placing it afresh, so a model that fits once room is made is
not refused on an old answer. "no placement for model" is logged when its
reason changes or every ten minutes, not on every pass: it was 250 identical
lines in two days for one model.
### Metrics and alerts for what goes wrong
New series: `llmgw_model_replicas_desired`, `llmgw_model_start_failures`,
`llmgw_model_parked`, `llmgw_model_unplaceable`, `llmgw_node_engine_install`,
`llmgw_control_plane_leader` and `llmgw_certificate_expiry_timestamp_seconds`.
`llmgw_model_replicas_ready` is now published at 0 for every model that should
be running; it was published only while a replica was ready, so
ModelHasNothingServing could never fire. New alerts: ModelParked,
ModelFailingToStart, ModelCannotBePlaced, EngineInstallFailed,
ControlPlaneLeadership, CertificateExpiresSoon and CertificateExpiresVerySoon.
`ctl status` counts a model against the replicas it should have now rather
than its minimum, so an on-demand model that was asked for and fitted nowhere
no longer reads as satisfied.
### Backups
`llmgw server backup` writes the store and `/etc/llmgw` — configuration,
secrets, and the certificate authority's key — to one archive readable by root
only, keeping the newest 14. `deploy/systemd/llmgw-backup.timer` runs it
nightly. The control plane publishes the newest archive's time as
`llmgw_backup_last_success_timestamp_seconds`, and BackupTooOld fires when it is
more than 26 hours old or there has never been one. docs/install.md now says how
to restore, including why running engines survive it: the engine-key secret
comes back with the store.
### Fixes
- Tests that slept a fixed time and then asserted now wait for their
condition. They are the likeliest cause of CI's intermittent failures on a
loaded runner.
- `ctl` reported a change the API had queued (202 Accepted) as an error, as
`ctl restart` did on its first real use.
### Releases
Small fixes are published as release candidates of the next version —
+91 -4
View File
@@ -42,10 +42,97 @@ groups:
annotations:
summary: "{{ $labels.model }} has no replica ready"
description: >-
The model is placed and nothing is serving it. Ten minutes, because
a large model on CPU legitimately takes minutes to load. A model
that is deliberately stopped publishes no series at all, so this
cannot fire for one nobody asked for.
The model should be running and nothing is serving it. Ten minutes,
because a large model on CPU legitimately takes minutes to load. The
series is published at 0 for every model that should be running —
before v0.10.4 it was published only while a replica was ready, so
this could not fire at all. A model deliberately stopped, or an
on-demand one nobody is asking for, publishes nothing.
- alert: ModelParked
expr: llmgw_model_parked == 1
labels: { severity: critical }
annotations:
summary: "{{ $labels.model }} failed every time it started and is no longer retried"
description: >-
Five replicas in a row failed at start. Before this existed a model
whose weights its engine could not load was started 155 times over
eight hours. `llmgw ctl status` has the last error; once it is dealt
with, `llmgw ctl retry <model>`. Changing the model's definition
also retries it.
- alert: ModelFailingToStart
expr: llmgw_model_start_failures >= 2
for: 5m
labels: { severity: warning }
annotations:
summary: "{{ $labels.model }} has failed to start {{ $value }} times in a row"
description: >-
On its way to being parked. The engine's own words are in
`llmgw ctl logs <node> --source <shard>` for the next attempt, or in
the control plane's log for the last one.
- alert: ModelCannotBePlaced
expr: llmgw_model_unplaceable == 1
for: 10m
labels: { severity: warning }
annotations:
summary: "{{ $labels.model }} is wanted and fits on no node"
description: >-
Short of replicas, and the last attempt found nowhere with the
memory, cards or engine it needs. `llmgw ctl explain-placement
<model>` says what each node lacks. For an on-demand model, requests
for it are told this rather than asked to retry.
- alert: EngineInstallFailed
expr: llmgw_node_engine_install{state="failed"} == 1
for: 15m
labels: { severity: warning }
annotations:
summary: "{{ $labels.engine }} {{ $labels.version }} will not install on {{ $labels.node }}"
description: >-
The node takes no model needing it until it does. The agent retries
on its own; `llmgw ctl nodes` and the node's logs on the cluster
page say why it failed.
- alert: ControlPlaneLeadership
expr: sum(llmgw_control_plane_leader) != 1
for: 2m
labels: { severity: critical }
annotations:
summary: "{{ $value }} control planes believe they are placing work"
description: >-
None means nothing is placed or replaced; two means both are, each
into GPUs the other believes free. The lease is in the store: check
it is reachable from every control plane.
- alert: CertificateExpiresSoon
expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 30 * 86400
labels: { severity: warning }
annotations:
summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
description: >-
Certificates last a year and nothing renews them. When the agent
server's runs out every agent refuses to connect at once. Reissue
with `llmgw ctl cert` and restart the control plane.
- alert: CertificateExpiresVerySoon
expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 7 * 86400
labels: { severity: critical }
annotations:
summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
description: See CertificateExpiresSoon; a week is not long.
- alert: BackupTooOld
expr: time() - llmgw_backup_last_success_timestamp_seconds > 26 * 3600
labels: { severity: warning }
annotations:
summary: "The newest llmgw backup is {{ $value | humanizeDuration }} old"
description: >-
llmgw-backup.timer runs nightly. Also fires when there has never
been a backup (the series is 0 then). `journalctl -u
llmgw-backup.service` on the control plane says why the last one
failed; docs/install.md, "Backing up", says how to set it up.
- alert: PoolSaturated
expr: llmgw_pool_saturation > 1
+11
View File
@@ -0,0 +1,11 @@
[Unit]
Description=llmgw backup: the store and /etc/llmgw, to /var/backups/llmgw-server
After=llmgw-server.service postgresql.service
[Service]
Type=oneshot
# Root, because the archive includes /etc/llmgw/secrets and the certificate
# authority's key, which the llmgw user deliberately cannot read all of.
ExecStart=/usr/local/bin/llmgw server backup --config /etc/llmgw/server.yaml --keep 14
Nice=10
IOSchedulingClass=idle
+12
View File
@@ -0,0 +1,12 @@
[Unit]
Description=Back up llmgw every night
[Timer]
OnCalendar=*-*-* 03:15:00
# A machine that was off at 03:15 backs up when it next starts, rather than
# waiting another day with a gap nobody notices until the alert.
Persistent=true
RandomizedDelaySec=10m
[Install]
WantedBy=timers.target
+56 -18
View File
@@ -961,36 +961,74 @@ the choice is yours rather than silent.
## Backing up
Three things, in descending order of how badly you want them.
`llmgw server backup` writes one archive holding everything a control plane
cannot be rebuilt without:
**The certificate authority.** `ca-key.pem`, `/etc/llmgw/pki` unless you kept it elsewhere (step 2).
Losing it does not stop the cluster, and it means every node identity has to be
reissued the next time you touch one. Nothing in llmgw can reconstruct it.
- **the store** — users, API keys, the tenancy tree, limits and budgets, usage
history, models added at runtime, measured capacity, and the secret every
engine key is derived from (`pg_dump` in its custom format for PostgreSQL, a
consistent `VACUUM INTO` snapshot for SQLite; both are safe against a running
gateway);
- **`/etc/llmgw`** — the configuration, the store's address, and the
certificate authority's key (`pki/`). Nothing in llmgw can reconstruct that
key; losing it means reissuing every node's identity.
**The store.** Users, API keys, organisations, teams, projects, every limit and
budget, all usage history, models added at runtime, and measured capacity.
Archives go to `/var/backups/llmgw-server`, readable by root only because of the CA
key, and the newest 14 are kept. Run it nightly with the timer:
```
# SQLite
sqlite3 /var/lib/llmgw/llmgw.db ".backup '/backup/llmgw-$(date +%F).db'"
# PostgreSQL
pg_dump --format=custom llmgw > /backup/llmgw-$(date +%F).dump
sudo install -m 0644 deploy/systemd/llmgw-backup.service deploy/systemd/llmgw-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now llmgw-backup.timer
sudo systemctl start llmgw-backup.service # one now, to see it work
ls -l /var/backups/llmgw-server
```
`.backup` and `pg_dump` are both safe against a running gateway. Copying the
SQLite file with `cp` while it is being written is not.
The control plane publishes when the newest archive was written as
`llmgw_backup_last_success_timestamp_seconds`, and **BackupTooOld** fires when
it is more than 26 hours old — including when there has never been one.
**Copy the archives off the machine.** They sit on the disk they back up and do
not survive losing it. Anything that copies files will do — rsync from the
mirror, a NAS mount, an offsite bucket — as long as it keeps the archives
readable by root only.
**Keys are stored hashed and cannot be recovered from a backup or from
anywhere else.** Restoring an old store restores which keys exist and who owns
them, not their secrets. A key created after the backup is gone and has to be
reissued — which is why the store is worth backing up on a schedule rather than
before upgrades.
reissued.
### Restoring
On a machine with llmgw installed (the same version or newer) and, for
PostgreSQL, an empty database and role as `server init --postgres` would create
them:
```
sudo systemctl stop llmgw-server
mkdir /tmp/restore && sudo tar -xzf /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz -C /tmp/restore
cat /tmp/restore/MANIFEST # version and time it was taken
# /etc/llmgw: configuration, secrets, certificates and the CA
sudo cp -a /tmp/restore/etc/llmgw/. /etc/llmgw/
# the store — PostgreSQL
sudo -u postgres pg_restore --clean --if-exists --no-owner --role=llmgw -d llmgw /tmp/restore/store.pgdump
# the store — SQLite
sudo install -o llmgw -g llmgw -m 0600 /tmp/restore/store.sqlite /var/lib/llmgw/llmgw.db
sudo systemctl start llmgw-server
sudo rm -rf /tmp/restore
```
Running engines keep serving throughout: agents reconnect to the restored
control plane and their replicas are adopted, and because the engine-key secret
came back with the store, the router can still reach them.
**The mirror's directory** is the one thing you can rebuild, by downloading it
all again. Worth remembering that a pinned version may no longer be upstream to
download: that is exactly what pinning it protected you from, and a backup is
what makes the protection outlive the disk.
all again. A pinned version may no longer be upstream to download: that is
exactly what pinning it protected you from, and a copy of the directory is what
makes the protection outlive the disk.
## Reclaiming disk
+9
View File
@@ -169,6 +169,7 @@ rather than repeated.
| `monitoring` | `Monitoring` | [Server](#server) |
| `mirror` | `Mirror` | [Server](#server) |
| `engines` | `map[string]string` | [Server](#server) |
| `backup_dir` | `string` | [Server](#server) |
| `router` | `Router` | [Server](#server) |
| `control` | `Control` | [Server](#server) |
| `models` | `[]catalog.Model` | [Server](#server) |
@@ -1265,6 +1266,14 @@ never trusted to already have what somebody once put there.
Unset keeps DefaultEngines. A version replaces the default; an empty
one leaves that engine to be installed by hand.
### `backup_dir`
`string`
BackupDir is where `llmgw server backup` writes, and where the newest
archive's age is read for llmgw_backup_last_success_timestamp_seconds.
Empty means /var/backups/llmgw-server.
### `router`
`Router`
+9 -4
View File
@@ -400,11 +400,16 @@ func TestHeartbeatKeepsNodeAlive(t *testing.T) {
n, _ := h.inv.Get("n1")
first := n.LastSeen
time.Sleep(200 * time.Millisecond)
n, _ = h.inv.Get("n1")
if !n.LastSeen.After(first) {
t.Error("heartbeats are not refreshing LastSeen; the node would be declared lost")
// Waited for rather than slept through: four heartbeat intervals was not
// always enough on a loaded runner for one to have landed.
deadline := time.Now().Add(3 * time.Second)
for time.Now().Before(deadline) {
if n, _ = h.inv.Get("n1"); n.LastSeen.After(first) {
return
}
time.Sleep(20 * time.Millisecond)
}
t.Error("heartbeats are not refreshing LastSeen; the node would be declared lost")
}
func TestStartAndStopShardEndToEnd(t *testing.T) {
+29 -8
View File
@@ -243,7 +243,9 @@ func TestWaitReadyTimeoutIncludesLogs(t *testing.T) {
if err != nil {
t.Fatal(err)
}
time.Sleep(100 * time.Millisecond) // let the line be captured
eventually(t, 3*time.Second, "the line was never captured", func() bool {
return strings.Contains(strings.Join(sh.Logs(10), "\n"), "CUDA out of memory")
})
err = s.WaitReady(t.Context(), sh, "http://x", "m", 200*time.Millisecond)
if err == nil {
@@ -295,7 +297,10 @@ func TestLogsAreCaptured(t *testing.T) {
if err != nil {
t.Fatal(err)
}
time.Sleep(400 * time.Millisecond)
eventually(t, 3*time.Second, "not every line arrived", func() bool {
l := strings.Join(sh.Logs(10), "\n")
return strings.Contains(l, "line1") && strings.Contains(l, "line2") && strings.Contains(l, "line3")
})
logs := strings.Join(sh.Logs(10), "\n")
for _, want := range []string{"line1", "line2", "line3"} {
@@ -372,11 +377,11 @@ func TestStateFileRemovedOnExit(t *testing.T) {
sh, _ := s.Start(t.Context(), "a", engines.ShardSpec{}, shellCommand("exit 0"))
waitState(t, sh, Failed, 3*time.Second)
time.Sleep(50 * time.Millisecond)
if _, err := os.Stat(s.statePath("a")); !os.IsNotExist(err) {
t.Error("the state file outlived the process; a later scan would find a dead orphan")
}
eventually(t, 3*time.Second,
"the state file outlived the process; a later scan would find a dead orphan", func() bool {
_, err := os.Stat(s.statePath("a"))
return os.IsNotExist(err)
})
}
// --- state machine and ring buffer ---
@@ -743,7 +748,9 @@ func TestWaitAliveFailsOnEarlyExit(t *testing.T) {
if err := s.WaitAlive(t.Context(), sh); err == nil {
t.Fatal("a worker that died immediately was reported ready")
}
if d := time.Since(start); d > 1500*time.Millisecond {
// Below AliveGrace (5s), which is what waiting it out would take; the
// headroom is for a loaded runner, where 1.5s was sometimes not enough.
if d := time.Since(start); d > 4*time.Second {
t.Errorf("waited %v for a process that had already exited", d)
}
}
@@ -827,3 +834,17 @@ func TestAWorkersOnlyProcessWaitsForTheLeader(t *testing.T) {
t.Errorf("the worker started after %v, before the leader was listening", d)
}
}
// eventually waits for cond, failing with msg if it does not hold within
// limit. Sleeping a fixed time and then checking measured the machine the
// test ran on, and a loaded CI runner under the race detector lost that race.
func eventually(t *testing.T, limit time.Duration, msg string, cond func() bool) {
t.Helper()
deadline := time.Now().Add(limit)
for !cond() {
if time.Now().After(deadline) {
t.Fatal(msg)
}
time.Sleep(10 * time.Millisecond)
}
}
+310
View File
@@ -0,0 +1,310 @@
package cli
import (
"archive/tar"
"compress/gzip"
"context"
"errors"
"fmt"
"io"
"io/fs"
"net/url"
"os"
"os/exec"
"path/filepath"
"slices"
"strings"
"time"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
)
// DefaultBackupDir is where backups are written and where the control plane
// looks for the newest one.
//
// Its own directory rather than /var/backups/llmgw, which a PostgreSQL move
// had already made for its own dumps, owned by postgres and closed to
// everyone else — so the control plane could not see a backup landing in it.
const DefaultBackupDir = "/var/backups/llmgw-server"
func newBackupCmd() *cobra.Command {
var (
configPath string
etcDir string
dir string
keep int
)
c := &cobra.Command{
Use: "backup",
Short: "Write the store and /etc/llmgw to one archive, and keep the last few",
Long: "Everything a control plane cannot be rebuilt without: the store — keys,\n" +
"people, quotas, usage, the models added at runtime, the secret engine\n" +
"keys derive from — and /etc/llmgw, which holds the configuration, the\n" +
"store's address and the certificate authority's key.\n\n" +
"The archive holds that key, so it is written readable by root only. It\n" +
"is on the same disk as what it backs up: copy it elsewhere, or it does\n" +
"not survive the loss of this machine.\n\n" +
"Run daily by llmgw-backup.timer (`llmgw server backup`). The control plane publishes the time of\n" +
"the newest archive as llmgw_backup_last_success_timestamp_seconds, and\n" +
"BackupTooOld fires when it is more than a day old.",
RunE: func(cmd *cobra.Command, _ []string) error {
cfg, err := config.Load(configPath)
if err != nil {
return err
}
path, err := writeBackup(cmd.Context(), cfg.Auth.Store, etcDir, dir)
if err != nil {
return err
}
if why := unlistable(dir); why != "" {
fmt.Fprintf(cmd.ErrOrStderr(), "warning: %s; the control plane will report no backup "+
"and BackupTooOld will fire\n", why)
}
removed, err := pruneBackups(dir, keep)
if err != nil {
return err
}
fmt.Fprintf(cmd.OutOrStdout(), "wrote %s", path)
if removed > 0 {
fmt.Fprintf(cmd.OutOrStdout(), "; removed %d older", removed)
}
fmt.Fprintln(cmd.OutOrStdout())
return nil
},
}
c.Flags().StringVar(&configPath, "config", "/etc/llmgw/server.yaml", "the control plane's configuration")
c.Flags().StringVar(&etcDir, "etc", "/etc/llmgw", "configuration directory to include")
c.Flags().StringVar(&dir, "dir", DefaultBackupDir, "where archives are written")
c.Flags().IntVar(&keep, "keep", 14, "how many archives to keep")
return c
}
const backupPrefix = "llmgw-backup-"
// writeBackup writes one archive and returns its path. Written beside its
// final name and renamed, so a backup interrupted halfway is never the newest
// one the control plane reports.
func writeBackup(ctx context.Context, storeAddr, etcDir, dir string) (string, error) {
// Listable by anyone, so the control plane — not root — can see when the
// newest archive was written. The archives themselves are root's alone.
if err := os.MkdirAll(dir, 0o755); err != nil {
return "", err
}
host, _ := os.Hostname()
name := backupPrefix + host + "-" + time.Now().UTC().Format("20060102T150405Z") + ".tar.gz"
final := filepath.Join(dir, name)
part := final + ".part"
f, err := os.OpenFile(part, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o600)
if err != nil {
return "", err
}
ok := false
defer func() {
if !ok {
_ = os.Remove(part)
}
}()
gz := gzip.NewWriter(f)
tw := tar.NewWriter(gz)
manifest := fmt.Sprintf("llmgw %s\nhost %s\ntime %s\n", version.Version, host,
time.Now().UTC().Format(time.RFC3339))
if err := addBytes(tw, "MANIFEST", []byte(manifest)); err != nil {
return "", err
}
if storeAddr != "" {
if err := addStore(ctx, tw, storeAddr, dir); err != nil {
return "", fmt.Errorf("the store: %w", err)
}
}
if err := addTree(tw, etcDir, "etc/llmgw"); err != nil {
return "", fmt.Errorf("%s: %w", etcDir, err)
}
if err := errors.Join(tw.Close(), gz.Close(), f.Sync(), f.Close()); err != nil {
return "", err
}
if err := os.Rename(part, final); err != nil {
return "", err
}
ok = true
return final, nil
}
// addStore dumps the store into the archive.
func addStore(ctx context.Context, tw *tar.Writer, addr, scratch string) error {
addr, err := store.ResolveAddress(addr)
if err != nil {
return err
}
tmp, err := os.CreateTemp(scratch, ".store-*")
if err != nil {
return err
}
tmpPath := tmp.Name()
_ = tmp.Close()
defer os.Remove(tmpPath)
name := "store.sqlite"
if store.IsPostgres(addr) {
name = "store.pgdump"
if err := pgDump(ctx, addr, tmpPath); err != nil {
return err
}
} else {
// VACUUM INTO refuses a file that exists.
_ = os.Remove(tmpPath)
db, err := store.Open(ctx, addr)
if err != nil {
return err
}
err = db.SnapshotTo(ctx, tmpPath)
_ = db.Close()
if err != nil {
return err
}
}
return addFile(tw, tmpPath, name)
}
// pgDump runs pg_dump in its custom format, which pg_restore can restore
// selectively. The connection is given through the environment, not the
// command line: a DSN carries the password, and every user on the machine
// can read another process's arguments.
func pgDump(ctx context.Context, dsn, out string) error {
u, err := url.Parse(dsn)
if err != nil {
return fmt.Errorf("the store address is not a URL")
}
env := os.Environ()
set := func(k, v string) {
if v != "" {
env = append(env, k+"="+v)
}
}
set("PGHOST", u.Hostname())
set("PGPORT", u.Port())
set("PGUSER", u.User.Username())
if pw, ok := u.User.Password(); ok {
set("PGPASSWORD", pw)
}
set("PGDATABASE", strings.TrimPrefix(u.Path, "/"))
set("PGSSLMODE", u.Query().Get("sslmode"))
cmd := exec.CommandContext(ctx, "pg_dump", "--format=custom", "--no-owner", "--file="+out) //nolint:gosec // fixed binary, our own file
cmd.Env = env
if b, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("pg_dump: %v: %s", err, strings.TrimSpace(string(b)))
}
return nil
}
func addTree(tw *tar.Writer, root, as string) error {
return filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
rel, _ := filepath.Rel(root, p)
name := filepath.ToSlash(filepath.Join(as, rel))
info, err := d.Info()
if err != nil {
return err
}
switch {
case d.IsDir():
return tw.WriteHeader(&tar.Header{Name: name + "/", Mode: int64(info.Mode().Perm()),
Typeflag: tar.TypeDir, ModTime: info.ModTime()})
case info.Mode().IsRegular():
return addFile(tw, p, name)
}
return nil // sockets, links: not configuration
})
}
func addFile(tw *tar.Writer, path, name string) error {
f, err := os.Open(path) //nolint:gosec // files we chose to back up
if err != nil {
return err
}
defer f.Close()
info, err := f.Stat()
if err != nil {
return err
}
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: int64(info.Mode().Perm()),
Size: info.Size(), ModTime: info.ModTime(), Typeflag: tar.TypeReg}); err != nil {
return err
}
_, err = io.Copy(tw, f)
return err
}
func addBytes(tw *tar.Writer, name string, b []byte) error {
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o600, Size: int64(len(b)),
ModTime: time.Now(), Typeflag: tar.TypeReg}); err != nil {
return err
}
_, err := tw.Write(b)
return err
}
// pruneBackups keeps the newest keep archives. Only files this command
// wrote are considered, so nothing else in the directory is ever removed.
func pruneBackups(dir string, keep int) (int, error) {
entries, err := os.ReadDir(dir)
if err != nil {
return 0, err
}
var names []string
for _, e := range entries {
if n := e.Name(); strings.HasPrefix(n, backupPrefix) && strings.HasSuffix(n, ".tar.gz") {
names = append(names, n)
}
}
// The name carries a UTC timestamp, so name order is age order.
slices.Sort(names)
removed := 0
for len(names) > max(keep, 1) {
if err := os.Remove(filepath.Join(dir, names[0])); err != nil {
return removed, err
}
names = names[1:]
removed++
}
return removed, nil
}
// NewestBackup is the modification time of the newest complete archive in
// dir, or zero when there is none.
func NewestBackup(dir string) time.Time {
entries, err := os.ReadDir(dir)
if err != nil {
return time.Time{}
}
var newest time.Time
for _, e := range entries {
if n := e.Name(); !strings.HasPrefix(n, backupPrefix) || !strings.HasSuffix(n, ".tar.gz") {
continue
}
if info, err := e.Info(); err == nil && info.ModTime().After(newest) {
newest = info.ModTime()
}
}
return newest
}
// unlistable says why the control plane, which does not run as root, could
// not see the archives in dir, or "" when it can.
func unlistable(dir string) string {
info, err := os.Stat(dir)
if err != nil {
return err.Error()
}
if info.Mode().Perm()&0o005 != 0o005 {
return fmt.Sprintf("%s is %v, so only its owner can list it", dir, info.Mode().Perm())
}
return ""
}
+118
View File
@@ -0,0 +1,118 @@
package cli
import (
"archive/tar"
"compress/gzip"
"io"
"os"
"path/filepath"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
func TestABackupHoldsTheStoreAndTheConfiguration(t *testing.T) {
tmp := t.TempDir()
dbPath := filepath.Join(tmp, "store.db")
db, err := store.Open(t.Context(), dbPath)
if err != nil {
t.Fatal(err)
}
secret, err := db.ClusterSecret(t.Context(), "engine-key")
if err != nil {
t.Fatal(err)
}
_ = db.Close()
etc := filepath.Join(tmp, "etc")
_ = os.MkdirAll(filepath.Join(etc, "pki"), 0o700)
_ = os.WriteFile(filepath.Join(etc, "server.yaml"), []byte("listen: :8080\n"), 0o644)
_ = os.WriteFile(filepath.Join(etc, "pki", "ca-key.pem"), []byte("KEY"), 0o600)
out := filepath.Join(tmp, "backups")
path, err := writeBackup(t.Context(), dbPath, etc, out)
if err != nil {
t.Fatal(err)
}
info, _ := os.Stat(path)
if info.Mode().Perm() != 0o600 {
t.Errorf("the archive holds the CA's key and is %v", info.Mode().Perm())
}
files := map[string][]byte{}
f, _ := os.Open(path)
defer f.Close()
gz, err := gzip.NewReader(f)
if err != nil {
t.Fatal(err)
}
tr := tar.NewReader(gz)
for {
h, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
t.Fatal(err)
}
b, _ := io.ReadAll(tr)
files[h.Name] = b
}
for _, want := range []string{"MANIFEST", "store.sqlite", "etc/llmgw/server.yaml", "etc/llmgw/pki/ca-key.pem"} {
if _, ok := files[want]; !ok {
t.Errorf("%s is not in the archive", want)
}
}
// And the store in it is a working store, with what was in it.
restored := filepath.Join(tmp, "restored.db")
_ = os.WriteFile(restored, files["store.sqlite"], 0o600)
rdb, err := store.Open(t.Context(), restored)
if err != nil {
t.Fatalf("the backed-up store does not open: %v", err)
}
defer rdb.Close()
if got, _ := rdb.ClusterSecret(t.Context(), "engine-key"); got != secret {
t.Error("the restored store lost the engine key secret; every engine would be locked out")
}
}
func TestOldBackupsArePrunedAndTheNewestIsFound(t *testing.T) {
dir := t.TempDir()
for i, stamp := range []string{"20260101T000000Z", "20260102T000000Z", "20260103T000000Z"} {
p := filepath.Join(dir, backupPrefix+"h-"+stamp+".tar.gz")
_ = os.WriteFile(p, []byte("x"), 0o600)
mt := time.Date(2026, 1, 1+i, 0, 0, 0, 0, time.UTC)
_ = os.Chtimes(p, mt, mt)
}
_ = os.WriteFile(filepath.Join(dir, "someone-elses-file"), []byte("x"), 0o600)
_ = os.WriteFile(filepath.Join(dir, backupPrefix+"h-20260104T000000Z.tar.gz.part"), []byte("x"), 0o600)
removed, err := pruneBackups(dir, 2)
if err != nil || removed != 1 {
t.Fatalf("removed %d, %v", removed, err)
}
if _, err := os.Stat(filepath.Join(dir, "someone-elses-file")); err != nil {
t.Error("a file the backup did not write was removed")
}
newest := NewestBackup(dir)
if !newest.Equal(time.Date(2026, 1, 3, 0, 0, 0, 0, time.UTC)) {
t.Errorf("newest %v; an interrupted .part must not count", newest)
}
if !NewestBackup(filepath.Join(dir, "none")).IsZero() {
t.Error("a missing directory reported a backup")
}
}
func TestABackupDirectoryTheControlPlaneCannotListIsSaid(t *testing.T) {
dir := t.TempDir()
_ = os.Chmod(dir, 0o700)
if unlistable(dir) == "" {
t.Error("a directory only its owner can list was taken as visible")
}
_ = os.Chmod(dir, 0o755)
if why := unlistable(dir); why != "" {
t.Errorf("a listable directory: %s", why)
}
}
+9 -1
View File
@@ -83,7 +83,10 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
// Any 2xx. The API answers a change it has queued rather than made — a
// restart — with 202, and treating that as a failure printed
// "Error: 202 Accepted" over a request that had worked.
if resp.StatusCode < 200 || resp.StatusCode > 299 {
var e struct {
Error string `json:"error"`
}
@@ -136,6 +139,7 @@ func newCtlCmd() *cobra.Command {
newModelCmd(client),
newScaleCmd(client),
newRestartCmd(client),
newRetryCmd(client),
newEngineCheckCmd(),
newToolCheckCmd(),
newWeightsCmd(client, &asJSON),
@@ -245,6 +249,10 @@ func newNodesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
// actually served, never what is merely catalogued.
func modelStatus(m adminapi.ModelView) string {
switch {
case m.Parked:
return "parked"
case m.Unplaceable:
return "cannot place"
case m.Min == 0 && m.Total == 0:
return "stopped"
case m.Ready >= m.Min && m.Min > 0:
+27
View File
@@ -46,3 +46,30 @@ func newRestartCmd(client func() *ctlClient) *cobra.Command {
c.Flags().BoolVar(&all, "all", false, "every ready replica in the cluster")
return c
}
func newRetryCmd(client func() *ctlClient) *cobra.Command {
return &cobra.Command{
Use: "retry MODEL",
Short: "Try a parked model again, and clear its backoff",
Long: "A model whose replicas fail every time they start is parked after five\n" +
"failures in a row: it is not placed again until its definition changes\n" +
"or this is run. Run it once the cause — the weights, the engine, the\n" +
"profile — has been dealt with; `llmgw ctl status` lists parked models with\n" +
"their last error.",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
var res struct {
WasParked bool `json:"was_parked"`
}
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/retry", []byte(`{}`), &res); err != nil {
return err
}
if res.WasParked {
fmt.Printf("%s unparked; it is placed on the next pass\n", args[0])
} else {
fmt.Printf("%s was not parked; its backoff is cleared\n", args[0])
}
return nil
},
}
}
+24
View File
@@ -0,0 +1,24 @@
package cli
import (
"net/http"
"net/http/httptest"
"testing"
)
func TestAQueuedChangeIsNotReportedAsAnError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusAccepted)
_, _ = w.Write([]byte(`{"queued":["r1"]}`))
}))
defer srv.Close()
var out struct {
Queued []string `json:"queued"`
}
if err := newCtlClient(srv.URL).post(t.Context(), "/api/v1/restart", []byte(`{}`), &out); err != nil {
t.Fatalf("202 read as a failure: %v", err)
}
if len(out.Queued) != 1 {
t.Errorf("the body of a 202 was not read: %+v", out)
}
}
+25
View File
@@ -7,6 +7,7 @@ import (
"fmt"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
"github.com/prometheus/client_golang/prometheus"
"log/slog"
"net"
"net/http"
@@ -36,6 +37,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
"gitea.mynodes.xyz/ilker/llmgw/internal/router"
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
"gitea.mynodes.xyz/ilker/llmgw/internal/web"
)
@@ -131,6 +133,28 @@ func newServerCmd() *cobra.Command {
// On the same /metrics as the traffic figures: correlating
// "the cluster was full" with "requests were queueing" should
// not need two systems.
// When the newest backup was written, so its absence alerts
// rather than being found on the day it is needed.
backupDir := cfg.BackupDir
if backupDir == "" {
backupDir = DefaultBackupDir
}
if err := rt.RegisterMetrics(prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "llmgw_backup_last_success_timestamp_seconds",
Help: "When the newest backup archive was written, as a Unix time; 0 when there is none.",
}, func() float64 {
if t := NewestBackup(backupDir); !t.IsZero() {
return float64(t.Unix())
}
return 0
})); err != nil {
return err
}
if err := rt.RegisterMetrics(tlsconf.ExpiryCollector(map[string]string{
"agent-server": cfg.AgentTLS.Cert, "agent-ca": cfg.AgentTLS.CA,
})); err != nil {
return err
}
if err := rt.RegisterMetrics(fleetmetrics.New(ctl)); err != nil {
return err
}
@@ -361,6 +385,7 @@ func newServerCmd() *cobra.Command {
f.StringArrayVar(&endpoints, "endpoint", nil, "static endpoint: id=URL=model[,model] (repeatable)")
cmd.AddCommand(newServerInitCmd())
cmd.AddCommand(newBackupCmd())
return cmd
}
+8
View File
@@ -94,6 +94,14 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
fmt.Fprintf(out, "\nMODELS %d of %d have the replicas they asked for\n",
s.Models.Satisfied, s.Models.Total)
if len(s.Parked) > 0 {
fmt.Fprintln(out, "\nPARKED — failed every time they started; not retried until fixed")
for _, p := range s.Parked {
fmt.Fprintf(out, " %s (%d failures): %s\n", p.Model, p.Failures, p.Reason)
}
fmt.Fprintln(out, "\n Once fixed: llmgw ctl retry <model>")
}
if len(s.Unplaceable) > 0 {
fmt.Fprintln(out, "\nNOT PLACED")
for _, m := range s.Unplaceable {
+5
View File
@@ -81,6 +81,11 @@ type Server struct {
// one leaves that engine to be installed by hand.
Engines map[string]string `koanf:"engines"`
// BackupDir is where `llmgw server backup` writes, and where the newest
// archive's age is read for llmgw_backup_last_success_timestamp_seconds.
// Empty means /var/backups/llmgw-server.
BackupDir string `koanf:"backup_dir"`
Router Router `koanf:"router"`
Control Control `koanf:"control"`
Models []catalog.Model `koanf:"models"`
+30
View File
@@ -78,6 +78,7 @@ func (a *API) Register(mux *http.ServeMux) {
mux.HandleFunc("POST /api/v1/fabric-test", a.fabricTest)
mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
mux.HandleFunc("POST /api/v1/restart", a.restart)
mux.HandleFunc("POST /api/v1/models/{name}/retry", a.retryModel)
}
// --- responses ---
@@ -233,6 +234,18 @@ type ModelView struct {
Max int `json:"max_replicas"`
Ready int `json:"ready_replicas"`
Total int `json:"total_replicas"`
// Desired is how many it should have now: Min, or for an on-demand
// model one while it is being asked for and none otherwise.
Desired int `json:"desired_replicas"`
// Failures counts replicas that failed to start in a row.
Failures int `json:"failures,omitempty"`
// Parked is set once Failures reached the limit; the model is not retried
// until its definition changes or `ctl retry` is run. Reason is the last
// error.
Parked bool `json:"parked,omitempty"`
Reason string `json:"reason,omitempty"`
// Unplaceable is set when it is short and the last attempt found no node.
Unplaceable bool `json:"unplaceable,omitempty"`
Fallbacks []string `json:"fallbacks,omitempty"`
Profiles []string `json:"profiles"`
}
@@ -442,13 +455,20 @@ func (a *API) Models() []ModelView {
}
}
health := map[string]reconciler.ModelHealth{}
for _, h := range a.ctl.Reconciler().ModelHealth() {
health[h.Model] = h
}
out := make([]ModelView, 0)
for _, m := range a.ctl.Catalog().Models() {
h := health[m.Name]
v := ModelView{
Name: m.Name, Description: m.Description, Aliases: m.Aliases, Source: m.Source,
Min: m.Scaling.Min, Max: m.Scaling.Max,
Ready: ready[m.Name], Total: total[m.Name],
Fallbacks: m.Fallbacks,
Desired: h.Desired, Failures: h.Failures,
Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
}
for _, p := range m.Profiles {
v.Profiles = append(v.Profiles, p.Name)
@@ -1085,3 +1105,13 @@ func (a *API) restart(w http.ResponseWriter, r *http.Request) {
}
writeJSON(w, http.StatusAccepted, RestartResult{Queued: ids})
}
func (a *API) retryModel(w http.ResponseWriter, r *http.Request) {
name := r.PathValue("name")
if _, ok := a.ctl.Catalog().Resolve(name); !ok {
writeError(w, http.StatusNotFound, "no model called "+name)
return
}
was := a.ctl.Reconciler().Retry(name)
writeJSON(w, http.StatusOK, map[string]bool{"was_parked": was})
}
+18 -2
View File
@@ -25,6 +25,9 @@ type Summary struct {
// Unplaceable names models the scheduler currently cannot satisfy. This is
// the line that turns "something is wrong" into a place to look.
Unplaceable []string `json:"unplaceable,omitempty"`
// Parked are models that failed every time they started and are no
// longer retried, with the last error.
Parked []ParkedModel `json:"parked,omitempty"`
// IdleReplicas are serving replicas nothing has asked for in a while, which
// on a fleet with more models than GPUs is the first place to find room.
IdleReplicas []IdleReplica `json:"idle_replicas,omitempty"`
@@ -201,17 +204,30 @@ func (a *API) Summary() Summary {
for _, m := range a.Models() {
s.Models.Total++
if m.Ready >= m.Min {
if m.Parked {
s.Parked = append(s.Parked, ParkedModel{Model: m.Name, Failures: m.Failures, Reason: m.Reason})
}
// Against what it should have now, not its minimum: an on-demand
// model's minimum is 0, so one that was asked for and fitted nowhere
// used to count as satisfied.
if m.Ready >= m.Desired && !m.Parked {
s.Models.Satisfied++
continue
}
s.Models.Short++
// Short of replicas and unable to place more is the actionable case;
// short and still starting is just a cluster doing its job.
if m.Total <= m.Ready {
if m.Unplaceable || (m.Total <= m.Ready && !m.Parked) {
s.Unplaceable = append(s.Unplaceable, m.Name)
}
}
sort.Strings(s.Unplaceable)
return s
}
// ParkedModel is a model no longer retried, and why.
type ParkedModel struct {
Model string `json:"model"`
Failures int `json:"failures"`
Reason string `json:"reason"`
}
+3
View File
@@ -171,6 +171,9 @@ func (c *Controller) Nodes() []inventory.Node { return c.inv.Nodes() }
func (c *Controller) FreeGPUs(nodeID string) []int { return c.inv.FreeGPUs(nodeID) }
func (c *Controller) Replicas() []reconciler.Replica { return c.rec.Replicas() }
// ModelHealth is how each model in the catalog stands.
func (c *Controller) ModelHealth() []reconciler.ModelHealth { return c.rec.ModelHealth() }
// Reconciler exposes the reconciliation loop, for status reports and ctl.
func (c *Controller) Reconciler() *reconciler.Reconciler { return c.rec }
+76 -4
View File
@@ -39,9 +39,25 @@ type Source interface {
// derived from state that already exists, so computing them when asked is both
// simpler and fresher than maintaining a parallel copy that can drift. Nothing
// runs between scrapes.
// HealthSource adds what the reconciler knows about each model and where
// engine installs stand, and whether this control plane is the one placing
// work. Asked for by type assertion, so a source without it — a test's —
// publishes what it did before.
type HealthSource interface {
ModelHealth() []reconciler.ModelHealth
IsLeader() bool
}
type Collector struct {
src Source
modelDesired *prometheus.Desc
modelFailures *prometheus.Desc
modelParked *prometheus.Desc
modelUnplaceable *prometheus.Desc
engineInstall *prometheus.Desc
leader *prometheus.Desc
nodeUp *prometheus.Desc
nodeGPUs *prometheus.Desc
nodeGPUsFree *prometheus.Desc
@@ -137,7 +153,26 @@ func New(src Source) *Collector {
replicaState: prometheus.NewDesc("llmgw_replicas",
"Replicas by model and state.", []string{"model", "state"}, nil),
replicaReady: prometheus.NewDesc("llmgw_model_replicas_ready",
"Replicas of a model that are serving.", []string{"model"}, nil),
"Replicas of a model that are serving. Published, as 0 when none are, for every "+
"model that should be running; a stopped model, or an on-demand one nobody is "+
"asking for, publishes nothing.", []string{"model"}, nil),
modelDesired: prometheus.NewDesc("llmgw_model_replicas_desired",
"Replicas a model should have now: its minimum, or for an on-demand model 1 while "+
"it is being asked for.", []string{"model"}, nil),
modelFailures: prometheus.NewDesc("llmgw_model_start_failures",
"Replicas of a model that failed to start in a row. Resets when one comes up.",
[]string{"model"}, nil),
modelParked: prometheus.NewDesc("llmgw_model_parked",
"1 when a model failed so often in a row that it is no longer retried.",
[]string{"model"}, nil),
modelUnplaceable: prometheus.NewDesc("llmgw_model_unplaceable",
"1 when a model is short of replicas and the last attempt found no node to place one on.",
[]string{"model"}, nil),
engineInstall: prometheus.NewDesc("llmgw_node_engine_install",
"Where installing an engine stands on a node: 1 for the current state.",
[]string{"node", "engine", "version", "state"}, nil),
leader: prometheus.NewDesc("llmgw_control_plane_leader",
"1 when this control plane is the one placing work, 0 when it is a standby.", nil, nil),
replicaIdle: prometheus.NewDesc("llmgw_replica_idle_seconds",
"Seconds since a replica was last sent a request. Absent when nothing has been reported, "+
"which is not the same as idle.", []string{"model", "replica"}, nil),
@@ -201,6 +236,8 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
c.gpuNVLink, c.gpuECC,
c.replicaState, c.replicaReady, c.replicaIdle,
c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable,
c.engineInstall, c.leader,
} {
ch <- d
}
@@ -387,8 +424,43 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
ch <- prometheus.MustNewConstMetric(c.replicaState, prometheus.GaugeValue,
float64(v), k[0], k[1])
}
for model, v := range ready {
ch <- prometheus.MustNewConstMetric(c.replicaReady, prometheus.GaugeValue,
float64(v), model)
hs, ok := c.src.(HealthSource)
if !ok {
for model, v := range ready {
ch <- prometheus.MustNewConstMetric(c.replicaReady, prometheus.GaugeValue,
float64(v), model)
}
return
}
for _, n := range c.src.Nodes() {
for _, p := range n.Provision {
ch <- prometheus.MustNewConstMetric(c.engineInstall, prometheus.GaugeValue, 1,
n.ID, p.Engine, p.Version, p.State)
}
}
lead := 0.0
if hs.IsLeader() {
lead = 1
}
ch <- prometheus.MustNewConstMetric(c.leader, prometheus.GaugeValue, lead)
for _, h := range hs.ModelHealth() {
// Ready is published at zero for what should be running — the one
// value ModelHasNothingServing exists to catch, and which, emitted
// only for replicas that were Ready, it never saw.
if h.Desired > 0 || ready[h.Model] > 0 {
ch <- prometheus.MustNewConstMetric(c.replicaReady, prometheus.GaugeValue,
float64(ready[h.Model]), h.Model)
}
ch <- prometheus.MustNewConstMetric(c.modelDesired, prometheus.GaugeValue, float64(h.Desired), h.Model)
ch <- prometheus.MustNewConstMetric(c.modelFailures, prometheus.GaugeValue, float64(h.Failures), h.Model)
ch <- prometheus.MustNewConstMetric(c.modelParked, prometheus.GaugeValue, b2f(h.Parked), h.Model)
ch <- prometheus.MustNewConstMetric(c.modelUnplaceable, prometheus.GaugeValue, b2f(h.Unplaceable), h.Model)
}
}
func b2f(b bool) float64 {
if b {
return 1
}
return 0
}
@@ -196,3 +196,35 @@ func TestEveryIBPortIsReportedWithItsCounters(t *testing.T) {
}
}
}
type healthSource struct {
fakeSource
health []reconciler.ModelHealth
}
func (h healthSource) ModelHealth() []reconciler.ModelHealth { return h.health }
func (h healthSource) IsLeader() bool { return true }
func TestAModelThatShouldRunPublishesZeroReplicasReady(t *testing.T) {
// ModelHasNothingServing alerts on llmgw_model_replicas_ready == 0, and
// the series was only ever published while a replica was ready — so the
// alert could not fire for the case it was written for.
src := healthSource{health: []reconciler.ModelHealth{
{Model: "wanted", Desired: 1, Failures: 5, Parked: true, Reason: "boom"},
{Model: "stopped", Desired: 0},
}}
out := gather(t, src)
for _, want := range []string{
"llmgw_model_replicas_ready model=wanted -> 0",
"llmgw_model_parked model=wanted -> 1",
"llmgw_model_start_failures model=wanted -> 5",
"llmgw_control_plane_leader -> 1",
} {
if !strings.Contains(out, want) {
t.Errorf("missing %s", want)
}
}
if strings.Contains(out, "llmgw_model_replicas_ready model=stopped") {
t.Error("a stopped model publishes a ready count, so the alert would fire for it")
}
}
+206 -14
View File
@@ -3,11 +3,15 @@ package reconciler
import (
"context"
crypterand "crypto/rand"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"log/slog"
"math/rand/v2"
"slices"
"strings"
"sync"
"time"
@@ -66,6 +70,15 @@ type Config struct {
// Without a bound, draining a node whose model cannot be placed anywhere
// else would never finish.
DrainGrace time.Duration
// ParkAfter is how many failures in a row stop a model being retried.
//
// Backoff alone never gives up: it grows to MaxBackoff and keeps trying,
// and a model whose weights its engine cannot load — an architecture the
// installed llama.cpp does not know — was started 155 times over eight
// hours on the live cluster, each time failing the same way, with nothing
// to show for it but the log. A parked model is not placed again until its
// definition changes or an operator runs `llmgw ctl retry`.
ParkAfter int
// PortRange is the per-node port range for engine HTTP and dist-init.
PortRange [2]int
// MgmtInterface is the ethernet interface used for NCCL bootstrap.
@@ -126,6 +139,7 @@ func DefaultConfig() Config {
EvictionGrace: 2 * time.Minute,
MinLifetime: 5 * time.Minute,
DrainGrace: 5 * time.Minute,
ParkAfter: 5,
PortRange: [2]int{20000, 20999},
MgmtInterface: "eth0",
}
@@ -165,6 +179,9 @@ func (c *Config) normalize() {
if c.DrainGrace <= 0 {
c.DrainGrace = d.DrainGrace
}
if c.ParkAfter <= 0 {
c.ParkAfter = d.ParkAfter
}
if c.PortRange[1] <= c.PortRange[0] {
c.PortRange = d.PortRange
}
@@ -209,6 +226,11 @@ type Reconciler struct {
// restarted and not yet begun. One is replaced at a time: replacing them
// all at once would need the memory of every replica twice over.
restartQueue map[string][]string
// parked holds models that failed ParkAfter times in a row.
parked map[string]parkRecord
// placementLogged remembers the last "no placement" said per model, so it
// is said when it changes rather than on every five-second pass.
placementLogged map[string]placementNote
wake chan struct{}
stop chan struct{}
@@ -230,19 +252,21 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
}
return &Reconciler{
cfg: cfg, cat: cat, inv: inv, disp: disp, log: log,
sched: placement.New(inv),
now: time.Now,
replicas: map[string]*Replica{},
nextAttempt: map[string]time.Time{},
attempts: map[string]int{},
reservedPorts: map[int]struct{}{},
wanted: map[string]time.Time{},
restartQueue: map[string][]string{},
lastExplain: map[string]placement.Result{},
wake: make(chan struct{}, 1),
stop: make(chan struct{}),
done: make(chan struct{}),
startedAt: time.Now(),
sched: placement.New(inv),
now: time.Now,
replicas: map[string]*Replica{},
nextAttempt: map[string]time.Time{},
attempts: map[string]int{},
reservedPorts: map[int]struct{}{},
wanted: map[string]time.Time{},
restartQueue: map[string][]string{},
parked: map[string]parkRecord{},
placementLogged: map[string]placementNote{},
lastExplain: map[string]placement.Result{},
wake: make(chan struct{}, 1),
stop: make(chan struct{}),
done: make(chan struct{}),
startedAt: time.Now(),
}
}
@@ -464,6 +488,12 @@ func (r *Reconciler) Want(model string) bool {
}
r.mu.Lock()
r.wanted[m.Name] = r.now()
// A placement that failed before this request says nothing about now:
// forgotten, so the pass this triggers decides afresh and Unservable
// does not answer from a stale result.
if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
delete(r.lastExplain, m.Name)
}
r.mu.Unlock()
// Immediately, not on the next tick. Someone is waiting on this request,
// and five seconds of ticker is five seconds they spend staring at nothing.
@@ -669,6 +699,17 @@ func (r *Reconciler) reapFailed(ctx context.Context) {
// replica is about to disappear, and the backoff has to survive it.
r.attempts[rep.Model]++
r.nextAttempt[rep.Model] = r.now().Add(r.backoff(r.attempts[rep.Model]))
if n := r.attempts[rep.Model]; n >= r.cfg.ParkAfter {
if _, already := r.parked[rep.Model]; !already {
m, _ := r.cat.Load().Resolve(rep.Model)
r.parked[rep.Model] = parkRecord{
Since: r.now(), Reason: rep.Err, Failures: n, fp: modelFingerprint(m),
}
r.log.Error("model parked: it failed every time it started, and is not retried "+
"until its definition changes or `llmgw ctl retry` is run",
"model", rep.Model, "failures", n, "last_error", rep.Err)
}
}
}
}
r.mu.Unlock()
@@ -746,10 +787,22 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
}
}
blockedUntil := r.nextAttempt[m.Name]
park, isParked := r.parked[m.Name]
if isParked && park.fp != modelFingerprint(m) {
// Somebody changed the model — a new source, an edited profile — so
// the reason it kept failing may be gone. Worth one more streak.
delete(r.parked, m.Name)
r.attempts[m.Name] = 0
delete(r.nextAttempt, m.Name)
isParked, blockedUntil = false, time.Time{}
r.log.Info("model unparked: its definition changed", "model", m.Name)
}
r.mu.Unlock()
desired := r.desiredFor(m)
switch {
case len(liveIDs) < desired && isParked:
// Not placed. Its failure is on every status the operator reads.
case len(liveIDs) < desired:
if r.now().Before(blockedUntil) {
// Still in backoff. Rebuilding immediately would hot-loop on a
@@ -925,7 +978,7 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
if m.Scaling.OnDemand() && r.makeRoomFor(ctx, m) {
return
}
r.log.Warn("no placement for model", "model", m.Name, "reasons", res.Explain())
r.notePlacementFailure(m.Name, res.Explain())
return
}
if err := r.startReplica(ctx, m, res.Placement); err != nil {
@@ -1523,3 +1576,142 @@ func (r *Reconciler) engineKey(replicaID string) string {
}
return r.cfg.EngineKey(replicaID)
}
// parkRecord is why a model stopped being retried.
type parkRecord struct {
Since time.Time
Reason string
Failures int
// fp is the model's definition when it was parked; a different one
// unparks it.
fp string
}
type placementNote struct {
text string
at time.Time
}
// modelFingerprint identifies a model's definition, so a change to it can be
// told from a retry of the same thing.
func modelFingerprint(m catalog.Model) string {
b, _ := json.Marshal(m)
sum := sha256.Sum256(b)
return hex.EncodeToString(sum[:8])
}
// placementLogEvery is how often an unchanged placement failure is repeated.
const placementLogEvery = 10 * time.Minute
// notePlacementFailure logs why a model could not be placed, when the reason
// is new or has not been said for a while. Said on every pass it was 250
// identical lines in two days for one model, burying everything else.
func (r *Reconciler) notePlacementFailure(model, why string) {
r.mu.Lock()
last := r.placementLogged[model]
say := last.text != why || r.now().Sub(last.at) >= placementLogEvery
if say {
r.placementLogged[model] = placementNote{text: why, at: r.now()}
}
r.mu.Unlock()
if say {
r.log.Warn("no placement for model", "model", model, "reasons", why)
}
}
// Retry clears a parked model and its backoff, so it is placed on the next
// pass.
func (r *Reconciler) Retry(model string) bool {
r.mu.Lock()
_, was := r.parked[model]
delete(r.parked, model)
r.attempts[model] = 0
delete(r.nextAttempt, model)
r.mu.Unlock()
r.Notify()
return was
}
// ModelHealth is how one model stands, for status and metrics.
type ModelHealth struct {
Model string
// Desired is how many replicas it should have now; 0 for a stopped model
// or an on-demand one nobody is asking for.
Desired int
Ready int
// Failures counts replicas that failed in a row.
Failures int
// Parked is set when Failures reached ParkAfter; Reason is the last error.
Parked bool
ParkedSince time.Time
Reason string
// Unplaceable is set when it is short of replicas and the last attempt to
// place one found nowhere; Placement says why.
Unplaceable bool
Placement string
}
// ModelHealth reports every model in the catalog.
func (r *Reconciler) ModelHealth() []ModelHealth {
models := r.cat.Load().Models()
desired := map[string]int{}
for _, m := range models {
desired[m.Name] = r.desiredFor(m)
}
r.mu.Lock()
defer r.mu.Unlock()
ready, live := map[string]int{}, map[string]int{}
for _, rep := range r.replicas {
if rep.State == Ready {
ready[rep.Model]++
}
if rep.State.Counts() {
live[rep.Model]++
}
}
out := make([]ModelHealth, 0, len(models))
for _, m := range models {
h := ModelHealth{Model: m.Name, Desired: desired[m.Name], Ready: ready[m.Name],
Failures: r.attempts[m.Name]}
if p, ok := r.parked[m.Name]; ok {
h.Parked, h.ParkedSince, h.Reason = true, p.Since, p.Reason
}
if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil && live[m.Name] < h.Desired {
h.Unplaceable, h.Placement = true, res.Explain()
}
out = append(out, h)
}
return out
}
// Unservable says why a model cannot be served at all, so a request for it is
// told now rather than asked to retry something that will never come up.
func (r *Reconciler) Unservable(model string) (string, bool) {
m, ok := r.cat.Load().Resolve(model)
if !ok {
return "", false
}
r.mu.Lock()
defer r.mu.Unlock()
if p, ok := r.parked[m.Name]; ok {
return fmt.Sprintf("%s failed to start %d times in a row and is not being retried: %s",
m.Name, p.Failures, p.Reason), true
}
for _, rep := range r.replicas {
if rep.Model == m.Name && rep.State.Counts() {
return "", false // something is on its way
}
}
if res, ok := r.lastExplain[m.Name]; ok && res.Placement == nil {
return m.Name + " cannot be placed on any node: " + firstLine(res.Explain()), true
}
return "", false
}
func firstLine(s string) string {
s = strings.TrimSpace(s)
if i := strings.IndexByte(s, '\n'); i >= 0 {
return strings.TrimSpace(s[:i]) + " (and others; see llmgw ctl explain-placement)"
}
return s
}
@@ -1,6 +1,7 @@
package reconciler
import (
"bytes"
"context"
"fmt"
"log/slog"
@@ -1836,3 +1837,98 @@ func TestReplicasOfAModelAreRestartedOneAtATime(t *testing.T) {
t.Error("an unknown replica was accepted")
}
}
// failRepeatedly starts and fails the model's replica n times over.
func failRepeatedly(t *testing.T, h *harness, n int, why string) {
t.Helper()
for i := 0; i < n; i++ {
h.disp.reset()
h.r.Reconcile(t.Context())
started := h.disp.startedShards()
if len(started) == 0 {
return // parked, or still in backoff
}
s := started[0]
h.r.OnShardStatus(s.ReplicaID, s.ShardID, supervisor.Failed, why)
h.r.Reconcile(t.Context())
h.advance(DefaultConfig().MaxBackoff + time.Second)
}
}
func TestAModelThatAlwaysFailsIsParkedNotRetriedForever(t *testing.T) {
// kev-0.5b: an architecture the node's llama.cpp did not know, started
// 155 times in eight hours, the same error every time.
h := newHarness(t, testModel(1), gpuNode("1", 8))
failRepeatedly(t, h, DefaultConfig().ParkAfter, "unknown model architecture: 'ggmlc'")
h.disp.reset()
h.advance(DefaultConfig().MaxBackoff + time.Second)
h.r.Reconcile(t.Context())
if n := len(h.disp.startedShards()); n != 0 {
t.Fatalf("a parked model was started again (%d shards)", n)
}
var mh ModelHealth
for _, x := range h.r.ModelHealth() {
if x.Model == "llama" {
mh = x
}
}
if !mh.Parked || !strings.Contains(mh.Reason, "ggmlc") || mh.Failures < DefaultConfig().ParkAfter {
t.Errorf("health %+v", mh)
}
if why, no := h.r.Unservable("llama"); !no || !strings.Contains(why, "ggmlc") {
t.Errorf("a request would not be told why: %q %v", why, no)
}
// Retried by the operator once fixed.
if !h.r.Retry("llama") {
t.Error("Retry did not report the model as parked")
}
h.r.Reconcile(t.Context())
if len(h.disp.startedShards()) == 0 {
t.Error("a retried model was not placed")
}
}
func TestChangingAParkedModelsDefinitionTriesItAgain(t *testing.T) {
h := newHarness(t, testModel(1), gpuNode("1", 8))
failRepeatedly(t, h, DefaultConfig().ParkAfter, "boom")
fixed := testModel(1)
fixed.Source = "/models/llama-fixed"
cat, err := catalog.New([]catalog.Model{fixed})
if err != nil {
t.Fatal(err)
}
h.r.cat.Store(cat)
h.disp.reset()
h.r.Reconcile(t.Context())
if len(h.disp.startedShards()) == 0 {
t.Error("a model whose definition changed stayed parked")
}
}
func TestAPlacementFailureIsLoggedWhenItChangesNotEveryPass(t *testing.T) {
var buf bytes.Buffer
h := newHarness(t, testModel(1), gpuNode("1", 1)) // tp4 needs 4 cards
h.r.log = slog.New(slog.NewTextHandler(&buf, nil))
for i := 0; i < 20; i++ {
h.r.Reconcile(t.Context())
h.advance(5 * time.Second)
}
if n := strings.Count(buf.String(), "no placement for model"); n != 1 {
t.Errorf("logged %d times over 20 passes", n)
}
h.advance(placementLogEvery)
h.r.Reconcile(t.Context())
if n := strings.Count(buf.String(), "no placement for model"); n != 2 {
t.Errorf("not repeated after %s: %d", placementLogEvery, n)
}
var mh ModelHealth
for _, x := range h.r.ModelHealth() {
mh = x
}
if !mh.Unplaceable || mh.Desired != 1 {
t.Errorf("health %+v", mh)
}
}
+23
View File
@@ -17,6 +17,22 @@ type Activator interface {
Want(model string) bool
}
// unservable is an activator that can also say a model will not come up at
// all — it failed every time it started, or fits on no node — so a request is
// told so at once rather than asked to retry something that never arrives.
type unservable interface {
Unservable(model string) (string, bool)
}
// whyUnservable is the activator's reason a model cannot be served, if any.
func (r *Router) whyUnservable(model string) (string, bool) {
u, ok := r.activator.(unservable)
if !ok {
return "", false
}
return u.Unservable(model)
}
// SetActivator supplies the control plane that can load models on demand.
func (r *Router) SetActivator(a Activator) { r.activator = a }
@@ -32,6 +48,8 @@ func (r *Router) SetActivator(a Activator) { r.activator = a }
// a slow request. A fifteen-minute wait is a job pretending to be one, and the
// client should be told to come back rather than left holding a socket.
func (r *Router) activate(ctx context.Context, model string) bool {
// Asked first, always: asking is what makes the control plane try again,
// and a model that fitted nowhere an hour ago may fit now.
if r.activator == nil || !r.activator.Want(model) {
return false
}
@@ -56,6 +74,11 @@ func (r *Router) activate(ctx context.Context, model string) bool {
if len(r.reg.Snapshots(model)) > 0 {
return true
}
// Asking set a placement going; if it found nowhere, there is
// nothing to wait for.
if _, no := r.whyUnservable(model); no {
return false
}
}
}
}
+15 -4
View File
@@ -42,14 +42,20 @@ func newController(t *testing.T, cfg Config, sat SaturationFunc) *Controller {
func TestAdmitsImmediatelyWhenIdle(t *testing.T) {
s := &fixedSat{}
c := newController(t, testConfig(), s.fn)
// A queue that is looked at every ten seconds, so a request that went
// through it rather than the fast path would take that long. A bound of
// 50ms against a 1ms tick measured the machine the test ran on, and a
// loaded CI runner under the race detector was slower than that.
cfg := testConfig()
cfg.Interval = 10 * time.Second
c := newController(t, cfg, s.fn)
start := time.Now()
lease, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
if err != nil {
t.Fatal(err)
}
if d := time.Since(start); d > 50*time.Millisecond {
if d := time.Since(start); d > 2*time.Second {
t.Errorf("idle admission took %v; the fast path is not being used", d)
}
lease.Release()
@@ -131,8 +137,13 @@ func TestQueuedRequestAdmittedWhenSaturationDrops(t *testing.T) {
done <- err
}()
// Give it time to actually queue, then open the gate.
time.Sleep(50 * time.Millisecond)
// Wait for it to actually queue, then open the gate. Waited for, not
// slept through: on a loaded runner 50ms was sometimes not long enough
// for the goroutine to have been scheduled at all.
deadline := time.Now().Add(2 * time.Second)
for c.Depths()[3].Depth != 1 && time.Now().Before(deadline) {
time.Sleep(5 * time.Millisecond)
}
if d := c.Depths(); d[3].Depth != 1 {
t.Fatalf("expected 1 queued bronze request, got %+v", d)
}
+5
View File
@@ -629,6 +629,11 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
}
}
if len(candidates) == 0 {
if why, no := r.whyUnservable(model); no && r.activator != nil {
// Not "loading; retry": it will not load, and a client told to
// retry keeps retrying something that never arrives.
return dispatchResult{err: fmt.Errorf("%w: %s", errNoCapacity, why)}
}
if r.activator != nil && r.activator.Want(model) {
// Asked for, still loading. Saying "no endpoint serves this" would
// be a lie, and a client told to retry will succeed where one told
+36
View File
@@ -619,3 +619,39 @@ func TestTheServedModelHeaderArrivesBeforeTheBody(t *testing.T) {
t.Errorf("served-model header = %q, want backup", got)
}
}
type unservableActivator struct {
fakeActivator
why map[string]string
}
func (u *unservableActivator) Unservable(model string) (string, bool) {
w, ok := u.why[model]
return w, ok
}
func TestAModelThatCannotComeUpIsSaidSoAtOnce(t *testing.T) {
// A model that fits on no node, or failed every time it started, was
// answered "loading; retry shortly" — and a client that retried did so
// forever.
e := newFakeEngine(t, "m")
cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}})
cfg.Router.ActivationWait = 10 * time.Second
rt := newTestRouter(t, cfg)
rt.SetActivator(&unservableActivator{
fakeActivator: fakeActivator{onDemand: map[string]bool{"phi-4": true}},
why: map[string]string{"phi-4": "phi-4 cannot be placed on any node: needs 15 GB"},
})
start := time.Now()
rec := post(t, rt.Handler(), map[string]any{"model": "phi-4", "messages": []any{}}, nil)
if took := time.Since(start); took > 2*time.Second {
t.Errorf("the request waited %s for a model that will not load", took)
}
if rec.Code != http.StatusServiceUnavailable || rec.Header().Get("Retry-After") != "" {
t.Errorf("status %d, Retry-After %q", rec.Code, rec.Header().Get("Retry-After"))
}
if !strings.Contains(rec.Body.String(), "needs 15 GB") {
t.Errorf("the reason was not given: %s", rec.Body.String())
}
}
+4 -1
View File
@@ -175,7 +175,10 @@ func TestABrokenContractIsReportedOnceAndLoudly(t *testing.T) {
in := endpoint.Info{ID: "e1", URL: srv.URL, Engine: "vllm", Models: []string{"m"}}
reg.Replace([]endpoint.Info{in})
p := NewPoller(reg, Config{Interval: 20 * time.Millisecond, Timeout: 10 * time.Millisecond},
// A generous timeout: a scrape that timed out would be reported as
// unreachable, and the violation reported again once it answered — two
// warnings where the test counts one, on a slow runner only.
p := NewPoller(reg, Config{Interval: 20 * time.Millisecond, Timeout: 2 * time.Second},
slog.New(cap))
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
+6
View File
@@ -59,3 +59,9 @@ func resolveAddress(address string) (string, error) {
}
return out, nil
}
// ResolveAddress expands an @file store address, as Open does.
func ResolveAddress(address string) (string, error) { return resolveAddress(address) }
// IsPostgres reports whether a resolved address names a PostgreSQL server.
func IsPostgres(address string) bool { return isPostgres(address) }
+11
View File
@@ -817,3 +817,14 @@ func (s *DB) UsageByModel(ctx context.Context, since time.Time) ([]ModelUsage, e
}
return out, rows.Err()
}
// SnapshotTo writes a consistent copy of a SQLite store to path, while it is
// in use. VACUUM INTO takes a read transaction, so the copy is of one moment
// even with writes going on, which copying the file is not.
func (s *DB) SnapshotTo(ctx context.Context, path string) error {
if s.dialect != SQLiteDialect {
return fmt.Errorf("a snapshot is for SQLite; back up PostgreSQL with pg_dump")
}
_, err := s.raw.ExecContext(ctx, `VACUUM INTO ?`, path)
return err
}
+58
View File
@@ -0,0 +1,58 @@
package tlsconf
import (
"crypto/x509"
"encoding/pem"
"os"
"github.com/prometheus/client_golang/prometheus"
)
// ExpiryCollector publishes when each certificate in a set of files expires.
//
// Certificates here last a year and nothing renews them; the first sign of one
// running out was going to be every agent refusing to connect on the same day.
// Read at every scrape rather than once, so a certificate replaced on disk is
// seen without a restart. A file that cannot be read publishes nothing, which
// is its own thing to notice.
func ExpiryCollector(files map[string]string) prometheus.Collector {
return &expiry{files: files, desc: prometheus.NewDesc("llmgw_certificate_expiry_timestamp_seconds",
"When a certificate this control plane uses expires, as a Unix time.",
[]string{"cert", "subject"}, nil)}
}
type expiry struct {
files map[string]string
desc *prometheus.Desc
}
func (e *expiry) Describe(ch chan<- *prometheus.Desc) { ch <- e.desc }
func (e *expiry) Collect(ch chan<- prometheus.Metric) {
for name, path := range e.files {
if path == "" {
continue
}
b, err := os.ReadFile(path) //nolint:gosec // configured certificate paths
if err != nil {
continue
}
// Every certificate in the file: a CA bundle may hold several.
for {
var blk *pem.Block
blk, b = pem.Decode(b)
if blk == nil {
break
}
if blk.Type != "CERTIFICATE" {
continue
}
c, err := x509.ParseCertificate(blk.Bytes)
if err != nil {
continue
}
ch <- prometheus.MustNewConstMetric(e.desc, prometheus.GaugeValue,
float64(c.NotAfter.Unix()), name, c.Subject.CommonName)
}
}
}