Compare commits

..
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 91fe24a105 Close what a second security review found still open
Release / build (push) Successful in 1m8s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m50s
A review of v0.10.4-rc.4, and a look at the live network, found:

- Alertmanager listening on every interface: from any node its API took
  silences, with no login. Bound to loopback on the monitoring host, and the
  monitoring README now installs it that way; Prometheus beside it is its only
  client, and one Alertmanager needs no cluster port.
- Login and signup unbounded in memory. Each attempt runs argon2 with 64 MiB,
  and nothing limited how many ran at once; the per-address throttle kept one
  entry per address ever tried. Hashing now takes one of a few slots and
  waits for one otherwise. The throttle expires entries whose window has
  passed when it fills, and if it is still full it refuses new addresses:
  evicting live ones instead would let a flood of made-up addresses reset the
  throttle on the address actually being guessed.
- Metric labels taken from the request. A refused request counted the model
  name it asked for, so any key holder could mint a series per request.
  Requests are counted under the catalog's name, or "unknown".
- Two in code written this week. A SQLite backup made its snapshot with the
  default mode in the listable backup directory, a full copy of the store
  readable while the backup ran; it is now made in a private directory, and
  the unit sets UMask=0077. And archive links were checked by name only, so
  "a -> ." then "a/a/b -> ../.." passed and a file written through them landed
  above the install directory, as root. Archives now unpack through os.Root,
  and a link may only point at a file beside it — which every llama.cpp
  b10991 archive, CPU and CUDA, was checked to follow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 02:25:07 +02:00
Ilker ManapandClaude Opus 5.5 9bf8256577 Keep backups where the control plane can see them
Release / build (push) Successful in 1m1s
Release / notes (push) Successful in 2s
CI / test (push) Successful in 3m35s
The first backup on the live control plane was written, and the control
plane reported none: /var/backups/llmgw already existed, made by an earlier
move to PostgreSQL for its own dumps, owned by postgres and 0700. MkdirAll
leaves an existing directory as it is, so the archive landed where the
llmgw user could not list it, and llmgw_backup_last_success_timestamp_seconds
said 0.

Opening that directory up would have exposed the dumps already in it, so
backups get their own, /var/backups/llmgw-server. And `llmgw server backup`
now warns when the directory it wrote to is one the control plane cannot
list, rather than leaving that to be found from an alert.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:50:33 +02:00
16 changed files with 298 additions and 41 deletions

No files matched your search

+22
View File
@@ -77,6 +77,28 @@ more than 26 hours old or there has never been one. docs/install.md now says how
to restore, including why running engines survive it: the engine-key secret
comes back with the store.
### Security
- **Alertmanager answered anyone on the network**, including requests to
silence alerts. The monitoring README now binds it to loopback, where
Prometheus beside it is its only client.
- **Login and signup could be made to use gigabytes.** Each attempt hashes with
64 MiB of argon2, with no bound on how many ran at once, and the per-address
throttle grew by one entry per address tried. Hashes now run a few at a time
and the rest wait; the throttle expires old entries, and when it is still
full it refuses new addresses rather than dropping live ones — which would
have let a flood of made-up addresses reset the throttle on a real one.
- **Any key holder could grow Prometheus without bound** by asking for made-up
model names, each of which became a metric label. Requests are counted under
the catalog's name for their model, or "unknown".
- **A SQLite backup left a full copy of the store readable** by local users
while it ran. The snapshot is now made in a private directory, and
llmgw-backup.service sets `UMask=0077`.
- **An engine archive could write outside its directory through chained
links.** Archives are unpacked through `os.Root`, and a link may only point
at a file beside it. Every llama.cpp b10991 archive, CPU and CUDA, already
follows that rule.
### Fixes
- Tests that slept a fixed time and then asserted now wait for their
+9 -2
View File
@@ -46,6 +46,12 @@ apt-get install prometheus prometheus-alertmanager
install -m 0644 prometheus.yml /etc/prometheus/prometheus.yml
install -m 0644 llmgw-rules.yml /etc/prometheus/rules/llmgw.yml
install -m 0644 alertmanager.yml /etc/prometheus/alertmanager.yml
# Alertmanager answers on loopback only. Its API takes silences from anyone who
# can reach it, with no login, and silencing BackupTooOld or ModelParked is how
# something that should be noticed is not. Prometheus beside it is its only
# client; one Alertmanager needs no cluster port.
echo 'ARGS="--web.listen-address=127.0.0.1:9093 --cluster.listen-address="' \
> /etc/default/prometheus-alertmanager
promtool check config /etc/prometheus/prometheus.yml
systemctl restart prometheus prometheus-alertmanager
@@ -75,7 +81,8 @@ Alertmanager groups and holds alerts and has no receiver.
Where an alert should go — an address, a Slack workspace, a webhook — is a
decision with credentials attached, and a monitoring system that quietly starts
mailing people is worse than one that is visibly not wired up. Alerts are
visible at `:9093` and in Prometheus; add a receiver to `alertmanager.yml` to
visible in Prometheus, and at `:9093` on the monitoring host itself (`ssh -L
9093:127.0.0.1:9093 <host>` to open its page); add a receiver to `alertmanager.yml` to
make them travel. `critical` already routes separately so it can be pointed
somewhere else without touching the rest.
@@ -86,7 +93,7 @@ An alerting system nobody has seen fire is an assumption.
```
llmgw ctl cordon <node> # work already there keeps running
# ~2 minutes later
curl -s http://<host>:9093/api/v2/alerts | jq '.[].labels'
ssh <host> curl -s http://127.0.0.1:9093/api/v2/alerts | jq '.[].labels'
llmgw ctl uncordon <node> # the alert resolves on its own
```
+4 -1
View File
@@ -1,9 +1,12 @@
[Unit]
Description=llmgw backup: the store and /etc/llmgw, to /var/backups/llmgw
Description=llmgw backup: the store and /etc/llmgw, to /var/backups/llmgw-server
After=llmgw-server.service postgresql.service
[Service]
Type=oneshot
# Nothing it writes is for anyone else: the archive holds the CA's key and a
# copy of the store.
UMask=0077
# Root, because the archive includes /etc/llmgw/secrets and the certificate
# authority's key, which the llmgw user deliberately cannot read all of.
ExecStart=/usr/local/bin/llmgw server backup --config /etc/llmgw/server.yaml --keep 14
+3 -3
View File
@@ -973,7 +973,7 @@ cannot be rebuilt without:
certificate authority's key (`pki/`). Nothing in llmgw can reconstruct that
key; losing it means reissuing every node's identity.
Archives go to `/var/backups/llmgw`, readable by root only because of the CA
Archives go to `/var/backups/llmgw-server`, readable by root only because of the CA
key, and the newest 14 are kept. Run it nightly with the timer:
```
@@ -981,7 +981,7 @@ sudo install -m 0644 deploy/systemd/llmgw-backup.service deploy/systemd/llmgw-ba
sudo systemctl daemon-reload
sudo systemctl enable --now llmgw-backup.timer
sudo systemctl start llmgw-backup.service # one now, to see it work
ls -l /var/backups/llmgw
ls -l /var/backups/llmgw-server
```
The control plane publishes when the newest archive was written as
@@ -1006,7 +1006,7 @@ them:
```
sudo systemctl stop llmgw-server
mkdir /tmp/restore && sudo tar -xzf /var/backups/llmgw/llmgw-backup-<host>-<time>.tar.gz -C /tmp/restore
mkdir /tmp/restore && sudo tar -xzf /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz -C /tmp/restore
cat /tmp/restore/MANIFEST # version and time it was taken
# /etc/llmgw: configuration, secrets, certificates and the CA
+1 -1
View File
@@ -1272,7 +1272,7 @@ one leaves that engine to be installed by hand.
BackupDir is where `llmgw server backup` writes, and where the newest
archive's age is read for llmgw_backup_last_success_timestamp_seconds.
Empty means /var/backups/llmgw.
Empty means /var/backups/llmgw-server.
### `router`
+16 -2
View File
@@ -15,6 +15,7 @@ import (
"encoding/base64"
"errors"
"fmt"
"runtime"
"strings"
"golang.org/x/crypto/argon2"
@@ -56,7 +57,7 @@ func HashPassword(password string) (string, error) {
if _, err := rand.Read(salt); err != nil {
return "", err
}
sum := argon2.IDKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
sum := idKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
b64 := base64.RawStdEncoding.EncodeToString
return fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s",
@@ -93,7 +94,7 @@ func VerifyPassword(password, encoded string) (bool, error) {
return false, ErrBadHash
}
got := argon2.IDKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
got := idKey([]byte(password), salt, time, memory, threads, uint32(len(want)))
return subtle.ConstantTimeCompare(got, want) == 1, nil
}
@@ -110,3 +111,16 @@ func init() {
}
dummyHash = h
}
// argonSlots bounds how many hashes run at once. Each takes 64 MiB, and the
// login and signup forms are on the client-facing port: without a bound, a
// few hundred parallel attempts at random addresses asked for gigabytes at
// once, and the per-address throttle does not see them — each address is new.
// Beyond the bound, attempts wait their turn instead.
var argonSlots = make(chan struct{}, max(2, runtime.NumCPU()/2))
func idKey(password, salt []byte, time, memory uint32, threads uint8, keyLen uint32) []byte {
argonSlots <- struct{}{}
defer func() { <-argonSlots }()
return argon2.IDKey(password, salt, time, memory, threads, keyLen)
}
+20
View File
@@ -230,6 +230,22 @@ func (s *Service) allowAttempt(email string) bool {
defer s.mu.Unlock()
w, ok := s.attempts[email]
if !ok && len(s.attempts) >= maxTrackedAddresses {
// Full of addresses nobody has signed in as for a while: forget the
// ones whose window has passed. Only expiring, never evicting early —
// dropping a live window is how an attacker spraying random addresses
// would buy unlimited guesses at the one they care about.
for k, v := range s.attempts {
if now.Sub(v.since) > s.cfg.AttemptWindow {
delete(s.attempts, k)
}
}
if len(s.attempts) >= maxTrackedAddresses {
// Still full, so somebody is spraying addresses faster than they
// expire. New ones wait; those already tracked are unaffected.
return false
}
}
if !ok || now.Sub(w.since) > s.cfg.AttemptWindow {
s.attempts[email] = &attemptWindow{count: 1, since: now}
return true
@@ -238,6 +254,10 @@ func (s *Service) allowAttempt(email string) bool {
return w.count <= s.cfg.MaxAttempts
}
// maxTrackedAddresses bounds the throttle's memory. Unbounded, it grew by one
// entry per address tried and shrank only on a successful sign-in.
const maxTrackedAddresses = 50000
func (s *Service) clearAttempts(email string) {
s.mu.Lock()
delete(s.attempts, email)
+33
View File
@@ -0,0 +1,33 @@
package account
import (
"fmt"
"testing"
"time"
)
func TestSprayingAddressesCannotGrowTheThrottleOrEscapeIt(t *testing.T) {
now := time.Unix(0, 0)
s := &Service{cfg: DefaultConfig(), attempts: map[string]*attemptWindow{}, now: func() time.Time { return now }}
// The target is being guessed at.
for i := 0; i < s.cfg.MaxAttempts; i++ {
s.allowAttempt("victim@example.com")
}
// Then the table is flooded with made-up addresses.
for i := 0; i < maxTrackedAddresses+100; i++ {
s.allowAttempt(fmt.Sprintf("x%d@example.com", i))
}
if n := len(s.attempts); n > maxTrackedAddresses {
t.Errorf("the table grew to %d", n)
}
if s.allowAttempt("victim@example.com") {
t.Error("flooding the table reset the target's throttle")
}
// Once the window passes, everything expires and sign-ins work again.
now = now.Add(s.cfg.AttemptWindow + time.Second)
if !s.allowAttempt("new@example.com") {
t.Error("a new address was refused after the flood expired")
}
}
+26 -16
View File
@@ -523,6 +523,11 @@ func (in *Installer) download(ctx context.Context, repo, tag, asset, dst, want s
// untarGz unpacks an archive under dir, refusing anything that would land
// outside it. A release archive is somebody else's file, and a path of
// "../../etc/cron.d/x" in it would otherwise be written as root.
//
// Every write goes through an os.Root, which refuses to leave dir however the
// path gets there. Checking names alone was not enough: "a -> ." followed by
// "a/a/b -> ../.." reads as inside and resolves two levels above it, and a
// file written through that lands wherever the links point.
func untarGz(archive, dir string) error {
f, err := os.Open(archive) //nolint:gosec // inside the install root
if err != nil {
@@ -533,6 +538,11 @@ func untarGz(archive, dir string) error {
if err != nil {
return err
}
root, err := os.OpenRoot(dir)
if err != nil {
return err
}
defer root.Close()
tr := tar.NewReader(gz)
for {
h, err := tr.Next()
@@ -542,22 +552,22 @@ func untarGz(archive, dir string) error {
if err != nil {
return err
}
target, err := within(dir, h.Name)
if err != nil {
name := filepath.Clean(h.Name)
if _, err := within(dir, name); err != nil {
return err
}
switch h.Typeflag {
case tar.TypeDir:
if err := os.MkdirAll(target, 0o755); err != nil {
if err := root.MkdirAll(name, 0o755); err != nil {
return err
}
case tar.TypeReg:
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
if err := root.MkdirAll(filepath.Dir(name), 0o755); err != nil {
return err
}
out, err := os.OpenFile(target, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, fs.FileMode(h.Mode)&0o755) //nolint:gosec // checked above
out, err := root.OpenFile(name, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, fs.FileMode(h.Mode)&0o755)
if err != nil {
return err
return fmt.Errorf("%s: %w", h.Name, err)
}
if _, err := io.Copy(out, tr); err != nil { //nolint:gosec // a release archive, size bounded by its checksum
out.Close()
@@ -567,19 +577,19 @@ func untarGz(archive, dir string) error {
return err
}
case tar.TypeSymlink:
// Libraries are versioned through symlinks, and those are
// relative. One pointing anywhere else is refused.
if filepath.IsAbs(h.Linkname) {
return fmt.Errorf("%s links outside the archive", h.Name)
// Libraries are versioned through symlinks to a file beside them —
// libllama.so -> libllama.so.0 — and that is all a release needs.
// A target with a separator in it could point anywhere once other
// links are followed, and a binary found through one would be run
// as root, so it is refused rather than reasoned about.
if h.Linkname == "" || strings.ContainsAny(h.Linkname, `/\`) || h.Linkname == ".." || h.Linkname == "." {
return fmt.Errorf("%s links to %q; only a file beside it is allowed", h.Name, h.Linkname)
}
if _, err := within(dir, filepath.Join(filepath.Dir(h.Name), h.Linkname)); err != nil {
return fmt.Errorf("%s links outside the archive", h.Name)
}
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
if err := root.MkdirAll(filepath.Dir(name), 0o755); err != nil {
return err
}
_ = os.Remove(target)
if err := os.Symlink(h.Linkname, target); err != nil {
_ = root.Remove(name)
if err := root.Symlink(h.Linkname, name); err != nil {
return err
}
default:
+46
View File
@@ -342,3 +342,49 @@ func TestAVersionThatIsAPathIsRefused(t *testing.T) {
}
}
}
func TestChainedLinksCannotCarryAFileOutOfTheInstall(t *testing.T) {
// Each link reads as inside the directory; together they point above it.
var buf bytes.Buffer
gz := gzip.NewWriter(&buf)
tw := tar.NewWriter(gz)
_ = tw.WriteHeader(&tar.Header{Name: "a", Linkname: ".", Typeflag: tar.TypeSymlink})
_ = tw.WriteHeader(&tar.Header{Name: "a/a/b", Linkname: "../..", Typeflag: tar.TypeSymlink})
_ = tw.WriteHeader(&tar.Header{Name: "a/a/b/escaped", Mode: 0o644, Size: 1, Typeflag: tar.TypeReg})
_, _ = tw.Write([]byte("x"))
_ = tw.Close()
_ = gz.Close()
parent := t.TempDir()
dir := filepath.Join(parent, "install")
_ = os.MkdirAll(dir, 0o755)
archive := filepath.Join(parent, "evil.tar.gz")
_ = os.WriteFile(archive, buf.Bytes(), 0o644)
if err := untarGz(archive, dir); err == nil {
t.Error("an archive of chained links was unpacked")
}
if _, err := os.Stat(filepath.Join(parent, "escaped")); err == nil {
t.Fatal("a file was written outside the install directory")
}
}
func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
var buf bytes.Buffer
gz := gzip.NewWriter(&buf)
tw := tar.NewWriter(gz)
_ = tw.WriteHeader(&tar.Header{Name: "lib/libllama.so.0", Mode: 0o755, Size: 1, Typeflag: tar.TypeReg})
_, _ = tw.Write([]byte("x"))
_ = tw.WriteHeader(&tar.Header{Name: "lib/libllama.so", Linkname: "libllama.so.0", Typeflag: tar.TypeSymlink})
_ = tw.Close()
_ = gz.Close()
dir := t.TempDir()
archive := filepath.Join(t.TempDir(), "ok.tar.gz")
_ = os.WriteFile(archive, buf.Bytes(), 0o644)
if err := untarGz(archive, dir); err != nil {
t.Fatal(err)
}
if target, err := os.Readlink(filepath.Join(dir, "lib", "libllama.so")); err != nil || target != "libllama.so.0" {
t.Errorf("link %q %v", target, err)
}
}
+30 -7
View File
@@ -25,7 +25,11 @@ import (
// DefaultBackupDir is where backups are written and where the control plane
// looks for the newest one.
const DefaultBackupDir = "/var/backups/llmgw"
//
// Its own directory rather than /var/backups/llmgw, which a PostgreSQL move
// had already made for its own dumps, owned by postgres and closed to
// everyone else — so the control plane could not see a backup landing in it.
const DefaultBackupDir = "/var/backups/llmgw-server"
func newBackupCmd() *cobra.Command {
var (
@@ -56,6 +60,10 @@ func newBackupCmd() *cobra.Command {
if err != nil {
return err
}
if why := unlistable(dir); why != "" {
fmt.Fprintf(cmd.ErrOrStderr(), "warning: %s; the control plane will report no backup "+
"and BackupTooOld will fire\n", why)
}
removed, err := pruneBackups(dir, keep)
if err != nil {
return err
@@ -133,13 +141,17 @@ func addStore(ctx context.Context, tw *tar.Writer, addr, scratch string) error {
if err != nil {
return err
}
tmp, err := os.CreateTemp(scratch, ".store-*")
// Inside a directory only its owner can enter. The archive directory is
// listable on purpose, and VACUUM INTO creates its file with the default
// mode: made there directly, a full copy of the store — password hashes,
// the engine-key secret — sat readable by any local user for as long as
// the backup took.
tmpDir, err := os.MkdirTemp(scratch, ".store-")
if err != nil {
return err
}
tmpPath := tmp.Name()
_ = tmp.Close()
defer os.Remove(tmpPath)
defer os.RemoveAll(tmpDir)
tmpPath := filepath.Join(tmpDir, "store")
name := "store.sqlite"
if store.IsPostgres(addr) {
@@ -148,8 +160,6 @@ func addStore(ctx context.Context, tw *tar.Writer, addr, scratch string) error {
return err
}
} else {
// VACUUM INTO refuses a file that exists.
_ = os.Remove(tmpPath)
db, err := store.Open(ctx, addr)
if err != nil {
return err
@@ -287,3 +297,16 @@ func NewestBackup(dir string) time.Time {
}
return newest
}
// unlistable says why the control plane, which does not run as root, could
// not see the archives in dir, or "" when it can.
func unlistable(dir string) string {
info, err := os.Stat(dir)
if err != nil {
return err.Error()
}
if info.Mode().Perm()&0o005 != 0o005 {
return fmt.Sprintf("%s is %v, so only its owner can list it", dir, info.Mode().Perm())
}
return ""
}
+35
View File
@@ -6,6 +6,7 @@ import (
"io"
"os"
"path/filepath"
"strings"
"testing"
"time"
@@ -104,3 +105,37 @@ func TestOldBackupsArePrunedAndTheNewestIsFound(t *testing.T) {
t.Error("a missing directory reported a backup")
}
}
func TestABackupDirectoryTheControlPlaneCannotListIsSaid(t *testing.T) {
dir := t.TempDir()
_ = os.Chmod(dir, 0o700)
if unlistable(dir) == "" {
t.Error("a directory only its owner can list was taken as visible")
}
_ = os.Chmod(dir, 0o755)
if why := unlistable(dir); why != "" {
t.Errorf("a listable directory: %s", why)
}
}
func TestNoScratchCopyOfTheStoreIsLeftBehind(t *testing.T) {
// The snapshot is made before it is archived. Made straight into the
// listable archive directory, it was readable by anyone for that time.
tmp := t.TempDir()
dbPath := filepath.Join(tmp, "store.db")
db, err := store.Open(t.Context(), dbPath)
if err != nil {
t.Fatal(err)
}
_ = db.Close()
out := filepath.Join(tmp, "backups")
if _, err := writeBackup(t.Context(), dbPath, t.TempDir(), out); err != nil {
t.Fatal(err)
}
entries, _ := os.ReadDir(out)
for _, e := range entries {
if strings.HasPrefix(e.Name(), ".store") {
t.Errorf("a scratch copy of the store was left behind: %s", e.Name())
}
}
}
+1 -1
View File
@@ -83,7 +83,7 @@ type Server struct {
// BackupDir is where `llmgw server backup` writes, and where the newest
// archive's age is read for llmgw_backup_last_success_timestamp_seconds.
// Empty means /var/backups/llmgw.
// Empty means /var/backups/llmgw-server.
BackupDir string `koanf:"backup_dir"`
Router Router `koanf:"router"`
+8 -8
View File
@@ -51,7 +51,7 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded):
// The client gave up while queued; nobody is listening.
r.m.observeAdmissionRejected(req.Band, "client_gone")
r.m.observeRequest(req.Model, "", 499, time.Since(start), nil, 0)
r.m.observeRequest(r.metricModel(req.Model), "", 499, time.Since(start), nil, 0)
return
case errors.Is(err, admission.ErrQueueTimeout):
status = http.StatusGatewayTimeout
@@ -67,7 +67,7 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
}
r.m.observeAdmissionRejected(req.Band, reason)
writeError(w, status, "server_error", err.Error())
r.m.observeRequest(req.Model, "", status, time.Since(start), nil, 0)
r.m.observeRequest(r.metricModel(req.Model), "", status, time.Since(start), nil, 0)
}
// prefixBytes is how much of the prompt is kept for prefix-aware routing.
@@ -289,7 +289,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
if _, entry := r.resolve(parsed.Model); entry.ContextLength > 0 {
if why := checkContextFits(entry, parsed.PromptTokens, parsed.MaxTokens); why != "" {
writeError(w, http.StatusBadRequest, "invalid_request_error", why)
r.m.observeRequest(parsed.Model, "", http.StatusBadRequest, time.Since(start), nil, 0)
r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
return
}
}
@@ -297,7 +297,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
if p, ok := auth.FromContext(req.Context()); ok {
if !r.permits(p, parsed.Model) {
auth.WriteForbiddenModel(w, parsed.Model)
r.m.observeRequest(parsed.Model, "", http.StatusForbidden, time.Since(start), nil, 0)
r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusForbidden, time.Since(start), nil, 0)
return
}
// Limits are checked before admission, not after: a request that is
@@ -339,7 +339,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
}
defer lease.Release()
queued := time.Since(start)
r.m.observeAdmission(lease.Band(), parsed.Model, queued)
r.m.observeAdmission(lease.Band(), r.metricModel(parsed.Model), queued)
if queued > time.Millisecond {
w.Header().Set("x-llmgw-queue-ms", strconv.FormatInt(queued.Milliseconds(), 10))
}
@@ -349,7 +349,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
// model that will actually serve, not the one that was asked for.
if dropped, err := r.loadThread(req.Context(), parsed); err != nil {
writeError(w, http.StatusBadRequest, "invalid_request_error", err.Error())
r.m.observeRequest(parsed.Model, "", http.StatusBadRequest, time.Since(start), nil, 0)
r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
return
} else if dropped > 0 {
// Dropping turns is the right answer and must not be a silent one.
@@ -420,7 +420,7 @@ func (r *Router) finish(w http.ResponseWriter, req *parsedRequest, res dispatchR
writeError(w, status, "server_error", res.err.Error())
}
r.m.observeRequest(req.Model, res.pool, status, time.Since(start), res.outcome.Usage, res.outcome.TTFT)
r.m.observeRequest(r.metricModel(req.Model), res.pool, status, time.Since(start), res.outcome.Usage, res.outcome.TTFT)
r.account(req, res, status, time.Since(start))
}
@@ -467,7 +467,7 @@ func (r *Router) rejectLimit(w http.ResponseWriter, req *parsedRequest, err erro
// already know how to surface that code. What differs is Retry-After,
// which is absent when waiting does not help.
writeError(w, http.StatusTooManyRequests, "rate_limit_exceeded", le.Error())
r.m.observeRequest(req.Model, "", http.StatusTooManyRequests, time.Since(start), nil, 0)
r.m.observeRequest(r.metricModel(req.Model), "", http.StatusTooManyRequests, time.Since(start), nil, 0)
}
// parse reads the body and extracts the routing-relevant fields.
+14
View File
@@ -151,3 +151,17 @@ func (m *metrics) observeScores(d scheduler.Decision) {
}
m.considered.Observe(float64(d.Considered))
}
// metricModel is the model label a request is counted under: the catalog's
// name for it, or "unknown". Taken as sent, any key holder could mint a new
// series per request — a refused request counted its made-up model name —
// and grow Prometheus without bound.
func (r *Router) metricModel(name string) string {
if name == "" {
return ""
}
if m, ok := r.modelMap()[name]; ok {
return m.Name
}
return "unknown"
}
+30
View File
@@ -4,6 +4,7 @@ import (
"bytes"
"encoding/json"
"fmt"
"github.com/prometheus/client_golang/prometheus"
"io"
"log/slog"
"net/http"
@@ -655,3 +656,32 @@ func TestAModelThatCannotComeUpIsSaidSoAtOnce(t *testing.T) {
t.Errorf("the reason was not given: %s", rec.Body.String())
}
}
func TestAMadeUpModelNameDoesNotBecomeAMetricLabel(t *testing.T) {
// Counted as sent, every request for a nonsense model minted a new series.
e := newFakeEngine(t, "m")
cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}})
cfg.Models = []catalog.Model{{Name: "m", Aliases: []string{"mm"}}}
rt := newTestRouter(t, cfg)
if got := rt.metricModel("mm"); got != "m" {
t.Errorf("an alias is counted as %q, not the model it names", got)
}
for i := 0; i < 20; i++ {
post(t, rt.Handler(), map[string]any{"model": fmt.Sprintf("junk-%d", i), "messages": []any{}}, nil)
}
// Twenty names, one series: all of them "unknown".
reg := prometheus.NewRegistry()
reg.MustRegister(rt.m.requests)
mfs, err := reg.Gather()
if err != nil {
t.Fatal(err)
}
series := 0
for _, mf := range mfs {
series += len(mf.GetMetric())
}
if series != 1 {
t.Errorf("%d series after twenty made-up names", series)
}
}