Compare commits

...
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 94eec557ad Serve a model from an external OpenAI-compatible provider
CI / test (push) Successful in 3m47s
Release / build (push) Successful in 1m11s
Release / notes (push) Successful in 3s
A catalog model with external {url, model, api_key_file} and no profiles
is published as an endpoint by the control plane, the key read from its
file each time; the router rewrites the request's model to the provider's
name and sends its key. It is never scraped nor stale, and nothing is
placed for it. Named in fallbacks, it serves when the cluster has no room.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:29:01 +02:00
Ilker ManapandClaude Opus 5.5 d5f02c0a4b Spill a saturated pool's requests to the next pool
CI / test (push) Successful in 3m40s
The pool-priority filter kept the highest-priority pool however full it
was. It now takes the first pool, by priority, that is not saturated by
admission's own measure, and the preferred one when all are.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:23:32 +02:00
Ilker ManapandClaude Opus 5.5 3d3c96e422 Displace the newest lowest-band request when the gateway queue is full
Release / notes (push) Successful in 3s
Release / build (push) Successful in 1m9s
CI / test (push) Successful in 3m50s
A total bound across bands (admission.queue_capacity_total, default half of
all bands' capacity together); reached, a request is not refused while a
lower band has requests queued, and the newest of the lowest band gets a 503
with Retry-After instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:07:53 +02:00
Ilker ManapandClaude Opus 5.5 623cdb5ba3 Sign releases, and install only signed ones once a key is trusted
CI / test (push) Successful in 4m6s
ed25519 over checksums.txt: llmgw release keygen makes the pair, CI signs
with the RELEASE_SIGNING_KEY secret when it is set, and release.Install
refuses an unsigned or wrongly signed release when the build embeds a
trusted key (internal/release/trusted.pub, empty for now) or
LLMGW_RELEASE_KEY names one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:03:48 +02:00
Ilker ManapandClaude Opus 5.5 561667c9be Name only the replicas that would make room in headroom
CI / test (push) Successful in 3m58s
Candidates tried on the way to finding no room were listed as whose replicas
a placement would stop.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:01:40 +02:00
Ilker ManapandClaude Opus 5.5 cf0178bf05 Test that a restart borrows a lower class's room and returns it
CI / test (push) Successful in 3m41s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 22:47:12 +02:00
33 changed files with 1098 additions and 48 deletions

No files matched your search

+18
View File
@@ -93,6 +93,24 @@ jobs:
./dist/llmgw-${version}-linux-amd64 --version
cat dist/checksums.txt
# Signed when the repository has a signing key, so a machine that trusts
# it installs nothing the server alone could have put there. Without the
# secret nothing is signed and releases are checked on their checksums,
# as before; a build that trusts a key refuses those, so the secret has
# to be in place before trusted.pub is filled in.
- name: Sign the checksums
env:
RELEASE_SIGNING_KEY: ${{ secrets.RELEASE_SIGNING_KEY }}
run: |
set -euo pipefail
version="${GITHUB_REF_NAME:-$(git describe --tags --always)}"
if [ -z "${RELEASE_SIGNING_KEY:-}" ]; then
echo "no RELEASE_SIGNING_KEY secret; releases are not signed"
exit 0
fi
./dist/llmgw-${version}-linux-amd64 release sign dist/checksums.txt
echo "signed: dist/checksums.txt.sig"
- name: Attach to the release
env:
TOKEN: ${{ secrets.GITEA_TOKEN }}
+51
View File
@@ -42,6 +42,57 @@ What v0.10.5 made settable from the command line is now on the pages too:
critical model's card no longer refuses a replica that preemption would make
room for.
### Models served outside the cluster
A catalog model can be an OpenAI-compatible API elsewhere: `external: {url,
model, api_key_file}` and no profiles. Nothing is placed for it; the router
sends its requests to the provider, rewritten to the provider's name for the
model and with its key, read from the file on every publish so a rotated key is
picked up. Named in another model's `fallbacks`, it is used only when the
cluster has no capacity left. It is never scraped for metrics, and never reads
as stale. Prices, budgets, limits, usage and key allowlists apply as to any
model — and since its prompts leave the cluster, the allowlist is how to decide
who may send them (docs/install.md). `ctl models` and the cluster page mark it
external.
### Pool spillover
A model served from pools of different priority — H100 before A100, say — was
routed to the preferred pool however full it was. Now a pool whose replicas are
saturated, by the same measure admission uses (queue past its healthy depth, or
KV cache past its ceiling), passes requests to the next pool that is not. When
every pool is saturated the preferred one is kept: a pool that is just as full
gains nothing.
### Displacement: a higher band is not refused while a lower one waits
Each band's queue was bounded, and nothing bounded them together, so a full
gateway turned an interactive request away while batch requests sat queued
behind it. `admission.queue_capacity_total` now bounds everything queued —
by default half of every band's capacity together. Reached, a request is not
refused while a lower band has any queued: the newest request of the lowest
band is turned away instead, with a 503 and `Retry-After: 5`, so it backs off
and returns. The newest, because the oldest have waited longest and are
nearest their TTL. Counted as `displaced` in the admission rejection metric.
### Releases can be signed
`llmgw release keygen` makes an ed25519 key pair. With the private key in the
CI secret `RELEASE_SIGNING_KEY`, releases carry `checksums.txt.sig`; with the
public key in `internal/release/trusted.pub`, every binary built after refuses
a release not signed with it, wherever it installs one. Off until then, and
docs/install.md says in which order to turn it on.
### Restarts borrow a lower class's room
A restart places the replacement beside the replica it replaces, and where
there is no free room it now makes some the way any placement does: by
stopping a replica its class outranks, which gets its room back once the old
replica has gone. The in-place restart — the model unserved while its
replacement loads — is left for when nothing of a lower class is running. On a
cluster where every model is standard, that is every full one: mark the models
that can wait `opportunistic` (`llmgw ctl model class`, or the model's card).
## v0.10.5 — 2026-09-30
Published first as release candidates v0.10.5-rc.1 to rc.14; rc.5 failed its
+61
View File
@@ -671,6 +671,40 @@ llmgw ctl explain-placement qwen2.5-coder-3b-instruct
which names every candidate and why each was rejected, rather than leaving you
to infer it.
### Models served outside the cluster
A model can be an OpenAI-compatible API elsewhere — a hosted provider, or
another gateway — rather than replicas here. Nothing is placed for it; the
router sends its requests on, with the provider's key and the provider's name
for the model:
```yaml
- name: cloud-mini
source: external
external:
url: https://api.openai.com # without /v1
model: gpt-4o-mini # what the provider calls it
api_key_file: /etc/llmgw/secrets/openai.key
```
Named in another model's `fallbacks`, it is used only when the cluster has no
capacity left for that model:
```yaml
- name: qwen2.5-coder-7b-instruct
fallbacks: [cloud-mini]
...
```
The key is read from its file on the control plane, never stored in the
catalog, and a key rotated in the file is used within seconds. A model whose key
file cannot be read is not offered. Everything else applies as to any model:
prices and budgets, rate limits, usage records, and key allowlists.
**Its prompts leave the cluster.** Decide who may send them: a key or project
whose allowlist leaves the external model out cannot reach it, as a fallback or
by name. The cluster page and `ctl models` mark it `external`.
### How many replicas, and how much each one carries
Replica count is changed while the cluster runs, without editing the
@@ -966,6 +1000,33 @@ replica alone on purpose, because churning one costs more than the difference.
`llmgw ctl restart <replica>` replaces it with one on the new profile, starting
the new one before stopping the old.
### Signed releases
Every install above checks a release against its published checksums, which
proves the download is what the server holds — not that the server holds what
was built. Signing closes that: CI signs the checksums, and a binary that
trusts a key installs only releases signed with it. Off until a key exists;
turning it on, once:
```
llmgw release keygen
```
1. Put the **private** key in the repository's Actions secret
`RELEASE_SIGNING_KEY`. From the next tag, releases carry
`checksums.txt.sig`.
2. Put the **public** key in `internal/release/trusted.pub` and release. Every
binary built from then on carries it and refuses releases that are not
signed with it — `ctl upgrade`, agents upgrading themselves, and
`llmgw upgrade` alike.
In that order: a build that trusts a key refuses unsigned releases, so the
secret has to be signing before the key is trusted. The first trusting release
is itself installed by a binary that trusts nothing; every one after is checked
by the one before. `LLMGW_RELEASE_KEY` overrides the built-in key, for rotating
it. The onboarding script fetches with `curl` and checks checksums only; a node
is checked from its first upgrade on.
## Backing up
`llmgw server backup` writes one archive holding everything a control plane
+50
View File
@@ -21,6 +21,7 @@ rather than repeated.
| `band_header` | `string` | [Admission](#admission) |
| `queue_ttl` | `time.Duration` | [Admission](#admission) |
| `queue_capacity_per_band` | `int` | [Admission](#admission) |
| `queue_capacity_total` | `int` | [Admission](#admission) |
| `interval` | `time.Duration` | [Admission](#admission) |
| `enabled` | `bool` | [Auth](#auth) |
| `store` | `string` | [Auth](#auth) |
@@ -77,6 +78,9 @@ rather than repeated.
| `pool` | `string` | [Endpoint](#endpoint) |
| `weight` | `float64` | [Endpoint](#endpoint) |
| `priority` | `int` | [Endpoint](#endpoint) |
| `url` | `string` | [External](#external) |
| `model` | `string` | [External](#external) |
| `api_key_file` | `string` | [External](#external) |
| `contact` | `string` | [Landing](#landing) |
| `notice` | `string` | [Landing](#landing) |
| `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
@@ -94,6 +98,7 @@ rather than repeated.
| `scaling` | `Scaling` | [Model](#model) |
| `class` | `string` | [Model](#model) |
| `owner` | `string` | [Model](#model) |
| `external` | `*External` | [Model](#model) |
| `fallbacks` | `[]string` | [Model](#model) |
| `routing_profile` | `string` | [Model](#model) |
| `slo` | `SLO` | [Model](#model) |
@@ -228,6 +233,15 @@ should not be able to promote themselves.
`int`
### `queue_capacity_total`
`int`
QueueCapacityTotal bounds everything queued across bands. Reached, a
request is not refused while a lower band has any queued: the newest of
the lowest band is turned away instead, with a 503 and Retry-After.
Zero takes half of all the bands' capacities together.
### `interval`
`time.Duration`
@@ -606,6 +620,28 @@ called. Empty means vLLM's names.
`int`
## External
### `url`
`string`
URL is the provider's base address, without /v1: https://api.openai.com.
### `model`
`string`
Model is what the provider calls it; requests are rewritten to it.
### `api_key_file`
`string`
APIKeyFile holds the provider's key, on the control plane. A file
rather than the key itself, so the key is in neither the catalog, the
store nor the audit log; empty for a provider that asks for none.
## Landing
### `contact`
@@ -755,6 +791,20 @@ Owner is the part of the tenancy tree whose capacity quota this model
counts against, written org:ID, team:ID or project:ID. Empty is shared:
held by the cluster, counted against nobody.
### `external`
`*External`
External is a model served by someone else — an OpenAI-compatible API
outside the cluster — rather than by replicas placed here. Such a model
has no profiles and nothing is placed for it; the router sends its
requests to the provider. Most useful named in another model's
fallbacks: used only when the cluster has no capacity left. Its
prompts leave the cluster, so which keys may name it is worth an
allowlist.
Takes a [External](#external) block.
### `fallbacks`
`[]string`
+2
View File
@@ -254,6 +254,8 @@ func modelStatus(m adminapi.ModelView) string {
switch {
case m.Parked:
return "parked"
case m.External != "":
return "external: " + m.External
case m.HeldFor != "":
return "gave room to " + m.HeldFor
case m.Unplaceable:
+67
View File
@@ -0,0 +1,67 @@
package cli
import (
"errors"
"fmt"
"os"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
)
func newReleaseCmd() *cobra.Command {
c := &cobra.Command{
Use: "release",
Short: "Sign releases, and make the key they are signed with",
}
c.AddCommand(&cobra.Command{
Use: "keygen",
Short: "Make a release signing key pair",
Long: "Prints a public key and a private key. The private key goes into the CI\n" +
"secret RELEASE_SIGNING_KEY and nowhere else; the public key into\n" +
"internal/release/trusted.pub, which every build from then on carries\n" +
"and checks releases against. Run once, on a machine you trust.",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
pub, priv, err := release.GenerateKey()
if err != nil {
return err
}
fmt.Fprintf(cmd.OutOrStdout(), "public: %s\nprivate: %s\n", pub, priv)
fmt.Fprintln(cmd.ErrOrStderr(), "\nThe private key signs every release: put it in the CI secret "+
"RELEASE_SIGNING_KEY, then clear this terminal's scrollback.")
return nil
},
})
c.AddCommand(&cobra.Command{
Use: "sign FILE",
Short: "Sign a release's checksums, writing FILE.sig",
Long: "Reads the private key from RELEASE_SIGNING_KEY, never from an argument,\n" +
"so it does not land in a shell history or a process list.",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
key := os.Getenv("RELEASE_SIGNING_KEY")
if key == "" {
return errors.New("RELEASE_SIGNING_KEY is not set")
}
data, err := os.ReadFile(args[0])
if err != nil {
return err
}
sig, err := release.Sign(key, data)
if err != nil {
return err
}
if pub := release.TrustedKey(); pub != "" {
// Signed with a key this build would not accept is a release
// every upgraded machine refuses: stopped here instead.
if err := release.Verify(pub, data, sig); err != nil {
return fmt.Errorf("the signing key does not match this build's trusted key: %w", err)
}
}
return os.WriteFile(args[0]+".sig", []byte(sig+"\n"), 0o644) //nolint:gosec // public
},
})
return c
}
+1
View File
@@ -48,6 +48,7 @@ func Root() *cobra.Command {
newDoctorCmd(),
newNodeCmd(),
newSelfUpgradeCmd(),
newReleaseCmd(),
)
return root
}
+6 -1
View File
@@ -378,7 +378,12 @@ type Admission struct {
QueueTTL time.Duration `koanf:"queue_ttl"`
QueueCapacityPerBand int `koanf:"queue_capacity_per_band"`
Interval time.Duration `koanf:"interval"`
// QueueCapacityTotal bounds everything queued across bands. Reached, a
// request is not refused while a lower band has any queued: the newest of
// the lowest band is turned away instead, with a 503 and Retry-After.
// Zero takes half of all the bands' capacities together.
QueueCapacityTotal int `koanf:"queue_capacity_total"`
Interval time.Duration `koanf:"interval"`
}
// Prefix configures approximate prefix routing.
+22 -4
View File
@@ -10,16 +10,18 @@ import (
"context"
"encoding/json"
"fmt"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"io"
"net/http"
"net/url"
"slices"
"strconv"
"strings"
"time"
"unicode"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
@@ -312,8 +314,11 @@ type ModelView struct {
Warnings []string `json:"warnings,omitempty"`
// Class is its capacity class; Preempted the replicas it lost to a higher
// class; HeldFor the model it is giving room to now, if any.
Class string `json:"class"`
Owner string `json:"owner,omitempty"`
Class string `json:"class"`
Owner string `json:"owner,omitempty"`
// External is where a model served outside the cluster goes, provider
// host and its name for the model; empty for one served here.
External string `json:"external,omitempty"`
Preempted int `json:"preempted,omitempty"`
HeldFor string `json:"held_for,omitempty"`
}
@@ -569,6 +574,7 @@ func (a *API) Models() []ModelView {
Desired: h.Desired, Failures: h.Failures,
Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
Class: m.EffectiveClass(), Owner: m.Owner, Preempted: h.Preempted, HeldFor: h.HeldFor,
External: externalLabel(m),
}
for _, p := range m.Profiles {
v.Profiles = append(v.Profiles, p.Name)
@@ -1443,3 +1449,15 @@ func reservationWarnings(m catalog.Model, measuredGB float64) []string {
}
return out
}
// externalLabel is "api.openai.com · gpt-4o-mini" for a model served elsewhere.
func externalLabel(m catalog.Model) string {
if !m.IsExternal() {
return ""
}
host := m.External.URL
if u, err := url.Parse(host); err == nil && u.Host != "" {
host = u.Host
}
return host + " · " + m.External.Model
}
+45 -3
View File
@@ -7,6 +7,7 @@ import (
"encoding/hex"
"encoding/json"
"fmt"
"net/url"
"slices"
"strings"
"time"
@@ -160,8 +161,16 @@ type Model struct {
// Owner is the part of the tenancy tree whose capacity quota this model
// counts against, written org:ID, team:ID or project:ID. Empty is shared:
// held by the cluster, counted against nobody.
Owner string `json:"owner,omitempty" koanf:"owner"`
Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
Owner string `json:"owner,omitempty" koanf:"owner"`
// External is a model served by someone else — an OpenAI-compatible API
// outside the cluster — rather than by replicas placed here. Such a model
// has no profiles and nothing is placed for it; the router sends its
// requests to the provider. Most useful named in another model's
// fallbacks: used only when the cluster has no capacity left. Its
// prompts leave the cluster, so which keys may name it is worth an
// allowlist.
External *External `json:"external,omitempty" koanf:"external"`
Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
// RoutingProfile selects how the router scores endpoints for this model.
RoutingProfile string `json:"routing_profile,omitempty" koanf:"routing_profile"`
SLO SLO `json:"slo,omitempty" koanf:"slo"`
@@ -305,7 +314,11 @@ func (m Model) validate() error {
if m.Source == "" {
return fmt.Errorf("model %q: source is required", m.Name)
}
if len(m.Profiles) == 0 {
if m.External != nil {
if err := m.External.validate(m.Name, len(m.Profiles)); err != nil {
return err
}
} else if len(m.Profiles) == 0 {
return fmt.Errorf("model %q: at least one profile is required", m.Name)
}
if m.Owner != "" && !validOwner(m.Owner) {
@@ -564,3 +577,32 @@ func validOwner(o string) bool {
}
return false
}
// External is a provider outside the cluster that serves a model.
type External struct {
// URL is the provider's base address, without /v1: https://api.openai.com.
URL string `json:"url" koanf:"url"`
// Model is what the provider calls it; requests are rewritten to it.
Model string `json:"model" koanf:"model"`
// APIKeyFile holds the provider's key, on the control plane. A file
// rather than the key itself, so the key is in neither the catalog, the
// store nor the audit log; empty for a provider that asks for none.
APIKeyFile string `json:"api_key_file,omitempty" koanf:"api_key_file"`
}
func (e *External) validate(model string, profiles int) error {
if profiles > 0 {
return fmt.Errorf("model %q: an external model is served elsewhere and has no profiles", model)
}
u, err := url.Parse(e.URL)
if err != nil || (u.Scheme != "https" && u.Scheme != "http") || u.Host == "" {
return fmt.Errorf("model %q: external url %q is not an http(s) address", model, e.URL)
}
if strings.TrimSpace(e.Model) == "" {
return fmt.Errorf("model %q: external model needs the name the provider knows it by", model)
}
return nil
}
// IsExternal reports whether a model is served outside the cluster.
func (m Model) IsExternal() bool { return m.External != nil }
+31 -1
View File
@@ -9,6 +9,7 @@ import (
"encoding/hex"
"fmt"
"log/slog"
"os"
"strings"
"sync"
"time"
@@ -338,6 +339,34 @@ func (c *Controller) endpoints() []endpoint.Info {
Priority: prof.Priority,
})
}
return append(out, c.externalEndpoints()...)
}
// externalEndpoints are the models served outside the cluster, as endpoints:
// one each, with the provider's key read from its file every time, so a key
// rotated on disk is used from the next publish on.
func (c *Controller) externalEndpoints() []endpoint.Info {
var out []endpoint.Info
for _, m := range c.cat.Load().Models() {
if !m.IsExternal() {
continue
}
key := ""
if f := m.External.APIKeyFile; f != "" {
b, err := os.ReadFile(f) //nolint:gosec // the operator's own path
if err != nil {
c.log.Warn("an external model's key could not be read; not offered", "model", m.Name, "err", err)
continue
}
key = strings.TrimSpace(string(b))
}
out = append(out, endpoint.Info{
ID: "external:" + m.Name, URL: strings.TrimRight(m.External.URL, "/"),
Models: append([]string{m.Name}, m.Aliases...), APIKey: key,
Engine: endpoint.EngineExternal, Pool: "external", Weight: 1,
UpstreamModel: m.External.Model,
})
}
return out
}
@@ -345,7 +374,8 @@ func (c *Controller) endpoints() []endpoint.Info {
func fingerprint(infos []endpoint.Info) string {
var b strings.Builder
for _, i := range infos {
fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight, i.Priority)
fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d|%s|%x;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight, i.Priority,
i.UpstreamModel, sha256.Sum256([]byte(i.APIKey)))
}
return b.String()
}
+75
View File
@@ -0,0 +1,75 @@
package control
import (
"context"
"log/slog"
"os"
"path/filepath"
"sync"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
)
type recordSink struct {
mu sync.Mutex
infos []endpoint.Info
times int
}
func (s *recordSink) SetEndpoints(_ context.Context, infos []endpoint.Info) {
s.mu.Lock()
defer s.mu.Unlock()
s.infos, s.times = infos, s.times+1
}
type noDispatch struct{}
func (noDispatch) StartShard(context.Context, reconciler.ShardCommand) error { return nil }
func (noDispatch) StopShard(context.Context, string, string) error { return nil }
func (noDispatch) DrainShard(context.Context, string, string) error { return nil }
func TestAnExternalModelIsPublishedWithItsKeyAndNothingIsPlacedForIt(t *testing.T) {
dir := t.TempDir()
keyFile := filepath.Join(dir, "provider.key")
if err := os.WriteFile(keyFile, []byte("sk-provider-1\n"), 0o600); err != nil {
t.Fatal(err)
}
models := []catalog.Model{
{Name: "cloud", Source: "external", Aliases: []string{"gpt"}, Scaling: catalog.Scaling{Min: 3, Max: 3},
External: &catalog.External{URL: "https://api.example.com/", Model: "gpt-4o-mini", APIKeyFile: keyFile}},
{Name: "gone", Source: "external",
External: &catalog.External{URL: "https://api.example.com", Model: "m", APIKeyFile: filepath.Join(dir, "missing")}},
}
cat, err := catalog.New(models)
if err != nil {
t.Fatal(err)
}
sink := &recordSink{}
c := New(DefaultConfig(), catalog.NewLive(cat), noDispatch{}, sink, slog.New(slog.DiscardHandler))
c.Publish(t.Context())
if len(sink.infos) != 1 {
t.Fatalf("published %+v; the model whose key is missing must not be offered", sink.infos)
}
e := sink.infos[0]
if e.ID != "external:cloud" || e.URL != "https://api.example.com" || e.APIKey != "sk-provider-1" ||
e.UpstreamModel != "gpt-4o-mini" || e.Engine != endpoint.EngineExternal || len(e.Models) != 2 {
t.Errorf("endpoint %+v", e)
}
// A key rotated on disk is published on the next pass.
_ = os.WriteFile(keyFile, []byte("sk-provider-2"), 0o600)
c.Publish(t.Context())
if sink.times != 2 || sink.infos[0].APIKey != "sk-provider-2" {
t.Errorf("the rotated key was not published: %d pushes, key %q", sink.times, sink.infos[0].APIKey)
}
// Scaling of 3 or not, nothing is wanted of the cluster for a model
// served elsewhere, and so it never reads as short or unplaceable.
for _, h := range c.Reconciler().ModelHealth() {
if h.Model == "cloud" && (h.Desired != 0 || h.Unplaceable) {
t.Errorf("an external model wants replicas: %+v", h)
}
}
}
+3
View File
@@ -94,6 +94,9 @@ func (r *Reconciler) desiredFor(m catalog.Model) int {
// baseFor is what a model runs without load: its minimum, or for an on-demand
// model one while somebody is asking and none once they stop.
func (r *Reconciler) baseFor(m catalog.Model) int {
if m.IsExternal() {
return 0 // served elsewhere: nothing to place, whatever its scaling says
}
if !m.Scaling.OnDemand() {
return m.Scaling.Min
}
+13 -4
View File
@@ -79,16 +79,25 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
victims, nextVictim, preempting := r.victims(m), 0, false
for h.More+h.ByPreempting < limit {
res := place()
// Named only once they have made room: candidates tried on the way
// to finding there is none would otherwise read as whose replicas a
// placement would stop.
var tried []string
for res.Placement == nil && nextVictim < len(victims) {
v := victims[nextVictim]
nextVictim++
inv.ReleaseReplica(v.ID)
if !slices.Contains(h.Preempts, v.Model) {
h.Preempts = append(h.Preempts, v.Model)
}
preempting = true
tried = append(tried, v.Model)
res = place()
}
if res.Placement != nil && len(tried) > 0 {
preempting = true
for _, m := range tried {
if !slices.Contains(h.Preempts, m) {
h.Preempts = append(h.Preempts, m)
}
}
}
if res.Placement == nil {
for _, rej := range res.Rejections {
h.Blocked = append(h.Blocked, rej.String())
@@ -331,3 +331,88 @@ func TestHeadroomCountsWhatAHigherClassCouldTake(t *testing.T) {
t.Errorf("a standard model counted room in another's minimum: %+v", got)
}
}
func TestARestartBorrowsALowerClassesRoomAndGivesItBack(t *testing.T) {
// chat fills node 1; the only other node holds an opportunistic replica.
chat := gpuModel("chat", "", 1)
extra := gpuModel("extra", catalog.ClassOpportunistic, 1)
h := newFleetHarness(t, []catalog.Model{chat, extra}, gpuNode("1", 8), gpuNode("2", 8))
h.settle(t, 3)
h.advance(h.r.cfg.MinLifetime + time.Minute)
var old string
for _, rep := range h.r.Replicas() {
if rep.Model == "chat" {
old = rep.ID
}
}
if old == "" || replicasOf(h, "extra") != 1 {
t.Fatalf("setup: %+v", h.r.Replicas())
}
if err := h.r.Restart(old); err != nil {
t.Fatal(err)
}
// Replacement first, beside the old one, in the room extra gives up;
// the old one keeps serving until then.
servedThroughout := true
for range 8 {
h.r.Reconcile(t.Context())
h.markReady()
ready := 0
for _, rep := range h.r.Replicas() {
if rep.Model == "chat" && rep.State == Ready {
ready++
}
}
if ready == 0 {
servedThroughout = false
}
h.advance(h.r.cfg.PreemptGrace + time.Second)
}
if !servedThroughout {
t.Error("chat had no ready replica during its restart")
}
var newChat bool
for _, rep := range h.r.Replicas() {
if rep.Model == "chat" && rep.ID != old && rep.State == Ready {
newChat = true
}
if rep.ID == old {
t.Errorf("the old replica is still there: %+v", rep)
}
}
if !newChat {
t.Fatalf("no new chat replica: %+v", h.r.Replicas())
}
// And extra has its replica back, in the room the old chat left.
h.settle(t, 4)
if replicasOf(h, "extra") != 1 {
t.Errorf("the opportunistic model did not get its room back: %+v", h.r.Replicas())
}
}
func TestHeadroomNamesOnlyTheReplicasThatWouldMakeRoom(t *testing.T) {
// vital needs a whole node and chat is on the only one; a second
// critical can never fit. Asking for 3: one by stopping chat, then no
// candidates left — nothing else may be named.
h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
got, _ := h.r.Headroom("vital", 3)
if got.ByPreempting != 1 || len(got.Preempts) != 1 {
t.Errorf("%+v", got)
}
// Free room for one, and candidates that cannot make room for a second:
// none of them may be listed.
small := catalog.Model{Name: "small", Source: "/m", Class: catalog.ClassOpportunistic,
Scaling: catalog.Scaling{Min: 1, Max: 1},
Profiles: []catalog.Profile{{Name: "tp2", Engine: "vllm", GPUClass: "h100-80g", TP: 2}}}
h2 := fillThenWant(t, gpuModel("big", "", 1), small, gpuNode("1", 8), gpuNode("2", 8))
got2, _ := h2.r.Headroom("big", 3)
if got2.More != 1 || got2.ByPreempting != 1 || len(got2.Preempts) != 1 {
t.Errorf("one free, one by stopping small: %+v", got2)
}
h3 := fillThenWant(t, gpuModel("big", "", 1), small, gpuNode("1", 8), gpuNode("2", 6))
got3, _ := h3.r.Headroom("big", 3)
if got3.ByPreempting != 0 || len(got3.Preempts) != 0 {
t.Errorf("stopping small frees a 6-GPU node, too small for big; nobody may be named: %+v", got3)
}
}
+27 -4
View File
@@ -9,6 +9,7 @@ package release
import (
"bufio"
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
@@ -51,7 +52,7 @@ func Install(ctx context.Context, client *http.Client, base, version, goos, goar
asset := AssetName(version, goos, goarch)
root := strings.TrimSuffix(base, "/") + "/" + version + "/"
want, err := checksum(ctx, client, root+"checksums.txt", asset)
want, err := checksum(ctx, client, root, asset)
if err != nil {
return "", err
}
@@ -85,13 +86,35 @@ func Install(ctx context.Context, client *http.Client, base, version, goos, goar
return backup, nil
}
func checksum(ctx context.Context, client *http.Client, url, asset string) (string, error) {
body, err := get(ctx, client, url)
// checksum reads the release's checksum for asset — after checking the
// checksums' signature, when this build trusts a signing key. A release with
// no signature is then refused rather than taken on its checksums: a missing
// signature is exactly what replacing a release would look like.
func checksum(ctx context.Context, client *http.Client, root, asset string) (string, error) {
body, err := get(ctx, client, root+"checksums.txt")
if err != nil {
return "", err
}
defer body.Close()
sc := bufio.NewScanner(io.LimitReader(body, 1<<20))
sums, err := io.ReadAll(io.LimitReader(body, 1<<20))
if err != nil {
return "", err
}
if key := TrustedKey(); key != "" {
sigBody, err := get(ctx, client, root+SignatureName)
if err != nil {
return "", fmt.Errorf("this build only installs signed releases, and %s could not be read: %w", SignatureName, err)
}
sig, err := io.ReadAll(io.LimitReader(sigBody, 4096))
sigBody.Close()
if err != nil {
return "", err
}
if err := Verify(key, sums, string(sig)); err != nil {
return "", err
}
}
sc := bufio.NewScanner(bytes.NewReader(sums))
for sc.Scan() {
f := strings.Fields(sc.Text())
if len(f) == 2 && f[1] == asset && len(f[0]) == 64 {
+72
View File
@@ -0,0 +1,72 @@
package release
import (
"crypto/ed25519"
"crypto/rand"
_ "embed"
"encoding/base64"
"errors"
"fmt"
"os"
"strings"
)
// trustedPub is the public key releases are signed with, base64, built into
// every binary from the repository. Empty — as it is until a key is made —
// trusts a release on its checksums alone, as every release before this did.
//
// Built in rather than fetched, because the point of a signature is that the
// server the release comes from cannot vouch for it: a machine that already
// runs llmgw checks the next release against the key the release it runs was
// built with, so replacing both the binaries and the checksums on the server
// is no longer enough.
//
//go:embed trusted.pub
var trustedPub string
// TrustedKey is the key releases are verified with: LLMGW_RELEASE_KEY when set
// (for rotating a key, or testing), the built-in one otherwise; empty is none.
func TrustedKey() string {
if k := strings.TrimSpace(os.Getenv("LLMGW_RELEASE_KEY")); k != "" {
return k
}
return strings.TrimSpace(trustedPub)
}
// SignatureName is the file a release's checksums are signed in.
const SignatureName = "checksums.txt.sig"
// GenerateKey makes a signing key pair, both base64: the private key is the
// 32-byte seed, which is what goes into the CI secret.
func GenerateKey() (public, private string, err error) {
pub, priv, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
return "", "", err
}
return base64.StdEncoding.EncodeToString(pub), base64.StdEncoding.EncodeToString(priv.Seed()), nil
}
// Sign signs data with a base64 seed, returning the signature in base64.
func Sign(privateSeed string, data []byte) (string, error) {
seed, err := base64.StdEncoding.DecodeString(strings.TrimSpace(privateSeed))
if err != nil || len(seed) != ed25519.SeedSize {
return "", errors.New("the signing key is not a base64 ed25519 seed")
}
return base64.StdEncoding.EncodeToString(ed25519.Sign(ed25519.NewKeyFromSeed(seed), data)), nil
}
// Verify checks a base64 signature of data against a base64 public key.
func Verify(public string, data []byte, signature string) error {
pub, err := base64.StdEncoding.DecodeString(strings.TrimSpace(public))
if err != nil || len(pub) != ed25519.PublicKeySize {
return errors.New("the trusted release key is not a base64 ed25519 public key")
}
sig, err := base64.StdEncoding.DecodeString(strings.TrimSpace(signature))
if err != nil {
return fmt.Errorf("the release signature is not base64: %w", err)
}
if !ed25519.Verify(pub, data, sig) {
return errors.New("the release's checksums are not signed by the trusted key")
}
return nil
}
+99
View File
@@ -0,0 +1,99 @@
package release
import (
"crypto/sha256"
"encoding/hex"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
)
// signedServer publishes a release whose checksums are signed with seed,
// or not signed at all when seed is empty; tamper rewrites the checksums
// after signing.
func signedServer(t *testing.T, seed string, body []byte, tamper bool) (*httptest.Server, string) {
t.Helper()
asset := AssetName("v2.0.0", "linux", "amd64")
sum := sha256.Sum256(body)
sums := hex.EncodeToString(sum[:]) + " " + asset + "\n"
sig := ""
if seed != "" {
var err error
if sig, err = Sign(seed, []byte(sums)); err != nil {
t.Fatal(err)
}
}
if tamper {
sums += strings.Repeat("0", 64) + " extra-file\n"
}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/v2.0.0/checksums.txt":
_, _ = w.Write([]byte(sums))
case "/v2.0.0/" + SignatureName:
if sig == "" {
http.NotFound(w, r)
return
}
_, _ = w.Write([]byte(sig + "\n"))
case "/v2.0.0/" + asset:
_, _ = w.Write(body)
default:
http.NotFound(w, r)
}
}))
t.Cleanup(srv.Close)
return srv, asset
}
func TestABuildThatTrustsAKeyInstallsOnlyReleasesSignedWithIt(t *testing.T) {
pub, priv, err := GenerateKey()
if err != nil {
t.Fatal(err)
}
_, otherPriv, _ := GenerateKey()
t.Setenv("LLMGW_RELEASE_KEY", pub)
install := func(srv *httptest.Server) error {
target := filepath.Join(t.TempDir(), "llmgw")
_ = os.WriteFile(target, []byte("old"), 0o755)
_, err := Install(t.Context(), srv.Client(), srv.URL, "v2.0.0", "linux", "amd64", target)
return err
}
body := []byte("new binary")
good, _ := signedServer(t, priv, body, false)
if err := install(good); err != nil {
t.Fatalf("a release signed with the trusted key was refused: %v", err)
}
for name, srv := range map[string]*httptest.Server{
"unsigned": func() *httptest.Server { s, _ := signedServer(t, "", body, false); return s }(),
"signed by another": func() *httptest.Server { s, _ := signedServer(t, otherPriv, body, false); return s }(),
"checksums rewritten": func() *httptest.Server { s, _ := signedServer(t, priv, body, true); return s }(),
} {
if err := install(srv); err == nil {
t.Errorf("%s: installed", name)
}
}
// A build that trusts no key installs on checksums, as before.
t.Setenv("LLMGW_RELEASE_KEY", "")
if TrustedKey() != "" {
t.Skip("this build embeds a trusted key; the unsigned path cannot be tested here")
}
unsigned, _ := signedServer(t, "", body, false)
if err := install(unsigned); err != nil {
t.Errorf("without a trusted key an unsigned release was refused: %v", err)
}
}
func TestAKeyThatIsNotOneIsSaid(t *testing.T) {
if _, err := Sign("not-a-key", []byte("x")); err == nil {
t.Error("signed with a malformed key")
}
if err := Verify("bm9wZQ==", []byte("x"), "c2ln"); err == nil || !strings.Contains(err.Error(), "public key") {
t.Errorf("a malformed trusted key: %v", err)
}
}
View File
Whitespace-only changes.
+54 -2
View File
@@ -34,6 +34,12 @@ type Config struct {
// QueueCapacityPerBand bounds each band's queue. Rejecting immediately
// beats accumulating silently.
QueueCapacityPerBand int `koanf:"queue_capacity_per_band"`
// QueueCapacityTotal bounds everything queued, across bands. When it is
// reached, a request is not refused while a lower band has requests
// queued: the newest of the lowest band is turned away instead
// (displacement), since the oldest have waited longest and are nearest
// their TTL. Zero takes half of every band's capacity together.
QueueCapacityTotal int `koanf:"queue_capacity_total"`
// Interval is how often the dispatcher re-checks saturation. Releases also
// wake it, so this is only the backstop for saturation falling on its own.
Interval time.Duration `koanf:"interval"`
@@ -94,6 +100,9 @@ func (c *Config) normalize() error {
if c.QueueCapacityPerBand <= 0 {
c.QueueCapacityPerBand = 2000
}
if c.QueueCapacityTotal <= 0 {
c.QueueCapacityTotal = max(c.QueueCapacityPerBand*len(c.Bands)/2, 1)
}
if c.Interval <= 0 {
c.Interval = 10 * time.Millisecond
}
@@ -123,6 +132,10 @@ var (
ErrQueueTimeout = errors.New("admission queue timeout")
// ErrClosed means the controller is shutting down.
ErrClosed = errors.New("admission controller closed")
// ErrDisplaced means a queued request was turned away to make room for
// one of a higher band. Worth retrying: the queue is full now, not for
// good.
ErrDisplaced = errors.New("displaced from the admission queue by higher-priority requests")
)
// Stats is a snapshot of one band's queue, for metrics.
@@ -259,6 +272,12 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
c.mu.Unlock()
return nil, fmt.Errorf("%w: band %q", ErrQueueFull, band.Name)
}
if c.queuedLocked() >= c.cfg.QueueCapacityTotal && !c.displaceBelowLocked(idx) {
c.rejected[band.Name]++
c.mu.Unlock()
return nil, fmt.Errorf("%w: %d queued in all, and nothing of a lower band to make room",
ErrQueueFull, c.cfg.QueueCapacityTotal)
}
w := &waiter{
tenant: tenantKey(r),
@@ -279,6 +298,10 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
c.mu.Unlock()
return nil, ErrClosed
}
if w.err != nil {
c.mu.Unlock()
return nil, w.err
}
c.waitTotal[band.Name] += time.Since(w.enqueued)
c.mu.Unlock()
return &Lease{c: c, band: band.Name, model: r.Model}, nil
@@ -297,8 +320,9 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
func (c *Controller) remove(idx int, w *waiter, reason string) {
c.mu.Lock()
defer c.mu.Unlock()
if w.admitted {
// It won the race; the caller will see ready closed next time round.
if w.admitted || w.err != nil {
// It won the race, or was displaced; the caller will see ready
// closed next time round.
return
}
c.queues[idx].remove(w)
@@ -410,3 +434,31 @@ func tenantKey(r Request) string {
}
return "anonymous"
}
// queuedLocked is how many requests wait in every band. Called with mu held.
func (c *Controller) queuedLocked() int {
n := 0
for _, q := range c.queues {
n += q.len()
}
return n
}
// displaceBelowLocked turns away the newest request of the lowest band below
// idx that has any queued, and says whether there was one. Called with mu held.
func (c *Controller) displaceBelowLocked(idx int) bool {
// Bands are sorted highest priority first, so lower bands have larger
// indices, and the lowest is last.
for j := len(c.queues) - 1; j > idx; j-- {
w := c.queues[j].newest()
if w == nil {
continue
}
c.queues[j].remove(w)
w.err = fmt.Errorf("%w (band %q)", ErrDisplaced, c.cfg.Bands[j].Name)
close(w.ready)
c.rejected[c.cfg.Bands[j].Name+"/displaced"]++
return true
}
return false
}
@@ -0,0 +1,89 @@
package admission
import (
"context"
"errors"
"testing"
"time"
)
// queueUntil starts an Acquire in the background and waits until the band
// holds depth requests, returning where its result will arrive.
func queueUntil(t *testing.T, c *Controller, band string, bandIdx, depth int) <-chan error {
t.Helper()
out := make(chan error, 1)
go func() {
l, err := c.Acquire(context.Background(), Request{Model: "m", Band: band})
if err == nil {
l.Release()
}
out <- err
}()
deadline := time.Now().Add(2 * time.Second)
for c.Depths()[bandIdx].Depth < depth && time.Now().Before(deadline) {
time.Sleep(time.Millisecond)
}
if got := c.Depths()[bandIdx].Depth; got != depth {
t.Fatalf("band %s holds %d, expected %d", band, got, depth)
}
return out
}
func TestAHigherBandDisplacesTheNewestOfTheLowest(t *testing.T) {
s := &fixedSat{}
s.set(5) // past every ceiling: everything queues
cfg := testConfig()
cfg.QueueCapacityTotal = 3
c := newController(t, cfg, s.fn)
// Bands, highest first: platinum 0, gold 1, silver 2, bronze 3.
oldest := queueUntil(t, c, "bronze", 3, 1)
time.Sleep(2 * time.Millisecond)
silver := queueUntil(t, c, "silver", 2, 1)
time.Sleep(2 * time.Millisecond)
newest := queueUntil(t, c, "bronze", 3, 2)
// Full at three. Gold arrives: the newest bronze goes, not the oldest,
// and not silver, which outranks bronze.
gold := queueUntil(t, c, "gold", 1, 1)
select {
case err := <-newest:
if !errors.Is(err, ErrDisplaced) {
t.Fatalf("the newest bronze got %v", err)
}
case <-time.After(time.Second):
t.Fatal("nothing was displaced")
}
if d := c.Depths(); d[3].Depth != 1 || d[2].Depth != 1 || d[1].Depth != 1 {
t.Errorf("depths %+v: one bronze, the silver and the gold should remain", d)
}
// Bronze cannot displace anything: nothing is below it.
_, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
if !errors.Is(err, ErrQueueFull) {
t.Errorf("a bronze request at a full queue got %v", err)
}
// Another gold takes the remaining bronze, then silver.
queueUntil(t, c, "gold", 1, 2)
if err := <-oldest; !errors.Is(err, ErrDisplaced) {
t.Errorf("the last bronze got %v", err)
}
queueUntil(t, c, "gold", 1, 3)
if err := <-silver; !errors.Is(err, ErrDisplaced) {
t.Errorf("silver got %v", err)
}
// All gold now: a fourth has nobody lower to displace.
if _, err := c.Acquire(t.Context(), Request{Model: "m", Band: "gold"}); !errors.Is(err, ErrQueueFull) {
t.Errorf("a gold request with only gold queued got %v", err)
}
// And they are served once there is room.
s.set(0)
select {
case err := <-gold:
if err != nil {
t.Errorf("the first gold was not served: %v", err)
}
case <-time.After(2 * time.Second):
t.Error("the queue did not drain")
}
}
+17
View File
@@ -12,6 +12,9 @@ type waiter struct {
// admitted guards the race between the dispatcher admitting a waiter and
// the waiter giving up on its own.
admitted bool
// err is why the waiter was turned away while queued — displaced by a
// higher band — set before ready is closed.
err error
}
// bandQueue holds one band's waiters, grouped by tenant.
@@ -166,3 +169,17 @@ func (q *bandQueue) dropExpired(now time.Time, onDrop func()) {
q.byTenant[tenant] = kept
}
}
// newest is the most recently queued waiter, across every tenant: the one
// displacement takes, since it has invested the least waiting.
func (q *bandQueue) newest() *waiter {
var out *waiter
for _, ws := range q.byTenant {
for _, w := range ws {
if out == nil || w.enqueued.After(out.enqueued) {
out = w
}
}
}
return out
}
+9 -1
View File
@@ -37,8 +37,16 @@ type Info struct {
Weight float64
// Priority expresses pool preference; higher pools are tried first.
Priority int
// UpstreamModel is what the endpoint calls the model, when that differs
// from the catalog's name: an external provider's. Requests are rewritten
// to it. Empty sends the name as the client did.
UpstreamModel string
}
// EngineExternal marks an endpoint that is a provider outside the cluster.
// It publishes no metrics, so it is never scraped and never reads as stale.
const EngineExternal = "external"
// Serves reports whether the replica serves the given model.
func (i Info) Serves(model string) bool {
return slices.Contains(i.Models, model)
@@ -242,7 +250,7 @@ func (r *Registry) Snapshots(model string) []Snapshot {
continue
}
var age time.Duration
stale := true
stale := l.info.Engine != EngineExternal
if !l.scrapedAt.IsZero() {
age = now.Sub(l.scrapedAt)
stale = age > r.cfg.StaleAfter
+89
View File
@@ -0,0 +1,89 @@
package router
import (
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
)
func TestALocalModelWithNoCapacityFallsBackToAnExternalProvider(t *testing.T) {
local := newFakeEngine(t, "local")
local.status.Store(http.StatusServiceUnavailable)
var calls atomic.Int64
var gotModel, gotAuth atomic.Value
provider := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/metrics" {
t.Error("the provider was scraped for metrics")
}
calls.Add(1)
gotAuth.Store(r.Header.Get("Authorization"))
var body map[string]any
b, _ := io.ReadAll(r.Body)
_ = json.Unmarshal(b, &body)
gotModel.Store(fmt.Sprint(body["model"]))
w.Header().Set("Content-Type", "application/json")
fmt.Fprint(w, `{"model":"gpt-4o-mini","choices":[{"message":{"content":"from outside"}}],"usage":{"prompt_tokens":3,"completion_tokens":2}}`)
}))
t.Cleanup(provider.Close)
cfg := baseConfig(config.Endpoint{ID: "l", URL: local.srv.URL, Models: []string{"local"}})
cloud := catalog.Model{Name: "cloud", Source: "external",
External: &catalog.External{URL: provider.URL, Model: "gpt-4o-mini"}}
cfg.Models = []catalog.Model{catModel("local", nil, "cloud"), cloud}
rt := newTestRouter(t, cfg)
rt.SetEndpoints(t.Context(), []endpoint.Info{
{ID: "l", URL: local.srv.URL, Models: []string{"local"}},
{ID: "external:cloud", URL: provider.URL, Models: []string{"cloud"}, APIKey: "provider-key",
Engine: endpoint.EngineExternal, Pool: "external", Weight: 1, UpstreamModel: "gpt-4o-mini"},
})
rec := post(t, rt.Handler(), map[string]any{"model": "local", "messages": []any{}}, nil)
if rec.Code != http.StatusOK {
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
}
if calls.Load() != 1 {
t.Fatalf("the provider was called %d times", calls.Load())
}
if gotModel.Load() != "gpt-4o-mini" {
t.Errorf("the provider was asked for %v, not its own name for the model", gotModel.Load())
}
if gotAuth.Load() != "Bearer provider-key" {
t.Errorf("the provider got %v, not its key", gotAuth.Load())
}
if got := rec.Header().Get("x-llmgw-served-model"); got != "cloud" {
t.Errorf("the client should be told it was served by cloud, got %q", got)
}
// Never scraped, and never stale for it.
for _, s := range rt.reg.Snapshots("cloud") {
if s.Stale {
t.Error("an external endpoint reads as stale")
}
}
}
func TestAnExternalModelIsNotAModelWithProfiles(t *testing.T) {
good := catalog.Model{Name: "cloud", Source: "external",
External: &catalog.External{URL: "https://api.example.com", Model: "gpt-4o-mini"}}
if _, err := catalog.New([]catalog.Model{good}); err != nil {
t.Fatalf("a valid external model was refused: %v", err)
}
for name, m := range map[string]catalog.Model{
"profiles": {Name: "x", Source: "external", External: &catalog.External{URL: "https://a.b", Model: "m"},
Profiles: []catalog.Profile{{Name: "p", Engine: "llamacpp"}}},
"no url": {Name: "x", Source: "external", External: &catalog.External{URL: "api.example.com", Model: "m"}},
"no model": {Name: "x", Source: "external", External: &catalog.External{URL: "https://a.b"}},
} {
if _, err := catalog.New([]catalog.Model{m}); err == nil {
t.Errorf("%s: accepted", name)
}
}
}
+3
View File
@@ -88,6 +88,9 @@ func TestARefusalBeforeDispatchSaysWhetherWaitingHelps(t *testing.T) {
{"queued too long", func(w http.ResponseWriter) {
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueTimeout, time.Now())
}, 504, ""},
{"displaced for a higher band", func(w http.ResponseWriter) {
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrDisplaced, time.Now())
}, 503, "5"},
{"shutting down", func(w http.ResponseWriter) {
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrClosed, time.Now())
}, 503, ""},
+26
View File
@@ -60,9 +60,15 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
return
case errors.Is(err, admission.ErrQueueTimeout):
status = http.StatusGatewayTimeout
case errors.Is(err, admission.ErrDisplaced):
// Turned away for higher-priority work, not for good: said so the
// client backs off and comes back rather than giving up.
w.Header().Set("Retry-After", "5")
}
reason := "unavailable"
switch {
case errors.Is(err, admission.ErrDisplaced):
reason = "displaced"
case errors.Is(err, admission.ErrQueueFull):
reason = "queue_full"
case errors.Is(err, admission.ErrQueueTimeout):
@@ -978,3 +984,23 @@ func (r *Router) rateLimitHeaders(w http.ResponseWriter, p auth.Principal) {
w.Header().Set("x-ratelimit-reset-tokens", reset)
}
}
// withModel is body with its model set to name, for an endpoint that knows the
// model by another name — an external provider. The body unchanged when it is
// not a JSON object, which the endpoint then refuses as it would have anyway.
func withModel(body []byte, name string) []byte {
var m map[string]json.RawMessage
if err := json.Unmarshal(body, &m); err != nil {
return body
}
encoded, err := json.Marshal(name)
if err != nil {
return body
}
m["model"] = encoded
out, err := json.Marshal(m)
if err != nil {
return body
}
return out
}
+6 -1
View File
@@ -347,6 +347,7 @@ func admissionConfig(rc config.Router) admission.Config {
},
QueueTTL: rc.Admission.QueueTTL,
QueueCapacityPerBand: rc.Admission.QueueCapacityPerBand,
QueueCapacityTotal: rc.Admission.QueueCapacityTotal,
Interval: rc.Admission.Interval,
}
}
@@ -714,10 +715,14 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
uctx, uspan := tracer().Start(ctx, "upstream", trace.WithSpanKind(trace.SpanKindClient),
trace.WithAttributes(attribute.String("llmgw.endpoint", dec.Endpoint.ID),
attribute.String("url.full", dec.Endpoint.URL+req.Path)))
body := req.UpstreamBody
if dec.Endpoint.UpstreamModel != "" {
body = withModel(body, dec.Endpoint.UpstreamModel)
}
out, perr := r.px.Do(uctx, w, proxy.Request{
URL: dec.Endpoint.URL + req.Path,
APIKey: dec.Endpoint.APIKey,
Body: req.UpstreamBody,
Body: body,
Header: withTraceparent(uctx, req.Header),
Stream: req.Stream,
SuppressUsageChunk: req.SuppressUsageChunk,
+28 -18
View File
@@ -1,6 +1,9 @@
package scheduler
import (
"sort"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/breaker"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
)
@@ -51,34 +54,41 @@ func KVSaturated(ceiling float64) Filter {
}
}
// PoolPriority keeps only the endpoints of the highest-priority pool
// (plan §7.3, the two-stage decision: pool first, then replica).
// PoolPriority keeps the endpoints of one pool: the highest-priority one,
// unless it is saturated and a lower one is not (plan §7.3, the two-stage
// decision — pool first, then replica).
//
// Spillover arrives in phase 5, at which point this filter will fall through
// to the next pool once the preferred one is saturated.
func PoolPriority() Filter {
return poolPriorityFilter{}
// Saturated is what admission means by it, the same formula over the same
// thresholds, so the two layers agree on when a pool is full: the H100 pool is
// used until it is, and only then the A100 one — never the A100 because it
// happens to be emptier. When every pool is saturated the preferred one is
// kept: moving to a pool that is just as full buys nothing and loses whatever
// made the first preferred.
func PoolPriority(targetWaiting, kvCeiling float64) Filter {
return poolPriorityFilter{t: admission.Thresholds{TargetWaitingPerReplica: targetWaiting, KVCeiling: kvCeiling}}
}
type poolPriorityFilter struct{}
type poolPriorityFilter struct{ t admission.Thresholds }
func (poolPriorityFilter) Name() string { return "pool-priority" }
func (poolPriorityFilter) Filter(_ *Ctx, eps []endpoint.Snapshot) []endpoint.Snapshot {
func (f poolPriorityFilter) Filter(_ *Ctx, eps []endpoint.Snapshot) []endpoint.Snapshot {
if len(eps) == 0 {
return eps
}
best := eps[0].Priority
for _, e := range eps[1:] {
if e.Priority > best {
best = e.Priority
}
}
out := make([]endpoint.Snapshot, 0, len(eps))
byPriority := map[int][]endpoint.Snapshot{}
var order []int
for _, e := range eps {
if e.Priority == best {
out = append(out, e)
if _, ok := byPriority[e.Priority]; !ok {
order = append(order, e.Priority)
}
byPriority[e.Priority] = append(byPriority[e.Priority], e)
}
sort.Sort(sort.Reverse(sort.IntSlice(order)))
for _, p := range order {
if admission.Saturation(byPriority[p], 0, f.t) < 1 {
return byPriority[p]
}
}
return out
return byPriority[order[0]]
}
+3 -3
View File
@@ -50,7 +50,7 @@ func Chat(c ProfileConfig) *Profile {
c.normalize()
return &Profile{
Name: "chat",
Filters: []Filter{BreakerOpen(), PoolPriority()},
Filters: []Filter{BreakerOpen(), PoolPriority(c.TargetWaiting, c.KVCeiling)},
Scorers: c.withPrefix([]WeightedScorer{
{Session(c.SessionLoadFactor), 3.0},
{Queue(c.TargetWaiting), 1.0},
@@ -81,7 +81,7 @@ func Batch(c ProfileConfig) *Profile {
c.normalize()
return &Profile{
Name: "batch",
Filters: []Filter{BreakerOpen(), PoolPriority()},
Filters: []Filter{BreakerOpen(), PoolPriority(c.TargetWaiting, c.KVCeiling)},
Scorers: []WeightedScorer{
{Queue(c.TargetWaiting), 3.0},
{KV(c.KVCeiling), 1.0},
@@ -99,7 +99,7 @@ func LongContext(c ProfileConfig) *Profile {
c.normalize()
return &Profile{
Name: "long-context",
Filters: []Filter{BreakerOpen(), PoolPriority()},
Filters: []Filter{BreakerOpen(), PoolPriority(c.TargetWaiting, c.KVCeiling)},
// Prefix matching earns more here than anywhere else: a long prompt is
// mostly prefill, and prefill is what a cache hit skips.
Scorers: c.withPrefix([]WeightedScorer{
+36 -2
View File
@@ -177,7 +177,7 @@ func TestStaleEndpointIsPenalizedNotDropped(t *testing.T) {
func TestPoolPriority(t *testing.T) {
p := &Profile{
Name: "t",
Filters: []Filter{PoolPriority()},
Filters: []Filter{PoolPriority(4, 0.9)},
Scorers: []WeightedScorer{{Queue(4), 1}},
Picker: MaxScore(),
}
@@ -189,7 +189,8 @@ func TestPoolPriority(t *testing.T) {
if err != nil {
t.Fatal(err)
}
// The lower-priority pool must not win despite being emptier (spillover is phase 5).
// The lower-priority pool must not win for being emptier: the preferred
// one is not saturated (3 waiting of a healthy 4).
if d.Endpoint.ID != "h100" {
t.Errorf("pool priority was ignored: %s", d.Endpoint.ID)
}
@@ -383,3 +384,36 @@ func TestBuiltinProfilesSchedule(t *testing.T) {
}
}
}
func TestASaturatedPoolSpillsToTheNext(t *testing.T) {
p := &Profile{Name: "t", Filters: []Filter{PoolPriority(4, 0.9)},
Scorers: []WeightedScorer{{Queue(4), 1}}, Picker: MaxScore()}
h100 := func(w int) endpoint.Snapshot {
return ep("h100", waiting(w), func(s *endpoint.Snapshot) { s.Priority = 100; s.Pool = "ib-a/h100" })
}
a100 := func(w int) endpoint.Snapshot {
return ep("a100", waiting(w), func(s *endpoint.Snapshot) { s.Priority = 50; s.Pool = "ib-b/a100" })
}
pick := func(eps ...endpoint.Snapshot) string {
d, err := p.Schedule(newCtx(Request{Model: "m"}), eps)
if err != nil {
t.Fatal(err)
}
return d.Endpoint.ID
}
if got := pick(h100(8), a100(0)); got != "a100" {
t.Errorf("the preferred pool at twice its healthy queue did not spill: %s", got)
}
if got := pick(h100(8), a100(6)); got != "h100" {
t.Errorf("both pools saturated: the preferred one should be kept, got %s", got)
}
if got := pick(h100(4), a100(0)); got != "a100" {
t.Errorf("at exactly its healthy limit a pool is full: %s", got)
}
// KV pressure saturates a pool as queue depth does.
full := h100(0)
full.Stats.KVCacheUsage = 0.95
if got := pick(full, a100(0)); got != "a100" {
t.Errorf("a pool out of KV cache did not spill: %s", got)
}
}
+3
View File
@@ -96,6 +96,9 @@ func (p *Poller) Sync(ctx context.Context) {
want := make(map[string]endpoint.Info, len(infos))
for _, in := range infos {
if in.Engine == endpoint.EngineExternal {
continue // a provider elsewhere publishes no /metrics
}
want[in.ID] = in
}
+1 -1
View File
@@ -89,7 +89,7 @@
<td{{if .HeldFor}} class="warn" title="gave room to {{.HeldFor}}"{{end}}>{{.Class}}{{if .Preempted}} <span class="note" title="replicas lost to a higher class">−{{.Preempted}}</span>{{end}}</td>
<td{{with .Warnings}} class="warn" title="{{join .}}"{{end}}>{{if or .MemoryGB .CPUCores}}{{.MemoryGB}}G{{if .CPUCores}} · {{.CPUCores}}c{{end}}{{else}}—{{end}}
{{- if .MeasuredMemoryGB}} <span class="note">{{printf "%.1f" .MeasuredMemoryGB}}G used</span>{{end}}{{if .Warnings}} ⚠{{end}}</td>
<td>{{join .Profiles}}</td>
<td>{{if .External}}<span class="tag" title="served outside the cluster; its prompts leave it">external</span> {{.External}}{{else}}{{join .Profiles}}{{end}}</td>
{{if $.AnyFallbacks}}<td>{{join .Fallbacks}}</td>{{end}}
{{/* Cut to the column, the whole sentence on hover: a row of its own for
each description made the table twice as long as the catalog. */}}
+6 -3
View File
@@ -1459,7 +1459,8 @@ The launch path is now built too:
hits. It does nothing when the working set fits in one replica's cache,
which is worth knowing rather than assuming.
- [x] Fallback models
- [ ] Pool spillover
- [x] Pool spillover: a saturated pool passes requests to the next one that
is not, by admission's own measure of saturation
- [x] `ctl bench`: a load ramp that reports capacity as the last concurrency
level meeting the SLO, plus the queue depth and KV usage observed there,
so `weight`, `target_waiting` and `kv_ceiling` stop being guesses
@@ -1477,7 +1478,8 @@ The launch path is now built too:
### Phase 6 — v2 ideas
- [ ] Precise KV index (§7.5 tier 2) — vLLM KV events plus tokenizer integration
- [ ] Dynamo-style block cost function, tuning the `softmax` picker
- [ ] Admission displacement (dropping requests from lower bands)
- [x] Admission displacement: a total queue bound, and the newest request of
the lowest band turned away (503, Retry-After) for a higher one
- [x] Autoscaling on load: a replica added while requests wait, up to a
model's max, and taken back after ten quiet minutes
- [x] Priority-based preemption: capacity classes (critical, standard,
@@ -1496,7 +1498,8 @@ The launch path is now built too:
checked against their digest, and reclaiming what nothing uses
- [ ] Prefill/decode disaggregation (§12)
- [ ] LoRA adapter routing (affinity to a replica that already has the adapter)
- [ ] External provider support (as a fallback)
- [x] External provider support: a catalog model served by an OpenAI-compatible
API elsewhere, usable as a fallback, under the same allowlists and budgets
- [x] Read-only dashboard on the admin listener (nodes, replicas, models,
placement failures, usage) and a public landing page
- [x] `/chat`: an in-browser playground for signed-in people, dispatched through