Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3d3c96e422 | ||
|
|
623cdb5ba3 | ||
|
|
561667c9be | ||
|
|
cf0178bf05 |
No files matched your search
@@ -93,6 +93,24 @@ jobs:
|
||||
./dist/llmgw-${version}-linux-amd64 --version
|
||||
cat dist/checksums.txt
|
||||
|
||||
# Signed when the repository has a signing key, so a machine that trusts
|
||||
# it installs nothing the server alone could have put there. Without the
|
||||
# secret nothing is signed and releases are checked on their checksums,
|
||||
# as before; a build that trusts a key refuses those, so the secret has
|
||||
# to be in place before trusted.pub is filled in.
|
||||
- name: Sign the checksums
|
||||
env:
|
||||
RELEASE_SIGNING_KEY: ${{ secrets.RELEASE_SIGNING_KEY }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
version="${GITHUB_REF_NAME:-$(git describe --tags --always)}"
|
||||
if [ -z "${RELEASE_SIGNING_KEY:-}" ]; then
|
||||
echo "no RELEASE_SIGNING_KEY secret; releases are not signed"
|
||||
exit 0
|
||||
fi
|
||||
./dist/llmgw-${version}-linux-amd64 release sign dist/checksums.txt
|
||||
echo "signed: dist/checksums.txt.sig"
|
||||
|
||||
- name: Attach to the release
|
||||
env:
|
||||
TOKEN: ${{ secrets.GITEA_TOKEN }}
|
||||
|
||||
@@ -42,6 +42,35 @@ What v0.10.5 made settable from the command line is now on the pages too:
|
||||
critical model's card no longer refuses a replica that preemption would make
|
||||
room for.
|
||||
|
||||
### Displacement: a higher band is not refused while a lower one waits
|
||||
|
||||
Each band's queue was bounded, and nothing bounded them together, so a full
|
||||
gateway turned an interactive request away while batch requests sat queued
|
||||
behind it. `admission.queue_capacity_total` now bounds everything queued —
|
||||
by default half of every band's capacity together. Reached, a request is not
|
||||
refused while a lower band has any queued: the newest request of the lowest
|
||||
band is turned away instead, with a 503 and `Retry-After: 5`, so it backs off
|
||||
and returns. The newest, because the oldest have waited longest and are
|
||||
nearest their TTL. Counted as `displaced` in the admission rejection metric.
|
||||
|
||||
### Releases can be signed
|
||||
|
||||
`llmgw release keygen` makes an ed25519 key pair. With the private key in the
|
||||
CI secret `RELEASE_SIGNING_KEY`, releases carry `checksums.txt.sig`; with the
|
||||
public key in `internal/release/trusted.pub`, every binary built after refuses
|
||||
a release not signed with it, wherever it installs one. Off until then, and
|
||||
docs/install.md says in which order to turn it on.
|
||||
|
||||
### Restarts borrow a lower class's room
|
||||
|
||||
A restart places the replacement beside the replica it replaces, and where
|
||||
there is no free room it now makes some the way any placement does: by
|
||||
stopping a replica its class outranks, which gets its room back once the old
|
||||
replica has gone. The in-place restart — the model unserved while its
|
||||
replacement loads — is left for when nothing of a lower class is running. On a
|
||||
cluster where every model is standard, that is every full one: mark the models
|
||||
that can wait `opportunistic` (`llmgw ctl model class`, or the model's card).
|
||||
|
||||
## v0.10.5 — 2026-09-30
|
||||
|
||||
Published first as release candidates v0.10.5-rc.1 to rc.14; rc.5 failed its
|
||||
|
||||
@@ -966,6 +966,33 @@ replica alone on purpose, because churning one costs more than the difference.
|
||||
`llmgw ctl restart <replica>` replaces it with one on the new profile, starting
|
||||
the new one before stopping the old.
|
||||
|
||||
### Signed releases
|
||||
|
||||
Every install above checks a release against its published checksums, which
|
||||
proves the download is what the server holds — not that the server holds what
|
||||
was built. Signing closes that: CI signs the checksums, and a binary that
|
||||
trusts a key installs only releases signed with it. Off until a key exists;
|
||||
turning it on, once:
|
||||
|
||||
```
|
||||
llmgw release keygen
|
||||
```
|
||||
|
||||
1. Put the **private** key in the repository's Actions secret
|
||||
`RELEASE_SIGNING_KEY`. From the next tag, releases carry
|
||||
`checksums.txt.sig`.
|
||||
2. Put the **public** key in `internal/release/trusted.pub` and release. Every
|
||||
binary built from then on carries it and refuses releases that are not
|
||||
signed with it — `ctl upgrade`, agents upgrading themselves, and
|
||||
`llmgw upgrade` alike.
|
||||
|
||||
In that order: a build that trusts a key refuses unsigned releases, so the
|
||||
secret has to be signing before the key is trusted. The first trusting release
|
||||
is itself installed by a binary that trusts nothing; every one after is checked
|
||||
by the one before. `LLMGW_RELEASE_KEY` overrides the built-in key, for rotating
|
||||
it. The onboarding script fetches with `curl` and checks checksums only; a node
|
||||
is checked from its first upgrade on.
|
||||
|
||||
## Backing up
|
||||
|
||||
`llmgw server backup` writes one archive holding everything a control plane
|
||||
|
||||
@@ -21,6 +21,7 @@ rather than repeated.
|
||||
| `band_header` | `string` | [Admission](#admission) |
|
||||
| `queue_ttl` | `time.Duration` | [Admission](#admission) |
|
||||
| `queue_capacity_per_band` | `int` | [Admission](#admission) |
|
||||
| `queue_capacity_total` | `int` | [Admission](#admission) |
|
||||
| `interval` | `time.Duration` | [Admission](#admission) |
|
||||
| `enabled` | `bool` | [Auth](#auth) |
|
||||
| `store` | `string` | [Auth](#auth) |
|
||||
@@ -228,6 +229,15 @@ should not be able to promote themselves.
|
||||
|
||||
`int`
|
||||
|
||||
### `queue_capacity_total`
|
||||
|
||||
`int`
|
||||
|
||||
QueueCapacityTotal bounds everything queued across bands. Reached, a
|
||||
request is not refused while a lower band has any queued: the newest of
|
||||
the lowest band is turned away instead, with a 503 and Retry-After.
|
||||
Zero takes half of all the bands' capacities together.
|
||||
|
||||
### `interval`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
)
|
||||
|
||||
func newReleaseCmd() *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: "release",
|
||||
Short: "Sign releases, and make the key they are signed with",
|
||||
}
|
||||
c.AddCommand(&cobra.Command{
|
||||
Use: "keygen",
|
||||
Short: "Make a release signing key pair",
|
||||
Long: "Prints a public key and a private key. The private key goes into the CI\n" +
|
||||
"secret RELEASE_SIGNING_KEY and nowhere else; the public key into\n" +
|
||||
"internal/release/trusted.pub, which every build from then on carries\n" +
|
||||
"and checks releases against. Run once, on a machine you trust.",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
pub, priv, err := release.GenerateKey()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "public: %s\nprivate: %s\n", pub, priv)
|
||||
fmt.Fprintln(cmd.ErrOrStderr(), "\nThe private key signs every release: put it in the CI secret "+
|
||||
"RELEASE_SIGNING_KEY, then clear this terminal's scrollback.")
|
||||
return nil
|
||||
},
|
||||
})
|
||||
c.AddCommand(&cobra.Command{
|
||||
Use: "sign FILE",
|
||||
Short: "Sign a release's checksums, writing FILE.sig",
|
||||
Long: "Reads the private key from RELEASE_SIGNING_KEY, never from an argument,\n" +
|
||||
"so it does not land in a shell history or a process list.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
key := os.Getenv("RELEASE_SIGNING_KEY")
|
||||
if key == "" {
|
||||
return errors.New("RELEASE_SIGNING_KEY is not set")
|
||||
}
|
||||
data, err := os.ReadFile(args[0])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sig, err := release.Sign(key, data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if pub := release.TrustedKey(); pub != "" {
|
||||
// Signed with a key this build would not accept is a release
|
||||
// every upgraded machine refuses: stopped here instead.
|
||||
if err := release.Verify(pub, data, sig); err != nil {
|
||||
return fmt.Errorf("the signing key does not match this build's trusted key: %w", err)
|
||||
}
|
||||
}
|
||||
return os.WriteFile(args[0]+".sig", []byte(sig+"\n"), 0o644) //nolint:gosec // public
|
||||
},
|
||||
})
|
||||
return c
|
||||
}
|
||||
@@ -48,6 +48,7 @@ func Root() *cobra.Command {
|
||||
newDoctorCmd(),
|
||||
newNodeCmd(),
|
||||
newSelfUpgradeCmd(),
|
||||
newReleaseCmd(),
|
||||
)
|
||||
return root
|
||||
}
|
||||
|
||||
@@ -378,7 +378,12 @@ type Admission struct {
|
||||
|
||||
QueueTTL time.Duration `koanf:"queue_ttl"`
|
||||
QueueCapacityPerBand int `koanf:"queue_capacity_per_band"`
|
||||
Interval time.Duration `koanf:"interval"`
|
||||
// QueueCapacityTotal bounds everything queued across bands. Reached, a
|
||||
// request is not refused while a lower band has any queued: the newest of
|
||||
// the lowest band is turned away instead, with a 503 and Retry-After.
|
||||
// Zero takes half of all the bands' capacities together.
|
||||
QueueCapacityTotal int `koanf:"queue_capacity_total"`
|
||||
Interval time.Duration `koanf:"interval"`
|
||||
}
|
||||
|
||||
// Prefix configures approximate prefix routing.
|
||||
|
||||
@@ -79,16 +79,25 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
|
||||
victims, nextVictim, preempting := r.victims(m), 0, false
|
||||
for h.More+h.ByPreempting < limit {
|
||||
res := place()
|
||||
// Named only once they have made room: candidates tried on the way
|
||||
// to finding there is none would otherwise read as whose replicas a
|
||||
// placement would stop.
|
||||
var tried []string
|
||||
for res.Placement == nil && nextVictim < len(victims) {
|
||||
v := victims[nextVictim]
|
||||
nextVictim++
|
||||
inv.ReleaseReplica(v.ID)
|
||||
if !slices.Contains(h.Preempts, v.Model) {
|
||||
h.Preempts = append(h.Preempts, v.Model)
|
||||
}
|
||||
preempting = true
|
||||
tried = append(tried, v.Model)
|
||||
res = place()
|
||||
}
|
||||
if res.Placement != nil && len(tried) > 0 {
|
||||
preempting = true
|
||||
for _, m := range tried {
|
||||
if !slices.Contains(h.Preempts, m) {
|
||||
h.Preempts = append(h.Preempts, m)
|
||||
}
|
||||
}
|
||||
}
|
||||
if res.Placement == nil {
|
||||
for _, rej := range res.Rejections {
|
||||
h.Blocked = append(h.Blocked, rej.String())
|
||||
|
||||
@@ -331,3 +331,88 @@ func TestHeadroomCountsWhatAHigherClassCouldTake(t *testing.T) {
|
||||
t.Errorf("a standard model counted room in another's minimum: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARestartBorrowsALowerClassesRoomAndGivesItBack(t *testing.T) {
|
||||
// chat fills node 1; the only other node holds an opportunistic replica.
|
||||
chat := gpuModel("chat", "", 1)
|
||||
extra := gpuModel("extra", catalog.ClassOpportunistic, 1)
|
||||
h := newFleetHarness(t, []catalog.Model{chat, extra}, gpuNode("1", 8), gpuNode("2", 8))
|
||||
h.settle(t, 3)
|
||||
h.advance(h.r.cfg.MinLifetime + time.Minute)
|
||||
var old string
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.Model == "chat" {
|
||||
old = rep.ID
|
||||
}
|
||||
}
|
||||
if old == "" || replicasOf(h, "extra") != 1 {
|
||||
t.Fatalf("setup: %+v", h.r.Replicas())
|
||||
}
|
||||
|
||||
if err := h.r.Restart(old); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Replacement first, beside the old one, in the room extra gives up;
|
||||
// the old one keeps serving until then.
|
||||
servedThroughout := true
|
||||
for range 8 {
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
ready := 0
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.Model == "chat" && rep.State == Ready {
|
||||
ready++
|
||||
}
|
||||
}
|
||||
if ready == 0 {
|
||||
servedThroughout = false
|
||||
}
|
||||
h.advance(h.r.cfg.PreemptGrace + time.Second)
|
||||
}
|
||||
if !servedThroughout {
|
||||
t.Error("chat had no ready replica during its restart")
|
||||
}
|
||||
var newChat bool
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.Model == "chat" && rep.ID != old && rep.State == Ready {
|
||||
newChat = true
|
||||
}
|
||||
if rep.ID == old {
|
||||
t.Errorf("the old replica is still there: %+v", rep)
|
||||
}
|
||||
}
|
||||
if !newChat {
|
||||
t.Fatalf("no new chat replica: %+v", h.r.Replicas())
|
||||
}
|
||||
// And extra has its replica back, in the room the old chat left.
|
||||
h.settle(t, 4)
|
||||
if replicasOf(h, "extra") != 1 {
|
||||
t.Errorf("the opportunistic model did not get its room back: %+v", h.r.Replicas())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHeadroomNamesOnlyTheReplicasThatWouldMakeRoom(t *testing.T) {
|
||||
// vital needs a whole node and chat is on the only one; a second
|
||||
// critical can never fit. Asking for 3: one by stopping chat, then no
|
||||
// candidates left — nothing else may be named.
|
||||
h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
|
||||
got, _ := h.r.Headroom("vital", 3)
|
||||
if got.ByPreempting != 1 || len(got.Preempts) != 1 {
|
||||
t.Errorf("%+v", got)
|
||||
}
|
||||
// Free room for one, and candidates that cannot make room for a second:
|
||||
// none of them may be listed.
|
||||
small := catalog.Model{Name: "small", Source: "/m", Class: catalog.ClassOpportunistic,
|
||||
Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "tp2", Engine: "vllm", GPUClass: "h100-80g", TP: 2}}}
|
||||
h2 := fillThenWant(t, gpuModel("big", "", 1), small, gpuNode("1", 8), gpuNode("2", 8))
|
||||
got2, _ := h2.r.Headroom("big", 3)
|
||||
if got2.More != 1 || got2.ByPreempting != 1 || len(got2.Preempts) != 1 {
|
||||
t.Errorf("one free, one by stopping small: %+v", got2)
|
||||
}
|
||||
h3 := fillThenWant(t, gpuModel("big", "", 1), small, gpuNode("1", 8), gpuNode("2", 6))
|
||||
got3, _ := h3.r.Headroom("big", 3)
|
||||
if got3.ByPreempting != 0 || len(got3.Preempts) != 0 {
|
||||
t.Errorf("stopping small frees a 6-GPU node, too small for big; nobody may be named: %+v", got3)
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,7 @@ package release
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
@@ -51,7 +52,7 @@ func Install(ctx context.Context, client *http.Client, base, version, goos, goar
|
||||
asset := AssetName(version, goos, goarch)
|
||||
root := strings.TrimSuffix(base, "/") + "/" + version + "/"
|
||||
|
||||
want, err := checksum(ctx, client, root+"checksums.txt", asset)
|
||||
want, err := checksum(ctx, client, root, asset)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
@@ -85,13 +86,35 @@ func Install(ctx context.Context, client *http.Client, base, version, goos, goar
|
||||
return backup, nil
|
||||
}
|
||||
|
||||
func checksum(ctx context.Context, client *http.Client, url, asset string) (string, error) {
|
||||
body, err := get(ctx, client, url)
|
||||
// checksum reads the release's checksum for asset — after checking the
|
||||
// checksums' signature, when this build trusts a signing key. A release with
|
||||
// no signature is then refused rather than taken on its checksums: a missing
|
||||
// signature is exactly what replacing a release would look like.
|
||||
func checksum(ctx context.Context, client *http.Client, root, asset string) (string, error) {
|
||||
body, err := get(ctx, client, root+"checksums.txt")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer body.Close()
|
||||
sc := bufio.NewScanner(io.LimitReader(body, 1<<20))
|
||||
sums, err := io.ReadAll(io.LimitReader(body, 1<<20))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if key := TrustedKey(); key != "" {
|
||||
sigBody, err := get(ctx, client, root+SignatureName)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("this build only installs signed releases, and %s could not be read: %w", SignatureName, err)
|
||||
}
|
||||
sig, err := io.ReadAll(io.LimitReader(sigBody, 4096))
|
||||
sigBody.Close()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := Verify(key, sums, string(sig)); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
sc := bufio.NewScanner(bytes.NewReader(sums))
|
||||
for sc.Scan() {
|
||||
f := strings.Fields(sc.Text())
|
||||
if len(f) == 2 && f[1] == asset && len(f[0]) == 64 {
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
package release
|
||||
|
||||
import (
|
||||
"crypto/ed25519"
|
||||
"crypto/rand"
|
||||
_ "embed"
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// trustedPub is the public key releases are signed with, base64, built into
|
||||
// every binary from the repository. Empty — as it is until a key is made —
|
||||
// trusts a release on its checksums alone, as every release before this did.
|
||||
//
|
||||
// Built in rather than fetched, because the point of a signature is that the
|
||||
// server the release comes from cannot vouch for it: a machine that already
|
||||
// runs llmgw checks the next release against the key the release it runs was
|
||||
// built with, so replacing both the binaries and the checksums on the server
|
||||
// is no longer enough.
|
||||
//
|
||||
//go:embed trusted.pub
|
||||
var trustedPub string
|
||||
|
||||
// TrustedKey is the key releases are verified with: LLMGW_RELEASE_KEY when set
|
||||
// (for rotating a key, or testing), the built-in one otherwise; empty is none.
|
||||
func TrustedKey() string {
|
||||
if k := strings.TrimSpace(os.Getenv("LLMGW_RELEASE_KEY")); k != "" {
|
||||
return k
|
||||
}
|
||||
return strings.TrimSpace(trustedPub)
|
||||
}
|
||||
|
||||
// SignatureName is the file a release's checksums are signed in.
|
||||
const SignatureName = "checksums.txt.sig"
|
||||
|
||||
// GenerateKey makes a signing key pair, both base64: the private key is the
|
||||
// 32-byte seed, which is what goes into the CI secret.
|
||||
func GenerateKey() (public, private string, err error) {
|
||||
pub, priv, err := ed25519.GenerateKey(rand.Reader)
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
return base64.StdEncoding.EncodeToString(pub), base64.StdEncoding.EncodeToString(priv.Seed()), nil
|
||||
}
|
||||
|
||||
// Sign signs data with a base64 seed, returning the signature in base64.
|
||||
func Sign(privateSeed string, data []byte) (string, error) {
|
||||
seed, err := base64.StdEncoding.DecodeString(strings.TrimSpace(privateSeed))
|
||||
if err != nil || len(seed) != ed25519.SeedSize {
|
||||
return "", errors.New("the signing key is not a base64 ed25519 seed")
|
||||
}
|
||||
return base64.StdEncoding.EncodeToString(ed25519.Sign(ed25519.NewKeyFromSeed(seed), data)), nil
|
||||
}
|
||||
|
||||
// Verify checks a base64 signature of data against a base64 public key.
|
||||
func Verify(public string, data []byte, signature string) error {
|
||||
pub, err := base64.StdEncoding.DecodeString(strings.TrimSpace(public))
|
||||
if err != nil || len(pub) != ed25519.PublicKeySize {
|
||||
return errors.New("the trusted release key is not a base64 ed25519 public key")
|
||||
}
|
||||
sig, err := base64.StdEncoding.DecodeString(strings.TrimSpace(signature))
|
||||
if err != nil {
|
||||
return fmt.Errorf("the release signature is not base64: %w", err)
|
||||
}
|
||||
if !ed25519.Verify(pub, data, sig) {
|
||||
return errors.New("the release's checksums are not signed by the trusted key")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
package release
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// signedServer publishes a release whose checksums are signed with seed,
|
||||
// or not signed at all when seed is empty; tamper rewrites the checksums
|
||||
// after signing.
|
||||
func signedServer(t *testing.T, seed string, body []byte, tamper bool) (*httptest.Server, string) {
|
||||
t.Helper()
|
||||
asset := AssetName("v2.0.0", "linux", "amd64")
|
||||
sum := sha256.Sum256(body)
|
||||
sums := hex.EncodeToString(sum[:]) + " " + asset + "\n"
|
||||
sig := ""
|
||||
if seed != "" {
|
||||
var err error
|
||||
if sig, err = Sign(seed, []byte(sums)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if tamper {
|
||||
sums += strings.Repeat("0", 64) + " extra-file\n"
|
||||
}
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/v2.0.0/checksums.txt":
|
||||
_, _ = w.Write([]byte(sums))
|
||||
case "/v2.0.0/" + SignatureName:
|
||||
if sig == "" {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte(sig + "\n"))
|
||||
case "/v2.0.0/" + asset:
|
||||
_, _ = w.Write(body)
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return srv, asset
|
||||
}
|
||||
|
||||
func TestABuildThatTrustsAKeyInstallsOnlyReleasesSignedWithIt(t *testing.T) {
|
||||
pub, priv, err := GenerateKey()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, otherPriv, _ := GenerateKey()
|
||||
t.Setenv("LLMGW_RELEASE_KEY", pub)
|
||||
install := func(srv *httptest.Server) error {
|
||||
target := filepath.Join(t.TempDir(), "llmgw")
|
||||
_ = os.WriteFile(target, []byte("old"), 0o755)
|
||||
_, err := Install(t.Context(), srv.Client(), srv.URL, "v2.0.0", "linux", "amd64", target)
|
||||
return err
|
||||
}
|
||||
body := []byte("new binary")
|
||||
|
||||
good, _ := signedServer(t, priv, body, false)
|
||||
if err := install(good); err != nil {
|
||||
t.Fatalf("a release signed with the trusted key was refused: %v", err)
|
||||
}
|
||||
for name, srv := range map[string]*httptest.Server{
|
||||
"unsigned": func() *httptest.Server { s, _ := signedServer(t, "", body, false); return s }(),
|
||||
"signed by another": func() *httptest.Server { s, _ := signedServer(t, otherPriv, body, false); return s }(),
|
||||
"checksums rewritten": func() *httptest.Server { s, _ := signedServer(t, priv, body, true); return s }(),
|
||||
} {
|
||||
if err := install(srv); err == nil {
|
||||
t.Errorf("%s: installed", name)
|
||||
}
|
||||
}
|
||||
|
||||
// A build that trusts no key installs on checksums, as before.
|
||||
t.Setenv("LLMGW_RELEASE_KEY", "")
|
||||
if TrustedKey() != "" {
|
||||
t.Skip("this build embeds a trusted key; the unsigned path cannot be tested here")
|
||||
}
|
||||
unsigned, _ := signedServer(t, "", body, false)
|
||||
if err := install(unsigned); err != nil {
|
||||
t.Errorf("without a trusted key an unsigned release was refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAKeyThatIsNotOneIsSaid(t *testing.T) {
|
||||
if _, err := Sign("not-a-key", []byte("x")); err == nil {
|
||||
t.Error("signed with a malformed key")
|
||||
}
|
||||
if err := Verify("bm9wZQ==", []byte("x"), "c2ln"); err == nil || !strings.Contains(err.Error(), "public key") {
|
||||
t.Errorf("a malformed trusted key: %v", err)
|
||||
}
|
||||
}
|
||||
Whitespace-only changes.
@@ -34,6 +34,12 @@ type Config struct {
|
||||
// QueueCapacityPerBand bounds each band's queue. Rejecting immediately
|
||||
// beats accumulating silently.
|
||||
QueueCapacityPerBand int `koanf:"queue_capacity_per_band"`
|
||||
// QueueCapacityTotal bounds everything queued, across bands. When it is
|
||||
// reached, a request is not refused while a lower band has requests
|
||||
// queued: the newest of the lowest band is turned away instead
|
||||
// (displacement), since the oldest have waited longest and are nearest
|
||||
// their TTL. Zero takes half of every band's capacity together.
|
||||
QueueCapacityTotal int `koanf:"queue_capacity_total"`
|
||||
// Interval is how often the dispatcher re-checks saturation. Releases also
|
||||
// wake it, so this is only the backstop for saturation falling on its own.
|
||||
Interval time.Duration `koanf:"interval"`
|
||||
@@ -94,6 +100,9 @@ func (c *Config) normalize() error {
|
||||
if c.QueueCapacityPerBand <= 0 {
|
||||
c.QueueCapacityPerBand = 2000
|
||||
}
|
||||
if c.QueueCapacityTotal <= 0 {
|
||||
c.QueueCapacityTotal = max(c.QueueCapacityPerBand*len(c.Bands)/2, 1)
|
||||
}
|
||||
if c.Interval <= 0 {
|
||||
c.Interval = 10 * time.Millisecond
|
||||
}
|
||||
@@ -123,6 +132,10 @@ var (
|
||||
ErrQueueTimeout = errors.New("admission queue timeout")
|
||||
// ErrClosed means the controller is shutting down.
|
||||
ErrClosed = errors.New("admission controller closed")
|
||||
// ErrDisplaced means a queued request was turned away to make room for
|
||||
// one of a higher band. Worth retrying: the queue is full now, not for
|
||||
// good.
|
||||
ErrDisplaced = errors.New("displaced from the admission queue by higher-priority requests")
|
||||
)
|
||||
|
||||
// Stats is a snapshot of one band's queue, for metrics.
|
||||
@@ -259,6 +272,12 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
|
||||
c.mu.Unlock()
|
||||
return nil, fmt.Errorf("%w: band %q", ErrQueueFull, band.Name)
|
||||
}
|
||||
if c.queuedLocked() >= c.cfg.QueueCapacityTotal && !c.displaceBelowLocked(idx) {
|
||||
c.rejected[band.Name]++
|
||||
c.mu.Unlock()
|
||||
return nil, fmt.Errorf("%w: %d queued in all, and nothing of a lower band to make room",
|
||||
ErrQueueFull, c.cfg.QueueCapacityTotal)
|
||||
}
|
||||
|
||||
w := &waiter{
|
||||
tenant: tenantKey(r),
|
||||
@@ -279,6 +298,10 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
|
||||
c.mu.Unlock()
|
||||
return nil, ErrClosed
|
||||
}
|
||||
if w.err != nil {
|
||||
c.mu.Unlock()
|
||||
return nil, w.err
|
||||
}
|
||||
c.waitTotal[band.Name] += time.Since(w.enqueued)
|
||||
c.mu.Unlock()
|
||||
return &Lease{c: c, band: band.Name, model: r.Model}, nil
|
||||
@@ -297,8 +320,9 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
|
||||
func (c *Controller) remove(idx int, w *waiter, reason string) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if w.admitted {
|
||||
// It won the race; the caller will see ready closed next time round.
|
||||
if w.admitted || w.err != nil {
|
||||
// It won the race, or was displaced; the caller will see ready
|
||||
// closed next time round.
|
||||
return
|
||||
}
|
||||
c.queues[idx].remove(w)
|
||||
@@ -410,3 +434,31 @@ func tenantKey(r Request) string {
|
||||
}
|
||||
return "anonymous"
|
||||
}
|
||||
|
||||
// queuedLocked is how many requests wait in every band. Called with mu held.
|
||||
func (c *Controller) queuedLocked() int {
|
||||
n := 0
|
||||
for _, q := range c.queues {
|
||||
n += q.len()
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// displaceBelowLocked turns away the newest request of the lowest band below
|
||||
// idx that has any queued, and says whether there was one. Called with mu held.
|
||||
func (c *Controller) displaceBelowLocked(idx int) bool {
|
||||
// Bands are sorted highest priority first, so lower bands have larger
|
||||
// indices, and the lowest is last.
|
||||
for j := len(c.queues) - 1; j > idx; j-- {
|
||||
w := c.queues[j].newest()
|
||||
if w == nil {
|
||||
continue
|
||||
}
|
||||
c.queues[j].remove(w)
|
||||
w.err = fmt.Errorf("%w (band %q)", ErrDisplaced, c.cfg.Bands[j].Name)
|
||||
close(w.ready)
|
||||
c.rejected[c.cfg.Bands[j].Name+"/displaced"]++
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
package admission
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// queueUntil starts an Acquire in the background and waits until the band
|
||||
// holds depth requests, returning where its result will arrive.
|
||||
func queueUntil(t *testing.T, c *Controller, band string, bandIdx, depth int) <-chan error {
|
||||
t.Helper()
|
||||
out := make(chan error, 1)
|
||||
go func() {
|
||||
l, err := c.Acquire(context.Background(), Request{Model: "m", Band: band})
|
||||
if err == nil {
|
||||
l.Release()
|
||||
}
|
||||
out <- err
|
||||
}()
|
||||
deadline := time.Now().Add(2 * time.Second)
|
||||
for c.Depths()[bandIdx].Depth < depth && time.Now().Before(deadline) {
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
if got := c.Depths()[bandIdx].Depth; got != depth {
|
||||
t.Fatalf("band %s holds %d, expected %d", band, got, depth)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestAHigherBandDisplacesTheNewestOfTheLowest(t *testing.T) {
|
||||
s := &fixedSat{}
|
||||
s.set(5) // past every ceiling: everything queues
|
||||
cfg := testConfig()
|
||||
cfg.QueueCapacityTotal = 3
|
||||
c := newController(t, cfg, s.fn)
|
||||
// Bands, highest first: platinum 0, gold 1, silver 2, bronze 3.
|
||||
oldest := queueUntil(t, c, "bronze", 3, 1)
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
silver := queueUntil(t, c, "silver", 2, 1)
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
newest := queueUntil(t, c, "bronze", 3, 2)
|
||||
|
||||
// Full at three. Gold arrives: the newest bronze goes, not the oldest,
|
||||
// and not silver, which outranks bronze.
|
||||
gold := queueUntil(t, c, "gold", 1, 1)
|
||||
select {
|
||||
case err := <-newest:
|
||||
if !errors.Is(err, ErrDisplaced) {
|
||||
t.Fatalf("the newest bronze got %v", err)
|
||||
}
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("nothing was displaced")
|
||||
}
|
||||
if d := c.Depths(); d[3].Depth != 1 || d[2].Depth != 1 || d[1].Depth != 1 {
|
||||
t.Errorf("depths %+v: one bronze, the silver and the gold should remain", d)
|
||||
}
|
||||
|
||||
// Bronze cannot displace anything: nothing is below it.
|
||||
_, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
|
||||
if !errors.Is(err, ErrQueueFull) {
|
||||
t.Errorf("a bronze request at a full queue got %v", err)
|
||||
}
|
||||
// Another gold takes the remaining bronze, then silver.
|
||||
queueUntil(t, c, "gold", 1, 2)
|
||||
if err := <-oldest; !errors.Is(err, ErrDisplaced) {
|
||||
t.Errorf("the last bronze got %v", err)
|
||||
}
|
||||
queueUntil(t, c, "gold", 1, 3)
|
||||
if err := <-silver; !errors.Is(err, ErrDisplaced) {
|
||||
t.Errorf("silver got %v", err)
|
||||
}
|
||||
// All gold now: a fourth has nobody lower to displace.
|
||||
if _, err := c.Acquire(t.Context(), Request{Model: "m", Band: "gold"}); !errors.Is(err, ErrQueueFull) {
|
||||
t.Errorf("a gold request with only gold queued got %v", err)
|
||||
}
|
||||
|
||||
// And they are served once there is room.
|
||||
s.set(0)
|
||||
select {
|
||||
case err := <-gold:
|
||||
if err != nil {
|
||||
t.Errorf("the first gold was not served: %v", err)
|
||||
}
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Error("the queue did not drain")
|
||||
}
|
||||
}
|
||||
@@ -12,6 +12,9 @@ type waiter struct {
|
||||
// admitted guards the race between the dispatcher admitting a waiter and
|
||||
// the waiter giving up on its own.
|
||||
admitted bool
|
||||
// err is why the waiter was turned away while queued — displaced by a
|
||||
// higher band — set before ready is closed.
|
||||
err error
|
||||
}
|
||||
|
||||
// bandQueue holds one band's waiters, grouped by tenant.
|
||||
@@ -166,3 +169,17 @@ func (q *bandQueue) dropExpired(now time.Time, onDrop func()) {
|
||||
q.byTenant[tenant] = kept
|
||||
}
|
||||
}
|
||||
|
||||
// newest is the most recently queued waiter, across every tenant: the one
|
||||
// displacement takes, since it has invested the least waiting.
|
||||
func (q *bandQueue) newest() *waiter {
|
||||
var out *waiter
|
||||
for _, ws := range q.byTenant {
|
||||
for _, w := range ws {
|
||||
if out == nil || w.enqueued.After(out.enqueued) {
|
||||
out = w
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -88,6 +88,9 @@ func TestARefusalBeforeDispatchSaysWhetherWaitingHelps(t *testing.T) {
|
||||
{"queued too long", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueTimeout, time.Now())
|
||||
}, 504, ""},
|
||||
{"displaced for a higher band", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrDisplaced, time.Now())
|
||||
}, 503, "5"},
|
||||
{"shutting down", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrClosed, time.Now())
|
||||
}, 503, ""},
|
||||
|
||||
@@ -60,9 +60,15 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
|
||||
return
|
||||
case errors.Is(err, admission.ErrQueueTimeout):
|
||||
status = http.StatusGatewayTimeout
|
||||
case errors.Is(err, admission.ErrDisplaced):
|
||||
// Turned away for higher-priority work, not for good: said so the
|
||||
// client backs off and comes back rather than giving up.
|
||||
w.Header().Set("Retry-After", "5")
|
||||
}
|
||||
reason := "unavailable"
|
||||
switch {
|
||||
case errors.Is(err, admission.ErrDisplaced):
|
||||
reason = "displaced"
|
||||
case errors.Is(err, admission.ErrQueueFull):
|
||||
reason = "queue_full"
|
||||
case errors.Is(err, admission.ErrQueueTimeout):
|
||||
|
||||
@@ -347,6 +347,7 @@ func admissionConfig(rc config.Router) admission.Config {
|
||||
},
|
||||
QueueTTL: rc.Admission.QueueTTL,
|
||||
QueueCapacityPerBand: rc.Admission.QueueCapacityPerBand,
|
||||
QueueCapacityTotal: rc.Admission.QueueCapacityTotal,
|
||||
Interval: rc.Admission.Interval,
|
||||
}
|
||||
}
|
||||
|
||||
+2
-1
@@ -1477,7 +1477,8 @@ The launch path is now built too:
|
||||
### Phase 6 — v2 ideas
|
||||
- [ ] Precise KV index (§7.5 tier 2) — vLLM KV events plus tokenizer integration
|
||||
- [ ] Dynamo-style block cost function, tuning the `softmax` picker
|
||||
- [ ] Admission displacement (dropping requests from lower bands)
|
||||
- [x] Admission displacement: a total queue bound, and the newest request of
|
||||
the lowest band turned away (503, Retry-After) for a higher one
|
||||
- [x] Autoscaling on load: a replica added while requests wait, up to a
|
||||
model's max, and taken back after ten quiet minutes
|
||||
- [x] Priority-based preemption: capacity classes (critical, standard,
|
||||
|
||||
Reference in new issue
Block a user