Merge admission-bulk into jobs-3d
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
ec233d3586
20 files changed
+328
-40
No files matched your search
+5
-4
@@ -1091,10 +1091,11 @@ at a third of its CPUs.
|
||||
FlashAttention and ubatch 1–2k; f16 no better. vLLM fp16 (`--runner
|
||||
pooling`) 6,200 direct, 4,700–4,900 through the gateway in the import,
|
||||
and its vectors match llama.cpp's (cosine 0.9995). The 103,579 chunks
|
||||
took 3 h. Wanted: an embedding model's GPU profile on vLLM and CPU
|
||||
profile on llama.cpp, which needs a source per profile (the catalog has
|
||||
one per model; the import swapped the entry by hand), and an admission
|
||||
queue limit that suits bulk imports (30 s turned some away).
|
||||
took 3 h. Done (2026-10-10, branch `admission-bulk`): a source per
|
||||
profile, so an embedding model's GPU profile runs vLLM and its CPU
|
||||
profile llama.cpp; a band's own `queue_ttl` for bulk imports, and
|
||||
`Retry-After` on a busy refusal, which is what had turned passages away
|
||||
(llmgw-kb retries only what says when to).
|
||||
- Retrieval on 40 questions written from the corpus, document in the top
|
||||
5: bge-m3 31/40 (Turkish questions 13/21), Qwen3-Embedding-4B 37/40
|
||||
(18/21), with its query prefix 36/40 and first place 31/40 instead of
|
||||
|
||||
@@ -41,6 +41,15 @@ APIs change between minors without a deprecation period.
|
||||
`router.batch.retain_days`. Migration 0044. No model makes 3D models
|
||||
yet: every request is refused at the door until the ComfyUI adapter
|
||||
learns meshes (BACKLOG G3).
|
||||
- **A profile's own `source`**: the weights it loads instead of the
|
||||
model's, so an embedding model can run on vLLM from its repository on a
|
||||
card and on llama.cpp from a GGUF on CPUs as one model. Placement prefers
|
||||
nodes holding each profile's weights, `ctl model add` checks every
|
||||
source, and `ctl weights` collects none a profile loads.
|
||||
- **A band's own `queue_ttl`** (`router.admission.bands[].queue_ttl`): how
|
||||
long its requests may wait for room, instead of the admission's 30 s. A
|
||||
band for bulk work — a knowledge-base import — waits behind people rather
|
||||
than being turned away.
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -59,6 +68,10 @@ APIs change between minors without a deprecation period.
|
||||
hour of quiet per event, so a budget spent on a team both served was
|
||||
announced twice. The control plane that claims an event in the store
|
||||
(migration 0043) sends it; with the store unreachable it is sent anyway.
|
||||
- **A request turned away for a busy cluster says when to come back.** A
|
||||
request that waited as long as it may (504) or found the queue full (503)
|
||||
had no `Retry-After`, so `llmgw-kb` counted the passage as failed instead
|
||||
of sending it again; both now say 10 seconds.
|
||||
- The usage sums and exports count pictures (`images`), as the request
|
||||
records did; pictures made before are added to the sums.
|
||||
|
||||
|
||||
@@ -8837,6 +8837,10 @@
|
||||
"gpu_class": {
|
||||
"type": "string"
|
||||
},
|
||||
"source": {
|
||||
"type": "string",
|
||||
"description": "Source is the weights this profile loads, when not the model's: an embedding model served by vLLM from its repository on a card and by llama.cpp from a GGUF on CPUs is one model with two sources. Empty loads the model's. Updates and canaries move the model's source; a profile with its own keeps it."
|
||||
},
|
||||
"tp": {
|
||||
"type": "integer"
|
||||
},
|
||||
|
||||
+2
-1
@@ -547,7 +547,8 @@ its usual exception with the gateway's message in it. The ones worth knowing:
|
||||
| 401 | No key, a mistyped one, or one that was revoked or has expired. |
|
||||
| 403 | The key may not use that model. |
|
||||
| 429 | A rate limit or budget of the key, its project or its team was reached. `Retry-After` says when to try again, and is absent when waiting will not help — a spent monthly budget. |
|
||||
| 503 | No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the request waited as long as it may. The SDKs retry by themselves; a name that is not served will not start to be. |
|
||||
| 503 | No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the queue is full. The SDKs retry by themselves; a name that is not served will not start to be. |
|
||||
| 504 | Every replica was busy for as long as the key's band may wait. `Retry-After` says when to send it again. |
|
||||
|
||||
## Priority
|
||||
|
||||
|
||||
@@ -1038,6 +1038,27 @@ qwen2.5-coder-3b-instruct yes 1/1 ready coder
|
||||
No restart, and it survives one: models added this way are stored, while
|
||||
anything in `server.yaml` is declarative and returns on every start.
|
||||
|
||||
A profile may load weights of its own. An embedding model, for example, can
|
||||
be served by vLLM from its repository on a card and by llama.cpp from a GGUF
|
||||
on CPUs, as one model:
|
||||
|
||||
```yaml
|
||||
name: qwen3-embed-4b
|
||||
source: Qwen/Qwen3-Embedding-4B # what a profile loads unless it says
|
||||
capabilities: { embeddings: true }
|
||||
profiles:
|
||||
- name: gpu
|
||||
engine: vllm
|
||||
gpu_class: 6000-24g
|
||||
args: ["--runner", "pooling"]
|
||||
- name: cpu
|
||||
engine: llamacpp
|
||||
source: hf:Qwen/Qwen3-Embedding-4B-GGUF/Qwen3-Embedding-4B-Q8_0.gguf
|
||||
```
|
||||
|
||||
Updates and canaries move the model's `source`; a profile with its own keeps
|
||||
it.
|
||||
|
||||
### Models that load when someone asks
|
||||
|
||||
A cluster with more models than GPUs can hold every model in its catalog and
|
||||
|
||||
@@ -146,6 +146,24 @@ Rules that apply:
|
||||
- Every document added or removed is in the audit log (`llmgw ctl audit`),
|
||||
by name. The text is never recorded.
|
||||
|
||||
### A large import
|
||||
|
||||
A few thousand documents ask the embedding model for far more than it can
|
||||
answer at once. Give the key that sends them a band of its own, below the
|
||||
people's, that may wait for room for minutes rather than the 30 seconds the
|
||||
others wait:
|
||||
|
||||
```sh
|
||||
llmgw ctl config set router.admission.bands '[{name: gold, priority: 300, saturation_ceiling: 0.85},
|
||||
{name: bronze, priority: 100, saturation_ceiling: 0.3}, {name: bulk, priority: 50, saturation_ceiling: 0.3, queue_ttl: 5m}]'
|
||||
llmgw keys create import --project Acme/docs --band bulk
|
||||
```
|
||||
|
||||
The people's questions then go first, and the import waits behind them. A
|
||||
request that waits as long as its band allows is answered 504 with
|
||||
`Retry-After`, and `llmgw-kb` sends it again rather than counting the
|
||||
passage as failed. It gives up after 15 minutes.
|
||||
|
||||
## For everyone: `llmgw-kb`
|
||||
|
||||
### Get it
|
||||
|
||||
@@ -220,6 +220,7 @@ rather than repeated.
|
||||
| `engine` | `string` | [Profile](#profile) |
|
||||
| `engine_version` | `string` | [Profile](#profile) |
|
||||
| `gpu_class` | `string` | [Profile](#profile) |
|
||||
| `source` | `string` | [Profile](#profile) |
|
||||
| `tp` | `int` | [Profile](#profile) |
|
||||
| `pp` | `int` | [Profile](#profile) |
|
||||
| `gpus_per_node` | `int` | [Profile](#profile) |
|
||||
@@ -356,6 +357,7 @@ rather than repeated.
|
||||
| `name` | `string` | [admission.Band](#admissionband) |
|
||||
| `priority` | `int` | [admission.Band](#admissionband) |
|
||||
| `saturation_ceiling` | `float64` | [admission.Band](#admissionband) |
|
||||
| `queue_ttl` | `time.Duration` | [admission.Band](#admissionband) |
|
||||
| `bands` | `[]Band` | [admission.Config](#admissionconfig) |
|
||||
| `default_band` | `string` | [admission.Config](#admissionconfig) |
|
||||
| `queue_ttl` | `time.Duration` | [admission.Config](#admissionconfig) |
|
||||
@@ -385,6 +387,7 @@ rather than repeated.
|
||||
| `name` | `string` | [config.Band](#configband) |
|
||||
| `priority` | `int` | [config.Band](#configband) |
|
||||
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
|
||||
| `queue_ttl` | `time.Duration` | [config.Band](#configband) |
|
||||
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `per_image` | `int64` | [config.Price](#configprice) |
|
||||
@@ -1923,6 +1926,16 @@ version starts with it: "0.11" takes vLLM 0.11.0 and 0.11.1, not
|
||||
|
||||
`string`
|
||||
|
||||
### `source`
|
||||
|
||||
`string`
|
||||
|
||||
Source is the weights this profile loads, when not the model's: an
|
||||
embedding model served by vLLM from its repository on a card and by
|
||||
llama.cpp from a GGUF on CPUs is one model with two sources. Empty
|
||||
loads the model's. Updates and canaries move the model's source; a
|
||||
profile with its own keeps it.
|
||||
|
||||
### `tp`
|
||||
|
||||
`int`
|
||||
@@ -3013,6 +3026,15 @@ Priority orders bands; higher is served first.
|
||||
SaturationCeiling is the pool saturation above which this band waits.
|
||||
1.0 means "never hold this band back".
|
||||
|
||||
### `queue_ttl`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
QueueTTL, set, is how long this band's requests may wait instead of
|
||||
the controller's: a band for bulk work — a knowledge-base import, a
|
||||
backfill — whose requests had rather wait ten minutes behind people
|
||||
than be turned away after thirty seconds.
|
||||
|
||||
## admission.Config
|
||||
|
||||
### `bands`
|
||||
@@ -3226,6 +3248,14 @@ Takes a [Tracing](#tracing) block.
|
||||
|
||||
`float64`
|
||||
|
||||
### `queue_ttl`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
QueueTTL is how long this band's requests may wait for room; the
|
||||
admission's queue_ttl unless set. Long for a band of bulk work, which
|
||||
had rather wait behind people than be turned away.
|
||||
|
||||
## config.Price
|
||||
|
||||
### `prompt_per_1k`
|
||||
|
||||
+11
-8
@@ -80,15 +80,18 @@ func newModelAddCmd(client func() *ctlClient) *cobra.Command {
|
||||
// resolve is otherwise discovered by a replica failing to start
|
||||
// minutes later on a node — with an error naming the node rather
|
||||
// than the typo.
|
||||
// Every source the model's profiles load, not the model's alone.
|
||||
if !skipCheck {
|
||||
size, err := sourcecheck.Check(cmd.Context(), m.Source)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w\n Pass --no-source-check to add it anyway, "+
|
||||
"which is right when the mirror is not up yet or the nodes "+
|
||||
"reach it by an address this machine does not.", err)
|
||||
}
|
||||
if _, isURL := sourcecheck.HTTPSource(m.Source); isURL {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), sourceNote(m.Source, size))
|
||||
for _, src := range m.WeightSources() {
|
||||
size, err := sourcecheck.Check(cmd.Context(), src)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w\n Pass --no-source-check to add it anyway, "+
|
||||
"which is right when the mirror is not up yet or the nodes "+
|
||||
"reach it by an address this machine does not.", err)
|
||||
}
|
||||
if _, isURL := sourcecheck.HTTPSource(src); isURL {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), sourceNote(src, size))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+10
-10
@@ -1173,18 +1173,18 @@ func servableCheck(llamaVersion string) func(context.Context, catalog.Model) err
|
||||
if llamaVersion == "" {
|
||||
return nil
|
||||
}
|
||||
usesLlama := false
|
||||
// Each source a llama.cpp profile loads: its own, or the model's.
|
||||
for _, p := range m.Profiles {
|
||||
if p.Engine == "llamacpp" {
|
||||
usesLlama = true
|
||||
if p.Engine != "llamacpp" {
|
||||
continue
|
||||
}
|
||||
url, ok := sourcecheck.HTTPSource(m.SourceOf(p))
|
||||
if !ok || !gguf.IsGGUF(url) {
|
||||
continue
|
||||
}
|
||||
if v := gguf.Check(ctx, url, llamaVersion); !v.Servable {
|
||||
return fmt.Errorf("%s cannot be served: %s", m.Name, v.Reason)
|
||||
}
|
||||
}
|
||||
url, ok := sourcecheck.HTTPSource(m.Source)
|
||||
if !usesLlama || !ok || !gguf.IsGGUF(url) {
|
||||
return nil
|
||||
}
|
||||
if v := gguf.Check(ctx, url, llamaVersion); !v.Servable {
|
||||
return fmt.Errorf("%s cannot be served: %s", m.Name, v.Reason)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -689,6 +689,10 @@ type Band struct {
|
||||
Name string `koanf:"name"`
|
||||
Priority int `koanf:"priority"`
|
||||
SaturationCeiling float64 `koanf:"saturation_ceiling"`
|
||||
// QueueTTL is how long this band's requests may wait for room; the
|
||||
// admission's queue_ttl unless set. Long for a band of bulk work, which
|
||||
// had rather wait behind people than be turned away.
|
||||
QueueTTL time.Duration `koanf:"queue_ttl"`
|
||||
}
|
||||
|
||||
// Timeouts bounds each phase of an upstream request.
|
||||
|
||||
@@ -30,7 +30,7 @@ type OrphanView struct {
|
||||
func (a *API) Orphans() []OrphanView {
|
||||
var wanted []string
|
||||
for _, m := range a.ctl.Catalog().Models() {
|
||||
wanted = append(wanted, m.Source)
|
||||
wanted = append(wanted, m.WeightSources()...)
|
||||
// A model's adapters are its weights too; collecting them would
|
||||
// leave replicas that cannot restart.
|
||||
for _, ad := range m.Adapters {
|
||||
|
||||
@@ -174,6 +174,12 @@ type Profile struct {
|
||||
// 0.10.2. Empty takes whatever version the node has.
|
||||
EngineVersion string `json:"engine_version,omitempty" koanf:"engine_version"`
|
||||
GPUClass string `json:"gpu_class,omitempty" koanf:"gpu_class"`
|
||||
// Source is the weights this profile loads, when not the model's: an
|
||||
// embedding model served by vLLM from its repository on a card and by
|
||||
// llama.cpp from a GGUF on CPUs is one model with two sources. Empty
|
||||
// loads the model's. Updates and canaries move the model's source; a
|
||||
// profile with its own keeps it.
|
||||
Source string `json:"source,omitempty" koanf:"source"`
|
||||
|
||||
TP int `json:"tp,omitempty" koanf:"tp"`
|
||||
PP int `json:"pp,omitempty" koanf:"pp"`
|
||||
@@ -762,9 +768,15 @@ func (m Model) normalize() Model {
|
||||
// repository it names: the engine downloads a repository id itself,
|
||||
// where the page was fetched as one file of HTML and failed at start.
|
||||
if repo, ok := repositoryPage(m.Source); ok && len(m.Profiles) > 0 &&
|
||||
!slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" }) {
|
||||
!slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" && p.Source == "" }) {
|
||||
m.Source = repo
|
||||
}
|
||||
for i := range m.Profiles {
|
||||
p := &m.Profiles[i]
|
||||
if repo, ok := repositoryPage(p.Source); ok && p.Engine != "llamacpp" {
|
||||
p.Source = repo
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
@@ -775,15 +787,26 @@ func (m Model) validate() error {
|
||||
if m.Source == "" {
|
||||
return fmt.Errorf("model %q: source is required", m.Name)
|
||||
}
|
||||
if repo, ok := repositoryPage(m.Source); ok {
|
||||
// A page, not weights: a node fetches the HTML as one file and the
|
||||
// engine fails at start, on a path that looks right. For vLLM it was
|
||||
// read as the repository id when the model was made; only a model
|
||||
// llama.cpp runs is left here, which needs a file.
|
||||
// A page, not weights: a node fetches the HTML as one file and the
|
||||
// engine fails at start, on a path that looks right. For vLLM it was
|
||||
// read as the repository id when the model was made; only a source
|
||||
// llama.cpp loads is left here, which needs a file.
|
||||
if repo, ok := repositoryPage(m.Source); ok && (len(m.Profiles) == 0 ||
|
||||
slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" && p.Source == "" })) {
|
||||
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not a file; "+
|
||||
"llama.cpp needs a .gguf: mirror one and use its source (repository %s)",
|
||||
m.Name, m.Source, repo)
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
if repo, ok := repositoryPage(p.Source); ok {
|
||||
return fmt.Errorf("model %q profile %q: source %s is a Hugging Face repository page, not a file; "+
|
||||
"llama.cpp needs a .gguf: mirror one and use its source (repository %s)",
|
||||
m.Name, p.Name, p.Source, repo)
|
||||
}
|
||||
if p.Source != "" && m.External != nil {
|
||||
return fmt.Errorf("model %q profile %q: an external model has no profiles to give a source", m.Name, p.Name)
|
||||
}
|
||||
}
|
||||
if err := validateAdapters(m); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -924,7 +947,7 @@ func (m Model) validate() error {
|
||||
return fmt.Errorf("model %q profile %q: trtllm runs a replica on one node; "+
|
||||
"use vllm or sglang for a %d-node replica", m.Name, p.Name, p.Nodes)
|
||||
}
|
||||
if strings.HasSuffix(strings.ToLower(strings.TrimSuffix(m.Source, "/")), ".gguf") {
|
||||
if strings.HasSuffix(strings.ToLower(strings.TrimSuffix(m.SourceOf(p), "/")), ".gguf") {
|
||||
return fmt.Errorf("model %q profile %q: trtllm loads a Hugging Face checkpoint, not a .gguf; "+
|
||||
"serve a GGUF with llamacpp", m.Name, p.Name)
|
||||
}
|
||||
@@ -1010,14 +1033,38 @@ func (m Model) PlacementProfiles() []placement.Profile {
|
||||
CPUCores: p.CPUCores,
|
||||
PowerWatts: p.PowerWatts,
|
||||
KeepWithin: facility.Level(strings.ToLower(strings.TrimSpace(p.KeepWithin))),
|
||||
// The model's source, so placement can prefer nodes that already
|
||||
// The profile's source, so placement can prefer nodes that already
|
||||
// hold the weights rather than one that must first move them.
|
||||
Source: m.Source,
|
||||
Source: m.SourceOf(p),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// SourceOf is the weights a profile loads: its own, or the model's.
|
||||
func (m Model) SourceOf(p Profile) string {
|
||||
if p.Source != "" {
|
||||
return p.Source
|
||||
}
|
||||
return m.Source
|
||||
}
|
||||
|
||||
// WeightSources are every source a model's replicas may load: the model's
|
||||
// and each profile's of its own, once each. What a node must not have
|
||||
// collected from under it.
|
||||
func (m Model) WeightSources() []string {
|
||||
out := []string{}
|
||||
if m.Source != "" {
|
||||
out = append(out, m.Source)
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
if p.Source != "" && !slices.Contains(out, p.Source) {
|
||||
out = append(out, p.Source)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// EngineProfile converts a catalog profile into the engine adapter's view.
|
||||
func (m Model) EngineProfile(name string) (engines.Profile, bool) {
|
||||
for _, p := range m.Profiles {
|
||||
@@ -1034,7 +1081,7 @@ func (m Model) EngineProfile(name string) (engines.Profile, bool) {
|
||||
Nodes: p.Nodes,
|
||||
Args: slices.Clone(p.Args),
|
||||
Env: p.Env,
|
||||
ModelPath: m.Source,
|
||||
ModelPath: m.SourceOf(p),
|
||||
ServedModelName: m.Name,
|
||||
Adapters: m.engineAdapters(),
|
||||
KVEvents: p.KVEvents,
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// embedModel is an embedding model the way it was wanted on the PoC: vLLM
|
||||
// on the card from the repository, llama.cpp on CPUs from a GGUF.
|
||||
func embedModel() Model {
|
||||
return Model{Name: "qwen3-embed-4b", Source: "Qwen/Qwen3-Embedding-4B",
|
||||
Profiles: []Profile{
|
||||
{Name: "gpu", Engine: "vllm", GPUClass: "6000-24g", TP: 1},
|
||||
{Name: "cpu", Engine: "llamacpp", Source: "hf:Qwen/Qwen3-Embedding-4B-GGUF/Qwen3-Embedding-4B-Q8_0.gguf"},
|
||||
}}
|
||||
}
|
||||
|
||||
func TestEachProfileLoadsItsOwnSource(t *testing.T) {
|
||||
// The catalog had one source a model, so serving an embedding model by
|
||||
// vLLM on a card and by llama.cpp on CPUs meant swapping the entry by
|
||||
// hand halfway through an import.
|
||||
cat, err := New([]Model{embedModel()})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m, _ := cat.Resolve("qwen3-embed-4b")
|
||||
gpu, _ := m.EngineProfile("gpu")
|
||||
cpu, _ := m.EngineProfile("cpu")
|
||||
if gpu.ModelPath != "Qwen/Qwen3-Embedding-4B" || !strings.HasSuffix(cpu.ModelPath, "Q8_0.gguf") {
|
||||
t.Errorf("gpu loads %q, cpu %q", gpu.ModelPath, cpu.ModelPath)
|
||||
}
|
||||
for _, p := range m.PlacementProfiles() {
|
||||
if want := map[string]string{"gpu": gpu.ModelPath, "cpu": cpu.ModelPath}[p.Name]; p.Source != want {
|
||||
t.Errorf("placement for %s looks for %q, want %q", p.Name, p.Source, want)
|
||||
}
|
||||
}
|
||||
if got := m.WeightSources(); len(got) != 2 || !slices.Contains(got, cpu.ModelPath) {
|
||||
t.Errorf("weight sources %v: a profile's own must not be collected", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAProfileWithoutASourceKeepsItsFingerprint(t *testing.T) {
|
||||
// Replicas are compared with their profile by fingerprint; a field added
|
||||
// to every profile must not make every running replica look stale.
|
||||
p := Profile{Name: "gpu", Engine: "vllm", GPUClass: "6000-24g", TP: 1}
|
||||
before := p.Fingerprint()
|
||||
if strings.Contains(mustJSON(t, p), "source") {
|
||||
t.Error("an empty source is written into the profile")
|
||||
}
|
||||
p.Source = "Qwen/Other"
|
||||
if p.Fingerprint() == before {
|
||||
t.Error("a profile whose source changed reads as unchanged")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAProfileSourceIsCheckedAsTheModelsIs(t *testing.T) {
|
||||
page := embedModel()
|
||||
page.Profiles[1].Source = "https://huggingface.co/Qwen/Qwen3-Embedding-4B-GGUF"
|
||||
if _, err := New([]Model{page}); err == nil || !strings.Contains(err.Error(), `profile "cpu"`) {
|
||||
t.Errorf("a repository page for llama.cpp: %v", err)
|
||||
}
|
||||
// The model's own source may be a page vLLM reads as a repository, now
|
||||
// that the llama.cpp profile loads something else.
|
||||
repo := embedModel()
|
||||
repo.Source = "https://huggingface.co/Qwen/Qwen3-Embedding-4B"
|
||||
cat, err := New([]Model{repo})
|
||||
if err != nil {
|
||||
t.Fatalf("a repository page for the vLLM profile beside a llama.cpp one with its own GGUF: %v", err)
|
||||
}
|
||||
m, _ := cat.Resolve("qwen3-embed-4b")
|
||||
if m.Source != "Qwen/Qwen3-Embedding-4B" {
|
||||
t.Errorf("the page was not read as the repository: %q", m.Source)
|
||||
}
|
||||
trt := embedModel()
|
||||
trt.Profiles[0] = Profile{Name: "gpu", Engine: "trtllm", GPUClass: "6000-24g", TP: 1, Source: "hf:a/b/m.gguf"}
|
||||
if _, err := New([]Model{trt}); err == nil || !strings.Contains(err.Error(), "not a .gguf") {
|
||||
t.Errorf("trtllm given a GGUF by its profile: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustJSON(t *testing.T, v any) string {
|
||||
t.Helper()
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
@@ -87,8 +87,8 @@ func (s *Service) Sources(ctx context.Context) (map[string][]catalog.Model, erro
|
||||
}
|
||||
out := map[string][]catalog.Model{}
|
||||
for _, m := range c.Models() {
|
||||
if m.Source != "" {
|
||||
out[m.Source] = append(out[m.Source], m)
|
||||
for _, src := range m.WeightSources() {
|
||||
out[src] = append(out[src], m)
|
||||
}
|
||||
for _, ad := range m.Adapters {
|
||||
out[ad.Source] = append(out[ad.Source], m)
|
||||
|
||||
@@ -284,6 +284,7 @@ var Reference = []Block{
|
||||
{Key: "engine", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "engine_version", Type: "string", Doc: "EngineVersion restricts the profile to nodes whose installed engine's\nversion starts with it: \"0.11\" takes vLLM 0.11.0 and 0.11.1, not\n0.10.2. Empty takes whatever version the node has.", Nested: ""},
|
||||
{Key: "gpu_class", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "source", Type: "string", Doc: "Source is the weights this profile loads, when not the model's: an\nembedding model served by vLLM from its repository on a card and by\nllama.cpp from a GGUF on CPUs is one model with two sources. Empty\nloads the model's. Updates and canaries move the model's source; a\nprofile with its own keeps it.", Nested: ""},
|
||||
{Key: "tp", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "pp", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "gpus_per_node", Type: "int", Doc: "GPUsPerNode defaults to TP for a single-node profile.", Nested: ""},
|
||||
@@ -464,6 +465,7 @@ var Reference = []Block{
|
||||
{Key: "name", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "priority", Type: "int", Doc: "Priority orders bands; higher is served first.", Nested: ""},
|
||||
{Key: "saturation_ceiling", Type: "float64", Doc: "SaturationCeiling is the pool saturation above which this band waits.\n1.0 means \"never hold this band back\".", Nested: ""},
|
||||
{Key: "queue_ttl", Type: "time.Duration", Doc: "QueueTTL, set, is how long this band's requests may wait instead of\nthe controller's: a band for bulk work — a knowledge-base import, a\nbackfill — whose requests had rather wait ten minutes behind people\nthan be turned away after thirty seconds.", Nested: ""},
|
||||
}},
|
||||
{Package: "admission", Name: "Config", Title: "admission.Config", Doc: "", Fields: []Field{
|
||||
{Key: "bands", Type: "[]Band", Doc: "", Nested: ""},
|
||||
@@ -499,6 +501,7 @@ var Reference = []Block{
|
||||
{Key: "name", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "priority", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "saturation_ceiling", Type: "float64", Doc: "", Nested: ""},
|
||||
{Key: "queue_ttl", Type: "time.Duration", Doc: "QueueTTL is how long this band's requests may wait for room; the\nadmission's queue_ttl unless set. Long for a band of bulk work, which\nhad rather wait behind people than be turned away.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Price", Title: "config.Price", Doc: "", Fields: []Field{
|
||||
{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
|
||||
@@ -511,3 +511,35 @@ func TestSaturationWithNoUsableEndpoints(t *testing.T) {
|
||||
t.Errorf("an empty pool should report full saturation, got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABandForBulkWorkWaitsLongerThanTheRest(t *testing.T) {
|
||||
// A knowledge-base import sends far more than the cluster can take at
|
||||
// once; turned away after the 30 s every band shared, its requests had
|
||||
// to be sent again. A band of its own may wait as long as it likes.
|
||||
s := &fixedSat{}
|
||||
s.set(1.0)
|
||||
cfg := testConfig()
|
||||
cfg.QueueTTL = 100 * time.Millisecond
|
||||
for i := range cfg.Bands {
|
||||
if cfg.Bands[i].Name == "bronze" {
|
||||
cfg.Bands[i].QueueTTL = 600 * time.Millisecond
|
||||
}
|
||||
}
|
||||
c := newController(t, cfg, s.fn)
|
||||
|
||||
start := time.Now()
|
||||
if _, err := c.Acquire(t.Context(), Request{Model: "m", Band: "gold"}); !errors.Is(err, ErrQueueTimeout) {
|
||||
t.Fatalf("gold: %v", err)
|
||||
}
|
||||
if d := time.Since(start); d > 400*time.Millisecond {
|
||||
t.Errorf("gold waited %v, past the controller's queue_ttl", d)
|
||||
}
|
||||
start = time.Now()
|
||||
_, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
|
||||
if !errors.Is(err, ErrQueueTimeout) || !strings.Contains(err.Error(), "600ms") {
|
||||
t.Fatalf("bronze: %v", err)
|
||||
}
|
||||
if d := time.Since(start); d < 500*time.Millisecond {
|
||||
t.Errorf("bronze gave up after %v, before its own queue_ttl", d)
|
||||
}
|
||||
}
|
||||
@@ -19,6 +19,11 @@ type Band struct {
|
||||
// SaturationCeiling is the pool saturation above which this band waits.
|
||||
// 1.0 means "never hold this band back".
|
||||
SaturationCeiling float64 `koanf:"saturation_ceiling"`
|
||||
// QueueTTL, set, is how long this band's requests may wait instead of
|
||||
// the controller's: a band for bulk work — a knowledge-base import, a
|
||||
// backfill — whose requests had rather wait ten minutes behind people
|
||||
// than be turned away after thirty seconds.
|
||||
QueueTTL time.Duration `koanf:"queue_ttl"`
|
||||
}
|
||||
|
||||
// Config configures the controller.
|
||||
@@ -86,6 +91,9 @@ func (c *Config) normalize() error {
|
||||
if b.SaturationCeiling <= 0 {
|
||||
b.SaturationCeiling = 1
|
||||
}
|
||||
if b.QueueTTL < 0 {
|
||||
return fmt.Errorf("band %q: queue_ttl %v is negative", b.Name, b.QueueTTL)
|
||||
}
|
||||
}
|
||||
if c.DefaultBand == "" {
|
||||
c.DefaultBand = c.Bands[len(c.Bands)-1].Name
|
||||
@@ -279,12 +287,16 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
|
||||
ErrQueueFull, c.cfg.QueueCapacityTotal)
|
||||
}
|
||||
|
||||
ttl := c.cfg.QueueTTL
|
||||
if band.QueueTTL > 0 {
|
||||
ttl = band.QueueTTL
|
||||
}
|
||||
w := &waiter{
|
||||
tenant: tenantKey(r),
|
||||
model: r.Model,
|
||||
ready: make(chan struct{}),
|
||||
enqueued: time.Now(),
|
||||
deadline: time.Now().Add(c.cfg.QueueTTL),
|
||||
deadline: time.Now().Add(ttl),
|
||||
}
|
||||
c.queues[idx].push(w)
|
||||
c.mu.Unlock()
|
||||
@@ -312,7 +324,7 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
|
||||
|
||||
case <-time.After(time.Until(w.deadline)):
|
||||
c.remove(idx, w, "ttl")
|
||||
return nil, fmt.Errorf("%w after %v", ErrQueueTimeout, c.cfg.QueueTTL)
|
||||
return nil, fmt.Errorf("%w after %v", ErrQueueTimeout, ttl)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -84,10 +84,12 @@ func TestARefusalBeforeDispatchSaysWhetherWaitingHelps(t *testing.T) {
|
||||
}{
|
||||
{"queue full", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueFull, time.Now())
|
||||
}, 503, ""},
|
||||
}, 503, "10"},
|
||||
// Busy, not broken: a client told when to come back comes back,
|
||||
// where one told nothing — llmgw-kb among them — gives up.
|
||||
{"queued too long", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueTimeout, time.Now())
|
||||
}, 504, ""},
|
||||
}, 504, "10"},
|
||||
{"displaced for a higher band", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrDisplaced, time.Now())
|
||||
}, 503, "5"},
|
||||
|
||||
@@ -62,7 +62,13 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
|
||||
r.m.observeRequest(r.metricModel(req.Model), "", 499, time.Since(start), nil, 0)
|
||||
return
|
||||
case errors.Is(err, admission.ErrQueueTimeout):
|
||||
// The cluster was busy for as long as the band may wait, not broken:
|
||||
// said so a client comes back. Without it llmgw-kb, which retries
|
||||
// only what says when to, counted every such passage as failed.
|
||||
status = http.StatusGatewayTimeout
|
||||
w.Header().Set("Retry-After", "10")
|
||||
case errors.Is(err, admission.ErrQueueFull):
|
||||
w.Header().Set("Retry-After", "10")
|
||||
case errors.Is(err, admission.ErrDisplaced):
|
||||
// Turned away for higher-priority work, not for good: said so the
|
||||
// client backs off and comes back rather than giving up.
|
||||
|
||||
@@ -435,6 +435,7 @@ func admissionConfig(rc config.Router) admission.Config {
|
||||
Name: b.Name,
|
||||
Priority: b.Priority,
|
||||
SaturationCeiling: b.SaturationCeiling,
|
||||
QueueTTL: b.QueueTTL,
|
||||
})
|
||||
}
|
||||
return admission.Config{
|
||||
|
||||
Reference in new issue
Block a user