Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
94eec557ad | ||
|
|
d5f02c0a4b |
No files matched your search
@@ -42,6 +42,28 @@ What v0.10.5 made settable from the command line is now on the pages too:
|
||||
critical model's card no longer refuses a replica that preemption would make
|
||||
room for.
|
||||
|
||||
### Models served outside the cluster
|
||||
|
||||
A catalog model can be an OpenAI-compatible API elsewhere: `external: {url,
|
||||
model, api_key_file}` and no profiles. Nothing is placed for it; the router
|
||||
sends its requests to the provider, rewritten to the provider's name for the
|
||||
model and with its key, read from the file on every publish so a rotated key is
|
||||
picked up. Named in another model's `fallbacks`, it is used only when the
|
||||
cluster has no capacity left. It is never scraped for metrics, and never reads
|
||||
as stale. Prices, budgets, limits, usage and key allowlists apply as to any
|
||||
model — and since its prompts leave the cluster, the allowlist is how to decide
|
||||
who may send them (docs/install.md). `ctl models` and the cluster page mark it
|
||||
external.
|
||||
|
||||
### Pool spillover
|
||||
|
||||
A model served from pools of different priority — H100 before A100, say — was
|
||||
routed to the preferred pool however full it was. Now a pool whose replicas are
|
||||
saturated, by the same measure admission uses (queue past its healthy depth, or
|
||||
KV cache past its ceiling), passes requests to the next pool that is not. When
|
||||
every pool is saturated the preferred one is kept: a pool that is just as full
|
||||
gains nothing.
|
||||
|
||||
### Displacement: a higher band is not refused while a lower one waits
|
||||
|
||||
Each band's queue was bounded, and nothing bounded them together, so a full
|
||||
|
||||
@@ -671,6 +671,40 @@ llmgw ctl explain-placement qwen2.5-coder-3b-instruct
|
||||
which names every candidate and why each was rejected, rather than leaving you
|
||||
to infer it.
|
||||
|
||||
### Models served outside the cluster
|
||||
|
||||
A model can be an OpenAI-compatible API elsewhere — a hosted provider, or
|
||||
another gateway — rather than replicas here. Nothing is placed for it; the
|
||||
router sends its requests on, with the provider's key and the provider's name
|
||||
for the model:
|
||||
|
||||
```yaml
|
||||
- name: cloud-mini
|
||||
source: external
|
||||
external:
|
||||
url: https://api.openai.com # without /v1
|
||||
model: gpt-4o-mini # what the provider calls it
|
||||
api_key_file: /etc/llmgw/secrets/openai.key
|
||||
```
|
||||
|
||||
Named in another model's `fallbacks`, it is used only when the cluster has no
|
||||
capacity left for that model:
|
||||
|
||||
```yaml
|
||||
- name: qwen2.5-coder-7b-instruct
|
||||
fallbacks: [cloud-mini]
|
||||
...
|
||||
```
|
||||
|
||||
The key is read from its file on the control plane, never stored in the
|
||||
catalog, and a key rotated in the file is used within seconds. A model whose key
|
||||
file cannot be read is not offered. Everything else applies as to any model:
|
||||
prices and budgets, rate limits, usage records, and key allowlists.
|
||||
|
||||
**Its prompts leave the cluster.** Decide who may send them: a key or project
|
||||
whose allowlist leaves the external model out cannot reach it, as a fallback or
|
||||
by name. The cluster page and `ctl models` mark it `external`.
|
||||
|
||||
### How many replicas, and how much each one carries
|
||||
|
||||
Replica count is changed while the cluster runs, without editing the
|
||||
|
||||
@@ -78,6 +78,9 @@ rather than repeated.
|
||||
| `pool` | `string` | [Endpoint](#endpoint) |
|
||||
| `weight` | `float64` | [Endpoint](#endpoint) |
|
||||
| `priority` | `int` | [Endpoint](#endpoint) |
|
||||
| `url` | `string` | [External](#external) |
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
| `contact` | `string` | [Landing](#landing) |
|
||||
| `notice` | `string` | [Landing](#landing) |
|
||||
| `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
|
||||
@@ -95,6 +98,7 @@ rather than repeated.
|
||||
| `scaling` | `Scaling` | [Model](#model) |
|
||||
| `class` | `string` | [Model](#model) |
|
||||
| `owner` | `string` | [Model](#model) |
|
||||
| `external` | `*External` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
| `routing_profile` | `string` | [Model](#model) |
|
||||
| `slo` | `SLO` | [Model](#model) |
|
||||
@@ -616,6 +620,28 @@ called. Empty means vLLM's names.
|
||||
|
||||
`int`
|
||||
|
||||
## External
|
||||
|
||||
### `url`
|
||||
|
||||
`string`
|
||||
|
||||
URL is the provider's base address, without /v1: https://api.openai.com.
|
||||
|
||||
### `model`
|
||||
|
||||
`string`
|
||||
|
||||
Model is what the provider calls it; requests are rewritten to it.
|
||||
|
||||
### `api_key_file`
|
||||
|
||||
`string`
|
||||
|
||||
APIKeyFile holds the provider's key, on the control plane. A file
|
||||
rather than the key itself, so the key is in neither the catalog, the
|
||||
store nor the audit log; empty for a provider that asks for none.
|
||||
|
||||
## Landing
|
||||
|
||||
### `contact`
|
||||
@@ -765,6 +791,20 @@ Owner is the part of the tenancy tree whose capacity quota this model
|
||||
counts against, written org:ID, team:ID or project:ID. Empty is shared:
|
||||
held by the cluster, counted against nobody.
|
||||
|
||||
### `external`
|
||||
|
||||
`*External`
|
||||
|
||||
External is a model served by someone else — an OpenAI-compatible API
|
||||
outside the cluster — rather than by replicas placed here. Such a model
|
||||
has no profiles and nothing is placed for it; the router sends its
|
||||
requests to the provider. Most useful named in another model's
|
||||
fallbacks: used only when the cluster has no capacity left. Its
|
||||
prompts leave the cluster, so which keys may name it is worth an
|
||||
allowlist.
|
||||
|
||||
Takes a [External](#external) block.
|
||||
|
||||
### `fallbacks`
|
||||
|
||||
`[]string`
|
||||
|
||||
@@ -254,6 +254,8 @@ func modelStatus(m adminapi.ModelView) string {
|
||||
switch {
|
||||
case m.Parked:
|
||||
return "parked"
|
||||
case m.External != "":
|
||||
return "external: " + m.External
|
||||
case m.HeldFor != "":
|
||||
return "gave room to " + m.HeldFor
|
||||
case m.Unplaceable:
|
||||
|
||||
@@ -10,16 +10,18 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
@@ -312,8 +314,11 @@ type ModelView struct {
|
||||
Warnings []string `json:"warnings,omitempty"`
|
||||
// Class is its capacity class; Preempted the replicas it lost to a higher
|
||||
// class; HeldFor the model it is giving room to now, if any.
|
||||
Class string `json:"class"`
|
||||
Owner string `json:"owner,omitempty"`
|
||||
Class string `json:"class"`
|
||||
Owner string `json:"owner,omitempty"`
|
||||
// External is where a model served outside the cluster goes, provider
|
||||
// host and its name for the model; empty for one served here.
|
||||
External string `json:"external,omitempty"`
|
||||
Preempted int `json:"preempted,omitempty"`
|
||||
HeldFor string `json:"held_for,omitempty"`
|
||||
}
|
||||
@@ -569,6 +574,7 @@ func (a *API) Models() []ModelView {
|
||||
Desired: h.Desired, Failures: h.Failures,
|
||||
Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
|
||||
Class: m.EffectiveClass(), Owner: m.Owner, Preempted: h.Preempted, HeldFor: h.HeldFor,
|
||||
External: externalLabel(m),
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
v.Profiles = append(v.Profiles, p.Name)
|
||||
@@ -1443,3 +1449,15 @@ func reservationWarnings(m catalog.Model, measuredGB float64) []string {
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// externalLabel is "api.openai.com · gpt-4o-mini" for a model served elsewhere.
|
||||
func externalLabel(m catalog.Model) string {
|
||||
if !m.IsExternal() {
|
||||
return ""
|
||||
}
|
||||
host := m.External.URL
|
||||
if u, err := url.Parse(host); err == nil && u.Host != "" {
|
||||
host = u.Host
|
||||
}
|
||||
return host + " · " + m.External.Model
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -160,8 +161,16 @@ type Model struct {
|
||||
// Owner is the part of the tenancy tree whose capacity quota this model
|
||||
// counts against, written org:ID, team:ID or project:ID. Empty is shared:
|
||||
// held by the cluster, counted against nobody.
|
||||
Owner string `json:"owner,omitempty" koanf:"owner"`
|
||||
Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
|
||||
Owner string `json:"owner,omitempty" koanf:"owner"`
|
||||
// External is a model served by someone else — an OpenAI-compatible API
|
||||
// outside the cluster — rather than by replicas placed here. Such a model
|
||||
// has no profiles and nothing is placed for it; the router sends its
|
||||
// requests to the provider. Most useful named in another model's
|
||||
// fallbacks: used only when the cluster has no capacity left. Its
|
||||
// prompts leave the cluster, so which keys may name it is worth an
|
||||
// allowlist.
|
||||
External *External `json:"external,omitempty" koanf:"external"`
|
||||
Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
|
||||
// RoutingProfile selects how the router scores endpoints for this model.
|
||||
RoutingProfile string `json:"routing_profile,omitempty" koanf:"routing_profile"`
|
||||
SLO SLO `json:"slo,omitempty" koanf:"slo"`
|
||||
@@ -305,7 +314,11 @@ func (m Model) validate() error {
|
||||
if m.Source == "" {
|
||||
return fmt.Errorf("model %q: source is required", m.Name)
|
||||
}
|
||||
if len(m.Profiles) == 0 {
|
||||
if m.External != nil {
|
||||
if err := m.External.validate(m.Name, len(m.Profiles)); err != nil {
|
||||
return err
|
||||
}
|
||||
} else if len(m.Profiles) == 0 {
|
||||
return fmt.Errorf("model %q: at least one profile is required", m.Name)
|
||||
}
|
||||
if m.Owner != "" && !validOwner(m.Owner) {
|
||||
@@ -564,3 +577,32 @@ func validOwner(o string) bool {
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// External is a provider outside the cluster that serves a model.
|
||||
type External struct {
|
||||
// URL is the provider's base address, without /v1: https://api.openai.com.
|
||||
URL string `json:"url" koanf:"url"`
|
||||
// Model is what the provider calls it; requests are rewritten to it.
|
||||
Model string `json:"model" koanf:"model"`
|
||||
// APIKeyFile holds the provider's key, on the control plane. A file
|
||||
// rather than the key itself, so the key is in neither the catalog, the
|
||||
// store nor the audit log; empty for a provider that asks for none.
|
||||
APIKeyFile string `json:"api_key_file,omitempty" koanf:"api_key_file"`
|
||||
}
|
||||
|
||||
func (e *External) validate(model string, profiles int) error {
|
||||
if profiles > 0 {
|
||||
return fmt.Errorf("model %q: an external model is served elsewhere and has no profiles", model)
|
||||
}
|
||||
u, err := url.Parse(e.URL)
|
||||
if err != nil || (u.Scheme != "https" && u.Scheme != "http") || u.Host == "" {
|
||||
return fmt.Errorf("model %q: external url %q is not an http(s) address", model, e.URL)
|
||||
}
|
||||
if strings.TrimSpace(e.Model) == "" {
|
||||
return fmt.Errorf("model %q: external model needs the name the provider knows it by", model)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// IsExternal reports whether a model is served outside the cluster.
|
||||
func (m Model) IsExternal() bool { return m.External != nil }
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -338,6 +339,34 @@ func (c *Controller) endpoints() []endpoint.Info {
|
||||
Priority: prof.Priority,
|
||||
})
|
||||
}
|
||||
return append(out, c.externalEndpoints()...)
|
||||
}
|
||||
|
||||
// externalEndpoints are the models served outside the cluster, as endpoints:
|
||||
// one each, with the provider's key read from its file every time, so a key
|
||||
// rotated on disk is used from the next publish on.
|
||||
func (c *Controller) externalEndpoints() []endpoint.Info {
|
||||
var out []endpoint.Info
|
||||
for _, m := range c.cat.Load().Models() {
|
||||
if !m.IsExternal() {
|
||||
continue
|
||||
}
|
||||
key := ""
|
||||
if f := m.External.APIKeyFile; f != "" {
|
||||
b, err := os.ReadFile(f) //nolint:gosec // the operator's own path
|
||||
if err != nil {
|
||||
c.log.Warn("an external model's key could not be read; not offered", "model", m.Name, "err", err)
|
||||
continue
|
||||
}
|
||||
key = strings.TrimSpace(string(b))
|
||||
}
|
||||
out = append(out, endpoint.Info{
|
||||
ID: "external:" + m.Name, URL: strings.TrimRight(m.External.URL, "/"),
|
||||
Models: append([]string{m.Name}, m.Aliases...), APIKey: key,
|
||||
Engine: endpoint.EngineExternal, Pool: "external", Weight: 1,
|
||||
UpstreamModel: m.External.Model,
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -345,7 +374,8 @@ func (c *Controller) endpoints() []endpoint.Info {
|
||||
func fingerprint(infos []endpoint.Info) string {
|
||||
var b strings.Builder
|
||||
for _, i := range infos {
|
||||
fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight, i.Priority)
|
||||
fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d|%s|%x;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight, i.Priority,
|
||||
i.UpstreamModel, sha256.Sum256([]byte(i.APIKey)))
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
package control
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
)
|
||||
|
||||
type recordSink struct {
|
||||
mu sync.Mutex
|
||||
infos []endpoint.Info
|
||||
times int
|
||||
}
|
||||
|
||||
func (s *recordSink) SetEndpoints(_ context.Context, infos []endpoint.Info) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.infos, s.times = infos, s.times+1
|
||||
}
|
||||
|
||||
type noDispatch struct{}
|
||||
|
||||
func (noDispatch) StartShard(context.Context, reconciler.ShardCommand) error { return nil }
|
||||
func (noDispatch) StopShard(context.Context, string, string) error { return nil }
|
||||
func (noDispatch) DrainShard(context.Context, string, string) error { return nil }
|
||||
|
||||
func TestAnExternalModelIsPublishedWithItsKeyAndNothingIsPlacedForIt(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
keyFile := filepath.Join(dir, "provider.key")
|
||||
if err := os.WriteFile(keyFile, []byte("sk-provider-1\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
models := []catalog.Model{
|
||||
{Name: "cloud", Source: "external", Aliases: []string{"gpt"}, Scaling: catalog.Scaling{Min: 3, Max: 3},
|
||||
External: &catalog.External{URL: "https://api.example.com/", Model: "gpt-4o-mini", APIKeyFile: keyFile}},
|
||||
{Name: "gone", Source: "external",
|
||||
External: &catalog.External{URL: "https://api.example.com", Model: "m", APIKeyFile: filepath.Join(dir, "missing")}},
|
||||
}
|
||||
cat, err := catalog.New(models)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sink := &recordSink{}
|
||||
c := New(DefaultConfig(), catalog.NewLive(cat), noDispatch{}, sink, slog.New(slog.DiscardHandler))
|
||||
c.Publish(t.Context())
|
||||
|
||||
if len(sink.infos) != 1 {
|
||||
t.Fatalf("published %+v; the model whose key is missing must not be offered", sink.infos)
|
||||
}
|
||||
e := sink.infos[0]
|
||||
if e.ID != "external:cloud" || e.URL != "https://api.example.com" || e.APIKey != "sk-provider-1" ||
|
||||
e.UpstreamModel != "gpt-4o-mini" || e.Engine != endpoint.EngineExternal || len(e.Models) != 2 {
|
||||
t.Errorf("endpoint %+v", e)
|
||||
}
|
||||
// A key rotated on disk is published on the next pass.
|
||||
_ = os.WriteFile(keyFile, []byte("sk-provider-2"), 0o600)
|
||||
c.Publish(t.Context())
|
||||
if sink.times != 2 || sink.infos[0].APIKey != "sk-provider-2" {
|
||||
t.Errorf("the rotated key was not published: %d pushes, key %q", sink.times, sink.infos[0].APIKey)
|
||||
}
|
||||
// Scaling of 3 or not, nothing is wanted of the cluster for a model
|
||||
// served elsewhere, and so it never reads as short or unplaceable.
|
||||
for _, h := range c.Reconciler().ModelHealth() {
|
||||
if h.Model == "cloud" && (h.Desired != 0 || h.Unplaceable) {
|
||||
t.Errorf("an external model wants replicas: %+v", h)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -94,6 +94,9 @@ func (r *Reconciler) desiredFor(m catalog.Model) int {
|
||||
// baseFor is what a model runs without load: its minimum, or for an on-demand
|
||||
// model one while somebody is asking and none once they stop.
|
||||
func (r *Reconciler) baseFor(m catalog.Model) int {
|
||||
if m.IsExternal() {
|
||||
return 0 // served elsewhere: nothing to place, whatever its scaling says
|
||||
}
|
||||
if !m.Scaling.OnDemand() {
|
||||
return m.Scaling.Min
|
||||
}
|
||||
|
||||
@@ -37,8 +37,16 @@ type Info struct {
|
||||
Weight float64
|
||||
// Priority expresses pool preference; higher pools are tried first.
|
||||
Priority int
|
||||
// UpstreamModel is what the endpoint calls the model, when that differs
|
||||
// from the catalog's name: an external provider's. Requests are rewritten
|
||||
// to it. Empty sends the name as the client did.
|
||||
UpstreamModel string
|
||||
}
|
||||
|
||||
// EngineExternal marks an endpoint that is a provider outside the cluster.
|
||||
// It publishes no metrics, so it is never scraped and never reads as stale.
|
||||
const EngineExternal = "external"
|
||||
|
||||
// Serves reports whether the replica serves the given model.
|
||||
func (i Info) Serves(model string) bool {
|
||||
return slices.Contains(i.Models, model)
|
||||
@@ -242,7 +250,7 @@ func (r *Registry) Snapshots(model string) []Snapshot {
|
||||
continue
|
||||
}
|
||||
var age time.Duration
|
||||
stale := true
|
||||
stale := l.info.Engine != EngineExternal
|
||||
if !l.scrapedAt.IsZero() {
|
||||
age = now.Sub(l.scrapedAt)
|
||||
stale = age > r.cfg.StaleAfter
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
)
|
||||
|
||||
func TestALocalModelWithNoCapacityFallsBackToAnExternalProvider(t *testing.T) {
|
||||
local := newFakeEngine(t, "local")
|
||||
local.status.Store(http.StatusServiceUnavailable)
|
||||
|
||||
var calls atomic.Int64
|
||||
var gotModel, gotAuth atomic.Value
|
||||
provider := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/metrics" {
|
||||
t.Error("the provider was scraped for metrics")
|
||||
}
|
||||
calls.Add(1)
|
||||
gotAuth.Store(r.Header.Get("Authorization"))
|
||||
var body map[string]any
|
||||
b, _ := io.ReadAll(r.Body)
|
||||
_ = json.Unmarshal(b, &body)
|
||||
gotModel.Store(fmt.Sprint(body["model"]))
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
fmt.Fprint(w, `{"model":"gpt-4o-mini","choices":[{"message":{"content":"from outside"}}],"usage":{"prompt_tokens":3,"completion_tokens":2}}`)
|
||||
}))
|
||||
t.Cleanup(provider.Close)
|
||||
|
||||
cfg := baseConfig(config.Endpoint{ID: "l", URL: local.srv.URL, Models: []string{"local"}})
|
||||
cloud := catalog.Model{Name: "cloud", Source: "external",
|
||||
External: &catalog.External{URL: provider.URL, Model: "gpt-4o-mini"}}
|
||||
cfg.Models = []catalog.Model{catModel("local", nil, "cloud"), cloud}
|
||||
rt := newTestRouter(t, cfg)
|
||||
rt.SetEndpoints(t.Context(), []endpoint.Info{
|
||||
{ID: "l", URL: local.srv.URL, Models: []string{"local"}},
|
||||
{ID: "external:cloud", URL: provider.URL, Models: []string{"cloud"}, APIKey: "provider-key",
|
||||
Engine: endpoint.EngineExternal, Pool: "external", Weight: 1, UpstreamModel: "gpt-4o-mini"},
|
||||
})
|
||||
|
||||
rec := post(t, rt.Handler(), map[string]any{"model": "local", "messages": []any{}}, nil)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if calls.Load() != 1 {
|
||||
t.Fatalf("the provider was called %d times", calls.Load())
|
||||
}
|
||||
if gotModel.Load() != "gpt-4o-mini" {
|
||||
t.Errorf("the provider was asked for %v, not its own name for the model", gotModel.Load())
|
||||
}
|
||||
if gotAuth.Load() != "Bearer provider-key" {
|
||||
t.Errorf("the provider got %v, not its key", gotAuth.Load())
|
||||
}
|
||||
if got := rec.Header().Get("x-llmgw-served-model"); got != "cloud" {
|
||||
t.Errorf("the client should be told it was served by cloud, got %q", got)
|
||||
}
|
||||
// Never scraped, and never stale for it.
|
||||
for _, s := range rt.reg.Snapshots("cloud") {
|
||||
if s.Stale {
|
||||
t.Error("an external endpoint reads as stale")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnExternalModelIsNotAModelWithProfiles(t *testing.T) {
|
||||
good := catalog.Model{Name: "cloud", Source: "external",
|
||||
External: &catalog.External{URL: "https://api.example.com", Model: "gpt-4o-mini"}}
|
||||
if _, err := catalog.New([]catalog.Model{good}); err != nil {
|
||||
t.Fatalf("a valid external model was refused: %v", err)
|
||||
}
|
||||
for name, m := range map[string]catalog.Model{
|
||||
"profiles": {Name: "x", Source: "external", External: &catalog.External{URL: "https://a.b", Model: "m"},
|
||||
Profiles: []catalog.Profile{{Name: "p", Engine: "llamacpp"}}},
|
||||
"no url": {Name: "x", Source: "external", External: &catalog.External{URL: "api.example.com", Model: "m"}},
|
||||
"no model": {Name: "x", Source: "external", External: &catalog.External{URL: "https://a.b"}},
|
||||
} {
|
||||
if _, err := catalog.New([]catalog.Model{m}); err == nil {
|
||||
t.Errorf("%s: accepted", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -984,3 +984,23 @@ func (r *Router) rateLimitHeaders(w http.ResponseWriter, p auth.Principal) {
|
||||
w.Header().Set("x-ratelimit-reset-tokens", reset)
|
||||
}
|
||||
}
|
||||
|
||||
// withModel is body with its model set to name, for an endpoint that knows the
|
||||
// model by another name — an external provider. The body unchanged when it is
|
||||
// not a JSON object, which the endpoint then refuses as it would have anyway.
|
||||
func withModel(body []byte, name string) []byte {
|
||||
var m map[string]json.RawMessage
|
||||
if err := json.Unmarshal(body, &m); err != nil {
|
||||
return body
|
||||
}
|
||||
encoded, err := json.Marshal(name)
|
||||
if err != nil {
|
||||
return body
|
||||
}
|
||||
m["model"] = encoded
|
||||
out, err := json.Marshal(m)
|
||||
if err != nil {
|
||||
return body
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -715,10 +715,14 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
|
||||
uctx, uspan := tracer().Start(ctx, "upstream", trace.WithSpanKind(trace.SpanKindClient),
|
||||
trace.WithAttributes(attribute.String("llmgw.endpoint", dec.Endpoint.ID),
|
||||
attribute.String("url.full", dec.Endpoint.URL+req.Path)))
|
||||
body := req.UpstreamBody
|
||||
if dec.Endpoint.UpstreamModel != "" {
|
||||
body = withModel(body, dec.Endpoint.UpstreamModel)
|
||||
}
|
||||
out, perr := r.px.Do(uctx, w, proxy.Request{
|
||||
URL: dec.Endpoint.URL + req.Path,
|
||||
APIKey: dec.Endpoint.APIKey,
|
||||
Body: req.UpstreamBody,
|
||||
Body: body,
|
||||
Header: withTraceparent(uctx, req.Header),
|
||||
Stream: req.Stream,
|
||||
SuppressUsageChunk: req.SuppressUsageChunk,
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
package scheduler
|
||||
|
||||
import (
|
||||
"sort"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/breaker"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
)
|
||||
@@ -51,34 +54,41 @@ func KVSaturated(ceiling float64) Filter {
|
||||
}
|
||||
}
|
||||
|
||||
// PoolPriority keeps only the endpoints of the highest-priority pool
|
||||
// (plan §7.3, the two-stage decision: pool first, then replica).
|
||||
// PoolPriority keeps the endpoints of one pool: the highest-priority one,
|
||||
// unless it is saturated and a lower one is not (plan §7.3, the two-stage
|
||||
// decision — pool first, then replica).
|
||||
//
|
||||
// Spillover arrives in phase 5, at which point this filter will fall through
|
||||
// to the next pool once the preferred one is saturated.
|
||||
func PoolPriority() Filter {
|
||||
return poolPriorityFilter{}
|
||||
// Saturated is what admission means by it, the same formula over the same
|
||||
// thresholds, so the two layers agree on when a pool is full: the H100 pool is
|
||||
// used until it is, and only then the A100 one — never the A100 because it
|
||||
// happens to be emptier. When every pool is saturated the preferred one is
|
||||
// kept: moving to a pool that is just as full buys nothing and loses whatever
|
||||
// made the first preferred.
|
||||
func PoolPriority(targetWaiting, kvCeiling float64) Filter {
|
||||
return poolPriorityFilter{t: admission.Thresholds{TargetWaitingPerReplica: targetWaiting, KVCeiling: kvCeiling}}
|
||||
}
|
||||
|
||||
type poolPriorityFilter struct{}
|
||||
type poolPriorityFilter struct{ t admission.Thresholds }
|
||||
|
||||
func (poolPriorityFilter) Name() string { return "pool-priority" }
|
||||
|
||||
func (poolPriorityFilter) Filter(_ *Ctx, eps []endpoint.Snapshot) []endpoint.Snapshot {
|
||||
func (f poolPriorityFilter) Filter(_ *Ctx, eps []endpoint.Snapshot) []endpoint.Snapshot {
|
||||
if len(eps) == 0 {
|
||||
return eps
|
||||
}
|
||||
best := eps[0].Priority
|
||||
for _, e := range eps[1:] {
|
||||
if e.Priority > best {
|
||||
best = e.Priority
|
||||
}
|
||||
}
|
||||
out := make([]endpoint.Snapshot, 0, len(eps))
|
||||
byPriority := map[int][]endpoint.Snapshot{}
|
||||
var order []int
|
||||
for _, e := range eps {
|
||||
if e.Priority == best {
|
||||
out = append(out, e)
|
||||
if _, ok := byPriority[e.Priority]; !ok {
|
||||
order = append(order, e.Priority)
|
||||
}
|
||||
byPriority[e.Priority] = append(byPriority[e.Priority], e)
|
||||
}
|
||||
sort.Sort(sort.Reverse(sort.IntSlice(order)))
|
||||
for _, p := range order {
|
||||
if admission.Saturation(byPriority[p], 0, f.t) < 1 {
|
||||
return byPriority[p]
|
||||
}
|
||||
}
|
||||
return out
|
||||
return byPriority[order[0]]
|
||||
}
|
||||
@@ -50,7 +50,7 @@ func Chat(c ProfileConfig) *Profile {
|
||||
c.normalize()
|
||||
return &Profile{
|
||||
Name: "chat",
|
||||
Filters: []Filter{BreakerOpen(), PoolPriority()},
|
||||
Filters: []Filter{BreakerOpen(), PoolPriority(c.TargetWaiting, c.KVCeiling)},
|
||||
Scorers: c.withPrefix([]WeightedScorer{
|
||||
{Session(c.SessionLoadFactor), 3.0},
|
||||
{Queue(c.TargetWaiting), 1.0},
|
||||
@@ -81,7 +81,7 @@ func Batch(c ProfileConfig) *Profile {
|
||||
c.normalize()
|
||||
return &Profile{
|
||||
Name: "batch",
|
||||
Filters: []Filter{BreakerOpen(), PoolPriority()},
|
||||
Filters: []Filter{BreakerOpen(), PoolPriority(c.TargetWaiting, c.KVCeiling)},
|
||||
Scorers: []WeightedScorer{
|
||||
{Queue(c.TargetWaiting), 3.0},
|
||||
{KV(c.KVCeiling), 1.0},
|
||||
@@ -99,7 +99,7 @@ func LongContext(c ProfileConfig) *Profile {
|
||||
c.normalize()
|
||||
return &Profile{
|
||||
Name: "long-context",
|
||||
Filters: []Filter{BreakerOpen(), PoolPriority()},
|
||||
Filters: []Filter{BreakerOpen(), PoolPriority(c.TargetWaiting, c.KVCeiling)},
|
||||
// Prefix matching earns more here than anywhere else: a long prompt is
|
||||
// mostly prefill, and prefill is what a cache hit skips.
|
||||
Scorers: c.withPrefix([]WeightedScorer{
|
||||
|
||||
@@ -177,7 +177,7 @@ func TestStaleEndpointIsPenalizedNotDropped(t *testing.T) {
|
||||
func TestPoolPriority(t *testing.T) {
|
||||
p := &Profile{
|
||||
Name: "t",
|
||||
Filters: []Filter{PoolPriority()},
|
||||
Filters: []Filter{PoolPriority(4, 0.9)},
|
||||
Scorers: []WeightedScorer{{Queue(4), 1}},
|
||||
Picker: MaxScore(),
|
||||
}
|
||||
@@ -189,7 +189,8 @@ func TestPoolPriority(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The lower-priority pool must not win despite being emptier (spillover is phase 5).
|
||||
// The lower-priority pool must not win for being emptier: the preferred
|
||||
// one is not saturated (3 waiting of a healthy 4).
|
||||
if d.Endpoint.ID != "h100" {
|
||||
t.Errorf("pool priority was ignored: %s", d.Endpoint.ID)
|
||||
}
|
||||
@@ -383,3 +384,36 @@ func TestBuiltinProfilesSchedule(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestASaturatedPoolSpillsToTheNext(t *testing.T) {
|
||||
p := &Profile{Name: "t", Filters: []Filter{PoolPriority(4, 0.9)},
|
||||
Scorers: []WeightedScorer{{Queue(4), 1}}, Picker: MaxScore()}
|
||||
h100 := func(w int) endpoint.Snapshot {
|
||||
return ep("h100", waiting(w), func(s *endpoint.Snapshot) { s.Priority = 100; s.Pool = "ib-a/h100" })
|
||||
}
|
||||
a100 := func(w int) endpoint.Snapshot {
|
||||
return ep("a100", waiting(w), func(s *endpoint.Snapshot) { s.Priority = 50; s.Pool = "ib-b/a100" })
|
||||
}
|
||||
pick := func(eps ...endpoint.Snapshot) string {
|
||||
d, err := p.Schedule(newCtx(Request{Model: "m"}), eps)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return d.Endpoint.ID
|
||||
}
|
||||
if got := pick(h100(8), a100(0)); got != "a100" {
|
||||
t.Errorf("the preferred pool at twice its healthy queue did not spill: %s", got)
|
||||
}
|
||||
if got := pick(h100(8), a100(6)); got != "h100" {
|
||||
t.Errorf("both pools saturated: the preferred one should be kept, got %s", got)
|
||||
}
|
||||
if got := pick(h100(4), a100(0)); got != "a100" {
|
||||
t.Errorf("at exactly its healthy limit a pool is full: %s", got)
|
||||
}
|
||||
// KV pressure saturates a pool as queue depth does.
|
||||
full := h100(0)
|
||||
full.Stats.KVCacheUsage = 0.95
|
||||
if got := pick(full, a100(0)); got != "a100" {
|
||||
t.Errorf("a pool out of KV cache did not spill: %s", got)
|
||||
}
|
||||
}
|
||||
@@ -96,6 +96,9 @@ func (p *Poller) Sync(ctx context.Context) {
|
||||
|
||||
want := make(map[string]endpoint.Info, len(infos))
|
||||
for _, in := range infos {
|
||||
if in.Engine == endpoint.EngineExternal {
|
||||
continue // a provider elsewhere publishes no /metrics
|
||||
}
|
||||
want[in.ID] = in
|
||||
}
|
||||
|
||||
|
||||
@@ -89,7 +89,7 @@
|
||||
<td{{if .HeldFor}} class="warn" title="gave room to {{.HeldFor}}"{{end}}>{{.Class}}{{if .Preempted}} <span class="note" title="replicas lost to a higher class">−{{.Preempted}}</span>{{end}}</td>
|
||||
<td{{with .Warnings}} class="warn" title="{{join .}}"{{end}}>{{if or .MemoryGB .CPUCores}}{{.MemoryGB}}G{{if .CPUCores}} · {{.CPUCores}}c{{end}}{{else}}—{{end}}
|
||||
{{- if .MeasuredMemoryGB}} <span class="note">{{printf "%.1f" .MeasuredMemoryGB}}G used</span>{{end}}{{if .Warnings}} ⚠{{end}}</td>
|
||||
<td>{{join .Profiles}}</td>
|
||||
<td>{{if .External}}<span class="tag" title="served outside the cluster; its prompts leave it">external</span> {{.External}}{{else}}{{join .Profiles}}{{end}}</td>
|
||||
{{if $.AnyFallbacks}}<td>{{join .Fallbacks}}</td>{{end}}
|
||||
{{/* Cut to the column, the whole sentence on hover: a row of its own for
|
||||
each description made the table twice as long as the catalog. */}}
|
||||
|
||||
+4
-2
@@ -1459,7 +1459,8 @@ The launch path is now built too:
|
||||
hits. It does nothing when the working set fits in one replica's cache,
|
||||
which is worth knowing rather than assuming.
|
||||
- [x] Fallback models
|
||||
- [ ] Pool spillover
|
||||
- [x] Pool spillover: a saturated pool passes requests to the next one that
|
||||
is not, by admission's own measure of saturation
|
||||
- [x] `ctl bench`: a load ramp that reports capacity as the last concurrency
|
||||
level meeting the SLO, plus the queue depth and KV usage observed there,
|
||||
so `weight`, `target_waiting` and `kv_ceiling` stop being guesses
|
||||
@@ -1497,7 +1498,8 @@ The launch path is now built too:
|
||||
checked against their digest, and reclaiming what nothing uses
|
||||
- [ ] Prefill/decode disaggregation (§12)
|
||||
- [ ] LoRA adapter routing (affinity to a replica that already has the adapter)
|
||||
- [ ] External provider support (as a fallback)
|
||||
- [x] External provider support: a catalog model served by an OpenAI-compatible
|
||||
API elsewhere, usable as a fallback, under the same allowlists and budgets
|
||||
- [x] Read-only dashboard on the admin listener (nodes, replicas, models,
|
||||
placement failures, usage) and a public landing page
|
||||
- [x] `/chat`: an in-browser playground for signed-in people, dispatched through
|
||||
|
||||
Reference in new issue
Block a user