Compare commits

...
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 78e41460f9 Scale and place by what a replica was measured to serve
Release / build (push) Successful in 1m8s
CI / test (push) Successful in 4m24s
Release / notes (push) Successful in 3s
Stored ctl bench results reach the reconciler: requests per replica past
the measured SLO concurrency scale a model up, a replica is taken back only
if the rest stay under 70% of theirs, and among profiles of equal priority
the most measured tokens per GPU or core is tried first. ctl model target
sizes a model for a throughput against the room there is.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 18:21:08 +02:00
Ilker ManapandClaude Opus 5.5 9dc1b62fdc Guarantee and limit capacity per part of the tenancy tree
Release / build (push) Successful in 1m8s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m51s
A model has an owner (org, team or project). Quotas guarantee a scope
GPUs, cores and memory and may limit it; what a scope holds is summed from
its models' replicas and those beneath it. A replica within its owner's
guarantee is not taken for another scope; what is held beyond a guarantee
is borrowed and taken back for a scope within its own. A limit stops a
model before it starts, and headroom stops there too. ctl quota and ctl
model owner set them; metrics and a CapacityAtLimit alert report them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 18:15:38 +02:00
34 changed files with 1978 additions and 31 deletions

No files matched your search

+48
View File
@@ -81,6 +81,54 @@ it gave way to has its replicas, so it does not take the room straight back.
cluster page shows the class and how many replicas were lost; the metric is
`llmgw_model_preempted_total{model,class}`, and every preemption is logged.
### Capacity quotas for the tenancy tree
An organization, team or project can be guaranteed GPUs, CPU cores and
memory, and limited to at most some amount of each. What it holds is what the
replicas of the models it owns hold — and those of the teams and projects
beneath it. `llmgw ctl model owner MODEL team:Acme/ml` says whose a model is;
one with no owner is shared and counts against nobody.
- **A guarantee is kept.** A replica within its owner's guarantee is never
taken for another scope's model, whatever the classes: a guarantee a higher
class elsewhere could take would not be one.
- **Beyond it is borrowing.** Free room is used; when a scope within its own
guarantee needs room, what others hold beyond theirs is taken back first,
whatever their class — except a critical model's minimum.
- **A limit is a ceiling.** A model whose owner would go past a limit anywhere
up the tree gets no further replica, and says which limit; headroom and the
mirror card stop there too.
`llmgw ctl quota set team:Acme/ml --gpus 8 --limit-gpus 16` (also `--cores`,
`--memory` and their limits), `llmgw ctl quota list` with what each scope holds,
`llmgw ctl quota clear`. Scopes are named by path or ID; a name that could mean
two is refused. Changes take effect within half a minute. Metrics:
`llmgw_capacity_holds`, `llmgw_capacity_guarantee` and `llmgw_capacity_limit`
by scope and resource; alert: CapacityAtLimit.
### Sized by what a replica was measured to serve
What `llmgw ctl bench MODEL --save` measured — the most requests at once one
replica served while meeting its SLO, and its throughput there — now drives
decisions rather than only being printed:
- **Scaling up** when requests per replica pass what was measured, not only
when the engines' queues fill: an engine takes requests into its slots long
after latency has gone.
- **Scaling down** only when the replicas left would stay under 70% of what
they were measured to hold, so a replica taken back does not put the rest
straight over.
- **Choosing a profile.** Among profiles of equal priority, the one measured
to serve the most per GPU — or per core — is tried first. Priority set by an
operator still wins.
- **Sizing.** `llmgw ctl model target MODEL --tokens-per-sec N` says how many
replicas that takes, how many run, and whether the rest fit, with the reason
when they do not; `--apply` sets it if they do. Headroom reports what one
more replica would serve.
A model never measured behaves exactly as before. Measurements are read from
the store every half minute.
### The mirror page in tabs
/admin/mirror was one long page: what the mirror holds, the Hugging Face
+11
View File
@@ -95,6 +95,17 @@ groups:
the cluster is too small for what is asked of it at these classes.
`llmgw ctl models` shows who it gave way to.
- alert: CapacityAtLimit
expr: llmgw_capacity_holds >= on(scope, resource) llmgw_capacity_limit
for: 15m
labels: { severity: info }
annotations:
summary: "{{ $labels.scope }} holds all the {{ $labels.resource }} it may"
description: >-
Its models get no further replicas, under load or not, until
something of theirs stops or the limit is raised: `llmgw ctl quota
list`, `llmgw ctl quota set`.
- alert: EngineInstallFailed
expr: llmgw_node_engine_install{state="failed"} == 1
for: 15m
+9
View File
@@ -93,6 +93,7 @@ rather than repeated.
| `profiles` | `[]Profile` | [Model](#model) |
| `scaling` | `Scaling` | [Model](#model) |
| `class` | `string` | [Model](#model) |
| `owner` | `string` | [Model](#model) |
| `fallbacks` | `[]string` | [Model](#model) |
| `routing_profile` | `string` | [Model](#model) |
| `slo` | `SLO` | [Model](#model) |
@@ -746,6 +747,14 @@ when there is not enough for both: critical, standard or opportunistic.
Empty is standard, or opportunistic for a model loaded on demand. See
EffectiveClass.
### `owner`
`string`
Owner is the part of the tenancy tree whose capacity quota this model
counts against, written org:ID, team:ID or project:ID. Empty is shared:
held by the cluster, counted against nobody.
### `fallbacks`
`[]string`
+2 -1
View File
@@ -137,7 +137,7 @@ func newCtlCmd() *cobra.Command {
// apply to it; it lives here because this is where operators look.
newBenchCmd(client, &asJSON),
newStatusCmd(client, &asJSON),
newModelCmd(client),
newModelCmd(client, &asJSON),
newScaleCmd(client),
newRestartCmd(client),
newRetryCmd(client),
@@ -146,6 +146,7 @@ func newCtlCmd() *cobra.Command {
newToolCheckCmd(),
newWeightsCmd(client, &asJSON),
newCapacityCmd(client, &asJSON),
newQuotaCmd(client, &asJSON),
newUsageCmd(client, &asJSON),
newCertCmd(),
)
+3 -2
View File
@@ -24,13 +24,14 @@ import (
// minutes of outage. For a fixed production set that is a reasonable trade; for
// a platform whose promise is that people can try things, it is the difference
// between trying one and not bothering.
func newModelCmd(client func() *ctlClient) *cobra.Command {
func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
c := &cobra.Command{
Use: "model",
Short: "Add or remove models while the cluster is running",
}
c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
newResourcesCmd(client), newClassCmd(client),
newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
newTargetCmd(client, asJSON),
newModelServeCmd(client))
return c
}
+230
View File
@@ -0,0 +1,230 @@
package cli
import (
"encoding/json"
"fmt"
"net/url"
"strconv"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
func newQuotaCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
c := &cobra.Command{
Use: "quota",
Short: "What parts of the tenancy tree are guaranteed of the cluster, and may hold",
Long: "A capacity quota belongs to an organization, team or project, and counts the\n" +
"GPUs, CPU cores and memory held by replicas of the models it owns (see\n" +
"`llmgw ctl model owner`), and of those its teams and projects own.\n\n" +
"The guarantee is kept for it: when it holds less and a model of its needs\n" +
"room, replicas other scopes hold beyond their own guarantees are stopped to\n" +
"make it — whatever their class. What it holds within its guarantee is\n" +
"never taken by another scope's model. Beyond it, it borrows: free room is\n" +
"used, and given back when asked for.\n\n" +
"The limit is the most it may hold, borrowing included; a model that would\n" +
"take it past is not given another replica.\n\n" +
"Scopes are written org:Acme, team:Acme/ml or project:Acme/ml/training, or\n" +
"with IDs. Changes take effect within half a minute.",
}
c.AddCommand(newQuotaListCmd(client, asJSON), newQuotaSetCmd(client), newQuotaClearCmd(client))
return c
}
func newQuotaListCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
return &cobra.Command{
Use: "list",
Short: "Every quota, with what its scope holds now",
RunE: func(cmd *cobra.Command, _ []string) error {
var qs []adminapi.QuotaView
if err := client().get(cmd.Context(), "/api/v1/quotas", &qs); err != nil {
return err
}
if *asJSON {
return json.NewEncoder(cmd.OutOrStdout()).Encode(qs)
}
if len(qs) == 0 {
fmt.Fprintln(cmd.OutOrStdout(), "No capacity quotas: everything shares the cluster by class alone.")
return nil
}
w := tab()
fmt.Fprintln(w, "SCOPE\tNAME\tGPUS\tCORES\tMEMORY GB")
for _, q := range qs {
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", q.Scope, q.Name,
quotaCell(q.Holds.GPUs, q.Guarantee.GPUs, q.Limit.GPUs),
quotaCell(q.Holds.CPUCores, q.Guarantee.CPUCores, q.Limit.CPUCores),
quotaCell(q.Holds.MemoryGB, q.Guarantee.MemoryGB, q.Limit.MemoryGB))
}
if err := w.Flush(); err != nil {
return err
}
fmt.Fprintln(cmd.OutOrStdout(), "\nEach cell: holds / guaranteed (at most limit). - is not governed.")
return nil
},
}
}
// quotaCell is "6 / 8 (≤ 16)": what is held, what is guaranteed, the limit.
func quotaCell(holds int, guarantee, limit *int) string {
if guarantee == nil && limit == nil {
return "-"
}
out := strconv.Itoa(holds) + " / "
if guarantee != nil {
out += strconv.Itoa(*guarantee)
} else {
out += "-"
}
if limit != nil {
out += fmt.Sprintf(" (≤ %d)", *limit)
}
return out
}
func newQuotaSetCmd(client func() *ctlClient) *cobra.Command {
var gpus, cores, mem, lgpus, lcores, lmem int
c := &cobra.Command{
Use: "set SCOPE [--gpus N] [--cores N] [--memory GB] [--limit-gpus N] ...",
Short: "Set a scope's guarantee and limit, replacing what it had",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
f := cmd.Flags()
pick := func(name string, v int) *int {
if !f.Changed(name) {
return nil
}
return &v
}
req := adminapi.QuotaRequest{Scope: args[0],
Guarantee: store.Resources{GPUs: pick("gpus", gpus), CPUCores: pick("cores", cores), MemoryGB: pick("memory", mem)},
Limit: store.Resources{GPUs: pick("limit-gpus", lgpus), CPUCores: pick("limit-cores", lcores), MemoryGB: pick("limit-memory", lmem)},
}
body, _ := json.Marshal(req)
var res map[string]string
if err := client().put(cmd.Context(), "/api/v1/quotas", body, &res); err != nil {
return err
}
fmt.Fprintf(cmd.OutOrStdout(), "%s: set; it takes effect within half a minute\n", res["scope"])
return nil
},
}
f := c.Flags()
f.IntVar(&gpus, "gpus", 0, "GPUs guaranteed")
f.IntVar(&cores, "cores", 0, "CPU cores guaranteed")
f.IntVar(&mem, "memory", 0, "memory guaranteed, in GB")
f.IntVar(&lgpus, "limit-gpus", 0, "the most GPUs it may hold")
f.IntVar(&lcores, "limit-cores", 0, "the most CPU cores it may hold")
f.IntVar(&lmem, "limit-memory", 0, "the most memory it may hold, in GB")
return c
}
func newQuotaClearCmd(client func() *ctlClient) *cobra.Command {
return &cobra.Command{
Use: "clear SCOPE",
Short: "Remove a scope's quota",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
var res map[string]string
if err := client().delete(cmd.Context(), "/api/v1/quotas/"+url.PathEscape(args[0]), &res); err != nil {
return err
}
fmt.Fprintf(cmd.OutOrStdout(), "%s: quota removed\n", res["removed"])
return nil
},
}
}
func newOwnerCmd(client func() *ctlClient) *cobra.Command {
return &cobra.Command{
Use: "owner <model> SCOPE|shared",
Short: "Set whose capacity quota a model counts against",
Long: "SCOPE is org:Acme, team:Acme/ml or project:Acme/ml/training. A shared\n" +
"model belongs to the cluster and counts against nobody's quota.",
Args: cobra.ExactArgs(2),
RunE: func(cmd *cobra.Command, args []string) error {
owner := args[1]
if owner == "shared" {
owner = ""
}
body, _ := json.Marshal(map[string]string{"owner": owner})
var res map[string]string
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/owner", body, &res); err != nil {
return err
}
if res["owner"] == "" {
fmt.Fprintf(cmd.OutOrStdout(), "%s is shared\n", res["model"])
} else {
fmt.Fprintf(cmd.OutOrStdout(), "%s counts against %s\n", res["model"], res["owner"])
}
return nil
},
}
}
func newTargetCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
var tps float64
var apply bool
c := &cobra.Command{
Use: "target <model> --tokens-per-sec N [--apply]",
Short: "How many replicas a model needs for a throughput, and whether they fit",
Long: "Sized from what one replica was measured to serve while meeting its SLO\n" +
"(`llmgw ctl bench MODEL --save`), on the profile its replicas run on.\n" +
"--apply makes that the number it always runs, if the cluster has room.",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
if tps <= 0 {
return fmt.Errorf("--tokens-per-sec is required")
}
var v adminapi.TargetView
path := "/api/v1/models/" + url.PathEscape(args[0]) + "/target?tokens_per_sec=" +
strconv.FormatFloat(tps, 'f', -1, 64)
if err := client().get(cmd.Context(), path, &v); err != nil {
return err
}
if *asJSON {
return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
}
out := cmd.OutOrStdout()
fmt.Fprintf(out, "%s: %.0f tokens/s needs %d replicas at %.1f each (measured on %s); %d running\n",
v.Model, v.TokPerSec, v.Needed, v.PerReplica, v.Profile, v.Current)
switch {
case v.More == 0:
fmt.Fprintln(out, "It already has them.")
case v.Fits:
fmt.Fprintf(out, "%d more, and there is room for them.\n", v.More)
default:
fmt.Fprintf(out, "%d more, and there is room for %d:\n", v.More, v.Room)
for _, b := range v.Blocked {
fmt.Fprintln(out, " "+b)
}
}
if !apply || v.More == 0 {
return nil
}
if !v.Fits {
return fmt.Errorf("not applied: the cluster has no room for them")
}
var models []adminapi.ModelView
if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
return err
}
top := v.Needed
for _, m := range models {
if m.Name == v.Model {
top = max(top, m.Max)
}
}
body, _ := json.Marshal(adminapi.ScaleRequest{Min: v.Needed, Max: top})
if err := client().post(cmd.Context(), "/api/v1/models/"+url.PathEscape(v.Model)+"/scale", body, nil); err != nil {
return err
}
fmt.Fprintf(out, "Applied: %s runs %d (up to %d under load).\n", v.Model, v.Needed, top)
return nil
},
}
c.Flags().Float64Var(&tps, "tokens-per-sec", 0, "the throughput to size for")
c.Flags().BoolVar(&apply, "apply", false, "set it as the number of replicas the model always runs")
return c
}
+8
View File
@@ -226,6 +226,8 @@ func newServerCmd() *cobra.Command {
// And what each group of the tenancy tree spent, which is
// the other half of a quota being reviewable.
api.SetUsage(st)
// Capacity quotas, set and read over this API.
api.SetQuotas(st)
}
api.Register(adminMux)
@@ -483,6 +485,12 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
if st := h.Store(); st != nil {
lead = control.NewLeadership(st, cfg.Control.LeaseTTL, log)
ctl.SetLeadership(lead)
// The tenancy tree's capacity quotas: what each part is
// guaranteed and may hold.
ctl.Reconciler().SetQuotas(control.QuotaPolicy(st, log))
// And what each model was measured to serve, for scaling and for
// choosing between profiles.
ctl.Reconciler().SetMeasurements(control.Measurements(st, log))
log.Info("competing to be the control plane that places work",
"owner", lead.Owner())
}
+9 -2
View File
@@ -42,6 +42,7 @@ type API struct {
models Models
capacity Capacity
usage Usage
quotas Quotas
ctl *control.Controller
}
@@ -58,6 +59,8 @@ func (a *API) Register(mux *http.ServeMux) {
mux.HandleFunc("DELETE /api/v1/models/{name}", a.removeModel)
mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
a.registerQuotas(mux)
mux.HandleFunc("GET /api/v1/models/{name}/target", a.target)
mux.HandleFunc("POST /api/v1/models/{name}/class", func(w http.ResponseWriter, r *http.Request) {
if a.models == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store")
@@ -304,6 +307,7 @@ type ModelView struct {
// Class is its capacity class; Preempted the replicas it lost to a higher
// class; HeldFor the model it is giving room to now, if any.
Class string `json:"class"`
Owner string `json:"owner,omitempty"`
Preempted int `json:"preempted,omitempty"`
HeldFor string `json:"held_for,omitempty"`
}
@@ -396,6 +400,8 @@ type HeadroomView struct {
Limited bool `json:"limited,omitempty"`
// Blocked says why one more does not fit, a line per node considered.
Blocked []string `json:"blocked,omitempty"`
// TokPerSec is what one more replica was measured to serve; 0 unmeasured.
TokPerSec float64 `json:"tok_per_sec,omitempty"`
}
// RejectionView is one rejected candidate.
@@ -552,7 +558,7 @@ func (a *API) Models() []ModelView {
Fallbacks: m.Fallbacks,
Desired: h.Desired, Failures: h.Failures,
Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
Class: m.EffectiveClass(), Preempted: h.Preempted, HeldFor: h.HeldFor,
Class: m.EffectiveClass(), Owner: m.Owner, Preempted: h.Preempted, HeldFor: h.HeldFor,
}
for _, p := range m.Profiles {
v.Profiles = append(v.Profiles, p.Name)
@@ -634,6 +640,7 @@ type Models interface {
Scale(ctx context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error)
Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
}
// ResourcesRequest sets what a model's profiles reserve of a node.
@@ -954,7 +961,7 @@ func (a *API) Headroom(model string, limit int) (HeadroomView, bool) {
return HeadroomView{}, false
}
return HeadroomView{Model: model, Current: h.Current, More: h.More, Nodes: h.Nodes,
Limited: h.Limited, Blocked: h.Blocked}, true
Limited: h.Limited, Blocked: h.Blocked, TokPerSec: h.TokPerSec}, true
}
func (a *API) Explain(model string) ExplainView {
+4
View File
@@ -258,6 +258,10 @@ type fakeModels struct {
err error
}
func (f *fakeModels) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
return catalog.Model{Name: name, Owner: owner}, f.err
}
func (f *fakeModels) SetClass(_ context.Context, name, class, _ string) (catalog.Model, error) {
return catalog.Model{Name: name, Class: class}, f.err
}
+237
View File
@@ -0,0 +1,237 @@
package adminapi
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// Quotas is where capacity quotas and the tenancy tree they name are kept.
// The store implements it.
type Quotas interface {
CapacityQuotas(ctx context.Context) ([]store.CapacityQuota, error)
PutCapacityQuota(ctx context.Context, q store.CapacityQuota) error
DeleteCapacityQuota(ctx context.Context, scope store.Scope, id string) error
ScopeParents(ctx context.Context) (map[string][]string, error)
AllOrgs(ctx context.Context) ([]store.Org, error)
AllTeams(ctx context.Context) ([]store.TeamPath, error)
AllProjects(ctx context.Context) ([]store.ProjectPath, error)
}
// SetQuotas supplies the quota store, which mounts the quota endpoints'
// behaviour; without it they answer 501.
func (a *API) SetQuotas(q Quotas) { a.quotas = q }
// QuotaView is one scope's quota with what it holds now.
type QuotaView struct {
// Scope is team:ID; Name is how people know it, "Acme / ml".
Scope string `json:"scope"`
Name string `json:"name"`
Guarantee store.Resources `json:"guarantee"`
Limit store.Resources `json:"limit"`
Holds quota.Amount `json:"holds"`
UpdatedBy string `json:"updated_by,omitempty"`
UpdatedAt time.Time `json:"updated_at"`
}
// QuotaRequest sets a scope's quota. Scope may be written with an ID or a
// name: org:Acme, team:Acme/ml, project:Acme/ml/training.
type QuotaRequest struct {
Scope string `json:"scope"`
Guarantee store.Resources `json:"guarantee"`
Limit store.Resources `json:"limit"`
}
func (a *API) registerQuotas(mux *http.ServeMux) {
mux.HandleFunc("GET /api/v1/quotas", func(w http.ResponseWriter, r *http.Request) {
if a.quotas == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store")
return
}
views, err := a.QuotaViews(r.Context())
if err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
writeJSON(w, http.StatusOK, views)
})
mux.HandleFunc("PUT /api/v1/quotas", func(w http.ResponseWriter, r *http.Request) {
if a.quotas == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store")
return
}
var req QuotaRequest
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<14)).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
return
}
scope, id, _, err := a.resolveScope(r.Context(), req.Scope)
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
if !req.Guarantee.Any() && !req.Limit.Any() {
writeError(w, http.StatusBadRequest, "a quota needs a guarantee or a limit; to remove one, DELETE it")
return
}
q := store.CapacityQuota{Scope: scope, ScopeID: id, Guarantee: req.Guarantee, Limit: req.Limit,
UpdatedBy: actorOf(r)}
if err := a.quotas.PutCapacityQuota(r.Context(), q); err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]string{"scope": quota.Key(scope, id)})
})
mux.HandleFunc("DELETE /api/v1/quotas/{scope}", func(w http.ResponseWriter, r *http.Request) {
if a.quotas == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store")
return
}
scope, id, _, err := a.resolveScope(r.Context(), r.PathValue("scope"))
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
if err := a.quotas.DeleteCapacityQuota(r.Context(), scope, id); err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]string{"removed": quota.Key(scope, id)})
})
mux.HandleFunc("POST /api/v1/models/{name}/owner", func(w http.ResponseWriter, r *http.Request) {
if a.models == nil || a.quotas == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store")
return
}
var req struct {
Owner string `json:"owner"`
}
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
return
}
owner := ""
if req.Owner != "" {
scope, id, _, err := a.resolveScope(r.Context(), req.Owner)
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
owner = quota.Key(scope, id)
}
m, err := a.models.SetOwner(r.Context(), r.PathValue("name"), owner, actorOf(r))
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]string{"model": m.Name, "owner": m.Owner})
})
}
// QuotaViews lists every quota with what its scope holds now.
func (a *API) QuotaViews(ctx context.Context) ([]QuotaView, error) {
qs, err := a.quotas.CapacityQuotas(ctx)
if err != nil {
return nil, err
}
parents, err := a.quotas.ScopeParents(ctx)
if err != nil {
return nil, err
}
names, err := a.scopeNames(ctx)
if err != nil {
return nil, err
}
usage := quota.New(qs, parents).Usage(a.ctl.Reconciler().Holdings())
out := make([]QuotaView, 0, len(qs))
for _, q := range qs {
key := quota.Key(q.Scope, q.ScopeID)
out = append(out, QuotaView{Scope: key, Name: names[key], Guarantee: q.Guarantee, Limit: q.Limit,
Holds: usage[key], UpdatedBy: q.UpdatedBy, UpdatedAt: q.UpdatedAt})
}
return out, nil
}
// scopeNames maps org:ID, team:ID and project:ID to their paths.
func (a *API) scopeNames(ctx context.Context) (map[string]string, error) {
out := map[string]string{}
orgs, err := a.quotas.AllOrgs(ctx)
if err != nil {
return nil, err
}
for _, o := range orgs {
out["org:"+o.ID] = o.Name
}
teams, err := a.quotas.AllTeams(ctx)
if err != nil {
return nil, err
}
for _, t := range teams {
out["team:"+t.ID] = t.OrgName + " / " + t.Name
}
projects, err := a.quotas.AllProjects(ctx)
if err != nil {
return nil, err
}
for _, p := range projects {
out["project:"+p.ID] = p.Path
}
return out, nil
}
// resolveScope reads org:X, team:X or project:X, where X is an ID or the
// scope's path — Acme, Acme/ml, Acme/ml/training — and returns its ID and
// path. A path that could mean two scopes is refused rather than guessed.
func (a *API) resolveScope(ctx context.Context, s string) (store.Scope, string, string, error) {
kind, what, ok := strings.Cut(strings.TrimSpace(s), ":")
scope := store.Scope(kind)
if !ok || what == "" || (scope != store.ScopeOrg && scope != store.ScopeTeam && scope != store.ScopeProject) {
return "", "", "", fmt.Errorf("%q is not org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT", s)
}
names, err := a.scopeNames(ctx)
if err != nil {
return "", "", "", err
}
norm := func(p string) string {
parts := strings.Split(p, "/")
for i := range parts {
parts[i] = strings.ToLower(strings.TrimSpace(parts[i]))
}
return strings.Join(parts, "/")
}
if name, ok := names[kind+":"+what]; ok {
return scope, what, name, nil
}
match := func(same func(name string) bool) []string {
var found []string
for key, name := range names {
if strings.HasPrefix(key, kind+":") && same(name) {
found = append(found, key)
}
}
return found
}
found := match(func(name string) bool { return norm(name) == norm(what) })
if len(found) == 0 {
// Its own name alone — ml rather than Acme/ml — when that is unique.
found = match(func(name string) bool {
parts := strings.Split(norm(name), "/")
return parts[len(parts)-1] == norm(what)
})
}
switch len(found) {
case 0:
return "", "", "", fmt.Errorf("there is no %s %q", kind, what)
case 1:
_, id, _ := strings.Cut(found[0], ":")
return scope, id, names[found[0]], nil
}
return "", "", "", fmt.Errorf("%q could be any of %d %ss; give its ID", what, len(found), kind)
}
+107
View File
@@ -0,0 +1,107 @@
package adminapi
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// fakeQuotaStore is Acme with teams ml and web, and Scania with another ml.
type fakeQuotaStore struct {
quotas []store.CapacityQuota
}
func (f *fakeQuotaStore) CapacityQuotas(context.Context) ([]store.CapacityQuota, error) {
return f.quotas, nil
}
func (f *fakeQuotaStore) PutCapacityQuota(_ context.Context, q store.CapacityQuota) error {
for i := range f.quotas {
if f.quotas[i].Scope == q.Scope && f.quotas[i].ScopeID == q.ScopeID {
f.quotas[i] = q
return nil
}
}
f.quotas = append(f.quotas, q)
return nil
}
func (f *fakeQuotaStore) DeleteCapacityQuota(_ context.Context, s store.Scope, id string) error {
for i, q := range f.quotas {
if q.Scope == s && q.ScopeID == id {
f.quotas = append(f.quotas[:i], f.quotas[i+1:]...)
}
}
return nil
}
func (f *fakeQuotaStore) ScopeParents(context.Context) (map[string][]string, error) {
return map[string][]string{"team:t-ml": {"org:o-acme"}, "team:t-web": {"org:o-acme"},
"team:t-sml": {"org:o-scania"}}, nil
}
func (f *fakeQuotaStore) AllOrgs(context.Context) ([]store.Org, error) {
return []store.Org{{ID: "o-acme", Name: "Acme"}, {ID: "o-scania", Name: "Scania"}}, nil
}
func (f *fakeQuotaStore) AllTeams(context.Context) ([]store.TeamPath, error) {
return []store.TeamPath{{ID: "t-ml", Name: "ml", OrgID: "o-acme", OrgName: "Acme"},
{ID: "t-web", Name: "web", OrgID: "o-acme", OrgName: "Acme"},
{ID: "t-sml", Name: "ml", OrgID: "o-scania", OrgName: "Scania"}}, nil
}
func (f *fakeQuotaStore) AllProjects(context.Context) ([]store.ProjectPath, error) { return nil, nil }
func TestAScopeIsNamedByPathOrIDAndNeverGuessed(t *testing.T) {
a, _ := setup(t, testModels())
a.SetQuotas(&fakeQuotaStore{})
for in, want := range map[string]string{
"org:Acme": "o-acme", "org:o-acme": "o-acme", "team:Acme/ml": "t-ml",
"team: acme / ML ": "t-ml", "team:Scania/ml": "t-sml", "team:web": "t-web",
} {
_, id, _, err := a.resolveScope(t.Context(), in)
if err != nil || id != want {
t.Errorf("%q: %q %v, expected %q", in, id, err, want)
}
}
for _, bad := range []string{"team:ml", "team:Acme/nope", "group:Acme", "Acme", "org:"} {
if _, _, _, err := a.resolveScope(t.Context(), bad); err == nil {
t.Errorf("%q was resolved", bad)
}
}
if _, _, _, err := a.resolveScope(t.Context(), "team:ml"); err == nil || !strings.Contains(err.Error(), "could be any of 2") {
t.Errorf("an ambiguous name should say so: %v", err)
}
}
func TestAQuotaIsSetListedAndCleared(t *testing.T) {
a, _ := setup(t, testModels())
qs := &fakeQuotaStore{}
a.SetQuotas(qs)
mux := http.NewServeMux()
a.Register(mux)
call := func(method, path, body string) *httptest.ResponseRecorder {
req := httptest.NewRequest(method, path, strings.NewReader(body))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("X-Llmgw-Actor", "ops")
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
return rec
}
if rec := call("PUT", "/api/v1/quotas", `{"scope":"team:Acme/ml","guarantee":{"gpus":8},"limit":{"gpus":16}}`); rec.Code != 200 {
t.Fatalf("set: %d %s", rec.Code, rec.Body)
}
if len(qs.quotas) != 1 || qs.quotas[0].ScopeID != "t-ml" || *qs.quotas[0].Guarantee.GPUs != 8 || qs.quotas[0].UpdatedBy != "ops" {
t.Errorf("stored %+v", qs.quotas)
}
rec := call("GET", "/api/v1/quotas", "")
var views []QuotaView
if err := json.Unmarshal(rec.Body.Bytes(), &views); err != nil || len(views) != 1 || views[0].Name != "Acme / ml" {
t.Errorf("list %d %s", rec.Code, rec.Body)
}
if rec := call("PUT", "/api/v1/quotas", `{"scope":"team:Acme/ml"}`); rec.Code != 400 {
t.Errorf("an empty quota was accepted: %d", rec.Code)
}
if rec := call("DELETE", "/api/v1/quotas/team:Acme%2Fml", ""); rec.Code != 200 || len(qs.quotas) != 0 {
t.Errorf("clear: %d %s %+v", rec.Code, rec.Body, qs.quotas)
}
}
+63
View File
@@ -0,0 +1,63 @@
package adminapi
import (
"fmt"
"math"
"net/http"
"strconv"
)
// TargetView is how many replicas a model needs to serve a throughput, from
// what one replica was measured to serve, and whether the cluster has room.
type TargetView struct {
Model string `json:"model"`
// TokPerSec is what was asked for; PerReplica what one replica was
// measured to serve at its SLO, on Profile.
TokPerSec float64 `json:"tok_per_sec"`
PerReplica float64 `json:"per_replica_tok_per_sec"`
Profile string `json:"profile"`
// Needed replicas in all; Current hold capacity now; More is how many of
// Needed are not running, and Room how many more fit.
Needed int `json:"needed"`
Current int `json:"current"`
More int `json:"more"`
Room int `json:"room"`
Fits bool `json:"fits"`
Blocked []string `json:"blocked,omitempty"`
}
// Target sizes a model for a throughput.
func (a *API) Target(model string, tokPerSec float64) (TargetView, error) {
meas, profile, ok := a.ctl.Reconciler().MeasuredFor(model)
if !ok || meas.TokPerSec <= 0 {
return TargetView{}, fmt.Errorf("%s has not been measured; run `llmgw ctl bench %s --save` first", model, model)
}
v := TargetView{Model: model, TokPerSec: tokPerSec, PerReplica: meas.TokPerSec, Profile: profile,
Needed: max(int(math.Ceil(tokPerSec/meas.TokPerSec)), 1)}
h, ok := a.Headroom(model, v.Needed)
if !ok {
return TargetView{}, fmt.Errorf("no model %q", model)
}
v.Current = h.Current
v.More = max(v.Needed-h.Current, 0)
v.Room = h.More
v.Fits = v.More <= h.More || h.Limited
if !v.Fits {
v.Blocked = h.Blocked
}
return v, nil
}
func (a *API) target(w http.ResponseWriter, r *http.Request) {
tps, err := strconv.ParseFloat(r.URL.Query().Get("tokens_per_sec"), 64)
if err != nil || tps <= 0 {
writeError(w, http.StatusBadRequest, "tokens_per_sec must be a positive number")
return
}
v, err := a.Target(r.PathValue("name"), tps)
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
writeJSON(w, http.StatusOK, v)
}
+30
View File
@@ -0,0 +1,30 @@
package adminapi
import (
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
)
func TestATargetIsSizedFromTheMeasurementAndTheRoom(t *testing.T) {
// Three 8-GPU nodes; one tp8 replica was measured at 500 tokens/s.
a, ctl := setup(t, testModels(), gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
if _, err := a.Target("llama", 1000); err == nil || !strings.Contains(err.Error(), "ctl bench llama --save") {
t.Errorf("an unmeasured model should say how to measure it: %v", err)
}
ctl.Reconciler().SetMeasurements(func() map[string]reconciler.Measured {
return map[string]reconciler.Measured{"llama/tp8": {MaxConcurrency: 16, TokPerSec: 500}}
})
v, err := a.Target("llama", 1200)
if err != nil {
t.Fatal(err)
}
if v.Needed != 3 || v.PerReplica != 500 || v.Profile != "tp8" || v.More != 3 || !v.Fits {
t.Errorf("1200 tok/s at 500 each is 3 replicas on 3 empty nodes: %+v", v)
}
v, _ = a.Target("llama", 2000)
if v.Needed != 4 || v.Fits || v.Room != 3 || len(v.Blocked) == 0 {
t.Errorf("a fourth does not fit on three nodes, and should say why: %+v", v)
}
}
+17 -1
View File
@@ -156,7 +156,11 @@ type Model struct {
// when there is not enough for both: critical, standard or opportunistic.
// Empty is standard, or opportunistic for a model loaded on demand. See
// EffectiveClass.
Class string `json:"class,omitempty" koanf:"class"`
Class string `json:"class,omitempty" koanf:"class"`
// Owner is the part of the tenancy tree whose capacity quota this model
// counts against, written org:ID, team:ID or project:ID. Empty is shared:
// held by the cluster, counted against nobody.
Owner string `json:"owner,omitempty" koanf:"owner"`
Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
// RoutingProfile selects how the router scores endpoints for this model.
RoutingProfile string `json:"routing_profile,omitempty" koanf:"routing_profile"`
@@ -304,6 +308,9 @@ func (m Model) validate() error {
if len(m.Profiles) == 0 {
return fmt.Errorf("model %q: at least one profile is required", m.Name)
}
if m.Owner != "" && !validOwner(m.Owner) {
return fmt.Errorf("model %q: owner %q is not org:ID, team:ID or project:ID", m.Name, m.Owner)
}
if !validClass(m.Class) {
return fmt.Errorf("model %q: class %q is not one of critical, standard, opportunistic", m.Name, m.Class)
}
@@ -548,3 +555,12 @@ func ClassRank(c string) int {
return 1
}
}
func validOwner(o string) bool {
for _, p := range []string{"org:", "team:", "project:"} {
if strings.HasPrefix(o, p) && len(o) > len(p) {
return true
}
}
return false
}
+3
View File
@@ -375,3 +375,6 @@ func EngineKeys(secret []byte) func(replicaID string) string {
return "llmgw-" + hex.EncodeToString(m.Sum(nil))[:40]
}
}
// QuotaUse is every capacity quota beside what its scope holds.
func (c *Controller) QuotaUse() []reconciler.QuotaUse { return c.rec.QuotaUse() }
+56 -7
View File
@@ -51,13 +51,16 @@ type HealthSource interface {
type Collector struct {
src Source
modelDesired *prometheus.Desc
modelFailures *prometheus.Desc
modelParked *prometheus.Desc
modelUnplaceable *prometheus.Desc
modelPreempted *prometheus.Desc
engineInstall *prometheus.Desc
leader *prometheus.Desc
modelDesired *prometheus.Desc
modelFailures *prometheus.Desc
modelParked *prometheus.Desc
modelUnplaceable *prometheus.Desc
modelPreempted *prometheus.Desc
capacityHolds *prometheus.Desc
capacityGuarantee *prometheus.Desc
capacityLimit *prometheus.Desc
engineInstall *prometheus.Desc
leader *prometheus.Desc
nodeUp *prometheus.Desc
nodeGPUs *prometheus.Desc
@@ -169,6 +172,15 @@ func New(src Source) *Collector {
modelUnplaceable: prometheus.NewDesc("llmgw_model_unplaceable",
"1 when a model is short of replicas and the last attempt found no node to place one on.",
[]string{"model"}, nil),
capacityHolds: prometheus.NewDesc("llmgw_capacity_holds",
"What a part of the tenancy tree holds of a resource, its own models' replicas and those beneath it.",
[]string{"scope", "resource"}, nil),
capacityGuarantee: prometheus.NewDesc("llmgw_capacity_guarantee",
"What a part of the tenancy tree is guaranteed of a resource.",
[]string{"scope", "resource"}, nil),
capacityLimit: prometheus.NewDesc("llmgw_capacity_limit",
"The most of a resource a part of the tenancy tree may hold.",
[]string{"scope", "resource"}, nil),
modelPreempted: prometheus.NewDesc("llmgw_model_preempted_total",
"Replicas of a model stopped to make room for a model of a higher capacity class.",
[]string{"model", "class"}, nil),
@@ -241,6 +253,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
c.gpuNVLink, c.gpuECC,
c.replicaState, c.replicaReady, c.replicaIdle,
c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable, c.modelPreempted,
c.capacityHolds, c.capacityGuarantee, c.capacityLimit,
c.engineInstall, c.leader,
} {
ch <- d
@@ -250,6 +263,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
// Collect implements prometheus.Collector.
func (c *Collector) Collect(ch chan<- prometheus.Metric) {
now := time.Now()
c.collectQuotas(ch)
loads := map[string]*inventory.Load{}
for _, n := range c.src.Nodes() {
@@ -469,3 +483,38 @@ func b2f(b bool) float64 {
}
return 0
}
// QuotaSource adds the tenancy tree's capacity quotas. Asked for by type
// assertion, like HealthSource.
type QuotaSource interface {
QuotaUse() []reconciler.QuotaUse
}
func (c *Collector) collectQuotas(ch chan<- prometheus.Metric) {
qs, ok := c.src.(QuotaSource)
if !ok {
return
}
for _, q := range qs.QuotaUse() {
for _, r := range []struct {
name string
holds int
gua, lim *int
}{
{"gpus", q.Holds.GPUs, q.Guarantee.GPUs, q.Limit.GPUs},
{"cpu_cores", q.Holds.CPUCores, q.Guarantee.CPUCores, q.Limit.CPUCores},
{"memory_gb", q.Holds.MemoryGB, q.Guarantee.MemoryGB, q.Limit.MemoryGB},
} {
if r.gua == nil && r.lim == nil {
continue
}
ch <- prometheus.MustNewConstMetric(c.capacityHolds, prometheus.GaugeValue, float64(r.holds), q.Scope, r.name)
if r.gua != nil {
ch <- prometheus.MustNewConstMetric(c.capacityGuarantee, prometheus.GaugeValue, float64(*r.gua), q.Scope, r.name)
}
if r.lim != nil {
ch <- prometheus.MustNewConstMetric(c.capacityLimit, prometheus.GaugeValue, float64(*r.lim), q.Scope, r.name)
}
}
}
}
+13
View File
@@ -255,6 +255,19 @@ func (s *Service) SetClass(ctx context.Context, name, class, by string) (catalog
return final, err
}
// SetOwner sets the part of the tenancy tree a model's capacity counts
// against, org:ID, team:ID or project:ID; empty makes it shared.
func (s *Service) SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error) {
final, err := s.update(ctx, name, by, "change its owner", func(m *catalog.Model) error {
m.Owner = owner
return nil
})
if err == nil {
s.log.Info("model owner set", "model", name, "owner", owner, "by", by)
}
return final, err
}
// update changes one model added at runtime and keeps the change.
//
// Built before anything is written, like Add: catalog.New is what applies the
+192
View File
@@ -0,0 +1,192 @@
// Package quota decides what the tenancy tree is guaranteed of the cluster
// and how much of it each part may hold.
//
// A scope — an organization, team or project — may be guaranteed GPUs, CPU
// cores and memory, and limited to at most some amount of each. What a scope
// holds is what the replicas of the models it owns, and of the models its
// teams and projects own, have reserved. Holding more than its guarantee is
// borrowing: allowed while the room is free, and taken back when another
// scope within its own guarantee needs it.
package quota
import (
"fmt"
"sort"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// Amount is how much of each resource something holds or needs.
type Amount struct {
GPUs, CPUCores, MemoryGB int
}
// Plus adds two amounts.
func (a Amount) Plus(b Amount) Amount {
return Amount{a.GPUs + b.GPUs, a.CPUCores + b.CPUCores, a.MemoryGB + b.MemoryGB}
}
// Over names the first resource in which a is above the governed amounts of
// l, with both figures; empty when it is above none.
func Over(a Amount, l store.Resources) string {
switch {
case l.GPUs != nil && a.GPUs > *l.GPUs:
return fmt.Sprintf("%d GPUs of %d", a.GPUs, *l.GPUs)
case l.CPUCores != nil && a.CPUCores > *l.CPUCores:
return fmt.Sprintf("%d cores of %d", a.CPUCores, *l.CPUCores)
case l.MemoryGB != nil && a.MemoryGB > *l.MemoryGB:
return fmt.Sprintf("%d GB of memory of %d", a.MemoryGB, *l.MemoryGB)
}
return ""
}
// Holding is one replica's hold on the cluster, for working out who holds
// what.
type Holding struct {
Replica string
Owner string
Amount Amount
Since time.Time
}
// Policy is the quotas and the tree they sit in, read at one moment.
type Policy struct {
quotas map[string]store.CapacityQuota
parents map[string][]string
}
// New builds a policy from stored quotas and each scope's parents.
func New(quotas []store.CapacityQuota, parents map[string][]string) *Policy {
p := &Policy{quotas: map[string]store.CapacityQuota{}, parents: parents}
for _, q := range quotas {
p.quotas[Key(q.Scope, q.ScopeID)] = q
}
return p
}
// Key writes a scope the way owners are written: team:ID.
func Key(s store.Scope, id string) string { return string(s) + ":" + id }
// Chain is an owner and every scope above it, nearest first.
func (p *Policy) Chain(owner string) []string {
if owner == "" {
return nil
}
return append([]string{owner}, p.parents[owner]...)
}
// Governing is the quotas along an owner's chain, nearest first.
func (p *Policy) Governing(owner string) []store.CapacityQuota {
if p == nil {
return nil
}
var out []store.CapacityQuota
for _, s := range p.Chain(owner) {
if q, ok := p.quotas[s]; ok {
out = append(out, q)
}
}
return out
}
// Quotas lists every quota.
func (p *Policy) Quotas() []store.CapacityQuota {
if p == nil {
return nil
}
out := make([]store.CapacityQuota, 0, len(p.quotas))
for _, q := range p.quotas {
out = append(out, q)
}
sort.Slice(out, func(i, j int) bool {
return Key(out[i].Scope, out[i].ScopeID) < Key(out[j].Scope, out[j].ScopeID)
})
return out
}
// Usage is what every scope holds, its own models' replicas and those of
// everything beneath it.
func (p *Policy) Usage(hs []Holding) map[string]Amount {
out := map[string]Amount{}
if p == nil {
return out
}
for _, h := range hs {
for _, s := range p.Chain(h.Owner) {
out[s] = out[s].Plus(h.Amount)
}
}
return out
}
// Borrowed is the replicas held beyond a guarantee: of each governed scope's
// replicas, oldest first, the ones that take it past what it is guaranteed.
// A replica beyond any guarantee along its chain is borrowed.
func (p *Policy) Borrowed(hs []Holding) map[string]bool {
out := map[string]bool{}
if p == nil || len(p.quotas) == 0 {
return out
}
sorted := append([]Holding(nil), hs...)
sort.Slice(sorted, func(i, j int) bool { return sorted[i].Since.Before(sorted[j].Since) })
for key, q := range p.quotas {
if !q.Guarantee.Any() {
continue
}
var sum Amount
for _, h := range sorted {
if !contains(p.Chain(h.Owner), key) {
continue
}
sum = sum.Plus(h.Amount)
if Over(sum, q.Guarantee) != "" {
out[h.Replica] = true
}
}
}
return out
}
// Guaranteed reports whether an owner, holding what it holds, is still within
// every guarantee along its chain with need added — so it may take back what
// others borrowed. An owner with no guarantee anywhere is never guaranteed.
func (p *Policy) Guaranteed(owner string, usage map[string]Amount, need Amount) bool {
gov := p.Governing(owner)
any := false
for _, q := range gov {
if !q.Guarantee.Any() {
continue
}
any = true
if Over(usage[Key(q.Scope, q.ScopeID)].Plus(need), q.Guarantee) != "" {
return false
}
}
return any
}
// AtLimit says which limit along an owner's chain need would take it past,
// empty when none.
func (p *Policy) AtLimit(owner string, usage map[string]Amount, need Amount) string {
for _, q := range p.Governing(owner) {
if !q.Limit.Any() {
continue
}
key := Key(q.Scope, q.ScopeID)
if over := Over(usage[key].Plus(need), q.Limit); over != "" {
return fmt.Sprintf("%s is at its capacity limit (%s)", key, strings.TrimSpace(over))
}
}
return ""
}
func contains(list []string, v string) bool {
for _, x := range list {
if x == v {
return true
}
}
return false
}
+83
View File
@@ -0,0 +1,83 @@
package quota
import (
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
func n(v int) *int { return &v }
// acme has two teams; ml is guaranteed 8 GPUs, the org as a whole 12 and
// no more than 16.
func policy() *Policy {
return New([]store.CapacityQuota{
{Scope: store.ScopeTeam, ScopeID: "ml", Guarantee: store.Resources{GPUs: n(8)}},
{Scope: store.ScopeOrg, ScopeID: "acme", Guarantee: store.Resources{GPUs: n(12)}, Limit: store.Resources{GPUs: n(16)}},
}, map[string][]string{
"team:ml": {"org:acme"}, "team:web": {"org:acme"},
"project:train": {"team:ml", "org:acme"},
})
}
func at(min int) time.Time { return time.Date(2026, 1, 1, 0, min, 0, 0, time.UTC) }
func TestAScopeHoldsWhatEverythingBeneathItHolds(t *testing.T) {
p := policy()
u := p.Usage([]Holding{
{Replica: "a", Owner: "project:train", Amount: Amount{GPUs: 4}},
{Replica: "b", Owner: "team:web", Amount: Amount{GPUs: 2, CPUCores: 3}},
{Replica: "c", Owner: "", Amount: Amount{GPUs: 8}}, // shared: nobody's
})
if u["team:ml"].GPUs != 4 || u["org:acme"].GPUs != 6 || u["org:acme"].CPUCores != 3 || u["project:train"].GPUs != 4 {
t.Errorf("usage %+v", u)
}
if _, ok := u[""]; ok {
t.Error("a shared model was counted against somebody")
}
}
func TestWhatGoesPastAGuaranteeIsBorrowedNewestFirst(t *testing.T) {
p := policy()
b := p.Borrowed([]Holding{
{Replica: "old", Owner: "team:ml", Amount: Amount{GPUs: 8}, Since: at(1)},
{Replica: "new", Owner: "team:ml", Amount: Amount{GPUs: 4}, Since: at(2)},
{Replica: "web", Owner: "team:web", Amount: Amount{GPUs: 4}, Since: at(3)},
})
// ml: 8 within its 8, then 4 past it. acme: 8 + 4 = 12 within, web's 4
// takes the org to 16, past its 12.
if b["old"] || !b["new"] || !b["web"] {
t.Errorf("borrowed %v", b)
}
}
func TestAGuaranteeIsOnlyForWhoHasOne(t *testing.T) {
p := policy()
u := map[string]Amount{"team:ml": {GPUs: 4}, "org:acme": {GPUs: 4}}
if !p.Guaranteed("project:train", u, Amount{GPUs: 4}) {
t.Error("8 of ml's 8 and 8 of acme's 12 is within both")
}
if p.Guaranteed("project:train", u, Amount{GPUs: 5}) {
t.Error("9 GPUs is past ml's 8")
}
if p.Guaranteed("", u, Amount{GPUs: 1}) || New(nil, nil).Guaranteed("team:ml", u, Amount{}) {
t.Error("an owner with no guarantee is not guaranteed anything")
}
}
func TestALimitStopsGrowthWhereverItIsSet(t *testing.T) {
p := policy()
u := map[string]Amount{"org:acme": {GPUs: 14}}
if why := p.AtLimit("team:web", u, Amount{GPUs: 2}); why != "" {
t.Errorf("16 of 16 is at the limit, not past it: %s", why)
}
why := p.AtLimit("team:web", u, Amount{GPUs: 4})
if !strings.Contains(why, "org:acme is at its capacity limit (18 GPUs of 16)") {
t.Errorf("the refusal should name the scope and the figures: %q", why)
}
if p.AtLimit("", u, Amount{GPUs: 99}) != "" {
t.Error("a shared model was limited")
}
}
+95
View File
@@ -0,0 +1,95 @@
package control
import (
"context"
"encoding/json"
"log/slog"
"sync"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// QuotaStore is where capacity quotas and the tenancy tree are kept.
type QuotaStore interface {
CapacityQuotas(ctx context.Context) ([]store.CapacityQuota, error)
ScopeParents(ctx context.Context) (map[string][]string, error)
}
// quotaRefresh is how stale the reconciler's view of quotas may get. A quota
// changed by an operator takes effect within this; the reconciler asks many
// times a pass and the store is not asked each time.
const quotaRefresh = 30 * time.Second
// QuotaPolicy returns what the reconciler asks for its quotas: the last
// policy read from the store, read again once it is older than quotaRefresh.
// A store that fails keeps the last good policy, so a database hiccup does
// not drop every guarantee and limit at once.
func QuotaPolicy(st QuotaStore, log *slog.Logger) func() *quota.Policy {
var mu sync.Mutex
var cur *quota.Policy
var at time.Time
return func() *quota.Policy {
mu.Lock()
defer mu.Unlock()
if cur != nil && time.Since(at) < quotaRefresh {
return cur
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
qs, err := st.CapacityQuotas(ctx)
if err == nil {
var parents map[string][]string
if parents, err = st.ScopeParents(ctx); err == nil {
cur, at = quota.New(qs, parents), time.Now()
return cur
}
}
log.Warn("could not read capacity quotas; keeping the last ones", "err", err)
at = time.Now() // not again on every call while the store is down
return cur
}
}
// MeasurementStore is where benchmark results are kept.
type MeasurementStore interface {
Capacities(ctx context.Context) ([]store.StoredCapacity, error)
}
// Measurements returns what the reconciler asks for benchmark results, keyed
// model/profile, read again once older than quotaRefresh; like QuotaPolicy it
// keeps the last good answer when the store fails.
func Measurements(st MeasurementStore, log *slog.Logger) func() map[string]reconciler.Measured {
var mu sync.Mutex
var cur map[string]reconciler.Measured
var at time.Time
return func() map[string]reconciler.Measured {
mu.Lock()
defer mu.Unlock()
if cur != nil && time.Since(at) < quotaRefresh {
return cur
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
at = time.Now()
rows, err := st.Capacities(ctx)
if err != nil {
log.Warn("could not read benchmark results; keeping the last ones", "err", err)
return cur
}
next := map[string]reconciler.Measured{}
for _, row := range rows {
var c bench.Capacity
if json.Unmarshal(row.Measurement, &c) != nil {
continue
}
next[row.Model+"/"+row.Profile] = reconciler.Measured{
MaxConcurrency: c.MaxConcurrencyAtSLO, TokPerSec: c.ThroughputTokPerSec}
}
cur = next
return cur
}
}
+37 -3
View File
@@ -157,8 +157,31 @@ type Load struct {
// the last one is still starting: on a small CPU fleet each replica is minutes
// of weights and gigabytes of memory, and a spike that is over by the time it
// loads should not leave three behind.
//
// With a benchmark stored for the profile a model runs on, its measured
// capacity is used too: more requests per replica than were measured to meet
// the SLO is busy, even with the engines' queues empty — an engine will take
// requests into its slots long after latency has gone — and a replica is only
// taken back when the rest would still be under scaleDownHeadroom of what they
// were measured to hold, so scaling down does not land on the edge.
func (r *Reconciler) RecordLoad(loads map[string]Load) {
now := r.now()
// The profile each model's replicas run on, for its measurement; read
// before the loop, which holds the lock.
running := map[string]string{}
r.mu.Lock()
for _, rep := range r.replicas {
if rep.State.Counts() {
running[rep.Model] = rep.Profile
}
}
r.mu.Unlock()
measured := map[string]Measured{}
for model, profile := range running {
if m, ok := r.measured(model, profile); ok {
measured[model] = m
}
}
for _, m := range r.cat.Load().Models() {
base := r.baseFor(m)
r.mu.Lock()
@@ -178,8 +201,13 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
l := loads[m.Name]
perReplica := float64(l.Waiting) / float64(max(l.Replicas, 1))
extra := r.extra[m.Name]
meas, hasMeas := measured[m.Name]
overMeasured := hasMeas &&
float64(l.Running+l.Waiting) > float64(meas.MaxConcurrency*max(l.Replicas, 1))
fitsInFewer := !hasMeas ||
float64(l.Running) <= scaleDownHeadroom*float64(meas.MaxConcurrency*max(l.Replicas-1, 0))
switch {
case perReplica >= r.cfg.ScaleUpWaiting:
case perReplica >= r.cfg.ScaleUpWaiting || overMeasured:
delete(r.quietSince, m.Name)
since, ok := r.busySince[m.Name]
if !ok {
@@ -190,9 +218,10 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
r.extra[m.Name] = extra + 1
r.busySince[m.Name] = now
r.log.Info("scaling up for load", "model", m.Name, "replicas", base+extra+1,
"waiting_per_replica", perReplica, "for", now.Sub(since).Round(time.Second))
"waiting_per_replica", perReplica, "in_flight", l.Running+l.Waiting,
"measured_per_replica", meas.MaxConcurrency, "for", now.Sub(since).Round(time.Second))
}
case l.Waiting == 0 && extra > 0:
case l.Waiting == 0 && extra > 0 && fitsInFewer:
delete(r.busySince, m.Name)
since, ok := r.quietSince[m.Name]
if !ok {
@@ -212,3 +241,8 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
r.mu.Unlock()
}
}
// scaleDownHeadroom is how full, of their measured capacity, the replicas left
// after a scale-down may be. Below one so that the load that was there a
// minute ago does not put them straight back over.
const scaleDownHeadroom = 0.7
+27 -2
View File
@@ -5,6 +5,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
)
// Headroom is how many more replicas of a model the cluster has room for now.
@@ -24,6 +25,9 @@ type Headroom struct {
// Blocked is why one more than More does not fit, a line per node that
// was considered. Empty when Limited.
Blocked []string
// TokPerSec is what one more replica was measured to serve, on the
// profile placement would give it; zero when it was never measured.
TokPerSec float64
}
// Headroom tries placing up to limit more replicas of a model on a copy of
@@ -52,11 +56,16 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
inv := r.inv.Clone()
sched := placement.New(inv)
pol := r.policy()
var usage map[string]quota.Amount
if pol != nil {
usage = pol.Usage(r.Holdings())
}
for h.More < limit {
res := sched.Place(m.PlacementProfiles(), avoid)
res := sched.Place(r.orderedProfiles(m), avoid)
if res.Placement == nil && len(avoid) > 0 {
// As scaleUp does: spreading is a preference, not a rule.
res = sched.Place(m.PlacementProfiles(), nil)
res = sched.Place(r.orderedProfiles(m), nil)
}
if res.Placement == nil {
for _, rej := range res.Rejections {
@@ -66,10 +75,26 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
h.Blocked = slices.Compact(h.Blocked)
return h, true
}
// A quota limit is as much an edge as a full node.
if pol != nil && m.Owner != "" {
need := placementNeed(m, res.Placement)
if why := pol.AtLimit(m.Owner, usage, need); why != "" {
h.Blocked = []string{why}
return h, true
}
for _, s := range pol.Chain(m.Owner) {
usage[s] = usage[s].Plus(need)
}
}
mem, cores := 0, 0
if prof, ok := m.ProfileByName(res.Placement.Profile.Name); ok {
mem, cores = prof.MinMemoryGB, cpuCores(prof)
}
if h.More == 0 {
if meas, ok := r.measured(m.Name, res.Placement.Profile.Name); ok {
h.TokPerSec = meas.TokPerSec
}
}
var nodes []string
for _, a := range res.Placement.Nodes {
inv.Allocate(a.NodeID, inventory.Allocation{
+108
View File
@@ -0,0 +1,108 @@
package reconciler
import (
"sort"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
)
// Measured is what one replica of a model on a profile was measured to serve
// while meeting its SLO, by `llmgw ctl bench --save`.
type Measured struct {
// MaxConcurrency is the most requests at once that still met the SLO.
MaxConcurrency int
// TokPerSec is the throughput at that concurrency.
TokPerSec float64
}
// SetMeasurements supplies benchmark results, keyed model/profile, read
// afresh each time it is called. Without them scaling goes by queue alone and
// profiles are chosen by priority alone.
func (r *Reconciler) SetMeasurements(f func() map[string]Measured) {
r.mu.Lock()
defer r.mu.Unlock()
r.measurements = f
}
// measured is a model's measurement on one profile.
func (r *Reconciler) measured(model, profile string) (Measured, bool) {
r.mu.Lock()
f := r.measurements
r.mu.Unlock()
if f == nil {
return Measured{}, false
}
m, ok := f()[model+"/"+profile]
return m, ok && m.MaxConcurrency > 0
}
// orderedProfiles is a model's profiles as placement should try them: by
// priority, and among equal priorities the one measured to serve the most per
// GPU — or per core, for a profile with none — first. Unmeasured profiles keep
// their catalog order behind measured ones of the same priority.
//
// Placement sorts by priority itself, so the order found here is written back
// as priorities, descending, for it to keep.
func (r *Reconciler) orderedProfiles(m catalog.Model) []placement.Profile {
ps := m.PlacementProfiles()
eff := make([]float64, len(ps))
for i, p := range ps {
meas, ok := r.measured(m.Name, p.Name)
if !ok {
continue
}
units := float64(max(p.GPUsPerNode, 0) * max(p.Nodes, 1))
if units == 0 {
units = float64(max(p.CPUCores, 1))
}
eff[i] = meas.TokPerSec / units
}
idx := make([]int, len(ps))
for i := range idx {
idx[i] = i
}
sort.SliceStable(idx, func(a, b int) bool {
pa, pb := ps[idx[a]], ps[idx[b]]
if pa.Priority != pb.Priority {
return pa.Priority > pb.Priority
}
return eff[idx[a]] > eff[idx[b]]
})
out := make([]placement.Profile, len(ps))
for rank, i := range idx {
out[rank] = ps[i]
out[rank].Priority = len(ps) - rank
}
return out
}
// MeasuredFor is the measurement that sizes a model: that of the profile its
// replicas run on, or, with none running, of the first profile placement
// would try that was measured.
func (r *Reconciler) MeasuredFor(model string) (Measured, string, bool) {
r.mu.Lock()
profile := ""
for _, rep := range r.replicas {
if rep.Model == model && rep.State.Counts() {
profile = rep.Profile
break
}
}
r.mu.Unlock()
if profile != "" {
if m, ok := r.measured(model, profile); ok {
return m, profile, true
}
}
cm, ok := r.cat.Load().Resolve(model)
if !ok {
return Measured{}, "", false
}
for _, p := range r.orderedProfiles(cm) {
if m, ok := r.measured(cm.Name, p.Name); ok {
return m, p.Name, true
}
}
return Measured{}, "", false
}
@@ -0,0 +1,96 @@
package reconciler
import (
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
)
func measure(h *harness, byKey map[string]Measured) {
h.r.SetMeasurements(func() map[string]Measured { return byKey })
}
func TestMoreRequestsThanWereMeasuredToMeetTheSLOIsBusy(t *testing.T) {
m := testModel(1)
m.Scaling.Max = 2
h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8))
h.r.Reconcile(t.Context())
h.markReady()
// Six in flight, nothing waiting at the engine: by the queue it is fine.
load := map[string]Load{"llama": {Replicas: 1, Running: 6}}
hold := func() {
h.r.RecordLoad(load)
h.advance(61 * time.Second)
h.r.RecordLoad(load)
h.r.Reconcile(t.Context())
}
hold()
if n := liveCount(h); n != 1 {
t.Fatalf("scaled up with no measurement and an empty queue: %d", n)
}
// Measured to meet its SLO at four at once: six is past it.
measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 4, TokPerSec: 200}})
hold()
if n := liveCount(h); n != 2 {
t.Errorf("%d replicas with 6 in flight against a measured 4", n)
}
}
func TestAReplicaIsTakenBackOnlyIfTheRestWouldHaveRoomToSpare(t *testing.T) {
m := testModel(1)
m.Scaling.Max = 2
h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8))
measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 4, TokPerSec: 200}})
h.r.Reconcile(t.Context())
h.markReady()
h.r.mu.Lock()
h.r.extra["llama"] = 1
h.r.mu.Unlock()
h.r.Reconcile(t.Context())
h.markReady()
if liveCount(h) != 2 {
t.Fatal("setup: not at two replicas")
}
quietFor := func(l Load) {
for range 3 {
h.r.RecordLoad(map[string]Load{"llama": l})
h.advance(DefaultConfig().ScaleDownAfter + time.Second)
h.r.RecordLoad(map[string]Load{"llama": l})
h.r.Reconcile(t.Context())
}
}
// Nothing waiting, but three running: one replica left would be at 3 of
// its 4, past 70%.
quietFor(Load{Replicas: 2, Running: 3})
if n := liveCount(h); n != 2 {
t.Errorf("scaled down to the edge of what one replica holds: %d", n)
}
quietFor(Load{Replicas: 2, Running: 2})
if n := liveCount(h); n != 1 {
t.Errorf("two running fits one replica's 70%% of 4; still %d", n)
}
}
func TestTheProfileMeasuredToServeMostPerGPUIsTriedFirst(t *testing.T) {
m := catalog.Model{Name: "llama", Source: "/models/llama", Scaling: catalog.Scaling{Min: 1, Max: 1},
Profiles: []catalog.Profile{
{Name: "tp4", GPUClass: "h100-80g", TP: 4},
{Name: "tp2", GPUClass: "h100-80g", TP: 2},
{Name: "tp8", GPUClass: "h100-80g", TP: 8, Priority: 10},
}}
h := newHarness(t, m, gpuNode("1", 8))
// tp4: 400/4 = 100 per GPU; tp2: 300/2 = 150. tp8 is unmeasured but has
// the higher priority, which an operator set and a benchmark does not
// overrule.
measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 8, TokPerSec: 400},
"llama/tp2": {MaxConcurrency: 6, TokPerSec: 300}})
cm, _ := h.r.cat.Load().Resolve("llama")
var order []string
for _, p := range h.r.orderedProfiles(cm) {
order = append(order, p.Name)
}
if len(order) != 3 || order[0] != "tp8" || order[1] != "tp2" || order[2] != "tp4" {
t.Errorf("order %v, expected tp8 (priority), then tp2 (150/GPU), then tp4 (100/GPU)", order)
}
}
+45 -8
View File
@@ -8,6 +8,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
)
// hold keeps a model that gave up room from taking it back.
@@ -62,9 +63,9 @@ func (r *Reconciler) preemptFor(ctx context.Context, m catalog.Model, avoid []st
}
trial.ReleaseReplica(v.ID)
chosen = append(chosen, v)
res := sched.Place(m.PlacementProfiles(), avoid)
res := sched.Place(r.orderedProfiles(m), avoid)
if res.Placement == nil && len(avoid) > 0 {
res = sched.Place(m.PlacementProfiles(), nil)
res = sched.Place(r.orderedProfiles(m), nil)
}
if res.Placement == nil {
continue
@@ -162,15 +163,29 @@ func (r *Reconciler) liveCount(model string) int {
}
// victims lists the replicas that may be stopped for m, in the order they
// should be: idle on-demand ones first, then what lower classes grew into
// under load, then the same within m's class, then lower classes' minimums;
// within each, the longest unused first.
// should be: idle on-demand ones first, then what other tenants borrowed past
// their guarantees when m's owner is within its own, then what lower classes
// grew into under load, then the same within m's class, then lower classes'
// minimums; within each, the longest unused first.
//
// A replica within its owner's guarantee is taken only for a model of the
// same owner: a guarantee that another tenant's higher class could take is
// not one.
func (r *Reconciler) victims(m catalog.Model) []victim {
cat := r.cat.Load()
now := r.now()
myRank := catalog.ClassRank(m.EffectiveClass())
belowMin := r.liveCount(m.Name) < m.Scaling.Min
pol := r.policy()
var borrowed map[string]bool
reclaim := false
if pol != nil {
hs := r.Holdings()
borrowed = pol.Borrowed(hs)
reclaim = pol.Guaranteed(m.Owner, pol.Usage(hs), profileNeed(m))
}
r.mu.Lock()
defer r.mu.Unlock()
@@ -201,11 +216,23 @@ func (r *Reconciler) victims(m catalog.Model) []victim {
case rank < myRank && other.Scaling.OnDemand() && idleFor(rep, now) >= r.cfg.EvictionGrace:
tier = 0
case rank < myRank && aboveMin:
tier = 1
case rank == myRank && aboveMin && belowMin:
tier = 2
case rank < myRank && class != catalog.ClassCritical:
case rank == myRank && aboveMin && belowMin:
tier = 3
case rank < myRank && class != catalog.ClassCritical:
tier = 4
}
if pol != nil && other.Owner != m.Owner {
switch {
case borrowed[rep.ID] && reclaim && (class != catalog.ClassCritical || aboveMin):
// Past its owner's guarantee, wanted by an owner within
// its own: taken back whatever the classes.
if tier != 0 {
tier = 1
}
case !borrowed[rep.ID] && guaranteed(pol, other.Owner):
continue // held within its owner's guarantee
}
}
if tier < 0 {
continue
@@ -247,3 +274,13 @@ func nodesOf(rep *Replica) map[string]struct{} {
}
return out
}
// guaranteed reports whether an owner has a guarantee anywhere up its tree.
func guaranteed(pol *quota.Policy, owner string) bool {
for _, q := range pol.Governing(owner) {
if q.Guarantee.Any() {
return true
}
}
return false
}
+105
View File
@@ -0,0 +1,105 @@
package reconciler
import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// SetQuotas supplies the tenancy tree's capacity quotas, read afresh each
// time it is called. Without it nothing is guaranteed or limited, and
// preemption goes by class alone.
func (r *Reconciler) SetQuotas(f func() *quota.Policy) {
r.mu.Lock()
defer r.mu.Unlock()
r.quotas = f
}
func (r *Reconciler) policy() *quota.Policy {
r.mu.Lock()
f := r.quotas
r.mu.Unlock()
if f == nil {
return nil
}
return f()
}
// Holdings is what every replica holds, and for whom.
func (r *Reconciler) Holdings() []quota.Holding {
cat := r.cat.Load()
r.mu.Lock()
defer r.mu.Unlock()
out := make([]quota.Holding, 0, len(r.replicas))
for _, rep := range r.replicas {
m, ok := cat.Resolve(rep.Model)
if !ok {
continue
}
prof, _ := m.ProfileByName(rep.Profile)
var a quota.Amount
for _, sh := range rep.Shards {
a = a.Plus(quota.Amount{GPUs: len(sh.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
}
out = append(out, quota.Holding{Replica: rep.ID, Owner: m.Owner, Amount: a, Since: rep.CreatedAt})
}
return out
}
// placementNeed is what a placement would hold.
func placementNeed(m catalog.Model, pl *placement.Placement) quota.Amount {
prof, _ := m.ProfileByName(pl.Profile.Name)
var a quota.Amount
for _, n := range pl.Nodes {
a = a.Plus(quota.Amount{GPUs: len(n.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
}
return a
}
// profileNeed is what a replica of a model's first profile would hold, for
// deciding before there is a placement.
func profileNeed(m catalog.Model) quota.Amount {
if len(m.Profiles) == 0 {
return quota.Amount{}
}
p := m.Profiles[0]
nodes := max(p.Nodes, 1)
gpus := p.GPUsPerNode
if gpus == 0 && !p.CPUOnly() {
gpus = max(p.TP, 1)
}
return quota.Amount{GPUs: gpus * nodes, CPUCores: cpuCores(p) * nodes, MemoryGB: p.MinMemoryGB * nodes}
}
// overLimit says why a placement would take its model's owner past a limit.
func (r *Reconciler) overLimit(m catalog.Model, pl *placement.Placement) string {
pol := r.policy()
if pol == nil || m.Owner == "" {
return ""
}
return pol.AtLimit(m.Owner, pol.Usage(r.Holdings()), placementNeed(m, pl))
}
// QuotaUse is one scope's quota beside what it holds.
type QuotaUse struct {
Scope string
Guarantee store.Resources
Limit store.Resources
Holds quota.Amount
}
// QuotaUse lists every quota with what its scope holds now.
func (r *Reconciler) QuotaUse() []QuotaUse {
pol := r.policy()
if pol == nil {
return nil
}
usage := pol.Usage(r.Holdings())
var out []QuotaUse
for _, q := range pol.Quotas() {
key := quota.Key(q.Scope, q.ScopeID)
out = append(out, QuotaUse{Scope: key, Guarantee: q.Guarantee, Limit: q.Limit, Holds: usage[key]})
}
return out
}
@@ -0,0 +1,81 @@
package reconciler
import (
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
func owned(m catalog.Model, owner string) catalog.Model { m.Owner = owner; return m }
func gpus(n int) store.Resources { return store.Resources{GPUs: &n} }
// Two teams in one org; each 8-GPU node holds one tp8 replica.
func withQuotas(h *harness, qs ...store.CapacityQuota) {
pol := quota.New(qs, map[string][]string{"team:ml": {"org:acme"}, "team:web": {"org:acme"}})
h.r.SetQuotas(func() *quota.Policy { return pol })
}
func TestATenantWithinItsGuaranteeTakesBackWhatAnotherBorrowed(t *testing.T) {
// web is guaranteed one node and borrowed the second; ml is guaranteed
// one node and wants it. Same class, and web's replica is at its minimum
// — by class alone nothing could be taken.
web := owned(gpuModel("web-chat", "", 2), "team:web")
ml := owned(gpuModel("ml-train", "", 1), "team:ml")
h := newFleetHarness(t, []catalog.Model{web, withMin(ml, 0)}, gpuNode("1", 8), gpuNode("2", 8))
withQuotas(h,
store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "web", Guarantee: gpus(8)},
store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "ml", Guarantee: gpus(8)})
h.settle(t, 4)
if replicasOf(h, "web-chat") != 2 {
t.Fatalf("setup: web did not borrow the second node: %+v", h.r.Replicas())
}
h.advance(h.r.cfg.MinLifetime + time.Minute)
cat, _ := catalog.New([]catalog.Model{web, ml})
h.r.cat.Store(cat)
h.settle(t, 5)
if replicasOf(h, "ml-train") != 1 || replicasOf(h, "web-chat") != 1 {
t.Errorf("the borrowed node was not taken back: web %d, ml %d",
replicasOf(h, "web-chat"), replicasOf(h, "ml-train"))
}
}
func TestAReplicaWithinItsGuaranteeIsNotTakenByAHigherClassElsewhere(t *testing.T) {
web := owned(gpuModel("web-chat", "", 1), "team:web")
vital := owned(gpuModel("vital", catalog.ClassCritical, 1), "team:ml")
h := fillThenWant(t, vital, web, gpuNode("1", 8))
withQuotas(h, store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "web", Guarantee: gpus(8)})
h.settle(t, 3)
if replicasOf(h, "web-chat") != 1 || replicasOf(h, "vital") != 0 {
t.Errorf("a guaranteed replica was preempted by another tenant's class: %+v", h.r.Replicas())
}
// Without the guarantee, critical takes it: the guarantee was the reason.
h2 := fillThenWant(t, vital, web, gpuNode("1", 8))
h2.settle(t, 3)
if replicasOf(h2, "vital") != 1 {
t.Error("setup: without a guarantee the critical model should have taken the node")
}
}
func TestALimitStopsAModelBeforeItStarts(t *testing.T) {
web := owned(gpuModel("web-chat", "", 2), "team:web")
h := newFleetHarness(t, []catalog.Model{web}, gpuNode("1", 8), gpuNode("2", 8))
withQuotas(h, store.CapacityQuota{Scope: store.ScopeOrg, ScopeID: "acme", Limit: gpus(8)})
h.settle(t, 4)
if replicasOf(h, "web-chat") != 1 {
t.Errorf("a second replica went past the org's 8-GPU limit: %+v", h.r.Replicas())
}
hs := health(h, "web-chat")
if !hs.Unplaceable || !strings.Contains(hs.Placement, "org:acme is at its capacity limit (16 GPUs of 8)") {
t.Errorf("the refusal should name the limit: %+v", hs)
}
room, _ := h.r.Headroom("web-chat", 5)
if room.More != 0 || len(room.Blocked) != 1 || !strings.Contains(room.Blocked[0], "capacity limit") {
t.Errorf("headroom should stop at the limit, not the empty node: %+v", room)
}
}
+18 -2
View File
@@ -15,6 +15,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
)
// ShardCommand is everything an agent needs to launch one shard.
@@ -270,6 +271,10 @@ type Reconciler struct {
preempting map[string][]string
held map[string]hold
preempted map[string]int
// quotas is the tenancy tree's capacity quotas as last read; nil is none.
quotas func() *quota.Policy
// measurements are benchmark results by model/profile; nil is none.
measurements func() map[string]Measured
wake chan struct{}
stop chan struct{}
@@ -781,12 +786,13 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
// allocated, and starting several at once from the same inventory snapshot
// would hand them the same GPUs.
func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) {
res := r.sched.Place(m.PlacementProfiles(), avoid)
profiles := r.orderedProfiles(m)
res := r.sched.Place(profiles, avoid)
if res.Placement == nil && len(avoid) > 0 {
// Spreading across failure domains is a preference, not a rule. If it
// is the only thing standing in the way, take the placement anyway:
// a second replica on one node still survives an engine crash.
res = r.sched.Place(m.PlacementProfiles(), nil)
res = r.sched.Place(profiles, nil)
}
r.mu.Lock()
@@ -811,6 +817,16 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
r.notePlacementFailure(m.Name, res.Explain())
return
}
if why := r.overLimit(m, res.Placement); why != "" {
// Recorded as the placement's answer, so the model reads as not
// placeable, and says why, wherever that is shown.
r.mu.Lock()
r.lastExplain[m.Name] = placement.Result{Rejections: []placement.Rejection{
{Profile: res.Placement.Profile.Name, Reason: why}}}
r.mu.Unlock()
r.notePlacementFailure(m.Name, why)
return
}
if err := r.startReplica(ctx, m, res.Placement); err != nil {
r.log.Error("starting replica failed", "model", m.Name, "err", err)
}
+3
View File
@@ -29,6 +29,9 @@ var copyOrder = []string{
"key_claims",
"sessions",
"limits",
// Beside limits, and like them keyed by scope with no foreign key: a
// quota on a scope that is later removed is inert, not an error.
"capacity_quotas",
"usage",
"models",
// No foreign key to models, deliberately: a measurement outlives the
@@ -0,0 +1,19 @@
-- Capacity a part of the tenancy tree is guaranteed, and the most it may hold.
--
-- Beside limits rather than in it: limits are counted on the request path, per
-- key, in requests and tokens; these are counted by the reconciler, per model,
-- in the GPUs, cores and memory its replicas hold. A NULL is a resource this
-- row does not govern: guaranteed nothing, or limited by nothing.
CREATE TABLE capacity_quotas (
scope_type TEXT NOT NULL,
scope_id TEXT NOT NULL,
gpus INTEGER,
cpu_cores INTEGER,
memory_gb INTEGER,
limit_gpus INTEGER,
limit_cpu_cores INTEGER,
limit_memory_gb INTEGER,
updated_at BIGINT NOT NULL,
updated_by TEXT NOT NULL,
PRIMARY KEY (scope_type, scope_id)
);
@@ -0,0 +1,19 @@
-- Capacity a part of the tenancy tree is guaranteed, and the most it may hold.
--
-- Beside limits rather than in it: limits are counted on the request path, per
-- key, in requests and tokens; these are counted by the reconciler, per model,
-- in the GPUs, cores and memory its replicas hold. A NULL is a resource this
-- row does not govern: guaranteed nothing, or limited by nothing.
CREATE TABLE capacity_quotas (
scope_type TEXT NOT NULL,
scope_id TEXT NOT NULL,
gpus INTEGER,
cpu_cores INTEGER,
memory_gb INTEGER,
limit_gpus INTEGER,
limit_cpu_cores INTEGER,
limit_memory_gb INTEGER,
updated_at BIGINT NOT NULL,
updated_by TEXT NOT NULL,
PRIMARY KEY (scope_type, scope_id)
);
+127
View File
@@ -0,0 +1,127 @@
package store
import (
"context"
"database/sql"
"fmt"
"time"
)
// Resources are amounts of the three things a replica holds of a cluster.
// Nil is not governed: nothing guaranteed, or nothing limited.
type Resources struct {
GPUs *int `json:"gpus,omitempty"`
CPUCores *int `json:"cpu_cores,omitempty"`
MemoryGB *int `json:"memory_gb,omitempty"`
}
// Any reports whether any resource is governed.
func (r Resources) Any() bool { return r.GPUs != nil || r.CPUCores != nil || r.MemoryGB != nil }
// CapacityQuota is what one part of the tenancy tree is guaranteed of the
// cluster, and the most it may hold.
type CapacityQuota struct {
Scope Scope `json:"scope"`
ScopeID string `json:"scope_id"`
// Guarantee is held for the scope: while it holds less, what others
// borrowed beyond their own guarantees is taken back for it.
Guarantee Resources `json:"guarantee"`
// Limit is the most it may hold, borrowing included.
Limit Resources `json:"limit"`
UpdatedAt time.Time `json:"updated_at"`
UpdatedBy string `json:"updated_by"`
}
// PutCapacityQuota sets a scope's quota, replacing what was there.
func (s *DB) PutCapacityQuota(ctx context.Context, q CapacityQuota) error {
if !validMembershipScope(q.Scope) || q.ScopeID == "" {
return fmt.Errorf("a capacity quota belongs to an organization, team or project")
}
for _, v := range []*int{q.Guarantee.GPUs, q.Guarantee.CPUCores, q.Guarantee.MemoryGB,
q.Limit.GPUs, q.Limit.CPUCores, q.Limit.MemoryGB} {
if v != nil && *v < 0 {
return fmt.Errorf("a quota cannot be negative")
}
}
if q.UpdatedAt.IsZero() {
q.UpdatedAt = time.Now()
}
_, err := s.db.ExecContext(ctx,
`INSERT INTO capacity_quotas (scope_type, scope_id, gpus, cpu_cores, memory_gb,
limit_gpus, limit_cpu_cores, limit_memory_gb, updated_at, updated_by)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT (scope_type, scope_id) DO UPDATE SET
gpus = excluded.gpus, cpu_cores = excluded.cpu_cores, memory_gb = excluded.memory_gb,
limit_gpus = excluded.limit_gpus, limit_cpu_cores = excluded.limit_cpu_cores,
limit_memory_gb = excluded.limit_memory_gb,
updated_at = excluded.updated_at, updated_by = excluded.updated_by`,
string(q.Scope), q.ScopeID,
nullInt(q.Guarantee.GPUs), nullInt(q.Guarantee.CPUCores), nullInt(q.Guarantee.MemoryGB),
nullInt(q.Limit.GPUs), nullInt(q.Limit.CPUCores), nullInt(q.Limit.MemoryGB),
q.UpdatedAt.UnixMilli(), q.UpdatedBy)
return err
}
// CapacityQuotas lists every quota.
func (s *DB) CapacityQuotas(ctx context.Context) ([]CapacityQuota, error) {
return queryAll(ctx, s.db,
`SELECT scope_type, scope_id, gpus, cpu_cores, memory_gb,
limit_gpus, limit_cpu_cores, limit_memory_gb, updated_at, updated_by
FROM capacity_quotas ORDER BY scope_type, scope_id`, nil,
func(rows *sql.Rows) (CapacityQuota, error) {
var q CapacityQuota
var scope string
var g [6]sql.NullInt64
var at int64
err := rows.Scan(&scope, &q.ScopeID, &g[0], &g[1], &g[2], &g[3], &g[4], &g[5], &at, &q.UpdatedBy)
q.Scope = Scope(scope)
q.Guarantee = Resources{GPUs: intOf(g[0]), CPUCores: intOf(g[1]), MemoryGB: intOf(g[2])}
q.Limit = Resources{GPUs: intOf(g[3]), CPUCores: intOf(g[4]), MemoryGB: intOf(g[5])}
q.UpdatedAt = time.UnixMilli(at)
return q, err
})
}
// DeleteCapacityQuota removes a scope's quota.
func (s *DB) DeleteCapacityQuota(ctx context.Context, scope Scope, id string) error {
_, err := s.db.ExecContext(ctx,
`DELETE FROM capacity_quotas WHERE scope_type = ? AND scope_id = ?`, string(scope), id)
return err
}
// ScopeParents maps every team and project to the scopes above it, nearest
// first: a project to its team and organization, a team to its organization.
// Keys and values are written scope:id.
func (s *DB) ScopeParents(ctx context.Context) (map[string][]string, error) {
out := map[string][]string{}
teams, err := s.AllTeams(ctx)
if err != nil {
return nil, err
}
orgOf := map[string]string{}
for _, t := range teams {
orgOf[t.ID] = t.OrgID
out["team:"+t.ID] = []string{"org:" + t.OrgID}
}
type pt struct{ project, team string }
rows, err := queryAll(ctx, s.db, `SELECT id, team_id FROM projects`, nil,
func(r *sql.Rows) (pt, error) {
var p pt
return p, r.Scan(&p.project, &p.team)
})
if err != nil {
return nil, err
}
for _, p := range rows {
out["project:"+p.project] = []string{"team:" + p.team, "org:" + orgOf[p.team]}
}
return out, nil
}
func intOf(n sql.NullInt64) *int {
if !n.Valid {
return nil
}
v := int(n.Int64)
return &v
}
+63
View File
@@ -0,0 +1,63 @@
package store
import "testing"
func ip(v int) *int { return &v }
func TestAQuotaKeepsWhatItGovernsAndNothingElse(t *testing.T) {
s := tenancyStore(t)
org, _ := s.CreateOrg(t.Context(), "Acme")
q := CapacityQuota{Scope: ScopeOrg, ScopeID: org, UpdatedBy: "ops",
Guarantee: Resources{GPUs: ip(8)}, Limit: Resources{GPUs: ip(16), CPUCores: ip(0)}}
if err := s.PutCapacityQuota(t.Context(), q); err != nil {
t.Fatal(err)
}
all, err := s.CapacityQuotas(t.Context())
if err != nil || len(all) != 1 {
t.Fatalf("%v %+v", err, all)
}
got := all[0]
if *got.Guarantee.GPUs != 8 || got.Guarantee.CPUCores != nil || got.Guarantee.MemoryGB != nil {
t.Errorf("guarantee %+v: only GPUs were set", got.Guarantee)
}
// Zero is a limit of nothing, not an absent one.
if *got.Limit.GPUs != 16 || got.Limit.CPUCores == nil || *got.Limit.CPUCores != 0 {
t.Errorf("limit %+v", got.Limit)
}
q.Guarantee = Resources{MemoryGB: ip(64)}
if err := s.PutCapacityQuota(t.Context(), q); err != nil {
t.Fatal(err)
}
all, _ = s.CapacityQuotas(t.Context())
if len(all) != 1 || all[0].Guarantee.GPUs != nil || *all[0].Guarantee.MemoryGB != 64 {
t.Errorf("a second put should replace the first: %+v", all)
}
if err := s.DeleteCapacityQuota(t.Context(), ScopeOrg, org); err != nil {
t.Fatal(err)
}
if all, _ = s.CapacityQuotas(t.Context()); len(all) != 0 {
t.Error("the quota survived its deletion")
}
if err := s.PutCapacityQuota(t.Context(), CapacityQuota{Scope: ScopeOrg, ScopeID: org,
Guarantee: Resources{GPUs: ip(-1)}}); err == nil {
t.Error("a negative quota was stored")
}
}
func TestEveryScopeKnowsWhatIsAboveIt(t *testing.T) {
s := tenancyStore(t)
org, _ := s.CreateOrg(t.Context(), "Acme")
team, _ := s.CreateTeam(t.Context(), org, "ml")
proj, _ := s.CreateProject(t.Context(), team, "training")
p, err := s.ScopeParents(t.Context())
if err != nil {
t.Fatal(err)
}
want := []string{"team:" + team, "org:" + org}
if got := p["project:"+proj]; len(got) != 2 || got[0] != want[0] || got[1] != want[1] {
t.Errorf("project's parents %v, expected %v", got, want)
}
if got := p["team:"+team]; len(got) != 1 || got[0] != "org:"+org {
t.Errorf("team's parents %v", got)
}
}
+10 -3
View File
@@ -1463,8 +1463,8 @@ The launch path is now built too:
- [x] `ctl bench`: a load ramp that reports capacity as the last concurrency
level meeting the SLO, plus the queue depth and KV usage observed there,
so `weight`, `target_waiting` and `kv_ceiling` stop being guesses
- [ ] `ctl plan <model> --target-rps N`, which needs capacity profiles stored
rather than printed
- [x] Sizing from stored capacity: `ctl model target <model> --tokens-per-sec N`,
and the autoscaler and profile choice use what `ctl bench --save` measured
- [x] OTel tracing: an inference span with admission, schedule and each
upstream attempt under it, over OTLP/HTTP when `tracing.endpoint` is
set, and the traceparent carried on to the engine
@@ -1480,7 +1480,14 @@ The launch path is now built too:
- [ ] Admission displacement (dropping requests from lower bands)
- [x] Autoscaling on load: a replica added while requests wait, up to a
model's max, and taken back after ten quiet minutes
- [ ] Priority-based preemption
- [x] Priority-based preemption: capacity classes (critical, standard,
opportunistic), preemption tried on a copy of the inventory first
- [x] Capacity quotas per organization, team and project: guarantees kept,
borrowing taken back, limits enforced
- [x] CPU cores reserved like memory, and memory required of CPU models
- [ ] Rack power, temperature and cooling as placement signals (a facility
source; not yet built)
- [ ] Defragmentation for large tensor-parallel replicas across NVLink domains
- [ ] Control plane HA (Raft)
- [x] Weight downloading and cache management: the mirror, agent fetches
checked against their digest, and reclaiming what nothing uses