Compare commits

...
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 9dc1b62fdc Guarantee and limit capacity per part of the tenancy tree
Release / build (push) Successful in 1m8s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m51s
A model has an owner (org, team or project). Quotas guarantee a scope
GPUs, cores and memory and may limit it; what a scope holds is summed from
its models' replicas and those beneath it. A replica within its owner's
guarantee is not taken for another scope; what is held beyond a guarantee
is borrowed and taken back for a scope within its own. A limit stops a
model before it starts, and headroom stops there too. ctl quota and ctl
model owner set them; metrics and a CapacityAtLimit alert report them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 18:15:38 +02:00
Ilker ManapandClaude Opus 5.5 ac24e204a6 Add capacity classes and preempt lower ones to place a model
Release / build (push) Successful in 1m7s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m46s
A model's class (critical, standard, opportunistic; unset follows its
scaling) decides what may be stopped to make room for it: a lower class
for a higher, and within a class only what another grew into under load,
for a model below its minimum. Victims are tried on a copy of the
inventory and stopped only if they make the room. Preempted replicas are
drained with a short grace and their model is held back until the room is
used. ctl model resources and class move under ctl model.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 18:02:59 +02:00
32 changed files with 2185 additions and 18 deletions

No files matched your search

+57
View File
@@ -49,6 +49,63 @@ holds of a node is reserved, not assumed.
smallest node the model fits on — beside the memory they already estimate.
Running replicas keep their old figures until restarted.
The commands are `llmgw ctl model resources` and `llmgw ctl model class`; in
rc.7 the first was briefly `llmgw ctl resources`.
### Capacity classes, and preemption
A model has a class — `critical`, `standard` or `opportunistic` — saying what
its capacity is worth against another's when there is not enough for both.
Unset, an on-demand model is opportunistic and every other model standard,
which is how the catalog behaved before. `llmgw ctl model class MODEL CLASS`
sets it.
When a model cannot be placed, replicas are stopped to make room for it, by
these rules:
- A lower class gives way to a higher one.
- Within a class, only replicas above a model's minimum — room it grew into
under load — and only for a model still below its own minimum.
- A critical model's minimum is never taken, nor a replica that has only just
loaded, nor one spanning several nodes.
- Idle on-demand replicas go first, then what lower classes grew into, then
the same within the class, then lower classes' minimums; the longest unused
first within each.
- The set is tried on a copy of the inventory first, at most three replicas,
and nothing is stopped unless it makes the room.
A replica stopped this way is drained and given 30 seconds to finish what it
is serving. Its model is held back for up to five minutes, or until the model
it gave way to has its replicas, so it does not take the room straight back.
`ctl models` shows each model's class and "gave room to X" while held; the
cluster page shows the class and how many replicas were lost; the metric is
`llmgw_model_preempted_total{model,class}`, and every preemption is logged.
### Capacity quotas for the tenancy tree
An organization, team or project can be guaranteed GPUs, CPU cores and
memory, and limited to at most some amount of each. What it holds is what the
replicas of the models it owns hold — and those of the teams and projects
beneath it. `llmgw ctl model owner MODEL team:Acme/ml` says whose a model is;
one with no owner is shared and counts against nobody.
- **A guarantee is kept.** A replica within its owner's guarantee is never
taken for another scope's model, whatever the classes: a guarantee a higher
class elsewhere could take would not be one.
- **Beyond it is borrowing.** Free room is used; when a scope within its own
guarantee needs room, what others hold beyond theirs is taken back first,
whatever their class — except a critical model's minimum.
- **A limit is a ceiling.** A model whose owner would go past a limit anywhere
up the tree gets no further replica, and says which limit; headroom and the
mirror card stop there too.
`llmgw ctl quota set team:Acme/ml --gpus 8 --limit-gpus 16` (also `--cores`,
`--memory` and their limits), `llmgw ctl quota list` with what each scope holds,
`llmgw ctl quota clear`. Scopes are named by path or ID; a name that could mean
two is refused. Changes take effect within half a minute. Metrics:
`llmgw_capacity_holds`, `llmgw_capacity_guarantee` and `llmgw_capacity_limit`
by scope and resource; alert: CapacityAtLimit.
### The mirror page in tabs
/admin/mirror was one long page: what the mirror holds, the Hugging Face
+22
View File
@@ -84,6 +84,28 @@ groups:
<model>` says what each node lacks. For an on-demand model, requests
for it are told this rather than asked to retry.
- alert: ModelPreempted
expr: increase(llmgw_model_preempted_total[15m]) > 0
labels: { severity: info }
annotations:
summary: "{{ $labels.model }} ({{ $labels.class }}) gave up replicas to a higher class"
description: >-
Capacity ran short and a model of a higher class took this one's
room. Expected now and then; if it keeps firing for the same model,
the cluster is too small for what is asked of it at these classes.
`llmgw ctl models` shows who it gave way to.
- alert: CapacityAtLimit
expr: llmgw_capacity_holds >= on(scope, resource) llmgw_capacity_limit
for: 15m
labels: { severity: info }
annotations:
summary: "{{ $labels.scope }} holds all the {{ $labels.resource }} it may"
description: >-
Its models get no further replicas, under load or not, until
something of theirs stops or the limit is raised: `llmgw ctl quota
list`, `llmgw ctl quota set`.
- alert: EngineInstallFailed
expr: llmgw_node_engine_install{state="failed"} == 1
for: 15m
+19
View File
@@ -92,6 +92,8 @@ rather than repeated.
| `source` | `string` | [Model](#model) |
| `profiles` | `[]Profile` | [Model](#model) |
| `scaling` | `Scaling` | [Model](#model) |
| `class` | `string` | [Model](#model) |
| `owner` | `string` | [Model](#model) |
| `fallbacks` | `[]string` | [Model](#model) |
| `routing_profile` | `string` | [Model](#model) |
| `slo` | `SLO` | [Model](#model) |
@@ -736,6 +738,23 @@ Source is the weights path or a HuggingFace id.
Takes a [Scaling](#scaling) block.
### `class`
`string`
Class is how much this model's capacity is worth against another's,
when there is not enough for both: critical, standard or opportunistic.
Empty is standard, or opportunistic for a model loaded on demand. See
EffectiveClass.
### `owner`
`string`
Owner is the part of the tenancy tree whose capacity quota this model
counts against, written org:ID, team:ID or project:ID. Empty is shared:
held by the cluster, counted against nobody.
### `fallbacks`
`[]string`
+6 -4
View File
@@ -139,7 +139,6 @@ func newCtlCmd() *cobra.Command {
newStatusCmd(client, &asJSON),
newModelCmd(client),
newScaleCmd(client),
newResourcesCmd(client),
newRestartCmd(client),
newRetryCmd(client),
newUpgradeCmd(client),
@@ -147,6 +146,7 @@ func newCtlCmd() *cobra.Command {
newToolCheckCmd(),
newWeightsCmd(client, &asJSON),
newCapacityCmd(client, &asJSON),
newQuotaCmd(client, &asJSON),
newUsageCmd(client, &asJSON),
newCertCmd(),
)
@@ -254,6 +254,8 @@ func modelStatus(m adminapi.ModelView) string {
switch {
case m.Parked:
return "parked"
case m.HeldFor != "":
return "gave room to " + m.HeldFor
case m.Unplaceable:
return "cannot place"
case m.Min == 0 && m.Total == 0:
@@ -324,15 +326,15 @@ func newModelsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
}
w := tab()
fmt.Fprintln(w, "MODEL\tSERVING\tREPLICAS\tSTATUS\tRESERVES\tALIASES\tPROFILES\tFALLBACKS")
fmt.Fprintln(w, "MODEL\tSERVING\tREPLICAS\tSTATUS\tCLASS\tRESERVES\tALIASES\tPROFILES\tFALLBACKS")
var warnings []string
for _, m := range models {
serving := "no"
if m.Ready > 0 {
serving = "yes"
}
fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\t%s\n",
m.Name, serving, m.Ready, m.Min, modelStatus(m), reserves(m),
fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\t%s\t%s\n",
m.Name, serving, m.Ready, m.Min, modelStatus(m), m.Class, reserves(m),
join(m.Aliases), join(m.Profiles), join(m.Fallbacks))
for _, x := range m.Warnings {
warnings = append(warnings, m.Name+": "+x)
+30
View File
@@ -30,6 +30,7 @@ func newModelCmd(client func() *ctlClient) *cobra.Command {
Short: "Add or remove models while the cluster is running",
}
c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
newModelServeCmd(client))
return c
}
@@ -233,3 +234,32 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
c.Flags().StringVar(&profile, "profile", "", "one profile; every profile when left out")
return c
}
func newClassCmd(client func() *ctlClient) *cobra.Command {
return &cobra.Command{
Use: "class <model> critical|standard|opportunistic|default",
Short: "Set how much a model's capacity is worth against another's",
Long: "When a model cannot be placed, replicas of a lower class are stopped to\n" +
"make room for it:\n\n" +
" critical never gives up the replicas of its minimum\n" +
" standard gives way to critical\n" +
" opportunistic gives way to both; the default for an on-demand model\n\n" +
"Within a class, a model below its minimum may take what another grew into\n" +
"under load, but not its minimum. \"default\" goes back to what the model's\n" +
"scaling implies: opportunistic when on demand, standard otherwise.",
Args: cobra.ExactArgs(2),
RunE: func(cmd *cobra.Command, args []string) error {
class := args[1]
if class == "default" {
class = ""
}
body, _ := json.Marshal(map[string]string{"class": class})
var res map[string]string
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/class", body, &res); err != nil {
return err
}
fmt.Fprintf(cmd.OutOrStdout(), "%s is %s\n", res["model"], res["class"])
return nil
},
}
}
+164
View File
@@ -0,0 +1,164 @@
package cli
import (
"encoding/json"
"fmt"
"net/url"
"strconv"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
func newQuotaCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
c := &cobra.Command{
Use: "quota",
Short: "What parts of the tenancy tree are guaranteed of the cluster, and may hold",
Long: "A capacity quota belongs to an organization, team or project, and counts the\n" +
"GPUs, CPU cores and memory held by replicas of the models it owns (see\n" +
"`llmgw ctl model owner`), and of those its teams and projects own.\n\n" +
"The guarantee is kept for it: when it holds less and a model of its needs\n" +
"room, replicas other scopes hold beyond their own guarantees are stopped to\n" +
"make it — whatever their class. What it holds within its guarantee is\n" +
"never taken by another scope's model. Beyond it, it borrows: free room is\n" +
"used, and given back when asked for.\n\n" +
"The limit is the most it may hold, borrowing included; a model that would\n" +
"take it past is not given another replica.\n\n" +
"Scopes are written org:Acme, team:Acme/ml or project:Acme/ml/training, or\n" +
"with IDs. Changes take effect within half a minute.",
}
c.AddCommand(newQuotaListCmd(client, asJSON), newQuotaSetCmd(client), newQuotaClearCmd(client))
return c
}
func newQuotaListCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
return &cobra.Command{
Use: "list",
Short: "Every quota, with what its scope holds now",
RunE: func(cmd *cobra.Command, _ []string) error {
var qs []adminapi.QuotaView
if err := client().get(cmd.Context(), "/api/v1/quotas", &qs); err != nil {
return err
}
if *asJSON {
return json.NewEncoder(cmd.OutOrStdout()).Encode(qs)
}
if len(qs) == 0 {
fmt.Fprintln(cmd.OutOrStdout(), "No capacity quotas: everything shares the cluster by class alone.")
return nil
}
w := tab()
fmt.Fprintln(w, "SCOPE\tNAME\tGPUS\tCORES\tMEMORY GB")
for _, q := range qs {
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", q.Scope, q.Name,
quotaCell(q.Holds.GPUs, q.Guarantee.GPUs, q.Limit.GPUs),
quotaCell(q.Holds.CPUCores, q.Guarantee.CPUCores, q.Limit.CPUCores),
quotaCell(q.Holds.MemoryGB, q.Guarantee.MemoryGB, q.Limit.MemoryGB))
}
if err := w.Flush(); err != nil {
return err
}
fmt.Fprintln(cmd.OutOrStdout(), "\nEach cell: holds / guaranteed (at most limit). - is not governed.")
return nil
},
}
}
// quotaCell is "6 / 8 (≤ 16)": what is held, what is guaranteed, the limit.
func quotaCell(holds int, guarantee, limit *int) string {
if guarantee == nil && limit == nil {
return "-"
}
out := strconv.Itoa(holds) + " / "
if guarantee != nil {
out += strconv.Itoa(*guarantee)
} else {
out += "-"
}
if limit != nil {
out += fmt.Sprintf(" (≤ %d)", *limit)
}
return out
}
func newQuotaSetCmd(client func() *ctlClient) *cobra.Command {
var gpus, cores, mem, lgpus, lcores, lmem int
c := &cobra.Command{
Use: "set SCOPE [--gpus N] [--cores N] [--memory GB] [--limit-gpus N] ...",
Short: "Set a scope's guarantee and limit, replacing what it had",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
f := cmd.Flags()
pick := func(name string, v int) *int {
if !f.Changed(name) {
return nil
}
return &v
}
req := adminapi.QuotaRequest{Scope: args[0],
Guarantee: store.Resources{GPUs: pick("gpus", gpus), CPUCores: pick("cores", cores), MemoryGB: pick("memory", mem)},
Limit: store.Resources{GPUs: pick("limit-gpus", lgpus), CPUCores: pick("limit-cores", lcores), MemoryGB: pick("limit-memory", lmem)},
}
body, _ := json.Marshal(req)
var res map[string]string
if err := client().put(cmd.Context(), "/api/v1/quotas", body, &res); err != nil {
return err
}
fmt.Fprintf(cmd.OutOrStdout(), "%s: set; it takes effect within half a minute\n", res["scope"])
return nil
},
}
f := c.Flags()
f.IntVar(&gpus, "gpus", 0, "GPUs guaranteed")
f.IntVar(&cores, "cores", 0, "CPU cores guaranteed")
f.IntVar(&mem, "memory", 0, "memory guaranteed, in GB")
f.IntVar(&lgpus, "limit-gpus", 0, "the most GPUs it may hold")
f.IntVar(&lcores, "limit-cores", 0, "the most CPU cores it may hold")
f.IntVar(&lmem, "limit-memory", 0, "the most memory it may hold, in GB")
return c
}
func newQuotaClearCmd(client func() *ctlClient) *cobra.Command {
return &cobra.Command{
Use: "clear SCOPE",
Short: "Remove a scope's quota",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
var res map[string]string
if err := client().delete(cmd.Context(), "/api/v1/quotas/"+url.PathEscape(args[0]), &res); err != nil {
return err
}
fmt.Fprintf(cmd.OutOrStdout(), "%s: quota removed\n", res["removed"])
return nil
},
}
}
func newOwnerCmd(client func() *ctlClient) *cobra.Command {
return &cobra.Command{
Use: "owner <model> SCOPE|shared",
Short: "Set whose capacity quota a model counts against",
Long: "SCOPE is org:Acme, team:Acme/ml or project:Acme/ml/training. A shared\n" +
"model belongs to the cluster and counts against nobody's quota.",
Args: cobra.ExactArgs(2),
RunE: func(cmd *cobra.Command, args []string) error {
owner := args[1]
if owner == "shared" {
owner = ""
}
body, _ := json.Marshal(map[string]string{"owner": owner})
var res map[string]string
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/owner", body, &res); err != nil {
return err
}
if res["owner"] == "" {
fmt.Fprintf(cmd.OutOrStdout(), "%s is shared\n", res["model"])
} else {
fmt.Fprintf(cmd.OutOrStdout(), "%s counts against %s\n", res["model"], res["owner"])
}
return nil
},
}
}
+5
View File
@@ -226,6 +226,8 @@ func newServerCmd() *cobra.Command {
// And what each group of the tenancy tree spent, which is
// the other half of a quota being reviewable.
api.SetUsage(st)
// Capacity quotas, set and read over this API.
api.SetQuotas(st)
}
api.Register(adminMux)
@@ -483,6 +485,9 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
if st := h.Store(); st != nil {
lead = control.NewLeadership(st, cfg.Control.LeaseTTL, log)
ctl.SetLeadership(lead)
// The tenancy tree's capacity quotas: what each part is
// guaranteed and may hold.
ctl.Reconciler().SetQuotas(control.QuotaPolicy(st, log))
log.Info("competing to be the control plane that places work",
"owner", lead.Owner())
}
+30
View File
@@ -42,6 +42,7 @@ type API struct {
models Models
capacity Capacity
usage Usage
quotas Quotas
ctl *control.Controller
}
@@ -58,6 +59,26 @@ func (a *API) Register(mux *http.ServeMux) {
mux.HandleFunc("DELETE /api/v1/models/{name}", a.removeModel)
mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
a.registerQuotas(mux)
mux.HandleFunc("POST /api/v1/models/{name}/class", func(w http.ResponseWriter, r *http.Request) {
if a.models == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store")
return
}
var req struct {
Class string `json:"class"`
}
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
return
}
m, err := a.models.SetClass(r.Context(), r.PathValue("name"), req.Class, actorOf(r))
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]string{"model": m.Name, "class": m.EffectiveClass()})
})
mux.HandleFunc("PUT /api/v1/models/{name}/capacity", a.putCapacity)
mux.HandleFunc("GET /api/v1/capacity", a.listCapacity)
mux.HandleFunc("GET /api/v1/usage", a.usageByScope)
@@ -282,6 +303,12 @@ type ModelView struct {
// Warnings are what makes its reservation wrong: nothing declared where
// placement needs a figure, or less declared than it uses.
Warnings []string `json:"warnings,omitempty"`
// Class is its capacity class; Preempted the replicas it lost to a higher
// class; HeldFor the model it is giving room to now, if any.
Class string `json:"class"`
Owner string `json:"owner,omitempty"`
Preempted int `json:"preempted,omitempty"`
HeldFor string `json:"held_for,omitempty"`
}
// ReplicaView is a replica as `ctl replicas` shows it.
@@ -528,6 +555,7 @@ func (a *API) Models() []ModelView {
Fallbacks: m.Fallbacks,
Desired: h.Desired, Failures: h.Failures,
Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
Class: m.EffectiveClass(), Owner: m.Owner, Preempted: h.Preempted, HeldFor: h.HeldFor,
}
for _, p := range m.Profiles {
v.Profiles = append(v.Profiles, p.Name)
@@ -608,6 +636,8 @@ type Models interface {
Remove(ctx context.Context, name string) error
Scale(ctx context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error)
Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
}
// ResourcesRequest sets what a model's profiles reserve of a node.
+8
View File
@@ -258,6 +258,14 @@ type fakeModels struct {
err error
}
func (f *fakeModels) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
return catalog.Model{Name: name, Owner: owner}, f.err
}
func (f *fakeModels) SetClass(_ context.Context, name, class, _ string) (catalog.Model, error) {
return catalog.Model{Name: name, Class: class}, f.err
}
func (f *fakeModels) Resources(_ context.Context, name, profile string, mem, cores int, _ string) (catalog.Model, error) {
if f.err != nil {
return catalog.Model{}, f.err
+237
View File
@@ -0,0 +1,237 @@
package adminapi
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// Quotas is where capacity quotas and the tenancy tree they name are kept.
// The store implements it.
type Quotas interface {
CapacityQuotas(ctx context.Context) ([]store.CapacityQuota, error)
PutCapacityQuota(ctx context.Context, q store.CapacityQuota) error
DeleteCapacityQuota(ctx context.Context, scope store.Scope, id string) error
ScopeParents(ctx context.Context) (map[string][]string, error)
AllOrgs(ctx context.Context) ([]store.Org, error)
AllTeams(ctx context.Context) ([]store.TeamPath, error)
AllProjects(ctx context.Context) ([]store.ProjectPath, error)
}
// SetQuotas supplies the quota store, which mounts the quota endpoints'
// behaviour; without it they answer 501.
func (a *API) SetQuotas(q Quotas) { a.quotas = q }
// QuotaView is one scope's quota with what it holds now.
type QuotaView struct {
// Scope is team:ID; Name is how people know it, "Acme / ml".
Scope string `json:"scope"`
Name string `json:"name"`
Guarantee store.Resources `json:"guarantee"`
Limit store.Resources `json:"limit"`
Holds quota.Amount `json:"holds"`
UpdatedBy string `json:"updated_by,omitempty"`
UpdatedAt time.Time `json:"updated_at"`
}
// QuotaRequest sets a scope's quota. Scope may be written with an ID or a
// name: org:Acme, team:Acme/ml, project:Acme/ml/training.
type QuotaRequest struct {
Scope string `json:"scope"`
Guarantee store.Resources `json:"guarantee"`
Limit store.Resources `json:"limit"`
}
func (a *API) registerQuotas(mux *http.ServeMux) {
mux.HandleFunc("GET /api/v1/quotas", func(w http.ResponseWriter, r *http.Request) {
if a.quotas == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store")
return
}
views, err := a.QuotaViews(r.Context())
if err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
writeJSON(w, http.StatusOK, views)
})
mux.HandleFunc("PUT /api/v1/quotas", func(w http.ResponseWriter, r *http.Request) {
if a.quotas == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store")
return
}
var req QuotaRequest
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<14)).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
return
}
scope, id, _, err := a.resolveScope(r.Context(), req.Scope)
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
if !req.Guarantee.Any() && !req.Limit.Any() {
writeError(w, http.StatusBadRequest, "a quota needs a guarantee or a limit; to remove one, DELETE it")
return
}
q := store.CapacityQuota{Scope: scope, ScopeID: id, Guarantee: req.Guarantee, Limit: req.Limit,
UpdatedBy: actorOf(r)}
if err := a.quotas.PutCapacityQuota(r.Context(), q); err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]string{"scope": quota.Key(scope, id)})
})
mux.HandleFunc("DELETE /api/v1/quotas/{scope}", func(w http.ResponseWriter, r *http.Request) {
if a.quotas == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store")
return
}
scope, id, _, err := a.resolveScope(r.Context(), r.PathValue("scope"))
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
if err := a.quotas.DeleteCapacityQuota(r.Context(), scope, id); err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]string{"removed": quota.Key(scope, id)})
})
mux.HandleFunc("POST /api/v1/models/{name}/owner", func(w http.ResponseWriter, r *http.Request) {
if a.models == nil || a.quotas == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store")
return
}
var req struct {
Owner string `json:"owner"`
}
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
return
}
owner := ""
if req.Owner != "" {
scope, id, _, err := a.resolveScope(r.Context(), req.Owner)
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
owner = quota.Key(scope, id)
}
m, err := a.models.SetOwner(r.Context(), r.PathValue("name"), owner, actorOf(r))
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
writeJSON(w, http.StatusOK, map[string]string{"model": m.Name, "owner": m.Owner})
})
}
// QuotaViews lists every quota with what its scope holds now.
func (a *API) QuotaViews(ctx context.Context) ([]QuotaView, error) {
qs, err := a.quotas.CapacityQuotas(ctx)
if err != nil {
return nil, err
}
parents, err := a.quotas.ScopeParents(ctx)
if err != nil {
return nil, err
}
names, err := a.scopeNames(ctx)
if err != nil {
return nil, err
}
usage := quota.New(qs, parents).Usage(a.ctl.Reconciler().Holdings())
out := make([]QuotaView, 0, len(qs))
for _, q := range qs {
key := quota.Key(q.Scope, q.ScopeID)
out = append(out, QuotaView{Scope: key, Name: names[key], Guarantee: q.Guarantee, Limit: q.Limit,
Holds: usage[key], UpdatedBy: q.UpdatedBy, UpdatedAt: q.UpdatedAt})
}
return out, nil
}
// scopeNames maps org:ID, team:ID and project:ID to their paths.
func (a *API) scopeNames(ctx context.Context) (map[string]string, error) {
out := map[string]string{}
orgs, err := a.quotas.AllOrgs(ctx)
if err != nil {
return nil, err
}
for _, o := range orgs {
out["org:"+o.ID] = o.Name
}
teams, err := a.quotas.AllTeams(ctx)
if err != nil {
return nil, err
}
for _, t := range teams {
out["team:"+t.ID] = t.OrgName + " / " + t.Name
}
projects, err := a.quotas.AllProjects(ctx)
if err != nil {
return nil, err
}
for _, p := range projects {
out["project:"+p.ID] = p.Path
}
return out, nil
}
// resolveScope reads org:X, team:X or project:X, where X is an ID or the
// scope's path — Acme, Acme/ml, Acme/ml/training — and returns its ID and
// path. A path that could mean two scopes is refused rather than guessed.
func (a *API) resolveScope(ctx context.Context, s string) (store.Scope, string, string, error) {
kind, what, ok := strings.Cut(strings.TrimSpace(s), ":")
scope := store.Scope(kind)
if !ok || what == "" || (scope != store.ScopeOrg && scope != store.ScopeTeam && scope != store.ScopeProject) {
return "", "", "", fmt.Errorf("%q is not org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT", s)
}
names, err := a.scopeNames(ctx)
if err != nil {
return "", "", "", err
}
norm := func(p string) string {
parts := strings.Split(p, "/")
for i := range parts {
parts[i] = strings.ToLower(strings.TrimSpace(parts[i]))
}
return strings.Join(parts, "/")
}
if name, ok := names[kind+":"+what]; ok {
return scope, what, name, nil
}
match := func(same func(name string) bool) []string {
var found []string
for key, name := range names {
if strings.HasPrefix(key, kind+":") && same(name) {
found = append(found, key)
}
}
return found
}
found := match(func(name string) bool { return norm(name) == norm(what) })
if len(found) == 0 {
// Its own name alone — ml rather than Acme/ml — when that is unique.
found = match(func(name string) bool {
parts := strings.Split(norm(name), "/")
return parts[len(parts)-1] == norm(what)
})
}
switch len(found) {
case 0:
return "", "", "", fmt.Errorf("there is no %s %q", kind, what)
case 1:
_, id, _ := strings.Cut(found[0], ":")
return scope, id, names[found[0]], nil
}
return "", "", "", fmt.Errorf("%q could be any of %d %ss; give its ID", what, len(found), kind)
}
+107
View File
@@ -0,0 +1,107 @@
package adminapi
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// fakeQuotaStore is Acme with teams ml and web, and Scania with another ml.
type fakeQuotaStore struct {
quotas []store.CapacityQuota
}
func (f *fakeQuotaStore) CapacityQuotas(context.Context) ([]store.CapacityQuota, error) {
return f.quotas, nil
}
func (f *fakeQuotaStore) PutCapacityQuota(_ context.Context, q store.CapacityQuota) error {
for i := range f.quotas {
if f.quotas[i].Scope == q.Scope && f.quotas[i].ScopeID == q.ScopeID {
f.quotas[i] = q
return nil
}
}
f.quotas = append(f.quotas, q)
return nil
}
func (f *fakeQuotaStore) DeleteCapacityQuota(_ context.Context, s store.Scope, id string) error {
for i, q := range f.quotas {
if q.Scope == s && q.ScopeID == id {
f.quotas = append(f.quotas[:i], f.quotas[i+1:]...)
}
}
return nil
}
func (f *fakeQuotaStore) ScopeParents(context.Context) (map[string][]string, error) {
return map[string][]string{"team:t-ml": {"org:o-acme"}, "team:t-web": {"org:o-acme"},
"team:t-sml": {"org:o-scania"}}, nil
}
func (f *fakeQuotaStore) AllOrgs(context.Context) ([]store.Org, error) {
return []store.Org{{ID: "o-acme", Name: "Acme"}, {ID: "o-scania", Name: "Scania"}}, nil
}
func (f *fakeQuotaStore) AllTeams(context.Context) ([]store.TeamPath, error) {
return []store.TeamPath{{ID: "t-ml", Name: "ml", OrgID: "o-acme", OrgName: "Acme"},
{ID: "t-web", Name: "web", OrgID: "o-acme", OrgName: "Acme"},
{ID: "t-sml", Name: "ml", OrgID: "o-scania", OrgName: "Scania"}}, nil
}
func (f *fakeQuotaStore) AllProjects(context.Context) ([]store.ProjectPath, error) { return nil, nil }
func TestAScopeIsNamedByPathOrIDAndNeverGuessed(t *testing.T) {
a, _ := setup(t, testModels())
a.SetQuotas(&fakeQuotaStore{})
for in, want := range map[string]string{
"org:Acme": "o-acme", "org:o-acme": "o-acme", "team:Acme/ml": "t-ml",
"team: acme / ML ": "t-ml", "team:Scania/ml": "t-sml", "team:web": "t-web",
} {
_, id, _, err := a.resolveScope(t.Context(), in)
if err != nil || id != want {
t.Errorf("%q: %q %v, expected %q", in, id, err, want)
}
}
for _, bad := range []string{"team:ml", "team:Acme/nope", "group:Acme", "Acme", "org:"} {
if _, _, _, err := a.resolveScope(t.Context(), bad); err == nil {
t.Errorf("%q was resolved", bad)
}
}
if _, _, _, err := a.resolveScope(t.Context(), "team:ml"); err == nil || !strings.Contains(err.Error(), "could be any of 2") {
t.Errorf("an ambiguous name should say so: %v", err)
}
}
func TestAQuotaIsSetListedAndCleared(t *testing.T) {
a, _ := setup(t, testModels())
qs := &fakeQuotaStore{}
a.SetQuotas(qs)
mux := http.NewServeMux()
a.Register(mux)
call := func(method, path, body string) *httptest.ResponseRecorder {
req := httptest.NewRequest(method, path, strings.NewReader(body))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("X-Llmgw-Actor", "ops")
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
return rec
}
if rec := call("PUT", "/api/v1/quotas", `{"scope":"team:Acme/ml","guarantee":{"gpus":8},"limit":{"gpus":16}}`); rec.Code != 200 {
t.Fatalf("set: %d %s", rec.Code, rec.Body)
}
if len(qs.quotas) != 1 || qs.quotas[0].ScopeID != "t-ml" || *qs.quotas[0].Guarantee.GPUs != 8 || qs.quotas[0].UpdatedBy != "ops" {
t.Errorf("stored %+v", qs.quotas)
}
rec := call("GET", "/api/v1/quotas", "")
var views []QuotaView
if err := json.Unmarshal(rec.Body.Bytes(), &views); err != nil || len(views) != 1 || views[0].Name != "Acme / ml" {
t.Errorf("list %d %s", rec.Code, rec.Body)
}
if rec := call("PUT", "/api/v1/quotas", `{"scope":"team:Acme/ml"}`); rec.Code != 400 {
t.Errorf("an empty quota was accepted: %d", rec.Code)
}
if rec := call("DELETE", "/api/v1/quotas/team:Acme%2Fml", ""); rec.Code != 200 || len(qs.quotas) != 0 {
t.Errorf("clear: %d %s %+v", rec.Code, rec.Body, qs.quotas)
}
}
+65 -4
View File
@@ -149,10 +149,19 @@ type Model struct {
// choice by guesswork.
Description string `json:"description,omitempty" koanf:"description"`
// Source is the weights path or a HuggingFace id.
Source string `json:"source,omitempty" koanf:"source"`
Profiles []Profile `json:"profiles,omitempty" koanf:"profiles"`
Scaling Scaling `json:"scaling,omitempty" koanf:"scaling"`
Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
Source string `json:"source,omitempty" koanf:"source"`
Profiles []Profile `json:"profiles,omitempty" koanf:"profiles"`
Scaling Scaling `json:"scaling,omitempty" koanf:"scaling"`
// Class is how much this model's capacity is worth against another's,
// when there is not enough for both: critical, standard or opportunistic.
// Empty is standard, or opportunistic for a model loaded on demand. See
// EffectiveClass.
Class string `json:"class,omitempty" koanf:"class"`
// Owner is the part of the tenancy tree whose capacity quota this model
// counts against, written org:ID, team:ID or project:ID. Empty is shared:
// held by the cluster, counted against nobody.
Owner string `json:"owner,omitempty" koanf:"owner"`
Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
// RoutingProfile selects how the router scores endpoints for this model.
RoutingProfile string `json:"routing_profile,omitempty" koanf:"routing_profile"`
SLO SLO `json:"slo,omitempty" koanf:"slo"`
@@ -299,6 +308,12 @@ func (m Model) validate() error {
if len(m.Profiles) == 0 {
return fmt.Errorf("model %q: at least one profile is required", m.Name)
}
if m.Owner != "" && !validOwner(m.Owner) {
return fmt.Errorf("model %q: owner %q is not org:ID, team:ID or project:ID", m.Name, m.Owner)
}
if !validClass(m.Class) {
return fmt.Errorf("model %q: class %q is not one of critical, standard, opportunistic", m.Name, m.Class)
}
if m.Scaling.Max < m.Scaling.Min {
return fmt.Errorf("model %q: scaling max %d is below min %d", m.Name, m.Scaling.Max, m.Scaling.Min)
}
@@ -503,3 +518,49 @@ func SuggestCores(cpuCounts []int) int {
}
return max(smallest/2, 1)
}
// The capacity classes, lowest first.
const (
ClassOpportunistic = "opportunistic"
ClassStandard = "standard"
ClassCritical = "critical"
)
func validClass(c string) bool {
return c == "" || c == ClassOpportunistic || c == ClassStandard || c == ClassCritical
}
// EffectiveClass is the class a model is treated as: what it says, or, when
// it says nothing, opportunistic for one loaded on demand — it holds capacity
// only while asked for, and has always been the first to give it up — and
// standard for everything else.
func (m Model) EffectiveClass() string {
if m.Class != "" {
return m.Class
}
if m.Scaling.OnDemand() {
return ClassOpportunistic
}
return ClassStandard
}
// ClassRank orders classes: a higher rank may take capacity from a lower one.
func ClassRank(c string) int {
switch c {
case ClassCritical:
return 3
case ClassStandard, "":
return 2
default:
return 1
}
}
func validOwner(o string) bool {
for _, p := range []string{"org:", "team:", "project:"} {
if strings.HasPrefix(o, p) && len(o) > len(p) {
return true
}
}
return false
}
+3
View File
@@ -375,3 +375,6 @@ func EngineKeys(secret []byte) func(replicaID string) string {
return "llmgw-" + hex.EncodeToString(m.Sum(nil))[:40]
}
}
// QuotaUse is every capacity quota beside what its scope holds.
func (c *Controller) QuotaUse() []reconciler.QuotaUse { return c.rec.QuotaUse() }
+61 -7
View File
@@ -51,12 +51,16 @@ type HealthSource interface {
type Collector struct {
src Source
modelDesired *prometheus.Desc
modelFailures *prometheus.Desc
modelParked *prometheus.Desc
modelUnplaceable *prometheus.Desc
engineInstall *prometheus.Desc
leader *prometheus.Desc
modelDesired *prometheus.Desc
modelFailures *prometheus.Desc
modelParked *prometheus.Desc
modelUnplaceable *prometheus.Desc
modelPreempted *prometheus.Desc
capacityHolds *prometheus.Desc
capacityGuarantee *prometheus.Desc
capacityLimit *prometheus.Desc
engineInstall *prometheus.Desc
leader *prometheus.Desc
nodeUp *prometheus.Desc
nodeGPUs *prometheus.Desc
@@ -168,6 +172,18 @@ func New(src Source) *Collector {
modelUnplaceable: prometheus.NewDesc("llmgw_model_unplaceable",
"1 when a model is short of replicas and the last attempt found no node to place one on.",
[]string{"model"}, nil),
capacityHolds: prometheus.NewDesc("llmgw_capacity_holds",
"What a part of the tenancy tree holds of a resource, its own models' replicas and those beneath it.",
[]string{"scope", "resource"}, nil),
capacityGuarantee: prometheus.NewDesc("llmgw_capacity_guarantee",
"What a part of the tenancy tree is guaranteed of a resource.",
[]string{"scope", "resource"}, nil),
capacityLimit: prometheus.NewDesc("llmgw_capacity_limit",
"The most of a resource a part of the tenancy tree may hold.",
[]string{"scope", "resource"}, nil),
modelPreempted: prometheus.NewDesc("llmgw_model_preempted_total",
"Replicas of a model stopped to make room for a model of a higher capacity class.",
[]string{"model", "class"}, nil),
engineInstall: prometheus.NewDesc("llmgw_node_engine_install",
"Where installing an engine stands on a node: 1 for the current state.",
[]string{"node", "engine", "version", "state"}, nil),
@@ -236,7 +252,8 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
c.gpuNVLink, c.gpuECC,
c.replicaState, c.replicaReady, c.replicaIdle,
c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable,
c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable, c.modelPreempted,
c.capacityHolds, c.capacityGuarantee, c.capacityLimit,
c.engineInstall, c.leader,
} {
ch <- d
@@ -246,6 +263,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
// Collect implements prometheus.Collector.
func (c *Collector) Collect(ch chan<- prometheus.Metric) {
now := time.Now()
c.collectQuotas(ch)
loads := map[string]*inventory.Load{}
for _, n := range c.src.Nodes() {
@@ -455,6 +473,7 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
ch <- prometheus.MustNewConstMetric(c.modelFailures, prometheus.GaugeValue, float64(h.Failures), h.Model)
ch <- prometheus.MustNewConstMetric(c.modelParked, prometheus.GaugeValue, b2f(h.Parked), h.Model)
ch <- prometheus.MustNewConstMetric(c.modelUnplaceable, prometheus.GaugeValue, b2f(h.Unplaceable), h.Model)
ch <- prometheus.MustNewConstMetric(c.modelPreempted, prometheus.CounterValue, float64(h.Preempted), h.Model, h.Class)
}
}
@@ -464,3 +483,38 @@ func b2f(b bool) float64 {
}
return 0
}
// QuotaSource adds the tenancy tree's capacity quotas. Asked for by type
// assertion, like HealthSource.
type QuotaSource interface {
QuotaUse() []reconciler.QuotaUse
}
func (c *Collector) collectQuotas(ch chan<- prometheus.Metric) {
qs, ok := c.src.(QuotaSource)
if !ok {
return
}
for _, q := range qs.QuotaUse() {
for _, r := range []struct {
name string
holds int
gua, lim *int
}{
{"gpus", q.Holds.GPUs, q.Guarantee.GPUs, q.Limit.GPUs},
{"cpu_cores", q.Holds.CPUCores, q.Guarantee.CPUCores, q.Limit.CPUCores},
{"memory_gb", q.Holds.MemoryGB, q.Guarantee.MemoryGB, q.Limit.MemoryGB},
} {
if r.gua == nil && r.lim == nil {
continue
}
ch <- prometheus.MustNewConstMetric(c.capacityHolds, prometheus.GaugeValue, float64(r.holds), q.Scope, r.name)
if r.gua != nil {
ch <- prometheus.MustNewConstMetric(c.capacityGuarantee, prometheus.GaugeValue, float64(*r.gua), q.Scope, r.name)
}
if r.lim != nil {
ch <- prometheus.MustNewConstMetric(c.capacityLimit, prometheus.GaugeValue, float64(*r.lim), q.Scope, r.name)
}
}
}
}
+26
View File
@@ -242,6 +242,32 @@ func (s *Service) Resources(ctx context.Context, name, profile string, memoryGB,
return final, err
}
// SetClass sets a model's capacity class: critical, standard or
// opportunistic, or empty to go back to what its scaling implies.
func (s *Service) SetClass(ctx context.Context, name, class, by string) (catalog.Model, error) {
final, err := s.update(ctx, name, by, "change its class", func(m *catalog.Model) error {
m.Class = class
return nil
})
if err == nil {
s.log.Info("model class set", "model", name, "class", final.EffectiveClass(), "by", by)
}
return final, err
}
// SetOwner sets the part of the tenancy tree a model's capacity counts
// against, org:ID, team:ID or project:ID; empty makes it shared.
func (s *Service) SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error) {
final, err := s.update(ctx, name, by, "change its owner", func(m *catalog.Model) error {
m.Owner = owner
return nil
})
if err == nil {
s.log.Info("model owner set", "model", name, "owner", owner, "by", by)
}
return final, err
}
// update changes one model added at runtime and keeps the change.
//
// Built before anything is written, like Add: catalog.New is what applies the
+192
View File
@@ -0,0 +1,192 @@
// Package quota decides what the tenancy tree is guaranteed of the cluster
// and how much of it each part may hold.
//
// A scope — an organization, team or project — may be guaranteed GPUs, CPU
// cores and memory, and limited to at most some amount of each. What a scope
// holds is what the replicas of the models it owns, and of the models its
// teams and projects own, have reserved. Holding more than its guarantee is
// borrowing: allowed while the room is free, and taken back when another
// scope within its own guarantee needs it.
package quota
import (
"fmt"
"sort"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// Amount is how much of each resource something holds or needs.
type Amount struct {
GPUs, CPUCores, MemoryGB int
}
// Plus adds two amounts.
func (a Amount) Plus(b Amount) Amount {
return Amount{a.GPUs + b.GPUs, a.CPUCores + b.CPUCores, a.MemoryGB + b.MemoryGB}
}
// Over names the first resource in which a is above the governed amounts of
// l, with both figures; empty when it is above none.
func Over(a Amount, l store.Resources) string {
switch {
case l.GPUs != nil && a.GPUs > *l.GPUs:
return fmt.Sprintf("%d GPUs of %d", a.GPUs, *l.GPUs)
case l.CPUCores != nil && a.CPUCores > *l.CPUCores:
return fmt.Sprintf("%d cores of %d", a.CPUCores, *l.CPUCores)
case l.MemoryGB != nil && a.MemoryGB > *l.MemoryGB:
return fmt.Sprintf("%d GB of memory of %d", a.MemoryGB, *l.MemoryGB)
}
return ""
}
// Holding is one replica's hold on the cluster, for working out who holds
// what.
type Holding struct {
Replica string
Owner string
Amount Amount
Since time.Time
}
// Policy is the quotas and the tree they sit in, read at one moment.
type Policy struct {
quotas map[string]store.CapacityQuota
parents map[string][]string
}
// New builds a policy from stored quotas and each scope's parents.
func New(quotas []store.CapacityQuota, parents map[string][]string) *Policy {
p := &Policy{quotas: map[string]store.CapacityQuota{}, parents: parents}
for _, q := range quotas {
p.quotas[Key(q.Scope, q.ScopeID)] = q
}
return p
}
// Key writes a scope the way owners are written: team:ID.
func Key(s store.Scope, id string) string { return string(s) + ":" + id }
// Chain is an owner and every scope above it, nearest first.
func (p *Policy) Chain(owner string) []string {
if owner == "" {
return nil
}
return append([]string{owner}, p.parents[owner]...)
}
// Governing is the quotas along an owner's chain, nearest first.
func (p *Policy) Governing(owner string) []store.CapacityQuota {
if p == nil {
return nil
}
var out []store.CapacityQuota
for _, s := range p.Chain(owner) {
if q, ok := p.quotas[s]; ok {
out = append(out, q)
}
}
return out
}
// Quotas lists every quota.
func (p *Policy) Quotas() []store.CapacityQuota {
if p == nil {
return nil
}
out := make([]store.CapacityQuota, 0, len(p.quotas))
for _, q := range p.quotas {
out = append(out, q)
}
sort.Slice(out, func(i, j int) bool {
return Key(out[i].Scope, out[i].ScopeID) < Key(out[j].Scope, out[j].ScopeID)
})
return out
}
// Usage is what every scope holds, its own models' replicas and those of
// everything beneath it.
func (p *Policy) Usage(hs []Holding) map[string]Amount {
out := map[string]Amount{}
if p == nil {
return out
}
for _, h := range hs {
for _, s := range p.Chain(h.Owner) {
out[s] = out[s].Plus(h.Amount)
}
}
return out
}
// Borrowed is the replicas held beyond a guarantee: of each governed scope's
// replicas, oldest first, the ones that take it past what it is guaranteed.
// A replica beyond any guarantee along its chain is borrowed.
func (p *Policy) Borrowed(hs []Holding) map[string]bool {
out := map[string]bool{}
if p == nil || len(p.quotas) == 0 {
return out
}
sorted := append([]Holding(nil), hs...)
sort.Slice(sorted, func(i, j int) bool { return sorted[i].Since.Before(sorted[j].Since) })
for key, q := range p.quotas {
if !q.Guarantee.Any() {
continue
}
var sum Amount
for _, h := range sorted {
if !contains(p.Chain(h.Owner), key) {
continue
}
sum = sum.Plus(h.Amount)
if Over(sum, q.Guarantee) != "" {
out[h.Replica] = true
}
}
}
return out
}
// Guaranteed reports whether an owner, holding what it holds, is still within
// every guarantee along its chain with need added — so it may take back what
// others borrowed. An owner with no guarantee anywhere is never guaranteed.
func (p *Policy) Guaranteed(owner string, usage map[string]Amount, need Amount) bool {
gov := p.Governing(owner)
any := false
for _, q := range gov {
if !q.Guarantee.Any() {
continue
}
any = true
if Over(usage[Key(q.Scope, q.ScopeID)].Plus(need), q.Guarantee) != "" {
return false
}
}
return any
}
// AtLimit says which limit along an owner's chain need would take it past,
// empty when none.
func (p *Policy) AtLimit(owner string, usage map[string]Amount, need Amount) string {
for _, q := range p.Governing(owner) {
if !q.Limit.Any() {
continue
}
key := Key(q.Scope, q.ScopeID)
if over := Over(usage[key].Plus(need), q.Limit); over != "" {
return fmt.Sprintf("%s is at its capacity limit (%s)", key, strings.TrimSpace(over))
}
}
return ""
}
func contains(list []string, v string) bool {
for _, x := range list {
if x == v {
return true
}
}
return false
}
+83
View File
@@ -0,0 +1,83 @@
package quota
import (
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
func n(v int) *int { return &v }
// acme has two teams; ml is guaranteed 8 GPUs, the org as a whole 12 and
// no more than 16.
func policy() *Policy {
return New([]store.CapacityQuota{
{Scope: store.ScopeTeam, ScopeID: "ml", Guarantee: store.Resources{GPUs: n(8)}},
{Scope: store.ScopeOrg, ScopeID: "acme", Guarantee: store.Resources{GPUs: n(12)}, Limit: store.Resources{GPUs: n(16)}},
}, map[string][]string{
"team:ml": {"org:acme"}, "team:web": {"org:acme"},
"project:train": {"team:ml", "org:acme"},
})
}
func at(min int) time.Time { return time.Date(2026, 1, 1, 0, min, 0, 0, time.UTC) }
func TestAScopeHoldsWhatEverythingBeneathItHolds(t *testing.T) {
p := policy()
u := p.Usage([]Holding{
{Replica: "a", Owner: "project:train", Amount: Amount{GPUs: 4}},
{Replica: "b", Owner: "team:web", Amount: Amount{GPUs: 2, CPUCores: 3}},
{Replica: "c", Owner: "", Amount: Amount{GPUs: 8}}, // shared: nobody's
})
if u["team:ml"].GPUs != 4 || u["org:acme"].GPUs != 6 || u["org:acme"].CPUCores != 3 || u["project:train"].GPUs != 4 {
t.Errorf("usage %+v", u)
}
if _, ok := u[""]; ok {
t.Error("a shared model was counted against somebody")
}
}
func TestWhatGoesPastAGuaranteeIsBorrowedNewestFirst(t *testing.T) {
p := policy()
b := p.Borrowed([]Holding{
{Replica: "old", Owner: "team:ml", Amount: Amount{GPUs: 8}, Since: at(1)},
{Replica: "new", Owner: "team:ml", Amount: Amount{GPUs: 4}, Since: at(2)},
{Replica: "web", Owner: "team:web", Amount: Amount{GPUs: 4}, Since: at(3)},
})
// ml: 8 within its 8, then 4 past it. acme: 8 + 4 = 12 within, web's 4
// takes the org to 16, past its 12.
if b["old"] || !b["new"] || !b["web"] {
t.Errorf("borrowed %v", b)
}
}
func TestAGuaranteeIsOnlyForWhoHasOne(t *testing.T) {
p := policy()
u := map[string]Amount{"team:ml": {GPUs: 4}, "org:acme": {GPUs: 4}}
if !p.Guaranteed("project:train", u, Amount{GPUs: 4}) {
t.Error("8 of ml's 8 and 8 of acme's 12 is within both")
}
if p.Guaranteed("project:train", u, Amount{GPUs: 5}) {
t.Error("9 GPUs is past ml's 8")
}
if p.Guaranteed("", u, Amount{GPUs: 1}) || New(nil, nil).Guaranteed("team:ml", u, Amount{}) {
t.Error("an owner with no guarantee is not guaranteed anything")
}
}
func TestALimitStopsGrowthWhereverItIsSet(t *testing.T) {
p := policy()
u := map[string]Amount{"org:acme": {GPUs: 14}}
if why := p.AtLimit("team:web", u, Amount{GPUs: 2}); why != "" {
t.Errorf("16 of 16 is at the limit, not past it: %s", why)
}
why := p.AtLimit("team:web", u, Amount{GPUs: 4})
if !strings.Contains(why, "org:acme is at its capacity limit (18 GPUs of 16)") {
t.Errorf("the refusal should name the scope and the figures: %q", why)
}
if p.AtLimit("", u, Amount{GPUs: 99}) != "" {
t.Error("a shared model was limited")
}
}
+52
View File
@@ -0,0 +1,52 @@
package control
import (
"context"
"log/slog"
"sync"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// QuotaStore is where capacity quotas and the tenancy tree are kept.
type QuotaStore interface {
CapacityQuotas(ctx context.Context) ([]store.CapacityQuota, error)
ScopeParents(ctx context.Context) (map[string][]string, error)
}
// quotaRefresh is how stale the reconciler's view of quotas may get. A quota
// changed by an operator takes effect within this; the reconciler asks many
// times a pass and the store is not asked each time.
const quotaRefresh = 30 * time.Second
// QuotaPolicy returns what the reconciler asks for its quotas: the last
// policy read from the store, read again once it is older than quotaRefresh.
// A store that fails keeps the last good policy, so a database hiccup does
// not drop every guarantee and limit at once.
func QuotaPolicy(st QuotaStore, log *slog.Logger) func() *quota.Policy {
var mu sync.Mutex
var cur *quota.Policy
var at time.Time
return func() *quota.Policy {
mu.Lock()
defer mu.Unlock()
if cur != nil && time.Since(at) < quotaRefresh {
return cur
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
qs, err := st.CapacityQuotas(ctx)
if err == nil {
var parents map[string][]string
if parents, err = st.ScopeParents(ctx); err == nil {
cur, at = quota.New(qs, parents), time.Now()
return cur
}
}
log.Warn("could not read capacity quotas; keeping the last ones", "err", err)
at = time.Now() // not again on every call while the store is down
return cur
}
}
+17
View File
@@ -5,6 +5,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
)
// Headroom is how many more replicas of a model the cluster has room for now.
@@ -52,6 +53,11 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
inv := r.inv.Clone()
sched := placement.New(inv)
pol := r.policy()
var usage map[string]quota.Amount
if pol != nil {
usage = pol.Usage(r.Holdings())
}
for h.More < limit {
res := sched.Place(m.PlacementProfiles(), avoid)
if res.Placement == nil && len(avoid) > 0 {
@@ -66,6 +72,17 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
h.Blocked = slices.Compact(h.Blocked)
return h, true
}
// A quota limit is as much an edge as a full node.
if pol != nil && m.Owner != "" {
need := placementNeed(m, res.Placement)
if why := pol.AtLimit(m.Owner, usage, need); why != "" {
h.Blocked = []string{why}
return h, true
}
for _, s := range pol.Chain(m.Owner) {
usage[s] = usage[s].Plus(need)
}
}
mem, cores := 0, 0
if prof, ok := m.ProfileByName(res.Placement.Profile.Name); ok {
mem, cores = prof.MinMemoryGB, cpuCores(prof)
+10 -1
View File
@@ -83,6 +83,12 @@ type ModelHealth struct {
// place one found nowhere; Placement says why.
Unplaceable bool
Placement string
// Class is the capacity class it is treated as. Preempted counts the
// replicas it has lost to models of a higher class since the control
// plane started; HeldFor names the model it is giving room to now.
Class string
Preempted int
HeldFor string
}
// ModelHealth reports every model in the catalog.
@@ -106,7 +112,10 @@ func (r *Reconciler) ModelHealth() []ModelHealth {
out := make([]ModelHealth, 0, len(models))
for _, m := range models {
h := ModelHealth{Model: m.Name, Desired: desired[m.Name], Ready: ready[m.Name],
Failures: r.attempts[m.Name]}
Failures: r.attempts[m.Name], Class: m.EffectiveClass(), Preempted: r.preempted[m.Name]}
if hd, ok := r.held[m.Name]; ok && r.now().Before(hd.Until) {
h.HeldFor = hd.For
}
if p, ok := r.parked[m.Name]; ok {
h.Parked, h.ParkedSince, h.Reason = true, p.Since, p.Reason
}
+286
View File
@@ -0,0 +1,286 @@
package reconciler
import (
"context"
"slices"
"sort"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
)
// hold keeps a model that gave up room from taking it back.
type hold struct {
Until time.Time
For string
}
// maxVictims bounds how many replicas one placement may stop. Past this the
// model is better reported as not fitting than paid for with half a cluster.
const maxVictims = 3
// victim is a replica that could be stopped for another model.
type victim struct {
ID string
Model string
Class string
AboveMin bool
Tier int
Idle time.Duration
}
// preemptFor stops replicas of lower classes when a model cannot otherwise be
// placed, and says whether it did.
//
// What may be taken from whom:
//
// - A lower class gives way to a higher one: opportunistic to standard,
// standard to critical.
// - Within a class, only a replica above its model's minimum, and only for
// a model still below its own. Room a model grew into under load is not a
// promise; its minimum is.
// - A critical model's minimum is never taken.
//
// Victims only ever come from a lower rank, or from above a minimum for a
// model below one, so two models cannot take turns stopping each other.
//
// The set is tried on a copy of the inventory first, and only stopped if it
// would let the model be placed: stopping replicas that do not make room is
// the one outcome worse than not preempting.
func (r *Reconciler) preemptFor(ctx context.Context, m catalog.Model, avoid []string) bool {
cands := r.victims(m)
if len(cands) == 0 {
return false
}
trial := r.inv.Clone()
sched := placement.New(trial)
var chosen []victim
for _, v := range cands {
if len(chosen) == maxVictims {
break
}
trial.ReleaseReplica(v.ID)
chosen = append(chosen, v)
res := sched.Place(m.PlacementProfiles(), avoid)
if res.Placement == nil && len(avoid) > 0 {
res = sched.Place(m.PlacementProfiles(), nil)
}
if res.Placement == nil {
continue
}
r.preempt(ctx, m, chosen)
return true
}
return false
}
// preempt drains the chosen replicas and holds their models back.
func (r *Reconciler) preempt(ctx context.Context, m catalog.Model, chosen []victim) {
now := r.now()
var ids []string
r.mu.Lock()
for _, v := range chosen {
if rep, ok := r.replicas[v.ID]; ok {
rep.PreemptedFor = m.Name
}
ids = append(ids, v.ID)
r.held[v.Model] = hold{Until: now.Add(r.cfg.PreemptHold), For: m.Name}
r.preempted[v.Model]++
}
r.preempting[m.Name] = ids
r.mu.Unlock()
for _, v := range chosen {
r.log.Warn("preempting a replica to make room",
"replica", v.ID, "model", v.Model, "class", v.Class, "above_min", v.AboveMin,
"for", m.Name, "for_class", m.EffectiveClass())
}
r.scaleDown(ctx, ids)
}
// awaitingPreemption reports whether replicas stopped for a model are still
// on their way out, so it waits for them rather than stopping more.
func (r *Reconciler) awaitingPreemption(model string) bool {
r.mu.Lock()
defer r.mu.Unlock()
ids := r.preempting[model]
ids = slices.DeleteFunc(slices.Clone(ids), func(id string) bool {
_, ok := r.replicas[id]
return !ok
})
if len(ids) == 0 {
delete(r.preempting, model)
return false
}
r.preempting[model] = ids
return true
}
// isHeld reports whether a model gave up room and may not take it back yet.
// The hold ends early once the model it was given to has what it asked for.
//
// Today a preempted replica is only removed by its own model's drain loop,
// which runs after that model has decided not to scale up, so the room it
// frees reaches the model it was freed for first anyway. The hold is what
// keeps that true for any other way a replica can go — a stop reported
// between passes, a reap at the start of one — rather than it resting on the
// order of lines in reconcileModel.
func (r *Reconciler) isHeld(model string) bool {
r.mu.Lock()
h, ok := r.held[model]
r.mu.Unlock()
if !ok {
return false
}
if r.now().After(h.Until) {
r.mu.Lock()
delete(r.held, model)
r.mu.Unlock()
return false
}
if to, ok := r.cat.Load().Resolve(h.For); ok && r.liveCount(h.For) >= r.desiredFor(to) {
r.mu.Lock()
delete(r.held, model)
r.mu.Unlock()
return false
}
return true
}
// liveCount is how many replicas of a model hold capacity.
func (r *Reconciler) liveCount(model string) int {
r.mu.Lock()
defer r.mu.Unlock()
n := 0
for _, rep := range r.replicas {
if rep.Model == model && rep.State.Counts() && rep.RetiringAt.IsZero() {
n++
}
}
return n
}
// victims lists the replicas that may be stopped for m, in the order they
// should be: idle on-demand ones first, then what other tenants borrowed past
// their guarantees when m's owner is within its own, then what lower classes
// grew into under load, then the same within m's class, then lower classes'
// minimums; within each, the longest unused first.
//
// A replica within its owner's guarantee is taken only for a model of the
// same owner: a guarantee that another tenant's higher class could take is
// not one.
func (r *Reconciler) victims(m catalog.Model) []victim {
cat := r.cat.Load()
now := r.now()
myRank := catalog.ClassRank(m.EffectiveClass())
belowMin := r.liveCount(m.Name) < m.Scaling.Min
pol := r.policy()
var borrowed map[string]bool
reclaim := false
if pol != nil {
hs := r.Holdings()
borrowed = pol.Borrowed(hs)
reclaim = pol.Guaranteed(m.Owner, pol.Usage(hs), profileNeed(m))
}
r.mu.Lock()
defer r.mu.Unlock()
// Per model, its live replicas newest first: the newest are the ones
// above its minimum.
byModel := map[string][]*Replica{}
for _, rep := range r.replicas {
if rep.Model == m.Name || !rep.State.Counts() || !rep.RetiringAt.IsZero() || rep.PreemptedFor != "" {
continue
}
byModel[rep.Model] = append(byModel[rep.Model], rep)
}
var out []victim
for model, reps := range byModel {
other, ok := cat.Resolve(model)
if !ok {
continue
}
class := other.EffectiveClass()
rank := catalog.ClassRank(class)
sort.Slice(reps, func(i, j int) bool { return reps[i].CreatedAt.After(reps[j].CreatedAt) })
above := max(len(reps)-other.Scaling.Min, 0)
for i, rep := range reps {
aboveMin := i < above
tier := -1
switch {
case rank < myRank && other.Scaling.OnDemand() && idleFor(rep, now) >= r.cfg.EvictionGrace:
tier = 0
case rank < myRank && aboveMin:
tier = 2
case rank == myRank && aboveMin && belowMin:
tier = 3
case rank < myRank && class != catalog.ClassCritical:
tier = 4
}
if pol != nil && other.Owner != m.Owner {
switch {
case borrowed[rep.ID] && reclaim && (class != catalog.ClassCritical || aboveMin):
// Past its owner's guarantee, wanted by an owner within
// its own: taken back whatever the classes.
if tier != 0 {
tier = 1
}
case !borrowed[rep.ID] && guaranteed(pol, other.Owner):
continue // held within its owner's guarantee
}
}
if tier < 0 {
continue
}
// Just started: stopping it throws away a model load that has
// only now been paid for.
if !rep.ReadyAt.IsZero() && now.Sub(rep.ReadyAt) < r.cfg.MinLifetime {
continue
}
if len(nodesOf(rep)) > 1 {
continue // see makeRoomFor: a multi-node replica is its own decision
}
out = append(out, victim{ID: rep.ID, Model: model, Class: class,
AboveMin: aboveMin, Tier: tier, Idle: idleFor(rep, now)})
}
}
sort.SliceStable(out, func(i, j int) bool {
if out[i].Tier != out[j].Tier {
return out[i].Tier < out[j].Tier
}
return out[i].Idle > out[j].Idle
})
return out
}
// idleFor is how long a replica has gone unused; zero when nobody has said,
// which is not the same as idle.
func idleFor(rep *Replica, now time.Time) time.Duration {
if rep.LastUsedAt.IsZero() {
return 0
}
return now.Sub(rep.LastUsedAt)
}
func nodesOf(rep *Replica) map[string]struct{} {
out := map[string]struct{}{}
for _, sh := range rep.Shards {
out[sh.NodeID] = struct{}{}
}
return out
}
// guaranteed reports whether an owner has a guarantee anywhere up its tree.
func guaranteed(pol *quota.Policy, owner string) bool {
for _, q := range pol.Governing(owner) {
if q.Guarantee.Any() {
return true
}
}
return false
}
+237
View File
@@ -0,0 +1,237 @@
package reconciler
import (
"log/slog"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
)
// newFleetHarness is newHarness with a catalog of several models.
func newFleetHarness(t *testing.T, models []catalog.Model, nodes ...inventory.Node) *harness {
t.Helper()
cat, err := catalog.New(models)
if err != nil {
t.Fatal(err)
}
inv := inventory.NewStore()
c := newClock()
for _, n := range nodes {
inv.Upsert(n)
}
disp := &fakeDispatcher{}
cfg := DefaultConfig()
cfg.UnreachableAfter = 15 * time.Second
cfg.LostAfter = 60 * time.Second
r := New(cfg, catalog.NewLive(cat), inv, disp, slog.New(slog.DiscardHandler))
r.SetClock(c.now)
c.advance(cfg.SettleFor)
for _, n := range nodes {
touchAt(inv, n.ID, c.now())
}
return &harness{r: r, inv: inv, disp: disp, clock: c}
}
// gpuModel runs on one whole 8-GPU node.
func gpuModel(name, class string, min int) catalog.Model {
return catalog.Model{Name: name, Source: "/models/" + name, Class: class,
Scaling: catalog.Scaling{Min: min, Max: min},
Profiles: []catalog.Profile{{Name: "tp8", Engine: "vllm", GPUClass: "h100-80g", TP: 8}}}
}
// settle runs passes until nothing changes, marking what starts ready and
// letting preempted replicas run out their grace.
func (h *harness) settle(t *testing.T, passes int) {
t.Helper()
for range passes {
h.r.Reconcile(t.Context())
h.markReady()
h.advance(h.r.cfg.PreemptGrace + time.Second)
}
}
func replicasOf(h *harness, model string) (live int) {
for _, rep := range h.r.Replicas() {
if rep.Model == model && rep.State.Counts() {
live++
}
}
return live
}
// fill places the standard model, then ages it past MinLifetime so it may be
// taken from, and only then adds the model that wants its node.
func fillThenWant(t *testing.T, want catalog.Model, have catalog.Model, nodes ...inventory.Node) *harness {
t.Helper()
h := newFleetHarness(t, []catalog.Model{have, withMin(want, 0)}, nodes...)
h.settle(t, 3)
if got := replicasOf(h, have.Name); got != have.Scaling.Min {
t.Fatalf("%s has %d replicas before anything wanted its room", have.Name, got)
}
h.advance(h.r.cfg.MinLifetime + time.Minute)
cat, err := catalog.New([]catalog.Model{have, want})
if err != nil {
t.Fatal(err)
}
h.r.cat.Store(cat)
return h
}
func withMin(m catalog.Model, n int) catalog.Model {
m.Scaling = catalog.Scaling{Min: n, Max: n}
return m
}
func TestACriticalModelTakesAStandardModelsNode(t *testing.T) {
h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
h.r.Reconcile(t.Context())
var draining int
for _, rep := range h.r.Replicas() {
if rep.Model == "chat" && rep.State == Draining && rep.PreemptedFor == "vital" {
draining++
}
}
if draining != 1 {
t.Fatalf("the standard replica was not drained for the critical one: %+v", h.r.Replicas())
}
if hs := health(h, "chat"); hs.HeldFor != "vital" || hs.Preempted != 1 {
t.Errorf("the victim's health does not say it gave way: %+v", hs)
}
h.settle(t, 4)
if replicasOf(h, "vital") != 1 {
t.Fatalf("the critical model was not placed once room was made: %+v", h.r.Replicas())
}
// Held back until the critical model had its replica, then short again
// because there is no room: it must not have taken the node back — not
// even for a moment, which would load a model only to preempt it again.
if replicasOf(h, "chat") != 0 {
t.Error("the standard model took the node back")
}
started := 0
for _, cmd := range h.disp.startedShards() {
if strings.HasPrefix(cmd.ReplicaID, "chat-") {
started++
}
}
if started != 1 {
t.Errorf("the standard model started %d replicas; it should not have tried again", started)
}
}
func TestAStandardModelDoesNotTakeAnothersMinimum(t *testing.T) {
h := fillThenWant(t, gpuModel("second", "", 1), gpuModel("first", "", 1), gpuNode("1", 8))
h.settle(t, 3)
if replicasOf(h, "first") != 1 || replicasOf(h, "second") != 0 {
t.Errorf("a model of the same class took another's minimum: %+v", h.r.Replicas())
}
if hs := health(h, "second"); !hs.Unplaceable {
t.Error("the model left without room should say it cannot be placed")
}
}
func TestWithinAClassOnlyWhatWasGrownIntoIsTaken(t *testing.T) {
grown := gpuModel("busy", "", 1)
grown.Scaling = catalog.Scaling{Min: 1, Max: 2}
h := newFleetHarness(t, []catalog.Model{grown, withMin(gpuModel("new", "", 1), 0)}, gpuNode("1", 8), gpuNode("2", 8))
h.settle(t, 2)
// Load grows it to its max: the second replica is above its minimum.
h.r.mu.Lock()
h.r.extra["busy"] = 1
h.r.mu.Unlock()
h.settle(t, 3)
if replicasOf(h, "busy") != 2 {
t.Fatalf("setup: busy did not grow to 2: %+v", h.r.Replicas())
}
h.advance(h.r.cfg.MinLifetime + time.Minute)
cat, _ := catalog.New([]catalog.Model{grown, gpuModel("new", "", 1)})
h.r.cat.Store(cat)
h.settle(t, 5)
if replicasOf(h, "new") != 1 || replicasOf(h, "busy") != 1 {
t.Errorf("the grown replica should give way to a minimum: busy %d, new %d",
replicasOf(h, "busy"), replicasOf(h, "new"))
}
}
func TestNothingIsStoppedThatWouldNotMakeRoom(t *testing.T) {
// Two nodes, each half used by an opportunistic model; the critical one
// needs a whole node. Stopping one frees half a node, which is no use —
// and both together are on two nodes, which is no use either.
half := catalog.Model{Name: "half", Source: "/m", Class: catalog.ClassOpportunistic,
Scaling: catalog.Scaling{Min: 2, Max: 2},
Profiles: []catalog.Profile{{Name: "tp4", Engine: "vllm", GPUClass: "h100-80g", TP: 4}}}
five := func(id string) inventory.Node { return gpuNode(id, 5) }
h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), half, five("1"), five("2"))
h.r.Reconcile(t.Context())
for _, rep := range h.r.Replicas() {
if rep.PreemptedFor != "" || rep.State == Draining {
t.Fatalf("a replica was stopped without making room: %+v", rep)
}
}
}
func TestACriticalMinimumIsNeverTaken(t *testing.T) {
h := fillThenWant(t, gpuModel("also-vital", catalog.ClassCritical, 1), gpuModel("vital", catalog.ClassCritical, 1), gpuNode("1", 8))
h.settle(t, 3)
if replicasOf(h, "vital") != 1 {
t.Error("a critical model's minimum was taken")
}
}
func TestAReplicaThatJustStartedIsLeftAlone(t *testing.T) {
h := newFleetHarness(t, []catalog.Model{gpuModel("chat", "", 1), withMin(gpuModel("vital", catalog.ClassCritical, 1), 0)}, gpuNode("1", 8))
h.settle(t, 2)
cat, _ := catalog.New([]catalog.Model{gpuModel("chat", "", 1), gpuModel("vital", catalog.ClassCritical, 1)})
h.r.cat.Store(cat)
h.r.Reconcile(t.Context())
for _, rep := range h.r.Replicas() {
if rep.State == Draining {
t.Fatal("a replica that had only just loaded was preempted")
}
}
}
func health(h *harness, model string) ModelHealth {
for _, x := range h.r.ModelHealth() {
if x.Model == model {
return x
}
}
return ModelHealth{}
}
func TestAHoldLastsUntilTheRoomIsUsedOrItExpires(t *testing.T) {
h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
h.r.Reconcile(t.Context())
if !h.r.isHeld("chat") {
t.Fatal("the model that gave way is not held")
}
if h.r.isHeld("vital") {
t.Error("the model room was made for is held")
}
// Released as soon as the model it was given to has what it asked for.
h.settle(t, 3)
if replicasOf(h, "vital") != 1 {
t.Fatalf("setup: vital was not placed: %+v", h.r.Replicas())
}
if h.r.isHeld("chat") {
t.Error("the hold outlasted its purpose")
}
// And, whatever happens, after PreemptHold.
h.r.mu.Lock()
h.r.held["other"] = hold{Until: h.r.now().Add(time.Minute), For: "never-placed"}
h.r.mu.Unlock()
if !h.r.isHeld("other") {
t.Fatal("a fresh hold is not in force")
}
h.advance(2 * time.Minute)
if h.r.isHeld("other") {
t.Error("a hold outlived its time")
}
}
+105
View File
@@ -0,0 +1,105 @@
package reconciler
import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// SetQuotas supplies the tenancy tree's capacity quotas, read afresh each
// time it is called. Without it nothing is guaranteed or limited, and
// preemption goes by class alone.
func (r *Reconciler) SetQuotas(f func() *quota.Policy) {
r.mu.Lock()
defer r.mu.Unlock()
r.quotas = f
}
func (r *Reconciler) policy() *quota.Policy {
r.mu.Lock()
f := r.quotas
r.mu.Unlock()
if f == nil {
return nil
}
return f()
}
// Holdings is what every replica holds, and for whom.
func (r *Reconciler) Holdings() []quota.Holding {
cat := r.cat.Load()
r.mu.Lock()
defer r.mu.Unlock()
out := make([]quota.Holding, 0, len(r.replicas))
for _, rep := range r.replicas {
m, ok := cat.Resolve(rep.Model)
if !ok {
continue
}
prof, _ := m.ProfileByName(rep.Profile)
var a quota.Amount
for _, sh := range rep.Shards {
a = a.Plus(quota.Amount{GPUs: len(sh.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
}
out = append(out, quota.Holding{Replica: rep.ID, Owner: m.Owner, Amount: a, Since: rep.CreatedAt})
}
return out
}
// placementNeed is what a placement would hold.
func placementNeed(m catalog.Model, pl *placement.Placement) quota.Amount {
prof, _ := m.ProfileByName(pl.Profile.Name)
var a quota.Amount
for _, n := range pl.Nodes {
a = a.Plus(quota.Amount{GPUs: len(n.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
}
return a
}
// profileNeed is what a replica of a model's first profile would hold, for
// deciding before there is a placement.
func profileNeed(m catalog.Model) quota.Amount {
if len(m.Profiles) == 0 {
return quota.Amount{}
}
p := m.Profiles[0]
nodes := max(p.Nodes, 1)
gpus := p.GPUsPerNode
if gpus == 0 && !p.CPUOnly() {
gpus = max(p.TP, 1)
}
return quota.Amount{GPUs: gpus * nodes, CPUCores: cpuCores(p) * nodes, MemoryGB: p.MinMemoryGB * nodes}
}
// overLimit says why a placement would take its model's owner past a limit.
func (r *Reconciler) overLimit(m catalog.Model, pl *placement.Placement) string {
pol := r.policy()
if pol == nil || m.Owner == "" {
return ""
}
return pol.AtLimit(m.Owner, pol.Usage(r.Holdings()), placementNeed(m, pl))
}
// QuotaUse is one scope's quota beside what it holds.
type QuotaUse struct {
Scope string
Guarantee store.Resources
Limit store.Resources
Holds quota.Amount
}
// QuotaUse lists every quota with what its scope holds now.
func (r *Reconciler) QuotaUse() []QuotaUse {
pol := r.policy()
if pol == nil {
return nil
}
usage := pol.Usage(r.Holdings())
var out []QuotaUse
for _, q := range pol.Quotas() {
key := quota.Key(q.Scope, q.ScopeID)
out = append(out, QuotaUse{Scope: key, Guarantee: q.Guarantee, Limit: q.Limit, Holds: usage[key]})
}
return out
}
@@ -0,0 +1,81 @@
package reconciler
import (
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
func owned(m catalog.Model, owner string) catalog.Model { m.Owner = owner; return m }
func gpus(n int) store.Resources { return store.Resources{GPUs: &n} }
// Two teams in one org; each 8-GPU node holds one tp8 replica.
func withQuotas(h *harness, qs ...store.CapacityQuota) {
pol := quota.New(qs, map[string][]string{"team:ml": {"org:acme"}, "team:web": {"org:acme"}})
h.r.SetQuotas(func() *quota.Policy { return pol })
}
func TestATenantWithinItsGuaranteeTakesBackWhatAnotherBorrowed(t *testing.T) {
// web is guaranteed one node and borrowed the second; ml is guaranteed
// one node and wants it. Same class, and web's replica is at its minimum
// — by class alone nothing could be taken.
web := owned(gpuModel("web-chat", "", 2), "team:web")
ml := owned(gpuModel("ml-train", "", 1), "team:ml")
h := newFleetHarness(t, []catalog.Model{web, withMin(ml, 0)}, gpuNode("1", 8), gpuNode("2", 8))
withQuotas(h,
store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "web", Guarantee: gpus(8)},
store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "ml", Guarantee: gpus(8)})
h.settle(t, 4)
if replicasOf(h, "web-chat") != 2 {
t.Fatalf("setup: web did not borrow the second node: %+v", h.r.Replicas())
}
h.advance(h.r.cfg.MinLifetime + time.Minute)
cat, _ := catalog.New([]catalog.Model{web, ml})
h.r.cat.Store(cat)
h.settle(t, 5)
if replicasOf(h, "ml-train") != 1 || replicasOf(h, "web-chat") != 1 {
t.Errorf("the borrowed node was not taken back: web %d, ml %d",
replicasOf(h, "web-chat"), replicasOf(h, "ml-train"))
}
}
func TestAReplicaWithinItsGuaranteeIsNotTakenByAHigherClassElsewhere(t *testing.T) {
web := owned(gpuModel("web-chat", "", 1), "team:web")
vital := owned(gpuModel("vital", catalog.ClassCritical, 1), "team:ml")
h := fillThenWant(t, vital, web, gpuNode("1", 8))
withQuotas(h, store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "web", Guarantee: gpus(8)})
h.settle(t, 3)
if replicasOf(h, "web-chat") != 1 || replicasOf(h, "vital") != 0 {
t.Errorf("a guaranteed replica was preempted by another tenant's class: %+v", h.r.Replicas())
}
// Without the guarantee, critical takes it: the guarantee was the reason.
h2 := fillThenWant(t, vital, web, gpuNode("1", 8))
h2.settle(t, 3)
if replicasOf(h2, "vital") != 1 {
t.Error("setup: without a guarantee the critical model should have taken the node")
}
}
func TestALimitStopsAModelBeforeItStarts(t *testing.T) {
web := owned(gpuModel("web-chat", "", 2), "team:web")
h := newFleetHarness(t, []catalog.Model{web}, gpuNode("1", 8), gpuNode("2", 8))
withQuotas(h, store.CapacityQuota{Scope: store.ScopeOrg, ScopeID: "acme", Limit: gpus(8)})
h.settle(t, 4)
if replicasOf(h, "web-chat") != 1 {
t.Errorf("a second replica went past the org's 8-GPU limit: %+v", h.r.Replicas())
}
hs := health(h, "web-chat")
if !hs.Unplaceable || !strings.Contains(hs.Placement, "org:acme is at its capacity limit (16 GPUs of 8)") {
t.Errorf("the refusal should name the limit: %+v", hs)
}
room, _ := h.r.Headroom("web-chat", 5)
if room.More != 0 || len(room.Blocked) != 1 || !strings.Contains(room.Blocked[0], "capacity limit") {
t.Errorf("headroom should stop at the limit, not the empty node: %+v", room)
}
}
+47 -1
View File
@@ -15,6 +15,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
)
// ShardCommand is everything an agent needs to launch one shard.
@@ -65,6 +66,12 @@ type Config struct {
// Without a bound, draining a node whose model cannot be placed anywhere
// else would never finish.
DrainGrace time.Duration
// PreemptGrace is how long a replica stopped to make room for another
// is given to finish what it is serving: a request, not a replacement.
PreemptGrace time.Duration
// PreemptHold is how long a model that gave up room may not take it back,
// so the replica it was given up for is placed first.
PreemptHold time.Duration
// ScaleUpWaiting is the average number of requests waiting per ready
// replica, held for ScaleUpAfter, that adds a replica — up to the model's
// max. ScaleDownAfter is how long with nothing waiting before one of the
@@ -141,6 +148,8 @@ func DefaultConfig() Config {
EvictionGrace: 2 * time.Minute,
MinLifetime: 5 * time.Minute,
DrainGrace: 5 * time.Minute,
PreemptGrace: 30 * time.Second,
PreemptHold: 5 * time.Minute,
ParkAfter: 5,
ScaleUpWaiting: 1,
ScaleUpAfter: time.Minute,
@@ -158,6 +167,12 @@ func (c *Config) normalize() {
if c.BaseBackoff <= 0 {
c.BaseBackoff = d.BaseBackoff
}
if c.PreemptGrace <= 0 {
c.PreemptGrace = d.PreemptGrace
}
if c.PreemptHold <= 0 {
c.PreemptHold = d.PreemptHold
}
if c.MaxBackoff < c.BaseBackoff {
c.MaxBackoff = max(c.BaseBackoff, d.MaxBackoff)
}
@@ -250,6 +265,14 @@ type Reconciler struct {
// placementLogged remembers the last "no placement" said per model, so it
// is said when it changes rather than on every five-second pass.
placementLogged map[string]placementNote
// preempting is, per model, the replicas stopped to make room for it and
// not gone yet; held is, per model that gave them up, until when it may
// not take the room back; preempted counts, per model, replicas it lost.
preempting map[string][]string
held map[string]hold
preempted map[string]int
// quotas is the tenancy tree's capacity quotas as last read; nil is none.
quotas func() *quota.Policy
wake chan struct{}
stop chan struct{}
@@ -285,6 +308,9 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
quietSince: map[string]time.Time{},
placementLogged: map[string]placementNote{},
lastExplain: map[string]placement.Result{},
preempting: map[string][]string{},
held: map[string]hold{},
preempted: map[string]int{},
wake: make(chan struct{}, 1),
stop: make(chan struct{}),
done: make(chan struct{}),
@@ -704,6 +730,9 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
switch {
case len(liveIDs) < desired && isParked:
// Not placed. Its failure is on every status the operator reads.
case len(liveIDs) < desired && r.isHeld(m.Name):
// It gave up room to a model of a higher class, which is placed
// first; taking it back now would undo the preemption.
case len(liveIDs) < desired:
if r.now().Before(blockedUntil) {
// Still in backoff. Rebuilding immediately would hot-loop on a
@@ -725,7 +754,11 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
rep, ok := r.replicas[id]
var expired bool
if ok {
expired = !rep.DrainingAt.IsZero() && r.now().Sub(rep.DrainingAt) > r.cfg.DrainGrace
grace := r.cfg.DrainGrace
if rep.PreemptedFor != "" {
grace = r.cfg.PreemptGrace
}
expired = !rep.DrainingAt.IsZero() && r.now().Sub(rep.DrainingAt) > grace
}
r.mu.Unlock()
@@ -775,9 +808,22 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
if m.Scaling.OnDemand() && r.makeRoomFor(ctx, m) {
return
}
if r.awaitingPreemption(m.Name) || r.preemptFor(ctx, m, avoid) {
return
}
r.notePlacementFailure(m.Name, res.Explain())
return
}
if why := r.overLimit(m, res.Placement); why != "" {
// Recorded as the placement's answer, so the model reads as not
// placeable, and says why, wherever that is shown.
r.mu.Lock()
r.lastExplain[m.Name] = placement.Result{Rejections: []placement.Rejection{
{Profile: res.Placement.Profile.Name, Reason: why}}}
r.mu.Unlock()
r.notePlacementFailure(m.Name, why)
return
}
if err := r.startReplica(ctx, m, res.Placement); err != nil {
r.log.Error("starting replica failed", "model", m.Name, "err", err)
}
+2
View File
@@ -156,6 +156,8 @@ type Replica struct {
// DrainingAt is when the replica stopped taking new work, so the wait for a
// replacement can be bounded.
DrainingAt time.Time
// PreemptedFor is the model this replica was stopped to make room for.
PreemptedFor string
// RetiringAt is set while the replica is being restarted: it keeps serving
// but no longer counts toward the model's replicas, so a replacement is
// placed beside it, and it is drained only once that replacement is ready.
+3
View File
@@ -29,6 +29,9 @@ var copyOrder = []string{
"key_claims",
"sessions",
"limits",
// Beside limits, and like them keyed by scope with no foreign key: a
// quota on a scope that is later removed is inert, not an error.
"capacity_quotas",
"usage",
"models",
// No foreign key to models, deliberately: a measurement outlives the
@@ -0,0 +1,19 @@
-- Capacity a part of the tenancy tree is guaranteed, and the most it may hold.
--
-- Beside limits rather than in it: limits are counted on the request path, per
-- key, in requests and tokens; these are counted by the reconciler, per model,
-- in the GPUs, cores and memory its replicas hold. A NULL is a resource this
-- row does not govern: guaranteed nothing, or limited by nothing.
CREATE TABLE capacity_quotas (
scope_type TEXT NOT NULL,
scope_id TEXT NOT NULL,
gpus INTEGER,
cpu_cores INTEGER,
memory_gb INTEGER,
limit_gpus INTEGER,
limit_cpu_cores INTEGER,
limit_memory_gb INTEGER,
updated_at BIGINT NOT NULL,
updated_by TEXT NOT NULL,
PRIMARY KEY (scope_type, scope_id)
);
@@ -0,0 +1,19 @@
-- Capacity a part of the tenancy tree is guaranteed, and the most it may hold.
--
-- Beside limits rather than in it: limits are counted on the request path, per
-- key, in requests and tokens; these are counted by the reconciler, per model,
-- in the GPUs, cores and memory its replicas hold. A NULL is a resource this
-- row does not govern: guaranteed nothing, or limited by nothing.
CREATE TABLE capacity_quotas (
scope_type TEXT NOT NULL,
scope_id TEXT NOT NULL,
gpus INTEGER,
cpu_cores INTEGER,
memory_gb INTEGER,
limit_gpus INTEGER,
limit_cpu_cores INTEGER,
limit_memory_gb INTEGER,
updated_at BIGINT NOT NULL,
updated_by TEXT NOT NULL,
PRIMARY KEY (scope_type, scope_id)
);
+127
View File
@@ -0,0 +1,127 @@
package store
import (
"context"
"database/sql"
"fmt"
"time"
)
// Resources are amounts of the three things a replica holds of a cluster.
// Nil is not governed: nothing guaranteed, or nothing limited.
type Resources struct {
GPUs *int `json:"gpus,omitempty"`
CPUCores *int `json:"cpu_cores,omitempty"`
MemoryGB *int `json:"memory_gb,omitempty"`
}
// Any reports whether any resource is governed.
func (r Resources) Any() bool { return r.GPUs != nil || r.CPUCores != nil || r.MemoryGB != nil }
// CapacityQuota is what one part of the tenancy tree is guaranteed of the
// cluster, and the most it may hold.
type CapacityQuota struct {
Scope Scope `json:"scope"`
ScopeID string `json:"scope_id"`
// Guarantee is held for the scope: while it holds less, what others
// borrowed beyond their own guarantees is taken back for it.
Guarantee Resources `json:"guarantee"`
// Limit is the most it may hold, borrowing included.
Limit Resources `json:"limit"`
UpdatedAt time.Time `json:"updated_at"`
UpdatedBy string `json:"updated_by"`
}
// PutCapacityQuota sets a scope's quota, replacing what was there.
func (s *DB) PutCapacityQuota(ctx context.Context, q CapacityQuota) error {
if !validMembershipScope(q.Scope) || q.ScopeID == "" {
return fmt.Errorf("a capacity quota belongs to an organization, team or project")
}
for _, v := range []*int{q.Guarantee.GPUs, q.Guarantee.CPUCores, q.Guarantee.MemoryGB,
q.Limit.GPUs, q.Limit.CPUCores, q.Limit.MemoryGB} {
if v != nil && *v < 0 {
return fmt.Errorf("a quota cannot be negative")
}
}
if q.UpdatedAt.IsZero() {
q.UpdatedAt = time.Now()
}
_, err := s.db.ExecContext(ctx,
`INSERT INTO capacity_quotas (scope_type, scope_id, gpus, cpu_cores, memory_gb,
limit_gpus, limit_cpu_cores, limit_memory_gb, updated_at, updated_by)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT (scope_type, scope_id) DO UPDATE SET
gpus = excluded.gpus, cpu_cores = excluded.cpu_cores, memory_gb = excluded.memory_gb,
limit_gpus = excluded.limit_gpus, limit_cpu_cores = excluded.limit_cpu_cores,
limit_memory_gb = excluded.limit_memory_gb,
updated_at = excluded.updated_at, updated_by = excluded.updated_by`,
string(q.Scope), q.ScopeID,
nullInt(q.Guarantee.GPUs), nullInt(q.Guarantee.CPUCores), nullInt(q.Guarantee.MemoryGB),
nullInt(q.Limit.GPUs), nullInt(q.Limit.CPUCores), nullInt(q.Limit.MemoryGB),
q.UpdatedAt.UnixMilli(), q.UpdatedBy)
return err
}
// CapacityQuotas lists every quota.
func (s *DB) CapacityQuotas(ctx context.Context) ([]CapacityQuota, error) {
return queryAll(ctx, s.db,
`SELECT scope_type, scope_id, gpus, cpu_cores, memory_gb,
limit_gpus, limit_cpu_cores, limit_memory_gb, updated_at, updated_by
FROM capacity_quotas ORDER BY scope_type, scope_id`, nil,
func(rows *sql.Rows) (CapacityQuota, error) {
var q CapacityQuota
var scope string
var g [6]sql.NullInt64
var at int64
err := rows.Scan(&scope, &q.ScopeID, &g[0], &g[1], &g[2], &g[3], &g[4], &g[5], &at, &q.UpdatedBy)
q.Scope = Scope(scope)
q.Guarantee = Resources{GPUs: intOf(g[0]), CPUCores: intOf(g[1]), MemoryGB: intOf(g[2])}
q.Limit = Resources{GPUs: intOf(g[3]), CPUCores: intOf(g[4]), MemoryGB: intOf(g[5])}
q.UpdatedAt = time.UnixMilli(at)
return q, err
})
}
// DeleteCapacityQuota removes a scope's quota.
func (s *DB) DeleteCapacityQuota(ctx context.Context, scope Scope, id string) error {
_, err := s.db.ExecContext(ctx,
`DELETE FROM capacity_quotas WHERE scope_type = ? AND scope_id = ?`, string(scope), id)
return err
}
// ScopeParents maps every team and project to the scopes above it, nearest
// first: a project to its team and organization, a team to its organization.
// Keys and values are written scope:id.
func (s *DB) ScopeParents(ctx context.Context) (map[string][]string, error) {
out := map[string][]string{}
teams, err := s.AllTeams(ctx)
if err != nil {
return nil, err
}
orgOf := map[string]string{}
for _, t := range teams {
orgOf[t.ID] = t.OrgID
out["team:"+t.ID] = []string{"org:" + t.OrgID}
}
type pt struct{ project, team string }
rows, err := queryAll(ctx, s.db, `SELECT id, team_id FROM projects`, nil,
func(r *sql.Rows) (pt, error) {
var p pt
return p, r.Scan(&p.project, &p.team)
})
if err != nil {
return nil, err
}
for _, p := range rows {
out["project:"+p.project] = []string{"team:" + p.team, "org:" + orgOf[p.team]}
}
return out, nil
}
func intOf(n sql.NullInt64) *int {
if !n.Valid {
return nil
}
v := int(n.Int64)
return &v
}
+63
View File
@@ -0,0 +1,63 @@
package store
import "testing"
func ip(v int) *int { return &v }
func TestAQuotaKeepsWhatItGovernsAndNothingElse(t *testing.T) {
s := tenancyStore(t)
org, _ := s.CreateOrg(t.Context(), "Acme")
q := CapacityQuota{Scope: ScopeOrg, ScopeID: org, UpdatedBy: "ops",
Guarantee: Resources{GPUs: ip(8)}, Limit: Resources{GPUs: ip(16), CPUCores: ip(0)}}
if err := s.PutCapacityQuota(t.Context(), q); err != nil {
t.Fatal(err)
}
all, err := s.CapacityQuotas(t.Context())
if err != nil || len(all) != 1 {
t.Fatalf("%v %+v", err, all)
}
got := all[0]
if *got.Guarantee.GPUs != 8 || got.Guarantee.CPUCores != nil || got.Guarantee.MemoryGB != nil {
t.Errorf("guarantee %+v: only GPUs were set", got.Guarantee)
}
// Zero is a limit of nothing, not an absent one.
if *got.Limit.GPUs != 16 || got.Limit.CPUCores == nil || *got.Limit.CPUCores != 0 {
t.Errorf("limit %+v", got.Limit)
}
q.Guarantee = Resources{MemoryGB: ip(64)}
if err := s.PutCapacityQuota(t.Context(), q); err != nil {
t.Fatal(err)
}
all, _ = s.CapacityQuotas(t.Context())
if len(all) != 1 || all[0].Guarantee.GPUs != nil || *all[0].Guarantee.MemoryGB != 64 {
t.Errorf("a second put should replace the first: %+v", all)
}
if err := s.DeleteCapacityQuota(t.Context(), ScopeOrg, org); err != nil {
t.Fatal(err)
}
if all, _ = s.CapacityQuotas(t.Context()); len(all) != 0 {
t.Error("the quota survived its deletion")
}
if err := s.PutCapacityQuota(t.Context(), CapacityQuota{Scope: ScopeOrg, ScopeID: org,
Guarantee: Resources{GPUs: ip(-1)}}); err == nil {
t.Error("a negative quota was stored")
}
}
func TestEveryScopeKnowsWhatIsAboveIt(t *testing.T) {
s := tenancyStore(t)
org, _ := s.CreateOrg(t.Context(), "Acme")
team, _ := s.CreateTeam(t.Context(), org, "ml")
proj, _ := s.CreateProject(t.Context(), team, "training")
p, err := s.ScopeParents(t.Context())
if err != nil {
t.Fatal(err)
}
want := []string{"team:" + team, "org:" + org}
if got := p["project:"+proj]; len(got) != 2 || got[0] != want[0] || got[1] != want[1] {
t.Errorf("project's parents %v, expected %v", got, want)
}
if got := p["team:"+team]; len(got) != 1 || got[0] != "org:"+org {
t.Errorf("team's parents %v", got)
}
}
+2 -1
View File
@@ -80,12 +80,13 @@
<h2>Models</h2>
{{if .Models}}
<table class="rows"><tr><th>Model</th><th>Aliases</th><th class="num">Ready</th><th class="num">Wanted</th><th>Reserves</th><th>Profiles</th>{{if .AnyFallbacks}}<th>Fallbacks</th>{{end}}<th>What it is for</th></tr>
<table class="rows"><tr><th>Model</th><th>Aliases</th><th class="num">Ready</th><th class="num">Wanted</th><th>Class</th><th>Reserves</th><th>Profiles</th>{{if .AnyFallbacks}}<th>Fallbacks</th>{{end}}<th>What it is for</th></tr>
{{range .Models}}<tr>
<td><code>{{.Name}}</code></td>
<td>{{join .Aliases}}</td>
<td class="num">{{.Ready}}</td>
<td class="num">{{.Min}}–{{.Max}}</td>
<td{{if .HeldFor}} class="warn" title="gave room to {{.HeldFor}}"{{end}}>{{.Class}}{{if .Preempted}} <span class="note" title="replicas lost to a higher class">−{{.Preempted}}</span>{{end}}</td>
<td{{with .Warnings}} class="warn" title="{{join .}}"{{end}}>{{if or .MemoryGB .CPUCores}}{{.MemoryGB}}G{{if .CPUCores}} · {{.CPUCores}}c{{end}}{{else}}—{{end}}
{{- if .MeasuredMemoryGB}} <span class="note">{{printf "%.1f" .MeasuredMemoryGB}}G used</span>{{end}}{{if .Warnings}} ⚠{{end}}</td>
<td>{{join .Profiles}}</td>