Guarantee and limit capacity per part of the tenancy tree
A model has an owner (org, team or project). Quotas guarantee a scope GPUs, cores and memory and may limit it; what a scope holds is summed from its models' replicas and those beneath it. A replica within its owner's guarantee is not taken for another scope; what is held beyond a guarantee is borrowed and taken back for a scope within its own. A limit stops a model before it starts, and headroom stops there too. ctl quota and ctl model owner set them; metrics and a CapacityAtLimit alert report them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
ac24e204a6
commit
f009bc5861
28 files changed
+1475
-16
No files matched your search
@@ -81,6 +81,31 @@ it gave way to has its replicas, so it does not take the room straight back.
|
||||
cluster page shows the class and how many replicas were lost; the metric is
|
||||
`llmgw_model_preempted_total{model,class}`, and every preemption is logged.
|
||||
|
||||
### Capacity quotas for the tenancy tree
|
||||
|
||||
An organization, team or project can be guaranteed GPUs, CPU cores and
|
||||
memory, and limited to at most some amount of each. What it holds is what the
|
||||
replicas of the models it owns hold — and those of the teams and projects
|
||||
beneath it. `llmgw ctl model owner MODEL team:Acme/ml` says whose a model is;
|
||||
one with no owner is shared and counts against nobody.
|
||||
|
||||
- **A guarantee is kept.** A replica within its owner's guarantee is never
|
||||
taken for another scope's model, whatever the classes: a guarantee a higher
|
||||
class elsewhere could take would not be one.
|
||||
- **Beyond it is borrowing.** Free room is used; when a scope within its own
|
||||
guarantee needs room, what others hold beyond theirs is taken back first,
|
||||
whatever their class — except a critical model's minimum.
|
||||
- **A limit is a ceiling.** A model whose owner would go past a limit anywhere
|
||||
up the tree gets no further replica, and says which limit; headroom and the
|
||||
mirror card stop there too.
|
||||
|
||||
`llmgw ctl quota set team:Acme/ml --gpus 8 --limit-gpus 16` (also `--cores`,
|
||||
`--memory` and their limits), `llmgw ctl quota list` with what each scope holds,
|
||||
`llmgw ctl quota clear`. Scopes are named by path or ID; a name that could mean
|
||||
two is refused. Changes take effect within half a minute. Metrics:
|
||||
`llmgw_capacity_holds`, `llmgw_capacity_guarantee` and `llmgw_capacity_limit`
|
||||
by scope and resource; alert: CapacityAtLimit.
|
||||
|
||||
### The mirror page in tabs
|
||||
|
||||
/admin/mirror was one long page: what the mirror holds, the Hugging Face
|
||||
|
||||
@@ -95,6 +95,17 @@ groups:
|
||||
the cluster is too small for what is asked of it at these classes.
|
||||
`llmgw ctl models` shows who it gave way to.
|
||||
|
||||
- alert: CapacityAtLimit
|
||||
expr: llmgw_capacity_holds >= on(scope, resource) llmgw_capacity_limit
|
||||
for: 15m
|
||||
labels: { severity: info }
|
||||
annotations:
|
||||
summary: "{{ $labels.scope }} holds all the {{ $labels.resource }} it may"
|
||||
description: >-
|
||||
Its models get no further replicas, under load or not, until
|
||||
something of theirs stops or the limit is raised: `llmgw ctl quota
|
||||
list`, `llmgw ctl quota set`.
|
||||
|
||||
- alert: EngineInstallFailed
|
||||
expr: llmgw_node_engine_install{state="failed"} == 1
|
||||
for: 15m
|
||||
|
||||
@@ -93,6 +93,7 @@ rather than repeated.
|
||||
| `profiles` | `[]Profile` | [Model](#model) |
|
||||
| `scaling` | `Scaling` | [Model](#model) |
|
||||
| `class` | `string` | [Model](#model) |
|
||||
| `owner` | `string` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
| `routing_profile` | `string` | [Model](#model) |
|
||||
| `slo` | `SLO` | [Model](#model) |
|
||||
@@ -746,6 +747,14 @@ when there is not enough for both: critical, standard or opportunistic.
|
||||
Empty is standard, or opportunistic for a model loaded on demand. See
|
||||
EffectiveClass.
|
||||
|
||||
### `owner`
|
||||
|
||||
`string`
|
||||
|
||||
Owner is the part of the tenancy tree whose capacity quota this model
|
||||
counts against, written org:ID, team:ID or project:ID. Empty is shared:
|
||||
held by the cluster, counted against nobody.
|
||||
|
||||
### `fallbacks`
|
||||
|
||||
`[]string`
|
||||
|
||||
@@ -146,6 +146,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newToolCheckCmd(),
|
||||
newWeightsCmd(client, &asJSON),
|
||||
newCapacityCmd(client, &asJSON),
|
||||
newQuotaCmd(client, &asJSON),
|
||||
newUsageCmd(client, &asJSON),
|
||||
newCertCmd(),
|
||||
)
|
||||
|
||||
@@ -30,7 +30,7 @@ func newModelCmd(client func() *ctlClient) *cobra.Command {
|
||||
Short: "Add or remove models while the cluster is running",
|
||||
}
|
||||
c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
|
||||
newResourcesCmd(client), newClassCmd(client),
|
||||
newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
|
||||
newModelServeCmd(client))
|
||||
return c
|
||||
}
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"strconv"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func newQuotaCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: "quota",
|
||||
Short: "What parts of the tenancy tree are guaranteed of the cluster, and may hold",
|
||||
Long: "A capacity quota belongs to an organization, team or project, and counts the\n" +
|
||||
"GPUs, CPU cores and memory held by replicas of the models it owns (see\n" +
|
||||
"`llmgw ctl model owner`), and of those its teams and projects own.\n\n" +
|
||||
"The guarantee is kept for it: when it holds less and a model of its needs\n" +
|
||||
"room, replicas other scopes hold beyond their own guarantees are stopped to\n" +
|
||||
"make it — whatever their class. What it holds within its guarantee is\n" +
|
||||
"never taken by another scope's model. Beyond it, it borrows: free room is\n" +
|
||||
"used, and given back when asked for.\n\n" +
|
||||
"The limit is the most it may hold, borrowing included; a model that would\n" +
|
||||
"take it past is not given another replica.\n\n" +
|
||||
"Scopes are written org:Acme, team:Acme/ml or project:Acme/ml/training, or\n" +
|
||||
"with IDs. Changes take effect within half a minute.",
|
||||
}
|
||||
c.AddCommand(newQuotaListCmd(client, asJSON), newQuotaSetCmd(client), newQuotaClearCmd(client))
|
||||
return c
|
||||
}
|
||||
|
||||
func newQuotaListCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "list",
|
||||
Short: "Every quota, with what its scope holds now",
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var qs []adminapi.QuotaView
|
||||
if err := client().get(cmd.Context(), "/api/v1/quotas", &qs); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return json.NewEncoder(cmd.OutOrStdout()).Encode(qs)
|
||||
}
|
||||
if len(qs) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "No capacity quotas: everything shares the cluster by class alone.")
|
||||
return nil
|
||||
}
|
||||
w := tab()
|
||||
fmt.Fprintln(w, "SCOPE\tNAME\tGPUS\tCORES\tMEMORY GB")
|
||||
for _, q := range qs {
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", q.Scope, q.Name,
|
||||
quotaCell(q.Holds.GPUs, q.Guarantee.GPUs, q.Limit.GPUs),
|
||||
quotaCell(q.Holds.CPUCores, q.Guarantee.CPUCores, q.Limit.CPUCores),
|
||||
quotaCell(q.Holds.MemoryGB, q.Guarantee.MemoryGB, q.Limit.MemoryGB))
|
||||
}
|
||||
if err := w.Flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "\nEach cell: holds / guaranteed (at most limit). - is not governed.")
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// quotaCell is "6 / 8 (≤ 16)": what is held, what is guaranteed, the limit.
|
||||
func quotaCell(holds int, guarantee, limit *int) string {
|
||||
if guarantee == nil && limit == nil {
|
||||
return "-"
|
||||
}
|
||||
out := strconv.Itoa(holds) + " / "
|
||||
if guarantee != nil {
|
||||
out += strconv.Itoa(*guarantee)
|
||||
} else {
|
||||
out += "-"
|
||||
}
|
||||
if limit != nil {
|
||||
out += fmt.Sprintf(" (≤ %d)", *limit)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func newQuotaSetCmd(client func() *ctlClient) *cobra.Command {
|
||||
var gpus, cores, mem, lgpus, lcores, lmem int
|
||||
c := &cobra.Command{
|
||||
Use: "set SCOPE [--gpus N] [--cores N] [--memory GB] [--limit-gpus N] ...",
|
||||
Short: "Set a scope's guarantee and limit, replacing what it had",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
f := cmd.Flags()
|
||||
pick := func(name string, v int) *int {
|
||||
if !f.Changed(name) {
|
||||
return nil
|
||||
}
|
||||
return &v
|
||||
}
|
||||
req := adminapi.QuotaRequest{Scope: args[0],
|
||||
Guarantee: store.Resources{GPUs: pick("gpus", gpus), CPUCores: pick("cores", cores), MemoryGB: pick("memory", mem)},
|
||||
Limit: store.Resources{GPUs: pick("limit-gpus", lgpus), CPUCores: pick("limit-cores", lcores), MemoryGB: pick("limit-memory", lmem)},
|
||||
}
|
||||
body, _ := json.Marshal(req)
|
||||
var res map[string]string
|
||||
if err := client().put(cmd.Context(), "/api/v1/quotas", body, &res); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: set; it takes effect within half a minute\n", res["scope"])
|
||||
return nil
|
||||
},
|
||||
}
|
||||
f := c.Flags()
|
||||
f.IntVar(&gpus, "gpus", 0, "GPUs guaranteed")
|
||||
f.IntVar(&cores, "cores", 0, "CPU cores guaranteed")
|
||||
f.IntVar(&mem, "memory", 0, "memory guaranteed, in GB")
|
||||
f.IntVar(&lgpus, "limit-gpus", 0, "the most GPUs it may hold")
|
||||
f.IntVar(&lcores, "limit-cores", 0, "the most CPU cores it may hold")
|
||||
f.IntVar(&lmem, "limit-memory", 0, "the most memory it may hold, in GB")
|
||||
return c
|
||||
}
|
||||
|
||||
func newQuotaClearCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "clear SCOPE",
|
||||
Short: "Remove a scope's quota",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var res map[string]string
|
||||
if err := client().delete(cmd.Context(), "/api/v1/quotas/"+url.PathEscape(args[0]), &res); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: quota removed\n", res["removed"])
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newOwnerCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "owner <model> SCOPE|shared",
|
||||
Short: "Set whose capacity quota a model counts against",
|
||||
Long: "SCOPE is org:Acme, team:Acme/ml or project:Acme/ml/training. A shared\n" +
|
||||
"model belongs to the cluster and counts against nobody's quota.",
|
||||
Args: cobra.ExactArgs(2),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
owner := args[1]
|
||||
if owner == "shared" {
|
||||
owner = ""
|
||||
}
|
||||
body, _ := json.Marshal(map[string]string{"owner": owner})
|
||||
var res map[string]string
|
||||
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/owner", body, &res); err != nil {
|
||||
return err
|
||||
}
|
||||
if res["owner"] == "" {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s is shared\n", res["model"])
|
||||
} else {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s counts against %s\n", res["model"], res["owner"])
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -226,6 +226,8 @@ func newServerCmd() *cobra.Command {
|
||||
// And what each group of the tenancy tree spent, which is
|
||||
// the other half of a quota being reviewable.
|
||||
api.SetUsage(st)
|
||||
// Capacity quotas, set and read over this API.
|
||||
api.SetQuotas(st)
|
||||
}
|
||||
api.Register(adminMux)
|
||||
|
||||
@@ -483,6 +485,9 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
|
||||
if st := h.Store(); st != nil {
|
||||
lead = control.NewLeadership(st, cfg.Control.LeaseTTL, log)
|
||||
ctl.SetLeadership(lead)
|
||||
// The tenancy tree's capacity quotas: what each part is
|
||||
// guaranteed and may hold.
|
||||
ctl.Reconciler().SetQuotas(control.QuotaPolicy(st, log))
|
||||
log.Info("competing to be the control plane that places work",
|
||||
"owner", lead.Owner())
|
||||
}
|
||||
|
||||
@@ -42,6 +42,7 @@ type API struct {
|
||||
models Models
|
||||
capacity Capacity
|
||||
usage Usage
|
||||
quotas Quotas
|
||||
ctl *control.Controller
|
||||
}
|
||||
|
||||
@@ -58,6 +59,7 @@ func (a *API) Register(mux *http.ServeMux) {
|
||||
mux.HandleFunc("DELETE /api/v1/models/{name}", a.removeModel)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
|
||||
a.registerQuotas(mux)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/class", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.models == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
@@ -304,6 +306,7 @@ type ModelView struct {
|
||||
// Class is its capacity class; Preempted the replicas it lost to a higher
|
||||
// class; HeldFor the model it is giving room to now, if any.
|
||||
Class string `json:"class"`
|
||||
Owner string `json:"owner,omitempty"`
|
||||
Preempted int `json:"preempted,omitempty"`
|
||||
HeldFor string `json:"held_for,omitempty"`
|
||||
}
|
||||
@@ -552,7 +555,7 @@ func (a *API) Models() []ModelView {
|
||||
Fallbacks: m.Fallbacks,
|
||||
Desired: h.Desired, Failures: h.Failures,
|
||||
Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
|
||||
Class: m.EffectiveClass(), Preempted: h.Preempted, HeldFor: h.HeldFor,
|
||||
Class: m.EffectiveClass(), Owner: m.Owner, Preempted: h.Preempted, HeldFor: h.HeldFor,
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
v.Profiles = append(v.Profiles, p.Name)
|
||||
@@ -634,6 +637,7 @@ type Models interface {
|
||||
Scale(ctx context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error)
|
||||
Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
|
||||
SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
|
||||
SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
|
||||
}
|
||||
|
||||
// ResourcesRequest sets what a model's profiles reserve of a node.
|
||||
|
||||
@@ -258,6 +258,10 @@ type fakeModels struct {
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeModels) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
|
||||
return catalog.Model{Name: name, Owner: owner}, f.err
|
||||
}
|
||||
|
||||
func (f *fakeModels) SetClass(_ context.Context, name, class, _ string) (catalog.Model, error) {
|
||||
return catalog.Model{Name: name, Class: class}, f.err
|
||||
}
|
||||
|
||||
@@ -0,0 +1,237 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Quotas is where capacity quotas and the tenancy tree they name are kept.
|
||||
// The store implements it.
|
||||
type Quotas interface {
|
||||
CapacityQuotas(ctx context.Context) ([]store.CapacityQuota, error)
|
||||
PutCapacityQuota(ctx context.Context, q store.CapacityQuota) error
|
||||
DeleteCapacityQuota(ctx context.Context, scope store.Scope, id string) error
|
||||
ScopeParents(ctx context.Context) (map[string][]string, error)
|
||||
AllOrgs(ctx context.Context) ([]store.Org, error)
|
||||
AllTeams(ctx context.Context) ([]store.TeamPath, error)
|
||||
AllProjects(ctx context.Context) ([]store.ProjectPath, error)
|
||||
}
|
||||
|
||||
// SetQuotas supplies the quota store, which mounts the quota endpoints'
|
||||
// behaviour; without it they answer 501.
|
||||
func (a *API) SetQuotas(q Quotas) { a.quotas = q }
|
||||
|
||||
// QuotaView is one scope's quota with what it holds now.
|
||||
type QuotaView struct {
|
||||
// Scope is team:ID; Name is how people know it, "Acme / ml".
|
||||
Scope string `json:"scope"`
|
||||
Name string `json:"name"`
|
||||
Guarantee store.Resources `json:"guarantee"`
|
||||
Limit store.Resources `json:"limit"`
|
||||
Holds quota.Amount `json:"holds"`
|
||||
UpdatedBy string `json:"updated_by,omitempty"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
}
|
||||
|
||||
// QuotaRequest sets a scope's quota. Scope may be written with an ID or a
|
||||
// name: org:Acme, team:Acme/ml, project:Acme/ml/training.
|
||||
type QuotaRequest struct {
|
||||
Scope string `json:"scope"`
|
||||
Guarantee store.Resources `json:"guarantee"`
|
||||
Limit store.Resources `json:"limit"`
|
||||
}
|
||||
|
||||
func (a *API) registerQuotas(mux *http.ServeMux) {
|
||||
mux.HandleFunc("GET /api/v1/quotas", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.quotas == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
views, err := a.QuotaViews(r.Context())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, views)
|
||||
})
|
||||
mux.HandleFunc("PUT /api/v1/quotas", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.quotas == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
var req QuotaRequest
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<14)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
scope, id, _, err := a.resolveScope(r.Context(), req.Scope)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
if !req.Guarantee.Any() && !req.Limit.Any() {
|
||||
writeError(w, http.StatusBadRequest, "a quota needs a guarantee or a limit; to remove one, DELETE it")
|
||||
return
|
||||
}
|
||||
q := store.CapacityQuota{Scope: scope, ScopeID: id, Guarantee: req.Guarantee, Limit: req.Limit,
|
||||
UpdatedBy: actorOf(r)}
|
||||
if err := a.quotas.PutCapacityQuota(r.Context(), q); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"scope": quota.Key(scope, id)})
|
||||
})
|
||||
mux.HandleFunc("DELETE /api/v1/quotas/{scope}", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.quotas == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
scope, id, _, err := a.resolveScope(r.Context(), r.PathValue("scope"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
if err := a.quotas.DeleteCapacityQuota(r.Context(), scope, id); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"removed": quota.Key(scope, id)})
|
||||
})
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/owner", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.models == nil || a.quotas == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
Owner string `json:"owner"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
owner := ""
|
||||
if req.Owner != "" {
|
||||
scope, id, _, err := a.resolveScope(r.Context(), req.Owner)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
owner = quota.Key(scope, id)
|
||||
}
|
||||
m, err := a.models.SetOwner(r.Context(), r.PathValue("name"), owner, actorOf(r))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"model": m.Name, "owner": m.Owner})
|
||||
})
|
||||
}
|
||||
|
||||
// QuotaViews lists every quota with what its scope holds now.
|
||||
func (a *API) QuotaViews(ctx context.Context) ([]QuotaView, error) {
|
||||
qs, err := a.quotas.CapacityQuotas(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
parents, err := a.quotas.ScopeParents(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
names, err := a.scopeNames(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
usage := quota.New(qs, parents).Usage(a.ctl.Reconciler().Holdings())
|
||||
out := make([]QuotaView, 0, len(qs))
|
||||
for _, q := range qs {
|
||||
key := quota.Key(q.Scope, q.ScopeID)
|
||||
out = append(out, QuotaView{Scope: key, Name: names[key], Guarantee: q.Guarantee, Limit: q.Limit,
|
||||
Holds: usage[key], UpdatedBy: q.UpdatedBy, UpdatedAt: q.UpdatedAt})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// scopeNames maps org:ID, team:ID and project:ID to their paths.
|
||||
func (a *API) scopeNames(ctx context.Context) (map[string]string, error) {
|
||||
out := map[string]string{}
|
||||
orgs, err := a.quotas.AllOrgs(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, o := range orgs {
|
||||
out["org:"+o.ID] = o.Name
|
||||
}
|
||||
teams, err := a.quotas.AllTeams(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, t := range teams {
|
||||
out["team:"+t.ID] = t.OrgName + " / " + t.Name
|
||||
}
|
||||
projects, err := a.quotas.AllProjects(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, p := range projects {
|
||||
out["project:"+p.ID] = p.Path
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// resolveScope reads org:X, team:X or project:X, where X is an ID or the
|
||||
// scope's path — Acme, Acme/ml, Acme/ml/training — and returns its ID and
|
||||
// path. A path that could mean two scopes is refused rather than guessed.
|
||||
func (a *API) resolveScope(ctx context.Context, s string) (store.Scope, string, string, error) {
|
||||
kind, what, ok := strings.Cut(strings.TrimSpace(s), ":")
|
||||
scope := store.Scope(kind)
|
||||
if !ok || what == "" || (scope != store.ScopeOrg && scope != store.ScopeTeam && scope != store.ScopeProject) {
|
||||
return "", "", "", fmt.Errorf("%q is not org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT", s)
|
||||
}
|
||||
names, err := a.scopeNames(ctx)
|
||||
if err != nil {
|
||||
return "", "", "", err
|
||||
}
|
||||
norm := func(p string) string {
|
||||
parts := strings.Split(p, "/")
|
||||
for i := range parts {
|
||||
parts[i] = strings.ToLower(strings.TrimSpace(parts[i]))
|
||||
}
|
||||
return strings.Join(parts, "/")
|
||||
}
|
||||
if name, ok := names[kind+":"+what]; ok {
|
||||
return scope, what, name, nil
|
||||
}
|
||||
match := func(same func(name string) bool) []string {
|
||||
var found []string
|
||||
for key, name := range names {
|
||||
if strings.HasPrefix(key, kind+":") && same(name) {
|
||||
found = append(found, key)
|
||||
}
|
||||
}
|
||||
return found
|
||||
}
|
||||
found := match(func(name string) bool { return norm(name) == norm(what) })
|
||||
if len(found) == 0 {
|
||||
// Its own name alone — ml rather than Acme/ml — when that is unique.
|
||||
found = match(func(name string) bool {
|
||||
parts := strings.Split(norm(name), "/")
|
||||
return parts[len(parts)-1] == norm(what)
|
||||
})
|
||||
}
|
||||
switch len(found) {
|
||||
case 0:
|
||||
return "", "", "", fmt.Errorf("there is no %s %q", kind, what)
|
||||
case 1:
|
||||
_, id, _ := strings.Cut(found[0], ":")
|
||||
return scope, id, names[found[0]], nil
|
||||
}
|
||||
return "", "", "", fmt.Errorf("%q could be any of %d %ss; give its ID", what, len(found), kind)
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// fakeQuotaStore is Acme with teams ml and web, and Scania with another ml.
|
||||
type fakeQuotaStore struct {
|
||||
quotas []store.CapacityQuota
|
||||
}
|
||||
|
||||
func (f *fakeQuotaStore) CapacityQuotas(context.Context) ([]store.CapacityQuota, error) {
|
||||
return f.quotas, nil
|
||||
}
|
||||
func (f *fakeQuotaStore) PutCapacityQuota(_ context.Context, q store.CapacityQuota) error {
|
||||
for i := range f.quotas {
|
||||
if f.quotas[i].Scope == q.Scope && f.quotas[i].ScopeID == q.ScopeID {
|
||||
f.quotas[i] = q
|
||||
return nil
|
||||
}
|
||||
}
|
||||
f.quotas = append(f.quotas, q)
|
||||
return nil
|
||||
}
|
||||
func (f *fakeQuotaStore) DeleteCapacityQuota(_ context.Context, s store.Scope, id string) error {
|
||||
for i, q := range f.quotas {
|
||||
if q.Scope == s && q.ScopeID == id {
|
||||
f.quotas = append(f.quotas[:i], f.quotas[i+1:]...)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (f *fakeQuotaStore) ScopeParents(context.Context) (map[string][]string, error) {
|
||||
return map[string][]string{"team:t-ml": {"org:o-acme"}, "team:t-web": {"org:o-acme"},
|
||||
"team:t-sml": {"org:o-scania"}}, nil
|
||||
}
|
||||
func (f *fakeQuotaStore) AllOrgs(context.Context) ([]store.Org, error) {
|
||||
return []store.Org{{ID: "o-acme", Name: "Acme"}, {ID: "o-scania", Name: "Scania"}}, nil
|
||||
}
|
||||
func (f *fakeQuotaStore) AllTeams(context.Context) ([]store.TeamPath, error) {
|
||||
return []store.TeamPath{{ID: "t-ml", Name: "ml", OrgID: "o-acme", OrgName: "Acme"},
|
||||
{ID: "t-web", Name: "web", OrgID: "o-acme", OrgName: "Acme"},
|
||||
{ID: "t-sml", Name: "ml", OrgID: "o-scania", OrgName: "Scania"}}, nil
|
||||
}
|
||||
func (f *fakeQuotaStore) AllProjects(context.Context) ([]store.ProjectPath, error) { return nil, nil }
|
||||
|
||||
func TestAScopeIsNamedByPathOrIDAndNeverGuessed(t *testing.T) {
|
||||
a, _ := setup(t, testModels())
|
||||
a.SetQuotas(&fakeQuotaStore{})
|
||||
for in, want := range map[string]string{
|
||||
"org:Acme": "o-acme", "org:o-acme": "o-acme", "team:Acme/ml": "t-ml",
|
||||
"team: acme / ML ": "t-ml", "team:Scania/ml": "t-sml", "team:web": "t-web",
|
||||
} {
|
||||
_, id, _, err := a.resolveScope(t.Context(), in)
|
||||
if err != nil || id != want {
|
||||
t.Errorf("%q: %q %v, expected %q", in, id, err, want)
|
||||
}
|
||||
}
|
||||
for _, bad := range []string{"team:ml", "team:Acme/nope", "group:Acme", "Acme", "org:"} {
|
||||
if _, _, _, err := a.resolveScope(t.Context(), bad); err == nil {
|
||||
t.Errorf("%q was resolved", bad)
|
||||
}
|
||||
}
|
||||
if _, _, _, err := a.resolveScope(t.Context(), "team:ml"); err == nil || !strings.Contains(err.Error(), "could be any of 2") {
|
||||
t.Errorf("an ambiguous name should say so: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAQuotaIsSetListedAndCleared(t *testing.T) {
|
||||
a, _ := setup(t, testModels())
|
||||
qs := &fakeQuotaStore{}
|
||||
a.SetQuotas(qs)
|
||||
mux := http.NewServeMux()
|
||||
a.Register(mux)
|
||||
call := func(method, path, body string) *httptest.ResponseRecorder {
|
||||
req := httptest.NewRequest(method, path, strings.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("X-Llmgw-Actor", "ops")
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, req)
|
||||
return rec
|
||||
}
|
||||
if rec := call("PUT", "/api/v1/quotas", `{"scope":"team:Acme/ml","guarantee":{"gpus":8},"limit":{"gpus":16}}`); rec.Code != 200 {
|
||||
t.Fatalf("set: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
if len(qs.quotas) != 1 || qs.quotas[0].ScopeID != "t-ml" || *qs.quotas[0].Guarantee.GPUs != 8 || qs.quotas[0].UpdatedBy != "ops" {
|
||||
t.Errorf("stored %+v", qs.quotas)
|
||||
}
|
||||
rec := call("GET", "/api/v1/quotas", "")
|
||||
var views []QuotaView
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &views); err != nil || len(views) != 1 || views[0].Name != "Acme / ml" {
|
||||
t.Errorf("list %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
if rec := call("PUT", "/api/v1/quotas", `{"scope":"team:Acme/ml"}`); rec.Code != 400 {
|
||||
t.Errorf("an empty quota was accepted: %d", rec.Code)
|
||||
}
|
||||
if rec := call("DELETE", "/api/v1/quotas/team:Acme%2Fml", ""); rec.Code != 200 || len(qs.quotas) != 0 {
|
||||
t.Errorf("clear: %d %s %+v", rec.Code, rec.Body, qs.quotas)
|
||||
}
|
||||
}
|
||||
@@ -156,7 +156,11 @@ type Model struct {
|
||||
// when there is not enough for both: critical, standard or opportunistic.
|
||||
// Empty is standard, or opportunistic for a model loaded on demand. See
|
||||
// EffectiveClass.
|
||||
Class string `json:"class,omitempty" koanf:"class"`
|
||||
Class string `json:"class,omitempty" koanf:"class"`
|
||||
// Owner is the part of the tenancy tree whose capacity quota this model
|
||||
// counts against, written org:ID, team:ID or project:ID. Empty is shared:
|
||||
// held by the cluster, counted against nobody.
|
||||
Owner string `json:"owner,omitempty" koanf:"owner"`
|
||||
Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
|
||||
// RoutingProfile selects how the router scores endpoints for this model.
|
||||
RoutingProfile string `json:"routing_profile,omitempty" koanf:"routing_profile"`
|
||||
@@ -304,6 +308,9 @@ func (m Model) validate() error {
|
||||
if len(m.Profiles) == 0 {
|
||||
return fmt.Errorf("model %q: at least one profile is required", m.Name)
|
||||
}
|
||||
if m.Owner != "" && !validOwner(m.Owner) {
|
||||
return fmt.Errorf("model %q: owner %q is not org:ID, team:ID or project:ID", m.Name, m.Owner)
|
||||
}
|
||||
if !validClass(m.Class) {
|
||||
return fmt.Errorf("model %q: class %q is not one of critical, standard, opportunistic", m.Name, m.Class)
|
||||
}
|
||||
@@ -548,3 +555,12 @@ func ClassRank(c string) int {
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
func validOwner(o string) bool {
|
||||
for _, p := range []string{"org:", "team:", "project:"} {
|
||||
if strings.HasPrefix(o, p) && len(o) > len(p) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -375,3 +375,6 @@ func EngineKeys(secret []byte) func(replicaID string) string {
|
||||
return "llmgw-" + hex.EncodeToString(m.Sum(nil))[:40]
|
||||
}
|
||||
}
|
||||
|
||||
// QuotaUse is every capacity quota beside what its scope holds.
|
||||
func (c *Controller) QuotaUse() []reconciler.QuotaUse { return c.rec.QuotaUse() }
|
||||
@@ -51,13 +51,16 @@ type HealthSource interface {
|
||||
type Collector struct {
|
||||
src Source
|
||||
|
||||
modelDesired *prometheus.Desc
|
||||
modelFailures *prometheus.Desc
|
||||
modelParked *prometheus.Desc
|
||||
modelUnplaceable *prometheus.Desc
|
||||
modelPreempted *prometheus.Desc
|
||||
engineInstall *prometheus.Desc
|
||||
leader *prometheus.Desc
|
||||
modelDesired *prometheus.Desc
|
||||
modelFailures *prometheus.Desc
|
||||
modelParked *prometheus.Desc
|
||||
modelUnplaceable *prometheus.Desc
|
||||
modelPreempted *prometheus.Desc
|
||||
capacityHolds *prometheus.Desc
|
||||
capacityGuarantee *prometheus.Desc
|
||||
capacityLimit *prometheus.Desc
|
||||
engineInstall *prometheus.Desc
|
||||
leader *prometheus.Desc
|
||||
|
||||
nodeUp *prometheus.Desc
|
||||
nodeGPUs *prometheus.Desc
|
||||
@@ -169,6 +172,15 @@ func New(src Source) *Collector {
|
||||
modelUnplaceable: prometheus.NewDesc("llmgw_model_unplaceable",
|
||||
"1 when a model is short of replicas and the last attempt found no node to place one on.",
|
||||
[]string{"model"}, nil),
|
||||
capacityHolds: prometheus.NewDesc("llmgw_capacity_holds",
|
||||
"What a part of the tenancy tree holds of a resource, its own models' replicas and those beneath it.",
|
||||
[]string{"scope", "resource"}, nil),
|
||||
capacityGuarantee: prometheus.NewDesc("llmgw_capacity_guarantee",
|
||||
"What a part of the tenancy tree is guaranteed of a resource.",
|
||||
[]string{"scope", "resource"}, nil),
|
||||
capacityLimit: prometheus.NewDesc("llmgw_capacity_limit",
|
||||
"The most of a resource a part of the tenancy tree may hold.",
|
||||
[]string{"scope", "resource"}, nil),
|
||||
modelPreempted: prometheus.NewDesc("llmgw_model_preempted_total",
|
||||
"Replicas of a model stopped to make room for a model of a higher capacity class.",
|
||||
[]string{"model", "class"}, nil),
|
||||
@@ -241,6 +253,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
|
||||
c.gpuNVLink, c.gpuECC,
|
||||
c.replicaState, c.replicaReady, c.replicaIdle,
|
||||
c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable, c.modelPreempted,
|
||||
c.capacityHolds, c.capacityGuarantee, c.capacityLimit,
|
||||
c.engineInstall, c.leader,
|
||||
} {
|
||||
ch <- d
|
||||
@@ -250,6 +263,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
|
||||
// Collect implements prometheus.Collector.
|
||||
func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
now := time.Now()
|
||||
c.collectQuotas(ch)
|
||||
|
||||
loads := map[string]*inventory.Load{}
|
||||
for _, n := range c.src.Nodes() {
|
||||
@@ -469,3 +483,38 @@ func b2f(b bool) float64 {
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// QuotaSource adds the tenancy tree's capacity quotas. Asked for by type
|
||||
// assertion, like HealthSource.
|
||||
type QuotaSource interface {
|
||||
QuotaUse() []reconciler.QuotaUse
|
||||
}
|
||||
|
||||
func (c *Collector) collectQuotas(ch chan<- prometheus.Metric) {
|
||||
qs, ok := c.src.(QuotaSource)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
for _, q := range qs.QuotaUse() {
|
||||
for _, r := range []struct {
|
||||
name string
|
||||
holds int
|
||||
gua, lim *int
|
||||
}{
|
||||
{"gpus", q.Holds.GPUs, q.Guarantee.GPUs, q.Limit.GPUs},
|
||||
{"cpu_cores", q.Holds.CPUCores, q.Guarantee.CPUCores, q.Limit.CPUCores},
|
||||
{"memory_gb", q.Holds.MemoryGB, q.Guarantee.MemoryGB, q.Limit.MemoryGB},
|
||||
} {
|
||||
if r.gua == nil && r.lim == nil {
|
||||
continue
|
||||
}
|
||||
ch <- prometheus.MustNewConstMetric(c.capacityHolds, prometheus.GaugeValue, float64(r.holds), q.Scope, r.name)
|
||||
if r.gua != nil {
|
||||
ch <- prometheus.MustNewConstMetric(c.capacityGuarantee, prometheus.GaugeValue, float64(*r.gua), q.Scope, r.name)
|
||||
}
|
||||
if r.lim != nil {
|
||||
ch <- prometheus.MustNewConstMetric(c.capacityLimit, prometheus.GaugeValue, float64(*r.lim), q.Scope, r.name)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -255,6 +255,19 @@ func (s *Service) SetClass(ctx context.Context, name, class, by string) (catalog
|
||||
return final, err
|
||||
}
|
||||
|
||||
// SetOwner sets the part of the tenancy tree a model's capacity counts
|
||||
// against, org:ID, team:ID or project:ID; empty makes it shared.
|
||||
func (s *Service) SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error) {
|
||||
final, err := s.update(ctx, name, by, "change its owner", func(m *catalog.Model) error {
|
||||
m.Owner = owner
|
||||
return nil
|
||||
})
|
||||
if err == nil {
|
||||
s.log.Info("model owner set", "model", name, "owner", owner, "by", by)
|
||||
}
|
||||
return final, err
|
||||
}
|
||||
|
||||
// update changes one model added at runtime and keeps the change.
|
||||
//
|
||||
// Built before anything is written, like Add: catalog.New is what applies the
|
||||
|
||||
@@ -0,0 +1,192 @@
|
||||
// Package quota decides what the tenancy tree is guaranteed of the cluster
|
||||
// and how much of it each part may hold.
|
||||
//
|
||||
// A scope — an organization, team or project — may be guaranteed GPUs, CPU
|
||||
// cores and memory, and limited to at most some amount of each. What a scope
|
||||
// holds is what the replicas of the models it owns, and of the models its
|
||||
// teams and projects own, have reserved. Holding more than its guarantee is
|
||||
// borrowing: allowed while the room is free, and taken back when another
|
||||
// scope within its own guarantee needs it.
|
||||
package quota
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Amount is how much of each resource something holds or needs.
|
||||
type Amount struct {
|
||||
GPUs, CPUCores, MemoryGB int
|
||||
}
|
||||
|
||||
// Plus adds two amounts.
|
||||
func (a Amount) Plus(b Amount) Amount {
|
||||
return Amount{a.GPUs + b.GPUs, a.CPUCores + b.CPUCores, a.MemoryGB + b.MemoryGB}
|
||||
}
|
||||
|
||||
// Over names the first resource in which a is above the governed amounts of
|
||||
// l, with both figures; empty when it is above none.
|
||||
func Over(a Amount, l store.Resources) string {
|
||||
switch {
|
||||
case l.GPUs != nil && a.GPUs > *l.GPUs:
|
||||
return fmt.Sprintf("%d GPUs of %d", a.GPUs, *l.GPUs)
|
||||
case l.CPUCores != nil && a.CPUCores > *l.CPUCores:
|
||||
return fmt.Sprintf("%d cores of %d", a.CPUCores, *l.CPUCores)
|
||||
case l.MemoryGB != nil && a.MemoryGB > *l.MemoryGB:
|
||||
return fmt.Sprintf("%d GB of memory of %d", a.MemoryGB, *l.MemoryGB)
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// Holding is one replica's hold on the cluster, for working out who holds
|
||||
// what.
|
||||
type Holding struct {
|
||||
Replica string
|
||||
Owner string
|
||||
Amount Amount
|
||||
Since time.Time
|
||||
}
|
||||
|
||||
// Policy is the quotas and the tree they sit in, read at one moment.
|
||||
type Policy struct {
|
||||
quotas map[string]store.CapacityQuota
|
||||
parents map[string][]string
|
||||
}
|
||||
|
||||
// New builds a policy from stored quotas and each scope's parents.
|
||||
func New(quotas []store.CapacityQuota, parents map[string][]string) *Policy {
|
||||
p := &Policy{quotas: map[string]store.CapacityQuota{}, parents: parents}
|
||||
for _, q := range quotas {
|
||||
p.quotas[Key(q.Scope, q.ScopeID)] = q
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// Key writes a scope the way owners are written: team:ID.
|
||||
func Key(s store.Scope, id string) string { return string(s) + ":" + id }
|
||||
|
||||
// Chain is an owner and every scope above it, nearest first.
|
||||
func (p *Policy) Chain(owner string) []string {
|
||||
if owner == "" {
|
||||
return nil
|
||||
}
|
||||
return append([]string{owner}, p.parents[owner]...)
|
||||
}
|
||||
|
||||
// Governing is the quotas along an owner's chain, nearest first.
|
||||
func (p *Policy) Governing(owner string) []store.CapacityQuota {
|
||||
if p == nil {
|
||||
return nil
|
||||
}
|
||||
var out []store.CapacityQuota
|
||||
for _, s := range p.Chain(owner) {
|
||||
if q, ok := p.quotas[s]; ok {
|
||||
out = append(out, q)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Quotas lists every quota.
|
||||
func (p *Policy) Quotas() []store.CapacityQuota {
|
||||
if p == nil {
|
||||
return nil
|
||||
}
|
||||
out := make([]store.CapacityQuota, 0, len(p.quotas))
|
||||
for _, q := range p.quotas {
|
||||
out = append(out, q)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
return Key(out[i].Scope, out[i].ScopeID) < Key(out[j].Scope, out[j].ScopeID)
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// Usage is what every scope holds, its own models' replicas and those of
|
||||
// everything beneath it.
|
||||
func (p *Policy) Usage(hs []Holding) map[string]Amount {
|
||||
out := map[string]Amount{}
|
||||
if p == nil {
|
||||
return out
|
||||
}
|
||||
for _, h := range hs {
|
||||
for _, s := range p.Chain(h.Owner) {
|
||||
out[s] = out[s].Plus(h.Amount)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Borrowed is the replicas held beyond a guarantee: of each governed scope's
|
||||
// replicas, oldest first, the ones that take it past what it is guaranteed.
|
||||
// A replica beyond any guarantee along its chain is borrowed.
|
||||
func (p *Policy) Borrowed(hs []Holding) map[string]bool {
|
||||
out := map[string]bool{}
|
||||
if p == nil || len(p.quotas) == 0 {
|
||||
return out
|
||||
}
|
||||
sorted := append([]Holding(nil), hs...)
|
||||
sort.Slice(sorted, func(i, j int) bool { return sorted[i].Since.Before(sorted[j].Since) })
|
||||
for key, q := range p.quotas {
|
||||
if !q.Guarantee.Any() {
|
||||
continue
|
||||
}
|
||||
var sum Amount
|
||||
for _, h := range sorted {
|
||||
if !contains(p.Chain(h.Owner), key) {
|
||||
continue
|
||||
}
|
||||
sum = sum.Plus(h.Amount)
|
||||
if Over(sum, q.Guarantee) != "" {
|
||||
out[h.Replica] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Guaranteed reports whether an owner, holding what it holds, is still within
|
||||
// every guarantee along its chain with need added — so it may take back what
|
||||
// others borrowed. An owner with no guarantee anywhere is never guaranteed.
|
||||
func (p *Policy) Guaranteed(owner string, usage map[string]Amount, need Amount) bool {
|
||||
gov := p.Governing(owner)
|
||||
any := false
|
||||
for _, q := range gov {
|
||||
if !q.Guarantee.Any() {
|
||||
continue
|
||||
}
|
||||
any = true
|
||||
if Over(usage[Key(q.Scope, q.ScopeID)].Plus(need), q.Guarantee) != "" {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return any
|
||||
}
|
||||
|
||||
// AtLimit says which limit along an owner's chain need would take it past,
|
||||
// empty when none.
|
||||
func (p *Policy) AtLimit(owner string, usage map[string]Amount, need Amount) string {
|
||||
for _, q := range p.Governing(owner) {
|
||||
if !q.Limit.Any() {
|
||||
continue
|
||||
}
|
||||
key := Key(q.Scope, q.ScopeID)
|
||||
if over := Over(usage[key].Plus(need), q.Limit); over != "" {
|
||||
return fmt.Sprintf("%s is at its capacity limit (%s)", key, strings.TrimSpace(over))
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func contains(list []string, v string) bool {
|
||||
for _, x := range list {
|
||||
if x == v {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package quota
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func n(v int) *int { return &v }
|
||||
|
||||
// acme has two teams; ml is guaranteed 8 GPUs, the org as a whole 12 and
|
||||
// no more than 16.
|
||||
func policy() *Policy {
|
||||
return New([]store.CapacityQuota{
|
||||
{Scope: store.ScopeTeam, ScopeID: "ml", Guarantee: store.Resources{GPUs: n(8)}},
|
||||
{Scope: store.ScopeOrg, ScopeID: "acme", Guarantee: store.Resources{GPUs: n(12)}, Limit: store.Resources{GPUs: n(16)}},
|
||||
}, map[string][]string{
|
||||
"team:ml": {"org:acme"}, "team:web": {"org:acme"},
|
||||
"project:train": {"team:ml", "org:acme"},
|
||||
})
|
||||
}
|
||||
|
||||
func at(min int) time.Time { return time.Date(2026, 1, 1, 0, min, 0, 0, time.UTC) }
|
||||
|
||||
func TestAScopeHoldsWhatEverythingBeneathItHolds(t *testing.T) {
|
||||
p := policy()
|
||||
u := p.Usage([]Holding{
|
||||
{Replica: "a", Owner: "project:train", Amount: Amount{GPUs: 4}},
|
||||
{Replica: "b", Owner: "team:web", Amount: Amount{GPUs: 2, CPUCores: 3}},
|
||||
{Replica: "c", Owner: "", Amount: Amount{GPUs: 8}}, // shared: nobody's
|
||||
})
|
||||
if u["team:ml"].GPUs != 4 || u["org:acme"].GPUs != 6 || u["org:acme"].CPUCores != 3 || u["project:train"].GPUs != 4 {
|
||||
t.Errorf("usage %+v", u)
|
||||
}
|
||||
if _, ok := u[""]; ok {
|
||||
t.Error("a shared model was counted against somebody")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWhatGoesPastAGuaranteeIsBorrowedNewestFirst(t *testing.T) {
|
||||
p := policy()
|
||||
b := p.Borrowed([]Holding{
|
||||
{Replica: "old", Owner: "team:ml", Amount: Amount{GPUs: 8}, Since: at(1)},
|
||||
{Replica: "new", Owner: "team:ml", Amount: Amount{GPUs: 4}, Since: at(2)},
|
||||
{Replica: "web", Owner: "team:web", Amount: Amount{GPUs: 4}, Since: at(3)},
|
||||
})
|
||||
// ml: 8 within its 8, then 4 past it. acme: 8 + 4 = 12 within, web's 4
|
||||
// takes the org to 16, past its 12.
|
||||
if b["old"] || !b["new"] || !b["web"] {
|
||||
t.Errorf("borrowed %v", b)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAGuaranteeIsOnlyForWhoHasOne(t *testing.T) {
|
||||
p := policy()
|
||||
u := map[string]Amount{"team:ml": {GPUs: 4}, "org:acme": {GPUs: 4}}
|
||||
if !p.Guaranteed("project:train", u, Amount{GPUs: 4}) {
|
||||
t.Error("8 of ml's 8 and 8 of acme's 12 is within both")
|
||||
}
|
||||
if p.Guaranteed("project:train", u, Amount{GPUs: 5}) {
|
||||
t.Error("9 GPUs is past ml's 8")
|
||||
}
|
||||
if p.Guaranteed("", u, Amount{GPUs: 1}) || New(nil, nil).Guaranteed("team:ml", u, Amount{}) {
|
||||
t.Error("an owner with no guarantee is not guaranteed anything")
|
||||
}
|
||||
}
|
||||
|
||||
func TestALimitStopsGrowthWhereverItIsSet(t *testing.T) {
|
||||
p := policy()
|
||||
u := map[string]Amount{"org:acme": {GPUs: 14}}
|
||||
if why := p.AtLimit("team:web", u, Amount{GPUs: 2}); why != "" {
|
||||
t.Errorf("16 of 16 is at the limit, not past it: %s", why)
|
||||
}
|
||||
why := p.AtLimit("team:web", u, Amount{GPUs: 4})
|
||||
if !strings.Contains(why, "org:acme is at its capacity limit (18 GPUs of 16)") {
|
||||
t.Errorf("the refusal should name the scope and the figures: %q", why)
|
||||
}
|
||||
if p.AtLimit("", u, Amount{GPUs: 99}) != "" {
|
||||
t.Error("a shared model was limited")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
package control
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// QuotaStore is where capacity quotas and the tenancy tree are kept.
|
||||
type QuotaStore interface {
|
||||
CapacityQuotas(ctx context.Context) ([]store.CapacityQuota, error)
|
||||
ScopeParents(ctx context.Context) (map[string][]string, error)
|
||||
}
|
||||
|
||||
// quotaRefresh is how stale the reconciler's view of quotas may get. A quota
|
||||
// changed by an operator takes effect within this; the reconciler asks many
|
||||
// times a pass and the store is not asked each time.
|
||||
const quotaRefresh = 30 * time.Second
|
||||
|
||||
// QuotaPolicy returns what the reconciler asks for its quotas: the last
|
||||
// policy read from the store, read again once it is older than quotaRefresh.
|
||||
// A store that fails keeps the last good policy, so a database hiccup does
|
||||
// not drop every guarantee and limit at once.
|
||||
func QuotaPolicy(st QuotaStore, log *slog.Logger) func() *quota.Policy {
|
||||
var mu sync.Mutex
|
||||
var cur *quota.Policy
|
||||
var at time.Time
|
||||
return func() *quota.Policy {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if cur != nil && time.Since(at) < quotaRefresh {
|
||||
return cur
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
qs, err := st.CapacityQuotas(ctx)
|
||||
if err == nil {
|
||||
var parents map[string][]string
|
||||
if parents, err = st.ScopeParents(ctx); err == nil {
|
||||
cur, at = quota.New(qs, parents), time.Now()
|
||||
return cur
|
||||
}
|
||||
}
|
||||
log.Warn("could not read capacity quotas; keeping the last ones", "err", err)
|
||||
at = time.Now() // not again on every call while the store is down
|
||||
return cur
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
)
|
||||
|
||||
// Headroom is how many more replicas of a model the cluster has room for now.
|
||||
@@ -52,6 +53,11 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
|
||||
|
||||
inv := r.inv.Clone()
|
||||
sched := placement.New(inv)
|
||||
pol := r.policy()
|
||||
var usage map[string]quota.Amount
|
||||
if pol != nil {
|
||||
usage = pol.Usage(r.Holdings())
|
||||
}
|
||||
for h.More < limit {
|
||||
res := sched.Place(m.PlacementProfiles(), avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
@@ -66,6 +72,17 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
|
||||
h.Blocked = slices.Compact(h.Blocked)
|
||||
return h, true
|
||||
}
|
||||
// A quota limit is as much an edge as a full node.
|
||||
if pol != nil && m.Owner != "" {
|
||||
need := placementNeed(m, res.Placement)
|
||||
if why := pol.AtLimit(m.Owner, usage, need); why != "" {
|
||||
h.Blocked = []string{why}
|
||||
return h, true
|
||||
}
|
||||
for _, s := range pol.Chain(m.Owner) {
|
||||
usage[s] = usage[s].Plus(need)
|
||||
}
|
||||
}
|
||||
mem, cores := 0, 0
|
||||
if prof, ok := m.ProfileByName(res.Placement.Profile.Name); ok {
|
||||
mem, cores = prof.MinMemoryGB, cpuCores(prof)
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
)
|
||||
|
||||
// hold keeps a model that gave up room from taking it back.
|
||||
@@ -162,15 +163,29 @@ func (r *Reconciler) liveCount(model string) int {
|
||||
}
|
||||
|
||||
// victims lists the replicas that may be stopped for m, in the order they
|
||||
// should be: idle on-demand ones first, then what lower classes grew into
|
||||
// under load, then the same within m's class, then lower classes' minimums;
|
||||
// within each, the longest unused first.
|
||||
// should be: idle on-demand ones first, then what other tenants borrowed past
|
||||
// their guarantees when m's owner is within its own, then what lower classes
|
||||
// grew into under load, then the same within m's class, then lower classes'
|
||||
// minimums; within each, the longest unused first.
|
||||
//
|
||||
// A replica within its owner's guarantee is taken only for a model of the
|
||||
// same owner: a guarantee that another tenant's higher class could take is
|
||||
// not one.
|
||||
func (r *Reconciler) victims(m catalog.Model) []victim {
|
||||
cat := r.cat.Load()
|
||||
now := r.now()
|
||||
myRank := catalog.ClassRank(m.EffectiveClass())
|
||||
belowMin := r.liveCount(m.Name) < m.Scaling.Min
|
||||
|
||||
pol := r.policy()
|
||||
var borrowed map[string]bool
|
||||
reclaim := false
|
||||
if pol != nil {
|
||||
hs := r.Holdings()
|
||||
borrowed = pol.Borrowed(hs)
|
||||
reclaim = pol.Guaranteed(m.Owner, pol.Usage(hs), profileNeed(m))
|
||||
}
|
||||
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
|
||||
@@ -201,11 +216,23 @@ func (r *Reconciler) victims(m catalog.Model) []victim {
|
||||
case rank < myRank && other.Scaling.OnDemand() && idleFor(rep, now) >= r.cfg.EvictionGrace:
|
||||
tier = 0
|
||||
case rank < myRank && aboveMin:
|
||||
tier = 1
|
||||
case rank == myRank && aboveMin && belowMin:
|
||||
tier = 2
|
||||
case rank < myRank && class != catalog.ClassCritical:
|
||||
case rank == myRank && aboveMin && belowMin:
|
||||
tier = 3
|
||||
case rank < myRank && class != catalog.ClassCritical:
|
||||
tier = 4
|
||||
}
|
||||
if pol != nil && other.Owner != m.Owner {
|
||||
switch {
|
||||
case borrowed[rep.ID] && reclaim && (class != catalog.ClassCritical || aboveMin):
|
||||
// Past its owner's guarantee, wanted by an owner within
|
||||
// its own: taken back whatever the classes.
|
||||
if tier != 0 {
|
||||
tier = 1
|
||||
}
|
||||
case !borrowed[rep.ID] && guaranteed(pol, other.Owner):
|
||||
continue // held within its owner's guarantee
|
||||
}
|
||||
}
|
||||
if tier < 0 {
|
||||
continue
|
||||
@@ -247,3 +274,13 @@ func nodesOf(rep *Replica) map[string]struct{} {
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// guaranteed reports whether an owner has a guarantee anywhere up its tree.
|
||||
func guaranteed(pol *quota.Policy, owner string) bool {
|
||||
for _, q := range pol.Governing(owner) {
|
||||
if q.Guarantee.Any() {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// SetQuotas supplies the tenancy tree's capacity quotas, read afresh each
|
||||
// time it is called. Without it nothing is guaranteed or limited, and
|
||||
// preemption goes by class alone.
|
||||
func (r *Reconciler) SetQuotas(f func() *quota.Policy) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.quotas = f
|
||||
}
|
||||
|
||||
func (r *Reconciler) policy() *quota.Policy {
|
||||
r.mu.Lock()
|
||||
f := r.quotas
|
||||
r.mu.Unlock()
|
||||
if f == nil {
|
||||
return nil
|
||||
}
|
||||
return f()
|
||||
}
|
||||
|
||||
// Holdings is what every replica holds, and for whom.
|
||||
func (r *Reconciler) Holdings() []quota.Holding {
|
||||
cat := r.cat.Load()
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
out := make([]quota.Holding, 0, len(r.replicas))
|
||||
for _, rep := range r.replicas {
|
||||
m, ok := cat.Resolve(rep.Model)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
prof, _ := m.ProfileByName(rep.Profile)
|
||||
var a quota.Amount
|
||||
for _, sh := range rep.Shards {
|
||||
a = a.Plus(quota.Amount{GPUs: len(sh.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
|
||||
}
|
||||
out = append(out, quota.Holding{Replica: rep.ID, Owner: m.Owner, Amount: a, Since: rep.CreatedAt})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// placementNeed is what a placement would hold.
|
||||
func placementNeed(m catalog.Model, pl *placement.Placement) quota.Amount {
|
||||
prof, _ := m.ProfileByName(pl.Profile.Name)
|
||||
var a quota.Amount
|
||||
for _, n := range pl.Nodes {
|
||||
a = a.Plus(quota.Amount{GPUs: len(n.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
|
||||
}
|
||||
return a
|
||||
}
|
||||
|
||||
// profileNeed is what a replica of a model's first profile would hold, for
|
||||
// deciding before there is a placement.
|
||||
func profileNeed(m catalog.Model) quota.Amount {
|
||||
if len(m.Profiles) == 0 {
|
||||
return quota.Amount{}
|
||||
}
|
||||
p := m.Profiles[0]
|
||||
nodes := max(p.Nodes, 1)
|
||||
gpus := p.GPUsPerNode
|
||||
if gpus == 0 && !p.CPUOnly() {
|
||||
gpus = max(p.TP, 1)
|
||||
}
|
||||
return quota.Amount{GPUs: gpus * nodes, CPUCores: cpuCores(p) * nodes, MemoryGB: p.MinMemoryGB * nodes}
|
||||
}
|
||||
|
||||
// overLimit says why a placement would take its model's owner past a limit.
|
||||
func (r *Reconciler) overLimit(m catalog.Model, pl *placement.Placement) string {
|
||||
pol := r.policy()
|
||||
if pol == nil || m.Owner == "" {
|
||||
return ""
|
||||
}
|
||||
return pol.AtLimit(m.Owner, pol.Usage(r.Holdings()), placementNeed(m, pl))
|
||||
}
|
||||
|
||||
// QuotaUse is one scope's quota beside what it holds.
|
||||
type QuotaUse struct {
|
||||
Scope string
|
||||
Guarantee store.Resources
|
||||
Limit store.Resources
|
||||
Holds quota.Amount
|
||||
}
|
||||
|
||||
// QuotaUse lists every quota with what its scope holds now.
|
||||
func (r *Reconciler) QuotaUse() []QuotaUse {
|
||||
pol := r.policy()
|
||||
if pol == nil {
|
||||
return nil
|
||||
}
|
||||
usage := pol.Usage(r.Holdings())
|
||||
var out []QuotaUse
|
||||
for _, q := range pol.Quotas() {
|
||||
key := quota.Key(q.Scope, q.ScopeID)
|
||||
out = append(out, QuotaUse{Scope: key, Guarantee: q.Guarantee, Limit: q.Limit, Holds: usage[key]})
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func owned(m catalog.Model, owner string) catalog.Model { m.Owner = owner; return m }
|
||||
|
||||
func gpus(n int) store.Resources { return store.Resources{GPUs: &n} }
|
||||
|
||||
// Two teams in one org; each 8-GPU node holds one tp8 replica.
|
||||
func withQuotas(h *harness, qs ...store.CapacityQuota) {
|
||||
pol := quota.New(qs, map[string][]string{"team:ml": {"org:acme"}, "team:web": {"org:acme"}})
|
||||
h.r.SetQuotas(func() *quota.Policy { return pol })
|
||||
}
|
||||
|
||||
func TestATenantWithinItsGuaranteeTakesBackWhatAnotherBorrowed(t *testing.T) {
|
||||
// web is guaranteed one node and borrowed the second; ml is guaranteed
|
||||
// one node and wants it. Same class, and web's replica is at its minimum
|
||||
// — by class alone nothing could be taken.
|
||||
web := owned(gpuModel("web-chat", "", 2), "team:web")
|
||||
ml := owned(gpuModel("ml-train", "", 1), "team:ml")
|
||||
h := newFleetHarness(t, []catalog.Model{web, withMin(ml, 0)}, gpuNode("1", 8), gpuNode("2", 8))
|
||||
withQuotas(h,
|
||||
store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "web", Guarantee: gpus(8)},
|
||||
store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "ml", Guarantee: gpus(8)})
|
||||
h.settle(t, 4)
|
||||
if replicasOf(h, "web-chat") != 2 {
|
||||
t.Fatalf("setup: web did not borrow the second node: %+v", h.r.Replicas())
|
||||
}
|
||||
h.advance(h.r.cfg.MinLifetime + time.Minute)
|
||||
cat, _ := catalog.New([]catalog.Model{web, ml})
|
||||
h.r.cat.Store(cat)
|
||||
|
||||
h.settle(t, 5)
|
||||
if replicasOf(h, "ml-train") != 1 || replicasOf(h, "web-chat") != 1 {
|
||||
t.Errorf("the borrowed node was not taken back: web %d, ml %d",
|
||||
replicasOf(h, "web-chat"), replicasOf(h, "ml-train"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReplicaWithinItsGuaranteeIsNotTakenByAHigherClassElsewhere(t *testing.T) {
|
||||
web := owned(gpuModel("web-chat", "", 1), "team:web")
|
||||
vital := owned(gpuModel("vital", catalog.ClassCritical, 1), "team:ml")
|
||||
h := fillThenWant(t, vital, web, gpuNode("1", 8))
|
||||
withQuotas(h, store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "web", Guarantee: gpus(8)})
|
||||
h.settle(t, 3)
|
||||
if replicasOf(h, "web-chat") != 1 || replicasOf(h, "vital") != 0 {
|
||||
t.Errorf("a guaranteed replica was preempted by another tenant's class: %+v", h.r.Replicas())
|
||||
}
|
||||
// Without the guarantee, critical takes it: the guarantee was the reason.
|
||||
h2 := fillThenWant(t, vital, web, gpuNode("1", 8))
|
||||
h2.settle(t, 3)
|
||||
if replicasOf(h2, "vital") != 1 {
|
||||
t.Error("setup: without a guarantee the critical model should have taken the node")
|
||||
}
|
||||
}
|
||||
|
||||
func TestALimitStopsAModelBeforeItStarts(t *testing.T) {
|
||||
web := owned(gpuModel("web-chat", "", 2), "team:web")
|
||||
h := newFleetHarness(t, []catalog.Model{web}, gpuNode("1", 8), gpuNode("2", 8))
|
||||
withQuotas(h, store.CapacityQuota{Scope: store.ScopeOrg, ScopeID: "acme", Limit: gpus(8)})
|
||||
h.settle(t, 4)
|
||||
if replicasOf(h, "web-chat") != 1 {
|
||||
t.Errorf("a second replica went past the org's 8-GPU limit: %+v", h.r.Replicas())
|
||||
}
|
||||
hs := health(h, "web-chat")
|
||||
if !hs.Unplaceable || !strings.Contains(hs.Placement, "org:acme is at its capacity limit (16 GPUs of 8)") {
|
||||
t.Errorf("the refusal should name the limit: %+v", hs)
|
||||
}
|
||||
room, _ := h.r.Headroom("web-chat", 5)
|
||||
if room.More != 0 || len(room.Blocked) != 1 || !strings.Contains(room.Blocked[0], "capacity limit") {
|
||||
t.Errorf("headroom should stop at the limit, not the empty node: %+v", room)
|
||||
}
|
||||
}
|
||||
@@ -15,6 +15,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
)
|
||||
|
||||
// ShardCommand is everything an agent needs to launch one shard.
|
||||
@@ -270,6 +271,8 @@ type Reconciler struct {
|
||||
preempting map[string][]string
|
||||
held map[string]hold
|
||||
preempted map[string]int
|
||||
// quotas is the tenancy tree's capacity quotas as last read; nil is none.
|
||||
quotas func() *quota.Policy
|
||||
|
||||
wake chan struct{}
|
||||
stop chan struct{}
|
||||
@@ -811,6 +814,16 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
|
||||
r.notePlacementFailure(m.Name, res.Explain())
|
||||
return
|
||||
}
|
||||
if why := r.overLimit(m, res.Placement); why != "" {
|
||||
// Recorded as the placement's answer, so the model reads as not
|
||||
// placeable, and says why, wherever that is shown.
|
||||
r.mu.Lock()
|
||||
r.lastExplain[m.Name] = placement.Result{Rejections: []placement.Rejection{
|
||||
{Profile: res.Placement.Profile.Name, Reason: why}}}
|
||||
r.mu.Unlock()
|
||||
r.notePlacementFailure(m.Name, why)
|
||||
return
|
||||
}
|
||||
if err := r.startReplica(ctx, m, res.Placement); err != nil {
|
||||
r.log.Error("starting replica failed", "model", m.Name, "err", err)
|
||||
}
|
||||
|
||||
@@ -29,6 +29,9 @@ var copyOrder = []string{
|
||||
"key_claims",
|
||||
"sessions",
|
||||
"limits",
|
||||
// Beside limits, and like them keyed by scope with no foreign key: a
|
||||
// quota on a scope that is later removed is inert, not an error.
|
||||
"capacity_quotas",
|
||||
"usage",
|
||||
"models",
|
||||
// No foreign key to models, deliberately: a measurement outlives the
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
-- Capacity a part of the tenancy tree is guaranteed, and the most it may hold.
|
||||
--
|
||||
-- Beside limits rather than in it: limits are counted on the request path, per
|
||||
-- key, in requests and tokens; these are counted by the reconciler, per model,
|
||||
-- in the GPUs, cores and memory its replicas hold. A NULL is a resource this
|
||||
-- row does not govern: guaranteed nothing, or limited by nothing.
|
||||
CREATE TABLE capacity_quotas (
|
||||
scope_type TEXT NOT NULL,
|
||||
scope_id TEXT NOT NULL,
|
||||
gpus INTEGER,
|
||||
cpu_cores INTEGER,
|
||||
memory_gb INTEGER,
|
||||
limit_gpus INTEGER,
|
||||
limit_cpu_cores INTEGER,
|
||||
limit_memory_gb INTEGER,
|
||||
updated_at BIGINT NOT NULL,
|
||||
updated_by TEXT NOT NULL,
|
||||
PRIMARY KEY (scope_type, scope_id)
|
||||
);
|
||||
@@ -0,0 +1,19 @@
|
||||
-- Capacity a part of the tenancy tree is guaranteed, and the most it may hold.
|
||||
--
|
||||
-- Beside limits rather than in it: limits are counted on the request path, per
|
||||
-- key, in requests and tokens; these are counted by the reconciler, per model,
|
||||
-- in the GPUs, cores and memory its replicas hold. A NULL is a resource this
|
||||
-- row does not govern: guaranteed nothing, or limited by nothing.
|
||||
CREATE TABLE capacity_quotas (
|
||||
scope_type TEXT NOT NULL,
|
||||
scope_id TEXT NOT NULL,
|
||||
gpus INTEGER,
|
||||
cpu_cores INTEGER,
|
||||
memory_gb INTEGER,
|
||||
limit_gpus INTEGER,
|
||||
limit_cpu_cores INTEGER,
|
||||
limit_memory_gb INTEGER,
|
||||
updated_at BIGINT NOT NULL,
|
||||
updated_by TEXT NOT NULL,
|
||||
PRIMARY KEY (scope_type, scope_id)
|
||||
);
|
||||
@@ -0,0 +1,127 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Resources are amounts of the three things a replica holds of a cluster.
|
||||
// Nil is not governed: nothing guaranteed, or nothing limited.
|
||||
type Resources struct {
|
||||
GPUs *int `json:"gpus,omitempty"`
|
||||
CPUCores *int `json:"cpu_cores,omitempty"`
|
||||
MemoryGB *int `json:"memory_gb,omitempty"`
|
||||
}
|
||||
|
||||
// Any reports whether any resource is governed.
|
||||
func (r Resources) Any() bool { return r.GPUs != nil || r.CPUCores != nil || r.MemoryGB != nil }
|
||||
|
||||
// CapacityQuota is what one part of the tenancy tree is guaranteed of the
|
||||
// cluster, and the most it may hold.
|
||||
type CapacityQuota struct {
|
||||
Scope Scope `json:"scope"`
|
||||
ScopeID string `json:"scope_id"`
|
||||
// Guarantee is held for the scope: while it holds less, what others
|
||||
// borrowed beyond their own guarantees is taken back for it.
|
||||
Guarantee Resources `json:"guarantee"`
|
||||
// Limit is the most it may hold, borrowing included.
|
||||
Limit Resources `json:"limit"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
UpdatedBy string `json:"updated_by"`
|
||||
}
|
||||
|
||||
// PutCapacityQuota sets a scope's quota, replacing what was there.
|
||||
func (s *DB) PutCapacityQuota(ctx context.Context, q CapacityQuota) error {
|
||||
if !validMembershipScope(q.Scope) || q.ScopeID == "" {
|
||||
return fmt.Errorf("a capacity quota belongs to an organization, team or project")
|
||||
}
|
||||
for _, v := range []*int{q.Guarantee.GPUs, q.Guarantee.CPUCores, q.Guarantee.MemoryGB,
|
||||
q.Limit.GPUs, q.Limit.CPUCores, q.Limit.MemoryGB} {
|
||||
if v != nil && *v < 0 {
|
||||
return fmt.Errorf("a quota cannot be negative")
|
||||
}
|
||||
}
|
||||
if q.UpdatedAt.IsZero() {
|
||||
q.UpdatedAt = time.Now()
|
||||
}
|
||||
_, err := s.db.ExecContext(ctx,
|
||||
`INSERT INTO capacity_quotas (scope_type, scope_id, gpus, cpu_cores, memory_gb,
|
||||
limit_gpus, limit_cpu_cores, limit_memory_gb, updated_at, updated_by)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT (scope_type, scope_id) DO UPDATE SET
|
||||
gpus = excluded.gpus, cpu_cores = excluded.cpu_cores, memory_gb = excluded.memory_gb,
|
||||
limit_gpus = excluded.limit_gpus, limit_cpu_cores = excluded.limit_cpu_cores,
|
||||
limit_memory_gb = excluded.limit_memory_gb,
|
||||
updated_at = excluded.updated_at, updated_by = excluded.updated_by`,
|
||||
string(q.Scope), q.ScopeID,
|
||||
nullInt(q.Guarantee.GPUs), nullInt(q.Guarantee.CPUCores), nullInt(q.Guarantee.MemoryGB),
|
||||
nullInt(q.Limit.GPUs), nullInt(q.Limit.CPUCores), nullInt(q.Limit.MemoryGB),
|
||||
q.UpdatedAt.UnixMilli(), q.UpdatedBy)
|
||||
return err
|
||||
}
|
||||
|
||||
// CapacityQuotas lists every quota.
|
||||
func (s *DB) CapacityQuotas(ctx context.Context) ([]CapacityQuota, error) {
|
||||
return queryAll(ctx, s.db,
|
||||
`SELECT scope_type, scope_id, gpus, cpu_cores, memory_gb,
|
||||
limit_gpus, limit_cpu_cores, limit_memory_gb, updated_at, updated_by
|
||||
FROM capacity_quotas ORDER BY scope_type, scope_id`, nil,
|
||||
func(rows *sql.Rows) (CapacityQuota, error) {
|
||||
var q CapacityQuota
|
||||
var scope string
|
||||
var g [6]sql.NullInt64
|
||||
var at int64
|
||||
err := rows.Scan(&scope, &q.ScopeID, &g[0], &g[1], &g[2], &g[3], &g[4], &g[5], &at, &q.UpdatedBy)
|
||||
q.Scope = Scope(scope)
|
||||
q.Guarantee = Resources{GPUs: intOf(g[0]), CPUCores: intOf(g[1]), MemoryGB: intOf(g[2])}
|
||||
q.Limit = Resources{GPUs: intOf(g[3]), CPUCores: intOf(g[4]), MemoryGB: intOf(g[5])}
|
||||
q.UpdatedAt = time.UnixMilli(at)
|
||||
return q, err
|
||||
})
|
||||
}
|
||||
|
||||
// DeleteCapacityQuota removes a scope's quota.
|
||||
func (s *DB) DeleteCapacityQuota(ctx context.Context, scope Scope, id string) error {
|
||||
_, err := s.db.ExecContext(ctx,
|
||||
`DELETE FROM capacity_quotas WHERE scope_type = ? AND scope_id = ?`, string(scope), id)
|
||||
return err
|
||||
}
|
||||
|
||||
// ScopeParents maps every team and project to the scopes above it, nearest
|
||||
// first: a project to its team and organization, a team to its organization.
|
||||
// Keys and values are written scope:id.
|
||||
func (s *DB) ScopeParents(ctx context.Context) (map[string][]string, error) {
|
||||
out := map[string][]string{}
|
||||
teams, err := s.AllTeams(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
orgOf := map[string]string{}
|
||||
for _, t := range teams {
|
||||
orgOf[t.ID] = t.OrgID
|
||||
out["team:"+t.ID] = []string{"org:" + t.OrgID}
|
||||
}
|
||||
type pt struct{ project, team string }
|
||||
rows, err := queryAll(ctx, s.db, `SELECT id, team_id FROM projects`, nil,
|
||||
func(r *sql.Rows) (pt, error) {
|
||||
var p pt
|
||||
return p, r.Scan(&p.project, &p.team)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, p := range rows {
|
||||
out["project:"+p.project] = []string{"team:" + p.team, "org:" + orgOf[p.team]}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func intOf(n sql.NullInt64) *int {
|
||||
if !n.Valid {
|
||||
return nil
|
||||
}
|
||||
v := int(n.Int64)
|
||||
return &v
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package store
|
||||
|
||||
import "testing"
|
||||
|
||||
func ip(v int) *int { return &v }
|
||||
|
||||
func TestAQuotaKeepsWhatItGovernsAndNothingElse(t *testing.T) {
|
||||
s := tenancyStore(t)
|
||||
org, _ := s.CreateOrg(t.Context(), "Acme")
|
||||
q := CapacityQuota{Scope: ScopeOrg, ScopeID: org, UpdatedBy: "ops",
|
||||
Guarantee: Resources{GPUs: ip(8)}, Limit: Resources{GPUs: ip(16), CPUCores: ip(0)}}
|
||||
if err := s.PutCapacityQuota(t.Context(), q); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
all, err := s.CapacityQuotas(t.Context())
|
||||
if err != nil || len(all) != 1 {
|
||||
t.Fatalf("%v %+v", err, all)
|
||||
}
|
||||
got := all[0]
|
||||
if *got.Guarantee.GPUs != 8 || got.Guarantee.CPUCores != nil || got.Guarantee.MemoryGB != nil {
|
||||
t.Errorf("guarantee %+v: only GPUs were set", got.Guarantee)
|
||||
}
|
||||
// Zero is a limit of nothing, not an absent one.
|
||||
if *got.Limit.GPUs != 16 || got.Limit.CPUCores == nil || *got.Limit.CPUCores != 0 {
|
||||
t.Errorf("limit %+v", got.Limit)
|
||||
}
|
||||
q.Guarantee = Resources{MemoryGB: ip(64)}
|
||||
if err := s.PutCapacityQuota(t.Context(), q); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
all, _ = s.CapacityQuotas(t.Context())
|
||||
if len(all) != 1 || all[0].Guarantee.GPUs != nil || *all[0].Guarantee.MemoryGB != 64 {
|
||||
t.Errorf("a second put should replace the first: %+v", all)
|
||||
}
|
||||
if err := s.DeleteCapacityQuota(t.Context(), ScopeOrg, org); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if all, _ = s.CapacityQuotas(t.Context()); len(all) != 0 {
|
||||
t.Error("the quota survived its deletion")
|
||||
}
|
||||
if err := s.PutCapacityQuota(t.Context(), CapacityQuota{Scope: ScopeOrg, ScopeID: org,
|
||||
Guarantee: Resources{GPUs: ip(-1)}}); err == nil {
|
||||
t.Error("a negative quota was stored")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEveryScopeKnowsWhatIsAboveIt(t *testing.T) {
|
||||
s := tenancyStore(t)
|
||||
org, _ := s.CreateOrg(t.Context(), "Acme")
|
||||
team, _ := s.CreateTeam(t.Context(), org, "ml")
|
||||
proj, _ := s.CreateProject(t.Context(), team, "training")
|
||||
p, err := s.ScopeParents(t.Context())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []string{"team:" + team, "org:" + org}
|
||||
if got := p["project:"+proj]; len(got) != 2 || got[0] != want[0] || got[1] != want[1] {
|
||||
t.Errorf("project's parents %v, expected %v", got, want)
|
||||
}
|
||||
if got := p["team:"+team]; len(got) != 1 || got[0] != "org:"+org {
|
||||
t.Errorf("team's parents %v", got)
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user