Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e7a8105aa4 | ||
|
|
d0e9813ce9 | ||
|
|
0e73f37128 | ||
|
|
78e41460f9 | ||
|
|
9dc1b62fdc | ||
|
|
ac24e204a6 |
No files matched your search
@@ -49,6 +49,89 @@ holds of a node is reserved, not assumed.
|
||||
smallest node the model fits on — beside the memory they already estimate.
|
||||
Running replicas keep their old figures until restarted.
|
||||
|
||||
The commands are `llmgw ctl model resources` and `llmgw ctl model class`; in
|
||||
rc.7 the first was briefly `llmgw ctl resources`.
|
||||
|
||||
### Capacity classes, and preemption
|
||||
|
||||
A model has a class — `critical`, `standard` or `opportunistic` — saying what
|
||||
its capacity is worth against another's when there is not enough for both.
|
||||
Unset, an on-demand model is opportunistic and every other model standard,
|
||||
which is how the catalog behaved before. `llmgw ctl model class MODEL CLASS`
|
||||
sets it.
|
||||
|
||||
When a model cannot be placed, replicas are stopped to make room for it, by
|
||||
these rules:
|
||||
|
||||
- A lower class gives way to a higher one.
|
||||
- Within a class, only replicas above a model's minimum — room it grew into
|
||||
under load — and only for a model still below its own minimum.
|
||||
- A critical model's minimum is never taken, nor a replica that has only just
|
||||
loaded, nor one spanning several nodes.
|
||||
- Idle on-demand replicas go first, then what lower classes grew into, then
|
||||
the same within the class, then lower classes' minimums; the longest unused
|
||||
first within each, and of equally idle ones the smallest.
|
||||
- The set is tried on a copy of the inventory first, at most three replicas,
|
||||
and nothing is stopped unless it makes the room. Of a set that does, each
|
||||
replica whose stopping turns out not to be needed is left running, so an
|
||||
idle replica tried first on the wrong node is not stopped with the one that
|
||||
made the room.
|
||||
|
||||
A replica stopped this way is drained and given 30 seconds to finish what it
|
||||
is serving. Its model is held back for up to five minutes, or until the model
|
||||
it gave way to has its replicas, so it does not take the room straight back.
|
||||
`ctl models` shows each model's class and "gave room to X" while held; the
|
||||
cluster page shows the class and how many replicas were lost; the metric is
|
||||
`llmgw_model_preempted_total{model,class}`, and every preemption is logged.
|
||||
|
||||
### Capacity quotas for the tenancy tree
|
||||
|
||||
An organization, team or project can be guaranteed GPUs, CPU cores and
|
||||
memory, and limited to at most some amount of each. What it holds is what the
|
||||
replicas of the models it owns hold — and those of the teams and projects
|
||||
beneath it. `llmgw ctl model owner MODEL team:Acme/ml` says whose a model is;
|
||||
one with no owner is shared and counts against nobody.
|
||||
|
||||
- **A guarantee is kept.** A replica within its owner's guarantee is never
|
||||
taken for another scope's model, whatever the classes: a guarantee a higher
|
||||
class elsewhere could take would not be one.
|
||||
- **Beyond it is borrowing.** Free room is used; when a scope within its own
|
||||
guarantee needs room, what others hold beyond theirs is taken back first,
|
||||
whatever their class — except a critical model's minimum.
|
||||
- **A limit is a ceiling.** A model whose owner would go past a limit anywhere
|
||||
up the tree gets no further replica, and says which limit; headroom and the
|
||||
mirror card stop there too.
|
||||
|
||||
`llmgw ctl quota set team:Acme/ml --gpus 8 --limit-gpus 16` (also `--cores`,
|
||||
`--memory` and their limits), `llmgw ctl quota list` with what each scope holds,
|
||||
`llmgw ctl quota clear`. Scopes are named by path or ID; a name that could mean
|
||||
two is refused. Changes take effect within half a minute. Metrics:
|
||||
`llmgw_capacity_holds`, `llmgw_capacity_guarantee` and `llmgw_capacity_limit`
|
||||
by scope and resource; alert: CapacityAtLimit.
|
||||
|
||||
### Sized by what a replica was measured to serve
|
||||
|
||||
What `llmgw ctl bench MODEL --save` measured — the most requests at once one
|
||||
replica served while meeting its SLO, and its throughput there — now drives
|
||||
decisions rather than only being printed:
|
||||
|
||||
- **Scaling up** when requests per replica pass what was measured, not only
|
||||
when the engines' queues fill: an engine takes requests into its slots long
|
||||
after latency has gone.
|
||||
- **Scaling down** only when the replicas left would stay under 70% of what
|
||||
they were measured to hold, so a replica taken back does not put the rest
|
||||
straight over.
|
||||
- **Choosing a profile.** Among profiles of equal priority, the one measured
|
||||
to serve the most per GPU — or per core — is tried first. Priority set by an
|
||||
operator still wins.
|
||||
- **Sizing.** `llmgw ctl model target MODEL --tokens-per-sec N` says how many
|
||||
replicas that takes, how many run, and whether the rest fit, with the reason
|
||||
when they do not; `--apply` sets it if they do. Headroom reports what one
|
||||
more replica would serve.
|
||||
|
||||
A model never measured behaves exactly as before. Measurements are read from
|
||||
the store every half minute.
|
||||
|
||||
### The mirror page in tabs
|
||||
|
||||
/admin/mirror was one long page: what the mirror holds, the Hugging Face
|
||||
|
||||
@@ -84,6 +84,28 @@ groups:
|
||||
<model>` says what each node lacks. For an on-demand model, requests
|
||||
for it are told this rather than asked to retry.
|
||||
|
||||
- alert: ModelPreempted
|
||||
expr: increase(llmgw_model_preempted_total[15m]) > 0
|
||||
labels: { severity: info }
|
||||
annotations:
|
||||
summary: "{{ $labels.model }} ({{ $labels.class }}) gave up replicas to a higher class"
|
||||
description: >-
|
||||
Capacity ran short and a model of a higher class took this one's
|
||||
room. Expected now and then; if it keeps firing for the same model,
|
||||
the cluster is too small for what is asked of it at these classes.
|
||||
`llmgw ctl models` shows who it gave way to.
|
||||
|
||||
- alert: CapacityAtLimit
|
||||
expr: llmgw_capacity_holds >= on(scope, resource) llmgw_capacity_limit
|
||||
for: 15m
|
||||
labels: { severity: info }
|
||||
annotations:
|
||||
summary: "{{ $labels.scope }} holds all the {{ $labels.resource }} it may"
|
||||
description: >-
|
||||
Its models get no further replicas, under load or not, until
|
||||
something of theirs stops or the limit is raised: `llmgw ctl quota
|
||||
list`, `llmgw ctl quota set`.
|
||||
|
||||
- alert: EngineInstallFailed
|
||||
expr: llmgw_node_engine_install{state="failed"} == 1
|
||||
for: 15m
|
||||
|
||||
@@ -92,6 +92,8 @@ rather than repeated.
|
||||
| `source` | `string` | [Model](#model) |
|
||||
| `profiles` | `[]Profile` | [Model](#model) |
|
||||
| `scaling` | `Scaling` | [Model](#model) |
|
||||
| `class` | `string` | [Model](#model) |
|
||||
| `owner` | `string` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
| `routing_profile` | `string` | [Model](#model) |
|
||||
| `slo` | `SLO` | [Model](#model) |
|
||||
@@ -736,6 +738,23 @@ Source is the weights path or a HuggingFace id.
|
||||
|
||||
Takes a [Scaling](#scaling) block.
|
||||
|
||||
### `class`
|
||||
|
||||
`string`
|
||||
|
||||
Class is how much this model's capacity is worth against another's,
|
||||
when there is not enough for both: critical, standard or opportunistic.
|
||||
Empty is standard, or opportunistic for a model loaded on demand. See
|
||||
EffectiveClass.
|
||||
|
||||
### `owner`
|
||||
|
||||
`string`
|
||||
|
||||
Owner is the part of the tenancy tree whose capacity quota this model
|
||||
counts against, written org:ID, team:ID or project:ID. Empty is shared:
|
||||
held by the cluster, counted against nobody.
|
||||
|
||||
### `fallbacks`
|
||||
|
||||
`[]string`
|
||||
|
||||
+7
-5
@@ -137,9 +137,8 @@ func newCtlCmd() *cobra.Command {
|
||||
// apply to it; it lives here because this is where operators look.
|
||||
newBenchCmd(client, &asJSON),
|
||||
newStatusCmd(client, &asJSON),
|
||||
newModelCmd(client),
|
||||
newModelCmd(client, &asJSON),
|
||||
newScaleCmd(client),
|
||||
newResourcesCmd(client),
|
||||
newRestartCmd(client),
|
||||
newRetryCmd(client),
|
||||
newUpgradeCmd(client),
|
||||
@@ -147,6 +146,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newToolCheckCmd(),
|
||||
newWeightsCmd(client, &asJSON),
|
||||
newCapacityCmd(client, &asJSON),
|
||||
newQuotaCmd(client, &asJSON),
|
||||
newUsageCmd(client, &asJSON),
|
||||
newCertCmd(),
|
||||
)
|
||||
@@ -254,6 +254,8 @@ func modelStatus(m adminapi.ModelView) string {
|
||||
switch {
|
||||
case m.Parked:
|
||||
return "parked"
|
||||
case m.HeldFor != "":
|
||||
return "gave room to " + m.HeldFor
|
||||
case m.Unplaceable:
|
||||
return "cannot place"
|
||||
case m.Min == 0 && m.Total == 0:
|
||||
@@ -324,15 +326,15 @@ func newModelsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
}
|
||||
|
||||
w := tab()
|
||||
fmt.Fprintln(w, "MODEL\tSERVING\tREPLICAS\tSTATUS\tRESERVES\tALIASES\tPROFILES\tFALLBACKS")
|
||||
fmt.Fprintln(w, "MODEL\tSERVING\tREPLICAS\tSTATUS\tCLASS\tRESERVES\tALIASES\tPROFILES\tFALLBACKS")
|
||||
var warnings []string
|
||||
for _, m := range models {
|
||||
serving := "no"
|
||||
if m.Ready > 0 {
|
||||
serving = "yes"
|
||||
}
|
||||
fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\t%s\n",
|
||||
m.Name, serving, m.Ready, m.Min, modelStatus(m), reserves(m),
|
||||
fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\t%s\t%s\n",
|
||||
m.Name, serving, m.Ready, m.Min, modelStatus(m), m.Class, reserves(m),
|
||||
join(m.Aliases), join(m.Profiles), join(m.Fallbacks))
|
||||
for _, x := range m.Warnings {
|
||||
warnings = append(warnings, m.Name+": "+x)
|
||||
|
||||
+32
-1
@@ -24,12 +24,14 @@ import (
|
||||
// minutes of outage. For a fixed production set that is a reasonable trade; for
|
||||
// a platform whose promise is that people can try things, it is the difference
|
||||
// between trying one and not bothering.
|
||||
func newModelCmd(client func() *ctlClient) *cobra.Command {
|
||||
func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: "model",
|
||||
Short: "Add or remove models while the cluster is running",
|
||||
}
|
||||
c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
|
||||
newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
|
||||
newTargetCmd(client, asJSON),
|
||||
newModelServeCmd(client))
|
||||
return c
|
||||
}
|
||||
@@ -233,3 +235,32 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
|
||||
c.Flags().StringVar(&profile, "profile", "", "one profile; every profile when left out")
|
||||
return c
|
||||
}
|
||||
|
||||
func newClassCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "class <model> critical|standard|opportunistic|default",
|
||||
Short: "Set how much a model's capacity is worth against another's",
|
||||
Long: "When a model cannot be placed, replicas of a lower class are stopped to\n" +
|
||||
"make room for it:\n\n" +
|
||||
" critical never gives up the replicas of its minimum\n" +
|
||||
" standard gives way to critical\n" +
|
||||
" opportunistic gives way to both; the default for an on-demand model\n\n" +
|
||||
"Within a class, a model below its minimum may take what another grew into\n" +
|
||||
"under load, but not its minimum. \"default\" goes back to what the model's\n" +
|
||||
"scaling implies: opportunistic when on demand, standard otherwise.",
|
||||
Args: cobra.ExactArgs(2),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
class := args[1]
|
||||
if class == "default" {
|
||||
class = ""
|
||||
}
|
||||
body, _ := json.Marshal(map[string]string{"class": class})
|
||||
var res map[string]string
|
||||
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/class", body, &res); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s is %s\n", res["model"], res["class"])
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,230 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"strconv"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func newQuotaCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: "quota",
|
||||
Short: "What parts of the tenancy tree are guaranteed of the cluster, and may hold",
|
||||
Long: "A capacity quota belongs to an organization, team or project, and counts the\n" +
|
||||
"GPUs, CPU cores and memory held by replicas of the models it owns (see\n" +
|
||||
"`llmgw ctl model owner`), and of those its teams and projects own.\n\n" +
|
||||
"The guarantee is kept for it: when it holds less and a model of its needs\n" +
|
||||
"room, replicas other scopes hold beyond their own guarantees are stopped to\n" +
|
||||
"make it — whatever their class. What it holds within its guarantee is\n" +
|
||||
"never taken by another scope's model. Beyond it, it borrows: free room is\n" +
|
||||
"used, and given back when asked for.\n\n" +
|
||||
"The limit is the most it may hold, borrowing included; a model that would\n" +
|
||||
"take it past is not given another replica.\n\n" +
|
||||
"Scopes are written org:Acme, team:Acme/ml or project:Acme/ml/training, or\n" +
|
||||
"with IDs. Changes take effect within half a minute.",
|
||||
}
|
||||
c.AddCommand(newQuotaListCmd(client, asJSON), newQuotaSetCmd(client), newQuotaClearCmd(client))
|
||||
return c
|
||||
}
|
||||
|
||||
func newQuotaListCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "list",
|
||||
Short: "Every quota, with what its scope holds now",
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var qs []adminapi.QuotaView
|
||||
if err := client().get(cmd.Context(), "/api/v1/quotas", &qs); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return json.NewEncoder(cmd.OutOrStdout()).Encode(qs)
|
||||
}
|
||||
if len(qs) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "No capacity quotas: everything shares the cluster by class alone.")
|
||||
return nil
|
||||
}
|
||||
w := tab()
|
||||
fmt.Fprintln(w, "SCOPE\tNAME\tGPUS\tCORES\tMEMORY GB")
|
||||
for _, q := range qs {
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", q.Scope, q.Name,
|
||||
quotaCell(q.Holds.GPUs, q.Guarantee.GPUs, q.Limit.GPUs),
|
||||
quotaCell(q.Holds.CPUCores, q.Guarantee.CPUCores, q.Limit.CPUCores),
|
||||
quotaCell(q.Holds.MemoryGB, q.Guarantee.MemoryGB, q.Limit.MemoryGB))
|
||||
}
|
||||
if err := w.Flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "\nEach cell: holds / guaranteed (at most limit). - is not governed.")
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// quotaCell is "6 / 8 (≤ 16)": what is held, what is guaranteed, the limit.
|
||||
func quotaCell(holds int, guarantee, limit *int) string {
|
||||
if guarantee == nil && limit == nil {
|
||||
return "-"
|
||||
}
|
||||
out := strconv.Itoa(holds) + " / "
|
||||
if guarantee != nil {
|
||||
out += strconv.Itoa(*guarantee)
|
||||
} else {
|
||||
out += "-"
|
||||
}
|
||||
if limit != nil {
|
||||
out += fmt.Sprintf(" (≤ %d)", *limit)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func newQuotaSetCmd(client func() *ctlClient) *cobra.Command {
|
||||
var gpus, cores, mem, lgpus, lcores, lmem int
|
||||
c := &cobra.Command{
|
||||
Use: "set SCOPE [--gpus N] [--cores N] [--memory GB] [--limit-gpus N] ...",
|
||||
Short: "Set a scope's guarantee and limit, replacing what it had",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
f := cmd.Flags()
|
||||
pick := func(name string, v int) *int {
|
||||
if !f.Changed(name) {
|
||||
return nil
|
||||
}
|
||||
return &v
|
||||
}
|
||||
req := adminapi.QuotaRequest{Scope: args[0],
|
||||
Guarantee: store.Resources{GPUs: pick("gpus", gpus), CPUCores: pick("cores", cores), MemoryGB: pick("memory", mem)},
|
||||
Limit: store.Resources{GPUs: pick("limit-gpus", lgpus), CPUCores: pick("limit-cores", lcores), MemoryGB: pick("limit-memory", lmem)},
|
||||
}
|
||||
body, _ := json.Marshal(req)
|
||||
var res map[string]string
|
||||
if err := client().put(cmd.Context(), "/api/v1/quotas", body, &res); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: set; it takes effect within half a minute\n", res["scope"])
|
||||
return nil
|
||||
},
|
||||
}
|
||||
f := c.Flags()
|
||||
f.IntVar(&gpus, "gpus", 0, "GPUs guaranteed")
|
||||
f.IntVar(&cores, "cores", 0, "CPU cores guaranteed")
|
||||
f.IntVar(&mem, "memory", 0, "memory guaranteed, in GB")
|
||||
f.IntVar(&lgpus, "limit-gpus", 0, "the most GPUs it may hold")
|
||||
f.IntVar(&lcores, "limit-cores", 0, "the most CPU cores it may hold")
|
||||
f.IntVar(&lmem, "limit-memory", 0, "the most memory it may hold, in GB")
|
||||
return c
|
||||
}
|
||||
|
||||
func newQuotaClearCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "clear SCOPE",
|
||||
Short: "Remove a scope's quota",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var res map[string]string
|
||||
if err := client().delete(cmd.Context(), "/api/v1/quotas/"+url.PathEscape(args[0]), &res); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: quota removed\n", res["removed"])
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newOwnerCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "owner <model> SCOPE|shared",
|
||||
Short: "Set whose capacity quota a model counts against",
|
||||
Long: "SCOPE is org:Acme, team:Acme/ml or project:Acme/ml/training. A shared\n" +
|
||||
"model belongs to the cluster and counts against nobody's quota.",
|
||||
Args: cobra.ExactArgs(2),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
owner := args[1]
|
||||
if owner == "shared" {
|
||||
owner = ""
|
||||
}
|
||||
body, _ := json.Marshal(map[string]string{"owner": owner})
|
||||
var res map[string]string
|
||||
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/owner", body, &res); err != nil {
|
||||
return err
|
||||
}
|
||||
if res["owner"] == "" {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s is shared\n", res["model"])
|
||||
} else {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s counts against %s\n", res["model"], res["owner"])
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newTargetCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
var tps float64
|
||||
var apply bool
|
||||
c := &cobra.Command{
|
||||
Use: "target <model> --tokens-per-sec N [--apply]",
|
||||
Short: "How many replicas a model needs for a throughput, and whether they fit",
|
||||
Long: "Sized from what one replica was measured to serve while meeting its SLO\n" +
|
||||
"(`llmgw ctl bench MODEL --save`), on the profile its replicas run on.\n" +
|
||||
"--apply makes that the number it always runs, if the cluster has room.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if tps <= 0 {
|
||||
return fmt.Errorf("--tokens-per-sec is required")
|
||||
}
|
||||
var v adminapi.TargetView
|
||||
path := "/api/v1/models/" + url.PathEscape(args[0]) + "/target?tokens_per_sec=" +
|
||||
strconv.FormatFloat(tps, 'f', -1, 64)
|
||||
if err := client().get(cmd.Context(), path, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
fmt.Fprintf(out, "%s: %.0f tokens/s needs %d replicas at %.1f each (measured on %s); %d running\n",
|
||||
v.Model, v.TokPerSec, v.Needed, v.PerReplica, v.Profile, v.Current)
|
||||
switch {
|
||||
case v.More == 0:
|
||||
fmt.Fprintln(out, "It already has them.")
|
||||
case v.Fits:
|
||||
fmt.Fprintf(out, "%d more, and there is room for them.\n", v.More)
|
||||
default:
|
||||
fmt.Fprintf(out, "%d more, and there is room for %d:\n", v.More, v.Room)
|
||||
for _, b := range v.Blocked {
|
||||
fmt.Fprintln(out, " "+b)
|
||||
}
|
||||
}
|
||||
if !apply || v.More == 0 {
|
||||
return nil
|
||||
}
|
||||
if !v.Fits {
|
||||
return fmt.Errorf("not applied: the cluster has no room for them")
|
||||
}
|
||||
var models []adminapi.ModelView
|
||||
if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
|
||||
return err
|
||||
}
|
||||
top := v.Needed
|
||||
for _, m := range models {
|
||||
if m.Name == v.Model {
|
||||
top = max(top, m.Max)
|
||||
}
|
||||
}
|
||||
body, _ := json.Marshal(adminapi.ScaleRequest{Min: v.Needed, Max: top})
|
||||
if err := client().post(cmd.Context(), "/api/v1/models/"+url.PathEscape(v.Model)+"/scale", body, nil); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(out, "Applied: %s runs %d (up to %d under load).\n", v.Model, v.Needed, top)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().Float64Var(&tps, "tokens-per-sec", 0, "the throughput to size for")
|
||||
c.Flags().BoolVar(&apply, "apply", false, "set it as the number of replicas the model always runs")
|
||||
return c
|
||||
}
|
||||
@@ -226,6 +226,8 @@ func newServerCmd() *cobra.Command {
|
||||
// And what each group of the tenancy tree spent, which is
|
||||
// the other half of a quota being reviewable.
|
||||
api.SetUsage(st)
|
||||
// Capacity quotas, set and read over this API.
|
||||
api.SetQuotas(st)
|
||||
}
|
||||
api.Register(adminMux)
|
||||
|
||||
@@ -483,6 +485,12 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
|
||||
if st := h.Store(); st != nil {
|
||||
lead = control.NewLeadership(st, cfg.Control.LeaseTTL, log)
|
||||
ctl.SetLeadership(lead)
|
||||
// The tenancy tree's capacity quotas: what each part is
|
||||
// guaranteed and may hold.
|
||||
ctl.Reconciler().SetQuotas(control.QuotaPolicy(st, log))
|
||||
// And what each model was measured to serve, for scaling and for
|
||||
// choosing between profiles.
|
||||
ctl.Reconciler().SetMeasurements(control.Measurements(st, log))
|
||||
log.Info("competing to be the control plane that places work",
|
||||
"owner", lead.Owner())
|
||||
}
|
||||
|
||||
@@ -42,6 +42,7 @@ type API struct {
|
||||
models Models
|
||||
capacity Capacity
|
||||
usage Usage
|
||||
quotas Quotas
|
||||
ctl *control.Controller
|
||||
}
|
||||
|
||||
@@ -58,6 +59,27 @@ func (a *API) Register(mux *http.ServeMux) {
|
||||
mux.HandleFunc("DELETE /api/v1/models/{name}", a.removeModel)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
|
||||
a.registerQuotas(mux)
|
||||
mux.HandleFunc("GET /api/v1/models/{name}/target", a.target)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/class", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.models == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
Class string `json:"class"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
m, err := a.models.SetClass(r.Context(), r.PathValue("name"), req.Class, actorOf(r))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"model": m.Name, "class": m.EffectiveClass()})
|
||||
})
|
||||
mux.HandleFunc("PUT /api/v1/models/{name}/capacity", a.putCapacity)
|
||||
mux.HandleFunc("GET /api/v1/capacity", a.listCapacity)
|
||||
mux.HandleFunc("GET /api/v1/usage", a.usageByScope)
|
||||
@@ -282,6 +304,12 @@ type ModelView struct {
|
||||
// Warnings are what makes its reservation wrong: nothing declared where
|
||||
// placement needs a figure, or less declared than it uses.
|
||||
Warnings []string `json:"warnings,omitempty"`
|
||||
// Class is its capacity class; Preempted the replicas it lost to a higher
|
||||
// class; HeldFor the model it is giving room to now, if any.
|
||||
Class string `json:"class"`
|
||||
Owner string `json:"owner,omitempty"`
|
||||
Preempted int `json:"preempted,omitempty"`
|
||||
HeldFor string `json:"held_for,omitempty"`
|
||||
}
|
||||
|
||||
// ReplicaView is a replica as `ctl replicas` shows it.
|
||||
@@ -372,6 +400,8 @@ type HeadroomView struct {
|
||||
Limited bool `json:"limited,omitempty"`
|
||||
// Blocked says why one more does not fit, a line per node considered.
|
||||
Blocked []string `json:"blocked,omitempty"`
|
||||
// TokPerSec is what one more replica was measured to serve; 0 unmeasured.
|
||||
TokPerSec float64 `json:"tok_per_sec,omitempty"`
|
||||
}
|
||||
|
||||
// RejectionView is one rejected candidate.
|
||||
@@ -528,6 +558,7 @@ func (a *API) Models() []ModelView {
|
||||
Fallbacks: m.Fallbacks,
|
||||
Desired: h.Desired, Failures: h.Failures,
|
||||
Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
|
||||
Class: m.EffectiveClass(), Owner: m.Owner, Preempted: h.Preempted, HeldFor: h.HeldFor,
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
v.Profiles = append(v.Profiles, p.Name)
|
||||
@@ -608,6 +639,8 @@ type Models interface {
|
||||
Remove(ctx context.Context, name string) error
|
||||
Scale(ctx context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error)
|
||||
Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
|
||||
SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
|
||||
SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
|
||||
}
|
||||
|
||||
// ResourcesRequest sets what a model's profiles reserve of a node.
|
||||
@@ -928,7 +961,7 @@ func (a *API) Headroom(model string, limit int) (HeadroomView, bool) {
|
||||
return HeadroomView{}, false
|
||||
}
|
||||
return HeadroomView{Model: model, Current: h.Current, More: h.More, Nodes: h.Nodes,
|
||||
Limited: h.Limited, Blocked: h.Blocked}, true
|
||||
Limited: h.Limited, Blocked: h.Blocked, TokPerSec: h.TokPerSec}, true
|
||||
}
|
||||
|
||||
func (a *API) Explain(model string) ExplainView {
|
||||
|
||||
@@ -258,6 +258,14 @@ type fakeModels struct {
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeModels) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
|
||||
return catalog.Model{Name: name, Owner: owner}, f.err
|
||||
}
|
||||
|
||||
func (f *fakeModels) SetClass(_ context.Context, name, class, _ string) (catalog.Model, error) {
|
||||
return catalog.Model{Name: name, Class: class}, f.err
|
||||
}
|
||||
|
||||
func (f *fakeModels) Resources(_ context.Context, name, profile string, mem, cores int, _ string) (catalog.Model, error) {
|
||||
if f.err != nil {
|
||||
return catalog.Model{}, f.err
|
||||
|
||||
@@ -0,0 +1,237 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Quotas is where capacity quotas and the tenancy tree they name are kept.
|
||||
// The store implements it.
|
||||
type Quotas interface {
|
||||
CapacityQuotas(ctx context.Context) ([]store.CapacityQuota, error)
|
||||
PutCapacityQuota(ctx context.Context, q store.CapacityQuota) error
|
||||
DeleteCapacityQuota(ctx context.Context, scope store.Scope, id string) error
|
||||
ScopeParents(ctx context.Context) (map[string][]string, error)
|
||||
AllOrgs(ctx context.Context) ([]store.Org, error)
|
||||
AllTeams(ctx context.Context) ([]store.TeamPath, error)
|
||||
AllProjects(ctx context.Context) ([]store.ProjectPath, error)
|
||||
}
|
||||
|
||||
// SetQuotas supplies the quota store, which mounts the quota endpoints'
|
||||
// behaviour; without it they answer 501.
|
||||
func (a *API) SetQuotas(q Quotas) { a.quotas = q }
|
||||
|
||||
// QuotaView is one scope's quota with what it holds now.
|
||||
type QuotaView struct {
|
||||
// Scope is team:ID; Name is how people know it, "Acme / ml".
|
||||
Scope string `json:"scope"`
|
||||
Name string `json:"name"`
|
||||
Guarantee store.Resources `json:"guarantee"`
|
||||
Limit store.Resources `json:"limit"`
|
||||
Holds quota.Amount `json:"holds"`
|
||||
UpdatedBy string `json:"updated_by,omitempty"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
}
|
||||
|
||||
// QuotaRequest sets a scope's quota. Scope may be written with an ID or a
|
||||
// name: org:Acme, team:Acme/ml, project:Acme/ml/training.
|
||||
type QuotaRequest struct {
|
||||
Scope string `json:"scope"`
|
||||
Guarantee store.Resources `json:"guarantee"`
|
||||
Limit store.Resources `json:"limit"`
|
||||
}
|
||||
|
||||
func (a *API) registerQuotas(mux *http.ServeMux) {
|
||||
mux.HandleFunc("GET /api/v1/quotas", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.quotas == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
views, err := a.QuotaViews(r.Context())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, views)
|
||||
})
|
||||
mux.HandleFunc("PUT /api/v1/quotas", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.quotas == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
var req QuotaRequest
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<14)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
scope, id, _, err := a.resolveScope(r.Context(), req.Scope)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
if !req.Guarantee.Any() && !req.Limit.Any() {
|
||||
writeError(w, http.StatusBadRequest, "a quota needs a guarantee or a limit; to remove one, DELETE it")
|
||||
return
|
||||
}
|
||||
q := store.CapacityQuota{Scope: scope, ScopeID: id, Guarantee: req.Guarantee, Limit: req.Limit,
|
||||
UpdatedBy: actorOf(r)}
|
||||
if err := a.quotas.PutCapacityQuota(r.Context(), q); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"scope": quota.Key(scope, id)})
|
||||
})
|
||||
mux.HandleFunc("DELETE /api/v1/quotas/{scope}", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.quotas == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
scope, id, _, err := a.resolveScope(r.Context(), r.PathValue("scope"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
if err := a.quotas.DeleteCapacityQuota(r.Context(), scope, id); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"removed": quota.Key(scope, id)})
|
||||
})
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/owner", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.models == nil || a.quotas == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
Owner string `json:"owner"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
owner := ""
|
||||
if req.Owner != "" {
|
||||
scope, id, _, err := a.resolveScope(r.Context(), req.Owner)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
owner = quota.Key(scope, id)
|
||||
}
|
||||
m, err := a.models.SetOwner(r.Context(), r.PathValue("name"), owner, actorOf(r))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"model": m.Name, "owner": m.Owner})
|
||||
})
|
||||
}
|
||||
|
||||
// QuotaViews lists every quota with what its scope holds now.
|
||||
func (a *API) QuotaViews(ctx context.Context) ([]QuotaView, error) {
|
||||
qs, err := a.quotas.CapacityQuotas(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
parents, err := a.quotas.ScopeParents(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
names, err := a.scopeNames(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
usage := quota.New(qs, parents).Usage(a.ctl.Reconciler().Holdings())
|
||||
out := make([]QuotaView, 0, len(qs))
|
||||
for _, q := range qs {
|
||||
key := quota.Key(q.Scope, q.ScopeID)
|
||||
out = append(out, QuotaView{Scope: key, Name: names[key], Guarantee: q.Guarantee, Limit: q.Limit,
|
||||
Holds: usage[key], UpdatedBy: q.UpdatedBy, UpdatedAt: q.UpdatedAt})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// scopeNames maps org:ID, team:ID and project:ID to their paths.
|
||||
func (a *API) scopeNames(ctx context.Context) (map[string]string, error) {
|
||||
out := map[string]string{}
|
||||
orgs, err := a.quotas.AllOrgs(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, o := range orgs {
|
||||
out["org:"+o.ID] = o.Name
|
||||
}
|
||||
teams, err := a.quotas.AllTeams(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, t := range teams {
|
||||
out["team:"+t.ID] = t.OrgName + " / " + t.Name
|
||||
}
|
||||
projects, err := a.quotas.AllProjects(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, p := range projects {
|
||||
out["project:"+p.ID] = p.Path
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// resolveScope reads org:X, team:X or project:X, where X is an ID or the
|
||||
// scope's path — Acme, Acme/ml, Acme/ml/training — and returns its ID and
|
||||
// path. A path that could mean two scopes is refused rather than guessed.
|
||||
func (a *API) resolveScope(ctx context.Context, s string) (store.Scope, string, string, error) {
|
||||
kind, what, ok := strings.Cut(strings.TrimSpace(s), ":")
|
||||
scope := store.Scope(kind)
|
||||
if !ok || what == "" || (scope != store.ScopeOrg && scope != store.ScopeTeam && scope != store.ScopeProject) {
|
||||
return "", "", "", fmt.Errorf("%q is not org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT", s)
|
||||
}
|
||||
names, err := a.scopeNames(ctx)
|
||||
if err != nil {
|
||||
return "", "", "", err
|
||||
}
|
||||
norm := func(p string) string {
|
||||
parts := strings.Split(p, "/")
|
||||
for i := range parts {
|
||||
parts[i] = strings.ToLower(strings.TrimSpace(parts[i]))
|
||||
}
|
||||
return strings.Join(parts, "/")
|
||||
}
|
||||
if name, ok := names[kind+":"+what]; ok {
|
||||
return scope, what, name, nil
|
||||
}
|
||||
match := func(same func(name string) bool) []string {
|
||||
var found []string
|
||||
for key, name := range names {
|
||||
if strings.HasPrefix(key, kind+":") && same(name) {
|
||||
found = append(found, key)
|
||||
}
|
||||
}
|
||||
return found
|
||||
}
|
||||
found := match(func(name string) bool { return norm(name) == norm(what) })
|
||||
if len(found) == 0 {
|
||||
// Its own name alone — ml rather than Acme/ml — when that is unique.
|
||||
found = match(func(name string) bool {
|
||||
parts := strings.Split(norm(name), "/")
|
||||
return parts[len(parts)-1] == norm(what)
|
||||
})
|
||||
}
|
||||
switch len(found) {
|
||||
case 0:
|
||||
return "", "", "", fmt.Errorf("there is no %s %q", kind, what)
|
||||
case 1:
|
||||
_, id, _ := strings.Cut(found[0], ":")
|
||||
return scope, id, names[found[0]], nil
|
||||
}
|
||||
return "", "", "", fmt.Errorf("%q could be any of %d %ss; give its ID", what, len(found), kind)
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// fakeQuotaStore is Acme with teams ml and web, and Scania with another ml.
|
||||
type fakeQuotaStore struct {
|
||||
quotas []store.CapacityQuota
|
||||
}
|
||||
|
||||
func (f *fakeQuotaStore) CapacityQuotas(context.Context) ([]store.CapacityQuota, error) {
|
||||
return f.quotas, nil
|
||||
}
|
||||
func (f *fakeQuotaStore) PutCapacityQuota(_ context.Context, q store.CapacityQuota) error {
|
||||
for i := range f.quotas {
|
||||
if f.quotas[i].Scope == q.Scope && f.quotas[i].ScopeID == q.ScopeID {
|
||||
f.quotas[i] = q
|
||||
return nil
|
||||
}
|
||||
}
|
||||
f.quotas = append(f.quotas, q)
|
||||
return nil
|
||||
}
|
||||
func (f *fakeQuotaStore) DeleteCapacityQuota(_ context.Context, s store.Scope, id string) error {
|
||||
for i, q := range f.quotas {
|
||||
if q.Scope == s && q.ScopeID == id {
|
||||
f.quotas = append(f.quotas[:i], f.quotas[i+1:]...)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (f *fakeQuotaStore) ScopeParents(context.Context) (map[string][]string, error) {
|
||||
return map[string][]string{"team:t-ml": {"org:o-acme"}, "team:t-web": {"org:o-acme"},
|
||||
"team:t-sml": {"org:o-scania"}}, nil
|
||||
}
|
||||
func (f *fakeQuotaStore) AllOrgs(context.Context) ([]store.Org, error) {
|
||||
return []store.Org{{ID: "o-acme", Name: "Acme"}, {ID: "o-scania", Name: "Scania"}}, nil
|
||||
}
|
||||
func (f *fakeQuotaStore) AllTeams(context.Context) ([]store.TeamPath, error) {
|
||||
return []store.TeamPath{{ID: "t-ml", Name: "ml", OrgID: "o-acme", OrgName: "Acme"},
|
||||
{ID: "t-web", Name: "web", OrgID: "o-acme", OrgName: "Acme"},
|
||||
{ID: "t-sml", Name: "ml", OrgID: "o-scania", OrgName: "Scania"}}, nil
|
||||
}
|
||||
func (f *fakeQuotaStore) AllProjects(context.Context) ([]store.ProjectPath, error) { return nil, nil }
|
||||
|
||||
func TestAScopeIsNamedByPathOrIDAndNeverGuessed(t *testing.T) {
|
||||
a, _ := setup(t, testModels())
|
||||
a.SetQuotas(&fakeQuotaStore{})
|
||||
for in, want := range map[string]string{
|
||||
"org:Acme": "o-acme", "org:o-acme": "o-acme", "team:Acme/ml": "t-ml",
|
||||
"team: acme / ML ": "t-ml", "team:Scania/ml": "t-sml", "team:web": "t-web",
|
||||
} {
|
||||
_, id, _, err := a.resolveScope(t.Context(), in)
|
||||
if err != nil || id != want {
|
||||
t.Errorf("%q: %q %v, expected %q", in, id, err, want)
|
||||
}
|
||||
}
|
||||
for _, bad := range []string{"team:ml", "team:Acme/nope", "group:Acme", "Acme", "org:"} {
|
||||
if _, _, _, err := a.resolveScope(t.Context(), bad); err == nil {
|
||||
t.Errorf("%q was resolved", bad)
|
||||
}
|
||||
}
|
||||
if _, _, _, err := a.resolveScope(t.Context(), "team:ml"); err == nil || !strings.Contains(err.Error(), "could be any of 2") {
|
||||
t.Errorf("an ambiguous name should say so: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAQuotaIsSetListedAndCleared(t *testing.T) {
|
||||
a, _ := setup(t, testModels())
|
||||
qs := &fakeQuotaStore{}
|
||||
a.SetQuotas(qs)
|
||||
mux := http.NewServeMux()
|
||||
a.Register(mux)
|
||||
call := func(method, path, body string) *httptest.ResponseRecorder {
|
||||
req := httptest.NewRequest(method, path, strings.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("X-Llmgw-Actor", "ops")
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, req)
|
||||
return rec
|
||||
}
|
||||
if rec := call("PUT", "/api/v1/quotas", `{"scope":"team:Acme/ml","guarantee":{"gpus":8},"limit":{"gpus":16}}`); rec.Code != 200 {
|
||||
t.Fatalf("set: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
if len(qs.quotas) != 1 || qs.quotas[0].ScopeID != "t-ml" || *qs.quotas[0].Guarantee.GPUs != 8 || qs.quotas[0].UpdatedBy != "ops" {
|
||||
t.Errorf("stored %+v", qs.quotas)
|
||||
}
|
||||
rec := call("GET", "/api/v1/quotas", "")
|
||||
var views []QuotaView
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &views); err != nil || len(views) != 1 || views[0].Name != "Acme / ml" {
|
||||
t.Errorf("list %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
if rec := call("PUT", "/api/v1/quotas", `{"scope":"team:Acme/ml"}`); rec.Code != 400 {
|
||||
t.Errorf("an empty quota was accepted: %d", rec.Code)
|
||||
}
|
||||
if rec := call("DELETE", "/api/v1/quotas/team:Acme%2Fml", ""); rec.Code != 200 || len(qs.quotas) != 0 {
|
||||
t.Errorf("clear: %d %s %+v", rec.Code, rec.Body, qs.quotas)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"net/http"
|
||||
"strconv"
|
||||
)
|
||||
|
||||
// TargetView is how many replicas a model needs to serve a throughput, from
|
||||
// what one replica was measured to serve, and whether the cluster has room.
|
||||
type TargetView struct {
|
||||
Model string `json:"model"`
|
||||
// TokPerSec is what was asked for; PerReplica what one replica was
|
||||
// measured to serve at its SLO, on Profile.
|
||||
TokPerSec float64 `json:"tok_per_sec"`
|
||||
PerReplica float64 `json:"per_replica_tok_per_sec"`
|
||||
Profile string `json:"profile"`
|
||||
// Needed replicas in all; Current hold capacity now; More is how many of
|
||||
// Needed are not running, and Room how many more fit.
|
||||
Needed int `json:"needed"`
|
||||
Current int `json:"current"`
|
||||
More int `json:"more"`
|
||||
Room int `json:"room"`
|
||||
Fits bool `json:"fits"`
|
||||
Blocked []string `json:"blocked,omitempty"`
|
||||
}
|
||||
|
||||
// Target sizes a model for a throughput.
|
||||
func (a *API) Target(model string, tokPerSec float64) (TargetView, error) {
|
||||
meas, profile, ok := a.ctl.Reconciler().MeasuredFor(model)
|
||||
if !ok || meas.TokPerSec <= 0 {
|
||||
return TargetView{}, fmt.Errorf("%s has not been measured; run `llmgw ctl bench %s --save` first", model, model)
|
||||
}
|
||||
v := TargetView{Model: model, TokPerSec: tokPerSec, PerReplica: meas.TokPerSec, Profile: profile,
|
||||
Needed: max(int(math.Ceil(tokPerSec/meas.TokPerSec)), 1)}
|
||||
h, ok := a.Headroom(model, v.Needed)
|
||||
if !ok {
|
||||
return TargetView{}, fmt.Errorf("no model %q", model)
|
||||
}
|
||||
v.Current = h.Current
|
||||
v.More = max(v.Needed-h.Current, 0)
|
||||
v.Room = h.More
|
||||
v.Fits = v.More <= h.More || h.Limited
|
||||
if !v.Fits {
|
||||
v.Blocked = h.Blocked
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
func (a *API) target(w http.ResponseWriter, r *http.Request) {
|
||||
tps, err := strconv.ParseFloat(r.URL.Query().Get("tokens_per_sec"), 64)
|
||||
if err != nil || tps <= 0 {
|
||||
writeError(w, http.StatusBadRequest, "tokens_per_sec must be a positive number")
|
||||
return
|
||||
}
|
||||
v, err := a.Target(r.PathValue("name"), tps)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
)
|
||||
|
||||
func TestATargetIsSizedFromTheMeasurementAndTheRoom(t *testing.T) {
|
||||
// Three 8-GPU nodes; one tp8 replica was measured at 500 tokens/s.
|
||||
a, ctl := setup(t, testModels(), gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
|
||||
if _, err := a.Target("llama", 1000); err == nil || !strings.Contains(err.Error(), "ctl bench llama --save") {
|
||||
t.Errorf("an unmeasured model should say how to measure it: %v", err)
|
||||
}
|
||||
ctl.Reconciler().SetMeasurements(func() map[string]reconciler.Measured {
|
||||
return map[string]reconciler.Measured{"llama/tp8": {MaxConcurrency: 16, TokPerSec: 500}}
|
||||
})
|
||||
v, err := a.Target("llama", 1200)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Needed != 3 || v.PerReplica != 500 || v.Profile != "tp8" || v.More != 3 || !v.Fits {
|
||||
t.Errorf("1200 tok/s at 500 each is 3 replicas on 3 empty nodes: %+v", v)
|
||||
}
|
||||
v, _ = a.Target("llama", 2000)
|
||||
if v.Needed != 4 || v.Fits || v.Room != 3 || len(v.Blocked) == 0 {
|
||||
t.Errorf("a fourth does not fit on three nodes, and should say why: %+v", v)
|
||||
}
|
||||
}
|
||||
@@ -149,10 +149,19 @@ type Model struct {
|
||||
// choice by guesswork.
|
||||
Description string `json:"description,omitempty" koanf:"description"`
|
||||
// Source is the weights path or a HuggingFace id.
|
||||
Source string `json:"source,omitempty" koanf:"source"`
|
||||
Profiles []Profile `json:"profiles,omitempty" koanf:"profiles"`
|
||||
Scaling Scaling `json:"scaling,omitempty" koanf:"scaling"`
|
||||
Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
|
||||
Source string `json:"source,omitempty" koanf:"source"`
|
||||
Profiles []Profile `json:"profiles,omitempty" koanf:"profiles"`
|
||||
Scaling Scaling `json:"scaling,omitempty" koanf:"scaling"`
|
||||
// Class is how much this model's capacity is worth against another's,
|
||||
// when there is not enough for both: critical, standard or opportunistic.
|
||||
// Empty is standard, or opportunistic for a model loaded on demand. See
|
||||
// EffectiveClass.
|
||||
Class string `json:"class,omitempty" koanf:"class"`
|
||||
// Owner is the part of the tenancy tree whose capacity quota this model
|
||||
// counts against, written org:ID, team:ID or project:ID. Empty is shared:
|
||||
// held by the cluster, counted against nobody.
|
||||
Owner string `json:"owner,omitempty" koanf:"owner"`
|
||||
Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
|
||||
// RoutingProfile selects how the router scores endpoints for this model.
|
||||
RoutingProfile string `json:"routing_profile,omitempty" koanf:"routing_profile"`
|
||||
SLO SLO `json:"slo,omitempty" koanf:"slo"`
|
||||
@@ -299,6 +308,12 @@ func (m Model) validate() error {
|
||||
if len(m.Profiles) == 0 {
|
||||
return fmt.Errorf("model %q: at least one profile is required", m.Name)
|
||||
}
|
||||
if m.Owner != "" && !validOwner(m.Owner) {
|
||||
return fmt.Errorf("model %q: owner %q is not org:ID, team:ID or project:ID", m.Name, m.Owner)
|
||||
}
|
||||
if !validClass(m.Class) {
|
||||
return fmt.Errorf("model %q: class %q is not one of critical, standard, opportunistic", m.Name, m.Class)
|
||||
}
|
||||
if m.Scaling.Max < m.Scaling.Min {
|
||||
return fmt.Errorf("model %q: scaling max %d is below min %d", m.Name, m.Scaling.Max, m.Scaling.Min)
|
||||
}
|
||||
@@ -503,3 +518,49 @@ func SuggestCores(cpuCounts []int) int {
|
||||
}
|
||||
return max(smallest/2, 1)
|
||||
}
|
||||
|
||||
// The capacity classes, lowest first.
|
||||
const (
|
||||
ClassOpportunistic = "opportunistic"
|
||||
ClassStandard = "standard"
|
||||
ClassCritical = "critical"
|
||||
)
|
||||
|
||||
func validClass(c string) bool {
|
||||
return c == "" || c == ClassOpportunistic || c == ClassStandard || c == ClassCritical
|
||||
}
|
||||
|
||||
// EffectiveClass is the class a model is treated as: what it says, or, when
|
||||
// it says nothing, opportunistic for one loaded on demand — it holds capacity
|
||||
// only while asked for, and has always been the first to give it up — and
|
||||
// standard for everything else.
|
||||
func (m Model) EffectiveClass() string {
|
||||
if m.Class != "" {
|
||||
return m.Class
|
||||
}
|
||||
if m.Scaling.OnDemand() {
|
||||
return ClassOpportunistic
|
||||
}
|
||||
return ClassStandard
|
||||
}
|
||||
|
||||
// ClassRank orders classes: a higher rank may take capacity from a lower one.
|
||||
func ClassRank(c string) int {
|
||||
switch c {
|
||||
case ClassCritical:
|
||||
return 3
|
||||
case ClassStandard, "":
|
||||
return 2
|
||||
default:
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
func validOwner(o string) bool {
|
||||
for _, p := range []string{"org:", "team:", "project:"} {
|
||||
if strings.HasPrefix(o, p) && len(o) > len(p) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -375,3 +375,6 @@ func EngineKeys(secret []byte) func(replicaID string) string {
|
||||
return "llmgw-" + hex.EncodeToString(m.Sum(nil))[:40]
|
||||
}
|
||||
}
|
||||
|
||||
// QuotaUse is every capacity quota beside what its scope holds.
|
||||
func (c *Controller) QuotaUse() []reconciler.QuotaUse { return c.rec.QuotaUse() }
|
||||
@@ -51,12 +51,16 @@ type HealthSource interface {
|
||||
type Collector struct {
|
||||
src Source
|
||||
|
||||
modelDesired *prometheus.Desc
|
||||
modelFailures *prometheus.Desc
|
||||
modelParked *prometheus.Desc
|
||||
modelUnplaceable *prometheus.Desc
|
||||
engineInstall *prometheus.Desc
|
||||
leader *prometheus.Desc
|
||||
modelDesired *prometheus.Desc
|
||||
modelFailures *prometheus.Desc
|
||||
modelParked *prometheus.Desc
|
||||
modelUnplaceable *prometheus.Desc
|
||||
modelPreempted *prometheus.Desc
|
||||
capacityHolds *prometheus.Desc
|
||||
capacityGuarantee *prometheus.Desc
|
||||
capacityLimit *prometheus.Desc
|
||||
engineInstall *prometheus.Desc
|
||||
leader *prometheus.Desc
|
||||
|
||||
nodeUp *prometheus.Desc
|
||||
nodeGPUs *prometheus.Desc
|
||||
@@ -168,6 +172,18 @@ func New(src Source) *Collector {
|
||||
modelUnplaceable: prometheus.NewDesc("llmgw_model_unplaceable",
|
||||
"1 when a model is short of replicas and the last attempt found no node to place one on.",
|
||||
[]string{"model"}, nil),
|
||||
capacityHolds: prometheus.NewDesc("llmgw_capacity_holds",
|
||||
"What a part of the tenancy tree holds of a resource, its own models' replicas and those beneath it.",
|
||||
[]string{"scope", "resource"}, nil),
|
||||
capacityGuarantee: prometheus.NewDesc("llmgw_capacity_guarantee",
|
||||
"What a part of the tenancy tree is guaranteed of a resource.",
|
||||
[]string{"scope", "resource"}, nil),
|
||||
capacityLimit: prometheus.NewDesc("llmgw_capacity_limit",
|
||||
"The most of a resource a part of the tenancy tree may hold.",
|
||||
[]string{"scope", "resource"}, nil),
|
||||
modelPreempted: prometheus.NewDesc("llmgw_model_preempted_total",
|
||||
"Replicas of a model stopped to make room for a model of a higher capacity class.",
|
||||
[]string{"model", "class"}, nil),
|
||||
engineInstall: prometheus.NewDesc("llmgw_node_engine_install",
|
||||
"Where installing an engine stands on a node: 1 for the current state.",
|
||||
[]string{"node", "engine", "version", "state"}, nil),
|
||||
@@ -236,7 +252,8 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
|
||||
c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
|
||||
c.gpuNVLink, c.gpuECC,
|
||||
c.replicaState, c.replicaReady, c.replicaIdle,
|
||||
c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable,
|
||||
c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable, c.modelPreempted,
|
||||
c.capacityHolds, c.capacityGuarantee, c.capacityLimit,
|
||||
c.engineInstall, c.leader,
|
||||
} {
|
||||
ch <- d
|
||||
@@ -246,6 +263,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
|
||||
// Collect implements prometheus.Collector.
|
||||
func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
now := time.Now()
|
||||
c.collectQuotas(ch)
|
||||
|
||||
loads := map[string]*inventory.Load{}
|
||||
for _, n := range c.src.Nodes() {
|
||||
@@ -455,6 +473,7 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
ch <- prometheus.MustNewConstMetric(c.modelFailures, prometheus.GaugeValue, float64(h.Failures), h.Model)
|
||||
ch <- prometheus.MustNewConstMetric(c.modelParked, prometheus.GaugeValue, b2f(h.Parked), h.Model)
|
||||
ch <- prometheus.MustNewConstMetric(c.modelUnplaceable, prometheus.GaugeValue, b2f(h.Unplaceable), h.Model)
|
||||
ch <- prometheus.MustNewConstMetric(c.modelPreempted, prometheus.CounterValue, float64(h.Preempted), h.Model, h.Class)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -464,3 +483,38 @@ func b2f(b bool) float64 {
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// QuotaSource adds the tenancy tree's capacity quotas. Asked for by type
|
||||
// assertion, like HealthSource.
|
||||
type QuotaSource interface {
|
||||
QuotaUse() []reconciler.QuotaUse
|
||||
}
|
||||
|
||||
func (c *Collector) collectQuotas(ch chan<- prometheus.Metric) {
|
||||
qs, ok := c.src.(QuotaSource)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
for _, q := range qs.QuotaUse() {
|
||||
for _, r := range []struct {
|
||||
name string
|
||||
holds int
|
||||
gua, lim *int
|
||||
}{
|
||||
{"gpus", q.Holds.GPUs, q.Guarantee.GPUs, q.Limit.GPUs},
|
||||
{"cpu_cores", q.Holds.CPUCores, q.Guarantee.CPUCores, q.Limit.CPUCores},
|
||||
{"memory_gb", q.Holds.MemoryGB, q.Guarantee.MemoryGB, q.Limit.MemoryGB},
|
||||
} {
|
||||
if r.gua == nil && r.lim == nil {
|
||||
continue
|
||||
}
|
||||
ch <- prometheus.MustNewConstMetric(c.capacityHolds, prometheus.GaugeValue, float64(r.holds), q.Scope, r.name)
|
||||
if r.gua != nil {
|
||||
ch <- prometheus.MustNewConstMetric(c.capacityGuarantee, prometheus.GaugeValue, float64(*r.gua), q.Scope, r.name)
|
||||
}
|
||||
if r.lim != nil {
|
||||
ch <- prometheus.MustNewConstMetric(c.capacityLimit, prometheus.GaugeValue, float64(*r.lim), q.Scope, r.name)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -242,6 +242,32 @@ func (s *Service) Resources(ctx context.Context, name, profile string, memoryGB,
|
||||
return final, err
|
||||
}
|
||||
|
||||
// SetClass sets a model's capacity class: critical, standard or
|
||||
// opportunistic, or empty to go back to what its scaling implies.
|
||||
func (s *Service) SetClass(ctx context.Context, name, class, by string) (catalog.Model, error) {
|
||||
final, err := s.update(ctx, name, by, "change its class", func(m *catalog.Model) error {
|
||||
m.Class = class
|
||||
return nil
|
||||
})
|
||||
if err == nil {
|
||||
s.log.Info("model class set", "model", name, "class", final.EffectiveClass(), "by", by)
|
||||
}
|
||||
return final, err
|
||||
}
|
||||
|
||||
// SetOwner sets the part of the tenancy tree a model's capacity counts
|
||||
// against, org:ID, team:ID or project:ID; empty makes it shared.
|
||||
func (s *Service) SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error) {
|
||||
final, err := s.update(ctx, name, by, "change its owner", func(m *catalog.Model) error {
|
||||
m.Owner = owner
|
||||
return nil
|
||||
})
|
||||
if err == nil {
|
||||
s.log.Info("model owner set", "model", name, "owner", owner, "by", by)
|
||||
}
|
||||
return final, err
|
||||
}
|
||||
|
||||
// update changes one model added at runtime and keeps the change.
|
||||
//
|
||||
// Built before anything is written, like Add: catalog.New is what applies the
|
||||
|
||||
@@ -0,0 +1,192 @@
|
||||
// Package quota decides what the tenancy tree is guaranteed of the cluster
|
||||
// and how much of it each part may hold.
|
||||
//
|
||||
// A scope — an organization, team or project — may be guaranteed GPUs, CPU
|
||||
// cores and memory, and limited to at most some amount of each. What a scope
|
||||
// holds is what the replicas of the models it owns, and of the models its
|
||||
// teams and projects own, have reserved. Holding more than its guarantee is
|
||||
// borrowing: allowed while the room is free, and taken back when another
|
||||
// scope within its own guarantee needs it.
|
||||
package quota
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Amount is how much of each resource something holds or needs.
|
||||
type Amount struct {
|
||||
GPUs, CPUCores, MemoryGB int
|
||||
}
|
||||
|
||||
// Plus adds two amounts.
|
||||
func (a Amount) Plus(b Amount) Amount {
|
||||
return Amount{a.GPUs + b.GPUs, a.CPUCores + b.CPUCores, a.MemoryGB + b.MemoryGB}
|
||||
}
|
||||
|
||||
// Over names the first resource in which a is above the governed amounts of
|
||||
// l, with both figures; empty when it is above none.
|
||||
func Over(a Amount, l store.Resources) string {
|
||||
switch {
|
||||
case l.GPUs != nil && a.GPUs > *l.GPUs:
|
||||
return fmt.Sprintf("%d GPUs of %d", a.GPUs, *l.GPUs)
|
||||
case l.CPUCores != nil && a.CPUCores > *l.CPUCores:
|
||||
return fmt.Sprintf("%d cores of %d", a.CPUCores, *l.CPUCores)
|
||||
case l.MemoryGB != nil && a.MemoryGB > *l.MemoryGB:
|
||||
return fmt.Sprintf("%d GB of memory of %d", a.MemoryGB, *l.MemoryGB)
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// Holding is one replica's hold on the cluster, for working out who holds
|
||||
// what.
|
||||
type Holding struct {
|
||||
Replica string
|
||||
Owner string
|
||||
Amount Amount
|
||||
Since time.Time
|
||||
}
|
||||
|
||||
// Policy is the quotas and the tree they sit in, read at one moment.
|
||||
type Policy struct {
|
||||
quotas map[string]store.CapacityQuota
|
||||
parents map[string][]string
|
||||
}
|
||||
|
||||
// New builds a policy from stored quotas and each scope's parents.
|
||||
func New(quotas []store.CapacityQuota, parents map[string][]string) *Policy {
|
||||
p := &Policy{quotas: map[string]store.CapacityQuota{}, parents: parents}
|
||||
for _, q := range quotas {
|
||||
p.quotas[Key(q.Scope, q.ScopeID)] = q
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// Key writes a scope the way owners are written: team:ID.
|
||||
func Key(s store.Scope, id string) string { return string(s) + ":" + id }
|
||||
|
||||
// Chain is an owner and every scope above it, nearest first.
|
||||
func (p *Policy) Chain(owner string) []string {
|
||||
if owner == "" {
|
||||
return nil
|
||||
}
|
||||
return append([]string{owner}, p.parents[owner]...)
|
||||
}
|
||||
|
||||
// Governing is the quotas along an owner's chain, nearest first.
|
||||
func (p *Policy) Governing(owner string) []store.CapacityQuota {
|
||||
if p == nil {
|
||||
return nil
|
||||
}
|
||||
var out []store.CapacityQuota
|
||||
for _, s := range p.Chain(owner) {
|
||||
if q, ok := p.quotas[s]; ok {
|
||||
out = append(out, q)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Quotas lists every quota.
|
||||
func (p *Policy) Quotas() []store.CapacityQuota {
|
||||
if p == nil {
|
||||
return nil
|
||||
}
|
||||
out := make([]store.CapacityQuota, 0, len(p.quotas))
|
||||
for _, q := range p.quotas {
|
||||
out = append(out, q)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
return Key(out[i].Scope, out[i].ScopeID) < Key(out[j].Scope, out[j].ScopeID)
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// Usage is what every scope holds, its own models' replicas and those of
|
||||
// everything beneath it.
|
||||
func (p *Policy) Usage(hs []Holding) map[string]Amount {
|
||||
out := map[string]Amount{}
|
||||
if p == nil {
|
||||
return out
|
||||
}
|
||||
for _, h := range hs {
|
||||
for _, s := range p.Chain(h.Owner) {
|
||||
out[s] = out[s].Plus(h.Amount)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Borrowed is the replicas held beyond a guarantee: of each governed scope's
|
||||
// replicas, oldest first, the ones that take it past what it is guaranteed.
|
||||
// A replica beyond any guarantee along its chain is borrowed.
|
||||
func (p *Policy) Borrowed(hs []Holding) map[string]bool {
|
||||
out := map[string]bool{}
|
||||
if p == nil || len(p.quotas) == 0 {
|
||||
return out
|
||||
}
|
||||
sorted := append([]Holding(nil), hs...)
|
||||
sort.Slice(sorted, func(i, j int) bool { return sorted[i].Since.Before(sorted[j].Since) })
|
||||
for key, q := range p.quotas {
|
||||
if !q.Guarantee.Any() {
|
||||
continue
|
||||
}
|
||||
var sum Amount
|
||||
for _, h := range sorted {
|
||||
if !contains(p.Chain(h.Owner), key) {
|
||||
continue
|
||||
}
|
||||
sum = sum.Plus(h.Amount)
|
||||
if Over(sum, q.Guarantee) != "" {
|
||||
out[h.Replica] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Guaranteed reports whether an owner, holding what it holds, is still within
|
||||
// every guarantee along its chain with need added — so it may take back what
|
||||
// others borrowed. An owner with no guarantee anywhere is never guaranteed.
|
||||
func (p *Policy) Guaranteed(owner string, usage map[string]Amount, need Amount) bool {
|
||||
gov := p.Governing(owner)
|
||||
any := false
|
||||
for _, q := range gov {
|
||||
if !q.Guarantee.Any() {
|
||||
continue
|
||||
}
|
||||
any = true
|
||||
if Over(usage[Key(q.Scope, q.ScopeID)].Plus(need), q.Guarantee) != "" {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return any
|
||||
}
|
||||
|
||||
// AtLimit says which limit along an owner's chain need would take it past,
|
||||
// empty when none.
|
||||
func (p *Policy) AtLimit(owner string, usage map[string]Amount, need Amount) string {
|
||||
for _, q := range p.Governing(owner) {
|
||||
if !q.Limit.Any() {
|
||||
continue
|
||||
}
|
||||
key := Key(q.Scope, q.ScopeID)
|
||||
if over := Over(usage[key].Plus(need), q.Limit); over != "" {
|
||||
return fmt.Sprintf("%s is at its capacity limit (%s)", key, strings.TrimSpace(over))
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func contains(list []string, v string) bool {
|
||||
for _, x := range list {
|
||||
if x == v {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package quota
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func n(v int) *int { return &v }
|
||||
|
||||
// acme has two teams; ml is guaranteed 8 GPUs, the org as a whole 12 and
|
||||
// no more than 16.
|
||||
func policy() *Policy {
|
||||
return New([]store.CapacityQuota{
|
||||
{Scope: store.ScopeTeam, ScopeID: "ml", Guarantee: store.Resources{GPUs: n(8)}},
|
||||
{Scope: store.ScopeOrg, ScopeID: "acme", Guarantee: store.Resources{GPUs: n(12)}, Limit: store.Resources{GPUs: n(16)}},
|
||||
}, map[string][]string{
|
||||
"team:ml": {"org:acme"}, "team:web": {"org:acme"},
|
||||
"project:train": {"team:ml", "org:acme"},
|
||||
})
|
||||
}
|
||||
|
||||
func at(min int) time.Time { return time.Date(2026, 1, 1, 0, min, 0, 0, time.UTC) }
|
||||
|
||||
func TestAScopeHoldsWhatEverythingBeneathItHolds(t *testing.T) {
|
||||
p := policy()
|
||||
u := p.Usage([]Holding{
|
||||
{Replica: "a", Owner: "project:train", Amount: Amount{GPUs: 4}},
|
||||
{Replica: "b", Owner: "team:web", Amount: Amount{GPUs: 2, CPUCores: 3}},
|
||||
{Replica: "c", Owner: "", Amount: Amount{GPUs: 8}}, // shared: nobody's
|
||||
})
|
||||
if u["team:ml"].GPUs != 4 || u["org:acme"].GPUs != 6 || u["org:acme"].CPUCores != 3 || u["project:train"].GPUs != 4 {
|
||||
t.Errorf("usage %+v", u)
|
||||
}
|
||||
if _, ok := u[""]; ok {
|
||||
t.Error("a shared model was counted against somebody")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWhatGoesPastAGuaranteeIsBorrowedNewestFirst(t *testing.T) {
|
||||
p := policy()
|
||||
b := p.Borrowed([]Holding{
|
||||
{Replica: "old", Owner: "team:ml", Amount: Amount{GPUs: 8}, Since: at(1)},
|
||||
{Replica: "new", Owner: "team:ml", Amount: Amount{GPUs: 4}, Since: at(2)},
|
||||
{Replica: "web", Owner: "team:web", Amount: Amount{GPUs: 4}, Since: at(3)},
|
||||
})
|
||||
// ml: 8 within its 8, then 4 past it. acme: 8 + 4 = 12 within, web's 4
|
||||
// takes the org to 16, past its 12.
|
||||
if b["old"] || !b["new"] || !b["web"] {
|
||||
t.Errorf("borrowed %v", b)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAGuaranteeIsOnlyForWhoHasOne(t *testing.T) {
|
||||
p := policy()
|
||||
u := map[string]Amount{"team:ml": {GPUs: 4}, "org:acme": {GPUs: 4}}
|
||||
if !p.Guaranteed("project:train", u, Amount{GPUs: 4}) {
|
||||
t.Error("8 of ml's 8 and 8 of acme's 12 is within both")
|
||||
}
|
||||
if p.Guaranteed("project:train", u, Amount{GPUs: 5}) {
|
||||
t.Error("9 GPUs is past ml's 8")
|
||||
}
|
||||
if p.Guaranteed("", u, Amount{GPUs: 1}) || New(nil, nil).Guaranteed("team:ml", u, Amount{}) {
|
||||
t.Error("an owner with no guarantee is not guaranteed anything")
|
||||
}
|
||||
}
|
||||
|
||||
func TestALimitStopsGrowthWhereverItIsSet(t *testing.T) {
|
||||
p := policy()
|
||||
u := map[string]Amount{"org:acme": {GPUs: 14}}
|
||||
if why := p.AtLimit("team:web", u, Amount{GPUs: 2}); why != "" {
|
||||
t.Errorf("16 of 16 is at the limit, not past it: %s", why)
|
||||
}
|
||||
why := p.AtLimit("team:web", u, Amount{GPUs: 4})
|
||||
if !strings.Contains(why, "org:acme is at its capacity limit (18 GPUs of 16)") {
|
||||
t.Errorf("the refusal should name the scope and the figures: %q", why)
|
||||
}
|
||||
if p.AtLimit("", u, Amount{GPUs: 99}) != "" {
|
||||
t.Error("a shared model was limited")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
package control
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// QuotaStore is where capacity quotas and the tenancy tree are kept.
|
||||
type QuotaStore interface {
|
||||
CapacityQuotas(ctx context.Context) ([]store.CapacityQuota, error)
|
||||
ScopeParents(ctx context.Context) (map[string][]string, error)
|
||||
}
|
||||
|
||||
// quotaRefresh is how stale the reconciler's view of quotas may get. A quota
|
||||
// changed by an operator takes effect within this; the reconciler asks many
|
||||
// times a pass and the store is not asked each time.
|
||||
const quotaRefresh = 30 * time.Second
|
||||
|
||||
// QuotaPolicy returns what the reconciler asks for its quotas: the last
|
||||
// policy read from the store, read again once it is older than quotaRefresh.
|
||||
// A store that fails keeps the last good policy, so a database hiccup does
|
||||
// not drop every guarantee and limit at once.
|
||||
func QuotaPolicy(st QuotaStore, log *slog.Logger) func() *quota.Policy {
|
||||
var mu sync.Mutex
|
||||
var cur *quota.Policy
|
||||
var at time.Time
|
||||
return func() *quota.Policy {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if cur != nil && time.Since(at) < quotaRefresh {
|
||||
return cur
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
qs, err := st.CapacityQuotas(ctx)
|
||||
if err == nil {
|
||||
var parents map[string][]string
|
||||
if parents, err = st.ScopeParents(ctx); err == nil {
|
||||
cur, at = quota.New(qs, parents), time.Now()
|
||||
return cur
|
||||
}
|
||||
}
|
||||
log.Warn("could not read capacity quotas; keeping the last ones", "err", err)
|
||||
at = time.Now() // not again on every call while the store is down
|
||||
return cur
|
||||
}
|
||||
}
|
||||
|
||||
// MeasurementStore is where benchmark results are kept.
|
||||
type MeasurementStore interface {
|
||||
Capacities(ctx context.Context) ([]store.StoredCapacity, error)
|
||||
}
|
||||
|
||||
// Measurements returns what the reconciler asks for benchmark results, keyed
|
||||
// model/profile, read again once older than quotaRefresh; like QuotaPolicy it
|
||||
// keeps the last good answer when the store fails.
|
||||
func Measurements(st MeasurementStore, log *slog.Logger) func() map[string]reconciler.Measured {
|
||||
var mu sync.Mutex
|
||||
var cur map[string]reconciler.Measured
|
||||
var at time.Time
|
||||
return func() map[string]reconciler.Measured {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if cur != nil && time.Since(at) < quotaRefresh {
|
||||
return cur
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
at = time.Now()
|
||||
rows, err := st.Capacities(ctx)
|
||||
if err != nil {
|
||||
log.Warn("could not read benchmark results; keeping the last ones", "err", err)
|
||||
return cur
|
||||
}
|
||||
next := map[string]reconciler.Measured{}
|
||||
for _, row := range rows {
|
||||
var c bench.Capacity
|
||||
if json.Unmarshal(row.Measurement, &c) != nil {
|
||||
continue
|
||||
}
|
||||
next[row.Model+"/"+row.Profile] = reconciler.Measured{
|
||||
MaxConcurrency: c.MaxConcurrencyAtSLO, TokPerSec: c.ThroughputTokPerSec}
|
||||
}
|
||||
cur = next
|
||||
return cur
|
||||
}
|
||||
}
|
||||
@@ -157,8 +157,31 @@ type Load struct {
|
||||
// the last one is still starting: on a small CPU fleet each replica is minutes
|
||||
// of weights and gigabytes of memory, and a spike that is over by the time it
|
||||
// loads should not leave three behind.
|
||||
//
|
||||
// With a benchmark stored for the profile a model runs on, its measured
|
||||
// capacity is used too: more requests per replica than were measured to meet
|
||||
// the SLO is busy, even with the engines' queues empty — an engine will take
|
||||
// requests into its slots long after latency has gone — and a replica is only
|
||||
// taken back when the rest would still be under scaleDownHeadroom of what they
|
||||
// were measured to hold, so scaling down does not land on the edge.
|
||||
func (r *Reconciler) RecordLoad(loads map[string]Load) {
|
||||
now := r.now()
|
||||
// The profile each model's replicas run on, for its measurement; read
|
||||
// before the loop, which holds the lock.
|
||||
running := map[string]string{}
|
||||
r.mu.Lock()
|
||||
for _, rep := range r.replicas {
|
||||
if rep.State.Counts() {
|
||||
running[rep.Model] = rep.Profile
|
||||
}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
measured := map[string]Measured{}
|
||||
for model, profile := range running {
|
||||
if m, ok := r.measured(model, profile); ok {
|
||||
measured[model] = m
|
||||
}
|
||||
}
|
||||
for _, m := range r.cat.Load().Models() {
|
||||
base := r.baseFor(m)
|
||||
r.mu.Lock()
|
||||
@@ -178,8 +201,13 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
|
||||
l := loads[m.Name]
|
||||
perReplica := float64(l.Waiting) / float64(max(l.Replicas, 1))
|
||||
extra := r.extra[m.Name]
|
||||
meas, hasMeas := measured[m.Name]
|
||||
overMeasured := hasMeas &&
|
||||
float64(l.Running+l.Waiting) > float64(meas.MaxConcurrency*max(l.Replicas, 1))
|
||||
fitsInFewer := !hasMeas ||
|
||||
float64(l.Running) <= scaleDownHeadroom*float64(meas.MaxConcurrency*max(l.Replicas-1, 0))
|
||||
switch {
|
||||
case perReplica >= r.cfg.ScaleUpWaiting:
|
||||
case perReplica >= r.cfg.ScaleUpWaiting || overMeasured:
|
||||
delete(r.quietSince, m.Name)
|
||||
since, ok := r.busySince[m.Name]
|
||||
if !ok {
|
||||
@@ -190,9 +218,10 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
|
||||
r.extra[m.Name] = extra + 1
|
||||
r.busySince[m.Name] = now
|
||||
r.log.Info("scaling up for load", "model", m.Name, "replicas", base+extra+1,
|
||||
"waiting_per_replica", perReplica, "for", now.Sub(since).Round(time.Second))
|
||||
"waiting_per_replica", perReplica, "in_flight", l.Running+l.Waiting,
|
||||
"measured_per_replica", meas.MaxConcurrency, "for", now.Sub(since).Round(time.Second))
|
||||
}
|
||||
case l.Waiting == 0 && extra > 0:
|
||||
case l.Waiting == 0 && extra > 0 && fitsInFewer:
|
||||
delete(r.busySince, m.Name)
|
||||
since, ok := r.quietSince[m.Name]
|
||||
if !ok {
|
||||
@@ -212,3 +241,8 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
|
||||
r.mu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
// scaleDownHeadroom is how full, of their measured capacity, the replicas left
|
||||
// after a scale-down may be. Below one so that the load that was there a
|
||||
// minute ago does not put them straight back over.
|
||||
const scaleDownHeadroom = 0.7
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
)
|
||||
|
||||
// Headroom is how many more replicas of a model the cluster has room for now.
|
||||
@@ -24,6 +25,9 @@ type Headroom struct {
|
||||
// Blocked is why one more than More does not fit, a line per node that
|
||||
// was considered. Empty when Limited.
|
||||
Blocked []string
|
||||
// TokPerSec is what one more replica was measured to serve, on the
|
||||
// profile placement would give it; zero when it was never measured.
|
||||
TokPerSec float64
|
||||
}
|
||||
|
||||
// Headroom tries placing up to limit more replicas of a model on a copy of
|
||||
@@ -52,11 +56,16 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
|
||||
|
||||
inv := r.inv.Clone()
|
||||
sched := placement.New(inv)
|
||||
pol := r.policy()
|
||||
var usage map[string]quota.Amount
|
||||
if pol != nil {
|
||||
usage = pol.Usage(r.Holdings())
|
||||
}
|
||||
for h.More < limit {
|
||||
res := sched.Place(m.PlacementProfiles(), avoid)
|
||||
res := sched.Place(r.orderedProfiles(m), avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
// As scaleUp does: spreading is a preference, not a rule.
|
||||
res = sched.Place(m.PlacementProfiles(), nil)
|
||||
res = sched.Place(r.orderedProfiles(m), nil)
|
||||
}
|
||||
if res.Placement == nil {
|
||||
for _, rej := range res.Rejections {
|
||||
@@ -66,10 +75,26 @@ func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
|
||||
h.Blocked = slices.Compact(h.Blocked)
|
||||
return h, true
|
||||
}
|
||||
// A quota limit is as much an edge as a full node.
|
||||
if pol != nil && m.Owner != "" {
|
||||
need := placementNeed(m, res.Placement)
|
||||
if why := pol.AtLimit(m.Owner, usage, need); why != "" {
|
||||
h.Blocked = []string{why}
|
||||
return h, true
|
||||
}
|
||||
for _, s := range pol.Chain(m.Owner) {
|
||||
usage[s] = usage[s].Plus(need)
|
||||
}
|
||||
}
|
||||
mem, cores := 0, 0
|
||||
if prof, ok := m.ProfileByName(res.Placement.Profile.Name); ok {
|
||||
mem, cores = prof.MinMemoryGB, cpuCores(prof)
|
||||
}
|
||||
if h.More == 0 {
|
||||
if meas, ok := r.measured(m.Name, res.Placement.Profile.Name); ok {
|
||||
h.TokPerSec = meas.TokPerSec
|
||||
}
|
||||
}
|
||||
var nodes []string
|
||||
for _, a := range res.Placement.Nodes {
|
||||
inv.Allocate(a.NodeID, inventory.Allocation{
|
||||
|
||||
@@ -83,6 +83,12 @@ type ModelHealth struct {
|
||||
// place one found nowhere; Placement says why.
|
||||
Unplaceable bool
|
||||
Placement string
|
||||
// Class is the capacity class it is treated as. Preempted counts the
|
||||
// replicas it has lost to models of a higher class since the control
|
||||
// plane started; HeldFor names the model it is giving room to now.
|
||||
Class string
|
||||
Preempted int
|
||||
HeldFor string
|
||||
}
|
||||
|
||||
// ModelHealth reports every model in the catalog.
|
||||
@@ -106,7 +112,10 @@ func (r *Reconciler) ModelHealth() []ModelHealth {
|
||||
out := make([]ModelHealth, 0, len(models))
|
||||
for _, m := range models {
|
||||
h := ModelHealth{Model: m.Name, Desired: desired[m.Name], Ready: ready[m.Name],
|
||||
Failures: r.attempts[m.Name]}
|
||||
Failures: r.attempts[m.Name], Class: m.EffectiveClass(), Preempted: r.preempted[m.Name]}
|
||||
if hd, ok := r.held[m.Name]; ok && r.now().Before(hd.Until) {
|
||||
h.HeldFor = hd.For
|
||||
}
|
||||
if p, ok := r.parked[m.Name]; ok {
|
||||
h.Parked, h.ParkedSince, h.Reason = true, p.Since, p.Reason
|
||||
}
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"sort"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
)
|
||||
|
||||
// Measured is what one replica of a model on a profile was measured to serve
|
||||
// while meeting its SLO, by `llmgw ctl bench --save`.
|
||||
type Measured struct {
|
||||
// MaxConcurrency is the most requests at once that still met the SLO.
|
||||
MaxConcurrency int
|
||||
// TokPerSec is the throughput at that concurrency.
|
||||
TokPerSec float64
|
||||
}
|
||||
|
||||
// SetMeasurements supplies benchmark results, keyed model/profile, read
|
||||
// afresh each time it is called. Without them scaling goes by queue alone and
|
||||
// profiles are chosen by priority alone.
|
||||
func (r *Reconciler) SetMeasurements(f func() map[string]Measured) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.measurements = f
|
||||
}
|
||||
|
||||
// measured is a model's measurement on one profile.
|
||||
func (r *Reconciler) measured(model, profile string) (Measured, bool) {
|
||||
r.mu.Lock()
|
||||
f := r.measurements
|
||||
r.mu.Unlock()
|
||||
if f == nil {
|
||||
return Measured{}, false
|
||||
}
|
||||
m, ok := f()[model+"/"+profile]
|
||||
return m, ok && m.MaxConcurrency > 0
|
||||
}
|
||||
|
||||
// orderedProfiles is a model's profiles as placement should try them: by
|
||||
// priority, and among equal priorities the one measured to serve the most per
|
||||
// GPU — or per core, for a profile with none — first. Unmeasured profiles keep
|
||||
// their catalog order behind measured ones of the same priority.
|
||||
//
|
||||
// Placement sorts by priority itself, so the order found here is written back
|
||||
// as priorities, descending, for it to keep.
|
||||
func (r *Reconciler) orderedProfiles(m catalog.Model) []placement.Profile {
|
||||
ps := m.PlacementProfiles()
|
||||
eff := make([]float64, len(ps))
|
||||
for i, p := range ps {
|
||||
meas, ok := r.measured(m.Name, p.Name)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
units := float64(max(p.GPUsPerNode, 0) * max(p.Nodes, 1))
|
||||
if units == 0 {
|
||||
units = float64(max(p.CPUCores, 1))
|
||||
}
|
||||
eff[i] = meas.TokPerSec / units
|
||||
}
|
||||
idx := make([]int, len(ps))
|
||||
for i := range idx {
|
||||
idx[i] = i
|
||||
}
|
||||
sort.SliceStable(idx, func(a, b int) bool {
|
||||
pa, pb := ps[idx[a]], ps[idx[b]]
|
||||
if pa.Priority != pb.Priority {
|
||||
return pa.Priority > pb.Priority
|
||||
}
|
||||
return eff[idx[a]] > eff[idx[b]]
|
||||
})
|
||||
out := make([]placement.Profile, len(ps))
|
||||
for rank, i := range idx {
|
||||
out[rank] = ps[i]
|
||||
out[rank].Priority = len(ps) - rank
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// MeasuredFor is the measurement that sizes a model: that of the profile its
|
||||
// replicas run on, or, with none running, of the first profile placement
|
||||
// would try that was measured.
|
||||
func (r *Reconciler) MeasuredFor(model string) (Measured, string, bool) {
|
||||
r.mu.Lock()
|
||||
profile := ""
|
||||
for _, rep := range r.replicas {
|
||||
if rep.Model == model && rep.State.Counts() {
|
||||
profile = rep.Profile
|
||||
break
|
||||
}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
if profile != "" {
|
||||
if m, ok := r.measured(model, profile); ok {
|
||||
return m, profile, true
|
||||
}
|
||||
}
|
||||
cm, ok := r.cat.Load().Resolve(model)
|
||||
if !ok {
|
||||
return Measured{}, "", false
|
||||
}
|
||||
for _, p := range r.orderedProfiles(cm) {
|
||||
if m, ok := r.measured(cm.Name, p.Name); ok {
|
||||
return m, p.Name, true
|
||||
}
|
||||
}
|
||||
return Measured{}, "", false
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
func measure(h *harness, byKey map[string]Measured) {
|
||||
h.r.SetMeasurements(func() map[string]Measured { return byKey })
|
||||
}
|
||||
|
||||
func TestMoreRequestsThanWereMeasuredToMeetTheSLOIsBusy(t *testing.T) {
|
||||
m := testModel(1)
|
||||
m.Scaling.Max = 2
|
||||
h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8))
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
// Six in flight, nothing waiting at the engine: by the queue it is fine.
|
||||
load := map[string]Load{"llama": {Replicas: 1, Running: 6}}
|
||||
hold := func() {
|
||||
h.r.RecordLoad(load)
|
||||
h.advance(61 * time.Second)
|
||||
h.r.RecordLoad(load)
|
||||
h.r.Reconcile(t.Context())
|
||||
}
|
||||
hold()
|
||||
if n := liveCount(h); n != 1 {
|
||||
t.Fatalf("scaled up with no measurement and an empty queue: %d", n)
|
||||
}
|
||||
// Measured to meet its SLO at four at once: six is past it.
|
||||
measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 4, TokPerSec: 200}})
|
||||
hold()
|
||||
if n := liveCount(h); n != 2 {
|
||||
t.Errorf("%d replicas with 6 in flight against a measured 4", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReplicaIsTakenBackOnlyIfTheRestWouldHaveRoomToSpare(t *testing.T) {
|
||||
m := testModel(1)
|
||||
m.Scaling.Max = 2
|
||||
h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8))
|
||||
measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 4, TokPerSec: 200}})
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
h.r.mu.Lock()
|
||||
h.r.extra["llama"] = 1
|
||||
h.r.mu.Unlock()
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
if liveCount(h) != 2 {
|
||||
t.Fatal("setup: not at two replicas")
|
||||
}
|
||||
quietFor := func(l Load) {
|
||||
for range 3 {
|
||||
h.r.RecordLoad(map[string]Load{"llama": l})
|
||||
h.advance(DefaultConfig().ScaleDownAfter + time.Second)
|
||||
h.r.RecordLoad(map[string]Load{"llama": l})
|
||||
h.r.Reconcile(t.Context())
|
||||
}
|
||||
}
|
||||
// Nothing waiting, but three running: one replica left would be at 3 of
|
||||
// its 4, past 70%.
|
||||
quietFor(Load{Replicas: 2, Running: 3})
|
||||
if n := liveCount(h); n != 2 {
|
||||
t.Errorf("scaled down to the edge of what one replica holds: %d", n)
|
||||
}
|
||||
quietFor(Load{Replicas: 2, Running: 2})
|
||||
if n := liveCount(h); n != 1 {
|
||||
t.Errorf("two running fits one replica's 70%% of 4; still %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheProfileMeasuredToServeMostPerGPUIsTriedFirst(t *testing.T) {
|
||||
m := catalog.Model{Name: "llama", Source: "/models/llama", Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{
|
||||
{Name: "tp4", GPUClass: "h100-80g", TP: 4},
|
||||
{Name: "tp2", GPUClass: "h100-80g", TP: 2},
|
||||
{Name: "tp8", GPUClass: "h100-80g", TP: 8, Priority: 10},
|
||||
}}
|
||||
h := newHarness(t, m, gpuNode("1", 8))
|
||||
// tp4: 400/4 = 100 per GPU; tp2: 300/2 = 150. tp8 is unmeasured but has
|
||||
// the higher priority, which an operator set and a benchmark does not
|
||||
// overrule.
|
||||
measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 8, TokPerSec: 400},
|
||||
"llama/tp2": {MaxConcurrency: 6, TokPerSec: 300}})
|
||||
cm, _ := h.r.cat.Load().Resolve("llama")
|
||||
var order []string
|
||||
for _, p := range h.r.orderedProfiles(cm) {
|
||||
order = append(order, p.Name)
|
||||
}
|
||||
if len(order) != 3 || order[0] != "tp8" || order[1] != "tp2" || order[2] != "tp4" {
|
||||
t.Errorf("order %v, expected tp8 (priority), then tp2 (150/GPU), then tp4 (100/GPU)", order)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,324 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
)
|
||||
|
||||
// hold keeps a model that gave up room from taking it back.
|
||||
type hold struct {
|
||||
Until time.Time
|
||||
For string
|
||||
}
|
||||
|
||||
// maxVictims bounds how many replicas one placement may stop. Past this the
|
||||
// model is better reported as not fitting than paid for with half a cluster.
|
||||
const maxVictims = 3
|
||||
|
||||
// victim is a replica that could be stopped for another model.
|
||||
type victim struct {
|
||||
ID string
|
||||
Model string
|
||||
Class string
|
||||
AboveMin bool
|
||||
Tier int
|
||||
Idle time.Duration
|
||||
// Size is what stopping it frees, for choosing the smaller of two
|
||||
// otherwise equal victims.
|
||||
Size quota.Amount
|
||||
}
|
||||
|
||||
// preemptFor stops replicas of lower classes when a model cannot otherwise be
|
||||
// placed, and says whether it did.
|
||||
//
|
||||
// What may be taken from whom:
|
||||
//
|
||||
// - A lower class gives way to a higher one: opportunistic to standard,
|
||||
// standard to critical.
|
||||
// - Within a class, only a replica above its model's minimum, and only for
|
||||
// a model still below its own. Room a model grew into under load is not a
|
||||
// promise; its minimum is.
|
||||
// - A critical model's minimum is never taken.
|
||||
//
|
||||
// Victims only ever come from a lower rank, or from above a minimum for a
|
||||
// model below one, so two models cannot take turns stopping each other.
|
||||
//
|
||||
// The set is tried on a copy of the inventory first, and only stopped if it
|
||||
// would let the model be placed: stopping replicas that do not make room is
|
||||
// the one outcome worse than not preempting.
|
||||
func (r *Reconciler) preemptFor(ctx context.Context, m catalog.Model, avoid []string) bool {
|
||||
cands := r.victims(m)
|
||||
if len(cands) == 0 {
|
||||
return false
|
||||
}
|
||||
// Candidates in order until the model fits; then every one whose
|
||||
// stopping turns out not to be needed is kept after all. Without that
|
||||
// second step an idle replica on the wrong node, tried first, would be
|
||||
// stopped along with the one that actually made the room.
|
||||
var chosen []victim
|
||||
for _, v := range cands {
|
||||
if len(chosen) == maxVictims {
|
||||
break
|
||||
}
|
||||
chosen = append(chosen, v)
|
||||
if !r.fitsWithout(m, avoid, chosen) {
|
||||
continue
|
||||
}
|
||||
for i := len(chosen) - 1; i >= 0 && len(chosen) > 1; i-- {
|
||||
fewer := append(append([]victim(nil), chosen[:i]...), chosen[i+1:]...)
|
||||
if r.fitsWithout(m, avoid, fewer) {
|
||||
chosen = fewer
|
||||
}
|
||||
}
|
||||
r.preempt(ctx, m, chosen)
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// fitsWithout reports whether m could be placed with these replicas gone,
|
||||
// tried on a copy of the inventory.
|
||||
func (r *Reconciler) fitsWithout(m catalog.Model, avoid []string, gone []victim) bool {
|
||||
trial := r.inv.Clone()
|
||||
for _, v := range gone {
|
||||
trial.ReleaseReplica(v.ID)
|
||||
}
|
||||
sched := placement.New(trial)
|
||||
res := sched.Place(r.orderedProfiles(m), avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
res = sched.Place(r.orderedProfiles(m), nil)
|
||||
}
|
||||
return res.Placement != nil
|
||||
}
|
||||
|
||||
// preempt drains the chosen replicas and holds their models back.
|
||||
func (r *Reconciler) preempt(ctx context.Context, m catalog.Model, chosen []victim) {
|
||||
now := r.now()
|
||||
var ids []string
|
||||
r.mu.Lock()
|
||||
for _, v := range chosen {
|
||||
if rep, ok := r.replicas[v.ID]; ok {
|
||||
rep.PreemptedFor = m.Name
|
||||
}
|
||||
ids = append(ids, v.ID)
|
||||
r.held[v.Model] = hold{Until: now.Add(r.cfg.PreemptHold), For: m.Name}
|
||||
r.preempted[v.Model]++
|
||||
}
|
||||
r.preempting[m.Name] = ids
|
||||
r.mu.Unlock()
|
||||
|
||||
for _, v := range chosen {
|
||||
r.log.Warn("preempting a replica to make room",
|
||||
"replica", v.ID, "model", v.Model, "class", v.Class, "above_min", v.AboveMin,
|
||||
"for", m.Name, "for_class", m.EffectiveClass())
|
||||
}
|
||||
r.scaleDown(ctx, ids)
|
||||
}
|
||||
|
||||
// awaitingPreemption reports whether replicas stopped for a model are still
|
||||
// on their way out, so it waits for them rather than stopping more.
|
||||
func (r *Reconciler) awaitingPreemption(model string) bool {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
ids := r.preempting[model]
|
||||
ids = slices.DeleteFunc(slices.Clone(ids), func(id string) bool {
|
||||
_, ok := r.replicas[id]
|
||||
return !ok
|
||||
})
|
||||
if len(ids) == 0 {
|
||||
delete(r.preempting, model)
|
||||
return false
|
||||
}
|
||||
r.preempting[model] = ids
|
||||
return true
|
||||
}
|
||||
|
||||
// isHeld reports whether a model gave up room and may not take it back yet.
|
||||
// The hold ends early once the model it was given to has what it asked for.
|
||||
//
|
||||
// Today a preempted replica is only removed by its own model's drain loop,
|
||||
// which runs after that model has decided not to scale up, so the room it
|
||||
// frees reaches the model it was freed for first anyway. The hold is what
|
||||
// keeps that true for any other way a replica can go — a stop reported
|
||||
// between passes, a reap at the start of one — rather than it resting on the
|
||||
// order of lines in reconcileModel.
|
||||
func (r *Reconciler) isHeld(model string) bool {
|
||||
r.mu.Lock()
|
||||
h, ok := r.held[model]
|
||||
r.mu.Unlock()
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
if r.now().After(h.Until) {
|
||||
r.mu.Lock()
|
||||
delete(r.held, model)
|
||||
r.mu.Unlock()
|
||||
return false
|
||||
}
|
||||
if to, ok := r.cat.Load().Resolve(h.For); ok && r.liveCount(h.For) >= r.desiredFor(to) {
|
||||
r.mu.Lock()
|
||||
delete(r.held, model)
|
||||
r.mu.Unlock()
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// liveCount is how many replicas of a model hold capacity.
|
||||
func (r *Reconciler) liveCount(model string) int {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
n := 0
|
||||
for _, rep := range r.replicas {
|
||||
if rep.Model == model && rep.State.Counts() && rep.RetiringAt.IsZero() {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// victims lists the replicas that may be stopped for m, in the order they
|
||||
// should be: idle on-demand ones first, then what other tenants borrowed past
|
||||
// their guarantees when m's owner is within its own, then what lower classes
|
||||
// grew into under load, then the same within m's class, then lower classes'
|
||||
// minimums; within each, the longest unused first.
|
||||
//
|
||||
// A replica within its owner's guarantee is taken only for a model of the
|
||||
// same owner: a guarantee that another tenant's higher class could take is
|
||||
// not one.
|
||||
func (r *Reconciler) victims(m catalog.Model) []victim {
|
||||
cat := r.cat.Load()
|
||||
now := r.now()
|
||||
myRank := catalog.ClassRank(m.EffectiveClass())
|
||||
belowMin := r.liveCount(m.Name) < m.Scaling.Min
|
||||
|
||||
pol := r.policy()
|
||||
var borrowed map[string]bool
|
||||
reclaim := false
|
||||
if pol != nil {
|
||||
hs := r.Holdings()
|
||||
borrowed = pol.Borrowed(hs)
|
||||
reclaim = pol.Guaranteed(m.Owner, pol.Usage(hs), profileNeed(m))
|
||||
}
|
||||
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
|
||||
// Per model, its live replicas newest first: the newest are the ones
|
||||
// above its minimum.
|
||||
byModel := map[string][]*Replica{}
|
||||
for _, rep := range r.replicas {
|
||||
if rep.Model == m.Name || !rep.State.Counts() || !rep.RetiringAt.IsZero() || rep.PreemptedFor != "" {
|
||||
continue
|
||||
}
|
||||
byModel[rep.Model] = append(byModel[rep.Model], rep)
|
||||
}
|
||||
|
||||
var out []victim
|
||||
for model, reps := range byModel {
|
||||
other, ok := cat.Resolve(model)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
class := other.EffectiveClass()
|
||||
rank := catalog.ClassRank(class)
|
||||
sort.Slice(reps, func(i, j int) bool { return reps[i].CreatedAt.After(reps[j].CreatedAt) })
|
||||
above := max(len(reps)-other.Scaling.Min, 0)
|
||||
for i, rep := range reps {
|
||||
aboveMin := i < above
|
||||
tier := -1
|
||||
switch {
|
||||
case rank < myRank && other.Scaling.OnDemand() && idleFor(rep, now) >= r.cfg.EvictionGrace:
|
||||
tier = 0
|
||||
case rank < myRank && aboveMin:
|
||||
tier = 2
|
||||
case rank == myRank && aboveMin && belowMin:
|
||||
tier = 3
|
||||
case rank < myRank && class != catalog.ClassCritical:
|
||||
tier = 4
|
||||
}
|
||||
if pol != nil && other.Owner != m.Owner {
|
||||
switch {
|
||||
case borrowed[rep.ID] && reclaim && (class != catalog.ClassCritical || aboveMin):
|
||||
// Past its owner's guarantee, wanted by an owner within
|
||||
// its own: taken back whatever the classes.
|
||||
if tier != 0 {
|
||||
tier = 1
|
||||
}
|
||||
case !borrowed[rep.ID] && guaranteed(pol, other.Owner):
|
||||
continue // held within its owner's guarantee
|
||||
}
|
||||
}
|
||||
if tier < 0 {
|
||||
continue
|
||||
}
|
||||
// Just started: stopping it throws away a model load that has
|
||||
// only now been paid for.
|
||||
if !rep.ReadyAt.IsZero() && now.Sub(rep.ReadyAt) < r.cfg.MinLifetime {
|
||||
continue
|
||||
}
|
||||
if len(nodesOf(rep)) > 1 {
|
||||
continue // see makeRoomFor: a multi-node replica is its own decision
|
||||
}
|
||||
prof, _ := other.ProfileByName(rep.Profile)
|
||||
var size quota.Amount
|
||||
for _, sh := range rep.Shards {
|
||||
size = size.Plus(quota.Amount{GPUs: len(sh.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
|
||||
}
|
||||
out = append(out, victim{ID: rep.ID, Model: model, Class: class,
|
||||
AboveMin: aboveMin, Tier: tier, Idle: idleFor(rep, now), Size: size})
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool {
|
||||
a, b := out[i], out[j]
|
||||
if a.Tier != b.Tier {
|
||||
return a.Tier < b.Tier
|
||||
}
|
||||
if a.Idle != b.Idle {
|
||||
return a.Idle > b.Idle
|
||||
}
|
||||
// The smaller first: the least served is lost if it is enough, and
|
||||
// the pruning after keeps a larger one only when it is needed.
|
||||
if a.Size.GPUs != b.Size.GPUs {
|
||||
return a.Size.GPUs < b.Size.GPUs
|
||||
}
|
||||
if a.Size.MemoryGB != b.Size.MemoryGB {
|
||||
return a.Size.MemoryGB < b.Size.MemoryGB
|
||||
}
|
||||
return a.ID < b.ID
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// idleFor is how long a replica has gone unused; zero when nobody has said,
|
||||
// which is not the same as idle.
|
||||
func idleFor(rep *Replica, now time.Time) time.Duration {
|
||||
if rep.LastUsedAt.IsZero() {
|
||||
return 0
|
||||
}
|
||||
return now.Sub(rep.LastUsedAt)
|
||||
}
|
||||
|
||||
func nodesOf(rep *Replica) map[string]struct{} {
|
||||
out := map[string]struct{}{}
|
||||
for _, sh := range rep.Shards {
|
||||
out[sh.NodeID] = struct{}{}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// guaranteed reports whether an owner has a guarantee anywhere up its tree.
|
||||
func guaranteed(pol *quota.Policy, owner string) bool {
|
||||
for _, q := range pol.Governing(owner) {
|
||||
if q.Guarantee.Any() {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,314 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
// newFleetHarness is newHarness with a catalog of several models.
|
||||
func newFleetHarness(t *testing.T, models []catalog.Model, nodes ...inventory.Node) *harness {
|
||||
t.Helper()
|
||||
cat, err := catalog.New(models)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inv := inventory.NewStore()
|
||||
c := newClock()
|
||||
for _, n := range nodes {
|
||||
inv.Upsert(n)
|
||||
}
|
||||
disp := &fakeDispatcher{}
|
||||
cfg := DefaultConfig()
|
||||
cfg.UnreachableAfter = 15 * time.Second
|
||||
cfg.LostAfter = 60 * time.Second
|
||||
r := New(cfg, catalog.NewLive(cat), inv, disp, slog.New(slog.DiscardHandler))
|
||||
r.SetClock(c.now)
|
||||
c.advance(cfg.SettleFor)
|
||||
for _, n := range nodes {
|
||||
touchAt(inv, n.ID, c.now())
|
||||
}
|
||||
return &harness{r: r, inv: inv, disp: disp, clock: c}
|
||||
}
|
||||
|
||||
// gpuModel runs on one whole 8-GPU node.
|
||||
func gpuModel(name, class string, min int) catalog.Model {
|
||||
return catalog.Model{Name: name, Source: "/models/" + name, Class: class,
|
||||
Scaling: catalog.Scaling{Min: min, Max: min},
|
||||
Profiles: []catalog.Profile{{Name: "tp8", Engine: "vllm", GPUClass: "h100-80g", TP: 8}}}
|
||||
}
|
||||
|
||||
// settle runs passes until nothing changes, marking what starts ready and
|
||||
// letting preempted replicas run out their grace.
|
||||
func (h *harness) settle(t *testing.T, passes int) {
|
||||
t.Helper()
|
||||
for range passes {
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
h.advance(h.r.cfg.PreemptGrace + time.Second)
|
||||
}
|
||||
}
|
||||
|
||||
func replicasOf(h *harness, model string) (live int) {
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.Model == model && rep.State.Counts() {
|
||||
live++
|
||||
}
|
||||
}
|
||||
return live
|
||||
}
|
||||
|
||||
// fill places the standard model, then ages it past MinLifetime so it may be
|
||||
// taken from, and only then adds the model that wants its node.
|
||||
func fillThenWant(t *testing.T, want catalog.Model, have catalog.Model, nodes ...inventory.Node) *harness {
|
||||
t.Helper()
|
||||
h := newFleetHarness(t, []catalog.Model{have, withMin(want, 0)}, nodes...)
|
||||
h.settle(t, 3)
|
||||
if got := replicasOf(h, have.Name); got != have.Scaling.Min {
|
||||
t.Fatalf("%s has %d replicas before anything wanted its room", have.Name, got)
|
||||
}
|
||||
h.advance(h.r.cfg.MinLifetime + time.Minute)
|
||||
cat, err := catalog.New([]catalog.Model{have, want})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.r.cat.Store(cat)
|
||||
return h
|
||||
}
|
||||
|
||||
func withMin(m catalog.Model, n int) catalog.Model {
|
||||
m.Scaling = catalog.Scaling{Min: n, Max: n}
|
||||
return m
|
||||
}
|
||||
|
||||
func TestACriticalModelTakesAStandardModelsNode(t *testing.T) {
|
||||
h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
|
||||
|
||||
h.r.Reconcile(t.Context())
|
||||
var draining int
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.Model == "chat" && rep.State == Draining && rep.PreemptedFor == "vital" {
|
||||
draining++
|
||||
}
|
||||
}
|
||||
if draining != 1 {
|
||||
t.Fatalf("the standard replica was not drained for the critical one: %+v", h.r.Replicas())
|
||||
}
|
||||
if hs := health(h, "chat"); hs.HeldFor != "vital" || hs.Preempted != 1 {
|
||||
t.Errorf("the victim's health does not say it gave way: %+v", hs)
|
||||
}
|
||||
|
||||
h.settle(t, 4)
|
||||
if replicasOf(h, "vital") != 1 {
|
||||
t.Fatalf("the critical model was not placed once room was made: %+v", h.r.Replicas())
|
||||
}
|
||||
// Held back until the critical model had its replica, then short again
|
||||
// because there is no room: it must not have taken the node back — not
|
||||
// even for a moment, which would load a model only to preempt it again.
|
||||
if replicasOf(h, "chat") != 0 {
|
||||
t.Error("the standard model took the node back")
|
||||
}
|
||||
started := 0
|
||||
for _, cmd := range h.disp.startedShards() {
|
||||
if strings.HasPrefix(cmd.ReplicaID, "chat-") {
|
||||
started++
|
||||
}
|
||||
}
|
||||
if started != 1 {
|
||||
t.Errorf("the standard model started %d replicas; it should not have tried again", started)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStandardModelDoesNotTakeAnothersMinimum(t *testing.T) {
|
||||
h := fillThenWant(t, gpuModel("second", "", 1), gpuModel("first", "", 1), gpuNode("1", 8))
|
||||
h.settle(t, 3)
|
||||
if replicasOf(h, "first") != 1 || replicasOf(h, "second") != 0 {
|
||||
t.Errorf("a model of the same class took another's minimum: %+v", h.r.Replicas())
|
||||
}
|
||||
if hs := health(h, "second"); !hs.Unplaceable {
|
||||
t.Error("the model left without room should say it cannot be placed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithinAClassOnlyWhatWasGrownIntoIsTaken(t *testing.T) {
|
||||
grown := gpuModel("busy", "", 1)
|
||||
grown.Scaling = catalog.Scaling{Min: 1, Max: 2}
|
||||
h := newFleetHarness(t, []catalog.Model{grown, withMin(gpuModel("new", "", 1), 0)}, gpuNode("1", 8), gpuNode("2", 8))
|
||||
h.settle(t, 2)
|
||||
// Load grows it to its max: the second replica is above its minimum.
|
||||
h.r.mu.Lock()
|
||||
h.r.extra["busy"] = 1
|
||||
h.r.mu.Unlock()
|
||||
h.settle(t, 3)
|
||||
if replicasOf(h, "busy") != 2 {
|
||||
t.Fatalf("setup: busy did not grow to 2: %+v", h.r.Replicas())
|
||||
}
|
||||
h.advance(h.r.cfg.MinLifetime + time.Minute)
|
||||
cat, _ := catalog.New([]catalog.Model{grown, gpuModel("new", "", 1)})
|
||||
h.r.cat.Store(cat)
|
||||
|
||||
h.settle(t, 5)
|
||||
if replicasOf(h, "new") != 1 || replicasOf(h, "busy") != 1 {
|
||||
t.Errorf("the grown replica should give way to a minimum: busy %d, new %d",
|
||||
replicasOf(h, "busy"), replicasOf(h, "new"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestNothingIsStoppedThatWouldNotMakeRoom(t *testing.T) {
|
||||
// Two nodes, each half used by an opportunistic model; the critical one
|
||||
// needs a whole node. Stopping one frees half a node, which is no use —
|
||||
// and both together are on two nodes, which is no use either.
|
||||
half := catalog.Model{Name: "half", Source: "/m", Class: catalog.ClassOpportunistic,
|
||||
Scaling: catalog.Scaling{Min: 2, Max: 2},
|
||||
Profiles: []catalog.Profile{{Name: "tp4", Engine: "vllm", GPUClass: "h100-80g", TP: 4}}}
|
||||
five := func(id string) inventory.Node { return gpuNode(id, 5) }
|
||||
h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), half, five("1"), five("2"))
|
||||
h.r.Reconcile(t.Context())
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.PreemptedFor != "" || rep.State == Draining {
|
||||
t.Fatalf("a replica was stopped without making room: %+v", rep)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestACriticalMinimumIsNeverTaken(t *testing.T) {
|
||||
h := fillThenWant(t, gpuModel("also-vital", catalog.ClassCritical, 1), gpuModel("vital", catalog.ClassCritical, 1), gpuNode("1", 8))
|
||||
h.settle(t, 3)
|
||||
if replicasOf(h, "vital") != 1 {
|
||||
t.Error("a critical model's minimum was taken")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReplicaThatJustStartedIsLeftAlone(t *testing.T) {
|
||||
h := newFleetHarness(t, []catalog.Model{gpuModel("chat", "", 1), withMin(gpuModel("vital", catalog.ClassCritical, 1), 0)}, gpuNode("1", 8))
|
||||
h.settle(t, 2)
|
||||
cat, _ := catalog.New([]catalog.Model{gpuModel("chat", "", 1), gpuModel("vital", catalog.ClassCritical, 1)})
|
||||
h.r.cat.Store(cat)
|
||||
h.r.Reconcile(t.Context())
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.State == Draining {
|
||||
t.Fatal("a replica that had only just loaded was preempted")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func health(h *harness, model string) ModelHealth {
|
||||
for _, x := range h.r.ModelHealth() {
|
||||
if x.Model == model {
|
||||
return x
|
||||
}
|
||||
}
|
||||
return ModelHealth{}
|
||||
}
|
||||
|
||||
func TestAHoldLastsUntilTheRoomIsUsedOrItExpires(t *testing.T) {
|
||||
h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
|
||||
h.r.Reconcile(t.Context())
|
||||
if !h.r.isHeld("chat") {
|
||||
t.Fatal("the model that gave way is not held")
|
||||
}
|
||||
if h.r.isHeld("vital") {
|
||||
t.Error("the model room was made for is held")
|
||||
}
|
||||
// Released as soon as the model it was given to has what it asked for.
|
||||
h.settle(t, 3)
|
||||
if replicasOf(h, "vital") != 1 {
|
||||
t.Fatalf("setup: vital was not placed: %+v", h.r.Replicas())
|
||||
}
|
||||
if h.r.isHeld("chat") {
|
||||
t.Error("the hold outlasted its purpose")
|
||||
}
|
||||
|
||||
// And, whatever happens, after PreemptHold.
|
||||
h.r.mu.Lock()
|
||||
h.r.held["other"] = hold{Until: h.r.now().Add(time.Minute), For: "never-placed"}
|
||||
h.r.mu.Unlock()
|
||||
if !h.r.isHeld("other") {
|
||||
t.Fatal("a fresh hold is not in force")
|
||||
}
|
||||
h.advance(2 * time.Minute)
|
||||
if h.r.isHeld("other") {
|
||||
t.Error("a hold outlived its time")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyTheReplicasThatMakeTheRoomAreStopped(t *testing.T) {
|
||||
// Two half-used nodes. "idle" on node 1 is on demand and long unused, so
|
||||
// it is first in line; but the critical model needs node 2's other half,
|
||||
// which only stopping "chat" frees.
|
||||
idle := catalog.Model{Name: "idle", Source: "/m", Scaling: catalog.Scaling{Min: 0, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "tp4", Engine: "vllm", GPUClass: "h100-80g", TP: 4}}}
|
||||
chat := catalog.Model{Name: "chat", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "tp6", Engine: "vllm", GPUClass: "h100-80g", TP: 6}}}
|
||||
vital := catalog.Model{Name: "vital", Source: "/m", Class: catalog.ClassCritical,
|
||||
Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "tp8", Engine: "vllm", GPUClass: "h100-80g", TP: 8}}}
|
||||
h := newFleetHarness(t, []catalog.Model{idle, chat}, gpuNode("1", 6), gpuNode("2", 8))
|
||||
h.r.Want("idle")
|
||||
h.settle(t, 4)
|
||||
var idleNode, chatNode string
|
||||
for _, rep := range h.r.Replicas() {
|
||||
switch rep.Model {
|
||||
case "idle":
|
||||
idleNode = rep.Shards[0].NodeID
|
||||
case "chat":
|
||||
chatNode = rep.Shards[0].NodeID
|
||||
}
|
||||
}
|
||||
if idleNode == "" || chatNode != "2" {
|
||||
t.Fatalf("setup: idle on %q, chat on %q; chat must hold node 2: %+v", idleNode, chatNode, h.r.Replicas())
|
||||
}
|
||||
h.r.mu.Lock()
|
||||
for _, rep := range h.r.replicas {
|
||||
rep.LastUsedAt = h.r.now().Add(-time.Hour)
|
||||
}
|
||||
h.r.mu.Unlock()
|
||||
h.advance(h.r.cfg.MinLifetime + time.Minute)
|
||||
cat, _ := catalog.New([]catalog.Model{idle, chat, vital})
|
||||
h.r.cat.Store(cat)
|
||||
|
||||
h.r.Reconcile(t.Context())
|
||||
var stopped []string
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.PreemptedFor == "vital" {
|
||||
stopped = append(stopped, rep.Model)
|
||||
}
|
||||
}
|
||||
if len(stopped) != 1 || stopped[0] != "chat" {
|
||||
t.Errorf("stopped %v; only chat's replica makes room for a whole node", stopped)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOfTwoThatWouldEachMakeRoomTheSmallerIsStopped(t *testing.T) {
|
||||
cpu := inventory.Node{ID: "a", MgmtAddr: "10.0.0.1", State: inventory.Ready, MemoryTotalMB: 31 * 1024, CPUCount: 8}
|
||||
big := catalog.Model{Name: "big", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 12, CPUCores: 4}}}
|
||||
small := catalog.Model{Name: "small", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 2, CPUCores: 1}}}
|
||||
vital := catalog.Model{Name: "vital", Source: "/m", Class: catalog.ClassCritical,
|
||||
Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 15, CPUCores: 4}}}
|
||||
// 8 cores: big 4 + small 1 leaves 3, and vital needs 4. Stopping either
|
||||
// is enough; stopping big loses far more.
|
||||
for range 5 { // the choice must not depend on map order
|
||||
h := newFleetHarness(t, []catalog.Model{big, small}, cpu)
|
||||
h.settle(t, 3)
|
||||
h.advance(h.r.cfg.MinLifetime + time.Minute)
|
||||
cat, _ := catalog.New([]catalog.Model{big, small, vital})
|
||||
h.r.cat.Store(cat)
|
||||
h.r.Reconcile(t.Context())
|
||||
var stopped []string
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.PreemptedFor == "vital" {
|
||||
stopped = append(stopped, rep.Model)
|
||||
}
|
||||
}
|
||||
if len(stopped) != 1 || stopped[0] != "small" {
|
||||
t.Fatalf("stopped %v; the small replica alone makes room", stopped)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// SetQuotas supplies the tenancy tree's capacity quotas, read afresh each
|
||||
// time it is called. Without it nothing is guaranteed or limited, and
|
||||
// preemption goes by class alone.
|
||||
func (r *Reconciler) SetQuotas(f func() *quota.Policy) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.quotas = f
|
||||
}
|
||||
|
||||
func (r *Reconciler) policy() *quota.Policy {
|
||||
r.mu.Lock()
|
||||
f := r.quotas
|
||||
r.mu.Unlock()
|
||||
if f == nil {
|
||||
return nil
|
||||
}
|
||||
return f()
|
||||
}
|
||||
|
||||
// Holdings is what every replica holds, and for whom.
|
||||
func (r *Reconciler) Holdings() []quota.Holding {
|
||||
cat := r.cat.Load()
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
out := make([]quota.Holding, 0, len(r.replicas))
|
||||
for _, rep := range r.replicas {
|
||||
m, ok := cat.Resolve(rep.Model)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
prof, _ := m.ProfileByName(rep.Profile)
|
||||
var a quota.Amount
|
||||
for _, sh := range rep.Shards {
|
||||
a = a.Plus(quota.Amount{GPUs: len(sh.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
|
||||
}
|
||||
out = append(out, quota.Holding{Replica: rep.ID, Owner: m.Owner, Amount: a, Since: rep.CreatedAt})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// placementNeed is what a placement would hold.
|
||||
func placementNeed(m catalog.Model, pl *placement.Placement) quota.Amount {
|
||||
prof, _ := m.ProfileByName(pl.Profile.Name)
|
||||
var a quota.Amount
|
||||
for _, n := range pl.Nodes {
|
||||
a = a.Plus(quota.Amount{GPUs: len(n.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
|
||||
}
|
||||
return a
|
||||
}
|
||||
|
||||
// profileNeed is what a replica of a model's first profile would hold, for
|
||||
// deciding before there is a placement.
|
||||
func profileNeed(m catalog.Model) quota.Amount {
|
||||
if len(m.Profiles) == 0 {
|
||||
return quota.Amount{}
|
||||
}
|
||||
p := m.Profiles[0]
|
||||
nodes := max(p.Nodes, 1)
|
||||
gpus := p.GPUsPerNode
|
||||
if gpus == 0 && !p.CPUOnly() {
|
||||
gpus = max(p.TP, 1)
|
||||
}
|
||||
return quota.Amount{GPUs: gpus * nodes, CPUCores: cpuCores(p) * nodes, MemoryGB: p.MinMemoryGB * nodes}
|
||||
}
|
||||
|
||||
// overLimit says why a placement would take its model's owner past a limit.
|
||||
func (r *Reconciler) overLimit(m catalog.Model, pl *placement.Placement) string {
|
||||
pol := r.policy()
|
||||
if pol == nil || m.Owner == "" {
|
||||
return ""
|
||||
}
|
||||
return pol.AtLimit(m.Owner, pol.Usage(r.Holdings()), placementNeed(m, pl))
|
||||
}
|
||||
|
||||
// QuotaUse is one scope's quota beside what it holds.
|
||||
type QuotaUse struct {
|
||||
Scope string
|
||||
Guarantee store.Resources
|
||||
Limit store.Resources
|
||||
Holds quota.Amount
|
||||
}
|
||||
|
||||
// QuotaUse lists every quota with what its scope holds now.
|
||||
func (r *Reconciler) QuotaUse() []QuotaUse {
|
||||
pol := r.policy()
|
||||
if pol == nil {
|
||||
return nil
|
||||
}
|
||||
usage := pol.Usage(r.Holdings())
|
||||
var out []QuotaUse
|
||||
for _, q := range pol.Quotas() {
|
||||
key := quota.Key(q.Scope, q.ScopeID)
|
||||
out = append(out, QuotaUse{Scope: key, Guarantee: q.Guarantee, Limit: q.Limit, Holds: usage[key]})
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func owned(m catalog.Model, owner string) catalog.Model { m.Owner = owner; return m }
|
||||
|
||||
func gpus(n int) store.Resources { return store.Resources{GPUs: &n} }
|
||||
|
||||
// Two teams in one org; each 8-GPU node holds one tp8 replica.
|
||||
func withQuotas(h *harness, qs ...store.CapacityQuota) {
|
||||
pol := quota.New(qs, map[string][]string{"team:ml": {"org:acme"}, "team:web": {"org:acme"}})
|
||||
h.r.SetQuotas(func() *quota.Policy { return pol })
|
||||
}
|
||||
|
||||
func TestATenantWithinItsGuaranteeTakesBackWhatAnotherBorrowed(t *testing.T) {
|
||||
// web is guaranteed one node and borrowed the second; ml is guaranteed
|
||||
// one node and wants it. Same class, and web's replica is at its minimum
|
||||
// — by class alone nothing could be taken.
|
||||
web := owned(gpuModel("web-chat", "", 2), "team:web")
|
||||
ml := owned(gpuModel("ml-train", "", 1), "team:ml")
|
||||
h := newFleetHarness(t, []catalog.Model{web, withMin(ml, 0)}, gpuNode("1", 8), gpuNode("2", 8))
|
||||
withQuotas(h,
|
||||
store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "web", Guarantee: gpus(8)},
|
||||
store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "ml", Guarantee: gpus(8)})
|
||||
h.settle(t, 4)
|
||||
if replicasOf(h, "web-chat") != 2 {
|
||||
t.Fatalf("setup: web did not borrow the second node: %+v", h.r.Replicas())
|
||||
}
|
||||
h.advance(h.r.cfg.MinLifetime + time.Minute)
|
||||
cat, _ := catalog.New([]catalog.Model{web, ml})
|
||||
h.r.cat.Store(cat)
|
||||
|
||||
h.settle(t, 5)
|
||||
if replicasOf(h, "ml-train") != 1 || replicasOf(h, "web-chat") != 1 {
|
||||
t.Errorf("the borrowed node was not taken back: web %d, ml %d",
|
||||
replicasOf(h, "web-chat"), replicasOf(h, "ml-train"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReplicaWithinItsGuaranteeIsNotTakenByAHigherClassElsewhere(t *testing.T) {
|
||||
web := owned(gpuModel("web-chat", "", 1), "team:web")
|
||||
vital := owned(gpuModel("vital", catalog.ClassCritical, 1), "team:ml")
|
||||
h := fillThenWant(t, vital, web, gpuNode("1", 8))
|
||||
withQuotas(h, store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "web", Guarantee: gpus(8)})
|
||||
h.settle(t, 3)
|
||||
if replicasOf(h, "web-chat") != 1 || replicasOf(h, "vital") != 0 {
|
||||
t.Errorf("a guaranteed replica was preempted by another tenant's class: %+v", h.r.Replicas())
|
||||
}
|
||||
// Without the guarantee, critical takes it: the guarantee was the reason.
|
||||
h2 := fillThenWant(t, vital, web, gpuNode("1", 8))
|
||||
h2.settle(t, 3)
|
||||
if replicasOf(h2, "vital") != 1 {
|
||||
t.Error("setup: without a guarantee the critical model should have taken the node")
|
||||
}
|
||||
}
|
||||
|
||||
func TestALimitStopsAModelBeforeItStarts(t *testing.T) {
|
||||
web := owned(gpuModel("web-chat", "", 2), "team:web")
|
||||
h := newFleetHarness(t, []catalog.Model{web}, gpuNode("1", 8), gpuNode("2", 8))
|
||||
withQuotas(h, store.CapacityQuota{Scope: store.ScopeOrg, ScopeID: "acme", Limit: gpus(8)})
|
||||
h.settle(t, 4)
|
||||
if replicasOf(h, "web-chat") != 1 {
|
||||
t.Errorf("a second replica went past the org's 8-GPU limit: %+v", h.r.Replicas())
|
||||
}
|
||||
hs := health(h, "web-chat")
|
||||
if !hs.Unplaceable || !strings.Contains(hs.Placement, "org:acme is at its capacity limit (16 GPUs of 8)") {
|
||||
t.Errorf("the refusal should name the limit: %+v", hs)
|
||||
}
|
||||
room, _ := h.r.Headroom("web-chat", 5)
|
||||
if room.More != 0 || len(room.Blocked) != 1 || !strings.Contains(room.Blocked[0], "capacity limit") {
|
||||
t.Errorf("headroom should stop at the limit, not the empty node: %+v", room)
|
||||
}
|
||||
}
|
||||
@@ -15,6 +15,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
)
|
||||
|
||||
// ShardCommand is everything an agent needs to launch one shard.
|
||||
@@ -65,6 +66,12 @@ type Config struct {
|
||||
// Without a bound, draining a node whose model cannot be placed anywhere
|
||||
// else would never finish.
|
||||
DrainGrace time.Duration
|
||||
// PreemptGrace is how long a replica stopped to make room for another
|
||||
// is given to finish what it is serving: a request, not a replacement.
|
||||
PreemptGrace time.Duration
|
||||
// PreemptHold is how long a model that gave up room may not take it back,
|
||||
// so the replica it was given up for is placed first.
|
||||
PreemptHold time.Duration
|
||||
// ScaleUpWaiting is the average number of requests waiting per ready
|
||||
// replica, held for ScaleUpAfter, that adds a replica — up to the model's
|
||||
// max. ScaleDownAfter is how long with nothing waiting before one of the
|
||||
@@ -141,6 +148,8 @@ func DefaultConfig() Config {
|
||||
EvictionGrace: 2 * time.Minute,
|
||||
MinLifetime: 5 * time.Minute,
|
||||
DrainGrace: 5 * time.Minute,
|
||||
PreemptGrace: 30 * time.Second,
|
||||
PreemptHold: 5 * time.Minute,
|
||||
ParkAfter: 5,
|
||||
ScaleUpWaiting: 1,
|
||||
ScaleUpAfter: time.Minute,
|
||||
@@ -158,6 +167,12 @@ func (c *Config) normalize() {
|
||||
if c.BaseBackoff <= 0 {
|
||||
c.BaseBackoff = d.BaseBackoff
|
||||
}
|
||||
if c.PreemptGrace <= 0 {
|
||||
c.PreemptGrace = d.PreemptGrace
|
||||
}
|
||||
if c.PreemptHold <= 0 {
|
||||
c.PreemptHold = d.PreemptHold
|
||||
}
|
||||
if c.MaxBackoff < c.BaseBackoff {
|
||||
c.MaxBackoff = max(c.BaseBackoff, d.MaxBackoff)
|
||||
}
|
||||
@@ -250,6 +265,16 @@ type Reconciler struct {
|
||||
// placementLogged remembers the last "no placement" said per model, so it
|
||||
// is said when it changes rather than on every five-second pass.
|
||||
placementLogged map[string]placementNote
|
||||
// preempting is, per model, the replicas stopped to make room for it and
|
||||
// not gone yet; held is, per model that gave them up, until when it may
|
||||
// not take the room back; preempted counts, per model, replicas it lost.
|
||||
preempting map[string][]string
|
||||
held map[string]hold
|
||||
preempted map[string]int
|
||||
// quotas is the tenancy tree's capacity quotas as last read; nil is none.
|
||||
quotas func() *quota.Policy
|
||||
// measurements are benchmark results by model/profile; nil is none.
|
||||
measurements func() map[string]Measured
|
||||
|
||||
wake chan struct{}
|
||||
stop chan struct{}
|
||||
@@ -285,6 +310,9 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
|
||||
quietSince: map[string]time.Time{},
|
||||
placementLogged: map[string]placementNote{},
|
||||
lastExplain: map[string]placement.Result{},
|
||||
preempting: map[string][]string{},
|
||||
held: map[string]hold{},
|
||||
preempted: map[string]int{},
|
||||
wake: make(chan struct{}, 1),
|
||||
stop: make(chan struct{}),
|
||||
done: make(chan struct{}),
|
||||
@@ -704,6 +732,9 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
switch {
|
||||
case len(liveIDs) < desired && isParked:
|
||||
// Not placed. Its failure is on every status the operator reads.
|
||||
case len(liveIDs) < desired && r.isHeld(m.Name):
|
||||
// It gave up room to a model of a higher class, which is placed
|
||||
// first; taking it back now would undo the preemption.
|
||||
case len(liveIDs) < desired:
|
||||
if r.now().Before(blockedUntil) {
|
||||
// Still in backoff. Rebuilding immediately would hot-loop on a
|
||||
@@ -724,8 +755,12 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
r.mu.Lock()
|
||||
rep, ok := r.replicas[id]
|
||||
var expired bool
|
||||
grace, preemptedFor := r.cfg.DrainGrace, ""
|
||||
if ok {
|
||||
expired = !rep.DrainingAt.IsZero() && r.now().Sub(rep.DrainingAt) > r.cfg.DrainGrace
|
||||
if rep.PreemptedFor != "" {
|
||||
grace, preemptedFor = r.cfg.PreemptGrace, rep.PreemptedFor
|
||||
}
|
||||
expired = !rep.DrainingAt.IsZero() && r.now().Sub(rep.DrainingAt) > grace
|
||||
}
|
||||
r.mu.Unlock()
|
||||
|
||||
@@ -733,11 +768,15 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
case !ok:
|
||||
case replaced:
|
||||
r.stopReplica(ctx, id)
|
||||
case expired && preemptedFor != "":
|
||||
r.log.Info("stopping a preempted replica", "replica", id,
|
||||
"for", preemptedFor, "after", grace)
|
||||
r.stopReplica(ctx, id)
|
||||
case expired:
|
||||
// The replacement never arrived — often because nowhere else can
|
||||
// host it. Draining forever would leave the node undrainable.
|
||||
r.log.Warn("stopping a drained replica without a replacement",
|
||||
"replica", id, "after", r.cfg.DrainGrace)
|
||||
"replica", id, "after", grace)
|
||||
r.stopReplica(ctx, id)
|
||||
}
|
||||
}
|
||||
@@ -751,12 +790,13 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
// allocated, and starting several at once from the same inventory snapshot
|
||||
// would hand them the same GPUs.
|
||||
func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) {
|
||||
res := r.sched.Place(m.PlacementProfiles(), avoid)
|
||||
profiles := r.orderedProfiles(m)
|
||||
res := r.sched.Place(profiles, avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
// Spreading across failure domains is a preference, not a rule. If it
|
||||
// is the only thing standing in the way, take the placement anyway:
|
||||
// a second replica on one node still survives an engine crash.
|
||||
res = r.sched.Place(m.PlacementProfiles(), nil)
|
||||
res = r.sched.Place(profiles, nil)
|
||||
}
|
||||
|
||||
r.mu.Lock()
|
||||
@@ -775,9 +815,22 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
|
||||
if m.Scaling.OnDemand() && r.makeRoomFor(ctx, m) {
|
||||
return
|
||||
}
|
||||
if r.awaitingPreemption(m.Name) || r.preemptFor(ctx, m, avoid) {
|
||||
return
|
||||
}
|
||||
r.notePlacementFailure(m.Name, res.Explain())
|
||||
return
|
||||
}
|
||||
if why := r.overLimit(m, res.Placement); why != "" {
|
||||
// Recorded as the placement's answer, so the model reads as not
|
||||
// placeable, and says why, wherever that is shown.
|
||||
r.mu.Lock()
|
||||
r.lastExplain[m.Name] = placement.Result{Rejections: []placement.Rejection{
|
||||
{Profile: res.Placement.Profile.Name, Reason: why}}}
|
||||
r.mu.Unlock()
|
||||
r.notePlacementFailure(m.Name, why)
|
||||
return
|
||||
}
|
||||
if err := r.startReplica(ctx, m, res.Placement); err != nil {
|
||||
r.log.Error("starting replica failed", "model", m.Name, "err", err)
|
||||
}
|
||||
|
||||
@@ -156,6 +156,8 @@ type Replica struct {
|
||||
// DrainingAt is when the replica stopped taking new work, so the wait for a
|
||||
// replacement can be bounded.
|
||||
DrainingAt time.Time
|
||||
// PreemptedFor is the model this replica was stopped to make room for.
|
||||
PreemptedFor string
|
||||
// RetiringAt is set while the replica is being restarted: it keeps serving
|
||||
// but no longer counts toward the model's replicas, so a replacement is
|
||||
// placed beside it, and it is drained only once that replacement is ready.
|
||||
|
||||
@@ -29,6 +29,9 @@ var copyOrder = []string{
|
||||
"key_claims",
|
||||
"sessions",
|
||||
"limits",
|
||||
// Beside limits, and like them keyed by scope with no foreign key: a
|
||||
// quota on a scope that is later removed is inert, not an error.
|
||||
"capacity_quotas",
|
||||
"usage",
|
||||
"models",
|
||||
// No foreign key to models, deliberately: a measurement outlives the
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
-- Capacity a part of the tenancy tree is guaranteed, and the most it may hold.
|
||||
--
|
||||
-- Beside limits rather than in it: limits are counted on the request path, per
|
||||
-- key, in requests and tokens; these are counted by the reconciler, per model,
|
||||
-- in the GPUs, cores and memory its replicas hold. A NULL is a resource this
|
||||
-- row does not govern: guaranteed nothing, or limited by nothing.
|
||||
CREATE TABLE capacity_quotas (
|
||||
scope_type TEXT NOT NULL,
|
||||
scope_id TEXT NOT NULL,
|
||||
gpus INTEGER,
|
||||
cpu_cores INTEGER,
|
||||
memory_gb INTEGER,
|
||||
limit_gpus INTEGER,
|
||||
limit_cpu_cores INTEGER,
|
||||
limit_memory_gb INTEGER,
|
||||
updated_at BIGINT NOT NULL,
|
||||
updated_by TEXT NOT NULL,
|
||||
PRIMARY KEY (scope_type, scope_id)
|
||||
);
|
||||
@@ -0,0 +1,19 @@
|
||||
-- Capacity a part of the tenancy tree is guaranteed, and the most it may hold.
|
||||
--
|
||||
-- Beside limits rather than in it: limits are counted on the request path, per
|
||||
-- key, in requests and tokens; these are counted by the reconciler, per model,
|
||||
-- in the GPUs, cores and memory its replicas hold. A NULL is a resource this
|
||||
-- row does not govern: guaranteed nothing, or limited by nothing.
|
||||
CREATE TABLE capacity_quotas (
|
||||
scope_type TEXT NOT NULL,
|
||||
scope_id TEXT NOT NULL,
|
||||
gpus INTEGER,
|
||||
cpu_cores INTEGER,
|
||||
memory_gb INTEGER,
|
||||
limit_gpus INTEGER,
|
||||
limit_cpu_cores INTEGER,
|
||||
limit_memory_gb INTEGER,
|
||||
updated_at BIGINT NOT NULL,
|
||||
updated_by TEXT NOT NULL,
|
||||
PRIMARY KEY (scope_type, scope_id)
|
||||
);
|
||||
@@ -0,0 +1,127 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Resources are amounts of the three things a replica holds of a cluster.
|
||||
// Nil is not governed: nothing guaranteed, or nothing limited.
|
||||
type Resources struct {
|
||||
GPUs *int `json:"gpus,omitempty"`
|
||||
CPUCores *int `json:"cpu_cores,omitempty"`
|
||||
MemoryGB *int `json:"memory_gb,omitempty"`
|
||||
}
|
||||
|
||||
// Any reports whether any resource is governed.
|
||||
func (r Resources) Any() bool { return r.GPUs != nil || r.CPUCores != nil || r.MemoryGB != nil }
|
||||
|
||||
// CapacityQuota is what one part of the tenancy tree is guaranteed of the
|
||||
// cluster, and the most it may hold.
|
||||
type CapacityQuota struct {
|
||||
Scope Scope `json:"scope"`
|
||||
ScopeID string `json:"scope_id"`
|
||||
// Guarantee is held for the scope: while it holds less, what others
|
||||
// borrowed beyond their own guarantees is taken back for it.
|
||||
Guarantee Resources `json:"guarantee"`
|
||||
// Limit is the most it may hold, borrowing included.
|
||||
Limit Resources `json:"limit"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
UpdatedBy string `json:"updated_by"`
|
||||
}
|
||||
|
||||
// PutCapacityQuota sets a scope's quota, replacing what was there.
|
||||
func (s *DB) PutCapacityQuota(ctx context.Context, q CapacityQuota) error {
|
||||
if !validMembershipScope(q.Scope) || q.ScopeID == "" {
|
||||
return fmt.Errorf("a capacity quota belongs to an organization, team or project")
|
||||
}
|
||||
for _, v := range []*int{q.Guarantee.GPUs, q.Guarantee.CPUCores, q.Guarantee.MemoryGB,
|
||||
q.Limit.GPUs, q.Limit.CPUCores, q.Limit.MemoryGB} {
|
||||
if v != nil && *v < 0 {
|
||||
return fmt.Errorf("a quota cannot be negative")
|
||||
}
|
||||
}
|
||||
if q.UpdatedAt.IsZero() {
|
||||
q.UpdatedAt = time.Now()
|
||||
}
|
||||
_, err := s.db.ExecContext(ctx,
|
||||
`INSERT INTO capacity_quotas (scope_type, scope_id, gpus, cpu_cores, memory_gb,
|
||||
limit_gpus, limit_cpu_cores, limit_memory_gb, updated_at, updated_by)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT (scope_type, scope_id) DO UPDATE SET
|
||||
gpus = excluded.gpus, cpu_cores = excluded.cpu_cores, memory_gb = excluded.memory_gb,
|
||||
limit_gpus = excluded.limit_gpus, limit_cpu_cores = excluded.limit_cpu_cores,
|
||||
limit_memory_gb = excluded.limit_memory_gb,
|
||||
updated_at = excluded.updated_at, updated_by = excluded.updated_by`,
|
||||
string(q.Scope), q.ScopeID,
|
||||
nullInt(q.Guarantee.GPUs), nullInt(q.Guarantee.CPUCores), nullInt(q.Guarantee.MemoryGB),
|
||||
nullInt(q.Limit.GPUs), nullInt(q.Limit.CPUCores), nullInt(q.Limit.MemoryGB),
|
||||
q.UpdatedAt.UnixMilli(), q.UpdatedBy)
|
||||
return err
|
||||
}
|
||||
|
||||
// CapacityQuotas lists every quota.
|
||||
func (s *DB) CapacityQuotas(ctx context.Context) ([]CapacityQuota, error) {
|
||||
return queryAll(ctx, s.db,
|
||||
`SELECT scope_type, scope_id, gpus, cpu_cores, memory_gb,
|
||||
limit_gpus, limit_cpu_cores, limit_memory_gb, updated_at, updated_by
|
||||
FROM capacity_quotas ORDER BY scope_type, scope_id`, nil,
|
||||
func(rows *sql.Rows) (CapacityQuota, error) {
|
||||
var q CapacityQuota
|
||||
var scope string
|
||||
var g [6]sql.NullInt64
|
||||
var at int64
|
||||
err := rows.Scan(&scope, &q.ScopeID, &g[0], &g[1], &g[2], &g[3], &g[4], &g[5], &at, &q.UpdatedBy)
|
||||
q.Scope = Scope(scope)
|
||||
q.Guarantee = Resources{GPUs: intOf(g[0]), CPUCores: intOf(g[1]), MemoryGB: intOf(g[2])}
|
||||
q.Limit = Resources{GPUs: intOf(g[3]), CPUCores: intOf(g[4]), MemoryGB: intOf(g[5])}
|
||||
q.UpdatedAt = time.UnixMilli(at)
|
||||
return q, err
|
||||
})
|
||||
}
|
||||
|
||||
// DeleteCapacityQuota removes a scope's quota.
|
||||
func (s *DB) DeleteCapacityQuota(ctx context.Context, scope Scope, id string) error {
|
||||
_, err := s.db.ExecContext(ctx,
|
||||
`DELETE FROM capacity_quotas WHERE scope_type = ? AND scope_id = ?`, string(scope), id)
|
||||
return err
|
||||
}
|
||||
|
||||
// ScopeParents maps every team and project to the scopes above it, nearest
|
||||
// first: a project to its team and organization, a team to its organization.
|
||||
// Keys and values are written scope:id.
|
||||
func (s *DB) ScopeParents(ctx context.Context) (map[string][]string, error) {
|
||||
out := map[string][]string{}
|
||||
teams, err := s.AllTeams(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
orgOf := map[string]string{}
|
||||
for _, t := range teams {
|
||||
orgOf[t.ID] = t.OrgID
|
||||
out["team:"+t.ID] = []string{"org:" + t.OrgID}
|
||||
}
|
||||
type pt struct{ project, team string }
|
||||
rows, err := queryAll(ctx, s.db, `SELECT id, team_id FROM projects`, nil,
|
||||
func(r *sql.Rows) (pt, error) {
|
||||
var p pt
|
||||
return p, r.Scan(&p.project, &p.team)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, p := range rows {
|
||||
out["project:"+p.project] = []string{"team:" + p.team, "org:" + orgOf[p.team]}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func intOf(n sql.NullInt64) *int {
|
||||
if !n.Valid {
|
||||
return nil
|
||||
}
|
||||
v := int(n.Int64)
|
||||
return &v
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package store
|
||||
|
||||
import "testing"
|
||||
|
||||
func ip(v int) *int { return &v }
|
||||
|
||||
func TestAQuotaKeepsWhatItGovernsAndNothingElse(t *testing.T) {
|
||||
s := tenancyStore(t)
|
||||
org, _ := s.CreateOrg(t.Context(), "Acme")
|
||||
q := CapacityQuota{Scope: ScopeOrg, ScopeID: org, UpdatedBy: "ops",
|
||||
Guarantee: Resources{GPUs: ip(8)}, Limit: Resources{GPUs: ip(16), CPUCores: ip(0)}}
|
||||
if err := s.PutCapacityQuota(t.Context(), q); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
all, err := s.CapacityQuotas(t.Context())
|
||||
if err != nil || len(all) != 1 {
|
||||
t.Fatalf("%v %+v", err, all)
|
||||
}
|
||||
got := all[0]
|
||||
if *got.Guarantee.GPUs != 8 || got.Guarantee.CPUCores != nil || got.Guarantee.MemoryGB != nil {
|
||||
t.Errorf("guarantee %+v: only GPUs were set", got.Guarantee)
|
||||
}
|
||||
// Zero is a limit of nothing, not an absent one.
|
||||
if *got.Limit.GPUs != 16 || got.Limit.CPUCores == nil || *got.Limit.CPUCores != 0 {
|
||||
t.Errorf("limit %+v", got.Limit)
|
||||
}
|
||||
q.Guarantee = Resources{MemoryGB: ip(64)}
|
||||
if err := s.PutCapacityQuota(t.Context(), q); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
all, _ = s.CapacityQuotas(t.Context())
|
||||
if len(all) != 1 || all[0].Guarantee.GPUs != nil || *all[0].Guarantee.MemoryGB != 64 {
|
||||
t.Errorf("a second put should replace the first: %+v", all)
|
||||
}
|
||||
if err := s.DeleteCapacityQuota(t.Context(), ScopeOrg, org); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if all, _ = s.CapacityQuotas(t.Context()); len(all) != 0 {
|
||||
t.Error("the quota survived its deletion")
|
||||
}
|
||||
if err := s.PutCapacityQuota(t.Context(), CapacityQuota{Scope: ScopeOrg, ScopeID: org,
|
||||
Guarantee: Resources{GPUs: ip(-1)}}); err == nil {
|
||||
t.Error("a negative quota was stored")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEveryScopeKnowsWhatIsAboveIt(t *testing.T) {
|
||||
s := tenancyStore(t)
|
||||
org, _ := s.CreateOrg(t.Context(), "Acme")
|
||||
team, _ := s.CreateTeam(t.Context(), org, "ml")
|
||||
proj, _ := s.CreateProject(t.Context(), team, "training")
|
||||
p, err := s.ScopeParents(t.Context())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []string{"team:" + team, "org:" + org}
|
||||
if got := p["project:"+proj]; len(got) != 2 || got[0] != want[0] || got[1] != want[1] {
|
||||
t.Errorf("project's parents %v, expected %v", got, want)
|
||||
}
|
||||
if got := p["team:"+team]; len(got) != 1 || got[0] != "org:"+org {
|
||||
t.Errorf("team's parents %v", got)
|
||||
}
|
||||
}
|
||||
@@ -80,12 +80,13 @@
|
||||
<h2>Models</h2>
|
||||
|
||||
{{if .Models}}
|
||||
<table class="rows"><tr><th>Model</th><th>Aliases</th><th class="num">Ready</th><th class="num">Wanted</th><th>Reserves</th><th>Profiles</th>{{if .AnyFallbacks}}<th>Fallbacks</th>{{end}}<th>What it is for</th></tr>
|
||||
<table class="rows"><tr><th>Model</th><th>Aliases</th><th class="num">Ready</th><th class="num">Wanted</th><th>Class</th><th>Reserves</th><th>Profiles</th>{{if .AnyFallbacks}}<th>Fallbacks</th>{{end}}<th>What it is for</th></tr>
|
||||
{{range .Models}}<tr>
|
||||
<td><code>{{.Name}}</code></td>
|
||||
<td>{{join .Aliases}}</td>
|
||||
<td class="num">{{.Ready}}</td>
|
||||
<td class="num">{{.Min}}–{{.Max}}</td>
|
||||
<td{{if .HeldFor}} class="warn" title="gave room to {{.HeldFor}}"{{end}}>{{.Class}}{{if .Preempted}} <span class="note" title="replicas lost to a higher class">−{{.Preempted}}</span>{{end}}</td>
|
||||
<td{{with .Warnings}} class="warn" title="{{join .}}"{{end}}>{{if or .MemoryGB .CPUCores}}{{.MemoryGB}}G{{if .CPUCores}} · {{.CPUCores}}c{{end}}{{else}}—{{end}}
|
||||
{{- if .MeasuredMemoryGB}} <span class="note">{{printf "%.1f" .MeasuredMemoryGB}}G used</span>{{end}}{{if .Warnings}} ⚠{{end}}</td>
|
||||
<td>{{join .Profiles}}</td>
|
||||
|
||||
+10
-3
@@ -1463,8 +1463,8 @@ The launch path is now built too:
|
||||
- [x] `ctl bench`: a load ramp that reports capacity as the last concurrency
|
||||
level meeting the SLO, plus the queue depth and KV usage observed there,
|
||||
so `weight`, `target_waiting` and `kv_ceiling` stop being guesses
|
||||
- [ ] `ctl plan <model> --target-rps N`, which needs capacity profiles stored
|
||||
rather than printed
|
||||
- [x] Sizing from stored capacity: `ctl model target <model> --tokens-per-sec N`,
|
||||
and the autoscaler and profile choice use what `ctl bench --save` measured
|
||||
- [x] OTel tracing: an inference span with admission, schedule and each
|
||||
upstream attempt under it, over OTLP/HTTP when `tracing.endpoint` is
|
||||
set, and the traceparent carried on to the engine
|
||||
@@ -1480,7 +1480,14 @@ The launch path is now built too:
|
||||
- [ ] Admission displacement (dropping requests from lower bands)
|
||||
- [x] Autoscaling on load: a replica added while requests wait, up to a
|
||||
model's max, and taken back after ten quiet minutes
|
||||
- [ ] Priority-based preemption
|
||||
- [x] Priority-based preemption: capacity classes (critical, standard,
|
||||
opportunistic), preemption tried on a copy of the inventory first
|
||||
- [x] Capacity quotas per organization, team and project: guarantees kept,
|
||||
borrowing taken back, limits enforced
|
||||
- [x] CPU cores reserved like memory, and memory required of CPU models
|
||||
- [ ] Rack power, temperature and cooling as placement signals (a facility
|
||||
source; not yet built)
|
||||
- [ ] Defragmentation for large tensor-parallel replicas across NVLink domains
|
||||
- [ ] Control plane HA (Raft)
|
||||
- [x] Weight downloading and cache management: the mirror, agent fetches
|
||||
checked against their digest, and reclaiming what nothing uses
|
||||
|
||||
Reference in new issue
Block a user