Compare commits
18
Commits
v0.10.4
...
v0.10.6-rc.1
No files matched your search
+216
@@ -20,6 +20,222 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
Changes since v0.10.5, published as release candidates (v0.10.6-rc.N) until
|
||||
there are enough for v0.10.6.
|
||||
|
||||
### Capacity from the admin pages
|
||||
|
||||
What v0.10.5 made settable from the command line is now on the pages too:
|
||||
|
||||
- **Quotas** has a Capacity section: every scope's guaranteed and most GPUs,
|
||||
cores and memory beside what it holds, with a form to set, change or remove
|
||||
one. It goes through the same checks as `llmgw ctl quota`.
|
||||
- **A model's card** on the mirror page sets its class, its owner (when there
|
||||
are quotas to count against), and the memory and cores a replica reserves,
|
||||
beside its replica counts. Only what changed is applied, and each change is
|
||||
said.
|
||||
- **Room counts what a class may take.** The card, `llmgw ctl model target` and
|
||||
the headroom API count the replicas a model could get by stopping those its
|
||||
class outranks, separately from free room, and name whose they would be. A
|
||||
critical model's card no longer refuses a replica that preemption would make
|
||||
room for.
|
||||
|
||||
## v0.10.5 — 2026-09-30
|
||||
|
||||
Published first as release candidates v0.10.5-rc.1 to rc.14; rc.5 failed its
|
||||
format check and has no binaries.
|
||||
|
||||
After upgrading:
|
||||
|
||||
- **Declare what CPU models need.** A CPU model added from now on must say its
|
||||
memory; one stored before this still loads and is flagged in `ctl models`
|
||||
until `llmgw ctl model resources MODEL --memory GB --cores N` is run. Cores
|
||||
are then reserved per replica and become the engine's thread count, for
|
||||
replicas started after the change: `llmgw ctl restart --model MODEL`.
|
||||
- **Upgrade the mirror on its own host**, now with one command:
|
||||
`sudo llmgw upgrade v0.10.5 --service llmgw-mirror` (by hand once, for a
|
||||
mirror older than this). `llmgw ctl upgrade` says so at the end.
|
||||
|
||||
### CPU cores are reserved, and memory has to be declared
|
||||
|
||||
The first step of resource allocation for larger clusters: what a replica
|
||||
holds of a node is reserved, not assumed.
|
||||
|
||||
- **Cores.** A profile's `cpu_cores` is reserved out of its node's CPUs, as
|
||||
memory is, and is what its engine computes on (llama.cpp's `--threads`). A
|
||||
node holds as many CPU replicas as it has cores for, and two on one node no
|
||||
longer spin against each other. Agents report their CPU count; a node that
|
||||
does not, and a profile that declares no cores, constrain nothing.
|
||||
- **Memory is required of new CPU models.** A model with no GPU is all host
|
||||
memory, and one declaring none looked to placement as if it fitted
|
||||
everywhere, any number of times. Adding one without `min_memory_gb` is now
|
||||
refused. Models stored before this still load, and say what is missing.
|
||||
- **Declared against measured.** Each model shows what one replica reserves
|
||||
and the most one of its replicas uses — in `ctl models` (RESERVES) and on the
|
||||
cluster page — with a warning when nothing is declared or when it uses more
|
||||
than a tenth over what it reserves.
|
||||
- **Setting them.** `llmgw ctl model resources MODEL --memory GB --cores N`
|
||||
(and `POST /api/v1/models/{name}/resources`) for a model added at runtime.
|
||||
`ctl model serve` and the mirror's serve form fill in cores — half the
|
||||
smallest node the model fits on — beside the memory they already estimate.
|
||||
Running replicas keep their old figures until restarted.
|
||||
|
||||
The commands are `llmgw ctl model resources` and `llmgw ctl model class`; in
|
||||
rc.7 the first was briefly `llmgw ctl resources`.
|
||||
|
||||
### Capacity classes, and preemption
|
||||
|
||||
A model has a class — `critical`, `standard` or `opportunistic` — saying what
|
||||
its capacity is worth against another's when there is not enough for both.
|
||||
Unset, an on-demand model is opportunistic and every other model standard,
|
||||
which is how the catalog behaved before. `llmgw ctl model class MODEL CLASS`
|
||||
sets it.
|
||||
|
||||
When a model cannot be placed, replicas are stopped to make room for it, by
|
||||
these rules:
|
||||
|
||||
- A lower class gives way to a higher one.
|
||||
- Within a class, only replicas above a model's minimum — room it grew into
|
||||
under load — and only for a model still below its own minimum.
|
||||
- A critical model's minimum is never taken, nor a replica that has only just
|
||||
loaded, nor one spanning several nodes.
|
||||
- Idle on-demand replicas go first, then what lower classes grew into, then
|
||||
the same within the class, then lower classes' minimums; the longest unused
|
||||
first within each, and of equally idle ones the smallest.
|
||||
- The set is tried on a copy of the inventory first, at most three replicas,
|
||||
and nothing is stopped unless it makes the room. Of a set that does, each
|
||||
replica whose stopping turns out not to be needed is left running, so an
|
||||
idle replica tried first on the wrong node is not stopped with the one that
|
||||
made the room.
|
||||
|
||||
A replica stopped this way is drained and given 30 seconds to finish what it
|
||||
is serving. Its model is held back for up to five minutes, or until the model
|
||||
it gave way to has its replicas, so it does not take the room straight back.
|
||||
`ctl models` shows each model's class and "gave room to X" while held; the
|
||||
cluster page shows the class and how many replicas were lost; the metric is
|
||||
`llmgw_model_preempted_total{model,class}`, and every preemption is logged.
|
||||
|
||||
### Capacity quotas for the tenancy tree
|
||||
|
||||
An organization, team or project can be guaranteed GPUs, CPU cores and
|
||||
memory, and limited to at most some amount of each. What it holds is what the
|
||||
replicas of the models it owns hold — and those of the teams and projects
|
||||
beneath it. `llmgw ctl model owner MODEL team:Acme/ml` says whose a model is;
|
||||
one with no owner is shared and counts against nobody.
|
||||
|
||||
- **A guarantee is kept.** A replica within its owner's guarantee is never
|
||||
taken for another scope's model, whatever the classes: a guarantee a higher
|
||||
class elsewhere could take would not be one.
|
||||
- **Beyond it is borrowing.** Free room is used; when a scope within its own
|
||||
guarantee needs room, what others hold beyond theirs is taken back first,
|
||||
whatever their class — except a critical model's minimum.
|
||||
- **A limit is a ceiling.** A model whose owner would go past a limit anywhere
|
||||
up the tree gets no further replica, and says which limit; headroom and the
|
||||
mirror card stop there too.
|
||||
|
||||
`llmgw ctl quota set team:Acme/ml --gpus 8 --limit-gpus 16` (also `--cores`,
|
||||
`--memory` and their limits), `llmgw ctl quota list` with what each scope holds,
|
||||
`llmgw ctl quota clear`. Scopes are named by path or ID; a name that could mean
|
||||
two is refused. Changes take effect within half a minute. Metrics:
|
||||
`llmgw_capacity_holds`, `llmgw_capacity_guarantee` and `llmgw_capacity_limit`
|
||||
by scope and resource; alert: CapacityAtLimit.
|
||||
|
||||
### Sized by what a replica was measured to serve
|
||||
|
||||
What `llmgw ctl bench MODEL --save` measured — the most requests at once one
|
||||
replica served while meeting its SLO, and its throughput there — now drives
|
||||
decisions rather than only being printed:
|
||||
|
||||
- **Scaling up** when requests per replica pass what was measured, not only
|
||||
when the engines' queues fill: an engine takes requests into its slots long
|
||||
after latency has gone.
|
||||
- **Scaling down** only when the replicas left would stay under 70% of what
|
||||
they were measured to hold, so a replica taken back does not put the rest
|
||||
straight over.
|
||||
- **Choosing a profile.** Among profiles of equal priority, the one measured
|
||||
to serve the most per GPU — or per core — is tried first. Priority set by an
|
||||
operator still wins.
|
||||
- **Sizing.** `llmgw ctl model target MODEL --tokens-per-sec N` says how many
|
||||
replicas that takes, how many run, and whether the rest fit, with the reason
|
||||
when they do not; `--apply` sets it if they do. Headroom reports what one
|
||||
more replica would serve.
|
||||
|
||||
A model never measured behaves exactly as before. Measurements are read from
|
||||
the store every half minute.
|
||||
|
||||
### Upgrading a machine the control plane does not reach
|
||||
|
||||
`sudo llmgw upgrade VERSION --service UNIT` upgrades the machine it runs on —
|
||||
the mirror's host above all, whose service runs unprivileged and cannot replace
|
||||
its own binary as an agent does. The release's checksums are checked, the old
|
||||
binary kept, the unit restarted and watched until it stays up.
|
||||
|
||||
### Nodes report their own heat and power
|
||||
|
||||
The first step of facility-aware placement (docs/facility-aware-placement.md),
|
||||
observe-only: nothing is decided on these yet. Every agent reads, at most every
|
||||
ten seconds, and sends with its heartbeat:
|
||||
|
||||
- **GPUs** from nvidia-smi: temperature, HBM temperature where the driver has
|
||||
it, power draw and the enforced limit, and what is holding the clocks down —
|
||||
power cap, thermal slowdown, power brake. A GPU's temperature was read once,
|
||||
at registration; it is now current.
|
||||
- **CPUs** from hwmon, which is what lm_sensors reads, with nothing to install:
|
||||
the package temperature, and the board's inlet air sensor where it has one.
|
||||
- **CPU power** from RAPL's energy counters.
|
||||
|
||||
A sensor a machine does not have is not reported, never shown as zero. New
|
||||
metrics: `llmgw_gpu_power_watts`, `llmgw_gpu_power_limit_watts`,
|
||||
`llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_throttled{reason}`,
|
||||
`llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts`,
|
||||
`llmgw_node_inlet_temperature_celsius`. Each node's page shows them, with a GPU
|
||||
table of temperature, power against its limit, and what is throttling it.
|
||||
|
||||
### The mirror page in tabs
|
||||
|
||||
/admin/mirror was one long page: what the mirror holds, the Hugging Face
|
||||
explorer and the fetch form, one under another, with every visit waiting on
|
||||
Hugging Face for a list it might not show. It is now three tabs — Held (with
|
||||
how many are held, downloading and failed), Explore Hugging Face, and Fetch by
|
||||
URL — each with its own address. Only the tab shown does its work, and only
|
||||
Held reloads while something downloads. A download started from the explorer
|
||||
links to Held, where its progress is; a fetch the mirror refused comes back on
|
||||
the form it was sent from.
|
||||
|
||||
### A held model opens to where it runs, and how many
|
||||
|
||||
Clicking a model under Held opens a card beneath it: each replica with its
|
||||
node, state, age and the CPU and memory its engine uses, and how many more
|
||||
replicas the cluster has room for now, on which nodes, and why not more. The
|
||||
replicas it always runs and the most it grows to under load are changed there.
|
||||
Room is found by trying the scheduler itself on a copy of the inventory — the
|
||||
same decisions the next passes would make, not an estimate beside them — and a
|
||||
number of always-running replicas it cannot place is refused with the reason,
|
||||
rather than accepted and left short. A ceiling above the room is taken, and
|
||||
said. The same answer is at `GET /api/v1/models/{name}/headroom` on the admin
|
||||
API.
|
||||
|
||||
Each row also says how its model runs without being opened: replicas ready of
|
||||
those it should have, how many are still starting, and the nodes they are on —
|
||||
in amber when short, in red when parked or when it does not fit. It is one line
|
||||
per model beside its mode, and nodes drop the part of the name every node
|
||||
shares (node05 for llmgw-node05, the full name on hover).
|
||||
|
||||
### One line per record, in every table
|
||||
|
||||
The same rule across the admin pages. Headers do not wrap; a record is one
|
||||
line; free text is cut to its column with the whole of it on hover rather than
|
||||
given a row of its own; a row across the whole table is only a note or an open
|
||||
form. On the cluster page that means a node is one line (its address and logs
|
||||
beside its name, not under it), a model's description is a column rather than a
|
||||
second row, node names drop the prefix they all share, and the GPUs, classes,
|
||||
fabrics and fallbacks columns appear only when some row has something in them —
|
||||
on a CPU-only cluster they were three columns of dashes pushing the table past
|
||||
the page. On People, a person in many teams shows two and a count, the rest on
|
||||
hover.
|
||||
|
||||
## v0.10.4 — 2026-09-30
|
||||
|
||||
Published first as release candidates v0.10.4-rc.1 to rc.9.
|
||||
|
||||
+338
-166
File diff suppressed because it is too large.
Load diff
@@ -120,6 +120,10 @@ message Register {
|
||||
// meant "an agent too old to say" and constrained nothing — so a freshly
|
||||
// built node with no engine installed yet was a candidate for every model.
|
||||
bool engines_probed = 14;
|
||||
// cpu_count is the CPUs the agent may run on. Placement reserves cores
|
||||
// for CPU replicas out of it, the way it reserves memory out of
|
||||
// memory_total_mb; zero, from an older agent, reserves nothing.
|
||||
int32 cpu_count = 18;
|
||||
// imex is the service that lets a multi-node NVLink domain share memory.
|
||||
IMEX imex = 15;
|
||||
|
||||
@@ -281,6 +285,25 @@ message NodeLoad {
|
||||
int64 memory_used_bytes = 6;
|
||||
int64 memory_available_bytes = 7;
|
||||
repeated ShardLoad shards = 8;
|
||||
// What the machine's own sensors say; zero is not measured. cpu_temp_c is
|
||||
// the hottest CPU sensor, cpu_power_w the CPU packages' draw from RAPL,
|
||||
// inlet_temp_c the board's inlet or ambient sensor where it has one.
|
||||
double cpu_temp_c = 9;
|
||||
double cpu_power_w = 10;
|
||||
double inlet_temp_c = 11;
|
||||
repeated GPUSensor gpus = 12;
|
||||
}
|
||||
|
||||
// GPUSensor is one card's heat, power, and what is holding it back.
|
||||
message GPUSensor {
|
||||
int32 index = 1;
|
||||
double temp_c = 2;
|
||||
double memory_temp_c = 3;
|
||||
double power_w = 4;
|
||||
double power_limit_w = 5;
|
||||
// throttle names what is holding the clocks down now: sw_power_cap,
|
||||
// hw_slowdown, sw_thermal, hw_thermal, power_brake.
|
||||
repeated string throttle = 6;
|
||||
}
|
||||
|
||||
// ShardLoad is one engine process group's use.
|
||||
@@ -420,6 +443,8 @@ message Heartbeat {
|
||||
// domain, and the node has to stop being offered work that spans it.
|
||||
repeated GPUFabricState gpu_fabric = 16;
|
||||
IMEX imex = 17;
|
||||
// cpu_count as in Register, for a VM given more CPUs while it runs.
|
||||
int32 cpu_count = 18;
|
||||
}
|
||||
|
||||
// Provision is the software a node should have: which engines, at which
|
||||
@@ -530,6 +555,9 @@ message StartShard {
|
||||
// control plane from the replica id. The router sends it; nothing else
|
||||
// that can reach the node's ports has it.
|
||||
string api_key = 16;
|
||||
// cpu_threads is how many CPU threads a shard with no GPU computes on:
|
||||
// the cores placement reserved for it. Zero leaves it to the agent.
|
||||
int32 cpu_threads = 17;
|
||||
}
|
||||
|
||||
message EngineProfile {
|
||||
|
||||
@@ -84,6 +84,28 @@ groups:
|
||||
<model>` says what each node lacks. For an on-demand model, requests
|
||||
for it are told this rather than asked to retry.
|
||||
|
||||
- alert: ModelPreempted
|
||||
expr: increase(llmgw_model_preempted_total[15m]) > 0
|
||||
labels: { severity: info }
|
||||
annotations:
|
||||
summary: "{{ $labels.model }} ({{ $labels.class }}) gave up replicas to a higher class"
|
||||
description: >-
|
||||
Capacity ran short and a model of a higher class took this one's
|
||||
room. Expected now and then; if it keeps firing for the same model,
|
||||
the cluster is too small for what is asked of it at these classes.
|
||||
`llmgw ctl models` shows who it gave way to.
|
||||
|
||||
- alert: CapacityAtLimit
|
||||
expr: llmgw_capacity_holds >= on(scope, resource) llmgw_capacity_limit
|
||||
for: 15m
|
||||
labels: { severity: info }
|
||||
annotations:
|
||||
summary: "{{ $labels.scope }} holds all the {{ $labels.resource }} it may"
|
||||
description: >-
|
||||
Its models get no further replicas, under load or not, until
|
||||
something of theirs stops or the limit is raised: `llmgw ctl quota
|
||||
list`, `llmgw ctl quota set`.
|
||||
|
||||
- alert: EngineInstallFailed
|
||||
expr: llmgw_node_engine_install{state="failed"} == 1
|
||||
for: 15m
|
||||
|
||||
@@ -0,0 +1,303 @@
|
||||
# Facility-aware placement: rack power, temperature and cooling
|
||||
|
||||
**Status: step 0, reading the nodes' own sensors, is built; steps 1–5 are
|
||||
deferred** until the rack, PDU and power information of a real site is at
|
||||
hand — without it they would be designed against guesses.
|
||||
This is the design for step 5 of resource allocation, after reserved CPU cores
|
||||
and memory (1), capacity classes and preemption (2), tenant capacity quotas (3)
|
||||
and sizing by measured throughput (4), all of which shipped in v0.10.5. Of
|
||||
what follows, only the node sensors (rollout step 0) exist in the code: they
|
||||
are read, published and shown, and change no decision.
|
||||
|
||||
## Why
|
||||
|
||||
In a large installation the scarce thing is often not a GPU but the room it
|
||||
sits in. Three failures are invisible to placement today:
|
||||
|
||||
- **A rack goes over its power budget.** Every node in it reports healthy;
|
||||
placement sees free cards and fills them; the breaker, or the facility's
|
||||
power cap, decides the rest. A rack with A and B feeds is worse: it runs
|
||||
fine until one feed fails and the other cannot carry the load.
|
||||
- **Hot hardware is slow hardware.** GPUs throttle long before they fail. A
|
||||
replica placed in a rack whose inlet air is too warm serves fewer tokens per
|
||||
second than its measured capacity says, and nothing connects the two.
|
||||
- **Correlated failure.** Two replicas of a model on two nodes in the same rack
|
||||
or on the same PDU are one failure away from zero. Placement spreads replicas
|
||||
across nodes, not across racks.
|
||||
|
||||
The goal is for placement to treat facility limits the way it already treats
|
||||
memory and cores: reserve against a budget, refuse what does not fit, say why,
|
||||
and never let missing data stop the cluster.
|
||||
|
||||
## What it builds on
|
||||
|
||||
| Already in place | Used for |
|
||||
|---|---|
|
||||
| Node labels, reported by agents, matched by `node_selector` | Where a node is: its rack, row, PDU |
|
||||
| `inventory.Allocation` carries GPUs, memory and cores per shard | Adding watts, summed per rack and PDU |
|
||||
| Placement rejections (`explain-placement`, headroom's "why not") | Naming the rack that is full or hot |
|
||||
| Headroom: the scheduler dry-run on a copy of the inventory | Room that respects power budgets, for free |
|
||||
| Preemption with classes, minimums and a dry run (step 2) | Shedding load from a hot rack in the right order |
|
||||
| Measured throughput per profile (step 4) | Tokens per watt as a placement preference |
|
||||
| Node sensors, read by every agent (step 0, built) | Heat and power with no facility integration: see below |
|
||||
| `internal/promq`, a Prometheus client | Reading facility metrics that DCIM exporters publish |
|
||||
|
||||
## Concepts
|
||||
|
||||
**Location.** Each node has a place in a tree: site, room, row, rack. Power has
|
||||
its own tree beside it: a rack draws from one or more PDUs, and each PDU from a
|
||||
feed. Both are read from node labels with well-known keys:
|
||||
|
||||
```yaml
|
||||
# agent.yaml
|
||||
labels:
|
||||
llmgw.io/site: sto1
|
||||
llmgw.io/room: hall-b
|
||||
llmgw.io/row: "12"
|
||||
llmgw.io/rack: r12-07
|
||||
llmgw.io/pdu: r12-07-a,r12-07-b # both feeds, for a dual-fed rack
|
||||
```
|
||||
|
||||
A node without these labels is its own rack of one: placement behaves as it
|
||||
does today.
|
||||
|
||||
**Domain.** A rack, a row, a PDU, a room: anything with a budget or a
|
||||
condition. Domains are declared in a facility file (below) or discovered from
|
||||
node labels; a label naming a domain the file does not declare creates it with
|
||||
no budget and no condition.
|
||||
|
||||
**Budget.** A domain may have a power cap in watts. Placement reserves each
|
||||
replica's estimated draw against every domain its node belongs to and refuses a
|
||||
placement that would take any of them past its cap less a safety margin. For a
|
||||
dual-fed rack the cap that matters is what one feed can carry alone, since that
|
||||
is what the rack must survive.
|
||||
|
||||
**Condition.** A domain has a thermal and cooling condition: `ok`, `warm`, `hot`
|
||||
or `unknown`, derived from readings against thresholds with hysteresis.
|
||||
Conditions change what placement prefers and allows; they are not amounts to
|
||||
fill.
|
||||
|
||||
**Reading.** A measured value with a time: inlet temperature, power draw, a
|
||||
cooling unit's alarm state. Readings older than their staleness limit are
|
||||
treated as absent.
|
||||
|
||||
## Estimating a replica's draw
|
||||
|
||||
What placement reserves has to be known before the replica runs, so it is an
|
||||
estimate first and a measurement later:
|
||||
|
||||
1. **Declared.** A profile may say `power_watts`. The operator's figure wins.
|
||||
2. **Derived.** Otherwise GPU profiles take each card's power limit from NVML
|
||||
(the agent already reads the cards; the power limit is one more field) plus
|
||||
a per-node share for CPU, memory and fans; CPU profiles take their reserved
|
||||
cores times a per-core figure for the node's CPU model.
|
||||
3. **Measured.** Once replicas run, the agent reports their cards' actual draw
|
||||
(NVML) and, where the BMC allows, the node's (Redfish or IPMI DCMI). A
|
||||
domain's headroom is computed from the larger of what is reserved and what
|
||||
is measured, so an estimate that was too low is corrected by the room it
|
||||
stops offering, not by an outage.
|
||||
|
||||
## Decisions
|
||||
|
||||
### Power
|
||||
|
||||
- **Hard limit.** A placement that would take any domain past `cap × (1 −
|
||||
margin)` is rejected, with the domain and the figures: *rack r12-07: 14.2 of
|
||||
16.0 kW reserved, replica needs ~2.8, margin 10%*.
|
||||
- **Feeds.** For a domain with two feeds the cap is the smaller feed's rating
|
||||
times its derating (typically 80%), not the sum.
|
||||
- **Already over.** A domain found over its cap, because the estimate was low
|
||||
or the cap was lowered, gets no new placements, and an alert. It is not
|
||||
emptied automatically; shedding power is the facility's decision, and
|
||||
stopping inference to do it is ours only if asked (see shedding below).
|
||||
|
||||
### Temperature and cooling
|
||||
|
||||
Thresholds are per domain with defaults, and each condition needs to hold for a
|
||||
while before it changes, in both directions, so a single reading neither
|
||||
starts nor ends anything:
|
||||
|
||||
| Condition | Enters when | Placement |
|
||||
|---|---|---|
|
||||
| `ok` | below the warm threshold for 5 min | normal |
|
||||
| `warm` | inlet above the warm threshold for 2 min, GPU temperatures near throttling, or any GPU thermally throttled | allowed, scored down: other domains first |
|
||||
| `hot` | inlet above the hot threshold for 2 min, a cooling unit in alarm, or a sustained share of the domain's GPUs thermally throttled | no new placements, as a cordon would, but automatic and reversible |
|
||||
| `unknown` | no fresh reading | as `ok`, and reported as unknown |
|
||||
|
||||
**Shedding** (optional, off by default): a domain `hot` for longer than its
|
||||
shed delay has load moved off it using step 2's machinery. The lowest class goes
|
||||
first, and replicas above their model's minimum go before any at it. A critical
|
||||
minimum is never touched, and moves are rate limited to one replica per domain
|
||||
per interval. The replica is placed elsewhere first where there is room, so
|
||||
shedding moves work rather than dropping it. GPU power capping (`nvidia-smi
|
||||
-pl`) is a gentler alternative worth measuring against shedding: less
|
||||
throughput, no restart.
|
||||
|
||||
### Spreading
|
||||
|
||||
Replicas of one model are spread across racks, and across PDUs, the way they
|
||||
are spread across nodes today. This is a preference, dropped when it is the only
|
||||
thing standing in the way, as node spreading already is.
|
||||
|
||||
### Scoring
|
||||
|
||||
Placement today filters, then takes the first candidate in a fixed order
|
||||
(fewest free GPUs for GPU profiles, fewest replicas for CPU ones, nodes holding
|
||||
the weights first). Facility signals are better as a score than as more
|
||||
filters. Candidates that pass the hard limits are ordered by:
|
||||
|
||||
1. condition: `ok` before `warm`;
|
||||
2. power headroom left in the rack after this replica, more first, so load
|
||||
levels across racks;
|
||||
3. the existing order.
|
||||
|
||||
Once measurements include power, step 4's profile preference can use tokens per
|
||||
watt instead of tokens per GPU where both are known.
|
||||
|
||||
## Where the data comes from
|
||||
|
||||
A `FacilitySource` interface, implemented several ways, chosen in
|
||||
`server.yaml`:
|
||||
|
||||
```go
|
||||
type FacilitySource interface {
|
||||
// Domains is what the source knows: budgets, thresholds, members.
|
||||
Domains(ctx) ([]Domain, error)
|
||||
// Readings is the latest value of each signal, per domain.
|
||||
Readings(ctx) ([]Reading, error)
|
||||
}
|
||||
```
|
||||
|
||||
- **Static file.** Domains, caps and thresholds, written by hand or generated
|
||||
from a DCIM export. Enough for power budgets with no live data at all.
|
||||
- **Prometheus.** DCIM, PDU (SNMP exporter) and BMC (Redfish exporter)
|
||||
readings usually end up in Prometheus. The source runs configured instant
|
||||
queries and maps their labels onto domains:
|
||||
|
||||
```yaml
|
||||
facility:
|
||||
source: prometheus
|
||||
prometheus: http://monitoring:9090
|
||||
signals:
|
||||
inlet_temp_c:
|
||||
query: max by (rack) (dcim_rack_inlet_temperature_celsius)
|
||||
domain_label: rack
|
||||
stale_after: 2m
|
||||
power_watts:
|
||||
query: sum by (pdu) (pdu_outlet_power_watts)
|
||||
domain_label: pdu
|
||||
cooling_alarm:
|
||||
query: max by (row) (crac_alarm)
|
||||
domain_label: row
|
||||
```
|
||||
- **Agents (built, step 0).** What each node reads about itself, every ten
|
||||
seconds at most, sent with its heartbeat:
|
||||
- **GPUs, from nvidia-smi:** temperature, memory (HBM) temperature where the
|
||||
driver reports it, power draw, the enforced power limit, and what is
|
||||
holding the clocks down now — `sw_power_cap`, `hw_slowdown`, `sw_thermal`,
|
||||
`hw_thermal`, `power_brake`.
|
||||
- **CPUs, from hwmon** (what lm_sensors reads, without it installed): the
|
||||
package temperature, or the hottest core; and the board's inlet or ambient
|
||||
sensor where it has one (`SYSTIN`, `Inlet`, `Ambient`).
|
||||
- **CPU power, from RAPL:** the packages' energy counters over time,
|
||||
subdomains not counted twice, counter wrap handled.
|
||||
|
||||
A machine without a sensor reports nothing for it, never a zero; a VM has
|
||||
none of them. Published as `llmgw_gpu_temperature_celsius`,
|
||||
`llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_power_watts`,
|
||||
`llmgw_gpu_power_limit_watts`, `llmgw_gpu_throttled{reason}`,
|
||||
`llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts` and
|
||||
`llmgw_node_inlet_temperature_celsius`, and on each node's page.
|
||||
|
||||
What they can and cannot say:
|
||||
- **Throttling is the strongest signal there is.** A temperature is a proxy;
|
||||
a card holding its clocks down is the fact that invalidates its measured
|
||||
capacity, and it needs no threshold to be chosen.
|
||||
- **Measured power** is GPU draw plus CPU packages; memory, disks, fans and
|
||||
power-supply losses are not in it, so a node's draw is that plus an
|
||||
estimated remainder. A rack's is its llmgw nodes' plus a declared figure
|
||||
for everything else in it: switches and storage report nothing here.
|
||||
- **CPU and GPU temperatures follow the load, not the room.** A hot rack is
|
||||
inferred from many nodes running hot at comparable load, or from
|
||||
throttling, not from one node's CPU. Only a board inlet sensor reads the
|
||||
air, and most boards have none.
|
||||
- **No budget comes from a sensor.** A reading is a draw; the cap it is
|
||||
measured against still comes from the facility.
|
||||
- **Simulated.** Readings from a script, for tests and for rehearsing a
|
||||
thermal event on a cluster without facility data.
|
||||
|
||||
## Safety
|
||||
|
||||
- **Missing data never blocks.** A source that is down, a reading that is stale,
|
||||
a node without location labels: placement behaves as it does today, and says
|
||||
the data is missing (a metric, a line on the cluster page). Facility
|
||||
awareness can make placement more careful, never impossible.
|
||||
- **Hysteresis and hold times** on every condition, so a noisy sensor does not
|
||||
flap a rack between allowed and forbidden.
|
||||
- **Observe-only first.** A mode where every decision is computed, logged and
|
||||
shown ("would have refused: rack r12-07 hot") but nothing is enforced, run
|
||||
for long enough to trust the thresholds before turning enforcement on.
|
||||
- **Manual override.** `llmgw ctl facility hold RACK` and `release RACK`, for the
|
||||
facility team to mark a domain unavailable, or to clear a condition they know
|
||||
to be a sensor fault.
|
||||
- **Shedding is opt-in**, per domain or cluster-wide, and rate limited.
|
||||
|
||||
## What people see
|
||||
|
||||
- `explain-placement` and headroom name the domain: *rack r12-07 is hot (inlet
|
||||
31°C for 6 min)*, *pdu r12-07-a: 7.1 of 7.4 kW*.
|
||||
- `llmgw ctl facility` lists domains with budget, reserved, measured and
|
||||
condition. The cluster page gets a rack summary beside the nodes table.
|
||||
- Metrics: `llmgw_domain_power_budget_watts`, `_reserved_watts`,
|
||||
`_measured_watts`, `llmgw_domain_condition{condition}`,
|
||||
`llmgw_domain_reading_age_seconds`; alerts for a domain near its cap, a
|
||||
domain hot, a domain over its cap, and facility data gone stale.
|
||||
|
||||
## Rollout
|
||||
|
||||
Each step ships on its own and is useful without the next:
|
||||
|
||||
0. **Node sensors, observe-only (built).** GPU heat, power and throttling,
|
||||
CPU heat and power, inlet air where the board has a sensor; published and
|
||||
shown, no decision changed.
|
||||
1. **Location and spreading.** Well-known labels, domains from labels, replicas
|
||||
spread across racks and PDUs, locations on the node views. No facility data
|
||||
needed.
|
||||
2. **Power budgets from a static file.** Caps and margins, estimated draw per
|
||||
replica (declared or derived), hard limit in placement and headroom.
|
||||
3. **Readings, observe-only.** The Prometheus source beside the agents',
|
||||
conditions with hysteresis (throttling first among them), measured power
|
||||
folded into headroom; decisions shown, not enforced.
|
||||
4. **Thermal enforcement.** `warm` scored down, `hot` refused, manual hold and
|
||||
release.
|
||||
5. **Shedding and tokens per watt.** Opt-in shedding through step 2's
|
||||
preemption; measured power in profile preference.
|
||||
|
||||
## Testing
|
||||
|
||||
- The simulated source drives unit tests of every condition transition, the
|
||||
hysteresis, staleness, and the placement outcomes, with mutation checks as
|
||||
for steps 1–4.
|
||||
- Property tests: whatever the readings, placement never exceeds a declared
|
||||
cap with fresh data, and never refuses everything because data is missing.
|
||||
- On the PoC cluster, with no facility data: step 1 with labelled VMs, step 2
|
||||
with a static file of made-up caps, steps 3–4 with the simulated source
|
||||
replaying a thermal event.
|
||||
- Before enforcement anywhere real: a period in observe-only mode against the
|
||||
site's actual data.
|
||||
|
||||
## Open questions
|
||||
|
||||
- **What the sites have.** Which DCIM, which PDUs, which BMCs, and whether any
|
||||
of it already reaches a Prometheus. The source design assumes Prometheus is
|
||||
the common ground; a site without it needs another source.
|
||||
- **Who owns the numbers.** Caps, deratings and thresholds belong to the
|
||||
facility team. Whether they edit llmgw's file, or llmgw reads theirs.
|
||||
- **Phase balance.** Three-phase PDUs have per-phase limits; whether placement
|
||||
should see phases or only the PDU total.
|
||||
- **Cooling granularity.** Cooling is often per row or per aisle, not per rack;
|
||||
the domain tree allows either, but the defaults should match what sites
|
||||
actually measure.
|
||||
- **Power capping instead of shedding.** Whether to lower GPU power limits in a
|
||||
warm rack rather than move replicas, and how that interacts with measured
|
||||
capacity from step 4, which would no longer hold at the lower limit.
|
||||
+12
-2
@@ -940,8 +940,18 @@ restart either — it rebuilds the fleet from what the agents report.
|
||||
fields, and both sides treat an absent one as unknown rather than as a default.
|
||||
Agents older than v0.10.4 cannot upgrade themselves: move those by hand once,
|
||||
installing the release binary and restarting `llmgw-agent`, and every upgrade
|
||||
after that is the command above. The mirror, which is not an agent, is
|
||||
upgraded the same way by hand.
|
||||
after that is the command above.
|
||||
|
||||
**The mirror is upgraded on its own host**, since it is not an agent and its
|
||||
service cannot replace its own binary:
|
||||
|
||||
```
|
||||
sudo llmgw upgrade v0.10.5 --service llmgw-mirror
|
||||
```
|
||||
|
||||
The same checks — the release's checksums, the old binary kept beside the new —
|
||||
then a restart of the unit, waiting for it to stay up. A mirror older than
|
||||
v0.10.5 does not have the command: install the release binary by hand once.
|
||||
|
||||
Then confirm what you have, rather than assuming:
|
||||
|
||||
|
||||
@@ -92,6 +92,8 @@ rather than repeated.
|
||||
| `source` | `string` | [Model](#model) |
|
||||
| `profiles` | `[]Profile` | [Model](#model) |
|
||||
| `scaling` | `Scaling` | [Model](#model) |
|
||||
| `class` | `string` | [Model](#model) |
|
||||
| `owner` | `string` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
| `routing_profile` | `string` | [Model](#model) |
|
||||
| `slo` | `SLO` | [Model](#model) |
|
||||
@@ -129,6 +131,7 @@ rather than repeated.
|
||||
| `args` | `[]string` | [Profile](#profile) |
|
||||
| `env` | `map[string]string` | [Profile](#profile) |
|
||||
| `min_memory_gb` | `int` | [Profile](#profile) |
|
||||
| `cpu_cores` | `int` | [Profile](#profile) |
|
||||
| `min_disk_gb` | `int` | [Profile](#profile) |
|
||||
| `node_selector` | `map[string]string` | [Profile](#profile) |
|
||||
| `weight` | `float64` | [Profile](#profile) |
|
||||
@@ -735,6 +738,23 @@ Source is the weights path or a HuggingFace id.
|
||||
|
||||
Takes a [Scaling](#scaling) block.
|
||||
|
||||
### `class`
|
||||
|
||||
`string`
|
||||
|
||||
Class is how much this model's capacity is worth against another's,
|
||||
when there is not enough for both: critical, standard or opportunistic.
|
||||
Empty is standard, or opportunistic for a model loaded on demand. See
|
||||
EffectiveClass.
|
||||
|
||||
### `owner`
|
||||
|
||||
`string`
|
||||
|
||||
Owner is the part of the tenancy tree whose capacity quota this model
|
||||
counts against, written org:ID, team:ID or project:ID. Empty is shared:
|
||||
held by the cluster, counted against nobody.
|
||||
|
||||
### `fallbacks`
|
||||
|
||||
`[]string`
|
||||
@@ -966,6 +986,20 @@ this. A node that does not report its memory is not excluded either —
|
||||
an older agent must not have its nodes quietly emptied by an upgrade of
|
||||
the control plane.
|
||||
|
||||
### `cpu_cores`
|
||||
|
||||
`int`
|
||||
|
||||
CPUCores is the cores a replica of a profile with no GPU is given:
|
||||
reserved out of its node's CPUs by placement, as memory is, and what
|
||||
its engine computes on (llama.cpp's --threads).
|
||||
|
||||
Reserved because an engine takes every core it is told about, and two
|
||||
on one node that each think they have all of them spin against each
|
||||
other: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second
|
||||
to under one each. Zero reserves nothing and leaves the thread count to
|
||||
the node, which is every profile from before this.
|
||||
|
||||
### `min_disk_gb`
|
||||
|
||||
`int`
|
||||
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"slices"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -100,6 +101,9 @@ type Agent struct {
|
||||
smsErr map[string]string
|
||||
// load measures the machine between heartbeats.
|
||||
load *discovery.LoadSampler
|
||||
// sensors reads heat and power: CPU and board from hwmon and RAPL, GPUs
|
||||
// from nvidia-smi.
|
||||
sensors *discovery.SensorSampler
|
||||
|
||||
// fetchWatchers are the shards waiting on each source, so progress on one
|
||||
// download reaches every shard that asked for it. A fetch is single-flight
|
||||
@@ -145,6 +149,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
|
||||
logRing: ring,
|
||||
provKick: make(chan struct{}, 1),
|
||||
load: discovery.NewLoadSampler(),
|
||||
sensors: discovery.NewSensorSampler(),
|
||||
}
|
||||
a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
|
||||
return a, nil
|
||||
@@ -549,6 +554,7 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
|
||||
// and a node that grew from 7 GB to 32 GB has to be seen to have grown
|
||||
// without anybody restarting anything.
|
||||
hb.MemoryTotalMb = int64(discovery.MemoryTotalMB())
|
||||
hb.CpuCount = int32(runtime.NumCPU())
|
||||
|
||||
a.reprobeEngines()
|
||||
a.mu.Lock()
|
||||
@@ -636,6 +642,16 @@ func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
|
||||
ShardId: sh.ID, CpuCores: sh.CPUCores, MemoryRssBytes: sh.RSSBytes,
|
||||
})
|
||||
}
|
||||
if a.sensors != nil {
|
||||
se := a.sensors.Sample(context.Background())
|
||||
out.CpuTempC, out.CpuPowerW, out.InletTempC = se.CPUTempC, se.CPUPowerW, se.InletTempC
|
||||
for _, g := range se.GPUs {
|
||||
out.Gpus = append(out.Gpus, &llmgwv1.GPUSensor{
|
||||
Index: int32(g.Index), TempC: g.TempC, MemoryTempC: g.MemoryTempC,
|
||||
PowerW: g.PowerW, PowerLimitW: g.PowerLimitW, Throttle: g.Throttle,
|
||||
})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,242 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Sensors is what a node's own hardware says about heat and power: the CPUs,
|
||||
// the board where it has an inlet sensor, and each GPU. A zero is not
|
||||
// measured, never a reading of zero — a VM has none of these, and must not
|
||||
// look like a machine running at 0 °C on no power.
|
||||
type Sensors struct {
|
||||
// CPUTempC is the hottest CPU sensor: the package, or the hottest core
|
||||
// where no package reading exists.
|
||||
CPUTempC float64
|
||||
// CPUPowerW is the CPU packages' draw, from RAPL: the change in their
|
||||
// energy counters over the time between two samples.
|
||||
CPUPowerW float64
|
||||
// InletTempC is the air coming in, where the board has a sensor it calls
|
||||
// that. Most do not, and a CPU's temperature is not a substitute: it
|
||||
// follows the load, not the room.
|
||||
InletTempC float64
|
||||
GPUs []GPUSensor
|
||||
}
|
||||
|
||||
// GPUSensor is one card's heat, power and whether it is slowing itself down.
|
||||
type GPUSensor struct {
|
||||
Index int
|
||||
TempC float64
|
||||
MemoryTempC float64
|
||||
PowerW float64
|
||||
PowerLimitW float64
|
||||
// Throttle names what is holding the card's clocks down now:
|
||||
// sw_power_cap, hw_slowdown, sw_thermal, hw_thermal, power_brake. Empty
|
||||
// is running free. Idle and application clock settings are left out;
|
||||
// they are not the card being held back.
|
||||
Throttle []string
|
||||
}
|
||||
|
||||
// sensorEvery bounds how often nvidia-smi is run. Heartbeats come more often
|
||||
// than temperatures change, and each run costs a few hundred milliseconds.
|
||||
const sensorEvery = 10 * time.Second
|
||||
|
||||
// SensorSampler reads Sensors. The first sample has no CPU power, which needs
|
||||
// two readings of the energy counters.
|
||||
type SensorSampler struct {
|
||||
sys string // /sys, or a test's copy of it
|
||||
smi func(ctx context.Context, fields string) ([]byte, error)
|
||||
now func() time.Time
|
||||
|
||||
mu sync.Mutex
|
||||
last Sensors
|
||||
lastAt time.Time
|
||||
prevEnergy map[string]uint64
|
||||
prevAt time.Time
|
||||
}
|
||||
|
||||
// NewSensorSampler reads this machine.
|
||||
func NewSensorSampler() *SensorSampler {
|
||||
return &SensorSampler{sys: "/sys", smi: runSMI, now: time.Now}
|
||||
}
|
||||
|
||||
func runSMI(ctx context.Context, fields string) ([]byte, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
//nolint:gosec // fixed binary, fields from this file
|
||||
return exec.CommandContext(ctx, "nvidia-smi", "--query-gpu="+fields,
|
||||
"--format=csv,noheader,nounits").Output()
|
||||
}
|
||||
|
||||
// Sample reads the sensors, or returns the last reading when that is recent.
|
||||
func (s *SensorSampler) Sample(ctx context.Context) Sensors {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
now := s.now()
|
||||
if !s.lastAt.IsZero() && now.Sub(s.lastAt) < sensorEvery {
|
||||
return s.last
|
||||
}
|
||||
out := Sensors{}
|
||||
out.CPUTempC, out.InletTempC = s.hwmon()
|
||||
out.CPUPowerW = s.rapl(now)
|
||||
out.GPUs = s.gpus(ctx)
|
||||
s.last, s.lastAt = out, now
|
||||
return out
|
||||
}
|
||||
|
||||
// hwmon reads the hottest CPU sensor and an inlet sensor, from every hwmon
|
||||
// device the kernel exposes — what lm_sensors reads, without it installed.
|
||||
func (s *SensorSampler) hwmon() (cpu, inlet float64) {
|
||||
devs, _ := filepath.Glob(filepath.Join(s.sys, "class/hwmon/hwmon*"))
|
||||
var pkg, core float64
|
||||
for _, d := range devs {
|
||||
name := readTrim(filepath.Join(d, "name"))
|
||||
inputs, _ := filepath.Glob(filepath.Join(d, "temp*_input"))
|
||||
for _, in := range inputs {
|
||||
milli, err := strconv.ParseFloat(readTrim(in), 64)
|
||||
if err != nil || milli <= 0 {
|
||||
continue
|
||||
}
|
||||
c := milli / 1000
|
||||
label := strings.ToLower(readTrim(strings.TrimSuffix(in, "_input") + "_label"))
|
||||
switch {
|
||||
case isCPUSensor(name) && (strings.HasPrefix(label, "package") || label == "tctl" || label == "tdie"):
|
||||
pkg = max(pkg, c)
|
||||
case isCPUSensor(name):
|
||||
core = max(core, c)
|
||||
case label == "cpu":
|
||||
// A board's own reading of the CPU, as Dell's SMM gives.
|
||||
core = max(core, c)
|
||||
case strings.Contains(label, "inlet") || strings.Contains(label, "ambient") ||
|
||||
strings.Contains(label, "intake") || label == "systin":
|
||||
inlet = max(inlet, c)
|
||||
}
|
||||
}
|
||||
}
|
||||
if pkg > 0 {
|
||||
return pkg, inlet
|
||||
}
|
||||
return core, inlet
|
||||
}
|
||||
|
||||
func isCPUSensor(name string) bool {
|
||||
switch name {
|
||||
case "coretemp", "k10temp", "zenpower", "cpu_thermal":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// rapl is the CPU packages' power from their energy counters. Package
|
||||
// domains only: the dram and core subdomains are parts of a package, and
|
||||
// adding them would count the same energy twice.
|
||||
func (s *SensorSampler) rapl(now time.Time) float64 {
|
||||
doms, _ := filepath.Glob(filepath.Join(s.sys, "class/powercap/intel-rapl:*"))
|
||||
energy := map[string]uint64{}
|
||||
wrap := map[string]uint64{}
|
||||
for _, d := range doms {
|
||||
if strings.Count(filepath.Base(d), ":") != 1 {
|
||||
continue // intel-rapl:0:0 is a subdomain of intel-rapl:0
|
||||
}
|
||||
e, err := strconv.ParseUint(readTrim(filepath.Join(d, "energy_uj")), 10, 64)
|
||||
if err != nil {
|
||||
continue // unreadable without root on newer kernels
|
||||
}
|
||||
energy[d] = e
|
||||
wrap[d], _ = strconv.ParseUint(readTrim(filepath.Join(d, "max_energy_range_uj")), 10, 64)
|
||||
}
|
||||
prev, prevAt := s.prevEnergy, s.prevAt
|
||||
s.prevEnergy, s.prevAt = energy, now
|
||||
dt := now.Sub(prevAt).Seconds()
|
||||
if prev == nil || dt <= 0 || len(energy) == 0 {
|
||||
return 0
|
||||
}
|
||||
var joules float64
|
||||
for d, e := range energy {
|
||||
p, ok := prev[d]
|
||||
if !ok {
|
||||
return 0 // a domain appeared: no fair difference this time
|
||||
}
|
||||
if e < p { // the counter wrapped
|
||||
e += wrap[d]
|
||||
}
|
||||
joules += float64(e-p) / 1e6
|
||||
}
|
||||
return joules / dt
|
||||
}
|
||||
|
||||
// The throttle reasons nvidia-smi reports, as bits. Only those that mean the
|
||||
// card is being held back are named.
|
||||
var throttleBits = []struct {
|
||||
bit uint64
|
||||
name string
|
||||
}{
|
||||
{0x4, "sw_power_cap"},
|
||||
{0x8, "hw_slowdown"},
|
||||
{0x20, "sw_thermal"},
|
||||
{0x40, "hw_thermal"},
|
||||
{0x80, "power_brake"},
|
||||
}
|
||||
|
||||
// gpus asks nvidia-smi for each card's heat and power. The memory temperature
|
||||
// is not known to every driver, and a field nvidia-smi does not know fails the
|
||||
// whole query, so it is asked for first and dropped on failure.
|
||||
func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
|
||||
withMem := "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active,temperature.memory"
|
||||
out, err := s.smi(ctx, withMem)
|
||||
if err != nil {
|
||||
if out, err = s.smi(ctx, strings.TrimSuffix(withMem, ",temperature.memory")); err != nil {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
var gs []GPUSensor
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
f := strings.Split(line, ",")
|
||||
if len(f) < 5 {
|
||||
continue
|
||||
}
|
||||
for i := range f {
|
||||
f[i] = strings.TrimSpace(f[i])
|
||||
}
|
||||
idx, err := strconv.Atoi(f[0])
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
g := GPUSensor{Index: idx, TempC: num(f[1]), PowerW: num(f[2]), PowerLimitW: num(f[3])}
|
||||
if bits, err := strconv.ParseUint(strings.TrimPrefix(f[4], "0x"), 16, 64); err == nil {
|
||||
for _, b := range throttleBits {
|
||||
if bits&b.bit != 0 {
|
||||
g.Throttle = append(g.Throttle, b.name)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(f) > 5 {
|
||||
g.MemoryTempC = num(f[5])
|
||||
}
|
||||
gs = append(gs, g)
|
||||
}
|
||||
return gs
|
||||
}
|
||||
|
||||
// num reads a figure nvidia-smi may give as "[N/A]" or "N/A".
|
||||
func num(s string) float64 {
|
||||
v, err := strconv.ParseFloat(s, 64)
|
||||
if err != nil || v < 0 {
|
||||
return 0
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func readTrim(path string) string {
|
||||
b, err := os.ReadFile(path) //nolint:gosec // sysfs
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
@@ -0,0 +1,169 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func put(t *testing.T, root, rel, body string) {
|
||||
t.Helper()
|
||||
p := filepath.Join(root, rel)
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(p, []byte(body+"\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// dellHost is the layout of the Proxmox host these tests were written
|
||||
// against: Dell's SMM with a CPU and SODIMM sensors, coretemp with a package
|
||||
// and cores, and RAPL with a package and its dram subdomain.
|
||||
func dellHost(t *testing.T) string {
|
||||
root := t.TempDir()
|
||||
put(t, root, "class/hwmon/hwmon0/name", "dell_smm")
|
||||
put(t, root, "class/hwmon/hwmon0/temp1_input", "55000")
|
||||
put(t, root, "class/hwmon/hwmon0/temp1_label", "CPU")
|
||||
put(t, root, "class/hwmon/hwmon0/temp2_input", "47000")
|
||||
put(t, root, "class/hwmon/hwmon0/temp2_label", "SODIMM")
|
||||
put(t, root, "class/hwmon/hwmon1/name", "coretemp")
|
||||
put(t, root, "class/hwmon/hwmon1/temp1_input", "58000")
|
||||
put(t, root, "class/hwmon/hwmon1/temp1_label", "Package id 0")
|
||||
put(t, root, "class/hwmon/hwmon1/temp10_input", "61000")
|
||||
put(t, root, "class/hwmon/hwmon1/temp10_label", "Core 8")
|
||||
put(t, root, "class/powercap/intel-rapl:0/energy_uj", "1000000000")
|
||||
put(t, root, "class/powercap/intel-rapl:0/max_energy_range_uj", "262143328850")
|
||||
put(t, root, "class/powercap/intel-rapl:0:0/energy_uj", "5000000")
|
||||
return root
|
||||
}
|
||||
|
||||
func sampler(sys string, smi func(context.Context, string) ([]byte, error), clock *time.Time) *SensorSampler {
|
||||
return &SensorSampler{sys: sys, smi: smi, now: func() time.Time { return *clock }}
|
||||
}
|
||||
|
||||
func noSMI(context.Context, string) ([]byte, error) { return nil, errors.New("not found") }
|
||||
|
||||
func TestAHostsCPUHeatAndPowerAreReadAsLmSensorsWould(t *testing.T) {
|
||||
root := dellHost(t)
|
||||
now := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||
s := sampler(root, noSMI, &now)
|
||||
|
||||
first := s.Sample(t.Context())
|
||||
// The package reading, not the hotter core: the package is what the
|
||||
// chip itself reports as its temperature.
|
||||
if first.CPUTempC != 58 {
|
||||
t.Errorf("CPU %.1f °C, expected the package's 58", first.CPUTempC)
|
||||
}
|
||||
if first.CPUPowerW != 0 {
|
||||
t.Error("power from one reading of a counter is invented")
|
||||
}
|
||||
if first.InletTempC != 0 || first.GPUs != nil {
|
||||
t.Errorf("sensors this host does not have were reported: %+v", first)
|
||||
}
|
||||
|
||||
// 10 s later the package has used 650 J, and its dram subdomain some of
|
||||
// that — counted once.
|
||||
put(t, root, "class/powercap/intel-rapl:0/energy_uj", "1650000000")
|
||||
put(t, root, "class/powercap/intel-rapl:0:0/energy_uj", "99000000")
|
||||
now = now.Add(10 * time.Second)
|
||||
if got := s.Sample(t.Context()).CPUPowerW; got < 64.9 || got > 65.1 {
|
||||
t.Errorf("CPU power %.2f W, expected 65", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACounterThatWrapsIsNotANegativeDraw(t *testing.T) {
|
||||
root := dellHost(t)
|
||||
now := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||
put(t, root, "class/powercap/intel-rapl:0/energy_uj", "262143000000")
|
||||
s := sampler(root, noSMI, &now)
|
||||
s.Sample(t.Context())
|
||||
// 328,850 µJ to the wrap, then 499,671,150 past it: 500 J in 10 s.
|
||||
put(t, root, "class/powercap/intel-rapl:0/energy_uj", "499671150")
|
||||
now = now.Add(10 * time.Second)
|
||||
if got := s.Sample(t.Context()).CPUPowerW; got < 49.9 || got > 50.1 {
|
||||
t.Errorf("across a wrap: %.2f W, expected 50", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAVMHasNoSensorsAndSaysNothing(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
put(t, root, "class/thermal/cooling_device0/type", "Processor")
|
||||
now := time.Now()
|
||||
got := sampler(root, noSMI, &now).Sample(t.Context())
|
||||
if got.CPUTempC != 0 || got.CPUPowerW != 0 || got.InletTempC != 0 || got.GPUs != nil {
|
||||
t.Errorf("a machine with no sensors reported %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABoardsInletSensorIsTheRoomNotTheCPU(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
put(t, root, "class/hwmon/hwmon2/name", "nct6798")
|
||||
put(t, root, "class/hwmon/hwmon2/temp1_input", "24500")
|
||||
put(t, root, "class/hwmon/hwmon2/temp1_label", "SYSTIN")
|
||||
put(t, root, "class/hwmon/hwmon3/name", "k10temp")
|
||||
put(t, root, "class/hwmon/hwmon3/temp1_input", "71250")
|
||||
put(t, root, "class/hwmon/hwmon3/temp1_label", "Tctl")
|
||||
now := time.Now()
|
||||
got := sampler(root, noSMI, &now).Sample(t.Context())
|
||||
if got.InletTempC != 24.5 || got.CPUTempC != 71.25 {
|
||||
t.Errorf("inlet %.2f, CPU %.2f", got.InletTempC, got.CPUTempC)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachGPUSaysHowHotHowHungryAndWhatHoldsItBack(t *testing.T) {
|
||||
var asked []string
|
||||
smi := func(_ context.Context, fields string) ([]byte, error) {
|
||||
asked = append(asked, fields)
|
||||
return []byte(
|
||||
"0, 71, 612.45, 700.00, 0x0000000000000000, 84\n" +
|
||||
"1, 88, 698.10, 700.00, 0x0000000000000024, 95\n" + // sw_power_cap + sw_thermal
|
||||
"2, 45, [N/A], 700.00, 0x0000000000000001, [N/A]\n"), nil // idle only
|
||||
}
|
||||
now := time.Now()
|
||||
ss := sampler(t.TempDir(), smi, &now)
|
||||
got := ss.Sample(t.Context())
|
||||
if len(got.GPUs) != 3 {
|
||||
t.Fatalf("%+v", got.GPUs)
|
||||
}
|
||||
g0, g1, g2 := got.GPUs[0], got.GPUs[1], got.GPUs[2]
|
||||
if g0.TempC != 71 || g0.PowerW != 612.45 || g0.PowerLimitW != 700 || g0.MemoryTempC != 84 || g0.Throttle != nil {
|
||||
t.Errorf("card 0 %+v", g0)
|
||||
}
|
||||
if strings.Join(g1.Throttle, ",") != "sw_power_cap,sw_thermal" {
|
||||
t.Errorf("card 1 throttled by %v", g1.Throttle)
|
||||
}
|
||||
if g2.PowerW != 0 || g2.MemoryTempC != 0 || g2.Throttle != nil {
|
||||
t.Errorf("card 2: N/A is not a reading, and idle is not being held back: %+v", g2)
|
||||
}
|
||||
// Within ten seconds the last reading is reused rather than running
|
||||
// nvidia-smi on every heartbeat; after, it is asked again.
|
||||
now = now.Add(5 * time.Second)
|
||||
ss.Sample(t.Context())
|
||||
if len(asked) != 1 {
|
||||
t.Errorf("nvidia-smi was asked %d times within ten seconds", len(asked))
|
||||
}
|
||||
now = now.Add(6 * time.Second)
|
||||
ss.Sample(t.Context())
|
||||
if len(asked) != 2 {
|
||||
t.Errorf("nvidia-smi was asked %d times after eleven seconds, expected 2", len(asked))
|
||||
}
|
||||
}
|
||||
|
||||
func TestADriverThatDoesNotKnowMemoryTemperatureStillReportsTheRest(t *testing.T) {
|
||||
smi := func(_ context.Context, fields string) ([]byte, error) {
|
||||
if strings.Contains(fields, "temperature.memory") {
|
||||
return nil, errors.New(`Field "temperature.memory" is not a valid field to query.`)
|
||||
}
|
||||
return []byte("0, 66, 250.0, 300.0, 0x0000000000000040\n"), nil
|
||||
}
|
||||
now := time.Now()
|
||||
got := sampler(t.TempDir(), smi, &now).Sample(t.Context())
|
||||
if len(got.GPUs) != 1 || got.GPUs[0].TempC != 66 || strings.Join(got.GPUs[0].Throttle, ",") != "hw_thermal" {
|
||||
t.Errorf("%+v", got.GPUs)
|
||||
}
|
||||
}
|
||||
@@ -76,6 +76,10 @@ type ShardSpec struct {
|
||||
// and accounting. Given to the engine in its environment rather than on
|
||||
// its command line, which every user on the node can read.
|
||||
APIKey string `json:"api_key,omitempty"`
|
||||
// Threads is the CPU cores placement reserved for a shard with no GPU,
|
||||
// and so what its engine computes on. Zero leaves it to the engine's own
|
||||
// configuration.
|
||||
Threads int `json:"threads,omitempty"`
|
||||
|
||||
// Pool and Fabric are control plane concepts the agent cannot derive. They
|
||||
// are carried so the agent can report them back after a restart; Fabric
|
||||
|
||||
@@ -133,7 +133,12 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
|
||||
args = append(args, "--n-gpu-layers", "999")
|
||||
}
|
||||
if len(s.GPUIndices) == 0 && !hasFlag(p.Args, "--threads", "-t") {
|
||||
n := l.cfg.Threads
|
||||
// What placement reserved first: it is what the node's other
|
||||
// replicas were placed around.
|
||||
n := s.Threads
|
||||
if n <= 0 {
|
||||
n = l.cfg.Threads
|
||||
}
|
||||
if n <= 0 {
|
||||
n = max(numCPU()/2, 1)
|
||||
}
|
||||
|
||||
@@ -294,3 +294,15 @@ func TestACPUShardLeavesHalfTheMachineForAnother(t *testing.T) {
|
||||
t.Errorf("--threads %q on one CPU", v)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheCoresPlacementReservedAreTheThreads(t *testing.T) {
|
||||
s := llamaSpec(nil)
|
||||
s.Threads = 3
|
||||
cmds, err := (&LlamaCpp{cfg: LlamaCppConfig{Threads: 8}}).Commands(s)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v, _ := flagValue(cmds[0].Args, "--threads"); v != "3" {
|
||||
t.Errorf("--threads %q; what placement reserved should win over the node's setting", v)
|
||||
}
|
||||
}
|
||||
+37
-5
@@ -137,7 +137,7 @@ func newCtlCmd() *cobra.Command {
|
||||
// apply to it; it lives here because this is where operators look.
|
||||
newBenchCmd(client, &asJSON),
|
||||
newStatusCmd(client, &asJSON),
|
||||
newModelCmd(client),
|
||||
newModelCmd(client, &asJSON),
|
||||
newScaleCmd(client),
|
||||
newRestartCmd(client),
|
||||
newRetryCmd(client),
|
||||
@@ -146,6 +146,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newToolCheckCmd(),
|
||||
newWeightsCmd(client, &asJSON),
|
||||
newCapacityCmd(client, &asJSON),
|
||||
newQuotaCmd(client, &asJSON),
|
||||
newUsageCmd(client, &asJSON),
|
||||
newCertCmd(),
|
||||
)
|
||||
@@ -253,6 +254,8 @@ func modelStatus(m adminapi.ModelView) string {
|
||||
switch {
|
||||
case m.Parked:
|
||||
return "parked"
|
||||
case m.HeldFor != "":
|
||||
return "gave room to " + m.HeldFor
|
||||
case m.Unplaceable:
|
||||
return "cannot place"
|
||||
case m.Min == 0 && m.Total == 0:
|
||||
@@ -323,17 +326,27 @@ func newModelsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
}
|
||||
|
||||
w := tab()
|
||||
fmt.Fprintln(w, "MODEL\tSERVING\tREPLICAS\tSTATUS\tALIASES\tPROFILES\tFALLBACKS")
|
||||
fmt.Fprintln(w, "MODEL\tSERVING\tREPLICAS\tSTATUS\tCLASS\tRESERVES\tALIASES\tPROFILES\tFALLBACKS")
|
||||
var warnings []string
|
||||
for _, m := range models {
|
||||
serving := "no"
|
||||
if m.Ready > 0 {
|
||||
serving = "yes"
|
||||
}
|
||||
fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\n",
|
||||
m.Name, serving, m.Ready, m.Min, modelStatus(m),
|
||||
fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\t%s\t%s\n",
|
||||
m.Name, serving, m.Ready, m.Min, modelStatus(m), m.Class, reserves(m),
|
||||
join(m.Aliases), join(m.Profiles), join(m.Fallbacks))
|
||||
for _, x := range m.Warnings {
|
||||
warnings = append(warnings, m.Name+": "+x)
|
||||
}
|
||||
}
|
||||
return w.Flush()
|
||||
if err := w.Flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, x := range warnings {
|
||||
fmt.Println("warning:", x)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -511,3 +524,22 @@ func dash(s string) string {
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// reserves is what one replica of a model holds of a node, with what it was
|
||||
// measured using beside it: "6G 2c (5.1G used)".
|
||||
func reserves(m adminapi.ModelView) string {
|
||||
if m.MemoryGB == 0 && m.CPUCores == 0 {
|
||||
return "-"
|
||||
}
|
||||
out := fmt.Sprintf("%dG", m.MemoryGB)
|
||||
if m.CPUCores > 0 {
|
||||
out += fmt.Sprintf(" %dc", m.CPUCores)
|
||||
}
|
||||
if m.MeasuredMemoryGB > 0 {
|
||||
out += fmt.Sprintf(" (%.1fG used)", m.MeasuredMemoryGB)
|
||||
}
|
||||
if len(m.Warnings) > 0 {
|
||||
out += " !"
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -172,6 +172,10 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(out, "%d node(s) upgraded to %s, %d already on it\n", done, v, skipped)
|
||||
// The one machine this does not reach. Said every time, because it is
|
||||
// exactly the kind of thing that stays three versions behind unnoticed.
|
||||
fmt.Fprintf(out, "The model mirror's host is not upgraded by this; on it, run:\n"+
|
||||
" sudo llmgw upgrade %s --service llmgw-mirror\n", v)
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
+73
-1
@@ -24,12 +24,14 @@ import (
|
||||
// minutes of outage. For a fixed production set that is a reasonable trade; for
|
||||
// a platform whose promise is that people can try things, it is the difference
|
||||
// between trying one and not bothering.
|
||||
func newModelCmd(client func() *ctlClient) *cobra.Command {
|
||||
func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: "model",
|
||||
Short: "Add or remove models while the cluster is running",
|
||||
}
|
||||
c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
|
||||
newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
|
||||
newTargetCmd(client, asJSON),
|
||||
newModelServeCmd(client))
|
||||
return c
|
||||
}
|
||||
@@ -192,3 +194,73 @@ func newScaleCmd(client func() *ctlClient) *cobra.Command {
|
||||
"how long an on-demand model stays up unasked-for; zero takes the cluster default")
|
||||
return c
|
||||
}
|
||||
|
||||
func newResourcesCmd(client func() *ctlClient) *cobra.Command {
|
||||
var memory, cores int
|
||||
var profile string
|
||||
c := &cobra.Command{
|
||||
Use: "resources <model> --memory GB [--cores N]",
|
||||
Short: "Set what a model reserves of a node: memory, and CPU cores",
|
||||
Long: "--memory is the host RAM a replica needs: for a model with no GPU, its\n" +
|
||||
"weights, KV cache and engine together. Placement reserves it on the node,\n" +
|
||||
"so a node holds only as many replicas as it has memory for.\n\n" +
|
||||
"--cores is the CPU cores a replica with no GPU computes on. They are\n" +
|
||||
"reserved the same way, and become the engine's thread count, so two\n" +
|
||||
"replicas on one node stop competing for the same cores. Zero reserves\n" +
|
||||
"none and leaves the thread count to the node.\n\n" +
|
||||
"Running replicas keep what they were started with; `llmgw ctl restart\n" +
|
||||
"--model M` moves them to the new figures.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if !cmd.Flags().Changed("memory") {
|
||||
return fmt.Errorf("--memory is required")
|
||||
}
|
||||
body, err := json.Marshal(adminapi.ResourcesRequest{Profile: profile, MemoryGB: memory, CPUCores: cores})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var m catalog.Model
|
||||
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/resources", body, &m); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s/%s: %d GB memory, %d cores\n", m.Name, p.Name, p.MinMemoryGB, p.CPUCores)
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "Running replicas keep their old figures until restarted: llmgw ctl restart --model %s\n", m.Name)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().IntVar(&memory, "memory", 0, "host RAM a replica needs, in GB")
|
||||
c.Flags().IntVar(&cores, "cores", 0, "CPU cores a replica with no GPU computes on; 0 reserves none")
|
||||
c.Flags().StringVar(&profile, "profile", "", "one profile; every profile when left out")
|
||||
return c
|
||||
}
|
||||
|
||||
func newClassCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "class <model> critical|standard|opportunistic|default",
|
||||
Short: "Set how much a model's capacity is worth against another's",
|
||||
Long: "When a model cannot be placed, replicas of a lower class are stopped to\n" +
|
||||
"make room for it:\n\n" +
|
||||
" critical never gives up the replicas of its minimum\n" +
|
||||
" standard gives way to critical\n" +
|
||||
" opportunistic gives way to both; the default for an on-demand model\n\n" +
|
||||
"Within a class, a model below its minimum may take what another grew into\n" +
|
||||
"under load, but not its minimum. \"default\" goes back to what the model's\n" +
|
||||
"scaling implies: opportunistic when on demand, standard otherwise.",
|
||||
Args: cobra.ExactArgs(2),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
class := args[1]
|
||||
if class == "default" {
|
||||
class = ""
|
||||
}
|
||||
body, _ := json.Marshal(map[string]string{"class": class})
|
||||
var res map[string]string
|
||||
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/class", body, &res); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s is %s\n", res["model"], res["class"])
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -75,6 +75,7 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
|
||||
f.IntVar(&req.Replicas, "replicas", 1, "how many replicas to run")
|
||||
f.BoolVar(&onDemand, "on-demand", false, "run none until first asked for")
|
||||
f.IntVar(&req.CtxPool, "ctx-pool", 16384, "llama.cpp's --ctx-size, shared by its parallel slots")
|
||||
f.IntVar(&req.CPUCores, "cores", 0, "CPU cores a replica computes on; 0 takes half the smallest node it fits")
|
||||
f.IntVar(&req.ContextLength, "context-length", 4096, "what one request may use")
|
||||
return c
|
||||
}
|
||||
@@ -0,0 +1,234 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func newQuotaCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: "quota",
|
||||
Short: "What parts of the tenancy tree are guaranteed of the cluster, and may hold",
|
||||
Long: "A capacity quota belongs to an organization, team or project, and counts the\n" +
|
||||
"GPUs, CPU cores and memory held by replicas of the models it owns (see\n" +
|
||||
"`llmgw ctl model owner`), and of those its teams and projects own.\n\n" +
|
||||
"The guarantee is kept for it: when it holds less and a model of its needs\n" +
|
||||
"room, replicas other scopes hold beyond their own guarantees are stopped to\n" +
|
||||
"make it — whatever their class. What it holds within its guarantee is\n" +
|
||||
"never taken by another scope's model. Beyond it, it borrows: free room is\n" +
|
||||
"used, and given back when asked for.\n\n" +
|
||||
"The limit is the most it may hold, borrowing included; a model that would\n" +
|
||||
"take it past is not given another replica.\n\n" +
|
||||
"Scopes are written org:Acme, team:Acme/ml or project:Acme/ml/training, or\n" +
|
||||
"with IDs. Changes take effect within half a minute.",
|
||||
}
|
||||
c.AddCommand(newQuotaListCmd(client, asJSON), newQuotaSetCmd(client), newQuotaClearCmd(client))
|
||||
return c
|
||||
}
|
||||
|
||||
func newQuotaListCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "list",
|
||||
Short: "Every quota, with what its scope holds now",
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var qs []adminapi.QuotaView
|
||||
if err := client().get(cmd.Context(), "/api/v1/quotas", &qs); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return json.NewEncoder(cmd.OutOrStdout()).Encode(qs)
|
||||
}
|
||||
if len(qs) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "No capacity quotas: everything shares the cluster by class alone.")
|
||||
return nil
|
||||
}
|
||||
w := tab()
|
||||
fmt.Fprintln(w, "SCOPE\tNAME\tGPUS\tCORES\tMEMORY GB")
|
||||
for _, q := range qs {
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", q.Scope, q.Name,
|
||||
quotaCell(q.Holds.GPUs, q.Guarantee.GPUs, q.Limit.GPUs),
|
||||
quotaCell(q.Holds.CPUCores, q.Guarantee.CPUCores, q.Limit.CPUCores),
|
||||
quotaCell(q.Holds.MemoryGB, q.Guarantee.MemoryGB, q.Limit.MemoryGB))
|
||||
}
|
||||
if err := w.Flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "\nEach cell: holds / guaranteed (at most limit). - is not governed.")
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// quotaCell is "6 / 8 (≤ 16)": what is held, what is guaranteed, the limit.
|
||||
func quotaCell(holds int, guarantee, limit *int) string {
|
||||
if guarantee == nil && limit == nil {
|
||||
return "-"
|
||||
}
|
||||
out := strconv.Itoa(holds) + " / "
|
||||
if guarantee != nil {
|
||||
out += strconv.Itoa(*guarantee)
|
||||
} else {
|
||||
out += "-"
|
||||
}
|
||||
if limit != nil {
|
||||
out += fmt.Sprintf(" (≤ %d)", *limit)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func newQuotaSetCmd(client func() *ctlClient) *cobra.Command {
|
||||
var gpus, cores, mem, lgpus, lcores, lmem int
|
||||
c := &cobra.Command{
|
||||
Use: "set SCOPE [--gpus N] [--cores N] [--memory GB] [--limit-gpus N] ...",
|
||||
Short: "Set a scope's guarantee and limit, replacing what it had",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
f := cmd.Flags()
|
||||
pick := func(name string, v int) *int {
|
||||
if !f.Changed(name) {
|
||||
return nil
|
||||
}
|
||||
return &v
|
||||
}
|
||||
req := adminapi.QuotaRequest{Scope: args[0],
|
||||
Guarantee: store.Resources{GPUs: pick("gpus", gpus), CPUCores: pick("cores", cores), MemoryGB: pick("memory", mem)},
|
||||
Limit: store.Resources{GPUs: pick("limit-gpus", lgpus), CPUCores: pick("limit-cores", lcores), MemoryGB: pick("limit-memory", lmem)},
|
||||
}
|
||||
body, _ := json.Marshal(req)
|
||||
var res map[string]string
|
||||
if err := client().put(cmd.Context(), "/api/v1/quotas", body, &res); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: set; it takes effect within half a minute\n", res["scope"])
|
||||
return nil
|
||||
},
|
||||
}
|
||||
f := c.Flags()
|
||||
f.IntVar(&gpus, "gpus", 0, "GPUs guaranteed")
|
||||
f.IntVar(&cores, "cores", 0, "CPU cores guaranteed")
|
||||
f.IntVar(&mem, "memory", 0, "memory guaranteed, in GB")
|
||||
f.IntVar(&lgpus, "limit-gpus", 0, "the most GPUs it may hold")
|
||||
f.IntVar(&lcores, "limit-cores", 0, "the most CPU cores it may hold")
|
||||
f.IntVar(&lmem, "limit-memory", 0, "the most memory it may hold, in GB")
|
||||
return c
|
||||
}
|
||||
|
||||
func newQuotaClearCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "clear SCOPE",
|
||||
Short: "Remove a scope's quota",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var res map[string]string
|
||||
if err := client().delete(cmd.Context(), "/api/v1/quotas/"+url.PathEscape(args[0]), &res); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: quota removed\n", res["removed"])
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newOwnerCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "owner <model> SCOPE|shared",
|
||||
Short: "Set whose capacity quota a model counts against",
|
||||
Long: "SCOPE is org:Acme, team:Acme/ml or project:Acme/ml/training. A shared\n" +
|
||||
"model belongs to the cluster and counts against nobody's quota.",
|
||||
Args: cobra.ExactArgs(2),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
owner := args[1]
|
||||
if owner == "shared" {
|
||||
owner = ""
|
||||
}
|
||||
body, _ := json.Marshal(map[string]string{"owner": owner})
|
||||
var res map[string]string
|
||||
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/owner", body, &res); err != nil {
|
||||
return err
|
||||
}
|
||||
if res["owner"] == "" {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s is shared\n", res["model"])
|
||||
} else {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s counts against %s\n", res["model"], res["owner"])
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newTargetCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
var tps float64
|
||||
var apply bool
|
||||
c := &cobra.Command{
|
||||
Use: "target <model> --tokens-per-sec N [--apply]",
|
||||
Short: "How many replicas a model needs for a throughput, and whether they fit",
|
||||
Long: "Sized from what one replica was measured to serve while meeting its SLO\n" +
|
||||
"(`llmgw ctl bench MODEL --save`), on the profile its replicas run on.\n" +
|
||||
"--apply makes that the number it always runs, if the cluster has room.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if tps <= 0 {
|
||||
return fmt.Errorf("--tokens-per-sec is required")
|
||||
}
|
||||
var v adminapi.TargetView
|
||||
path := "/api/v1/models/" + url.PathEscape(args[0]) + "/target?tokens_per_sec=" +
|
||||
strconv.FormatFloat(tps, 'f', -1, 64)
|
||||
if err := client().get(cmd.Context(), path, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return json.NewEncoder(cmd.OutOrStdout()).Encode(v)
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
fmt.Fprintf(out, "%s: %.0f tokens/s needs %d replicas at %.1f each (measured on %s); %d running\n",
|
||||
v.Model, v.TokPerSec, v.Needed, v.PerReplica, v.Profile, v.Current)
|
||||
switch {
|
||||
case v.More == 0:
|
||||
fmt.Fprintln(out, "It already has them.")
|
||||
case v.Fits && len(v.Preempts) > 0:
|
||||
fmt.Fprintf(out, "%d more; there is room once replicas of %s are stopped, which its class may do.\n",
|
||||
v.More, strings.Join(v.Preempts, ", "))
|
||||
case v.Fits:
|
||||
fmt.Fprintf(out, "%d more, and there is room for them.\n", v.More)
|
||||
default:
|
||||
fmt.Fprintf(out, "%d more, and there is room for %d:\n", v.More, v.Room)
|
||||
for _, b := range v.Blocked {
|
||||
fmt.Fprintln(out, " "+b)
|
||||
}
|
||||
}
|
||||
if !apply || v.More == 0 {
|
||||
return nil
|
||||
}
|
||||
if !v.Fits {
|
||||
return fmt.Errorf("not applied: the cluster has no room for them")
|
||||
}
|
||||
var models []adminapi.ModelView
|
||||
if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
|
||||
return err
|
||||
}
|
||||
top := v.Needed
|
||||
for _, m := range models {
|
||||
if m.Name == v.Model {
|
||||
top = max(top, m.Max)
|
||||
}
|
||||
}
|
||||
body, _ := json.Marshal(adminapi.ScaleRequest{Min: v.Needed, Max: top})
|
||||
if err := client().post(cmd.Context(), "/api/v1/models/"+url.PathEscape(v.Model)+"/scale", body, nil); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(out, "Applied: %s runs %d (up to %d under load).\n", v.Model, v.Needed, top)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().Float64Var(&tps, "tokens-per-sec", 0, "the throughput to size for")
|
||||
c.Flags().BoolVar(&apply, "apply", false, "set it as the number of replicas the model always runs")
|
||||
return c
|
||||
}
|
||||
@@ -47,6 +47,7 @@ func Root() *cobra.Command {
|
||||
newSimEngineCmd(),
|
||||
newDoctorCmd(),
|
||||
newNodeCmd(),
|
||||
newSelfUpgradeCmd(),
|
||||
)
|
||||
return root
|
||||
}
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"runtime"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
)
|
||||
|
||||
// newSelfUpgradeCmd upgrades this machine's llmgw on its own, for the machines
|
||||
// `llmgw ctl upgrade` does not reach: a mirror host, above all, whose service
|
||||
// runs unprivileged with the filesystem read-only and so cannot replace its own
|
||||
// binary the way an agent does.
|
||||
func newSelfUpgradeCmd() *cobra.Command {
|
||||
var releases, service string
|
||||
c := &cobra.Command{
|
||||
Use: "upgrade VERSION [--service UNIT]",
|
||||
Short: "Upgrade this machine's llmgw, and restart the service that runs it",
|
||||
Long: "For a machine the control plane does not upgrade — the model mirror's host.\n" +
|
||||
"The control plane and the nodes are upgraded with `llmgw ctl upgrade`.\n\n" +
|
||||
"Fetches the release for this platform, checks it against the release's\n" +
|
||||
"checksums, keeps the old binary beside the new one, and restarts --service,\n" +
|
||||
"waiting for it to be running again. Run it with sudo.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
v := args[0]
|
||||
out := cmd.OutOrStdout()
|
||||
if !release.Tag.MatchString(v) {
|
||||
return fmt.Errorf("%q is not a release version (v1.2.3 or v1.2.3-rc.4)", v)
|
||||
}
|
||||
if version.Version == v {
|
||||
fmt.Fprintf(out, "this machine already runs %s\n", v)
|
||||
return nil
|
||||
}
|
||||
if os.Geteuid() != 0 {
|
||||
return errors.New("replacing this machine's binary needs root: run it with sudo")
|
||||
}
|
||||
self, err := os.Executable()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
backup, err := release.Install(cmd.Context(), nil, releases, v, runtime.GOOS, runtime.GOARCH, self)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(out, "%s -> %s: installed, checksum verified; the old binary is %s\n",
|
||||
version.Version, v, backup)
|
||||
if service == "" {
|
||||
return nil
|
||||
}
|
||||
//nolint:gosec // the unit name is the operator's own flag
|
||||
if b, err := exec.CommandContext(cmd.Context(), "systemctl", "restart", service).CombinedOutput(); err != nil {
|
||||
return fmt.Errorf("restarting %s: %v: %s", service, err, strings.TrimSpace(string(b)))
|
||||
}
|
||||
// Running, and still running a few seconds on: a unit that starts
|
||||
// and dies on a bad flag is "active" for a moment first.
|
||||
deadline := time.Now().Add(30 * time.Second)
|
||||
steady := 0
|
||||
for time.Now().Before(deadline) {
|
||||
//nolint:gosec // as above
|
||||
b, _ := exec.CommandContext(cmd.Context(), "systemctl", "is-active", service).Output()
|
||||
if strings.TrimSpace(string(b)) == "active" {
|
||||
if steady++; steady >= 3 {
|
||||
fmt.Fprintf(out, "%s restarted and running\n", service)
|
||||
return nil
|
||||
}
|
||||
} else {
|
||||
steady = 0
|
||||
}
|
||||
time.Sleep(2 * time.Second)
|
||||
}
|
||||
return fmt.Errorf("%s is not running after the upgrade; see journalctl -u %s, and %s holds the previous binary",
|
||||
service, service, backup)
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(&releases, "releases", release.DefaultBase, "where release binaries are fetched from")
|
||||
c.Flags().StringVar(&service, "service", "", "the systemd unit to restart afterwards, e.g. llmgw-mirror")
|
||||
return c
|
||||
}
|
||||
@@ -226,6 +226,8 @@ func newServerCmd() *cobra.Command {
|
||||
// And what each group of the tenancy tree spent, which is
|
||||
// the other half of a quota being reviewable.
|
||||
api.SetUsage(st)
|
||||
// Capacity quotas, set and read over this API.
|
||||
api.SetQuotas(st)
|
||||
}
|
||||
api.Register(adminMux)
|
||||
|
||||
@@ -483,6 +485,12 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
|
||||
if st := h.Store(); st != nil {
|
||||
lead = control.NewLeadership(st, cfg.Control.LeaseTTL, log)
|
||||
ctl.SetLeadership(lead)
|
||||
// The tenancy tree's capacity quotas: what each part is
|
||||
// guaranteed and may hold.
|
||||
ctl.Reconciler().SetQuotas(control.QuotaPolicy(st, log))
|
||||
// And what each model was measured to serve, for scaling and for
|
||||
// choosing between profiles.
|
||||
ctl.Reconciler().SetMeasurements(control.Measurements(st, log))
|
||||
log.Info("competing to be the control plane that places work",
|
||||
"owner", lead.Owner())
|
||||
}
|
||||
|
||||
@@ -4,6 +4,8 @@ import (
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
func TestAClaimedActorIsOneShortName(t *testing.T) {
|
||||
@@ -22,3 +24,22 @@ func TestAClaimedActorIsOneShortName(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReservationThatPlacementCannotTrustIsSaid(t *testing.T) {
|
||||
cpu := func(mem int) catalog.Model {
|
||||
return catalog.Model{Name: "m", Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: mem}}}
|
||||
}
|
||||
if w := reservationWarnings(cpu(0), 0); len(w) != 1 || !strings.Contains(w[0], "declares no memory") {
|
||||
t.Errorf("no memory declared: %v", w)
|
||||
}
|
||||
if w := reservationWarnings(cpu(6), 8.2); len(w) != 1 || !strings.Contains(w[0], "uses 8.2 GB and reserves 6") {
|
||||
t.Errorf("under-declared: %v", w)
|
||||
}
|
||||
if w := reservationWarnings(cpu(6), 6.4); len(w) != 0 {
|
||||
t.Errorf("within a tenth is mapped weights, not a wrong figure: %v", w)
|
||||
}
|
||||
gpu := catalog.Model{Name: "g", Profiles: []catalog.Profile{{Name: "tp1", Engine: "vllm", GPUClass: "h100-80g"}}}
|
||||
if w := reservationWarnings(gpu, 3); len(w) != 0 {
|
||||
t.Errorf("a GPU model needs no host memory figure: %v", w)
|
||||
}
|
||||
}
|
||||
@@ -15,6 +15,7 @@ import (
|
||||
"io"
|
||||
"net/http"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
@@ -41,6 +42,7 @@ type API struct {
|
||||
models Models
|
||||
capacity Capacity
|
||||
usage Usage
|
||||
quotas Quotas
|
||||
ctl *control.Controller
|
||||
}
|
||||
|
||||
@@ -56,6 +58,28 @@ func (a *API) Register(mux *http.ServeMux) {
|
||||
mux.HandleFunc("POST /api/v1/models", a.addModel)
|
||||
mux.HandleFunc("DELETE /api/v1/models/{name}", a.removeModel)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
|
||||
a.registerQuotas(mux)
|
||||
mux.HandleFunc("GET /api/v1/models/{name}/target", a.target)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/class", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.models == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
Class string `json:"class"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
m, err := a.models.SetClass(r.Context(), r.PathValue("name"), req.Class, actorOf(r))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"model": m.Name, "class": m.EffectiveClass()})
|
||||
})
|
||||
mux.HandleFunc("PUT /api/v1/models/{name}/capacity", a.putCapacity)
|
||||
mux.HandleFunc("GET /api/v1/capacity", a.listCapacity)
|
||||
mux.HandleFunc("GET /api/v1/usage", a.usageByScope)
|
||||
@@ -91,6 +115,15 @@ func (a *API) Register(mux *http.ServeMux) {
|
||||
mux.HandleFunc("GET /api/v1/jobs/{id}", a.job)
|
||||
mux.HandleFunc("GET /api/v1/requests/{id}", a.request)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/retry", a.retryModel)
|
||||
mux.HandleFunc("GET /api/v1/models/{name}/headroom", func(w http.ResponseWriter, r *http.Request) {
|
||||
limit, _ := strconv.Atoi(r.URL.Query().Get("limit"))
|
||||
v, ok := a.Headroom(r.PathValue("name"), limit)
|
||||
if !ok {
|
||||
http.Error(w, "no such model", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
})
|
||||
}
|
||||
|
||||
// --- responses ---
|
||||
@@ -109,6 +142,8 @@ type NodeView struct {
|
||||
// what decides whether a CPU profile can be placed here, so it belongs
|
||||
// where someone looks when a model will not schedule.
|
||||
MemoryGB int `json:"memory_gb,omitempty"`
|
||||
// CPUCount is the machine's CPUs, 0 when the agent did not report them.
|
||||
CPUCount int `json:"cpu_count,omitempty"`
|
||||
// Engines are the inference engines installed here, as the agent found
|
||||
// them. Absent means the agent is older than the probe and said nothing,
|
||||
// which is not the same as having none — placement treats it that way too,
|
||||
@@ -214,6 +249,11 @@ type LoadView struct {
|
||||
// MemPercent is used over used plus available.
|
||||
MemPercent float64 `json:"memory_percent"`
|
||||
AgeSec int64 `json:"age_seconds"`
|
||||
// What the node's sensors say; zero is not measured.
|
||||
CPUTempC float64 `json:"cpu_temp_c,omitempty"`
|
||||
CPUPowerW float64 `json:"cpu_power_w,omitempty"`
|
||||
InletTempC float64 `json:"inlet_temp_c,omitempty"`
|
||||
GPUs []inventory.GPUSensor `json:"gpu_sensors,omitempty"`
|
||||
}
|
||||
|
||||
// loadFresh is how old a measurement may be and still be shown as current.
|
||||
@@ -227,7 +267,8 @@ func loadView(l *inventory.Load, now time.Time) *LoadView {
|
||||
CPUPercent: l.CPUPercent, CPUs: l.CPUCount,
|
||||
Load1: l.Load1, Load5: l.Load5, Load15: l.Load15,
|
||||
MemUsed: l.MemUsedBytes, MemTotal: l.MemUsedBytes + l.MemAvailBytes,
|
||||
AgeSec: int64(now.Sub(l.At).Seconds()),
|
||||
AgeSec: int64(now.Sub(l.At).Seconds()),
|
||||
CPUTempC: l.CPUTempC, CPUPowerW: l.CPUPowerW, InletTempC: l.InletTempC, GPUs: l.GPUs,
|
||||
}
|
||||
if v.MemTotal > 0 {
|
||||
v.MemPercent = float64(v.MemUsed) * 100 / float64(v.MemTotal)
|
||||
@@ -260,6 +301,21 @@ type ModelView struct {
|
||||
Unplaceable bool `json:"unplaceable,omitempty"`
|
||||
Fallbacks []string `json:"fallbacks,omitempty"`
|
||||
Profiles []string `json:"profiles"`
|
||||
// MemoryGB and CPUCores are what one replica reserves, from its first
|
||||
// profile; MeasuredMemoryGB is the most one of its replicas uses now, 0
|
||||
// when none is measured.
|
||||
MemoryGB int `json:"memory_gb,omitempty"`
|
||||
CPUCores int `json:"cpu_cores,omitempty"`
|
||||
MeasuredMemoryGB float64 `json:"measured_memory_gb,omitempty"`
|
||||
// Warnings are what makes its reservation wrong: nothing declared where
|
||||
// placement needs a figure, or less declared than it uses.
|
||||
Warnings []string `json:"warnings,omitempty"`
|
||||
// Class is its capacity class; Preempted the replicas it lost to a higher
|
||||
// class; HeldFor the model it is giving room to now, if any.
|
||||
Class string `json:"class"`
|
||||
Owner string `json:"owner,omitempty"`
|
||||
Preempted int `json:"preempted,omitempty"`
|
||||
HeldFor string `json:"held_for,omitempty"`
|
||||
}
|
||||
|
||||
// ReplicaView is a replica as `ctl replicas` shows it.
|
||||
@@ -336,6 +392,28 @@ type ExplainView struct {
|
||||
Rejections []RejectionView `json:"rejections,omitempty"`
|
||||
}
|
||||
|
||||
// HeadroomView is how many more replicas of a model fit now, and where.
|
||||
type HeadroomView struct {
|
||||
Model string `json:"model"`
|
||||
// Current is the replicas holding capacity: starting or ready.
|
||||
Current int `json:"current"`
|
||||
// More is how many more placement would find room for now.
|
||||
More int `json:"more"`
|
||||
// Nodes is where each of those would go.
|
||||
Nodes [][]string `json:"nodes,omitempty"`
|
||||
// Limited is set when More stopped at the limit asked rather than at the
|
||||
// cluster's edge: there may be room for more.
|
||||
Limited bool `json:"limited,omitempty"`
|
||||
// Blocked says why one more does not fit, a line per node considered.
|
||||
Blocked []string `json:"blocked,omitempty"`
|
||||
// TokPerSec is what one more replica was measured to serve; 0 unmeasured.
|
||||
TokPerSec float64 `json:"tok_per_sec,omitempty"`
|
||||
// ByPreempting is how many more would fit by stopping replicas this
|
||||
// model's class may take; Preempts the models those belong to.
|
||||
ByPreempting int `json:"by_preempting,omitempty"`
|
||||
Preempts []string `json:"preempts,omitempty"`
|
||||
}
|
||||
|
||||
// RejectionView is one rejected candidate.
|
||||
type RejectionView struct {
|
||||
Profile string `json:"profile"`
|
||||
@@ -370,8 +448,8 @@ func (a *API) Nodes() []NodeView {
|
||||
ID: n.ID, Hostname: n.Hostname, MgmtAddr: n.MgmtAddr,
|
||||
State: n.State.String(), Labels: n.Labels,
|
||||
GPUTotal: len(n.GPUs), GPUFree: len(inv.FreeGPUs(n.ID)),
|
||||
MemoryGB: n.MemoryTotalMB / 1024,
|
||||
Fabrics: n.FabricNames(), AgentVersion: n.AgentVersion, Features: n.Features,
|
||||
MemoryGB: n.MemoryTotalMB / 1024, CPUCount: n.CPUCount,
|
||||
Fabrics: n.FabricNames(), AgentVersion: n.AgentVersion, Features: n.Features,
|
||||
Replicas: replicasOn[n.ID],
|
||||
}
|
||||
if free, total := n.LocalDisk(); total > 0 {
|
||||
@@ -471,6 +549,15 @@ func (a *API) Models() []ModelView {
|
||||
for _, h := range a.ctl.Reconciler().ModelHealth() {
|
||||
health[h.Model] = h
|
||||
}
|
||||
// The most one replica of each model uses, summed over its shards.
|
||||
measured := map[string]int64{}
|
||||
for _, rep := range a.Replicas("") {
|
||||
var b int64
|
||||
for _, sh := range rep.Shards {
|
||||
b += sh.MemoryBytes
|
||||
}
|
||||
measured[rep.Model] = max(measured[rep.Model], b)
|
||||
}
|
||||
out := make([]ModelView, 0)
|
||||
for _, m := range a.ctl.Catalog().Models() {
|
||||
h := health[m.Name]
|
||||
@@ -481,10 +568,16 @@ func (a *API) Models() []ModelView {
|
||||
Fallbacks: m.Fallbacks,
|
||||
Desired: h.Desired, Failures: h.Failures,
|
||||
Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
|
||||
Class: m.EffectiveClass(), Owner: m.Owner, Preempted: h.Preempted, HeldFor: h.HeldFor,
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
v.Profiles = append(v.Profiles, p.Name)
|
||||
}
|
||||
if len(m.Profiles) > 0 {
|
||||
v.MemoryGB, v.CPUCores = m.Profiles[0].MinMemoryGB, m.Profiles[0].CPUCores
|
||||
}
|
||||
v.MeasuredMemoryGB = float64(measured[m.Name]) / (1 << 30)
|
||||
v.Warnings = reservationWarnings(m, v.MeasuredMemoryGB)
|
||||
out = append(out, v)
|
||||
}
|
||||
return out
|
||||
@@ -555,6 +648,17 @@ type Models interface {
|
||||
Add(ctx context.Context, m catalog.Model, by string) error
|
||||
Remove(ctx context.Context, name string) error
|
||||
Scale(ctx context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error)
|
||||
Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
|
||||
SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
|
||||
SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
|
||||
}
|
||||
|
||||
// ResourcesRequest sets what a model's profiles reserve of a node.
|
||||
type ResourcesRequest struct {
|
||||
// Profile is one profile's name; empty is every profile.
|
||||
Profile string `json:"profile,omitempty"`
|
||||
MemoryGB int `json:"memory_gb"`
|
||||
CPUCores int `json:"cpu_cores"`
|
||||
}
|
||||
|
||||
// Usage reports what each group of the tenancy tree spent. The store
|
||||
@@ -800,6 +904,26 @@ func (a *API) scaleModel(w http.ResponseWriter, r *http.Request) {
|
||||
})
|
||||
}
|
||||
|
||||
// setResources sets a model's reserved memory and cores. Placed replicas keep
|
||||
// what they were started with until they are restarted.
|
||||
func (a *API) setResources(w http.ResponseWriter, r *http.Request) {
|
||||
if a.models == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
var req ResourcesRequest
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
m, err := a.models.Resources(r.Context(), r.PathValue("name"), req.Profile, req.MemoryGB, req.CPUCores, actorOf(r))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, m)
|
||||
}
|
||||
|
||||
func (a *API) removeModel(w http.ResponseWriter, r *http.Request) {
|
||||
if a.models == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
@@ -836,6 +960,21 @@ func (a *API) explainPlacement(w http.ResponseWriter, r *http.Request) {
|
||||
// It does not check that the model exists; the handler above does that so it
|
||||
// can answer 404, while the dashboard only ever asks about models it just read
|
||||
// out of the catalog.
|
||||
// Headroom tries placing up to limit more replicas of a model, on a copy of
|
||||
// the inventory, and says how many fit. A limit of zero or less takes 16.
|
||||
func (a *API) Headroom(model string, limit int) (HeadroomView, bool) {
|
||||
if limit <= 0 || limit > 64 {
|
||||
limit = 16
|
||||
}
|
||||
h, ok := a.ctl.Reconciler().Headroom(model, limit)
|
||||
if !ok {
|
||||
return HeadroomView{}, false
|
||||
}
|
||||
return HeadroomView{Model: model, Current: h.Current, More: h.More, Nodes: h.Nodes,
|
||||
Limited: h.Limited, Blocked: h.Blocked, TokPerSec: h.TokPerSec,
|
||||
ByPreempting: h.ByPreempting, Preempts: h.Preempts}, true
|
||||
}
|
||||
|
||||
func (a *API) Explain(model string) ExplainView {
|
||||
v := ExplainView{Model: model}
|
||||
for _, rep := range a.ctl.Reconciler().Replicas() {
|
||||
@@ -1267,3 +1406,40 @@ func actorOf(r *http.Request) string {
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// NodeCPUs is the CPU counts of ready nodes with at least this much memory:
|
||||
// where a replica needing it could go.
|
||||
func (a *API) NodeCPUs(minMemoryGB int) []int {
|
||||
var out []int
|
||||
for _, n := range a.ctl.Inventory().Nodes() {
|
||||
if n.State.String() != "ready" || n.CPUCount == 0 {
|
||||
continue
|
||||
}
|
||||
if n.MemoryTotalMB > 0 && n.MemoryTotalMB/1024 < minMemoryGB {
|
||||
continue
|
||||
}
|
||||
out = append(out, n.CPUCount)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// reservationWarnings says what is wrong with what a model reserves.
|
||||
//
|
||||
// Resident memory includes the weights mapped from disk, which the kernel can
|
||||
// take back, so it overstates what a model needs by the part of the weights
|
||||
// not in use. A tenth over is let pass for that; more is a figure too low for
|
||||
// placement to be trusted with.
|
||||
func reservationWarnings(m catalog.Model, measuredGB float64) []string {
|
||||
var out []string
|
||||
for _, p := range m.Profiles {
|
||||
if p.CPUOnly() && p.MinMemoryGB <= 0 {
|
||||
out = append(out, fmt.Sprintf("profile %s declares no memory: placement cannot tell when a node is full "+
|
||||
"(llmgw ctl model resources %s --memory GB)", p.Name, m.Name))
|
||||
}
|
||||
}
|
||||
if len(m.Profiles) == 1 && m.Profiles[0].MinMemoryGB > 0 && measuredGB > float64(m.Profiles[0].MinMemoryGB)*1.1 {
|
||||
out = append(out, fmt.Sprintf("uses %.1f GB and reserves %d: raise it, or nodes will be filled past their memory",
|
||||
measuredGB, m.Profiles[0].MinMemoryGB))
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -258,6 +258,21 @@ type fakeModels struct {
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeModels) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
|
||||
return catalog.Model{Name: name, Owner: owner}, f.err
|
||||
}
|
||||
|
||||
func (f *fakeModels) SetClass(_ context.Context, name, class, _ string) (catalog.Model, error) {
|
||||
return catalog.Model{Name: name, Class: class}, f.err
|
||||
}
|
||||
|
||||
func (f *fakeModels) Resources(_ context.Context, name, profile string, mem, cores int, _ string) (catalog.Model, error) {
|
||||
if f.err != nil {
|
||||
return catalog.Model{}, f.err
|
||||
}
|
||||
return catalog.Model{Name: name, Profiles: []catalog.Profile{{Name: "p", MinMemoryGB: mem, CPUCores: cores}}}, nil
|
||||
}
|
||||
|
||||
func (f *fakeModels) Scale(_ context.Context, name string, s catalog.Scaling, by string) (catalog.Model, error) {
|
||||
if f.err != nil {
|
||||
return catalog.Model{}, f.err
|
||||
|
||||
@@ -0,0 +1,285 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Quotas is where capacity quotas and the tenancy tree they name are kept.
|
||||
// The store implements it.
|
||||
type Quotas interface {
|
||||
CapacityQuotas(ctx context.Context) ([]store.CapacityQuota, error)
|
||||
PutCapacityQuota(ctx context.Context, q store.CapacityQuota) error
|
||||
DeleteCapacityQuota(ctx context.Context, scope store.Scope, id string) error
|
||||
ScopeParents(ctx context.Context) (map[string][]string, error)
|
||||
AllOrgs(ctx context.Context) ([]store.Org, error)
|
||||
AllTeams(ctx context.Context) ([]store.TeamPath, error)
|
||||
AllProjects(ctx context.Context) ([]store.ProjectPath, error)
|
||||
}
|
||||
|
||||
// SetQuotas supplies the quota store, which mounts the quota endpoints'
|
||||
// behaviour; without it they answer 501.
|
||||
func (a *API) SetQuotas(q Quotas) { a.quotas = q }
|
||||
|
||||
// QuotaView is one scope's quota with what it holds now.
|
||||
type QuotaView struct {
|
||||
// Scope is team:ID; Name is how people know it, "Acme / ml".
|
||||
Scope string `json:"scope"`
|
||||
Name string `json:"name"`
|
||||
Guarantee store.Resources `json:"guarantee"`
|
||||
Limit store.Resources `json:"limit"`
|
||||
Holds quota.Amount `json:"holds"`
|
||||
UpdatedBy string `json:"updated_by,omitempty"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
}
|
||||
|
||||
// QuotaRequest sets a scope's quota. Scope may be written with an ID or a
|
||||
// name: org:Acme, team:Acme/ml, project:Acme/ml/training.
|
||||
type QuotaRequest struct {
|
||||
Scope string `json:"scope"`
|
||||
Guarantee store.Resources `json:"guarantee"`
|
||||
Limit store.Resources `json:"limit"`
|
||||
}
|
||||
|
||||
func (a *API) registerQuotas(mux *http.ServeMux) {
|
||||
mux.HandleFunc("GET /api/v1/quotas", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.quotas == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
views, err := a.QuotaViews(r.Context())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, views)
|
||||
})
|
||||
mux.HandleFunc("PUT /api/v1/quotas", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.quotas == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
var req QuotaRequest
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<14)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
key, err := a.SetQuota(r.Context(), req, actorOf(r))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"scope": key})
|
||||
})
|
||||
mux.HandleFunc("DELETE /api/v1/quotas/{scope}", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.quotas == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
key, err := a.ClearQuota(r.Context(), r.PathValue("scope"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"removed": key})
|
||||
})
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/owner", func(w http.ResponseWriter, r *http.Request) {
|
||||
if a.models == nil || a.quotas == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
Owner string `json:"owner"`
|
||||
}
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
owner := ""
|
||||
if req.Owner != "" {
|
||||
scope, id, _, err := a.resolveScope(r.Context(), req.Owner)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
owner = quota.Key(scope, id)
|
||||
}
|
||||
m, err := a.models.SetOwner(r.Context(), r.PathValue("name"), owner, actorOf(r))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"model": m.Name, "owner": m.Owner})
|
||||
})
|
||||
}
|
||||
|
||||
// QuotaViews lists every quota with what its scope holds now.
|
||||
func (a *API) QuotaViews(ctx context.Context) ([]QuotaView, error) {
|
||||
if a.quotas == nil {
|
||||
return nil, nil
|
||||
}
|
||||
qs, err := a.quotas.CapacityQuotas(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
parents, err := a.quotas.ScopeParents(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
names, err := a.scopeNames(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
usage := quota.New(qs, parents).Usage(a.ctl.Reconciler().Holdings())
|
||||
out := make([]QuotaView, 0, len(qs))
|
||||
for _, q := range qs {
|
||||
key := quota.Key(q.Scope, q.ScopeID)
|
||||
out = append(out, QuotaView{Scope: key, Name: names[key], Guarantee: q.Guarantee, Limit: q.Limit,
|
||||
Holds: usage[key], UpdatedBy: q.UpdatedBy, UpdatedAt: q.UpdatedAt})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// scopeNames maps org:ID, team:ID and project:ID to their paths.
|
||||
func (a *API) scopeNames(ctx context.Context) (map[string]string, error) {
|
||||
out := map[string]string{}
|
||||
orgs, err := a.quotas.AllOrgs(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, o := range orgs {
|
||||
out["org:"+o.ID] = o.Name
|
||||
}
|
||||
teams, err := a.quotas.AllTeams(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, t := range teams {
|
||||
out["team:"+t.ID] = t.OrgName + " / " + t.Name
|
||||
}
|
||||
projects, err := a.quotas.AllProjects(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, p := range projects {
|
||||
out["project:"+p.ID] = p.Path
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// resolveScope reads org:X, team:X or project:X, where X is an ID or the
|
||||
// scope's path — Acme, Acme/ml, Acme/ml/training — and returns its ID and
|
||||
// path. A path that could mean two scopes is refused rather than guessed.
|
||||
func (a *API) resolveScope(ctx context.Context, s string) (store.Scope, string, string, error) {
|
||||
kind, what, ok := strings.Cut(strings.TrimSpace(s), ":")
|
||||
scope := store.Scope(kind)
|
||||
if !ok || what == "" || (scope != store.ScopeOrg && scope != store.ScopeTeam && scope != store.ScopeProject) {
|
||||
return "", "", "", fmt.Errorf("%q is not org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT", s)
|
||||
}
|
||||
names, err := a.scopeNames(ctx)
|
||||
if err != nil {
|
||||
return "", "", "", err
|
||||
}
|
||||
norm := func(p string) string {
|
||||
parts := strings.Split(p, "/")
|
||||
for i := range parts {
|
||||
parts[i] = strings.ToLower(strings.TrimSpace(parts[i]))
|
||||
}
|
||||
return strings.Join(parts, "/")
|
||||
}
|
||||
if name, ok := names[kind+":"+what]; ok {
|
||||
return scope, what, name, nil
|
||||
}
|
||||
match := func(same func(name string) bool) []string {
|
||||
var found []string
|
||||
for key, name := range names {
|
||||
if strings.HasPrefix(key, kind+":") && same(name) {
|
||||
found = append(found, key)
|
||||
}
|
||||
}
|
||||
return found
|
||||
}
|
||||
found := match(func(name string) bool { return norm(name) == norm(what) })
|
||||
if len(found) == 0 {
|
||||
// Its own name alone — ml rather than Acme/ml — when that is unique.
|
||||
found = match(func(name string) bool {
|
||||
parts := strings.Split(norm(name), "/")
|
||||
return parts[len(parts)-1] == norm(what)
|
||||
})
|
||||
}
|
||||
switch len(found) {
|
||||
case 0:
|
||||
return "", "", "", fmt.Errorf("there is no %s %q", kind, what)
|
||||
case 1:
|
||||
_, id, _ := strings.Cut(found[0], ":")
|
||||
return scope, id, names[found[0]], nil
|
||||
}
|
||||
return "", "", "", fmt.Errorf("%q could be any of %d %ss; give its ID", what, len(found), kind)
|
||||
}
|
||||
|
||||
// SetQuota sets a scope's capacity quota, named by path or ID, and returns
|
||||
// the scope as org:ID.
|
||||
func (a *API) SetQuota(ctx context.Context, req QuotaRequest, by string) (string, error) {
|
||||
if a.quotas == nil {
|
||||
return "", fmt.Errorf("this control plane has no store")
|
||||
}
|
||||
scope, id, _, err := a.resolveScope(ctx, req.Scope)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if !req.Guarantee.Any() && !req.Limit.Any() {
|
||||
return "", fmt.Errorf("a quota needs a guarantee or a limit; to remove one, clear it")
|
||||
}
|
||||
q := store.CapacityQuota{Scope: scope, ScopeID: id, Guarantee: req.Guarantee, Limit: req.Limit, UpdatedBy: by}
|
||||
if err := a.quotas.PutCapacityQuota(ctx, q); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return quota.Key(scope, id), nil
|
||||
}
|
||||
|
||||
// ClearQuota removes a scope's capacity quota.
|
||||
func (a *API) ClearQuota(ctx context.Context, scope string) (string, error) {
|
||||
if a.quotas == nil {
|
||||
return "", fmt.Errorf("this control plane has no store")
|
||||
}
|
||||
s, id, _, err := a.resolveScope(ctx, scope)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := a.quotas.DeleteCapacityQuota(ctx, s, id); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return quota.Key(s, id), nil
|
||||
}
|
||||
|
||||
// ScopeName is a part of the tenancy tree: org:ID and its path.
|
||||
type ScopeName struct {
|
||||
Key string `json:"scope"`
|
||||
Name string `json:"name"`
|
||||
}
|
||||
|
||||
// Scopes lists every organization, team and project, by path.
|
||||
func (a *API) Scopes(ctx context.Context) ([]ScopeName, error) {
|
||||
if a.quotas == nil {
|
||||
return nil, nil
|
||||
}
|
||||
names, err := a.scopeNames(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]ScopeName, 0, len(names))
|
||||
for k, n := range names {
|
||||
out = append(out, ScopeName{Key: k, Name: n})
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// fakeQuotaStore is Acme with teams ml and web, and Scania with another ml.
|
||||
type fakeQuotaStore struct {
|
||||
quotas []store.CapacityQuota
|
||||
}
|
||||
|
||||
func (f *fakeQuotaStore) CapacityQuotas(context.Context) ([]store.CapacityQuota, error) {
|
||||
return f.quotas, nil
|
||||
}
|
||||
func (f *fakeQuotaStore) PutCapacityQuota(_ context.Context, q store.CapacityQuota) error {
|
||||
for i := range f.quotas {
|
||||
if f.quotas[i].Scope == q.Scope && f.quotas[i].ScopeID == q.ScopeID {
|
||||
f.quotas[i] = q
|
||||
return nil
|
||||
}
|
||||
}
|
||||
f.quotas = append(f.quotas, q)
|
||||
return nil
|
||||
}
|
||||
func (f *fakeQuotaStore) DeleteCapacityQuota(_ context.Context, s store.Scope, id string) error {
|
||||
for i, q := range f.quotas {
|
||||
if q.Scope == s && q.ScopeID == id {
|
||||
f.quotas = append(f.quotas[:i], f.quotas[i+1:]...)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (f *fakeQuotaStore) ScopeParents(context.Context) (map[string][]string, error) {
|
||||
return map[string][]string{"team:t-ml": {"org:o-acme"}, "team:t-web": {"org:o-acme"},
|
||||
"team:t-sml": {"org:o-scania"}}, nil
|
||||
}
|
||||
func (f *fakeQuotaStore) AllOrgs(context.Context) ([]store.Org, error) {
|
||||
return []store.Org{{ID: "o-acme", Name: "Acme"}, {ID: "o-scania", Name: "Scania"}}, nil
|
||||
}
|
||||
func (f *fakeQuotaStore) AllTeams(context.Context) ([]store.TeamPath, error) {
|
||||
return []store.TeamPath{{ID: "t-ml", Name: "ml", OrgID: "o-acme", OrgName: "Acme"},
|
||||
{ID: "t-web", Name: "web", OrgID: "o-acme", OrgName: "Acme"},
|
||||
{ID: "t-sml", Name: "ml", OrgID: "o-scania", OrgName: "Scania"}}, nil
|
||||
}
|
||||
func (f *fakeQuotaStore) AllProjects(context.Context) ([]store.ProjectPath, error) { return nil, nil }
|
||||
|
||||
func TestAScopeIsNamedByPathOrIDAndNeverGuessed(t *testing.T) {
|
||||
a, _ := setup(t, testModels())
|
||||
a.SetQuotas(&fakeQuotaStore{})
|
||||
for in, want := range map[string]string{
|
||||
"org:Acme": "o-acme", "org:o-acme": "o-acme", "team:Acme/ml": "t-ml",
|
||||
"team: acme / ML ": "t-ml", "team:Scania/ml": "t-sml", "team:web": "t-web",
|
||||
} {
|
||||
_, id, _, err := a.resolveScope(t.Context(), in)
|
||||
if err != nil || id != want {
|
||||
t.Errorf("%q: %q %v, expected %q", in, id, err, want)
|
||||
}
|
||||
}
|
||||
for _, bad := range []string{"team:ml", "team:Acme/nope", "group:Acme", "Acme", "org:"} {
|
||||
if _, _, _, err := a.resolveScope(t.Context(), bad); err == nil {
|
||||
t.Errorf("%q was resolved", bad)
|
||||
}
|
||||
}
|
||||
if _, _, _, err := a.resolveScope(t.Context(), "team:ml"); err == nil || !strings.Contains(err.Error(), "could be any of 2") {
|
||||
t.Errorf("an ambiguous name should say so: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAQuotaIsSetListedAndCleared(t *testing.T) {
|
||||
a, _ := setup(t, testModels())
|
||||
qs := &fakeQuotaStore{}
|
||||
a.SetQuotas(qs)
|
||||
mux := http.NewServeMux()
|
||||
a.Register(mux)
|
||||
call := func(method, path, body string) *httptest.ResponseRecorder {
|
||||
req := httptest.NewRequest(method, path, strings.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("X-Llmgw-Actor", "ops")
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, req)
|
||||
return rec
|
||||
}
|
||||
if rec := call("PUT", "/api/v1/quotas", `{"scope":"team:Acme/ml","guarantee":{"gpus":8},"limit":{"gpus":16}}`); rec.Code != 200 {
|
||||
t.Fatalf("set: %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
if len(qs.quotas) != 1 || qs.quotas[0].ScopeID != "t-ml" || *qs.quotas[0].Guarantee.GPUs != 8 || qs.quotas[0].UpdatedBy != "ops" {
|
||||
t.Errorf("stored %+v", qs.quotas)
|
||||
}
|
||||
rec := call("GET", "/api/v1/quotas", "")
|
||||
var views []QuotaView
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &views); err != nil || len(views) != 1 || views[0].Name != "Acme / ml" {
|
||||
t.Errorf("list %d %s", rec.Code, rec.Body)
|
||||
}
|
||||
if rec := call("PUT", "/api/v1/quotas", `{"scope":"team:Acme/ml"}`); rec.Code != 400 {
|
||||
t.Errorf("an empty quota was accepted: %d", rec.Code)
|
||||
}
|
||||
if rec := call("DELETE", "/api/v1/quotas/team:Acme%2Fml", ""); rec.Code != 200 || len(qs.quotas) != 0 {
|
||||
t.Errorf("clear: %d %s %+v", rec.Code, rec.Body, qs.quotas)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"net/http"
|
||||
"strconv"
|
||||
)
|
||||
|
||||
// TargetView is how many replicas a model needs to serve a throughput, from
|
||||
// what one replica was measured to serve, and whether the cluster has room.
|
||||
type TargetView struct {
|
||||
Model string `json:"model"`
|
||||
// TokPerSec is what was asked for; PerReplica what one replica was
|
||||
// measured to serve at its SLO, on Profile.
|
||||
TokPerSec float64 `json:"tok_per_sec"`
|
||||
PerReplica float64 `json:"per_replica_tok_per_sec"`
|
||||
Profile string `json:"profile"`
|
||||
// Needed replicas in all; Current hold capacity now; More is how many of
|
||||
// Needed are not running, and Room how many more fit.
|
||||
Needed int `json:"needed"`
|
||||
Current int `json:"current"`
|
||||
More int `json:"more"`
|
||||
Room int `json:"room"`
|
||||
Fits bool `json:"fits"`
|
||||
Blocked []string `json:"blocked,omitempty"`
|
||||
// Preempts is whose replicas would be stopped to make the room, when
|
||||
// free room alone is not enough; Room counts that room too.
|
||||
Preempts []string `json:"preempts,omitempty"`
|
||||
}
|
||||
|
||||
// Target sizes a model for a throughput.
|
||||
func (a *API) Target(model string, tokPerSec float64) (TargetView, error) {
|
||||
meas, profile, ok := a.ctl.Reconciler().MeasuredFor(model)
|
||||
if !ok || meas.TokPerSec <= 0 {
|
||||
return TargetView{}, fmt.Errorf("%s has not been measured; run `llmgw ctl bench %s --save` first", model, model)
|
||||
}
|
||||
v := TargetView{Model: model, TokPerSec: tokPerSec, PerReplica: meas.TokPerSec, Profile: profile,
|
||||
Needed: max(int(math.Ceil(tokPerSec/meas.TokPerSec)), 1)}
|
||||
h, ok := a.Headroom(model, v.Needed)
|
||||
if !ok {
|
||||
return TargetView{}, fmt.Errorf("no model %q", model)
|
||||
}
|
||||
v.Current = h.Current
|
||||
v.More = max(v.Needed-h.Current, 0)
|
||||
v.Room = h.More + h.ByPreempting
|
||||
v.Fits = v.More <= v.Room || h.Limited
|
||||
if v.More > h.More {
|
||||
v.Preempts = h.Preempts
|
||||
}
|
||||
if !v.Fits {
|
||||
v.Blocked = h.Blocked
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
func (a *API) target(w http.ResponseWriter, r *http.Request) {
|
||||
tps, err := strconv.ParseFloat(r.URL.Query().Get("tokens_per_sec"), 64)
|
||||
if err != nil || tps <= 0 {
|
||||
writeError(w, http.StatusBadRequest, "tokens_per_sec must be a positive number")
|
||||
return
|
||||
}
|
||||
v, err := a.Target(r.PathValue("name"), tps)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
)
|
||||
|
||||
func TestATargetIsSizedFromTheMeasurementAndTheRoom(t *testing.T) {
|
||||
// Three 8-GPU nodes; one tp8 replica was measured at 500 tokens/s.
|
||||
a, ctl := setup(t, testModels(), gpuNode("1", 8), gpuNode("2", 8), gpuNode("3", 8))
|
||||
if _, err := a.Target("llama", 1000); err == nil || !strings.Contains(err.Error(), "ctl bench llama --save") {
|
||||
t.Errorf("an unmeasured model should say how to measure it: %v", err)
|
||||
}
|
||||
ctl.Reconciler().SetMeasurements(func() map[string]reconciler.Measured {
|
||||
return map[string]reconciler.Measured{"llama/tp8": {MaxConcurrency: 16, TokPerSec: 500}}
|
||||
})
|
||||
v, err := a.Target("llama", 1200)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v.Needed != 3 || v.PerReplica != 500 || v.Profile != "tp8" || v.More != 3 || !v.Fits {
|
||||
t.Errorf("1200 tok/s at 500 each is 3 replicas on 3 empty nodes: %+v", v)
|
||||
}
|
||||
v, _ = a.Target("llama", 2000)
|
||||
if v.Needed != 4 || v.Fits || v.Room != 3 || len(v.Blocked) == 0 {
|
||||
t.Errorf("a fourth does not fit on three nodes, and should say why: %+v", v)
|
||||
}
|
||||
}
|
||||
@@ -4,6 +4,7 @@ package agentsvc
|
||||
import (
|
||||
"fmt"
|
||||
"maps"
|
||||
"runtime"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
@@ -27,6 +28,7 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
|
||||
EngineVersions: r.GetEngineVersions(),
|
||||
EnginesProbed: r.GetEnginesProbed(),
|
||||
MemoryTotalMB: int(r.GetMemoryTotalMb()),
|
||||
CPUCount: int(r.GetCpuCount()),
|
||||
}
|
||||
if h := r.GetHost(); h != nil {
|
||||
n.Host = discovery.Host{
|
||||
@@ -90,6 +92,7 @@ func RegisterFromInventory(nodeID, hostname, mgmtAddr, agentVersion string,
|
||||
NodeId: nodeID, Hostname: hostname, MgmtAddr: mgmtAddr,
|
||||
Labels: labels, AgentVersion: agentVersion,
|
||||
MemoryTotalMb: int64(inv.MemoryTotalMB),
|
||||
CpuCount: int32(runtime.NumCPU()),
|
||||
}
|
||||
for _, w := range inv.Weights {
|
||||
r.Weights = append(r.Weights, &llmgwv1.LocalWeights{
|
||||
@@ -192,6 +195,7 @@ func StartShardFromCommand(cmd reconciler.ShardCommand) *llmgwv1.StartShard {
|
||||
Fabric: cmd.Fabric,
|
||||
ProfileFp: cmd.ProfileFP,
|
||||
ApiKey: cmd.Spec.APIKey,
|
||||
CpuThreads: int32(cmd.Spec.Threads),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -213,6 +217,7 @@ func ShardSpecFromStart(m *llmgwv1.StartShard) (engines.ShardSpec, time.Duration
|
||||
Fabric: m.GetFabric(),
|
||||
ProfileFP: m.GetProfileFp(),
|
||||
APIKey: m.GetApiKey(),
|
||||
Threads: int(m.GetCpuThreads()),
|
||||
Profile: engines.Profile{
|
||||
Name: p.GetName(), Engine: p.GetEngine(), GPUClass: p.GetGpuClass(),
|
||||
TP: int(p.GetTp()), PP: int(p.GetPp()),
|
||||
|
||||
@@ -345,8 +345,9 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
|
||||
problems := hb.GetNewProblems()
|
||||
storage := hb.GetStorage()
|
||||
memMB := int(hb.GetMemoryTotalMb())
|
||||
cpus := int(hb.GetCpuCount())
|
||||
if len(used) == 0 && len(problems) == 0 && len(storage) == 0 &&
|
||||
!hb.GetWeightsChanged() && memMB == 0 && !hb.GetEnginesChanged() &&
|
||||
!hb.GetWeightsChanged() && memMB == 0 && cpus == 0 && !hb.GetEnginesChanged() &&
|
||||
!hb.GetProvisionChanged() && hb.GetLoad() == nil && len(hb.GetIbPorts()) == 0 &&
|
||||
!hb.GetFabricsChanged() && len(hb.GetGpuFabric()) == 0 && hb.GetImex() == nil {
|
||||
return
|
||||
@@ -406,6 +407,12 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
|
||||
"from_mb", n.MemoryTotalMB, "to_mb", memMB)
|
||||
n.MemoryTotalMB = memMB
|
||||
}
|
||||
if cpus > 0 && cpus != n.CPUCount {
|
||||
if n.CPUCount > 0 {
|
||||
s.log.Info("node CPUs changed", "node", nodeID, "from", n.CPUCount, "to", cpus)
|
||||
}
|
||||
n.CPUCount = cpus
|
||||
}
|
||||
if hb.GetEnginesChanged() {
|
||||
vers := maps.Clone(hb.GetEngineVersions())
|
||||
if !maps.Equal(vers, n.EngineVersions) {
|
||||
@@ -538,6 +545,13 @@ func loadFrom(l *llmgwv1.NodeLoad, at time.Time) *inventory.Load {
|
||||
CPUCores: sh.GetCpuCores(), RSSBytes: sh.GetMemoryRssBytes(),
|
||||
}
|
||||
}
|
||||
out.CPUTempC, out.CPUPowerW, out.InletTempC = l.GetCpuTempC(), l.GetCpuPowerW(), l.GetInletTempC()
|
||||
for _, g := range l.GetGpus() {
|
||||
out.GPUs = append(out.GPUs, inventory.GPUSensor{
|
||||
Index: int(g.GetIndex()), TempC: g.GetTempC(), MemoryTempC: g.GetMemoryTempC(),
|
||||
PowerW: g.GetPowerW(), PowerLimitW: g.GetPowerLimitW(), Throttle: g.GetThrottle(),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
|
||||
@@ -74,6 +74,17 @@ type Profile struct {
|
||||
// the control plane.
|
||||
MinMemoryGB int `json:"min_memory_gb,omitempty" koanf:"min_memory_gb"`
|
||||
|
||||
// CPUCores is the cores a replica of a profile with no GPU is given:
|
||||
// reserved out of its node's CPUs by placement, as memory is, and what
|
||||
// its engine computes on (llama.cpp's --threads).
|
||||
//
|
||||
// Reserved because an engine takes every core it is told about, and two
|
||||
// on one node that each think they have all of them spin against each
|
||||
// other: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second
|
||||
// to under one each. Zero reserves nothing and leaves the thread count to
|
||||
// the node, which is every profile from before this.
|
||||
CPUCores int `json:"cpu_cores,omitempty" koanf:"cpu_cores"`
|
||||
|
||||
// MinDiskGB is the free space a node needs before it can hold these
|
||||
// weights.
|
||||
//
|
||||
@@ -138,10 +149,19 @@ type Model struct {
|
||||
// choice by guesswork.
|
||||
Description string `json:"description,omitempty" koanf:"description"`
|
||||
// Source is the weights path or a HuggingFace id.
|
||||
Source string `json:"source,omitempty" koanf:"source"`
|
||||
Profiles []Profile `json:"profiles,omitempty" koanf:"profiles"`
|
||||
Scaling Scaling `json:"scaling,omitempty" koanf:"scaling"`
|
||||
Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
|
||||
Source string `json:"source,omitempty" koanf:"source"`
|
||||
Profiles []Profile `json:"profiles,omitempty" koanf:"profiles"`
|
||||
Scaling Scaling `json:"scaling,omitempty" koanf:"scaling"`
|
||||
// Class is how much this model's capacity is worth against another's,
|
||||
// when there is not enough for both: critical, standard or opportunistic.
|
||||
// Empty is standard, or opportunistic for a model loaded on demand. See
|
||||
// EffectiveClass.
|
||||
Class string `json:"class,omitempty" koanf:"class"`
|
||||
// Owner is the part of the tenancy tree whose capacity quota this model
|
||||
// counts against, written org:ID, team:ID or project:ID. Empty is shared:
|
||||
// held by the cluster, counted against nobody.
|
||||
Owner string `json:"owner,omitempty" koanf:"owner"`
|
||||
Fallbacks []string `json:"fallbacks,omitempty" koanf:"fallbacks"`
|
||||
// RoutingProfile selects how the router scores endpoints for this model.
|
||||
RoutingProfile string `json:"routing_profile,omitempty" koanf:"routing_profile"`
|
||||
SLO SLO `json:"slo,omitempty" koanf:"slo"`
|
||||
@@ -288,6 +308,12 @@ func (m Model) validate() error {
|
||||
if len(m.Profiles) == 0 {
|
||||
return fmt.Errorf("model %q: at least one profile is required", m.Name)
|
||||
}
|
||||
if m.Owner != "" && !validOwner(m.Owner) {
|
||||
return fmt.Errorf("model %q: owner %q is not org:ID, team:ID or project:ID", m.Name, m.Owner)
|
||||
}
|
||||
if !validClass(m.Class) {
|
||||
return fmt.Errorf("model %q: class %q is not one of critical, standard, opportunistic", m.Name, m.Class)
|
||||
}
|
||||
if m.Scaling.Max < m.Scaling.Min {
|
||||
return fmt.Errorf("model %q: scaling max %d is below min %d", m.Name, m.Scaling.Max, m.Scaling.Min)
|
||||
}
|
||||
@@ -380,6 +406,7 @@ func (m Model) PlacementProfiles() []placement.Profile {
|
||||
NodeSelector: p.NodeSelector,
|
||||
MinMemoryGB: p.MinMemoryGB,
|
||||
MinDiskGB: p.MinDiskGB,
|
||||
CPUCores: p.CPUCores,
|
||||
// The model's source, so placement can prefer nodes that already
|
||||
// hold the weights rather than one that must first move them.
|
||||
Source: m.Source,
|
||||
@@ -475,3 +502,65 @@ func (p Profile) Fingerprint() string {
|
||||
sum := sha256.Sum256(b)
|
||||
return hex.EncodeToString(sum[:8])
|
||||
}
|
||||
|
||||
// SuggestCores is the cores to give a replica with no GPU on nodes of these
|
||||
// CPU counts: half of the smallest, so two replicas share any of them without
|
||||
// competing. Zero when no node says, which reserves nothing.
|
||||
func SuggestCores(cpuCounts []int) int {
|
||||
smallest := 0
|
||||
for _, c := range cpuCounts {
|
||||
if c > 0 && (smallest == 0 || c < smallest) {
|
||||
smallest = c
|
||||
}
|
||||
}
|
||||
if smallest == 0 {
|
||||
return 0
|
||||
}
|
||||
return max(smallest/2, 1)
|
||||
}
|
||||
|
||||
// The capacity classes, lowest first.
|
||||
const (
|
||||
ClassOpportunistic = "opportunistic"
|
||||
ClassStandard = "standard"
|
||||
ClassCritical = "critical"
|
||||
)
|
||||
|
||||
func validClass(c string) bool {
|
||||
return c == "" || c == ClassOpportunistic || c == ClassStandard || c == ClassCritical
|
||||
}
|
||||
|
||||
// EffectiveClass is the class a model is treated as: what it says, or, when
|
||||
// it says nothing, opportunistic for one loaded on demand — it holds capacity
|
||||
// only while asked for, and has always been the first to give it up — and
|
||||
// standard for everything else.
|
||||
func (m Model) EffectiveClass() string {
|
||||
if m.Class != "" {
|
||||
return m.Class
|
||||
}
|
||||
if m.Scaling.OnDemand() {
|
||||
return ClassOpportunistic
|
||||
}
|
||||
return ClassStandard
|
||||
}
|
||||
|
||||
// ClassRank orders classes: a higher rank may take capacity from a lower one.
|
||||
func ClassRank(c string) int {
|
||||
switch c {
|
||||
case ClassCritical:
|
||||
return 3
|
||||
case ClassStandard, "":
|
||||
return 2
|
||||
default:
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
func validOwner(o string) bool {
|
||||
for _, p := range []string{"org:", "team:", "project:"} {
|
||||
if strings.HasPrefix(o, p) && len(o) > len(p) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -375,3 +375,6 @@ func EngineKeys(secret []byte) func(replicaID string) string {
|
||||
return "llmgw-" + hex.EncodeToString(m.Sum(nil))[:40]
|
||||
}
|
||||
}
|
||||
|
||||
// QuotaUse is every capacity quota beside what its scope holds.
|
||||
func (c *Controller) QuotaUse() []reconciler.QuotaUse { return c.rec.QuotaUse() }
|
||||
@@ -51,12 +51,16 @@ type HealthSource interface {
|
||||
type Collector struct {
|
||||
src Source
|
||||
|
||||
modelDesired *prometheus.Desc
|
||||
modelFailures *prometheus.Desc
|
||||
modelParked *prometheus.Desc
|
||||
modelUnplaceable *prometheus.Desc
|
||||
engineInstall *prometheus.Desc
|
||||
leader *prometheus.Desc
|
||||
modelDesired *prometheus.Desc
|
||||
modelFailures *prometheus.Desc
|
||||
modelParked *prometheus.Desc
|
||||
modelUnplaceable *prometheus.Desc
|
||||
modelPreempted *prometheus.Desc
|
||||
capacityHolds *prometheus.Desc
|
||||
capacityGuarantee *prometheus.Desc
|
||||
capacityLimit *prometheus.Desc
|
||||
engineInstall *prometheus.Desc
|
||||
leader *prometheus.Desc
|
||||
|
||||
nodeUp *prometheus.Desc
|
||||
nodeGPUs *prometheus.Desc
|
||||
@@ -70,6 +74,13 @@ type Collector struct {
|
||||
gpuMemoryUsed *prometheus.Desc
|
||||
gpuUtil *prometheus.Desc
|
||||
gpuTemp *prometheus.Desc
|
||||
gpuMemTemp *prometheus.Desc
|
||||
gpuPower *prometheus.Desc
|
||||
gpuPowerLimit *prometheus.Desc
|
||||
gpuThrottled *prometheus.Desc
|
||||
nodeCPUTemp *prometheus.Desc
|
||||
nodeCPUPower *prometheus.Desc
|
||||
nodeInletTemp *prometheus.Desc
|
||||
gpuHealthy *prometheus.Desc
|
||||
gpuNVLink *prometheus.Desc
|
||||
gpuECC *prometheus.Desc
|
||||
@@ -138,6 +149,21 @@ func New(src Source) *Collector {
|
||||
"How busy a GPU was over the last sample period. Absent when the driver does not "+
|
||||
"report it, which is not the same as idle. Memory used says a model is loaded; "+
|
||||
"this says whether it is doing anything.", gpu, nil),
|
||||
gpuMemTemp: prometheus.NewDesc("llmgw_gpu_memory_temperature_celsius",
|
||||
"A GPU's memory (HBM) temperature, where the driver reports it.", gpu, nil),
|
||||
gpuPower: prometheus.NewDesc("llmgw_gpu_power_watts",
|
||||
"A GPU's power draw now.", gpu, nil),
|
||||
gpuPowerLimit: prometheus.NewDesc("llmgw_gpu_power_limit_watts",
|
||||
"The power limit a GPU is held to.", gpu, nil),
|
||||
gpuThrottled: prometheus.NewDesc("llmgw_gpu_throttled",
|
||||
"1 while something holds a GPU's clocks down: sw_power_cap, hw_slowdown, sw_thermal, hw_thermal, power_brake.",
|
||||
append(gpu, "reason"), nil),
|
||||
nodeCPUTemp: prometheus.NewDesc("llmgw_node_cpu_temperature_celsius",
|
||||
"A node's hottest CPU sensor.", node, nil),
|
||||
nodeCPUPower: prometheus.NewDesc("llmgw_node_cpu_power_watts",
|
||||
"A node's CPU packages' power draw, from RAPL.", node, nil),
|
||||
nodeInletTemp: prometheus.NewDesc("llmgw_node_inlet_temperature_celsius",
|
||||
"A node's inlet or ambient air sensor, where its board has one.", node, nil),
|
||||
gpuTemp: prometheus.NewDesc("llmgw_gpu_temperature_celsius",
|
||||
"A GPU's reported temperature.", gpu, nil),
|
||||
gpuHealthy: prometheus.NewDesc("llmgw_gpu_healthy",
|
||||
@@ -168,6 +194,18 @@ func New(src Source) *Collector {
|
||||
modelUnplaceable: prometheus.NewDesc("llmgw_model_unplaceable",
|
||||
"1 when a model is short of replicas and the last attempt found no node to place one on.",
|
||||
[]string{"model"}, nil),
|
||||
capacityHolds: prometheus.NewDesc("llmgw_capacity_holds",
|
||||
"What a part of the tenancy tree holds of a resource, its own models' replicas and those beneath it.",
|
||||
[]string{"scope", "resource"}, nil),
|
||||
capacityGuarantee: prometheus.NewDesc("llmgw_capacity_guarantee",
|
||||
"What a part of the tenancy tree is guaranteed of a resource.",
|
||||
[]string{"scope", "resource"}, nil),
|
||||
capacityLimit: prometheus.NewDesc("llmgw_capacity_limit",
|
||||
"The most of a resource a part of the tenancy tree may hold.",
|
||||
[]string{"scope", "resource"}, nil),
|
||||
modelPreempted: prometheus.NewDesc("llmgw_model_preempted_total",
|
||||
"Replicas of a model stopped to make room for a model of a higher capacity class.",
|
||||
[]string{"model", "class"}, nil),
|
||||
engineInstall: prometheus.NewDesc("llmgw_node_engine_install",
|
||||
"Where installing an engine stands on a node: 1 for the current state.",
|
||||
[]string{"node", "engine", "version", "state"}, nil),
|
||||
@@ -234,9 +272,11 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
|
||||
c.ibActive, c.ibRate, c.ibCounter, c.fabricSMs, c.fabricMaster, c.fabricProblems,
|
||||
c.gpuNVLJoined, c.imexUp,
|
||||
c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
|
||||
c.gpuMemTemp, c.gpuPower, c.gpuPowerLimit, c.gpuThrottled, c.nodeCPUTemp, c.nodeCPUPower, c.nodeInletTemp,
|
||||
c.gpuNVLink, c.gpuECC,
|
||||
c.replicaState, c.replicaReady, c.replicaIdle,
|
||||
c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable,
|
||||
c.modelDesired, c.modelFailures, c.modelParked, c.modelUnplaceable, c.modelPreempted,
|
||||
c.capacityHolds, c.capacityGuarantee, c.capacityLimit,
|
||||
c.engineInstall, c.leader,
|
||||
} {
|
||||
ch <- d
|
||||
@@ -246,6 +286,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
|
||||
// Collect implements prometheus.Collector.
|
||||
func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
now := time.Now()
|
||||
c.collectQuotas(ch)
|
||||
|
||||
loads := map[string]*inventory.Load{}
|
||||
for _, n := range c.src.Nodes() {
|
||||
@@ -279,6 +320,16 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
}
|
||||
|
||||
if l := n.Load; l != nil && now.Sub(l.At) < loadFresh {
|
||||
// Each only when measured: a VM has none of these, and a zero
|
||||
// would read as a machine at 0 °C drawing nothing.
|
||||
for _, v := range []struct {
|
||||
d *prometheus.Desc
|
||||
val float64
|
||||
}{{c.nodeCPUTemp, l.CPUTempC}, {c.nodeCPUPower, l.CPUPowerW}, {c.nodeInletTemp, l.InletTempC}} {
|
||||
if v.val > 0 {
|
||||
ch <- prometheus.MustNewConstMetric(v.d, prometheus.GaugeValue, v.val, n.ID)
|
||||
}
|
||||
}
|
||||
ch <- prometheus.MustNewConstMetric(c.nodeCPU, prometheus.GaugeValue, l.CPUPercent, n.ID)
|
||||
if l.CPUCount > 0 {
|
||||
ch <- prometheus.MustNewConstMetric(c.nodeCPUs, prometheus.GaugeValue, float64(l.CPUCount), n.ID)
|
||||
@@ -348,7 +399,11 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
|
||||
float64(g.UtilizationPercent), labels...)
|
||||
}
|
||||
if g.Health.TemperatureC > 0 {
|
||||
// The sensors' fresh reading when there is one; the temperature
|
||||
// read at registration otherwise, which is all an older agent sends.
|
||||
if s, ok := sensorFor(n.Load, g.Index, now); ok {
|
||||
c.gpuSensor(ch, s, labels)
|
||||
} else if g.Health.TemperatureC > 0 {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuTemp, prometheus.GaugeValue,
|
||||
float64(g.Health.TemperatureC), labels...)
|
||||
}
|
||||
@@ -455,6 +510,7 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
ch <- prometheus.MustNewConstMetric(c.modelFailures, prometheus.GaugeValue, float64(h.Failures), h.Model)
|
||||
ch <- prometheus.MustNewConstMetric(c.modelParked, prometheus.GaugeValue, b2f(h.Parked), h.Model)
|
||||
ch <- prometheus.MustNewConstMetric(c.modelUnplaceable, prometheus.GaugeValue, b2f(h.Unplaceable), h.Model)
|
||||
ch <- prometheus.MustNewConstMetric(c.modelPreempted, prometheus.CounterValue, float64(h.Preempted), h.Model, h.Class)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -464,3 +520,65 @@ func b2f(b bool) float64 {
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// QuotaSource adds the tenancy tree's capacity quotas. Asked for by type
|
||||
// assertion, like HealthSource.
|
||||
type QuotaSource interface {
|
||||
QuotaUse() []reconciler.QuotaUse
|
||||
}
|
||||
|
||||
func (c *Collector) collectQuotas(ch chan<- prometheus.Metric) {
|
||||
qs, ok := c.src.(QuotaSource)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
for _, q := range qs.QuotaUse() {
|
||||
for _, r := range []struct {
|
||||
name string
|
||||
holds int
|
||||
gua, lim *int
|
||||
}{
|
||||
{"gpus", q.Holds.GPUs, q.Guarantee.GPUs, q.Limit.GPUs},
|
||||
{"cpu_cores", q.Holds.CPUCores, q.Guarantee.CPUCores, q.Limit.CPUCores},
|
||||
{"memory_gb", q.Holds.MemoryGB, q.Guarantee.MemoryGB, q.Limit.MemoryGB},
|
||||
} {
|
||||
if r.gua == nil && r.lim == nil {
|
||||
continue
|
||||
}
|
||||
ch <- prometheus.MustNewConstMetric(c.capacityHolds, prometheus.GaugeValue, float64(r.holds), q.Scope, r.name)
|
||||
if r.gua != nil {
|
||||
ch <- prometheus.MustNewConstMetric(c.capacityGuarantee, prometheus.GaugeValue, float64(*r.gua), q.Scope, r.name)
|
||||
}
|
||||
if r.lim != nil {
|
||||
ch <- prometheus.MustNewConstMetric(c.capacityLimit, prometheus.GaugeValue, float64(*r.lim), q.Scope, r.name)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// sensorFor is a GPU's fresh sensor reading, if the node sent one.
|
||||
func sensorFor(l *inventory.Load, index int, now time.Time) (inventory.GPUSensor, bool) {
|
||||
if l == nil || now.Sub(l.At) >= loadFresh {
|
||||
return inventory.GPUSensor{}, false
|
||||
}
|
||||
for _, g := range l.GPUs {
|
||||
if g.Index == index {
|
||||
return g, true
|
||||
}
|
||||
}
|
||||
return inventory.GPUSensor{}, false
|
||||
}
|
||||
|
||||
func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
|
||||
for _, v := range []struct {
|
||||
d *prometheus.Desc
|
||||
val float64
|
||||
}{{c.gpuTemp, s.TempC}, {c.gpuMemTemp, s.MemoryTempC}, {c.gpuPower, s.PowerW}, {c.gpuPowerLimit, s.PowerLimitW}} {
|
||||
if v.val > 0 {
|
||||
ch <- prometheus.MustNewConstMetric(v.d, prometheus.GaugeValue, v.val, labels...)
|
||||
}
|
||||
}
|
||||
for _, reason := range s.Throttle {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuThrottled, prometheus.GaugeValue, 1, append(labels, reason)...)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
package fleetmetrics
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
|
||||
gpuNode := inventory.Node{ID: "g1", State: inventory.Ready,
|
||||
GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559, Health: discovery.GPUHealth{TemperatureC: 40}}},
|
||||
Load: &inventory.Load{At: time.Now(), CPUTempC: 58, CPUPowerW: 65,
|
||||
GPUs: []inventory.GPUSensor{{Index: 0, TempC: 88, PowerW: 698, PowerLimitW: 700,
|
||||
Throttle: []string{"sw_power_cap", "sw_thermal"}}}}}
|
||||
vm := inventory.Node{ID: "vm1", State: inventory.Ready, Load: &inventory.Load{At: time.Now(), CPUPercent: 3}}
|
||||
out := gather(t, fakeSource{nodes: []inventory.Node{gpuNode, vm}})
|
||||
|
||||
for _, want := range []string{
|
||||
"llmgw_node_cpu_temperature_celsius node=g1 -> 58",
|
||||
"llmgw_node_cpu_power_watts node=g1 -> 65",
|
||||
"llmgw_gpu_power_watts class=h100-80g gpu=0 node=g1 -> 698",
|
||||
"llmgw_gpu_power_limit_watts class=h100-80g gpu=0 node=g1 -> 700",
|
||||
"llmgw_gpu_throttled class=h100-80g gpu=0 node=g1 reason=sw_thermal -> 1",
|
||||
// The fresh reading, not the 40 °C from registration.
|
||||
"llmgw_gpu_temperature_celsius class=h100-80g gpu=0 node=g1 -> 88",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("missing %q", want)
|
||||
}
|
||||
}
|
||||
for _, absent := range []string{"node=vm1 -> 0", "llmgw_node_inlet_temperature_celsius", "llmgw_gpu_memory_temperature_celsius"} {
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if strings.Contains(line, absent) && (strings.Contains(line, "temperature") || strings.Contains(line, "power")) {
|
||||
t.Errorf("published what was not measured: %s", line)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -79,6 +79,9 @@ type Node struct {
|
||||
// it, which is treated as no constraint — the same as every node before
|
||||
// this field existed.
|
||||
MemoryTotalMB int `json:"memory_total_mb,omitempty"`
|
||||
// CPUCount is the CPUs the agent may run on. Zero is an agent too old to
|
||||
// say, which, like memory, constrains nothing.
|
||||
CPUCount int `json:"cpu_count,omitempty"`
|
||||
|
||||
// Weights are the model files this node can see, however they got there.
|
||||
// Empty means the node was not asked to look, which is not the same as
|
||||
@@ -301,6 +304,21 @@ type Load struct {
|
||||
MemAvailBytes int64
|
||||
// Shards is each engine's use, by shard ID.
|
||||
Shards map[string]ShardLoad
|
||||
// What the node's sensors say; zero is not measured.
|
||||
CPUTempC float64
|
||||
CPUPowerW float64
|
||||
InletTempC float64
|
||||
GPUs []GPUSensor
|
||||
}
|
||||
|
||||
// GPUSensor is one card's heat and power, and what is throttling it.
|
||||
type GPUSensor struct {
|
||||
Index int `json:"index"`
|
||||
TempC float64 `json:"temp_c,omitempty"`
|
||||
MemoryTempC float64 `json:"memory_temp_c,omitempty"`
|
||||
PowerW float64 `json:"power_w,omitempty"`
|
||||
PowerLimitW float64 `json:"power_limit_w,omitempty"`
|
||||
Throttle []string `json:"throttle,omitempty"`
|
||||
}
|
||||
|
||||
// ShardLoad is one engine's use: cores busy, and resident memory.
|
||||
@@ -373,6 +391,9 @@ type Allocation struct {
|
||||
// Zero for a profile that declared nothing, which constrains nothing and
|
||||
// is counted as nothing.
|
||||
MemoryGB int
|
||||
// CPUCores is the cores this shard was given to compute on, for a shard
|
||||
// with no GPU; zero for one that shares whatever is free.
|
||||
CPUCores int
|
||||
}
|
||||
|
||||
// Store holds the cluster inventory. It is safe for concurrent use.
|
||||
@@ -392,6 +413,23 @@ func NewStore() *Store {
|
||||
return &Store{nodes: map[string]*Node{}, allocs: map[string][]Allocation{}}
|
||||
}
|
||||
|
||||
// Clone is a copy to try decisions on without making them: allocations made
|
||||
// on it are its own. Nodes are copied by value; the slices inside them are
|
||||
// shared, and a clone must not write through them.
|
||||
func (s *Store) Clone() *Store {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
c := NewStore()
|
||||
for id, n := range s.nodes {
|
||||
cp := *n
|
||||
c.nodes[id] = &cp
|
||||
}
|
||||
for id, list := range s.allocs {
|
||||
c.allocs[id] = slices.Clone(list)
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// Upsert records a node's registration, preserving operator-set state such as
|
||||
// a cordon: an agent reconnecting must not silently undo an operator's decision.
|
||||
// RecordSMChange keeps a port's change of master subnet manager, the last
|
||||
|
||||
@@ -91,6 +91,9 @@ type Request struct {
|
||||
// use. They default to 16384 and 4096, what four parallel slots divide.
|
||||
CtxPool int `json:"ctx_pool,omitempty"`
|
||||
ContextLength int `json:"context_length,omitempty"`
|
||||
// CPUCores is what a replica computes on; zero takes half the smallest
|
||||
// node it fits on.
|
||||
CPUCores int `json:"cpu_cores,omitempty"`
|
||||
// By is who asked, for the catalog's record.
|
||||
By string `json:"-"`
|
||||
}
|
||||
@@ -278,13 +281,20 @@ func (s *Server) run(j *Job) {
|
||||
}
|
||||
s.step(j, "check", Done, fmt.Sprintf("%s, needs about %d GB (%s)", orUnknown(header.Architecture), memGB, basis))
|
||||
|
||||
cores := req.CPUCores
|
||||
if cores <= 0 {
|
||||
if nc, ok := s.deps.Fleet.(interface{ NodeCPUs(int) []int }); ok {
|
||||
cores = catalog.SuggestCores(nc.NodeCPUs(memGB))
|
||||
}
|
||||
}
|
||||
|
||||
// 4. The catalog.
|
||||
m := catalog.Model{
|
||||
Name: name, Aliases: req.Aliases, Description: req.Description, Source: source,
|
||||
RoutingProfile: "chat", ContextLength: req.ContextLength, MaxOutput: 1024,
|
||||
Profiles: []catalog.Profile{{
|
||||
Name: "cpu", Engine: "llamacpp", Weight: 1, Priority: 100,
|
||||
Args: []string{"--ctx-size", strconv.Itoa(req.CtxPool)}, MinMemoryGB: memGB,
|
||||
Args: []string{"--ctx-size", strconv.Itoa(req.CtxPool)}, MinMemoryGB: memGB, CPUCores: cores,
|
||||
}},
|
||||
Scaling: catalog.Scaling{Min: req.Replicas, Max: max(req.Replicas, 1)},
|
||||
}
|
||||
|
||||
@@ -18,6 +18,7 @@ package modelsvc
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"sync"
|
||||
@@ -104,6 +105,16 @@ func (s *Service) Add(ctx context.Context, m catalog.Model, by string) error {
|
||||
if s.Declared(m.Name) {
|
||||
return fmt.Errorf("model %q is declared in configuration; edit that rather than adding a second definition", m.Name)
|
||||
}
|
||||
// A model with no GPU is all host memory, and one that declares none
|
||||
// fits everywhere as far as placement can tell — however many of it
|
||||
// there already are. Required of what is added now; models stored before
|
||||
// this still load, and say what they are missing.
|
||||
for _, p := range m.Profiles {
|
||||
if p.CPUOnly() && p.MinMemoryGB <= 0 {
|
||||
return fmt.Errorf("model %q profile %q runs without a GPU and declares no memory (min_memory_gb): "+
|
||||
"placement could not tell when a node is full", m.Name, p.Name)
|
||||
}
|
||||
}
|
||||
if s.check != nil {
|
||||
if err := s.check(ctx, m); err != nil {
|
||||
return err
|
||||
@@ -189,36 +200,98 @@ func (s *Service) Remove(ctx context.Context, name string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Scale changes how many replicas a model runs, without a restart.
|
||||
//
|
||||
// Only the scaling block moves. Everything else about the model — its source,
|
||||
// its profiles, the engine arguments somebody tuned — is read back from what
|
||||
// was stored and written out again untouched, so changing a number cannot
|
||||
// quietly become a redefinition.
|
||||
//
|
||||
// A model declared in configuration is refused, for the same reason Remove
|
||||
// refuses one: the YAML is read on every start, so the new number would hold
|
||||
// until the next restart and then silently revert. A cluster that disagrees
|
||||
// with its own config file is worse than an error message saying where to edit.
|
||||
// Scale changes how many replicas a model added at runtime runs.
|
||||
func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by string) (catalog.Model, error) {
|
||||
final, err := s.update(ctx, name, by, "change its scaling", func(m *catalog.Model) error {
|
||||
m.Scaling = sc
|
||||
return nil
|
||||
})
|
||||
if err == nil {
|
||||
s.log.Info("model scaled", "model", name,
|
||||
"min", final.Scaling.Min, "max", final.Scaling.Max, "by", by)
|
||||
}
|
||||
return final, err
|
||||
}
|
||||
|
||||
// Resources sets what a model's profile reserves of a node: host memory, and
|
||||
// the CPU cores a replica without a GPU computes on. An empty profile name
|
||||
// sets every profile.
|
||||
func (s *Service) Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error) {
|
||||
if memoryGB < 0 || cpuCores < 0 {
|
||||
return catalog.Model{}, errors.New("memory and cores cannot be negative")
|
||||
}
|
||||
final, err := s.update(ctx, name, by, "change its resources", func(m *catalog.Model) error {
|
||||
found := false
|
||||
for i := range m.Profiles {
|
||||
if profile != "" && m.Profiles[i].Name != profile {
|
||||
continue
|
||||
}
|
||||
found = true
|
||||
m.Profiles[i].MinMemoryGB = memoryGB
|
||||
m.Profiles[i].CPUCores = cpuCores
|
||||
}
|
||||
if !found {
|
||||
return fmt.Errorf("model %q has no profile %q", name, profile)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err == nil {
|
||||
s.log.Info("model resources set", "model", name, "profile", profile,
|
||||
"memory_gb", memoryGB, "cpu_cores", cpuCores, "by", by)
|
||||
}
|
||||
return final, err
|
||||
}
|
||||
|
||||
// SetClass sets a model's capacity class: critical, standard or
|
||||
// opportunistic, or empty to go back to what its scaling implies.
|
||||
func (s *Service) SetClass(ctx context.Context, name, class, by string) (catalog.Model, error) {
|
||||
final, err := s.update(ctx, name, by, "change its class", func(m *catalog.Model) error {
|
||||
m.Class = class
|
||||
return nil
|
||||
})
|
||||
if err == nil {
|
||||
s.log.Info("model class set", "model", name, "class", final.EffectiveClass(), "by", by)
|
||||
}
|
||||
return final, err
|
||||
}
|
||||
|
||||
// SetOwner sets the part of the tenancy tree a model's capacity counts
|
||||
// against, org:ID, team:ID or project:ID; empty makes it shared.
|
||||
func (s *Service) SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error) {
|
||||
final, err := s.update(ctx, name, by, "change its owner", func(m *catalog.Model) error {
|
||||
m.Owner = owner
|
||||
return nil
|
||||
})
|
||||
if err == nil {
|
||||
s.log.Info("model owner set", "model", name, "owner", owner, "by", by)
|
||||
}
|
||||
return final, err
|
||||
}
|
||||
|
||||
// update changes one model added at runtime and keeps the change.
|
||||
//
|
||||
// Built before anything is written, like Add: catalog.New is what applies the
|
||||
// defaults and refuses what does not validate, so the model that comes back
|
||||
// is the one that will run rather than the one that was asked for.
|
||||
func (s *Service) update(ctx context.Context, name, by, what string, change func(*catalog.Model) error) (catalog.Model, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
|
||||
if s.Declared(name) {
|
||||
return catalog.Model{}, fmt.Errorf(
|
||||
"model %q is declared in configuration; change its scaling there, or it reverts on the next restart", name)
|
||||
"model %q is declared in configuration; %s there, or it reverts on the next restart", name, what)
|
||||
}
|
||||
|
||||
stored, err := s.storedModels(ctx)
|
||||
if err != nil {
|
||||
return catalog.Model{}, err
|
||||
}
|
||||
|
||||
next := append([]catalog.Model{}, s.declared...)
|
||||
found := false
|
||||
for _, e := range stored {
|
||||
if e.Name == name {
|
||||
e.Scaling = sc
|
||||
if err := change(&e); err != nil {
|
||||
return catalog.Model{}, err
|
||||
}
|
||||
found = true
|
||||
}
|
||||
next = append(next, e)
|
||||
@@ -226,20 +299,14 @@ func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by
|
||||
if !found {
|
||||
return catalog.Model{}, fmt.Errorf("model %q was not added at runtime", name)
|
||||
}
|
||||
|
||||
// Built before anything is written, like Add: catalog.New is what applies
|
||||
// the defaults (max follows min when it is left out) and what refuses a
|
||||
// max below a min, so the model that comes back is the one that will run
|
||||
// rather than the one that was asked for.
|
||||
built, err := catalog.New(next)
|
||||
if err != nil {
|
||||
return catalog.Model{}, fmt.Errorf("scaling %q that way would make the catalog invalid: %w", name, err)
|
||||
return catalog.Model{}, fmt.Errorf("that would make the catalog invalid: %w", err)
|
||||
}
|
||||
final, ok := built.Resolve(name)
|
||||
if !ok {
|
||||
return catalog.Model{}, fmt.Errorf("model %q vanished from the catalog it was just built into", name)
|
||||
}
|
||||
|
||||
def, err := json.Marshal(final)
|
||||
if err != nil {
|
||||
return catalog.Model{}, err
|
||||
@@ -249,10 +316,7 @@ func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by
|
||||
}); err != nil {
|
||||
return catalog.Model{}, err
|
||||
}
|
||||
|
||||
s.apply.ReplaceCatalog(ctx, built)
|
||||
s.log.Info("model scaled", "model", name,
|
||||
"min", final.Scaling.Min, "max", final.Scaling.Max, "by", by)
|
||||
return final, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -48,7 +48,7 @@ func model(name string) catalog.Model {
|
||||
return catalog.Model{
|
||||
Name: name, Source: "/models/" + name,
|
||||
Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "p", Engine: "llamacpp", Weight: 1}},
|
||||
Profiles: []catalog.Profile{{Name: "p", Engine: "llamacpp", Weight: 1, MinMemoryGB: 2}},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -337,3 +337,43 @@ func TestMinZeroWithAMaxIsOnDemand(t *testing.T) {
|
||||
t.Errorf("the idle timeout was lost; got %v", got.Scaling.IdleTimeout)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACPUModelAddedNowMustSayHowMuchMemoryItNeeds(t *testing.T) {
|
||||
s, st, _ := newService(t)
|
||||
m := model("bare")
|
||||
m.Profiles[0].MinMemoryGB = 0
|
||||
err := s.Add(t.Context(), m, "tester")
|
||||
if err == nil || !strings.Contains(err.Error(), "declares no memory") {
|
||||
t.Fatalf("a CPU model with no memory was added: %v", err)
|
||||
}
|
||||
if len(st.rows) != 0 {
|
||||
t.Error("it was stored anyway")
|
||||
}
|
||||
}
|
||||
|
||||
func TestResourcesAreSetOnAStoredModelAndSurviveARestart(t *testing.T) {
|
||||
s, _, _ := newService(t)
|
||||
if err := s.Add(t.Context(), model("m"), "tester"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := s.Resources(t.Context(), "m", "", 6, 3, "tester")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if p := got.Profiles[0]; p.MinMemoryGB != 6 || p.CPUCores != 3 {
|
||||
t.Errorf("profile %+v", p)
|
||||
}
|
||||
if _, err := s.Resources(t.Context(), "m", "nope", 6, 3, "tester"); err == nil {
|
||||
t.Error("a profile the model does not have was accepted")
|
||||
}
|
||||
if _, err := s.Resources(t.Context(), "m", "", -1, 0, "tester"); err == nil {
|
||||
t.Error("negative memory was accepted")
|
||||
}
|
||||
cat, err := s.Load(t.Context())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if m, _ := cat.Resolve("m"); m.Profiles[0].CPUCores != 3 {
|
||||
t.Error("the cores were lost on reload")
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
package placement
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
@@ -100,3 +101,38 @@ func TestCPUProfileRespectsAvoid(t *testing.T) {
|
||||
t.Errorf("placed on %q despite being told to avoid it", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCPUCoresAreReservedAsMemoryIs(t *testing.T) {
|
||||
n := node("n1", "", 0)
|
||||
n.CPUCount = 8
|
||||
st := store(n)
|
||||
s := New(st)
|
||||
p := cpuProfile("cpu")
|
||||
p.CPUCores = 3
|
||||
|
||||
for i := range 2 {
|
||||
res := s.Place([]Profile{p}, nil)
|
||||
if res.Placement == nil {
|
||||
t.Fatalf("replica %d did not fit in 8 cores:\n%s", i+1, res.Explain())
|
||||
}
|
||||
st.Allocate("n1", inventory.Allocation{ShardID: string(rune('a' + i)), CPUCores: 3})
|
||||
}
|
||||
// Six of eight given away: a third would need three.
|
||||
res := s.Place([]Profile{p}, nil)
|
||||
if res.Placement != nil {
|
||||
t.Fatal("a third replica was given cores the node does not have")
|
||||
}
|
||||
if got := res.Explain(); !strings.Contains(got, "has 8 CPUs with 2 free, needs 3") {
|
||||
t.Errorf("the refusal should say how many cores are left:\n%s", got)
|
||||
}
|
||||
|
||||
// Unknown counts constrain nothing: an older agent, or a profile that
|
||||
// reserves no cores.
|
||||
old := node("n2", "", 0)
|
||||
if res := New(store(old)).Place([]Profile{p}, nil); res.Placement == nil {
|
||||
t.Error("a node that does not report its CPUs was excluded")
|
||||
}
|
||||
if res := s.Place([]Profile{cpuProfile("cpu")}, nil); res.Placement == nil {
|
||||
t.Error("a profile reserving no cores was refused")
|
||||
}
|
||||
}
|
||||
@@ -39,6 +39,9 @@ type Profile struct {
|
||||
// MinDiskGB is the free space needed to hold the weights, checked only on
|
||||
// nodes that do not already have them. Zero is unconstrained.
|
||||
MinDiskGB int
|
||||
// CPUCores is the cores a profile with no GPU reserves. Zero is
|
||||
// unconstrained.
|
||||
CPUCores int
|
||||
// Source is the model's weights, as the engine would be given it. Used to
|
||||
// prefer nodes that already hold them; empty asks for no preference.
|
||||
Source string
|
||||
@@ -582,6 +585,18 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
|
||||
}
|
||||
}
|
||||
|
||||
// Cores, for a replica that computes on them: what is left after the
|
||||
// cores already given to the node's other replicas. Same exemptions
|
||||
// as memory: zero declares nothing, and a node that does not report
|
||||
// its CPUs is an older agent, not one with none.
|
||||
if p.CPUOnly() && p.CPUCores > 0 && n.CPUCount > 0 {
|
||||
free := n.CPUCount - s.claimedCores(n.ID)
|
||||
if free < p.CPUCores {
|
||||
reject(fmt.Sprintf("has %d CPUs with %d free, needs %d", n.CPUCount, free, p.CPUCores))
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
if p.CPUOnly() {
|
||||
// Every ready node can run it; there is no card to be free.
|
||||
out = append(out, candidate{node: n})
|
||||
@@ -723,6 +738,15 @@ func (s *Scheduler) claimedMemory(nodeID string) int {
|
||||
return total
|
||||
}
|
||||
|
||||
// claimedCores is the CPU cores already given to replicas on a node.
|
||||
func (s *Scheduler) claimedCores(nodeID string) int {
|
||||
total := 0
|
||||
for _, a := range s.inv.Allocations(nodeID) {
|
||||
total += a.CPUCores
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
// installing says why an engine is missing, when the node has said.
|
||||
func installing(n inventory.Node, engine string) string {
|
||||
for _, p := range n.Provision {
|
||||
|
||||
@@ -0,0 +1,192 @@
|
||||
// Package quota decides what the tenancy tree is guaranteed of the cluster
|
||||
// and how much of it each part may hold.
|
||||
//
|
||||
// A scope — an organization, team or project — may be guaranteed GPUs, CPU
|
||||
// cores and memory, and limited to at most some amount of each. What a scope
|
||||
// holds is what the replicas of the models it owns, and of the models its
|
||||
// teams and projects own, have reserved. Holding more than its guarantee is
|
||||
// borrowing: allowed while the room is free, and taken back when another
|
||||
// scope within its own guarantee needs it.
|
||||
package quota
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Amount is how much of each resource something holds or needs.
|
||||
type Amount struct {
|
||||
GPUs, CPUCores, MemoryGB int
|
||||
}
|
||||
|
||||
// Plus adds two amounts.
|
||||
func (a Amount) Plus(b Amount) Amount {
|
||||
return Amount{a.GPUs + b.GPUs, a.CPUCores + b.CPUCores, a.MemoryGB + b.MemoryGB}
|
||||
}
|
||||
|
||||
// Over names the first resource in which a is above the governed amounts of
|
||||
// l, with both figures; empty when it is above none.
|
||||
func Over(a Amount, l store.Resources) string {
|
||||
switch {
|
||||
case l.GPUs != nil && a.GPUs > *l.GPUs:
|
||||
return fmt.Sprintf("%d GPUs of %d", a.GPUs, *l.GPUs)
|
||||
case l.CPUCores != nil && a.CPUCores > *l.CPUCores:
|
||||
return fmt.Sprintf("%d cores of %d", a.CPUCores, *l.CPUCores)
|
||||
case l.MemoryGB != nil && a.MemoryGB > *l.MemoryGB:
|
||||
return fmt.Sprintf("%d GB of memory of %d", a.MemoryGB, *l.MemoryGB)
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// Holding is one replica's hold on the cluster, for working out who holds
|
||||
// what.
|
||||
type Holding struct {
|
||||
Replica string
|
||||
Owner string
|
||||
Amount Amount
|
||||
Since time.Time
|
||||
}
|
||||
|
||||
// Policy is the quotas and the tree they sit in, read at one moment.
|
||||
type Policy struct {
|
||||
quotas map[string]store.CapacityQuota
|
||||
parents map[string][]string
|
||||
}
|
||||
|
||||
// New builds a policy from stored quotas and each scope's parents.
|
||||
func New(quotas []store.CapacityQuota, parents map[string][]string) *Policy {
|
||||
p := &Policy{quotas: map[string]store.CapacityQuota{}, parents: parents}
|
||||
for _, q := range quotas {
|
||||
p.quotas[Key(q.Scope, q.ScopeID)] = q
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// Key writes a scope the way owners are written: team:ID.
|
||||
func Key(s store.Scope, id string) string { return string(s) + ":" + id }
|
||||
|
||||
// Chain is an owner and every scope above it, nearest first.
|
||||
func (p *Policy) Chain(owner string) []string {
|
||||
if owner == "" {
|
||||
return nil
|
||||
}
|
||||
return append([]string{owner}, p.parents[owner]...)
|
||||
}
|
||||
|
||||
// Governing is the quotas along an owner's chain, nearest first.
|
||||
func (p *Policy) Governing(owner string) []store.CapacityQuota {
|
||||
if p == nil {
|
||||
return nil
|
||||
}
|
||||
var out []store.CapacityQuota
|
||||
for _, s := range p.Chain(owner) {
|
||||
if q, ok := p.quotas[s]; ok {
|
||||
out = append(out, q)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Quotas lists every quota.
|
||||
func (p *Policy) Quotas() []store.CapacityQuota {
|
||||
if p == nil {
|
||||
return nil
|
||||
}
|
||||
out := make([]store.CapacityQuota, 0, len(p.quotas))
|
||||
for _, q := range p.quotas {
|
||||
out = append(out, q)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
return Key(out[i].Scope, out[i].ScopeID) < Key(out[j].Scope, out[j].ScopeID)
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// Usage is what every scope holds, its own models' replicas and those of
|
||||
// everything beneath it.
|
||||
func (p *Policy) Usage(hs []Holding) map[string]Amount {
|
||||
out := map[string]Amount{}
|
||||
if p == nil {
|
||||
return out
|
||||
}
|
||||
for _, h := range hs {
|
||||
for _, s := range p.Chain(h.Owner) {
|
||||
out[s] = out[s].Plus(h.Amount)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Borrowed is the replicas held beyond a guarantee: of each governed scope's
|
||||
// replicas, oldest first, the ones that take it past what it is guaranteed.
|
||||
// A replica beyond any guarantee along its chain is borrowed.
|
||||
func (p *Policy) Borrowed(hs []Holding) map[string]bool {
|
||||
out := map[string]bool{}
|
||||
if p == nil || len(p.quotas) == 0 {
|
||||
return out
|
||||
}
|
||||
sorted := append([]Holding(nil), hs...)
|
||||
sort.Slice(sorted, func(i, j int) bool { return sorted[i].Since.Before(sorted[j].Since) })
|
||||
for key, q := range p.quotas {
|
||||
if !q.Guarantee.Any() {
|
||||
continue
|
||||
}
|
||||
var sum Amount
|
||||
for _, h := range sorted {
|
||||
if !contains(p.Chain(h.Owner), key) {
|
||||
continue
|
||||
}
|
||||
sum = sum.Plus(h.Amount)
|
||||
if Over(sum, q.Guarantee) != "" {
|
||||
out[h.Replica] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Guaranteed reports whether an owner, holding what it holds, is still within
|
||||
// every guarantee along its chain with need added — so it may take back what
|
||||
// others borrowed. An owner with no guarantee anywhere is never guaranteed.
|
||||
func (p *Policy) Guaranteed(owner string, usage map[string]Amount, need Amount) bool {
|
||||
gov := p.Governing(owner)
|
||||
any := false
|
||||
for _, q := range gov {
|
||||
if !q.Guarantee.Any() {
|
||||
continue
|
||||
}
|
||||
any = true
|
||||
if Over(usage[Key(q.Scope, q.ScopeID)].Plus(need), q.Guarantee) != "" {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return any
|
||||
}
|
||||
|
||||
// AtLimit says which limit along an owner's chain need would take it past,
|
||||
// empty when none.
|
||||
func (p *Policy) AtLimit(owner string, usage map[string]Amount, need Amount) string {
|
||||
for _, q := range p.Governing(owner) {
|
||||
if !q.Limit.Any() {
|
||||
continue
|
||||
}
|
||||
key := Key(q.Scope, q.ScopeID)
|
||||
if over := Over(usage[key].Plus(need), q.Limit); over != "" {
|
||||
return fmt.Sprintf("%s is at its capacity limit (%s)", key, strings.TrimSpace(over))
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func contains(list []string, v string) bool {
|
||||
for _, x := range list {
|
||||
if x == v {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package quota
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func n(v int) *int { return &v }
|
||||
|
||||
// acme has two teams; ml is guaranteed 8 GPUs, the org as a whole 12 and
|
||||
// no more than 16.
|
||||
func policy() *Policy {
|
||||
return New([]store.CapacityQuota{
|
||||
{Scope: store.ScopeTeam, ScopeID: "ml", Guarantee: store.Resources{GPUs: n(8)}},
|
||||
{Scope: store.ScopeOrg, ScopeID: "acme", Guarantee: store.Resources{GPUs: n(12)}, Limit: store.Resources{GPUs: n(16)}},
|
||||
}, map[string][]string{
|
||||
"team:ml": {"org:acme"}, "team:web": {"org:acme"},
|
||||
"project:train": {"team:ml", "org:acme"},
|
||||
})
|
||||
}
|
||||
|
||||
func at(min int) time.Time { return time.Date(2026, 1, 1, 0, min, 0, 0, time.UTC) }
|
||||
|
||||
func TestAScopeHoldsWhatEverythingBeneathItHolds(t *testing.T) {
|
||||
p := policy()
|
||||
u := p.Usage([]Holding{
|
||||
{Replica: "a", Owner: "project:train", Amount: Amount{GPUs: 4}},
|
||||
{Replica: "b", Owner: "team:web", Amount: Amount{GPUs: 2, CPUCores: 3}},
|
||||
{Replica: "c", Owner: "", Amount: Amount{GPUs: 8}}, // shared: nobody's
|
||||
})
|
||||
if u["team:ml"].GPUs != 4 || u["org:acme"].GPUs != 6 || u["org:acme"].CPUCores != 3 || u["project:train"].GPUs != 4 {
|
||||
t.Errorf("usage %+v", u)
|
||||
}
|
||||
if _, ok := u[""]; ok {
|
||||
t.Error("a shared model was counted against somebody")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWhatGoesPastAGuaranteeIsBorrowedNewestFirst(t *testing.T) {
|
||||
p := policy()
|
||||
b := p.Borrowed([]Holding{
|
||||
{Replica: "old", Owner: "team:ml", Amount: Amount{GPUs: 8}, Since: at(1)},
|
||||
{Replica: "new", Owner: "team:ml", Amount: Amount{GPUs: 4}, Since: at(2)},
|
||||
{Replica: "web", Owner: "team:web", Amount: Amount{GPUs: 4}, Since: at(3)},
|
||||
})
|
||||
// ml: 8 within its 8, then 4 past it. acme: 8 + 4 = 12 within, web's 4
|
||||
// takes the org to 16, past its 12.
|
||||
if b["old"] || !b["new"] || !b["web"] {
|
||||
t.Errorf("borrowed %v", b)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAGuaranteeIsOnlyForWhoHasOne(t *testing.T) {
|
||||
p := policy()
|
||||
u := map[string]Amount{"team:ml": {GPUs: 4}, "org:acme": {GPUs: 4}}
|
||||
if !p.Guaranteed("project:train", u, Amount{GPUs: 4}) {
|
||||
t.Error("8 of ml's 8 and 8 of acme's 12 is within both")
|
||||
}
|
||||
if p.Guaranteed("project:train", u, Amount{GPUs: 5}) {
|
||||
t.Error("9 GPUs is past ml's 8")
|
||||
}
|
||||
if p.Guaranteed("", u, Amount{GPUs: 1}) || New(nil, nil).Guaranteed("team:ml", u, Amount{}) {
|
||||
t.Error("an owner with no guarantee is not guaranteed anything")
|
||||
}
|
||||
}
|
||||
|
||||
func TestALimitStopsGrowthWhereverItIsSet(t *testing.T) {
|
||||
p := policy()
|
||||
u := map[string]Amount{"org:acme": {GPUs: 14}}
|
||||
if why := p.AtLimit("team:web", u, Amount{GPUs: 2}); why != "" {
|
||||
t.Errorf("16 of 16 is at the limit, not past it: %s", why)
|
||||
}
|
||||
why := p.AtLimit("team:web", u, Amount{GPUs: 4})
|
||||
if !strings.Contains(why, "org:acme is at its capacity limit (18 GPUs of 16)") {
|
||||
t.Errorf("the refusal should name the scope and the figures: %q", why)
|
||||
}
|
||||
if p.AtLimit("", u, Amount{GPUs: 99}) != "" {
|
||||
t.Error("a shared model was limited")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
package control
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/bench"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// QuotaStore is where capacity quotas and the tenancy tree are kept.
|
||||
type QuotaStore interface {
|
||||
CapacityQuotas(ctx context.Context) ([]store.CapacityQuota, error)
|
||||
ScopeParents(ctx context.Context) (map[string][]string, error)
|
||||
}
|
||||
|
||||
// quotaRefresh is how stale the reconciler's view of quotas may get. A quota
|
||||
// changed by an operator takes effect within this; the reconciler asks many
|
||||
// times a pass and the store is not asked each time.
|
||||
const quotaRefresh = 30 * time.Second
|
||||
|
||||
// QuotaPolicy returns what the reconciler asks for its quotas: the last
|
||||
// policy read from the store, read again once it is older than quotaRefresh.
|
||||
// A store that fails keeps the last good policy, so a database hiccup does
|
||||
// not drop every guarantee and limit at once.
|
||||
func QuotaPolicy(st QuotaStore, log *slog.Logger) func() *quota.Policy {
|
||||
var mu sync.Mutex
|
||||
var cur *quota.Policy
|
||||
var at time.Time
|
||||
return func() *quota.Policy {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if cur != nil && time.Since(at) < quotaRefresh {
|
||||
return cur
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
qs, err := st.CapacityQuotas(ctx)
|
||||
if err == nil {
|
||||
var parents map[string][]string
|
||||
if parents, err = st.ScopeParents(ctx); err == nil {
|
||||
cur, at = quota.New(qs, parents), time.Now()
|
||||
return cur
|
||||
}
|
||||
}
|
||||
log.Warn("could not read capacity quotas; keeping the last ones", "err", err)
|
||||
at = time.Now() // not again on every call while the store is down
|
||||
return cur
|
||||
}
|
||||
}
|
||||
|
||||
// MeasurementStore is where benchmark results are kept.
|
||||
type MeasurementStore interface {
|
||||
Capacities(ctx context.Context) ([]store.StoredCapacity, error)
|
||||
}
|
||||
|
||||
// Measurements returns what the reconciler asks for benchmark results, keyed
|
||||
// model/profile, read again once older than quotaRefresh; like QuotaPolicy it
|
||||
// keeps the last good answer when the store fails.
|
||||
func Measurements(st MeasurementStore, log *slog.Logger) func() map[string]reconciler.Measured {
|
||||
var mu sync.Mutex
|
||||
var cur map[string]reconciler.Measured
|
||||
var at time.Time
|
||||
return func() map[string]reconciler.Measured {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if cur != nil && time.Since(at) < quotaRefresh {
|
||||
return cur
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
at = time.Now()
|
||||
rows, err := st.Capacities(ctx)
|
||||
if err != nil {
|
||||
log.Warn("could not read benchmark results; keeping the last ones", "err", err)
|
||||
return cur
|
||||
}
|
||||
next := map[string]reconciler.Measured{}
|
||||
for _, row := range rows {
|
||||
var c bench.Capacity
|
||||
if json.Unmarshal(row.Measurement, &c) != nil {
|
||||
continue
|
||||
}
|
||||
next[row.Model+"/"+row.Profile] = reconciler.Measured{
|
||||
MaxConcurrency: c.MaxConcurrencyAtSLO, TokPerSec: c.ThroughputTokPerSec}
|
||||
}
|
||||
cur = next
|
||||
return cur
|
||||
}
|
||||
}
|
||||
@@ -144,6 +144,7 @@ func (r *Reconciler) mergeAdopted(rep *Replica, group []AdoptedShard) {
|
||||
r.inv.Allocate(s.NodeID, inventory.Allocation{
|
||||
ShardID: s.ShardID, ReplicaID: rep.ID, GPUIndices: s.GPUIndices,
|
||||
MemoryGB: r.profileMemory(rep.Model, rep.Profile),
|
||||
CPUCores: r.adoptedCores(rep.Model, rep.Profile),
|
||||
})
|
||||
// The endpoint comes from the leader, which may be in any of the
|
||||
// groups — the rank 0 node is not necessarily the first to register.
|
||||
@@ -253,3 +254,9 @@ type AdoptedShard struct {
|
||||
// Endpoint is the leader's base URL, set on the rank 0 shard.
|
||||
Endpoint string
|
||||
}
|
||||
|
||||
// adoptedCores is what an adopted shard is taken to hold of its node's CPUs.
|
||||
func (r *Reconciler) adoptedCores(model, profile string) int {
|
||||
p, _ := r.profileOf(model, profile)
|
||||
return cpuCores(p)
|
||||
}
|
||||
@@ -157,8 +157,31 @@ type Load struct {
|
||||
// the last one is still starting: on a small CPU fleet each replica is minutes
|
||||
// of weights and gigabytes of memory, and a spike that is over by the time it
|
||||
// loads should not leave three behind.
|
||||
//
|
||||
// With a benchmark stored for the profile a model runs on, its measured
|
||||
// capacity is used too: more requests per replica than were measured to meet
|
||||
// the SLO is busy, even with the engines' queues empty — an engine will take
|
||||
// requests into its slots long after latency has gone — and a replica is only
|
||||
// taken back when the rest would still be under scaleDownHeadroom of what they
|
||||
// were measured to hold, so scaling down does not land on the edge.
|
||||
func (r *Reconciler) RecordLoad(loads map[string]Load) {
|
||||
now := r.now()
|
||||
// The profile each model's replicas run on, for its measurement; read
|
||||
// before the loop, which holds the lock.
|
||||
running := map[string]string{}
|
||||
r.mu.Lock()
|
||||
for _, rep := range r.replicas {
|
||||
if rep.State.Counts() {
|
||||
running[rep.Model] = rep.Profile
|
||||
}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
measured := map[string]Measured{}
|
||||
for model, profile := range running {
|
||||
if m, ok := r.measured(model, profile); ok {
|
||||
measured[model] = m
|
||||
}
|
||||
}
|
||||
for _, m := range r.cat.Load().Models() {
|
||||
base := r.baseFor(m)
|
||||
r.mu.Lock()
|
||||
@@ -178,8 +201,13 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
|
||||
l := loads[m.Name]
|
||||
perReplica := float64(l.Waiting) / float64(max(l.Replicas, 1))
|
||||
extra := r.extra[m.Name]
|
||||
meas, hasMeas := measured[m.Name]
|
||||
overMeasured := hasMeas &&
|
||||
float64(l.Running+l.Waiting) > float64(meas.MaxConcurrency*max(l.Replicas, 1))
|
||||
fitsInFewer := !hasMeas ||
|
||||
float64(l.Running) <= scaleDownHeadroom*float64(meas.MaxConcurrency*max(l.Replicas-1, 0))
|
||||
switch {
|
||||
case perReplica >= r.cfg.ScaleUpWaiting:
|
||||
case perReplica >= r.cfg.ScaleUpWaiting || overMeasured:
|
||||
delete(r.quietSince, m.Name)
|
||||
since, ok := r.busySince[m.Name]
|
||||
if !ok {
|
||||
@@ -190,9 +218,10 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
|
||||
r.extra[m.Name] = extra + 1
|
||||
r.busySince[m.Name] = now
|
||||
r.log.Info("scaling up for load", "model", m.Name, "replicas", base+extra+1,
|
||||
"waiting_per_replica", perReplica, "for", now.Sub(since).Round(time.Second))
|
||||
"waiting_per_replica", perReplica, "in_flight", l.Running+l.Waiting,
|
||||
"measured_per_replica", meas.MaxConcurrency, "for", now.Sub(since).Round(time.Second))
|
||||
}
|
||||
case l.Waiting == 0 && extra > 0:
|
||||
case l.Waiting == 0 && extra > 0 && fitsInFewer:
|
||||
delete(r.busySince, m.Name)
|
||||
since, ok := r.quietSince[m.Name]
|
||||
if !ok {
|
||||
@@ -212,3 +241,8 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
|
||||
r.mu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
// scaleDownHeadroom is how full, of their measured capacity, the replicas left
|
||||
// after a scale-down may be. Below one so that the load that was there a
|
||||
// minute ago does not put them straight back over.
|
||||
const scaleDownHeadroom = 0.7
|
||||
@@ -0,0 +1,139 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"slices"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
)
|
||||
|
||||
// Headroom is how many more replicas of a model the cluster has room for now.
|
||||
type Headroom struct {
|
||||
// Current is the replicas the model has that hold capacity: starting or
|
||||
// ready, not those on their way out.
|
||||
Current int
|
||||
// More is how many more could be placed now, at most the limit asked.
|
||||
More int
|
||||
// Nodes is where each of those would go, in order, a replica's nodes
|
||||
// joined when it spans several.
|
||||
Nodes [][]string
|
||||
// Limited is set when More reached the limit asked, so there may be room
|
||||
// for more than that. A CPU profile that declares no memory is always
|
||||
// limited: nothing it asks for runs out.
|
||||
Limited bool
|
||||
// Blocked is why one more than More does not fit, a line per node that
|
||||
// was considered. Empty when Limited.
|
||||
Blocked []string
|
||||
// TokPerSec is what one more replica was measured to serve, on the
|
||||
// profile placement would give it; zero when it was never measured.
|
||||
TokPerSec float64
|
||||
// ByPreempting is how many more, beyond More, would fit by stopping
|
||||
// replicas the model's class may take, and Preempts the models those
|
||||
// replicas belong to: what "room" means for a model that outranks others.
|
||||
ByPreempting int
|
||||
Preempts []string
|
||||
}
|
||||
|
||||
// Headroom tries placing up to limit more replicas of a model on a copy of
|
||||
// the inventory, each one's allocation made on the copy before the next is
|
||||
// tried, exactly as the reconciler would place them one pass at a time.
|
||||
//
|
||||
// The scheduler's own decisions, not an estimate beside them: what the page
|
||||
// offers is then what the next passes will do, and a count that the
|
||||
// arithmetic of a second implementation allowed but placement refuses would
|
||||
// leave a model short with nobody told why.
|
||||
func (r *Reconciler) Headroom(model string, limit int) (Headroom, bool) {
|
||||
m, ok := r.cat.Load().Resolve(model)
|
||||
if !ok {
|
||||
return Headroom{}, false
|
||||
}
|
||||
var h Headroom
|
||||
var avoid []string
|
||||
r.mu.Lock()
|
||||
for _, rep := range r.replicas {
|
||||
if rep.Model == m.Name && rep.State.Counts() && rep.RetiringAt.IsZero() {
|
||||
h.Current++
|
||||
avoid = append(avoid, rep.NodeIDs()...)
|
||||
}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
|
||||
inv := r.inv.Clone()
|
||||
sched := placement.New(inv)
|
||||
pol := r.policy()
|
||||
var usage map[string]quota.Amount
|
||||
if pol != nil {
|
||||
usage = pol.Usage(r.Holdings())
|
||||
}
|
||||
place := func() placement.Result {
|
||||
res := sched.Place(r.orderedProfiles(m), avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
// As scaleUp does: spreading is a preference, not a rule.
|
||||
res = sched.Place(r.orderedProfiles(m), nil)
|
||||
}
|
||||
return res
|
||||
}
|
||||
// What the model's class may take, in the order preemption would, each
|
||||
// released on the copy only when free room has run out.
|
||||
victims, nextVictim, preempting := r.victims(m), 0, false
|
||||
for h.More+h.ByPreempting < limit {
|
||||
res := place()
|
||||
for res.Placement == nil && nextVictim < len(victims) {
|
||||
v := victims[nextVictim]
|
||||
nextVictim++
|
||||
inv.ReleaseReplica(v.ID)
|
||||
if !slices.Contains(h.Preempts, v.Model) {
|
||||
h.Preempts = append(h.Preempts, v.Model)
|
||||
}
|
||||
preempting = true
|
||||
res = place()
|
||||
}
|
||||
if res.Placement == nil {
|
||||
for _, rej := range res.Rejections {
|
||||
h.Blocked = append(h.Blocked, rej.String())
|
||||
}
|
||||
slices.Sort(h.Blocked)
|
||||
h.Blocked = slices.Compact(h.Blocked)
|
||||
return h, true
|
||||
}
|
||||
// A quota limit is as much an edge as a full node.
|
||||
if pol != nil && m.Owner != "" {
|
||||
need := placementNeed(m, res.Placement)
|
||||
if why := pol.AtLimit(m.Owner, usage, need); why != "" {
|
||||
h.Blocked = []string{why}
|
||||
return h, true
|
||||
}
|
||||
for _, s := range pol.Chain(m.Owner) {
|
||||
usage[s] = usage[s].Plus(need)
|
||||
}
|
||||
}
|
||||
mem, cores := 0, 0
|
||||
if prof, ok := m.ProfileByName(res.Placement.Profile.Name); ok {
|
||||
mem, cores = prof.MinMemoryGB, cpuCores(prof)
|
||||
}
|
||||
if h.More == 0 {
|
||||
if meas, ok := r.measured(m.Name, res.Placement.Profile.Name); ok {
|
||||
h.TokPerSec = meas.TokPerSec
|
||||
}
|
||||
}
|
||||
var nodes []string
|
||||
for _, a := range res.Placement.Nodes {
|
||||
inv.Allocate(a.NodeID, inventory.Allocation{
|
||||
ShardID: "headroom", ReplicaID: "headroom", GPUIndices: a.GPUIndices, MemoryGB: mem, CPUCores: cores,
|
||||
})
|
||||
nodes = append(nodes, a.NodeID)
|
||||
}
|
||||
avoid = append(avoid, nodes...)
|
||||
// Once anything was stopped, what fits after is counted as fitting
|
||||
// by preempting: it may be sitting in the room that was freed.
|
||||
if preempting {
|
||||
h.ByPreempting++
|
||||
} else {
|
||||
h.Nodes = append(h.Nodes, nodes)
|
||||
h.More++
|
||||
}
|
||||
}
|
||||
h.Limited = true
|
||||
return h, true
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
func TestHeadroomIsWhatPlacementWouldDoNext(t *testing.T) {
|
||||
// Two 8-GPU nodes and a 4-GPU profile: four replicas in all, one placed.
|
||||
h := newHarness(t, testModel(1), gpuNode("1", 8), gpuNode("2", 8))
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
|
||||
got, ok := h.r.Headroom("llama", 10)
|
||||
if !ok {
|
||||
t.Fatal("the model was not found")
|
||||
}
|
||||
if got.Current != 1 || got.More != 3 || got.Limited {
|
||||
t.Fatalf("headroom %+v, expected 1 placed and room for 3 more", got)
|
||||
}
|
||||
// Spread first: the next one goes to the node without one.
|
||||
if first := got.Nodes[0][0]; first == h.r.Replicas()[0].NodeIDs()[0] {
|
||||
t.Errorf("the next replica would stack on %s beside the first", first)
|
||||
}
|
||||
if len(got.Blocked) == 0 || !strings.Contains(strings.Join(got.Blocked, "\n"), "GPU") {
|
||||
t.Errorf("no reason given for the fifth: %v", got.Blocked)
|
||||
}
|
||||
// Tried on a copy: nothing was taken from the real inventory.
|
||||
if free := len(h.inv.FreeGPUs("1")) + len(h.inv.FreeGPUs("2")); free != 12 {
|
||||
t.Errorf("%d GPUs free after asking, expected the 12 the one replica leaves", free)
|
||||
}
|
||||
|
||||
capped, _ := h.r.Headroom("llama", 2)
|
||||
if capped.More != 2 || !capped.Limited || capped.Blocked != nil {
|
||||
t.Errorf("asked for at most 2: %+v", capped)
|
||||
}
|
||||
if _, ok := h.r.Headroom("nope", 5); ok {
|
||||
t.Error("an unknown model has headroom")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHeadroomCountsMemoryAlreadyClaimed(t *testing.T) {
|
||||
cpu := func(id string) inventory.Node {
|
||||
return inventory.Node{ID: id, MgmtAddr: "10.0.0." + id, State: inventory.Ready, MemoryTotalMB: 16 * 1024}
|
||||
}
|
||||
m := catalog.Model{Name: "small", Source: "/m.gguf", Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 6}}}
|
||||
h := newHarness(t, m, cpu("a"), cpu("b"))
|
||||
h.r.Reconcile(t.Context())
|
||||
|
||||
// 16 GB each at 6 GB a replica: two per node, one of them taken.
|
||||
got, _ := h.r.Headroom("small", 10)
|
||||
if got.Current != 1 || got.More != 3 {
|
||||
t.Errorf("headroom %+v, expected room for 3 more", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReservedCoresAreTheEnginesThreadsAndBoundTheRoom(t *testing.T) {
|
||||
cpu := inventory.Node{ID: "a", MgmtAddr: "10.0.0.1", State: inventory.Ready, MemoryTotalMB: 64 * 1024, CPUCount: 8}
|
||||
m := catalog.Model{Name: "coder", Source: "/m.gguf", Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 6, CPUCores: 3}}}
|
||||
h := newHarness(t, m, cpu)
|
||||
h.r.Reconcile(t.Context())
|
||||
|
||||
started := h.disp.startedShards()
|
||||
if len(started) != 1 || started[0].Spec.Threads != 3 {
|
||||
t.Fatalf("the engine was not told its 3 cores: %+v", started)
|
||||
}
|
||||
if a := h.inv.Allocations("a"); len(a) != 1 || a[0].CPUCores != 3 {
|
||||
t.Errorf("the cores were not reserved: %+v", a)
|
||||
}
|
||||
// Memory would allow nine more; cores allow one: 8 - 3 = 5, then 2.
|
||||
got, _ := h.r.Headroom("coder", 10)
|
||||
if got.More != 1 {
|
||||
t.Errorf("room for %d more, expected 1 by cores: %+v", got.More, got)
|
||||
}
|
||||
}
|
||||
@@ -83,6 +83,12 @@ type ModelHealth struct {
|
||||
// place one found nowhere; Placement says why.
|
||||
Unplaceable bool
|
||||
Placement string
|
||||
// Class is the capacity class it is treated as. Preempted counts the
|
||||
// replicas it has lost to models of a higher class since the control
|
||||
// plane started; HeldFor names the model it is giving room to now.
|
||||
Class string
|
||||
Preempted int
|
||||
HeldFor string
|
||||
}
|
||||
|
||||
// ModelHealth reports every model in the catalog.
|
||||
@@ -106,7 +112,10 @@ func (r *Reconciler) ModelHealth() []ModelHealth {
|
||||
out := make([]ModelHealth, 0, len(models))
|
||||
for _, m := range models {
|
||||
h := ModelHealth{Model: m.Name, Desired: desired[m.Name], Ready: ready[m.Name],
|
||||
Failures: r.attempts[m.Name]}
|
||||
Failures: r.attempts[m.Name], Class: m.EffectiveClass(), Preempted: r.preempted[m.Name]}
|
||||
if hd, ok := r.held[m.Name]; ok && r.now().Before(hd.Until) {
|
||||
h.HeldFor = hd.For
|
||||
}
|
||||
if p, ok := r.parked[m.Name]; ok {
|
||||
h.Parked, h.ParkedSince, h.Reason = true, p.Since, p.Reason
|
||||
}
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"sort"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
)
|
||||
|
||||
// Measured is what one replica of a model on a profile was measured to serve
|
||||
// while meeting its SLO, by `llmgw ctl bench --save`.
|
||||
type Measured struct {
|
||||
// MaxConcurrency is the most requests at once that still met the SLO.
|
||||
MaxConcurrency int
|
||||
// TokPerSec is the throughput at that concurrency.
|
||||
TokPerSec float64
|
||||
}
|
||||
|
||||
// SetMeasurements supplies benchmark results, keyed model/profile, read
|
||||
// afresh each time it is called. Without them scaling goes by queue alone and
|
||||
// profiles are chosen by priority alone.
|
||||
func (r *Reconciler) SetMeasurements(f func() map[string]Measured) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.measurements = f
|
||||
}
|
||||
|
||||
// measured is a model's measurement on one profile.
|
||||
func (r *Reconciler) measured(model, profile string) (Measured, bool) {
|
||||
r.mu.Lock()
|
||||
f := r.measurements
|
||||
r.mu.Unlock()
|
||||
if f == nil {
|
||||
return Measured{}, false
|
||||
}
|
||||
m, ok := f()[model+"/"+profile]
|
||||
return m, ok && m.MaxConcurrency > 0
|
||||
}
|
||||
|
||||
// orderedProfiles is a model's profiles as placement should try them: by
|
||||
// priority, and among equal priorities the one measured to serve the most per
|
||||
// GPU — or per core, for a profile with none — first. Unmeasured profiles keep
|
||||
// their catalog order behind measured ones of the same priority.
|
||||
//
|
||||
// Placement sorts by priority itself, so the order found here is written back
|
||||
// as priorities, descending, for it to keep.
|
||||
func (r *Reconciler) orderedProfiles(m catalog.Model) []placement.Profile {
|
||||
ps := m.PlacementProfiles()
|
||||
eff := make([]float64, len(ps))
|
||||
for i, p := range ps {
|
||||
meas, ok := r.measured(m.Name, p.Name)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
units := float64(max(p.GPUsPerNode, 0) * max(p.Nodes, 1))
|
||||
if units == 0 {
|
||||
units = float64(max(p.CPUCores, 1))
|
||||
}
|
||||
eff[i] = meas.TokPerSec / units
|
||||
}
|
||||
idx := make([]int, len(ps))
|
||||
for i := range idx {
|
||||
idx[i] = i
|
||||
}
|
||||
sort.SliceStable(idx, func(a, b int) bool {
|
||||
pa, pb := ps[idx[a]], ps[idx[b]]
|
||||
if pa.Priority != pb.Priority {
|
||||
return pa.Priority > pb.Priority
|
||||
}
|
||||
return eff[idx[a]] > eff[idx[b]]
|
||||
})
|
||||
out := make([]placement.Profile, len(ps))
|
||||
for rank, i := range idx {
|
||||
out[rank] = ps[i]
|
||||
out[rank].Priority = len(ps) - rank
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// MeasuredFor is the measurement that sizes a model: that of the profile its
|
||||
// replicas run on, or, with none running, of the first profile placement
|
||||
// would try that was measured.
|
||||
func (r *Reconciler) MeasuredFor(model string) (Measured, string, bool) {
|
||||
r.mu.Lock()
|
||||
profile := ""
|
||||
for _, rep := range r.replicas {
|
||||
if rep.Model == model && rep.State.Counts() {
|
||||
profile = rep.Profile
|
||||
break
|
||||
}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
if profile != "" {
|
||||
if m, ok := r.measured(model, profile); ok {
|
||||
return m, profile, true
|
||||
}
|
||||
}
|
||||
cm, ok := r.cat.Load().Resolve(model)
|
||||
if !ok {
|
||||
return Measured{}, "", false
|
||||
}
|
||||
for _, p := range r.orderedProfiles(cm) {
|
||||
if m, ok := r.measured(cm.Name, p.Name); ok {
|
||||
return m, p.Name, true
|
||||
}
|
||||
}
|
||||
return Measured{}, "", false
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
func measure(h *harness, byKey map[string]Measured) {
|
||||
h.r.SetMeasurements(func() map[string]Measured { return byKey })
|
||||
}
|
||||
|
||||
func TestMoreRequestsThanWereMeasuredToMeetTheSLOIsBusy(t *testing.T) {
|
||||
m := testModel(1)
|
||||
m.Scaling.Max = 2
|
||||
h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8))
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
// Six in flight, nothing waiting at the engine: by the queue it is fine.
|
||||
load := map[string]Load{"llama": {Replicas: 1, Running: 6}}
|
||||
hold := func() {
|
||||
h.r.RecordLoad(load)
|
||||
h.advance(61 * time.Second)
|
||||
h.r.RecordLoad(load)
|
||||
h.r.Reconcile(t.Context())
|
||||
}
|
||||
hold()
|
||||
if n := liveCount(h); n != 1 {
|
||||
t.Fatalf("scaled up with no measurement and an empty queue: %d", n)
|
||||
}
|
||||
// Measured to meet its SLO at four at once: six is past it.
|
||||
measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 4, TokPerSec: 200}})
|
||||
hold()
|
||||
if n := liveCount(h); n != 2 {
|
||||
t.Errorf("%d replicas with 6 in flight against a measured 4", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReplicaIsTakenBackOnlyIfTheRestWouldHaveRoomToSpare(t *testing.T) {
|
||||
m := testModel(1)
|
||||
m.Scaling.Max = 2
|
||||
h := newHarness(t, m, gpuNode("1", 8), gpuNode("2", 8))
|
||||
measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 4, TokPerSec: 200}})
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
h.r.mu.Lock()
|
||||
h.r.extra["llama"] = 1
|
||||
h.r.mu.Unlock()
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
if liveCount(h) != 2 {
|
||||
t.Fatal("setup: not at two replicas")
|
||||
}
|
||||
quietFor := func(l Load) {
|
||||
for range 3 {
|
||||
h.r.RecordLoad(map[string]Load{"llama": l})
|
||||
h.advance(DefaultConfig().ScaleDownAfter + time.Second)
|
||||
h.r.RecordLoad(map[string]Load{"llama": l})
|
||||
h.r.Reconcile(t.Context())
|
||||
}
|
||||
}
|
||||
// Nothing waiting, but three running: one replica left would be at 3 of
|
||||
// its 4, past 70%.
|
||||
quietFor(Load{Replicas: 2, Running: 3})
|
||||
if n := liveCount(h); n != 2 {
|
||||
t.Errorf("scaled down to the edge of what one replica holds: %d", n)
|
||||
}
|
||||
quietFor(Load{Replicas: 2, Running: 2})
|
||||
if n := liveCount(h); n != 1 {
|
||||
t.Errorf("two running fits one replica's 70%% of 4; still %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheProfileMeasuredToServeMostPerGPUIsTriedFirst(t *testing.T) {
|
||||
m := catalog.Model{Name: "llama", Source: "/models/llama", Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{
|
||||
{Name: "tp4", GPUClass: "h100-80g", TP: 4},
|
||||
{Name: "tp2", GPUClass: "h100-80g", TP: 2},
|
||||
{Name: "tp8", GPUClass: "h100-80g", TP: 8, Priority: 10},
|
||||
}}
|
||||
h := newHarness(t, m, gpuNode("1", 8))
|
||||
// tp4: 400/4 = 100 per GPU; tp2: 300/2 = 150. tp8 is unmeasured but has
|
||||
// the higher priority, which an operator set and a benchmark does not
|
||||
// overrule.
|
||||
measure(h, map[string]Measured{"llama/tp4": {MaxConcurrency: 8, TokPerSec: 400},
|
||||
"llama/tp2": {MaxConcurrency: 6, TokPerSec: 300}})
|
||||
cm, _ := h.r.cat.Load().Resolve("llama")
|
||||
var order []string
|
||||
for _, p := range h.r.orderedProfiles(cm) {
|
||||
order = append(order, p.Name)
|
||||
}
|
||||
if len(order) != 3 || order[0] != "tp8" || order[1] != "tp2" || order[2] != "tp4" {
|
||||
t.Errorf("order %v, expected tp8 (priority), then tp2 (150/GPU), then tp4 (100/GPU)", order)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,324 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
)
|
||||
|
||||
// hold keeps a model that gave up room from taking it back.
|
||||
type hold struct {
|
||||
Until time.Time
|
||||
For string
|
||||
}
|
||||
|
||||
// maxVictims bounds how many replicas one placement may stop. Past this the
|
||||
// model is better reported as not fitting than paid for with half a cluster.
|
||||
const maxVictims = 3
|
||||
|
||||
// victim is a replica that could be stopped for another model.
|
||||
type victim struct {
|
||||
ID string
|
||||
Model string
|
||||
Class string
|
||||
AboveMin bool
|
||||
Tier int
|
||||
Idle time.Duration
|
||||
// Size is what stopping it frees, for choosing the smaller of two
|
||||
// otherwise equal victims.
|
||||
Size quota.Amount
|
||||
}
|
||||
|
||||
// preemptFor stops replicas of lower classes when a model cannot otherwise be
|
||||
// placed, and says whether it did.
|
||||
//
|
||||
// What may be taken from whom:
|
||||
//
|
||||
// - A lower class gives way to a higher one: opportunistic to standard,
|
||||
// standard to critical.
|
||||
// - Within a class, only a replica above its model's minimum, and only for
|
||||
// a model still below its own. Room a model grew into under load is not a
|
||||
// promise; its minimum is.
|
||||
// - A critical model's minimum is never taken.
|
||||
//
|
||||
// Victims only ever come from a lower rank, or from above a minimum for a
|
||||
// model below one, so two models cannot take turns stopping each other.
|
||||
//
|
||||
// The set is tried on a copy of the inventory first, and only stopped if it
|
||||
// would let the model be placed: stopping replicas that do not make room is
|
||||
// the one outcome worse than not preempting.
|
||||
func (r *Reconciler) preemptFor(ctx context.Context, m catalog.Model, avoid []string) bool {
|
||||
cands := r.victims(m)
|
||||
if len(cands) == 0 {
|
||||
return false
|
||||
}
|
||||
// Candidates in order until the model fits; then every one whose
|
||||
// stopping turns out not to be needed is kept after all. Without that
|
||||
// second step an idle replica on the wrong node, tried first, would be
|
||||
// stopped along with the one that actually made the room.
|
||||
var chosen []victim
|
||||
for _, v := range cands {
|
||||
if len(chosen) == maxVictims {
|
||||
break
|
||||
}
|
||||
chosen = append(chosen, v)
|
||||
if !r.fitsWithout(m, avoid, chosen) {
|
||||
continue
|
||||
}
|
||||
for i := len(chosen) - 1; i >= 0 && len(chosen) > 1; i-- {
|
||||
fewer := append(append([]victim(nil), chosen[:i]...), chosen[i+1:]...)
|
||||
if r.fitsWithout(m, avoid, fewer) {
|
||||
chosen = fewer
|
||||
}
|
||||
}
|
||||
r.preempt(ctx, m, chosen)
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// fitsWithout reports whether m could be placed with these replicas gone,
|
||||
// tried on a copy of the inventory.
|
||||
func (r *Reconciler) fitsWithout(m catalog.Model, avoid []string, gone []victim) bool {
|
||||
trial := r.inv.Clone()
|
||||
for _, v := range gone {
|
||||
trial.ReleaseReplica(v.ID)
|
||||
}
|
||||
sched := placement.New(trial)
|
||||
res := sched.Place(r.orderedProfiles(m), avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
res = sched.Place(r.orderedProfiles(m), nil)
|
||||
}
|
||||
return res.Placement != nil
|
||||
}
|
||||
|
||||
// preempt drains the chosen replicas and holds their models back.
|
||||
func (r *Reconciler) preempt(ctx context.Context, m catalog.Model, chosen []victim) {
|
||||
now := r.now()
|
||||
var ids []string
|
||||
r.mu.Lock()
|
||||
for _, v := range chosen {
|
||||
if rep, ok := r.replicas[v.ID]; ok {
|
||||
rep.PreemptedFor = m.Name
|
||||
}
|
||||
ids = append(ids, v.ID)
|
||||
r.held[v.Model] = hold{Until: now.Add(r.cfg.PreemptHold), For: m.Name}
|
||||
r.preempted[v.Model]++
|
||||
}
|
||||
r.preempting[m.Name] = ids
|
||||
r.mu.Unlock()
|
||||
|
||||
for _, v := range chosen {
|
||||
r.log.Warn("preempting a replica to make room",
|
||||
"replica", v.ID, "model", v.Model, "class", v.Class, "above_min", v.AboveMin,
|
||||
"for", m.Name, "for_class", m.EffectiveClass())
|
||||
}
|
||||
r.scaleDown(ctx, ids)
|
||||
}
|
||||
|
||||
// awaitingPreemption reports whether replicas stopped for a model are still
|
||||
// on their way out, so it waits for them rather than stopping more.
|
||||
func (r *Reconciler) awaitingPreemption(model string) bool {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
ids := r.preempting[model]
|
||||
ids = slices.DeleteFunc(slices.Clone(ids), func(id string) bool {
|
||||
_, ok := r.replicas[id]
|
||||
return !ok
|
||||
})
|
||||
if len(ids) == 0 {
|
||||
delete(r.preempting, model)
|
||||
return false
|
||||
}
|
||||
r.preempting[model] = ids
|
||||
return true
|
||||
}
|
||||
|
||||
// isHeld reports whether a model gave up room and may not take it back yet.
|
||||
// The hold ends early once the model it was given to has what it asked for.
|
||||
//
|
||||
// Today a preempted replica is only removed by its own model's drain loop,
|
||||
// which runs after that model has decided not to scale up, so the room it
|
||||
// frees reaches the model it was freed for first anyway. The hold is what
|
||||
// keeps that true for any other way a replica can go — a stop reported
|
||||
// between passes, a reap at the start of one — rather than it resting on the
|
||||
// order of lines in reconcileModel.
|
||||
func (r *Reconciler) isHeld(model string) bool {
|
||||
r.mu.Lock()
|
||||
h, ok := r.held[model]
|
||||
r.mu.Unlock()
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
if r.now().After(h.Until) {
|
||||
r.mu.Lock()
|
||||
delete(r.held, model)
|
||||
r.mu.Unlock()
|
||||
return false
|
||||
}
|
||||
if to, ok := r.cat.Load().Resolve(h.For); ok && r.liveCount(h.For) >= r.desiredFor(to) {
|
||||
r.mu.Lock()
|
||||
delete(r.held, model)
|
||||
r.mu.Unlock()
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// liveCount is how many replicas of a model hold capacity.
|
||||
func (r *Reconciler) liveCount(model string) int {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
n := 0
|
||||
for _, rep := range r.replicas {
|
||||
if rep.Model == model && rep.State.Counts() && rep.RetiringAt.IsZero() {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// victims lists the replicas that may be stopped for m, in the order they
|
||||
// should be: idle on-demand ones first, then what other tenants borrowed past
|
||||
// their guarantees when m's owner is within its own, then what lower classes
|
||||
// grew into under load, then the same within m's class, then lower classes'
|
||||
// minimums; within each, the longest unused first.
|
||||
//
|
||||
// A replica within its owner's guarantee is taken only for a model of the
|
||||
// same owner: a guarantee that another tenant's higher class could take is
|
||||
// not one.
|
||||
func (r *Reconciler) victims(m catalog.Model) []victim {
|
||||
cat := r.cat.Load()
|
||||
now := r.now()
|
||||
myRank := catalog.ClassRank(m.EffectiveClass())
|
||||
belowMin := r.liveCount(m.Name) < m.Scaling.Min
|
||||
|
||||
pol := r.policy()
|
||||
var borrowed map[string]bool
|
||||
reclaim := false
|
||||
if pol != nil {
|
||||
hs := r.Holdings()
|
||||
borrowed = pol.Borrowed(hs)
|
||||
reclaim = pol.Guaranteed(m.Owner, pol.Usage(hs), profileNeed(m))
|
||||
}
|
||||
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
|
||||
// Per model, its live replicas newest first: the newest are the ones
|
||||
// above its minimum.
|
||||
byModel := map[string][]*Replica{}
|
||||
for _, rep := range r.replicas {
|
||||
if rep.Model == m.Name || !rep.State.Counts() || !rep.RetiringAt.IsZero() || rep.PreemptedFor != "" {
|
||||
continue
|
||||
}
|
||||
byModel[rep.Model] = append(byModel[rep.Model], rep)
|
||||
}
|
||||
|
||||
var out []victim
|
||||
for model, reps := range byModel {
|
||||
other, ok := cat.Resolve(model)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
class := other.EffectiveClass()
|
||||
rank := catalog.ClassRank(class)
|
||||
sort.Slice(reps, func(i, j int) bool { return reps[i].CreatedAt.After(reps[j].CreatedAt) })
|
||||
above := max(len(reps)-other.Scaling.Min, 0)
|
||||
for i, rep := range reps {
|
||||
aboveMin := i < above
|
||||
tier := -1
|
||||
switch {
|
||||
case rank < myRank && other.Scaling.OnDemand() && idleFor(rep, now) >= r.cfg.EvictionGrace:
|
||||
tier = 0
|
||||
case rank < myRank && aboveMin:
|
||||
tier = 2
|
||||
case rank == myRank && aboveMin && belowMin:
|
||||
tier = 3
|
||||
case rank < myRank && class != catalog.ClassCritical:
|
||||
tier = 4
|
||||
}
|
||||
if pol != nil && other.Owner != m.Owner {
|
||||
switch {
|
||||
case borrowed[rep.ID] && reclaim && (class != catalog.ClassCritical || aboveMin):
|
||||
// Past its owner's guarantee, wanted by an owner within
|
||||
// its own: taken back whatever the classes.
|
||||
if tier != 0 {
|
||||
tier = 1
|
||||
}
|
||||
case !borrowed[rep.ID] && guaranteed(pol, other.Owner):
|
||||
continue // held within its owner's guarantee
|
||||
}
|
||||
}
|
||||
if tier < 0 {
|
||||
continue
|
||||
}
|
||||
// Just started: stopping it throws away a model load that has
|
||||
// only now been paid for.
|
||||
if !rep.ReadyAt.IsZero() && now.Sub(rep.ReadyAt) < r.cfg.MinLifetime {
|
||||
continue
|
||||
}
|
||||
if len(nodesOf(rep)) > 1 {
|
||||
continue // see makeRoomFor: a multi-node replica is its own decision
|
||||
}
|
||||
prof, _ := other.ProfileByName(rep.Profile)
|
||||
var size quota.Amount
|
||||
for _, sh := range rep.Shards {
|
||||
size = size.Plus(quota.Amount{GPUs: len(sh.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
|
||||
}
|
||||
out = append(out, victim{ID: rep.ID, Model: model, Class: class,
|
||||
AboveMin: aboveMin, Tier: tier, Idle: idleFor(rep, now), Size: size})
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool {
|
||||
a, b := out[i], out[j]
|
||||
if a.Tier != b.Tier {
|
||||
return a.Tier < b.Tier
|
||||
}
|
||||
if a.Idle != b.Idle {
|
||||
return a.Idle > b.Idle
|
||||
}
|
||||
// The smaller first: the least served is lost if it is enough, and
|
||||
// the pruning after keeps a larger one only when it is needed.
|
||||
if a.Size.GPUs != b.Size.GPUs {
|
||||
return a.Size.GPUs < b.Size.GPUs
|
||||
}
|
||||
if a.Size.MemoryGB != b.Size.MemoryGB {
|
||||
return a.Size.MemoryGB < b.Size.MemoryGB
|
||||
}
|
||||
return a.ID < b.ID
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// idleFor is how long a replica has gone unused; zero when nobody has said,
|
||||
// which is not the same as idle.
|
||||
func idleFor(rep *Replica, now time.Time) time.Duration {
|
||||
if rep.LastUsedAt.IsZero() {
|
||||
return 0
|
||||
}
|
||||
return now.Sub(rep.LastUsedAt)
|
||||
}
|
||||
|
||||
func nodesOf(rep *Replica) map[string]struct{} {
|
||||
out := map[string]struct{}{}
|
||||
for _, sh := range rep.Shards {
|
||||
out[sh.NodeID] = struct{}{}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// guaranteed reports whether an owner has a guarantee anywhere up its tree.
|
||||
func guaranteed(pol *quota.Policy, owner string) bool {
|
||||
for _, q := range pol.Governing(owner) {
|
||||
if q.Guarantee.Any() {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,333 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
// newFleetHarness is newHarness with a catalog of several models.
|
||||
func newFleetHarness(t *testing.T, models []catalog.Model, nodes ...inventory.Node) *harness {
|
||||
t.Helper()
|
||||
cat, err := catalog.New(models)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inv := inventory.NewStore()
|
||||
c := newClock()
|
||||
for _, n := range nodes {
|
||||
inv.Upsert(n)
|
||||
}
|
||||
disp := &fakeDispatcher{}
|
||||
cfg := DefaultConfig()
|
||||
cfg.UnreachableAfter = 15 * time.Second
|
||||
cfg.LostAfter = 60 * time.Second
|
||||
r := New(cfg, catalog.NewLive(cat), inv, disp, slog.New(slog.DiscardHandler))
|
||||
r.SetClock(c.now)
|
||||
c.advance(cfg.SettleFor)
|
||||
for _, n := range nodes {
|
||||
touchAt(inv, n.ID, c.now())
|
||||
}
|
||||
return &harness{r: r, inv: inv, disp: disp, clock: c}
|
||||
}
|
||||
|
||||
// gpuModel runs on one whole 8-GPU node.
|
||||
func gpuModel(name, class string, min int) catalog.Model {
|
||||
return catalog.Model{Name: name, Source: "/models/" + name, Class: class,
|
||||
Scaling: catalog.Scaling{Min: min, Max: min},
|
||||
Profiles: []catalog.Profile{{Name: "tp8", Engine: "vllm", GPUClass: "h100-80g", TP: 8}}}
|
||||
}
|
||||
|
||||
// settle runs passes until nothing changes, marking what starts ready and
|
||||
// letting preempted replicas run out their grace.
|
||||
func (h *harness) settle(t *testing.T, passes int) {
|
||||
t.Helper()
|
||||
for range passes {
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
h.advance(h.r.cfg.PreemptGrace + time.Second)
|
||||
}
|
||||
}
|
||||
|
||||
func replicasOf(h *harness, model string) (live int) {
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.Model == model && rep.State.Counts() {
|
||||
live++
|
||||
}
|
||||
}
|
||||
return live
|
||||
}
|
||||
|
||||
// fill places the standard model, then ages it past MinLifetime so it may be
|
||||
// taken from, and only then adds the model that wants its node.
|
||||
func fillThenWant(t *testing.T, want catalog.Model, have catalog.Model, nodes ...inventory.Node) *harness {
|
||||
t.Helper()
|
||||
h := newFleetHarness(t, []catalog.Model{have, withMin(want, 0)}, nodes...)
|
||||
h.settle(t, 3)
|
||||
if got := replicasOf(h, have.Name); got != have.Scaling.Min {
|
||||
t.Fatalf("%s has %d replicas before anything wanted its room", have.Name, got)
|
||||
}
|
||||
h.advance(h.r.cfg.MinLifetime + time.Minute)
|
||||
cat, err := catalog.New([]catalog.Model{have, want})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.r.cat.Store(cat)
|
||||
return h
|
||||
}
|
||||
|
||||
func withMin(m catalog.Model, n int) catalog.Model {
|
||||
m.Scaling = catalog.Scaling{Min: n, Max: n}
|
||||
return m
|
||||
}
|
||||
|
||||
func TestACriticalModelTakesAStandardModelsNode(t *testing.T) {
|
||||
h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
|
||||
|
||||
h.r.Reconcile(t.Context())
|
||||
var draining int
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.Model == "chat" && rep.State == Draining && rep.PreemptedFor == "vital" {
|
||||
draining++
|
||||
}
|
||||
}
|
||||
if draining != 1 {
|
||||
t.Fatalf("the standard replica was not drained for the critical one: %+v", h.r.Replicas())
|
||||
}
|
||||
if hs := health(h, "chat"); hs.HeldFor != "vital" || hs.Preempted != 1 {
|
||||
t.Errorf("the victim's health does not say it gave way: %+v", hs)
|
||||
}
|
||||
|
||||
h.settle(t, 4)
|
||||
if replicasOf(h, "vital") != 1 {
|
||||
t.Fatalf("the critical model was not placed once room was made: %+v", h.r.Replicas())
|
||||
}
|
||||
// Held back until the critical model had its replica, then short again
|
||||
// because there is no room: it must not have taken the node back — not
|
||||
// even for a moment, which would load a model only to preempt it again.
|
||||
if replicasOf(h, "chat") != 0 {
|
||||
t.Error("the standard model took the node back")
|
||||
}
|
||||
started := 0
|
||||
for _, cmd := range h.disp.startedShards() {
|
||||
if strings.HasPrefix(cmd.ReplicaID, "chat-") {
|
||||
started++
|
||||
}
|
||||
}
|
||||
if started != 1 {
|
||||
t.Errorf("the standard model started %d replicas; it should not have tried again", started)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStandardModelDoesNotTakeAnothersMinimum(t *testing.T) {
|
||||
h := fillThenWant(t, gpuModel("second", "", 1), gpuModel("first", "", 1), gpuNode("1", 8))
|
||||
h.settle(t, 3)
|
||||
if replicasOf(h, "first") != 1 || replicasOf(h, "second") != 0 {
|
||||
t.Errorf("a model of the same class took another's minimum: %+v", h.r.Replicas())
|
||||
}
|
||||
if hs := health(h, "second"); !hs.Unplaceable {
|
||||
t.Error("the model left without room should say it cannot be placed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithinAClassOnlyWhatWasGrownIntoIsTaken(t *testing.T) {
|
||||
grown := gpuModel("busy", "", 1)
|
||||
grown.Scaling = catalog.Scaling{Min: 1, Max: 2}
|
||||
h := newFleetHarness(t, []catalog.Model{grown, withMin(gpuModel("new", "", 1), 0)}, gpuNode("1", 8), gpuNode("2", 8))
|
||||
h.settle(t, 2)
|
||||
// Load grows it to its max: the second replica is above its minimum.
|
||||
h.r.mu.Lock()
|
||||
h.r.extra["busy"] = 1
|
||||
h.r.mu.Unlock()
|
||||
h.settle(t, 3)
|
||||
if replicasOf(h, "busy") != 2 {
|
||||
t.Fatalf("setup: busy did not grow to 2: %+v", h.r.Replicas())
|
||||
}
|
||||
h.advance(h.r.cfg.MinLifetime + time.Minute)
|
||||
cat, _ := catalog.New([]catalog.Model{grown, gpuModel("new", "", 1)})
|
||||
h.r.cat.Store(cat)
|
||||
|
||||
h.settle(t, 5)
|
||||
if replicasOf(h, "new") != 1 || replicasOf(h, "busy") != 1 {
|
||||
t.Errorf("the grown replica should give way to a minimum: busy %d, new %d",
|
||||
replicasOf(h, "busy"), replicasOf(h, "new"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestNothingIsStoppedThatWouldNotMakeRoom(t *testing.T) {
|
||||
// Two nodes, each half used by an opportunistic model; the critical one
|
||||
// needs a whole node. Stopping one frees half a node, which is no use —
|
||||
// and both together are on two nodes, which is no use either.
|
||||
half := catalog.Model{Name: "half", Source: "/m", Class: catalog.ClassOpportunistic,
|
||||
Scaling: catalog.Scaling{Min: 2, Max: 2},
|
||||
Profiles: []catalog.Profile{{Name: "tp4", Engine: "vllm", GPUClass: "h100-80g", TP: 4}}}
|
||||
five := func(id string) inventory.Node { return gpuNode(id, 5) }
|
||||
h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), half, five("1"), five("2"))
|
||||
h.r.Reconcile(t.Context())
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.PreemptedFor != "" || rep.State == Draining {
|
||||
t.Fatalf("a replica was stopped without making room: %+v", rep)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestACriticalMinimumIsNeverTaken(t *testing.T) {
|
||||
h := fillThenWant(t, gpuModel("also-vital", catalog.ClassCritical, 1), gpuModel("vital", catalog.ClassCritical, 1), gpuNode("1", 8))
|
||||
h.settle(t, 3)
|
||||
if replicasOf(h, "vital") != 1 {
|
||||
t.Error("a critical model's minimum was taken")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReplicaThatJustStartedIsLeftAlone(t *testing.T) {
|
||||
h := newFleetHarness(t, []catalog.Model{gpuModel("chat", "", 1), withMin(gpuModel("vital", catalog.ClassCritical, 1), 0)}, gpuNode("1", 8))
|
||||
h.settle(t, 2)
|
||||
cat, _ := catalog.New([]catalog.Model{gpuModel("chat", "", 1), gpuModel("vital", catalog.ClassCritical, 1)})
|
||||
h.r.cat.Store(cat)
|
||||
h.r.Reconcile(t.Context())
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.State == Draining {
|
||||
t.Fatal("a replica that had only just loaded was preempted")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func health(h *harness, model string) ModelHealth {
|
||||
for _, x := range h.r.ModelHealth() {
|
||||
if x.Model == model {
|
||||
return x
|
||||
}
|
||||
}
|
||||
return ModelHealth{}
|
||||
}
|
||||
|
||||
func TestAHoldLastsUntilTheRoomIsUsedOrItExpires(t *testing.T) {
|
||||
h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
|
||||
h.r.Reconcile(t.Context())
|
||||
if !h.r.isHeld("chat") {
|
||||
t.Fatal("the model that gave way is not held")
|
||||
}
|
||||
if h.r.isHeld("vital") {
|
||||
t.Error("the model room was made for is held")
|
||||
}
|
||||
// Released as soon as the model it was given to has what it asked for.
|
||||
h.settle(t, 3)
|
||||
if replicasOf(h, "vital") != 1 {
|
||||
t.Fatalf("setup: vital was not placed: %+v", h.r.Replicas())
|
||||
}
|
||||
if h.r.isHeld("chat") {
|
||||
t.Error("the hold outlasted its purpose")
|
||||
}
|
||||
|
||||
// And, whatever happens, after PreemptHold.
|
||||
h.r.mu.Lock()
|
||||
h.r.held["other"] = hold{Until: h.r.now().Add(time.Minute), For: "never-placed"}
|
||||
h.r.mu.Unlock()
|
||||
if !h.r.isHeld("other") {
|
||||
t.Fatal("a fresh hold is not in force")
|
||||
}
|
||||
h.advance(2 * time.Minute)
|
||||
if h.r.isHeld("other") {
|
||||
t.Error("a hold outlived its time")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyTheReplicasThatMakeTheRoomAreStopped(t *testing.T) {
|
||||
// Two half-used nodes. "idle" on node 1 is on demand and long unused, so
|
||||
// it is first in line; but the critical model needs node 2's other half,
|
||||
// which only stopping "chat" frees.
|
||||
idle := catalog.Model{Name: "idle", Source: "/m", Scaling: catalog.Scaling{Min: 0, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "tp4", Engine: "vllm", GPUClass: "h100-80g", TP: 4}}}
|
||||
chat := catalog.Model{Name: "chat", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "tp6", Engine: "vllm", GPUClass: "h100-80g", TP: 6}}}
|
||||
vital := catalog.Model{Name: "vital", Source: "/m", Class: catalog.ClassCritical,
|
||||
Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "tp8", Engine: "vllm", GPUClass: "h100-80g", TP: 8}}}
|
||||
h := newFleetHarness(t, []catalog.Model{idle, chat}, gpuNode("1", 6), gpuNode("2", 8))
|
||||
h.r.Want("idle")
|
||||
h.settle(t, 4)
|
||||
var idleNode, chatNode string
|
||||
for _, rep := range h.r.Replicas() {
|
||||
switch rep.Model {
|
||||
case "idle":
|
||||
idleNode = rep.Shards[0].NodeID
|
||||
case "chat":
|
||||
chatNode = rep.Shards[0].NodeID
|
||||
}
|
||||
}
|
||||
if idleNode == "" || chatNode != "2" {
|
||||
t.Fatalf("setup: idle on %q, chat on %q; chat must hold node 2: %+v", idleNode, chatNode, h.r.Replicas())
|
||||
}
|
||||
h.r.mu.Lock()
|
||||
for _, rep := range h.r.replicas {
|
||||
rep.LastUsedAt = h.r.now().Add(-time.Hour)
|
||||
}
|
||||
h.r.mu.Unlock()
|
||||
h.advance(h.r.cfg.MinLifetime + time.Minute)
|
||||
cat, _ := catalog.New([]catalog.Model{idle, chat, vital})
|
||||
h.r.cat.Store(cat)
|
||||
|
||||
h.r.Reconcile(t.Context())
|
||||
var stopped []string
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.PreemptedFor == "vital" {
|
||||
stopped = append(stopped, rep.Model)
|
||||
}
|
||||
}
|
||||
if len(stopped) != 1 || stopped[0] != "chat" {
|
||||
t.Errorf("stopped %v; only chat's replica makes room for a whole node", stopped)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOfTwoThatWouldEachMakeRoomTheSmallerIsStopped(t *testing.T) {
|
||||
cpu := inventory.Node{ID: "a", MgmtAddr: "10.0.0.1", State: inventory.Ready, MemoryTotalMB: 31 * 1024, CPUCount: 8}
|
||||
big := catalog.Model{Name: "big", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 12, CPUCores: 4}}}
|
||||
small := catalog.Model{Name: "small", Source: "/m", Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 2, CPUCores: 1}}}
|
||||
vital := catalog.Model{Name: "vital", Source: "/m", Class: catalog.ClassCritical,
|
||||
Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 15, CPUCores: 4}}}
|
||||
// 8 cores: big 4 + small 1 leaves 3, and vital needs 4. Stopping either
|
||||
// is enough; stopping big loses far more.
|
||||
for range 5 { // the choice must not depend on map order
|
||||
h := newFleetHarness(t, []catalog.Model{big, small}, cpu)
|
||||
h.settle(t, 3)
|
||||
h.advance(h.r.cfg.MinLifetime + time.Minute)
|
||||
cat, _ := catalog.New([]catalog.Model{big, small, vital})
|
||||
h.r.cat.Store(cat)
|
||||
h.r.Reconcile(t.Context())
|
||||
var stopped []string
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.PreemptedFor == "vital" {
|
||||
stopped = append(stopped, rep.Model)
|
||||
}
|
||||
}
|
||||
if len(stopped) != 1 || stopped[0] != "small" {
|
||||
t.Fatalf("stopped %v; the small replica alone makes room", stopped)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestHeadroomCountsWhatAHigherClassCouldTake(t *testing.T) {
|
||||
h := fillThenWant(t, gpuModel("vital", catalog.ClassCritical, 1), gpuModel("chat", "", 1), gpuNode("1", 8))
|
||||
got, _ := h.r.Headroom("vital", 3)
|
||||
if got.More != 0 || got.ByPreempting != 1 || len(got.Preempts) != 1 || got.Preempts[0] != "chat" {
|
||||
t.Errorf("a critical model should see one more by stopping chat: %+v", got)
|
||||
}
|
||||
// Tried on a copy: nothing was stopped.
|
||||
if replicasOf(h, "chat") != 1 {
|
||||
t.Fatal("asking about room stopped a replica")
|
||||
}
|
||||
// A model of chat's own class may not take its minimum, so has no such room.
|
||||
peer := newFleetHarness(t, []catalog.Model{gpuModel("chat", "", 1), withMin(gpuModel("peer", "", 1), 0)}, gpuNode("1", 8))
|
||||
peer.settle(t, 3)
|
||||
peer.advance(peer.r.cfg.MinLifetime + time.Minute)
|
||||
if got, _ := peer.r.Headroom("peer", 3); got.ByPreempting != 0 {
|
||||
t.Errorf("a standard model counted room in another's minimum: %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// SetQuotas supplies the tenancy tree's capacity quotas, read afresh each
|
||||
// time it is called. Without it nothing is guaranteed or limited, and
|
||||
// preemption goes by class alone.
|
||||
func (r *Reconciler) SetQuotas(f func() *quota.Policy) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.quotas = f
|
||||
}
|
||||
|
||||
func (r *Reconciler) policy() *quota.Policy {
|
||||
r.mu.Lock()
|
||||
f := r.quotas
|
||||
r.mu.Unlock()
|
||||
if f == nil {
|
||||
return nil
|
||||
}
|
||||
return f()
|
||||
}
|
||||
|
||||
// Holdings is what every replica holds, and for whom.
|
||||
func (r *Reconciler) Holdings() []quota.Holding {
|
||||
cat := r.cat.Load()
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
out := make([]quota.Holding, 0, len(r.replicas))
|
||||
for _, rep := range r.replicas {
|
||||
m, ok := cat.Resolve(rep.Model)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
prof, _ := m.ProfileByName(rep.Profile)
|
||||
var a quota.Amount
|
||||
for _, sh := range rep.Shards {
|
||||
a = a.Plus(quota.Amount{GPUs: len(sh.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
|
||||
}
|
||||
out = append(out, quota.Holding{Replica: rep.ID, Owner: m.Owner, Amount: a, Since: rep.CreatedAt})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// placementNeed is what a placement would hold.
|
||||
func placementNeed(m catalog.Model, pl *placement.Placement) quota.Amount {
|
||||
prof, _ := m.ProfileByName(pl.Profile.Name)
|
||||
var a quota.Amount
|
||||
for _, n := range pl.Nodes {
|
||||
a = a.Plus(quota.Amount{GPUs: len(n.GPUIndices), CPUCores: cpuCores(prof), MemoryGB: prof.MinMemoryGB})
|
||||
}
|
||||
return a
|
||||
}
|
||||
|
||||
// profileNeed is what a replica of a model's first profile would hold, for
|
||||
// deciding before there is a placement.
|
||||
func profileNeed(m catalog.Model) quota.Amount {
|
||||
if len(m.Profiles) == 0 {
|
||||
return quota.Amount{}
|
||||
}
|
||||
p := m.Profiles[0]
|
||||
nodes := max(p.Nodes, 1)
|
||||
gpus := p.GPUsPerNode
|
||||
if gpus == 0 && !p.CPUOnly() {
|
||||
gpus = max(p.TP, 1)
|
||||
}
|
||||
return quota.Amount{GPUs: gpus * nodes, CPUCores: cpuCores(p) * nodes, MemoryGB: p.MinMemoryGB * nodes}
|
||||
}
|
||||
|
||||
// overLimit says why a placement would take its model's owner past a limit.
|
||||
func (r *Reconciler) overLimit(m catalog.Model, pl *placement.Placement) string {
|
||||
pol := r.policy()
|
||||
if pol == nil || m.Owner == "" {
|
||||
return ""
|
||||
}
|
||||
return pol.AtLimit(m.Owner, pol.Usage(r.Holdings()), placementNeed(m, pl))
|
||||
}
|
||||
|
||||
// QuotaUse is one scope's quota beside what it holds.
|
||||
type QuotaUse struct {
|
||||
Scope string
|
||||
Guarantee store.Resources
|
||||
Limit store.Resources
|
||||
Holds quota.Amount
|
||||
}
|
||||
|
||||
// QuotaUse lists every quota with what its scope holds now.
|
||||
func (r *Reconciler) QuotaUse() []QuotaUse {
|
||||
pol := r.policy()
|
||||
if pol == nil {
|
||||
return nil
|
||||
}
|
||||
usage := pol.Usage(r.Holdings())
|
||||
var out []QuotaUse
|
||||
for _, q := range pol.Quotas() {
|
||||
key := quota.Key(q.Scope, q.ScopeID)
|
||||
out = append(out, QuotaUse{Scope: key, Guarantee: q.Guarantee, Limit: q.Limit, Holds: usage[key]})
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func owned(m catalog.Model, owner string) catalog.Model { m.Owner = owner; return m }
|
||||
|
||||
func gpus(n int) store.Resources { return store.Resources{GPUs: &n} }
|
||||
|
||||
// Two teams in one org; each 8-GPU node holds one tp8 replica.
|
||||
func withQuotas(h *harness, qs ...store.CapacityQuota) {
|
||||
pol := quota.New(qs, map[string][]string{"team:ml": {"org:acme"}, "team:web": {"org:acme"}})
|
||||
h.r.SetQuotas(func() *quota.Policy { return pol })
|
||||
}
|
||||
|
||||
func TestATenantWithinItsGuaranteeTakesBackWhatAnotherBorrowed(t *testing.T) {
|
||||
// web is guaranteed one node and borrowed the second; ml is guaranteed
|
||||
// one node and wants it. Same class, and web's replica is at its minimum
|
||||
// — by class alone nothing could be taken.
|
||||
web := owned(gpuModel("web-chat", "", 2), "team:web")
|
||||
ml := owned(gpuModel("ml-train", "", 1), "team:ml")
|
||||
h := newFleetHarness(t, []catalog.Model{web, withMin(ml, 0)}, gpuNode("1", 8), gpuNode("2", 8))
|
||||
withQuotas(h,
|
||||
store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "web", Guarantee: gpus(8)},
|
||||
store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "ml", Guarantee: gpus(8)})
|
||||
h.settle(t, 4)
|
||||
if replicasOf(h, "web-chat") != 2 {
|
||||
t.Fatalf("setup: web did not borrow the second node: %+v", h.r.Replicas())
|
||||
}
|
||||
h.advance(h.r.cfg.MinLifetime + time.Minute)
|
||||
cat, _ := catalog.New([]catalog.Model{web, ml})
|
||||
h.r.cat.Store(cat)
|
||||
|
||||
h.settle(t, 5)
|
||||
if replicasOf(h, "ml-train") != 1 || replicasOf(h, "web-chat") != 1 {
|
||||
t.Errorf("the borrowed node was not taken back: web %d, ml %d",
|
||||
replicasOf(h, "web-chat"), replicasOf(h, "ml-train"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReplicaWithinItsGuaranteeIsNotTakenByAHigherClassElsewhere(t *testing.T) {
|
||||
web := owned(gpuModel("web-chat", "", 1), "team:web")
|
||||
vital := owned(gpuModel("vital", catalog.ClassCritical, 1), "team:ml")
|
||||
h := fillThenWant(t, vital, web, gpuNode("1", 8))
|
||||
withQuotas(h, store.CapacityQuota{Scope: store.ScopeTeam, ScopeID: "web", Guarantee: gpus(8)})
|
||||
h.settle(t, 3)
|
||||
if replicasOf(h, "web-chat") != 1 || replicasOf(h, "vital") != 0 {
|
||||
t.Errorf("a guaranteed replica was preempted by another tenant's class: %+v", h.r.Replicas())
|
||||
}
|
||||
// Without the guarantee, critical takes it: the guarantee was the reason.
|
||||
h2 := fillThenWant(t, vital, web, gpuNode("1", 8))
|
||||
h2.settle(t, 3)
|
||||
if replicasOf(h2, "vital") != 1 {
|
||||
t.Error("setup: without a guarantee the critical model should have taken the node")
|
||||
}
|
||||
}
|
||||
|
||||
func TestALimitStopsAModelBeforeItStarts(t *testing.T) {
|
||||
web := owned(gpuModel("web-chat", "", 2), "team:web")
|
||||
h := newFleetHarness(t, []catalog.Model{web}, gpuNode("1", 8), gpuNode("2", 8))
|
||||
withQuotas(h, store.CapacityQuota{Scope: store.ScopeOrg, ScopeID: "acme", Limit: gpus(8)})
|
||||
h.settle(t, 4)
|
||||
if replicasOf(h, "web-chat") != 1 {
|
||||
t.Errorf("a second replica went past the org's 8-GPU limit: %+v", h.r.Replicas())
|
||||
}
|
||||
hs := health(h, "web-chat")
|
||||
if !hs.Unplaceable || !strings.Contains(hs.Placement, "org:acme is at its capacity limit (16 GPUs of 8)") {
|
||||
t.Errorf("the refusal should name the limit: %+v", hs)
|
||||
}
|
||||
room, _ := h.r.Headroom("web-chat", 5)
|
||||
if room.More != 0 || len(room.Blocked) != 1 || !strings.Contains(room.Blocked[0], "capacity limit") {
|
||||
t.Errorf("headroom should stop at the limit, not the empty node: %+v", room)
|
||||
}
|
||||
}
|
||||
@@ -15,6 +15,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
)
|
||||
|
||||
// ShardCommand is everything an agent needs to launch one shard.
|
||||
@@ -65,6 +66,12 @@ type Config struct {
|
||||
// Without a bound, draining a node whose model cannot be placed anywhere
|
||||
// else would never finish.
|
||||
DrainGrace time.Duration
|
||||
// PreemptGrace is how long a replica stopped to make room for another
|
||||
// is given to finish what it is serving: a request, not a replacement.
|
||||
PreemptGrace time.Duration
|
||||
// PreemptHold is how long a model that gave up room may not take it back,
|
||||
// so the replica it was given up for is placed first.
|
||||
PreemptHold time.Duration
|
||||
// ScaleUpWaiting is the average number of requests waiting per ready
|
||||
// replica, held for ScaleUpAfter, that adds a replica — up to the model's
|
||||
// max. ScaleDownAfter is how long with nothing waiting before one of the
|
||||
@@ -141,6 +148,8 @@ func DefaultConfig() Config {
|
||||
EvictionGrace: 2 * time.Minute,
|
||||
MinLifetime: 5 * time.Minute,
|
||||
DrainGrace: 5 * time.Minute,
|
||||
PreemptGrace: 30 * time.Second,
|
||||
PreemptHold: 5 * time.Minute,
|
||||
ParkAfter: 5,
|
||||
ScaleUpWaiting: 1,
|
||||
ScaleUpAfter: time.Minute,
|
||||
@@ -158,6 +167,12 @@ func (c *Config) normalize() {
|
||||
if c.BaseBackoff <= 0 {
|
||||
c.BaseBackoff = d.BaseBackoff
|
||||
}
|
||||
if c.PreemptGrace <= 0 {
|
||||
c.PreemptGrace = d.PreemptGrace
|
||||
}
|
||||
if c.PreemptHold <= 0 {
|
||||
c.PreemptHold = d.PreemptHold
|
||||
}
|
||||
if c.MaxBackoff < c.BaseBackoff {
|
||||
c.MaxBackoff = max(c.BaseBackoff, d.MaxBackoff)
|
||||
}
|
||||
@@ -250,6 +265,16 @@ type Reconciler struct {
|
||||
// placementLogged remembers the last "no placement" said per model, so it
|
||||
// is said when it changes rather than on every five-second pass.
|
||||
placementLogged map[string]placementNote
|
||||
// preempting is, per model, the replicas stopped to make room for it and
|
||||
// not gone yet; held is, per model that gave them up, until when it may
|
||||
// not take the room back; preempted counts, per model, replicas it lost.
|
||||
preempting map[string][]string
|
||||
held map[string]hold
|
||||
preempted map[string]int
|
||||
// quotas is the tenancy tree's capacity quotas as last read; nil is none.
|
||||
quotas func() *quota.Policy
|
||||
// measurements are benchmark results by model/profile; nil is none.
|
||||
measurements func() map[string]Measured
|
||||
|
||||
wake chan struct{}
|
||||
stop chan struct{}
|
||||
@@ -285,6 +310,9 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
|
||||
quietSince: map[string]time.Time{},
|
||||
placementLogged: map[string]placementNote{},
|
||||
lastExplain: map[string]placement.Result{},
|
||||
preempting: map[string][]string{},
|
||||
held: map[string]hold{},
|
||||
preempted: map[string]int{},
|
||||
wake: make(chan struct{}, 1),
|
||||
stop: make(chan struct{}),
|
||||
done: make(chan struct{}),
|
||||
@@ -704,6 +732,9 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
switch {
|
||||
case len(liveIDs) < desired && isParked:
|
||||
// Not placed. Its failure is on every status the operator reads.
|
||||
case len(liveIDs) < desired && r.isHeld(m.Name):
|
||||
// It gave up room to a model of a higher class, which is placed
|
||||
// first; taking it back now would undo the preemption.
|
||||
case len(liveIDs) < desired:
|
||||
if r.now().Before(blockedUntil) {
|
||||
// Still in backoff. Rebuilding immediately would hot-loop on a
|
||||
@@ -724,8 +755,12 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
r.mu.Lock()
|
||||
rep, ok := r.replicas[id]
|
||||
var expired bool
|
||||
grace, preemptedFor := r.cfg.DrainGrace, ""
|
||||
if ok {
|
||||
expired = !rep.DrainingAt.IsZero() && r.now().Sub(rep.DrainingAt) > r.cfg.DrainGrace
|
||||
if rep.PreemptedFor != "" {
|
||||
grace, preemptedFor = r.cfg.PreemptGrace, rep.PreemptedFor
|
||||
}
|
||||
expired = !rep.DrainingAt.IsZero() && r.now().Sub(rep.DrainingAt) > grace
|
||||
}
|
||||
r.mu.Unlock()
|
||||
|
||||
@@ -733,11 +768,15 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
case !ok:
|
||||
case replaced:
|
||||
r.stopReplica(ctx, id)
|
||||
case expired && preemptedFor != "":
|
||||
r.log.Info("stopping a preempted replica", "replica", id,
|
||||
"for", preemptedFor, "after", grace)
|
||||
r.stopReplica(ctx, id)
|
||||
case expired:
|
||||
// The replacement never arrived — often because nowhere else can
|
||||
// host it. Draining forever would leave the node undrainable.
|
||||
r.log.Warn("stopping a drained replica without a replacement",
|
||||
"replica", id, "after", r.cfg.DrainGrace)
|
||||
"replica", id, "after", grace)
|
||||
r.stopReplica(ctx, id)
|
||||
}
|
||||
}
|
||||
@@ -751,12 +790,13 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
// allocated, and starting several at once from the same inventory snapshot
|
||||
// would hand them the same GPUs.
|
||||
func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []string) {
|
||||
res := r.sched.Place(m.PlacementProfiles(), avoid)
|
||||
profiles := r.orderedProfiles(m)
|
||||
res := r.sched.Place(profiles, avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
// Spreading across failure domains is a preference, not a rule. If it
|
||||
// is the only thing standing in the way, take the placement anyway:
|
||||
// a second replica on one node still survives an engine crash.
|
||||
res = r.sched.Place(m.PlacementProfiles(), nil)
|
||||
res = r.sched.Place(profiles, nil)
|
||||
}
|
||||
|
||||
r.mu.Lock()
|
||||
@@ -775,9 +815,22 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
|
||||
if m.Scaling.OnDemand() && r.makeRoomFor(ctx, m) {
|
||||
return
|
||||
}
|
||||
if r.awaitingPreemption(m.Name) || r.preemptFor(ctx, m, avoid) {
|
||||
return
|
||||
}
|
||||
r.notePlacementFailure(m.Name, res.Explain())
|
||||
return
|
||||
}
|
||||
if why := r.overLimit(m, res.Placement); why != "" {
|
||||
// Recorded as the placement's answer, so the model reads as not
|
||||
// placeable, and says why, wherever that is shown.
|
||||
r.mu.Lock()
|
||||
r.lastExplain[m.Name] = placement.Result{Rejections: []placement.Rejection{
|
||||
{Profile: res.Placement.Profile.Name, Reason: why}}}
|
||||
r.mu.Unlock()
|
||||
r.notePlacementFailure(m.Name, why)
|
||||
return
|
||||
}
|
||||
if err := r.startReplica(ctx, m, res.Placement); err != nil {
|
||||
r.log.Error("starting replica failed", "model", m.Name, "err", err)
|
||||
}
|
||||
@@ -825,7 +878,7 @@ func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *plac
|
||||
})
|
||||
r.inv.Allocate(a.NodeID, inventory.Allocation{
|
||||
ShardID: shardID, ReplicaID: id, GPUIndices: a.GPUIndices,
|
||||
MemoryGB: prof.MinMemoryGB,
|
||||
MemoryGB: prof.MinMemoryGB, CPUCores: cpuCores(prof),
|
||||
})
|
||||
}
|
||||
rep.Endpoint = fmt.Sprintf("http://%s:%d", addrOf(leaderNode), basePort)
|
||||
@@ -869,6 +922,7 @@ func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *plac
|
||||
HTTPPort: s.HTTPPort,
|
||||
Profile: espec,
|
||||
APIKey: r.engineKey(id),
|
||||
Threads: cpuCores(prof),
|
||||
},
|
||||
StartupTimeout: prof.StartupTimeout,
|
||||
Pool: pl.Pool,
|
||||
@@ -1108,16 +1162,31 @@ func addrOf(n inventory.Node) string {
|
||||
// claim on restart would place a second large model onto a node already full of
|
||||
// the first.
|
||||
func (r *Reconciler) profileMemory(model, profile string) int {
|
||||
p, _ := r.profileOf(model, profile)
|
||||
return p.MinMemoryGB
|
||||
}
|
||||
|
||||
// profileOf is a replica's profile as the catalog holds it now.
|
||||
func (r *Reconciler) profileOf(model, profile string) (catalog.Profile, bool) {
|
||||
m, ok := r.cat.Load().Resolve(model)
|
||||
if !ok {
|
||||
return 0
|
||||
return catalog.Profile{}, false
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
if p.Name == profile {
|
||||
return p.MinMemoryGB
|
||||
return p, true
|
||||
}
|
||||
}
|
||||
return 0
|
||||
return catalog.Profile{}, false
|
||||
}
|
||||
|
||||
// cpuCores is what a profile reserves of a node's CPUs: its cores when it
|
||||
// computes on them, nothing when a GPU does the work.
|
||||
func cpuCores(p catalog.Profile) int {
|
||||
if !p.CPUOnly() {
|
||||
return 0
|
||||
}
|
||||
return p.CPUCores
|
||||
}
|
||||
|
||||
func (r *Reconciler) engineKey(replicaID string) string {
|
||||
|
||||
@@ -156,6 +156,8 @@ type Replica struct {
|
||||
// DrainingAt is when the replica stopped taking new work, so the wait for a
|
||||
// replacement can be bounded.
|
||||
DrainingAt time.Time
|
||||
// PreemptedFor is the model this replica was stopped to make room for.
|
||||
PreemptedFor string
|
||||
// RetiringAt is set while the replica is being restarted: it keeps serving
|
||||
// but no longer counts toward the model's replicas, so a replacement is
|
||||
// placed beside it, and it is drained only once that replacement is ready.
|
||||
|
||||
@@ -185,6 +185,11 @@ func (r *Router) Handler() http.Handler {
|
||||
pages.RegisterAdminUsers(mux, r.store)
|
||||
// Quotas are written on the same tree the accounts page builds, so
|
||||
// they are mounted together.
|
||||
// Capacity quotas are the control plane's, reached through the
|
||||
// same admin API the CLI uses.
|
||||
if c, ok := r.clusterView.(web.CapacityAdmin); ok {
|
||||
pages.SetCapacity(c)
|
||||
}
|
||||
pages.RegisterAdminQuotas(mux, r.store, r.acct)
|
||||
// And the report of what that tree actually used.
|
||||
pages.RegisterAdminUsage(mux, r.store, r.acct)
|
||||
|
||||
@@ -29,6 +29,9 @@ var copyOrder = []string{
|
||||
"key_claims",
|
||||
"sessions",
|
||||
"limits",
|
||||
// Beside limits, and like them keyed by scope with no foreign key: a
|
||||
// quota on a scope that is later removed is inert, not an error.
|
||||
"capacity_quotas",
|
||||
"usage",
|
||||
"models",
|
||||
// No foreign key to models, deliberately: a measurement outlives the
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
-- Capacity a part of the tenancy tree is guaranteed, and the most it may hold.
|
||||
--
|
||||
-- Beside limits rather than in it: limits are counted on the request path, per
|
||||
-- key, in requests and tokens; these are counted by the reconciler, per model,
|
||||
-- in the GPUs, cores and memory its replicas hold. A NULL is a resource this
|
||||
-- row does not govern: guaranteed nothing, or limited by nothing.
|
||||
CREATE TABLE capacity_quotas (
|
||||
scope_type TEXT NOT NULL,
|
||||
scope_id TEXT NOT NULL,
|
||||
gpus INTEGER,
|
||||
cpu_cores INTEGER,
|
||||
memory_gb INTEGER,
|
||||
limit_gpus INTEGER,
|
||||
limit_cpu_cores INTEGER,
|
||||
limit_memory_gb INTEGER,
|
||||
updated_at BIGINT NOT NULL,
|
||||
updated_by TEXT NOT NULL,
|
||||
PRIMARY KEY (scope_type, scope_id)
|
||||
);
|
||||
@@ -0,0 +1,19 @@
|
||||
-- Capacity a part of the tenancy tree is guaranteed, and the most it may hold.
|
||||
--
|
||||
-- Beside limits rather than in it: limits are counted on the request path, per
|
||||
-- key, in requests and tokens; these are counted by the reconciler, per model,
|
||||
-- in the GPUs, cores and memory its replicas hold. A NULL is a resource this
|
||||
-- row does not govern: guaranteed nothing, or limited by nothing.
|
||||
CREATE TABLE capacity_quotas (
|
||||
scope_type TEXT NOT NULL,
|
||||
scope_id TEXT NOT NULL,
|
||||
gpus INTEGER,
|
||||
cpu_cores INTEGER,
|
||||
memory_gb INTEGER,
|
||||
limit_gpus INTEGER,
|
||||
limit_cpu_cores INTEGER,
|
||||
limit_memory_gb INTEGER,
|
||||
updated_at BIGINT NOT NULL,
|
||||
updated_by TEXT NOT NULL,
|
||||
PRIMARY KEY (scope_type, scope_id)
|
||||
);
|
||||
@@ -0,0 +1,127 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Resources are amounts of the three things a replica holds of a cluster.
|
||||
// Nil is not governed: nothing guaranteed, or nothing limited.
|
||||
type Resources struct {
|
||||
GPUs *int `json:"gpus,omitempty"`
|
||||
CPUCores *int `json:"cpu_cores,omitempty"`
|
||||
MemoryGB *int `json:"memory_gb,omitempty"`
|
||||
}
|
||||
|
||||
// Any reports whether any resource is governed.
|
||||
func (r Resources) Any() bool { return r.GPUs != nil || r.CPUCores != nil || r.MemoryGB != nil }
|
||||
|
||||
// CapacityQuota is what one part of the tenancy tree is guaranteed of the
|
||||
// cluster, and the most it may hold.
|
||||
type CapacityQuota struct {
|
||||
Scope Scope `json:"scope"`
|
||||
ScopeID string `json:"scope_id"`
|
||||
// Guarantee is held for the scope: while it holds less, what others
|
||||
// borrowed beyond their own guarantees is taken back for it.
|
||||
Guarantee Resources `json:"guarantee"`
|
||||
// Limit is the most it may hold, borrowing included.
|
||||
Limit Resources `json:"limit"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
UpdatedBy string `json:"updated_by"`
|
||||
}
|
||||
|
||||
// PutCapacityQuota sets a scope's quota, replacing what was there.
|
||||
func (s *DB) PutCapacityQuota(ctx context.Context, q CapacityQuota) error {
|
||||
if !validMembershipScope(q.Scope) || q.ScopeID == "" {
|
||||
return fmt.Errorf("a capacity quota belongs to an organization, team or project")
|
||||
}
|
||||
for _, v := range []*int{q.Guarantee.GPUs, q.Guarantee.CPUCores, q.Guarantee.MemoryGB,
|
||||
q.Limit.GPUs, q.Limit.CPUCores, q.Limit.MemoryGB} {
|
||||
if v != nil && *v < 0 {
|
||||
return fmt.Errorf("a quota cannot be negative")
|
||||
}
|
||||
}
|
||||
if q.UpdatedAt.IsZero() {
|
||||
q.UpdatedAt = time.Now()
|
||||
}
|
||||
_, err := s.db.ExecContext(ctx,
|
||||
`INSERT INTO capacity_quotas (scope_type, scope_id, gpus, cpu_cores, memory_gb,
|
||||
limit_gpus, limit_cpu_cores, limit_memory_gb, updated_at, updated_by)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT (scope_type, scope_id) DO UPDATE SET
|
||||
gpus = excluded.gpus, cpu_cores = excluded.cpu_cores, memory_gb = excluded.memory_gb,
|
||||
limit_gpus = excluded.limit_gpus, limit_cpu_cores = excluded.limit_cpu_cores,
|
||||
limit_memory_gb = excluded.limit_memory_gb,
|
||||
updated_at = excluded.updated_at, updated_by = excluded.updated_by`,
|
||||
string(q.Scope), q.ScopeID,
|
||||
nullInt(q.Guarantee.GPUs), nullInt(q.Guarantee.CPUCores), nullInt(q.Guarantee.MemoryGB),
|
||||
nullInt(q.Limit.GPUs), nullInt(q.Limit.CPUCores), nullInt(q.Limit.MemoryGB),
|
||||
q.UpdatedAt.UnixMilli(), q.UpdatedBy)
|
||||
return err
|
||||
}
|
||||
|
||||
// CapacityQuotas lists every quota.
|
||||
func (s *DB) CapacityQuotas(ctx context.Context) ([]CapacityQuota, error) {
|
||||
return queryAll(ctx, s.db,
|
||||
`SELECT scope_type, scope_id, gpus, cpu_cores, memory_gb,
|
||||
limit_gpus, limit_cpu_cores, limit_memory_gb, updated_at, updated_by
|
||||
FROM capacity_quotas ORDER BY scope_type, scope_id`, nil,
|
||||
func(rows *sql.Rows) (CapacityQuota, error) {
|
||||
var q CapacityQuota
|
||||
var scope string
|
||||
var g [6]sql.NullInt64
|
||||
var at int64
|
||||
err := rows.Scan(&scope, &q.ScopeID, &g[0], &g[1], &g[2], &g[3], &g[4], &g[5], &at, &q.UpdatedBy)
|
||||
q.Scope = Scope(scope)
|
||||
q.Guarantee = Resources{GPUs: intOf(g[0]), CPUCores: intOf(g[1]), MemoryGB: intOf(g[2])}
|
||||
q.Limit = Resources{GPUs: intOf(g[3]), CPUCores: intOf(g[4]), MemoryGB: intOf(g[5])}
|
||||
q.UpdatedAt = time.UnixMilli(at)
|
||||
return q, err
|
||||
})
|
||||
}
|
||||
|
||||
// DeleteCapacityQuota removes a scope's quota.
|
||||
func (s *DB) DeleteCapacityQuota(ctx context.Context, scope Scope, id string) error {
|
||||
_, err := s.db.ExecContext(ctx,
|
||||
`DELETE FROM capacity_quotas WHERE scope_type = ? AND scope_id = ?`, string(scope), id)
|
||||
return err
|
||||
}
|
||||
|
||||
// ScopeParents maps every team and project to the scopes above it, nearest
|
||||
// first: a project to its team and organization, a team to its organization.
|
||||
// Keys and values are written scope:id.
|
||||
func (s *DB) ScopeParents(ctx context.Context) (map[string][]string, error) {
|
||||
out := map[string][]string{}
|
||||
teams, err := s.AllTeams(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
orgOf := map[string]string{}
|
||||
for _, t := range teams {
|
||||
orgOf[t.ID] = t.OrgID
|
||||
out["team:"+t.ID] = []string{"org:" + t.OrgID}
|
||||
}
|
||||
type pt struct{ project, team string }
|
||||
rows, err := queryAll(ctx, s.db, `SELECT id, team_id FROM projects`, nil,
|
||||
func(r *sql.Rows) (pt, error) {
|
||||
var p pt
|
||||
return p, r.Scan(&p.project, &p.team)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, p := range rows {
|
||||
out["project:"+p.project] = []string{"team:" + p.team, "org:" + orgOf[p.team]}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func intOf(n sql.NullInt64) *int {
|
||||
if !n.Valid {
|
||||
return nil
|
||||
}
|
||||
v := int(n.Int64)
|
||||
return &v
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package store
|
||||
|
||||
import "testing"
|
||||
|
||||
func ip(v int) *int { return &v }
|
||||
|
||||
func TestAQuotaKeepsWhatItGovernsAndNothingElse(t *testing.T) {
|
||||
s := tenancyStore(t)
|
||||
org, _ := s.CreateOrg(t.Context(), "Acme")
|
||||
q := CapacityQuota{Scope: ScopeOrg, ScopeID: org, UpdatedBy: "ops",
|
||||
Guarantee: Resources{GPUs: ip(8)}, Limit: Resources{GPUs: ip(16), CPUCores: ip(0)}}
|
||||
if err := s.PutCapacityQuota(t.Context(), q); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
all, err := s.CapacityQuotas(t.Context())
|
||||
if err != nil || len(all) != 1 {
|
||||
t.Fatalf("%v %+v", err, all)
|
||||
}
|
||||
got := all[0]
|
||||
if *got.Guarantee.GPUs != 8 || got.Guarantee.CPUCores != nil || got.Guarantee.MemoryGB != nil {
|
||||
t.Errorf("guarantee %+v: only GPUs were set", got.Guarantee)
|
||||
}
|
||||
// Zero is a limit of nothing, not an absent one.
|
||||
if *got.Limit.GPUs != 16 || got.Limit.CPUCores == nil || *got.Limit.CPUCores != 0 {
|
||||
t.Errorf("limit %+v", got.Limit)
|
||||
}
|
||||
q.Guarantee = Resources{MemoryGB: ip(64)}
|
||||
if err := s.PutCapacityQuota(t.Context(), q); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
all, _ = s.CapacityQuotas(t.Context())
|
||||
if len(all) != 1 || all[0].Guarantee.GPUs != nil || *all[0].Guarantee.MemoryGB != 64 {
|
||||
t.Errorf("a second put should replace the first: %+v", all)
|
||||
}
|
||||
if err := s.DeleteCapacityQuota(t.Context(), ScopeOrg, org); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if all, _ = s.CapacityQuotas(t.Context()); len(all) != 0 {
|
||||
t.Error("the quota survived its deletion")
|
||||
}
|
||||
if err := s.PutCapacityQuota(t.Context(), CapacityQuota{Scope: ScopeOrg, ScopeID: org,
|
||||
Guarantee: Resources{GPUs: ip(-1)}}); err == nil {
|
||||
t.Error("a negative quota was stored")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEveryScopeKnowsWhatIsAboveIt(t *testing.T) {
|
||||
s := tenancyStore(t)
|
||||
org, _ := s.CreateOrg(t.Context(), "Acme")
|
||||
team, _ := s.CreateTeam(t.Context(), org, "ml")
|
||||
proj, _ := s.CreateProject(t.Context(), team, "training")
|
||||
p, err := s.ScopeParents(t.Context())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []string{"team:" + team, "org:" + org}
|
||||
if got := p["project:"+proj]; len(got) != 2 || got[0] != want[0] || got[1] != want[1] {
|
||||
t.Errorf("project's parents %v, expected %v", got, want)
|
||||
}
|
||||
if got := p["team:"+team]; len(got) != 1 || got[0] != "org:"+org {
|
||||
t.Errorf("team's parents %v", got)
|
||||
}
|
||||
}
|
||||
@@ -61,6 +61,8 @@ type AccountPages struct {
|
||||
hasMirror bool
|
||||
// hasQuotas is set when /admin/quotas is registered.
|
||||
hasQuotas bool
|
||||
// capacity sets capacity quotas, from the quotas page; nil leaves them off.
|
||||
capacity CapacityAdmin
|
||||
// personalOff refuses personal keys; see SetPersonalOff.
|
||||
personalOff bool
|
||||
// personalDefaults is what personal use falls under by default.
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// CapacityAdmin sets what parts of the tenancy tree are guaranteed of the
|
||||
// cluster and may hold. The admin API implements it; the page and the CLI go
|
||||
// through the same checks.
|
||||
type CapacityAdmin interface {
|
||||
QuotaViews(ctx context.Context) ([]adminapi.QuotaView, error)
|
||||
SetQuota(ctx context.Context, req adminapi.QuotaRequest, by string) (string, error)
|
||||
ClearQuota(ctx context.Context, scope string) (string, error)
|
||||
Scopes(ctx context.Context) ([]adminapi.ScopeName, error)
|
||||
}
|
||||
|
||||
// SetCapacity gives the quotas page capacity quotas to show and set. Without
|
||||
// it the section is left off.
|
||||
func (p *AccountPages) SetCapacity(c CapacityAdmin) { p.capacity = c }
|
||||
|
||||
// capacityRow is one scope's capacity quota as the page shows it.
|
||||
type capacityRow struct {
|
||||
adminapi.QuotaView
|
||||
GPUs, Cores, Memory string
|
||||
}
|
||||
|
||||
// capacityForm is what the form starts with: a scope's quota to change, or
|
||||
// nothing for a new one.
|
||||
type capacityForm struct {
|
||||
Scope string
|
||||
GPUs, Cores, Memory string
|
||||
LGPUs, LCores, LMem string
|
||||
}
|
||||
|
||||
func (p *AccountPages) capacitySection(ctx context.Context, d *adminQuotasData, editing string) {
|
||||
if p.capacity == nil {
|
||||
return
|
||||
}
|
||||
d.HasCapacity = true
|
||||
views, err := p.capacity.QuotaViews(ctx)
|
||||
if err != nil {
|
||||
d.Error = err.Error()
|
||||
return
|
||||
}
|
||||
for _, v := range views {
|
||||
d.Capacity = append(d.Capacity, capacityRow{QuotaView: v,
|
||||
GPUs: capCell(v.Holds.GPUs, v.Guarantee.GPUs, v.Limit.GPUs),
|
||||
Cores: capCell(v.Holds.CPUCores, v.Guarantee.CPUCores, v.Limit.CPUCores),
|
||||
Memory: capCell(v.Holds.MemoryGB, v.Guarantee.MemoryGB, v.Limit.MemoryGB)})
|
||||
if v.Scope == editing {
|
||||
d.CapForm = capacityForm{Scope: v.Scope,
|
||||
GPUs: intStr(v.Guarantee.GPUs), Cores: intStr(v.Guarantee.CPUCores), Memory: intStr(v.Guarantee.MemoryGB),
|
||||
LGPUs: intStr(v.Limit.GPUs), LCores: intStr(v.Limit.CPUCores), LMem: intStr(v.Limit.MemoryGB)}
|
||||
}
|
||||
}
|
||||
if d.Scopes, err = p.capacity.Scopes(ctx); err != nil {
|
||||
d.Error = err.Error()
|
||||
}
|
||||
}
|
||||
|
||||
// capCell is "6 of 8 (at most 16)": held, guaranteed, limit.
|
||||
func capCell(holds int, g, l *int) string {
|
||||
if g == nil && l == nil {
|
||||
return ""
|
||||
}
|
||||
out := strconv.Itoa(holds)
|
||||
if g != nil {
|
||||
out += " of " + strconv.Itoa(*g)
|
||||
}
|
||||
if l != nil {
|
||||
out += fmt.Sprintf(" (at most %d)", *l)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func intStr(v *int) string {
|
||||
if v == nil {
|
||||
return ""
|
||||
}
|
||||
return strconv.Itoa(*v)
|
||||
}
|
||||
|
||||
// applyCapacity sets or clears one scope's capacity quota from the form.
|
||||
func (p *AccountPages) applyCapacity(r *http.Request, admin store.User) (string, error) {
|
||||
scope := strings.TrimSpace(r.FormValue("scope"))
|
||||
if scope == "" {
|
||||
return "", fmt.Errorf("choose where the quota applies")
|
||||
}
|
||||
if r.FormValue("clear") != "" {
|
||||
key, err := p.capacity.ClearQuota(r.Context(), scope)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return "capacity quota removed from " + key, nil
|
||||
}
|
||||
field := func(name, what string) (*int, error) {
|
||||
v := strings.TrimSpace(r.FormValue(name))
|
||||
if v == "" {
|
||||
return nil, nil
|
||||
}
|
||||
n, err := strconv.Atoi(v)
|
||||
if err != nil || n < 0 {
|
||||
return nil, fmt.Errorf("%s must be a whole number, or blank for none", what)
|
||||
}
|
||||
return &n, nil
|
||||
}
|
||||
var req adminapi.QuotaRequest
|
||||
req.Scope = scope
|
||||
var err error
|
||||
for _, f := range []struct {
|
||||
dst **int
|
||||
name string
|
||||
what string
|
||||
}{
|
||||
{&req.Guarantee.GPUs, "gpus", "guaranteed GPUs"},
|
||||
{&req.Guarantee.CPUCores, "cores", "guaranteed cores"},
|
||||
{&req.Guarantee.MemoryGB, "memory", "guaranteed memory"},
|
||||
{&req.Limit.GPUs, "limit_gpus", "the GPU limit"},
|
||||
{&req.Limit.CPUCores, "limit_cores", "the core limit"},
|
||||
{&req.Limit.MemoryGB, "limit_memory", "the memory limit"},
|
||||
} {
|
||||
if *f.dst, err = field(f.name, f.what); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
key, err := p.capacity.SetQuota(r.Context(), req, admin.Email)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return "capacity quota set on " + key + "; it takes effect within half a minute", nil
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/quota"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
type fakeCapacity struct {
|
||||
views []adminapi.QuotaView
|
||||
set []adminapi.QuotaRequest
|
||||
cleared []string
|
||||
}
|
||||
|
||||
func (f *fakeCapacity) QuotaViews(context.Context) ([]adminapi.QuotaView, error) { return f.views, nil }
|
||||
func (f *fakeCapacity) SetQuota(_ context.Context, r adminapi.QuotaRequest, _ string) (string, error) {
|
||||
if r.Scope == "team:nope" {
|
||||
return "", errors.New(`there is no team "nope"`)
|
||||
}
|
||||
f.set = append(f.set, r)
|
||||
return r.Scope, nil
|
||||
}
|
||||
func (f *fakeCapacity) ClearQuota(_ context.Context, s string) (string, error) {
|
||||
f.cleared = append(f.cleared, s)
|
||||
return s, nil
|
||||
}
|
||||
func (f *fakeCapacity) Scopes(context.Context) ([]adminapi.ScopeName, error) {
|
||||
return []adminapi.ScopeName{{Key: "org:o1", Name: "Acme"}, {Key: "team:t1", Name: "Acme / ml"}}, nil
|
||||
}
|
||||
|
||||
func TestCapacityQuotasAreShownAndSetFromTheQuotasPage(t *testing.T) {
|
||||
eight, sixteen := 8, 16
|
||||
fc := &fakeCapacity{views: []adminapi.QuotaView{{Scope: "team:t1", Name: "Acme / ml",
|
||||
Guarantee: store.Resources{GPUs: &eight}, Limit: store.Resources{GPUs: &sixteen},
|
||||
Holds: quota.Amount{GPUs: 6}, UpdatedBy: "ops@acme"}}}
|
||||
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
|
||||
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.DiscardHandler))
|
||||
p.SetCapacity(fc)
|
||||
mux := http.NewServeMux()
|
||||
p.RegisterAdminQuotas(mux, &fakeQuotas{}, pricedYes{})
|
||||
|
||||
body := mirrorGet(t, mux, "/admin/quotas").Body.String()
|
||||
for _, want := range []string{`id="capacity"`, "Acme / ml", "6 of 8 (at most 16)", "ops@acme",
|
||||
`<option value="team:t1">Acme / ml</option>`} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the page lacks %q", want)
|
||||
}
|
||||
}
|
||||
// "change" fills the form with the scope's figures.
|
||||
edit := mirrorGet(t, mux, "/admin/quotas?cap=team:t1").Body.String()
|
||||
if !strings.Contains(edit, `<option value="team:t1" selected>`) || !strings.Contains(edit, `name="limit_gpus" value="16"`) {
|
||||
t.Error("change did not open the form on the scope's quota")
|
||||
}
|
||||
|
||||
res := mirrorPost(t, mux, "/admin/quotas/capacity", url.Values{"csrf": {"tok"}, "scope": {"team:t1"},
|
||||
"gpus": {"8"}, "cores": {""}, "limit_memory": {"64"}}).Body.String()
|
||||
if len(fc.set) != 1 || *fc.set[0].Guarantee.GPUs != 8 || fc.set[0].Guarantee.CPUCores != nil || *fc.set[0].Limit.MemoryGB != 64 {
|
||||
t.Errorf("sent %+v", fc.set)
|
||||
}
|
||||
if !strings.Contains(res, "capacity quota set on team:t1") {
|
||||
t.Errorf("no word of it: %s", res)
|
||||
}
|
||||
for _, bad := range []url.Values{{"scope": {"team:t1"}, "gpus": {"eight"}}, {"scope": {"team:nope"}, "gpus": {"1"}}, {"gpus": {"1"}}} {
|
||||
bad.Set("csrf", "tok")
|
||||
n := len(fc.set)
|
||||
out := mirrorPost(t, mux, "/admin/quotas/capacity", bad).Body.String()
|
||||
if len(fc.set) != n || !strings.Contains(out, `class="error"`) {
|
||||
t.Errorf("%v was not refused with a reason", bad)
|
||||
}
|
||||
}
|
||||
mirrorPost(t, mux, "/admin/quotas/capacity", url.Values{"csrf": {"tok"}, "scope": {"team:t1"}, "clear": {"1"}})
|
||||
if len(fc.cleared) != 1 || fc.cleared[0] != "team:t1" {
|
||||
t.Errorf("cleared %v", fc.cleared)
|
||||
}
|
||||
// Without a CSRF token nothing happens.
|
||||
mirrorPost(t, mux, "/admin/quotas/capacity", url.Values{"scope": {"team:t1"}, "clear": {"1"}})
|
||||
if len(fc.cleared) != 1 {
|
||||
t.Error("a change was made without the CSRF token")
|
||||
}
|
||||
}
|
||||
@@ -7,11 +7,13 @@ import (
|
||||
"fmt"
|
||||
"html/template"
|
||||
"net/http"
|
||||
"slices"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
@@ -67,6 +69,23 @@ type modelUse struct {
|
||||
Name string
|
||||
Mode string // always, ondemand or stopped
|
||||
Replicas int
|
||||
// Min and Max are its scaling as the catalog holds it.
|
||||
Min, Max int
|
||||
// Class is what it set, empty for its default; EffClass what that comes
|
||||
// to. Owner is org:ID or empty for shared. MemoryGB and CPUCores are what
|
||||
// a replica of its first profile reserves.
|
||||
Class, EffClass, Owner string
|
||||
MemoryGB, CPUCores int
|
||||
// Ready, Wanted and Nodes are how it runs now, for the row itself:
|
||||
// replicas ready of those it should have, and the nodes they are on.
|
||||
// Fleet is false where nothing could be asked, so the row says nothing
|
||||
// rather than "0 of 0".
|
||||
Fleet bool
|
||||
Ready, Wanted int
|
||||
Starting int
|
||||
Nodes []nodeName
|
||||
Parked bool
|
||||
Unplaceable bool
|
||||
// Declared models are read from server.yaml on every start, so a mode
|
||||
// changed here would revert; the page says where to change it instead.
|
||||
Declared bool
|
||||
@@ -83,6 +102,11 @@ type mirrorRow struct {
|
||||
// operator clicked. Nil on a row that is still arriving: a catalog entry
|
||||
// pointing at a partial file is the thing versioned paths exist to prevent.
|
||||
Serve *serveForm
|
||||
// Detail is the card opened beneath the row: what runs it, on which
|
||||
// nodes, and how many replicas it runs.
|
||||
Detail *heldDetail
|
||||
// Open is the row's ?open= value.
|
||||
Open string
|
||||
|
||||
// HasPercent is false when the upstream never sent a length. The bar is
|
||||
// then not drawn at all: a progress bar that has to invent its own total is
|
||||
@@ -100,7 +124,11 @@ type mirrorRow struct {
|
||||
}
|
||||
|
||||
type adminMirrorData struct {
|
||||
Nav Nav
|
||||
Nav Nav
|
||||
// Tab is the part of the page shown: held, explore or fetch. Tabs are the
|
||||
// links to each.
|
||||
Tab string
|
||||
Tabs []mirrorTab
|
||||
Addr string
|
||||
Base string
|
||||
Rows []mirrorRow
|
||||
@@ -270,6 +298,8 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
|
||||
return name + " now runs all the time", nil
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /admin/mirror/replicas", p.mirrorAction(m, p.setModel))
|
||||
|
||||
mux.HandleFunc("POST /admin/mirror/serve", p.mirrorAction(m,
|
||||
func(ctx context.Context, admin store.User, r *http.Request) (string, error) {
|
||||
model, err := buildModel(r.FormValue)
|
||||
@@ -361,7 +391,14 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
|
||||
}
|
||||
|
||||
d.HasCatalog = p.catalog != nil
|
||||
p.explore(r, &d)
|
||||
d.HasRegistry = p.registry != nil
|
||||
d.Tab = mirrorTabFor(r, errMsg)
|
||||
// Only the tab shown does its work: the registry is a server across the
|
||||
// internet, and asking it for sixty repositories to show somebody who came
|
||||
// to look at a download's progress made every visit wait on it.
|
||||
if d.Tab == tabExplore {
|
||||
p.explore(r, &d)
|
||||
}
|
||||
|
||||
got, err := m.List(r.Context())
|
||||
if err != nil {
|
||||
@@ -370,6 +407,7 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
|
||||
if d.Error == "" {
|
||||
d.Error = err.Error()
|
||||
}
|
||||
d.Tabs = mirrorTabs(d)
|
||||
render(w, "admin_mirror", d)
|
||||
return
|
||||
}
|
||||
@@ -412,12 +450,50 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
|
||||
}
|
||||
d.Rows = append(d.Rows, row)
|
||||
}
|
||||
var views map[string]adminapi.ModelView
|
||||
prefix := ""
|
||||
if p.fleet != nil && len(sources) > 0 {
|
||||
views = map[string]adminapi.ModelView{}
|
||||
for _, v := range p.fleet.Models() {
|
||||
views[v.Name] = v
|
||||
}
|
||||
var ids []string
|
||||
for _, n := range p.fleet.Nodes() {
|
||||
ids = append(ids, n.ID)
|
||||
}
|
||||
prefix = sharedPrefix(ids)
|
||||
}
|
||||
for i := range d.Rows {
|
||||
for _, m := range sources[d.Rows[i].URL] {
|
||||
d.Rows[i].UsedBy = append(d.Rows[i].UsedBy, modelUse{
|
||||
u := modelUse{
|
||||
Name: m.Name, Mode: modeOf(m.Scaling), Replicas: max(m.Scaling.Max, 1),
|
||||
Min: m.Scaling.Min, Max: m.Scaling.Max,
|
||||
Class: m.Class, EffClass: m.EffectiveClass(), Owner: m.Owner,
|
||||
Declared: p.catalog.Declared(m.Name),
|
||||
})
|
||||
}
|
||||
if len(m.Profiles) > 0 {
|
||||
u.MemoryGB, u.CPUCores = m.Profiles[0].MinMemoryGB, m.Profiles[0].CPUCores
|
||||
}
|
||||
if v, ok := views[m.Name]; ok {
|
||||
u.Fleet, u.Ready, u.Wanted = true, v.Ready, v.Desired
|
||||
u.Parked, u.Unplaceable = v.Parked, v.Unplaceable
|
||||
var nodes []string
|
||||
nodes, u.Starting = runningOn(p.fleet.Replicas(m.Name))
|
||||
for _, n := range nodes {
|
||||
u.Nodes = append(u.Nodes, nodeName{Full: n, Short: strings.TrimPrefix(n, prefix)})
|
||||
}
|
||||
}
|
||||
d.Rows[i].UsedBy = append(d.Rows[i].UsedBy, u)
|
||||
}
|
||||
}
|
||||
|
||||
// The one row whose card is open. From the form too, so a change made in
|
||||
// the card comes back with the card still open to show it.
|
||||
open := r.FormValue("open")
|
||||
for i := range d.Rows {
|
||||
d.Rows[i].Open = openKey(d.Rows[i].Model, d.Rows[i].Version)
|
||||
if open != "" && d.Rows[i].Open == open && d.Rows[i].State == mirror.Present {
|
||||
d.Rows[i].Detail = p.heldDetailFor(r.Context(), d.Rows[i])
|
||||
}
|
||||
}
|
||||
|
||||
@@ -462,12 +538,78 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
|
||||
|
||||
d.Groups = groupByModel(d.Rows)
|
||||
|
||||
if d.Fetching > 0 {
|
||||
// Only on the list: a reload anywhere else takes with it a search, an
|
||||
// open repository or a URL half typed.
|
||||
if d.Fetching > 0 && d.Tab == tabHeld && open == "" {
|
||||
d.RefreshSec = 5
|
||||
}
|
||||
d.Tabs = mirrorTabs(d)
|
||||
render(w, "admin_mirror", d)
|
||||
}
|
||||
|
||||
const (
|
||||
tabHeld = "held"
|
||||
tabExplore = "explore"
|
||||
tabFetch = "fetch"
|
||||
)
|
||||
|
||||
// mirrorTab is one of the page's tabs.
|
||||
type mirrorTab struct {
|
||||
Label string
|
||||
Count string // beside the label, empty for none
|
||||
Href string
|
||||
Active bool
|
||||
}
|
||||
|
||||
// mirrorTabFor is the tab a request is for.
|
||||
//
|
||||
// Named in the address when it is chosen, and otherwise read from what the
|
||||
// address carries, so links made before the page had tabs — a search, an
|
||||
// open repository — still land where they meant. After a change the list is
|
||||
// shown, where its effect is, unless it was refused: then the form it came
|
||||
// from, to try again.
|
||||
func mirrorTabFor(r *http.Request, errMsg string) string {
|
||||
valid := func(t string) bool { return t == tabHeld || t == tabExplore || t == tabFetch }
|
||||
if r.Method == http.MethodPost {
|
||||
if t := r.PostFormValue("tab"); errMsg != "" && valid(t) {
|
||||
return t
|
||||
}
|
||||
return tabHeld
|
||||
}
|
||||
q := r.URL.Query()
|
||||
if t := q.Get("tab"); valid(t) {
|
||||
return t
|
||||
}
|
||||
for _, k := range []string{"q", "repo", "view", "task", "size", "x"} {
|
||||
if q.Has(k) {
|
||||
return tabExplore
|
||||
}
|
||||
}
|
||||
return tabHeld
|
||||
}
|
||||
|
||||
func mirrorTabs(d adminMirrorData) []mirrorTab {
|
||||
held := mirrorTab{Label: "Held", Href: "/admin/mirror?tab=held", Active: d.Tab == tabHeld}
|
||||
switch {
|
||||
case d.Unreachable:
|
||||
held.Count = "unreachable"
|
||||
case d.Fetching > 0:
|
||||
held.Count = fmt.Sprintf("%d · %d downloading", d.Present, d.Fetching)
|
||||
default:
|
||||
held.Count = strconv.Itoa(d.Present)
|
||||
}
|
||||
if d.Failed > 0 {
|
||||
held.Count += fmt.Sprintf(" · %d failed", d.Failed)
|
||||
}
|
||||
tabs := []mirrorTab{held}
|
||||
if d.HasRegistry {
|
||||
tabs = append(tabs, mirrorTab{Label: "Explore Hugging Face",
|
||||
Href: "/admin/mirror?tab=explore", Active: d.Tab == tabExplore})
|
||||
}
|
||||
return append(tabs, mirrorTab{Label: "Fetch by URL",
|
||||
Href: "/admin/mirror?tab=fetch", Active: d.Tab == tabFetch})
|
||||
}
|
||||
|
||||
// mirrorGroup is a family of models the mirror holds — every qwen, every
|
||||
// llama — and their versions.
|
||||
type mirrorGroup struct {
|
||||
@@ -663,3 +805,43 @@ func writeJSONStatus(w http.ResponseWriter, code int, v any) {
|
||||
w.WriteHeader(code)
|
||||
_ = json.NewEncoder(w).Encode(v)
|
||||
}
|
||||
|
||||
// runningOn is the nodes a model's replicas are on, each once and in order,
|
||||
// and how many of its replicas are not ready yet.
|
||||
func runningOn(reps []adminapi.ReplicaView) (nodes []string, starting int) {
|
||||
for _, r := range reps {
|
||||
if r.State != "ready" {
|
||||
starting++
|
||||
}
|
||||
for _, sh := range r.Shards {
|
||||
if !slices.Contains(nodes, sh.Node) {
|
||||
nodes = append(nodes, sh.Node)
|
||||
}
|
||||
}
|
||||
}
|
||||
slices.Sort(nodes)
|
||||
return nodes, starting
|
||||
}
|
||||
|
||||
// nodeName is a node as a row names it: short, with the full name beside.
|
||||
type nodeName struct{ Short, Full string }
|
||||
|
||||
// sharedPrefix is what every node's name starts with, up to and including a
|
||||
// separator — "llmgw-" of llmgw-node01 and llmgw-node02 — so a row can say
|
||||
// node01 and fit on a line. Nothing when the names share no such part, or
|
||||
// when there is one node and dropping it would leave only a number.
|
||||
func sharedPrefix(ids []string) string {
|
||||
if len(ids) < 2 {
|
||||
return ""
|
||||
}
|
||||
p := ids[0]
|
||||
for _, id := range ids[1:] {
|
||||
for !strings.HasPrefix(id, p) {
|
||||
p = p[:len(p)-1]
|
||||
}
|
||||
}
|
||||
if i := strings.LastIndexAny(p, "-._"); i >= 0 {
|
||||
return p[:i+1]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -341,16 +341,18 @@ func TestAFailedRowOffersRetryRatherThanRemove(t *testing.T) {
|
||||
}
|
||||
|
||||
type fakeRegistry struct {
|
||||
results []hf.Model
|
||||
repo hf.Repo
|
||||
err error
|
||||
asked string
|
||||
query hf.Query
|
||||
results []hf.Model
|
||||
repo hf.Repo
|
||||
err error
|
||||
asked string
|
||||
query hf.Query
|
||||
explored int
|
||||
}
|
||||
|
||||
func (f *fakeRegistry) Explore(_ context.Context, q hf.Query) ([]hf.Model, error) {
|
||||
f.asked = q.Search
|
||||
f.query = q
|
||||
f.explored++
|
||||
return f.results, f.err
|
||||
}
|
||||
|
||||
@@ -441,7 +443,11 @@ func TestWithNoRegistryThereIsNoSearchBox(t *testing.T) {
|
||||
if strings.Contains(body, "Search Hugging Face") {
|
||||
t.Error("the explorer should be absent without a registry")
|
||||
}
|
||||
if !strings.Contains(body, "Fetch a model") {
|
||||
if strings.Contains(body, "Explore Hugging Face") {
|
||||
t.Error("a tab for an explorer that is not there")
|
||||
}
|
||||
fetch := mirrorGet(t, mirrorPage(t, &fakeMirror{}, true), "/admin/mirror?tab=fetch").Body.String()
|
||||
if !strings.Contains(fetch, `name="source" required`) {
|
||||
t.Error("fetching by hand must still be there")
|
||||
}
|
||||
}
|
||||
@@ -464,8 +470,7 @@ func TestAShardedModelIsOneRowAndOneButton(t *testing.T) {
|
||||
"/admin/mirror?repo=Qwen/Qwen2.5-7B-Instruct").Body.String()
|
||||
|
||||
// Fifteen shards is one decision, not fifteen. Four here, and one form.
|
||||
if n := strings.Count(body, `action="/admin/mirror/fetch"`); n != 2 {
|
||||
// One for the set, one for the manual form at the bottom.
|
||||
if n := strings.Count(body, `action="/admin/mirror/fetch"`); n != 1 {
|
||||
t.Errorf("expected the four shards as a single form, found %d forms", n)
|
||||
}
|
||||
if !strings.Contains(body, "4 shards, fetched together") {
|
||||
@@ -534,8 +539,8 @@ func TestAlternativeQuantisationsStaySeparateRows(t *testing.T) {
|
||||
|
||||
// Two GGUFs are alternatives, not halves. Offering "fetch all" here would
|
||||
// spend gigabytes on a copy nobody asked for.
|
||||
if n := strings.Count(body, `action="/admin/mirror/fetch"`); n != 3 {
|
||||
t.Errorf("expected a form each plus the manual one, found %d", n)
|
||||
if n := strings.Count(body, `action="/admin/mirror/fetch"`); n != 2 {
|
||||
t.Errorf("expected a form each, found %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -587,7 +592,7 @@ func TestHowAModelRunsIsChangedWhereItsFileIsShown(t *testing.T) {
|
||||
t.Error("the running model's mode is not shown as chosen")
|
||||
}
|
||||
// Declared in server.yaml: shown, not offered, since it would revert.
|
||||
if !strings.Contains(body, "on demand<br>set in server.yaml") {
|
||||
if !strings.Contains(body, `title="set in server.yaml, which is read again on every start">on demand · yaml</span>`) {
|
||||
t.Error("a declared model should say where its mode is set")
|
||||
}
|
||||
if strings.Count(body, `name="mode" class="mode"`) != 1 {
|
||||
@@ -655,3 +660,53 @@ func privateOK(m *mirror.Manager) *mirror.Manager {
|
||||
m.AllowPrivateSources()
|
||||
return m
|
||||
}
|
||||
|
||||
func TestEachTabDoesOnlyItsOwnWork(t *testing.T) {
|
||||
reg := &fakeRegistry{}
|
||||
mux := mirrorPageWith(t, &fakeMirror{models: []mirror.Status{
|
||||
{Model: "m", Version: "v", File: "m.gguf", State: mirror.Present, BytesDone: 1},
|
||||
{Model: "n", Version: "v", File: "n.gguf", State: mirror.Fetching, BytesDone: 1},
|
||||
}}, reg, true)
|
||||
|
||||
held := mirrorGet(t, mux, "/admin/mirror").Body.String()
|
||||
if reg.explored != 0 {
|
||||
t.Error("the list asked the registry for repositories it does not show")
|
||||
}
|
||||
if !strings.Contains(held, `class="on" aria-current="page">Held <span class="count">1 · 1 downloading</span>`) {
|
||||
t.Errorf("the Held tab should be chosen and count what it holds: %s", held)
|
||||
}
|
||||
if strings.Contains(held, `name="source" required`) || strings.Contains(held, "Search Hugging Face") {
|
||||
t.Error("other tabs' forms are on the list")
|
||||
}
|
||||
if !strings.Contains(held, `http-equiv="refresh"`) {
|
||||
t.Error("the list stopped following a download")
|
||||
}
|
||||
|
||||
explore := mirrorGet(t, mux, "/admin/mirror?tab=explore").Body.String()
|
||||
if reg.explored != 1 || !strings.Contains(explore, "Search Hugging Face") {
|
||||
t.Error("the explorer tab did not explore")
|
||||
}
|
||||
if strings.Contains(explore, `id="held"`) || strings.Contains(explore, `http-equiv="refresh"`) {
|
||||
t.Error("the explorer shows the list, or reloads away a search")
|
||||
}
|
||||
// A link from before there were tabs still lands on the explorer.
|
||||
if old := mirrorGet(t, mux, "/admin/mirror?q=qwen").Body.String(); !strings.Contains(old, "Search Hugging Face") {
|
||||
t.Error("a search address no longer opens the explorer")
|
||||
}
|
||||
}
|
||||
|
||||
func TestARefusedFetchReturnsToItsFormAndAStartedOneToTheList(t *testing.T) {
|
||||
m := &fakeMirror{fetchErr: errors.New("upstream said 404")}
|
||||
mux := mirrorPage(t, m, true)
|
||||
body := mirrorPost(t, mux, "/admin/mirror/fetch", url.Values{"csrf": {"tok"}, "source": {"https://h.example/m.gguf"}, "model": {"m"},
|
||||
"version": {"v"}, "force": {"1"}, "tab": {"fetch"}}).Body.String()
|
||||
if !strings.Contains(body, "upstream said 404") || !strings.Contains(body, `name="source" required`) {
|
||||
t.Errorf("a refusal should come back on the form it came from: %s", body)
|
||||
}
|
||||
m.fetchErr = nil
|
||||
body = mirrorPost(t, mux, "/admin/mirror/fetch", url.Values{"csrf": {"tok"}, "source": {"https://h.example/m.gguf"}, "model": {"m"},
|
||||
"version": {"v"}, "force": {"1"}, "tab": {"fetch"}}).Body.String()
|
||||
if !strings.Contains(body, `id="held"`) {
|
||||
t.Errorf("a started download should be shown where it is followed: %s", body)
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
@@ -129,12 +130,17 @@ type adminQuotasData struct {
|
||||
// Editing is the "scope:id" whose form is open, so the template can keep
|
||||
// the link on every other row.
|
||||
Editing string
|
||||
Since string
|
||||
Message string
|
||||
Error string
|
||||
CSRF string
|
||||
Version string
|
||||
Now string
|
||||
// The capacity section: quotas on GPUs, cores and memory.
|
||||
HasCapacity bool
|
||||
Capacity []capacityRow
|
||||
Scopes []adminapi.ScopeName
|
||||
CapForm capacityForm
|
||||
Since string
|
||||
Message string
|
||||
Error string
|
||||
CSRF string
|
||||
Version string
|
||||
Now string
|
||||
}
|
||||
|
||||
// RegisterAdminQuotas mounts quota administration under /admin/quotas.
|
||||
@@ -176,6 +182,24 @@ func (p *AccountPages) RegisterAdminQuotas(mux *http.ServeMux, s QuotaAdmin, pri
|
||||
}
|
||||
p.renderQuotas(w, r, s, msg, "")
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /admin/quotas/capacity", p.adminOnly(
|
||||
func(w http.ResponseWriter, r *http.Request, admin store.User) {
|
||||
sess, _, _ := p.session(r)
|
||||
if p.capacity == nil || !p.accounts.CheckCSRF(sess, r.FormValue("csrf")) {
|
||||
http.Error(w, "bad request", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
msg, err := p.applyCapacity(r, admin)
|
||||
if err != nil {
|
||||
p.log.Warn("a capacity quota change was refused",
|
||||
"admin", admin.Email, "scope", r.FormValue("scope"), "err", err)
|
||||
p.renderQuotas(w, r, s, "", err.Error())
|
||||
return
|
||||
}
|
||||
p.log.Info("capacity quota changed", "admin", admin.Email, "what", msg)
|
||||
p.renderQuotas(w, r, s, msg, "")
|
||||
}))
|
||||
}
|
||||
|
||||
// applyQuota writes one scope's limits, and says what it did.
|
||||
@@ -259,6 +283,7 @@ func (p *AccountPages) renderQuotas(w http.ResponseWriter, r *http.Request,
|
||||
|
||||
d.BudgetsBind = p.priced != nil && p.priced.AnythingPriced()
|
||||
d.Editing = r.URL.Query().Get("edit")
|
||||
p.capacitySection(r.Context(), &d, r.FormValue("cap"))
|
||||
|
||||
limits, err := s.AllLimits(r.Context())
|
||||
if err != nil {
|
||||
|
||||
@@ -22,6 +22,11 @@ type CatalogAdmin interface {
|
||||
// Scale changes how a model added at runtime runs: always, on demand, or
|
||||
// not at all. One declared in configuration is refused.
|
||||
Scale(ctx context.Context, name string, sc catalog.Scaling, by string) (catalog.Model, error)
|
||||
// SetClass, SetOwner and Resources change what a model's capacity is
|
||||
// worth, whose quota it counts against, and what a replica reserves.
|
||||
SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
|
||||
SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
|
||||
Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
|
||||
}
|
||||
|
||||
// serveForm is the offer to put one mirrored version into service.
|
||||
@@ -49,6 +54,9 @@ type serveForm struct {
|
||||
// 3.5 GB for a KV pool. Nothing was wrong with placement — it was never
|
||||
// told what the model needed.
|
||||
MinMemoryGB int
|
||||
// CPUCores is the cores a replica computes on: half the smallest node it
|
||||
// fits on, or zero when no node says how many it has.
|
||||
CPUCores int
|
||||
// MemoryBasis says how MinMemoryGB was arrived at, for the form to show.
|
||||
MemoryBasis string
|
||||
// CtxPool is the engine context pool the form starts with, and the one
|
||||
@@ -152,6 +160,15 @@ func planServe(st mirror.Status, files []mirror.Status, base string,
|
||||
defaultCtxPool, header.BlockCount, kvHeads,
|
||||
float64(int64(1<<30)+weights/20)/(1<<30))
|
||||
}
|
||||
if fleet != nil {
|
||||
var cpus []int
|
||||
for _, n := range fleet.Nodes() {
|
||||
if n.State == "ready" && (n.MemoryGB == 0 || n.MemoryGB >= f.MinMemoryGB) {
|
||||
cpus = append(cpus, n.CPUCount)
|
||||
}
|
||||
}
|
||||
f.CPUCores = catalog.SuggestCores(cpus)
|
||||
}
|
||||
if biggest := biggestNode(fleet); biggest > 0 && f.MinMemoryGB > biggest {
|
||||
f.Warning = strings.TrimSuffix(f.Warning, ".") +
|
||||
fmt.Sprintf("%sthe largest node has %d GB and this needs about %d",
|
||||
@@ -277,6 +294,11 @@ func buildModel(get func(string) string) (catalog.Model, error) {
|
||||
return m, err
|
||||
}
|
||||
}
|
||||
if v := strings.TrimSpace(get("cpu_cores")); v != "" {
|
||||
if p.CPUCores, err = positive(v, "the cores"); err != nil {
|
||||
return m, err
|
||||
}
|
||||
}
|
||||
if k, v := strings.TrimSpace(get("selector_key")), strings.TrimSpace(get("selector_value")); k != "" && v != "" {
|
||||
// What keeps a large model off the small nodes. A CPU profile has no
|
||||
// card to be short of, so without this every ready node looks like a
|
||||
|
||||
@@ -3,6 +3,7 @@ package web
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
@@ -21,6 +22,9 @@ type fakeCatalog struct {
|
||||
scaled []string
|
||||
lastScale catalog.Scaling
|
||||
err error
|
||||
classes []string
|
||||
owners []string
|
||||
resources []string
|
||||
}
|
||||
|
||||
func (f *fakeCatalog) Add(_ context.Context, m catalog.Model, _ string) error {
|
||||
@@ -35,6 +39,21 @@ func (f *fakeCatalog) Sources(context.Context) (map[string][]catalog.Model, erro
|
||||
return f.sources, nil
|
||||
}
|
||||
|
||||
func (f *fakeCatalog) SetClass(_ context.Context, name, class, _ string) (catalog.Model, error) {
|
||||
f.classes = append(f.classes, name+"="+class)
|
||||
return catalog.Model{Name: name, Class: class}, f.err
|
||||
}
|
||||
|
||||
func (f *fakeCatalog) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
|
||||
f.owners = append(f.owners, name+"="+owner)
|
||||
return catalog.Model{Name: name, Owner: owner}, f.err
|
||||
}
|
||||
|
||||
func (f *fakeCatalog) Resources(_ context.Context, name, _ string, mem, cores int, _ string) (catalog.Model, error) {
|
||||
f.resources = append(f.resources, fmt.Sprintf("%s=%d/%d", name, mem, cores))
|
||||
return catalog.Model{Name: name}, f.err
|
||||
}
|
||||
|
||||
func (f *fakeCatalog) Scale(_ context.Context, name string, sc catalog.Scaling, _ string) (catalog.Model, error) {
|
||||
f.scaled = append(f.scaled, name)
|
||||
f.lastScale = sc
|
||||
|
||||
@@ -45,6 +45,13 @@ type DashboardData struct {
|
||||
|
||||
HasPeople bool
|
||||
|
||||
// NodePrefix is what every node's name starts with, left off in tables
|
||||
// so a row fits on a line; the full name is on hover.
|
||||
NodePrefix string
|
||||
// Columns shown only when some row has something in them: a CPU-only
|
||||
// cluster's nodes table was three columns of dashes wide.
|
||||
AnyGPUs, AnyClasses, AnyFabrics, AnyFallbacks bool
|
||||
|
||||
// Charts are pre-rendered rather than assembled in the template: building
|
||||
// SVG out of template fragments is how a chart ends up unescaped.
|
||||
RequestChart template.HTML
|
||||
@@ -177,6 +184,17 @@ func Dashboard(src Source, usage UsageSource, version string, refresh time.Durat
|
||||
}
|
||||
}
|
||||
d.Unplaced = unplaced(src, d.Models)
|
||||
ids := make([]string, 0, len(d.Nodes))
|
||||
for _, n := range d.Nodes {
|
||||
ids = append(ids, n.ID)
|
||||
d.AnyGPUs = d.AnyGPUs || n.GPUTotal > 0
|
||||
d.AnyClasses = d.AnyClasses || len(n.Classes) > 0
|
||||
d.AnyFabrics = d.AnyFabrics || len(n.Fabrics) > 0
|
||||
}
|
||||
d.NodePrefix = sharedPrefix(ids)
|
||||
for _, m := range d.Models {
|
||||
d.AnyFallbacks = d.AnyFallbacks || len(m.Fallbacks) > 0
|
||||
}
|
||||
d.Summary = src.Summary()
|
||||
if fs, ok := src.(FabricSource); ok {
|
||||
r := fs.Fabrics()
|
||||
|
||||
@@ -0,0 +1,284 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// HeadroomSource is a fleet that can say how many more replicas of a model
|
||||
// fit now. Optional: without it the card shows what runs and offers no change
|
||||
// it cannot check.
|
||||
type HeadroomSource interface {
|
||||
Headroom(model string, limit int) (adminapi.HeadroomView, bool)
|
||||
}
|
||||
|
||||
// maxReplicas bounds what the form accepts. Not a cluster limit, a typing one:
|
||||
// 300 where 3 was meant would otherwise be taken as a ceiling.
|
||||
const maxReplicas = 64
|
||||
|
||||
// heldDetail is the card opened beneath a held version: what runs it, where,
|
||||
// and room to change it.
|
||||
type heldDetail struct {
|
||||
Models []heldModel
|
||||
// Scopes are the parts of the tenancy tree a model can belong to; empty
|
||||
// when capacity quotas are not set up, and the owner choice is left off.
|
||||
Scopes []adminapi.ScopeName
|
||||
}
|
||||
|
||||
// heldModel is one catalog model running a held version.
|
||||
type heldModel struct {
|
||||
modelUse
|
||||
View adminapi.ModelView
|
||||
Replicas []heldReplica
|
||||
Headroom *adminapi.HeadroomView
|
||||
// MostNow is how many replicas could always run, Current plus the room
|
||||
// found; zero when the room was not measured or had no end in sight.
|
||||
MostNow int
|
||||
}
|
||||
|
||||
// heldReplica is one replica, by the nodes it runs on.
|
||||
type heldReplica struct {
|
||||
ID string
|
||||
State string
|
||||
Restart string
|
||||
Age int64
|
||||
Endpoint string
|
||||
Nodes []heldShard
|
||||
}
|
||||
|
||||
type heldShard struct {
|
||||
Node string
|
||||
MemoryGB int // the node's RAM, 0 when unknown
|
||||
CPU float64
|
||||
MemBytes int64
|
||||
}
|
||||
|
||||
// openKey names a row the way ?open= does.
|
||||
func openKey(model, version string) string { return model + ":" + version }
|
||||
|
||||
// heldDetailFor reads, for each model using a row, how it runs and how much
|
||||
// more room it has.
|
||||
func (p *AccountPages) heldDetailFor(ctx context.Context, row mirrorRow) *heldDetail {
|
||||
d := &heldDetail{}
|
||||
if p.capacity != nil {
|
||||
d.Scopes, _ = p.capacity.Scopes(ctx)
|
||||
}
|
||||
if p.fleet == nil {
|
||||
for _, u := range row.UsedBy {
|
||||
d.Models = append(d.Models, heldModel{modelUse: u})
|
||||
}
|
||||
return d
|
||||
}
|
||||
views := map[string]adminapi.ModelView{}
|
||||
for _, v := range p.fleet.Models() {
|
||||
views[v.Name] = v
|
||||
}
|
||||
memory := map[string]int{}
|
||||
for _, n := range p.fleet.Nodes() {
|
||||
memory[n.ID] = n.MemoryGB
|
||||
}
|
||||
hs, _ := p.fleet.(HeadroomSource)
|
||||
for _, u := range row.UsedBy {
|
||||
hm := heldModel{modelUse: u, View: views[u.Name]}
|
||||
for _, rep := range p.fleet.Replicas(u.Name) {
|
||||
// Built field by field: a ReplicaView carries the engine's key,
|
||||
// which is for the admin listener and never for a page.
|
||||
hr := heldReplica{ID: rep.ID, State: rep.State, Restart: rep.Restart,
|
||||
Age: rep.AgeSec, Endpoint: rep.Endpoint}
|
||||
for _, sh := range rep.Shards {
|
||||
hr.Nodes = append(hr.Nodes, heldShard{Node: sh.Node, MemoryGB: memory[sh.Node],
|
||||
CPU: sh.CPUCores, MemBytes: sh.MemoryBytes})
|
||||
}
|
||||
hm.Replicas = append(hm.Replicas, hr)
|
||||
}
|
||||
if hs != nil {
|
||||
if h, ok := hs.Headroom(u.Name, 16); ok {
|
||||
hm.Headroom = &h
|
||||
if !h.Limited {
|
||||
hm.MostNow = h.Current + h.More + h.ByPreempting
|
||||
}
|
||||
}
|
||||
}
|
||||
d.Models = append(d.Models, hm)
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// setReplicas changes how many replicas a model always runs and how far it
|
||||
// may grow, refusing an always-on count the cluster has no room for.
|
||||
//
|
||||
// Refused rather than taken and left short: a minimum placement cannot meet
|
||||
// is a model reported as unplaceable on every pass from then on, which is the
|
||||
// same answer given later and less plainly. A maximum above the room is
|
||||
// accepted and said: it is reached only under load, by which time something
|
||||
// may have made room, and refusing it would make the ceiling depend on what
|
||||
// happened to be running when it was set.
|
||||
func (p *AccountPages) setReplicas(ctx context.Context, admin store.User, r *http.Request) (string, error) {
|
||||
name := r.FormValue("model")
|
||||
lo, err1 := strconv.Atoi(strings.TrimSpace(r.FormValue("min")))
|
||||
hi, err2 := strconv.Atoi(strings.TrimSpace(r.FormValue("max")))
|
||||
switch {
|
||||
case name == "":
|
||||
return "", errors.New("which model?")
|
||||
case err1 != nil || err2 != nil:
|
||||
return "", errors.New("replicas are whole numbers")
|
||||
case lo < 0 || hi < lo:
|
||||
return "", fmt.Errorf("always running (%d) cannot be more than the most under load (%d)", lo, hi)
|
||||
case hi > maxReplicas:
|
||||
return "", fmt.Errorf("at most %d replicas", maxReplicas)
|
||||
}
|
||||
current, err := p.currentModel(ctx, name)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if p.catalog.Declared(name) {
|
||||
return "", fmt.Errorf("%s is set in server.yaml; change it there", name)
|
||||
}
|
||||
|
||||
note := ""
|
||||
if hs, ok := p.fleet.(HeadroomSource); ok && hi > 0 {
|
||||
h, ok := hs.Headroom(name, hi)
|
||||
if !ok {
|
||||
return "", fmt.Errorf("no model %q", name)
|
||||
}
|
||||
fits := h.Current + h.More + h.ByPreempting
|
||||
if lo > fits && !h.Limited {
|
||||
why := "nothing more fits"
|
||||
if len(h.Blocked) > 0 {
|
||||
why = strings.Join(h.Blocked, "; ")
|
||||
}
|
||||
return "", fmt.Errorf("%s can always run at most %d now (%d running, room for %d more): %s",
|
||||
name, fits, h.Current, h.More, why)
|
||||
}
|
||||
if hi > fits && !h.Limited {
|
||||
note = fmt.Sprintf("; there is room for %d now, so it grows past that only if room is made", fits)
|
||||
}
|
||||
// Room its class would make by stopping others: said, since it is
|
||||
// somebody else's replica that goes.
|
||||
if lo > h.Current+h.More && len(h.Preempts) > 0 {
|
||||
note += "; to make room, replicas of " + strings.Join(h.Preempts, ", ") + " will be stopped"
|
||||
}
|
||||
}
|
||||
|
||||
sc := catalog.Scaling{Min: lo, Max: hi, IdleTimeout: current.Scaling.IdleTimeout}
|
||||
if sc.OnDemand() && sc.IdleTimeout <= 0 {
|
||||
sc.IdleTimeout = defaultIdle
|
||||
}
|
||||
if _, err := p.catalog.Scale(ctx, name, sc, admin.Email); err != nil {
|
||||
return "", err
|
||||
}
|
||||
switch {
|
||||
case hi == 0:
|
||||
return name + " is stopped; its replicas are being taken down", nil
|
||||
case lo == 0:
|
||||
return fmt.Sprintf("%s loads when asked for, up to %d under load%s", name, hi, note), nil
|
||||
case lo == hi:
|
||||
return fmt.Sprintf("%s runs %s%s", name, plural(lo, "replica", "replicas"), note), nil
|
||||
}
|
||||
return fmt.Sprintf("%s runs %d, and up to %d under load%s", name, lo, hi, note), nil
|
||||
}
|
||||
|
||||
// currentModel is a model as the catalog holds it now.
|
||||
func (p *AccountPages) currentModel(ctx context.Context, name string) (catalog.Model, error) {
|
||||
sources, err := p.catalog.Sources(ctx)
|
||||
if err != nil {
|
||||
return catalog.Model{}, err
|
||||
}
|
||||
for _, ms := range sources {
|
||||
for _, m := range ms {
|
||||
if m.Name == name {
|
||||
return m, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
return catalog.Model{}, fmt.Errorf("no model %q runs from the mirror", name)
|
||||
}
|
||||
|
||||
// age renders seconds the way the replicas table does.
|
||||
func age(sec int64) string {
|
||||
d := time.Duration(sec) * time.Second
|
||||
switch {
|
||||
case d < time.Minute:
|
||||
return fmt.Sprintf("%ds", sec)
|
||||
case d < time.Hour:
|
||||
return fmt.Sprintf("%dm", int(d.Minutes()))
|
||||
case d < 48*time.Hour:
|
||||
return fmt.Sprintf("%dh%02dm", int(d.Hours()), int(d.Minutes())%60)
|
||||
}
|
||||
return fmt.Sprintf("%dd", int(d.Hours()/24))
|
||||
}
|
||||
|
||||
// setModel applies the card's settings form: what a replica reserves, the
|
||||
// model's class and owner, then how many replicas it runs — in that order, so
|
||||
// the room checked for the replicas is the room its new class and reservation
|
||||
// would get. Only what changed is applied, and each change is said.
|
||||
func (p *AccountPages) setModel(ctx context.Context, admin store.User, r *http.Request) (string, error) {
|
||||
name := r.FormValue("model")
|
||||
current, err := p.currentModel(ctx, name)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if p.catalog.Declared(name) {
|
||||
return "", fmt.Errorf("%s is set in server.yaml; change it there", name)
|
||||
}
|
||||
var done []string
|
||||
if r.Form.Has("memory") && len(current.Profiles) > 0 {
|
||||
mem, err1 := strconv.Atoi(strings.TrimSpace(r.FormValue("memory")))
|
||||
cores, err2 := strconv.Atoi(strings.TrimSpace(r.FormValue("cores")))
|
||||
if err1 != nil || err2 != nil || mem < 0 || cores < 0 {
|
||||
return "", errors.New("memory and cores are whole numbers")
|
||||
}
|
||||
if cur := current.Profiles[0]; mem != cur.MinMemoryGB || cores != cur.CPUCores {
|
||||
if _, err := p.catalog.Resources(ctx, name, "", mem, cores, admin.Email); err != nil {
|
||||
return "", err
|
||||
}
|
||||
done = append(done, fmt.Sprintf("a replica reserves %d GB and %d cores (running ones keep theirs until restarted)", mem, cores))
|
||||
}
|
||||
}
|
||||
if r.Form.Has("class") {
|
||||
if c := r.FormValue("class"); c != current.Class {
|
||||
m, err := p.catalog.SetClass(ctx, name, c, admin.Email)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
done = append(done, "is "+m.EffectiveClass())
|
||||
}
|
||||
}
|
||||
if r.Form.Has("owner") && p.capacity != nil {
|
||||
if o := r.FormValue("owner"); o != current.Owner {
|
||||
if _, err := p.catalog.SetOwner(ctx, name, o, admin.Email); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if o == "" {
|
||||
done = append(done, "is shared")
|
||||
} else {
|
||||
done = append(done, "counts against "+o)
|
||||
}
|
||||
}
|
||||
}
|
||||
lo, _ := strconv.Atoi(strings.TrimSpace(r.FormValue("min")))
|
||||
hi, _ := strconv.Atoi(strings.TrimSpace(r.FormValue("max")))
|
||||
if r.Form.Has("min") && (lo != current.Scaling.Min || hi != current.Scaling.Max) {
|
||||
msg, err := p.setReplicas(ctx, admin, r)
|
||||
if err != nil {
|
||||
if len(done) > 0 {
|
||||
return "", fmt.Errorf("%s %s, but not the replicas: %w", name, strings.Join(done, ", "), err)
|
||||
}
|
||||
return "", err
|
||||
}
|
||||
done = append(done, strings.TrimPrefix(msg, name+" "))
|
||||
}
|
||||
if len(done) == 0 {
|
||||
return name + ": nothing changed", nil
|
||||
}
|
||||
return name + " " + strings.Join(done, "; "), nil
|
||||
}
|
||||
@@ -0,0 +1,271 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// roomyFleet runs one replica of "coder" on n2 and has room for `more`.
|
||||
type roomyFleet struct {
|
||||
fakeFleet
|
||||
more int
|
||||
blocked []string
|
||||
asked int
|
||||
preempt int
|
||||
preempts []string
|
||||
}
|
||||
|
||||
func (f *roomyFleet) Nodes() []adminapi.NodeView {
|
||||
return []adminapi.NodeView{{ID: "n1", MemoryGB: 16}, {ID: "n2", MemoryGB: 32}}
|
||||
}
|
||||
func (f *roomyFleet) Models() []adminapi.ModelView {
|
||||
return []adminapi.ModelView{{Name: "coder", Min: 1, Max: 1, Ready: 1, Total: 1, Desired: 1}}
|
||||
}
|
||||
func (f *roomyFleet) Replicas(string) []adminapi.ReplicaView {
|
||||
return []adminapi.ReplicaView{{ID: "coder-abc", Model: "coder", State: "ready", AgeSec: 3700,
|
||||
EngineKey: "secret-engine-key",
|
||||
Shards: []adminapi.ShardView{{Node: "n2", CPUCores: 3.5, MemoryBytes: 6 << 30}}}}
|
||||
}
|
||||
func (f *roomyFleet) Headroom(model string, limit int) (adminapi.HeadroomView, bool) {
|
||||
f.asked = limit
|
||||
more := min(f.more, limit)
|
||||
h := adminapi.HeadroomView{Model: model, Current: 1, More: more}
|
||||
for range more {
|
||||
h.Nodes = append(h.Nodes, []string{"n1"})
|
||||
}
|
||||
if more < limit {
|
||||
h.Blocked = f.blocked
|
||||
} else {
|
||||
h.Limited = true
|
||||
}
|
||||
h.ByPreempting, h.Preempts = f.preempt, f.preempts
|
||||
return h, true
|
||||
}
|
||||
|
||||
func detailPage(t *testing.T, fleet *roomyFleet, cat *fakeCatalog) func(string) string {
|
||||
t.Helper()
|
||||
m := &fakeMirror{models: []mirror.Status{
|
||||
{Model: "coder", Version: "v1", File: "c.gguf", State: mirror.Present, BytesDone: 5 << 30},
|
||||
}}
|
||||
mux := mirrorPageDeps(t, MirrorDeps{Mirror: m, Catalog: cat, Fleet: fleet}, true)
|
||||
return func(path string) string { return mirrorGet(t, mux, path).Body.String() }
|
||||
}
|
||||
|
||||
func coderCatalog() *fakeCatalog {
|
||||
return &fakeCatalog{sources: map[string][]catalog.Model{"http://mirror:8081/coder/v1/c.gguf": {
|
||||
{Name: "coder", Scaling: catalog.Scaling{Min: 1, Max: 1}}}}}
|
||||
}
|
||||
|
||||
func TestAHeldModelOpensToWhereItRunsAndHowMuchRoomIsLeft(t *testing.T) {
|
||||
fleet := &roomyFleet{more: 2, blocked: []string{"n2: 2 GB free of 32, profile needs 6"}}
|
||||
get := detailPage(t, fleet, coderCatalog())
|
||||
|
||||
closed := get("/admin/mirror")
|
||||
if strings.Contains(closed, "held-card") {
|
||||
t.Error("a card is open before anything was clicked")
|
||||
}
|
||||
if !strings.Contains(closed, `href="/admin/mirror?tab=held&open=coder%3av1"`) {
|
||||
t.Error("the model's name does not open its card")
|
||||
}
|
||||
|
||||
body := get("/admin/mirror?tab=held&open=coder:v1")
|
||||
for _, want := range []string{
|
||||
"held-card", `<a href="/admin/nodes/n2">n2</a>`, "32 GB", "coder-abc", "1h01m", "3.5 cores", "6.0 GiB",
|
||||
"Room for 2 more now, on n1, n1.", "n2: 2 GB free of 32, profile needs 6",
|
||||
`name="min" value="1" min="0" max="3"`, `action="/admin/mirror/replicas"`,
|
||||
`name="open" value="coder:v1"`,
|
||||
} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the card is missing %q", want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(body, "secret-engine-key") {
|
||||
t.Error("the engine's key reached the page")
|
||||
}
|
||||
if strings.Contains(body, `http-equiv="refresh"`) {
|
||||
t.Error("a page with an open form reloads under the operator")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReplicaCountTheClusterHasNoRoomForIsRefused(t *testing.T) {
|
||||
fleet := &roomyFleet{more: 1, blocked: []string{"n1: 1 GB free of 16, profile needs 6"}}
|
||||
cat := coderCatalog()
|
||||
mux := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{models: []mirror.Status{
|
||||
{Model: "coder", Version: "v1", File: "c.gguf", State: mirror.Present, BytesDone: 1},
|
||||
}}, Catalog: cat, Fleet: fleet}, true)
|
||||
post := func(lo, hi string) string {
|
||||
return mirrorPost(t, mux, "/admin/mirror/replicas", url.Values{"csrf": {"tok"},
|
||||
"model": {"coder"}, "min": {lo}, "max": {hi}, "open": {"coder:v1"}}).Body.String()
|
||||
}
|
||||
|
||||
body := post("3", "3")
|
||||
if len(cat.scaled) != 0 {
|
||||
t.Fatalf("three always-on replicas were set with room for two: %+v", cat.lastScale)
|
||||
}
|
||||
if !strings.Contains(body, "can always run at most 2 now") || !strings.Contains(body, "1 GB free of 16") {
|
||||
t.Errorf("the refusal should say how many fit and why: %s", body)
|
||||
}
|
||||
if !strings.Contains(body, "held-card") {
|
||||
t.Error("the card closed on the refusal")
|
||||
}
|
||||
|
||||
body = post("2", "4")
|
||||
if len(cat.scaled) != 1 || cat.lastScale.Min != 2 || cat.lastScale.Max != 4 {
|
||||
t.Fatalf("what fits was not set: %+v", cat.lastScale)
|
||||
}
|
||||
if !strings.Contains(body, "coder runs 2, and up to 4 under load; there is room for 2 now") {
|
||||
t.Errorf("a ceiling above the room should be taken and said: %s", body)
|
||||
}
|
||||
|
||||
post("0", "2")
|
||||
if cat.lastScale.Min != 0 || cat.lastScale.Max != 2 || cat.lastScale.IdleTimeout == 0 {
|
||||
t.Errorf("on demand was set as %+v", cat.lastScale)
|
||||
}
|
||||
for _, bad := range [][2]string{{"3", "2"}, {"-1", "1"}, {"x", "1"}, {"1", "100"}} {
|
||||
n := len(cat.scaled)
|
||||
post(bad[0], bad[1])
|
||||
if len(cat.scaled) != n {
|
||||
t.Errorf("min %s max %s was accepted", bad[0], bad[1])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAModelSetInConfigurationIsShownButNotChanged(t *testing.T) {
|
||||
cat := coderCatalog()
|
||||
cat.declared = "coder"
|
||||
body := detailPage(t, &roomyFleet{more: 20}, cat)("/admin/mirror?open=coder:v1")
|
||||
if strings.Contains(body, `action="/admin/mirror/replicas"`) || !strings.Contains(body, "change its replicas there") {
|
||||
t.Error("a declared model offered a change that would revert")
|
||||
}
|
||||
if !strings.Contains(body, "Room for at least 16 more") {
|
||||
t.Error("room that reached the limit asked should read as at least")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAHeldRowSaysHowItsModelRunsWithoutOpening(t *testing.T) {
|
||||
body := detailPage(t, &roomyFleet{more: 1}, coderCatalog())("/admin/mirror")
|
||||
if strings.Contains(body, "held-card") {
|
||||
t.Fatal("the card opened by itself")
|
||||
}
|
||||
if !strings.Contains(body, `title="replicas ready / wanted">1/1 · <span class="node" title="n2">n2</span></span>`) {
|
||||
t.Errorf("the row does not say where its replicas run: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunningOnNamesEachNodeOnceAndCountsWhatIsStarting(t *testing.T) {
|
||||
nodes, starting := runningOn([]adminapi.ReplicaView{
|
||||
{State: "ready", Shards: []adminapi.ShardView{{Node: "n2"}}},
|
||||
{State: "starting", Shards: []adminapi.ShardView{{Node: "n1"}}},
|
||||
{State: "ready", Shards: []adminapi.ShardView{{Node: "n2"}}},
|
||||
})
|
||||
if strings.Join(nodes, ",") != "n1,n2" || starting != 1 {
|
||||
t.Errorf("nodes %v, starting %d", nodes, starting)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNodeNamesLoseOnlyWhatEveryNodeShares(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
ids []string
|
||||
want string
|
||||
}{
|
||||
{[]string{"llmgw-node01", "llmgw-node02", "llmgw-node06"}, "llmgw-"},
|
||||
{[]string{"gpu-a.dc1", "gpu-b.dc1"}, "gpu-"},
|
||||
{[]string{"n1", "n2"}, ""},
|
||||
{[]string{"llmgw-node01"}, ""},
|
||||
{[]string{"alpha", "beta"}, ""},
|
||||
} {
|
||||
if got := sharedPrefix(c.ids); got != c.want {
|
||||
t.Errorf("%v: %q, expected %q", c.ids, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func settingsPage(t *testing.T, cat *fakeCatalog, withQuotas bool) *http.ServeMux {
|
||||
t.Helper()
|
||||
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
|
||||
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.DiscardHandler))
|
||||
if withQuotas {
|
||||
p.SetCapacity(&fakeCapacity{})
|
||||
}
|
||||
mux := http.NewServeMux()
|
||||
p.RegisterAdminMirror(mux, MirrorDeps{Mirror: &fakeMirror{models: []mirror.Status{
|
||||
{Model: "coder", Version: "v1", File: "c.gguf", State: mirror.Present, BytesDone: 1},
|
||||
}}, Catalog: cat, Fleet: &roomyFleet{more: 3}})
|
||||
return mux
|
||||
}
|
||||
|
||||
func TestACardSetsClassOwnerAndReservationAndOnlyWhatChanged(t *testing.T) {
|
||||
cat := coderCatalog()
|
||||
cat.sources["http://mirror:8081/coder/v1/c.gguf"][0].Profiles = []catalog.Profile{{Name: "cpu", MinMemoryGB: 12, CPUCores: 4}}
|
||||
mux := settingsPage(t, cat, true)
|
||||
|
||||
card := mirrorGet(t, mux, "/admin/mirror?open=coder:v1").Body.String()
|
||||
for _, want := range []string{`name="class"`, `<option value="" selected>default (standard)</option>`,
|
||||
`name="owner"`, `<option value="team:t1">Acme / ml</option>`, `name="memory" value="12"`, `name="cores" value="4"`} {
|
||||
if !strings.Contains(card, want) {
|
||||
t.Errorf("the card lacks %q", want)
|
||||
}
|
||||
}
|
||||
post := func(v url.Values) string {
|
||||
v.Set("csrf", "tok")
|
||||
v.Set("model", "coder")
|
||||
v.Set("open", "coder:v1")
|
||||
return mirrorPost(t, mux, "/admin/mirror/replicas", v).Body.String()
|
||||
}
|
||||
same := url.Values{"min": {"1"}, "max": {"1"}, "class": {""}, "owner": {""}, "memory": {"12"}, "cores": {"4"}}
|
||||
if body := post(same); !strings.Contains(body, "coder: nothing changed") ||
|
||||
len(cat.classes)+len(cat.owners)+len(cat.resources)+len(cat.scaled) != 0 {
|
||||
t.Errorf("an unchanged form changed something: %v %v %v %v", cat.classes, cat.owners, cat.resources, cat.scaled)
|
||||
}
|
||||
changed := url.Values{"min": {"1"}, "max": {"1"}, "class": {"critical"}, "owner": {"team:t1"}, "memory": {"14"}, "cores": {"4"}}
|
||||
body := post(changed)
|
||||
if strings.Join(cat.resources, ",") != "coder=14/4" || strings.Join(cat.classes, ",") != "coder=critical" ||
|
||||
strings.Join(cat.owners, ",") != "coder=team:t1" || len(cat.scaled) != 0 {
|
||||
t.Errorf("applied %v %v %v %v", cat.resources, cat.classes, cat.owners, cat.scaled)
|
||||
}
|
||||
if !strings.Contains(body, "a replica reserves 14 GB and 4 cores") || !strings.Contains(body, "counts against team:t1") {
|
||||
t.Errorf("the change was not said: %s", body)
|
||||
}
|
||||
|
||||
// Without capacity quotas there is no owner to choose, and none is sent.
|
||||
plain := mirrorGet(t, settingsPage(t, coderCatalog(), false), "/admin/mirror?open=coder:v1").Body.String()
|
||||
if strings.Contains(plain, `name="owner"`) {
|
||||
t.Error("an owner choice with no quotas to count against")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRoomAHigherClassWouldMakeIsOfferedAndSaid(t *testing.T) {
|
||||
fleet := &roomyFleet{more: 0, preempt: 1, preempts: []string{"chat"}, blocked: []string{"n1: full"}}
|
||||
cat := coderCatalog()
|
||||
mux := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{models: []mirror.Status{
|
||||
{Model: "coder", Version: "v1", File: "c.gguf", State: mirror.Present, BytesDone: 1},
|
||||
}}, Catalog: cat, Fleet: fleet}, true)
|
||||
|
||||
card := mirrorGet(t, mux, "/admin/mirror?open=coder:v1").Body.String()
|
||||
if !strings.Contains(card, "No free room for another replica now.") ||
|
||||
!strings.Contains(card, "1 by stopping replicas its class outranks: chat.") ||
|
||||
!strings.Contains(card, `name="min" value="1" min="0" max="2"`) {
|
||||
t.Errorf("the card should offer the room preemption makes, and say whose")
|
||||
}
|
||||
body := mirrorPost(t, mux, "/admin/mirror/replicas", url.Values{"csrf": {"tok"},
|
||||
"model": {"coder"}, "min": {"2"}, "max": {"2"}, "open": {"coder:v1"}}).Body.String()
|
||||
if len(cat.scaled) != 1 || cat.lastScale.Min != 2 {
|
||||
t.Fatalf("two were refused though preemption makes room: %s", body)
|
||||
}
|
||||
if !strings.Contains(body, "replicas of chat will be stopped") {
|
||||
t.Errorf("stopping somebody else's replica should be said: %s", body)
|
||||
}
|
||||
body = mirrorPost(t, mux, "/admin/mirror/replicas", url.Values{"csrf": {"tok"},
|
||||
"model": {"coder"}, "min": {"3"}, "max": {"3"}, "open": {"coder:v1"}}).Body.String()
|
||||
if len(cat.scaled) != 1 || !strings.Contains(body, "at most 2") {
|
||||
t.Errorf("three is past even what preemption makes: %s", body)
|
||||
}
|
||||
}
|
||||
@@ -155,7 +155,7 @@ func oneOf(s string, options ...string) bool {
|
||||
|
||||
// href is the page with these settings and one of them changed.
|
||||
func (e explorer) href(query, key, value string) template.URL {
|
||||
v := url.Values{"view": {e.View}, "task": {e.Task}, "size": {e.Size}, "x": {"1"}}
|
||||
v := url.Values{"tab": {tabExplore}, "view": {e.View}, "task": {e.Task}, "size": {e.Size}, "x": {"1"}}
|
||||
if e.Here {
|
||||
v.Set("here", "1")
|
||||
}
|
||||
|
||||
@@ -27,7 +27,7 @@ func TestTheExplorerOpensOnTrendingModelsTheNodesCanRun(t *testing.T) {
|
||||
ggufModel("odd/Kev-GGUF", "ggmlc", 5e8, now.Add(-2*24*time.Hour)),
|
||||
}}
|
||||
body := mirrorGet(t, mirrorPageLlama(t, MirrorDeps{Mirror: &fakeMirror{}, Registry: reg}),
|
||||
"/admin/mirror").Body.String()
|
||||
"/admin/mirror?tab=explore").Body.String()
|
||||
|
||||
q := reg.query
|
||||
if q.Sort != hf.SortTrending || q.Task != "text-generation" || !q.GGUF || q.Search != "" {
|
||||
@@ -96,7 +96,7 @@ func TestAnAdapterIsNotListedAsAModel(t *testing.T) {
|
||||
ggufModel("a/Tiny-GGUF", "qwen2", 135e6, time.Now()),
|
||||
}}
|
||||
body := mirrorGet(t, mirrorPageLlama(t, MirrorDeps{Mirror: &fakeMirror{}, Registry: reg}),
|
||||
"/admin/mirror").Body.String()
|
||||
"/admin/mirror?tab=explore").Body.String()
|
||||
if strings.Contains(body, "Some-LoRA") || !strings.Contains(body, "a/Tiny-GGUF") {
|
||||
t.Errorf("adapters should be left out and small models kept: %s", body)
|
||||
}
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/topology"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
@@ -174,3 +175,33 @@ func TestTheNVLinkPageDrawsEachDomainAsARack(t *testing.T) {
|
||||
t.Error("the cluster page does not lead to the racks")
|
||||
}
|
||||
}
|
||||
|
||||
// hotSource is a GPU node whose sensors report, one card held back.
|
||||
type hotSource struct{ fakeSource }
|
||||
|
||||
func (hotSource) Nodes() []adminapi.NodeView {
|
||||
n := fakeSource{}.Nodes()[0]
|
||||
n.Load = &adminapi.LoadView{CPUPercent: 40, CPUs: 64, MemTotal: 1 << 40, MemUsed: 1 << 39,
|
||||
CPUTempC: 58, CPUPowerW: 312, InletTempC: 24.5,
|
||||
GPUs: []inventory.GPUSensor{
|
||||
{Index: 0, TempC: 71, MemoryTempC: 84, PowerW: 612, PowerLimitW: 700},
|
||||
{Index: 1, TempC: 88, PowerW: 698, PowerLimitW: 700, Throttle: []string{"sw_power_cap", "sw_thermal"}},
|
||||
}}
|
||||
return []adminapi.NodeView{n}
|
||||
}
|
||||
|
||||
func TestANodesSensorsAreOnItsPageAndNothingItCannotMeasure(t *testing.T) {
|
||||
_, body := get(t, nodeMux(hotSource{}, nil), "http://x/admin/nodes/llmgw-node01", nil)
|
||||
for _, want := range []string{"58 °C", "312 W", "24.5 °C", "inlet air", "84 °C", "612 W", "of 700",
|
||||
`<span class="bad">sw_power_cap, sw_thermal</span>`} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the page lacks %q", want)
|
||||
}
|
||||
}
|
||||
_, vm := get(t, nodeMux(busySource{}, nil), "http://x/admin/nodes/llmgw-node01", nil)
|
||||
for _, absent := range []string{"°C", "<h2>GPUs</h2>", "inlet air"} {
|
||||
if strings.Contains(vm, absent) {
|
||||
t.Errorf("a node with no sensors shows %q", absent)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
// cpuFleet is a CPU-only cluster whose nodes share a name prefix.
|
||||
type cpuFleet struct{ fakeSource }
|
||||
|
||||
func (cpuFleet) Nodes() []adminapi.NodeView {
|
||||
return []adminapi.NodeView{
|
||||
{ID: "llmgw-node01", State: "ready", MgmtAddr: "10.0.0.141", MemoryGB: 8},
|
||||
{ID: "llmgw-node02", State: "ready", MgmtAddr: "10.0.0.142", MemoryGB: 8},
|
||||
}
|
||||
}
|
||||
func (cpuFleet) Summary() adminapi.Summary { return adminapi.Summary{} }
|
||||
func (cpuFleet) Models() []adminapi.ModelView {
|
||||
return []adminapi.ModelView{{Name: "qwen", Min: 1, Max: 1, Ready: 1,
|
||||
Description: "A sentence long enough that it once had a row of its own under the model."}}
|
||||
}
|
||||
func (cpuFleet) Replicas(string) []adminapi.ReplicaView {
|
||||
return []adminapi.ReplicaView{{ID: "qwen-1", Model: "qwen", State: "ready",
|
||||
Shards: []adminapi.ShardView{{Node: "llmgw-node02"}}}}
|
||||
}
|
||||
|
||||
func TestTheClusterPageKeepsEachRecordOnOneLine(t *testing.T) {
|
||||
h := Dashboard(cpuFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil)
|
||||
_, body := get(t, h, "http://127.0.0.1:9090/", nil)
|
||||
|
||||
for _, gone := range []string{"<th>Classes</th>", "<th>Fabrics</th>", `<th class="num">GPUs</th>`, "<th>Fallbacks</th>"} {
|
||||
if strings.Contains(body, gone) {
|
||||
t.Errorf("%s is shown with nothing in it on any row", gone)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(body, `title="llmgw-node01"><code>node01</code>`) {
|
||||
t.Error("node names should drop what every node shares, keeping the whole on hover")
|
||||
}
|
||||
if !strings.Contains(body, `<code title="llmgw-node02">node02</code>`) {
|
||||
t.Error("the replicas table should name nodes the same way")
|
||||
}
|
||||
if strings.Contains(body, `class="url-row"`) || !strings.Contains(body, `<td class="clip" title="A sentence long enough`) {
|
||||
t.Error("a model's description should be a clipped cell, not a row of its own")
|
||||
}
|
||||
|
||||
// A cluster with GPUs still gets its GPU column.
|
||||
_, gpu := get(t, Dashboard(fakeSource{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if !strings.Contains(gpu, `<th class="num">GPUs</th>`) || !strings.Contains(gpu, "<th>Fabrics</th>") {
|
||||
t.Error("columns with something in them were hidden")
|
||||
}
|
||||
}
|
||||
@@ -60,7 +60,7 @@ li { margin: .25rem 0; }
|
||||
a { color: var(--accent); }
|
||||
.wrap.narrow { max-width: 30rem; }
|
||||
label { display: block; margin: .9rem 0; font-size: .9rem; font-weight: 600; }
|
||||
input[type=text], input[type=email], input[type=password], input[type=search] {
|
||||
input[type=text], input[type=email], input[type=password], input[type=search], input[type=number] {
|
||||
display: block; width: 100%; margin-top: .3rem; font: inherit; font-weight: 400;
|
||||
padding: .5rem .6rem; border: 1px solid var(--line); border-radius: 6px;
|
||||
background: var(--card); color: var(--fg);
|
||||
@@ -444,18 +444,29 @@ h3.limits-for { font-size: .92rem; margin: 1rem 0 .4rem; }
|
||||
table.narrow-table { max-width: 30rem; }
|
||||
.q-act a + a { margin-left: .8rem; }
|
||||
.chip-role { font-size: .68rem; font-weight: 600; color: var(--accent); text-transform: uppercase; letter-spacing: .04em; }
|
||||
/* Runs: one fixed-width selector per model, so they line up down the column
|
||||
whatever the names; a catalog name differing from the mirror's sits above. */
|
||||
table.held .use { width: 9.5rem; margin: .1rem 0; }
|
||||
table.held .use .as { display: block; font-size: .72rem; color: var(--muted);
|
||||
overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
table.held .use .opt { font-size: .75rem; line-height: 1.3; }
|
||||
/* Runs: one line per model — a catalog name that differs from the mirror's,
|
||||
how it runs, and where — with the selector at a fixed width so the
|
||||
replica counts line up down the column. */
|
||||
table.held .use { display: flex; align-items: center; gap: .5rem; margin: .1rem 0; white-space: nowrap; }
|
||||
table.held .use .as { font-size: .72rem; color: var(--muted); max-width: 10rem;
|
||||
overflow: hidden; text-overflow: ellipsis; }
|
||||
table.held .use form.inline { margin: 0; }
|
||||
table.held .use .mode-fixed { width: 7.5rem; font-size: .8rem; color: var(--muted); }
|
||||
table.held td.num { white-space: nowrap; }
|
||||
table.held select.mode { width: 100%; padding: .15rem .3rem; font-size: .8rem; }
|
||||
table.held select.mode { width: 7.5rem; margin: 0; padding: .1rem .3rem; font-size: .8rem; }
|
||||
|
||||
/* Link pills: the same look as .tabs, for choices that are addresses rather
|
||||
than panels on one page — a filtered listing should survive a reload and be
|
||||
worth sending to somebody. */
|
||||
/* A page's own sections, one address each. Underlined rather than pills so
|
||||
* they do not read as one more filter beside the ones inside a section. */
|
||||
.page-tabs { display: flex; flex-wrap: wrap; gap: 0 1.5rem; margin: -1rem 0 1.5rem;
|
||||
border-bottom: 1px solid var(--line); }
|
||||
.page-tabs a { padding: .55rem 0; margin-bottom: -1px; font-weight: 600; color: var(--muted);
|
||||
text-decoration: none; border-bottom: 2px solid transparent; }
|
||||
.page-tabs a:hover { color: var(--fg); }
|
||||
.page-tabs a.on { color: var(--fg); border-bottom-color: var(--accent); }
|
||||
.page-tabs .count { font-weight: 500; font-size: .8rem; color: var(--muted); margin-left: .25rem; }
|
||||
.pills { display: flex; flex-wrap: wrap; gap: .35rem; margin: .5rem 0; }
|
||||
.pills a { font-size: .85rem; font-weight: 600; color: var(--muted); padding: .35rem .85rem;
|
||||
border: 1px solid transparent; border-radius: 999px; text-decoration: none; }
|
||||
@@ -804,3 +815,46 @@ details.chat-settings output { font-variant-numeric: tabular-nums; font-weight:
|
||||
.copyable button { margin-right: .75rem; font-size: .8rem; }
|
||||
.connect-form .field-row label { flex: 1 1 12rem; }
|
||||
table.connect-fields th { white-space: nowrap; }
|
||||
|
||||
/* The card opened beneath a held version on the mirror page. */
|
||||
tr.held-detail > td { background: var(--bg); padding: .75rem 1rem 1rem; }
|
||||
.held-card .held-model { margin-top: .75rem; padding-top: .75rem; border-top: 1px solid var(--line); }
|
||||
.held-card .held-model:first-of-type { border-top: 0; padding-top: 0; }
|
||||
.held-card h3 { font-size: 1rem; margin: 0 0 .5rem; display: flex; gap: .6rem; align-items: baseline; flex-wrap: wrap; }
|
||||
.held-card table.replicas { margin: .25rem 0 .5rem; }
|
||||
.held-card .headroom { margin: .5rem 0 .25rem; font-weight: 600; }
|
||||
.held-card details.blocked { font-size: .85rem; color: var(--muted); margin-bottom: .5rem; }
|
||||
.held-card details.blocked ul { margin: .35rem 0 0 1.1rem; padding: 0; }
|
||||
.held-card .replicas-form .field-row { align-items: flex-end; }
|
||||
.held-card .replicas-form .field-row label { flex: 0 0 auto; }
|
||||
.held-card .replicas-form input[type=number] { width: 8rem; margin: .25rem 0 0; }
|
||||
.held-card .replicas-form button { margin: 0; }
|
||||
a.open-held { text-decoration: none; color: inherit; }
|
||||
a.open-held:hover strong { text-decoration: underline; }
|
||||
a.open-held[aria-expanded=true]::before { content: "▾ "; color: var(--muted); }
|
||||
a.open-held[aria-expanded=false]::before { content: "▸ "; color: var(--muted); }
|
||||
.use .runs-now { font-size: .78rem; color: var(--muted); }
|
||||
.use .runs-now.short { color: var(--warn); }
|
||||
.use .runs-now.bad { color: var(--bad); }
|
||||
.use .runs-now .node { white-space: nowrap; }
|
||||
|
||||
/* One line per record, in every table. A header never wraps; a cell of free
|
||||
* text is cut to its column with the whole of it on hover, instead of a row of
|
||||
* its own or a line break that makes one record look like two. */
|
||||
th { white-space: nowrap; }
|
||||
table.rows td { white-space: nowrap; }
|
||||
td.clip { max-width: 22rem; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
table.rows td.clip { color: var(--muted); font-size: .85rem; }
|
||||
/* A row across the whole table is a note or an open form, not a record: it
|
||||
* wraps as prose does. */
|
||||
table.rows td[colspan] { white-space: normal; }
|
||||
table.rows td.where { max-width: none; }
|
||||
table.rows .more { cursor: help; margin-left: .2rem; }
|
||||
table.explore td:first-child { max-width: 34rem; overflow: hidden; text-overflow: ellipsis; }
|
||||
table.explore .from { margin-left: .5rem; font-size: .78rem; color: var(--muted); }
|
||||
.clip-inline { display: inline-block; max-width: 24rem; overflow: hidden; text-overflow: ellipsis;
|
||||
vertical-align: bottom; font-size: .8rem; color: var(--muted); margin-left: .4rem; }
|
||||
table.rows td.warn { color: var(--warn); cursor: help; }
|
||||
.held-card .replicas-form select, .capacity-form select { display: block; margin-top: .25rem; }
|
||||
.held-card .replicas-form .field-row { flex-wrap: wrap; gap: .75rem 1rem; }
|
||||
.quota-tree td.q-act form.inline { margin-left: .75rem; }
|
||||
@@ -84,7 +84,7 @@ Tell whoever you asked for access that you have signed up.</p>
|
||||
{{end}}
|
||||
{{if .Fields}}
|
||||
<h3>Settings to fill in</h3>
|
||||
<table class="compact narrow-table connect-fields">{{range .Fields}}
|
||||
<table class="rows compact narrow-table connect-fields">{{range .Fields}}
|
||||
<tr><th>{{.Label}}</th><td><code>{{.Value}}</code></td><td><button type="button" class="link copy" data-value="{{.Value}}">copy</button></td></tr>{{end}}
|
||||
</table>
|
||||
{{end}}
|
||||
@@ -149,7 +149,7 @@ document.addEventListener("click", (e) => {
|
||||
|
||||
<h2>Keys</h2>
|
||||
{{if .Keys}}
|
||||
<table><tr><th>Name</th><th>Key</th><th>Spends against</th><th>Created</th><th>Last used</th><th>State</th><th></th></tr>
|
||||
<table class="rows"><tr><th>Name</th><th>Key</th><th>Spends against</th><th>Created</th><th>Last used</th><th>State</th><th></th></tr>
|
||||
{{range .Keys}}<tr>
|
||||
<td>{{.Name}}</td>
|
||||
<td><code>{{.Prefix}}</code></td>
|
||||
@@ -192,7 +192,7 @@ old one to stop moving, then revoke it.</p>
|
||||
</div>
|
||||
{{end}}
|
||||
{{if .Rows}}
|
||||
<table class="compact narrow-table"><tr><th>Limit</th><th class="num">Value</th></tr>
|
||||
<table class="rows compact narrow-table"><tr><th>Limit</th><th class="num">Value</th></tr>
|
||||
{{range .Rows}}<tr><td>{{.Name}}</td><td class="num">{{.Value}}</td></tr>{{end}}</table>
|
||||
{{else}}<p class="empty">No limits here.</p>{{end}}
|
||||
{{else}}<p class="empty">Nowhere to spend yet.</p>{{end}}
|
||||
|
||||
@@ -8,16 +8,20 @@
|
||||
</head><body><div class="wrap">
|
||||
<header>
|
||||
<h1>Model mirror</h1>
|
||||
<p class="sub">
|
||||
{{if .Unreachable}}{{.Addr}} did not answer{{else}}{{.Present}} held ·
|
||||
{{.Fetching}} downloading · {{.Failed}} failed · {{.Addr}}{{end}}
|
||||
</p>
|
||||
<p class="sub">{{.Addr}}{{if .Unreachable}} did not answer{{end}}</p>
|
||||
{{template "nav" .Nav}}
|
||||
</header>
|
||||
|
||||
{{/* Links rather than a script's panels: each tab has an address, the back
|
||||
button works, and only the one shown asks anything of the registry. */}}
|
||||
<nav class="page-tabs" aria-label="Mirror">
|
||||
{{range .Tabs}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="page"{{end}}>{{.Label}}{{with .Count}} <span class="count">{{.}}</span>{{end}}</a>
|
||||
{{end}}</nav>
|
||||
|
||||
{{if .Error}}<p class="error">{{.Error}}</p>{{end}}
|
||||
{{if .Message}}<p class="ok">{{.Message}}</p>{{end}}
|
||||
|
||||
{{if eq .Tab "held"}}
|
||||
{{if .Unreachable}}
|
||||
{{/* Said outright rather than shown as an empty table. A mirror holding
|
||||
nothing and a mirror that is not there look the same in a list of zero
|
||||
@@ -29,9 +33,8 @@ reachable from this machine.</p>
|
||||
{{else}}
|
||||
|
||||
<div id="held">
|
||||
<h2>What the mirror holds</h2>
|
||||
{{if .Rows}}
|
||||
<table class="compact held">
|
||||
<table class="rows compact held">
|
||||
<tr><th>Model</th><th>Files</th><th class="num">Size</th><th>Runs</th><th></th></tr>
|
||||
{{range .Groups}}{{$grouped := gt .Models 1}}
|
||||
{{if $grouped}}
|
||||
@@ -41,8 +44,10 @@ reachable from this machine.</p>
|
||||
{{if .Failed}}<span class="tag bad">failed</span>{{end}}
|
||||
{{if .InUse}}<span class="tag">in use</span>{{end}}</td></tr>
|
||||
{{end}}
|
||||
{{range .Rows}}<tr{{if $grouped}} class="model-version"{{end}}>
|
||||
<td><strong>{{.Model}}</strong> <span class="opt">{{.Version}}</span></td>
|
||||
{{range .Rows}}{{$row0 := .}}<tr{{if $grouped}} class="model-version"{{end}}{{if .Detail}} class="open"{{end}}>
|
||||
<td>{{if eq (printf "%s" .State) "present"}}<a class="open-held" aria-expanded="{{if .Detail}}true{{else}}false{{end}}"
|
||||
href="/admin/mirror?tab=held{{if not .Detail}}&open={{.Open}}{{end}}"><strong>{{.Model}}</strong></a>
|
||||
{{else}}<strong>{{.Model}}</strong>{{end}} <span class="opt">{{.Version}}</span></td>
|
||||
<td class="file" title="{{.File}}{{if .SHA256}} · sha256 {{.SHA256}}{{end}}">
|
||||
{{if gt .Files 1}}{{.Files}} files{{else}}<code>{{.File}}</code>{{end}}</td>
|
||||
<td class="num">
|
||||
@@ -62,7 +67,8 @@ reachable from this machine.</p>
|
||||
{{/* Named only when the catalog calls it something other than the
|
||||
mirror does; otherwise it is the model column again. */}}
|
||||
{{if ne .Name $row.Model}}<span class="as" title="{{.Name}}">as {{.Name}}</span>{{end}}
|
||||
{{if .Declared}}<span class="opt">{{if eq .Mode "always"}}always on{{else if eq .Mode "ondemand"}}on demand{{else}}stopped{{end}}<br>set in server.yaml</span>
|
||||
{{if .Declared}}<span class="mode-fixed" title="set in server.yaml, which is read again on every start">
|
||||
{{- if eq .Mode "always"}}always on{{else if eq .Mode "ondemand"}}on demand{{else}}stopped{{end}} · yaml</span>
|
||||
{{else}}
|
||||
{{/* Submitted on change: the choice is the action, and a second click
|
||||
to confirm it would only be a way to forget to. */}}
|
||||
@@ -78,6 +84,12 @@ reachable from this machine.</p>
|
||||
<noscript><button type="submit" class="link">set</button></noscript>
|
||||
</form>
|
||||
{{end}}
|
||||
{{if .Fleet}}<span class="runs-now{{if or .Parked .Unplaceable}} bad{{else if lt .Ready .Wanted}} short{{end}}" title="replicas ready / wanted{{if .Starting}}, + starting{{end}}">
|
||||
{{- if .Parked}}parked
|
||||
{{- else if and (eq .Wanted 0) (eq .Ready 0)}}none running
|
||||
{{- else}}{{.Ready}}/{{.Wanted}}{{if .Starting}} +{{.Starting}}{{end}}{{end}}
|
||||
{{- if .Unplaceable}} · no room{{end}}
|
||||
{{- with .Nodes}} · {{range $i, $n := .}}{{if $i}}, {{end}}<span class="node" title="{{$n.Full}}">{{$n.Short}}</span>{{end}}{{end}}</span>{{end}}
|
||||
</div>{{end}}
|
||||
{{else}}<span class="opt">not served</span>{{end}}
|
||||
</td>
|
||||
@@ -109,6 +121,7 @@ reachable from this machine.</p>
|
||||
</td>
|
||||
</tr>
|
||||
{{if .Err}}<tr><td colspan="5" class="error-row">{{.Err}}</td></tr>{{end}}
|
||||
{{with .Detail}}<tr class="held-detail"><td colspan="5">{{template "mirror_held_detail" dict "D" . "Row" $row0 "CSRF" $.CSRF}}</td></tr>{{end}}
|
||||
|
||||
{{with .Serve}}
|
||||
<tr class="editing"><td colspan="5">
|
||||
@@ -152,7 +165,9 @@ reachable from this machine.</p>
|
||||
<div class="field-row">
|
||||
<label>Profile name <input type="text" name="profile" value="{{.Profile}}"></label>
|
||||
<label>Needs RAM (GB) <span class="opt">for the context pool beside it</span>
|
||||
<input type="text" name="min_memory_gb" value="{{.MinMemoryGB}}"></label>
|
||||
<input type="text" name="min_memory_gb" value="{{.MinMemoryGB}}" required></label>
|
||||
<label>CPU cores <span class="opt">reserved; the engine's threads</span>
|
||||
<input type="text" name="cpu_cores" value="{{.CPUCores}}"></label>
|
||||
<label>Only nodes labelled <span class="opt">key / value, optional</span>
|
||||
<input type="text" name="selector_key" placeholder="role" size="8">
|
||||
<input type="text" name="selector_value" placeholder="cpu-large" size="10"></label>
|
||||
@@ -190,11 +205,14 @@ document.addEventListener('click', function (e) {
|
||||
} else { window.prompt('Address', b.dataset.url); }
|
||||
});
|
||||
</script>
|
||||
{{else}}<p class="empty">The mirror is reachable and holds nothing yet.</p>{{end}}
|
||||
{{else}}<p class="empty">The mirror is reachable and holds nothing yet.
|
||||
{{if .HasRegistry}}<a href="/admin/mirror?tab=explore">Explore Hugging Face</a> for a model, or
|
||||
{{end}}<a href="/admin/mirror?tab=fetch">fetch one by URL</a>.</p>{{end}}
|
||||
</div>
|
||||
{{end}}
|
||||
{{end}}
|
||||
|
||||
{{if .HasRegistry}}
|
||||
<h2>Explore models</h2>
|
||||
{{if and (eq .Tab "explore") .HasRegistry}}
|
||||
{{if not .Repo}}{{with .Explore}}
|
||||
<nav class="pills" aria-label="Which models">{{range .Views}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="page"{{end}}>{{.Label}}</a>{{end}}</nav>
|
||||
<div class="explore-filters">
|
||||
@@ -203,6 +221,7 @@ document.addEventListener('click', function (e) {
|
||||
</div>
|
||||
{{end}}
|
||||
<form method="get" action="/admin/mirror" class="explore-search">
|
||||
<input type="hidden" name="tab" value="explore">
|
||||
{{with .Explore}}<input type="hidden" name="view" value="{{.View}}">
|
||||
<input type="hidden" name="task" value="{{.Task}}">
|
||||
<input type="hidden" name="size" value="{{.Size}}">
|
||||
@@ -222,13 +241,13 @@ document.addEventListener('click', function (e) {
|
||||
|
||||
{{if not .Repo}}{{with .Explore}}
|
||||
{{if .Rows}}
|
||||
<table class="explore">
|
||||
<table class="rows explore">
|
||||
<tr><th>Repository</th><th class="num">Size</th><th>Architecture</th><th class="num">Context</th>
|
||||
<th class="num">Downloads</th><th class="num">Likes</th><th class="num">Updated</th></tr>
|
||||
{{range .Rows}}<tr class="explore-row">
|
||||
<td><a href="{{.Href}}" class="open-repo" data-repo="{{.ID}}" aria-expanded="false"><code>{{.ID}}</code></a>
|
||||
{{if .New}}<span class="tag new">new</span>{{end}}{{if .Gated.Is}}<span class="tag">gated</span>{{end}}
|
||||
{{if or .BaseModel .License}}<div class="from">{{with .BaseModel}}from {{.}}{{end}}{{if and .BaseModel .License}} · {{end}}{{.License}}</div>{{end}}</td>
|
||||
{{if or .BaseModel .License}}<span class="from" title="{{with .BaseModel}}from {{.}}{{end}}{{if and .BaseModel .License}} · {{end}}{{.License}}">{{with .BaseModel}}from {{.}}{{end}}{{if and .BaseModel .License}} · {{end}}{{.License}}</span>{{end}}</td>
|
||||
<td class="num">{{.ParamsLabel}}</td>
|
||||
<td>{{if .Architecture}}<span{{if not .Verdict.Servable}} class="cannot"{{end}} title="{{.Verdict.Reason}}">{{.Architecture}}{{if .Verdict.Checked}}{{if .Verdict.Servable}} ✓{{else}} ✗{{end}}{{end}}</span>{{else}}—{{end}}</td>
|
||||
<td class="num">{{.ContextLabel}}</td>
|
||||
@@ -307,9 +326,12 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
return;
|
||||
}
|
||||
note.classList.remove("bad");
|
||||
note.textContent = "✓ " + out.message + " — its progress is in the table above.";
|
||||
note.textContent = "✓ " + out.message + " — ";
|
||||
const link = document.createElement("a");
|
||||
link.href = "/admin/mirror?tab=held";
|
||||
link.textContent = "follow it under Held";
|
||||
note.appendChild(link);
|
||||
btn.textContent = "Started";
|
||||
followHeld();
|
||||
} catch (err) {
|
||||
note.textContent = "Could not reach the gateway: " + err;
|
||||
note.classList.add("bad");
|
||||
@@ -317,28 +339,6 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
btn.textContent = was;
|
||||
}
|
||||
});
|
||||
|
||||
// Refreshes only the mirror's own table, every five seconds while anything
|
||||
// in it is downloading.
|
||||
let following = false;
|
||||
async function followHeld() {
|
||||
if (following) return;
|
||||
following = true;
|
||||
try {
|
||||
for (;;) {
|
||||
const r = await fetch(location.pathname + location.search, { headers: { Accept: "text/html" } });
|
||||
if (!r.ok) break;
|
||||
const doc = new DOMParser().parseFromString(await r.text(), "text/html");
|
||||
const fresh = doc.getElementById("held"), held = document.getElementById("held");
|
||||
if (!fresh || !held) break;
|
||||
held.replaceWith(fresh);
|
||||
if (!fresh.querySelector(".state-starting")) break;
|
||||
await new Promise((ok) => setTimeout(ok, 5000));
|
||||
}
|
||||
} finally {
|
||||
following = false;
|
||||
}
|
||||
}
|
||||
</script>
|
||||
{{if .Hidden}}<p class="note">{{.Hidden}} more {{if eq .Hidden 1}}is{{else}}are{{end}} hidden: their architecture is one the nodes' llama.cpp cannot load.
|
||||
Untick “only what these nodes can run” to see them.</p>{{end}}
|
||||
@@ -354,10 +354,11 @@ directory, or give the mirror a token.</p>
|
||||
{{if .Repo}}{{template "mirror_repo" .}}{{end}}
|
||||
{{end}}
|
||||
|
||||
<h2>Fetch a model</h2>
|
||||
{{if eq .Tab "fetch"}}
|
||||
<p class="note">By hand, for anything the search does not reach.</p>
|
||||
<form method="post" action="/admin/mirror/fetch">
|
||||
<input type="hidden" name="csrf" value="{{.CSRF}}">
|
||||
<input type="hidden" name="tab" value="fetch">
|
||||
<label>Source URL
|
||||
<input type="text" name="source" required
|
||||
placeholder="https://huggingface.co/…/resolve/main/model.gguf"></label>
|
||||
@@ -373,7 +374,7 @@ directory, or give the mirror a token.</p>
|
||||
</form>
|
||||
<p class="note">The mirror answers as soon as the download is under way, not when
|
||||
it finishes — a model is tens of gigabytes and nothing here holds a connection
|
||||
open for that. The row above will show it arriving. A version is never
|
||||
open for that. The Held tab shows it arriving. A version is never
|
||||
overwritten: fetching into one that exists is refused, so a new build is a new
|
||||
version and the old weights stay where they are.</p>
|
||||
{{end}}
|
||||
@@ -400,7 +401,7 @@ version and the old weights stay where they are.</p>
|
||||
</details>
|
||||
{{end}}
|
||||
{{if .RepoSets}}
|
||||
<table class="repo-files">
|
||||
<table class="rows repo-files">
|
||||
<tr><th>What</th><th class="num">Size</th><th>Published checksum</th><th></th></tr>
|
||||
{{range .RepoSets}}<tr>
|
||||
<td><code>{{.Name}}</code>
|
||||
@@ -410,11 +411,12 @@ version and the old weights stay where they are.</p>
|
||||
<td>
|
||||
{{if not .Verdict.Servable}}
|
||||
<span class="state state-failed" title="{{.Verdict.Reason}}">cannot be served</span>
|
||||
<div class="note">{{.Verdict.Reason}}</div>
|
||||
<span class="clip-inline" title="{{.Verdict.Reason}}">{{.Verdict.Reason}}</span>
|
||||
{{else}}
|
||||
{{if .Verdict.Architecture}}<span class="opt" title="{{.Verdict.Reason}}">{{.Verdict.Architecture}}{{if .Verdict.Checked}} ✓{{end}}</span>{{end}}
|
||||
<form method="post" action="/admin/mirror/fetch" class="inline">
|
||||
<input type="hidden" name="csrf" value="{{$.CSRF}}">
|
||||
<input type="hidden" name="tab" value="explore">
|
||||
{{range .Sources}}<input type="hidden" name="source" value="{{.URL}}">
|
||||
<input type="hidden" name="file" value="{{.Path}}">
|
||||
{{end}}
|
||||
@@ -439,3 +441,93 @@ computes once it has it.</p>
|
||||
<p class="empty">No weight files in {{.Repo.ID}}.</p>
|
||||
{{end}}
|
||||
{{end}}
|
||||
|
||||
{{/* The card beneath a held version: what runs it, where, and how many. */}}
|
||||
{{define "mirror_held_detail"}}{{$row := .Row}}{{$csrf := .CSRF}}{{$scopes := .D.Scopes}}
|
||||
<div class="held-card">
|
||||
<p class="repo-meta">
|
||||
<span>{{if gt $row.Files 1}}{{$row.Files}} files{{else}}<code>{{$row.File}}</code>{{end}}</span>
|
||||
<span>{{bytes $row.BytesDone}}</span>
|
||||
{{with $row.SHA256}}<span>sha256 <code title="{{.}}">{{slice . 0 12}}</code></span>{{end}}
|
||||
<span><code>{{$row.URL}}</code></span>
|
||||
</p>
|
||||
{{if not .D.Models}}
|
||||
<p class="note">Not in the catalog: nothing runs this version.
|
||||
{{if $row.Serve}}{{else}}Use <strong>serve</strong> on its row to put it into service.{{end}}</p>
|
||||
{{end}}
|
||||
{{range .D.Models}}
|
||||
<section class="held-model">
|
||||
<h3>{{.Name}}
|
||||
{{with .View}}<span class="opt">{{.Ready}} of {{.Desired}} ready{{if ne .Total .Ready}} · {{.Total}} in all{{end}}</span>
|
||||
{{if .Parked}}<span class="tag bad">parked</span>{{else if .Unplaceable}}<span class="tag bad">does not fit</span>{{end}}{{end}}
|
||||
</h3>
|
||||
{{with .View.Reason}}<p class="note error-row">{{.}}</p>{{end}}
|
||||
|
||||
{{if .Replicas}}
|
||||
<table class="rows compact replicas">
|
||||
<tr><th>Replica</th><th>Node</th><th>State</th><th class="num">Age</th><th class="num">CPU</th><th class="num">Memory</th></tr>
|
||||
{{range .Replicas}}{{$rep := .}}{{range $i, $s := .Nodes}}<tr>
|
||||
<td>{{if eq $i 0}}<code>{{$rep.ID}}</code>{{end}}</td>
|
||||
<td><a href="/admin/nodes/{{$s.Node}}">{{$s.Node}}</a>{{if $s.MemoryGB}} <span class="opt">{{$s.MemoryGB}} GB</span>{{end}}</td>
|
||||
<td>{{if eq $i 0}}<span class="state state-{{if eq $rep.State "ready"}}ready{{else if eq $rep.State "failed"}}failed{{else}}starting{{end}}">{{$rep.State}}</span>
|
||||
{{with $rep.Restart}}<span class="opt">restart {{.}}</span>{{end}}{{end}}</td>
|
||||
<td class="num">{{if eq $i 0}}{{age $rep.Age}}{{end}}</td>
|
||||
<td class="num">{{if $s.CPU}}{{printf "%.1f" $s.CPU}} cores{{else}}—{{end}}</td>
|
||||
<td class="num">{{if $s.MemBytes}}{{bytes $s.MemBytes}}{{else}}—{{end}}</td>
|
||||
</tr>{{end}}{{end}}
|
||||
</table>
|
||||
{{else}}<p class="note">No replica runs now{{if eq .Mode "ondemand"}}: it starts when somebody asks for it{{end}}.</p>{{end}}
|
||||
|
||||
{{with .Headroom}}
|
||||
<p class="headroom">
|
||||
{{if .Limited}}Room for at least {{.More}} more replicas.
|
||||
{{else if .More}}Room for {{.More}} more now{{if .Nodes}}, on {{range $i, $n := .Nodes}}{{if $i}}, {{end}}{{join $n}}{{end}}{{end}}.
|
||||
{{else}}No free room for another replica now.{{end}}
|
||||
{{if .ByPreempting}}<br><span class="note">{{if .More}}And {{end}}{{.ByPreempting}} {{if .More}}more {{end}}by stopping replicas its class outranks: {{join .Preempts}}.</span>{{end}}
|
||||
</p>
|
||||
{{if and (not .Limited) .Blocked}}<details class="blocked"><summary>Why not {{if .More}}more{{else}}another{{end}}</summary>
|
||||
<ul>{{range .Blocked}}<li>{{.}}</li>{{end}}</ul></details>{{end}}
|
||||
{{end}}
|
||||
|
||||
{{if .Declared}}
|
||||
<p class="note">Set in server.yaml, which is read again on every start: change its replicas there.</p>
|
||||
{{else}}
|
||||
<form method="post" action="/admin/mirror/replicas" class="replicas-form">
|
||||
<input type="hidden" name="csrf" value="{{$csrf}}">
|
||||
<input type="hidden" name="model" value="{{.Name}}">
|
||||
<input type="hidden" name="open" value="{{$row.Open}}">
|
||||
<div class="field-row">
|
||||
<label>Always running
|
||||
<input type="number" name="min" value="{{.Min}}" min="0"{{if .MostNow}} max="{{.MostNow}}"{{end}} required></label>
|
||||
<label>Up to, under load
|
||||
<input type="number" name="max" value="{{.Max}}" min="0" max="64" required></label>
|
||||
<label>Class
|
||||
<select name="class">
|
||||
<option value=""{{if eq .Class ""}} selected{{end}}>default ({{if eq .Mode "ondemand"}}opportunistic{{else}}standard{{end}})</option>
|
||||
<option value="critical"{{if eq .Class "critical"}} selected{{end}}>critical</option>
|
||||
<option value="standard"{{if eq .Class "standard"}} selected{{end}}>standard</option>
|
||||
<option value="opportunistic"{{if eq .Class "opportunistic"}} selected{{end}}>opportunistic</option>
|
||||
</select></label>
|
||||
{{if $scopes}}{{$owner := .Owner}}<label>Owner
|
||||
<select name="owner">
|
||||
<option value=""{{if eq $owner ""}} selected{{end}}>shared</option>
|
||||
{{range $scopes}}<option value="{{.Key}}"{{if eq .Key $owner}} selected{{end}}>{{.Name}}</option>{{end}}
|
||||
</select></label>{{end}}
|
||||
</div>
|
||||
<div class="field-row">
|
||||
<label>Memory GB <span class="opt">per replica</span>
|
||||
<input type="number" name="memory" value="{{.MemoryGB}}" min="0"></label>
|
||||
<label>Cores <span class="opt">per replica, its threads</span>
|
||||
<input type="number" name="cores" value="{{.CPUCores}}" min="0"></label>
|
||||
<button type="submit">Save</button>
|
||||
</div>
|
||||
<p class="note">{{if .MostNow}}At most {{.MostNow}} can always run now{{if .Headroom}}{{if .Headroom.ByPreempting}}, counting what its class may stop to make room{{end}}{{end}}. {{end}}0 always running with a
|
||||
higher limit loads it when asked for; 0 and 0 stops it. A replica is added under load when requests wait, and
|
||||
taken away after ten quiet minutes. A higher class may stop replicas of lower ones to make room; memory and
|
||||
cores apply to replicas started after the change.</p>
|
||||
</form>
|
||||
{{end}}
|
||||
</section>
|
||||
{{end}}
|
||||
</div>
|
||||
{{end}}
|
||||
@@ -34,7 +34,7 @@ budgets bind.</p>
|
||||
{{/* The tree is the interface. A quota belongs to a branch, so the form to set
|
||||
one opens on that branch rather than asking somebody to name it again in a
|
||||
box at the bottom. */}}
|
||||
<table class="quota-tree">
|
||||
<table class="rows quota-tree">
|
||||
<tr><th>Where</th><th class="num">Requests/min</th><th class="num">Tokens/min</th>
|
||||
<th class="num">Budget</th><th>Spent</th><th></th></tr>
|
||||
{{range .Rows}}
|
||||
@@ -119,7 +119,7 @@ up — hover it to see where. A value set here is in normal type.</p>
|
||||
<p class="note">A person's own limit covers everything they do, across all their
|
||||
keys, projects and the chat page. {{.Personal}}</p>
|
||||
{{if .People}}
|
||||
<table class="quota-tree people-limits">
|
||||
<table class="rows quota-tree people-limits">
|
||||
<tr><th>Person</th><th class="num">Requests/min</th><th class="num">Tokens/min</th>
|
||||
<th class="num">Budget</th><th>Spent</th><th></th></tr>
|
||||
{{range .People}}
|
||||
@@ -147,6 +147,60 @@ keys, projects and the chat page. {{.Personal}}</p>
|
||||
</table>
|
||||
{{else}}<p class="empty">Nobody has been approved yet.</p>{{end}}
|
||||
|
||||
{{if .HasCapacity}}
|
||||
<h2 id="capacity">Capacity</h2>
|
||||
<p class="note">What a part of the tree is guaranteed of the cluster, and the most
|
||||
it may hold: GPUs, CPU cores and memory, counted from the replicas of the models
|
||||
it owns and of those beneath it. What it holds within its guarantee is never taken
|
||||
for someone else's model; beyond it, it borrows free room and gives it back when
|
||||
asked. A model's owner is set on its card on the mirror page.</p>
|
||||
{{if .Capacity}}
|
||||
<table class="rows quota-tree">
|
||||
<tr><th>Where</th><th class="num">GPUs</th><th class="num">Cores</th><th class="num">Memory GB</th><th>Set by</th><th></th></tr>
|
||||
{{range .Capacity}}<tr>
|
||||
<td class="q-where"><span class="q-name">{{or .Name .Scope}}</span></td>
|
||||
<td class="num">{{or .GPUs "—"}}</td>
|
||||
<td class="num">{{or .Cores "—"}}</td>
|
||||
<td class="num">{{or .Memory "—"}}</td>
|
||||
<td class="note">{{.UpdatedBy}}</td>
|
||||
<td class="q-act"><a href="/admin/quotas?cap={{.Scope}}#capacity">change</a>
|
||||
<form method="post" action="/admin/quotas/capacity" class="inline">
|
||||
<input type="hidden" name="csrf" value="{{$.CSRF}}">
|
||||
<input type="hidden" name="scope" value="{{.Scope}}">
|
||||
<button type="submit" name="clear" value="1" class="link danger">remove</button>
|
||||
</form></td>
|
||||
</tr>{{end}}
|
||||
</table>
|
||||
<p class="q-legend">Each figure is held of guaranteed (at most the limit); — is not governed.</p>
|
||||
{{else}}<p class="empty">No capacity quotas: every model shares the cluster by its class alone.</p>{{end}}
|
||||
|
||||
{{if .Scopes}}
|
||||
<form method="post" action="/admin/quotas/capacity" class="q-form capacity-form">
|
||||
<input type="hidden" name="csrf" value="{{.CSRF}}">
|
||||
<p class="q-form-title"><strong>{{if .CapForm.Scope}}Change{{else}}Set{{end}} a capacity quota.</strong>
|
||||
A blank field is not governed; 0 is none at all.</p>
|
||||
<label>Where
|
||||
<select name="scope" required>
|
||||
<option value="">choose…</option>
|
||||
{{range .Scopes}}<option value="{{.Key}}"{{if eq .Key $.CapForm.Scope}} selected{{end}}>{{.Name}}</option>{{end}}
|
||||
</select></label>
|
||||
<div class="field-row">
|
||||
<label>Guaranteed GPUs <input type="text" name="gpus" value="{{.CapForm.GPUs}}" placeholder="—"></label>
|
||||
<label>Cores <input type="text" name="cores" value="{{.CapForm.Cores}}" placeholder="—"></label>
|
||||
<label>Memory GB <input type="text" name="memory" value="{{.CapForm.Memory}}" placeholder="—"></label>
|
||||
</div>
|
||||
<div class="field-row">
|
||||
<label>At most GPUs <input type="text" name="limit_gpus" value="{{.CapForm.LGPUs}}" placeholder="—"></label>
|
||||
<label>Cores <input type="text" name="limit_cores" value="{{.CapForm.LCores}}" placeholder="—"></label>
|
||||
<label>Memory GB <input type="text" name="limit_memory" value="{{.CapForm.LMem}}" placeholder="—"></label>
|
||||
</div>
|
||||
<div class="q-form-actions"><button type="submit">Save</button>
|
||||
{{if .CapForm.Scope}}<a href="/admin/quotas#capacity">Cancel</a>{{end}}</div>
|
||||
</form>
|
||||
{{else}}<p class="empty">Nothing to give a capacity quota to yet. Make an organization on the
|
||||
<a href="/admin/users">people page</a> first.</p>{{end}}
|
||||
{{end}}
|
||||
|
||||
<details class="explain">
|
||||
<summary>How limits combine</summary>
|
||||
<p>Along a branch the most restrictive value wins, so a team cannot raise what
|
||||
|
||||
@@ -47,7 +47,7 @@
|
||||
</div>
|
||||
|
||||
{{if .Rows}}
|
||||
<table>
|
||||
<table class="rows">
|
||||
<tr><th>Group</th><th class="num">Requests</th><th class="num">Tokens</th><th class="num">Spend</th><th class="num">Errors</th></tr>
|
||||
{{range .Rows}}
|
||||
<tr>
|
||||
|
||||
@@ -22,7 +22,7 @@
|
||||
|
||||
{{if .Pending}}
|
||||
<h2>Waiting for approval</h2>
|
||||
<table><tr><th>Email</th><th>Name</th><th>Signed up</th><th>Approve, and put them in</th><th></th></tr>
|
||||
<table class="rows"><tr><th>Email</th><th>Name</th><th>Signed up</th><th>Approve, and put them in</th><th></th></tr>
|
||||
{{range .Pending}}<tr>
|
||||
<td>{{.Email}}</td><td>{{.Name}}</td><td>{{ago .CreatedAt}}</td>
|
||||
<td>
|
||||
@@ -149,14 +149,14 @@ project in the team — an account can only be approved into a project.</p>
|
||||
</div>
|
||||
|
||||
{{with index $.Members (printf "project:%s" $p.ID)}}
|
||||
<table class="compact"><tr><th>Member</th><th>State</th><th>Last sign-in</th><th></th></tr>
|
||||
<table class="rows compact"><tr><th>Member</th><th>State</th><th>Last sign-in</th><th></th></tr>
|
||||
{{range .}}{{template "member-row" (dict "U" . "D" $ "Scope" (printf "project:%s" $p.ID) "Open" $p.ID)}}{{end}}
|
||||
</table>
|
||||
{{else}}<p class="empty">No members of its own. Members of its team or organization can use it too.</p>{{end}}
|
||||
{{template "add-member" (dict "Kind" "project" "ID" $p.ID "D" $ "Open" $p.ID)}}
|
||||
|
||||
{{with index $.Keys $p.ID}}
|
||||
<table class="compact"><tr><th>Key</th><th>Owner</th><th>State</th><th>Last used</th><th></th></tr>
|
||||
<table class="rows compact"><tr><th>Key</th><th>Owner</th><th>State</th><th>Last used</th><th></th></tr>
|
||||
{{range .}}<tr>
|
||||
<td>{{.Name}} <code>{{.Prefix}}</code></td>
|
||||
<td>{{or .OwnerEmail "issued by an operator"}}</td>
|
||||
@@ -210,7 +210,7 @@ project in the team — an account can only be approved into a project.</p>
|
||||
|
||||
<h2 id="accounts">All accounts</h2>
|
||||
{{if .Accounts}}
|
||||
<table id="account-list"><tr><th>Person</th><th>State</th><th>Member of</th><th>Last sign-in</th><th>Add to</th><th></th></tr>
|
||||
<table class="rows" id="account-list"><tr><th>Person</th><th>State</th><th>Member of</th><th>Last sign-in</th><th>Add to</th><th></th></tr>
|
||||
{{range .Accounts}}{{template "account-row" (dict "U" . "D" $)}}{{end}}
|
||||
</table>
|
||||
<p class="note">Someone who is a member of nothing uses the gateway on their own:
|
||||
@@ -298,7 +298,7 @@ access than it had, so they need a shell on the control plane —
|
||||
<td>{{$u.Email}}{{if $u.IsAdmin}} <span class="tag">admin</span>{{end}}</td>
|
||||
<td class="state state-{{$u.State}}">{{$u.State}}</td>
|
||||
<td class="where">
|
||||
{{range $u.Memberships}}<span class="chip-m">{{.Path}}{{if eq .Role "manager"}} <b class="chip-role">manager</b>{{end}}
|
||||
{{range $i, $m := $u.Memberships}}{{if lt $i 2}}{{with $m}}<span class="chip-m">{{.Path}}{{if eq .Role "manager"}} <b class="chip-role">manager</b>{{end}}
|
||||
{{if eq (printf "%s" .Scope) "org"}}
|
||||
<form method="post" action="/admin/users/{{$u.ID}}/role" class="inline">
|
||||
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
|
||||
@@ -313,8 +313,12 @@ access than it had, so they need a shell on the control plane —
|
||||
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
|
||||
<input type="hidden" name="scope" value="{{.Scope}}:{{.ScopeID}}">
|
||||
<button type="submit" class="chip-x" aria-label="Take out of {{.Path}}" title="Take out of {{.Path}}">×</button>
|
||||
</form></span>
|
||||
</form></span>{{end}}{{end}}
|
||||
{{else}}<span class="opt">on their own</span>{{end}}
|
||||
{{/* Two, then a count: a person in six teams was six chips wrapping the
|
||||
row into a paragraph. The rest are named on hover and in their own
|
||||
scopes' member lists, where taking them out is done. */}}
|
||||
{{if gt (len $u.Memberships) 2}}<span class="opt more" title="{{range $i, $m := $u.Memberships}}{{if ge $i 2}}{{$m.Path}} {{end}}{{end}}">+{{sub (len $u.Memberships) 2}}</span>{{end}}
|
||||
</td>
|
||||
<td>{{ago $u.LastLoginAt}}</td>
|
||||
<td>
|
||||
@@ -365,7 +369,7 @@ access than it had, so they need a shell on the control plane —
|
||||
<details class="edit"><summary>{{len (index $d.Members $k)}} member{{if ne (len (index $d.Members $k)) 1}}s{{end}}</summary>
|
||||
<div class="scope-members">
|
||||
{{with index $d.Members $k}}
|
||||
<table class="compact">
|
||||
<table class="rows compact">
|
||||
{{range .}}{{template "member-row" (dict "U" . "D" $d "Scope" $k "Open" "")}}{{end}}
|
||||
</table>
|
||||
{{else}}<p class="note">Nobody is a member of {{.Name}} itself; a member here could use every project in it.</p>{{end}}
|
||||
|
||||
@@ -51,23 +51,23 @@
|
||||
|
||||
<h2>Nodes</h2>
|
||||
{{if .Nodes}}
|
||||
<table class="nodes"><tr><th>Node</th><th>State</th><th>CPU</th><th>RAM</th><th class="num">GPUs</th><th class="num">Disk</th><th>Engines</th><th>Classes</th><th>Fabrics</th><th class="num">Models</th><th class="num">Replicas</th><th>Last seen</th></tr>
|
||||
<table class="nodes rows"><tr><th>Node</th><th>State</th><th>CPU</th><th>RAM</th>{{if .AnyGPUs}}<th class="num">GPUs</th>{{end}}<th class="num">Disk</th><th>Engines</th>{{if .AnyClasses}}<th>Classes</th>{{end}}{{if .AnyFabrics}}<th>Fabrics</th>{{end}}<th class="num">Models</th><th class="num">Replicas</th><th>Seen</th></tr>
|
||||
{{range .Nodes}}<tr>
|
||||
<td><a href="{{if $.OnAdminListener}}/nodes/{{.ID}}{{else}}/admin/nodes/{{.ID}}{{end}}"><code>{{.ID}}</code></a>{{if .MgmtAddr}} <span class="note">{{.MgmtAddr}}</span>{{end}}
|
||||
<td><a href="{{if $.OnAdminListener}}/nodes/{{.ID}}{{else}}/admin/nodes/{{.ID}}{{end}}" title="{{.ID}}"><code>{{shortNode .ID $.NodePrefix}}</code></a>{{if .MgmtAddr}} <span class="note">{{.MgmtAddr}}</span>{{end}}
|
||||
<a class="note" href="{{if $.OnAdminListener}}/nodes/{{.ID}}/logs{{else}}/admin/nodes/{{.ID}}/logs{{end}}">logs</a></td>
|
||||
<td class="state state-{{.State}}">{{.State}}</td>
|
||||
{{/* A dash, not an empty bar, when the agent has not measured it lately:
|
||||
unknown is not idle. */}}
|
||||
<td class="load">{{with .Load}}{{template "minibar" pctbar .CPUPercent}}<span class="load-v">{{printf "%.0f" .CPUPercent}}%{{if .CPUs}} <span class="note">of {{.CPUs}}</span>{{end}}</span>{{else}}—{{end}}</td>
|
||||
<td class="load">{{with .Load}}{{template "minibar" pctbar .MemPercent}}<span class="load-v">{{bytes .MemUsed}} <span class="note">of {{bytes .MemTotal}}</span></span>{{else}}{{if .MemoryGB}}{{.MemoryGB}}G{{else}}—{{end}}{{end}}</td>
|
||||
<td class="num">{{if .GPUTotal}}{{.GPUFree}}/{{.GPUTotal}}{{else}}—{{end}}</td>
|
||||
{{if $.AnyGPUs}}<td class="num">{{if .GPUTotal}}{{.GPUFree}}/{{.GPUTotal}}{{else}}—{{end}}</td>{{end}}
|
||||
{{/* Free out of total, because the first is meaningless without the
|
||||
second, and an em dash where nothing was reported: an agent that
|
||||
was never asked about its disk must not read as a full one. */}}
|
||||
<td class="num">{{if .DiskTotalGB}}{{.DiskFreeGB}}G/{{.DiskTotalGB}}G{{else}}—{{end}}</td>
|
||||
<td>{{if .Engines}}{{join .Engines}}{{else}}<span class="note">none yet</span>{{end}}</td>
|
||||
<td>{{join .Classes}}</td>
|
||||
<td>{{join .Fabrics}}</td>
|
||||
{{if $.AnyClasses}}<td>{{join .Classes}}</td>{{end}}
|
||||
{{if $.AnyFabrics}}<td>{{join .Fabrics}}</td>{{end}}
|
||||
<td class="num">{{if .Models}}{{len .Models}}{{else}}—{{end}}</td>
|
||||
<td class="num">{{.Replicas}}</td>
|
||||
<td>{{or .LastSeenAgo "—"}}</td>
|
||||
@@ -80,18 +80,21 @@
|
||||
<h2>Models</h2>
|
||||
|
||||
{{if .Models}}
|
||||
<table><tr><th>Model</th><th>Aliases</th><th class="num">Ready</th><th class="num">Wanted</th><th>Profiles</th><th>Fallbacks</th></tr>
|
||||
<table class="rows"><tr><th>Model</th><th>Aliases</th><th class="num">Ready</th><th class="num">Wanted</th><th>Class</th><th>Reserves</th><th>Profiles</th>{{if .AnyFallbacks}}<th>Fallbacks</th>{{end}}<th>What it is for</th></tr>
|
||||
{{range .Models}}<tr>
|
||||
<td><code>{{.Name}}</code></td>
|
||||
<td>{{join .Aliases}}</td>
|
||||
<td class="num">{{.Ready}}</td>
|
||||
<td class="num">{{.Min}}–{{.Max}}</td>
|
||||
<td{{if .HeldFor}} class="warn" title="gave room to {{.HeldFor}}"{{end}}>{{.Class}}{{if .Preempted}} <span class="note" title="replicas lost to a higher class">−{{.Preempted}}</span>{{end}}</td>
|
||||
<td{{with .Warnings}} class="warn" title="{{join .}}"{{end}}>{{if or .MemoryGB .CPUCores}}{{.MemoryGB}}G{{if .CPUCores}} · {{.CPUCores}}c{{end}}{{else}}—{{end}}
|
||||
{{- if .MeasuredMemoryGB}} <span class="note">{{printf "%.1f" .MeasuredMemoryGB}}G used</span>{{end}}{{if .Warnings}} ⚠{{end}}</td>
|
||||
<td>{{join .Profiles}}</td>
|
||||
<td>{{join .Fallbacks}}</td>
|
||||
{{if $.AnyFallbacks}}<td>{{join .Fallbacks}}</td>{{end}}
|
||||
{{/* Cut to the column, the whole sentence on hover: a row of its own for
|
||||
each description made the table twice as long as the catalog. */}}
|
||||
<td class="clip"{{with .Description}} title="{{.}}"{{end}}>{{or .Description "—"}}</td>
|
||||
</tr>
|
||||
{{/* Under the row rather than in a column: a sentence in a table cell either
|
||||
wraps every other column into a stripe or gets cut off. */}}
|
||||
{{if .Description}}<tr><td colspan="6" class="url-row">{{.Description}}</td></tr>{{end}}
|
||||
{{end}}</table>
|
||||
{{else}}<p class="empty">The catalog is empty.</p>{{end}}
|
||||
|
||||
@@ -106,7 +109,7 @@
|
||||
|
||||
<h2>Replicas</h2>
|
||||
{{if .Replicas}}
|
||||
<table><tr><th>Replica</th><th>Model</th><th>State</th><th>Pool</th><th>Endpoint</th><th class="num">Age</th><th>Nodes</th></tr>
|
||||
<table class="rows"><tr><th>Replica</th><th>Model</th><th>State</th><th>Pool</th><th>Endpoint</th><th class="num">Age</th><th>Nodes</th></tr>
|
||||
{{range .Replicas}}<tr>
|
||||
<td><code>{{.ID}}</code></td>
|
||||
<td>{{.Model}}</td>
|
||||
@@ -114,7 +117,7 @@
|
||||
<td>{{or .Pool "—"}}</td>
|
||||
<td>{{if .Endpoint}}<code>{{.Endpoint}}</code>{{else}}—{{end}}</td>
|
||||
<td class="num">{{duration .AgeSec}}</td>
|
||||
<td>{{range .Shards}}<code>{{.Node}}</code> {{end}}</td>
|
||||
<td>{{range .Shards}}<code title="{{.Node}}">{{shortNode .Node $.NodePrefix}}</code> {{end}}</td>
|
||||
</tr>{{if .Err}}<tr><td colspan="7" class="note">{{.Err}}</td></tr>{{end}}
|
||||
{{end}}</table>
|
||||
{{else}}<p class="empty">No replicas.</p>{{end}}
|
||||
|
||||
@@ -21,6 +21,10 @@
|
||||
<div class="tile"><div class="v">{{printf "%.0f" .CPUPercent}}%</div><div class="k">CPU{{if .CPUs}} · {{.CPUs}} cores{{end}}</div>{{template "minibar" pctbar .CPUPercent}}</div>
|
||||
<div class="tile"><div class="v">{{bytes .MemUsed}}</div><div class="k">RAM in use · of {{bytes .MemTotal}}</div>{{template "minibar" pctbar .MemPercent}}</div>
|
||||
<div class="tile"><div class="v">{{printf "%.2f" .Load1}}</div><div class="k">load · {{printf "%.2f" .Load5}} 5m · {{printf "%.2f" .Load15}} 15m</div></div>
|
||||
{{/* Only what this machine has sensors for: a VM has none, and a tile
|
||||
saying 0 °C would be a reading nobody took. */}}
|
||||
{{if or .CPUTempC .CPUPowerW}}<div class="tile"><div class="v">{{if .CPUTempC}}{{printf "%.0f" .CPUTempC}} °C{{else}}—{{end}}</div><div class="k">CPU{{if .CPUPowerW}} · {{printf "%.0f" .CPUPowerW}} W{{end}}</div></div>{{end}}
|
||||
{{if .InletTempC}}<div class="tile"><div class="v">{{printf "%.1f" .InletTempC}} °C</div><div class="k">inlet air</div></div>{{end}}
|
||||
{{else}}
|
||||
<div class="tile"><div class="v">—</div><div class="k">CPU and RAM not measured lately</div></div>
|
||||
{{end}}
|
||||
@@ -29,13 +33,24 @@
|
||||
</div>
|
||||
{{if .Warnings}}{{range .Warnings}}<p class="note bad">{{.}}</p>{{end}}{{end}}
|
||||
{{if .Unhealthy}}<p class="note bad">{{join .Unhealthy}}</p>{{end}}
|
||||
{{with .Load}}{{if .GPUs}}
|
||||
<h2>GPUs</h2>
|
||||
<table class="rows"><tr><th class="num">GPU</th><th class="num">Temperature</th><th class="num">Memory</th><th>Power</th><th>Held back by</th></tr>
|
||||
{{range .GPUs}}<tr>
|
||||
<td class="num">{{.Index}}</td>
|
||||
<td class="num">{{if .TempC}}{{printf "%.0f" .TempC}} °C{{else}}—{{end}}</td>
|
||||
<td class="num">{{if .MemoryTempC}}{{printf "%.0f" .MemoryTempC}} °C{{else}}—{{end}}</td>
|
||||
<td class="load">{{if and .PowerW .PowerLimitW}}{{template "minibar" pctbar (percent .PowerW .PowerLimitW)}}<span class="load-v">{{printf "%.0f" .PowerW}} W <span class="note">of {{printf "%.0f" .PowerLimitW}}</span></span>{{else if .PowerW}}{{printf "%.0f" .PowerW}} W{{else}}—{{end}}</td>
|
||||
<td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}</td>
|
||||
</tr>{{end}}</table>
|
||||
{{end}}{{end}}
|
||||
{{end}}
|
||||
|
||||
{{if or .Node.IBPorts .Node.Fabrics}}
|
||||
<h2>Fabric</h2>
|
||||
<p class="sub">{{if .Node.Fabrics}}On {{join .Node.Fabrics}} · <a href="{{if $.OnAdminListener}}/fabrics{{else}}/admin/fabrics{{end}}">the map →</a>{{else}}<span class="bad">On no fabric: no port a declared fabric names can carry traffic, so no multi-node replica is placed here.</span>{{end}}</p>
|
||||
{{if .Node.IBPorts}}
|
||||
<table><tr><th>Adapter</th><th>State</th><th class="num">Rate</th><th>Fabric</th><th>Physically on</th><th>Errors since boot</th><th class="num">Sent</th><th class="num">Received</th></tr>
|
||||
<table class="rows"><tr><th>Adapter</th><th>State</th><th class="num">Rate</th><th>Fabric</th><th>Physically on</th><th>Errors since boot</th><th class="num">Sent</th><th class="num">Received</th></tr>
|
||||
{{range .Node.IBPorts}}<tr>
|
||||
<td><code>{{.HCA}}/{{.Port}}</code>{{if eq .LinkLayer "Ethernet"}} <span class="tag">RoCE</span>{{end}}</td>
|
||||
<td class="{{if .Active}}state-ready{{else}}bad{{end}}">{{.State}}{{with .PhysState}} <span class="note">{{.}}</span>{{end}}</td>
|
||||
@@ -60,13 +75,13 @@ the adapter came up: a number that grows is the one to look at.</p>
|
||||
{{if or .Node.NVLinkDomain .Node.IMEX}} · IMEX {{if eq .Node.IMEX "active"}}<span class="state-ready">active</span>, {{.Node.IMEXChannels}} channel{{if ne .Node.IMEXChannels 1}}s{{end}}{{else}}<span class="bad">{{or .Node.IMEX "not installed"}}</span>{{end}}{{end}}
|
||||
</p>
|
||||
<div class="gpu-wiring">
|
||||
<table class="nvl-matrix" aria-label="NVLink between GPUs">
|
||||
<table class="rows nvl-matrix" aria-label="NVLink between GPUs">
|
||||
<tr><th></th>{{range .Node.GPUs}}<th>{{.Index}}</th>{{end}}</tr>
|
||||
{{range $a := .Node.GPUs}}<tr><th>GPU {{$a.Index}}</th>
|
||||
{{range $b := $.Node.GPUs}}{{if eq $a.Index $b.Index}}<td class="self"></td>{{else if hasInt $a.NVLinkPeers $b.Index}}<td class="nv" title="GPU {{$a.Index}} ↔ GPU {{$b.Index}} over NVLink">NV</td>{{else}}<td class="pcie" title="GPU {{$a.Index}} ↔ GPU {{$b.Index}} over PCIe">·</td>{{end}}{{end}}
|
||||
</tr>{{end}}
|
||||
</table>
|
||||
<table class="compact gpu-list">
|
||||
<table class="rows compact gpu-list">
|
||||
<tr><th>GPU</th><th>Name</th><th class="num">NUMA</th><th>Nearest adapter</th><th>Multi-node NVLink</th></tr>
|
||||
{{range .Node.GPUs}}<tr>
|
||||
<td>{{.Index}}</td><td>{{.Name}}</td><td class="num">{{.NUMA}}</td>
|
||||
@@ -82,7 +97,7 @@ traffic between nodes should go: PIX and PXB are under the same PCIe switch, SYS
|
||||
|
||||
<h2>What runs here</h2>
|
||||
{{if .Engines}}
|
||||
<table><tr><th>Model</th><th>Replica</th><th>State</th><th>CPU</th><th class="num">Memory</th></tr>
|
||||
<table class="rows"><tr><th>Model</th><th>Replica</th><th>State</th><th>CPU</th><th class="num">Memory</th></tr>
|
||||
{{range .Engines}}<tr>
|
||||
<td>{{.Model}}</td>
|
||||
<td><code>{{.Replica}}</code></td>
|
||||
|
||||
@@ -42,7 +42,7 @@
|
||||
{{end}}
|
||||
|
||||
{{if .Rows}}
|
||||
<table class="compact">
|
||||
<table class="rows compact">
|
||||
<tr><th>{{if eq .By "user"}}Person{{else if eq .By "project"}}Project{{else}}Team{{end}}</th>
|
||||
<th class="num">Requests</th><th class="num">Tokens</th><th class="num">Spend</th><th class="num">Errors</th></tr>
|
||||
{{range .Rows}}<tr>
|
||||
@@ -56,7 +56,7 @@ people do elsewhere — in another organization, or on their own — is not show
|
||||
|
||||
<h2>Teams, projects and limits</h2>
|
||||
{{if .Quotas}}
|
||||
<table class="quota-tree">
|
||||
<table class="rows quota-tree">
|
||||
<tr><th>Where</th><th class="num">Requests/min</th><th class="num">Tokens/min</th>
|
||||
<th class="num">Budget</th><th>Spent · 30 days</th></tr>
|
||||
{{range .Quotas}}
|
||||
@@ -79,7 +79,7 @@ people do elsewhere — in another organization, or on their own — is not show
|
||||
|
||||
<h2>Members</h2>
|
||||
{{if .Members}}
|
||||
<table class="compact">
|
||||
<table class="rows compact">
|
||||
<tr><th>Person</th><th>Member of</th><th>State</th></tr>
|
||||
{{range .Members}}<tr>
|
||||
<td>{{.Email}}</td>
|
||||
|
||||
@@ -109,6 +109,16 @@ var funcs = template.FuncMap{
|
||||
return p
|
||||
},
|
||||
"ago": ago,
|
||||
// percent is part of whole, 0 to 100, for a bar.
|
||||
"percent": func(part, whole float64) float64 {
|
||||
if whole <= 0 {
|
||||
return 0
|
||||
}
|
||||
return part * 100 / whole
|
||||
},
|
||||
// shortNode is a node's name without what every node's name starts with.
|
||||
"shortNode": strings.TrimPrefix,
|
||||
"age": age,
|
||||
// pctbar is a percentage as the minibar template takes it: whole, and
|
||||
// held to 0..100 so a measurement a little over does not draw past its box.
|
||||
"pctbar": func(v float64) struct{ P int } {
|
||||
|
||||
+13
-3
@@ -1463,8 +1463,8 @@ The launch path is now built too:
|
||||
- [x] `ctl bench`: a load ramp that reports capacity as the last concurrency
|
||||
level meeting the SLO, plus the queue depth and KV usage observed there,
|
||||
so `weight`, `target_waiting` and `kv_ceiling` stop being guesses
|
||||
- [ ] `ctl plan <model> --target-rps N`, which needs capacity profiles stored
|
||||
rather than printed
|
||||
- [x] Sizing from stored capacity: `ctl model target <model> --tokens-per-sec N`,
|
||||
and the autoscaler and profile choice use what `ctl bench --save` measured
|
||||
- [x] OTel tracing: an inference span with admission, schedule and each
|
||||
upstream attempt under it, over OTLP/HTTP when `tracing.endpoint` is
|
||||
set, and the traceparent carried on to the engine
|
||||
@@ -1480,7 +1480,17 @@ The launch path is now built too:
|
||||
- [ ] Admission displacement (dropping requests from lower bands)
|
||||
- [x] Autoscaling on load: a replica added while requests wait, up to a
|
||||
model's max, and taken back after ten quiet minutes
|
||||
- [ ] Priority-based preemption
|
||||
- [x] Priority-based preemption: capacity classes (critical, standard,
|
||||
opportunistic), preemption tried on a copy of the inventory first
|
||||
- [x] Capacity quotas per organization, team and project: guarantees kept,
|
||||
borrowing taken back, limits enforced
|
||||
- [x] CPU cores reserved like memory, and memory required of CPU models
|
||||
- [x] Node sensors collected: GPU temperature, power and throttling, CPU
|
||||
temperature and RAPL power, board inlet air; published and shown
|
||||
- [ ] Rack power, temperature and cooling as placement signals: planned in
|
||||
[docs/facility-aware-placement.md](docs/facility-aware-placement.md),
|
||||
deferred — it needs each site's rack, PDU and power data first
|
||||
- [ ] Defragmentation for large tensor-parallel replicas across NVLink domains
|
||||
- [ ] Control plane HA (Raft)
|
||||
- [x] Weight downloading and cache management: the mirror, agent fetches
|
||||
checked against their digest, and reclaiming what nothing uses
|
||||
|
||||
Reference in new issue
Block a user