Compare commits

...
27 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 0774e901c0 Slurm: a new leader with nothing kept waits for the nodes a while
CI / test (push) Failing after 7m57s
On a cluster upgraded to the kept nodes, the first leader finds none and
wrote slurm.conf without the nodes it had not heard from yet, as before.
A node neither reported nor kept now holds the controller's files back
for slurmsvc.HoldFor (2 minutes) from the leader's first pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:37:52 +01:00
Ilker ManapandClaude Opus 5.5 141a1f46c5 Slurm: a new leader keeps the nodes it has not heard from yet
CI / test (push) Successful in 10m50s
Tried on the PoC: with a job on a lent node, the leader restarted and its
first slurm.conf had no node in it (none had reported to it yet), so
slurmctld dropped the node and the job ended in NODE_FAIL.

The leader keeps slurm.conf's nodes and the controller's address in the
store's cluster state (slurm-nodes); the next one starts from them and
writes the same files until the nodes report again. With the store
unread, the controller gets no files that leave a node out. A node is
told nothing, rather than that it has no part, until its controller's
address is known: no part stops slurmd, and its jobs with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:33:53 +01:00
Ilker ManapandClaude Opus 5.5 3387f97f6a Slurm's ports for the cluster only, and srun in a range of its own
CI / test (push) Successful in 9m40s
slurm.conf sets SrunPortRange=60001-63000, so a site's firewall can let
srun's step connections in. A node given a part in Slurm adds 6817-6818
and that range to llmgw's nftables table, open to the cluster's networks
and closed to everyone else, as the engines' ports are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:10:37 +01:00
Ilker ManapandClaude Opus 5.5 914985bf9f Slurm in the first install: server init --slurm, and slurm.all_nodes
CI / test (push) Successful in 9m30s
The choices screen has a Slurm part, and --slurm writes slurmctld on this
machine's node and slurmd on every node, those that join later too, into
server.yaml. slurm.all_nodes (ctl slurm setup --all-nodes) takes the nodes
from the leader's inventory. No node is lent at install.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 14:50:35 +01:00
Ilker ManapandClaude Opus 5.5 b15944aa1f A package install that is no engine's clears its progress; slurmd retried
CI / test (push) Successful in 10m15s
Found on the PoC: InstallPackages reported 'installing munge, slurm-client,
slurmd' under no engine and never cleared it, so the node's page kept
saying so (infiniband-diags and perftest did the same). slurmd started
before the leader has written its node into slurm.conf is refused; the
drop-in now has systemd try it again every 15 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:23:53 +01:00
Ilker ManapandClaude Opus 5.5 dcf4e256e6 BACKLOG P: tried on the PoC
CI / test (push) Failing after 7m26s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:20:32 +01:00
Ilker ManapandClaude Opus 5.5 849760c67d Slurm: no second restart; backfill every 10 s, MailProg set
The InvalidAccount seen on the PoC was not the reconfigure: 23.11.4 without
accounting storage has its main scheduler mark a batch job so, and backfill
starts it on its next pass (job 4 waited 15 s). bf_interval=10 keeps that
short; the extra restart is taken out again. MailProg=/bin/true quiets the
warning on every start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:18:35 +01:00
Ilker ManapandClaude Opus 5.5 087552136f Slurm: slurmctld started again after a reconfigure; no accounting gathered
Found on the PoC: Slurm 23.11.4 without accounting storage left batch jobs
pending InvalidAccount after scontrol reconfigure until slurmctld was
restarted. And jobacct_gather with nowhere to keep it logs an error per job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:16:31 +01:00
Ilker ManapandClaude Opus 5.5 d2c27e2c76 Slurm: slurmd started as its unit expects, and slurmd's own stepd not a job
Found on the PoC (Ubuntu 24.04, Slurm 23.11.4): the packaged unit is
Type=notify, so slurmd needs --systemd or systemd times it out; and slurmd
keeps a 'slurmstepd infinity' for its cgroup v2 scope, which was counted as
a job and kept the node from llmgw.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:11:41 +01:00
Ilker ManapandClaude Opus 5.5 8a5a04ed77 Slurm: the version from the daemon, not scontrol
Found on the PoC: scontrol looks for its files even for --version, and a
configless node has none before slurmd has fetched them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:06:42 +01:00
Ilker ManapandClaude Opus 5.5 410fe545ce Slurm beside llmgw, P3 and P4: the leader hands nodes over; ctl slurm
CI / test (push) Successful in 9m40s
internal/control/slurmsvc tells each node what it is, writes slurm.conf
and gres.conf from the nodes' reports and the inventory, and moves each node
of slurm.nodes between llmgw and Slurm: lent, drained of replicas first, then
resumed in Slurm; taken back, drained in Slurm and cordoned in llmgw until
its last job step ends. Kept every tick from the settings and the agents'
reports, so a new leader carries on. A broken Slurm never keeps a node from
llmgw. GET /api/v1/slurm; ctl slurm status, setup, lend, reclaim, off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:33:14 +01:00
Ilker ManapandClaude Opus 5.5 35de47444c Slurm beside llmgw, P2: the agent keeps its node's part
A Slurm message says whether the node runs slurmctld on the leader's files,
slurmd configless, both or neither, and whether it is lent. The agent
installs munge and Slurm from the machine's sources (22.05 or newer, for
cgroup v2), writes the munge key, starts slurmd through a drop-in with
--conf-server, resumes a lent node that llmgw drained and drains one that is
not, never touching a drain a Slurm administrator made, and reports its
version, slurmd -C, state and jobs in SlurmStatus. Told it has no part, it
stops what it started. New nodes enter slurm.conf drained.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:25:41 +01:00
Ilker ManapandClaude Opus 5.5 d439d507d7 Slurm beside llmgw, P1: its settings and the files a controller runs on
slurm.{enabled,cluster_name,controller,nodes,lent,partition} in the cluster's
settings, checked; internal/slurm writes slurm.conf (configless), gres.conf
and cgroup.conf from the nodes' slurmd -C lines and GPUs, derives the munge
key from a cluster secret, and reads scontrol and version output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:20:31 +01:00
Ilker ManapandClaude Opus 5.5 89767c8be1 BACKLOG P: training beyond one card, in llmgw or Slurm beside it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:12:07 +01:00
Ilker ManapandClaude Opus 5.5 d864438272 A model that failed to start is placed elsewhere next
CI / test (push) Successful in 9m31s
The same inputs place a replica on the same node and card, so an on-demand
model that failed to load there was started there again after every
backoff until it was parked. The reconciler remembers where a model's
replicas failed and places the next away from them when anything else
fits (Scheduler.Away); with nowhere else, there again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 09:22:11 +01:00
Ilker ManapandClaude Opus 5.5 6320dd5b13 A new stream is told the node's files again
CI / test (push) Successful in 9m31s
Registration carries what the agent's startup scan found, and a heartbeat
sends the list only when it changed since last sent; so a file deleted
after the agent started came back on a control plane that restarted. The
fingerprint is now forgotten with the provisioning one on every stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:14:52 +01:00
Ilker ManapandClaude Opus 5.5 5732a5ebda The serve forms start at the longest context that fits
CI / test (push) Successful in 9m26s
The mirror page's form began every GGUF model at 4096 tokens, the
console's at 4096 or the plan's figure. Both now start at what the model
was trained for, cut to the most whose KV cache fits beside the weights
where the form places it; the mirror page's slots are as many of that, up
to four, as the room left holds. Plan carries the trained context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 14:38:41 +01:00
Ilker ManapandClaude Opus 5.5 561a536477 The weight sweep names a set as its fetch did
CI / test (push) Successful in 9m33s
A set is fetched into a directory named from its address without the
trailing slash; Orphans named it from the catalog's address with it, so
every set in the catalog was offered for deletion: 330 GB of six models on
the PoC. One function, cacheSource, now gives both the name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 14:01:27 +01:00
Ilker ManapandClaude Opus 5.5 45fe7daaa0 The admin API document regenerated for CollectResult's left counts
CI / test (push) Successful in 9m32s
cba4897 added fields without regenerating it, and
TestTheAdminAPIDocumentIsNotStale failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 11:38:08 +01:00
Ilker ManapandClaude Opus 5.5 102ca9556f An organization's admin, and forms only for those who may send them
C1. A membership of an organization may be admin as well as member and
manager. An admin sees /org as a manager does and changes the
organization from it: its teams and projects, who of the approved
accounts is in them, its members' roles, and the keys spent in its
projects. Every id a form names is checked against the organization;
limits, accounts, the organization's name and the cluster stay with the
cluster's staff, and an admin cannot take their own role away.

A group from OIDC, SCIM or a directory no longer rewrites a membership
it names as a plain member at every sign-in, which took a manager's role
away (EnsureMembership).

Pages draw a form only for someone who may send it: Nav.May asks the
staff rule staffMay keeps, and a page whose forms a person may not send
gets no form token. A viewer sees the people, limits and mirror pages
without forms; an operator the mirror's.

Forms are compact: labels close to their fields, fields as wide as what
goes in them, smaller buttons, and the short forms on one line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 11:38:06 +01:00
Ilker ManapandClaude Opus 5.5 2cb1afda55 The introduction's comparison on one slide
CI / test (push) Failing after 6m38s
Four slides of products and cards become one table: llmgw beside API
gateways (LiteLLM, Kong AI Gateway, Agent Router) and serving stacks
(llm-d, NVIDIA Dynamo, vLLM production-stack), six rows. The PDF is
18 slides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 17:06:49 +01:00
Ilker ManapandClaude Opus 5.5 b6478e24b7 An introduction for decision makers, linked from the website
CI / test (push) Failing after 6m43s
docs/llmgw-intro.pdf is a 21-slide presentation: what llmgw adds on
premise, screens from a running cluster, and how it compares with
LiteLLM, Kong AI Gateway, Agent Router, llm-d, NVIDIA Dynamo and vLLM
production-stack from their own documentation. The /about page and the
published site link to it in the repository, so the binary does not
carry it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 14:36:47 +01:00
Ilker ManapandClaude Opus 5.5 cba4897987 Deleting unused model files says what went, and sets go whole
CI / test (push) Failing after 6m44s
A sweep counted every file the nodes were asked to delete as freed as soon
as it asked, and a node refused a set's directory as not empty, so tens of
gigabytes were reported freed and nothing was. The agent removes a set's
directory whole (a link only as a link), and the control plane waits for
each node to list its files again, saying what it no longer lists and what
it kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:29:16 +01:00
Ilker ManapandClaude Opus 5.5 3bb52742a4 v0.16.3: changelog and README
Release / build (push) Successful in 5m11s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 10m10s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:43:40 +01:00
Ilker ManapandClaude Opus 5.5 1a3f385be4 Every setting of a served model from the mirror page, and its replicas per profile
CI / test (push) Successful in 10m51s
The row of a served model leads to its card, which now changes its context
and longest answer, its cards, its replicas on each profile with a profile
on CPUs to add and its speed beside, stops, starts and restarts it, and
removes it with its files or without. Stopped on the row stops the model
rather than setting its scaling to nothing. The console gains Replicas per
profile, and ctl models shows the counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:40:06 +01:00
Ilker ManapandClaude Opus 5.5 e23732fbbb Replicas counted per profile: seven on cards and four on CPUs under one name
A profile's replicas, set on any of a model's profiles, is how many of its
replicas run there; the scaling becomes the sum. The reconciler starts the
ones short of their profile before stopping the ones over, so a change of
split never runs fewer. ctl model replicas sets them, adding a profile on
CPUs to a GGUF model; the router already fills the higher-priority pool
first, which the CPU profile is put below.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:21:50 +01:00
Ilker ManapandClaude Opus 5.5 6403e61f4b How fast a model answers on the CPU, before it is served
CI / test (push) Successful in 9m31s
Each agent measures one core's memory rate beside the whole machine's; the
estimate takes a replica's threads at about one core's rate each, up to the
machine's, calibrated by llama-bench on the PoC. The serve form shows it
beside a card's, warns when slow, and asks for the CPU on purpose when a
reply would take minutes; ctl model plan, the console and serve jobs say
it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 08:52:15 +01:00
129 changed files with 7852 additions and 431 deletions

No files matched your search

+75 -4
View File
@@ -49,7 +49,8 @@ Face, done 2026-10-30), then M5 (engines without the internet), M3, M4, M6.
**Scale.** O: more than one control plane answering clients; O1 (measuring one router's ceiling) first.
**Governance.**
7. **C1**: a tenant-admin role, and forms hidden from viewers.
7. ~~**C1**~~: done 2026-11-08. An organization's admin (`--role admin` on an org membership) changes its teams,
projects, people and keys from /org; viewers are drawn no forms; forms made compact.
8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
**When there is room.**
@@ -435,9 +436,9 @@ portal is not done — the page shows; the console and ctl change.
API middleware and the /admin pages (with each change's result), `ctl
audit`, the console's Audit view; staff roles viewer / operator /
billing / admin. Left: before/after values (the request's fields are
recorded, not the state they replaced); a tenant-admin role that changes
its own organization (managers still only view it); hiding the forms a
viewer cannot submit. **Audit log and roles (M).** Today "audit" is slog lines
recorded, not the state they replaced). Done 2026-11-08: an organization's
admin who changes its own organization from /org (managers still only view
it), and the forms a viewer cannot submit no longer drawn. **Audit log and roles (M).** Today "audit" is slog lines
(`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag
plus an org "manager" who can view. Add `audit_events` (who, what,
before/after, source) written by the `adminAction`/`tenancyAction`
@@ -1982,6 +1983,76 @@ one control plane's router can. In order:
- O3. Only if O2 is not enough: stateless gateway machines (`llmgw server --role gateway`) in front of the leader's
control plane, as many as the traffic needs.
## P. Training beyond one card (written 2026-11-11) — decision open
F4 trains a LoRA on one card of a node the caller names (`ctl train start --node`), one job per node, no resume.
Larger work (full fine-tuning, a LoRA on a 70B, several GPUs or machines) needs more. Two ways:
**A. In llmgw.** Add, in order: a job that takes N whole GPUs on one node (torchrun, FSDP/DeepSpeed in the agent's
venv); placement by the control plane instead of `--node`, freeing GPUs from replicas by the scaling schedule or by
draining; checkpoints to shared storage and resume when an agent or node goes; datasets uploaded or read from
storage, not only KB pairs; then gang placement of N nodes with a rendezvous on the first, the NCCL environment from
the fabric discovery (IB/NVLink, `llmgw doctor`); a console screen with logs and loss. Size: L for one node, XL for
several. For: one product, one install, one set of users and grants (KB rules carry over), GPUs shared with
inference without a second scheduler. Against: it is Slurm's gang scheduling, requeue and accounting written again;
multi-node NCCL cannot be tried on the PoC (one GPU node, no IB); every customer's training code would have to fit
our job format.
**B. Slurm beside llmgw.** The TUI installs Slurm on the nodes the customer picks (munge, slurmctld on the control
planes, slurmd, users from the same LDAP/AD), with pyxis/enroot for containers. A node is handed between the two:
llmgw drains its replicas and `scontrol update state=resume`, and back. llmgw's part is what comes out: a checkpoint
or adapter from a Slurm job published to the mirror (`ctl train import`), served through canary and the eval. Size:
M. For: what research and university customers already run (`sbatch`, `srun torchrun`), any training code, mature
multi-node and requeue; we write only the install and the handover. Against: a second system with its own failures,
upgrades and users to keep in step; GPUs split by hand or by the handover, not by demand; access rules on training
data are Slurm's and the file system's, not the KB's.
Both need shared storage (NFS, BeeGFS, Lustre) the customer provides; neither changes F4.
Recommendation: keep F4 in llmgw for small single-card adapters tied to KB grants; take B for anything larger;
build only A's first step (N GPUs on one node, placed by the control plane, with resume) if customers ask for
mid-size jobs without wanting Slurm.
**B taken (2026-11-11).** Design: the cluster's settings say it (`slurm: enabled, cluster_name, controller, nodes,
lent, partition`); the leader tells each named node's agent (root) what to be, the agents install and run it.
- The controller is an llmgw node (its agent runs slurmctld); `nodes` run slurmd configless
(`--conf-server`), so slurm.conf, gres.conf and cgroup.conf live only on the controller.
- The munge key is derived from the cluster secret (HKDF), so nothing new is stored or copied by hand.
- Each node reports `slurmd -C`; the leader writes slurm.conf from those lines, its mgmt address and its GPUs
(`Gres=gpu:<type>:N`, gres.conf with the device files), and sends it to the controller again when one changes.
- A node in `nodes` belongs to llmgw until it is in `lent`: drained in Slurm (reason "llmgw: serving models").
Lent: llmgw drains it as `ctl drain` does (kept in force each tick, as facility holds are, so a new leader
keeps it) and, once no replica is left, its agent resumes it in Slurm. Taken out of `lent`: the agent drains
it in Slurm, reports the jobs left, and when none are the leader makes it ready for llmgw again.
- Phases: P1 the `slurm` package (conf, key, parsers) and settings; P2 the agent side (proto, install, munge,
slurmd/slurmctld, status); P3 the leader side (conf from reports, lend/reclaim); P4 `ctl slurm`
(status, lend, reclaim) and a TUI screen; P5 `ctl train import` of a Slurm job's output into the mirror.
Later: a backup slurmctld on a second node, accounting (slurmdbd on the Patroni PG), users from LDAP/AD.
- Tried on the PoC 2026-11-11 (dev build v0.16.3-17-g849760c-slurm on 140, 150, node05, node06; Ubuntu
24.04, Slurm 23.11.4 from the archive): `ctl slurm setup --controller llmgw-node06 --nodes llmgw-node05`
installed munge, slurmctld and slurmd, node05 entered drained with its slurmd -C hardware (8 CPUs,
31132 MiB after the 3%). `lend`: its 2 replicas moved off, then IDLE in Slurm; srun and sbatch ran there.
`reclaim` with a 90 s job running: MIXED+DRAIN, llmgw kept off, ready again within a minute of the job
ending. Found and fixed on the way: a client command needs its files even for --version (version now
from the daemon); the packaged unit is Type=notify (slurmd needs --systemd); slurmd's own "slurmstepd
infinity" was counted as a job; without accounting storage 23.11's main scheduler marks batch jobs
InvalidAccount and backfill starts them, one a pass (bf_interval=10: 6-10 s each). So many small jobs
want slurmdbd. Not tried yet: a GPU node (gres), a leader failover while lent.
- Built 2026-11-11: P1 (`internal/slurm`, `slurm.*` settings), P2
(`internal/agent/slurmnode`, proto Slurm/SlurmStatus), P3 (`internal/control/slurmsvc`, wired in the
leader; `GET /api/v1/slurm`), P4's commands (`ctl slurm status|setup|lend|reclaim|off`). Decisions on the
way: a node not heard from or with Slurm jobs is cordoned, not drained (nothing of llmgw's to move, and a
new leader must not move models off every Slurm node); jobs are counted from the node's own slurmstepd,
so a controller that is down neither hides them nor, with none, keeps the node from llmgw; a drain whose
reason does not start with "llmgw" is a Slurm administrator's and never resumed. Left: the TUI screen, P5,
a try on the PoC (needs agents built from this; slurm-wlm 23.11 on Ubuntu 24.04), GPU minor numbers are
taken to be nvidia-smi's index.
- In the first install too (2026-11-11): `llmgw server init --slurm`, and a Slurm part on the choices
screen, write `slurm: {enabled, controller: <this hostname>, all_nodes: true}` into server.yaml.
`slurm.all_nodes` (also `ctl slurm setup --all-nodes`) runs slurmd on every node in the leader's
inventory, those that join later too; none is lent. Refused with `--no-node`, or a hostname Slurm does
not take.
## D. Facts worth knowing before starting
- Admin API: `127.0.0.1:9090`, behind the token in
+136 -1
View File
@@ -20,12 +20,147 @@ is only released once its **exit criterion** is met, not when its code lands.
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node) |
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node; v0.16.3: replicas per profile, a served model's settings on the mirror page, the CPU's speed before serving) |
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased
### Added
- **An organization's admin.** Besides member and manager, a membership of
an organization may now be `admin` (`llmgw users join … --org Acme --role
admin`, the role chosen on a membership on the people page, the console,
the admin API). An admin sees the organization's page as a manager does and
also changes the organization from it: adds, renames and removes its teams
and projects, adds people with an approved account by their address and
takes them out, makes its members managers or admins, and revokes keys spent
in its projects. Limits, approving and suspending accounts, the
organization's name and the rest of the cluster stay with the cluster's
administrators; an admin cannot take away their own role. Every change is in
the audit log.
- **Slurm beside llmgw, for training and batch work.** `llmgw ctl slurm setup
--controller NODE --nodes NODE,...` (or `--all-nodes`, or `server init
--slurm`) has the agents install munge and Slurm from the machines' package
sources: slurmctld on one node, slurmd configless on the others, slurm.conf
and gres.conf written from what each node reports. Every node stays llmgw's,
drained in Slurm, until `ctl slurm lend NODE`: its replicas move off, then
Slurm runs jobs there. `ctl slurm reclaim NODE` takes it back once its jobs
end. `ctl slurm status` shows each node. Slurm's ports (6817–6818, srun's
60001–63000) answer the cluster's networks only, as llmgw's own do.
### Changed
- **Forms only for those who may send them.** A viewer sees the people,
limits, mirror and cluster pages without their forms, an operator the
mirror's and a node's engines but not the rest; before, they were drawn and
answered "not found".
- **Smaller forms.** A label sits close to its field, a field is as wide as
what goes in it rather than the page, buttons are smaller, and the short
forms — a new key, a password, connecting an editor, a capacity or a limit —
sit on one line or two instead of a screen.
### Fixed
- **The serve forms start at the longest context that fits.** The mirror
page's form began every GGUF model at 4096 tokens, and the console's at
4096 or the plan's figure, whatever the model was trained for. Both now
start at what it was trained for, as long as its KV cache fits where the
form places it beside the weights, else at the most that does, in whole
thousands; the mirror page's llama.cpp slots are as many of that, up to
four, as the room left holds, and its memory figure is for that pool. A
plan says what the model was trained for (`trained`). Strata stays at
its 32768; vLLM's form is unchanged.
- **A model that failed to start was started in the same place again.**
Placement given the same inputs chooses the same node and card, so an
on-demand model whose replica failed to load was retried there, past each
backoff, until it was parked, with another card or node free all along.
The control plane now remembers where a model's replicas failed (node and
cards, up to eight places) and places the next one elsewhere when anywhere
else fits; with nowhere else it is tried there as before. Coming up, a
change to its definition or `ctl retry` forgets the places. The log says
`placing a model away from where it failed to start`, with both places.
- **Deleted model files came back after a control plane restart.** An agent
registers with the files its startup scan found, and sends the list again
only when it changes; a file deleted after the agent started was listed
again by a control plane that restarted, until the next change. A new
stream now gets the list on its first heartbeat.
- **`ctl weights` offered models in use for deletion.** A model of several
files (a set, its address ending in `/`) is fetched into a directory named
from the address without the slash; the sweep named it from the address
with it, so every set in the catalog looked like nobody's. On the PoC that
was 330 GB of six catalog models, the qwen3.8-flash-next Q8_0 among them;
`--collect`, and `ctl model remove --purge` of a set, would have deleted
them. Both now compute the name the fetch does.
- **A sign-in took an organization's manager role away.** A group from single
sign-on, SCIM or Active Directory that puts someone in an organization wrote
the membership again at every sign-in or sync, as a plain member: a manager
made one on the people page was one only until they next signed in. A
membership the directory names is now added if missing and otherwise left
as it is.
- **The limits form on the people page's panel** was laid out on one line, its
title squeezed into a column.
- **Deleting the model files nothing uses said what it was asked to free,
not what it freed.** The console's "Delete model files nothing uses" and
`ctl weights --collect` added up every file the nodes were asked to
delete as soon as the request was sent, and a model of several files —
fetched into a directory of its own — was refused by the node as "not
empty", so a sweep said tens of gigabytes freed and nothing went. The
node now deletes such a directory whole (a link only as a link), and the
sweep waits for the nodes to list their files again: freed is what they
no longer list, and what they kept is said beside, with where to read
why. Removing a model with its files reports the same way.
## v0.16.3 — 2026-11-03
Replicas counted per profile — seven on cards and four on CPUs under one name, the cards filled first — and every
setting of a served model from the mirror page; how fast a model answers on the CPU, said before it is served, by
the threads it is given rather than the whole machine's memory.
**Upgrading.**
- No migrations. `llmgw ctl upgrade v0.16.3`: the control planes carry the replicas per profile, the model card and
the speed in the serve form and in `ctl model plan`; the agents carry the measurement of one core's memory rate,
without which the speed is reckoned the old way, from the whole machine's.
### Added
- **Replicas counted per profile: seven on cards and four on CPUs, under one
name.** `llmgw ctl model replicas <model> gpu=7 cpu=4`, the console's
Replicas per profile… and the mirror page's model card set how many of a
model's replicas run on each of its profiles, rather than every replica
on the best profile that fits. For a GGUF model with no profile on CPUs
one is added (`--cpu-memory`), preferred below the cards, so the router
fills the cards first and sends to the CPUs once they are full. A change
starts the replicas short before stopping those over: the model never
runs fewer. `ctl models` shows the counts; a scaling of another total
ends them.
- **Every setting of a served model from the mirror page.** A served
model's row has a Settings link, and its card now changes the context
and longest answer, the cards a replica takes, the replicas on each
profile (with the CPU's speed beside), stops and starts it, restarts its
replicas, and takes it out of the catalog with its files or without —
beside the replicas, class, owner, memory and adapters it held before.
Stopped on the row is now the model stopped and kept, started again as
it was, rather than its scaling set to nothing.
- **How fast a model answers on the CPU, before it is served.** The serve
form says, as the cores are changed, about how many tokens a second one
reply gets on the CPU and on one card of the fleet's fastest class, and
warns when it is slow (under 15) or very slow (under 5, a reply of 500
tokens taking minutes). Very slow on the CPU is refused until "run on the
CPU anyway" is ticked, GPUs free or not. `llmgw ctl model plan` and the
console's Hugging Face screen show the same figure for every way a model
can run, and `ctl model serve --cpu` says it in its log.
- **The speed follows the threads a replica is given.** Each agent now
measures what one core reads beside what all of them read: a replica of
four threads on a large server reads at about four cores' rate, not the
server's, which the estimate used to assume and overstated several times.
Calibrated on the PoC by llama-bench at 1, 2, 4 and 8 threads (within 15%),
and a model of experts reads its experts' share, from the GGUF header.
Agents need upgrading to report one core's rate; until then the old
estimate is used.
## v0.16.2 — 2026-11-02
Standby control planes that answer clients themselves, with rate limits and budgets kept across them; FreeToken, an
+1 -1
View File
@@ -42,7 +42,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node) |
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node; v0.16.3: replicas per profile, a served model's settings on the mirror page, the CPU's speed before serving) |
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
+395 -81
View File
@@ -108,6 +108,7 @@ type AgentMessage struct {
// *AgentMessage_KvEvents
// *AgentMessage_TrainStatus
// *AgentMessage_TrainArtifact
// *AgentMessage_SlurmStatus
Msg isAgentMessage_Msg `protobuf_oneof:"msg"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
@@ -240,6 +241,15 @@ func (x *AgentMessage) GetTrainArtifact() *TrainArtifact {
return nil
}
func (x *AgentMessage) GetSlurmStatus() *SlurmStatus {
if x != nil {
if x, ok := x.Msg.(*AgentMessage_SlurmStatus); ok {
return x.SlurmStatus
}
}
return nil
}
type isAgentMessage_Msg interface {
isAgentMessage_Msg()
}
@@ -284,6 +294,10 @@ type AgentMessage_TrainArtifact struct {
TrainArtifact *TrainArtifact `protobuf:"bytes,10,opt,name=train_artifact,json=trainArtifact,proto3,oneof"`
}
type AgentMessage_SlurmStatus struct {
SlurmStatus *SlurmStatus `protobuf:"bytes,11,opt,name=slurm_status,json=slurmStatus,proto3,oneof"`
}
func (*AgentMessage_Register) isAgentMessage_Msg() {}
func (*AgentMessage_Heartbeat) isAgentMessage_Msg() {}
@@ -304,6 +318,8 @@ func (*AgentMessage_TrainStatus) isAgentMessage_Msg() {}
func (*AgentMessage_TrainArtifact) isAgentMessage_Msg() {}
func (*AgentMessage_SlurmStatus) isAgentMessage_Msg() {}
// KVEvents is a change to what one vLLM replica's prefix cache holds, read
// by the agent from the engine's KV cache events and sent on for the router's
// precise index (plan §7.5 tier 2).
@@ -426,6 +442,7 @@ type ControlMessage struct {
// *ControlMessage_AgentSettings
// *ControlMessage_TrainStart
// *ControlMessage_TrainCancel
// *ControlMessage_Slurm
Msg isControlMessage_Msg `protobuf_oneof:"msg"`
// trace is the W3C trace context (traceparent, and tracestate when there
// is one) of the control plane's span that sent this command, so what the
@@ -618,6 +635,15 @@ func (x *ControlMessage) GetTrainCancel() *TrainCancel {
return nil
}
func (x *ControlMessage) GetSlurm() *Slurm {
if x != nil {
if x, ok := x.Msg.(*ControlMessage_Slurm); ok {
return x.Slurm
}
}
return nil
}
func (x *ControlMessage) GetTrace() map[string]string {
if x != nil {
return x.Trace
@@ -693,6 +719,10 @@ type ControlMessage_TrainCancel struct {
TrainCancel *TrainCancel `protobuf:"bytes,17,opt,name=train_cancel,json=trainCancel,proto3,oneof"`
}
type ControlMessage_Slurm struct {
Slurm *Slurm `protobuf:"bytes,18,opt,name=slurm,proto3,oneof"`
}
func (*ControlMessage_Start) isControlMessage_Msg() {}
func (*ControlMessage_Stop) isControlMessage_Msg() {}
@@ -725,6 +755,8 @@ func (*ControlMessage_TrainStart) isControlMessage_Msg() {}
func (*ControlMessage_TrainCancel) isControlMessage_Msg() {}
func (*ControlMessage_Slurm) isControlMessage_Msg() {}
// RestartAgent has the agent start again, as llmgw restart asks of every
// node: its engines keep serving and are taken over again.
type RestartAgent struct {
@@ -1210,8 +1242,12 @@ type Host struct {
// mem_bandwidth_gbs is how fast the memory reads, in GB/s, measured when
// the agent started: what decides a model's speed on the CPU. 0 unknown.
MemBandwidthGbs float64 `protobuf:"fixed64,7,opt,name=mem_bandwidth_gbs,json=memBandwidthGbs,proto3" json:"mem_bandwidth_gbs,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
// mem_bandwidth_core_gbs is what one core reads alone, in GB/s: a replica
// given a few threads of a large machine reads at a few times this, not at
// the whole machine's rate. 0 unknown.
MemBandwidthCoreGbs float64 `protobuf:"fixed64,8,opt,name=mem_bandwidth_core_gbs,json=memBandwidthCoreGbs,proto3" json:"mem_bandwidth_core_gbs,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *Host) Reset() {
@@ -1293,6 +1329,13 @@ func (x *Host) GetMemBandwidthGbs() float64 {
return 0
}
func (x *Host) GetMemBandwidthCoreGbs() float64 {
if x != nil {
return x.MemBandwidthCoreGbs
}
return 0
}
// Storage is one filesystem a node keeps weights on.
//
// Per filesystem rather than per configured directory: free space belongs to
@@ -3049,8 +3092,10 @@ type Heartbeat struct {
// the agent has measured it — in the background after it starts, so a
// registration usually has none yet. 0 means not measured.
MemBandwidthGbs float64 `protobuf:"fixed64,20,opt,name=mem_bandwidth_gbs,json=memBandwidthGbs,proto3" json:"mem_bandwidth_gbs,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
// mem_bandwidth_core_gbs is one core's read rate, as in Host.
MemBandwidthCoreGbs float64 `protobuf:"fixed64,21,opt,name=mem_bandwidth_core_gbs,json=memBandwidthCoreGbs,proto3" json:"mem_bandwidth_core_gbs,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *Heartbeat) Reset() {
@@ -3223,6 +3268,13 @@ func (x *Heartbeat) GetMemBandwidthGbs() float64 {
return 0
}
func (x *Heartbeat) GetMemBandwidthCoreGbs() float64 {
if x != nil {
return x.MemBandwidthCoreGbs
}
return 0
}
// Provision is the software a node should have: which engines, at which
// versions. Nothing more.
//
@@ -5231,11 +5283,236 @@ func (x *TrainArtifact) GetLast() bool {
return false
}
// Slurm is what this node is in the Slurm cluster run beside llmgw
// (BACKLOG P): its controller, one of its nodes, both or neither. Sent when
// an agent registers and whenever it changes; the agent installs what is
// missing and keeps the daemons running as it says.
type Slurm struct {
state protoimpl.MessageState `protogen:"open.v1"`
// controller runs slurmctld here, on files.
Controller bool `protobuf:"varint,1,opt,name=controller,proto3" json:"controller,omitempty"`
// node runs slurmd here, configless, fetching its files from conf_server.
Node bool `protobuf:"varint,2,opt,name=node,proto3" json:"node,omitempty"`
// name is this node's NodeName in slurm.conf.
Name string `protobuf:"bytes,3,opt,name=name,proto3" json:"name,omitempty"`
// conf_server is slurmctld's host:port.
ConfServer string `protobuf:"bytes,4,opt,name=conf_server,json=confServer,proto3" json:"conf_server,omitempty"`
// munge_key is the key every machine of the cluster shares.
MungeKey []byte `protobuf:"bytes,5,opt,name=munge_key,json=mungeKey,proto3" json:"munge_key,omitempty"`
// files are the controller's, by name in /etc/slurm; empty for a node
// that is not the controller.
Files map[string]string `protobuf:"bytes,6,rep,name=files,proto3" json:"files,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
// lent says Slurm may run jobs here now: resume the node if llmgw drained
// it. Otherwise it is drained, its jobs left to finish.
Lent bool `protobuf:"varint,7,opt,name=lent,proto3" json:"lent,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *Slurm) Reset() {
*x = Slurm{}
mi := &file_llmgw_v1_agent_proto_msgTypes[48]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *Slurm) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*Slurm) ProtoMessage() {}
func (x *Slurm) ProtoReflect() protoreflect.Message {
mi := &file_llmgw_v1_agent_proto_msgTypes[48]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use Slurm.ProtoReflect.Descriptor instead.
func (*Slurm) Descriptor() ([]byte, []int) {
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{48}
}
func (x *Slurm) GetController() bool {
if x != nil {
return x.Controller
}
return false
}
func (x *Slurm) GetNode() bool {
if x != nil {
return x.Node
}
return false
}
func (x *Slurm) GetName() string {
if x != nil {
return x.Name
}
return ""
}
func (x *Slurm) GetConfServer() string {
if x != nil {
return x.ConfServer
}
return ""
}
func (x *Slurm) GetMungeKey() []byte {
if x != nil {
return x.MungeKey
}
return nil
}
func (x *Slurm) GetFiles() map[string]string {
if x != nil {
return x.Files
}
return nil
}
func (x *Slurm) GetLent() bool {
if x != nil {
return x.Lent
}
return false
}
// SlurmStatus is what this node's Slurm daemons are doing, sent when it
// changes and every few minutes while the node has a part in Slurm.
type SlurmStatus struct {
state protoimpl.MessageState `protogen:"open.v1"`
// version is Slurm's, as "23.11.4"; empty when it is not installed.
Version string `protobuf:"bytes,1,opt,name=version,proto3" json:"version,omitempty"`
// hardware is `slurmd -C` on a node, which slurm.conf must match.
Hardware string `protobuf:"bytes,2,opt,name=hardware,proto3" json:"hardware,omitempty"`
SlurmdRunning bool `protobuf:"varint,3,opt,name=slurmd_running,json=slurmdRunning,proto3" json:"slurmd_running,omitempty"`
SlurmctldRunning bool `protobuf:"varint,4,opt,name=slurmctld_running,json=slurmctldRunning,proto3" json:"slurmctld_running,omitempty"`
// state and reason are the controller's for this node, from scontrol.
State string `protobuf:"bytes,5,opt,name=state,proto3" json:"state,omitempty"`
Reason string `protobuf:"bytes,6,opt,name=reason,proto3" json:"reason,omitempty"`
// jobs is how many job steps run on this node now, counted from their
// slurmstepd so a controller that is down does not hide them; -1 when the
// agent cannot tell.
Jobs int32 `protobuf:"varint,7,opt,name=jobs,proto3" json:"jobs,omitempty"`
// error is the last thing that went wrong keeping Slurm, empty when
// nothing did.
Error string `protobuf:"bytes,8,opt,name=error,proto3" json:"error,omitempty"`
// files_sha256 is of the controller's files as written, so the leader
// knows they arrived.
FilesSha256 string `protobuf:"bytes,9,opt,name=files_sha256,json=filesSha256,proto3" json:"files_sha256,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *SlurmStatus) Reset() {
*x = SlurmStatus{}
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *SlurmStatus) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*SlurmStatus) ProtoMessage() {}
func (x *SlurmStatus) ProtoReflect() protoreflect.Message {
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use SlurmStatus.ProtoReflect.Descriptor instead.
func (*SlurmStatus) Descriptor() ([]byte, []int) {
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{49}
}
func (x *SlurmStatus) GetVersion() string {
if x != nil {
return x.Version
}
return ""
}
func (x *SlurmStatus) GetHardware() string {
if x != nil {
return x.Hardware
}
return ""
}
func (x *SlurmStatus) GetSlurmdRunning() bool {
if x != nil {
return x.SlurmdRunning
}
return false
}
func (x *SlurmStatus) GetSlurmctldRunning() bool {
if x != nil {
return x.SlurmctldRunning
}
return false
}
func (x *SlurmStatus) GetState() string {
if x != nil {
return x.State
}
return ""
}
func (x *SlurmStatus) GetReason() string {
if x != nil {
return x.Reason
}
return ""
}
func (x *SlurmStatus) GetJobs() int32 {
if x != nil {
return x.Jobs
}
return 0
}
func (x *SlurmStatus) GetError() string {
if x != nil {
return x.Error
}
return ""
}
func (x *SlurmStatus) GetFilesSha256() string {
if x != nil {
return x.FilesSha256
}
return ""
}
var File_llmgw_v1_agent_proto protoreflect.FileDescriptor
const file_llmgw_v1_agent_proto_rawDesc = "" +
"\n" +
"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xb7\x04\n" +
"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xf3\x04\n" +
"\fAgentMessage\x120\n" +
"\bregister\x18\x01 \x01(\v2\x12.llmgw.v1.RegisterH\x00R\bregister\x123\n" +
"\theartbeat\x18\x02 \x01(\v2\x13.llmgw.v1.HeartbeatH\x00R\theartbeat\x12-\n" +
@@ -5249,7 +5526,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\tkv_events\x18\b \x01(\v2\x12.llmgw.v1.KVEventsH\x00R\bkvEvents\x12:\n" +
"\ftrain_status\x18\t \x01(\v2\x15.llmgw.v1.TrainStatusH\x00R\vtrainStatus\x12@\n" +
"\x0etrain_artifact\x18\n" +
" \x01(\v2\x17.llmgw.v1.TrainArtifactH\x00R\rtrainArtifactB\x05\n" +
" \x01(\v2\x17.llmgw.v1.TrainArtifactH\x00R\rtrainArtifact\x12:\n" +
"\fslurm_status\x18\v \x01(\v2\x15.llmgw.v1.SlurmStatusH\x00R\vslurmStatusB\x05\n" +
"\x03msg\"\xb5\x01\n" +
"\bKVEvents\x12\x1d\n" +
"\n" +
@@ -5260,7 +5538,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"clearFirst\x12\x16\n" +
"\x06stored\x18\x04 \x03(\x06R\x06stored\x12\x18\n" +
"\aremoved\x18\x05 \x03(\x06R\aremoved\x12\x18\n" +
"\aproblem\x18\x06 \x01(\tR\aproblem\"\xf8\a\n" +
"\aproblem\x18\x06 \x01(\tR\aproblem\"\xa1\b\n" +
"\x0eControlMessage\x12,\n" +
"\x05start\x18\x01 \x01(\v2\x14.llmgw.v1.StartShardH\x00R\x05start\x12)\n" +
"\x04stop\x18\x02 \x01(\v2\x13.llmgw.v1.StopShardH\x00R\x04stop\x12,\n" +
@@ -5281,7 +5559,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\x0eagent_settings\x18\x0f \x01(\v2\x17.llmgw.v1.AgentSettingsH\x00R\ragentSettings\x127\n" +
"\vtrain_start\x18\x10 \x01(\v2\x14.llmgw.v1.TrainStartH\x00R\n" +
"trainStart\x12:\n" +
"\ftrain_cancel\x18\x11 \x01(\v2\x15.llmgw.v1.TrainCancelH\x00R\vtrainCancel\x129\n" +
"\ftrain_cancel\x18\x11 \x01(\v2\x15.llmgw.v1.TrainCancelH\x00R\vtrainCancel\x12'\n" +
"\x05slurm\x18\x12 \x01(\v2\x0f.llmgw.v1.SlurmH\x00R\x05slurm\x129\n" +
"\x05trace\x18\f \x03(\v2#.llmgw.v1.ControlMessage.TraceEntryR\x05trace\x1a8\n" +
"\n" +
"TraceEntry\x12\x10\n" +
@@ -5324,7 +5603,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\x1aA\n" +
"\x13EngineVersionsEntry\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\xdd\x01\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\x92\x02\n" +
"\x04Host\x12\x13\n" +
"\x05os_id\x18\x01 \x01(\tR\x04osId\x12\x1d\n" +
"\n" +
@@ -5334,7 +5613,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\x04arch\x18\x04 \x01(\tR\x04arch\x12\x1b\n" +
"\tcpu_flags\x18\x05 \x03(\tR\bcpuFlags\x12#\n" +
"\rnvidia_driver\x18\x06 \x01(\tR\fnvidiaDriver\x12*\n" +
"\x11mem_bandwidth_gbs\x18\a \x01(\x01R\x0fmemBandwidthGbs\"u\n" +
"\x11mem_bandwidth_gbs\x18\a \x01(\x01R\x0fmemBandwidthGbs\x123\n" +
"\x16mem_bandwidth_core_gbs\x18\b \x01(\x01R\x13memBandwidthCoreGbs\"u\n" +
"\aStorage\x12\x12\n" +
"\x04path\x18\x01 \x01(\tR\x04path\x12\x1d\n" +
"\n" +
@@ -5520,7 +5800,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\x06status\x18\x03 \x01(\tR\x06status\x12!\n" +
"\fcluster_uuid\x18\x04 \x01(\tR\vclusterUuid\x12\x1b\n" +
"\tclique_id\x18\x05 \x01(\tR\bcliqueId\x12\x1c\n" +
"\tbandwidth\x18\x06 \x01(\tR\tbandwidth\"\xd5\a\n" +
"\tbandwidth\x18\x06 \x01(\tR\tbandwidth\"\x8a\b\n" +
"\tHeartbeat\x12\x17\n" +
"\anode_id\x18\x01 \x01(\tR\x06nodeId\x12+\n" +
"\x12gpu_memory_used_mb\x18\x02 \x03(\x05R\x0fgpuMemoryUsedMb\x12!\n" +
@@ -5543,7 +5823,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\x04imex\x18\x11 \x01(\v2\x0e.llmgw.v1.IMEXR\x04imex\x12\x1b\n" +
"\tcpu_count\x18\x12 \x01(\x05R\bcpuCount\x121\n" +
"\x15gpu_foreign_memory_mb\x18\x13 \x03(\x05R\x12gpuForeignMemoryMb\x12*\n" +
"\x11mem_bandwidth_gbs\x18\x14 \x01(\x01R\x0fmemBandwidthGbs\x1aA\n" +
"\x11mem_bandwidth_gbs\x18\x14 \x01(\x01R\x0fmemBandwidthGbs\x123\n" +
"\x16mem_bandwidth_core_gbs\x18\x15 \x01(\x01R\x13memBandwidthCoreGbs\x1aA\n" +
"\x13EngineVersionsEntry\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\";\n" +
@@ -5732,7 +6013,32 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\x04kind\x18\x02 \x01(\tR\x04kind\x12\x10\n" +
"\x03seq\x18\x03 \x01(\x05R\x03seq\x12\x12\n" +
"\x04data\x18\x04 \x01(\fR\x04data\x12\x12\n" +
"\x04last\x18\x05 \x01(\bR\x04last2O\n" +
"\x04last\x18\x05 \x01(\bR\x04last\"\x8d\x02\n" +
"\x05Slurm\x12\x1e\n" +
"\n" +
"controller\x18\x01 \x01(\bR\n" +
"controller\x12\x12\n" +
"\x04node\x18\x02 \x01(\bR\x04node\x12\x12\n" +
"\x04name\x18\x03 \x01(\tR\x04name\x12\x1f\n" +
"\vconf_server\x18\x04 \x01(\tR\n" +
"confServer\x12\x1b\n" +
"\tmunge_key\x18\x05 \x01(\fR\bmungeKey\x120\n" +
"\x05files\x18\x06 \x03(\v2\x1a.llmgw.v1.Slurm.FilesEntryR\x05files\x12\x12\n" +
"\x04lent\x18\a \x01(\bR\x04lent\x1a8\n" +
"\n" +
"FilesEntry\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\x92\x02\n" +
"\vSlurmStatus\x12\x18\n" +
"\aversion\x18\x01 \x01(\tR\aversion\x12\x1a\n" +
"\bhardware\x18\x02 \x01(\tR\bhardware\x12%\n" +
"\x0eslurmd_running\x18\x03 \x01(\bR\rslurmdRunning\x12+\n" +
"\x11slurmctld_running\x18\x04 \x01(\bR\x10slurmctldRunning\x12\x14\n" +
"\x05state\x18\x05 \x01(\tR\x05state\x12\x16\n" +
"\x06reason\x18\x06 \x01(\tR\x06reason\x12\x12\n" +
"\x04jobs\x18\a \x01(\x05R\x04jobs\x12\x14\n" +
"\x05error\x18\b \x01(\tR\x05error\x12!\n" +
"\ffiles_sha256\x18\t \x01(\tR\vfilesSha2562O\n" +
"\fAgentService\x12?\n" +
"\aConnect\x12\x16.llmgw.v1.AgentMessage\x1a\x18.llmgw.v1.ControlMessage(\x010\x01B\x93\x01\n" +
"\fcom.llmgw.v1B\n" +
@@ -5751,7 +6057,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
}
var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 56)
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 59)
var file_llmgw_v1_agent_proto_goTypes = []any{
(ShardStatus_State)(0), // 0: llmgw.v1.ShardStatus.State
(*AgentMessage)(nil), // 1: llmgw.v1.AgentMessage
@@ -5802,14 +6108,17 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
(*TrainCancel)(nil), // 46: llmgw.v1.TrainCancel
(*TrainStatus)(nil), // 47: llmgw.v1.TrainStatus
(*TrainArtifact)(nil), // 48: llmgw.v1.TrainArtifact
nil, // 49: llmgw.v1.ControlMessage.TraceEntry
nil, // 50: llmgw.v1.Register.LabelsEntry
nil, // 51: llmgw.v1.Register.EngineVersionsEntry
nil, // 52: llmgw.v1.GPU.NicPathsEntry
nil, // 53: llmgw.v1.IBPort.CountersEntry
nil, // 54: llmgw.v1.Heartbeat.EngineVersionsEntry
nil, // 55: llmgw.v1.EngineWant.OfflineSha256Entry
nil, // 56: llmgw.v1.EngineProfile.EnvEntry
(*Slurm)(nil), // 49: llmgw.v1.Slurm
(*SlurmStatus)(nil), // 50: llmgw.v1.SlurmStatus
nil, // 51: llmgw.v1.ControlMessage.TraceEntry
nil, // 52: llmgw.v1.Register.LabelsEntry
nil, // 53: llmgw.v1.Register.EngineVersionsEntry
nil, // 54: llmgw.v1.GPU.NicPathsEntry
nil, // 55: llmgw.v1.IBPort.CountersEntry
nil, // 56: llmgw.v1.Heartbeat.EngineVersionsEntry
nil, // 57: llmgw.v1.EngineWant.OfflineSha256Entry
nil, // 58: llmgw.v1.EngineProfile.EnvEntry
nil, // 59: llmgw.v1.Slurm.FilesEntry
}
var file_llmgw_v1_agent_proto_depIdxs = []int32{
8, // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
@@ -5822,64 +6131,67 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
2, // 7: llmgw.v1.AgentMessage.kv_events:type_name -> llmgw.v1.KVEvents
47, // 8: llmgw.v1.AgentMessage.train_status:type_name -> llmgw.v1.TrainStatus
48, // 9: llmgw.v1.AgentMessage.train_artifact:type_name -> llmgw.v1.TrainArtifact
30, // 10: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
33, // 11: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
34, // 12: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
35, // 13: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
32, // 14: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
24, // 15: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
36, // 16: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
38, // 17: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
40, // 18: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
42, // 19: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
7, // 20: llmgw.v1.ControlMessage.control_planes:type_name -> llmgw.v1.ControlPlanes
5, // 21: llmgw.v1.ControlMessage.firewall:type_name -> llmgw.v1.Firewall
4, // 22: llmgw.v1.ControlMessage.restart_agent:type_name -> llmgw.v1.RestartAgent
6, // 23: llmgw.v1.ControlMessage.agent_settings:type_name -> llmgw.v1.AgentSettings
45, // 24: llmgw.v1.ControlMessage.train_start:type_name -> llmgw.v1.TrainStart
46, // 25: llmgw.v1.ControlMessage.train_cancel:type_name -> llmgw.v1.TrainCancel
49, // 26: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
50, // 27: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
12, // 28: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
14, // 29: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
51, // 30: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
11, // 31: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
15, // 32: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
10, // 33: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
9, // 34: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
21, // 35: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
0, // 36: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
27, // 37: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
52, // 38: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
13, // 39: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
18, // 40: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
17, // 41: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
53, // 42: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
20, // 43: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
10, // 44: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
15, // 45: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
54, // 46: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
26, // 47: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
16, // 48: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
19, // 49: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
14, // 50: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
22, // 51: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
21, // 52: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
25, // 53: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
55, // 54: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
0, // 55: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
29, // 56: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
27, // 57: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
31, // 58: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
56, // 59: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
44, // 60: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
1, // 61: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
3, // 62: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
62, // [62:63] is the sub-list for method output_type
61, // [61:62] is the sub-list for method input_type
61, // [61:61] is the sub-list for extension type_name
61, // [61:61] is the sub-list for extension extendee
0, // [0:61] is the sub-list for field type_name
50, // 10: llmgw.v1.AgentMessage.slurm_status:type_name -> llmgw.v1.SlurmStatus
30, // 11: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
33, // 12: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
34, // 13: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
35, // 14: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
32, // 15: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
24, // 16: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
36, // 17: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
38, // 18: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
40, // 19: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
42, // 20: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
7, // 21: llmgw.v1.ControlMessage.control_planes:type_name -> llmgw.v1.ControlPlanes
5, // 22: llmgw.v1.ControlMessage.firewall:type_name -> llmgw.v1.Firewall
4, // 23: llmgw.v1.ControlMessage.restart_agent:type_name -> llmgw.v1.RestartAgent
6, // 24: llmgw.v1.ControlMessage.agent_settings:type_name -> llmgw.v1.AgentSettings
45, // 25: llmgw.v1.ControlMessage.train_start:type_name -> llmgw.v1.TrainStart
46, // 26: llmgw.v1.ControlMessage.train_cancel:type_name -> llmgw.v1.TrainCancel
49, // 27: llmgw.v1.ControlMessage.slurm:type_name -> llmgw.v1.Slurm
51, // 28: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
52, // 29: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
12, // 30: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
14, // 31: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
53, // 32: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
11, // 33: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
15, // 34: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
10, // 35: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
9, // 36: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
21, // 37: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
0, // 38: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
27, // 39: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
54, // 40: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
13, // 41: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
18, // 42: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
17, // 43: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
55, // 44: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
20, // 45: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
10, // 46: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
15, // 47: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
56, // 48: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
26, // 49: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
16, // 50: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
19, // 51: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
14, // 52: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
22, // 53: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
21, // 54: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
25, // 55: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
57, // 56: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
0, // 57: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
29, // 58: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
27, // 59: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
31, // 60: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
58, // 61: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
44, // 62: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
59, // 63: llmgw.v1.Slurm.files:type_name -> llmgw.v1.Slurm.FilesEntry
1, // 64: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
3, // 65: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
65, // [65:66] is the sub-list for method output_type
64, // [64:65] is the sub-list for method input_type
64, // [64:64] is the sub-list for extension type_name
64, // [64:64] is the sub-list for extension extendee
0, // [0:64] is the sub-list for field type_name
}
func init() { file_llmgw_v1_agent_proto_init() }
@@ -5898,6 +6210,7 @@ func file_llmgw_v1_agent_proto_init() {
(*AgentMessage_KvEvents)(nil),
(*AgentMessage_TrainStatus)(nil),
(*AgentMessage_TrainArtifact)(nil),
(*AgentMessage_SlurmStatus)(nil),
}
file_llmgw_v1_agent_proto_msgTypes[2].OneofWrappers = []any{
(*ControlMessage_Start)(nil),
@@ -5916,6 +6229,7 @@ func file_llmgw_v1_agent_proto_init() {
(*ControlMessage_AgentSettings)(nil),
(*ControlMessage_TrainStart)(nil),
(*ControlMessage_TrainCancel)(nil),
(*ControlMessage_Slurm)(nil),
}
type x struct{}
out := protoimpl.TypeBuilder{
@@ -5923,7 +6237,7 @@ func file_llmgw_v1_agent_proto_init() {
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
NumEnums: 1,
NumMessages: 56,
NumMessages: 59,
NumExtensions: 0,
NumServices: 1,
},
+55
View File
@@ -30,6 +30,7 @@ message AgentMessage {
KVEvents kv_events = 8;
TrainStatus train_status = 9;
TrainArtifact train_artifact = 10;
SlurmStatus slurm_status = 11;
}
}
@@ -78,6 +79,7 @@ message ControlMessage {
AgentSettings agent_settings = 15;
TrainStart train_start = 16;
TrainCancel train_cancel = 17;
Slurm slurm = 18;
}
// trace is the W3C trace context (traceparent, and tracestate when there
// is one) of the control plane's span that sent this command, so what the
@@ -241,6 +243,10 @@ message Host {
// mem_bandwidth_gbs is how fast the memory reads, in GB/s, measured when
// the agent started: what decides a model's speed on the CPU. 0 unknown.
double mem_bandwidth_gbs = 7;
// mem_bandwidth_core_gbs is what one core reads alone, in GB/s: a replica
// given a few threads of a large machine reads at a few times this, not at
// the whole machine's rate. 0 unknown.
double mem_bandwidth_core_gbs = 8;
}
// Storage is one filesystem a node keeps weights on.
@@ -618,6 +624,8 @@ message Heartbeat {
// the agent has measured it — in the background after it starts, so a
// registration usually has none yet. 0 means not measured.
double mem_bandwidth_gbs = 20;
// mem_bandwidth_core_gbs is one core's read rate, as in Host.
double mem_bandwidth_core_gbs = 21;
}
// Provision is the software a node should have: which engines, at which
@@ -980,3 +988,50 @@ message TrainArtifact {
bytes data = 4;
bool last = 5;
}
// Slurm is what this node is in the Slurm cluster run beside llmgw
// (BACKLOG P): its controller, one of its nodes, both or neither. Sent when
// an agent registers and whenever it changes; the agent installs what is
// missing and keeps the daemons running as it says.
message Slurm {
// controller runs slurmctld here, on files.
bool controller = 1;
// node runs slurmd here, configless, fetching its files from conf_server.
bool node = 2;
// name is this node's NodeName in slurm.conf.
string name = 3;
// conf_server is slurmctld's host:port.
string conf_server = 4;
// munge_key is the key every machine of the cluster shares.
bytes munge_key = 5;
// files are the controller's, by name in /etc/slurm; empty for a node
// that is not the controller.
map<string, string> files = 6;
// lent says Slurm may run jobs here now: resume the node if llmgw drained
// it. Otherwise it is drained, its jobs left to finish.
bool lent = 7;
}
// SlurmStatus is what this node's Slurm daemons are doing, sent when it
// changes and every few minutes while the node has a part in Slurm.
message SlurmStatus {
// version is Slurm's, as "23.11.4"; empty when it is not installed.
string version = 1;
// hardware is `slurmd -C` on a node, which slurm.conf must match.
string hardware = 2;
bool slurmd_running = 3;
bool slurmctld_running = 4;
// state and reason are the controller's for this node, from scontrol.
string state = 5;
string reason = 6;
// jobs is how many job steps run on this node now, counted from their
// slurmstepd so a controller that is down does not hide them; -1 when the
// agent cannot tell.
int32 jobs = 7;
// error is the last thing that went wrong keeping Slurm, empty when
// nothing did.
string error = 8;
// files_sha256 is of the controller's files as written, so the leader
// knows they arrived.
string files_sha256 = 9;
}
+237 -1
View File
@@ -116,6 +116,9 @@
{
"name": "settings"
},
{
"name": "slurm"
},
{
"name": "summary"
},
@@ -3227,6 +3230,58 @@
}
}
},
"/api/v1/models/{name}/replicas": {
"post": {
"operationId": "postModelsNameReplicas",
"summary": "How many replicas run on each profile",
"description": "Seven on cards and four on CPUs, under the model's name. A profile not named keeps its count; cpu adds a profile on CPUs to a llama.cpp model that has none. Its scaling becomes the sum.",
"tags": [
"models"
],
"parameters": [
{
"name": "name",
"in": "path",
"required": true,
"description": "The model's name in the catalog.",
"schema": {
"type": "string"
}
},
{
"$ref": "#/components/parameters/Actor"
}
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ProfileReplicasRequest"
}
}
}
},
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/catalog.Model"
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/models/{name}/resources": {
"post": {
"operationId": "postModelsNameResources",
@@ -5026,6 +5081,34 @@
}
}
},
"/api/v1/slurm": {
"get": {
"operationId": "getSlurm",
"summary": "The Slurm cluster beside llmgw, and where each of its nodes is",
"description": "From the cluster's settings (slurm.*) and what each node's agent last said. A node's phase is llmgw (drained in Slurm, no job left), lending (its replicas moving off), lent, reclaiming (drained in Slurm, jobs still running) or unknown (its agent has not said).",
"tags": [
"slurm"
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/slurmsvc.Status"
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/summary": {
"get": {
"operationId": "getSummary",
@@ -6182,7 +6265,7 @@
},
"CollectResult": {
"type": "object",
"description": "CollectResult is what was deleted on one node, or why nothing was.",
"description": "CollectResult is what was deleted on one node, or why nothing was: what the node no longer lists once asked, not what it was asked to delete.",
"properties": {
"node": {
"type": "string"
@@ -6194,6 +6277,14 @@
"type": "integer",
"format": "int64"
},
"left": {
"type": "integer",
"description": "Left and LeftBytes are what the node still lists: refused, or not gone by the time the sweep stopped waiting. Its agent's log says why."
},
"left_bytes": {
"type": "integer",
"format": "int64"
},
"error": {
"type": "string"
}
@@ -8103,6 +8194,13 @@
"type": "string"
}
},
"profile_replicas": {
"type": "object",
"additionalProperties": {
"type": "integer"
},
"description": "ProfileReplicas is how many replicas run on each profile, for a model whose replicas are counted per profile; absent otherwise."
},
"memory_gb": {
"type": "integer",
"description": "MemoryGB and CPUCores are what one replica reserves, from its first profile; MeasuredMemoryGB is the most one of its replicas uses now, 0 when none is measured."
@@ -8305,6 +8403,10 @@
"type": "number",
"description": "MemBandwidthGBs is how fast its memory reads, in GB/s, measured when its agent started; 0 when not reported. A model's speed on the CPU."
},
"mem_bandwidth_core_gbs": {
"type": "number",
"description": "MemBandwidthCoreGBs is what one core reads alone, in GB/s; 0 when not reported. A CPU replica's threads read at about this each."
},
"engines": {
"type": "array",
"items": {
@@ -8661,6 +8763,21 @@
}
}
},
"ProfileReplicasRequest": {
"type": "object",
"description": "ProfileReplicasRequest counts a model's replicas per profile: profile name to replicas, and CPU, for a llama.cpp model with no profile on CPUs, the one to add.",
"properties": {
"profiles": {
"type": "object",
"additionalProperties": {
"type": "integer"
}
},
"cpu": {
"$ref": "#/components/schemas/modelsvc.CPUProfile"
}
}
},
"ProjectView": {
"type": "object",
"description": "ProjectView is a project, with how many people and keys are in it.",
@@ -10506,6 +10623,10 @@
"type": "integer",
"description": "Priority orders profiles during placement; highest that fits wins."
},
"replicas": {
"type": "integer",
"description": "Replicas, set on any of a model's profiles, is how many of its replicas run on this one: seven on cards and four on CPUs, all answering under the model's name, rather than each replica on the best profile that fits. The model's scaling is then the sum, min and max alike; a profile left at zero runs none."
},
"startup_timeout": {
"type": "integer",
"format": "int64",
@@ -11406,6 +11527,10 @@
},
"strata_size": {
"type": "string"
},
"tokens_per_second": {
"type": "number",
"description": "TokensPerSecond is about how many tokens a second one reply gets this way, on the fastest host it fits; 0 when not known. On the CPU with the threads a replica is given by default: half the node's."
}
}
},
@@ -11651,6 +11776,10 @@
"context": {
"type": "integer"
},
"trained": {
"type": "integer",
"description": "Trained is the context the model was trained for, the most a request may use; 0 when neither its header nor its config says."
},
"kv_bytes_per_token": {
"type": "integer",
"format": "int64",
@@ -11898,6 +12027,21 @@
}
}
},
"modelsvc.CPUProfile": {
"type": "object",
"description": "CPUProfile is a profile on CPUs to add to a llama.cpp model, for replicas counted per profile: what a replica of it reserves.",
"properties": {
"name": {
"type": "string"
},
"memory_gb": {
"type": "integer"
},
"cpu_cores": {
"type": "integer"
}
}
},
"notify.Result": {
"type": "object",
"description": "Result is how one webhook answered a test.",
@@ -12065,6 +12209,98 @@
}
}
},
"slurmsvc.NodeStatus": {
"type": "object",
"description": "NodeStatus is one node as the service sees it.",
"properties": {
"node": {
"type": "string"
},
"phase": {
"type": "string"
},
"replicas": {
"type": "integer"
},
"slurm": {
"$ref": "#/components/schemas/slurmsvc.Reported"
},
"connected": {
"type": "boolean"
}
}
},
"slurmsvc.Reported": {
"type": "object",
"description": "Reported is the agent's SlurmStatus, for the admin API.",
"properties": {
"version": {
"type": "string"
},
"hardware": {
"type": "string"
},
"slurmd_running": {
"type": "boolean"
},
"slurmctld_running": {
"type": "boolean"
},
"state": {
"type": "string"
},
"reason": {
"type": "string"
},
"jobs": {
"type": "integer"
},
"error": {
"type": "string"
},
"files_sha256": {
"type": "string"
},
"at": {
"type": "string",
"format": "date-time"
}
}
},
"slurmsvc.Status": {
"type": "object",
"description": "Status is the whole Slurm cluster, for `ctl slurm status`.",
"properties": {
"enabled": {
"type": "boolean"
},
"cluster": {
"type": "string"
},
"controller": {
"type": "string"
},
"partition": {
"type": "string"
},
"controller_status": {
"$ref": "#/components/schemas/slurmsvc.Reported"
},
"nodes": {
"type": "array",
"items": {
"$ref": "#/components/schemas/slurmsvc.NodeStatus"
}
},
"waiting": {
"type": "array",
"items": {
"type": "string"
},
"description": "Waiting are nodes left out of slurm.conf until they report their hardware."
}
}
},
"store.AuditEvent": {
"type": "object",
"description": "AuditEvent is one change: who made it, through what, to what, and how it ended. Detail never holds a secret; the writer takes them out.",
+2 -1
View File
@@ -220,7 +220,8 @@ and change it with the same commands as `llmgw users`, `llmgw keys` and
dashboard, and what they belong to. **A** creates an account, already
approved. On a row: approve one waiting, suspend or reinstate, give or take
the cluster dashboard, add to or take out of an organization, team or
project, their limits, a new password (their sessions close), delete, and
project (on an organization as a member, its manager, who sees its page, or
its admin, who also changes it), their limits, a new password (their sessions close), delete, and
**Forget them…** for a request to be forgotten (`llmgw users forget`).
**API keys** — every key, its project, owner, state and last use. **A**
+17 -2
View File
@@ -166,13 +166,28 @@ is a flag too, and the plan shows the command with them: `--starter qwen3-8b`
`--judge-fail open`, `--judge-categories`, `--judge-replicas`,
`--judge-budget`, `--language tr`, `--signup-open`, `--default-rpm`,
`--default-tpm`, `--max-key-ttl`, `--media` (pictures and 3D: ComfyUI on the
nodes, and `presets.auto`), `--evict-after-idle` and `--min-lifetime`. They
nodes, and `presets.auto`), `--slurm` (see below), `--evict-after-idle` and `--min-lifetime`. They
go into `server.yaml` as init writes it, and become the cluster's first
settings when it starts; a `server.yaml` already there is never overwritten,
so run again they change nothing — the settings page does. An NVIDIA card with no working
driver is reported rather than silently ignored; install the driver and run
init again to serve on it.
**Slurm, for training and batch jobs.** `--slurm`, or the Slurm part on the
choices screen, runs a Slurm cluster beside llmgw: slurmctld on this
machine's node, slurmd on every node, those that join later too
(`slurm.all_nodes`). The agents install munge and Slurm (22.05 or newer)
from the machines' own package sources. No node is Slurm's at first: each is
drained in Slurm and serves models until `llmgw ctl slurm lend NODE` gives it
to Slurm — its models move off first — and `llmgw ctl slurm reclaim NODE`
takes it back once its jobs end. Jobs are submitted on this machine with
`sbatch` and `srun`; `llmgw ctl slurm status` shows where each node is. It
needs this machine to be a node, and its hostname to be a name Slurm takes.
On a cluster installed without it, `llmgw ctl slurm setup --controller NODE
--all-nodes` does the same later. Slurm's ports (6817–6818, and srun's 60001–63000) answer the cluster's
networks only, as llmgw's own do; a firewall of the site's own between the
nodes must let them through.
**Secure by default.** What init writes is the hardened configuration, not a
demonstration one:
@@ -3145,7 +3160,7 @@ loaded at boot by `llmgw-firewall.service`):
| Machine | Ports, for the cluster only |
| --- | --- |
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
| node | engine ports (`control.port_range`, 20000–20999), 9100 |
| node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6818 (slurmctld, slurmd) and 60001–63000 (srun's `SrunPortRange`) |
| store member | 5432, 8008, 2379, 2380, 9100 |
| mirror | 8081, 8082, 9100 |
Binary file not shown.
+32 -11
View File
@@ -192,22 +192,41 @@ auth:
A personal key has no project, so it keeps no conversations or agent notes;
those are kept per project.
### Organization managers
### Organization managers and admins
A member of an organization can be made its manager:
A member of an organization can be made its manager, who sees it, or its
admin, who also changes it:
```
llmgw users join boss@acme.com --org Acme --role manager
llmgw users join lead@acme.com --org Acme --role admin
```
or with the star on their organization membership on `/admin/users`. A manager
gets an **Organization** link and a page at `/org`: usage over time by team,
or with the role chosen on their organization membership on `/admin/users`.
Both get an **Organization** link and a page at `/org`: usage over time by team,
project or person, the teams and projects with their limits and spend, and every
member. Only what was spent in that organization's projects is on it; what its
people do elsewhere is not. The page is read-only — a manager is not an
administrator of the cluster, and limits and memberships stay with those who
are. An administrator following `/org` lands on the same organization's panel
on the people page instead, where all of that can be changed.
people do elsewhere is not.
A manager's page is read-only. An organization's admin also changes the
organization from it:
- adds and renames its teams and projects, and removes an empty one;
- adds a person by their sign-in address to the organization, a team or a
project, and takes them out (their keys there are revoked with it);
- makes a member of the organization a manager or an admin, or neither;
- revokes a key spent in its projects.
What reaches past the organization stays with the cluster's administrators:
its limits and budgets, approving or suspending an account (someone not yet
approved cannot be added), the organization's own name, and anything else on
the cluster. An admin cannot take away their own role. A group from single
sign-on or Active Directory that puts someone in the organization does not, by
saying so again at the next sign-in, take a manager's or admin's role away.
An administrator of the cluster following `/org` lands on the same
organization's panel on the people page instead, where all of that can be
changed.
`llmgw keys init` still works for provisioning scripts and is idempotent, but it
will no longer move a team or project that already exists under another parent;
@@ -517,8 +536,10 @@ included: `llmgw ctl audit`, or **Audit** in the console.
suspends accounts from it; see [install.md](install.md#single-sign-on).
Without them accounts are local, with local passwords.
- **Staff roles are coarse.** `viewer`, `operator`, `billing` and `admin`
(`llmgw users role`) decide what someone may see and do under /admin; an
organization's manager can view it but not change it. Every change is in
the audit log (`llmgw ctl audit`).
(`llmgw users role`) decide what someone may see and do under /admin, and a
page draws only the forms that person may send: a viewer sees no forms. An
organization's manager can view it; its admin also changes its teams,
projects, people and keys. Every change is in the audit log (`llmgw ctl
audit`).
- **Rate limits are per gateway instance.** With two instances each counts its own,
so the effective limit doubles. v1 is single-instance by design.
+79
View File
@@ -295,6 +295,7 @@ rather than repeated.
| `keep_within` | `string` | [Profile](#profile) |
| `weight` | `float64` | [Profile](#profile) |
| `priority` | `int` | [Profile](#profile) |
| `replicas` | `int` | [Profile](#profile) |
| `startup_timeout` | `time.Duration` | [Profile](#profile) |
| `kv_events` | `bool` | [Profile](#profile) |
| `gpu` | `*GPUNeed` | [Profile](#profile) |
@@ -375,6 +376,7 @@ rather than repeated.
| `mail` | `mail.Config` | [Server](#server) |
| `serving` | `ServingSettings` | [Server](#server) |
| `firewall` | `FirewallSettings` | [Server](#server) |
| `slurm` | `SlurmSettings` | [Server](#server) |
| `presets` | `PresetSettings` | [Server](#server) |
| `on_demand` | `OnDemandSettings` | [Server](#server) |
| `agents` | `AgentSettings` | [Server](#server) |
@@ -400,6 +402,13 @@ rather than repeated.
| `timeout` | `time.Duration` | [Service](#service) |
| `stopped` | `bool` | [ServingSettings](#servingsettings) |
| `bin` | `string` | [SimConfig](#simconfig) |
| `enabled` | `bool` | [SlurmSettings](#slurmsettings) |
| `cluster_name` | `string` | [SlurmSettings](#slurmsettings) |
| `controller` | `string` | [SlurmSettings](#slurmsettings) |
| `nodes` | `[]string` | [SlurmSettings](#slurmsettings) |
| `all_nodes` | `bool` | [SlurmSettings](#slurmsettings) |
| `lent` | `[]string` | [SlurmSettings](#slurmsettings) |
| `partition` | `string` | [SlurmSettings](#slurmsettings) |
| `dir` | `string` | [StrataConfig](#strataconfig) |
| `python` | `string` | [StrataConfig](#strataconfig) |
| `engine` | `string` | [StrataConfig](#strataconfig) |
@@ -2611,6 +2620,16 @@ rather than a guess (plan §6.2.1).
Priority orders profiles during placement; highest that fits wins.
### `replicas`
`int`
Replicas, set on any of a model's profiles, is how many of its
replicas run on this one: seven on cards and four on CPUs, all
answering under the model's name, rather than each replica on the
best profile that fits. The model's scaling is then the sum, min and
max alike; a profile left at zero runs none.
### `startup_timeout`
`time.Duration`
@@ -3268,6 +3287,15 @@ Firewall turns llmgw's firewall on every machine on or off.
Takes a [FirewallSettings](#firewallsettings) block.
### `slurm`
`SlurmSettings`
Slurm is a Slurm cluster run beside llmgw on some of its nodes, and
which of them it may use now.
Takes a [SlurmSettings](#slurmsettings) block.
### `presets`
`PresetSettings`
@@ -3469,6 +3497,57 @@ entries ask. In force on every control plane within seconds.
Bin is the llmgw binary itself; the engine is one of its subcommands.
Empty means whatever started this agent, which is right almost always.
## SlurmSettings
### `enabled`
`bool`
Enabled runs Slurm on the nodes below. False stops llmgw keeping it;
what was installed is left installed, and every node is llmgw's.
### `cluster_name`
`string`
ClusterName is Slurm's ClusterName. Empty is "llmgw".
### `controller`
`string`
Controller is the llmgw node whose agent runs slurmctld. It needs no
GPU, and need not be one of Nodes.
### `nodes`
`[]string`
Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained
in Slurm, until it is also in Lent.
### `all_nodes`
`bool`
AllNodes runs slurmd on every llmgw node, those that join later too,
besides any named in Nodes. Each is llmgw's until it is lent, as any.
### `lent`
`[]string`
Lent are the nodes of Nodes Slurm may run jobs on now. A node put here
is drained of llmgw's replicas first, as `ctl drain` does, and resumed
in Slurm once none is left; a node taken out is drained in Slurm and
given back to llmgw when its last job ends.
### `partition`
`string`
Partition is the one partition every node is in. Empty is "gpu".
## StrataConfig
### `dir`
+72 -16
View File
@@ -35,6 +35,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/slurmnode"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
@@ -97,7 +98,10 @@ type Agent struct {
// (nil: this machine's shell).
fwMu sync.Mutex
fwRunner firewall.Runner
inv discovery.Inventory
// fwLast is the last firewall message, applied again when the node
// is given a part in Slurm. Under fwMu.
fwLast *llmgwv1.Firewall
inv discovery.Inventory
// shardReplica remembers which replica a shard belongs to, so a status
// report can name both. The control plane keys on the replica.
shardReplica map[string]string
@@ -148,6 +152,10 @@ type Agent struct {
// trainer runs LoRA training jobs, one at a time (BACKLOG F4).
trainer *train.Runner
// slurm keeps the node's part in Slurm (BACKLOG P); slurmOut is where
// its status goes, under mu.
slurm *slurmnode.Keeper
slurmOut chan<- *llmgwv1.AgentMessage
}
// connectHTTPClient is the HTTP client connect uses; an interface so tests can
@@ -206,6 +214,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
}
a.trainer = train.New(train.Env{EnginesDir: enginesDir, WorkDir: filepath.Join(cfg.Superv.WorkDir, "trainer"),
RunAs: runAs, Log: log})
a.slurm = slurmnode.New(slurmnode.Local{}, a.prov, slurmMarker, log)
return a, nil
}
@@ -232,6 +241,7 @@ func (a *Agent) Run(ctx context.Context) error {
}
go a.provisionLoop(ctx)
go a.ensureDiags(ctx)
go a.slurm.Run(ctx, a.sendSlurm)
attempt := 0
next := 0
@@ -519,7 +529,8 @@ func (a *Agent) scan(ctx context.Context) error {
inv, err := discovery.Discover(ctx, opts, a.log)
host := discovery.DiscoverHost()
host.MinComputeCapability = discovery.MinComputeCapability(inv.GPUs)
host.MemBandwidthGBs = a.memBandwidth()
bw := a.memBandwidth()
host.MemBandwidthGBs, host.MemBandwidthCoreGBs = bw.All, bw.Core
a.mu.Lock()
a.inv = inv
a.prov.Host = host
@@ -529,6 +540,18 @@ func (a *Agent) scan(ctx context.Context) error {
return err
}
// forgetSent is a new stream: a control plane that knows nothing of where
// installs stand, nor which files are here beyond what registration says —
// and registration says what the startup scan found. So the next heartbeat
// tells it both again. Without it, files deleted after the agent started came
// back on a control plane that restarted, and stayed until the next change.
func (a *Agent) forgetSent() {
a.mu.Lock()
defer a.mu.Unlock()
a.provSent = ""
a.lastWeights = ""
}
// session runs one connection until it breaks.
func (a *Agent) session(ctx context.Context, server string) error {
client := llmgwv1connect.NewAgentServiceClient(a.http, server)
@@ -565,20 +588,19 @@ func (a *Agent) session(ctx context.Context, server string) error {
reg.EngineVersions = a.engines(parent)
reg.EnginesProbed = true
reg.InstallableEngines = slices.DeleteFunc(provision.Installable(), a.disabled)
reg.Features = []string{FeatureFabricTest}
reg.Features = []string{FeatureFabricTest, FeatureSlurm}
reg.RunningShards = a.runningShards()
a.mu.Lock()
h := a.prov.Host
// A new stream is a control plane that knows nothing of where installs
// stand, so the next heartbeat tells it again.
a.provSent = ""
a.mu.Unlock()
a.forgetSent()
reg.Host = &llmgwv1.Host{
OsId: h.OSID, OsVersion: h.OSVersion, OsCodename: h.OSCodename,
Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver,
// What is measured by now, not what the startup scan found before
// the measurement ended: a reconnect would otherwise say 0.
MemBandwidthGbs: a.memBandwidth(),
MemBandwidthGbs: a.memBandwidth().All,
MemBandwidthCoreGbs: a.memBandwidth().Core,
}
if err := stream.Send(&llmgwv1.AgentMessage{
@@ -616,6 +638,8 @@ func (a *Agent) session(ctx context.Context, server string) error {
go a.watchExits(ctx, out, unready(reg.RunningShards))
go a.forwardKV(ctx, out)
go a.reportCompute(ctx, out)
a.setSlurmOut(out)
defer a.setSlurmOut(nil)
for {
msg, err := stream.Receive()
@@ -676,7 +700,8 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
// without anybody restarting anything.
hb.MemoryTotalMb = int64(discovery.MemoryTotalMB())
hb.CpuCount = int32(runtime.NumCPU())
hb.MemBandwidthGbs = a.memBandwidth()
bw := a.memBandwidth()
hb.MemBandwidthGbs, hb.MemBandwidthCoreGbs = bw.All, bw.Core
a.reprobeEngines()
a.mu.Lock()
@@ -881,7 +906,23 @@ func (a *Agent) removeWeights(paths []string) {
a.log.Warn("refused to remove a file llmgw did not fetch", "path", p)
continue
}
if err := os.Remove(abs); err != nil && !os.IsNotExist(err) {
// A set — every file of one version, fetched into a directory of its
// own — goes whole: os.Remove refused it as not empty, and nothing
// was freed of what a model of several files left. Lstat, so a link
// is taken away and never what it points at.
info, err := os.Lstat(abs)
switch {
case os.IsNotExist(err):
continue
case err != nil:
a.log.Warn("could not look at cached weights", "path", p, "err", err)
continue
case info.IsDir():
err = os.RemoveAll(abs)
default:
err = os.Remove(abs)
}
if err != nil {
a.log.Warn("could not remove cached weights", "path", p, "err", err)
continue
}
@@ -922,6 +963,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
go a.applySettings(m.AgentSettings)
case *llmgwv1.ControlMessage_TrainStart:
go a.trainJob(ctx, m.TrainStart, out)
case *llmgwv1.ControlMessage_Slurm:
a.slurmMessage(m.Slurm)
case *llmgwv1.ControlMessage_TrainCancel:
a.trainer.Cancel(m.TrainCancel.GetJobId())
case *llmgwv1.ControlMessage_RestartAgent:
@@ -1573,20 +1616,32 @@ func (a *Agent) reportCompute(ctx context.Context, out chan<- *llmgwv1.AgentMess
// VMs on one host that all restart together measure each other, and read a
// third less than any of them gets alone. Once a process rather than an
// agent, so tests running many agents in one read it once.
func (a *Agent) memBandwidth() float64 {
func (a *Agent) memBandwidth() discovery.Bandwidth {
bwOnce.Do(func() {
go func() {
time.Sleep(bandwidthDelay())
for {
if bw := measureBandwidth(); bw > math.Float64frombits(bwBits.Load()) {
bwBits.Store(math.Float64bits(bw))
a.log.Info("this node's memory", "read_gbs", math.Round(bw*10)/10)
// The best of each seen so far: an hourly pass on a node
// busy serving reads less, and that is not the memory.
bw := measureBandwidth()
cur := a.memBandwidthNow()
if bw.All > cur.All || bw.Core > cur.Core {
bwBits.Store(math.Float64bits(max(bw.All, cur.All)))
coreBits.Store(math.Float64bits(max(bw.Core, cur.Core)))
a.log.Info("this node's memory", "read_gbs", math.Round(bw.All*10)/10,
"one_core_gbs", math.Round(bw.Core*10)/10)
}
time.Sleep(time.Hour)
}
}()
})
return math.Float64frombits(bwBits.Load())
return a.memBandwidthNow()
}
// memBandwidthNow is what has been measured so far, without starting a
// measurement.
func (a *Agent) memBandwidthNow() discovery.Bandwidth {
return discovery.Bandwidth{All: math.Float64frombits(bwBits.Load()), Core: math.Float64frombits(coreBits.Load())}
}
// bandwidthDelay is how long an agent waits before measuring, so agents
@@ -1594,8 +1649,9 @@ func (a *Agent) memBandwidth() float64 {
var bandwidthDelay = func() time.Duration { return 10*time.Second + rand.N(110*time.Second) }
var (
bwOnce sync.Once
bwBits atomic.Uint64
bwOnce sync.Once
bwBits atomic.Uint64
coreBits atomic.Uint64
// measureBandwidth is replaced in tests, which have no use for it.
measureBandwidth = discovery.MeasureMemoryBandwidth
)
+20
View File
@@ -925,3 +925,23 @@ func TestABandwidthTestIsRunOnTheAdapterAskedFor(t *testing.T) {
t.Errorf("reply %+v, ran %+v", r, got)
}
}
func TestANewStreamIsToldTheFilesAgain(t *testing.T) {
// Registration carries what the startup scan found; a file deleted since
// came back on a control plane that restarted, because the next
// heartbeat saw the list unchanged since it was last sent.
dir := t.TempDir()
h := start(t, func(c *Config) { c.ModelDirs = []string{dir} })
if err := os.WriteFile(filepath.Join(dir, "0123456789abcdef-m.gguf"), []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
h.ag.heartbeat()
if hb := h.ag.heartbeat().GetHeartbeat(); hb.GetWeightsChanged() {
t.Fatal("an unchanged list should not be sent again on the same stream")
}
h.ag.forgetSent()
hb := h.ag.heartbeat().GetHeartbeat()
if !hb.GetWeightsChanged() || len(hb.GetWeights()) != 1 {
t.Errorf("a new stream should be told the files: changed=%v %v", hb.GetWeightsChanged(), hb.GetWeights())
}
}
+4
View File
@@ -35,6 +35,10 @@ type Host struct {
// measured: what decides a model's speed on the CPU. Not read here but
// measured by the agent when it starts (MeasureMemoryBandwidth).
MemBandwidthGBs float64 `json:"mem_bandwidth_gbs,omitempty"`
// MemBandwidthCoreGBs is what one core reads alone, in GB/s, 0 when not
// measured: a replica given n threads reads at about n times this, up to
// MemBandwidthGBs.
MemBandwidthCoreGBs float64 `json:"mem_bandwidth_core_gbs,omitempty"`
}
// Family is the packaging family: "deb", "rpm", or "" when neither is known.
+28 -5
View File
@@ -28,9 +28,22 @@ const (
minBuffer = 256 << 20
)
// MeasureMemoryBandwidth is the memory's read rate in GB/s, the best of a
// few passes; 0 when the machine has too little free memory to measure.
func MeasureMemoryBandwidth() float64 {
// Bandwidth is how fast this machine's memory reads, in GB/s: with every
// CPU reading, and with one alone. A replica given a few threads of a large
// machine reads at a few times the second, not at the first: on a server of
// many memory channels one core cannot keep more than a fraction of them
// busy.
type Bandwidth struct {
All, Core float64
}
// coreBuffer is what the one-core passes read: past any cache, and short
// enough that one core reads it in a fraction of a second.
const coreBuffer = 512 << 20
// MeasureMemoryBandwidth is the memory's read rate, the best of a few passes
// each way; zero when the machine has too little free memory to measure.
func MeasureMemoryBandwidth() Bandwidth {
size := int64(bandwidthBuffer)
// Never more than a quarter of what is free: a node already holding
// models must not be pushed into swap by a measurement. Without room
@@ -40,7 +53,7 @@ func MeasureMemoryBandwidth() float64 {
size = min(size, avail/4)
}
if size < max(minBuffer, 8*lastCacheBytes("/sys/devices/system/cpu")) {
return 0
return Bandwidth{}
}
buf := make([]uint64, size/8)
for i := range buf {
@@ -68,7 +81,17 @@ func MeasureMemoryBandwidth() float64 {
best = max(best, float64(size)/s/1e9)
}
}
return best
// And one core alone, over a part of the same buffer.
part := buf[:min(len(buf), coreBuffer/8)]
core := 0.0
for pass := 0; pass < 3; pass++ {
start := time.Now()
sink(sum(part))
if s := time.Since(start).Seconds(); s > 0 {
core = max(core, float64(len(part)*8)/s/1e9)
}
}
return Bandwidth{All: best, Core: min(core, best)}
}
// sum reads every word, eight at a time so the loop is not what limits it.
+4 -4
View File
@@ -12,11 +12,11 @@ func TestMemoryBandwidthIsMeasured(t *testing.T) {
}
bw := MeasureMemoryBandwidth()
// Any machine running this reads more than a gigabyte a second and less
// than a few terabytes.
if bw < 1 || bw > 5000 {
t.Errorf("measured %.1f GB/s", bw)
// than a few terabytes, and one core no more than all of them.
if bw.All < 1 || bw.All > 5000 || bw.Core < 0.5 || bw.Core > bw.All {
t.Errorf("measured %.1f GB/s, %.1f on one core", bw.All, bw.Core)
}
t.Logf("%.1f GB/s", bw)
t.Logf("%.1f GB/s, %.1f on one core", bw.All, bw.Core)
}
func TestTheLastCacheIsReadFromSys(t *testing.T) {
+30
View File
@@ -10,8 +10,16 @@ import (
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
// slurmPorts are Slurm's on a node that runs any of it: slurmctld's and
// slurmd's, and srun's range for the steps it starts.
var slurmPorts = []firewall.Port{
{From: slurm.CtldPort, To: slurm.SlurmdPort, What: "slurm: slurmctld and slurmd"},
{From: slurm.SrunPortsFrom, To: slurm.SrunPortsTo, What: "slurm: srun, for the steps it starts"},
}
// applyFirewall keeps llmgw's firewall on this node as the cluster's
// settings say: the engines' ports and the node exporter's for the networks
// the node reaches its control planes on, and those named, or no table at
@@ -19,6 +27,7 @@ import (
func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
a.fwMu.Lock()
defer a.fwMu.Unlock()
a.fwLast = m
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()
r := a.fwRunner
@@ -55,6 +64,11 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
if from > 0 && to >= from {
ports = append(ports, firewall.Engines(from, to))
}
// Once there, they stay: the ruleset keeps every port it held, so a
// node that leaves Slurm keeps them closed to everyone but the cluster.
if a.slurm != nil && a.slurm.HasPart() {
ports = append(ports, slurmPorts...)
}
changed, err := firewall.Install(ctx, r, firewall.Spec{Ports: ports, Allow: nets})
switch {
case err != nil:
@@ -64,6 +78,22 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
}
}
// slurmMessage is the leader's word on this node's part in Slurm. A node
// that gets one has the firewall applied again, with Slurm's ports.
func (a *Agent) slurmMessage(m *llmgwv1.Slurm) {
had := a.slurm.HasPart()
a.slurm.Apply(m)
if had || !a.slurm.HasPart() {
return
}
a.fwMu.Lock()
last := a.fwLast
a.fwMu.Unlock()
if last != nil && last.GetEnabled() {
go a.applyFirewall(last)
}
}
// controlPlaneIPs are the addresses of the control planes an agent dials.
func controlPlaneIPs(ctx context.Context, servers []string) []string {
var out []string
+27
View File
@@ -6,8 +6,10 @@ import (
"log/slog"
"strings"
"testing"
"time"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/slurmnode"
)
// fakeShell answers the firewall package's scripts and records what was
@@ -55,3 +57,28 @@ func TestANodeKeepsOrDropsItsFirewallAsTheClusterSays(t *testing.T) {
t.Error("turned off, the table stayed")
}
}
func TestANodeGivenAPartInSlurmLetsSlurmInFromTheClusterOnly(t *testing.T) {
sh := &fakeShell{}
a := &Agent{log: slog.New(slog.DiscardHandler), fwRunner: sh, serverList: []string{"https://10.0.0.140:7443"},
slurm: slurmnode.New(nil, nil, "", slog.New(slog.DiscardHandler))}
a.applyFirewall(&llmgwv1.Firewall{Enabled: true, EnginePortsFrom: 20000, EnginePortsTo: 20999})
if strings.Contains(sh.installed, "6817") {
t.Fatalf("Slurm's ports before the node has a part in it:\n%s", sh.installed)
}
a.slurmMessage(&llmgwv1.Slurm{Name: "node05", Node: true, ConfServer: "10.0.0.148:6817"})
deadline := time.Now().Add(5 * time.Second)
for {
a.fwMu.Lock()
got := sh.installed
a.fwMu.Unlock()
if strings.Contains(got, "6817-6818") && strings.Contains(got, "60001-63000") {
break
}
if time.Now().After(deadline) {
t.Fatalf("given a part in Slurm, the rules still lack its ports:\n%s", got)
}
time.Sleep(10 * time.Millisecond)
}
}
+3 -1
View File
@@ -4,12 +4,14 @@ import (
"os"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
)
func TestMain(m *testing.M) {
// Half a gigabyte read per test binary, and contending with the tests
// for the CPUs: these never look at the figure.
measureBandwidth = func() float64 { return 0 }
measureBandwidth = func() discovery.Bandwidth { return discovery.Bandwidth{} }
bandwidthDelay = func() time.Duration { return 0 }
os.Exit(m.Run())
}
+7
View File
@@ -164,6 +164,13 @@ func (a *Agent) provisionLoop(ctx context.Context) {
// provisionStatus records a change in one engine's installation.
func (a *Agent) provisionStatus(st provision.Status) {
if st == (provision.Status{}) {
// A package install that was no engine's has ended.
a.mu.Lock()
delete(a.provStatus, "")
a.mu.Unlock()
return
}
a.mu.Lock()
prev, had := a.provStatus[st.Engine]
a.provStatus[st.Engine] = st
+7 -1
View File
@@ -497,7 +497,13 @@ func (in *Installer) packages(ctx context.Context, byFamily map[string][]string,
// byFamily is keyed by discovery.Host.Family: "deb" or "rpm".
func (in *Installer) InstallPackages(ctx context.Context, byFamily map[string][]string) error {
var st Status
return in.packages(ctx, byFamily, &st)
err := in.packages(ctx, byFamily, &st)
if st.State == Installing {
// Its progress was no engine's: an empty status takes it off the
// node's page again, done or failed (the caller says which).
in.report(Status{})
}
return err
}
// lockWait bounds how long a package command waits for another to finish.
@@ -0,0 +1,36 @@
package provision
import (
"context"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
)
// A package install that is no engine's says it is done, so its progress
// does not stay on the node's page (found on the PoC installing Slurm).
func TestInstallPackagesClearsItsProgress(t *testing.T) {
var got []Status
in := &Installer{Host: discovery.Host{OSID: "ubuntu"}, OnStatus: func(s Status) { got = append(got, s) },
Run: func(_ context.Context, _ []string, name string, _ ...string) ([]byte, error) {
if name == "sh" { // the query: not installed
return nil, context.Canceled
}
return nil, nil
}}
if err := in.InstallPackages(context.Background(), map[string][]string{"deb": {"munge"}}); err != nil {
t.Fatal(err)
}
if len(got) < 2 || got[0].State != Installing || got[len(got)-1] != (Status{}) {
t.Fatalf("statuses %+v", got)
}
got = nil
in.Run = func(context.Context, []string, string, ...string) ([]byte, error) { return nil, nil }
if err := in.InstallPackages(context.Background(), map[string][]string{"deb": {"munge"}}); err != nil {
t.Fatal(err)
}
if len(got) != 0 {
t.Fatalf("installed already, yet said %+v", got)
}
}
+32
View File
@@ -94,3 +94,35 @@ func TestWithNoCacheConfiguredNothingIsDeleted(t *testing.T) {
t.Errorf("an agent with no cache directory deleted something: %v", err)
}
}
// A model of several files is fetched into a directory of its own, and an
// unused one goes whole: it was refused as not empty, and nothing was freed.
func TestAnUnusedSetIsRemovedWhole(t *testing.T) {
dir := t.TempDir()
a := agentWithCache(t, dir)
set := weights.CachePathFor(dir, "https://mirror/coder/v1")
if err := os.MkdirAll(set, 0o755); err != nil {
t.Fatal(err)
}
write(t, filepath.Join(set, "model-00001-of-00002.gguf"))
write(t, filepath.Join(set, ".complete"))
// A link in the cache is taken away, not what it points at.
target := t.TempDir()
write(t, filepath.Join(target, "keep.gguf"))
link := weights.CachePathFor(dir, "https://mirror/linked/v1")
if err := os.Symlink(target, link); err != nil {
t.Fatal(err)
}
a.removeWeights([]string{set, link})
if _, err := os.Stat(set); !os.IsNotExist(err) {
t.Errorf("the set is still there: %v", err)
}
if _, err := os.Lstat(link); !os.IsNotExist(err) {
t.Errorf("the link is still there: %v", err)
}
if _, err := os.Stat(filepath.Join(target, "keep.gguf")); err != nil {
t.Errorf("what the link pointed at was deleted: %v", err)
}
}
+38
View File
@@ -0,0 +1,38 @@
package agent
import (
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
)
// FeatureSlurm is the agent keeping its node's part in Slurm (BACKLOG P).
const FeatureSlurm = "slurm"
// slurmMarker says llmgw started Slurm's daemons on this machine.
var slurmMarker = "/var/lib/llmgw-agent/slurm"
// setSlurmOut is where Slurm's status goes now: this session's queue, or
// nowhere between sessions.
func (a *Agent) setSlurmOut(out chan<- *llmgwv1.AgentMessage) {
a.mu.Lock()
a.slurmOut = out
a.mu.Unlock()
if out != nil {
a.slurm.Resend()
}
}
// sendSlurm queues a status without waiting: a full queue or no session
// drops it, and the next session is sent it again.
func (a *Agent) sendSlurm(st *llmgwv1.SlurmStatus) {
a.mu.Lock()
out := a.slurmOut
a.mu.Unlock()
if out == nil {
return
}
select {
case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_SlurmStatus{SlurmStatus: st}}:
default:
a.log.Warn("Slurm's status not sent: the queue to the control plane is full")
}
}
+612
View File
@@ -0,0 +1,612 @@
// Package slurmnode keeps this machine's part in the Slurm cluster run beside
// llmgw (BACKLOG P), as the leader says: slurmctld on the controller, slurmd
// on a node, munge on both, installed from the machine's own package sources.
//
// What the leader sends is wanted state, kept by Keeper's loop: sent again,
// or checked again a minute later, it changes nothing that is already so.
package slurmnode
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io/fs"
"log/slog"
"os"
"os/exec"
"os/user"
"path/filepath"
"slices"
"strconv"
"strings"
"sync"
"time"
"google.golang.org/protobuf/proto"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
// Host is what Keeper does to the machine; Local in the agent, a fake in
// tests.
type Host interface {
Run(ctx context.Context, env []string, name string, args ...string) ([]byte, error)
LookPath(name string) (string, error)
ReadFile(path string) ([]byte, error)
// WriteFile writes b at path with mode, owned by owner ("" for root),
// making its directory.
WriteFile(path string, b []byte, mode fs.FileMode, owner string) error
// MkdirAll makes path, owned by owner.
MkdirAll(path string, mode fs.FileMode, owner string) error
Remove(path string) error
}
// Installer installs packages by family, "deb" and "rpm".
type Installer interface {
InstallPackages(ctx context.Context, byFamily map[string][]string) error
}
// Reasons llmgw drains a node with; one drained for another reason, by a
// Slurm administrator, is never resumed by llmgw.
const (
ReasonServing = "llmgw: serving models"
reasonPrefix = "llmgw"
)
// DropIn is the unit drop-in that starts slurmd configless.
const DropIn = "/etc/systemd/system/slurmd.service.d/llmgw.conf"
// Keeper keeps the machine as the last Slurm message says.
type Keeper struct {
host Host
inst Installer
log *slog.Logger
marker string
mu sync.Mutex
want *llmgwv1.Slurm
kick chan struct{}
// resend has the next status sent even if unchanged: a new stream's
// control plane may never have had it.
resend bool
// installed is true once the packages were found or put there, so a
// minute's check does not look again.
installed map[string]bool
files string
}
// New is a Keeper; marker is a file it keeps while it runs Slurm here, so
// an agent started again knows to stop what it started when told it has no
// part any more.
func New(host Host, inst Installer, marker string, log *slog.Logger) *Keeper {
if log == nil {
log = slog.Default()
}
return &Keeper{host: host, inst: inst, log: log, marker: marker,
kick: make(chan struct{}, 1), installed: map[string]bool{}}
}
// Apply is a new message from the leader: kept from now on.
func (k *Keeper) Apply(m *llmgwv1.Slurm) {
k.mu.Lock()
k.want = proto.CloneOf(m)
k.mu.Unlock()
select {
case k.kick <- struct{}{}:
default:
}
}
// HasPart is whether the last message gives this machine a part in Slurm.
func (k *Keeper) HasPart() bool {
k.mu.Lock()
defer k.mu.Unlock()
return k.want.GetController() || k.want.GetNode()
}
// Resend has the status sent again on the next pass, which it starts now.
func (k *Keeper) Resend() {
k.mu.Lock()
k.resend = true
k.mu.Unlock()
select {
case k.kick <- struct{}{}:
default:
}
}
// Every is how often the machine is checked again without a new message.
var Every = time.Minute
// resendAfter is how often an unchanged status is sent anyway.
const resendAfter = 5 * time.Minute
// Run keeps the machine until ctx ends, sending its status when it changes
// and every few minutes while it has a part.
func (k *Keeper) Run(ctx context.Context, send func(*llmgwv1.SlurmStatus)) {
t := time.NewTicker(Every)
defer t.Stop()
var last *llmgwv1.SlurmStatus
var sent time.Time
for {
select {
case <-ctx.Done():
return
case <-k.kick:
case <-t.C:
}
k.mu.Lock()
want, again := k.want, k.resend
k.resend = false
k.mu.Unlock()
if want == nil {
continue
}
st := k.Reconcile(ctx, want)
if st == nil {
last = nil
continue
}
if again || !proto.Equal(st, last) || time.Since(sent) > resendAfter {
send(st)
last, sent = st, time.Now()
}
}
}
// Reconcile makes the machine what want says once, and answers its status;
// nil when it has no part in Slurm.
func (k *Keeper) Reconcile(ctx context.Context, want *llmgwv1.Slurm) *llmgwv1.SlurmStatus {
ctx, cancel := context.WithTimeout(ctx, 20*time.Minute)
defer cancel()
if !want.GetController() && !want.GetNode() {
k.stop(ctx)
return nil
}
st := &llmgwv1.SlurmStatus{}
fail := func(what string, err error) *llmgwv1.SlurmStatus {
st.Error = what + ": " + err.Error()
k.log.Warn("keeping Slurm on this node: "+what, "err", err)
return st
}
if err := k.ensurePackages(ctx, want); err != nil {
return fail("installing Slurm", err)
}
// From the daemon, not scontrol: a client command looks for its files
// even to say its version, and a configless node has none before slurmd
// has fetched them.
daemon := "slurmd"
if !want.GetNode() {
daemon = "slurmctld"
}
out, err := k.host.Run(ctx, nil, daemon, "-V")
if err != nil {
return fail("reading Slurm's version", fmt.Errorf("%v: %s", err, lastLine(out)))
}
v, err := slurm.ParseVersion(string(out))
if err != nil {
return fail("reading Slurm's version", err)
}
st.Version = strings.TrimSpace(string(out))
if i := strings.LastIndexByte(st.Version, ' '); i >= 0 {
st.Version = st.Version[i+1:]
}
if slurm.TooOld(v) {
return fail("Slurm", fmt.Errorf("%d.%02d is older than %d.%02d, the first to run jobs in cgroup v2; "+
"a newer release of the distribution, or Slurm's own packages, has one", v[0], v[1],
slurm.MinVersion[0], slurm.MinVersion[1]))
}
if err := k.host.WriteFile(k.marker, []byte("llmgw runs Slurm here (BACKLOG P)\n"), 0o644, ""); err != nil {
return fail("keeping its marker", err)
}
if err := k.munge(ctx, want.GetMungeKey()); err != nil {
return fail("munge", err)
}
if want.GetController() {
sum, err := k.controller(ctx, want.GetFiles())
st.FilesSha256 = sum
if err != nil {
return fail("slurmctld", err)
}
st.SlurmctldRunning = k.active(ctx, "slurmctld")
}
if want.GetNode() {
if err := k.node(ctx, want, st); err != nil {
return fail("slurmd", err)
}
}
return st
}
func (k *Keeper) ensurePackages(ctx context.Context, want *llmgwv1.Slurm) error {
need := map[string][2]string{
"munged": {"munge", "munge"},
"scontrol": {"slurm-client", "slurm"},
}
if want.GetController() {
need["slurmctld"] = [2]string{"slurmctld", "slurm-slurmctld"}
}
if want.GetNode() {
need["slurmd"] = [2]string{"slurmd", "slurm-slurmd"}
}
pkgs := map[string][]string{}
for bin, p := range need {
if k.installed[bin] {
continue
}
if _, err := k.host.LookPath(bin); err == nil {
k.installed[bin] = true
continue
}
pkgs["deb"] = append(pkgs["deb"], p[0])
pkgs["rpm"] = append(pkgs["rpm"], p[1])
}
if len(pkgs) == 0 {
return nil
}
slices.Sort(pkgs["deb"])
slices.Sort(pkgs["rpm"])
if k.inst == nil {
return fmt.Errorf("%s missing, and this agent installs nothing", strings.Join(pkgs["deb"], ", "))
}
k.log.Info("installing Slurm, as the cluster's settings say", "packages", pkgs["deb"])
if err := k.inst.InstallPackages(ctx, pkgs); err != nil {
return err
}
for bin := range need {
if _, err := k.host.LookPath(bin); err != nil {
return fmt.Errorf("%s is still missing after installing %s", bin, strings.Join(pkgs["deb"], ", "))
}
k.installed[bin] = true
}
return nil
}
// munge writes the cluster's key and keeps munged running on it.
func (k *Keeper) munge(ctx context.Context, key []byte) error {
if len(key) < 32 {
return errors.New("the leader sent no key")
}
changed, err := k.write(slurm.MungeKeyPath, key, 0o400, "munge")
if err != nil {
return err
}
if changed {
k.log.Info("munge key written; munge starts again on it")
return k.systemctl(ctx, "enable", "--now", "munge", "&&restart")
}
if !k.active(ctx, "munge") {
return k.systemctl(ctx, "enable", "--now", "munge")
}
return nil
}
// controller writes the files and keeps slurmctld running on them,
// answering their digest.
func (k *Keeper) controller(ctx context.Context, files map[string]string) (string, error) {
if files["slurm.conf"] == "" {
return "", errors.New("the leader sent no slurm.conf")
}
for _, d := range []string{slurm.StateDir, slurm.LogDir} {
if err := k.host.MkdirAll(d, 0o755, "slurm"); err != nil {
return "", err
}
}
names := make([]string, 0, len(files))
for n := range files {
names = append(names, n)
}
slices.Sort(names)
h := sha256.New()
changed := false
for _, n := range names {
if strings.ContainsAny(n, "/\\") || n == "" || n == "." || n == ".." {
return "", fmt.Errorf("the leader sent a file named %q", n)
}
fmt.Fprintf(h, "%s\x00%s\x00", n, files[n])
c, err := k.write(filepath.Join(slurm.ConfDir, n), []byte(files[n]), 0o644, "")
if err != nil {
return "", err
}
changed = changed || c
}
sum := hex.EncodeToString(h.Sum(nil))
if changed || sum != k.files {
if err := k.systemctl(ctx, "enable", "slurmctld"); err != nil {
return sum, err
}
// Nodes added or taken away need a restart before 23.11 and a
// reconfigure after it; a restart does both, and running jobs live
// through it.
k.log.Info("slurmctld starts again on new files")
if err := k.systemctl(ctx, "restart", "slurmctld"); err != nil {
return sum, err
}
if err := k.waitPing(ctx); err != nil {
return sum, err
}
// Hands the configless slurmd the new files.
if out, err := k.host.Run(ctx, nil, "scontrol", "reconfigure"); err != nil {
return sum, fmt.Errorf("scontrol reconfigure: %v: %s", err, lastLine(out))
}
k.files = sum
return sum, nil
}
if !k.active(ctx, "slurmctld") {
return sum, k.systemctl(ctx, "enable", "--now", "slurmctld")
}
return sum, nil
}
func (k *Keeper) waitPing(ctx context.Context) error {
var last []byte
for range 30 {
out, err := k.host.Run(ctx, nil, "scontrol", "ping")
if err == nil && bytes.Contains(out, []byte("UP")) {
return nil
}
last = out
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(pingWait):
}
}
return fmt.Errorf("slurmctld did not answer after starting: %s", lastLine(last))
}
var pingWait = time.Second
// node keeps slurmd running configless and puts the node in or out of the
// controller's hands as lent says.
func (k *Keeper) node(ctx context.Context, want *llmgwv1.Slurm, st *llmgwv1.SlurmStatus) error {
name, server := want.GetName(), want.GetConfServer()
if name == "" || server == "" {
return errors.New("the leader sent no name or controller")
}
if strings.ContainsAny(name+server, " \n\t\"'\\$`") {
return fmt.Errorf("the leader sent a name or controller with spaces or quotes in it")
}
for _, d := range []string{slurm.SpoolDir, slurm.LogDir} {
if err := k.host.MkdirAll(d, 0o755, "slurm"); err != nil {
return err
}
}
bin, err := k.host.LookPath("slurmd")
if err != nil {
return err
}
unit := "# Written by llmgw (BACKLOG P): slurmd fetches its files from slurmctld.\n" +
"[Unit]\nConditionPathExists=\nStartLimitIntervalSec=0\n" +
// --systemd as the packaged unit has it (Type=notify, since 22.05):
// without it systemd waits for a readiness that never comes.
"[Service]\nExecStart=\nExecStart=" + bin + " --systemd --conf-server " + server + " -N " + name + "\n" +
// A node new to the controller is refused until the leader has
// written it into slurm.conf; tried again until it is.
"Restart=on-failure\nRestartSec=15\n"
changed, err := k.write(DropIn, []byte(unit), 0o644, "")
if err != nil {
return err
}
if out, err := k.host.Run(ctx, nil, "slurmd", "-C"); err == nil {
st.Hardware = strings.TrimSpace(string(out))
}
if changed {
if err := k.systemctl(ctx, "daemon-reload"); err != nil {
return err
}
if err := k.systemctl(ctx, "enable", "slurmd"); err != nil {
return err
}
if err := k.systemctl(ctx, "restart", "slurmd"); err != nil {
return err
}
} else if !k.active(ctx, "slurmd") {
if err := k.systemctl(ctx, "enable", "--now", "slurmd"); err != nil {
return err
}
}
st.SlurmdRunning = k.active(ctx, "slurmd")
st.Jobs = k.localJobs(ctx)
env := k.clientEnv()
out, err := k.host.Run(ctx, env, "scontrol", "show", "node", "-o", name)
if err != nil {
// Not in slurm.conf yet: the leader writes it once this node's
// hardware is in.
return nil
}
ns, err := slurm.ParseNodeState(string(out))
if err != nil {
return nil
}
ours := ns.Reason == "" || strings.HasPrefix(ns.Reason, reasonPrefix)
switch {
case want.GetLent() && ns.Drained() && ours:
k.log.Info("this node is lent to Slurm: resumed there")
if out, err := k.host.Run(ctx, env, "scontrol", "update", "NodeName="+name, "State=RESUME"); err != nil {
return fmt.Errorf("resuming the node: %v: %s", err, lastLine(out))
}
case !want.GetLent() && !strings.Contains(ns.State, "DRAIN"):
k.log.Info("this node is llmgw's: drained in Slurm, its jobs left to finish")
if out, err := k.host.Run(ctx, env, "scontrol", "update", "NodeName="+name, "State=DRAIN",
"Reason="+ReasonServing); err != nil {
return fmt.Errorf("draining the node: %v: %s", err, lastLine(out))
}
}
if out, err := k.host.Run(ctx, env, "scontrol", "show", "node", "-o", name); err == nil {
if ns2, err := slurm.ParseNodeState(string(out)); err == nil {
ns = ns2
}
}
st.State, st.Reason = ns.State, ns.Reason
return nil
}
// localJobs counts the job steps running here, from their slurmstepd, not
// from the controller: a job outlives a controller that is down, and is
// what decides whether llmgw may have the cards back. The one slurmstepd
// that is no job's, "slurmstepd infinity" holding slurmd's cgroup v2 scope,
// is not counted. -1 when it cannot tell.
func (k *Keeper) localJobs(ctx context.Context) int32 {
// pgrep exits 1 with no output when there is none.
out, err := k.host.Run(ctx, nil, "pgrep", "-a", "-x", "slurmstepd")
if err != nil && len(bytes.TrimSpace(out)) > 0 {
return -1
}
n := int32(0)
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
f := strings.Fields(line)
if len(f) < 2 || f[len(f)-1] == "infinity" {
continue
}
n++
}
return n
}
// clientEnv points scontrol at slurmd's copy of the files on a node that is
// not the controller, which has none of its own.
func (k *Keeper) clientEnv() []string {
if _, err := k.host.ReadFile(filepath.Join(slurm.ConfDir, "slurm.conf")); err == nil {
return nil
}
return []string{"SLURM_CONF=" + filepath.Join(slurm.SpoolDir, "conf-cache", "slurm.conf")}
}
// stop stops what llmgw started here, if it did.
func (k *Keeper) stop(ctx context.Context) {
if _, err := k.host.ReadFile(k.marker); err != nil {
return
}
k.log.Info("this node has no part in Slurm any more: slurmd and slurmctld stopped; the packages stay")
for _, u := range []string{"slurmd", "slurmctld"} {
_ = k.systemctl(ctx, "disable", "--now", u)
}
_ = k.host.Remove(DropIn)
_ = k.systemctl(ctx, "daemon-reload")
_ = k.host.Remove(k.marker)
k.files = ""
}
// write writes a file when it differs, saying whether it did.
func (k *Keeper) write(path string, b []byte, mode fs.FileMode, owner string) (bool, error) {
if old, err := k.host.ReadFile(path); err == nil && bytes.Equal(old, b) {
return false, nil
}
return true, k.host.WriteFile(path, b, mode, owner)
}
// systemctl runs systemctl; a last argument "&&restart" restarts the unit
// after.
func (k *Keeper) systemctl(ctx context.Context, args ...string) error {
restart := false
if n := len(args); n > 0 && args[n-1] == "&&restart" {
args, restart = args[:n-1], true
}
if out, err := k.host.Run(ctx, nil, "systemctl", args...); err != nil {
return fmt.Errorf("systemctl %s: %v: %s", strings.Join(args, " "), err, lastLine(out))
}
if restart {
u := args[len(args)-1]
if out, err := k.host.Run(ctx, nil, "systemctl", "restart", u); err != nil {
return fmt.Errorf("systemctl restart %s: %v: %s", u, err, lastLine(out))
}
}
return nil
}
func (k *Keeper) active(ctx context.Context, unit string) bool {
out, err := k.host.Run(ctx, nil, "systemctl", "is-active", unit)
return err == nil && strings.TrimSpace(string(out)) == "active"
}
func lastLine(b []byte) string {
s := strings.TrimSpace(string(b))
if i := strings.LastIndexByte(s, '\n'); i >= 0 {
s = s[i+1:]
}
return s
}
// Local is the machine the agent runs on.
type Local struct{}
// Run runs a command, with env added to the agent's.
func (Local) Run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
cmd := exec.CommandContext(ctx, name, args...)
if len(env) > 0 {
cmd.Env = append(os.Environ(), env...)
}
return cmd.CombinedOutput()
}
// LookPath finds a command, in the sbin directories too: an agent started
// by systemd has them, one started by hand may not.
func (Local) LookPath(name string) (string, error) {
if p, err := exec.LookPath(name); err == nil {
return p, nil
}
for _, d := range []string{"/usr/sbin", "/sbin", "/usr/local/sbin"} {
p := filepath.Join(d, name)
if fi, err := os.Stat(p); err == nil && !fi.IsDir() {
return p, nil
}
}
return "", fmt.Errorf("%s is not installed", name)
}
// ReadFile reads a file.
func (Local) ReadFile(path string) ([]byte, error) { return os.ReadFile(path) } //nolint:gosec // our paths
// WriteFile writes a file whole, through a temporary one beside it.
func (Local) WriteFile(path string, b []byte, mode fs.FileMode, owner string) error {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { //nolint:gosec // system directories
return err
}
tmp := path + ".llmgw-new"
if err := os.WriteFile(tmp, b, mode); err != nil {
return err
}
if err := os.Chmod(tmp, mode); err != nil {
return err
}
if err := chown(tmp, owner); err != nil {
_ = os.Remove(tmp)
return err
}
return os.Rename(tmp, path)
}
// MkdirAll makes a directory owned by owner.
func (Local) MkdirAll(path string, mode fs.FileMode, owner string) error {
if err := os.MkdirAll(path, mode); err != nil {
return err
}
return chown(path, owner)
}
// Remove removes a file, and is content when there is none.
func (Local) Remove(path string) error {
if err := os.Remove(path); err != nil && !errors.Is(err, fs.ErrNotExist) {
return err
}
return nil
}
func chown(path, owner string) error {
if owner == "" {
return nil
}
u, err := user.Lookup(owner)
if err != nil {
return fmt.Errorf("user %s: %w (installed with its package)", owner, err)
}
uid, _ := strconv.Atoi(u.Uid)
gid, _ := strconv.Atoi(u.Gid)
return os.Chown(path, uid, gid)
}
+308
View File
@@ -0,0 +1,308 @@
package slurmnode
import (
"context"
"errors"
"io/fs"
"path/filepath"
"slices"
"strings"
"testing"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
type fakeHost struct {
files map[string][]byte
owners map[string]string
bins map[string]bool
active map[string]bool
ran []string
outputs map[string]string
fail map[string]bool
state string
reason string
}
func newFake() *fakeHost {
return &fakeHost{files: map[string][]byte{}, owners: map[string]string{}, bins: map[string]bool{},
active: map[string]bool{}, outputs: map[string]string{
"slurmd -V": "slurm-wlm 23.11.4\n",
"slurmctld -V": "slurm-wlm 23.11.4\n",
"scontrol ping": "Slurmctld(primary) at node06 is UP\n",
"slurmd -C": "NodeName=cuda01 CPUs=16 RealMemory=64000\nUpTime=1-00:00:00\n",
}, fail: map[string]bool{}, state: "IDLE+DRAIN", reason: slurm.ReasonNew}
}
func (f *fakeHost) Run(_ context.Context, env []string, name string, args ...string) ([]byte, error) {
c := strings.Join(append([]string{name}, args...), " ")
f.ran = append(f.ran, c)
if f.fail[c] {
return []byte("it failed"), errors.New("exit status 1")
}
switch {
case name == "systemctl" && len(args) >= 1:
u := args[len(args)-1]
switch args[0] {
case "is-active":
if f.active[u] {
return []byte("active\n"), nil
}
return []byte("inactive\n"), errors.New("exit status 3")
case "restart", "start":
f.active[u] = true
case "enable":
if slices.Contains(args, "--now") {
f.active[u] = true
}
case "disable":
f.active[u] = false
}
return nil, nil
case strings.HasPrefix(c, "scontrol show node"):
return []byte("NodeName=cuda01 Arch=x86_64 State=" + f.state + " Partitions=gpu Reason=" + f.reason + " [root@now]\n"), nil
case strings.HasPrefix(c, "scontrol update"):
if slices.Contains(args, "State=RESUME") {
f.state, f.reason = "IDLE", ""
} else {
f.state, f.reason = "IDLE+DRAIN", ReasonServing
}
return nil, nil
case strings.HasPrefix(c, "pgrep"):
out := "1800358 /usr/sbin/slurmstepd infinity\n" + f.outputs["jobs"]
return []byte(out), nil
}
return []byte(f.outputs[c]), nil
}
func (f *fakeHost) LookPath(name string) (string, error) {
if f.bins[name] {
return "/usr/sbin/" + name, nil
}
return "", errors.New("not found")
}
func (f *fakeHost) ReadFile(path string) ([]byte, error) {
b, ok := f.files[path]
if !ok {
return nil, fs.ErrNotExist
}
return b, nil
}
func (f *fakeHost) WriteFile(path string, b []byte, _ fs.FileMode, owner string) error {
f.files[path] = append([]byte(nil), b...)
f.owners[path] = owner
return nil
}
func (f *fakeHost) MkdirAll(string, fs.FileMode, string) error { return nil }
func (f *fakeHost) Remove(path string) error { delete(f.files, path); return nil }
func (f *fakeHost) count(prefix string) int {
n := 0
for _, c := range f.ran {
if strings.HasPrefix(c, prefix) {
n++
}
}
return n
}
type fakeInst struct {
host *fakeHost
got map[string][]string
}
func (i *fakeInst) InstallPackages(_ context.Context, by map[string][]string) error {
i.got = by
for _, b := range []string{"munged", "scontrol", "slurmd", "slurmctld"} {
i.host.bins[b] = true
}
return nil
}
var key = []byte(strings.Repeat("k", 1024))
func nodeMsg(lent bool) *llmgwv1.Slurm {
return &llmgwv1.Slurm{Node: true, Name: "cuda01", ConfServer: "10.0.0.146:6817", MungeKey: key, Lent: lent}
}
func TestNodeInstallsAndStaysDrained(t *testing.T) {
h := newFake()
inst := &fakeInst{host: h}
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
st := k.Reconcile(context.Background(), nodeMsg(false))
if st.GetError() != "" {
t.Fatal(st.GetError())
}
if !slices.Equal(inst.got["deb"], []string{"munge", "slurm-client", "slurmd"}) ||
!slices.Equal(inst.got["rpm"], []string{"munge", "slurm", "slurm-slurmd"}) {
t.Fatalf("installed %v", inst.got)
}
if string(h.files[slurm.MungeKeyPath]) != string(key) || h.owners[slurm.MungeKeyPath] != "munge" {
t.Fatal("the munge key was not written for munge")
}
unit := string(h.files[DropIn])
if !strings.Contains(unit, "ExecStart=/usr/sbin/slurmd --systemd --conf-server 10.0.0.146:6817 -N cuda01\n") ||
!strings.Contains(unit, "ConditionPathExists=\n") {
t.Fatalf("drop-in:\n%s", unit)
}
if !st.GetSlurmdRunning() || st.GetVersion() != "23.11.4" || !strings.HasPrefix(st.GetHardware(), "NodeName=cuda01 CPUs=16") {
t.Fatalf("status %v", st)
}
// Not lent: left drained, with llmgw's first reason.
if h.count("scontrol update") != 0 || st.GetState() != "IDLE+DRAIN" {
t.Fatalf("a node not lent was changed: %v, %v", h.ran, st)
}
if _, ok := h.files["/var/lib/llmgw-agent/slurm"]; !ok {
t.Fatal("no marker")
}
// Again, nothing changed: no install, no restart.
h.ran = nil
inst.got = nil
k.Reconcile(context.Background(), nodeMsg(false))
if inst.got != nil || h.count("systemctl restart") != 0 || h.count("systemctl enable") != 0 {
t.Fatalf("a second pass did work: %v", h.ran)
}
}
func TestLendAndReclaim(t *testing.T) {
h := newFake()
k := New(h, &fakeInst{host: h}, "/m", nil)
st := k.Reconcile(context.Background(), nodeMsg(true))
if h.count("scontrol update NodeName=cuda01 State=RESUME") != 1 || st.GetState() != "IDLE" {
t.Fatalf("a lent node was not resumed: %v %v", h.ran, st)
}
h.outputs["jobs"] = "2001 slurmstepd: [101.batch]\n2002 slurmstepd: [102.0]\n"
st = k.Reconcile(context.Background(), nodeMsg(false))
if h.count("scontrol update NodeName=cuda01 State=DRAIN Reason="+ReasonServing) != 1 {
t.Fatalf("a node taken back was not drained: %v", h.ran)
}
if st.GetJobs() != 2 || !strings.Contains(st.GetState(), "DRAIN") {
t.Fatalf("status %v", st)
}
// Drained already: not drained again.
h.ran = nil
k.Reconcile(context.Background(), nodeMsg(false))
if h.count("scontrol update") != 0 {
t.Fatalf("drained twice: %v", h.ran)
}
}
func TestAdminsDrainIsLeft(t *testing.T) {
h := newFake()
h.reason = "bad DIMM, ticket 42"
k := New(h, &fakeInst{host: h}, "/m", nil)
k.Reconcile(context.Background(), nodeMsg(true))
if h.count("scontrol update") != 0 {
t.Fatalf("llmgw resumed a node a Slurm administrator drained: %v", h.ran)
}
}
func TestClientEnvOnANodeWithoutFiles(t *testing.T) {
h := newFake()
k := New(h, &fakeInst{host: h}, "/m", nil)
if env := k.clientEnv(); len(env) != 1 || !strings.HasSuffix(env[0], "conf-cache/slurm.conf") {
t.Fatalf("env %v", env)
}
h.files[filepath.Join(slurm.ConfDir, "slurm.conf")] = []byte("x")
if env := k.clientEnv(); env != nil {
t.Fatalf("env %v on the controller", env)
}
}
func TestController(t *testing.T) {
h := newFake()
inst := &fakeInst{host: h}
k := New(h, inst, "/m", nil)
files := map[string]string{"slurm.conf": "ClusterName=llmgw\n", "gres.conf": "#\n", "cgroup.conf": "#\n"}
m := &llmgwv1.Slurm{Controller: true, MungeKey: key, Files: files}
st := k.Reconcile(context.Background(), m)
if st.GetError() != "" {
t.Fatal(st.GetError())
}
if !slices.Contains(inst.got["deb"], "slurmctld") || slices.Contains(inst.got["deb"], "slurmd") {
t.Fatalf("installed %v", inst.got)
}
if string(h.files["/etc/slurm/slurm.conf"]) != "ClusterName=llmgw\n" {
t.Fatal("slurm.conf not written")
}
if h.count("systemctl restart slurmctld") != 1 || h.count("scontrol reconfigure") != 1 {
t.Fatalf("slurmctld not started on its files: %v", h.ran)
}
if !st.GetSlurmctldRunning() || len(st.GetFilesSha256()) != 64 {
t.Fatalf("status %v", st)
}
sum := st.GetFilesSha256()
h.ran = nil
k.Reconcile(context.Background(), m)
if h.count("systemctl restart") != 0 {
t.Fatalf("restarted on the same files: %v", h.ran)
}
files["gres.conf"] = "NodeName=cuda01 Name=gpu File=/dev/nvidia0\n"
h.ran = nil
st = k.Reconcile(context.Background(), m)
if h.count("systemctl restart slurmctld") != 1 || st.GetFilesSha256() == sum {
t.Fatalf("a changed file did not restart slurmctld: %v", h.ran)
}
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Controller: true, MungeKey: key,
Files: map[string]string{"slurm.conf": "x", "../evil": "y"}}); !strings.Contains(st.GetError(), "named") {
t.Fatalf("a path in a file name was taken: %v", st)
}
}
func TestTooOld(t *testing.T) {
h := newFake()
h.outputs["slurmd -V"] = "slurm-wlm 21.08.5\n"
k := New(h, &fakeInst{host: h}, "/m", nil)
st := k.Reconcile(context.Background(), nodeMsg(false))
if !strings.Contains(st.GetError(), "older than 22.05") {
t.Fatalf("status %v", st)
}
if _, ok := h.files[slurm.MungeKeyPath]; ok {
t.Fatal("went on with a Slurm too old")
}
}
func TestStop(t *testing.T) {
h := newFake()
k := New(h, &fakeInst{host: h}, "/m", nil)
k.Reconcile(context.Background(), nodeMsg(false))
h.ran = nil
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{}); st != nil {
t.Fatalf("status %v for no part", st)
}
if h.count("systemctl disable --now slurmd") != 1 {
t.Fatalf("slurmd not stopped: %v", h.ran)
}
if _, ok := h.files[DropIn]; ok {
t.Fatal("drop-in left")
}
// Never started here: nothing touched.
h2 := newFake()
k2 := New(h2, &fakeInst{host: h2}, "/m", nil)
k2.Reconcile(context.Background(), &llmgwv1.Slurm{})
if len(h2.ran) != 0 {
t.Fatalf("a machine llmgw never ran Slurm on was touched: %v", h2.ran)
}
}
func TestBadNames(t *testing.T) {
h := newFake()
k := New(h, &fakeInst{host: h}, "/m", nil)
m := nodeMsg(false)
m.Name = "cuda01 -c /etc/evil"
if st := k.Reconcile(context.Background(), m); !strings.Contains(st.GetError(), "spaces") {
t.Fatalf("status %v", st)
}
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Node: true, Name: "a", ConfServer: "b"}); !strings.Contains(st.GetError(), "no key") {
t.Fatalf("status %v", st)
}
}
+12 -1
View File
@@ -33,6 +33,17 @@ func CachePathFor(dir, source string) string {
return filepath.Join(dir, hex.EncodeToString(sum[:8])+"-"+name)
}
// cacheSource is the source a cache name is computed from: a set's address
// without its trailing slash, as its directory was named when it was fetched.
// Computed from the slash-ended address, every set in the catalog looked like
// nobody's, and a sweep offered 330 GB of models still in use for deletion.
func cacheSource(source string) string {
if IsSet(source) {
return strings.TrimSuffix(source, "/")
}
return source
}
// Owned reports whether a path is one this package created.
//
// The dividing line of the whole feature. A file without the cache name was
@@ -65,7 +76,7 @@ func Orphans(have []string, wanted []string) []string {
// By base name rather than full path: the control plane does not know
// each node's cache directory, and the name is what identifies the
// file wherever the node keeps it.
keep[filepath.Base(CachePathFor("", src))] = struct{}{}
keep[filepath.Base(CachePathFor("", cacheSource(src)))] = struct{}{}
}
var out []string
+1 -1
View File
@@ -51,7 +51,7 @@ type manifest struct {
// as a broken model rather than failing to load, so a partial set is left
// unfinished under its temporary name and the next attempt starts again.
func (f *Fetcher) ensureSet(ctx context.Context, source string, allowPickle bool) (string, error) {
dir, err := f.cachePath(strings.TrimSuffix(source, "/"))
dir, err := f.cachePath(cacheSource(source))
if err != nil {
return "", err
}
+17
View File
@@ -74,6 +74,23 @@ func TestASetIsFetchedIntoOneDirectory(t *testing.T) {
}
}
func TestASetInTheCatalogIsNotAnOrphan(t *testing.T) {
// The catalog names a set by its address with the trailing slash; the
// fetch names the directory without it. A sweep that computed the name
// from the catalog's spelling offered every set in use for deletion.
f, srv, _ := mirrorOf(t, map[string]string{"model.safetensors": "aaaa", "config.json": "{}"})
dir, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false)
if err != nil {
t.Fatal(err)
}
if got := Orphans([]string{dir}, []string{srv.URL + "/m/v/"}); len(got) != 0 {
t.Errorf("a set the catalog still names was called an orphan: %v", got)
}
if got := Orphans([]string{dir}, nil); len(got) != 1 {
t.Errorf("a set nothing names was not found: %v", got)
}
}
func TestAFinishedSetIsNotFetchedAgain(t *testing.T) {
f, srv, hits := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
+2
View File
@@ -247,6 +247,8 @@ var liveSettings = []string{
"presets",
// Handed to the reconciler as they change.
"on_demand",
// Read by the Slurm service each pass, kicked when they change.
"slurm",
// Read by server init, node add and backups, not by the server.
"backup_recipients", "backup_copy_to",
}
+15 -1
View File
@@ -189,6 +189,7 @@ func newCtlCmd() *cobra.Command {
newMCPCmd(client, &asJSON),
newKBCmd(client, &asJSON),
newTrainCmd(client, &asJSON),
newSlurmCmd(client, &asJSON),
newConfigCmd(client, &asJSON),
newAuditCmd(client, &asJSON),
newNotifyCmd(client, &asJSON),
@@ -418,7 +419,7 @@ func newModelsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
}
fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\t%s\t%s\n",
name, serving, m.Ready, m.Min, modelStatus(m), m.Class, reserves(m),
join(m.Aliases), join(m.Profiles), join(m.Fallbacks))
join(m.Aliases), join(profilesWithCounts(m)), join(m.Fallbacks))
for _, x := range m.Warnings {
warnings = append(warnings, m.Name+": "+x)
}
@@ -696,3 +697,16 @@ func reserves(m adminapi.ModelView) string {
}
return out
}
// profilesWithCounts are a model's profiles, each with its replicas when
// they are counted per profile: gpu=7, cpu=4.
func profilesWithCounts(m adminapi.ModelView) []string {
if len(m.ProfileReplicas) == 0 {
return m.Profiles
}
out := make([]string, len(m.Profiles))
for i, p := range m.Profiles {
out[i] = fmt.Sprintf("%s=%d", p, m.ProfileReplicas[p])
}
return out
}
+242
View File
@@ -0,0 +1,242 @@
package cli
import (
"encoding/json"
"fmt"
"slices"
"strings"
"text/tabwriter"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
)
// newSlurmCmd is the Slurm cluster run beside llmgw (BACKLOG P): set up on
// nodes llmgw already has, and nodes handed between the two.
func newSlurmCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
cmd := &cobra.Command{
Use: "slurm",
Short: "A Slurm cluster beside llmgw, for training and other batch work",
Long: "Slurm runs on llmgw's own nodes, installed and kept by their agents: slurmctld on one node,\n" +
"slurmd on those named. A node is llmgw's, drained in Slurm, until it is lent; lent, its\n" +
"replicas move off first, then Slurm may run jobs there. Taken back, Slurm drains it and\n" +
"llmgw has it again when its last job ends. Jobs are submitted on the controller node\n" +
"(sbatch, srun) by its own users, as on any Slurm cluster.",
}
cmd.AddCommand(newSlurmStatusCmd(client, asJSON), newSlurmSetupCmd(client), newSlurmOffCmd(client),
newSlurmListCmd(client, "lend", "Let Slurm run jobs on nodes; llmgw's replicas move off first", true),
newSlurmListCmd(client, "reclaim", "Take nodes back from Slurm; llmgw has them when their jobs end", false))
return cmd
}
func newSlurmStatusCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
return &cobra.Command{
Use: "status",
Short: "The Slurm cluster, and where each of its nodes is",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
var st slurmsvc.Status
if err := client().get(cmd.Context(), "/api/v1/slurm", &st); err != nil {
return err
}
w := cmd.OutOrStdout()
if *asJSON {
return json.NewEncoder(w).Encode(st)
}
if !st.Enabled {
fmt.Fprintln(w, "Slurm is off. `llmgw ctl slurm setup --controller NODE --nodes NODE,...` turns it on.")
return nil
}
fmt.Fprintf(w, "cluster %s, partition %s, controller %s", st.Cluster, st.Partition, st.Controller)
switch c := st.Ctld; {
case c == nil:
fmt.Fprintln(w, ": not heard from yet")
case c.Error != "":
fmt.Fprintf(w, ": %s\n", c.Error)
case c.SlurmctldRunning:
fmt.Fprintf(w, ": slurmctld %s running\n", c.Version)
default:
fmt.Fprintln(w, ": slurmctld not running")
}
tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
fmt.Fprintln(tw, "NODE\tPHASE\tSLURM\tJOBS\tSLURMD\tNOTE")
for _, n := range st.Nodes {
state, jobs, slurmd, note := "-", "-", "-", n.Describe()
if r := n.Status; r != nil {
state = or(r.State, "-")
jobs = fmt.Sprint(r.Jobs)
slurmd = map[bool]string{true: "running", false: "stopped"}[r.SlurmdRunning]
if r.Error != "" {
note = r.Error
}
}
if !n.Connected {
note = "agent not connected; " + note
}
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\t%s\n", n.Node, n.Phase, state, jobs, slurmd, note)
}
if err := tw.Flush(); err != nil {
return err
}
if len(st.Waiting) > 0 {
fmt.Fprintf(w, "not in slurm.conf until they report their hardware: %s\n", strings.Join(st.Waiting, ", "))
}
return nil
},
}
}
func newSlurmSetupCmd(client func() *ctlClient) *cobra.Command {
var controller, cluster, partition, note string
var nodes []string
var all bool
c := &cobra.Command{
Use: "setup --controller NODE --nodes NODE,... | --all-nodes",
Short: "Run Slurm on llmgw's nodes: slurmctld on one, slurmd on those named",
Long: "Turns slurm.enabled on and names the nodes, or with --all-nodes every node. Their agents install munge and Slurm (22.05 or\n" +
"newer) from the machines' own package sources. Every node starts drained in Slurm and stays\n" +
"llmgw's until `llmgw ctl slurm lend` gives it to Slurm.",
Example: " llmgw ctl slurm setup --controller node06 --nodes cuda01,cuda02\n" +
" llmgw ctl slurm setup --controller node06 --all-nodes",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
if controller == "" || (len(nodes) == 0 && !all) {
return fmt.Errorf("give --controller, and --nodes or --all-nodes")
}
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
if err := doc.SetYAML([]string{"slurm", "enabled"}, "true"); err != nil {
return err
}
if err := doc.SetYAML([]string{"slurm", "controller"}, yamlString(controller)); err != nil {
return err
}
if err := setList(doc, []string{"slurm", "nodes"}, nodes); err != nil {
return err
}
if err := doc.SetYAML([]string{"slurm", "all_nodes"}, fmt.Sprint(all)); err != nil {
return err
}
// Lent nodes no longer named are not lent.
lent := slices.DeleteFunc(docList(doc, []string{"slurm", "lent"}),
func(n string) bool { return !all && !slices.Contains(nodes, n) })
if err := setList(doc, []string{"slurm", "lent"}, lent); err != nil {
return err
}
for k, v := range map[string]string{"cluster_name": cluster, "partition": partition} {
if v != "" {
if err := doc.SetYAML([]string{"slurm", k}, yamlString(v)); err != nil {
return err
}
}
}
return nil
})
},
}
c.Flags().StringVar(&controller, "controller", "", "the node that runs slurmctld")
c.Flags().StringSliceVar(&nodes, "nodes", nil, "the nodes that run slurmd")
c.Flags().BoolVar(&all, "all-nodes", false, "every node runs slurmd, those that join later too")
c.Flags().StringVar(&cluster, "cluster-name", "", "Slurm's ClusterName (llmgw)")
c.Flags().StringVar(&partition, "partition", "", "the partition every node is in (gpu)")
c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
return c
}
func newSlurmOffCmd(client func() *ctlClient) *cobra.Command {
var note string
var force bool
c := &cobra.Command{
Use: "off",
Short: "Stop running Slurm; every node is llmgw's again",
Long: "slurmd and slurmctld are stopped where llmgw started them; the packages stay. A node still\n" +
"lent is refused unless --force: take it back first, so its jobs end before the daemons do.",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
if lent := docList(doc, []string{"slurm", "lent"}); len(lent) > 0 && !force {
return fmt.Errorf("%s still lent: `llmgw ctl slurm reclaim` them first, or --force",
strings.Join(lent, ", "))
}
setList(doc, []string{"slurm", "lent"}, nil) //nolint:errcheck // an empty list
return doc.SetYAML([]string{"slurm", "enabled"}, "false")
})
},
}
c.Flags().BoolVar(&force, "force", false, "even with nodes still lent")
c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
return c
}
// newSlurmListCmd adds nodes to slurm.lent, or takes them out.
func newSlurmListCmd(client func() *ctlClient, use, short string, add bool) *cobra.Command {
var note string
c := &cobra.Command{
Use: use + " NODE...",
Short: short,
Args: cobra.MinimumNArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
nodes := docList(doc, []string{"slurm", "nodes"})
all := docBool(doc, []string{"slurm", "all_nodes"})
lent := docList(doc, []string{"slurm", "lent"})
for _, n := range args {
switch {
case add && !all && !slices.Contains(nodes, n):
return fmt.Errorf("%s does not run slurmd: add it with `llmgw ctl slurm setup --nodes`", n)
case add && !slices.Contains(lent, n):
lent = append(lent, n)
case !add:
lent = slices.DeleteFunc(lent, func(l string) bool { return l == n })
}
}
return setList(doc, []string{"slurm", "lent"}, lent)
})
},
}
c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
return c
}
// docList reads a list of strings from the settings, empty when unset.
func docList(doc *settings.Doc, path []string) []string {
n := doc.Node(path)
if n == nil {
return nil
}
var out []string
if err := n.Decode(&out); err != nil {
return nil
}
return out
}
// docBool reads a boolean from the settings, false when unset.
func docBool(doc *settings.Doc, path []string) bool {
var b bool
if n := doc.Node(path); n != nil {
_ = n.Decode(&b)
}
return b
}
func setList(doc *settings.Doc, path []string, list []string) error {
if len(list) == 0 {
if doc.Node(path) != nil {
doc.Unset(path)
}
return nil
}
return doc.SetYAML(path, yamlList(list))
}
func yamlString(s string) string {
b, _ := json.Marshal(s)
return string(b)
}
func yamlList(list []string) string {
b, _ := json.Marshal(list)
return string(b)
}
+35
View File
@@ -0,0 +1,35 @@
package cli
import (
"slices"
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
)
func TestSlurmDocLists(t *testing.T) {
doc, err := settings.FromBytes("settings", []byte("slurm:\n enabled: true\n nodes: [cuda01, site2]\n"))
if err != nil {
t.Fatal(err)
}
if got := docList(doc, []string{"slurm", "nodes"}); !slices.Equal(got, []string{"cuda01", "site2"}) {
t.Fatalf("nodes %v", got)
}
if got := docList(doc, []string{"slurm", "lent"}); got != nil {
t.Fatalf("lent %v", got)
}
if err := setList(doc, []string{"slurm", "lent"}, []string{"cuda01"}); err != nil {
t.Fatal(err)
}
if got := docList(doc, []string{"slurm", "lent"}); !slices.Equal(got, []string{"cuda01"}) {
t.Fatalf("lent %v", got)
}
if err := setList(doc, []string{"slurm", "lent"}, nil); err != nil {
t.Fatal(err)
}
b, _ := doc.Bytes()
if strings.Contains(string(b), "lent") {
t.Fatalf("an empty list left:\n%s", b)
}
}
+19 -15
View File
@@ -4,6 +4,7 @@ import (
"encoding/json"
"fmt"
"strings"
"time"
"github.com/spf13/cobra"
@@ -87,13 +88,10 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
if err != nil {
return err
}
var results []struct {
Node string `json:"node"`
Files int `json:"files"`
Bytes int64 `json:"bytes"`
Error string `json:"error,omitempty"`
}
if err := c.post(cmd.Context(), "/api/v1/weights/collect", body, &results); err != nil {
var results []adminapi.CollectResult
// The control plane waits for the nodes to list their files without the
// deleted ones: half a minute at most.
if err := c.patient(time.Minute).post(cmd.Context(), "/api/v1/weights/collect", body, &results); err != nil {
return err
}
if asJSON {
@@ -103,7 +101,7 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
fmt.Fprintln(cmd.OutOrStdout(), "Nothing to reclaim.")
return nil
}
var freed int64
var freed, left int64
for _, r := range results {
if r.Error != "" {
// Reported rather than fatal: the other nodes' space is still worth
@@ -111,14 +109,20 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
fmt.Fprintf(cmd.OutOrStdout(), "%s: %s\n", r.Node, r.Error)
continue
}
fmt.Fprintf(cmd.OutOrStdout(), "%s: asked to remove %d files, %s\n",
r.Node, r.Files, fileSize(r.Bytes))
freed += r.Bytes
line := fmt.Sprintf("%s: removed %d files, %s", r.Node, r.Files, fileSize(r.Bytes))
if r.Left > 0 {
line += fmt.Sprintf("; %d still there, %s", r.Left, fileSize(r.LeftBytes))
}
fmt.Fprintln(cmd.OutOrStdout(), line)
freed, left = freed+r.Bytes, left+r.LeftBytes
}
fmt.Fprintf(cmd.OutOrStdout(), "\n%s freed.\n", fileSize(freed))
if left > 0 {
// Each node decides: one that keeps a path it cannot prove it
// fetched is the feature working. Its agent's log names each.
fmt.Fprintf(cmd.OutOrStdout(), "%s the nodes kept; each node's agent log says why "+
"(journalctl -u llmgw-agent).\n", fileSize(left))
}
// "asked", not "removed": each node decides, and one that refuses a path it
// cannot prove it fetched is the feature working rather than failing.
fmt.Fprintf(cmd.OutOrStdout(), "\n%s requested. The nodes verify each path before deleting it.\n",
fileSize(freed))
return nil
}
+49 -1
View File
@@ -19,6 +19,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
)
@@ -34,7 +35,7 @@ func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
Use: "model",
Short: "Add or remove models while the cluster is running",
}
c.AddCommand(newModelStopCmd(client), newModelStartCmd(client), newModelContextCmd(client), newModelGPUsCmd(client), newModelAddCmd(client), newModelPresetsCmd(client, asJSON), newModelRemoveCmd(client), newModelPlanCmd(client),
c.AddCommand(newModelStopCmd(client), newModelStartCmd(client), newModelReplicasCmd(client), newModelContextCmd(client), newModelGPUsCmd(client), newModelAddCmd(client), newModelPresetsCmd(client, asJSON), newModelRemoveCmd(client), newModelPlanCmd(client),
newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
newTargetCmd(client, asJSON), newAdaptersCmd(client),
newModelServeCmd(client), newModelUpdatesCmd(client, asJSON), newModelUpdateCmd(client),
@@ -598,3 +599,50 @@ func newModelGPUsCmd(client func() *ctlClient) *cobra.Command {
c.Flags().BoolVar(&inPlace, "in-place", false, "stop each replica before its replacement starts, when there is no room beside it")
return c
}
func newModelReplicasCmd(client func() *ctlClient) *cobra.Command {
var cpuMemory, cpuCores int
c := &cobra.Command{
Use: "replicas <model> <profile>=<n>...",
Short: "How many of a model's replicas run on each of its profiles: gpu=7 cpu=4",
Long: "Each replica runs on the best profile that fits unless counted here: seven on\n" +
"cards and four on CPUs, all answering under the model's name, the router filling\n" +
"the cards before the CPUs. A profile not named keeps its count. For a GGUF model\n" +
"with no profile on CPUs, --cpu-memory adds one named cpu. Its scaling becomes the\n" +
"sum; llmgw ctl model scale with another total ends the count per profile.",
Example: " llmgw ctl model replicas qwen-coder gpu-1xl40s-48g=7 cpu=4 --cpu-memory 24 --cpu-cores 8",
Args: cobra.MinimumNArgs(2),
RunE: func(cmd *cobra.Command, args []string) error {
req := adminapi.ProfileReplicasRequest{Profiles: map[string]int{}}
for _, a := range args[1:] {
name, v, ok := strings.Cut(a, "=")
n, err := strconv.Atoi(v)
if !ok || err != nil || name == "" {
return fmt.Errorf("%q: a profile and a count, as gpu=7", a)
}
req.Profiles[name] = n
}
if cpuMemory > 0 {
req.CPU = &modelsvc.CPUProfile{MemoryGB: cpuMemory, CPUCores: cpuCores}
}
body, err := json.Marshal(req)
if err != nil {
return err
}
var m catalog.Model
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/replicas", body, &m); err != nil {
return err
}
var parts []string
for _, p := range m.Profiles {
parts = append(parts, fmt.Sprintf("%d on %s", p.Replicas, p.Name))
}
fmt.Fprintf(cmd.OutOrStdout(), "%s runs %s: %d in all. Those short start first, then those over stop.\n",
m.Name, strings.Join(parts, ", "), m.SplitTotal())
return nil
},
}
c.Flags().IntVar(&cpuMemory, "cpu-memory", 0, "add a profile on CPUs named cpu, reserving this many GB a replica")
c.Flags().IntVar(&cpuCores, "cpu-cores", 0, "the cores a replica on CPUs computes on (default: half the node's)")
return c
}
+40 -2
View File
@@ -321,7 +321,10 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
fmt.Fprintln(out, "\nNo file fits this cluster as it is: a smaller context (--context), or more memory.")
return errDoesNotFit
}
fmt.Fprintf(out, "\nBest: %s on %s: %s.\n", rec.Artifact, rec.Label(), rec.MemoryText())
fmt.Fprintf(out, "\nBest: %s on %s: %s%s.\n", rec.Artifact, rec.Label(), rec.MemoryText(), speedNote(rec))
if w := slowWarning(rec); w != "" {
fmt.Fprintln(out, w)
}
if p.KVBytesPerToken > 0 {
fmt.Fprintf(out, "The context costs %.2f GB per 1000 tokens, the whole model (--context to plan another).\n",
modelplan.GB(p.KVBytesPerToken*1000))
@@ -334,7 +337,8 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
if o.FreeNow {
now = "free"
}
fmt.Fprintf(w, " %s\t%s\t%s\t%d host(s)\t%s\n", o.Artifact, o.Label(), o.MemoryText(), o.Hosts, now)
fmt.Fprintf(w, " %s\t%s\t%s\t%d host(s)\t%s\t%s\n", o.Artifact, o.Label(), o.MemoryText(), o.Hosts, now,
speedCell(o.TokensPerSecond))
}
_ = w.Flush()
}
@@ -349,3 +353,37 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
fmt.Fprintln(out, " --fallbacks")
return nil
}
// speedCell is a way's speed in a table: about how many tokens a second one
// reply gets, and whether that is slow.
func speedCell(tps float64) string {
switch {
case tps <= 0:
return ""
case tps < modelplan.CPUTooSlowTPS:
return fmt.Sprintf("~%.1f tok/s, very slow", tps)
case tps < modelplan.CPUSlowTPS:
return fmt.Sprintf("~%.0f tok/s, slow", tps)
}
return fmt.Sprintf("~%.0f tok/s", tps)
}
// speedNote is the same for a sentence, empty when not known.
func speedNote(o modelplan.Option) string {
if o.TokensPerSecond <= 0 {
return ""
}
return fmt.Sprintf(", about %s tokens a second for one reply", strconv.FormatFloat(o.TokensPerSecond, 'f', 1, 64))
}
// slowWarning says what a slow way means for somebody waiting on a reply.
func slowWarning(o modelplan.Option) string {
tps := o.TokensPerSecond
switch {
case tps <= 0 || tps >= modelplan.CPUSlowTPS:
return ""
case tps < modelplan.CPUTooSlowTPS:
return fmt.Sprintf("Very slow: a reply of 500 tokens takes about %.0f minutes.", max(1, 500/tps/60))
}
return fmt.Sprintf("Slow: a reply of 500 tokens takes about %.0f seconds.", 500/tps)
}
+22
View File
@@ -49,6 +49,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
@@ -59,6 +60,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
"gitea.mynodes.xyz/ilker/llmgw/internal/router"
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
@@ -261,6 +263,8 @@ func newServerCmd() *cobra.Command {
// What the settings say of the firewall, for the mirror and the
// store's members; nil without a control plane.
var fwPub *firewallPublisher
// The Slurm cluster beside llmgw, when there is a store (BACKLOG P).
var slurmSvc *slurmsvc.Service
updateMetrics := &modelserve.UpdateCollector{}
// The store's cluster, when it is one; read by the metrics and
// by `ctl status`.
@@ -436,6 +440,21 @@ func newServerCmd() *cobra.Command {
agentSvc.SetTrain(trainer)
go trainer.Run(ctx)
api.SetTrain(&adminapi.TrainAdmin{Store: st, Service: trainer})
// Slurm on the nodes the settings name, its munge key
// made from a cluster secret.
slurmSvc = slurmsvc.New(func() config.SlurmSettings { return current.Load().Slurm },
admin.Inventory(), agentSvc, func(ctx context.Context) ([]byte, error) {
secret, err := st.ClusterSecret(ctx, "slurm-munge")
if err != nil {
return nil, err
}
return slurm.MungeKey(secret)
}, log)
// What slurm.conf held, for the leader that comes next.
slurmSvc.SetMemory(slurmsvc.StoreMemory(st))
agentSvc.SetSlurm(slurmSvc)
go slurmSvc.Run(ctx)
api.SetSlurm(slurmSvc)
}
// The cluster's settings, each change checked against this
// machine's own keys before it is kept.
@@ -841,6 +860,9 @@ func newServerCmd() *cobra.Command {
fwPub.Kick()
}
}
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "slurm") }) && slurmSvc != nil {
slurmSvc.Kick()
}
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "on_demand") }) && admin != nil {
admin.Reconciler().SetEvictionTimes(next.OnDemand.EvictAfterIdle, next.OnDemand.MinLifetime)
}
+5
View File
@@ -124,6 +124,9 @@ type serverInit struct {
// media has the nodes install ComfyUI and the leader add a picture model
// and a 3D model once a card fits one.
media bool
// slurm runs Slurm beside llmgw: slurmctld on this machine's node,
// slurmd on every node, each llmgw's until it is lent.
slurm bool
// evictAfterIdle and minLifetime are on_demand's; zero leaves its own.
evictAfterIdle, minLifetime time.Duration
// standbys makes each machine of the replicated store a control plane
@@ -258,6 +261,8 @@ func newServerInitCmd() *cobra.Command {
f.DurationVar(&in.maxKeyTTL, "max-key-ttl", defaultMaxKeyTTL, "the longest an API key lives")
f.BoolVar(&in.media, "media", false,
"pictures and 3D: the nodes install ComfyUI, and a picture model and a 3D model are added once a card fits one")
f.BoolVar(&in.slurm, "slurm", false,
"Slurm for training and batch jobs: slurmctld on this machine, slurmd on every node, each llmgw's until 'llmgw ctl slurm lend'")
f.DurationVar(&in.evictAfterIdle, "evict-after-idle", 0,
"how long a model loaded on demand goes unasked-for before another may have its card (0: 2m)")
f.DurationVar(&in.minLifetime, "min-lifetime", 0,
+34
View File
@@ -13,6 +13,7 @@ import (
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
)
@@ -132,9 +133,18 @@ func (in *serverInit) checkChoices() error {
return fmt.Errorf("max key ttl %s: an hour at least", in.maxKeyTTL)
case in.evictAfterIdle < 0 || in.minLifetime < 0:
return fmt.Errorf("on-demand times cannot be negative")
case in.slurm && in.noNode:
return fmt.Errorf("slurm runs slurmctld on this machine's node: it needs the node (no --no-node)")
case in.vip != "" && !in.standbys:
return fmt.Errorf("a floating address moves between control planes: it needs the standbys")
}
if in.slurm {
// The controller is this machine's node, named by its hostname: a
// name Slurm does not take would keep the server from starting.
if err := (config.SlurmSettings{Enabled: true, Controller: localNodeID()}).Check(); err != nil {
return fmt.Errorf("slurm: this machine's hostname: %w", err)
}
}
if in.vip != "" {
ip, _, err := net.ParseCIDR(in.vip)
if err != nil || ip.To4() == nil {
@@ -318,6 +328,9 @@ func (in *serverInit) tuneConfig(body string) string {
"# one ('llmgw ctl model presets auto off' stops it).\n"+
"engines:\n comfyui: "+provision.ComfyUIVersion()+"\npresets:\n auto: true\n")
}
if in.slurm {
extra = append(extra, slurmSection(localNodeID()))
}
if in.evictAfterIdle > 0 || in.minLifetime > 0 {
b := "# How models that load when asked for take turns on a card, chosen at\n# install.\non_demand:\n"
if in.evictAfterIdle > 0 {
@@ -334,6 +347,24 @@ func (in *serverInit) tuneConfig(body string) string {
return body
}
// slurmSection is server.yaml's slurm block as an install chooses it:
// slurmctld on this machine's node, slurmd on every node.
func slurmSection(controller string) string {
return "# Slurm beside llmgw, chosen at install: slurmctld on this machine, slurmd on\n" +
"# every node, those that join later too. Each node is llmgw's, drained in\n" +
"# Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm; 'llmgw ctl slurm\n" +
"# reclaim NODE' takes it back once its jobs end.\n" +
"slurm:\n enabled: true\n controller: " + yamlString(controller) + "\n all_nodes: true\n"
}
// localNodeID is the node id this machine's agent takes: its hostname.
func localNodeID() string {
if h, err := os.Hostname(); err == nil && h != "" {
return h
}
return "localhost"
}
// durationYAML is a duration as a person writes it: 2h, 90m, 1m30s.
func durationYAML(d time.Duration) string {
switch {
@@ -467,6 +498,9 @@ func (in *serverInit) initFlags() []string {
if in.media {
add("--media")
}
if in.slurm {
add("--slurm")
}
if in.evictAfterIdle > 0 {
add("--evict-after-idle", durationYAML(in.evictAfterIdle))
}
+7 -4
View File
@@ -17,7 +17,7 @@ import (
func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
in := &serverInit{
language: "tr", signupOpen: true, defaultRPM: 120, defaultTPM: 50000, maxKeyTTL: 30 * 24 * time.Hour,
media: true, evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
media: true, slurm: true, evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
judge: judgeSpec{Fail: "open", Categories: []string{"Violent", "PII"}, Replicas: 3, Budget: 3 * time.Second},
}
j := in.judgeFor()
@@ -41,6 +41,8 @@ func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
t.Errorf("limits %d %d %s", cfg.Auth.DefaultRPM, cfg.Auth.DefaultTPM, cfg.Auth.MaxKeyTTL)
case !cfg.Presets.Auto || cfg.Engines["comfyui"] == "":
t.Errorf("pictures and 3D: presets %v comfyui %q", cfg.Presets.Auto, cfg.Engines["comfyui"])
case !cfg.Slurm.Enabled || !cfg.Slurm.AllNodes || cfg.Slurm.Controller != localNodeID():
t.Errorf("slurm %+v", cfg.Slurm)
case cfg.OnDemand.EvictAfterIdle != time.Minute || cfg.OnDemand.MinLifetime != 90*time.Second:
t.Errorf("on demand %+v", cfg.OnDemand)
}
@@ -95,14 +97,14 @@ func TestAStarterIsChosenByNameOrByItsFile(t *testing.T) {
func TestTheChoicesAreTheirFlagsAgain(t *testing.T) {
in := &serverInit{admin: "ops@example.com", hosts: []string{"llm.example.com"}, postgres: true,
mirrorPath: "/data/models", noNode: true, language: "tr", media: true,
mirrorPath: "/data/models", language: "tr", media: true, slurm: true,
webhooks: []string{"https://hooks.slack.com/services/T/B/x, json:https://a.example.com/in"},
judge: judgeSpec{Categories: []string{"Violent", "Jailbreak"}}, maxKeyTTL: 7 * 24 * time.Hour}
in.starter, _ = starterFrom("qwen3-4b")
got := strings.Join(in.initFlags(), " ")
want := "--admin ops@example.com --host llm.example.com --language tr --max-key-ttl 168h --postgres " +
"--mirror-dir /data/models --no-node --starter qwen3-4b --judge-categories Violent,Jailbreak " +
"--webhook https://hooks.slack.com/services/T/B/x --webhook json:https://a.example.com/in --media"
"--mirror-dir /data/models --starter qwen3-4b --judge-categories Violent,Jailbreak " +
"--webhook https://hooks.slack.com/services/T/B/x --webhook json:https://a.example.com/in --media --slurm"
if got != want {
t.Errorf("flags\n got %s\nwant %s", got, want)
}
@@ -124,6 +126,7 @@ func TestChoicesThatCannotBeInstalledAreRefused(t *testing.T) {
{maxKeyTTL: time.Minute},
{vip: "10.0.0.160/24"},
{standbys: true, vip: "10.0.0.160"},
{slurm: true, noNode: true},
} {
if err := in.checkChoices(); err == nil {
t.Errorf("taken: %+v", in)
+64 -8
View File
@@ -20,6 +20,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
)
@@ -406,6 +407,48 @@ func (m consoleModel) modelMenu(md adminapi.ModelView) *menuDialog {
}})
return nil
}},
{"Replicas per profile…", func(m *consoleModel) tea.Cmd {
var now []string
for _, p := range md.Profiles {
now = append(now, fmt.Sprintf("%s=%d", p, md.ProfileReplicas[p]))
}
m.openForm(&formDialog{title: "Replicas per profile of " + md.Name,
note: "How many replicas run on each profile, all answering under the model's name: seven on cards and " +
"four on CPUs, the router filling the cards first. A profile not named keeps its count; cpu=N with " +
"the memory below adds a profile on CPUs to a GGUF model that has none. Those short start first, " +
"then those over stop.",
fields: []formField{
textF("replicas", strings.Join(now, " "), "profile=count, separated by spaces"),
textF("CPU memory GB", "", "only to add a profile on CPUs: what a replica reserves"),
textF("CPU cores", "", "of a replica on CPUs; empty is half the node's"),
},
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
req := adminapi.ProfileReplicasRequest{Profiles: map[string]int{}}
for _, f := range strings.Fields(v[0]) {
name, c, ok := strings.Cut(f, "=")
n, err := strconv.Atoi(c)
if !ok || err != nil || n < 0 || name == "" {
return nil, fmt.Errorf("%q: a profile and a count, as gpu=7", f)
}
req.Profiles[name] = n
}
if v[1] != "" {
mem, err := strconv.Atoi(v[1])
if err != nil || mem < 1 {
return nil, fmt.Errorf("the CPU memory is a whole number of GB")
}
cores := 0
if v[2] != "" {
if cores, err = strconv.Atoi(v[2]); err != nil || cores < 1 {
return nil, fmt.Errorf("the CPU cores are a whole number from 1")
}
}
req.CPU = &modelsvc.CPUProfile{MemoryGB: mem, CPUCores: cores}
}
return m.call("Replicas of "+md.Name, http.MethodPost, path+"/replicas", req, nil), nil
}})
return nil
}},
{"Resources…", func(m *consoleModel) tea.Cmd {
profiles := md.Profiles
if len(profiles) == 0 {
@@ -600,17 +643,30 @@ func (m consoleModel) nodeMenu(n adminapi.NodeView) *menuDialog {
menuItem{"Delete model files nothing uses", func(m *consoleModel) tea.Cmd {
return m.call("Unused files on "+n.ID, http.MethodPost, "/api/v1/weights/collect", map[string]string{"node": n.ID},
func(raw json.RawMessage) string {
var rs []struct {
Files int `json:"files"`
Bytes int64 `json:"bytes"`
}
// What the node no longer lists, and what it kept.
var rs []adminapi.CollectResult
_ = json.Unmarshal(raw, &rs)
var files int
var b int64
var files, left int
var b, lb int64
var errs []string
for _, r := range rs {
files, b = files+r.Files, b+r.Bytes
files, b, left, lb = files+r.Files, b+r.Bytes, left+r.Left, lb+r.LeftBytes
if r.Error != "" {
errs = append(errs, r.Error)
}
}
return fmt.Sprintf("%d files, %.1f GB freed.", files, float64(b)/(1<<30))
if len(rs) == 0 {
return "Nothing on " + n.ID + " that no model uses."
}
say := fmt.Sprintf("%d files removed, %.1f GB freed.", files, float64(b)/(1<<30))
if left > 0 {
say += fmt.Sprintf(" %d files (%.1f GB) are still there: the node kept them, and its agent's log "+
"says why (journalctl -u llmgw-agent there).", left, float64(lb)/(1<<30))
}
if len(errs) > 0 {
say += " " + strings.Join(errs, "; ")
}
return say
})
}},
menuItem{"Remove from the cluster…", func(m *consoleModel) tea.Cmd {
+22 -2
View File
@@ -57,7 +57,7 @@ type installForm struct {
standby *installPart
vipF *setupField
control, store, mirror, node, starter, media, judge, firewall, alerts, limits, turns, bundle *installPart
control, store, mirror, node, starter, media, slurm, judge, firewall, alerts, limits, turns, bundle *installPart
admin, hosts, language, signup *setupField
storeKind *setupField
@@ -282,6 +282,25 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
summary: func() string { return "ComfyUI; a picture model and a 3D model once a card fits" },
}
// --- Slurm ---
f.slurm = &installPart{
title: "Slurm", on: in.slurm,
about: "A Slurm cluster beside llmgw, for training and other batch work: slurmctld on this machine, " +
"slurmd on every node, those that join later too. The agents install munge and Slurm from the " +
"machines' own package sources.\n\n" +
"Each node stays llmgw's, drained in Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm: " +
"its models move off first. 'llmgw ctl slurm reclaim NODE' takes it back once its jobs end. Jobs " +
"are submitted on this machine with sbatch and srun.",
off: "no Slurm; 'llmgw ctl slurm setup' adds it later",
summary: func() string { return "slurmctld here, slurmd on every node; none lent" },
needs: func() string {
if !f.node.on {
return "slurmctld runs on this machine's node: turn the node on, or Slurm off"
}
return ""
},
}
// --- the guardrails' judge ---
j := in.judgeFor()
f.judgeFail = choiceField("When no judge answers in time", []string{"refuse", "serve flagged"},
@@ -421,7 +440,7 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
if ha {
f.parts = append(f.parts, f.standby)
}
f.parts = append(f.parts, f.mirror, f.node, f.starter, f.media, f.judge, f.firewall,
f.parts = append(f.parts, f.mirror, f.node, f.starter, f.media, f.slurm, f.judge, f.firewall,
f.alerts, f.limits, f.turns, f.bundle)
return f
}
@@ -653,6 +672,7 @@ func (f *installForm) apply(in *serverInit) error {
in.noModel = !f.starter.on
in.starter, in.starterCtx = starter, ctx
in.media = f.media.on
in.slurm = f.slurm.on
in.noGuardModel = !f.judge.on
in.judge = judge
in.noFirewall = !f.firewall.on
+24 -1
View File
@@ -107,7 +107,15 @@ func optionLabel(o modelplan.Option) string {
if o.FreeNow {
now = sOK.Render("free")
}
return fmt.Sprintf("%s · %s · %s", o.Label(), o.MemoryText(), now)
label := fmt.Sprintf("%s · %s · %s", o.Label(), o.MemoryText(), now)
if c := speedCell(o.TokensPerSecond); c != "" {
style := sDim
if o.TokensPerSecond < modelplan.CPUTooSlowTPS {
style = sWarn
}
label += " · " + style.Render(c)
}
return label
}
// repoMenu offers a repository's files, each with the best way it runs here;
@@ -196,16 +204,26 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
note += fmt.Sprintf("\nMemory for a %d-token context, each way it runs:", p.Context)
for _, o := range opts {
note += "\n " + o.Label() + ": " + o.MemoryText()
if w := slowWarning(o); w != "" && o.CPU {
note += "\n " + sWarn.Render(w)
}
}
if p.KVBytesPerToken > 0 {
note += fmt.Sprintf("\nThe context costs %.2f GB per 1000 tokens, the whole model; a longer one is checked\n"+
"before it is served.", modelplan.GB(p.KVBytesPerToken*1000))
}
note += "\nThe mirror fetches it and checks it, it joins the catalog, and a replica is started\nand asked to say hello."
// The longest context the first way holds, up to what the model was
// trained for: a choice of another way is checked again on submitting.
ctxDefault := "4096"
if a.Format == "safetensors" {
ctxDefault = strconv.Itoa(p.Context)
}
if len(opts) > 0 {
if most := opts[0].MostContext(p.KVBytesPerToken, p.Trained); most > 0 {
ctxDefault = strconv.Itoa(most)
}
}
if len(opts) > 0 && opts[0].Engine == "strata" {
ctxDefault = "32768"
}
@@ -238,6 +256,11 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
Description: v[3], Format: a.Format}
if a.Format == "gguf" {
req.File = a.Name
// A context longer than the default pool is the pool: one
// slot of it, where the server would refuse the request.
if ctxLen > modelserve.DefaultCtxPool {
req.CtxPool = ctxLen
}
}
o := opts[slices.Index(where, v[4])]
// Asked for a context other than planned: does it still fit?
+2 -2
View File
@@ -221,11 +221,11 @@ func (m consoleModel) userMenu(u store.User) *menuDialog {
items = append(items,
menuItem{"Add to an organization, team or project…", func(m *consoleModel) tea.Cmd {
m.openForm(&formDialog{title: "Membership for " + who,
note: "A member of a team or organization may use every project in it.",
note: "A member of a team or organization may use every project in it. On an organization, a manager also sees its page and an admin also changes it.",
fields: []formField{
choiceF("level", []string{"org", "team", "project"}, "team"),
textF("which", "", "Acme, Acme/ml or Acme/ml/training"),
choiceF("role", []string{store.RoleMember, "manager"}, store.RoleMember),
choiceF("role", store.MembershipRoles, store.RoleMember),
},
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
if v[1] == "" {
+4 -2
View File
@@ -391,7 +391,8 @@ func newUserJoinCmd(open userOpener) *cobra.Command {
Long: "A person can be a member of any number of them. A member of a team or\n" +
"organization may use every project in it.",
Example: " llmgw users join ayse@example.com --team Acme/ml\n" +
" llmgw users join ayse@example.com --project Scania/research/vision",
" llmgw users join ayse@example.com --project Scania/research/vision\n" +
" llmgw users join lars@example.com --org Scania --role admin",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
s, err := open(cmd)
@@ -415,7 +416,8 @@ func newUserJoinCmd(open userOpener) *cobra.Command {
},
}
where.bind(cmd)
cmd.Flags().StringVar(&role, "role", store.RoleMember, "member, or manager: with --org, may see the organization's page")
cmd.Flags().StringVar(&role, "role", store.RoleMember,
"member; with --org also manager, who sees the organization's page, or admin, who also changes its teams, projects, people and keys")
cmd.Flags().StringVar(&by, "by", "", "who added them, for the record")
return cmd
}
+7
View File
@@ -158,6 +158,10 @@ type Server struct {
// Firewall turns llmgw's firewall on every machine on or off.
Firewall FirewallSettings `koanf:"firewall"`
// Slurm is a Slurm cluster run beside llmgw on some of its nodes, and
// which of them it may use now.
Slurm SlurmSettings `koanf:"slurm"`
// Presets is how the models that come with llmgw reach the catalog.
Presets PresetSettings `koanf:"presets"`
@@ -960,6 +964,9 @@ func (s Server) Validate() error {
if _, err := s.Facility.Policy(); err != nil {
return err
}
if err := s.Slurm.Check(); err != nil {
return err
}
if s.Control.Timezone != "" {
if _, err := time.LoadLocation(s.Control.Timezone); err != nil {
return fmt.Errorf("control.timezone %q: %w", s.Control.Timezone, err)
+115
View File
@@ -0,0 +1,115 @@
package config
import (
"fmt"
"regexp"
"slices"
)
// SlurmSettings is a Slurm cluster run beside llmgw on some of its nodes
// (BACKLOG P), for training and other batch work llmgw does not schedule.
// The leader tells each named node's agent what to run; the agents install
// it from the machine's package sources and keep it going.
type SlurmSettings struct {
// Enabled runs Slurm on the nodes below. False stops llmgw keeping it;
// what was installed is left installed, and every node is llmgw's.
Enabled bool `koanf:"enabled"`
// ClusterName is Slurm's ClusterName. Empty is "llmgw".
ClusterName string `koanf:"cluster_name"`
// Controller is the llmgw node whose agent runs slurmctld. It needs no
// GPU, and need not be one of Nodes.
Controller string `koanf:"controller"`
// Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained
// in Slurm, until it is also in Lent.
Nodes []string `koanf:"nodes"`
// AllNodes runs slurmd on every llmgw node, those that join later too,
// besides any named in Nodes. Each is llmgw's until it is lent, as any.
AllNodes bool `koanf:"all_nodes"`
// Lent are the nodes of Nodes Slurm may run jobs on now. A node put here
// is drained of llmgw's replicas first, as `ctl drain` does, and resumed
// in Slurm once none is left; a node taken out is drained in Slurm and
// given back to llmgw when its last job ends.
Lent []string `koanf:"lent"`
// Partition is the one partition every node is in. Empty is "gpu".
Partition string `koanf:"partition"`
}
// slurmName is what Slurm takes as a node, cluster or partition name.
var slurmName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]{0,62}$`)
// Cluster is ClusterName or its default.
func (s SlurmSettings) Cluster() string {
if s.ClusterName == "" {
return "llmgw"
}
return s.ClusterName
}
// PartitionName is Partition or its default.
func (s SlurmSettings) PartitionName() string {
if s.Partition == "" {
return "gpu"
}
return s.Partition
}
// IsLent is whether Slurm may use the node now.
func (s SlurmSettings) IsLent(node string) bool {
return s.Enabled && (s.AllNodes || slices.Contains(s.Nodes, node)) && slices.Contains(s.Lent, node)
}
// WithNodes is the settings with every node of ids in Nodes when AllNodes:
// the nodes that run slurmd, as the leader's inventory has them.
func (s SlurmSettings) WithNodes(ids []string) SlurmSettings {
if !s.AllNodes {
return s
}
nodes := slices.Clone(s.Nodes)
for _, id := range ids {
if !slices.Contains(nodes, id) {
nodes = append(nodes, id)
}
}
s.Nodes = nodes
return s
}
// Check refuses settings Slurm could not run on.
func (s SlurmSettings) Check() error {
if !s.Enabled {
return nil
}
if s.Controller == "" {
return fmt.Errorf("slurm.controller is empty: name the node that runs slurmctld")
}
if !slurmName.MatchString(s.Controller) {
return fmt.Errorf("slurm.controller %q is not a name Slurm takes", s.Controller)
}
for _, n := range []struct{ key, v string }{{"slurm.cluster_name", s.ClusterName}, {"slurm.partition", s.Partition}} {
if n.v != "" && !slurmName.MatchString(n.v) {
return fmt.Errorf("%s %q is not a name Slurm takes: letters, digits, _ . -", n.key, n.v)
}
}
seen := map[string]bool{}
for i, n := range s.Nodes {
if !slurmName.MatchString(n) {
return fmt.Errorf("slurm.nodes[%d] %q is not a name Slurm takes", i, n)
}
if seen[n] {
return fmt.Errorf("slurm.nodes names %s twice", n)
}
seen[n] = true
}
for i, n := range s.Lent {
if !seen[n] && !s.AllNodes {
return fmt.Errorf("slurm.lent[%d] %q is not in slurm.nodes: only a node that runs slurmd can be lent", i, n)
}
}
return nil
}
+53
View File
@@ -0,0 +1,53 @@
package config
import (
"strings"
"testing"
)
func TestSlurmCheck(t *testing.T) {
ok := SlurmSettings{Enabled: true, Controller: "node06", Nodes: []string{"cuda01", "site2"}, Lent: []string{"cuda01"}}
if err := ok.Check(); err != nil {
t.Fatal(err)
}
if !ok.IsLent("cuda01") || ok.IsLent("site2") || ok.IsLent("node06") {
t.Error("IsLent is wrong")
}
if ok.Cluster() != "llmgw" || ok.PartitionName() != "gpu" {
t.Error("defaults are wrong")
}
if (SlurmSettings{Lent: []string{"x"}}).Check() != nil {
t.Error("settings that are off were checked")
}
for want, s := range map[string]SlurmSettings{
"controller is empty": {Enabled: true},
"not a name": {Enabled: true, Controller: "bad name"},
"twice": {Enabled: true, Controller: "c", Nodes: []string{"a", "a"}},
"not in slurm.nodes": {Enabled: true, Controller: "c", Nodes: []string{"a"}, Lent: []string{"b"}},
"slurm.partition": {Enabled: true, Controller: "c", Partition: "a b"},
} {
if err := s.Check(); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("%+v: got %v, want %q", s, err, want)
}
}
if (SlurmSettings{Enabled: false, Lent: []string{"a"}}).IsLent("a") {
t.Error("a node is lent with Slurm off")
}
}
func TestSlurmAllNodes(t *testing.T) {
s := SlurmSettings{Enabled: true, Controller: "c", AllNodes: true, Lent: []string{"joined"}}
if err := s.Check(); err != nil {
t.Fatalf("a node lent with all_nodes refused: %v", err)
}
if !s.IsLent("joined") {
t.Fatal("not lent with all_nodes")
}
if got := s.WithNodes([]string{"a", "joined"}).Nodes; len(got) != 2 {
t.Fatalf("nodes %v", got)
}
s.AllNodes = false
if s.Check() == nil || len(s.WithNodes([]string{"a"}).Nodes) != 0 {
t.Fatal("all_nodes off still takes the inventory's nodes")
}
}
+56 -2
View File
@@ -22,6 +22,7 @@ import (
"unicode"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
@@ -77,7 +78,9 @@ type API struct {
// tenancy is organizations, keys and people; nil without a store.
tenancy TenancyStore
// train is adapter training; nil without a store and agents.
train *TrainAdmin
train *TrainAdmin
// slurm is the Slurm cluster beside llmgw; nil without a store and agents.
slurm SlurmService
settings *Settings
auth *AuthAdmin
mcp MCPStore
@@ -130,10 +133,12 @@ func (a *API) Register(mux Mux) {
mux.HandleFunc("POST /api/v1/models/{name}/start", a.startModel)
mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
mux.HandleFunc("POST /api/v1/models/{name}/replicas", a.setProfileReplicas)
a.registerQuotas(mux)
a.registerKB(mux)
a.registerTenancy(mux)
a.registerTrain(mux)
a.registerSlurm(mux)
a.registerGuard(mux)
a.registerSettings(mux)
a.registerAuth(mux)
@@ -259,6 +264,9 @@ type NodeView struct {
// MemBandwidthGBs is how fast its memory reads, in GB/s, measured when
// its agent started; 0 when not reported. A model's speed on the CPU.
MemBandwidthGBs float64 `json:"mem_bandwidth_gbs,omitempty"`
// MemBandwidthCoreGBs is what one core reads alone, in GB/s; 0 when not
// reported. A CPU replica's threads read at about this each.
MemBandwidthCoreGBs float64 `json:"mem_bandwidth_core_gbs,omitempty"`
// Engines are the inference engines installed here, as the agent found
// them. Absent means the agent is older than the probe and said nothing,
// which is not the same as having none — placement treats it that way too,
@@ -489,6 +497,9 @@ type ModelView struct {
Unplaceable bool `json:"unplaceable,omitempty"`
Fallbacks []string `json:"fallbacks,omitempty"`
Profiles []string `json:"profiles"`
// ProfileReplicas is how many replicas run on each profile, for a model
// whose replicas are counted per profile; absent otherwise.
ProfileReplicas map[string]int `json:"profile_replicas,omitempty"`
// MemoryGB and CPUCores are what one replica reserves, from its first
// profile; MeasuredMemoryGB is the most one of its replicas uses now, 0
// when none is measured.
@@ -723,7 +734,7 @@ func (a *API) Nodes() []NodeView {
v.EngineBuilds[e] = b
}
}
v.MemBandwidthGBs = n.Host.MemBandwidthGBs
v.MemBandwidthGBs, v.MemBandwidthCoreGBs = n.Host.MemBandwidthGBs, n.Host.MemBandwidthCoreGBs
for _, g := range n.GPUs {
if g.MIG != nil {
continue // said of the card it is carved from
@@ -865,6 +876,12 @@ func (a *API) Models() []ModelView {
Canary: m.Canary, CanaryOf: a.ctl.Catalog().CanaryOf(m.Name), Eval: m.Eval,
Role: m.Role,
}
if m.Split() {
v.ProfileReplicas = map[string]int{}
for _, p := range m.Profiles {
v.ProfileReplicas[p.Name] = p.Replicas
}
}
for _, p := range m.Profiles {
v.Profiles = append(v.Profiles, p.Name)
}
@@ -962,6 +979,15 @@ type Models interface {
SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
SetAdapters(ctx context.Context, name string, adapters []catalog.Adapter, by string) (catalog.Model, error)
SetProfileReplicas(ctx context.Context, name string, counts map[string]int, cpu *modelsvc.CPUProfile, by string) (catalog.Model, error)
}
// ProfileReplicasRequest counts a model's replicas per profile: profile
// name to replicas, and CPU, for a llama.cpp model with no profile on CPUs,
// the one to add.
type ProfileReplicasRequest struct {
Profiles map[string]int `json:"profiles"`
CPU *modelsvc.CPUProfile `json:"cpu,omitempty"`
}
// ResourcesRequest sets what a model's profiles reserve of a node.
@@ -1288,6 +1314,25 @@ func (a *API) setResources(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, m)
}
// setProfileReplicas counts a model's replicas per profile.
func (a *API) setProfileReplicas(w http.ResponseWriter, r *http.Request) {
if a.models == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store")
return
}
var req ProfileReplicasRequest
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
return
}
m, err := a.models.SetProfileReplicas(r.Context(), r.PathValue("name"), req.Profiles, req.CPU, actorOf(r))
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
writeJSON(w, http.StatusOK, m)
}
func (a *API) removeModel(w http.ResponseWriter, r *http.Request) {
if a.models == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store")
@@ -1966,6 +2011,15 @@ func PoolNodesOf(nodes []NodeView) []modelplan.PoolNode {
if l := n.Load; l != nil && l.MemTotal > 0 {
p.HostFreeGB = int((l.MemTotal - l.MemUsed) >> 30)
}
p.CPUs, p.MemBWGBs, p.MemBWCoreGBs = n.CPUCount, n.MemBandwidthGBs, n.MemBandwidthCoreGBs
for _, g := range n.GPUs {
if bw := modelplan.GPUBandwidthGBs(g.Name); bw > 0 {
if p.CardBWGBs == nil {
p.CardBWGBs = map[string]float64{}
}
p.CardBWGBs[g.Class] = bw
}
}
out = append(out, p)
}
return out
+11
View File
@@ -16,6 +16,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
@@ -263,6 +264,16 @@ func (f *fakeModels) SetAdapters(_ context.Context, name string, as []catalog.Ad
return catalog.Model{Name: name, Adapters: as}, f.err
}
func (f *fakeModels) SetProfileReplicas(_ context.Context, name string, counts map[string]int, _ *modelsvc.CPUProfile,
_ string,
) (catalog.Model, error) {
m := catalog.Model{Name: name}
for p, n := range counts {
m.Profiles = append(m.Profiles, catalog.Profile{Name: p, Replicas: n})
}
return m, f.err
}
func (f *fakeModels) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
return catalog.Model{Name: name, Owner: owner}, f.err
}
+40 -22
View File
@@ -2,6 +2,8 @@ package adminapi
import (
"context"
"errors"
"fmt"
"log/slog"
"net/http"
"net/url"
@@ -48,10 +50,29 @@ type PurgeView struct {
// purgeModel removes a model from the catalog and its files from the mirror
// and the nodes.
func (a *API) purgeModel(w http.ResponseWriter, r *http.Request, name string) {
out, err := a.Purge(r.Context(), name, actorOf(r))
switch {
case errors.Is(err, errNoSuchModel):
writeError(w, http.StatusNotFound, err.Error())
case err != nil:
writeError(w, http.StatusBadRequest, err.Error())
default:
writeJSON(w, http.StatusOK, out)
}
}
// errNoSuchModel is a model the catalog does not have.
var errNoSuchModel = errors.New("no such model in the catalog")
// Purge removes a model from the catalog and its files from the mirror and
// the nodes, waiting a while for its replicas to stop before the nodes'.
func (a *API) Purge(ctx context.Context, name, by string) (PurgeView, error) {
if a.models == nil {
return PurgeView{}, errors.New("this control plane has no store")
}
m, ok := a.ctl.Catalog().Resolve(name)
if !ok {
writeError(w, http.StatusNotFound, "no model "+name+" in the catalog")
return
return PurgeView{}, fmt.Errorf("%w: %s", errNoSuchModel, name)
}
name = m.Name
others := []catalog.Model{}
@@ -60,37 +81,35 @@ func (a *API) purgeModel(w http.ResponseWriter, r *http.Request, name string) {
others = append(others, o)
}
}
if err := a.models.Remove(r.Context(), name); err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
if err := a.models.Remove(ctx, name); err != nil {
return PurgeView{}, err
}
a.presetDeclined(r.Context(), name, actorOf(r))
a.presetDeclined(ctx, name, by)
out := PurgeView{Removed: name}
sources := purgeSources(m)
if a.purgeMirror == nil {
out.Note = "this control plane has no mirror; its files there, if any, are left"
} else {
out.Mirror, out.Kept, out.Note = purgeFromMirror(r.Context(), a.purgeMirror, sources, others)
out.Mirror, out.Kept, out.Note = purgeFromMirror(ctx, a.purgeMirror, sources, others)
}
// The nodes' copies, once nothing runs on them.
deadline := time.Now().Add(purgeWait)
for a.runs(name) && time.Now().Before(deadline) {
select {
case <-r.Context().Done():
return
case <-ctx.Done():
return out, ctx.Err()
case <-time.After(time.Second):
}
}
if a.runs(name) {
out.Note = strings.TrimSpace(out.Note + " Its replicas had not stopped yet: its copies on the nodes are " +
"left, and llmgw ctl weights --collect deletes them.")
writeJSON(w, http.StatusOK, out)
return
return out, nil
}
out.Nodes = a.purgeFromNodes(r.Context(), sources, others)
writeJSON(w, http.StatusOK, out)
out.Nodes = a.purgeFromNodes(ctx, sources, others)
return out, nil
}
// runs says whether any replica of a model is still on a node.
@@ -197,7 +216,7 @@ func (a *API) purgeFromNodes(ctx context.Context, sources []string, others []cat
names[filepath.Base(s)] = true
}
wanted := inUse(others)
var out []CollectResult
var rs []removal
for _, n := range a.ctl.Inventory().Nodes() {
size := map[string]int64{}
have := make([]string, 0, len(n.Weights))
@@ -216,15 +235,14 @@ func (a *API) purgeFromNodes(ctx context.Context, sources []string, others []cat
if len(paths) == 0 {
continue
}
res := CollectResult{Node: n.ID, Files: len(paths)}
for _, p := range paths {
res.Bytes += size[p]
rs = append(rs, removal{node: n.ID, paths: paths, size: size})
}
out := a.removeConfirmed(ctx, rm, rs)
for _, r := range out {
if r.Error != "" || r.Left > 0 {
slog.Warn("a purged model's copies were not all deleted on a node", "node", r.Node,
"left", r.Left, "err", r.Error)
}
if err := rm.RemoveWeights(ctx, n.ID, paths); err != nil {
res.Error, res.Files, res.Bytes = err.Error(), 0, 0
slog.Warn("a purged model's copies could not be deleted on a node", "node", n.ID, "err", err)
}
out = append(out, res)
}
return out
}
+11
View File
@@ -12,6 +12,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/federation"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
@@ -168,6 +169,10 @@ var Routes = []Route{
Response: ScaleResponse{}},
{Method: "POST", Path: "/api/v1/models/{name}/scale", Summary: "Set how many replicas a model runs",
Request: ScaleRequest{}, Response: ScaleResponse{}},
{Method: "POST", Path: "/api/v1/models/{name}/replicas", Summary: "How many replicas run on each profile",
Description: "Seven on cards and four on CPUs, under the model's name. A profile not named keeps its count; " +
"cpu adds a profile on CPUs to a llama.cpp model that has none. Its scaling becomes the sum.",
Request: ProfileReplicasRequest{}, Response: catalog.Model{}},
{Method: "POST", Path: "/api/v1/models/{name}/resources", Summary: "Set what a model's profiles reserve of a node",
Request: ResourcesRequest{}, Response: catalog.Model{}},
{Method: "POST", Path: "/api/v1/models/{name}/class", Summary: "Set a model's capacity class",
@@ -388,6 +393,12 @@ var Routes = []Route{
Description: "Their keys in projects they no longer reach are revoked.",
Request: MembershipRequest{}, Response: UserView{}},
{Method: "GET", Path: "/api/v1/slurm", Summary: "The Slurm cluster beside llmgw, and where each of its nodes is",
Description: "From the cluster's settings (slurm.*) and what each node's agent last said. A node's phase is llmgw " +
"(drained in Slurm, no job left), lending (its replicas moving off), lent, reclaiming (drained in Slurm, jobs " +
"still running) or unknown (its agent has not said).",
Response: slurmsvc.Status{}},
{Method: "POST", Path: "/api/v1/train/jobs", Summary: "Queue a LoRA adapter's training on a node",
Description: "The node trains when its agent is there, nothing else trains on it, and a card has gpu_memory_mib free; " +
"the pairs are NDJSON of {\"q\", \"a\"}. An adapter learns how answers sound far better than what they say.",
+28
View File
@@ -0,0 +1,28 @@
package adminapi
import (
"net/http"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
)
// SlurmService is the Slurm cluster beside llmgw (BACKLOG P), as the leader
// keeps it.
type SlurmService interface {
Status() slurmsvc.Status
}
// SetSlurm gives the API the Slurm service.
func (a *API) SetSlurm(s SlurmService) { a.slurm = s }
func (a *API) registerSlurm(mux Mux) {
mux.HandleFunc("GET /api/v1/slurm", a.slurmStatus)
}
func (a *API) slurmStatus(w http.ResponseWriter, _ *http.Request) {
if a.slurm == nil {
writeError(w, http.StatusNotImplemented, "Slurm needs the store and the nodes' agents, which this control plane runs without")
return
}
writeJSON(w, http.StatusOK, a.slurm.Status())
}
+6 -2
View File
@@ -634,8 +634,12 @@ func (a *API) membership(add bool) http.HandlerFunc {
if role == "" {
role = store.RoleMember
}
if role != store.RoleMember && role != store.RoleManager {
writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member or manager", role))
if role != store.RoleMember && role != store.RoleManager && role != store.RoleOrgAdmin {
writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member, manager or admin", role))
return
}
if role != store.RoleMember && scope != store.ScopeOrg {
writeError(w, http.StatusBadRequest, fmt.Sprintf("%s is a role on an organization", role))
return
}
if err := a.tenancy.AddMembership(ctx, u.ID, scope, id, role, actorOf(r), time.Now()); err != nil {
+59
View File
@@ -0,0 +1,59 @@
package adminapi
import (
"context"
"errors"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
)
// What the portal changes of a served model beside what the catalog holds:
// the same as the API's routes, called in the process.
// SetModelContext changes a served model's context and longest answer.
func (a *API) SetModelContext(ctx context.Context, name string, req modelserve.ContextRequest, by string) (modelserve.ContextChange, error) {
if a.serve == nil {
return modelserve.ContextChange{}, errors.New("this control plane cannot change a model's context")
}
return a.serve.SetContext(ctx, name, req, by)
}
// SetModelGPUs spreads a served model over more cards a replica, or fewer.
func (a *API) SetModelGPUs(ctx context.Context, name string, req modelserve.GPUsRequest, by string) (modelserve.GPUsChange, error) {
if a.serve == nil {
return modelserve.GPUsChange{}, errors.New("this control plane cannot change a model's cards")
}
return a.serve.SetGPUs(ctx, name, req, by)
}
// RestartModel starts each of a model's ready replicas again, one at a time,
// and says how many.
func (a *API) RestartModel(name string) (int, error) {
var ids []string
for _, rep := range a.ctl.Reconciler().ReadyReplicas() {
if rep.Model == name {
ids = append(ids, rep.ID)
}
}
if len(ids) == 0 {
return 0, errors.New("none of its replicas is ready to restart")
}
return len(ids), a.ctl.Reconciler().RestartPaced(reconciler.RestartPace{}, false, ids...)
}
// RemoveModel takes a model out of the catalog, with its files on the mirror
// and the nodes when purge.
func (a *API) RemoveModel(ctx context.Context, name string, purge bool, by string) (PurgeView, error) {
if purge {
return a.Purge(ctx, name, by)
}
if a.models == nil {
return PurgeView{}, errors.New("this control plane has no store")
}
if err := a.models.Remove(ctx, name); err != nil {
return PurgeView{}, err
}
a.presetDeclined(ctx, name, by)
return PurgeView{Removed: name}, nil
}
+110 -13
View File
@@ -1,11 +1,14 @@
package adminapi
import (
"context"
"encoding/json"
"net/http"
"sort"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
)
// OrphanView is cached weights on one node that no catalog model accounts for.
@@ -71,12 +74,100 @@ type CollectRequest struct {
Node string `json:"node,omitempty"`
}
// CollectResult is what was deleted on one node, or why nothing was.
// CollectResult is what was deleted on one node, or why nothing was: what
// the node no longer lists once asked, not what it was asked to delete.
type CollectResult struct {
Node string `json:"node"`
Files int `json:"files"`
Bytes int64 `json:"bytes"`
Error string `json:"error,omitempty"`
// Left and LeftBytes are what the node still lists: refused, or not
// gone by the time the sweep stopped waiting. Its agent's log says why.
Left int `json:"left,omitempty"`
LeftBytes int64 `json:"left_bytes,omitempty"`
Error string `json:"error,omitempty"`
}
// confirmWait is how long a sweep waits for the nodes to list their files
// without the ones deleted: they scan theirs every heartbeat, five seconds
// by default, and a large set takes a moment to delete.
var confirmWait = 30 * time.Second
// removal is what one node is asked to delete, with each file's size.
type removal struct {
node string
paths []string
size map[string]int64
}
// removeConfirmed asks each node to delete its paths and says what each then
// no longer lists. Counted as asked, a sweep said "25 GB freed" of files the
// node had refused and kept.
func (a *API) removeConfirmed(ctx context.Context, rm control.WeightRemover, rs []removal) []CollectResult {
out := make([]CollectResult, len(rs))
pending := map[int]bool{}
for i, r := range rs {
out[i].Node = r.node
if err := rm.RemoveWeights(ctx, r.node, r.paths); err != nil {
out[i].Error = err.Error()
for _, p := range r.paths {
out[i].Left++
out[i].LeftBytes += r.size[p]
}
continue
}
pending[i] = true
}
deadline := time.Now().Add(confirmWait)
for {
for i := range pending {
if left := a.stillListed(rs[i]); len(left) == 0 {
delete(pending, i)
}
}
if len(pending) == 0 || !time.Now().Before(deadline) {
break
}
select {
case <-ctx.Done():
deadline = time.Now()
case <-time.After(time.Second):
}
}
for i, r := range rs {
if out[i].Error != "" {
continue
}
left := a.stillListed(r)
for _, p := range r.paths {
if left[p] {
out[i].Left++
out[i].LeftBytes += r.size[p]
} else {
out[i].Files++
out[i].Bytes += r.size[p]
}
}
}
return out
}
// stillListed is which of a removal's paths its node still lists.
func (a *API) stillListed(r removal) map[string]bool {
left := map[string]bool{}
n, ok := a.ctl.Inventory().Get(r.node)
if !ok {
return left
}
want := map[string]bool{}
for _, p := range r.paths {
want[p] = true
}
for _, w := range n.Weights {
if want[w.Path] {
left[w.Path] = true
}
}
return left
}
// collectWeights deletes them.
@@ -97,20 +188,26 @@ func (a *API) collectWeights(w http.ResponseWriter, r *http.Request) {
// An empty body is the "all nodes" case and must not be an error.
_ = json.NewDecoder(r.Body).Decode(&body)
out := make([]CollectResult, 0)
// A node that has gone since the list was built is reported rather than
// failing the whole sweep: the other nodes' space is still worth
// freeing, and this one will be offered again next time.
var rs []removal
for _, o := range a.Orphans() {
if body.Node != "" && o.Node != body.Node {
continue
}
res := CollectResult{Node: o.Node, Files: len(o.Paths), Bytes: o.Bytes}
// A node that has gone since the list was built is reported rather than
// failing the whole sweep: the other nodes' space is still worth
// freeing, and this one will be offered again next time.
if err := rm.RemoveWeights(r.Context(), o.Node, o.Paths); err != nil {
res.Error = err.Error()
res.Files, res.Bytes = 0, 0
}
out = append(out, res)
rs = append(rs, removal{node: o.Node, paths: o.Paths, size: a.sizesOn(o.Node)})
}
writeJSON(w, http.StatusOK, out)
writeJSON(w, http.StatusOK, a.removeConfirmed(r.Context(), rm, rs))
}
// sizesOn is each file a node lists, by path, with its size.
func (a *API) sizesOn(node string) map[string]int64 {
out := map[string]int64{}
if n, ok := a.ctl.Inventory().Get(node); ok {
for _, w := range n.Weights {
out[w.Path] = w.SizeBytes
}
}
return out
}
+71
View File
@@ -0,0 +1,71 @@
package adminapi
import (
"context"
"encoding/json"
"log/slog"
"net/http"
"net/http/httptest"
"slices"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
)
// deletingDispatcher deletes what it is asked to as a node would, by its
// files dropping out of what the node lists — but for refused, which it
// keeps, as an agent keeps a file it cannot prove it fetched.
type deletingDispatcher struct {
nullDispatcher
inv *inventory.Store
refused string
}
func (d *deletingDispatcher) RemoveWeights(_ context.Context, node string, paths []string) error {
n, _ := d.inv.Get(node)
n.Weights = slices.DeleteFunc(n.Weights, func(w discovery.LocalWeights) bool {
return w.Path != d.refused && slices.Contains(paths, w.Path)
})
d.inv.Upsert(n)
return nil
}
// What a sweep says it freed is what the node no longer lists: a file it
// kept is said to be left, not counted as freed.
func TestASweepSaysWhatWasDeletedNotWhatWasAsked(t *testing.T) {
confirmWait = 1500 * time.Millisecond
t.Cleanup(func() { confirmWait = 30 * time.Second })
gone := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/old/v1/old.gguf")
set := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/big/v1")
kept := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/stuck/v1/stuck.gguf")
cat, _ := catalog.New(nil)
d := &deletingDispatcher{refused: kept}
ctl := control.New(control.DefaultConfig(), catalog.NewLive(cat), d, nullSink{}, slog.New(slog.DiscardHandler))
d.inv = ctl.Inventory()
node := gpuNode("n1", 1)
node.Weights = []discovery.LocalWeights{{Path: gone, SizeBytes: 1 << 30}, {Path: set, SizeBytes: 20 << 30},
{Path: kept, SizeBytes: 4 << 30}}
ctl.Inventory().Upsert(node)
a := New(ctl)
rec := httptest.NewRecorder()
a.collectWeights(rec, httptest.NewRequest(http.MethodPost, "/api/v1/weights/collect", strings.NewReader(`{"node":"n1"}`)))
var got []CollectResult
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatalf("%v: %s", err, rec.Body.String())
}
if len(got) != 1 {
t.Fatalf("results %+v", got)
}
r := got[0]
if r.Files != 2 || r.Bytes != 21<<30 || r.Left != 1 || r.LeftBytes != 4<<30 {
t.Errorf("result %+v: want 2 files and 21 GB freed, 1 file of 4 GB left", r)
}
}
+1 -1
View File
@@ -36,7 +36,7 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
n.Host = discovery.Host{
OSID: h.GetOsId(), OSVersion: h.GetOsVersion(), OSCodename: h.GetOsCodename(),
Arch: h.GetArch(), CPUFlags: h.GetCpuFlags(), NVIDIADriver: h.GetNvidiaDriver(),
MemBandwidthGBs: h.GetMemBandwidthGbs(),
MemBandwidthGBs: h.GetMemBandwidthGbs(), MemBandwidthCoreGBs: h.GetMemBandwidthCoreGbs(),
}
}
for _, w := range r.GetWeights() {
+15 -1
View File
@@ -84,6 +84,8 @@ type Service struct {
agentSettings func(node string) string
// train is told how training jobs go; see SetTrain.
train TrainHandler
// slurm is told of the nodes' part in Slurm; see SetSlurm.
slurm SlurmHandler
// logWaits are the log requests awaiting a reply, by request id.
logMu sync.Mutex
@@ -308,6 +310,10 @@ func (s *Service) Connect(
return err
}
}
// Its part in Slurm, sent by the Slurm service's next pass.
if h := s.slurmHandler(); h != nil {
h.OnRegister(nodeID)
}
// Recover replica state from what this node is already running, before any
// reconciliation pass can conclude the cluster is empty and place
@@ -515,6 +521,11 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
h.OnTrainStatus(nodeID, m.TrainStatus)
}
case *llmgwv1.AgentMessage_SlurmStatus:
if h := s.slurmHandler(); h != nil {
h.OnSlurmStatus(nodeID, m.SlurmStatus)
}
case *llmgwv1.AgentMessage_TrainArtifact:
if h := s.trainHandler(); h != nil {
h.OnTrainArtifact(nodeID, m.TrainArtifact)
@@ -547,7 +558,7 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
storage := hb.GetStorage()
memMB := int(hb.GetMemoryTotalMb())
cpus := int(hb.GetCpuCount())
bw := hb.GetMemBandwidthGbs()
bw, coreBW := hb.GetMemBandwidthGbs(), hb.GetMemBandwidthCoreGbs()
if len(used) == 0 && len(foreign) == 0 && len(problems) == 0 && len(storage) == 0 &&
!hb.GetWeightsChanged() && memMB == 0 && cpus == 0 && !hb.GetEnginesChanged() &&
!hb.GetProvisionChanged() && hb.GetLoad() == nil && len(hb.GetIbPorts()) == 0 &&
@@ -626,6 +637,9 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
if bw > 0 {
n.Host.MemBandwidthGBs = bw
}
if coreBW > 0 {
n.Host.MemBandwidthCoreGBs = coreBW
}
if cpus > 0 && cpus != n.CPUCount {
if n.CPUCount > 0 {
s.log.Info("node CPUs changed", "node", nodeID, "from", n.CPUCount, "to", cpus)
+25
View File
@@ -0,0 +1,25 @@
package agentsvc
import (
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
)
// SlurmHandler is told of the nodes' part in Slurm (BACKLOG P).
type SlurmHandler interface {
OnSlurmStatus(node string, st *llmgwv1.SlurmStatus)
// OnRegister is a node's new stream, which is told again what it is.
OnRegister(node string)
}
// SetSlurm gives the service what Slurm's messages go to.
func (s *Service) SetSlurm(h SlurmHandler) {
s.mu.Lock()
s.slurm = h
s.mu.Unlock()
}
func (s *Service) slurmHandler() SlurmHandler {
s.mu.RLock()
defer s.mu.RUnlock()
return s.slurm
}
+56
View File
@@ -267,6 +267,12 @@ type Profile struct {
Weight float64 `json:"weight,omitempty" koanf:"weight"`
// Priority orders profiles during placement; highest that fits wins.
Priority int `json:"priority,omitempty" koanf:"priority"`
// Replicas, set on any of a model's profiles, is how many of its
// replicas run on this one: seven on cards and four on CPUs, all
// answering under the model's name, rather than each replica on the
// best profile that fits. The model's scaling is then the sum, min and
// max alike; a profile left at zero runs none.
Replicas int `json:"replicas,omitempty" koanf:"replicas"`
// StartupTimeout bounds model loading. A 70B model across two nodes takes
// tens of minutes, so this is per profile rather than global.
StartupTimeout time.Duration `json:"startup_timeout,omitempty" koanf:"startup_timeout"`
@@ -1024,10 +1030,57 @@ func (m Model) normalize() Model {
return m
}
// Split says whether the model's replicas are counted per profile: any of
// its profiles sets replicas.
func (m Model) Split() bool {
return slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Replicas > 0 })
}
// SplitTotal is the replicas its profiles ask for together.
func (m Model) SplitTotal() int {
n := 0
for _, p := range m.Profiles {
n += max(p.Replicas, 0)
}
return n
}
// ProfileBase is the catalog profile a placement profile stands for: one
// named by the card it needs is placed as name@class.
func ProfileBase(name string) string {
base, _, _ := strings.Cut(name, "@")
return base
}
// validateSplit refuses replicas per profile that the model's scaling does
// not add up to.
func (m Model) validateSplit() error {
for _, p := range m.Profiles {
if p.Replicas < 0 {
return fmt.Errorf("model %q profile %q: replicas %d is negative", m.Name, p.Name, p.Replicas)
}
}
if !m.Split() || m.Stopped != nil {
return nil
}
n := m.SplitTotal()
if m.Scaling.Min != n || m.Scaling.Max != n {
return fmt.Errorf("model %q: its profiles' replicas add up to %d, so its scaling is min %d, max %d, not %d and %d",
m.Name, n, n, n, m.Scaling.Min, m.Scaling.Max)
}
if len(m.Scaling.Schedule) > 0 {
return fmt.Errorf("model %q: replicas per profile and a schedule are one or the other", m.Name)
}
return nil
}
func (m Model) validate() error {
if m.Name == "" {
return fmt.Errorf("a model has no name")
}
if err := m.validateSplit(); err != nil {
return err
}
if m.Source == "" {
return fmt.Errorf("model %q: source is required", m.Name)
}
@@ -1510,6 +1563,9 @@ func (m Model) Simulated() bool {
// happened, and the answer to "why is this one on the wrong node" was in
// somebody's memory of what they edited an hour ago.
func (p Profile) Fingerprint() string {
// How many replicas run on it is not how each runs: changing the count
// must not mark the running ones stale.
p.Replicas = 0
b, err := json.Marshal(p)
if err != nil {
// A profile that will not marshal cannot be compared, and saying so by
+13 -1
View File
@@ -296,7 +296,7 @@ func (s *Server) Start(req Request) (Job, error) {
}
req.sized = req.CtxPool > 0 || req.ContextLength > 0
if req.CtxPool <= 0 {
req.CtxPool = 16384
req.CtxPool = DefaultCtxPool
}
if req.ContextLength <= 0 {
req.ContextLength = 4096
@@ -572,6 +572,14 @@ func (s *Server) run(j *Job) {
// On one card, the weights and the context's cache have to fit it: a
// 70B with a 65536-token pool on a 24 GB card loaded its weights and then
// failed to make its context, every time, as did an 8B with 131072.
// Asked for the CPU, and slow there: said, not refused — the request
// named it — so the job's log says why its replies take minutes.
if len(layout) > 0 && layout[0].CPU && layout[0].TokensPerSecond > 0 &&
layout[0].TokensPerSecond < modelplan.CPUSlowTPS {
tps := layout[0].TokensPerSecond
s.step(j, "check", Running, fmt.Sprintf("on the CPU about %.1f tokens a second for one reply: "+
"a reply of 500 tokens takes about %.0f seconds; a card would be many times faster", tps, 500/tps))
}
if len(layout) > 0 && layout[0].Class != "" && !layout[0].CPU {
fitted, why, err := fitPoolToCard(req, header, set.Bytes, layout[0].Class, max(layout[0].GPUsPerNode, 1))
if err != nil {
@@ -723,6 +731,10 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
return out
}
// DefaultCtxPool is llama.cpp's pool when a request names none: a context
// asked for longer than it needs a pool of its own.
const DefaultCtxPool = 16384
// llamaArgs size llama.cpp's pool and its slots together. The pool is shared
// by the slots, so slots of the context the catalog promises are as many as
// the pool holds: the engine's default of four turned a 32768 pool into
+9
View File
@@ -43,6 +43,9 @@ type Plan struct {
License string `json:"license,omitempty"`
Params int64 `json:"params,omitempty"`
Context int `json:"context"`
// Trained is the context the model was trained for, the most a request
// may use; 0 when neither its header nor its config says.
Trained int `json:"trained,omitempty"`
// KVBytesPerToken is one token of context, the whole model: what an
// option needs for another context is worked out from it. 0 is unknown.
KVBytesPerToken int64 `json:"kv_bytes_per_token,omitempty"`
@@ -149,7 +152,9 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
p.Servable, p.Reason = false, v.Reason
}
shape.KVBytesPerToken = v.Header.KVBytesPerToken()
p.Trained = int(v.Header.ContextLength) //nolint:gosec // a header's context length
shape.KVHeads = int(v.Header.HeadCountKV) //nolint:gosec // a header's head count
shape.ActiveShare = v.Header.ActiveShare()
if p.Arch == "" {
p.Arch = v.Header.Architecture
}
@@ -166,6 +171,7 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
m, err := modelplan.FetchModel(ctx, repo.ID)
if err == nil {
shape.KVBytesPerToken = m.KVBytesPerToken(modelplan.BF16)
p.Trained = m.MaxContext
shape.KVHeads = m.KVHeads
shape.Experts = m.Experts
if p.Params == 0 {
@@ -181,6 +187,9 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
p.Note = "the model's shape was not readable, so the context's memory is not counted"
}
shape.Params = p.Params
if shape.ActiveShare == 0 || shape.ActiveShare == 1 {
shape.ActiveShare = modelplan.ActiveShare(repo.ID, p.Params)
}
p.KVBytesPerToken, p.Experts = shape.KVBytesPerToken, shape.Experts
p.Options = modelplan.Options(arts, shape, s.deps.Pool(), p.Context)
if !p.Servable {
+100
View File
@@ -22,6 +22,7 @@ import (
"fmt"
"log/slog"
"slices"
"strings"
"sync"
"time"
@@ -249,6 +250,13 @@ func (s *Service) Remove(ctx context.Context, name string) error {
// Scale changes how many replicas a model added at runtime runs.
func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by string) (catalog.Model, error) {
final, err := s.update(ctx, name, by, "change its scaling", func(m *catalog.Model) error {
// Replicas counted per profile stay so only for the same total, one
// way: any other scaling is the model's again, on whatever fits.
if m.Split() && (sc.Min != sc.Max || sc.Min != m.SplitTotal()) {
for i := range m.Profiles {
m.Profiles[i].Replicas = 0
}
}
m.Scaling = sc
// A number set is what it runs now: a stopped model is started by it.
m.Stopped = nil
@@ -261,6 +269,98 @@ func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by
return final, err
}
// CPUProfile is a profile on CPUs to add to a llama.cpp model, for replicas
// counted per profile: what a replica of it reserves.
type CPUProfile struct {
Name string `json:"name,omitempty"`
MemoryGB int `json:"memory_gb"`
CPUCores int `json:"cpu_cores,omitempty"`
}
// SetProfileReplicas sets how many of a model's replicas run on each of its
// profiles: seven on cards and four on CPUs, under the one name. A profile
// not named keeps its count. cpu, for a llama.cpp model with no profile on
// CPUs, adds one — its card profile without the card, preferred below every
// other, so the router fills the cards first and the CPUs after. Running
// replicas carry on; those over a count stop once the ones short have
// started.
func (s *Service) SetProfileReplicas(ctx context.Context, name string, counts map[string]int, cpu *CPUProfile,
by string,
) (catalog.Model, error) {
final, err := s.update(ctx, name, by, "count its replicas per profile", func(m *catalog.Model) error {
if cpu != nil {
if cpu.Name == "" {
cpu.Name = "cpu"
}
if !slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Name == cpu.Name }) {
p, err := cpuProfileOf(*m, *cpu)
if err != nil {
return err
}
m.Profiles = append(m.Profiles, p)
}
}
for prof, n := range counts {
i := slices.IndexFunc(m.Profiles, func(p catalog.Profile) bool { return p.Name == prof })
if i < 0 {
return fmt.Errorf("model %q has no profile %q", name, prof)
}
if n < 0 {
return fmt.Errorf("profile %q: %d replicas", prof, n)
}
m.Profiles[i].Replicas = n
}
n := m.SplitTotal()
if n == 0 {
return fmt.Errorf("none of %q's profiles would run a replica; llmgw ctl model stop %s keeps it without", name, name)
}
sc := catalog.Scaling{Min: n, Max: n, IdleTimeout: m.Scaling.IdleTimeout}
if m.Stopped != nil {
sc.IdleTimeout = m.Stopped.IdleTimeout
m.Stopped = &sc
} else {
m.Scaling = sc
}
return nil
})
if err == nil {
s.log.Info("replicas counted per profile", "model", name, "counts", counts, "by", by)
}
return final, err
}
// cpuProfileOf is a model's llama.cpp profile on CPUs: the same engine and
// arguments, without the card or what spreads layers over cards.
func cpuProfileOf(m catalog.Model, cpu CPUProfile) (catalog.Profile, error) {
i := slices.IndexFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" })
if i < 0 {
return catalog.Profile{}, fmt.Errorf("model %q is not served by llama.cpp; only a GGUF model runs on CPUs", m.Name)
}
if cpu.MemoryGB <= 0 {
return catalog.Profile{}, errors.New("a profile on CPUs needs the memory a replica reserves")
}
p := m.Profiles[i]
lowest := p.Priority
for _, o := range m.Profiles {
lowest = min(lowest, o.Priority)
}
out := catalog.Profile{Name: cpu.Name, Engine: p.Engine, EngineVersion: p.EngineVersion, Source: p.Source,
Env: p.Env, NodeSelector: p.NodeSelector, StartupTimeout: p.StartupTimeout, Weight: p.Weight,
Priority: lowest - 10, MinMemoryGB: cpu.MemoryGB, CPUCores: cpu.CPUCores}
for j := 0; j < len(p.Args); j++ {
switch a := p.Args[j]; {
case slices.Contains([]string{"--n-gpu-layers", "-ngl", "--split-mode", "-sm", "--tensor-split", "-ts",
"--main-gpu", "-mg"}, a):
j++ // and its value
case strings.HasPrefix(a, "--n-gpu-layers=") || strings.HasPrefix(a, "--split-mode=") ||
strings.HasPrefix(a, "--tensor-split=") || strings.HasPrefix(a, "--main-gpu="):
default:
out.Args = append(out.Args, a)
}
}
return out, nil
}
// Stop takes a model out of service and keeps it: none of its replicas run,
// and what it was scaled to is kept for Start.
func (s *Service) Stop(ctx context.Context, name, by string) (catalog.Model, error) {
@@ -5,6 +5,7 @@ import (
"encoding/json"
"errors"
"log/slog"
"slices"
"strings"
"testing"
"time"
@@ -458,3 +459,44 @@ func TestAStrataModelReservesTheMemoryItsExpertsTake(t *testing.T) {
t.Errorf("an unknown size: %v", err)
}
}
// A GGUF model on a card gets a profile on CPUs, and seven replicas on cards
// beside four on CPUs; a scaling of another total puts it back on whatever
// fits.
func TestReplicasCountedPerProfile(t *testing.T) {
s, _, _ := newService(t)
m := catalog.Model{Name: "g", Source: "/models/g.gguf", Scaling: catalog.Scaling{Min: 1, Max: 1},
Profiles: []catalog.Profile{{Name: "gpu", Engine: "llamacpp", GPUClass: "l40s-48g", GPUsPerNode: 2,
Priority: 100, Args: []string{"--ctx-size", "16384", "--n-gpu-layers", "999", "--split-mode=layer"}}}}
if err := s.Add(t.Context(), m, "ilker"); err != nil {
t.Fatal(err)
}
if _, err := s.SetProfileReplicas(t.Context(), "g", map[string]int{"gpu": 7, "cpu": 4}, nil, "ilker"); err == nil {
t.Fatal("a count for a profile it does not have was taken")
}
got, err := s.SetProfileReplicas(t.Context(), "g", map[string]int{"gpu": 7, "cpu": 4},
&CPUProfile{MemoryGB: 24, CPUCores: 8}, "ilker")
if err != nil {
t.Fatal(err)
}
if len(got.Profiles) != 2 || got.Scaling.Min != 11 || got.Scaling.Max != 11 {
t.Fatalf("profiles %+v, scaling %+v", got.Profiles, got.Scaling)
}
cpu := got.Profiles[1]
if cpu.Name != "cpu" || cpu.GPUClass != "" || cpu.GPUsPerNode != 0 || cpu.Replicas != 4 || cpu.Priority >= 100 ||
cpu.MinMemoryGB != 24 || cpu.CPUCores != 8 || !slices.Equal(cpu.Args, []string{"--ctx-size", "16384"}) {
t.Errorf("the profile on CPUs: %+v", cpu)
}
// Stopped and started, it comes back split.
_, _ = s.Stop(t.Context(), "g", "ilker")
if got, _ = s.Start(t.Context(), "g", "ilker"); got.Scaling.Min != 11 || got.Profiles[0].Replicas != 7 {
t.Errorf("started: %+v", got)
}
// The same total keeps the split; another one ends it.
if got, _ = s.Scale(t.Context(), "g", catalog.Scaling{Min: 11, Max: 11}, "ilker"); !got.Split() {
t.Error("the same total ended the split")
}
if got, _ = s.Scale(t.Context(), "g", catalog.Scaling{Min: 3, Max: 3}, "ilker"); got.Split() {
t.Error("another total kept the split")
}
}
+51
View File
@@ -154,6 +154,9 @@ func (r Result) Explain() string {
// Scheduler places replicas onto an inventory.
type Scheduler struct {
inv *inventory.Store
// shun is where a replica of the model being placed failed to start;
// set on a copy by Away, never on the scheduler the reconciler shares.
shun []Spot
// now is the clock power readings are judged fresh by; nil is the wall
// clock.
now func() time.Time
@@ -162,6 +165,41 @@ type Scheduler struct {
// New builds a scheduler over an inventory.
func New(inv *inventory.Store) *Scheduler { return &Scheduler{inv: inv} }
// Spot is where a replica ran: a node, and the cards it held there (none
// for one on the node's CPU).
type Spot struct {
Node string
GPUs []int
}
// Away is this scheduler placing nowhere in spots: not those cards, nor a
// CPU replica on those nodes. A copy, so the one shared is not changed.
//
// For a model that failed to start somewhere: the same inputs pick the same
// node and the same card every time, so without it every retry went back to
// where the last one had failed until the model was parked, with another
// card or node free all along.
func (s *Scheduler) Away(spots []Spot) *Scheduler {
c := *s
c.shun = spots
return &c
}
// shunned is the cards of node n a model failed on, and whether it failed
// on the node's CPU.
func (s *Scheduler) shunned(node string) (gpus []int, cpu bool) {
for _, sp := range s.shun {
if sp.Node != node {
continue
}
if len(sp.GPUs) == 0 {
cpu = true
}
gpus = append(gpus, sp.GPUs...)
}
return gpus, cpu
}
// Place picks the first profile, in priority order, that fits.
//
// avoid lists nodes already hosting this model, so replicas of one model spread
@@ -686,7 +724,12 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
}
}
badGPUs, badCPU := s.shunned(n.ID)
if p.CPUOnly() {
if badCPU {
reject("a replica of it failed to start here; trying elsewhere first")
continue
}
// Every ready node can run it; there is no card to be free.
c := candidate{node: n}
if why := budget.check(n, s.shardWatts(p, c)); why != "" {
@@ -703,8 +746,16 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
if g.Class() != p.GPUClass || !slices.Contains(freeIdx, g.Index) {
continue
}
if slices.Contains(badGPUs, g.Index) {
continue
}
free = append(free, g)
}
if len(free) < p.GPUsPerNode && len(badGPUs) > 0 {
reject(fmt.Sprintf("has %d free %s GPUs besides the ones a replica of it failed to start on, needs %d",
len(free), p.GPUClass, p.GPUsPerNode))
continue
}
if len(free) < p.GPUsPerNode {
reject(fmt.Sprintf("has %d free %s GPUs, needs %d", len(free), p.GPUClass, p.GPUsPerNode))
continue
+80
View File
@@ -0,0 +1,80 @@
package reconciler
import (
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
)
// failAndRetry fails the last replica started, and returns where the next
// one is started once the backoff is over.
func failAndRetry(t *testing.T, h *harness) ShardCommand {
t.Helper()
started := h.disp.startedShards()
if len(started) == 0 {
t.Fatal("nothing was started")
}
s := started[len(started)-1]
h.r.OnShardStatus(s.ReplicaID, s.ShardID, supervisor.Failed, "the engine exited")
h.r.Reconcile(t.Context()) // reaps it
h.disp.reset()
h.advance(DefaultConfig().MaxBackoff)
h.r.Reconcile(t.Context())
again := h.disp.startedShards()
if len(again) == 0 {
t.Fatal("it was not tried again")
}
return again[0]
}
func oneCard() catalog.Model {
return testModel(1, catalog.Profile{Name: "one", GPUClass: "h100-80g", GPUsPerNode: 1})
}
func TestAReplicaThatFailedIsTriedOnAnotherNode(t *testing.T) {
// The same inputs give the same node every time: an on-demand 3D model
// that failed to load was started on that node again, and again, until
// it was parked, with another node free all along.
h := newHarness(t, oneCard(), gpuNode("1", 1), gpuNode("2", 1))
h.r.Reconcile(t.Context())
first := h.disp.startedShards()[0].NodeID
if next := failAndRetry(t, h); next.NodeID == first {
t.Errorf("tried on node %s again, where it had just failed", first)
}
}
func TestAReplicaThatFailedIsTriedOnAnotherCardOfTheNode(t *testing.T) {
h := newHarness(t, oneCard(), gpuNode("1", 2))
h.r.Reconcile(t.Context())
first := h.disp.startedShards()[0].Spec.GPUIndices
next := failAndRetry(t, h)
if len(next.Spec.GPUIndices) != 1 || len(first) != 1 || next.Spec.GPUIndices[0] == first[0] {
t.Errorf("tried on card %v again, where it had just failed", next.Spec.GPUIndices)
}
}
func TestWithNowhereElseItIsTriedWhereItFailed(t *testing.T) {
// A preference, not a rule: a model that has one place is not left
// unplaced because it failed there once.
h := newHarness(t, oneCard(), gpuNode("1", 1))
h.r.Reconcile(t.Context())
if next := failAndRetry(t, h); next.NodeID != "1" {
t.Errorf("started on %q", next.NodeID)
}
}
func TestComingUpForgetsWhereItFailed(t *testing.T) {
h := newHarness(t, oneCard(), gpuNode("1", 1), gpuNode("2", 1))
h.r.Reconcile(t.Context())
next := failAndRetry(t, h)
h.r.OnShardStatus(next.ReplicaID, next.ShardID, supervisor.Ready, "")
h.r.mu.Lock()
left := len(h.r.failedAt["llama"])
h.r.mu.Unlock()
if left != 0 {
t.Errorf("a model that came up still remembers %d failures' places", left)
}
}
+1
View File
@@ -61,6 +61,7 @@ func (r *Reconciler) Retry(model string) bool {
delete(r.parked, model)
r.attempts[model] = 0
delete(r.nextAttempt, model)
delete(r.failedAt, model)
r.mu.Unlock()
r.Notify()
return was
+107 -5
View File
@@ -7,6 +7,8 @@ import (
"log/slog"
"math/rand/v2"
"slices"
"strconv"
"strings"
"sync"
"sync/atomic"
"time"
@@ -251,6 +253,9 @@ type Reconciler struct {
nextAttempt map[string]time.Time
// attempts counts consecutive failures per model, for the backoff.
attempts map[string]int
// failedAt is where a model's replicas failed to start since it last
// came up, most recent last: the next one is placed elsewhere if it can.
failedAt map[string][]placement.Spot
// lastExplain keeps the most recent placement failure per model, so
// `ctl explain-placement` can answer without re-running placement.
lastExplain map[string]placement.Result
@@ -334,6 +339,7 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
replicas: map[string]*Replica{},
nextAttempt: map[string]time.Time{},
attempts: map[string]int{},
failedAt: map[string][]placement.Spot{},
reservedPorts: map[int]struct{}{},
wanted: map[string]time.Time{},
coldStart: map[string]time.Duration{},
@@ -726,6 +732,7 @@ func (r *Reconciler) reapFailed(ctx context.Context) {
// replica is about to disappear, and the backoff has to survive it.
r.attempts[rep.Model]++
r.nextAttempt[rep.Model] = r.now().Add(r.backoff(r.attempts[rep.Model]))
r.noteFailedAt(rep)
if n := r.attempts[rep.Model]; n >= r.cfg.ParkAfter {
if _, already := r.parked[rep.Model]; !already {
m, _ := r.cat.Load().Resolve(rep.Model)
@@ -875,6 +882,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
delete(r.parked, m.Name)
r.attempts[m.Name] = 0
delete(r.nextAttempt, m.Name)
delete(r.failedAt, m.Name)
isParked, blockedUntil = false, time.Time{}
r.log.Info("model unparked: its definition changed", "model", m.Name)
}
@@ -886,14 +894,25 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
}
desired := r.desiredFor(m)
// Replicas counted per profile: one short on its profile is started even
// with the total right — before one of a profile with too many is
// stopped, which the total going over then does — so changing seven on
// cards and four on CPUs to nine and two never runs fewer than eleven.
var deficit map[string]bool
var surplus []string
split := m.Split() && desired == m.SplitTotal()
if split {
deficit, surplus = r.splitGaps(m)
}
needUp := len(liveIDs) < desired || split && len(deficit) > 0 && len(liveIDs) == desired
outcome := notTried
switch {
case len(liveIDs) < desired && isParked:
case needUp && isParked:
// Not placed. Its failure is on every status the operator reads.
case len(liveIDs) < desired && r.isHeld(m.Name):
case needUp && r.isHeld(m.Name):
// It gave up room to a model of a higher class, which is placed
// first; taking it back now would undo the preemption.
case len(liveIDs) < desired:
case needUp:
if r.now().Before(blockedUntil) {
// Still in backoff. Rebuilding immediately would hot-loop on a
// model that cannot start — a missing weights path, say — and bury
@@ -912,7 +931,11 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
attribute.String("llmgw.model", m.Name),
attribute.Int("llmgw.replicas", len(liveIDs)),
attribute.Int("llmgw.desired", desired))
r.scaleDown(sctx, liveIDs[desired:], "more replicas than desired")
excess := liveIDs[desired:]
if split {
excess = pickSurplus(surplus, liveIDs, len(liveIDs)-desired)
}
r.scaleDown(sctx, excess, "more replicas than desired")
span.End()
}
@@ -976,7 +999,40 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
attribute.String("llmgw.model", m.Name),
attribute.StringSlice("llmgw.avoid", avoid))
profiles := r.orderedProfiles(m)
res := r.sched.Place(profiles, avoid)
if m.Split() {
// Only the profiles short of their replicas.
deficit, _ := r.splitGaps(m)
profiles = slices.DeleteFunc(profiles, func(p placement.Profile) bool {
return !deficit[catalog.ProfileBase(p.Name)]
})
if len(profiles) == 0 {
pspan.End()
return notTried
}
}
// Away from where it last failed to start, when there is anywhere else:
// the same inputs give the same node and card, and an on-demand model
// that failed there was otherwise started there again until it was
// parked. Only a preference too: nowhere else, and it is tried there.
r.mu.Lock()
failed := slices.Clone(r.failedAt[m.Name])
r.mu.Unlock()
var res placement.Result
if len(failed) > 0 {
away := r.sched.Away(failed)
res = away.Place(profiles, avoid)
if res.Placement == nil && len(avoid) > 0 {
res = away.Place(profiles, nil)
}
pspan.SetAttributes(attribute.Bool("llmgw.away_from_failure", res.Placement != nil))
if res.Placement != nil {
r.log.Info("placing a model away from where it failed to start", "model", m.Name,
"failed_on", spotsText(failed), "now_on", spotsText(spotsOf(res.Placement)))
}
}
if res.Placement == nil {
res = r.sched.Place(profiles, avoid)
}
if res.Placement == nil && len(avoid) > 0 {
// Spreading across failure domains is a preference, not a rule. If it
// is the only thing standing in the way, take the placement anyway:
@@ -1028,6 +1084,51 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
return placed
}
// maxFailedSpots is how many places a model's failures are remembered at.
const maxFailedSpots = 8
// noteFailedAt remembers where a failed replica ran. Called under r.mu.
func (r *Reconciler) noteFailedAt(rep *Replica) {
spots := r.failedAt[rep.Model]
for _, sh := range rep.Shards {
sp := placement.Spot{Node: sh.NodeID, GPUs: slices.Clone(sh.GPUIndices)}
spots = slices.DeleteFunc(spots, func(o placement.Spot) bool {
return o.Node == sp.Node && slices.Equal(o.GPUs, sp.GPUs)
})
spots = append(spots, sp)
}
if len(spots) > maxFailedSpots {
spots = spots[len(spots)-maxFailedSpots:]
}
r.failedAt[rep.Model] = spots
}
// spotsOf is where a placement puts its shards.
func spotsOf(pl *placement.Placement) []placement.Spot {
var out []placement.Spot
for _, n := range pl.Nodes {
out = append(out, placement.Spot{Node: n.NodeID, GPUs: n.GPUIndices})
}
return out
}
// spotsText is spots as a log reads them: "node01 gpu 0,1; node02 cpu".
func spotsText(spots []placement.Spot) string {
parts := make([]string, 0, len(spots))
for _, sp := range spots {
if len(sp.GPUs) == 0 {
parts = append(parts, sp.Node+" cpu")
continue
}
gs := make([]string, len(sp.GPUs))
for i, g := range sp.GPUs {
gs[i] = strconv.Itoa(g)
}
parts = append(parts, sp.Node+" gpu "+strings.Join(gs, ","))
}
return strings.Join(parts, "; ")
}
// scaleOutcome is what one scaleUp did.
type scaleOutcome int
@@ -1407,6 +1508,7 @@ func (r *Reconciler) markShard(replicaID, shardID string, st supervisor.State, e
// counting from scratch rather than from an old streak.
r.attempts[rep.Model] = 0
delete(r.nextAttempt, rep.Model)
delete(r.failedAt, rep.Model)
if !rep.CreatedAt.IsZero() {
r.coldStart[rep.Model] = r.now().Sub(rep.CreatedAt)
}
+64
View File
@@ -0,0 +1,64 @@
package reconciler
import (
"slices"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
)
// splitGaps is, for a model whose replicas are counted per profile, the
// profiles short of theirs and the replicas over: of a profile with more
// than it asks, or of one it no longer has. Counted as reconcileModel
// counts — a replica being retired is not one it has — so a replica
// restarted is replaced on its own profile.
func (r *Reconciler) splitGaps(m catalog.Model) (deficit map[string]bool, surplus []string) {
live := map[string][]string{}
r.mu.Lock()
for _, rep := range r.replicas {
if rep.Model == m.Name && rep.State.Counts() && rep.RetiringAt.IsZero() {
b := catalog.ProfileBase(rep.Profile)
live[b] = append(live[b], rep.ID)
}
}
r.mu.Unlock()
deficit = map[string]bool{}
for _, p := range m.Profiles {
ids := live[p.Name]
delete(live, p.Name)
slices.Sort(ids)
switch {
case len(ids) < p.Replicas:
deficit[p.Name] = true
case len(ids) > p.Replicas:
surplus = append(surplus, ids[p.Replicas:]...)
}
}
for _, ids := range live {
surplus = append(surplus, ids...)
}
slices.Sort(surplus)
return deficit, surplus
}
// pickSurplus is n of the live replicas to stop: those over their
// profile's count first, then any.
func pickSurplus(surplus, live []string, n int) []string {
out := make([]string, 0, n)
for _, id := range surplus {
if len(out) == n {
return out
}
if slices.Contains(live, id) {
out = append(out, id)
}
}
for _, id := range live {
if len(out) == n {
break
}
if !slices.Contains(out, id) {
out = append(out, id)
}
}
return out
}
+85
View File
@@ -0,0 +1,85 @@
package reconciler
import (
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
)
// splitModel runs gpu replicas on cards and cpu on CPUs, under one name.
func splitModel(gpu, cpu int) catalog.Model {
return catalog.Model{Name: "m", Source: "/models/m.gguf",
Scaling: catalog.Scaling{Min: gpu + cpu, Max: gpu + cpu},
Profiles: []catalog.Profile{
{Name: "gpu", Engine: "llamacpp", GPUClass: "h100-80g", GPUsPerNode: 1, Priority: 100, Replicas: gpu},
{Name: "cpu", Engine: "llamacpp", Priority: 10, Replicas: cpu},
}}
}
// byProfile counts the model's live replicas on each profile.
func byProfile(h *harness) map[string]int {
out := map[string]int{}
for _, rep := range h.r.Replicas() {
if rep.State.Counts() && rep.RetiringAt.IsZero() {
out[catalog.ProfileBase(rep.Profile)]++
}
}
return out
}
// Seven on cards and four on CPUs is what runs, not eleven on whatever fits
// best: with cards free for all eleven, four still go to the CPUs.
func TestReplicasRunAsManyOnEachProfileAsItAsks(t *testing.T) {
h := newHarness(t, splitModel(7, 4), gpuNode("1", 8), gpuNode("2", 8), cpuNode("3"), cpuNode("4"))
for range 14 {
h.r.Reconcile(t.Context())
h.markReady()
h.advance(time.Second)
}
if got := byProfile(h); got["gpu"] != 7 || got["cpu"] != 4 {
t.Fatalf("running %v, want 7 on cards and 4 on CPUs", got)
}
}
// Moving replicas from one profile to the other starts the new ones before
// stopping the old: the model never runs fewer than it asks.
func TestChangingTheSplitStartsBeforeItStops(t *testing.T) {
h := newHarness(t, splitModel(2, 1), gpuNode("1", 8), cpuNode("3"), cpuNode("4"))
for range 6 {
h.r.Reconcile(t.Context())
h.markReady()
h.advance(time.Second)
}
if got := byProfile(h); got["gpu"] != 2 || got["cpu"] != 1 {
t.Fatalf("running %v before the change", got)
}
h.setModels(t, splitModel(1, 2))
for range 10 {
h.r.Reconcile(t.Context())
h.markReady()
h.advance(h.r.cfg.DrainGrace + time.Second)
if n := replicasOf(h, "m"); n < 3 {
t.Fatalf("only %d replicas while the split changed", n)
}
}
if got := byProfile(h); got["gpu"] != 1 || got["cpu"] != 2 {
t.Errorf("running %v, want 1 on cards and 2 on CPUs", got)
}
}
// The scaling has to be the profiles' sum, so a count in one place cannot
// disagree with the other.
func TestReplicasPerProfileAndScalingAgree(t *testing.T) {
m := splitModel(2, 1)
m.Scaling = catalog.Scaling{Min: 1, Max: 5}
if _, err := catalog.New([]catalog.Model{m}); err == nil {
t.Error("profiles asking for 3 with scaling 1..5 were accepted")
}
stopped := splitModel(2, 1)
was := stopped.Scaling
stopped.Stopped, stopped.Scaling = &was, catalog.Scaling{}
if _, err := catalog.New([]catalog.Model{stopped}); err != nil {
t.Errorf("a stopped model with replicas per profile was refused: %v", err)
}
}
+133
View File
@@ -0,0 +1,133 @@
package slurmsvc
import (
"context"
"errors"
"strings"
"testing"
"time"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// fakeKV is the store's cluster state.
type fakeKV struct {
m map[string]string
err error
}
func (f *fakeKV) PutClusterState(_ context.Context, k, v string) (bool, error) {
if f.err != nil {
return false, f.err
}
f.m[k] = v
return true, nil
}
func (f *fakeKV) ClusterState(_ context.Context, k string) (string, error) {
if f.err != nil {
return "", f.err
}
v, ok := f.m[k]
if !ok {
return "", store.ErrNotFound
}
return v, nil
}
// takeOver is the leader that comes next: the same settings and store, an
// inventory with only the nodes connected to it so far.
func takeOver(r *rig, kv *fakeKV, connected ...string) *rig {
n := &rig{set: r.set, inv: &fakeInv{nodes: map[string]*inventory.Node{}, allocs: map[string]int{}},
agents: &fakeAgents{conn: connected, got: map[string][]*llmgwv1.Slurm{}}}
for _, c := range connected {
node := *r.inv.nodes[c]
node.State = inventory.Ready
n.inv.nodes[c] = &node
}
n.svc = New(func() config.SlurmSettings { return n.set }, n.inv, n.agents,
func(context.Context) ([]byte, error) { return []byte(strings.Repeat("k", 1024)), nil }, nil)
n.svc.SetMemory(StoreMemory(kv))
return n
}
func TestANewLeaderKeepsTheNodesItHasNotHeardFromInTheControllersFiles(t *testing.T) {
kv := &fakeKV{m: map[string]string{}}
r := newRig()
r.svc.SetMemory(StoreMemory(kv))
r.set.Lent = []string{"cuda01"}
r.inv.allocs = map[string]int{}
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "MIXED", Jobs: 1})
r.tick()
want := r.agents.last("node06").GetFiles()["slurm.conf"]
if !strings.Contains(want, "NodeName=cuda01") || !strings.Contains(want, "Gres=gpu:rtx3090:1") {
t.Fatalf("slurm.conf:\n%s", want)
}
// The leader goes; the next hears from the controller first.
n := takeOver(r, kv, "node06")
n.tick()
if got := n.agents.last("node06").GetFiles()["slurm.conf"]; got != want {
t.Fatalf("the new leader's slurm.conf differs; the controller would drop the node and its job:\n%s\nwas:\n%s",
got, want)
}
}
func TestANodeIsToldNothingUntilItsControllerIsKnown(t *testing.T) {
r := newRig()
r.svc.SetMemory(StoreMemory(&fakeKV{m: map[string]string{}}))
delete(r.inv.nodes, "node06")
r.agents.conn = []string{"cuda01"}
r.tick()
if m := r.agents.last("cuda01"); m != nil {
t.Fatalf("a node was told %v before its controller was known; an empty part stops its slurmd and its jobs", m)
}
}
func TestANewLeaderTellsANodeItsControllerFromTheStore(t *testing.T) {
kv := &fakeKV{m: map[string]string{}}
r := newRig()
r.svc.SetMemory(StoreMemory(kv))
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
r.tick()
n := takeOver(r, kv, "cuda01")
n.tick()
if m := n.agents.last("cuda01"); !m.GetNode() || m.GetConfServer() != "10.0.0.146:6817" {
t.Fatalf("the node was told %v", m)
}
}
func TestWithTheStoreUnreadTheControllerIsSentNothingThatLeavesANodeOut(t *testing.T) {
kv := &fakeKV{m: map[string]string{}, err: errors.New("store down")}
r := newRig()
r.svc.SetMemory(StoreMemory(kv))
r.tick()
if m := r.agents.last("node06"); m != nil {
t.Fatalf("the controller was sent %v without cuda01, with the store unread", m)
}
// Once every node has reported, nothing is left out: sent.
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
r.tick()
if !strings.Contains(r.agents.last("node06").GetFiles()["slurm.conf"], "NodeName=cuda01") {
t.Fatal("with every node reported, the controller still got no files")
}
}
func TestWithNothingKeptANewLeaderWaitsForTheNodesAWhile(t *testing.T) {
r := newRig()
r.svc.SetMemory(StoreMemory(&fakeKV{m: map[string]string{}}))
r.tick()
if m := r.agents.last("node06"); m != nil {
t.Fatalf("with nothing kept, the controller was sent %v before cuda01 reported", m)
}
defer func(d time.Duration) { HoldFor = d }(HoldFor)
HoldFor = 0
r.tick()
if m := r.agents.last("node06"); !m.GetController() || strings.Contains(m.GetFiles()["slurm.conf"], "NodeName=cuda01") {
t.Fatalf("after the wait, the controller was told %v", m)
}
}
+77
View File
@@ -0,0 +1,77 @@
package slurmsvc
import (
"context"
"encoding/json"
"errors"
"slices"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// Memory keeps the nodes slurm.conf last held, and the controller's
// address, where a leader that takes over finds them. Without it a new
// leader's first slurm.conf has none of the nodes it has not heard from
// yet, and the controller, restarted on it, drops them and the jobs
// running there.
type Memory interface {
Recall(ctx context.Context) (Kept, error)
Keep(ctx context.Context, k Kept) error
}
// Kept is what Memory holds.
type Kept struct {
Controller string `json:"controller,omitempty"`
ControllerAddr string `json:"controller_addr,omitempty"`
Nodes map[string]slurm.Node `json:"nodes,omitempty"`
}
func (k Kept) equal(o Kept) bool {
if k.Controller != o.Controller || k.ControllerAddr != o.ControllerAddr || len(k.Nodes) != len(o.Nodes) {
return false
}
for name, n := range k.Nodes {
m, ok := o.Nodes[name]
if !ok || n.Addr != m.Addr || n.Hardware != m.Hardware || !slices.Equal(n.GPUs, m.GPUs) {
return false
}
}
return true
}
// KV is the store's cluster state.
type KV interface {
PutClusterState(ctx context.Context, key, value string) (bool, error)
ClusterState(ctx context.Context, key string) (string, error)
}
// keptKey is the cluster state key it is kept under.
const keptKey = "slurm-nodes"
// StoreMemory keeps it in the store's cluster state.
func StoreMemory(kv KV) Memory { return kvMemory{kv} }
type kvMemory struct{ kv KV }
func (m kvMemory) Recall(ctx context.Context) (Kept, error) {
var k Kept
v, err := m.kv.ClusterState(ctx, keptKey)
if errors.Is(err, store.ErrNotFound) {
return k, nil
}
if err != nil {
return k, err
}
err = json.Unmarshal([]byte(v), &k)
return k, err
}
func (m kvMemory) Keep(ctx context.Context, k Kept) error {
b, err := json.Marshal(k)
if err != nil {
return err
}
_, err = m.kv.PutClusterState(ctx, keptKey, string(b))
return err
}
+534
View File
@@ -0,0 +1,534 @@
// Package slurmsvc is the leader's side of the Slurm cluster run beside
// llmgw (BACKLOG P): it tells each node's agent what it is, writes the
// controller's files from what the nodes report, and hands nodes between
// llmgw and Slurm as the cluster's settings say.
//
// A node in slurm.lent is drained of llmgw's replicas, as `ctl drain` does,
// and lent to Slurm once none is left. A node taken out of slurm.lent is
// drained in Slurm, and is llmgw's again when its last job has ended. Both
// are kept in force every tick, from the settings and what the agents say,
// so a leader that takes over carries on where the last one was.
package slurmsvc
import (
"context"
"fmt"
"log/slog"
"maps"
"net"
"slices"
"strconv"
"strings"
"sync"
"time"
"google.golang.org/protobuf/proto"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
// Agents is how the service reaches the nodes.
type Agents interface {
Send(node string, msg *llmgwv1.ControlMessage) error
ConnectedNodes() []string
}
// Inventory is what the service reads and changes of llmgw's nodes.
type Inventory interface {
Get(id string) (inventory.Node, bool)
SetState(id string, st inventory.NodeState) bool
Allocations(nodeID string) []inventory.Allocation
Nodes() []inventory.Node
}
// Phase is where a node of slurm.nodes is between llmgw and Slurm.
type Phase string
const (
// Llmgw: drained in Slurm with no job left, llmgw's to place on.
Llmgw Phase = "llmgw"
// Lending: in slurm.lent, its replicas being moved off.
Lending Phase = "lending"
// Lent: Slurm's; llmgw places nothing there.
Lent Phase = "lent"
// Reclaiming: out of slurm.lent, drained in Slurm, jobs still running.
Reclaiming Phase = "reclaiming"
// Unknown: its agent has not said how it is in Slurm yet.
Unknown Phase = "unknown"
)
// NodeStatus is one node as the service sees it.
type NodeStatus struct {
Node string `json:"node"`
Phase Phase `json:"phase"`
Replicas int `json:"replicas"`
Status *Reported `json:"slurm,omitempty"`
Connected bool `json:"connected"`
}
// Reported is the agent's SlurmStatus, for the admin API.
type Reported struct {
Version string `json:"version,omitempty"`
Hardware string `json:"hardware,omitempty"`
SlurmdRunning bool `json:"slurmd_running"`
SlurmctldRunning bool `json:"slurmctld_running"`
State string `json:"state,omitempty"`
Reason string `json:"reason,omitempty"`
Jobs int `json:"jobs"`
Error string `json:"error,omitempty"`
FilesSHA256 string `json:"files_sha256,omitempty"`
At time.Time `json:"at"`
}
// Status is the whole Slurm cluster, for `ctl slurm status`.
type Status struct {
Enabled bool `json:"enabled"`
Cluster string `json:"cluster"`
Controller string `json:"controller"`
Partition string `json:"partition"`
Ctld *Reported `json:"controller_status,omitempty"`
Nodes []NodeStatus `json:"nodes"`
// Waiting are nodes left out of slurm.conf until they report their
// hardware.
Waiting []string `json:"waiting,omitempty"`
}
// Service keeps the Slurm cluster.
type Service struct {
settings func() config.SlurmSettings
inv Inventory
agents Agents
key func(ctx context.Context) ([]byte, error)
log *slog.Logger
mu sync.Mutex
// status is each node's last SlurmStatus.
status map[string]*Reported
// drained are nodes this service put in Draining, which it alone makes
// ready again.
drained map[string]bool
// sent is what each node was last told, so it is told again only when
// that changes or its stream is new.
sent map[string]*llmgwv1.Slurm
mungeKy []byte
kick chan struct{}
// mem keeps slurm.conf's nodes and the controller's address for the
// next leader; known is what it held when read, and what was kept
// since. loaded is false until it was read: until then the controller
// is sent no files that leave out a node, which would drop that node
// and its jobs.
mem Memory
known Kept
loaded bool
// started is the first pass, from which a node neither heard from nor
// kept is waited for, for HoldFor.
started time.Time
}
// HoldFor is how long a new leader waits for a node it neither has heard
// from nor finds kept before it writes the controller's files without it:
// on a cluster that kept nothing yet (one upgraded to this), the nodes
// report within seconds of a new leader, and the controller keeps the
// node and its jobs meanwhile.
var HoldFor = 2 * time.Minute
// New is a Service; key gives the munge key, made once from a cluster
// secret.
func New(settings func() config.SlurmSettings, inv Inventory, agents Agents,
key func(ctx context.Context) ([]byte, error), log *slog.Logger) *Service {
if log == nil {
log = slog.Default()
}
return &Service{settings: settings, inv: inv, agents: agents, key: key, log: log,
status: map[string]*Reported{}, drained: map[string]bool{}, sent: map[string]*llmgwv1.Slurm{},
kick: make(chan struct{}, 1)}
}
// SetMemory has slurm.conf's nodes kept in mem, and read from it first.
func (s *Service) SetMemory(mem Memory) {
s.mu.Lock()
s.mem = mem
s.mu.Unlock()
}
// current is the settings, slurm.all_nodes resolved to the nodes the
// inventory has.
func (s *Service) current() config.SlurmSettings {
set := s.settings()
if !set.AllNodes {
return set
}
var ids []string
for _, n := range s.inv.Nodes() {
ids = append(ids, n.ID)
}
return set.WithNodes(ids)
}
// Every is how often the service looks again without being asked.
var Every = 10 * time.Second
// Run keeps the cluster until ctx ends.
func (s *Service) Run(ctx context.Context) {
t := time.NewTicker(Every)
defer t.Stop()
for {
s.Tick(ctx)
select {
case <-ctx.Done():
return
case <-t.C:
case <-s.kick:
}
}
}
// Kick has the service look again now: the settings changed.
func (s *Service) Kick() {
select {
case s.kick <- struct{}{}:
default:
}
}
// OnSlurmStatus records what a node's agent says.
func (s *Service) OnSlurmStatus(node string, st *llmgwv1.SlurmStatus) {
r := &Reported{Version: st.GetVersion(), Hardware: st.GetHardware(), SlurmdRunning: st.GetSlurmdRunning(),
SlurmctldRunning: st.GetSlurmctldRunning(), State: st.GetState(), Reason: st.GetReason(),
Jobs: int(st.GetJobs()), Error: st.GetError(), FilesSHA256: st.GetFilesSha256(), At: time.Now()}
s.mu.Lock()
prev := s.status[node]
s.status[node] = r
s.mu.Unlock()
if r.Error != "" && (prev == nil || prev.Error != r.Error) {
s.log.Warn("a node could not keep its part in Slurm", "node", node, "err", r.Error)
}
if prev == nil || prev.Hardware != r.Hardware || prev.State != r.State || prev.Jobs != r.Jobs {
s.Kick()
}
}
// OnRegister is a node's new stream: it is told again what it is.
func (s *Service) OnRegister(node string) {
s.mu.Lock()
delete(s.sent, node)
s.mu.Unlock()
s.Kick()
}
// Tick does one pass: the nodes handed over as far as they can be, and
// each told what it is.
func (s *Service) Tick(ctx context.Context) {
set := s.current()
s.mu.Lock()
defer s.mu.Unlock()
s.recall(ctx)
if s.started.IsZero() {
s.started = time.Now()
}
phases := s.handOver(set)
var key []byte
if set.Enabled {
if s.mungeKy == nil {
k, err := s.key(ctx)
if err != nil {
s.log.Warn("no munge key for Slurm; nothing is sent to the nodes", "err", err)
return
}
s.mungeKy = k
}
key = s.mungeKy
}
files, waiting := s.files(set)
// Not yet knowing what the last leader knew, a slurm.conf without some
// node would take it, and its jobs, from the controller: kept back.
hold := s.mem != nil && len(waiting) > 0 && (!s.loaded || time.Since(s.started) < HoldFor)
for _, node := range s.agents.ConnectedNodes() {
if hold && node == set.Controller {
continue
}
m := s.message(set, node, phases[node] == Lent, key, files)
if m == nil {
continue
}
if old, ok := s.sent[node]; ok && proto.Equal(old, m) {
continue
}
if err := s.agents.Send(node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_Slurm{Slurm: m}}); err != nil {
s.log.Warn("could not tell a node its part in Slurm", "node", node, "err", err)
continue
}
s.sent[node] = m
}
s.remember(ctx, set)
}
// recall reads what the last leader kept, once. Under s.mu.
func (s *Service) recall(ctx context.Context) {
if s.mem == nil || s.loaded {
return
}
k, err := s.mem.Recall(ctx)
if err != nil {
s.log.Warn("could not read Slurm's nodes from the store; the controller's files wait for every node",
"err", err)
return
}
s.known, s.loaded = k, true
}
// remember keeps slurm.conf's nodes and the controller's address when they
// changed. Under s.mu.
func (s *Service) remember(ctx context.Context, set config.SlurmSettings) {
if s.mem == nil || !s.loaded || !set.Enabled {
return
}
c, _ := s.cluster(set)
now := Kept{Controller: set.Controller, ControllerAddr: c.ControllerAddr, Nodes: map[string]slurm.Node{}}
for _, n := range c.Nodes {
now.Nodes[n.Name] = n
}
if now.equal(s.known) {
return
}
if err := s.mem.Keep(ctx, now); err != nil {
s.log.Warn("could not keep Slurm's nodes in the store", "err", err)
return
}
s.known = now
}
// controllerAddr is where the nodes reach slurmctld: the controller's
// management address, else the one kept. Under s.mu.
func (s *Service) controllerAddr(set config.SlurmSettings) string {
if n, ok := s.inv.Get(set.Controller); ok && n.MgmtAddr != "" {
return n.MgmtAddr
}
if s.known.Controller == set.Controller {
return s.known.ControllerAddr
}
return ""
}
// handOver moves each node of slurm.nodes a step between llmgw and Slurm,
// answering where each is. Under s.mu.
func (s *Service) handOver(set config.SlurmSettings) map[string]Phase {
out := map[string]Phase{}
if !set.Enabled {
// Off: every node is llmgw's again. What Slurm still runs is left
// to finish where it is; turning Slurm off with work lent is the
// operator's choice.
for node := range s.drained {
s.ready(node)
}
return out
}
for _, node := range set.Nodes {
out[node] = s.phase(set, node)
}
// A node no longer in slurm.nodes is llmgw's.
for node := range s.drained {
if !slices.Contains(set.Nodes, node) {
s.ready(node)
}
}
return out
}
func (s *Service) phase(set config.SlurmSettings, node string) Phase {
n, ok := s.inv.Get(node)
st := s.status[node]
if set.IsLent(node) {
if ok && (n.State == inventory.Ready || n.State == inventory.Cordoned) {
s.inv.SetState(node, inventory.Draining)
s.drained[node] = true
s.log.Info("lending a node to Slurm: its replicas are moved off first", "node", node)
}
if !ok || len(s.inv.Allocations(node)) > 0 {
return Lending
}
return Lent
}
// Not lent. Until the agent says Slurm has drained it with no job left,
// llmgw places nothing new on it: a new leader starts with every node
// ready. Cordoned, not drained: llmgw has nothing there to move.
if st == nil || slurmBusy(st) {
if ok && n.State == inventory.Ready {
s.inv.SetState(node, inventory.Cordoned)
s.drained[node] = true
if st != nil {
s.log.Info("taking a node back from Slurm: llmgw places nothing there until its jobs end",
"node", node, "jobs", st.Jobs)
}
}
if st == nil {
return Unknown
}
return Reclaiming
}
if s.drained[node] {
s.ready(node)
}
return Llmgw
}
// slurmBusy is whether Slurm may still run work on a node: a job step left
// on it (or no telling), or a controller that would still give it one. A
// node the controller does not know of, or cannot be asked about, gets no
// job from it, so a Slurm that is broken never keeps a node from llmgw.
func slurmBusy(st *Reported) bool {
if st.Jobs != 0 {
return true
}
if st.State == "" {
return false
}
return !strings.Contains(st.State, "DRAIN") && !strings.Contains(st.State, "DOWN")
}
func (s *Service) ready(node string) {
if n, ok := s.inv.Get(node); ok && (n.State == inventory.Draining || n.State == inventory.Cordoned) {
s.inv.SetState(node, inventory.Ready)
s.log.Info("a node is llmgw's again: Slurm has no job there", "node", node)
}
delete(s.drained, node)
}
// files are the controller's files, and the nodes left out of them for not
// having reported their hardware. Under s.mu.
func (s *Service) files(set config.SlurmSettings) (map[string]string, []string) {
c, waiting := s.cluster(set)
if c.ControllerAddr == "" {
return nil, waiting
}
return c.Files(), waiting
}
// cluster is what slurm.conf says: each node as it last reported, else as
// kept from before, and the nodes left out for having neither. Under s.mu.
func (s *Service) cluster(set config.SlurmSettings) (slurm.Cluster, []string) {
c := slurm.Cluster{Name: set.Cluster(), Controller: set.Controller, Partition: set.PartitionName()}
if !set.Enabled {
return c, nil
}
c.ControllerAddr = s.controllerAddr(set)
var waiting []string
for _, node := range set.Nodes {
if sn, ok := s.reported(node); ok {
c.Nodes = append(c.Nodes, sn)
} else if sn, ok := s.known.Nodes[node]; ok {
c.Nodes = append(c.Nodes, sn)
} else {
waiting = append(waiting, node)
}
}
return c, waiting
}
// reported is a node as its agent and the inventory say now. Under s.mu.
func (s *Service) reported(node string) (slurm.Node, bool) {
n, ok := s.inv.Get(node)
st := s.status[node]
if !ok || n.MgmtAddr == "" || st == nil || st.Hardware == "" {
return slurm.Node{}, false
}
hw, err := slurm.ParseHardware(st.Hardware)
if err != nil {
return slurm.Node{}, false
}
sn := slurm.Node{Name: node, Addr: n.MgmtAddr, Hardware: hw}
for _, g := range n.GPUs {
sn.GPUs = append(sn.GPUs, slurm.GPU{Minor: g.Index, Name: g.Name})
}
return sn, true
}
// message is what a node is told. Under s.mu.
func (s *Service) message(set config.SlurmSettings, node string, lent bool, key []byte,
files map[string]string) *llmgwv1.Slurm {
m := &llmgwv1.Slurm{}
if !set.Enabled {
return m
}
isNode := slices.Contains(set.Nodes, node)
isCtl := set.Controller == node && files != nil
if !isNode && !isCtl {
return m
}
m.Name, m.MungeKey, m.Lent = node, key, lent
m.Node = isNode
if addr := s.controllerAddr(set); addr != "" {
m.ConfServer = net.JoinHostPort(addr, strconv.Itoa(slurm.CtldPort))
}
if isNode && m.ConfServer == "" {
// Without its controller a node cannot start; it is told when the
// controller registers. Told nothing until then, not "no part": a
// node still running Slurm's jobs would stop them.
return nil
}
if isCtl {
m.Controller = true
m.Files = maps.Clone(files)
}
return m
}
// Status is the cluster as the service sees it now.
func (s *Service) Status() Status {
set := s.current()
s.mu.Lock()
defer s.mu.Unlock()
out := Status{Enabled: set.Enabled, Cluster: set.Cluster(), Controller: set.Controller,
Partition: set.PartitionName(), Ctld: s.status[set.Controller]}
if !set.Enabled {
return out
}
_, out.Waiting = s.files(set)
conn := s.agents.ConnectedNodes()
for _, node := range set.Nodes {
ns := NodeStatus{Node: node, Status: s.status[node], Replicas: len(s.inv.Allocations(node)),
Connected: slices.Contains(conn, node)}
ns.Phase = s.peek(set, node)
out.Nodes = append(out.Nodes, ns)
}
return out
}
// peek is phase without changing anything. Under s.mu.
func (s *Service) peek(set config.SlurmSettings, node string) Phase {
st := s.status[node]
if set.IsLent(node) {
if len(s.inv.Allocations(node)) > 0 {
return Lending
}
return Lent
}
switch {
case st == nil:
return Unknown
case slurmBusy(st):
return Reclaiming
}
return Llmgw
}
// Describe is one line for a log or a screen.
func (n NodeStatus) Describe() string {
switch n.Phase {
case Lending:
return fmt.Sprintf("being lent: %d replica(s) still to move off", n.Replicas)
case Lent:
return "lent to Slurm"
case Reclaiming:
if n.Status != nil {
return fmt.Sprintf("being taken back: %d job(s) still running", n.Status.Jobs)
}
return "being taken back"
case Llmgw:
return "llmgw's"
}
return "not heard from yet"
}
+274
View File
@@ -0,0 +1,274 @@
package slurmsvc
import (
"context"
"slices"
"strings"
"testing"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
)
type fakeInv struct {
nodes map[string]*inventory.Node
allocs map[string]int
}
func (f *fakeInv) Get(id string) (inventory.Node, bool) {
n, ok := f.nodes[id]
if !ok {
return inventory.Node{}, false
}
return *n, true
}
func (f *fakeInv) SetState(id string, st inventory.NodeState) bool {
n, ok := f.nodes[id]
if ok {
n.State = st
}
return ok
}
func (f *fakeInv) Allocations(id string) []inventory.Allocation {
return make([]inventory.Allocation, f.allocs[id])
}
func (f *fakeInv) Nodes() []inventory.Node {
var out []inventory.Node
for _, n := range f.nodes {
out = append(out, *n)
}
slices.SortFunc(out, func(a, b inventory.Node) int { return strings.Compare(a.ID, b.ID) })
return out
}
type fakeAgents struct {
conn []string
got map[string][]*llmgwv1.Slurm
}
func (f *fakeAgents) Send(node string, m *llmgwv1.ControlMessage) error {
f.got[node] = append(f.got[node], m.GetSlurm())
return nil
}
func (f *fakeAgents) ConnectedNodes() []string { return f.conn }
func (f *fakeAgents) last(node string) *llmgwv1.Slurm {
ms := f.got[node]
if len(ms) == 0 {
return nil
}
return ms[len(ms)-1]
}
type rig struct {
set config.SlurmSettings
inv *fakeInv
agents *fakeAgents
svc *Service
}
func newRig() *rig {
r := &rig{
set: config.SlurmSettings{Enabled: true, Controller: "node06", Nodes: []string{"cuda01"}},
inv: &fakeInv{nodes: map[string]*inventory.Node{
"node06": {ID: "node06", MgmtAddr: "10.0.0.146"},
"cuda01": {ID: "cuda01", MgmtAddr: "10.0.0.147", GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA GeForce RTX 3090"}}},
}, allocs: map[string]int{"cuda01": 2}},
agents: &fakeAgents{conn: []string{"node06", "cuda01"}, got: map[string][]*llmgwv1.Slurm{}},
}
r.svc = New(func() config.SlurmSettings { return r.set }, r.inv, r.agents,
func(context.Context) ([]byte, error) { return []byte(strings.Repeat("k", 1024)), nil }, nil)
return r
}
func (r *rig) report(node string, st *llmgwv1.SlurmStatus) { r.svc.OnSlurmStatus(node, st) }
func (r *rig) tick() { r.svc.Tick(context.Background()) }
const hw = "NodeName=cuda01 CPUs=16 RealMemory=64000"
func TestMessagesAndFiles(t *testing.T) {
r := newRig()
r.tick()
ctl, node := r.agents.last("node06"), r.agents.last("cuda01")
if !ctl.GetController() || ctl.GetNode() || len(ctl.GetMungeKey()) != 1024 {
t.Fatalf("controller told %v", ctl)
}
if !node.GetNode() || node.GetController() || node.GetConfServer() != "10.0.0.146:6817" || node.GetLent() ||
node.GetName() != "cuda01" || len(node.GetFiles()) != 0 {
t.Fatalf("node told %v", node)
}
if strings.Contains(ctl.GetFiles()["slurm.conf"], "NodeName=cuda01") {
t.Fatal("a node with no hardware reported is in slurm.conf")
}
if w := r.svc.Status().Waiting; len(w) != 1 || w[0] != "cuda01" {
t.Fatalf("waiting %v", w)
}
// Same again: nobody is told twice.
r.tick()
if len(r.agents.got["node06"]) != 1 || len(r.agents.got["cuda01"]) != 1 {
t.Fatal("told again with nothing changed")
}
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
r.tick()
conf := r.agents.last("node06").GetFiles()["slurm.conf"]
if !strings.Contains(conf, "NodeName=cuda01 NodeAddr=10.0.0.147 CPUs=16 RealMemory=62080 Gres=gpu:rtx3090:1") {
t.Fatalf("slurm.conf:\n%s", conf)
}
if !strings.Contains(r.agents.last("node06").GetFiles()["gres.conf"], "File=/dev/nvidia0") {
t.Fatal("gres.conf has no card")
}
if len(r.agents.got["cuda01"]) != 1 {
t.Fatal("a node was told again when only the controller's files changed")
}
// A new stream is told again.
r.svc.OnRegister("cuda01")
r.tick()
if len(r.agents.got["cuda01"]) != 2 {
t.Fatal("a new stream was not told")
}
if r.inv.nodes["cuda01"].State != inventory.Ready {
t.Fatal("a node drained in Slurm with no job was taken from llmgw")
}
}
func TestLendAndReclaim(t *testing.T) {
r := newRig()
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
r.set.Lent = []string{"cuda01"}
r.tick()
if r.inv.nodes["cuda01"].State != inventory.Draining {
t.Fatal("a lent node was not drained of its replicas")
}
if r.agents.last("cuda01").GetLent() {
t.Fatal("lent to Slurm with replicas still on it")
}
if p := r.svc.Status().Nodes[0].Phase; p != Lending {
t.Fatalf("phase %s", p)
}
r.inv.allocs["cuda01"] = 0
r.tick()
if !r.agents.last("cuda01").GetLent() || r.svc.Status().Nodes[0].Phase != Lent {
t.Fatal("not lent once its replicas were gone")
}
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "ALLOCATED", Jobs: 3})
// Taken back: Slurm drains it; llmgw keeps off until its jobs end.
r.set.Lent = nil
r.tick()
if r.agents.last("cuda01").GetLent() {
t.Fatal("still lent")
}
if s := r.inv.nodes["cuda01"].State; s != inventory.Draining && s != inventory.Cordoned {
t.Fatalf("llmgw may place on a node with Slurm jobs: %v", s)
}
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "ALLOCATED+DRAIN", Jobs: 1})
r.tick()
if r.svc.Status().Nodes[0].Phase != Reclaiming || r.inv.nodes["cuda01"].State == inventory.Ready {
t.Fatal("given back with a job left")
}
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: serving models"})
r.tick()
if r.inv.nodes["cuda01"].State != inventory.Ready || r.svc.Status().Nodes[0].Phase != Llmgw {
t.Fatal("not llmgw's when its last job ended")
}
}
func TestNewLeaderKeepsOffUntilHeard(t *testing.T) {
r := newRig()
r.inv.allocs["cuda01"] = 0
r.tick()
// Nothing heard yet: cordoned, nothing moved.
if r.inv.nodes["cuda01"].State != inventory.Cordoned {
t.Fatalf("state %v before the agent said anything", r.inv.nodes["cuda01"].State)
}
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "MIXED", Jobs: 2})
r.tick()
if r.inv.nodes["cuda01"].State == inventory.Ready {
t.Fatal("ready with Slurm jobs on it")
}
}
func TestBrokenSlurmLeavesTheNodeToLlmgw(t *testing.T) {
r := newRig()
r.report("cuda01", &llmgwv1.SlurmStatus{Error: "installing Slurm: no package", Jobs: 0})
r.tick()
if r.inv.nodes["cuda01"].State != inventory.Ready {
t.Fatal("a Slurm that does not work took a node from llmgw")
}
}
func TestOperatorsCordonIsLeft(t *testing.T) {
r := newRig()
r.inv.nodes["cuda01"].State = inventory.Cordoned
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN"})
r.tick()
if r.inv.nodes["cuda01"].State != inventory.Cordoned {
t.Fatal("llmgw uncordoned a node an operator cordoned")
}
}
func TestOffAndOut(t *testing.T) {
r := newRig()
r.inv.allocs["cuda01"] = 0
r.set.Lent = []string{"cuda01"}
r.tick()
r.set.Enabled = false
r.tick()
if m := r.agents.last("cuda01"); m.GetNode() || m.GetController() {
t.Fatalf("told %v with Slurm off", m)
}
if r.inv.nodes["cuda01"].State != inventory.Ready {
t.Fatal("not llmgw's with Slurm off")
}
r = newRig()
r.inv.allocs["cuda01"] = 0
r.set.Lent = []string{"cuda01"}
r.tick()
r.set.Nodes, r.set.Lent = nil, nil
r.tick()
if r.inv.nodes["cuda01"].State != inventory.Ready || r.agents.last("cuda01").GetNode() {
t.Fatal("a node taken out of slurm.nodes is not llmgw's")
}
if !r.agents.last("node06").GetController() {
t.Fatal("the controller stopped with no nodes")
}
}
func TestNoControllerYet(t *testing.T) {
r := newRig()
delete(r.inv.nodes, "node06")
r.agents.conn = []string{"cuda01"}
r.tick()
if m := r.agents.last("cuda01"); m.GetNode() {
t.Fatalf("a node was started with no controller to fetch its files from: %v", m)
}
}
func TestAllNodes(t *testing.T) {
r := newRig()
r.set.Nodes, r.set.AllNodes = nil, true
r.inv.nodes["cuda02"] = &inventory.Node{ID: "cuda02", MgmtAddr: "10.0.0.150"}
r.agents.conn = append(r.agents.conn, "cuda02")
r.tick()
for _, n := range []string{"cuda01", "cuda02", "node06"} {
if !r.agents.last(n).GetNode() {
t.Fatalf("%s, a node that joined, does not run slurmd", n)
}
}
if got := len(r.svc.Status().Nodes); got != 3 {
t.Fatalf("%d nodes in the status", got)
}
r.set.Lent = []string{"cuda02"}
r.tick()
if p := r.svc.Status().Nodes[1]; p.Node != "cuda02" || p.Phase != Lent {
t.Fatalf("cuda02 not lent: %+v", p)
}
}
+2 -2
View File
@@ -94,7 +94,7 @@ type Store interface {
SetStaffRole(ctx context.Context, id, role string) error
SetUserState(ctx context.Context, id string, state store.UserState) error
DeleteUserSessions(ctx context.Context, userID string) error
AddMembership(ctx context.Context, user string, scope store.Scope, id, role, by string, at time.Time) error
EnsureMembership(ctx context.Context, user string, scope store.Scope, id, by string, at time.Time) error
RemoveMembership(ctx context.Context, userID string, scope store.Scope, scopeID string, at time.Time) (int, error)
MembershipsOf(ctx context.Context, userID string) ([]store.Membership, error)
AllOrgs(ctx context.Context) ([]store.Org, error)
@@ -221,7 +221,7 @@ func (k *Keeper) apply(ctx context.Context, u *store.User, id Identity, remove b
continue // a mapping to a place the tree lacks: said by Sync
}
want[string(scope)+"/"+sid] = scope
if err := k.St.AddMembership(ctx, u.ID, scope, sid, store.RoleMember, k.Name, k.now()); err != nil {
if err := k.St.EnsureMembership(ctx, u.ID, scope, sid, k.Name, k.now()); err != nil {
return err
}
}
+11
View File
@@ -377,6 +377,7 @@ var Reference = []Block{
{Key: "keep_within", Type: "string", Doc: "KeepWithin keeps a multi-node replica's nodes inside one rack, row,\nroom or site, read from their llmgw.io/rack (and so on) labels: for a\nfabric whose leaves do not reach across racks, or cables that do not.\nNodes without the label are passed over for such a profile. Empty is\nno constraint.", Nested: ""},
{Key: "weight", Type: "float64", Doc: "Weight is the router's capacity weight. It should come from ctl bench\nrather than a guess (plan §6.2.1).", Nested: ""},
{Key: "priority", Type: "int", Doc: "Priority orders profiles during placement; highest that fits wins.", Nested: ""},
{Key: "replicas", Type: "int", Doc: "Replicas, set on any of a model's profiles, is how many of its\nreplicas run on this one: seven on cards and four on CPUs, all\nanswering under the model's name, rather than each replica on the\nbest profile that fits. The model's scaling is then the sum, min and\nmax alike; a profile left at zero runs none.", Nested: ""},
{Key: "startup_timeout", Type: "time.Duration", Doc: "StartupTimeout bounds model loading. A 70B model across two nodes takes\ntens of minutes, so this is per profile rather than global.", Nested: ""},
{Key: "kv_events", Type: "bool", Doc: "KVEvents has vLLM publish what its prefix cache holds, and the router\nroute this model's requests by it: to the replica holding the most of\na prompt's leading blocks (plan §7.5 tier 2). vLLM only.\n\nOff by default, because it costs something on every request: the\nrouter asks a replica to tokenize the prompt before placing it, a\nround trip and the chat template run twice. It pays where prompts\nshare long prefixes that the approximate tree guesses wrong — many\nreplicas, conversations that move between them, evictions. The engine\nis started with sha256_cbor block hashing and a fixed PYTHONHASHSEED,\nwhich the router's hashing assumes; arguments overriding either leave\nthe index empty, and the agent says so in its log.", Nested: ""},
{Key: "gpu", Type: "*GPUNeed", Doc: "GPU says what card the profile needs, in place of naming its class:\nit stands for every class in the cluster that meets it. A catalog\nentry written once then places sensibly on any site's cards.", Nested: "catalog.GPUNeed"},
@@ -479,6 +480,7 @@ var Reference = []Block{
{Key: "mail", Type: "mail.Config", Doc: "Mail is where the gateway's few emails go out from: a link to set a\nforgotten password. Unset, none is sent and people ask an operator.", Nested: "mail.Config"},
{Key: "serving", Type: "ServingSettings", Doc: "Serving stops and starts the whole cluster's service (llmgw stop, llmgw\nstart).", Nested: "config.ServingSettings"},
{Key: "firewall", Type: "FirewallSettings", Doc: "Firewall turns llmgw's firewall on every machine on or off.", Nested: "config.FirewallSettings"},
{Key: "slurm", Type: "SlurmSettings", Doc: "Slurm is a Slurm cluster run beside llmgw on some of its nodes, and\nwhich of them it may use now.", Nested: "config.SlurmSettings"},
{Key: "presets", Type: "PresetSettings", Doc: "Presets is how the models that come with llmgw reach the catalog.", Nested: "config.PresetSettings"},
{Key: "on_demand", Type: "OnDemandSettings", Doc: "OnDemand is how models that load when asked for take turns on the\ncards.", Nested: "config.OnDemandSettings"},
{Key: "agents", Type: "AgentSettings", Doc: "Agents are the nodes' agent settings: agent.yaml's keys but those an\nagent needs to start, for every node (default) and for one\n(nodes.<id>). Sent to the agents as they change; an agent whose\nsettings changed starts again on them, its engines serving\nthroughout.", Nested: "config.AgentSettings"},
@@ -511,6 +513,15 @@ var Reference = []Block{
{Package: "engines", Name: "SimConfig", Title: "SimConfig", Doc: "", Fields: []Field{
{Key: "bin", Type: "string", Doc: "Bin is the llmgw binary itself; the engine is one of its subcommands.\nEmpty means whatever started this agent, which is right almost always.", Nested: ""},
}},
{Package: "config", Name: "SlurmSettings", Title: "SlurmSettings", Doc: "", Fields: []Field{
{Key: "enabled", Type: "bool", Doc: "Enabled runs Slurm on the nodes below. False stops llmgw keeping it;\nwhat was installed is left installed, and every node is llmgw's.", Nested: ""},
{Key: "cluster_name", Type: "string", Doc: "ClusterName is Slurm's ClusterName. Empty is \"llmgw\".", Nested: ""},
{Key: "controller", Type: "string", Doc: "Controller is the llmgw node whose agent runs slurmctld. It needs no\nGPU, and need not be one of Nodes.", Nested: ""},
{Key: "nodes", Type: "[]string", Doc: "Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained\nin Slurm, until it is also in Lent.", Nested: ""},
{Key: "all_nodes", Type: "bool", Doc: "AllNodes runs slurmd on every llmgw node, those that join later too,\nbesides any named in Nodes. Each is llmgw's until it is lent, as any.", Nested: ""},
{Key: "lent", Type: "[]string", Doc: "Lent are the nodes of Nodes Slurm may run jobs on now. A node put here\nis drained of llmgw's replicas first, as `ctl drain` does, and resumed\nin Slurm once none is left; a node taken out is drained in Slurm and\ngiven back to llmgw when its last job ends.", Nested: ""},
{Key: "partition", Type: "string", Doc: "Partition is the one partition every node is in. Empty is \"gpu\".", Nested: ""},
}},
{Package: "engines", Name: "StrataConfig", Title: "StrataConfig", Doc: "", Fields: []Field{
{Key: "dir", Type: "string", Doc: "Dir is Strata's source directory, where setup.py and serve/ are.", Nested: ""},
{Key: "python", Type: "string", Doc: "Python is the interpreter of its environment; Dir's .venv when unset.", Nested: ""},
+1
View File
@@ -41,6 +41,7 @@ var Sources = []string{
"internal/config/serving.go",
"internal/config/presets.go",
"internal/config/ondemand.go",
"internal/config/slurm.go",
"internal/config/agents.go",
"internal/mail/mail.go",
"internal/notify/notify.go",
+20
View File
@@ -48,6 +48,22 @@ type Header struct {
// (<arch>.pooling_type set and not none): an embedding model, which
// llama.cpp serves with --embedding and does not chat.
Embedding bool
// ExpertCount and ExpertUsedCount are a model of experts' experts and
// how many each token uses (<arch>.expert_count, expert_used_count);
// zero for a dense model.
ExpertCount, ExpertUsedCount uint64
}
// ActiveShare is the part of the weights a token reads: for a model of
// experts its experts' share plus a tenth for what every token reads
// whatever the experts — attention, embeddings, routing, shared experts.
// Qwen3-Coder-30B-A3B, eight of 128, read 0.16 of its weights a token on
// cuda01's card, which is 0.0625 and the tenth. 1 for a dense model.
func (h Header) ActiveShare() float64 {
if h.ExpertCount == 0 || h.ExpertUsedCount == 0 || h.ExpertUsedCount >= h.ExpertCount {
return 1
}
return min(1, float64(h.ExpertUsedCount)/float64(h.ExpertCount)+0.1)
}
// KVBytesPerToken is what one token of context costs in llama.cpp's default
@@ -276,6 +292,10 @@ func Read(r io.Reader) (Header, error) {
h.Embedding = num > 0
case "context_length":
h.ContextLength = num
case "expert_count":
h.ExpertCount = num
case "expert_used_count":
h.ExpertUsedCount = num
}
}
// The tokenizer's arrays — megabytes of vocabulary — come after the
+11
View File
@@ -210,3 +210,14 @@ func TestAPoolIsSizedToTheCard(t *testing.T) {
t.Error("class memory misread")
}
}
// A model of experts reads its experts' share a token and what every token
// reads besides; a dense one reads it all.
func TestAModelOfExpertsReadsAShareOfItsWeights(t *testing.T) {
if got := (Header{ExpertCount: 128, ExpertUsedCount: 8}).ActiveShare(); got < 0.16 || got > 0.17 {
t.Errorf("Qwen3-30B-A3B's shape reads %.3f of its weights a token, measured 0.16", got)
}
if (Header{}).ActiveShare() != 1 || (Header{ExpertCount: 8, ExpertUsedCount: 8}).ActiveShare() != 1 {
t.Error("a dense model reads all its weights")
}
}
+72 -12
View File
@@ -1,6 +1,7 @@
package modelplan
import (
"math"
"regexp"
"strconv"
)
@@ -9,28 +10,45 @@ import (
//
// Writing a token reads every weight it uses once, so on a CPU the memory
// decides the speed: a token takes the bytes read over the rate llama.cpp
// reads them at, plus a little that does not shrink with the model. Measured
// on the PoC on 2026-10-26, each node's bandwidth by MeasureMemoryBandwidth
// against llama.cpp's own predicted_per_second for 128 tokens:
// reads them at, plus a little that does not shrink with the model.
//
// node01 4 vCPU 24.9 GB/s qwen2.5-1.5b Q4_K_M 1.1 GB 14.5 tok/s 0.65 of the bandwidth
// node05 8 vCPU 41.9 GB/s qwen2.5-14b Q4_K_M 9.0 GB 3.3 tok/s 0.71
// cuda01 12 vCPU 42.4 GB/s qwen2.5-0.5b Q4_K_M 0.5 GB 36.0 tok/s (the fixed cost shows)
// The rate is the threads': each reads at about what one core reads alone,
// a little less with every one added, up to what the whole machine reads.
// On a server of many memory channels a replica given four threads reads at
// a fraction of the machine's rate, and the machine's rate alone overstated
// it several times. Measured on the PoC's node05 on 2026-11-03 (8 vCPU,
// 43.9 GB/s with every CPU, 9.7 with one; llama-bench, 64 tokens):
//
// So 0.68 of the measured rate, and 5 ms a token besides. One reply at a
// time; several at once share the same reads and each is slower. The PoC's
// host has far fewer cores than its VMs, so these are what a busy host
// gives, and a machine of its own does better.
// threads 1 2 4 8
// qwen2.5-coder-7b 1.9 3.6 6.3 9.3 tok/s 4.7 GB: 8.9 17.0 30.7 45.6 GB/s read
// qwen2.5-1.5b 8.3 15.2 25.4 37.0 tok/s 1.1 GB: 9.7 18.3 32.6 —
//
// so 0.95 of one core's rate times threads^0.9, up to the machine's, and
// 5 ms a token besides. Against what was measured before (2026-10-26):
// node01's 1.5b on two threads, 14.5 tok/s, is taken for 15.0; node05's 14b
// on four, 3.3, for 3.5. One reply at a time; several at once share the same
// reads and each is slower. The PoC's host has far fewer cores than its VMs,
// so these are what a busy host gives, and a machine of its own does better.
// cpuReadShare is the part of the measured bandwidth llama.cpp reads at.
// cpuReadShare is the part of the whole machine's rate llama.cpp reads at,
// for a node whose one-core rate is not known: an agent older than its
// measurement.
const cpuReadShare = 0.68
// coreShare and threadExp are how threads add up: each about one core's
// rate, a little less with every one added.
const (
coreShare = 0.95
threadExp = 0.9
)
// cpuTokenFixed is a token's cost that does not grow with the model.
const cpuTokenFixed = 0.005
// CPUTokensPerSecond is about how many tokens a second one reply gets on a
// CPU whose memory reads bwGBs, from a model whose tokens each read bytes.
// 0 when either is unknown.
// 0 when either is unknown. Use CPUTokensPerSecondOn where the threads and
// one core's rate are known.
func CPUTokensPerSecond(bytes int64, bwGBs float64) float64 {
if bytes <= 0 || bwGBs <= 0 {
return 0
@@ -38,6 +56,48 @@ func CPUTokensPerSecond(bytes int64, bwGBs float64) float64 {
return 1 / (float64(bytes)/(bwGBs*1e9*cpuReadShare) + cpuTokenFixed)
}
// CPUReadGBs is the rate a replica given threads reads its weights at, on a
// node whose memory reads allGBs with every CPU and coreGBs with one. Without
// one core's rate or the threads, the share of the whole that small machines
// gave.
func CPUReadGBs(allGBs, coreGBs float64, threads int) float64 {
if allGBs <= 0 {
return 0
}
if coreGBs <= 0 || threads <= 0 {
return allGBs * cpuReadShare
}
return min(coreShare*coreGBs*math.Pow(float64(threads), threadExp), allGBs)
}
// CPUTokensPerSecondOn is about how many tokens a second one reply gets
// from a model whose tokens each read bytes, on threads of a node whose
// memory reads allGBs with every CPU and coreGBs with one. 0 when unknown.
func CPUTokensPerSecondOn(bytes int64, allGBs, coreGBs float64, threads int) float64 {
read := CPUReadGBs(allGBs, coreGBs, threads)
if bytes <= 0 || read <= 0 {
return 0
}
return 1 / (float64(bytes)/(read*1e9) + cpuTokenFixed)
}
// CPUThreads is how many threads a CPU replica runs on: the cores its
// profile reserves, else half the node's, which is what the agent gives
// llama.cpp then.
func CPUThreads(cpuCores, nodeCPUs int) int {
if cpuCores > 0 {
return cpuCores
}
return max(nodeCPUs/2, 1)
}
// Slow speeds: under the first a reply is a wait somebody notices, under the
// second one of a few hundred tokens takes minutes.
const (
CPUSlowTPS = 15
CPUTooSlowTPS = 5
)
// moeRead is what a token of a model of experts reads over its active
// share.
const moeRead = 1.6
+61
View File
@@ -23,6 +23,44 @@ func TestCPUSpeedMatchesWhatThePoCMeasured(t *testing.T) {
}
}
// Each thread reads at about one core's rate, up to the machine's: what
// llama-bench measured on node05 by thread count, and what was measured on
// node01 and node05 before, all within 15%.
func TestCPUSpeedFollowsTheThreads(t *testing.T) {
for _, c := range []struct {
bytes int64
all, core float64
threads int
want float64
}{
{4683073536, 43.9, 9.7, 1, 1.88}, // node05, qwen2.5-coder-7b
{4683073536, 43.9, 9.7, 2, 3.56},
{4683073536, 43.9, 9.7, 4, 6.35},
{4683073536, 43.9, 9.7, 8, 9.29},
{1117320736, 43.9, 9.7, 1, 8.31}, // node05, qwen2.5-1.5b
{1117320736, 43.9, 9.7, 4, 25.4},
{1117320736, 34.2, 10.1, 2, 14.5}, // node01 on its two cores
{9_000_000_000, 43.9, 9.7, 4, 3.3}, // node05, qwen2.5-14b on four
} {
got := CPUTokensPerSecondOn(c.bytes, c.all, c.core, c.threads)
if math.Abs(got-c.want)/c.want > 0.15 {
t.Errorf("%d bytes, %d threads: %.1f tok/s, measured %.1f", c.bytes, c.threads, got, c.want)
}
}
// A large machine, a few threads: the threads decide, not the machine.
few := CPUTokensPerSecondOn(18e9, 300, 10, 4)
if few > 2.5 {
t.Errorf("four threads of a 300 GB/s machine: %.1f tok/s, as though they read at the whole machine's rate", few)
}
// An agent that did not measure one core: the old reckoning.
if got, want := CPUTokensPerSecondOn(1e9, 40, 0, 4), CPUTokensPerSecond(1e9, 40); got != want {
t.Errorf("without one core's rate %.2f, want %.2f", got, want)
}
if CPUThreads(0, 64) != 32 || CPUThreads(6, 64) != 6 || CPUThreads(0, 1) != 1 {
t.Error("threads: the profile's cores, else half the node's")
}
}
func TestAModelOfExpertsReadsItsActiveShare(t *testing.T) {
for name, want := range map[string]float64{
"Qwen/Qwen3-30B-A3B-Instruct-2507-GGUF": 0.16,
@@ -35,3 +73,26 @@ func TestAModelOfExpertsReadsItsActiveShare(t *testing.T) {
}
}
}
// A plan's ways say how fast they answer: on the CPU by the node's memory
// and half its threads, on a card by its bandwidth.
func TestAPlansWaysSayHowFastTheyAnswer(t *testing.T) {
nodes := []PoolNode{{ID: "big", Class: "l40s-48g", GPUMemGB: 48, GPUs: 2, FreeGPUs: 2, HostMemGB: 512, HostFreeGB: 400,
CPUs: 64, MemBWGBs: 300, MemBWCoreGBs: 10, CardBWGBs: map[string]float64{"l40s-48g": 864}}}
opts := Options([]Artifact{{Name: "m-Q4_K_M.gguf", Format: "gguf", Bytes: 18e9, Quant: "Q4_K_M"}},
Shape{Params: 32e9}, nodes, 4096)
var cpu, gpu float64
for _, o := range opts {
if o.CPU {
cpu = o.TokensPerSecond
} else {
gpu = o.TokensPerSecond
}
}
if want := CPUTokensPerSecondOn(18e9, 300, 10, 32); math.Abs(cpu-want) > 0.01 {
t.Errorf("on the CPU %.2f tok/s, want %.2f", cpu, want)
}
if gpu < 3*cpu {
t.Errorf("on a card %.1f tok/s against %.1f on the CPU", gpu, cpu)
}
}
+82
View File
@@ -35,6 +35,17 @@ type Shape struct {
Params int64
// Experts is a mixture-of-experts model's routed experts; 0 for dense.
Experts int
// ActiveShare is the part of the weights a token reads; 0 is all of
// them.
ActiveShare float64
}
// share is ActiveShare, all of the weights when not known.
func (sh Shape) share() float64 {
if sh.ActiveShare <= 0 || sh.ActiveShare > 1 {
return 1
}
return sh.ActiveShare
}
// PoolNode is one ready node, as a plan sees it.
@@ -57,6 +68,13 @@ type PoolNode struct {
Engines []string
// CC is its newest card's compute capability, 0 when not known.
CC float64
// CPUs, MemBWGBs and MemBWCoreGBs are its CPUs and how fast its memory
// reads, with every CPU and with one: a model's speed on its CPU. Zero
// when not measured.
CPUs int
MemBWGBs, MemBWCoreGBs float64
// CardBWGBs is a card's memory bandwidth by class, for its speed there.
CardBWGBs map[string]float64
}
// PoolKind is one class of a node's devices.
@@ -119,6 +137,10 @@ type Option struct {
// NeedGB and HaveGB are then about.
StrataFamily string `json:"strata_family,omitempty"`
StrataSize string `json:"strata_size,omitempty"`
// TokensPerSecond is about how many tokens a second one reply gets this
// way, on the fastest host it fits; 0 when not known. On the CPU with
// the threads a replica is given by default: half the node's.
TokensPerSecond float64 `json:"tokens_per_second,omitempty"`
}
// GPUs is every card one replica takes.
@@ -151,6 +173,29 @@ func (o Option) WithContext(kvPerToken int64, tokens int) Option {
return o
}
// MostContext is the longest context, up to trained, this option still holds
// for one request, in whole thousands of tokens below trained: trained when
// the KV cache's size is unknown, 0 when trained is, for Strata (its context
// is its own), or when not even 1024 tokens fit.
func (o Option) MostContext(kvPerToken int64, trained int) int {
switch {
case trained <= 0 || o.Engine == "strata":
return 0
case kvPerToken <= 0 || o.WithContext(kvPerToken, trained).Fits():
return trained
}
lo, hi := 0, trained/1024+1 // lo fits (or is none), hi does not
for hi-lo > 1 {
mid := (lo + hi) / 2
if o.WithContext(kvPerToken, mid*1024).Fits() {
lo = mid
} else {
hi = mid
}
}
return lo * 1024
}
// MemoryText says what it needs, part by part.
func (o Option) MemoryText() string {
if o.Engine == "strata" {
@@ -222,6 +267,9 @@ func Options(arts []Artifact, sh Shape, nodes []PoolNode, context int) []Option
out = append(out, vllmOptions(a, kv, sh, nodes, bpw)...)
}
}
for i := range out {
out[i].TokensPerSecond = speedOf(out[i], bytesOf(arts, out[i].Artifact), sh, nodes)
}
Rank(out)
return out
}
@@ -539,3 +587,37 @@ func CardBudget(class string) float64 { return float64(MemFromClass(class)) * gp
func SplitsHeads(kvHeads, width int) bool {
return kvHeads == 0 || width <= 1 || kvHeads%width == 0 || width%kvHeads == 0
}
func bytesOf(arts []Artifact, name string) int64 {
for _, a := range arts {
if a.Name == name {
return a.Bytes
}
}
return 0
}
// speedOf is about how many tokens a second one reply gets in o, on the
// fastest node it fits: on the CPU by the node's memory and the threads a
// replica is given, on cards by their bandwidth.
func speedOf(o Option, bytes int64, sh Shape, nodes []PoolNode) float64 {
best := 0.0
for _, n := range nodes {
var tps float64
switch {
case o.CPU:
if float64(n.HostMemGB)*hostShare < o.NeedGB {
continue
}
tps = CPUTokensPerSecondOn(int64(float64(bytes)*sh.share()), n.MemBWGBs, n.MemBWCoreGBs, CPUThreads(0, n.CPUs))
case o.Engine != "strata":
bw := n.CardBWGBs[o.Class]
if bw <= 0 {
continue
}
tps = GPUTokensPerSecond(o, bytes, sh.share(), sh.Params, bw, Links{})
}
best = max(best, tps)
}
return best
}
+20
View File
@@ -193,3 +193,23 @@ func TestAnOptionSaysWhatItNeedsOfEachCardAndOfAll(t *testing.T) {
t.Errorf("a 2M-token context fits four cards: %s", huge.MemoryText())
}
}
func TestTheMostContextAnOptionHolds(t *testing.T) {
// 4 GB of weights on a 24 GB card, a token's cache 1 MiB.
o := Option{Engine: "llamacpp", Class: "6000-24g", GPUsPerNode: 1, Nodes: 1,
WeightsGB: 4, EngineGB: 1, NeedGB: 5, HaveGB: 21.6}
const kv = 1 << 20
if got := o.MostContext(kv, 8192); got != 8192 {
t.Errorf("a context that fits whole: %d, want what it was trained for", got)
}
got := o.MostContext(kv, 131072)
if got <= 0 || got%1024 != 0 || !o.WithContext(kv, got).Fits() || o.WithContext(kv, got+1024).Fits() {
t.Errorf("the most that fits is %d", got)
}
if got := o.MostContext(0, 32768); got != 32768 {
t.Errorf("an unknown cache size: %d, want what it was trained for", got)
}
if got := (Option{Engine: "strata"}).MostContext(kv, 32768); got != 0 {
t.Errorf("Strata's context is its own: %d", got)
}
}
+2 -2
View File
@@ -39,7 +39,7 @@ type Store interface {
SetUserState(ctx context.Context, id string, state store.UserState) error
DeleteUserSessions(ctx context.Context, userID string) error
SetStaffRole(ctx context.Context, id, role string) error
AddMembership(ctx context.Context, user string, scope store.Scope, id, role, by string, at time.Time) error
EnsureMembership(ctx context.Context, user string, scope store.Scope, id, by string, at time.Time) error
RemoveMembership(ctx context.Context, user string, scope store.Scope, id string, at time.Time) (int, error)
AllOrgs(ctx context.Context) ([]store.Org, error)
AllTeams(ctx context.Context) ([]store.TeamPath, error)
@@ -548,7 +548,7 @@ func (s *Server) member(ctx context.Context, group, userID string, joined bool)
return
}
if joined {
err = s.st.AddMembership(ctx, u.ID, scope, id, store.RoleMember, "scim", time.Now())
err = s.st.EnsureMembership(ctx, u.ID, scope, id, "scim", time.Now())
} else {
_, err = s.st.RemoveMembership(ctx, u.ID, scope, id, time.Now())
}
+1 -1
View File
@@ -58,7 +58,7 @@ func (m *mem) DeleteUserSessions(_ context.Context, id string) error {
return nil
}
func (m *mem) SetStaffRole(_ context.Context, id, role string) error { m.roles[id] = role; return nil }
func (m *mem) AddMembership(_ context.Context, u string, sc store.Scope, id, _, _ string, _ time.Time) error {
func (m *mem) EnsureMembership(_ context.Context, u string, sc store.Scope, id, _ string, _ time.Time) error {
m.members[u+"@"+string(sc)+":"+id] = true
return nil
}
+344
View File
@@ -0,0 +1,344 @@
// Package slurm is what llmgw knows of Slurm (BACKLOG P): the files a
// controller runs on, written from what llmgw knows of its nodes, the munge
// key every machine shares, and reading what slurmd and scontrol say.
//
// Nothing here runs anything. The agent installs and starts the daemons; the
// leader decides what each node is and writes the files with this package.
package slurm
import (
"crypto/hkdf"
"crypto/sha256"
"fmt"
"regexp"
"slices"
"strconv"
"strings"
)
// Paths are where the files are on every distribution llmgw installs on:
// Debian and Ubuntu since Slurm 20.11, and EPEL.
const (
ConfDir = "/etc/slurm"
MungeKeyPath = "/etc/munge/munge.key"
StateDir = "/var/spool/slurmctld"
SpoolDir = "/var/spool/slurmd"
LogDir = "/var/log/slurm"
// CtldPort is slurmctld's, which configless slurmd fetch their files
// from.
CtldPort = 6817
// SlurmdPort is slurmd's, which the controller and srun reach.
SlurmdPort = 6818
// SrunPortsFrom and SrunPortsTo are where srun listens for the steps it
// starts: a range of its own, so a firewall can let it in (Slurm's own
// example range). Without one srun takes any ephemeral port.
SrunPortsFrom = 60001
SrunPortsTo = 63000
)
// ReasonNew is why a node new to slurm.conf starts drained. Every reason
// llmgw drains with starts with "llmgw", and only those are resumed by it.
const ReasonNew = "llmgw: not lent yet"
// MinVersion is the oldest Slurm llmgw runs: the first to put jobs in
// cgroup v2, which every distribution llmgw supports now mounts alone.
var MinVersion = [2]int{22, 5}
// MungeKeyLen is the length munge's own tool writes.
const MungeKeyLen = 1024
// MungeKey is the key every machine of the Slurm cluster shares, derived
// from a cluster secret, so a control plane that takes over writes the same
// one and nobody copies it by hand.
func MungeKey(secret string) ([]byte, error) {
if len(secret) < 32 {
return nil, fmt.Errorf("the secret a munge key is made from is too short")
}
return hkdf.Key(sha256.New, []byte(secret), nil, "llmgw slurm munge key v1", MungeKeyLen)
}
// Hardware is a node's line from `slurmd -C`: how slurmd counts it, which is
// what slurm.conf must say or the controller refuses the node.
type Hardware struct {
CPUs int
Boards int
SocketsPerBoard int
CoresPerSocket int
ThreadsPerCore int
// RealMemory is in MiB.
RealMemory int64
}
// ParseHardware reads `slurmd -C`.
func ParseHardware(out string) (Hardware, error) {
var h Hardware
for _, line := range strings.Split(out, "\n") {
if !strings.HasPrefix(strings.TrimSpace(line), "NodeName=") {
continue
}
for _, f := range strings.Fields(line) {
k, v, _ := strings.Cut(f, "=")
n, err := strconv.ParseInt(v, 10, 64)
if err != nil {
continue
}
switch k {
case "CPUs":
h.CPUs = int(n)
case "Boards":
h.Boards = int(n)
case "SocketsPerBoard":
h.SocketsPerBoard = int(n)
case "CoresPerSocket":
h.CoresPerSocket = int(n)
case "ThreadsPerCore":
h.ThreadsPerCore = int(n)
case "RealMemory":
h.RealMemory = n
}
}
break
}
if h.CPUs <= 0 || h.RealMemory <= 0 {
return Hardware{}, fmt.Errorf("slurmd -C said no CPUs or memory: %q", strings.TrimSpace(out))
}
return h, nil
}
// GPU is one of a node's cards.
type GPU struct {
// Minor is N of /dev/nvidiaN.
Minor int
// Name is the driver's, as nvidia-smi gives it.
Name string
}
// Node is one machine slurmd runs on.
type Node struct {
Name string
// Addr is where slurmctld reaches it: its llmgw management address.
Addr string
Hardware
GPUs []GPU
}
// Cluster is everything slurm.conf says.
type Cluster struct {
Name string
Controller string
ControllerAddr string
Partition string
// Nodes without hardware (not reported yet) are left out of the files
// until they are.
Nodes []Node
}
// gpuModel is a word that names a card's model: A100, H100, V100, T4, L40S.
var gpuModel = regexp.MustCompile(`^[a-z]{1,2}\d{1,4}[a-z]?$`)
// GPUType is a card's name as a Slurm GRES type: "NVIDIA A100-SXM4-80GB" is
// a100, "NVIDIA GeForce RTX 3090" rtx3090, "Tesla T4" t4. A name it cannot
// read is "gpu".
func GPUType(name string) string {
words := strings.Fields(strings.ToLower(strings.NewReplacer("-", " ", "_", " ").Replace(name)))
for i, w := range words {
if (w == "rtx" || w == "gtx") && i+1 < len(words) {
return w + words[i+1]
}
if gpuModel.MatchString(w) {
return w
}
}
return "gpu"
}
// gpuGroups are a node's cards by type, the types sorted.
func gpuGroups(gs []GPU) ([]string, map[string][]int) {
by := map[string][]int{}
for _, g := range gs {
t := GPUType(g.Name)
by[t] = append(by[t], g.Minor)
}
types := make([]string, 0, len(by))
for t, ms := range by {
slices.Sort(ms)
types = append(types, t)
}
slices.Sort(types)
return types, by
}
// memoryKept is what is left out of a node's RealMemory: slurmd -C reads
// what the kernel says now, which is a little less after a reboot, and a
// node whose memory is below what slurm.conf says is refused.
const memoryKept = 0.97
func (c Cluster) ready() []Node {
var out []Node
for _, n := range c.Nodes {
if n.CPUs > 0 && n.RealMemory > 0 {
out = append(out, n)
}
}
slices.SortFunc(out, func(a, b Node) int { return strings.Compare(a.Name, b.Name) })
return out
}
const header = "# Written by llmgw (BACKLOG P), on the node that runs slurmctld; the nodes read it\n" +
"# from there. Change it through the cluster's settings (slurm.*), not here.\n"
// SlurmConf is slurm.conf.
func (c Cluster) SlurmConf() string {
var b strings.Builder
b.WriteString(header)
fmt.Fprintf(&b, "ClusterName=%s\n", c.Name)
fmt.Fprintf(&b, "SlurmctldHost=%s(%s)\n", c.Controller, c.ControllerAddr)
b.WriteString(`SlurmctldParameters=enable_configless
SlurmctldPort=6817
SlurmdPort=6818
SrunPortRange=60001-63000
AuthType=auth/munge
CredType=cred/munge
SlurmUser=slurm
SlurmdUser=root
StateSaveLocation=` + StateDir + `
SlurmdSpoolDir=` + SpoolDir + `
SlurmctldLogFile=` + LogDir + `/slurmctld.log
SlurmdLogFile=` + LogDir + `/slurmd.log
SlurmctldPidFile=/run/slurmctld.pid
SlurmdPidFile=/run/slurmd.pid
ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup,task/affinity
# Nothing gathered for accounting until there is somewhere to keep it
# (slurmdbd, later); gathered and not kept is an error in every log.
JobAcctGatherType=jobacct_gather/none
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory
SchedulerType=sched/backfill
GresTypes=gpu
MpiDefault=none
# No mail sent; without it every start says MailProg is invalid.
MailProg=/bin/true
# Without accounting storage, 23.11's main scheduler marks a batch job
# InvalidAccount; backfill starts it on its next pass, so it is kept short.
SchedulerParameters=bf_interval=10
# A node that went down and comes back is taken again; one llmgw drained
# stays drained until llmgw resumes it.
ReturnToService=2
SlurmctldTimeout=120
SlurmdTimeout=300
`)
nodes := c.ready()
names := make([]string, 0, len(nodes))
for _, n := range nodes {
names = append(names, n.Name)
fmt.Fprintf(&b, "NodeName=%s NodeAddr=%s CPUs=%d", n.Name, n.Addr, n.CPUs)
if n.Boards > 0 && n.SocketsPerBoard > 0 && n.CoresPerSocket > 0 && n.ThreadsPerCore > 0 {
fmt.Fprintf(&b, " Boards=%d SocketsPerBoard=%d CoresPerSocket=%d ThreadsPerCore=%d",
n.Boards, n.SocketsPerBoard, n.CoresPerSocket, n.ThreadsPerCore)
}
fmt.Fprintf(&b, " RealMemory=%d", int64(float64(n.RealMemory)*memoryKept))
if types, by := gpuGroups(n.GPUs); len(types) > 0 {
gres := make([]string, 0, len(types))
for _, t := range types {
gres = append(gres, fmt.Sprintf("gpu:%s:%d", t, len(by[t])))
}
fmt.Fprintf(&b, " Gres=%s", strings.Join(gres, ","))
}
// Drained until llmgw lends it: a node new to the controller would
// otherwise take jobs in the minute before its agent drains it.
fmt.Fprintf(&b, " State=DRAIN Reason=%q\n", ReasonNew)
}
if len(names) > 0 {
fmt.Fprintf(&b, "PartitionName=%s Nodes=%s Default=YES MaxTime=INFINITE State=UP\n",
c.Partition, strings.Join(names, ","))
}
return b.String()
}
// GresConf is gres.conf: each node's cards by device file, so a job is given
// the cards it asked for and kept off the others.
func (c Cluster) GresConf() string {
var b strings.Builder
b.WriteString(header)
for _, n := range c.ready() {
types, by := gpuGroups(n.GPUs)
for _, t := range types {
files := make([]string, 0, len(by[t]))
for _, m := range by[t] {
files = append(files, fmt.Sprintf("/dev/nvidia%d", m))
}
fmt.Fprintf(&b, "NodeName=%s Name=gpu Type=%s File=%s\n", n.Name, t, strings.Join(files, ","))
}
}
return b.String()
}
// CgroupConf is cgroup.conf: a job kept to its cores, memory and cards.
func (c Cluster) CgroupConf() string {
return header + `CgroupPlugin=autodetect
ConstrainCores=yes
ConstrainDevices=yes
ConstrainRAMSpace=yes
`
}
// Files are the controller's files, by name in ConfDir.
func (c Cluster) Files() map[string]string {
return map[string]string{"slurm.conf": c.SlurmConf(), "gres.conf": c.GresConf(), "cgroup.conf": c.CgroupConf()}
}
// NodeState is what `scontrol show node -o NAME` says of a node.
type NodeState struct {
// State is Slurm's, with its flags: IDLE, ALLOCATED, MIXED+DRAIN,
// DOWN+NOT_RESPONDING.
State string
Reason string
}
// Drained is whether the controller gives the node nothing new.
func (s NodeState) Drained() bool {
return strings.Contains(s.State, "DRAIN") || strings.Contains(s.State, "DOWN")
}
// ParseNodeState reads `scontrol show node -o NAME`, one line of key=value.
func ParseNodeState(out string) (NodeState, error) {
line := strings.TrimSpace(out)
if !strings.HasPrefix(line, "NodeName=") {
return NodeState{}, fmt.Errorf("scontrol said %q", line)
}
var s NodeState
if i := strings.Index(line, " State="); i >= 0 {
s.State, _, _ = strings.Cut(line[i+len(" State="):], " ")
}
// Reason is last but for its [user@time] and may have spaces.
if i := strings.Index(line, " Reason="); i >= 0 {
r := line[i+len(" Reason="):]
if j := strings.LastIndex(r, " ["); j >= 0 {
r = r[:j]
}
s.Reason = strings.TrimSpace(r)
}
return s, nil
}
// ParseVersion reads "slurm 23.11.4" or "slurm-wlm 21.08.5" as major, minor.
func ParseVersion(out string) ([2]int, error) {
for _, f := range strings.Fields(out) {
parts := strings.Split(f, ".")
if len(parts) < 2 {
continue
}
major, err1 := strconv.Atoi(parts[0])
minor, err2 := strconv.Atoi(parts[1])
if err1 == nil && err2 == nil {
return [2]int{major, minor}, nil
}
}
return [2]int{}, fmt.Errorf("no version in %q", strings.TrimSpace(out))
}
// TooOld is whether v is older than MinVersion.
func TooOld(v [2]int) bool {
return v[0] < MinVersion[0] || (v[0] == MinVersion[0] && v[1] < MinVersion[1])
}
+151
View File
@@ -0,0 +1,151 @@
package slurm
import (
"bytes"
"strings"
"testing"
)
func TestMungeKey(t *testing.T) {
s := strings.Repeat("ab", 32)
a, err := MungeKey(s)
if err != nil {
t.Fatal(err)
}
b, _ := MungeKey(s)
if len(a) != MungeKeyLen || !bytes.Equal(a, b) {
t.Fatalf("a key is %d bytes and the same each time: got %d, equal %v", MungeKeyLen, len(a), bytes.Equal(a, b))
}
c, _ := MungeKey(strings.Repeat("cd", 32))
if bytes.Equal(a, c) {
t.Fatal("two secrets made one key")
}
if _, err := MungeKey("short"); err == nil {
t.Fatal("a short secret was taken")
}
}
func TestParseHardware(t *testing.T) {
out := "NodeName=cuda01 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=64302\n" +
"UpTime=3-01:02:03\n"
h, err := ParseHardware(out)
if err != nil {
t.Fatal(err)
}
want := Hardware{CPUs: 16, Boards: 1, SocketsPerBoard: 2, CoresPerSocket: 4, ThreadsPerCore: 2, RealMemory: 64302}
if h != want {
t.Fatalf("got %+v", h)
}
if _, err := ParseHardware("slurmd: error: something\n"); err == nil {
t.Fatal("an error was read as hardware")
}
}
func TestGPUType(t *testing.T) {
for name, want := range map[string]string{
"NVIDIA A100-SXM4-80GB": "a100",
"NVIDIA H100 80GB HBM3": "h100",
"Tesla V100-SXM2-32GB": "v100",
"Tesla T4": "t4",
"NVIDIA GeForce RTX 3090": "rtx3090",
"NVIDIA RTX A6000": "rtxa6000",
"NVIDIA L40S": "l40s",
"Quadro RTX 8000": "rtx8000",
"NVIDIA GeForce GTX 1080": "gtx1080",
"": "gpu",
"Some Accelerator Unknown": "gpu",
} {
if got := GPUType(name); got != want {
t.Errorf("GPUType(%q) = %q, want %q", name, got, want)
}
}
}
func testCluster() Cluster {
return Cluster{Name: "llmgw", Controller: "node06", ControllerAddr: "10.0.0.146", Partition: "gpu",
Nodes: []Node{
{Name: "site2", Addr: "10.1.0.5",
Hardware: Hardware{CPUs: 32, Boards: 1, SocketsPerBoard: 2, CoresPerSocket: 8, ThreadsPerCore: 2, RealMemory: 100000},
GPUs: []GPU{{Minor: 2, Name: "Tesla T4"}, {Minor: 0, Name: "Tesla V100-SXM2-32GB"}, {Minor: 1, Name: "Tesla V100-SXM2-32GB"}}},
{Name: "cuda01", Addr: "10.0.0.147", Hardware: Hardware{CPUs: 16, RealMemory: 64000},
GPUs: []GPU{{Minor: 0, Name: "NVIDIA GeForce RTX 3090"}}},
{Name: "waiting", Addr: "10.0.0.9"},
}}
}
func TestSlurmConf(t *testing.T) {
conf := testCluster().SlurmConf()
for _, want := range []string{
"ClusterName=llmgw\n",
"SlurmctldHost=node06(10.0.0.146)\n",
"SlurmctldParameters=enable_configless\n",
"NodeName=cuda01 NodeAddr=10.0.0.147 CPUs=16 RealMemory=62080 Gres=gpu:rtx3090:1 State=DRAIN Reason=\"llmgw: not lent yet\"\n",
"NodeName=site2 NodeAddr=10.1.0.5 CPUs=32 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=97000 Gres=gpu:t4:1,gpu:v100:2 State=DRAIN Reason=\"llmgw: not lent yet\"\n",
"PartitionName=gpu Nodes=cuda01,site2 Default=YES",
} {
if !strings.Contains(conf, want) {
t.Errorf("slurm.conf has no %q:\n%s", want, conf)
}
}
if strings.Contains(conf, "waiting") {
t.Error("a node that has not reported its hardware is in slurm.conf")
}
if strings.Index(conf, "NodeName=cuda01") > strings.Index(conf, "NodeName=site2") {
t.Error("nodes are not in order")
}
}
func TestSlurmConfNoNodes(t *testing.T) {
c := testCluster()
c.Nodes = nil
if strings.Contains(c.SlurmConf(), "PartitionName") {
t.Error("a partition with no nodes would stop slurmctld starting")
}
}
func TestGresConf(t *testing.T) {
g := testCluster().GresConf()
for _, want := range []string{
"NodeName=cuda01 Name=gpu Type=rtx3090 File=/dev/nvidia0\n",
"NodeName=site2 Name=gpu Type=t4 File=/dev/nvidia2\n",
"NodeName=site2 Name=gpu Type=v100 File=/dev/nvidia0,/dev/nvidia1\n",
} {
if !strings.Contains(g, want) {
t.Errorf("gres.conf has no %q:\n%s", want, g)
}
}
}
func TestParseNodeState(t *testing.T) {
out := "NodeName=cuda01 Arch=x86_64 CoresPerSocket=8 CPUAlloc=0 State=IDLE+DRAIN ThreadsPerCore=2 " +
"Partitions=gpu Reason=llmgw: serving models [root@2026-11-11T10:00:00]\n"
s, err := ParseNodeState(out)
if err != nil {
t.Fatal(err)
}
if s.State != "IDLE+DRAIN" || s.Reason != "llmgw: serving models" || !s.Drained() {
t.Fatalf("got %+v", s)
}
s, _ = ParseNodeState("NodeName=cuda01 State=MIXED Partitions=gpu\n")
if s.Drained() || s.Reason != "" {
t.Fatalf("got %+v", s)
}
if _, err := ParseNodeState("Node cuda02 not found\n"); err == nil {
t.Fatal("an unknown node was read")
}
}
func TestParseVersion(t *testing.T) {
for out, want := range map[string][2]int{"slurm 23.11.4\n": {23, 11}, "slurm-wlm 21.08.5": {21, 8}, "slurm 22.05.9": {22, 5}} {
v, err := ParseVersion(out)
if err != nil || v != want {
t.Errorf("ParseVersion(%q) = %v, %v", out, v, err)
}
}
if !TooOld([2]int{21, 8}) || TooOld([2]int{22, 5}) || TooOld([2]int{23, 2}) {
t.Error("TooOld is wrong")
}
if _, err := ParseVersion("command not found"); err == nil {
t.Error("no version was read as one")
}
}
+54 -5
View File
@@ -14,10 +14,19 @@ var ErrNotMember = errors.New("not a member of that project")
// Roles a membership can carry.
const (
RoleMember = "member"
// RoleManager is reserved for delegated administration of a scope.
// RoleManager, on an organization, may look at that organization's page:
// its use, members and limits. It changes nothing.
RoleManager = "manager"
// RoleOrgAdmin, on an organization, also changes it: its teams and
// projects, who is in them, and their keys. Its limits, the accounts
// themselves and the rest of the cluster stay with the cluster's staff.
RoleOrgAdmin = "admin"
)
// MembershipRoles are the roles a membership may carry; manager and admin mean
// something only on an organization.
var MembershipRoles = []string{RoleMember, RoleManager, RoleOrgAdmin}
// Membership is a person's place at one level of the tree.
type Membership struct {
UserID string
@@ -43,6 +52,12 @@ func (s *DB) AddMembership(ctx context.Context, userID string, scope Scope, scop
if role == "" {
role = RoleMember
}
if role != RoleMember && role != RoleManager && role != RoleOrgAdmin {
return fmt.Errorf("a membership's role is member, manager or admin, not %q", role)
}
if role != RoleMember && scope != ScopeOrg {
return fmt.Errorf("%s is a role on an organization, not on a %s", role, scope)
}
table := map[Scope]string{ScopeOrg: "organizations", ScopeTeam: "teams", ScopeProject: "projects"}[scope]
if err := s.exists(ctx, string(scope), `SELECT COUNT(*) FROM `+table+` WHERE id = ?`, scopeID); err != nil {
return err
@@ -55,6 +70,27 @@ func (s *DB) AddMembership(ctx context.Context, userID string, scope Scope, scop
return err
}
// EnsureMembership puts a person in a scope as a member if they are not there
// yet, and leaves a membership that exists as it is. It is for what a
// directory or an identity provider says on every sign-in or sync: a group
// that places someone in an organization must not, by saying so again, take
// away the manager or admin role an administrator gave them there.
func (s *DB) EnsureMembership(ctx context.Context, userID string, scope Scope, scopeID, by string, at time.Time) error {
if !validMembershipScope(scope) {
return fmt.Errorf("a membership is in an org, team or project, not %q", scope)
}
table := map[Scope]string{ScopeOrg: "organizations", ScopeTeam: "teams", ScopeProject: "projects"}[scope]
if err := s.exists(ctx, string(scope), `SELECT COUNT(*) FROM `+table+` WHERE id = ?`, scopeID); err != nil {
return err
}
_, err := s.db.ExecContext(ctx, `
INSERT INTO memberships (user_id, scope_type, scope_id, role, added_at, added_by)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT (user_id, scope_type, scope_id) DO NOTHING`,
userID, string(scope), scopeID, RoleMember, millis(at), nullString(by))
return err
}
// RemoveMembership takes a person out of a scope and revokes the keys they
// made in any project they can no longer use, returning how many.
//
@@ -306,23 +342,36 @@ func (s *DB) resolveChain(ctx context.Context, e *Effective, keyLevel bool) erro
return nil
}
// ManagedOrgs lists the organizations a person manages.
// ManagedOrgs lists the organizations a person manages or administers: those
// whose page they may see.
func (s *DB) ManagedOrgs(ctx context.Context, userID string) ([]Org, error) {
return queryAll(ctx, s.db, `
SELECT o.id, o.name FROM memberships m JOIN organizations o ON o.id = m.scope_id
WHERE m.user_id = ? AND m.scope_type = 'org' AND m.role = 'manager'
WHERE m.user_id = ? AND m.scope_type = 'org' AND m.role IN ('manager', 'admin')
ORDER BY o.name`, []any{userID}, func(rows *sql.Rows) (Org, error) {
var o Org
return o, rows.Scan(&o.ID, &o.Name)
})
}
// ManagesOrg reports whether a person manages an organization.
// ManagesOrg reports whether a person manages or administers an
// organization: whether they may see its page.
func (s *DB) ManagesOrg(ctx context.Context, userID, orgID string) (bool, error) {
var n int
err := s.db.QueryRowContext(ctx, `
SELECT COUNT(*) FROM memberships
WHERE user_id = ? AND scope_type = 'org' AND scope_id = ? AND role = 'manager'`,
WHERE user_id = ? AND scope_type = 'org' AND scope_id = ? AND role IN ('manager', 'admin')`,
userID, orgID).Scan(&n)
return n > 0, err
}
// AdministersOrg reports whether a person administers an organization:
// whether they may change it.
func (s *DB) AdministersOrg(ctx context.Context, userID, orgID string) (bool, error) {
var n int
err := s.db.QueryRowContext(ctx, `
SELECT COUNT(*) FROM memberships
WHERE user_id = ? AND scope_type = 'org' AND scope_id = ? AND role = 'admin'`,
userID, orgID).Scan(&n)
return n > 0, err
}
+75
View File
@@ -307,3 +307,78 @@ func TestASuspendedPersonsKeysStopWorkingAndComeBackWithThem(t *testing.T) {
t.Errorf("reinstated, the key still refused: %v", err)
}
}
func TestAnOrganizationsAdminAlsoManagesIt(t *testing.T) {
f := newFixture(t)
ctx := t.Context()
if err := f.s.AddMembership(ctx, f.user, ScopeOrg, f.acme, RoleOrgAdmin, "t", time.Now()); err != nil {
t.Fatal(err)
}
if ok, _ := f.s.AdministersOrg(ctx, f.user, f.acme); !ok {
t.Error("an admin of Acme is not recognised as one")
}
if ok, _ := f.s.ManagesOrg(ctx, f.user, f.acme); !ok {
t.Error("an admin of Acme may not see its page")
}
if ok, _ := f.s.AdministersOrg(ctx, f.user, f.scania); ok {
t.Error("administering Acme made them an admin of Scania")
}
if orgs, _ := f.s.ManagedOrgs(ctx, f.user); len(orgs) != 1 || orgs[0].Name != "Acme" {
t.Errorf("ManagedOrgs = %+v", orgs)
}
// A manager sees the page and changes nothing.
if err := f.s.AddMembership(ctx, f.user, ScopeOrg, f.acme, RoleManager, "t", time.Now()); err != nil {
t.Fatal(err)
}
if ok, _ := f.s.AdministersOrg(ctx, f.user, f.acme); ok {
t.Error("a manager is taken for an admin")
}
}
func TestManagerAndAdminAreRolesOnAnOrganizationOnly(t *testing.T) {
f := newFixture(t)
ctx := t.Context()
for _, c := range []struct {
scope Scope
id string
role string
}{
{ScopeProject, f.gateway, RoleOrgAdmin},
{ScopeTeam, f.platform, RoleManager},
{ScopeOrg, f.acme, "owner"},
} {
if err := f.s.AddMembership(ctx, f.user, c.scope, c.id, c.role, "t", time.Now()); err == nil {
t.Errorf("%s on a %s was taken", c.role, c.scope)
}
}
}
func TestASyncedGroupLeavesARoleAsItIs(t *testing.T) {
// A directory group that places someone in Acme says so at every sign-in;
// saying it again must not take away the admin role they were given there.
f := newFixture(t)
ctx := t.Context()
if err := f.s.AddMembership(ctx, f.user, ScopeOrg, f.acme, RoleOrgAdmin, "t", time.Now()); err != nil {
t.Fatal(err)
}
if err := f.s.EnsureMembership(ctx, f.user, ScopeOrg, f.acme, "oidc", time.Now()); err != nil {
t.Fatal(err)
}
if ok, _ := f.s.AdministersOrg(ctx, f.user, f.acme); !ok {
t.Error("a sign-in's group took the admin role away")
}
// And one that is not there yet is added as a member.
if err := f.s.EnsureMembership(ctx, f.user, ScopeOrg, f.scania, "oidc", time.Now()); err != nil {
t.Fatal(err)
}
ms, _ := f.s.AllMemberships(ctx)
found := false
for _, m := range ms {
if m.UserID == f.user && m.Scope == ScopeOrg && m.ScopeID == f.scania {
found = m.Role == RoleMember
}
}
if !found {
t.Error("a new group did not make them a member of Scania")
}
}
+7
View File
@@ -50,6 +50,13 @@ func (d AboutData) InstallURL() string {
return "/install"
}
// DeckURL is the introduction for decision makers, as a PDF. It lives in the
// repository rather than in the binary, so every gateway does not carry a
// megabyte of slides; gateway and website link to the same file.
func (AboutData) DeckURL() string {
return repoURL + "/raw/branch/main/docs/llmgw-intro.pdf"
}
// Still is a screen of the install guide's recording, for the steps here.
func (d AboutData) Still(name string) template.HTML {
return InstallData{Site: d.Site}.Still(name)
+2
View File
@@ -123,6 +123,8 @@ type AccountPages struct {
personalDefaults *store.Limits
// orgs is set when /org is mounted, so the menu can offer it to managers.
orgs OrgViewer
// orgEditor changes an organization from its page, for its admin.
orgEditor OrgEditor
// llamaVersion is the llama.cpp build nodes install, for judging whether a
// GGUF can be served before it is fetched or catalogued.
llamaVersion string
+49 -2
View File
@@ -108,6 +108,23 @@ type modelUse struct {
// Declared models are read from server.yaml on every start, so a mode
// changed here would revert; the page says where to change it instead.
Declared bool
// What its card changes beside: the context and longest answer, whether
// it is stopped, its profile on cards and the cards a replica takes.
ContextLength, MaxOutput int
Stopped bool
GPUProfile string
GPUs int
// Profiles are where its replicas run and, Split, how many on each.
Profiles []profileUse
Split bool
// CanCPU says a profile on CPUs can be added — a llama.cpp model with
// none — reserving CPUMemoryGB and NewCPUCores, answering about
// CPUSpeed tokens a second on CPUSpeedNode.
CanCPU bool
CPUMemoryGB int
NewCPUCores int
CPUSpeed float64
CPUSpeedNode string
}
// mirrorRow is one model version as the page shows it.
@@ -250,7 +267,7 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
http.NotFound(w, r)
return
}
d := adminMirrorData{CSRF: csrfFor(r), Inline: true, HasRegistry: true}
d := adminMirrorData{CSRF: csrfIfMay(r, "/admin/mirror/x"), Inline: true, HasRegistry: true}
p.openRepo(r, &d, repo)
if d.RegistryErr != "" {
// Read by the page's own script, which shows it beneath the
@@ -324,6 +341,18 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
if name == "" || (mode != modeAlways && mode != modeOnDemand && mode != modeStopped) {
return "", errors.New(l.T("mirror.err.choose_mode"))
}
// Stopped is the model kept as it was, and started again so:
// not its scaling set to nothing, which forgot what it was —
// and its replicas per profile with it.
if mode == modeStopped {
if _, err := d.Catalog.Stop(ctx, name, admin.Email); err != nil {
return "", err
}
return l.T("mirror.msg.stopped", name), nil
}
if _, err := d.Catalog.Start(ctx, name, admin.Email); err == nil && mode == modeAlways {
return l.T("mirror.msg.model_started", name), nil
}
replicas, _ := strconv.Atoi(r.FormValue("replicas"))
sc := scalingFor(mode, replicas, 0)
if _, err := d.Catalog.Scale(ctx, name, sc, admin.Email); err != nil {
@@ -339,6 +368,7 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
}))
mux.HandleFunc("POST /admin/mirror/replicas", p.mirrorAction(m, p.setModel))
p.registerModelSettings(mux, m)
mux.HandleFunc("POST /admin/mirror/adapters", p.mirrorAction(m, p.setAdapters))
mux.HandleFunc("POST /admin/mirror/serve", p.mirrorAction(m,
@@ -370,6 +400,17 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
if err != nil {
return "", err
}
// Slow enough that a reply is minutes is a choice to make on
// purpose too, GPUs free or not: on a fleet whose every node
// has cards, the CPU is theirs, and some are slow.
if prof := model.Profiles[0]; !judge && prof.CPUOnly() && r.FormValue("cpu_ok") == "" {
share := p.checkSource(ctx, model.Source).Header.ActiveShare()
hosts := cpuHosts(p.fleet, prof.MinMemoryGB)
if tps, node := bestCPUSpeed(hosts, int64(float64(size)*share), prof.CPUCores); tps > 0 && tps < modelplan.CPUTooSlowTPS {
return "", errors.New(l.T("serve.err.cpu_too_slow", model.Name, l.Num(tps, 1), node,
l.Int(int64(500/tps/60+0.5))))
}
}
// The weights and the context's cache have to fit the card:
// an 8B asked for a 131072-token pool on 24 GB loaded its
// weights and failed to make its context, again and again.
@@ -461,7 +502,7 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
Addr: m.Addr(),
Message: msg,
Error: errMsg,
CSRF: csrfFor(r),
CSRF: csrfIfMay(r, "/admin/mirror/x"),
Version: p.version,
Now: time.Now().UTC().Format(time.RFC3339),
}
@@ -554,6 +595,7 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
if len(m.Profiles) > 0 {
u.MemoryGB, u.CPUCores = m.Profiles[0].MinMemoryGB, m.Profiles[0].CPUCores
}
p.describeProfiles(&u, m, d.Rows[i].Status)
if v, ok := views[m.Name]; ok {
u.Fleet, u.Ready, u.Wanted = true, v.Ready, v.Desired
u.Parked, u.Unplaceable = v.Parked, v.Unplaceable
@@ -694,6 +736,11 @@ func mirrorTabs(l *Lang, d adminMirrorData) []mirrorTab {
tabs = append(tabs, mirrorTab{Label: l.T("mirror.tab.strata"),
Href: "/admin/mirror?tab=strata", Active: d.Tab == tabStrata})
}
if d.CSRF == "" {
// Fetching is all that tab does: someone who may not fetch is not
// offered an empty page.
return tabs
}
return append(tabs, mirrorTab{Label: l.T("mirror.tab.fetch"),
Href: "/admin/mirror?tab=fetch", Active: d.Tab == tabFetch})
}
+1 -1
View File
@@ -149,7 +149,7 @@ func (p *AccountPages) detail(r *http.Request, sel string, u *adminUsersData) *p
}
ctx := r.Context()
l := langOf(r)
d := &peopleDetail{Kind: store.Scope(kind), ID: id, CSRF: csrfFor(r),
d := &peopleDetail{Kind: store.Scope(kind), ID: id, CSRF: csrfIfMay(r, "/admin/users"),
UsageWindow: l.T("usage.window.7d"), QuotaWindow: l.T("quotas.since")}
if !p.nameDetail(d, u) {
return nil
+1 -1
View File
@@ -353,7 +353,7 @@ func (p *AccountPages) buildQuotas(r *http.Request, s QuotaAdmin, msg, errMsg st
d := adminQuotasData{
Nav: navFrom(r, "quotas"),
Since: l.T("quotas.since"), Message: msg, Error: errMsg,
CSRF: csrfFor(r), Version: p.version,
CSRF: csrfIfMay(r, "/admin/quotas"), Version: p.version,
Now: time.Now().UTC().Format(time.RFC3339),
}
+135 -3
View File
@@ -12,9 +12,11 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
@@ -33,6 +35,12 @@ type CatalogAdmin interface {
SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
SetAdapters(ctx context.Context, name string, adapters []catalog.Adapter, by string) (catalog.Model, error)
// Stop takes a model out of service and keeps it; Start runs it again
// as it was.
Stop(ctx context.Context, name, by string) (catalog.Model, error)
Start(ctx context.Context, name, by string) (catalog.Model, error)
// SetProfileReplicas counts its replicas per profile, adding cpu.
SetProfileReplicas(ctx context.Context, name string, counts map[string]int, cpu *modelsvc.CPUProfile, by string) (catalog.Model, error)
}
// serveForm is the offer to put one mirrored version into service.
@@ -86,6 +94,20 @@ type serveForm struct {
// version of a family Strata runs, served on it.
StrataFamily, StrataSize string
// CPUHosts are the nodes a CPU replica of it could run on, with their
// memory's rates, and ReadBytes what a token of it reads: what the page
// reckons its speed on the CPU from, live, as the cores change.
CPUHosts []speedHost
ReadBytes int64
// CPUSpeed is about how many tokens a second one reply gets on the CPU,
// on CPUSpeedNode with the cores the form starts with; 0 when unknown.
CPUSpeed float64
CPUSpeedNode string
// GPUSpeed is the same on one card of the fleet's fastest class,
// GPUSpeedClass; 0 when the fleet has none it knows.
GPUSpeed float64
GPUSpeedClass string
// Refusal is set when this version cannot be served as it stands, and says
// why. Offered and then failing at placement is worse than not offered.
Refusal string
@@ -202,7 +224,10 @@ func planServe(l *Lang, st mirror.Status, files []mirror.Status, base string,
if weights <= 0 {
weights = st.BytesDone
}
if b := header.MemoryBytes(weights, defaultCtxPool); b > 0 {
if c, par := mostContext(header, weights, f.Placements, f.GPUs, fleet); c > 0 {
f.Context, f.Parallel, f.CtxPool = c, par, c*par
}
if b := header.MemoryBytes(weights, f.CtxPool); b > 0 {
kvHeads := header.HeadCountKV
if kvHeads == 0 {
kvHeads = header.HeadCount
@@ -213,8 +238,8 @@ func planServe(l *Lang, st mirror.Status, files []mirror.Status, base string,
// flat "half again" could not see.
f.MinMemoryGB = int((b + 1<<30 - 1) >> 30)
f.MemoryBasis = l.T("serve.basis.shape",
l.Num(float64(weights)/(1<<30), 1), l.Num(float64(int64(defaultCtxPool)*header.KVBytesPerToken())/(1<<30), 1),
l.Int(defaultCtxPool), header.BlockCount, kvHeads,
l.Num(float64(weights)/(1<<30), 1), l.Num(float64(int64(f.CtxPool)*header.KVBytesPerToken())/(1<<30), 1),
l.Int(int64(f.CtxPool)), header.BlockCount, kvHeads,
l.Num(float64(int64(1<<30)+weights/20)/(1<<30), 1))
}
if fleet != nil {
@@ -226,6 +251,13 @@ func planServe(l *Lang, st mirror.Status, files []mirror.Status, base string,
}
f.CPUCores = catalog.SuggestCores(cpus)
}
// How fast it would answer: on the CPU, and on a card for comparison.
if f.Engine == "llamacpp" {
f.ReadBytes = int64(float64(weights) * header.ActiveShare())
f.CPUHosts = cpuHosts(fleet, f.MinMemoryGB)
f.CPUSpeed, f.CPUSpeedNode = bestCPUSpeed(f.CPUHosts, f.ReadBytes, f.CPUCores)
f.GPUSpeed, f.GPUSpeedClass = bestGPUSpeed(fleet, weights, header.ActiveShare())
}
if biggest := biggestNode(fleet); biggest > 0 && f.MinMemoryGB > biggest {
big := l.T("serve.warn.too_big", biggest, f.MinMemoryGB)
if f.Warning != "" {
@@ -523,6 +555,46 @@ func modeOf(sc catalog.Scaling) string {
// --parallel divides it among slots; it does not multiply it.
const defaultCtxPool = 16384
// mostContext is the longest context the serve form starts with: what the
// model was trained for, as long as its KV cache fits where the form places
// it beside the weights, else the most that does; and as many slots of it,
// up to defaultParallel, as the room left holds. 0 when the header does not
// say what it was trained for, or the weights alone do not fit.
//
// The longest, because a context is the one thing about a served model a
// client runs into without being told: a 4096 default on a model trained for
// 40960 cut documents short that the model could have read whole.
func mostContext(h gguf.Header, weights int64, places []placeChoice, gpus int, fleet Source) (tokens, slots int) {
trained := int(h.ContextLength) //nolint:gosec // a header's context length
if trained <= 0 {
return 0, 0
}
pool := 0
for _, pc := range places {
if !pc.Selected {
continue
}
if class, ok := strings.CutPrefix(pc.Value, "gpu:"); ok {
pool = h.PoolThatFits(weights, gguf.ClassMemGB(class), gpus)
} else {
pool = h.PoolThatFits(weights, biggestNode(fleet), 1)
}
}
switch {
case pool < 0:
return 0, 0
case pool == 0:
// Nothing to measure it against: what it was trained for, as one slot
// would hold it; the checks on submitting still see the card.
return trained, 1
}
tokens = min(trained, pool/1024*1024)
if tokens < 1024 {
return 0, 0
}
return tokens, max(1, min(defaultParallel, pool/tokens))
}
// defaultParallel is the requests a llama.cpp replica serves at once when
// the form is not changed: the agent's own default.
const defaultParallel = 4
@@ -675,3 +747,63 @@ func (p *AccountPages) asJudge(m *catalog.Model) (judge bool, renamed string) {
}
return true, renamed
}
// speedHost is a node a CPU replica could run on, as its speed is reckoned:
// its memory's rate with every CPU and with one, and its CPUs.
type speedHost struct {
ID string `json:"id"`
All float64 `json:"all"`
Core float64 `json:"core"`
CPUs int `json:"cpus"`
}
// cpuHosts are the ready nodes whose RAM holds memGB and whose agents
// measured their memory.
func cpuHosts(fleet Source, memGB int) []speedHost {
if fleet == nil {
return nil
}
var out []speedHost
for _, n := range fleet.Nodes() {
if n.State != "ready" || n.MemBandwidthGBs <= 0 || (n.MemoryGB > 0 && n.MemoryGB < memGB) {
continue
}
out = append(out, speedHost{ID: n.ID, All: n.MemBandwidthGBs, Core: n.MemBandwidthCoreGBs, CPUs: n.CPUCount})
}
return out
}
// bestCPUSpeed is about how many tokens a second one reply gets on the
// fastest of hosts, reading read bytes a token on cores threads (zero takes
// what the agent gives: half the node's), and that node.
func bestCPUSpeed(hosts []speedHost, read int64, cores int) (float64, string) {
best, node := 0.0, ""
for _, h := range hosts {
if tps := modelplan.CPUTokensPerSecondOn(read, h.All, h.Core, modelplan.CPUThreads(cores, h.CPUs)); tps > best {
best, node = tps, h.ID
}
}
return best, node
}
// bestGPUSpeed is the same for one card of the fastest class the fleet has
// whose memory holds the weights, by its bandwidth, and that class.
func bestGPUSpeed(fleet Source, weights int64, share float64) (float64, string) {
if fleet == nil || weights <= 0 {
return 0, ""
}
best, class := 0.0, ""
for _, n := range fleet.Nodes() {
for _, g := range n.GPUs {
bw := modelplan.GPUBandwidthGBs(g.Name)
if bw <= 0 || g.MemoryTotalMB > 0 && int64(g.MemoryTotalMB)<<20 < weights+weights/10 {
continue
}
o := modelplan.Option{Engine: "llamacpp", Class: g.Class, GPUsPerNode: 1}
if tps := modelplan.GPUTokensPerSecond(o, weights, share, 0, bw, modelplan.Links{}); tps > best {
best, class = tps, g.Class
}
}
}
return best, class
}
+108
View File
@@ -12,6 +12,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
)
@@ -27,6 +28,22 @@ type fakeCatalog struct {
owners []string
resources []string
adapterSets [][]catalog.Adapter
// stops, starts and splits are the models stopped, started and counted
// per profile, splitCPU the profile on CPUs last asked for.
stops, starts []string
splits []map[string]int
splitCPU *modelsvc.CPUProfile
}
func (f *fakeCatalog) SetProfileReplicas(_ context.Context, name string, counts map[string]int, cpu *modelsvc.CPUProfile,
_ string,
) (catalog.Model, error) {
f.splits, f.splitCPU = append(f.splits, counts), cpu
m := catalog.Model{Name: name}
for p, n := range counts {
m.Profiles = append(m.Profiles, catalog.Profile{Name: p, Replicas: n})
}
return m, f.err
}
func (f *fakeCatalog) Add(_ context.Context, m catalog.Model, _ string) error {
@@ -549,10 +566,12 @@ func TestTheCPUWithGPUsFreeIsAChoiceMadeOnPurpose(t *testing.T) {
}
func (f *fakeCatalog) Stop(_ context.Context, name, _ string) (catalog.Model, error) {
f.stops = append(f.stops, name)
return catalog.Model{Name: name, Stopped: &catalog.Scaling{Min: 1, Max: 1}}, f.err
}
func (f *fakeCatalog) Start(_ context.Context, name, _ string) (catalog.Model, error) {
f.starts = append(f.starts, name)
return catalog.Model{Name: name, Scaling: catalog.Scaling{Min: 1, Max: 1}}, f.err
}
@@ -590,3 +609,92 @@ func TestAStrataFamilysVersionIsServedOnStrata(t *testing.T) {
t.Error("an unknown size was taken")
}
}
// speedFleet is two nodes with GPUs whose memory rates were measured: a
// large server and a small one, as on a fleet where every node has cards.
type speedFleet struct{ fakeFleet }
func (speedFleet) Nodes() []adminapi.NodeView {
return []adminapi.NodeView{
{ID: "big", State: "ready", MemoryGB: 512, CPUCount: 64, MemBandwidthGBs: 300, MemBandwidthCoreGBs: 10,
Engines: []string{"llamacpp"}},
{ID: "small", State: "ready", MemoryGB: 64, CPUCount: 8, MemBandwidthGBs: 40, MemBandwidthCoreGBs: 10,
Engines: []string{"llamacpp"}},
}
}
// The speed on the CPU follows the threads a replica is given: four of a
// large server read at four cores' rate, not the server's.
func TestTheCPUsSpeedFollowsTheCoresGiven(t *testing.T) {
hosts := cpuHosts(speedFleet{}, 32)
if len(hosts) != 2 {
t.Fatalf("hosts %+v", hosts)
}
few, _ := bestCPUSpeed(hosts, 18e9, 4)
half, node := bestCPUSpeed(hosts, 18e9, 0)
if few > 2.5 || half < 8 || node != "big" {
t.Errorf("four cores %.1f tok/s, half the big node's %.1f on %s", few, half, node)
}
if hosts := cpuHosts(speedFleet{}, 100); len(hosts) != 1 || hosts[0].ID != "big" {
t.Errorf("a model of 100 GB fits the big node only: %+v", hosts)
}
}
// Slow enough that a reply is minutes, the CPU is a choice made on purpose,
// GPUs free or not.
func TestAVerySlowModelOnTheCPUIsAChoiceMadeOnPurpose(t *testing.T) {
big := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Length", "18000000000")
w.WriteHeader(http.StatusOK)
}))
t.Cleanup(big.Close)
values := func(extra ...string) url.Values {
v := url.Values{
"csrf": {"tok"}, "name": {"big-dense"}, "source": {big.URL + "/m/v1/m.gguf"}, "engine": {"llamacpp"},
"replicas": {"1"}, "context_length": {"4096"}, "placement": {"cpu"}, "min_memory_gb": {"24"},
"cpu_cores": {"4"},
}
for i := 0; i+1 < len(extra); i += 2 {
v.Set(extra[i], extra[i+1])
}
return v
}
cat := &fakeCatalog{}
page := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{}, Catalog: cat, Fleet: speedFleet{}}, true)
body := mirrorPost(t, page, "/admin/mirror/serve", values()).Body.String()
if len(cat.added) != 0 || !strings.Contains(body, "tokens a second on the CPU") {
t.Fatalf("an 18 GB model on four cores went through unasked: %s", body)
}
mirrorPost(t, page, "/admin/mirror/serve", values("cpu_ok", "yes"))
if len(cat.added) != 1 {
t.Fatal("ticking the box should add it")
}
}
func TestTheServeFormStartsAtTheLongestContextThatFits(t *testing.T) {
// An 8B's shape: 36 layers, 8 KV heads of 128, so 144 KiB a token.
h := gguf.Header{BlockCount: 36, HeadCount: 32, HeadCountKV: 8, KeyLength: 128, ValueLength: 128,
ContextLength: 40960}
card := []placeChoice{{Value: "gpu:6000-24g", Selected: true}, {Value: "cpu"}}
cpu := []placeChoice{{Value: "gpu:6000-24g"}, {Value: "cpu", Selected: true}}
for _, c := range []struct {
name string
weights int64
places []placeChoice
header gguf.Header
tokens, slots int
}{
// All it was trained for, and two slots of it in the room left.
{"on a card with room", 5 << 30, card, h, 40960, 2},
// The weights take most of the card: the most that fits, one slot.
{"on a card nearly full", 18 << 30, card, h, 18432, 1},
{"on the CPU of a 64 GB node", 5 << 30, cpu, h, 40960, 4},
{"weights larger than the card", 30 << 30, card, h, 0, 0},
{"a header that does not say", 5 << 30, card, gguf.Header{BlockCount: 36, HeadCountKV: 8, HeadCount: 32, KeyLength: 128}, 0, 0},
} {
tokens, slots := mostContext(c.header, c.weights, c.places, 1, fakeFleet{memoryGB: 64})
if tokens != c.tokens || slots != c.slots {
t.Errorf("%s: %d tokens × %d slots, want %d × %d", c.name, tokens, slots, c.tokens, c.slots)
}
}
}
+24 -10
View File
@@ -4,6 +4,7 @@ import (
"context"
"errors"
"net/http"
"slices"
"sort"
"strings"
"time"
@@ -92,7 +93,10 @@ type adminUsersData struct {
// Open is the project whose section is shown expanded: the one just acted
// on, so the result of a change is in view without the reader hunting for
// it. Scripted pages keep every open section open; this is for the rest.
Open string
Open string
// Edit is whether this person may change what the page shows: a viewer
// is drawn the page without its forms.
Edit bool
AdminID string
Message string
Error string
@@ -173,23 +177,19 @@ func (p *AccountPages) RegisterAdminUsers(mux *http.ServeMux, s AccountAdmin) {
return say("users.msg.added_to", scopeLabel(ctx, s, store.Scope(scope), scopeID)), nil
}))
// Managing is a role on an organization membership: it lets the person see
// that organization's page, and nothing else changes.
// A role on an organization membership: a manager sees that organization's
// page, an admin also changes it; a member neither.
mux.HandleFunc("POST /admin/users/{id}/role", p.adminAction(s,
func(ctx context.Context, admin store.User, id string, r *http.Request) (notice, error) {
scope, scopeID, ok := strings.Cut(r.FormValue("scope"), ":")
role := r.FormValue("role")
if !ok || store.Scope(scope) != store.ScopeOrg || (role != store.RoleManager && role != store.RoleMember) {
if !ok || store.Scope(scope) != store.ScopeOrg || !slices.Contains(store.MembershipRoles, role) {
return notice{}, refuse("users.err.manager_role")
}
if err := s.AddMembership(ctx, id, store.ScopeOrg, scopeID, role, admin.Email, time.Now()); err != nil {
return notice{}, err
}
label := scopeLabel(ctx, s, store.ScopeOrg, scopeID)
if role == store.RoleManager {
return say("users.msg.now_manages", label), nil
}
return say("users.msg.no_longer_manages", label), nil
return roleNotice(role, scopeLabel(ctx, s, store.ScopeOrg, scopeID)), nil
}))
mux.HandleFunc("POST /admin/users/{id}/leave", p.adminAction(s,
@@ -351,6 +351,17 @@ func (p *AccountPages) RegisterAdminUsers(mux *http.ServeMux, s AccountAdmin) {
}))
}
// roleNotice says what a person may now do in an organization.
func roleNotice(role, org string) notice {
switch role {
case store.RoleManager:
return say("users.msg.now_manages", org)
case store.RoleOrgAdmin:
return say("users.msg.now_administers", org)
}
return say("users.msg.no_longer_manages", org)
}
// notice is what the page says about a change, kept as a catalogue key and
// its arguments rather than as text: the log and the audit record take it in
// English, which is what they are searched in, and the page in the language
@@ -394,7 +405,7 @@ func init() {
registerKeys(
"users.msg.approved_alone", "users.msg.approved_into", "users.msg.suspended",
"users.msg.reinstated", "users.msg.added_to", "users.msg.now_manages",
"users.msg.no_longer_manages", "users.msg.removed_from", "users.msg.removed_from_one_key",
"users.msg.no_longer_manages", "users.msg.now_administers", "users.msg.removed_from", "users.msg.removed_from_one_key",
"users.msg.removed_from_keys", "users.msg.removed_keys_revoked", "users.msg.org_created",
"users.msg.team_created", "users.msg.project_created", "users.msg.renamed",
"users.msg.team_moved", "users.msg.project_moved", "users.msg.removed",
@@ -408,6 +419,8 @@ func init() {
// state, which is also the row's class.
"users.state.active", "users.state.pending", "users.state.suspended",
"users.key_state.active", "users.key_state.revoked", "users.key_state.expired",
// A membership's role, looked up by the role.
"users.role.member", "users.role.manager", "users.role.admin",
)
}
@@ -573,6 +586,7 @@ func (p *AccountPages) buildUsers(r *http.Request, s AccountAdmin, admin store.U
Members: map[string][]userRow{},
Keys: map[string][]keyRow{},
}
d.Edit = d.Nav.May("/admin/users")
var err error
// logErr logs what could not be read in English and says so on the page,
// once, in the reader's language.
+18 -10
View File
@@ -43,9 +43,17 @@ type clusterFit struct {
// cpuHost is a node as a model on its CPU sees it.
type cpuHost struct {
id string
ramGB int
bwGBs float64
id string
ramGB int
cpus int
bwGBs float64
coreBW float64 // one core's rate, 0 from an older agent
}
// tps is about how many tokens a second one reply of a model whose tokens
// each read bytes gets here, with the threads a replica is given by default.
func (h cpuHost) tps(bytes int64) float64 {
return modelplan.CPUTokensPerSecondOn(bytes, h.bwGBs, h.coreBW, modelplan.CPUThreads(0, h.cpus))
}
// fp8Capability is the oldest card vLLM runs an FP8 checkpoint on: Ampere,
@@ -100,7 +108,8 @@ func (p *AccountPages) fitOf() *clusterFit {
}
}
if n.State == "ready" && n.MemBandwidthGBs > 0 {
c.cpus = append(c.cpus, cpuHost{id: n.ID, ramGB: n.MemoryGB, bwGBs: n.MemBandwidthGBs})
c.cpus = append(c.cpus, cpuHost{id: n.ID, ramGB: n.MemoryGB, cpus: n.CPUCount,
bwGBs: n.MemBandwidthGBs, coreBW: n.MemBandwidthCoreGBs})
}
for _, s := range n.ComputeCapabilities {
if v, err := strconv.ParseFloat(s, 64); err == nil && v > newest[n.ID] {
@@ -268,17 +277,16 @@ func (c *clusterFit) holds(l *Lang, name string, bytes, params int64) (gguf.Verd
// bytes on the fastest node whose RAM holds it, and that node; share is the
// part a token reads (ActiveShare). 0 when no measured node holds it.
func (c *clusterFit) cpuSpeed(bytes int64, share float64) (float64, string) {
read := int64(float64(bytes) * share)
var best cpuHost
bestTPS := 0.0
for _, n := range c.cpus {
need := float64(bytes) + float64(1<<30) + float64(bytes)/20
if need <= float64(n.ramGB)*0.85*(1<<30) && n.bwGBs > best.bwGBs {
best = n
if tps := n.tps(read); need <= float64(n.ramGB)*0.85*(1<<30) && tps > bestTPS {
best, bestTPS = n, tps
}
}
if best.id == "" {
return 0, ""
}
return modelplan.CPUTokensPerSecond(int64(float64(bytes)*share), best.bwGBs), best.id
return bestTPS, best.id
}
// speedLabel is a rate as the page shows it: whole tokens from ten up, one
+3 -3
View File
@@ -134,13 +134,13 @@ func TestTheUsageReportWritesTurkishNumbers(t *testing.T) {
func TestTheOrganizationPageSpeaksTurkishWhenAsked(t *testing.T) {
body := inTurkish(t, orgPage(t, store.User{ID: "u1", Email: "boss@acme"}), "/org/org-a", nil)
for _, want := range []string{"2 kişi", "Kullanım", "Ekipler, projeler ve sınırlar", "Üyeler",
">yönetici</b>", "kuruluşun tamamı", "son 7 gün", `lang="tr"`} {
for _, want := range []string{"2 kişi", "Kullanım", "Sınırlar", "Üyeler",
">sorumlu</b>", "kuruluşun tamamı", "son 7 gün", `lang="tr"`} {
if !strings.Contains(body, want) {
t.Errorf("the organization page in Turkish lacks %q", want)
}
}
if strings.Contains(body, "Teams, projects and limits") {
if strings.Contains(body, ">Limits<") {
t.Error("the organization page in Turkish still has English headings")
}
}
Loaded 100 of 129 files, more files were not shown because too many files have changed in this diff. Show more