Compare commits

..
53 Commits
Author SHA1 Message Date
Ilker ManapandClaude Fable 5.1 32d4d9c3af The admin API document regenerated for Slurm's jobs and the training page
CI / test (push) Failing after 8m24s
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-19 12:41:57 +01:00
Ilker ManapandClaude Fable 5.1 81f6ab3596 Go 1.27, no patch level pinned
CI / test (push) Failing after 10m54s
Release / build (push) Successful in 6m45s
Release / notes (push) Successful in 7s
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-19 12:38:07 +01:00
Ilker ManapandClaude Fable 5.1 bd9860a98d v0.16.4: changelog and README
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-19 12:33:42 +01:00
Ilker ManapandClaude Fable 5.1 5cc719fcde Go 1.27.4
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-19 12:33:42 +01:00
Ilker ManapandClaude Fable 5.1 c2fedeb326 An NVIDIA card with no driver is reported, not onboarded as a CPU node
CI / test (push) Failing after 9m12s
The agent found GPUs through nvidia-smi alone, so a machine whose driver
was never installed looked exactly like one with no card and joined
quietly as a CPU node. Only init looked at the PCI bus.

The host discovery now reads the bus too (vendor 0x10de, class 0x03) and
carries it to the control plane. The doctor fails the pre-install checks
on a card without a driver, with the install and the reboot named, and
tells a driver that is installed but not loaded (Secure Boot, nouveau)
apart from a missing one. A node already in shows the warning in ctl
nodes and the console. init uses the same check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-19 12:29:02 +01:00
Ilker ManapandClaude Opus 5.5 a53b2f4be9 Customise a model from the portal: a file, a model, a button
A page for anyone in a project, built for the one who knows least: an
Excel, CSV or JSONL file of questions and answers, previewed; a model; a
name; who may use it; and a button. The weights to train on are found
from where the model's came (a GGUF repository's base model), the card's
memory from their size, the node as the one with the most free. When it
is done the adapter is published by itself (a llama.cpp model's replicas
replaced to load it) and is a model of its own in the chat and the API.

The project's by default; its members share it wider (confirmed, and
never wider than the knowledge bases its pairs came from) or delete it.
One trains at a time per project. ctl train start needs only --model and
--data now, and --publish publishes when done.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 11:31:12 +01:00
Ilker ManapandClaude Opus 5.5 fcda4d3f85 Slurm's jobs in llmgw's accounting: projects as accounts, jobs priced
CI / test (push) Failing after 7m24s
With slurm.users llmgw, each project is a Slurm account, its organization's
and team's above it, with the project's members and its team's and
organization's on it; slurm.conf holds a job to its user's accounts. The
controller's agent keeps them with sacctmgr, only what it made, and reports
the jobs that ended in the last day; the leader records each once, its
GPU-hours (core-hours without cards) priced at auth.gpu_hour_cost into its
project's usage as model slurm/PARTITION.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 11:54:01 +01:00
Ilker ManapandClaude Opus 5.5 8547528fb7 Prices from what the cluster costs: ctl prices suggest
CI / test (push) Successful in 9m51s
auth.gpu_hour_cost (and core_hour_cost) is what an hour of a GPU (a core)
costs the operator. The leader keeps, by the hour, the GPU- and core-hours
each model's replicas hold (held_1h); llmgw ctl prices suggest spreads a
model's over the tokens it served, an output token counted as four input
ones and a cached prompt token as a tenth of one, and --apply writes the
prices into auth.pricing. Idle hours are in the cost, so a model held and
little used comes out dear; one held and billed nothing is shown with what
it held.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 06:54:55 +01:00
Ilker ManapandClaude Opus 5.5 b8d3c084c5 BACKLOG P: AD sign-in to Slurm user, tried end to end
CI / test (push) Failing after 7m31s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 23:00:28 +01:00
Ilker ManapandClaude Opus 5.5 57f0fd2ed0 Slurm's jobs in containers: pyxis and enroot, installed by the agents
CI / test (push) Failing after 6m53s
slurm.containers (ctl slurm containers on) has every Slurm machine's agent
install enroot from NVIDIA's release and build pyxis against its own
Slurm, both pinned and checked against their digests, and on GPU machines
nvidia-container-cli from NVIDIA's repository. enroot keeps each user's
containers under /run/enroot and /var/lib/enroot, needing no login
session, with the shared path in every container. The leader has Slurm
load pyxis only once every machine reports it: plugstack.conf's
"required" fails every job on a machine without the plugin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:31:20 +01:00
Ilker ManapandClaude Opus 5.5 c3a75dcf3f A node being lent says when a replica with nowhere to go is stopped
CI / test (push) Successful in 9m37s
Tried on the PoC: lending cuda01, the one GPU node, its model had nowhere
else to go and waited out the drain's five-minute grace, while ctl slurm
status said only "1 replica(s) still to move off". It now says the time
that replica is stopped at. Also recorded: Slurm on a GPU node, tried on
cuda01 (gres, ConstrainDevices, accounting of gres/gpu).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:45:57 +01:00
Ilker ManapandClaude Opus 5.5 4b4306bb6a Slurm users reach a node only through a job of theirs
CI / test (push) Successful in 9m41s
llmgw's Slurm users log in to the controller freely and to another Slurm
machine only while a job of theirs runs there: the agent installs
pam_slurm_adopt and puts it in sshd's account stack for the users it made
alone, so root and the machine's own users are never locked out. Their
session is put in the job's extern step (PrologFlags=Contain), and
pam_systemd, which would move it out, is skipped for them by taking the
session file into sshd's stack with a guard in front of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:30:49 +01:00
Ilker ManapandClaude Opus 5.5 a00c92648e Status says when the cluster's alerts go nowhere
CI / test (push) Failing after 6m57s
Setup asks for webhooks, but one left empty was never mentioned again: a
lost node or a judge down was told nobody. With no notify.webhooks in the
cluster's settings, ctl status and the console's Overview say so, and how
to set one. BACKLOG: the judge's refusal after a placement change, tried
again on idle VMs, did not happen; a cold judge's first answer is 2-3x
slower but inside its budget.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:49:37 +01:00
Ilker ManapandClaude Opus 5.5 2c32a58a06 Suspending a person says their keys are refused, which they are
CI / test (push) Successful in 9m41s
The message said their API keys still worked and should be revoked; the
store has refused a key whose owner is not active for a while. Tried on
the test cluster: refused 26 s after the suspend (auth.cache_ttl, 30 s),
answered again 6 s after reinstating.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:38:54 +01:00
Ilker ManapandClaude Opus 5.5 d8ce573bdb Slurm's shared files and users: one path everywhere, accounts as Linux users
CI / test (push) Failing after 7m1s
slurm.shared mounts the site's NFS export, has an llmgw node export the path
over NFS to the Slurm machines alone (2049 in the firewall), or looks for a
file system the site mounts. slurm.users "llmgw" makes each active account a
Linux user on every Slurm machine: login from its e-mail, a UID from 200000
kept in slurm_users and never reused, its home on the shared path, its SSH
keys from the portal's Profile (or ctl slurm users keys) read by sshd from
/etc/llmgw/ssh-keys. server init --slurm chooses /shared from the first
machine and llmgw's users; --slurm-shared and --slurm-users change that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:34:34 +01:00
Ilker ManapandClaude Fable 5.1 b9d20e7bba Licence: FSL-1.1 with an Apache 2.0 future licence
CI / test (push) Successful in 9m48s
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-12 19:55:24 +01:00
Ilker ManapandClaude Opus 5.5 0d78f9bb34 A machine given by address is reached through its ssh config Host
node add (and server add, mirror peer, patroni init) is usually given an
address, while the ssh config names the machine: `Host gpu01` with
`HostName 10.0.0.21` and the User, IdentityFile or ProxyJump it needs. That
block did not match the address, so the address had to be added to its
Host line by hand. Now the concrete Host names in the config and its
Includes are read, ssh -G says each one's HostName, and the first that is
the address (or resolves to it) is connected through.

Under sudo, the caller's name is now a fallback: a User in the config's
block for the machine comes first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:42:23 +01:00
Ilker ManapandClaude Opus 5.5 5ac6bef463 BACKLOG P: the fresh install tried, and what it left open
CI / test (push) Successful in 10m50s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:42:22 +01:00
Ilker ManapandClaude Opus 5.5 557a5a1ef3 Found on a fresh three-machine install: what the screens said wrong
CI / test (push) Successful in 11m5s
- server init printed every engine's line again on each look while the
  node installed (some 280 lines); each engine's line now once, and again
  only when it changes, as node add does.
- A package install that is no engine's (Slurm's, MariaDB's) read
  ": installing installing mariadb-server, ..."; it is "system
  packages: installing: mariadb-server, ..., with apt-get".
- The slow-judge advice says "ctl model resources JUDGE --cores 4",
  which was refused for want of --memory: cores alone now keep the
  memory as it is.
- server add said both control planes for three.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:38:18 +01:00
Ilker ManapandClaude Opus 5.5 21828fefce BACKLOG P: slurmdbd done; P6, Slurm's jobs in llmgw's accounting
CI / test (push) Failing after 7m52s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:21:40 +01:00
Ilker ManapandClaude Opus 5.5 748f777a56 Slurm's accounting: slurmdbd and a MariaDB on the controller
CI / test (push) Successful in 10m57s
Since 23.11 Slurm runs badly without accounting storage: the main
scheduler passes every job by as InvalidAccount. slurmdbd takes MySQL or
MariaDB only, so the controller's agent installs MariaDB beside it,
answering that machine alone, makes the database and its user (the
password made from the munge key, never on a command line), runs
slurmdbd and adds the cluster, all before slurmctld. A MariaDB that was
on the machine before keeps its settings. slurm.conf accounts the jobs
there, with what each used from its cgroup and the cards, and the
priority is multifactor again. Port 6819 joins Slurm's in the firewall.
ctl slurm status and the console say whether slurmdbd runs; the install
screen, server init --slurm and the guide say what is put where.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:59:08 +01:00
Ilker ManapandClaude Opus 5.5 02774769a7 Slurm: why priority/basic, said rightly
CI / test (push) Failing after 8m3s
priority/basic did not end InvalidAccount: 23.11 dropped
accounting_storage/none, and without slurmdbd the main scheduler passes
jobs by whatever the priority plugin. Basic stays, fair share needing
slurmdbd anyway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:21:24 +01:00
Ilker ManapandClaude Opus 5.5 bab0766ee1 Slurm: jobs start on submission, and a lent node stays open across restarts
CI / test (push) Failing after 8m35s
Without accounting storage, 23.11's default priority/multifactor has the
main scheduler pass every job by as InvalidAccount; backfill started it a
pass later. slurm.conf sets priority/basic.

Every node was written State=DRAIN, so each start of slurmctld on new
files drained a lent node again until its agent resumed it. A lent node
is written without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:18:22 +01:00
Ilker ManapandClaude Opus 5.5 5db920bf2c Slurm: a job that asks for no memory gets a share, not the whole node
CI / test (push) Successful in 10m34s
Tried on the PoC: five one-second jobs on an 8-CPU node started 10 s
apart. Each asked for no memory, so each was given the node's all, ran
alone, and the next waited for backfill's pass. Not accounting, as
thought. The partition's DefMemPerCPU is now the leanest node's memory
per CPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:11:24 +01:00
Ilker ManapandClaude Opus 5.5 356f7e0ced Slurm in the console: each node, and lend or take back from its row
CI / test (push) Successful in 9m39s
The console's Slurm section shows the controller, and each node that runs
slurmd with whose it is now, its state in Slurm and its jobs. Enter on a
row lends the node (its replicas move off first) or takes it back, by the
same change to slurm.lent as ctl slurm lend and reclaim.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:53:26 +01:00
Ilker ManapandClaude Opus 5.5 17287ee8b0 BACKLOG P: failover while lent tried, and fixed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:53:26 +01:00
Ilker ManapandClaude Opus 5.5 0774e901c0 Slurm: a new leader with nothing kept waits for the nodes a while
CI / test (push) Failing after 7m57s
On a cluster upgraded to the kept nodes, the first leader finds none and
wrote slurm.conf without the nodes it had not heard from yet, as before.
A node neither reported nor kept now holds the controller's files back
for slurmsvc.HoldFor (2 minutes) from the leader's first pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:37:52 +01:00
Ilker ManapandClaude Opus 5.5 141a1f46c5 Slurm: a new leader keeps the nodes it has not heard from yet
CI / test (push) Successful in 10m50s
Tried on the PoC: with a job on a lent node, the leader restarted and its
first slurm.conf had no node in it (none had reported to it yet), so
slurmctld dropped the node and the job ended in NODE_FAIL.

The leader keeps slurm.conf's nodes and the controller's address in the
store's cluster state (slurm-nodes); the next one starts from them and
writes the same files until the nodes report again. With the store
unread, the controller gets no files that leave a node out. A node is
told nothing, rather than that it has no part, until its controller's
address is known: no part stops slurmd, and its jobs with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:33:53 +01:00
Ilker ManapandClaude Opus 5.5 3387f97f6a Slurm's ports for the cluster only, and srun in a range of its own
CI / test (push) Successful in 9m40s
slurm.conf sets SrunPortRange=60001-63000, so a site's firewall can let
srun's step connections in. A node given a part in Slurm adds 6817-6818
and that range to llmgw's nftables table, open to the cluster's networks
and closed to everyone else, as the engines' ports are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:10:37 +01:00
Ilker ManapandClaude Opus 5.5 914985bf9f Slurm in the first install: server init --slurm, and slurm.all_nodes
CI / test (push) Successful in 9m30s
The choices screen has a Slurm part, and --slurm writes slurmctld on this
machine's node and slurmd on every node, those that join later too, into
server.yaml. slurm.all_nodes (ctl slurm setup --all-nodes) takes the nodes
from the leader's inventory. No node is lent at install.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 14:50:35 +01:00
Ilker ManapandClaude Opus 5.5 b15944aa1f A package install that is no engine's clears its progress; slurmd retried
CI / test (push) Successful in 10m15s
Found on the PoC: InstallPackages reported 'installing munge, slurm-client,
slurmd' under no engine and never cleared it, so the node's page kept
saying so (infiniband-diags and perftest did the same). slurmd started
before the leader has written its node into slurm.conf is refused; the
drop-in now has systemd try it again every 15 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:23:53 +01:00
Ilker ManapandClaude Opus 5.5 dcf4e256e6 BACKLOG P: tried on the PoC
CI / test (push) Failing after 7m26s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:20:32 +01:00
Ilker ManapandClaude Opus 5.5 849760c67d Slurm: no second restart; backfill every 10 s, MailProg set
The InvalidAccount seen on the PoC was not the reconfigure: 23.11.4 without
accounting storage has its main scheduler mark a batch job so, and backfill
starts it on its next pass (job 4 waited 15 s). bf_interval=10 keeps that
short; the extra restart is taken out again. MailProg=/bin/true quiets the
warning on every start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:18:35 +01:00
Ilker ManapandClaude Opus 5.5 087552136f Slurm: slurmctld started again after a reconfigure; no accounting gathered
Found on the PoC: Slurm 23.11.4 without accounting storage left batch jobs
pending InvalidAccount after scontrol reconfigure until slurmctld was
restarted. And jobacct_gather with nowhere to keep it logs an error per job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:16:31 +01:00
Ilker ManapandClaude Opus 5.5 d2c27e2c76 Slurm: slurmd started as its unit expects, and slurmd's own stepd not a job
Found on the PoC (Ubuntu 24.04, Slurm 23.11.4): the packaged unit is
Type=notify, so slurmd needs --systemd or systemd times it out; and slurmd
keeps a 'slurmstepd infinity' for its cgroup v2 scope, which was counted as
a job and kept the node from llmgw.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:11:41 +01:00
Ilker ManapandClaude Opus 5.5 8a5a04ed77 Slurm: the version from the daemon, not scontrol
Found on the PoC: scontrol looks for its files even for --version, and a
configless node has none before slurmd has fetched them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:06:42 +01:00
Ilker ManapandClaude Opus 5.5 410fe545ce Slurm beside llmgw, P3 and P4: the leader hands nodes over; ctl slurm
CI / test (push) Successful in 9m40s
internal/control/slurmsvc tells each node what it is, writes slurm.conf
and gres.conf from the nodes' reports and the inventory, and moves each node
of slurm.nodes between llmgw and Slurm: lent, drained of replicas first, then
resumed in Slurm; taken back, drained in Slurm and cordoned in llmgw until
its last job step ends. Kept every tick from the settings and the agents'
reports, so a new leader carries on. A broken Slurm never keeps a node from
llmgw. GET /api/v1/slurm; ctl slurm status, setup, lend, reclaim, off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:33:14 +01:00
Ilker ManapandClaude Opus 5.5 35de47444c Slurm beside llmgw, P2: the agent keeps its node's part
A Slurm message says whether the node runs slurmctld on the leader's files,
slurmd configless, both or neither, and whether it is lent. The agent
installs munge and Slurm from the machine's sources (22.05 or newer, for
cgroup v2), writes the munge key, starts slurmd through a drop-in with
--conf-server, resumes a lent node that llmgw drained and drains one that is
not, never touching a drain a Slurm administrator made, and reports its
version, slurmd -C, state and jobs in SlurmStatus. Told it has no part, it
stops what it started. New nodes enter slurm.conf drained.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:25:41 +01:00
Ilker ManapandClaude Opus 5.5 d439d507d7 Slurm beside llmgw, P1: its settings and the files a controller runs on
slurm.{enabled,cluster_name,controller,nodes,lent,partition} in the cluster's
settings, checked; internal/slurm writes slurm.conf (configless), gres.conf
and cgroup.conf from the nodes' slurmd -C lines and GPUs, derives the munge
key from a cluster secret, and reads scontrol and version output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:20:31 +01:00
Ilker ManapandClaude Opus 5.5 89767c8be1 BACKLOG P: training beyond one card, in llmgw or Slurm beside it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:12:07 +01:00
Ilker ManapandClaude Opus 5.5 d864438272 A model that failed to start is placed elsewhere next
CI / test (push) Successful in 9m31s
The same inputs place a replica on the same node and card, so an on-demand
model that failed to load there was started there again after every
backoff until it was parked. The reconciler remembers where a model's
replicas failed and places the next away from them when anything else
fits (Scheduler.Away); with nowhere else, there again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 09:22:11 +01:00
Ilker ManapandClaude Opus 5.5 6320dd5b13 A new stream is told the node's files again
CI / test (push) Successful in 9m31s
Registration carries what the agent's startup scan found, and a heartbeat
sends the list only when it changed since last sent; so a file deleted
after the agent started came back on a control plane that restarted. The
fingerprint is now forgotten with the provisioning one on every stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:14:52 +01:00
Ilker ManapandClaude Opus 5.5 5732a5ebda The serve forms start at the longest context that fits
CI / test (push) Successful in 9m26s
The mirror page's form began every GGUF model at 4096 tokens, the
console's at 4096 or the plan's figure. Both now start at what the model
was trained for, cut to the most whose KV cache fits beside the weights
where the form places it; the mirror page's slots are as many of that, up
to four, as the room left holds. Plan carries the trained context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 14:38:41 +01:00
Ilker ManapandClaude Opus 5.5 561a536477 The weight sweep names a set as its fetch did
CI / test (push) Successful in 9m33s
A set is fetched into a directory named from its address without the
trailing slash; Orphans named it from the catalog's address with it, so
every set in the catalog was offered for deletion: 330 GB of six models on
the PoC. One function, cacheSource, now gives both the name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 14:01:27 +01:00
Ilker ManapandClaude Opus 5.5 45fe7daaa0 The admin API document regenerated for CollectResult's left counts
CI / test (push) Successful in 9m32s
cba4897 added fields without regenerating it, and
TestTheAdminAPIDocumentIsNotStale failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 11:38:08 +01:00
Ilker ManapandClaude Opus 5.5 102ca9556f An organization's admin, and forms only for those who may send them
C1. A membership of an organization may be admin as well as member and
manager. An admin sees /org as a manager does and changes the
organization from it: its teams and projects, who of the approved
accounts is in them, its members' roles, and the keys spent in its
projects. Every id a form names is checked against the organization;
limits, accounts, the organization's name and the cluster stay with the
cluster's staff, and an admin cannot take their own role away.

A group from OIDC, SCIM or a directory no longer rewrites a membership
it names as a plain member at every sign-in, which took a manager's role
away (EnsureMembership).

Pages draw a form only for someone who may send it: Nav.May asks the
staff rule staffMay keeps, and a page whose forms a person may not send
gets no form token. A viewer sees the people, limits and mirror pages
without forms; an operator the mirror's.

Forms are compact: labels close to their fields, fields as wide as what
goes in them, smaller buttons, and the short forms on one line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 11:38:06 +01:00
Ilker ManapandClaude Opus 5.5 2cb1afda55 The introduction's comparison on one slide
CI / test (push) Failing after 6m38s
Four slides of products and cards become one table: llmgw beside API
gateways (LiteLLM, Kong AI Gateway, Agent Router) and serving stacks
(llm-d, NVIDIA Dynamo, vLLM production-stack), six rows. The PDF is
18 slides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 17:06:49 +01:00
Ilker ManapandClaude Opus 5.5 b6478e24b7 An introduction for decision makers, linked from the website
CI / test (push) Failing after 6m43s
docs/llmgw-intro.pdf is a 21-slide presentation: what llmgw adds on
premise, screens from a running cluster, and how it compares with
LiteLLM, Kong AI Gateway, Agent Router, llm-d, NVIDIA Dynamo and vLLM
production-stack from their own documentation. The /about page and the
published site link to it in the repository, so the binary does not
carry it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 14:36:47 +01:00
Ilker ManapandClaude Opus 5.5 cba4897987 Deleting unused model files says what went, and sets go whole
CI / test (push) Failing after 6m44s
A sweep counted every file the nodes were asked to delete as freed as soon
as it asked, and a node refused a set's directory as not empty, so tens of
gigabytes were reported freed and nothing was. The agent removes a set's
directory whole (a link only as a link), and the control plane waits for
each node to list its files again, saying what it no longer lists and what
it kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:29:16 +01:00
Ilker ManapandClaude Opus 5.5 3bb52742a4 v0.16.3: changelog and README
Release / build (push) Successful in 5m11s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 10m10s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:43:40 +01:00
Ilker ManapandClaude Opus 5.5 1a3f385be4 Every setting of a served model from the mirror page, and its replicas per profile
CI / test (push) Successful in 10m51s
The row of a served model leads to its card, which now changes its context
and longest answer, its cards, its replicas on each profile with a profile
on CPUs to add and its speed beside, stops, starts and restarts it, and
removes it with its files or without. Stopped on the row stops the model
rather than setting its scaling to nothing. The console gains Replicas per
profile, and ctl models shows the counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:40:06 +01:00
Ilker ManapandClaude Opus 5.5 e23732fbbb Replicas counted per profile: seven on cards and four on CPUs under one name
A profile's replicas, set on any of a model's profiles, is how many of its
replicas run there; the scaling becomes the sum. The reconciler starts the
ones short of their profile before stopping the ones over, so a change of
split never runs fewer. ctl model replicas sets them, adding a profile on
CPUs to a GGUF model; the router already fills the higher-priority pool
first, which the CPU profile is put below.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:21:50 +01:00
Ilker ManapandClaude Opus 5.5 6403e61f4b How fast a model answers on the CPU, before it is served
CI / test (push) Successful in 9m31s
Each agent measures one core's memory rate beside the whole machine's; the
estimate takes a replica's threads at about one core's rate each, up to the
machine's, calibrated by llama-bench on the PoC. The serve form shows it
beside a card's, warns when slow, and asks for the CPU on purpose when a
reply would take minutes; ctl model plan, the console and serve jobs say
it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 08:52:15 +01:00
224 changed files with 17456 additions and 641 deletions

No files matched your search

+1
View File
@@ -11,3 +11,4 @@ coverage.out
/terraform-provider-llmgw/examples/terraform.tfstate* /terraform-provider-llmgw/examples/terraform.tfstate*
# What a development server run from the checkout writes beside it. # What a development server run from the checkout writes beside it.
/secrets/ /secrets/
linkedin/
+186 -6
View File
@@ -49,7 +49,8 @@ Face, done 2026-10-30), then M5 (engines without the internet), M3, M4, M6.
**Scale.** O: more than one control plane answering clients; O1 (measuring one router's ceiling) first. **Scale.** O: more than one control plane answering clients; O1 (measuring one router's ceiling) first.
**Governance.** **Governance.**
7. **C1**: a tenant-admin role, and forms hidden from viewers. 7. ~~**C1**~~: done 2026-11-08. An organization's admin (`--role admin` on an org membership) changes its teams,
projects, people and keys from /org; viewers are drawn no forms; forms made compact.
8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too. 8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
**When there is room.** **When there is room.**
@@ -435,9 +436,9 @@ portal is not done — the page shows; the console and ctl change.
API middleware and the /admin pages (with each change's result), `ctl API middleware and the /admin pages (with each change's result), `ctl
audit`, the console's Audit view; staff roles viewer / operator / audit`, the console's Audit view; staff roles viewer / operator /
billing / admin. Left: before/after values (the request's fields are billing / admin. Left: before/after values (the request's fields are
recorded, not the state they replaced); a tenant-admin role that changes recorded, not the state they replaced). Done 2026-11-08: an organization's
its own organization (managers still only view it); hiding the forms a admin who changes its own organization from /org (managers still only view
viewer cannot submit. **Audit log and roles (M).** Today "audit" is slog lines it), and the forms a viewer cannot submit no longer drawn. **Audit log and roles (M).** Today "audit" is slog lines
(`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag (`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag
plus an org "manager" who can view. Add `audit_events` (who, what, plus an org "manager" who can view. Add `audit_events` (who, what,
before/after, source) written by the `adminAction`/`tenancyAction` before/after, source) written by the `adminAction`/`tenancyAction`
@@ -462,8 +463,9 @@ portal is not done — the page shows; the console and ctl change.
5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node 5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node
lost, budget spent, update available; signed JSON or Slack; retries, lost, budget spent, update available; signed JSON or Slack; retries,
hourly de-duplication), `ctl notify test`, Alertmanager receiver hourly de-duplication), `ctl notify test`, Alertmanager receiver
templates. Left: `server init` and the console do not yet ask for a templates. `server init` and both setup screens ask for webhooks and test
receiver — the templates are filled in by hand. Events are claimed in each (d1e9186, 2026-10-14); with none set, `ctl status` and the console's
Overview say the alerts go nowhere (2026-11-12). Events are claimed in
the store before they are sent (2026-10-10), so two control planes send the store before they are sent (2026-10-10), so two control planes send
each once. each once.
**Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml` **Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
@@ -1982,6 +1984,184 @@ one control plane's router can. In order:
- O3. Only if O2 is not enough: stateless gateway machines (`llmgw server --role gateway`) in front of the leader's - O3. Only if O2 is not enough: stateless gateway machines (`llmgw server --role gateway`) in front of the leader's
control plane, as many as the traffic needs. control plane, as many as the traffic needs.
## P. Training beyond one card (written 2026-11-11) — decision open
F4 trains a LoRA on one card of a node the caller names (`ctl train start --node`), one job per node, no resume.
Larger work (full fine-tuning, a LoRA on a 70B, several GPUs or machines) needs more. Two ways:
**A. In llmgw.** Add, in order: a job that takes N whole GPUs on one node (torchrun, FSDP/DeepSpeed in the agent's
venv); placement by the control plane instead of `--node`, freeing GPUs from replicas by the scaling schedule or by
draining; checkpoints to shared storage and resume when an agent or node goes; datasets uploaded or read from
storage, not only KB pairs; then gang placement of N nodes with a rendezvous on the first, the NCCL environment from
the fabric discovery (IB/NVLink, `llmgw doctor`); a console screen with logs and loss. Size: L for one node, XL for
several. For: one product, one install, one set of users and grants (KB rules carry over), GPUs shared with
inference without a second scheduler. Against: it is Slurm's gang scheduling, requeue and accounting written again;
multi-node NCCL cannot be tried on the PoC (one GPU node, no IB); every customer's training code would have to fit
our job format.
**B. Slurm beside llmgw.** The TUI installs Slurm on the nodes the customer picks (munge, slurmctld on the control
planes, slurmd, users from the same LDAP/AD), with pyxis/enroot for containers. A node is handed between the two:
llmgw drains its replicas and `scontrol update state=resume`, and back. llmgw's part is what comes out: a checkpoint
or adapter from a Slurm job published to the mirror (`ctl train import`), served through canary and the eval. Size:
M. For: what research and university customers already run (`sbatch`, `srun torchrun`), any training code, mature
multi-node and requeue; we write only the install and the handover. Against: a second system with its own failures,
upgrades and users to keep in step; GPUs split by hand or by the handover, not by demand; access rules on training
data are Slurm's and the file system's, not the KB's.
Both need shared storage (NFS, BeeGFS, Lustre) the customer provides; neither changes F4.
Recommendation: keep F4 in llmgw for small single-card adapters tied to KB grants; take B for anything larger;
build only A's first step (N GPUs on one node, placed by the control plane, with resume) if customers ask for
mid-size jobs without wanting Slurm.
**B taken (2026-11-11).** Design: the cluster's settings say it (`slurm: enabled, cluster_name, controller, nodes,
lent, partition`); the leader tells each named node's agent (root) what to be, the agents install and run it.
- The controller is an llmgw node (its agent runs slurmctld); `nodes` run slurmd configless
(`--conf-server`), so slurm.conf, gres.conf and cgroup.conf live only on the controller.
- The munge key is derived from the cluster secret (HKDF), so nothing new is stored or copied by hand.
- Each node reports `slurmd -C`; the leader writes slurm.conf from those lines, its mgmt address and its GPUs
(`Gres=gpu:<type>:N`, gres.conf with the device files), and sends it to the controller again when one changes.
- A node in `nodes` belongs to llmgw until it is in `lent`: drained in Slurm (reason "llmgw: serving models").
Lent: llmgw drains it as `ctl drain` does (kept in force each tick, as facility holds are, so a new leader
keeps it) and, once no replica is left, its agent resumes it in Slurm. Taken out of `lent`: the agent drains
it in Slurm, reports the jobs left, and when none are the leader makes it ready for llmgw again.
- Phases: P1 the `slurm` package (conf, key, parsers) and settings; P2 the agent side (proto, install, munge,
slurmd/slurmctld, status); P3 the leader side (conf from reports, lend/reclaim); P4 `ctl slurm`
(status, lend, reclaim) and a TUI screen; P5 `ctl train import` of a Slurm job's output into the mirror.
Later: a backup slurmctld on a second node, accounting (slurmdbd; it takes MySQL/MariaDB only, not the Patroni PG), users from LDAP/AD.
- Tried on the PoC 2026-11-11 (dev build v0.16.3-17-g849760c-slurm on 140, 150, node05, node06; Ubuntu
24.04, Slurm 23.11.4 from the archive): `ctl slurm setup --controller llmgw-node06 --nodes llmgw-node05`
installed munge, slurmctld and slurmd, node05 entered drained with its slurmd -C hardware (8 CPUs,
31132 MiB after the 3%). `lend`: its 2 replicas moved off, then IDLE in Slurm; srun and sbatch ran there.
`reclaim` with a 90 s job running: MIXED+DRAIN, llmgw kept off, ready again within a minute of the job
ending. Found and fixed on the way: a client command needs its files even for --version (version now
from the daemon); the packaged unit is Type=notify (slurmd needs --systemd); slurmd's own "slurmstepd
infinity" was counted as a job; without accounting storage 23.11's main scheduler marks batch jobs
InvalidAccount and backfill starts them, one a pass (bf_interval=10: 6-10 s each). So many small jobs
want slurmdbd. — Wrong cause, found 2026-11-11: a job asking no memory got the whole node's (ReqTRES
mem=all), so a node ran one job at a time and each next one waited for backfill's pass. Fixed: the
partition's DefMemPerCPU is the leanest node's share; slurmdbd is not needed for this. InvalidAccount is real too:
23.11 dropped accounting_storage/none, and without slurmdbd the main scheduler passes every job by;
backfill starts it within bf_interval (10 s). priority/basic did not change that (fair share needs
slurmdbd anyway). Starting on submission needs slurmdbd, on MySQL/MariaDB.
- slurmdbd, done 2026-11-12 (748f777): the controller's agent installs MariaDB (bound to 127.0.0.1; a
MariaDB that was there keeps its settings), makes `slurm_acct_db` and its user (password from the munge
key, through a 0600 file, never argv), runs slurmdbd (6819, in the firewall for the cluster) and adds
the cluster, before slurmctld. slurm.conf: AccountingStorageType=slurmdbd, jobacct_gather/cgroup,
AccountingStorageTRES=gres/gpu, priority/multifactor. Tried on the PoC: five jobs started on
submission, `sacct -a` has them with their TRES. Not tried: a fresh `server init --slurm` (needs VMs);
a Red Hat controller.
- Fresh install tried 2026-11-12 on VMs (3 control planes with Patroni, 2 CPU nodes, `server init
--slurm`): Slurm, slurmdbd and MariaDB came up with the install; two-node srun/sbatch on the lent nodes,
`sacct` has them; leader failover kept serving. Open from it: `--slurm` needs the first machine to be a
node (the controller is its node), so a site wanting control planes without nodes chooses the
controller later with `ctl slurm setup`; batch output lands on the job's first node until there is
shared storage; on shared CPUs the judge refuses the first request after a placement change (fail
closed, ~10 s), and its advice repeats `--cores 4` to a judge that has 4. Tried again on the idle
VMs: one judge restarted in place, then both at once, requests every second: 48 of 48 served; the
first after a cold start took 7.6–8.9 s end to end against ~3 s, within the 5 s judge budget. So the
refusal was the install's load on the CPUs; if it is seen again, warm a new judge with one judgment
before it takes requests.
- Shared files and users, built 2026-11-12: `slurm.shared` {path, nfs | server} — the site's export
mounted (a systemd mount unit, hard, nconnect=4), an llmgw node exporting the path itself
(/etc/exports.d, to the Slurm machines' addresses, root_squash; 2049 in the firewall), or the site's
own mount only looked for; `slurm.users: llmgw` — every active account a Linux user everywhere
(`slurm_users`: login from the e-mail, UID from 200000, never reused; homes made on the exporting
machine, else the controller; keys in /etc/llmgw/ssh-keys/%u through an sshd drop-in), SSH keys
from the portal's Profile or `ctl slurm users keys`. A user already on a machine with another UID
is left alone and said. The site has AD (not yet live): users from it, through sssd, are the next
step (`slurm.users` empty today leaves them to the site).
- Logging in to a node, done 2026-11-14: llmgw's users reach the controller freely, another Slurm
machine only while a job of theirs runs there (pam_slurm_adopt, installed by the agent; PrologFlags=
Contain). Only the users listed in the agent's users file are held: root and the machine's own users
log in as before, so a wrong line never locks out who would fix it. pam_systemd would move the session
out of the job, so sshd's stack takes the session file in place of its include, with pam_systemd
skipped for the listed users alone; where that cannot be done safely (nested includes, jumps across
it) they are held to having a job, not to its cores and cards, and the agent says so. Tried on the
VMs: ayse without a job refused ("no active jobs on this node"), with one in
job_N/step_extern (nproc 1), cut off when it ended; mehmet refused; admin's session unchanged.
Left: sites whose users come from AD (sssd) are not held yet; a group of theirs would be the list.
- A GPU node, tried 2026-11-14 on the PoC (cuda01, Quadro RTX 6000, Slurm 23.11.4): gres.conf
`Type=rtx6000 File=/dev/nvidia0`, slurm.conf `Gres=gpu:rtx6000:1`. `srun --gres=gpu:1`:
CUDA_VISIBLE_DEVICES=0, the one card; torch found it and ran 14.1 TFLOPS (fp32 matmul). Without
--gres: "No devices found", torch none (ConstrainDevices holds). `--gres=gpu:2`: refused. sacct has
`gres/gpu=1` in AllocTRES. Found: lending a node whose GPU model has nowhere else to go waits out
the drain grace (5 min) before the replica stops, and `ctl slurm status` says only "1 replica(s)
still to move off"; it should say it waits for a replacement that cannot be placed, and until when.
Done (c3a75dc): it says the time.
- Jobs in containers, done 2026-11-14: `slurm.containers` (`ctl slurm containers on`). Each agent
installs enroot 4.2.1 (NVIDIA's .deb/.rpm, sha256 pinned) and builds pyxis 0.24.0 (source tarball,
sha256 pinned) against its own Slurm (gcc, make, libslurm-dev); on a machine with nvidia-smi,
nvidia-container-cli from NVIDIA's repository, its key's digest pinned. enroot.conf keeps each
user's runtime, cache and data under /run/enroot and /var/lib/enroot (no login session needed), the
shared path bind-mounted in every container. The leader writes plugstack.conf ("required") only
once every machine reports both, since a machine without the plugin would fail every job. Tried on
the VMs: on in ~30 s; ayse's srun --container-image=ubuntu:24.04 on n1 ran as uid 200001 with
/shared inside and nproc 1; a two-node sbatch ran a rank on each. Seen once: a node's pull timed out
reaching auth.docker.io and the job failed (network, not pyxis). Not tried: GPUs in a container
(cuda01's agent is an older build; needs the peer protocol), Red Hat.
- AD end to end, tried 2026-11-14 (decided: one AD per install, the existing auth.ldap group mapping, no
tree from AD, no sssd unless a site asks): the VMs' auth.ldap against Samba DC VM 165 (ldaps by IP,
CA in the settings, ml-team -> team:Test/arge). ada signed in on the portal with her AD password: an
account, in Test/arge, Slurm user ada UID 200003; her key added in the portal's Profile, ssh to the
controller ~20 s later, sbatch ran on n1 as ada. Disabled in AD: suspended in 56 s, the Linux user
gone in 89 s, ssh refused, home kept. Enabled: back in 32 s, same UID, key kept. Taken out of
ml-team: out of the team in 11 s; put back: in again in 56 s.
- P5, replaced 2026-11-17 by the portal (decided 2026-11-15: users of every level, build for the one who
knows least first; any project member may start, no approval, one per project, the project's by default):
**Customise a model** (`/train`): an xlsx/CSV/JSONL of questions and answers, checked and previewed; base
weights from the model's origin (a GGUF repo's card base_model), memory from their size (2x + 1 GiB),
the node with the most free; published by itself when done (llama.cpp replicas replaced to load it);
shared me/project/team/org/everyone (wider than the project only confirmed, never wider than its
knowledge bases), deleted from the page. Left: pairs generated from a knowledge base on the page, a
base-vs-new comparison, the training's GPU time charged to the project, big jobs through Slurm, the
server's own error messages in Turkish.
- P6, done 2026-11-15 but for budgets: each project a Slurm account (org and team accounts above it,
ids keyed by kind: the first install's org, team and project share the id "default"), its members and its
team's and org's on it; AccountingStorageEnforce=associations; the controller's agent keeps them with
sacctmgr (only what it made; a user's default account moved before an association goes, the user taken out
when none is left) and reports the jobs that ended in the last day; the leader records each once
(slurm_jobs, keyed cluster/id) with a usage row, model slurm/PARTITION, priced at gpu_hour_cost or
core_hour_cost. Tried on the VMs: accounts made, ayse's job on her project's account ran and another
project's was refused, both jobs in deneme's usage at 345 and 0 micro-units. Left: budgets (a job's cost
is not counted against a budget, which binds keys; GrpTRESMins from a project's budget would hold its jobs).
- P6 as written 2026-11-12: The tenancy tree to Slurm:
each project an account (`sacctmgr add account`, parent its team and organization), each member a user
of it, kept by the leader from the store; `sbatch --account=PROJECT`. The records back: the leader reads
`sacct --json` for jobs ended since the last read (through the controller's agent) into a
`batch_usage` table (job, user, project, start, end, GPU-seconds by class, CPU-seconds, memory), priced
per GPU-hour and core-hour (auth.gpu_hour_cost, core_hour_cost: the same basis `ctl prices suggest`
prices tokens from, 2026-11-15) into cost_micros, summed with the requests' usage on the usage pages and in
budgets. Budgets enforced: a project's remaining budget as GPU-minutes in its account's GrpTRESMins, so
Slurm holds its new jobs once spent. Needs the same users on every node (open: LDAP/sssd or
llmgw-made). Also found: slurm.conf
wrote every node State=DRAIN, so each slurmctld start (on any change of the files) drained a lent node
until its agent resumed it, up to a minute; a lent node is now written without it. Not tried yet: a GPU node (gres).
- Failover while lent, tried 2026-11-11 (v0.16.3-23): the first try killed the job (NODE_FAIL). A new
leader had heard from no node yet and sent the controller a slurm.conf without them; slurmctld,
restarted on it, dropped node05 and its job. Fixed (141a1f4, 0774e90): slurm.conf's nodes and the
controller's address are kept in cluster_state (`slurm-nodes`) and a new leader starts from them; with
nothing kept (an upgraded cluster) or the store unread, the controller's files wait up to 2 min for the
nodes; a node is told nothing until its controller's address is known (no part stops its slurmd). Tried
again: a 5-minute job on lent node05, leader 140 stopped, 150 took over; slurmctld not restarted, the job
ran to its end; reclaimed during it, node05 was llmgw's within seconds of its end. Also 3387f97: srun in
SrunPortRange=60001-63000, and Slurm's ports in llmgw's firewall for the cluster's networks.
- Built 2026-11-11: P1 (`internal/slurm`, `slurm.*` settings), P2
(`internal/agent/slurmnode`, proto Slurm/SlurmStatus), P3 (`internal/control/slurmsvc`, wired in the
leader; `GET /api/v1/slurm`), P4's commands (`ctl slurm status|setup|lend|reclaim|off`). Decisions on the
way: a node not heard from or with Slurm jobs is cordoned, not drained (nothing of llmgw's to move, and a
new leader must not move models off every Slurm node); jobs are counted from the node's own slurmstepd,
so a controller that is down neither hides them nor, with none, keeps the node from llmgw; a drain whose
reason does not start with "llmgw" is a Slurm administrator's and never resumed. Left: ~~the TUI screen~~
(done 2026-11-11: console **Slurm**, lend/take back on a row), P5,
a try on the PoC (needs agents built from this; slurm-wlm 23.11 on Ubuntu 24.04), GPU minor numbers are
taken to be nvidia-smi's index.
- In the first install too (2026-11-11): `llmgw server init --slurm`, and a Slurm part on the choices
screen, write `slurm: {enabled, controller: <this hostname>, all_nodes: true}` into server.yaml.
`slurm.all_nodes` (also `ctl slurm setup --all-nodes`) runs slurmd on every node in the leader's
inventory, those that join later too; none is lent. Refused with `--no-node`, or a hostname Slurm does
not take.
## D. Facts worth knowing before starting ## D. Facts worth knowing before starting
- Admin API: `127.0.0.1:9090`, behind the token in - Admin API: `127.0.0.1:9090`, behind the token in
+204 -1
View File
@@ -20,12 +20,215 @@ is only released once its **exit criterion** is met, not when its code lands.
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released | | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) | | v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) | | v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node) | | v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node; v0.16.3: replicas per profile, a served model's settings on the mirror page, the CPU's speed before serving; v0.16.4: Slurm beside llmgw, a model customised from the portal, a GPU node without a driver reported) |
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned | | v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
Until v1.0.0 the minor version is the breaking one: configuration formats and Go Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period. APIs change between minors without a deprecation period.
## v0.16.4 — 2026-11-19
Slurm beside llmgw for training and batch work — its users, shared files, containers, accounting priced in llmgw's
books — a model customised from the portal with a file and a button, an organization's admin, and a GPU node whose
driver was never installed told apart from a CPU node.
**Upgrading.**
- Four migrations (0052–0055: Slurm's users and jobs, held capacity, the portal's training jobs), applied by the
first control plane to start on this version; nothing to do by hand. `llmgw ctl upgrade v0.16.4`: the control
planes carry Slurm, the training page, the admin role and the prices; the agents carry Slurm's node part, the
containers, and the reading of the PCI bus for a card without a driver, without which such a node still joins
as a CPU node. Built with Go 1.27, the newest patch at build time.
### Added
- **Customise a model from the portal.** A new page, **Customise a model**
(`/train`), lets anyone in a project teach a model how to answer without a
terminal: an Excel, CSV or JSONL file of questions and answers (columns
headed Soru/Cevap or Question/Answer, else the first two), checked and
shown before anything trains; a model; a name; who may use it; and a
button. The weights to train on are found from where the model's came (a
GGUF repository's base model on Hugging Face), the card's memory from their
size, the node as the one with the most free; when it is done the adapter
is published by itself and appears as a model of its own in the chat and
the API, a llama.cpp model's replicas replaced one at a time to load it.
It is the project's by default; its members may share it with the person,
the team, the organization or everyone (wider than the project only once
they confirm the file holds nothing that group must not see, and never
wider than the knowledge bases its pairs came from), and delete it. One
trains at a time per project. `llmgw ctl train start` needs only `--model`
and `--data` now, and `--publish` publishes when done.
- **An organization's admin.** Besides member and manager, a membership of
an organization may now be `admin` (`llmgw users join … --org Acme --role
admin`, the role chosen on a membership on the people page, the console,
the admin API). An admin sees the organization's page as a manager does and
also changes the organization from it: adds, renames and removes its teams
and projects, adds people with an approved account by their address and
takes them out, makes its members managers or admins, and revokes keys spent
in its projects. Limits, approving and suspending accounts, the
organization's name and the rest of the cluster stay with the cluster's
administrators; an admin cannot take away their own role. Every change is in
the audit log.
- **Slurm beside llmgw, for training and batch work.** `llmgw ctl slurm setup
--controller NODE --nodes NODE,...` (or `--all-nodes`, or `server init
--slurm`) has the agents install munge and Slurm from the machines' package
sources: slurmctld on one node, slurmd configless on the others, slurm.conf
and gres.conf written from what each node reports. Every node stays llmgw's,
drained in Slurm, until `ctl slurm lend NODE`: its replicas move off, then
Slurm runs jobs there. `ctl slurm reclaim NODE` takes it back once its jobs
end. `ctl slurm status` and the console's **Slurm** show each node. The
controller also keeps the jobs' records: slurmdbd, in a MariaDB on that
machine alone (Slurm takes MySQL or MariaDB only). Slurm's ports
(6817–6819, srun's 60001–63000) answer the cluster's networks only, as
llmgw's own do.
- **Slurm's shared files and users.** `slurm.shared` (`ctl slurm shared`) is
one path on every Slurm machine: the site's NFS export mounted, an llmgw
node exporting it over NFS (what `server init --slurm` chooses, `/shared`
from the first machine), or a file system the site mounts. `slurm.users:
llmgw` (`ctl slurm users on`) makes each active account a Linux user on
every Slurm machine, the same login and UID (from 200000, never reused)
everywhere, its home on the shared path; people add SSH keys in the
portal's Profile (`ctl slurm users keys` for an operator). They log in to
the controller; to another Slurm machine only while a job of theirs runs
there, their session put in the job (pam_slurm_adopt), so nobody uses a
node's cards outside Slurm, nor on a node llmgw serves from. Root and the
machines' own users are not held to it.
- **Slurm's jobs in llmgw's accounting.** With `slurm.users: llmgw` each
project is a Slurm account (organizations and teams above it), its members
and its team's and organization's on it, and a job must run on one of its
user's (`AccountingStorageEnforce=associations`). Each job that ends is
recorded once, its GPU-hours (or core-hours) priced at `auth.gpu_hour_cost`
into its project's usage as model `slurm/PARTITION`. Accounts a Slurm
administrator made are not touched.
- **Prices from what the cluster costs.** `auth.gpu_hour_cost` (and
`core_hour_cost`) is what an hour of a GPU (a core) costs; the leader keeps
the hours each model's replicas hold, and `llmgw ctl prices suggest` prices
each model from them over the tokens it served, an output token four input
ones and a cached one a tenth. `--apply` writes the prices.
- **Slurm's jobs in containers.** `llmgw ctl slurm containers on`
(`slurm.containers`) installs enroot and pyxis on every Slurm machine, at
pinned versions checked against their digests, and nvidia-container-cli on
GPU machines; Slurm loads pyxis once every machine has it, and then `srun
--container-image=...` runs a job in that image, as its user, with the
shared path inside.
### Changed
- **Forms only for those who may send them.** A viewer sees the people,
limits, mirror and cluster pages without their forms, an operator the
mirror's and a node's engines but not the rest; before, they were drawn and
answered "not found".
- **Smaller forms.** A label sits close to its field, a field is as wide as
what goes in it rather than the page, buttons are smaller, and the short
forms — a new key, a password, connecting an editor, a capacity or a limit —
sit on one line or two instead of a screen.
### Fixed
- **A GPU node with no driver joined quietly as a CPU node.** The agent found
GPUs through `nvidia-smi` alone, so a machine whose driver was never
installed looked exactly like one with no card. The PCI bus is read as
well: a card there with no driver fails the checks before the agent is
installed, with the install and the reboot named (and a driver installed
but not loaded — Secure Boot, nouveau — told apart from a missing one), and
a node already in shows the warning in `ctl nodes` and the console until
the driver is in and its agent restarted.
- **The serve forms start at the longest context that fits.** The mirror
page's form began every GGUF model at 4096 tokens, and the console's at
4096 or the plan's figure, whatever the model was trained for. Both now
start at what it was trained for, as long as its KV cache fits where the
form places it beside the weights, else at the most that does, in whole
thousands; the mirror page's llama.cpp slots are as many of that, up to
four, as the room left holds, and its memory figure is for that pool. A
plan says what the model was trained for (`trained`). Strata stays at
its 32768; vLLM's form is unchanged.
- **A model that failed to start was started in the same place again.**
Placement given the same inputs chooses the same node and card, so an
on-demand model whose replica failed to load was retried there, past each
backoff, until it was parked, with another card or node free all along.
The control plane now remembers where a model's replicas failed (node and
cards, up to eight places) and places the next one elsewhere when anywhere
else fits; with nowhere else it is tried there as before. Coming up, a
change to its definition or `ctl retry` forgets the places. The log says
`placing a model away from where it failed to start`, with both places.
- **Deleted model files came back after a control plane restart.** An agent
registers with the files its startup scan found, and sends the list again
only when it changes; a file deleted after the agent started was listed
again by a control plane that restarted, until the next change. A new
stream now gets the list on its first heartbeat.
- **`ctl weights` offered models in use for deletion.** A model of several
files (a set, its address ending in `/`) is fetched into a directory named
from the address without the slash; the sweep named it from the address
with it, so every set in the catalog looked like nobody's. On the PoC that
was 330 GB of six catalog models, the qwen3.8-flash-next Q8_0 among them;
`--collect`, and `ctl model remove --purge` of a set, would have deleted
them. Both now compute the name the fetch does.
- **A sign-in took an organization's manager role away.** A group from single
sign-on, SCIM or Active Directory that puts someone in an organization wrote
the membership again at every sign-in or sync, as a plain member: a manager
made one on the people page was one only until they next signed in. A
membership the directory names is now added if missing and otherwise left
as it is.
- **The limits form on the people page's panel** was laid out on one line, its
title squeezed into a column.
- **Deleting the model files nothing uses said what it was asked to free,
not what it freed.** The console's "Delete model files nothing uses" and
`ctl weights --collect` added up every file the nodes were asked to
delete as soon as the request was sent, and a model of several files —
fetched into a directory of its own — was refused by the node as "not
empty", so a sweep said tens of gigabytes freed and nothing went. The
node now deletes such a directory whole (a link only as a link), and the
sweep waits for the nodes to list their files again: freed is what they
no longer list, and what they kept is said beside, with where to read
why. Removing a model with its files reports the same way.
## v0.16.3 — 2026-11-03
Replicas counted per profile — seven on cards and four on CPUs under one name, the cards filled first — and every
setting of a served model from the mirror page; how fast a model answers on the CPU, said before it is served, by
the threads it is given rather than the whole machine's memory.
**Upgrading.**
- No migrations. `llmgw ctl upgrade v0.16.3`: the control planes carry the replicas per profile, the model card and
the speed in the serve form and in `ctl model plan`; the agents carry the measurement of one core's memory rate,
without which the speed is reckoned the old way, from the whole machine's.
### Added
- **Replicas counted per profile: seven on cards and four on CPUs, under one
name.** `llmgw ctl model replicas <model> gpu=7 cpu=4`, the console's
Replicas per profile… and the mirror page's model card set how many of a
model's replicas run on each of its profiles, rather than every replica
on the best profile that fits. For a GGUF model with no profile on CPUs
one is added (`--cpu-memory`), preferred below the cards, so the router
fills the cards first and sends to the CPUs once they are full. A change
starts the replicas short before stopping those over: the model never
runs fewer. `ctl models` shows the counts; a scaling of another total
ends them.
- **Every setting of a served model from the mirror page.** A served
model's row has a Settings link, and its card now changes the context
and longest answer, the cards a replica takes, the replicas on each
profile (with the CPU's speed beside), stops and starts it, restarts its
replicas, and takes it out of the catalog with its files or without —
beside the replicas, class, owner, memory and adapters it held before.
Stopped on the row is now the model stopped and kept, started again as
it was, rather than its scaling set to nothing.
- **How fast a model answers on the CPU, before it is served.** The serve
form says, as the cores are changed, about how many tokens a second one
reply gets on the CPU and on one card of the fleet's fastest class, and
warns when it is slow (under 15) or very slow (under 5, a reply of 500
tokens taking minutes). Very slow on the CPU is refused until "run on the
CPU anyway" is ticked, GPUs free or not. `llmgw ctl model plan` and the
console's Hugging Face screen show the same figure for every way a model
can run, and `ctl model serve --cpu` says it in its log.
- **The speed follows the threads a replica is given.** Each agent now
measures what one core reads beside what all of them read: a replica of
four threads on a large server reads at about four cores' rate, not the
server's, which the estimate used to assume and overstated several times.
Calibrated on the PoC by llama-bench at 1, 2, 4 and 8 threads (within 15%),
and a model of experts reads its experts' share, from the GGUF header.
Agents need upgrading to report one core's rate; until then the old
estimate is used.
## v0.16.2 — 2026-11-02 ## v0.16.2 — 2026-11-02
Standby control planes that answer clients themselves, with rate limits and budgets kept across them; FreeToken, an Standby control planes that answer clients themselves, with rate limits and budgets kept across them; FreeToken, an
+105
View File
@@ -0,0 +1,105 @@
# Functional Source License, Version 1.1, ALv2 Future License
## Abbreviation
FSL-1.1-ALv2
## Notice
Copyright 2026 Ilker Manap
## Terms and Conditions
### Licensor ("We")
The party offering the Software under these Terms and Conditions.
### The Software
The "Software" is each version of the software that we make available under
these Terms and Conditions, as indicated by our inclusion of these Terms and
Conditions with the Software.
### License Grant
Subject to your compliance with this License Grant and the Patents,
Redistribution and Trademark clauses below, we hereby grant you the right to
use, copy, modify, create derivative works, publicly perform, publicly display
and redistribute the Software for any Permitted Purpose identified below.
### Permitted Purpose
A Permitted Purpose is any purpose other than a Competing Use. A Competing Use
means making the Software available to others in a commercial product or
service that:
1. substitutes for the Software;
2. substitutes for any other product or service we offer using the Software
that exists as of the date we make the Software available; or
3. offers the same or substantially similar functionality as the Software.
Permitted Purposes specifically include using the Software:
1. for your internal use and access;
2. for non-commercial education;
3. for non-commercial research; and
4. in connection with professional services that you provide to a licensee
using the Software in accordance with these Terms and Conditions.
### Patents
To the extent your use for a Permitted Purpose would necessarily infringe our
patents, the license grant above includes a license under our patents. If you
make a claim against any party that the Software infringes or contributes to
the infringement of any patent, then your patent license to the Software ends
immediately.
### Redistribution
The Terms and Conditions apply to all copies, modifications and derivatives of
the Software.
If you redistribute any copies, modifications or derivatives of the Software,
you must include a copy of or a link to these Terms and Conditions and not
remove any copyright notices provided in or with the Software.
### Disclaimer
THE SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTIES OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING WITHOUT LIMITATION WARRANTIES OF FITNESS FOR A PARTICULAR
PURPOSE, MERCHANTABILITY, TITLE OR NON-INFRINGEMENT.
IN NO EVENT WILL WE HAVE ANY LIABILITY TO YOU ARISING OUT OF OR RELATED TO THE
SOFTWARE, INCLUDING INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES,
EVEN IF WE HAVE BEEN INFORMED OF THEIR POSSIBILITY IN ADVANCE.
### Trademarks
Except for displaying the License Details and identifying us as the origin of
the Software, you have no right under these Terms and Conditions to use our
trademarks, trade names, service marks or product names.
## Grant of Future License
We hereby irrevocably grant you an additional license to use the Software under
the Apache License, Version 2.0 that is effective on the second anniversary of
the date we make the Software available. On or after that date, you may use the
Software under the Apache License, Version 2.0, in which case the following
will apply:
Licensed under the Apache License, Version 2.0 (the "License"); you may not use
this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
+8 -1
View File
@@ -42,7 +42,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released | | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) | | v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) | | v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node) | | v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node; v0.16.3: replicas per profile, a served model's settings on the mirror page, the CPU's speed before serving; v0.16.4: Slurm beside llmgw, a model customised from the portal, a GPU node without a driver reported) |
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned | | v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
As of v0.6.0 a cluster runs unattended: agents report their hardware, the As of v0.6.0 a cluster runs unattended: agents report their hardware, the
@@ -372,3 +372,10 @@ make rename NEW=git.example.com/team/llmgw
## Language ## Language
Code, comments, tests, commit messages and documentation are in English. Code, comments, tests, commit messages and documentation are in English.
## Licence
Functional Source License 1.1 with an Apache 2.0 future licence
(FSL-1.1-ALv2); see [LICENSE](LICENSE). Each release may be used, read and
changed for anything except offering it as a competing product or service,
and becomes Apache 2.0 two years after it is published.
File diff suppressed because it is too large. Load diff
+138
View File
@@ -30,6 +30,7 @@ message AgentMessage {
KVEvents kv_events = 8; KVEvents kv_events = 8;
TrainStatus train_status = 9; TrainStatus train_status = 9;
TrainArtifact train_artifact = 10; TrainArtifact train_artifact = 10;
SlurmStatus slurm_status = 11;
} }
} }
@@ -78,6 +79,7 @@ message ControlMessage {
AgentSettings agent_settings = 15; AgentSettings agent_settings = 15;
TrainStart train_start = 16; TrainStart train_start = 16;
TrainCancel train_cancel = 17; TrainCancel train_cancel = 17;
Slurm slurm = 18;
} }
// trace is the W3C trace context (traceparent, and tracestate when there // trace is the W3C trace context (traceparent, and tracestate when there
// is one) of the control plane's span that sent this command, so what the // is one) of the control plane's span that sent this command, so what the
@@ -241,6 +243,14 @@ message Host {
// mem_bandwidth_gbs is how fast the memory reads, in GB/s, measured when // mem_bandwidth_gbs is how fast the memory reads, in GB/s, measured when
// the agent started: what decides a model's speed on the CPU. 0 unknown. // the agent started: what decides a model's speed on the CPU. 0 unknown.
double mem_bandwidth_gbs = 7; double mem_bandwidth_gbs = 7;
// mem_bandwidth_core_gbs is what one core reads alone, in GB/s: a replica
// given a few threads of a large machine reads at a few times this, not at
// the whole machine's rate. 0 unknown.
double mem_bandwidth_core_gbs = 8;
// nvidia_device is whether an NVIDIA display controller sits on the PCI
// bus, driver or not. With nvidia_driver empty it is a card nothing can
// use yet, which must be said rather than counted as no card at all.
bool nvidia_device = 9;
} }
// Storage is one filesystem a node keeps weights on. // Storage is one filesystem a node keeps weights on.
@@ -618,6 +628,8 @@ message Heartbeat {
// the agent has measured it — in the background after it starts, so a // the agent has measured it — in the background after it starts, so a
// registration usually has none yet. 0 means not measured. // registration usually has none yet. 0 means not measured.
double mem_bandwidth_gbs = 20; double mem_bandwidth_gbs = 20;
// mem_bandwidth_core_gbs is one core's read rate, as in Host.
double mem_bandwidth_core_gbs = 21;
} }
// Provision is the software a node should have: which engines, at which // Provision is the software a node should have: which engines, at which
@@ -980,3 +992,129 @@ message TrainArtifact {
bytes data = 4; bytes data = 4;
bool last = 5; bool last = 5;
} }
// Slurm is what this node is in the Slurm cluster run beside llmgw
// (BACKLOG P): its controller, one of its nodes, both or neither. Sent when
// an agent registers and whenever it changes; the agent installs what is
// missing and keeps the daemons running as it says.
message Slurm {
// controller runs slurmctld here, on files.
bool controller = 1;
// node runs slurmd here, configless, fetching its files from conf_server.
bool node = 2;
// name is this node's NodeName in slurm.conf.
string name = 3;
// conf_server is slurmctld's host:port.
string conf_server = 4;
// munge_key is the key every machine of the cluster shares.
bytes munge_key = 5;
// files are the controller's, by name in /etc/slurm; empty for a node
// that is not the controller.
map<string, string> files = 6;
// lent says Slurm may run jobs here now: resume the node if llmgw drained
// it. Otherwise it is drained, its jobs left to finish.
bool lent = 7;
// shared is the file system every machine of the Slurm cluster sees at
// the same path; unset or with no path, none.
SlurmShared shared = 8;
// manage_users has llmgw keep the Linux users below on this machine,
// and take away those it made that are no longer among them.
bool manage_users = 9;
repeated SlurmUser users = 10;
// make_homes has this machine make the users' home directories: the one
// that exports the shared path, else the controller.
bool make_homes = 11;
// containers has the machine install pyxis and enroot, so jobs may run
// in containers (srun --container-image), and set them up.
bool containers = 12;
// manage_accounts has the controller keep Slurm's accounts as accounts
// says: one for each organization, team and project, nested, the users
// on each project's. Only those llmgw made are changed or taken away.
bool manage_accounts = 13;
repeated SlurmAccount accounts = 14;
}
// SlurmAccount is an account in Slurm's accounting, parents before their
// children.
message SlurmAccount {
string name = 1;
string parent = 2;
repeated string users = 3;
}
// SlurmJob is a job that ended, as sacct has it, for llmgw's accounting.
message SlurmJob {
string id = 1;
string user = 2;
string account = 3;
string partition = 4;
string state = 5;
int64 start_unix = 6;
int64 end_unix = 7;
int64 elapsed_seconds = 8;
int32 cpus = 9;
int32 gpus = 10;
}
// SlurmShared is where the Slurm cluster's shared files come from.
message SlurmShared {
// path is where every machine sees them, as /shared.
string path = 1;
// nfs is an export, host:/path, to mount at path; empty when this
// machine exports path itself (serve) or the site mounts it.
string nfs = 2;
// serve has this machine export path over NFS to clients.
bool serve = 3;
// clients are the addresses the export answers.
repeated string clients = 4;
}
// SlurmUser is a Linux user that runs jobs, the same on every machine.
message SlurmUser {
string login = 1;
// uid is also the user's own group's gid.
uint32 uid = 2;
// name is the account's, for the GECOS field.
string name = 3;
string home = 4;
// ssh_keys are OpenSSH public keys, one a line of authorized_keys.
repeated string ssh_keys = 5;
}
// SlurmStatus is what this node's Slurm daemons are doing, sent when it
// changes and every few minutes while the node has a part in Slurm.
message SlurmStatus {
// version is Slurm's, as "23.11.4"; empty when it is not installed.
string version = 1;
// hardware is `slurmd -C` on a node, which slurm.conf must match.
string hardware = 2;
bool slurmd_running = 3;
bool slurmctld_running = 4;
// state and reason are the controller's for this node, from scontrol.
string state = 5;
string reason = 6;
// jobs is how many job steps run on this node now, counted from their
// slurmstepd so a controller that is down does not hide them; -1 when the
// agent cannot tell.
int32 jobs = 7;
// error is the last thing that went wrong keeping Slurm, empty when
// nothing did.
string error = 8;
// files_sha256 is of the controller's files as written, so the leader
// knows they arrived.
string files_sha256 = 9;
// slurmdbd_running is the accounting daemon on the controller, which keeps
// the jobs' records in a MariaDB beside it.
bool slurmdbd_running = 10;
// shared is the shared path as this machine has it: "mounted",
// "exported", "missing", or empty when there is none.
string shared = 11;
// users is how many of llmgw's users this machine has.
int32 users = 12;
// containers is what this machine has for jobs in containers, as
// "pyxis 0.24.0, enroot 4.2.1"; empty while it has not got both.
string containers = 13;
// ended are, on the controller, the jobs that ended in the last day; the
// leader records each once.
repeated SlurmJob ended = 14;
}
+1 -1
View File
@@ -16,7 +16,7 @@
# machine, manages its GPUs and supervises processes there as root, which is # machine, manages its GPUs and supervises processes there as root, which is
# what a container is built to prevent: docs/kubernetes.md. # what a container is built to prevent: docs/kubernetes.md.
FROM golang:1.26 AS build FROM golang:1.27 AS build
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
+556 -3
View File
@@ -95,6 +95,9 @@
{ {
"name": "presets" "name": "presets"
}, },
{
"name": "prices"
},
{ {
"name": "projects" "name": "projects"
}, },
@@ -116,6 +119,9 @@
{ {
"name": "settings" "name": "settings"
}, },
{
"name": "slurm"
},
{ {
"name": "summary" "name": "summary"
}, },
@@ -3227,6 +3233,58 @@
} }
} }
}, },
"/api/v1/models/{name}/replicas": {
"post": {
"operationId": "postModelsNameReplicas",
"summary": "How many replicas run on each profile",
"description": "Seven on cards and four on CPUs, under the model's name. A profile not named keeps its count; cpu adds a profile on CPUs to a llama.cpp model that has none. Its scaling becomes the sum.",
"tags": [
"models"
],
"parameters": [
{
"name": "name",
"in": "path",
"required": true,
"description": "The model's name in the catalog.",
"schema": {
"type": "string"
}
},
{
"$ref": "#/components/parameters/Actor"
}
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ProfileReplicasRequest"
}
}
}
},
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/catalog.Model"
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/models/{name}/resources": { "/api/v1/models/{name}/resources": {
"post": { "post": {
"operationId": "postModelsNameResources", "operationId": "postModelsNameResources",
@@ -4341,6 +4399,34 @@
} }
} }
}, },
"/api/v1/prices/suggest": {
"get": {
"operationId": "getPricesSuggest",
"summary": "Each model's price, from what it held of the cluster",
"description": "days (30 if not given) is the window. A model's cost is the GPU- and core-hours its replicas held, idle ones too, at auth.gpu_hour_cost and auth.core_hour_cost; spread over its tokens, an output token four input ones and a cached prompt token a tenth of one, it is price, in micro-units a thousand tokens as auth.pricing takes them. current is the price in force. 409 with no cost set.",
"tags": [
"prices"
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/PriceSuggestions"
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/projects": { "/api/v1/projects": {
"post": { "post": {
"operationId": "postProjects", "operationId": "postProjects",
@@ -5026,6 +5112,117 @@
} }
} }
}, },
"/api/v1/slurm": {
"get": {
"operationId": "getSlurm",
"summary": "The Slurm cluster beside llmgw, and where each of its nodes is",
"description": "From the cluster's settings (slurm.*) and what each node's agent last said. A node's phase is llmgw (drained in Slurm, no job left), lending (its replicas moving off), lent, reclaiming (drained in Slurm, jobs still running) or unknown (its agent has not said).",
"tags": [
"slurm"
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/slurmsvc.Status"
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/slurm/users": {
"get": {
"operationId": "getSlurmUsers",
"summary": "The Linux users made from llmgw's accounts for the Slurm cluster",
"description": "Every active account, with its login, UID and SSH keys; an account is given its login and UID the first time it is listed. The machines have them when slurm.users is \"llmgw\".",
"tags": [
"slurm"
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/slurmsvc.User"
}
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/slurm/users/{user}/ssh-keys": {
"put": {
"operationId": "putSlurmUsersUserSshKeys",
"summary": "Replace an account's SSH public keys",
"description": "{user} is an ID or e-mail. keys is OpenSSH public keys, one a line, without options; empty takes them all away. The machines have them within a minute.",
"tags": [
"slurm"
],
"parameters": [
{
"name": "user",
"in": "path",
"required": true,
"description": "The person's ID or email.",
"schema": {
"type": "string"
}
},
{
"$ref": "#/components/parameters/Actor"
}
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/SlurmKeysRequest"
}
}
}
},
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/slurmsvc.User"
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/summary": { "/api/v1/summary": {
"get": { "get": {
"operationId": "getSummary", "operationId": "getSummary",
@@ -6182,7 +6379,7 @@
}, },
"CollectResult": { "CollectResult": {
"type": "object", "type": "object",
"description": "CollectResult is what was deleted on one node, or why nothing was.", "description": "CollectResult is what was deleted on one node, or why nothing was: what the node no longer lists once asked, not what it was asked to delete.",
"properties": { "properties": {
"node": { "node": {
"type": "string" "type": "string"
@@ -6194,6 +6391,14 @@
"type": "integer", "type": "integer",
"format": "int64" "format": "int64"
}, },
"left": {
"type": "integer",
"description": "Left and LeftBytes are what the node still lists: refused, or not gone by the time the sweep stopped waiting. Its agent's log says why."
},
"left_bytes": {
"type": "integer",
"format": "int64"
},
"error": { "error": {
"type": "string" "type": "string"
} }
@@ -8103,6 +8308,13 @@
"type": "string" "type": "string"
} }
}, },
"profile_replicas": {
"type": "object",
"additionalProperties": {
"type": "integer"
},
"description": "ProfileReplicas is how many replicas run on each profile, for a model whose replicas are counted per profile; absent otherwise."
},
"memory_gb": { "memory_gb": {
"type": "integer", "type": "integer",
"description": "MemoryGB and CPUCores are what one replica reserves, from its first profile; MeasuredMemoryGB is the most one of its replicas uses now, 0 when none is measured." "description": "MemoryGB and CPUCores are what one replica reserves, from its first profile; MeasuredMemoryGB is the most one of its replicas uses now, 0 when none is measured."
@@ -8305,6 +8517,10 @@
"type": "number", "type": "number",
"description": "MemBandwidthGBs is how fast its memory reads, in GB/s, measured when its agent started; 0 when not reported. A model's speed on the CPU." "description": "MemBandwidthGBs is how fast its memory reads, in GB/s, measured when its agent started; 0 when not reported. A model's speed on the CPU."
}, },
"mem_bandwidth_core_gbs": {
"type": "number",
"description": "MemBandwidthCoreGBs is what one core reads alone, in GB/s; 0 when not reported. A CPU replica's threads read at about this each."
},
"engines": { "engines": {
"type": "array", "type": "array",
"items": { "items": {
@@ -8661,6 +8877,88 @@
} }
} }
}, },
"PriceSuggestion": {
"type": "object",
"description": "PriceSuggestion is one model's.",
"properties": {
"model": {
"type": "string"
},
"gpu_hours": {
"type": "number"
},
"core_hours": {
"type": "number"
},
"cost": {
"type": "number",
"description": "Cost is what it held cost, in budget units."
},
"prompt_tokens": {
"type": "integer",
"format": "int64"
},
"completion_tokens": {
"type": "integer",
"format": "int64"
},
"price": {
"$ref": "#/components/schemas/accounting.Price",
"description": "Price is the suggestion; zero with a Note saying why there is none."
},
"note": {
"type": "string"
},
"current": {
"$ref": "#/components/schemas/accounting.Price"
}
}
},
"PriceSuggestions": {
"type": "object",
"description": "PriceSuggestions is a model's price from its cost, for each model.",
"properties": {
"days": {
"type": "integer",
"description": "Days is the window; RecordedFrom the first hour held is known for, later than the window's start on a cluster that began keeping it since, which makes the window shorter than asked."
},
"since": {
"type": "string",
"format": "date-time"
},
"recorded_from": {
"type": "string",
"format": "date-time"
},
"gpu_hour_cost": {
"type": "number"
},
"core_hour_cost": {
"type": "number"
},
"models": {
"type": "array",
"items": {
"$ref": "#/components/schemas/PriceSuggestion"
}
}
}
},
"ProfileReplicasRequest": {
"type": "object",
"description": "ProfileReplicasRequest counts a model's replicas per profile: profile name to replicas, and CPU, for a llama.cpp model with no profile on CPUs, the one to add.",
"properties": {
"profiles": {
"type": "object",
"additionalProperties": {
"type": "integer"
}
},
"cpu": {
"$ref": "#/components/schemas/modelsvc.CPUProfile"
}
}
},
"ProjectView": { "ProjectView": {
"type": "object", "type": "object",
"description": "ProjectView is a project, with how many people and keys are in it.", "description": "ProjectView is a project, with how many people and keys are in it.",
@@ -9284,6 +9582,15 @@
} }
} }
}, },
"SlurmKeysRequest": {
"type": "object",
"description": "SlurmKeysRequest is a person's SSH public keys, one a line, as authorized_keys has them; empty takes them all away.",
"properties": {
"keys": {
"type": "string"
}
}
},
"StoreMember": { "StoreMember": {
"type": "object", "type": "object",
"description": "StoreMember is one member of a replicated store, as its Patroni reports it.", "description": "StoreMember is one member of a replicated store, as its Patroni reports it.",
@@ -9386,6 +9693,10 @@
"guard_off": { "guard_off": {
"type": "boolean", "type": "boolean",
"description": "GuardOff says the guardrails are turned off (guard.enabled: false): no request is checked, whatever the policies say." "description": "GuardOff says the guardrails are turned off (guard.enabled: false): no request is checked, whatever the policies say."
},
"no_alerts": {
"type": "boolean",
"description": "NoAlerts says the cluster's settings name no webhook: a model parked, a node lost or a judge down is told nobody."
} }
} }
}, },
@@ -9633,6 +9944,12 @@
"created_by": { "created_by": {
"type": "string" "type": "string"
}, },
"project": {
"type": "string"
},
"publish": {
"type": "boolean"
},
"artifacts": { "artifacts": {
"type": "array", "type": "array",
"items": { "items": {
@@ -9678,18 +9995,26 @@
}, },
"base_repo": { "base_repo": {
"type": "string", "type": "string",
"description": "BaseRepo and BaseRevision are the weights trained on: the model's own on its registry, in safetensors (Hugging Face), not a GGUF." "description": "BaseRepo and BaseRevision are the weights trained on: the model's own on its registry, in safetensors (Hugging Face), not a GGUF. Empty finds them from where the model's came: the repository itself, or the one its card says it was made from."
}, },
"base_revision": { "base_revision": {
"type": "string" "type": "string"
}, },
"node": { "node": {
"type": "string", "type": "string",
"description": "Node trains it; GPUMemoryMiB is what it needs of a card, which can be what an engine there leaves free." "description": "Node trains it; empty is the first with a card that has enough free. GPUMemoryMiB is what it needs of a card, which can be what an engine there leaves free; 0 is worked out from the weights' size."
}, },
"gpu_memory_mib": { "gpu_memory_mib": {
"type": "integer" "type": "integer"
}, },
"project": {
"type": "string",
"description": "Project is the project the job is started in: its members see and manage it, and only one of its jobs trains at a time."
},
"publish": {
"type": "boolean",
"description": "Publish puts what the job makes on the mirror and makes it the model's adapter by itself when the job is done, restarting a llama.cpp model's replicas to load it."
},
"owner": { "owner": {
"type": "string", "type": "string",
"description": "Owner is who the adapter will belong to: org:, team:, project: (by path or ID), user: or key:. Only callers there see it. When the pairs come from one knowledge base, it defaults to that collection's owner; it must be able to read every collection they come from." "description": "Owner is who the adapter will belong to: org:, team:, project: (by path or ID), user: or key:. Only callers there see it. When the pairs come from one knowledge base, it defaults to that collection's owner; it must be able to read every collection they come from."
@@ -9794,6 +10119,35 @@
} }
} }
}, },
"accounting.Price": {
"type": "object",
"description": "Price is what one model costs, in micro-units per thousand tokens.\n\nMicro-units rather than a float because spend is accumulated by repeated addition, and repeated addition of floats drifts. What a unit means is the operator's: self-hosted models have no per-token price, so it may be GPU-seconds, a notional rate, or 1000 everywhere — which turns a budget into a token quota.",
"properties": {
"PromptPer1k": {
"type": "integer",
"format": "int64"
},
"CompletionPer1k": {
"type": "integer",
"format": "int64"
},
"CachedPer1k": {
"type": "integer",
"format": "int64",
"description": "CachedPer1k is what a prompt token costs that the engine or provider read from its cache instead of computing: the part of prompt_tokens the answer counts as cached_tokens. Zero charges those at PromptPer1k, as before there was a rate for them."
},
"PerImage": {
"type": "integer",
"format": "int64",
"description": "PerImage is what one picture costs, for a model that makes them: a picture's cost is the card's time, which its prompt's few tokens do not measure."
},
"PerMesh": {
"type": "integer",
"format": "int64",
"description": "PerMesh is what one 3D model costs: minutes of a card, which no token count measures."
}
}
},
"accounting.Rates": { "accounting.Rates": {
"type": "object", "type": "object",
"description": "Rates are an accountant's two limiters' counts.", "description": "Rates are an accountant's two limiters' counts.",
@@ -10506,6 +10860,10 @@
"type": "integer", "type": "integer",
"description": "Priority orders profiles during placement; highest that fits wins." "description": "Priority orders profiles during placement; highest that fits wins."
}, },
"replicas": {
"type": "integer",
"description": "Replicas, set on any of a model's profiles, is how many of its replicas run on this one: seven on cards and four on CPUs, all answering under the model's name, rather than each replica on the best profile that fits. The model's scaling is then the sum, min and max alike; a profile left at zero runs none."
},
"startup_timeout": { "startup_timeout": {
"type": "integer", "type": "integer",
"format": "int64", "format": "int64",
@@ -10664,6 +11022,24 @@
} }
} }
}, },
"config.SharedStorage": {
"type": "object",
"description": "SharedStorage is where the Slurm cluster's shared files come from: the site's NFS export (NFS), an NFS export llmgw keeps on one of its nodes (Server), or, with neither, a file system the site mounts itself (Lustre, BeeGFS, GPFS), which llmgw only looks for.",
"properties": {
"Path": {
"type": "string",
"description": "Path is where every machine sees it, as /shared."
},
"NFS": {
"type": "string",
"description": "NFS is the site's export, host:/path, mounted at Path on every machine of the Slurm cluster."
},
"Server": {
"type": "string",
"description": "Server is the llmgw node, the controller or one of the nodes, that exports Path over NFS to the others: for a site with no storage of its own. One machine: not highly available, and no faster than its disk and network."
}
}
},
"control.Activity": { "control.Activity": {
"type": "object", "type": "object",
"description": "Activity is what a standby's router reports back: when it last sent each replica a request, and the on-demand models it was asked for. The leader evicts and loads by these, and without them a replica busy behind a standby looks idle to the one machine that decides.", "description": "Activity is what a standby's router reports back: when it last sent each replica a request, and the on-demand models it was asked for. The leader evicts and loads by these, and without them a replica busy behind a standby looks idle to the one machine that decides.",
@@ -11406,6 +11782,10 @@
}, },
"strata_size": { "strata_size": {
"type": "string" "type": "string"
},
"tokens_per_second": {
"type": "number",
"description": "TokensPerSecond is about how many tokens a second one reply gets this way, on the fastest host it fits; 0 when not known. On the CPU with the threads a replica is given by default: half the node's."
} }
} }
}, },
@@ -11651,6 +12031,10 @@
"context": { "context": {
"type": "integer" "type": "integer"
}, },
"trained": {
"type": "integer",
"description": "Trained is the context the model was trained for, the most a request may use; 0 when neither its header nor its config says."
},
"kv_bytes_per_token": { "kv_bytes_per_token": {
"type": "integer", "type": "integer",
"format": "int64", "format": "int64",
@@ -11898,6 +12282,21 @@
} }
} }
}, },
"modelsvc.CPUProfile": {
"type": "object",
"description": "CPUProfile is a profile on CPUs to add to a llama.cpp model, for replicas counted per profile: what a replica of it reserves.",
"properties": {
"name": {
"type": "string"
},
"memory_gb": {
"type": "integer"
},
"cpu_cores": {
"type": "integer"
}
}
},
"notify.Result": { "notify.Result": {
"type": "object", "type": "object",
"description": "Result is how one webhook answered a test.", "description": "Result is how one webhook answered a test.",
@@ -12065,6 +12464,160 @@
} }
} }
}, },
"slurmsvc.NodeStatus": {
"type": "object",
"description": "NodeStatus is one node as the service sees it.",
"properties": {
"node": {
"type": "string"
},
"phase": {
"type": "string"
},
"replicas": {
"type": "integer"
},
"stops_by": {
"type": "string",
"format": "date-time",
"description": "StopsBy is, while it is being lent, when a replica still on it is stopped even with nowhere else to go: the drain's grace after the lend began. Zero when not known (a leader that took over meanwhile)."
},
"slurm": {
"$ref": "#/components/schemas/slurmsvc.Reported"
},
"connected": {
"type": "boolean"
}
}
},
"slurmsvc.Reported": {
"type": "object",
"description": "Reported is the agent's SlurmStatus, for the admin API.",
"properties": {
"version": {
"type": "string"
},
"hardware": {
"type": "string"
},
"slurmd_running": {
"type": "boolean"
},
"slurmctld_running": {
"type": "boolean"
},
"slurmdbd_running": {
"type": "boolean"
},
"shared": {
"type": "string",
"description": "Shared is the shared path here: \"mounted\", \"exported\", \"missing\"."
},
"containers": {
"type": "string",
"description": "Containers is what the machine has for jobs in containers, empty while not both of pyxis and enroot."
},
"users": {
"type": "integer",
"description": "Users is how many of llmgw's users the machine has."
},
"state": {
"type": "string"
},
"reason": {
"type": "string"
},
"jobs": {
"type": "integer"
},
"error": {
"type": "string"
},
"files_sha256": {
"type": "string"
},
"at": {
"type": "string",
"format": "date-time"
}
}
},
"slurmsvc.Status": {
"type": "object",
"description": "Status is the whole Slurm cluster, for `ctl slurm status`.",
"properties": {
"enabled": {
"type": "boolean"
},
"cluster": {
"type": "string"
},
"controller": {
"type": "string"
},
"partition": {
"type": "string"
},
"controller_status": {
"$ref": "#/components/schemas/slurmsvc.Reported"
},
"nodes": {
"type": "array",
"items": {
"$ref": "#/components/schemas/slurmsvc.NodeStatus"
}
},
"waiting": {
"type": "array",
"items": {
"type": "string"
},
"description": "Waiting are nodes left out of slurm.conf until they report their hardware."
},
"shared": {
"$ref": "#/components/schemas/config.SharedStorage",
"description": "Shared is where the shared files are and come from; Users who keeps the users, and how many llmgw has."
},
"users": {
"type": "string"
},
"user_count": {
"type": "integer"
},
"containers": {
"type": "string",
"description": "Containers is whether jobs may run in containers: \"on\" once every machine has pyxis and enroot, \"installing\" until then, empty when slurm.containers is off."
},
"accounts": {
"type": "integer",
"description": "Accounts is how many of Slurm's accounts are llmgw's projects'."
}
}
},
"slurmsvc.User": {
"type": "object",
"description": "User is a Linux user made from an llmgw account.",
"properties": {
"login": {
"type": "string"
},
"uid": {
"type": "integer"
},
"name": {
"type": "string"
},
"email": {
"type": "string"
},
"ssh_keys": {
"type": "array",
"items": {
"type": "string"
}
}
}
},
"store.AuditEvent": { "store.AuditEvent": {
"type": "object", "type": "object",
"description": "AuditEvent is one change: who made it, through what, to what, and how it ended. Detail never holds a secret; the writer takes them out.", "description": "AuditEvent is one change: who made it, through what, to what, and how it ended. Detail never holds a secret; the writer takes them out.",
+17 -2
View File
@@ -67,7 +67,8 @@ or with the mirror unticked, is said in the form and nothing starts.
## Watch ## Watch
**Overview** is `llmgw ctl status`: nodes, GPUs by class, fabrics, replicas, **Overview** is `llmgw ctl status`: nodes, GPUs by class, fabrics, replicas,
models, the store's members, and anything parked, unplaced or idle. models, the store's members, and anything parked, unplaced or idle. With no
webhook in the cluster's settings it says the alerts go nowhere.
**Monitor** draws the cluster the way btop draws a machine: CPU, memory, **Monitor** draws the cluster the way btop draws a machine: CPU, memory,
GPU busy and GPU memory, requests and tokens a second, time to first token, GPU busy and GPU memory, requests and tokens a second, time to first token,
@@ -207,6 +208,19 @@ starting — and whether it ended done, failed, or waiting for room on busy
cards. The entry shows how many are running. Enter on one shows its steps. cards. The entry shows how many are running. Enter on one shows its steps.
`llmgw ctl jobs` is the same, and `llmgw ctl jobs ID` follows one. `llmgw ctl jobs` is the same, and `llmgw ctl jobs ID` follows one.
**Slurm** is the Slurm cluster run beside llmgw, when there is one: its
controller and whether slurmctld runs, and each node that runs slurmd —
whose it is now (llmgw's, Slurm's, or on its way between them), its state in
Slurm, the jobs running there. On a row, **Lend to Slurm…** moves the node's
replicas off and then lets Slurm run jobs there; **Take back from Slurm…**
starts no new job there and gives it back to llmgw when its jobs have ended.
Both change `slurm.lent` in the cluster's settings, as `llmgw ctl slurm lend`
and `reclaim` do. Turning Slurm on and choosing its nodes is `llmgw ctl slurm
setup`, or the Slurm part of the first install. Above the nodes it says where
the shared files come from and how many users llmgw made; the SHARED column
is each node's (mounted, exported, missing). `llmgw ctl slurm shared` and
`llmgw ctl slurm users` change them.
**Logs** shows a node's agent log, newest at the bottom, errors and warnings **Logs** shows a node's agent log, newest at the bottom, errors and warnings
coloured. coloured.
@@ -220,7 +234,8 @@ and change it with the same commands as `llmgw users`, `llmgw keys` and
dashboard, and what they belong to. **A** creates an account, already dashboard, and what they belong to. **A** creates an account, already
approved. On a row: approve one waiting, suspend or reinstate, give or take approved. On a row: approve one waiting, suspend or reinstate, give or take
the cluster dashboard, add to or take out of an organization, team or the cluster dashboard, add to or take out of an organization, team or
project, their limits, a new password (their sessions close), delete, and project (on an organization as a member, its manager, who sees its page, or
its admin, who also changes it), their limits, a new password (their sessions close), delete, and
**Forget them…** for a request to be forgotten (`llmgw users forget`). **Forget them…** for a request to be forgotten (`llmgw users forget`).
**API keys** — every key, its project, owner, state and last use. **A** **API keys** — every key, its project, owner, state and last use. **A**
+1 -1
View File
@@ -27,7 +27,7 @@ make test
./bin/llmgw --version ./bin/llmgw --version
``` ```
`make build` writes `./bin/llmgw`. Go 1.26.6 or later is the only requirement (an older Go fetches it itself). `make build` writes `./bin/llmgw`. Go 1.27 or later is the only requirement (an older Go fetches it itself); the patch level is never pinned, so CI and the containers build with the newest 1.27.
## 2. Check the path with a fake engine ## 2. Check the path with a fake engine
+91 -3
View File
@@ -166,13 +166,85 @@ is a flag too, and the plan shows the command with them: `--starter qwen3-8b`
`--judge-fail open`, `--judge-categories`, `--judge-replicas`, `--judge-fail open`, `--judge-categories`, `--judge-replicas`,
`--judge-budget`, `--language tr`, `--signup-open`, `--default-rpm`, `--judge-budget`, `--language tr`, `--signup-open`, `--default-rpm`,
`--default-tpm`, `--max-key-ttl`, `--media` (pictures and 3D: ComfyUI on the `--default-tpm`, `--max-key-ttl`, `--media` (pictures and 3D: ComfyUI on the
nodes, and `presets.auto`), `--evict-after-idle` and `--min-lifetime`. They nodes, and `presets.auto`), `--slurm` (see below), `--evict-after-idle` and `--min-lifetime`. They
go into `server.yaml` as init writes it, and become the cluster's first go into `server.yaml` as init writes it, and become the cluster's first
settings when it starts; a `server.yaml` already there is never overwritten, settings when it starts; a `server.yaml` already there is never overwritten,
so run again they change nothing — the settings page does. An NVIDIA card with no working so run again they change nothing — the settings page does. An NVIDIA card with no working
driver is reported rather than silently ignored; install the driver and run driver is reported rather than silently ignored; install the driver and run
init again to serve on it. init again to serve on it.
**Slurm, for training and batch jobs.** `--slurm`, or the Slurm part on the
choices screen, runs a Slurm cluster beside llmgw: slurmctld on this
machine's node, slurmd on every node, those that join later too
(`slurm.all_nodes`). The agents install munge and Slurm (22.05 or newer)
from the machines' own package sources, and on this machine slurmdbd, which
keeps the jobs' records (`sacct`), in a MariaDB installed beside it that
answers this machine alone. Slurm keeps its records in MySQL or MariaDB
only, not in the cluster's PostgreSQL; the database's password is made from
the cluster's munge key, so nothing new is stored. A MariaDB the machine
already had keeps its settings and only gets the database `slurm_acct_db`
and its user. Back it up with the machine (`mariadb-dump slurm_acct_db`);
records older than a year are purged. No node is Slurm's at first: each is
drained in Slurm and serves models until `llmgw ctl slurm lend NODE` gives it
to Slurm — its models move off first — and `llmgw ctl slurm reclaim NODE`
takes it back once its jobs end. Jobs are submitted on this machine with
`sbatch` and `srun`; `llmgw ctl slurm status` shows where each node is. It
needs this machine to be a node, and its hostname to be a name Slurm takes.
On a cluster installed without it, `llmgw ctl slurm setup --controller NODE
--all-nodes` does the same later. Slurm's ports (6817–6819, and srun's 60001–63000) answer the cluster's
networks only, as llmgw's own do; a firewall of the site's own between the
nodes must let them through.
**Slurm's shared files and users.** Jobs need the same files on every
machine: homes, datasets, output, checkpoints. With `--slurm` this machine
exports `/shared` over NFS to every Slurm machine and they mount it
(`slurm.shared`); NFS's port 2049 answers the cluster's networks only, the
export the Slurm machines alone. That is one machine, not highly available
and no faster than its disk and network: enough for a few nodes. A site with
storage of its own uses it instead — `--slurm-shared HOST:/EXPORT` at
install, or `llmgw ctl slurm shared --path /shared --nfs HOST:/EXPORT`
later; Lustre, BeeGFS or GPFS the site mounts itself, and `llmgw ctl slurm
shared --path /lustre/hpc` only has llmgw look that it is there. Every
active llmgw account is a Linux user on every Slurm machine
(`slurm.users: llmgw`): its login from its e-mail address, a UID from
200000 up that is never given to anyone else, its home under
`/shared/home`. People add their SSH public keys in the portal's
**Profile** and log in to this machine to run `sbatch`; an account suspended
or deleted loses its user, its files stay. On the other Slurm machines they
are let in only while a job of theirs runs there, and their ssh session is
put in that job, held to its cores and cards (pam_slurm_adopt, which the
agents install); root and the machines' own users log in as before. A site whose users come from its
directory (sssd against Active Directory or LDAP) passes `--slurm-users
site`, or later `llmgw ctl slurm users off`.
With llmgw's users, Slurm's accounts are llmgw's projects: an account for
each organization, team and project, nested (`acme`, `acme-ml`,
`acme-ml-training`), and on each project's account its members and its
team's and organization's. A job runs on one of its user's accounts and no
other (`sbatch --account=acme-ml-training`; without it, the user's first),
and when it ends its cards' hours (a job without cards, its cores') are
priced at `auth.gpu_hour_cost` (`core_hour_cost`) into the project's usage,
where it shows as model `slurm/PARTITION` beside the project's requests.
Accounts a Slurm administrator made are left alone. A job's cost does not
yet count against a budget, which binds API keys only.
Jobs may run in containers: `llmgw ctl slurm containers on` has the agents
install enroot and pyxis on every Slurm machine (NVIDIA's releases, at
versions llmgw pins and checks; pyxis is built against each machine's own
Slurm) and, on GPU machines, nvidia-container-cli from NVIDIA's repository.
Once every machine has them Slurm loads pyxis, and `ctl slurm status` says
`containers: on`:
```sh
srun --gres=gpu:1 --container-image=nvcr.io#nvidia/pytorch:25.01-py3 python train.py
```
The shared path is in every container where it is on the host, and the job
runs as its user, held to its cores and cards. Each node pulls the image
from its registry for each job; for a large image used often, import it
once (`enroot import -o /shared/images/pytorch.sqsh docker://nvcr.io#...`)
and give that file to `--container-image`.
**Secure by default.** What init writes is the hardened configuration, not a **Secure by default.** What init writes is the hardened configuration, not a
demonstration one: demonstration one:
@@ -396,6 +468,19 @@ by itself. Three things are worth knowing before you edit it:
cache (the answer's `cached_tokens`) are charged at it, the rest at the cache (the answer's `cached_tokens`) are charged at it, the rest at the
prompt rate. A provider that bills cached reads at a tenth is prompt rate. A provider that bills cached reads at a tenth is
`{prompt_per_1k: 2500, cached_per_1k: 250, completion_per_1k: 10000}`. `{prompt_per_1k: 2500, cached_per_1k: 250, completion_per_1k: 10000}`.
- What to price a self-hosted model at follows from what the cluster costs.
Set `auth.gpu_hour_cost`, what an hour of one GPU costs in the units budgets
are in: the hardware written off over its years, its power and room, and the
people who run it, over the GPUs and the 730 hours of a month (an 8-GPU
server at 250,000 over four years, 10 kW, and some upkeep is about 1.35);
`auth.core_hour_cost` likewise for models on CPUs. The leader keeps, by the
hour, the GPU- and core-hours each model's replicas hold, and `llmgw ctl
prices suggest` spreads them over the tokens the model served: an output
token counted as four input ones (making one is that much dearer), a cached
prompt token as a tenth of one. Idle hours are in the cost, so a model held
all month and seldom used comes out dear, which is the point; one held and
billed nothing (a guard's judge) is shown with what it held. `--apply`
writes the prices into `auth.pricing`.
- `control.enabled: true` is what makes this a control plane while its catalog - `control.enabled: true` is what makes this a control plane while its catalog
is empty. Without it, a file with no models is refused as having nothing to is empty. Without it, a file with no models is refused as having nothing to
serve. serve.
@@ -582,7 +667,10 @@ set `discovery.nvidia_smi` in the agent configuration rather than moving it.
A node with no GPU at all is fine and is not an error. It reports zero cards and A node with no GPU at all is fine and is not an error. It reports zero cards and
can host CPU profiles, which is a good way to try llmgw before committing can host CPU profiles, which is a good way to try llmgw before committing
hardware to it. hardware to it. A node with an NVIDIA card and no driver is an error: adding it
stops at the checks and says so, and a node already added shows the warning in
`llmgw ctl nodes` and the console until the driver is installed and its agent
restarted.
### NVLink, if your GPUs have it ### NVLink, if your GPUs have it
@@ -3145,7 +3233,7 @@ loaded at boot by `llmgw-firewall.service`):
| Machine | Ports, for the cluster only | | Machine | Ports, for the cluster only |
| --- | --- | | --- | --- |
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote | | control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
| node | engine ports (`control.port_range`, 20000–20999), 9100 | | node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6819 (slurmctld, slurmd, slurmdbd) and 60001–63000 (srun's `SrunPortRange`); exporting Slurm's shared files, 2049 (NFS) |
| store member | 5432, 8008, 2379, 2380, 9100 | | store member | 5432, 8008, 2379, 2380, 9100 |
| mirror | 8081, 8082, 9100 | | mirror | 8081, 8082, 9100 |
+12 -5
View File
@@ -385,18 +385,25 @@ right and mixed facts up on the rest, while the same questions through the
knowledge base got 9 of 9. Use both: the knowledge base for what answers knowledge base got 9 of 9. Use both: the knowledge base for what answers
say, an adapter for how they sound. say, an adapter for how they sound.
Anyone in a project can do it from the portal: **Customise a model**
(`/train`) takes an Excel, CSV or JSONL file of questions and answers, a
model, a name and who may use it, and publishes the adapter by itself when it
is trained. The weights, the node and the card's memory are worked out. From
the command line:
```sh ```sh
# 1. Questions and answers from the collection, written by a model of the # 1. Questions and answers from the collection, written by a model of the
# cluster. Read the file through and correct it. # cluster. Read the file through and correct it.
llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl
# 2. Train on a node. --gpu-memory is what the job needs of a card; it # 2. Train. The weights to train on are found from where the model's came
# runs beside an engine when the engine leaves that much free. # (--base-repo names others), the card's memory from their size
llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \ # (--gpu-memory), the node as the one whose card has the most free
--base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl # (--node); it runs beside an engine when the engine leaves that much.
llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct --data policies.jsonl
llmgw ctl train show JOB --follow llmgw ctl train show JOB --follow
# 3. Make it the model's adapter, kept on the mirror. # 3. Make it the model's adapter, kept on the mirror (or --publish in step 2).
llmgw ctl train publish JOB --restart llmgw ctl train publish JOB --restart
``` ```
Binary file not shown.
+6
View File
@@ -181,6 +181,12 @@ Both default to the usual places. Containerized driver installs routinely put
`nvidia-smi` somewhere else, and an agent that cannot find it registers with no `nvidia-smi` somewhere else, and an agent that cannot find it registers with no
GPUs — which looks exactly like a node that has none. GPUs — which looks exactly like a node that has none.
A node with no driver at all is told apart from one with no card: the PCI bus
is read as well, and an NVIDIA card there with no driver loaded is a failure in
the checks before the agent is installed (what to install, and that a reboot
follows), and a warning on the node in `llmgw ctl nodes` and the console
afterwards. It serves on CPU until the driver is in and its agent restarted.
## Removing a node ## Removing a node
``` ```
+32 -11
View File
@@ -192,22 +192,41 @@ auth:
A personal key has no project, so it keeps no conversations or agent notes; A personal key has no project, so it keeps no conversations or agent notes;
those are kept per project. those are kept per project.
### Organization managers ### Organization managers and admins
A member of an organization can be made its manager: A member of an organization can be made its manager, who sees it, or its
admin, who also changes it:
``` ```
llmgw users join boss@acme.com --org Acme --role manager llmgw users join boss@acme.com --org Acme --role manager
llmgw users join lead@acme.com --org Acme --role admin
``` ```
or with the star on their organization membership on `/admin/users`. A manager or with the role chosen on their organization membership on `/admin/users`.
gets an **Organization** link and a page at `/org`: usage over time by team, Both get an **Organization** link and a page at `/org`: usage over time by team,
project or person, the teams and projects with their limits and spend, and every project or person, the teams and projects with their limits and spend, and every
member. Only what was spent in that organization's projects is on it; what its member. Only what was spent in that organization's projects is on it; what its
people do elsewhere is not. The page is read-only — a manager is not an people do elsewhere is not.
administrator of the cluster, and limits and memberships stay with those who
are. An administrator following `/org` lands on the same organization's panel A manager's page is read-only. An organization's admin also changes the
on the people page instead, where all of that can be changed. organization from it:
- adds and renames its teams and projects, and removes an empty one;
- adds a person by their sign-in address to the organization, a team or a
project, and takes them out (their keys there are revoked with it);
- makes a member of the organization a manager or an admin, or neither;
- revokes a key spent in its projects.
What reaches past the organization stays with the cluster's administrators:
its limits and budgets, approving or suspending an account (someone not yet
approved cannot be added), the organization's own name, and anything else on
the cluster. An admin cannot take away their own role. A group from single
sign-on or Active Directory that puts someone in the organization does not, by
saying so again at the next sign-in, take a manager's or admin's role away.
An administrator of the cluster following `/org` lands on the same
organization's panel on the people page instead, where all of that can be
changed.
`llmgw keys init` still works for provisioning scripts and is idempotent, but it `llmgw keys init` still works for provisioning scripts and is idempotent, but it
will no longer move a team or project that already exists under another parent; will no longer move a team or project that already exists under another parent;
@@ -517,8 +536,10 @@ included: `llmgw ctl audit`, or **Audit** in the console.
suspends accounts from it; see [install.md](install.md#single-sign-on). suspends accounts from it; see [install.md](install.md#single-sign-on).
Without them accounts are local, with local passwords. Without them accounts are local, with local passwords.
- **Staff roles are coarse.** `viewer`, `operator`, `billing` and `admin` - **Staff roles are coarse.** `viewer`, `operator`, `billing` and `admin`
(`llmgw users role`) decide what someone may see and do under /admin; an (`llmgw users role`) decide what someone may see and do under /admin, and a
organization's manager can view it but not change it. Every change is in page draws only the forms that person may send: a viewer sees no forms. An
the audit log (`llmgw ctl audit`). organization's manager can view it; its admin also changes its teams,
projects, people and keys. Every change is in the audit log (`llmgw ctl
audit`).
- **Rate limits are per gateway instance.** With two instances each counts its own, - **Rate limits are per gateway instance.** With two instances each counts its own,
so the effective limit doubles. v1 is single-instance by design. so the effective limit doubles. v1 is single-instance by design.
+157
View File
@@ -43,6 +43,8 @@ rather than repeated.
| `personal` | `Personal` | [Auth](#auth) | | `personal` | `Personal` | [Auth](#auth) |
| `pricing` | `map[string]Price` | [Auth](#auth) | | `pricing` | `map[string]Price` | [Auth](#auth) |
| `default_price` | `Price` | [Auth](#auth) | | `default_price` | `Price` | [Auth](#auth) |
| `gpu_hour_cost` | `float64` | [Auth](#auth) |
| `core_hour_cost` | `float64` | [Auth](#auth) |
| `flush_interval` | `time.Duration` | [Auth](#auth) | | `flush_interval` | `time.Duration` | [Auth](#auth) |
| `batch_size` | `int` | [Auth](#auth) | | `batch_size` | `int` | [Auth](#auth) |
| `buffer_size` | `int` | [Auth](#auth) | | `buffer_size` | `int` | [Auth](#auth) |
@@ -295,6 +297,7 @@ rather than repeated.
| `keep_within` | `string` | [Profile](#profile) | | `keep_within` | `string` | [Profile](#profile) |
| `weight` | `float64` | [Profile](#profile) | | `weight` | `float64` | [Profile](#profile) |
| `priority` | `int` | [Profile](#profile) | | `priority` | `int` | [Profile](#profile) |
| `replicas` | `int` | [Profile](#profile) |
| `startup_timeout` | `time.Duration` | [Profile](#profile) | | `startup_timeout` | `time.Duration` | [Profile](#profile) |
| `kv_events` | `bool` | [Profile](#profile) | | `kv_events` | `bool` | [Profile](#profile) |
| `gpu` | `*GPUNeed` | [Profile](#profile) | | `gpu` | `*GPUNeed` | [Profile](#profile) |
@@ -375,6 +378,7 @@ rather than repeated.
| `mail` | `mail.Config` | [Server](#server) | | `mail` | `mail.Config` | [Server](#server) |
| `serving` | `ServingSettings` | [Server](#server) | | `serving` | `ServingSettings` | [Server](#server) |
| `firewall` | `FirewallSettings` | [Server](#server) | | `firewall` | `FirewallSettings` | [Server](#server) |
| `slurm` | `SlurmSettings` | [Server](#server) |
| `presets` | `PresetSettings` | [Server](#server) | | `presets` | `PresetSettings` | [Server](#server) |
| `on_demand` | `OnDemandSettings` | [Server](#server) | | `on_demand` | `OnDemandSettings` | [Server](#server) |
| `agents` | `AgentSettings` | [Server](#server) | | `agents` | `AgentSettings` | [Server](#server) |
@@ -399,7 +403,20 @@ rather than repeated.
| `action` | `string` | [Service](#service) | | `action` | `string` | [Service](#service) |
| `timeout` | `time.Duration` | [Service](#service) | | `timeout` | `time.Duration` | [Service](#service) |
| `stopped` | `bool` | [ServingSettings](#servingsettings) | | `stopped` | `bool` | [ServingSettings](#servingsettings) |
| `path` | `string` | [SharedStorage](#sharedstorage) |
| `nfs` | `string` | [SharedStorage](#sharedstorage) |
| `server` | `string` | [SharedStorage](#sharedstorage) |
| `bin` | `string` | [SimConfig](#simconfig) | | `bin` | `string` | [SimConfig](#simconfig) |
| `enabled` | `bool` | [SlurmSettings](#slurmsettings) |
| `cluster_name` | `string` | [SlurmSettings](#slurmsettings) |
| `controller` | `string` | [SlurmSettings](#slurmsettings) |
| `nodes` | `[]string` | [SlurmSettings](#slurmsettings) |
| `all_nodes` | `bool` | [SlurmSettings](#slurmsettings) |
| `lent` | `[]string` | [SlurmSettings](#slurmsettings) |
| `partition` | `string` | [SlurmSettings](#slurmsettings) |
| `shared` | `SharedStorage` | [SlurmSettings](#slurmsettings) |
| `users` | `string` | [SlurmSettings](#slurmsettings) |
| `containers` | `bool` | [SlurmSettings](#slurmsettings) |
| `dir` | `string` | [StrataConfig](#strataconfig) | | `dir` | `string` | [StrataConfig](#strataconfig) |
| `python` | `string` | [StrataConfig](#strataconfig) | | `python` | `string` | [StrataConfig](#strataconfig) |
| `engine` | `string` | [StrataConfig](#strataconfig) | | `engine` | `string` | [StrataConfig](#strataconfig) |
@@ -707,6 +724,21 @@ table costs whatever DefaultPrice says, which is nothing unless set.
Takes a [config.Price](#configprice) block. Takes a [config.Price](#configprice) block.
### `gpu_hour_cost`
`float64`
GPUHourCost is what an hour of one GPU costs the operator, in the
units budgets are in: the hardware written off over its years, its
power and room, and the people who run it, over the GPUs and the
hours of a month. With CoreHourCost for a CPU core, it is what
`llmgw ctl prices suggest` prices each model from: the hours its
replicas held, idle ones too, over the tokens it served.
### `core_hour_cost`
`float64`
### `flush_interval` ### `flush_interval`
`time.Duration` `time.Duration`
@@ -2611,6 +2643,16 @@ rather than a guess (plan §6.2.1).
Priority orders profiles during placement; highest that fits wins. Priority orders profiles during placement; highest that fits wins.
### `replicas`
`int`
Replicas, set on any of a model's profiles, is how many of its
replicas run on this one: seven on cards and four on CPUs, all
answering under the model's name, rather than each replica on the
best profile that fits. The model's scaling is then the sum, min and
max alike; a profile left at zero runs none.
### `startup_timeout` ### `startup_timeout`
`time.Duration` `time.Duration`
@@ -3268,6 +3310,15 @@ Firewall turns llmgw's firewall on every machine on or off.
Takes a [FirewallSettings](#firewallsettings) block. Takes a [FirewallSettings](#firewallsettings) block.
### `slurm`
`SlurmSettings`
Slurm is a Slurm cluster run beside llmgw on some of its nodes, and
which of them it may use now.
Takes a [SlurmSettings](#slurmsettings) block.
### `presets` ### `presets`
`PresetSettings` `PresetSettings`
@@ -3460,6 +3511,30 @@ running and every model's definition as it is: llmgw stop sets it,
llmgw start clears it, and the models come back as their catalog llmgw start clears it, and the models come back as their catalog
entries ask. In force on every control plane within seconds. entries ask. In force on every control plane within seconds.
## SharedStorage
### `path`
`string`
Path is where every machine sees it, as /shared.
### `nfs`
`string`
NFS is the site's export, host:/path, mounted at Path on every
machine of the Slurm cluster.
### `server`
`string`
Server is the llmgw node, the controller or one of the nodes, that
exports Path over NFS to the others: for a site with no storage of
its own. One machine: not highly available, and no faster than its
disk and network.
## SimConfig ## SimConfig
### `bin` ### `bin`
@@ -3469,6 +3544,88 @@ entries ask. In force on every control plane within seconds.
Bin is the llmgw binary itself; the engine is one of its subcommands. Bin is the llmgw binary itself; the engine is one of its subcommands.
Empty means whatever started this agent, which is right almost always. Empty means whatever started this agent, which is right almost always.
## SlurmSettings
### `enabled`
`bool`
Enabled runs Slurm on the nodes below. False stops llmgw keeping it;
what was installed is left installed, and every node is llmgw's.
### `cluster_name`
`string`
ClusterName is Slurm's ClusterName. Empty is "llmgw".
### `controller`
`string`
Controller is the llmgw node whose agent runs slurmctld. It needs no
GPU, and need not be one of Nodes.
### `nodes`
`[]string`
Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained
in Slurm, until it is also in Lent.
### `all_nodes`
`bool`
AllNodes runs slurmd on every llmgw node, those that join later too,
besides any named in Nodes. Each is llmgw's until it is lent, as any.
### `lent`
`[]string`
Lent are the nodes of Nodes Slurm may run jobs on now. A node put here
is drained of llmgw's replicas first, as `ctl drain` does, and resumed
in Slurm once none is left; a node taken out is drained in Slurm and
given back to llmgw when its last job ends.
### `partition`
`string`
Partition is the one partition every node is in. Empty is "gpu".
### `shared`
`SharedStorage`
Shared is the file system every machine of the Slurm cluster sees at
the same path: home directories, datasets, the jobs' output and
checkpoints. Empty path: none, and a batch job's output stays on the
node it ran on.
Takes a [SharedStorage](#sharedstorage) block.
### `users`
`string`
Users is who keeps the Linux users that run jobs, the same on every
machine of the Slurm cluster: "llmgw" makes one for each active llmgw
account (its login from the e-mail address, a UID of llmgw's from
200000 up, its SSH keys from the portal), let in to the controller,
and to a node only while a job of theirs runs there (pam_slurm_adopt);
empty leaves users to the site (a directory through sssd, or by hand).
### `containers`
`bool`
Containers lets jobs run in containers: `srun --container-image
nvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),
the GPUs and the shared path inside. The agents install both on
every Slurm machine; jobs take it once every machine has them.
## StrataConfig ## StrataConfig
### `dir` ### `dir`
+1 -1
View File
@@ -1,6 +1,6 @@
module gitea.mynodes.xyz/ilker/llmgw module gitea.mynodes.xyz/ilker/llmgw
go 1.26.6 go 1.27
require ( require (
connectrpc.com/connect v1.21.0 connectrpc.com/connect v1.21.0
+91
View File
@@ -0,0 +1,91 @@
package accounting
import (
"math"
"slices"
)
// Prices from what the cluster costs: what each model held of it (its
// replicas' GPU- and core-hours) at what an hour of a GPU or a core costs
// the operator, spread over the tokens it served. Idle time is in it: a
// model held all month and used an hour costs what it held, not the hour.
//
// An output token is priced OutputPerInput input tokens, as producing one
// is that much dearer (one step a token, against a prompt read in one
// pass); a cached prompt token a tenth of an input one.
const (
OutputPerInput = 4
CachedPerInput = 0.1
)
// Basis is what an hour costs, in the units budgets are in.
type Basis struct {
GPUHour float64
CoreHour float64
}
// Held is what a model held, and Served what it answered with.
type Held struct {
GPUSeconds, CoreSeconds int64
}
// Served is a model's tokens.
type Served struct {
Prompt, Completion int64
}
// Suggestion is one model's price from its cost.
type Suggestion struct {
Model string `json:"model"`
GPUHours float64 `json:"gpu_hours"`
CoreHours float64 `json:"core_hours"`
// Cost is what it held cost, in budget units.
Cost float64 `json:"cost"`
Prompt int64 `json:"prompt_tokens"`
Completion int64 `json:"completion_tokens"`
// Price is the suggestion; zero with a Note saying why there is none.
Price Price `json:"price"`
Note string `json:"note,omitempty"`
}
// Suggest prices every model that held something or served something.
func Suggest(held map[string]Held, served map[string]Served, b Basis) []Suggestion {
var names []string
for m := range held {
names = append(names, m)
}
for m := range served {
if _, ok := held[m]; !ok {
names = append(names, m)
}
}
slices.Sort(names)
out := make([]Suggestion, 0, len(names))
for _, m := range names {
h, s := held[m], served[m]
g := Suggestion{Model: m, GPUHours: float64(h.GPUSeconds) / 3600, CoreHours: float64(h.CoreSeconds) / 3600,
Prompt: s.Prompt, Completion: s.Completion}
g.Cost = g.GPUHours*b.GPUHour + g.CoreHours*b.CoreHour
// Tokens counted as output ones: an input token is a fraction of one.
units := float64(s.Completion) + float64(s.Prompt)/OutputPerInput
switch {
case g.Cost == 0 && units > 0:
g.Note = "held nothing recorded here: an outside provider's, or served before the hours were kept"
case g.Cost > 0 && units == 0:
g.Note = "held the cluster and was billed no tokens (a guard's judge, or a model priced by the picture or 3D model): no one pays what it held"
case g.Cost > 0:
// Micro-units per thousand output tokens, rounded up so a
// cheap model is never free.
out1k := int64(math.Ceil(g.Cost / units * 1000 * 1e6))
g.Price = Price{
CompletionPer1k: out1k,
PromptPer1k: ceilDiv(out1k, OutputPerInput),
CachedPer1k: max(int64(math.Ceil(float64(ceilDiv(out1k, OutputPerInput))*CachedPerInput)), 1),
}
}
out = append(out, g)
}
return out
}
func ceilDiv(a, b int64) int64 { return (a + b - 1) / b }
+45
View File
@@ -0,0 +1,45 @@
package accounting
import (
"strings"
"testing"
)
func TestSuggestSpreadsWhatAModelHeldOverItsTokens(t *testing.T) {
// The worked example: two cards for 1000 hours at 1.35 an hour is 2700;
// 3.0 billion output tokens and 4.0 billion input ones, counted as 1.0
// billion output ones, are 4.0 billion: 0.675 a million output tokens.
held := map[string]Held{
"llama-70b": {GPUSeconds: 2 * 1000 * 3600},
"idle": {GPUSeconds: 3600},
"small-cpu": {CoreSeconds: 8 * 100 * 3600},
}
served := map[string]Served{
"llama-70b": {Prompt: 4_000_000_000, Completion: 3_000_000_000},
"small-cpu": {Prompt: 1_000_000, Completion: 1_000_000},
"outside": {Prompt: 10, Completion: 10},
}
got := Suggest(held, served, Basis{GPUHour: 1.35, CoreHour: 0.01})
by := map[string]Suggestion{}
for _, s := range got {
by[s.Model] = s
}
l := by["llama-70b"]
if l.Cost != 2700 || l.Price.CompletionPer1k != 675 || l.Price.PromptPer1k != 169 || l.Price.CachedPer1k != 17 {
t.Errorf("llama: %+v", l)
}
// What it costs back at that price is what it held, within the rounding.
back := float64(l.Price.Cost(l.Prompt, 0, l.Completion, 0, 0)) / 1e6
if back < l.Cost || back > l.Cost*1.01 {
t.Errorf("charged %.2f for a cost of %.2f", back, l.Cost)
}
if s := by["small-cpu"]; s.Cost != 8 || s.Price.CompletionPer1k == 0 {
t.Errorf("cpu: %+v", s)
}
if s := by["idle"]; !s.Price.Zero() || !strings.Contains(s.Note, "billed no tokens") {
t.Errorf("idle: %+v", s)
}
if s := by["outside"]; !s.Price.Zero() || !strings.Contains(s.Note, "held nothing") {
t.Errorf("outside: %+v", s)
}
}
+73 -17
View File
@@ -35,6 +35,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery" "gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines" "gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision" "gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/slurmnode"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor" "gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train" "gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights" "gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
@@ -97,7 +98,10 @@ type Agent struct {
// (nil: this machine's shell). // (nil: this machine's shell).
fwMu sync.Mutex fwMu sync.Mutex
fwRunner firewall.Runner fwRunner firewall.Runner
inv discovery.Inventory // fwLast is the last firewall message, applied again when the node
// is given a part in Slurm. Under fwMu.
fwLast *llmgwv1.Firewall
inv discovery.Inventory
// shardReplica remembers which replica a shard belongs to, so a status // shardReplica remembers which replica a shard belongs to, so a status
// report can name both. The control plane keys on the replica. // report can name both. The control plane keys on the replica.
shardReplica map[string]string shardReplica map[string]string
@@ -148,6 +152,10 @@ type Agent struct {
// trainer runs LoRA training jobs, one at a time (BACKLOG F4). // trainer runs LoRA training jobs, one at a time (BACKLOG F4).
trainer *train.Runner trainer *train.Runner
// slurm keeps the node's part in Slurm (BACKLOG P); slurmOut is where
// its status goes, under mu.
slurm *slurmnode.Keeper
slurmOut chan<- *llmgwv1.AgentMessage
} }
// connectHTTPClient is the HTTP client connect uses; an interface so tests can // connectHTTPClient is the HTTP client connect uses; an interface so tests can
@@ -206,6 +214,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
} }
a.trainer = train.New(train.Env{EnginesDir: enginesDir, WorkDir: filepath.Join(cfg.Superv.WorkDir, "trainer"), a.trainer = train.New(train.Env{EnginesDir: enginesDir, WorkDir: filepath.Join(cfg.Superv.WorkDir, "trainer"),
RunAs: runAs, Log: log}) RunAs: runAs, Log: log})
a.slurm = slurmnode.New(slurmnode.Local{}, a.prov, slurmMarker, log)
return a, nil return a, nil
} }
@@ -232,6 +241,7 @@ func (a *Agent) Run(ctx context.Context) error {
} }
go a.provisionLoop(ctx) go a.provisionLoop(ctx)
go a.ensureDiags(ctx) go a.ensureDiags(ctx)
go a.slurm.Run(ctx, a.sendSlurm)
attempt := 0 attempt := 0
next := 0 next := 0
@@ -519,7 +529,8 @@ func (a *Agent) scan(ctx context.Context) error {
inv, err := discovery.Discover(ctx, opts, a.log) inv, err := discovery.Discover(ctx, opts, a.log)
host := discovery.DiscoverHost() host := discovery.DiscoverHost()
host.MinComputeCapability = discovery.MinComputeCapability(inv.GPUs) host.MinComputeCapability = discovery.MinComputeCapability(inv.GPUs)
host.MemBandwidthGBs = a.memBandwidth() bw := a.memBandwidth()
host.MemBandwidthGBs, host.MemBandwidthCoreGBs = bw.All, bw.Core
a.mu.Lock() a.mu.Lock()
a.inv = inv a.inv = inv
a.prov.Host = host a.prov.Host = host
@@ -529,6 +540,18 @@ func (a *Agent) scan(ctx context.Context) error {
return err return err
} }
// forgetSent is a new stream: a control plane that knows nothing of where
// installs stand, nor which files are here beyond what registration says —
// and registration says what the startup scan found. So the next heartbeat
// tells it both again. Without it, files deleted after the agent started came
// back on a control plane that restarted, and stayed until the next change.
func (a *Agent) forgetSent() {
a.mu.Lock()
defer a.mu.Unlock()
a.provSent = ""
a.lastWeights = ""
}
// session runs one connection until it breaks. // session runs one connection until it breaks.
func (a *Agent) session(ctx context.Context, server string) error { func (a *Agent) session(ctx context.Context, server string) error {
client := llmgwv1connect.NewAgentServiceClient(a.http, server) client := llmgwv1connect.NewAgentServiceClient(a.http, server)
@@ -565,20 +588,19 @@ func (a *Agent) session(ctx context.Context, server string) error {
reg.EngineVersions = a.engines(parent) reg.EngineVersions = a.engines(parent)
reg.EnginesProbed = true reg.EnginesProbed = true
reg.InstallableEngines = slices.DeleteFunc(provision.Installable(), a.disabled) reg.InstallableEngines = slices.DeleteFunc(provision.Installable(), a.disabled)
reg.Features = []string{FeatureFabricTest} reg.Features = []string{FeatureFabricTest, FeatureSlurm}
reg.RunningShards = a.runningShards() reg.RunningShards = a.runningShards()
a.mu.Lock() a.mu.Lock()
h := a.prov.Host h := a.prov.Host
// A new stream is a control plane that knows nothing of where installs
// stand, so the next heartbeat tells it again.
a.provSent = ""
a.mu.Unlock() a.mu.Unlock()
a.forgetSent()
reg.Host = &llmgwv1.Host{ reg.Host = &llmgwv1.Host{
OsId: h.OSID, OsVersion: h.OSVersion, OsCodename: h.OSCodename, OsId: h.OSID, OsVersion: h.OSVersion, OsCodename: h.OSCodename,
Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver, Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver, NvidiaDevice: h.NVIDIADevice,
// What is measured by now, not what the startup scan found before // What is measured by now, not what the startup scan found before
// the measurement ended: a reconnect would otherwise say 0. // the measurement ended: a reconnect would otherwise say 0.
MemBandwidthGbs: a.memBandwidth(), MemBandwidthGbs: a.memBandwidth().All,
MemBandwidthCoreGbs: a.memBandwidth().Core,
} }
if err := stream.Send(&llmgwv1.AgentMessage{ if err := stream.Send(&llmgwv1.AgentMessage{
@@ -616,6 +638,8 @@ func (a *Agent) session(ctx context.Context, server string) error {
go a.watchExits(ctx, out, unready(reg.RunningShards)) go a.watchExits(ctx, out, unready(reg.RunningShards))
go a.forwardKV(ctx, out) go a.forwardKV(ctx, out)
go a.reportCompute(ctx, out) go a.reportCompute(ctx, out)
a.setSlurmOut(out)
defer a.setSlurmOut(nil)
for { for {
msg, err := stream.Receive() msg, err := stream.Receive()
@@ -676,7 +700,8 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
// without anybody restarting anything. // without anybody restarting anything.
hb.MemoryTotalMb = int64(discovery.MemoryTotalMB()) hb.MemoryTotalMb = int64(discovery.MemoryTotalMB())
hb.CpuCount = int32(runtime.NumCPU()) hb.CpuCount = int32(runtime.NumCPU())
hb.MemBandwidthGbs = a.memBandwidth() bw := a.memBandwidth()
hb.MemBandwidthGbs, hb.MemBandwidthCoreGbs = bw.All, bw.Core
a.reprobeEngines() a.reprobeEngines()
a.mu.Lock() a.mu.Lock()
@@ -881,7 +906,23 @@ func (a *Agent) removeWeights(paths []string) {
a.log.Warn("refused to remove a file llmgw did not fetch", "path", p) a.log.Warn("refused to remove a file llmgw did not fetch", "path", p)
continue continue
} }
if err := os.Remove(abs); err != nil && !os.IsNotExist(err) { // A set — every file of one version, fetched into a directory of its
// own — goes whole: os.Remove refused it as not empty, and nothing
// was freed of what a model of several files left. Lstat, so a link
// is taken away and never what it points at.
info, err := os.Lstat(abs)
switch {
case os.IsNotExist(err):
continue
case err != nil:
a.log.Warn("could not look at cached weights", "path", p, "err", err)
continue
case info.IsDir():
err = os.RemoveAll(abs)
default:
err = os.Remove(abs)
}
if err != nil {
a.log.Warn("could not remove cached weights", "path", p, "err", err) a.log.Warn("could not remove cached weights", "path", p, "err", err)
continue continue
} }
@@ -922,6 +963,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
go a.applySettings(m.AgentSettings) go a.applySettings(m.AgentSettings)
case *llmgwv1.ControlMessage_TrainStart: case *llmgwv1.ControlMessage_TrainStart:
go a.trainJob(ctx, m.TrainStart, out) go a.trainJob(ctx, m.TrainStart, out)
case *llmgwv1.ControlMessage_Slurm:
a.slurmMessage(m.Slurm)
case *llmgwv1.ControlMessage_TrainCancel: case *llmgwv1.ControlMessage_TrainCancel:
a.trainer.Cancel(m.TrainCancel.GetJobId()) a.trainer.Cancel(m.TrainCancel.GetJobId())
case *llmgwv1.ControlMessage_RestartAgent: case *llmgwv1.ControlMessage_RestartAgent:
@@ -1573,20 +1616,32 @@ func (a *Agent) reportCompute(ctx context.Context, out chan<- *llmgwv1.AgentMess
// VMs on one host that all restart together measure each other, and read a // VMs on one host that all restart together measure each other, and read a
// third less than any of them gets alone. Once a process rather than an // third less than any of them gets alone. Once a process rather than an
// agent, so tests running many agents in one read it once. // agent, so tests running many agents in one read it once.
func (a *Agent) memBandwidth() float64 { func (a *Agent) memBandwidth() discovery.Bandwidth {
bwOnce.Do(func() { bwOnce.Do(func() {
go func() { go func() {
time.Sleep(bandwidthDelay()) time.Sleep(bandwidthDelay())
for { for {
if bw := measureBandwidth(); bw > math.Float64frombits(bwBits.Load()) { // The best of each seen so far: an hourly pass on a node
bwBits.Store(math.Float64bits(bw)) // busy serving reads less, and that is not the memory.
a.log.Info("this node's memory", "read_gbs", math.Round(bw*10)/10) bw := measureBandwidth()
cur := a.memBandwidthNow()
if bw.All > cur.All || bw.Core > cur.Core {
bwBits.Store(math.Float64bits(max(bw.All, cur.All)))
coreBits.Store(math.Float64bits(max(bw.Core, cur.Core)))
a.log.Info("this node's memory", "read_gbs", math.Round(bw.All*10)/10,
"one_core_gbs", math.Round(bw.Core*10)/10)
} }
time.Sleep(time.Hour) time.Sleep(time.Hour)
} }
}() }()
}) })
return math.Float64frombits(bwBits.Load()) return a.memBandwidthNow()
}
// memBandwidthNow is what has been measured so far, without starting a
// measurement.
func (a *Agent) memBandwidthNow() discovery.Bandwidth {
return discovery.Bandwidth{All: math.Float64frombits(bwBits.Load()), Core: math.Float64frombits(coreBits.Load())}
} }
// bandwidthDelay is how long an agent waits before measuring, so agents // bandwidthDelay is how long an agent waits before measuring, so agents
@@ -1594,8 +1649,9 @@ func (a *Agent) memBandwidth() float64 {
var bandwidthDelay = func() time.Duration { return 10*time.Second + rand.N(110*time.Second) } var bandwidthDelay = func() time.Duration { return 10*time.Second + rand.N(110*time.Second) }
var ( var (
bwOnce sync.Once bwOnce sync.Once
bwBits atomic.Uint64 bwBits atomic.Uint64
coreBits atomic.Uint64
// measureBandwidth is replaced in tests, which have no use for it. // measureBandwidth is replaced in tests, which have no use for it.
measureBandwidth = discovery.MeasureMemoryBandwidth measureBandwidth = discovery.MeasureMemoryBandwidth
) )
+20
View File
@@ -925,3 +925,23 @@ func TestABandwidthTestIsRunOnTheAdapterAskedFor(t *testing.T) {
t.Errorf("reply %+v, ran %+v", r, got) t.Errorf("reply %+v, ran %+v", r, got)
} }
} }
func TestANewStreamIsToldTheFilesAgain(t *testing.T) {
// Registration carries what the startup scan found; a file deleted since
// came back on a control plane that restarted, because the next
// heartbeat saw the list unchanged since it was last sent.
dir := t.TempDir()
h := start(t, func(c *Config) { c.ModelDirs = []string{dir} })
if err := os.WriteFile(filepath.Join(dir, "0123456789abcdef-m.gguf"), []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
h.ag.heartbeat()
if hb := h.ag.heartbeat().GetHeartbeat(); hb.GetWeightsChanged() {
t.Fatal("an unchanged list should not be sent again on the same stream")
}
h.ag.forgetSent()
hb := h.ag.heartbeat().GetHeartbeat()
if !hb.GetWeightsChanged() || len(hb.GetWeights()) != 1 {
t.Errorf("a new stream should be told the files: changed=%v %v", hb.GetWeightsChanged(), hb.GetWeights())
}
}
+35
View File
@@ -3,6 +3,7 @@ package discovery
import ( import (
"bufio" "bufio"
"os" "os"
"path/filepath"
"regexp" "regexp"
"runtime" "runtime"
"strconv" "strconv"
@@ -28,6 +29,10 @@ type Host struct {
CPUFlags []string `json:"cpu_flags,omitempty"` CPUFlags []string `json:"cpu_flags,omitempty"`
// NVIDIADriver is the loaded driver's version, empty when there is none. // NVIDIADriver is the loaded driver's version, empty when there is none.
NVIDIADriver string `json:"nvidia_driver,omitempty"` NVIDIADriver string `json:"nvidia_driver,omitempty"`
// NVIDIADevice is whether an NVIDIA display controller is on the PCI bus,
// driver or not. True with NVIDIADriver empty is a card without a driver:
// a machine that would otherwise serve on CPU with nothing saying why.
NVIDIADevice bool `json:"nvidia_device,omitempty"`
// MinComputeCapability is the oldest GPU's, 0 when there are none or it // MinComputeCapability is the oldest GPU's, 0 when there are none or it
// is not known. Not read here but from the GPU inventory, by the agent. // is not known. Not read here but from the GPU inventory, by the agent.
MinComputeCapability float64 `json:"min_compute_capability,omitempty"` MinComputeCapability float64 `json:"min_compute_capability,omitempty"`
@@ -35,6 +40,10 @@ type Host struct {
// measured: what decides a model's speed on the CPU. Not read here but // measured: what decides a model's speed on the CPU. Not read here but
// measured by the agent when it starts (MeasureMemoryBandwidth). // measured by the agent when it starts (MeasureMemoryBandwidth).
MemBandwidthGBs float64 `json:"mem_bandwidth_gbs,omitempty"` MemBandwidthGBs float64 `json:"mem_bandwidth_gbs,omitempty"`
// MemBandwidthCoreGBs is what one core reads alone, in GB/s, 0 when not
// measured: a replica given n threads reads at about n times this, up to
// MemBandwidthGBs.
MemBandwidthCoreGBs float64 `json:"mem_bandwidth_core_gbs,omitempty"`
} }
// Family is the packaging family: "deb", "rpm", or "" when neither is known. // Family is the packaging family: "deb", "rpm", or "" when neither is known.
@@ -71,9 +80,35 @@ func DiscoverHost() Host {
h.Arch = debianArch(runtime.GOARCH) h.Arch = debianArch(runtime.GOARCH)
h.CPUFlags = cpuFlagsFrom("/proc/cpuinfo") h.CPUFlags = cpuFlagsFrom("/proc/cpuinfo")
h.NVIDIADriver = nvidiaDriverFrom("/proc/driver/nvidia/version") h.NVIDIADriver = nvidiaDriverFrom("/proc/driver/nvidia/version")
h.NVIDIADevice = HasNVIDIADevice()
return h return h
} }
// CardWithoutDriver reports an NVIDIA card on the PCI bus with no driver
// loaded: a node that looks like a CPU node to nvidia-smi and is not one.
func (h Host) CardWithoutDriver() bool { return h.NVIDIADevice && h.NVIDIADriver == "" }
// HasNVIDIADevice reports an NVIDIA display or 3D controller on the PCI bus,
// read from sysfs, which needs no driver.
func HasNVIDIADevice() bool { return hasNVIDIADeviceIn("/sys/bus/pci/devices") }
func hasNVIDIADeviceIn(root string) bool {
vendors, _ := filepath.Glob(filepath.Join(root, "*", "vendor"))
for _, v := range vendors {
b, err := os.ReadFile(v) //nolint:gosec // sysfs
if err != nil || strings.TrimSpace(string(b)) != "0x10de" {
continue
}
class, _ := os.ReadFile(filepath.Join(filepath.Dir(v), "class")) //nolint:gosec // sysfs
// 0x03xxxx is a display or 3D controller; NVIDIA also makes bridges
// and audio devices.
if strings.HasPrefix(strings.TrimSpace(string(class)), "0x03") {
return true
}
}
return false
}
func hostFromOSRelease(path string) Host { func hostFromOSRelease(path string) Host {
var h Host var h Host
f, err := os.Open(path) //nolint:gosec // a fixed system path f, err := os.Open(path) //nolint:gosec // a fixed system path
@@ -23,3 +23,35 @@ func TestNVIDIADriverFrom(t *testing.T) {
} }
} }
} }
func TestACardOnTheBusIsSeenWithoutADriver(t *testing.T) {
root := t.TempDir()
dev := func(name, vendor, class string) {
d := filepath.Join(root, name)
if err := os.MkdirAll(d, 0o750); err != nil {
t.Fatal(err)
}
_ = os.WriteFile(filepath.Join(d, "vendor"), []byte(vendor+"\n"), 0o600)
_ = os.WriteFile(filepath.Join(d, "class"), []byte(class+"\n"), 0o600)
}
dev("0000:00:1f.0", "0x8086", "0x060100") // the chipset
if hasNVIDIADeviceIn(root) {
t.Fatal("a machine with no NVIDIA device said it had one")
}
dev("0000:01:00.1", "0x10de", "0x040300") // the card's HDMI audio alone is not a GPU
if hasNVIDIADeviceIn(root) {
t.Fatal("NVIDIA's audio device was taken for a GPU")
}
dev("0000:01:00.0", "0x10de", "0x030000")
if !hasNVIDIADeviceIn(root) {
t.Fatal("a VGA controller of NVIDIA's was not seen")
}
h := Host{NVIDIADevice: true}
if !h.CardWithoutDriver() {
t.Error("a card with no driver version is a card without a driver")
}
h.NVIDIADriver = "580.82.07"
if h.CardWithoutDriver() {
t.Error("a card with a driver was reported as without one")
}
}
+28 -5
View File
@@ -28,9 +28,22 @@ const (
minBuffer = 256 << 20 minBuffer = 256 << 20
) )
// MeasureMemoryBandwidth is the memory's read rate in GB/s, the best of a // Bandwidth is how fast this machine's memory reads, in GB/s: with every
// few passes; 0 when the machine has too little free memory to measure. // CPU reading, and with one alone. A replica given a few threads of a large
func MeasureMemoryBandwidth() float64 { // machine reads at a few times the second, not at the first: on a server of
// many memory channels one core cannot keep more than a fraction of them
// busy.
type Bandwidth struct {
All, Core float64
}
// coreBuffer is what the one-core passes read: past any cache, and short
// enough that one core reads it in a fraction of a second.
const coreBuffer = 512 << 20
// MeasureMemoryBandwidth is the memory's read rate, the best of a few passes
// each way; zero when the machine has too little free memory to measure.
func MeasureMemoryBandwidth() Bandwidth {
size := int64(bandwidthBuffer) size := int64(bandwidthBuffer)
// Never more than a quarter of what is free: a node already holding // Never more than a quarter of what is free: a node already holding
// models must not be pushed into swap by a measurement. Without room // models must not be pushed into swap by a measurement. Without room
@@ -40,7 +53,7 @@ func MeasureMemoryBandwidth() float64 {
size = min(size, avail/4) size = min(size, avail/4)
} }
if size < max(minBuffer, 8*lastCacheBytes("/sys/devices/system/cpu")) { if size < max(minBuffer, 8*lastCacheBytes("/sys/devices/system/cpu")) {
return 0 return Bandwidth{}
} }
buf := make([]uint64, size/8) buf := make([]uint64, size/8)
for i := range buf { for i := range buf {
@@ -68,7 +81,17 @@ func MeasureMemoryBandwidth() float64 {
best = max(best, float64(size)/s/1e9) best = max(best, float64(size)/s/1e9)
} }
} }
return best // And one core alone, over a part of the same buffer.
part := buf[:min(len(buf), coreBuffer/8)]
core := 0.0
for pass := 0; pass < 3; pass++ {
start := time.Now()
sink(sum(part))
if s := time.Since(start).Seconds(); s > 0 {
core = max(core, float64(len(part)*8)/s/1e9)
}
}
return Bandwidth{All: best, Core: min(core, best)}
} }
// sum reads every word, eight at a time so the loop is not what limits it. // sum reads every word, eight at a time so the loop is not what limits it.
+4 -4
View File
@@ -12,11 +12,11 @@ func TestMemoryBandwidthIsMeasured(t *testing.T) {
} }
bw := MeasureMemoryBandwidth() bw := MeasureMemoryBandwidth()
// Any machine running this reads more than a gigabyte a second and less // Any machine running this reads more than a gigabyte a second and less
// than a few terabytes. // than a few terabytes, and one core no more than all of them.
if bw < 1 || bw > 5000 { if bw.All < 1 || bw.All > 5000 || bw.Core < 0.5 || bw.Core > bw.All {
t.Errorf("measured %.1f GB/s", bw) t.Errorf("measured %.1f GB/s, %.1f on one core", bw.All, bw.Core)
} }
t.Logf("%.1f GB/s", bw) t.Logf("%.1f GB/s, %.1f on one core", bw.All, bw.Core)
} }
func TestTheLastCacheIsReadFromSys(t *testing.T) { func TestTheLastCacheIsReadFromSys(t *testing.T) {
+38
View File
@@ -10,8 +10,20 @@ import (
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1" llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall" "gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
) )
// slurmPorts are Slurm's on a node that runs any of it: slurmctld's and
// slurmd's, and srun's range for the steps it starts.
var slurmPorts = []firewall.Port{
{From: slurm.CtldPort, To: slurm.DbdPort, What: "slurm: slurmctld, slurmd and slurmdbd"},
{From: slurm.SrunPortsFrom, To: slurm.SrunPortsTo, What: "slurm: srun, for the steps it starts"},
}
// nfsPort is NFSv4's, on the machine that exports the Slurm cluster's
// shared files.
var nfsPort = firewall.Port{From: slurm.NFSPort, To: slurm.NFSPort, What: "nfs: the Slurm cluster's shared files"}
// applyFirewall keeps llmgw's firewall on this node as the cluster's // applyFirewall keeps llmgw's firewall on this node as the cluster's
// settings say: the engines' ports and the node exporter's for the networks // settings say: the engines' ports and the node exporter's for the networks
// the node reaches its control planes on, and those named, or no table at // the node reaches its control planes on, and those named, or no table at
@@ -19,6 +31,7 @@ import (
func (a *Agent) applyFirewall(m *llmgwv1.Firewall) { func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
a.fwMu.Lock() a.fwMu.Lock()
defer a.fwMu.Unlock() defer a.fwMu.Unlock()
a.fwLast = m
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel() defer cancel()
r := a.fwRunner r := a.fwRunner
@@ -55,6 +68,14 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
if from > 0 && to >= from { if from > 0 && to >= from {
ports = append(ports, firewall.Engines(from, to)) ports = append(ports, firewall.Engines(from, to))
} }
// Once there, they stay: the ruleset keeps every port it held, so a
// node that leaves Slurm keeps them closed to everyone but the cluster.
if a.slurm != nil && a.slurm.HasPart() {
ports = append(ports, slurmPorts...)
}
if a.slurm != nil && a.slurm.Serves() {
ports = append(ports, nfsPort)
}
changed, err := firewall.Install(ctx, r, firewall.Spec{Ports: ports, Allow: nets}) changed, err := firewall.Install(ctx, r, firewall.Spec{Ports: ports, Allow: nets})
switch { switch {
case err != nil: case err != nil:
@@ -64,6 +85,23 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
} }
} }
// slurmMessage is the leader's word on this node's part in Slurm. A node
// that gets one, or starts or stops exporting the shared files, has the
// firewall applied again, with Slurm's ports and NFS's.
func (a *Agent) slurmMessage(m *llmgwv1.Slurm) {
had, served := a.slurm.HasPart(), a.slurm.Serves()
a.slurm.Apply(m)
if (had || !a.slurm.HasPart()) && served == a.slurm.Serves() {
return
}
a.fwMu.Lock()
last := a.fwLast
a.fwMu.Unlock()
if last != nil && last.GetEnabled() {
go a.applyFirewall(last)
}
}
// controlPlaneIPs are the addresses of the control planes an agent dials. // controlPlaneIPs are the addresses of the control planes an agent dials.
func controlPlaneIPs(ctx context.Context, servers []string) []string { func controlPlaneIPs(ctx context.Context, servers []string) []string {
var out []string var out []string
+27
View File
@@ -6,8 +6,10 @@ import (
"log/slog" "log/slog"
"strings" "strings"
"testing" "testing"
"time"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1" llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/slurmnode"
) )
// fakeShell answers the firewall package's scripts and records what was // fakeShell answers the firewall package's scripts and records what was
@@ -55,3 +57,28 @@ func TestANodeKeepsOrDropsItsFirewallAsTheClusterSays(t *testing.T) {
t.Error("turned off, the table stayed") t.Error("turned off, the table stayed")
} }
} }
func TestANodeGivenAPartInSlurmLetsSlurmInFromTheClusterOnly(t *testing.T) {
sh := &fakeShell{}
a := &Agent{log: slog.New(slog.DiscardHandler), fwRunner: sh, serverList: []string{"https://10.0.0.140:7443"},
slurm: slurmnode.New(nil, nil, "", slog.New(slog.DiscardHandler))}
a.applyFirewall(&llmgwv1.Firewall{Enabled: true, EnginePortsFrom: 20000, EnginePortsTo: 20999})
if strings.Contains(sh.installed, "6817") {
t.Fatalf("Slurm's ports before the node has a part in it:\n%s", sh.installed)
}
a.slurmMessage(&llmgwv1.Slurm{Name: "node05", Node: true, ConfServer: "10.0.0.148:6817"})
deadline := time.Now().Add(5 * time.Second)
for {
a.fwMu.Lock()
got := sh.installed
a.fwMu.Unlock()
if strings.Contains(got, "6817-6819") && strings.Contains(got, "60001-63000") {
break
}
if time.Now().After(deadline) {
t.Fatalf("given a part in Slurm, the rules still lack its ports:\n%s", got)
}
time.Sleep(10 * time.Millisecond)
}
}
+3 -1
View File
@@ -4,12 +4,14 @@ import (
"os" "os"
"testing" "testing"
"time" "time"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
) )
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
// Half a gigabyte read per test binary, and contending with the tests // Half a gigabyte read per test binary, and contending with the tests
// for the CPUs: these never look at the figure. // for the CPUs: these never look at the figure.
measureBandwidth = func() float64 { return 0 } measureBandwidth = func() discovery.Bandwidth { return discovery.Bandwidth{} }
bandwidthDelay = func() time.Duration { return 0 } bandwidthDelay = func() time.Duration { return 0 }
os.Exit(m.Run()) os.Exit(m.Run())
} }
+7
View File
@@ -164,6 +164,13 @@ func (a *Agent) provisionLoop(ctx context.Context) {
// provisionStatus records a change in one engine's installation. // provisionStatus records a change in one engine's installation.
func (a *Agent) provisionStatus(st provision.Status) { func (a *Agent) provisionStatus(st provision.Status) {
if st == (provision.Status{}) {
// A package install that was no engine's has ended.
a.mu.Lock()
delete(a.provStatus, "")
a.mu.Unlock()
return
}
a.mu.Lock() a.mu.Lock()
prev, had := a.provStatus[st.Engine] prev, had := a.provStatus[st.Engine]
a.provStatus[st.Engine] = st a.provStatus[st.Engine] = st
+9 -2
View File
@@ -483,7 +483,8 @@ func (in *Installer) packages(ctx context.Context, byFamily map[string][]string,
return fmt.Errorf("%s is missing and no package manager was found to install it", return fmt.Errorf("%s is missing and no package manager was found to install it",
strings.Join(missing, ", ")) strings.Join(missing, ", "))
} }
in.progress(st, "installing %s with %s", strings.Join(missing, ", "), tool) // The state already says installing.
in.progress(st, "%s, with %s", strings.Join(missing, ", "), tool)
for _, c := range cmds { for _, c := range cmds {
if err := in.runWaitingForLock(ctx, pm.env, c, st); err != nil { if err := in.runWaitingForLock(ctx, pm.env, c, st); err != nil {
return err return err
@@ -497,7 +498,13 @@ func (in *Installer) packages(ctx context.Context, byFamily map[string][]string,
// byFamily is keyed by discovery.Host.Family: "deb" or "rpm". // byFamily is keyed by discovery.Host.Family: "deb" or "rpm".
func (in *Installer) InstallPackages(ctx context.Context, byFamily map[string][]string) error { func (in *Installer) InstallPackages(ctx context.Context, byFamily map[string][]string) error {
var st Status var st Status
return in.packages(ctx, byFamily, &st) err := in.packages(ctx, byFamily, &st)
if st.State == Installing {
// Its progress was no engine's: an empty status takes it off the
// node's page again, done or failed (the caller says which).
in.report(Status{})
}
return err
} }
// lockWait bounds how long a package command waits for another to finish. // lockWait bounds how long a package command waits for another to finish.
@@ -0,0 +1,36 @@
package provision
import (
"context"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
)
// A package install that is no engine's says it is done, so its progress
// does not stay on the node's page (found on the PoC installing Slurm).
func TestInstallPackagesClearsItsProgress(t *testing.T) {
var got []Status
in := &Installer{Host: discovery.Host{OSID: "ubuntu"}, OnStatus: func(s Status) { got = append(got, s) },
Run: func(_ context.Context, _ []string, name string, _ ...string) ([]byte, error) {
if name == "sh" { // the query: not installed
return nil, context.Canceled
}
return nil, nil
}}
if err := in.InstallPackages(context.Background(), map[string][]string{"deb": {"munge"}}); err != nil {
t.Fatal(err)
}
if len(got) < 2 || got[0].State != Installing || got[len(got)-1] != (Status{}) {
t.Fatalf("statuses %+v", got)
}
got = nil
in.Run = func(context.Context, []string, string, ...string) ([]byte, error) { return nil, nil }
if err := in.InstallPackages(context.Background(), map[string][]string{"deb": {"munge"}}); err != nil {
t.Fatal(err)
}
if len(got) != 0 {
t.Fatalf("installed already, yet said %+v", got)
}
}
+32
View File
@@ -94,3 +94,35 @@ func TestWithNoCacheConfiguredNothingIsDeleted(t *testing.T) {
t.Errorf("an agent with no cache directory deleted something: %v", err) t.Errorf("an agent with no cache directory deleted something: %v", err)
} }
} }
// A model of several files is fetched into a directory of its own, and an
// unused one goes whole: it was refused as not empty, and nothing was freed.
func TestAnUnusedSetIsRemovedWhole(t *testing.T) {
dir := t.TempDir()
a := agentWithCache(t, dir)
set := weights.CachePathFor(dir, "https://mirror/coder/v1")
if err := os.MkdirAll(set, 0o755); err != nil {
t.Fatal(err)
}
write(t, filepath.Join(set, "model-00001-of-00002.gguf"))
write(t, filepath.Join(set, ".complete"))
// A link in the cache is taken away, not what it points at.
target := t.TempDir()
write(t, filepath.Join(target, "keep.gguf"))
link := weights.CachePathFor(dir, "https://mirror/linked/v1")
if err := os.Symlink(target, link); err != nil {
t.Fatal(err)
}
a.removeWeights([]string{set, link})
if _, err := os.Stat(set); !os.IsNotExist(err) {
t.Errorf("the set is still there: %v", err)
}
if _, err := os.Lstat(link); !os.IsNotExist(err) {
t.Errorf("the link is still there: %v", err)
}
if _, err := os.Stat(filepath.Join(target, "keep.gguf")); err != nil {
t.Errorf("what the link pointed at was deleted: %v", err)
}
}
+38
View File
@@ -0,0 +1,38 @@
package agent
import (
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
)
// FeatureSlurm is the agent keeping its node's part in Slurm (BACKLOG P).
const FeatureSlurm = "slurm"
// slurmMarker says llmgw started Slurm's daemons on this machine.
var slurmMarker = "/var/lib/llmgw-agent/slurm"
// setSlurmOut is where Slurm's status goes now: this session's queue, or
// nowhere between sessions.
func (a *Agent) setSlurmOut(out chan<- *llmgwv1.AgentMessage) {
a.mu.Lock()
a.slurmOut = out
a.mu.Unlock()
if out != nil {
a.slurm.Resend()
}
}
// sendSlurm queues a status without waiting: a full queue or no session
// drops it, and the next session is sent it again.
func (a *Agent) sendSlurm(st *llmgwv1.SlurmStatus) {
a.mu.Lock()
out := a.slurmOut
a.mu.Unlock()
if out == nil {
return
}
select {
case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_SlurmStatus{SlurmStatus: st}}:
default:
a.log.Warn("Slurm's status not sent: the queue to the control plane is full")
}
}
+214
View File
@@ -0,0 +1,214 @@
package slurmnode
import (
"context"
"errors"
"fmt"
"slices"
"strings"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
// accountsMarker lists what llmgw made in Slurm's accounting, a line each:
// "account NAME" and "user LOGIN NAME". Nothing else is changed or taken
// away: an account a Slurm administrator made is theirs.
func (k *Keeper) accountsMarker() string { return k.marker + "-accounts" }
// maxEnded is the most ended jobs sent at once: a day's on a busy cluster
// would otherwise be a status message of megabytes.
const maxEnded = 2000
// accounts keeps Slurm's accounts and their users as want says, on the
// controller.
func (k *Keeper) accounts(ctx context.Context, want *llmgwv1.Slurm) error {
made := k.madeAccounts()
if !want.GetManageAccounts() {
if len(made) == 0 {
return nil
}
return k.applyAccounts(ctx, nil, made)
}
return k.applyAccounts(ctx, want.GetAccounts(), made)
}
func (k *Keeper) madeAccounts() []string {
b, err := k.host.ReadFile(k.accountsMarker())
if err != nil {
return nil
}
var out []string
for _, l := range strings.Split(strings.TrimSpace(string(b)), "\n") {
if l != "" {
out = append(out, l)
}
}
return out
}
func (k *Keeper) sacctmgr(ctx context.Context, args ...string) error {
out, err := k.host.Run(ctx, nil, "sacctmgr", append([]string{"-i"}, args...)...)
if err != nil {
return fmt.Errorf("sacctmgr %s: %v: %s", strings.Join(args, " "), err, lastLine(out))
}
return nil
}
func (k *Keeper) applyAccounts(ctx context.Context, want []*llmgwv1.SlurmAccount, made []string) error {
out, err := k.host.Run(ctx, nil, "sacctmgr", slurm.AssocFormat...)
if err != nil {
return fmt.Errorf("reading Slurm's accounts: %v: %s", err, lastLine(out))
}
have := map[string]bool{} // "account NAME", "user LOGIN NAME"
for _, a := range slurm.ParseAssociations(string(out)) {
if a.User == "" {
have["account "+a.Account] = true
} else {
have["user "+a.User+" "+a.Account] = true
}
}
keep := map[string]bool{}
var errs []error
ours := slices.Clone(made)
note := func(line string) {
if !slices.Contains(ours, line) {
ours = append(ours, line)
}
}
for _, a := range want {
name, parent := a.GetName(), a.GetParent()
if !validAccount(name) || (parent != "root" && !validAccount(parent)) {
errs = append(errs, fmt.Errorf("the leader sent an account %q under %q", name, parent))
continue
}
line := "account " + name
keep[line] = true
if !have[line] {
k.log.Info("a Slurm account is made, as llmgw's projects say", "account", name, "parent", parent)
if err := k.sacctmgr(ctx, "add", "account", name, "parent="+parent, "description=llmgw"); err != nil {
errs = append(errs, err)
continue
}
note(line)
}
for _, u := range a.GetUsers() {
if !slurm.ValidLogin(u) {
continue
}
ul := "user " + u + " " + name
keep[ul] = true
if have[ul] {
continue
}
if err := k.sacctmgr(ctx, "add", "user", u, "account="+name); err != nil {
errs = append(errs, err)
continue
}
note(ul)
}
}
// Taken away: what llmgw made and no longer wants, users before their
// accounts, children before their parents (made later, listed later).
var left []string
for i := len(ours) - 1; i >= 0; i-- {
line := ours[i]
if keep[line] || !strings.HasPrefix(line, "user ") {
continue
}
f := strings.Fields(line)
if !have[line] {
continue
}
if err := k.removeAssoc(ctx, f[1], f[2], have, keep); err != nil {
errs = append(errs, err)
left = append(left, line)
}
}
for i := len(ours) - 1; i >= 0; i-- {
line := ours[i]
if keep[line] || !strings.HasPrefix(line, "account ") {
continue
}
if have[line] {
k.log.Info("a Slurm account llmgw made is taken away; its jobs' records stay", "account", line[len("account "):])
if err := k.sacctmgr(ctx, "remove", "account", line[len("account "):]); err != nil {
errs = append(errs, err)
left = append(left, line)
}
}
}
var lines []string
for _, l := range ours {
if keep[l] || slices.Contains(left, l) {
lines = append(lines, l)
}
}
if !slices.Equal(lines, made) {
var err error
if len(lines) == 0 {
err = k.host.Remove(k.accountsMarker())
} else {
_, err = k.write(k.accountsMarker(), []byte(strings.Join(lines, "\n")+"\n"), 0o644, "")
}
errs = append(errs, err)
}
return errors.Join(errs...)
}
func validAccount(s string) bool {
return s != "" && len(s) <= 64 && strings.Trim(s, "abcdefghijklmnopqrstuvwxyz0123456789_-") == ""
}
// ended are the jobs that ended in the last day, newest last.
func (k *Keeper) ended(ctx context.Context) ([]*llmgwv1.SlurmJob, error) {
out, err := k.host.Run(ctx, nil, "sacct", slurm.SacctArgs...)
if err != nil {
return nil, fmt.Errorf("sacct: %v: %s", err, lastLine(out))
}
jobs, err := slurm.ParseJobs(string(out))
if err != nil {
return nil, err
}
if len(jobs) > maxEnded {
jobs = jobs[len(jobs)-maxEnded:]
}
ps := make([]*llmgwv1.SlurmJob, 0, len(jobs))
for _, j := range jobs {
ps = append(ps, &llmgwv1.SlurmJob{Id: j.ID, User: j.User, Account: j.Account, Partition: j.Partition,
State: j.State, StartUnix: j.Start.Unix(), EndUnix: j.End.Unix(), ElapsedSeconds: int64(j.Elapsed.Seconds()),
Cpus: int32(j.CPUs), Gpus: int32(j.GPUs)}) //nolint:gosec // counts of a node's
}
return ps, nil
}
// removeAssoc takes login off account. Slurm refuses to take away a
// user's default account, so another of theirs is made the default first,
// one llmgw keeps if there is one; with none left the user goes from the
// accounting entirely.
func (k *Keeper) removeAssoc(ctx context.Context, login, account string, have, keep map[string]bool) error {
var others []string
for line := range have {
f := strings.Fields(line)
if len(f) == 3 && f[0] == "user" && f[1] == login && f[2] != account {
others = append(others, f[2])
}
}
if len(others) == 0 {
return k.sacctmgr(ctx, "remove", "user", login)
}
slices.SortFunc(others, func(a, b string) int {
ka, kb := keep["user "+login+" "+a], keep["user "+login+" "+b]
switch {
case ka && !kb:
return -1
case kb && !ka:
return 1
}
return strings.Compare(a, b)
})
if err := k.sacctmgr(ctx, "modify", "user", login, "set", "defaultaccount="+others[0]); err != nil {
return err
}
return k.sacctmgr(ctx, "remove", "user", login, "account="+account)
}
+83
View File
@@ -0,0 +1,83 @@
package slurmnode
import (
"context"
"slices"
"strings"
"testing"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
func TestAccountsMadeAndTakenAwayOnlyIfLlmgwsOwn(t *testing.T) {
h := newFake()
k := New(h, nil, "/var/lib/llmgw-agent/slurm", nil)
assoc := "sacctmgr " + strings.Join(slurm.AssocFormat, " ")
// root, and an account a Slurm administrator made.
h.outputs[assoc] = "root||\nroot|root|\nphysics||root\n"
m := &llmgwv1.Slurm{ManageAccounts: true, Accounts: []*llmgwv1.SlurmAccount{
{Name: "test", Parent: "root"}, {Name: "test-arge", Parent: "test"},
{Name: "test-arge-deneme", Parent: "test-arge", Users: []string{"ada", "ayse"}},
}}
if err := k.accounts(context.Background(), m); err != nil {
t.Fatal(err)
}
for _, want := range []string{"sacctmgr -i add account test parent=root description=llmgw",
"sacctmgr -i add account test-arge-deneme parent=test-arge description=llmgw",
"sacctmgr -i add user ayse account=test-arge-deneme"} {
if !slices.Contains(h.ran, want) {
t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
}
}
if !strings.Contains(string(h.files[k.accountsMarker()]), "user ada test-arge-deneme") {
t.Errorf("marker %q", h.files[k.accountsMarker()])
}
// Now there; ayse leaves the project, and the team goes.
h.outputs[assoc] = "root||\nphysics||root\ntest||root\ntest-arge||test\ntest-arge-deneme||test-arge\n" +
"test-arge-deneme|ada|test-arge\ntest-arge-deneme|ayse|test-arge\n"
h.ran = nil
m.Accounts = []*llmgwv1.SlurmAccount{{Name: "test", Parent: "root"}}
if err := k.accounts(context.Background(), m); err != nil {
t.Fatal(err)
}
// Neither has another account: they go from the accounting.
want := []string{"sacctmgr -i remove user ayse", "sacctmgr -i remove user ada",
"sacctmgr -i remove account test-arge-deneme", "sacctmgr -i remove account test-arge"}
var got []string
for _, c := range h.ran {
if strings.Contains(c, "remove") || strings.Contains(c, " add ") {
got = append(got, c)
}
}
if !slices.Equal(got, want) {
t.Errorf("ran:\n%s\nwant:\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
}
if slices.ContainsFunc(h.ran, func(c string) bool { return strings.Contains(c, "physics") }) {
t.Error("touched an account llmgw did not make")
}
// One still on another account: that one is made the default first.
h.outputs[assoc] = "root||\nphysics||root\nphysics|ada|\ntest||root\nx||test\nx|ada|\n"
h.files[k.accountsMarker()] = []byte("account test\naccount x\nuser ada x\n")
h.ran = nil
m.Accounts = []*llmgwv1.SlurmAccount{{Name: "test", Parent: "root"}}
if err := k.accounts(context.Background(), m); err != nil {
t.Fatal(err)
}
if i, j := slices.Index(h.ran, "sacctmgr -i modify user ada set defaultaccount=physics"),
slices.Index(h.ran, "sacctmgr -i remove user ada account=x"); i < 0 || j < i {
t.Errorf("ran:\n%s", strings.Join(h.ran, "\n"))
}
// Off: what llmgw made goes.
h.outputs[assoc] = "root||\nphysics||root\ntest||root\n"
h.ran = nil
if err := k.accounts(context.Background(), &llmgwv1.Slurm{}); err != nil {
t.Fatal(err)
}
if !slices.Contains(h.ran, "sacctmgr -i remove account test") || h.files[k.accountsMarker()] != nil {
t.Errorf("not taken away:\n%s", strings.Join(h.ran, "\n"))
}
}
+198
View File
@@ -0,0 +1,198 @@
package slurmnode
import (
"context"
"fmt"
"path/filepath"
"runtime"
"strings"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
// cacheDir holds what is fetched for Slurm: enroot's packages, pyxis's
// source and its build.
const cacheDir = "/var/cache/llmgw-agent/slurm"
// pyxisMarker is the pyxis version built here, against this Slurm.
func (k *Keeper) pyxisMarker() string { return k.marker + "-pyxis" }
// containers has enroot and pyxis here, set up, when want says so; what
// is there is left when it no longer does (Slurm stops loading pyxis once
// the controller's plugstack.conf goes). Answers the status line, empty
// while either is missing.
func (k *Keeper) containers(ctx context.Context, want *llmgwv1.Slurm) (string, error) {
if !want.GetContainers() {
return "", nil
}
family := k.family()
if family == "" {
return "", fmt.Errorf("neither dpkg nor rpm here: enroot and pyxis come as deb or rpm")
}
if err := k.enroot(ctx, family); err != nil {
return "", fmt.Errorf("enroot: %w", err)
}
if err := k.enrootConf(ctx, want.GetShared().GetPath()); err != nil {
return "", fmt.Errorf("enroot's settings: %w", err)
}
if _, err := k.host.LookPath("nvidia-smi"); err == nil {
if err := k.nvidiaContainer(ctx, family); err != nil {
return "", fmt.Errorf("the GPUs in containers (nvidia-container-cli): %w", err)
}
}
if err := k.pyxis(ctx, family); err != nil {
return "", fmt.Errorf("pyxis: %w", err)
}
return slurm.ContainersStatus(), nil
}
func (k *Keeper) family() string {
if _, err := k.host.LookPath("dpkg"); err == nil {
return "deb"
}
if _, err := k.host.LookPath("rpm"); err == nil {
return "rpm"
}
return ""
}
// enroot installs the pinned release when another, or none, is here.
func (k *Keeper) enroot(ctx context.Context, family string) error {
if k.installed["enroot"] {
return nil
}
if out, err := k.host.Run(ctx, nil, "enroot", "version"); err == nil && strings.TrimSpace(string(out)) == slurm.EnrootVersion {
k.installed["enroot"] = true
return nil
}
pkgs, err := slurm.EnrootPackages(family, runtime.GOARCH)
if err != nil {
return err
}
if k.inst == nil {
return fmt.Errorf("enroot missing, and this agent installs nothing")
}
var files []string
for _, p := range pkgs {
f := filepath.Join(cacheDir, p.Name)
if err := k.host.Download(ctx, p.URL(), f, p.SHA256); err != nil {
return err
}
files = append(files, f)
}
k.log.Info("installing enroot, for Slurm's jobs in containers", "version", slurm.EnrootVersion)
if err := k.inst.InstallPackages(ctx, map[string][]string{family: files}); err != nil {
return err
}
out, err := k.host.Run(ctx, nil, "enroot", "version")
if v := strings.TrimSpace(string(out)); err != nil || v != slurm.EnrootVersion {
return fmt.Errorf("enroot says %q after installing %s: %v", v, slurm.EnrootVersion, err)
}
k.installed["enroot"] = true
return nil
}
// enrootConf writes enroot's settings and makes its directories.
func (k *Keeper) enrootConf(ctx context.Context, shared string) error {
if _, err := k.write(slurm.EnrootConfPath, []byte(slurm.EnrootConf), 0o644, ""); err != nil {
return err
}
changed, err := k.write(slurm.EnrootTmpfiles, []byte(slurm.EnrootTmpfilesConf), 0o644, "")
if err != nil {
return err
}
if changed {
if out, err := k.host.Run(ctx, nil, "systemd-tmpfiles", "--create", slurm.EnrootTmpfiles); err != nil {
return fmt.Errorf("systemd-tmpfiles: %v: %s", err, lastLine(out))
}
}
for _, d := range []string{slurm.EnrootLib + "/cache", slurm.EnrootLib + "/data"} {
if _, err := k.host.Run(ctx, nil, "test", "-d", d); err == nil {
continue
}
// Each user makes their own beneath, as in /tmp.
if out, err := k.host.Run(ctx, nil, "install", "-d", "-m", "1777", d); err != nil {
return fmt.Errorf("making %s: %v: %s", d, err, lastLine(out))
}
}
if shared == "" {
return k.host.Remove(slurm.EnrootMounts)
}
_, err = k.write(slurm.EnrootMounts, []byte(slurm.EnrootSharedMount(shared)), 0o644, "")
return err
}
// nvidiaContainer installs nvidia-container-cli, which enroot's hook uses
// to give a container the job's GPUs, from NVIDIA's repository.
func (k *Keeper) nvidiaContainer(ctx context.Context, family string) error {
if k.installed["nvidia-container-cli"] {
return nil
}
if _, err := k.host.LookPath("nvidia-container-cli"); err == nil {
k.installed["nvidia-container-cli"] = true
return nil
}
if k.inst == nil {
return fmt.Errorf("nvidia-container-cli missing, and this agent installs nothing")
}
switch family {
case "deb":
if err := k.host.Download(ctx, slurm.NvidiaContainerKeyURL, slurm.NvidiaContainerKeyPath,
slurm.NvidiaContainerKeySHA256); err != nil {
return err
}
if _, err := k.write(slurm.NvidiaContainerAptList, []byte(slurm.NvidiaContainerApt), 0o644, ""); err != nil {
return err
}
case "rpm":
if _, err := k.write(slurm.NvidiaContainerYumRepo, []byte(slurm.NvidiaContainerYum), 0o644, ""); err != nil {
return err
}
}
k.log.Info("installing nvidia-container-cli, so containers get the job's GPUs")
if err := k.inst.InstallPackages(ctx, slurm.NvidiaContainerPackages); err != nil {
return err
}
if _, err := k.host.LookPath("nvidia-container-cli"); err != nil {
return fmt.Errorf("still missing after installing libnvidia-container-tools")
}
k.installed["nvidia-container-cli"] = true
return nil
}
// pyxis builds the pinned pyxis against this machine's Slurm and installs
// it, when it has not been, or was for another version.
func (k *Keeper) pyxis(ctx context.Context, family string) error {
if b, err := k.host.ReadFile(k.pyxisMarker()); err == nil && strings.TrimSpace(string(b)) == slurm.PyxisVersion {
if _, err := k.host.Run(ctx, nil, "test", "-f", slurm.PyxisPlugin); err == nil {
return nil
}
}
if k.inst == nil {
return fmt.Errorf("pyxis missing, and this agent installs nothing")
}
if err := k.inst.InstallPackages(ctx, map[string][]string{family: slurm.PyxisBuildPackages[family]}); err != nil {
return fmt.Errorf("installing what builds it: %w", err)
}
tarball := filepath.Join(cacheDir, "pyxis-"+slurm.PyxisVersion+".tar.gz")
if err := k.host.Download(ctx, slurm.PyxisSource, tarball, slurm.PyxisSHA256); err != nil {
return err
}
src := filepath.Join(cacheDir, "pyxis-"+slurm.PyxisVersion)
k.log.Info("building pyxis against this machine's Slurm", "version", slurm.PyxisVersion)
for _, c := range [][]string{
{"rm", "-rf", src},
{"tar", "-xzf", tarball, "-C", cacheDir},
{"make", "-C", src, "install"},
} {
if out, err := k.host.Run(ctx, nil, c[0], c[1:]...); err != nil {
return fmt.Errorf("%s: %v: %s", strings.Join(c, " "), err, lastLine(out))
}
}
if _, err := k.host.Run(ctx, nil, "test", "-f", slurm.PyxisPlugin); err != nil {
return fmt.Errorf("built, yet %s is not there", slurm.PyxisPlugin)
}
_, err := k.write(k.pyxisMarker(), []byte(slurm.PyxisVersion+"\n"), 0o644, "")
return err
}
@@ -0,0 +1,92 @@
package slurmnode
import (
"context"
"runtime"
"slices"
"strings"
"testing"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
func TestContainersInstalledFromPinnedReleases(t *testing.T) {
if _, err := slurm.EnrootPackages("deb", runtime.GOARCH); err != nil {
t.Skip(err)
}
h := newFake()
h.bins["dpkg"], h.bins["nvidia-smi"] = true, true
inst := &fakeInst{host: h}
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
// enroot is not here until installed, and then says its version.
h.fail["enroot version"] = true
h.fail["test -d "+slurm.EnrootLib+"/cache"] = true
h.fail["test -f "+slurm.PyxisPlugin] = true
m := nodeMsg(true)
m.Containers = true
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Nfs: "10.0.0.11:/shared"}
st := k.Reconcile(context.Background(), m)
if st.GetContainers() != "" || !strings.Contains(st.GetError(), "containers: enroot") {
t.Fatalf("ready with enroot saying nothing: %v", st)
}
pkgs, _ := slurm.EnrootPackages("deb", runtime.GOARCH)
for _, p := range pkgs {
if !slices.Contains(h.ran, "download "+p.URL()+" "+cacheDir+"/"+p.Name+" "+p.SHA256) {
t.Errorf("%s not fetched with its digest:\n%s", p.Name, strings.Join(h.ran, "\n"))
}
}
if len(inst.got["deb"]) != 2 || !strings.HasPrefix(inst.got["deb"][0], cacheDir) {
t.Errorf("installed %v", inst.got)
}
delete(h.fail, "enroot version")
h.outputs["enroot version"] = slurm.EnrootVersion + "\n"
h.bins["nvidia-container-cli"] = false
st = k.Reconcile(context.Background(), m)
if !strings.Contains(st.GetError(), "nvidia-container-cli") {
t.Errorf("a GPU machine without nvidia-container-cli is ready: %v", st)
}
if string(h.files[slurm.NvidiaContainerAptList]) != slurm.NvidiaContainerApt ||
!slices.Contains(h.ran, "download "+slurm.NvidiaContainerKeyURL+" "+slurm.NvidiaContainerKeyPath+" "+slurm.NvidiaContainerKeySHA256) {
t.Errorf("NVIDIA's repository not added with its key checked")
}
h.bins["nvidia-container-cli"] = true
st = k.Reconcile(context.Background(), m)
// Built, but the plugin is still not there.
if st.GetContainers() != "" || !strings.Contains(st.GetError(), "pyxis") {
t.Fatalf("status %v", st)
}
for _, want := range []string{"tar -xzf " + cacheDir + "/pyxis-" + slurm.PyxisVersion + ".tar.gz -C " + cacheDir,
"make -C " + cacheDir + "/pyxis-" + slurm.PyxisVersion + " install", "install -d -m 1777 " + slurm.EnrootLib + "/cache"} {
if !slices.Contains(h.ran, want) {
t.Errorf("did not run %q", want)
}
}
delete(h.fail, "test -f "+slurm.PyxisPlugin)
st = k.Reconcile(context.Background(), m)
if st.GetContainers() != slurm.ContainersStatus() || st.GetError() != "" {
t.Fatalf("status %v", st)
}
if string(h.files[slurm.EnrootConfPath]) != slurm.EnrootConf ||
!strings.Contains(string(h.files[slurm.EnrootMounts]), "/shared /shared none") {
t.Errorf("enroot's settings not written")
}
// Done: nothing fetched or built again.
h.ran = nil
k.Reconcile(context.Background(), m)
for _, c := range h.ran {
if strings.HasPrefix(c, "download") || strings.HasPrefix(c, "make") {
t.Errorf("ran %q again", c)
}
}
// Off: nothing said, nothing taken away.
m.Containers = false
if st := k.Reconcile(context.Background(), m); st.GetContainers() != "" {
t.Errorf("status %v", st)
}
}
+170
View File
@@ -0,0 +1,170 @@
package slurmnode
import (
"bufio"
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
// mariaDBConfDirs are where each family's MariaDB reads its settings: Debian
// and Ubuntu's, then Red Hat's.
var mariaDBConfDirs = []string{"/etc/mysql/mariadb.conf.d", "/etc/my.cnf.d"}
const mariaDBConfName = "90-llmgw-slurm.cnf"
// acctSQLPath is where the database's statements are put for the client to
// read, never on its command line: they hold the password.
const acctSQLPath = "/run/llmgw-slurm-acct.sql"
// accounting keeps slurmdbd on the controller: MariaDB on this machine
// alone, the database and its user, slurmdbd on them, and the cluster in it.
// Before slurmctld, which reports to it from its first start.
func (k *Keeper) accounting(ctx context.Context, key []byte, files map[string]string) error {
password := slurm.DbdPassword(key)
changed, err := k.mariaDBSettings(ctx)
if err != nil {
return err
}
switch {
case changed:
k.log.Info("MariaDB, for Slurm's accounting, starts on this machine alone")
if err := k.systemctl(ctx, "enable", "--now", "mariadb", "&&restart"); err != nil {
return err
}
case !k.active(ctx, "mariadb"):
if err := k.systemctl(ctx, "enable", "--now", "mariadb"); err != nil {
return err
}
}
client, err := k.mariaDBClient()
if err != nil {
return err
}
if err := k.waitFor(ctx, "MariaDB", client, "-e", "SELECT 1"); err != nil {
return err
}
// The database and its user, once each time the agent starts or the
// password is new: cheap, and a database restored from elsewhere gets
// its user back.
sum := sha256.Sum256([]byte(password))
if mark := hex.EncodeToString(sum[:]); k.acctDone != mark {
if err := k.host.WriteFile(acctSQLPath, []byte(slurm.AcctSQL(password)), 0o600, ""); err != nil {
return err
}
out, err := k.host.Run(ctx, nil, client, "-e", "source "+acctSQLPath)
_ = k.host.Remove(acctSQLPath)
if err != nil {
return fmt.Errorf("making Slurm's database: %v: %s", err, lastLine(out))
}
k.acctDone = mark
}
out, err := k.host.Run(ctx, nil, "hostname", "-s")
host := strings.TrimSpace(string(out))
if err != nil || host == "" {
return fmt.Errorf("this machine's short hostname: %v: %s", err, lastLine(out))
}
changed, err = k.write(slurm.DbdConfPath, []byte(slurm.DbdConf(host, password)), 0o600, "slurm")
if err != nil {
return err
}
switch {
case changed:
k.log.Info("slurmdbd starts again on new settings")
if err := k.systemctl(ctx, "enable", "--now", "slurmdbd", "&&restart"); err != nil {
return err
}
case !k.active(ctx, "slurmdbd"):
if err := k.systemctl(ctx, "enable", "--now", "slurmdbd"); err != nil {
return err
}
}
if err := k.waitFor(ctx, "slurmdbd", "sacctmgr", "-n", "-P", "show", "cluster", "format=cluster"); err != nil {
return err
}
name := clusterName(files["slurm.conf"])
if name == "" || k.clusterAdded == name {
return nil
}
out, err = k.host.Run(ctx, nil, "sacctmgr", "-n", "-P", "show", "cluster", "format=cluster")
if err != nil {
return fmt.Errorf("sacctmgr show cluster: %v: %s", err, lastLine(out))
}
for _, l := range strings.Split(string(out), "\n") {
if strings.TrimSpace(l) == name {
k.clusterAdded = name
return nil
}
}
k.log.Info("the cluster is added to Slurm's accounting", "cluster", name)
if out, err := k.host.Run(ctx, nil, "sacctmgr", "-i", "add", "cluster", name); err != nil {
return fmt.Errorf("sacctmgr add cluster %s: %v: %s", name, err, lastLine(out))
}
k.clusterAdded = name
return nil
}
// mariaDBSettings keeps MariaDB to this machine, when llmgw installed it;
// one that was here before llmgw keeps its own settings, and only gets the
// database and its user.
func (k *Keeper) mariaDBSettings(ctx context.Context) (bool, error) {
if _, err := k.host.ReadFile(k.mariaDBMarker()); err != nil {
return false, nil
}
for _, d := range mariaDBConfDirs {
if _, err := k.host.Run(ctx, nil, "test", "-d", d); err == nil {
return k.write(d+"/"+mariaDBConfName, []byte(slurm.MariaDBConf), 0o644, "")
}
}
return false, fmt.Errorf("MariaDB has none of %s to put its settings in", strings.Join(mariaDBConfDirs, ", "))
}
// mariaDBMarker is there when llmgw installed MariaDB on this machine.
func (k *Keeper) mariaDBMarker() string { return k.marker + "-mariadb" }
// mariaDBClient is MariaDB's command-line client, under either of its
// names.
func (k *Keeper) mariaDBClient() (string, error) {
for _, c := range []string{"mariadb", "mysql"} {
if _, err := k.host.LookPath(c); err == nil {
return c, nil
}
}
return "", errors.New("MariaDB's client (mariadb or mysql) is not installed")
}
// waitFor runs a command until it succeeds, half a minute at most.
func (k *Keeper) waitFor(ctx context.Context, what, name string, args ...string) error {
var last []byte
for range 30 {
out, err := k.host.Run(ctx, nil, name, args...)
if err == nil {
return nil
}
last = out
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(pingWait):
}
}
return fmt.Errorf("%s did not answer after starting: %s", what, lastLine(last))
}
// clusterName is ClusterName in slurm.conf.
func clusterName(conf string) string {
sc := bufio.NewScanner(strings.NewReader(conf))
for sc.Scan() {
if v, ok := strings.CutPrefix(strings.TrimSpace(sc.Text()), "ClusterName="); ok {
return strings.TrimSpace(v)
}
}
return ""
}
+150
View File
@@ -0,0 +1,150 @@
package slurmnode
import (
"context"
"fmt"
"path/filepath"
"strings"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
// What a machine says of the shared path.
const (
SharedMounted = "mounted"
SharedExported = "exported"
SharedMissing = "missing"
)
// shared keeps the shared path as want says: exported from here, mounted
// from an export, or looked for where the site mounts it. Answers how it
// is.
func (k *Keeper) shared(ctx context.Context, want *llmgwv1.Slurm) (string, error) {
sh := want.GetShared()
path := sh.GetPath()
k.unshare(ctx, path, sh.GetServe(), sh.GetNfs())
if path == "" {
return "", nil
}
if !strings.HasPrefix(path, "/") || strings.Contains(path, "..") || strings.ContainsAny(path, " \n\t\"'\\$`") {
return SharedMissing, fmt.Errorf("the leader sent a path %q", path)
}
switch {
case sh.GetServe():
if err := k.host.MkdirAll(path, 0o755, ""); err != nil {
return SharedMissing, err
}
changed, err := k.write(slurm.ExportsPath, []byte(slurm.Exports(path, sh.GetClients())), 0o644, "")
if err != nil {
return SharedMissing, err
}
if !k.active(ctx, "nfs-server") {
k.log.Info("NFS starts on this machine, exporting the Slurm cluster's shared files", "path", path)
if err := k.systemctl(ctx, "enable", "--now", "nfs-server"); err != nil {
return SharedMissing, err
}
changed = true
}
if changed {
if out, err := k.host.Run(ctx, nil, "exportfs", "-ra"); err != nil {
return SharedMissing, fmt.Errorf("exportfs -ra: %v: %s", err, lastLine(out))
}
}
return SharedExported, nil
case sh.GetNfs() != "":
nfs := sh.GetNfs()
if strings.ContainsAny(nfs, " \n\t\"'\\$`") || !strings.Contains(nfs, ":/") {
return SharedMissing, fmt.Errorf("the leader sent an export %q", nfs)
}
if err := k.host.MkdirAll(path, 0o755, ""); err != nil {
return SharedMissing, err
}
unit := slurm.MountUnitName(path)
changed, err := k.write(filepath.Join(slurm.MountUnitDir, unit), []byte(slurm.MountUnit(nfs, path)), 0o644, "")
if err != nil {
return SharedMissing, err
}
if changed {
k.log.Info("the Slurm cluster's shared files are mounted here", "from", nfs, "at", path)
if err := k.systemctl(ctx, "daemon-reload"); err != nil {
return SharedMissing, err
}
if err := k.systemctl(ctx, "enable", unit); err != nil {
return SharedMissing, err
}
// From another export: a restart mounts the new one, and fails
// while a job still has files open on the old.
if err := k.systemctl(ctx, "restart", unit); err != nil {
return SharedMissing, err
}
} else if !k.active(ctx, unit) {
if err := k.systemctl(ctx, "start", unit); err != nil {
return SharedMissing, err
}
}
}
if _, err := k.host.Run(ctx, nil, "mountpoint", "-q", path); err != nil {
if sh.GetNfs() == "" {
return SharedMissing, fmt.Errorf("%s is not mounted here: the site mounts it (slurm.shared names no export)", path)
}
return SharedMissing, fmt.Errorf("%s is not mounted from %s", path, sh.GetNfs())
}
return SharedMounted, nil
}
// unshare takes away the export or the mount llmgw kept here that want no
// longer has: path is where the files are wanted now, serve and nfs how.
func (k *Keeper) unshare(ctx context.Context, path string, serve bool, nfs string) {
if _, err := k.host.ReadFile(slurm.ExportsPath); err == nil && (!serve || path == "") {
k.log.Info("this machine no longer exports the Slurm cluster's shared files")
_ = k.host.Remove(slurm.ExportsPath)
_, _ = k.host.Run(ctx, nil, "exportfs", "-ra")
}
out, err := k.host.Run(ctx, nil, "systemctl", "list-units", "--all", "--plain", "--no-legend", "--type=mount")
if err != nil {
return
}
keep := ""
if path != "" && nfs != "" && !serve {
keep = slurm.MountUnitName(path)
}
for _, l := range strings.Split(string(out), "\n") {
f := strings.Fields(l)
if len(f) == 0 || f[0] == keep {
continue
}
file := filepath.Join(slurm.MountUnitDir, f[0])
b, err := k.host.ReadFile(file)
if err != nil || !strings.HasPrefix(string(b), "# Written by llmgw (BACKLOG P): the Slurm cluster's shared files.") {
continue
}
k.log.Info("the Slurm cluster's old shared mount is taken away", "unit", f[0])
if err := k.systemctl(ctx, "disable", "--now", f[0]); err != nil {
k.log.Warn("could not unmount the old shared files", "unit", f[0], "err", err)
continue
}
_ = k.host.Remove(file)
_ = k.systemctl(ctx, "daemon-reload")
}
}
// sharedPackages are what the shared path needs here, by the command that
// shows each is installed.
func sharedPackages(want *llmgwv1.Slurm, need map[string][2]string) {
switch sh := want.GetShared(); {
case sh.GetPath() == "":
case sh.GetServe():
need["exportfs"] = [2]string{"nfs-kernel-server", "nfs-utils"}
case sh.GetNfs() != "":
need["mount.nfs"] = [2]string{"nfs-common", "nfs-utils"}
}
}
// Serves is whether this machine exports the shared files, so NFS's port
// is kept for the cluster's networks.
func (k *Keeper) Serves() bool {
k.mu.Lock()
defer k.mu.Unlock()
return k.want.GetShared().GetServe() && k.want.GetShared().GetPath() != ""
}
+171
View File
@@ -0,0 +1,171 @@
package slurmnode
import (
"context"
"slices"
"strings"
"testing"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
func ayse() *llmgwv1.SlurmUser {
return &llmgwv1.SlurmUser{Login: "ayse", Uid: 200000, Name: "Ayşe Y.", Home: "/shared/home/ayse",
SshKeys: []string{"ssh-ed25519 AAAA ayse@laptop"}}
}
func TestTheServerExportsAndMakesUsersWithHomes(t *testing.T) {
h := newFake()
h.active["ssh"] = true
inst := &fakeInst{host: h}
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
m := nodeMsg(false)
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Serve: true, Clients: []string{"10.0.0.22", "10.0.0.21"}}
m.ManageUsers, m.MakeHomes, m.Users = true, true, []*llmgwv1.SlurmUser{ayse()}
h.fail["test -d /shared/home/ayse"] = true
h.files[slurm.PAMSSHD] = []byte(pamStack)
st := k.Reconcile(context.Background(), m)
if st.GetError() != "" {
t.Fatal(st.GetError())
}
if st.GetShared() != SharedExported || st.GetUsers() != 1 {
t.Fatalf("status %v", st)
}
if !slices.Contains(inst.got["deb"], "nfs-kernel-server") {
t.Errorf("packages %v", inst.got)
}
if e := string(h.files[slurm.ExportsPath]); !strings.Contains(e, `"/shared" 10.0.0.21(`) {
t.Errorf("exports:\n%s", e)
}
for _, want := range []string{"systemctl enable --now nfs-server", "exportfs -ra", "groupadd -g 200000 ayse",
"useradd -u 200000 -g 200000 -M -d /shared/home/ayse -s /bin/bash -c Ayşe Y. ayse",
"install -d -m 0700 -o 200000 -g 200000 /shared/home/ayse", "sshd -t", "systemctl reload ssh"} {
if !slices.Contains(h.ran, want) {
t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
}
}
if string(h.files[slurm.AuthorizedKeysDir+"/ayse"]) != "ssh-ed25519 AAAA ayse@laptop\n" ||
string(h.files[k.usersMarker()]) != "ayse\n" || h.files[slurm.SSHDropIn] == nil {
t.Errorf("files: keys %q, marker %q", h.files[slurm.AuthorizedKeysDir+"/ayse"], h.files[k.usersMarker()])
}
// Again, made: nothing is run again.
h.outputs["getent passwd ayse"] = "ayse:x:200000:200000:Ayşe Y.:/shared/home/ayse:/bin/bash\n"
delete(h.fail, "test -d /shared/home/ayse")
h.ran = nil
k.Reconcile(context.Background(), m)
for _, c := range h.ran {
if strings.HasPrefix(c, "useradd") || strings.HasPrefix(c, "usermod") || strings.HasPrefix(c, "exportfs") ||
strings.HasPrefix(c, "install") || c == "systemctl reload ssh" {
t.Errorf("ran %q again", c)
}
}
// Gone from the accounts: taken away, its home left.
m.Users = nil
k.Reconcile(context.Background(), m)
if !slices.Contains(h.ran, "userdel ayse") || h.files[k.usersMarker()] != nil || h.files[slurm.SSHDropIn] != nil ||
h.files[slurm.AuthorizedKeysDir+"/ayse"] != nil {
t.Errorf("not taken away:\n%s", strings.Join(h.ran, "\n"))
}
if slices.ContainsFunc(h.ran, func(c string) bool { return strings.HasPrefix(c, "rm") || strings.Contains(c, "userdel -r") }) {
t.Error("a home was removed")
}
}
func TestAClientMountsAndLeavesOthersUsersAlone(t *testing.T) {
h := newFake()
inst := &fakeInst{host: h}
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
m := nodeMsg(false)
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Nfs: "10.0.0.11:/shared"}
bob := &llmgwv1.SlurmUser{Login: "bob", Uid: 200001, Home: "/shared/home/bob"}
m.ManageUsers, m.Users = true, []*llmgwv1.SlurmUser{ayse(), bob}
// The site's own ayse, with its own UID.
h.outputs["getent passwd ayse"] = "ayse:x:1001:1001::/home/ayse:/bin/bash\n"
st := k.Reconcile(context.Background(), m)
if st.GetShared() != SharedMounted || st.GetUsers() != 1 {
t.Fatalf("status %v", st)
}
if !strings.Contains(st.GetError(), "ayse is already a user here with UID 1001") {
t.Errorf("error %q", st.GetError())
}
if u := string(h.files["/etc/systemd/system/shared.mount"]); !strings.Contains(u, "What=10.0.0.11:/shared") {
t.Fatalf("unit:\n%s", u)
}
for _, want := range []string{"systemctl enable shared.mount", "systemctl restart shared.mount", "mountpoint -q /shared",
"useradd -u 200001 -g 200001 -M -d /shared/home/bob -s /bin/bash -c bob"} {
if !slices.Contains(h.ran, want) {
t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
}
}
if slices.ContainsFunc(h.ran, func(c string) bool { return strings.Contains(c, "ayse") && !strings.HasPrefix(c, "getent") }) ||
slices.ContainsFunc(h.ran, func(c string) bool { return strings.HasPrefix(c, "install -d") }) {
t.Errorf("touched the site's ayse, or made a home:\n%s", strings.Join(h.ran, "\n"))
}
if string(h.files[k.usersMarker()]) != "bob\n" {
t.Errorf("marker %q", h.files[k.usersMarker()])
}
// Not mounted: said.
h.fail["mountpoint -q /shared"] = true
if st := k.Reconcile(context.Background(), m); st.GetShared() != SharedMissing ||
!strings.Contains(st.GetError(), "not mounted from 10.0.0.11:/shared") {
t.Errorf("status %v", st)
}
// No shared path any more: the mount is taken away.
h.outputs["systemctl list-units --all --plain --no-legend --type=mount"] = "shared.mount loaded active mounted llmgw\n"
m.Shared = nil
k.Reconcile(context.Background(), m)
if !slices.Contains(h.ran, "systemctl disable --now shared.mount") || h.files["/etc/systemd/system/shared.mount"] != nil {
t.Errorf("mount kept:\n%s", strings.Join(h.ran, "\n"))
}
}
const pamStack = "@include common-auth\naccount required pam_nologin.so\n@include common-account\n@include common-session\n"
func TestUsersReachANodeOnlyThroughTheirJobs(t *testing.T) {
h := newFake()
inst := &fakeInst{host: h}
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
h.files[slurm.PAMSSHD] = []byte(pamStack)
// The module is not there until its package is installed.
for _, d := range slurm.ModuleDirs {
h.fail["test -f "+d+"/"+slurm.AdoptModule] = true
}
m := nodeMsg(true)
m.ManageUsers, m.Users = true, []*llmgwv1.SlurmUser{ayse()}
k.Reconcile(context.Background(), m)
if !slices.Contains(inst.got["deb"], "libpam-slurm-adopt") {
t.Fatalf("not installed: %v", inst.got)
}
if got := string(h.files[slurm.PAMSSHD]); got != pamStack {
t.Fatalf("lines put in with the module still missing:\n%s", got)
}
delete(h.fail, "test -f "+slurm.ModuleDirs[0]+"/"+slurm.AdoptModule)
st := k.Reconcile(context.Background(), m)
got := string(h.files[slurm.PAMSSHD])
if !strings.Contains(got, "pam_listfile.so item=user sense=allow file="+k.usersMarker()) ||
!strings.Contains(got, "account required "+slurm.AdoptModule) {
t.Fatalf("not held to their jobs (%s):\n%s", st.GetError(), got)
}
// The controller is where they log in: not held there.
m.Controller = true
m.Files = map[string]string{"slurm.conf": "x"}
k.Reconcile(context.Background(), m)
if string(h.files[slurm.PAMSSHD]) != pamStack {
t.Errorf("held on the controller:\n%s", h.files[slurm.PAMSSHD])
}
// No users of llmgw's any more: the lines go.
m.Controller, m.Files = false, nil
k.Reconcile(context.Background(), m)
m.Users = nil
k.Reconcile(context.Background(), m)
if string(h.files[slurm.PAMSSHD]) != pamStack {
t.Errorf("lines left with no users:\n%s", h.files[slurm.PAMSSHD])
}
}
+747
View File
@@ -0,0 +1,747 @@
// Package slurmnode keeps this machine's part in the Slurm cluster run beside
// llmgw (BACKLOG P), as the leader says: slurmctld on the controller, slurmd
// on a node, munge on both, installed from the machine's own package sources.
//
// What the leader sends is wanted state, kept by Keeper's loop: sent again,
// or checked again a minute later, it changes nothing that is already so.
package slurmnode
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"io/fs"
"log/slog"
"net/http"
"os"
"os/exec"
"os/user"
"path/filepath"
"slices"
"strconv"
"strings"
"sync"
"time"
"google.golang.org/protobuf/proto"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
// Host is what Keeper does to the machine; Local in the agent, a fake in
// tests.
type Host interface {
Run(ctx context.Context, env []string, name string, args ...string) ([]byte, error)
LookPath(name string) (string, error)
ReadFile(path string) ([]byte, error)
// WriteFile writes b at path with mode, owned by owner ("" for root),
// making its directory.
WriteFile(path string, b []byte, mode fs.FileMode, owner string) error
// MkdirAll makes path, owned by owner.
MkdirAll(path string, mode fs.FileMode, owner string) error
Remove(path string) error
// Download fetches url to path, kept only if its SHA-256 is sha; a file
// already there with it is not fetched again.
Download(ctx context.Context, url, path, sha string) error
}
// Installer installs packages by family, "deb" and "rpm".
type Installer interface {
InstallPackages(ctx context.Context, byFamily map[string][]string) error
}
// Reasons llmgw drains a node with; one drained for another reason, by a
// Slurm administrator, is never resumed by llmgw.
const (
ReasonServing = "llmgw: serving models"
reasonPrefix = "llmgw"
)
// DropIn is the unit drop-in that starts slurmd configless.
const DropIn = "/etc/systemd/system/slurmd.service.d/llmgw.conf"
// Keeper keeps the machine as the last Slurm message says.
type Keeper struct {
host Host
inst Installer
log *slog.Logger
marker string
mu sync.Mutex
want *llmgwv1.Slurm
kick chan struct{}
// resend has the next status sent even if unchanged: a new stream's
// control plane may never have had it.
resend bool
// installed is true once the packages were found or put there, so a
// minute's check does not look again.
installed map[string]bool
files string
// acctDone is the digest of the password the database's user was last
// given; clusterAdded the cluster last found in the accounting.
acctDone string
clusterAdded string
}
// New is a Keeper; marker is a file it keeps while it runs Slurm here, so
// an agent started again knows to stop what it started when told it has no
// part any more.
func New(host Host, inst Installer, marker string, log *slog.Logger) *Keeper {
if log == nil {
log = slog.Default()
}
return &Keeper{host: host, inst: inst, log: log, marker: marker,
kick: make(chan struct{}, 1), installed: map[string]bool{}}
}
// Apply is a new message from the leader: kept from now on.
func (k *Keeper) Apply(m *llmgwv1.Slurm) {
k.mu.Lock()
k.want = proto.CloneOf(m)
k.mu.Unlock()
select {
case k.kick <- struct{}{}:
default:
}
}
// HasPart is whether the last message gives this machine a part in Slurm.
func (k *Keeper) HasPart() bool {
k.mu.Lock()
defer k.mu.Unlock()
return k.want.GetController() || k.want.GetNode()
}
// Resend has the status sent again on the next pass, which it starts now.
func (k *Keeper) Resend() {
k.mu.Lock()
k.resend = true
k.mu.Unlock()
select {
case k.kick <- struct{}{}:
default:
}
}
// Every is how often the machine is checked again without a new message.
var Every = time.Minute
// resendAfter is how often an unchanged status is sent anyway.
const resendAfter = 5 * time.Minute
// Run keeps the machine until ctx ends, sending its status when it changes
// and every few minutes while it has a part.
func (k *Keeper) Run(ctx context.Context, send func(*llmgwv1.SlurmStatus)) {
t := time.NewTicker(Every)
defer t.Stop()
var last *llmgwv1.SlurmStatus
var sent time.Time
for {
select {
case <-ctx.Done():
return
case <-k.kick:
case <-t.C:
}
k.mu.Lock()
want, again := k.want, k.resend
k.resend = false
k.mu.Unlock()
if want == nil {
continue
}
st := k.Reconcile(ctx, want)
if st == nil {
last = nil
continue
}
if again || !proto.Equal(st, last) || time.Since(sent) > resendAfter {
send(st)
last, sent = st, time.Now()
}
}
}
// Reconcile makes the machine what want says once, and answers its status;
// nil when it has no part in Slurm.
func (k *Keeper) Reconcile(ctx context.Context, want *llmgwv1.Slurm) *llmgwv1.SlurmStatus {
ctx, cancel := context.WithTimeout(ctx, 20*time.Minute)
defer cancel()
if !want.GetController() && !want.GetNode() {
k.stop(ctx)
return nil
}
st := &llmgwv1.SlurmStatus{}
fail := func(what string, err error) *llmgwv1.SlurmStatus {
st.Error = what + ": " + err.Error()
k.log.Warn("keeping Slurm on this node: "+what, "err", err)
return st
}
if err := k.ensurePackages(ctx, want); err != nil {
return fail("installing Slurm", err)
}
// From the daemon, not scontrol: a client command looks for its files
// even to say its version, and a configless node has none before slurmd
// has fetched them.
daemon := "slurmd"
if !want.GetNode() {
daemon = "slurmctld"
}
out, err := k.host.Run(ctx, nil, daemon, "-V")
if err != nil {
return fail("reading Slurm's version", fmt.Errorf("%v: %s", err, lastLine(out)))
}
v, err := slurm.ParseVersion(string(out))
if err != nil {
return fail("reading Slurm's version", err)
}
st.Version = strings.TrimSpace(string(out))
if i := strings.LastIndexByte(st.Version, ' '); i >= 0 {
st.Version = st.Version[i+1:]
}
if slurm.TooOld(v) {
return fail("Slurm", fmt.Errorf("%d.%02d is older than %d.%02d, the first to run jobs in cgroup v2; "+
"a newer release of the distribution, or Slurm's own packages, has one", v[0], v[1],
slurm.MinVersion[0], slurm.MinVersion[1]))
}
if err := k.host.WriteFile(k.marker, []byte("llmgw runs Slurm here (BACKLOG P)\n"), 0o644, ""); err != nil {
return fail("keeping its marker", err)
}
if err := k.munge(ctx, want.GetMungeKey()); err != nil {
return fail("munge", err)
}
// The shared files, then the users whose homes are on them. Neither
// keeps Slurm from running: what goes wrong is said, and Slurm kept.
var shErr, usErr error
st.Shared, shErr = k.shared(ctx, want)
if shErr != nil {
k.log.Warn("keeping the Slurm cluster's shared files on this node", "err", shErr)
}
if shErr == nil || !want.GetMakeHomes() || want.GetShared().GetPath() == "" {
st.Users, usErr = k.users(ctx, want)
} else {
// Homes made now would be made beside the shared files, not on them.
usErr = errors.New("waiting for the shared files, where the homes are")
}
if usErr != nil {
k.log.Warn("keeping the Slurm users on this node", "err", usErr)
}
// Jobs in containers; Slurm runs without them, and is kept anyway.
var ctErr error
if st.Containers, ctErr = k.containers(ctx, want); ctErr != nil {
k.log.Warn("setting up jobs in containers on this node", "err", ctErr)
}
defer func() {
switch {
case st.Error != "":
case shErr != nil:
st.Error = "shared files: " + shErr.Error()
case usErr != nil:
st.Error = "users: " + usErr.Error()
case ctErr != nil:
st.Error = "containers: " + ctErr.Error()
}
}()
if want.GetController() {
// Accounting first: slurmctld reports to it from its first start.
// Without it slurmctld still runs jobs, a pass of backfill late
// each; so a failure here is said, and slurmctld is kept anyway.
var acctErr error
if acctErr = k.accounting(ctx, want.GetMungeKey(), want.GetFiles()); acctErr != nil {
k.log.Warn("keeping Slurm's accounting on this node", "err", acctErr)
}
st.SlurmdbdRunning = k.active(ctx, "slurmdbd")
defer func() {
if acctErr != nil && st.Error == "" {
st.Error = "slurmdbd: " + acctErr.Error()
}
}()
sum, err := k.controller(ctx, want.GetFiles())
st.FilesSha256 = sum
if err != nil {
return fail("slurmctld", err)
}
st.SlurmctldRunning = k.active(ctx, "slurmctld")
// The accounts, and the jobs that ended, need the accounting.
if st.SlurmdbdRunning && st.SlurmctldRunning {
if err := k.accounts(ctx, want); err != nil {
k.log.Warn("keeping Slurm's accounts", "err", err)
if acctErr == nil {
acctErr = fmt.Errorf("accounts: %w", err)
}
}
if ended, err := k.ended(ctx); err != nil {
k.log.Warn("reading the jobs that ended", "err", err)
} else {
st.Ended = ended
}
}
}
if want.GetNode() {
if err := k.node(ctx, want, st); err != nil {
return fail("slurmd", err)
}
}
return st
}
func (k *Keeper) ensurePackages(ctx context.Context, want *llmgwv1.Slurm) error {
need := map[string][2]string{
"munged": {"munge", "munge"},
"scontrol": {"slurm-client", "slurm"},
}
if want.GetController() {
need["slurmctld"] = [2]string{"slurmctld", "slurm-slurmctld"}
need["slurmdbd"] = [2]string{"slurmdbd", "slurm-slurmdbd"}
need["mariadbd"] = [2]string{"mariadb-server", "mariadb-server"}
}
if want.GetNode() {
need["slurmd"] = [2]string{"slurmd", "slurm-slurmd"}
}
sharedPackages(want, need)
pkgs := map[string][]string{}
for bin, p := range need {
if k.installed[bin] {
continue
}
if _, err := k.host.LookPath(bin); err == nil {
k.installed[bin] = true
continue
}
pkgs["deb"] = append(pkgs["deb"], p[0])
pkgs["rpm"] = append(pkgs["rpm"], p[1])
}
if len(pkgs) == 0 {
return nil
}
slices.Sort(pkgs["deb"])
slices.Sort(pkgs["rpm"])
if k.inst == nil {
return fmt.Errorf("%s missing, and this agent installs nothing", strings.Join(pkgs["deb"], ", "))
}
k.log.Info("installing Slurm, as the cluster's settings say", "packages", pkgs["deb"])
if err := k.inst.InstallPackages(ctx, pkgs); err != nil {
return err
}
if slices.Contains(pkgs["deb"], "mariadb-server") {
// Installed by llmgw: its settings are llmgw's to keep. One that was
// here before is another's, and is left as it is.
if err := k.host.WriteFile(k.mariaDBMarker(), []byte("llmgw installed MariaDB here, for Slurm's accounting\n"),
0o644, ""); err != nil {
return err
}
}
for bin := range need {
if _, err := k.host.LookPath(bin); err != nil {
return fmt.Errorf("%s is still missing after installing %s", bin, strings.Join(pkgs["deb"], ", "))
}
k.installed[bin] = true
}
return nil
}
// munge writes the cluster's key and keeps munged running on it.
func (k *Keeper) munge(ctx context.Context, key []byte) error {
if len(key) < 32 {
return errors.New("the leader sent no key")
}
changed, err := k.write(slurm.MungeKeyPath, key, 0o400, "munge")
if err != nil {
return err
}
if changed {
k.log.Info("munge key written; munge starts again on it")
return k.systemctl(ctx, "enable", "--now", "munge", "&&restart")
}
if !k.active(ctx, "munge") {
return k.systemctl(ctx, "enable", "--now", "munge")
}
return nil
}
// controller writes the files and keeps slurmctld running on them,
// answering their digest.
func (k *Keeper) controller(ctx context.Context, files map[string]string) (string, error) {
if files["slurm.conf"] == "" {
return "", errors.New("the leader sent no slurm.conf")
}
for _, d := range []string{slurm.StateDir, slurm.LogDir} {
if err := k.host.MkdirAll(d, 0o755, "slurm"); err != nil {
return "", err
}
}
names := make([]string, 0, len(files))
for n := range files {
names = append(names, n)
}
slices.Sort(names)
h := sha256.New()
changed := false
for _, n := range names {
if strings.ContainsAny(n, "/\\") || n == "" || n == "." || n == ".." {
return "", fmt.Errorf("the leader sent a file named %q", n)
}
fmt.Fprintf(h, "%s\x00%s\x00", n, files[n])
c, err := k.write(filepath.Join(slurm.ConfDir, n), []byte(files[n]), 0o644, "")
if err != nil {
return "", err
}
changed = changed || c
}
sum := hex.EncodeToString(h.Sum(nil))
if changed || sum != k.files {
if err := k.systemctl(ctx, "enable", "slurmctld"); err != nil {
return sum, err
}
// Nodes added or taken away need a restart before 23.11 and a
// reconfigure after it; a restart does both, and running jobs live
// through it.
k.log.Info("slurmctld starts again on new files")
if err := k.systemctl(ctx, "restart", "slurmctld"); err != nil {
return sum, err
}
if err := k.waitPing(ctx); err != nil {
return sum, err
}
// Hands the configless slurmd the new files.
if out, err := k.host.Run(ctx, nil, "scontrol", "reconfigure"); err != nil {
return sum, fmt.Errorf("scontrol reconfigure: %v: %s", err, lastLine(out))
}
k.files = sum
return sum, nil
}
if !k.active(ctx, "slurmctld") {
return sum, k.systemctl(ctx, "enable", "--now", "slurmctld")
}
return sum, nil
}
func (k *Keeper) waitPing(ctx context.Context) error {
var last []byte
for range 30 {
out, err := k.host.Run(ctx, nil, "scontrol", "ping")
if err == nil && bytes.Contains(out, []byte("UP")) {
return nil
}
last = out
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(pingWait):
}
}
return fmt.Errorf("slurmctld did not answer after starting: %s", lastLine(last))
}
var pingWait = time.Second
// node keeps slurmd running configless and puts the node in or out of the
// controller's hands as lent says.
func (k *Keeper) node(ctx context.Context, want *llmgwv1.Slurm, st *llmgwv1.SlurmStatus) error {
name, server := want.GetName(), want.GetConfServer()
if name == "" || server == "" {
return errors.New("the leader sent no name or controller")
}
if strings.ContainsAny(name+server, " \n\t\"'\\$`") {
return fmt.Errorf("the leader sent a name or controller with spaces or quotes in it")
}
for _, d := range []string{slurm.SpoolDir, slurm.LogDir} {
if err := k.host.MkdirAll(d, 0o755, "slurm"); err != nil {
return err
}
}
bin, err := k.host.LookPath("slurmd")
if err != nil {
return err
}
unit := "# Written by llmgw (BACKLOG P): slurmd fetches its files from slurmctld.\n" +
"[Unit]\nConditionPathExists=\nStartLimitIntervalSec=0\n" +
// --systemd as the packaged unit has it (Type=notify, since 22.05):
// without it systemd waits for a readiness that never comes.
"[Service]\nExecStart=\nExecStart=" + bin + " --systemd --conf-server " + server + " -N " + name + "\n" +
// A node new to the controller is refused until the leader has
// written it into slurm.conf; tried again until it is.
"Restart=on-failure\nRestartSec=15\n"
changed, err := k.write(DropIn, []byte(unit), 0o644, "")
if err != nil {
return err
}
if out, err := k.host.Run(ctx, nil, "slurmd", "-C"); err == nil {
st.Hardware = strings.TrimSpace(string(out))
}
if changed {
if err := k.systemctl(ctx, "daemon-reload"); err != nil {
return err
}
if err := k.systemctl(ctx, "enable", "slurmd"); err != nil {
return err
}
if err := k.systemctl(ctx, "restart", "slurmd"); err != nil {
return err
}
} else if !k.active(ctx, "slurmd") {
if err := k.systemctl(ctx, "enable", "--now", "slurmd"); err != nil {
return err
}
}
st.SlurmdRunning = k.active(ctx, "slurmd")
st.Jobs = k.localJobs(ctx)
env := k.clientEnv()
out, err := k.host.Run(ctx, env, "scontrol", "show", "node", "-o", name)
if err != nil {
// Not in slurm.conf yet: the leader writes it once this node's
// hardware is in.
return nil
}
ns, err := slurm.ParseNodeState(string(out))
if err != nil {
return nil
}
ours := ns.Reason == "" || strings.HasPrefix(ns.Reason, reasonPrefix)
switch {
case want.GetLent() && ns.Drained() && ours:
k.log.Info("this node is lent to Slurm: resumed there")
if out, err := k.host.Run(ctx, env, "scontrol", "update", "NodeName="+name, "State=RESUME"); err != nil {
return fmt.Errorf("resuming the node: %v: %s", err, lastLine(out))
}
case !want.GetLent() && !strings.Contains(ns.State, "DRAIN"):
k.log.Info("this node is llmgw's: drained in Slurm, its jobs left to finish")
if out, err := k.host.Run(ctx, env, "scontrol", "update", "NodeName="+name, "State=DRAIN",
"Reason="+ReasonServing); err != nil {
return fmt.Errorf("draining the node: %v: %s", err, lastLine(out))
}
}
if out, err := k.host.Run(ctx, env, "scontrol", "show", "node", "-o", name); err == nil {
if ns2, err := slurm.ParseNodeState(string(out)); err == nil {
ns = ns2
}
}
st.State, st.Reason = ns.State, ns.Reason
return nil
}
// localJobs counts the job steps running here, from their slurmstepd, not
// from the controller: a job outlives a controller that is down, and is
// what decides whether llmgw may have the cards back. The one slurmstepd
// that is no job's, "slurmstepd infinity" holding slurmd's cgroup v2 scope,
// is not counted. -1 when it cannot tell.
func (k *Keeper) localJobs(ctx context.Context) int32 {
// pgrep exits 1 with no output when there is none.
out, err := k.host.Run(ctx, nil, "pgrep", "-a", "-x", "slurmstepd")
if err != nil && len(bytes.TrimSpace(out)) > 0 {
return -1
}
n := int32(0)
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
f := strings.Fields(line)
if len(f) < 2 || f[len(f)-1] == "infinity" {
continue
}
n++
}
return n
}
// clientEnv points scontrol at slurmd's copy of the files on a node that is
// not the controller, which has none of its own.
func (k *Keeper) clientEnv() []string {
if _, err := k.host.ReadFile(filepath.Join(slurm.ConfDir, "slurm.conf")); err == nil {
return nil
}
return []string{"SLURM_CONF=" + filepath.Join(slurm.SpoolDir, "conf-cache", "slurm.conf")}
}
// stop stops what llmgw started here, if it did.
func (k *Keeper) stop(ctx context.Context) {
if _, err := k.host.ReadFile(k.marker); err != nil {
return
}
k.log.Info("this node has no part in Slurm any more: slurmd, slurmctld and slurmdbd stopped; the packages and MariaDB stay")
// MariaDB stays: it may have been the machine's before, and holds the
// jobs' records.
for _, u := range []string{"slurmd", "slurmctld", "slurmdbd"} {
_ = k.systemctl(ctx, "disable", "--now", u)
}
_ = k.host.Remove(DropIn)
_ = k.systemctl(ctx, "daemon-reload")
k.unshare(ctx, "", false, "")
if err := k.removeUsers(ctx, k.madeUsers(), nil); err != nil {
k.log.Warn("could not take away every Slurm user llmgw made", "err", err)
}
_ = k.host.Remove(k.marker)
k.files, k.acctDone, k.clusterAdded = "", "", ""
}
// write writes a file when it differs, saying whether it did.
func (k *Keeper) write(path string, b []byte, mode fs.FileMode, owner string) (bool, error) {
if old, err := k.host.ReadFile(path); err == nil && bytes.Equal(old, b) {
return false, nil
}
return true, k.host.WriteFile(path, b, mode, owner)
}
// systemctl runs systemctl; a last argument "&&restart" restarts the unit
// after.
func (k *Keeper) systemctl(ctx context.Context, args ...string) error {
restart := false
if n := len(args); n > 0 && args[n-1] == "&&restart" {
args, restart = args[:n-1], true
}
if out, err := k.host.Run(ctx, nil, "systemctl", args...); err != nil {
return fmt.Errorf("systemctl %s: %v: %s", strings.Join(args, " "), err, lastLine(out))
}
if restart {
u := args[len(args)-1]
if out, err := k.host.Run(ctx, nil, "systemctl", "restart", u); err != nil {
return fmt.Errorf("systemctl restart %s: %v: %s", u, err, lastLine(out))
}
}
return nil
}
func (k *Keeper) active(ctx context.Context, unit string) bool {
out, err := k.host.Run(ctx, nil, "systemctl", "is-active", unit)
return err == nil && strings.TrimSpace(string(out)) == "active"
}
func lastLine(b []byte) string {
s := strings.TrimSpace(string(b))
if i := strings.LastIndexByte(s, '\n'); i >= 0 {
s = s[i+1:]
}
return s
}
// Local is the machine the agent runs on.
type Local struct{}
// Run runs a command, with env added to the agent's.
func (Local) Run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
cmd := exec.CommandContext(ctx, name, args...)
if len(env) > 0 {
cmd.Env = append(os.Environ(), env...)
}
return cmd.CombinedOutput()
}
// LookPath finds a command, in the sbin directories too: an agent started
// by systemd has them, one started by hand may not.
func (Local) LookPath(name string) (string, error) {
if p, err := exec.LookPath(name); err == nil {
return p, nil
}
// /usr/libexec: Red Hat's mariadbd.
for _, d := range []string{"/usr/sbin", "/sbin", "/usr/local/sbin", "/usr/libexec"} {
p := filepath.Join(d, name)
if fi, err := os.Stat(p); err == nil && !fi.IsDir() {
return p, nil
}
}
return "", fmt.Errorf("%s is not installed", name)
}
// ReadFile reads a file.
func (Local) ReadFile(path string) ([]byte, error) { return os.ReadFile(path) } //nolint:gosec // our paths
// WriteFile writes a file whole, through a temporary one beside it.
func (Local) WriteFile(path string, b []byte, mode fs.FileMode, owner string) error {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { //nolint:gosec // system directories
return err
}
tmp := path + ".llmgw-new"
if err := os.WriteFile(tmp, b, mode); err != nil {
return err
}
if err := os.Chmod(tmp, mode); err != nil {
return err
}
if err := chown(tmp, owner); err != nil {
_ = os.Remove(tmp)
return err
}
return os.Rename(tmp, path)
}
// Download fetches url to path through a temporary file beside it, kept
// only when its SHA-256 is sha.
func (Local) Download(ctx context.Context, url, path, sha string) error {
if f, err := os.Open(path); err == nil { //nolint:gosec // our paths
h := sha256.New()
_, cerr := io.Copy(h, f)
_ = f.Close()
if cerr == nil && hex.EncodeToString(h.Sum(nil)) == sha {
return nil
}
}
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { //nolint:gosec // system directories
return err
}
ctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return fmt.Errorf("fetching %s: %w", url, err)
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("fetching %s: %s", url, resp.Status)
}
tmp := path + ".llmgw-new"
f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o644) //nolint:gosec // our paths
if err != nil {
return err
}
h := sha256.New()
_, err = io.Copy(io.MultiWriter(f, h), io.LimitReader(resp.Body, 1<<30))
if cerr := f.Close(); err == nil {
err = cerr
}
if err != nil {
_ = os.Remove(tmp)
return fmt.Errorf("fetching %s: %w", url, err)
}
if got := hex.EncodeToString(h.Sum(nil)); got != sha {
_ = os.Remove(tmp)
return fmt.Errorf("%s has SHA-256 %s, not the %s llmgw was built with: not used", url, got, sha)
}
return os.Rename(tmp, path)
}
// MkdirAll makes a directory owned by owner.
func (Local) MkdirAll(path string, mode fs.FileMode, owner string) error {
if err := os.MkdirAll(path, mode); err != nil {
return err
}
return chown(path, owner)
}
// Remove removes a file, and is content when there is none.
func (Local) Remove(path string) error {
if err := os.Remove(path); err != nil && !errors.Is(err, fs.ErrNotExist) {
return err
}
return nil
}
func chown(path, owner string) error {
if owner == "" {
return nil
}
u, err := user.Lookup(owner)
if err != nil {
return fmt.Errorf("user %s: %w (installed with its package)", owner, err)
}
uid, _ := strconv.Atoi(u.Uid)
gid, _ := strconv.Atoi(u.Gid)
return os.Chown(path, uid, gid)
}
+367
View File
@@ -0,0 +1,367 @@
package slurmnode
import (
"context"
"errors"
"io/fs"
"path/filepath"
"slices"
"strings"
"testing"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
type fakeHost struct {
files map[string][]byte
owners map[string]string
bins map[string]bool
active map[string]bool
ran []string
outputs map[string]string
fail map[string]bool
state string
reason string
}
func newFake() *fakeHost {
return &fakeHost{files: map[string][]byte{}, owners: map[string]string{}, bins: map[string]bool{},
active: map[string]bool{}, outputs: map[string]string{
"slurmd -V": "slurm-wlm 23.11.4\n",
"slurmctld -V": "slurm-wlm 23.11.4\n",
"scontrol ping": "Slurmctld(primary) at node06 is UP\n",
"slurmd -C": "NodeName=cuda01 CPUs=16 RealMemory=64000\nUpTime=1-00:00:00\n",
"hostname -s": "node06\n",
}, fail: map[string]bool{}, state: "IDLE+DRAIN", reason: slurm.ReasonNew}
}
func (f *fakeHost) Run(_ context.Context, env []string, name string, args ...string) ([]byte, error) {
c := strings.Join(append([]string{name}, args...), " ")
f.ran = append(f.ran, c)
if f.fail[c] {
return []byte("it failed"), errors.New("exit status 1")
}
switch {
case name == "systemctl" && len(args) >= 1:
u := args[len(args)-1]
switch args[0] {
case "is-active":
if f.active[u] {
return []byte("active\n"), nil
}
return []byte("inactive\n"), errors.New("exit status 3")
case "restart", "start":
f.active[u] = true
case "enable":
if slices.Contains(args, "--now") {
f.active[u] = true
}
case "disable":
f.active[u] = false
}
return []byte(f.outputs[c]), nil
case strings.HasPrefix(c, "scontrol show node"):
return []byte("NodeName=cuda01 Arch=x86_64 State=" + f.state + " Partitions=gpu Reason=" + f.reason + " [root@now]\n"), nil
case strings.HasPrefix(c, "scontrol update"):
if slices.Contains(args, "State=RESUME") {
f.state, f.reason = "IDLE", ""
} else {
f.state, f.reason = "IDLE+DRAIN", ReasonServing
}
return nil, nil
case strings.HasPrefix(c, "pgrep"):
out := "1800358 /usr/sbin/slurmstepd infinity\n" + f.outputs["jobs"]
return []byte(out), nil
}
return []byte(f.outputs[c]), nil
}
func (f *fakeHost) LookPath(name string) (string, error) {
if f.bins[name] {
return "/usr/sbin/" + name, nil
}
return "", errors.New("not found")
}
func (f *fakeHost) ReadFile(path string) ([]byte, error) {
b, ok := f.files[path]
if !ok {
return nil, fs.ErrNotExist
}
return b, nil
}
func (f *fakeHost) WriteFile(path string, b []byte, _ fs.FileMode, owner string) error {
f.files[path] = append([]byte(nil), b...)
f.owners[path] = owner
return nil
}
func (f *fakeHost) MkdirAll(string, fs.FileMode, string) error { return nil }
func (f *fakeHost) Remove(path string) error { delete(f.files, path); return nil }
func (f *fakeHost) Download(_ context.Context, url, path, sha string) error {
c := "download " + url + " " + path + " " + sha
f.ran = append(f.ran, c)
if f.fail[c] || f.fail["download "+url] {
return errors.New("fetch failed")
}
f.files[path] = []byte("fetched " + url)
return nil
}
func (f *fakeHost) count(prefix string) int {
n := 0
for _, c := range f.ran {
if strings.HasPrefix(c, prefix) {
n++
}
}
return n
}
type fakeInst struct {
host *fakeHost
got map[string][]string
}
func (i *fakeInst) InstallPackages(_ context.Context, by map[string][]string) error {
i.got = by
for _, b := range []string{"munged", "scontrol", "slurmd", "slurmctld", "slurmdbd", "mariadbd", "mariadb", "exportfs", "mount.nfs"} {
i.host.bins[b] = true
}
return nil
}
var key = []byte(strings.Repeat("k", 1024))
func nodeMsg(lent bool) *llmgwv1.Slurm {
return &llmgwv1.Slurm{Node: true, Name: "cuda01", ConfServer: "10.0.0.146:6817", MungeKey: key, Lent: lent}
}
func TestNodeInstallsAndStaysDrained(t *testing.T) {
h := newFake()
inst := &fakeInst{host: h}
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
st := k.Reconcile(context.Background(), nodeMsg(false))
if st.GetError() != "" {
t.Fatal(st.GetError())
}
if !slices.Equal(inst.got["deb"], []string{"munge", "slurm-client", "slurmd"}) ||
!slices.Equal(inst.got["rpm"], []string{"munge", "slurm", "slurm-slurmd"}) {
t.Fatalf("installed %v", inst.got)
}
if string(h.files[slurm.MungeKeyPath]) != string(key) || h.owners[slurm.MungeKeyPath] != "munge" {
t.Fatal("the munge key was not written for munge")
}
unit := string(h.files[DropIn])
if !strings.Contains(unit, "ExecStart=/usr/sbin/slurmd --systemd --conf-server 10.0.0.146:6817 -N cuda01\n") ||
!strings.Contains(unit, "ConditionPathExists=\n") {
t.Fatalf("drop-in:\n%s", unit)
}
if !st.GetSlurmdRunning() || st.GetVersion() != "23.11.4" || !strings.HasPrefix(st.GetHardware(), "NodeName=cuda01 CPUs=16") {
t.Fatalf("status %v", st)
}
// Not lent: left drained, with llmgw's first reason.
if h.count("scontrol update") != 0 || st.GetState() != "IDLE+DRAIN" {
t.Fatalf("a node not lent was changed: %v, %v", h.ran, st)
}
if _, ok := h.files["/var/lib/llmgw-agent/slurm"]; !ok {
t.Fatal("no marker")
}
// Again, nothing changed: no install, no restart.
h.ran = nil
inst.got = nil
k.Reconcile(context.Background(), nodeMsg(false))
if inst.got != nil || h.count("systemctl restart") != 0 || h.count("systemctl enable") != 0 {
t.Fatalf("a second pass did work: %v", h.ran)
}
}
func TestLendAndReclaim(t *testing.T) {
h := newFake()
k := New(h, &fakeInst{host: h}, "/m", nil)
st := k.Reconcile(context.Background(), nodeMsg(true))
if h.count("scontrol update NodeName=cuda01 State=RESUME") != 1 || st.GetState() != "IDLE" {
t.Fatalf("a lent node was not resumed: %v %v", h.ran, st)
}
h.outputs["jobs"] = "2001 slurmstepd: [101.batch]\n2002 slurmstepd: [102.0]\n"
st = k.Reconcile(context.Background(), nodeMsg(false))
if h.count("scontrol update NodeName=cuda01 State=DRAIN Reason="+ReasonServing) != 1 {
t.Fatalf("a node taken back was not drained: %v", h.ran)
}
if st.GetJobs() != 2 || !strings.Contains(st.GetState(), "DRAIN") {
t.Fatalf("status %v", st)
}
// Drained already: not drained again.
h.ran = nil
k.Reconcile(context.Background(), nodeMsg(false))
if h.count("scontrol update") != 0 {
t.Fatalf("drained twice: %v", h.ran)
}
}
func TestAdminsDrainIsLeft(t *testing.T) {
h := newFake()
h.reason = "bad DIMM, ticket 42"
k := New(h, &fakeInst{host: h}, "/m", nil)
k.Reconcile(context.Background(), nodeMsg(true))
if h.count("scontrol update") != 0 {
t.Fatalf("llmgw resumed a node a Slurm administrator drained: %v", h.ran)
}
}
func TestClientEnvOnANodeWithoutFiles(t *testing.T) {
h := newFake()
k := New(h, &fakeInst{host: h}, "/m", nil)
if env := k.clientEnv(); len(env) != 1 || !strings.HasSuffix(env[0], "conf-cache/slurm.conf") {
t.Fatalf("env %v", env)
}
h.files[filepath.Join(slurm.ConfDir, "slurm.conf")] = []byte("x")
if env := k.clientEnv(); env != nil {
t.Fatalf("env %v on the controller", env)
}
}
func TestController(t *testing.T) {
h := newFake()
inst := &fakeInst{host: h}
k := New(h, inst, "/m", nil)
files := map[string]string{"slurm.conf": "ClusterName=llmgw\n", "gres.conf": "#\n", "cgroup.conf": "#\n"}
m := &llmgwv1.Slurm{Controller: true, MungeKey: key, Files: files}
st := k.Reconcile(context.Background(), m)
if st.GetError() != "" {
t.Fatal(st.GetError())
}
if !slices.Contains(inst.got["deb"], "slurmctld") || slices.Contains(inst.got["deb"], "slurmd") {
t.Fatalf("installed %v", inst.got)
}
if string(h.files["/etc/slurm/slurm.conf"]) != "ClusterName=llmgw\n" {
t.Fatal("slurm.conf not written")
}
if h.count("systemctl restart slurmctld") != 1 || h.count("scontrol reconfigure") != 1 {
t.Fatalf("slurmctld not started on its files: %v", h.ran)
}
if !st.GetSlurmctldRunning() || len(st.GetFilesSha256()) != 64 {
t.Fatalf("status %v", st)
}
// Accounting: MariaDB on this machine alone, the database, slurmdbd on
// it, the cluster added, all before slurmctld.
if !slices.Contains(inst.got["deb"], "slurmdbd") || !slices.Contains(inst.got["deb"], "mariadb-server") {
t.Fatalf("installed %v", inst.got)
}
if !strings.Contains(string(h.files["/etc/mysql/mariadb.conf.d/90-llmgw-slurm.cnf"]), "bind-address=127.0.0.1") {
t.Fatal("MariaDB, installed by llmgw, not kept to this machine")
}
dbd := string(h.files[slurm.DbdConfPath])
if !strings.Contains(dbd, "DbdHost=node06\n") || !strings.Contains(dbd, "StoragePass="+slurm.DbdPassword(key)+"\n") ||
h.owners[slurm.DbdConfPath] != "slurm" {
t.Fatalf("slurmdbd.conf for %q:\n%s", h.owners[slurm.DbdConfPath], dbd)
}
if _, ok := h.files[acctSQLPath]; ok {
t.Fatal("the database's statements, password and all, were left on disk")
}
if strings.Contains(strings.Join(h.ran, "\n"), slurm.DbdPassword(key)) {
t.Fatal("the password was on a command line")
}
order := strings.Join(h.ran, "\n")
if !strings.Contains(order, "sacctmgr -i add cluster llmgw") ||
strings.Index(order, "slurmdbd") > strings.Index(order, "systemctl restart slurmctld") {
t.Fatalf("accounting not before slurmctld: %v", h.ran)
}
if !st.GetSlurmdbdRunning() {
t.Fatalf("status %v", st)
}
sum := st.GetFilesSha256()
h.ran = nil
k.Reconcile(context.Background(), m)
if h.count("systemctl restart") != 0 {
t.Fatalf("restarted on the same files: %v", h.ran)
}
files["gres.conf"] = "NodeName=cuda01 Name=gpu File=/dev/nvidia0\n"
h.ran = nil
st = k.Reconcile(context.Background(), m)
if h.count("systemctl restart slurmctld") != 1 || st.GetFilesSha256() == sum {
t.Fatalf("a changed file did not restart slurmctld: %v", h.ran)
}
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Controller: true, MungeKey: key,
Files: map[string]string{"slurm.conf": "x", "../evil": "y"}}); !strings.Contains(st.GetError(), "named") {
t.Fatalf("a path in a file name was taken: %v", st)
}
}
func TestTooOld(t *testing.T) {
h := newFake()
h.outputs["slurmd -V"] = "slurm-wlm 21.08.5\n"
k := New(h, &fakeInst{host: h}, "/m", nil)
st := k.Reconcile(context.Background(), nodeMsg(false))
if !strings.Contains(st.GetError(), "older than 22.05") {
t.Fatalf("status %v", st)
}
if _, ok := h.files[slurm.MungeKeyPath]; ok {
t.Fatal("went on with a Slurm too old")
}
}
func TestStop(t *testing.T) {
h := newFake()
k := New(h, &fakeInst{host: h}, "/m", nil)
k.Reconcile(context.Background(), nodeMsg(false))
h.ran = nil
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{}); st != nil {
t.Fatalf("status %v for no part", st)
}
if !slices.Contains(h.ran, "systemctl disable --now slurmd") {
t.Fatalf("slurmd not stopped: %v", h.ran)
}
if _, ok := h.files[DropIn]; ok {
t.Fatal("drop-in left")
}
// Never started here: nothing touched.
h2 := newFake()
k2 := New(h2, &fakeInst{host: h2}, "/m", nil)
k2.Reconcile(context.Background(), &llmgwv1.Slurm{})
if len(h2.ran) != 0 {
t.Fatalf("a machine llmgw never ran Slurm on was touched: %v", h2.ran)
}
}
func TestBadNames(t *testing.T) {
h := newFake()
k := New(h, &fakeInst{host: h}, "/m", nil)
m := nodeMsg(false)
m.Name = "cuda01 -c /etc/evil"
if st := k.Reconcile(context.Background(), m); !strings.Contains(st.GetError(), "spaces") {
t.Fatalf("status %v", st)
}
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Node: true, Name: "a", ConfServer: "b"}); !strings.Contains(st.GetError(), "no key") {
t.Fatalf("status %v", st)
}
}
func TestAMariaDBThatWasThereKeepsItsSettings(t *testing.T) {
h := newFake()
for _, b := range []string{"munged", "scontrol", "slurmctld", "slurmdbd", "mariadbd", "mariadb", "exportfs", "mount.nfs"} {
h.bins[b] = true
}
k := New(h, &fakeInst{host: h}, "/m", nil)
st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Controller: true, MungeKey: key,
Files: map[string]string{"slurm.conf": "ClusterName=llmgw\n"}})
if st.GetError() != "" {
t.Fatal(st.GetError())
}
for p := range h.files {
if strings.HasSuffix(p, ".cnf") {
t.Fatalf("a MariaDB llmgw did not install had its settings changed: %s", p)
}
}
if h.count("mariadb -e source") != 1 || !st.GetSlurmdbdRunning() {
t.Fatalf("the database was not made in the MariaDB that was there: %v", h.ran)
}
}
+319
View File
@@ -0,0 +1,319 @@
package slurmnode
import (
"context"
"errors"
"fmt"
"path/filepath"
"slices"
"strconv"
"strings"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
// usersMarker lists, a line each, the logins llmgw made here: the ones it
// may change and take away. A user that was here before is never touched.
func (k *Keeper) usersMarker() string { return k.marker + "-users" }
// users keeps the Linux users want has, the same login and UID as on every
// other machine, their keys where sshd reads them, and their homes when
// this machine makes them; and takes away those llmgw made that are gone.
// Answers how many it has. A user that cannot be made is said, the others
// are made anyway.
func (k *Keeper) users(ctx context.Context, want *llmgwv1.Slurm) (int32, error) {
made := k.madeUsers()
if !want.GetManageUsers() {
return 0, k.removeUsers(ctx, made, nil)
}
var errs []error
var keep []string
n := int32(0)
for _, u := range want.GetUsers() {
err := k.user(ctx, u, slices.Contains(made, u.GetLogin()), want.GetMakeHomes())
if errors.Is(err, errNotMade) {
errs = append(errs, err)
continue
}
keep = append(keep, u.GetLogin())
if err != nil {
errs = append(errs, err)
continue
}
n++
}
if err := k.keepMade(keep); err != nil {
return n, err
}
if err := k.removeUsers(ctx, made, keep); err != nil {
errs = append(errs, err)
}
if err := k.sshConf(ctx, len(keep) > 0); err != nil {
errs = append(errs, err)
}
// The controller is where they log in and submit from; a node only
// while a job of theirs runs there.
if err := k.nodeLogin(ctx, len(keep) > 0 && want.GetNode() && !want.GetController()); err != nil {
errs = append(errs, err)
}
return n, errors.Join(errs...)
}
// nodeLogin holds the users llmgw made here to their jobs when they log in
// (pam_slurm_adopt), or no longer. Only they are: root and the site's
// administrators log in as before, so a mistake here never locks out the
// people who would put it right.
func (k *Keeper) nodeLogin(ctx context.Context, on bool) error {
old, err := k.host.ReadFile(slurm.PAMSSHD)
if err != nil {
if !on {
return nil
}
return fmt.Errorf("reading %s: %w", slurm.PAMSSHD, err)
}
if on {
if err := k.adoptModule(ctx); err != nil {
return err
}
}
read := func(name string) (string, error) {
b, err := k.host.ReadFile(filepath.Join(filepath.Dir(slurm.PAMSSHD), name))
return string(b), err
}
stack, contained, err := slurm.PAMAdopt(string(old), k.usersMarker(), read, on)
if err != nil {
return err
}
if stack == string(old) {
return nil
}
if on {
k.log.Info("llmgw's Slurm users log in to this node only while a job of theirs runs here",
"in_the_job", contained)
if !contained {
k.log.Warn("pam_systemd could not be skipped for them safely here: their ssh sessions are not held " +
"to their job's cores and cards, only to having one")
}
} else {
k.log.Info("llmgw's Slurm users are no longer held to their jobs when they log in here")
}
// Read by sshd for each login: nothing to reload.
_, err = k.write(slurm.PAMSSHD, []byte(stack), 0o644, "")
return err
}
// adoptModule is there, or is installed.
func (k *Keeper) adoptModule(ctx context.Context) error {
if k.installed[slurm.AdoptModule] {
return nil
}
find := func() bool {
for _, d := range slurm.ModuleDirs {
if _, err := k.host.Run(ctx, nil, "test", "-f", filepath.Join(d, slurm.AdoptModule)); err == nil {
return true
}
}
return false
}
if !find() {
if k.inst == nil {
return fmt.Errorf("%s missing, and this agent installs nothing", slurm.AdoptModule)
}
k.log.Info("installing pam_slurm_adopt, which holds llmgw's Slurm users to their jobs here")
if err := k.inst.InstallPackages(ctx, slurm.AdoptPackages); err != nil {
return err
}
if !find() {
return fmt.Errorf("%s is still missing after installing %s", slurm.AdoptModule,
strings.Join(slurm.AdoptPackages["deb"], ", "))
}
}
k.installed[slurm.AdoptModule] = true
return nil
}
// errNotMade is a user llmgw could not make here, nor took for its own.
var errNotMade = errors.New("not made")
type notMade struct{ msg string }
func (e notMade) Error() string { return e.msg }
func (e notMade) Is(target error) bool { return target == errNotMade }
// user keeps one user; ours is whether llmgw made it here before.
func (k *Keeper) user(ctx context.Context, u *llmgwv1.SlurmUser, ours, makeHome bool) error {
login, uid, home := u.GetLogin(), u.GetUid(), u.GetHome()
if !slurm.ValidLogin(login) || uid < slurm.MinUID || !filepath.IsAbs(home) || strings.Contains(home, "..") {
return notMade{fmt.Sprintf("the leader sent a user %q, UID %d, home %q", login, uid, home)}
}
id := strconv.FormatUint(uint64(uid), 10)
gecos := strings.Map(func(r rune) rune {
if r == ':' || r == ',' || r == '\n' || r < ' ' {
return -1
}
return r
}, u.GetName())
byName, okName := k.passwd(ctx, login)
byID, okID := k.passwd(ctx, id)
switch {
case okName && byName[2] != id:
return notMade{fmt.Sprintf("%s is already a user here with UID %s, not llmgw's %s", login, byName[2], id)}
case !okName && okID:
return notMade{fmt.Sprintf("UID %s is already %s's here, so %s is not made", id, byID[0], login)}
case !okName:
if _, ok := k.group(ctx, login); !ok {
if out, err := k.host.Run(ctx, nil, "groupadd", "-g", id, login); err != nil {
return notMade{fmt.Sprintf("groupadd %s: %v: %s", login, err, lastLine(out))}
}
}
k.log.Info("a Slurm user is made here, as llmgw's accounts say", "login", login, "uid", uid)
if out, err := k.host.Run(ctx, nil, "useradd", "-u", id, "-g", id, "-M", "-d", home, "-s", "/bin/bash",
"-c", gecos, login); err != nil {
return notMade{fmt.Sprintf("useradd %s: %v: %s", login, err, lastLine(out))}
}
case ours && (byName[4] != gecos || byName[5] != home):
if out, err := k.host.Run(ctx, nil, "usermod", "-c", gecos, "-d", home, login); err != nil {
return fmt.Errorf("usermod %s: %v: %s", login, err, lastLine(out))
}
}
keys := strings.Join(u.GetSshKeys(), "\n")
file := filepath.Join(slurm.AuthorizedKeysDir, login)
if keys == "" {
_ = k.host.Remove(file)
} else if _, err := k.write(file, []byte(keys+"\n"), 0o644, ""); err != nil {
return err
}
if makeHome {
return k.home(ctx, login, id, home)
}
return nil
}
// home makes a user's home directory, from /etc/skel, when it is not there.
func (k *Keeper) home(ctx context.Context, login, id, home string) error {
if _, err := k.host.Run(ctx, nil, "test", "-d", home); err == nil {
return nil
}
if err := k.host.MkdirAll(filepath.Dir(home), 0o755, ""); err != nil {
return err
}
k.log.Info("a Slurm user's home is made", "login", login, "home", home)
for _, c := range [][]string{
{"install", "-d", "-m", "0700", "-o", id, "-g", id, home},
{"cp", "-rT", "/etc/skel", home},
{"chown", "-R", id + ":" + id, home},
} {
if out, err := k.host.Run(ctx, nil, c[0], c[1:]...); err != nil {
return fmt.Errorf("making %s's home: %s: %v: %s", login, c[0], err, lastLine(out))
}
}
return nil
}
// removeUsers takes away the users llmgw made here that are not in keep.
// Their homes and files stay; their UID is never given to another.
func (k *Keeper) removeUsers(ctx context.Context, made, keep []string) error {
var errs []error
left := slices.Clone(keep)
for _, login := range made {
if slices.Contains(keep, login) {
continue
}
k.log.Info("a Slurm user llmgw made is taken away; its home stays", "login", login)
_ = k.host.Remove(filepath.Join(slurm.AuthorizedKeysDir, login))
if _, ok := k.passwd(ctx, login); ok {
if out, err := k.host.Run(ctx, nil, "userdel", login); err != nil {
// Still logged in, or running something: tried again.
errs = append(errs, fmt.Errorf("userdel %s: %v: %s", login, err, lastLine(out)))
left = append(left, login)
continue
}
}
}
if !slices.Equal(left, made) {
if err := k.keepMade(left); err != nil {
errs = append(errs, err)
}
}
if len(left) == 0 {
if err := k.sshConf(ctx, false); err != nil {
errs = append(errs, err)
}
if err := k.nodeLogin(ctx, false); err != nil {
errs = append(errs, err)
}
}
return errors.Join(errs...)
}
func (k *Keeper) madeUsers() []string {
b, err := k.host.ReadFile(k.usersMarker())
if err != nil {
return nil
}
return strings.Fields(string(b))
}
func (k *Keeper) keepMade(logins []string) error {
s := slices.Clone(logins)
slices.Sort(s)
s = slices.Compact(s)
if len(s) == 0 {
return k.host.Remove(k.usersMarker())
}
_, err := k.write(k.usersMarker(), []byte(strings.Join(s, "\n")+"\n"), 0o644, "")
return err
}
// sshConf has sshd read the users' keys from AuthorizedKeysDir, or no
// longer.
func (k *Keeper) sshConf(ctx context.Context, on bool) error {
if !on {
if _, err := k.host.ReadFile(slurm.SSHDropIn); err != nil {
return nil
}
_ = k.host.Remove(slurm.SSHDropIn)
return k.reloadSSH(ctx)
}
changed, err := k.write(slurm.SSHDropIn, []byte(slurm.SSHConf), 0o644, "")
if err != nil || !changed {
return err
}
if out, err := k.host.Run(ctx, nil, "sshd", "-t"); err != nil {
_ = k.host.Remove(slurm.SSHDropIn)
return fmt.Errorf("sshd refuses llmgw's setting for the users' keys, so it is not kept: %v: %s", err, lastLine(out))
}
k.log.Info("sshd reads the Slurm users' keys from llmgw too", "dir", slurm.AuthorizedKeysDir)
return k.reloadSSH(ctx)
}
// reloadSSH has sshd read its settings again, under Debian's name for it
// or Red Hat's; one that is not running reads them when it starts.
func (k *Keeper) reloadSSH(ctx context.Context) error {
for _, u := range []string{"ssh", "sshd"} {
if k.active(ctx, u) {
return k.systemctl(ctx, "reload", u)
}
}
return nil
}
// passwd is getent's line for a user, by login or UID, in its fields.
func (k *Keeper) passwd(ctx context.Context, key string) ([]string, bool) {
out, err := k.host.Run(ctx, nil, "getent", "passwd", key)
f := strings.Split(strings.TrimSpace(string(out)), ":")
if err != nil || len(f) < 7 {
return nil, false
}
return f, true
}
func (k *Keeper) group(ctx context.Context, key string) ([]string, bool) {
out, err := k.host.Run(ctx, nil, "getent", "group", key)
f := strings.Split(strings.TrimSpace(string(out)), ":")
if err != nil || len(f) < 3 {
return nil, false
}
return f, true
}
+12 -1
View File
@@ -33,6 +33,17 @@ func CachePathFor(dir, source string) string {
return filepath.Join(dir, hex.EncodeToString(sum[:8])+"-"+name) return filepath.Join(dir, hex.EncodeToString(sum[:8])+"-"+name)
} }
// cacheSource is the source a cache name is computed from: a set's address
// without its trailing slash, as its directory was named when it was fetched.
// Computed from the slash-ended address, every set in the catalog looked like
// nobody's, and a sweep offered 330 GB of models still in use for deletion.
func cacheSource(source string) string {
if IsSet(source) {
return strings.TrimSuffix(source, "/")
}
return source
}
// Owned reports whether a path is one this package created. // Owned reports whether a path is one this package created.
// //
// The dividing line of the whole feature. A file without the cache name was // The dividing line of the whole feature. A file without the cache name was
@@ -65,7 +76,7 @@ func Orphans(have []string, wanted []string) []string {
// By base name rather than full path: the control plane does not know // By base name rather than full path: the control plane does not know
// each node's cache directory, and the name is what identifies the // each node's cache directory, and the name is what identifies the
// file wherever the node keeps it. // file wherever the node keeps it.
keep[filepath.Base(CachePathFor("", src))] = struct{}{} keep[filepath.Base(CachePathFor("", cacheSource(src)))] = struct{}{}
} }
var out []string var out []string
+1 -1
View File
@@ -51,7 +51,7 @@ type manifest struct {
// as a broken model rather than failing to load, so a partial set is left // as a broken model rather than failing to load, so a partial set is left
// unfinished under its temporary name and the next attempt starts again. // unfinished under its temporary name and the next attempt starts again.
func (f *Fetcher) ensureSet(ctx context.Context, source string, allowPickle bool) (string, error) { func (f *Fetcher) ensureSet(ctx context.Context, source string, allowPickle bool) (string, error) {
dir, err := f.cachePath(strings.TrimSuffix(source, "/")) dir, err := f.cachePath(cacheSource(source))
if err != nil { if err != nil {
return "", err return "", err
} }
+17
View File
@@ -74,6 +74,23 @@ func TestASetIsFetchedIntoOneDirectory(t *testing.T) {
} }
} }
func TestASetInTheCatalogIsNotAnOrphan(t *testing.T) {
// The catalog names a set by its address with the trailing slash; the
// fetch names the directory without it. A sweep that computed the name
// from the catalog's spelling offered every set in use for deletion.
f, srv, _ := mirrorOf(t, map[string]string{"model.safetensors": "aaaa", "config.json": "{}"})
dir, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false)
if err != nil {
t.Fatal(err)
}
if got := Orphans([]string{dir}, []string{srv.URL + "/m/v/"}); len(got) != 0 {
t.Errorf("a set the catalog still names was called an orphan: %v", got)
}
if got := Orphans([]string{dir}, nil); len(got) != 1 {
t.Errorf("a set nothing names was not found: %v", got)
}
}
func TestAFinishedSetIsNotFetchedAgain(t *testing.T) { func TestAFinishedSetIsNotFetchedAgain(t *testing.T) {
f, srv, hits := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"}) f, srv, hits := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
+2
View File
@@ -247,6 +247,8 @@ var liveSettings = []string{
"presets", "presets",
// Handed to the reconciler as they change. // Handed to the reconciler as they change.
"on_demand", "on_demand",
// Read by the Slurm service each pass, kicked when they change.
"slurm",
// Read by server init, node add and backups, not by the server. // Read by server init, node add and backups, not by the server.
"backup_recipients", "backup_copy_to", "backup_recipients", "backup_copy_to",
} }
+16 -1
View File
@@ -189,6 +189,8 @@ func newCtlCmd() *cobra.Command {
newMCPCmd(client, &asJSON), newMCPCmd(client, &asJSON),
newKBCmd(client, &asJSON), newKBCmd(client, &asJSON),
newTrainCmd(client, &asJSON), newTrainCmd(client, &asJSON),
newSlurmCmd(client, &asJSON),
newPricesCmd(client, &asJSON),
newConfigCmd(client, &asJSON), newConfigCmd(client, &asJSON),
newAuditCmd(client, &asJSON), newAuditCmd(client, &asJSON),
newNotifyCmd(client, &asJSON), newNotifyCmd(client, &asJSON),
@@ -418,7 +420,7 @@ func newModelsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
} }
fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\t%s\t%s\n", fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\t%s\t%s\n",
name, serving, m.Ready, m.Min, modelStatus(m), m.Class, reserves(m), name, serving, m.Ready, m.Min, modelStatus(m), m.Class, reserves(m),
join(m.Aliases), join(m.Profiles), join(m.Fallbacks)) join(m.Aliases), join(profilesWithCounts(m)), join(m.Fallbacks))
for _, x := range m.Warnings { for _, x := range m.Warnings {
warnings = append(warnings, m.Name+": "+x) warnings = append(warnings, m.Name+": "+x)
} }
@@ -696,3 +698,16 @@ func reserves(m adminapi.ModelView) string {
} }
return out return out
} }
// profilesWithCounts are a model's profiles, each with its replicas when
// they are counted per profile: gpu=7, cpu=4.
func profilesWithCounts(m adminapi.ModelView) []string {
if len(m.ProfileReplicas) == 0 {
return m.Profiles
}
out := make([]string, len(m.Profiles))
for i, p := range m.Profiles {
out[i] = fmt.Sprintf("%s=%d", p, m.ProfileReplicas[p])
}
return out
}
+121
View File
@@ -0,0 +1,121 @@
package cli
import (
"encoding/json"
"fmt"
"net/url"
"strconv"
"text/tabwriter"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
)
func newPricesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
cmd := &cobra.Command{
Use: "prices",
Short: "What each model's tokens cost, from what the cluster costs",
}
var days int
var apply bool
var note string
suggest := &cobra.Command{
Use: "suggest",
Short: "Each model's price from the GPU- and core-hours it held, over the tokens it served",
Long: "Set auth.gpu_hour_cost first: what an hour of one GPU costs, in the units budgets are in (the\n" +
"hardware written off over its years, power and room, the people who run it, over the GPUs and\n" +
"the hours of a month); auth.core_hour_cost likewise for models on CPUs. Each model's cost is\n" +
"the hours its replicas held, idle ones too, kept by the hour on the leader; spread over its\n" +
"tokens, an output token counted as four input ones and a cached prompt token as a tenth of one,\n" +
"it is its price. A model held and not used shows what that cost. --apply writes the prices\n" +
"into auth.pricing, one settings version, as `ctl config set` does.",
Example: " llmgw ctl config set auth.gpu_hour_cost 1.35\n" +
" llmgw ctl prices suggest --days 30\n" +
" llmgw ctl prices suggest --apply --note \"November's costs\"",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
var s adminapi.PriceSuggestions
if err := client().get(cmd.Context(), "/api/v1/prices/suggest?days="+url.QueryEscape(strconv.Itoa(days)), &s); err != nil {
return err
}
w := cmd.OutOrStdout()
if *asJSON {
return json.NewEncoder(w).Encode(s)
}
fmt.Fprintf(w, "cost: %g a GPU-hour, %g a core-hour; the last %d days", s.GPUHourCost, s.CoreHourCost, s.Days)
if !s.RecordedFrom.IsZero() && s.RecordedFrom.After(s.Since) {
fmt.Fprintf(w, " (held kept only since %s)", s.RecordedFrom.Local().Format("2006-01-02 15:04"))
}
fmt.Fprintln(w)
fmt.Fprintln(w, "prices a million tokens: input / output / cached input, in budget units")
tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
fmt.Fprintln(tw, "MODEL\tGPU-H\tCORE-H\tCOST\tTOKENS IN/OUT\tSUGGESTED\tNOW\tNOTE")
var changed []adminapi.PriceSuggestion
for _, m := range s.Models {
sug := "-"
if !m.Price.Zero() {
sug = perMillion(m.Price.PromptPer1k, m.Price.CompletionPer1k, m.Price.CachedPer1k)
if m.Price.PromptPer1k != m.Current.PromptPer1k || m.Price.CompletionPer1k != m.Current.CompletionPer1k ||
m.Price.CachedPer1k != m.Current.CachedPer1k {
changed = append(changed, m)
}
}
fmt.Fprintf(tw, "%s\t%.1f\t%.1f\t%.2f\t%s/%s\t%s\t%s\t%s\n", m.Model, m.GPUHours, m.CoreHours, m.Cost,
count(m.Prompt), count(m.Completion), sug,
perMillion(m.Current.PromptPer1k, m.Current.CompletionPer1k, m.Current.CachedPer1k), m.Note)
}
if err := tw.Flush(); err != nil {
return err
}
if !apply {
if len(changed) > 0 {
fmt.Fprintf(w, "%d model(s) priced otherwise now; --apply writes the suggestions\n", len(changed))
}
return nil
}
if len(changed) == 0 {
fmt.Fprintln(w, "nothing to change")
return nil
}
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
for _, m := range changed {
for k, v := range map[string]int64{"prompt_per_1k": m.Price.PromptPer1k,
"completion_per_1k": m.Price.CompletionPer1k, "cached_per_1k": m.Price.CachedPer1k} {
if err := doc.SetYAML([]string{"auth", "pricing", m.Model, k}, strconv.FormatInt(v, 10)); err != nil {
return err
}
}
}
return nil
})
},
}
suggest.Flags().IntVar(&days, "days", 30, "the window, in days")
suggest.Flags().BoolVar(&apply, "apply", false, "write the suggested prices into auth.pricing")
suggest.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
cmd.AddCommand(suggest)
return cmd
}
// perMillion is micro-units a thousand tokens as budget units a million.
func perMillion(in, out, cached int64) string {
if in == 0 && out == 0 && cached == 0 {
return "-"
}
f := func(v int64) string { return strconv.FormatFloat(float64(v)/1000, 'f', -1, 64) }
return f(in) + " / " + f(out) + " / " + f(cached)
}
func count(n int64) string {
switch {
case n >= 1e9:
return fmt.Sprintf("%.1fB", float64(n)/1e9)
case n >= 1e6:
return fmt.Sprintf("%.1fM", float64(n)/1e6)
case n >= 1e3:
return fmt.Sprintf("%.1fk", float64(n)/1e3)
}
return strconv.FormatInt(n, 10)
}
+480
View File
@@ -0,0 +1,480 @@
package cli
import (
"encoding/json"
"fmt"
"io"
"net/url"
"os"
"slices"
"strings"
"text/tabwriter"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
)
// newSlurmCmd is the Slurm cluster run beside llmgw (BACKLOG P): set up on
// nodes llmgw already has, and nodes handed between the two.
func newSlurmCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
cmd := &cobra.Command{
Use: "slurm",
Short: "A Slurm cluster beside llmgw, for training and other batch work",
Long: "Slurm runs on llmgw's own nodes, installed and kept by their agents: slurmctld on one node,\n" +
"slurmd on those named. A node is llmgw's, drained in Slurm, until it is lent; lent, its\n" +
"replicas move off first, then Slurm may run jobs there. Taken back, Slurm drains it and\n" +
"llmgw has it again when its last job ends. Jobs are submitted on the controller node\n" +
"(sbatch, srun) by its own users, as on any Slurm cluster.",
}
cmd.AddCommand(newSlurmStatusCmd(client, asJSON), newSlurmSetupCmd(client), newSlurmOffCmd(client),
newSlurmListCmd(client, "lend", "Let Slurm run jobs on nodes; llmgw's replicas move off first", true),
newSlurmListCmd(client, "reclaim", "Take nodes back from Slurm; llmgw has them when their jobs end", false),
newSlurmSharedCmd(client), newSlurmUsersCmd(client, asJSON), newSlurmContainersCmd(client))
return cmd
}
func newSlurmContainersCmd(client func() *ctlClient) *cobra.Command {
cmd := &cobra.Command{
Use: "containers",
Short: "Jobs in containers: srun --container-image (pyxis and enroot)",
Long: "On, the agents install enroot and pyxis on every machine of the Slurm cluster (from NVIDIA's\n" +
"releases, at versions llmgw pins and checks) and, on GPU machines, nvidia-container-cli; once\n" +
"every machine has them, Slurm loads pyxis and jobs may run in a container:\n\n" +
" srun --gres=gpu:1 --container-image=nvcr.io#nvidia/pytorch:25.01-py3 python train.py\n\n" +
"The shared path is in every container where it is on the host. Off, Slurm stops loading pyxis;\n" +
"what was installed stays.",
}
var note string
toggle := func(use, short string, on bool) *cobra.Command {
c := &cobra.Command{
Use: use,
Short: short,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
if !on {
if doc.Node([]string{"slurm", "containers"}) != nil {
doc.Unset([]string{"slurm", "containers"})
}
return nil
}
return doc.SetYAML([]string{"slurm", "containers"}, "true")
})
},
}
c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
return c
}
cmd.AddCommand(toggle("on", "Install pyxis and enroot everywhere, and let jobs run in containers", true),
toggle("off", "Slurm stops loading pyxis; what was installed stays", false))
return cmd
}
func newSlurmSharedCmd(client func() *ctlClient) *cobra.Command {
var path, server, nfs, note string
var off bool
c := &cobra.Command{
Use: "shared --path DIR (--server NODE | --nfs HOST:/EXPORT) | --off",
Short: "Files every machine of the Slurm cluster sees at one path",
Long: "Home directories, datasets, the jobs' output and checkpoints, at the same path on the controller\n" +
"and every node. Three ways:\n" +
" --nfs HOST:/EXPORT the site's own NFS export, mounted at --path on every machine;\n" +
" --server NODE an llmgw node (the controller or a node) exports --path over NFS to the\n" +
" others, for a site with no storage of its own: one machine, not highly\n" +
" available, no faster than its disk and network;\n" +
" neither the site mounts --path itself (Lustre, BeeGFS, GPFS); llmgw only looks\n" +
" that it is there.\n" +
"The export answers the Slurm machines alone, NFS's port the cluster's networks. A machine\n" +
"that gives up its mount does so when no job holds files on it.",
Example: " llmgw ctl slurm shared --path /shared --server node06\n" +
" llmgw ctl slurm shared --path /shared --nfs filer.site:/export/hpc\n" +
" llmgw ctl slurm shared --path /lustre/hpc\n" +
" llmgw ctl slurm shared --off",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
if off == (path != "") {
return fmt.Errorf("give --path, or --off")
}
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
if off {
if doc.Node([]string{"slurm", "shared"}) != nil {
doc.Unset([]string{"slurm", "shared"})
}
return nil
}
v := map[string]string{"path": path}
if server != "" {
v["server"] = server
}
if nfs != "" {
v["nfs"] = nfs
}
b, _ := json.Marshal(v)
return doc.SetYAML([]string{"slurm", "shared"}, string(b))
})
},
}
c.Flags().StringVar(&path, "path", "", "where every machine sees the files, as /shared")
c.Flags().StringVar(&server, "server", "", "the llmgw node that exports them over NFS")
c.Flags().StringVar(&nfs, "nfs", "", "the site's NFS export, HOST:/EXPORT")
c.Flags().BoolVar(&off, "off", false, "no shared files; mounts and exports llmgw made are taken away")
c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
return c
}
func newSlurmUsersCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
cmd := &cobra.Command{
Use: "users",
Short: "The Linux users that run jobs, made from llmgw's accounts",
Long: "With slurm.users \"llmgw\" every active account is a Linux user on every machine of the Slurm\n" +
"cluster: its login from its e-mail address, a UID of llmgw's from 200000 up (never given to\n" +
"another, even when the account goes), its home under the shared path. It logs in over SSH\n" +
"with the keys given in the portal (Profile) or here. An account suspended or deleted loses\n" +
"its user; its home and files stay. Off, users are the site's: from its directory (sssd), or\n" +
"made by hand.",
}
list := &cobra.Command{
Use: "list",
Short: "Every user, with its login, UID and how many keys it has",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
var us []slurmsvc.User
if err := client().get(cmd.Context(), "/api/v1/slurm/users", &us); err != nil {
return err
}
w := cmd.OutOrStdout()
if *asJSON {
return json.NewEncoder(w).Encode(us)
}
tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
fmt.Fprintln(tw, "LOGIN\tUID\tACCOUNT\tSSH KEYS")
for _, u := range us {
fmt.Fprintf(tw, "%s\t%d\t%s\t%d\n", u.Login, u.UID, u.Email, len(u.SSHKeys))
}
return tw.Flush()
},
}
var note string
toggle := func(use, short, value string) *cobra.Command {
c := &cobra.Command{
Use: use,
Short: short,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
if value == "" {
if doc.Node([]string{"slurm", "users"}) != nil {
doc.Unset([]string{"slurm", "users"})
}
return nil
}
return doc.SetYAML([]string{"slurm", "users"}, yamlString(value))
})
},
}
c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
return c
}
keys := &cobra.Command{
Use: "keys ACCOUNT FILE",
Short: "Replace an account's SSH public keys, from a file (- for standard input)",
Long: "FILE holds OpenSSH public keys, one a line, as ~/.ssh/id_ed25519.pub; an empty one takes them all away.",
Example: " llmgw ctl slurm users keys ayse@site.example ayse.pub\n" +
" ssh-add -L | llmgw ctl slurm users keys ayse@site.example -",
Args: cobra.ExactArgs(2),
RunE: func(cmd *cobra.Command, args []string) error {
var b []byte
var err error
if args[1] == "-" {
b, err = io.ReadAll(io.LimitReader(cmd.InOrStdin(), 64<<10))
} else {
b, err = os.ReadFile(args[1])
}
if err != nil {
return err
}
body, _ := json.Marshal(map[string]string{"keys": string(b)})
var u slurmsvc.User
if err := client().put(cmd.Context(), "/api/v1/slurm/users/"+url.PathEscape(args[0])+"/ssh-keys", body, &u); err != nil {
return err
}
fmt.Fprintf(cmd.OutOrStdout(), "%s (UID %d) has %d SSH key(s); the machines have them within a minute\n",
u.Login, u.UID, len(u.SSHKeys))
return nil
},
}
cmd.AddCommand(list,
toggle("on", "Make a Linux user for every active account, on every machine of the Slurm cluster", "llmgw"),
toggle("off", "Leave users to the site; those llmgw made are taken away, their homes left", ""),
keys)
return cmd
}
func newSlurmStatusCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
return &cobra.Command{
Use: "status",
Short: "The Slurm cluster, and where each of its nodes is",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
var st slurmsvc.Status
if err := client().get(cmd.Context(), "/api/v1/slurm", &st); err != nil {
return err
}
w := cmd.OutOrStdout()
if *asJSON {
return json.NewEncoder(w).Encode(st)
}
if !st.Enabled {
fmt.Fprintln(w, "Slurm is off. `llmgw ctl slurm setup --controller NODE --nodes NODE,...` turns it on.")
return nil
}
fmt.Fprintf(w, "cluster %s, partition %s, controller %s", st.Cluster, st.Partition, st.Controller)
switch c := st.Ctld; {
case c == nil:
fmt.Fprintln(w, ": not heard from yet")
case c.Error != "":
fmt.Fprintf(w, ": %s\n", c.Error)
case c.SlurmctldRunning && c.SlurmdbdRunning:
fmt.Fprintf(w, ": slurmctld %s and slurmdbd running\n", c.Version)
case c.SlurmctldRunning:
fmt.Fprintf(w, ": slurmctld %s running, slurmdbd not (the jobs' records)\n", c.Version)
default:
fmt.Fprintln(w, ": slurmctld not running")
}
fmt.Fprintf(w, "shared files: %s\n", sharedLine(st.Shared))
if st.Users != "" {
fmt.Fprintf(w, "users: %d, made from llmgw's accounts\n", st.UserCount)
if st.Accounts > 0 {
fmt.Fprintf(w, "accounts: %d, one for each project (sbatch --account=ORG-TEAM-PROJECT)\n", st.Accounts)
}
}
if line := containersLine(st); line != "" {
fmt.Fprintf(w, "containers: %s\n", line)
}
tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
fmt.Fprintln(tw, "NODE\tPHASE\tSLURM\tJOBS\tSLURMD\tSHARED\tNOTE")
for _, n := range st.Nodes {
state, jobs, slurmd, shared, note := "-", "-", "-", "-", n.Describe()
if r := n.Status; r != nil {
state = or(r.State, "-")
jobs = fmt.Sprint(r.Jobs)
slurmd = map[bool]string{true: "running", false: "stopped"}[r.SlurmdRunning]
shared = or(r.Shared, "-")
if r.Error != "" {
note = r.Error
}
}
if !n.Connected {
note = "agent not connected; " + note
}
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n", n.Node, n.Phase, state, jobs, slurmd, shared, note)
}
if err := tw.Flush(); err != nil {
return err
}
if len(st.Waiting) > 0 {
fmt.Fprintf(w, "not in slurm.conf until they report their hardware: %s\n", strings.Join(st.Waiting, ", "))
}
return nil
},
}
}
func newSlurmSetupCmd(client func() *ctlClient) *cobra.Command {
var controller, cluster, partition, note string
var nodes []string
var all bool
c := &cobra.Command{
Use: "setup --controller NODE --nodes NODE,... | --all-nodes",
Short: "Run Slurm on llmgw's nodes: slurmctld on one, slurmd on those named",
Long: "Turns slurm.enabled on and names the nodes, or with --all-nodes every node. Their agents install munge and Slurm (22.05 or\n" +
"newer) from the machines' own package sources; the controller also slurmdbd, the jobs' records, in a\n" +
"MariaDB on that machine alone (one that was there keeps its settings). Every node starts drained in Slurm and stays\n" +
"llmgw's until `llmgw ctl slurm lend` gives it to Slurm.",
Example: " llmgw ctl slurm setup --controller node06 --nodes cuda01,cuda02\n" +
" llmgw ctl slurm setup --controller node06 --all-nodes",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
if controller == "" || (len(nodes) == 0 && !all) {
return fmt.Errorf("give --controller, and --nodes or --all-nodes")
}
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
if err := doc.SetYAML([]string{"slurm", "enabled"}, "true"); err != nil {
return err
}
if err := doc.SetYAML([]string{"slurm", "controller"}, yamlString(controller)); err != nil {
return err
}
if err := setList(doc, []string{"slurm", "nodes"}, nodes); err != nil {
return err
}
if err := doc.SetYAML([]string{"slurm", "all_nodes"}, fmt.Sprint(all)); err != nil {
return err
}
// Lent nodes no longer named are not lent.
lent := slices.DeleteFunc(docList(doc, []string{"slurm", "lent"}),
func(n string) bool { return !all && !slices.Contains(nodes, n) })
if err := setList(doc, []string{"slurm", "lent"}, lent); err != nil {
return err
}
for k, v := range map[string]string{"cluster_name": cluster, "partition": partition} {
if v != "" {
if err := doc.SetYAML([]string{"slurm", k}, yamlString(v)); err != nil {
return err
}
}
}
return nil
})
},
}
c.Flags().StringVar(&controller, "controller", "", "the node that runs slurmctld")
c.Flags().StringSliceVar(&nodes, "nodes", nil, "the nodes that run slurmd")
c.Flags().BoolVar(&all, "all-nodes", false, "every node runs slurmd, those that join later too")
c.Flags().StringVar(&cluster, "cluster-name", "", "Slurm's ClusterName (llmgw)")
c.Flags().StringVar(&partition, "partition", "", "the partition every node is in (gpu)")
c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
return c
}
func newSlurmOffCmd(client func() *ctlClient) *cobra.Command {
var note string
var force bool
c := &cobra.Command{
Use: "off",
Short: "Stop running Slurm; every node is llmgw's again",
Long: "slurmd and slurmctld are stopped where llmgw started them; the packages stay. A node still\n" +
"lent is refused unless --force: take it back first, so its jobs end before the daemons do.",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
if lent := docList(doc, []string{"slurm", "lent"}); len(lent) > 0 && !force {
return fmt.Errorf("%s still lent: `llmgw ctl slurm reclaim` them first, or --force",
strings.Join(lent, ", "))
}
setList(doc, []string{"slurm", "lent"}, nil) //nolint:errcheck // an empty list
return doc.SetYAML([]string{"slurm", "enabled"}, "false")
})
},
}
c.Flags().BoolVar(&force, "force", false, "even with nodes still lent")
c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
return c
}
// containersLine says whether jobs may run in containers, and which
// machines are not ready yet; empty when containers are off.
func containersLine(st slurmsvc.Status) string {
switch st.Containers {
case "on":
return "on (srun --container-image)"
case "":
return ""
}
var waiting []string
for _, n := range st.Nodes {
if n.Status == nil || n.Status.Containers == "" {
waiting = append(waiting, n.Node)
}
}
if st.Ctld == nil || st.Ctld.Containers == "" {
if !slices.Contains(waiting, st.Controller) {
waiting = append(waiting, st.Controller)
}
}
return "installing pyxis and enroot; jobs take them once every machine has them (waiting: " +
strings.Join(waiting, ", ") + ")"
}
// sharedLine says where the shared files are and come from.
func sharedLine(sh config.SharedStorage) string {
switch {
case sh.Path == "":
return "none (`llmgw ctl slurm shared` adds them; a batch job's output stays on its node)"
case sh.Server != "":
return fmt.Sprintf("%s, exported over NFS by %s", sh.Path, sh.Server)
case sh.NFS != "":
return fmt.Sprintf("%s, mounted from %s", sh.Path, sh.NFS)
}
return sh.Path + ", mounted by the site"
}
// newSlurmListCmd adds nodes to slurm.lent, or takes them out.
func newSlurmListCmd(client func() *ctlClient, use, short string, add bool) *cobra.Command {
var note string
c := &cobra.Command{
Use: use + " NODE...",
Short: short,
Args: cobra.MinimumNArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
return lendChange(doc, args, add)
})
},
}
c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
return c
}
// lendChange adds nodes to slurm.lent, or with add false takes them out.
func lendChange(doc *settings.Doc, names []string, add bool) error {
nodes := docList(doc, []string{"slurm", "nodes"})
all := docBool(doc, []string{"slurm", "all_nodes"})
lent := docList(doc, []string{"slurm", "lent"})
for _, n := range names {
switch {
case add && !all && !slices.Contains(nodes, n):
return fmt.Errorf("%s does not run slurmd: add it with `llmgw ctl slurm setup --nodes`", n)
case add && !slices.Contains(lent, n):
lent = append(lent, n)
case !add:
lent = slices.DeleteFunc(lent, func(l string) bool { return l == n })
}
}
return setList(doc, []string{"slurm", "lent"}, lent)
}
// docList reads a list of strings from the settings, empty when unset.
func docList(doc *settings.Doc, path []string) []string {
n := doc.Node(path)
if n == nil {
return nil
}
var out []string
if err := n.Decode(&out); err != nil {
return nil
}
return out
}
// docBool reads a boolean from the settings, false when unset.
func docBool(doc *settings.Doc, path []string) bool {
var b bool
if n := doc.Node(path); n != nil {
_ = n.Decode(&b)
}
return b
}
func setList(doc *settings.Doc, path []string, list []string) error {
if len(list) == 0 {
if doc.Node(path) != nil {
doc.Unset(path)
}
return nil
}
return doc.SetYAML(path, yamlList(list))
}
func yamlString(s string) string {
b, _ := json.Marshal(s)
return string(b)
}
func yamlList(list []string) string {
b, _ := json.Marshal(list)
return string(b)
}
+35
View File
@@ -0,0 +1,35 @@
package cli
import (
"slices"
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
)
func TestSlurmDocLists(t *testing.T) {
doc, err := settings.FromBytes("settings", []byte("slurm:\n enabled: true\n nodes: [cuda01, site2]\n"))
if err != nil {
t.Fatal(err)
}
if got := docList(doc, []string{"slurm", "nodes"}); !slices.Equal(got, []string{"cuda01", "site2"}) {
t.Fatalf("nodes %v", got)
}
if got := docList(doc, []string{"slurm", "lent"}); got != nil {
t.Fatalf("lent %v", got)
}
if err := setList(doc, []string{"slurm", "lent"}, []string{"cuda01"}); err != nil {
t.Fatal(err)
}
if got := docList(doc, []string{"slurm", "lent"}); !slices.Equal(got, []string{"cuda01"}) {
t.Fatalf("lent %v", got)
}
if err := setList(doc, []string{"slurm", "lent"}, nil); err != nil {
t.Fatal(err)
}
b, _ := doc.Bytes()
if strings.Contains(string(b), "lent") {
t.Fatalf("an empty list left:\n%s", b)
}
}
+16
View File
@@ -84,3 +84,19 @@ func TestTheSummaryIsWrittenWhereItIsToldNotToTheTerminal(t *testing.T) {
} }
} }
} }
func TestStatusSaysWhenAlertsGoNowhere(t *testing.T) {
render := func(s adminapi.Summary) string {
cmd := &cobra.Command{}
var b bytes.Buffer
cmd.SetOut(&b)
printSummary(cmd, s)
return b.String()
}
if out := render(adminapi.Summary{NoAlerts: true}); !strings.Contains(out, "ALERTS nowhere") {
t.Errorf("no webhook was not called out:\n%s", out)
}
if out := render(adminapi.Summary{}); strings.Contains(out, "ALERTS") {
t.Errorf("an alerts line with webhooks set:\n%s", out)
}
}
+19 -15
View File
@@ -4,6 +4,7 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"strings" "strings"
"time"
"github.com/spf13/cobra" "github.com/spf13/cobra"
@@ -87,13 +88,10 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
if err != nil { if err != nil {
return err return err
} }
var results []struct { var results []adminapi.CollectResult
Node string `json:"node"` // The control plane waits for the nodes to list their files without the
Files int `json:"files"` // deleted ones: half a minute at most.
Bytes int64 `json:"bytes"` if err := c.patient(time.Minute).post(cmd.Context(), "/api/v1/weights/collect", body, &results); err != nil {
Error string `json:"error,omitempty"`
}
if err := c.post(cmd.Context(), "/api/v1/weights/collect", body, &results); err != nil {
return err return err
} }
if asJSON { if asJSON {
@@ -103,7 +101,7 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
fmt.Fprintln(cmd.OutOrStdout(), "Nothing to reclaim.") fmt.Fprintln(cmd.OutOrStdout(), "Nothing to reclaim.")
return nil return nil
} }
var freed int64 var freed, left int64
for _, r := range results { for _, r := range results {
if r.Error != "" { if r.Error != "" {
// Reported rather than fatal: the other nodes' space is still worth // Reported rather than fatal: the other nodes' space is still worth
@@ -111,14 +109,20 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
fmt.Fprintf(cmd.OutOrStdout(), "%s: %s\n", r.Node, r.Error) fmt.Fprintf(cmd.OutOrStdout(), "%s: %s\n", r.Node, r.Error)
continue continue
} }
fmt.Fprintf(cmd.OutOrStdout(), "%s: asked to remove %d files, %s\n", line := fmt.Sprintf("%s: removed %d files, %s", r.Node, r.Files, fileSize(r.Bytes))
r.Node, r.Files, fileSize(r.Bytes)) if r.Left > 0 {
freed += r.Bytes line += fmt.Sprintf("; %d still there, %s", r.Left, fileSize(r.LeftBytes))
}
fmt.Fprintln(cmd.OutOrStdout(), line)
freed, left = freed+r.Bytes, left+r.LeftBytes
}
fmt.Fprintf(cmd.OutOrStdout(), "\n%s freed.\n", fileSize(freed))
if left > 0 {
// Each node decides: one that keeps a path it cannot prove it
// fetched is the feature working. Its agent's log names each.
fmt.Fprintf(cmd.OutOrStdout(), "%s the nodes kept; each node's agent log says why "+
"(journalctl -u llmgw-agent).\n", fileSize(left))
} }
// "asked", not "removed": each node decides, and one that refuses a path it
// cannot prove it fetched is the feature working rather than failing.
fmt.Fprintf(cmd.OutOrStdout(), "\n%s requested. The nodes verify each path before deleting it.\n",
fileSize(freed))
return nil return nil
} }
+34
View File
@@ -0,0 +1,34 @@
package cli
import (
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
)
func TestAPackageInstallThatIsNoEnginesIsNamed(t *testing.T) {
if got := provisionLine("", "", "installing", "mariadb-server, munge, with apt-get"); got !=
"system packages: installing: mariadb-server, munge, with apt-get" {
t.Fatalf("got %q", got)
}
if got := provisionLine("llamacpp", "b10991", "ready", ""); got != "llamacpp b10991: ready" {
t.Fatalf("got %q", got)
}
}
func TestCoresAloneKeepTheMemory(t *testing.T) {
m := catalog.Model{Name: "judge", Profiles: []catalog.Profile{{Name: "cpu", MinMemoryGB: 2}, {Name: "cpu-big", MinMemoryGB: 2}}}
if got, err := keptMemory(m, ""); err != nil || got != 2 {
t.Fatalf("got %d, %v", got, err)
}
m.Profiles[1].MinMemoryGB = 4
if _, err := keptMemory(m, ""); err == nil {
t.Fatal("profiles that differ were given one memory")
}
if got, err := keptMemory(m, "cpu-big"); err != nil || got != 4 {
t.Fatalf("got %d, %v", got, err)
}
if _, err := keptMemory(m, "gpu"); err == nil {
t.Fatal("a profile the model has not was taken")
}
}
+83 -4
View File
@@ -19,6 +19,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi" "gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve" "gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck" "gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
) )
@@ -34,7 +35,7 @@ func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
Use: "model", Use: "model",
Short: "Add or remove models while the cluster is running", Short: "Add or remove models while the cluster is running",
} }
c.AddCommand(newModelStopCmd(client), newModelStartCmd(client), newModelContextCmd(client), newModelGPUsCmd(client), newModelAddCmd(client), newModelPresetsCmd(client, asJSON), newModelRemoveCmd(client), newModelPlanCmd(client), c.AddCommand(newModelStopCmd(client), newModelStartCmd(client), newModelReplicasCmd(client), newModelContextCmd(client), newModelGPUsCmd(client), newModelAddCmd(client), newModelPresetsCmd(client, asJSON), newModelRemoveCmd(client), newModelPlanCmd(client),
newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client), newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
newTargetCmd(client, asJSON), newAdaptersCmd(client), newTargetCmd(client, asJSON), newAdaptersCmd(client),
newModelServeCmd(client), newModelUpdatesCmd(client, asJSON), newModelUpdateCmd(client), newModelServeCmd(client), newModelUpdatesCmd(client, asJSON), newModelUpdateCmd(client),
@@ -337,7 +338,7 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
var memory, cores int var memory, cores int
var profile string var profile string
c := &cobra.Command{ c := &cobra.Command{
Use: "resources <model> --memory GB [--cores N]", Use: "resources <model> [--memory GB] [--cores N]",
Short: "Set what a model reserves of a node: memory, and CPU cores", Short: "Set what a model reserves of a node: memory, and CPU cores",
Long: "--memory is the host RAM a replica needs: for a model with no GPU, its\n" + Long: "--memory is the host RAM a replica needs: for a model with no GPU, its\n" +
"weights, KV cache and engine together. Placement reserves it on the node,\n" + "weights, KV cache and engine together. Placement reserves it on the node,\n" +
@@ -350,8 +351,20 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
"--model M` moves them to the new figures.", "--model M` moves them to the new figures.",
Args: cobra.ExactArgs(1), Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error { RunE: func(cmd *cobra.Command, args []string) error {
if !cmd.Flags().Changed("memory") && !cmd.Flags().Changed("cores") {
return fmt.Errorf("give --memory, --cores or both")
}
if !cmd.Flags().Changed("memory") { if !cmd.Flags().Changed("memory") {
return fmt.Errorf("--memory is required") // Cores alone: the memory stays what it is.
var cur catalog.Model
if err := client().get(cmd.Context(), "/api/v1/models/"+args[0], &cur); err != nil {
return err
}
kept, err := keptMemory(cur, profile)
if err != nil {
return err
}
memory = kept
} }
body, err := json.Marshal(adminapi.ResourcesRequest{Profile: profile, MemoryGB: memory, CPUCores: cores}) body, err := json.Marshal(adminapi.ResourcesRequest{Profile: profile, MemoryGB: memory, CPUCores: cores})
if err != nil { if err != nil {
@@ -368,12 +381,31 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
return nil return nil
}, },
} }
c.Flags().IntVar(&memory, "memory", 0, "host RAM a replica needs, in GB") c.Flags().IntVar(&memory, "memory", 0, "host RAM a replica needs, in GB; kept as it is when left out")
c.Flags().IntVar(&cores, "cores", 0, "CPU cores a replica with no GPU computes on; 0 reserves none") c.Flags().IntVar(&cores, "cores", 0, "CPU cores a replica with no GPU computes on; 0 reserves none")
c.Flags().StringVar(&profile, "profile", "", "one profile; every profile when left out") c.Flags().StringVar(&profile, "profile", "", "one profile; every profile when left out")
return c return c
} }
// keptMemory is the memory a model's profiles reserve now, for a change of
// cores alone: one profile's, or every profile's when they agree.
func keptMemory(m catalog.Model, profile string) (int, error) {
memory := -1
for _, p := range m.Profiles {
if profile != "" && p.Name != profile {
continue
}
if memory >= 0 && p.MinMemoryGB != memory {
return 0, fmt.Errorf("%s's profiles reserve different memory: give --memory, or --profile", m.Name)
}
memory = p.MinMemoryGB
}
if memory < 0 {
return 0, fmt.Errorf("%s has no profile %q", m.Name, profile)
}
return memory, nil
}
func newClassCmd(client func() *ctlClient) *cobra.Command { func newClassCmd(client func() *ctlClient) *cobra.Command {
return &cobra.Command{ return &cobra.Command{
Use: "class <model> critical|standard|opportunistic|default", Use: "class <model> critical|standard|opportunistic|default",
@@ -598,3 +630,50 @@ func newModelGPUsCmd(client func() *ctlClient) *cobra.Command {
c.Flags().BoolVar(&inPlace, "in-place", false, "stop each replica before its replacement starts, when there is no room beside it") c.Flags().BoolVar(&inPlace, "in-place", false, "stop each replica before its replacement starts, when there is no room beside it")
return c return c
} }
func newModelReplicasCmd(client func() *ctlClient) *cobra.Command {
var cpuMemory, cpuCores int
c := &cobra.Command{
Use: "replicas <model> <profile>=<n>...",
Short: "How many of a model's replicas run on each of its profiles: gpu=7 cpu=4",
Long: "Each replica runs on the best profile that fits unless counted here: seven on\n" +
"cards and four on CPUs, all answering under the model's name, the router filling\n" +
"the cards before the CPUs. A profile not named keeps its count. For a GGUF model\n" +
"with no profile on CPUs, --cpu-memory adds one named cpu. Its scaling becomes the\n" +
"sum; llmgw ctl model scale with another total ends the count per profile.",
Example: " llmgw ctl model replicas qwen-coder gpu-1xl40s-48g=7 cpu=4 --cpu-memory 24 --cpu-cores 8",
Args: cobra.MinimumNArgs(2),
RunE: func(cmd *cobra.Command, args []string) error {
req := adminapi.ProfileReplicasRequest{Profiles: map[string]int{}}
for _, a := range args[1:] {
name, v, ok := strings.Cut(a, "=")
n, err := strconv.Atoi(v)
if !ok || err != nil || name == "" {
return fmt.Errorf("%q: a profile and a count, as gpu=7", a)
}
req.Profiles[name] = n
}
if cpuMemory > 0 {
req.CPU = &modelsvc.CPUProfile{MemoryGB: cpuMemory, CPUCores: cpuCores}
}
body, err := json.Marshal(req)
if err != nil {
return err
}
var m catalog.Model
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/replicas", body, &m); err != nil {
return err
}
var parts []string
for _, p := range m.Profiles {
parts = append(parts, fmt.Sprintf("%d on %s", p.Replicas, p.Name))
}
fmt.Fprintf(cmd.OutOrStdout(), "%s runs %s: %d in all. Those short start first, then those over stop.\n",
m.Name, strings.Join(parts, ", "), m.SplitTotal())
return nil
},
}
c.Flags().IntVar(&cpuMemory, "cpu-memory", 0, "add a profile on CPUs named cpu, reserving this many GB a replica")
c.Flags().IntVar(&cpuCores, "cpu-cores", 0, "the cores a replica on CPUs computes on (default: half the node's)")
return c
}
+40 -2
View File
@@ -321,7 +321,10 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
fmt.Fprintln(out, "\nNo file fits this cluster as it is: a smaller context (--context), or more memory.") fmt.Fprintln(out, "\nNo file fits this cluster as it is: a smaller context (--context), or more memory.")
return errDoesNotFit return errDoesNotFit
} }
fmt.Fprintf(out, "\nBest: %s on %s: %s.\n", rec.Artifact, rec.Label(), rec.MemoryText()) fmt.Fprintf(out, "\nBest: %s on %s: %s%s.\n", rec.Artifact, rec.Label(), rec.MemoryText(), speedNote(rec))
if w := slowWarning(rec); w != "" {
fmt.Fprintln(out, w)
}
if p.KVBytesPerToken > 0 { if p.KVBytesPerToken > 0 {
fmt.Fprintf(out, "The context costs %.2f GB per 1000 tokens, the whole model (--context to plan another).\n", fmt.Fprintf(out, "The context costs %.2f GB per 1000 tokens, the whole model (--context to plan another).\n",
modelplan.GB(p.KVBytesPerToken*1000)) modelplan.GB(p.KVBytesPerToken*1000))
@@ -334,7 +337,8 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
if o.FreeNow { if o.FreeNow {
now = "free" now = "free"
} }
fmt.Fprintf(w, " %s\t%s\t%s\t%d host(s)\t%s\n", o.Artifact, o.Label(), o.MemoryText(), o.Hosts, now) fmt.Fprintf(w, " %s\t%s\t%s\t%d host(s)\t%s\t%s\n", o.Artifact, o.Label(), o.MemoryText(), o.Hosts, now,
speedCell(o.TokensPerSecond))
} }
_ = w.Flush() _ = w.Flush()
} }
@@ -349,3 +353,37 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
fmt.Fprintln(out, " --fallbacks") fmt.Fprintln(out, " --fallbacks")
return nil return nil
} }
// speedCell is a way's speed in a table: about how many tokens a second one
// reply gets, and whether that is slow.
func speedCell(tps float64) string {
switch {
case tps <= 0:
return ""
case tps < modelplan.CPUTooSlowTPS:
return fmt.Sprintf("~%.1f tok/s, very slow", tps)
case tps < modelplan.CPUSlowTPS:
return fmt.Sprintf("~%.0f tok/s, slow", tps)
}
return fmt.Sprintf("~%.0f tok/s", tps)
}
// speedNote is the same for a sentence, empty when not known.
func speedNote(o modelplan.Option) string {
if o.TokensPerSecond <= 0 {
return ""
}
return fmt.Sprintf(", about %s tokens a second for one reply", strconv.FormatFloat(o.TokensPerSecond, 'f', 1, 64))
}
// slowWarning says what a slow way means for somebody waiting on a reply.
func slowWarning(o modelplan.Option) string {
tps := o.TokensPerSecond
switch {
case tps <= 0 || tps >= modelplan.CPUSlowTPS:
return ""
case tps < modelplan.CPUTooSlowTPS:
return fmt.Sprintf("Very slow: a reply of 500 tokens takes about %.0f minutes.", max(1, 500/tps/60))
}
return fmt.Sprintf("Slow: a reply of 500 tokens takes about %.0f seconds.", 500/tps)
}
+15 -4
View File
@@ -114,7 +114,7 @@ engines are kept.`,
} }
f := cmd.Flags() f := cmd.Flags()
f.StringVar(&na.opts.Node, "node", "", "node id (default: the machine's hostname)") f.StringVar(&na.opts.Node, "node", "", "node id (default: the machine's hostname)")
f.StringVar(&na.opts.User, "ssh-user", "", "user to log in as (default: the one before @ in the address, else whoever ran sudo, else you)") f.StringVar(&na.opts.User, "ssh-user", "", "user to log in as (default: the one before @ in the address, else the ssh config's for the machine, else whoever ran sudo, else you)")
f.StringArrayVar(&na.servers, "server", nil, "control plane address the agent dials, https://host:7443; repeatable for failover (default: this machine)") f.StringArrayVar(&na.servers, "server", nil, "control plane address the agent dials, https://host:7443; repeatable for failover (default: this machine)")
f.StringArrayVar(&na.fabrics, "fabric", nil, "InfiniBand or RoCE fabric and its adapters, name:mlx5_0,mlx5_1; repeatable") f.StringArrayVar(&na.fabrics, "fabric", nil, "InfiniBand or RoCE fabric and its adapters, name:mlx5_0,mlx5_1; repeatable")
f.StringVar(&na.opts.MgmtAddr, "mgmt-addr", "", "address engines are reached at (default: found by the agent)") f.StringVar(&na.opts.MgmtAddr, "mgmt-addr", "", "address engines are reached at (default: found by the agent)")
@@ -228,8 +228,9 @@ func (na *nodeAdd) target(addr string) onboard.Options {
if o.User == "" { if o.User == "" {
// Under sudo, the person who ran it — their account is usually the // Under sudo, the person who ran it — their account is usually the
// one that exists on the nodes too. root's own keys still do the // one that exists on the nodes too. root's own keys still do the
// logging in, since that is who is running ssh. // logging in, since that is who is running ssh. Only a fallback: a
o.User = os.Getenv("SUDO_USER") // User in the ssh config's block for the machine comes first.
o.FallbackUser = os.Getenv("SUDO_USER")
} }
return o return o
} }
@@ -495,7 +496,17 @@ func engineEvent(engine, version, state, detail string) onboard.Event {
level = onboard.Warn level = onboard.Warn
} }
return onboard.Event{Step: onboard.StepEngines, Level: level, return onboard.Event{Step: onboard.StepEngines, Level: level,
Detail: fmt.Sprintf("%s %s: %s%s", engine, version, state, prefixed(": ", detail))} Detail: provisionLine(engine, version, state, detail)}
}
// provisionLine is one line of what a node is installing. A package
// install that is no engine's, Slurm's or MariaDB's, has no name or version.
func provisionLine(engine, version, state, detail string) string {
what := strings.TrimSpace(engine + " " + version)
if what == "" {
what = "system packages"
}
return fmt.Sprintf("%s: %s%s", what, state, prefixed(": ", detail))
} }
func prefixed(p, s string) string { func prefixed(p, s string) string {
+153 -2
View File
@@ -10,6 +10,7 @@ import (
"fmt" "fmt"
"log/slog" "log/slog"
"maps" "maps"
"math"
"net" "net"
"net/http" "net/http"
"os" "os"
@@ -36,6 +37,7 @@ import (
"golang.org/x/sync/errgroup" "golang.org/x/sync/errgroup"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1" llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
"gitea.mynodes.xyz/ilker/llmgw/internal/audit" "gitea.mynodes.xyz/ilker/llmgw/internal/audit"
"gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval" "gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval"
"gitea.mynodes.xyz/ilker/llmgw/internal/bundle" "gitea.mynodes.xyz/ilker/llmgw/internal/bundle"
@@ -47,8 +49,10 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrictest" "gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrictest"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/federation" "gitea.mynodes.xyz/ilker/llmgw/internal/control/federation"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics" "gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/held"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc" "gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler" "gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train" "gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard" "gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf" "gitea.mynodes.xyz/ilker/llmgw/internal/hf"
@@ -59,6 +63,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/promq" "gitea.mynodes.xyz/ilker/llmgw/internal/promq"
"gitea.mynodes.xyz/ilker/llmgw/internal/router" "gitea.mynodes.xyz/ilker/llmgw/internal/router"
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets" "gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store" storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf" "gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
"gitea.mynodes.xyz/ilker/llmgw/internal/version" "gitea.mynodes.xyz/ilker/llmgw/internal/version"
@@ -254,6 +259,7 @@ func newServerCmd() *cobra.Command {
// The settings in force, for what reads them at each use. // The settings in force, for what reads them at each use.
var current atomic.Pointer[config.Server] var current atomic.Pointer[config.Server]
current.Store(&cfg) current.Store(&cfg)
rt.SetSlurm(func() config.SlurmSettings { return current.Load().Slurm })
var agentSvc *agentsvc.Service var agentSvc *agentsvc.Service
var agentHandler http.Handler var agentHandler http.Handler
var agentPath string var agentPath string
@@ -261,6 +267,8 @@ func newServerCmd() *cobra.Command {
// What the settings say of the firewall, for the mirror and the // What the settings say of the firewall, for the mirror and the
// store's members; nil without a control plane. // store's members; nil without a control plane.
var fwPub *firewallPublisher var fwPub *firewallPublisher
// The Slurm cluster beside llmgw, when there is a store (BACKLOG P).
var slurmSvc *slurmsvc.Service
updateMetrics := &modelserve.UpdateCollector{} updateMetrics := &modelserve.UpdateCollector{}
// The store's cluster, when it is one; read by the metrics and // The store's cluster, when it is one; read by the metrics and
// by `ctl status`. // by `ctl status`.
@@ -412,6 +420,7 @@ func newServerCmd() *cobra.Command {
adminAPI = api adminAPI = api
api.SetReleases(cfg.Releases) api.SetReleases(cfg.Releases)
api.SetNotifier(notifier) api.SetNotifier(notifier)
api.SetWebhooks(func() int { return len(current.Load().Notify.Webhooks) })
api.SetReplayer(rt.Replay) api.SetReplayer(rt.Replay)
// Quality suites, asked of a model the way a replay is. // Quality suites, asked of a model the way a replay is.
evals = newEvalService(cfg, rt, admin, log) evals = newEvalService(cfg, rt, admin, log)
@@ -427,6 +436,27 @@ func newServerCmd() *cobra.Command {
api.SetAgentRestarter(agentSvc.RestartAgents) api.SetAgentRestarter(agentSvc.RestartAgents)
} }
api.SetAudit(st) api.SetAudit(st)
// Prices from what the cluster costs: what each model
// holds, kept by the hour on the leader.
api.SetPrices(st, func() adminapi.PriceConfig {
au := current.Load().Auth
cur := make(map[string]accounting.Price, len(au.Pricing))
for m, p := range au.Pricing {
cur[m] = accountingPrice(p)
}
return adminapi.PriceConfig{Basis: accounting.Basis{GPUHour: au.GPUHourCost, CoreHour: au.CoreHourCost},
Current: cur, Default: accountingPrice(au.DefaultPrice)}
})
go (&held.Recorder{Inv: admin.Inventory(), Store: st, Log: log,
Models: func() map[string]string {
m := map[string]string{}
for _, rep := range admin.Reconciler().Replicas() {
m[rep.ID] = rep.Model
}
return m
},
Leader: func() bool { return admin.Leadership() == nil || admin.Leadership().IsLeader() },
}).Run(ctx)
// Knowledge bases, with the cluster's own models to embed a // Knowledge bases, with the cluster's own models to embed a
// probe (a collection's dimension) and an operator's search. // probe (a collection's dimension) and an operator's search.
api.SetKB(&adminapi.KBAdmin{Store: st, Embed: embedder(rt.Replay)}) api.SetKB(&adminapi.KBAdmin{Store: st, Embed: embedder(rt.Replay)})
@@ -435,7 +465,46 @@ func newServerCmd() *cobra.Command {
trainer := train.New(st, agentSvc, log) trainer := train.New(st, agentSvc, log)
agentSvc.SetTrain(trainer) agentSvc.SetTrain(trainer)
go trainer.Run(ctx) go trainer.Run(ctx)
api.SetTrain(&adminapi.TrainAdmin{Store: st, Service: trainer}) api.SetTrain(&adminapi.TrainAdmin{Store: st, Service: trainer, Log: log})
// From the portal: a node with a card free enough, and
// what was made published when it is done.
trainer.SetPicker(api.PickTrainNode)
trainer.SetOnDone(api.TrainDone)
rt.SetTraining(api)
// Slurm on the nodes the settings name, its munge key
// made from a cluster secret.
slurmSvc = slurmsvc.New(func() config.SlurmSettings { return current.Load().Slurm },
admin.Inventory(), agentSvc, func(ctx context.Context) ([]byte, error) {
secret, err := st.ClusterSecret(ctx, "slurm-munge")
if err != nil {
return nil, err
}
return slurm.MungeKey(secret)
}, log)
// What slurm.conf held, for the leader that comes next.
slurmSvc.SetMemory(slurmsvc.StoreMemory(st))
// slurm.users "llmgw": a Linux user for each active account.
slurmSvc.SetUsers(func(ctx context.Context) ([]slurmsvc.User, error) {
us, err := st.SlurmUsers(ctx)
out := make([]slurmsvc.User, 0, len(us))
for _, u := range us {
out = append(out, adminapi.SlurmUserView(u))
}
return out, err
})
api.SetSlurmUsers(st)
// Slurm's accounts from the projects, and the jobs
// that ended priced as the cluster's hours are.
slurmSvc.SetAccounts(func(ctx context.Context) ([]slurm.Account, error) {
return slurmAccounts(ctx, st)
})
slurmSvc.SetJobs(func(ctx context.Context, jobs []slurmsvc.JobRecord) error {
au := current.Load().Auth
return recordSlurmJobs(ctx, st, current.Load().Slurm.Cluster(), au.GPUHourCost, au.CoreHourCost, jobs, log)
})
agentSvc.SetSlurm(slurmSvc)
go slurmSvc.Run(ctx)
api.SetSlurm(slurmSvc)
} }
// The cluster's settings, each change checked against this // The cluster's settings, each change checked against this
// machine's own keys before it is kept. // machine's own keys before it is kept.
@@ -597,7 +666,7 @@ func newServerCmd() *cobra.Command {
// takes longer than the mirror's usual answers. // takes longer than the mirror's usual answers.
if adminAPI != nil { if adminAPI != nil {
if up, err := mirror.NewFailoverClient(mirrors, cfg.Mirror.TLS, 10*time.Minute); err == nil { if up, err := mirror.NewFailoverClient(mirrors, cfg.Mirror.TLS, 10*time.Minute); err == nil {
adminAPI.SetTrainMirror(up) adminAPI.SetTrainMirror(up, hf.New(cfg.Mirror.Registry, cfg.Mirror.Timeout))
} }
} }
// Browsing the registry is what turns "paste a resolve URL" // Browsing the registry is what turns "paste a resolve URL"
@@ -841,6 +910,9 @@ func newServerCmd() *cobra.Command {
fwPub.Kick() fwPub.Kick()
} }
} }
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "slurm") }) && slurmSvc != nil {
slurmSvc.Kick()
}
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "on_demand") }) && admin != nil { if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "on_demand") }) && admin != nil {
admin.Reconciler().SetEvictionTimes(next.OnDemand.EvictAfterIdle, next.OnDemand.MinLifetime) admin.Reconciler().SetEvictionTimes(next.OnDemand.EvictAfterIdle, next.OnDemand.MinLifetime)
} }
@@ -1384,3 +1456,82 @@ func runPresetsPass(ctx context.Context, api *adminapi.API, lead *control.Leader
} }
} }
} }
// accountingPrice is a configured price as accounting has it.
func accountingPrice(p config.Price) accounting.Price {
return accounting.Price{PromptPer1k: p.PromptPer1k, CompletionPer1k: p.CompletionPer1k, CachedPer1k: p.CachedPer1k,
PerImage: p.PerImage, PerMesh: p.PerMesh}
}
// slurmAccounts are Slurm's accounts from the store's projects, members and
// Slurm users.
func slurmAccounts(ctx context.Context, st *storepkg.DB) ([]slurm.Account, error) {
ps, err := st.SlurmProjects(ctx)
if err != nil {
return nil, err
}
ms, err := st.AllMemberships(ctx)
if err != nil {
return nil, err
}
us, err := st.SlurmUsers(ctx)
if err != nil {
return nil, err
}
logins := make(map[string]string, len(us))
for _, u := range us {
logins[u.UserID] = u.Login
}
tps := make([]slurm.TenancyProject, 0, len(ps))
for _, p := range ps {
tps = append(tps, slurm.TenancyProject{ID: p.ID, Name: p.Name, TeamID: p.TeamID, TeamName: p.TeamName,
OrgID: p.OrgID, OrgName: p.OrgName})
}
tms := make([]slurm.TenancyMember, 0, len(ms))
for _, m := range ms {
tms = append(tms, slurm.TenancyMember{UserID: m.UserID, Scope: string(m.Scope), ScopeID: m.ScopeID})
}
return slurm.Accounts(tps, tms, logins), nil
}
// recordSlurmJobs records the jobs that ended, each with its project (from
// its account) and person (from its login), priced as the hours the
// cluster's models hold are: a GPU job its cards' hours, any other its
// cores'.
func recordSlurmJobs(ctx context.Context, st *storepkg.DB, cluster string, gpuHour, coreHour float64,
jobs []slurmsvc.JobRecord, log *slog.Logger) error {
as, err := slurmAccounts(ctx, st)
if err != nil {
return err
}
project := map[string]string{}
for _, a := range as {
if a.Project != "" {
project[a.Name] = a.Project
}
}
us, err := st.SlurmUsers(ctx)
if err != nil {
return err
}
person := map[string]string{}
for _, u := range us {
person[u.Login] = u.UserID
}
rows := make([]storepkg.SlurmJob, 0, len(jobs))
for _, j := range jobs {
secs := j.Elapsed.Seconds()
cost := float64(j.CPUs) * secs / 3600 * coreHour
if j.GPUs > 0 {
cost = float64(j.GPUs) * secs / 3600 * gpuHour
}
rows = append(rows, storepkg.SlurmJob{Key: cluster + "/" + j.ID, UserID: person[j.Login], ProjectID: project[j.Account],
Login: j.Login, Account: j.Account, Partition: j.Partition, State: j.State, Start: j.Start, End: j.End,
ElapsedSeconds: int64(secs), CPUs: int64(j.CPUs), GPUs: int64(j.GPUs), CostMicros: int64(math.Ceil(cost * 1e6))})
}
n, err := st.RecordSlurmJobs(ctx, rows)
if n > 0 {
log.Info("Slurm jobs that ended recorded, with their cost", "jobs", n)
}
return err
}
+1 -1
View File
@@ -443,7 +443,7 @@ func (sa *serverAdd) storeAccess(ctx context.Context) error {
return err return err
} }
} }
sa.done("nodes will be told of both control planes within half a minute") sa.done("nodes will be told of every control plane within half a minute")
return nil return nil
} }
+13
View File
@@ -124,6 +124,13 @@ type serverInit struct {
// media has the nodes install ComfyUI and the leader add a picture model // media has the nodes install ComfyUI and the leader add a picture model
// and a 3D model once a card fits one. // and a 3D model once a card fits one.
media bool media bool
// slurm runs Slurm beside llmgw: slurmctld on this machine's node,
// slurmd on every node, each llmgw's until it is lent.
slurm bool
// slurmShared is where Slurm's shared files come from: "here" (this
// machine exports /shared), the site's export as HOST:/EXPORT, or
// "none". slurmUsers is "llmgw" (users from the accounts) or "site".
slurmShared, slurmUsers string
// evictAfterIdle and minLifetime are on_demand's; zero leaves its own. // evictAfterIdle and minLifetime are on_demand's; zero leaves its own.
evictAfterIdle, minLifetime time.Duration evictAfterIdle, minLifetime time.Duration
// standbys makes each machine of the replicated store a control plane // standbys makes each machine of the replicated store a control plane
@@ -258,6 +265,12 @@ func newServerInitCmd() *cobra.Command {
f.DurationVar(&in.maxKeyTTL, "max-key-ttl", defaultMaxKeyTTL, "the longest an API key lives") f.DurationVar(&in.maxKeyTTL, "max-key-ttl", defaultMaxKeyTTL, "the longest an API key lives")
f.BoolVar(&in.media, "media", false, f.BoolVar(&in.media, "media", false,
"pictures and 3D: the nodes install ComfyUI, and a picture model and a 3D model are added once a card fits one") "pictures and 3D: the nodes install ComfyUI, and a picture model and a 3D model are added once a card fits one")
f.BoolVar(&in.slurm, "slurm", false,
"Slurm for training and batch jobs: slurmctld and slurmdbd (with a MariaDB) on this machine, slurmd on every node, each llmgw's until 'llmgw ctl slurm lend'")
f.StringVar(&in.slurmShared, "slurm-shared", "here",
"with --slurm, the jobs' shared files at /shared: here (this machine exports them over NFS), the site's HOST:/EXPORT, or none")
f.StringVar(&in.slurmUsers, "slurm-users", config.UsersLlmgw,
"with --slurm, who makes the Linux users that run jobs: llmgw (one for each account, SSH keys in the portal) or site")
f.DurationVar(&in.evictAfterIdle, "evict-after-idle", 0, f.DurationVar(&in.evictAfterIdle, "evict-after-idle", 0,
"how long a model loaded on demand goes unasked-for before another may have its card (0: 2m)") "how long a model loaded on demand goes unasked-for before another may have its card (0: 2m)")
f.DurationVar(&in.minLifetime, "min-lifetime", 0, f.DurationVar(&in.minLifetime, "min-lifetime", 0,
+76
View File
@@ -13,6 +13,7 @@ import (
"time" "time"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision" "gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha" "gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
"gitea.mynodes.xyz/ilker/llmgw/internal/preflight" "gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
) )
@@ -132,9 +133,25 @@ func (in *serverInit) checkChoices() error {
return fmt.Errorf("max key ttl %s: an hour at least", in.maxKeyTTL) return fmt.Errorf("max key ttl %s: an hour at least", in.maxKeyTTL)
case in.evictAfterIdle < 0 || in.minLifetime < 0: case in.evictAfterIdle < 0 || in.minLifetime < 0:
return fmt.Errorf("on-demand times cannot be negative") return fmt.Errorf("on-demand times cannot be negative")
case in.slurm && in.noNode:
return fmt.Errorf("slurm runs slurmctld on this machine's node: it needs the node (no --no-node)")
case in.vip != "" && !in.standbys: case in.vip != "" && !in.standbys:
return fmt.Errorf("a floating address moves between control planes: it needs the standbys") return fmt.Errorf("a floating address moves between control planes: it needs the standbys")
} }
if in.slurm {
// The controller is this machine's node, named by its hostname: a
// name Slurm does not take would keep the server from starting.
if err := (config.SlurmSettings{Enabled: true, Controller: localNodeID()}).Check(); err != nil {
return fmt.Errorf("slurm: this machine's hostname: %w", err)
}
set := config.SlurmSettings{Enabled: true, Controller: localNodeID(), Shared: slurmShared(in.slurmShared)}
if err := set.Check(); err != nil {
return fmt.Errorf("--slurm-shared %q: here, none, or the site's HOST:/EXPORT: %w", in.slurmShared, err)
}
if u := in.slurmUsers; u != "" && u != config.UsersLlmgw && u != "site" {
return fmt.Errorf("--slurm-users %q: llmgw or site", u)
}
}
if in.vip != "" { if in.vip != "" {
ip, _, err := net.ParseCIDR(in.vip) ip, _, err := net.ParseCIDR(in.vip)
if err != nil || ip.To4() == nil { if err != nil || ip.To4() == nil {
@@ -318,6 +335,9 @@ func (in *serverInit) tuneConfig(body string) string {
"# one ('llmgw ctl model presets auto off' stops it).\n"+ "# one ('llmgw ctl model presets auto off' stops it).\n"+
"engines:\n comfyui: "+provision.ComfyUIVersion()+"\npresets:\n auto: true\n") "engines:\n comfyui: "+provision.ComfyUIVersion()+"\npresets:\n auto: true\n")
} }
if in.slurm {
extra = append(extra, slurmSection(localNodeID(), slurmShared(in.slurmShared), in.slurmUsers != "site"))
}
if in.evictAfterIdle > 0 || in.minLifetime > 0 { if in.evictAfterIdle > 0 || in.minLifetime > 0 {
b := "# How models that load when asked for take turns on a card, chosen at\n# install.\non_demand:\n" b := "# How models that load when asked for take turns on a card, chosen at\n# install.\non_demand:\n"
if in.evictAfterIdle > 0 { if in.evictAfterIdle > 0 {
@@ -334,6 +354,53 @@ func (in *serverInit) tuneConfig(body string) string {
return body return body
} }
// slurmShared is --slurm-shared as settings: "here" is /shared exported
// by this machine's node, "none" nothing, anything else the site's export.
func slurmShared(v string) config.SharedStorage {
switch v {
case "", "none":
return config.SharedStorage{}
case "here":
return config.SharedStorage{Path: "/shared", Server: localNodeID()}
}
return config.SharedStorage{Path: "/shared", NFS: v}
}
// slurmSection is server.yaml's slurm block as an install chooses it:
// slurmctld on this machine's node, slurmd on every node, the shared files
// and the users as chosen.
func slurmSection(controller string, shared config.SharedStorage, users bool) string {
b := "# Slurm beside llmgw, chosen at install: slurmctld, and slurmdbd with a MariaDB\n" +
"# for the jobs' records, on this machine; slurmd on\n" +
"# every node, those that join later too. Each node is llmgw's, drained in\n" +
"# Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm; 'llmgw ctl slurm\n" +
"# reclaim NODE' takes it back once its jobs end.\n" +
"slurm:\n enabled: true\n controller: " + yamlString(controller) + "\n all_nodes: true\n"
switch {
case shared.Server != "":
b += " # /shared on every Slurm machine, exported over NFS from this one\n" +
" # ('llmgw ctl slurm shared' points it at the site's storage instead).\n" +
" shared:\n path: " + yamlString(shared.Path) + "\n server: " + yamlString(shared.Server) + "\n"
case shared.NFS != "":
b += " # /shared on every Slurm machine, mounted from the site's export.\n" +
" shared:\n path: " + yamlString(shared.Path) + "\n nfs: " + yamlString(shared.NFS) + "\n"
}
if users {
b += " # A Linux user for each llmgw account, the same UID everywhere; SSH keys\n" +
" # in the portal's Profile. Empty leaves users to the site (sssd, a directory).\n" +
" users: llmgw\n"
}
return b
}
// localNodeID is the node id this machine's agent takes: its hostname.
func localNodeID() string {
if h, err := os.Hostname(); err == nil && h != "" {
return h
}
return "localhost"
}
// durationYAML is a duration as a person writes it: 2h, 90m, 1m30s. // durationYAML is a duration as a person writes it: 2h, 90m, 1m30s.
func durationYAML(d time.Duration) string { func durationYAML(d time.Duration) string {
switch { switch {
@@ -467,6 +534,15 @@ func (in *serverInit) initFlags() []string {
if in.media { if in.media {
add("--media") add("--media")
} }
if in.slurm {
add("--slurm")
if in.slurmShared != "" && in.slurmShared != "here" {
add("--slurm-shared", in.slurmShared)
}
if in.slurmUsers == "site" {
add("--slurm-users", "site")
}
}
if in.evictAfterIdle > 0 { if in.evictAfterIdle > 0 {
add("--evict-after-idle", durationYAML(in.evictAfterIdle)) add("--evict-after-idle", durationYAML(in.evictAfterIdle))
} }
+9 -4
View File
@@ -17,7 +17,7 @@ import (
func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) { func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
in := &serverInit{ in := &serverInit{
language: "tr", signupOpen: true, defaultRPM: 120, defaultTPM: 50000, maxKeyTTL: 30 * 24 * time.Hour, language: "tr", signupOpen: true, defaultRPM: 120, defaultTPM: 50000, maxKeyTTL: 30 * 24 * time.Hour,
media: true, evictAfterIdle: time.Minute, minLifetime: 90 * time.Second, media: true, slurm: true, slurmShared: "here", slurmUsers: "llmgw", evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
judge: judgeSpec{Fail: "open", Categories: []string{"Violent", "PII"}, Replicas: 3, Budget: 3 * time.Second}, judge: judgeSpec{Fail: "open", Categories: []string{"Violent", "PII"}, Replicas: 3, Budget: 3 * time.Second},
} }
j := in.judgeFor() j := in.judgeFor()
@@ -41,6 +41,10 @@ func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
t.Errorf("limits %d %d %s", cfg.Auth.DefaultRPM, cfg.Auth.DefaultTPM, cfg.Auth.MaxKeyTTL) t.Errorf("limits %d %d %s", cfg.Auth.DefaultRPM, cfg.Auth.DefaultTPM, cfg.Auth.MaxKeyTTL)
case !cfg.Presets.Auto || cfg.Engines["comfyui"] == "": case !cfg.Presets.Auto || cfg.Engines["comfyui"] == "":
t.Errorf("pictures and 3D: presets %v comfyui %q", cfg.Presets.Auto, cfg.Engines["comfyui"]) t.Errorf("pictures and 3D: presets %v comfyui %q", cfg.Presets.Auto, cfg.Engines["comfyui"])
case !cfg.Slurm.Enabled || !cfg.Slurm.AllNodes || cfg.Slurm.Controller != localNodeID():
t.Errorf("slurm %+v", cfg.Slurm)
case cfg.Slurm.Shared.Path != "/shared" || cfg.Slurm.Shared.Server != localNodeID() || cfg.Slurm.Users != "llmgw":
t.Errorf("slurm's shared files and users %+v", cfg.Slurm)
case cfg.OnDemand.EvictAfterIdle != time.Minute || cfg.OnDemand.MinLifetime != 90*time.Second: case cfg.OnDemand.EvictAfterIdle != time.Minute || cfg.OnDemand.MinLifetime != 90*time.Second:
t.Errorf("on demand %+v", cfg.OnDemand) t.Errorf("on demand %+v", cfg.OnDemand)
} }
@@ -95,14 +99,14 @@ func TestAStarterIsChosenByNameOrByItsFile(t *testing.T) {
func TestTheChoicesAreTheirFlagsAgain(t *testing.T) { func TestTheChoicesAreTheirFlagsAgain(t *testing.T) {
in := &serverInit{admin: "ops@example.com", hosts: []string{"llm.example.com"}, postgres: true, in := &serverInit{admin: "ops@example.com", hosts: []string{"llm.example.com"}, postgres: true,
mirrorPath: "/data/models", noNode: true, language: "tr", media: true, mirrorPath: "/data/models", language: "tr", media: true, slurm: true,
webhooks: []string{"https://hooks.slack.com/services/T/B/x, json:https://a.example.com/in"}, webhooks: []string{"https://hooks.slack.com/services/T/B/x, json:https://a.example.com/in"},
judge: judgeSpec{Categories: []string{"Violent", "Jailbreak"}}, maxKeyTTL: 7 * 24 * time.Hour} judge: judgeSpec{Categories: []string{"Violent", "Jailbreak"}}, maxKeyTTL: 7 * 24 * time.Hour}
in.starter, _ = starterFrom("qwen3-4b") in.starter, _ = starterFrom("qwen3-4b")
got := strings.Join(in.initFlags(), " ") got := strings.Join(in.initFlags(), " ")
want := "--admin ops@example.com --host llm.example.com --language tr --max-key-ttl 168h --postgres " + want := "--admin ops@example.com --host llm.example.com --language tr --max-key-ttl 168h --postgres " +
"--mirror-dir /data/models --no-node --starter qwen3-4b --judge-categories Violent,Jailbreak " + "--mirror-dir /data/models --starter qwen3-4b --judge-categories Violent,Jailbreak " +
"--webhook https://hooks.slack.com/services/T/B/x --webhook json:https://a.example.com/in --media" "--webhook https://hooks.slack.com/services/T/B/x --webhook json:https://a.example.com/in --media --slurm"
if got != want { if got != want {
t.Errorf("flags\n got %s\nwant %s", got, want) t.Errorf("flags\n got %s\nwant %s", got, want)
} }
@@ -124,6 +128,7 @@ func TestChoicesThatCannotBeInstalledAreRefused(t *testing.T) {
{maxKeyTTL: time.Minute}, {maxKeyTTL: time.Minute},
{vip: "10.0.0.160/24"}, {vip: "10.0.0.160/24"},
{standbys: true, vip: "10.0.0.160"}, {standbys: true, vip: "10.0.0.160"},
{slurm: true, noNode: true},
} { } {
if err := in.checkChoices(); err == nil { if err := in.checkChoices(); err == nil {
t.Errorf("taken: %+v", in) t.Errorf("taken: %+v", in)
+7 -23
View File
@@ -329,7 +329,7 @@ func (in *serverInit) ensureNode(ctx context.Context) (string, error) {
if classes, n := gpuClasses(ctx); n > 0 { if classes, n := gpuClasses(ctx); n > 0 {
in.kept("%d NVIDIA GPU(s) here: %s — the node will use them", n, strings.Join(classes, ", ")) in.kept("%d NVIDIA GPU(s) here: %s — the node will use them", n, strings.Join(classes, ", "))
} else if hasNVIDIADevice() { } else if discovery.HasNVIDIADevice() {
fmt.Fprintf(in.out, " !! an NVIDIA card is on the PCI bus but nvidia-smi does not answer: "+ fmt.Fprintf(in.out, " !! an NVIDIA card is on the PCI bus but nvidia-smi does not answer: "+
"install the driver and run init again to serve on it; until then this node is CPU only\n") "install the driver and run init again to serve on it; until then this node is CPU only\n")
} else { } else {
@@ -398,7 +398,9 @@ labels:
// checks its own llama.cpp, the CUDA build when it has a GPU, and a new // checks its own llama.cpp, the CUDA build when it has a GPU, and a new
// machine is often busy with its own package updates first. // machine is often busy with its own package updates first.
client := in.adminAPI() client := in.adminAPI()
last := "" // Each engine's line once, and again only when it changes: the engines'
// states come back together on every look.
seen := map[string]string{}
err = waitFor(ctx, 20*time.Minute, "this node to be ready with an engine", func() (bool, error) { err = waitFor(ctx, 20*time.Minute, "this node to be ready with an engine", func() (bool, error) {
var nodes []adminapi.NodeView var nodes []adminapi.NodeView
if err := client.get(ctx, "/api/v1/nodes", &nodes); err != nil { if err := client.get(ctx, "/api/v1/nodes", &nodes); err != nil {
@@ -412,9 +414,9 @@ labels:
return true, nil return true, nil
} }
for _, p := range n.Provision { for _, p := range n.Provision {
if line := fmt.Sprintf("%s %s: %s %s", p.Engine, p.Version, p.State, p.Detail); line != last { if line := provisionLine(p.Engine, p.Version, p.State, p.Detail); seen[p.Engine] != line {
fmt.Fprintf(in.out, " .. %s\n", strings.TrimSpace(line)) fmt.Fprintf(in.out, " .. %s\n", line)
last = line seen[p.Engine] = line
} }
// Only llama.cpp is waited for: vLLM installs for minutes after, // Only llama.cpp is waited for: vLLM installs for minutes after,
// and one this node cannot run is not a reason to stop here. // and one this node cannot run is not a reason to stop here.
@@ -432,24 +434,6 @@ labels:
return nodeID, nil return nodeID, nil
} }
// hasNVIDIADevice reports an NVIDIA card on the PCI bus, driver or not, so a
// machine whose GPU is invisible for want of a driver says so instead of
// quietly serving on CPU.
func hasNVIDIADevice() bool {
vendors, _ := filepath.Glob("/sys/bus/pci/devices/*/vendor")
for _, v := range vendors {
b, err := os.ReadFile(v)
if err == nil && strings.TrimSpace(string(b)) == "0x10de" {
class, _ := os.ReadFile(filepath.Join(filepath.Dir(v), "class"))
// 0x03xxxx is a display or 3D controller; NVIDIA also makes bridges.
if strings.HasPrefix(strings.TrimSpace(string(class)), "0x03") {
return true
}
}
}
return false
}
// --- the starter model --- // --- the starter model ---
// starterModel is the catalog entry: a GPU profile first when there is a card, // starterModel is the catalog entry: a GPU profile first when there is a card,
+4
View File
@@ -156,6 +156,10 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
if s.Stopped { if s.Stopped {
fmt.Fprintln(out, "SERVICE STOPPED: no model runs and requests are answered 503 (llmgw start)") fmt.Fprintln(out, "SERVICE STOPPED: no model runs and requests are answered 503 (llmgw start)")
} }
if s.NoAlerts {
fmt.Fprintln(out, "ALERTS nowhere: a lost node or a judge down is told nobody "+
"(llmgw ctl config set notify.webhooks, then llmgw ctl notify test)")
}
if s.GuardOff { if s.GuardOff {
fmt.Fprintln(out, "GUARD off: no request is checked, whatever the policies say (llmgw ctl guard on)") fmt.Fprintln(out, "GUARD off: no request is checked, whatever the policies say (llmgw ctl guard on)")
} }
+23 -14
View File
@@ -21,9 +21,13 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
Long: "A node trains an adapter for a catalog model on pairs (llmgw ctl kb dataset\n" + Long: "A node trains an adapter for a catalog model on pairs (llmgw ctl kb dataset\n" +
"writes them from a knowledge base; read them through first). It trains with\n" + "writes them from a knowledge base; read them through first). It trains with\n" +
"its installed vLLM's torch and a small overlay, as the engines' user, on a card\n" + "its installed vLLM's torch and a small overlay, as the engines' user, on a card\n" +
"with --gpu-memory free — beside an engine, if it leaves that much. What it\n" + "with --gpu-memory free — beside an engine, if it leaves that much. Without\n" +
"makes is kept by the control plane: llmgw ctl train download, or publish it\n" + "--base-repo the weights are found from where the model's came, without\n" +
"as the model's adapter.\n\n" + "--gpu-memory from their size, and without --node it goes to the node whose\n" +
"card has the most free. What it makes is kept by the control plane: llmgw ctl\n" +
"train download, or publish it as the model's adapter (--publish does that by\n" +
"itself when it is done). People in a project can do all this from the portal's\n" +
"\"Customise a model\" page.\n\n" +
"An adapter learns how answers sound — words, format, tone — far better than\n" + "An adapter learns how answers sound — words, format, tone — far better than\n" +
"what they say. Facts belong to the knowledge base.", "what they say. Facts belong to the knowledge base.",
} }
@@ -33,11 +37,12 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
var gpuMem int var gpuMem int
var epochs, rank int var epochs, rank int
var lr float64 var lr float64
var noGGUF bool var noGGUF, publishWhenDone bool
start := &cobra.Command{ start := &cobra.Command{
Use: "start NAME --model MODEL --base-repo OWNER/NAME --node NODE --data FILE", Use: "start NAME --model MODEL --data FILE",
Short: "Queue an adapter's training on a node", Short: "Queue an adapter's training on a node",
Example: " llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \\\n" + Example: " llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct --data policies.jsonl --public --publish\n" +
" llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \\\n" +
" --base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl", " --base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl",
Args: cobra.ExactArgs(1), Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error { RunE: func(cmd *cobra.Command, args []string) error {
@@ -46,7 +51,7 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
return err return err
} }
req := adminapi.TrainRequest{Name: args[0], Model: model, BaseRepo: base, BaseRevision: rev, Node: node, req := adminapi.TrainRequest{Name: args[0], Model: model, BaseRepo: base, BaseRevision: rev, Node: node,
GPUMemoryMiB: gpuMem, Data: string(b), Params: map[string]any{}, Owner: owner, Public: public} GPUMemoryMiB: gpuMem, Data: string(b), Params: map[string]any{}, Owner: owner, Public: public, Publish: publishWhenDone}
if cmd.Flags().Changed("epochs") { if cmd.Flags().Changed("epochs") {
req.Params["epochs"] = epochs req.Params["epochs"] = epochs
} }
@@ -69,8 +74,8 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
if v.Owner != "" { if v.Owner != "" {
who = v.Owner + " only" who = v.Owner + " only"
} }
fmt.Fprintf(cmd.OutOrStdout(), "queued %s: %s for %s on %s, %d pairs; it will be seen by %s\n", fmt.Fprintf(cmd.OutOrStdout(), "queued %s: %s for %s on %s, %d pairs from %s, %d MiB of a card; it will be seen by %s\n",
v.ID, v.Name, v.Model, v.Node, v.Pairs, who) v.ID, v.Name, v.Model, or(v.Node, "the node with a card free enough"), v.Pairs, v.BaseRepo, v.GPUMemoryMiB, who)
fmt.Fprintf(cmd.OutOrStdout(), "follow it with: llmgw ctl train show %s --follow\n", v.ID) fmt.Fprintf(cmd.OutOrStdout(), "follow it with: llmgw ctl train show %s --follow\n", v.ID)
return nil return nil
}, },
@@ -79,8 +84,9 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
f.StringVar(&model, "model", "", "the catalog model the adapter is for") f.StringVar(&model, "model", "", "the catalog model the adapter is for")
f.StringVar(&base, "base-repo", "", "the weights to train on, as safetensors on Hugging Face (owner/name)") f.StringVar(&base, "base-repo", "", "the weights to train on, as safetensors on Hugging Face (owner/name)")
f.StringVar(&rev, "base-revision", "", "a revision of them (default: the newest)") f.StringVar(&rev, "base-revision", "", "a revision of them (default: the newest)")
f.StringVar(&node, "node", "", "the node that trains") f.StringVar(&node, "node", "", "the node that trains (default: the one whose card has the most free)")
f.IntVar(&gpuMem, "gpu-memory", 4096, "MiB of a card the job needs free") f.IntVar(&gpuMem, "gpu-memory", 0, "MiB of a card the job needs free (default: from the weights' size)")
f.BoolVar(&publishWhenDone, "publish", false, "publish it as the model's adapter when it is done, restarting a llama.cpp model's replicas")
f.StringVar(&data, "data", "", "the pairs, NDJSON of {\"q\", \"a\"}") f.StringVar(&data, "data", "", "the pairs, NDJSON of {\"q\", \"a\"}")
f.IntVar(&epochs, "epochs", 6, "passes over the pairs") f.IntVar(&epochs, "epochs", 6, "passes over the pairs")
f.IntVar(&rank, "rank", 16, "the adapter's rank") f.IntVar(&rank, "rank", 16, "the adapter's rank")
@@ -89,7 +95,7 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
f.StringVar(&owner, "owner", "", "who the adapter belongs to — org:, team:, project: (path or ID), user: or key: — and only they see it "+ f.StringVar(&owner, "owner", "", "who the adapter belongs to — org:, team:, project: (path or ID), user: or key: — and only they see it "+
"(default: the knowledge base's owner, when the pairs come from one)") "(default: the knowledge base's owner, when the pairs come from one)")
f.BoolVar(&public, "public", false, "every caller of the model sees it; refused for pairs from a knowledge base") f.BoolVar(&public, "public", false, "every caller of the model sees it; refused for pairs from a knowledge base")
for _, req := range []string{"model", "base-repo", "node", "data"} { for _, req := range []string{"model", "data"} {
_ = start.MarkFlagRequired(req) _ = start.MarkFlagRequired(req)
} }
c.AddCommand(start) c.AddCommand(start)
@@ -138,8 +144,11 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
} }
line := fmt.Sprintf("%s %s %s loss %s peak %s %s", j.State, trainProgress(j), j.Node, trainLoss(j), trainPeak(j), j.Message) line := fmt.Sprintf("%s %s %s loss %s peak %s %s", j.State, trainProgress(j), j.Node, trainLoss(j), trainPeak(j), j.Message)
if !follow { if !follow {
fmt.Fprintf(out, "%s: adapter %s for %s, owned by %s, on %s (base %s), %d pairs\n", j.ID, j.Name, j.Model, fmt.Fprintf(out, "%s: adapter %s for %s, owned by %s, on %s (base %s), %d pairs, %d MiB of a card\n", j.ID, j.Name, j.Model,
or(j.Owner, "everyone"), j.Node, j.BaseRepo, j.Pairs) or(j.Owner, "everyone"), or(j.Node, "the node whose card has the most free"), j.BaseRepo, j.Pairs, j.GPUMemoryMiB)
if j.Project != "" {
fmt.Fprintf(out, "started from the portal in project %s by %s; published when done: %t\n", j.Project, j.CreatedBy, j.Publish)
}
fmt.Fprintln(out, line) fmt.Fprintln(out, line)
if len(j.Artifacts) > 0 { if len(j.Artifacts) > 0 {
fmt.Fprintf(out, "made: %s (llmgw ctl train download %s --kind KIND)\n", strings.Join(j.Artifacts, ", "), j.ID) fmt.Fprintf(out, "made: %s (llmgw ctl train download %s --kind KIND)\n", strings.Join(j.Artifacts, ", "), j.ID)
+64 -8
View File
@@ -20,6 +20,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility" "gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve" "gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/release" "gitea.mynodes.xyz/ilker/llmgw/internal/release"
"gitea.mynodes.xyz/ilker/llmgw/internal/version" "gitea.mynodes.xyz/ilker/llmgw/internal/version"
) )
@@ -406,6 +407,48 @@ func (m consoleModel) modelMenu(md adminapi.ModelView) *menuDialog {
}}) }})
return nil return nil
}}, }},
{"Replicas per profile…", func(m *consoleModel) tea.Cmd {
var now []string
for _, p := range md.Profiles {
now = append(now, fmt.Sprintf("%s=%d", p, md.ProfileReplicas[p]))
}
m.openForm(&formDialog{title: "Replicas per profile of " + md.Name,
note: "How many replicas run on each profile, all answering under the model's name: seven on cards and " +
"four on CPUs, the router filling the cards first. A profile not named keeps its count; cpu=N with " +
"the memory below adds a profile on CPUs to a GGUF model that has none. Those short start first, " +
"then those over stop.",
fields: []formField{
textF("replicas", strings.Join(now, " "), "profile=count, separated by spaces"),
textF("CPU memory GB", "", "only to add a profile on CPUs: what a replica reserves"),
textF("CPU cores", "", "of a replica on CPUs; empty is half the node's"),
},
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
req := adminapi.ProfileReplicasRequest{Profiles: map[string]int{}}
for _, f := range strings.Fields(v[0]) {
name, c, ok := strings.Cut(f, "=")
n, err := strconv.Atoi(c)
if !ok || err != nil || n < 0 || name == "" {
return nil, fmt.Errorf("%q: a profile and a count, as gpu=7", f)
}
req.Profiles[name] = n
}
if v[1] != "" {
mem, err := strconv.Atoi(v[1])
if err != nil || mem < 1 {
return nil, fmt.Errorf("the CPU memory is a whole number of GB")
}
cores := 0
if v[2] != "" {
if cores, err = strconv.Atoi(v[2]); err != nil || cores < 1 {
return nil, fmt.Errorf("the CPU cores are a whole number from 1")
}
}
req.CPU = &modelsvc.CPUProfile{MemoryGB: mem, CPUCores: cores}
}
return m.call("Replicas of "+md.Name, http.MethodPost, path+"/replicas", req, nil), nil
}})
return nil
}},
{"Resources…", func(m *consoleModel) tea.Cmd { {"Resources…", func(m *consoleModel) tea.Cmd {
profiles := md.Profiles profiles := md.Profiles
if len(profiles) == 0 { if len(profiles) == 0 {
@@ -600,17 +643,30 @@ func (m consoleModel) nodeMenu(n adminapi.NodeView) *menuDialog {
menuItem{"Delete model files nothing uses", func(m *consoleModel) tea.Cmd { menuItem{"Delete model files nothing uses", func(m *consoleModel) tea.Cmd {
return m.call("Unused files on "+n.ID, http.MethodPost, "/api/v1/weights/collect", map[string]string{"node": n.ID}, return m.call("Unused files on "+n.ID, http.MethodPost, "/api/v1/weights/collect", map[string]string{"node": n.ID},
func(raw json.RawMessage) string { func(raw json.RawMessage) string {
var rs []struct { // What the node no longer lists, and what it kept.
Files int `json:"files"` var rs []adminapi.CollectResult
Bytes int64 `json:"bytes"`
}
_ = json.Unmarshal(raw, &rs) _ = json.Unmarshal(raw, &rs)
var files int var files, left int
var b int64 var b, lb int64
var errs []string
for _, r := range rs { for _, r := range rs {
files, b = files+r.Files, b+r.Bytes files, b, left, lb = files+r.Files, b+r.Bytes, left+r.Left, lb+r.LeftBytes
if r.Error != "" {
errs = append(errs, r.Error)
}
} }
return fmt.Sprintf("%d files, %.1f GB freed.", files, float64(b)/(1<<30)) if len(rs) == 0 {
return "Nothing on " + n.ID + " that no model uses."
}
say := fmt.Sprintf("%d files removed, %.1f GB freed.", files, float64(b)/(1<<30))
if left > 0 {
say += fmt.Sprintf(" %d files (%.1f GB) are still there: the node kept them, and its agent's log "+
"says why (journalctl -u llmgw-agent there).", left, float64(lb)/(1<<30))
}
if len(errs) > 0 {
say += " " + strings.Join(errs, "; ")
}
return say
}) })
}}, }},
menuItem{"Remove from the cluster…", func(m *consoleModel) tea.Cmd { menuItem{"Remove from the cluster…", func(m *consoleModel) tea.Cmd {
+28 -2
View File
@@ -57,7 +57,7 @@ type installForm struct {
standby *installPart standby *installPart
vipF *setupField vipF *setupField
control, store, mirror, node, starter, media, judge, firewall, alerts, limits, turns, bundle *installPart control, store, mirror, node, starter, media, slurm, judge, firewall, alerts, limits, turns, bundle *installPart
admin, hosts, language, signup *setupField admin, hosts, language, signup *setupField
storeKind *setupField storeKind *setupField
@@ -282,6 +282,31 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
summary: func() string { return "ComfyUI; a picture model and a 3D model once a card fits" }, summary: func() string { return "ComfyUI; a picture model and a 3D model once a card fits" },
} }
// --- Slurm ---
f.slurm = &installPart{
title: "Slurm", on: in.slurm,
about: "A Slurm cluster beside llmgw, for training and other batch work: slurmctld on this machine, " +
"slurmd on every node, those that join later too. The agents install munge and Slurm from the " +
"machines' own package sources, and on this machine slurmdbd, the jobs' records, with a MariaDB " +
"that answers this machine alone (Slurm keeps its records in MySQL or MariaDB only, not in the " +
"cluster's PostgreSQL). Its port 6819, with Slurm's others, answers the cluster's networks only.\n\n" +
"Each node stays llmgw's, drained in Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm: " +
"its models move off first. 'llmgw ctl slurm reclaim NODE' takes it back once its jobs end. Jobs " +
"are submitted on this machine with sbatch and srun.\n\n" +
"Every Slurm machine sees /shared, exported over NFS from this machine (homes, datasets, the jobs' " +
"output); 'llmgw ctl slurm shared' points it at the site's own storage instead. Each llmgw account " +
"is a Linux user there, the same UID on every machine, logging in with the SSH keys given in the " +
"portal; 'llmgw ctl slurm users off' leaves users to the site's directory.",
off: "no Slurm; 'llmgw ctl slurm setup' adds it later",
summary: func() string { return "slurmctld here, slurmd on every node, /shared from here; none lent" },
needs: func() string {
if !f.node.on {
return "slurmctld runs on this machine's node: turn the node on, or Slurm off"
}
return ""
},
}
// --- the guardrails' judge --- // --- the guardrails' judge ---
j := in.judgeFor() j := in.judgeFor()
f.judgeFail = choiceField("When no judge answers in time", []string{"refuse", "serve flagged"}, f.judgeFail = choiceField("When no judge answers in time", []string{"refuse", "serve flagged"},
@@ -421,7 +446,7 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
if ha { if ha {
f.parts = append(f.parts, f.standby) f.parts = append(f.parts, f.standby)
} }
f.parts = append(f.parts, f.mirror, f.node, f.starter, f.media, f.judge, f.firewall, f.parts = append(f.parts, f.mirror, f.node, f.starter, f.media, f.slurm, f.judge, f.firewall,
f.alerts, f.limits, f.turns, f.bundle) f.alerts, f.limits, f.turns, f.bundle)
return f return f
} }
@@ -653,6 +678,7 @@ func (f *installForm) apply(in *serverInit) error {
in.noModel = !f.starter.on in.noModel = !f.starter.on
in.starter, in.starterCtx = starter, ctx in.starter, in.starterCtx = starter, ctx
in.media = f.media.on in.media = f.media.on
in.slurm = f.slurm.on
in.noGuardModel = !f.judge.on in.noGuardModel = !f.judge.on
in.judge = judge in.judge = judge
in.noFirewall = !f.firewall.on in.noFirewall = !f.firewall.on
+12 -1
View File
@@ -24,6 +24,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi" "gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve" "gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/docgen" "gitea.mynodes.xyz/ilker/llmgw/internal/docgen"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf" "gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/settings" "gitea.mynodes.xyz/ilker/llmgw/internal/settings"
@@ -50,6 +51,7 @@ const (
cFederation cFederation
cMirror cMirror
cJobs cJobs
cSlurm
cLogs cLogs
cAccounts cAccounts
cKeys cKeys
@@ -71,7 +73,7 @@ const (
cCount cCount
) )
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "MCP servers", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Sign-in", "Audit", "Manual", var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "MCP servers", "Federation", "Mirror", "Jobs", "Slurm", "Logs", "Accounts", "API keys", "Organizations", "Sign-in", "Audit", "Manual",
"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Serve the guardrails' judge", "Upgrade the cluster", "server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Serve the guardrails' judge", "Upgrade the cluster",
"Restart all replicas", "Back up now", "Wipe this machine", "Leave"} "Restart all replicas", "Back up now", "Wipe this machine", "Leave"}
@@ -92,6 +94,8 @@ type consoleData struct {
// federation is the other llmgw clusters whose models are offered // federation is the other llmgw clusters whose models are offered
// here; empty on a control plane without a store. // here; empty on a control plane without a store.
federation adminapi.FederationView federation adminapi.FederationView
// slurm is the Slurm cluster beside llmgw; off when it has none.
slurm slurmsvc.Status
// presets is the models that come with llmgw as they stand here; nil // presets is the models that come with llmgw as they stand here; nil
// from a control plane that cannot say. // from a control plane that cannot say.
presets *adminapi.PresetsView presets *adminapi.PresetsView
@@ -284,6 +288,7 @@ func (m consoleModel) fetch() tea.Cmd {
_ = c.get(ctx, "/api/v1/jobs", &d.jobs) _ = c.get(ctx, "/api/v1/jobs", &d.jobs)
_ = c.get(ctx, "/api/v1/audit?since=7d&limit=300", &d.audit) _ = c.get(ctx, "/api/v1/audit?since=7d&limit=300", &d.audit)
_ = c.get(ctx, "/api/v1/federation", &d.federation) _ = c.get(ctx, "/api/v1/federation", &d.federation)
_ = c.get(ctx, "/api/v1/slurm", &d.slurm)
var presets adminapi.PresetsView var presets adminapi.PresetsView
if c.get(ctx, "/api/v1/presets", &presets) == nil { if c.get(ctx, "/api/v1/presets", &presets) == nil {
d.presets = &presets d.presets = &presets
@@ -588,6 +593,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
case cFederation: case cFederation:
m.dialog = "error" m.dialog = "error"
m.dialogTitle, m.dialogText = peerDialog(m.data.federation.Peers[m.row]) m.dialogTitle, m.dialogText = peerDialog(m.data.federation.Peers[m.row])
case cSlurm:
m.menuD = m.slurmMenu(m.data.slurm.Nodes[m.row])
case cServe: case cServe:
return m, m.fetchRepo(m.hf.models[m.row].ID) return m, m.fetchRepo(m.hf.models[m.row].ID)
case cAccounts: case cAccounts:
@@ -1110,6 +1117,8 @@ func (m consoleModel) rowCount() int {
return len(m.data.audit) return len(m.data.audit)
case cFederation: case cFederation:
return len(m.data.federation.Peers) return len(m.data.federation.Peers)
case cSlurm:
return len(m.data.slurm.Nodes)
case cServe: case cServe:
return len(m.hf.models) return len(m.hf.models)
case cAccounts: case cAccounts:
@@ -1369,6 +1378,8 @@ func (m consoleModel) panel() (title, body, keys string) {
return m.auditPanel(pw) return m.auditPanel(pw)
case cFederation: case cFederation:
return m.federationPanel(pw) return m.federationPanel(pw)
case cSlurm:
return m.slurmPanel(pw)
case cMonitor: case cMonitor:
return m.monitorPanel(pw) return m.monitorPanel(pw)
case cServe: case cServe:
+24 -1
View File
@@ -107,7 +107,15 @@ func optionLabel(o modelplan.Option) string {
if o.FreeNow { if o.FreeNow {
now = sOK.Render("free") now = sOK.Render("free")
} }
return fmt.Sprintf("%s · %s · %s", o.Label(), o.MemoryText(), now) label := fmt.Sprintf("%s · %s · %s", o.Label(), o.MemoryText(), now)
if c := speedCell(o.TokensPerSecond); c != "" {
style := sDim
if o.TokensPerSecond < modelplan.CPUTooSlowTPS {
style = sWarn
}
label += " · " + style.Render(c)
}
return label
} }
// repoMenu offers a repository's files, each with the best way it runs here; // repoMenu offers a repository's files, each with the best way it runs here;
@@ -196,16 +204,26 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
note += fmt.Sprintf("\nMemory for a %d-token context, each way it runs:", p.Context) note += fmt.Sprintf("\nMemory for a %d-token context, each way it runs:", p.Context)
for _, o := range opts { for _, o := range opts {
note += "\n " + o.Label() + ": " + o.MemoryText() note += "\n " + o.Label() + ": " + o.MemoryText()
if w := slowWarning(o); w != "" && o.CPU {
note += "\n " + sWarn.Render(w)
}
} }
if p.KVBytesPerToken > 0 { if p.KVBytesPerToken > 0 {
note += fmt.Sprintf("\nThe context costs %.2f GB per 1000 tokens, the whole model; a longer one is checked\n"+ note += fmt.Sprintf("\nThe context costs %.2f GB per 1000 tokens, the whole model; a longer one is checked\n"+
"before it is served.", modelplan.GB(p.KVBytesPerToken*1000)) "before it is served.", modelplan.GB(p.KVBytesPerToken*1000))
} }
note += "\nThe mirror fetches it and checks it, it joins the catalog, and a replica is started\nand asked to say hello." note += "\nThe mirror fetches it and checks it, it joins the catalog, and a replica is started\nand asked to say hello."
// The longest context the first way holds, up to what the model was
// trained for: a choice of another way is checked again on submitting.
ctxDefault := "4096" ctxDefault := "4096"
if a.Format == "safetensors" { if a.Format == "safetensors" {
ctxDefault = strconv.Itoa(p.Context) ctxDefault = strconv.Itoa(p.Context)
} }
if len(opts) > 0 {
if most := opts[0].MostContext(p.KVBytesPerToken, p.Trained); most > 0 {
ctxDefault = strconv.Itoa(most)
}
}
if len(opts) > 0 && opts[0].Engine == "strata" { if len(opts) > 0 && opts[0].Engine == "strata" {
ctxDefault = "32768" ctxDefault = "32768"
} }
@@ -238,6 +256,11 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
Description: v[3], Format: a.Format} Description: v[3], Format: a.Format}
if a.Format == "gguf" { if a.Format == "gguf" {
req.File = a.Name req.File = a.Name
// A context longer than the default pool is the pool: one
// slot of it, where the server would refuse the request.
if ctxLen > modelserve.DefaultCtxPool {
req.CtxPool = ctxLen
}
} }
o := opts[slices.Index(where, v[4])] o := opts[slices.Index(where, v[4])]
// Asked for a context other than planned: does it still fit? // Asked for a context other than planned: does it still fit?
+2 -2
View File
@@ -221,11 +221,11 @@ func (m consoleModel) userMenu(u store.User) *menuDialog {
items = append(items, items = append(items,
menuItem{"Add to an organization, team or project…", func(m *consoleModel) tea.Cmd { menuItem{"Add to an organization, team or project…", func(m *consoleModel) tea.Cmd {
m.openForm(&formDialog{title: "Membership for " + who, m.openForm(&formDialog{title: "Membership for " + who,
note: "A member of a team or organization may use every project in it.", note: "A member of a team or organization may use every project in it. On an organization, a manager also sees its page and an admin also changes it.",
fields: []formField{ fields: []formField{
choiceF("level", []string{"org", "team", "project"}, "team"), choiceF("level", []string{"org", "team", "project"}, "team"),
textF("which", "", "Acme, Acme/ml or Acme/ml/training"), textF("which", "", "Acme, Acme/ml or Acme/ml/training"),
choiceF("role", []string{store.RoleMember, "manager"}, store.RoleMember), choiceF("role", store.MembershipRoles, store.RoleMember),
}, },
submit: func(m *consoleModel, v []string) (tea.Cmd, error) { submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
if v[1] == "" { if v[1] == "" {
+170
View File
@@ -0,0 +1,170 @@
package cli
import (
"context"
"encoding/json"
"fmt"
"time"
tea "github.com/charmbracelet/bubbletea"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
)
// The console's Slurm view: the Slurm cluster run beside llmgw (BACKLOG P),
// each of its nodes and whose it is now, as `llmgw ctl slurm status` shows
// them. A node is lent to Slurm, or taken back, from its row. Turning Slurm
// on, and choosing its nodes, is `ctl slurm setup` or server.yaml's slurm
// part: a form for it would be server.yaml written again.
func (m consoleModel) slurmPanel(pw int) (string, string, string) {
keys := "→ or Enter open · ↑↓ choose · Q leave"
if m.inPanel {
keys = "↑↓ node · Enter lend or take back · ← back · Q leave"
}
st := m.data.slurm
head := "\n " + sDim.Render("Slurm beside llmgw, on its nodes: training and batch jobs.")
if !st.Enabled {
return "Slurm", head + "\n\n " + sDim.Render("Slurm is off. Turn it on with: "+
"llmgw ctl slurm setup --controller NODE --nodes NODE,... (or --all-nodes)"), keys
}
ctl := sDim.Render("not heard from yet")
switch c := st.Ctld; {
case c == nil:
case c.Error != "":
ctl = sBad.Render(c.Error)
case c.SlurmctldRunning && c.SlurmdbdRunning:
ctl = sOK.Render("slurmctld " + c.Version + " and slurmdbd running")
case c.SlurmctldRunning:
ctl = sOK.Render("slurmctld "+c.Version+" running") + ", " + sWarn.Render("slurmdbd not")
default:
ctl = sBad.Render("slurmctld not running")
}
head += fmt.Sprintf("\n controller %s: %s", st.Controller, ctl)
head += "\n " + sDim.Render(fmt.Sprintf("cluster %s, partition %s; sbatch and srun on %s", st.Cluster, st.Partition, st.Controller))
if st.Shared.Path == "" {
head += "\n shared files: " + sWarn.Render("none") + sDim.Render(" (llmgw ctl slurm shared; output stays on the job's node)")
} else {
head += "\n shared files: " + sharedLine(st.Shared)
}
if st.Users != "" {
head += "\n " + fmt.Sprintf("users: %d from llmgw's accounts", st.UserCount) + sDim.Render(", SSH keys in the portal's Profile")
} else {
head += "\n " + sDim.Render("users: the site's (llmgw ctl slurm users on makes them from llmgw's accounts)")
}
if line := containersLine(st); line != "" {
head += "\n containers: " + line
}
head += "\n " + sDim.Render("A node is llmgw's until lent; lent, its replicas move off first.")
if len(st.Waiting) > 0 {
head += "\n " + sWarn.Render(fmt.Sprintf("not in slurm.conf until they report their hardware: %v", st.Waiting))
}
head += "\n\n"
if len(st.Nodes) == 0 {
return "Slurm", head + " " + sDim.Render("No node runs slurmd."), keys
}
var rows [][]string
for _, n := range st.Nodes {
rows = append(rows, slurmRow(n))
}
return "Slurm", head + table([]string{"NODE", "WHOSE", "IN SLURM", "JOBS", "SLURMD", "SHARED", "NOTE"}, rows, m.row, pw, m.inPanel), keys
}
func slurmRow(n slurmsvc.NodeStatus) []string {
state, jobs, slurmd, shared, note := "-", "-", "-", "-", n.Describe()
if r := n.Status; r != nil {
state = or(r.State, "-")
switch r.Shared {
case "":
case "missing":
shared = sBad.Render(r.Shared)
default:
shared = sOK.Render(r.Shared)
}
jobs = fmt.Sprint(r.Jobs)
slurmd = sBad.Render("stopped")
if r.SlurmdRunning {
slurmd = sOK.Render("running")
}
if r.Error != "" {
note = sBad.Render(r.Error)
}
}
if !n.Connected {
note = sWarn.Render("agent not connected") + "; " + note
}
whose := string(n.Phase)
switch n.Phase {
case slurmsvc.Lent:
whose = sOK.Render("Slurm's")
case slurmsvc.Llmgw:
whose = "llmgw's"
case slurmsvc.Lending, slurmsvc.Reclaiming:
whose = sWarn.Render(whose)
}
return []string{n.Node, whose, state, jobs, slurmd, shared, note}
}
// slurmMenu is what can be done with one of Slurm's nodes.
func (m consoleModel) slurmMenu(n slurmsvc.NodeStatus) *menuDialog {
var items []menuItem
if n.Phase == slurmsvc.Lent || n.Phase == slurmsvc.Lending {
items = append(items, menuItem{"Take back from Slurm…", func(m *consoleModel) tea.Cmd {
m.confirmD = &confirmDialog{title: "Take " + n.Node + " back",
text: "Slurm starts no new job there. llmgw places replicas on it again once the jobs running there have ended.",
yes: func(m *consoleModel) tea.Cmd { return m.slurmLend(n.Node, false) }}
return nil
}})
} else {
items = append(items, menuItem{"Lend to Slurm…", func(m *consoleModel) tea.Cmd {
text := "Its replicas move off first, each replacement up before the original stops; then Slurm runs jobs there."
if n.Replicas > 0 {
text = fmt.Sprintf("Its %d replica(s) move off first, each replacement up before the original stops; "+
"a model with nowhere else to go stops serving. Then Slurm runs jobs there.", n.Replicas)
}
m.confirmD = &confirmDialog{title: "Lend " + n.Node + " to Slurm", text: text,
yes: func(m *consoleModel) tea.Cmd { return m.slurmLend(n.Node, true) }}
return nil
}})
}
return &menuDialog{title: n.Node, items: items}
}
// slurmLend puts a node in slurm.lent, or takes it out, as `llmgw ctl slurm
// lend|reclaim` does: the newest settings, changed and put back.
func (m *consoleModel) slurmLend(node string, add bool) tea.Cmd {
title, note, done := "Lend "+node, "lent to Slurm from the console", node+" is being lent to Slurm: its replicas move off first."
if !add {
title, note, done = "Take back "+node, "taken back from Slurm in the console",
node+" is being taken back: llmgw has it once its Slurm jobs end."
}
m.busy = title
c := m.client
return func() tea.Msg {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
var cur adminapi.SettingsView
if err := c.get(ctx, "/api/v1/settings", &cur); err != nil {
return actionDoneMsg{title: title, err: err}
}
doc, err := settings.FromBytes("settings", []byte(cur.Body))
if err == nil {
err = lendChange(doc, []string{node}, add)
}
var body []byte
if err == nil {
body, err = doc.Bytes()
}
if err != nil {
return actionDoneMsg{title: title, err: err}
}
req, _ := json.Marshal(adminapi.SettingsRequest{Base: cur.Version, Body: string(body), Note: note})
var v adminapi.SettingsView
if err := c.put(ctx, "/api/v1/settings", req, &v); err != nil {
return actionDoneMsg{title: title, err: err}
}
return actionDoneMsg{title: title, text: done}
}
}
+94
View File
@@ -0,0 +1,94 @@
package cli
import (
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
tea "github.com/charmbracelet/bubbletea"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
)
func TestTheConsoleLendsANodeToSlurmFromItsRow(t *testing.T) {
var mu sync.Mutex
var put adminapi.SettingsRequest
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v1/settings":
_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 7,
Body: "slurm:\n enabled: true\n controller: node06\n nodes: [node05, cuda01]\n"})
case r.Method == http.MethodPut && r.URL.Path == "/api/v1/settings":
b, _ := io.ReadAll(r.Body)
mu.Lock()
_ = json.Unmarshal(b, &put)
mu.Unlock()
_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 8})
default:
_, _ = w.Write([]byte("[]"))
}
}))
t.Cleanup(srv.Close)
m := newConsoleModel(srv.URL, "/nonexistent/server.yaml")
m.loaded, m.menu, m.inPanel = true, cSlurm, true
m.data.slurm = slurmsvc.Status{Enabled: true, Cluster: "llmgw", Controller: "node06", Partition: "gpu",
Ctld: &slurmsvc.Reported{SlurmctldRunning: true, SlurmdbdRunning: true, Version: "23.11.4"},
Shared: config.SharedStorage{Path: "/shared", Server: "node06"}, Users: "llmgw", UserCount: 2,
Nodes: []slurmsvc.NodeStatus{
{Node: "node05", Phase: slurmsvc.Lent, Connected: true,
Status: &slurmsvc.Reported{State: "MIXED", Jobs: 2, SlurmdRunning: true, Shared: "mounted"}},
{Node: "cuda01", Phase: slurmsvc.Llmgw, Replicas: 3, Connected: true,
Status: &slurmsvc.Reported{State: "IDLE+DRAIN", SlurmdRunning: true}},
}}
var tm tea.Model = m
view := tm.View()
for _, want := range []string{"slurmctld 23.11.4 and slurmdbd running", "/shared, exported over NFS by node06", "users: 2", "node05", "Slurm's", "MIXED", "cuda01", "llmgw's"} {
if !strings.Contains(view, want) {
t.Errorf("the Slurm view has no %q:\n%s", want, view)
}
}
// cuda01, the second row: lent, after saying its replicas move.
tm = press(tm, "down", "enter")
if !strings.Contains(tm.View(), "Lend to Slurm") {
t.Fatalf("no lend in the menu:\n%s", tm.View())
}
tm = press(tm, "enter")
if !strings.Contains(tm.View(), "3 replica(s) move off first") {
t.Fatalf("the question does not say what moves:\n%s", tm.View())
}
tm = pressRun(tm, "y")
mu.Lock()
defer mu.Unlock()
if put.Base != 7 || !strings.Contains(put.Body, "lent:") || !strings.Contains(put.Body, "cuda01") {
t.Fatalf("the settings put back: %+v", put)
}
if !strings.Contains(tm.View(), "cuda01 is being lent to Slurm") {
t.Errorf("not said:\n%s", tm.View())
}
}
func TestTheConsoleTakesALentNodeBack(t *testing.T) {
m, _ := consoleWith(t, cSlurm)
m.data.slurm = slurmsvc.Status{Enabled: true, Controller: "node06",
Nodes: []slurmsvc.NodeStatus{{Node: "node05", Phase: slurmsvc.Lent, Connected: true}}}
var tm tea.Model = m
tm = press(tm, "enter")
if v := tm.View(); !strings.Contains(v, "Take back from Slurm") || strings.Contains(v, "Lend to Slurm") {
t.Fatalf("a lent node's menu:\n%s", v)
}
}
func TestTheConsoleSaysHowToTurnSlurmOn(t *testing.T) {
m, _ := consoleWith(t, cSlurm)
var tm tea.Model = m
if !strings.Contains(tm.View(), "llmgw ctl slurm setup") {
t.Fatalf("Slurm off says nothing of turning it on:\n%s", tm.View())
}
}
+8 -3
View File
@@ -171,7 +171,10 @@ func newUserStateCmd(open userOpener, verb string, state store.UserState, short
if err := s.DeleteUserSessions(cmd.Context(), u.ID); err != nil { if err := s.DeleteUserSessions(cmd.Context(), u.ID); err != nil {
return err return err
} }
cmd.PrintErrln("Their sessions were closed. Their API keys still work — revoke those separately.") // The keys are refused by their owner's state, which every
// router reads again within auth.cache_ttl.
cmd.PrintErrln("Their sessions were closed. Their API keys are refused within auth.cache_ttl " +
"(30 s by default), and work again if they are reinstated.")
} }
fmt.Fprintf(cmd.OutOrStdout(), "%s is now %s\n", u.Email, state) fmt.Fprintf(cmd.OutOrStdout(), "%s is now %s\n", u.Email, state)
return nil return nil
@@ -391,7 +394,8 @@ func newUserJoinCmd(open userOpener) *cobra.Command {
Long: "A person can be a member of any number of them. A member of a team or\n" + Long: "A person can be a member of any number of them. A member of a team or\n" +
"organization may use every project in it.", "organization may use every project in it.",
Example: " llmgw users join ayse@example.com --team Acme/ml\n" + Example: " llmgw users join ayse@example.com --team Acme/ml\n" +
" llmgw users join ayse@example.com --project Scania/research/vision", " llmgw users join ayse@example.com --project Scania/research/vision\n" +
" llmgw users join lars@example.com --org Scania --role admin",
Args: cobra.ExactArgs(1), Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error { RunE: func(cmd *cobra.Command, args []string) error {
s, err := open(cmd) s, err := open(cmd)
@@ -415,7 +419,8 @@ func newUserJoinCmd(open userOpener) *cobra.Command {
}, },
} }
where.bind(cmd) where.bind(cmd)
cmd.Flags().StringVar(&role, "role", store.RoleMember, "member, or manager: with --org, may see the organization's page") cmd.Flags().StringVar(&role, "role", store.RoleMember,
"member; with --org also manager, who sees the organization's page, or admin, who also changes its teams, projects, people and keys")
cmd.Flags().StringVar(&by, "by", "", "who added them, for the record") cmd.Flags().StringVar(&by, "by", "", "who added them, for the record")
return cmd return cmd
} }
+16
View File
@@ -158,6 +158,10 @@ type Server struct {
// Firewall turns llmgw's firewall on every machine on or off. // Firewall turns llmgw's firewall on every machine on or off.
Firewall FirewallSettings `koanf:"firewall"` Firewall FirewallSettings `koanf:"firewall"`
// Slurm is a Slurm cluster run beside llmgw on some of its nodes, and
// which of them it may use now.
Slurm SlurmSettings `koanf:"slurm"`
// Presets is how the models that come with llmgw reach the catalog. // Presets is how the models that come with llmgw reach the catalog.
Presets PresetSettings `koanf:"presets"` Presets PresetSettings `koanf:"presets"`
@@ -466,6 +470,15 @@ type Auth struct {
Pricing map[string]Price `koanf:"pricing"` Pricing map[string]Price `koanf:"pricing"`
DefaultPrice Price `koanf:"default_price"` DefaultPrice Price `koanf:"default_price"`
// GPUHourCost is what an hour of one GPU costs the operator, in the
// units budgets are in: the hardware written off over its years, its
// power and room, and the people who run it, over the GPUs and the
// hours of a month. With CoreHourCost for a CPU core, it is what
// `llmgw ctl prices suggest` prices each model from: the hours its
// replicas held, idle ones too, over the tokens it served.
GPUHourCost float64 `koanf:"gpu_hour_cost"`
CoreHourCost float64 `koanf:"core_hour_cost"`
// FlushInterval and BatchSize decide when usage records are written. // FlushInterval and BatchSize decide when usage records are written.
FlushInterval time.Duration `koanf:"flush_interval"` FlushInterval time.Duration `koanf:"flush_interval"`
BatchSize int `koanf:"batch_size"` BatchSize int `koanf:"batch_size"`
@@ -960,6 +973,9 @@ func (s Server) Validate() error {
if _, err := s.Facility.Policy(); err != nil { if _, err := s.Facility.Policy(); err != nil {
return err return err
} }
if err := s.Slurm.Check(); err != nil {
return err
}
if s.Control.Timezone != "" { if s.Control.Timezone != "" {
if _, err := time.LoadLocation(s.Control.Timezone); err != nil { if _, err := time.LoadLocation(s.Control.Timezone); err != nil {
return fmt.Errorf("control.timezone %q: %w", s.Control.Timezone, err) return fmt.Errorf("control.timezone %q: %w", s.Control.Timezone, err)
+204
View File
@@ -0,0 +1,204 @@
package config
import (
"fmt"
"path"
"regexp"
"slices"
"strings"
)
// SlurmSettings is a Slurm cluster run beside llmgw on some of its nodes
// (BACKLOG P), for training and other batch work llmgw does not schedule.
// The leader tells each named node's agent what to run; the agents install
// it from the machine's package sources and keep it going.
type SlurmSettings struct {
// Enabled runs Slurm on the nodes below. False stops llmgw keeping it;
// what was installed is left installed, and every node is llmgw's.
Enabled bool `koanf:"enabled"`
// ClusterName is Slurm's ClusterName. Empty is "llmgw".
ClusterName string `koanf:"cluster_name"`
// Controller is the llmgw node whose agent runs slurmctld. It needs no
// GPU, and need not be one of Nodes.
Controller string `koanf:"controller"`
// Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained
// in Slurm, until it is also in Lent.
Nodes []string `koanf:"nodes"`
// AllNodes runs slurmd on every llmgw node, those that join later too,
// besides any named in Nodes. Each is llmgw's until it is lent, as any.
AllNodes bool `koanf:"all_nodes"`
// Lent are the nodes of Nodes Slurm may run jobs on now. A node put here
// is drained of llmgw's replicas first, as `ctl drain` does, and resumed
// in Slurm once none is left; a node taken out is drained in Slurm and
// given back to llmgw when its last job ends.
Lent []string `koanf:"lent"`
// Partition is the one partition every node is in. Empty is "gpu".
Partition string `koanf:"partition"`
// Shared is the file system every machine of the Slurm cluster sees at
// the same path: home directories, datasets, the jobs' output and
// checkpoints. Empty path: none, and a batch job's output stays on the
// node it ran on.
Shared SharedStorage `koanf:"shared"`
// Users is who keeps the Linux users that run jobs, the same on every
// machine of the Slurm cluster: "llmgw" makes one for each active llmgw
// account (its login from the e-mail address, a UID of llmgw's from
// 200000 up, its SSH keys from the portal), let in to the controller,
// and to a node only while a job of theirs runs there (pam_slurm_adopt);
// empty leaves users to the site (a directory through sssd, or by hand).
Users string `koanf:"users"`
// Containers lets jobs run in containers: `srun --container-image
// nvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),
// the GPUs and the shared path inside. The agents install both on
// every Slurm machine; jobs take it once every machine has them.
Containers bool `koanf:"containers"`
}
// SharedStorage is where the Slurm cluster's shared files come from: the
// site's NFS export (NFS), an NFS export llmgw keeps on one of its nodes
// (Server), or, with neither, a file system the site mounts itself
// (Lustre, BeeGFS, GPFS), which llmgw only looks for.
type SharedStorage struct {
// Path is where every machine sees it, as /shared.
Path string `koanf:"path"`
// NFS is the site's export, host:/path, mounted at Path on every
// machine of the Slurm cluster.
NFS string `koanf:"nfs"`
// Server is the llmgw node, the controller or one of the nodes, that
// exports Path over NFS to the others: for a site with no storage of
// its own. One machine: not highly available, and no faster than its
// disk and network.
Server string `koanf:"server"`
}
// UsersLlmgw has llmgw make the Slurm cluster's users from its accounts.
const UsersLlmgw = "llmgw"
// HomeDir is where a user's home directory is: under the shared path when
// there is one, else on each machine's own disk.
func (s SlurmSettings) HomeDir(login string) string {
if s.Shared.Path != "" {
return path.Join(s.Shared.Path, "home", login)
}
return path.Join("/home", login)
}
// nfsExport is host:/path.
var nfsExport = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.:\[\]-]*:/[A-Za-z0-9_./-]*$`)
// sharedPath is an absolute path of plain names, which a mount unit's name
// is made from.
var sharedPath = regexp.MustCompile(`^(/[A-Za-z0-9_.-]+)+$`)
func (s SlurmSettings) checkShared() error {
sh := s.Shared
if sh.Path == "" {
if sh.NFS != "" || sh.Server != "" {
return fmt.Errorf("slurm.shared.path is empty: say where every machine sees the shared files, as /shared")
}
return nil
}
if !sharedPath.MatchString(sh.Path) || strings.Contains(sh.Path, "/.") {
return fmt.Errorf("slurm.shared.path %q is not an absolute path of plain names, as /shared", sh.Path)
}
for _, sys := range []string{"/", "/etc", "/usr", "/var", "/var/lib", "/home", "/root", "/boot", "/opt", "/srv", "/tmp", "/proc", "/sys", "/dev", "/run"} {
if path.Clean(sh.Path) == sys {
return fmt.Errorf("slurm.shared.path %s is the system's own directory: take one of its own, as /shared", sh.Path)
}
}
if sh.NFS != "" && sh.Server != "" {
return fmt.Errorf("slurm.shared names both the site's export (nfs) and an llmgw node to export it (server): one of them")
}
if sh.NFS != "" && !nfsExport.MatchString(sh.NFS) {
return fmt.Errorf("slurm.shared.nfs %q is not host:/path", sh.NFS)
}
if sh.Server != "" && sh.Server != s.Controller && !s.AllNodes && !slices.Contains(s.Nodes, sh.Server) {
return fmt.Errorf("slurm.shared.server %q is neither the controller nor one of slurm.nodes", sh.Server)
}
return nil
}
// slurmName is what Slurm takes as a node, cluster or partition name.
var slurmName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]{0,62}$`)
// Cluster is ClusterName or its default.
func (s SlurmSettings) Cluster() string {
if s.ClusterName == "" {
return "llmgw"
}
return s.ClusterName
}
// PartitionName is Partition or its default.
func (s SlurmSettings) PartitionName() string {
if s.Partition == "" {
return "gpu"
}
return s.Partition
}
// IsLent is whether Slurm may use the node now.
func (s SlurmSettings) IsLent(node string) bool {
return s.Enabled && (s.AllNodes || slices.Contains(s.Nodes, node)) && slices.Contains(s.Lent, node)
}
// WithNodes is the settings with every node of ids in Nodes when AllNodes:
// the nodes that run slurmd, as the leader's inventory has them.
func (s SlurmSettings) WithNodes(ids []string) SlurmSettings {
if !s.AllNodes {
return s
}
nodes := slices.Clone(s.Nodes)
for _, id := range ids {
if !slices.Contains(nodes, id) {
nodes = append(nodes, id)
}
}
s.Nodes = nodes
return s
}
// Check refuses settings Slurm could not run on.
func (s SlurmSettings) Check() error {
if !s.Enabled {
return nil
}
if s.Controller == "" {
return fmt.Errorf("slurm.controller is empty: name the node that runs slurmctld")
}
if !slurmName.MatchString(s.Controller) {
return fmt.Errorf("slurm.controller %q is not a name Slurm takes", s.Controller)
}
for _, n := range []struct{ key, v string }{{"slurm.cluster_name", s.ClusterName}, {"slurm.partition", s.Partition}} {
if n.v != "" && !slurmName.MatchString(n.v) {
return fmt.Errorf("%s %q is not a name Slurm takes: letters, digits, _ . -", n.key, n.v)
}
}
seen := map[string]bool{}
for i, n := range s.Nodes {
if !slurmName.MatchString(n) {
return fmt.Errorf("slurm.nodes[%d] %q is not a name Slurm takes", i, n)
}
if seen[n] {
return fmt.Errorf("slurm.nodes names %s twice", n)
}
seen[n] = true
}
for i, n := range s.Lent {
if !seen[n] && !s.AllNodes {
return fmt.Errorf("slurm.lent[%d] %q is not in slurm.nodes: only a node that runs slurmd can be lent", i, n)
}
}
if s.Users != "" && s.Users != UsersLlmgw {
return fmt.Errorf("slurm.users %q: %q has llmgw make them, empty leaves them to the site", s.Users, UsersLlmgw)
}
return s.checkShared()
}
+96
View File
@@ -0,0 +1,96 @@
package config
import (
"strings"
"testing"
)
func TestSlurmCheck(t *testing.T) {
ok := SlurmSettings{Enabled: true, Controller: "node06", Nodes: []string{"cuda01", "site2"}, Lent: []string{"cuda01"}}
if err := ok.Check(); err != nil {
t.Fatal(err)
}
if !ok.IsLent("cuda01") || ok.IsLent("site2") || ok.IsLent("node06") {
t.Error("IsLent is wrong")
}
if ok.Cluster() != "llmgw" || ok.PartitionName() != "gpu" {
t.Error("defaults are wrong")
}
if (SlurmSettings{Lent: []string{"x"}}).Check() != nil {
t.Error("settings that are off were checked")
}
for want, s := range map[string]SlurmSettings{
"controller is empty": {Enabled: true},
"not a name": {Enabled: true, Controller: "bad name"},
"twice": {Enabled: true, Controller: "c", Nodes: []string{"a", "a"}},
"not in slurm.nodes": {Enabled: true, Controller: "c", Nodes: []string{"a"}, Lent: []string{"b"}},
"slurm.partition": {Enabled: true, Controller: "c", Partition: "a b"},
} {
if err := s.Check(); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("%+v: got %v, want %q", s, err, want)
}
}
if (SlurmSettings{Enabled: false, Lent: []string{"a"}}).IsLent("a") {
t.Error("a node is lent with Slurm off")
}
}
func TestSlurmShared(t *testing.T) {
base := SlurmSettings{Enabled: true, Controller: "c", Nodes: []string{"a"}}
for _, sh := range []SharedStorage{{}, {Path: "/shared"}, {Path: "/shared", NFS: "filer.site:/export/hpc"},
{Path: "/data/hpc", Server: "c"}, {Path: "/shared", Server: "a"}} {
s := base
s.Shared = sh
if err := s.Check(); err != nil {
t.Errorf("%+v refused: %v", sh, err)
}
}
for want, sh := range map[string]SharedStorage{
"path is empty": {NFS: "f:/x"},
"plain names": {Path: "shared"},
"system's own": {Path: "/home"},
"one of them": {Path: "/shared", NFS: "f:/x", Server: "c"},
"not host:/path": {Path: "/shared", NFS: "f/x"},
"neither": {Path: "/shared", Server: "z"},
} {
s := base
s.Shared = sh
if err := s.Check(); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("%+v: got %v, want %q", sh, err, want)
}
}
s := base
s.Shared = SharedStorage{Path: "/a/../etc"}
if s.Check() == nil {
t.Error("a path with .. taken")
}
s.Shared, s.Users = SharedStorage{}, "ldap"
if s.Check() == nil {
t.Error("slurm.users ldap taken")
}
s.Users, s.Shared = UsersLlmgw, SharedStorage{Path: "/shared"}
if s.Check() != nil || s.HomeDir("ayse") != "/shared/home/ayse" {
t.Errorf("home %s", s.HomeDir("ayse"))
}
s.Shared = SharedStorage{}
if s.HomeDir("ayse") != "/home/ayse" {
t.Errorf("home %s", s.HomeDir("ayse"))
}
}
func TestSlurmAllNodes(t *testing.T) {
s := SlurmSettings{Enabled: true, Controller: "c", AllNodes: true, Lent: []string{"joined"}}
if err := s.Check(); err != nil {
t.Fatalf("a node lent with all_nodes refused: %v", err)
}
if !s.IsLent("joined") {
t.Fatal("not lent with all_nodes")
}
if got := s.WithNodes([]string{"a", "joined"}).Nodes; len(got) != 2 {
t.Fatalf("nodes %v", got)
}
s.AllNodes = false
if s.Check() == nil || len(s.WithNodes([]string{"a"}).Nodes) != 0 {
t.Fatal("all_nodes off still takes the inventory's nodes")
}
}
+69 -2
View File
@@ -22,6 +22,7 @@ import (
"unicode" "unicode"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve" "gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf" "gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan" "gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
"gitea.mynodes.xyz/ilker/llmgw/internal/release" "gitea.mynodes.xyz/ilker/llmgw/internal/release"
@@ -77,7 +78,9 @@ type API struct {
// tenancy is organizations, keys and people; nil without a store. // tenancy is organizations, keys and people; nil without a store.
tenancy TenancyStore tenancy TenancyStore
// train is adapter training; nil without a store and agents. // train is adapter training; nil without a store and agents.
train *TrainAdmin train *TrainAdmin
// slurm is the Slurm cluster beside llmgw; nil without a store and agents.
slurm SlurmService
settings *Settings settings *Settings
auth *AuthAdmin auth *AuthAdmin
mcp MCPStore mcp MCPStore
@@ -95,6 +98,14 @@ type API struct {
usage Usage usage Usage
quotas Quotas quotas Quotas
ctl *control.Controller ctl *control.Controller
// slurmUsers are the Linux users made from the accounts; nil without a store.
slurmUsers SlurmUsers
// prices are suggested from what each model held and served; nil
// without a store.
prices PriceStore
priceCfg func() PriceConfig
// webhooks counts the settings' notify.webhooks; nil does not say.
webhooks func() int
} }
// New builds the API over a controller. // New builds the API over a controller.
@@ -130,10 +141,13 @@ func (a *API) Register(mux Mux) {
mux.HandleFunc("POST /api/v1/models/{name}/start", a.startModel) mux.HandleFunc("POST /api/v1/models/{name}/start", a.startModel)
mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel) mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources) mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
mux.HandleFunc("POST /api/v1/models/{name}/replicas", a.setProfileReplicas)
a.registerQuotas(mux) a.registerQuotas(mux)
a.registerKB(mux) a.registerKB(mux)
a.registerTenancy(mux) a.registerTenancy(mux)
a.registerTrain(mux) a.registerTrain(mux)
a.registerSlurm(mux)
a.registerPrices(mux)
a.registerGuard(mux) a.registerGuard(mux)
a.registerSettings(mux) a.registerSettings(mux)
a.registerAuth(mux) a.registerAuth(mux)
@@ -259,6 +273,9 @@ type NodeView struct {
// MemBandwidthGBs is how fast its memory reads, in GB/s, measured when // MemBandwidthGBs is how fast its memory reads, in GB/s, measured when
// its agent started; 0 when not reported. A model's speed on the CPU. // its agent started; 0 when not reported. A model's speed on the CPU.
MemBandwidthGBs float64 `json:"mem_bandwidth_gbs,omitempty"` MemBandwidthGBs float64 `json:"mem_bandwidth_gbs,omitempty"`
// MemBandwidthCoreGBs is what one core reads alone, in GB/s; 0 when not
// reported. A CPU replica's threads read at about this each.
MemBandwidthCoreGBs float64 `json:"mem_bandwidth_core_gbs,omitempty"`
// Engines are the inference engines installed here, as the agent found // Engines are the inference engines installed here, as the agent found
// them. Absent means the agent is older than the probe and said nothing, // them. Absent means the agent is older than the probe and said nothing,
// which is not the same as having none — placement treats it that way too, // which is not the same as having none — placement treats it that way too,
@@ -489,6 +506,9 @@ type ModelView struct {
Unplaceable bool `json:"unplaceable,omitempty"` Unplaceable bool `json:"unplaceable,omitempty"`
Fallbacks []string `json:"fallbacks,omitempty"` Fallbacks []string `json:"fallbacks,omitempty"`
Profiles []string `json:"profiles"` Profiles []string `json:"profiles"`
// ProfileReplicas is how many replicas run on each profile, for a model
// whose replicas are counted per profile; absent otherwise.
ProfileReplicas map[string]int `json:"profile_replicas,omitempty"`
// MemoryGB and CPUCores are what one replica reserves, from its first // MemoryGB and CPUCores are what one replica reserves, from its first
// profile; MeasuredMemoryGB is the most one of its replicas uses now, 0 // profile; MeasuredMemoryGB is the most one of its replicas uses now, 0
// when none is measured. // when none is measured.
@@ -723,7 +743,7 @@ func (a *API) Nodes() []NodeView {
v.EngineBuilds[e] = b v.EngineBuilds[e] = b
} }
} }
v.MemBandwidthGBs = n.Host.MemBandwidthGBs v.MemBandwidthGBs, v.MemBandwidthCoreGBs = n.Host.MemBandwidthGBs, n.Host.MemBandwidthCoreGBs
for _, g := range n.GPUs { for _, g := range n.GPUs {
if g.MIG != nil { if g.MIG != nil {
continue // said of the card it is carved from continue // said of the card it is carved from
@@ -734,6 +754,10 @@ func (a *API) Nodes() []NodeView {
} }
if h := n.Host; h.OSID != "" { if h := n.Host; h.OSID != "" {
v.OS = strings.TrimSpace(h.OSID + " " + h.OSVersion + " " + h.Arch) v.OS = strings.TrimSpace(h.OSID + " " + h.OSVersion + " " + h.Arch)
if h.CardWithoutDriver() {
v.Warnings = append(v.Warnings, "an NVIDIA card is on the PCI bus but no driver is loaded: "+
"this node serves on CPU only (install the driver, reboot, check nvidia-smi, restart the agent)")
}
if h.Arch == "amd64" && !h.HasCPUFlag("avx2") { if h.Arch == "amd64" && !h.HasCPUFlag("avx2") {
v.Warnings = append(v.Warnings, "no AVX2: inference runs several times slower "+ v.Warnings = append(v.Warnings, "no AVX2: inference runs several times slower "+
"(a VM needs CPU type \"host\")") "(a VM needs CPU type \"host\")")
@@ -865,6 +889,12 @@ func (a *API) Models() []ModelView {
Canary: m.Canary, CanaryOf: a.ctl.Catalog().CanaryOf(m.Name), Eval: m.Eval, Canary: m.Canary, CanaryOf: a.ctl.Catalog().CanaryOf(m.Name), Eval: m.Eval,
Role: m.Role, Role: m.Role,
} }
if m.Split() {
v.ProfileReplicas = map[string]int{}
for _, p := range m.Profiles {
v.ProfileReplicas[p.Name] = p.Replicas
}
}
for _, p := range m.Profiles { for _, p := range m.Profiles {
v.Profiles = append(v.Profiles, p.Name) v.Profiles = append(v.Profiles, p.Name)
} }
@@ -962,6 +992,15 @@ type Models interface {
SetClass(ctx context.Context, name, class, by string) (catalog.Model, error) SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error) SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
SetAdapters(ctx context.Context, name string, adapters []catalog.Adapter, by string) (catalog.Model, error) SetAdapters(ctx context.Context, name string, adapters []catalog.Adapter, by string) (catalog.Model, error)
SetProfileReplicas(ctx context.Context, name string, counts map[string]int, cpu *modelsvc.CPUProfile, by string) (catalog.Model, error)
}
// ProfileReplicasRequest counts a model's replicas per profile: profile
// name to replicas, and CPU, for a llama.cpp model with no profile on CPUs,
// the one to add.
type ProfileReplicasRequest struct {
Profiles map[string]int `json:"profiles"`
CPU *modelsvc.CPUProfile `json:"cpu,omitempty"`
} }
// ResourcesRequest sets what a model's profiles reserve of a node. // ResourcesRequest sets what a model's profiles reserve of a node.
@@ -1288,6 +1327,25 @@ func (a *API) setResources(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, m) writeJSON(w, http.StatusOK, m)
} }
// setProfileReplicas counts a model's replicas per profile.
func (a *API) setProfileReplicas(w http.ResponseWriter, r *http.Request) {
if a.models == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store")
return
}
var req ProfileReplicasRequest
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
return
}
m, err := a.models.SetProfileReplicas(r.Context(), r.PathValue("name"), req.Profiles, req.CPU, actorOf(r))
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
writeJSON(w, http.StatusOK, m)
}
func (a *API) removeModel(w http.ResponseWriter, r *http.Request) { func (a *API) removeModel(w http.ResponseWriter, r *http.Request) {
if a.models == nil { if a.models == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store") writeError(w, http.StatusNotImplemented, "this control plane has no store")
@@ -1966,6 +2024,15 @@ func PoolNodesOf(nodes []NodeView) []modelplan.PoolNode {
if l := n.Load; l != nil && l.MemTotal > 0 { if l := n.Load; l != nil && l.MemTotal > 0 {
p.HostFreeGB = int((l.MemTotal - l.MemUsed) >> 30) p.HostFreeGB = int((l.MemTotal - l.MemUsed) >> 30)
} }
p.CPUs, p.MemBWGBs, p.MemBWCoreGBs = n.CPUCount, n.MemBandwidthGBs, n.MemBandwidthCoreGBs
for _, g := range n.GPUs {
if bw := modelplan.GPUBandwidthGBs(g.Name); bw > 0 {
if p.CardBWGBs == nil {
p.CardBWGBs = map[string]float64{}
}
p.CardBWGBs[g.Class] = bw
}
}
out = append(out, p) out = append(out, p)
} }
return out return out
+11
View File
@@ -16,6 +16,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control" "gitea.mynodes.xyz/ilker/llmgw/internal/control"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory" "gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler" "gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint" "gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
"gitea.mynodes.xyz/ilker/llmgw/internal/store" "gitea.mynodes.xyz/ilker/llmgw/internal/store"
@@ -263,6 +264,16 @@ func (f *fakeModels) SetAdapters(_ context.Context, name string, as []catalog.Ad
return catalog.Model{Name: name, Adapters: as}, f.err return catalog.Model{Name: name, Adapters: as}, f.err
} }
func (f *fakeModels) SetProfileReplicas(_ context.Context, name string, counts map[string]int, _ *modelsvc.CPUProfile,
_ string,
) (catalog.Model, error) {
m := catalog.Model{Name: name}
for p, n := range counts {
m.Profiles = append(m.Profiles, catalog.Profile{Name: p, Replicas: n})
}
return m, f.err
}
func (f *fakeModels) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) { func (f *fakeModels) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
return catalog.Model{Name: name, Owner: owner}, f.err return catalog.Model{Name: name, Owner: owner}, f.err
} }
+4
View File
@@ -6,6 +6,10 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/notify" "gitea.mynodes.xyz/ilker/llmgw/internal/notify"
) )
// SetWebhooks has the summary say when the settings name no webhook:
// count is how many they name now.
func (a *API) SetWebhooks(count func() int) { a.webhooks = count }
// SetNotifier gives the API the webhooks `ctl notify test` reaches. // SetNotifier gives the API the webhooks `ctl notify test` reaches.
func (a *API) SetNotifier(n *notify.Notifier) { a.notifier = n } func (a *API) SetNotifier(n *notify.Notifier) { a.notifier = n }
+108
View File
@@ -0,0 +1,108 @@
package adminapi
import (
"context"
"net/http"
"strconv"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// PriceStore is what a price is suggested from: what each model held of
// the cluster, and the tokens it served.
type PriceStore interface {
HeldSince(ctx context.Context, since time.Time) ([]store.Held, time.Time, error)
UsageByModel(ctx context.Context, since time.Time) ([]store.ModelUsage, error)
}
// PriceConfig is the cost basis and the prices in force.
type PriceConfig struct {
Basis accounting.Basis
Current map[string]accounting.Price
Default accounting.Price
}
// PriceSuggestions is a model's price from its cost, for each model.
type PriceSuggestions struct {
// Days is the window; RecordedFrom the first hour held is known for,
// later than the window's start on a cluster that began keeping it
// since, which makes the window shorter than asked.
Days int `json:"days"`
Since time.Time `json:"since"`
RecordedFrom time.Time `json:"recorded_from,omitempty"`
GPUHourCost float64 `json:"gpu_hour_cost"`
CoreHourCost float64 `json:"core_hour_cost"`
Models []PriceSuggestion `json:"models"`
}
// PriceSuggestion is one model's.
type PriceSuggestion struct {
accounting.Suggestion
Current accounting.Price `json:"current"`
}
// SetPrices gives the API what prices are suggested from.
func (a *API) SetPrices(st PriceStore, cfg func() PriceConfig) { a.prices, a.priceCfg = st, cfg }
func (a *API) registerPrices(mux Mux) {
mux.HandleFunc("GET /api/v1/prices/suggest", a.suggestPrices)
}
func (a *API) suggestPrices(w http.ResponseWriter, r *http.Request) {
if a.prices == nil || a.priceCfg == nil {
writeError(w, http.StatusNotImplemented, "prices are suggested from the store, which this control plane runs without")
return
}
days := 30
if v := r.URL.Query().Get("days"); v != "" {
n, err := strconv.Atoi(v)
if err != nil || n < 1 || n > 366 {
writeError(w, http.StatusBadRequest, "days is a whole number of days, 1 to 366")
return
}
days = n
}
cfg := a.priceCfg()
if cfg.Basis.GPUHour <= 0 && cfg.Basis.CoreHour <= 0 {
writeError(w, http.StatusConflict, "no cost to price from: set auth.gpu_hour_cost (and auth.core_hour_cost for "+
"models on CPUs), what an hour of one costs, in the units budgets are in")
return
}
since := time.Now().Add(-time.Duration(days) * 24 * time.Hour)
rows, first, err := a.prices.HeldSince(r.Context(), since)
if err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
// Tokens from when holding was first recorded, so both sides cover the
// same hours.
from := since
if first.After(from) {
from = first
}
usage, err := a.prices.UsageByModel(r.Context(), from)
if err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
held := map[string]accounting.Held{}
for _, h := range rows {
held[h.Model] = accounting.Held{GPUSeconds: h.GPUSeconds, CoreSeconds: h.CoreSeconds}
}
served := map[string]accounting.Served{}
for _, u := range usage {
served[u.Model] = accounting.Served{Prompt: u.Prompt, Completion: u.Completion}
}
out := PriceSuggestions{Days: days, Since: since.UTC(), RecordedFrom: first,
GPUHourCost: cfg.Basis.GPUHour, CoreHourCost: cfg.Basis.CoreHour, Models: []PriceSuggestion{}}
for _, s := range accounting.Suggest(held, served, cfg.Basis) {
cur, ok := cfg.Current[s.Model]
if !ok {
cur = cfg.Default
}
out.Models = append(out.Models, PriceSuggestion{Suggestion: s, Current: cur})
}
writeJSON(w, http.StatusOK, out)
}
+40 -22
View File
@@ -2,6 +2,8 @@ package adminapi
import ( import (
"context" "context"
"errors"
"fmt"
"log/slog" "log/slog"
"net/http" "net/http"
"net/url" "net/url"
@@ -48,10 +50,29 @@ type PurgeView struct {
// purgeModel removes a model from the catalog and its files from the mirror // purgeModel removes a model from the catalog and its files from the mirror
// and the nodes. // and the nodes.
func (a *API) purgeModel(w http.ResponseWriter, r *http.Request, name string) { func (a *API) purgeModel(w http.ResponseWriter, r *http.Request, name string) {
out, err := a.Purge(r.Context(), name, actorOf(r))
switch {
case errors.Is(err, errNoSuchModel):
writeError(w, http.StatusNotFound, err.Error())
case err != nil:
writeError(w, http.StatusBadRequest, err.Error())
default:
writeJSON(w, http.StatusOK, out)
}
}
// errNoSuchModel is a model the catalog does not have.
var errNoSuchModel = errors.New("no such model in the catalog")
// Purge removes a model from the catalog and its files from the mirror and
// the nodes, waiting a while for its replicas to stop before the nodes'.
func (a *API) Purge(ctx context.Context, name, by string) (PurgeView, error) {
if a.models == nil {
return PurgeView{}, errors.New("this control plane has no store")
}
m, ok := a.ctl.Catalog().Resolve(name) m, ok := a.ctl.Catalog().Resolve(name)
if !ok { if !ok {
writeError(w, http.StatusNotFound, "no model "+name+" in the catalog") return PurgeView{}, fmt.Errorf("%w: %s", errNoSuchModel, name)
return
} }
name = m.Name name = m.Name
others := []catalog.Model{} others := []catalog.Model{}
@@ -60,37 +81,35 @@ func (a *API) purgeModel(w http.ResponseWriter, r *http.Request, name string) {
others = append(others, o) others = append(others, o)
} }
} }
if err := a.models.Remove(r.Context(), name); err != nil { if err := a.models.Remove(ctx, name); err != nil {
writeError(w, http.StatusBadRequest, err.Error()) return PurgeView{}, err
return
} }
a.presetDeclined(r.Context(), name, actorOf(r)) a.presetDeclined(ctx, name, by)
out := PurgeView{Removed: name} out := PurgeView{Removed: name}
sources := purgeSources(m) sources := purgeSources(m)
if a.purgeMirror == nil { if a.purgeMirror == nil {
out.Note = "this control plane has no mirror; its files there, if any, are left" out.Note = "this control plane has no mirror; its files there, if any, are left"
} else { } else {
out.Mirror, out.Kept, out.Note = purgeFromMirror(r.Context(), a.purgeMirror, sources, others) out.Mirror, out.Kept, out.Note = purgeFromMirror(ctx, a.purgeMirror, sources, others)
} }
// The nodes' copies, once nothing runs on them. // The nodes' copies, once nothing runs on them.
deadline := time.Now().Add(purgeWait) deadline := time.Now().Add(purgeWait)
for a.runs(name) && time.Now().Before(deadline) { for a.runs(name) && time.Now().Before(deadline) {
select { select {
case <-r.Context().Done(): case <-ctx.Done():
return return out, ctx.Err()
case <-time.After(time.Second): case <-time.After(time.Second):
} }
} }
if a.runs(name) { if a.runs(name) {
out.Note = strings.TrimSpace(out.Note + " Its replicas had not stopped yet: its copies on the nodes are " + out.Note = strings.TrimSpace(out.Note + " Its replicas had not stopped yet: its copies on the nodes are " +
"left, and llmgw ctl weights --collect deletes them.") "left, and llmgw ctl weights --collect deletes them.")
writeJSON(w, http.StatusOK, out) return out, nil
return
} }
out.Nodes = a.purgeFromNodes(r.Context(), sources, others) out.Nodes = a.purgeFromNodes(ctx, sources, others)
writeJSON(w, http.StatusOK, out) return out, nil
} }
// runs says whether any replica of a model is still on a node. // runs says whether any replica of a model is still on a node.
@@ -197,7 +216,7 @@ func (a *API) purgeFromNodes(ctx context.Context, sources []string, others []cat
names[filepath.Base(s)] = true names[filepath.Base(s)] = true
} }
wanted := inUse(others) wanted := inUse(others)
var out []CollectResult var rs []removal
for _, n := range a.ctl.Inventory().Nodes() { for _, n := range a.ctl.Inventory().Nodes() {
size := map[string]int64{} size := map[string]int64{}
have := make([]string, 0, len(n.Weights)) have := make([]string, 0, len(n.Weights))
@@ -216,15 +235,14 @@ func (a *API) purgeFromNodes(ctx context.Context, sources []string, others []cat
if len(paths) == 0 { if len(paths) == 0 {
continue continue
} }
res := CollectResult{Node: n.ID, Files: len(paths)} rs = append(rs, removal{node: n.ID, paths: paths, size: size})
for _, p := range paths { }
res.Bytes += size[p] out := a.removeConfirmed(ctx, rm, rs)
for _, r := range out {
if r.Error != "" || r.Left > 0 {
slog.Warn("a purged model's copies were not all deleted on a node", "node", r.Node,
"left", r.Left, "err", r.Error)
} }
if err := rm.RemoveWeights(ctx, n.ID, paths); err != nil {
res.Error, res.Files, res.Bytes = err.Error(), 0, 0
slog.Warn("a purged model's copies could not be deleted on a node", "node", n.ID, "err", err)
}
out = append(out, res)
} }
return out return out
} }
+27
View File
@@ -12,6 +12,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/federation" "gitea.mynodes.xyz/ilker/llmgw/internal/control/federation"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve" "gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink" "gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard" "gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify" "gitea.mynodes.xyz/ilker/llmgw/internal/notify"
"gitea.mynodes.xyz/ilker/llmgw/internal/store" "gitea.mynodes.xyz/ilker/llmgw/internal/store"
@@ -84,6 +85,7 @@ var PathParams = map[string]string{
"projects/{id}": "The project's ID.", "projects/{id}": "The project's ID.",
"keys/{id}": "The key's ID, as listed (not the key itself).", "keys/{id}": "The key's ID, as listed (not the key itself).",
"users/{id}": "The person's ID or email.", "users/{id}": "The person's ID or email.",
"users/{user}": "The person's ID or email.",
"artifacts/{kind}": "What the job made: peft, a gzipped tar of the PEFT adapter, or gguf, the adapter for llama.cpp.", "artifacts/{kind}": "What the job made: peft, a gzipped tar of the PEFT adapter, or gguf, the adapter for llama.cpp.",
} }
@@ -168,6 +170,10 @@ var Routes = []Route{
Response: ScaleResponse{}}, Response: ScaleResponse{}},
{Method: "POST", Path: "/api/v1/models/{name}/scale", Summary: "Set how many replicas a model runs", {Method: "POST", Path: "/api/v1/models/{name}/scale", Summary: "Set how many replicas a model runs",
Request: ScaleRequest{}, Response: ScaleResponse{}}, Request: ScaleRequest{}, Response: ScaleResponse{}},
{Method: "POST", Path: "/api/v1/models/{name}/replicas", Summary: "How many replicas run on each profile",
Description: "Seven on cards and four on CPUs, under the model's name. A profile not named keeps its count; " +
"cpu adds a profile on CPUs to a llama.cpp model that has none. Its scaling becomes the sum.",
Request: ProfileReplicasRequest{}, Response: catalog.Model{}},
{Method: "POST", Path: "/api/v1/models/{name}/resources", Summary: "Set what a model's profiles reserve of a node", {Method: "POST", Path: "/api/v1/models/{name}/resources", Summary: "Set what a model's profiles reserve of a node",
Request: ResourcesRequest{}, Response: catalog.Model{}}, Request: ResourcesRequest{}, Response: catalog.Model{}},
{Method: "POST", Path: "/api/v1/models/{name}/class", Summary: "Set a model's capacity class", {Method: "POST", Path: "/api/v1/models/{name}/class", Summary: "Set a model's capacity class",
@@ -388,6 +394,27 @@ var Routes = []Route{
Description: "Their keys in projects they no longer reach are revoked.", Description: "Their keys in projects they no longer reach are revoked.",
Request: MembershipRequest{}, Response: UserView{}}, Request: MembershipRequest{}, Response: UserView{}},
{Method: "GET", Path: "/api/v1/slurm", Summary: "The Slurm cluster beside llmgw, and where each of its nodes is",
Description: "From the cluster's settings (slurm.*) and what each node's agent last said. A node's phase is llmgw " +
"(drained in Slurm, no job left), lending (its replicas moving off), lent, reclaiming (drained in Slurm, jobs " +
"still running) or unknown (its agent has not said).",
Response: slurmsvc.Status{}},
{Method: "GET", Path: "/api/v1/slurm/users", Summary: "The Linux users made from llmgw's accounts for the Slurm cluster",
Description: "Every active account, with its login, UID and SSH keys; an account is given its login and UID the first " +
"time it is listed. The machines have them when slurm.users is \"llmgw\".",
Response: []slurmsvc.User{}},
{Method: "PUT", Path: "/api/v1/slurm/users/{user}/ssh-keys", Summary: "Replace an account's SSH public keys",
Description: "{user} is an ID or e-mail. keys is OpenSSH public keys, one a line, without options; empty takes them " +
"all away. The machines have them within a minute.",
Request: SlurmKeysRequest{}, Response: slurmsvc.User{}},
{Method: "GET", Path: "/api/v1/prices/suggest", Summary: "Each model's price, from what it held of the cluster",
Description: "days (30 if not given) is the window. A model's cost is the GPU- and core-hours its replicas held, idle " +
"ones too, at auth.gpu_hour_cost and auth.core_hour_cost; spread over its tokens, an output token four input " +
"ones and a cached prompt token a tenth of one, it is price, in micro-units a thousand tokens as " +
"auth.pricing takes them. current is the price in force. 409 with no cost set.",
Response: PriceSuggestions{}},
{Method: "POST", Path: "/api/v1/train/jobs", Summary: "Queue a LoRA adapter's training on a node", {Method: "POST", Path: "/api/v1/train/jobs", Summary: "Queue a LoRA adapter's training on a node",
Description: "The node trains when its agent is there, nothing else trains on it, and a card has gpu_memory_mib free; " + Description: "The node trains when its agent is there, nothing else trains on it, and a card has gpu_memory_mib free; " +
"the pairs are NDJSON of {\"q\", \"a\"}. An adapter learns how answers sound far better than what they say.", "the pairs are NDJSON of {\"q\", \"a\"}. An adapter learns how answers sound far better than what they say.",
+119
View File
@@ -0,0 +1,119 @@
package adminapi
import (
"context"
"encoding/json"
"errors"
"net/http"
"strings"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// SlurmService is the Slurm cluster beside llmgw (BACKLOG P), as the leader
// keeps it.
type SlurmService interface {
Status() slurmsvc.Status
UsersChanged()
}
// SlurmUsers are the Linux users llmgw makes from its accounts.
type SlurmUsers interface {
SlurmUsers(ctx context.Context) ([]store.SlurmUser, error)
SlurmUserByID(ctx context.Context, userID string) (store.SlurmUser, error)
SetSlurmSSHKeys(ctx context.Context, userID string, keys []string) error
}
// SetSlurmUsers enables the Slurm users' endpoints.
func (a *API) SetSlurmUsers(u SlurmUsers) { a.slurmUsers = u }
// SlurmKeysRequest is a person's SSH public keys, one a line, as
// authorized_keys has them; empty takes them all away.
type SlurmKeysRequest struct {
Keys string `json:"keys"`
}
// SlurmUserView is the user from a store row.
func SlurmUserView(u store.SlurmUser) slurmsvc.User {
return slurmsvc.User{Login: u.Login, UID: u.UID, Name: u.Name, Email: u.Email, SSHKeys: u.SSHKeys}
}
// SetSlurm gives the API the Slurm service.
func (a *API) SetSlurm(s SlurmService) { a.slurm = s }
func (a *API) registerSlurm(mux Mux) {
mux.HandleFunc("GET /api/v1/slurm", a.slurmStatus)
mux.HandleFunc("GET /api/v1/slurm/users", a.slurmUserList)
mux.HandleFunc("PUT /api/v1/slurm/users/{user}/ssh-keys", a.slurmUserKeys)
}
func (a *API) slurmUserList(w http.ResponseWriter, r *http.Request) {
if a.slurmUsers == nil {
writeError(w, http.StatusNotImplemented, "the Slurm users need the store, which this control plane runs without")
return
}
us, err := a.slurmUsers.SlurmUsers(r.Context())
if err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
out := make([]slurmsvc.User, 0, len(us))
for _, u := range us {
out = append(out, SlurmUserView(u))
}
writeJSON(w, http.StatusOK, out)
}
// slurmUserKeys replaces a person's SSH keys; {user} is an ID or e-mail.
func (a *API) slurmUserKeys(w http.ResponseWriter, r *http.Request) {
if a.slurmUsers == nil {
writeError(w, http.StatusNotImplemented, "the Slurm users need the store, which this control plane runs without")
return
}
if !a.tenancyReady(w) {
return
}
var in SlurmKeysRequest
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 64<<10)).Decode(&in); err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
keys, err := slurm.ParseSSHKeys(in.Keys)
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
ctx := r.Context()
u, err := a.findUser(ctx, r.PathValue("user"))
if err != nil {
writeError(w, http.StatusNotFound, strings.TrimPrefix(err.Error(), store.ErrNotFound.Error()+": "))
return
}
if err := a.slurmUsers.SetSlurmSSHKeys(ctx, u.ID, keys); err != nil {
code := http.StatusInternalServerError
if errors.Is(err, store.ErrNotFound) {
code = http.StatusNotFound
}
writeError(w, code, err.Error())
return
}
if a.slurm != nil {
a.slurm.UsersChanged()
}
su, err := a.slurmUsers.SlurmUserByID(ctx, u.ID)
if err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
writeJSON(w, http.StatusOK, SlurmUserView(su))
}
func (a *API) slurmStatus(w http.ResponseWriter, _ *http.Request) {
if a.slurm == nil {
writeError(w, http.StatusNotImplemented, "Slurm needs the store and the nodes' agents, which this control plane runs without")
return
}
writeJSON(w, http.StatusOK, a.slurm.Status())
}
+6
View File
@@ -50,6 +50,9 @@ type Summary struct {
// GuardOff says the guardrails are turned off (guard.enabled: false): // GuardOff says the guardrails are turned off (guard.enabled: false):
// no request is checked, whatever the policies say. // no request is checked, whatever the policies say.
GuardOff bool `json:"guard_off,omitempty"` GuardOff bool `json:"guard_off,omitempty"`
// NoAlerts says the cluster's settings name no webhook: a model parked,
// a node lost or a judge down is told nobody.
NoAlerts bool `json:"no_alerts,omitempty"`
} }
// StoreMember is one member of a replicated store, as its Patroni reports it. // StoreMember is one member of a replicated store, as its Patroni reports it.
@@ -160,6 +163,9 @@ func (a *API) Summary() Summary {
if a.guard != nil && a.guard.Library != nil && !a.guard.Library().On() { if a.guard != nil && a.guard.Library != nil && !a.guard.Library().On() {
s.GuardOff = true s.GuardOff = true
} }
if a.webhooks != nil && a.webhooks() == 0 {
s.NoAlerts = true
}
for _, j := range a.Judges() { for _, j := range a.Judges() {
if j.Problem != "" { if j.Problem != "" {
s.Judges = append(s.Judges, j) s.Judges = append(s.Judges, j)
+13
View File
@@ -153,3 +153,16 @@ func TestOneSharedCopyCountsAsMuchAsTwenty(t *testing.T) {
t.Errorf("counted %d models; three nodes seeing one file is one model", f.Models) t.Errorf("counted %d models; three nodes seeing one file is one model", f.Models)
} }
} }
func TestTheSummarySaysWhenNoWebhookIsSet(t *testing.T) {
a, _ := setup(t, testModels(), gpuNode("n1", 8))
n := 0
a.SetWebhooks(func() int { return n })
if !a.Summary().NoAlerts {
t.Error("no webhook, not said")
}
n = 1
if a.Summary().NoAlerts {
t.Error("a webhook set, yet said there is none")
}
}
+6 -2
View File
@@ -634,8 +634,12 @@ func (a *API) membership(add bool) http.HandlerFunc {
if role == "" { if role == "" {
role = store.RoleMember role = store.RoleMember
} }
if role != store.RoleMember && role != store.RoleManager { if role != store.RoleMember && role != store.RoleManager && role != store.RoleOrgAdmin {
writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member or manager", role)) writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member, manager or admin", role))
return
}
if role != store.RoleMember && scope != store.ScopeOrg {
writeError(w, http.StatusBadRequest, fmt.Sprintf("%s is a role on an organization", role))
return return
} }
if err := a.tenancy.AddMembership(ctx, u.ID, scope, id, role, actorOf(r), time.Now()); err != nil { if err := a.tenancy.AddMembership(ctx, u.ID, scope, id, role, actorOf(r), time.Now()); err != nil {
+347 -72
View File
@@ -10,15 +10,20 @@ import (
"errors" "errors"
"fmt" "fmt"
"io" "io"
"log/slog"
"net/http" "net/http"
"regexp" "regexp"
"slices" "slices"
"strconv" "strconv"
"strings" "strings"
"sync"
"time" "time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train" "gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror" "gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
"gitea.mynodes.xyz/ilker/llmgw/internal/store" "gitea.mynodes.xyz/ilker/llmgw/internal/store"
) )
@@ -36,6 +41,11 @@ type TrainStore interface {
TrainArtifact(ctx context.Context, job, kind string) ([]byte, error) TrainArtifact(ctx context.Context, job, kind string) ([]byte, error)
TrainArtifactKinds(ctx context.Context, job string) ([]string, error) TrainArtifactKinds(ctx context.Context, job string) ([]string, error)
DeleteTrainJob(ctx context.Context, id string) error DeleteTrainJob(ctx context.Context, id string) error
QueuedTrainJobs(ctx context.Context) ([]store.TrainJob, error)
ActiveTrainJobs(ctx context.Context) ([]store.TrainJob, error)
TrainJobsFor(ctx context.Context, projects []string, createdBy string, limit int) ([]store.TrainJob, error)
SetTrainOwner(ctx context.Context, id, owner string) error
SetTrainNote(ctx context.Context, id, message string) error
} }
// TrainAdmin is what the training endpoints need. // TrainAdmin is what the training endpoints need.
@@ -45,6 +55,27 @@ type TrainAdmin struct {
// Mirror keeps what a job made where the nodes fetch weights; nil // Mirror keeps what a job made where the nodes fetch weights; nil
// leaves publishing off. // leaves publishing off.
Mirror TrainMirror Mirror TrainMirror
// Registry finds the weights to train a catalog model on, from where
// its own came; nil has every job name them.
Registry TrainRegistry
// Log says what publishing by itself could not do.
Log *slog.Logger
mu sync.Mutex
bases map[string]trainBaseInfo // by the model's origin repository
}
// TrainRegistry reads a repository of the model registry.
type TrainRegistry interface {
RepoAt(ctx context.Context, id, rev string) (hf.Repo, error)
}
// trainBaseInfo is the safetensors weights a model is trained on.
type trainBaseInfo struct {
repo, revision string
bytes int64
err error
at time.Time
} }
// TrainMirror is the model mirror, as publishing needs it. // TrainMirror is the model mirror, as publishing needs it.
@@ -66,11 +97,11 @@ type TrainPublishView struct {
// SetTrain enables the training endpoints. // SetTrain enables the training endpoints.
func (a *API) SetTrain(t *TrainAdmin) { a.train = t } func (a *API) SetTrain(t *TrainAdmin) { a.train = t }
// SetTrainMirror gives publishing the mirror, which is set up after the // SetTrainMirror gives publishing the mirror, and finding a model's
// training endpoints are. // weights the registry, which are set up after the training endpoints are.
func (a *API) SetTrainMirror(m TrainMirror) { func (a *API) SetTrainMirror(m TrainMirror, reg TrainRegistry) {
if a.train != nil { if a.train != nil {
a.train.Mirror = m a.train.Mirror, a.train.Registry = m, reg
} }
} }
@@ -81,13 +112,23 @@ type TrainRequest struct {
// Model is the catalog model the adapter is for. // Model is the catalog model the adapter is for.
Model string `json:"model"` Model string `json:"model"`
// BaseRepo and BaseRevision are the weights trained on: the model's own // BaseRepo and BaseRevision are the weights trained on: the model's own
// on its registry, in safetensors (Hugging Face), not a GGUF. // on its registry, in safetensors (Hugging Face), not a GGUF. Empty
BaseRepo string `json:"base_repo"` // finds them from where the model's came: the repository itself, or the
// one its card says it was made from.
BaseRepo string `json:"base_repo,omitempty"`
BaseRevision string `json:"base_revision,omitempty"` BaseRevision string `json:"base_revision,omitempty"`
// Node trains it; GPUMemoryMiB is what it needs of a card, which can be // Node trains it; empty is the first with a card that has enough free.
// what an engine there leaves free. // GPUMemoryMiB is what it needs of a card, which can be what an engine
Node string `json:"node"` // there leaves free; 0 is worked out from the weights' size.
GPUMemoryMiB int `json:"gpu_memory_mib"` Node string `json:"node,omitempty"`
GPUMemoryMiB int `json:"gpu_memory_mib,omitempty"`
// Project is the project the job is started in: its members see and
// manage it, and only one of its jobs trains at a time.
Project string `json:"project,omitempty"`
// Publish puts what the job makes on the mirror and makes it the
// model's adapter by itself when the job is done, restarting a llama.cpp
// model's replicas to load it.
Publish bool `json:"publish,omitempty"`
// Owner is who the adapter will belong to: org:, team:, project: (by // Owner is who the adapter will belong to: org:, team:, project: (by
// path or ID), user: or key:. Only callers there see it. When the // path or ID), user: or key:. Only callers there see it. When the
// pairs come from one knowledge base, it defaults to that collection's // pairs come from one knowledge base, it defaults to that collection's
@@ -129,6 +170,8 @@ type TrainJobView struct {
StartedAt time.Time `json:"started_at,omitzero"` StartedAt time.Time `json:"started_at,omitzero"`
FinishedAt time.Time `json:"finished_at,omitzero"` FinishedAt time.Time `json:"finished_at,omitzero"`
CreatedBy string `json:"created_by,omitempty"` CreatedBy string `json:"created_by,omitempty"`
Project string `json:"project,omitempty"`
Publish bool `json:"publish,omitempty"`
// Artifacts are what it made: peft (a gzipped tar), gguf. // Artifacts are what it made: peft (a gzipped tar), gguf.
Artifacts []string `json:"artifacts,omitempty"` Artifacts []string `json:"artifacts,omitempty"`
} }
@@ -137,7 +180,8 @@ func trainView(j store.TrainJob) TrainJobView {
return TrainJobView{ID: j.ID, Name: j.Name, Model: j.Model, BaseRepo: j.BaseRepo, BaseRevision: j.BaseRevision, return TrainJobView{ID: j.ID, Name: j.Name, Model: j.Model, BaseRepo: j.BaseRepo, BaseRevision: j.BaseRevision,
Node: j.Node, Owner: j.Owner, GPUMemoryMiB: j.GPUMemoryMiB, Params: j.Params, GGUF: j.GGUF, Pairs: j.Pairs, State: j.State, Node: j.Node, Owner: j.Owner, GPUMemoryMiB: j.GPUMemoryMiB, Params: j.Params, GGUF: j.GGUF, Pairs: j.Pairs, State: j.State,
Step: j.Step, Steps: j.Steps, Loss: j.Loss, Message: j.Message, PeakMiB: j.PeakMiB, CreatedAt: j.CreatedAt, Step: j.Step, Steps: j.Steps, Loss: j.Loss, Message: j.Message, PeakMiB: j.PeakMiB, CreatedAt: j.CreatedAt,
StartedAt: j.StartedAt, FinishedAt: j.FinishedAt, CreatedBy: j.CreatedBy} StartedAt: j.StartedAt, FinishedAt: j.FinishedAt, CreatedBy: j.CreatedBy,
Project: j.Project, Publish: j.Publish}
} }
func (a *API) registerTrain(mux Mux) { func (a *API) registerTrain(mux Mux) {
@@ -178,50 +222,122 @@ func (a *API) trainCreate(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, err.Error()) writeError(w, http.StatusBadRequest, err.Error())
return return
} }
bad := func(format string, args ...any) { j, err := a.StartTraining(r.Context(), in, actorOf(r))
writeError(w, http.StatusBadRequest, fmt.Sprintf(format, args...)) if err != nil {
writeTrainErr(w, err)
return
}
writeJSON(w, http.StatusCreated, trainView(j))
}
// trainErr is a refusal, with the status it is answered with.
type trainErr struct {
code int
msg string
}
func (e *trainErr) Error() string { return e.msg }
func refused(code int, format string, args ...any) error {
return &trainErr{code: code, msg: fmt.Sprintf(format, args...)}
}
func writeTrainErr(w http.ResponseWriter, err error) {
var te *trainErr
if errors.As(err, &te) {
writeError(w, te.code, te.msg)
return
}
writeError(w, http.StatusInternalServerError, err.Error())
}
// TrainRefusal is the reason a training request was refused, for someone
// to read, and whether it was the request's fault rather than the
// cluster's.
func TrainRefusal(err error) (string, bool) {
if err == nil {
return "", false
}
var te *trainErr
if errors.As(err, &te) {
return te.msg, te.code < 500
}
return err.Error(), false
}
// StartTraining checks a job and queues it, as actor.
func (a *API) StartTraining(ctx context.Context, in TrainRequest, actor string) (store.TrainJob, error) {
if a.train == nil {
return store.TrainJob{}, refused(http.StatusNotImplemented,
"training needs the store and the nodes' agents, which this control plane runs without")
}
bad := func(format string, args ...any) (store.TrainJob, error) {
return store.TrainJob{}, refused(http.StatusBadRequest, format, args...)
} }
if !adapterName.MatchString(in.Name) { if !adapterName.MatchString(in.Name) {
bad("an adapter's name is letters, digits, . _ and -, up to 100: %q is not", in.Name) return bad("an adapter's name is letters, digits, . _ and -, up to 100: %q is not", in.Name)
return
} }
m, ok := a.ctl.Catalog().Resolve(in.Model) m, ok := a.ctl.Catalog().Resolve(in.Model)
if !ok { if !ok {
bad("no model %s in the catalog", in.Model) return bad("no model %s in the catalog", in.Model)
return
} }
if _, _, _, taken := a.ctl.Catalog().ResolveAdapter(in.Name); taken || in.Name == m.Name { if _, _, _, taken := a.ctl.Catalog().ResolveAdapter(in.Name); taken || in.Name == m.Name {
bad("%s is already a model's or an adapter's name", in.Name) return bad("%s is already a model's or an adapter's name", in.Name)
return
} }
if strings.Count(in.BaseRepo, "/") != 1 || strings.ContainsAny(in.BaseRepo, " :") { if _, taken := a.ctl.Catalog().Resolve(in.Name); taken {
bad("base_repo is the weights' registry repository, owner/name: %q is not", in.BaseRepo) return bad("%s is already a model's or an adapter's name", in.Name)
return
} }
if _, ok := a.ctl.Inventory().Get(in.Node); !ok { queued, err := a.train.Store.QueuedTrainJobs(ctx)
bad("no node %s", in.Node) if err != nil {
return return store.TrainJob{}, err
}
active, err := a.train.Store.ActiveTrainJobs(ctx)
if err != nil {
return store.TrainJob{}, err
}
for _, j := range append(queued, active...) {
if j.Name == in.Name {
return bad("%s is already being trained (job %s)", in.Name, j.ID)
}
if in.Project != "" && j.Project == in.Project {
return store.TrainJob{}, refused(http.StatusConflict,
"this project already has a model training (%s): one at a time, so that one project does not hold every card", j.Name)
}
}
var weights int64
if in.BaseRepo == "" {
b := a.trainBase(ctx, m)
if b.err != nil {
return store.TrainJob{}, b.err
}
in.BaseRepo, in.BaseRevision, weights = b.repo, b.revision, b.bytes
} else if strings.Count(in.BaseRepo, "/") != 1 || strings.ContainsAny(in.BaseRepo, " :") {
return bad("base_repo is the weights' registry repository, owner/name: %q is not", in.BaseRepo)
}
if in.Node != "" {
if _, ok := a.ctl.Inventory().Get(in.Node); !ok {
return bad("no node %s", in.Node)
}
} }
if in.GPUMemoryMiB <= 0 { if in.GPUMemoryMiB <= 0 {
bad("gpu_memory_mib is what the job needs of a card, in MiB") if weights <= 0 {
return return bad("gpu_memory_mib is what the job needs of a card, in MiB")
}
in.GPUMemoryMiB = TrainMemoryMiB(weights)
} }
pairs, colls, err := countPairs(in.Data) pairs, colls, err := countPairs(in.Data)
if err != nil { if err != nil {
bad("%v", err) return bad("%v", err)
return
} }
owner, err := a.trainOwner(r.Context(), in, colls) owner, err := a.trainOwner(ctx, in, colls)
if err != nil { if err != nil {
bad("%v", err) return bad("%v", err)
return
} }
for k, v := range in.Params { for k, v := range in.Params {
lim, known := trainParams[k] lim, known := trainParams[k]
f, isNum := v.(float64) f, isNum := v.(float64)
if !known || !isNum || f < lim[0] || f > lim[1] { if !known || !isNum || f < lim[0] || f > lim[1] {
bad("params.%s: the trainer takes rank, alpha, dropout, epochs, lr, batch, max_len and seed, each a number in its range", k) return bad("params.%s: the trainer takes rank, alpha, dropout, epochs, lr, batch, max_len and seed, each a number in its range", k)
return
} }
} }
params := "" params := ""
@@ -240,15 +356,132 @@ func (a *API) trainCreate(w http.ResponseWriter, r *http.Request) {
} }
j := store.TrainJob{ID: store.NewID("train-"), Name: in.Name, Model: m.Name, BaseRepo: in.BaseRepo, j := store.TrainJob{ID: store.NewID("train-"), Name: in.Name, Model: m.Name, BaseRepo: in.BaseRepo,
BaseRevision: in.BaseRevision, Node: in.Node, Owner: owner, GPUMemoryMiB: in.GPUMemoryMiB, Params: params, GGUF: gguf, BaseRevision: in.BaseRevision, Node: in.Node, Owner: owner, GPUMemoryMiB: in.GPUMemoryMiB, Params: params, GGUF: gguf,
Data: []byte(in.Data), Pairs: pairs, CreatedBy: actorOf(r)} Data: []byte(in.Data), Pairs: pairs, CreatedBy: actor, Project: in.Project, Publish: in.Publish}
if err := a.train.Store.CreateTrainJob(r.Context(), j); err != nil { if err := a.train.Store.CreateTrainJob(ctx, j); err != nil {
writeError(w, http.StatusInternalServerError, err.Error()) return store.TrainJob{}, err
return
} }
// At once rather than at the next look, when the node is here. // At once rather than at the next look, when the node is here.
go a.train.Service.Dispatch(context.WithoutCancel(r.Context())) go a.train.Service.Dispatch(context.WithoutCancel(ctx))
j.State = store.TrainQueued j.State = store.TrainQueued
writeJSON(w, http.StatusCreated, trainView(j)) return j, nil
}
// TrainMemoryMiB is what training a LoRA adapter on weights of this size
// takes of a card: the weights in 16 bits, as much again for what passes
// through them, and the CUDA context. Measured at 2.3 GB for a 0.5B model.
func TrainMemoryMiB(weightBytes int64) int {
mib := int(weightBytes>>20)*2 + 1024
return (mib + 255) / 256 * 256
}
// trainBaseFor is how long what the registry said of a model's weights is
// kept: it changes when the model is updated, which is rare.
const trainBaseFor = time.Hour
// trainBase is the safetensors weights to train m on: its own origin's,
// when they are safetensors, or those its card says it was made from — a
// GGUF's base model.
func (a *API) trainBase(ctx context.Context, m catalog.Model) trainBaseInfo {
if m.Origin == nil || m.Origin.Repo == "" {
return trainBaseInfo{err: refused(http.StatusBadRequest, "%s's weights did not come from the model registry, so "+
"what to train it on is not known: an administrator can name them (llmgw ctl train start --base-repo)", m.Name)}
}
if a.train.Registry == nil {
return trainBaseInfo{err: refused(http.StatusNotImplemented, "the model registry is not reachable from here "+
"(mirror.registry): the weights to train %s on cannot be found", m.Name)}
}
a.train.mu.Lock()
b, ok := a.train.bases[m.Origin.Repo]
a.train.mu.Unlock()
if ok && time.Since(b.at) < trainBaseFor {
return b
}
b = a.findTrainBase(ctx, m)
if b.err != nil {
var te *trainErr
if !errors.As(b.err, &te) {
return b // the registry did not answer: asked again next time
}
}
b.at = time.Now()
a.train.mu.Lock()
if a.train.bases == nil {
a.train.bases = map[string]trainBaseInfo{}
}
a.train.bases[m.Origin.Repo] = b
a.train.mu.Unlock()
return b
}
func (a *API) findTrainBase(ctx context.Context, m catalog.Model) trainBaseInfo {
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
repo, err := a.train.Registry.RepoAt(ctx, m.Origin.Repo, "")
if err != nil {
return trainBaseInfo{err: fmt.Errorf("asking the model registry about %s: %w", m.Origin.Repo, err)}
}
candidates := []hf.Repo{repo}
for _, id := range repo.BaseModel {
if id == repo.ID || strings.Count(id, "/") != 1 {
continue
}
r, err := a.train.Registry.RepoAt(ctx, id, "")
if err != nil {
continue
}
candidates = append(candidates, r)
}
gated := ""
for _, r := range candidates {
var n int64
for _, f := range r.Files {
if strings.HasSuffix(strings.ToLower(f.Path), ".safetensors") {
n += f.Bytes
}
}
if n == 0 {
continue
}
if r.Gated.Is() {
gated = r.ID
continue
}
return trainBaseInfo{repo: r.ID, revision: r.Revision, bytes: n}
}
if gated != "" {
return trainBaseInfo{err: refused(http.StatusBadRequest, "the weights to train %s on, %s, are gated on the registry: "+
"the nodes cannot fetch them without an account that accepted their terms", m.Name, gated)}
}
return trainBaseInfo{err: refused(http.StatusBadRequest, "neither %s, where %s's weights came from, nor the model its card "+
"names has safetensors weights to train on", m.Origin.Repo, m.Name)}
}
// PickTrainNode is the node whose freest card has the most memory free,
// if that is need, among those ready and not busy.
func (a *API) PickTrainNode(need int, busy map[string]bool) string {
best, bestFree := "", 0
for _, n := range a.ctl.Inventory().Nodes() {
if busy[n.ID] || n.State != inventory.Ready {
continue
}
free := 0
if n.Load != nil {
for _, g := range n.Load.GPUs {
if g.MemoryTotalMB > 0 {
free = max(free, int(g.MemoryTotalMB-g.MemoryUsedMB))
}
}
}
if free == 0 {
for _, g := range n.GPUs {
free = max(free, g.MemoryTotalMB-g.MemoryUsedMB)
}
}
if free >= need && free > bestFree {
best, bestFree = n.ID, free
}
}
return best
} }
// trainOwner is who a job's adapter will belong to, as scope:ID: one that // trainOwner is who a job's adapter will belong to, as scope:ID: one that
@@ -453,73 +686,74 @@ func (a *API) trainPublish(w http.ResponseWriter, r *http.Request) {
if !ok { if !ok {
return return
} }
if a.train.Mirror == nil { v, err := a.publishTrained(r.Context(), j, actorOf(r))
writeError(w, http.StatusNotImplemented, "publishing needs the model mirror (mirror.api in the settings); "+ if err != nil {
"llmgw ctl train download, then llmgw mirror import there, does it by hand") writeTrainErr(w, err)
return return
} }
writeJSON(w, http.StatusOK, v)
}
// publishTrained puts what a job made on the mirror and makes it the
// model's adapter: the GGUF for a model llama.cpp serves, the PEFT
// directory for one vLLM serves.
func (a *API) publishTrained(ctx context.Context, j store.TrainJob, actor string) (TrainPublishView, error) {
var none TrainPublishView
if a.train.Mirror == nil {
return none, refused(http.StatusNotImplemented, "publishing needs the model mirror (mirror.api in the settings); "+
"llmgw ctl train download, then llmgw mirror import there, does it by hand")
}
if a.models == nil { if a.models == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store for the catalog") return none, refused(http.StatusNotImplemented, "this control plane has no store for the catalog")
return
} }
if j.State != store.TrainDone { if j.State != store.TrainDone {
writeError(w, http.StatusConflict, "the job has not finished: it is "+j.State) return none, refused(http.StatusConflict, "%s", "the job has not finished: it is "+j.State)
return
} }
m, found := a.ctl.Catalog().Resolve(j.Model) m, found := a.ctl.Catalog().Resolve(j.Model)
if !found { if !found {
writeError(w, http.StatusConflict, "the model "+j.Model+" is no longer in the catalog") return none, refused(http.StatusConflict, "%s", "the model "+j.Model+" is no longer in the catalog")
return
} }
for _, ad := range m.Adapters { for _, ad := range m.Adapters {
if ad.Name == j.Name { if ad.Name == j.Name {
writeError(w, http.StatusConflict, j.Name+" is already "+m.Name+"'s adapter; train under another name") return none, refused(http.StatusConflict, "%s", j.Name+" is already "+m.Name+"'s adapter; train under another name")
return
} }
} }
llama := slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" }) llama := slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" })
source := "trained by llmgw job " + j.ID + " on " + j.Node + " from " + j.BaseRepo source := "trained by llmgw job " + j.ID + " on " + j.Node + " from " + j.BaseRepo
var url string var url string
if llama { if llama {
b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, "gguf") b, err := a.train.Store.TrainArtifact(ctx, j.ID, "gguf")
if err != nil { if err != nil {
writeError(w, http.StatusConflict, "the job made no GGUF, which llama.cpp needs; train again without --no-gguf") return none, refused(http.StatusConflict, "the job made no GGUF, which llama.cpp needs; train again without --no-gguf")
return
} }
res, err := a.train.Mirror.Upload(r.Context(), j.Name, j.ID, j.Name+".gguf", b, source) res, err := a.train.Mirror.Upload(ctx, j.Name, j.ID, j.Name+".gguf", b, source)
if err != nil { if err != nil {
writeError(w, http.StatusBadGateway, "keeping it on the mirror: "+err.Error()) return none, refused(http.StatusBadGateway, "%s", "keeping it on the mirror: "+err.Error())
return
} }
url = res.URL url = res.URL
} else { } else {
b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, "peft") b, err := a.train.Store.TrainArtifact(ctx, j.ID, "peft")
if err != nil { if err != nil {
writeError(w, http.StatusConflict, "the job made no PEFT adapter") return none, refused(http.StatusConflict, "the job made no PEFT adapter")
return
} }
files, err := untarAll(b) files, err := untarAll(b)
if err != nil { if err != nil {
writeError(w, http.StatusInternalServerError, err.Error()) return none, err
return
} }
for name, data := range files { for name, data := range files {
res, err := a.train.Mirror.Upload(r.Context(), j.Name, j.ID, name, data, source) res, err := a.train.Mirror.Upload(ctx, j.Name, j.ID, name, data, source)
if err != nil { if err != nil {
writeError(w, http.StatusBadGateway, "keeping "+name+" on the mirror: "+err.Error()) return none, refused(http.StatusBadGateway, "%s", "keeping "+name+" on the mirror: "+err.Error())
return
} }
url = res.URL[:strings.LastIndex(res.URL, "/")+1] url = res.URL[:strings.LastIndex(res.URL, "/")+1]
} }
} }
ad := catalog.Adapter{Name: j.Name, Source: url, Owner: j.Owner} ad := catalog.Adapter{Name: j.Name, Source: url, Owner: j.Owner}
if err := checkAdapters(r.Context(), m, []catalog.Adapter{ad}); err != nil { if err := checkAdapters(ctx, m, []catalog.Adapter{ad}); err != nil {
writeError(w, http.StatusBadRequest, err.Error()) return none, refused(http.StatusBadRequest, "%v", err)
return
} }
if _, err := a.models.SetAdapters(r.Context(), m.Name, append(slices.Clone(m.Adapters), ad), actorOf(r)); err != nil { if _, err := a.models.SetAdapters(ctx, m.Name, append(slices.Clone(m.Adapters), ad), actor); err != nil {
writeError(w, http.StatusBadRequest, err.Error()) return none, refused(http.StatusBadRequest, "%v", err)
return
} }
v := TrainPublishView{Adapter: j.Name, Model: m.Name, Source: url, Owner: j.Owner} v := TrainPublishView{Adapter: j.Name, Model: m.Name, Source: url, Owner: j.Owner}
if llama { if llama {
@@ -528,7 +762,48 @@ func (a *API) trainPublish(w http.ResponseWriter, r *http.Request) {
} else { } else {
v.Note = "ask for model " + j.Name + "; a replica loads it on the first request" v.Note = "ask for model " + j.Name + "; a replica loads it on the first request"
} }
writeJSON(w, http.StatusOK, v) return v, nil
}
func (a *API) trainLog() *slog.Logger {
if a.train.Log == nil {
return slog.New(slog.DiscardHandler)
}
return a.train.Log
}
// TrainDone publishes what a job made, when it asked for that: on the
// mirror, as its model's adapter, and for a llama.cpp model with the
// model's replicas replaced, one at a time beside themselves, to load it.
func (a *API) TrainDone(j store.TrainJob) {
if !j.Publish || a.train == nil {
return
}
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Minute)
defer cancel()
note := "published: ask for model " + j.Name
v, err := a.publishTrained(ctx, j, "training:"+j.ID)
if err != nil {
note = "trained, but publishing it failed: " + err.Error()
a.trainLog().Warn("a trained adapter could not be published", "job", j.ID, "adapter", j.Name, "err", err)
} else if strings.HasPrefix(v.Note, "llama.cpp") {
var ids []string
for _, rep := range a.ctl.Reconciler().ReadyReplicas() {
if rep.Model == v.Model {
ids = append(ids, rep.ID)
}
}
if len(ids) > 0 {
if err := a.ctl.Reconciler().RestartPaced(reconciler.RestartPace{}, false, ids...); err != nil {
note = "published, but " + v.Model + "'s replicas could not be restarted to load it: " + err.Error()
} else {
note = "published: " + v.Model + "'s replicas are restarting to load it; then ask for model " + j.Name
}
}
}
if err := a.train.Store.SetTrainNote(ctx, j.ID, note); err != nil {
a.trainLog().Warn("could not record a trained adapter's publishing", "job", j.ID, "err", err)
}
} }
// untarAll reads a gzipped tar of a PEFT directory's files. // untarAll reads a gzipped tar of a PEFT directory's files.
+114
View File
@@ -0,0 +1,114 @@
package adminapi
import (
"context"
"errors"
"net/http"
"slices"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// Training from the portal's "customise a model" page: the page decides who
// may do what (members of the job's project), and these do it.
// TrainableModels are the catalog's models an adapter can be trained for
// from the portal: chat models llama.cpp or vLLM serve whose weights came
// from the registry, so the weights to train on can be found.
func (a *API) TrainableModels() []string {
var out []string
for _, m := range a.ctl.Catalog().Models() {
if m.IsGuard() || m.IsExternal() || m.Capabilities.Embeddings || m.MakesImages() || m.MakesMeshes() ||
m.Origin == nil || m.Origin.Repo == "" {
continue
}
if slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" || p.Engine == "vllm" }) {
out = append(out, m.Name)
}
}
slices.Sort(out)
return out
}
// Trainings are the jobs started in projects, or by createdBy outside any.
func (a *API) Trainings(ctx context.Context, projects []string, createdBy string) ([]store.TrainJob, error) {
if a.train == nil {
return nil, nil
}
return a.train.Store.TrainJobsFor(ctx, projects, createdBy, 100)
}
// Training is one job.
func (a *API) Training(ctx context.Context, id string) (store.TrainJob, error) {
if a.train == nil {
return store.TrainJob{}, store.ErrNotFound
}
return a.train.Store.TrainJob(ctx, id)
}
// Published is whether a job's adapter is its model's in the catalog.
func (a *API) Published(j store.TrainJob) bool {
m, ok := a.ctl.Catalog().Resolve(j.Model)
return ok && slices.ContainsFunc(m.Adapters, func(ad catalog.Adapter) bool { return ad.Name == j.Name })
}
// ShareTraining changes who a job's adapter belongs to — owner as scope:ID,
// or "" for every caller its model allows — on the job and, once
// published, in the catalog. Refused wider than the knowledge bases its
// pairs came from may be read.
func (a *API) ShareTraining(ctx context.Context, id, owner, actor string) error {
j, err := a.Training(ctx, id)
if err != nil {
return err
}
_, colls, err := countPairs(string(j.Data))
if err != nil {
return err
}
resolved, err := a.trainOwner(ctx, TrainRequest{Owner: owner, Public: owner == ""}, colls)
if err != nil {
return refused(http.StatusBadRequest, "%v", err)
}
if err := a.train.Store.SetTrainOwner(ctx, id, resolved); err != nil {
return err
}
m, ok := a.ctl.Catalog().Resolve(j.Model)
if !ok || a.models == nil {
return nil
}
ads := slices.Clone(m.Adapters)
i := slices.IndexFunc(ads, func(ad catalog.Adapter) bool { return ad.Name == j.Name })
if i < 0 {
return nil
}
ads[i].Owner = resolved
_, err = a.models.SetAdapters(ctx, m.Name, ads, actor)
return err
}
// DeleteTraining stops a job if it trains, takes its adapter out of the
// catalog if it was published, and forgets it.
func (a *API) DeleteTraining(ctx context.Context, id, actor string) error {
j, err := a.Training(ctx, id)
if err != nil {
return err
}
if !store.TrainOver(j.State) {
if err := a.train.Service.Cancel(ctx, id); err != nil {
return err
}
}
if m, ok := a.ctl.Catalog().Resolve(j.Model); ok && a.models != nil {
ads := slices.DeleteFunc(slices.Clone(m.Adapters), func(ad catalog.Adapter) bool { return ad.Name == j.Name })
if len(ads) != len(m.Adapters) {
if _, err := a.models.SetAdapters(ctx, m.Name, ads, actor); err != nil {
return err
}
}
}
if err := a.train.Store.DeleteTrainJob(ctx, id); err != nil && !errors.Is(err, store.ErrNotFound) {
return err
}
return nil
}
@@ -0,0 +1,137 @@
package adminapi
import (
"context"
"errors"
"path/filepath"
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
type fakeRegistry struct {
repos map[string]hf.Repo
asked int
}
func (f *fakeRegistry) RepoAt(_ context.Context, id, _ string) (hf.Repo, error) {
f.asked++
r, ok := f.repos[id]
if !ok {
return hf.Repo{}, errors.New("404")
}
return r, nil
}
func TestAPortalTrainingFindsItsWeightsAndMemoryAndOneTrainsPerProject(t *testing.T) {
ctx := context.Background()
models := append(testModels(),
catalog.Model{Name: "small", Source: "/s", Origin: &catalog.Origin{Repo: "bartowski/Small-GGUF"},
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}},
catalog.Model{Name: "gated", Source: "/g", Origin: &catalog.Origin{Repo: "meta/Gated"},
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}},
catalog.Model{Name: "local", Source: "/l", Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}})
a, _ := setup(t, models, gpuNode("cuda01", 1))
s, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "s.db"))
if err != nil {
t.Fatal(err)
}
defer s.Close()
reg := &fakeRegistry{repos: map[string]hf.Repo{
"bartowski/Small-GGUF": {ID: "bartowski/Small-GGUF", Revision: "g1", BaseModel: []string{"Org/Small"},
Files: []hf.File{{Path: "small-Q4_K_M.gguf", Bytes: 400 << 20}}},
"Org/Small": {ID: "Org/Small", Revision: "abc", Files: []hf.File{{Path: "model.safetensors", Bytes: 1000 << 20},
{Path: "config.json", Bytes: 1 << 10}}},
"meta/Gated": {ID: "meta/Gated", Gated: hf.Gated{Reason: "manual"},
Files: []hf.File{{Path: "model.safetensors", Bytes: 1 << 30}}},
}}
a.SetTrain(&TrainAdmin{Store: s, Service: train.New(s, noAgents{}, nil)})
a.SetTrainMirror(nil, reg)
if got := a.TrainableModels(); strings.Join(got, ",") != "gated,small" {
t.Errorf("trainable: %v", got)
}
data := train.NDJSON([]train.Pair{{Q: "a", A: "b"}})
req := TrainRequest{Name: "tone", Model: "small", Data: data, Owner: "project:p1", Project: "p1", Publish: true}
j, err := a.StartTraining(ctx, req, "ayse@x")
if err != nil {
t.Fatal(err)
}
if j.BaseRepo != "Org/Small" || j.BaseRevision != "abc" || j.Node != "" || j.GPUMemoryMiB != TrainMemoryMiB(1000<<20) ||
!j.Publish || j.Project != "p1" || j.CreatedBy != "ayse@x" || !j.GGUF {
t.Errorf("queued %+v", j)
}
if j.GPUMemoryMiB != 3072 {
t.Errorf("1000 MiB of weights take %d MiB to train", j.GPUMemoryMiB)
}
req.Name = "tone2"
if _, err := a.StartTraining(ctx, req, "mehmet@x"); err == nil || !strings.Contains(err.Error(), "already has a model training") {
t.Errorf("a second in the project: %v", err)
}
req.Name, req.Project, req.Owner = "tone", "p2", "project:p2"
if _, err := a.StartTraining(ctx, req, "x"); err == nil || !strings.Contains(err.Error(), "already being trained") {
t.Errorf("the same name twice: %v", err)
}
req.Name = "tone3"
asked := reg.asked
if _, err := a.StartTraining(ctx, req, "x"); err != nil {
t.Errorf("another project: %v", err)
}
if reg.asked != asked {
t.Errorf("the registry was asked again: %d", reg.asked-asked)
}
for model, says := range map[string]string{"gated": "gated", "local": "did not come from the model registry"} {
req.Model, req.Name, req.Project = model, "x-"+model, "p-"+model
_, err := a.StartTraining(ctx, req, "x")
if err == nil || !strings.Contains(err.Error(), says) {
t.Errorf("%s: %v", model, err)
}
if msg, theirs := TrainRefusal(err); !theirs || msg == "" {
t.Errorf("%s is not the request's fault: %q", model, msg)
}
}
jobs, err := a.Trainings(ctx, []string{"p1"}, "nobody")
if err != nil || len(jobs) != 1 || jobs[0].Name != "tone" {
t.Errorf("p1's: %+v %v", jobs, err)
}
if err := a.ShareTraining(ctx, j.ID, "", "ayse@x"); err != nil {
t.Fatal(err)
}
if got, _ := a.Training(ctx, j.ID); got.Owner != "" {
t.Errorf("shared with everyone: %q", got.Owner)
}
if err := a.DeleteTraining(ctx, j.ID, "ayse@x"); err != nil {
t.Fatal(err)
}
if _, err := a.Training(ctx, j.ID); !errors.Is(err, store.ErrNotFound) {
t.Errorf("deleted: %v", err)
}
}
func TestATrainingGoesToTheNodeWithTheMostFree(t *testing.T) {
busy := gpuNode("busy", 1)
busy.Load = &inventory.Load{GPUs: []inventory.GPUSensor{{Index: 0, MemoryTotalMB: 24000, MemoryUsedMB: 23000}}}
roomy := gpuNode("roomy", 1)
roomy.Load = &inventory.Load{GPUs: []inventory.GPUSensor{{Index: 0, MemoryTotalMB: 24000, MemoryUsedMB: 4000}}}
cpu := inventory.Node{ID: "cpu", State: inventory.Ready}
small := gpuNode("small", 1)
small.GPUs = []discovery.GPU{{Index: 0, MemoryTotalMB: 8000}}
a, _ := setup(t, testModels(), busy, roomy, cpu, small)
if got := a.PickTrainNode(3000, nil); got != "roomy" {
t.Errorf("picked %q", got)
}
if got := a.PickTrainNode(3000, map[string]bool{"roomy": true}); got != "small" {
t.Errorf("with roomy busy, picked %q", got)
}
if got := a.PickTrainNode(30000, nil); got != "" {
t.Errorf("none has 30 GB, picked %q", got)
}
}
+59
View File
@@ -0,0 +1,59 @@
package adminapi
import (
"context"
"errors"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
)
// What the portal changes of a served model beside what the catalog holds:
// the same as the API's routes, called in the process.
// SetModelContext changes a served model's context and longest answer.
func (a *API) SetModelContext(ctx context.Context, name string, req modelserve.ContextRequest, by string) (modelserve.ContextChange, error) {
if a.serve == nil {
return modelserve.ContextChange{}, errors.New("this control plane cannot change a model's context")
}
return a.serve.SetContext(ctx, name, req, by)
}
// SetModelGPUs spreads a served model over more cards a replica, or fewer.
func (a *API) SetModelGPUs(ctx context.Context, name string, req modelserve.GPUsRequest, by string) (modelserve.GPUsChange, error) {
if a.serve == nil {
return modelserve.GPUsChange{}, errors.New("this control plane cannot change a model's cards")
}
return a.serve.SetGPUs(ctx, name, req, by)
}
// RestartModel starts each of a model's ready replicas again, one at a time,
// and says how many.
func (a *API) RestartModel(name string) (int, error) {
var ids []string
for _, rep := range a.ctl.Reconciler().ReadyReplicas() {
if rep.Model == name {
ids = append(ids, rep.ID)
}
}
if len(ids) == 0 {
return 0, errors.New("none of its replicas is ready to restart")
}
return len(ids), a.ctl.Reconciler().RestartPaced(reconciler.RestartPace{}, false, ids...)
}
// RemoveModel takes a model out of the catalog, with its files on the mirror
// and the nodes when purge.
func (a *API) RemoveModel(ctx context.Context, name string, purge bool, by string) (PurgeView, error) {
if purge {
return a.Purge(ctx, name, by)
}
if a.models == nil {
return PurgeView{}, errors.New("this control plane has no store")
}
if err := a.models.Remove(ctx, name); err != nil {
return PurgeView{}, err
}
a.presetDeclined(ctx, name, by)
return PurgeView{Removed: name}, nil
}
+110 -13
View File
@@ -1,11 +1,14 @@
package adminapi package adminapi
import ( import (
"context"
"encoding/json" "encoding/json"
"net/http" "net/http"
"sort" "sort"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights" "gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
) )
// OrphanView is cached weights on one node that no catalog model accounts for. // OrphanView is cached weights on one node that no catalog model accounts for.
@@ -71,12 +74,100 @@ type CollectRequest struct {
Node string `json:"node,omitempty"` Node string `json:"node,omitempty"`
} }
// CollectResult is what was deleted on one node, or why nothing was. // CollectResult is what was deleted on one node, or why nothing was: what
// the node no longer lists once asked, not what it was asked to delete.
type CollectResult struct { type CollectResult struct {
Node string `json:"node"` Node string `json:"node"`
Files int `json:"files"` Files int `json:"files"`
Bytes int64 `json:"bytes"` Bytes int64 `json:"bytes"`
Error string `json:"error,omitempty"` // Left and LeftBytes are what the node still lists: refused, or not
// gone by the time the sweep stopped waiting. Its agent's log says why.
Left int `json:"left,omitempty"`
LeftBytes int64 `json:"left_bytes,omitempty"`
Error string `json:"error,omitempty"`
}
// confirmWait is how long a sweep waits for the nodes to list their files
// without the ones deleted: they scan theirs every heartbeat, five seconds
// by default, and a large set takes a moment to delete.
var confirmWait = 30 * time.Second
// removal is what one node is asked to delete, with each file's size.
type removal struct {
node string
paths []string
size map[string]int64
}
// removeConfirmed asks each node to delete its paths and says what each then
// no longer lists. Counted as asked, a sweep said "25 GB freed" of files the
// node had refused and kept.
func (a *API) removeConfirmed(ctx context.Context, rm control.WeightRemover, rs []removal) []CollectResult {
out := make([]CollectResult, len(rs))
pending := map[int]bool{}
for i, r := range rs {
out[i].Node = r.node
if err := rm.RemoveWeights(ctx, r.node, r.paths); err != nil {
out[i].Error = err.Error()
for _, p := range r.paths {
out[i].Left++
out[i].LeftBytes += r.size[p]
}
continue
}
pending[i] = true
}
deadline := time.Now().Add(confirmWait)
for {
for i := range pending {
if left := a.stillListed(rs[i]); len(left) == 0 {
delete(pending, i)
}
}
if len(pending) == 0 || !time.Now().Before(deadline) {
break
}
select {
case <-ctx.Done():
deadline = time.Now()
case <-time.After(time.Second):
}
}
for i, r := range rs {
if out[i].Error != "" {
continue
}
left := a.stillListed(r)
for _, p := range r.paths {
if left[p] {
out[i].Left++
out[i].LeftBytes += r.size[p]
} else {
out[i].Files++
out[i].Bytes += r.size[p]
}
}
}
return out
}
// stillListed is which of a removal's paths its node still lists.
func (a *API) stillListed(r removal) map[string]bool {
left := map[string]bool{}
n, ok := a.ctl.Inventory().Get(r.node)
if !ok {
return left
}
want := map[string]bool{}
for _, p := range r.paths {
want[p] = true
}
for _, w := range n.Weights {
if want[w.Path] {
left[w.Path] = true
}
}
return left
} }
// collectWeights deletes them. // collectWeights deletes them.
@@ -97,20 +188,26 @@ func (a *API) collectWeights(w http.ResponseWriter, r *http.Request) {
// An empty body is the "all nodes" case and must not be an error. // An empty body is the "all nodes" case and must not be an error.
_ = json.NewDecoder(r.Body).Decode(&body) _ = json.NewDecoder(r.Body).Decode(&body)
out := make([]CollectResult, 0) // A node that has gone since the list was built is reported rather than
// failing the whole sweep: the other nodes' space is still worth
// freeing, and this one will be offered again next time.
var rs []removal
for _, o := range a.Orphans() { for _, o := range a.Orphans() {
if body.Node != "" && o.Node != body.Node { if body.Node != "" && o.Node != body.Node {
continue continue
} }
res := CollectResult{Node: o.Node, Files: len(o.Paths), Bytes: o.Bytes} rs = append(rs, removal{node: o.Node, paths: o.Paths, size: a.sizesOn(o.Node)})
// A node that has gone since the list was built is reported rather than
// failing the whole sweep: the other nodes' space is still worth
// freeing, and this one will be offered again next time.
if err := rm.RemoveWeights(r.Context(), o.Node, o.Paths); err != nil {
res.Error = err.Error()
res.Files, res.Bytes = 0, 0
}
out = append(out, res)
} }
writeJSON(w, http.StatusOK, out) writeJSON(w, http.StatusOK, a.removeConfirmed(r.Context(), rm, rs))
}
// sizesOn is each file a node lists, by path, with its size.
func (a *API) sizesOn(node string) map[string]int64 {
out := map[string]int64{}
if n, ok := a.ctl.Inventory().Get(node); ok {
for _, w := range n.Weights {
out[w.Path] = w.SizeBytes
}
}
return out
} }
+71
View File
@@ -0,0 +1,71 @@
package adminapi
import (
"context"
"encoding/json"
"log/slog"
"net/http"
"net/http/httptest"
"slices"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
)
// deletingDispatcher deletes what it is asked to as a node would, by its
// files dropping out of what the node lists — but for refused, which it
// keeps, as an agent keeps a file it cannot prove it fetched.
type deletingDispatcher struct {
nullDispatcher
inv *inventory.Store
refused string
}
func (d *deletingDispatcher) RemoveWeights(_ context.Context, node string, paths []string) error {
n, _ := d.inv.Get(node)
n.Weights = slices.DeleteFunc(n.Weights, func(w discovery.LocalWeights) bool {
return w.Path != d.refused && slices.Contains(paths, w.Path)
})
d.inv.Upsert(n)
return nil
}
// What a sweep says it freed is what the node no longer lists: a file it
// kept is said to be left, not counted as freed.
func TestASweepSaysWhatWasDeletedNotWhatWasAsked(t *testing.T) {
confirmWait = 1500 * time.Millisecond
t.Cleanup(func() { confirmWait = 30 * time.Second })
gone := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/old/v1/old.gguf")
set := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/big/v1")
kept := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/stuck/v1/stuck.gguf")
cat, _ := catalog.New(nil)
d := &deletingDispatcher{refused: kept}
ctl := control.New(control.DefaultConfig(), catalog.NewLive(cat), d, nullSink{}, slog.New(slog.DiscardHandler))
d.inv = ctl.Inventory()
node := gpuNode("n1", 1)
node.Weights = []discovery.LocalWeights{{Path: gone, SizeBytes: 1 << 30}, {Path: set, SizeBytes: 20 << 30},
{Path: kept, SizeBytes: 4 << 30}}
ctl.Inventory().Upsert(node)
a := New(ctl)
rec := httptest.NewRecorder()
a.collectWeights(rec, httptest.NewRequest(http.MethodPost, "/api/v1/weights/collect", strings.NewReader(`{"node":"n1"}`)))
var got []CollectResult
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatalf("%v: %s", err, rec.Body.String())
}
if len(got) != 1 {
t.Fatalf("results %+v", got)
}
r := got[0]
if r.Files != 2 || r.Bytes != 21<<30 || r.Left != 1 || r.LeftBytes != 4<<30 {
t.Errorf("result %+v: want 2 files and 21 GB freed, 1 file of 4 GB left", r)
}
}
+2 -1
View File
@@ -36,7 +36,8 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
n.Host = discovery.Host{ n.Host = discovery.Host{
OSID: h.GetOsId(), OSVersion: h.GetOsVersion(), OSCodename: h.GetOsCodename(), OSID: h.GetOsId(), OSVersion: h.GetOsVersion(), OSCodename: h.GetOsCodename(),
Arch: h.GetArch(), CPUFlags: h.GetCpuFlags(), NVIDIADriver: h.GetNvidiaDriver(), Arch: h.GetArch(), CPUFlags: h.GetCpuFlags(), NVIDIADriver: h.GetNvidiaDriver(),
MemBandwidthGBs: h.GetMemBandwidthGbs(), NVIDIADevice: h.GetNvidiaDevice(),
MemBandwidthGBs: h.GetMemBandwidthGbs(), MemBandwidthCoreGBs: h.GetMemBandwidthCoreGbs(),
} }
} }
for _, w := range r.GetWeights() { for _, w := range r.GetWeights() {
+15 -1
View File
@@ -84,6 +84,8 @@ type Service struct {
agentSettings func(node string) string agentSettings func(node string) string
// train is told how training jobs go; see SetTrain. // train is told how training jobs go; see SetTrain.
train TrainHandler train TrainHandler
// slurm is told of the nodes' part in Slurm; see SetSlurm.
slurm SlurmHandler
// logWaits are the log requests awaiting a reply, by request id. // logWaits are the log requests awaiting a reply, by request id.
logMu sync.Mutex logMu sync.Mutex
@@ -308,6 +310,10 @@ func (s *Service) Connect(
return err return err
} }
} }
// Its part in Slurm, sent by the Slurm service's next pass.
if h := s.slurmHandler(); h != nil {
h.OnRegister(nodeID)
}
// Recover replica state from what this node is already running, before any // Recover replica state from what this node is already running, before any
// reconciliation pass can conclude the cluster is empty and place // reconciliation pass can conclude the cluster is empty and place
@@ -515,6 +521,11 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
h.OnTrainStatus(nodeID, m.TrainStatus) h.OnTrainStatus(nodeID, m.TrainStatus)
} }
case *llmgwv1.AgentMessage_SlurmStatus:
if h := s.slurmHandler(); h != nil {
h.OnSlurmStatus(nodeID, m.SlurmStatus)
}
case *llmgwv1.AgentMessage_TrainArtifact: case *llmgwv1.AgentMessage_TrainArtifact:
if h := s.trainHandler(); h != nil { if h := s.trainHandler(); h != nil {
h.OnTrainArtifact(nodeID, m.TrainArtifact) h.OnTrainArtifact(nodeID, m.TrainArtifact)
@@ -547,7 +558,7 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
storage := hb.GetStorage() storage := hb.GetStorage()
memMB := int(hb.GetMemoryTotalMb()) memMB := int(hb.GetMemoryTotalMb())
cpus := int(hb.GetCpuCount()) cpus := int(hb.GetCpuCount())
bw := hb.GetMemBandwidthGbs() bw, coreBW := hb.GetMemBandwidthGbs(), hb.GetMemBandwidthCoreGbs()
if len(used) == 0 && len(foreign) == 0 && len(problems) == 0 && len(storage) == 0 && if len(used) == 0 && len(foreign) == 0 && len(problems) == 0 && len(storage) == 0 &&
!hb.GetWeightsChanged() && memMB == 0 && cpus == 0 && !hb.GetEnginesChanged() && !hb.GetWeightsChanged() && memMB == 0 && cpus == 0 && !hb.GetEnginesChanged() &&
!hb.GetProvisionChanged() && hb.GetLoad() == nil && len(hb.GetIbPorts()) == 0 && !hb.GetProvisionChanged() && hb.GetLoad() == nil && len(hb.GetIbPorts()) == 0 &&
@@ -626,6 +637,9 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
if bw > 0 { if bw > 0 {
n.Host.MemBandwidthGBs = bw n.Host.MemBandwidthGBs = bw
} }
if coreBW > 0 {
n.Host.MemBandwidthCoreGBs = coreBW
}
if cpus > 0 && cpus != n.CPUCount { if cpus > 0 && cpus != n.CPUCount {
if n.CPUCount > 0 { if n.CPUCount > 0 {
s.log.Info("node CPUs changed", "node", nodeID, "from", n.CPUCount, "to", cpus) s.log.Info("node CPUs changed", "node", nodeID, "from", n.CPUCount, "to", cpus)
+25
View File
@@ -0,0 +1,25 @@
package agentsvc
import (
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
)
// SlurmHandler is told of the nodes' part in Slurm (BACKLOG P).
type SlurmHandler interface {
OnSlurmStatus(node string, st *llmgwv1.SlurmStatus)
// OnRegister is a node's new stream, which is told again what it is.
OnRegister(node string)
}
// SetSlurm gives the service what Slurm's messages go to.
func (s *Service) SetSlurm(h SlurmHandler) {
s.mu.Lock()
s.slurm = h
s.mu.Unlock()
}
func (s *Service) slurmHandler() SlurmHandler {
s.mu.RLock()
defer s.mu.RUnlock()
return s.slurm
}
+56
View File
@@ -267,6 +267,12 @@ type Profile struct {
Weight float64 `json:"weight,omitempty" koanf:"weight"` Weight float64 `json:"weight,omitempty" koanf:"weight"`
// Priority orders profiles during placement; highest that fits wins. // Priority orders profiles during placement; highest that fits wins.
Priority int `json:"priority,omitempty" koanf:"priority"` Priority int `json:"priority,omitempty" koanf:"priority"`
// Replicas, set on any of a model's profiles, is how many of its
// replicas run on this one: seven on cards and four on CPUs, all
// answering under the model's name, rather than each replica on the
// best profile that fits. The model's scaling is then the sum, min and
// max alike; a profile left at zero runs none.
Replicas int `json:"replicas,omitempty" koanf:"replicas"`
// StartupTimeout bounds model loading. A 70B model across two nodes takes // StartupTimeout bounds model loading. A 70B model across two nodes takes
// tens of minutes, so this is per profile rather than global. // tens of minutes, so this is per profile rather than global.
StartupTimeout time.Duration `json:"startup_timeout,omitempty" koanf:"startup_timeout"` StartupTimeout time.Duration `json:"startup_timeout,omitempty" koanf:"startup_timeout"`
@@ -1024,10 +1030,57 @@ func (m Model) normalize() Model {
return m return m
} }
// Split says whether the model's replicas are counted per profile: any of
// its profiles sets replicas.
func (m Model) Split() bool {
return slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Replicas > 0 })
}
// SplitTotal is the replicas its profiles ask for together.
func (m Model) SplitTotal() int {
n := 0
for _, p := range m.Profiles {
n += max(p.Replicas, 0)
}
return n
}
// ProfileBase is the catalog profile a placement profile stands for: one
// named by the card it needs is placed as name@class.
func ProfileBase(name string) string {
base, _, _ := strings.Cut(name, "@")
return base
}
// validateSplit refuses replicas per profile that the model's scaling does
// not add up to.
func (m Model) validateSplit() error {
for _, p := range m.Profiles {
if p.Replicas < 0 {
return fmt.Errorf("model %q profile %q: replicas %d is negative", m.Name, p.Name, p.Replicas)
}
}
if !m.Split() || m.Stopped != nil {
return nil
}
n := m.SplitTotal()
if m.Scaling.Min != n || m.Scaling.Max != n {
return fmt.Errorf("model %q: its profiles' replicas add up to %d, so its scaling is min %d, max %d, not %d and %d",
m.Name, n, n, n, m.Scaling.Min, m.Scaling.Max)
}
if len(m.Scaling.Schedule) > 0 {
return fmt.Errorf("model %q: replicas per profile and a schedule are one or the other", m.Name)
}
return nil
}
func (m Model) validate() error { func (m Model) validate() error {
if m.Name == "" { if m.Name == "" {
return fmt.Errorf("a model has no name") return fmt.Errorf("a model has no name")
} }
if err := m.validateSplit(); err != nil {
return err
}
if m.Source == "" { if m.Source == "" {
return fmt.Errorf("model %q: source is required", m.Name) return fmt.Errorf("model %q: source is required", m.Name)
} }
@@ -1510,6 +1563,9 @@ func (m Model) Simulated() bool {
// happened, and the answer to "why is this one on the wrong node" was in // happened, and the answer to "why is this one on the wrong node" was in
// somebody's memory of what they edited an hour ago. // somebody's memory of what they edited an hour ago.
func (p Profile) Fingerprint() string { func (p Profile) Fingerprint() string {
// How many replicas run on it is not how each runs: changing the count
// must not mark the running ones stale.
p.Replicas = 0
b, err := json.Marshal(p) b, err := json.Marshal(p)
if err != nil { if err != nil {
// A profile that will not marshal cannot be compared, and saying so by // A profile that will not marshal cannot be compared, and saying so by
+93
View File
@@ -0,0 +1,93 @@
// Package held records, on the leader, what each model holds of the
// cluster: its replicas' GPUs and, for one on CPUs, its cores. Kept by the
// hour, it is the cost side of a token's price (llmgw ctl prices suggest).
package held
import (
"context"
"log/slog"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// Inventory is where the allocations are.
type Inventory interface {
Nodes() []inventory.Node
Allocations(nodeID string) []inventory.Allocation
}
// Store keeps what was held.
type Store interface {
AddHeld(ctx context.Context, at time.Time, held []store.Held) error
}
// Recorder adds, every Every, what each model holds for that long.
type Recorder struct {
Inv Inventory
// Models is each replica's model, by replica id.
Models func() map[string]string
Store Store
Leader func() bool
Every time.Duration
Log *slog.Logger
}
// Run records until ctx ends.
func (r *Recorder) Run(ctx context.Context) {
every := r.Every
if every <= 0 {
every = 5 * time.Minute
}
t := time.NewTicker(every)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case now := <-t.C:
if r.Leader != nil && !r.Leader() {
continue
}
held := Sample(r.Inv, r.Models(), every)
if err := r.Store.AddHeld(ctx, now, held); err != nil && r.Log != nil {
r.Log.Warn("could not record what the models held, for their prices", "err", err)
}
}
}
}
// Sample is what each model holds now, for d: a GPU for each card a shard
// was given, and the cores of a shard given cores and no GPU. A replica
// whose model is not known (headroom kept for another, one just reaped) is
// left out.
func Sample(inv Inventory, models map[string]string, d time.Duration) []store.Held {
by := map[string]*store.Held{}
var order []string
secs := int64(d.Seconds())
for _, n := range inv.Nodes() {
for _, a := range inv.Allocations(n.ID) {
m, ok := models[a.ReplicaID]
if !ok {
continue
}
h := by[m]
if h == nil {
h = &store.Held{Model: m}
by[m] = h
order = append(order, m)
}
if g := len(a.GPUIndices); g > 0 {
h.GPUSeconds += int64(g) * secs
} else {
h.CoreSeconds += int64(max(a.CPUCores, 1)) * secs
}
}
}
out := make([]store.Held, 0, len(order))
for _, m := range order {
out = append(out, *by[m])
}
return out
}
+40
View File
@@ -0,0 +1,40 @@
package held
import (
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
type inv map[string][]inventory.Allocation
func (i inv) Nodes() []inventory.Node {
var out []inventory.Node
for id := range i {
out = append(out, inventory.Node{ID: id})
}
return out
}
func (i inv) Allocations(id string) []inventory.Allocation { return i[id] }
func TestSampleCountsCardsAndCores(t *testing.T) {
in := inv{
"gpu1": {{ReplicaID: "big-1", GPUIndices: []int{0, 1}}, {ReplicaID: "headroom", GPUIndices: []int{2}}},
"gpu2": {{ReplicaID: "big-2", GPUIndices: []int{0, 1}}},
"cpu1": {{ReplicaID: "small-1", CPUCores: 4}, {ReplicaID: "tiny-1"}},
}
models := map[string]string{"big-1": "big", "big-2": "big", "small-1": "small", "tiny-1": "tiny"}
got := map[string]store.Held{}
for _, h := range Sample(in, models, 5*time.Minute) {
got[h.Model] = h
}
if got["big"].GPUSeconds != 4*300 || got["small"].CoreSeconds != 4*300 || got["tiny"].CoreSeconds != 300 {
t.Errorf("held %+v", got)
}
if len(got) != 3 {
t.Errorf("headroom counted: %+v", got)
}
}
+13 -1
View File
@@ -296,7 +296,7 @@ func (s *Server) Start(req Request) (Job, error) {
} }
req.sized = req.CtxPool > 0 || req.ContextLength > 0 req.sized = req.CtxPool > 0 || req.ContextLength > 0
if req.CtxPool <= 0 { if req.CtxPool <= 0 {
req.CtxPool = 16384 req.CtxPool = DefaultCtxPool
} }
if req.ContextLength <= 0 { if req.ContextLength <= 0 {
req.ContextLength = 4096 req.ContextLength = 4096
@@ -572,6 +572,14 @@ func (s *Server) run(j *Job) {
// On one card, the weights and the context's cache have to fit it: a // On one card, the weights and the context's cache have to fit it: a
// 70B with a 65536-token pool on a 24 GB card loaded its weights and then // 70B with a 65536-token pool on a 24 GB card loaded its weights and then
// failed to make its context, every time, as did an 8B with 131072. // failed to make its context, every time, as did an 8B with 131072.
// Asked for the CPU, and slow there: said, not refused — the request
// named it — so the job's log says why its replies take minutes.
if len(layout) > 0 && layout[0].CPU && layout[0].TokensPerSecond > 0 &&
layout[0].TokensPerSecond < modelplan.CPUSlowTPS {
tps := layout[0].TokensPerSecond
s.step(j, "check", Running, fmt.Sprintf("on the CPU about %.1f tokens a second for one reply: "+
"a reply of 500 tokens takes about %.0f seconds; a card would be many times faster", tps, 500/tps))
}
if len(layout) > 0 && layout[0].Class != "" && !layout[0].CPU { if len(layout) > 0 && layout[0].Class != "" && !layout[0].CPU {
fitted, why, err := fitPoolToCard(req, header, set.Bytes, layout[0].Class, max(layout[0].GPUsPerNode, 1)) fitted, why, err := fitPoolToCard(req, header, set.Bytes, layout[0].Class, max(layout[0].GPUsPerNode, 1))
if err != nil { if err != nil {
@@ -723,6 +731,10 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
return out return out
} }
// DefaultCtxPool is llama.cpp's pool when a request names none: a context
// asked for longer than it needs a pool of its own.
const DefaultCtxPool = 16384
// llamaArgs size llama.cpp's pool and its slots together. The pool is shared // llamaArgs size llama.cpp's pool and its slots together. The pool is shared
// by the slots, so slots of the context the catalog promises are as many as // by the slots, so slots of the context the catalog promises are as many as
// the pool holds: the engine's default of four turned a 32768 pool into // the pool holds: the engine's default of four turned a 32768 pool into
Loaded 100 of 224 files, more files were not shown because too many files have changed in this diff. Show more