Compare commits

...
23 Commits
Author SHA1 Message Date
Ilker ManapandClaude Fable 5.1 81f6ab3596 Go 1.27, no patch level pinned
CI / test (push) Failing after 10m54s
Release / build (push) Successful in 6m45s
Release / notes (push) Successful in 7s
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-19 12:38:07 +01:00
Ilker ManapandClaude Fable 5.1 bd9860a98d v0.16.4: changelog and README
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-19 12:33:42 +01:00
Ilker ManapandClaude Fable 5.1 5cc719fcde Go 1.27.4
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-19 12:33:42 +01:00
Ilker ManapandClaude Fable 5.1 c2fedeb326 An NVIDIA card with no driver is reported, not onboarded as a CPU node
CI / test (push) Failing after 9m12s
The agent found GPUs through nvidia-smi alone, so a machine whose driver
was never installed looked exactly like one with no card and joined
quietly as a CPU node. Only init looked at the PCI bus.

The host discovery now reads the bus too (vendor 0x10de, class 0x03) and
carries it to the control plane. The doctor fails the pre-install checks
on a card without a driver, with the install and the reboot named, and
tells a driver that is installed but not loaded (Secure Boot, nouveau)
apart from a missing one. A node already in shows the warning in ctl
nodes and the console. init uses the same check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-19 12:29:02 +01:00
Ilker ManapandClaude Opus 5.5 a53b2f4be9 Customise a model from the portal: a file, a model, a button
A page for anyone in a project, built for the one who knows least: an
Excel, CSV or JSONL file of questions and answers, previewed; a model; a
name; who may use it; and a button. The weights to train on are found
from where the model's came (a GGUF repository's base model), the card's
memory from their size, the node as the one with the most free. When it
is done the adapter is published by itself (a llama.cpp model's replicas
replaced to load it) and is a model of its own in the chat and the API.

The project's by default; its members share it wider (confirmed, and
never wider than the knowledge bases its pairs came from) or delete it.
One trains at a time per project. ctl train start needs only --model and
--data now, and --publish publishes when done.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 11:31:12 +01:00
Ilker ManapandClaude Opus 5.5 fcda4d3f85 Slurm's jobs in llmgw's accounting: projects as accounts, jobs priced
CI / test (push) Failing after 7m24s
With slurm.users llmgw, each project is a Slurm account, its organization's
and team's above it, with the project's members and its team's and
organization's on it; slurm.conf holds a job to its user's accounts. The
controller's agent keeps them with sacctmgr, only what it made, and reports
the jobs that ended in the last day; the leader records each once, its
GPU-hours (core-hours without cards) priced at auth.gpu_hour_cost into its
project's usage as model slurm/PARTITION.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 11:54:01 +01:00
Ilker ManapandClaude Opus 5.5 8547528fb7 Prices from what the cluster costs: ctl prices suggest
CI / test (push) Successful in 9m51s
auth.gpu_hour_cost (and core_hour_cost) is what an hour of a GPU (a core)
costs the operator. The leader keeps, by the hour, the GPU- and core-hours
each model's replicas hold (held_1h); llmgw ctl prices suggest spreads a
model's over the tokens it served, an output token counted as four input
ones and a cached prompt token as a tenth of one, and --apply writes the
prices into auth.pricing. Idle hours are in the cost, so a model held and
little used comes out dear; one held and billed nothing is shown with what
it held.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 06:54:55 +01:00
Ilker ManapandClaude Opus 5.5 b8d3c084c5 BACKLOG P: AD sign-in to Slurm user, tried end to end
CI / test (push) Failing after 7m31s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 23:00:28 +01:00
Ilker ManapandClaude Opus 5.5 57f0fd2ed0 Slurm's jobs in containers: pyxis and enroot, installed by the agents
CI / test (push) Failing after 6m53s
slurm.containers (ctl slurm containers on) has every Slurm machine's agent
install enroot from NVIDIA's release and build pyxis against its own
Slurm, both pinned and checked against their digests, and on GPU machines
nvidia-container-cli from NVIDIA's repository. enroot keeps each user's
containers under /run/enroot and /var/lib/enroot, needing no login
session, with the shared path in every container. The leader has Slurm
load pyxis only once every machine reports it: plugstack.conf's
"required" fails every job on a machine without the plugin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:31:20 +01:00
Ilker ManapandClaude Opus 5.5 c3a75dcf3f A node being lent says when a replica with nowhere to go is stopped
CI / test (push) Successful in 9m37s
Tried on the PoC: lending cuda01, the one GPU node, its model had nowhere
else to go and waited out the drain's five-minute grace, while ctl slurm
status said only "1 replica(s) still to move off". It now says the time
that replica is stopped at. Also recorded: Slurm on a GPU node, tried on
cuda01 (gres, ConstrainDevices, accounting of gres/gpu).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:45:57 +01:00
Ilker ManapandClaude Opus 5.5 4b4306bb6a Slurm users reach a node only through a job of theirs
CI / test (push) Successful in 9m41s
llmgw's Slurm users log in to the controller freely and to another Slurm
machine only while a job of theirs runs there: the agent installs
pam_slurm_adopt and puts it in sshd's account stack for the users it made
alone, so root and the machine's own users are never locked out. Their
session is put in the job's extern step (PrologFlags=Contain), and
pam_systemd, which would move it out, is skipped for them by taking the
session file into sshd's stack with a guard in front of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:30:49 +01:00
Ilker ManapandClaude Opus 5.5 a00c92648e Status says when the cluster's alerts go nowhere
CI / test (push) Failing after 6m57s
Setup asks for webhooks, but one left empty was never mentioned again: a
lost node or a judge down was told nobody. With no notify.webhooks in the
cluster's settings, ctl status and the console's Overview say so, and how
to set one. BACKLOG: the judge's refusal after a placement change, tried
again on idle VMs, did not happen; a cold judge's first answer is 2-3x
slower but inside its budget.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:49:37 +01:00
Ilker ManapandClaude Opus 5.5 2c32a58a06 Suspending a person says their keys are refused, which they are
CI / test (push) Successful in 9m41s
The message said their API keys still worked and should be revoked; the
store has refused a key whose owner is not active for a while. Tried on
the test cluster: refused 26 s after the suspend (auth.cache_ttl, 30 s),
answered again 6 s after reinstating.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:38:54 +01:00
Ilker ManapandClaude Opus 5.5 d8ce573bdb Slurm's shared files and users: one path everywhere, accounts as Linux users
CI / test (push) Failing after 7m1s
slurm.shared mounts the site's NFS export, has an llmgw node export the path
over NFS to the Slurm machines alone (2049 in the firewall), or looks for a
file system the site mounts. slurm.users "llmgw" makes each active account a
Linux user on every Slurm machine: login from its e-mail, a UID from 200000
kept in slurm_users and never reused, its home on the shared path, its SSH
keys from the portal's Profile (or ctl slurm users keys) read by sshd from
/etc/llmgw/ssh-keys. server init --slurm chooses /shared from the first
machine and llmgw's users; --slurm-shared and --slurm-users change that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:34:34 +01:00
Ilker ManapandClaude Fable 5.1 b9d20e7bba Licence: FSL-1.1 with an Apache 2.0 future licence
CI / test (push) Successful in 9m48s
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-12 19:55:24 +01:00
Ilker ManapandClaude Opus 5.5 0d78f9bb34 A machine given by address is reached through its ssh config Host
node add (and server add, mirror peer, patroni init) is usually given an
address, while the ssh config names the machine: `Host gpu01` with
`HostName 10.0.0.21` and the User, IdentityFile or ProxyJump it needs. That
block did not match the address, so the address had to be added to its
Host line by hand. Now the concrete Host names in the config and its
Includes are read, ssh -G says each one's HostName, and the first that is
the address (or resolves to it) is connected through.

Under sudo, the caller's name is now a fallback: a User in the config's
block for the machine comes first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:42:23 +01:00
Ilker ManapandClaude Opus 5.5 5ac6bef463 BACKLOG P: the fresh install tried, and what it left open
CI / test (push) Successful in 10m50s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:42:22 +01:00
Ilker ManapandClaude Opus 5.5 557a5a1ef3 Found on a fresh three-machine install: what the screens said wrong
CI / test (push) Successful in 11m5s
- server init printed every engine's line again on each look while the
  node installed (some 280 lines); each engine's line now once, and again
  only when it changes, as node add does.
- A package install that is no engine's (Slurm's, MariaDB's) read
  ": installing installing mariadb-server, ..."; it is "system
  packages: installing: mariadb-server, ..., with apt-get".
- The slow-judge advice says "ctl model resources JUDGE --cores 4",
  which was refused for want of --memory: cores alone now keep the
  memory as it is.
- server add said both control planes for three.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:38:18 +01:00
Ilker ManapandClaude Opus 5.5 21828fefce BACKLOG P: slurmdbd done; P6, Slurm's jobs in llmgw's accounting
CI / test (push) Failing after 7m52s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:21:40 +01:00
Ilker ManapandClaude Opus 5.5 748f777a56 Slurm's accounting: slurmdbd and a MariaDB on the controller
CI / test (push) Successful in 10m57s
Since 23.11 Slurm runs badly without accounting storage: the main
scheduler passes every job by as InvalidAccount. slurmdbd takes MySQL or
MariaDB only, so the controller's agent installs MariaDB beside it,
answering that machine alone, makes the database and its user (the
password made from the munge key, never on a command line), runs
slurmdbd and adds the cluster, all before slurmctld. A MariaDB that was
on the machine before keeps its settings. slurm.conf accounts the jobs
there, with what each used from its cgroup and the cards, and the
priority is multifactor again. Port 6819 joins Slurm's in the firewall.
ctl slurm status and the console say whether slurmdbd runs; the install
screen, server init --slurm and the guide say what is put where.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:59:08 +01:00
Ilker ManapandClaude Opus 5.5 02774769a7 Slurm: why priority/basic, said rightly
CI / test (push) Failing after 8m3s
priority/basic did not end InvalidAccount: 23.11 dropped
accounting_storage/none, and without slurmdbd the main scheduler passes
jobs by whatever the priority plugin. Basic stays, fair share needing
slurmdbd anyway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:21:24 +01:00
Ilker ManapandClaude Opus 5.5 bab0766ee1 Slurm: jobs start on submission, and a lent node stays open across restarts
CI / test (push) Failing after 8m35s
Without accounting storage, 23.11's default priority/multifactor has the
main scheduler pass every job by as InvalidAccount; backfill started it a
pass later. slurm.conf sets priority/basic.

Every node was written State=DRAIN, so each start of slurmctld on new
files drained a lent node again until its agent resumed it. A lent node
is written without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:18:22 +01:00
Ilker ManapandClaude Opus 5.5 5db920bf2c Slurm: a job that asks for no memory gets a share, not the whole node
CI / test (push) Successful in 10m34s
Tried on the PoC: five one-second jobs on an 8-CPU node started 10 s
apart. Each asked for no memory, so each was given the node's all, ran
alone, and the next waited for backfill's pass. Not accounting, as
thought. The partition's DefMemPerCPU is now the leanest node's memory
per CPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:11:24 +01:00
133 changed files with 9433 additions and 329 deletions

No files matched your search

+1
View File
@@ -11,3 +11,4 @@ coverage.out
/terraform-provider-llmgw/examples/terraform.tfstate*
# What a development server run from the checkout writes beside it.
/secrets/
linkedin/
+103 -4
View File
@@ -463,8 +463,9 @@ portal is not done — the page shows; the console and ctl change.
5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node
lost, budget spent, update available; signed JSON or Slack; retries,
hourly de-duplication), `ctl notify test`, Alertmanager receiver
templates. Left: `server init` and the console do not yet ask for a
receiver — the templates are filled in by hand. Events are claimed in
templates. `server init` and both setup screens ask for webhooks and test
each (d1e9186, 2026-10-14); with none set, `ctl status` and the console's
Overview say the alerts go nowhere (2026-11-12). Events are claimed in
the store before they are sent (2026-10-10), so two control planes send
each once.
**Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
@@ -2027,7 +2028,7 @@ lent, partition`); the leader tells each named node's agent (root) what to be, t
- Phases: P1 the `slurm` package (conf, key, parsers) and settings; P2 the agent side (proto, install, munge,
slurmd/slurmctld, status); P3 the leader side (conf from reports, lend/reclaim); P4 `ctl slurm`
(status, lend, reclaim) and a TUI screen; P5 `ctl train import` of a Slurm job's output into the mirror.
Later: a backup slurmctld on a second node, accounting (slurmdbd on the Patroni PG), users from LDAP/AD.
Later: a backup slurmctld on a second node, accounting (slurmdbd; it takes MySQL/MariaDB only, not the Patroni PG), users from LDAP/AD.
- Tried on the PoC 2026-11-11 (dev build v0.16.3-17-g849760c-slurm on 140, 150, node05, node06; Ubuntu
24.04, Slurm 23.11.4 from the archive): `ctl slurm setup --controller llmgw-node06 --nodes llmgw-node05`
installed munge, slurmctld and slurmd, node05 entered drained with its slurmd -C hardware (8 CPUs,
@@ -2037,7 +2038,105 @@ lent, partition`); the leader tells each named node's agent (root) what to be, t
from the daemon); the packaged unit is Type=notify (slurmd needs --systemd); slurmd's own "slurmstepd
infinity" was counted as a job; without accounting storage 23.11's main scheduler marks batch jobs
InvalidAccount and backfill starts them, one a pass (bf_interval=10: 6-10 s each). So many small jobs
want slurmdbd. Not tried yet: a GPU node (gres).
want slurmdbd. — Wrong cause, found 2026-11-11: a job asking no memory got the whole node's (ReqTRES
mem=all), so a node ran one job at a time and each next one waited for backfill's pass. Fixed: the
partition's DefMemPerCPU is the leanest node's share; slurmdbd is not needed for this. InvalidAccount is real too:
23.11 dropped accounting_storage/none, and without slurmdbd the main scheduler passes every job by;
backfill starts it within bf_interval (10 s). priority/basic did not change that (fair share needs
slurmdbd anyway). Starting on submission needs slurmdbd, on MySQL/MariaDB.
- slurmdbd, done 2026-11-12 (748f777): the controller's agent installs MariaDB (bound to 127.0.0.1; a
MariaDB that was there keeps its settings), makes `slurm_acct_db` and its user (password from the munge
key, through a 0600 file, never argv), runs slurmdbd (6819, in the firewall for the cluster) and adds
the cluster, before slurmctld. slurm.conf: AccountingStorageType=slurmdbd, jobacct_gather/cgroup,
AccountingStorageTRES=gres/gpu, priority/multifactor. Tried on the PoC: five jobs started on
submission, `sacct -a` has them with their TRES. Not tried: a fresh `server init --slurm` (needs VMs);
a Red Hat controller.
- Fresh install tried 2026-11-12 on VMs (3 control planes with Patroni, 2 CPU nodes, `server init
--slurm`): Slurm, slurmdbd and MariaDB came up with the install; two-node srun/sbatch on the lent nodes,
`sacct` has them; leader failover kept serving. Open from it: `--slurm` needs the first machine to be a
node (the controller is its node), so a site wanting control planes without nodes chooses the
controller later with `ctl slurm setup`; batch output lands on the job's first node until there is
shared storage; on shared CPUs the judge refuses the first request after a placement change (fail
closed, ~10 s), and its advice repeats `--cores 4` to a judge that has 4. Tried again on the idle
VMs: one judge restarted in place, then both at once, requests every second: 48 of 48 served; the
first after a cold start took 7.6–8.9 s end to end against ~3 s, within the 5 s judge budget. So the
refusal was the install's load on the CPUs; if it is seen again, warm a new judge with one judgment
before it takes requests.
- Shared files and users, built 2026-11-12: `slurm.shared` {path, nfs | server} — the site's export
mounted (a systemd mount unit, hard, nconnect=4), an llmgw node exporting the path itself
(/etc/exports.d, to the Slurm machines' addresses, root_squash; 2049 in the firewall), or the site's
own mount only looked for; `slurm.users: llmgw` — every active account a Linux user everywhere
(`slurm_users`: login from the e-mail, UID from 200000, never reused; homes made on the exporting
machine, else the controller; keys in /etc/llmgw/ssh-keys/%u through an sshd drop-in), SSH keys
from the portal's Profile or `ctl slurm users keys`. A user already on a machine with another UID
is left alone and said. The site has AD (not yet live): users from it, through sssd, are the next
step (`slurm.users` empty today leaves them to the site).
- Logging in to a node, done 2026-11-14: llmgw's users reach the controller freely, another Slurm
machine only while a job of theirs runs there (pam_slurm_adopt, installed by the agent; PrologFlags=
Contain). Only the users listed in the agent's users file are held: root and the machine's own users
log in as before, so a wrong line never locks out who would fix it. pam_systemd would move the session
out of the job, so sshd's stack takes the session file in place of its include, with pam_systemd
skipped for the listed users alone; where that cannot be done safely (nested includes, jumps across
it) they are held to having a job, not to its cores and cards, and the agent says so. Tried on the
VMs: ayse without a job refused ("no active jobs on this node"), with one in
job_N/step_extern (nproc 1), cut off when it ended; mehmet refused; admin's session unchanged.
Left: sites whose users come from AD (sssd) are not held yet; a group of theirs would be the list.
- A GPU node, tried 2026-11-14 on the PoC (cuda01, Quadro RTX 6000, Slurm 23.11.4): gres.conf
`Type=rtx6000 File=/dev/nvidia0`, slurm.conf `Gres=gpu:rtx6000:1`. `srun --gres=gpu:1`:
CUDA_VISIBLE_DEVICES=0, the one card; torch found it and ran 14.1 TFLOPS (fp32 matmul). Without
--gres: "No devices found", torch none (ConstrainDevices holds). `--gres=gpu:2`: refused. sacct has
`gres/gpu=1` in AllocTRES. Found: lending a node whose GPU model has nowhere else to go waits out
the drain grace (5 min) before the replica stops, and `ctl slurm status` says only "1 replica(s)
still to move off"; it should say it waits for a replacement that cannot be placed, and until when.
Done (c3a75dc): it says the time.
- Jobs in containers, done 2026-11-14: `slurm.containers` (`ctl slurm containers on`). Each agent
installs enroot 4.2.1 (NVIDIA's .deb/.rpm, sha256 pinned) and builds pyxis 0.24.0 (source tarball,
sha256 pinned) against its own Slurm (gcc, make, libslurm-dev); on a machine with nvidia-smi,
nvidia-container-cli from NVIDIA's repository, its key's digest pinned. enroot.conf keeps each
user's runtime, cache and data under /run/enroot and /var/lib/enroot (no login session needed), the
shared path bind-mounted in every container. The leader writes plugstack.conf ("required") only
once every machine reports both, since a machine without the plugin would fail every job. Tried on
the VMs: on in ~30 s; ayse's srun --container-image=ubuntu:24.04 on n1 ran as uid 200001 with
/shared inside and nproc 1; a two-node sbatch ran a rank on each. Seen once: a node's pull timed out
reaching auth.docker.io and the job failed (network, not pyxis). Not tried: GPUs in a container
(cuda01's agent is an older build; needs the peer protocol), Red Hat.
- AD end to end, tried 2026-11-14 (decided: one AD per install, the existing auth.ldap group mapping, no
tree from AD, no sssd unless a site asks): the VMs' auth.ldap against Samba DC VM 165 (ldaps by IP,
CA in the settings, ml-team -> team:Test/arge). ada signed in on the portal with her AD password: an
account, in Test/arge, Slurm user ada UID 200003; her key added in the portal's Profile, ssh to the
controller ~20 s later, sbatch ran on n1 as ada. Disabled in AD: suspended in 56 s, the Linux user
gone in 89 s, ssh refused, home kept. Enabled: back in 32 s, same UID, key kept. Taken out of
ml-team: out of the team in 11 s; put back: in again in 56 s.
- P5, replaced 2026-11-17 by the portal (decided 2026-11-15: users of every level, build for the one who
knows least first; any project member may start, no approval, one per project, the project's by default):
**Customise a model** (`/train`): an xlsx/CSV/JSONL of questions and answers, checked and previewed; base
weights from the model's origin (a GGUF repo's card base_model), memory from their size (2x + 1 GiB),
the node with the most free; published by itself when done (llama.cpp replicas replaced to load it);
shared me/project/team/org/everyone (wider than the project only confirmed, never wider than its
knowledge bases), deleted from the page. Left: pairs generated from a knowledge base on the page, a
base-vs-new comparison, the training's GPU time charged to the project, big jobs through Slurm, the
server's own error messages in Turkish.
- P6, done 2026-11-15 but for budgets: each project a Slurm account (org and team accounts above it,
ids keyed by kind: the first install's org, team and project share the id "default"), its members and its
team's and org's on it; AccountingStorageEnforce=associations; the controller's agent keeps them with
sacctmgr (only what it made; a user's default account moved before an association goes, the user taken out
when none is left) and reports the jobs that ended in the last day; the leader records each once
(slurm_jobs, keyed cluster/id) with a usage row, model slurm/PARTITION, priced at gpu_hour_cost or
core_hour_cost. Tried on the VMs: accounts made, ayse's job on her project's account ran and another
project's was refused, both jobs in deneme's usage at 345 and 0 micro-units. Left: budgets (a job's cost
is not counted against a budget, which binds keys; GrpTRESMins from a project's budget would hold its jobs).
- P6 as written 2026-11-12: The tenancy tree to Slurm:
each project an account (`sacctmgr add account`, parent its team and organization), each member a user
of it, kept by the leader from the store; `sbatch --account=PROJECT`. The records back: the leader reads
`sacct --json` for jobs ended since the last read (through the controller's agent) into a
`batch_usage` table (job, user, project, start, end, GPU-seconds by class, CPU-seconds, memory), priced
per GPU-hour and core-hour (auth.gpu_hour_cost, core_hour_cost: the same basis `ctl prices suggest`
prices tokens from, 2026-11-15) into cost_micros, summed with the requests' usage on the usage pages and in
budgets. Budgets enforced: a project's remaining budget as GPU-minutes in its account's GrpTRESMins, so
Slurm holds its new jobs once spent. Needs the same users on every node (open: LDAP/sssd or
llmgw-made). Also found: slurm.conf
wrote every node State=DRAIN, so each slurmctld start (on any change of the files) drained a lent node
until its agent resumed it, up to a minute; a lent node is now written without it. Not tried yet: a GPU node (gres).
- Failover while lent, tried 2026-11-11 (v0.16.3-23): the first try killed the job (NODE_FAIL). A new
leader had heard from no node yet and sent the controller a slurm.conf without them; slurmctld,
restarted on it, dropped node05 and its job. Fixed (141a1f4, 0774e90): slurm.conf's nodes and the
+72 -4
View File
@@ -20,16 +20,43 @@ is only released once its **exit criterion** is met, not when its code lands.
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node; v0.16.3: replicas per profile, a served model's settings on the mirror page, the CPU's speed before serving) |
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node; v0.16.3: replicas per profile, a served model's settings on the mirror page, the CPU's speed before serving; v0.16.4: Slurm beside llmgw, a model customised from the portal, a GPU node without a driver reported) |
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased
## v0.16.4 — 2026-11-19
Slurm beside llmgw for training and batch work — its users, shared files, containers, accounting priced in llmgw's
books — a model customised from the portal with a file and a button, an organization's admin, and a GPU node whose
driver was never installed told apart from a CPU node.
**Upgrading.**
- Four migrations (0052–0055: Slurm's users and jobs, held capacity, the portal's training jobs), applied by the
first control plane to start on this version; nothing to do by hand. `llmgw ctl upgrade v0.16.4`: the control
planes carry Slurm, the training page, the admin role and the prices; the agents carry Slurm's node part, the
containers, and the reading of the PCI bus for a card without a driver, without which such a node still joins
as a CPU node. Built with Go 1.27, the newest patch at build time.
### Added
- **Customise a model from the portal.** A new page, **Customise a model**
(`/train`), lets anyone in a project teach a model how to answer without a
terminal: an Excel, CSV or JSONL file of questions and answers (columns
headed Soru/Cevap or Question/Answer, else the first two), checked and
shown before anything trains; a model; a name; who may use it; and a
button. The weights to train on are found from where the model's came (a
GGUF repository's base model on Hugging Face), the card's memory from their
size, the node as the one with the most free; when it is done the adapter
is published by itself and appears as a model of its own in the chat and
the API, a llama.cpp model's replicas replaced one at a time to load it.
It is the project's by default; its members may share it with the person,
the team, the organization or everyone (wider than the project only once
they confirm the file holds nothing that group must not see, and never
wider than the knowledge bases its pairs came from), and delete it. One
trains at a time per project. `llmgw ctl train start` needs only `--model`
and `--data` now, and `--publish` publishes when done.
- **An organization's admin.** Besides member and manager, a membership of
an organization may now be `admin` (`llmgw users join … --org Acme --role
admin`, the role chosen on a membership on the people page, the console,
@@ -48,8 +75,41 @@ APIs change between minors without a deprecation period.
and gres.conf written from what each node reports. Every node stays llmgw's,
drained in Slurm, until `ctl slurm lend NODE`: its replicas move off, then
Slurm runs jobs there. `ctl slurm reclaim NODE` takes it back once its jobs
end. `ctl slurm status` shows each node. Slurm's ports (6817–6818, srun's
60001–63000) answer the cluster's networks only, as llmgw's own do.
end. `ctl slurm status` and the console's **Slurm** show each node. The
controller also keeps the jobs' records: slurmdbd, in a MariaDB on that
machine alone (Slurm takes MySQL or MariaDB only). Slurm's ports
(6817–6819, srun's 60001–63000) answer the cluster's networks only, as
llmgw's own do.
- **Slurm's shared files and users.** `slurm.shared` (`ctl slurm shared`) is
one path on every Slurm machine: the site's NFS export mounted, an llmgw
node exporting it over NFS (what `server init --slurm` chooses, `/shared`
from the first machine), or a file system the site mounts. `slurm.users:
llmgw` (`ctl slurm users on`) makes each active account a Linux user on
every Slurm machine, the same login and UID (from 200000, never reused)
everywhere, its home on the shared path; people add SSH keys in the
portal's Profile (`ctl slurm users keys` for an operator). They log in to
the controller; to another Slurm machine only while a job of theirs runs
there, their session put in the job (pam_slurm_adopt), so nobody uses a
node's cards outside Slurm, nor on a node llmgw serves from. Root and the
machines' own users are not held to it.
- **Slurm's jobs in llmgw's accounting.** With `slurm.users: llmgw` each
project is a Slurm account (organizations and teams above it), its members
and its team's and organization's on it, and a job must run on one of its
user's (`AccountingStorageEnforce=associations`). Each job that ends is
recorded once, its GPU-hours (or core-hours) priced at `auth.gpu_hour_cost`
into its project's usage as model `slurm/PARTITION`. Accounts a Slurm
administrator made are not touched.
- **Prices from what the cluster costs.** `auth.gpu_hour_cost` (and
`core_hour_cost`) is what an hour of a GPU (a core) costs; the leader keeps
the hours each model's replicas hold, and `llmgw ctl prices suggest` prices
each model from them over the tokens it served, an output token four input
ones and a cached one a tenth. `--apply` writes the prices.
- **Slurm's jobs in containers.** `llmgw ctl slurm containers on`
(`slurm.containers`) installs enroot and pyxis on every Slurm machine, at
pinned versions checked against their digests, and nvidia-container-cli on
GPU machines; Slurm loads pyxis once every machine has it, and then `srun
--container-image=...` runs a job in that image, as its user, with the
shared path inside.
### Changed
@@ -64,6 +124,14 @@ APIs change between minors without a deprecation period.
### Fixed
- **A GPU node with no driver joined quietly as a CPU node.** The agent found
GPUs through `nvidia-smi` alone, so a machine whose driver was never
installed looked exactly like one with no card. The PCI bus is read as
well: a card there with no driver fails the checks before the agent is
installed, with the install and the reboot named (and a driver installed
but not loaded — Secure Boot, nouveau — told apart from a missing one), and
a node already in shows the warning in `ctl nodes` and the console until
the driver is in and its agent restarted.
- **The serve forms start at the longest context that fits.** The mirror
page's form began every GGUF model at 4096 tokens, and the console's at
4096 or the plan's figure, whatever the model was trained for. Both now
+105
View File
@@ -0,0 +1,105 @@
# Functional Source License, Version 1.1, ALv2 Future License
## Abbreviation
FSL-1.1-ALv2
## Notice
Copyright 2026 Ilker Manap
## Terms and Conditions
### Licensor ("We")
The party offering the Software under these Terms and Conditions.
### The Software
The "Software" is each version of the software that we make available under
these Terms and Conditions, as indicated by our inclusion of these Terms and
Conditions with the Software.
### License Grant
Subject to your compliance with this License Grant and the Patents,
Redistribution and Trademark clauses below, we hereby grant you the right to
use, copy, modify, create derivative works, publicly perform, publicly display
and redistribute the Software for any Permitted Purpose identified below.
### Permitted Purpose
A Permitted Purpose is any purpose other than a Competing Use. A Competing Use
means making the Software available to others in a commercial product or
service that:
1. substitutes for the Software;
2. substitutes for any other product or service we offer using the Software
that exists as of the date we make the Software available; or
3. offers the same or substantially similar functionality as the Software.
Permitted Purposes specifically include using the Software:
1. for your internal use and access;
2. for non-commercial education;
3. for non-commercial research; and
4. in connection with professional services that you provide to a licensee
using the Software in accordance with these Terms and Conditions.
### Patents
To the extent your use for a Permitted Purpose would necessarily infringe our
patents, the license grant above includes a license under our patents. If you
make a claim against any party that the Software infringes or contributes to
the infringement of any patent, then your patent license to the Software ends
immediately.
### Redistribution
The Terms and Conditions apply to all copies, modifications and derivatives of
the Software.
If you redistribute any copies, modifications or derivatives of the Software,
you must include a copy of or a link to these Terms and Conditions and not
remove any copyright notices provided in or with the Software.
### Disclaimer
THE SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTIES OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING WITHOUT LIMITATION WARRANTIES OF FITNESS FOR A PARTICULAR
PURPOSE, MERCHANTABILITY, TITLE OR NON-INFRINGEMENT.
IN NO EVENT WILL WE HAVE ANY LIABILITY TO YOU ARISING OUT OF OR RELATED TO THE
SOFTWARE, INCLUDING INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES,
EVEN IF WE HAVE BEEN INFORMED OF THEIR POSSIBILITY IN ADVANCE.
### Trademarks
Except for displaying the License Details and identifying us as the origin of
the Software, you have no right under these Terms and Conditions to use our
trademarks, trade names, service marks or product names.
## Grant of Future License
We hereby irrevocably grant you an additional license to use the Software under
the Apache License, Version 2.0 that is effective on the second anniversary of
the date we make the Software available. On or after that date, you may use the
Software under the Apache License, Version 2.0, in which case the following
will apply:
Licensed under the Apache License, Version 2.0 (the "License"); you may not use
this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
+8 -1
View File
@@ -42,7 +42,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node; v0.16.3: replicas per profile, a served model's settings on the mirror page, the CPU's speed before serving) |
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node; v0.16.3: replicas per profile, a served model's settings on the mirror page, the CPU's speed before serving; v0.16.4: Slurm beside llmgw, a model customised from the portal, a GPU node without a driver reported) |
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
@@ -372,3 +372,10 @@ make rename NEW=git.example.com/team/llmgw
## Language
Code, comments, tests, commit messages and documentation are in English.
## Licence
Functional Source License 1.1 with an Apache 2.0 future licence
(FSL-1.1-ALv2); see [LICENSE](LICENSE). Each release may be used, read and
changed for anything except offering it as a competing product or service,
and becomes Apache 2.0 two years after it is published.
+560 -44
View File
@@ -1246,8 +1246,12 @@ type Host struct {
// given a few threads of a large machine reads at a few times this, not at
// the whole machine's rate. 0 unknown.
MemBandwidthCoreGbs float64 `protobuf:"fixed64,8,opt,name=mem_bandwidth_core_gbs,json=memBandwidthCoreGbs,proto3" json:"mem_bandwidth_core_gbs,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
// nvidia_device is whether an NVIDIA display controller sits on the PCI
// bus, driver or not. With nvidia_driver empty it is a card nothing can
// use yet, which must be said rather than counted as no card at all.
NvidiaDevice bool `protobuf:"varint,9,opt,name=nvidia_device,json=nvidiaDevice,proto3" json:"nvidia_device,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *Host) Reset() {
@@ -1336,6 +1340,13 @@ func (x *Host) GetMemBandwidthCoreGbs() float64 {
return 0
}
func (x *Host) GetNvidiaDevice() bool {
if x != nil {
return x.NvidiaDevice
}
return false
}
// Storage is one filesystem a node keeps weights on.
//
// Per filesystem rather than per configured directory: free space belongs to
@@ -5304,9 +5315,27 @@ type Slurm struct {
Files map[string]string `protobuf:"bytes,6,rep,name=files,proto3" json:"files,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
// lent says Slurm may run jobs here now: resume the node if llmgw drained
// it. Otherwise it is drained, its jobs left to finish.
Lent bool `protobuf:"varint,7,opt,name=lent,proto3" json:"lent,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
Lent bool `protobuf:"varint,7,opt,name=lent,proto3" json:"lent,omitempty"`
// shared is the file system every machine of the Slurm cluster sees at
// the same path; unset or with no path, none.
Shared *SlurmShared `protobuf:"bytes,8,opt,name=shared,proto3" json:"shared,omitempty"`
// manage_users has llmgw keep the Linux users below on this machine,
// and take away those it made that are no longer among them.
ManageUsers bool `protobuf:"varint,9,opt,name=manage_users,json=manageUsers,proto3" json:"manage_users,omitempty"`
Users []*SlurmUser `protobuf:"bytes,10,rep,name=users,proto3" json:"users,omitempty"`
// make_homes has this machine make the users' home directories: the one
// that exports the shared path, else the controller.
MakeHomes bool `protobuf:"varint,11,opt,name=make_homes,json=makeHomes,proto3" json:"make_homes,omitempty"`
// containers has the machine install pyxis and enroot, so jobs may run
// in containers (srun --container-image), and set them up.
Containers bool `protobuf:"varint,12,opt,name=containers,proto3" json:"containers,omitempty"`
// manage_accounts has the controller keep Slurm's accounts as accounts
// says: one for each organization, team and project, nested, the users
// on each project's. Only those llmgw made are changed or taken away.
ManageAccounts bool `protobuf:"varint,13,opt,name=manage_accounts,json=manageAccounts,proto3" json:"manage_accounts,omitempty"`
Accounts []*SlurmAccount `protobuf:"bytes,14,rep,name=accounts,proto3" json:"accounts,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *Slurm) Reset() {
@@ -5388,6 +5417,388 @@ func (x *Slurm) GetLent() bool {
return false
}
func (x *Slurm) GetShared() *SlurmShared {
if x != nil {
return x.Shared
}
return nil
}
func (x *Slurm) GetManageUsers() bool {
if x != nil {
return x.ManageUsers
}
return false
}
func (x *Slurm) GetUsers() []*SlurmUser {
if x != nil {
return x.Users
}
return nil
}
func (x *Slurm) GetMakeHomes() bool {
if x != nil {
return x.MakeHomes
}
return false
}
func (x *Slurm) GetContainers() bool {
if x != nil {
return x.Containers
}
return false
}
func (x *Slurm) GetManageAccounts() bool {
if x != nil {
return x.ManageAccounts
}
return false
}
func (x *Slurm) GetAccounts() []*SlurmAccount {
if x != nil {
return x.Accounts
}
return nil
}
// SlurmAccount is an account in Slurm's accounting, parents before their
// children.
type SlurmAccount struct {
state protoimpl.MessageState `protogen:"open.v1"`
Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
Parent string `protobuf:"bytes,2,opt,name=parent,proto3" json:"parent,omitempty"`
Users []string `protobuf:"bytes,3,rep,name=users,proto3" json:"users,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *SlurmAccount) Reset() {
*x = SlurmAccount{}
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *SlurmAccount) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*SlurmAccount) ProtoMessage() {}
func (x *SlurmAccount) ProtoReflect() protoreflect.Message {
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use SlurmAccount.ProtoReflect.Descriptor instead.
func (*SlurmAccount) Descriptor() ([]byte, []int) {
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{49}
}
func (x *SlurmAccount) GetName() string {
if x != nil {
return x.Name
}
return ""
}
func (x *SlurmAccount) GetParent() string {
if x != nil {
return x.Parent
}
return ""
}
func (x *SlurmAccount) GetUsers() []string {
if x != nil {
return x.Users
}
return nil
}
// SlurmJob is a job that ended, as sacct has it, for llmgw's accounting.
type SlurmJob struct {
state protoimpl.MessageState `protogen:"open.v1"`
Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"`
User string `protobuf:"bytes,2,opt,name=user,proto3" json:"user,omitempty"`
Account string `protobuf:"bytes,3,opt,name=account,proto3" json:"account,omitempty"`
Partition string `protobuf:"bytes,4,opt,name=partition,proto3" json:"partition,omitempty"`
State string `protobuf:"bytes,5,opt,name=state,proto3" json:"state,omitempty"`
StartUnix int64 `protobuf:"varint,6,opt,name=start_unix,json=startUnix,proto3" json:"start_unix,omitempty"`
EndUnix int64 `protobuf:"varint,7,opt,name=end_unix,json=endUnix,proto3" json:"end_unix,omitempty"`
ElapsedSeconds int64 `protobuf:"varint,8,opt,name=elapsed_seconds,json=elapsedSeconds,proto3" json:"elapsed_seconds,omitempty"`
Cpus int32 `protobuf:"varint,9,opt,name=cpus,proto3" json:"cpus,omitempty"`
Gpus int32 `protobuf:"varint,10,opt,name=gpus,proto3" json:"gpus,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *SlurmJob) Reset() {
*x = SlurmJob{}
mi := &file_llmgw_v1_agent_proto_msgTypes[50]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *SlurmJob) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*SlurmJob) ProtoMessage() {}
func (x *SlurmJob) ProtoReflect() protoreflect.Message {
mi := &file_llmgw_v1_agent_proto_msgTypes[50]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use SlurmJob.ProtoReflect.Descriptor instead.
func (*SlurmJob) Descriptor() ([]byte, []int) {
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{50}
}
func (x *SlurmJob) GetId() string {
if x != nil {
return x.Id
}
return ""
}
func (x *SlurmJob) GetUser() string {
if x != nil {
return x.User
}
return ""
}
func (x *SlurmJob) GetAccount() string {
if x != nil {
return x.Account
}
return ""
}
func (x *SlurmJob) GetPartition() string {
if x != nil {
return x.Partition
}
return ""
}
func (x *SlurmJob) GetState() string {
if x != nil {
return x.State
}
return ""
}
func (x *SlurmJob) GetStartUnix() int64 {
if x != nil {
return x.StartUnix
}
return 0
}
func (x *SlurmJob) GetEndUnix() int64 {
if x != nil {
return x.EndUnix
}
return 0
}
func (x *SlurmJob) GetElapsedSeconds() int64 {
if x != nil {
return x.ElapsedSeconds
}
return 0
}
func (x *SlurmJob) GetCpus() int32 {
if x != nil {
return x.Cpus
}
return 0
}
func (x *SlurmJob) GetGpus() int32 {
if x != nil {
return x.Gpus
}
return 0
}
// SlurmShared is where the Slurm cluster's shared files come from.
type SlurmShared struct {
state protoimpl.MessageState `protogen:"open.v1"`
// path is where every machine sees them, as /shared.
Path string `protobuf:"bytes,1,opt,name=path,proto3" json:"path,omitempty"`
// nfs is an export, host:/path, to mount at path; empty when this
// machine exports path itself (serve) or the site mounts it.
Nfs string `protobuf:"bytes,2,opt,name=nfs,proto3" json:"nfs,omitempty"`
// serve has this machine export path over NFS to clients.
Serve bool `protobuf:"varint,3,opt,name=serve,proto3" json:"serve,omitempty"`
// clients are the addresses the export answers.
Clients []string `protobuf:"bytes,4,rep,name=clients,proto3" json:"clients,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *SlurmShared) Reset() {
*x = SlurmShared{}
mi := &file_llmgw_v1_agent_proto_msgTypes[51]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *SlurmShared) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*SlurmShared) ProtoMessage() {}
func (x *SlurmShared) ProtoReflect() protoreflect.Message {
mi := &file_llmgw_v1_agent_proto_msgTypes[51]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use SlurmShared.ProtoReflect.Descriptor instead.
func (*SlurmShared) Descriptor() ([]byte, []int) {
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{51}
}
func (x *SlurmShared) GetPath() string {
if x != nil {
return x.Path
}
return ""
}
func (x *SlurmShared) GetNfs() string {
if x != nil {
return x.Nfs
}
return ""
}
func (x *SlurmShared) GetServe() bool {
if x != nil {
return x.Serve
}
return false
}
func (x *SlurmShared) GetClients() []string {
if x != nil {
return x.Clients
}
return nil
}
// SlurmUser is a Linux user that runs jobs, the same on every machine.
type SlurmUser struct {
state protoimpl.MessageState `protogen:"open.v1"`
Login string `protobuf:"bytes,1,opt,name=login,proto3" json:"login,omitempty"`
// uid is also the user's own group's gid.
Uid uint32 `protobuf:"varint,2,opt,name=uid,proto3" json:"uid,omitempty"`
// name is the account's, for the GECOS field.
Name string `protobuf:"bytes,3,opt,name=name,proto3" json:"name,omitempty"`
Home string `protobuf:"bytes,4,opt,name=home,proto3" json:"home,omitempty"`
// ssh_keys are OpenSSH public keys, one a line of authorized_keys.
SshKeys []string `protobuf:"bytes,5,rep,name=ssh_keys,json=sshKeys,proto3" json:"ssh_keys,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *SlurmUser) Reset() {
*x = SlurmUser{}
mi := &file_llmgw_v1_agent_proto_msgTypes[52]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *SlurmUser) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*SlurmUser) ProtoMessage() {}
func (x *SlurmUser) ProtoReflect() protoreflect.Message {
mi := &file_llmgw_v1_agent_proto_msgTypes[52]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use SlurmUser.ProtoReflect.Descriptor instead.
func (*SlurmUser) Descriptor() ([]byte, []int) {
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{52}
}
func (x *SlurmUser) GetLogin() string {
if x != nil {
return x.Login
}
return ""
}
func (x *SlurmUser) GetUid() uint32 {
if x != nil {
return x.Uid
}
return 0
}
func (x *SlurmUser) GetName() string {
if x != nil {
return x.Name
}
return ""
}
func (x *SlurmUser) GetHome() string {
if x != nil {
return x.Home
}
return ""
}
func (x *SlurmUser) GetSshKeys() []string {
if x != nil {
return x.SshKeys
}
return nil
}
// SlurmStatus is what this node's Slurm daemons are doing, sent when it
// changes and every few minutes while the node has a part in Slurm.
type SlurmStatus struct {
@@ -5410,14 +5821,28 @@ type SlurmStatus struct {
Error string `protobuf:"bytes,8,opt,name=error,proto3" json:"error,omitempty"`
// files_sha256 is of the controller's files as written, so the leader
// knows they arrived.
FilesSha256 string `protobuf:"bytes,9,opt,name=files_sha256,json=filesSha256,proto3" json:"files_sha256,omitempty"`
FilesSha256 string `protobuf:"bytes,9,opt,name=files_sha256,json=filesSha256,proto3" json:"files_sha256,omitempty"`
// slurmdbd_running is the accounting daemon on the controller, which keeps
// the jobs' records in a MariaDB beside it.
SlurmdbdRunning bool `protobuf:"varint,10,opt,name=slurmdbd_running,json=slurmdbdRunning,proto3" json:"slurmdbd_running,omitempty"`
// shared is the shared path as this machine has it: "mounted",
// "exported", "missing", or empty when there is none.
Shared string `protobuf:"bytes,11,opt,name=shared,proto3" json:"shared,omitempty"`
// users is how many of llmgw's users this machine has.
Users int32 `protobuf:"varint,12,opt,name=users,proto3" json:"users,omitempty"`
// containers is what this machine has for jobs in containers, as
// "pyxis 0.24.0, enroot 4.2.1"; empty while it has not got both.
Containers string `protobuf:"bytes,13,opt,name=containers,proto3" json:"containers,omitempty"`
// ended are, on the controller, the jobs that ended in the last day; the
// leader records each once.
Ended []*SlurmJob `protobuf:"bytes,14,rep,name=ended,proto3" json:"ended,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *SlurmStatus) Reset() {
*x = SlurmStatus{}
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
mi := &file_llmgw_v1_agent_proto_msgTypes[53]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -5429,7 +5854,7 @@ func (x *SlurmStatus) String() string {
func (*SlurmStatus) ProtoMessage() {}
func (x *SlurmStatus) ProtoReflect() protoreflect.Message {
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
mi := &file_llmgw_v1_agent_proto_msgTypes[53]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -5442,7 +5867,7 @@ func (x *SlurmStatus) ProtoReflect() protoreflect.Message {
// Deprecated: Use SlurmStatus.ProtoReflect.Descriptor instead.
func (*SlurmStatus) Descriptor() ([]byte, []int) {
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{49}
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{53}
}
func (x *SlurmStatus) GetVersion() string {
@@ -5508,6 +5933,41 @@ func (x *SlurmStatus) GetFilesSha256() string {
return ""
}
func (x *SlurmStatus) GetSlurmdbdRunning() bool {
if x != nil {
return x.SlurmdbdRunning
}
return false
}
func (x *SlurmStatus) GetShared() string {
if x != nil {
return x.Shared
}
return ""
}
func (x *SlurmStatus) GetUsers() int32 {
if x != nil {
return x.Users
}
return 0
}
func (x *SlurmStatus) GetContainers() string {
if x != nil {
return x.Containers
}
return ""
}
func (x *SlurmStatus) GetEnded() []*SlurmJob {
if x != nil {
return x.Ended
}
return nil
}
var File_llmgw_v1_agent_proto protoreflect.FileDescriptor
const file_llmgw_v1_agent_proto_rawDesc = "" +
@@ -5603,7 +6063,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\x1aA\n" +
"\x13EngineVersionsEntry\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\x92\x02\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\xb7\x02\n" +
"\x04Host\x12\x13\n" +
"\x05os_id\x18\x01 \x01(\tR\x04osId\x12\x1d\n" +
"\n" +
@@ -5614,7 +6074,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\tcpu_flags\x18\x05 \x03(\tR\bcpuFlags\x12#\n" +
"\rnvidia_driver\x18\x06 \x01(\tR\fnvidiaDriver\x12*\n" +
"\x11mem_bandwidth_gbs\x18\a \x01(\x01R\x0fmemBandwidthGbs\x123\n" +
"\x16mem_bandwidth_core_gbs\x18\b \x01(\x01R\x13memBandwidthCoreGbs\"u\n" +
"\x16mem_bandwidth_core_gbs\x18\b \x01(\x01R\x13memBandwidthCoreGbs\x12#\n" +
"\rnvidia_device\x18\t \x01(\bR\fnvidiaDevice\"u\n" +
"\aStorage\x12\x12\n" +
"\x04path\x18\x01 \x01(\tR\x04path\x12\x1d\n" +
"\n" +
@@ -6013,7 +6474,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\x04kind\x18\x02 \x01(\tR\x04kind\x12\x10\n" +
"\x03seq\x18\x03 \x01(\x05R\x03seq\x12\x12\n" +
"\x04data\x18\x04 \x01(\fR\x04data\x12\x12\n" +
"\x04last\x18\x05 \x01(\bR\x04last\"\x8d\x02\n" +
"\x04last\x18\x05 \x01(\bR\x04last\"\xa6\x04\n" +
"\x05Slurm\x12\x1e\n" +
"\n" +
"controller\x18\x01 \x01(\bR\n" +
@@ -6024,11 +6485,50 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"confServer\x12\x1b\n" +
"\tmunge_key\x18\x05 \x01(\fR\bmungeKey\x120\n" +
"\x05files\x18\x06 \x03(\v2\x1a.llmgw.v1.Slurm.FilesEntryR\x05files\x12\x12\n" +
"\x04lent\x18\a \x01(\bR\x04lent\x1a8\n" +
"\x04lent\x18\a \x01(\bR\x04lent\x12-\n" +
"\x06shared\x18\b \x01(\v2\x15.llmgw.v1.SlurmSharedR\x06shared\x12!\n" +
"\fmanage_users\x18\t \x01(\bR\vmanageUsers\x12)\n" +
"\x05users\x18\n" +
" \x03(\v2\x13.llmgw.v1.SlurmUserR\x05users\x12\x1d\n" +
"\n" +
"make_homes\x18\v \x01(\bR\tmakeHomes\x12\x1e\n" +
"\n" +
"containers\x18\f \x01(\bR\n" +
"containers\x12'\n" +
"\x0fmanage_accounts\x18\r \x01(\bR\x0emanageAccounts\x122\n" +
"\baccounts\x18\x0e \x03(\v2\x16.llmgw.v1.SlurmAccountR\baccounts\x1a8\n" +
"\n" +
"FilesEntry\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\x92\x02\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"P\n" +
"\fSlurmAccount\x12\x12\n" +
"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
"\x06parent\x18\x02 \x01(\tR\x06parent\x12\x14\n" +
"\x05users\x18\x03 \x03(\tR\x05users\"\x87\x02\n" +
"\bSlurmJob\x12\x0e\n" +
"\x02id\x18\x01 \x01(\tR\x02id\x12\x12\n" +
"\x04user\x18\x02 \x01(\tR\x04user\x12\x18\n" +
"\aaccount\x18\x03 \x01(\tR\aaccount\x12\x1c\n" +
"\tpartition\x18\x04 \x01(\tR\tpartition\x12\x14\n" +
"\x05state\x18\x05 \x01(\tR\x05state\x12\x1d\n" +
"\n" +
"start_unix\x18\x06 \x01(\x03R\tstartUnix\x12\x19\n" +
"\bend_unix\x18\a \x01(\x03R\aendUnix\x12'\n" +
"\x0felapsed_seconds\x18\b \x01(\x03R\x0eelapsedSeconds\x12\x12\n" +
"\x04cpus\x18\t \x01(\x05R\x04cpus\x12\x12\n" +
"\x04gpus\x18\n" +
" \x01(\x05R\x04gpus\"c\n" +
"\vSlurmShared\x12\x12\n" +
"\x04path\x18\x01 \x01(\tR\x04path\x12\x10\n" +
"\x03nfs\x18\x02 \x01(\tR\x03nfs\x12\x14\n" +
"\x05serve\x18\x03 \x01(\bR\x05serve\x12\x18\n" +
"\aclients\x18\x04 \x03(\tR\aclients\"v\n" +
"\tSlurmUser\x12\x14\n" +
"\x05login\x18\x01 \x01(\tR\x05login\x12\x10\n" +
"\x03uid\x18\x02 \x01(\rR\x03uid\x12\x12\n" +
"\x04name\x18\x03 \x01(\tR\x04name\x12\x12\n" +
"\x04home\x18\x04 \x01(\tR\x04home\x12\x19\n" +
"\bssh_keys\x18\x05 \x03(\tR\asshKeys\"\xb5\x03\n" +
"\vSlurmStatus\x12\x18\n" +
"\aversion\x18\x01 \x01(\tR\aversion\x12\x1a\n" +
"\bhardware\x18\x02 \x01(\tR\bhardware\x12%\n" +
@@ -6038,7 +6538,15 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\x06reason\x18\x06 \x01(\tR\x06reason\x12\x12\n" +
"\x04jobs\x18\a \x01(\x05R\x04jobs\x12\x14\n" +
"\x05error\x18\b \x01(\tR\x05error\x12!\n" +
"\ffiles_sha256\x18\t \x01(\tR\vfilesSha2562O\n" +
"\ffiles_sha256\x18\t \x01(\tR\vfilesSha256\x12)\n" +
"\x10slurmdbd_running\x18\n" +
" \x01(\bR\x0fslurmdbdRunning\x12\x16\n" +
"\x06shared\x18\v \x01(\tR\x06shared\x12\x14\n" +
"\x05users\x18\f \x01(\x05R\x05users\x12\x1e\n" +
"\n" +
"containers\x18\r \x01(\tR\n" +
"containers\x12(\n" +
"\x05ended\x18\x0e \x03(\v2\x12.llmgw.v1.SlurmJobR\x05ended2O\n" +
"\fAgentService\x12?\n" +
"\aConnect\x12\x16.llmgw.v1.AgentMessage\x1a\x18.llmgw.v1.ControlMessage(\x010\x01B\x93\x01\n" +
"\fcom.llmgw.v1B\n" +
@@ -6057,7 +6565,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
}
var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 59)
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 63)
var file_llmgw_v1_agent_proto_goTypes = []any{
(ShardStatus_State)(0), // 0: llmgw.v1.ShardStatus.State
(*AgentMessage)(nil), // 1: llmgw.v1.AgentMessage
@@ -6109,16 +6617,20 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
(*TrainStatus)(nil), // 47: llmgw.v1.TrainStatus
(*TrainArtifact)(nil), // 48: llmgw.v1.TrainArtifact
(*Slurm)(nil), // 49: llmgw.v1.Slurm
(*SlurmStatus)(nil), // 50: llmgw.v1.SlurmStatus
nil, // 51: llmgw.v1.ControlMessage.TraceEntry
nil, // 52: llmgw.v1.Register.LabelsEntry
nil, // 53: llmgw.v1.Register.EngineVersionsEntry
nil, // 54: llmgw.v1.GPU.NicPathsEntry
nil, // 55: llmgw.v1.IBPort.CountersEntry
nil, // 56: llmgw.v1.Heartbeat.EngineVersionsEntry
nil, // 57: llmgw.v1.EngineWant.OfflineSha256Entry
nil, // 58: llmgw.v1.EngineProfile.EnvEntry
nil, // 59: llmgw.v1.Slurm.FilesEntry
(*SlurmAccount)(nil), // 50: llmgw.v1.SlurmAccount
(*SlurmJob)(nil), // 51: llmgw.v1.SlurmJob
(*SlurmShared)(nil), // 52: llmgw.v1.SlurmShared
(*SlurmUser)(nil), // 53: llmgw.v1.SlurmUser
(*SlurmStatus)(nil), // 54: llmgw.v1.SlurmStatus
nil, // 55: llmgw.v1.ControlMessage.TraceEntry
nil, // 56: llmgw.v1.Register.LabelsEntry
nil, // 57: llmgw.v1.Register.EngineVersionsEntry
nil, // 58: llmgw.v1.GPU.NicPathsEntry
nil, // 59: llmgw.v1.IBPort.CountersEntry
nil, // 60: llmgw.v1.Heartbeat.EngineVersionsEntry
nil, // 61: llmgw.v1.EngineWant.OfflineSha256Entry
nil, // 62: llmgw.v1.EngineProfile.EnvEntry
nil, // 63: llmgw.v1.Slurm.FilesEntry
}
var file_llmgw_v1_agent_proto_depIdxs = []int32{
8, // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
@@ -6131,7 +6643,7 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
2, // 7: llmgw.v1.AgentMessage.kv_events:type_name -> llmgw.v1.KVEvents
47, // 8: llmgw.v1.AgentMessage.train_status:type_name -> llmgw.v1.TrainStatus
48, // 9: llmgw.v1.AgentMessage.train_artifact:type_name -> llmgw.v1.TrainArtifact
50, // 10: llmgw.v1.AgentMessage.slurm_status:type_name -> llmgw.v1.SlurmStatus
54, // 10: llmgw.v1.AgentMessage.slurm_status:type_name -> llmgw.v1.SlurmStatus
30, // 11: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
33, // 12: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
34, // 13: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
@@ -6149,11 +6661,11 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
45, // 25: llmgw.v1.ControlMessage.train_start:type_name -> llmgw.v1.TrainStart
46, // 26: llmgw.v1.ControlMessage.train_cancel:type_name -> llmgw.v1.TrainCancel
49, // 27: llmgw.v1.ControlMessage.slurm:type_name -> llmgw.v1.Slurm
51, // 28: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
52, // 29: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
55, // 28: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
56, // 29: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
12, // 30: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
14, // 31: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
53, // 32: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
57, // 32: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
11, // 33: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
15, // 34: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
10, // 35: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
@@ -6161,15 +6673,15 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
21, // 37: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
0, // 38: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
27, // 39: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
54, // 40: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
58, // 40: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
13, // 41: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
18, // 42: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
17, // 43: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
55, // 44: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
59, // 44: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
20, // 45: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
10, // 46: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
15, // 47: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
56, // 48: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
60, // 48: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
26, // 49: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
16, // 50: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
19, // 51: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
@@ -6177,21 +6689,25 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
22, // 53: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
21, // 54: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
25, // 55: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
57, // 56: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
61, // 56: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
0, // 57: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
29, // 58: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
27, // 59: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
31, // 60: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
58, // 61: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
62, // 61: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
44, // 62: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
59, // 63: llmgw.v1.Slurm.files:type_name -> llmgw.v1.Slurm.FilesEntry
1, // 64: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
3, // 65: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
65, // [65:66] is the sub-list for method output_type
64, // [64:65] is the sub-list for method input_type
64, // [64:64] is the sub-list for extension type_name
64, // [64:64] is the sub-list for extension extendee
0, // [0:64] is the sub-list for field type_name
63, // 63: llmgw.v1.Slurm.files:type_name -> llmgw.v1.Slurm.FilesEntry
52, // 64: llmgw.v1.Slurm.shared:type_name -> llmgw.v1.SlurmShared
53, // 65: llmgw.v1.Slurm.users:type_name -> llmgw.v1.SlurmUser
50, // 66: llmgw.v1.Slurm.accounts:type_name -> llmgw.v1.SlurmAccount
51, // 67: llmgw.v1.SlurmStatus.ended:type_name -> llmgw.v1.SlurmJob
1, // 68: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
3, // 69: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
69, // [69:70] is the sub-list for method output_type
68, // [68:69] is the sub-list for method input_type
68, // [68:68] is the sub-list for extension type_name
68, // [68:68] is the sub-list for extension extendee
0, // [0:68] is the sub-list for field type_name
}
func init() { file_llmgw_v1_agent_proto_init() }
@@ -6237,7 +6753,7 @@ func file_llmgw_v1_agent_proto_init() {
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
NumEnums: 1,
NumMessages: 59,
NumMessages: 63,
NumExtensions: 0,
NumServices: 1,
},
+83
View File
@@ -247,6 +247,10 @@ message Host {
// given a few threads of a large machine reads at a few times this, not at
// the whole machine's rate. 0 unknown.
double mem_bandwidth_core_gbs = 8;
// nvidia_device is whether an NVIDIA display controller sits on the PCI
// bus, driver or not. With nvidia_driver empty it is a card nothing can
// use yet, which must be said rather than counted as no card at all.
bool nvidia_device = 9;
}
// Storage is one filesystem a node keeps weights on.
@@ -1010,6 +1014,71 @@ message Slurm {
// lent says Slurm may run jobs here now: resume the node if llmgw drained
// it. Otherwise it is drained, its jobs left to finish.
bool lent = 7;
// shared is the file system every machine of the Slurm cluster sees at
// the same path; unset or with no path, none.
SlurmShared shared = 8;
// manage_users has llmgw keep the Linux users below on this machine,
// and take away those it made that are no longer among them.
bool manage_users = 9;
repeated SlurmUser users = 10;
// make_homes has this machine make the users' home directories: the one
// that exports the shared path, else the controller.
bool make_homes = 11;
// containers has the machine install pyxis and enroot, so jobs may run
// in containers (srun --container-image), and set them up.
bool containers = 12;
// manage_accounts has the controller keep Slurm's accounts as accounts
// says: one for each organization, team and project, nested, the users
// on each project's. Only those llmgw made are changed or taken away.
bool manage_accounts = 13;
repeated SlurmAccount accounts = 14;
}
// SlurmAccount is an account in Slurm's accounting, parents before their
// children.
message SlurmAccount {
string name = 1;
string parent = 2;
repeated string users = 3;
}
// SlurmJob is a job that ended, as sacct has it, for llmgw's accounting.
message SlurmJob {
string id = 1;
string user = 2;
string account = 3;
string partition = 4;
string state = 5;
int64 start_unix = 6;
int64 end_unix = 7;
int64 elapsed_seconds = 8;
int32 cpus = 9;
int32 gpus = 10;
}
// SlurmShared is where the Slurm cluster's shared files come from.
message SlurmShared {
// path is where every machine sees them, as /shared.
string path = 1;
// nfs is an export, host:/path, to mount at path; empty when this
// machine exports path itself (serve) or the site mounts it.
string nfs = 2;
// serve has this machine export path over NFS to clients.
bool serve = 3;
// clients are the addresses the export answers.
repeated string clients = 4;
}
// SlurmUser is a Linux user that runs jobs, the same on every machine.
message SlurmUser {
string login = 1;
// uid is also the user's own group's gid.
uint32 uid = 2;
// name is the account's, for the GECOS field.
string name = 3;
string home = 4;
// ssh_keys are OpenSSH public keys, one a line of authorized_keys.
repeated string ssh_keys = 5;
}
// SlurmStatus is what this node's Slurm daemons are doing, sent when it
@@ -1034,4 +1103,18 @@ message SlurmStatus {
// files_sha256 is of the controller's files as written, so the leader
// knows they arrived.
string files_sha256 = 9;
// slurmdbd_running is the accounting daemon on the controller, which keeps
// the jobs' records in a MariaDB beside it.
bool slurmdbd_running = 10;
// shared is the shared path as this machine has it: "mounted",
// "exported", "missing", or empty when there is none.
string shared = 11;
// users is how many of llmgw's users this machine has.
int32 users = 12;
// containers is what this machine has for jobs in containers, as
// "pyxis 0.24.0, enroot 4.2.1"; empty while it has not got both.
string containers = 13;
// ended are, on the controller, the jobs that ended in the last day; the
// leader records each once.
repeated SlurmJob ended = 14;
}
+1 -1
View File
@@ -16,7 +16,7 @@
# machine, manages its GPUs and supervises processes there as root, which is
# what a container is built to prevent: docs/kubernetes.md.
FROM golang:1.26 AS build
FROM golang:1.27 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
+303
View File
@@ -95,6 +95,9 @@
{
"name": "presets"
},
{
"name": "prices"
},
{
"name": "projects"
},
@@ -4396,6 +4399,34 @@
}
}
},
"/api/v1/prices/suggest": {
"get": {
"operationId": "getPricesSuggest",
"summary": "Each model's price, from what it held of the cluster",
"description": "days (30 if not given) is the window. A model's cost is the GPU- and core-hours its replicas held, idle ones too, at auth.gpu_hour_cost and auth.core_hour_cost; spread over its tokens, an output token four input ones and a cached prompt token a tenth of one, it is price, in micro-units a thousand tokens as auth.pricing takes them. current is the price in force. 409 with no cost set.",
"tags": [
"prices"
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/PriceSuggestions"
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/projects": {
"post": {
"operationId": "postProjects",
@@ -5109,6 +5140,89 @@
}
}
},
"/api/v1/slurm/users": {
"get": {
"operationId": "getSlurmUsers",
"summary": "The Linux users made from llmgw's accounts for the Slurm cluster",
"description": "Every active account, with its login, UID and SSH keys; an account is given its login and UID the first time it is listed. The machines have them when slurm.users is \"llmgw\".",
"tags": [
"slurm"
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/slurmsvc.User"
}
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/slurm/users/{user}/ssh-keys": {
"put": {
"operationId": "putSlurmUsersUserSshKeys",
"summary": "Replace an account's SSH public keys",
"description": "{user} is an ID or e-mail. keys is OpenSSH public keys, one a line, without options; empty takes them all away. The machines have them within a minute.",
"tags": [
"slurm"
],
"parameters": [
{
"name": "user",
"in": "path",
"required": true,
"description": "The person's ID or email.",
"schema": {
"type": "string"
}
},
{
"$ref": "#/components/parameters/Actor"
}
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/SlurmKeysRequest"
}
}
}
},
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/slurmsvc.User"
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/summary": {
"get": {
"operationId": "getSummary",
@@ -8763,6 +8877,73 @@
}
}
},
"PriceSuggestion": {
"type": "object",
"description": "PriceSuggestion is one model's.",
"properties": {
"model": {
"type": "string"
},
"gpu_hours": {
"type": "number"
},
"core_hours": {
"type": "number"
},
"cost": {
"type": "number",
"description": "Cost is what it held cost, in budget units."
},
"prompt_tokens": {
"type": "integer",
"format": "int64"
},
"completion_tokens": {
"type": "integer",
"format": "int64"
},
"price": {
"$ref": "#/components/schemas/accounting.Price",
"description": "Price is the suggestion; zero with a Note saying why there is none."
},
"note": {
"type": "string"
},
"current": {
"$ref": "#/components/schemas/accounting.Price"
}
}
},
"PriceSuggestions": {
"type": "object",
"description": "PriceSuggestions is a model's price from its cost, for each model.",
"properties": {
"days": {
"type": "integer",
"description": "Days is the window; RecordedFrom the first hour held is known for, later than the window's start on a cluster that began keeping it since, which makes the window shorter than asked."
},
"since": {
"type": "string",
"format": "date-time"
},
"recorded_from": {
"type": "string",
"format": "date-time"
},
"gpu_hour_cost": {
"type": "number"
},
"core_hour_cost": {
"type": "number"
},
"models": {
"type": "array",
"items": {
"$ref": "#/components/schemas/PriceSuggestion"
}
}
}
},
"ProfileReplicasRequest": {
"type": "object",
"description": "ProfileReplicasRequest counts a model's replicas per profile: profile name to replicas, and CPU, for a llama.cpp model with no profile on CPUs, the one to add.",
@@ -9401,6 +9582,15 @@
}
}
},
"SlurmKeysRequest": {
"type": "object",
"description": "SlurmKeysRequest is a person's SSH public keys, one a line, as authorized_keys has them; empty takes them all away.",
"properties": {
"keys": {
"type": "string"
}
}
},
"StoreMember": {
"type": "object",
"description": "StoreMember is one member of a replicated store, as its Patroni reports it.",
@@ -9503,6 +9693,10 @@
"guard_off": {
"type": "boolean",
"description": "GuardOff says the guardrails are turned off (guard.enabled: false): no request is checked, whatever the policies say."
},
"no_alerts": {
"type": "boolean",
"description": "NoAlerts says the cluster's settings name no webhook: a model parked, a node lost or a judge down is told nobody."
}
}
},
@@ -9911,6 +10105,35 @@
}
}
},
"accounting.Price": {
"type": "object",
"description": "Price is what one model costs, in micro-units per thousand tokens.\n\nMicro-units rather than a float because spend is accumulated by repeated addition, and repeated addition of floats drifts. What a unit means is the operator's: self-hosted models have no per-token price, so it may be GPU-seconds, a notional rate, or 1000 everywhere — which turns a budget into a token quota.",
"properties": {
"PromptPer1k": {
"type": "integer",
"format": "int64"
},
"CompletionPer1k": {
"type": "integer",
"format": "int64"
},
"CachedPer1k": {
"type": "integer",
"format": "int64",
"description": "CachedPer1k is what a prompt token costs that the engine or provider read from its cache instead of computing: the part of prompt_tokens the answer counts as cached_tokens. Zero charges those at PromptPer1k, as before there was a rate for them."
},
"PerImage": {
"type": "integer",
"format": "int64",
"description": "PerImage is what one picture costs, for a model that makes them: a picture's cost is the card's time, which its prompt's few tokens do not measure."
},
"PerMesh": {
"type": "integer",
"format": "int64",
"description": "PerMesh is what one 3D model costs: minutes of a card, which no token count measures."
}
}
},
"accounting.Rates": {
"type": "object",
"description": "Rates are an accountant's two limiters' counts.",
@@ -10785,6 +11008,24 @@
}
}
},
"config.SharedStorage": {
"type": "object",
"description": "SharedStorage is where the Slurm cluster's shared files come from: the site's NFS export (NFS), an NFS export llmgw keeps on one of its nodes (Server), or, with neither, a file system the site mounts itself (Lustre, BeeGFS, GPFS), which llmgw only looks for.",
"properties": {
"Path": {
"type": "string",
"description": "Path is where every machine sees it, as /shared."
},
"NFS": {
"type": "string",
"description": "NFS is the site's export, host:/path, mounted at Path on every machine of the Slurm cluster."
},
"Server": {
"type": "string",
"description": "Server is the llmgw node, the controller or one of the nodes, that exports Path over NFS to the others: for a site with no storage of its own. One machine: not highly available, and no faster than its disk and network."
}
}
},
"control.Activity": {
"type": "object",
"description": "Activity is what a standby's router reports back: when it last sent each replica a request, and the on-demand models it was asked for. The leader evicts and loads by these, and without them a replica busy behind a standby looks idle to the one machine that decides.",
@@ -12222,6 +12463,11 @@
"replicas": {
"type": "integer"
},
"stops_by": {
"type": "string",
"format": "date-time",
"description": "StopsBy is, while it is being lent, when a replica still on it is stopped even with nowhere else to go: the drain's grace after the lend began. Zero when not known (a leader that took over meanwhile)."
},
"slurm": {
"$ref": "#/components/schemas/slurmsvc.Reported"
},
@@ -12246,6 +12492,21 @@
"slurmctld_running": {
"type": "boolean"
},
"slurmdbd_running": {
"type": "boolean"
},
"shared": {
"type": "string",
"description": "Shared is the shared path here: \"mounted\", \"exported\", \"missing\"."
},
"containers": {
"type": "string",
"description": "Containers is what the machine has for jobs in containers, empty while not both of pyxis and enroot."
},
"users": {
"type": "integer",
"description": "Users is how many of llmgw's users the machine has."
},
"state": {
"type": "string"
},
@@ -12298,6 +12559,48 @@
"type": "string"
},
"description": "Waiting are nodes left out of slurm.conf until they report their hardware."
},
"shared": {
"$ref": "#/components/schemas/config.SharedStorage",
"description": "Shared is where the shared files are and come from; Users who keeps the users, and how many llmgw has."
},
"users": {
"type": "string"
},
"user_count": {
"type": "integer"
},
"containers": {
"type": "string",
"description": "Containers is whether jobs may run in containers: \"on\" once every machine has pyxis and enroot, \"installing\" until then, empty when slurm.containers is off."
},
"accounts": {
"type": "integer",
"description": "Accounts is how many of Slurm's accounts are llmgw's projects'."
}
}
},
"slurmsvc.User": {
"type": "object",
"description": "User is a Linux user made from an llmgw account.",
"properties": {
"login": {
"type": "string"
},
"uid": {
"type": "integer"
},
"name": {
"type": "string"
},
"email": {
"type": "string"
},
"ssh_keys": {
"type": "array",
"items": {
"type": "string"
}
}
}
},
+6 -2
View File
@@ -67,7 +67,8 @@ or with the mirror unticked, is said in the form and nothing starts.
## Watch
**Overview** is `llmgw ctl status`: nodes, GPUs by class, fabrics, replicas,
models, the store's members, and anything parked, unplaced or idle.
models, the store's members, and anything parked, unplaced or idle. With no
webhook in the cluster's settings it says the alerts go nowhere.
**Monitor** draws the cluster the way btop draws a machine: CPU, memory,
GPU busy and GPU memory, requests and tokens a second, time to first token,
@@ -215,7 +216,10 @@ replicas off and then lets Slurm run jobs there; **Take back from Slurm…**
starts no new job there and gives it back to llmgw when its jobs have ended.
Both change `slurm.lent` in the cluster's settings, as `llmgw ctl slurm lend`
and `reclaim` do. Turning Slurm on and choosing its nodes is `llmgw ctl slurm
setup`, or the Slurm part of the first install.
setup`, or the Slurm part of the first install. Above the nodes it says where
the shared files come from and how many users llmgw made; the SHARED column
is each node's (mounted, exported, missing). `llmgw ctl slurm shared` and
`llmgw ctl slurm users` change them.
**Logs** shows a node's agent log, newest at the bottom, errors and warnings
coloured.
+1 -1
View File
@@ -27,7 +27,7 @@ make test
./bin/llmgw --version
```
`make build` writes `./bin/llmgw`. Go 1.26.6 or later is the only requirement (an older Go fetches it itself).
`make build` writes `./bin/llmgw`. Go 1.27 or later is the only requirement (an older Go fetches it itself); the patch level is never pinned, so CI and the containers build with the newest 1.27.
## 2. Check the path with a fake engine
+77 -4
View File
@@ -177,17 +177,74 @@ init again to serve on it.
choices screen, runs a Slurm cluster beside llmgw: slurmctld on this
machine's node, slurmd on every node, those that join later too
(`slurm.all_nodes`). The agents install munge and Slurm (22.05 or newer)
from the machines' own package sources. No node is Slurm's at first: each is
from the machines' own package sources, and on this machine slurmdbd, which
keeps the jobs' records (`sacct`), in a MariaDB installed beside it that
answers this machine alone. Slurm keeps its records in MySQL or MariaDB
only, not in the cluster's PostgreSQL; the database's password is made from
the cluster's munge key, so nothing new is stored. A MariaDB the machine
already had keeps its settings and only gets the database `slurm_acct_db`
and its user. Back it up with the machine (`mariadb-dump slurm_acct_db`);
records older than a year are purged. No node is Slurm's at first: each is
drained in Slurm and serves models until `llmgw ctl slurm lend NODE` gives it
to Slurm — its models move off first — and `llmgw ctl slurm reclaim NODE`
takes it back once its jobs end. Jobs are submitted on this machine with
`sbatch` and `srun`; `llmgw ctl slurm status` shows where each node is. It
needs this machine to be a node, and its hostname to be a name Slurm takes.
On a cluster installed without it, `llmgw ctl slurm setup --controller NODE
--all-nodes` does the same later. Slurm's ports (6817–6818, and srun's 60001–63000) answer the cluster's
--all-nodes` does the same later. Slurm's ports (6817–6819, and srun's 60001–63000) answer the cluster's
networks only, as llmgw's own do; a firewall of the site's own between the
nodes must let them through.
**Slurm's shared files and users.** Jobs need the same files on every
machine: homes, datasets, output, checkpoints. With `--slurm` this machine
exports `/shared` over NFS to every Slurm machine and they mount it
(`slurm.shared`); NFS's port 2049 answers the cluster's networks only, the
export the Slurm machines alone. That is one machine, not highly available
and no faster than its disk and network: enough for a few nodes. A site with
storage of its own uses it instead — `--slurm-shared HOST:/EXPORT` at
install, or `llmgw ctl slurm shared --path /shared --nfs HOST:/EXPORT`
later; Lustre, BeeGFS or GPFS the site mounts itself, and `llmgw ctl slurm
shared --path /lustre/hpc` only has llmgw look that it is there. Every
active llmgw account is a Linux user on every Slurm machine
(`slurm.users: llmgw`): its login from its e-mail address, a UID from
200000 up that is never given to anyone else, its home under
`/shared/home`. People add their SSH public keys in the portal's
**Profile** and log in to this machine to run `sbatch`; an account suspended
or deleted loses its user, its files stay. On the other Slurm machines they
are let in only while a job of theirs runs there, and their ssh session is
put in that job, held to its cores and cards (pam_slurm_adopt, which the
agents install); root and the machines' own users log in as before. A site whose users come from its
directory (sssd against Active Directory or LDAP) passes `--slurm-users
site`, or later `llmgw ctl slurm users off`.
With llmgw's users, Slurm's accounts are llmgw's projects: an account for
each organization, team and project, nested (`acme`, `acme-ml`,
`acme-ml-training`), and on each project's account its members and its
team's and organization's. A job runs on one of its user's accounts and no
other (`sbatch --account=acme-ml-training`; without it, the user's first),
and when it ends its cards' hours (a job without cards, its cores') are
priced at `auth.gpu_hour_cost` (`core_hour_cost`) into the project's usage,
where it shows as model `slurm/PARTITION` beside the project's requests.
Accounts a Slurm administrator made are left alone. A job's cost does not
yet count against a budget, which binds API keys only.
Jobs may run in containers: `llmgw ctl slurm containers on` has the agents
install enroot and pyxis on every Slurm machine (NVIDIA's releases, at
versions llmgw pins and checks; pyxis is built against each machine's own
Slurm) and, on GPU machines, nvidia-container-cli from NVIDIA's repository.
Once every machine has them Slurm loads pyxis, and `ctl slurm status` says
`containers: on`:
```sh
srun --gres=gpu:1 --container-image=nvcr.io#nvidia/pytorch:25.01-py3 python train.py
```
The shared path is in every container where it is on the host, and the job
runs as its user, held to its cores and cards. Each node pulls the image
from its registry for each job; for a large image used often, import it
once (`enroot import -o /shared/images/pytorch.sqsh docker://nvcr.io#...`)
and give that file to `--container-image`.
**Secure by default.** What init writes is the hardened configuration, not a
demonstration one:
@@ -411,6 +468,19 @@ by itself. Three things are worth knowing before you edit it:
cache (the answer's `cached_tokens`) are charged at it, the rest at the
prompt rate. A provider that bills cached reads at a tenth is
`{prompt_per_1k: 2500, cached_per_1k: 250, completion_per_1k: 10000}`.
- What to price a self-hosted model at follows from what the cluster costs.
Set `auth.gpu_hour_cost`, what an hour of one GPU costs in the units budgets
are in: the hardware written off over its years, its power and room, and the
people who run it, over the GPUs and the 730 hours of a month (an 8-GPU
server at 250,000 over four years, 10 kW, and some upkeep is about 1.35);
`auth.core_hour_cost` likewise for models on CPUs. The leader keeps, by the
hour, the GPU- and core-hours each model's replicas hold, and `llmgw ctl
prices suggest` spreads them over the tokens the model served: an output
token counted as four input ones (making one is that much dearer), a cached
prompt token as a tenth of one. Idle hours are in the cost, so a model held
all month and seldom used comes out dear, which is the point; one held and
billed nothing (a guard's judge) is shown with what it held. `--apply`
writes the prices into `auth.pricing`.
- `control.enabled: true` is what makes this a control plane while its catalog
is empty. Without it, a file with no models is refused as having nothing to
serve.
@@ -597,7 +667,10 @@ set `discovery.nvidia_smi` in the agent configuration rather than moving it.
A node with no GPU at all is fine and is not an error. It reports zero cards and
can host CPU profiles, which is a good way to try llmgw before committing
hardware to it.
hardware to it. A node with an NVIDIA card and no driver is an error: adding it
stops at the checks and says so, and a node already added shows the warning in
`llmgw ctl nodes` and the console until the driver is installed and its agent
restarted.
### NVLink, if your GPUs have it
@@ -3160,7 +3233,7 @@ loaded at boot by `llmgw-firewall.service`):
| Machine | Ports, for the cluster only |
| --- | --- |
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
| node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6818 (slurmctld, slurmd) and 60001–63000 (srun's `SrunPortRange`) |
| node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6819 (slurmctld, slurmd, slurmdbd) and 60001–63000 (srun's `SrunPortRange`); exporting Slurm's shared files, 2049 (NFS) |
| store member | 5432, 8008, 2379, 2380, 9100 |
| mirror | 8081, 8082, 9100 |
+12 -5
View File
@@ -385,18 +385,25 @@ right and mixed facts up on the rest, while the same questions through the
knowledge base got 9 of 9. Use both: the knowledge base for what answers
say, an adapter for how they sound.
Anyone in a project can do it from the portal: **Customise a model**
(`/train`) takes an Excel, CSV or JSONL file of questions and answers, a
model, a name and who may use it, and publishes the adapter by itself when it
is trained. The weights, the node and the card's memory are worked out. From
the command line:
```sh
# 1. Questions and answers from the collection, written by a model of the
# cluster. Read the file through and correct it.
llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl
# 2. Train on a node. --gpu-memory is what the job needs of a card; it
# runs beside an engine when the engine leaves that much free.
llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \
--base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl
# 2. Train. The weights to train on are found from where the model's came
# (--base-repo names others), the card's memory from their size
# (--gpu-memory), the node as the one whose card has the most free
# (--node); it runs beside an engine when the engine leaves that much.
llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct --data policies.jsonl
llmgw ctl train show JOB --follow
# 3. Make it the model's adapter, kept on the mirror.
# 3. Make it the model's adapter, kept on the mirror (or --publish in step 2).
llmgw ctl train publish JOB --restart
```
+6
View File
@@ -181,6 +181,12 @@ Both default to the usual places. Containerized driver installs routinely put
`nvidia-smi` somewhere else, and an agent that cannot find it registers with no
GPUs — which looks exactly like a node that has none.
A node with no driver at all is told apart from one with no card: the PCI bus
is read as well, and an NVIDIA card there with no driver loaded is a failure in
the checks before the agent is installed (what to install, and that a reboot
follows), and a warning on the node in `llmgw ctl nodes` and the console
afterwards. It serves on CPU until the driver is in and its agent restarted.
## Removing a node
```
+78
View File
@@ -43,6 +43,8 @@ rather than repeated.
| `personal` | `Personal` | [Auth](#auth) |
| `pricing` | `map[string]Price` | [Auth](#auth) |
| `default_price` | `Price` | [Auth](#auth) |
| `gpu_hour_cost` | `float64` | [Auth](#auth) |
| `core_hour_cost` | `float64` | [Auth](#auth) |
| `flush_interval` | `time.Duration` | [Auth](#auth) |
| `batch_size` | `int` | [Auth](#auth) |
| `buffer_size` | `int` | [Auth](#auth) |
@@ -401,6 +403,9 @@ rather than repeated.
| `action` | `string` | [Service](#service) |
| `timeout` | `time.Duration` | [Service](#service) |
| `stopped` | `bool` | [ServingSettings](#servingsettings) |
| `path` | `string` | [SharedStorage](#sharedstorage) |
| `nfs` | `string` | [SharedStorage](#sharedstorage) |
| `server` | `string` | [SharedStorage](#sharedstorage) |
| `bin` | `string` | [SimConfig](#simconfig) |
| `enabled` | `bool` | [SlurmSettings](#slurmsettings) |
| `cluster_name` | `string` | [SlurmSettings](#slurmsettings) |
@@ -409,6 +414,9 @@ rather than repeated.
| `all_nodes` | `bool` | [SlurmSettings](#slurmsettings) |
| `lent` | `[]string` | [SlurmSettings](#slurmsettings) |
| `partition` | `string` | [SlurmSettings](#slurmsettings) |
| `shared` | `SharedStorage` | [SlurmSettings](#slurmsettings) |
| `users` | `string` | [SlurmSettings](#slurmsettings) |
| `containers` | `bool` | [SlurmSettings](#slurmsettings) |
| `dir` | `string` | [StrataConfig](#strataconfig) |
| `python` | `string` | [StrataConfig](#strataconfig) |
| `engine` | `string` | [StrataConfig](#strataconfig) |
@@ -716,6 +724,21 @@ table costs whatever DefaultPrice says, which is nothing unless set.
Takes a [config.Price](#configprice) block.
### `gpu_hour_cost`
`float64`
GPUHourCost is what an hour of one GPU costs the operator, in the
units budgets are in: the hardware written off over its years, its
power and room, and the people who run it, over the GPUs and the
hours of a month. With CoreHourCost for a CPU core, it is what
`llmgw ctl prices suggest` prices each model from: the hours its
replicas held, idle ones too, over the tokens it served.
### `core_hour_cost`
`float64`
### `flush_interval`
`time.Duration`
@@ -3488,6 +3511,30 @@ running and every model's definition as it is: llmgw stop sets it,
llmgw start clears it, and the models come back as their catalog
entries ask. In force on every control plane within seconds.
## SharedStorage
### `path`
`string`
Path is where every machine sees it, as /shared.
### `nfs`
`string`
NFS is the site's export, host:/path, mounted at Path on every
machine of the Slurm cluster.
### `server`
`string`
Server is the llmgw node, the controller or one of the nodes, that
exports Path over NFS to the others: for a site with no storage of
its own. One machine: not highly available, and no faster than its
disk and network.
## SimConfig
### `bin`
@@ -3548,6 +3595,37 @@ given back to llmgw when its last job ends.
Partition is the one partition every node is in. Empty is "gpu".
### `shared`
`SharedStorage`
Shared is the file system every machine of the Slurm cluster sees at
the same path: home directories, datasets, the jobs' output and
checkpoints. Empty path: none, and a batch job's output stays on the
node it ran on.
Takes a [SharedStorage](#sharedstorage) block.
### `users`
`string`
Users is who keeps the Linux users that run jobs, the same on every
machine of the Slurm cluster: "llmgw" makes one for each active llmgw
account (its login from the e-mail address, a UID of llmgw's from
200000 up, its SSH keys from the portal), let in to the controller,
and to a node only while a job of theirs runs there (pam_slurm_adopt);
empty leaves users to the site (a directory through sssd, or by hand).
### `containers`
`bool`
Containers lets jobs run in containers: `srun --container-image
nvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),
the GPUs and the shared path inside. The agents install both on
every Slurm machine; jobs take it once every machine has them.
## StrataConfig
### `dir`
+1 -1
View File
@@ -1,6 +1,6 @@
module gitea.mynodes.xyz/ilker/llmgw
go 1.26.6
go 1.27
require (
connectrpc.com/connect v1.21.0
+91
View File
@@ -0,0 +1,91 @@
package accounting
import (
"math"
"slices"
)
// Prices from what the cluster costs: what each model held of it (its
// replicas' GPU- and core-hours) at what an hour of a GPU or a core costs
// the operator, spread over the tokens it served. Idle time is in it: a
// model held all month and used an hour costs what it held, not the hour.
//
// An output token is priced OutputPerInput input tokens, as producing one
// is that much dearer (one step a token, against a prompt read in one
// pass); a cached prompt token a tenth of an input one.
const (
OutputPerInput = 4
CachedPerInput = 0.1
)
// Basis is what an hour costs, in the units budgets are in.
type Basis struct {
GPUHour float64
CoreHour float64
}
// Held is what a model held, and Served what it answered with.
type Held struct {
GPUSeconds, CoreSeconds int64
}
// Served is a model's tokens.
type Served struct {
Prompt, Completion int64
}
// Suggestion is one model's price from its cost.
type Suggestion struct {
Model string `json:"model"`
GPUHours float64 `json:"gpu_hours"`
CoreHours float64 `json:"core_hours"`
// Cost is what it held cost, in budget units.
Cost float64 `json:"cost"`
Prompt int64 `json:"prompt_tokens"`
Completion int64 `json:"completion_tokens"`
// Price is the suggestion; zero with a Note saying why there is none.
Price Price `json:"price"`
Note string `json:"note,omitempty"`
}
// Suggest prices every model that held something or served something.
func Suggest(held map[string]Held, served map[string]Served, b Basis) []Suggestion {
var names []string
for m := range held {
names = append(names, m)
}
for m := range served {
if _, ok := held[m]; !ok {
names = append(names, m)
}
}
slices.Sort(names)
out := make([]Suggestion, 0, len(names))
for _, m := range names {
h, s := held[m], served[m]
g := Suggestion{Model: m, GPUHours: float64(h.GPUSeconds) / 3600, CoreHours: float64(h.CoreSeconds) / 3600,
Prompt: s.Prompt, Completion: s.Completion}
g.Cost = g.GPUHours*b.GPUHour + g.CoreHours*b.CoreHour
// Tokens counted as output ones: an input token is a fraction of one.
units := float64(s.Completion) + float64(s.Prompt)/OutputPerInput
switch {
case g.Cost == 0 && units > 0:
g.Note = "held nothing recorded here: an outside provider's, or served before the hours were kept"
case g.Cost > 0 && units == 0:
g.Note = "held the cluster and was billed no tokens (a guard's judge, or a model priced by the picture or 3D model): no one pays what it held"
case g.Cost > 0:
// Micro-units per thousand output tokens, rounded up so a
// cheap model is never free.
out1k := int64(math.Ceil(g.Cost / units * 1000 * 1e6))
g.Price = Price{
CompletionPer1k: out1k,
PromptPer1k: ceilDiv(out1k, OutputPerInput),
CachedPer1k: max(int64(math.Ceil(float64(ceilDiv(out1k, OutputPerInput))*CachedPerInput)), 1),
}
}
out = append(out, g)
}
return out
}
func ceilDiv(a, b int64) int64 { return (a + b - 1) / b }
+45
View File
@@ -0,0 +1,45 @@
package accounting
import (
"strings"
"testing"
)
func TestSuggestSpreadsWhatAModelHeldOverItsTokens(t *testing.T) {
// The worked example: two cards for 1000 hours at 1.35 an hour is 2700;
// 3.0 billion output tokens and 4.0 billion input ones, counted as 1.0
// billion output ones, are 4.0 billion: 0.675 a million output tokens.
held := map[string]Held{
"llama-70b": {GPUSeconds: 2 * 1000 * 3600},
"idle": {GPUSeconds: 3600},
"small-cpu": {CoreSeconds: 8 * 100 * 3600},
}
served := map[string]Served{
"llama-70b": {Prompt: 4_000_000_000, Completion: 3_000_000_000},
"small-cpu": {Prompt: 1_000_000, Completion: 1_000_000},
"outside": {Prompt: 10, Completion: 10},
}
got := Suggest(held, served, Basis{GPUHour: 1.35, CoreHour: 0.01})
by := map[string]Suggestion{}
for _, s := range got {
by[s.Model] = s
}
l := by["llama-70b"]
if l.Cost != 2700 || l.Price.CompletionPer1k != 675 || l.Price.PromptPer1k != 169 || l.Price.CachedPer1k != 17 {
t.Errorf("llama: %+v", l)
}
// What it costs back at that price is what it held, within the rounding.
back := float64(l.Price.Cost(l.Prompt, 0, l.Completion, 0, 0)) / 1e6
if back < l.Cost || back > l.Cost*1.01 {
t.Errorf("charged %.2f for a cost of %.2f", back, l.Cost)
}
if s := by["small-cpu"]; s.Cost != 8 || s.Price.CompletionPer1k == 0 {
t.Errorf("cpu: %+v", s)
}
if s := by["idle"]; !s.Price.Zero() || !strings.Contains(s.Note, "billed no tokens") {
t.Errorf("idle: %+v", s)
}
if s := by["outside"]; !s.Price.Zero() || !strings.Contains(s.Note, "held nothing") {
t.Errorf("outside: %+v", s)
}
}
+1 -1
View File
@@ -596,7 +596,7 @@ func (a *Agent) session(ctx context.Context, server string) error {
a.forgetSent()
reg.Host = &llmgwv1.Host{
OsId: h.OSID, OsVersion: h.OSVersion, OsCodename: h.OSCodename,
Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver,
Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver, NvidiaDevice: h.NVIDIADevice,
// What is measured by now, not what the startup scan found before
// the measurement ended: a reconnect would otherwise say 0.
MemBandwidthGbs: a.memBandwidth().All,
+31
View File
@@ -3,6 +3,7 @@ package discovery
import (
"bufio"
"os"
"path/filepath"
"regexp"
"runtime"
"strconv"
@@ -28,6 +29,10 @@ type Host struct {
CPUFlags []string `json:"cpu_flags,omitempty"`
// NVIDIADriver is the loaded driver's version, empty when there is none.
NVIDIADriver string `json:"nvidia_driver,omitempty"`
// NVIDIADevice is whether an NVIDIA display controller is on the PCI bus,
// driver or not. True with NVIDIADriver empty is a card without a driver:
// a machine that would otherwise serve on CPU with nothing saying why.
NVIDIADevice bool `json:"nvidia_device,omitempty"`
// MinComputeCapability is the oldest GPU's, 0 when there are none or it
// is not known. Not read here but from the GPU inventory, by the agent.
MinComputeCapability float64 `json:"min_compute_capability,omitempty"`
@@ -75,9 +80,35 @@ func DiscoverHost() Host {
h.Arch = debianArch(runtime.GOARCH)
h.CPUFlags = cpuFlagsFrom("/proc/cpuinfo")
h.NVIDIADriver = nvidiaDriverFrom("/proc/driver/nvidia/version")
h.NVIDIADevice = HasNVIDIADevice()
return h
}
// CardWithoutDriver reports an NVIDIA card on the PCI bus with no driver
// loaded: a node that looks like a CPU node to nvidia-smi and is not one.
func (h Host) CardWithoutDriver() bool { return h.NVIDIADevice && h.NVIDIADriver == "" }
// HasNVIDIADevice reports an NVIDIA display or 3D controller on the PCI bus,
// read from sysfs, which needs no driver.
func HasNVIDIADevice() bool { return hasNVIDIADeviceIn("/sys/bus/pci/devices") }
func hasNVIDIADeviceIn(root string) bool {
vendors, _ := filepath.Glob(filepath.Join(root, "*", "vendor"))
for _, v := range vendors {
b, err := os.ReadFile(v) //nolint:gosec // sysfs
if err != nil || strings.TrimSpace(string(b)) != "0x10de" {
continue
}
class, _ := os.ReadFile(filepath.Join(filepath.Dir(v), "class")) //nolint:gosec // sysfs
// 0x03xxxx is a display or 3D controller; NVIDIA also makes bridges
// and audio devices.
if strings.HasPrefix(strings.TrimSpace(string(class)), "0x03") {
return true
}
}
return false
}
func hostFromOSRelease(path string) Host {
var h Host
f, err := os.Open(path) //nolint:gosec // a fixed system path
@@ -23,3 +23,35 @@ func TestNVIDIADriverFrom(t *testing.T) {
}
}
}
func TestACardOnTheBusIsSeenWithoutADriver(t *testing.T) {
root := t.TempDir()
dev := func(name, vendor, class string) {
d := filepath.Join(root, name)
if err := os.MkdirAll(d, 0o750); err != nil {
t.Fatal(err)
}
_ = os.WriteFile(filepath.Join(d, "vendor"), []byte(vendor+"\n"), 0o600)
_ = os.WriteFile(filepath.Join(d, "class"), []byte(class+"\n"), 0o600)
}
dev("0000:00:1f.0", "0x8086", "0x060100") // the chipset
if hasNVIDIADeviceIn(root) {
t.Fatal("a machine with no NVIDIA device said it had one")
}
dev("0000:01:00.1", "0x10de", "0x040300") // the card's HDMI audio alone is not a GPU
if hasNVIDIADeviceIn(root) {
t.Fatal("NVIDIA's audio device was taken for a GPU")
}
dev("0000:01:00.0", "0x10de", "0x030000")
if !hasNVIDIADeviceIn(root) {
t.Fatal("a VGA controller of NVIDIA's was not seen")
}
h := Host{NVIDIADevice: true}
if !h.CardWithoutDriver() {
t.Error("a card with no driver version is a card without a driver")
}
h.NVIDIADriver = "580.82.07"
if h.CardWithoutDriver() {
t.Error("a card with a driver was reported as without one")
}
}
+12 -4
View File
@@ -16,10 +16,14 @@ import (
// slurmPorts are Slurm's on a node that runs any of it: slurmctld's and
// slurmd's, and srun's range for the steps it starts.
var slurmPorts = []firewall.Port{
{From: slurm.CtldPort, To: slurm.SlurmdPort, What: "slurm: slurmctld and slurmd"},
{From: slurm.CtldPort, To: slurm.DbdPort, What: "slurm: slurmctld, slurmd and slurmdbd"},
{From: slurm.SrunPortsFrom, To: slurm.SrunPortsTo, What: "slurm: srun, for the steps it starts"},
}
// nfsPort is NFSv4's, on the machine that exports the Slurm cluster's
// shared files.
var nfsPort = firewall.Port{From: slurm.NFSPort, To: slurm.NFSPort, What: "nfs: the Slurm cluster's shared files"}
// applyFirewall keeps llmgw's firewall on this node as the cluster's
// settings say: the engines' ports and the node exporter's for the networks
// the node reaches its control planes on, and those named, or no table at
@@ -69,6 +73,9 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
if a.slurm != nil && a.slurm.HasPart() {
ports = append(ports, slurmPorts...)
}
if a.slurm != nil && a.slurm.Serves() {
ports = append(ports, nfsPort)
}
changed, err := firewall.Install(ctx, r, firewall.Spec{Ports: ports, Allow: nets})
switch {
case err != nil:
@@ -79,11 +86,12 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
}
// slurmMessage is the leader's word on this node's part in Slurm. A node
// that gets one has the firewall applied again, with Slurm's ports.
// that gets one, or starts or stops exporting the shared files, has the
// firewall applied again, with Slurm's ports and NFS's.
func (a *Agent) slurmMessage(m *llmgwv1.Slurm) {
had := a.slurm.HasPart()
had, served := a.slurm.HasPart(), a.slurm.Serves()
a.slurm.Apply(m)
if had || !a.slurm.HasPart() {
if (had || !a.slurm.HasPart()) && served == a.slurm.Serves() {
return
}
a.fwMu.Lock()
+1 -1
View File
@@ -73,7 +73,7 @@ func TestANodeGivenAPartInSlurmLetsSlurmInFromTheClusterOnly(t *testing.T) {
a.fwMu.Lock()
got := sh.installed
a.fwMu.Unlock()
if strings.Contains(got, "6817-6818") && strings.Contains(got, "60001-63000") {
if strings.Contains(got, "6817-6819") && strings.Contains(got, "60001-63000") {
break
}
if time.Now().After(deadline) {
+2 -1
View File
@@ -483,7 +483,8 @@ func (in *Installer) packages(ctx context.Context, byFamily map[string][]string,
return fmt.Errorf("%s is missing and no package manager was found to install it",
strings.Join(missing, ", "))
}
in.progress(st, "installing %s with %s", strings.Join(missing, ", "), tool)
// The state already says installing.
in.progress(st, "%s, with %s", strings.Join(missing, ", "), tool)
for _, c := range cmds {
if err := in.runWaitingForLock(ctx, pm.env, c, st); err != nil {
return err
+214
View File
@@ -0,0 +1,214 @@
package slurmnode
import (
"context"
"errors"
"fmt"
"slices"
"strings"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
// accountsMarker lists what llmgw made in Slurm's accounting, a line each:
// "account NAME" and "user LOGIN NAME". Nothing else is changed or taken
// away: an account a Slurm administrator made is theirs.
func (k *Keeper) accountsMarker() string { return k.marker + "-accounts" }
// maxEnded is the most ended jobs sent at once: a day's on a busy cluster
// would otherwise be a status message of megabytes.
const maxEnded = 2000
// accounts keeps Slurm's accounts and their users as want says, on the
// controller.
func (k *Keeper) accounts(ctx context.Context, want *llmgwv1.Slurm) error {
made := k.madeAccounts()
if !want.GetManageAccounts() {
if len(made) == 0 {
return nil
}
return k.applyAccounts(ctx, nil, made)
}
return k.applyAccounts(ctx, want.GetAccounts(), made)
}
func (k *Keeper) madeAccounts() []string {
b, err := k.host.ReadFile(k.accountsMarker())
if err != nil {
return nil
}
var out []string
for _, l := range strings.Split(strings.TrimSpace(string(b)), "\n") {
if l != "" {
out = append(out, l)
}
}
return out
}
func (k *Keeper) sacctmgr(ctx context.Context, args ...string) error {
out, err := k.host.Run(ctx, nil, "sacctmgr", append([]string{"-i"}, args...)...)
if err != nil {
return fmt.Errorf("sacctmgr %s: %v: %s", strings.Join(args, " "), err, lastLine(out))
}
return nil
}
func (k *Keeper) applyAccounts(ctx context.Context, want []*llmgwv1.SlurmAccount, made []string) error {
out, err := k.host.Run(ctx, nil, "sacctmgr", slurm.AssocFormat...)
if err != nil {
return fmt.Errorf("reading Slurm's accounts: %v: %s", err, lastLine(out))
}
have := map[string]bool{} // "account NAME", "user LOGIN NAME"
for _, a := range slurm.ParseAssociations(string(out)) {
if a.User == "" {
have["account "+a.Account] = true
} else {
have["user "+a.User+" "+a.Account] = true
}
}
keep := map[string]bool{}
var errs []error
ours := slices.Clone(made)
note := func(line string) {
if !slices.Contains(ours, line) {
ours = append(ours, line)
}
}
for _, a := range want {
name, parent := a.GetName(), a.GetParent()
if !validAccount(name) || (parent != "root" && !validAccount(parent)) {
errs = append(errs, fmt.Errorf("the leader sent an account %q under %q", name, parent))
continue
}
line := "account " + name
keep[line] = true
if !have[line] {
k.log.Info("a Slurm account is made, as llmgw's projects say", "account", name, "parent", parent)
if err := k.sacctmgr(ctx, "add", "account", name, "parent="+parent, "description=llmgw"); err != nil {
errs = append(errs, err)
continue
}
note(line)
}
for _, u := range a.GetUsers() {
if !slurm.ValidLogin(u) {
continue
}
ul := "user " + u + " " + name
keep[ul] = true
if have[ul] {
continue
}
if err := k.sacctmgr(ctx, "add", "user", u, "account="+name); err != nil {
errs = append(errs, err)
continue
}
note(ul)
}
}
// Taken away: what llmgw made and no longer wants, users before their
// accounts, children before their parents (made later, listed later).
var left []string
for i := len(ours) - 1; i >= 0; i-- {
line := ours[i]
if keep[line] || !strings.HasPrefix(line, "user ") {
continue
}
f := strings.Fields(line)
if !have[line] {
continue
}
if err := k.removeAssoc(ctx, f[1], f[2], have, keep); err != nil {
errs = append(errs, err)
left = append(left, line)
}
}
for i := len(ours) - 1; i >= 0; i-- {
line := ours[i]
if keep[line] || !strings.HasPrefix(line, "account ") {
continue
}
if have[line] {
k.log.Info("a Slurm account llmgw made is taken away; its jobs' records stay", "account", line[len("account "):])
if err := k.sacctmgr(ctx, "remove", "account", line[len("account "):]); err != nil {
errs = append(errs, err)
left = append(left, line)
}
}
}
var lines []string
for _, l := range ours {
if keep[l] || slices.Contains(left, l) {
lines = append(lines, l)
}
}
if !slices.Equal(lines, made) {
var err error
if len(lines) == 0 {
err = k.host.Remove(k.accountsMarker())
} else {
_, err = k.write(k.accountsMarker(), []byte(strings.Join(lines, "\n")+"\n"), 0o644, "")
}
errs = append(errs, err)
}
return errors.Join(errs...)
}
func validAccount(s string) bool {
return s != "" && len(s) <= 64 && strings.Trim(s, "abcdefghijklmnopqrstuvwxyz0123456789_-") == ""
}
// ended are the jobs that ended in the last day, newest last.
func (k *Keeper) ended(ctx context.Context) ([]*llmgwv1.SlurmJob, error) {
out, err := k.host.Run(ctx, nil, "sacct", slurm.SacctArgs...)
if err != nil {
return nil, fmt.Errorf("sacct: %v: %s", err, lastLine(out))
}
jobs, err := slurm.ParseJobs(string(out))
if err != nil {
return nil, err
}
if len(jobs) > maxEnded {
jobs = jobs[len(jobs)-maxEnded:]
}
ps := make([]*llmgwv1.SlurmJob, 0, len(jobs))
for _, j := range jobs {
ps = append(ps, &llmgwv1.SlurmJob{Id: j.ID, User: j.User, Account: j.Account, Partition: j.Partition,
State: j.State, StartUnix: j.Start.Unix(), EndUnix: j.End.Unix(), ElapsedSeconds: int64(j.Elapsed.Seconds()),
Cpus: int32(j.CPUs), Gpus: int32(j.GPUs)}) //nolint:gosec // counts of a node's
}
return ps, nil
}
// removeAssoc takes login off account. Slurm refuses to take away a
// user's default account, so another of theirs is made the default first,
// one llmgw keeps if there is one; with none left the user goes from the
// accounting entirely.
func (k *Keeper) removeAssoc(ctx context.Context, login, account string, have, keep map[string]bool) error {
var others []string
for line := range have {
f := strings.Fields(line)
if len(f) == 3 && f[0] == "user" && f[1] == login && f[2] != account {
others = append(others, f[2])
}
}
if len(others) == 0 {
return k.sacctmgr(ctx, "remove", "user", login)
}
slices.SortFunc(others, func(a, b string) int {
ka, kb := keep["user "+login+" "+a], keep["user "+login+" "+b]
switch {
case ka && !kb:
return -1
case kb && !ka:
return 1
}
return strings.Compare(a, b)
})
if err := k.sacctmgr(ctx, "modify", "user", login, "set", "defaultaccount="+others[0]); err != nil {
return err
}
return k.sacctmgr(ctx, "remove", "user", login, "account="+account)
}
+83
View File
@@ -0,0 +1,83 @@
package slurmnode
import (
"context"
"slices"
"strings"
"testing"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
func TestAccountsMadeAndTakenAwayOnlyIfLlmgwsOwn(t *testing.T) {
h := newFake()
k := New(h, nil, "/var/lib/llmgw-agent/slurm", nil)
assoc := "sacctmgr " + strings.Join(slurm.AssocFormat, " ")
// root, and an account a Slurm administrator made.
h.outputs[assoc] = "root||\nroot|root|\nphysics||root\n"
m := &llmgwv1.Slurm{ManageAccounts: true, Accounts: []*llmgwv1.SlurmAccount{
{Name: "test", Parent: "root"}, {Name: "test-arge", Parent: "test"},
{Name: "test-arge-deneme", Parent: "test-arge", Users: []string{"ada", "ayse"}},
}}
if err := k.accounts(context.Background(), m); err != nil {
t.Fatal(err)
}
for _, want := range []string{"sacctmgr -i add account test parent=root description=llmgw",
"sacctmgr -i add account test-arge-deneme parent=test-arge description=llmgw",
"sacctmgr -i add user ayse account=test-arge-deneme"} {
if !slices.Contains(h.ran, want) {
t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
}
}
if !strings.Contains(string(h.files[k.accountsMarker()]), "user ada test-arge-deneme") {
t.Errorf("marker %q", h.files[k.accountsMarker()])
}
// Now there; ayse leaves the project, and the team goes.
h.outputs[assoc] = "root||\nphysics||root\ntest||root\ntest-arge||test\ntest-arge-deneme||test-arge\n" +
"test-arge-deneme|ada|test-arge\ntest-arge-deneme|ayse|test-arge\n"
h.ran = nil
m.Accounts = []*llmgwv1.SlurmAccount{{Name: "test", Parent: "root"}}
if err := k.accounts(context.Background(), m); err != nil {
t.Fatal(err)
}
// Neither has another account: they go from the accounting.
want := []string{"sacctmgr -i remove user ayse", "sacctmgr -i remove user ada",
"sacctmgr -i remove account test-arge-deneme", "sacctmgr -i remove account test-arge"}
var got []string
for _, c := range h.ran {
if strings.Contains(c, "remove") || strings.Contains(c, " add ") {
got = append(got, c)
}
}
if !slices.Equal(got, want) {
t.Errorf("ran:\n%s\nwant:\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
}
if slices.ContainsFunc(h.ran, func(c string) bool { return strings.Contains(c, "physics") }) {
t.Error("touched an account llmgw did not make")
}
// One still on another account: that one is made the default first.
h.outputs[assoc] = "root||\nphysics||root\nphysics|ada|\ntest||root\nx||test\nx|ada|\n"
h.files[k.accountsMarker()] = []byte("account test\naccount x\nuser ada x\n")
h.ran = nil
m.Accounts = []*llmgwv1.SlurmAccount{{Name: "test", Parent: "root"}}
if err := k.accounts(context.Background(), m); err != nil {
t.Fatal(err)
}
if i, j := slices.Index(h.ran, "sacctmgr -i modify user ada set defaultaccount=physics"),
slices.Index(h.ran, "sacctmgr -i remove user ada account=x"); i < 0 || j < i {
t.Errorf("ran:\n%s", strings.Join(h.ran, "\n"))
}
// Off: what llmgw made goes.
h.outputs[assoc] = "root||\nphysics||root\ntest||root\n"
h.ran = nil
if err := k.accounts(context.Background(), &llmgwv1.Slurm{}); err != nil {
t.Fatal(err)
}
if !slices.Contains(h.ran, "sacctmgr -i remove account test") || h.files[k.accountsMarker()] != nil {
t.Errorf("not taken away:\n%s", strings.Join(h.ran, "\n"))
}
}
+198
View File
@@ -0,0 +1,198 @@
package slurmnode
import (
"context"
"fmt"
"path/filepath"
"runtime"
"strings"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
// cacheDir holds what is fetched for Slurm: enroot's packages, pyxis's
// source and its build.
const cacheDir = "/var/cache/llmgw-agent/slurm"
// pyxisMarker is the pyxis version built here, against this Slurm.
func (k *Keeper) pyxisMarker() string { return k.marker + "-pyxis" }
// containers has enroot and pyxis here, set up, when want says so; what
// is there is left when it no longer does (Slurm stops loading pyxis once
// the controller's plugstack.conf goes). Answers the status line, empty
// while either is missing.
func (k *Keeper) containers(ctx context.Context, want *llmgwv1.Slurm) (string, error) {
if !want.GetContainers() {
return "", nil
}
family := k.family()
if family == "" {
return "", fmt.Errorf("neither dpkg nor rpm here: enroot and pyxis come as deb or rpm")
}
if err := k.enroot(ctx, family); err != nil {
return "", fmt.Errorf("enroot: %w", err)
}
if err := k.enrootConf(ctx, want.GetShared().GetPath()); err != nil {
return "", fmt.Errorf("enroot's settings: %w", err)
}
if _, err := k.host.LookPath("nvidia-smi"); err == nil {
if err := k.nvidiaContainer(ctx, family); err != nil {
return "", fmt.Errorf("the GPUs in containers (nvidia-container-cli): %w", err)
}
}
if err := k.pyxis(ctx, family); err != nil {
return "", fmt.Errorf("pyxis: %w", err)
}
return slurm.ContainersStatus(), nil
}
func (k *Keeper) family() string {
if _, err := k.host.LookPath("dpkg"); err == nil {
return "deb"
}
if _, err := k.host.LookPath("rpm"); err == nil {
return "rpm"
}
return ""
}
// enroot installs the pinned release when another, or none, is here.
func (k *Keeper) enroot(ctx context.Context, family string) error {
if k.installed["enroot"] {
return nil
}
if out, err := k.host.Run(ctx, nil, "enroot", "version"); err == nil && strings.TrimSpace(string(out)) == slurm.EnrootVersion {
k.installed["enroot"] = true
return nil
}
pkgs, err := slurm.EnrootPackages(family, runtime.GOARCH)
if err != nil {
return err
}
if k.inst == nil {
return fmt.Errorf("enroot missing, and this agent installs nothing")
}
var files []string
for _, p := range pkgs {
f := filepath.Join(cacheDir, p.Name)
if err := k.host.Download(ctx, p.URL(), f, p.SHA256); err != nil {
return err
}
files = append(files, f)
}
k.log.Info("installing enroot, for Slurm's jobs in containers", "version", slurm.EnrootVersion)
if err := k.inst.InstallPackages(ctx, map[string][]string{family: files}); err != nil {
return err
}
out, err := k.host.Run(ctx, nil, "enroot", "version")
if v := strings.TrimSpace(string(out)); err != nil || v != slurm.EnrootVersion {
return fmt.Errorf("enroot says %q after installing %s: %v", v, slurm.EnrootVersion, err)
}
k.installed["enroot"] = true
return nil
}
// enrootConf writes enroot's settings and makes its directories.
func (k *Keeper) enrootConf(ctx context.Context, shared string) error {
if _, err := k.write(slurm.EnrootConfPath, []byte(slurm.EnrootConf), 0o644, ""); err != nil {
return err
}
changed, err := k.write(slurm.EnrootTmpfiles, []byte(slurm.EnrootTmpfilesConf), 0o644, "")
if err != nil {
return err
}
if changed {
if out, err := k.host.Run(ctx, nil, "systemd-tmpfiles", "--create", slurm.EnrootTmpfiles); err != nil {
return fmt.Errorf("systemd-tmpfiles: %v: %s", err, lastLine(out))
}
}
for _, d := range []string{slurm.EnrootLib + "/cache", slurm.EnrootLib + "/data"} {
if _, err := k.host.Run(ctx, nil, "test", "-d", d); err == nil {
continue
}
// Each user makes their own beneath, as in /tmp.
if out, err := k.host.Run(ctx, nil, "install", "-d", "-m", "1777", d); err != nil {
return fmt.Errorf("making %s: %v: %s", d, err, lastLine(out))
}
}
if shared == "" {
return k.host.Remove(slurm.EnrootMounts)
}
_, err = k.write(slurm.EnrootMounts, []byte(slurm.EnrootSharedMount(shared)), 0o644, "")
return err
}
// nvidiaContainer installs nvidia-container-cli, which enroot's hook uses
// to give a container the job's GPUs, from NVIDIA's repository.
func (k *Keeper) nvidiaContainer(ctx context.Context, family string) error {
if k.installed["nvidia-container-cli"] {
return nil
}
if _, err := k.host.LookPath("nvidia-container-cli"); err == nil {
k.installed["nvidia-container-cli"] = true
return nil
}
if k.inst == nil {
return fmt.Errorf("nvidia-container-cli missing, and this agent installs nothing")
}
switch family {
case "deb":
if err := k.host.Download(ctx, slurm.NvidiaContainerKeyURL, slurm.NvidiaContainerKeyPath,
slurm.NvidiaContainerKeySHA256); err != nil {
return err
}
if _, err := k.write(slurm.NvidiaContainerAptList, []byte(slurm.NvidiaContainerApt), 0o644, ""); err != nil {
return err
}
case "rpm":
if _, err := k.write(slurm.NvidiaContainerYumRepo, []byte(slurm.NvidiaContainerYum), 0o644, ""); err != nil {
return err
}
}
k.log.Info("installing nvidia-container-cli, so containers get the job's GPUs")
if err := k.inst.InstallPackages(ctx, slurm.NvidiaContainerPackages); err != nil {
return err
}
if _, err := k.host.LookPath("nvidia-container-cli"); err != nil {
return fmt.Errorf("still missing after installing libnvidia-container-tools")
}
k.installed["nvidia-container-cli"] = true
return nil
}
// pyxis builds the pinned pyxis against this machine's Slurm and installs
// it, when it has not been, or was for another version.
func (k *Keeper) pyxis(ctx context.Context, family string) error {
if b, err := k.host.ReadFile(k.pyxisMarker()); err == nil && strings.TrimSpace(string(b)) == slurm.PyxisVersion {
if _, err := k.host.Run(ctx, nil, "test", "-f", slurm.PyxisPlugin); err == nil {
return nil
}
}
if k.inst == nil {
return fmt.Errorf("pyxis missing, and this agent installs nothing")
}
if err := k.inst.InstallPackages(ctx, map[string][]string{family: slurm.PyxisBuildPackages[family]}); err != nil {
return fmt.Errorf("installing what builds it: %w", err)
}
tarball := filepath.Join(cacheDir, "pyxis-"+slurm.PyxisVersion+".tar.gz")
if err := k.host.Download(ctx, slurm.PyxisSource, tarball, slurm.PyxisSHA256); err != nil {
return err
}
src := filepath.Join(cacheDir, "pyxis-"+slurm.PyxisVersion)
k.log.Info("building pyxis against this machine's Slurm", "version", slurm.PyxisVersion)
for _, c := range [][]string{
{"rm", "-rf", src},
{"tar", "-xzf", tarball, "-C", cacheDir},
{"make", "-C", src, "install"},
} {
if out, err := k.host.Run(ctx, nil, c[0], c[1:]...); err != nil {
return fmt.Errorf("%s: %v: %s", strings.Join(c, " "), err, lastLine(out))
}
}
if _, err := k.host.Run(ctx, nil, "test", "-f", slurm.PyxisPlugin); err != nil {
return fmt.Errorf("built, yet %s is not there", slurm.PyxisPlugin)
}
_, err := k.write(k.pyxisMarker(), []byte(slurm.PyxisVersion+"\n"), 0o644, "")
return err
}
@@ -0,0 +1,92 @@
package slurmnode
import (
"context"
"runtime"
"slices"
"strings"
"testing"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
func TestContainersInstalledFromPinnedReleases(t *testing.T) {
if _, err := slurm.EnrootPackages("deb", runtime.GOARCH); err != nil {
t.Skip(err)
}
h := newFake()
h.bins["dpkg"], h.bins["nvidia-smi"] = true, true
inst := &fakeInst{host: h}
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
// enroot is not here until installed, and then says its version.
h.fail["enroot version"] = true
h.fail["test -d "+slurm.EnrootLib+"/cache"] = true
h.fail["test -f "+slurm.PyxisPlugin] = true
m := nodeMsg(true)
m.Containers = true
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Nfs: "10.0.0.11:/shared"}
st := k.Reconcile(context.Background(), m)
if st.GetContainers() != "" || !strings.Contains(st.GetError(), "containers: enroot") {
t.Fatalf("ready with enroot saying nothing: %v", st)
}
pkgs, _ := slurm.EnrootPackages("deb", runtime.GOARCH)
for _, p := range pkgs {
if !slices.Contains(h.ran, "download "+p.URL()+" "+cacheDir+"/"+p.Name+" "+p.SHA256) {
t.Errorf("%s not fetched with its digest:\n%s", p.Name, strings.Join(h.ran, "\n"))
}
}
if len(inst.got["deb"]) != 2 || !strings.HasPrefix(inst.got["deb"][0], cacheDir) {
t.Errorf("installed %v", inst.got)
}
delete(h.fail, "enroot version")
h.outputs["enroot version"] = slurm.EnrootVersion + "\n"
h.bins["nvidia-container-cli"] = false
st = k.Reconcile(context.Background(), m)
if !strings.Contains(st.GetError(), "nvidia-container-cli") {
t.Errorf("a GPU machine without nvidia-container-cli is ready: %v", st)
}
if string(h.files[slurm.NvidiaContainerAptList]) != slurm.NvidiaContainerApt ||
!slices.Contains(h.ran, "download "+slurm.NvidiaContainerKeyURL+" "+slurm.NvidiaContainerKeyPath+" "+slurm.NvidiaContainerKeySHA256) {
t.Errorf("NVIDIA's repository not added with its key checked")
}
h.bins["nvidia-container-cli"] = true
st = k.Reconcile(context.Background(), m)
// Built, but the plugin is still not there.
if st.GetContainers() != "" || !strings.Contains(st.GetError(), "pyxis") {
t.Fatalf("status %v", st)
}
for _, want := range []string{"tar -xzf " + cacheDir + "/pyxis-" + slurm.PyxisVersion + ".tar.gz -C " + cacheDir,
"make -C " + cacheDir + "/pyxis-" + slurm.PyxisVersion + " install", "install -d -m 1777 " + slurm.EnrootLib + "/cache"} {
if !slices.Contains(h.ran, want) {
t.Errorf("did not run %q", want)
}
}
delete(h.fail, "test -f "+slurm.PyxisPlugin)
st = k.Reconcile(context.Background(), m)
if st.GetContainers() != slurm.ContainersStatus() || st.GetError() != "" {
t.Fatalf("status %v", st)
}
if string(h.files[slurm.EnrootConfPath]) != slurm.EnrootConf ||
!strings.Contains(string(h.files[slurm.EnrootMounts]), "/shared /shared none") {
t.Errorf("enroot's settings not written")
}
// Done: nothing fetched or built again.
h.ran = nil
k.Reconcile(context.Background(), m)
for _, c := range h.ran {
if strings.HasPrefix(c, "download") || strings.HasPrefix(c, "make") {
t.Errorf("ran %q again", c)
}
}
// Off: nothing said, nothing taken away.
m.Containers = false
if st := k.Reconcile(context.Background(), m); st.GetContainers() != "" {
t.Errorf("status %v", st)
}
}
+170
View File
@@ -0,0 +1,170 @@
package slurmnode
import (
"bufio"
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
// mariaDBConfDirs are where each family's MariaDB reads its settings: Debian
// and Ubuntu's, then Red Hat's.
var mariaDBConfDirs = []string{"/etc/mysql/mariadb.conf.d", "/etc/my.cnf.d"}
const mariaDBConfName = "90-llmgw-slurm.cnf"
// acctSQLPath is where the database's statements are put for the client to
// read, never on its command line: they hold the password.
const acctSQLPath = "/run/llmgw-slurm-acct.sql"
// accounting keeps slurmdbd on the controller: MariaDB on this machine
// alone, the database and its user, slurmdbd on them, and the cluster in it.
// Before slurmctld, which reports to it from its first start.
func (k *Keeper) accounting(ctx context.Context, key []byte, files map[string]string) error {
password := slurm.DbdPassword(key)
changed, err := k.mariaDBSettings(ctx)
if err != nil {
return err
}
switch {
case changed:
k.log.Info("MariaDB, for Slurm's accounting, starts on this machine alone")
if err := k.systemctl(ctx, "enable", "--now", "mariadb", "&&restart"); err != nil {
return err
}
case !k.active(ctx, "mariadb"):
if err := k.systemctl(ctx, "enable", "--now", "mariadb"); err != nil {
return err
}
}
client, err := k.mariaDBClient()
if err != nil {
return err
}
if err := k.waitFor(ctx, "MariaDB", client, "-e", "SELECT 1"); err != nil {
return err
}
// The database and its user, once each time the agent starts or the
// password is new: cheap, and a database restored from elsewhere gets
// its user back.
sum := sha256.Sum256([]byte(password))
if mark := hex.EncodeToString(sum[:]); k.acctDone != mark {
if err := k.host.WriteFile(acctSQLPath, []byte(slurm.AcctSQL(password)), 0o600, ""); err != nil {
return err
}
out, err := k.host.Run(ctx, nil, client, "-e", "source "+acctSQLPath)
_ = k.host.Remove(acctSQLPath)
if err != nil {
return fmt.Errorf("making Slurm's database: %v: %s", err, lastLine(out))
}
k.acctDone = mark
}
out, err := k.host.Run(ctx, nil, "hostname", "-s")
host := strings.TrimSpace(string(out))
if err != nil || host == "" {
return fmt.Errorf("this machine's short hostname: %v: %s", err, lastLine(out))
}
changed, err = k.write(slurm.DbdConfPath, []byte(slurm.DbdConf(host, password)), 0o600, "slurm")
if err != nil {
return err
}
switch {
case changed:
k.log.Info("slurmdbd starts again on new settings")
if err := k.systemctl(ctx, "enable", "--now", "slurmdbd", "&&restart"); err != nil {
return err
}
case !k.active(ctx, "slurmdbd"):
if err := k.systemctl(ctx, "enable", "--now", "slurmdbd"); err != nil {
return err
}
}
if err := k.waitFor(ctx, "slurmdbd", "sacctmgr", "-n", "-P", "show", "cluster", "format=cluster"); err != nil {
return err
}
name := clusterName(files["slurm.conf"])
if name == "" || k.clusterAdded == name {
return nil
}
out, err = k.host.Run(ctx, nil, "sacctmgr", "-n", "-P", "show", "cluster", "format=cluster")
if err != nil {
return fmt.Errorf("sacctmgr show cluster: %v: %s", err, lastLine(out))
}
for _, l := range strings.Split(string(out), "\n") {
if strings.TrimSpace(l) == name {
k.clusterAdded = name
return nil
}
}
k.log.Info("the cluster is added to Slurm's accounting", "cluster", name)
if out, err := k.host.Run(ctx, nil, "sacctmgr", "-i", "add", "cluster", name); err != nil {
return fmt.Errorf("sacctmgr add cluster %s: %v: %s", name, err, lastLine(out))
}
k.clusterAdded = name
return nil
}
// mariaDBSettings keeps MariaDB to this machine, when llmgw installed it;
// one that was here before llmgw keeps its own settings, and only gets the
// database and its user.
func (k *Keeper) mariaDBSettings(ctx context.Context) (bool, error) {
if _, err := k.host.ReadFile(k.mariaDBMarker()); err != nil {
return false, nil
}
for _, d := range mariaDBConfDirs {
if _, err := k.host.Run(ctx, nil, "test", "-d", d); err == nil {
return k.write(d+"/"+mariaDBConfName, []byte(slurm.MariaDBConf), 0o644, "")
}
}
return false, fmt.Errorf("MariaDB has none of %s to put its settings in", strings.Join(mariaDBConfDirs, ", "))
}
// mariaDBMarker is there when llmgw installed MariaDB on this machine.
func (k *Keeper) mariaDBMarker() string { return k.marker + "-mariadb" }
// mariaDBClient is MariaDB's command-line client, under either of its
// names.
func (k *Keeper) mariaDBClient() (string, error) {
for _, c := range []string{"mariadb", "mysql"} {
if _, err := k.host.LookPath(c); err == nil {
return c, nil
}
}
return "", errors.New("MariaDB's client (mariadb or mysql) is not installed")
}
// waitFor runs a command until it succeeds, half a minute at most.
func (k *Keeper) waitFor(ctx context.Context, what, name string, args ...string) error {
var last []byte
for range 30 {
out, err := k.host.Run(ctx, nil, name, args...)
if err == nil {
return nil
}
last = out
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(pingWait):
}
}
return fmt.Errorf("%s did not answer after starting: %s", what, lastLine(last))
}
// clusterName is ClusterName in slurm.conf.
func clusterName(conf string) string {
sc := bufio.NewScanner(strings.NewReader(conf))
for sc.Scan() {
if v, ok := strings.CutPrefix(strings.TrimSpace(sc.Text()), "ClusterName="); ok {
return strings.TrimSpace(v)
}
}
return ""
}
+150
View File
@@ -0,0 +1,150 @@
package slurmnode
import (
"context"
"fmt"
"path/filepath"
"strings"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
// What a machine says of the shared path.
const (
SharedMounted = "mounted"
SharedExported = "exported"
SharedMissing = "missing"
)
// shared keeps the shared path as want says: exported from here, mounted
// from an export, or looked for where the site mounts it. Answers how it
// is.
func (k *Keeper) shared(ctx context.Context, want *llmgwv1.Slurm) (string, error) {
sh := want.GetShared()
path := sh.GetPath()
k.unshare(ctx, path, sh.GetServe(), sh.GetNfs())
if path == "" {
return "", nil
}
if !strings.HasPrefix(path, "/") || strings.Contains(path, "..") || strings.ContainsAny(path, " \n\t\"'\\$`") {
return SharedMissing, fmt.Errorf("the leader sent a path %q", path)
}
switch {
case sh.GetServe():
if err := k.host.MkdirAll(path, 0o755, ""); err != nil {
return SharedMissing, err
}
changed, err := k.write(slurm.ExportsPath, []byte(slurm.Exports(path, sh.GetClients())), 0o644, "")
if err != nil {
return SharedMissing, err
}
if !k.active(ctx, "nfs-server") {
k.log.Info("NFS starts on this machine, exporting the Slurm cluster's shared files", "path", path)
if err := k.systemctl(ctx, "enable", "--now", "nfs-server"); err != nil {
return SharedMissing, err
}
changed = true
}
if changed {
if out, err := k.host.Run(ctx, nil, "exportfs", "-ra"); err != nil {
return SharedMissing, fmt.Errorf("exportfs -ra: %v: %s", err, lastLine(out))
}
}
return SharedExported, nil
case sh.GetNfs() != "":
nfs := sh.GetNfs()
if strings.ContainsAny(nfs, " \n\t\"'\\$`") || !strings.Contains(nfs, ":/") {
return SharedMissing, fmt.Errorf("the leader sent an export %q", nfs)
}
if err := k.host.MkdirAll(path, 0o755, ""); err != nil {
return SharedMissing, err
}
unit := slurm.MountUnitName(path)
changed, err := k.write(filepath.Join(slurm.MountUnitDir, unit), []byte(slurm.MountUnit(nfs, path)), 0o644, "")
if err != nil {
return SharedMissing, err
}
if changed {
k.log.Info("the Slurm cluster's shared files are mounted here", "from", nfs, "at", path)
if err := k.systemctl(ctx, "daemon-reload"); err != nil {
return SharedMissing, err
}
if err := k.systemctl(ctx, "enable", unit); err != nil {
return SharedMissing, err
}
// From another export: a restart mounts the new one, and fails
// while a job still has files open on the old.
if err := k.systemctl(ctx, "restart", unit); err != nil {
return SharedMissing, err
}
} else if !k.active(ctx, unit) {
if err := k.systemctl(ctx, "start", unit); err != nil {
return SharedMissing, err
}
}
}
if _, err := k.host.Run(ctx, nil, "mountpoint", "-q", path); err != nil {
if sh.GetNfs() == "" {
return SharedMissing, fmt.Errorf("%s is not mounted here: the site mounts it (slurm.shared names no export)", path)
}
return SharedMissing, fmt.Errorf("%s is not mounted from %s", path, sh.GetNfs())
}
return SharedMounted, nil
}
// unshare takes away the export or the mount llmgw kept here that want no
// longer has: path is where the files are wanted now, serve and nfs how.
func (k *Keeper) unshare(ctx context.Context, path string, serve bool, nfs string) {
if _, err := k.host.ReadFile(slurm.ExportsPath); err == nil && (!serve || path == "") {
k.log.Info("this machine no longer exports the Slurm cluster's shared files")
_ = k.host.Remove(slurm.ExportsPath)
_, _ = k.host.Run(ctx, nil, "exportfs", "-ra")
}
out, err := k.host.Run(ctx, nil, "systemctl", "list-units", "--all", "--plain", "--no-legend", "--type=mount")
if err != nil {
return
}
keep := ""
if path != "" && nfs != "" && !serve {
keep = slurm.MountUnitName(path)
}
for _, l := range strings.Split(string(out), "\n") {
f := strings.Fields(l)
if len(f) == 0 || f[0] == keep {
continue
}
file := filepath.Join(slurm.MountUnitDir, f[0])
b, err := k.host.ReadFile(file)
if err != nil || !strings.HasPrefix(string(b), "# Written by llmgw (BACKLOG P): the Slurm cluster's shared files.") {
continue
}
k.log.Info("the Slurm cluster's old shared mount is taken away", "unit", f[0])
if err := k.systemctl(ctx, "disable", "--now", f[0]); err != nil {
k.log.Warn("could not unmount the old shared files", "unit", f[0], "err", err)
continue
}
_ = k.host.Remove(file)
_ = k.systemctl(ctx, "daemon-reload")
}
}
// sharedPackages are what the shared path needs here, by the command that
// shows each is installed.
func sharedPackages(want *llmgwv1.Slurm, need map[string][2]string) {
switch sh := want.GetShared(); {
case sh.GetPath() == "":
case sh.GetServe():
need["exportfs"] = [2]string{"nfs-kernel-server", "nfs-utils"}
case sh.GetNfs() != "":
need["mount.nfs"] = [2]string{"nfs-common", "nfs-utils"}
}
}
// Serves is whether this machine exports the shared files, so NFS's port
// is kept for the cluster's networks.
func (k *Keeper) Serves() bool {
k.mu.Lock()
defer k.mu.Unlock()
return k.want.GetShared().GetServe() && k.want.GetShared().GetPath() != ""
}
+171
View File
@@ -0,0 +1,171 @@
package slurmnode
import (
"context"
"slices"
"strings"
"testing"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
func ayse() *llmgwv1.SlurmUser {
return &llmgwv1.SlurmUser{Login: "ayse", Uid: 200000, Name: "Ayşe Y.", Home: "/shared/home/ayse",
SshKeys: []string{"ssh-ed25519 AAAA ayse@laptop"}}
}
func TestTheServerExportsAndMakesUsersWithHomes(t *testing.T) {
h := newFake()
h.active["ssh"] = true
inst := &fakeInst{host: h}
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
m := nodeMsg(false)
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Serve: true, Clients: []string{"10.0.0.22", "10.0.0.21"}}
m.ManageUsers, m.MakeHomes, m.Users = true, true, []*llmgwv1.SlurmUser{ayse()}
h.fail["test -d /shared/home/ayse"] = true
h.files[slurm.PAMSSHD] = []byte(pamStack)
st := k.Reconcile(context.Background(), m)
if st.GetError() != "" {
t.Fatal(st.GetError())
}
if st.GetShared() != SharedExported || st.GetUsers() != 1 {
t.Fatalf("status %v", st)
}
if !slices.Contains(inst.got["deb"], "nfs-kernel-server") {
t.Errorf("packages %v", inst.got)
}
if e := string(h.files[slurm.ExportsPath]); !strings.Contains(e, `"/shared" 10.0.0.21(`) {
t.Errorf("exports:\n%s", e)
}
for _, want := range []string{"systemctl enable --now nfs-server", "exportfs -ra", "groupadd -g 200000 ayse",
"useradd -u 200000 -g 200000 -M -d /shared/home/ayse -s /bin/bash -c Ayşe Y. ayse",
"install -d -m 0700 -o 200000 -g 200000 /shared/home/ayse", "sshd -t", "systemctl reload ssh"} {
if !slices.Contains(h.ran, want) {
t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
}
}
if string(h.files[slurm.AuthorizedKeysDir+"/ayse"]) != "ssh-ed25519 AAAA ayse@laptop\n" ||
string(h.files[k.usersMarker()]) != "ayse\n" || h.files[slurm.SSHDropIn] == nil {
t.Errorf("files: keys %q, marker %q", h.files[slurm.AuthorizedKeysDir+"/ayse"], h.files[k.usersMarker()])
}
// Again, made: nothing is run again.
h.outputs["getent passwd ayse"] = "ayse:x:200000:200000:Ayşe Y.:/shared/home/ayse:/bin/bash\n"
delete(h.fail, "test -d /shared/home/ayse")
h.ran = nil
k.Reconcile(context.Background(), m)
for _, c := range h.ran {
if strings.HasPrefix(c, "useradd") || strings.HasPrefix(c, "usermod") || strings.HasPrefix(c, "exportfs") ||
strings.HasPrefix(c, "install") || c == "systemctl reload ssh" {
t.Errorf("ran %q again", c)
}
}
// Gone from the accounts: taken away, its home left.
m.Users = nil
k.Reconcile(context.Background(), m)
if !slices.Contains(h.ran, "userdel ayse") || h.files[k.usersMarker()] != nil || h.files[slurm.SSHDropIn] != nil ||
h.files[slurm.AuthorizedKeysDir+"/ayse"] != nil {
t.Errorf("not taken away:\n%s", strings.Join(h.ran, "\n"))
}
if slices.ContainsFunc(h.ran, func(c string) bool { return strings.HasPrefix(c, "rm") || strings.Contains(c, "userdel -r") }) {
t.Error("a home was removed")
}
}
func TestAClientMountsAndLeavesOthersUsersAlone(t *testing.T) {
h := newFake()
inst := &fakeInst{host: h}
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
m := nodeMsg(false)
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Nfs: "10.0.0.11:/shared"}
bob := &llmgwv1.SlurmUser{Login: "bob", Uid: 200001, Home: "/shared/home/bob"}
m.ManageUsers, m.Users = true, []*llmgwv1.SlurmUser{ayse(), bob}
// The site's own ayse, with its own UID.
h.outputs["getent passwd ayse"] = "ayse:x:1001:1001::/home/ayse:/bin/bash\n"
st := k.Reconcile(context.Background(), m)
if st.GetShared() != SharedMounted || st.GetUsers() != 1 {
t.Fatalf("status %v", st)
}
if !strings.Contains(st.GetError(), "ayse is already a user here with UID 1001") {
t.Errorf("error %q", st.GetError())
}
if u := string(h.files["/etc/systemd/system/shared.mount"]); !strings.Contains(u, "What=10.0.0.11:/shared") {
t.Fatalf("unit:\n%s", u)
}
for _, want := range []string{"systemctl enable shared.mount", "systemctl restart shared.mount", "mountpoint -q /shared",
"useradd -u 200001 -g 200001 -M -d /shared/home/bob -s /bin/bash -c bob"} {
if !slices.Contains(h.ran, want) {
t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
}
}
if slices.ContainsFunc(h.ran, func(c string) bool { return strings.Contains(c, "ayse") && !strings.HasPrefix(c, "getent") }) ||
slices.ContainsFunc(h.ran, func(c string) bool { return strings.HasPrefix(c, "install -d") }) {
t.Errorf("touched the site's ayse, or made a home:\n%s", strings.Join(h.ran, "\n"))
}
if string(h.files[k.usersMarker()]) != "bob\n" {
t.Errorf("marker %q", h.files[k.usersMarker()])
}
// Not mounted: said.
h.fail["mountpoint -q /shared"] = true
if st := k.Reconcile(context.Background(), m); st.GetShared() != SharedMissing ||
!strings.Contains(st.GetError(), "not mounted from 10.0.0.11:/shared") {
t.Errorf("status %v", st)
}
// No shared path any more: the mount is taken away.
h.outputs["systemctl list-units --all --plain --no-legend --type=mount"] = "shared.mount loaded active mounted llmgw\n"
m.Shared = nil
k.Reconcile(context.Background(), m)
if !slices.Contains(h.ran, "systemctl disable --now shared.mount") || h.files["/etc/systemd/system/shared.mount"] != nil {
t.Errorf("mount kept:\n%s", strings.Join(h.ran, "\n"))
}
}
const pamStack = "@include common-auth\naccount required pam_nologin.so\n@include common-account\n@include common-session\n"
func TestUsersReachANodeOnlyThroughTheirJobs(t *testing.T) {
h := newFake()
inst := &fakeInst{host: h}
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
h.files[slurm.PAMSSHD] = []byte(pamStack)
// The module is not there until its package is installed.
for _, d := range slurm.ModuleDirs {
h.fail["test -f "+d+"/"+slurm.AdoptModule] = true
}
m := nodeMsg(true)
m.ManageUsers, m.Users = true, []*llmgwv1.SlurmUser{ayse()}
k.Reconcile(context.Background(), m)
if !slices.Contains(inst.got["deb"], "libpam-slurm-adopt") {
t.Fatalf("not installed: %v", inst.got)
}
if got := string(h.files[slurm.PAMSSHD]); got != pamStack {
t.Fatalf("lines put in with the module still missing:\n%s", got)
}
delete(h.fail, "test -f "+slurm.ModuleDirs[0]+"/"+slurm.AdoptModule)
st := k.Reconcile(context.Background(), m)
got := string(h.files[slurm.PAMSSHD])
if !strings.Contains(got, "pam_listfile.so item=user sense=allow file="+k.usersMarker()) ||
!strings.Contains(got, "account required "+slurm.AdoptModule) {
t.Fatalf("not held to their jobs (%s):\n%s", st.GetError(), got)
}
// The controller is where they log in: not held there.
m.Controller = true
m.Files = map[string]string{"slurm.conf": "x"}
k.Reconcile(context.Background(), m)
if string(h.files[slurm.PAMSSHD]) != pamStack {
t.Errorf("held on the controller:\n%s", h.files[slurm.PAMSSHD])
}
// No users of llmgw's any more: the lines go.
m.Controller, m.Files = false, nil
k.Reconcile(context.Background(), m)
m.Users = nil
k.Reconcile(context.Background(), m)
if string(h.files[slurm.PAMSSHD]) != pamStack {
t.Errorf("lines left with no users:\n%s", h.files[slurm.PAMSSHD])
}
}
+139 -4
View File
@@ -13,8 +13,10 @@ import (
"encoding/hex"
"errors"
"fmt"
"io"
"io/fs"
"log/slog"
"net/http"
"os"
"os/exec"
"os/user"
@@ -43,6 +45,9 @@ type Host interface {
// MkdirAll makes path, owned by owner.
MkdirAll(path string, mode fs.FileMode, owner string) error
Remove(path string) error
// Download fetches url to path, kept only if its SHA-256 is sha; a file
// already there with it is not fetched again.
Download(ctx context.Context, url, path, sha string) error
}
// Installer installs packages by family, "deb" and "rpm".
@@ -77,6 +82,10 @@ type Keeper struct {
// minute's check does not look again.
installed map[string]bool
files string
// acctDone is the digest of the password the database's user was last
// given; clusterAdded the cluster last found in the accounting.
acctDone string
clusterAdded string
}
// New is a Keeper; marker is a file it keeps while it runs Slurm here, so
@@ -206,13 +215,72 @@ func (k *Keeper) Reconcile(ctx context.Context, want *llmgwv1.Slurm) *llmgwv1.Sl
if err := k.munge(ctx, want.GetMungeKey()); err != nil {
return fail("munge", err)
}
// The shared files, then the users whose homes are on them. Neither
// keeps Slurm from running: what goes wrong is said, and Slurm kept.
var shErr, usErr error
st.Shared, shErr = k.shared(ctx, want)
if shErr != nil {
k.log.Warn("keeping the Slurm cluster's shared files on this node", "err", shErr)
}
if shErr == nil || !want.GetMakeHomes() || want.GetShared().GetPath() == "" {
st.Users, usErr = k.users(ctx, want)
} else {
// Homes made now would be made beside the shared files, not on them.
usErr = errors.New("waiting for the shared files, where the homes are")
}
if usErr != nil {
k.log.Warn("keeping the Slurm users on this node", "err", usErr)
}
// Jobs in containers; Slurm runs without them, and is kept anyway.
var ctErr error
if st.Containers, ctErr = k.containers(ctx, want); ctErr != nil {
k.log.Warn("setting up jobs in containers on this node", "err", ctErr)
}
defer func() {
switch {
case st.Error != "":
case shErr != nil:
st.Error = "shared files: " + shErr.Error()
case usErr != nil:
st.Error = "users: " + usErr.Error()
case ctErr != nil:
st.Error = "containers: " + ctErr.Error()
}
}()
if want.GetController() {
// Accounting first: slurmctld reports to it from its first start.
// Without it slurmctld still runs jobs, a pass of backfill late
// each; so a failure here is said, and slurmctld is kept anyway.
var acctErr error
if acctErr = k.accounting(ctx, want.GetMungeKey(), want.GetFiles()); acctErr != nil {
k.log.Warn("keeping Slurm's accounting on this node", "err", acctErr)
}
st.SlurmdbdRunning = k.active(ctx, "slurmdbd")
defer func() {
if acctErr != nil && st.Error == "" {
st.Error = "slurmdbd: " + acctErr.Error()
}
}()
sum, err := k.controller(ctx, want.GetFiles())
st.FilesSha256 = sum
if err != nil {
return fail("slurmctld", err)
}
st.SlurmctldRunning = k.active(ctx, "slurmctld")
// The accounts, and the jobs that ended, need the accounting.
if st.SlurmdbdRunning && st.SlurmctldRunning {
if err := k.accounts(ctx, want); err != nil {
k.log.Warn("keeping Slurm's accounts", "err", err)
if acctErr == nil {
acctErr = fmt.Errorf("accounts: %w", err)
}
}
if ended, err := k.ended(ctx); err != nil {
k.log.Warn("reading the jobs that ended", "err", err)
} else {
st.Ended = ended
}
}
}
if want.GetNode() {
if err := k.node(ctx, want, st); err != nil {
@@ -229,10 +297,13 @@ func (k *Keeper) ensurePackages(ctx context.Context, want *llmgwv1.Slurm) error
}
if want.GetController() {
need["slurmctld"] = [2]string{"slurmctld", "slurm-slurmctld"}
need["slurmdbd"] = [2]string{"slurmdbd", "slurm-slurmdbd"}
need["mariadbd"] = [2]string{"mariadb-server", "mariadb-server"}
}
if want.GetNode() {
need["slurmd"] = [2]string{"slurmd", "slurm-slurmd"}
}
sharedPackages(want, need)
pkgs := map[string][]string{}
for bin, p := range need {
if k.installed[bin] {
@@ -257,6 +328,14 @@ func (k *Keeper) ensurePackages(ctx context.Context, want *llmgwv1.Slurm) error
if err := k.inst.InstallPackages(ctx, pkgs); err != nil {
return err
}
if slices.Contains(pkgs["deb"], "mariadb-server") {
// Installed by llmgw: its settings are llmgw's to keep. One that was
// here before is another's, and is left as it is.
if err := k.host.WriteFile(k.mariaDBMarker(), []byte("llmgw installed MariaDB here, for Slurm's accounting\n"),
0o644, ""); err != nil {
return err
}
}
for bin := range need {
if _, err := k.host.LookPath(bin); err != nil {
return fmt.Errorf("%s is still missing after installing %s", bin, strings.Join(pkgs["deb"], ", "))
@@ -483,14 +562,20 @@ func (k *Keeper) stop(ctx context.Context) {
if _, err := k.host.ReadFile(k.marker); err != nil {
return
}
k.log.Info("this node has no part in Slurm any more: slurmd and slurmctld stopped; the packages stay")
for _, u := range []string{"slurmd", "slurmctld"} {
k.log.Info("this node has no part in Slurm any more: slurmd, slurmctld and slurmdbd stopped; the packages and MariaDB stay")
// MariaDB stays: it may have been the machine's before, and holds the
// jobs' records.
for _, u := range []string{"slurmd", "slurmctld", "slurmdbd"} {
_ = k.systemctl(ctx, "disable", "--now", u)
}
_ = k.host.Remove(DropIn)
_ = k.systemctl(ctx, "daemon-reload")
k.unshare(ctx, "", false, "")
if err := k.removeUsers(ctx, k.madeUsers(), nil); err != nil {
k.log.Warn("could not take away every Slurm user llmgw made", "err", err)
}
_ = k.host.Remove(k.marker)
k.files = ""
k.files, k.acctDone, k.clusterAdded = "", "", ""
}
// write writes a file when it differs, saying whether it did.
@@ -551,7 +636,8 @@ func (Local) LookPath(name string) (string, error) {
if p, err := exec.LookPath(name); err == nil {
return p, nil
}
for _, d := range []string{"/usr/sbin", "/sbin", "/usr/local/sbin"} {
// /usr/libexec: Red Hat's mariadbd.
for _, d := range []string{"/usr/sbin", "/sbin", "/usr/local/sbin", "/usr/libexec"} {
p := filepath.Join(d, name)
if fi, err := os.Stat(p); err == nil && !fi.IsDir() {
return p, nil
@@ -582,6 +668,55 @@ func (Local) WriteFile(path string, b []byte, mode fs.FileMode, owner string) er
return os.Rename(tmp, path)
}
// Download fetches url to path through a temporary file beside it, kept
// only when its SHA-256 is sha.
func (Local) Download(ctx context.Context, url, path, sha string) error {
if f, err := os.Open(path); err == nil { //nolint:gosec // our paths
h := sha256.New()
_, cerr := io.Copy(h, f)
_ = f.Close()
if cerr == nil && hex.EncodeToString(h.Sum(nil)) == sha {
return nil
}
}
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { //nolint:gosec // system directories
return err
}
ctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return fmt.Errorf("fetching %s: %w", url, err)
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("fetching %s: %s", url, resp.Status)
}
tmp := path + ".llmgw-new"
f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o644) //nolint:gosec // our paths
if err != nil {
return err
}
h := sha256.New()
_, err = io.Copy(io.MultiWriter(f, h), io.LimitReader(resp.Body, 1<<30))
if cerr := f.Close(); err == nil {
err = cerr
}
if err != nil {
_ = os.Remove(tmp)
return fmt.Errorf("fetching %s: %w", url, err)
}
if got := hex.EncodeToString(h.Sum(nil)); got != sha {
_ = os.Remove(tmp)
return fmt.Errorf("%s has SHA-256 %s, not the %s llmgw was built with: not used", url, got, sha)
}
return os.Rename(tmp, path)
}
// MkdirAll makes a directory owned by owner.
func (Local) MkdirAll(path string, mode fs.FileMode, owner string) error {
if err := os.MkdirAll(path, mode); err != nil {
+62 -3
View File
@@ -32,6 +32,7 @@ func newFake() *fakeHost {
"slurmctld -V": "slurm-wlm 23.11.4\n",
"scontrol ping": "Slurmctld(primary) at node06 is UP\n",
"slurmd -C": "NodeName=cuda01 CPUs=16 RealMemory=64000\nUpTime=1-00:00:00\n",
"hostname -s": "node06\n",
}, fail: map[string]bool{}, state: "IDLE+DRAIN", reason: slurm.ReasonNew}
}
@@ -59,7 +60,7 @@ func (f *fakeHost) Run(_ context.Context, env []string, name string, args ...str
case "disable":
f.active[u] = false
}
return nil, nil
return []byte(f.outputs[c]), nil
case strings.HasPrefix(c, "scontrol show node"):
return []byte("NodeName=cuda01 Arch=x86_64 State=" + f.state + " Partitions=gpu Reason=" + f.reason + " [root@now]\n"), nil
case strings.HasPrefix(c, "scontrol update"):
@@ -101,6 +102,16 @@ func (f *fakeHost) MkdirAll(string, fs.FileMode, string) error { return nil }
func (f *fakeHost) Remove(path string) error { delete(f.files, path); return nil }
func (f *fakeHost) Download(_ context.Context, url, path, sha string) error {
c := "download " + url + " " + path + " " + sha
f.ran = append(f.ran, c)
if f.fail[c] || f.fail["download "+url] {
return errors.New("fetch failed")
}
f.files[path] = []byte("fetched " + url)
return nil
}
func (f *fakeHost) count(prefix string) int {
n := 0
for _, c := range f.ran {
@@ -118,7 +129,7 @@ type fakeInst struct {
func (i *fakeInst) InstallPackages(_ context.Context, by map[string][]string) error {
i.got = by
for _, b := range []string{"munged", "scontrol", "slurmd", "slurmctld"} {
for _, b := range []string{"munged", "scontrol", "slurmd", "slurmctld", "slurmdbd", "mariadbd", "mariadb", "exportfs", "mount.nfs"} {
i.host.bins[b] = true
}
return nil
@@ -237,6 +248,33 @@ func TestController(t *testing.T) {
if !st.GetSlurmctldRunning() || len(st.GetFilesSha256()) != 64 {
t.Fatalf("status %v", st)
}
// Accounting: MariaDB on this machine alone, the database, slurmdbd on
// it, the cluster added, all before slurmctld.
if !slices.Contains(inst.got["deb"], "slurmdbd") || !slices.Contains(inst.got["deb"], "mariadb-server") {
t.Fatalf("installed %v", inst.got)
}
if !strings.Contains(string(h.files["/etc/mysql/mariadb.conf.d/90-llmgw-slurm.cnf"]), "bind-address=127.0.0.1") {
t.Fatal("MariaDB, installed by llmgw, not kept to this machine")
}
dbd := string(h.files[slurm.DbdConfPath])
if !strings.Contains(dbd, "DbdHost=node06\n") || !strings.Contains(dbd, "StoragePass="+slurm.DbdPassword(key)+"\n") ||
h.owners[slurm.DbdConfPath] != "slurm" {
t.Fatalf("slurmdbd.conf for %q:\n%s", h.owners[slurm.DbdConfPath], dbd)
}
if _, ok := h.files[acctSQLPath]; ok {
t.Fatal("the database's statements, password and all, were left on disk")
}
if strings.Contains(strings.Join(h.ran, "\n"), slurm.DbdPassword(key)) {
t.Fatal("the password was on a command line")
}
order := strings.Join(h.ran, "\n")
if !strings.Contains(order, "sacctmgr -i add cluster llmgw") ||
strings.Index(order, "slurmdbd") > strings.Index(order, "systemctl restart slurmctld") {
t.Fatalf("accounting not before slurmctld: %v", h.ran)
}
if !st.GetSlurmdbdRunning() {
t.Fatalf("status %v", st)
}
sum := st.GetFilesSha256()
h.ran = nil
@@ -279,7 +317,7 @@ func TestStop(t *testing.T) {
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{}); st != nil {
t.Fatalf("status %v for no part", st)
}
if h.count("systemctl disable --now slurmd") != 1 {
if !slices.Contains(h.ran, "systemctl disable --now slurmd") {
t.Fatalf("slurmd not stopped: %v", h.ran)
}
if _, ok := h.files[DropIn]; ok {
@@ -306,3 +344,24 @@ func TestBadNames(t *testing.T) {
t.Fatalf("status %v", st)
}
}
func TestAMariaDBThatWasThereKeepsItsSettings(t *testing.T) {
h := newFake()
for _, b := range []string{"munged", "scontrol", "slurmctld", "slurmdbd", "mariadbd", "mariadb", "exportfs", "mount.nfs"} {
h.bins[b] = true
}
k := New(h, &fakeInst{host: h}, "/m", nil)
st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Controller: true, MungeKey: key,
Files: map[string]string{"slurm.conf": "ClusterName=llmgw\n"}})
if st.GetError() != "" {
t.Fatal(st.GetError())
}
for p := range h.files {
if strings.HasSuffix(p, ".cnf") {
t.Fatalf("a MariaDB llmgw did not install had its settings changed: %s", p)
}
}
if h.count("mariadb -e source") != 1 || !st.GetSlurmdbdRunning() {
t.Fatalf("the database was not made in the MariaDB that was there: %v", h.ran)
}
}
+319
View File
@@ -0,0 +1,319 @@
package slurmnode
import (
"context"
"errors"
"fmt"
"path/filepath"
"slices"
"strconv"
"strings"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
// usersMarker lists, a line each, the logins llmgw made here: the ones it
// may change and take away. A user that was here before is never touched.
func (k *Keeper) usersMarker() string { return k.marker + "-users" }
// users keeps the Linux users want has, the same login and UID as on every
// other machine, their keys where sshd reads them, and their homes when
// this machine makes them; and takes away those llmgw made that are gone.
// Answers how many it has. A user that cannot be made is said, the others
// are made anyway.
func (k *Keeper) users(ctx context.Context, want *llmgwv1.Slurm) (int32, error) {
made := k.madeUsers()
if !want.GetManageUsers() {
return 0, k.removeUsers(ctx, made, nil)
}
var errs []error
var keep []string
n := int32(0)
for _, u := range want.GetUsers() {
err := k.user(ctx, u, slices.Contains(made, u.GetLogin()), want.GetMakeHomes())
if errors.Is(err, errNotMade) {
errs = append(errs, err)
continue
}
keep = append(keep, u.GetLogin())
if err != nil {
errs = append(errs, err)
continue
}
n++
}
if err := k.keepMade(keep); err != nil {
return n, err
}
if err := k.removeUsers(ctx, made, keep); err != nil {
errs = append(errs, err)
}
if err := k.sshConf(ctx, len(keep) > 0); err != nil {
errs = append(errs, err)
}
// The controller is where they log in and submit from; a node only
// while a job of theirs runs there.
if err := k.nodeLogin(ctx, len(keep) > 0 && want.GetNode() && !want.GetController()); err != nil {
errs = append(errs, err)
}
return n, errors.Join(errs...)
}
// nodeLogin holds the users llmgw made here to their jobs when they log in
// (pam_slurm_adopt), or no longer. Only they are: root and the site's
// administrators log in as before, so a mistake here never locks out the
// people who would put it right.
func (k *Keeper) nodeLogin(ctx context.Context, on bool) error {
old, err := k.host.ReadFile(slurm.PAMSSHD)
if err != nil {
if !on {
return nil
}
return fmt.Errorf("reading %s: %w", slurm.PAMSSHD, err)
}
if on {
if err := k.adoptModule(ctx); err != nil {
return err
}
}
read := func(name string) (string, error) {
b, err := k.host.ReadFile(filepath.Join(filepath.Dir(slurm.PAMSSHD), name))
return string(b), err
}
stack, contained, err := slurm.PAMAdopt(string(old), k.usersMarker(), read, on)
if err != nil {
return err
}
if stack == string(old) {
return nil
}
if on {
k.log.Info("llmgw's Slurm users log in to this node only while a job of theirs runs here",
"in_the_job", contained)
if !contained {
k.log.Warn("pam_systemd could not be skipped for them safely here: their ssh sessions are not held " +
"to their job's cores and cards, only to having one")
}
} else {
k.log.Info("llmgw's Slurm users are no longer held to their jobs when they log in here")
}
// Read by sshd for each login: nothing to reload.
_, err = k.write(slurm.PAMSSHD, []byte(stack), 0o644, "")
return err
}
// adoptModule is there, or is installed.
func (k *Keeper) adoptModule(ctx context.Context) error {
if k.installed[slurm.AdoptModule] {
return nil
}
find := func() bool {
for _, d := range slurm.ModuleDirs {
if _, err := k.host.Run(ctx, nil, "test", "-f", filepath.Join(d, slurm.AdoptModule)); err == nil {
return true
}
}
return false
}
if !find() {
if k.inst == nil {
return fmt.Errorf("%s missing, and this agent installs nothing", slurm.AdoptModule)
}
k.log.Info("installing pam_slurm_adopt, which holds llmgw's Slurm users to their jobs here")
if err := k.inst.InstallPackages(ctx, slurm.AdoptPackages); err != nil {
return err
}
if !find() {
return fmt.Errorf("%s is still missing after installing %s", slurm.AdoptModule,
strings.Join(slurm.AdoptPackages["deb"], ", "))
}
}
k.installed[slurm.AdoptModule] = true
return nil
}
// errNotMade is a user llmgw could not make here, nor took for its own.
var errNotMade = errors.New("not made")
type notMade struct{ msg string }
func (e notMade) Error() string { return e.msg }
func (e notMade) Is(target error) bool { return target == errNotMade }
// user keeps one user; ours is whether llmgw made it here before.
func (k *Keeper) user(ctx context.Context, u *llmgwv1.SlurmUser, ours, makeHome bool) error {
login, uid, home := u.GetLogin(), u.GetUid(), u.GetHome()
if !slurm.ValidLogin(login) || uid < slurm.MinUID || !filepath.IsAbs(home) || strings.Contains(home, "..") {
return notMade{fmt.Sprintf("the leader sent a user %q, UID %d, home %q", login, uid, home)}
}
id := strconv.FormatUint(uint64(uid), 10)
gecos := strings.Map(func(r rune) rune {
if r == ':' || r == ',' || r == '\n' || r < ' ' {
return -1
}
return r
}, u.GetName())
byName, okName := k.passwd(ctx, login)
byID, okID := k.passwd(ctx, id)
switch {
case okName && byName[2] != id:
return notMade{fmt.Sprintf("%s is already a user here with UID %s, not llmgw's %s", login, byName[2], id)}
case !okName && okID:
return notMade{fmt.Sprintf("UID %s is already %s's here, so %s is not made", id, byID[0], login)}
case !okName:
if _, ok := k.group(ctx, login); !ok {
if out, err := k.host.Run(ctx, nil, "groupadd", "-g", id, login); err != nil {
return notMade{fmt.Sprintf("groupadd %s: %v: %s", login, err, lastLine(out))}
}
}
k.log.Info("a Slurm user is made here, as llmgw's accounts say", "login", login, "uid", uid)
if out, err := k.host.Run(ctx, nil, "useradd", "-u", id, "-g", id, "-M", "-d", home, "-s", "/bin/bash",
"-c", gecos, login); err != nil {
return notMade{fmt.Sprintf("useradd %s: %v: %s", login, err, lastLine(out))}
}
case ours && (byName[4] != gecos || byName[5] != home):
if out, err := k.host.Run(ctx, nil, "usermod", "-c", gecos, "-d", home, login); err != nil {
return fmt.Errorf("usermod %s: %v: %s", login, err, lastLine(out))
}
}
keys := strings.Join(u.GetSshKeys(), "\n")
file := filepath.Join(slurm.AuthorizedKeysDir, login)
if keys == "" {
_ = k.host.Remove(file)
} else if _, err := k.write(file, []byte(keys+"\n"), 0o644, ""); err != nil {
return err
}
if makeHome {
return k.home(ctx, login, id, home)
}
return nil
}
// home makes a user's home directory, from /etc/skel, when it is not there.
func (k *Keeper) home(ctx context.Context, login, id, home string) error {
if _, err := k.host.Run(ctx, nil, "test", "-d", home); err == nil {
return nil
}
if err := k.host.MkdirAll(filepath.Dir(home), 0o755, ""); err != nil {
return err
}
k.log.Info("a Slurm user's home is made", "login", login, "home", home)
for _, c := range [][]string{
{"install", "-d", "-m", "0700", "-o", id, "-g", id, home},
{"cp", "-rT", "/etc/skel", home},
{"chown", "-R", id + ":" + id, home},
} {
if out, err := k.host.Run(ctx, nil, c[0], c[1:]...); err != nil {
return fmt.Errorf("making %s's home: %s: %v: %s", login, c[0], err, lastLine(out))
}
}
return nil
}
// removeUsers takes away the users llmgw made here that are not in keep.
// Their homes and files stay; their UID is never given to another.
func (k *Keeper) removeUsers(ctx context.Context, made, keep []string) error {
var errs []error
left := slices.Clone(keep)
for _, login := range made {
if slices.Contains(keep, login) {
continue
}
k.log.Info("a Slurm user llmgw made is taken away; its home stays", "login", login)
_ = k.host.Remove(filepath.Join(slurm.AuthorizedKeysDir, login))
if _, ok := k.passwd(ctx, login); ok {
if out, err := k.host.Run(ctx, nil, "userdel", login); err != nil {
// Still logged in, or running something: tried again.
errs = append(errs, fmt.Errorf("userdel %s: %v: %s", login, err, lastLine(out)))
left = append(left, login)
continue
}
}
}
if !slices.Equal(left, made) {
if err := k.keepMade(left); err != nil {
errs = append(errs, err)
}
}
if len(left) == 0 {
if err := k.sshConf(ctx, false); err != nil {
errs = append(errs, err)
}
if err := k.nodeLogin(ctx, false); err != nil {
errs = append(errs, err)
}
}
return errors.Join(errs...)
}
func (k *Keeper) madeUsers() []string {
b, err := k.host.ReadFile(k.usersMarker())
if err != nil {
return nil
}
return strings.Fields(string(b))
}
func (k *Keeper) keepMade(logins []string) error {
s := slices.Clone(logins)
slices.Sort(s)
s = slices.Compact(s)
if len(s) == 0 {
return k.host.Remove(k.usersMarker())
}
_, err := k.write(k.usersMarker(), []byte(strings.Join(s, "\n")+"\n"), 0o644, "")
return err
}
// sshConf has sshd read the users' keys from AuthorizedKeysDir, or no
// longer.
func (k *Keeper) sshConf(ctx context.Context, on bool) error {
if !on {
if _, err := k.host.ReadFile(slurm.SSHDropIn); err != nil {
return nil
}
_ = k.host.Remove(slurm.SSHDropIn)
return k.reloadSSH(ctx)
}
changed, err := k.write(slurm.SSHDropIn, []byte(slurm.SSHConf), 0o644, "")
if err != nil || !changed {
return err
}
if out, err := k.host.Run(ctx, nil, "sshd", "-t"); err != nil {
_ = k.host.Remove(slurm.SSHDropIn)
return fmt.Errorf("sshd refuses llmgw's setting for the users' keys, so it is not kept: %v: %s", err, lastLine(out))
}
k.log.Info("sshd reads the Slurm users' keys from llmgw too", "dir", slurm.AuthorizedKeysDir)
return k.reloadSSH(ctx)
}
// reloadSSH has sshd read its settings again, under Debian's name for it
// or Red Hat's; one that is not running reads them when it starts.
func (k *Keeper) reloadSSH(ctx context.Context) error {
for _, u := range []string{"ssh", "sshd"} {
if k.active(ctx, u) {
return k.systemctl(ctx, "reload", u)
}
}
return nil
}
// passwd is getent's line for a user, by login or UID, in its fields.
func (k *Keeper) passwd(ctx context.Context, key string) ([]string, bool) {
out, err := k.host.Run(ctx, nil, "getent", "passwd", key)
f := strings.Split(strings.TrimSpace(string(out)), ":")
if err != nil || len(f) < 7 {
return nil, false
}
return f, true
}
func (k *Keeper) group(ctx context.Context, key string) ([]string, bool) {
out, err := k.host.Run(ctx, nil, "getent", "group", key)
f := strings.Split(strings.TrimSpace(string(out)), ":")
if err != nil || len(f) < 3 {
return nil, false
}
return f, true
}
+1
View File
@@ -190,6 +190,7 @@ func newCtlCmd() *cobra.Command {
newKBCmd(client, &asJSON),
newTrainCmd(client, &asJSON),
newSlurmCmd(client, &asJSON),
newPricesCmd(client, &asJSON),
newConfigCmd(client, &asJSON),
newAuditCmd(client, &asJSON),
newNotifyCmd(client, &asJSON),
+121
View File
@@ -0,0 +1,121 @@
package cli
import (
"encoding/json"
"fmt"
"net/url"
"strconv"
"text/tabwriter"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
)
func newPricesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
cmd := &cobra.Command{
Use: "prices",
Short: "What each model's tokens cost, from what the cluster costs",
}
var days int
var apply bool
var note string
suggest := &cobra.Command{
Use: "suggest",
Short: "Each model's price from the GPU- and core-hours it held, over the tokens it served",
Long: "Set auth.gpu_hour_cost first: what an hour of one GPU costs, in the units budgets are in (the\n" +
"hardware written off over its years, power and room, the people who run it, over the GPUs and\n" +
"the hours of a month); auth.core_hour_cost likewise for models on CPUs. Each model's cost is\n" +
"the hours its replicas held, idle ones too, kept by the hour on the leader; spread over its\n" +
"tokens, an output token counted as four input ones and a cached prompt token as a tenth of one,\n" +
"it is its price. A model held and not used shows what that cost. --apply writes the prices\n" +
"into auth.pricing, one settings version, as `ctl config set` does.",
Example: " llmgw ctl config set auth.gpu_hour_cost 1.35\n" +
" llmgw ctl prices suggest --days 30\n" +
" llmgw ctl prices suggest --apply --note \"November's costs\"",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
var s adminapi.PriceSuggestions
if err := client().get(cmd.Context(), "/api/v1/prices/suggest?days="+url.QueryEscape(strconv.Itoa(days)), &s); err != nil {
return err
}
w := cmd.OutOrStdout()
if *asJSON {
return json.NewEncoder(w).Encode(s)
}
fmt.Fprintf(w, "cost: %g a GPU-hour, %g a core-hour; the last %d days", s.GPUHourCost, s.CoreHourCost, s.Days)
if !s.RecordedFrom.IsZero() && s.RecordedFrom.After(s.Since) {
fmt.Fprintf(w, " (held kept only since %s)", s.RecordedFrom.Local().Format("2006-01-02 15:04"))
}
fmt.Fprintln(w)
fmt.Fprintln(w, "prices a million tokens: input / output / cached input, in budget units")
tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
fmt.Fprintln(tw, "MODEL\tGPU-H\tCORE-H\tCOST\tTOKENS IN/OUT\tSUGGESTED\tNOW\tNOTE")
var changed []adminapi.PriceSuggestion
for _, m := range s.Models {
sug := "-"
if !m.Price.Zero() {
sug = perMillion(m.Price.PromptPer1k, m.Price.CompletionPer1k, m.Price.CachedPer1k)
if m.Price.PromptPer1k != m.Current.PromptPer1k || m.Price.CompletionPer1k != m.Current.CompletionPer1k ||
m.Price.CachedPer1k != m.Current.CachedPer1k {
changed = append(changed, m)
}
}
fmt.Fprintf(tw, "%s\t%.1f\t%.1f\t%.2f\t%s/%s\t%s\t%s\t%s\n", m.Model, m.GPUHours, m.CoreHours, m.Cost,
count(m.Prompt), count(m.Completion), sug,
perMillion(m.Current.PromptPer1k, m.Current.CompletionPer1k, m.Current.CachedPer1k), m.Note)
}
if err := tw.Flush(); err != nil {
return err
}
if !apply {
if len(changed) > 0 {
fmt.Fprintf(w, "%d model(s) priced otherwise now; --apply writes the suggestions\n", len(changed))
}
return nil
}
if len(changed) == 0 {
fmt.Fprintln(w, "nothing to change")
return nil
}
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
for _, m := range changed {
for k, v := range map[string]int64{"prompt_per_1k": m.Price.PromptPer1k,
"completion_per_1k": m.Price.CompletionPer1k, "cached_per_1k": m.Price.CachedPer1k} {
if err := doc.SetYAML([]string{"auth", "pricing", m.Model, k}, strconv.FormatInt(v, 10)); err != nil {
return err
}
}
}
return nil
})
},
}
suggest.Flags().IntVar(&days, "days", 30, "the window, in days")
suggest.Flags().BoolVar(&apply, "apply", false, "write the suggested prices into auth.pricing")
suggest.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
cmd.AddCommand(suggest)
return cmd
}
// perMillion is micro-units a thousand tokens as budget units a million.
func perMillion(in, out, cached int64) string {
if in == 0 && out == 0 && cached == 0 {
return "-"
}
f := func(v int64) string { return strconv.FormatFloat(float64(v)/1000, 'f', -1, 64) }
return f(in) + " / " + f(out) + " / " + f(cached)
}
func count(n int64) string {
switch {
case n >= 1e9:
return fmt.Sprintf("%.1fB", float64(n)/1e9)
case n >= 1e6:
return fmt.Sprintf("%.1fM", float64(n)/1e6)
case n >= 1e3:
return fmt.Sprintf("%.1fk", float64(n)/1e3)
}
return strconv.FormatInt(n, 10)
}
+239 -6
View File
@@ -3,12 +3,16 @@ package cli
import (
"encoding/json"
"fmt"
"io"
"net/url"
"os"
"slices"
"strings"
"text/tabwriter"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
)
@@ -27,7 +31,185 @@ func newSlurmCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
}
cmd.AddCommand(newSlurmStatusCmd(client, asJSON), newSlurmSetupCmd(client), newSlurmOffCmd(client),
newSlurmListCmd(client, "lend", "Let Slurm run jobs on nodes; llmgw's replicas move off first", true),
newSlurmListCmd(client, "reclaim", "Take nodes back from Slurm; llmgw has them when their jobs end", false))
newSlurmListCmd(client, "reclaim", "Take nodes back from Slurm; llmgw has them when their jobs end", false),
newSlurmSharedCmd(client), newSlurmUsersCmd(client, asJSON), newSlurmContainersCmd(client))
return cmd
}
func newSlurmContainersCmd(client func() *ctlClient) *cobra.Command {
cmd := &cobra.Command{
Use: "containers",
Short: "Jobs in containers: srun --container-image (pyxis and enroot)",
Long: "On, the agents install enroot and pyxis on every machine of the Slurm cluster (from NVIDIA's\n" +
"releases, at versions llmgw pins and checks) and, on GPU machines, nvidia-container-cli; once\n" +
"every machine has them, Slurm loads pyxis and jobs may run in a container:\n\n" +
" srun --gres=gpu:1 --container-image=nvcr.io#nvidia/pytorch:25.01-py3 python train.py\n\n" +
"The shared path is in every container where it is on the host. Off, Slurm stops loading pyxis;\n" +
"what was installed stays.",
}
var note string
toggle := func(use, short string, on bool) *cobra.Command {
c := &cobra.Command{
Use: use,
Short: short,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
if !on {
if doc.Node([]string{"slurm", "containers"}) != nil {
doc.Unset([]string{"slurm", "containers"})
}
return nil
}
return doc.SetYAML([]string{"slurm", "containers"}, "true")
})
},
}
c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
return c
}
cmd.AddCommand(toggle("on", "Install pyxis and enroot everywhere, and let jobs run in containers", true),
toggle("off", "Slurm stops loading pyxis; what was installed stays", false))
return cmd
}
func newSlurmSharedCmd(client func() *ctlClient) *cobra.Command {
var path, server, nfs, note string
var off bool
c := &cobra.Command{
Use: "shared --path DIR (--server NODE | --nfs HOST:/EXPORT) | --off",
Short: "Files every machine of the Slurm cluster sees at one path",
Long: "Home directories, datasets, the jobs' output and checkpoints, at the same path on the controller\n" +
"and every node. Three ways:\n" +
" --nfs HOST:/EXPORT the site's own NFS export, mounted at --path on every machine;\n" +
" --server NODE an llmgw node (the controller or a node) exports --path over NFS to the\n" +
" others, for a site with no storage of its own: one machine, not highly\n" +
" available, no faster than its disk and network;\n" +
" neither the site mounts --path itself (Lustre, BeeGFS, GPFS); llmgw only looks\n" +
" that it is there.\n" +
"The export answers the Slurm machines alone, NFS's port the cluster's networks. A machine\n" +
"that gives up its mount does so when no job holds files on it.",
Example: " llmgw ctl slurm shared --path /shared --server node06\n" +
" llmgw ctl slurm shared --path /shared --nfs filer.site:/export/hpc\n" +
" llmgw ctl slurm shared --path /lustre/hpc\n" +
" llmgw ctl slurm shared --off",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
if off == (path != "") {
return fmt.Errorf("give --path, or --off")
}
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
if off {
if doc.Node([]string{"slurm", "shared"}) != nil {
doc.Unset([]string{"slurm", "shared"})
}
return nil
}
v := map[string]string{"path": path}
if server != "" {
v["server"] = server
}
if nfs != "" {
v["nfs"] = nfs
}
b, _ := json.Marshal(v)
return doc.SetYAML([]string{"slurm", "shared"}, string(b))
})
},
}
c.Flags().StringVar(&path, "path", "", "where every machine sees the files, as /shared")
c.Flags().StringVar(&server, "server", "", "the llmgw node that exports them over NFS")
c.Flags().StringVar(&nfs, "nfs", "", "the site's NFS export, HOST:/EXPORT")
c.Flags().BoolVar(&off, "off", false, "no shared files; mounts and exports llmgw made are taken away")
c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
return c
}
func newSlurmUsersCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
cmd := &cobra.Command{
Use: "users",
Short: "The Linux users that run jobs, made from llmgw's accounts",
Long: "With slurm.users \"llmgw\" every active account is a Linux user on every machine of the Slurm\n" +
"cluster: its login from its e-mail address, a UID of llmgw's from 200000 up (never given to\n" +
"another, even when the account goes), its home under the shared path. It logs in over SSH\n" +
"with the keys given in the portal (Profile) or here. An account suspended or deleted loses\n" +
"its user; its home and files stay. Off, users are the site's: from its directory (sssd), or\n" +
"made by hand.",
}
list := &cobra.Command{
Use: "list",
Short: "Every user, with its login, UID and how many keys it has",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
var us []slurmsvc.User
if err := client().get(cmd.Context(), "/api/v1/slurm/users", &us); err != nil {
return err
}
w := cmd.OutOrStdout()
if *asJSON {
return json.NewEncoder(w).Encode(us)
}
tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
fmt.Fprintln(tw, "LOGIN\tUID\tACCOUNT\tSSH KEYS")
for _, u := range us {
fmt.Fprintf(tw, "%s\t%d\t%s\t%d\n", u.Login, u.UID, u.Email, len(u.SSHKeys))
}
return tw.Flush()
},
}
var note string
toggle := func(use, short, value string) *cobra.Command {
c := &cobra.Command{
Use: use,
Short: short,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
if value == "" {
if doc.Node([]string{"slurm", "users"}) != nil {
doc.Unset([]string{"slurm", "users"})
}
return nil
}
return doc.SetYAML([]string{"slurm", "users"}, yamlString(value))
})
},
}
c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
return c
}
keys := &cobra.Command{
Use: "keys ACCOUNT FILE",
Short: "Replace an account's SSH public keys, from a file (- for standard input)",
Long: "FILE holds OpenSSH public keys, one a line, as ~/.ssh/id_ed25519.pub; an empty one takes them all away.",
Example: " llmgw ctl slurm users keys ayse@site.example ayse.pub\n" +
" ssh-add -L | llmgw ctl slurm users keys ayse@site.example -",
Args: cobra.ExactArgs(2),
RunE: func(cmd *cobra.Command, args []string) error {
var b []byte
var err error
if args[1] == "-" {
b, err = io.ReadAll(io.LimitReader(cmd.InOrStdin(), 64<<10))
} else {
b, err = os.ReadFile(args[1])
}
if err != nil {
return err
}
body, _ := json.Marshal(map[string]string{"keys": string(b)})
var u slurmsvc.User
if err := client().put(cmd.Context(), "/api/v1/slurm/users/"+url.PathEscape(args[0])+"/ssh-keys", body, &u); err != nil {
return err
}
fmt.Fprintf(cmd.OutOrStdout(), "%s (UID %d) has %d SSH key(s); the machines have them within a minute\n",
u.Login, u.UID, len(u.SSHKeys))
return nil
},
}
cmd.AddCommand(list,
toggle("on", "Make a Linux user for every active account, on every machine of the Slurm cluster", "llmgw"),
toggle("off", "Leave users to the site; those llmgw made are taken away, their homes left", ""),
keys)
return cmd
}
@@ -55,19 +237,32 @@ func newSlurmStatusCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
fmt.Fprintln(w, ": not heard from yet")
case c.Error != "":
fmt.Fprintf(w, ": %s\n", c.Error)
case c.SlurmctldRunning && c.SlurmdbdRunning:
fmt.Fprintf(w, ": slurmctld %s and slurmdbd running\n", c.Version)
case c.SlurmctldRunning:
fmt.Fprintf(w, ": slurmctld %s running\n", c.Version)
fmt.Fprintf(w, ": slurmctld %s running, slurmdbd not (the jobs' records)\n", c.Version)
default:
fmt.Fprintln(w, ": slurmctld not running")
}
fmt.Fprintf(w, "shared files: %s\n", sharedLine(st.Shared))
if st.Users != "" {
fmt.Fprintf(w, "users: %d, made from llmgw's accounts\n", st.UserCount)
if st.Accounts > 0 {
fmt.Fprintf(w, "accounts: %d, one for each project (sbatch --account=ORG-TEAM-PROJECT)\n", st.Accounts)
}
}
if line := containersLine(st); line != "" {
fmt.Fprintf(w, "containers: %s\n", line)
}
tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
fmt.Fprintln(tw, "NODE\tPHASE\tSLURM\tJOBS\tSLURMD\tNOTE")
fmt.Fprintln(tw, "NODE\tPHASE\tSLURM\tJOBS\tSLURMD\tSHARED\tNOTE")
for _, n := range st.Nodes {
state, jobs, slurmd, note := "-", "-", "-", n.Describe()
state, jobs, slurmd, shared, note := "-", "-", "-", "-", n.Describe()
if r := n.Status; r != nil {
state = or(r.State, "-")
jobs = fmt.Sprint(r.Jobs)
slurmd = map[bool]string{true: "running", false: "stopped"}[r.SlurmdRunning]
shared = or(r.Shared, "-")
if r.Error != "" {
note = r.Error
}
@@ -75,7 +270,7 @@ func newSlurmStatusCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
if !n.Connected {
note = "agent not connected; " + note
}
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\t%s\n", n.Node, n.Phase, state, jobs, slurmd, note)
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n", n.Node, n.Phase, state, jobs, slurmd, shared, note)
}
if err := tw.Flush(); err != nil {
return err
@@ -96,7 +291,8 @@ func newSlurmSetupCmd(client func() *ctlClient) *cobra.Command {
Use: "setup --controller NODE --nodes NODE,... | --all-nodes",
Short: "Run Slurm on llmgw's nodes: slurmctld on one, slurmd on those named",
Long: "Turns slurm.enabled on and names the nodes, or with --all-nodes every node. Their agents install munge and Slurm (22.05 or\n" +
"newer) from the machines' own package sources. Every node starts drained in Slurm and stays\n" +
"newer) from the machines' own package sources; the controller also slurmdbd, the jobs' records, in a\n" +
"MariaDB on that machine alone (one that was there keeps its settings). Every node starts drained in Slurm and stays\n" +
"llmgw's until `llmgw ctl slurm lend` gives it to Slurm.",
Example: " llmgw ctl slurm setup --controller node06 --nodes cuda01,cuda02\n" +
" llmgw ctl slurm setup --controller node06 --all-nodes",
@@ -169,6 +365,43 @@ func newSlurmOffCmd(client func() *ctlClient) *cobra.Command {
return c
}
// containersLine says whether jobs may run in containers, and which
// machines are not ready yet; empty when containers are off.
func containersLine(st slurmsvc.Status) string {
switch st.Containers {
case "on":
return "on (srun --container-image)"
case "":
return ""
}
var waiting []string
for _, n := range st.Nodes {
if n.Status == nil || n.Status.Containers == "" {
waiting = append(waiting, n.Node)
}
}
if st.Ctld == nil || st.Ctld.Containers == "" {
if !slices.Contains(waiting, st.Controller) {
waiting = append(waiting, st.Controller)
}
}
return "installing pyxis and enroot; jobs take them once every machine has them (waiting: " +
strings.Join(waiting, ", ") + ")"
}
// sharedLine says where the shared files are and come from.
func sharedLine(sh config.SharedStorage) string {
switch {
case sh.Path == "":
return "none (`llmgw ctl slurm shared` adds them; a batch job's output stays on its node)"
case sh.Server != "":
return fmt.Sprintf("%s, exported over NFS by %s", sh.Path, sh.Server)
case sh.NFS != "":
return fmt.Sprintf("%s, mounted from %s", sh.Path, sh.NFS)
}
return sh.Path + ", mounted by the site"
}
// newSlurmListCmd adds nodes to slurm.lent, or takes them out.
func newSlurmListCmd(client func() *ctlClient, use, short string, add bool) *cobra.Command {
var note string
+16
View File
@@ -84,3 +84,19 @@ func TestTheSummaryIsWrittenWhereItIsToldNotToTheTerminal(t *testing.T) {
}
}
}
func TestStatusSaysWhenAlertsGoNowhere(t *testing.T) {
render := func(s adminapi.Summary) string {
cmd := &cobra.Command{}
var b bytes.Buffer
cmd.SetOut(&b)
printSummary(cmd, s)
return b.String()
}
if out := render(adminapi.Summary{NoAlerts: true}); !strings.Contains(out, "ALERTS nowhere") {
t.Errorf("no webhook was not called out:\n%s", out)
}
if out := render(adminapi.Summary{}); strings.Contains(out, "ALERTS") {
t.Errorf("an alerts line with webhooks set:\n%s", out)
}
}
+34
View File
@@ -0,0 +1,34 @@
package cli
import (
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
)
func TestAPackageInstallThatIsNoEnginesIsNamed(t *testing.T) {
if got := provisionLine("", "", "installing", "mariadb-server, munge, with apt-get"); got !=
"system packages: installing: mariadb-server, munge, with apt-get" {
t.Fatalf("got %q", got)
}
if got := provisionLine("llamacpp", "b10991", "ready", ""); got != "llamacpp b10991: ready" {
t.Fatalf("got %q", got)
}
}
func TestCoresAloneKeepTheMemory(t *testing.T) {
m := catalog.Model{Name: "judge", Profiles: []catalog.Profile{{Name: "cpu", MinMemoryGB: 2}, {Name: "cpu-big", MinMemoryGB: 2}}}
if got, err := keptMemory(m, ""); err != nil || got != 2 {
t.Fatalf("got %d, %v", got, err)
}
m.Profiles[1].MinMemoryGB = 4
if _, err := keptMemory(m, ""); err == nil {
t.Fatal("profiles that differ were given one memory")
}
if got, err := keptMemory(m, "cpu-big"); err != nil || got != 4 {
t.Fatalf("got %d, %v", got, err)
}
if _, err := keptMemory(m, "gpu"); err == nil {
t.Fatal("a profile the model has not was taken")
}
}
+34 -3
View File
@@ -338,7 +338,7 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
var memory, cores int
var profile string
c := &cobra.Command{
Use: "resources <model> --memory GB [--cores N]",
Use: "resources <model> [--memory GB] [--cores N]",
Short: "Set what a model reserves of a node: memory, and CPU cores",
Long: "--memory is the host RAM a replica needs: for a model with no GPU, its\n" +
"weights, KV cache and engine together. Placement reserves it on the node,\n" +
@@ -351,8 +351,20 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
"--model M` moves them to the new figures.",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
if !cmd.Flags().Changed("memory") && !cmd.Flags().Changed("cores") {
return fmt.Errorf("give --memory, --cores or both")
}
if !cmd.Flags().Changed("memory") {
return fmt.Errorf("--memory is required")
// Cores alone: the memory stays what it is.
var cur catalog.Model
if err := client().get(cmd.Context(), "/api/v1/models/"+args[0], &cur); err != nil {
return err
}
kept, err := keptMemory(cur, profile)
if err != nil {
return err
}
memory = kept
}
body, err := json.Marshal(adminapi.ResourcesRequest{Profile: profile, MemoryGB: memory, CPUCores: cores})
if err != nil {
@@ -369,12 +381,31 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
return nil
},
}
c.Flags().IntVar(&memory, "memory", 0, "host RAM a replica needs, in GB")
c.Flags().IntVar(&memory, "memory", 0, "host RAM a replica needs, in GB; kept as it is when left out")
c.Flags().IntVar(&cores, "cores", 0, "CPU cores a replica with no GPU computes on; 0 reserves none")
c.Flags().StringVar(&profile, "profile", "", "one profile; every profile when left out")
return c
}
// keptMemory is the memory a model's profiles reserve now, for a change of
// cores alone: one profile's, or every profile's when they agree.
func keptMemory(m catalog.Model, profile string) (int, error) {
memory := -1
for _, p := range m.Profiles {
if profile != "" && p.Name != profile {
continue
}
if memory >= 0 && p.MinMemoryGB != memory {
return 0, fmt.Errorf("%s's profiles reserve different memory: give --memory, or --profile", m.Name)
}
memory = p.MinMemoryGB
}
if memory < 0 {
return 0, fmt.Errorf("%s has no profile %q", m.Name, profile)
}
return memory, nil
}
func newClassCmd(client func() *ctlClient) *cobra.Command {
return &cobra.Command{
Use: "class <model> critical|standard|opportunistic|default",
+15 -4
View File
@@ -114,7 +114,7 @@ engines are kept.`,
}
f := cmd.Flags()
f.StringVar(&na.opts.Node, "node", "", "node id (default: the machine's hostname)")
f.StringVar(&na.opts.User, "ssh-user", "", "user to log in as (default: the one before @ in the address, else whoever ran sudo, else you)")
f.StringVar(&na.opts.User, "ssh-user", "", "user to log in as (default: the one before @ in the address, else the ssh config's for the machine, else whoever ran sudo, else you)")
f.StringArrayVar(&na.servers, "server", nil, "control plane address the agent dials, https://host:7443; repeatable for failover (default: this machine)")
f.StringArrayVar(&na.fabrics, "fabric", nil, "InfiniBand or RoCE fabric and its adapters, name:mlx5_0,mlx5_1; repeatable")
f.StringVar(&na.opts.MgmtAddr, "mgmt-addr", "", "address engines are reached at (default: found by the agent)")
@@ -228,8 +228,9 @@ func (na *nodeAdd) target(addr string) onboard.Options {
if o.User == "" {
// Under sudo, the person who ran it — their account is usually the
// one that exists on the nodes too. root's own keys still do the
// logging in, since that is who is running ssh.
o.User = os.Getenv("SUDO_USER")
// logging in, since that is who is running ssh. Only a fallback: a
// User in the ssh config's block for the machine comes first.
o.FallbackUser = os.Getenv("SUDO_USER")
}
return o
}
@@ -495,7 +496,17 @@ func engineEvent(engine, version, state, detail string) onboard.Event {
level = onboard.Warn
}
return onboard.Event{Step: onboard.StepEngines, Level: level,
Detail: fmt.Sprintf("%s %s: %s%s", engine, version, state, prefixed(": ", detail))}
Detail: provisionLine(engine, version, state, detail)}
}
// provisionLine is one line of what a node is installing. A package
// install that is no engine's, Slurm's or MariaDB's, has no name or version.
func provisionLine(engine, version, state, detail string) string {
what := strings.TrimSpace(engine + " " + version)
if what == "" {
what = "system packages"
}
return fmt.Sprintf("%s: %s%s", what, state, prefixed(": ", detail))
}
func prefixed(p, s string) string {
+131 -2
View File
@@ -10,6 +10,7 @@ import (
"fmt"
"log/slog"
"maps"
"math"
"net"
"net/http"
"os"
@@ -36,6 +37,7 @@ import (
"golang.org/x/sync/errgroup"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
"gitea.mynodes.xyz/ilker/llmgw/internal/audit"
"gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval"
"gitea.mynodes.xyz/ilker/llmgw/internal/bundle"
@@ -47,6 +49,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrictest"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/federation"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/held"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
@@ -256,6 +259,7 @@ func newServerCmd() *cobra.Command {
// The settings in force, for what reads them at each use.
var current atomic.Pointer[config.Server]
current.Store(&cfg)
rt.SetSlurm(func() config.SlurmSettings { return current.Load().Slurm })
var agentSvc *agentsvc.Service
var agentHandler http.Handler
var agentPath string
@@ -416,6 +420,7 @@ func newServerCmd() *cobra.Command {
adminAPI = api
api.SetReleases(cfg.Releases)
api.SetNotifier(notifier)
api.SetWebhooks(func() int { return len(current.Load().Notify.Webhooks) })
api.SetReplayer(rt.Replay)
// Quality suites, asked of a model the way a replay is.
evals = newEvalService(cfg, rt, admin, log)
@@ -431,6 +436,27 @@ func newServerCmd() *cobra.Command {
api.SetAgentRestarter(agentSvc.RestartAgents)
}
api.SetAudit(st)
// Prices from what the cluster costs: what each model
// holds, kept by the hour on the leader.
api.SetPrices(st, func() adminapi.PriceConfig {
au := current.Load().Auth
cur := make(map[string]accounting.Price, len(au.Pricing))
for m, p := range au.Pricing {
cur[m] = accountingPrice(p)
}
return adminapi.PriceConfig{Basis: accounting.Basis{GPUHour: au.GPUHourCost, CoreHour: au.CoreHourCost},
Current: cur, Default: accountingPrice(au.DefaultPrice)}
})
go (&held.Recorder{Inv: admin.Inventory(), Store: st, Log: log,
Models: func() map[string]string {
m := map[string]string{}
for _, rep := range admin.Reconciler().Replicas() {
m[rep.ID] = rep.Model
}
return m
},
Leader: func() bool { return admin.Leadership() == nil || admin.Leadership().IsLeader() },
}).Run(ctx)
// Knowledge bases, with the cluster's own models to embed a
// probe (a collection's dimension) and an operator's search.
api.SetKB(&adminapi.KBAdmin{Store: st, Embed: embedder(rt.Replay)})
@@ -439,7 +465,12 @@ func newServerCmd() *cobra.Command {
trainer := train.New(st, agentSvc, log)
agentSvc.SetTrain(trainer)
go trainer.Run(ctx)
api.SetTrain(&adminapi.TrainAdmin{Store: st, Service: trainer})
api.SetTrain(&adminapi.TrainAdmin{Store: st, Service: trainer, Log: log})
// From the portal: a node with a card free enough, and
// what was made published when it is done.
trainer.SetPicker(api.PickTrainNode)
trainer.SetOnDone(api.TrainDone)
rt.SetTraining(api)
// Slurm on the nodes the settings name, its munge key
// made from a cluster secret.
slurmSvc = slurmsvc.New(func() config.SlurmSettings { return current.Load().Slurm },
@@ -452,6 +483,25 @@ func newServerCmd() *cobra.Command {
}, log)
// What slurm.conf held, for the leader that comes next.
slurmSvc.SetMemory(slurmsvc.StoreMemory(st))
// slurm.users "llmgw": a Linux user for each active account.
slurmSvc.SetUsers(func(ctx context.Context) ([]slurmsvc.User, error) {
us, err := st.SlurmUsers(ctx)
out := make([]slurmsvc.User, 0, len(us))
for _, u := range us {
out = append(out, adminapi.SlurmUserView(u))
}
return out, err
})
api.SetSlurmUsers(st)
// Slurm's accounts from the projects, and the jobs
// that ended priced as the cluster's hours are.
slurmSvc.SetAccounts(func(ctx context.Context) ([]slurm.Account, error) {
return slurmAccounts(ctx, st)
})
slurmSvc.SetJobs(func(ctx context.Context, jobs []slurmsvc.JobRecord) error {
au := current.Load().Auth
return recordSlurmJobs(ctx, st, current.Load().Slurm.Cluster(), au.GPUHourCost, au.CoreHourCost, jobs, log)
})
agentSvc.SetSlurm(slurmSvc)
go slurmSvc.Run(ctx)
api.SetSlurm(slurmSvc)
@@ -616,7 +666,7 @@ func newServerCmd() *cobra.Command {
// takes longer than the mirror's usual answers.
if adminAPI != nil {
if up, err := mirror.NewFailoverClient(mirrors, cfg.Mirror.TLS, 10*time.Minute); err == nil {
adminAPI.SetTrainMirror(up)
adminAPI.SetTrainMirror(up, hf.New(cfg.Mirror.Registry, cfg.Mirror.Timeout))
}
}
// Browsing the registry is what turns "paste a resolve URL"
@@ -1406,3 +1456,82 @@ func runPresetsPass(ctx context.Context, api *adminapi.API, lead *control.Leader
}
}
}
// accountingPrice is a configured price as accounting has it.
func accountingPrice(p config.Price) accounting.Price {
return accounting.Price{PromptPer1k: p.PromptPer1k, CompletionPer1k: p.CompletionPer1k, CachedPer1k: p.CachedPer1k,
PerImage: p.PerImage, PerMesh: p.PerMesh}
}
// slurmAccounts are Slurm's accounts from the store's projects, members and
// Slurm users.
func slurmAccounts(ctx context.Context, st *storepkg.DB) ([]slurm.Account, error) {
ps, err := st.SlurmProjects(ctx)
if err != nil {
return nil, err
}
ms, err := st.AllMemberships(ctx)
if err != nil {
return nil, err
}
us, err := st.SlurmUsers(ctx)
if err != nil {
return nil, err
}
logins := make(map[string]string, len(us))
for _, u := range us {
logins[u.UserID] = u.Login
}
tps := make([]slurm.TenancyProject, 0, len(ps))
for _, p := range ps {
tps = append(tps, slurm.TenancyProject{ID: p.ID, Name: p.Name, TeamID: p.TeamID, TeamName: p.TeamName,
OrgID: p.OrgID, OrgName: p.OrgName})
}
tms := make([]slurm.TenancyMember, 0, len(ms))
for _, m := range ms {
tms = append(tms, slurm.TenancyMember{UserID: m.UserID, Scope: string(m.Scope), ScopeID: m.ScopeID})
}
return slurm.Accounts(tps, tms, logins), nil
}
// recordSlurmJobs records the jobs that ended, each with its project (from
// its account) and person (from its login), priced as the hours the
// cluster's models hold are: a GPU job its cards' hours, any other its
// cores'.
func recordSlurmJobs(ctx context.Context, st *storepkg.DB, cluster string, gpuHour, coreHour float64,
jobs []slurmsvc.JobRecord, log *slog.Logger) error {
as, err := slurmAccounts(ctx, st)
if err != nil {
return err
}
project := map[string]string{}
for _, a := range as {
if a.Project != "" {
project[a.Name] = a.Project
}
}
us, err := st.SlurmUsers(ctx)
if err != nil {
return err
}
person := map[string]string{}
for _, u := range us {
person[u.Login] = u.UserID
}
rows := make([]storepkg.SlurmJob, 0, len(jobs))
for _, j := range jobs {
secs := j.Elapsed.Seconds()
cost := float64(j.CPUs) * secs / 3600 * coreHour
if j.GPUs > 0 {
cost = float64(j.GPUs) * secs / 3600 * gpuHour
}
rows = append(rows, storepkg.SlurmJob{Key: cluster + "/" + j.ID, UserID: person[j.Login], ProjectID: project[j.Account],
Login: j.Login, Account: j.Account, Partition: j.Partition, State: j.State, Start: j.Start, End: j.End,
ElapsedSeconds: int64(secs), CPUs: int64(j.CPUs), GPUs: int64(j.GPUs), CostMicros: int64(math.Ceil(cost * 1e6))})
}
n, err := st.RecordSlurmJobs(ctx, rows)
if n > 0 {
log.Info("Slurm jobs that ended recorded, with their cost", "jobs", n)
}
return err
}
+1 -1
View File
@@ -443,7 +443,7 @@ func (sa *serverAdd) storeAccess(ctx context.Context) error {
return err
}
}
sa.done("nodes will be told of both control planes within half a minute")
sa.done("nodes will be told of every control plane within half a minute")
return nil
}
+9 -1
View File
@@ -127,6 +127,10 @@ type serverInit struct {
// slurm runs Slurm beside llmgw: slurmctld on this machine's node,
// slurmd on every node, each llmgw's until it is lent.
slurm bool
// slurmShared is where Slurm's shared files come from: "here" (this
// machine exports /shared), the site's export as HOST:/EXPORT, or
// "none". slurmUsers is "llmgw" (users from the accounts) or "site".
slurmShared, slurmUsers string
// evictAfterIdle and minLifetime are on_demand's; zero leaves its own.
evictAfterIdle, minLifetime time.Duration
// standbys makes each machine of the replicated store a control plane
@@ -262,7 +266,11 @@ func newServerInitCmd() *cobra.Command {
f.BoolVar(&in.media, "media", false,
"pictures and 3D: the nodes install ComfyUI, and a picture model and a 3D model are added once a card fits one")
f.BoolVar(&in.slurm, "slurm", false,
"Slurm for training and batch jobs: slurmctld on this machine, slurmd on every node, each llmgw's until 'llmgw ctl slurm lend'")
"Slurm for training and batch jobs: slurmctld and slurmdbd (with a MariaDB) on this machine, slurmd on every node, each llmgw's until 'llmgw ctl slurm lend'")
f.StringVar(&in.slurmShared, "slurm-shared", "here",
"with --slurm, the jobs' shared files at /shared: here (this machine exports them over NFS), the site's HOST:/EXPORT, or none")
f.StringVar(&in.slurmUsers, "slurm-users", config.UsersLlmgw,
"with --slurm, who makes the Linux users that run jobs: llmgw (one for each account, SSH keys in the portal) or site")
f.DurationVar(&in.evictAfterIdle, "evict-after-idle", 0,
"how long a model loaded on demand goes unasked-for before another may have its card (0: 2m)")
f.DurationVar(&in.minLifetime, "min-lifetime", 0,
+46 -4
View File
@@ -144,6 +144,13 @@ func (in *serverInit) checkChoices() error {
if err := (config.SlurmSettings{Enabled: true, Controller: localNodeID()}).Check(); err != nil {
return fmt.Errorf("slurm: this machine's hostname: %w", err)
}
set := config.SlurmSettings{Enabled: true, Controller: localNodeID(), Shared: slurmShared(in.slurmShared)}
if err := set.Check(); err != nil {
return fmt.Errorf("--slurm-shared %q: here, none, or the site's HOST:/EXPORT: %w", in.slurmShared, err)
}
if u := in.slurmUsers; u != "" && u != config.UsersLlmgw && u != "site" {
return fmt.Errorf("--slurm-users %q: llmgw or site", u)
}
}
if in.vip != "" {
ip, _, err := net.ParseCIDR(in.vip)
@@ -329,7 +336,7 @@ func (in *serverInit) tuneConfig(body string) string {
"engines:\n comfyui: "+provision.ComfyUIVersion()+"\npresets:\n auto: true\n")
}
if in.slurm {
extra = append(extra, slurmSection(localNodeID()))
extra = append(extra, slurmSection(localNodeID(), slurmShared(in.slurmShared), in.slurmUsers != "site"))
}
if in.evictAfterIdle > 0 || in.minLifetime > 0 {
b := "# How models that load when asked for take turns on a card, chosen at\n# install.\non_demand:\n"
@@ -347,14 +354,43 @@ func (in *serverInit) tuneConfig(body string) string {
return body
}
// slurmShared is --slurm-shared as settings: "here" is /shared exported
// by this machine's node, "none" nothing, anything else the site's export.
func slurmShared(v string) config.SharedStorage {
switch v {
case "", "none":
return config.SharedStorage{}
case "here":
return config.SharedStorage{Path: "/shared", Server: localNodeID()}
}
return config.SharedStorage{Path: "/shared", NFS: v}
}
// slurmSection is server.yaml's slurm block as an install chooses it:
// slurmctld on this machine's node, slurmd on every node.
func slurmSection(controller string) string {
return "# Slurm beside llmgw, chosen at install: slurmctld on this machine, slurmd on\n" +
// slurmctld on this machine's node, slurmd on every node, the shared files
// and the users as chosen.
func slurmSection(controller string, shared config.SharedStorage, users bool) string {
b := "# Slurm beside llmgw, chosen at install: slurmctld, and slurmdbd with a MariaDB\n" +
"# for the jobs' records, on this machine; slurmd on\n" +
"# every node, those that join later too. Each node is llmgw's, drained in\n" +
"# Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm; 'llmgw ctl slurm\n" +
"# reclaim NODE' takes it back once its jobs end.\n" +
"slurm:\n enabled: true\n controller: " + yamlString(controller) + "\n all_nodes: true\n"
switch {
case shared.Server != "":
b += " # /shared on every Slurm machine, exported over NFS from this one\n" +
" # ('llmgw ctl slurm shared' points it at the site's storage instead).\n" +
" shared:\n path: " + yamlString(shared.Path) + "\n server: " + yamlString(shared.Server) + "\n"
case shared.NFS != "":
b += " # /shared on every Slurm machine, mounted from the site's export.\n" +
" shared:\n path: " + yamlString(shared.Path) + "\n nfs: " + yamlString(shared.NFS) + "\n"
}
if users {
b += " # A Linux user for each llmgw account, the same UID everywhere; SSH keys\n" +
" # in the portal's Profile. Empty leaves users to the site (sssd, a directory).\n" +
" users: llmgw\n"
}
return b
}
// localNodeID is the node id this machine's agent takes: its hostname.
@@ -500,6 +536,12 @@ func (in *serverInit) initFlags() []string {
}
if in.slurm {
add("--slurm")
if in.slurmShared != "" && in.slurmShared != "here" {
add("--slurm-shared", in.slurmShared)
}
if in.slurmUsers == "site" {
add("--slurm-users", "site")
}
}
if in.evictAfterIdle > 0 {
add("--evict-after-idle", durationYAML(in.evictAfterIdle))
+3 -1
View File
@@ -17,7 +17,7 @@ import (
func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
in := &serverInit{
language: "tr", signupOpen: true, defaultRPM: 120, defaultTPM: 50000, maxKeyTTL: 30 * 24 * time.Hour,
media: true, slurm: true, evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
media: true, slurm: true, slurmShared: "here", slurmUsers: "llmgw", evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
judge: judgeSpec{Fail: "open", Categories: []string{"Violent", "PII"}, Replicas: 3, Budget: 3 * time.Second},
}
j := in.judgeFor()
@@ -43,6 +43,8 @@ func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
t.Errorf("pictures and 3D: presets %v comfyui %q", cfg.Presets.Auto, cfg.Engines["comfyui"])
case !cfg.Slurm.Enabled || !cfg.Slurm.AllNodes || cfg.Slurm.Controller != localNodeID():
t.Errorf("slurm %+v", cfg.Slurm)
case cfg.Slurm.Shared.Path != "/shared" || cfg.Slurm.Shared.Server != localNodeID() || cfg.Slurm.Users != "llmgw":
t.Errorf("slurm's shared files and users %+v", cfg.Slurm)
case cfg.OnDemand.EvictAfterIdle != time.Minute || cfg.OnDemand.MinLifetime != 90*time.Second:
t.Errorf("on demand %+v", cfg.OnDemand)
}
+7 -23
View File
@@ -329,7 +329,7 @@ func (in *serverInit) ensureNode(ctx context.Context) (string, error) {
if classes, n := gpuClasses(ctx); n > 0 {
in.kept("%d NVIDIA GPU(s) here: %s — the node will use them", n, strings.Join(classes, ", "))
} else if hasNVIDIADevice() {
} else if discovery.HasNVIDIADevice() {
fmt.Fprintf(in.out, " !! an NVIDIA card is on the PCI bus but nvidia-smi does not answer: "+
"install the driver and run init again to serve on it; until then this node is CPU only\n")
} else {
@@ -398,7 +398,9 @@ labels:
// checks its own llama.cpp, the CUDA build when it has a GPU, and a new
// machine is often busy with its own package updates first.
client := in.adminAPI()
last := ""
// Each engine's line once, and again only when it changes: the engines'
// states come back together on every look.
seen := map[string]string{}
err = waitFor(ctx, 20*time.Minute, "this node to be ready with an engine", func() (bool, error) {
var nodes []adminapi.NodeView
if err := client.get(ctx, "/api/v1/nodes", &nodes); err != nil {
@@ -412,9 +414,9 @@ labels:
return true, nil
}
for _, p := range n.Provision {
if line := fmt.Sprintf("%s %s: %s %s", p.Engine, p.Version, p.State, p.Detail); line != last {
fmt.Fprintf(in.out, " .. %s\n", strings.TrimSpace(line))
last = line
if line := provisionLine(p.Engine, p.Version, p.State, p.Detail); seen[p.Engine] != line {
fmt.Fprintf(in.out, " .. %s\n", line)
seen[p.Engine] = line
}
// Only llama.cpp is waited for: vLLM installs for minutes after,
// and one this node cannot run is not a reason to stop here.
@@ -432,24 +434,6 @@ labels:
return nodeID, nil
}
// hasNVIDIADevice reports an NVIDIA card on the PCI bus, driver or not, so a
// machine whose GPU is invisible for want of a driver says so instead of
// quietly serving on CPU.
func hasNVIDIADevice() bool {
vendors, _ := filepath.Glob("/sys/bus/pci/devices/*/vendor")
for _, v := range vendors {
b, err := os.ReadFile(v)
if err == nil && strings.TrimSpace(string(b)) == "0x10de" {
class, _ := os.ReadFile(filepath.Join(filepath.Dir(v), "class"))
// 0x03xxxx is a display or 3D controller; NVIDIA also makes bridges.
if strings.HasPrefix(strings.TrimSpace(string(class)), "0x03") {
return true
}
}
}
return false
}
// --- the starter model ---
// starterModel is the catalog entry: a GPU profile first when there is a card,
+4
View File
@@ -156,6 +156,10 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
if s.Stopped {
fmt.Fprintln(out, "SERVICE STOPPED: no model runs and requests are answered 503 (llmgw start)")
}
if s.NoAlerts {
fmt.Fprintln(out, "ALERTS nowhere: a lost node or a judge down is told nobody "+
"(llmgw ctl config set notify.webhooks, then llmgw ctl notify test)")
}
if s.GuardOff {
fmt.Fprintln(out, "GUARD off: no request is checked, whatever the policies say (llmgw ctl guard on)")
}
+23 -14
View File
@@ -21,9 +21,13 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
Long: "A node trains an adapter for a catalog model on pairs (llmgw ctl kb dataset\n" +
"writes them from a knowledge base; read them through first). It trains with\n" +
"its installed vLLM's torch and a small overlay, as the engines' user, on a card\n" +
"with --gpu-memory free — beside an engine, if it leaves that much. What it\n" +
"makes is kept by the control plane: llmgw ctl train download, or publish it\n" +
"as the model's adapter.\n\n" +
"with --gpu-memory free — beside an engine, if it leaves that much. Without\n" +
"--base-repo the weights are found from where the model's came, without\n" +
"--gpu-memory from their size, and without --node it goes to the node whose\n" +
"card has the most free. What it makes is kept by the control plane: llmgw ctl\n" +
"train download, or publish it as the model's adapter (--publish does that by\n" +
"itself when it is done). People in a project can do all this from the portal's\n" +
"\"Customise a model\" page.\n\n" +
"An adapter learns how answers sound — words, format, tone — far better than\n" +
"what they say. Facts belong to the knowledge base.",
}
@@ -33,11 +37,12 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
var gpuMem int
var epochs, rank int
var lr float64
var noGGUF bool
var noGGUF, publishWhenDone bool
start := &cobra.Command{
Use: "start NAME --model MODEL --base-repo OWNER/NAME --node NODE --data FILE",
Use: "start NAME --model MODEL --data FILE",
Short: "Queue an adapter's training on a node",
Example: " llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \\\n" +
Example: " llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct --data policies.jsonl --public --publish\n" +
" llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \\\n" +
" --base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
@@ -46,7 +51,7 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
return err
}
req := adminapi.TrainRequest{Name: args[0], Model: model, BaseRepo: base, BaseRevision: rev, Node: node,
GPUMemoryMiB: gpuMem, Data: string(b), Params: map[string]any{}, Owner: owner, Public: public}
GPUMemoryMiB: gpuMem, Data: string(b), Params: map[string]any{}, Owner: owner, Public: public, Publish: publishWhenDone}
if cmd.Flags().Changed("epochs") {
req.Params["epochs"] = epochs
}
@@ -69,8 +74,8 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
if v.Owner != "" {
who = v.Owner + " only"
}
fmt.Fprintf(cmd.OutOrStdout(), "queued %s: %s for %s on %s, %d pairs; it will be seen by %s\n",
v.ID, v.Name, v.Model, v.Node, v.Pairs, who)
fmt.Fprintf(cmd.OutOrStdout(), "queued %s: %s for %s on %s, %d pairs from %s, %d MiB of a card; it will be seen by %s\n",
v.ID, v.Name, v.Model, or(v.Node, "the node with a card free enough"), v.Pairs, v.BaseRepo, v.GPUMemoryMiB, who)
fmt.Fprintf(cmd.OutOrStdout(), "follow it with: llmgw ctl train show %s --follow\n", v.ID)
return nil
},
@@ -79,8 +84,9 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
f.StringVar(&model, "model", "", "the catalog model the adapter is for")
f.StringVar(&base, "base-repo", "", "the weights to train on, as safetensors on Hugging Face (owner/name)")
f.StringVar(&rev, "base-revision", "", "a revision of them (default: the newest)")
f.StringVar(&node, "node", "", "the node that trains")
f.IntVar(&gpuMem, "gpu-memory", 4096, "MiB of a card the job needs free")
f.StringVar(&node, "node", "", "the node that trains (default: the one whose card has the most free)")
f.IntVar(&gpuMem, "gpu-memory", 0, "MiB of a card the job needs free (default: from the weights' size)")
f.BoolVar(&publishWhenDone, "publish", false, "publish it as the model's adapter when it is done, restarting a llama.cpp model's replicas")
f.StringVar(&data, "data", "", "the pairs, NDJSON of {\"q\", \"a\"}")
f.IntVar(&epochs, "epochs", 6, "passes over the pairs")
f.IntVar(&rank, "rank", 16, "the adapter's rank")
@@ -89,7 +95,7 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
f.StringVar(&owner, "owner", "", "who the adapter belongs to — org:, team:, project: (path or ID), user: or key: — and only they see it "+
"(default: the knowledge base's owner, when the pairs come from one)")
f.BoolVar(&public, "public", false, "every caller of the model sees it; refused for pairs from a knowledge base")
for _, req := range []string{"model", "base-repo", "node", "data"} {
for _, req := range []string{"model", "data"} {
_ = start.MarkFlagRequired(req)
}
c.AddCommand(start)
@@ -138,8 +144,11 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
}
line := fmt.Sprintf("%s %s %s loss %s peak %s %s", j.State, trainProgress(j), j.Node, trainLoss(j), trainPeak(j), j.Message)
if !follow {
fmt.Fprintf(out, "%s: adapter %s for %s, owned by %s, on %s (base %s), %d pairs\n", j.ID, j.Name, j.Model,
or(j.Owner, "everyone"), j.Node, j.BaseRepo, j.Pairs)
fmt.Fprintf(out, "%s: adapter %s for %s, owned by %s, on %s (base %s), %d pairs, %d MiB of a card\n", j.ID, j.Name, j.Model,
or(j.Owner, "everyone"), or(j.Node, "the node whose card has the most free"), j.BaseRepo, j.Pairs, j.GPUMemoryMiB)
if j.Project != "" {
fmt.Fprintf(out, "started from the portal in project %s by %s; published when done: %t\n", j.Project, j.CreatedBy, j.Publish)
}
fmt.Fprintln(out, line)
if len(j.Artifacts) > 0 {
fmt.Fprintf(out, "made: %s (llmgw ctl train download %s --kind KIND)\n", strings.Join(j.Artifacts, ", "), j.ID)
+9 -3
View File
@@ -287,12 +287,18 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
title: "Slurm", on: in.slurm,
about: "A Slurm cluster beside llmgw, for training and other batch work: slurmctld on this machine, " +
"slurmd on every node, those that join later too. The agents install munge and Slurm from the " +
"machines' own package sources.\n\n" +
"machines' own package sources, and on this machine slurmdbd, the jobs' records, with a MariaDB " +
"that answers this machine alone (Slurm keeps its records in MySQL or MariaDB only, not in the " +
"cluster's PostgreSQL). Its port 6819, with Slurm's others, answers the cluster's networks only.\n\n" +
"Each node stays llmgw's, drained in Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm: " +
"its models move off first. 'llmgw ctl slurm reclaim NODE' takes it back once its jobs end. Jobs " +
"are submitted on this machine with sbatch and srun.",
"are submitted on this machine with sbatch and srun.\n\n" +
"Every Slurm machine sees /shared, exported over NFS from this machine (homes, datasets, the jobs' " +
"output); 'llmgw ctl slurm shared' points it at the site's own storage instead. Each llmgw account " +
"is a Linux user there, the same UID on every machine, logging in with the SSH keys given in the " +
"portal; 'llmgw ctl slurm users off' leaves users to the site's directory.",
off: "no Slurm; 'llmgw ctl slurm setup' adds it later",
summary: func() string { return "slurmctld here, slurmd on every node; none lent" },
summary: func() string { return "slurmctld here, slurmd on every node, /shared from here; none lent" },
needs: func() string {
if !f.node.on {
return "slurmctld runs on this machine's node: turn the node on, or Slurm off"
+26 -4
View File
@@ -35,13 +35,28 @@ func (m consoleModel) slurmPanel(pw int) (string, string, string) {
case c == nil:
case c.Error != "":
ctl = sBad.Render(c.Error)
case c.SlurmctldRunning && c.SlurmdbdRunning:
ctl = sOK.Render("slurmctld " + c.Version + " and slurmdbd running")
case c.SlurmctldRunning:
ctl = sOK.Render("slurmctld " + c.Version + " running")
ctl = sOK.Render("slurmctld "+c.Version+" running") + ", " + sWarn.Render("slurmdbd not")
default:
ctl = sBad.Render("slurmctld not running")
}
head += fmt.Sprintf("\n controller %s: %s", st.Controller, ctl)
head += "\n " + sDim.Render(fmt.Sprintf("cluster %s, partition %s; sbatch and srun on %s", st.Cluster, st.Partition, st.Controller))
if st.Shared.Path == "" {
head += "\n shared files: " + sWarn.Render("none") + sDim.Render(" (llmgw ctl slurm shared; output stays on the job's node)")
} else {
head += "\n shared files: " + sharedLine(st.Shared)
}
if st.Users != "" {
head += "\n " + fmt.Sprintf("users: %d from llmgw's accounts", st.UserCount) + sDim.Render(", SSH keys in the portal's Profile")
} else {
head += "\n " + sDim.Render("users: the site's (llmgw ctl slurm users on makes them from llmgw's accounts)")
}
if line := containersLine(st); line != "" {
head += "\n containers: " + line
}
head += "\n " + sDim.Render("A node is llmgw's until lent; lent, its replicas move off first.")
if len(st.Waiting) > 0 {
head += "\n " + sWarn.Render(fmt.Sprintf("not in slurm.conf until they report their hardware: %v", st.Waiting))
@@ -54,13 +69,20 @@ func (m consoleModel) slurmPanel(pw int) (string, string, string) {
for _, n := range st.Nodes {
rows = append(rows, slurmRow(n))
}
return "Slurm", head + table([]string{"NODE", "WHOSE", "IN SLURM", "JOBS", "SLURMD", "NOTE"}, rows, m.row, pw, m.inPanel), keys
return "Slurm", head + table([]string{"NODE", "WHOSE", "IN SLURM", "JOBS", "SLURMD", "SHARED", "NOTE"}, rows, m.row, pw, m.inPanel), keys
}
func slurmRow(n slurmsvc.NodeStatus) []string {
state, jobs, slurmd, note := "-", "-", "-", n.Describe()
state, jobs, slurmd, shared, note := "-", "-", "-", "-", n.Describe()
if r := n.Status; r != nil {
state = or(r.State, "-")
switch r.Shared {
case "":
case "missing":
shared = sBad.Render(r.Shared)
default:
shared = sOK.Render(r.Shared)
}
jobs = fmt.Sprint(r.Jobs)
slurmd = sBad.Render("stopped")
if r.SlurmdRunning {
@@ -82,7 +104,7 @@ func slurmRow(n slurmsvc.NodeStatus) []string {
case slurmsvc.Lending, slurmsvc.Reclaiming:
whose = sWarn.Render(whose)
}
return []string{n.Node, whose, state, jobs, slurmd, note}
return []string{n.Node, whose, state, jobs, slurmd, shared, note}
}
// slurmMenu is what can be done with one of Slurm's nodes.
+5 -3
View File
@@ -11,6 +11,7 @@ import (
tea "github.com/charmbracelet/bubbletea"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
)
@@ -37,16 +38,17 @@ func TestTheConsoleLendsANodeToSlurmFromItsRow(t *testing.T) {
m := newConsoleModel(srv.URL, "/nonexistent/server.yaml")
m.loaded, m.menu, m.inPanel = true, cSlurm, true
m.data.slurm = slurmsvc.Status{Enabled: true, Cluster: "llmgw", Controller: "node06", Partition: "gpu",
Ctld: &slurmsvc.Reported{SlurmctldRunning: true, Version: "23.11.4"},
Ctld: &slurmsvc.Reported{SlurmctldRunning: true, SlurmdbdRunning: true, Version: "23.11.4"},
Shared: config.SharedStorage{Path: "/shared", Server: "node06"}, Users: "llmgw", UserCount: 2,
Nodes: []slurmsvc.NodeStatus{
{Node: "node05", Phase: slurmsvc.Lent, Connected: true,
Status: &slurmsvc.Reported{State: "MIXED", Jobs: 2, SlurmdRunning: true}},
Status: &slurmsvc.Reported{State: "MIXED", Jobs: 2, SlurmdRunning: true, Shared: "mounted"}},
{Node: "cuda01", Phase: slurmsvc.Llmgw, Replicas: 3, Connected: true,
Status: &slurmsvc.Reported{State: "IDLE+DRAIN", SlurmdRunning: true}},
}}
var tm tea.Model = m
view := tm.View()
for _, want := range []string{"slurmctld 23.11.4 running", "node05", "Slurm's", "MIXED", "cuda01", "llmgw's"} {
for _, want := range []string{"slurmctld 23.11.4 and slurmdbd running", "/shared, exported over NFS by node06", "users: 2", "node05", "Slurm's", "MIXED", "cuda01", "llmgw's"} {
if !strings.Contains(view, want) {
t.Errorf("the Slurm view has no %q:\n%s", want, view)
}
+4 -1
View File
@@ -171,7 +171,10 @@ func newUserStateCmd(open userOpener, verb string, state store.UserState, short
if err := s.DeleteUserSessions(cmd.Context(), u.ID); err != nil {
return err
}
cmd.PrintErrln("Their sessions were closed. Their API keys still work — revoke those separately.")
// The keys are refused by their owner's state, which every
// router reads again within auth.cache_ttl.
cmd.PrintErrln("Their sessions were closed. Their API keys are refused within auth.cache_ttl " +
"(30 s by default), and work again if they are reinstated.")
}
fmt.Fprintf(cmd.OutOrStdout(), "%s is now %s\n", u.Email, state)
return nil
+9
View File
@@ -470,6 +470,15 @@ type Auth struct {
Pricing map[string]Price `koanf:"pricing"`
DefaultPrice Price `koanf:"default_price"`
// GPUHourCost is what an hour of one GPU costs the operator, in the
// units budgets are in: the hardware written off over its years, its
// power and room, and the people who run it, over the GPUs and the
// hours of a month. With CoreHourCost for a CPU core, it is what
// `llmgw ctl prices suggest` prices each model from: the hours its
// replicas held, idle ones too, over the tokens it served.
GPUHourCost float64 `koanf:"gpu_hour_cost"`
CoreHourCost float64 `koanf:"core_hour_cost"`
// FlushInterval and BatchSize decide when usage records are written.
FlushInterval time.Duration `koanf:"flush_interval"`
BatchSize int `koanf:"batch_size"`
+90 -1
View File
@@ -2,8 +2,10 @@ package config
import (
"fmt"
"path"
"regexp"
"slices"
"strings"
)
// SlurmSettings is a Slurm cluster run beside llmgw on some of its nodes
@@ -38,6 +40,90 @@ type SlurmSettings struct {
// Partition is the one partition every node is in. Empty is "gpu".
Partition string `koanf:"partition"`
// Shared is the file system every machine of the Slurm cluster sees at
// the same path: home directories, datasets, the jobs' output and
// checkpoints. Empty path: none, and a batch job's output stays on the
// node it ran on.
Shared SharedStorage `koanf:"shared"`
// Users is who keeps the Linux users that run jobs, the same on every
// machine of the Slurm cluster: "llmgw" makes one for each active llmgw
// account (its login from the e-mail address, a UID of llmgw's from
// 200000 up, its SSH keys from the portal), let in to the controller,
// and to a node only while a job of theirs runs there (pam_slurm_adopt);
// empty leaves users to the site (a directory through sssd, or by hand).
Users string `koanf:"users"`
// Containers lets jobs run in containers: `srun --container-image
// nvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),
// the GPUs and the shared path inside. The agents install both on
// every Slurm machine; jobs take it once every machine has them.
Containers bool `koanf:"containers"`
}
// SharedStorage is where the Slurm cluster's shared files come from: the
// site's NFS export (NFS), an NFS export llmgw keeps on one of its nodes
// (Server), or, with neither, a file system the site mounts itself
// (Lustre, BeeGFS, GPFS), which llmgw only looks for.
type SharedStorage struct {
// Path is where every machine sees it, as /shared.
Path string `koanf:"path"`
// NFS is the site's export, host:/path, mounted at Path on every
// machine of the Slurm cluster.
NFS string `koanf:"nfs"`
// Server is the llmgw node, the controller or one of the nodes, that
// exports Path over NFS to the others: for a site with no storage of
// its own. One machine: not highly available, and no faster than its
// disk and network.
Server string `koanf:"server"`
}
// UsersLlmgw has llmgw make the Slurm cluster's users from its accounts.
const UsersLlmgw = "llmgw"
// HomeDir is where a user's home directory is: under the shared path when
// there is one, else on each machine's own disk.
func (s SlurmSettings) HomeDir(login string) string {
if s.Shared.Path != "" {
return path.Join(s.Shared.Path, "home", login)
}
return path.Join("/home", login)
}
// nfsExport is host:/path.
var nfsExport = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.:\[\]-]*:/[A-Za-z0-9_./-]*$`)
// sharedPath is an absolute path of plain names, which a mount unit's name
// is made from.
var sharedPath = regexp.MustCompile(`^(/[A-Za-z0-9_.-]+)+$`)
func (s SlurmSettings) checkShared() error {
sh := s.Shared
if sh.Path == "" {
if sh.NFS != "" || sh.Server != "" {
return fmt.Errorf("slurm.shared.path is empty: say where every machine sees the shared files, as /shared")
}
return nil
}
if !sharedPath.MatchString(sh.Path) || strings.Contains(sh.Path, "/.") {
return fmt.Errorf("slurm.shared.path %q is not an absolute path of plain names, as /shared", sh.Path)
}
for _, sys := range []string{"/", "/etc", "/usr", "/var", "/var/lib", "/home", "/root", "/boot", "/opt", "/srv", "/tmp", "/proc", "/sys", "/dev", "/run"} {
if path.Clean(sh.Path) == sys {
return fmt.Errorf("slurm.shared.path %s is the system's own directory: take one of its own, as /shared", sh.Path)
}
}
if sh.NFS != "" && sh.Server != "" {
return fmt.Errorf("slurm.shared names both the site's export (nfs) and an llmgw node to export it (server): one of them")
}
if sh.NFS != "" && !nfsExport.MatchString(sh.NFS) {
return fmt.Errorf("slurm.shared.nfs %q is not host:/path", sh.NFS)
}
if sh.Server != "" && sh.Server != s.Controller && !s.AllNodes && !slices.Contains(s.Nodes, sh.Server) {
return fmt.Errorf("slurm.shared.server %q is neither the controller nor one of slurm.nodes", sh.Server)
}
return nil
}
// slurmName is what Slurm takes as a node, cluster or partition name.
@@ -111,5 +197,8 @@ func (s SlurmSettings) Check() error {
return fmt.Errorf("slurm.lent[%d] %q is not in slurm.nodes: only a node that runs slurmd can be lent", i, n)
}
}
return nil
if s.Users != "" && s.Users != UsersLlmgw {
return fmt.Errorf("slurm.users %q: %q has llmgw make them, empty leaves them to the site", s.Users, UsersLlmgw)
}
return s.checkShared()
}
+43
View File
@@ -35,6 +35,49 @@ func TestSlurmCheck(t *testing.T) {
}
}
func TestSlurmShared(t *testing.T) {
base := SlurmSettings{Enabled: true, Controller: "c", Nodes: []string{"a"}}
for _, sh := range []SharedStorage{{}, {Path: "/shared"}, {Path: "/shared", NFS: "filer.site:/export/hpc"},
{Path: "/data/hpc", Server: "c"}, {Path: "/shared", Server: "a"}} {
s := base
s.Shared = sh
if err := s.Check(); err != nil {
t.Errorf("%+v refused: %v", sh, err)
}
}
for want, sh := range map[string]SharedStorage{
"path is empty": {NFS: "f:/x"},
"plain names": {Path: "shared"},
"system's own": {Path: "/home"},
"one of them": {Path: "/shared", NFS: "f:/x", Server: "c"},
"not host:/path": {Path: "/shared", NFS: "f/x"},
"neither": {Path: "/shared", Server: "z"},
} {
s := base
s.Shared = sh
if err := s.Check(); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("%+v: got %v, want %q", sh, err, want)
}
}
s := base
s.Shared = SharedStorage{Path: "/a/../etc"}
if s.Check() == nil {
t.Error("a path with .. taken")
}
s.Shared, s.Users = SharedStorage{}, "ldap"
if s.Check() == nil {
t.Error("slurm.users ldap taken")
}
s.Users, s.Shared = UsersLlmgw, SharedStorage{Path: "/shared"}
if s.Check() != nil || s.HomeDir("ayse") != "/shared/home/ayse" {
t.Errorf("home %s", s.HomeDir("ayse"))
}
s.Shared = SharedStorage{}
if s.HomeDir("ayse") != "/home/ayse" {
t.Errorf("home %s", s.HomeDir("ayse"))
}
}
func TestSlurmAllNodes(t *testing.T) {
s := SlurmSettings{Enabled: true, Controller: "c", AllNodes: true, Lent: []string{"joined"}}
if err := s.Check(); err != nil {
+13
View File
@@ -98,6 +98,14 @@ type API struct {
usage Usage
quotas Quotas
ctl *control.Controller
// slurmUsers are the Linux users made from the accounts; nil without a store.
slurmUsers SlurmUsers
// prices are suggested from what each model held and served; nil
// without a store.
prices PriceStore
priceCfg func() PriceConfig
// webhooks counts the settings' notify.webhooks; nil does not say.
webhooks func() int
}
// New builds the API over a controller.
@@ -139,6 +147,7 @@ func (a *API) Register(mux Mux) {
a.registerTenancy(mux)
a.registerTrain(mux)
a.registerSlurm(mux)
a.registerPrices(mux)
a.registerGuard(mux)
a.registerSettings(mux)
a.registerAuth(mux)
@@ -745,6 +754,10 @@ func (a *API) Nodes() []NodeView {
}
if h := n.Host; h.OSID != "" {
v.OS = strings.TrimSpace(h.OSID + " " + h.OSVersion + " " + h.Arch)
if h.CardWithoutDriver() {
v.Warnings = append(v.Warnings, "an NVIDIA card is on the PCI bus but no driver is loaded: "+
"this node serves on CPU only (install the driver, reboot, check nvidia-smi, restart the agent)")
}
if h.Arch == "amd64" && !h.HasCPUFlag("avx2") {
v.Warnings = append(v.Warnings, "no AVX2: inference runs several times slower "+
"(a VM needs CPU type \"host\")")
+4
View File
@@ -6,6 +6,10 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
)
// SetWebhooks has the summary say when the settings name no webhook:
// count is how many they name now.
func (a *API) SetWebhooks(count func() int) { a.webhooks = count }
// SetNotifier gives the API the webhooks `ctl notify test` reaches.
func (a *API) SetNotifier(n *notify.Notifier) { a.notifier = n }
+108
View File
@@ -0,0 +1,108 @@
package adminapi
import (
"context"
"net/http"
"strconv"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// PriceStore is what a price is suggested from: what each model held of
// the cluster, and the tokens it served.
type PriceStore interface {
HeldSince(ctx context.Context, since time.Time) ([]store.Held, time.Time, error)
UsageByModel(ctx context.Context, since time.Time) ([]store.ModelUsage, error)
}
// PriceConfig is the cost basis and the prices in force.
type PriceConfig struct {
Basis accounting.Basis
Current map[string]accounting.Price
Default accounting.Price
}
// PriceSuggestions is a model's price from its cost, for each model.
type PriceSuggestions struct {
// Days is the window; RecordedFrom the first hour held is known for,
// later than the window's start on a cluster that began keeping it
// since, which makes the window shorter than asked.
Days int `json:"days"`
Since time.Time `json:"since"`
RecordedFrom time.Time `json:"recorded_from,omitempty"`
GPUHourCost float64 `json:"gpu_hour_cost"`
CoreHourCost float64 `json:"core_hour_cost"`
Models []PriceSuggestion `json:"models"`
}
// PriceSuggestion is one model's.
type PriceSuggestion struct {
accounting.Suggestion
Current accounting.Price `json:"current"`
}
// SetPrices gives the API what prices are suggested from.
func (a *API) SetPrices(st PriceStore, cfg func() PriceConfig) { a.prices, a.priceCfg = st, cfg }
func (a *API) registerPrices(mux Mux) {
mux.HandleFunc("GET /api/v1/prices/suggest", a.suggestPrices)
}
func (a *API) suggestPrices(w http.ResponseWriter, r *http.Request) {
if a.prices == nil || a.priceCfg == nil {
writeError(w, http.StatusNotImplemented, "prices are suggested from the store, which this control plane runs without")
return
}
days := 30
if v := r.URL.Query().Get("days"); v != "" {
n, err := strconv.Atoi(v)
if err != nil || n < 1 || n > 366 {
writeError(w, http.StatusBadRequest, "days is a whole number of days, 1 to 366")
return
}
days = n
}
cfg := a.priceCfg()
if cfg.Basis.GPUHour <= 0 && cfg.Basis.CoreHour <= 0 {
writeError(w, http.StatusConflict, "no cost to price from: set auth.gpu_hour_cost (and auth.core_hour_cost for "+
"models on CPUs), what an hour of one costs, in the units budgets are in")
return
}
since := time.Now().Add(-time.Duration(days) * 24 * time.Hour)
rows, first, err := a.prices.HeldSince(r.Context(), since)
if err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
// Tokens from when holding was first recorded, so both sides cover the
// same hours.
from := since
if first.After(from) {
from = first
}
usage, err := a.prices.UsageByModel(r.Context(), from)
if err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
held := map[string]accounting.Held{}
for _, h := range rows {
held[h.Model] = accounting.Held{GPUSeconds: h.GPUSeconds, CoreSeconds: h.CoreSeconds}
}
served := map[string]accounting.Served{}
for _, u := range usage {
served[u.Model] = accounting.Served{Prompt: u.Prompt, Completion: u.Completion}
}
out := PriceSuggestions{Days: days, Since: since.UTC(), RecordedFrom: first,
GPUHourCost: cfg.Basis.GPUHour, CoreHourCost: cfg.Basis.CoreHour, Models: []PriceSuggestion{}}
for _, s := range accounting.Suggest(held, served, cfg.Basis) {
cur, ok := cfg.Current[s.Model]
if !ok {
cur = cfg.Default
}
out.Models = append(out.Models, PriceSuggestion{Suggestion: s, Current: cur})
}
writeJSON(w, http.StatusOK, out)
}
+16
View File
@@ -85,6 +85,7 @@ var PathParams = map[string]string{
"projects/{id}": "The project's ID.",
"keys/{id}": "The key's ID, as listed (not the key itself).",
"users/{id}": "The person's ID or email.",
"users/{user}": "The person's ID or email.",
"artifacts/{kind}": "What the job made: peft, a gzipped tar of the PEFT adapter, or gguf, the adapter for llama.cpp.",
}
@@ -398,6 +399,21 @@ var Routes = []Route{
"(drained in Slurm, no job left), lending (its replicas moving off), lent, reclaiming (drained in Slurm, jobs " +
"still running) or unknown (its agent has not said).",
Response: slurmsvc.Status{}},
{Method: "GET", Path: "/api/v1/slurm/users", Summary: "The Linux users made from llmgw's accounts for the Slurm cluster",
Description: "Every active account, with its login, UID and SSH keys; an account is given its login and UID the first " +
"time it is listed. The machines have them when slurm.users is \"llmgw\".",
Response: []slurmsvc.User{}},
{Method: "PUT", Path: "/api/v1/slurm/users/{user}/ssh-keys", Summary: "Replace an account's SSH public keys",
Description: "{user} is an ID or e-mail. keys is OpenSSH public keys, one a line, without options; empty takes them " +
"all away. The machines have them within a minute.",
Request: SlurmKeysRequest{}, Response: slurmsvc.User{}},
{Method: "GET", Path: "/api/v1/prices/suggest", Summary: "Each model's price, from what it held of the cluster",
Description: "days (30 if not given) is the window. A model's cost is the GPU- and core-hours its replicas held, idle " +
"ones too, at auth.gpu_hour_cost and auth.core_hour_cost; spread over its tokens, an output token four input " +
"ones and a cached prompt token a tenth of one, it is price, in micro-units a thousand tokens as " +
"auth.pricing takes them. current is the price in force. 409 with no cost set.",
Response: PriceSuggestions{}},
{Method: "POST", Path: "/api/v1/train/jobs", Summary: "Queue a LoRA adapter's training on a node",
Description: "The node trains when its agent is there, nothing else trains on it, and a card has gpu_memory_mib free; " +
+91
View File
@@ -1,15 +1,43 @@
package adminapi
import (
"context"
"encoding/json"
"errors"
"net/http"
"strings"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// SlurmService is the Slurm cluster beside llmgw (BACKLOG P), as the leader
// keeps it.
type SlurmService interface {
Status() slurmsvc.Status
UsersChanged()
}
// SlurmUsers are the Linux users llmgw makes from its accounts.
type SlurmUsers interface {
SlurmUsers(ctx context.Context) ([]store.SlurmUser, error)
SlurmUserByID(ctx context.Context, userID string) (store.SlurmUser, error)
SetSlurmSSHKeys(ctx context.Context, userID string, keys []string) error
}
// SetSlurmUsers enables the Slurm users' endpoints.
func (a *API) SetSlurmUsers(u SlurmUsers) { a.slurmUsers = u }
// SlurmKeysRequest is a person's SSH public keys, one a line, as
// authorized_keys has them; empty takes them all away.
type SlurmKeysRequest struct {
Keys string `json:"keys"`
}
// SlurmUserView is the user from a store row.
func SlurmUserView(u store.SlurmUser) slurmsvc.User {
return slurmsvc.User{Login: u.Login, UID: u.UID, Name: u.Name, Email: u.Email, SSHKeys: u.SSHKeys}
}
// SetSlurm gives the API the Slurm service.
@@ -17,6 +45,69 @@ func (a *API) SetSlurm(s SlurmService) { a.slurm = s }
func (a *API) registerSlurm(mux Mux) {
mux.HandleFunc("GET /api/v1/slurm", a.slurmStatus)
mux.HandleFunc("GET /api/v1/slurm/users", a.slurmUserList)
mux.HandleFunc("PUT /api/v1/slurm/users/{user}/ssh-keys", a.slurmUserKeys)
}
func (a *API) slurmUserList(w http.ResponseWriter, r *http.Request) {
if a.slurmUsers == nil {
writeError(w, http.StatusNotImplemented, "the Slurm users need the store, which this control plane runs without")
return
}
us, err := a.slurmUsers.SlurmUsers(r.Context())
if err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
out := make([]slurmsvc.User, 0, len(us))
for _, u := range us {
out = append(out, SlurmUserView(u))
}
writeJSON(w, http.StatusOK, out)
}
// slurmUserKeys replaces a person's SSH keys; {user} is an ID or e-mail.
func (a *API) slurmUserKeys(w http.ResponseWriter, r *http.Request) {
if a.slurmUsers == nil {
writeError(w, http.StatusNotImplemented, "the Slurm users need the store, which this control plane runs without")
return
}
if !a.tenancyReady(w) {
return
}
var in SlurmKeysRequest
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 64<<10)).Decode(&in); err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
keys, err := slurm.ParseSSHKeys(in.Keys)
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
ctx := r.Context()
u, err := a.findUser(ctx, r.PathValue("user"))
if err != nil {
writeError(w, http.StatusNotFound, strings.TrimPrefix(err.Error(), store.ErrNotFound.Error()+": "))
return
}
if err := a.slurmUsers.SetSlurmSSHKeys(ctx, u.ID, keys); err != nil {
code := http.StatusInternalServerError
if errors.Is(err, store.ErrNotFound) {
code = http.StatusNotFound
}
writeError(w, code, err.Error())
return
}
if a.slurm != nil {
a.slurm.UsersChanged()
}
su, err := a.slurmUsers.SlurmUserByID(ctx, u.ID)
if err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
writeJSON(w, http.StatusOK, SlurmUserView(su))
}
func (a *API) slurmStatus(w http.ResponseWriter, _ *http.Request) {
+6
View File
@@ -50,6 +50,9 @@ type Summary struct {
// GuardOff says the guardrails are turned off (guard.enabled: false):
// no request is checked, whatever the policies say.
GuardOff bool `json:"guard_off,omitempty"`
// NoAlerts says the cluster's settings name no webhook: a model parked,
// a node lost or a judge down is told nobody.
NoAlerts bool `json:"no_alerts,omitempty"`
}
// StoreMember is one member of a replicated store, as its Patroni reports it.
@@ -160,6 +163,9 @@ func (a *API) Summary() Summary {
if a.guard != nil && a.guard.Library != nil && !a.guard.Library().On() {
s.GuardOff = true
}
if a.webhooks != nil && a.webhooks() == 0 {
s.NoAlerts = true
}
for _, j := range a.Judges() {
if j.Problem != "" {
s.Judges = append(s.Judges, j)
+13
View File
@@ -153,3 +153,16 @@ func TestOneSharedCopyCountsAsMuchAsTwenty(t *testing.T) {
t.Errorf("counted %d models; three nodes seeing one file is one model", f.Models)
}
}
func TestTheSummarySaysWhenNoWebhookIsSet(t *testing.T) {
a, _ := setup(t, testModels(), gpuNode("n1", 8))
n := 0
a.SetWebhooks(func() int { return n })
if !a.Summary().NoAlerts {
t.Error("no webhook, not said")
}
n = 1
if a.Summary().NoAlerts {
t.Error("a webhook set, yet said there is none")
}
}
+347 -72
View File
@@ -10,15 +10,20 @@ import (
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"regexp"
"slices"
"strconv"
"strings"
"sync"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
@@ -36,6 +41,11 @@ type TrainStore interface {
TrainArtifact(ctx context.Context, job, kind string) ([]byte, error)
TrainArtifactKinds(ctx context.Context, job string) ([]string, error)
DeleteTrainJob(ctx context.Context, id string) error
QueuedTrainJobs(ctx context.Context) ([]store.TrainJob, error)
ActiveTrainJobs(ctx context.Context) ([]store.TrainJob, error)
TrainJobsFor(ctx context.Context, projects []string, createdBy string, limit int) ([]store.TrainJob, error)
SetTrainOwner(ctx context.Context, id, owner string) error
SetTrainNote(ctx context.Context, id, message string) error
}
// TrainAdmin is what the training endpoints need.
@@ -45,6 +55,27 @@ type TrainAdmin struct {
// Mirror keeps what a job made where the nodes fetch weights; nil
// leaves publishing off.
Mirror TrainMirror
// Registry finds the weights to train a catalog model on, from where
// its own came; nil has every job name them.
Registry TrainRegistry
// Log says what publishing by itself could not do.
Log *slog.Logger
mu sync.Mutex
bases map[string]trainBaseInfo // by the model's origin repository
}
// TrainRegistry reads a repository of the model registry.
type TrainRegistry interface {
RepoAt(ctx context.Context, id, rev string) (hf.Repo, error)
}
// trainBaseInfo is the safetensors weights a model is trained on.
type trainBaseInfo struct {
repo, revision string
bytes int64
err error
at time.Time
}
// TrainMirror is the model mirror, as publishing needs it.
@@ -66,11 +97,11 @@ type TrainPublishView struct {
// SetTrain enables the training endpoints.
func (a *API) SetTrain(t *TrainAdmin) { a.train = t }
// SetTrainMirror gives publishing the mirror, which is set up after the
// training endpoints are.
func (a *API) SetTrainMirror(m TrainMirror) {
// SetTrainMirror gives publishing the mirror, and finding a model's
// weights the registry, which are set up after the training endpoints are.
func (a *API) SetTrainMirror(m TrainMirror, reg TrainRegistry) {
if a.train != nil {
a.train.Mirror = m
a.train.Mirror, a.train.Registry = m, reg
}
}
@@ -81,13 +112,23 @@ type TrainRequest struct {
// Model is the catalog model the adapter is for.
Model string `json:"model"`
// BaseRepo and BaseRevision are the weights trained on: the model's own
// on its registry, in safetensors (Hugging Face), not a GGUF.
BaseRepo string `json:"base_repo"`
// on its registry, in safetensors (Hugging Face), not a GGUF. Empty
// finds them from where the model's came: the repository itself, or the
// one its card says it was made from.
BaseRepo string `json:"base_repo,omitempty"`
BaseRevision string `json:"base_revision,omitempty"`
// Node trains it; GPUMemoryMiB is what it needs of a card, which can be
// what an engine there leaves free.
Node string `json:"node"`
GPUMemoryMiB int `json:"gpu_memory_mib"`
// Node trains it; empty is the first with a card that has enough free.
// GPUMemoryMiB is what it needs of a card, which can be what an engine
// there leaves free; 0 is worked out from the weights' size.
Node string `json:"node,omitempty"`
GPUMemoryMiB int `json:"gpu_memory_mib,omitempty"`
// Project is the project the job is started in: its members see and
// manage it, and only one of its jobs trains at a time.
Project string `json:"project,omitempty"`
// Publish puts what the job makes on the mirror and makes it the
// model's adapter by itself when the job is done, restarting a llama.cpp
// model's replicas to load it.
Publish bool `json:"publish,omitempty"`
// Owner is who the adapter will belong to: org:, team:, project: (by
// path or ID), user: or key:. Only callers there see it. When the
// pairs come from one knowledge base, it defaults to that collection's
@@ -129,6 +170,8 @@ type TrainJobView struct {
StartedAt time.Time `json:"started_at,omitzero"`
FinishedAt time.Time `json:"finished_at,omitzero"`
CreatedBy string `json:"created_by,omitempty"`
Project string `json:"project,omitempty"`
Publish bool `json:"publish,omitempty"`
// Artifacts are what it made: peft (a gzipped tar), gguf.
Artifacts []string `json:"artifacts,omitempty"`
}
@@ -137,7 +180,8 @@ func trainView(j store.TrainJob) TrainJobView {
return TrainJobView{ID: j.ID, Name: j.Name, Model: j.Model, BaseRepo: j.BaseRepo, BaseRevision: j.BaseRevision,
Node: j.Node, Owner: j.Owner, GPUMemoryMiB: j.GPUMemoryMiB, Params: j.Params, GGUF: j.GGUF, Pairs: j.Pairs, State: j.State,
Step: j.Step, Steps: j.Steps, Loss: j.Loss, Message: j.Message, PeakMiB: j.PeakMiB, CreatedAt: j.CreatedAt,
StartedAt: j.StartedAt, FinishedAt: j.FinishedAt, CreatedBy: j.CreatedBy}
StartedAt: j.StartedAt, FinishedAt: j.FinishedAt, CreatedBy: j.CreatedBy,
Project: j.Project, Publish: j.Publish}
}
func (a *API) registerTrain(mux Mux) {
@@ -178,50 +222,122 @@ func (a *API) trainCreate(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, err.Error())
return
}
bad := func(format string, args ...any) {
writeError(w, http.StatusBadRequest, fmt.Sprintf(format, args...))
j, err := a.StartTraining(r.Context(), in, actorOf(r))
if err != nil {
writeTrainErr(w, err)
return
}
writeJSON(w, http.StatusCreated, trainView(j))
}
// trainErr is a refusal, with the status it is answered with.
type trainErr struct {
code int
msg string
}
func (e *trainErr) Error() string { return e.msg }
func refused(code int, format string, args ...any) error {
return &trainErr{code: code, msg: fmt.Sprintf(format, args...)}
}
func writeTrainErr(w http.ResponseWriter, err error) {
var te *trainErr
if errors.As(err, &te) {
writeError(w, te.code, te.msg)
return
}
writeError(w, http.StatusInternalServerError, err.Error())
}
// TrainRefusal is the reason a training request was refused, for someone
// to read, and whether it was the request's fault rather than the
// cluster's.
func TrainRefusal(err error) (string, bool) {
if err == nil {
return "", false
}
var te *trainErr
if errors.As(err, &te) {
return te.msg, te.code < 500
}
return err.Error(), false
}
// StartTraining checks a job and queues it, as actor.
func (a *API) StartTraining(ctx context.Context, in TrainRequest, actor string) (store.TrainJob, error) {
if a.train == nil {
return store.TrainJob{}, refused(http.StatusNotImplemented,
"training needs the store and the nodes' agents, which this control plane runs without")
}
bad := func(format string, args ...any) (store.TrainJob, error) {
return store.TrainJob{}, refused(http.StatusBadRequest, format, args...)
}
if !adapterName.MatchString(in.Name) {
bad("an adapter's name is letters, digits, . _ and -, up to 100: %q is not", in.Name)
return
return bad("an adapter's name is letters, digits, . _ and -, up to 100: %q is not", in.Name)
}
m, ok := a.ctl.Catalog().Resolve(in.Model)
if !ok {
bad("no model %s in the catalog", in.Model)
return
return bad("no model %s in the catalog", in.Model)
}
if _, _, _, taken := a.ctl.Catalog().ResolveAdapter(in.Name); taken || in.Name == m.Name {
bad("%s is already a model's or an adapter's name", in.Name)
return
return bad("%s is already a model's or an adapter's name", in.Name)
}
if strings.Count(in.BaseRepo, "/") != 1 || strings.ContainsAny(in.BaseRepo, " :") {
bad("base_repo is the weights' registry repository, owner/name: %q is not", in.BaseRepo)
return
if _, taken := a.ctl.Catalog().Resolve(in.Name); taken {
return bad("%s is already a model's or an adapter's name", in.Name)
}
if _, ok := a.ctl.Inventory().Get(in.Node); !ok {
bad("no node %s", in.Node)
return
queued, err := a.train.Store.QueuedTrainJobs(ctx)
if err != nil {
return store.TrainJob{}, err
}
active, err := a.train.Store.ActiveTrainJobs(ctx)
if err != nil {
return store.TrainJob{}, err
}
for _, j := range append(queued, active...) {
if j.Name == in.Name {
return bad("%s is already being trained (job %s)", in.Name, j.ID)
}
if in.Project != "" && j.Project == in.Project {
return store.TrainJob{}, refused(http.StatusConflict,
"this project already has a model training (%s): one at a time, so that one project does not hold every card", j.Name)
}
}
var weights int64
if in.BaseRepo == "" {
b := a.trainBase(ctx, m)
if b.err != nil {
return store.TrainJob{}, b.err
}
in.BaseRepo, in.BaseRevision, weights = b.repo, b.revision, b.bytes
} else if strings.Count(in.BaseRepo, "/") != 1 || strings.ContainsAny(in.BaseRepo, " :") {
return bad("base_repo is the weights' registry repository, owner/name: %q is not", in.BaseRepo)
}
if in.Node != "" {
if _, ok := a.ctl.Inventory().Get(in.Node); !ok {
return bad("no node %s", in.Node)
}
}
if in.GPUMemoryMiB <= 0 {
bad("gpu_memory_mib is what the job needs of a card, in MiB")
return
if weights <= 0 {
return bad("gpu_memory_mib is what the job needs of a card, in MiB")
}
in.GPUMemoryMiB = TrainMemoryMiB(weights)
}
pairs, colls, err := countPairs(in.Data)
if err != nil {
bad("%v", err)
return
return bad("%v", err)
}
owner, err := a.trainOwner(r.Context(), in, colls)
owner, err := a.trainOwner(ctx, in, colls)
if err != nil {
bad("%v", err)
return
return bad("%v", err)
}
for k, v := range in.Params {
lim, known := trainParams[k]
f, isNum := v.(float64)
if !known || !isNum || f < lim[0] || f > lim[1] {
bad("params.%s: the trainer takes rank, alpha, dropout, epochs, lr, batch, max_len and seed, each a number in its range", k)
return
return bad("params.%s: the trainer takes rank, alpha, dropout, epochs, lr, batch, max_len and seed, each a number in its range", k)
}
}
params := ""
@@ -240,15 +356,132 @@ func (a *API) trainCreate(w http.ResponseWriter, r *http.Request) {
}
j := store.TrainJob{ID: store.NewID("train-"), Name: in.Name, Model: m.Name, BaseRepo: in.BaseRepo,
BaseRevision: in.BaseRevision, Node: in.Node, Owner: owner, GPUMemoryMiB: in.GPUMemoryMiB, Params: params, GGUF: gguf,
Data: []byte(in.Data), Pairs: pairs, CreatedBy: actorOf(r)}
if err := a.train.Store.CreateTrainJob(r.Context(), j); err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
Data: []byte(in.Data), Pairs: pairs, CreatedBy: actor, Project: in.Project, Publish: in.Publish}
if err := a.train.Store.CreateTrainJob(ctx, j); err != nil {
return store.TrainJob{}, err
}
// At once rather than at the next look, when the node is here.
go a.train.Service.Dispatch(context.WithoutCancel(r.Context()))
go a.train.Service.Dispatch(context.WithoutCancel(ctx))
j.State = store.TrainQueued
writeJSON(w, http.StatusCreated, trainView(j))
return j, nil
}
// TrainMemoryMiB is what training a LoRA adapter on weights of this size
// takes of a card: the weights in 16 bits, as much again for what passes
// through them, and the CUDA context. Measured at 2.3 GB for a 0.5B model.
func TrainMemoryMiB(weightBytes int64) int {
mib := int(weightBytes>>20)*2 + 1024
return (mib + 255) / 256 * 256
}
// trainBaseFor is how long what the registry said of a model's weights is
// kept: it changes when the model is updated, which is rare.
const trainBaseFor = time.Hour
// trainBase is the safetensors weights to train m on: its own origin's,
// when they are safetensors, or those its card says it was made from — a
// GGUF's base model.
func (a *API) trainBase(ctx context.Context, m catalog.Model) trainBaseInfo {
if m.Origin == nil || m.Origin.Repo == "" {
return trainBaseInfo{err: refused(http.StatusBadRequest, "%s's weights did not come from the model registry, so "+
"what to train it on is not known: an administrator can name them (llmgw ctl train start --base-repo)", m.Name)}
}
if a.train.Registry == nil {
return trainBaseInfo{err: refused(http.StatusNotImplemented, "the model registry is not reachable from here "+
"(mirror.registry): the weights to train %s on cannot be found", m.Name)}
}
a.train.mu.Lock()
b, ok := a.train.bases[m.Origin.Repo]
a.train.mu.Unlock()
if ok && time.Since(b.at) < trainBaseFor {
return b
}
b = a.findTrainBase(ctx, m)
if b.err != nil {
var te *trainErr
if !errors.As(b.err, &te) {
return b // the registry did not answer: asked again next time
}
}
b.at = time.Now()
a.train.mu.Lock()
if a.train.bases == nil {
a.train.bases = map[string]trainBaseInfo{}
}
a.train.bases[m.Origin.Repo] = b
a.train.mu.Unlock()
return b
}
func (a *API) findTrainBase(ctx context.Context, m catalog.Model) trainBaseInfo {
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
repo, err := a.train.Registry.RepoAt(ctx, m.Origin.Repo, "")
if err != nil {
return trainBaseInfo{err: fmt.Errorf("asking the model registry about %s: %w", m.Origin.Repo, err)}
}
candidates := []hf.Repo{repo}
for _, id := range repo.BaseModel {
if id == repo.ID || strings.Count(id, "/") != 1 {
continue
}
r, err := a.train.Registry.RepoAt(ctx, id, "")
if err != nil {
continue
}
candidates = append(candidates, r)
}
gated := ""
for _, r := range candidates {
var n int64
for _, f := range r.Files {
if strings.HasSuffix(strings.ToLower(f.Path), ".safetensors") {
n += f.Bytes
}
}
if n == 0 {
continue
}
if r.Gated.Is() {
gated = r.ID
continue
}
return trainBaseInfo{repo: r.ID, revision: r.Revision, bytes: n}
}
if gated != "" {
return trainBaseInfo{err: refused(http.StatusBadRequest, "the weights to train %s on, %s, are gated on the registry: "+
"the nodes cannot fetch them without an account that accepted their terms", m.Name, gated)}
}
return trainBaseInfo{err: refused(http.StatusBadRequest, "neither %s, where %s's weights came from, nor the model its card "+
"names has safetensors weights to train on", m.Origin.Repo, m.Name)}
}
// PickTrainNode is the node whose freest card has the most memory free,
// if that is need, among those ready and not busy.
func (a *API) PickTrainNode(need int, busy map[string]bool) string {
best, bestFree := "", 0
for _, n := range a.ctl.Inventory().Nodes() {
if busy[n.ID] || n.State != inventory.Ready {
continue
}
free := 0
if n.Load != nil {
for _, g := range n.Load.GPUs {
if g.MemoryTotalMB > 0 {
free = max(free, int(g.MemoryTotalMB-g.MemoryUsedMB))
}
}
}
if free == 0 {
for _, g := range n.GPUs {
free = max(free, g.MemoryTotalMB-g.MemoryUsedMB)
}
}
if free >= need && free > bestFree {
best, bestFree = n.ID, free
}
}
return best
}
// trainOwner is who a job's adapter will belong to, as scope:ID: one that
@@ -453,73 +686,74 @@ func (a *API) trainPublish(w http.ResponseWriter, r *http.Request) {
if !ok {
return
}
if a.train.Mirror == nil {
writeError(w, http.StatusNotImplemented, "publishing needs the model mirror (mirror.api in the settings); "+
"llmgw ctl train download, then llmgw mirror import there, does it by hand")
v, err := a.publishTrained(r.Context(), j, actorOf(r))
if err != nil {
writeTrainErr(w, err)
return
}
writeJSON(w, http.StatusOK, v)
}
// publishTrained puts what a job made on the mirror and makes it the
// model's adapter: the GGUF for a model llama.cpp serves, the PEFT
// directory for one vLLM serves.
func (a *API) publishTrained(ctx context.Context, j store.TrainJob, actor string) (TrainPublishView, error) {
var none TrainPublishView
if a.train.Mirror == nil {
return none, refused(http.StatusNotImplemented, "publishing needs the model mirror (mirror.api in the settings); "+
"llmgw ctl train download, then llmgw mirror import there, does it by hand")
}
if a.models == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store for the catalog")
return
return none, refused(http.StatusNotImplemented, "this control plane has no store for the catalog")
}
if j.State != store.TrainDone {
writeError(w, http.StatusConflict, "the job has not finished: it is "+j.State)
return
return none, refused(http.StatusConflict, "%s", "the job has not finished: it is "+j.State)
}
m, found := a.ctl.Catalog().Resolve(j.Model)
if !found {
writeError(w, http.StatusConflict, "the model "+j.Model+" is no longer in the catalog")
return
return none, refused(http.StatusConflict, "%s", "the model "+j.Model+" is no longer in the catalog")
}
for _, ad := range m.Adapters {
if ad.Name == j.Name {
writeError(w, http.StatusConflict, j.Name+" is already "+m.Name+"'s adapter; train under another name")
return
return none, refused(http.StatusConflict, "%s", j.Name+" is already "+m.Name+"'s adapter; train under another name")
}
}
llama := slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" })
source := "trained by llmgw job " + j.ID + " on " + j.Node + " from " + j.BaseRepo
var url string
if llama {
b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, "gguf")
b, err := a.train.Store.TrainArtifact(ctx, j.ID, "gguf")
if err != nil {
writeError(w, http.StatusConflict, "the job made no GGUF, which llama.cpp needs; train again without --no-gguf")
return
return none, refused(http.StatusConflict, "the job made no GGUF, which llama.cpp needs; train again without --no-gguf")
}
res, err := a.train.Mirror.Upload(r.Context(), j.Name, j.ID, j.Name+".gguf", b, source)
res, err := a.train.Mirror.Upload(ctx, j.Name, j.ID, j.Name+".gguf", b, source)
if err != nil {
writeError(w, http.StatusBadGateway, "keeping it on the mirror: "+err.Error())
return
return none, refused(http.StatusBadGateway, "%s", "keeping it on the mirror: "+err.Error())
}
url = res.URL
} else {
b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, "peft")
b, err := a.train.Store.TrainArtifact(ctx, j.ID, "peft")
if err != nil {
writeError(w, http.StatusConflict, "the job made no PEFT adapter")
return
return none, refused(http.StatusConflict, "the job made no PEFT adapter")
}
files, err := untarAll(b)
if err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
return none, err
}
for name, data := range files {
res, err := a.train.Mirror.Upload(r.Context(), j.Name, j.ID, name, data, source)
res, err := a.train.Mirror.Upload(ctx, j.Name, j.ID, name, data, source)
if err != nil {
writeError(w, http.StatusBadGateway, "keeping "+name+" on the mirror: "+err.Error())
return
return none, refused(http.StatusBadGateway, "%s", "keeping "+name+" on the mirror: "+err.Error())
}
url = res.URL[:strings.LastIndex(res.URL, "/")+1]
}
}
ad := catalog.Adapter{Name: j.Name, Source: url, Owner: j.Owner}
if err := checkAdapters(r.Context(), m, []catalog.Adapter{ad}); err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
if err := checkAdapters(ctx, m, []catalog.Adapter{ad}); err != nil {
return none, refused(http.StatusBadRequest, "%v", err)
}
if _, err := a.models.SetAdapters(r.Context(), m.Name, append(slices.Clone(m.Adapters), ad), actorOf(r)); err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
if _, err := a.models.SetAdapters(ctx, m.Name, append(slices.Clone(m.Adapters), ad), actor); err != nil {
return none, refused(http.StatusBadRequest, "%v", err)
}
v := TrainPublishView{Adapter: j.Name, Model: m.Name, Source: url, Owner: j.Owner}
if llama {
@@ -528,7 +762,48 @@ func (a *API) trainPublish(w http.ResponseWriter, r *http.Request) {
} else {
v.Note = "ask for model " + j.Name + "; a replica loads it on the first request"
}
writeJSON(w, http.StatusOK, v)
return v, nil
}
func (a *API) trainLog() *slog.Logger {
if a.train.Log == nil {
return slog.New(slog.DiscardHandler)
}
return a.train.Log
}
// TrainDone publishes what a job made, when it asked for that: on the
// mirror, as its model's adapter, and for a llama.cpp model with the
// model's replicas replaced, one at a time beside themselves, to load it.
func (a *API) TrainDone(j store.TrainJob) {
if !j.Publish || a.train == nil {
return
}
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Minute)
defer cancel()
note := "published: ask for model " + j.Name
v, err := a.publishTrained(ctx, j, "training:"+j.ID)
if err != nil {
note = "trained, but publishing it failed: " + err.Error()
a.trainLog().Warn("a trained adapter could not be published", "job", j.ID, "adapter", j.Name, "err", err)
} else if strings.HasPrefix(v.Note, "llama.cpp") {
var ids []string
for _, rep := range a.ctl.Reconciler().ReadyReplicas() {
if rep.Model == v.Model {
ids = append(ids, rep.ID)
}
}
if len(ids) > 0 {
if err := a.ctl.Reconciler().RestartPaced(reconciler.RestartPace{}, false, ids...); err != nil {
note = "published, but " + v.Model + "'s replicas could not be restarted to load it: " + err.Error()
} else {
note = "published: " + v.Model + "'s replicas are restarting to load it; then ask for model " + j.Name
}
}
}
if err := a.train.Store.SetTrainNote(ctx, j.ID, note); err != nil {
a.trainLog().Warn("could not record a trained adapter's publishing", "job", j.ID, "err", err)
}
}
// untarAll reads a gzipped tar of a PEFT directory's files.
+114
View File
@@ -0,0 +1,114 @@
package adminapi
import (
"context"
"errors"
"net/http"
"slices"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// Training from the portal's "customise a model" page: the page decides who
// may do what (members of the job's project), and these do it.
// TrainableModels are the catalog's models an adapter can be trained for
// from the portal: chat models llama.cpp or vLLM serve whose weights came
// from the registry, so the weights to train on can be found.
func (a *API) TrainableModels() []string {
var out []string
for _, m := range a.ctl.Catalog().Models() {
if m.IsGuard() || m.IsExternal() || m.Capabilities.Embeddings || m.MakesImages() || m.MakesMeshes() ||
m.Origin == nil || m.Origin.Repo == "" {
continue
}
if slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" || p.Engine == "vllm" }) {
out = append(out, m.Name)
}
}
slices.Sort(out)
return out
}
// Trainings are the jobs started in projects, or by createdBy outside any.
func (a *API) Trainings(ctx context.Context, projects []string, createdBy string) ([]store.TrainJob, error) {
if a.train == nil {
return nil, nil
}
return a.train.Store.TrainJobsFor(ctx, projects, createdBy, 100)
}
// Training is one job.
func (a *API) Training(ctx context.Context, id string) (store.TrainJob, error) {
if a.train == nil {
return store.TrainJob{}, store.ErrNotFound
}
return a.train.Store.TrainJob(ctx, id)
}
// Published is whether a job's adapter is its model's in the catalog.
func (a *API) Published(j store.TrainJob) bool {
m, ok := a.ctl.Catalog().Resolve(j.Model)
return ok && slices.ContainsFunc(m.Adapters, func(ad catalog.Adapter) bool { return ad.Name == j.Name })
}
// ShareTraining changes who a job's adapter belongs to — owner as scope:ID,
// or "" for every caller its model allows — on the job and, once
// published, in the catalog. Refused wider than the knowledge bases its
// pairs came from may be read.
func (a *API) ShareTraining(ctx context.Context, id, owner, actor string) error {
j, err := a.Training(ctx, id)
if err != nil {
return err
}
_, colls, err := countPairs(string(j.Data))
if err != nil {
return err
}
resolved, err := a.trainOwner(ctx, TrainRequest{Owner: owner, Public: owner == ""}, colls)
if err != nil {
return refused(http.StatusBadRequest, "%v", err)
}
if err := a.train.Store.SetTrainOwner(ctx, id, resolved); err != nil {
return err
}
m, ok := a.ctl.Catalog().Resolve(j.Model)
if !ok || a.models == nil {
return nil
}
ads := slices.Clone(m.Adapters)
i := slices.IndexFunc(ads, func(ad catalog.Adapter) bool { return ad.Name == j.Name })
if i < 0 {
return nil
}
ads[i].Owner = resolved
_, err = a.models.SetAdapters(ctx, m.Name, ads, actor)
return err
}
// DeleteTraining stops a job if it trains, takes its adapter out of the
// catalog if it was published, and forgets it.
func (a *API) DeleteTraining(ctx context.Context, id, actor string) error {
j, err := a.Training(ctx, id)
if err != nil {
return err
}
if !store.TrainOver(j.State) {
if err := a.train.Service.Cancel(ctx, id); err != nil {
return err
}
}
if m, ok := a.ctl.Catalog().Resolve(j.Model); ok && a.models != nil {
ads := slices.DeleteFunc(slices.Clone(m.Adapters), func(ad catalog.Adapter) bool { return ad.Name == j.Name })
if len(ads) != len(m.Adapters) {
if _, err := a.models.SetAdapters(ctx, m.Name, ads, actor); err != nil {
return err
}
}
}
if err := a.train.Store.DeleteTrainJob(ctx, id); err != nil && !errors.Is(err, store.ErrNotFound) {
return err
}
return nil
}
@@ -0,0 +1,137 @@
package adminapi
import (
"context"
"errors"
"path/filepath"
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
type fakeRegistry struct {
repos map[string]hf.Repo
asked int
}
func (f *fakeRegistry) RepoAt(_ context.Context, id, _ string) (hf.Repo, error) {
f.asked++
r, ok := f.repos[id]
if !ok {
return hf.Repo{}, errors.New("404")
}
return r, nil
}
func TestAPortalTrainingFindsItsWeightsAndMemoryAndOneTrainsPerProject(t *testing.T) {
ctx := context.Background()
models := append(testModels(),
catalog.Model{Name: "small", Source: "/s", Origin: &catalog.Origin{Repo: "bartowski/Small-GGUF"},
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}},
catalog.Model{Name: "gated", Source: "/g", Origin: &catalog.Origin{Repo: "meta/Gated"},
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}},
catalog.Model{Name: "local", Source: "/l", Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}})
a, _ := setup(t, models, gpuNode("cuda01", 1))
s, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "s.db"))
if err != nil {
t.Fatal(err)
}
defer s.Close()
reg := &fakeRegistry{repos: map[string]hf.Repo{
"bartowski/Small-GGUF": {ID: "bartowski/Small-GGUF", Revision: "g1", BaseModel: []string{"Org/Small"},
Files: []hf.File{{Path: "small-Q4_K_M.gguf", Bytes: 400 << 20}}},
"Org/Small": {ID: "Org/Small", Revision: "abc", Files: []hf.File{{Path: "model.safetensors", Bytes: 1000 << 20},
{Path: "config.json", Bytes: 1 << 10}}},
"meta/Gated": {ID: "meta/Gated", Gated: hf.Gated{Reason: "manual"},
Files: []hf.File{{Path: "model.safetensors", Bytes: 1 << 30}}},
}}
a.SetTrain(&TrainAdmin{Store: s, Service: train.New(s, noAgents{}, nil)})
a.SetTrainMirror(nil, reg)
if got := a.TrainableModels(); strings.Join(got, ",") != "gated,small" {
t.Errorf("trainable: %v", got)
}
data := train.NDJSON([]train.Pair{{Q: "a", A: "b"}})
req := TrainRequest{Name: "tone", Model: "small", Data: data, Owner: "project:p1", Project: "p1", Publish: true}
j, err := a.StartTraining(ctx, req, "ayse@x")
if err != nil {
t.Fatal(err)
}
if j.BaseRepo != "Org/Small" || j.BaseRevision != "abc" || j.Node != "" || j.GPUMemoryMiB != TrainMemoryMiB(1000<<20) ||
!j.Publish || j.Project != "p1" || j.CreatedBy != "ayse@x" || !j.GGUF {
t.Errorf("queued %+v", j)
}
if j.GPUMemoryMiB != 3072 {
t.Errorf("1000 MiB of weights take %d MiB to train", j.GPUMemoryMiB)
}
req.Name = "tone2"
if _, err := a.StartTraining(ctx, req, "mehmet@x"); err == nil || !strings.Contains(err.Error(), "already has a model training") {
t.Errorf("a second in the project: %v", err)
}
req.Name, req.Project, req.Owner = "tone", "p2", "project:p2"
if _, err := a.StartTraining(ctx, req, "x"); err == nil || !strings.Contains(err.Error(), "already being trained") {
t.Errorf("the same name twice: %v", err)
}
req.Name = "tone3"
asked := reg.asked
if _, err := a.StartTraining(ctx, req, "x"); err != nil {
t.Errorf("another project: %v", err)
}
if reg.asked != asked {
t.Errorf("the registry was asked again: %d", reg.asked-asked)
}
for model, says := range map[string]string{"gated": "gated", "local": "did not come from the model registry"} {
req.Model, req.Name, req.Project = model, "x-"+model, "p-"+model
_, err := a.StartTraining(ctx, req, "x")
if err == nil || !strings.Contains(err.Error(), says) {
t.Errorf("%s: %v", model, err)
}
if msg, theirs := TrainRefusal(err); !theirs || msg == "" {
t.Errorf("%s is not the request's fault: %q", model, msg)
}
}
jobs, err := a.Trainings(ctx, []string{"p1"}, "nobody")
if err != nil || len(jobs) != 1 || jobs[0].Name != "tone" {
t.Errorf("p1's: %+v %v", jobs, err)
}
if err := a.ShareTraining(ctx, j.ID, "", "ayse@x"); err != nil {
t.Fatal(err)
}
if got, _ := a.Training(ctx, j.ID); got.Owner != "" {
t.Errorf("shared with everyone: %q", got.Owner)
}
if err := a.DeleteTraining(ctx, j.ID, "ayse@x"); err != nil {
t.Fatal(err)
}
if _, err := a.Training(ctx, j.ID); !errors.Is(err, store.ErrNotFound) {
t.Errorf("deleted: %v", err)
}
}
func TestATrainingGoesToTheNodeWithTheMostFree(t *testing.T) {
busy := gpuNode("busy", 1)
busy.Load = &inventory.Load{GPUs: []inventory.GPUSensor{{Index: 0, MemoryTotalMB: 24000, MemoryUsedMB: 23000}}}
roomy := gpuNode("roomy", 1)
roomy.Load = &inventory.Load{GPUs: []inventory.GPUSensor{{Index: 0, MemoryTotalMB: 24000, MemoryUsedMB: 4000}}}
cpu := inventory.Node{ID: "cpu", State: inventory.Ready}
small := gpuNode("small", 1)
small.GPUs = []discovery.GPU{{Index: 0, MemoryTotalMB: 8000}}
a, _ := setup(t, testModels(), busy, roomy, cpu, small)
if got := a.PickTrainNode(3000, nil); got != "roomy" {
t.Errorf("picked %q", got)
}
if got := a.PickTrainNode(3000, map[string]bool{"roomy": true}); got != "small" {
t.Errorf("with roomy busy, picked %q", got)
}
if got := a.PickTrainNode(30000, nil); got != "" {
t.Errorf("none has 30 GB, picked %q", got)
}
}
+1
View File
@@ -36,6 +36,7 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
n.Host = discovery.Host{
OSID: h.GetOsId(), OSVersion: h.GetOsVersion(), OSCodename: h.GetOsCodename(),
Arch: h.GetArch(), CPUFlags: h.GetCpuFlags(), NVIDIADriver: h.GetNvidiaDriver(),
NVIDIADevice: h.GetNvidiaDevice(),
MemBandwidthGBs: h.GetMemBandwidthGbs(), MemBandwidthCoreGBs: h.GetMemBandwidthCoreGbs(),
}
}
+93
View File
@@ -0,0 +1,93 @@
// Package held records, on the leader, what each model holds of the
// cluster: its replicas' GPUs and, for one on CPUs, its cores. Kept by the
// hour, it is the cost side of a token's price (llmgw ctl prices suggest).
package held
import (
"context"
"log/slog"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// Inventory is where the allocations are.
type Inventory interface {
Nodes() []inventory.Node
Allocations(nodeID string) []inventory.Allocation
}
// Store keeps what was held.
type Store interface {
AddHeld(ctx context.Context, at time.Time, held []store.Held) error
}
// Recorder adds, every Every, what each model holds for that long.
type Recorder struct {
Inv Inventory
// Models is each replica's model, by replica id.
Models func() map[string]string
Store Store
Leader func() bool
Every time.Duration
Log *slog.Logger
}
// Run records until ctx ends.
func (r *Recorder) Run(ctx context.Context) {
every := r.Every
if every <= 0 {
every = 5 * time.Minute
}
t := time.NewTicker(every)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case now := <-t.C:
if r.Leader != nil && !r.Leader() {
continue
}
held := Sample(r.Inv, r.Models(), every)
if err := r.Store.AddHeld(ctx, now, held); err != nil && r.Log != nil {
r.Log.Warn("could not record what the models held, for their prices", "err", err)
}
}
}
}
// Sample is what each model holds now, for d: a GPU for each card a shard
// was given, and the cores of a shard given cores and no GPU. A replica
// whose model is not known (headroom kept for another, one just reaped) is
// left out.
func Sample(inv Inventory, models map[string]string, d time.Duration) []store.Held {
by := map[string]*store.Held{}
var order []string
secs := int64(d.Seconds())
for _, n := range inv.Nodes() {
for _, a := range inv.Allocations(n.ID) {
m, ok := models[a.ReplicaID]
if !ok {
continue
}
h := by[m]
if h == nil {
h = &store.Held{Model: m}
by[m] = h
order = append(order, m)
}
if g := len(a.GPUIndices); g > 0 {
h.GPUSeconds += int64(g) * secs
} else {
h.CoreSeconds += int64(max(a.CPUCores, 1)) * secs
}
}
}
out := make([]store.Held, 0, len(order))
for _, m := range order {
out = append(out, *by[m])
}
return out
}
+40
View File
@@ -0,0 +1,40 @@
package held
import (
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
type inv map[string][]inventory.Allocation
func (i inv) Nodes() []inventory.Node {
var out []inventory.Node
for id := range i {
out = append(out, inventory.Node{ID: id})
}
return out
}
func (i inv) Allocations(id string) []inventory.Allocation { return i[id] }
func TestSampleCountsCardsAndCores(t *testing.T) {
in := inv{
"gpu1": {{ReplicaID: "big-1", GPUIndices: []int{0, 1}}, {ReplicaID: "headroom", GPUIndices: []int{2}}},
"gpu2": {{ReplicaID: "big-2", GPUIndices: []int{0, 1}}},
"cpu1": {{ReplicaID: "small-1", CPUCores: 4}, {ReplicaID: "tiny-1"}},
}
models := map[string]string{"big-1": "big", "big-2": "big", "small-1": "small", "tiny-1": "tiny"}
got := map[string]store.Held{}
for _, h := range Sample(in, models, 5*time.Minute) {
got[h.Model] = h
}
if got["big"].GPUSeconds != 4*300 || got["small"].CoreSeconds != 4*300 || got["tiny"].CoreSeconds != 300 {
t.Errorf("held %+v", got)
}
if len(got) != 3 {
t.Errorf("headroom counted: %+v", got)
}
}
+358 -24
View File
@@ -27,6 +27,7 @@ import (
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
@@ -62,25 +63,37 @@ const (
// NodeStatus is one node as the service sees it.
type NodeStatus struct {
Node string `json:"node"`
Phase Phase `json:"phase"`
Replicas int `json:"replicas"`
Node string `json:"node"`
Phase Phase `json:"phase"`
Replicas int `json:"replicas"`
// StopsBy is, while it is being lent, when a replica still on it is
// stopped even with nowhere else to go: the drain's grace after the
// lend began. Zero when not known (a leader that took over meanwhile).
StopsBy time.Time `json:"stops_by,omitempty"`
Status *Reported `json:"slurm,omitempty"`
Connected bool `json:"connected"`
}
// Reported is the agent's SlurmStatus, for the admin API.
type Reported struct {
Version string `json:"version,omitempty"`
Hardware string `json:"hardware,omitempty"`
SlurmdRunning bool `json:"slurmd_running"`
SlurmctldRunning bool `json:"slurmctld_running"`
State string `json:"state,omitempty"`
Reason string `json:"reason,omitempty"`
Jobs int `json:"jobs"`
Error string `json:"error,omitempty"`
FilesSHA256 string `json:"files_sha256,omitempty"`
At time.Time `json:"at"`
Version string `json:"version,omitempty"`
Hardware string `json:"hardware,omitempty"`
SlurmdRunning bool `json:"slurmd_running"`
SlurmctldRunning bool `json:"slurmctld_running"`
SlurmdbdRunning bool `json:"slurmdbd_running"`
// Shared is the shared path here: "mounted", "exported", "missing".
Shared string `json:"shared,omitempty"`
// Containers is what the machine has for jobs in containers, empty
// while not both of pyxis and enroot.
Containers string `json:"containers,omitempty"`
// Users is how many of llmgw's users the machine has.
Users int `json:"users"`
State string `json:"state,omitempty"`
Reason string `json:"reason,omitempty"`
Jobs int `json:"jobs"`
Error string `json:"error,omitempty"`
FilesSHA256 string `json:"files_sha256,omitempty"`
At time.Time `json:"at"`
}
// Status is the whole Slurm cluster, for `ctl slurm status`.
@@ -94,6 +107,26 @@ type Status struct {
// Waiting are nodes left out of slurm.conf until they report their
// hardware.
Waiting []string `json:"waiting,omitempty"`
// Shared is where the shared files are and come from; Users who keeps
// the users, and how many llmgw has.
Shared config.SharedStorage `json:"shared"`
Users string `json:"users,omitempty"`
UserCount int `json:"user_count,omitempty"`
// Containers is whether jobs may run in containers: "on" once every
// machine has pyxis and enroot, "installing" until then, empty when
// slurm.containers is off.
Containers string `json:"containers,omitempty"`
// Accounts is how many of Slurm's accounts are llmgw's projects'.
Accounts int `json:"accounts,omitempty"`
}
// User is a Linux user made from an llmgw account.
type User struct {
Login string `json:"login"`
UID int `json:"uid"`
Name string `json:"name"`
Email string `json:"email"`
SSHKeys []string `json:"ssh_keys,omitempty"`
}
// Service keeps the Slurm cluster.
@@ -110,6 +143,8 @@ type Service struct {
// drained are nodes this service put in Draining, which it alone makes
// ready again.
drained map[string]bool
// lendingAt is when this leader began moving a node's replicas off.
lendingAt map[string]time.Time
// sent is what each node was last told, so it is told again only when
// that changes or its stream is new.
sent map[string]*llmgwv1.Slurm
@@ -126,6 +161,151 @@ type Service struct {
// started is the first pass, from which a node neither heard from nor
// kept is waited for, for HoldFor.
started time.Time
// userList reads the users from the accounts; users is what it last
// said, read at usersAt. Until it has been read once no node is told
// anything of users: an empty list would take every user away.
userList func(ctx context.Context) ([]User, error)
users []User
usersAt time.Time
usersOK bool
// accountList reads Slurm's accounts from the projects, with the users;
// accountsOK is false until it has been read, and the controller is
// told nothing of accounts until then: none would take every one away.
accountList func(ctx context.Context) ([]slurm.Account, error)
accounts []slurm.Account
accountsOK bool
// jobSink records the jobs that ended; recorded are those it took,
// recording whether a recording is under way.
jobSink func(ctx context.Context, jobs []JobRecord) error
recorded map[string]time.Time
recording bool
}
// JobRecord is a job that ended, for llmgw's accounting.
type JobRecord struct {
ID, Login, Account, Partition, State string
Start, End time.Time
Elapsed time.Duration
CPUs, GPUs int
}
// SetAccounts has Slurm's accounts read with list, beside the users.
func (s *Service) SetAccounts(list func(ctx context.Context) ([]slurm.Account, error)) {
s.mu.Lock()
s.accountList = list
s.mu.Unlock()
}
// SetJobs has the jobs that ended recorded with record; each is given to it
// once it took it without an error.
func (s *Service) SetJobs(record func(ctx context.Context, jobs []JobRecord) error) {
s.mu.Lock()
s.jobSink = record
s.mu.Unlock()
}
// AccountsOf are the accounts a login may run jobs on.
func (s *Service) AccountsOf(login string) []string {
s.mu.Lock()
defer s.mu.Unlock()
var out []string
for _, a := range s.accounts {
if slices.Contains(a.Users, login) {
out = append(out, a.Name)
}
}
return out
}
// UsersEvery is how often the users are read again without being asked.
var UsersEvery = 30 * time.Second
// SetUsers has the users read with list, for slurm.users "llmgw".
func (s *Service) SetUsers(list func(ctx context.Context) ([]User, error)) {
s.mu.Lock()
s.userList = list
s.mu.Unlock()
}
// UsersChanged has the users read again now: an account or its keys
// changed.
func (s *Service) UsersChanged() {
s.mu.Lock()
s.usersAt = time.Time{}
s.mu.Unlock()
s.Kick()
}
// readUsers reads the users when they are due. Under s.mu.
func (s *Service) readUsers(ctx context.Context, set config.SlurmSettings) {
if set.Users != config.UsersLlmgw || s.userList == nil || time.Since(s.usersAt) < UsersEvery {
return
}
us, err := s.userList(ctx)
s.usersAt = time.Now()
if err != nil {
s.log.Warn("could not read the Slurm users from the accounts; the nodes keep the last ones", "err", err)
return
}
s.users, s.usersOK = us, true
if s.accountList == nil {
return
}
as, err := s.accountList(ctx)
if err != nil {
s.log.Warn("could not read Slurm's accounts from the projects; the controller keeps the last ones", "err", err)
return
}
s.accounts, s.accountsOK = as, true
}
// takeJobs records the jobs a controller says ended that are not recorded
// yet, away from the agent's stream.
func (s *Service) takeJobs(ended []*llmgwv1.SlurmJob) {
s.mu.Lock()
if s.jobSink == nil || s.recording {
s.mu.Unlock()
return
}
if s.recorded == nil {
s.recorded = map[string]time.Time{}
}
for id, at := range s.recorded {
if time.Since(at) > 3*24*time.Hour {
delete(s.recorded, id)
}
}
var jobs []JobRecord
for _, j := range ended {
if _, ok := s.recorded[j.GetId()]; ok || j.GetId() == "" {
continue
}
jobs = append(jobs, JobRecord{ID: j.GetId(), Login: j.GetUser(), Account: j.GetAccount(), Partition: j.GetPartition(),
State: j.GetState(), Start: time.Unix(j.GetStartUnix(), 0), End: time.Unix(j.GetEndUnix(), 0),
Elapsed: time.Duration(j.GetElapsedSeconds()) * time.Second, CPUs: int(j.GetCpus()), GPUs: int(j.GetGpus())})
}
if len(jobs) == 0 {
s.mu.Unlock()
return
}
sink := s.jobSink
s.recording = true
s.mu.Unlock()
go func() {
ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
defer cancel()
err := sink(ctx, jobs)
s.mu.Lock()
defer s.mu.Unlock()
s.recording = false
if err != nil {
s.log.Warn("could not record the Slurm jobs that ended; tried again with the next status", "jobs", len(jobs), "err", err)
return
}
for _, j := range jobs {
s.recorded[j.ID] = time.Now()
}
}()
}
// HoldFor is how long a new leader waits for a node it neither has heard
@@ -143,7 +323,7 @@ func New(settings func() config.SlurmSettings, inv Inventory, agents Agents,
log = slog.Default()
}
return &Service{settings: settings, inv: inv, agents: agents, key: key, log: log,
status: map[string]*Reported{}, drained: map[string]bool{}, sent: map[string]*llmgwv1.Slurm{},
status: map[string]*Reported{}, drained: map[string]bool{}, lendingAt: map[string]time.Time{}, sent: map[string]*llmgwv1.Slurm{},
kick: make(chan struct{}, 1)}
}
@@ -197,12 +377,17 @@ func (s *Service) Kick() {
// OnSlurmStatus records what a node's agent says.
func (s *Service) OnSlurmStatus(node string, st *llmgwv1.SlurmStatus) {
r := &Reported{Version: st.GetVersion(), Hardware: st.GetHardware(), SlurmdRunning: st.GetSlurmdRunning(),
SlurmctldRunning: st.GetSlurmctldRunning(), State: st.GetState(), Reason: st.GetReason(),
Shared: st.GetShared(), Users: int(st.GetUsers()), Containers: st.GetContainers(),
SlurmctldRunning: st.GetSlurmctldRunning(), SlurmdbdRunning: st.GetSlurmdbdRunning(), State: st.GetState(), Reason: st.GetReason(),
Jobs: int(st.GetJobs()), Error: st.GetError(), FilesSHA256: st.GetFilesSha256(), At: time.Now()}
s.mu.Lock()
prev := s.status[node]
s.status[node] = r
isCtl := s.settings().Controller == node
s.mu.Unlock()
if isCtl && len(st.GetEnded()) > 0 {
s.takeJobs(st.GetEnded())
}
if r.Error != "" && (prev == nil || prev.Error != r.Error) {
s.log.Warn("a node could not keep its part in Slurm", "node", node, "err", r.Error)
}
@@ -226,6 +411,7 @@ func (s *Service) Tick(ctx context.Context) {
s.mu.Lock()
defer s.mu.Unlock()
s.recall(ctx)
s.readUsers(ctx, set)
if s.started.IsZero() {
s.started = time.Now()
}
@@ -345,11 +531,14 @@ func (s *Service) phase(set config.SlurmSettings, node string) Phase {
if ok && (n.State == inventory.Ready || n.State == inventory.Cordoned) {
s.inv.SetState(node, inventory.Draining)
s.drained[node] = true
s.log.Info("lending a node to Slurm: its replicas are moved off first", "node", node)
s.lendingAt[node] = time.Now()
s.log.Info("lending a node to Slurm: its replicas are moved off first", "node", node,
"stopped_by", time.Now().Add(drainGrace).Format(time.TimeOnly))
}
if !ok || len(s.inv.Allocations(node)) > 0 {
return Lending
}
delete(s.lendingAt, node)
return Lent
}
// Not lent. Until the agent says Slurm has drained it with no job left,
@@ -369,12 +558,17 @@ func (s *Service) phase(set config.SlurmSettings, node string) Phase {
}
return Reclaiming
}
delete(s.lendingAt, node)
if s.drained[node] {
s.ready(node)
}
return Llmgw
}
// drainGrace is how long a drained replica waits for its replacement
// before it is stopped anyway: the reconciler's, which is not a setting.
var drainGrace = reconciler.DefaultConfig().DrainGrace
// slurmBusy is whether Slurm may still run work on a node: a job step left
// on it (or no telling), or a controller that would still give it one. A
// node the controller does not know of, or cannot be asked about, gets no
@@ -404,7 +598,26 @@ func (s *Service) files(set config.SlurmSettings) (map[string]string, []string)
if c.ControllerAddr == "" {
return nil, waiting
}
return c.Files(), waiting
f := c.Files()
if s.containersReady(set) {
f["plugstack.conf"] = slurm.PlugstackConf
}
return f, waiting
}
// containersReady is whether jobs may ask for containers: every machine of
// the Slurm cluster says it has pyxis and enroot. Slurm fails a job on a
// machine without the plugin it was told to load. Under s.mu.
func (s *Service) containersReady(set config.SlurmSettings) bool {
if !set.Containers {
return false
}
for _, n := range s.machines(set) {
if st := s.status[n]; st == nil || st.Containers == "" {
return false
}
}
return true
}
// cluster is what slurm.conf says: each node as it last reported, else as
@@ -415,15 +628,19 @@ func (s *Service) cluster(set config.SlurmSettings) (slurm.Cluster, []string) {
return c, nil
}
c.ControllerAddr = s.controllerAddr(set)
c.EnforceAccounts = set.Users == config.UsersLlmgw && s.accountList != nil
var waiting []string
for _, node := range set.Nodes {
if sn, ok := s.reported(node); ok {
c.Nodes = append(c.Nodes, sn)
} else if sn, ok := s.known.Nodes[node]; ok {
c.Nodes = append(c.Nodes, sn)
} else {
waiting = append(waiting, node)
sn, ok := s.reported(node)
if !ok {
sn, ok = s.known.Nodes[node]
}
if !ok {
waiting = append(waiting, node)
continue
}
sn.Lent = set.IsLent(node)
c.Nodes = append(c.Nodes, sn)
}
return c, waiting
}
@@ -459,6 +676,7 @@ func (s *Service) message(set config.SlurmSettings, node string, lent bool, key
return m
}
m.Name, m.MungeKey, m.Lent = node, key, lent
m.Containers = set.Containers
m.Node = isNode
if addr := s.controllerAddr(set); addr != "" {
m.ConfServer = net.JoinHostPort(addr, strconv.Itoa(slurm.CtldPort))
@@ -473,25 +691,135 @@ func (s *Service) message(set config.SlurmSettings, node string, lent bool, key
m.Controller = true
m.Files = maps.Clone(files)
}
if !s.sharedPart(set, node, m) || !s.usersPart(set, node, m) {
return nil
}
if isCtl && set.Users == config.UsersLlmgw && s.accountList != nil {
if !s.accountsOK {
return nil
}
m.ManageAccounts = true
for _, a := range s.accounts {
m.Accounts = append(m.Accounts, &llmgwv1.SlurmAccount{Name: a.Name, Parent: a.Parent, Users: slices.Clone(a.Users)})
}
}
return m
}
// sharedPart puts the shared files in m, false when where they come from
// is not known yet: told nothing rather than no shared files, which would
// unmount them under running jobs. Under s.mu.
func (s *Service) sharedPart(set config.SlurmSettings, node string, m *llmgwv1.Slurm) bool {
sh := set.Shared
if sh.Path == "" {
return true
}
m.Shared = &llmgwv1.SlurmShared{Path: sh.Path, Nfs: sh.NFS}
switch {
case sh.Server == "":
case sh.Server == node:
m.Shared.Serve = true
for _, n := range s.machines(set) {
if n == node {
continue
}
if a := s.addr(set, n); a != "" {
m.Shared.Clients = append(m.Shared.Clients, a)
}
}
slices.Sort(m.Shared.Clients)
default:
a := s.addr(set, sh.Server)
if a == "" {
return false
}
if strings.Contains(a, ":") {
a = "[" + a + "]"
}
m.Shared.Nfs = a + ":" + sh.Path
}
return true
}
// usersPart puts the users in m, false while they have not been read.
// Under s.mu.
func (s *Service) usersPart(set config.SlurmSettings, node string, m *llmgwv1.Slurm) bool {
if set.Users != config.UsersLlmgw {
return true
}
if !s.usersOK {
return false
}
m.ManageUsers = true
switch {
case set.Shared.Path == "":
m.MakeHomes = true // each machine's own /home
case set.Shared.Server != "":
m.MakeHomes = set.Shared.Server == node
default:
m.MakeHomes = set.Controller == node
}
for _, u := range s.users {
m.Users = append(m.Users, &llmgwv1.SlurmUser{Login: u.Login, Uid: uint32(u.UID), Name: u.Name, //nolint:gosec // from FirstSlurmUID up
Home: set.HomeDir(u.Login), SshKeys: slices.Clone(u.SSHKeys)})
}
return true
}
// machines are the controller and the nodes.
func (s *Service) machines(set config.SlurmSettings) []string {
out := slices.Clone(set.Nodes)
if !slices.Contains(out, set.Controller) {
out = append(out, set.Controller)
}
return out
}
// addr is a machine's management address, from the inventory, else kept
// from before. Under s.mu.
func (s *Service) addr(set config.SlurmSettings, node string) string {
if n, ok := s.inv.Get(node); ok && n.MgmtAddr != "" {
return n.MgmtAddr
}
if node == set.Controller {
return s.controllerAddr(set)
}
return s.known.Nodes[node].Addr
}
// Status is the cluster as the service sees it now.
func (s *Service) Status() Status {
set := s.current()
s.mu.Lock()
defer s.mu.Unlock()
out := Status{Enabled: set.Enabled, Cluster: set.Cluster(), Controller: set.Controller,
Partition: set.PartitionName(), Ctld: s.status[set.Controller]}
Partition: set.PartitionName(), Ctld: s.status[set.Controller], Shared: set.Shared, Users: set.Users}
if set.Users == config.UsersLlmgw {
out.UserCount = len(s.users)
for _, a := range s.accounts {
if a.Project != "" {
out.Accounts++
}
}
}
if !set.Enabled {
return out
}
if set.Containers {
out.Containers = "installing"
if s.containersReady(set) {
out.Containers = "on"
}
}
_, out.Waiting = s.files(set)
conn := s.agents.ConnectedNodes()
for _, node := range set.Nodes {
ns := NodeStatus{Node: node, Status: s.status[node], Replicas: len(s.inv.Allocations(node)),
Connected: slices.Contains(conn, node)}
ns.Phase = s.peek(set, node)
if at, ok := s.lendingAt[node]; ok && ns.Phase == Lending {
ns.StopsBy = at.Add(drainGrace)
}
out.Nodes = append(out.Nodes, ns)
}
return out
@@ -519,6 +847,12 @@ func (s *Service) peek(set config.SlurmSettings, node string) Phase {
func (n NodeStatus) Describe() string {
switch n.Phase {
case Lending:
if !n.StopsBy.IsZero() {
// A replica moves once its replacement serves; one with nowhere
// else to go waits out the grace, and that wait is not a hang.
return fmt.Sprintf("being lent: %d replica(s) still to move off; one with nowhere else to go is stopped at %s",
n.Replicas, n.StopsBy.Local().Format(time.TimeOnly))
}
return fmt.Sprintf("being lent: %d replica(s) still to move off", n.Replicas)
case Lent:
return "lent to Slurm"
+108
View File
@@ -5,11 +5,13 @@ import (
"slices"
"strings"
"testing"
"time"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
type fakeInv struct {
@@ -272,3 +274,109 @@ func TestAllNodes(t *testing.T) {
t.Fatalf("cuda02 not lent: %+v", p)
}
}
func TestLendingSaysWhenAReplicaWithNowhereToGoStops(t *testing.T) {
// Seen on the PoC: a GPU model with no other card waited out the drain's
// grace, and five minutes of "1 replica(s) still to move off" read as a hang.
at := time.Date(2026, 11, 14, 19, 38, 34, 0, time.Local)
n := NodeStatus{Node: "cuda01", Phase: Lending, Replicas: 1, StopsBy: at}
if d := n.Describe(); !strings.Contains(d, "nowhere else to go is stopped at 19:38:34") {
t.Errorf("%q", d)
}
n.StopsBy = time.Time{}
if d := n.Describe(); d != "being lent: 1 replica(s) still to move off" {
t.Errorf("%q", d)
}
}
func TestPyxisIsLoadedOnlyOnceEveryMachineHasIt(t *testing.T) {
// "required" in plugstack.conf fails every job on a machine without the
// plugin; sent early, it would stop Slurm's jobs everywhere it is missing.
r := newRig()
r.set.Containers = true
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw})
r.tick()
if !r.agents.last("cuda01").GetContainers() || !r.agents.last("node06").GetContainers() {
t.Fatal("the machines were not told to set containers up")
}
if _, ok := r.agents.last("node06").GetFiles()["plugstack.conf"]; ok {
t.Fatal("pyxis loaded before any machine has it")
}
if s := r.svc.Status().Containers; s != "installing" {
t.Errorf("status %q", s)
}
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, Containers: "pyxis 0.24.0, enroot 4.2.1"})
r.tick()
if _, ok := r.agents.last("node06").GetFiles()["plugstack.conf"]; ok {
t.Fatal("pyxis loaded with the controller still without it")
}
r.report("node06", &llmgwv1.SlurmStatus{Containers: "pyxis 0.24.0, enroot 4.2.1"})
r.tick()
if p := r.agents.last("node06").GetFiles()["plugstack.conf"]; !strings.Contains(p, "required /usr/local/lib/slurm/spank_pyxis.so") {
t.Fatalf("plugstack.conf %q", p)
}
if s := r.svc.Status().Containers; s != "on" {
t.Errorf("status %q", s)
}
r.set.Containers = false
r.tick()
if _, ok := r.agents.last("node06").GetFiles()["plugstack.conf"]; ok || r.svc.Status().Containers != "" {
t.Fatal("pyxis still loaded with containers off")
}
}
func TestTheControllerKeepsTheProjectsAccountsAndJobsAreRecordedOnce(t *testing.T) {
r := newRig()
r.set.Users = config.UsersLlmgw
r.svc.SetUsers(func(context.Context) ([]User, error) { return []User{{Login: "ayse", UID: 200001}}, nil })
read := 0
r.svc.SetAccounts(func(context.Context) ([]slurm.Account, error) {
read++
return []slurm.Account{{Name: "test", Parent: "root"},
{Name: "test-arge-deneme", Parent: "test", Project: "p1", Users: []string{"ayse"}}}, nil
})
got := make(chan []JobRecord, 4)
r.svc.SetJobs(func(_ context.Context, jobs []JobRecord) error { got <- jobs; return nil })
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw})
r.tick()
ctl := r.agents.last("node06")
if !ctl.GetManageAccounts() || len(ctl.GetAccounts()) != 2 || ctl.GetAccounts()[1].GetUsers()[0] != "ayse" {
t.Fatalf("controller told %v", ctl.GetAccounts())
}
if r.agents.last("cuda01").GetManageAccounts() {
t.Error("a node was told the accounts")
}
if !strings.Contains(ctl.GetFiles()["slurm.conf"], "AccountingStorageEnforce=associations") {
t.Error("jobs not held to accounts")
}
if r.svc.Status().Accounts != 1 || len(r.svc.AccountsOf("ayse")) != 1 {
t.Errorf("status %d, ayse's %v", r.svc.Status().Accounts, r.svc.AccountsOf("ayse"))
}
ended := []*llmgwv1.SlurmJob{{Id: "41", User: "ayse", Account: "test-arge-deneme", ElapsedSeconds: 60, Gpus: 1}}
r.report("node06", &llmgwv1.SlurmStatus{Ended: ended})
select {
case jobs := <-got:
if len(jobs) != 1 || jobs[0].Login != "ayse" || jobs[0].GPUs != 1 {
t.Fatalf("jobs %+v", jobs)
}
case <-time.After(5 * time.Second):
t.Fatal("not recorded")
}
// The next status says it again: not recorded twice.
deadline := time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) {
r.svc.mu.Lock()
busy := r.svc.recording
r.svc.mu.Unlock()
if !busy {
break
}
time.Sleep(10 * time.Millisecond)
}
r.report("node06", &llmgwv1.SlurmStatus{Ended: ended})
select {
case jobs := <-got:
t.Fatalf("recorded again: %+v", jobs)
case <-time.After(200 * time.Millisecond):
}
}
+85
View File
@@ -0,0 +1,85 @@
package slurmsvc
import (
"context"
"errors"
"slices"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
)
func TestSharedFilesFromTheController(t *testing.T) {
r := newRig()
r.set.Shared = config.SharedStorage{Path: "/shared", Server: "node06"}
r.tick()
ctl, node := r.agents.last("node06").GetShared(), r.agents.last("cuda01").GetShared()
if !ctl.GetServe() || !slices.Equal(ctl.GetClients(), []string{"10.0.0.147"}) || ctl.GetPath() != "/shared" {
t.Fatalf("server told %v", ctl)
}
if node.GetServe() || node.GetNfs() != "10.0.0.146:/shared" {
t.Fatalf("client told %v", node)
}
// The site's own export: every machine mounts it.
r.set.Shared = config.SharedStorage{Path: "/shared", NFS: "filer:/hpc"}
r.tick()
if r.agents.last("node06").GetShared().GetNfs() != "filer:/hpc" || r.agents.last("cuda01").GetShared().GetServe() {
t.Fatal("the site's export not sent")
}
}
func TestSharedServerNotKnownTellsNothing(t *testing.T) {
r := newRig()
r.set.Shared = config.SharedStorage{Path: "/shared", Server: "node06"}
delete(r.inv.nodes, "node06")
r.svc.known.ControllerAddr, r.svc.known.Controller = "10.0.0.146", "node06"
r.tick()
if r.agents.last("cuda01").GetShared().GetNfs() != "10.0.0.146:/shared" {
t.Fatalf("the kept address not used: %v", r.agents.last("cuda01"))
}
r2 := newRig()
r2.set.Nodes = []string{"cuda01", "site2"}
r2.agents.conn = append(r2.agents.conn, "site2")
r2.set.Shared = config.SharedStorage{Path: "/shared", Server: "site2"}
r2.tick()
if r2.agents.last("cuda01") != nil {
t.Fatal("told with no address for the server: its mount would go")
}
}
func TestUsersSentOnceRead(t *testing.T) {
r := newRig()
r.set.Users = config.UsersLlmgw
r.set.Shared = config.SharedStorage{Path: "/shared", Server: "node06"}
fail := true
r.svc.SetUsers(func(context.Context) ([]User, error) {
if fail {
return nil, errors.New("store down")
}
return []User{{Login: "ayse", UID: 200000, Name: "Ayşe", SSHKeys: []string{"ssh-ed25519 AAAA"}}}, nil
})
r.tick()
if len(r.agents.got["cuda01"]) != 0 {
t.Fatal("told before the users were read: every user would go")
}
fail = false
r.svc.UsersChanged()
r.tick()
ctl, node := r.agents.last("node06"), r.agents.last("cuda01")
if !ctl.GetManageUsers() || !ctl.GetMakeHomes() || node.GetMakeHomes() || len(node.GetUsers()) != 1 {
t.Fatalf("controller %v\nnode %v", ctl, node)
}
if u := node.GetUsers()[0]; u.GetHome() != "/shared/home/ayse" || u.GetUid() != 200000 || len(u.GetSshKeys()) != 1 {
t.Fatalf("user %v", u)
}
// The store down later: the last users are kept.
fail = true
r.svc.UsersChanged()
r.tick()
if len(r.agents.last("cuda01").GetUsers()) != 1 {
t.Fatal("users lost when the store could not be read")
}
if st := r.svc.Status(); st.UserCount != 1 || st.Shared.Server != "node06" {
t.Fatalf("status %+v", st)
}
}
+205
View File
@@ -0,0 +1,205 @@
package train
import (
"bufio"
"bytes"
"encoding/csv"
"encoding/json"
"errors"
"fmt"
"path"
"strings"
"unicode/utf8"
"gitea.mynodes.xyz/ilker/llmgw/internal/kb/extract"
)
// Pairs from a file a person uploads: a spreadsheet of questions and
// answers (Excel, CSV, tab-separated) or llmgw's own NDJSON of {"q", "a"}.
// The columns are found by their header — Soru/Cevap, Question/Answer and
// the like — or are the first two when the first row is not a header.
// Pair is one question and the answer to learn for it.
type Pair struct {
Q string `json:"q"`
A string `json:"a"`
}
// Read is what a file gave.
type Read struct {
Pairs []Pair
// Skipped counts rows with a question and no answer, or the other way.
Skipped int
// Columns names the columns read, as the header has them; empty for
// NDJSON and for a table with no header.
Columns [2]string
}
// MaxPairs bounds one file: past it a job trains for hours on a page that
// was meant to take minutes.
const MaxPairs = 20000
var (
questionNames = []string{"soru", "sorular", "question", "questions", "q", "prompt", "instruction", "input", "girdi"}
answerNames = []string{"cevap", "cevaplar", "yanıt", "yanit", "answer", "answers", "a", "response", "output", "completion", "çıktı", "cikti"}
)
// ReadPairs reads a file's pairs; name says what kind it is.
func ReadPairs(name string, b []byte) (Read, error) {
var rows [][]string
switch strings.ToLower(path.Ext(name)) {
case ".xlsx", ".xlsm":
r, err := extract.XLSXRows(b)
if err != nil {
return Read{}, fmt.Errorf("the Excel file could not be read: %w", err)
}
rows = r
case ".jsonl", ".ndjson", ".json":
return readNDJSON(b)
case ".csv", ".tsv", ".txt":
r, err := readDelimited(b)
if err != nil {
return Read{}, err
}
rows = r
case ".xls":
return Read{}, errors.New("an old Excel file (.xls): save it as .xlsx and choose it again")
default:
return Read{}, errors.New("choose an Excel (.xlsx), CSV or JSONL file")
}
return fromRows(rows)
}
func readDelimited(b []byte) ([][]string, error) {
b = bytes.TrimPrefix(b, []byte("\xef\xbb\xbf"))
if !utf8.Valid(b) {
return nil, errors.New("the file is not UTF-8 text: save it as \"CSV UTF-8\"")
}
first, _, _ := strings.Cut(string(b), "\n")
// Whichever separator the header line has most of: Excel writes ";"
// where the decimal mark is a comma, as it is in Turkish.
sep, most := ',', strings.Count(first, ",")
for _, c := range []rune{';', '\t'} {
if n := strings.Count(first, string(c)); n > most {
sep, most = c, n
}
}
r := csv.NewReader(bytes.NewReader(b))
r.Comma, r.FieldsPerRecord, r.LazyQuotes = sep, -1, true
rows, err := r.ReadAll()
if err != nil {
return nil, fmt.Errorf("the file could not be read as a table: %w", err)
}
return rows, nil
}
func fromRows(rows [][]string) (Read, error) {
var out Read
qc, ac := 0, 1
if len(rows) > 0 {
q, a := -1, -1
for i, c := range rows[0] {
n := strings.ToLower(strings.TrimSpace(c))
switch {
case q < 0 && contains(questionNames, n):
q = i
case a < 0 && contains(answerNames, n):
a = i
}
}
if q >= 0 && a >= 0 {
qc, ac = q, a
out.Columns = [2]string{strings.TrimSpace(rows[0][q]), strings.TrimSpace(rows[0][a])}
rows = rows[1:]
}
}
for _, r := range rows {
var q, a string
if qc < len(r) {
q = strings.TrimSpace(r[qc])
}
if ac < len(r) {
a = strings.TrimSpace(r[ac])
}
switch {
case q == "" && a == "":
case q == "" || a == "":
out.Skipped++
default:
out.Pairs = append(out.Pairs, Pair{Q: q, A: a})
}
}
return out, check(out)
}
func contains(names []string, n string) bool {
for _, x := range names {
if x == n {
return true
}
}
return false
}
func readNDJSON(b []byte) (Read, error) {
var out Read
sc := bufio.NewScanner(bytes.NewReader(b))
sc.Buffer(make([]byte, 64<<10), 1<<20)
line := 0
for sc.Scan() {
line++
l := bytes.TrimSpace(sc.Bytes())
if len(l) == 0 {
continue
}
var p struct {
Q, A string
Question, Answer string
Prompt, Completion string
}
if err := json.Unmarshal(l, &p); err != nil {
return Read{}, fmt.Errorf("line %d is not JSON: %v", line, err)
}
q, a := first(p.Q, p.Question, p.Prompt), first(p.A, p.Answer, p.Completion)
if q == "" || a == "" {
out.Skipped++
continue
}
out.Pairs = append(out.Pairs, Pair{Q: q, A: a})
}
if err := sc.Err(); err != nil {
return Read{}, err
}
return out, check(out)
}
func first(s ...string) string {
for _, x := range s {
if x = strings.TrimSpace(x); x != "" {
return x
}
}
return ""
}
func check(r Read) error {
switch {
case len(r.Pairs) == 0:
return errors.New("no question and answer found: the first column is the question and the second the answer, " +
"or columns headed Soru and Cevap")
case len(r.Pairs) > MaxPairs:
return fmt.Errorf("%d pairs, more than the %d one training takes", len(r.Pairs), MaxPairs)
}
return nil
}
// NDJSON is the pairs as a job's data.
func NDJSON(pairs []Pair) string {
var b strings.Builder
enc := json.NewEncoder(&b)
enc.SetEscapeHTML(false)
for _, p := range pairs {
_ = enc.Encode(p)
}
return b.String()
}
+65
View File
@@ -0,0 +1,65 @@
package train
import (
"strings"
"testing"
)
func TestPairsFromATableByItsHeaderOrItsFirstColumns(t *testing.T) {
for _, c := range []struct {
name, file string
want []Pair
skipped int
cols [2]string
}{
{"headed.csv", "No;Cevap;Soru\n1;Yirmi gün.;İzin kaç gün?\n2;;Boş cevap\n3;Evet;Uzaktan çalışabilir miyim?\n",
[]Pair{{"İzin kaç gün?", "Yirmi gün."}, {"Uzaktan çalışabilir miyim?", "Evet"}}, 1, [2]string{"Soru", "Cevap"}},
{"plain.csv", "\ufeffİzin kaç gün?,\"Yirmi gün, yılda.\"\n\nMasraf?,Formla.\n",
[]Pair{{"İzin kaç gün?", "Yirmi gün, yılda."}, {"Masraf?", "Formla."}}, 0, [2]string{}},
{"tabs.tsv", "Question\tAnswer\nA?\tB.\n", []Pair{{"A?", "B."}}, 0, [2]string{"Question", "Answer"}},
{"d.jsonl", `{"q":"A?","a":"B."}` + "\n" + `{"question":"C?","answer":"D."}` + "\n" + `{"q":"E?"}` + "\n",
[]Pair{{"A?", "B."}, {"C?", "D."}}, 1, [2]string{}},
} {
got, err := ReadPairs(c.name, []byte(c.file))
if err != nil {
t.Errorf("%s: %v", c.name, err)
continue
}
if len(got.Pairs) != len(c.want) || got.Skipped != c.skipped || got.Columns != c.cols {
t.Errorf("%s: %+v", c.name, got)
continue
}
for i := range c.want {
if got.Pairs[i] != c.want[i] {
t.Errorf("%s: pair %d = %+v, want %+v", c.name, i, got.Pairs[i], c.want[i])
}
}
}
}
func TestPairsRefusedWithAReason(t *testing.T) {
for name, file := range map[string]string{
"a.csv": "only one column\n",
"a.xls": "x",
"a.pdf": "x",
"a.json": "not json\n",
} {
if _, err := ReadPairs(name, []byte(file)); err == nil {
t.Errorf("%s read", name)
}
}
var b strings.Builder
for range MaxPairs + 1 {
b.WriteString("x,y\n")
}
if _, err := ReadPairs("big.csv", []byte(b.String())); err == nil || !strings.Contains(err.Error(), "more than") {
t.Errorf("too many: %v", err)
}
}
func TestNDJSONIsWhatAJobReads(t *testing.T) {
got := NDJSON([]Pair{{"<b>?", "a & b"}})
if got != `{"q":"<b>?","a":"a & b"}`+"\n" {
t.Errorf("%q", got)
}
}
+35 -1
View File
@@ -35,11 +35,26 @@ type Service struct {
agents Agents
log *slog.Logger
// pick chooses the node for a job that names none, among those not
// busy; "" is none can take it now. Nil leaves such jobs queued.
pick func(needMiB int, busy map[string]bool) string
// done is told of a job that finished, after what it made is kept.
done func(j store.TrainJob)
mu sync.Mutex
// pieces are artifacts arriving, by job and kind.
pieces map[[2]string][]byte
}
// SetPicker has jobs that name no node sent where pick says.
func (s *Service) SetPicker(pick func(needMiB int, busy map[string]bool) string) { s.pick = pick }
// SetOnDone has done told of each job that finishes.
func (s *Service) SetOnDone(done func(j store.TrainJob)) { s.done = done }
// Waiting is what a job that names no node says while no node can take it.
const Waiting = "waiting for a card with enough memory free"
// New is a service.
func New(s Store, a Agents, log *slog.Logger) *Service {
if log == nil {
@@ -81,13 +96,27 @@ func (s *Service) Dispatch(ctx context.Context) {
busy[j.Node] = true
}
for _, q := range queued {
if busy[q.Node] || !s.agents.Connected(q.Node) {
node := q.Node
if node == "" {
if s.pick != nil {
node = s.pick(q.GPUMemoryMiB, busy)
}
if node == "" {
if q.Message != Waiting {
q.Message = Waiting
_ = s.store.UpdateTrainJob(ctx, q)
}
continue
}
}
if busy[node] || !s.agents.Connected(node) {
continue
}
j, err := s.store.TrainJob(ctx, q.ID)
if err != nil {
continue
}
j.Node = node
err = s.agents.Send(j.Node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_TrainStart{TrainStart: &llmgwv1.TrainStart{
JobId: j.ID, BaseRepo: j.BaseRepo, BaseRevision: j.BaseRevision, Data: j.Data, Params: j.Params,
GpuMemoryMib: int32(j.GPUMemoryMiB), Gguf: j.GGUF}}}) //nolint:gosec // MiB
@@ -135,6 +164,11 @@ func (s *Service) OnTrainStatus(node string, st *llmgwv1.TrainStatus) {
}
if err := s.store.UpdateTrainJob(ctx, j); err != nil {
s.log.Warn("could not record a training job's progress", "job", j.ID, "err", err)
return
}
// What it made came before it said it was done, on the same stream.
if j.State == store.TrainDone && s.done != nil {
go s.done(j)
}
}
+38
View File
@@ -122,3 +122,41 @@ func TestCancellingAJob(t *testing.T) {
t.Errorf("the node was not told: %+v", a.sent)
}
}
func TestAJobNamingNoNodeGoesWhereThePickerSaysAndIsToldWhenDone(t *testing.T) {
ctx := context.Background()
svc, s, a := setup(t)
job(t, s, "j1", "")
asked := 0
free := ""
svc.SetPicker(func(need int, busy map[string]bool) string {
asked++
if need != 2000 {
t.Errorf("asked for %d MiB", need)
}
return free
})
done := make(chan store.TrainJob, 1)
svc.SetOnDone(func(j store.TrainJob) { done <- j })
svc.Dispatch(ctx)
if j, _ := s.TrainJob(ctx, "j1"); j.State != store.TrainQueued || j.Message != Waiting || len(a.sent) != 0 {
t.Fatalf("with no card free: %+v, sent %d", j, len(a.sent))
}
free = "gpu2"
a.connected["gpu2"] = true
svc.Dispatch(ctx)
j, _ := s.TrainJob(ctx, "j1")
if len(a.sent) != 1 || j.Node != "gpu2" || j.State != store.TrainStarting {
t.Fatalf("picked: %+v, sent %d", j, len(a.sent))
}
// Its node is the one it was sent to, which alone reports for it.
svc.OnTrainStatus("cuda01", &llmgwv1.TrainStatus{JobId: "j1", State: "done"})
if j, _ := s.TrainJob(ctx, "j1"); j.State == store.TrainDone {
t.Fatal("another node finished it")
}
svc.OnTrainStatus("gpu2", &llmgwv1.TrainStatus{JobId: "j1", State: "done"})
if got := <-done; got.ID != "j1" || got.Node != "gpu2" {
t.Errorf("told of %+v", got)
}
}
+10
View File
@@ -39,6 +39,8 @@ var Reference = []Block{
{Key: "personal", Type: "Personal", Doc: "Personal is what a person may do on their own, outside any project: make\npersonal keys and use the chat page personally, within these limits.", Nested: "config.Personal"},
{Key: "pricing", Type: "map[string]Price", Doc: "Pricing is what each model costs, for budgets. A model absent from the\ntable costs whatever DefaultPrice says, which is nothing unless set.", Nested: ""},
{Key: "default_price", Type: "Price", Doc: "", Nested: "config.Price"},
{Key: "gpu_hour_cost", Type: "float64", Doc: "GPUHourCost is what an hour of one GPU costs the operator, in the\nunits budgets are in: the hardware written off over its years, its\npower and room, and the people who run it, over the GPUs and the\nhours of a month. With CoreHourCost for a CPU core, it is what\n`llmgw ctl prices suggest` prices each model from: the hours its\nreplicas held, idle ones too, over the tokens it served.", Nested: ""},
{Key: "core_hour_cost", Type: "float64", Doc: "", Nested: ""},
{Key: "flush_interval", Type: "time.Duration", Doc: "FlushInterval and BatchSize decide when usage records are written.", Nested: ""},
{Key: "batch_size", Type: "int", Doc: "", Nested: ""},
{Key: "buffer_size", Type: "int", Doc: "BufferSize bounds the queue. Beyond it records are dropped rather than\nblocking a request: losing accounting is bad, turning a stalled database\ninto an outage is worse.", Nested: ""},
@@ -510,6 +512,11 @@ var Reference = []Block{
{Package: "config", Name: "ServingSettings", Title: "ServingSettings", Doc: "", Fields: []Field{
{Key: "stopped", Type: "bool", Doc: "Stopped takes every model down and answers every request for one with\n503 (system_stopped), leaving the control planes, agents and mirror\nrunning and every model's definition as it is: llmgw stop sets it,\nllmgw start clears it, and the models come back as their catalog\nentries ask. In force on every control plane within seconds.", Nested: ""},
}},
{Package: "config", Name: "SharedStorage", Title: "SharedStorage", Doc: "", Fields: []Field{
{Key: "path", Type: "string", Doc: "Path is where every machine sees it, as /shared.", Nested: ""},
{Key: "nfs", Type: "string", Doc: "NFS is the site's export, host:/path, mounted at Path on every\nmachine of the Slurm cluster.", Nested: ""},
{Key: "server", Type: "string", Doc: "Server is the llmgw node, the controller or one of the nodes, that\nexports Path over NFS to the others: for a site with no storage of\nits own. One machine: not highly available, and no faster than its\ndisk and network.", Nested: ""},
}},
{Package: "engines", Name: "SimConfig", Title: "SimConfig", Doc: "", Fields: []Field{
{Key: "bin", Type: "string", Doc: "Bin is the llmgw binary itself; the engine is one of its subcommands.\nEmpty means whatever started this agent, which is right almost always.", Nested: ""},
}},
@@ -521,6 +528,9 @@ var Reference = []Block{
{Key: "all_nodes", Type: "bool", Doc: "AllNodes runs slurmd on every llmgw node, those that join later too,\nbesides any named in Nodes. Each is llmgw's until it is lent, as any.", Nested: ""},
{Key: "lent", Type: "[]string", Doc: "Lent are the nodes of Nodes Slurm may run jobs on now. A node put here\nis drained of llmgw's replicas first, as `ctl drain` does, and resumed\nin Slurm once none is left; a node taken out is drained in Slurm and\ngiven back to llmgw when its last job ends.", Nested: ""},
{Key: "partition", Type: "string", Doc: "Partition is the one partition every node is in. Empty is \"gpu\".", Nested: ""},
{Key: "shared", Type: "SharedStorage", Doc: "Shared is the file system every machine of the Slurm cluster sees at\nthe same path: home directories, datasets, the jobs' output and\ncheckpoints. Empty path: none, and a batch job's output stays on the\nnode it ran on.", Nested: "config.SharedStorage"},
{Key: "users", Type: "string", Doc: "Users is who keeps the Linux users that run jobs, the same on every\nmachine of the Slurm cluster: \"llmgw\" makes one for each active llmgw\naccount (its login from the e-mail address, a UID of llmgw's from\n200000 up, its SSH keys from the portal), let in to the controller,\nand to a node only while a job of theirs runs there (pam_slurm_adopt);\nempty leaves users to the site (a directory through sssd, or by hand).", Nested: ""},
{Key: "containers", Type: "bool", Doc: "Containers lets jobs run in containers: `srun --container-image\nnvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),\nthe GPUs and the shared path inside. The agents install both on\nevery Slurm machine; jobs take it once every machine has them.", Nested: ""},
}},
{Package: "engines", Name: "StrataConfig", Title: "StrataConfig", Doc: "", Fields: []Field{
{Key: "dir", Type: "string", Doc: "Dir is Strata's source directory, where setup.py and serve/ are.", Nested: ""},
+14 -1
View File
@@ -122,6 +122,10 @@ type Options struct {
}
// Collect reads the node.
// driverFix is what to do about a card with no working driver.
const driverFix = "install the NVIDIA driver the card needs (on Ubuntu: apt-get install nvidia-driver-580-open), reboot, " +
"and nvidia-smi must list it; then add the node again, or restart its agent if it was added already"
func Collect(ctx context.Context, o Options) Facts {
f := Facts{At: time.Now().UTC(), Modules: map[string]bool{}, Engines: map[string]string{}}
f.Hostname, _ = os.Hostname()
@@ -372,10 +376,19 @@ func Evaluate(f Facts) []Finding {
// --- GPUs ---
switch {
case !f.SMIFound && f.Host.NVIDIADevice:
add(Fail, "gpu", "an NVIDIA card is on the PCI bus but no driver is installed: this node would serve on CPU only",
driverFix)
case !f.SMIFound:
add(Info, "gpu", "no nvidia-smi: a CPU node, or the driver is not installed", "")
add(Info, "gpu", "no nvidia-smi: a CPU node", "")
case f.GPUError != "" && f.Host.CardWithoutDriver():
add(Fail, "gpu", "an NVIDIA card is on the PCI bus but its driver is not loaded: "+f.GPUError,
"reboot if the driver was just installed; with Secure Boot on, the module must be signed (mokutil --sb-state); "+
"check that nouveau is not holding the card (lsmod | grep nouveau)")
case f.GPUError != "":
add(Fail, "gpu", "nvidia-smi fails: "+f.GPUError, "check the driver: run nvidia-smi by hand")
case len(f.GPUs) == 0 && f.Host.NVIDIADevice:
add(Fail, "gpu", "an NVIDIA card is on the PCI bus but nvidia-smi reports no GPUs", driverFix)
case len(f.GPUs) == 0:
add(Warn, "gpu", "nvidia-smi is installed but reports no GPUs", "")
default:
+24
View File
@@ -276,3 +276,27 @@ func TestTheNodeAndItsFabricsAreReadySeparately(t *testing.T) {
t.Errorf("%+v", r)
}
}
func TestACardWithoutADriverIsAFailureNotACPUNode(t *testing.T) {
// The mistake this guards: a GPU node whose driver was never installed
// onboarded quietly as a CPU node, with nothing anywhere saying why.
f := healthy()
f.SMIFound, f.GPUs, f.GPUError = false, nil, ""
f.Host.NVIDIADevice, f.Host.NVIDIADriver = true, ""
fs := Evaluate(f)
has(t, fs, Fail, "an NVIDIA card is on the PCI bus but no driver is installed")
f.Host.NVIDIADevice = false
fs = Evaluate(f)
if bad := levels(fs, Fail); len(bad) > 0 {
t.Errorf("a plain CPU node was faulted: %v", bad)
}
has(t, fs, Info, "no nvidia-smi: a CPU node")
// nvidia-smi installed, the module not loaded: the message points at the
// usual reasons rather than at a generic "run nvidia-smi".
f.SMIFound, f.GPUError = true, "running nvidia-smi: exit status 9"
f.Host.NVIDIADevice = true
fs = Evaluate(f)
has(t, fs, Fail, "its driver is not loaded")
}
+23
View File
@@ -7,6 +7,7 @@ import (
"encoding/base64"
"errors"
"os"
"reflect"
"strings"
"testing"
@@ -321,3 +322,25 @@ func TestHelpersWhenInstalled(t *testing.T) {
}
wants(t, got, "LibreOffice yolu")
}
func TestWorkbookRowsKeepTheirColumns(t *testing.T) {
b := zipOf(t,
"xl/workbook.xml", `<workbook xmlns:r="r"><sheets><sheet name="Sorular" sheetId="1" r:id="rId1"/></sheets></workbook>`,
"xl/_rels/workbook.xml.rels", `<Relationships><Relationship Id="rId1" Target="worksheets/sheet1.xml"/></Relationships>`,
"xl/sharedStrings.xml", `<sst><si><t>Soru</t></si><si><t>Cevap</t></si><si><t>İzin kaç gün?</t></si></sst>`,
"xl/worksheets/sheet1.xml", `<worksheet><sheetData>`+
`<row r="1"><c r="A1" t="s"><v>0</v></c><c r="B1" t="s"><v>1</v></c></row>`+
`<row r="2"/>`+
`<row r="3"><c r="A3" t="s"><v>2</v></c><c r="C3" t="inlineStr"><is><t>not</t></is></c></row>`+
`<row r="4"><c r="B4"><v>20</v></c></row>`+
`</sheetData></worksheet>`,
)
rows, err := XLSXRows(b)
if err != nil {
t.Fatal(err)
}
want := [][]string{{"Soru", "Cevap"}, {"İzin kaç gün?", "", "not"}, {"", "20"}}
if !reflect.DeepEqual(rows, want) {
t.Errorf("rows = %q, want %q", rows, want)
}
}
+114 -23
View File
@@ -9,6 +9,7 @@ import (
"io"
"path"
"regexp"
"slices"
"sort"
"strconv"
"strings"
@@ -227,29 +228,7 @@ func xlsxText(b []byte) (string, error) {
if err != nil {
return "", err
}
var shared []string
if ss, _ := member(zr, "xl/sharedStrings.xml"); ss != nil {
var cur strings.Builder
inT, inRPh := false, 0
_ = walkXML(ss, func(e xmlEvent) {
switch {
case e.start && e.name == "si":
cur.Reset()
case e.end && e.name == "si":
shared = append(shared, cur.String())
case e.start && e.name == "rPh":
inRPh++ // phonetic guides, not the cell's text
case e.end && e.name == "rPh":
inRPh--
case e.start && e.name == "t":
inT = true
case e.end && e.name == "t":
inT = false
case !e.start && !e.end && inT && inRPh == 0:
cur.WriteString(e.text)
}
})
}
shared := xlsxShared(zr)
sheets, err := xlsxSheets(zr)
if err != nil {
return "", err
@@ -307,6 +286,118 @@ func xlsxText(b []byte) (string, error) {
return o.String(), nil
}
// xlsxShared is a workbook's shared strings, which cells name by index.
func xlsxShared(zr *zip.Reader) []string {
var shared []string
ss, _ := member(zr, "xl/sharedStrings.xml")
if ss == nil {
return nil
}
var cur strings.Builder
inT, inRPh := false, 0
_ = walkXML(ss, func(e xmlEvent) {
switch {
case e.start && e.name == "si":
cur.Reset()
case e.end && e.name == "si":
shared = append(shared, cur.String())
case e.start && e.name == "rPh":
inRPh++ // phonetic guides, not the cell's text
case e.end && e.name == "rPh":
inRPh--
case e.start && e.name == "t":
inT = true
case e.end && e.name == "t":
inT = false
case !e.start && !e.end && inT && inRPh == 0:
cur.WriteString(e.text)
}
})
return shared
}
// XLSXRows reads a workbook's first sheet as rows of cells, each cell in
// its column (an empty one is ""), for a table such as questions and
// answers. Rows with nothing in them are left out.
func XLSXRows(b []byte) ([][]string, error) {
zr, err := openZip(b)
if err != nil {
return nil, err
}
shared := xlsxShared(zr)
sheets, err := xlsxSheets(zr)
if err != nil {
return nil, err
}
if len(sheets) == 0 {
return nil, errors.New("the workbook has no sheets")
}
data, err := member(zr, sheets[0].path)
if err != nil || data == nil {
return nil, fmt.Errorf("the workbook's first sheet, %s, could not be read", sheets[0].name)
}
var rows [][]string
var row []string
var cell strings.Builder
cellType, col, next := "", 0, 0
inV, inIs := false, false
_ = walkXML(data, func(e xmlEvent) {
switch {
case e.start && e.name == "row":
row, next = nil, 0
case e.end && e.name == "row":
if slices.ContainsFunc(row, func(s string) bool { return s != "" }) {
rows = append(rows, row)
}
case e.start && e.name == "c":
cellType = e.get("t")
col = next
if c, ok := columnOf(e.get("r")); ok {
col = c
}
next = col + 1
cell.Reset()
case e.end && e.name == "c":
v := cell.String()
if cellType == "s" {
if i, err := strconv.Atoi(strings.TrimSpace(v)); err == nil && i >= 0 && i < len(shared) {
v = shared[i]
}
}
if col < 0 || col > 16383 {
return
}
for len(row) <= col {
row = append(row, "")
}
row[col] = strings.TrimSpace(v)
case e.start && e.name == "v":
inV = true
case e.end && e.name == "v":
inV = false
case e.start && e.name == "is":
inIs = true
case e.end && e.name == "is":
inIs = false
case !e.start && !e.end && (inV || inIs):
cell.WriteString(e.text)
}
})
return rows, nil
}
// columnOf is a cell reference's column, from 0: "B7" is 1.
func columnOf(ref string) (int, bool) {
n, i := 0, 0
for ; i < len(ref) && ref[i] >= 'A' && ref[i] <= 'Z'; i++ {
n = n*26 + int(ref[i]-'A'+1)
}
if i == 0 {
return 0, false
}
return n - 1, true
}
type sheetRef struct{ name, path string }
// xlsxSheets lists a workbook's sheets in their order, with the part each is
+7 -1
View File
@@ -79,6 +79,9 @@ type Options struct {
// Addr is how to reach the machine over SSH, and User who to log in as.
Addr, User string
// FallbackUser logs in when User is empty and the ssh config names no
// user for the machine.
FallbackUser string
// Node is its id in the cluster. Empty means its hostname, which is what
// it should be: the node id, the hostname and the certificate's name
// being one string saves an afternoon each time they would differ.
@@ -193,7 +196,7 @@ type Runner struct {
// New makes a Runner that reaches the machine with the system's ssh.
func New(o Options, emit func(Event)) *Runner {
return &Runner{Opts: o, Remote: &SSH{Addr: o.Addr, User: o.User}, Emit: emit}
return &Runner{Opts: o, Remote: &SSH{Addr: o.Addr, User: o.User, FallbackUser: o.FallbackUser}, Emit: emit}
}
func (r *Runner) emit(step, level, detail, fix string) {
@@ -302,6 +305,9 @@ func (r *Runner) connect(ctx context.Context, res *Result) error {
}
func (r *Runner) target() string {
if s, ok := r.Remote.(*SSH); ok {
return s.Target(context.Background())
}
if r.Opts.User == "" {
return r.Opts.Addr
}
+33 -4
View File
@@ -5,6 +5,7 @@ import (
"context"
"os/exec"
"runtime"
"sync"
)
// Remote runs a shell command on the machine being onboarded, with input.
@@ -14,8 +15,39 @@ type Remote interface {
// SSH is Remote through the system's ssh, so the operator's keys, agent and
// ~/.ssh/config apply exactly as they do when they log in by hand.
//
// Addr is looked up in the ssh config first: a Host block whose HostName is
// Addr is connected through by its name, so its User, IdentityFile and
// ProxyJump apply to a machine given by address, without the address
// having to be added to its Host line.
type SSH struct {
Addr, User string
// FallbackUser logs in when User is empty and the ssh config names no
// user for the machine: under sudo, the person who ran it.
FallbackUser string
once sync.Once
target string
}
// Target is what ssh is given: [user@]host, host being the ssh config's
// name for Addr when it has one.
func (s *SSH) Target(ctx context.Context) string {
s.once.Do(func() {
host := s.Addr
if a := configAlias(ctx, s.Addr); a != "" {
host = a
}
u := s.User
if u == "" && configUser(ctx, host) == "" {
u = s.FallbackUser
}
s.target = host
if u != "" {
s.target = u + "@" + host
}
})
return s.target
}
// sshOptions never prompt: a question nobody is there to answer would hang
@@ -37,10 +69,7 @@ var sshOptions = []string{
// Run runs script with sh on the machine. Files travel on standard input,
// not scp, which a hardened sshd may not offer.
func (s *SSH) Run(ctx context.Context, script string, stdin []byte) ([]byte, error) {
target := s.Addr
if s.User != "" {
target = s.User + "@" + s.Addr
}
target := s.Target(ctx)
// "--" before the target: an address beginning with "-" is not read as
// an option.
args := append(append([]string{}, sshOptions...), "--", target, "sh", "-c", shellQuote(script))
+223
View File
@@ -0,0 +1,223 @@
package onboard
import (
"bufio"
"bytes"
"context"
"net"
"os"
"os/exec"
"os/user"
"path/filepath"
"strings"
"time"
)
// Where ssh reads its configuration, and arguments every `ssh -G` here is
// given; variables so a test can point them at a file of its own.
var (
userSSHConfig = func() string { return filepath.Join(sshHome(), ".ssh", "config") }
systemSSHConfig = "/etc/ssh/ssh_config"
sshExtraArgs []string
)
// sshHome is the home ssh itself reads ~/.ssh from: the account's, from the
// password database, not $HOME, which sudo may leave as the caller's.
func sshHome() string {
if u, err := user.Current(); err == nil && u.HomeDir != "" {
return u.HomeDir
}
h, _ := os.UserHomeDir()
return h
}
// configAlias is the ssh config's name for addr: the first concrete Host
// whose HostName is addr, or resolves to the same address. A machine is
// usually written in the config under its name, `Host gpu01` with
// `HostName 10.0.0.21` and the User, IdentityFile or ProxyJump it needs,
// and given to node add by its address, which that block does not match.
// Empty when addr is itself named on a Host line, or no block points at it.
func configAlias(ctx context.Context, addr string) string {
aliases := map[string]bool{}
var order []string
seen := map[string]bool{}
collect := func(path, base string) {
for _, h := range hostNames(path, base, seen, 0) {
if !aliases[h] {
aliases[h] = true
order = append(order, h)
}
}
}
collect(userSSHConfig(), filepath.Join(sshHome(), ".ssh"))
if systemSSHConfig != "" {
collect(systemSSHConfig, filepath.Dir(systemSSHConfig))
}
for h := range aliases {
if strings.EqualFold(h, addr) {
return ""
}
}
if len(order) > 256 {
order = order[:256]
}
// Names are looked up only when no HostName is the address as written,
// and within one budget: a config of many hosts must not hold up a
// connection by seconds each.
var pending []string
hostOf := map[string]string{}
for _, a := range order {
h := sshGet(ctx, a)["hostname"]
if strings.EqualFold(h, addr) {
return a
}
if h != "" {
hostOf[a] = h
pending = append(pending, a)
}
}
if len(pending) == 0 {
return ""
}
lctx, cancel := context.WithTimeout(ctx, 3*time.Second)
defer cancel()
want := lookup(lctx, addr)
if len(want) == 0 {
return ""
}
for _, a := range pending {
for ip := range lookup(lctx, hostOf[a]) {
if want[ip] {
return a
}
}
}
return ""
}
// configUser is the user the ssh config logs in to host as, empty when it
// names none. ssh -G reports the local account when nothing set one, so
// that is taken as none: the config saying `User root` to a root caller is
// the one case read wrongly, and it gets the fallback user instead.
func configUser(ctx context.Context, host string) string {
u := sshGet(ctx, host)["user"]
if cur, err := user.Current(); err == nil && u == cur.Username {
return ""
}
return u
}
// sshGet is ssh's own reading of its configuration for host, keys
// lowercased; empty when ssh cannot say.
func sshGet(ctx context.Context, host string) map[string]string {
args := append(append([]string{"-G"}, sshExtraArgs...), "--", host)
out, err := exec.CommandContext(ctx, "ssh", args...).Output()
if err != nil {
return nil
}
m := map[string]string{}
sc := bufio.NewScanner(bytes.NewReader(out))
for sc.Scan() {
k, v, ok := strings.Cut(sc.Text(), " ")
if k = strings.ToLower(k); ok {
if _, dup := m[k]; !dup {
m[k] = v
}
}
}
return m
}
func lookup(ctx context.Context, host string) map[string]bool {
if ip := net.ParseIP(host); ip != nil {
return map[string]bool{ip.String(): true}
}
addrs, err := net.DefaultResolver.LookupHost(ctx, host)
if err != nil {
return nil
}
m := map[string]bool{}
for _, a := range addrs {
if ip := net.ParseIP(a); ip != nil {
m[ip.String()] = true
}
}
return m
}
// hostNames are the concrete names on the Host lines of an ssh config file
// and the files it includes — no patterns, no negations. base is where a
// relative Include is looked for, as ssh does: ~/.ssh for the user's file,
// /etc/ssh for the system's.
func hostNames(path, base string, seen map[string]bool, depth int) []string {
if depth > 16 || seen[path] {
return nil
}
seen[path] = true
data, err := os.ReadFile(path)
if err != nil {
return nil
}
var names []string
for _, line := range strings.Split(string(data), "\n") {
key, args := configLine(line)
switch key {
case "host":
for _, a := range args {
if a != "" && !strings.ContainsAny(a, "*?!") {
names = append(names, a)
}
}
case "include":
for _, a := range args {
if strings.HasPrefix(a, "~/") {
a = filepath.Join(sshHome(), a[2:])
} else if !filepath.IsAbs(a) {
a = filepath.Join(base, a)
}
matches, _ := filepath.Glob(a)
for _, m := range matches {
names = append(names, hostNames(m, base, seen, depth+1)...)
}
}
}
}
return names
}
// configLine splits one line of ssh config into its keyword, lowercased,
// and arguments: "Host a b", "Host=a b" and quoted arguments alike.
func configLine(line string) (string, []string) {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
return "", nil
}
i := strings.IndexAny(line, " \t=")
if i < 0 {
return strings.ToLower(line), nil
}
key := strings.ToLower(line[:i])
rest := strings.TrimLeft(line[i:], " \t")
rest = strings.TrimPrefix(rest, "=")
var args []string
for rest = strings.TrimSpace(rest); rest != ""; rest = strings.TrimSpace(rest) {
if rest[0] == '"' {
end := strings.IndexByte(rest[1:], '"')
if end < 0 {
args = append(args, rest[1:])
break
}
args = append(args, rest[1:1+end])
rest = rest[2+end:]
continue
}
j := strings.IndexAny(rest, " \t")
if j < 0 {
args = append(args, rest)
break
}
args = append(args, rest[:j])
rest = rest[j:]
}
return key, args
}
+87
View File
@@ -0,0 +1,87 @@
package onboard
import (
"context"
"os"
"os/exec"
"os/user"
"path/filepath"
"testing"
)
// withSSHConfig points configAlias and ssh -G at one file, and nothing else.
func withSSHConfig(t *testing.T, config string, extra map[string]string) {
t.Helper()
if _, err := exec.LookPath("ssh"); err != nil {
t.Skip("no ssh")
}
dir := t.TempDir()
path := filepath.Join(dir, "config")
if err := os.WriteFile(path, []byte(config), 0o600); err != nil {
t.Fatal(err)
}
for name, body := range extra {
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o600); err != nil {
t.Fatal(err)
}
}
oldUser, oldSys, oldArgs := userSSHConfig, systemSSHConfig, sshExtraArgs
userSSHConfig = func() string { return path }
systemSSHConfig = ""
sshExtraArgs = []string{"-F", path}
t.Cleanup(func() { userSSHConfig, systemSSHConfig, sshExtraArgs = oldUser, oldSys, oldArgs })
}
// A machine given by address is reached through the Host block whose
// HostName it is, with that block's user — not the sudo caller's.
func TestSSHTargetUsesConfigAlias(t *testing.T) {
withSSHConfig(t, `
Host *.example
User nobody-wild
Host gpu01 gpu01-alt
HostName 10.9.8.7
User ops
Host gpu02
HostName=10.9.8.8
`, nil)
ctx := context.Background()
cases := []struct {
s *SSH
want string
}{
// The config's User applies by ssh itself: no fallback is added.
{&SSH{Addr: "10.9.8.7", FallbackUser: "ilker"}, "gpu01"},
{&SSH{Addr: "10.9.8.7", User: "root", FallbackUser: "ilker"}, "root@gpu01"},
{&SSH{Addr: "10.9.8.8", FallbackUser: "ilker"}, "ilker@gpu02"},
{&SSH{Addr: "gpu01", FallbackUser: "ilker"}, "gpu01"},
{&SSH{Addr: "10.9.8.9", FallbackUser: "ilker"}, "ilker@10.9.8.9"},
{&SSH{Addr: "10.9.8.9"}, "10.9.8.9"},
}
for _, c := range cases {
if got := c.s.Target(ctx); got != c.want {
t.Errorf("%+v: target %q, want %q", c.s.Addr, got, c.want)
}
}
}
// A block in an included file counts, as it does for ssh.
func TestSSHTargetFollowsInclude(t *testing.T) {
withSSHConfig(t, "Include nodes.conf\n", map[string]string{
"nodes.conf": "Host \"gpu03\"\n HostName 10.9.8.10\n",
})
// ssh resolves a relative Include against ~/.ssh, not -F's directory;
// the scan does the same, so point base at the test's directory.
names := hostNames(userSSHConfig(), filepath.Dir(userSSHConfig()), map[string]bool{}, 0)
if len(names) != 1 || names[0] != "gpu03" {
t.Fatalf("names %q", names)
}
}
func TestConfigUserIgnoresLocalDefault(t *testing.T) {
withSSHConfig(t, "Host x\n HostName 10.9.8.11\n", nil)
if cur, err := user.Current(); err == nil {
if u := configUser(context.Background(), "x"); u != "" {
t.Errorf("config user %q for a block naming none (local %q)", u, cur.Username)
}
}
}
+1
View File
@@ -58,6 +58,7 @@ func (r *Router) registerChat(mux *http.ServeMux, pages *web.AccountPages) {
}))
r.registerImagesPage(mux, pages)
r.registerMeshesPage(mux, pages)
r.registerTrainPage(mux, pages)
if r.store != nil {
pages.SetLicences(r.licenceChoices, r.store)
}
+4 -1
View File
@@ -253,6 +253,9 @@ func (r *Router) Handler() http.Handler {
pages.SetPersonalDefaults(r.personalDefaults)
if r.store != nil {
pages.SetAudit(r.store)
if r.slurmSettings != nil {
pages.SetSlurm(r.store, r.slurmSettings)
}
// A forgotten password: a link mailed when the cluster's
// settings say how, otherwise asked of an operator.
pages.SetPasswordReset(r.store, r.mail, r.mail.BaseURL)
@@ -381,7 +384,7 @@ func (r *Router) Handler() http.Handler {
}
exempt := []string{"/health", "/ready", "/leader", "/v1/openapi.json", "/", "/about", "/install", "/install/", "/_static/", "/_lang/", "/claim/",
"/signup", "/login", "/login/", "/logout", "/profile", "/profile/", "/scim/",
"/admin", "/admin/", "/chat", "/chat/", "/images", "/images/", "/3d", "/3d/", "/org", "/org/", "/setup/"}
"/admin", "/admin/", "/chat", "/chat/", "/images", "/images/", "/3d", "/3d/", "/train", "/train/", "/org", "/org/", "/setup/"}
// The docs are public unless the site said otherwise, and then guarded by
// the session check in front of them — or, on a gateway with keys and no
// accounts to sign in to, by the key like any other path.
+1 -1
View File
@@ -17,7 +17,7 @@ func TestSessionPagesAreNotAskedForAnAPIKey(t *testing.T) {
h := rt.Handler()
// The /admin pages are mounted only with a cluster view, which this
// router has none of; these are the session pages it does mount.
for _, path := range []string{"/profile", "/chat", "/images", "/3d", "/org", "/org/org-x"} {
for _, path := range []string{"/profile", "/chat", "/images", "/3d", "/train", "/org", "/org/org-x"} {
w := httptest.NewRecorder()
h.ServeHTTP(w, httptest.NewRequest(http.MethodGet, path, nil))
if strings.Contains(w.Body.String(), "api key") {
+10
View File
@@ -192,6 +192,9 @@ type Router struct {
// registry browses a model registry from the mirror page. Nil where the
// deployment has no way out to the internet.
registry web.Registry
// training is the "customise a model" page's control plane; nil
// leaves the page off.
training atomic.Pointer[Training]
// catalogAdmin puts a mirrored model into service. Nil without a store,
// where a model added at runtime would not survive a restart.
catalogAdmin web.CatalogAdmin
@@ -208,6 +211,9 @@ type Router struct {
// settingsCheck and settingsLocal mount the portal's settings page.
settingsCheck func([]byte) error
settingsLocal []string
// slurmSettings is the cluster's Slurm settings, for the portal; nil
// offers nothing of Slurm.
slurmSettings func() config.SlurmSettings
}
// SetClusterView gives the router the data the /admin page renders.
@@ -238,6 +244,10 @@ func (r *Router) SetSettingsCheck(check func([]byte) error, local []string) {
r.settingsCheck, r.settingsLocal = check, local
}
// SetSlurm has the portal offer the Slurm cluster's login and SSH keys as
// settings say.
func (r *Router) SetSlurm(s func() config.SlurmSettings) { r.slurmSettings = s }
// SetStopped refuses, or again serves, every request for a model.
func (r *Router) SetStopped(stopped bool) { r.stopped.Store(stopped) }
+331
View File
@@ -0,0 +1,331 @@
package router
import (
"context"
"encoding/json"
"errors"
"io"
"net/http"
"slices"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
"gitea.mynodes.xyz/ilker/llmgw/internal/web"
)
// The "customise a model" page: a person in a project uploads questions
// and answers, picks a model, and presses a button. The node, the weights
// to train on and how much of a card it takes are worked out; what it
// makes is published as a model of its own, seen by the project unless it
// is shared wider. Members of the project see its jobs and manage them;
// one trains at a time per project.
// Training is what the page needs of the control plane.
type Training interface {
TrainableModels() []string
StartTraining(ctx context.Context, in adminapi.TrainRequest, actor string) (store.TrainJob, error)
Trainings(ctx context.Context, projects []string, createdBy string) ([]store.TrainJob, error)
Training(ctx context.Context, id string) (store.TrainJob, error)
Published(j store.TrainJob) bool
ShareTraining(ctx context.Context, id, owner, actor string) error
DeleteTraining(ctx context.Context, id, actor string) error
}
// SetTraining mounts the page's actions on t.
func (r *Router) SetTraining(t Training) { r.training.Store(&t) }
func (r *Router) trainer() Training {
if t := r.training.Load(); t != nil {
return *t
}
return nil
}
// trainMaxFile bounds an uploaded file of pairs.
const trainMaxFile = 32 << 20
// The widths an adapter can be shared at, narrowest first.
var trainShares = []string{"me", "project", "team", "org", "all"}
func (r *Router) registerTrainPage(mux *http.ServeMux, pages *web.AccountPages) {
pages.SetTrain(func() bool {
t := r.trainer()
return t != nil && r.store != nil && len(t.TrainableModels()) > 0
})
mux.HandleFunc("GET /train", pages.Guard(func(w http.ResponseWriter, req *http.Request, user store.User) {
t := r.trainer()
if t == nil || r.store == nil {
http.NotFound(w, req)
return
}
projects, _ := r.store.ProjectsFor(req.Context(), user.ID)
var ps []web.ChatContext
for _, p := range projects {
ps = append(ps, web.ChatContext{Value: p.ID, Label: p.Path})
}
web.Train(user, t.TrainableModels(), ps, version.Version)(w, req)
}))
mux.HandleFunc("POST /train/preview", pages.Guard(r.trainPreview))
mux.HandleFunc("POST /train/start", pages.Guard(r.trainStart))
mux.HandleFunc("GET /train/jobs", pages.Guard(r.trainJobs))
mux.HandleFunc("POST /train/jobs/{id}/share", pages.Guard(r.trainShare))
mux.HandleFunc("POST /train/jobs/{id}/delete", pages.Guard(r.trainDelete))
}
func trainFail(w http.ResponseWriter, code int, msg string) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(code)
_ = json.NewEncoder(w).Encode(map[string]string{"error": msg})
}
// trainFile reads the pairs of the uploaded file.
func trainFile(w http.ResponseWriter, req *http.Request) (train.Read, bool) {
req.Body = http.MaxBytesReader(w, req.Body, trainMaxFile+1<<20)
f, h, err := req.FormFile("file")
if err != nil {
trainFail(w, http.StatusBadRequest, "choose a file of questions and answers")
return train.Read{}, false
}
defer f.Close()
b, err := io.ReadAll(io.LimitReader(f, trainMaxFile+1))
if err != nil || len(b) > trainMaxFile {
trainFail(w, http.StatusBadRequest, "the file is over 32 MB")
return train.Read{}, false
}
rd, err := train.ReadPairs(h.Filename, b)
if err != nil {
trainFail(w, http.StatusBadRequest, err.Error())
return train.Read{}, false
}
return rd, true
}
// trainPreview says what a file holds before anything is trained on it.
func (r *Router) trainPreview(w http.ResponseWriter, req *http.Request, _ store.User) {
rd, ok := trainFile(w, req)
if !ok {
return
}
sample := rd.Pairs[:min(len(rd.Pairs), 5)]
writeJSONBody(w, map[string]any{"pairs": len(rd.Pairs), "skipped": rd.Skipped, "columns": rd.Columns, "sample": sample})
}
// trainPlace is where a project is in the tree.
func (r *Router) trainPlace(ctx context.Context, user store.User, project string) (store.Effective, error) {
projects, err := r.store.ProjectsFor(ctx, user.ID)
if err != nil {
return store.Effective{}, err
}
if !slices.ContainsFunc(projects, func(p store.ProjectPath) bool { return p.ID == project }) {
return store.Effective{}, errors.New("you are not a member of that project")
}
return r.store.ResolveFor(ctx, "", "", project)
}
// trainOwner is the adapter's owner for a share width, in a place.
func trainOwner(share string, user store.User, e store.Effective) (string, bool) {
switch share {
case "me":
return "user:" + user.ID, true
case "project":
return "project:" + e.ProjectID, true
case "team":
return "team:" + e.TeamID, true
case "org":
return "org:" + e.OrgID, true
case "all":
return "", true
}
return "", false
}
// trainShareOf is the width an owner is, seen from the job's place.
func trainShareOf(owner string, e store.Effective) string {
switch {
case owner == "":
return "all"
case strings.HasPrefix(owner, "user:"):
return "me"
case owner == "project:"+e.ProjectID:
return "project"
case owner == "team:"+e.TeamID:
return "team"
case owner == "org:"+e.OrgID:
return "org"
}
return "other"
}
func (r *Router) trainStart(w http.ResponseWriter, req *http.Request, user store.User) {
t := r.trainer()
if t == nil {
trainFail(w, http.StatusNotFound, "training is not offered here")
return
}
rd, ok := trainFile(w, req)
if !ok {
return
}
e, err := r.trainPlace(req.Context(), user, req.FormValue("project"))
if err != nil {
trainFail(w, http.StatusForbidden, err.Error())
return
}
share := req.FormValue("share")
owner, ok := trainOwner(share, user, e)
if !ok {
trainFail(w, http.StatusBadRequest, "who may use it: me, the project, the team, the organization or everyone")
return
}
if slices.Index(trainShares, share) > slices.Index(trainShares, "project") && req.FormValue("confirm") != "yes" {
trainFail(w, http.StatusBadRequest, "confirm that the file holds nothing the wider group must not see")
return
}
in := adminapi.TrainRequest{Name: strings.TrimSpace(req.FormValue("name")), Model: req.FormValue("model"),
Owner: owner, Public: owner == "", Data: train.NDJSON(rd.Pairs), Project: e.ProjectID, Publish: true}
if !slices.Contains(t.TrainableModels(), in.Model) {
trainFail(w, http.StatusBadRequest, "choose one of the models offered")
return
}
j, err := t.StartTraining(req.Context(), in, user.Email)
if err != nil {
msg, theirs := adminapi.TrainRefusal(err)
code := http.StatusBadRequest
if !theirs {
code = http.StatusInternalServerError
r.log.Warn("training could not be started from the portal", "user", user.Email, "err", err)
}
trainFail(w, code, msg)
return
}
r.log.Info("training started from the portal", "job", j.ID, "adapter", j.Name, "model", j.Model,
"project", j.Project, "user", user.Email, "pairs", j.Pairs)
writeJSONBody(w, map[string]string{"id": j.ID})
}
// trainRow is a job as the page lists it.
type trainRow struct {
ID string `json:"id"`
Name string `json:"name"`
Model string `json:"model"`
Project string `json:"project"`
State string `json:"state"`
Step int `json:"step"`
Steps int `json:"steps"`
Message string `json:"message,omitempty"`
Pairs int `json:"pairs"`
Share string `json:"share"`
Published bool `json:"published"`
CreatedBy string `json:"created_by"`
CreatedAt time.Time `json:"created_at"`
Minutes int `json:"minutes"`
}
func (r *Router) trainJobs(w http.ResponseWriter, req *http.Request, user store.User) {
t := r.trainer()
if t == nil || r.store == nil {
writeJSONBody(w, []trainRow{})
return
}
projects, err := r.store.ProjectsFor(req.Context(), user.ID)
if err != nil {
trainFail(w, http.StatusInternalServerError, err.Error())
return
}
ids, names := []string{}, map[string]string{}
for _, p := range projects {
ids = append(ids, p.ID)
names[p.ID] = p.Path
}
jobs, err := t.Trainings(req.Context(), ids, user.Email)
if err != nil {
trainFail(w, http.StatusInternalServerError, err.Error())
return
}
out := []trainRow{}
for _, j := range jobs {
e, _ := r.store.ResolveFor(req.Context(), "", "", j.Project)
row := trainRow{ID: j.ID, Name: j.Name, Model: j.Model, Project: names[j.Project], State: j.State, Step: j.Step,
Steps: j.Steps, Message: j.Message, Pairs: j.Pairs, Share: trainShareOf(j.Owner, e),
Published: j.State == store.TrainDone && t.Published(j), CreatedBy: j.CreatedBy, CreatedAt: j.CreatedAt}
if !j.StartedAt.IsZero() {
end := j.FinishedAt
if end.IsZero() {
end = time.Now()
}
row.Minutes = int(end.Sub(j.StartedAt).Minutes())
}
out = append(out, row)
}
writeJSONBody(w, out)
}
// trainMine is a job a person may manage: their project's, or their own.
func (r *Router) trainMine(w http.ResponseWriter, req *http.Request, user store.User) (store.TrainJob, Training, bool) {
t := r.trainer()
if t == nil || r.store == nil {
trainFail(w, http.StatusNotFound, "training is not offered here")
return store.TrainJob{}, nil, false
}
j, err := t.Training(req.Context(), req.PathValue("id"))
if err == nil {
if j.Project == "" {
if j.CreatedBy == user.Email {
return j, t, true
}
} else if _, err := r.trainPlace(req.Context(), user, j.Project); err == nil {
return j, t, true
}
}
trainFail(w, http.StatusNotFound, "no such training")
return store.TrainJob{}, nil, false
}
func (r *Router) trainShare(w http.ResponseWriter, req *http.Request, user store.User) {
j, t, ok := r.trainMine(w, req, user)
if !ok {
return
}
e, _ := r.store.ResolveFor(req.Context(), "", "", j.Project)
share := req.FormValue("share")
owner, ok := trainOwner(share, user, e)
if !ok || (j.Project == "" && share != "me" && share != "all") {
trainFail(w, http.StatusBadRequest, "who may use it: me, the project, the team, the organization or everyone")
return
}
if slices.Index(trainShares, share) > slices.Index(trainShares, "project") && req.FormValue("confirm") != "yes" {
trainFail(w, http.StatusBadRequest, "confirm that the data holds nothing the wider group must not see")
return
}
if err := t.ShareTraining(req.Context(), j.ID, owner, user.Email); err != nil {
msg, _ := adminapi.TrainRefusal(err)
trainFail(w, http.StatusBadRequest, msg)
return
}
r.log.Info("a trained model's sharing changed from the portal", "job", j.ID, "adapter", j.Name, "share", share,
"user", user.Email)
writeJSONBody(w, map[string]string{"share": share})
}
func (r *Router) trainDelete(w http.ResponseWriter, req *http.Request, user store.User) {
j, t, ok := r.trainMine(w, req, user)
if !ok {
return
}
if err := t.DeleteTraining(req.Context(), j.ID, user.Email); err != nil {
msg, _ := adminapi.TrainRefusal(err)
trainFail(w, http.StatusInternalServerError, msg)
return
}
r.log.Info("a trained model was deleted from the portal", "job", j.ID, "adapter", j.Name, "user", user.Email)
writeJSONBody(w, map[string]bool{"deleted": true})
}
func writeJSONBody(w http.ResponseWriter, v any) {
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(v)
}
+165
View File
@@ -0,0 +1,165 @@
package router
import (
"bytes"
"context"
"encoding/json"
"mime/multipart"
"net/http"
"net/http/httptest"
"path/filepath"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
type fakeTraining struct {
started []adminapi.TrainRequest
jobs map[string]store.TrainJob
shared map[string]string
deleted []string
}
func (f *fakeTraining) TrainableModels() []string { return []string{"qwen"} }
func (f *fakeTraining) StartTraining(_ context.Context, in adminapi.TrainRequest, actor string) (store.TrainJob, error) {
f.started = append(f.started, in)
j := store.TrainJob{ID: "train-" + in.Name, Name: in.Name, Model: in.Model, Owner: in.Owner, Project: in.Project,
CreatedBy: actor, State: store.TrainQueued, Pairs: strings.Count(in.Data, "\n"), CreatedAt: time.Now()}
f.jobs[j.ID] = j
return j, nil
}
func (f *fakeTraining) Trainings(_ context.Context, projects []string, by string) ([]store.TrainJob, error) {
var out []store.TrainJob
for _, j := range f.jobs {
for _, p := range projects {
if j.Project == p {
out = append(out, j)
}
}
}
return out, nil
}
func (f *fakeTraining) Training(_ context.Context, id string) (store.TrainJob, error) {
j, ok := f.jobs[id]
if !ok {
return j, store.ErrNotFound
}
return j, nil
}
func (f *fakeTraining) Published(store.TrainJob) bool { return false }
func (f *fakeTraining) ShareTraining(_ context.Context, id, owner, _ string) error {
f.shared[id] = owner
return nil
}
func (f *fakeTraining) DeleteTraining(_ context.Context, id, _ string) error {
f.deleted = append(f.deleted, id)
return nil
}
func trainForm(t *testing.T, fields map[string]string, file string) *http.Request {
t.Helper()
var b bytes.Buffer
mw := multipart.NewWriter(&b)
for k, v := range fields {
_ = mw.WriteField(k, v)
}
if file != "" {
w, _ := mw.CreateFormFile("file", "ornekler.csv")
_, _ = w.Write([]byte(file))
}
_ = mw.Close()
req := httptest.NewRequest("POST", "/train/start", &b)
req.Header.Set("Content-Type", mw.FormDataContentType())
return req
}
func TestThePortalTrainsForAProjectItsMembersManage(t *testing.T) {
cfg := baseConfig()
cfg.Auth.Enabled = true
cfg.Auth.Store = filepath.Join(t.TempDir(), "store.db")
rt := newTestRouter(t, cfg)
ft := &fakeTraining{jobs: map[string]store.TrainJob{}, shared: map[string]string{}}
rt.SetTraining(ft)
ctx := t.Context()
org, _ := rt.store.CreateOrg(ctx, "acme")
team, _ := rt.store.CreateTeam(ctx, org, "support")
project, _ := rt.store.CreateProject(ctx, team, "bot")
ayse := store.User{ID: "usr-a", Email: "ayse@acme.example", State: store.UserActive, CreatedAt: time.Now()}
bob := store.User{ID: "usr-b", Email: "bob@else.example", State: store.UserActive, CreatedAt: time.Now()}
for _, u := range []store.User{ayse, bob} {
if err := rt.store.CreateUser(ctx, u); err != nil {
t.Fatal(err)
}
}
_ = rt.store.AddMembership(ctx, ayse.ID, store.ScopeProject, project, "member", "test", time.Now())
csv := "Soru,Cevap\nİzin kaç gün?,Yirmi gün.\nMasraf nasıl?,Formla.\n"
start := func(u store.User, fields map[string]string) *httptest.ResponseRecorder {
rec := httptest.NewRecorder()
rt.trainStart(rec, trainForm(t, fields, csv), u)
return rec
}
good := map[string]string{"name": "destek", "model": "qwen", "project": project, "share": "project"}
if rec := start(bob, good); rec.Code != http.StatusForbidden {
t.Errorf("someone outside the project started one: %d %s", rec.Code, rec.Body.String())
}
wide := map[string]string{"name": "destek", "model": "qwen", "project": project, "share": "org"}
if rec := start(ayse, wide); rec.Code != http.StatusBadRequest || !strings.Contains(rec.Body.String(), "confirm") {
t.Errorf("shared with the organization unconfirmed: %d %s", rec.Code, rec.Body.String())
}
other := map[string]string{"name": "destek", "model": "llama-70b", "project": project, "share": "project"}
if rec := start(ayse, other); rec.Code != http.StatusBadRequest {
t.Errorf("a model not offered: %d", rec.Code)
}
if rec := start(ayse, good); rec.Code != http.StatusOK {
t.Fatalf("%d %s", rec.Code, rec.Body.String())
}
in := ft.started[0]
if in.Owner != "project:"+project || in.Project != project || !in.Publish || in.Public ||
in.Data != `{"q":"İzin kaç gün?","a":"Yirmi gün."}`+"\n"+`{"q":"Masraf nasıl?","a":"Formla."}`+"\n" {
t.Errorf("asked for %+v", in)
}
list := func(u store.User) []trainRow {
rec := httptest.NewRecorder()
rt.trainJobs(rec, httptest.NewRequest("GET", "/train/jobs", nil), u)
var rows []trainRow
_ = json.Unmarshal(rec.Body.Bytes(), &rows)
return rows
}
if rows := list(ayse); len(rows) != 1 || rows[0].Share != "project" || rows[0].Project != "acme / support / bot" ||
rows[0].CreatedBy != ayse.Email {
t.Errorf("ayse sees %+v", rows)
}
if rows := list(bob); len(rows) != 0 {
t.Errorf("bob sees %+v", rows)
}
act := func(u store.User, what string, fields map[string]string) int {
req := trainForm(t, fields, "")
req.SetPathValue("id", "train-destek")
rec := httptest.NewRecorder()
if what == "share" {
rt.trainShare(rec, req, u)
} else {
rt.trainDelete(rec, req, u)
}
return rec.Code
}
if code := act(bob, "delete", nil); code != http.StatusNotFound || len(ft.deleted) != 0 {
t.Errorf("bob deleted it: %d", code)
}
if code := act(ayse, "share", map[string]string{"share": "team", "confirm": "yes"}); code != http.StatusOK ||
ft.shared["train-destek"] != "team:"+team {
t.Errorf("shared with the team: %d %v", code, ft.shared)
}
if code := act(ayse, "share", map[string]string{"share": "all"}); code != http.StatusBadRequest {
t.Errorf("shared with everyone unconfirmed: %d", code)
}
if code := act(ayse, "delete", nil); code != http.StatusOK || len(ft.deleted) != 1 {
t.Errorf("ayse deleting: %d", code)
}
}
+193
View File
@@ -0,0 +1,193 @@
package slurm
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"slices"
"strconv"
"strings"
"time"
)
// Slurm's accounts from llmgw's tree: an account for each organization,
// team and project, nested as they are, and on each project's the users
// that may use it, its members and its team's and organization's. A job
// is charged to the project its --account names (`sbatch --account
// acme-ml-training`), and what it used to that project's spend.
// Account is one Slurm account llmgw keeps.
type Account struct {
Name string
Parent string // "root" at the top
// Project is the llmgw project it is for; empty for an organization's
// or a team's.
Project string
// Users are the logins that may run jobs on it.
Users []string
}
// TenancyProject is a project with where it is.
type TenancyProject struct {
ID, Name, TeamID, TeamName, OrgID, OrgName string
}
// TenancyMember is a membership: a person in an organization, team or
// project ("org", "team", "project").
type TenancyMember struct {
UserID, Scope, ScopeID string
}
// Accounts are the accounts for projects, with logins by user id. Parents
// come before their children, as sacctmgr needs them.
func Accounts(projects []TenancyProject, members []TenancyMember, logins map[string]string) []Account {
// By kind and id: an organization, its team and its project may share
// an id (the first install's are all "default").
names := map[string]string{} // kind/id -> name
taken := map[string]string{} // name -> kind/id
name := func(id string, parts ...string) string {
if n, ok := names[id]; ok {
return n
}
n := AccountName(parts...)
if other, ok := taken[n]; ok && other != id {
// Two places whose names come out the same: told apart by id.
h := sha256.Sum256([]byte(id))
n = AccountName(append(parts, hex.EncodeToString(h[:3]))...)
}
names[id], taken[n] = n, id
return n
}
in := map[string][]string{} // kind/id -> logins
for _, m := range members {
if l, ok := logins[m.UserID]; ok {
in[m.Scope+"/"+m.ScopeID] = append(in[m.Scope+"/"+m.ScopeID], l)
}
}
ps := slices.Clone(projects)
slices.SortFunc(ps, func(a, b TenancyProject) int {
return strings.Compare(a.OrgName+"\x00"+a.TeamName+"\x00"+a.Name, b.OrgName+"\x00"+b.TeamName+"\x00"+b.Name)
})
var out []Account
seen := map[string]bool{}
for _, p := range ps {
org := name("org/"+p.OrgID, p.OrgName)
team := name("team/"+p.TeamID, p.OrgName, p.TeamName)
proj := name("project/"+p.ID, p.OrgName, p.TeamName, p.Name)
if !seen[org] {
out = append(out, Account{Name: org, Parent: "root"})
seen[org] = true
}
if !seen[team] {
out = append(out, Account{Name: team, Parent: org})
seen[team] = true
}
var users []string
for _, id := range []string{"project/" + p.ID, "team/" + p.TeamID, "org/" + p.OrgID} {
users = append(users, in[id]...)
}
slices.Sort(users)
out = append(out, Account{Name: proj, Parent: team, Project: p.ID, Users: slices.Compact(users)})
}
return out
}
// AccountName is a Slurm account name from names: lower case, ASCII,
// joined with "-", at most 64 characters.
func AccountName(parts ...string) string {
var segs []string
for _, p := range parts {
var b strings.Builder
for _, r := range strings.ToLower(p) {
switch {
case r >= 'a' && r <= 'z', r >= '0' && r <= '9', r == '_':
b.WriteRune(r)
case strings.ContainsRune("çğıöşüâîû", r):
b.WriteString(map[rune]string{'ç': "c", 'ğ': "g", 'ı': "i", 'ö': "o", 'ş': "s", 'ü': "u",
'â': "a", 'î': "i", 'û': "u"}[r])
default:
b.WriteByte('_')
}
}
if s := strings.Trim(b.String(), "_"); s != "" {
segs = append(segs, s)
}
}
n := strings.Join(segs, "-")
if n == "" {
n = "x"
}
if len(n) > 64 {
h := sha256.Sum256([]byte(n))
n = n[:57] + "-" + hex.EncodeToString(h[:3])
}
return n
}
// Association is one line of `sacctmgr show assoc format=account,user,parentname`.
type Association struct {
Account, User, Parent string
}
// AssocFormat is what ParseAssociations reads.
var AssocFormat = []string{"show", "assoc", "format=account,user,parentname", "-n", "-P"}
// ParseAssociations reads sacctmgr's parsable lines.
func ParseAssociations(out string) []Association {
var as []Association
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
f := strings.Split(line, "|")
if len(f) < 3 || f[0] == "" {
continue
}
as = append(as, Association{Account: f[0], User: f[1], Parent: f[2]})
}
return as
}
// Job is a finished job, as sacct has it.
type Job struct {
ID, User, Account, Partition, State string
Start, End time.Time
Elapsed time.Duration
CPUs, GPUs int
}
// SacctArgs are sacct's for the jobs that ended in the last day.
var SacctArgs = []string{"-a", "-X", "-n", "-P", "--noconvert", "-S", "now-1days", "-E", "now",
"-s", "CD,F,TO,CA,NF,OOM,PR,DL,BF",
"-o", "JobIDRaw,User,Account,Partition,Start,End,ElapsedRaw,AllocTRES,State"}
// ParseJobs reads sacct's lines; a job that has not ended is left out.
func ParseJobs(out string) ([]Job, error) {
var jobs []Job
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
if line == "" {
continue
}
f := strings.Split(line, "|")
if len(f) < 9 {
return nil, fmt.Errorf("sacct said %q, not 9 fields", line)
}
end, err := time.ParseInLocation("2006-01-02T15:04:05", f[5], time.Local)
if err != nil {
continue // Unknown: still running, or never started
}
start, _ := time.ParseInLocation("2006-01-02T15:04:05", f[4], time.Local)
secs, _ := strconv.ParseInt(f[6], 10, 64)
j := Job{ID: f[0], User: f[1], Account: f[2], Partition: f[3], Start: start, End: end,
Elapsed: time.Duration(secs) * time.Second, State: strings.Fields(f[8] + " ")[0]}
for _, t := range strings.Split(f[7], ",") {
k, v, _ := strings.Cut(t, "=")
n, _ := strconv.Atoi(v)
switch k {
case "cpu":
j.CPUs = n
case "gres/gpu":
j.GPUs = n
}
}
jobs = append(jobs, j)
}
return jobs, nil
}
+89
View File
@@ -0,0 +1,89 @@
package slurm
import (
"slices"
"testing"
"time"
)
func TestAccountsFollowTheTree(t *testing.T) {
projects := []TenancyProject{
{ID: "p1", Name: "deneme", TeamID: "t1", TeamName: "arge", OrgID: "o1", OrgName: "Test"},
{ID: "p2", Name: "Görüntü İşleme", TeamID: "t1", TeamName: "arge", OrgID: "o1", OrgName: "Test"},
{ID: "p3", Name: "x", TeamID: "t2", TeamName: "ops", OrgID: "o2", OrgName: "Other"},
}
members := []TenancyMember{
{UserID: "u-ayse", Scope: "project", ScopeID: "p1"},
{UserID: "u-ada", Scope: "team", ScopeID: "t1"},
{UserID: "u-bob", Scope: "org", ScopeID: "o2"},
{UserID: "u-nologin", Scope: "project", ScopeID: "p1"},
}
logins := map[string]string{"u-ayse": "ayse", "u-ada": "ada", "u-bob": "bob"}
got := Accounts(projects, members, logins)
var names []string
by := map[string]Account{}
for _, a := range got {
names = append(names, a.Name)
by[a.Name] = a
}
// Parents first, each before its children.
want := []string{"other", "other-ops", "other-ops-x", "test", "test-arge", "test-arge-goruntu_isleme", "test-arge-deneme"}
if !slices.Equal(names, want) {
t.Fatalf("names %v", names)
}
if a := by["test-arge-deneme"]; a.Parent != "test-arge" || a.Project != "p1" || !slices.Equal(a.Users, []string{"ada", "ayse"}) {
t.Errorf("deneme %+v", a)
}
if a := by["other-ops-x"]; !slices.Equal(a.Users, []string{"bob"}) {
t.Errorf("x %+v", a)
}
if a := by["test"]; a.Parent != "root" || a.Project != "" || len(a.Users) != 0 {
t.Errorf("org %+v", a)
}
if n := AccountName("Test", "arge", "Görüntü İşleme"); n != "test-arge-goruntu_isleme" {
t.Errorf("name %q", n)
}
// The first install's tree: one id for all three.
def := Accounts([]TenancyProject{{ID: "default", Name: "default", TeamID: "default", TeamName: "default",
OrgID: "default", OrgName: "default"}}, []TenancyMember{{UserID: "u", Scope: "project", ScopeID: "default"}},
map[string]string{"u": "admin2"})
if len(def) != 3 || def[2].Name != "default-default-default" || def[2].Parent != "default-default" ||
def[1].Parent != "default" || !slices.Equal(def[2].Users, []string{"admin2"}) {
t.Errorf("default tree %+v", def)
}
// Two that come out the same are told apart.
same := Accounts([]TenancyProject{
{ID: "a", Name: "ml", TeamID: "t", TeamName: "x", OrgID: "o", OrgName: "A"},
{ID: "b", Name: "ML", TeamID: "t", TeamName: "x", OrgID: "o", OrgName: "A"},
}, nil, nil)
if same[2].Name == same[3].Name {
t.Errorf("same name twice: %+v", same)
}
}
func TestParseJobs(t *testing.T) {
out := "41|ayse|test-arge-deneme|gpu|2026-11-15T10:00:00|2026-11-15T10:30:00|1800|billing=4,cpu=4,gres/gpu=1,mem=16G,node=1|COMPLETED\n" +
"42|ada|test-arge-deneme|gpu|2026-11-15T10:00:00|Unknown|0|cpu=2|RUNNING\n" +
"43|root|root|gpu|2026-11-15T09:00:00|2026-11-15T09:00:05|5|cpu=1,mem=1G|CANCELLED by 0\n"
jobs, err := ParseJobs(out)
if err != nil {
t.Fatal(err)
}
if len(jobs) != 2 {
t.Fatalf("jobs %+v", jobs)
}
j := jobs[0]
if j.ID != "41" || j.User != "ayse" || j.GPUs != 1 || j.CPUs != 4 || j.Elapsed != 30*time.Minute || j.State != "COMPLETED" {
t.Errorf("job %+v", j)
}
if jobs[1].State != "CANCELLED" {
t.Errorf("state %q", jobs[1].State)
}
if _, err := ParseJobs("short|line\n"); err == nil {
t.Error("a short line taken")
}
as := ParseAssociations("root||\ntest|ayse|test-arge\n")
if len(as) != 2 || as[1] != (Association{"test", "ayse", "test-arge"}) {
t.Errorf("assoc %+v", as)
}
}
+139
View File
@@ -0,0 +1,139 @@
package slurm
import (
"fmt"
"strings"
)
// Jobs in containers: pyxis, Slurm's plugin that gives srun and sbatch
// --container-image, and enroot under it, which turns an image into a
// directory a job's user runs in without a daemon or root. Both are
// NVIDIA's and in no distribution's archive, so each is fetched from its
// release at a pinned version and checked against the digest here.
const (
EnrootVersion = "4.2.1"
PyxisVersion = "0.24.0"
// PyxisSource is pyxis's source, built on each machine against its own
// Slurm: a plugin built for another Slurm is refused by it.
PyxisSource = "https://github.com/NVIDIA/pyxis/archive/refs/tags/v" + PyxisVersion + ".tar.gz"
PyxisSHA256 = "9c4cdb79a67301d8ea05951aa4d2c205f40cf6145e338214b0191add8b36d6c4"
// PyxisPlugin is where it is installed (pyxis's own make install).
PyxisPlugin = "/usr/local/lib/slurm/spank_pyxis.so"
// PlugstackConf has Slurm load pyxis. Written on the controller, from
// which configless nodes fetch it, only once every machine has the
// plugin: "required" fails every job on a machine without it, and
// "optional" would run a job asking for a container on the host.
PlugstackConf = "# Written by llmgw (BACKLOG P): jobs in containers, srun --container-image.\n" +
"required " + PyxisPlugin + "\n"
// EnrootConfPath is enroot's settings.
EnrootConfPath = "/etc/enroot/enroot.conf"
// EnrootMounts has every container see the shared path.
EnrootMounts = "/etc/enroot/mounts.d/50-llmgw-shared.fstab"
// EnrootTmpfiles makes enroot's runtime directory at each boot.
EnrootTmpfiles = "/etc/tmpfiles.d/llmgw-enroot.conf"
// EnrootRun is where containers are unpacked while they run; EnrootLib
// holds each user's image cache and named containers.
EnrootRun = "/run/enroot"
EnrootLib = "/var/lib/enroot"
)
// EnrootPackage is one of enroot's release files.
type EnrootPackage struct {
Name, SHA256 string
}
// URL is where the file is.
func (p EnrootPackage) URL() string {
return "https://github.com/NVIDIA/enroot/releases/download/v" + EnrootVersion + "/" +
strings.ReplaceAll(p.Name, "+", "%2B")
}
// EnrootPackages are the files for a family and architecture (Go's
// names): enroot, and enroot+caps, which lets its helpers unpack images
// for a user without root.
func EnrootPackages(family, arch string) ([]EnrootPackage, error) {
ps, ok := enrootPackages[family+"/"+arch]
if !ok {
return nil, fmt.Errorf("enroot %s has no package for %s on %s", EnrootVersion, family, arch)
}
return ps, nil
}
var enrootPackages = map[string][]EnrootPackage{
"deb/amd64": {
{"enroot_4.2.1-1_amd64.deb", "5955c51e88df0c5ed538d87abb5264609d4bad55fa8570745ed5d4ec66c8ab91"},
{"enroot+caps_4.2.1-1_amd64.deb", "ca22652dcbdd3ebb6e2c68df781ef4301647c79dd7a41ed62899c2fd0f692750"},
},
"deb/arm64": {
{"enroot_4.2.1-1_arm64.deb", "267018b335815a6b32a562915de1f3fad579f7c9d6748e98c62e7369b40be421"},
{"enroot+caps_4.2.1-1_arm64.deb", "12b811d0c95e879e42e4ac762fa85869a5f3cc9dbb06ab483c01e12e911284a1"},
},
"rpm/amd64": {
{"enroot-4.2.1-1.el8.x86_64.rpm", "6749195e37e3a98071ec349884900b12de7677332599da0fd5d8afb5bed9233c"},
{"enroot+caps-4.2.1-1.el8.x86_64.rpm", "56321a03a1f6c4e4273bf664917eb9a174714499d5506c498db4b64275a2b9c0"},
},
"rpm/arm64": {
{"enroot-4.2.1-1.el8.aarch64.rpm", "6b6bc2526f0f0daa7d3cebe6dd33c0468b209e81eebd92def5777520042ec6fe"},
{"enroot+caps-4.2.1-1.el8.aarch64.rpm", "38cf9ae9602a9a8c3e94637bc1b34e736e8176bda656ba71d371f5e1d1e1eed9"},
},
}
// PyxisBuildPackages build pyxis: a compiler and Slurm's headers.
var PyxisBuildPackages = map[string][]string{
"deb": {"gcc", "make", "libslurm-dev"},
"rpm": {"gcc", "make", "slurm-devel"},
}
// EnrootConf is enroot's settings: each user's containers, cache and
// data apart, in directories that need no login session (a job has none,
// and llmgw's users have no pam_systemd on a node).
const EnrootConf = "# Written by llmgw (BACKLOG P): enroot, for Slurm's jobs in containers.\n" +
"ENROOT_RUNTIME_PATH " + EnrootRun + "/user-$(id -u)\n" +
"ENROOT_CACHE_PATH " + EnrootLib + "/cache/user-$(id -u)\n" +
"ENROOT_DATA_PATH " + EnrootLib + "/data/user-$(id -u)\n" +
"ENROOT_TEMP_PATH ${TMPDIR:-/tmp}\n" +
// A container's root is the user's, as on the host: no fake root.
"ENROOT_REMAP_ROOT no\n" +
"ENROOT_ROOTFS_WRITABLE yes\n"
// EnrootTmpfilesConf makes EnrootRun, open to every user as /tmp is.
const EnrootTmpfilesConf = "# Written by llmgw (BACKLOG P): enroot's runtime directory.\n" +
"d " + EnrootRun + " 1777 root root -\n"
// EnrootSharedMount has a container see path where the host does.
func EnrootSharedMount(path string) string {
return "# Written by llmgw (BACKLOG P): the Slurm cluster's shared files, in every container.\n" +
path + " " + path + " none x-create=dir,rbind,rw,nosuid\n"
}
// The NVIDIA container library, for enroot's hook that gives a container
// the job's GPUs: from NVIDIA's own repository, its key checked.
const (
NvidiaContainerKeyURL = "https://nvidia.github.io/libnvidia-container/gpgkey"
NvidiaContainerKeySHA256 = "c880576d6cf75a48e5027a871bac70fd0421ab07d2b55f30877b21f1c87959c9"
NvidiaContainerKeyPath = "/etc/apt/keyrings/llmgw-nvidia-container.asc"
NvidiaContainerAptList = "/etc/apt/sources.list.d/llmgw-nvidia-container.list"
NvidiaContainerYumRepo = "/etc/yum.repos.d/llmgw-nvidia-container.repo"
)
// NvidiaContainerApt is the apt source.
const NvidiaContainerApt = "# Written by llmgw (BACKLOG P): nvidia-container-cli, for GPUs in Slurm's containers.\n" +
"deb [signed-by=" + NvidiaContainerKeyPath + "] https://nvidia.github.io/libnvidia-container/stable/deb/$(ARCH) /\n"
// NvidiaContainerYum is the yum/dnf repository.
const NvidiaContainerYum = "# Written by llmgw (BACKLOG P): nvidia-container-cli, for GPUs in Slurm's containers.\n" +
"[llmgw-nvidia-container]\nname=NVIDIA container library\n" +
"baseurl=https://nvidia.github.io/libnvidia-container/stable/rpm/$basearch\n" +
"repo_gpgcheck=1\ngpgcheck=0\nenabled=1\ngpgkey=" + NvidiaContainerKeyURL + "\n"
// NvidiaContainerPackages has nvidia-container-cli.
var NvidiaContainerPackages = map[string][]string{"deb": {"libnvidia-container-tools"}, "rpm": {"libnvidia-container-tools"}}
// ContainersStatus is what a machine says it has.
func ContainersStatus() string {
return "pyxis " + PyxisVersion + ", enroot " + EnrootVersion
}
+81
View File
@@ -0,0 +1,81 @@
package slurm
import (
"crypto/sha256"
"encoding/hex"
"fmt"
)
// Accounting: slurmdbd on the controller, keeping the jobs' records in a
// MariaDB beside it that answers this machine alone. slurmdbd takes MySQL or
// MariaDB only, so it is not the cluster's PostgreSQL. Since 23.11 Slurm
// cannot run without it well: its main scheduler passes every job by as
// InvalidAccount, and backfill starts it a pass later.
const (
// DbdPort is slurmdbd's, which slurmctld and sacct reach.
DbdPort = 6819
// DbdConfPath is slurmdbd's file, readable by the slurm user alone: it
// holds the database's password.
DbdConfPath = ConfDir + "/slurmdbd.conf"
// AcctDB and AcctUser are the database and its user.
AcctDB = "slurm_acct_db"
AcctUser = "slurm"
)
// DbdPassword is the database user's password, made from the munge key:
// every controller the cluster has gets the same one, and nothing new is
// kept anywhere.
func DbdPassword(mungeKey []byte) string {
h := sha256.New()
h.Write([]byte("llmgw slurmdbd password\x00"))
h.Write(mungeKey)
return hex.EncodeToString(h.Sum(nil))[:40]
}
// DbdConf is slurmdbd.conf for the controller host, named as `hostname -s`
// says, which slurmdbd checks it runs on.
func DbdConf(host, password string) string {
return header + fmt.Sprintf(`AuthType=auth/munge
DbdHost=%s
DbdPort=%d
SlurmUser=slurm
LogFile=%s/slurmdbd.log
PidFile=/run/slurmdbd.pid
StorageType=accounting_storage/mysql
StorageHost=localhost
StorageUser=%s
StoragePass=%s
StorageLoc=%s
# Kept a year; older records are purged, a month at a time.
PurgeEventAfter=12months
PurgeJobAfter=12months
PurgeResvAfter=12months
PurgeStepAfter=12months
PurgeSuspendAfter=12months
PurgeTXNAfter=12months
PurgeUsageAfter=24months
`, host, DbdPort, LogDir, AcctUser, password, AcctDB)
}
// MariaDBConf is the database server's settings: this machine alone, and
// what Slurm's documentation asks of InnoDB.
const MariaDBConf = `# Written by llmgw (BACKLOG P): the database of Slurm's accounting
# (slurmdbd), on this machine alone.
[mysqld]
bind-address=127.0.0.1
skip-networking=0
innodb_buffer_pool_size=256M
innodb_lock_wait_timeout=900
`
// AcctSQL makes the database and its user, and gives the user the password
// again: run on every start of the controller's part, it changes nothing
// that is already so.
func AcctSQL(password string) string {
return fmt.Sprintf(`CREATE DATABASE IF NOT EXISTS %[1]s;
CREATE USER IF NOT EXISTS '%[2]s'@'localhost' IDENTIFIED BY '%[3]s';
ALTER USER '%[2]s'@'localhost' IDENTIFIED BY '%[3]s';
GRANT ALL ON %[1]s.* TO '%[2]s'@'localhost';
FLUSH PRIVILEGES;
`, AcctDB, AcctUser, password)
}
+229
View File
@@ -0,0 +1,229 @@
package slurm
import (
"errors"
"fmt"
"path"
"regexp"
"strconv"
"strings"
)
// Logging in to a node: llmgw's users reach the controller, the machine
// they submit from, whenever they like, and a node only while a job of
// theirs runs there, their session put in that job (pam_slurm_adopt).
// Without it a user could ssh to a node and use its cards outside Slurm,
// on a node llmgw serves models from too.
const (
// PAMSSHD is sshd's PAM stack, where the lines go.
PAMSSHD = "/etc/pam.d/sshd"
// AdoptModule is the module's file name.
AdoptModule = "pam_slurm_adopt.so"
pamBegin = "# llmgw (BACKLOG P): llmgw's Slurm users reach this node only while a job of theirs runs here."
pamSession = "# llmgw (BACKLOG P): pam_systemd skipped for llmgw's Slurm users, so their session stays in the job pam_slurm_adopt put it in."
pamWas = "# llmgw, in place of: "
pamEnd = "# llmgw end"
)
// AdoptPackages are the packages with AdoptModule, by family.
var AdoptPackages = map[string][]string{"deb": {"libpam-slurm-adopt"}, "rpm": {"slurm-pam_slurm"}}
// ModuleDirs are where PAM's modules are, on Debian's and Red Hat's.
var ModuleDirs = []string{
"/usr/lib/x86_64-linux-gnu/security", "/lib/x86_64-linux-gnu/security",
"/usr/lib/aarch64-linux-gnu/security", "/lib/aarch64-linux-gnu/security",
"/usr/lib64/security", "/lib64/security", "/usr/lib/security", "/lib/security",
}
// PAMAdopt is sshd's PAM stack with llmgw's lines in it (on) or taken out
// and the stack as it was. Only the users listed in the file list, one a
// line, are held to their jobs; anyone else, root and the site's
// administrators, logs in as before.
//
// The account lines go after the last account line, so a user the
// machine's own rules refuse is still refused. pam_systemd, which would
// move the session out of the job into a scope of its own, is skipped for
// the listed users: the file sshd includes it from (read through read) is
// put in sshd's stack in its place, with the one line in front of
// pam_systemd. contained is whether that was done; when it cannot be done
// safely, the users are still held to their jobs, their sessions not to
// the job's cards and cores.
func PAMAdopt(stack, list string, read func(name string) (string, error), on bool) (out string, contained bool, err error) {
lines, err := pamStrip(stack)
if err != nil {
return "", false, err
}
if !on {
return join(lines), false, nil
}
last := -1
for i, l := range lines {
f := strings.Fields(l)
if len(f) > 0 && (f[0] == "account" || (f[0] == "@include" && len(f) > 1 && strings.Contains(f[1], "account"))) {
last = i
}
}
if last < 0 {
return "", false, errors.New(PAMSSHD + " has no account lines to put llmgw's after")
}
block := []string{
pamBegin,
// Listed: on to the next line; not listed, or no list: past it.
"account [success=ignore default=1] pam_listfile.so item=user sense=allow file=" + list + " onerr=fail",
"account required " + AdoptModule,
pamEnd,
}
lines = append(lines[:last+1:last+1], append(block, lines[last+1:]...)...)
lines, contained = pamSkipSystemd(lines, list, read)
return join(lines), contained, nil
}
func join(lines []string) string { return strings.Join(lines, "\n") + "\n" }
// pamStrip takes llmgw's blocks out, putting back the lines they stand in
// place of.
func pamStrip(stack string) ([]string, error) {
var lines []string
in := false
for _, l := range strings.Split(strings.TrimSuffix(stack, "\n"), "\n") {
switch {
case !in && (l == pamBegin || l == pamSession):
in = true
case in && l == pamEnd:
in = false
case in && strings.HasPrefix(l, pamWas):
lines = append(lines, strings.TrimPrefix(l, pamWas))
case !in:
lines = append(lines, l)
}
}
if in {
return nil, errors.New("llmgw's lines in " + PAMSSHD + " have no end; put them right by hand")
}
return lines, nil
}
// pamLine is a PAM line's type, control and module, the control with its
// brackets.
var pamLine = regexp.MustCompile(`^\s*(-?[a-z]+)\s+(\[[^\]]*\]|\S+)\s+(\S+)`)
// pamJump is a jump of N in a bracketed control.
var pamJump = regexp.MustCompile(`=(\d+)`)
func isSession(l string) bool {
m := pamLine.FindStringSubmatch(l)
return m != nil && strings.TrimPrefix(m[1], "-") == "session"
}
// pamSkipSystemd puts the guard in front of pam_systemd, in the session
// file sshd includes it from or in sshd's own lines; false, and lines as
// they were, when it is in neither or cannot be done without changing
// what any other jump in the stack lands on.
func pamSkipSystemd(lines []string, list string, read func(string) (string, error)) ([]string, bool) {
guard := "session [success=1 default=ignore] pam_listfile.so item=user sense=allow file=" + list + " onerr=fail"
hasSystemd := func(l string) bool {
m := pamLine.FindStringSubmatch(l)
return m != nil && strings.TrimPrefix(m[1], "-") == "session" && path.Base(m[3]) == "pam_systemd.so"
}
for i, l := range lines {
f := strings.Fields(l)
var body []string
switch {
case hasSystemd(l):
body = []string{l}
case len(f) == 2 && f[0] == "@include" && strings.Contains(f[1], "session"),
len(f) == 3 && f[0] == "session" && f[1] == "include":
name := f[len(f)-1]
if strings.Contains(name, "/") {
continue
}
text, err := read(name)
if err != nil {
return lines, false
}
for _, il := range strings.Split(text, "\n") {
t := strings.TrimSpace(il)
if t == "" || strings.HasPrefix(t, "#") {
continue
}
if f[0] == "session" && !isSession(t) {
continue // include takes only the lines of its own type
}
body = append(body, t)
}
default:
continue
}
at := -1
for j, b := range body {
if hasSystemd(b) {
at = j
}
if g := strings.Fields(b); len(g) > 0 && (g[0] == "@include" || strings.Contains(b, "include") || strings.Contains(b, "substack")) {
return lines, false // nested: not followed
}
}
if at < 0 {
continue
}
// A jump in sshd's own session lines may cross what is put in.
for j, o := range lines {
if j != i && isSession(o) && pamJump.MatchString(pamControl(o)) {
return lines, false
}
}
body, ok := pamInsert(body, at, guard)
if !ok {
return lines, false
}
block := append([]string{pamSession, pamWas + l}, body...)
block = append(block, pamEnd)
return append(lines[:i:i], append(block, lines[i+1:]...)...), true
}
return lines, false
}
func pamControl(l string) string {
if m := pamLine.FindStringSubmatch(l); m != nil {
return m[2]
}
return ""
}
// pamInsert puts guard in front of body[at], a jump that crosses it made
// one longer so it lands where it did.
func pamInsert(body []string, at int, guard string) ([]string, bool) {
// Each session line's place among the session lines.
idx := make([]int, len(body))
n := 0
for j, b := range body {
idx[j] = n
if isSession(b) {
n++
}
}
out := make([]string, 0, len(body)+1)
for j, b := range body {
if j == at {
out = append(out, guard)
}
ctl := pamControl(b)
if isSession(b) && j < at && strings.HasPrefix(ctl, "[") {
from, target := idx[j], idx[at]
fixed := pamJump.ReplaceAllStringFunc(ctl, func(s string) string {
k, err := strconv.Atoi(s[1:])
if err != nil {
return s
}
if from < target && target <= from+k {
k++
}
return fmt.Sprintf("=%d", k)
})
b = strings.Replace(b, ctl, fixed, 1)
}
out = append(out, b)
}
return out, true
}
+113
View File
@@ -0,0 +1,113 @@
package slurm
import (
"errors"
"strings"
"testing"
)
const ubuntuSSHD = `# PAM configuration for the Secure Shell service
@include common-auth
account required pam_nologin.so
@include common-account
session [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so close
session required pam_loginuid.so
@include common-session
session required pam_limits.so
@include common-password
`
const ubuntuSession = `# here are the per-package modules (the "Primary" block)
session [default=1] pam_permit.so
session requisite pam_deny.so
session required pam_permit.so
session optional pam_umask.so
session required pam_unix.so
session optional pam_systemd.so
# end of pam-auth-update config
`
const redhatSSHD = `#%PAM-1.0
auth substack password-auth
account required pam_sepermit.so
account required pam_nologin.so
account include password-auth
password include password-auth
session required pam_loginuid.so
session include password-auth
session include postlogin
`
const redhatPasswordAuth = `auth required pam_env.so
account required pam_unix.so
-session optional pam_keyinit.so revoke
session required pam_limits.so
-session optional pam_systemd.so
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session required pam_unix.so
`
func files(m map[string]string) func(string) (string, error) {
return func(n string) (string, error) {
if s, ok := m[n]; ok {
return s, nil
}
return "", errors.New("no such file")
}
}
func TestPAMAdopt(t *testing.T) {
read := files(map[string]string{"common-session": ubuntuSession, "password-auth": redhatPasswordAuth, "postlogin": "session optional pam_lastlog.so\n"})
for name, stack := range map[string]string{"ubuntu": ubuntuSSHD, "redhat": redhatSSHD} {
on, contained, err := PAMAdopt(stack, "/var/lib/x-users", read, true)
if err != nil {
t.Fatalf("%s: %v", name, err)
}
if !contained {
t.Errorf("%s: pam_systemd not skipped:\n%s", name, on)
}
lines := strings.Split(on, "\n")
if !strings.Contains(on, "account [success=ignore default=1] pam_listfile.so item=user sense=allow file=/var/lib/x-users onerr=fail") {
t.Fatalf("%s: lines not put in:\n%s", name, on)
}
for j, l := range lines {
if l == pamBegin && !strings.Contains(lines[j-1], "account") {
t.Errorf("%s: put after %q", name, lines[j-1])
}
if strings.Contains(l, "pam_systemd.so") &&
lines[j-1] != "session [success=1 default=ignore] pam_listfile.so item=user sense=allow file=/var/lib/x-users onerr=fail" {
t.Errorf("%s: pam_systemd not guarded, after %q", name, lines[j-1])
}
}
again, _, _ := PAMAdopt(on, "/var/lib/x-users", read, true)
if again != on {
t.Errorf("%s: put in twice:\n%s", name, again)
}
off, _, err := PAMAdopt(on, "", read, false)
if err != nil || off != stack {
t.Errorf("%s: taken out, not as it was (%v):\n%s", name, err, off)
}
}
// The jump over pam_deny lands where it did; the one past crond's line too.
on, _, _ := PAMAdopt(ubuntuSSHD, "/l", read, true)
if !strings.Contains(on, "[default=1]\t\t\tpam_permit.so") {
t.Errorf("a jump that does not cross the guard was changed:\n%s", on)
}
body := []string{"session [default=2] pam_permit.so", "session required pam_unix.so", "session optional pam_systemd.so", "session required pam_x.so"}
got, _ := pamInsert(body, 2, "G")
if got[0] != "session [default=3] pam_permit.so" || got[2] != "G" {
t.Errorf("a jump across the guard not made longer: %q", got)
}
if _, _, err := PAMAdopt("auth required pam_unix.so\n", "/x", read, true); err == nil {
t.Error("a stack with no account lines taken")
}
if _, _, err := PAMAdopt(pamBegin+"\naccount required x\n", "/x", read, false); err == nil {
t.Error("lines with no end taken")
}
// The session file unreadable: held to their jobs all the same.
on, contained, err := PAMAdopt(ubuntuSSHD, "/l", files(nil), true)
if err != nil || contained || !strings.Contains(on, AdoptModule) {
t.Errorf("contained %v, err %v:\n%s", contained, err, on)
}
}
+127
View File
@@ -0,0 +1,127 @@
package slurm
import (
"fmt"
"regexp"
"slices"
"strings"
"golang.org/x/crypto/ssh"
)
// Shared storage: one path every machine of the Slurm cluster sees, from
// the site's NFS export, from an llmgw node that exports it, or mounted by
// the site itself.
const (
// NFSPort is NFSv4's one port, which is all a client needs.
NFSPort = 2049
// ExportsPath is the export llmgw keeps on the machine that serves.
ExportsPath = "/etc/exports.d/llmgw-slurm.exports"
// MountUnitDir is where the mount unit is written.
MountUnitDir = "/etc/systemd/system"
)
// Exports is the exports file for path, read-write to clients alone. Root
// on a client is nobody here: a client's root cannot take another's files.
func Exports(path string, clients []string) string {
cs := slices.Clone(clients)
slices.Sort(cs)
cs = slices.Compact(cs)
var b strings.Builder
b.WriteString("# Written by llmgw (BACKLOG P): the Slurm cluster's shared files, to its machines alone.\n")
if len(cs) == 0 {
return b.String()
}
fmt.Fprintf(&b, "%q", path)
for _, c := range cs {
fmt.Fprintf(&b, " %s(rw,sync,no_subtree_check,root_squash)", c)
}
b.WriteString("\n")
return b.String()
}
// MountUnitName is systemd's name for the mount unit of path, as
// systemd-escape --path makes it.
func MountUnitName(path string) string {
p := strings.Trim(path, "/")
var b strings.Builder
for i := 0; i < len(p); i++ {
c := p[i]
switch {
case c == '/':
b.WriteByte('-')
case c >= 'a' && c <= 'z', c >= 'A' && c <= 'Z', c >= '0' && c <= '9', c == '_', c == '.' && i > 0:
b.WriteByte(c)
default:
fmt.Fprintf(&b, `\x%02x`, c)
}
}
return b.String() + ".mount"
}
// MountUnit mounts the export nfs at path. hard: a job waits for a server
// that is gone rather than reading errors; nconnect spreads a client's
// traffic over several connections. The version is the highest both
// sides speak.
func MountUnit(nfs, path string) string {
return "# Written by llmgw (BACKLOG P): the Slurm cluster's shared files.\n" +
"[Unit]\nDescription=llmgw: the Slurm cluster's shared files\n" +
"After=network-online.target\nWants=network-online.target\n\n" +
"[Mount]\nWhat=" + nfs + "\nWhere=" + path + "\nType=nfs\n" +
"Options=hard,nconnect=4,_netdev\nTimeoutSec=60\n\n" +
"[Install]\nWantedBy=remote-fs.target\n"
}
// Users: Linux users llmgw makes from its accounts, the same login and UID
// on every machine, logging in with the SSH keys given in the portal.
const (
// AuthorizedKeysDir holds each user's keys, root's to write, so a
// user's own home (on a file system shared with others) need hold none.
AuthorizedKeysDir = "/etc/llmgw/ssh-keys"
// SSHDropIn has sshd look there too, after the user's own files.
SSHDropIn = "/etc/ssh/sshd_config.d/50-llmgw-slurm.conf"
// MinUID is the lowest UID llmgw gives.
MinUID = 200000
)
// SSHConf is SSHDropIn's settings.
const SSHConf = "# Written by llmgw (BACKLOG P): the Slurm users' keys, from llmgw's portal.\n" +
"AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 " + AuthorizedKeysDir + "/%u\n"
// login is what useradd takes, as llmgw gives it.
var login = regexp.MustCompile(`^[a-z][a-z0-9_-]{0,31}$`)
// ValidLogin is whether a login is one llmgw gives.
func ValidLogin(s string) bool { return login.MatchString(s) }
// ParseSSHKeys reads OpenSSH public keys, one a line, as authorized_keys
// takes them, without options: what a person pastes into the portal.
// Blank lines and # comments are skipped; each key comes back as its type,
// its base64 and its comment.
func ParseSSHKeys(text string) ([]string, error) {
var out []string
for i, line := range strings.Split(text, "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
pk, comment, opts, _, err := ssh.ParseAuthorizedKey([]byte(line))
if err != nil {
return nil, fmt.Errorf("line %d is not an SSH public key (as ~/.ssh/id_ed25519.pub has it)", i+1)
}
if len(opts) > 0 {
return nil, fmt.Errorf("line %d has options before its key; only the key is taken", i+1)
}
k := strings.TrimSpace(string(ssh.MarshalAuthorizedKey(pk)))
if comment = strings.Join(strings.Fields(comment), " "); comment != "" {
k += " " + comment
}
if !slices.Contains(out, k) {
out = append(out, k)
}
}
if len(out) > 20 {
return nil, fmt.Errorf("%d keys: 20 at most", len(out))
}
return out, nil
}
+42
View File
@@ -0,0 +1,42 @@
package slurm
import (
"strings"
"testing"
)
func TestExportsAndMountUnit(t *testing.T) {
e := Exports("/shared", []string{"10.0.0.22", "10.0.0.21", "10.0.0.22"})
if !strings.Contains(e, `"/shared" 10.0.0.21(rw,sync,no_subtree_check,root_squash) 10.0.0.22(rw,`) ||
strings.Count(e, "10.0.0.22") != 1 {
t.Fatalf("exports:\n%s", e)
}
if strings.Contains(Exports("/shared", nil), "/shared\"") {
t.Fatal("an export to nobody")
}
for path, want := range map[string]string{"/shared": "shared.mount", "/data/hpc": "data-hpc.mount",
"/hpc-data": `hpc\x2ddata.mount`, "/.b": `\x2eb.mount`} {
if got := MountUnitName(path); got != want {
t.Errorf("%s: %s, want %s", path, got, want)
}
}
if u := MountUnit("10.0.0.11:/shared", "/shared"); !strings.Contains(u, "What=10.0.0.11:/shared\nWhere=/shared\nType=nfs") {
t.Fatalf("unit:\n%s", u)
}
}
func TestParseSSHKeys(t *testing.T) {
key := "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILq3vMdLAKlaDdxwrSgCtP/j/qrF4nzxNBNGuJmK5vxh"
got, err := ParseSSHKeys("# mine\n\n" + key + " ayse@laptop \n" + key + " ayse@laptop\n")
if err != nil || len(got) != 1 || got[0] != key+" ayse@laptop" {
t.Fatalf("%q %v", got, err)
}
for _, bad := range []string{"not a key", `command="rm -rf /" ` + key} {
if _, err := ParseSSHKeys(bad); err == nil {
t.Errorf("%q taken", bad)
}
}
if !ValidLogin("ali_veli") || ValidLogin("9x") || ValidLogin("Ayse") {
t.Error("ValidLogin")
}
}
+37 -8
View File
@@ -120,6 +120,9 @@ type Node struct {
Addr string
Hardware
GPUs []GPU
// Lent is whether llmgw has lent it to Slurm: it is not written
// drained, or every start of slurmctld on these files drains it again.
Lent bool `json:"-"`
}
// Cluster is everything slurm.conf says.
@@ -128,6 +131,10 @@ type Cluster struct {
Controller string
ControllerAddr string
Partition string
// EnforceAccounts refuses a job whose user has no account in Slurm's
// accounting: with llmgw's users, each is on the accounts of the
// projects they are in, and a job is charged to one of them.
EnforceAccounts bool
// Nodes without hardware (not reported yet) are left out of the files
// until they are.
Nodes []Node
@@ -193,6 +200,12 @@ func (c Cluster) SlurmConf() string {
b.WriteString(header)
fmt.Fprintf(&b, "ClusterName=%s\n", c.Name)
fmt.Fprintf(&b, "SlurmctldHost=%s(%s)\n", c.Controller, c.ControllerAddr)
// The jobs' records, in slurmdbd beside slurmctld (DbdConf).
fmt.Fprintf(&b, "AccountingStorageType=accounting_storage/slurmdbd\nAccountingStorageHost=%s\nAccountingStoragePort=%d\n",
c.ControllerAddr, DbdPort)
if c.EnforceAccounts {
b.WriteString("# A job runs on an account of a project its user is in.\nAccountingStorageEnforce=associations\n")
}
b.WriteString(`SlurmctldParameters=enable_configless
SlurmctldPort=6817
SlurmdPort=6818
@@ -208,19 +221,22 @@ SlurmdLogFile=` + LogDir + `/slurmd.log
SlurmctldPidFile=/run/slurmctld.pid
SlurmdPidFile=/run/slurmd.pid
ProctrackType=proctrack/cgroup
# Each job has a step from its start, where pam_slurm_adopt puts a user's
# ssh session on the node.
PrologFlags=Contain
TaskPlugin=task/cgroup,task/affinity
# Nothing gathered for accounting until there is somewhere to keep it
# (slurmdbd, later); gathered and not kept is an error in every log.
JobAcctGatherType=jobacct_gather/none
# What each job used, from its cgroup, for the records; the cards too.
JobAcctGatherType=jobacct_gather/cgroup
AccountingStorageTRES=gres/gpu
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory
SchedulerType=sched/backfill
PriorityType=priority/multifactor
GresTypes=gpu
MpiDefault=none
# No mail sent; without it every start says MailProg is invalid.
MailProg=/bin/true
# Without accounting storage, 23.11's main scheduler marks a batch job
# InvalidAccount; backfill starts it on its next pass, so it is kept short.
# Backfill looks again every 10 s, not every 30.
SchedulerParameters=bf_interval=10
# A node that went down and comes back is taken again; one llmgw drained
# stays drained until llmgw resumes it.
@@ -230,6 +246,15 @@ SlurmdTimeout=300
`)
nodes := c.ready()
names := make([]string, 0, len(nodes))
// A job that asks for no memory gets this much per CPU: the share of
// the leanest node. Without it it gets the whole node's, and the jobs
// on a node run one at a time.
var perCPU int64
for _, n := range nodes {
if share := int64(float64(n.RealMemory)*memoryKept) / int64(n.CPUs); perCPU == 0 || share < perCPU {
perCPU = share
}
}
for _, n := range nodes {
names = append(names, n.Name)
fmt.Fprintf(&b, "NodeName=%s NodeAddr=%s CPUs=%d", n.Name, n.Addr, n.CPUs)
@@ -247,11 +272,15 @@ SlurmdTimeout=300
}
// Drained until llmgw lends it: a node new to the controller would
// otherwise take jobs in the minute before its agent drains it.
fmt.Fprintf(&b, " State=DRAIN Reason=%q\n", ReasonNew)
if n.Lent {
b.WriteString("\n")
} else {
fmt.Fprintf(&b, " State=DRAIN Reason=%q\n", ReasonNew)
}
}
if len(names) > 0 {
fmt.Fprintf(&b, "PartitionName=%s Nodes=%s Default=YES MaxTime=INFINITE State=UP\n",
c.Partition, strings.Join(names, ","))
fmt.Fprintf(&b, "PartitionName=%s Nodes=%s Default=YES MaxTime=INFINITE State=UP DefMemPerCPU=%d\n",
c.Partition, strings.Join(names, ","), perCPU)
}
return b.String()
}
Loaded 100 of 133 files, more files were not shown because too many files have changed in this diff. Show more