Slurm's jobs in llmgw's accounting: projects as accounts, jobs priced

With slurm.users llmgw, each project is a Slurm account, its organization's
and team's above it, with the project's members and its team's and
organization's on it; slurm.conf holds a job to its user's accounts. The
controller's agent keeps them with sacctmgr, only what it made, and reports
the jobs that ended in the last day; the leader records each once, its
GPU-hours (core-hours without cards) priced at auth.gpu_hour_cost into its
project's usage as model slurm/PARTITION.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

BACKLOG.md

@@ -49,7 +49,8 @@ Face, done 2026-10-30), then M5 (engines without the internet), M3, M4, M6.
 **Scale.** O: more than one control plane answering clients; O1 (measuring one router's ceiling) first.
 
 **Governance.**
-7. **C1**: a tenant-admin role, and forms hidden from viewers.
+7. ~~**C1**~~: done 2026-11-08. An organization's admin (`--role admin` on an org membership) changes its teams,
+   projects, people and keys from /org; viewers are drawn no forms; forms made compact.
 8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
 
 **When there is room.**
@@ -435,9 +436,9 @@ portal is not done — the page shows; the console and ctl change.
    API middleware and the /admin pages (with each change's result), `ctl
    audit`, the console's Audit view; staff roles viewer / operator /
    billing / admin. Left: before/after values (the request's fields are
-   recorded, not the state they replaced); a tenant-admin role that changes
-   its own organization (managers still only view it); hiding the forms a
-   viewer cannot submit. **Audit log and roles (M).** Today "audit" is slog lines
+   recorded, not the state they replaced). Done 2026-11-08: an organization's
+   admin who changes its own organization from /org (managers still only view
+   it), and the forms a viewer cannot submit no longer drawn. **Audit log and roles (M).** Today "audit" is slog lines
    (`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag
    plus an org "manager" who can view. Add `audit_events` (who, what,
    before/after, source) written by the `adminAction`/`tenancyAction`
@@ -462,8 +463,9 @@ portal is not done — the page shows; the console and ctl change.
 5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node
    lost, budget spent, update available; signed JSON or Slack; retries,
    hourly de-duplication), `ctl notify test`, Alertmanager receiver
-   templates. Left: `server init` and the console do not yet ask for a
-   receiver — the templates are filled in by hand. Events are claimed in
+   templates. `server init` and both setup screens ask for webhooks and test
+   each (d1e9186, 2026-10-14); with none set, `ctl status` and the console's
+   Overview say the alerts go nowhere (2026-11-12). Events are claimed in
    the store before they are sent (2026-10-10), so two control planes send
    each once.
    **Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
@@ -1982,6 +1984,175 @@ one control plane's router can. In order:
 - O3. Only if O2 is not enough: stateless gateway machines (`llmgw server --role gateway`) in front of the leader's
   control plane, as many as the traffic needs.
 
+## P. Training beyond one card (written 2026-11-11) — decision open
+
+F4 trains a LoRA on one card of a node the caller names (`ctl train start --node`), one job per node, no resume.
+Larger work (full fine-tuning, a LoRA on a 70B, several GPUs or machines) needs more. Two ways:
+
+**A. In llmgw.** Add, in order: a job that takes N whole GPUs on one node (torchrun, FSDP/DeepSpeed in the agent's
+venv); placement by the control plane instead of `--node`, freeing GPUs from replicas by the scaling schedule or by
+draining; checkpoints to shared storage and resume when an agent or node goes; datasets uploaded or read from
+storage, not only KB pairs; then gang placement of N nodes with a rendezvous on the first, the NCCL environment from
+the fabric discovery (IB/NVLink, `llmgw doctor`); a console screen with logs and loss. Size: L for one node, XL for
+several. For: one product, one install, one set of users and grants (KB rules carry over), GPUs shared with
+inference without a second scheduler. Against: it is Slurm's gang scheduling, requeue and accounting written again;
+multi-node NCCL cannot be tried on the PoC (one GPU node, no IB); every customer's training code would have to fit
+our job format.
+
+**B. Slurm beside llmgw.** The TUI installs Slurm on the nodes the customer picks (munge, slurmctld on the control
+planes, slurmd, users from the same LDAP/AD), with pyxis/enroot for containers. A node is handed between the two:
+llmgw drains its replicas and `scontrol update state=resume`, and back. llmgw's part is what comes out: a checkpoint
+or adapter from a Slurm job published to the mirror (`ctl train import`), served through canary and the eval. Size:
+M. For: what research and university customers already run (`sbatch`, `srun torchrun`), any training code, mature
+multi-node and requeue; we write only the install and the handover. Against: a second system with its own failures,
+upgrades and users to keep in step; GPUs split by hand or by the handover, not by demand; access rules on training
+data are Slurm's and the file system's, not the KB's.
+
+Both need shared storage (NFS, BeeGFS, Lustre) the customer provides; neither changes F4.
+
+Recommendation: keep F4 in llmgw for small single-card adapters tied to KB grants; take B for anything larger;
+build only A's first step (N GPUs on one node, placed by the control plane, with resume) if customers ask for
+mid-size jobs without wanting Slurm.
+
+**B taken (2026-11-11).** Design: the cluster's settings say it (`slurm: enabled, cluster_name, controller, nodes,
+lent, partition`); the leader tells each named node's agent (root) what to be, the agents install and run it.
+- The controller is an llmgw node (its agent runs slurmctld); `nodes` run slurmd configless
+  (`--conf-server`), so slurm.conf, gres.conf and cgroup.conf live only on the controller.
+- The munge key is derived from the cluster secret (HKDF), so nothing new is stored or copied by hand.
+- Each node reports `slurmd -C`; the leader writes slurm.conf from those lines, its mgmt address and its GPUs
+  (`Gres=gpu:<type>:N`, gres.conf with the device files), and sends it to the controller again when one changes.
+- A node in `nodes` belongs to llmgw until it is in `lent`: drained in Slurm (reason "llmgw: serving models").
+  Lent: llmgw drains it as `ctl drain` does (kept in force each tick, as facility holds are, so a new leader
+  keeps it) and, once no replica is left, its agent resumes it in Slurm. Taken out of `lent`: the agent drains
+  it in Slurm, reports the jobs left, and when none are the leader makes it ready for llmgw again.
+- Phases: P1 the `slurm` package (conf, key, parsers) and settings; P2 the agent side (proto, install, munge,
+  slurmd/slurmctld, status); P3 the leader side (conf from reports, lend/reclaim); P4 `ctl slurm`
+  (status, lend, reclaim) and a TUI screen; P5 `ctl train import` of a Slurm job's output into the mirror.
+  Later: a backup slurmctld on a second node, accounting (slurmdbd; it takes MySQL/MariaDB only, not the Patroni PG), users from LDAP/AD.
+- Tried on the PoC 2026-11-11 (dev build v0.16.3-17-g849760c-slurm on 140, 150, node05, node06; Ubuntu
+  24.04, Slurm 23.11.4 from the archive): `ctl slurm setup --controller llmgw-node06 --nodes llmgw-node05`
+  installed munge, slurmctld and slurmd, node05 entered drained with its slurmd -C hardware (8 CPUs,
+  31132 MiB after the 3%). `lend`: its 2 replicas moved off, then IDLE in Slurm; srun and sbatch ran there.
+  `reclaim` with a 90 s job running: MIXED+DRAIN, llmgw kept off, ready again within a minute of the job
+  ending. Found and fixed on the way: a client command needs its files even for --version (version now
+  from the daemon); the packaged unit is Type=notify (slurmd needs --systemd); slurmd's own "slurmstepd
+  infinity" was counted as a job; without accounting storage 23.11's main scheduler marks batch jobs
+  InvalidAccount and backfill starts them, one a pass (bf_interval=10: 6-10 s each). So many small jobs
+  want slurmdbd. — Wrong cause, found 2026-11-11: a job asking no memory got the whole node's (ReqTRES
+  mem=all), so a node ran one job at a time and each next one waited for backfill's pass. Fixed: the
+  partition's DefMemPerCPU is the leanest node's share; slurmdbd is not needed for this. InvalidAccount is real too:
+  23.11 dropped accounting_storage/none, and without slurmdbd the main scheduler passes every job by;
+  backfill starts it within bf_interval (10 s). priority/basic did not change that (fair share needs
+  slurmdbd anyway). Starting on submission needs slurmdbd, on MySQL/MariaDB.
+- slurmdbd, done 2026-11-12 (748f777): the controller's agent installs MariaDB (bound to 127.0.0.1; a
+  MariaDB that was there keeps its settings), makes `slurm_acct_db` and its user (password from the munge
+  key, through a 0600 file, never argv), runs slurmdbd (6819, in the firewall for the cluster) and adds
+  the cluster, before slurmctld. slurm.conf: AccountingStorageType=slurmdbd, jobacct_gather/cgroup,
+  AccountingStorageTRES=gres/gpu, priority/multifactor. Tried on the PoC: five jobs started on
+  submission, `sacct -a` has them with their TRES. Not tried: a fresh `server init --slurm` (needs VMs);
+  a Red Hat controller.
+- Fresh install tried 2026-11-12 on VMs (3 control planes with Patroni, 2 CPU nodes, `server init
+  --slurm`): Slurm, slurmdbd and MariaDB came up with the install; two-node srun/sbatch on the lent nodes,
+  `sacct` has them; leader failover kept serving. Open from it: `--slurm` needs the first machine to be a
+  node (the controller is its node), so a site wanting control planes without nodes chooses the
+  controller later with `ctl slurm setup`; batch output lands on the job's first node until there is
+  shared storage; on shared CPUs the judge refuses the first request after a placement change (fail
+  closed, ~10 s), and its advice repeats `--cores 4` to a judge that has 4. Tried again on the idle
+  VMs: one judge restarted in place, then both at once, requests every second: 48 of 48 served; the
+  first after a cold start took 7.6–8.9 s end to end against ~3 s, within the 5 s judge budget. So the
+  refusal was the install's load on the CPUs; if it is seen again, warm a new judge with one judgment
+  before it takes requests.
+- Shared files and users, built 2026-11-12: `slurm.shared` {path, nfs | server} — the site's export
+  mounted (a systemd mount unit, hard, nconnect=4), an llmgw node exporting the path itself
+  (/etc/exports.d, to the Slurm machines' addresses, root_squash; 2049 in the firewall), or the site's
+  own mount only looked for; `slurm.users: llmgw` — every active account a Linux user everywhere
+  (`slurm_users`: login from the e-mail, UID from 200000, never reused; homes made on the exporting
+  machine, else the controller; keys in /etc/llmgw/ssh-keys/%u through an sshd drop-in), SSH keys
+  from the portal's Profile or `ctl slurm users keys`. A user already on a machine with another UID
+  is left alone and said. The site has AD (not yet live): users from it, through sssd, are the next
+  step (`slurm.users` empty today leaves them to the site).
+- Logging in to a node, done 2026-11-14: llmgw's users reach the controller freely, another Slurm
+  machine only while a job of theirs runs there (pam_slurm_adopt, installed by the agent; PrologFlags=
+  Contain). Only the users listed in the agent's users file are held: root and the machine's own users
+  log in as before, so a wrong line never locks out who would fix it. pam_systemd would move the session
+  out of the job, so sshd's stack takes the session file in place of its include, with pam_systemd
+  skipped for the listed users alone; where that cannot be done safely (nested includes, jumps across
+  it) they are held to having a job, not to its cores and cards, and the agent says so. Tried on the
+  VMs: ayse without a job refused ("no active jobs on this node"), with one in
+  job_N/step_extern (nproc 1), cut off when it ended; mehmet refused; admin's session unchanged.
+  Left: sites whose users come from AD (sssd) are not held yet; a group of theirs would be the list.
+- A GPU node, tried 2026-11-14 on the PoC (cuda01, Quadro RTX 6000, Slurm 23.11.4): gres.conf
+  `Type=rtx6000 File=/dev/nvidia0`, slurm.conf `Gres=gpu:rtx6000:1`. `srun --gres=gpu:1`:
+  CUDA_VISIBLE_DEVICES=0, the one card; torch found it and ran 14.1 TFLOPS (fp32 matmul). Without
+  --gres: "No devices found", torch none (ConstrainDevices holds). `--gres=gpu:2`: refused. sacct has
+  `gres/gpu=1` in AllocTRES. Found: lending a node whose GPU model has nowhere else to go waits out
+  the drain grace (5 min) before the replica stops, and `ctl slurm status` says only "1 replica(s)
+  still to move off"; it should say it waits for a replacement that cannot be placed, and until when.
+  Done (c3a75dc): it says the time.
+- Jobs in containers, done 2026-11-14: `slurm.containers` (`ctl slurm containers on`). Each agent
+  installs enroot 4.2.1 (NVIDIA's .deb/.rpm, sha256 pinned) and builds pyxis 0.24.0 (source tarball,
+  sha256 pinned) against its own Slurm (gcc, make, libslurm-dev); on a machine with nvidia-smi,
+  nvidia-container-cli from NVIDIA's repository, its key's digest pinned. enroot.conf keeps each
+  user's runtime, cache and data under /run/enroot and /var/lib/enroot (no login session needed), the
+  shared path bind-mounted in every container. The leader writes plugstack.conf ("required") only
+  once every machine reports both, since a machine without the plugin would fail every job. Tried on
+  the VMs: on in ~30 s; ayse's srun --container-image=ubuntu:24.04 on n1 ran as uid 200001 with
+  /shared inside and nproc 1; a two-node sbatch ran a rank on each. Seen once: a node's pull timed out
+  reaching auth.docker.io and the job failed (network, not pyxis). Not tried: GPUs in a container
+  (cuda01's agent is an older build; needs the peer protocol), Red Hat.
+- AD end to end, tried 2026-11-14 (decided: one AD per install, the existing auth.ldap group mapping, no
+  tree from AD, no sssd unless a site asks): the VMs' auth.ldap against Samba DC VM 165 (ldaps by IP,
+  CA in the settings, ml-team -> team:Test/arge). ada signed in on the portal with her AD password: an
+  account, in Test/arge, Slurm user ada UID 200003; her key added in the portal's Profile, ssh to the
+  controller ~20 s later, sbatch ran on n1 as ada. Disabled in AD: suspended in 56 s, the Linux user
+  gone in 89 s, ssh refused, home kept. Enabled: back in 32 s, same UID, key kept. Taken out of
+  ml-team: out of the team in 11 s; put back: in again in 56 s.
+- P6, done 2026-11-15 but for budgets: each project a Slurm account (org and team accounts above it,
+  ids keyed by kind: the first install's org, team and project share the id "default"), its members and its
+  team's and org's on it; AccountingStorageEnforce=associations; the controller's agent keeps them with
+  sacctmgr (only what it made; a user's default account moved before an association goes, the user taken out
+  when none is left) and reports the jobs that ended in the last day; the leader records each once
+  (slurm_jobs, keyed cluster/id) with a usage row, model slurm/PARTITION, priced at gpu_hour_cost or
+  core_hour_cost. Tried on the VMs: accounts made, ayse's job on her project's account ran and another
+  project's was refused, both jobs in deneme's usage at 345 and 0 micro-units. Left: budgets (a job's cost
+  is not counted against a budget, which binds keys; GrpTRESMins from a project's budget would hold its jobs).
+- P6 as written 2026-11-12: The tenancy tree to Slurm:
+  each project an account (`sacctmgr add account`, parent its team and organization), each member a user
+  of it, kept by the leader from the store; `sbatch --account=PROJECT`. The records back: the leader reads
+  `sacct --json` for jobs ended since the last read (through the controller's agent) into a
+  `batch_usage` table (job, user, project, start, end, GPU-seconds by class, CPU-seconds, memory), priced
+  per GPU-hour and core-hour (auth.gpu_hour_cost, core_hour_cost: the same basis `ctl prices suggest`
+  prices tokens from, 2026-11-15) into cost_micros, summed with the requests' usage on the usage pages and in
+  budgets. Budgets enforced: a project's remaining budget as GPU-minutes in its account's GrpTRESMins, so
+  Slurm holds its new jobs once spent. Needs the same users on every node (open: LDAP/sssd or
+  llmgw-made). Also found: slurm.conf
+  wrote every node State=DRAIN, so each slurmctld start (on any change of the files) drained a lent node
+  until its agent resumed it, up to a minute; a lent node is now written without it. Not tried yet: a GPU node (gres).
+- Failover while lent, tried 2026-11-11 (v0.16.3-23): the first try killed the job (NODE_FAIL). A new
+  leader had heard from no node yet and sent the controller a slurm.conf without them; slurmctld,
+  restarted on it, dropped node05 and its job. Fixed (141a1f4, 0774e90): slurm.conf's nodes and the
+  controller's address are kept in cluster_state (`slurm-nodes`) and a new leader starts from them; with
+  nothing kept (an upgraded cluster) or the store unread, the controller's files wait up to 2 min for the
+  nodes; a node is told nothing until its controller's address is known (no part stops its slurmd). Tried
+  again: a 5-minute job on lent node05, leader 140 stopped, 150 took over; slurmctld not restarted, the job
+  ran to its end; reclaimed during it, node05 was llmgw's within seconds of its end. Also 3387f97: srun in
+  SrunPortRange=60001-63000, and Slurm's ports in llmgw's firewall for the cluster's networks.
+- Built 2026-11-11: P1 (`internal/slurm`, `slurm.*` settings), P2
+  (`internal/agent/slurmnode`, proto Slurm/SlurmStatus), P3 (`internal/control/slurmsvc`, wired in the
+  leader; `GET /api/v1/slurm`), P4's commands (`ctl slurm status|setup|lend|reclaim|off`). Decisions on the
+  way: a node not heard from or with Slurm jobs is cordoned, not drained (nothing of llmgw's to move, and a
+  new leader must not move models off every Slurm node); jobs are counted from the node's own slurmstepd,
+  so a controller that is down neither hides them nor, with none, keeps the node from llmgw; a drain whose
+  reason does not start with "llmgw" is a Slurm administrator's and never resumed. Left: ~~the TUI screen~~
+  (done 2026-11-11: console **Slurm**, lend/take back on a row), P5,
+  a try on the PoC (needs agents built from this; slurm-wlm 23.11 on Ubuntu 24.04), GPU minor numbers are
+  taken to be nvidia-smi's index.
+- In the first install too (2026-11-11): `llmgw server init --slurm`, and a Slurm part on the choices
+  screen, write `slurm: {enabled, controller: <this hostname>, all_nodes: true}` into server.yaml.
+  `slurm.all_nodes` (also `ctl slurm setup --all-nodes`) runs slurmd on every node in the leader's
+  inventory, those that join later too; none is lent. Refused with `--no-node`, or a hostname Slurm does
+  not take.
+
 ## D. Facts worth knowing before starting
 
 - Admin API: `127.0.0.1:9090`, behind the token in

CHANGELOG.md

@@ -26,6 +26,126 @@ is only released once its **exit criterion** is met, not when its code lands.
 Until v1.0.0 the minor version is the breaking one: configuration formats and Go
 APIs change between minors without a deprecation period.
 
+## Unreleased
+
+### Added
+
+- **An organization's admin.** Besides member and manager, a membership of
+  an organization may now be `admin` (`llmgw users join … --org Acme --role
+  admin`, the role chosen on a membership on the people page, the console,
+  the admin API). An admin sees the organization's page as a manager does and
+  also changes the organization from it: adds, renames and removes its teams
+  and projects, adds people with an approved account by their address and
+  takes them out, makes its members managers or admins, and revokes keys spent
+  in its projects. Limits, approving and suspending accounts, the
+  organization's name and the rest of the cluster stay with the cluster's
+  administrators; an admin cannot take away their own role. Every change is in
+  the audit log.
+- **Slurm beside llmgw, for training and batch work.** `llmgw ctl slurm setup
+  --controller NODE --nodes NODE,...` (or `--all-nodes`, or `server init
+  --slurm`) has the agents install munge and Slurm from the machines' package
+  sources: slurmctld on one node, slurmd configless on the others, slurm.conf
+  and gres.conf written from what each node reports. Every node stays llmgw's,
+  drained in Slurm, until `ctl slurm lend NODE`: its replicas move off, then
+  Slurm runs jobs there. `ctl slurm reclaim NODE` takes it back once its jobs
+  end. `ctl slurm status` and the console's **Slurm** show each node. The
+  controller also keeps the jobs' records: slurmdbd, in a MariaDB on that
+  machine alone (Slurm takes MySQL or MariaDB only). Slurm's ports
+  (6817–6819, srun's 60001–63000) answer the cluster's networks only, as
+  llmgw's own do.
+- **Slurm's shared files and users.** `slurm.shared` (`ctl slurm shared`) is
+  one path on every Slurm machine: the site's NFS export mounted, an llmgw
+  node exporting it over NFS (what `server init --slurm` chooses, `/shared`
+  from the first machine), or a file system the site mounts. `slurm.users:
+  llmgw` (`ctl slurm users on`) makes each active account a Linux user on
+  every Slurm machine, the same login and UID (from 200000, never reused)
+  everywhere, its home on the shared path; people add SSH keys in the
+  portal's Profile (`ctl slurm users keys` for an operator). They log in to
+  the controller; to another Slurm machine only while a job of theirs runs
+  there, their session put in the job (pam_slurm_adopt), so nobody uses a
+  node's cards outside Slurm, nor on a node llmgw serves from. Root and the
+  machines' own users are not held to it.
+- **Slurm's jobs in llmgw's accounting.** With `slurm.users: llmgw` each
+  project is a Slurm account (organizations and teams above it), its members
+  and its team's and organization's on it, and a job must run on one of its
+  user's (`AccountingStorageEnforce=associations`). Each job that ends is
+  recorded once, its GPU-hours (or core-hours) priced at `auth.gpu_hour_cost`
+  into its project's usage as model `slurm/PARTITION`. Accounts a Slurm
+  administrator made are not touched.
+- **Prices from what the cluster costs.** `auth.gpu_hour_cost` (and
+  `core_hour_cost`) is what an hour of a GPU (a core) costs; the leader keeps
+  the hours each model's replicas hold, and `llmgw ctl prices suggest` prices
+  each model from them over the tokens it served, an output token four input
+  ones and a cached one a tenth. `--apply` writes the prices.
+- **Slurm's jobs in containers.** `llmgw ctl slurm containers on`
+  (`slurm.containers`) installs enroot and pyxis on every Slurm machine, at
+  pinned versions checked against their digests, and nvidia-container-cli on
+  GPU machines; Slurm loads pyxis once every machine has it, and then `srun
+  --container-image=...` runs a job in that image, as its user, with the
+  shared path inside.
+
+### Changed
+
+- **Forms only for those who may send them.** A viewer sees the people,
+  limits, mirror and cluster pages without their forms, an operator the
+  mirror's and a node's engines but not the rest; before, they were drawn and
+  answered "not found".
+- **Smaller forms.** A label sits close to its field, a field is as wide as
+  what goes in it rather than the page, buttons are smaller, and the short
+  forms — a new key, a password, connecting an editor, a capacity or a limit —
+  sit on one line or two instead of a screen.
+
+### Fixed
+
+- **The serve forms start at the longest context that fits.** The mirror
+  page's form began every GGUF model at 4096 tokens, and the console's at
+  4096 or the plan's figure, whatever the model was trained for. Both now
+  start at what it was trained for, as long as its KV cache fits where the
+  form places it beside the weights, else at the most that does, in whole
+  thousands; the mirror page's llama.cpp slots are as many of that, up to
+  four, as the room left holds, and its memory figure is for that pool. A
+  plan says what the model was trained for (`trained`). Strata stays at
+  its 32768; vLLM's form is unchanged.
+- **A model that failed to start was started in the same place again.**
+  Placement given the same inputs chooses the same node and card, so an
+  on-demand model whose replica failed to load was retried there, past each
+  backoff, until it was parked, with another card or node free all along.
+  The control plane now remembers where a model's replicas failed (node and
+  cards, up to eight places) and places the next one elsewhere when anywhere
+  else fits; with nowhere else it is tried there as before. Coming up, a
+  change to its definition or `ctl retry` forgets the places. The log says
+  `placing a model away from where it failed to start`, with both places.
+- **Deleted model files came back after a control plane restart.** An agent
+  registers with the files its startup scan found, and sends the list again
+  only when it changes; a file deleted after the agent started was listed
+  again by a control plane that restarted, until the next change. A new
+  stream now gets the list on its first heartbeat.
+- **`ctl weights` offered models in use for deletion.** A model of several
+  files (a set, its address ending in `/`) is fetched into a directory named
+  from the address without the slash; the sweep named it from the address
+  with it, so every set in the catalog looked like nobody's. On the PoC that
+  was 330 GB of six catalog models, the qwen3.8-flash-next Q8_0 among them;
+  `--collect`, and `ctl model remove --purge` of a set, would have deleted
+  them. Both now compute the name the fetch does.
+- **A sign-in took an organization's manager role away.** A group from single
+  sign-on, SCIM or Active Directory that puts someone in an organization wrote
+  the membership again at every sign-in or sync, as a plain member: a manager
+  made one on the people page was one only until they next signed in. A
+  membership the directory names is now added if missing and otherwise left
+  as it is.
+- **The limits form on the people page's panel** was laid out on one line, its
+  title squeezed into a column.
+- **Deleting the model files nothing uses said what it was asked to free,
+  not what it freed.** The console's "Delete model files nothing uses" and
+  `ctl weights --collect` added up every file the nodes were asked to
+  delete as soon as the request was sent, and a model of several files —
+  fetched into a directory of its own — was refused by the node as "not
+  empty", so a sweep said tens of gigabytes freed and nothing went. The
+  node now deletes such a directory whole (a link only as a link), and the
+  sweep waits for the nodes to list their files again: freed is what they
+  no longer list, and what they kept is said beside, with where to read
+  why. Removing a model with its files reports the same way.
+
 ## v0.16.3 — 2026-11-03
 
 Replicas counted per profile — seven on cards and four on CPUs under one name, the cards filled first — and every

LICENSE

@@ -0,0 +1,105 @@
+# Functional Source License, Version 1.1, ALv2 Future License
+
+## Abbreviation
+
+FSL-1.1-ALv2
+
+## Notice
+
+Copyright 2026 Ilker Manap
+
+## Terms and Conditions
+
+### Licensor ("We")
+
+The party offering the Software under these Terms and Conditions.
+
+### The Software
+
+The "Software" is each version of the software that we make available under
+these Terms and Conditions, as indicated by our inclusion of these Terms and
+Conditions with the Software.
+
+### License Grant
+
+Subject to your compliance with this License Grant and the Patents,
+Redistribution and Trademark clauses below, we hereby grant you the right to
+use, copy, modify, create derivative works, publicly perform, publicly display
+and redistribute the Software for any Permitted Purpose identified below.
+
+### Permitted Purpose
+
+A Permitted Purpose is any purpose other than a Competing Use. A Competing Use
+means making the Software available to others in a commercial product or
+service that:
+
+1. substitutes for the Software;
+
+2. substitutes for any other product or service we offer using the Software
+   that exists as of the date we make the Software available; or
+
+3. offers the same or substantially similar functionality as the Software.
+
+Permitted Purposes specifically include using the Software:
+
+1. for your internal use and access;
+
+2. for non-commercial education;
+
+3. for non-commercial research; and
+
+4. in connection with professional services that you provide to a licensee
+   using the Software in accordance with these Terms and Conditions.
+
+### Patents
+
+To the extent your use for a Permitted Purpose would necessarily infringe our
+patents, the license grant above includes a license under our patents. If you
+make a claim against any party that the Software infringes or contributes to
+the infringement of any patent, then your patent license to the Software ends
+immediately.
+
+### Redistribution
+
+The Terms and Conditions apply to all copies, modifications and derivatives of
+the Software.
+
+If you redistribute any copies, modifications or derivatives of the Software,
+you must include a copy of or a link to these Terms and Conditions and not
+remove any copyright notices provided in or with the Software.
+
+### Disclaimer
+
+THE SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTIES OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING WITHOUT LIMITATION WARRANTIES OF FITNESS FOR A PARTICULAR
+PURPOSE, MERCHANTABILITY, TITLE OR NON-INFRINGEMENT.
+
+IN NO EVENT WILL WE HAVE ANY LIABILITY TO YOU ARISING OUT OF OR RELATED TO THE
+SOFTWARE, INCLUDING INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES,
+EVEN IF WE HAVE BEEN INFORMED OF THEIR POSSIBILITY IN ADVANCE.
+
+### Trademarks
+
+Except for displaying the License Details and identifying us as the origin of
+the Software, you have no right under these Terms and Conditions to use our
+trademarks, trade names, service marks or product names.
+
+## Grant of Future License
+
+We hereby irrevocably grant you an additional license to use the Software under
+the Apache License, Version 2.0 that is effective on the second anniversary of
+the date we make the Software available. On or after that date, you may use the
+Software under the Apache License, Version 2.0, in which case the following
+will apply:
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use
+this file except in compliance with the License.
+
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed
+under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
+CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.

README.md

@@ -372,3 +372,10 @@ make rename NEW=git.example.com/team/llmgw
 ## Language
 
 Code, comments, tests, commit messages and documentation are in English.
+
+## Licence
+
+Functional Source License 1.1 with an Apache 2.0 future licence
+(FSL-1.1-ALv2); see [LICENSE](LICENSE). Each release may be used, read and
+changed for anything except offering it as a competing product or service,
+and becomes Apache 2.0 two years after it is published.

api/proto/llmgw/v1/agent.proto

@@ -30,6 +30,7 @@ message AgentMessage {
     KVEvents kv_events = 8;
     TrainStatus train_status = 9;
     TrainArtifact train_artifact = 10;
+    SlurmStatus slurm_status = 11;
   }
 }
 
@@ -78,6 +79,7 @@ message ControlMessage {
     AgentSettings agent_settings = 15;
     TrainStart train_start = 16;
     TrainCancel train_cancel = 17;
+    Slurm slurm = 18;
   }
   // trace is the W3C trace context (traceparent, and tracestate when there
   // is one) of the control plane's span that sent this command, so what the
@@ -986,3 +988,129 @@ message TrainArtifact {
   bytes data = 4;
   bool last = 5;
 }
+
+// Slurm is what this node is in the Slurm cluster run beside llmgw
+// (BACKLOG P): its controller, one of its nodes, both or neither. Sent when
+// an agent registers and whenever it changes; the agent installs what is
+// missing and keeps the daemons running as it says.
+message Slurm {
+  // controller runs slurmctld here, on files.
+  bool controller = 1;
+  // node runs slurmd here, configless, fetching its files from conf_server.
+  bool node = 2;
+  // name is this node's NodeName in slurm.conf.
+  string name = 3;
+  // conf_server is slurmctld's host:port.
+  string conf_server = 4;
+  // munge_key is the key every machine of the cluster shares.
+  bytes munge_key = 5;
+  // files are the controller's, by name in /etc/slurm; empty for a node
+  // that is not the controller.
+  map<string, string> files = 6;
+  // lent says Slurm may run jobs here now: resume the node if llmgw drained
+  // it. Otherwise it is drained, its jobs left to finish.
+  bool lent = 7;
+  // shared is the file system every machine of the Slurm cluster sees at
+  // the same path; unset or with no path, none.
+  SlurmShared shared = 8;
+  // manage_users has llmgw keep the Linux users below on this machine,
+  // and take away those it made that are no longer among them.
+  bool manage_users = 9;
+  repeated SlurmUser users = 10;
+  // make_homes has this machine make the users' home directories: the one
+  // that exports the shared path, else the controller.
+  bool make_homes = 11;
+  // containers has the machine install pyxis and enroot, so jobs may run
+  // in containers (srun --container-image), and set them up.
+  bool containers = 12;
+  // manage_accounts has the controller keep Slurm's accounts as accounts
+  // says: one for each organization, team and project, nested, the users
+  // on each project's. Only those llmgw made are changed or taken away.
+  bool manage_accounts = 13;
+  repeated SlurmAccount accounts = 14;
+}
+
+// SlurmAccount is an account in Slurm's accounting, parents before their
+// children.
+message SlurmAccount {
+  string name = 1;
+  string parent = 2;
+  repeated string users = 3;
+}
+
+// SlurmJob is a job that ended, as sacct has it, for llmgw's accounting.
+message SlurmJob {
+  string id = 1;
+  string user = 2;
+  string account = 3;
+  string partition = 4;
+  string state = 5;
+  int64 start_unix = 6;
+  int64 end_unix = 7;
+  int64 elapsed_seconds = 8;
+  int32 cpus = 9;
+  int32 gpus = 10;
+}
+
+// SlurmShared is where the Slurm cluster's shared files come from.
+message SlurmShared {
+  // path is where every machine sees them, as /shared.
+  string path = 1;
+  // nfs is an export, host:/path, to mount at path; empty when this
+  // machine exports path itself (serve) or the site mounts it.
+  string nfs = 2;
+  // serve has this machine export path over NFS to clients.
+  bool serve = 3;
+  // clients are the addresses the export answers.
+  repeated string clients = 4;
+}
+
+// SlurmUser is a Linux user that runs jobs, the same on every machine.
+message SlurmUser {
+  string login = 1;
+  // uid is also the user's own group's gid.
+  uint32 uid = 2;
+  // name is the account's, for the GECOS field.
+  string name = 3;
+  string home = 4;
+  // ssh_keys are OpenSSH public keys, one a line of authorized_keys.
+  repeated string ssh_keys = 5;
+}
+
+// SlurmStatus is what this node's Slurm daemons are doing, sent when it
+// changes and every few minutes while the node has a part in Slurm.
+message SlurmStatus {
+  // version is Slurm's, as "23.11.4"; empty when it is not installed.
+  string version = 1;
+  // hardware is `slurmd -C` on a node, which slurm.conf must match.
+  string hardware = 2;
+  bool slurmd_running = 3;
+  bool slurmctld_running = 4;
+  // state and reason are the controller's for this node, from scontrol.
+  string state = 5;
+  string reason = 6;
+  // jobs is how many job steps run on this node now, counted from their
+  // slurmstepd so a controller that is down does not hide them; -1 when the
+  // agent cannot tell.
+  int32 jobs = 7;
+  // error is the last thing that went wrong keeping Slurm, empty when
+  // nothing did.
+  string error = 8;
+  // files_sha256 is of the controller's files as written, so the leader
+  // knows they arrived.
+  string files_sha256 = 9;
+  // slurmdbd_running is the accounting daemon on the controller, which keeps
+  // the jobs' records in a MariaDB beside it.
+  bool slurmdbd_running = 10;
+  // shared is the shared path as this machine has it: "mounted",
+  // "exported", "missing", or empty when there is none.
+  string shared = 11;
+  // users is how many of llmgw's users this machine has.
+  int32 users = 12;
+  // containers is what this machine has for jobs in containers, as
+  // "pyxis 0.24.0, enroot 4.2.1"; empty while it has not got both.
+  string containers = 13;
+  // ended are, on the controller, the jobs that ended in the last day; the
+  // leader records each once.
+  repeated SlurmJob ended = 14;
+}

docs/admin-api.openapi.json

@@ -95,6 +95,9 @@
     {
       "name": "presets"
     },
+    {
+      "name": "prices"
+    },
     {
       "name": "projects"
     },
@@ -116,6 +119,9 @@
     {
       "name": "settings"
     },
+    {
+      "name": "slurm"
+    },
     {
       "name": "summary"
     },
@@ -4393,6 +4399,34 @@
         }
       }
     },
+    "/api/v1/prices/suggest": {
+      "get": {
+        "operationId": "getPricesSuggest",
+        "summary": "Each model's price, from what it held of the cluster",
+        "description": "days (30 if not given) is the window. A model's cost is the GPU- and core-hours its replicas held, idle ones too, at auth.gpu_hour_cost and auth.core_hour_cost; spread over its tokens, an output token four input ones and a cached prompt token a tenth of one, it is price, in micro-units a thousand tokens as auth.pricing takes them. current is the price in force. 409 with no cost set.",
+        "tags": [
+          "prices"
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/PriceSuggestions"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
     "/api/v1/projects": {
       "post": {
         "operationId": "postProjects",
@@ -5078,6 +5112,117 @@
         }
       }
     },
+    "/api/v1/slurm": {
+      "get": {
+        "operationId": "getSlurm",
+        "summary": "The Slurm cluster beside llmgw, and where each of its nodes is",
+        "description": "From the cluster's settings (slurm.*) and what each node's agent last said. A node's phase is llmgw (drained in Slurm, no job left), lending (its replicas moving off), lent, reclaiming (drained in Slurm, jobs still running) or unknown (its agent has not said).",
+        "tags": [
+          "slurm"
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/slurmsvc.Status"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/slurm/users": {
+      "get": {
+        "operationId": "getSlurmUsers",
+        "summary": "The Linux users made from llmgw's accounts for the Slurm cluster",
+        "description": "Every active account, with its login, UID and SSH keys; an account is given its login and UID the first time it is listed. The machines have them when slurm.users is \"llmgw\".",
+        "tags": [
+          "slurm"
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "array",
+                  "items": {
+                    "$ref": "#/components/schemas/slurmsvc.User"
+                  }
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
+    "/api/v1/slurm/users/{user}/ssh-keys": {
+      "put": {
+        "operationId": "putSlurmUsersUserSshKeys",
+        "summary": "Replace an account's SSH public keys",
+        "description": "{user} is an ID or e-mail. keys is OpenSSH public keys, one a line, without options; empty takes them all away. The machines have them within a minute.",
+        "tags": [
+          "slurm"
+        ],
+        "parameters": [
+          {
+            "name": "user",
+            "in": "path",
+            "required": true,
+            "description": "The person's ID or email.",
+            "schema": {
+              "type": "string"
+            }
+          },
+          {
+            "$ref": "#/components/parameters/Actor"
+          }
+        ],
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/SlurmKeysRequest"
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "OK",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/slurmsvc.User"
+                }
+              }
+            }
+          },
+          "401": {
+            "$ref": "#/components/responses/Unauthorized"
+          },
+          "default": {
+            "$ref": "#/components/responses/Error"
+          }
+        }
+      }
+    },
     "/api/v1/summary": {
       "get": {
         "operationId": "getSummary",
@@ -6234,7 +6379,7 @@
       },
       "CollectResult": {
         "type": "object",
-        "description": "CollectResult is what was deleted on one node, or why nothing was.",
+        "description": "CollectResult is what was deleted on one node, or why nothing was: what the node no longer lists once asked, not what it was asked to delete.",
         "properties": {
           "node": {
             "type": "string"
@@ -6246,6 +6391,14 @@
             "type": "integer",
             "format": "int64"
           },
+          "left": {
+            "type": "integer",
+            "description": "Left and LeftBytes are what the node still lists: refused, or not gone by the time the sweep stopped waiting. Its agent's log says why."
+          },
+          "left_bytes": {
+            "type": "integer",
+            "format": "int64"
+          },
           "error": {
             "type": "string"
           }
@@ -8724,6 +8877,73 @@
           }
         }
       },
+      "PriceSuggestion": {
+        "type": "object",
+        "description": "PriceSuggestion is one model's.",
+        "properties": {
+          "model": {
+            "type": "string"
+          },
+          "gpu_hours": {
+            "type": "number"
+          },
+          "core_hours": {
+            "type": "number"
+          },
+          "cost": {
+            "type": "number",
+            "description": "Cost is what it held cost, in budget units."
+          },
+          "prompt_tokens": {
+            "type": "integer",
+            "format": "int64"
+          },
+          "completion_tokens": {
+            "type": "integer",
+            "format": "int64"
+          },
+          "price": {
+            "$ref": "#/components/schemas/accounting.Price",
+            "description": "Price is the suggestion; zero with a Note saying why there is none."
+          },
+          "note": {
+            "type": "string"
+          },
+          "current": {
+            "$ref": "#/components/schemas/accounting.Price"
+          }
+        }
+      },
+      "PriceSuggestions": {
+        "type": "object",
+        "description": "PriceSuggestions is a model's price from its cost, for each model.",
+        "properties": {
+          "days": {
+            "type": "integer",
+            "description": "Days is the window; RecordedFrom the first hour held is known for, later than the window's start on a cluster that began keeping it since, which makes the window shorter than asked."
+          },
+          "since": {
+            "type": "string",
+            "format": "date-time"
+          },
+          "recorded_from": {
+            "type": "string",
+            "format": "date-time"
+          },
+          "gpu_hour_cost": {
+            "type": "number"
+          },
+          "core_hour_cost": {
+            "type": "number"
+          },
+          "models": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/PriceSuggestion"
+            }
+          }
+        }
+      },
       "ProfileReplicasRequest": {
         "type": "object",
         "description": "ProfileReplicasRequest counts a model's replicas per profile: profile name to replicas, and CPU, for a llama.cpp model with no profile on CPUs, the one to add.",
@@ -9362,6 +9582,15 @@
           }
         }
       },
+      "SlurmKeysRequest": {
+        "type": "object",
+        "description": "SlurmKeysRequest is a person's SSH public keys, one a line, as authorized_keys has them; empty takes them all away.",
+        "properties": {
+          "keys": {
+            "type": "string"
+          }
+        }
+      },
       "StoreMember": {
         "type": "object",
         "description": "StoreMember is one member of a replicated store, as its Patroni reports it.",
@@ -9464,6 +9693,10 @@
           "guard_off": {
             "type": "boolean",
             "description": "GuardOff says the guardrails are turned off (guard.enabled: false): no request is checked, whatever the policies say."
+          },
+          "no_alerts": {
+            "type": "boolean",
+            "description": "NoAlerts says the cluster's settings name no webhook: a model parked, a node lost or a judge down is told nobody."
           }
         }
       },
@@ -9872,6 +10105,35 @@
           }
         }
       },
+      "accounting.Price": {
+        "type": "object",
+        "description": "Price is what one model costs, in micro-units per thousand tokens.\n\nMicro-units rather than a float because spend is accumulated by repeated addition, and repeated addition of floats drifts. What a unit means is the operator's: self-hosted models have no per-token price, so it may be GPU-seconds, a notional rate, or 1000 everywhere — which turns a budget into a token quota.",
+        "properties": {
+          "PromptPer1k": {
+            "type": "integer",
+            "format": "int64"
+          },
+          "CompletionPer1k": {
+            "type": "integer",
+            "format": "int64"
+          },
+          "CachedPer1k": {
+            "type": "integer",
+            "format": "int64",
+            "description": "CachedPer1k is what a prompt token costs that the engine or provider read from its cache instead of computing: the part of prompt_tokens the answer counts as cached_tokens. Zero charges those at PromptPer1k, as before there was a rate for them."
+          },
+          "PerImage": {
+            "type": "integer",
+            "format": "int64",
+            "description": "PerImage is what one picture costs, for a model that makes them: a picture's cost is the card's time, which its prompt's few tokens do not measure."
+          },
+          "PerMesh": {
+            "type": "integer",
+            "format": "int64",
+            "description": "PerMesh is what one 3D model costs: minutes of a card, which no token count measures."
+          }
+        }
+      },
       "accounting.Rates": {
         "type": "object",
         "description": "Rates are an accountant's two limiters' counts.",
@@ -10746,6 +11008,24 @@
           }
         }
       },
+      "config.SharedStorage": {
+        "type": "object",
+        "description": "SharedStorage is where the Slurm cluster's shared files come from: the site's NFS export (NFS), an NFS export llmgw keeps on one of its nodes (Server), or, with neither, a file system the site mounts itself (Lustre, BeeGFS, GPFS), which llmgw only looks for.",
+        "properties": {
+          "Path": {
+            "type": "string",
+            "description": "Path is where every machine sees it, as /shared."
+          },
+          "NFS": {
+            "type": "string",
+            "description": "NFS is the site's export, host:/path, mounted at Path on every machine of the Slurm cluster."
+          },
+          "Server": {
+            "type": "string",
+            "description": "Server is the llmgw node, the controller or one of the nodes, that exports Path over NFS to the others: for a site with no storage of its own. One machine: not highly available, and no faster than its disk and network."
+          }
+        }
+      },
       "control.Activity": {
         "type": "object",
         "description": "Activity is what a standby's router reports back: when it last sent each replica a request, and the on-demand models it was asked for. The leader evicts and loads by these, and without them a replica busy behind a standby looks idle to the one machine that decides.",
@@ -11737,6 +12017,10 @@
           "context": {
             "type": "integer"
           },
+          "trained": {
+            "type": "integer",
+            "description": "Trained is the context the model was trained for, the most a request may use; 0 when neither its header nor its config says."
+          },
           "kv_bytes_per_token": {
             "type": "integer",
             "format": "int64",
@@ -12166,6 +12450,160 @@
           }
         }
       },
+      "slurmsvc.NodeStatus": {
+        "type": "object",
+        "description": "NodeStatus is one node as the service sees it.",
+        "properties": {
+          "node": {
+            "type": "string"
+          },
+          "phase": {
+            "type": "string"
+          },
+          "replicas": {
+            "type": "integer"
+          },
+          "stops_by": {
+            "type": "string",
+            "format": "date-time",
+            "description": "StopsBy is, while it is being lent, when a replica still on it is stopped even with nowhere else to go: the drain's grace after the lend began. Zero when not known (a leader that took over meanwhile)."
+          },
+          "slurm": {
+            "$ref": "#/components/schemas/slurmsvc.Reported"
+          },
+          "connected": {
+            "type": "boolean"
+          }
+        }
+      },
+      "slurmsvc.Reported": {
+        "type": "object",
+        "description": "Reported is the agent's SlurmStatus, for the admin API.",
+        "properties": {
+          "version": {
+            "type": "string"
+          },
+          "hardware": {
+            "type": "string"
+          },
+          "slurmd_running": {
+            "type": "boolean"
+          },
+          "slurmctld_running": {
+            "type": "boolean"
+          },
+          "slurmdbd_running": {
+            "type": "boolean"
+          },
+          "shared": {
+            "type": "string",
+            "description": "Shared is the shared path here: \"mounted\", \"exported\", \"missing\"."
+          },
+          "containers": {
+            "type": "string",
+            "description": "Containers is what the machine has for jobs in containers, empty while not both of pyxis and enroot."
+          },
+          "users": {
+            "type": "integer",
+            "description": "Users is how many of llmgw's users the machine has."
+          },
+          "state": {
+            "type": "string"
+          },
+          "reason": {
+            "type": "string"
+          },
+          "jobs": {
+            "type": "integer"
+          },
+          "error": {
+            "type": "string"
+          },
+          "files_sha256": {
+            "type": "string"
+          },
+          "at": {
+            "type": "string",
+            "format": "date-time"
+          }
+        }
+      },
+      "slurmsvc.Status": {
+        "type": "object",
+        "description": "Status is the whole Slurm cluster, for `ctl slurm status`.",
+        "properties": {
+          "enabled": {
+            "type": "boolean"
+          },
+          "cluster": {
+            "type": "string"
+          },
+          "controller": {
+            "type": "string"
+          },
+          "partition": {
+            "type": "string"
+          },
+          "controller_status": {
+            "$ref": "#/components/schemas/slurmsvc.Reported"
+          },
+          "nodes": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/slurmsvc.NodeStatus"
+            }
+          },
+          "waiting": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            },
+            "description": "Waiting are nodes left out of slurm.conf until they report their hardware."
+          },
+          "shared": {
+            "$ref": "#/components/schemas/config.SharedStorage",
+            "description": "Shared is where the shared files are and come from; Users who keeps the users, and how many llmgw has."
+          },
+          "users": {
+            "type": "string"
+          },
+          "user_count": {
+            "type": "integer"
+          },
+          "containers": {
+            "type": "string",
+            "description": "Containers is whether jobs may run in containers: \"on\" once every machine has pyxis and enroot, \"installing\" until then, empty when slurm.containers is off."
+          },
+          "accounts": {
+            "type": "integer",
+            "description": "Accounts is how many of Slurm's accounts are llmgw's projects'."
+          }
+        }
+      },
+      "slurmsvc.User": {
+        "type": "object",
+        "description": "User is a Linux user made from an llmgw account.",
+        "properties": {
+          "login": {
+            "type": "string"
+          },
+          "uid": {
+            "type": "integer"
+          },
+          "name": {
+            "type": "string"
+          },
+          "email": {
+            "type": "string"
+          },
+          "ssh_keys": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          }
+        }
+      },
       "store.AuditEvent": {
         "type": "object",
         "description": "AuditEvent is one change: who made it, through what, to what, and how it ended. Detail never holds a secret; the writer takes them out.",

docs/console.md

@@ -67,7 +67,8 @@ or with the mirror unticked, is said in the form and nothing starts.
 ## Watch
 
 **Overview** is `llmgw ctl status`: nodes, GPUs by class, fabrics, replicas,
-models, the store's members, and anything parked, unplaced or idle.
+models, the store's members, and anything parked, unplaced or idle. With no
+webhook in the cluster's settings it says the alerts go nowhere.
 
 **Monitor** draws the cluster the way btop draws a machine: CPU, memory,
 GPU busy and GPU memory, requests and tokens a second, time to first token,
@@ -207,6 +208,19 @@ starting — and whether it ended done, failed, or waiting for room on busy
 cards. The entry shows how many are running. Enter on one shows its steps.
 `llmgw ctl jobs` is the same, and `llmgw ctl jobs ID` follows one.
 
+**Slurm** is the Slurm cluster run beside llmgw, when there is one: its
+controller and whether slurmctld runs, and each node that runs slurmd —
+whose it is now (llmgw's, Slurm's, or on its way between them), its state in
+Slurm, the jobs running there. On a row, **Lend to Slurm…** moves the node's
+replicas off and then lets Slurm run jobs there; **Take back from Slurm…**
+starts no new job there and gives it back to llmgw when its jobs have ended.
+Both change `slurm.lent` in the cluster's settings, as `llmgw ctl slurm lend`
+and `reclaim` do. Turning Slurm on and choosing its nodes is `llmgw ctl slurm
+setup`, or the Slurm part of the first install. Above the nodes it says where
+the shared files come from and how many users llmgw made; the SHARED column
+is each node's (mounted, exported, missing). `llmgw ctl slurm shared` and
+`llmgw ctl slurm users` change them.
+
 **Logs** shows a node's agent log, newest at the bottom, errors and warnings
 coloured.
 
@@ -220,7 +234,8 @@ and change it with the same commands as `llmgw users`, `llmgw keys` and
 dashboard, and what they belong to. **A** creates an account, already
 approved. On a row: approve one waiting, suspend or reinstate, give or take
 the cluster dashboard, add to or take out of an organization, team or
-project, their limits, a new password (their sessions close), delete, and
+project (on an organization as a member, its manager, who sees its page, or
+its admin, who also changes it), their limits, a new password (their sessions close), delete, and
 **Forget them…** for a request to be forgotten (`llmgw users forget`).
 
 **API keys** — every key, its project, owner, state and last use. **A**

docs/install.md

@@ -166,13 +166,85 @@ is a flag too, and the plan shows the command with them: `--starter qwen3-8b`
 `--judge-fail open`, `--judge-categories`, `--judge-replicas`,
 `--judge-budget`, `--language tr`, `--signup-open`, `--default-rpm`,
 `--default-tpm`, `--max-key-ttl`, `--media` (pictures and 3D: ComfyUI on the
-nodes, and `presets.auto`), `--evict-after-idle` and `--min-lifetime`. They
+nodes, and `presets.auto`), `--slurm` (see below), `--evict-after-idle` and `--min-lifetime`. They
 go into `server.yaml` as init writes it, and become the cluster's first
 settings when it starts; a `server.yaml` already there is never overwritten,
 so run again they change nothing — the settings page does. An NVIDIA card with no working
 driver is reported rather than silently ignored; install the driver and run
 init again to serve on it.
 
+**Slurm, for training and batch jobs.** `--slurm`, or the Slurm part on the
+choices screen, runs a Slurm cluster beside llmgw: slurmctld on this
+machine's node, slurmd on every node, those that join later too
+(`slurm.all_nodes`). The agents install munge and Slurm (22.05 or newer)
+from the machines' own package sources, and on this machine slurmdbd, which
+keeps the jobs' records (`sacct`), in a MariaDB installed beside it that
+answers this machine alone. Slurm keeps its records in MySQL or MariaDB
+only, not in the cluster's PostgreSQL; the database's password is made from
+the cluster's munge key, so nothing new is stored. A MariaDB the machine
+already had keeps its settings and only gets the database `slurm_acct_db`
+and its user. Back it up with the machine (`mariadb-dump slurm_acct_db`);
+records older than a year are purged. No node is Slurm's at first: each is
+drained in Slurm and serves models until `llmgw ctl slurm lend NODE` gives it
+to Slurm — its models move off first — and `llmgw ctl slurm reclaim NODE`
+takes it back once its jobs end. Jobs are submitted on this machine with
+`sbatch` and `srun`; `llmgw ctl slurm status` shows where each node is. It
+needs this machine to be a node, and its hostname to be a name Slurm takes.
+On a cluster installed without it, `llmgw ctl slurm setup --controller NODE
+--all-nodes` does the same later. Slurm's ports (6817–6819, and srun's 60001–63000) answer the cluster's
+networks only, as llmgw's own do; a firewall of the site's own between the
+nodes must let them through.
+
+**Slurm's shared files and users.** Jobs need the same files on every
+machine: homes, datasets, output, checkpoints. With `--slurm` this machine
+exports `/shared` over NFS to every Slurm machine and they mount it
+(`slurm.shared`); NFS's port 2049 answers the cluster's networks only, the
+export the Slurm machines alone. That is one machine, not highly available
+and no faster than its disk and network: enough for a few nodes. A site with
+storage of its own uses it instead — `--slurm-shared HOST:/EXPORT` at
+install, or `llmgw ctl slurm shared --path /shared --nfs HOST:/EXPORT`
+later; Lustre, BeeGFS or GPFS the site mounts itself, and `llmgw ctl slurm
+shared --path /lustre/hpc` only has llmgw look that it is there. Every
+active llmgw account is a Linux user on every Slurm machine
+(`slurm.users: llmgw`): its login from its e-mail address, a UID from
+200000 up that is never given to anyone else, its home under
+`/shared/home`. People add their SSH public keys in the portal's
+**Profile** and log in to this machine to run `sbatch`; an account suspended
+or deleted loses its user, its files stay. On the other Slurm machines they
+are let in only while a job of theirs runs there, and their ssh session is
+put in that job, held to its cores and cards (pam_slurm_adopt, which the
+agents install); root and the machines' own users log in as before. A site whose users come from its
+directory (sssd against Active Directory or LDAP) passes `--slurm-users
+site`, or later `llmgw ctl slurm users off`.
+
+With llmgw's users, Slurm's accounts are llmgw's projects: an account for
+each organization, team and project, nested (`acme`, `acme-ml`,
+`acme-ml-training`), and on each project's account its members and its
+team's and organization's. A job runs on one of its user's accounts and no
+other (`sbatch --account=acme-ml-training`; without it, the user's first),
+and when it ends its cards' hours (a job without cards, its cores') are
+priced at `auth.gpu_hour_cost` (`core_hour_cost`) into the project's usage,
+where it shows as model `slurm/PARTITION` beside the project's requests.
+Accounts a Slurm administrator made are left alone. A job's cost does not
+yet count against a budget, which binds API keys only.
+
+Jobs may run in containers: `llmgw ctl slurm containers on` has the agents
+install enroot and pyxis on every Slurm machine (NVIDIA's releases, at
+versions llmgw pins and checks; pyxis is built against each machine's own
+Slurm) and, on GPU machines, nvidia-container-cli from NVIDIA's repository.
+Once every machine has them Slurm loads pyxis, and `ctl slurm status` says
+`containers: on`:
+
+```sh
+srun --gres=gpu:1 --container-image=nvcr.io#nvidia/pytorch:25.01-py3 python train.py
+```
+
+The shared path is in every container where it is on the host, and the job
+runs as its user, held to its cores and cards. Each node pulls the image
+from its registry for each job; for a large image used often, import it
+once (`enroot import -o /shared/images/pytorch.sqsh docker://nvcr.io#...`)
+and give that file to `--container-image`.
+
 **Secure by default.** What init writes is the hardened configuration, not a
 demonstration one:
 
@@ -396,6 +468,19 @@ by itself. Three things are worth knowing before you edit it:
   cache (the answer's `cached_tokens`) are charged at it, the rest at the
   prompt rate. A provider that bills cached reads at a tenth is
   `{prompt_per_1k: 2500, cached_per_1k: 250, completion_per_1k: 10000}`.
+- What to price a self-hosted model at follows from what the cluster costs.
+  Set `auth.gpu_hour_cost`, what an hour of one GPU costs in the units budgets
+  are in: the hardware written off over its years, its power and room, and the
+  people who run it, over the GPUs and the 730 hours of a month (an 8-GPU
+  server at 250,000 over four years, 10 kW, and some upkeep is about 1.35);
+  `auth.core_hour_cost` likewise for models on CPUs. The leader keeps, by the
+  hour, the GPU- and core-hours each model's replicas hold, and `llmgw ctl
+  prices suggest` spreads them over the tokens the model served: an output
+  token counted as four input ones (making one is that much dearer), a cached
+  prompt token as a tenth of one. Idle hours are in the cost, so a model held
+  all month and seldom used comes out dear, which is the point; one held and
+  billed nothing (a guard's judge) is shown with what it held. `--apply`
+  writes the prices into `auth.pricing`.
 - `control.enabled: true` is what makes this a control plane while its catalog
   is empty. Without it, a file with no models is refused as having nothing to
   serve.
@@ -3145,7 +3230,7 @@ loaded at boot by `llmgw-firewall.service`):
 | Machine | Ports, for the cluster only |
 | --- | --- |
 | control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
-| node | engine ports (`control.port_range`, 20000–20999), 9100 |
+| node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6819 (slurmctld, slurmd, slurmdbd) and 60001–63000 (srun's `SrunPortRange`); exporting Slurm's shared files, 2049 (NFS) |
 | store member | 5432, 8008, 2379, 2380, 9100 |
 | mirror | 8081, 8082, 9100 |
 

docs/onboarding.md

@@ -192,22 +192,41 @@ auth:
 A personal key has no project, so it keeps no conversations or agent notes;
 those are kept per project.
 
-### Organization managers
+### Organization managers and admins
 
-A member of an organization can be made its manager:
+A member of an organization can be made its manager, who sees it, or its
+admin, who also changes it:
 
 ```
 llmgw users join boss@acme.com --org Acme --role manager
+llmgw users join lead@acme.com --org Acme --role admin
 ```
 
-or with the star on their organization membership on `/admin/users`. A manager
-gets an **Organization** link and a page at `/org`: usage over time by team,
+or with the role chosen on their organization membership on `/admin/users`.
+Both get an **Organization** link and a page at `/org`: usage over time by team,
 project or person, the teams and projects with their limits and spend, and every
 member. Only what was spent in that organization's projects is on it; what its
-people do elsewhere is not. The page is read-only — a manager is not an
-administrator of the cluster, and limits and memberships stay with those who
-are. An administrator following `/org` lands on the same organization's panel
-on the people page instead, where all of that can be changed.
+people do elsewhere is not.
+
+A manager's page is read-only. An organization's admin also changes the
+organization from it:
+
+- adds and renames its teams and projects, and removes an empty one;
+- adds a person by their sign-in address to the organization, a team or a
+  project, and takes them out (their keys there are revoked with it);
+- makes a member of the organization a manager or an admin, or neither;
+- revokes a key spent in its projects.
+
+What reaches past the organization stays with the cluster's administrators:
+its limits and budgets, approving or suspending an account (someone not yet
+approved cannot be added), the organization's own name, and anything else on
+the cluster. An admin cannot take away their own role. A group from single
+sign-on or Active Directory that puts someone in the organization does not, by
+saying so again at the next sign-in, take a manager's or admin's role away.
+
+An administrator of the cluster following `/org` lands on the same
+organization's panel on the people page instead, where all of that can be
+changed.
 
 `llmgw keys init` still works for provisioning scripts and is idempotent, but it
 will no longer move a team or project that already exists under another parent;
@@ -517,8 +536,10 @@ included: `llmgw ctl audit`, or **Audit** in the console.
   suspends accounts from it; see [install.md](install.md#single-sign-on).
   Without them accounts are local, with local passwords.
 - **Staff roles are coarse.** `viewer`, `operator`, `billing` and `admin`
-  (`llmgw users role`) decide what someone may see and do under /admin; an
-  organization's manager can view it but not change it. Every change is in
-  the audit log (`llmgw ctl audit`).
+  (`llmgw users role`) decide what someone may see and do under /admin, and a
+  page draws only the forms that person may send: a viewer sees no forms. An
+  organization's manager can view it; its admin also changes its teams,
+  projects, people and keys. Every change is in the audit log (`llmgw ctl
+  audit`).
 - **Rate limits are per gateway instance.** With two instances each counts its own,
   so the effective limit doubles. v1 is single-instance by design.

docs/reference.md

@@ -43,6 +43,8 @@ rather than repeated.
 | `personal` | `Personal` | [Auth](#auth) |
 | `pricing` | `map[string]Price` | [Auth](#auth) |
 | `default_price` | `Price` | [Auth](#auth) |
+| `gpu_hour_cost` | `float64` | [Auth](#auth) |
+| `core_hour_cost` | `float64` | [Auth](#auth) |
 | `flush_interval` | `time.Duration` | [Auth](#auth) |
 | `batch_size` | `int` | [Auth](#auth) |
 | `buffer_size` | `int` | [Auth](#auth) |
@@ -376,6 +378,7 @@ rather than repeated.
 | `mail` | `mail.Config` | [Server](#server) |
 | `serving` | `ServingSettings` | [Server](#server) |
 | `firewall` | `FirewallSettings` | [Server](#server) |
+| `slurm` | `SlurmSettings` | [Server](#server) |
 | `presets` | `PresetSettings` | [Server](#server) |
 | `on_demand` | `OnDemandSettings` | [Server](#server) |
 | `agents` | `AgentSettings` | [Server](#server) |
@@ -400,7 +403,20 @@ rather than repeated.
 | `action` | `string` | [Service](#service) |
 | `timeout` | `time.Duration` | [Service](#service) |
 | `stopped` | `bool` | [ServingSettings](#servingsettings) |
+| `path` | `string` | [SharedStorage](#sharedstorage) |
+| `nfs` | `string` | [SharedStorage](#sharedstorage) |
+| `server` | `string` | [SharedStorage](#sharedstorage) |
 | `bin` | `string` | [SimConfig](#simconfig) |
+| `enabled` | `bool` | [SlurmSettings](#slurmsettings) |
+| `cluster_name` | `string` | [SlurmSettings](#slurmsettings) |
+| `controller` | `string` | [SlurmSettings](#slurmsettings) |
+| `nodes` | `[]string` | [SlurmSettings](#slurmsettings) |
+| `all_nodes` | `bool` | [SlurmSettings](#slurmsettings) |
+| `lent` | `[]string` | [SlurmSettings](#slurmsettings) |
+| `partition` | `string` | [SlurmSettings](#slurmsettings) |
+| `shared` | `SharedStorage` | [SlurmSettings](#slurmsettings) |
+| `users` | `string` | [SlurmSettings](#slurmsettings) |
+| `containers` | `bool` | [SlurmSettings](#slurmsettings) |
 | `dir` | `string` | [StrataConfig](#strataconfig) |
 | `python` | `string` | [StrataConfig](#strataconfig) |
 | `engine` | `string` | [StrataConfig](#strataconfig) |
@@ -708,6 +724,21 @@ table costs whatever DefaultPrice says, which is nothing unless set.
 
 Takes a [config.Price](#configprice) block.
 
+### `gpu_hour_cost`
+
+`float64`
+
+GPUHourCost is what an hour of one GPU costs the operator, in the
+units budgets are in: the hardware written off over its years, its
+power and room, and the people who run it, over the GPUs and the
+hours of a month. With CoreHourCost for a CPU core, it is what
+`llmgw ctl prices suggest` prices each model from: the hours its
+replicas held, idle ones too, over the tokens it served.
+
+### `core_hour_cost`
+
+`float64`
+
 ### `flush_interval`
 
 `time.Duration`
@@ -3279,6 +3310,15 @@ Firewall turns llmgw's firewall on every machine on or off.
 
 Takes a [FirewallSettings](#firewallsettings) block.
 
+### `slurm`
+
+`SlurmSettings`
+
+Slurm is a Slurm cluster run beside llmgw on some of its nodes, and
+which of them it may use now.
+
+Takes a [SlurmSettings](#slurmsettings) block.
+
 ### `presets`
 
 `PresetSettings`
@@ -3471,6 +3511,30 @@ running and every model's definition as it is: llmgw stop sets it,
 llmgw start clears it, and the models come back as their catalog
 entries ask. In force on every control plane within seconds.
 
+## SharedStorage
+
+### `path`
+
+`string`
+
+Path is where every machine sees it, as /shared.
+
+### `nfs`
+
+`string`
+
+NFS is the site's export, host:/path, mounted at Path on every
+machine of the Slurm cluster.
+
+### `server`
+
+`string`
+
+Server is the llmgw node, the controller or one of the nodes, that
+exports Path over NFS to the others: for a site with no storage of
+its own. One machine: not highly available, and no faster than its
+disk and network.
+
 ## SimConfig
 
 ### `bin`
@@ -3480,6 +3544,88 @@ entries ask. In force on every control plane within seconds.
 Bin is the llmgw binary itself; the engine is one of its subcommands.
 Empty means whatever started this agent, which is right almost always.
 
+## SlurmSettings
+
+### `enabled`
+
+`bool`
+
+Enabled runs Slurm on the nodes below. False stops llmgw keeping it;
+what was installed is left installed, and every node is llmgw's.
+
+### `cluster_name`
+
+`string`
+
+ClusterName is Slurm's ClusterName. Empty is "llmgw".
+
+### `controller`
+
+`string`
+
+Controller is the llmgw node whose agent runs slurmctld. It needs no
+GPU, and need not be one of Nodes.
+
+### `nodes`
+
+`[]string`
+
+Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained
+in Slurm, until it is also in Lent.
+
+### `all_nodes`
+
+`bool`
+
+AllNodes runs slurmd on every llmgw node, those that join later too,
+besides any named in Nodes. Each is llmgw's until it is lent, as any.
+
+### `lent`
+
+`[]string`
+
+Lent are the nodes of Nodes Slurm may run jobs on now. A node put here
+is drained of llmgw's replicas first, as `ctl drain` does, and resumed
+in Slurm once none is left; a node taken out is drained in Slurm and
+given back to llmgw when its last job ends.
+
+### `partition`
+
+`string`
+
+Partition is the one partition every node is in. Empty is "gpu".
+
+### `shared`
+
+`SharedStorage`
+
+Shared is the file system every machine of the Slurm cluster sees at
+the same path: home directories, datasets, the jobs' output and
+checkpoints. Empty path: none, and a batch job's output stays on the
+node it ran on.
+
+Takes a [SharedStorage](#sharedstorage) block.
+
+### `users`
+
+`string`
+
+Users is who keeps the Linux users that run jobs, the same on every
+machine of the Slurm cluster: "llmgw" makes one for each active llmgw
+account (its login from the e-mail address, a UID of llmgw's from
+200000 up, its SSH keys from the portal), let in to the controller,
+and to a node only while a job of theirs runs there (pam_slurm_adopt);
+empty leaves users to the site (a directory through sssd, or by hand).
+
+### `containers`
+
+`bool`
+
+Containers lets jobs run in containers: `srun --container-image
+nvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),
+the GPUs and the shared path inside. The agents install both on
+every Slurm machine; jobs take it once every machine has them.
+
 ## StrataConfig
 
 ### `dir`

internal/accounting/suggest.go

@@ -0,0 +1,91 @@
+package accounting
+
+import (
+	"math"
+	"slices"
+)
+
+// Prices from what the cluster costs: what each model held of it (its
+// replicas' GPU- and core-hours) at what an hour of a GPU or a core costs
+// the operator, spread over the tokens it served. Idle time is in it: a
+// model held all month and used an hour costs what it held, not the hour.
+//
+// An output token is priced OutputPerInput input tokens, as producing one
+// is that much dearer (one step a token, against a prompt read in one
+// pass); a cached prompt token a tenth of an input one.
+const (
+	OutputPerInput = 4
+	CachedPerInput = 0.1
+)
+
+// Basis is what an hour costs, in the units budgets are in.
+type Basis struct {
+	GPUHour  float64
+	CoreHour float64
+}
+
+// Held is what a model held, and Served what it answered with.
+type Held struct {
+	GPUSeconds, CoreSeconds int64
+}
+
+// Served is a model's tokens.
+type Served struct {
+	Prompt, Completion int64
+}
+
+// Suggestion is one model's price from its cost.
+type Suggestion struct {
+	Model     string  `json:"model"`
+	GPUHours  float64 `json:"gpu_hours"`
+	CoreHours float64 `json:"core_hours"`
+	// Cost is what it held cost, in budget units.
+	Cost       float64 `json:"cost"`
+	Prompt     int64   `json:"prompt_tokens"`
+	Completion int64   `json:"completion_tokens"`
+	// Price is the suggestion; zero with a Note saying why there is none.
+	Price Price  `json:"price"`
+	Note  string `json:"note,omitempty"`
+}
+
+// Suggest prices every model that held something or served something.
+func Suggest(held map[string]Held, served map[string]Served, b Basis) []Suggestion {
+	var names []string
+	for m := range held {
+		names = append(names, m)
+	}
+	for m := range served {
+		if _, ok := held[m]; !ok {
+			names = append(names, m)
+		}
+	}
+	slices.Sort(names)
+	out := make([]Suggestion, 0, len(names))
+	for _, m := range names {
+		h, s := held[m], served[m]
+		g := Suggestion{Model: m, GPUHours: float64(h.GPUSeconds) / 3600, CoreHours: float64(h.CoreSeconds) / 3600,
+			Prompt: s.Prompt, Completion: s.Completion}
+		g.Cost = g.GPUHours*b.GPUHour + g.CoreHours*b.CoreHour
+		// Tokens counted as output ones: an input token is a fraction of one.
+		units := float64(s.Completion) + float64(s.Prompt)/OutputPerInput
+		switch {
+		case g.Cost == 0 && units > 0:
+			g.Note = "held nothing recorded here: an outside provider's, or served before the hours were kept"
+		case g.Cost > 0 && units == 0:
+			g.Note = "held the cluster and was billed no tokens (a guard's judge, or a model priced by the picture or 3D model): no one pays what it held"
+		case g.Cost > 0:
+			// Micro-units per thousand output tokens, rounded up so a
+			// cheap model is never free.
+			out1k := int64(math.Ceil(g.Cost / units * 1000 * 1e6))
+			g.Price = Price{
+				CompletionPer1k: out1k,
+				PromptPer1k:     ceilDiv(out1k, OutputPerInput),
+				CachedPer1k:     max(int64(math.Ceil(float64(ceilDiv(out1k, OutputPerInput))*CachedPerInput)), 1),
+			}
+		}
+		out = append(out, g)
+	}
+	return out
+}
+
+func ceilDiv(a, b int64) int64 { return (a + b - 1) / b }

internal/accounting/suggest_test.go

@@ -0,0 +1,45 @@
+package accounting
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestSuggestSpreadsWhatAModelHeldOverItsTokens(t *testing.T) {
+	// The worked example: two cards for 1000 hours at 1.35 an hour is 2700;
+	// 3.0 billion output tokens and 4.0 billion input ones, counted as 1.0
+	// billion output ones, are 4.0 billion: 0.675 a million output tokens.
+	held := map[string]Held{
+		"llama-70b": {GPUSeconds: 2 * 1000 * 3600},
+		"idle":      {GPUSeconds: 3600},
+		"small-cpu": {CoreSeconds: 8 * 100 * 3600},
+	}
+	served := map[string]Served{
+		"llama-70b": {Prompt: 4_000_000_000, Completion: 3_000_000_000},
+		"small-cpu": {Prompt: 1_000_000, Completion: 1_000_000},
+		"outside":   {Prompt: 10, Completion: 10},
+	}
+	got := Suggest(held, served, Basis{GPUHour: 1.35, CoreHour: 0.01})
+	by := map[string]Suggestion{}
+	for _, s := range got {
+		by[s.Model] = s
+	}
+	l := by["llama-70b"]
+	if l.Cost != 2700 || l.Price.CompletionPer1k != 675 || l.Price.PromptPer1k != 169 || l.Price.CachedPer1k != 17 {
+		t.Errorf("llama: %+v", l)
+	}
+	// What it costs back at that price is what it held, within the rounding.
+	back := float64(l.Price.Cost(l.Prompt, 0, l.Completion, 0, 0)) / 1e6
+	if back < l.Cost || back > l.Cost*1.01 {
+		t.Errorf("charged %.2f for a cost of %.2f", back, l.Cost)
+	}
+	if s := by["small-cpu"]; s.Cost != 8 || s.Price.CompletionPer1k == 0 {
+		t.Errorf("cpu: %+v", s)
+	}
+	if s := by["idle"]; !s.Price.Zero() || !strings.Contains(s.Note, "billed no tokens") {
+		t.Errorf("idle: %+v", s)
+	}
+	if s := by["outside"]; !s.Price.Zero() || !strings.Contains(s.Note, "held nothing") {
+		t.Errorf("outside: %+v", s)
+	}
+}

internal/agent/agent.go

@@ -35,6 +35,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/slurmnode"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
@@ -97,7 +98,10 @@ type Agent struct {
 	// (nil: this machine's shell).
 	fwMu     sync.Mutex
 	fwRunner firewall.Runner
-	inv      discovery.Inventory
+	// fwLast is the last firewall message, applied again when the node
+	// is given a part in Slurm. Under fwMu.
+	fwLast *llmgwv1.Firewall
+	inv    discovery.Inventory
 	// shardReplica remembers which replica a shard belongs to, so a status
 	// report can name both. The control plane keys on the replica.
 	shardReplica map[string]string
@@ -148,6 +152,10 @@ type Agent struct {
 
 	// trainer runs LoRA training jobs, one at a time (BACKLOG F4).
 	trainer *train.Runner
+	// slurm keeps the node's part in Slurm (BACKLOG P); slurmOut is where
+	// its status goes, under mu.
+	slurm    *slurmnode.Keeper
+	slurmOut chan<- *llmgwv1.AgentMessage
 }
 
 // connectHTTPClient is the HTTP client connect uses; an interface so tests can
@@ -206,6 +214,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
 	}
 	a.trainer = train.New(train.Env{EnginesDir: enginesDir, WorkDir: filepath.Join(cfg.Superv.WorkDir, "trainer"),
 		RunAs: runAs, Log: log})
+	a.slurm = slurmnode.New(slurmnode.Local{}, a.prov, slurmMarker, log)
 	return a, nil
 }
 
@@ -232,6 +241,7 @@ func (a *Agent) Run(ctx context.Context) error {
 	}
 	go a.provisionLoop(ctx)
 	go a.ensureDiags(ctx)
+	go a.slurm.Run(ctx, a.sendSlurm)
 
 	attempt := 0
 	next := 0
@@ -530,6 +540,18 @@ func (a *Agent) scan(ctx context.Context) error {
 	return err
 }
 
+// forgetSent is a new stream: a control plane that knows nothing of where
+// installs stand, nor which files are here beyond what registration says —
+// and registration says what the startup scan found. So the next heartbeat
+// tells it both again. Without it, files deleted after the agent started came
+// back on a control plane that restarted, and stayed until the next change.
+func (a *Agent) forgetSent() {
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	a.provSent = ""
+	a.lastWeights = ""
+}
+
 // session runs one connection until it breaks.
 func (a *Agent) session(ctx context.Context, server string) error {
 	client := llmgwv1connect.NewAgentServiceClient(a.http, server)
@@ -566,14 +588,12 @@ func (a *Agent) session(ctx context.Context, server string) error {
 	reg.EngineVersions = a.engines(parent)
 	reg.EnginesProbed = true
 	reg.InstallableEngines = slices.DeleteFunc(provision.Installable(), a.disabled)
-	reg.Features = []string{FeatureFabricTest}
+	reg.Features = []string{FeatureFabricTest, FeatureSlurm}
 	reg.RunningShards = a.runningShards()
 	a.mu.Lock()
 	h := a.prov.Host
-	// A new stream is a control plane that knows nothing of where installs
-	// stand, so the next heartbeat tells it again.
-	a.provSent = ""
 	a.mu.Unlock()
+	a.forgetSent()
 	reg.Host = &llmgwv1.Host{
 		OsId: h.OSID, OsVersion: h.OSVersion, OsCodename: h.OSCodename,
 		Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver,
@@ -618,6 +638,8 @@ func (a *Agent) session(ctx context.Context, server string) error {
 	go a.watchExits(ctx, out, unready(reg.RunningShards))
 	go a.forwardKV(ctx, out)
 	go a.reportCompute(ctx, out)
+	a.setSlurmOut(out)
+	defer a.setSlurmOut(nil)
 
 	for {
 		msg, err := stream.Receive()
@@ -884,7 +906,23 @@ func (a *Agent) removeWeights(paths []string) {
 			a.log.Warn("refused to remove a file llmgw did not fetch", "path", p)
 			continue
 		}
-		if err := os.Remove(abs); err != nil && !os.IsNotExist(err) {
+		// A set — every file of one version, fetched into a directory of its
+		// own — goes whole: os.Remove refused it as not empty, and nothing
+		// was freed of what a model of several files left. Lstat, so a link
+		// is taken away and never what it points at.
+		info, err := os.Lstat(abs)
+		switch {
+		case os.IsNotExist(err):
+			continue
+		case err != nil:
+			a.log.Warn("could not look at cached weights", "path", p, "err", err)
+			continue
+		case info.IsDir():
+			err = os.RemoveAll(abs)
+		default:
+			err = os.Remove(abs)
+		}
+		if err != nil {
 			a.log.Warn("could not remove cached weights", "path", p, "err", err)
 			continue
 		}
@@ -925,6 +963,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 		go a.applySettings(m.AgentSettings)
 	case *llmgwv1.ControlMessage_TrainStart:
 		go a.trainJob(ctx, m.TrainStart, out)
+	case *llmgwv1.ControlMessage_Slurm:
+		a.slurmMessage(m.Slurm)
 	case *llmgwv1.ControlMessage_TrainCancel:
 		a.trainer.Cancel(m.TrainCancel.GetJobId())
 	case *llmgwv1.ControlMessage_RestartAgent:

internal/agent/agent_test.go

@@ -925,3 +925,23 @@ func TestABandwidthTestIsRunOnTheAdapterAskedFor(t *testing.T) {
 		t.Errorf("reply %+v, ran %+v", r, got)
 	}
 }
+
+func TestANewStreamIsToldTheFilesAgain(t *testing.T) {
+	// Registration carries what the startup scan found; a file deleted since
+	// came back on a control plane that restarted, because the next
+	// heartbeat saw the list unchanged since it was last sent.
+	dir := t.TempDir()
+	h := start(t, func(c *Config) { c.ModelDirs = []string{dir} })
+	if err := os.WriteFile(filepath.Join(dir, "0123456789abcdef-m.gguf"), []byte("x"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	h.ag.heartbeat()
+	if hb := h.ag.heartbeat().GetHeartbeat(); hb.GetWeightsChanged() {
+		t.Fatal("an unchanged list should not be sent again on the same stream")
+	}
+	h.ag.forgetSent()
+	hb := h.ag.heartbeat().GetHeartbeat()
+	if !hb.GetWeightsChanged() || len(hb.GetWeights()) != 1 {
+		t.Errorf("a new stream should be told the files: changed=%v %v", hb.GetWeightsChanged(), hb.GetWeights())
+	}
+}

internal/agent/firewall.go

@@ -10,8 +10,20 @@ import (
 
 	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
 )
 
+// slurmPorts are Slurm's on a node that runs any of it: slurmctld's and
+// slurmd's, and srun's range for the steps it starts.
+var slurmPorts = []firewall.Port{
+	{From: slurm.CtldPort, To: slurm.DbdPort, What: "slurm: slurmctld, slurmd and slurmdbd"},
+	{From: slurm.SrunPortsFrom, To: slurm.SrunPortsTo, What: "slurm: srun, for the steps it starts"},
+}
+
+// nfsPort is NFSv4's, on the machine that exports the Slurm cluster's
+// shared files.
+var nfsPort = firewall.Port{From: slurm.NFSPort, To: slurm.NFSPort, What: "nfs: the Slurm cluster's shared files"}
+
 // applyFirewall keeps llmgw's firewall on this node as the cluster's
 // settings say: the engines' ports and the node exporter's for the networks
 // the node reaches its control planes on, and those named, or no table at
@@ -19,6 +31,7 @@ import (
 func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
 	a.fwMu.Lock()
 	defer a.fwMu.Unlock()
+	a.fwLast = m
 	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
 	defer cancel()
 	r := a.fwRunner
@@ -55,6 +68,14 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
 	if from > 0 && to >= from {
 		ports = append(ports, firewall.Engines(from, to))
 	}
+	// Once there, they stay: the ruleset keeps every port it held, so a
+	// node that leaves Slurm keeps them closed to everyone but the cluster.
+	if a.slurm != nil && a.slurm.HasPart() {
+		ports = append(ports, slurmPorts...)
+	}
+	if a.slurm != nil && a.slurm.Serves() {
+		ports = append(ports, nfsPort)
+	}
 	changed, err := firewall.Install(ctx, r, firewall.Spec{Ports: ports, Allow: nets})
 	switch {
 	case err != nil:
@@ -64,6 +85,23 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
 	}
 }
 
+// slurmMessage is the leader's word on this node's part in Slurm. A node
+// that gets one, or starts or stops exporting the shared files, has the
+// firewall applied again, with Slurm's ports and NFS's.
+func (a *Agent) slurmMessage(m *llmgwv1.Slurm) {
+	had, served := a.slurm.HasPart(), a.slurm.Serves()
+	a.slurm.Apply(m)
+	if (had || !a.slurm.HasPart()) && served == a.slurm.Serves() {
+		return
+	}
+	a.fwMu.Lock()
+	last := a.fwLast
+	a.fwMu.Unlock()
+	if last != nil && last.GetEnabled() {
+		go a.applyFirewall(last)
+	}
+}
+
 // controlPlaneIPs are the addresses of the control planes an agent dials.
 func controlPlaneIPs(ctx context.Context, servers []string) []string {
 	var out []string

internal/agent/firewall_test.go

@@ -6,8 +6,10 @@ import (
 	"log/slog"
 	"strings"
 	"testing"
+	"time"
 
 	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/slurmnode"
 )
 
 // fakeShell answers the firewall package's scripts and records what was
@@ -55,3 +57,28 @@ func TestANodeKeepsOrDropsItsFirewallAsTheClusterSays(t *testing.T) {
 		t.Error("turned off, the table stayed")
 	}
 }
+
+func TestANodeGivenAPartInSlurmLetsSlurmInFromTheClusterOnly(t *testing.T) {
+	sh := &fakeShell{}
+	a := &Agent{log: slog.New(slog.DiscardHandler), fwRunner: sh, serverList: []string{"https://10.0.0.140:7443"},
+		slurm: slurmnode.New(nil, nil, "", slog.New(slog.DiscardHandler))}
+	a.applyFirewall(&llmgwv1.Firewall{Enabled: true, EnginePortsFrom: 20000, EnginePortsTo: 20999})
+	if strings.Contains(sh.installed, "6817") {
+		t.Fatalf("Slurm's ports before the node has a part in it:\n%s", sh.installed)
+	}
+
+	a.slurmMessage(&llmgwv1.Slurm{Name: "node05", Node: true, ConfServer: "10.0.0.148:6817"})
+	deadline := time.Now().Add(5 * time.Second)
+	for {
+		a.fwMu.Lock()
+		got := sh.installed
+		a.fwMu.Unlock()
+		if strings.Contains(got, "6817-6819") && strings.Contains(got, "60001-63000") {
+			break
+		}
+		if time.Now().After(deadline) {
+			t.Fatalf("given a part in Slurm, the rules still lack its ports:\n%s", got)
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+}

internal/agent/provision.go

@@ -164,6 +164,13 @@ func (a *Agent) provisionLoop(ctx context.Context) {
 
 // provisionStatus records a change in one engine's installation.
 func (a *Agent) provisionStatus(st provision.Status) {
+	if st == (provision.Status{}) {
+		// A package install that was no engine's has ended.
+		a.mu.Lock()
+		delete(a.provStatus, "")
+		a.mu.Unlock()
+		return
+	}
 	a.mu.Lock()
 	prev, had := a.provStatus[st.Engine]
 	a.provStatus[st.Engine] = st

internal/agent/provision/install.go

@@ -483,7 +483,8 @@ func (in *Installer) packages(ctx context.Context, byFamily map[string][]string,
 		return fmt.Errorf("%s is missing and no package manager was found to install it",
 			strings.Join(missing, ", "))
 	}
-	in.progress(st, "installing %s with %s", strings.Join(missing, ", "), tool)
+	// The state already says installing.
+	in.progress(st, "%s, with %s", strings.Join(missing, ", "), tool)
 	for _, c := range cmds {
 		if err := in.runWaitingForLock(ctx, pm.env, c, st); err != nil {
 			return err
@@ -497,7 +498,13 @@ func (in *Installer) packages(ctx context.Context, byFamily map[string][]string,
 // byFamily is keyed by discovery.Host.Family: "deb" or "rpm".
 func (in *Installer) InstallPackages(ctx context.Context, byFamily map[string][]string) error {
 	var st Status
-	return in.packages(ctx, byFamily, &st)
+	err := in.packages(ctx, byFamily, &st)
+	if st.State == Installing {
+		// Its progress was no engine's: an empty status takes it off the
+		// node's page again, done or failed (the caller says which).
+		in.report(Status{})
+	}
+	return err
 }
 
 // lockWait bounds how long a package command waits for another to finish.

internal/agent/provision/packages_status_test.go

@@ -0,0 +1,36 @@
+package provision
+
+import (
+	"context"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+)
+
+// A package install that is no engine's says it is done, so its progress
+// does not stay on the node's page (found on the PoC installing Slurm).
+func TestInstallPackagesClearsItsProgress(t *testing.T) {
+	var got []Status
+	in := &Installer{Host: discovery.Host{OSID: "ubuntu"}, OnStatus: func(s Status) { got = append(got, s) },
+		Run: func(_ context.Context, _ []string, name string, _ ...string) ([]byte, error) {
+			if name == "sh" { // the query: not installed
+				return nil, context.Canceled
+			}
+			return nil, nil
+		}}
+	if err := in.InstallPackages(context.Background(), map[string][]string{"deb": {"munge"}}); err != nil {
+		t.Fatal(err)
+	}
+	if len(got) < 2 || got[0].State != Installing || got[len(got)-1] != (Status{}) {
+		t.Fatalf("statuses %+v", got)
+	}
+
+	got = nil
+	in.Run = func(context.Context, []string, string, ...string) ([]byte, error) { return nil, nil }
+	if err := in.InstallPackages(context.Background(), map[string][]string{"deb": {"munge"}}); err != nil {
+		t.Fatal(err)
+	}
+	if len(got) != 0 {
+		t.Fatalf("installed already, yet said %+v", got)
+	}
+}

internal/agent/removeweights_test.go

@@ -94,3 +94,35 @@ func TestWithNoCacheConfiguredNothingIsDeleted(t *testing.T) {
 		t.Errorf("an agent with no cache directory deleted something: %v", err)
 	}
 }
+
+// A model of several files is fetched into a directory of its own, and an
+// unused one goes whole: it was refused as not empty, and nothing was freed.
+func TestAnUnusedSetIsRemovedWhole(t *testing.T) {
+	dir := t.TempDir()
+	a := agentWithCache(t, dir)
+	set := weights.CachePathFor(dir, "https://mirror/coder/v1")
+	if err := os.MkdirAll(set, 0o755); err != nil {
+		t.Fatal(err)
+	}
+	write(t, filepath.Join(set, "model-00001-of-00002.gguf"))
+	write(t, filepath.Join(set, ".complete"))
+	// A link in the cache is taken away, not what it points at.
+	target := t.TempDir()
+	write(t, filepath.Join(target, "keep.gguf"))
+	link := weights.CachePathFor(dir, "https://mirror/linked/v1")
+	if err := os.Symlink(target, link); err != nil {
+		t.Fatal(err)
+	}
+
+	a.removeWeights([]string{set, link})
+
+	if _, err := os.Stat(set); !os.IsNotExist(err) {
+		t.Errorf("the set is still there: %v", err)
+	}
+	if _, err := os.Lstat(link); !os.IsNotExist(err) {
+		t.Errorf("the link is still there: %v", err)
+	}
+	if _, err := os.Stat(filepath.Join(target, "keep.gguf")); err != nil {
+		t.Errorf("what the link pointed at was deleted: %v", err)
+	}
+}

internal/agent/slurm.go

@@ -0,0 +1,38 @@
+package agent
+
+import (
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// FeatureSlurm is the agent keeping its node's part in Slurm (BACKLOG P).
+const FeatureSlurm = "slurm"
+
+// slurmMarker says llmgw started Slurm's daemons on this machine.
+var slurmMarker = "/var/lib/llmgw-agent/slurm"
+
+// setSlurmOut is where Slurm's status goes now: this session's queue, or
+// nowhere between sessions.
+func (a *Agent) setSlurmOut(out chan<- *llmgwv1.AgentMessage) {
+	a.mu.Lock()
+	a.slurmOut = out
+	a.mu.Unlock()
+	if out != nil {
+		a.slurm.Resend()
+	}
+}
+
+// sendSlurm queues a status without waiting: a full queue or no session
+// drops it, and the next session is sent it again.
+func (a *Agent) sendSlurm(st *llmgwv1.SlurmStatus) {
+	a.mu.Lock()
+	out := a.slurmOut
+	a.mu.Unlock()
+	if out == nil {
+		return
+	}
+	select {
+	case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_SlurmStatus{SlurmStatus: st}}:
+	default:
+		a.log.Warn("Slurm's status not sent: the queue to the control plane is full")
+	}
+}

internal/agent/slurmnode/accounts.go

@@ -0,0 +1,214 @@
+package slurmnode
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"slices"
+	"strings"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
+)
+
+// accountsMarker lists what llmgw made in Slurm's accounting, a line each:
+// "account NAME" and "user LOGIN NAME". Nothing else is changed or taken
+// away: an account a Slurm administrator made is theirs.
+func (k *Keeper) accountsMarker() string { return k.marker + "-accounts" }
+
+// maxEnded is the most ended jobs sent at once: a day's on a busy cluster
+// would otherwise be a status message of megabytes.
+const maxEnded = 2000
+
+// accounts keeps Slurm's accounts and their users as want says, on the
+// controller.
+func (k *Keeper) accounts(ctx context.Context, want *llmgwv1.Slurm) error {
+	made := k.madeAccounts()
+	if !want.GetManageAccounts() {
+		if len(made) == 0 {
+			return nil
+		}
+		return k.applyAccounts(ctx, nil, made)
+	}
+	return k.applyAccounts(ctx, want.GetAccounts(), made)
+}
+
+func (k *Keeper) madeAccounts() []string {
+	b, err := k.host.ReadFile(k.accountsMarker())
+	if err != nil {
+		return nil
+	}
+	var out []string
+	for _, l := range strings.Split(strings.TrimSpace(string(b)), "\n") {
+		if l != "" {
+			out = append(out, l)
+		}
+	}
+	return out
+}
+
+func (k *Keeper) sacctmgr(ctx context.Context, args ...string) error {
+	out, err := k.host.Run(ctx, nil, "sacctmgr", append([]string{"-i"}, args...)...)
+	if err != nil {
+		return fmt.Errorf("sacctmgr %s: %v: %s", strings.Join(args, " "), err, lastLine(out))
+	}
+	return nil
+}
+
+func (k *Keeper) applyAccounts(ctx context.Context, want []*llmgwv1.SlurmAccount, made []string) error {
+	out, err := k.host.Run(ctx, nil, "sacctmgr", slurm.AssocFormat...)
+	if err != nil {
+		return fmt.Errorf("reading Slurm's accounts: %v: %s", err, lastLine(out))
+	}
+	have := map[string]bool{} // "account NAME", "user LOGIN NAME"
+	for _, a := range slurm.ParseAssociations(string(out)) {
+		if a.User == "" {
+			have["account "+a.Account] = true
+		} else {
+			have["user "+a.User+" "+a.Account] = true
+		}
+	}
+	keep := map[string]bool{}
+	var errs []error
+	ours := slices.Clone(made)
+	note := func(line string) {
+		if !slices.Contains(ours, line) {
+			ours = append(ours, line)
+		}
+	}
+	for _, a := range want {
+		name, parent := a.GetName(), a.GetParent()
+		if !validAccount(name) || (parent != "root" && !validAccount(parent)) {
+			errs = append(errs, fmt.Errorf("the leader sent an account %q under %q", name, parent))
+			continue
+		}
+		line := "account " + name
+		keep[line] = true
+		if !have[line] {
+			k.log.Info("a Slurm account is made, as llmgw's projects say", "account", name, "parent", parent)
+			if err := k.sacctmgr(ctx, "add", "account", name, "parent="+parent, "description=llmgw"); err != nil {
+				errs = append(errs, err)
+				continue
+			}
+			note(line)
+		}
+		for _, u := range a.GetUsers() {
+			if !slurm.ValidLogin(u) {
+				continue
+			}
+			ul := "user " + u + " " + name
+			keep[ul] = true
+			if have[ul] {
+				continue
+			}
+			if err := k.sacctmgr(ctx, "add", "user", u, "account="+name); err != nil {
+				errs = append(errs, err)
+				continue
+			}
+			note(ul)
+		}
+	}
+	// Taken away: what llmgw made and no longer wants, users before their
+	// accounts, children before their parents (made later, listed later).
+	var left []string
+	for i := len(ours) - 1; i >= 0; i-- {
+		line := ours[i]
+		if keep[line] || !strings.HasPrefix(line, "user ") {
+			continue
+		}
+		f := strings.Fields(line)
+		if !have[line] {
+			continue
+		}
+		if err := k.removeAssoc(ctx, f[1], f[2], have, keep); err != nil {
+			errs = append(errs, err)
+			left = append(left, line)
+		}
+	}
+	for i := len(ours) - 1; i >= 0; i-- {
+		line := ours[i]
+		if keep[line] || !strings.HasPrefix(line, "account ") {
+			continue
+		}
+		if have[line] {
+			k.log.Info("a Slurm account llmgw made is taken away; its jobs' records stay", "account", line[len("account "):])
+			if err := k.sacctmgr(ctx, "remove", "account", line[len("account "):]); err != nil {
+				errs = append(errs, err)
+				left = append(left, line)
+			}
+		}
+	}
+	var lines []string
+	for _, l := range ours {
+		if keep[l] || slices.Contains(left, l) {
+			lines = append(lines, l)
+		}
+	}
+	if !slices.Equal(lines, made) {
+		var err error
+		if len(lines) == 0 {
+			err = k.host.Remove(k.accountsMarker())
+		} else {
+			_, err = k.write(k.accountsMarker(), []byte(strings.Join(lines, "\n")+"\n"), 0o644, "")
+		}
+		errs = append(errs, err)
+	}
+	return errors.Join(errs...)
+}
+
+func validAccount(s string) bool {
+	return s != "" && len(s) <= 64 && strings.Trim(s, "abcdefghijklmnopqrstuvwxyz0123456789_-") == ""
+}
+
+// ended are the jobs that ended in the last day, newest last.
+func (k *Keeper) ended(ctx context.Context) ([]*llmgwv1.SlurmJob, error) {
+	out, err := k.host.Run(ctx, nil, "sacct", slurm.SacctArgs...)
+	if err != nil {
+		return nil, fmt.Errorf("sacct: %v: %s", err, lastLine(out))
+	}
+	jobs, err := slurm.ParseJobs(string(out))
+	if err != nil {
+		return nil, err
+	}
+	if len(jobs) > maxEnded {
+		jobs = jobs[len(jobs)-maxEnded:]
+	}
+	ps := make([]*llmgwv1.SlurmJob, 0, len(jobs))
+	for _, j := range jobs {
+		ps = append(ps, &llmgwv1.SlurmJob{Id: j.ID, User: j.User, Account: j.Account, Partition: j.Partition,
+			State: j.State, StartUnix: j.Start.Unix(), EndUnix: j.End.Unix(), ElapsedSeconds: int64(j.Elapsed.Seconds()),
+			Cpus: int32(j.CPUs), Gpus: int32(j.GPUs)}) //nolint:gosec // counts of a node's
+	}
+	return ps, nil
+}
+
+// removeAssoc takes login off account. Slurm refuses to take away a
+// user's default account, so another of theirs is made the default first,
+// one llmgw keeps if there is one; with none left the user goes from the
+// accounting entirely.
+func (k *Keeper) removeAssoc(ctx context.Context, login, account string, have, keep map[string]bool) error {
+	var others []string
+	for line := range have {
+		f := strings.Fields(line)
+		if len(f) == 3 && f[0] == "user" && f[1] == login && f[2] != account {
+			others = append(others, f[2])
+		}
+	}
+	if len(others) == 0 {
+		return k.sacctmgr(ctx, "remove", "user", login)
+	}
+	slices.SortFunc(others, func(a, b string) int {
+		ka, kb := keep["user "+login+" "+a], keep["user "+login+" "+b]
+		switch {
+		case ka && !kb:
+			return -1
+		case kb && !ka:
+			return 1
+		}
+		return strings.Compare(a, b)
+	})
+	if err := k.sacctmgr(ctx, "modify", "user", login, "set", "defaultaccount="+others[0]); err != nil {
+		return err
+	}
+	return k.sacctmgr(ctx, "remove", "user", login, "account="+account)
+}

internal/agent/slurmnode/accounts_test.go

@@ -0,0 +1,83 @@
+package slurmnode
+
+import (
+	"context"
+	"slices"
+	"strings"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
+)
+
+func TestAccountsMadeAndTakenAwayOnlyIfLlmgwsOwn(t *testing.T) {
+	h := newFake()
+	k := New(h, nil, "/var/lib/llmgw-agent/slurm", nil)
+	assoc := "sacctmgr " + strings.Join(slurm.AssocFormat, " ")
+	// root, and an account a Slurm administrator made.
+	h.outputs[assoc] = "root||\nroot|root|\nphysics||root\n"
+	m := &llmgwv1.Slurm{ManageAccounts: true, Accounts: []*llmgwv1.SlurmAccount{
+		{Name: "test", Parent: "root"}, {Name: "test-arge", Parent: "test"},
+		{Name: "test-arge-deneme", Parent: "test-arge", Users: []string{"ada", "ayse"}},
+	}}
+	if err := k.accounts(context.Background(), m); err != nil {
+		t.Fatal(err)
+	}
+	for _, want := range []string{"sacctmgr -i add account test parent=root description=llmgw",
+		"sacctmgr -i add account test-arge-deneme parent=test-arge description=llmgw",
+		"sacctmgr -i add user ayse account=test-arge-deneme"} {
+		if !slices.Contains(h.ran, want) {
+			t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
+		}
+	}
+	if !strings.Contains(string(h.files[k.accountsMarker()]), "user ada test-arge-deneme") {
+		t.Errorf("marker %q", h.files[k.accountsMarker()])
+	}
+
+	// Now there; ayse leaves the project, and the team goes.
+	h.outputs[assoc] = "root||\nphysics||root\ntest||root\ntest-arge||test\ntest-arge-deneme||test-arge\n" +
+		"test-arge-deneme|ada|test-arge\ntest-arge-deneme|ayse|test-arge\n"
+	h.ran = nil
+	m.Accounts = []*llmgwv1.SlurmAccount{{Name: "test", Parent: "root"}}
+	if err := k.accounts(context.Background(), m); err != nil {
+		t.Fatal(err)
+	}
+	// Neither has another account: they go from the accounting.
+	want := []string{"sacctmgr -i remove user ayse", "sacctmgr -i remove user ada",
+		"sacctmgr -i remove account test-arge-deneme", "sacctmgr -i remove account test-arge"}
+	var got []string
+	for _, c := range h.ran {
+		if strings.Contains(c, "remove") || strings.Contains(c, " add ") {
+			got = append(got, c)
+		}
+	}
+	if !slices.Equal(got, want) {
+		t.Errorf("ran:\n%s\nwant:\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
+	}
+	if slices.ContainsFunc(h.ran, func(c string) bool { return strings.Contains(c, "physics") }) {
+		t.Error("touched an account llmgw did not make")
+	}
+
+	// One still on another account: that one is made the default first.
+	h.outputs[assoc] = "root||\nphysics||root\nphysics|ada|\ntest||root\nx||test\nx|ada|\n"
+	h.files[k.accountsMarker()] = []byte("account test\naccount x\nuser ada x\n")
+	h.ran = nil
+	m.Accounts = []*llmgwv1.SlurmAccount{{Name: "test", Parent: "root"}}
+	if err := k.accounts(context.Background(), m); err != nil {
+		t.Fatal(err)
+	}
+	if i, j := slices.Index(h.ran, "sacctmgr -i modify user ada set defaultaccount=physics"),
+		slices.Index(h.ran, "sacctmgr -i remove user ada account=x"); i < 0 || j < i {
+		t.Errorf("ran:\n%s", strings.Join(h.ran, "\n"))
+	}
+
+	// Off: what llmgw made goes.
+	h.outputs[assoc] = "root||\nphysics||root\ntest||root\n"
+	h.ran = nil
+	if err := k.accounts(context.Background(), &llmgwv1.Slurm{}); err != nil {
+		t.Fatal(err)
+	}
+	if !slices.Contains(h.ran, "sacctmgr -i remove account test") || h.files[k.accountsMarker()] != nil {
+		t.Errorf("not taken away:\n%s", strings.Join(h.ran, "\n"))
+	}
+}

internal/agent/slurmnode/containers.go

@@ -0,0 +1,198 @@
+package slurmnode
+
+import (
+	"context"
+	"fmt"
+	"path/filepath"
+	"runtime"
+	"strings"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
+)
+
+// cacheDir holds what is fetched for Slurm: enroot's packages, pyxis's
+// source and its build.
+const cacheDir = "/var/cache/llmgw-agent/slurm"
+
+// pyxisMarker is the pyxis version built here, against this Slurm.
+func (k *Keeper) pyxisMarker() string { return k.marker + "-pyxis" }
+
+// containers has enroot and pyxis here, set up, when want says so; what
+// is there is left when it no longer does (Slurm stops loading pyxis once
+// the controller's plugstack.conf goes). Answers the status line, empty
+// while either is missing.
+func (k *Keeper) containers(ctx context.Context, want *llmgwv1.Slurm) (string, error) {
+	if !want.GetContainers() {
+		return "", nil
+	}
+	family := k.family()
+	if family == "" {
+		return "", fmt.Errorf("neither dpkg nor rpm here: enroot and pyxis come as deb or rpm")
+	}
+	if err := k.enroot(ctx, family); err != nil {
+		return "", fmt.Errorf("enroot: %w", err)
+	}
+	if err := k.enrootConf(ctx, want.GetShared().GetPath()); err != nil {
+		return "", fmt.Errorf("enroot's settings: %w", err)
+	}
+	if _, err := k.host.LookPath("nvidia-smi"); err == nil {
+		if err := k.nvidiaContainer(ctx, family); err != nil {
+			return "", fmt.Errorf("the GPUs in containers (nvidia-container-cli): %w", err)
+		}
+	}
+	if err := k.pyxis(ctx, family); err != nil {
+		return "", fmt.Errorf("pyxis: %w", err)
+	}
+	return slurm.ContainersStatus(), nil
+}
+
+func (k *Keeper) family() string {
+	if _, err := k.host.LookPath("dpkg"); err == nil {
+		return "deb"
+	}
+	if _, err := k.host.LookPath("rpm"); err == nil {
+		return "rpm"
+	}
+	return ""
+}
+
+// enroot installs the pinned release when another, or none, is here.
+func (k *Keeper) enroot(ctx context.Context, family string) error {
+	if k.installed["enroot"] {
+		return nil
+	}
+	if out, err := k.host.Run(ctx, nil, "enroot", "version"); err == nil && strings.TrimSpace(string(out)) == slurm.EnrootVersion {
+		k.installed["enroot"] = true
+		return nil
+	}
+	pkgs, err := slurm.EnrootPackages(family, runtime.GOARCH)
+	if err != nil {
+		return err
+	}
+	if k.inst == nil {
+		return fmt.Errorf("enroot missing, and this agent installs nothing")
+	}
+	var files []string
+	for _, p := range pkgs {
+		f := filepath.Join(cacheDir, p.Name)
+		if err := k.host.Download(ctx, p.URL(), f, p.SHA256); err != nil {
+			return err
+		}
+		files = append(files, f)
+	}
+	k.log.Info("installing enroot, for Slurm's jobs in containers", "version", slurm.EnrootVersion)
+	if err := k.inst.InstallPackages(ctx, map[string][]string{family: files}); err != nil {
+		return err
+	}
+	out, err := k.host.Run(ctx, nil, "enroot", "version")
+	if v := strings.TrimSpace(string(out)); err != nil || v != slurm.EnrootVersion {
+		return fmt.Errorf("enroot says %q after installing %s: %v", v, slurm.EnrootVersion, err)
+	}
+	k.installed["enroot"] = true
+	return nil
+}
+
+// enrootConf writes enroot's settings and makes its directories.
+func (k *Keeper) enrootConf(ctx context.Context, shared string) error {
+	if _, err := k.write(slurm.EnrootConfPath, []byte(slurm.EnrootConf), 0o644, ""); err != nil {
+		return err
+	}
+	changed, err := k.write(slurm.EnrootTmpfiles, []byte(slurm.EnrootTmpfilesConf), 0o644, "")
+	if err != nil {
+		return err
+	}
+	if changed {
+		if out, err := k.host.Run(ctx, nil, "systemd-tmpfiles", "--create", slurm.EnrootTmpfiles); err != nil {
+			return fmt.Errorf("systemd-tmpfiles: %v: %s", err, lastLine(out))
+		}
+	}
+	for _, d := range []string{slurm.EnrootLib + "/cache", slurm.EnrootLib + "/data"} {
+		if _, err := k.host.Run(ctx, nil, "test", "-d", d); err == nil {
+			continue
+		}
+		// Each user makes their own beneath, as in /tmp.
+		if out, err := k.host.Run(ctx, nil, "install", "-d", "-m", "1777", d); err != nil {
+			return fmt.Errorf("making %s: %v: %s", d, err, lastLine(out))
+		}
+	}
+	if shared == "" {
+		return k.host.Remove(slurm.EnrootMounts)
+	}
+	_, err = k.write(slurm.EnrootMounts, []byte(slurm.EnrootSharedMount(shared)), 0o644, "")
+	return err
+}
+
+// nvidiaContainer installs nvidia-container-cli, which enroot's hook uses
+// to give a container the job's GPUs, from NVIDIA's repository.
+func (k *Keeper) nvidiaContainer(ctx context.Context, family string) error {
+	if k.installed["nvidia-container-cli"] {
+		return nil
+	}
+	if _, err := k.host.LookPath("nvidia-container-cli"); err == nil {
+		k.installed["nvidia-container-cli"] = true
+		return nil
+	}
+	if k.inst == nil {
+		return fmt.Errorf("nvidia-container-cli missing, and this agent installs nothing")
+	}
+	switch family {
+	case "deb":
+		if err := k.host.Download(ctx, slurm.NvidiaContainerKeyURL, slurm.NvidiaContainerKeyPath,
+			slurm.NvidiaContainerKeySHA256); err != nil {
+			return err
+		}
+		if _, err := k.write(slurm.NvidiaContainerAptList, []byte(slurm.NvidiaContainerApt), 0o644, ""); err != nil {
+			return err
+		}
+	case "rpm":
+		if _, err := k.write(slurm.NvidiaContainerYumRepo, []byte(slurm.NvidiaContainerYum), 0o644, ""); err != nil {
+			return err
+		}
+	}
+	k.log.Info("installing nvidia-container-cli, so containers get the job's GPUs")
+	if err := k.inst.InstallPackages(ctx, slurm.NvidiaContainerPackages); err != nil {
+		return err
+	}
+	if _, err := k.host.LookPath("nvidia-container-cli"); err != nil {
+		return fmt.Errorf("still missing after installing libnvidia-container-tools")
+	}
+	k.installed["nvidia-container-cli"] = true
+	return nil
+}
+
+// pyxis builds the pinned pyxis against this machine's Slurm and installs
+// it, when it has not been, or was for another version.
+func (k *Keeper) pyxis(ctx context.Context, family string) error {
+	if b, err := k.host.ReadFile(k.pyxisMarker()); err == nil && strings.TrimSpace(string(b)) == slurm.PyxisVersion {
+		if _, err := k.host.Run(ctx, nil, "test", "-f", slurm.PyxisPlugin); err == nil {
+			return nil
+		}
+	}
+	if k.inst == nil {
+		return fmt.Errorf("pyxis missing, and this agent installs nothing")
+	}
+	if err := k.inst.InstallPackages(ctx, map[string][]string{family: slurm.PyxisBuildPackages[family]}); err != nil {
+		return fmt.Errorf("installing what builds it: %w", err)
+	}
+	tarball := filepath.Join(cacheDir, "pyxis-"+slurm.PyxisVersion+".tar.gz")
+	if err := k.host.Download(ctx, slurm.PyxisSource, tarball, slurm.PyxisSHA256); err != nil {
+		return err
+	}
+	src := filepath.Join(cacheDir, "pyxis-"+slurm.PyxisVersion)
+	k.log.Info("building pyxis against this machine's Slurm", "version", slurm.PyxisVersion)
+	for _, c := range [][]string{
+		{"rm", "-rf", src},
+		{"tar", "-xzf", tarball, "-C", cacheDir},
+		{"make", "-C", src, "install"},
+	} {
+		if out, err := k.host.Run(ctx, nil, c[0], c[1:]...); err != nil {
+			return fmt.Errorf("%s: %v: %s", strings.Join(c, " "), err, lastLine(out))
+		}
+	}
+	if _, err := k.host.Run(ctx, nil, "test", "-f", slurm.PyxisPlugin); err != nil {
+		return fmt.Errorf("built, yet %s is not there", slurm.PyxisPlugin)
+	}
+	_, err := k.write(k.pyxisMarker(), []byte(slurm.PyxisVersion+"\n"), 0o644, "")
+	return err
+}

internal/agent/slurmnode/containers_test.go

@@ -0,0 +1,92 @@
+package slurmnode
+
+import (
+	"context"
+	"runtime"
+	"slices"
+	"strings"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
+)
+
+func TestContainersInstalledFromPinnedReleases(t *testing.T) {
+	if _, err := slurm.EnrootPackages("deb", runtime.GOARCH); err != nil {
+		t.Skip(err)
+	}
+	h := newFake()
+	h.bins["dpkg"], h.bins["nvidia-smi"] = true, true
+	inst := &fakeInst{host: h}
+	k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
+	// enroot is not here until installed, and then says its version.
+	h.fail["enroot version"] = true
+	h.fail["test -d "+slurm.EnrootLib+"/cache"] = true
+	h.fail["test -f "+slurm.PyxisPlugin] = true
+	m := nodeMsg(true)
+	m.Containers = true
+	m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Nfs: "10.0.0.11:/shared"}
+
+	st := k.Reconcile(context.Background(), m)
+	if st.GetContainers() != "" || !strings.Contains(st.GetError(), "containers: enroot") {
+		t.Fatalf("ready with enroot saying nothing: %v", st)
+	}
+	pkgs, _ := slurm.EnrootPackages("deb", runtime.GOARCH)
+	for _, p := range pkgs {
+		if !slices.Contains(h.ran, "download "+p.URL()+" "+cacheDir+"/"+p.Name+" "+p.SHA256) {
+			t.Errorf("%s not fetched with its digest:\n%s", p.Name, strings.Join(h.ran, "\n"))
+		}
+	}
+	if len(inst.got["deb"]) != 2 || !strings.HasPrefix(inst.got["deb"][0], cacheDir) {
+		t.Errorf("installed %v", inst.got)
+	}
+
+	delete(h.fail, "enroot version")
+	h.outputs["enroot version"] = slurm.EnrootVersion + "\n"
+	h.bins["nvidia-container-cli"] = false
+	st = k.Reconcile(context.Background(), m)
+	if !strings.Contains(st.GetError(), "nvidia-container-cli") {
+		t.Errorf("a GPU machine without nvidia-container-cli is ready: %v", st)
+	}
+	if string(h.files[slurm.NvidiaContainerAptList]) != slurm.NvidiaContainerApt ||
+		!slices.Contains(h.ran, "download "+slurm.NvidiaContainerKeyURL+" "+slurm.NvidiaContainerKeyPath+" "+slurm.NvidiaContainerKeySHA256) {
+		t.Errorf("NVIDIA's repository not added with its key checked")
+	}
+
+	h.bins["nvidia-container-cli"] = true
+	st = k.Reconcile(context.Background(), m)
+	// Built, but the plugin is still not there.
+	if st.GetContainers() != "" || !strings.Contains(st.GetError(), "pyxis") {
+		t.Fatalf("status %v", st)
+	}
+	for _, want := range []string{"tar -xzf " + cacheDir + "/pyxis-" + slurm.PyxisVersion + ".tar.gz -C " + cacheDir,
+		"make -C " + cacheDir + "/pyxis-" + slurm.PyxisVersion + " install", "install -d -m 1777 " + slurm.EnrootLib + "/cache"} {
+		if !slices.Contains(h.ran, want) {
+			t.Errorf("did not run %q", want)
+		}
+	}
+	delete(h.fail, "test -f "+slurm.PyxisPlugin)
+	st = k.Reconcile(context.Background(), m)
+	if st.GetContainers() != slurm.ContainersStatus() || st.GetError() != "" {
+		t.Fatalf("status %v", st)
+	}
+	if string(h.files[slurm.EnrootConfPath]) != slurm.EnrootConf ||
+		!strings.Contains(string(h.files[slurm.EnrootMounts]), "/shared /shared none") {
+		t.Errorf("enroot's settings not written")
+	}
+
+	// Done: nothing fetched or built again.
+	h.ran = nil
+	k.Reconcile(context.Background(), m)
+	for _, c := range h.ran {
+		if strings.HasPrefix(c, "download") || strings.HasPrefix(c, "make") {
+			t.Errorf("ran %q again", c)
+		}
+	}
+
+	// Off: nothing said, nothing taken away.
+	m.Containers = false
+	if st := k.Reconcile(context.Background(), m); st.GetContainers() != "" {
+		t.Errorf("status %v", st)
+	}
+}

internal/agent/slurmnode/dbd.go

@@ -0,0 +1,170 @@
+package slurmnode
+
+import (
+	"bufio"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"strings"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
+)
+
+// mariaDBConfDirs are where each family's MariaDB reads its settings: Debian
+// and Ubuntu's, then Red Hat's.
+var mariaDBConfDirs = []string{"/etc/mysql/mariadb.conf.d", "/etc/my.cnf.d"}
+
+const mariaDBConfName = "90-llmgw-slurm.cnf"
+
+// acctSQLPath is where the database's statements are put for the client to
+// read, never on its command line: they hold the password.
+const acctSQLPath = "/run/llmgw-slurm-acct.sql"
+
+// accounting keeps slurmdbd on the controller: MariaDB on this machine
+// alone, the database and its user, slurmdbd on them, and the cluster in it.
+// Before slurmctld, which reports to it from its first start.
+func (k *Keeper) accounting(ctx context.Context, key []byte, files map[string]string) error {
+	password := slurm.DbdPassword(key)
+	changed, err := k.mariaDBSettings(ctx)
+	if err != nil {
+		return err
+	}
+	switch {
+	case changed:
+		k.log.Info("MariaDB, for Slurm's accounting, starts on this machine alone")
+		if err := k.systemctl(ctx, "enable", "--now", "mariadb", "&&restart"); err != nil {
+			return err
+		}
+	case !k.active(ctx, "mariadb"):
+		if err := k.systemctl(ctx, "enable", "--now", "mariadb"); err != nil {
+			return err
+		}
+	}
+	client, err := k.mariaDBClient()
+	if err != nil {
+		return err
+	}
+	if err := k.waitFor(ctx, "MariaDB", client, "-e", "SELECT 1"); err != nil {
+		return err
+	}
+	// The database and its user, once each time the agent starts or the
+	// password is new: cheap, and a database restored from elsewhere gets
+	// its user back.
+	sum := sha256.Sum256([]byte(password))
+	if mark := hex.EncodeToString(sum[:]); k.acctDone != mark {
+		if err := k.host.WriteFile(acctSQLPath, []byte(slurm.AcctSQL(password)), 0o600, ""); err != nil {
+			return err
+		}
+		out, err := k.host.Run(ctx, nil, client, "-e", "source "+acctSQLPath)
+		_ = k.host.Remove(acctSQLPath)
+		if err != nil {
+			return fmt.Errorf("making Slurm's database: %v: %s", err, lastLine(out))
+		}
+		k.acctDone = mark
+	}
+
+	out, err := k.host.Run(ctx, nil, "hostname", "-s")
+	host := strings.TrimSpace(string(out))
+	if err != nil || host == "" {
+		return fmt.Errorf("this machine's short hostname: %v: %s", err, lastLine(out))
+	}
+	changed, err = k.write(slurm.DbdConfPath, []byte(slurm.DbdConf(host, password)), 0o600, "slurm")
+	if err != nil {
+		return err
+	}
+	switch {
+	case changed:
+		k.log.Info("slurmdbd starts again on new settings")
+		if err := k.systemctl(ctx, "enable", "--now", "slurmdbd", "&&restart"); err != nil {
+			return err
+		}
+	case !k.active(ctx, "slurmdbd"):
+		if err := k.systemctl(ctx, "enable", "--now", "slurmdbd"); err != nil {
+			return err
+		}
+	}
+	if err := k.waitFor(ctx, "slurmdbd", "sacctmgr", "-n", "-P", "show", "cluster", "format=cluster"); err != nil {
+		return err
+	}
+	name := clusterName(files["slurm.conf"])
+	if name == "" || k.clusterAdded == name {
+		return nil
+	}
+	out, err = k.host.Run(ctx, nil, "sacctmgr", "-n", "-P", "show", "cluster", "format=cluster")
+	if err != nil {
+		return fmt.Errorf("sacctmgr show cluster: %v: %s", err, lastLine(out))
+	}
+	for _, l := range strings.Split(string(out), "\n") {
+		if strings.TrimSpace(l) == name {
+			k.clusterAdded = name
+			return nil
+		}
+	}
+	k.log.Info("the cluster is added to Slurm's accounting", "cluster", name)
+	if out, err := k.host.Run(ctx, nil, "sacctmgr", "-i", "add", "cluster", name); err != nil {
+		return fmt.Errorf("sacctmgr add cluster %s: %v: %s", name, err, lastLine(out))
+	}
+	k.clusterAdded = name
+	return nil
+}
+
+// mariaDBSettings keeps MariaDB to this machine, when llmgw installed it;
+// one that was here before llmgw keeps its own settings, and only gets the
+// database and its user.
+func (k *Keeper) mariaDBSettings(ctx context.Context) (bool, error) {
+	if _, err := k.host.ReadFile(k.mariaDBMarker()); err != nil {
+		return false, nil
+	}
+	for _, d := range mariaDBConfDirs {
+		if _, err := k.host.Run(ctx, nil, "test", "-d", d); err == nil {
+			return k.write(d+"/"+mariaDBConfName, []byte(slurm.MariaDBConf), 0o644, "")
+		}
+	}
+	return false, fmt.Errorf("MariaDB has none of %s to put its settings in", strings.Join(mariaDBConfDirs, ", "))
+}
+
+// mariaDBMarker is there when llmgw installed MariaDB on this machine.
+func (k *Keeper) mariaDBMarker() string { return k.marker + "-mariadb" }
+
+// mariaDBClient is MariaDB's command-line client, under either of its
+// names.
+func (k *Keeper) mariaDBClient() (string, error) {
+	for _, c := range []string{"mariadb", "mysql"} {
+		if _, err := k.host.LookPath(c); err == nil {
+			return c, nil
+		}
+	}
+	return "", errors.New("MariaDB's client (mariadb or mysql) is not installed")
+}
+
+// waitFor runs a command until it succeeds, half a minute at most.
+func (k *Keeper) waitFor(ctx context.Context, what, name string, args ...string) error {
+	var last []byte
+	for range 30 {
+		out, err := k.host.Run(ctx, nil, name, args...)
+		if err == nil {
+			return nil
+		}
+		last = out
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-time.After(pingWait):
+		}
+	}
+	return fmt.Errorf("%s did not answer after starting: %s", what, lastLine(last))
+}
+
+// clusterName is ClusterName in slurm.conf.
+func clusterName(conf string) string {
+	sc := bufio.NewScanner(strings.NewReader(conf))
+	for sc.Scan() {
+		if v, ok := strings.CutPrefix(strings.TrimSpace(sc.Text()), "ClusterName="); ok {
+			return strings.TrimSpace(v)
+		}
+	}
+	return ""
+}

internal/agent/slurmnode/shared.go

@@ -0,0 +1,150 @@
+package slurmnode
+
+import (
+	"context"
+	"fmt"
+	"path/filepath"
+	"strings"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
+)
+
+// What a machine says of the shared path.
+const (
+	SharedMounted  = "mounted"
+	SharedExported = "exported"
+	SharedMissing  = "missing"
+)
+
+// shared keeps the shared path as want says: exported from here, mounted
+// from an export, or looked for where the site mounts it. Answers how it
+// is.
+func (k *Keeper) shared(ctx context.Context, want *llmgwv1.Slurm) (string, error) {
+	sh := want.GetShared()
+	path := sh.GetPath()
+	k.unshare(ctx, path, sh.GetServe(), sh.GetNfs())
+	if path == "" {
+		return "", nil
+	}
+	if !strings.HasPrefix(path, "/") || strings.Contains(path, "..") || strings.ContainsAny(path, " \n\t\"'\\$`") {
+		return SharedMissing, fmt.Errorf("the leader sent a path %q", path)
+	}
+	switch {
+	case sh.GetServe():
+		if err := k.host.MkdirAll(path, 0o755, ""); err != nil {
+			return SharedMissing, err
+		}
+		changed, err := k.write(slurm.ExportsPath, []byte(slurm.Exports(path, sh.GetClients())), 0o644, "")
+		if err != nil {
+			return SharedMissing, err
+		}
+		if !k.active(ctx, "nfs-server") {
+			k.log.Info("NFS starts on this machine, exporting the Slurm cluster's shared files", "path", path)
+			if err := k.systemctl(ctx, "enable", "--now", "nfs-server"); err != nil {
+				return SharedMissing, err
+			}
+			changed = true
+		}
+		if changed {
+			if out, err := k.host.Run(ctx, nil, "exportfs", "-ra"); err != nil {
+				return SharedMissing, fmt.Errorf("exportfs -ra: %v: %s", err, lastLine(out))
+			}
+		}
+		return SharedExported, nil
+	case sh.GetNfs() != "":
+		nfs := sh.GetNfs()
+		if strings.ContainsAny(nfs, " \n\t\"'\\$`") || !strings.Contains(nfs, ":/") {
+			return SharedMissing, fmt.Errorf("the leader sent an export %q", nfs)
+		}
+		if err := k.host.MkdirAll(path, 0o755, ""); err != nil {
+			return SharedMissing, err
+		}
+		unit := slurm.MountUnitName(path)
+		changed, err := k.write(filepath.Join(slurm.MountUnitDir, unit), []byte(slurm.MountUnit(nfs, path)), 0o644, "")
+		if err != nil {
+			return SharedMissing, err
+		}
+		if changed {
+			k.log.Info("the Slurm cluster's shared files are mounted here", "from", nfs, "at", path)
+			if err := k.systemctl(ctx, "daemon-reload"); err != nil {
+				return SharedMissing, err
+			}
+			if err := k.systemctl(ctx, "enable", unit); err != nil {
+				return SharedMissing, err
+			}
+			// From another export: a restart mounts the new one, and fails
+			// while a job still has files open on the old.
+			if err := k.systemctl(ctx, "restart", unit); err != nil {
+				return SharedMissing, err
+			}
+		} else if !k.active(ctx, unit) {
+			if err := k.systemctl(ctx, "start", unit); err != nil {
+				return SharedMissing, err
+			}
+		}
+	}
+	if _, err := k.host.Run(ctx, nil, "mountpoint", "-q", path); err != nil {
+		if sh.GetNfs() == "" {
+			return SharedMissing, fmt.Errorf("%s is not mounted here: the site mounts it (slurm.shared names no export)", path)
+		}
+		return SharedMissing, fmt.Errorf("%s is not mounted from %s", path, sh.GetNfs())
+	}
+	return SharedMounted, nil
+}
+
+// unshare takes away the export or the mount llmgw kept here that want no
+// longer has: path is where the files are wanted now, serve and nfs how.
+func (k *Keeper) unshare(ctx context.Context, path string, serve bool, nfs string) {
+	if _, err := k.host.ReadFile(slurm.ExportsPath); err == nil && (!serve || path == "") {
+		k.log.Info("this machine no longer exports the Slurm cluster's shared files")
+		_ = k.host.Remove(slurm.ExportsPath)
+		_, _ = k.host.Run(ctx, nil, "exportfs", "-ra")
+	}
+	out, err := k.host.Run(ctx, nil, "systemctl", "list-units", "--all", "--plain", "--no-legend", "--type=mount")
+	if err != nil {
+		return
+	}
+	keep := ""
+	if path != "" && nfs != "" && !serve {
+		keep = slurm.MountUnitName(path)
+	}
+	for _, l := range strings.Split(string(out), "\n") {
+		f := strings.Fields(l)
+		if len(f) == 0 || f[0] == keep {
+			continue
+		}
+		file := filepath.Join(slurm.MountUnitDir, f[0])
+		b, err := k.host.ReadFile(file)
+		if err != nil || !strings.HasPrefix(string(b), "# Written by llmgw (BACKLOG P): the Slurm cluster's shared files.") {
+			continue
+		}
+		k.log.Info("the Slurm cluster's old shared mount is taken away", "unit", f[0])
+		if err := k.systemctl(ctx, "disable", "--now", f[0]); err != nil {
+			k.log.Warn("could not unmount the old shared files", "unit", f[0], "err", err)
+			continue
+		}
+		_ = k.host.Remove(file)
+		_ = k.systemctl(ctx, "daemon-reload")
+	}
+}
+
+// sharedPackages are what the shared path needs here, by the command that
+// shows each is installed.
+func sharedPackages(want *llmgwv1.Slurm, need map[string][2]string) {
+	switch sh := want.GetShared(); {
+	case sh.GetPath() == "":
+	case sh.GetServe():
+		need["exportfs"] = [2]string{"nfs-kernel-server", "nfs-utils"}
+	case sh.GetNfs() != "":
+		need["mount.nfs"] = [2]string{"nfs-common", "nfs-utils"}
+	}
+}
+
+// Serves is whether this machine exports the shared files, so NFS's port
+// is kept for the cluster's networks.
+func (k *Keeper) Serves() bool {
+	k.mu.Lock()
+	defer k.mu.Unlock()
+	return k.want.GetShared().GetServe() && k.want.GetShared().GetPath() != ""
+}

internal/agent/slurmnode/shared_test.go

@@ -0,0 +1,171 @@
+package slurmnode
+
+import (
+	"context"
+	"slices"
+	"strings"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
+)
+
+func ayse() *llmgwv1.SlurmUser {
+	return &llmgwv1.SlurmUser{Login: "ayse", Uid: 200000, Name: "Ayşe Y.", Home: "/shared/home/ayse",
+		SshKeys: []string{"ssh-ed25519 AAAA ayse@laptop"}}
+}
+
+func TestTheServerExportsAndMakesUsersWithHomes(t *testing.T) {
+	h := newFake()
+	h.active["ssh"] = true
+	inst := &fakeInst{host: h}
+	k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
+	m := nodeMsg(false)
+	m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Serve: true, Clients: []string{"10.0.0.22", "10.0.0.21"}}
+	m.ManageUsers, m.MakeHomes, m.Users = true, true, []*llmgwv1.SlurmUser{ayse()}
+	h.fail["test -d /shared/home/ayse"] = true
+	h.files[slurm.PAMSSHD] = []byte(pamStack)
+	st := k.Reconcile(context.Background(), m)
+	if st.GetError() != "" {
+		t.Fatal(st.GetError())
+	}
+	if st.GetShared() != SharedExported || st.GetUsers() != 1 {
+		t.Fatalf("status %v", st)
+	}
+	if !slices.Contains(inst.got["deb"], "nfs-kernel-server") {
+		t.Errorf("packages %v", inst.got)
+	}
+	if e := string(h.files[slurm.ExportsPath]); !strings.Contains(e, `"/shared" 10.0.0.21(`) {
+		t.Errorf("exports:\n%s", e)
+	}
+	for _, want := range []string{"systemctl enable --now nfs-server", "exportfs -ra", "groupadd -g 200000 ayse",
+		"useradd -u 200000 -g 200000 -M -d /shared/home/ayse -s /bin/bash -c Ayşe Y. ayse",
+		"install -d -m 0700 -o 200000 -g 200000 /shared/home/ayse", "sshd -t", "systemctl reload ssh"} {
+		if !slices.Contains(h.ran, want) {
+			t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
+		}
+	}
+	if string(h.files[slurm.AuthorizedKeysDir+"/ayse"]) != "ssh-ed25519 AAAA ayse@laptop\n" ||
+		string(h.files[k.usersMarker()]) != "ayse\n" || h.files[slurm.SSHDropIn] == nil {
+		t.Errorf("files: keys %q, marker %q", h.files[slurm.AuthorizedKeysDir+"/ayse"], h.files[k.usersMarker()])
+	}
+
+	// Again, made: nothing is run again.
+	h.outputs["getent passwd ayse"] = "ayse:x:200000:200000:Ayşe Y.:/shared/home/ayse:/bin/bash\n"
+	delete(h.fail, "test -d /shared/home/ayse")
+	h.ran = nil
+	k.Reconcile(context.Background(), m)
+	for _, c := range h.ran {
+		if strings.HasPrefix(c, "useradd") || strings.HasPrefix(c, "usermod") || strings.HasPrefix(c, "exportfs") ||
+			strings.HasPrefix(c, "install") || c == "systemctl reload ssh" {
+			t.Errorf("ran %q again", c)
+		}
+	}
+
+	// Gone from the accounts: taken away, its home left.
+	m.Users = nil
+	k.Reconcile(context.Background(), m)
+	if !slices.Contains(h.ran, "userdel ayse") || h.files[k.usersMarker()] != nil || h.files[slurm.SSHDropIn] != nil ||
+		h.files[slurm.AuthorizedKeysDir+"/ayse"] != nil {
+		t.Errorf("not taken away:\n%s", strings.Join(h.ran, "\n"))
+	}
+	if slices.ContainsFunc(h.ran, func(c string) bool { return strings.HasPrefix(c, "rm") || strings.Contains(c, "userdel -r") }) {
+		t.Error("a home was removed")
+	}
+}
+
+func TestAClientMountsAndLeavesOthersUsersAlone(t *testing.T) {
+	h := newFake()
+	inst := &fakeInst{host: h}
+	k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
+	m := nodeMsg(false)
+	m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Nfs: "10.0.0.11:/shared"}
+	bob := &llmgwv1.SlurmUser{Login: "bob", Uid: 200001, Home: "/shared/home/bob"}
+	m.ManageUsers, m.Users = true, []*llmgwv1.SlurmUser{ayse(), bob}
+	// The site's own ayse, with its own UID.
+	h.outputs["getent passwd ayse"] = "ayse:x:1001:1001::/home/ayse:/bin/bash\n"
+	st := k.Reconcile(context.Background(), m)
+	if st.GetShared() != SharedMounted || st.GetUsers() != 1 {
+		t.Fatalf("status %v", st)
+	}
+	if !strings.Contains(st.GetError(), "ayse is already a user here with UID 1001") {
+		t.Errorf("error %q", st.GetError())
+	}
+	if u := string(h.files["/etc/systemd/system/shared.mount"]); !strings.Contains(u, "What=10.0.0.11:/shared") {
+		t.Fatalf("unit:\n%s", u)
+	}
+	for _, want := range []string{"systemctl enable shared.mount", "systemctl restart shared.mount", "mountpoint -q /shared",
+		"useradd -u 200001 -g 200001 -M -d /shared/home/bob -s /bin/bash -c  bob"} {
+		if !slices.Contains(h.ran, want) {
+			t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
+		}
+	}
+	if slices.ContainsFunc(h.ran, func(c string) bool { return strings.Contains(c, "ayse") && !strings.HasPrefix(c, "getent") }) ||
+		slices.ContainsFunc(h.ran, func(c string) bool { return strings.HasPrefix(c, "install -d") }) {
+		t.Errorf("touched the site's ayse, or made a home:\n%s", strings.Join(h.ran, "\n"))
+	}
+	if string(h.files[k.usersMarker()]) != "bob\n" {
+		t.Errorf("marker %q", h.files[k.usersMarker()])
+	}
+
+	// Not mounted: said.
+	h.fail["mountpoint -q /shared"] = true
+	if st := k.Reconcile(context.Background(), m); st.GetShared() != SharedMissing ||
+		!strings.Contains(st.GetError(), "not mounted from 10.0.0.11:/shared") {
+		t.Errorf("status %v", st)
+	}
+
+	// No shared path any more: the mount is taken away.
+	h.outputs["systemctl list-units --all --plain --no-legend --type=mount"] = "shared.mount loaded active mounted llmgw\n"
+	m.Shared = nil
+	k.Reconcile(context.Background(), m)
+	if !slices.Contains(h.ran, "systemctl disable --now shared.mount") || h.files["/etc/systemd/system/shared.mount"] != nil {
+		t.Errorf("mount kept:\n%s", strings.Join(h.ran, "\n"))
+	}
+}
+
+const pamStack = "@include common-auth\naccount required pam_nologin.so\n@include common-account\n@include common-session\n"
+
+func TestUsersReachANodeOnlyThroughTheirJobs(t *testing.T) {
+	h := newFake()
+	inst := &fakeInst{host: h}
+	k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
+	h.files[slurm.PAMSSHD] = []byte(pamStack)
+	// The module is not there until its package is installed.
+	for _, d := range slurm.ModuleDirs {
+		h.fail["test -f "+d+"/"+slurm.AdoptModule] = true
+	}
+	m := nodeMsg(true)
+	m.ManageUsers, m.Users = true, []*llmgwv1.SlurmUser{ayse()}
+	k.Reconcile(context.Background(), m)
+	if !slices.Contains(inst.got["deb"], "libpam-slurm-adopt") {
+		t.Fatalf("not installed: %v", inst.got)
+	}
+	if got := string(h.files[slurm.PAMSSHD]); got != pamStack {
+		t.Fatalf("lines put in with the module still missing:\n%s", got)
+	}
+	delete(h.fail, "test -f "+slurm.ModuleDirs[0]+"/"+slurm.AdoptModule)
+	st := k.Reconcile(context.Background(), m)
+	got := string(h.files[slurm.PAMSSHD])
+	if !strings.Contains(got, "pam_listfile.so item=user sense=allow file="+k.usersMarker()) ||
+		!strings.Contains(got, "account required "+slurm.AdoptModule) {
+		t.Fatalf("not held to their jobs (%s):\n%s", st.GetError(), got)
+	}
+
+	// The controller is where they log in: not held there.
+	m.Controller = true
+	m.Files = map[string]string{"slurm.conf": "x"}
+	k.Reconcile(context.Background(), m)
+	if string(h.files[slurm.PAMSSHD]) != pamStack {
+		t.Errorf("held on the controller:\n%s", h.files[slurm.PAMSSHD])
+	}
+
+	// No users of llmgw's any more: the lines go.
+	m.Controller, m.Files = false, nil
+	k.Reconcile(context.Background(), m)
+	m.Users = nil
+	k.Reconcile(context.Background(), m)
+	if string(h.files[slurm.PAMSSHD]) != pamStack {
+		t.Errorf("lines left with no users:\n%s", h.files[slurm.PAMSSHD])
+	}
+}

internal/agent/slurmnode/slurmnode.go

@@ -0,0 +1,747 @@
+// Package slurmnode keeps this machine's part in the Slurm cluster run beside
+// llmgw (BACKLOG P), as the leader says: slurmctld on the controller, slurmd
+// on a node, munge on both, installed from the machine's own package sources.
+//
+// What the leader sends is wanted state, kept by Keeper's loop: sent again,
+// or checked again a minute later, it changes nothing that is already so.
+package slurmnode
+
+import (
+	"bytes"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"io"
+	"io/fs"
+	"log/slog"
+	"net/http"
+	"os"
+	"os/exec"
+	"os/user"
+	"path/filepath"
+	"slices"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"google.golang.org/protobuf/proto"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
+)
+
+// Host is what Keeper does to the machine; Local in the agent, a fake in
+// tests.
+type Host interface {
+	Run(ctx context.Context, env []string, name string, args ...string) ([]byte, error)
+	LookPath(name string) (string, error)
+	ReadFile(path string) ([]byte, error)
+	// WriteFile writes b at path with mode, owned by owner ("" for root),
+	// making its directory.
+	WriteFile(path string, b []byte, mode fs.FileMode, owner string) error
+	// MkdirAll makes path, owned by owner.
+	MkdirAll(path string, mode fs.FileMode, owner string) error
+	Remove(path string) error
+	// Download fetches url to path, kept only if its SHA-256 is sha; a file
+	// already there with it is not fetched again.
+	Download(ctx context.Context, url, path, sha string) error
+}
+
+// Installer installs packages by family, "deb" and "rpm".
+type Installer interface {
+	InstallPackages(ctx context.Context, byFamily map[string][]string) error
+}
+
+// Reasons llmgw drains a node with; one drained for another reason, by a
+// Slurm administrator, is never resumed by llmgw.
+const (
+	ReasonServing = "llmgw: serving models"
+	reasonPrefix  = "llmgw"
+)
+
+// DropIn is the unit drop-in that starts slurmd configless.
+const DropIn = "/etc/systemd/system/slurmd.service.d/llmgw.conf"
+
+// Keeper keeps the machine as the last Slurm message says.
+type Keeper struct {
+	host   Host
+	inst   Installer
+	log    *slog.Logger
+	marker string
+
+	mu   sync.Mutex
+	want *llmgwv1.Slurm
+	kick chan struct{}
+	// resend has the next status sent even if unchanged: a new stream's
+	// control plane may never have had it.
+	resend bool
+	// installed is true once the packages were found or put there, so a
+	// minute's check does not look again.
+	installed map[string]bool
+	files     string
+	// acctDone is the digest of the password the database's user was last
+	// given; clusterAdded the cluster last found in the accounting.
+	acctDone     string
+	clusterAdded string
+}
+
+// New is a Keeper; marker is a file it keeps while it runs Slurm here, so
+// an agent started again knows to stop what it started when told it has no
+// part any more.
+func New(host Host, inst Installer, marker string, log *slog.Logger) *Keeper {
+	if log == nil {
+		log = slog.Default()
+	}
+	return &Keeper{host: host, inst: inst, log: log, marker: marker,
+		kick: make(chan struct{}, 1), installed: map[string]bool{}}
+}
+
+// Apply is a new message from the leader: kept from now on.
+func (k *Keeper) Apply(m *llmgwv1.Slurm) {
+	k.mu.Lock()
+	k.want = proto.CloneOf(m)
+	k.mu.Unlock()
+	select {
+	case k.kick <- struct{}{}:
+	default:
+	}
+}
+
+// HasPart is whether the last message gives this machine a part in Slurm.
+func (k *Keeper) HasPart() bool {
+	k.mu.Lock()
+	defer k.mu.Unlock()
+	return k.want.GetController() || k.want.GetNode()
+}
+
+// Resend has the status sent again on the next pass, which it starts now.
+func (k *Keeper) Resend() {
+	k.mu.Lock()
+	k.resend = true
+	k.mu.Unlock()
+	select {
+	case k.kick <- struct{}{}:
+	default:
+	}
+}
+
+// Every is how often the machine is checked again without a new message.
+var Every = time.Minute
+
+// resendAfter is how often an unchanged status is sent anyway.
+const resendAfter = 5 * time.Minute
+
+// Run keeps the machine until ctx ends, sending its status when it changes
+// and every few minutes while it has a part.
+func (k *Keeper) Run(ctx context.Context, send func(*llmgwv1.SlurmStatus)) {
+	t := time.NewTicker(Every)
+	defer t.Stop()
+	var last *llmgwv1.SlurmStatus
+	var sent time.Time
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-k.kick:
+		case <-t.C:
+		}
+		k.mu.Lock()
+		want, again := k.want, k.resend
+		k.resend = false
+		k.mu.Unlock()
+		if want == nil {
+			continue
+		}
+		st := k.Reconcile(ctx, want)
+		if st == nil {
+			last = nil
+			continue
+		}
+		if again || !proto.Equal(st, last) || time.Since(sent) > resendAfter {
+			send(st)
+			last, sent = st, time.Now()
+		}
+	}
+}
+
+// Reconcile makes the machine what want says once, and answers its status;
+// nil when it has no part in Slurm.
+func (k *Keeper) Reconcile(ctx context.Context, want *llmgwv1.Slurm) *llmgwv1.SlurmStatus {
+	ctx, cancel := context.WithTimeout(ctx, 20*time.Minute)
+	defer cancel()
+	if !want.GetController() && !want.GetNode() {
+		k.stop(ctx)
+		return nil
+	}
+	st := &llmgwv1.SlurmStatus{}
+	fail := func(what string, err error) *llmgwv1.SlurmStatus {
+		st.Error = what + ": " + err.Error()
+		k.log.Warn("keeping Slurm on this node: "+what, "err", err)
+		return st
+	}
+	if err := k.ensurePackages(ctx, want); err != nil {
+		return fail("installing Slurm", err)
+	}
+	// From the daemon, not scontrol: a client command looks for its files
+	// even to say its version, and a configless node has none before slurmd
+	// has fetched them.
+	daemon := "slurmd"
+	if !want.GetNode() {
+		daemon = "slurmctld"
+	}
+	out, err := k.host.Run(ctx, nil, daemon, "-V")
+	if err != nil {
+		return fail("reading Slurm's version", fmt.Errorf("%v: %s", err, lastLine(out)))
+	}
+	v, err := slurm.ParseVersion(string(out))
+	if err != nil {
+		return fail("reading Slurm's version", err)
+	}
+	st.Version = strings.TrimSpace(string(out))
+	if i := strings.LastIndexByte(st.Version, ' '); i >= 0 {
+		st.Version = st.Version[i+1:]
+	}
+	if slurm.TooOld(v) {
+		return fail("Slurm", fmt.Errorf("%d.%02d is older than %d.%02d, the first to run jobs in cgroup v2; "+
+			"a newer release of the distribution, or Slurm's own packages, has one", v[0], v[1],
+			slurm.MinVersion[0], slurm.MinVersion[1]))
+	}
+	if err := k.host.WriteFile(k.marker, []byte("llmgw runs Slurm here (BACKLOG P)\n"), 0o644, ""); err != nil {
+		return fail("keeping its marker", err)
+	}
+	if err := k.munge(ctx, want.GetMungeKey()); err != nil {
+		return fail("munge", err)
+	}
+	// The shared files, then the users whose homes are on them. Neither
+	// keeps Slurm from running: what goes wrong is said, and Slurm kept.
+	var shErr, usErr error
+	st.Shared, shErr = k.shared(ctx, want)
+	if shErr != nil {
+		k.log.Warn("keeping the Slurm cluster's shared files on this node", "err", shErr)
+	}
+	if shErr == nil || !want.GetMakeHomes() || want.GetShared().GetPath() == "" {
+		st.Users, usErr = k.users(ctx, want)
+	} else {
+		// Homes made now would be made beside the shared files, not on them.
+		usErr = errors.New("waiting for the shared files, where the homes are")
+	}
+	if usErr != nil {
+		k.log.Warn("keeping the Slurm users on this node", "err", usErr)
+	}
+	// Jobs in containers; Slurm runs without them, and is kept anyway.
+	var ctErr error
+	if st.Containers, ctErr = k.containers(ctx, want); ctErr != nil {
+		k.log.Warn("setting up jobs in containers on this node", "err", ctErr)
+	}
+	defer func() {
+		switch {
+		case st.Error != "":
+		case shErr != nil:
+			st.Error = "shared files: " + shErr.Error()
+		case usErr != nil:
+			st.Error = "users: " + usErr.Error()
+		case ctErr != nil:
+			st.Error = "containers: " + ctErr.Error()
+		}
+	}()
+	if want.GetController() {
+		// Accounting first: slurmctld reports to it from its first start.
+		// Without it slurmctld still runs jobs, a pass of backfill late
+		// each; so a failure here is said, and slurmctld is kept anyway.
+		var acctErr error
+		if acctErr = k.accounting(ctx, want.GetMungeKey(), want.GetFiles()); acctErr != nil {
+			k.log.Warn("keeping Slurm's accounting on this node", "err", acctErr)
+		}
+		st.SlurmdbdRunning = k.active(ctx, "slurmdbd")
+		defer func() {
+			if acctErr != nil && st.Error == "" {
+				st.Error = "slurmdbd: " + acctErr.Error()
+			}
+		}()
+		sum, err := k.controller(ctx, want.GetFiles())
+		st.FilesSha256 = sum
+		if err != nil {
+			return fail("slurmctld", err)
+		}
+		st.SlurmctldRunning = k.active(ctx, "slurmctld")
+		// The accounts, and the jobs that ended, need the accounting.
+		if st.SlurmdbdRunning && st.SlurmctldRunning {
+			if err := k.accounts(ctx, want); err != nil {
+				k.log.Warn("keeping Slurm's accounts", "err", err)
+				if acctErr == nil {
+					acctErr = fmt.Errorf("accounts: %w", err)
+				}
+			}
+			if ended, err := k.ended(ctx); err != nil {
+				k.log.Warn("reading the jobs that ended", "err", err)
+			} else {
+				st.Ended = ended
+			}
+		}
+	}
+	if want.GetNode() {
+		if err := k.node(ctx, want, st); err != nil {
+			return fail("slurmd", err)
+		}
+	}
+	return st
+}
+
+func (k *Keeper) ensurePackages(ctx context.Context, want *llmgwv1.Slurm) error {
+	need := map[string][2]string{
+		"munged":   {"munge", "munge"},
+		"scontrol": {"slurm-client", "slurm"},
+	}
+	if want.GetController() {
+		need["slurmctld"] = [2]string{"slurmctld", "slurm-slurmctld"}
+		need["slurmdbd"] = [2]string{"slurmdbd", "slurm-slurmdbd"}
+		need["mariadbd"] = [2]string{"mariadb-server", "mariadb-server"}
+	}
+	if want.GetNode() {
+		need["slurmd"] = [2]string{"slurmd", "slurm-slurmd"}
+	}
+	sharedPackages(want, need)
+	pkgs := map[string][]string{}
+	for bin, p := range need {
+		if k.installed[bin] {
+			continue
+		}
+		if _, err := k.host.LookPath(bin); err == nil {
+			k.installed[bin] = true
+			continue
+		}
+		pkgs["deb"] = append(pkgs["deb"], p[0])
+		pkgs["rpm"] = append(pkgs["rpm"], p[1])
+	}
+	if len(pkgs) == 0 {
+		return nil
+	}
+	slices.Sort(pkgs["deb"])
+	slices.Sort(pkgs["rpm"])
+	if k.inst == nil {
+		return fmt.Errorf("%s missing, and this agent installs nothing", strings.Join(pkgs["deb"], ", "))
+	}
+	k.log.Info("installing Slurm, as the cluster's settings say", "packages", pkgs["deb"])
+	if err := k.inst.InstallPackages(ctx, pkgs); err != nil {
+		return err
+	}
+	if slices.Contains(pkgs["deb"], "mariadb-server") {
+		// Installed by llmgw: its settings are llmgw's to keep. One that was
+		// here before is another's, and is left as it is.
+		if err := k.host.WriteFile(k.mariaDBMarker(), []byte("llmgw installed MariaDB here, for Slurm's accounting\n"),
+			0o644, ""); err != nil {
+			return err
+		}
+	}
+	for bin := range need {
+		if _, err := k.host.LookPath(bin); err != nil {
+			return fmt.Errorf("%s is still missing after installing %s", bin, strings.Join(pkgs["deb"], ", "))
+		}
+		k.installed[bin] = true
+	}
+	return nil
+}
+
+// munge writes the cluster's key and keeps munged running on it.
+func (k *Keeper) munge(ctx context.Context, key []byte) error {
+	if len(key) < 32 {
+		return errors.New("the leader sent no key")
+	}
+	changed, err := k.write(slurm.MungeKeyPath, key, 0o400, "munge")
+	if err != nil {
+		return err
+	}
+	if changed {
+		k.log.Info("munge key written; munge starts again on it")
+		return k.systemctl(ctx, "enable", "--now", "munge", "&&restart")
+	}
+	if !k.active(ctx, "munge") {
+		return k.systemctl(ctx, "enable", "--now", "munge")
+	}
+	return nil
+}
+
+// controller writes the files and keeps slurmctld running on them,
+// answering their digest.
+func (k *Keeper) controller(ctx context.Context, files map[string]string) (string, error) {
+	if files["slurm.conf"] == "" {
+		return "", errors.New("the leader sent no slurm.conf")
+	}
+	for _, d := range []string{slurm.StateDir, slurm.LogDir} {
+		if err := k.host.MkdirAll(d, 0o755, "slurm"); err != nil {
+			return "", err
+		}
+	}
+	names := make([]string, 0, len(files))
+	for n := range files {
+		names = append(names, n)
+	}
+	slices.Sort(names)
+	h := sha256.New()
+	changed := false
+	for _, n := range names {
+		if strings.ContainsAny(n, "/\\") || n == "" || n == "." || n == ".." {
+			return "", fmt.Errorf("the leader sent a file named %q", n)
+		}
+		fmt.Fprintf(h, "%s\x00%s\x00", n, files[n])
+		c, err := k.write(filepath.Join(slurm.ConfDir, n), []byte(files[n]), 0o644, "")
+		if err != nil {
+			return "", err
+		}
+		changed = changed || c
+	}
+	sum := hex.EncodeToString(h.Sum(nil))
+	if changed || sum != k.files {
+		if err := k.systemctl(ctx, "enable", "slurmctld"); err != nil {
+			return sum, err
+		}
+		// Nodes added or taken away need a restart before 23.11 and a
+		// reconfigure after it; a restart does both, and running jobs live
+		// through it.
+		k.log.Info("slurmctld starts again on new files")
+		if err := k.systemctl(ctx, "restart", "slurmctld"); err != nil {
+			return sum, err
+		}
+		if err := k.waitPing(ctx); err != nil {
+			return sum, err
+		}
+		// Hands the configless slurmd the new files.
+		if out, err := k.host.Run(ctx, nil, "scontrol", "reconfigure"); err != nil {
+			return sum, fmt.Errorf("scontrol reconfigure: %v: %s", err, lastLine(out))
+		}
+		k.files = sum
+		return sum, nil
+	}
+	if !k.active(ctx, "slurmctld") {
+		return sum, k.systemctl(ctx, "enable", "--now", "slurmctld")
+	}
+	return sum, nil
+}
+
+func (k *Keeper) waitPing(ctx context.Context) error {
+	var last []byte
+	for range 30 {
+		out, err := k.host.Run(ctx, nil, "scontrol", "ping")
+		if err == nil && bytes.Contains(out, []byte("UP")) {
+			return nil
+		}
+		last = out
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-time.After(pingWait):
+		}
+	}
+	return fmt.Errorf("slurmctld did not answer after starting: %s", lastLine(last))
+}
+
+var pingWait = time.Second
+
+// node keeps slurmd running configless and puts the node in or out of the
+// controller's hands as lent says.
+func (k *Keeper) node(ctx context.Context, want *llmgwv1.Slurm, st *llmgwv1.SlurmStatus) error {
+	name, server := want.GetName(), want.GetConfServer()
+	if name == "" || server == "" {
+		return errors.New("the leader sent no name or controller")
+	}
+	if strings.ContainsAny(name+server, " \n\t\"'\\$`") {
+		return fmt.Errorf("the leader sent a name or controller with spaces or quotes in it")
+	}
+	for _, d := range []string{slurm.SpoolDir, slurm.LogDir} {
+		if err := k.host.MkdirAll(d, 0o755, "slurm"); err != nil {
+			return err
+		}
+	}
+	bin, err := k.host.LookPath("slurmd")
+	if err != nil {
+		return err
+	}
+	unit := "# Written by llmgw (BACKLOG P): slurmd fetches its files from slurmctld.\n" +
+		"[Unit]\nConditionPathExists=\nStartLimitIntervalSec=0\n" +
+		// --systemd as the packaged unit has it (Type=notify, since 22.05):
+		// without it systemd waits for a readiness that never comes.
+		"[Service]\nExecStart=\nExecStart=" + bin + " --systemd --conf-server " + server + " -N " + name + "\n" +
+		// A node new to the controller is refused until the leader has
+		// written it into slurm.conf; tried again until it is.
+		"Restart=on-failure\nRestartSec=15\n"
+	changed, err := k.write(DropIn, []byte(unit), 0o644, "")
+	if err != nil {
+		return err
+	}
+	if out, err := k.host.Run(ctx, nil, "slurmd", "-C"); err == nil {
+		st.Hardware = strings.TrimSpace(string(out))
+	}
+	if changed {
+		if err := k.systemctl(ctx, "daemon-reload"); err != nil {
+			return err
+		}
+		if err := k.systemctl(ctx, "enable", "slurmd"); err != nil {
+			return err
+		}
+		if err := k.systemctl(ctx, "restart", "slurmd"); err != nil {
+			return err
+		}
+	} else if !k.active(ctx, "slurmd") {
+		if err := k.systemctl(ctx, "enable", "--now", "slurmd"); err != nil {
+			return err
+		}
+	}
+	st.SlurmdRunning = k.active(ctx, "slurmd")
+
+	st.Jobs = k.localJobs(ctx)
+	env := k.clientEnv()
+	out, err := k.host.Run(ctx, env, "scontrol", "show", "node", "-o", name)
+	if err != nil {
+		// Not in slurm.conf yet: the leader writes it once this node's
+		// hardware is in.
+		return nil
+	}
+	ns, err := slurm.ParseNodeState(string(out))
+	if err != nil {
+		return nil
+	}
+	ours := ns.Reason == "" || strings.HasPrefix(ns.Reason, reasonPrefix)
+	switch {
+	case want.GetLent() && ns.Drained() && ours:
+		k.log.Info("this node is lent to Slurm: resumed there")
+		if out, err := k.host.Run(ctx, env, "scontrol", "update", "NodeName="+name, "State=RESUME"); err != nil {
+			return fmt.Errorf("resuming the node: %v: %s", err, lastLine(out))
+		}
+	case !want.GetLent() && !strings.Contains(ns.State, "DRAIN"):
+		k.log.Info("this node is llmgw's: drained in Slurm, its jobs left to finish")
+		if out, err := k.host.Run(ctx, env, "scontrol", "update", "NodeName="+name, "State=DRAIN",
+			"Reason="+ReasonServing); err != nil {
+			return fmt.Errorf("draining the node: %v: %s", err, lastLine(out))
+		}
+	}
+	if out, err := k.host.Run(ctx, env, "scontrol", "show", "node", "-o", name); err == nil {
+		if ns2, err := slurm.ParseNodeState(string(out)); err == nil {
+			ns = ns2
+		}
+	}
+	st.State, st.Reason = ns.State, ns.Reason
+	return nil
+}
+
+// localJobs counts the job steps running here, from their slurmstepd, not
+// from the controller: a job outlives a controller that is down, and is
+// what decides whether llmgw may have the cards back. The one slurmstepd
+// that is no job's, "slurmstepd infinity" holding slurmd's cgroup v2 scope,
+// is not counted. -1 when it cannot tell.
+func (k *Keeper) localJobs(ctx context.Context) int32 {
+	// pgrep exits 1 with no output when there is none.
+	out, err := k.host.Run(ctx, nil, "pgrep", "-a", "-x", "slurmstepd")
+	if err != nil && len(bytes.TrimSpace(out)) > 0 {
+		return -1
+	}
+	n := int32(0)
+	for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
+		f := strings.Fields(line)
+		if len(f) < 2 || f[len(f)-1] == "infinity" {
+			continue
+		}
+		n++
+	}
+	return n
+}
+
+// clientEnv points scontrol at slurmd's copy of the files on a node that is
+// not the controller, which has none of its own.
+func (k *Keeper) clientEnv() []string {
+	if _, err := k.host.ReadFile(filepath.Join(slurm.ConfDir, "slurm.conf")); err == nil {
+		return nil
+	}
+	return []string{"SLURM_CONF=" + filepath.Join(slurm.SpoolDir, "conf-cache", "slurm.conf")}
+}
+
+// stop stops what llmgw started here, if it did.
+func (k *Keeper) stop(ctx context.Context) {
+	if _, err := k.host.ReadFile(k.marker); err != nil {
+		return
+	}
+	k.log.Info("this node has no part in Slurm any more: slurmd, slurmctld and slurmdbd stopped; the packages and MariaDB stay")
+	// MariaDB stays: it may have been the machine's before, and holds the
+	// jobs' records.
+	for _, u := range []string{"slurmd", "slurmctld", "slurmdbd"} {
+		_ = k.systemctl(ctx, "disable", "--now", u)
+	}
+	_ = k.host.Remove(DropIn)
+	_ = k.systemctl(ctx, "daemon-reload")
+	k.unshare(ctx, "", false, "")
+	if err := k.removeUsers(ctx, k.madeUsers(), nil); err != nil {
+		k.log.Warn("could not take away every Slurm user llmgw made", "err", err)
+	}
+	_ = k.host.Remove(k.marker)
+	k.files, k.acctDone, k.clusterAdded = "", "", ""
+}
+
+// write writes a file when it differs, saying whether it did.
+func (k *Keeper) write(path string, b []byte, mode fs.FileMode, owner string) (bool, error) {
+	if old, err := k.host.ReadFile(path); err == nil && bytes.Equal(old, b) {
+		return false, nil
+	}
+	return true, k.host.WriteFile(path, b, mode, owner)
+}
+
+// systemctl runs systemctl; a last argument "&&restart" restarts the unit
+// after.
+func (k *Keeper) systemctl(ctx context.Context, args ...string) error {
+	restart := false
+	if n := len(args); n > 0 && args[n-1] == "&&restart" {
+		args, restart = args[:n-1], true
+	}
+	if out, err := k.host.Run(ctx, nil, "systemctl", args...); err != nil {
+		return fmt.Errorf("systemctl %s: %v: %s", strings.Join(args, " "), err, lastLine(out))
+	}
+	if restart {
+		u := args[len(args)-1]
+		if out, err := k.host.Run(ctx, nil, "systemctl", "restart", u); err != nil {
+			return fmt.Errorf("systemctl restart %s: %v: %s", u, err, lastLine(out))
+		}
+	}
+	return nil
+}
+
+func (k *Keeper) active(ctx context.Context, unit string) bool {
+	out, err := k.host.Run(ctx, nil, "systemctl", "is-active", unit)
+	return err == nil && strings.TrimSpace(string(out)) == "active"
+}
+
+func lastLine(b []byte) string {
+	s := strings.TrimSpace(string(b))
+	if i := strings.LastIndexByte(s, '\n'); i >= 0 {
+		s = s[i+1:]
+	}
+	return s
+}
+
+// Local is the machine the agent runs on.
+type Local struct{}
+
+// Run runs a command, with env added to the agent's.
+func (Local) Run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
+	cmd := exec.CommandContext(ctx, name, args...)
+	if len(env) > 0 {
+		cmd.Env = append(os.Environ(), env...)
+	}
+	return cmd.CombinedOutput()
+}
+
+// LookPath finds a command, in the sbin directories too: an agent started
+// by systemd has them, one started by hand may not.
+func (Local) LookPath(name string) (string, error) {
+	if p, err := exec.LookPath(name); err == nil {
+		return p, nil
+	}
+	// /usr/libexec: Red Hat's mariadbd.
+	for _, d := range []string{"/usr/sbin", "/sbin", "/usr/local/sbin", "/usr/libexec"} {
+		p := filepath.Join(d, name)
+		if fi, err := os.Stat(p); err == nil && !fi.IsDir() {
+			return p, nil
+		}
+	}
+	return "", fmt.Errorf("%s is not installed", name)
+}
+
+// ReadFile reads a file.
+func (Local) ReadFile(path string) ([]byte, error) { return os.ReadFile(path) } //nolint:gosec // our paths
+
+// WriteFile writes a file whole, through a temporary one beside it.
+func (Local) WriteFile(path string, b []byte, mode fs.FileMode, owner string) error {
+	if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { //nolint:gosec // system directories
+		return err
+	}
+	tmp := path + ".llmgw-new"
+	if err := os.WriteFile(tmp, b, mode); err != nil {
+		return err
+	}
+	if err := os.Chmod(tmp, mode); err != nil {
+		return err
+	}
+	if err := chown(tmp, owner); err != nil {
+		_ = os.Remove(tmp)
+		return err
+	}
+	return os.Rename(tmp, path)
+}
+
+// Download fetches url to path through a temporary file beside it, kept
+// only when its SHA-256 is sha.
+func (Local) Download(ctx context.Context, url, path, sha string) error {
+	if f, err := os.Open(path); err == nil { //nolint:gosec // our paths
+		h := sha256.New()
+		_, cerr := io.Copy(h, f)
+		_ = f.Close()
+		if cerr == nil && hex.EncodeToString(h.Sum(nil)) == sha {
+			return nil
+		}
+	}
+	if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { //nolint:gosec // system directories
+		return err
+	}
+	ctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
+	defer cancel()
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return err
+	}
+	resp, err := http.DefaultClient.Do(req)
+	if err != nil {
+		return fmt.Errorf("fetching %s: %w", url, err)
+	}
+	defer func() { _ = resp.Body.Close() }()
+	if resp.StatusCode != http.StatusOK {
+		return fmt.Errorf("fetching %s: %s", url, resp.Status)
+	}
+	tmp := path + ".llmgw-new"
+	f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o644) //nolint:gosec // our paths
+	if err != nil {
+		return err
+	}
+	h := sha256.New()
+	_, err = io.Copy(io.MultiWriter(f, h), io.LimitReader(resp.Body, 1<<30))
+	if cerr := f.Close(); err == nil {
+		err = cerr
+	}
+	if err != nil {
+		_ = os.Remove(tmp)
+		return fmt.Errorf("fetching %s: %w", url, err)
+	}
+	if got := hex.EncodeToString(h.Sum(nil)); got != sha {
+		_ = os.Remove(tmp)
+		return fmt.Errorf("%s has SHA-256 %s, not the %s llmgw was built with: not used", url, got, sha)
+	}
+	return os.Rename(tmp, path)
+}
+
+// MkdirAll makes a directory owned by owner.
+func (Local) MkdirAll(path string, mode fs.FileMode, owner string) error {
+	if err := os.MkdirAll(path, mode); err != nil {
+		return err
+	}
+	return chown(path, owner)
+}
+
+// Remove removes a file, and is content when there is none.
+func (Local) Remove(path string) error {
+	if err := os.Remove(path); err != nil && !errors.Is(err, fs.ErrNotExist) {
+		return err
+	}
+	return nil
+}
+
+func chown(path, owner string) error {
+	if owner == "" {
+		return nil
+	}
+	u, err := user.Lookup(owner)
+	if err != nil {
+		return fmt.Errorf("user %s: %w (installed with its package)", owner, err)
+	}
+	uid, _ := strconv.Atoi(u.Uid)
+	gid, _ := strconv.Atoi(u.Gid)
+	return os.Chown(path, uid, gid)
+}

internal/agent/slurmnode/slurmnode_test.go

@@ -0,0 +1,367 @@
+package slurmnode
+
+import (
+	"context"
+	"errors"
+	"io/fs"
+	"path/filepath"
+	"slices"
+	"strings"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
+)
+
+type fakeHost struct {
+	files   map[string][]byte
+	owners  map[string]string
+	bins    map[string]bool
+	active  map[string]bool
+	ran     []string
+	outputs map[string]string
+	fail    map[string]bool
+	state   string
+	reason  string
+}
+
+func newFake() *fakeHost {
+	return &fakeHost{files: map[string][]byte{}, owners: map[string]string{}, bins: map[string]bool{},
+		active: map[string]bool{}, outputs: map[string]string{
+			"slurmd -V":     "slurm-wlm 23.11.4\n",
+			"slurmctld -V":  "slurm-wlm 23.11.4\n",
+			"scontrol ping": "Slurmctld(primary) at node06 is UP\n",
+			"slurmd -C":     "NodeName=cuda01 CPUs=16 RealMemory=64000\nUpTime=1-00:00:00\n",
+			"hostname -s":   "node06\n",
+		}, fail: map[string]bool{}, state: "IDLE+DRAIN", reason: slurm.ReasonNew}
+}
+
+func (f *fakeHost) Run(_ context.Context, env []string, name string, args ...string) ([]byte, error) {
+	c := strings.Join(append([]string{name}, args...), " ")
+	f.ran = append(f.ran, c)
+	if f.fail[c] {
+		return []byte("it failed"), errors.New("exit status 1")
+	}
+	switch {
+	case name == "systemctl" && len(args) >= 1:
+		u := args[len(args)-1]
+		switch args[0] {
+		case "is-active":
+			if f.active[u] {
+				return []byte("active\n"), nil
+			}
+			return []byte("inactive\n"), errors.New("exit status 3")
+		case "restart", "start":
+			f.active[u] = true
+		case "enable":
+			if slices.Contains(args, "--now") {
+				f.active[u] = true
+			}
+		case "disable":
+			f.active[u] = false
+		}
+		return []byte(f.outputs[c]), nil
+	case strings.HasPrefix(c, "scontrol show node"):
+		return []byte("NodeName=cuda01 Arch=x86_64 State=" + f.state + " Partitions=gpu Reason=" + f.reason + " [root@now]\n"), nil
+	case strings.HasPrefix(c, "scontrol update"):
+		if slices.Contains(args, "State=RESUME") {
+			f.state, f.reason = "IDLE", ""
+		} else {
+			f.state, f.reason = "IDLE+DRAIN", ReasonServing
+		}
+		return nil, nil
+	case strings.HasPrefix(c, "pgrep"):
+		out := "1800358 /usr/sbin/slurmstepd infinity\n" + f.outputs["jobs"]
+		return []byte(out), nil
+	}
+	return []byte(f.outputs[c]), nil
+}
+
+func (f *fakeHost) LookPath(name string) (string, error) {
+	if f.bins[name] {
+		return "/usr/sbin/" + name, nil
+	}
+	return "", errors.New("not found")
+}
+
+func (f *fakeHost) ReadFile(path string) ([]byte, error) {
+	b, ok := f.files[path]
+	if !ok {
+		return nil, fs.ErrNotExist
+	}
+	return b, nil
+}
+
+func (f *fakeHost) WriteFile(path string, b []byte, _ fs.FileMode, owner string) error {
+	f.files[path] = append([]byte(nil), b...)
+	f.owners[path] = owner
+	return nil
+}
+
+func (f *fakeHost) MkdirAll(string, fs.FileMode, string) error { return nil }
+
+func (f *fakeHost) Remove(path string) error { delete(f.files, path); return nil }
+
+func (f *fakeHost) Download(_ context.Context, url, path, sha string) error {
+	c := "download " + url + " " + path + " " + sha
+	f.ran = append(f.ran, c)
+	if f.fail[c] || f.fail["download "+url] {
+		return errors.New("fetch failed")
+	}
+	f.files[path] = []byte("fetched " + url)
+	return nil
+}
+
+func (f *fakeHost) count(prefix string) int {
+	n := 0
+	for _, c := range f.ran {
+		if strings.HasPrefix(c, prefix) {
+			n++
+		}
+	}
+	return n
+}
+
+type fakeInst struct {
+	host *fakeHost
+	got  map[string][]string
+}
+
+func (i *fakeInst) InstallPackages(_ context.Context, by map[string][]string) error {
+	i.got = by
+	for _, b := range []string{"munged", "scontrol", "slurmd", "slurmctld", "slurmdbd", "mariadbd", "mariadb", "exportfs", "mount.nfs"} {
+		i.host.bins[b] = true
+	}
+	return nil
+}
+
+var key = []byte(strings.Repeat("k", 1024))
+
+func nodeMsg(lent bool) *llmgwv1.Slurm {
+	return &llmgwv1.Slurm{Node: true, Name: "cuda01", ConfServer: "10.0.0.146:6817", MungeKey: key, Lent: lent}
+}
+
+func TestNodeInstallsAndStaysDrained(t *testing.T) {
+	h := newFake()
+	inst := &fakeInst{host: h}
+	k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
+	st := k.Reconcile(context.Background(), nodeMsg(false))
+	if st.GetError() != "" {
+		t.Fatal(st.GetError())
+	}
+	if !slices.Equal(inst.got["deb"], []string{"munge", "slurm-client", "slurmd"}) ||
+		!slices.Equal(inst.got["rpm"], []string{"munge", "slurm", "slurm-slurmd"}) {
+		t.Fatalf("installed %v", inst.got)
+	}
+	if string(h.files[slurm.MungeKeyPath]) != string(key) || h.owners[slurm.MungeKeyPath] != "munge" {
+		t.Fatal("the munge key was not written for munge")
+	}
+	unit := string(h.files[DropIn])
+	if !strings.Contains(unit, "ExecStart=/usr/sbin/slurmd --systemd --conf-server 10.0.0.146:6817 -N cuda01\n") ||
+		!strings.Contains(unit, "ConditionPathExists=\n") {
+		t.Fatalf("drop-in:\n%s", unit)
+	}
+	if !st.GetSlurmdRunning() || st.GetVersion() != "23.11.4" || !strings.HasPrefix(st.GetHardware(), "NodeName=cuda01 CPUs=16") {
+		t.Fatalf("status %v", st)
+	}
+	// Not lent: left drained, with llmgw's first reason.
+	if h.count("scontrol update") != 0 || st.GetState() != "IDLE+DRAIN" {
+		t.Fatalf("a node not lent was changed: %v, %v", h.ran, st)
+	}
+	if _, ok := h.files["/var/lib/llmgw-agent/slurm"]; !ok {
+		t.Fatal("no marker")
+	}
+
+	// Again, nothing changed: no install, no restart.
+	h.ran = nil
+	inst.got = nil
+	k.Reconcile(context.Background(), nodeMsg(false))
+	if inst.got != nil || h.count("systemctl restart") != 0 || h.count("systemctl enable") != 0 {
+		t.Fatalf("a second pass did work: %v", h.ran)
+	}
+}
+
+func TestLendAndReclaim(t *testing.T) {
+	h := newFake()
+	k := New(h, &fakeInst{host: h}, "/m", nil)
+	st := k.Reconcile(context.Background(), nodeMsg(true))
+	if h.count("scontrol update NodeName=cuda01 State=RESUME") != 1 || st.GetState() != "IDLE" {
+		t.Fatalf("a lent node was not resumed: %v %v", h.ran, st)
+	}
+	h.outputs["jobs"] = "2001 slurmstepd: [101.batch]\n2002 slurmstepd: [102.0]\n"
+	st = k.Reconcile(context.Background(), nodeMsg(false))
+	if h.count("scontrol update NodeName=cuda01 State=DRAIN Reason="+ReasonServing) != 1 {
+		t.Fatalf("a node taken back was not drained: %v", h.ran)
+	}
+	if st.GetJobs() != 2 || !strings.Contains(st.GetState(), "DRAIN") {
+		t.Fatalf("status %v", st)
+	}
+	// Drained already: not drained again.
+	h.ran = nil
+	k.Reconcile(context.Background(), nodeMsg(false))
+	if h.count("scontrol update") != 0 {
+		t.Fatalf("drained twice: %v", h.ran)
+	}
+}
+
+func TestAdminsDrainIsLeft(t *testing.T) {
+	h := newFake()
+	h.reason = "bad DIMM, ticket 42"
+	k := New(h, &fakeInst{host: h}, "/m", nil)
+	k.Reconcile(context.Background(), nodeMsg(true))
+	if h.count("scontrol update") != 0 {
+		t.Fatalf("llmgw resumed a node a Slurm administrator drained: %v", h.ran)
+	}
+}
+
+func TestClientEnvOnANodeWithoutFiles(t *testing.T) {
+	h := newFake()
+	k := New(h, &fakeInst{host: h}, "/m", nil)
+	if env := k.clientEnv(); len(env) != 1 || !strings.HasSuffix(env[0], "conf-cache/slurm.conf") {
+		t.Fatalf("env %v", env)
+	}
+	h.files[filepath.Join(slurm.ConfDir, "slurm.conf")] = []byte("x")
+	if env := k.clientEnv(); env != nil {
+		t.Fatalf("env %v on the controller", env)
+	}
+}
+
+func TestController(t *testing.T) {
+	h := newFake()
+	inst := &fakeInst{host: h}
+	k := New(h, inst, "/m", nil)
+	files := map[string]string{"slurm.conf": "ClusterName=llmgw\n", "gres.conf": "#\n", "cgroup.conf": "#\n"}
+	m := &llmgwv1.Slurm{Controller: true, MungeKey: key, Files: files}
+	st := k.Reconcile(context.Background(), m)
+	if st.GetError() != "" {
+		t.Fatal(st.GetError())
+	}
+	if !slices.Contains(inst.got["deb"], "slurmctld") || slices.Contains(inst.got["deb"], "slurmd") {
+		t.Fatalf("installed %v", inst.got)
+	}
+	if string(h.files["/etc/slurm/slurm.conf"]) != "ClusterName=llmgw\n" {
+		t.Fatal("slurm.conf not written")
+	}
+	if h.count("systemctl restart slurmctld") != 1 || h.count("scontrol reconfigure") != 1 {
+		t.Fatalf("slurmctld not started on its files: %v", h.ran)
+	}
+	if !st.GetSlurmctldRunning() || len(st.GetFilesSha256()) != 64 {
+		t.Fatalf("status %v", st)
+	}
+	// Accounting: MariaDB on this machine alone, the database, slurmdbd on
+	// it, the cluster added, all before slurmctld.
+	if !slices.Contains(inst.got["deb"], "slurmdbd") || !slices.Contains(inst.got["deb"], "mariadb-server") {
+		t.Fatalf("installed %v", inst.got)
+	}
+	if !strings.Contains(string(h.files["/etc/mysql/mariadb.conf.d/90-llmgw-slurm.cnf"]), "bind-address=127.0.0.1") {
+		t.Fatal("MariaDB, installed by llmgw, not kept to this machine")
+	}
+	dbd := string(h.files[slurm.DbdConfPath])
+	if !strings.Contains(dbd, "DbdHost=node06\n") || !strings.Contains(dbd, "StoragePass="+slurm.DbdPassword(key)+"\n") ||
+		h.owners[slurm.DbdConfPath] != "slurm" {
+		t.Fatalf("slurmdbd.conf for %q:\n%s", h.owners[slurm.DbdConfPath], dbd)
+	}
+	if _, ok := h.files[acctSQLPath]; ok {
+		t.Fatal("the database's statements, password and all, were left on disk")
+	}
+	if strings.Contains(strings.Join(h.ran, "\n"), slurm.DbdPassword(key)) {
+		t.Fatal("the password was on a command line")
+	}
+	order := strings.Join(h.ran, "\n")
+	if !strings.Contains(order, "sacctmgr -i add cluster llmgw") ||
+		strings.Index(order, "slurmdbd") > strings.Index(order, "systemctl restart slurmctld") {
+		t.Fatalf("accounting not before slurmctld: %v", h.ran)
+	}
+	if !st.GetSlurmdbdRunning() {
+		t.Fatalf("status %v", st)
+	}
+	sum := st.GetFilesSha256()
+
+	h.ran = nil
+	k.Reconcile(context.Background(), m)
+	if h.count("systemctl restart") != 0 {
+		t.Fatalf("restarted on the same files: %v", h.ran)
+	}
+
+	files["gres.conf"] = "NodeName=cuda01 Name=gpu File=/dev/nvidia0\n"
+	h.ran = nil
+	st = k.Reconcile(context.Background(), m)
+	if h.count("systemctl restart slurmctld") != 1 || st.GetFilesSha256() == sum {
+		t.Fatalf("a changed file did not restart slurmctld: %v", h.ran)
+	}
+
+	if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Controller: true, MungeKey: key,
+		Files: map[string]string{"slurm.conf": "x", "../evil": "y"}}); !strings.Contains(st.GetError(), "named") {
+		t.Fatalf("a path in a file name was taken: %v", st)
+	}
+}
+
+func TestTooOld(t *testing.T) {
+	h := newFake()
+	h.outputs["slurmd -V"] = "slurm-wlm 21.08.5\n"
+	k := New(h, &fakeInst{host: h}, "/m", nil)
+	st := k.Reconcile(context.Background(), nodeMsg(false))
+	if !strings.Contains(st.GetError(), "older than 22.05") {
+		t.Fatalf("status %v", st)
+	}
+	if _, ok := h.files[slurm.MungeKeyPath]; ok {
+		t.Fatal("went on with a Slurm too old")
+	}
+}
+
+func TestStop(t *testing.T) {
+	h := newFake()
+	k := New(h, &fakeInst{host: h}, "/m", nil)
+	k.Reconcile(context.Background(), nodeMsg(false))
+	h.ran = nil
+	if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{}); st != nil {
+		t.Fatalf("status %v for no part", st)
+	}
+	if !slices.Contains(h.ran, "systemctl disable --now slurmd") {
+		t.Fatalf("slurmd not stopped: %v", h.ran)
+	}
+	if _, ok := h.files[DropIn]; ok {
+		t.Fatal("drop-in left")
+	}
+	// Never started here: nothing touched.
+	h2 := newFake()
+	k2 := New(h2, &fakeInst{host: h2}, "/m", nil)
+	k2.Reconcile(context.Background(), &llmgwv1.Slurm{})
+	if len(h2.ran) != 0 {
+		t.Fatalf("a machine llmgw never ran Slurm on was touched: %v", h2.ran)
+	}
+}
+
+func TestBadNames(t *testing.T) {
+	h := newFake()
+	k := New(h, &fakeInst{host: h}, "/m", nil)
+	m := nodeMsg(false)
+	m.Name = "cuda01 -c /etc/evil"
+	if st := k.Reconcile(context.Background(), m); !strings.Contains(st.GetError(), "spaces") {
+		t.Fatalf("status %v", st)
+	}
+	if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Node: true, Name: "a", ConfServer: "b"}); !strings.Contains(st.GetError(), "no key") {
+		t.Fatalf("status %v", st)
+	}
+}
+
+func TestAMariaDBThatWasThereKeepsItsSettings(t *testing.T) {
+	h := newFake()
+	for _, b := range []string{"munged", "scontrol", "slurmctld", "slurmdbd", "mariadbd", "mariadb", "exportfs", "mount.nfs"} {
+		h.bins[b] = true
+	}
+	k := New(h, &fakeInst{host: h}, "/m", nil)
+	st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Controller: true, MungeKey: key,
+		Files: map[string]string{"slurm.conf": "ClusterName=llmgw\n"}})
+	if st.GetError() != "" {
+		t.Fatal(st.GetError())
+	}
+	for p := range h.files {
+		if strings.HasSuffix(p, ".cnf") {
+			t.Fatalf("a MariaDB llmgw did not install had its settings changed: %s", p)
+		}
+	}
+	if h.count("mariadb -e source") != 1 || !st.GetSlurmdbdRunning() {
+		t.Fatalf("the database was not made in the MariaDB that was there: %v", h.ran)
+	}
+}

internal/agent/slurmnode/users.go

@@ -0,0 +1,319 @@
+package slurmnode
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"path/filepath"
+	"slices"
+	"strconv"
+	"strings"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
+)
+
+// usersMarker lists, a line each, the logins llmgw made here: the ones it
+// may change and take away. A user that was here before is never touched.
+func (k *Keeper) usersMarker() string { return k.marker + "-users" }
+
+// users keeps the Linux users want has, the same login and UID as on every
+// other machine, their keys where sshd reads them, and their homes when
+// this machine makes them; and takes away those llmgw made that are gone.
+// Answers how many it has. A user that cannot be made is said, the others
+// are made anyway.
+func (k *Keeper) users(ctx context.Context, want *llmgwv1.Slurm) (int32, error) {
+	made := k.madeUsers()
+	if !want.GetManageUsers() {
+		return 0, k.removeUsers(ctx, made, nil)
+	}
+	var errs []error
+	var keep []string
+	n := int32(0)
+	for _, u := range want.GetUsers() {
+		err := k.user(ctx, u, slices.Contains(made, u.GetLogin()), want.GetMakeHomes())
+		if errors.Is(err, errNotMade) {
+			errs = append(errs, err)
+			continue
+		}
+		keep = append(keep, u.GetLogin())
+		if err != nil {
+			errs = append(errs, err)
+			continue
+		}
+		n++
+	}
+	if err := k.keepMade(keep); err != nil {
+		return n, err
+	}
+	if err := k.removeUsers(ctx, made, keep); err != nil {
+		errs = append(errs, err)
+	}
+	if err := k.sshConf(ctx, len(keep) > 0); err != nil {
+		errs = append(errs, err)
+	}
+	// The controller is where they log in and submit from; a node only
+	// while a job of theirs runs there.
+	if err := k.nodeLogin(ctx, len(keep) > 0 && want.GetNode() && !want.GetController()); err != nil {
+		errs = append(errs, err)
+	}
+	return n, errors.Join(errs...)
+}
+
+// nodeLogin holds the users llmgw made here to their jobs when they log in
+// (pam_slurm_adopt), or no longer. Only they are: root and the site's
+// administrators log in as before, so a mistake here never locks out the
+// people who would put it right.
+func (k *Keeper) nodeLogin(ctx context.Context, on bool) error {
+	old, err := k.host.ReadFile(slurm.PAMSSHD)
+	if err != nil {
+		if !on {
+			return nil
+		}
+		return fmt.Errorf("reading %s: %w", slurm.PAMSSHD, err)
+	}
+	if on {
+		if err := k.adoptModule(ctx); err != nil {
+			return err
+		}
+	}
+	read := func(name string) (string, error) {
+		b, err := k.host.ReadFile(filepath.Join(filepath.Dir(slurm.PAMSSHD), name))
+		return string(b), err
+	}
+	stack, contained, err := slurm.PAMAdopt(string(old), k.usersMarker(), read, on)
+	if err != nil {
+		return err
+	}
+	if stack == string(old) {
+		return nil
+	}
+	if on {
+		k.log.Info("llmgw's Slurm users log in to this node only while a job of theirs runs here",
+			"in_the_job", contained)
+		if !contained {
+			k.log.Warn("pam_systemd could not be skipped for them safely here: their ssh sessions are not held " +
+				"to their job's cores and cards, only to having one")
+		}
+	} else {
+		k.log.Info("llmgw's Slurm users are no longer held to their jobs when they log in here")
+	}
+	// Read by sshd for each login: nothing to reload.
+	_, err = k.write(slurm.PAMSSHD, []byte(stack), 0o644, "")
+	return err
+}
+
+// adoptModule is there, or is installed.
+func (k *Keeper) adoptModule(ctx context.Context) error {
+	if k.installed[slurm.AdoptModule] {
+		return nil
+	}
+	find := func() bool {
+		for _, d := range slurm.ModuleDirs {
+			if _, err := k.host.Run(ctx, nil, "test", "-f", filepath.Join(d, slurm.AdoptModule)); err == nil {
+				return true
+			}
+		}
+		return false
+	}
+	if !find() {
+		if k.inst == nil {
+			return fmt.Errorf("%s missing, and this agent installs nothing", slurm.AdoptModule)
+		}
+		k.log.Info("installing pam_slurm_adopt, which holds llmgw's Slurm users to their jobs here")
+		if err := k.inst.InstallPackages(ctx, slurm.AdoptPackages); err != nil {
+			return err
+		}
+		if !find() {
+			return fmt.Errorf("%s is still missing after installing %s", slurm.AdoptModule,
+				strings.Join(slurm.AdoptPackages["deb"], ", "))
+		}
+	}
+	k.installed[slurm.AdoptModule] = true
+	return nil
+}
+
+// errNotMade is a user llmgw could not make here, nor took for its own.
+var errNotMade = errors.New("not made")
+
+type notMade struct{ msg string }
+
+func (e notMade) Error() string        { return e.msg }
+func (e notMade) Is(target error) bool { return target == errNotMade }
+
+// user keeps one user; ours is whether llmgw made it here before.
+func (k *Keeper) user(ctx context.Context, u *llmgwv1.SlurmUser, ours, makeHome bool) error {
+	login, uid, home := u.GetLogin(), u.GetUid(), u.GetHome()
+	if !slurm.ValidLogin(login) || uid < slurm.MinUID || !filepath.IsAbs(home) || strings.Contains(home, "..") {
+		return notMade{fmt.Sprintf("the leader sent a user %q, UID %d, home %q", login, uid, home)}
+	}
+	id := strconv.FormatUint(uint64(uid), 10)
+	gecos := strings.Map(func(r rune) rune {
+		if r == ':' || r == ',' || r == '\n' || r < ' ' {
+			return -1
+		}
+		return r
+	}, u.GetName())
+	byName, okName := k.passwd(ctx, login)
+	byID, okID := k.passwd(ctx, id)
+	switch {
+	case okName && byName[2] != id:
+		return notMade{fmt.Sprintf("%s is already a user here with UID %s, not llmgw's %s", login, byName[2], id)}
+	case !okName && okID:
+		return notMade{fmt.Sprintf("UID %s is already %s's here, so %s is not made", id, byID[0], login)}
+	case !okName:
+		if _, ok := k.group(ctx, login); !ok {
+			if out, err := k.host.Run(ctx, nil, "groupadd", "-g", id, login); err != nil {
+				return notMade{fmt.Sprintf("groupadd %s: %v: %s", login, err, lastLine(out))}
+			}
+		}
+		k.log.Info("a Slurm user is made here, as llmgw's accounts say", "login", login, "uid", uid)
+		if out, err := k.host.Run(ctx, nil, "useradd", "-u", id, "-g", id, "-M", "-d", home, "-s", "/bin/bash",
+			"-c", gecos, login); err != nil {
+			return notMade{fmt.Sprintf("useradd %s: %v: %s", login, err, lastLine(out))}
+		}
+	case ours && (byName[4] != gecos || byName[5] != home):
+		if out, err := k.host.Run(ctx, nil, "usermod", "-c", gecos, "-d", home, login); err != nil {
+			return fmt.Errorf("usermod %s: %v: %s", login, err, lastLine(out))
+		}
+	}
+	keys := strings.Join(u.GetSshKeys(), "\n")
+	file := filepath.Join(slurm.AuthorizedKeysDir, login)
+	if keys == "" {
+		_ = k.host.Remove(file)
+	} else if _, err := k.write(file, []byte(keys+"\n"), 0o644, ""); err != nil {
+		return err
+	}
+	if makeHome {
+		return k.home(ctx, login, id, home)
+	}
+	return nil
+}
+
+// home makes a user's home directory, from /etc/skel, when it is not there.
+func (k *Keeper) home(ctx context.Context, login, id, home string) error {
+	if _, err := k.host.Run(ctx, nil, "test", "-d", home); err == nil {
+		return nil
+	}
+	if err := k.host.MkdirAll(filepath.Dir(home), 0o755, ""); err != nil {
+		return err
+	}
+	k.log.Info("a Slurm user's home is made", "login", login, "home", home)
+	for _, c := range [][]string{
+		{"install", "-d", "-m", "0700", "-o", id, "-g", id, home},
+		{"cp", "-rT", "/etc/skel", home},
+		{"chown", "-R", id + ":" + id, home},
+	} {
+		if out, err := k.host.Run(ctx, nil, c[0], c[1:]...); err != nil {
+			return fmt.Errorf("making %s's home: %s: %v: %s", login, c[0], err, lastLine(out))
+		}
+	}
+	return nil
+}
+
+// removeUsers takes away the users llmgw made here that are not in keep.
+// Their homes and files stay; their UID is never given to another.
+func (k *Keeper) removeUsers(ctx context.Context, made, keep []string) error {
+	var errs []error
+	left := slices.Clone(keep)
+	for _, login := range made {
+		if slices.Contains(keep, login) {
+			continue
+		}
+		k.log.Info("a Slurm user llmgw made is taken away; its home stays", "login", login)
+		_ = k.host.Remove(filepath.Join(slurm.AuthorizedKeysDir, login))
+		if _, ok := k.passwd(ctx, login); ok {
+			if out, err := k.host.Run(ctx, nil, "userdel", login); err != nil {
+				// Still logged in, or running something: tried again.
+				errs = append(errs, fmt.Errorf("userdel %s: %v: %s", login, err, lastLine(out)))
+				left = append(left, login)
+				continue
+			}
+		}
+	}
+	if !slices.Equal(left, made) {
+		if err := k.keepMade(left); err != nil {
+			errs = append(errs, err)
+		}
+	}
+	if len(left) == 0 {
+		if err := k.sshConf(ctx, false); err != nil {
+			errs = append(errs, err)
+		}
+		if err := k.nodeLogin(ctx, false); err != nil {
+			errs = append(errs, err)
+		}
+	}
+	return errors.Join(errs...)
+}
+
+func (k *Keeper) madeUsers() []string {
+	b, err := k.host.ReadFile(k.usersMarker())
+	if err != nil {
+		return nil
+	}
+	return strings.Fields(string(b))
+}
+
+func (k *Keeper) keepMade(logins []string) error {
+	s := slices.Clone(logins)
+	slices.Sort(s)
+	s = slices.Compact(s)
+	if len(s) == 0 {
+		return k.host.Remove(k.usersMarker())
+	}
+	_, err := k.write(k.usersMarker(), []byte(strings.Join(s, "\n")+"\n"), 0o644, "")
+	return err
+}
+
+// sshConf has sshd read the users' keys from AuthorizedKeysDir, or no
+// longer.
+func (k *Keeper) sshConf(ctx context.Context, on bool) error {
+	if !on {
+		if _, err := k.host.ReadFile(slurm.SSHDropIn); err != nil {
+			return nil
+		}
+		_ = k.host.Remove(slurm.SSHDropIn)
+		return k.reloadSSH(ctx)
+	}
+	changed, err := k.write(slurm.SSHDropIn, []byte(slurm.SSHConf), 0o644, "")
+	if err != nil || !changed {
+		return err
+	}
+	if out, err := k.host.Run(ctx, nil, "sshd", "-t"); err != nil {
+		_ = k.host.Remove(slurm.SSHDropIn)
+		return fmt.Errorf("sshd refuses llmgw's setting for the users' keys, so it is not kept: %v: %s", err, lastLine(out))
+	}
+	k.log.Info("sshd reads the Slurm users' keys from llmgw too", "dir", slurm.AuthorizedKeysDir)
+	return k.reloadSSH(ctx)
+}
+
+// reloadSSH has sshd read its settings again, under Debian's name for it
+// or Red Hat's; one that is not running reads them when it starts.
+func (k *Keeper) reloadSSH(ctx context.Context) error {
+	for _, u := range []string{"ssh", "sshd"} {
+		if k.active(ctx, u) {
+			return k.systemctl(ctx, "reload", u)
+		}
+	}
+	return nil
+}
+
+// passwd is getent's line for a user, by login or UID, in its fields.
+func (k *Keeper) passwd(ctx context.Context, key string) ([]string, bool) {
+	out, err := k.host.Run(ctx, nil, "getent", "passwd", key)
+	f := strings.Split(strings.TrimSpace(string(out)), ":")
+	if err != nil || len(f) < 7 {
+		return nil, false
+	}
+	return f, true
+}
+
+func (k *Keeper) group(ctx context.Context, key string) ([]string, bool) {
+	out, err := k.host.Run(ctx, nil, "getent", "group", key)
+	f := strings.Split(strings.TrimSpace(string(out)), ":")
+	if err != nil || len(f) < 3 {
+		return nil, false
+	}
+	return f, true
+}

internal/agent/weights/gc.go

@@ -33,6 +33,17 @@ func CachePathFor(dir, source string) string {
 	return filepath.Join(dir, hex.EncodeToString(sum[:8])+"-"+name)
 }
 
+// cacheSource is the source a cache name is computed from: a set's address
+// without its trailing slash, as its directory was named when it was fetched.
+// Computed from the slash-ended address, every set in the catalog looked like
+// nobody's, and a sweep offered 330 GB of models still in use for deletion.
+func cacheSource(source string) string {
+	if IsSet(source) {
+		return strings.TrimSuffix(source, "/")
+	}
+	return source
+}
+
 // Owned reports whether a path is one this package created.
 //
 // The dividing line of the whole feature. A file without the cache name was
@@ -65,7 +76,7 @@ func Orphans(have []string, wanted []string) []string {
 		// By base name rather than full path: the control plane does not know
 		// each node's cache directory, and the name is what identifies the
 		// file wherever the node keeps it.
-		keep[filepath.Base(CachePathFor("", src))] = struct{}{}
+		keep[filepath.Base(CachePathFor("", cacheSource(src)))] = struct{}{}
 	}
 
 	var out []string

internal/agent/weights/set.go

@@ -51,7 +51,7 @@ type manifest struct {
 // as a broken model rather than failing to load, so a partial set is left
 // unfinished under its temporary name and the next attempt starts again.
 func (f *Fetcher) ensureSet(ctx context.Context, source string, allowPickle bool) (string, error) {
-	dir, err := f.cachePath(strings.TrimSuffix(source, "/"))
+	dir, err := f.cachePath(cacheSource(source))
 	if err != nil {
 		return "", err
 	}

internal/agent/weights/set_test.go

@@ -74,6 +74,23 @@ func TestASetIsFetchedIntoOneDirectory(t *testing.T) {
 	}
 }
 
+func TestASetInTheCatalogIsNotAnOrphan(t *testing.T) {
+	// The catalog names a set by its address with the trailing slash; the
+	// fetch names the directory without it. A sweep that computed the name
+	// from the catalog's spelling offered every set in use for deletion.
+	f, srv, _ := mirrorOf(t, map[string]string{"model.safetensors": "aaaa", "config.json": "{}"})
+	dir, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got := Orphans([]string{dir}, []string{srv.URL + "/m/v/"}); len(got) != 0 {
+		t.Errorf("a set the catalog still names was called an orphan: %v", got)
+	}
+	if got := Orphans([]string{dir}, nil); len(got) != 1 {
+		t.Errorf("a set nothing names was not found: %v", got)
+	}
+}
+
 func TestAFinishedSetIsNotFetchedAgain(t *testing.T) {
 	f, srv, hits := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
 

internal/cli/clustersettings.go

@@ -247,6 +247,8 @@ var liveSettings = []string{
 	"presets",
 	// Handed to the reconciler as they change.
 	"on_demand",
+	// Read by the Slurm service each pass, kicked when they change.
+	"slurm",
 	// Read by server init, node add and backups, not by the server.
 	"backup_recipients", "backup_copy_to",
 }

internal/cli/ctl.go

@@ -189,6 +189,8 @@ func newCtlCmd() *cobra.Command {
 		newMCPCmd(client, &asJSON),
 		newKBCmd(client, &asJSON),
 		newTrainCmd(client, &asJSON),
+		newSlurmCmd(client, &asJSON),
+		newPricesCmd(client, &asJSON),
 		newConfigCmd(client, &asJSON),
 		newAuditCmd(client, &asJSON),
 		newNotifyCmd(client, &asJSON),

internal/cli/ctl_prices.go

@@ -0,0 +1,121 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/url"
+	"strconv"
+	"text/tabwriter"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
+)
+
+func newPricesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "prices",
+		Short: "What each model's tokens cost, from what the cluster costs",
+	}
+	var days int
+	var apply bool
+	var note string
+	suggest := &cobra.Command{
+		Use:   "suggest",
+		Short: "Each model's price from the GPU- and core-hours it held, over the tokens it served",
+		Long: "Set auth.gpu_hour_cost first: what an hour of one GPU costs, in the units budgets are in (the\n" +
+			"hardware written off over its years, power and room, the people who run it, over the GPUs and\n" +
+			"the hours of a month); auth.core_hour_cost likewise for models on CPUs. Each model's cost is\n" +
+			"the hours its replicas held, idle ones too, kept by the hour on the leader; spread over its\n" +
+			"tokens, an output token counted as four input ones and a cached prompt token as a tenth of one,\n" +
+			"it is its price. A model held and not used shows what that cost. --apply writes the prices\n" +
+			"into auth.pricing, one settings version, as `ctl config set` does.",
+		Example: "  llmgw ctl config set auth.gpu_hour_cost 1.35\n" +
+			"  llmgw ctl prices suggest --days 30\n" +
+			"  llmgw ctl prices suggest --apply --note \"November's costs\"",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var s adminapi.PriceSuggestions
+			if err := client().get(cmd.Context(), "/api/v1/prices/suggest?days="+url.QueryEscape(strconv.Itoa(days)), &s); err != nil {
+				return err
+			}
+			w := cmd.OutOrStdout()
+			if *asJSON {
+				return json.NewEncoder(w).Encode(s)
+			}
+			fmt.Fprintf(w, "cost: %g a GPU-hour, %g a core-hour; the last %d days", s.GPUHourCost, s.CoreHourCost, s.Days)
+			if !s.RecordedFrom.IsZero() && s.RecordedFrom.After(s.Since) {
+				fmt.Fprintf(w, " (held kept only since %s)", s.RecordedFrom.Local().Format("2006-01-02 15:04"))
+			}
+			fmt.Fprintln(w)
+			fmt.Fprintln(w, "prices a million tokens: input / output / cached input, in budget units")
+			tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
+			fmt.Fprintln(tw, "MODEL\tGPU-H\tCORE-H\tCOST\tTOKENS IN/OUT\tSUGGESTED\tNOW\tNOTE")
+			var changed []adminapi.PriceSuggestion
+			for _, m := range s.Models {
+				sug := "-"
+				if !m.Price.Zero() {
+					sug = perMillion(m.Price.PromptPer1k, m.Price.CompletionPer1k, m.Price.CachedPer1k)
+					if m.Price.PromptPer1k != m.Current.PromptPer1k || m.Price.CompletionPer1k != m.Current.CompletionPer1k ||
+						m.Price.CachedPer1k != m.Current.CachedPer1k {
+						changed = append(changed, m)
+					}
+				}
+				fmt.Fprintf(tw, "%s\t%.1f\t%.1f\t%.2f\t%s/%s\t%s\t%s\t%s\n", m.Model, m.GPUHours, m.CoreHours, m.Cost,
+					count(m.Prompt), count(m.Completion), sug,
+					perMillion(m.Current.PromptPer1k, m.Current.CompletionPer1k, m.Current.CachedPer1k), m.Note)
+			}
+			if err := tw.Flush(); err != nil {
+				return err
+			}
+			if !apply {
+				if len(changed) > 0 {
+					fmt.Fprintf(w, "%d model(s) priced otherwise now; --apply writes the suggestions\n", len(changed))
+				}
+				return nil
+			}
+			if len(changed) == 0 {
+				fmt.Fprintln(w, "nothing to change")
+				return nil
+			}
+			return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
+				for _, m := range changed {
+					for k, v := range map[string]int64{"prompt_per_1k": m.Price.PromptPer1k,
+						"completion_per_1k": m.Price.CompletionPer1k, "cached_per_1k": m.Price.CachedPer1k} {
+						if err := doc.SetYAML([]string{"auth", "pricing", m.Model, k}, strconv.FormatInt(v, 10)); err != nil {
+							return err
+						}
+					}
+				}
+				return nil
+			})
+		},
+	}
+	suggest.Flags().IntVar(&days, "days", 30, "the window, in days")
+	suggest.Flags().BoolVar(&apply, "apply", false, "write the suggested prices into auth.pricing")
+	suggest.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
+	cmd.AddCommand(suggest)
+	return cmd
+}
+
+// perMillion is micro-units a thousand tokens as budget units a million.
+func perMillion(in, out, cached int64) string {
+	if in == 0 && out == 0 && cached == 0 {
+		return "-"
+	}
+	f := func(v int64) string { return strconv.FormatFloat(float64(v)/1000, 'f', -1, 64) }
+	return f(in) + " / " + f(out) + " / " + f(cached)
+}
+
+func count(n int64) string {
+	switch {
+	case n >= 1e9:
+		return fmt.Sprintf("%.1fB", float64(n)/1e9)
+	case n >= 1e6:
+		return fmt.Sprintf("%.1fM", float64(n)/1e6)
+	case n >= 1e3:
+		return fmt.Sprintf("%.1fk", float64(n)/1e3)
+	}
+	return strconv.FormatInt(n, 10)
+}

internal/cli/ctl_slurm.go

@@ -0,0 +1,480 @@
+package cli
+
+import (
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/url"
+	"os"
+	"slices"
+	"strings"
+	"text/tabwriter"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
+)
+
+// newSlurmCmd is the Slurm cluster run beside llmgw (BACKLOG P): set up on
+// nodes llmgw already has, and nodes handed between the two.
+func newSlurmCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "slurm",
+		Short: "A Slurm cluster beside llmgw, for training and other batch work",
+		Long: "Slurm runs on llmgw's own nodes, installed and kept by their agents: slurmctld on one node,\n" +
+			"slurmd on those named. A node is llmgw's, drained in Slurm, until it is lent; lent, its\n" +
+			"replicas move off first, then Slurm may run jobs there. Taken back, Slurm drains it and\n" +
+			"llmgw has it again when its last job ends. Jobs are submitted on the controller node\n" +
+			"(sbatch, srun) by its own users, as on any Slurm cluster.",
+	}
+	cmd.AddCommand(newSlurmStatusCmd(client, asJSON), newSlurmSetupCmd(client), newSlurmOffCmd(client),
+		newSlurmListCmd(client, "lend", "Let Slurm run jobs on nodes; llmgw's replicas move off first", true),
+		newSlurmListCmd(client, "reclaim", "Take nodes back from Slurm; llmgw has them when their jobs end", false),
+		newSlurmSharedCmd(client), newSlurmUsersCmd(client, asJSON), newSlurmContainersCmd(client))
+	return cmd
+}
+
+func newSlurmContainersCmd(client func() *ctlClient) *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "containers",
+		Short: "Jobs in containers: srun --container-image (pyxis and enroot)",
+		Long: "On, the agents install enroot and pyxis on every machine of the Slurm cluster (from NVIDIA's\n" +
+			"releases, at versions llmgw pins and checks) and, on GPU machines, nvidia-container-cli; once\n" +
+			"every machine has them, Slurm loads pyxis and jobs may run in a container:\n\n" +
+			"  srun --gres=gpu:1 --container-image=nvcr.io#nvidia/pytorch:25.01-py3 python train.py\n\n" +
+			"The shared path is in every container where it is on the host. Off, Slurm stops loading pyxis;\n" +
+			"what was installed stays.",
+	}
+	var note string
+	toggle := func(use, short string, on bool) *cobra.Command {
+		c := &cobra.Command{
+			Use:   use,
+			Short: short,
+			Args:  cobra.NoArgs,
+			RunE: func(cmd *cobra.Command, _ []string) error {
+				return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
+					if !on {
+						if doc.Node([]string{"slurm", "containers"}) != nil {
+							doc.Unset([]string{"slurm", "containers"})
+						}
+						return nil
+					}
+					return doc.SetYAML([]string{"slurm", "containers"}, "true")
+				})
+			},
+		}
+		c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
+		return c
+	}
+	cmd.AddCommand(toggle("on", "Install pyxis and enroot everywhere, and let jobs run in containers", true),
+		toggle("off", "Slurm stops loading pyxis; what was installed stays", false))
+	return cmd
+}
+
+func newSlurmSharedCmd(client func() *ctlClient) *cobra.Command {
+	var path, server, nfs, note string
+	var off bool
+	c := &cobra.Command{
+		Use:   "shared --path DIR (--server NODE | --nfs HOST:/EXPORT) | --off",
+		Short: "Files every machine of the Slurm cluster sees at one path",
+		Long: "Home directories, datasets, the jobs' output and checkpoints, at the same path on the controller\n" +
+			"and every node. Three ways:\n" +
+			"  --nfs HOST:/EXPORT  the site's own NFS export, mounted at --path on every machine;\n" +
+			"  --server NODE       an llmgw node (the controller or a node) exports --path over NFS to the\n" +
+			"                      others, for a site with no storage of its own: one machine, not highly\n" +
+			"                      available, no faster than its disk and network;\n" +
+			"  neither             the site mounts --path itself (Lustre, BeeGFS, GPFS); llmgw only looks\n" +
+			"                      that it is there.\n" +
+			"The export answers the Slurm machines alone, NFS's port the cluster's networks. A machine\n" +
+			"that gives up its mount does so when no job holds files on it.",
+		Example: "  llmgw ctl slurm shared --path /shared --server node06\n" +
+			"  llmgw ctl slurm shared --path /shared --nfs filer.site:/export/hpc\n" +
+			"  llmgw ctl slurm shared --path /lustre/hpc\n" +
+			"  llmgw ctl slurm shared --off",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if off == (path != "") {
+				return fmt.Errorf("give --path, or --off")
+			}
+			return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
+				if off {
+					if doc.Node([]string{"slurm", "shared"}) != nil {
+						doc.Unset([]string{"slurm", "shared"})
+					}
+					return nil
+				}
+				v := map[string]string{"path": path}
+				if server != "" {
+					v["server"] = server
+				}
+				if nfs != "" {
+					v["nfs"] = nfs
+				}
+				b, _ := json.Marshal(v)
+				return doc.SetYAML([]string{"slurm", "shared"}, string(b))
+			})
+		},
+	}
+	c.Flags().StringVar(&path, "path", "", "where every machine sees the files, as /shared")
+	c.Flags().StringVar(&server, "server", "", "the llmgw node that exports them over NFS")
+	c.Flags().StringVar(&nfs, "nfs", "", "the site's NFS export, HOST:/EXPORT")
+	c.Flags().BoolVar(&off, "off", false, "no shared files; mounts and exports llmgw made are taken away")
+	c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
+	return c
+}
+
+func newSlurmUsersCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "users",
+		Short: "The Linux users that run jobs, made from llmgw's accounts",
+		Long: "With slurm.users \"llmgw\" every active account is a Linux user on every machine of the Slurm\n" +
+			"cluster: its login from its e-mail address, a UID of llmgw's from 200000 up (never given to\n" +
+			"another, even when the account goes), its home under the shared path. It logs in over SSH\n" +
+			"with the keys given in the portal (Profile) or here. An account suspended or deleted loses\n" +
+			"its user; its home and files stay. Off, users are the site's: from its directory (sssd), or\n" +
+			"made by hand.",
+	}
+	list := &cobra.Command{
+		Use:   "list",
+		Short: "Every user, with its login, UID and how many keys it has",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var us []slurmsvc.User
+			if err := client().get(cmd.Context(), "/api/v1/slurm/users", &us); err != nil {
+				return err
+			}
+			w := cmd.OutOrStdout()
+			if *asJSON {
+				return json.NewEncoder(w).Encode(us)
+			}
+			tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
+			fmt.Fprintln(tw, "LOGIN\tUID\tACCOUNT\tSSH KEYS")
+			for _, u := range us {
+				fmt.Fprintf(tw, "%s\t%d\t%s\t%d\n", u.Login, u.UID, u.Email, len(u.SSHKeys))
+			}
+			return tw.Flush()
+		},
+	}
+	var note string
+	toggle := func(use, short, value string) *cobra.Command {
+		c := &cobra.Command{
+			Use:   use,
+			Short: short,
+			Args:  cobra.NoArgs,
+			RunE: func(cmd *cobra.Command, _ []string) error {
+				return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
+					if value == "" {
+						if doc.Node([]string{"slurm", "users"}) != nil {
+							doc.Unset([]string{"slurm", "users"})
+						}
+						return nil
+					}
+					return doc.SetYAML([]string{"slurm", "users"}, yamlString(value))
+				})
+			},
+		}
+		c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
+		return c
+	}
+	keys := &cobra.Command{
+		Use:   "keys ACCOUNT FILE",
+		Short: "Replace an account's SSH public keys, from a file (- for standard input)",
+		Long:  "FILE holds OpenSSH public keys, one a line, as ~/.ssh/id_ed25519.pub; an empty one takes them all away.",
+		Example: "  llmgw ctl slurm users keys ayse@site.example ayse.pub\n" +
+			"  ssh-add -L | llmgw ctl slurm users keys ayse@site.example -",
+		Args: cobra.ExactArgs(2),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			var b []byte
+			var err error
+			if args[1] == "-" {
+				b, err = io.ReadAll(io.LimitReader(cmd.InOrStdin(), 64<<10))
+			} else {
+				b, err = os.ReadFile(args[1])
+			}
+			if err != nil {
+				return err
+			}
+			body, _ := json.Marshal(map[string]string{"keys": string(b)})
+			var u slurmsvc.User
+			if err := client().put(cmd.Context(), "/api/v1/slurm/users/"+url.PathEscape(args[0])+"/ssh-keys", body, &u); err != nil {
+				return err
+			}
+			fmt.Fprintf(cmd.OutOrStdout(), "%s (UID %d) has %d SSH key(s); the machines have them within a minute\n",
+				u.Login, u.UID, len(u.SSHKeys))
+			return nil
+		},
+	}
+	cmd.AddCommand(list,
+		toggle("on", "Make a Linux user for every active account, on every machine of the Slurm cluster", "llmgw"),
+		toggle("off", "Leave users to the site; those llmgw made are taken away, their homes left", ""),
+		keys)
+	return cmd
+}
+
+func newSlurmStatusCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
+	return &cobra.Command{
+		Use:   "status",
+		Short: "The Slurm cluster, and where each of its nodes is",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			var st slurmsvc.Status
+			if err := client().get(cmd.Context(), "/api/v1/slurm", &st); err != nil {
+				return err
+			}
+			w := cmd.OutOrStdout()
+			if *asJSON {
+				return json.NewEncoder(w).Encode(st)
+			}
+			if !st.Enabled {
+				fmt.Fprintln(w, "Slurm is off. `llmgw ctl slurm setup --controller NODE --nodes NODE,...` turns it on.")
+				return nil
+			}
+			fmt.Fprintf(w, "cluster %s, partition %s, controller %s", st.Cluster, st.Partition, st.Controller)
+			switch c := st.Ctld; {
+			case c == nil:
+				fmt.Fprintln(w, ": not heard from yet")
+			case c.Error != "":
+				fmt.Fprintf(w, ": %s\n", c.Error)
+			case c.SlurmctldRunning && c.SlurmdbdRunning:
+				fmt.Fprintf(w, ": slurmctld %s and slurmdbd running\n", c.Version)
+			case c.SlurmctldRunning:
+				fmt.Fprintf(w, ": slurmctld %s running, slurmdbd not (the jobs' records)\n", c.Version)
+			default:
+				fmt.Fprintln(w, ": slurmctld not running")
+			}
+			fmt.Fprintf(w, "shared files: %s\n", sharedLine(st.Shared))
+			if st.Users != "" {
+				fmt.Fprintf(w, "users: %d, made from llmgw's accounts\n", st.UserCount)
+				if st.Accounts > 0 {
+					fmt.Fprintf(w, "accounts: %d, one for each project (sbatch --account=ORG-TEAM-PROJECT)\n", st.Accounts)
+				}
+			}
+			if line := containersLine(st); line != "" {
+				fmt.Fprintf(w, "containers: %s\n", line)
+			}
+			tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
+			fmt.Fprintln(tw, "NODE\tPHASE\tSLURM\tJOBS\tSLURMD\tSHARED\tNOTE")
+			for _, n := range st.Nodes {
+				state, jobs, slurmd, shared, note := "-", "-", "-", "-", n.Describe()
+				if r := n.Status; r != nil {
+					state = or(r.State, "-")
+					jobs = fmt.Sprint(r.Jobs)
+					slurmd = map[bool]string{true: "running", false: "stopped"}[r.SlurmdRunning]
+					shared = or(r.Shared, "-")
+					if r.Error != "" {
+						note = r.Error
+					}
+				}
+				if !n.Connected {
+					note = "agent not connected; " + note
+				}
+				fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n", n.Node, n.Phase, state, jobs, slurmd, shared, note)
+			}
+			if err := tw.Flush(); err != nil {
+				return err
+			}
+			if len(st.Waiting) > 0 {
+				fmt.Fprintf(w, "not in slurm.conf until they report their hardware: %s\n", strings.Join(st.Waiting, ", "))
+			}
+			return nil
+		},
+	}
+}
+
+func newSlurmSetupCmd(client func() *ctlClient) *cobra.Command {
+	var controller, cluster, partition, note string
+	var nodes []string
+	var all bool
+	c := &cobra.Command{
+		Use:   "setup --controller NODE --nodes NODE,... | --all-nodes",
+		Short: "Run Slurm on llmgw's nodes: slurmctld on one, slurmd on those named",
+		Long: "Turns slurm.enabled on and names the nodes, or with --all-nodes every node. Their agents install munge and Slurm (22.05 or\n" +
+			"newer) from the machines' own package sources; the controller also slurmdbd, the jobs' records, in a\n" +
+			"MariaDB on that machine alone (one that was there keeps its settings). Every node starts drained in Slurm and stays\n" +
+			"llmgw's until `llmgw ctl slurm lend` gives it to Slurm.",
+		Example: "  llmgw ctl slurm setup --controller node06 --nodes cuda01,cuda02\n" +
+			"  llmgw ctl slurm setup --controller node06 --all-nodes",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if controller == "" || (len(nodes) == 0 && !all) {
+				return fmt.Errorf("give --controller, and --nodes or --all-nodes")
+			}
+			return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
+				if err := doc.SetYAML([]string{"slurm", "enabled"}, "true"); err != nil {
+					return err
+				}
+				if err := doc.SetYAML([]string{"slurm", "controller"}, yamlString(controller)); err != nil {
+					return err
+				}
+				if err := setList(doc, []string{"slurm", "nodes"}, nodes); err != nil {
+					return err
+				}
+				if err := doc.SetYAML([]string{"slurm", "all_nodes"}, fmt.Sprint(all)); err != nil {
+					return err
+				}
+				// Lent nodes no longer named are not lent.
+				lent := slices.DeleteFunc(docList(doc, []string{"slurm", "lent"}),
+					func(n string) bool { return !all && !slices.Contains(nodes, n) })
+				if err := setList(doc, []string{"slurm", "lent"}, lent); err != nil {
+					return err
+				}
+				for k, v := range map[string]string{"cluster_name": cluster, "partition": partition} {
+					if v != "" {
+						if err := doc.SetYAML([]string{"slurm", k}, yamlString(v)); err != nil {
+							return err
+						}
+					}
+				}
+				return nil
+			})
+		},
+	}
+	c.Flags().StringVar(&controller, "controller", "", "the node that runs slurmctld")
+	c.Flags().StringSliceVar(&nodes, "nodes", nil, "the nodes that run slurmd")
+	c.Flags().BoolVar(&all, "all-nodes", false, "every node runs slurmd, those that join later too")
+	c.Flags().StringVar(&cluster, "cluster-name", "", "Slurm's ClusterName (llmgw)")
+	c.Flags().StringVar(&partition, "partition", "", "the partition every node is in (gpu)")
+	c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
+	return c
+}
+
+func newSlurmOffCmd(client func() *ctlClient) *cobra.Command {
+	var note string
+	var force bool
+	c := &cobra.Command{
+		Use:   "off",
+		Short: "Stop running Slurm; every node is llmgw's again",
+		Long: "slurmd and slurmctld are stopped where llmgw started them; the packages stay. A node still\n" +
+			"lent is refused unless --force: take it back first, so its jobs end before the daemons do.",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
+				if lent := docList(doc, []string{"slurm", "lent"}); len(lent) > 0 && !force {
+					return fmt.Errorf("%s still lent: `llmgw ctl slurm reclaim` them first, or --force",
+						strings.Join(lent, ", "))
+				}
+				setList(doc, []string{"slurm", "lent"}, nil) //nolint:errcheck // an empty list
+				return doc.SetYAML([]string{"slurm", "enabled"}, "false")
+			})
+		},
+	}
+	c.Flags().BoolVar(&force, "force", false, "even with nodes still lent")
+	c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
+	return c
+}
+
+// containersLine says whether jobs may run in containers, and which
+// machines are not ready yet; empty when containers are off.
+func containersLine(st slurmsvc.Status) string {
+	switch st.Containers {
+	case "on":
+		return "on (srun --container-image)"
+	case "":
+		return ""
+	}
+	var waiting []string
+	for _, n := range st.Nodes {
+		if n.Status == nil || n.Status.Containers == "" {
+			waiting = append(waiting, n.Node)
+		}
+	}
+	if st.Ctld == nil || st.Ctld.Containers == "" {
+		if !slices.Contains(waiting, st.Controller) {
+			waiting = append(waiting, st.Controller)
+		}
+	}
+	return "installing pyxis and enroot; jobs take them once every machine has them (waiting: " +
+		strings.Join(waiting, ", ") + ")"
+}
+
+// sharedLine says where the shared files are and come from.
+func sharedLine(sh config.SharedStorage) string {
+	switch {
+	case sh.Path == "":
+		return "none (`llmgw ctl slurm shared` adds them; a batch job's output stays on its node)"
+	case sh.Server != "":
+		return fmt.Sprintf("%s, exported over NFS by %s", sh.Path, sh.Server)
+	case sh.NFS != "":
+		return fmt.Sprintf("%s, mounted from %s", sh.Path, sh.NFS)
+	}
+	return sh.Path + ", mounted by the site"
+}
+
+// newSlurmListCmd adds nodes to slurm.lent, or takes them out.
+func newSlurmListCmd(client func() *ctlClient, use, short string, add bool) *cobra.Command {
+	var note string
+	c := &cobra.Command{
+		Use:   use + " NODE...",
+		Short: short,
+		Args:  cobra.MinimumNArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
+				return lendChange(doc, args, add)
+			})
+		},
+	}
+	c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
+	return c
+}
+
+// lendChange adds nodes to slurm.lent, or with add false takes them out.
+func lendChange(doc *settings.Doc, names []string, add bool) error {
+	nodes := docList(doc, []string{"slurm", "nodes"})
+	all := docBool(doc, []string{"slurm", "all_nodes"})
+	lent := docList(doc, []string{"slurm", "lent"})
+	for _, n := range names {
+		switch {
+		case add && !all && !slices.Contains(nodes, n):
+			return fmt.Errorf("%s does not run slurmd: add it with `llmgw ctl slurm setup --nodes`", n)
+		case add && !slices.Contains(lent, n):
+			lent = append(lent, n)
+		case !add:
+			lent = slices.DeleteFunc(lent, func(l string) bool { return l == n })
+		}
+	}
+	return setList(doc, []string{"slurm", "lent"}, lent)
+}
+
+// docList reads a list of strings from the settings, empty when unset.
+func docList(doc *settings.Doc, path []string) []string {
+	n := doc.Node(path)
+	if n == nil {
+		return nil
+	}
+	var out []string
+	if err := n.Decode(&out); err != nil {
+		return nil
+	}
+	return out
+}
+
+// docBool reads a boolean from the settings, false when unset.
+func docBool(doc *settings.Doc, path []string) bool {
+	var b bool
+	if n := doc.Node(path); n != nil {
+		_ = n.Decode(&b)
+	}
+	return b
+}
+
+func setList(doc *settings.Doc, path []string, list []string) error {
+	if len(list) == 0 {
+		if doc.Node(path) != nil {
+			doc.Unset(path)
+		}
+		return nil
+	}
+	return doc.SetYAML(path, yamlList(list))
+}
+
+func yamlString(s string) string {
+	b, _ := json.Marshal(s)
+	return string(b)
+}
+
+func yamlList(list []string) string {
+	b, _ := json.Marshal(list)
+	return string(b)
+}

internal/cli/ctl_slurm_test.go

@@ -0,0 +1,35 @@
+package cli
+
+import (
+	"slices"
+	"strings"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
+)
+
+func TestSlurmDocLists(t *testing.T) {
+	doc, err := settings.FromBytes("settings", []byte("slurm:\n  enabled: true\n  nodes: [cuda01, site2]\n"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got := docList(doc, []string{"slurm", "nodes"}); !slices.Equal(got, []string{"cuda01", "site2"}) {
+		t.Fatalf("nodes %v", got)
+	}
+	if got := docList(doc, []string{"slurm", "lent"}); got != nil {
+		t.Fatalf("lent %v", got)
+	}
+	if err := setList(doc, []string{"slurm", "lent"}, []string{"cuda01"}); err != nil {
+		t.Fatal(err)
+	}
+	if got := docList(doc, []string{"slurm", "lent"}); !slices.Equal(got, []string{"cuda01"}) {
+		t.Fatalf("lent %v", got)
+	}
+	if err := setList(doc, []string{"slurm", "lent"}, nil); err != nil {
+		t.Fatal(err)
+	}
+	b, _ := doc.Bytes()
+	if strings.Contains(string(b), "lent") {
+		t.Fatalf("an empty list left:\n%s", b)
+	}
+}

internal/cli/ctl_status_test.go

@@ -84,3 +84,19 @@ func TestTheSummaryIsWrittenWhereItIsToldNotToTheTerminal(t *testing.T) {
 		}
 	}
 }
+
+func TestStatusSaysWhenAlertsGoNowhere(t *testing.T) {
+	render := func(s adminapi.Summary) string {
+		cmd := &cobra.Command{}
+		var b bytes.Buffer
+		cmd.SetOut(&b)
+		printSummary(cmd, s)
+		return b.String()
+	}
+	if out := render(adminapi.Summary{NoAlerts: true}); !strings.Contains(out, "ALERTS    nowhere") {
+		t.Errorf("no webhook was not called out:\n%s", out)
+	}
+	if out := render(adminapi.Summary{}); strings.Contains(out, "ALERTS") {
+		t.Errorf("an alerts line with webhooks set:\n%s", out)
+	}
+}

internal/cli/ctlweights.go

@@ -4,6 +4,7 @@ import (
 	"encoding/json"
 	"fmt"
 	"strings"
+	"time"
 
 	"github.com/spf13/cobra"
 
@@ -87,13 +88,10 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
 	if err != nil {
 		return err
 	}
-	var results []struct {
-		Node  string `json:"node"`
-		Files int    `json:"files"`
-		Bytes int64  `json:"bytes"`
-		Error string `json:"error,omitempty"`
-	}
-	if err := c.post(cmd.Context(), "/api/v1/weights/collect", body, &results); err != nil {
+	var results []adminapi.CollectResult
+	// The control plane waits for the nodes to list their files without the
+	// deleted ones: half a minute at most.
+	if err := c.patient(time.Minute).post(cmd.Context(), "/api/v1/weights/collect", body, &results); err != nil {
 		return err
 	}
 	if asJSON {
@@ -103,7 +101,7 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
 		fmt.Fprintln(cmd.OutOrStdout(), "Nothing to reclaim.")
 		return nil
 	}
-	var freed int64
+	var freed, left int64
 	for _, r := range results {
 		if r.Error != "" {
 			// Reported rather than fatal: the other nodes' space is still worth
@@ -111,14 +109,20 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
 			fmt.Fprintf(cmd.OutOrStdout(), "%s: %s\n", r.Node, r.Error)
 			continue
 		}
-		fmt.Fprintf(cmd.OutOrStdout(), "%s: asked to remove %d files, %s\n",
-			r.Node, r.Files, fileSize(r.Bytes))
-		freed += r.Bytes
+		line := fmt.Sprintf("%s: removed %d files, %s", r.Node, r.Files, fileSize(r.Bytes))
+		if r.Left > 0 {
+			line += fmt.Sprintf("; %d still there, %s", r.Left, fileSize(r.LeftBytes))
+		}
+		fmt.Fprintln(cmd.OutOrStdout(), line)
+		freed, left = freed+r.Bytes, left+r.LeftBytes
+	}
+	fmt.Fprintf(cmd.OutOrStdout(), "\n%s freed.\n", fileSize(freed))
+	if left > 0 {
+		// Each node decides: one that keeps a path it cannot prove it
+		// fetched is the feature working. Its agent's log names each.
+		fmt.Fprintf(cmd.OutOrStdout(), "%s the nodes kept; each node's agent log says why "+
+			"(journalctl -u llmgw-agent).\n", fileSize(left))
 	}
-	// "asked", not "removed": each node decides, and one that refuses a path it
-	// cannot prove it fetched is the feature working rather than failing.
-	fmt.Fprintf(cmd.OutOrStdout(), "\n%s requested. The nodes verify each path before deleting it.\n",
-		fileSize(freed))
 	return nil
 }
 

internal/cli/install_lines_test.go

@@ -0,0 +1,34 @@
+package cli
+
+import (
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+func TestAPackageInstallThatIsNoEnginesIsNamed(t *testing.T) {
+	if got := provisionLine("", "", "installing", "mariadb-server, munge, with apt-get"); got !=
+		"system packages: installing: mariadb-server, munge, with apt-get" {
+		t.Fatalf("got %q", got)
+	}
+	if got := provisionLine("llamacpp", "b10991", "ready", ""); got != "llamacpp b10991: ready" {
+		t.Fatalf("got %q", got)
+	}
+}
+
+func TestCoresAloneKeepTheMemory(t *testing.T) {
+	m := catalog.Model{Name: "judge", Profiles: []catalog.Profile{{Name: "cpu", MinMemoryGB: 2}, {Name: "cpu-big", MinMemoryGB: 2}}}
+	if got, err := keptMemory(m, ""); err != nil || got != 2 {
+		t.Fatalf("got %d, %v", got, err)
+	}
+	m.Profiles[1].MinMemoryGB = 4
+	if _, err := keptMemory(m, ""); err == nil {
+		t.Fatal("profiles that differ were given one memory")
+	}
+	if got, err := keptMemory(m, "cpu-big"); err != nil || got != 4 {
+		t.Fatalf("got %d, %v", got, err)
+	}
+	if _, err := keptMemory(m, "gpu"); err == nil {
+		t.Fatal("a profile the model has not was taken")
+	}
+}

internal/cli/model.go

@@ -338,7 +338,7 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
 	var memory, cores int
 	var profile string
 	c := &cobra.Command{
-		Use:   "resources <model> --memory GB [--cores N]",
+		Use:   "resources <model> [--memory GB] [--cores N]",
 		Short: "Set what a model reserves of a node: memory, and CPU cores",
 		Long: "--memory is the host RAM a replica needs: for a model with no GPU, its\n" +
 			"weights, KV cache and engine together. Placement reserves it on the node,\n" +
@@ -351,8 +351,20 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
 			"--model M` moves them to the new figures.",
 		Args: cobra.ExactArgs(1),
 		RunE: func(cmd *cobra.Command, args []string) error {
+			if !cmd.Flags().Changed("memory") && !cmd.Flags().Changed("cores") {
+				return fmt.Errorf("give --memory, --cores or both")
+			}
 			if !cmd.Flags().Changed("memory") {
-				return fmt.Errorf("--memory is required")
+				// Cores alone: the memory stays what it is.
+				var cur catalog.Model
+				if err := client().get(cmd.Context(), "/api/v1/models/"+args[0], &cur); err != nil {
+					return err
+				}
+				kept, err := keptMemory(cur, profile)
+				if err != nil {
+					return err
+				}
+				memory = kept
 			}
 			body, err := json.Marshal(adminapi.ResourcesRequest{Profile: profile, MemoryGB: memory, CPUCores: cores})
 			if err != nil {
@@ -369,12 +381,31 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
 			return nil
 		},
 	}
-	c.Flags().IntVar(&memory, "memory", 0, "host RAM a replica needs, in GB")
+	c.Flags().IntVar(&memory, "memory", 0, "host RAM a replica needs, in GB; kept as it is when left out")
 	c.Flags().IntVar(&cores, "cores", 0, "CPU cores a replica with no GPU computes on; 0 reserves none")
 	c.Flags().StringVar(&profile, "profile", "", "one profile; every profile when left out")
 	return c
 }
 
+// keptMemory is the memory a model's profiles reserve now, for a change of
+// cores alone: one profile's, or every profile's when they agree.
+func keptMemory(m catalog.Model, profile string) (int, error) {
+	memory := -1
+	for _, p := range m.Profiles {
+		if profile != "" && p.Name != profile {
+			continue
+		}
+		if memory >= 0 && p.MinMemoryGB != memory {
+			return 0, fmt.Errorf("%s's profiles reserve different memory: give --memory, or --profile", m.Name)
+		}
+		memory = p.MinMemoryGB
+	}
+	if memory < 0 {
+		return 0, fmt.Errorf("%s has no profile %q", m.Name, profile)
+	}
+	return memory, nil
+}
+
 func newClassCmd(client func() *ctlClient) *cobra.Command {
 	return &cobra.Command{
 		Use:   "class <model> critical|standard|opportunistic|default",

internal/cli/node.go

@@ -114,7 +114,7 @@ engines are kept.`,
 	}
 	f := cmd.Flags()
 	f.StringVar(&na.opts.Node, "node", "", "node id (default: the machine's hostname)")
-	f.StringVar(&na.opts.User, "ssh-user", "", "user to log in as (default: the one before @ in the address, else whoever ran sudo, else you)")
+	f.StringVar(&na.opts.User, "ssh-user", "", "user to log in as (default: the one before @ in the address, else the ssh config's for the machine, else whoever ran sudo, else you)")
 	f.StringArrayVar(&na.servers, "server", nil, "control plane address the agent dials, https://host:7443; repeatable for failover (default: this machine)")
 	f.StringArrayVar(&na.fabrics, "fabric", nil, "InfiniBand or RoCE fabric and its adapters, name:mlx5_0,mlx5_1; repeatable")
 	f.StringVar(&na.opts.MgmtAddr, "mgmt-addr", "", "address engines are reached at (default: found by the agent)")
@@ -228,8 +228,9 @@ func (na *nodeAdd) target(addr string) onboard.Options {
 	if o.User == "" {
 		// Under sudo, the person who ran it — their account is usually the
 		// one that exists on the nodes too. root's own keys still do the
-		// logging in, since that is who is running ssh.
-		o.User = os.Getenv("SUDO_USER")
+		// logging in, since that is who is running ssh. Only a fallback: a
+		// User in the ssh config's block for the machine comes first.
+		o.FallbackUser = os.Getenv("SUDO_USER")
 	}
 	return o
 }
@@ -495,7 +496,17 @@ func engineEvent(engine, version, state, detail string) onboard.Event {
 		level = onboard.Warn
 	}
 	return onboard.Event{Step: onboard.StepEngines, Level: level,
-		Detail: fmt.Sprintf("%s %s: %s%s", engine, version, state, prefixed(": ", detail))}
+		Detail: provisionLine(engine, version, state, detail)}
+}
+
+// provisionLine is one line of what a node is installing. A package
+// install that is no engine's, Slurm's or MariaDB's, has no name or version.
+func provisionLine(engine, version, state, detail string) string {
+	what := strings.TrimSpace(engine + " " + version)
+	if what == "" {
+		what = "system packages"
+	}
+	return fmt.Sprintf("%s: %s%s", what, state, prefixed(": ", detail))
 }
 
 func prefixed(p, s string) string {

internal/cli/server.go

@@ -10,6 +10,7 @@ import (
 	"fmt"
 	"log/slog"
 	"maps"
+	"math"
 	"net"
 	"net/http"
 	"os"
@@ -36,6 +37,7 @@ import (
 	"golang.org/x/sync/errgroup"
 
 	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/audit"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/bundle"
@@ -47,8 +49,10 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrictest"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/federation"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/held"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
@@ -59,6 +63,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/router"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
 	storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/version"
@@ -254,6 +259,7 @@ func newServerCmd() *cobra.Command {
 			// The settings in force, for what reads them at each use.
 			var current atomic.Pointer[config.Server]
 			current.Store(&cfg)
+			rt.SetSlurm(func() config.SlurmSettings { return current.Load().Slurm })
 			var agentSvc *agentsvc.Service
 			var agentHandler http.Handler
 			var agentPath string
@@ -261,6 +267,8 @@ func newServerCmd() *cobra.Command {
 			// What the settings say of the firewall, for the mirror and the
 			// store's members; nil without a control plane.
 			var fwPub *firewallPublisher
+			// The Slurm cluster beside llmgw, when there is a store (BACKLOG P).
+			var slurmSvc *slurmsvc.Service
 			updateMetrics := &modelserve.UpdateCollector{}
 			// The store's cluster, when it is one; read by the metrics and
 			// by `ctl status`.
@@ -412,6 +420,7 @@ func newServerCmd() *cobra.Command {
 				adminAPI = api
 				api.SetReleases(cfg.Releases)
 				api.SetNotifier(notifier)
+				api.SetWebhooks(func() int { return len(current.Load().Notify.Webhooks) })
 				api.SetReplayer(rt.Replay)
 				// Quality suites, asked of a model the way a replay is.
 				evals = newEvalService(cfg, rt, admin, log)
@@ -427,6 +436,27 @@ func newServerCmd() *cobra.Command {
 						api.SetAgentRestarter(agentSvc.RestartAgents)
 					}
 					api.SetAudit(st)
+					// Prices from what the cluster costs: what each model
+					// holds, kept by the hour on the leader.
+					api.SetPrices(st, func() adminapi.PriceConfig {
+						au := current.Load().Auth
+						cur := make(map[string]accounting.Price, len(au.Pricing))
+						for m, p := range au.Pricing {
+							cur[m] = accountingPrice(p)
+						}
+						return adminapi.PriceConfig{Basis: accounting.Basis{GPUHour: au.GPUHourCost, CoreHour: au.CoreHourCost},
+							Current: cur, Default: accountingPrice(au.DefaultPrice)}
+					})
+					go (&held.Recorder{Inv: admin.Inventory(), Store: st, Log: log,
+						Models: func() map[string]string {
+							m := map[string]string{}
+							for _, rep := range admin.Reconciler().Replicas() {
+								m[rep.ID] = rep.Model
+							}
+							return m
+						},
+						Leader: func() bool { return admin.Leadership() == nil || admin.Leadership().IsLeader() },
+					}).Run(ctx)
 					// Knowledge bases, with the cluster's own models to embed a
 					// probe (a collection's dimension) and an operator's search.
 					api.SetKB(&adminapi.KBAdmin{Store: st, Embed: embedder(rt.Replay)})
@@ -436,6 +466,40 @@ func newServerCmd() *cobra.Command {
 						agentSvc.SetTrain(trainer)
 						go trainer.Run(ctx)
 						api.SetTrain(&adminapi.TrainAdmin{Store: st, Service: trainer})
+						// Slurm on the nodes the settings name, its munge key
+						// made from a cluster secret.
+						slurmSvc = slurmsvc.New(func() config.SlurmSettings { return current.Load().Slurm },
+							admin.Inventory(), agentSvc, func(ctx context.Context) ([]byte, error) {
+								secret, err := st.ClusterSecret(ctx, "slurm-munge")
+								if err != nil {
+									return nil, err
+								}
+								return slurm.MungeKey(secret)
+							}, log)
+						// What slurm.conf held, for the leader that comes next.
+						slurmSvc.SetMemory(slurmsvc.StoreMemory(st))
+						// slurm.users "llmgw": a Linux user for each active account.
+						slurmSvc.SetUsers(func(ctx context.Context) ([]slurmsvc.User, error) {
+							us, err := st.SlurmUsers(ctx)
+							out := make([]slurmsvc.User, 0, len(us))
+							for _, u := range us {
+								out = append(out, adminapi.SlurmUserView(u))
+							}
+							return out, err
+						})
+						api.SetSlurmUsers(st)
+						// Slurm's accounts from the projects, and the jobs
+						// that ended priced as the cluster's hours are.
+						slurmSvc.SetAccounts(func(ctx context.Context) ([]slurm.Account, error) {
+							return slurmAccounts(ctx, st)
+						})
+						slurmSvc.SetJobs(func(ctx context.Context, jobs []slurmsvc.JobRecord) error {
+							au := current.Load().Auth
+							return recordSlurmJobs(ctx, st, current.Load().Slurm.Cluster(), au.GPUHourCost, au.CoreHourCost, jobs, log)
+						})
+						agentSvc.SetSlurm(slurmSvc)
+						go slurmSvc.Run(ctx)
+						api.SetSlurm(slurmSvc)
 					}
 					// The cluster's settings, each change checked against this
 					// machine's own keys before it is kept.
@@ -841,6 +905,9 @@ func newServerCmd() *cobra.Command {
 							fwPub.Kick()
 						}
 					}
+					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "slurm") }) && slurmSvc != nil {
+						slurmSvc.Kick()
+					}
 					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "on_demand") }) && admin != nil {
 						admin.Reconciler().SetEvictionTimes(next.OnDemand.EvictAfterIdle, next.OnDemand.MinLifetime)
 					}
@@ -1384,3 +1451,82 @@ func runPresetsPass(ctx context.Context, api *adminapi.API, lead *control.Leader
 		}
 	}
 }
+
+// accountingPrice is a configured price as accounting has it.
+func accountingPrice(p config.Price) accounting.Price {
+	return accounting.Price{PromptPer1k: p.PromptPer1k, CompletionPer1k: p.CompletionPer1k, CachedPer1k: p.CachedPer1k,
+		PerImage: p.PerImage, PerMesh: p.PerMesh}
+}
+
+// slurmAccounts are Slurm's accounts from the store's projects, members and
+// Slurm users.
+func slurmAccounts(ctx context.Context, st *storepkg.DB) ([]slurm.Account, error) {
+	ps, err := st.SlurmProjects(ctx)
+	if err != nil {
+		return nil, err
+	}
+	ms, err := st.AllMemberships(ctx)
+	if err != nil {
+		return nil, err
+	}
+	us, err := st.SlurmUsers(ctx)
+	if err != nil {
+		return nil, err
+	}
+	logins := make(map[string]string, len(us))
+	for _, u := range us {
+		logins[u.UserID] = u.Login
+	}
+	tps := make([]slurm.TenancyProject, 0, len(ps))
+	for _, p := range ps {
+		tps = append(tps, slurm.TenancyProject{ID: p.ID, Name: p.Name, TeamID: p.TeamID, TeamName: p.TeamName,
+			OrgID: p.OrgID, OrgName: p.OrgName})
+	}
+	tms := make([]slurm.TenancyMember, 0, len(ms))
+	for _, m := range ms {
+		tms = append(tms, slurm.TenancyMember{UserID: m.UserID, Scope: string(m.Scope), ScopeID: m.ScopeID})
+	}
+	return slurm.Accounts(tps, tms, logins), nil
+}
+
+// recordSlurmJobs records the jobs that ended, each with its project (from
+// its account) and person (from its login), priced as the hours the
+// cluster's models hold are: a GPU job its cards' hours, any other its
+// cores'.
+func recordSlurmJobs(ctx context.Context, st *storepkg.DB, cluster string, gpuHour, coreHour float64,
+	jobs []slurmsvc.JobRecord, log *slog.Logger) error {
+	as, err := slurmAccounts(ctx, st)
+	if err != nil {
+		return err
+	}
+	project := map[string]string{}
+	for _, a := range as {
+		if a.Project != "" {
+			project[a.Name] = a.Project
+		}
+	}
+	us, err := st.SlurmUsers(ctx)
+	if err != nil {
+		return err
+	}
+	person := map[string]string{}
+	for _, u := range us {
+		person[u.Login] = u.UserID
+	}
+	rows := make([]storepkg.SlurmJob, 0, len(jobs))
+	for _, j := range jobs {
+		secs := j.Elapsed.Seconds()
+		cost := float64(j.CPUs) * secs / 3600 * coreHour
+		if j.GPUs > 0 {
+			cost = float64(j.GPUs) * secs / 3600 * gpuHour
+		}
+		rows = append(rows, storepkg.SlurmJob{Key: cluster + "/" + j.ID, UserID: person[j.Login], ProjectID: project[j.Account],
+			Login: j.Login, Account: j.Account, Partition: j.Partition, State: j.State, Start: j.Start, End: j.End,
+			ElapsedSeconds: int64(secs), CPUs: int64(j.CPUs), GPUs: int64(j.GPUs), CostMicros: int64(math.Ceil(cost * 1e6))})
+	}
+	n, err := st.RecordSlurmJobs(ctx, rows)
+	if n > 0 {
+		log.Info("Slurm jobs that ended recorded, with their cost", "jobs", n)
+	}
+	return err
+}

internal/cli/serveradd.go

@@ -443,7 +443,7 @@ func (sa *serverAdd) storeAccess(ctx context.Context) error {
 			return err
 		}
 	}
-	sa.done("nodes will be told of both control planes within half a minute")
+	sa.done("nodes will be told of every control plane within half a minute")
 	return nil
 }
 

internal/cli/serverinit.go

@@ -124,6 +124,13 @@ type serverInit struct {
 	// media has the nodes install ComfyUI and the leader add a picture model
 	// and a 3D model once a card fits one.
 	media bool
+	// slurm runs Slurm beside llmgw: slurmctld on this machine's node,
+	// slurmd on every node, each llmgw's until it is lent.
+	slurm bool
+	// slurmShared is where Slurm's shared files come from: "here" (this
+	// machine exports /shared), the site's export as HOST:/EXPORT, or
+	// "none". slurmUsers is "llmgw" (users from the accounts) or "site".
+	slurmShared, slurmUsers string
 	// evictAfterIdle and minLifetime are on_demand's; zero leaves its own.
 	evictAfterIdle, minLifetime time.Duration
 	// standbys makes each machine of the replicated store a control plane
@@ -258,6 +265,12 @@ func newServerInitCmd() *cobra.Command {
 	f.DurationVar(&in.maxKeyTTL, "max-key-ttl", defaultMaxKeyTTL, "the longest an API key lives")
 	f.BoolVar(&in.media, "media", false,
 		"pictures and 3D: the nodes install ComfyUI, and a picture model and a 3D model are added once a card fits one")
+	f.BoolVar(&in.slurm, "slurm", false,
+		"Slurm for training and batch jobs: slurmctld and slurmdbd (with a MariaDB) on this machine, slurmd on every node, each llmgw's until 'llmgw ctl slurm lend'")
+	f.StringVar(&in.slurmShared, "slurm-shared", "here",
+		"with --slurm, the jobs' shared files at /shared: here (this machine exports them over NFS), the site's HOST:/EXPORT, or none")
+	f.StringVar(&in.slurmUsers, "slurm-users", config.UsersLlmgw,
+		"with --slurm, who makes the Linux users that run jobs: llmgw (one for each account, SSH keys in the portal) or site")
 	f.DurationVar(&in.evictAfterIdle, "evict-after-idle", 0,
 		"how long a model loaded on demand goes unasked-for before another may have its card (0: 2m)")
 	f.DurationVar(&in.minLifetime, "min-lifetime", 0,

internal/cli/serverinit_choices.go

@@ -13,6 +13,7 @@ import (
 	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
 )
@@ -132,9 +133,25 @@ func (in *serverInit) checkChoices() error {
 		return fmt.Errorf("max key ttl %s: an hour at least", in.maxKeyTTL)
 	case in.evictAfterIdle < 0 || in.minLifetime < 0:
 		return fmt.Errorf("on-demand times cannot be negative")
+	case in.slurm && in.noNode:
+		return fmt.Errorf("slurm runs slurmctld on this machine's node: it needs the node (no --no-node)")
 	case in.vip != "" && !in.standbys:
 		return fmt.Errorf("a floating address moves between control planes: it needs the standbys")
 	}
+	if in.slurm {
+		// The controller is this machine's node, named by its hostname: a
+		// name Slurm does not take would keep the server from starting.
+		if err := (config.SlurmSettings{Enabled: true, Controller: localNodeID()}).Check(); err != nil {
+			return fmt.Errorf("slurm: this machine's hostname: %w", err)
+		}
+		set := config.SlurmSettings{Enabled: true, Controller: localNodeID(), Shared: slurmShared(in.slurmShared)}
+		if err := set.Check(); err != nil {
+			return fmt.Errorf("--slurm-shared %q: here, none, or the site's HOST:/EXPORT: %w", in.slurmShared, err)
+		}
+		if u := in.slurmUsers; u != "" && u != config.UsersLlmgw && u != "site" {
+			return fmt.Errorf("--slurm-users %q: llmgw or site", u)
+		}
+	}
 	if in.vip != "" {
 		ip, _, err := net.ParseCIDR(in.vip)
 		if err != nil || ip.To4() == nil {
@@ -318,6 +335,9 @@ func (in *serverInit) tuneConfig(body string) string {
 			"# one ('llmgw ctl model presets auto off' stops it).\n"+
 			"engines:\n  comfyui: "+provision.ComfyUIVersion()+"\npresets:\n  auto: true\n")
 	}
+	if in.slurm {
+		extra = append(extra, slurmSection(localNodeID(), slurmShared(in.slurmShared), in.slurmUsers != "site"))
+	}
 	if in.evictAfterIdle > 0 || in.minLifetime > 0 {
 		b := "# How models that load when asked for take turns on a card, chosen at\n# install.\non_demand:\n"
 		if in.evictAfterIdle > 0 {
@@ -334,6 +354,53 @@ func (in *serverInit) tuneConfig(body string) string {
 	return body
 }
 
+// slurmShared is --slurm-shared as settings: "here" is /shared exported
+// by this machine's node, "none" nothing, anything else the site's export.
+func slurmShared(v string) config.SharedStorage {
+	switch v {
+	case "", "none":
+		return config.SharedStorage{}
+	case "here":
+		return config.SharedStorage{Path: "/shared", Server: localNodeID()}
+	}
+	return config.SharedStorage{Path: "/shared", NFS: v}
+}
+
+// slurmSection is server.yaml's slurm block as an install chooses it:
+// slurmctld on this machine's node, slurmd on every node, the shared files
+// and the users as chosen.
+func slurmSection(controller string, shared config.SharedStorage, users bool) string {
+	b := "# Slurm beside llmgw, chosen at install: slurmctld, and slurmdbd with a MariaDB\n" +
+		"# for the jobs' records, on this machine; slurmd on\n" +
+		"# every node, those that join later too. Each node is llmgw's, drained in\n" +
+		"# Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm; 'llmgw ctl slurm\n" +
+		"# reclaim NODE' takes it back once its jobs end.\n" +
+		"slurm:\n  enabled: true\n  controller: " + yamlString(controller) + "\n  all_nodes: true\n"
+	switch {
+	case shared.Server != "":
+		b += "  # /shared on every Slurm machine, exported over NFS from this one\n" +
+			"  # ('llmgw ctl slurm shared' points it at the site's storage instead).\n" +
+			"  shared:\n    path: " + yamlString(shared.Path) + "\n    server: " + yamlString(shared.Server) + "\n"
+	case shared.NFS != "":
+		b += "  # /shared on every Slurm machine, mounted from the site's export.\n" +
+			"  shared:\n    path: " + yamlString(shared.Path) + "\n    nfs: " + yamlString(shared.NFS) + "\n"
+	}
+	if users {
+		b += "  # A Linux user for each llmgw account, the same UID everywhere; SSH keys\n" +
+			"  # in the portal's Profile. Empty leaves users to the site (sssd, a directory).\n" +
+			"  users: llmgw\n"
+	}
+	return b
+}
+
+// localNodeID is the node id this machine's agent takes: its hostname.
+func localNodeID() string {
+	if h, err := os.Hostname(); err == nil && h != "" {
+		return h
+	}
+	return "localhost"
+}
+
 // durationYAML is a duration as a person writes it: 2h, 90m, 1m30s.
 func durationYAML(d time.Duration) string {
 	switch {
@@ -467,6 +534,15 @@ func (in *serverInit) initFlags() []string {
 	if in.media {
 		add("--media")
 	}
+	if in.slurm {
+		add("--slurm")
+		if in.slurmShared != "" && in.slurmShared != "here" {
+			add("--slurm-shared", in.slurmShared)
+		}
+		if in.slurmUsers == "site" {
+			add("--slurm-users", "site")
+		}
+	}
 	if in.evictAfterIdle > 0 {
 		add("--evict-after-idle", durationYAML(in.evictAfterIdle))
 	}

internal/cli/serverinit_choices_test.go

@@ -17,7 +17,7 @@ import (
 func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
 	in := &serverInit{
 		language: "tr", signupOpen: true, defaultRPM: 120, defaultTPM: 50000, maxKeyTTL: 30 * 24 * time.Hour,
-		media: true, evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
+		media: true, slurm: true, slurmShared: "here", slurmUsers: "llmgw", evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
 		judge: judgeSpec{Fail: "open", Categories: []string{"Violent", "PII"}, Replicas: 3, Budget: 3 * time.Second},
 	}
 	j := in.judgeFor()
@@ -41,6 +41,10 @@ func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
 		t.Errorf("limits %d %d %s", cfg.Auth.DefaultRPM, cfg.Auth.DefaultTPM, cfg.Auth.MaxKeyTTL)
 	case !cfg.Presets.Auto || cfg.Engines["comfyui"] == "":
 		t.Errorf("pictures and 3D: presets %v comfyui %q", cfg.Presets.Auto, cfg.Engines["comfyui"])
+	case !cfg.Slurm.Enabled || !cfg.Slurm.AllNodes || cfg.Slurm.Controller != localNodeID():
+		t.Errorf("slurm %+v", cfg.Slurm)
+	case cfg.Slurm.Shared.Path != "/shared" || cfg.Slurm.Shared.Server != localNodeID() || cfg.Slurm.Users != "llmgw":
+		t.Errorf("slurm's shared files and users %+v", cfg.Slurm)
 	case cfg.OnDemand.EvictAfterIdle != time.Minute || cfg.OnDemand.MinLifetime != 90*time.Second:
 		t.Errorf("on demand %+v", cfg.OnDemand)
 	}
@@ -95,14 +99,14 @@ func TestAStarterIsChosenByNameOrByItsFile(t *testing.T) {
 
 func TestTheChoicesAreTheirFlagsAgain(t *testing.T) {
 	in := &serverInit{admin: "ops@example.com", hosts: []string{"llm.example.com"}, postgres: true,
-		mirrorPath: "/data/models", noNode: true, language: "tr", media: true,
+		mirrorPath: "/data/models", language: "tr", media: true, slurm: true,
 		webhooks: []string{"https://hooks.slack.com/services/T/B/x, json:https://a.example.com/in"},
 		judge:    judgeSpec{Categories: []string{"Violent", "Jailbreak"}}, maxKeyTTL: 7 * 24 * time.Hour}
 	in.starter, _ = starterFrom("qwen3-4b")
 	got := strings.Join(in.initFlags(), " ")
 	want := "--admin ops@example.com --host llm.example.com --language tr --max-key-ttl 168h --postgres " +
-		"--mirror-dir /data/models --no-node --starter qwen3-4b --judge-categories Violent,Jailbreak " +
-		"--webhook https://hooks.slack.com/services/T/B/x --webhook json:https://a.example.com/in --media"
+		"--mirror-dir /data/models --starter qwen3-4b --judge-categories Violent,Jailbreak " +
+		"--webhook https://hooks.slack.com/services/T/B/x --webhook json:https://a.example.com/in --media --slurm"
 	if got != want {
 		t.Errorf("flags\n got %s\nwant %s", got, want)
 	}
@@ -124,6 +128,7 @@ func TestChoicesThatCannotBeInstalledAreRefused(t *testing.T) {
 		{maxKeyTTL: time.Minute},
 		{vip: "10.0.0.160/24"},
 		{standbys: true, vip: "10.0.0.160"},
+		{slurm: true, noNode: true},
 	} {
 		if err := in.checkChoices(); err == nil {
 			t.Errorf("taken: %+v", in)

internal/cli/serverinit_cluster.go

@@ -398,7 +398,9 @@ labels:
 	// checks its own llama.cpp, the CUDA build when it has a GPU, and a new
 	// machine is often busy with its own package updates first.
 	client := in.adminAPI()
-	last := ""
+	// Each engine's line once, and again only when it changes: the engines'
+	// states come back together on every look.
+	seen := map[string]string{}
 	err = waitFor(ctx, 20*time.Minute, "this node to be ready with an engine", func() (bool, error) {
 		var nodes []adminapi.NodeView
 		if err := client.get(ctx, "/api/v1/nodes", &nodes); err != nil {
@@ -412,9 +414,9 @@ labels:
 				return true, nil
 			}
 			for _, p := range n.Provision {
-				if line := fmt.Sprintf("%s %s: %s %s", p.Engine, p.Version, p.State, p.Detail); line != last {
-					fmt.Fprintf(in.out, "  ..    %s\n", strings.TrimSpace(line))
-					last = line
+				if line := provisionLine(p.Engine, p.Version, p.State, p.Detail); seen[p.Engine] != line {
+					fmt.Fprintf(in.out, "  ..    %s\n", line)
+					seen[p.Engine] = line
 				}
 				// Only llama.cpp is waited for: vLLM installs for minutes after,
 				// and one this node cannot run is not a reason to stop here.

internal/cli/status.go

@@ -156,6 +156,10 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
 	if s.Stopped {
 		fmt.Fprintln(out, "SERVICE   STOPPED: no model runs and requests are answered 503 (llmgw start)")
 	}
+	if s.NoAlerts {
+		fmt.Fprintln(out, "ALERTS    nowhere: a lost node or a judge down is told nobody "+
+			"(llmgw ctl config set notify.webhooks, then llmgw ctl notify test)")
+	}
 	if s.GuardOff {
 		fmt.Fprintln(out, "GUARD     off: no request is checked, whatever the policies say (llmgw ctl guard on)")
 	}

internal/cli/tui_actions.go

@@ -643,17 +643,30 @@ func (m consoleModel) nodeMenu(n adminapi.NodeView) *menuDialog {
 		menuItem{"Delete model files nothing uses", func(m *consoleModel) tea.Cmd {
 			return m.call("Unused files on "+n.ID, http.MethodPost, "/api/v1/weights/collect", map[string]string{"node": n.ID},
 				func(raw json.RawMessage) string {
-					var rs []struct {
-						Files int   `json:"files"`
-						Bytes int64 `json:"bytes"`
-					}
+					// What the node no longer lists, and what it kept.
+					var rs []adminapi.CollectResult
 					_ = json.Unmarshal(raw, &rs)
-					var files int
-					var b int64
+					var files, left int
+					var b, lb int64
+					var errs []string
 					for _, r := range rs {
-						files, b = files+r.Files, b+r.Bytes
+						files, b, left, lb = files+r.Files, b+r.Bytes, left+r.Left, lb+r.LeftBytes
+						if r.Error != "" {
+							errs = append(errs, r.Error)
+						}
 					}
-					return fmt.Sprintf("%d files, %.1f GB freed.", files, float64(b)/(1<<30))
+					if len(rs) == 0 {
+						return "Nothing on " + n.ID + " that no model uses."
+					}
+					say := fmt.Sprintf("%d files removed, %.1f GB freed.", files, float64(b)/(1<<30))
+					if left > 0 {
+						say += fmt.Sprintf(" %d files (%.1f GB) are still there: the node kept them, and its agent's log "+
+							"says why (journalctl -u llmgw-agent there).", left, float64(lb)/(1<<30))
+					}
+					if len(errs) > 0 {
+						say += " " + strings.Join(errs, "; ")
+					}
+					return say
 				})
 		}},
 		menuItem{"Remove from the cluster…", func(m *consoleModel) tea.Cmd {

internal/cli/tui_choices.go

@@ -57,7 +57,7 @@ type installForm struct {
 	standby *installPart
 	vipF    *setupField
 
-	control, store, mirror, node, starter, media, judge, firewall, alerts, limits, turns, bundle *installPart
+	control, store, mirror, node, starter, media, slurm, judge, firewall, alerts, limits, turns, bundle *installPart
 
 	admin, hosts, language, signup         *setupField
 	storeKind                              *setupField
@@ -282,6 +282,31 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
 		summary: func() string { return "ComfyUI; a picture model and a 3D model once a card fits" },
 	}
 
+	// --- Slurm ---
+	f.slurm = &installPart{
+		title: "Slurm", on: in.slurm,
+		about: "A Slurm cluster beside llmgw, for training and other batch work: slurmctld on this machine, " +
+			"slurmd on every node, those that join later too. The agents install munge and Slurm from the " +
+			"machines' own package sources, and on this machine slurmdbd, the jobs' records, with a MariaDB " +
+			"that answers this machine alone (Slurm keeps its records in MySQL or MariaDB only, not in the " +
+			"cluster's PostgreSQL). Its port 6819, with Slurm's others, answers the cluster's networks only.\n\n" +
+			"Each node stays llmgw's, drained in Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm: " +
+			"its models move off first. 'llmgw ctl slurm reclaim NODE' takes it back once its jobs end. Jobs " +
+			"are submitted on this machine with sbatch and srun.\n\n" +
+			"Every Slurm machine sees /shared, exported over NFS from this machine (homes, datasets, the jobs' " +
+			"output); 'llmgw ctl slurm shared' points it at the site's own storage instead. Each llmgw account " +
+			"is a Linux user there, the same UID on every machine, logging in with the SSH keys given in the " +
+			"portal; 'llmgw ctl slurm users off' leaves users to the site's directory.",
+		off:     "no Slurm; 'llmgw ctl slurm setup' adds it later",
+		summary: func() string { return "slurmctld here, slurmd on every node, /shared from here; none lent" },
+		needs: func() string {
+			if !f.node.on {
+				return "slurmctld runs on this machine's node: turn the node on, or Slurm off"
+			}
+			return ""
+		},
+	}
+
 	// --- the guardrails' judge ---
 	j := in.judgeFor()
 	f.judgeFail = choiceField("When no judge answers in time", []string{"refuse", "serve flagged"},
@@ -421,7 +446,7 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
 	if ha {
 		f.parts = append(f.parts, f.standby)
 	}
-	f.parts = append(f.parts, f.mirror, f.node, f.starter, f.media, f.judge, f.firewall,
+	f.parts = append(f.parts, f.mirror, f.node, f.starter, f.media, f.slurm, f.judge, f.firewall,
 		f.alerts, f.limits, f.turns, f.bundle)
 	return f
 }
@@ -653,6 +678,7 @@ func (f *installForm) apply(in *serverInit) error {
 	in.noModel = !f.starter.on
 	in.starter, in.starterCtx = starter, ctx
 	in.media = f.media.on
+	in.slurm = f.slurm.on
 	in.noGuardModel = !f.judge.on
 	in.judge = judge
 	in.noFirewall = !f.firewall.on

internal/cli/tui_console.go

@@ -24,6 +24,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/docgen"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
@@ -50,6 +51,7 @@ const (
 	cFederation
 	cMirror
 	cJobs
+	cSlurm
 	cLogs
 	cAccounts
 	cKeys
@@ -71,7 +73,7 @@ const (
 	cCount
 )
 
-var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "MCP servers", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Sign-in", "Audit", "Manual",
+var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "MCP servers", "Federation", "Mirror", "Jobs", "Slurm", "Logs", "Accounts", "API keys", "Organizations", "Sign-in", "Audit", "Manual",
 	"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Serve the guardrails' judge", "Upgrade the cluster",
 	"Restart all replicas", "Back up now", "Wipe this machine", "Leave"}
 
@@ -92,6 +94,8 @@ type consoleData struct {
 	// federation is the other llmgw clusters whose models are offered
 	// here; empty on a control plane without a store.
 	federation adminapi.FederationView
+	// slurm is the Slurm cluster beside llmgw; off when it has none.
+	slurm slurmsvc.Status
 	// presets is the models that come with llmgw as they stand here; nil
 	// from a control plane that cannot say.
 	presets *adminapi.PresetsView
@@ -284,6 +288,7 @@ func (m consoleModel) fetch() tea.Cmd {
 		_ = c.get(ctx, "/api/v1/jobs", &d.jobs)
 		_ = c.get(ctx, "/api/v1/audit?since=7d&limit=300", &d.audit)
 		_ = c.get(ctx, "/api/v1/federation", &d.federation)
+		_ = c.get(ctx, "/api/v1/slurm", &d.slurm)
 		var presets adminapi.PresetsView
 		if c.get(ctx, "/api/v1/presets", &presets) == nil {
 			d.presets = &presets
@@ -588,6 +593,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 		case cFederation:
 			m.dialog = "error"
 			m.dialogTitle, m.dialogText = peerDialog(m.data.federation.Peers[m.row])
+		case cSlurm:
+			m.menuD = m.slurmMenu(m.data.slurm.Nodes[m.row])
 		case cServe:
 			return m, m.fetchRepo(m.hf.models[m.row].ID)
 		case cAccounts:
@@ -1110,6 +1117,8 @@ func (m consoleModel) rowCount() int {
 		return len(m.data.audit)
 	case cFederation:
 		return len(m.data.federation.Peers)
+	case cSlurm:
+		return len(m.data.slurm.Nodes)
 	case cServe:
 		return len(m.hf.models)
 	case cAccounts:
@@ -1369,6 +1378,8 @@ func (m consoleModel) panel() (title, body, keys string) {
 		return m.auditPanel(pw)
 	case cFederation:
 		return m.federationPanel(pw)
+	case cSlurm:
+		return m.slurmPanel(pw)
 	case cMonitor:
 		return m.monitorPanel(pw)
 	case cServe:

internal/cli/tui_hf.go

@@ -213,10 +213,17 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
 			"before it is served.", modelplan.GB(p.KVBytesPerToken*1000))
 	}
 	note += "\nThe mirror fetches it and checks it, it joins the catalog, and a replica is started\nand asked to say hello."
+	// The longest context the first way holds, up to what the model was
+	// trained for: a choice of another way is checked again on submitting.
 	ctxDefault := "4096"
 	if a.Format == "safetensors" {
 		ctxDefault = strconv.Itoa(p.Context)
 	}
+	if len(opts) > 0 {
+		if most := opts[0].MostContext(p.KVBytesPerToken, p.Trained); most > 0 {
+			ctxDefault = strconv.Itoa(most)
+		}
+	}
 	if len(opts) > 0 && opts[0].Engine == "strata" {
 		ctxDefault = "32768"
 	}
@@ -249,6 +256,11 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
 				Description: v[3], Format: a.Format}
 			if a.Format == "gguf" {
 				req.File = a.Name
+				// A context longer than the default pool is the pool: one
+				// slot of it, where the server would refuse the request.
+				if ctxLen > modelserve.DefaultCtxPool {
+					req.CtxPool = ctxLen
+				}
 			}
 			o := opts[slices.Index(where, v[4])]
 			// Asked for a context other than planned: does it still fit?

internal/cli/tui_people.go

@@ -221,11 +221,11 @@ func (m consoleModel) userMenu(u store.User) *menuDialog {
 	items = append(items,
 		menuItem{"Add to an organization, team or project…", func(m *consoleModel) tea.Cmd {
 			m.openForm(&formDialog{title: "Membership for " + who,
-				note: "A member of a team or organization may use every project in it.",
+				note: "A member of a team or organization may use every project in it. On an organization, a manager also sees its page and an admin also changes it.",
 				fields: []formField{
 					choiceF("level", []string{"org", "team", "project"}, "team"),
 					textF("which", "", "Acme, Acme/ml or Acme/ml/training"),
-					choiceF("role", []string{store.RoleMember, "manager"}, store.RoleMember),
+					choiceF("role", store.MembershipRoles, store.RoleMember),
 				},
 				submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
 					if v[1] == "" {

internal/cli/tui_slurm.go

@@ -0,0 +1,170 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"time"
+
+	tea "github.com/charmbracelet/bubbletea"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
+)
+
+// The console's Slurm view: the Slurm cluster run beside llmgw (BACKLOG P),
+// each of its nodes and whose it is now, as `llmgw ctl slurm status` shows
+// them. A node is lent to Slurm, or taken back, from its row. Turning Slurm
+// on, and choosing its nodes, is `ctl slurm setup` or server.yaml's slurm
+// part: a form for it would be server.yaml written again.
+
+func (m consoleModel) slurmPanel(pw int) (string, string, string) {
+	keys := "→ or Enter open · ↑↓ choose · Q leave"
+	if m.inPanel {
+		keys = "↑↓ node · Enter lend or take back · ← back · Q leave"
+	}
+	st := m.data.slurm
+	head := "\n  " + sDim.Render("Slurm beside llmgw, on its nodes: training and batch jobs.")
+	if !st.Enabled {
+		return "Slurm", head + "\n\n  " + sDim.Render("Slurm is off. Turn it on with: "+
+			"llmgw ctl slurm setup --controller NODE --nodes NODE,... (or --all-nodes)"), keys
+	}
+	ctl := sDim.Render("not heard from yet")
+	switch c := st.Ctld; {
+	case c == nil:
+	case c.Error != "":
+		ctl = sBad.Render(c.Error)
+	case c.SlurmctldRunning && c.SlurmdbdRunning:
+		ctl = sOK.Render("slurmctld " + c.Version + " and slurmdbd running")
+	case c.SlurmctldRunning:
+		ctl = sOK.Render("slurmctld "+c.Version+" running") + ", " + sWarn.Render("slurmdbd not")
+	default:
+		ctl = sBad.Render("slurmctld not running")
+	}
+	head += fmt.Sprintf("\n  controller %s: %s", st.Controller, ctl)
+	head += "\n  " + sDim.Render(fmt.Sprintf("cluster %s, partition %s; sbatch and srun on %s", st.Cluster, st.Partition, st.Controller))
+	if st.Shared.Path == "" {
+		head += "\n  shared files: " + sWarn.Render("none") + sDim.Render(" (llmgw ctl slurm shared; output stays on the job's node)")
+	} else {
+		head += "\n  shared files: " + sharedLine(st.Shared)
+	}
+	if st.Users != "" {
+		head += "\n  " + fmt.Sprintf("users: %d from llmgw's accounts", st.UserCount) + sDim.Render(", SSH keys in the portal's Profile")
+	} else {
+		head += "\n  " + sDim.Render("users: the site's (llmgw ctl slurm users on makes them from llmgw's accounts)")
+	}
+	if line := containersLine(st); line != "" {
+		head += "\n  containers: " + line
+	}
+	head += "\n  " + sDim.Render("A node is llmgw's until lent; lent, its replicas move off first.")
+	if len(st.Waiting) > 0 {
+		head += "\n  " + sWarn.Render(fmt.Sprintf("not in slurm.conf until they report their hardware: %v", st.Waiting))
+	}
+	head += "\n\n"
+	if len(st.Nodes) == 0 {
+		return "Slurm", head + "  " + sDim.Render("No node runs slurmd."), keys
+	}
+	var rows [][]string
+	for _, n := range st.Nodes {
+		rows = append(rows, slurmRow(n))
+	}
+	return "Slurm", head + table([]string{"NODE", "WHOSE", "IN SLURM", "JOBS", "SLURMD", "SHARED", "NOTE"}, rows, m.row, pw, m.inPanel), keys
+}
+
+func slurmRow(n slurmsvc.NodeStatus) []string {
+	state, jobs, slurmd, shared, note := "-", "-", "-", "-", n.Describe()
+	if r := n.Status; r != nil {
+		state = or(r.State, "-")
+		switch r.Shared {
+		case "":
+		case "missing":
+			shared = sBad.Render(r.Shared)
+		default:
+			shared = sOK.Render(r.Shared)
+		}
+		jobs = fmt.Sprint(r.Jobs)
+		slurmd = sBad.Render("stopped")
+		if r.SlurmdRunning {
+			slurmd = sOK.Render("running")
+		}
+		if r.Error != "" {
+			note = sBad.Render(r.Error)
+		}
+	}
+	if !n.Connected {
+		note = sWarn.Render("agent not connected") + "; " + note
+	}
+	whose := string(n.Phase)
+	switch n.Phase {
+	case slurmsvc.Lent:
+		whose = sOK.Render("Slurm's")
+	case slurmsvc.Llmgw:
+		whose = "llmgw's"
+	case slurmsvc.Lending, slurmsvc.Reclaiming:
+		whose = sWarn.Render(whose)
+	}
+	return []string{n.Node, whose, state, jobs, slurmd, shared, note}
+}
+
+// slurmMenu is what can be done with one of Slurm's nodes.
+func (m consoleModel) slurmMenu(n slurmsvc.NodeStatus) *menuDialog {
+	var items []menuItem
+	if n.Phase == slurmsvc.Lent || n.Phase == slurmsvc.Lending {
+		items = append(items, menuItem{"Take back from Slurm…", func(m *consoleModel) tea.Cmd {
+			m.confirmD = &confirmDialog{title: "Take " + n.Node + " back",
+				text: "Slurm starts no new job there. llmgw places replicas on it again once the jobs running there have ended.",
+				yes:  func(m *consoleModel) tea.Cmd { return m.slurmLend(n.Node, false) }}
+			return nil
+		}})
+	} else {
+		items = append(items, menuItem{"Lend to Slurm…", func(m *consoleModel) tea.Cmd {
+			text := "Its replicas move off first, each replacement up before the original stops; then Slurm runs jobs there."
+			if n.Replicas > 0 {
+				text = fmt.Sprintf("Its %d replica(s) move off first, each replacement up before the original stops; "+
+					"a model with nowhere else to go stops serving. Then Slurm runs jobs there.", n.Replicas)
+			}
+			m.confirmD = &confirmDialog{title: "Lend " + n.Node + " to Slurm", text: text,
+				yes: func(m *consoleModel) tea.Cmd { return m.slurmLend(n.Node, true) }}
+			return nil
+		}})
+	}
+	return &menuDialog{title: n.Node, items: items}
+}
+
+// slurmLend puts a node in slurm.lent, or takes it out, as `llmgw ctl slurm
+// lend|reclaim` does: the newest settings, changed and put back.
+func (m *consoleModel) slurmLend(node string, add bool) tea.Cmd {
+	title, note, done := "Lend "+node, "lent to Slurm from the console", node+" is being lent to Slurm: its replicas move off first."
+	if !add {
+		title, note, done = "Take back "+node, "taken back from Slurm in the console",
+			node+" is being taken back: llmgw has it once its Slurm jobs end."
+	}
+	m.busy = title
+	c := m.client
+	return func() tea.Msg {
+		ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+		defer cancel()
+		var cur adminapi.SettingsView
+		if err := c.get(ctx, "/api/v1/settings", &cur); err != nil {
+			return actionDoneMsg{title: title, err: err}
+		}
+		doc, err := settings.FromBytes("settings", []byte(cur.Body))
+		if err == nil {
+			err = lendChange(doc, []string{node}, add)
+		}
+		var body []byte
+		if err == nil {
+			body, err = doc.Bytes()
+		}
+		if err != nil {
+			return actionDoneMsg{title: title, err: err}
+		}
+		req, _ := json.Marshal(adminapi.SettingsRequest{Base: cur.Version, Body: string(body), Note: note})
+		var v adminapi.SettingsView
+		if err := c.put(ctx, "/api/v1/settings", req, &v); err != nil {
+			return actionDoneMsg{title: title, err: err}
+		}
+		return actionDoneMsg{title: title, text: done}
+	}
+}

internal/cli/tui_slurm_test.go

@@ -0,0 +1,94 @@
+package cli
+
+import (
+	"encoding/json"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"sync"
+	"testing"
+
+	tea "github.com/charmbracelet/bubbletea"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
+)
+
+func TestTheConsoleLendsANodeToSlurmFromItsRow(t *testing.T) {
+	var mu sync.Mutex
+	var put adminapi.SettingsRequest
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch {
+		case r.Method == http.MethodGet && r.URL.Path == "/api/v1/settings":
+			_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 7,
+				Body: "slurm:\n  enabled: true\n  controller: node06\n  nodes: [node05, cuda01]\n"})
+		case r.Method == http.MethodPut && r.URL.Path == "/api/v1/settings":
+			b, _ := io.ReadAll(r.Body)
+			mu.Lock()
+			_ = json.Unmarshal(b, &put)
+			mu.Unlock()
+			_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 8})
+		default:
+			_, _ = w.Write([]byte("[]"))
+		}
+	}))
+	t.Cleanup(srv.Close)
+	m := newConsoleModel(srv.URL, "/nonexistent/server.yaml")
+	m.loaded, m.menu, m.inPanel = true, cSlurm, true
+	m.data.slurm = slurmsvc.Status{Enabled: true, Cluster: "llmgw", Controller: "node06", Partition: "gpu",
+		Ctld:   &slurmsvc.Reported{SlurmctldRunning: true, SlurmdbdRunning: true, Version: "23.11.4"},
+		Shared: config.SharedStorage{Path: "/shared", Server: "node06"}, Users: "llmgw", UserCount: 2,
+		Nodes: []slurmsvc.NodeStatus{
+			{Node: "node05", Phase: slurmsvc.Lent, Connected: true,
+				Status: &slurmsvc.Reported{State: "MIXED", Jobs: 2, SlurmdRunning: true, Shared: "mounted"}},
+			{Node: "cuda01", Phase: slurmsvc.Llmgw, Replicas: 3, Connected: true,
+				Status: &slurmsvc.Reported{State: "IDLE+DRAIN", SlurmdRunning: true}},
+		}}
+	var tm tea.Model = m
+	view := tm.View()
+	for _, want := range []string{"slurmctld 23.11.4 and slurmdbd running", "/shared, exported over NFS by node06", "users: 2", "node05", "Slurm's", "MIXED", "cuda01", "llmgw's"} {
+		if !strings.Contains(view, want) {
+			t.Errorf("the Slurm view has no %q:\n%s", want, view)
+		}
+	}
+
+	// cuda01, the second row: lent, after saying its replicas move.
+	tm = press(tm, "down", "enter")
+	if !strings.Contains(tm.View(), "Lend to Slurm") {
+		t.Fatalf("no lend in the menu:\n%s", tm.View())
+	}
+	tm = press(tm, "enter")
+	if !strings.Contains(tm.View(), "3 replica(s) move off first") {
+		t.Fatalf("the question does not say what moves:\n%s", tm.View())
+	}
+	tm = pressRun(tm, "y")
+	mu.Lock()
+	defer mu.Unlock()
+	if put.Base != 7 || !strings.Contains(put.Body, "lent:") || !strings.Contains(put.Body, "cuda01") {
+		t.Fatalf("the settings put back: %+v", put)
+	}
+	if !strings.Contains(tm.View(), "cuda01 is being lent to Slurm") {
+		t.Errorf("not said:\n%s", tm.View())
+	}
+}
+
+func TestTheConsoleTakesALentNodeBack(t *testing.T) {
+	m, _ := consoleWith(t, cSlurm)
+	m.data.slurm = slurmsvc.Status{Enabled: true, Controller: "node06",
+		Nodes: []slurmsvc.NodeStatus{{Node: "node05", Phase: slurmsvc.Lent, Connected: true}}}
+	var tm tea.Model = m
+	tm = press(tm, "enter")
+	if v := tm.View(); !strings.Contains(v, "Take back from Slurm") || strings.Contains(v, "Lend to Slurm") {
+		t.Fatalf("a lent node's menu:\n%s", v)
+	}
+}
+
+func TestTheConsoleSaysHowToTurnSlurmOn(t *testing.T) {
+	m, _ := consoleWith(t, cSlurm)
+	var tm tea.Model = m
+	if !strings.Contains(tm.View(), "llmgw ctl slurm setup") {
+		t.Fatalf("Slurm off says nothing of turning it on:\n%s", tm.View())
+	}
+}

internal/cli/users.go

@@ -171,7 +171,10 @@ func newUserStateCmd(open userOpener, verb string, state store.UserState, short
 				if err := s.DeleteUserSessions(cmd.Context(), u.ID); err != nil {
 					return err
 				}
-				cmd.PrintErrln("Their sessions were closed. Their API keys still work — revoke those separately.")
+				// The keys are refused by their owner's state, which every
+				// router reads again within auth.cache_ttl.
+				cmd.PrintErrln("Their sessions were closed. Their API keys are refused within auth.cache_ttl " +
+					"(30 s by default), and work again if they are reinstated.")
 			}
 			fmt.Fprintf(cmd.OutOrStdout(), "%s is now %s\n", u.Email, state)
 			return nil
@@ -391,7 +394,8 @@ func newUserJoinCmd(open userOpener) *cobra.Command {
 		Long: "A person can be a member of any number of them. A member of a team or\n" +
 			"organization may use every project in it.",
 		Example: "  llmgw users join ayse@example.com --team Acme/ml\n" +
-			"  llmgw users join ayse@example.com --project Scania/research/vision",
+			"  llmgw users join ayse@example.com --project Scania/research/vision\n" +
+			"  llmgw users join lars@example.com --org Scania --role admin",
 		Args: cobra.ExactArgs(1),
 		RunE: func(cmd *cobra.Command, args []string) error {
 			s, err := open(cmd)
@@ -415,7 +419,8 @@ func newUserJoinCmd(open userOpener) *cobra.Command {
 		},
 	}
 	where.bind(cmd)
-	cmd.Flags().StringVar(&role, "role", store.RoleMember, "member, or manager: with --org, may see the organization's page")
+	cmd.Flags().StringVar(&role, "role", store.RoleMember,
+		"member; with --org also manager, who sees the organization's page, or admin, who also changes its teams, projects, people and keys")
 	cmd.Flags().StringVar(&by, "by", "", "who added them, for the record")
 	return cmd
 }

internal/config/config.go

@@ -158,6 +158,10 @@ type Server struct {
 	// Firewall turns llmgw's firewall on every machine on or off.
 	Firewall FirewallSettings `koanf:"firewall"`
 
+	// Slurm is a Slurm cluster run beside llmgw on some of its nodes, and
+	// which of them it may use now.
+	Slurm SlurmSettings `koanf:"slurm"`
+
 	// Presets is how the models that come with llmgw reach the catalog.
 	Presets PresetSettings `koanf:"presets"`
 
@@ -466,6 +470,15 @@ type Auth struct {
 	Pricing      map[string]Price `koanf:"pricing"`
 	DefaultPrice Price            `koanf:"default_price"`
 
+	// GPUHourCost is what an hour of one GPU costs the operator, in the
+	// units budgets are in: the hardware written off over its years, its
+	// power and room, and the people who run it, over the GPUs and the
+	// hours of a month. With CoreHourCost for a CPU core, it is what
+	// `llmgw ctl prices suggest` prices each model from: the hours its
+	// replicas held, idle ones too, over the tokens it served.
+	GPUHourCost  float64 `koanf:"gpu_hour_cost"`
+	CoreHourCost float64 `koanf:"core_hour_cost"`
+
 	// FlushInterval and BatchSize decide when usage records are written.
 	FlushInterval time.Duration `koanf:"flush_interval"`
 	BatchSize     int           `koanf:"batch_size"`
@@ -960,6 +973,9 @@ func (s Server) Validate() error {
 	if _, err := s.Facility.Policy(); err != nil {
 		return err
 	}
+	if err := s.Slurm.Check(); err != nil {
+		return err
+	}
 	if s.Control.Timezone != "" {
 		if _, err := time.LoadLocation(s.Control.Timezone); err != nil {
 			return fmt.Errorf("control.timezone %q: %w", s.Control.Timezone, err)

internal/config/slurm.go

@@ -0,0 +1,204 @@
+package config
+
+import (
+	"fmt"
+	"path"
+	"regexp"
+	"slices"
+	"strings"
+)
+
+// SlurmSettings is a Slurm cluster run beside llmgw on some of its nodes
+// (BACKLOG P), for training and other batch work llmgw does not schedule.
+// The leader tells each named node's agent what to run; the agents install
+// it from the machine's package sources and keep it going.
+type SlurmSettings struct {
+	// Enabled runs Slurm on the nodes below. False stops llmgw keeping it;
+	// what was installed is left installed, and every node is llmgw's.
+	Enabled bool `koanf:"enabled"`
+
+	// ClusterName is Slurm's ClusterName. Empty is "llmgw".
+	ClusterName string `koanf:"cluster_name"`
+
+	// Controller is the llmgw node whose agent runs slurmctld. It needs no
+	// GPU, and need not be one of Nodes.
+	Controller string `koanf:"controller"`
+
+	// Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained
+	// in Slurm, until it is also in Lent.
+	Nodes []string `koanf:"nodes"`
+
+	// AllNodes runs slurmd on every llmgw node, those that join later too,
+	// besides any named in Nodes. Each is llmgw's until it is lent, as any.
+	AllNodes bool `koanf:"all_nodes"`
+
+	// Lent are the nodes of Nodes Slurm may run jobs on now. A node put here
+	// is drained of llmgw's replicas first, as `ctl drain` does, and resumed
+	// in Slurm once none is left; a node taken out is drained in Slurm and
+	// given back to llmgw when its last job ends.
+	Lent []string `koanf:"lent"`
+
+	// Partition is the one partition every node is in. Empty is "gpu".
+	Partition string `koanf:"partition"`
+
+	// Shared is the file system every machine of the Slurm cluster sees at
+	// the same path: home directories, datasets, the jobs' output and
+	// checkpoints. Empty path: none, and a batch job's output stays on the
+	// node it ran on.
+	Shared SharedStorage `koanf:"shared"`
+
+	// Users is who keeps the Linux users that run jobs, the same on every
+	// machine of the Slurm cluster: "llmgw" makes one for each active llmgw
+	// account (its login from the e-mail address, a UID of llmgw's from
+	// 200000 up, its SSH keys from the portal), let in to the controller,
+	// and to a node only while a job of theirs runs there (pam_slurm_adopt);
+	// empty leaves users to the site (a directory through sssd, or by hand).
+	Users string `koanf:"users"`
+
+	// Containers lets jobs run in containers: `srun --container-image
+	// nvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),
+	// the GPUs and the shared path inside. The agents install both on
+	// every Slurm machine; jobs take it once every machine has them.
+	Containers bool `koanf:"containers"`
+}
+
+// SharedStorage is where the Slurm cluster's shared files come from: the
+// site's NFS export (NFS), an NFS export llmgw keeps on one of its nodes
+// (Server), or, with neither, a file system the site mounts itself
+// (Lustre, BeeGFS, GPFS), which llmgw only looks for.
+type SharedStorage struct {
+	// Path is where every machine sees it, as /shared.
+	Path string `koanf:"path"`
+	// NFS is the site's export, host:/path, mounted at Path on every
+	// machine of the Slurm cluster.
+	NFS string `koanf:"nfs"`
+	// Server is the llmgw node, the controller or one of the nodes, that
+	// exports Path over NFS to the others: for a site with no storage of
+	// its own. One machine: not highly available, and no faster than its
+	// disk and network.
+	Server string `koanf:"server"`
+}
+
+// UsersLlmgw has llmgw make the Slurm cluster's users from its accounts.
+const UsersLlmgw = "llmgw"
+
+// HomeDir is where a user's home directory is: under the shared path when
+// there is one, else on each machine's own disk.
+func (s SlurmSettings) HomeDir(login string) string {
+	if s.Shared.Path != "" {
+		return path.Join(s.Shared.Path, "home", login)
+	}
+	return path.Join("/home", login)
+}
+
+// nfsExport is host:/path.
+var nfsExport = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.:\[\]-]*:/[A-Za-z0-9_./-]*$`)
+
+// sharedPath is an absolute path of plain names, which a mount unit's name
+// is made from.
+var sharedPath = regexp.MustCompile(`^(/[A-Za-z0-9_.-]+)+$`)
+
+func (s SlurmSettings) checkShared() error {
+	sh := s.Shared
+	if sh.Path == "" {
+		if sh.NFS != "" || sh.Server != "" {
+			return fmt.Errorf("slurm.shared.path is empty: say where every machine sees the shared files, as /shared")
+		}
+		return nil
+	}
+	if !sharedPath.MatchString(sh.Path) || strings.Contains(sh.Path, "/.") {
+		return fmt.Errorf("slurm.shared.path %q is not an absolute path of plain names, as /shared", sh.Path)
+	}
+	for _, sys := range []string{"/", "/etc", "/usr", "/var", "/var/lib", "/home", "/root", "/boot", "/opt", "/srv", "/tmp", "/proc", "/sys", "/dev", "/run"} {
+		if path.Clean(sh.Path) == sys {
+			return fmt.Errorf("slurm.shared.path %s is the system's own directory: take one of its own, as /shared", sh.Path)
+		}
+	}
+	if sh.NFS != "" && sh.Server != "" {
+		return fmt.Errorf("slurm.shared names both the site's export (nfs) and an llmgw node to export it (server): one of them")
+	}
+	if sh.NFS != "" && !nfsExport.MatchString(sh.NFS) {
+		return fmt.Errorf("slurm.shared.nfs %q is not host:/path", sh.NFS)
+	}
+	if sh.Server != "" && sh.Server != s.Controller && !s.AllNodes && !slices.Contains(s.Nodes, sh.Server) {
+		return fmt.Errorf("slurm.shared.server %q is neither the controller nor one of slurm.nodes", sh.Server)
+	}
+	return nil
+}
+
+// slurmName is what Slurm takes as a node, cluster or partition name.
+var slurmName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]{0,62}$`)
+
+// Cluster is ClusterName or its default.
+func (s SlurmSettings) Cluster() string {
+	if s.ClusterName == "" {
+		return "llmgw"
+	}
+	return s.ClusterName
+}
+
+// PartitionName is Partition or its default.
+func (s SlurmSettings) PartitionName() string {
+	if s.Partition == "" {
+		return "gpu"
+	}
+	return s.Partition
+}
+
+// IsLent is whether Slurm may use the node now.
+func (s SlurmSettings) IsLent(node string) bool {
+	return s.Enabled && (s.AllNodes || slices.Contains(s.Nodes, node)) && slices.Contains(s.Lent, node)
+}
+
+// WithNodes is the settings with every node of ids in Nodes when AllNodes:
+// the nodes that run slurmd, as the leader's inventory has them.
+func (s SlurmSettings) WithNodes(ids []string) SlurmSettings {
+	if !s.AllNodes {
+		return s
+	}
+	nodes := slices.Clone(s.Nodes)
+	for _, id := range ids {
+		if !slices.Contains(nodes, id) {
+			nodes = append(nodes, id)
+		}
+	}
+	s.Nodes = nodes
+	return s
+}
+
+// Check refuses settings Slurm could not run on.
+func (s SlurmSettings) Check() error {
+	if !s.Enabled {
+		return nil
+	}
+	if s.Controller == "" {
+		return fmt.Errorf("slurm.controller is empty: name the node that runs slurmctld")
+	}
+	if !slurmName.MatchString(s.Controller) {
+		return fmt.Errorf("slurm.controller %q is not a name Slurm takes", s.Controller)
+	}
+	for _, n := range []struct{ key, v string }{{"slurm.cluster_name", s.ClusterName}, {"slurm.partition", s.Partition}} {
+		if n.v != "" && !slurmName.MatchString(n.v) {
+			return fmt.Errorf("%s %q is not a name Slurm takes: letters, digits, _ . -", n.key, n.v)
+		}
+	}
+	seen := map[string]bool{}
+	for i, n := range s.Nodes {
+		if !slurmName.MatchString(n) {
+			return fmt.Errorf("slurm.nodes[%d] %q is not a name Slurm takes", i, n)
+		}
+		if seen[n] {
+			return fmt.Errorf("slurm.nodes names %s twice", n)
+		}
+		seen[n] = true
+	}
+	for i, n := range s.Lent {
+		if !seen[n] && !s.AllNodes {
+			return fmt.Errorf("slurm.lent[%d] %q is not in slurm.nodes: only a node that runs slurmd can be lent", i, n)
+		}
+	}
+	if s.Users != "" && s.Users != UsersLlmgw {
+		return fmt.Errorf("slurm.users %q: %q has llmgw make them, empty leaves them to the site", s.Users, UsersLlmgw)
+	}
+	return s.checkShared()
+}

internal/config/slurm_test.go

@@ -0,0 +1,96 @@
+package config
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestSlurmCheck(t *testing.T) {
+	ok := SlurmSettings{Enabled: true, Controller: "node06", Nodes: []string{"cuda01", "site2"}, Lent: []string{"cuda01"}}
+	if err := ok.Check(); err != nil {
+		t.Fatal(err)
+	}
+	if !ok.IsLent("cuda01") || ok.IsLent("site2") || ok.IsLent("node06") {
+		t.Error("IsLent is wrong")
+	}
+	if ok.Cluster() != "llmgw" || ok.PartitionName() != "gpu" {
+		t.Error("defaults are wrong")
+	}
+	if (SlurmSettings{Lent: []string{"x"}}).Check() != nil {
+		t.Error("settings that are off were checked")
+	}
+	for want, s := range map[string]SlurmSettings{
+		"controller is empty": {Enabled: true},
+		"not a name":          {Enabled: true, Controller: "bad name"},
+		"twice":               {Enabled: true, Controller: "c", Nodes: []string{"a", "a"}},
+		"not in slurm.nodes":  {Enabled: true, Controller: "c", Nodes: []string{"a"}, Lent: []string{"b"}},
+		"slurm.partition":     {Enabled: true, Controller: "c", Partition: "a b"},
+	} {
+		if err := s.Check(); err == nil || !strings.Contains(err.Error(), want) {
+			t.Errorf("%+v: got %v, want %q", s, err, want)
+		}
+	}
+	if (SlurmSettings{Enabled: false, Lent: []string{"a"}}).IsLent("a") {
+		t.Error("a node is lent with Slurm off")
+	}
+}
+
+func TestSlurmShared(t *testing.T) {
+	base := SlurmSettings{Enabled: true, Controller: "c", Nodes: []string{"a"}}
+	for _, sh := range []SharedStorage{{}, {Path: "/shared"}, {Path: "/shared", NFS: "filer.site:/export/hpc"},
+		{Path: "/data/hpc", Server: "c"}, {Path: "/shared", Server: "a"}} {
+		s := base
+		s.Shared = sh
+		if err := s.Check(); err != nil {
+			t.Errorf("%+v refused: %v", sh, err)
+		}
+	}
+	for want, sh := range map[string]SharedStorage{
+		"path is empty":  {NFS: "f:/x"},
+		"plain names":    {Path: "shared"},
+		"system's own":   {Path: "/home"},
+		"one of them":    {Path: "/shared", NFS: "f:/x", Server: "c"},
+		"not host:/path": {Path: "/shared", NFS: "f/x"},
+		"neither":        {Path: "/shared", Server: "z"},
+	} {
+		s := base
+		s.Shared = sh
+		if err := s.Check(); err == nil || !strings.Contains(err.Error(), want) {
+			t.Errorf("%+v: got %v, want %q", sh, err, want)
+		}
+	}
+	s := base
+	s.Shared = SharedStorage{Path: "/a/../etc"}
+	if s.Check() == nil {
+		t.Error("a path with .. taken")
+	}
+	s.Shared, s.Users = SharedStorage{}, "ldap"
+	if s.Check() == nil {
+		t.Error("slurm.users ldap taken")
+	}
+	s.Users, s.Shared = UsersLlmgw, SharedStorage{Path: "/shared"}
+	if s.Check() != nil || s.HomeDir("ayse") != "/shared/home/ayse" {
+		t.Errorf("home %s", s.HomeDir("ayse"))
+	}
+	s.Shared = SharedStorage{}
+	if s.HomeDir("ayse") != "/home/ayse" {
+		t.Errorf("home %s", s.HomeDir("ayse"))
+	}
+}
+
+func TestSlurmAllNodes(t *testing.T) {
+	s := SlurmSettings{Enabled: true, Controller: "c", AllNodes: true, Lent: []string{"joined"}}
+	if err := s.Check(); err != nil {
+		t.Fatalf("a node lent with all_nodes refused: %v", err)
+	}
+	if !s.IsLent("joined") {
+		t.Fatal("not lent with all_nodes")
+	}
+	if got := s.WithNodes([]string{"a", "joined"}).Nodes; len(got) != 2 {
+		t.Fatalf("nodes %v", got)
+	}
+	s.AllNodes = false
+	if s.Check() == nil || len(s.WithNodes([]string{"a"}).Nodes) != 0 {
+		t.Fatal("all_nodes off still takes the inventory's nodes")
+	}
+}

internal/control/adminapi/api.go

@@ -78,7 +78,9 @@ type API struct {
 	// tenancy is organizations, keys and people; nil without a store.
 	tenancy TenancyStore
 	// train is adapter training; nil without a store and agents.
-	train    *TrainAdmin
+	train *TrainAdmin
+	// slurm is the Slurm cluster beside llmgw; nil without a store and agents.
+	slurm    SlurmService
 	settings *Settings
 	auth     *AuthAdmin
 	mcp      MCPStore
@@ -96,6 +98,14 @@ type API struct {
 	usage     Usage
 	quotas    Quotas
 	ctl       *control.Controller
+	// slurmUsers are the Linux users made from the accounts; nil without a store.
+	slurmUsers SlurmUsers
+	// prices are suggested from what each model held and served; nil
+	// without a store.
+	prices   PriceStore
+	priceCfg func() PriceConfig
+	// webhooks counts the settings' notify.webhooks; nil does not say.
+	webhooks func() int
 }
 
 // New builds the API over a controller.
@@ -136,6 +146,8 @@ func (a *API) Register(mux Mux) {
 	a.registerKB(mux)
 	a.registerTenancy(mux)
 	a.registerTrain(mux)
+	a.registerSlurm(mux)
+	a.registerPrices(mux)
 	a.registerGuard(mux)
 	a.registerSettings(mux)
 	a.registerAuth(mux)

internal/control/adminapi/notify.go

@@ -6,6 +6,10 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
 )
 
+// SetWebhooks has the summary say when the settings name no webhook:
+// count is how many they name now.
+func (a *API) SetWebhooks(count func() int) { a.webhooks = count }
+
 // SetNotifier gives the API the webhooks `ctl notify test` reaches.
 func (a *API) SetNotifier(n *notify.Notifier) { a.notifier = n }
 

internal/control/adminapi/prices.go

@@ -0,0 +1,108 @@
+package adminapi
+
+import (
+	"context"
+	"net/http"
+	"strconv"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// PriceStore is what a price is suggested from: what each model held of
+// the cluster, and the tokens it served.
+type PriceStore interface {
+	HeldSince(ctx context.Context, since time.Time) ([]store.Held, time.Time, error)
+	UsageByModel(ctx context.Context, since time.Time) ([]store.ModelUsage, error)
+}
+
+// PriceConfig is the cost basis and the prices in force.
+type PriceConfig struct {
+	Basis   accounting.Basis
+	Current map[string]accounting.Price
+	Default accounting.Price
+}
+
+// PriceSuggestions is a model's price from its cost, for each model.
+type PriceSuggestions struct {
+	// Days is the window; RecordedFrom the first hour held is known for,
+	// later than the window's start on a cluster that began keeping it
+	// since, which makes the window shorter than asked.
+	Days         int               `json:"days"`
+	Since        time.Time         `json:"since"`
+	RecordedFrom time.Time         `json:"recorded_from,omitempty"`
+	GPUHourCost  float64           `json:"gpu_hour_cost"`
+	CoreHourCost float64           `json:"core_hour_cost"`
+	Models       []PriceSuggestion `json:"models"`
+}
+
+// PriceSuggestion is one model's.
+type PriceSuggestion struct {
+	accounting.Suggestion
+	Current accounting.Price `json:"current"`
+}
+
+// SetPrices gives the API what prices are suggested from.
+func (a *API) SetPrices(st PriceStore, cfg func() PriceConfig) { a.prices, a.priceCfg = st, cfg }
+
+func (a *API) registerPrices(mux Mux) {
+	mux.HandleFunc("GET /api/v1/prices/suggest", a.suggestPrices)
+}
+
+func (a *API) suggestPrices(w http.ResponseWriter, r *http.Request) {
+	if a.prices == nil || a.priceCfg == nil {
+		writeError(w, http.StatusNotImplemented, "prices are suggested from the store, which this control plane runs without")
+		return
+	}
+	days := 30
+	if v := r.URL.Query().Get("days"); v != "" {
+		n, err := strconv.Atoi(v)
+		if err != nil || n < 1 || n > 366 {
+			writeError(w, http.StatusBadRequest, "days is a whole number of days, 1 to 366")
+			return
+		}
+		days = n
+	}
+	cfg := a.priceCfg()
+	if cfg.Basis.GPUHour <= 0 && cfg.Basis.CoreHour <= 0 {
+		writeError(w, http.StatusConflict, "no cost to price from: set auth.gpu_hour_cost (and auth.core_hour_cost for "+
+			"models on CPUs), what an hour of one costs, in the units budgets are in")
+		return
+	}
+	since := time.Now().Add(-time.Duration(days) * 24 * time.Hour)
+	rows, first, err := a.prices.HeldSince(r.Context(), since)
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	// Tokens from when holding was first recorded, so both sides cover the
+	// same hours.
+	from := since
+	if first.After(from) {
+		from = first
+	}
+	usage, err := a.prices.UsageByModel(r.Context(), from)
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	held := map[string]accounting.Held{}
+	for _, h := range rows {
+		held[h.Model] = accounting.Held{GPUSeconds: h.GPUSeconds, CoreSeconds: h.CoreSeconds}
+	}
+	served := map[string]accounting.Served{}
+	for _, u := range usage {
+		served[u.Model] = accounting.Served{Prompt: u.Prompt, Completion: u.Completion}
+	}
+	out := PriceSuggestions{Days: days, Since: since.UTC(), RecordedFrom: first,
+		GPUHourCost: cfg.Basis.GPUHour, CoreHourCost: cfg.Basis.CoreHour, Models: []PriceSuggestion{}}
+	for _, s := range accounting.Suggest(held, served, cfg.Basis) {
+		cur, ok := cfg.Current[s.Model]
+		if !ok {
+			cur = cfg.Default
+		}
+		out.Models = append(out.Models, PriceSuggestion{Suggestion: s, Current: cur})
+	}
+	writeJSON(w, http.StatusOK, out)
+}

internal/control/adminapi/purge.go

@@ -216,7 +216,7 @@ func (a *API) purgeFromNodes(ctx context.Context, sources []string, others []cat
 		names[filepath.Base(s)] = true
 	}
 	wanted := inUse(others)
-	var out []CollectResult
+	var rs []removal
 	for _, n := range a.ctl.Inventory().Nodes() {
 		size := map[string]int64{}
 		have := make([]string, 0, len(n.Weights))
@@ -235,15 +235,14 @@ func (a *API) purgeFromNodes(ctx context.Context, sources []string, others []cat
 		if len(paths) == 0 {
 			continue
 		}
-		res := CollectResult{Node: n.ID, Files: len(paths)}
-		for _, p := range paths {
-			res.Bytes += size[p]
+		rs = append(rs, removal{node: n.ID, paths: paths, size: size})
+	}
+	out := a.removeConfirmed(ctx, rm, rs)
+	for _, r := range out {
+		if r.Error != "" || r.Left > 0 {
+			slog.Warn("a purged model's copies were not all deleted on a node", "node", r.Node,
+				"left", r.Left, "err", r.Error)
 		}
-		if err := rm.RemoveWeights(ctx, n.ID, paths); err != nil {
-			res.Error, res.Files, res.Bytes = err.Error(), 0, 0
-			slog.Warn("a purged model's copies could not be deleted on a node", "node", n.ID, "err", err)
-		}
-		out = append(out, res)
 	}
 	return out
 }

internal/control/adminapi/routes.go

@@ -12,6 +12,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/federation"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
@@ -84,6 +85,7 @@ var PathParams = map[string]string{
 	"projects/{id}":     "The project's ID.",
 	"keys/{id}":         "The key's ID, as listed (not the key itself).",
 	"users/{id}":        "The person's ID or email.",
+	"users/{user}":      "The person's ID or email.",
 	"artifacts/{kind}":  "What the job made: peft, a gzipped tar of the PEFT adapter, or gguf, the adapter for llama.cpp.",
 }
 
@@ -392,6 +394,27 @@ var Routes = []Route{
 		Description: "Their keys in projects they no longer reach are revoked.",
 		Request:     MembershipRequest{}, Response: UserView{}},
 
+	{Method: "GET", Path: "/api/v1/slurm", Summary: "The Slurm cluster beside llmgw, and where each of its nodes is",
+		Description: "From the cluster's settings (slurm.*) and what each node's agent last said. A node's phase is llmgw " +
+			"(drained in Slurm, no job left), lending (its replicas moving off), lent, reclaiming (drained in Slurm, jobs " +
+			"still running) or unknown (its agent has not said).",
+		Response: slurmsvc.Status{}},
+	{Method: "GET", Path: "/api/v1/slurm/users", Summary: "The Linux users made from llmgw's accounts for the Slurm cluster",
+		Description: "Every active account, with its login, UID and SSH keys; an account is given its login and UID the first " +
+			"time it is listed. The machines have them when slurm.users is \"llmgw\".",
+		Response: []slurmsvc.User{}},
+	{Method: "PUT", Path: "/api/v1/slurm/users/{user}/ssh-keys", Summary: "Replace an account's SSH public keys",
+		Description: "{user} is an ID or e-mail. keys is OpenSSH public keys, one a line, without options; empty takes them " +
+			"all away. The machines have them within a minute.",
+		Request: SlurmKeysRequest{}, Response: slurmsvc.User{}},
+
+	{Method: "GET", Path: "/api/v1/prices/suggest", Summary: "Each model's price, from what it held of the cluster",
+		Description: "days (30 if not given) is the window. A model's cost is the GPU- and core-hours its replicas held, idle " +
+			"ones too, at auth.gpu_hour_cost and auth.core_hour_cost; spread over its tokens, an output token four input " +
+			"ones and a cached prompt token a tenth of one, it is price, in micro-units a thousand tokens as " +
+			"auth.pricing takes them. current is the price in force. 409 with no cost set.",
+		Response: PriceSuggestions{}},
+
 	{Method: "POST", Path: "/api/v1/train/jobs", Summary: "Queue a LoRA adapter's training on a node",
 		Description: "The node trains when its agent is there, nothing else trains on it, and a card has gpu_memory_mib free; " +
 			"the pairs are NDJSON of {\"q\", \"a\"}. An adapter learns how answers sound far better than what they say.",

internal/control/adminapi/slurm.go

@@ -0,0 +1,119 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"net/http"
+	"strings"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// SlurmService is the Slurm cluster beside llmgw (BACKLOG P), as the leader
+// keeps it.
+type SlurmService interface {
+	Status() slurmsvc.Status
+	UsersChanged()
+}
+
+// SlurmUsers are the Linux users llmgw makes from its accounts.
+type SlurmUsers interface {
+	SlurmUsers(ctx context.Context) ([]store.SlurmUser, error)
+	SlurmUserByID(ctx context.Context, userID string) (store.SlurmUser, error)
+	SetSlurmSSHKeys(ctx context.Context, userID string, keys []string) error
+}
+
+// SetSlurmUsers enables the Slurm users' endpoints.
+func (a *API) SetSlurmUsers(u SlurmUsers) { a.slurmUsers = u }
+
+// SlurmKeysRequest is a person's SSH public keys, one a line, as
+// authorized_keys has them; empty takes them all away.
+type SlurmKeysRequest struct {
+	Keys string `json:"keys"`
+}
+
+// SlurmUserView is the user from a store row.
+func SlurmUserView(u store.SlurmUser) slurmsvc.User {
+	return slurmsvc.User{Login: u.Login, UID: u.UID, Name: u.Name, Email: u.Email, SSHKeys: u.SSHKeys}
+}
+
+// SetSlurm gives the API the Slurm service.
+func (a *API) SetSlurm(s SlurmService) { a.slurm = s }
+
+func (a *API) registerSlurm(mux Mux) {
+	mux.HandleFunc("GET /api/v1/slurm", a.slurmStatus)
+	mux.HandleFunc("GET /api/v1/slurm/users", a.slurmUserList)
+	mux.HandleFunc("PUT /api/v1/slurm/users/{user}/ssh-keys", a.slurmUserKeys)
+}
+
+func (a *API) slurmUserList(w http.ResponseWriter, r *http.Request) {
+	if a.slurmUsers == nil {
+		writeError(w, http.StatusNotImplemented, "the Slurm users need the store, which this control plane runs without")
+		return
+	}
+	us, err := a.slurmUsers.SlurmUsers(r.Context())
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	out := make([]slurmsvc.User, 0, len(us))
+	for _, u := range us {
+		out = append(out, SlurmUserView(u))
+	}
+	writeJSON(w, http.StatusOK, out)
+}
+
+// slurmUserKeys replaces a person's SSH keys; {user} is an ID or e-mail.
+func (a *API) slurmUserKeys(w http.ResponseWriter, r *http.Request) {
+	if a.slurmUsers == nil {
+		writeError(w, http.StatusNotImplemented, "the Slurm users need the store, which this control plane runs without")
+		return
+	}
+	if !a.tenancyReady(w) {
+		return
+	}
+	var in SlurmKeysRequest
+	if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 64<<10)).Decode(&in); err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	keys, err := slurm.ParseSSHKeys(in.Keys)
+	if err != nil {
+		writeError(w, http.StatusBadRequest, err.Error())
+		return
+	}
+	ctx := r.Context()
+	u, err := a.findUser(ctx, r.PathValue("user"))
+	if err != nil {
+		writeError(w, http.StatusNotFound, strings.TrimPrefix(err.Error(), store.ErrNotFound.Error()+": "))
+		return
+	}
+	if err := a.slurmUsers.SetSlurmSSHKeys(ctx, u.ID, keys); err != nil {
+		code := http.StatusInternalServerError
+		if errors.Is(err, store.ErrNotFound) {
+			code = http.StatusNotFound
+		}
+		writeError(w, code, err.Error())
+		return
+	}
+	if a.slurm != nil {
+		a.slurm.UsersChanged()
+	}
+	su, err := a.slurmUsers.SlurmUserByID(ctx, u.ID)
+	if err != nil {
+		writeError(w, http.StatusInternalServerError, err.Error())
+		return
+	}
+	writeJSON(w, http.StatusOK, SlurmUserView(su))
+}
+
+func (a *API) slurmStatus(w http.ResponseWriter, _ *http.Request) {
+	if a.slurm == nil {
+		writeError(w, http.StatusNotImplemented, "Slurm needs the store and the nodes' agents, which this control plane runs without")
+		return
+	}
+	writeJSON(w, http.StatusOK, a.slurm.Status())
+}

internal/control/adminapi/summary.go

@@ -50,6 +50,9 @@ type Summary struct {
 	// GuardOff says the guardrails are turned off (guard.enabled: false):
 	// no request is checked, whatever the policies say.
 	GuardOff bool `json:"guard_off,omitempty"`
+	// NoAlerts says the cluster's settings name no webhook: a model parked,
+	// a node lost or a judge down is told nobody.
+	NoAlerts bool `json:"no_alerts,omitempty"`
 }
 
 // StoreMember is one member of a replicated store, as its Patroni reports it.
@@ -160,6 +163,9 @@ func (a *API) Summary() Summary {
 	if a.guard != nil && a.guard.Library != nil && !a.guard.Library().On() {
 		s.GuardOff = true
 	}
+	if a.webhooks != nil && a.webhooks() == 0 {
+		s.NoAlerts = true
+	}
 	for _, j := range a.Judges() {
 		if j.Problem != "" {
 			s.Judges = append(s.Judges, j)

internal/control/adminapi/summary_test.go

@@ -153,3 +153,16 @@ func TestOneSharedCopyCountsAsMuchAsTwenty(t *testing.T) {
 		t.Errorf("counted %d models; three nodes seeing one file is one model", f.Models)
 	}
 }
+
+func TestTheSummarySaysWhenNoWebhookIsSet(t *testing.T) {
+	a, _ := setup(t, testModels(), gpuNode("n1", 8))
+	n := 0
+	a.SetWebhooks(func() int { return n })
+	if !a.Summary().NoAlerts {
+		t.Error("no webhook, not said")
+	}
+	n = 1
+	if a.Summary().NoAlerts {
+		t.Error("a webhook set, yet said there is none")
+	}
+}

internal/control/adminapi/tenancy.go

@@ -634,8 +634,12 @@ func (a *API) membership(add bool) http.HandlerFunc {
 			if role == "" {
 				role = store.RoleMember
 			}
-			if role != store.RoleMember && role != store.RoleManager {
-				writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member or manager", role))
+			if role != store.RoleMember && role != store.RoleManager && role != store.RoleOrgAdmin {
+				writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member, manager or admin", role))
+				return
+			}
+			if role != store.RoleMember && scope != store.ScopeOrg {
+				writeError(w, http.StatusBadRequest, fmt.Sprintf("%s is a role on an organization", role))
 				return
 			}
 			if err := a.tenancy.AddMembership(ctx, u.ID, scope, id, role, actorOf(r), time.Now()); err != nil {

internal/control/adminapi/weights.go

@@ -1,11 +1,14 @@
 package adminapi
 
 import (
+	"context"
 	"encoding/json"
 	"net/http"
 	"sort"
+	"time"
 
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
 )
 
 // OrphanView is cached weights on one node that no catalog model accounts for.
@@ -71,12 +74,100 @@ type CollectRequest struct {
 	Node string `json:"node,omitempty"`
 }
 
-// CollectResult is what was deleted on one node, or why nothing was.
+// CollectResult is what was deleted on one node, or why nothing was: what
+// the node no longer lists once asked, not what it was asked to delete.
 type CollectResult struct {
 	Node  string `json:"node"`
 	Files int    `json:"files"`
 	Bytes int64  `json:"bytes"`
-	Error string `json:"error,omitempty"`
+	// Left and LeftBytes are what the node still lists: refused, or not
+	// gone by the time the sweep stopped waiting. Its agent's log says why.
+	Left      int    `json:"left,omitempty"`
+	LeftBytes int64  `json:"left_bytes,omitempty"`
+	Error     string `json:"error,omitempty"`
+}
+
+// confirmWait is how long a sweep waits for the nodes to list their files
+// without the ones deleted: they scan theirs every heartbeat, five seconds
+// by default, and a large set takes a moment to delete.
+var confirmWait = 30 * time.Second
+
+// removal is what one node is asked to delete, with each file's size.
+type removal struct {
+	node  string
+	paths []string
+	size  map[string]int64
+}
+
+// removeConfirmed asks each node to delete its paths and says what each then
+// no longer lists. Counted as asked, a sweep said "25 GB freed" of files the
+// node had refused and kept.
+func (a *API) removeConfirmed(ctx context.Context, rm control.WeightRemover, rs []removal) []CollectResult {
+	out := make([]CollectResult, len(rs))
+	pending := map[int]bool{}
+	for i, r := range rs {
+		out[i].Node = r.node
+		if err := rm.RemoveWeights(ctx, r.node, r.paths); err != nil {
+			out[i].Error = err.Error()
+			for _, p := range r.paths {
+				out[i].Left++
+				out[i].LeftBytes += r.size[p]
+			}
+			continue
+		}
+		pending[i] = true
+	}
+	deadline := time.Now().Add(confirmWait)
+	for {
+		for i := range pending {
+			if left := a.stillListed(rs[i]); len(left) == 0 {
+				delete(pending, i)
+			}
+		}
+		if len(pending) == 0 || !time.Now().Before(deadline) {
+			break
+		}
+		select {
+		case <-ctx.Done():
+			deadline = time.Now()
+		case <-time.After(time.Second):
+		}
+	}
+	for i, r := range rs {
+		if out[i].Error != "" {
+			continue
+		}
+		left := a.stillListed(r)
+		for _, p := range r.paths {
+			if left[p] {
+				out[i].Left++
+				out[i].LeftBytes += r.size[p]
+			} else {
+				out[i].Files++
+				out[i].Bytes += r.size[p]
+			}
+		}
+	}
+	return out
+}
+
+// stillListed is which of a removal's paths its node still lists.
+func (a *API) stillListed(r removal) map[string]bool {
+	left := map[string]bool{}
+	n, ok := a.ctl.Inventory().Get(r.node)
+	if !ok {
+		return left
+	}
+	want := map[string]bool{}
+	for _, p := range r.paths {
+		want[p] = true
+	}
+	for _, w := range n.Weights {
+		if want[w.Path] {
+			left[w.Path] = true
+		}
+	}
+	return left
 }
 
 // collectWeights deletes them.
@@ -97,20 +188,26 @@ func (a *API) collectWeights(w http.ResponseWriter, r *http.Request) {
 	// An empty body is the "all nodes" case and must not be an error.
 	_ = json.NewDecoder(r.Body).Decode(&body)
 
-	out := make([]CollectResult, 0)
+	// A node that has gone since the list was built is reported rather than
+	// failing the whole sweep: the other nodes' space is still worth
+	// freeing, and this one will be offered again next time.
+	var rs []removal
 	for _, o := range a.Orphans() {
 		if body.Node != "" && o.Node != body.Node {
 			continue
 		}
-		res := CollectResult{Node: o.Node, Files: len(o.Paths), Bytes: o.Bytes}
-		// A node that has gone since the list was built is reported rather than
-		// failing the whole sweep: the other nodes' space is still worth
-		// freeing, and this one will be offered again next time.
-		if err := rm.RemoveWeights(r.Context(), o.Node, o.Paths); err != nil {
-			res.Error = err.Error()
-			res.Files, res.Bytes = 0, 0
-		}
-		out = append(out, res)
+		rs = append(rs, removal{node: o.Node, paths: o.Paths, size: a.sizesOn(o.Node)})
 	}
-	writeJSON(w, http.StatusOK, out)
+	writeJSON(w, http.StatusOK, a.removeConfirmed(r.Context(), rm, rs))
+}
+
+// sizesOn is each file a node lists, by path, with its size.
+func (a *API) sizesOn(node string) map[string]int64 {
+	out := map[string]int64{}
+	if n, ok := a.ctl.Inventory().Get(node); ok {
+		for _, w := range n.Weights {
+			out[w.Path] = w.SizeBytes
+		}
+	}
+	return out
 }

internal/control/adminapi/weights_test.go

@@ -0,0 +1,71 @@
+package adminapi
+
+import (
+	"context"
+	"encoding/json"
+	"log/slog"
+	"net/http"
+	"net/http/httptest"
+	"slices"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+// deletingDispatcher deletes what it is asked to as a node would, by its
+// files dropping out of what the node lists — but for refused, which it
+// keeps, as an agent keeps a file it cannot prove it fetched.
+type deletingDispatcher struct {
+	nullDispatcher
+	inv     *inventory.Store
+	refused string
+}
+
+func (d *deletingDispatcher) RemoveWeights(_ context.Context, node string, paths []string) error {
+	n, _ := d.inv.Get(node)
+	n.Weights = slices.DeleteFunc(n.Weights, func(w discovery.LocalWeights) bool {
+		return w.Path != d.refused && slices.Contains(paths, w.Path)
+	})
+	d.inv.Upsert(n)
+	return nil
+}
+
+// What a sweep says it freed is what the node no longer lists: a file it
+// kept is said to be left, not counted as freed.
+func TestASweepSaysWhatWasDeletedNotWhatWasAsked(t *testing.T) {
+	confirmWait = 1500 * time.Millisecond
+	t.Cleanup(func() { confirmWait = 30 * time.Second })
+
+	gone := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/old/v1/old.gguf")
+	set := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/big/v1")
+	kept := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/stuck/v1/stuck.gguf")
+	cat, _ := catalog.New(nil)
+	d := &deletingDispatcher{refused: kept}
+	ctl := control.New(control.DefaultConfig(), catalog.NewLive(cat), d, nullSink{}, slog.New(slog.DiscardHandler))
+	d.inv = ctl.Inventory()
+	node := gpuNode("n1", 1)
+	node.Weights = []discovery.LocalWeights{{Path: gone, SizeBytes: 1 << 30}, {Path: set, SizeBytes: 20 << 30},
+		{Path: kept, SizeBytes: 4 << 30}}
+	ctl.Inventory().Upsert(node)
+	a := New(ctl)
+
+	rec := httptest.NewRecorder()
+	a.collectWeights(rec, httptest.NewRequest(http.MethodPost, "/api/v1/weights/collect", strings.NewReader(`{"node":"n1"}`)))
+	var got []CollectResult
+	if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
+		t.Fatalf("%v: %s", err, rec.Body.String())
+	}
+	if len(got) != 1 {
+		t.Fatalf("results %+v", got)
+	}
+	r := got[0]
+	if r.Files != 2 || r.Bytes != 21<<30 || r.Left != 1 || r.LeftBytes != 4<<30 {
+		t.Errorf("result %+v: want 2 files and 21 GB freed, 1 file of 4 GB left", r)
+	}
+}

internal/control/agentsvc/service.go

@@ -84,6 +84,8 @@ type Service struct {
 	agentSettings func(node string) string
 	// train is told how training jobs go; see SetTrain.
 	train TrainHandler
+	// slurm is told of the nodes' part in Slurm; see SetSlurm.
+	slurm SlurmHandler
 
 	// logWaits are the log requests awaiting a reply, by request id.
 	logMu    sync.Mutex
@@ -308,6 +310,10 @@ func (s *Service) Connect(
 			return err
 		}
 	}
+	// Its part in Slurm, sent by the Slurm service's next pass.
+	if h := s.slurmHandler(); h != nil {
+		h.OnRegister(nodeID)
+	}
 
 	// Recover replica state from what this node is already running, before any
 	// reconciliation pass can conclude the cluster is empty and place
@@ -515,6 +521,11 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
 			h.OnTrainStatus(nodeID, m.TrainStatus)
 		}
 
+	case *llmgwv1.AgentMessage_SlurmStatus:
+		if h := s.slurmHandler(); h != nil {
+			h.OnSlurmStatus(nodeID, m.SlurmStatus)
+		}
+
 	case *llmgwv1.AgentMessage_TrainArtifact:
 		if h := s.trainHandler(); h != nil {
 			h.OnTrainArtifact(nodeID, m.TrainArtifact)

internal/control/agentsvc/slurm.go

@@ -0,0 +1,25 @@
+package agentsvc
+
+import (
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// SlurmHandler is told of the nodes' part in Slurm (BACKLOG P).
+type SlurmHandler interface {
+	OnSlurmStatus(node string, st *llmgwv1.SlurmStatus)
+	// OnRegister is a node's new stream, which is told again what it is.
+	OnRegister(node string)
+}
+
+// SetSlurm gives the service what Slurm's messages go to.
+func (s *Service) SetSlurm(h SlurmHandler) {
+	s.mu.Lock()
+	s.slurm = h
+	s.mu.Unlock()
+}
+
+func (s *Service) slurmHandler() SlurmHandler {
+	s.mu.RLock()
+	defer s.mu.RUnlock()
+	return s.slurm
+}

internal/control/held/held.go

@@ -0,0 +1,93 @@
+// Package held records, on the leader, what each model holds of the
+// cluster: its replicas' GPUs and, for one on CPUs, its cores. Kept by the
+// hour, it is the cost side of a token's price (llmgw ctl prices suggest).
+package held
+
+import (
+	"context"
+	"log/slog"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// Inventory is where the allocations are.
+type Inventory interface {
+	Nodes() []inventory.Node
+	Allocations(nodeID string) []inventory.Allocation
+}
+
+// Store keeps what was held.
+type Store interface {
+	AddHeld(ctx context.Context, at time.Time, held []store.Held) error
+}
+
+// Recorder adds, every Every, what each model holds for that long.
+type Recorder struct {
+	Inv Inventory
+	// Models is each replica's model, by replica id.
+	Models func() map[string]string
+	Store  Store
+	Leader func() bool
+	Every  time.Duration
+	Log    *slog.Logger
+}
+
+// Run records until ctx ends.
+func (r *Recorder) Run(ctx context.Context) {
+	every := r.Every
+	if every <= 0 {
+		every = 5 * time.Minute
+	}
+	t := time.NewTicker(every)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case now := <-t.C:
+			if r.Leader != nil && !r.Leader() {
+				continue
+			}
+			held := Sample(r.Inv, r.Models(), every)
+			if err := r.Store.AddHeld(ctx, now, held); err != nil && r.Log != nil {
+				r.Log.Warn("could not record what the models held, for their prices", "err", err)
+			}
+		}
+	}
+}
+
+// Sample is what each model holds now, for d: a GPU for each card a shard
+// was given, and the cores of a shard given cores and no GPU. A replica
+// whose model is not known (headroom kept for another, one just reaped) is
+// left out.
+func Sample(inv Inventory, models map[string]string, d time.Duration) []store.Held {
+	by := map[string]*store.Held{}
+	var order []string
+	secs := int64(d.Seconds())
+	for _, n := range inv.Nodes() {
+		for _, a := range inv.Allocations(n.ID) {
+			m, ok := models[a.ReplicaID]
+			if !ok {
+				continue
+			}
+			h := by[m]
+			if h == nil {
+				h = &store.Held{Model: m}
+				by[m] = h
+				order = append(order, m)
+			}
+			if g := len(a.GPUIndices); g > 0 {
+				h.GPUSeconds += int64(g) * secs
+			} else {
+				h.CoreSeconds += int64(max(a.CPUCores, 1)) * secs
+			}
+		}
+	}
+	out := make([]store.Held, 0, len(order))
+	for _, m := range order {
+		out = append(out, *by[m])
+	}
+	return out
+}

internal/control/held/held_test.go

@@ -0,0 +1,40 @@
+package held
+
+import (
+	"testing"
+	"time"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+type inv map[string][]inventory.Allocation
+
+func (i inv) Nodes() []inventory.Node {
+	var out []inventory.Node
+	for id := range i {
+		out = append(out, inventory.Node{ID: id})
+	}
+	return out
+}
+
+func (i inv) Allocations(id string) []inventory.Allocation { return i[id] }
+
+func TestSampleCountsCardsAndCores(t *testing.T) {
+	in := inv{
+		"gpu1": {{ReplicaID: "big-1", GPUIndices: []int{0, 1}}, {ReplicaID: "headroom", GPUIndices: []int{2}}},
+		"gpu2": {{ReplicaID: "big-2", GPUIndices: []int{0, 1}}},
+		"cpu1": {{ReplicaID: "small-1", CPUCores: 4}, {ReplicaID: "tiny-1"}},
+	}
+	models := map[string]string{"big-1": "big", "big-2": "big", "small-1": "small", "tiny-1": "tiny"}
+	got := map[string]store.Held{}
+	for _, h := range Sample(in, models, 5*time.Minute) {
+		got[h.Model] = h
+	}
+	if got["big"].GPUSeconds != 4*300 || got["small"].CoreSeconds != 4*300 || got["tiny"].CoreSeconds != 300 {
+		t.Errorf("held %+v", got)
+	}
+	if len(got) != 3 {
+		t.Errorf("headroom counted: %+v", got)
+	}
+}

internal/control/modelserve/modelserve.go

@@ -296,7 +296,7 @@ func (s *Server) Start(req Request) (Job, error) {
 	}
 	req.sized = req.CtxPool > 0 || req.ContextLength > 0
 	if req.CtxPool <= 0 {
-		req.CtxPool = 16384
+		req.CtxPool = DefaultCtxPool
 	}
 	if req.ContextLength <= 0 {
 		req.ContextLength = 4096
@@ -731,6 +731,10 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
 	return out
 }
 
+// DefaultCtxPool is llama.cpp's pool when a request names none: a context
+// asked for longer than it needs a pool of its own.
+const DefaultCtxPool = 16384
+
 // llamaArgs size llama.cpp's pool and its slots together. The pool is shared
 // by the slots, so slots of the context the catalog promises are as many as
 // the pool holds: the engine's default of four turned a 32768 pool into

internal/control/modelserve/plan.go

@@ -43,6 +43,9 @@ type Plan struct {
 	License  string `json:"license,omitempty"`
 	Params   int64  `json:"params,omitempty"`
 	Context  int    `json:"context"`
+	// Trained is the context the model was trained for, the most a request
+	// may use; 0 when neither its header nor its config says.
+	Trained int `json:"trained,omitempty"`
 	// KVBytesPerToken is one token of context, the whole model: what an
 	// option needs for another context is worked out from it. 0 is unknown.
 	KVBytesPerToken int64 `json:"kv_bytes_per_token,omitempty"`
@@ -149,6 +152,7 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
 			p.Servable, p.Reason = false, v.Reason
 		}
 		shape.KVBytesPerToken = v.Header.KVBytesPerToken()
+		p.Trained = int(v.Header.ContextLength)   //nolint:gosec // a header's context length
 		shape.KVHeads = int(v.Header.HeadCountKV) //nolint:gosec // a header's head count
 		shape.ActiveShare = v.Header.ActiveShare()
 		if p.Arch == "" {
@@ -167,6 +171,7 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
 		m, err := modelplan.FetchModel(ctx, repo.ID)
 		if err == nil {
 			shape.KVBytesPerToken = m.KVBytesPerToken(modelplan.BF16)
+			p.Trained = m.MaxContext
 			shape.KVHeads = m.KVHeads
 			shape.Experts = m.Experts
 			if p.Params == 0 {

internal/control/placement/placement.go

@@ -154,6 +154,9 @@ func (r Result) Explain() string {
 // Scheduler places replicas onto an inventory.
 type Scheduler struct {
 	inv *inventory.Store
+	// shun is where a replica of the model being placed failed to start;
+	// set on a copy by Away, never on the scheduler the reconciler shares.
+	shun []Spot
 	// now is the clock power readings are judged fresh by; nil is the wall
 	// clock.
 	now func() time.Time
@@ -162,6 +165,41 @@ type Scheduler struct {
 // New builds a scheduler over an inventory.
 func New(inv *inventory.Store) *Scheduler { return &Scheduler{inv: inv} }
 
+// Spot is where a replica ran: a node, and the cards it held there (none
+// for one on the node's CPU).
+type Spot struct {
+	Node string
+	GPUs []int
+}
+
+// Away is this scheduler placing nowhere in spots: not those cards, nor a
+// CPU replica on those nodes. A copy, so the one shared is not changed.
+//
+// For a model that failed to start somewhere: the same inputs pick the same
+// node and the same card every time, so without it every retry went back to
+// where the last one had failed until the model was parked, with another
+// card or node free all along.
+func (s *Scheduler) Away(spots []Spot) *Scheduler {
+	c := *s
+	c.shun = spots
+	return &c
+}
+
+// shunned is the cards of node n a model failed on, and whether it failed
+// on the node's CPU.
+func (s *Scheduler) shunned(node string) (gpus []int, cpu bool) {
+	for _, sp := range s.shun {
+		if sp.Node != node {
+			continue
+		}
+		if len(sp.GPUs) == 0 {
+			cpu = true
+		}
+		gpus = append(gpus, sp.GPUs...)
+	}
+	return gpus, cpu
+}
+
 // Place picks the first profile, in priority order, that fits.
 //
 // avoid lists nodes already hosting this model, so replicas of one model spread
@@ -686,7 +724,12 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
 			}
 		}
 
+		badGPUs, badCPU := s.shunned(n.ID)
 		if p.CPUOnly() {
+			if badCPU {
+				reject("a replica of it failed to start here; trying elsewhere first")
+				continue
+			}
 			// Every ready node can run it; there is no card to be free.
 			c := candidate{node: n}
 			if why := budget.check(n, s.shardWatts(p, c)); why != "" {
@@ -703,8 +746,16 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
 			if g.Class() != p.GPUClass || !slices.Contains(freeIdx, g.Index) {
 				continue
 			}
+			if slices.Contains(badGPUs, g.Index) {
+				continue
+			}
 			free = append(free, g)
 		}
+		if len(free) < p.GPUsPerNode && len(badGPUs) > 0 {
+			reject(fmt.Sprintf("has %d free %s GPUs besides the ones a replica of it failed to start on, needs %d",
+				len(free), p.GPUClass, p.GPUsPerNode))
+			continue
+		}
 		if len(free) < p.GPUsPerNode {
 			reject(fmt.Sprintf("has %d free %s GPUs, needs %d", len(free), p.GPUClass, p.GPUsPerNode))
 			continue

internal/control/reconciler/away_test.go

@@ -0,0 +1,80 @@
+package reconciler
+
+import (
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+)
+
+// failAndRetry fails the last replica started, and returns where the next
+// one is started once the backoff is over.
+func failAndRetry(t *testing.T, h *harness) ShardCommand {
+	t.Helper()
+	started := h.disp.startedShards()
+	if len(started) == 0 {
+		t.Fatal("nothing was started")
+	}
+	s := started[len(started)-1]
+	h.r.OnShardStatus(s.ReplicaID, s.ShardID, supervisor.Failed, "the engine exited")
+	h.r.Reconcile(t.Context()) // reaps it
+	h.disp.reset()
+	h.advance(DefaultConfig().MaxBackoff)
+	h.r.Reconcile(t.Context())
+	again := h.disp.startedShards()
+	if len(again) == 0 {
+		t.Fatal("it was not tried again")
+	}
+	return again[0]
+}
+
+func oneCard() catalog.Model {
+	return testModel(1, catalog.Profile{Name: "one", GPUClass: "h100-80g", GPUsPerNode: 1})
+}
+
+func TestAReplicaThatFailedIsTriedOnAnotherNode(t *testing.T) {
+	// The same inputs give the same node every time: an on-demand 3D model
+	// that failed to load was started on that node again, and again, until
+	// it was parked, with another node free all along.
+	h := newHarness(t, oneCard(), gpuNode("1", 1), gpuNode("2", 1))
+	h.r.Reconcile(t.Context())
+	first := h.disp.startedShards()[0].NodeID
+
+	if next := failAndRetry(t, h); next.NodeID == first {
+		t.Errorf("tried on node %s again, where it had just failed", first)
+	}
+}
+
+func TestAReplicaThatFailedIsTriedOnAnotherCardOfTheNode(t *testing.T) {
+	h := newHarness(t, oneCard(), gpuNode("1", 2))
+	h.r.Reconcile(t.Context())
+	first := h.disp.startedShards()[0].Spec.GPUIndices
+
+	next := failAndRetry(t, h)
+	if len(next.Spec.GPUIndices) != 1 || len(first) != 1 || next.Spec.GPUIndices[0] == first[0] {
+		t.Errorf("tried on card %v again, where it had just failed", next.Spec.GPUIndices)
+	}
+}
+
+func TestWithNowhereElseItIsTriedWhereItFailed(t *testing.T) {
+	// A preference, not a rule: a model that has one place is not left
+	// unplaced because it failed there once.
+	h := newHarness(t, oneCard(), gpuNode("1", 1))
+	h.r.Reconcile(t.Context())
+	if next := failAndRetry(t, h); next.NodeID != "1" {
+		t.Errorf("started on %q", next.NodeID)
+	}
+}
+
+func TestComingUpForgetsWhereItFailed(t *testing.T) {
+	h := newHarness(t, oneCard(), gpuNode("1", 1), gpuNode("2", 1))
+	h.r.Reconcile(t.Context())
+	next := failAndRetry(t, h)
+	h.r.OnShardStatus(next.ReplicaID, next.ShardID, supervisor.Ready, "")
+	h.r.mu.Lock()
+	left := len(h.r.failedAt["llama"])
+	h.r.mu.Unlock()
+	if left != 0 {
+		t.Errorf("a model that came up still remembers %d failures' places", left)
+	}
+}

internal/control/reconciler/health.go

@@ -61,6 +61,7 @@ func (r *Reconciler) Retry(model string) bool {
 	delete(r.parked, model)
 	r.attempts[model] = 0
 	delete(r.nextAttempt, model)
+	delete(r.failedAt, model)
 	r.mu.Unlock()
 	r.Notify()
 	return was

internal/control/reconciler/reconciler.go

@@ -7,6 +7,8 @@ import (
 	"log/slog"
 	"math/rand/v2"
 	"slices"
+	"strconv"
+	"strings"
 	"sync"
 	"sync/atomic"
 	"time"
@@ -251,6 +253,9 @@ type Reconciler struct {
 	nextAttempt map[string]time.Time
 	// attempts counts consecutive failures per model, for the backoff.
 	attempts map[string]int
+	// failedAt is where a model's replicas failed to start since it last
+	// came up, most recent last: the next one is placed elsewhere if it can.
+	failedAt map[string][]placement.Spot
 	// lastExplain keeps the most recent placement failure per model, so
 	// `ctl explain-placement` can answer without re-running placement.
 	lastExplain map[string]placement.Result
@@ -334,6 +339,7 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
 		replicas:        map[string]*Replica{},
 		nextAttempt:     map[string]time.Time{},
 		attempts:        map[string]int{},
+		failedAt:        map[string][]placement.Spot{},
 		reservedPorts:   map[int]struct{}{},
 		wanted:          map[string]time.Time{},
 		coldStart:       map[string]time.Duration{},
@@ -726,6 +732,7 @@ func (r *Reconciler) reapFailed(ctx context.Context) {
 			// replica is about to disappear, and the backoff has to survive it.
 			r.attempts[rep.Model]++
 			r.nextAttempt[rep.Model] = r.now().Add(r.backoff(r.attempts[rep.Model]))
+			r.noteFailedAt(rep)
 			if n := r.attempts[rep.Model]; n >= r.cfg.ParkAfter {
 				if _, already := r.parked[rep.Model]; !already {
 					m, _ := r.cat.Load().Resolve(rep.Model)
@@ -875,6 +882,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
 		delete(r.parked, m.Name)
 		r.attempts[m.Name] = 0
 		delete(r.nextAttempt, m.Name)
+		delete(r.failedAt, m.Name)
 		isParked, blockedUntil = false, time.Time{}
 		r.log.Info("model unparked: its definition changed", "model", m.Name)
 	}
@@ -1002,7 +1010,29 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
 			return notTried
 		}
 	}
-	res := r.sched.Place(profiles, avoid)
+	// Away from where it last failed to start, when there is anywhere else:
+	// the same inputs give the same node and card, and an on-demand model
+	// that failed there was otherwise started there again until it was
+	// parked. Only a preference too: nowhere else, and it is tried there.
+	r.mu.Lock()
+	failed := slices.Clone(r.failedAt[m.Name])
+	r.mu.Unlock()
+	var res placement.Result
+	if len(failed) > 0 {
+		away := r.sched.Away(failed)
+		res = away.Place(profiles, avoid)
+		if res.Placement == nil && len(avoid) > 0 {
+			res = away.Place(profiles, nil)
+		}
+		pspan.SetAttributes(attribute.Bool("llmgw.away_from_failure", res.Placement != nil))
+		if res.Placement != nil {
+			r.log.Info("placing a model away from where it failed to start", "model", m.Name,
+				"failed_on", spotsText(failed), "now_on", spotsText(spotsOf(res.Placement)))
+		}
+	}
+	if res.Placement == nil {
+		res = r.sched.Place(profiles, avoid)
+	}
 	if res.Placement == nil && len(avoid) > 0 {
 		// Spreading across failure domains is a preference, not a rule. If it
 		// is the only thing standing in the way, take the placement anyway:
@@ -1054,6 +1084,51 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
 	return placed
 }
 
+// maxFailedSpots is how many places a model's failures are remembered at.
+const maxFailedSpots = 8
+
+// noteFailedAt remembers where a failed replica ran. Called under r.mu.
+func (r *Reconciler) noteFailedAt(rep *Replica) {
+	spots := r.failedAt[rep.Model]
+	for _, sh := range rep.Shards {
+		sp := placement.Spot{Node: sh.NodeID, GPUs: slices.Clone(sh.GPUIndices)}
+		spots = slices.DeleteFunc(spots, func(o placement.Spot) bool {
+			return o.Node == sp.Node && slices.Equal(o.GPUs, sp.GPUs)
+		})
+		spots = append(spots, sp)
+	}
+	if len(spots) > maxFailedSpots {
+		spots = spots[len(spots)-maxFailedSpots:]
+	}
+	r.failedAt[rep.Model] = spots
+}
+
+// spotsOf is where a placement puts its shards.
+func spotsOf(pl *placement.Placement) []placement.Spot {
+	var out []placement.Spot
+	for _, n := range pl.Nodes {
+		out = append(out, placement.Spot{Node: n.NodeID, GPUs: n.GPUIndices})
+	}
+	return out
+}
+
+// spotsText is spots as a log reads them: "node01 gpu 0,1; node02 cpu".
+func spotsText(spots []placement.Spot) string {
+	parts := make([]string, 0, len(spots))
+	for _, sp := range spots {
+		if len(sp.GPUs) == 0 {
+			parts = append(parts, sp.Node+" cpu")
+			continue
+		}
+		gs := make([]string, len(sp.GPUs))
+		for i, g := range sp.GPUs {
+			gs[i] = strconv.Itoa(g)
+		}
+		parts = append(parts, sp.Node+" gpu "+strings.Join(gs, ","))
+	}
+	return strings.Join(parts, "; ")
+}
+
 // scaleOutcome is what one scaleUp did.
 type scaleOutcome int
 
@@ -1433,6 +1508,7 @@ func (r *Reconciler) markShard(replicaID, shardID string, st supervisor.State, e
 		// counting from scratch rather than from an old streak.
 		r.attempts[rep.Model] = 0
 		delete(r.nextAttempt, rep.Model)
+		delete(r.failedAt, rep.Model)
 		if !rep.CreatedAt.IsZero() {
 			r.coldStart[rep.Model] = r.now().Sub(rep.CreatedAt)
 		}

internal/control/slurmsvc/failover_test.go

@@ -0,0 +1,133 @@
+package slurmsvc
+
+import (
+	"context"
+	"errors"
+	"strings"
+	"testing"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// fakeKV is the store's cluster state.
+type fakeKV struct {
+	m   map[string]string
+	err error
+}
+
+func (f *fakeKV) PutClusterState(_ context.Context, k, v string) (bool, error) {
+	if f.err != nil {
+		return false, f.err
+	}
+	f.m[k] = v
+	return true, nil
+}
+
+func (f *fakeKV) ClusterState(_ context.Context, k string) (string, error) {
+	if f.err != nil {
+		return "", f.err
+	}
+	v, ok := f.m[k]
+	if !ok {
+		return "", store.ErrNotFound
+	}
+	return v, nil
+}
+
+// takeOver is the leader that comes next: the same settings and store, an
+// inventory with only the nodes connected to it so far.
+func takeOver(r *rig, kv *fakeKV, connected ...string) *rig {
+	n := &rig{set: r.set, inv: &fakeInv{nodes: map[string]*inventory.Node{}, allocs: map[string]int{}},
+		agents: &fakeAgents{conn: connected, got: map[string][]*llmgwv1.Slurm{}}}
+	for _, c := range connected {
+		node := *r.inv.nodes[c]
+		node.State = inventory.Ready
+		n.inv.nodes[c] = &node
+	}
+	n.svc = New(func() config.SlurmSettings { return n.set }, n.inv, n.agents,
+		func(context.Context) ([]byte, error) { return []byte(strings.Repeat("k", 1024)), nil }, nil)
+	n.svc.SetMemory(StoreMemory(kv))
+	return n
+}
+
+func TestANewLeaderKeepsTheNodesItHasNotHeardFromInTheControllersFiles(t *testing.T) {
+	kv := &fakeKV{m: map[string]string{}}
+	r := newRig()
+	r.svc.SetMemory(StoreMemory(kv))
+	r.set.Lent = []string{"cuda01"}
+	r.inv.allocs = map[string]int{}
+	r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "MIXED", Jobs: 1})
+	r.tick()
+	want := r.agents.last("node06").GetFiles()["slurm.conf"]
+	if !strings.Contains(want, "NodeName=cuda01") || !strings.Contains(want, "Gres=gpu:rtx3090:1") {
+		t.Fatalf("slurm.conf:\n%s", want)
+	}
+
+	// The leader goes; the next hears from the controller first.
+	n := takeOver(r, kv, "node06")
+	n.tick()
+	if got := n.agents.last("node06").GetFiles()["slurm.conf"]; got != want {
+		t.Fatalf("the new leader's slurm.conf differs; the controller would drop the node and its job:\n%s\nwas:\n%s",
+			got, want)
+	}
+}
+
+func TestANodeIsToldNothingUntilItsControllerIsKnown(t *testing.T) {
+	r := newRig()
+	r.svc.SetMemory(StoreMemory(&fakeKV{m: map[string]string{}}))
+	delete(r.inv.nodes, "node06")
+	r.agents.conn = []string{"cuda01"}
+	r.tick()
+	if m := r.agents.last("cuda01"); m != nil {
+		t.Fatalf("a node was told %v before its controller was known; an empty part stops its slurmd and its jobs", m)
+	}
+}
+
+func TestANewLeaderTellsANodeItsControllerFromTheStore(t *testing.T) {
+	kv := &fakeKV{m: map[string]string{}}
+	r := newRig()
+	r.svc.SetMemory(StoreMemory(kv))
+	r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
+	r.tick()
+
+	n := takeOver(r, kv, "cuda01")
+	n.tick()
+	if m := n.agents.last("cuda01"); !m.GetNode() || m.GetConfServer() != "10.0.0.146:6817" {
+		t.Fatalf("the node was told %v", m)
+	}
+}
+
+func TestWithTheStoreUnreadTheControllerIsSentNothingThatLeavesANodeOut(t *testing.T) {
+	kv := &fakeKV{m: map[string]string{}, err: errors.New("store down")}
+	r := newRig()
+	r.svc.SetMemory(StoreMemory(kv))
+	r.tick()
+	if m := r.agents.last("node06"); m != nil {
+		t.Fatalf("the controller was sent %v without cuda01, with the store unread", m)
+	}
+	// Once every node has reported, nothing is left out: sent.
+	r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
+	r.tick()
+	if !strings.Contains(r.agents.last("node06").GetFiles()["slurm.conf"], "NodeName=cuda01") {
+		t.Fatal("with every node reported, the controller still got no files")
+	}
+}
+
+func TestWithNothingKeptANewLeaderWaitsForTheNodesAWhile(t *testing.T) {
+	r := newRig()
+	r.svc.SetMemory(StoreMemory(&fakeKV{m: map[string]string{}}))
+	r.tick()
+	if m := r.agents.last("node06"); m != nil {
+		t.Fatalf("with nothing kept, the controller was sent %v before cuda01 reported", m)
+	}
+	defer func(d time.Duration) { HoldFor = d }(HoldFor)
+	HoldFor = 0
+	r.tick()
+	if m := r.agents.last("node06"); !m.GetController() || strings.Contains(m.GetFiles()["slurm.conf"], "NodeName=cuda01") {
+		t.Fatalf("after the wait, the controller was told %v", m)
+	}
+}

internal/control/slurmsvc/memory.go

@@ -0,0 +1,77 @@
+package slurmsvc
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"slices"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/store"
+)
+
+// Memory keeps the nodes slurm.conf last held, and the controller's
+// address, where a leader that takes over finds them. Without it a new
+// leader's first slurm.conf has none of the nodes it has not heard from
+// yet, and the controller, restarted on it, drops them and the jobs
+// running there.
+type Memory interface {
+	Recall(ctx context.Context) (Kept, error)
+	Keep(ctx context.Context, k Kept) error
+}
+
+// Kept is what Memory holds.
+type Kept struct {
+	Controller     string                `json:"controller,omitempty"`
+	ControllerAddr string                `json:"controller_addr,omitempty"`
+	Nodes          map[string]slurm.Node `json:"nodes,omitempty"`
+}
+
+func (k Kept) equal(o Kept) bool {
+	if k.Controller != o.Controller || k.ControllerAddr != o.ControllerAddr || len(k.Nodes) != len(o.Nodes) {
+		return false
+	}
+	for name, n := range k.Nodes {
+		m, ok := o.Nodes[name]
+		if !ok || n.Addr != m.Addr || n.Hardware != m.Hardware || !slices.Equal(n.GPUs, m.GPUs) {
+			return false
+		}
+	}
+	return true
+}
+
+// KV is the store's cluster state.
+type KV interface {
+	PutClusterState(ctx context.Context, key, value string) (bool, error)
+	ClusterState(ctx context.Context, key string) (string, error)
+}
+
+// keptKey is the cluster state key it is kept under.
+const keptKey = "slurm-nodes"
+
+// StoreMemory keeps it in the store's cluster state.
+func StoreMemory(kv KV) Memory { return kvMemory{kv} }
+
+type kvMemory struct{ kv KV }
+
+func (m kvMemory) Recall(ctx context.Context) (Kept, error) {
+	var k Kept
+	v, err := m.kv.ClusterState(ctx, keptKey)
+	if errors.Is(err, store.ErrNotFound) {
+		return k, nil
+	}
+	if err != nil {
+		return k, err
+	}
+	err = json.Unmarshal([]byte(v), &k)
+	return k, err
+}
+
+func (m kvMemory) Keep(ctx context.Context, k Kept) error {
+	b, err := json.Marshal(k)
+	if err != nil {
+		return err
+	}
+	_, err = m.kv.PutClusterState(ctx, keptKey, string(b))
+	return err
+}

internal/control/slurmsvc/service.go

@@ -0,0 +1,868 @@
+// Package slurmsvc is the leader's side of the Slurm cluster run beside
+// llmgw (BACKLOG P): it tells each node's agent what it is, writes the
+// controller's files from what the nodes report, and hands nodes between
+// llmgw and Slurm as the cluster's settings say.
+//
+// A node in slurm.lent is drained of llmgw's replicas, as `ctl drain` does,
+// and lent to Slurm once none is left. A node taken out of slurm.lent is
+// drained in Slurm, and is llmgw's again when its last job has ended. Both
+// are kept in force every tick, from the settings and what the agents say,
+// so a leader that takes over carries on where the last one was.
+package slurmsvc
+
+import (
+	"context"
+	"fmt"
+	"log/slog"
+	"maps"
+	"net"
+	"slices"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"google.golang.org/protobuf/proto"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
+)
+
+// Agents is how the service reaches the nodes.
+type Agents interface {
+	Send(node string, msg *llmgwv1.ControlMessage) error
+	ConnectedNodes() []string
+}
+
+// Inventory is what the service reads and changes of llmgw's nodes.
+type Inventory interface {
+	Get(id string) (inventory.Node, bool)
+	SetState(id string, st inventory.NodeState) bool
+	Allocations(nodeID string) []inventory.Allocation
+	Nodes() []inventory.Node
+}
+
+// Phase is where a node of slurm.nodes is between llmgw and Slurm.
+type Phase string
+
+const (
+	// Llmgw: drained in Slurm with no job left, llmgw's to place on.
+	Llmgw Phase = "llmgw"
+	// Lending: in slurm.lent, its replicas being moved off.
+	Lending Phase = "lending"
+	// Lent: Slurm's; llmgw places nothing there.
+	Lent Phase = "lent"
+	// Reclaiming: out of slurm.lent, drained in Slurm, jobs still running.
+	Reclaiming Phase = "reclaiming"
+	// Unknown: its agent has not said how it is in Slurm yet.
+	Unknown Phase = "unknown"
+)
+
+// NodeStatus is one node as the service sees it.
+type NodeStatus struct {
+	Node     string `json:"node"`
+	Phase    Phase  `json:"phase"`
+	Replicas int    `json:"replicas"`
+	// StopsBy is, while it is being lent, when a replica still on it is
+	// stopped even with nowhere else to go: the drain's grace after the
+	// lend began. Zero when not known (a leader that took over meanwhile).
+	StopsBy   time.Time `json:"stops_by,omitempty"`
+	Status    *Reported `json:"slurm,omitempty"`
+	Connected bool      `json:"connected"`
+}
+
+// Reported is the agent's SlurmStatus, for the admin API.
+type Reported struct {
+	Version          string `json:"version,omitempty"`
+	Hardware         string `json:"hardware,omitempty"`
+	SlurmdRunning    bool   `json:"slurmd_running"`
+	SlurmctldRunning bool   `json:"slurmctld_running"`
+	SlurmdbdRunning  bool   `json:"slurmdbd_running"`
+	// Shared is the shared path here: "mounted", "exported", "missing".
+	Shared string `json:"shared,omitempty"`
+	// Containers is what the machine has for jobs in containers, empty
+	// while not both of pyxis and enroot.
+	Containers string `json:"containers,omitempty"`
+	// Users is how many of llmgw's users the machine has.
+	Users       int       `json:"users"`
+	State       string    `json:"state,omitempty"`
+	Reason      string    `json:"reason,omitempty"`
+	Jobs        int       `json:"jobs"`
+	Error       string    `json:"error,omitempty"`
+	FilesSHA256 string    `json:"files_sha256,omitempty"`
+	At          time.Time `json:"at"`
+}
+
+// Status is the whole Slurm cluster, for `ctl slurm status`.
+type Status struct {
+	Enabled    bool         `json:"enabled"`
+	Cluster    string       `json:"cluster"`
+	Controller string       `json:"controller"`
+	Partition  string       `json:"partition"`
+	Ctld       *Reported    `json:"controller_status,omitempty"`
+	Nodes      []NodeStatus `json:"nodes"`
+	// Waiting are nodes left out of slurm.conf until they report their
+	// hardware.
+	Waiting []string `json:"waiting,omitempty"`
+	// Shared is where the shared files are and come from; Users who keeps
+	// the users, and how many llmgw has.
+	Shared    config.SharedStorage `json:"shared"`
+	Users     string               `json:"users,omitempty"`
+	UserCount int                  `json:"user_count,omitempty"`
+	// Containers is whether jobs may run in containers: "on" once every
+	// machine has pyxis and enroot, "installing" until then, empty when
+	// slurm.containers is off.
+	Containers string `json:"containers,omitempty"`
+	// Accounts is how many of Slurm's accounts are llmgw's projects'.
+	Accounts int `json:"accounts,omitempty"`
+}
+
+// User is a Linux user made from an llmgw account.
+type User struct {
+	Login   string   `json:"login"`
+	UID     int      `json:"uid"`
+	Name    string   `json:"name"`
+	Email   string   `json:"email"`
+	SSHKeys []string `json:"ssh_keys,omitempty"`
+}
+
+// Service keeps the Slurm cluster.
+type Service struct {
+	settings func() config.SlurmSettings
+	inv      Inventory
+	agents   Agents
+	key      func(ctx context.Context) ([]byte, error)
+	log      *slog.Logger
+
+	mu sync.Mutex
+	// status is each node's last SlurmStatus.
+	status map[string]*Reported
+	// drained are nodes this service put in Draining, which it alone makes
+	// ready again.
+	drained map[string]bool
+	// lendingAt is when this leader began moving a node's replicas off.
+	lendingAt map[string]time.Time
+	// sent is what each node was last told, so it is told again only when
+	// that changes or its stream is new.
+	sent    map[string]*llmgwv1.Slurm
+	mungeKy []byte
+	kick    chan struct{}
+	// mem keeps slurm.conf's nodes and the controller's address for the
+	// next leader; known is what it held when read, and what was kept
+	// since. loaded is false until it was read: until then the controller
+	// is sent no files that leave out a node, which would drop that node
+	// and its jobs.
+	mem    Memory
+	known  Kept
+	loaded bool
+	// started is the first pass, from which a node neither heard from nor
+	// kept is waited for, for HoldFor.
+	started time.Time
+	// userList reads the users from the accounts; users is what it last
+	// said, read at usersAt. Until it has been read once no node is told
+	// anything of users: an empty list would take every user away.
+	userList func(ctx context.Context) ([]User, error)
+	users    []User
+	usersAt  time.Time
+	usersOK  bool
+	// accountList reads Slurm's accounts from the projects, with the users;
+	// accountsOK is false until it has been read, and the controller is
+	// told nothing of accounts until then: none would take every one away.
+	accountList func(ctx context.Context) ([]slurm.Account, error)
+	accounts    []slurm.Account
+	accountsOK  bool
+	// jobSink records the jobs that ended; recorded are those it took,
+	// recording whether a recording is under way.
+	jobSink   func(ctx context.Context, jobs []JobRecord) error
+	recorded  map[string]time.Time
+	recording bool
+}
+
+// JobRecord is a job that ended, for llmgw's accounting.
+type JobRecord struct {
+	ID, Login, Account, Partition, State string
+	Start, End                           time.Time
+	Elapsed                              time.Duration
+	CPUs, GPUs                           int
+}
+
+// SetAccounts has Slurm's accounts read with list, beside the users.
+func (s *Service) SetAccounts(list func(ctx context.Context) ([]slurm.Account, error)) {
+	s.mu.Lock()
+	s.accountList = list
+	s.mu.Unlock()
+}
+
+// SetJobs has the jobs that ended recorded with record; each is given to it
+// once it took it without an error.
+func (s *Service) SetJobs(record func(ctx context.Context, jobs []JobRecord) error) {
+	s.mu.Lock()
+	s.jobSink = record
+	s.mu.Unlock()
+}
+
+// AccountsOf are the accounts a login may run jobs on.
+func (s *Service) AccountsOf(login string) []string {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	var out []string
+	for _, a := range s.accounts {
+		if slices.Contains(a.Users, login) {
+			out = append(out, a.Name)
+		}
+	}
+	return out
+}
+
+// UsersEvery is how often the users are read again without being asked.
+var UsersEvery = 30 * time.Second
+
+// SetUsers has the users read with list, for slurm.users "llmgw".
+func (s *Service) SetUsers(list func(ctx context.Context) ([]User, error)) {
+	s.mu.Lock()
+	s.userList = list
+	s.mu.Unlock()
+}
+
+// UsersChanged has the users read again now: an account or its keys
+// changed.
+func (s *Service) UsersChanged() {
+	s.mu.Lock()
+	s.usersAt = time.Time{}
+	s.mu.Unlock()
+	s.Kick()
+}
+
+// readUsers reads the users when they are due. Under s.mu.
+func (s *Service) readUsers(ctx context.Context, set config.SlurmSettings) {
+	if set.Users != config.UsersLlmgw || s.userList == nil || time.Since(s.usersAt) < UsersEvery {
+		return
+	}
+	us, err := s.userList(ctx)
+	s.usersAt = time.Now()
+	if err != nil {
+		s.log.Warn("could not read the Slurm users from the accounts; the nodes keep the last ones", "err", err)
+		return
+	}
+	s.users, s.usersOK = us, true
+	if s.accountList == nil {
+		return
+	}
+	as, err := s.accountList(ctx)
+	if err != nil {
+		s.log.Warn("could not read Slurm's accounts from the projects; the controller keeps the last ones", "err", err)
+		return
+	}
+	s.accounts, s.accountsOK = as, true
+}
+
+// takeJobs records the jobs a controller says ended that are not recorded
+// yet, away from the agent's stream.
+func (s *Service) takeJobs(ended []*llmgwv1.SlurmJob) {
+	s.mu.Lock()
+	if s.jobSink == nil || s.recording {
+		s.mu.Unlock()
+		return
+	}
+	if s.recorded == nil {
+		s.recorded = map[string]time.Time{}
+	}
+	for id, at := range s.recorded {
+		if time.Since(at) > 3*24*time.Hour {
+			delete(s.recorded, id)
+		}
+	}
+	var jobs []JobRecord
+	for _, j := range ended {
+		if _, ok := s.recorded[j.GetId()]; ok || j.GetId() == "" {
+			continue
+		}
+		jobs = append(jobs, JobRecord{ID: j.GetId(), Login: j.GetUser(), Account: j.GetAccount(), Partition: j.GetPartition(),
+			State: j.GetState(), Start: time.Unix(j.GetStartUnix(), 0), End: time.Unix(j.GetEndUnix(), 0),
+			Elapsed: time.Duration(j.GetElapsedSeconds()) * time.Second, CPUs: int(j.GetCpus()), GPUs: int(j.GetGpus())})
+	}
+	if len(jobs) == 0 {
+		s.mu.Unlock()
+		return
+	}
+	sink := s.jobSink
+	s.recording = true
+	s.mu.Unlock()
+	go func() {
+		ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
+		defer cancel()
+		err := sink(ctx, jobs)
+		s.mu.Lock()
+		defer s.mu.Unlock()
+		s.recording = false
+		if err != nil {
+			s.log.Warn("could not record the Slurm jobs that ended; tried again with the next status", "jobs", len(jobs), "err", err)
+			return
+		}
+		for _, j := range jobs {
+			s.recorded[j.ID] = time.Now()
+		}
+	}()
+}
+
+// HoldFor is how long a new leader waits for a node it neither has heard
+// from nor finds kept before it writes the controller's files without it:
+// on a cluster that kept nothing yet (one upgraded to this), the nodes
+// report within seconds of a new leader, and the controller keeps the
+// node and its jobs meanwhile.
+var HoldFor = 2 * time.Minute
+
+// New is a Service; key gives the munge key, made once from a cluster
+// secret.
+func New(settings func() config.SlurmSettings, inv Inventory, agents Agents,
+	key func(ctx context.Context) ([]byte, error), log *slog.Logger) *Service {
+	if log == nil {
+		log = slog.Default()
+	}
+	return &Service{settings: settings, inv: inv, agents: agents, key: key, log: log,
+		status: map[string]*Reported{}, drained: map[string]bool{}, lendingAt: map[string]time.Time{}, sent: map[string]*llmgwv1.Slurm{},
+		kick: make(chan struct{}, 1)}
+}
+
+// SetMemory has slurm.conf's nodes kept in mem, and read from it first.
+func (s *Service) SetMemory(mem Memory) {
+	s.mu.Lock()
+	s.mem = mem
+	s.mu.Unlock()
+}
+
+// current is the settings, slurm.all_nodes resolved to the nodes the
+// inventory has.
+func (s *Service) current() config.SlurmSettings {
+	set := s.settings()
+	if !set.AllNodes {
+		return set
+	}
+	var ids []string
+	for _, n := range s.inv.Nodes() {
+		ids = append(ids, n.ID)
+	}
+	return set.WithNodes(ids)
+}
+
+// Every is how often the service looks again without being asked.
+var Every = 10 * time.Second
+
+// Run keeps the cluster until ctx ends.
+func (s *Service) Run(ctx context.Context) {
+	t := time.NewTicker(Every)
+	defer t.Stop()
+	for {
+		s.Tick(ctx)
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+		case <-s.kick:
+		}
+	}
+}
+
+// Kick has the service look again now: the settings changed.
+func (s *Service) Kick() {
+	select {
+	case s.kick <- struct{}{}:
+	default:
+	}
+}
+
+// OnSlurmStatus records what a node's agent says.
+func (s *Service) OnSlurmStatus(node string, st *llmgwv1.SlurmStatus) {
+	r := &Reported{Version: st.GetVersion(), Hardware: st.GetHardware(), SlurmdRunning: st.GetSlurmdRunning(),
+		Shared: st.GetShared(), Users: int(st.GetUsers()), Containers: st.GetContainers(),
+		SlurmctldRunning: st.GetSlurmctldRunning(), SlurmdbdRunning: st.GetSlurmdbdRunning(), State: st.GetState(), Reason: st.GetReason(),
+		Jobs: int(st.GetJobs()), Error: st.GetError(), FilesSHA256: st.GetFilesSha256(), At: time.Now()}
+	s.mu.Lock()
+	prev := s.status[node]
+	s.status[node] = r
+	isCtl := s.settings().Controller == node
+	s.mu.Unlock()
+	if isCtl && len(st.GetEnded()) > 0 {
+		s.takeJobs(st.GetEnded())
+	}
+	if r.Error != "" && (prev == nil || prev.Error != r.Error) {
+		s.log.Warn("a node could not keep its part in Slurm", "node", node, "err", r.Error)
+	}
+	if prev == nil || prev.Hardware != r.Hardware || prev.State != r.State || prev.Jobs != r.Jobs {
+		s.Kick()
+	}
+}
+
+// OnRegister is a node's new stream: it is told again what it is.
+func (s *Service) OnRegister(node string) {
+	s.mu.Lock()
+	delete(s.sent, node)
+	s.mu.Unlock()
+	s.Kick()
+}
+
+// Tick does one pass: the nodes handed over as far as they can be, and
+// each told what it is.
+func (s *Service) Tick(ctx context.Context) {
+	set := s.current()
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	s.recall(ctx)
+	s.readUsers(ctx, set)
+	if s.started.IsZero() {
+		s.started = time.Now()
+	}
+	phases := s.handOver(set)
+	var key []byte
+	if set.Enabled {
+		if s.mungeKy == nil {
+			k, err := s.key(ctx)
+			if err != nil {
+				s.log.Warn("no munge key for Slurm; nothing is sent to the nodes", "err", err)
+				return
+			}
+			s.mungeKy = k
+		}
+		key = s.mungeKy
+	}
+	files, waiting := s.files(set)
+	// Not yet knowing what the last leader knew, a slurm.conf without some
+	// node would take it, and its jobs, from the controller: kept back.
+	hold := s.mem != nil && len(waiting) > 0 && (!s.loaded || time.Since(s.started) < HoldFor)
+	for _, node := range s.agents.ConnectedNodes() {
+		if hold && node == set.Controller {
+			continue
+		}
+		m := s.message(set, node, phases[node] == Lent, key, files)
+		if m == nil {
+			continue
+		}
+		if old, ok := s.sent[node]; ok && proto.Equal(old, m) {
+			continue
+		}
+		if err := s.agents.Send(node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_Slurm{Slurm: m}}); err != nil {
+			s.log.Warn("could not tell a node its part in Slurm", "node", node, "err", err)
+			continue
+		}
+		s.sent[node] = m
+	}
+	s.remember(ctx, set)
+}
+
+// recall reads what the last leader kept, once. Under s.mu.
+func (s *Service) recall(ctx context.Context) {
+	if s.mem == nil || s.loaded {
+		return
+	}
+	k, err := s.mem.Recall(ctx)
+	if err != nil {
+		s.log.Warn("could not read Slurm's nodes from the store; the controller's files wait for every node",
+			"err", err)
+		return
+	}
+	s.known, s.loaded = k, true
+}
+
+// remember keeps slurm.conf's nodes and the controller's address when they
+// changed. Under s.mu.
+func (s *Service) remember(ctx context.Context, set config.SlurmSettings) {
+	if s.mem == nil || !s.loaded || !set.Enabled {
+		return
+	}
+	c, _ := s.cluster(set)
+	now := Kept{Controller: set.Controller, ControllerAddr: c.ControllerAddr, Nodes: map[string]slurm.Node{}}
+	for _, n := range c.Nodes {
+		now.Nodes[n.Name] = n
+	}
+	if now.equal(s.known) {
+		return
+	}
+	if err := s.mem.Keep(ctx, now); err != nil {
+		s.log.Warn("could not keep Slurm's nodes in the store", "err", err)
+		return
+	}
+	s.known = now
+}
+
+// controllerAddr is where the nodes reach slurmctld: the controller's
+// management address, else the one kept. Under s.mu.
+func (s *Service) controllerAddr(set config.SlurmSettings) string {
+	if n, ok := s.inv.Get(set.Controller); ok && n.MgmtAddr != "" {
+		return n.MgmtAddr
+	}
+	if s.known.Controller == set.Controller {
+		return s.known.ControllerAddr
+	}
+	return ""
+}
+
+// handOver moves each node of slurm.nodes a step between llmgw and Slurm,
+// answering where each is. Under s.mu.
+func (s *Service) handOver(set config.SlurmSettings) map[string]Phase {
+	out := map[string]Phase{}
+	if !set.Enabled {
+		// Off: every node is llmgw's again. What Slurm still runs is left
+		// to finish where it is; turning Slurm off with work lent is the
+		// operator's choice.
+		for node := range s.drained {
+			s.ready(node)
+		}
+		return out
+	}
+	for _, node := range set.Nodes {
+		out[node] = s.phase(set, node)
+	}
+	// A node no longer in slurm.nodes is llmgw's.
+	for node := range s.drained {
+		if !slices.Contains(set.Nodes, node) {
+			s.ready(node)
+		}
+	}
+	return out
+}
+
+func (s *Service) phase(set config.SlurmSettings, node string) Phase {
+	n, ok := s.inv.Get(node)
+	st := s.status[node]
+	if set.IsLent(node) {
+		if ok && (n.State == inventory.Ready || n.State == inventory.Cordoned) {
+			s.inv.SetState(node, inventory.Draining)
+			s.drained[node] = true
+			s.lendingAt[node] = time.Now()
+			s.log.Info("lending a node to Slurm: its replicas are moved off first", "node", node,
+				"stopped_by", time.Now().Add(drainGrace).Format(time.TimeOnly))
+		}
+		if !ok || len(s.inv.Allocations(node)) > 0 {
+			return Lending
+		}
+		delete(s.lendingAt, node)
+		return Lent
+	}
+	// Not lent. Until the agent says Slurm has drained it with no job left,
+	// llmgw places nothing new on it: a new leader starts with every node
+	// ready. Cordoned, not drained: llmgw has nothing there to move.
+	if st == nil || slurmBusy(st) {
+		if ok && n.State == inventory.Ready {
+			s.inv.SetState(node, inventory.Cordoned)
+			s.drained[node] = true
+			if st != nil {
+				s.log.Info("taking a node back from Slurm: llmgw places nothing there until its jobs end",
+					"node", node, "jobs", st.Jobs)
+			}
+		}
+		if st == nil {
+			return Unknown
+		}
+		return Reclaiming
+	}
+	delete(s.lendingAt, node)
+	if s.drained[node] {
+		s.ready(node)
+	}
+	return Llmgw
+}
+
+// drainGrace is how long a drained replica waits for its replacement
+// before it is stopped anyway: the reconciler's, which is not a setting.
+var drainGrace = reconciler.DefaultConfig().DrainGrace
+
+// slurmBusy is whether Slurm may still run work on a node: a job step left
+// on it (or no telling), or a controller that would still give it one. A
+// node the controller does not know of, or cannot be asked about, gets no
+// job from it, so a Slurm that is broken never keeps a node from llmgw.
+func slurmBusy(st *Reported) bool {
+	if st.Jobs != 0 {
+		return true
+	}
+	if st.State == "" {
+		return false
+	}
+	return !strings.Contains(st.State, "DRAIN") && !strings.Contains(st.State, "DOWN")
+}
+
+func (s *Service) ready(node string) {
+	if n, ok := s.inv.Get(node); ok && (n.State == inventory.Draining || n.State == inventory.Cordoned) {
+		s.inv.SetState(node, inventory.Ready)
+		s.log.Info("a node is llmgw's again: Slurm has no job there", "node", node)
+	}
+	delete(s.drained, node)
+}
+
+// files are the controller's files, and the nodes left out of them for not
+// having reported their hardware. Under s.mu.
+func (s *Service) files(set config.SlurmSettings) (map[string]string, []string) {
+	c, waiting := s.cluster(set)
+	if c.ControllerAddr == "" {
+		return nil, waiting
+	}
+	f := c.Files()
+	if s.containersReady(set) {
+		f["plugstack.conf"] = slurm.PlugstackConf
+	}
+	return f, waiting
+}
+
+// containersReady is whether jobs may ask for containers: every machine of
+// the Slurm cluster says it has pyxis and enroot. Slurm fails a job on a
+// machine without the plugin it was told to load. Under s.mu.
+func (s *Service) containersReady(set config.SlurmSettings) bool {
+	if !set.Containers {
+		return false
+	}
+	for _, n := range s.machines(set) {
+		if st := s.status[n]; st == nil || st.Containers == "" {
+			return false
+		}
+	}
+	return true
+}
+
+// cluster is what slurm.conf says: each node as it last reported, else as
+// kept from before, and the nodes left out for having neither. Under s.mu.
+func (s *Service) cluster(set config.SlurmSettings) (slurm.Cluster, []string) {
+	c := slurm.Cluster{Name: set.Cluster(), Controller: set.Controller, Partition: set.PartitionName()}
+	if !set.Enabled {
+		return c, nil
+	}
+	c.ControllerAddr = s.controllerAddr(set)
+	c.EnforceAccounts = set.Users == config.UsersLlmgw && s.accountList != nil
+	var waiting []string
+	for _, node := range set.Nodes {
+		sn, ok := s.reported(node)
+		if !ok {
+			sn, ok = s.known.Nodes[node]
+		}
+		if !ok {
+			waiting = append(waiting, node)
+			continue
+		}
+		sn.Lent = set.IsLent(node)
+		c.Nodes = append(c.Nodes, sn)
+	}
+	return c, waiting
+}
+
+// reported is a node as its agent and the inventory say now. Under s.mu.
+func (s *Service) reported(node string) (slurm.Node, bool) {
+	n, ok := s.inv.Get(node)
+	st := s.status[node]
+	if !ok || n.MgmtAddr == "" || st == nil || st.Hardware == "" {
+		return slurm.Node{}, false
+	}
+	hw, err := slurm.ParseHardware(st.Hardware)
+	if err != nil {
+		return slurm.Node{}, false
+	}
+	sn := slurm.Node{Name: node, Addr: n.MgmtAddr, Hardware: hw}
+	for _, g := range n.GPUs {
+		sn.GPUs = append(sn.GPUs, slurm.GPU{Minor: g.Index, Name: g.Name})
+	}
+	return sn, true
+}
+
+// message is what a node is told. Under s.mu.
+func (s *Service) message(set config.SlurmSettings, node string, lent bool, key []byte,
+	files map[string]string) *llmgwv1.Slurm {
+	m := &llmgwv1.Slurm{}
+	if !set.Enabled {
+		return m
+	}
+	isNode := slices.Contains(set.Nodes, node)
+	isCtl := set.Controller == node && files != nil
+	if !isNode && !isCtl {
+		return m
+	}
+	m.Name, m.MungeKey, m.Lent = node, key, lent
+	m.Containers = set.Containers
+	m.Node = isNode
+	if addr := s.controllerAddr(set); addr != "" {
+		m.ConfServer = net.JoinHostPort(addr, strconv.Itoa(slurm.CtldPort))
+	}
+	if isNode && m.ConfServer == "" {
+		// Without its controller a node cannot start; it is told when the
+		// controller registers. Told nothing until then, not "no part": a
+		// node still running Slurm's jobs would stop them.
+		return nil
+	}
+	if isCtl {
+		m.Controller = true
+		m.Files = maps.Clone(files)
+	}
+	if !s.sharedPart(set, node, m) || !s.usersPart(set, node, m) {
+		return nil
+	}
+	if isCtl && set.Users == config.UsersLlmgw && s.accountList != nil {
+		if !s.accountsOK {
+			return nil
+		}
+		m.ManageAccounts = true
+		for _, a := range s.accounts {
+			m.Accounts = append(m.Accounts, &llmgwv1.SlurmAccount{Name: a.Name, Parent: a.Parent, Users: slices.Clone(a.Users)})
+		}
+	}
+	return m
+}
+
+// sharedPart puts the shared files in m, false when where they come from
+// is not known yet: told nothing rather than no shared files, which would
+// unmount them under running jobs. Under s.mu.
+func (s *Service) sharedPart(set config.SlurmSettings, node string, m *llmgwv1.Slurm) bool {
+	sh := set.Shared
+	if sh.Path == "" {
+		return true
+	}
+	m.Shared = &llmgwv1.SlurmShared{Path: sh.Path, Nfs: sh.NFS}
+	switch {
+	case sh.Server == "":
+	case sh.Server == node:
+		m.Shared.Serve = true
+		for _, n := range s.machines(set) {
+			if n == node {
+				continue
+			}
+			if a := s.addr(set, n); a != "" {
+				m.Shared.Clients = append(m.Shared.Clients, a)
+			}
+		}
+		slices.Sort(m.Shared.Clients)
+	default:
+		a := s.addr(set, sh.Server)
+		if a == "" {
+			return false
+		}
+		if strings.Contains(a, ":") {
+			a = "[" + a + "]"
+		}
+		m.Shared.Nfs = a + ":" + sh.Path
+	}
+	return true
+}
+
+// usersPart puts the users in m, false while they have not been read.
+// Under s.mu.
+func (s *Service) usersPart(set config.SlurmSettings, node string, m *llmgwv1.Slurm) bool {
+	if set.Users != config.UsersLlmgw {
+		return true
+	}
+	if !s.usersOK {
+		return false
+	}
+	m.ManageUsers = true
+	switch {
+	case set.Shared.Path == "":
+		m.MakeHomes = true // each machine's own /home
+	case set.Shared.Server != "":
+		m.MakeHomes = set.Shared.Server == node
+	default:
+		m.MakeHomes = set.Controller == node
+	}
+	for _, u := range s.users {
+		m.Users = append(m.Users, &llmgwv1.SlurmUser{Login: u.Login, Uid: uint32(u.UID), Name: u.Name, //nolint:gosec // from FirstSlurmUID up
+			Home: set.HomeDir(u.Login), SshKeys: slices.Clone(u.SSHKeys)})
+	}
+	return true
+}
+
+// machines are the controller and the nodes.
+func (s *Service) machines(set config.SlurmSettings) []string {
+	out := slices.Clone(set.Nodes)
+	if !slices.Contains(out, set.Controller) {
+		out = append(out, set.Controller)
+	}
+	return out
+}
+
+// addr is a machine's management address, from the inventory, else kept
+// from before. Under s.mu.
+func (s *Service) addr(set config.SlurmSettings, node string) string {
+	if n, ok := s.inv.Get(node); ok && n.MgmtAddr != "" {
+		return n.MgmtAddr
+	}
+	if node == set.Controller {
+		return s.controllerAddr(set)
+	}
+	return s.known.Nodes[node].Addr
+}
+
+// Status is the cluster as the service sees it now.
+func (s *Service) Status() Status {
+	set := s.current()
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	out := Status{Enabled: set.Enabled, Cluster: set.Cluster(), Controller: set.Controller,
+		Partition: set.PartitionName(), Ctld: s.status[set.Controller], Shared: set.Shared, Users: set.Users}
+	if set.Users == config.UsersLlmgw {
+		out.UserCount = len(s.users)
+		for _, a := range s.accounts {
+			if a.Project != "" {
+				out.Accounts++
+			}
+		}
+	}
+	if !set.Enabled {
+		return out
+	}
+	if set.Containers {
+		out.Containers = "installing"
+		if s.containersReady(set) {
+			out.Containers = "on"
+		}
+	}
+	_, out.Waiting = s.files(set)
+	conn := s.agents.ConnectedNodes()
+	for _, node := range set.Nodes {
+		ns := NodeStatus{Node: node, Status: s.status[node], Replicas: len(s.inv.Allocations(node)),
+			Connected: slices.Contains(conn, node)}
+		ns.Phase = s.peek(set, node)
+		if at, ok := s.lendingAt[node]; ok && ns.Phase == Lending {
+			ns.StopsBy = at.Add(drainGrace)
+		}
+		out.Nodes = append(out.Nodes, ns)
+	}
+	return out
+}
+
+// peek is phase without changing anything. Under s.mu.
+func (s *Service) peek(set config.SlurmSettings, node string) Phase {
+	st := s.status[node]
+	if set.IsLent(node) {
+		if len(s.inv.Allocations(node)) > 0 {
+			return Lending
+		}
+		return Lent
+	}
+	switch {
+	case st == nil:
+		return Unknown
+	case slurmBusy(st):
+		return Reclaiming
+	}
+	return Llmgw
+}
+
+// Describe is one line for a log or a screen.
+func (n NodeStatus) Describe() string {
+	switch n.Phase {
+	case Lending:
+		if !n.StopsBy.IsZero() {
+			// A replica moves once its replacement serves; one with nowhere
+			// else to go waits out the grace, and that wait is not a hang.
+			return fmt.Sprintf("being lent: %d replica(s) still to move off; one with nowhere else to go is stopped at %s",
+				n.Replicas, n.StopsBy.Local().Format(time.TimeOnly))
+		}
+		return fmt.Sprintf("being lent: %d replica(s) still to move off", n.Replicas)
+	case Lent:
+		return "lent to Slurm"
+	case Reclaiming:
+		if n.Status != nil {
+			return fmt.Sprintf("being taken back: %d job(s) still running", n.Status.Jobs)
+		}
+		return "being taken back"
+	case Llmgw:
+		return "llmgw's"
+	}
+	return "not heard from yet"
+}

internal/control/slurmsvc/service_test.go

@@ -0,0 +1,382 @@
+package slurmsvc
+
+import (
+	"context"
+	"slices"
+	"strings"
+	"testing"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
+)
+
+type fakeInv struct {
+	nodes  map[string]*inventory.Node
+	allocs map[string]int
+}
+
+func (f *fakeInv) Get(id string) (inventory.Node, bool) {
+	n, ok := f.nodes[id]
+	if !ok {
+		return inventory.Node{}, false
+	}
+	return *n, true
+}
+
+func (f *fakeInv) SetState(id string, st inventory.NodeState) bool {
+	n, ok := f.nodes[id]
+	if ok {
+		n.State = st
+	}
+	return ok
+}
+
+func (f *fakeInv) Allocations(id string) []inventory.Allocation {
+	return make([]inventory.Allocation, f.allocs[id])
+}
+
+func (f *fakeInv) Nodes() []inventory.Node {
+	var out []inventory.Node
+	for _, n := range f.nodes {
+		out = append(out, *n)
+	}
+	slices.SortFunc(out, func(a, b inventory.Node) int { return strings.Compare(a.ID, b.ID) })
+	return out
+}
+
+type fakeAgents struct {
+	conn []string
+	got  map[string][]*llmgwv1.Slurm
+}
+
+func (f *fakeAgents) Send(node string, m *llmgwv1.ControlMessage) error {
+	f.got[node] = append(f.got[node], m.GetSlurm())
+	return nil
+}
+
+func (f *fakeAgents) ConnectedNodes() []string { return f.conn }
+
+func (f *fakeAgents) last(node string) *llmgwv1.Slurm {
+	ms := f.got[node]
+	if len(ms) == 0 {
+		return nil
+	}
+	return ms[len(ms)-1]
+}
+
+type rig struct {
+	set    config.SlurmSettings
+	inv    *fakeInv
+	agents *fakeAgents
+	svc    *Service
+}
+
+func newRig() *rig {
+	r := &rig{
+		set: config.SlurmSettings{Enabled: true, Controller: "node06", Nodes: []string{"cuda01"}},
+		inv: &fakeInv{nodes: map[string]*inventory.Node{
+			"node06": {ID: "node06", MgmtAddr: "10.0.0.146"},
+			"cuda01": {ID: "cuda01", MgmtAddr: "10.0.0.147", GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA GeForce RTX 3090"}}},
+		}, allocs: map[string]int{"cuda01": 2}},
+		agents: &fakeAgents{conn: []string{"node06", "cuda01"}, got: map[string][]*llmgwv1.Slurm{}},
+	}
+	r.svc = New(func() config.SlurmSettings { return r.set }, r.inv, r.agents,
+		func(context.Context) ([]byte, error) { return []byte(strings.Repeat("k", 1024)), nil }, nil)
+	return r
+}
+
+func (r *rig) report(node string, st *llmgwv1.SlurmStatus) { r.svc.OnSlurmStatus(node, st) }
+
+func (r *rig) tick() { r.svc.Tick(context.Background()) }
+
+const hw = "NodeName=cuda01 CPUs=16 RealMemory=64000"
+
+func TestMessagesAndFiles(t *testing.T) {
+	r := newRig()
+	r.tick()
+	ctl, node := r.agents.last("node06"), r.agents.last("cuda01")
+	if !ctl.GetController() || ctl.GetNode() || len(ctl.GetMungeKey()) != 1024 {
+		t.Fatalf("controller told %v", ctl)
+	}
+	if !node.GetNode() || node.GetController() || node.GetConfServer() != "10.0.0.146:6817" || node.GetLent() ||
+		node.GetName() != "cuda01" || len(node.GetFiles()) != 0 {
+		t.Fatalf("node told %v", node)
+	}
+	if strings.Contains(ctl.GetFiles()["slurm.conf"], "NodeName=cuda01") {
+		t.Fatal("a node with no hardware reported is in slurm.conf")
+	}
+	if w := r.svc.Status().Waiting; len(w) != 1 || w[0] != "cuda01" {
+		t.Fatalf("waiting %v", w)
+	}
+	// Same again: nobody is told twice.
+	r.tick()
+	if len(r.agents.got["node06"]) != 1 || len(r.agents.got["cuda01"]) != 1 {
+		t.Fatal("told again with nothing changed")
+	}
+	r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
+	r.tick()
+	conf := r.agents.last("node06").GetFiles()["slurm.conf"]
+	if !strings.Contains(conf, "NodeName=cuda01 NodeAddr=10.0.0.147 CPUs=16 RealMemory=62080 Gres=gpu:rtx3090:1") {
+		t.Fatalf("slurm.conf:\n%s", conf)
+	}
+	if !strings.Contains(r.agents.last("node06").GetFiles()["gres.conf"], "File=/dev/nvidia0") {
+		t.Fatal("gres.conf has no card")
+	}
+	if len(r.agents.got["cuda01"]) != 1 {
+		t.Fatal("a node was told again when only the controller's files changed")
+	}
+	// A new stream is told again.
+	r.svc.OnRegister("cuda01")
+	r.tick()
+	if len(r.agents.got["cuda01"]) != 2 {
+		t.Fatal("a new stream was not told")
+	}
+	if r.inv.nodes["cuda01"].State != inventory.Ready {
+		t.Fatal("a node drained in Slurm with no job was taken from llmgw")
+	}
+}
+
+func TestLendAndReclaim(t *testing.T) {
+	r := newRig()
+	r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
+	r.set.Lent = []string{"cuda01"}
+	r.tick()
+	if r.inv.nodes["cuda01"].State != inventory.Draining {
+		t.Fatal("a lent node was not drained of its replicas")
+	}
+	if r.agents.last("cuda01").GetLent() {
+		t.Fatal("lent to Slurm with replicas still on it")
+	}
+	if p := r.svc.Status().Nodes[0].Phase; p != Lending {
+		t.Fatalf("phase %s", p)
+	}
+	r.inv.allocs["cuda01"] = 0
+	r.tick()
+	if !r.agents.last("cuda01").GetLent() || r.svc.Status().Nodes[0].Phase != Lent {
+		t.Fatal("not lent once its replicas were gone")
+	}
+	r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "ALLOCATED", Jobs: 3})
+
+	// Taken back: Slurm drains it; llmgw keeps off until its jobs end.
+	r.set.Lent = nil
+	r.tick()
+	if r.agents.last("cuda01").GetLent() {
+		t.Fatal("still lent")
+	}
+	if s := r.inv.nodes["cuda01"].State; s != inventory.Draining && s != inventory.Cordoned {
+		t.Fatalf("llmgw may place on a node with Slurm jobs: %v", s)
+	}
+	r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "ALLOCATED+DRAIN", Jobs: 1})
+	r.tick()
+	if r.svc.Status().Nodes[0].Phase != Reclaiming || r.inv.nodes["cuda01"].State == inventory.Ready {
+		t.Fatal("given back with a job left")
+	}
+	r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: serving models"})
+	r.tick()
+	if r.inv.nodes["cuda01"].State != inventory.Ready || r.svc.Status().Nodes[0].Phase != Llmgw {
+		t.Fatal("not llmgw's when its last job ended")
+	}
+}
+
+func TestNewLeaderKeepsOffUntilHeard(t *testing.T) {
+	r := newRig()
+	r.inv.allocs["cuda01"] = 0
+	r.tick()
+	// Nothing heard yet: cordoned, nothing moved.
+	if r.inv.nodes["cuda01"].State != inventory.Cordoned {
+		t.Fatalf("state %v before the agent said anything", r.inv.nodes["cuda01"].State)
+	}
+	r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "MIXED", Jobs: 2})
+	r.tick()
+	if r.inv.nodes["cuda01"].State == inventory.Ready {
+		t.Fatal("ready with Slurm jobs on it")
+	}
+}
+
+func TestBrokenSlurmLeavesTheNodeToLlmgw(t *testing.T) {
+	r := newRig()
+	r.report("cuda01", &llmgwv1.SlurmStatus{Error: "installing Slurm: no package", Jobs: 0})
+	r.tick()
+	if r.inv.nodes["cuda01"].State != inventory.Ready {
+		t.Fatal("a Slurm that does not work took a node from llmgw")
+	}
+}
+
+func TestOperatorsCordonIsLeft(t *testing.T) {
+	r := newRig()
+	r.inv.nodes["cuda01"].State = inventory.Cordoned
+	r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN"})
+	r.tick()
+	if r.inv.nodes["cuda01"].State != inventory.Cordoned {
+		t.Fatal("llmgw uncordoned a node an operator cordoned")
+	}
+}
+
+func TestOffAndOut(t *testing.T) {
+	r := newRig()
+	r.inv.allocs["cuda01"] = 0
+	r.set.Lent = []string{"cuda01"}
+	r.tick()
+	r.set.Enabled = false
+	r.tick()
+	if m := r.agents.last("cuda01"); m.GetNode() || m.GetController() {
+		t.Fatalf("told %v with Slurm off", m)
+	}
+	if r.inv.nodes["cuda01"].State != inventory.Ready {
+		t.Fatal("not llmgw's with Slurm off")
+	}
+
+	r = newRig()
+	r.inv.allocs["cuda01"] = 0
+	r.set.Lent = []string{"cuda01"}
+	r.tick()
+	r.set.Nodes, r.set.Lent = nil, nil
+	r.tick()
+	if r.inv.nodes["cuda01"].State != inventory.Ready || r.agents.last("cuda01").GetNode() {
+		t.Fatal("a node taken out of slurm.nodes is not llmgw's")
+	}
+	if !r.agents.last("node06").GetController() {
+		t.Fatal("the controller stopped with no nodes")
+	}
+}
+
+func TestNoControllerYet(t *testing.T) {
+	r := newRig()
+	delete(r.inv.nodes, "node06")
+	r.agents.conn = []string{"cuda01"}
+	r.tick()
+	if m := r.agents.last("cuda01"); m.GetNode() {
+		t.Fatalf("a node was started with no controller to fetch its files from: %v", m)
+	}
+}
+
+func TestAllNodes(t *testing.T) {
+	r := newRig()
+	r.set.Nodes, r.set.AllNodes = nil, true
+	r.inv.nodes["cuda02"] = &inventory.Node{ID: "cuda02", MgmtAddr: "10.0.0.150"}
+	r.agents.conn = append(r.agents.conn, "cuda02")
+	r.tick()
+	for _, n := range []string{"cuda01", "cuda02", "node06"} {
+		if !r.agents.last(n).GetNode() {
+			t.Fatalf("%s, a node that joined, does not run slurmd", n)
+		}
+	}
+	if got := len(r.svc.Status().Nodes); got != 3 {
+		t.Fatalf("%d nodes in the status", got)
+	}
+	r.set.Lent = []string{"cuda02"}
+	r.tick()
+	if p := r.svc.Status().Nodes[1]; p.Node != "cuda02" || p.Phase != Lent {
+		t.Fatalf("cuda02 not lent: %+v", p)
+	}
+}
+
+func TestLendingSaysWhenAReplicaWithNowhereToGoStops(t *testing.T) {
+	// Seen on the PoC: a GPU model with no other card waited out the drain's
+	// grace, and five minutes of "1 replica(s) still to move off" read as a hang.
+	at := time.Date(2026, 11, 14, 19, 38, 34, 0, time.Local)
+	n := NodeStatus{Node: "cuda01", Phase: Lending, Replicas: 1, StopsBy: at}
+	if d := n.Describe(); !strings.Contains(d, "nowhere else to go is stopped at 19:38:34") {
+		t.Errorf("%q", d)
+	}
+	n.StopsBy = time.Time{}
+	if d := n.Describe(); d != "being lent: 1 replica(s) still to move off" {
+		t.Errorf("%q", d)
+	}
+}
+
+func TestPyxisIsLoadedOnlyOnceEveryMachineHasIt(t *testing.T) {
+	// "required" in plugstack.conf fails every job on a machine without the
+	// plugin; sent early, it would stop Slurm's jobs everywhere it is missing.
+	r := newRig()
+	r.set.Containers = true
+	r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw})
+	r.tick()
+	if !r.agents.last("cuda01").GetContainers() || !r.agents.last("node06").GetContainers() {
+		t.Fatal("the machines were not told to set containers up")
+	}
+	if _, ok := r.agents.last("node06").GetFiles()["plugstack.conf"]; ok {
+		t.Fatal("pyxis loaded before any machine has it")
+	}
+	if s := r.svc.Status().Containers; s != "installing" {
+		t.Errorf("status %q", s)
+	}
+	r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, Containers: "pyxis 0.24.0, enroot 4.2.1"})
+	r.tick()
+	if _, ok := r.agents.last("node06").GetFiles()["plugstack.conf"]; ok {
+		t.Fatal("pyxis loaded with the controller still without it")
+	}
+	r.report("node06", &llmgwv1.SlurmStatus{Containers: "pyxis 0.24.0, enroot 4.2.1"})
+	r.tick()
+	if p := r.agents.last("node06").GetFiles()["plugstack.conf"]; !strings.Contains(p, "required /usr/local/lib/slurm/spank_pyxis.so") {
+		t.Fatalf("plugstack.conf %q", p)
+	}
+	if s := r.svc.Status().Containers; s != "on" {
+		t.Errorf("status %q", s)
+	}
+	r.set.Containers = false
+	r.tick()
+	if _, ok := r.agents.last("node06").GetFiles()["plugstack.conf"]; ok || r.svc.Status().Containers != "" {
+		t.Fatal("pyxis still loaded with containers off")
+	}
+}
+
+func TestTheControllerKeepsTheProjectsAccountsAndJobsAreRecordedOnce(t *testing.T) {
+	r := newRig()
+	r.set.Users = config.UsersLlmgw
+	r.svc.SetUsers(func(context.Context) ([]User, error) { return []User{{Login: "ayse", UID: 200001}}, nil })
+	read := 0
+	r.svc.SetAccounts(func(context.Context) ([]slurm.Account, error) {
+		read++
+		return []slurm.Account{{Name: "test", Parent: "root"},
+			{Name: "test-arge-deneme", Parent: "test", Project: "p1", Users: []string{"ayse"}}}, nil
+	})
+	got := make(chan []JobRecord, 4)
+	r.svc.SetJobs(func(_ context.Context, jobs []JobRecord) error { got <- jobs; return nil })
+	r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw})
+	r.tick()
+	ctl := r.agents.last("node06")
+	if !ctl.GetManageAccounts() || len(ctl.GetAccounts()) != 2 || ctl.GetAccounts()[1].GetUsers()[0] != "ayse" {
+		t.Fatalf("controller told %v", ctl.GetAccounts())
+	}
+	if r.agents.last("cuda01").GetManageAccounts() {
+		t.Error("a node was told the accounts")
+	}
+	if !strings.Contains(ctl.GetFiles()["slurm.conf"], "AccountingStorageEnforce=associations") {
+		t.Error("jobs not held to accounts")
+	}
+	if r.svc.Status().Accounts != 1 || len(r.svc.AccountsOf("ayse")) != 1 {
+		t.Errorf("status %d, ayse's %v", r.svc.Status().Accounts, r.svc.AccountsOf("ayse"))
+	}
+	ended := []*llmgwv1.SlurmJob{{Id: "41", User: "ayse", Account: "test-arge-deneme", ElapsedSeconds: 60, Gpus: 1}}
+	r.report("node06", &llmgwv1.SlurmStatus{Ended: ended})
+	select {
+	case jobs := <-got:
+		if len(jobs) != 1 || jobs[0].Login != "ayse" || jobs[0].GPUs != 1 {
+			t.Fatalf("jobs %+v", jobs)
+		}
+	case <-time.After(5 * time.Second):
+		t.Fatal("not recorded")
+	}
+	// The next status says it again: not recorded twice.
+	deadline := time.Now().Add(2 * time.Second)
+	for time.Now().Before(deadline) {
+		r.svc.mu.Lock()
+		busy := r.svc.recording
+		r.svc.mu.Unlock()
+		if !busy {
+			break
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+	r.report("node06", &llmgwv1.SlurmStatus{Ended: ended})
+	select {
+	case jobs := <-got:
+		t.Fatalf("recorded again: %+v", jobs)
+	case <-time.After(200 * time.Millisecond):
+	}
+}

internal/control/slurmsvc/shared_test.go

@@ -0,0 +1,85 @@
+package slurmsvc
+
+import (
+	"context"
+	"errors"
+	"slices"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+)
+
+func TestSharedFilesFromTheController(t *testing.T) {
+	r := newRig()
+	r.set.Shared = config.SharedStorage{Path: "/shared", Server: "node06"}
+	r.tick()
+	ctl, node := r.agents.last("node06").GetShared(), r.agents.last("cuda01").GetShared()
+	if !ctl.GetServe() || !slices.Equal(ctl.GetClients(), []string{"10.0.0.147"}) || ctl.GetPath() != "/shared" {
+		t.Fatalf("server told %v", ctl)
+	}
+	if node.GetServe() || node.GetNfs() != "10.0.0.146:/shared" {
+		t.Fatalf("client told %v", node)
+	}
+	// The site's own export: every machine mounts it.
+	r.set.Shared = config.SharedStorage{Path: "/shared", NFS: "filer:/hpc"}
+	r.tick()
+	if r.agents.last("node06").GetShared().GetNfs() != "filer:/hpc" || r.agents.last("cuda01").GetShared().GetServe() {
+		t.Fatal("the site's export not sent")
+	}
+}
+
+func TestSharedServerNotKnownTellsNothing(t *testing.T) {
+	r := newRig()
+	r.set.Shared = config.SharedStorage{Path: "/shared", Server: "node06"}
+	delete(r.inv.nodes, "node06")
+	r.svc.known.ControllerAddr, r.svc.known.Controller = "10.0.0.146", "node06"
+	r.tick()
+	if r.agents.last("cuda01").GetShared().GetNfs() != "10.0.0.146:/shared" {
+		t.Fatalf("the kept address not used: %v", r.agents.last("cuda01"))
+	}
+	r2 := newRig()
+	r2.set.Nodes = []string{"cuda01", "site2"}
+	r2.agents.conn = append(r2.agents.conn, "site2")
+	r2.set.Shared = config.SharedStorage{Path: "/shared", Server: "site2"}
+	r2.tick()
+	if r2.agents.last("cuda01") != nil {
+		t.Fatal("told with no address for the server: its mount would go")
+	}
+}
+
+func TestUsersSentOnceRead(t *testing.T) {
+	r := newRig()
+	r.set.Users = config.UsersLlmgw
+	r.set.Shared = config.SharedStorage{Path: "/shared", Server: "node06"}
+	fail := true
+	r.svc.SetUsers(func(context.Context) ([]User, error) {
+		if fail {
+			return nil, errors.New("store down")
+		}
+		return []User{{Login: "ayse", UID: 200000, Name: "Ayşe", SSHKeys: []string{"ssh-ed25519 AAAA"}}}, nil
+	})
+	r.tick()
+	if len(r.agents.got["cuda01"]) != 0 {
+		t.Fatal("told before the users were read: every user would go")
+	}
+	fail = false
+	r.svc.UsersChanged()
+	r.tick()
+	ctl, node := r.agents.last("node06"), r.agents.last("cuda01")
+	if !ctl.GetManageUsers() || !ctl.GetMakeHomes() || node.GetMakeHomes() || len(node.GetUsers()) != 1 {
+		t.Fatalf("controller %v\nnode %v", ctl, node)
+	}
+	if u := node.GetUsers()[0]; u.GetHome() != "/shared/home/ayse" || u.GetUid() != 200000 || len(u.GetSshKeys()) != 1 {
+		t.Fatalf("user %v", u)
+	}
+	// The store down later: the last users are kept.
+	fail = true
+	r.svc.UsersChanged()
+	r.tick()
+	if len(r.agents.last("cuda01").GetUsers()) != 1 {
+		t.Fatal("users lost when the store could not be read")
+	}
+	if st := r.svc.Status(); st.UserCount != 1 || st.Shared.Server != "node06" {
+		t.Fatalf("status %+v", st)
+	}
+}

internal/directory/directory.go

@@ -94,7 +94,7 @@ type Store interface {
 	SetStaffRole(ctx context.Context, id, role string) error
 	SetUserState(ctx context.Context, id string, state store.UserState) error
 	DeleteUserSessions(ctx context.Context, userID string) error
-	AddMembership(ctx context.Context, user string, scope store.Scope, id, role, by string, at time.Time) error
+	EnsureMembership(ctx context.Context, user string, scope store.Scope, id, by string, at time.Time) error
 	RemoveMembership(ctx context.Context, userID string, scope store.Scope, scopeID string, at time.Time) (int, error)
 	MembershipsOf(ctx context.Context, userID string) ([]store.Membership, error)
 	AllOrgs(ctx context.Context) ([]store.Org, error)
@@ -221,7 +221,7 @@ func (k *Keeper) apply(ctx context.Context, u *store.User, id Identity, remove b
 			continue // a mapping to a place the tree lacks: said by Sync
 		}
 		want[string(scope)+"/"+sid] = scope
-		if err := k.St.AddMembership(ctx, u.ID, scope, sid, store.RoleMember, k.Name, k.now()); err != nil {
+		if err := k.St.EnsureMembership(ctx, u.ID, scope, sid, k.Name, k.now()); err != nil {
 			return err
 		}
 	}

internal/docgen/reference_gen.go

@@ -39,6 +39,8 @@ var Reference = []Block{
 		{Key: "personal", Type: "Personal", Doc: "Personal is what a person may do on their own, outside any project: make\npersonal keys and use the chat page personally, within these limits.", Nested: "config.Personal"},
 		{Key: "pricing", Type: "map[string]Price", Doc: "Pricing is what each model costs, for budgets. A model absent from the\ntable costs whatever DefaultPrice says, which is nothing unless set.", Nested: ""},
 		{Key: "default_price", Type: "Price", Doc: "", Nested: "config.Price"},
+		{Key: "gpu_hour_cost", Type: "float64", Doc: "GPUHourCost is what an hour of one GPU costs the operator, in the\nunits budgets are in: the hardware written off over its years, its\npower and room, and the people who run it, over the GPUs and the\nhours of a month. With CoreHourCost for a CPU core, it is what\n`llmgw ctl prices suggest` prices each model from: the hours its\nreplicas held, idle ones too, over the tokens it served.", Nested: ""},
+		{Key: "core_hour_cost", Type: "float64", Doc: "", Nested: ""},
 		{Key: "flush_interval", Type: "time.Duration", Doc: "FlushInterval and BatchSize decide when usage records are written.", Nested: ""},
 		{Key: "batch_size", Type: "int", Doc: "", Nested: ""},
 		{Key: "buffer_size", Type: "int", Doc: "BufferSize bounds the queue. Beyond it records are dropped rather than\nblocking a request: losing accounting is bad, turning a stalled database\ninto an outage is worse.", Nested: ""},
@@ -480,6 +482,7 @@ var Reference = []Block{
 		{Key: "mail", Type: "mail.Config", Doc: "Mail is where the gateway's few emails go out from: a link to set a\nforgotten password. Unset, none is sent and people ask an operator.", Nested: "mail.Config"},
 		{Key: "serving", Type: "ServingSettings", Doc: "Serving stops and starts the whole cluster's service (llmgw stop, llmgw\nstart).", Nested: "config.ServingSettings"},
 		{Key: "firewall", Type: "FirewallSettings", Doc: "Firewall turns llmgw's firewall on every machine on or off.", Nested: "config.FirewallSettings"},
+		{Key: "slurm", Type: "SlurmSettings", Doc: "Slurm is a Slurm cluster run beside llmgw on some of its nodes, and\nwhich of them it may use now.", Nested: "config.SlurmSettings"},
 		{Key: "presets", Type: "PresetSettings", Doc: "Presets is how the models that come with llmgw reach the catalog.", Nested: "config.PresetSettings"},
 		{Key: "on_demand", Type: "OnDemandSettings", Doc: "OnDemand is how models that load when asked for take turns on the\ncards.", Nested: "config.OnDemandSettings"},
 		{Key: "agents", Type: "AgentSettings", Doc: "Agents are the nodes' agent settings: agent.yaml's keys but those an\nagent needs to start, for every node (default) and for one\n(nodes.<id>). Sent to the agents as they change; an agent whose\nsettings changed starts again on them, its engines serving\nthroughout.", Nested: "config.AgentSettings"},
@@ -509,9 +512,26 @@ var Reference = []Block{
 	{Package: "config", Name: "ServingSettings", Title: "ServingSettings", Doc: "", Fields: []Field{
 		{Key: "stopped", Type: "bool", Doc: "Stopped takes every model down and answers every request for one with\n503 (system_stopped), leaving the control planes, agents and mirror\nrunning and every model's definition as it is: llmgw stop sets it,\nllmgw start clears it, and the models come back as their catalog\nentries ask. In force on every control plane within seconds.", Nested: ""},
 	}},
+	{Package: "config", Name: "SharedStorage", Title: "SharedStorage", Doc: "", Fields: []Field{
+		{Key: "path", Type: "string", Doc: "Path is where every machine sees it, as /shared.", Nested: ""},
+		{Key: "nfs", Type: "string", Doc: "NFS is the site's export, host:/path, mounted at Path on every\nmachine of the Slurm cluster.", Nested: ""},
+		{Key: "server", Type: "string", Doc: "Server is the llmgw node, the controller or one of the nodes, that\nexports Path over NFS to the others: for a site with no storage of\nits own. One machine: not highly available, and no faster than its\ndisk and network.", Nested: ""},
+	}},
 	{Package: "engines", Name: "SimConfig", Title: "SimConfig", Doc: "", Fields: []Field{
 		{Key: "bin", Type: "string", Doc: "Bin is the llmgw binary itself; the engine is one of its subcommands.\nEmpty means whatever started this agent, which is right almost always.", Nested: ""},
 	}},
+	{Package: "config", Name: "SlurmSettings", Title: "SlurmSettings", Doc: "", Fields: []Field{
+		{Key: "enabled", Type: "bool", Doc: "Enabled runs Slurm on the nodes below. False stops llmgw keeping it;\nwhat was installed is left installed, and every node is llmgw's.", Nested: ""},
+		{Key: "cluster_name", Type: "string", Doc: "ClusterName is Slurm's ClusterName. Empty is \"llmgw\".", Nested: ""},
+		{Key: "controller", Type: "string", Doc: "Controller is the llmgw node whose agent runs slurmctld. It needs no\nGPU, and need not be one of Nodes.", Nested: ""},
+		{Key: "nodes", Type: "[]string", Doc: "Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained\nin Slurm, until it is also in Lent.", Nested: ""},
+		{Key: "all_nodes", Type: "bool", Doc: "AllNodes runs slurmd on every llmgw node, those that join later too,\nbesides any named in Nodes. Each is llmgw's until it is lent, as any.", Nested: ""},
+		{Key: "lent", Type: "[]string", Doc: "Lent are the nodes of Nodes Slurm may run jobs on now. A node put here\nis drained of llmgw's replicas first, as `ctl drain` does, and resumed\nin Slurm once none is left; a node taken out is drained in Slurm and\ngiven back to llmgw when its last job ends.", Nested: ""},
+		{Key: "partition", Type: "string", Doc: "Partition is the one partition every node is in. Empty is \"gpu\".", Nested: ""},
+		{Key: "shared", Type: "SharedStorage", Doc: "Shared is the file system every machine of the Slurm cluster sees at\nthe same path: home directories, datasets, the jobs' output and\ncheckpoints. Empty path: none, and a batch job's output stays on the\nnode it ran on.", Nested: "config.SharedStorage"},
+		{Key: "users", Type: "string", Doc: "Users is who keeps the Linux users that run jobs, the same on every\nmachine of the Slurm cluster: \"llmgw\" makes one for each active llmgw\naccount (its login from the e-mail address, a UID of llmgw's from\n200000 up, its SSH keys from the portal), let in to the controller,\nand to a node only while a job of theirs runs there (pam_slurm_adopt);\nempty leaves users to the site (a directory through sssd, or by hand).", Nested: ""},
+		{Key: "containers", Type: "bool", Doc: "Containers lets jobs run in containers: `srun --container-image\nnvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),\nthe GPUs and the shared path inside. The agents install both on\nevery Slurm machine; jobs take it once every machine has them.", Nested: ""},
+	}},
 	{Package: "engines", Name: "StrataConfig", Title: "StrataConfig", Doc: "", Fields: []Field{
 		{Key: "dir", Type: "string", Doc: "Dir is Strata's source directory, where setup.py and serve/ are.", Nested: ""},
 		{Key: "python", Type: "string", Doc: "Python is the interpreter of its environment; Dir's .venv when unset.", Nested: ""},

internal/docgen/render.go

@@ -41,6 +41,7 @@ var Sources = []string{
 	"internal/config/serving.go",
 	"internal/config/presets.go",
 	"internal/config/ondemand.go",
+	"internal/config/slurm.go",
 	"internal/config/agents.go",
 	"internal/mail/mail.go",
 	"internal/notify/notify.go",

internal/modelplan/options.go

@@ -173,6 +173,29 @@ func (o Option) WithContext(kvPerToken int64, tokens int) Option {
 	return o
 }
 
+// MostContext is the longest context, up to trained, this option still holds
+// for one request, in whole thousands of tokens below trained: trained when
+// the KV cache's size is unknown, 0 when trained is, for Strata (its context
+// is its own), or when not even 1024 tokens fit.
+func (o Option) MostContext(kvPerToken int64, trained int) int {
+	switch {
+	case trained <= 0 || o.Engine == "strata":
+		return 0
+	case kvPerToken <= 0 || o.WithContext(kvPerToken, trained).Fits():
+		return trained
+	}
+	lo, hi := 0, trained/1024+1 // lo fits (or is none), hi does not
+	for hi-lo > 1 {
+		mid := (lo + hi) / 2
+		if o.WithContext(kvPerToken, mid*1024).Fits() {
+			lo = mid
+		} else {
+			hi = mid
+		}
+	}
+	return lo * 1024
+}
+
 // MemoryText says what it needs, part by part.
 func (o Option) MemoryText() string {
 	if o.Engine == "strata" {

internal/modelplan/options_test.go

@@ -193,3 +193,23 @@ func TestAnOptionSaysWhatItNeedsOfEachCardAndOfAll(t *testing.T) {
 		t.Errorf("a 2M-token context fits four cards: %s", huge.MemoryText())
 	}
 }
+
+func TestTheMostContextAnOptionHolds(t *testing.T) {
+	// 4 GB of weights on a 24 GB card, a token's cache 1 MiB.
+	o := Option{Engine: "llamacpp", Class: "6000-24g", GPUsPerNode: 1, Nodes: 1,
+		WeightsGB: 4, EngineGB: 1, NeedGB: 5, HaveGB: 21.6}
+	const kv = 1 << 20
+	if got := o.MostContext(kv, 8192); got != 8192 {
+		t.Errorf("a context that fits whole: %d, want what it was trained for", got)
+	}
+	got := o.MostContext(kv, 131072)
+	if got <= 0 || got%1024 != 0 || !o.WithContext(kv, got).Fits() || o.WithContext(kv, got+1024).Fits() {
+		t.Errorf("the most that fits is %d", got)
+	}
+	if got := o.MostContext(0, 32768); got != 32768 {
+		t.Errorf("an unknown cache size: %d, want what it was trained for", got)
+	}
+	if got := (Option{Engine: "strata"}).MostContext(kv, 32768); got != 0 {
+		t.Errorf("Strata's context is its own: %d", got)
+	}
+}

internal/onboard/onboard.go

@@ -79,6 +79,9 @@ type Options struct {
 
 	// Addr is how to reach the machine over SSH, and User who to log in as.
 	Addr, User string
+	// FallbackUser logs in when User is empty and the ssh config names no
+	// user for the machine.
+	FallbackUser string
 	// Node is its id in the cluster. Empty means its hostname, which is what
 	// it should be: the node id, the hostname and the certificate's name
 	// being one string saves an afternoon each time they would differ.
@@ -193,7 +196,7 @@ type Runner struct {
 
 // New makes a Runner that reaches the machine with the system's ssh.
 func New(o Options, emit func(Event)) *Runner {
-	return &Runner{Opts: o, Remote: &SSH{Addr: o.Addr, User: o.User}, Emit: emit}
+	return &Runner{Opts: o, Remote: &SSH{Addr: o.Addr, User: o.User, FallbackUser: o.FallbackUser}, Emit: emit}
 }
 
 func (r *Runner) emit(step, level, detail, fix string) {
@@ -302,6 +305,9 @@ func (r *Runner) connect(ctx context.Context, res *Result) error {
 }
 
 func (r *Runner) target() string {
+	if s, ok := r.Remote.(*SSH); ok {
+		return s.Target(context.Background())
+	}
 	if r.Opts.User == "" {
 		return r.Opts.Addr
 	}

internal/onboard/ssh.go

@@ -5,6 +5,7 @@ import (
 	"context"
 	"os/exec"
 	"runtime"
+	"sync"
 )
 
 // Remote runs a shell command on the machine being onboarded, with input.
@@ -14,8 +15,39 @@ type Remote interface {
 
 // SSH is Remote through the system's ssh, so the operator's keys, agent and
 // ~/.ssh/config apply exactly as they do when they log in by hand.
+//
+// Addr is looked up in the ssh config first: a Host block whose HostName is
+// Addr is connected through by its name, so its User, IdentityFile and
+// ProxyJump apply to a machine given by address, without the address
+// having to be added to its Host line.
 type SSH struct {
 	Addr, User string
+	// FallbackUser logs in when User is empty and the ssh config names no
+	// user for the machine: under sudo, the person who ran it.
+	FallbackUser string
+
+	once   sync.Once
+	target string
+}
+
+// Target is what ssh is given: [user@]host, host being the ssh config's
+// name for Addr when it has one.
+func (s *SSH) Target(ctx context.Context) string {
+	s.once.Do(func() {
+		host := s.Addr
+		if a := configAlias(ctx, s.Addr); a != "" {
+			host = a
+		}
+		u := s.User
+		if u == "" && configUser(ctx, host) == "" {
+			u = s.FallbackUser
+		}
+		s.target = host
+		if u != "" {
+			s.target = u + "@" + host
+		}
+	})
+	return s.target
 }
 
 // sshOptions never prompt: a question nobody is there to answer would hang
@@ -37,10 +69,7 @@ var sshOptions = []string{
 // Run runs script with sh on the machine. Files travel on standard input,
 // not scp, which a hardened sshd may not offer.
 func (s *SSH) Run(ctx context.Context, script string, stdin []byte) ([]byte, error) {
-	target := s.Addr
-	if s.User != "" {
-		target = s.User + "@" + s.Addr
-	}
+	target := s.Target(ctx)
 	// "--" before the target: an address beginning with "-" is not read as
 	// an option.
 	args := append(append([]string{}, sshOptions...), "--", target, "sh", "-c", shellQuote(script))

internal/onboard/sshconfig.go

@@ -0,0 +1,223 @@
+package onboard
+
+import (
+	"bufio"
+	"bytes"
+	"context"
+	"net"
+	"os"
+	"os/exec"
+	"os/user"
+	"path/filepath"
+	"strings"
+	"time"
+)
+
+// Where ssh reads its configuration, and arguments every `ssh -G` here is
+// given; variables so a test can point them at a file of its own.
+var (
+	userSSHConfig   = func() string { return filepath.Join(sshHome(), ".ssh", "config") }
+	systemSSHConfig = "/etc/ssh/ssh_config"
+	sshExtraArgs    []string
+)
+
+// sshHome is the home ssh itself reads ~/.ssh from: the account's, from the
+// password database, not $HOME, which sudo may leave as the caller's.
+func sshHome() string {
+	if u, err := user.Current(); err == nil && u.HomeDir != "" {
+		return u.HomeDir
+	}
+	h, _ := os.UserHomeDir()
+	return h
+}
+
+// configAlias is the ssh config's name for addr: the first concrete Host
+// whose HostName is addr, or resolves to the same address. A machine is
+// usually written in the config under its name, `Host gpu01` with
+// `HostName 10.0.0.21` and the User, IdentityFile or ProxyJump it needs,
+// and given to node add by its address, which that block does not match.
+// Empty when addr is itself named on a Host line, or no block points at it.
+func configAlias(ctx context.Context, addr string) string {
+	aliases := map[string]bool{}
+	var order []string
+	seen := map[string]bool{}
+	collect := func(path, base string) {
+		for _, h := range hostNames(path, base, seen, 0) {
+			if !aliases[h] {
+				aliases[h] = true
+				order = append(order, h)
+			}
+		}
+	}
+	collect(userSSHConfig(), filepath.Join(sshHome(), ".ssh"))
+	if systemSSHConfig != "" {
+		collect(systemSSHConfig, filepath.Dir(systemSSHConfig))
+	}
+	for h := range aliases {
+		if strings.EqualFold(h, addr) {
+			return ""
+		}
+	}
+	if len(order) > 256 {
+		order = order[:256]
+	}
+	// Names are looked up only when no HostName is the address as written,
+	// and within one budget: a config of many hosts must not hold up a
+	// connection by seconds each.
+	var pending []string
+	hostOf := map[string]string{}
+	for _, a := range order {
+		h := sshGet(ctx, a)["hostname"]
+		if strings.EqualFold(h, addr) {
+			return a
+		}
+		if h != "" {
+			hostOf[a] = h
+			pending = append(pending, a)
+		}
+	}
+	if len(pending) == 0 {
+		return ""
+	}
+	lctx, cancel := context.WithTimeout(ctx, 3*time.Second)
+	defer cancel()
+	want := lookup(lctx, addr)
+	if len(want) == 0 {
+		return ""
+	}
+	for _, a := range pending {
+		for ip := range lookup(lctx, hostOf[a]) {
+			if want[ip] {
+				return a
+			}
+		}
+	}
+	return ""
+}
+
+// configUser is the user the ssh config logs in to host as, empty when it
+// names none. ssh -G reports the local account when nothing set one, so
+// that is taken as none: the config saying `User root` to a root caller is
+// the one case read wrongly, and it gets the fallback user instead.
+func configUser(ctx context.Context, host string) string {
+	u := sshGet(ctx, host)["user"]
+	if cur, err := user.Current(); err == nil && u == cur.Username {
+		return ""
+	}
+	return u
+}
+
+// sshGet is ssh's own reading of its configuration for host, keys
+// lowercased; empty when ssh cannot say.
+func sshGet(ctx context.Context, host string) map[string]string {
+	args := append(append([]string{"-G"}, sshExtraArgs...), "--", host)
+	out, err := exec.CommandContext(ctx, "ssh", args...).Output()
+	if err != nil {
+		return nil
+	}
+	m := map[string]string{}
+	sc := bufio.NewScanner(bytes.NewReader(out))
+	for sc.Scan() {
+		k, v, ok := strings.Cut(sc.Text(), " ")
+		if k = strings.ToLower(k); ok {
+			if _, dup := m[k]; !dup {
+				m[k] = v
+			}
+		}
+	}
+	return m
+}
+
+func lookup(ctx context.Context, host string) map[string]bool {
+	if ip := net.ParseIP(host); ip != nil {
+		return map[string]bool{ip.String(): true}
+	}
+	addrs, err := net.DefaultResolver.LookupHost(ctx, host)
+	if err != nil {
+		return nil
+	}
+	m := map[string]bool{}
+	for _, a := range addrs {
+		if ip := net.ParseIP(a); ip != nil {
+			m[ip.String()] = true
+		}
+	}
+	return m
+}
+
+// hostNames are the concrete names on the Host lines of an ssh config file
+// and the files it includes — no patterns, no negations. base is where a
+// relative Include is looked for, as ssh does: ~/.ssh for the user's file,
+// /etc/ssh for the system's.
+func hostNames(path, base string, seen map[string]bool, depth int) []string {
+	if depth > 16 || seen[path] {
+		return nil
+	}
+	seen[path] = true
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return nil
+	}
+	var names []string
+	for _, line := range strings.Split(string(data), "\n") {
+		key, args := configLine(line)
+		switch key {
+		case "host":
+			for _, a := range args {
+				if a != "" && !strings.ContainsAny(a, "*?!") {
+					names = append(names, a)
+				}
+			}
+		case "include":
+			for _, a := range args {
+				if strings.HasPrefix(a, "~/") {
+					a = filepath.Join(sshHome(), a[2:])
+				} else if !filepath.IsAbs(a) {
+					a = filepath.Join(base, a)
+				}
+				matches, _ := filepath.Glob(a)
+				for _, m := range matches {
+					names = append(names, hostNames(m, base, seen, depth+1)...)
+				}
+			}
+		}
+	}
+	return names
+}
+
+// configLine splits one line of ssh config into its keyword, lowercased,
+// and arguments: "Host a b", "Host=a b" and quoted arguments alike.
+func configLine(line string) (string, []string) {
+	line = strings.TrimSpace(line)
+	if line == "" || strings.HasPrefix(line, "#") {
+		return "", nil
+	}
+	i := strings.IndexAny(line, " \t=")
+	if i < 0 {
+		return strings.ToLower(line), nil
+	}
+	key := strings.ToLower(line[:i])
+	rest := strings.TrimLeft(line[i:], " \t")
+	rest = strings.TrimPrefix(rest, "=")
+	var args []string
+	for rest = strings.TrimSpace(rest); rest != ""; rest = strings.TrimSpace(rest) {
+		if rest[0] == '"' {
+			end := strings.IndexByte(rest[1:], '"')
+			if end < 0 {
+				args = append(args, rest[1:])
+				break
+			}
+			args = append(args, rest[1:1+end])
+			rest = rest[2+end:]
+			continue
+		}
+		j := strings.IndexAny(rest, " \t")
+		if j < 0 {
+			args = append(args, rest)
+			break
+		}
+		args = append(args, rest[:j])
+		rest = rest[j:]
+	}
+	return key, args
+}

internal/onboard/sshconfig_test.go

@@ -0,0 +1,87 @@
+package onboard
+
+import (
+	"context"
+	"os"
+	"os/exec"
+	"os/user"
+	"path/filepath"
+	"testing"
+)
+
+// withSSHConfig points configAlias and ssh -G at one file, and nothing else.
+func withSSHConfig(t *testing.T, config string, extra map[string]string) {
+	t.Helper()
+	if _, err := exec.LookPath("ssh"); err != nil {
+		t.Skip("no ssh")
+	}
+	dir := t.TempDir()
+	path := filepath.Join(dir, "config")
+	if err := os.WriteFile(path, []byte(config), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	for name, body := range extra {
+		if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o600); err != nil {
+			t.Fatal(err)
+		}
+	}
+	oldUser, oldSys, oldArgs := userSSHConfig, systemSSHConfig, sshExtraArgs
+	userSSHConfig = func() string { return path }
+	systemSSHConfig = ""
+	sshExtraArgs = []string{"-F", path}
+	t.Cleanup(func() { userSSHConfig, systemSSHConfig, sshExtraArgs = oldUser, oldSys, oldArgs })
+}
+
+// A machine given by address is reached through the Host block whose
+// HostName it is, with that block's user — not the sudo caller's.
+func TestSSHTargetUsesConfigAlias(t *testing.T) {
+	withSSHConfig(t, `
+Host *.example
+    User nobody-wild
+Host gpu01 gpu01-alt
+    HostName 10.9.8.7
+    User ops
+Host gpu02
+    HostName=10.9.8.8
+`, nil)
+	ctx := context.Background()
+	cases := []struct {
+		s    *SSH
+		want string
+	}{
+		// The config's User applies by ssh itself: no fallback is added.
+		{&SSH{Addr: "10.9.8.7", FallbackUser: "ilker"}, "gpu01"},
+		{&SSH{Addr: "10.9.8.7", User: "root", FallbackUser: "ilker"}, "root@gpu01"},
+		{&SSH{Addr: "10.9.8.8", FallbackUser: "ilker"}, "ilker@gpu02"},
+		{&SSH{Addr: "gpu01", FallbackUser: "ilker"}, "gpu01"},
+		{&SSH{Addr: "10.9.8.9", FallbackUser: "ilker"}, "ilker@10.9.8.9"},
+		{&SSH{Addr: "10.9.8.9"}, "10.9.8.9"},
+	}
+	for _, c := range cases {
+		if got := c.s.Target(ctx); got != c.want {
+			t.Errorf("%+v: target %q, want %q", c.s.Addr, got, c.want)
+		}
+	}
+}
+
+// A block in an included file counts, as it does for ssh.
+func TestSSHTargetFollowsInclude(t *testing.T) {
+	withSSHConfig(t, "Include nodes.conf\n", map[string]string{
+		"nodes.conf": "Host \"gpu03\"\n  HostName 10.9.8.10\n",
+	})
+	// ssh resolves a relative Include against ~/.ssh, not -F's directory;
+	// the scan does the same, so point base at the test's directory.
+	names := hostNames(userSSHConfig(), filepath.Dir(userSSHConfig()), map[string]bool{}, 0)
+	if len(names) != 1 || names[0] != "gpu03" {
+		t.Fatalf("names %q", names)
+	}
+}
+
+func TestConfigUserIgnoresLocalDefault(t *testing.T) {
+	withSSHConfig(t, "Host x\n  HostName 10.9.8.11\n", nil)
+	if cur, err := user.Current(); err == nil {
+		if u := configUser(context.Background(), "x"); u != "" {
+			t.Errorf("config user %q for a block naming none (local %q)", u, cur.Username)
+		}
+	}
+}

internal/router/handlers.go

@@ -253,6 +253,9 @@ func (r *Router) Handler() http.Handler {
 		pages.SetPersonalDefaults(r.personalDefaults)
 		if r.store != nil {
 			pages.SetAudit(r.store)
+			if r.slurmSettings != nil {
+				pages.SetSlurm(r.store, r.slurmSettings)
+			}
 			// A forgotten password: a link mailed when the cluster's
 			// settings say how, otherwise asked of an operator.
 			pages.SetPasswordReset(r.store, r.mail, r.mail.BaseURL)