Commit Graph
13 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 fcda4d3f85 Slurm's jobs in llmgw's accounting: projects as accounts, jobs priced
CI / test (push) Failing after 7m24s
With slurm.users llmgw, each project is a Slurm account, its organization's
and team's above it, with the project's members and its team's and
organization's on it; slurm.conf holds a job to its user's accounts. The
controller's agent keeps them with sacctmgr, only what it made, and reports
the jobs that ended in the last day; the leader records each once, its
GPU-hours (core-hours without cards) priced at auth.gpu_hour_cost into its
project's usage as model slurm/PARTITION.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 11:54:01 +01:00
Ilker ManapandClaude Opus 5.5 57f0fd2ed0 Slurm's jobs in containers: pyxis and enroot, installed by the agents
CI / test (push) Failing after 6m53s
slurm.containers (ctl slurm containers on) has every Slurm machine's agent
install enroot from NVIDIA's release and build pyxis against its own
Slurm, both pinned and checked against their digests, and on GPU machines
nvidia-container-cli from NVIDIA's repository. enroot keeps each user's
containers under /run/enroot and /var/lib/enroot, needing no login
session, with the shared path in every container. The leader has Slurm
load pyxis only once every machine reports it: plugstack.conf's
"required" fails every job on a machine without the plugin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:31:20 +01:00
Ilker ManapandClaude Opus 5.5 4b4306bb6a Slurm users reach a node only through a job of theirs
CI / test (push) Successful in 9m41s
llmgw's Slurm users log in to the controller freely and to another Slurm
machine only while a job of theirs runs there: the agent installs
pam_slurm_adopt and puts it in sshd's account stack for the users it made
alone, so root and the machine's own users are never locked out. Their
session is put in the job's extern step (PrologFlags=Contain), and
pam_systemd, which would move it out, is skipped for them by taking the
session file into sshd's stack with a guard in front of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:30:49 +01:00
Ilker ManapandClaude Opus 5.5 d8ce573bdb Slurm's shared files and users: one path everywhere, accounts as Linux users
CI / test (push) Failing after 7m1s
slurm.shared mounts the site's NFS export, has an llmgw node export the path
over NFS to the Slurm machines alone (2049 in the firewall), or looks for a
file system the site mounts. slurm.users "llmgw" makes each active account a
Linux user on every Slurm machine: login from its e-mail, a UID from 200000
kept in slurm_users and never reused, its home on the shared path, its SSH
keys from the portal's Profile (or ctl slurm users keys) read by sshd from
/etc/llmgw/ssh-keys. server init --slurm chooses /shared from the first
machine and llmgw's users; --slurm-shared and --slurm-users change that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:34:34 +01:00
Ilker ManapandClaude Opus 5.5 748f777a56 Slurm's accounting: slurmdbd and a MariaDB on the controller
CI / test (push) Successful in 10m57s
Since 23.11 Slurm runs badly without accounting storage: the main
scheduler passes every job by as InvalidAccount. slurmdbd takes MySQL or
MariaDB only, so the controller's agent installs MariaDB beside it,
answering that machine alone, makes the database and its user (the
password made from the munge key, never on a command line), runs
slurmdbd and adds the cluster, all before slurmctld. A MariaDB that was
on the machine before keeps its settings. slurm.conf accounts the jobs
there, with what each used from its cgroup and the cards, and the
priority is multifactor again. Port 6819 joins Slurm's in the firewall.
ctl slurm status and the console say whether slurmdbd runs; the install
screen, server init --slurm and the guide say what is put where.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:59:08 +01:00
Ilker ManapandClaude Opus 5.5 02774769a7 Slurm: why priority/basic, said rightly
CI / test (push) Failing after 8m3s
priority/basic did not end InvalidAccount: 23.11 dropped
accounting_storage/none, and without slurmdbd the main scheduler passes
jobs by whatever the priority plugin. Basic stays, fair share needing
slurmdbd anyway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:21:24 +01:00
Ilker ManapandClaude Opus 5.5 bab0766ee1 Slurm: jobs start on submission, and a lent node stays open across restarts
CI / test (push) Failing after 8m35s
Without accounting storage, 23.11's default priority/multifactor has the
main scheduler pass every job by as InvalidAccount; backfill started it a
pass later. slurm.conf sets priority/basic.

Every node was written State=DRAIN, so each start of slurmctld on new
files drained a lent node again until its agent resumed it. A lent node
is written without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:18:22 +01:00
Ilker ManapandClaude Opus 5.5 5db920bf2c Slurm: a job that asks for no memory gets a share, not the whole node
CI / test (push) Successful in 10m34s
Tried on the PoC: five one-second jobs on an 8-CPU node started 10 s
apart. Each asked for no memory, so each was given the node's all, ran
alone, and the next waited for backfill's pass. Not accounting, as
thought. The partition's DefMemPerCPU is now the leanest node's memory
per CPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:11:24 +01:00
Ilker ManapandClaude Opus 5.5 3387f97f6a Slurm's ports for the cluster only, and srun in a range of its own
CI / test (push) Successful in 9m40s
slurm.conf sets SrunPortRange=60001-63000, so a site's firewall can let
srun's step connections in. A node given a part in Slurm adds 6817-6818
and that range to llmgw's nftables table, open to the cluster's networks
and closed to everyone else, as the engines' ports are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:10:37 +01:00
Ilker ManapandClaude Opus 5.5 849760c67d Slurm: no second restart; backfill every 10 s, MailProg set
The InvalidAccount seen on the PoC was not the reconfigure: 23.11.4 without
accounting storage has its main scheduler mark a batch job so, and backfill
starts it on its next pass (job 4 waited 15 s). bf_interval=10 keeps that
short; the extra restart is taken out again. MailProg=/bin/true quiets the
warning on every start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:18:35 +01:00
Ilker ManapandClaude Opus 5.5 087552136f Slurm: slurmctld started again after a reconfigure; no accounting gathered
Found on the PoC: Slurm 23.11.4 without accounting storage left batch jobs
pending InvalidAccount after scontrol reconfigure until slurmctld was
restarted. And jobacct_gather with nowhere to keep it logs an error per job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:16:31 +01:00
Ilker ManapandClaude Opus 5.5 35de47444c Slurm beside llmgw, P2: the agent keeps its node's part
A Slurm message says whether the node runs slurmctld on the leader's files,
slurmd configless, both or neither, and whether it is lent. The agent
installs munge and Slurm from the machine's sources (22.05 or newer, for
cgroup v2), writes the munge key, starts slurmd through a drop-in with
--conf-server, resumes a lent node that llmgw drained and drains one that is
not, never touching a drain a Slurm administrator made, and reports its
version, slurmd -C, state and jobs in SlurmStatus. Told it has no part, it
stops what it started. New nodes enter slurm.conf drained.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:25:41 +01:00
Ilker ManapandClaude Opus 5.5 d439d507d7 Slurm beside llmgw, P1: its settings and the files a controller runs on
slurm.{enabled,cluster_name,controller,nodes,lent,partition} in the cluster's
settings, checked; internal/slurm writes slurm.conf (configless), gres.conf
and cgroup.conf from the nodes' slurmd -C lines and GPUs, derives the munge
key from a cluster secret, and reads scontrol and version output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:20:31 +01:00