From 21828fefce0d598e6e51b6466938acfa40401d3f Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Thu, 12 Nov 2026 08:21:40 +0100 Subject: [PATCH] BACKLOG P: slurmdbd done; P6, Slurm's jobs in llmgw's accounting Co-Authored-By: Claude Opus 5.5 --- BACKLOG.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/BACKLOG.md b/BACKLOG.md index 31b850c..35aa6f4 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -2042,7 +2042,23 @@ lent, partition`); the leader tells each named node's agent (root) what to be, t partition's DefMemPerCPU is the leanest node's share; slurmdbd is not needed for this. InvalidAccount is real too: 23.11 dropped accounting_storage/none, and without slurmdbd the main scheduler passes every job by; backfill starts it within bf_interval (10 s). priority/basic did not change that (fair share needs - slurmdbd anyway). Starting on submission needs slurmdbd, on MySQL/MariaDB. Also found: slurm.conf + slurmdbd anyway). Starting on submission needs slurmdbd, on MySQL/MariaDB. +- slurmdbd, done 2026-11-12 (748f777): the controller's agent installs MariaDB (bound to 127.0.0.1; a + MariaDB that was there keeps its settings), makes `slurm_acct_db` and its user (password from the munge + key, through a 0600 file, never argv), runs slurmdbd (6819, in the firewall for the cluster) and adds + the cluster, before slurmctld. slurm.conf: AccountingStorageType=slurmdbd, jobacct_gather/cgroup, + AccountingStorageTRES=gres/gpu, priority/multifactor. Tried on the PoC: five jobs started on + submission, `sacct -a` has them with their TRES. Not tried: a fresh `server init --slurm` (needs VMs); + a Red Hat controller. +- P6, Slurm's jobs in llmgw's accounting (written 2026-11-12, not started). The tenancy tree to Slurm: + each project an account (`sacctmgr add account`, parent its team and organization), each member a user + of it, kept by the leader from the store; `sbatch --account=PROJECT`. The records back: the leader reads + `sacct --json` for jobs ended since the last read (through the controller's agent) into a + `batch_usage` table (job, user, project, start, end, GPU-seconds by class, CPU-seconds, memory), priced + per GPU-hour and core-hour into cost_micros, summed with the requests' usage on the usage pages and in + budgets. Budgets enforced: a project's remaining budget as GPU-minutes in its account's GrpTRESMins, so + Slurm holds its new jobs once spent. Needs the same users on every node (open: LDAP/sssd or + llmgw-made). Also found: slurm.conf wrote every node State=DRAIN, so each slurmctld start (on any change of the files) drained a lent node until its agent resumed it, up to a minute; a lent node is now written without it. Not tried yet: a GPU node (gres). - Failover while lent, tried 2026-11-11 (v0.16.3-23): the first try killed the job (NODE_FAIL). A new