BACKLOG P: slurmdbd done; P6, Slurm's jobs in llmgw's accounting
CI / test (push) Failing after 7m52s
CI / test (push) Failing after 7m52s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
748f777a56
commit
21828fefce
1 file changed
+17
-1
+17
-1
@@ -2042,7 +2042,23 @@ lent, partition`); the leader tells each named node's agent (root) what to be, t
|
||||
partition's DefMemPerCPU is the leanest node's share; slurmdbd is not needed for this. InvalidAccount is real too:
|
||||
23.11 dropped accounting_storage/none, and without slurmdbd the main scheduler passes every job by;
|
||||
backfill starts it within bf_interval (10 s). priority/basic did not change that (fair share needs
|
||||
slurmdbd anyway). Starting on submission needs slurmdbd, on MySQL/MariaDB. Also found: slurm.conf
|
||||
slurmdbd anyway). Starting on submission needs slurmdbd, on MySQL/MariaDB.
|
||||
- slurmdbd, done 2026-11-12 (748f777): the controller's agent installs MariaDB (bound to 127.0.0.1; a
|
||||
MariaDB that was there keeps its settings), makes `slurm_acct_db` and its user (password from the munge
|
||||
key, through a 0600 file, never argv), runs slurmdbd (6819, in the firewall for the cluster) and adds
|
||||
the cluster, before slurmctld. slurm.conf: AccountingStorageType=slurmdbd, jobacct_gather/cgroup,
|
||||
AccountingStorageTRES=gres/gpu, priority/multifactor. Tried on the PoC: five jobs started on
|
||||
submission, `sacct -a` has them with their TRES. Not tried: a fresh `server init --slurm` (needs VMs);
|
||||
a Red Hat controller.
|
||||
- P6, Slurm's jobs in llmgw's accounting (written 2026-11-12, not started). The tenancy tree to Slurm:
|
||||
each project an account (`sacctmgr add account`, parent its team and organization), each member a user
|
||||
of it, kept by the leader from the store; `sbatch --account=PROJECT`. The records back: the leader reads
|
||||
`sacct --json` for jobs ended since the last read (through the controller's agent) into a
|
||||
`batch_usage` table (job, user, project, start, end, GPU-seconds by class, CPU-seconds, memory), priced
|
||||
per GPU-hour and core-hour into cost_micros, summed with the requests' usage on the usage pages and in
|
||||
budgets. Budgets enforced: a project's remaining budget as GPU-minutes in its account's GrpTRESMins, so
|
||||
Slurm holds its new jobs once spent. Needs the same users on every node (open: LDAP/sssd or
|
||||
llmgw-made). Also found: slurm.conf
|
||||
wrote every node State=DRAIN, so each slurmctld start (on any change of the files) drained a lent node
|
||||
until its agent resumed it, up to a minute; a lent node is now written without it. Not tried yet: a GPU node (gres).
|
||||
- Failover while lent, tried 2026-11-11 (v0.16.3-23): the first try killed the job (NODE_FAIL). A new
|
||||
|
||||
Reference in new issue
Block a user