diff --git a/BACKLOG.md b/BACKLOG.md index b5dd08d..31b850c 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -2039,9 +2039,10 @@ lent, partition`); the leader tells each named node's agent (root) what to be, t InvalidAccount and backfill starts them, one a pass (bf_interval=10: 6-10 s each). So many small jobs want slurmdbd. — Wrong cause, found 2026-11-11: a job asking no memory got the whole node's (ReqTRES mem=all), so a node ran one job at a time and each next one waited for backfill's pass. Fixed: the - partition's DefMemPerCPU is the leanest node's share; slurmdbd is not needed for this. And InvalidAccount is - real: 23.11 defaults to priority/multifactor, whose main scheduler passes every job by without - accounting storage; PriorityType=priority/basic starts a job on submission. Also found: slurm.conf + partition's DefMemPerCPU is the leanest node's share; slurmdbd is not needed for this. InvalidAccount is real too: + 23.11 dropped accounting_storage/none, and without slurmdbd the main scheduler passes every job by; + backfill starts it within bf_interval (10 s). priority/basic did not change that (fair share needs + slurmdbd anyway). Starting on submission needs slurmdbd, on MySQL/MariaDB. Also found: slurm.conf wrote every node State=DRAIN, so each slurmctld start (on any change of the files) drained a lent node until its agent resumed it, up to a minute; a lent node is now written without it. Not tried yet: a GPU node (gres). - Failover while lent, tried 2026-11-11 (v0.16.3-23): the first try killed the job (NODE_FAIL). A new diff --git a/internal/slurm/slurm.go b/internal/slurm/slurm.go index b13f578..411d00f 100644 --- a/internal/slurm/slurm.go +++ b/internal/slurm/slurm.go @@ -218,9 +218,10 @@ JobAcctGatherType=jobacct_gather/none SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory SchedulerType=sched/backfill -# Without accounting storage, 23.11's default (multifactor) has the main -# scheduler pass every job by as InvalidAccount, for backfill to start a -# pass later; basic is first come, first served, started on submission. +# First come, first served: multifactor's fair share needs accounting +# storage (slurmdbd), which there is none of yet. Without it, 23.11's main +# scheduler passes every job by as InvalidAccount, and backfill starts it +# on its next pass (bf_interval above). PriorityType=priority/basic GresTypes=gpu MpiDefault=none