Since 23.11 Slurm runs badly without accounting storage: the main
scheduler passes every job by as InvalidAccount. slurmdbd takes MySQL or
MariaDB only, so the controller's agent installs MariaDB beside it,
answering that machine alone, makes the database and its user (the
password made from the munge key, never on a command line), runs
slurmdbd and adds the cluster, all before slurmctld. A MariaDB that was
on the machine before keeps its settings. slurm.conf accounts the jobs
there, with what each used from its cgroup and the cards, and the
priority is multifactor again. Port 6819 joins Slurm's in the firewall.
ctl slurm status and the console say whether slurmdbd runs; the install
screen, server init --slurm and the guide say what is put where.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without accounting storage, 23.11's default priority/multifactor has the
main scheduler pass every job by as InvalidAccount; backfill started it a
pass later. slurm.conf sets priority/basic.
Every node was written State=DRAIN, so each start of slurmctld on new
files drained a lent node again until its agent resumed it. A lent node
is written without it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tried on the PoC: five one-second jobs on an 8-CPU node started 10 s
apart. Each asked for no memory, so each was given the node's all, ran
alone, and the next waited for backfill's pass. Not accounting, as
thought. The partition's DefMemPerCPU is now the leanest node's memory
per CPU.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Slurm message says whether the node runs slurmctld on the leader's files,
slurmd configless, both or neither, and whether it is lent. The agent
installs munge and Slurm from the machine's sources (22.05 or newer, for
cgroup v2), writes the munge key, starts slurmd through a drop-in with
--conf-server, resumes a lent node that llmgw drained and drains one that is
not, never touching a drain a Slurm administrator made, and reports its
version, slurmd -C, state and jobs in SlurmStatus. Told it has no part, it
stops what it started. New nodes enter slurm.conf drained.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
slurm.{enabled,cluster_name,controller,nodes,lent,partition} in the cluster's
settings, checked; internal/slurm writes slurm.conf (configless), gres.conf
and cgroup.conf from the nodes' slurmd -C lines and GPUs, derives the munge
key from a cluster secret, and reads scontrol and version output.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>