The InvalidAccount seen on the PoC was not the reconfigure: 23.11.4 without
accounting storage has its main scheduler mark a batch job so, and backfill
starts it on its next pass (job 4 waited 15 s). bf_interval=10 keeps that
short; the extra restart is taken out again. MailProg=/bin/true quiets the
warning on every start.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found on the PoC: Slurm 23.11.4 without accounting storage left batch jobs
pending InvalidAccount after scontrol reconfigure until slurmctld was
restarted. And jobacct_gather with nowhere to keep it logs an error per job.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found on the PoC (Ubuntu 24.04, Slurm 23.11.4): the packaged unit is
Type=notify, so slurmd needs --systemd or systemd times it out; and slurmd
keeps a 'slurmstepd infinity' for its cgroup v2 scope, which was counted as
a job and kept the node from llmgw.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found on the PoC: scontrol looks for its files even for --version, and a
configless node has none before slurmd has fetched them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
internal/control/slurmsvc tells each node what it is, writes slurm.conf
and gres.conf from the nodes' reports and the inventory, and moves each node
of slurm.nodes between llmgw and Slurm: lent, drained of replicas first, then
resumed in Slurm; taken back, drained in Slurm and cordoned in llmgw until
its last job step ends. Kept every tick from the settings and the agents'
reports, so a new leader carries on. A broken Slurm never keeps a node from
llmgw. GET /api/v1/slurm; ctl slurm status, setup, lend, reclaim, off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Slurm message says whether the node runs slurmctld on the leader's files,
slurmd configless, both or neither, and whether it is lent. The agent
installs munge and Slurm from the machine's sources (22.05 or newer, for
cgroup v2), writes the munge key, starts slurmd through a drop-in with
--conf-server, resumes a lent node that llmgw drained and drains one that is
not, never touching a drain a Slurm administrator made, and reports its
version, slurmd -C, state and jobs in SlurmStatus. Told it has no part, it
stops what it started. New nodes enter slurm.conf drained.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>