BACKLOG P: failover while lent tried, and fixed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-11 22:53:26 +01:00
1 parent 0774e901c0
commit 17287ee8b0
1 file changed
+10 -1
+10 -1
View File
@@ -2037,7 +2037,16 @@ lent, partition`); the leader tells each named node's agent (root) what to be, t
from the daemon); the packaged unit is Type=notify (slurmd needs --systemd); slurmd's own "slurmstepd
infinity" was counted as a job; without accounting storage 23.11's main scheduler marks batch jobs
InvalidAccount and backfill starts them, one a pass (bf_interval=10: 6-10 s each). So many small jobs
want slurmdbd. Not tried yet: a GPU node (gres), a leader failover while lent.
want slurmdbd. Not tried yet: a GPU node (gres).
- Failover while lent, tried 2026-11-11 (v0.16.3-23): the first try killed the job (NODE_FAIL). A new
leader had heard from no node yet and sent the controller a slurm.conf without them; slurmctld,
restarted on it, dropped node05 and its job. Fixed (141a1f4, 0774e90): slurm.conf's nodes and the
controller's address are kept in cluster_state (`slurm-nodes`) and a new leader starts from them; with
nothing kept (an upgraded cluster) or the store unread, the controller's files wait up to 2 min for the
nodes; a node is told nothing until its controller's address is known (no part stops its slurmd). Tried
again: a 5-minute job on lent node05, leader 140 stopped, 150 took over; slurmctld not restarted, the job
ran to its end; reclaimed during it, node05 was llmgw's within seconds of its end. Also 3387f97: srun in
SrunPortRange=60001-63000, and Slurm's ports in llmgw's firewall for the cluster's networks.
- Built 2026-11-11: P1 (`internal/slurm`, `slurm.*` settings), P2
(`internal/agent/slurmnode`, proto Slurm/SlurmStatus), P3 (`internal/control/slurmsvc`, wired in the
leader; `GET /api/v1/slurm`), P4's commands (`ctl slurm status|setup|lend|reclaim|off`). Decisions on the