diff --git a/BACKLOG.md b/BACKLOG.md index 2bfc274..0567330 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -2037,7 +2037,16 @@ lent, partition`); the leader tells each named node's agent (root) what to be, t from the daemon); the packaged unit is Type=notify (slurmd needs --systemd); slurmd's own "slurmstepd infinity" was counted as a job; without accounting storage 23.11's main scheduler marks batch jobs InvalidAccount and backfill starts them, one a pass (bf_interval=10: 6-10 s each). So many small jobs - want slurmdbd. Not tried yet: a GPU node (gres), a leader failover while lent. + want slurmdbd. Not tried yet: a GPU node (gres). +- Failover while lent, tried 2026-11-11 (v0.16.3-23): the first try killed the job (NODE_FAIL). A new + leader had heard from no node yet and sent the controller a slurm.conf without them; slurmctld, + restarted on it, dropped node05 and its job. Fixed (141a1f4, 0774e90): slurm.conf's nodes and the + controller's address are kept in cluster_state (`slurm-nodes`) and a new leader starts from them; with + nothing kept (an upgraded cluster) or the store unread, the controller's files wait up to 2 min for the + nodes; a node is told nothing until its controller's address is known (no part stops its slurmd). Tried + again: a 5-minute job on lent node05, leader 140 stopped, 150 took over; slurmctld not restarted, the job + ran to its end; reclaimed during it, node05 was llmgw's within seconds of its end. Also 3387f97: srun in + SrunPortRange=60001-63000, and Slurm's ports in llmgw's firewall for the cluster's networks. - Built 2026-11-11: P1 (`internal/slurm`, `slurm.*` settings), P2 (`internal/agent/slurmnode`, proto Slurm/SlurmStatus), P3 (`internal/control/slurmsvc`, wired in the leader; `GET /api/v1/slurm`), P4's commands (`ctl slurm status|setup|lend|reclaim|off`). Decisions on the