diff --git a/BACKLOG.md b/BACKLOG.md index b74c9e1..0755f29 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -2028,7 +2028,17 @@ lent, partition`); the leader tells each named node's agent (root) what to be, t slurmd/slurmctld, status); P3 the leader side (conf from reports, lend/reclaim); P4 `ctl slurm` (status, lend, reclaim) and a TUI screen; P5 `ctl train import` of a Slurm job's output into the mirror. Later: a backup slurmctld on a second node, accounting (slurmdbd on the Patroni PG), users from LDAP/AD. -- Built 2026-11-11, not tried on a machine yet: P1 (`internal/slurm`, `slurm.*` settings), P2 +- Tried on the PoC 2026-11-11 (dev build v0.16.3-17-g849760c-slurm on 140, 150, node05, node06; Ubuntu + 24.04, Slurm 23.11.4 from the archive): `ctl slurm setup --controller llmgw-node06 --nodes llmgw-node05` + installed munge, slurmctld and slurmd, node05 entered drained with its slurmd -C hardware (8 CPUs, + 31132 MiB after the 3%). `lend`: its 2 replicas moved off, then IDLE in Slurm; srun and sbatch ran there. + `reclaim` with a 90 s job running: MIXED+DRAIN, llmgw kept off, ready again within a minute of the job + ending. Found and fixed on the way: a client command needs its files even for --version (version now + from the daemon); the packaged unit is Type=notify (slurmd needs --systemd); slurmd's own "slurmstepd + infinity" was counted as a job; without accounting storage 23.11's main scheduler marks batch jobs + InvalidAccount and backfill starts them, one a pass (bf_interval=10: 6-10 s each). So many small jobs + want slurmdbd. Not tried yet: a GPU node (gres), a leader failover while lent. +- Built 2026-11-11: P1 (`internal/slurm`, `slurm.*` settings), P2 (`internal/agent/slurmnode`, proto Slurm/SlurmStatus), P3 (`internal/control/slurmsvc`, wired in the leader; `GET /api/v1/slurm`), P4's commands (`ctl slurm status|setup|lend|reclaim|off`). Decisions on the way: a node not heard from or with Slurm jobs is cordoned, not drained (nothing of llmgw's to move, and a