BACKLOG P: tried on the PoC
CI / test (push) Failing after 7m26s

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-11 13:20:32 +01:00
1 parent 849760c67d
commit dcf4e256e6
1 file changed
+11 -1
+11 -1
View File
@@ -2028,7 +2028,17 @@ lent, partition`); the leader tells each named node's agent (root) what to be, t
slurmd/slurmctld, status); P3 the leader side (conf from reports, lend/reclaim); P4 `ctl slurm`
(status, lend, reclaim) and a TUI screen; P5 `ctl train import` of a Slurm job's output into the mirror.
Later: a backup slurmctld on a second node, accounting (slurmdbd on the Patroni PG), users from LDAP/AD.
- Built 2026-11-11, not tried on a machine yet: P1 (`internal/slurm`, `slurm.*` settings), P2
- Tried on the PoC 2026-11-11 (dev build v0.16.3-17-g849760c-slurm on 140, 150, node05, node06; Ubuntu
24.04, Slurm 23.11.4 from the archive): `ctl slurm setup --controller llmgw-node06 --nodes llmgw-node05`
installed munge, slurmctld and slurmd, node05 entered drained with its slurmd -C hardware (8 CPUs,
31132 MiB after the 3%). `lend`: its 2 replicas moved off, then IDLE in Slurm; srun and sbatch ran there.
`reclaim` with a 90 s job running: MIXED+DRAIN, llmgw kept off, ready again within a minute of the job
ending. Found and fixed on the way: a client command needs its files even for --version (version now
from the daemon); the packaged unit is Type=notify (slurmd needs --systemd); slurmd's own "slurmstepd
infinity" was counted as a job; without accounting storage 23.11's main scheduler marks batch jobs
InvalidAccount and backfill starts them, one a pass (bf_interval=10: 6-10 s each). So many small jobs
want slurmdbd. Not tried yet: a GPU node (gres), a leader failover while lent.
- Built 2026-11-11: P1 (`internal/slurm`, `slurm.*` settings), P2
(`internal/agent/slurmnode`, proto Slurm/SlurmStatus), P3 (`internal/control/slurmsvc`, wired in the
leader; `GET /api/v1/slurm`), P4's commands (`ctl slurm status|setup|lend|reclaim|off`). Decisions on the
way: a node not heard from or with Slurm jobs is cordoned, not drained (nothing of llmgw's to move, and a