A node being lent says when a replica with nowhere to go is stopped
CI / test (push) Successful in 9m37s

Tried on the PoC: lending cuda01, the one GPU node, its model had nowhere
else to go and waited out the drain's five-minute grace, while ctl slurm
status said only "1 replica(s) still to move off". It now says the time
that replica is stopped at. Also recorded: Slurm on a GPU node, tried on
cuda01 (gres, ConstrainDevices, accounting of gres/gpu).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-14 20:45:57 +01:00
1 parent 4b4306bb6a
commit c3a75dcf3f
4 files changed
+56 -5

No files matched your search

+7
View File
@@ -2081,6 +2081,13 @@ lent, partition`); the leader tells each named node's agent (root) what to be, t
VMs: ayse without a job refused ("no active jobs on this node"), with one in
job_N/step_extern (nproc 1), cut off when it ended; mehmet refused; admin's session unchanged.
Left: sites whose users come from AD (sssd) are not held yet; a group of theirs would be the list.
- A GPU node, tried 2026-11-14 on the PoC (cuda01, Quadro RTX 6000, Slurm 23.11.4): gres.conf
`Type=rtx6000 File=/dev/nvidia0`, slurm.conf `Gres=gpu:rtx6000:1`. `srun --gres=gpu:1`:
CUDA_VISIBLE_DEVICES=0, the one card; torch found it and ran 14.1 TFLOPS (fp32 matmul). Without
--gres: "No devices found", torch none (ConstrainDevices holds). `--gres=gpu:2`: refused. sacct has
`gres/gpu=1` in AllocTRES. Found: lending a node whose GPU model has nowhere else to go waits out
the drain grace (5 min) before the replica stops, and `ctl slurm status` says only "1 replica(s)
still to move off"; it should say it waits for a replacement that cannot be placed, and until when.
- P6, Slurm's jobs in llmgw's accounting (written 2026-11-12, not started). The tenancy tree to Slurm:
each project an account (`sacctmgr add account`, parent its team and organization), each member a user
of it, kept by the leader from the store; `sbatch --account=PROJECT`. The records back: the leader reads