A node being lent says when a replica with nowhere to go is stopped
CI / test (push) Successful in 9m37s
CI / test (push) Successful in 9m37s
Tried on the PoC: lending cuda01, the one GPU node, its model had nowhere else to go and waited out the drain's five-minute grace, while ctl slurm status said only "1 replica(s) still to move off". It now says the time that replica is stopped at. Also recorded: Slurm on a GPU node, tried on cuda01 (gres, ConstrainDevices, accounting of gres/gpu). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
4b4306bb6a
commit
c3a75dcf3f
4 files changed
+56
-5
No files matched your search
@@ -2081,6 +2081,13 @@ lent, partition`); the leader tells each named node's agent (root) what to be, t
|
||||
VMs: ayse without a job refused ("no active jobs on this node"), with one in
|
||||
job_N/step_extern (nproc 1), cut off when it ended; mehmet refused; admin's session unchanged.
|
||||
Left: sites whose users come from AD (sssd) are not held yet; a group of theirs would be the list.
|
||||
- A GPU node, tried 2026-11-14 on the PoC (cuda01, Quadro RTX 6000, Slurm 23.11.4): gres.conf
|
||||
`Type=rtx6000 File=/dev/nvidia0`, slurm.conf `Gres=gpu:rtx6000:1`. `srun --gres=gpu:1`:
|
||||
CUDA_VISIBLE_DEVICES=0, the one card; torch found it and ran 14.1 TFLOPS (fp32 matmul). Without
|
||||
--gres: "No devices found", torch none (ConstrainDevices holds). `--gres=gpu:2`: refused. sacct has
|
||||
`gres/gpu=1` in AllocTRES. Found: lending a node whose GPU model has nowhere else to go waits out
|
||||
the drain grace (5 min) before the replica stops, and `ctl slurm status` says only "1 replica(s)
|
||||
still to move off"; it should say it waits for a replacement that cannot be placed, and until when.
|
||||
- P6, Slurm's jobs in llmgw's accounting (written 2026-11-12, not started). The tenancy tree to Slurm:
|
||||
each project an account (`sacctmgr add account`, parent its team and organization), each member a user
|
||||
of it, kept by the leader from the store; `sbatch --account=PROJECT`. The records back: the leader reads
|
||||
|
||||
Reference in new issue
Block a user