Slurm's jobs in containers: pyxis and enroot, installed by the agents
CI / test (push) Failing after 6m53s

slurm.containers (ctl slurm containers on) has every Slurm machine's agent
install enroot from NVIDIA's release and build pyxis against its own
Slurm, both pinned and checked against their digests, and on GPU machines
nvidia-container-cli from NVIDIA's repository. enroot keeps each user's
containers under /run/enroot and /var/lib/enroot, needing no login
session, with the shared path in every container. The leader has Slurm
load pyxis only once every machine reports it: plugstack.conf's
"required" fails every job on a machine without the plugin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-14 21:31:20 +01:00
1 parent c3a75dcf3f
commit 57f0fd2ed0
18 files changed
+737 -9

No files matched your search

+12
View File
@@ -2088,6 +2088,18 @@ lent, partition`); the leader tells each named node's agent (root) what to be, t
`gres/gpu=1` in AllocTRES. Found: lending a node whose GPU model has nowhere else to go waits out
the drain grace (5 min) before the replica stops, and `ctl slurm status` says only "1 replica(s)
still to move off"; it should say it waits for a replacement that cannot be placed, and until when.
Done (c3a75dc): it says the time.
- Jobs in containers, done 2026-11-14: `slurm.containers` (`ctl slurm containers on`). Each agent
installs enroot 4.2.1 (NVIDIA's .deb/.rpm, sha256 pinned) and builds pyxis 0.24.0 (source tarball,
sha256 pinned) against its own Slurm (gcc, make, libslurm-dev); on a machine with nvidia-smi,
nvidia-container-cli from NVIDIA's repository, its key's digest pinned. enroot.conf keeps each
user's runtime, cache and data under /run/enroot and /var/lib/enroot (no login session needed), the
shared path bind-mounted in every container. The leader writes plugstack.conf ("required") only
once every machine reports both, since a machine without the plugin would fail every job. Tried on
the VMs: on in ~30 s; ayse's srun --container-image=ubuntu:24.04 on n1 ran as uid 200001 with
/shared inside and nproc 1; a two-node sbatch ran a rank on each. Seen once: a node's pull timed out
reaching auth.docker.io and the job failed (network, not pyxis). Not tried: GPUs in a container
(cuda01's agent is an older build; needs the peer protocol), Red Hat.
- P6, Slurm's jobs in llmgw's accounting (written 2026-11-12, not started). The tenancy tree to Slurm:
each project an account (`sacctmgr add account`, parent its team and organization), each member a user
of it, kept by the leader from the store; `sbatch --account=PROJECT`. The records back: the leader reads