Slurm's jobs in containers: pyxis and enroot, installed by the agents
CI / test (push) Failing after 6m53s
CI / test (push) Failing after 6m53s
slurm.containers (ctl slurm containers on) has every Slurm machine's agent install enroot from NVIDIA's release and build pyxis against its own Slurm, both pinned and checked against their digests, and on GPU machines nvidia-container-cli from NVIDIA's repository. enroot keeps each user's containers under /run/enroot and /var/lib/enroot, needing no login session, with the shared path in every container. The leader has Slurm load pyxis only once every machine reports it: plugstack.conf's "required" fails every job on a machine without the plugin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c3a75dcf3f
commit
57f0fd2ed0
18 files changed
+737
-9
No files matched your search
+12
@@ -2088,6 +2088,18 @@ lent, partition`); the leader tells each named node's agent (root) what to be, t
|
||||
`gres/gpu=1` in AllocTRES. Found: lending a node whose GPU model has nowhere else to go waits out
|
||||
the drain grace (5 min) before the replica stops, and `ctl slurm status` says only "1 replica(s)
|
||||
still to move off"; it should say it waits for a replacement that cannot be placed, and until when.
|
||||
Done (c3a75dc): it says the time.
|
||||
- Jobs in containers, done 2026-11-14: `slurm.containers` (`ctl slurm containers on`). Each agent
|
||||
installs enroot 4.2.1 (NVIDIA's .deb/.rpm, sha256 pinned) and builds pyxis 0.24.0 (source tarball,
|
||||
sha256 pinned) against its own Slurm (gcc, make, libslurm-dev); on a machine with nvidia-smi,
|
||||
nvidia-container-cli from NVIDIA's repository, its key's digest pinned. enroot.conf keeps each
|
||||
user's runtime, cache and data under /run/enroot and /var/lib/enroot (no login session needed), the
|
||||
shared path bind-mounted in every container. The leader writes plugstack.conf ("required") only
|
||||
once every machine reports both, since a machine without the plugin would fail every job. Tried on
|
||||
the VMs: on in ~30 s; ayse's srun --container-image=ubuntu:24.04 on n1 ran as uid 200001 with
|
||||
/shared inside and nproc 1; a two-node sbatch ran a rank on each. Seen once: a node's pull timed out
|
||||
reaching auth.docker.io and the job failed (network, not pyxis). Not tried: GPUs in a container
|
||||
(cuda01's agent is an older build; needs the peer protocol), Red Hat.
|
||||
- P6, Slurm's jobs in llmgw's accounting (written 2026-11-12, not started). The tenancy tree to Slurm:
|
||||
each project an account (`sacctmgr add account`, parent its team and organization), each member a user
|
||||
of it, kept by the leader from the store; `sbatch --account=PROJECT`. The records back: the leader reads
|
||||
|
||||
Reference in new issue
Block a user