Slurm's jobs in containers: pyxis and enroot, installed by the agents
CI / test (push) Failing after 6m53s

slurm.containers (ctl slurm containers on) has every Slurm machine's agent
install enroot from NVIDIA's release and build pyxis against its own
Slurm, both pinned and checked against their digests, and on GPU machines
nvidia-container-cli from NVIDIA's repository. enroot keeps each user's
containers under /run/enroot and /var/lib/enroot, needing no login
session, with the shared path in every container. The leader has Slurm
load pyxis only once every machine reports it: plugstack.conf's
"required" fails every job on a machine without the plugin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-14 21:31:20 +01:00
1 parent c3a75dcf3f
commit 57f0fd2ed0
18 files changed
+737 -9

No files matched your search

+17
View File
@@ -217,6 +217,23 @@ agents install); root and the machines' own users log in as before. A site whose
directory (sssd against Active Directory or LDAP) passes `--slurm-users
site`, or later `llmgw ctl slurm users off`.
Jobs may run in containers: `llmgw ctl slurm containers on` has the agents
install enroot and pyxis on every Slurm machine (NVIDIA's releases, at
versions llmgw pins and checks; pyxis is built against each machine's own
Slurm) and, on GPU machines, nvidia-container-cli from NVIDIA's repository.
Once every machine has them Slurm loads pyxis, and `ctl slurm status` says
`containers: on`:
```sh
srun --gres=gpu:1 --container-image=nvcr.io#nvidia/pytorch:25.01-py3 python train.py
```
The shared path is in every container where it is on the host, and the job
runs as its user, held to its cores and cards. Each node pulls the image
from its registry for each job; for a large image used often, import it
once (`enroot import -o /shared/images/pytorch.sqsh docker://nvcr.io#...`)
and give that file to `--container-image`.
**Secure by default.** What init writes is the hardened configuration, not a
demonstration one: