Slurm's jobs in containers: pyxis and enroot, installed by the agents
CI / test (push) Failing after 6m53s

slurm.containers (ctl slurm containers on) has every Slurm machine's agent
install enroot from NVIDIA's release and build pyxis against its own
Slurm, both pinned and checked against their digests, and on GPU machines
nvidia-container-cli from NVIDIA's repository. enroot keeps each user's
containers under /run/enroot and /var/lib/enroot, needing no login
session, with the shared path in every container. The leader has Slurm
load pyxis only once every machine reports it: plugstack.conf's
"required" fails every job on a machine without the plugin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-14 21:31:20 +01:00
1 parent c3a75dcf3f
commit 57f0fd2ed0
18 files changed
+737 -9

No files matched your search

+8
View File
@@ -12372,6 +12372,10 @@
"type": "string",
"description": "Shared is the shared path here: \"mounted\", \"exported\", \"missing\"."
},
"containers": {
"type": "string",
"description": "Containers is what the machine has for jobs in containers, empty while not both of pyxis and enroot."
},
"users": {
"type": "integer",
"description": "Users is how many of llmgw's users the machine has."
@@ -12438,6 +12442,10 @@
},
"user_count": {
"type": "integer"
},
"containers": {
"type": "string",
"description": "Containers is whether jobs may run in containers: \"on\" once every machine has pyxis and enroot, \"installing\" until then, empty when slurm.containers is off."
}
}
},
+17
View File
@@ -217,6 +217,23 @@ agents install); root and the machines' own users log in as before. A site whose
directory (sssd against Active Directory or LDAP) passes `--slurm-users
site`, or later `llmgw ctl slurm users off`.
Jobs may run in containers: `llmgw ctl slurm containers on` has the agents
install enroot and pyxis on every Slurm machine (NVIDIA's releases, at
versions llmgw pins and checks; pyxis is built against each machine's own
Slurm) and, on GPU machines, nvidia-container-cli from NVIDIA's repository.
Once every machine has them Slurm loads pyxis, and `ctl slurm status` says
`containers: on`:
```sh
srun --gres=gpu:1 --container-image=nvcr.io#nvidia/pytorch:25.01-py3 python train.py
```
The shared path is in every container where it is on the host, and the job
runs as its user, held to its cores and cards. Each node pulls the image
from its registry for each job; for a large image used often, import it
once (`enroot import -o /shared/images/pytorch.sqsh docker://nvcr.io#...`)
and give that file to `--container-image`.
**Secure by default.** What init writes is the hardened configuration, not a
demonstration one:
+10
View File
@@ -414,6 +414,7 @@ rather than repeated.
| `partition` | `string` | [SlurmSettings](#slurmsettings) |
| `shared` | `SharedStorage` | [SlurmSettings](#slurmsettings) |
| `users` | `string` | [SlurmSettings](#slurmsettings) |
| `containers` | `bool` | [SlurmSettings](#slurmsettings) |
| `dir` | `string` | [StrataConfig](#strataconfig) |
| `python` | `string` | [StrataConfig](#strataconfig) |
| `engine` | `string` | [StrataConfig](#strataconfig) |
@@ -3599,6 +3600,15 @@ account (its login from the e-mail address, a UID of llmgw's from
and to a node only while a job of theirs runs there (pam_slurm_adopt);
empty leaves users to the site (a directory through sssd, or by hand).
### `containers`
`bool`
Containers lets jobs run in containers: `srun --container-image
nvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),
the GPUs and the shared path inside. The agents install both on
every Slurm machine; jobs take it once every machine has them.
## StrataConfig
### `dir`