Slurm's jobs in containers: pyxis and enroot, installed by the agents
CI / test (push) Failing after 6m53s
CI / test (push) Failing after 6m53s
slurm.containers (ctl slurm containers on) has every Slurm machine's agent install enroot from NVIDIA's release and build pyxis against its own Slurm, both pinned and checked against their digests, and on GPU machines nvidia-container-cli from NVIDIA's repository. enroot keeps each user's containers under /run/enroot and /var/lib/enroot, needing no login session, with the shared path in every container. The leader has Slurm load pyxis only once every machine reports it: plugstack.conf's "required" fails every job on a machine without the plugin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c3a75dcf3f
commit
57f0fd2ed0
18 files changed
+737
-9
No files matched your search
@@ -12372,6 +12372,10 @@
|
||||
"type": "string",
|
||||
"description": "Shared is the shared path here: \"mounted\", \"exported\", \"missing\"."
|
||||
},
|
||||
"containers": {
|
||||
"type": "string",
|
||||
"description": "Containers is what the machine has for jobs in containers, empty while not both of pyxis and enroot."
|
||||
},
|
||||
"users": {
|
||||
"type": "integer",
|
||||
"description": "Users is how many of llmgw's users the machine has."
|
||||
@@ -12438,6 +12442,10 @@
|
||||
},
|
||||
"user_count": {
|
||||
"type": "integer"
|
||||
},
|
||||
"containers": {
|
||||
"type": "string",
|
||||
"description": "Containers is whether jobs may run in containers: \"on\" once every machine has pyxis and enroot, \"installing\" until then, empty when slurm.containers is off."
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
@@ -217,6 +217,23 @@ agents install); root and the machines' own users log in as before. A site whose
|
||||
directory (sssd against Active Directory or LDAP) passes `--slurm-users
|
||||
site`, or later `llmgw ctl slurm users off`.
|
||||
|
||||
Jobs may run in containers: `llmgw ctl slurm containers on` has the agents
|
||||
install enroot and pyxis on every Slurm machine (NVIDIA's releases, at
|
||||
versions llmgw pins and checks; pyxis is built against each machine's own
|
||||
Slurm) and, on GPU machines, nvidia-container-cli from NVIDIA's repository.
|
||||
Once every machine has them Slurm loads pyxis, and `ctl slurm status` says
|
||||
`containers: on`:
|
||||
|
||||
```sh
|
||||
srun --gres=gpu:1 --container-image=nvcr.io#nvidia/pytorch:25.01-py3 python train.py
|
||||
```
|
||||
|
||||
The shared path is in every container where it is on the host, and the job
|
||||
runs as its user, held to its cores and cards. Each node pulls the image
|
||||
from its registry for each job; for a large image used often, import it
|
||||
once (`enroot import -o /shared/images/pytorch.sqsh docker://nvcr.io#...`)
|
||||
and give that file to `--container-image`.
|
||||
|
||||
**Secure by default.** What init writes is the hardened configuration, not a
|
||||
demonstration one:
|
||||
|
||||
|
||||
@@ -414,6 +414,7 @@ rather than repeated.
|
||||
| `partition` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `shared` | `SharedStorage` | [SlurmSettings](#slurmsettings) |
|
||||
| `users` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `containers` | `bool` | [SlurmSettings](#slurmsettings) |
|
||||
| `dir` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `python` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `engine` | `string` | [StrataConfig](#strataconfig) |
|
||||
@@ -3599,6 +3600,15 @@ account (its login from the e-mail address, a UID of llmgw's from
|
||||
and to a node only while a job of theirs runs there (pam_slurm_adopt);
|
||||
empty leaves users to the site (a directory through sssd, or by hand).
|
||||
|
||||
### `containers`
|
||||
|
||||
`bool`
|
||||
|
||||
Containers lets jobs run in containers: `srun --container-image
|
||||
nvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),
|
||||
the GPUs and the shared path inside. The agents install both on
|
||||
every Slurm machine; jobs take it once every machine has them.
|
||||
|
||||
## StrataConfig
|
||||
|
||||
### `dir`
|
||||
|
||||
Reference in new issue
Block a user