Slurm's jobs in containers: pyxis and enroot, installed by the agents

slurm.containers (ctl slurm containers on) has every Slurm machine's agent
install enroot from NVIDIA's release and build pyxis against its own
Slurm, both pinned and checked against their digests, and on GPU machines
nvidia-container-cli from NVIDIA's repository. enroot keeps each user's
containers under /run/enroot and /var/lib/enroot, needing no login
session, with the shared path in every container. The leader has Slurm
load pyxis only once every machine reports it: plugstack.conf's
"required" fails every job on a machine without the plugin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/admin-api.openapi.json

@@ -12372,6 +12372,10 @@
             "type": "string",
             "description": "Shared is the shared path here: \"mounted\", \"exported\", \"missing\"."
           },
+          "containers": {
+            "type": "string",
+            "description": "Containers is what the machine has for jobs in containers, empty while not both of pyxis and enroot."
+          },
           "users": {
             "type": "integer",
             "description": "Users is how many of llmgw's users the machine has."
@@ -12438,6 +12442,10 @@
           },
           "user_count": {
             "type": "integer"
+          },
+          "containers": {
+            "type": "string",
+            "description": "Containers is whether jobs may run in containers: \"on\" once every machine has pyxis and enroot, \"installing\" until then, empty when slurm.containers is off."
           }
         }
       },