Slurm's shared files and users: one path everywhere, accounts as Linux users

slurm.shared mounts the site's NFS export, has an llmgw node export the path
over NFS to the Slurm machines alone (2049 in the firewall), or looks for a
file system the site mounts. slurm.users "llmgw" makes each active account a
Linux user on every Slurm machine: login from its e-mail, a UID from 200000
kept in slurm_users and never reused, its home on the shared path, its SSH
keys from the portal's Profile (or ctl slurm users keys) read by sshd from
/etc/llmgw/ssh-keys. server init --slurm chooses /shared from the first
machine and llmgw's users; --slurm-shared and --slurm-users change that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/reference.md

@@ -401,6 +401,9 @@ rather than repeated.
 | `action` | `string` | [Service](#service) |
 | `timeout` | `time.Duration` | [Service](#service) |
 | `stopped` | `bool` | [ServingSettings](#servingsettings) |
+| `path` | `string` | [SharedStorage](#sharedstorage) |
+| `nfs` | `string` | [SharedStorage](#sharedstorage) |
+| `server` | `string` | [SharedStorage](#sharedstorage) |
 | `bin` | `string` | [SimConfig](#simconfig) |
 | `enabled` | `bool` | [SlurmSettings](#slurmsettings) |
 | `cluster_name` | `string` | [SlurmSettings](#slurmsettings) |
@@ -409,6 +412,8 @@ rather than repeated.
 | `all_nodes` | `bool` | [SlurmSettings](#slurmsettings) |
 | `lent` | `[]string` | [SlurmSettings](#slurmsettings) |
 | `partition` | `string` | [SlurmSettings](#slurmsettings) |
+| `shared` | `SharedStorage` | [SlurmSettings](#slurmsettings) |
+| `users` | `string` | [SlurmSettings](#slurmsettings) |
 | `dir` | `string` | [StrataConfig](#strataconfig) |
 | `python` | `string` | [StrataConfig](#strataconfig) |
 | `engine` | `string` | [StrataConfig](#strataconfig) |
@@ -3488,6 +3493,30 @@ running and every model's definition as it is: llmgw stop sets it,
 llmgw start clears it, and the models come back as their catalog
 entries ask. In force on every control plane within seconds.
 
+## SharedStorage
+
+### `path`
+
+`string`
+
+Path is where every machine sees it, as /shared.
+
+### `nfs`
+
+`string`
+
+NFS is the site's export, host:/path, mounted at Path on every
+machine of the Slurm cluster.
+
+### `server`
+
+`string`
+
+Server is the llmgw node, the controller or one of the nodes, that
+exports Path over NFS to the others: for a site with no storage of
+its own. One machine: not highly available, and no faster than its
+disk and network.
+
 ## SimConfig
 
 ### `bin`
@@ -3548,6 +3577,27 @@ given back to llmgw when its last job ends.
 
 Partition is the one partition every node is in. Empty is "gpu".
 
+### `shared`
+
+`SharedStorage`
+
+Shared is the file system every machine of the Slurm cluster sees at
+the same path: home directories, datasets, the jobs' output and
+checkpoints. Empty path: none, and a batch job's output stays on the
+node it ran on.
+
+Takes a [SharedStorage](#sharedstorage) block.
+
+### `users`
+
+`string`
+
+Users is who keeps the Linux users that run jobs, the same on every
+machine of the Slurm cluster: "llmgw" makes one for each active llmgw
+account (its login from the e-mail address, a UID of llmgw's from
+200000 up, its SSH keys from the portal); empty leaves users to the
+site (a directory through sssd, or by hand).
+
 ## StrataConfig
 
 ### `dir`