Slurm's shared files and users: one path everywhere, accounts as Linux users
CI / test (push) Failing after 7m1s

slurm.shared mounts the site's NFS export, has an llmgw node export the path
over NFS to the Slurm machines alone (2049 in the firewall), or looks for a
file system the site mounts. slurm.users "llmgw" makes each active account a
Linux user on every Slurm machine: login from its e-mail, a UID from 200000
kept in slurm_users and never reused, its home on the shared path, its SSH
keys from the portal's Profile (or ctl slurm users keys) read by sshd from
/etc/llmgw/ssh-keys. server init --slurm chooses /shared from the first
machine and llmgw's users; --slurm-shared and --slurm-users change that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-12 21:34:34 +01:00
1 parent b9d20e7bba
commit d8ce573bdb
45 files changed
+2505 -79

No files matched your search

+152
View File
@@ -5109,6 +5109,89 @@
}
}
},
"/api/v1/slurm/users": {
"get": {
"operationId": "getSlurmUsers",
"summary": "The Linux users made from llmgw's accounts for the Slurm cluster",
"description": "Every active account, with its login, UID and SSH keys; an account is given its login and UID the first time it is listed. The machines have them when slurm.users is \"llmgw\".",
"tags": [
"slurm"
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/slurmsvc.User"
}
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/slurm/users/{user}/ssh-keys": {
"put": {
"operationId": "putSlurmUsersUserSshKeys",
"summary": "Replace an account's SSH public keys",
"description": "{user} is an ID or e-mail. keys is OpenSSH public keys, one a line, without options; empty takes them all away. The machines have them within a minute.",
"tags": [
"slurm"
],
"parameters": [
{
"name": "user",
"in": "path",
"required": true,
"description": "The person's ID or email.",
"schema": {
"type": "string"
}
},
{
"$ref": "#/components/parameters/Actor"
}
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/SlurmKeysRequest"
}
}
}
},
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/slurmsvc.User"
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/summary": {
"get": {
"operationId": "getSummary",
@@ -9401,6 +9484,15 @@
}
}
},
"SlurmKeysRequest": {
"type": "object",
"description": "SlurmKeysRequest is a person's SSH public keys, one a line, as authorized_keys has them; empty takes them all away.",
"properties": {
"keys": {
"type": "string"
}
}
},
"StoreMember": {
"type": "object",
"description": "StoreMember is one member of a replicated store, as its Patroni reports it.",
@@ -10785,6 +10877,24 @@
}
}
},
"config.SharedStorage": {
"type": "object",
"description": "SharedStorage is where the Slurm cluster's shared files come from: the site's NFS export (NFS), an NFS export llmgw keeps on one of its nodes (Server), or, with neither, a file system the site mounts itself (Lustre, BeeGFS, GPFS), which llmgw only looks for.",
"properties": {
"Path": {
"type": "string",
"description": "Path is where every machine sees it, as /shared."
},
"NFS": {
"type": "string",
"description": "NFS is the site's export, host:/path, mounted at Path on every machine of the Slurm cluster."
},
"Server": {
"type": "string",
"description": "Server is the llmgw node, the controller or one of the nodes, that exports Path over NFS to the others: for a site with no storage of its own. One machine: not highly available, and no faster than its disk and network."
}
}
},
"control.Activity": {
"type": "object",
"description": "Activity is what a standby's router reports back: when it last sent each replica a request, and the on-demand models it was asked for. The leader evicts and loads by these, and without them a replica busy behind a standby looks idle to the one machine that decides.",
@@ -12249,6 +12359,14 @@
"slurmdbd_running": {
"type": "boolean"
},
"shared": {
"type": "string",
"description": "Shared is the shared path here: \"mounted\", \"exported\", \"missing\"."
},
"users": {
"type": "integer",
"description": "Users is how many of llmgw's users the machine has."
},
"state": {
"type": "string"
},
@@ -12301,6 +12419,40 @@
"type": "string"
},
"description": "Waiting are nodes left out of slurm.conf until they report their hardware."
},
"shared": {
"$ref": "#/components/schemas/config.SharedStorage",
"description": "Shared is where the shared files are and come from; Users who keeps the users, and how many llmgw has."
},
"users": {
"type": "string"
},
"user_count": {
"type": "integer"
}
}
},
"slurmsvc.User": {
"type": "object",
"description": "User is a Linux user made from an llmgw account.",
"properties": {
"login": {
"type": "string"
},
"uid": {
"type": "integer"
},
"name": {
"type": "string"
},
"email": {
"type": "string"
},
"ssh_keys": {
"type": "array",
"items": {
"type": "string"
}
}
}
},
+4 -1
View File
@@ -215,7 +215,10 @@ replicas off and then lets Slurm run jobs there; **Take back from Slurm…**
starts no new job there and gives it back to llmgw when its jobs have ended.
Both change `slurm.lent` in the cluster's settings, as `llmgw ctl slurm lend`
and `reclaim` do. Turning Slurm on and choosing its nodes is `llmgw ctl slurm
setup`, or the Slurm part of the first install.
setup`, or the Slurm part of the first install. Above the nodes it says where
the shared files come from and how many users llmgw made; the SHARED column
is each node's (mounted, exported, missing). `llmgw ctl slurm shared` and
`llmgw ctl slurm users` change them.
**Logs** shows a node's agent log, newest at the bottom, errors and warnings
coloured.
+20 -1
View File
@@ -195,6 +195,25 @@ On a cluster installed without it, `llmgw ctl slurm setup --controller NODE
networks only, as llmgw's own do; a firewall of the site's own between the
nodes must let them through.
**Slurm's shared files and users.** Jobs need the same files on every
machine: homes, datasets, output, checkpoints. With `--slurm` this machine
exports `/shared` over NFS to every Slurm machine and they mount it
(`slurm.shared`); NFS's port 2049 answers the cluster's networks only, the
export the Slurm machines alone. That is one machine, not highly available
and no faster than its disk and network: enough for a few nodes. A site with
storage of its own uses it instead — `--slurm-shared HOST:/EXPORT` at
install, or `llmgw ctl slurm shared --path /shared --nfs HOST:/EXPORT`
later; Lustre, BeeGFS or GPFS the site mounts itself, and `llmgw ctl slurm
shared --path /lustre/hpc` only has llmgw look that it is there. Every
active llmgw account is a Linux user on every Slurm machine
(`slurm.users: llmgw`): its login from its e-mail address, a UID from
200000 up that is never given to anyone else, its home under
`/shared/home`. People add their SSH public keys in the portal's
**Profile** and log in to this machine to run `sbatch`; an account suspended
or deleted loses its user, its files stay. A site whose users come from its
directory (sssd against Active Directory or LDAP) passes `--slurm-users
site`, or later `llmgw ctl slurm users off`.
**Secure by default.** What init writes is the hardened configuration, not a
demonstration one:
@@ -3167,7 +3186,7 @@ loaded at boot by `llmgw-firewall.service`):
| Machine | Ports, for the cluster only |
| --- | --- |
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
| node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6819 (slurmctld, slurmd, slurmdbd) and 60001–63000 (srun's `SrunPortRange`) |
| node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6819 (slurmctld, slurmd, slurmdbd) and 60001–63000 (srun's `SrunPortRange`); exporting Slurm's shared files, 2049 (NFS) |
| store member | 5432, 8008, 2379, 2380, 9100 |
| mirror | 8081, 8082, 9100 |
+50
View File
@@ -401,6 +401,9 @@ rather than repeated.
| `action` | `string` | [Service](#service) |
| `timeout` | `time.Duration` | [Service](#service) |
| `stopped` | `bool` | [ServingSettings](#servingsettings) |
| `path` | `string` | [SharedStorage](#sharedstorage) |
| `nfs` | `string` | [SharedStorage](#sharedstorage) |
| `server` | `string` | [SharedStorage](#sharedstorage) |
| `bin` | `string` | [SimConfig](#simconfig) |
| `enabled` | `bool` | [SlurmSettings](#slurmsettings) |
| `cluster_name` | `string` | [SlurmSettings](#slurmsettings) |
@@ -409,6 +412,8 @@ rather than repeated.
| `all_nodes` | `bool` | [SlurmSettings](#slurmsettings) |
| `lent` | `[]string` | [SlurmSettings](#slurmsettings) |
| `partition` | `string` | [SlurmSettings](#slurmsettings) |
| `shared` | `SharedStorage` | [SlurmSettings](#slurmsettings) |
| `users` | `string` | [SlurmSettings](#slurmsettings) |
| `dir` | `string` | [StrataConfig](#strataconfig) |
| `python` | `string` | [StrataConfig](#strataconfig) |
| `engine` | `string` | [StrataConfig](#strataconfig) |
@@ -3488,6 +3493,30 @@ running and every model's definition as it is: llmgw stop sets it,
llmgw start clears it, and the models come back as their catalog
entries ask. In force on every control plane within seconds.
## SharedStorage
### `path`
`string`
Path is where every machine sees it, as /shared.
### `nfs`
`string`
NFS is the site's export, host:/path, mounted at Path on every
machine of the Slurm cluster.
### `server`
`string`
Server is the llmgw node, the controller or one of the nodes, that
exports Path over NFS to the others: for a site with no storage of
its own. One machine: not highly available, and no faster than its
disk and network.
## SimConfig
### `bin`
@@ -3548,6 +3577,27 @@ given back to llmgw when its last job ends.
Partition is the one partition every node is in. Empty is "gpu".
### `shared`
`SharedStorage`
Shared is the file system every machine of the Slurm cluster sees at
the same path: home directories, datasets, the jobs' output and
checkpoints. Empty path: none, and a batch job's output stays on the
node it ran on.
Takes a [SharedStorage](#sharedstorage) block.
### `users`
`string`
Users is who keeps the Linux users that run jobs, the same on every
machine of the Slurm cluster: "llmgw" makes one for each active llmgw
account (its login from the e-mail address, a UID of llmgw's from
200000 up, its SSH keys from the portal); empty leaves users to the
site (a directory through sssd, or by hand).
## StrataConfig
### `dir`