Slurm's shared files and users: one path everywhere, accounts as Linux users
CI / test (push) Failing after 7m1s
CI / test (push) Failing after 7m1s
slurm.shared mounts the site's NFS export, has an llmgw node export the path over NFS to the Slurm machines alone (2049 in the firewall), or looks for a file system the site mounts. slurm.users "llmgw" makes each active account a Linux user on every Slurm machine: login from its e-mail, a UID from 200000 kept in slurm_users and never reused, its home on the shared path, its SSH keys from the portal's Profile (or ctl slurm users keys) read by sshd from /etc/llmgw/ssh-keys. server init --slurm chooses /shared from the first machine and llmgw's users; --slurm-shared and --slurm-users change that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b9d20e7bba
commit
d8ce573bdb
45 files changed
+2505
-79
No files matched your search
@@ -5109,6 +5109,89 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/slurm/users": {
|
||||
"get": {
|
||||
"operationId": "getSlurmUsers",
|
||||
"summary": "The Linux users made from llmgw's accounts for the Slurm cluster",
|
||||
"description": "Every active account, with its login, UID and SSH keys; an account is given its login and UID the first time it is listed. The machines have them when slurm.users is \"llmgw\".",
|
||||
"tags": [
|
||||
"slurm"
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/slurmsvc.User"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/slurm/users/{user}/ssh-keys": {
|
||||
"put": {
|
||||
"operationId": "putSlurmUsersUserSshKeys",
|
||||
"summary": "Replace an account's SSH public keys",
|
||||
"description": "{user} is an ID or e-mail. keys is OpenSSH public keys, one a line, without options; empty takes them all away. The machines have them within a minute.",
|
||||
"tags": [
|
||||
"slurm"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "user",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "The person's ID or email.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/SlurmKeysRequest"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/slurmsvc.User"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/summary": {
|
||||
"get": {
|
||||
"operationId": "getSummary",
|
||||
@@ -9401,6 +9484,15 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"SlurmKeysRequest": {
|
||||
"type": "object",
|
||||
"description": "SlurmKeysRequest is a person's SSH public keys, one a line, as authorized_keys has them; empty takes them all away.",
|
||||
"properties": {
|
||||
"keys": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"StoreMember": {
|
||||
"type": "object",
|
||||
"description": "StoreMember is one member of a replicated store, as its Patroni reports it.",
|
||||
@@ -10785,6 +10877,24 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"config.SharedStorage": {
|
||||
"type": "object",
|
||||
"description": "SharedStorage is where the Slurm cluster's shared files come from: the site's NFS export (NFS), an NFS export llmgw keeps on one of its nodes (Server), or, with neither, a file system the site mounts itself (Lustre, BeeGFS, GPFS), which llmgw only looks for.",
|
||||
"properties": {
|
||||
"Path": {
|
||||
"type": "string",
|
||||
"description": "Path is where every machine sees it, as /shared."
|
||||
},
|
||||
"NFS": {
|
||||
"type": "string",
|
||||
"description": "NFS is the site's export, host:/path, mounted at Path on every machine of the Slurm cluster."
|
||||
},
|
||||
"Server": {
|
||||
"type": "string",
|
||||
"description": "Server is the llmgw node, the controller or one of the nodes, that exports Path over NFS to the others: for a site with no storage of its own. One machine: not highly available, and no faster than its disk and network."
|
||||
}
|
||||
}
|
||||
},
|
||||
"control.Activity": {
|
||||
"type": "object",
|
||||
"description": "Activity is what a standby's router reports back: when it last sent each replica a request, and the on-demand models it was asked for. The leader evicts and loads by these, and without them a replica busy behind a standby looks idle to the one machine that decides.",
|
||||
@@ -12249,6 +12359,14 @@
|
||||
"slurmdbd_running": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"shared": {
|
||||
"type": "string",
|
||||
"description": "Shared is the shared path here: \"mounted\", \"exported\", \"missing\"."
|
||||
},
|
||||
"users": {
|
||||
"type": "integer",
|
||||
"description": "Users is how many of llmgw's users the machine has."
|
||||
},
|
||||
"state": {
|
||||
"type": "string"
|
||||
},
|
||||
@@ -12301,6 +12419,40 @@
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Waiting are nodes left out of slurm.conf until they report their hardware."
|
||||
},
|
||||
"shared": {
|
||||
"$ref": "#/components/schemas/config.SharedStorage",
|
||||
"description": "Shared is where the shared files are and come from; Users who keeps the users, and how many llmgw has."
|
||||
},
|
||||
"users": {
|
||||
"type": "string"
|
||||
},
|
||||
"user_count": {
|
||||
"type": "integer"
|
||||
}
|
||||
}
|
||||
},
|
||||
"slurmsvc.User": {
|
||||
"type": "object",
|
||||
"description": "User is a Linux user made from an llmgw account.",
|
||||
"properties": {
|
||||
"login": {
|
||||
"type": "string"
|
||||
},
|
||||
"uid": {
|
||||
"type": "integer"
|
||||
},
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"email": {
|
||||
"type": "string"
|
||||
},
|
||||
"ssh_keys": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
+4
-1
@@ -215,7 +215,10 @@ replicas off and then lets Slurm run jobs there; **Take back from Slurm…**
|
||||
starts no new job there and gives it back to llmgw when its jobs have ended.
|
||||
Both change `slurm.lent` in the cluster's settings, as `llmgw ctl slurm lend`
|
||||
and `reclaim` do. Turning Slurm on and choosing its nodes is `llmgw ctl slurm
|
||||
setup`, or the Slurm part of the first install.
|
||||
setup`, or the Slurm part of the first install. Above the nodes it says where
|
||||
the shared files come from and how many users llmgw made; the SHARED column
|
||||
is each node's (mounted, exported, missing). `llmgw ctl slurm shared` and
|
||||
`llmgw ctl slurm users` change them.
|
||||
|
||||
**Logs** shows a node's agent log, newest at the bottom, errors and warnings
|
||||
coloured.
|
||||
|
||||
+20
-1
@@ -195,6 +195,25 @@ On a cluster installed without it, `llmgw ctl slurm setup --controller NODE
|
||||
networks only, as llmgw's own do; a firewall of the site's own between the
|
||||
nodes must let them through.
|
||||
|
||||
**Slurm's shared files and users.** Jobs need the same files on every
|
||||
machine: homes, datasets, output, checkpoints. With `--slurm` this machine
|
||||
exports `/shared` over NFS to every Slurm machine and they mount it
|
||||
(`slurm.shared`); NFS's port 2049 answers the cluster's networks only, the
|
||||
export the Slurm machines alone. That is one machine, not highly available
|
||||
and no faster than its disk and network: enough for a few nodes. A site with
|
||||
storage of its own uses it instead — `--slurm-shared HOST:/EXPORT` at
|
||||
install, or `llmgw ctl slurm shared --path /shared --nfs HOST:/EXPORT`
|
||||
later; Lustre, BeeGFS or GPFS the site mounts itself, and `llmgw ctl slurm
|
||||
shared --path /lustre/hpc` only has llmgw look that it is there. Every
|
||||
active llmgw account is a Linux user on every Slurm machine
|
||||
(`slurm.users: llmgw`): its login from its e-mail address, a UID from
|
||||
200000 up that is never given to anyone else, its home under
|
||||
`/shared/home`. People add their SSH public keys in the portal's
|
||||
**Profile** and log in to this machine to run `sbatch`; an account suspended
|
||||
or deleted loses its user, its files stay. A site whose users come from its
|
||||
directory (sssd against Active Directory or LDAP) passes `--slurm-users
|
||||
site`, or later `llmgw ctl slurm users off`.
|
||||
|
||||
**Secure by default.** What init writes is the hardened configuration, not a
|
||||
demonstration one:
|
||||
|
||||
@@ -3167,7 +3186,7 @@ loaded at boot by `llmgw-firewall.service`):
|
||||
| Machine | Ports, for the cluster only |
|
||||
| --- | --- |
|
||||
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
|
||||
| node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6819 (slurmctld, slurmd, slurmdbd) and 60001–63000 (srun's `SrunPortRange`) |
|
||||
| node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6819 (slurmctld, slurmd, slurmdbd) and 60001–63000 (srun's `SrunPortRange`); exporting Slurm's shared files, 2049 (NFS) |
|
||||
| store member | 5432, 8008, 2379, 2380, 9100 |
|
||||
| mirror | 8081, 8082, 9100 |
|
||||
|
||||
|
||||
@@ -401,6 +401,9 @@ rather than repeated.
|
||||
| `action` | `string` | [Service](#service) |
|
||||
| `timeout` | `time.Duration` | [Service](#service) |
|
||||
| `stopped` | `bool` | [ServingSettings](#servingsettings) |
|
||||
| `path` | `string` | [SharedStorage](#sharedstorage) |
|
||||
| `nfs` | `string` | [SharedStorage](#sharedstorage) |
|
||||
| `server` | `string` | [SharedStorage](#sharedstorage) |
|
||||
| `bin` | `string` | [SimConfig](#simconfig) |
|
||||
| `enabled` | `bool` | [SlurmSettings](#slurmsettings) |
|
||||
| `cluster_name` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
@@ -409,6 +412,8 @@ rather than repeated.
|
||||
| `all_nodes` | `bool` | [SlurmSettings](#slurmsettings) |
|
||||
| `lent` | `[]string` | [SlurmSettings](#slurmsettings) |
|
||||
| `partition` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `shared` | `SharedStorage` | [SlurmSettings](#slurmsettings) |
|
||||
| `users` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `dir` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `python` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `engine` | `string` | [StrataConfig](#strataconfig) |
|
||||
@@ -3488,6 +3493,30 @@ running and every model's definition as it is: llmgw stop sets it,
|
||||
llmgw start clears it, and the models come back as their catalog
|
||||
entries ask. In force on every control plane within seconds.
|
||||
|
||||
## SharedStorage
|
||||
|
||||
### `path`
|
||||
|
||||
`string`
|
||||
|
||||
Path is where every machine sees it, as /shared.
|
||||
|
||||
### `nfs`
|
||||
|
||||
`string`
|
||||
|
||||
NFS is the site's export, host:/path, mounted at Path on every
|
||||
machine of the Slurm cluster.
|
||||
|
||||
### `server`
|
||||
|
||||
`string`
|
||||
|
||||
Server is the llmgw node, the controller or one of the nodes, that
|
||||
exports Path over NFS to the others: for a site with no storage of
|
||||
its own. One machine: not highly available, and no faster than its
|
||||
disk and network.
|
||||
|
||||
## SimConfig
|
||||
|
||||
### `bin`
|
||||
@@ -3548,6 +3577,27 @@ given back to llmgw when its last job ends.
|
||||
|
||||
Partition is the one partition every node is in. Empty is "gpu".
|
||||
|
||||
### `shared`
|
||||
|
||||
`SharedStorage`
|
||||
|
||||
Shared is the file system every machine of the Slurm cluster sees at
|
||||
the same path: home directories, datasets, the jobs' output and
|
||||
checkpoints. Empty path: none, and a batch job's output stays on the
|
||||
node it ran on.
|
||||
|
||||
Takes a [SharedStorage](#sharedstorage) block.
|
||||
|
||||
### `users`
|
||||
|
||||
`string`
|
||||
|
||||
Users is who keeps the Linux users that run jobs, the same on every
|
||||
machine of the Slurm cluster: "llmgw" makes one for each active llmgw
|
||||
account (its login from the e-mail address, a UID of llmgw's from
|
||||
200000 up, its SSH keys from the portal); empty leaves users to the
|
||||
site (a directory through sssd, or by hand).
|
||||
|
||||
## StrataConfig
|
||||
|
||||
### `dir`
|
||||
|
||||
Reference in new issue
Block a user