Since 23.11 Slurm runs badly without accounting storage: the main scheduler passes every job by as InvalidAccount. slurmdbd takes MySQL or MariaDB only, so the controller's agent installs MariaDB beside it, answering that machine alone, makes the database and its user (the password made from the munge key, never on a command line), runs slurmdbd and adds the cluster, all before slurmctld. A MariaDB that was on the machine before keeps its settings. slurm.conf accounts the jobs there, with what each used from its cgroup and the cards, and the priority is multifactor again. Port 6819 joins Slurm's in the firewall. ctl slurm status and the console say whether slurmdbd runs; the install screen, server init --slurm and the guide say what is put where. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
02774769a7
commit
748f777a56
20 files changed
+425
-37
No files matched your search
+10
-3
@@ -177,14 +177,21 @@ init again to serve on it.
|
||||
choices screen, runs a Slurm cluster beside llmgw: slurmctld on this
|
||||
machine's node, slurmd on every node, those that join later too
|
||||
(`slurm.all_nodes`). The agents install munge and Slurm (22.05 or newer)
|
||||
from the machines' own package sources. No node is Slurm's at first: each is
|
||||
from the machines' own package sources, and on this machine slurmdbd, which
|
||||
keeps the jobs' records (`sacct`), in a MariaDB installed beside it that
|
||||
answers this machine alone. Slurm keeps its records in MySQL or MariaDB
|
||||
only, not in the cluster's PostgreSQL; the database's password is made from
|
||||
the cluster's munge key, so nothing new is stored. A MariaDB the machine
|
||||
already had keeps its settings and only gets the database `slurm_acct_db`
|
||||
and its user. Back it up with the machine (`mariadb-dump slurm_acct_db`);
|
||||
records older than a year are purged. No node is Slurm's at first: each is
|
||||
drained in Slurm and serves models until `llmgw ctl slurm lend NODE` gives it
|
||||
to Slurm — its models move off first — and `llmgw ctl slurm reclaim NODE`
|
||||
takes it back once its jobs end. Jobs are submitted on this machine with
|
||||
`sbatch` and `srun`; `llmgw ctl slurm status` shows where each node is. It
|
||||
needs this machine to be a node, and its hostname to be a name Slurm takes.
|
||||
On a cluster installed without it, `llmgw ctl slurm setup --controller NODE
|
||||
--all-nodes` does the same later. Slurm's ports (6817–6818, and srun's 60001–63000) answer the cluster's
|
||||
--all-nodes` does the same later. Slurm's ports (6817–6819, and srun's 60001–63000) answer the cluster's
|
||||
networks only, as llmgw's own do; a firewall of the site's own between the
|
||||
nodes must let them through.
|
||||
|
||||
@@ -3160,7 +3167,7 @@ loaded at boot by `llmgw-firewall.service`):
|
||||
| Machine | Ports, for the cluster only |
|
||||
| --- | --- |
|
||||
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
|
||||
| node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6818 (slurmctld, slurmd) and 60001–63000 (srun's `SrunPortRange`) |
|
||||
| node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6819 (slurmctld, slurmd, slurmdbd) and 60001–63000 (srun's `SrunPortRange`) |
|
||||
| store member | 5432, 8008, 2379, 2380, 9100 |
|
||||
| mirror | 8081, 8082, 9100 |
|
||||
|
||||
|
||||
Reference in new issue
Block a user