Slurm in the first install: server init --slurm, and slurm.all_nodes
CI / test (push) Successful in 9m30s

The choices screen has a Slurm part, and --slurm writes slurmctld on this
machine's node and slurmd on every node, those that join later too, into
server.yaml. slurm.all_nodes (ctl slurm setup --all-nodes) takes the nodes
from the leader's inventory. No node is lent at install.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-11 14:50:35 +01:00
1 parent b15944aa1f
commit 914985bf9f
13 files changed
+208 -20

No files matched your search

+14 -1
View File
@@ -166,13 +166,26 @@ is a flag too, and the plan shows the command with them: `--starter qwen3-8b`
`--judge-fail open`, `--judge-categories`, `--judge-replicas`,
`--judge-budget`, `--language tr`, `--signup-open`, `--default-rpm`,
`--default-tpm`, `--max-key-ttl`, `--media` (pictures and 3D: ComfyUI on the
nodes, and `presets.auto`), `--evict-after-idle` and `--min-lifetime`. They
nodes, and `presets.auto`), `--slurm` (see below), `--evict-after-idle` and `--min-lifetime`. They
go into `server.yaml` as init writes it, and become the cluster's first
settings when it starts; a `server.yaml` already there is never overwritten,
so run again they change nothing — the settings page does. An NVIDIA card with no working
driver is reported rather than silently ignored; install the driver and run
init again to serve on it.
**Slurm, for training and batch jobs.** `--slurm`, or the Slurm part on the
choices screen, runs a Slurm cluster beside llmgw: slurmctld on this
machine's node, slurmd on every node, those that join later too
(`slurm.all_nodes`). The agents install munge and Slurm (22.05 or newer)
from the machines' own package sources. No node is Slurm's at first: each is
drained in Slurm and serves models until `llmgw ctl slurm lend NODE` gives it
to Slurm — its models move off first — and `llmgw ctl slurm reclaim NODE`
takes it back once its jobs end. Jobs are submitted on this machine with
`sbatch` and `srun`; `llmgw ctl slurm status` shows where each node is. It
needs this machine to be a node, and its hostname to be a name Slurm takes.
On a cluster installed without it, `llmgw ctl slurm setup --controller NODE
--all-nodes` does the same later.
**Secure by default.** What init writes is the hardened configuration, not a
demonstration one:
+8
View File
@@ -406,6 +406,7 @@ rather than repeated.
| `cluster_name` | `string` | [SlurmSettings](#slurmsettings) |
| `controller` | `string` | [SlurmSettings](#slurmsettings) |
| `nodes` | `[]string` | [SlurmSettings](#slurmsettings) |
| `all_nodes` | `bool` | [SlurmSettings](#slurmsettings) |
| `lent` | `[]string` | [SlurmSettings](#slurmsettings) |
| `partition` | `string` | [SlurmSettings](#slurmsettings) |
| `dir` | `string` | [StrataConfig](#strataconfig) |
@@ -3525,6 +3526,13 @@ GPU, and need not be one of Nodes.
Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained
in Slurm, until it is also in Lent.
### `all_nodes`
`bool`
AllNodes runs slurmd on every llmgw node, those that join later too,
besides any named in Nodes. Each is llmgw's until it is lent, as any.
### `lent`
`[]string`