Slurm beside llmgw, P1: its settings and the files a controller runs on
slurm.{enabled,cluster_name,controller,nodes,lent,partition} in the cluster's
settings, checked; internal/slurm writes slurm.conf (configless), gres.conf
and cgroup.conf from the nodes' slurmd -C lines and GPUs, derives the munge
key from a cluster secret, and reads scontrol and version output.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
89767c8be1
commit
d439d507d7
9 files changed
+698
No files matched your search
+16
@@ -2013,6 +2013,22 @@ Recommendation: keep F4 in llmgw for small single-card adapters tied to KB grant
|
||||
build only A's first step (N GPUs on one node, placed by the control plane, with resume) if customers ask for
|
||||
mid-size jobs without wanting Slurm.
|
||||
|
||||
**B taken (2026-11-11).** Design: the cluster's settings say it (`slurm: enabled, cluster_name, controller, nodes,
|
||||
lent, partition`); the leader tells each named node's agent (root) what to be, the agents install and run it.
|
||||
- The controller is an llmgw node (its agent runs slurmctld); `nodes` run slurmd configless
|
||||
(`--conf-server`), so slurm.conf, gres.conf and cgroup.conf live only on the controller.
|
||||
- The munge key is derived from the cluster secret (HKDF), so nothing new is stored or copied by hand.
|
||||
- Each node reports `slurmd -C`; the leader writes slurm.conf from those lines, its mgmt address and its GPUs
|
||||
(`Gres=gpu:<type>:N`, gres.conf with the device files), and sends it to the controller again when one changes.
|
||||
- A node in `nodes` belongs to llmgw until it is in `lent`: drained in Slurm (reason "llmgw: serving models").
|
||||
Lent: llmgw drains it as `ctl drain` does (kept in force each tick, as facility holds are, so a new leader
|
||||
keeps it) and, once no replica is left, its agent resumes it in Slurm. Taken out of `lent`: the agent drains
|
||||
it in Slurm, reports the jobs left, and when none are the leader makes it ready for llmgw again.
|
||||
- Phases: P1 the `slurm` package (conf, key, parsers) and settings; P2 the agent side (proto, install, munge,
|
||||
slurmd/slurmctld, status); P3 the leader side (conf from reports, lend/reclaim); P4 `ctl slurm`
|
||||
(status, lend, reclaim) and a TUI screen; P5 `ctl train import` of a Slurm job's output into the mirror.
|
||||
Later: a backup slurmctld on a second node, accounting (slurmdbd on the Patroni PG), users from LDAP/AD.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, behind the token in
|
||||
|
||||
Reference in new issue
Block a user