Compare commits
No files matched your search
+167
-6
@@ -49,7 +49,8 @@ Face, done 2026-10-30), then M5 (engines without the internet), M3, M4, M6.
|
||||
**Scale.** O: more than one control plane answering clients; O1 (measuring one router's ceiling) first.
|
||||
|
||||
**Governance.**
|
||||
7. **C1**: a tenant-admin role, and forms hidden from viewers.
|
||||
7. ~~**C1**~~: done 2026-11-08. An organization's admin (`--role admin` on an org membership) changes its teams,
|
||||
projects, people and keys from /org; viewers are drawn no forms; forms made compact.
|
||||
8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
|
||||
|
||||
**When there is room.**
|
||||
@@ -435,9 +436,9 @@ portal is not done — the page shows; the console and ctl change.
|
||||
API middleware and the /admin pages (with each change's result), `ctl
|
||||
audit`, the console's Audit view; staff roles viewer / operator /
|
||||
billing / admin. Left: before/after values (the request's fields are
|
||||
recorded, not the state they replaced); a tenant-admin role that changes
|
||||
its own organization (managers still only view it); hiding the forms a
|
||||
viewer cannot submit. **Audit log and roles (M).** Today "audit" is slog lines
|
||||
recorded, not the state they replaced). Done 2026-11-08: an organization's
|
||||
admin who changes its own organization from /org (managers still only view
|
||||
it), and the forms a viewer cannot submit no longer drawn. **Audit log and roles (M).** Today "audit" is slog lines
|
||||
(`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag
|
||||
plus an org "manager" who can view. Add `audit_events` (who, what,
|
||||
before/after, source) written by the `adminAction`/`tenancyAction`
|
||||
@@ -462,8 +463,9 @@ portal is not done — the page shows; the console and ctl change.
|
||||
5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node
|
||||
lost, budget spent, update available; signed JSON or Slack; retries,
|
||||
hourly de-duplication), `ctl notify test`, Alertmanager receiver
|
||||
templates. Left: `server init` and the console do not yet ask for a
|
||||
receiver — the templates are filled in by hand. Events are claimed in
|
||||
templates. `server init` and both setup screens ask for webhooks and test
|
||||
each (d1e9186, 2026-10-14); with none set, `ctl status` and the console's
|
||||
Overview say the alerts go nowhere (2026-11-12). Events are claimed in
|
||||
the store before they are sent (2026-10-10), so two control planes send
|
||||
each once.
|
||||
**Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
|
||||
@@ -1982,6 +1984,165 @@ one control plane's router can. In order:
|
||||
- O3. Only if O2 is not enough: stateless gateway machines (`llmgw server --role gateway`) in front of the leader's
|
||||
control plane, as many as the traffic needs.
|
||||
|
||||
## P. Training beyond one card (written 2026-11-11) — decision open
|
||||
|
||||
F4 trains a LoRA on one card of a node the caller names (`ctl train start --node`), one job per node, no resume.
|
||||
Larger work (full fine-tuning, a LoRA on a 70B, several GPUs or machines) needs more. Two ways:
|
||||
|
||||
**A. In llmgw.** Add, in order: a job that takes N whole GPUs on one node (torchrun, FSDP/DeepSpeed in the agent's
|
||||
venv); placement by the control plane instead of `--node`, freeing GPUs from replicas by the scaling schedule or by
|
||||
draining; checkpoints to shared storage and resume when an agent or node goes; datasets uploaded or read from
|
||||
storage, not only KB pairs; then gang placement of N nodes with a rendezvous on the first, the NCCL environment from
|
||||
the fabric discovery (IB/NVLink, `llmgw doctor`); a console screen with logs and loss. Size: L for one node, XL for
|
||||
several. For: one product, one install, one set of users and grants (KB rules carry over), GPUs shared with
|
||||
inference without a second scheduler. Against: it is Slurm's gang scheduling, requeue and accounting written again;
|
||||
multi-node NCCL cannot be tried on the PoC (one GPU node, no IB); every customer's training code would have to fit
|
||||
our job format.
|
||||
|
||||
**B. Slurm beside llmgw.** The TUI installs Slurm on the nodes the customer picks (munge, slurmctld on the control
|
||||
planes, slurmd, users from the same LDAP/AD), with pyxis/enroot for containers. A node is handed between the two:
|
||||
llmgw drains its replicas and `scontrol update state=resume`, and back. llmgw's part is what comes out: a checkpoint
|
||||
or adapter from a Slurm job published to the mirror (`ctl train import`), served through canary and the eval. Size:
|
||||
M. For: what research and university customers already run (`sbatch`, `srun torchrun`), any training code, mature
|
||||
multi-node and requeue; we write only the install and the handover. Against: a second system with its own failures,
|
||||
upgrades and users to keep in step; GPUs split by hand or by the handover, not by demand; access rules on training
|
||||
data are Slurm's and the file system's, not the KB's.
|
||||
|
||||
Both need shared storage (NFS, BeeGFS, Lustre) the customer provides; neither changes F4.
|
||||
|
||||
Recommendation: keep F4 in llmgw for small single-card adapters tied to KB grants; take B for anything larger;
|
||||
build only A's first step (N GPUs on one node, placed by the control plane, with resume) if customers ask for
|
||||
mid-size jobs without wanting Slurm.
|
||||
|
||||
**B taken (2026-11-11).** Design: the cluster's settings say it (`slurm: enabled, cluster_name, controller, nodes,
|
||||
lent, partition`); the leader tells each named node's agent (root) what to be, the agents install and run it.
|
||||
- The controller is an llmgw node (its agent runs slurmctld); `nodes` run slurmd configless
|
||||
(`--conf-server`), so slurm.conf, gres.conf and cgroup.conf live only on the controller.
|
||||
- The munge key is derived from the cluster secret (HKDF), so nothing new is stored or copied by hand.
|
||||
- Each node reports `slurmd -C`; the leader writes slurm.conf from those lines, its mgmt address and its GPUs
|
||||
(`Gres=gpu:<type>:N`, gres.conf with the device files), and sends it to the controller again when one changes.
|
||||
- A node in `nodes` belongs to llmgw until it is in `lent`: drained in Slurm (reason "llmgw: serving models").
|
||||
Lent: llmgw drains it as `ctl drain` does (kept in force each tick, as facility holds are, so a new leader
|
||||
keeps it) and, once no replica is left, its agent resumes it in Slurm. Taken out of `lent`: the agent drains
|
||||
it in Slurm, reports the jobs left, and when none are the leader makes it ready for llmgw again.
|
||||
- Phases: P1 the `slurm` package (conf, key, parsers) and settings; P2 the agent side (proto, install, munge,
|
||||
slurmd/slurmctld, status); P3 the leader side (conf from reports, lend/reclaim); P4 `ctl slurm`
|
||||
(status, lend, reclaim) and a TUI screen; P5 `ctl train import` of a Slurm job's output into the mirror.
|
||||
Later: a backup slurmctld on a second node, accounting (slurmdbd; it takes MySQL/MariaDB only, not the Patroni PG), users from LDAP/AD.
|
||||
- Tried on the PoC 2026-11-11 (dev build v0.16.3-17-g849760c-slurm on 140, 150, node05, node06; Ubuntu
|
||||
24.04, Slurm 23.11.4 from the archive): `ctl slurm setup --controller llmgw-node06 --nodes llmgw-node05`
|
||||
installed munge, slurmctld and slurmd, node05 entered drained with its slurmd -C hardware (8 CPUs,
|
||||
31132 MiB after the 3%). `lend`: its 2 replicas moved off, then IDLE in Slurm; srun and sbatch ran there.
|
||||
`reclaim` with a 90 s job running: MIXED+DRAIN, llmgw kept off, ready again within a minute of the job
|
||||
ending. Found and fixed on the way: a client command needs its files even for --version (version now
|
||||
from the daemon); the packaged unit is Type=notify (slurmd needs --systemd); slurmd's own "slurmstepd
|
||||
infinity" was counted as a job; without accounting storage 23.11's main scheduler marks batch jobs
|
||||
InvalidAccount and backfill starts them, one a pass (bf_interval=10: 6-10 s each). So many small jobs
|
||||
want slurmdbd. — Wrong cause, found 2026-11-11: a job asking no memory got the whole node's (ReqTRES
|
||||
mem=all), so a node ran one job at a time and each next one waited for backfill's pass. Fixed: the
|
||||
partition's DefMemPerCPU is the leanest node's share; slurmdbd is not needed for this. InvalidAccount is real too:
|
||||
23.11 dropped accounting_storage/none, and without slurmdbd the main scheduler passes every job by;
|
||||
backfill starts it within bf_interval (10 s). priority/basic did not change that (fair share needs
|
||||
slurmdbd anyway). Starting on submission needs slurmdbd, on MySQL/MariaDB.
|
||||
- slurmdbd, done 2026-11-12 (748f777): the controller's agent installs MariaDB (bound to 127.0.0.1; a
|
||||
MariaDB that was there keeps its settings), makes `slurm_acct_db` and its user (password from the munge
|
||||
key, through a 0600 file, never argv), runs slurmdbd (6819, in the firewall for the cluster) and adds
|
||||
the cluster, before slurmctld. slurm.conf: AccountingStorageType=slurmdbd, jobacct_gather/cgroup,
|
||||
AccountingStorageTRES=gres/gpu, priority/multifactor. Tried on the PoC: five jobs started on
|
||||
submission, `sacct -a` has them with their TRES. Not tried: a fresh `server init --slurm` (needs VMs);
|
||||
a Red Hat controller.
|
||||
- Fresh install tried 2026-11-12 on VMs (3 control planes with Patroni, 2 CPU nodes, `server init
|
||||
--slurm`): Slurm, slurmdbd and MariaDB came up with the install; two-node srun/sbatch on the lent nodes,
|
||||
`sacct` has them; leader failover kept serving. Open from it: `--slurm` needs the first machine to be a
|
||||
node (the controller is its node), so a site wanting control planes without nodes chooses the
|
||||
controller later with `ctl slurm setup`; batch output lands on the job's first node until there is
|
||||
shared storage; on shared CPUs the judge refuses the first request after a placement change (fail
|
||||
closed, ~10 s), and its advice repeats `--cores 4` to a judge that has 4. Tried again on the idle
|
||||
VMs: one judge restarted in place, then both at once, requests every second: 48 of 48 served; the
|
||||
first after a cold start took 7.6–8.9 s end to end against ~3 s, within the 5 s judge budget. So the
|
||||
refusal was the install's load on the CPUs; if it is seen again, warm a new judge with one judgment
|
||||
before it takes requests.
|
||||
- Shared files and users, built 2026-11-12: `slurm.shared` {path, nfs | server} — the site's export
|
||||
mounted (a systemd mount unit, hard, nconnect=4), an llmgw node exporting the path itself
|
||||
(/etc/exports.d, to the Slurm machines' addresses, root_squash; 2049 in the firewall), or the site's
|
||||
own mount only looked for; `slurm.users: llmgw` — every active account a Linux user everywhere
|
||||
(`slurm_users`: login from the e-mail, UID from 200000, never reused; homes made on the exporting
|
||||
machine, else the controller; keys in /etc/llmgw/ssh-keys/%u through an sshd drop-in), SSH keys
|
||||
from the portal's Profile or `ctl slurm users keys`. A user already on a machine with another UID
|
||||
is left alone and said. The site has AD (not yet live): users from it, through sssd, are the next
|
||||
step (`slurm.users` empty today leaves them to the site).
|
||||
- Logging in to a node, done 2026-11-14: llmgw's users reach the controller freely, another Slurm
|
||||
machine only while a job of theirs runs there (pam_slurm_adopt, installed by the agent; PrologFlags=
|
||||
Contain). Only the users listed in the agent's users file are held: root and the machine's own users
|
||||
log in as before, so a wrong line never locks out who would fix it. pam_systemd would move the session
|
||||
out of the job, so sshd's stack takes the session file in place of its include, with pam_systemd
|
||||
skipped for the listed users alone; where that cannot be done safely (nested includes, jumps across
|
||||
it) they are held to having a job, not to its cores and cards, and the agent says so. Tried on the
|
||||
VMs: ayse without a job refused ("no active jobs on this node"), with one in
|
||||
job_N/step_extern (nproc 1), cut off when it ended; mehmet refused; admin's session unchanged.
|
||||
Left: sites whose users come from AD (sssd) are not held yet; a group of theirs would be the list.
|
||||
- A GPU node, tried 2026-11-14 on the PoC (cuda01, Quadro RTX 6000, Slurm 23.11.4): gres.conf
|
||||
`Type=rtx6000 File=/dev/nvidia0`, slurm.conf `Gres=gpu:rtx6000:1`. `srun --gres=gpu:1`:
|
||||
CUDA_VISIBLE_DEVICES=0, the one card; torch found it and ran 14.1 TFLOPS (fp32 matmul). Without
|
||||
--gres: "No devices found", torch none (ConstrainDevices holds). `--gres=gpu:2`: refused. sacct has
|
||||
`gres/gpu=1` in AllocTRES. Found: lending a node whose GPU model has nowhere else to go waits out
|
||||
the drain grace (5 min) before the replica stops, and `ctl slurm status` says only "1 replica(s)
|
||||
still to move off"; it should say it waits for a replacement that cannot be placed, and until when.
|
||||
Done (c3a75dc): it says the time.
|
||||
- Jobs in containers, done 2026-11-14: `slurm.containers` (`ctl slurm containers on`). Each agent
|
||||
installs enroot 4.2.1 (NVIDIA's .deb/.rpm, sha256 pinned) and builds pyxis 0.24.0 (source tarball,
|
||||
sha256 pinned) against its own Slurm (gcc, make, libslurm-dev); on a machine with nvidia-smi,
|
||||
nvidia-container-cli from NVIDIA's repository, its key's digest pinned. enroot.conf keeps each
|
||||
user's runtime, cache and data under /run/enroot and /var/lib/enroot (no login session needed), the
|
||||
shared path bind-mounted in every container. The leader writes plugstack.conf ("required") only
|
||||
once every machine reports both, since a machine without the plugin would fail every job. Tried on
|
||||
the VMs: on in ~30 s; ayse's srun --container-image=ubuntu:24.04 on n1 ran as uid 200001 with
|
||||
/shared inside and nproc 1; a two-node sbatch ran a rank on each. Seen once: a node's pull timed out
|
||||
reaching auth.docker.io and the job failed (network, not pyxis). Not tried: GPUs in a container
|
||||
(cuda01's agent is an older build; needs the peer protocol), Red Hat.
|
||||
- AD end to end, tried 2026-11-14 (decided: one AD per install, the existing auth.ldap group mapping, no
|
||||
tree from AD, no sssd unless a site asks): the VMs' auth.ldap against Samba DC VM 165 (ldaps by IP,
|
||||
CA in the settings, ml-team -> team:Test/arge). ada signed in on the portal with her AD password: an
|
||||
account, in Test/arge, Slurm user ada UID 200003; her key added in the portal's Profile, ssh to the
|
||||
controller ~20 s later, sbatch ran on n1 as ada. Disabled in AD: suspended in 56 s, the Linux user
|
||||
gone in 89 s, ssh refused, home kept. Enabled: back in 32 s, same UID, key kept. Taken out of
|
||||
ml-team: out of the team in 11 s; put back: in again in 56 s.
|
||||
- P6, Slurm's jobs in llmgw's accounting (written 2026-11-12, not started). The tenancy tree to Slurm:
|
||||
each project an account (`sacctmgr add account`, parent its team and organization), each member a user
|
||||
of it, kept by the leader from the store; `sbatch --account=PROJECT`. The records back: the leader reads
|
||||
`sacct --json` for jobs ended since the last read (through the controller's agent) into a
|
||||
`batch_usage` table (job, user, project, start, end, GPU-seconds by class, CPU-seconds, memory), priced
|
||||
per GPU-hour and core-hour into cost_micros, summed with the requests' usage on the usage pages and in
|
||||
budgets. Budgets enforced: a project's remaining budget as GPU-minutes in its account's GrpTRESMins, so
|
||||
Slurm holds its new jobs once spent. Needs the same users on every node (open: LDAP/sssd or
|
||||
llmgw-made). Also found: slurm.conf
|
||||
wrote every node State=DRAIN, so each slurmctld start (on any change of the files) drained a lent node
|
||||
until its agent resumed it, up to a minute; a lent node is now written without it. Not tried yet: a GPU node (gres).
|
||||
- Failover while lent, tried 2026-11-11 (v0.16.3-23): the first try killed the job (NODE_FAIL). A new
|
||||
leader had heard from no node yet and sent the controller a slurm.conf without them; slurmctld,
|
||||
restarted on it, dropped node05 and its job. Fixed (141a1f4, 0774e90): slurm.conf's nodes and the
|
||||
controller's address are kept in cluster_state (`slurm-nodes`) and a new leader starts from them; with
|
||||
nothing kept (an upgraded cluster) or the store unread, the controller's files wait up to 2 min for the
|
||||
nodes; a node is told nothing until its controller's address is known (no part stops its slurmd). Tried
|
||||
again: a 5-minute job on lent node05, leader 140 stopped, 150 took over; slurmctld not restarted, the job
|
||||
ran to its end; reclaimed during it, node05 was llmgw's within seconds of its end. Also 3387f97: srun in
|
||||
SrunPortRange=60001-63000, and Slurm's ports in llmgw's firewall for the cluster's networks.
|
||||
- Built 2026-11-11: P1 (`internal/slurm`, `slurm.*` settings), P2
|
||||
(`internal/agent/slurmnode`, proto Slurm/SlurmStatus), P3 (`internal/control/slurmsvc`, wired in the
|
||||
leader; `GET /api/v1/slurm`), P4's commands (`ctl slurm status|setup|lend|reclaim|off`). Decisions on the
|
||||
way: a node not heard from or with Slurm jobs is cordoned, not drained (nothing of llmgw's to move, and a
|
||||
new leader must not move models off every Slurm node); jobs are counted from the node's own slurmstepd,
|
||||
so a controller that is down neither hides them nor, with none, keeps the node from llmgw; a drain whose
|
||||
reason does not start with "llmgw" is a Slurm administrator's and never resumed. Left: ~~the TUI screen~~
|
||||
(done 2026-11-11: console **Slurm**, lend/take back on a row), P5,
|
||||
a try on the PoC (needs agents built from this; slurm-wlm 23.11 on Ubuntu 24.04), GPU minor numbers are
|
||||
taken to be nvidia-smi's index.
|
||||
- In the first install too (2026-11-11): `llmgw server init --slurm`, and a Slurm part on the choices
|
||||
screen, write `slurm: {enabled, controller: <this hostname>, all_nodes: true}` into server.yaml.
|
||||
`slurm.all_nodes` (also `ctl slurm setup --all-nodes`) runs slurmd on every node in the leader's
|
||||
inventory, those that join later too; none is lent. Refused with `--no-node`, or a hostname Slurm does
|
||||
not take.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, behind the token in
|
||||
|
||||
+108
@@ -26,6 +26,114 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Added
|
||||
|
||||
- **An organization's admin.** Besides member and manager, a membership of
|
||||
an organization may now be `admin` (`llmgw users join … --org Acme --role
|
||||
admin`, the role chosen on a membership on the people page, the console,
|
||||
the admin API). An admin sees the organization's page as a manager does and
|
||||
also changes the organization from it: adds, renames and removes its teams
|
||||
and projects, adds people with an approved account by their address and
|
||||
takes them out, makes its members managers or admins, and revokes keys spent
|
||||
in its projects. Limits, approving and suspending accounts, the
|
||||
organization's name and the rest of the cluster stay with the cluster's
|
||||
administrators; an admin cannot take away their own role. Every change is in
|
||||
the audit log.
|
||||
- **Slurm beside llmgw, for training and batch work.** `llmgw ctl slurm setup
|
||||
--controller NODE --nodes NODE,...` (or `--all-nodes`, or `server init
|
||||
--slurm`) has the agents install munge and Slurm from the machines' package
|
||||
sources: slurmctld on one node, slurmd configless on the others, slurm.conf
|
||||
and gres.conf written from what each node reports. Every node stays llmgw's,
|
||||
drained in Slurm, until `ctl slurm lend NODE`: its replicas move off, then
|
||||
Slurm runs jobs there. `ctl slurm reclaim NODE` takes it back once its jobs
|
||||
end. `ctl slurm status` and the console's **Slurm** show each node. The
|
||||
controller also keeps the jobs' records: slurmdbd, in a MariaDB on that
|
||||
machine alone (Slurm takes MySQL or MariaDB only). Slurm's ports
|
||||
(6817–6819, srun's 60001–63000) answer the cluster's networks only, as
|
||||
llmgw's own do.
|
||||
- **Slurm's shared files and users.** `slurm.shared` (`ctl slurm shared`) is
|
||||
one path on every Slurm machine: the site's NFS export mounted, an llmgw
|
||||
node exporting it over NFS (what `server init --slurm` chooses, `/shared`
|
||||
from the first machine), or a file system the site mounts. `slurm.users:
|
||||
llmgw` (`ctl slurm users on`) makes each active account a Linux user on
|
||||
every Slurm machine, the same login and UID (from 200000, never reused)
|
||||
everywhere, its home on the shared path; people add SSH keys in the
|
||||
portal's Profile (`ctl slurm users keys` for an operator). They log in to
|
||||
the controller; to another Slurm machine only while a job of theirs runs
|
||||
there, their session put in the job (pam_slurm_adopt), so nobody uses a
|
||||
node's cards outside Slurm, nor on a node llmgw serves from. Root and the
|
||||
machines' own users are not held to it.
|
||||
- **Slurm's jobs in containers.** `llmgw ctl slurm containers on`
|
||||
(`slurm.containers`) installs enroot and pyxis on every Slurm machine, at
|
||||
pinned versions checked against their digests, and nvidia-container-cli on
|
||||
GPU machines; Slurm loads pyxis once every machine has it, and then `srun
|
||||
--container-image=...` runs a job in that image, as its user, with the
|
||||
shared path inside.
|
||||
|
||||
### Changed
|
||||
|
||||
- **Forms only for those who may send them.** A viewer sees the people,
|
||||
limits, mirror and cluster pages without their forms, an operator the
|
||||
mirror's and a node's engines but not the rest; before, they were drawn and
|
||||
answered "not found".
|
||||
- **Smaller forms.** A label sits close to its field, a field is as wide as
|
||||
what goes in it rather than the page, buttons are smaller, and the short
|
||||
forms — a new key, a password, connecting an editor, a capacity or a limit —
|
||||
sit on one line or two instead of a screen.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **The serve forms start at the longest context that fits.** The mirror
|
||||
page's form began every GGUF model at 4096 tokens, and the console's at
|
||||
4096 or the plan's figure, whatever the model was trained for. Both now
|
||||
start at what it was trained for, as long as its KV cache fits where the
|
||||
form places it beside the weights, else at the most that does, in whole
|
||||
thousands; the mirror page's llama.cpp slots are as many of that, up to
|
||||
four, as the room left holds, and its memory figure is for that pool. A
|
||||
plan says what the model was trained for (`trained`). Strata stays at
|
||||
its 32768; vLLM's form is unchanged.
|
||||
- **A model that failed to start was started in the same place again.**
|
||||
Placement given the same inputs chooses the same node and card, so an
|
||||
on-demand model whose replica failed to load was retried there, past each
|
||||
backoff, until it was parked, with another card or node free all along.
|
||||
The control plane now remembers where a model's replicas failed (node and
|
||||
cards, up to eight places) and places the next one elsewhere when anywhere
|
||||
else fits; with nowhere else it is tried there as before. Coming up, a
|
||||
change to its definition or `ctl retry` forgets the places. The log says
|
||||
`placing a model away from where it failed to start`, with both places.
|
||||
- **Deleted model files came back after a control plane restart.** An agent
|
||||
registers with the files its startup scan found, and sends the list again
|
||||
only when it changes; a file deleted after the agent started was listed
|
||||
again by a control plane that restarted, until the next change. A new
|
||||
stream now gets the list on its first heartbeat.
|
||||
- **`ctl weights` offered models in use for deletion.** A model of several
|
||||
files (a set, its address ending in `/`) is fetched into a directory named
|
||||
from the address without the slash; the sweep named it from the address
|
||||
with it, so every set in the catalog looked like nobody's. On the PoC that
|
||||
was 330 GB of six catalog models, the qwen3.8-flash-next Q8_0 among them;
|
||||
`--collect`, and `ctl model remove --purge` of a set, would have deleted
|
||||
them. Both now compute the name the fetch does.
|
||||
- **A sign-in took an organization's manager role away.** A group from single
|
||||
sign-on, SCIM or Active Directory that puts someone in an organization wrote
|
||||
the membership again at every sign-in or sync, as a plain member: a manager
|
||||
made one on the people page was one only until they next signed in. A
|
||||
membership the directory names is now added if missing and otherwise left
|
||||
as it is.
|
||||
- **The limits form on the people page's panel** was laid out on one line, its
|
||||
title squeezed into a column.
|
||||
- **Deleting the model files nothing uses said what it was asked to free,
|
||||
not what it freed.** The console's "Delete model files nothing uses" and
|
||||
`ctl weights --collect` added up every file the nodes were asked to
|
||||
delete as soon as the request was sent, and a model of several files —
|
||||
fetched into a directory of its own — was refused by the node as "not
|
||||
empty", so a sweep said tens of gigabytes freed and nothing went. The
|
||||
node now deletes such a directory whole (a link only as a link), and the
|
||||
sweep waits for the nodes to list their files again: freed is what they
|
||||
no longer list, and what they kept is said beside, with where to read
|
||||
why. Removing a model with its files reports the same way.
|
||||
|
||||
## v0.16.3 — 2026-11-03
|
||||
|
||||
Replicas counted per profile — seven on cards and four on CPUs under one name, the cards filled first — and every
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
# Functional Source License, Version 1.1, ALv2 Future License
|
||||
|
||||
## Abbreviation
|
||||
|
||||
FSL-1.1-ALv2
|
||||
|
||||
## Notice
|
||||
|
||||
Copyright 2026 Ilker Manap
|
||||
|
||||
## Terms and Conditions
|
||||
|
||||
### Licensor ("We")
|
||||
|
||||
The party offering the Software under these Terms and Conditions.
|
||||
|
||||
### The Software
|
||||
|
||||
The "Software" is each version of the software that we make available under
|
||||
these Terms and Conditions, as indicated by our inclusion of these Terms and
|
||||
Conditions with the Software.
|
||||
|
||||
### License Grant
|
||||
|
||||
Subject to your compliance with this License Grant and the Patents,
|
||||
Redistribution and Trademark clauses below, we hereby grant you the right to
|
||||
use, copy, modify, create derivative works, publicly perform, publicly display
|
||||
and redistribute the Software for any Permitted Purpose identified below.
|
||||
|
||||
### Permitted Purpose
|
||||
|
||||
A Permitted Purpose is any purpose other than a Competing Use. A Competing Use
|
||||
means making the Software available to others in a commercial product or
|
||||
service that:
|
||||
|
||||
1. substitutes for the Software;
|
||||
|
||||
2. substitutes for any other product or service we offer using the Software
|
||||
that exists as of the date we make the Software available; or
|
||||
|
||||
3. offers the same or substantially similar functionality as the Software.
|
||||
|
||||
Permitted Purposes specifically include using the Software:
|
||||
|
||||
1. for your internal use and access;
|
||||
|
||||
2. for non-commercial education;
|
||||
|
||||
3. for non-commercial research; and
|
||||
|
||||
4. in connection with professional services that you provide to a licensee
|
||||
using the Software in accordance with these Terms and Conditions.
|
||||
|
||||
### Patents
|
||||
|
||||
To the extent your use for a Permitted Purpose would necessarily infringe our
|
||||
patents, the license grant above includes a license under our patents. If you
|
||||
make a claim against any party that the Software infringes or contributes to
|
||||
the infringement of any patent, then your patent license to the Software ends
|
||||
immediately.
|
||||
|
||||
### Redistribution
|
||||
|
||||
The Terms and Conditions apply to all copies, modifications and derivatives of
|
||||
the Software.
|
||||
|
||||
If you redistribute any copies, modifications or derivatives of the Software,
|
||||
you must include a copy of or a link to these Terms and Conditions and not
|
||||
remove any copyright notices provided in or with the Software.
|
||||
|
||||
### Disclaimer
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTIES OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING WITHOUT LIMITATION WARRANTIES OF FITNESS FOR A PARTICULAR
|
||||
PURPOSE, MERCHANTABILITY, TITLE OR NON-INFRINGEMENT.
|
||||
|
||||
IN NO EVENT WILL WE HAVE ANY LIABILITY TO YOU ARISING OUT OF OR RELATED TO THE
|
||||
SOFTWARE, INCLUDING INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES,
|
||||
EVEN IF WE HAVE BEEN INFORMED OF THEIR POSSIBILITY IN ADVANCE.
|
||||
|
||||
### Trademarks
|
||||
|
||||
Except for displaying the License Details and identifying us as the origin of
|
||||
the Software, you have no right under these Terms and Conditions to use our
|
||||
trademarks, trade names, service marks or product names.
|
||||
|
||||
## Grant of Future License
|
||||
|
||||
We hereby irrevocably grant you an additional license to use the Software under
|
||||
the Apache License, Version 2.0 that is effective on the second anniversary of
|
||||
the date we make the Software available. On or after that date, you may use the
|
||||
Software under the Apache License, Version 2.0, in which case the following
|
||||
will apply:
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use
|
||||
this file except in compliance with the License.
|
||||
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed
|
||||
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
|
||||
CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
@@ -372,3 +372,10 @@ make rename NEW=git.example.com/team/llmgw
|
||||
## Language
|
||||
|
||||
Code, comments, tests, commit messages and documentation are in English.
|
||||
|
||||
## Licence
|
||||
|
||||
Functional Source License 1.1 with an Apache 2.0 future licence
|
||||
(FSL-1.1-ALv2); see [LICENSE](LICENSE). Each release may be used, read and
|
||||
changed for anything except offering it as a competing product or service,
|
||||
and becomes Apache 2.0 two years after it is published.
|
||||
+637
-73
@@ -108,6 +108,7 @@ type AgentMessage struct {
|
||||
// *AgentMessage_KvEvents
|
||||
// *AgentMessage_TrainStatus
|
||||
// *AgentMessage_TrainArtifact
|
||||
// *AgentMessage_SlurmStatus
|
||||
Msg isAgentMessage_Msg `protobuf_oneof:"msg"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
@@ -240,6 +241,15 @@ func (x *AgentMessage) GetTrainArtifact() *TrainArtifact {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *AgentMessage) GetSlurmStatus() *SlurmStatus {
|
||||
if x != nil {
|
||||
if x, ok := x.Msg.(*AgentMessage_SlurmStatus); ok {
|
||||
return x.SlurmStatus
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type isAgentMessage_Msg interface {
|
||||
isAgentMessage_Msg()
|
||||
}
|
||||
@@ -284,6 +294,10 @@ type AgentMessage_TrainArtifact struct {
|
||||
TrainArtifact *TrainArtifact `protobuf:"bytes,10,opt,name=train_artifact,json=trainArtifact,proto3,oneof"`
|
||||
}
|
||||
|
||||
type AgentMessage_SlurmStatus struct {
|
||||
SlurmStatus *SlurmStatus `protobuf:"bytes,11,opt,name=slurm_status,json=slurmStatus,proto3,oneof"`
|
||||
}
|
||||
|
||||
func (*AgentMessage_Register) isAgentMessage_Msg() {}
|
||||
|
||||
func (*AgentMessage_Heartbeat) isAgentMessage_Msg() {}
|
||||
@@ -304,6 +318,8 @@ func (*AgentMessage_TrainStatus) isAgentMessage_Msg() {}
|
||||
|
||||
func (*AgentMessage_TrainArtifact) isAgentMessage_Msg() {}
|
||||
|
||||
func (*AgentMessage_SlurmStatus) isAgentMessage_Msg() {}
|
||||
|
||||
// KVEvents is a change to what one vLLM replica's prefix cache holds, read
|
||||
// by the agent from the engine's KV cache events and sent on for the router's
|
||||
// precise index (plan §7.5 tier 2).
|
||||
@@ -426,6 +442,7 @@ type ControlMessage struct {
|
||||
// *ControlMessage_AgentSettings
|
||||
// *ControlMessage_TrainStart
|
||||
// *ControlMessage_TrainCancel
|
||||
// *ControlMessage_Slurm
|
||||
Msg isControlMessage_Msg `protobuf_oneof:"msg"`
|
||||
// trace is the W3C trace context (traceparent, and tracestate when there
|
||||
// is one) of the control plane's span that sent this command, so what the
|
||||
@@ -618,6 +635,15 @@ func (x *ControlMessage) GetTrainCancel() *TrainCancel {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *ControlMessage) GetSlurm() *Slurm {
|
||||
if x != nil {
|
||||
if x, ok := x.Msg.(*ControlMessage_Slurm); ok {
|
||||
return x.Slurm
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *ControlMessage) GetTrace() map[string]string {
|
||||
if x != nil {
|
||||
return x.Trace
|
||||
@@ -693,6 +719,10 @@ type ControlMessage_TrainCancel struct {
|
||||
TrainCancel *TrainCancel `protobuf:"bytes,17,opt,name=train_cancel,json=trainCancel,proto3,oneof"`
|
||||
}
|
||||
|
||||
type ControlMessage_Slurm struct {
|
||||
Slurm *Slurm `protobuf:"bytes,18,opt,name=slurm,proto3,oneof"`
|
||||
}
|
||||
|
||||
func (*ControlMessage_Start) isControlMessage_Msg() {}
|
||||
|
||||
func (*ControlMessage_Stop) isControlMessage_Msg() {}
|
||||
@@ -725,6 +755,8 @@ func (*ControlMessage_TrainStart) isControlMessage_Msg() {}
|
||||
|
||||
func (*ControlMessage_TrainCancel) isControlMessage_Msg() {}
|
||||
|
||||
func (*ControlMessage_Slurm) isControlMessage_Msg() {}
|
||||
|
||||
// RestartAgent has the agent start again, as llmgw restart asks of every
|
||||
// node: its engines keep serving and are taken over again.
|
||||
type RestartAgent struct {
|
||||
@@ -5251,11 +5283,477 @@ func (x *TrainArtifact) GetLast() bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// Slurm is what this node is in the Slurm cluster run beside llmgw
|
||||
// (BACKLOG P): its controller, one of its nodes, both or neither. Sent when
|
||||
// an agent registers and whenever it changes; the agent installs what is
|
||||
// missing and keeps the daemons running as it says.
|
||||
type Slurm struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
// controller runs slurmctld here, on files.
|
||||
Controller bool `protobuf:"varint,1,opt,name=controller,proto3" json:"controller,omitempty"`
|
||||
// node runs slurmd here, configless, fetching its files from conf_server.
|
||||
Node bool `protobuf:"varint,2,opt,name=node,proto3" json:"node,omitempty"`
|
||||
// name is this node's NodeName in slurm.conf.
|
||||
Name string `protobuf:"bytes,3,opt,name=name,proto3" json:"name,omitempty"`
|
||||
// conf_server is slurmctld's host:port.
|
||||
ConfServer string `protobuf:"bytes,4,opt,name=conf_server,json=confServer,proto3" json:"conf_server,omitempty"`
|
||||
// munge_key is the key every machine of the cluster shares.
|
||||
MungeKey []byte `protobuf:"bytes,5,opt,name=munge_key,json=mungeKey,proto3" json:"munge_key,omitempty"`
|
||||
// files are the controller's, by name in /etc/slurm; empty for a node
|
||||
// that is not the controller.
|
||||
Files map[string]string `protobuf:"bytes,6,rep,name=files,proto3" json:"files,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
|
||||
// lent says Slurm may run jobs here now: resume the node if llmgw drained
|
||||
// it. Otherwise it is drained, its jobs left to finish.
|
||||
Lent bool `protobuf:"varint,7,opt,name=lent,proto3" json:"lent,omitempty"`
|
||||
// shared is the file system every machine of the Slurm cluster sees at
|
||||
// the same path; unset or with no path, none.
|
||||
Shared *SlurmShared `protobuf:"bytes,8,opt,name=shared,proto3" json:"shared,omitempty"`
|
||||
// manage_users has llmgw keep the Linux users below on this machine,
|
||||
// and take away those it made that are no longer among them.
|
||||
ManageUsers bool `protobuf:"varint,9,opt,name=manage_users,json=manageUsers,proto3" json:"manage_users,omitempty"`
|
||||
Users []*SlurmUser `protobuf:"bytes,10,rep,name=users,proto3" json:"users,omitempty"`
|
||||
// make_homes has this machine make the users' home directories: the one
|
||||
// that exports the shared path, else the controller.
|
||||
MakeHomes bool `protobuf:"varint,11,opt,name=make_homes,json=makeHomes,proto3" json:"make_homes,omitempty"`
|
||||
// containers has the machine install pyxis and enroot, so jobs may run
|
||||
// in containers (srun --container-image), and set them up.
|
||||
Containers bool `protobuf:"varint,12,opt,name=containers,proto3" json:"containers,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *Slurm) Reset() {
|
||||
*x = Slurm{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[48]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *Slurm) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*Slurm) ProtoMessage() {}
|
||||
|
||||
func (x *Slurm) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[48]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use Slurm.ProtoReflect.Descriptor instead.
|
||||
func (*Slurm) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{48}
|
||||
}
|
||||
|
||||
func (x *Slurm) GetController() bool {
|
||||
if x != nil {
|
||||
return x.Controller
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Slurm) GetNode() bool {
|
||||
if x != nil {
|
||||
return x.Node
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Slurm) GetName() string {
|
||||
if x != nil {
|
||||
return x.Name
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *Slurm) GetConfServer() string {
|
||||
if x != nil {
|
||||
return x.ConfServer
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *Slurm) GetMungeKey() []byte {
|
||||
if x != nil {
|
||||
return x.MungeKey
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *Slurm) GetFiles() map[string]string {
|
||||
if x != nil {
|
||||
return x.Files
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *Slurm) GetLent() bool {
|
||||
if x != nil {
|
||||
return x.Lent
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Slurm) GetShared() *SlurmShared {
|
||||
if x != nil {
|
||||
return x.Shared
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *Slurm) GetManageUsers() bool {
|
||||
if x != nil {
|
||||
return x.ManageUsers
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Slurm) GetUsers() []*SlurmUser {
|
||||
if x != nil {
|
||||
return x.Users
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *Slurm) GetMakeHomes() bool {
|
||||
if x != nil {
|
||||
return x.MakeHomes
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Slurm) GetContainers() bool {
|
||||
if x != nil {
|
||||
return x.Containers
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// SlurmShared is where the Slurm cluster's shared files come from.
|
||||
type SlurmShared struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
// path is where every machine sees them, as /shared.
|
||||
Path string `protobuf:"bytes,1,opt,name=path,proto3" json:"path,omitempty"`
|
||||
// nfs is an export, host:/path, to mount at path; empty when this
|
||||
// machine exports path itself (serve) or the site mounts it.
|
||||
Nfs string `protobuf:"bytes,2,opt,name=nfs,proto3" json:"nfs,omitempty"`
|
||||
// serve has this machine export path over NFS to clients.
|
||||
Serve bool `protobuf:"varint,3,opt,name=serve,proto3" json:"serve,omitempty"`
|
||||
// clients are the addresses the export answers.
|
||||
Clients []string `protobuf:"bytes,4,rep,name=clients,proto3" json:"clients,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *SlurmShared) Reset() {
|
||||
*x = SlurmShared{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *SlurmShared) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*SlurmShared) ProtoMessage() {}
|
||||
|
||||
func (x *SlurmShared) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use SlurmShared.ProtoReflect.Descriptor instead.
|
||||
func (*SlurmShared) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{49}
|
||||
}
|
||||
|
||||
func (x *SlurmShared) GetPath() string {
|
||||
if x != nil {
|
||||
return x.Path
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmShared) GetNfs() string {
|
||||
if x != nil {
|
||||
return x.Nfs
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmShared) GetServe() bool {
|
||||
if x != nil {
|
||||
return x.Serve
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *SlurmShared) GetClients() []string {
|
||||
if x != nil {
|
||||
return x.Clients
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SlurmUser is a Linux user that runs jobs, the same on every machine.
|
||||
type SlurmUser struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Login string `protobuf:"bytes,1,opt,name=login,proto3" json:"login,omitempty"`
|
||||
// uid is also the user's own group's gid.
|
||||
Uid uint32 `protobuf:"varint,2,opt,name=uid,proto3" json:"uid,omitempty"`
|
||||
// name is the account's, for the GECOS field.
|
||||
Name string `protobuf:"bytes,3,opt,name=name,proto3" json:"name,omitempty"`
|
||||
Home string `protobuf:"bytes,4,opt,name=home,proto3" json:"home,omitempty"`
|
||||
// ssh_keys are OpenSSH public keys, one a line of authorized_keys.
|
||||
SshKeys []string `protobuf:"bytes,5,rep,name=ssh_keys,json=sshKeys,proto3" json:"ssh_keys,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *SlurmUser) Reset() {
|
||||
*x = SlurmUser{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[50]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *SlurmUser) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*SlurmUser) ProtoMessage() {}
|
||||
|
||||
func (x *SlurmUser) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[50]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use SlurmUser.ProtoReflect.Descriptor instead.
|
||||
func (*SlurmUser) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{50}
|
||||
}
|
||||
|
||||
func (x *SlurmUser) GetLogin() string {
|
||||
if x != nil {
|
||||
return x.Login
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmUser) GetUid() uint32 {
|
||||
if x != nil {
|
||||
return x.Uid
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *SlurmUser) GetName() string {
|
||||
if x != nil {
|
||||
return x.Name
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmUser) GetHome() string {
|
||||
if x != nil {
|
||||
return x.Home
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmUser) GetSshKeys() []string {
|
||||
if x != nil {
|
||||
return x.SshKeys
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SlurmStatus is what this node's Slurm daemons are doing, sent when it
|
||||
// changes and every few minutes while the node has a part in Slurm.
|
||||
type SlurmStatus struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
// version is Slurm's, as "23.11.4"; empty when it is not installed.
|
||||
Version string `protobuf:"bytes,1,opt,name=version,proto3" json:"version,omitempty"`
|
||||
// hardware is `slurmd -C` on a node, which slurm.conf must match.
|
||||
Hardware string `protobuf:"bytes,2,opt,name=hardware,proto3" json:"hardware,omitempty"`
|
||||
SlurmdRunning bool `protobuf:"varint,3,opt,name=slurmd_running,json=slurmdRunning,proto3" json:"slurmd_running,omitempty"`
|
||||
SlurmctldRunning bool `protobuf:"varint,4,opt,name=slurmctld_running,json=slurmctldRunning,proto3" json:"slurmctld_running,omitempty"`
|
||||
// state and reason are the controller's for this node, from scontrol.
|
||||
State string `protobuf:"bytes,5,opt,name=state,proto3" json:"state,omitempty"`
|
||||
Reason string `protobuf:"bytes,6,opt,name=reason,proto3" json:"reason,omitempty"`
|
||||
// jobs is how many job steps run on this node now, counted from their
|
||||
// slurmstepd so a controller that is down does not hide them; -1 when the
|
||||
// agent cannot tell.
|
||||
Jobs int32 `protobuf:"varint,7,opt,name=jobs,proto3" json:"jobs,omitempty"`
|
||||
// error is the last thing that went wrong keeping Slurm, empty when
|
||||
// nothing did.
|
||||
Error string `protobuf:"bytes,8,opt,name=error,proto3" json:"error,omitempty"`
|
||||
// files_sha256 is of the controller's files as written, so the leader
|
||||
// knows they arrived.
|
||||
FilesSha256 string `protobuf:"bytes,9,opt,name=files_sha256,json=filesSha256,proto3" json:"files_sha256,omitempty"`
|
||||
// slurmdbd_running is the accounting daemon on the controller, which keeps
|
||||
// the jobs' records in a MariaDB beside it.
|
||||
SlurmdbdRunning bool `protobuf:"varint,10,opt,name=slurmdbd_running,json=slurmdbdRunning,proto3" json:"slurmdbd_running,omitempty"`
|
||||
// shared is the shared path as this machine has it: "mounted",
|
||||
// "exported", "missing", or empty when there is none.
|
||||
Shared string `protobuf:"bytes,11,opt,name=shared,proto3" json:"shared,omitempty"`
|
||||
// users is how many of llmgw's users this machine has.
|
||||
Users int32 `protobuf:"varint,12,opt,name=users,proto3" json:"users,omitempty"`
|
||||
// containers is what this machine has for jobs in containers, as
|
||||
// "pyxis 0.24.0, enroot 4.2.1"; empty while it has not got both.
|
||||
Containers string `protobuf:"bytes,13,opt,name=containers,proto3" json:"containers,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) Reset() {
|
||||
*x = SlurmStatus{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[51]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*SlurmStatus) ProtoMessage() {}
|
||||
|
||||
func (x *SlurmStatus) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[51]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use SlurmStatus.ProtoReflect.Descriptor instead.
|
||||
func (*SlurmStatus) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{51}
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetVersion() string {
|
||||
if x != nil {
|
||||
return x.Version
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetHardware() string {
|
||||
if x != nil {
|
||||
return x.Hardware
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetSlurmdRunning() bool {
|
||||
if x != nil {
|
||||
return x.SlurmdRunning
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetSlurmctldRunning() bool {
|
||||
if x != nil {
|
||||
return x.SlurmctldRunning
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetState() string {
|
||||
if x != nil {
|
||||
return x.State
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetReason() string {
|
||||
if x != nil {
|
||||
return x.Reason
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetJobs() int32 {
|
||||
if x != nil {
|
||||
return x.Jobs
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetError() string {
|
||||
if x != nil {
|
||||
return x.Error
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetFilesSha256() string {
|
||||
if x != nil {
|
||||
return x.FilesSha256
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetSlurmdbdRunning() bool {
|
||||
if x != nil {
|
||||
return x.SlurmdbdRunning
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetShared() string {
|
||||
if x != nil {
|
||||
return x.Shared
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetUsers() int32 {
|
||||
if x != nil {
|
||||
return x.Users
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetContainers() string {
|
||||
if x != nil {
|
||||
return x.Containers
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
var File_llmgw_v1_agent_proto protoreflect.FileDescriptor
|
||||
|
||||
const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\n" +
|
||||
"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xb7\x04\n" +
|
||||
"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xf3\x04\n" +
|
||||
"\fAgentMessage\x120\n" +
|
||||
"\bregister\x18\x01 \x01(\v2\x12.llmgw.v1.RegisterH\x00R\bregister\x123\n" +
|
||||
"\theartbeat\x18\x02 \x01(\v2\x13.llmgw.v1.HeartbeatH\x00R\theartbeat\x12-\n" +
|
||||
@@ -5269,7 +5767,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\tkv_events\x18\b \x01(\v2\x12.llmgw.v1.KVEventsH\x00R\bkvEvents\x12:\n" +
|
||||
"\ftrain_status\x18\t \x01(\v2\x15.llmgw.v1.TrainStatusH\x00R\vtrainStatus\x12@\n" +
|
||||
"\x0etrain_artifact\x18\n" +
|
||||
" \x01(\v2\x17.llmgw.v1.TrainArtifactH\x00R\rtrainArtifactB\x05\n" +
|
||||
" \x01(\v2\x17.llmgw.v1.TrainArtifactH\x00R\rtrainArtifact\x12:\n" +
|
||||
"\fslurm_status\x18\v \x01(\v2\x15.llmgw.v1.SlurmStatusH\x00R\vslurmStatusB\x05\n" +
|
||||
"\x03msg\"\xb5\x01\n" +
|
||||
"\bKVEvents\x12\x1d\n" +
|
||||
"\n" +
|
||||
@@ -5280,7 +5779,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"clearFirst\x12\x16\n" +
|
||||
"\x06stored\x18\x04 \x03(\x06R\x06stored\x12\x18\n" +
|
||||
"\aremoved\x18\x05 \x03(\x06R\aremoved\x12\x18\n" +
|
||||
"\aproblem\x18\x06 \x01(\tR\aproblem\"\xf8\a\n" +
|
||||
"\aproblem\x18\x06 \x01(\tR\aproblem\"\xa1\b\n" +
|
||||
"\x0eControlMessage\x12,\n" +
|
||||
"\x05start\x18\x01 \x01(\v2\x14.llmgw.v1.StartShardH\x00R\x05start\x12)\n" +
|
||||
"\x04stop\x18\x02 \x01(\v2\x13.llmgw.v1.StopShardH\x00R\x04stop\x12,\n" +
|
||||
@@ -5301,7 +5800,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x0eagent_settings\x18\x0f \x01(\v2\x17.llmgw.v1.AgentSettingsH\x00R\ragentSettings\x127\n" +
|
||||
"\vtrain_start\x18\x10 \x01(\v2\x14.llmgw.v1.TrainStartH\x00R\n" +
|
||||
"trainStart\x12:\n" +
|
||||
"\ftrain_cancel\x18\x11 \x01(\v2\x15.llmgw.v1.TrainCancelH\x00R\vtrainCancel\x129\n" +
|
||||
"\ftrain_cancel\x18\x11 \x01(\v2\x15.llmgw.v1.TrainCancelH\x00R\vtrainCancel\x12'\n" +
|
||||
"\x05slurm\x18\x12 \x01(\v2\x0f.llmgw.v1.SlurmH\x00R\x05slurm\x129\n" +
|
||||
"\x05trace\x18\f \x03(\v2#.llmgw.v1.ControlMessage.TraceEntryR\x05trace\x1a8\n" +
|
||||
"\n" +
|
||||
"TraceEntry\x12\x10\n" +
|
||||
@@ -5754,7 +6254,59 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x04kind\x18\x02 \x01(\tR\x04kind\x12\x10\n" +
|
||||
"\x03seq\x18\x03 \x01(\x05R\x03seq\x12\x12\n" +
|
||||
"\x04data\x18\x04 \x01(\fR\x04data\x12\x12\n" +
|
||||
"\x04last\x18\x05 \x01(\bR\x04last2O\n" +
|
||||
"\x04last\x18\x05 \x01(\bR\x04last\"\xc9\x03\n" +
|
||||
"\x05Slurm\x12\x1e\n" +
|
||||
"\n" +
|
||||
"controller\x18\x01 \x01(\bR\n" +
|
||||
"controller\x12\x12\n" +
|
||||
"\x04node\x18\x02 \x01(\bR\x04node\x12\x12\n" +
|
||||
"\x04name\x18\x03 \x01(\tR\x04name\x12\x1f\n" +
|
||||
"\vconf_server\x18\x04 \x01(\tR\n" +
|
||||
"confServer\x12\x1b\n" +
|
||||
"\tmunge_key\x18\x05 \x01(\fR\bmungeKey\x120\n" +
|
||||
"\x05files\x18\x06 \x03(\v2\x1a.llmgw.v1.Slurm.FilesEntryR\x05files\x12\x12\n" +
|
||||
"\x04lent\x18\a \x01(\bR\x04lent\x12-\n" +
|
||||
"\x06shared\x18\b \x01(\v2\x15.llmgw.v1.SlurmSharedR\x06shared\x12!\n" +
|
||||
"\fmanage_users\x18\t \x01(\bR\vmanageUsers\x12)\n" +
|
||||
"\x05users\x18\n" +
|
||||
" \x03(\v2\x13.llmgw.v1.SlurmUserR\x05users\x12\x1d\n" +
|
||||
"\n" +
|
||||
"make_homes\x18\v \x01(\bR\tmakeHomes\x12\x1e\n" +
|
||||
"\n" +
|
||||
"containers\x18\f \x01(\bR\n" +
|
||||
"containers\x1a8\n" +
|
||||
"\n" +
|
||||
"FilesEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"c\n" +
|
||||
"\vSlurmShared\x12\x12\n" +
|
||||
"\x04path\x18\x01 \x01(\tR\x04path\x12\x10\n" +
|
||||
"\x03nfs\x18\x02 \x01(\tR\x03nfs\x12\x14\n" +
|
||||
"\x05serve\x18\x03 \x01(\bR\x05serve\x12\x18\n" +
|
||||
"\aclients\x18\x04 \x03(\tR\aclients\"v\n" +
|
||||
"\tSlurmUser\x12\x14\n" +
|
||||
"\x05login\x18\x01 \x01(\tR\x05login\x12\x10\n" +
|
||||
"\x03uid\x18\x02 \x01(\rR\x03uid\x12\x12\n" +
|
||||
"\x04name\x18\x03 \x01(\tR\x04name\x12\x12\n" +
|
||||
"\x04home\x18\x04 \x01(\tR\x04home\x12\x19\n" +
|
||||
"\bssh_keys\x18\x05 \x03(\tR\asshKeys\"\x8b\x03\n" +
|
||||
"\vSlurmStatus\x12\x18\n" +
|
||||
"\aversion\x18\x01 \x01(\tR\aversion\x12\x1a\n" +
|
||||
"\bhardware\x18\x02 \x01(\tR\bhardware\x12%\n" +
|
||||
"\x0eslurmd_running\x18\x03 \x01(\bR\rslurmdRunning\x12+\n" +
|
||||
"\x11slurmctld_running\x18\x04 \x01(\bR\x10slurmctldRunning\x12\x14\n" +
|
||||
"\x05state\x18\x05 \x01(\tR\x05state\x12\x16\n" +
|
||||
"\x06reason\x18\x06 \x01(\tR\x06reason\x12\x12\n" +
|
||||
"\x04jobs\x18\a \x01(\x05R\x04jobs\x12\x14\n" +
|
||||
"\x05error\x18\b \x01(\tR\x05error\x12!\n" +
|
||||
"\ffiles_sha256\x18\t \x01(\tR\vfilesSha256\x12)\n" +
|
||||
"\x10slurmdbd_running\x18\n" +
|
||||
" \x01(\bR\x0fslurmdbdRunning\x12\x16\n" +
|
||||
"\x06shared\x18\v \x01(\tR\x06shared\x12\x14\n" +
|
||||
"\x05users\x18\f \x01(\x05R\x05users\x12\x1e\n" +
|
||||
"\n" +
|
||||
"containers\x18\r \x01(\tR\n" +
|
||||
"containers2O\n" +
|
||||
"\fAgentService\x12?\n" +
|
||||
"\aConnect\x12\x16.llmgw.v1.AgentMessage\x1a\x18.llmgw.v1.ControlMessage(\x010\x01B\x93\x01\n" +
|
||||
"\fcom.llmgw.v1B\n" +
|
||||
@@ -5773,7 +6325,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
|
||||
}
|
||||
|
||||
var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
|
||||
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 56)
|
||||
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 61)
|
||||
var file_llmgw_v1_agent_proto_goTypes = []any{
|
||||
(ShardStatus_State)(0), // 0: llmgw.v1.ShardStatus.State
|
||||
(*AgentMessage)(nil), // 1: llmgw.v1.AgentMessage
|
||||
@@ -5824,14 +6376,19 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
|
||||
(*TrainCancel)(nil), // 46: llmgw.v1.TrainCancel
|
||||
(*TrainStatus)(nil), // 47: llmgw.v1.TrainStatus
|
||||
(*TrainArtifact)(nil), // 48: llmgw.v1.TrainArtifact
|
||||
nil, // 49: llmgw.v1.ControlMessage.TraceEntry
|
||||
nil, // 50: llmgw.v1.Register.LabelsEntry
|
||||
nil, // 51: llmgw.v1.Register.EngineVersionsEntry
|
||||
nil, // 52: llmgw.v1.GPU.NicPathsEntry
|
||||
nil, // 53: llmgw.v1.IBPort.CountersEntry
|
||||
nil, // 54: llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
nil, // 55: llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
nil, // 56: llmgw.v1.EngineProfile.EnvEntry
|
||||
(*Slurm)(nil), // 49: llmgw.v1.Slurm
|
||||
(*SlurmShared)(nil), // 50: llmgw.v1.SlurmShared
|
||||
(*SlurmUser)(nil), // 51: llmgw.v1.SlurmUser
|
||||
(*SlurmStatus)(nil), // 52: llmgw.v1.SlurmStatus
|
||||
nil, // 53: llmgw.v1.ControlMessage.TraceEntry
|
||||
nil, // 54: llmgw.v1.Register.LabelsEntry
|
||||
nil, // 55: llmgw.v1.Register.EngineVersionsEntry
|
||||
nil, // 56: llmgw.v1.GPU.NicPathsEntry
|
||||
nil, // 57: llmgw.v1.IBPort.CountersEntry
|
||||
nil, // 58: llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
nil, // 59: llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
nil, // 60: llmgw.v1.EngineProfile.EnvEntry
|
||||
nil, // 61: llmgw.v1.Slurm.FilesEntry
|
||||
}
|
||||
var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
8, // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
|
||||
@@ -5844,64 +6401,69 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
2, // 7: llmgw.v1.AgentMessage.kv_events:type_name -> llmgw.v1.KVEvents
|
||||
47, // 8: llmgw.v1.AgentMessage.train_status:type_name -> llmgw.v1.TrainStatus
|
||||
48, // 9: llmgw.v1.AgentMessage.train_artifact:type_name -> llmgw.v1.TrainArtifact
|
||||
30, // 10: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
|
||||
33, // 11: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
|
||||
34, // 12: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
|
||||
35, // 13: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
|
||||
32, // 14: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
|
||||
24, // 15: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
|
||||
36, // 16: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
|
||||
38, // 17: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
|
||||
40, // 18: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
|
||||
42, // 19: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
|
||||
7, // 20: llmgw.v1.ControlMessage.control_planes:type_name -> llmgw.v1.ControlPlanes
|
||||
5, // 21: llmgw.v1.ControlMessage.firewall:type_name -> llmgw.v1.Firewall
|
||||
4, // 22: llmgw.v1.ControlMessage.restart_agent:type_name -> llmgw.v1.RestartAgent
|
||||
6, // 23: llmgw.v1.ControlMessage.agent_settings:type_name -> llmgw.v1.AgentSettings
|
||||
45, // 24: llmgw.v1.ControlMessage.train_start:type_name -> llmgw.v1.TrainStart
|
||||
46, // 25: llmgw.v1.ControlMessage.train_cancel:type_name -> llmgw.v1.TrainCancel
|
||||
49, // 26: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
|
||||
50, // 27: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
|
||||
12, // 28: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
|
||||
14, // 29: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
51, // 30: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
|
||||
11, // 31: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
|
||||
15, // 32: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
|
||||
10, // 33: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
|
||||
9, // 34: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
|
||||
21, // 35: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
|
||||
0, // 36: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
27, // 37: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
|
||||
52, // 38: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
|
||||
13, // 39: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
|
||||
18, // 40: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
|
||||
17, // 41: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
|
||||
53, // 42: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
|
||||
20, // 43: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
|
||||
10, // 44: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
|
||||
15, // 45: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
|
||||
54, // 46: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
26, // 47: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
|
||||
16, // 48: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
|
||||
19, // 49: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
|
||||
14, // 50: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
22, // 51: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
|
||||
21, // 52: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
|
||||
25, // 53: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
|
||||
55, // 54: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
0, // 55: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
29, // 56: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
|
||||
27, // 57: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
|
||||
31, // 58: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
|
||||
56, // 59: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
|
||||
44, // 60: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
|
||||
1, // 61: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
|
||||
3, // 62: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
|
||||
62, // [62:63] is the sub-list for method output_type
|
||||
61, // [61:62] is the sub-list for method input_type
|
||||
61, // [61:61] is the sub-list for extension type_name
|
||||
61, // [61:61] is the sub-list for extension extendee
|
||||
0, // [0:61] is the sub-list for field type_name
|
||||
52, // 10: llmgw.v1.AgentMessage.slurm_status:type_name -> llmgw.v1.SlurmStatus
|
||||
30, // 11: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
|
||||
33, // 12: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
|
||||
34, // 13: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
|
||||
35, // 14: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
|
||||
32, // 15: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
|
||||
24, // 16: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
|
||||
36, // 17: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
|
||||
38, // 18: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
|
||||
40, // 19: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
|
||||
42, // 20: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
|
||||
7, // 21: llmgw.v1.ControlMessage.control_planes:type_name -> llmgw.v1.ControlPlanes
|
||||
5, // 22: llmgw.v1.ControlMessage.firewall:type_name -> llmgw.v1.Firewall
|
||||
4, // 23: llmgw.v1.ControlMessage.restart_agent:type_name -> llmgw.v1.RestartAgent
|
||||
6, // 24: llmgw.v1.ControlMessage.agent_settings:type_name -> llmgw.v1.AgentSettings
|
||||
45, // 25: llmgw.v1.ControlMessage.train_start:type_name -> llmgw.v1.TrainStart
|
||||
46, // 26: llmgw.v1.ControlMessage.train_cancel:type_name -> llmgw.v1.TrainCancel
|
||||
49, // 27: llmgw.v1.ControlMessage.slurm:type_name -> llmgw.v1.Slurm
|
||||
53, // 28: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
|
||||
54, // 29: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
|
||||
12, // 30: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
|
||||
14, // 31: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
55, // 32: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
|
||||
11, // 33: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
|
||||
15, // 34: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
|
||||
10, // 35: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
|
||||
9, // 36: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
|
||||
21, // 37: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
|
||||
0, // 38: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
27, // 39: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
|
||||
56, // 40: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
|
||||
13, // 41: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
|
||||
18, // 42: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
|
||||
17, // 43: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
|
||||
57, // 44: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
|
||||
20, // 45: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
|
||||
10, // 46: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
|
||||
15, // 47: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
|
||||
58, // 48: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
26, // 49: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
|
||||
16, // 50: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
|
||||
19, // 51: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
|
||||
14, // 52: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
22, // 53: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
|
||||
21, // 54: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
|
||||
25, // 55: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
|
||||
59, // 56: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
0, // 57: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
29, // 58: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
|
||||
27, // 59: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
|
||||
31, // 60: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
|
||||
60, // 61: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
|
||||
44, // 62: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
|
||||
61, // 63: llmgw.v1.Slurm.files:type_name -> llmgw.v1.Slurm.FilesEntry
|
||||
50, // 64: llmgw.v1.Slurm.shared:type_name -> llmgw.v1.SlurmShared
|
||||
51, // 65: llmgw.v1.Slurm.users:type_name -> llmgw.v1.SlurmUser
|
||||
1, // 66: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
|
||||
3, // 67: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
|
||||
67, // [67:68] is the sub-list for method output_type
|
||||
66, // [66:67] is the sub-list for method input_type
|
||||
66, // [66:66] is the sub-list for extension type_name
|
||||
66, // [66:66] is the sub-list for extension extendee
|
||||
0, // [0:66] is the sub-list for field type_name
|
||||
}
|
||||
|
||||
func init() { file_llmgw_v1_agent_proto_init() }
|
||||
@@ -5920,6 +6482,7 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
(*AgentMessage_KvEvents)(nil),
|
||||
(*AgentMessage_TrainStatus)(nil),
|
||||
(*AgentMessage_TrainArtifact)(nil),
|
||||
(*AgentMessage_SlurmStatus)(nil),
|
||||
}
|
||||
file_llmgw_v1_agent_proto_msgTypes[2].OneofWrappers = []any{
|
||||
(*ControlMessage_Start)(nil),
|
||||
@@ -5938,6 +6501,7 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
(*ControlMessage_AgentSettings)(nil),
|
||||
(*ControlMessage_TrainStart)(nil),
|
||||
(*ControlMessage_TrainCancel)(nil),
|
||||
(*ControlMessage_Slurm)(nil),
|
||||
}
|
||||
type x struct{}
|
||||
out := protoimpl.TypeBuilder{
|
||||
@@ -5945,7 +6509,7 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
|
||||
RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
|
||||
NumEnums: 1,
|
||||
NumMessages: 56,
|
||||
NumMessages: 61,
|
||||
NumExtensions: 0,
|
||||
NumServices: 1,
|
||||
},
|
||||
|
||||
@@ -30,6 +30,7 @@ message AgentMessage {
|
||||
KVEvents kv_events = 8;
|
||||
TrainStatus train_status = 9;
|
||||
TrainArtifact train_artifact = 10;
|
||||
SlurmStatus slurm_status = 11;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -78,6 +79,7 @@ message ControlMessage {
|
||||
AgentSettings agent_settings = 15;
|
||||
TrainStart train_start = 16;
|
||||
TrainCancel train_cancel = 17;
|
||||
Slurm slurm = 18;
|
||||
}
|
||||
// trace is the W3C trace context (traceparent, and tracestate when there
|
||||
// is one) of the control plane's span that sent this command, so what the
|
||||
@@ -986,3 +988,99 @@ message TrainArtifact {
|
||||
bytes data = 4;
|
||||
bool last = 5;
|
||||
}
|
||||
|
||||
// Slurm is what this node is in the Slurm cluster run beside llmgw
|
||||
// (BACKLOG P): its controller, one of its nodes, both or neither. Sent when
|
||||
// an agent registers and whenever it changes; the agent installs what is
|
||||
// missing and keeps the daemons running as it says.
|
||||
message Slurm {
|
||||
// controller runs slurmctld here, on files.
|
||||
bool controller = 1;
|
||||
// node runs slurmd here, configless, fetching its files from conf_server.
|
||||
bool node = 2;
|
||||
// name is this node's NodeName in slurm.conf.
|
||||
string name = 3;
|
||||
// conf_server is slurmctld's host:port.
|
||||
string conf_server = 4;
|
||||
// munge_key is the key every machine of the cluster shares.
|
||||
bytes munge_key = 5;
|
||||
// files are the controller's, by name in /etc/slurm; empty for a node
|
||||
// that is not the controller.
|
||||
map<string, string> files = 6;
|
||||
// lent says Slurm may run jobs here now: resume the node if llmgw drained
|
||||
// it. Otherwise it is drained, its jobs left to finish.
|
||||
bool lent = 7;
|
||||
// shared is the file system every machine of the Slurm cluster sees at
|
||||
// the same path; unset or with no path, none.
|
||||
SlurmShared shared = 8;
|
||||
// manage_users has llmgw keep the Linux users below on this machine,
|
||||
// and take away those it made that are no longer among them.
|
||||
bool manage_users = 9;
|
||||
repeated SlurmUser users = 10;
|
||||
// make_homes has this machine make the users' home directories: the one
|
||||
// that exports the shared path, else the controller.
|
||||
bool make_homes = 11;
|
||||
// containers has the machine install pyxis and enroot, so jobs may run
|
||||
// in containers (srun --container-image), and set them up.
|
||||
bool containers = 12;
|
||||
}
|
||||
|
||||
// SlurmShared is where the Slurm cluster's shared files come from.
|
||||
message SlurmShared {
|
||||
// path is where every machine sees them, as /shared.
|
||||
string path = 1;
|
||||
// nfs is an export, host:/path, to mount at path; empty when this
|
||||
// machine exports path itself (serve) or the site mounts it.
|
||||
string nfs = 2;
|
||||
// serve has this machine export path over NFS to clients.
|
||||
bool serve = 3;
|
||||
// clients are the addresses the export answers.
|
||||
repeated string clients = 4;
|
||||
}
|
||||
|
||||
// SlurmUser is a Linux user that runs jobs, the same on every machine.
|
||||
message SlurmUser {
|
||||
string login = 1;
|
||||
// uid is also the user's own group's gid.
|
||||
uint32 uid = 2;
|
||||
// name is the account's, for the GECOS field.
|
||||
string name = 3;
|
||||
string home = 4;
|
||||
// ssh_keys are OpenSSH public keys, one a line of authorized_keys.
|
||||
repeated string ssh_keys = 5;
|
||||
}
|
||||
|
||||
// SlurmStatus is what this node's Slurm daemons are doing, sent when it
|
||||
// changes and every few minutes while the node has a part in Slurm.
|
||||
message SlurmStatus {
|
||||
// version is Slurm's, as "23.11.4"; empty when it is not installed.
|
||||
string version = 1;
|
||||
// hardware is `slurmd -C` on a node, which slurm.conf must match.
|
||||
string hardware = 2;
|
||||
bool slurmd_running = 3;
|
||||
bool slurmctld_running = 4;
|
||||
// state and reason are the controller's for this node, from scontrol.
|
||||
string state = 5;
|
||||
string reason = 6;
|
||||
// jobs is how many job steps run on this node now, counted from their
|
||||
// slurmstepd so a controller that is down does not hide them; -1 when the
|
||||
// agent cannot tell.
|
||||
int32 jobs = 7;
|
||||
// error is the last thing that went wrong keeping Slurm, empty when
|
||||
// nothing did.
|
||||
string error = 8;
|
||||
// files_sha256 is of the controller's files as written, so the leader
|
||||
// knows they arrived.
|
||||
string files_sha256 = 9;
|
||||
// slurmdbd_running is the accounting daemon on the controller, which keeps
|
||||
// the jobs' records in a MariaDB beside it.
|
||||
bool slurmdbd_running = 10;
|
||||
// shared is the shared path as this machine has it: "mounted",
|
||||
// "exported", "missing", or empty when there is none.
|
||||
string shared = 11;
|
||||
// users is how many of llmgw's users this machine has.
|
||||
int32 users = 12;
|
||||
// containers is what this machine has for jobs in containers, as
|
||||
// "pyxis 0.24.0, enroot 4.2.1"; empty while it has not got both.
|
||||
string containers = 13;
|
||||
}
|
||||
+308
-1
@@ -116,6 +116,9 @@
|
||||
{
|
||||
"name": "settings"
|
||||
},
|
||||
{
|
||||
"name": "slurm"
|
||||
},
|
||||
{
|
||||
"name": "summary"
|
||||
},
|
||||
@@ -5078,6 +5081,117 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/slurm": {
|
||||
"get": {
|
||||
"operationId": "getSlurm",
|
||||
"summary": "The Slurm cluster beside llmgw, and where each of its nodes is",
|
||||
"description": "From the cluster's settings (slurm.*) and what each node's agent last said. A node's phase is llmgw (drained in Slurm, no job left), lending (its replicas moving off), lent, reclaiming (drained in Slurm, jobs still running) or unknown (its agent has not said).",
|
||||
"tags": [
|
||||
"slurm"
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/slurmsvc.Status"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/slurm/users": {
|
||||
"get": {
|
||||
"operationId": "getSlurmUsers",
|
||||
"summary": "The Linux users made from llmgw's accounts for the Slurm cluster",
|
||||
"description": "Every active account, with its login, UID and SSH keys; an account is given its login and UID the first time it is listed. The machines have them when slurm.users is \"llmgw\".",
|
||||
"tags": [
|
||||
"slurm"
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/slurmsvc.User"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/slurm/users/{user}/ssh-keys": {
|
||||
"put": {
|
||||
"operationId": "putSlurmUsersUserSshKeys",
|
||||
"summary": "Replace an account's SSH public keys",
|
||||
"description": "{user} is an ID or e-mail. keys is OpenSSH public keys, one a line, without options; empty takes them all away. The machines have them within a minute.",
|
||||
"tags": [
|
||||
"slurm"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "user",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "The person's ID or email.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/SlurmKeysRequest"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/slurmsvc.User"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/summary": {
|
||||
"get": {
|
||||
"operationId": "getSummary",
|
||||
@@ -6234,7 +6348,7 @@
|
||||
},
|
||||
"CollectResult": {
|
||||
"type": "object",
|
||||
"description": "CollectResult is what was deleted on one node, or why nothing was.",
|
||||
"description": "CollectResult is what was deleted on one node, or why nothing was: what the node no longer lists once asked, not what it was asked to delete.",
|
||||
"properties": {
|
||||
"node": {
|
||||
"type": "string"
|
||||
@@ -6246,6 +6360,14 @@
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"left": {
|
||||
"type": "integer",
|
||||
"description": "Left and LeftBytes are what the node still lists: refused, or not gone by the time the sweep stopped waiting. Its agent's log says why."
|
||||
},
|
||||
"left_bytes": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"error": {
|
||||
"type": "string"
|
||||
}
|
||||
@@ -9362,6 +9484,15 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"SlurmKeysRequest": {
|
||||
"type": "object",
|
||||
"description": "SlurmKeysRequest is a person's SSH public keys, one a line, as authorized_keys has them; empty takes them all away.",
|
||||
"properties": {
|
||||
"keys": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"StoreMember": {
|
||||
"type": "object",
|
||||
"description": "StoreMember is one member of a replicated store, as its Patroni reports it.",
|
||||
@@ -9464,6 +9595,10 @@
|
||||
"guard_off": {
|
||||
"type": "boolean",
|
||||
"description": "GuardOff says the guardrails are turned off (guard.enabled: false): no request is checked, whatever the policies say."
|
||||
},
|
||||
"no_alerts": {
|
||||
"type": "boolean",
|
||||
"description": "NoAlerts says the cluster's settings name no webhook: a model parked, a node lost or a judge down is told nobody."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -10746,6 +10881,24 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"config.SharedStorage": {
|
||||
"type": "object",
|
||||
"description": "SharedStorage is where the Slurm cluster's shared files come from: the site's NFS export (NFS), an NFS export llmgw keeps on one of its nodes (Server), or, with neither, a file system the site mounts itself (Lustre, BeeGFS, GPFS), which llmgw only looks for.",
|
||||
"properties": {
|
||||
"Path": {
|
||||
"type": "string",
|
||||
"description": "Path is where every machine sees it, as /shared."
|
||||
},
|
||||
"NFS": {
|
||||
"type": "string",
|
||||
"description": "NFS is the site's export, host:/path, mounted at Path on every machine of the Slurm cluster."
|
||||
},
|
||||
"Server": {
|
||||
"type": "string",
|
||||
"description": "Server is the llmgw node, the controller or one of the nodes, that exports Path over NFS to the others: for a site with no storage of its own. One machine: not highly available, and no faster than its disk and network."
|
||||
}
|
||||
}
|
||||
},
|
||||
"control.Activity": {
|
||||
"type": "object",
|
||||
"description": "Activity is what a standby's router reports back: when it last sent each replica a request, and the on-demand models it was asked for. The leader evicts and loads by these, and without them a replica busy behind a standby looks idle to the one machine that decides.",
|
||||
@@ -11737,6 +11890,10 @@
|
||||
"context": {
|
||||
"type": "integer"
|
||||
},
|
||||
"trained": {
|
||||
"type": "integer",
|
||||
"description": "Trained is the context the model was trained for, the most a request may use; 0 when neither its header nor its config says."
|
||||
},
|
||||
"kv_bytes_per_token": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
@@ -12166,6 +12323,156 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"slurmsvc.NodeStatus": {
|
||||
"type": "object",
|
||||
"description": "NodeStatus is one node as the service sees it.",
|
||||
"properties": {
|
||||
"node": {
|
||||
"type": "string"
|
||||
},
|
||||
"phase": {
|
||||
"type": "string"
|
||||
},
|
||||
"replicas": {
|
||||
"type": "integer"
|
||||
},
|
||||
"stops_by": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "StopsBy is, while it is being lent, when a replica still on it is stopped even with nowhere else to go: the drain's grace after the lend began. Zero when not known (a leader that took over meanwhile)."
|
||||
},
|
||||
"slurm": {
|
||||
"$ref": "#/components/schemas/slurmsvc.Reported"
|
||||
},
|
||||
"connected": {
|
||||
"type": "boolean"
|
||||
}
|
||||
}
|
||||
},
|
||||
"slurmsvc.Reported": {
|
||||
"type": "object",
|
||||
"description": "Reported is the agent's SlurmStatus, for the admin API.",
|
||||
"properties": {
|
||||
"version": {
|
||||
"type": "string"
|
||||
},
|
||||
"hardware": {
|
||||
"type": "string"
|
||||
},
|
||||
"slurmd_running": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"slurmctld_running": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"slurmdbd_running": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"shared": {
|
||||
"type": "string",
|
||||
"description": "Shared is the shared path here: \"mounted\", \"exported\", \"missing\"."
|
||||
},
|
||||
"containers": {
|
||||
"type": "string",
|
||||
"description": "Containers is what the machine has for jobs in containers, empty while not both of pyxis and enroot."
|
||||
},
|
||||
"users": {
|
||||
"type": "integer",
|
||||
"description": "Users is how many of llmgw's users the machine has."
|
||||
},
|
||||
"state": {
|
||||
"type": "string"
|
||||
},
|
||||
"reason": {
|
||||
"type": "string"
|
||||
},
|
||||
"jobs": {
|
||||
"type": "integer"
|
||||
},
|
||||
"error": {
|
||||
"type": "string"
|
||||
},
|
||||
"files_sha256": {
|
||||
"type": "string"
|
||||
},
|
||||
"at": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
}
|
||||
}
|
||||
},
|
||||
"slurmsvc.Status": {
|
||||
"type": "object",
|
||||
"description": "Status is the whole Slurm cluster, for `ctl slurm status`.",
|
||||
"properties": {
|
||||
"enabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"cluster": {
|
||||
"type": "string"
|
||||
},
|
||||
"controller": {
|
||||
"type": "string"
|
||||
},
|
||||
"partition": {
|
||||
"type": "string"
|
||||
},
|
||||
"controller_status": {
|
||||
"$ref": "#/components/schemas/slurmsvc.Reported"
|
||||
},
|
||||
"nodes": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/slurmsvc.NodeStatus"
|
||||
}
|
||||
},
|
||||
"waiting": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Waiting are nodes left out of slurm.conf until they report their hardware."
|
||||
},
|
||||
"shared": {
|
||||
"$ref": "#/components/schemas/config.SharedStorage",
|
||||
"description": "Shared is where the shared files are and come from; Users who keeps the users, and how many llmgw has."
|
||||
},
|
||||
"users": {
|
||||
"type": "string"
|
||||
},
|
||||
"user_count": {
|
||||
"type": "integer"
|
||||
},
|
||||
"containers": {
|
||||
"type": "string",
|
||||
"description": "Containers is whether jobs may run in containers: \"on\" once every machine has pyxis and enroot, \"installing\" until then, empty when slurm.containers is off."
|
||||
}
|
||||
}
|
||||
},
|
||||
"slurmsvc.User": {
|
||||
"type": "object",
|
||||
"description": "User is a Linux user made from an llmgw account.",
|
||||
"properties": {
|
||||
"login": {
|
||||
"type": "string"
|
||||
},
|
||||
"uid": {
|
||||
"type": "integer"
|
||||
},
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"email": {
|
||||
"type": "string"
|
||||
},
|
||||
"ssh_keys": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"store.AuditEvent": {
|
||||
"type": "object",
|
||||
"description": "AuditEvent is one change: who made it, through what, to what, and how it ended. Detail never holds a secret; the writer takes them out.",
|
||||
|
||||
+17
-2
@@ -67,7 +67,8 @@ or with the mirror unticked, is said in the form and nothing starts.
|
||||
## Watch
|
||||
|
||||
**Overview** is `llmgw ctl status`: nodes, GPUs by class, fabrics, replicas,
|
||||
models, the store's members, and anything parked, unplaced or idle.
|
||||
models, the store's members, and anything parked, unplaced or idle. With no
|
||||
webhook in the cluster's settings it says the alerts go nowhere.
|
||||
|
||||
**Monitor** draws the cluster the way btop draws a machine: CPU, memory,
|
||||
GPU busy and GPU memory, requests and tokens a second, time to first token,
|
||||
@@ -207,6 +208,19 @@ starting — and whether it ended done, failed, or waiting for room on busy
|
||||
cards. The entry shows how many are running. Enter on one shows its steps.
|
||||
`llmgw ctl jobs` is the same, and `llmgw ctl jobs ID` follows one.
|
||||
|
||||
**Slurm** is the Slurm cluster run beside llmgw, when there is one: its
|
||||
controller and whether slurmctld runs, and each node that runs slurmd —
|
||||
whose it is now (llmgw's, Slurm's, or on its way between them), its state in
|
||||
Slurm, the jobs running there. On a row, **Lend to Slurm…** moves the node's
|
||||
replicas off and then lets Slurm run jobs there; **Take back from Slurm…**
|
||||
starts no new job there and gives it back to llmgw when its jobs have ended.
|
||||
Both change `slurm.lent` in the cluster's settings, as `llmgw ctl slurm lend`
|
||||
and `reclaim` do. Turning Slurm on and choosing its nodes is `llmgw ctl slurm
|
||||
setup`, or the Slurm part of the first install. Above the nodes it says where
|
||||
the shared files come from and how many users llmgw made; the SHARED column
|
||||
is each node's (mounted, exported, missing). `llmgw ctl slurm shared` and
|
||||
`llmgw ctl slurm users` change them.
|
||||
|
||||
**Logs** shows a node's agent log, newest at the bottom, errors and warnings
|
||||
coloured.
|
||||
|
||||
@@ -220,7 +234,8 @@ and change it with the same commands as `llmgw users`, `llmgw keys` and
|
||||
dashboard, and what they belong to. **A** creates an account, already
|
||||
approved. On a row: approve one waiting, suspend or reinstate, give or take
|
||||
the cluster dashboard, add to or take out of an organization, team or
|
||||
project, their limits, a new password (their sessions close), delete, and
|
||||
project (on an organization as a member, its manager, who sees its page, or
|
||||
its admin, who also changes it), their limits, a new password (their sessions close), delete, and
|
||||
**Forget them…** for a request to be forgotten (`llmgw users forget`).
|
||||
|
||||
**API keys** — every key, its project, owner, state and last use. **A**
|
||||
|
||||
+63
-2
@@ -166,13 +166,74 @@ is a flag too, and the plan shows the command with them: `--starter qwen3-8b`
|
||||
`--judge-fail open`, `--judge-categories`, `--judge-replicas`,
|
||||
`--judge-budget`, `--language tr`, `--signup-open`, `--default-rpm`,
|
||||
`--default-tpm`, `--max-key-ttl`, `--media` (pictures and 3D: ComfyUI on the
|
||||
nodes, and `presets.auto`), `--evict-after-idle` and `--min-lifetime`. They
|
||||
nodes, and `presets.auto`), `--slurm` (see below), `--evict-after-idle` and `--min-lifetime`. They
|
||||
go into `server.yaml` as init writes it, and become the cluster's first
|
||||
settings when it starts; a `server.yaml` already there is never overwritten,
|
||||
so run again they change nothing — the settings page does. An NVIDIA card with no working
|
||||
driver is reported rather than silently ignored; install the driver and run
|
||||
init again to serve on it.
|
||||
|
||||
**Slurm, for training and batch jobs.** `--slurm`, or the Slurm part on the
|
||||
choices screen, runs a Slurm cluster beside llmgw: slurmctld on this
|
||||
machine's node, slurmd on every node, those that join later too
|
||||
(`slurm.all_nodes`). The agents install munge and Slurm (22.05 or newer)
|
||||
from the machines' own package sources, and on this machine slurmdbd, which
|
||||
keeps the jobs' records (`sacct`), in a MariaDB installed beside it that
|
||||
answers this machine alone. Slurm keeps its records in MySQL or MariaDB
|
||||
only, not in the cluster's PostgreSQL; the database's password is made from
|
||||
the cluster's munge key, so nothing new is stored. A MariaDB the machine
|
||||
already had keeps its settings and only gets the database `slurm_acct_db`
|
||||
and its user. Back it up with the machine (`mariadb-dump slurm_acct_db`);
|
||||
records older than a year are purged. No node is Slurm's at first: each is
|
||||
drained in Slurm and serves models until `llmgw ctl slurm lend NODE` gives it
|
||||
to Slurm — its models move off first — and `llmgw ctl slurm reclaim NODE`
|
||||
takes it back once its jobs end. Jobs are submitted on this machine with
|
||||
`sbatch` and `srun`; `llmgw ctl slurm status` shows where each node is. It
|
||||
needs this machine to be a node, and its hostname to be a name Slurm takes.
|
||||
On a cluster installed without it, `llmgw ctl slurm setup --controller NODE
|
||||
--all-nodes` does the same later. Slurm's ports (6817–6819, and srun's 60001–63000) answer the cluster's
|
||||
networks only, as llmgw's own do; a firewall of the site's own between the
|
||||
nodes must let them through.
|
||||
|
||||
**Slurm's shared files and users.** Jobs need the same files on every
|
||||
machine: homes, datasets, output, checkpoints. With `--slurm` this machine
|
||||
exports `/shared` over NFS to every Slurm machine and they mount it
|
||||
(`slurm.shared`); NFS's port 2049 answers the cluster's networks only, the
|
||||
export the Slurm machines alone. That is one machine, not highly available
|
||||
and no faster than its disk and network: enough for a few nodes. A site with
|
||||
storage of its own uses it instead — `--slurm-shared HOST:/EXPORT` at
|
||||
install, or `llmgw ctl slurm shared --path /shared --nfs HOST:/EXPORT`
|
||||
later; Lustre, BeeGFS or GPFS the site mounts itself, and `llmgw ctl slurm
|
||||
shared --path /lustre/hpc` only has llmgw look that it is there. Every
|
||||
active llmgw account is a Linux user on every Slurm machine
|
||||
(`slurm.users: llmgw`): its login from its e-mail address, a UID from
|
||||
200000 up that is never given to anyone else, its home under
|
||||
`/shared/home`. People add their SSH public keys in the portal's
|
||||
**Profile** and log in to this machine to run `sbatch`; an account suspended
|
||||
or deleted loses its user, its files stay. On the other Slurm machines they
|
||||
are let in only while a job of theirs runs there, and their ssh session is
|
||||
put in that job, held to its cores and cards (pam_slurm_adopt, which the
|
||||
agents install); root and the machines' own users log in as before. A site whose users come from its
|
||||
directory (sssd against Active Directory or LDAP) passes `--slurm-users
|
||||
site`, or later `llmgw ctl slurm users off`.
|
||||
|
||||
Jobs may run in containers: `llmgw ctl slurm containers on` has the agents
|
||||
install enroot and pyxis on every Slurm machine (NVIDIA's releases, at
|
||||
versions llmgw pins and checks; pyxis is built against each machine's own
|
||||
Slurm) and, on GPU machines, nvidia-container-cli from NVIDIA's repository.
|
||||
Once every machine has them Slurm loads pyxis, and `ctl slurm status` says
|
||||
`containers: on`:
|
||||
|
||||
```sh
|
||||
srun --gres=gpu:1 --container-image=nvcr.io#nvidia/pytorch:25.01-py3 python train.py
|
||||
```
|
||||
|
||||
The shared path is in every container where it is on the host, and the job
|
||||
runs as its user, held to its cores and cards. Each node pulls the image
|
||||
from its registry for each job; for a large image used often, import it
|
||||
once (`enroot import -o /shared/images/pytorch.sqsh docker://nvcr.io#...`)
|
||||
and give that file to `--container-image`.
|
||||
|
||||
**Secure by default.** What init writes is the hardened configuration, not a
|
||||
demonstration one:
|
||||
|
||||
@@ -3145,7 +3206,7 @@ loaded at boot by `llmgw-firewall.service`):
|
||||
| Machine | Ports, for the cluster only |
|
||||
| --- | --- |
|
||||
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
|
||||
| node | engine ports (`control.port_range`, 20000–20999), 9100 |
|
||||
| node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6819 (slurmctld, slurmd, slurmdbd) and 60001–63000 (srun's `SrunPortRange`); exporting Slurm's shared files, 2049 (NFS) |
|
||||
| store member | 5432, 8008, 2379, 2380, 9100 |
|
||||
| mirror | 8081, 8082, 9100 |
|
||||
|
||||
|
||||
Binary file not shown.
+32
-11
@@ -192,22 +192,41 @@ auth:
|
||||
A personal key has no project, so it keeps no conversations or agent notes;
|
||||
those are kept per project.
|
||||
|
||||
### Organization managers
|
||||
### Organization managers and admins
|
||||
|
||||
A member of an organization can be made its manager:
|
||||
A member of an organization can be made its manager, who sees it, or its
|
||||
admin, who also changes it:
|
||||
|
||||
```
|
||||
llmgw users join boss@acme.com --org Acme --role manager
|
||||
llmgw users join lead@acme.com --org Acme --role admin
|
||||
```
|
||||
|
||||
or with the star on their organization membership on `/admin/users`. A manager
|
||||
gets an **Organization** link and a page at `/org`: usage over time by team,
|
||||
or with the role chosen on their organization membership on `/admin/users`.
|
||||
Both get an **Organization** link and a page at `/org`: usage over time by team,
|
||||
project or person, the teams and projects with their limits and spend, and every
|
||||
member. Only what was spent in that organization's projects is on it; what its
|
||||
people do elsewhere is not. The page is read-only — a manager is not an
|
||||
administrator of the cluster, and limits and memberships stay with those who
|
||||
are. An administrator following `/org` lands on the same organization's panel
|
||||
on the people page instead, where all of that can be changed.
|
||||
people do elsewhere is not.
|
||||
|
||||
A manager's page is read-only. An organization's admin also changes the
|
||||
organization from it:
|
||||
|
||||
- adds and renames its teams and projects, and removes an empty one;
|
||||
- adds a person by their sign-in address to the organization, a team or a
|
||||
project, and takes them out (their keys there are revoked with it);
|
||||
- makes a member of the organization a manager or an admin, or neither;
|
||||
- revokes a key spent in its projects.
|
||||
|
||||
What reaches past the organization stays with the cluster's administrators:
|
||||
its limits and budgets, approving or suspending an account (someone not yet
|
||||
approved cannot be added), the organization's own name, and anything else on
|
||||
the cluster. An admin cannot take away their own role. A group from single
|
||||
sign-on or Active Directory that puts someone in the organization does not, by
|
||||
saying so again at the next sign-in, take a manager's or admin's role away.
|
||||
|
||||
An administrator of the cluster following `/org` lands on the same
|
||||
organization's panel on the people page instead, where all of that can be
|
||||
changed.
|
||||
|
||||
`llmgw keys init` still works for provisioning scripts and is idempotent, but it
|
||||
will no longer move a team or project that already exists under another parent;
|
||||
@@ -517,8 +536,10 @@ included: `llmgw ctl audit`, or **Audit** in the console.
|
||||
suspends accounts from it; see [install.md](install.md#single-sign-on).
|
||||
Without them accounts are local, with local passwords.
|
||||
- **Staff roles are coarse.** `viewer`, `operator`, `billing` and `admin`
|
||||
(`llmgw users role`) decide what someone may see and do under /admin; an
|
||||
organization's manager can view it but not change it. Every change is in
|
||||
the audit log (`llmgw ctl audit`).
|
||||
(`llmgw users role`) decide what someone may see and do under /admin, and a
|
||||
page draws only the forms that person may send: a viewer sees no forms. An
|
||||
organization's manager can view it; its admin also changes its teams,
|
||||
projects, people and keys. Every change is in the audit log (`llmgw ctl
|
||||
audit`).
|
||||
- **Rate limits are per gateway instance.** With two instances each counts its own,
|
||||
so the effective limit doubles. v1 is single-instance by design.
|
||||
@@ -376,6 +376,7 @@ rather than repeated.
|
||||
| `mail` | `mail.Config` | [Server](#server) |
|
||||
| `serving` | `ServingSettings` | [Server](#server) |
|
||||
| `firewall` | `FirewallSettings` | [Server](#server) |
|
||||
| `slurm` | `SlurmSettings` | [Server](#server) |
|
||||
| `presets` | `PresetSettings` | [Server](#server) |
|
||||
| `on_demand` | `OnDemandSettings` | [Server](#server) |
|
||||
| `agents` | `AgentSettings` | [Server](#server) |
|
||||
@@ -400,7 +401,20 @@ rather than repeated.
|
||||
| `action` | `string` | [Service](#service) |
|
||||
| `timeout` | `time.Duration` | [Service](#service) |
|
||||
| `stopped` | `bool` | [ServingSettings](#servingsettings) |
|
||||
| `path` | `string` | [SharedStorage](#sharedstorage) |
|
||||
| `nfs` | `string` | [SharedStorage](#sharedstorage) |
|
||||
| `server` | `string` | [SharedStorage](#sharedstorage) |
|
||||
| `bin` | `string` | [SimConfig](#simconfig) |
|
||||
| `enabled` | `bool` | [SlurmSettings](#slurmsettings) |
|
||||
| `cluster_name` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `controller` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `nodes` | `[]string` | [SlurmSettings](#slurmsettings) |
|
||||
| `all_nodes` | `bool` | [SlurmSettings](#slurmsettings) |
|
||||
| `lent` | `[]string` | [SlurmSettings](#slurmsettings) |
|
||||
| `partition` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `shared` | `SharedStorage` | [SlurmSettings](#slurmsettings) |
|
||||
| `users` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `containers` | `bool` | [SlurmSettings](#slurmsettings) |
|
||||
| `dir` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `python` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `engine` | `string` | [StrataConfig](#strataconfig) |
|
||||
@@ -3279,6 +3293,15 @@ Firewall turns llmgw's firewall on every machine on or off.
|
||||
|
||||
Takes a [FirewallSettings](#firewallsettings) block.
|
||||
|
||||
### `slurm`
|
||||
|
||||
`SlurmSettings`
|
||||
|
||||
Slurm is a Slurm cluster run beside llmgw on some of its nodes, and
|
||||
which of them it may use now.
|
||||
|
||||
Takes a [SlurmSettings](#slurmsettings) block.
|
||||
|
||||
### `presets`
|
||||
|
||||
`PresetSettings`
|
||||
@@ -3471,6 +3494,30 @@ running and every model's definition as it is: llmgw stop sets it,
|
||||
llmgw start clears it, and the models come back as their catalog
|
||||
entries ask. In force on every control plane within seconds.
|
||||
|
||||
## SharedStorage
|
||||
|
||||
### `path`
|
||||
|
||||
`string`
|
||||
|
||||
Path is where every machine sees it, as /shared.
|
||||
|
||||
### `nfs`
|
||||
|
||||
`string`
|
||||
|
||||
NFS is the site's export, host:/path, mounted at Path on every
|
||||
machine of the Slurm cluster.
|
||||
|
||||
### `server`
|
||||
|
||||
`string`
|
||||
|
||||
Server is the llmgw node, the controller or one of the nodes, that
|
||||
exports Path over NFS to the others: for a site with no storage of
|
||||
its own. One machine: not highly available, and no faster than its
|
||||
disk and network.
|
||||
|
||||
## SimConfig
|
||||
|
||||
### `bin`
|
||||
@@ -3480,6 +3527,88 @@ entries ask. In force on every control plane within seconds.
|
||||
Bin is the llmgw binary itself; the engine is one of its subcommands.
|
||||
Empty means whatever started this agent, which is right almost always.
|
||||
|
||||
## SlurmSettings
|
||||
|
||||
### `enabled`
|
||||
|
||||
`bool`
|
||||
|
||||
Enabled runs Slurm on the nodes below. False stops llmgw keeping it;
|
||||
what was installed is left installed, and every node is llmgw's.
|
||||
|
||||
### `cluster_name`
|
||||
|
||||
`string`
|
||||
|
||||
ClusterName is Slurm's ClusterName. Empty is "llmgw".
|
||||
|
||||
### `controller`
|
||||
|
||||
`string`
|
||||
|
||||
Controller is the llmgw node whose agent runs slurmctld. It needs no
|
||||
GPU, and need not be one of Nodes.
|
||||
|
||||
### `nodes`
|
||||
|
||||
`[]string`
|
||||
|
||||
Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained
|
||||
in Slurm, until it is also in Lent.
|
||||
|
||||
### `all_nodes`
|
||||
|
||||
`bool`
|
||||
|
||||
AllNodes runs slurmd on every llmgw node, those that join later too,
|
||||
besides any named in Nodes. Each is llmgw's until it is lent, as any.
|
||||
|
||||
### `lent`
|
||||
|
||||
`[]string`
|
||||
|
||||
Lent are the nodes of Nodes Slurm may run jobs on now. A node put here
|
||||
is drained of llmgw's replicas first, as `ctl drain` does, and resumed
|
||||
in Slurm once none is left; a node taken out is drained in Slurm and
|
||||
given back to llmgw when its last job ends.
|
||||
|
||||
### `partition`
|
||||
|
||||
`string`
|
||||
|
||||
Partition is the one partition every node is in. Empty is "gpu".
|
||||
|
||||
### `shared`
|
||||
|
||||
`SharedStorage`
|
||||
|
||||
Shared is the file system every machine of the Slurm cluster sees at
|
||||
the same path: home directories, datasets, the jobs' output and
|
||||
checkpoints. Empty path: none, and a batch job's output stays on the
|
||||
node it ran on.
|
||||
|
||||
Takes a [SharedStorage](#sharedstorage) block.
|
||||
|
||||
### `users`
|
||||
|
||||
`string`
|
||||
|
||||
Users is who keeps the Linux users that run jobs, the same on every
|
||||
machine of the Slurm cluster: "llmgw" makes one for each active llmgw
|
||||
account (its login from the e-mail address, a UID of llmgw's from
|
||||
200000 up, its SSH keys from the portal), let in to the controller,
|
||||
and to a node only while a job of theirs runs there (pam_slurm_adopt);
|
||||
empty leaves users to the site (a directory through sssd, or by hand).
|
||||
|
||||
### `containers`
|
||||
|
||||
`bool`
|
||||
|
||||
Containers lets jobs run in containers: `srun --container-image
|
||||
nvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),
|
||||
the GPUs and the shared path inside. The agents install both on
|
||||
every Slurm machine; jobs take it once every machine has them.
|
||||
|
||||
## StrataConfig
|
||||
|
||||
### `dir`
|
||||
|
||||
+46
-6
@@ -35,6 +35,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/slurmnode"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
|
||||
@@ -97,7 +98,10 @@ type Agent struct {
|
||||
// (nil: this machine's shell).
|
||||
fwMu sync.Mutex
|
||||
fwRunner firewall.Runner
|
||||
inv discovery.Inventory
|
||||
// fwLast is the last firewall message, applied again when the node
|
||||
// is given a part in Slurm. Under fwMu.
|
||||
fwLast *llmgwv1.Firewall
|
||||
inv discovery.Inventory
|
||||
// shardReplica remembers which replica a shard belongs to, so a status
|
||||
// report can name both. The control plane keys on the replica.
|
||||
shardReplica map[string]string
|
||||
@@ -148,6 +152,10 @@ type Agent struct {
|
||||
|
||||
// trainer runs LoRA training jobs, one at a time (BACKLOG F4).
|
||||
trainer *train.Runner
|
||||
// slurm keeps the node's part in Slurm (BACKLOG P); slurmOut is where
|
||||
// its status goes, under mu.
|
||||
slurm *slurmnode.Keeper
|
||||
slurmOut chan<- *llmgwv1.AgentMessage
|
||||
}
|
||||
|
||||
// connectHTTPClient is the HTTP client connect uses; an interface so tests can
|
||||
@@ -206,6 +214,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
|
||||
}
|
||||
a.trainer = train.New(train.Env{EnginesDir: enginesDir, WorkDir: filepath.Join(cfg.Superv.WorkDir, "trainer"),
|
||||
RunAs: runAs, Log: log})
|
||||
a.slurm = slurmnode.New(slurmnode.Local{}, a.prov, slurmMarker, log)
|
||||
return a, nil
|
||||
}
|
||||
|
||||
@@ -232,6 +241,7 @@ func (a *Agent) Run(ctx context.Context) error {
|
||||
}
|
||||
go a.provisionLoop(ctx)
|
||||
go a.ensureDiags(ctx)
|
||||
go a.slurm.Run(ctx, a.sendSlurm)
|
||||
|
||||
attempt := 0
|
||||
next := 0
|
||||
@@ -530,6 +540,18 @@ func (a *Agent) scan(ctx context.Context) error {
|
||||
return err
|
||||
}
|
||||
|
||||
// forgetSent is a new stream: a control plane that knows nothing of where
|
||||
// installs stand, nor which files are here beyond what registration says —
|
||||
// and registration says what the startup scan found. So the next heartbeat
|
||||
// tells it both again. Without it, files deleted after the agent started came
|
||||
// back on a control plane that restarted, and stayed until the next change.
|
||||
func (a *Agent) forgetSent() {
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
a.provSent = ""
|
||||
a.lastWeights = ""
|
||||
}
|
||||
|
||||
// session runs one connection until it breaks.
|
||||
func (a *Agent) session(ctx context.Context, server string) error {
|
||||
client := llmgwv1connect.NewAgentServiceClient(a.http, server)
|
||||
@@ -566,14 +588,12 @@ func (a *Agent) session(ctx context.Context, server string) error {
|
||||
reg.EngineVersions = a.engines(parent)
|
||||
reg.EnginesProbed = true
|
||||
reg.InstallableEngines = slices.DeleteFunc(provision.Installable(), a.disabled)
|
||||
reg.Features = []string{FeatureFabricTest}
|
||||
reg.Features = []string{FeatureFabricTest, FeatureSlurm}
|
||||
reg.RunningShards = a.runningShards()
|
||||
a.mu.Lock()
|
||||
h := a.prov.Host
|
||||
// A new stream is a control plane that knows nothing of where installs
|
||||
// stand, so the next heartbeat tells it again.
|
||||
a.provSent = ""
|
||||
a.mu.Unlock()
|
||||
a.forgetSent()
|
||||
reg.Host = &llmgwv1.Host{
|
||||
OsId: h.OSID, OsVersion: h.OSVersion, OsCodename: h.OSCodename,
|
||||
Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver,
|
||||
@@ -618,6 +638,8 @@ func (a *Agent) session(ctx context.Context, server string) error {
|
||||
go a.watchExits(ctx, out, unready(reg.RunningShards))
|
||||
go a.forwardKV(ctx, out)
|
||||
go a.reportCompute(ctx, out)
|
||||
a.setSlurmOut(out)
|
||||
defer a.setSlurmOut(nil)
|
||||
|
||||
for {
|
||||
msg, err := stream.Receive()
|
||||
@@ -884,7 +906,23 @@ func (a *Agent) removeWeights(paths []string) {
|
||||
a.log.Warn("refused to remove a file llmgw did not fetch", "path", p)
|
||||
continue
|
||||
}
|
||||
if err := os.Remove(abs); err != nil && !os.IsNotExist(err) {
|
||||
// A set — every file of one version, fetched into a directory of its
|
||||
// own — goes whole: os.Remove refused it as not empty, and nothing
|
||||
// was freed of what a model of several files left. Lstat, so a link
|
||||
// is taken away and never what it points at.
|
||||
info, err := os.Lstat(abs)
|
||||
switch {
|
||||
case os.IsNotExist(err):
|
||||
continue
|
||||
case err != nil:
|
||||
a.log.Warn("could not look at cached weights", "path", p, "err", err)
|
||||
continue
|
||||
case info.IsDir():
|
||||
err = os.RemoveAll(abs)
|
||||
default:
|
||||
err = os.Remove(abs)
|
||||
}
|
||||
if err != nil {
|
||||
a.log.Warn("could not remove cached weights", "path", p, "err", err)
|
||||
continue
|
||||
}
|
||||
@@ -925,6 +963,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
|
||||
go a.applySettings(m.AgentSettings)
|
||||
case *llmgwv1.ControlMessage_TrainStart:
|
||||
go a.trainJob(ctx, m.TrainStart, out)
|
||||
case *llmgwv1.ControlMessage_Slurm:
|
||||
a.slurmMessage(m.Slurm)
|
||||
case *llmgwv1.ControlMessage_TrainCancel:
|
||||
a.trainer.Cancel(m.TrainCancel.GetJobId())
|
||||
case *llmgwv1.ControlMessage_RestartAgent:
|
||||
|
||||
@@ -925,3 +925,23 @@ func TestABandwidthTestIsRunOnTheAdapterAskedFor(t *testing.T) {
|
||||
t.Errorf("reply %+v, ran %+v", r, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANewStreamIsToldTheFilesAgain(t *testing.T) {
|
||||
// Registration carries what the startup scan found; a file deleted since
|
||||
// came back on a control plane that restarted, because the next
|
||||
// heartbeat saw the list unchanged since it was last sent.
|
||||
dir := t.TempDir()
|
||||
h := start(t, func(c *Config) { c.ModelDirs = []string{dir} })
|
||||
if err := os.WriteFile(filepath.Join(dir, "0123456789abcdef-m.gguf"), []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.ag.heartbeat()
|
||||
if hb := h.ag.heartbeat().GetHeartbeat(); hb.GetWeightsChanged() {
|
||||
t.Fatal("an unchanged list should not be sent again on the same stream")
|
||||
}
|
||||
h.ag.forgetSent()
|
||||
hb := h.ag.heartbeat().GetHeartbeat()
|
||||
if !hb.GetWeightsChanged() || len(hb.GetWeights()) != 1 {
|
||||
t.Errorf("a new stream should be told the files: changed=%v %v", hb.GetWeightsChanged(), hb.GetWeights())
|
||||
}
|
||||
}
|
||||
@@ -10,8 +10,20 @@ import (
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// slurmPorts are Slurm's on a node that runs any of it: slurmctld's and
|
||||
// slurmd's, and srun's range for the steps it starts.
|
||||
var slurmPorts = []firewall.Port{
|
||||
{From: slurm.CtldPort, To: slurm.DbdPort, What: "slurm: slurmctld, slurmd and slurmdbd"},
|
||||
{From: slurm.SrunPortsFrom, To: slurm.SrunPortsTo, What: "slurm: srun, for the steps it starts"},
|
||||
}
|
||||
|
||||
// nfsPort is NFSv4's, on the machine that exports the Slurm cluster's
|
||||
// shared files.
|
||||
var nfsPort = firewall.Port{From: slurm.NFSPort, To: slurm.NFSPort, What: "nfs: the Slurm cluster's shared files"}
|
||||
|
||||
// applyFirewall keeps llmgw's firewall on this node as the cluster's
|
||||
// settings say: the engines' ports and the node exporter's for the networks
|
||||
// the node reaches its control planes on, and those named, or no table at
|
||||
@@ -19,6 +31,7 @@ import (
|
||||
func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
|
||||
a.fwMu.Lock()
|
||||
defer a.fwMu.Unlock()
|
||||
a.fwLast = m
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
defer cancel()
|
||||
r := a.fwRunner
|
||||
@@ -55,6 +68,14 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
|
||||
if from > 0 && to >= from {
|
||||
ports = append(ports, firewall.Engines(from, to))
|
||||
}
|
||||
// Once there, they stay: the ruleset keeps every port it held, so a
|
||||
// node that leaves Slurm keeps them closed to everyone but the cluster.
|
||||
if a.slurm != nil && a.slurm.HasPart() {
|
||||
ports = append(ports, slurmPorts...)
|
||||
}
|
||||
if a.slurm != nil && a.slurm.Serves() {
|
||||
ports = append(ports, nfsPort)
|
||||
}
|
||||
changed, err := firewall.Install(ctx, r, firewall.Spec{Ports: ports, Allow: nets})
|
||||
switch {
|
||||
case err != nil:
|
||||
@@ -64,6 +85,23 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
|
||||
}
|
||||
}
|
||||
|
||||
// slurmMessage is the leader's word on this node's part in Slurm. A node
|
||||
// that gets one, or starts or stops exporting the shared files, has the
|
||||
// firewall applied again, with Slurm's ports and NFS's.
|
||||
func (a *Agent) slurmMessage(m *llmgwv1.Slurm) {
|
||||
had, served := a.slurm.HasPart(), a.slurm.Serves()
|
||||
a.slurm.Apply(m)
|
||||
if (had || !a.slurm.HasPart()) && served == a.slurm.Serves() {
|
||||
return
|
||||
}
|
||||
a.fwMu.Lock()
|
||||
last := a.fwLast
|
||||
a.fwMu.Unlock()
|
||||
if last != nil && last.GetEnabled() {
|
||||
go a.applyFirewall(last)
|
||||
}
|
||||
}
|
||||
|
||||
// controlPlaneIPs are the addresses of the control planes an agent dials.
|
||||
func controlPlaneIPs(ctx context.Context, servers []string) []string {
|
||||
var out []string
|
||||
|
||||
@@ -6,8 +6,10 @@ import (
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/slurmnode"
|
||||
)
|
||||
|
||||
// fakeShell answers the firewall package's scripts and records what was
|
||||
@@ -55,3 +57,28 @@ func TestANodeKeepsOrDropsItsFirewallAsTheClusterSays(t *testing.T) {
|
||||
t.Error("turned off, the table stayed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeGivenAPartInSlurmLetsSlurmInFromTheClusterOnly(t *testing.T) {
|
||||
sh := &fakeShell{}
|
||||
a := &Agent{log: slog.New(slog.DiscardHandler), fwRunner: sh, serverList: []string{"https://10.0.0.140:7443"},
|
||||
slurm: slurmnode.New(nil, nil, "", slog.New(slog.DiscardHandler))}
|
||||
a.applyFirewall(&llmgwv1.Firewall{Enabled: true, EnginePortsFrom: 20000, EnginePortsTo: 20999})
|
||||
if strings.Contains(sh.installed, "6817") {
|
||||
t.Fatalf("Slurm's ports before the node has a part in it:\n%s", sh.installed)
|
||||
}
|
||||
|
||||
a.slurmMessage(&llmgwv1.Slurm{Name: "node05", Node: true, ConfServer: "10.0.0.148:6817"})
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
a.fwMu.Lock()
|
||||
got := sh.installed
|
||||
a.fwMu.Unlock()
|
||||
if strings.Contains(got, "6817-6819") && strings.Contains(got, "60001-63000") {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("given a part in Slurm, the rules still lack its ports:\n%s", got)
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
@@ -164,6 +164,13 @@ func (a *Agent) provisionLoop(ctx context.Context) {
|
||||
|
||||
// provisionStatus records a change in one engine's installation.
|
||||
func (a *Agent) provisionStatus(st provision.Status) {
|
||||
if st == (provision.Status{}) {
|
||||
// A package install that was no engine's has ended.
|
||||
a.mu.Lock()
|
||||
delete(a.provStatus, "")
|
||||
a.mu.Unlock()
|
||||
return
|
||||
}
|
||||
a.mu.Lock()
|
||||
prev, had := a.provStatus[st.Engine]
|
||||
a.provStatus[st.Engine] = st
|
||||
|
||||
@@ -483,7 +483,8 @@ func (in *Installer) packages(ctx context.Context, byFamily map[string][]string,
|
||||
return fmt.Errorf("%s is missing and no package manager was found to install it",
|
||||
strings.Join(missing, ", "))
|
||||
}
|
||||
in.progress(st, "installing %s with %s", strings.Join(missing, ", "), tool)
|
||||
// The state already says installing.
|
||||
in.progress(st, "%s, with %s", strings.Join(missing, ", "), tool)
|
||||
for _, c := range cmds {
|
||||
if err := in.runWaitingForLock(ctx, pm.env, c, st); err != nil {
|
||||
return err
|
||||
@@ -497,7 +498,13 @@ func (in *Installer) packages(ctx context.Context, byFamily map[string][]string,
|
||||
// byFamily is keyed by discovery.Host.Family: "deb" or "rpm".
|
||||
func (in *Installer) InstallPackages(ctx context.Context, byFamily map[string][]string) error {
|
||||
var st Status
|
||||
return in.packages(ctx, byFamily, &st)
|
||||
err := in.packages(ctx, byFamily, &st)
|
||||
if st.State == Installing {
|
||||
// Its progress was no engine's: an empty status takes it off the
|
||||
// node's page again, done or failed (the caller says which).
|
||||
in.report(Status{})
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// lockWait bounds how long a package command waits for another to finish.
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
package provision
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
// A package install that is no engine's says it is done, so its progress
|
||||
// does not stay on the node's page (found on the PoC installing Slurm).
|
||||
func TestInstallPackagesClearsItsProgress(t *testing.T) {
|
||||
var got []Status
|
||||
in := &Installer{Host: discovery.Host{OSID: "ubuntu"}, OnStatus: func(s Status) { got = append(got, s) },
|
||||
Run: func(_ context.Context, _ []string, name string, _ ...string) ([]byte, error) {
|
||||
if name == "sh" { // the query: not installed
|
||||
return nil, context.Canceled
|
||||
}
|
||||
return nil, nil
|
||||
}}
|
||||
if err := in.InstallPackages(context.Background(), map[string][]string{"deb": {"munge"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) < 2 || got[0].State != Installing || got[len(got)-1] != (Status{}) {
|
||||
t.Fatalf("statuses %+v", got)
|
||||
}
|
||||
|
||||
got = nil
|
||||
in.Run = func(context.Context, []string, string, ...string) ([]byte, error) { return nil, nil }
|
||||
if err := in.InstallPackages(context.Background(), map[string][]string{"deb": {"munge"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("installed already, yet said %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -94,3 +94,35 @@ func TestWithNoCacheConfiguredNothingIsDeleted(t *testing.T) {
|
||||
t.Errorf("an agent with no cache directory deleted something: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A model of several files is fetched into a directory of its own, and an
|
||||
// unused one goes whole: it was refused as not empty, and nothing was freed.
|
||||
func TestAnUnusedSetIsRemovedWhole(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
a := agentWithCache(t, dir)
|
||||
set := weights.CachePathFor(dir, "https://mirror/coder/v1")
|
||||
if err := os.MkdirAll(set, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
write(t, filepath.Join(set, "model-00001-of-00002.gguf"))
|
||||
write(t, filepath.Join(set, ".complete"))
|
||||
// A link in the cache is taken away, not what it points at.
|
||||
target := t.TempDir()
|
||||
write(t, filepath.Join(target, "keep.gguf"))
|
||||
link := weights.CachePathFor(dir, "https://mirror/linked/v1")
|
||||
if err := os.Symlink(target, link); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
a.removeWeights([]string{set, link})
|
||||
|
||||
if _, err := os.Stat(set); !os.IsNotExist(err) {
|
||||
t.Errorf("the set is still there: %v", err)
|
||||
}
|
||||
if _, err := os.Lstat(link); !os.IsNotExist(err) {
|
||||
t.Errorf("the link is still there: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(target, "keep.gguf")); err != nil {
|
||||
t.Errorf("what the link pointed at was deleted: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
)
|
||||
|
||||
// FeatureSlurm is the agent keeping its node's part in Slurm (BACKLOG P).
|
||||
const FeatureSlurm = "slurm"
|
||||
|
||||
// slurmMarker says llmgw started Slurm's daemons on this machine.
|
||||
var slurmMarker = "/var/lib/llmgw-agent/slurm"
|
||||
|
||||
// setSlurmOut is where Slurm's status goes now: this session's queue, or
|
||||
// nowhere between sessions.
|
||||
func (a *Agent) setSlurmOut(out chan<- *llmgwv1.AgentMessage) {
|
||||
a.mu.Lock()
|
||||
a.slurmOut = out
|
||||
a.mu.Unlock()
|
||||
if out != nil {
|
||||
a.slurm.Resend()
|
||||
}
|
||||
}
|
||||
|
||||
// sendSlurm queues a status without waiting: a full queue or no session
|
||||
// drops it, and the next session is sent it again.
|
||||
func (a *Agent) sendSlurm(st *llmgwv1.SlurmStatus) {
|
||||
a.mu.Lock()
|
||||
out := a.slurmOut
|
||||
a.mu.Unlock()
|
||||
if out == nil {
|
||||
return
|
||||
}
|
||||
select {
|
||||
case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_SlurmStatus{SlurmStatus: st}}:
|
||||
default:
|
||||
a.log.Warn("Slurm's status not sent: the queue to the control plane is full")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,198 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// cacheDir holds what is fetched for Slurm: enroot's packages, pyxis's
|
||||
// source and its build.
|
||||
const cacheDir = "/var/cache/llmgw-agent/slurm"
|
||||
|
||||
// pyxisMarker is the pyxis version built here, against this Slurm.
|
||||
func (k *Keeper) pyxisMarker() string { return k.marker + "-pyxis" }
|
||||
|
||||
// containers has enroot and pyxis here, set up, when want says so; what
|
||||
// is there is left when it no longer does (Slurm stops loading pyxis once
|
||||
// the controller's plugstack.conf goes). Answers the status line, empty
|
||||
// while either is missing.
|
||||
func (k *Keeper) containers(ctx context.Context, want *llmgwv1.Slurm) (string, error) {
|
||||
if !want.GetContainers() {
|
||||
return "", nil
|
||||
}
|
||||
family := k.family()
|
||||
if family == "" {
|
||||
return "", fmt.Errorf("neither dpkg nor rpm here: enroot and pyxis come as deb or rpm")
|
||||
}
|
||||
if err := k.enroot(ctx, family); err != nil {
|
||||
return "", fmt.Errorf("enroot: %w", err)
|
||||
}
|
||||
if err := k.enrootConf(ctx, want.GetShared().GetPath()); err != nil {
|
||||
return "", fmt.Errorf("enroot's settings: %w", err)
|
||||
}
|
||||
if _, err := k.host.LookPath("nvidia-smi"); err == nil {
|
||||
if err := k.nvidiaContainer(ctx, family); err != nil {
|
||||
return "", fmt.Errorf("the GPUs in containers (nvidia-container-cli): %w", err)
|
||||
}
|
||||
}
|
||||
if err := k.pyxis(ctx, family); err != nil {
|
||||
return "", fmt.Errorf("pyxis: %w", err)
|
||||
}
|
||||
return slurm.ContainersStatus(), nil
|
||||
}
|
||||
|
||||
func (k *Keeper) family() string {
|
||||
if _, err := k.host.LookPath("dpkg"); err == nil {
|
||||
return "deb"
|
||||
}
|
||||
if _, err := k.host.LookPath("rpm"); err == nil {
|
||||
return "rpm"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// enroot installs the pinned release when another, or none, is here.
|
||||
func (k *Keeper) enroot(ctx context.Context, family string) error {
|
||||
if k.installed["enroot"] {
|
||||
return nil
|
||||
}
|
||||
if out, err := k.host.Run(ctx, nil, "enroot", "version"); err == nil && strings.TrimSpace(string(out)) == slurm.EnrootVersion {
|
||||
k.installed["enroot"] = true
|
||||
return nil
|
||||
}
|
||||
pkgs, err := slurm.EnrootPackages(family, runtime.GOARCH)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("enroot missing, and this agent installs nothing")
|
||||
}
|
||||
var files []string
|
||||
for _, p := range pkgs {
|
||||
f := filepath.Join(cacheDir, p.Name)
|
||||
if err := k.host.Download(ctx, p.URL(), f, p.SHA256); err != nil {
|
||||
return err
|
||||
}
|
||||
files = append(files, f)
|
||||
}
|
||||
k.log.Info("installing enroot, for Slurm's jobs in containers", "version", slurm.EnrootVersion)
|
||||
if err := k.inst.InstallPackages(ctx, map[string][]string{family: files}); err != nil {
|
||||
return err
|
||||
}
|
||||
out, err := k.host.Run(ctx, nil, "enroot", "version")
|
||||
if v := strings.TrimSpace(string(out)); err != nil || v != slurm.EnrootVersion {
|
||||
return fmt.Errorf("enroot says %q after installing %s: %v", v, slurm.EnrootVersion, err)
|
||||
}
|
||||
k.installed["enroot"] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// enrootConf writes enroot's settings and makes its directories.
|
||||
func (k *Keeper) enrootConf(ctx context.Context, shared string) error {
|
||||
if _, err := k.write(slurm.EnrootConfPath, []byte(slurm.EnrootConf), 0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
changed, err := k.write(slurm.EnrootTmpfiles, []byte(slurm.EnrootTmpfilesConf), 0o644, "")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if changed {
|
||||
if out, err := k.host.Run(ctx, nil, "systemd-tmpfiles", "--create", slurm.EnrootTmpfiles); err != nil {
|
||||
return fmt.Errorf("systemd-tmpfiles: %v: %s", err, lastLine(out))
|
||||
}
|
||||
}
|
||||
for _, d := range []string{slurm.EnrootLib + "/cache", slurm.EnrootLib + "/data"} {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-d", d); err == nil {
|
||||
continue
|
||||
}
|
||||
// Each user makes their own beneath, as in /tmp.
|
||||
if out, err := k.host.Run(ctx, nil, "install", "-d", "-m", "1777", d); err != nil {
|
||||
return fmt.Errorf("making %s: %v: %s", d, err, lastLine(out))
|
||||
}
|
||||
}
|
||||
if shared == "" {
|
||||
return k.host.Remove(slurm.EnrootMounts)
|
||||
}
|
||||
_, err = k.write(slurm.EnrootMounts, []byte(slurm.EnrootSharedMount(shared)), 0o644, "")
|
||||
return err
|
||||
}
|
||||
|
||||
// nvidiaContainer installs nvidia-container-cli, which enroot's hook uses
|
||||
// to give a container the job's GPUs, from NVIDIA's repository.
|
||||
func (k *Keeper) nvidiaContainer(ctx context.Context, family string) error {
|
||||
if k.installed["nvidia-container-cli"] {
|
||||
return nil
|
||||
}
|
||||
if _, err := k.host.LookPath("nvidia-container-cli"); err == nil {
|
||||
k.installed["nvidia-container-cli"] = true
|
||||
return nil
|
||||
}
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("nvidia-container-cli missing, and this agent installs nothing")
|
||||
}
|
||||
switch family {
|
||||
case "deb":
|
||||
if err := k.host.Download(ctx, slurm.NvidiaContainerKeyURL, slurm.NvidiaContainerKeyPath,
|
||||
slurm.NvidiaContainerKeySHA256); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := k.write(slurm.NvidiaContainerAptList, []byte(slurm.NvidiaContainerApt), 0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
case "rpm":
|
||||
if _, err := k.write(slurm.NvidiaContainerYumRepo, []byte(slurm.NvidiaContainerYum), 0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
k.log.Info("installing nvidia-container-cli, so containers get the job's GPUs")
|
||||
if err := k.inst.InstallPackages(ctx, slurm.NvidiaContainerPackages); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := k.host.LookPath("nvidia-container-cli"); err != nil {
|
||||
return fmt.Errorf("still missing after installing libnvidia-container-tools")
|
||||
}
|
||||
k.installed["nvidia-container-cli"] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// pyxis builds the pinned pyxis against this machine's Slurm and installs
|
||||
// it, when it has not been, or was for another version.
|
||||
func (k *Keeper) pyxis(ctx context.Context, family string) error {
|
||||
if b, err := k.host.ReadFile(k.pyxisMarker()); err == nil && strings.TrimSpace(string(b)) == slurm.PyxisVersion {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-f", slurm.PyxisPlugin); err == nil {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("pyxis missing, and this agent installs nothing")
|
||||
}
|
||||
if err := k.inst.InstallPackages(ctx, map[string][]string{family: slurm.PyxisBuildPackages[family]}); err != nil {
|
||||
return fmt.Errorf("installing what builds it: %w", err)
|
||||
}
|
||||
tarball := filepath.Join(cacheDir, "pyxis-"+slurm.PyxisVersion+".tar.gz")
|
||||
if err := k.host.Download(ctx, slurm.PyxisSource, tarball, slurm.PyxisSHA256); err != nil {
|
||||
return err
|
||||
}
|
||||
src := filepath.Join(cacheDir, "pyxis-"+slurm.PyxisVersion)
|
||||
k.log.Info("building pyxis against this machine's Slurm", "version", slurm.PyxisVersion)
|
||||
for _, c := range [][]string{
|
||||
{"rm", "-rf", src},
|
||||
{"tar", "-xzf", tarball, "-C", cacheDir},
|
||||
{"make", "-C", src, "install"},
|
||||
} {
|
||||
if out, err := k.host.Run(ctx, nil, c[0], c[1:]...); err != nil {
|
||||
return fmt.Errorf("%s: %v: %s", strings.Join(c, " "), err, lastLine(out))
|
||||
}
|
||||
}
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-f", slurm.PyxisPlugin); err != nil {
|
||||
return fmt.Errorf("built, yet %s is not there", slurm.PyxisPlugin)
|
||||
}
|
||||
_, err := k.write(k.pyxisMarker(), []byte(slurm.PyxisVersion+"\n"), 0o644, "")
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"runtime"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
func TestContainersInstalledFromPinnedReleases(t *testing.T) {
|
||||
if _, err := slurm.EnrootPackages("deb", runtime.GOARCH); err != nil {
|
||||
t.Skip(err)
|
||||
}
|
||||
h := newFake()
|
||||
h.bins["dpkg"], h.bins["nvidia-smi"] = true, true
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
// enroot is not here until installed, and then says its version.
|
||||
h.fail["enroot version"] = true
|
||||
h.fail["test -d "+slurm.EnrootLib+"/cache"] = true
|
||||
h.fail["test -f "+slurm.PyxisPlugin] = true
|
||||
m := nodeMsg(true)
|
||||
m.Containers = true
|
||||
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Nfs: "10.0.0.11:/shared"}
|
||||
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
if st.GetContainers() != "" || !strings.Contains(st.GetError(), "containers: enroot") {
|
||||
t.Fatalf("ready with enroot saying nothing: %v", st)
|
||||
}
|
||||
pkgs, _ := slurm.EnrootPackages("deb", runtime.GOARCH)
|
||||
for _, p := range pkgs {
|
||||
if !slices.Contains(h.ran, "download "+p.URL()+" "+cacheDir+"/"+p.Name+" "+p.SHA256) {
|
||||
t.Errorf("%s not fetched with its digest:\n%s", p.Name, strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
if len(inst.got["deb"]) != 2 || !strings.HasPrefix(inst.got["deb"][0], cacheDir) {
|
||||
t.Errorf("installed %v", inst.got)
|
||||
}
|
||||
|
||||
delete(h.fail, "enroot version")
|
||||
h.outputs["enroot version"] = slurm.EnrootVersion + "\n"
|
||||
h.bins["nvidia-container-cli"] = false
|
||||
st = k.Reconcile(context.Background(), m)
|
||||
if !strings.Contains(st.GetError(), "nvidia-container-cli") {
|
||||
t.Errorf("a GPU machine without nvidia-container-cli is ready: %v", st)
|
||||
}
|
||||
if string(h.files[slurm.NvidiaContainerAptList]) != slurm.NvidiaContainerApt ||
|
||||
!slices.Contains(h.ran, "download "+slurm.NvidiaContainerKeyURL+" "+slurm.NvidiaContainerKeyPath+" "+slurm.NvidiaContainerKeySHA256) {
|
||||
t.Errorf("NVIDIA's repository not added with its key checked")
|
||||
}
|
||||
|
||||
h.bins["nvidia-container-cli"] = true
|
||||
st = k.Reconcile(context.Background(), m)
|
||||
// Built, but the plugin is still not there.
|
||||
if st.GetContainers() != "" || !strings.Contains(st.GetError(), "pyxis") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
for _, want := range []string{"tar -xzf " + cacheDir + "/pyxis-" + slurm.PyxisVersion + ".tar.gz -C " + cacheDir,
|
||||
"make -C " + cacheDir + "/pyxis-" + slurm.PyxisVersion + " install", "install -d -m 1777 " + slurm.EnrootLib + "/cache"} {
|
||||
if !slices.Contains(h.ran, want) {
|
||||
t.Errorf("did not run %q", want)
|
||||
}
|
||||
}
|
||||
delete(h.fail, "test -f "+slurm.PyxisPlugin)
|
||||
st = k.Reconcile(context.Background(), m)
|
||||
if st.GetContainers() != slurm.ContainersStatus() || st.GetError() != "" {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if string(h.files[slurm.EnrootConfPath]) != slurm.EnrootConf ||
|
||||
!strings.Contains(string(h.files[slurm.EnrootMounts]), "/shared /shared none") {
|
||||
t.Errorf("enroot's settings not written")
|
||||
}
|
||||
|
||||
// Done: nothing fetched or built again.
|
||||
h.ran = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
for _, c := range h.ran {
|
||||
if strings.HasPrefix(c, "download") || strings.HasPrefix(c, "make") {
|
||||
t.Errorf("ran %q again", c)
|
||||
}
|
||||
}
|
||||
|
||||
// Off: nothing said, nothing taken away.
|
||||
m.Containers = false
|
||||
if st := k.Reconcile(context.Background(), m); st.GetContainers() != "" {
|
||||
t.Errorf("status %v", st)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// mariaDBConfDirs are where each family's MariaDB reads its settings: Debian
|
||||
// and Ubuntu's, then Red Hat's.
|
||||
var mariaDBConfDirs = []string{"/etc/mysql/mariadb.conf.d", "/etc/my.cnf.d"}
|
||||
|
||||
const mariaDBConfName = "90-llmgw-slurm.cnf"
|
||||
|
||||
// acctSQLPath is where the database's statements are put for the client to
|
||||
// read, never on its command line: they hold the password.
|
||||
const acctSQLPath = "/run/llmgw-slurm-acct.sql"
|
||||
|
||||
// accounting keeps slurmdbd on the controller: MariaDB on this machine
|
||||
// alone, the database and its user, slurmdbd on them, and the cluster in it.
|
||||
// Before slurmctld, which reports to it from its first start.
|
||||
func (k *Keeper) accounting(ctx context.Context, key []byte, files map[string]string) error {
|
||||
password := slurm.DbdPassword(key)
|
||||
changed, err := k.mariaDBSettings(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch {
|
||||
case changed:
|
||||
k.log.Info("MariaDB, for Slurm's accounting, starts on this machine alone")
|
||||
if err := k.systemctl(ctx, "enable", "--now", "mariadb", "&&restart"); err != nil {
|
||||
return err
|
||||
}
|
||||
case !k.active(ctx, "mariadb"):
|
||||
if err := k.systemctl(ctx, "enable", "--now", "mariadb"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
client, err := k.mariaDBClient()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := k.waitFor(ctx, "MariaDB", client, "-e", "SELECT 1"); err != nil {
|
||||
return err
|
||||
}
|
||||
// The database and its user, once each time the agent starts or the
|
||||
// password is new: cheap, and a database restored from elsewhere gets
|
||||
// its user back.
|
||||
sum := sha256.Sum256([]byte(password))
|
||||
if mark := hex.EncodeToString(sum[:]); k.acctDone != mark {
|
||||
if err := k.host.WriteFile(acctSQLPath, []byte(slurm.AcctSQL(password)), 0o600, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
out, err := k.host.Run(ctx, nil, client, "-e", "source "+acctSQLPath)
|
||||
_ = k.host.Remove(acctSQLPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("making Slurm's database: %v: %s", err, lastLine(out))
|
||||
}
|
||||
k.acctDone = mark
|
||||
}
|
||||
|
||||
out, err := k.host.Run(ctx, nil, "hostname", "-s")
|
||||
host := strings.TrimSpace(string(out))
|
||||
if err != nil || host == "" {
|
||||
return fmt.Errorf("this machine's short hostname: %v: %s", err, lastLine(out))
|
||||
}
|
||||
changed, err = k.write(slurm.DbdConfPath, []byte(slurm.DbdConf(host, password)), 0o600, "slurm")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch {
|
||||
case changed:
|
||||
k.log.Info("slurmdbd starts again on new settings")
|
||||
if err := k.systemctl(ctx, "enable", "--now", "slurmdbd", "&&restart"); err != nil {
|
||||
return err
|
||||
}
|
||||
case !k.active(ctx, "slurmdbd"):
|
||||
if err := k.systemctl(ctx, "enable", "--now", "slurmdbd"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := k.waitFor(ctx, "slurmdbd", "sacctmgr", "-n", "-P", "show", "cluster", "format=cluster"); err != nil {
|
||||
return err
|
||||
}
|
||||
name := clusterName(files["slurm.conf"])
|
||||
if name == "" || k.clusterAdded == name {
|
||||
return nil
|
||||
}
|
||||
out, err = k.host.Run(ctx, nil, "sacctmgr", "-n", "-P", "show", "cluster", "format=cluster")
|
||||
if err != nil {
|
||||
return fmt.Errorf("sacctmgr show cluster: %v: %s", err, lastLine(out))
|
||||
}
|
||||
for _, l := range strings.Split(string(out), "\n") {
|
||||
if strings.TrimSpace(l) == name {
|
||||
k.clusterAdded = name
|
||||
return nil
|
||||
}
|
||||
}
|
||||
k.log.Info("the cluster is added to Slurm's accounting", "cluster", name)
|
||||
if out, err := k.host.Run(ctx, nil, "sacctmgr", "-i", "add", "cluster", name); err != nil {
|
||||
return fmt.Errorf("sacctmgr add cluster %s: %v: %s", name, err, lastLine(out))
|
||||
}
|
||||
k.clusterAdded = name
|
||||
return nil
|
||||
}
|
||||
|
||||
// mariaDBSettings keeps MariaDB to this machine, when llmgw installed it;
|
||||
// one that was here before llmgw keeps its own settings, and only gets the
|
||||
// database and its user.
|
||||
func (k *Keeper) mariaDBSettings(ctx context.Context) (bool, error) {
|
||||
if _, err := k.host.ReadFile(k.mariaDBMarker()); err != nil {
|
||||
return false, nil
|
||||
}
|
||||
for _, d := range mariaDBConfDirs {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-d", d); err == nil {
|
||||
return k.write(d+"/"+mariaDBConfName, []byte(slurm.MariaDBConf), 0o644, "")
|
||||
}
|
||||
}
|
||||
return false, fmt.Errorf("MariaDB has none of %s to put its settings in", strings.Join(mariaDBConfDirs, ", "))
|
||||
}
|
||||
|
||||
// mariaDBMarker is there when llmgw installed MariaDB on this machine.
|
||||
func (k *Keeper) mariaDBMarker() string { return k.marker + "-mariadb" }
|
||||
|
||||
// mariaDBClient is MariaDB's command-line client, under either of its
|
||||
// names.
|
||||
func (k *Keeper) mariaDBClient() (string, error) {
|
||||
for _, c := range []string{"mariadb", "mysql"} {
|
||||
if _, err := k.host.LookPath(c); err == nil {
|
||||
return c, nil
|
||||
}
|
||||
}
|
||||
return "", errors.New("MariaDB's client (mariadb or mysql) is not installed")
|
||||
}
|
||||
|
||||
// waitFor runs a command until it succeeds, half a minute at most.
|
||||
func (k *Keeper) waitFor(ctx context.Context, what, name string, args ...string) error {
|
||||
var last []byte
|
||||
for range 30 {
|
||||
out, err := k.host.Run(ctx, nil, name, args...)
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
last = out
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(pingWait):
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("%s did not answer after starting: %s", what, lastLine(last))
|
||||
}
|
||||
|
||||
// clusterName is ClusterName in slurm.conf.
|
||||
func clusterName(conf string) string {
|
||||
sc := bufio.NewScanner(strings.NewReader(conf))
|
||||
for sc.Scan() {
|
||||
if v, ok := strings.CutPrefix(strings.TrimSpace(sc.Text()), "ClusterName="); ok {
|
||||
return strings.TrimSpace(v)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,150 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// What a machine says of the shared path.
|
||||
const (
|
||||
SharedMounted = "mounted"
|
||||
SharedExported = "exported"
|
||||
SharedMissing = "missing"
|
||||
)
|
||||
|
||||
// shared keeps the shared path as want says: exported from here, mounted
|
||||
// from an export, or looked for where the site mounts it. Answers how it
|
||||
// is.
|
||||
func (k *Keeper) shared(ctx context.Context, want *llmgwv1.Slurm) (string, error) {
|
||||
sh := want.GetShared()
|
||||
path := sh.GetPath()
|
||||
k.unshare(ctx, path, sh.GetServe(), sh.GetNfs())
|
||||
if path == "" {
|
||||
return "", nil
|
||||
}
|
||||
if !strings.HasPrefix(path, "/") || strings.Contains(path, "..") || strings.ContainsAny(path, " \n\t\"'\\$`") {
|
||||
return SharedMissing, fmt.Errorf("the leader sent a path %q", path)
|
||||
}
|
||||
switch {
|
||||
case sh.GetServe():
|
||||
if err := k.host.MkdirAll(path, 0o755, ""); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
changed, err := k.write(slurm.ExportsPath, []byte(slurm.Exports(path, sh.GetClients())), 0o644, "")
|
||||
if err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
if !k.active(ctx, "nfs-server") {
|
||||
k.log.Info("NFS starts on this machine, exporting the Slurm cluster's shared files", "path", path)
|
||||
if err := k.systemctl(ctx, "enable", "--now", "nfs-server"); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
changed = true
|
||||
}
|
||||
if changed {
|
||||
if out, err := k.host.Run(ctx, nil, "exportfs", "-ra"); err != nil {
|
||||
return SharedMissing, fmt.Errorf("exportfs -ra: %v: %s", err, lastLine(out))
|
||||
}
|
||||
}
|
||||
return SharedExported, nil
|
||||
case sh.GetNfs() != "":
|
||||
nfs := sh.GetNfs()
|
||||
if strings.ContainsAny(nfs, " \n\t\"'\\$`") || !strings.Contains(nfs, ":/") {
|
||||
return SharedMissing, fmt.Errorf("the leader sent an export %q", nfs)
|
||||
}
|
||||
if err := k.host.MkdirAll(path, 0o755, ""); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
unit := slurm.MountUnitName(path)
|
||||
changed, err := k.write(filepath.Join(slurm.MountUnitDir, unit), []byte(slurm.MountUnit(nfs, path)), 0o644, "")
|
||||
if err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
if changed {
|
||||
k.log.Info("the Slurm cluster's shared files are mounted here", "from", nfs, "at", path)
|
||||
if err := k.systemctl(ctx, "daemon-reload"); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
if err := k.systemctl(ctx, "enable", unit); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
// From another export: a restart mounts the new one, and fails
|
||||
// while a job still has files open on the old.
|
||||
if err := k.systemctl(ctx, "restart", unit); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
} else if !k.active(ctx, unit) {
|
||||
if err := k.systemctl(ctx, "start", unit); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
}
|
||||
}
|
||||
if _, err := k.host.Run(ctx, nil, "mountpoint", "-q", path); err != nil {
|
||||
if sh.GetNfs() == "" {
|
||||
return SharedMissing, fmt.Errorf("%s is not mounted here: the site mounts it (slurm.shared names no export)", path)
|
||||
}
|
||||
return SharedMissing, fmt.Errorf("%s is not mounted from %s", path, sh.GetNfs())
|
||||
}
|
||||
return SharedMounted, nil
|
||||
}
|
||||
|
||||
// unshare takes away the export or the mount llmgw kept here that want no
|
||||
// longer has: path is where the files are wanted now, serve and nfs how.
|
||||
func (k *Keeper) unshare(ctx context.Context, path string, serve bool, nfs string) {
|
||||
if _, err := k.host.ReadFile(slurm.ExportsPath); err == nil && (!serve || path == "") {
|
||||
k.log.Info("this machine no longer exports the Slurm cluster's shared files")
|
||||
_ = k.host.Remove(slurm.ExportsPath)
|
||||
_, _ = k.host.Run(ctx, nil, "exportfs", "-ra")
|
||||
}
|
||||
out, err := k.host.Run(ctx, nil, "systemctl", "list-units", "--all", "--plain", "--no-legend", "--type=mount")
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
keep := ""
|
||||
if path != "" && nfs != "" && !serve {
|
||||
keep = slurm.MountUnitName(path)
|
||||
}
|
||||
for _, l := range strings.Split(string(out), "\n") {
|
||||
f := strings.Fields(l)
|
||||
if len(f) == 0 || f[0] == keep {
|
||||
continue
|
||||
}
|
||||
file := filepath.Join(slurm.MountUnitDir, f[0])
|
||||
b, err := k.host.ReadFile(file)
|
||||
if err != nil || !strings.HasPrefix(string(b), "# Written by llmgw (BACKLOG P): the Slurm cluster's shared files.") {
|
||||
continue
|
||||
}
|
||||
k.log.Info("the Slurm cluster's old shared mount is taken away", "unit", f[0])
|
||||
if err := k.systemctl(ctx, "disable", "--now", f[0]); err != nil {
|
||||
k.log.Warn("could not unmount the old shared files", "unit", f[0], "err", err)
|
||||
continue
|
||||
}
|
||||
_ = k.host.Remove(file)
|
||||
_ = k.systemctl(ctx, "daemon-reload")
|
||||
}
|
||||
}
|
||||
|
||||
// sharedPackages are what the shared path needs here, by the command that
|
||||
// shows each is installed.
|
||||
func sharedPackages(want *llmgwv1.Slurm, need map[string][2]string) {
|
||||
switch sh := want.GetShared(); {
|
||||
case sh.GetPath() == "":
|
||||
case sh.GetServe():
|
||||
need["exportfs"] = [2]string{"nfs-kernel-server", "nfs-utils"}
|
||||
case sh.GetNfs() != "":
|
||||
need["mount.nfs"] = [2]string{"nfs-common", "nfs-utils"}
|
||||
}
|
||||
}
|
||||
|
||||
// Serves is whether this machine exports the shared files, so NFS's port
|
||||
// is kept for the cluster's networks.
|
||||
func (k *Keeper) Serves() bool {
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
return k.want.GetShared().GetServe() && k.want.GetShared().GetPath() != ""
|
||||
}
|
||||
@@ -0,0 +1,171 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
func ayse() *llmgwv1.SlurmUser {
|
||||
return &llmgwv1.SlurmUser{Login: "ayse", Uid: 200000, Name: "Ayşe Y.", Home: "/shared/home/ayse",
|
||||
SshKeys: []string{"ssh-ed25519 AAAA ayse@laptop"}}
|
||||
}
|
||||
|
||||
func TestTheServerExportsAndMakesUsersWithHomes(t *testing.T) {
|
||||
h := newFake()
|
||||
h.active["ssh"] = true
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
m := nodeMsg(false)
|
||||
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Serve: true, Clients: []string{"10.0.0.22", "10.0.0.21"}}
|
||||
m.ManageUsers, m.MakeHomes, m.Users = true, true, []*llmgwv1.SlurmUser{ayse()}
|
||||
h.fail["test -d /shared/home/ayse"] = true
|
||||
h.files[slurm.PAMSSHD] = []byte(pamStack)
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
if st.GetError() != "" {
|
||||
t.Fatal(st.GetError())
|
||||
}
|
||||
if st.GetShared() != SharedExported || st.GetUsers() != 1 {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if !slices.Contains(inst.got["deb"], "nfs-kernel-server") {
|
||||
t.Errorf("packages %v", inst.got)
|
||||
}
|
||||
if e := string(h.files[slurm.ExportsPath]); !strings.Contains(e, `"/shared" 10.0.0.21(`) {
|
||||
t.Errorf("exports:\n%s", e)
|
||||
}
|
||||
for _, want := range []string{"systemctl enable --now nfs-server", "exportfs -ra", "groupadd -g 200000 ayse",
|
||||
"useradd -u 200000 -g 200000 -M -d /shared/home/ayse -s /bin/bash -c Ayşe Y. ayse",
|
||||
"install -d -m 0700 -o 200000 -g 200000 /shared/home/ayse", "sshd -t", "systemctl reload ssh"} {
|
||||
if !slices.Contains(h.ran, want) {
|
||||
t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
if string(h.files[slurm.AuthorizedKeysDir+"/ayse"]) != "ssh-ed25519 AAAA ayse@laptop\n" ||
|
||||
string(h.files[k.usersMarker()]) != "ayse\n" || h.files[slurm.SSHDropIn] == nil {
|
||||
t.Errorf("files: keys %q, marker %q", h.files[slurm.AuthorizedKeysDir+"/ayse"], h.files[k.usersMarker()])
|
||||
}
|
||||
|
||||
// Again, made: nothing is run again.
|
||||
h.outputs["getent passwd ayse"] = "ayse:x:200000:200000:Ayşe Y.:/shared/home/ayse:/bin/bash\n"
|
||||
delete(h.fail, "test -d /shared/home/ayse")
|
||||
h.ran = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
for _, c := range h.ran {
|
||||
if strings.HasPrefix(c, "useradd") || strings.HasPrefix(c, "usermod") || strings.HasPrefix(c, "exportfs") ||
|
||||
strings.HasPrefix(c, "install") || c == "systemctl reload ssh" {
|
||||
t.Errorf("ran %q again", c)
|
||||
}
|
||||
}
|
||||
|
||||
// Gone from the accounts: taken away, its home left.
|
||||
m.Users = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
if !slices.Contains(h.ran, "userdel ayse") || h.files[k.usersMarker()] != nil || h.files[slurm.SSHDropIn] != nil ||
|
||||
h.files[slurm.AuthorizedKeysDir+"/ayse"] != nil {
|
||||
t.Errorf("not taken away:\n%s", strings.Join(h.ran, "\n"))
|
||||
}
|
||||
if slices.ContainsFunc(h.ran, func(c string) bool { return strings.HasPrefix(c, "rm") || strings.Contains(c, "userdel -r") }) {
|
||||
t.Error("a home was removed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAClientMountsAndLeavesOthersUsersAlone(t *testing.T) {
|
||||
h := newFake()
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
m := nodeMsg(false)
|
||||
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Nfs: "10.0.0.11:/shared"}
|
||||
bob := &llmgwv1.SlurmUser{Login: "bob", Uid: 200001, Home: "/shared/home/bob"}
|
||||
m.ManageUsers, m.Users = true, []*llmgwv1.SlurmUser{ayse(), bob}
|
||||
// The site's own ayse, with its own UID.
|
||||
h.outputs["getent passwd ayse"] = "ayse:x:1001:1001::/home/ayse:/bin/bash\n"
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
if st.GetShared() != SharedMounted || st.GetUsers() != 1 {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if !strings.Contains(st.GetError(), "ayse is already a user here with UID 1001") {
|
||||
t.Errorf("error %q", st.GetError())
|
||||
}
|
||||
if u := string(h.files["/etc/systemd/system/shared.mount"]); !strings.Contains(u, "What=10.0.0.11:/shared") {
|
||||
t.Fatalf("unit:\n%s", u)
|
||||
}
|
||||
for _, want := range []string{"systemctl enable shared.mount", "systemctl restart shared.mount", "mountpoint -q /shared",
|
||||
"useradd -u 200001 -g 200001 -M -d /shared/home/bob -s /bin/bash -c bob"} {
|
||||
if !slices.Contains(h.ran, want) {
|
||||
t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
if slices.ContainsFunc(h.ran, func(c string) bool { return strings.Contains(c, "ayse") && !strings.HasPrefix(c, "getent") }) ||
|
||||
slices.ContainsFunc(h.ran, func(c string) bool { return strings.HasPrefix(c, "install -d") }) {
|
||||
t.Errorf("touched the site's ayse, or made a home:\n%s", strings.Join(h.ran, "\n"))
|
||||
}
|
||||
if string(h.files[k.usersMarker()]) != "bob\n" {
|
||||
t.Errorf("marker %q", h.files[k.usersMarker()])
|
||||
}
|
||||
|
||||
// Not mounted: said.
|
||||
h.fail["mountpoint -q /shared"] = true
|
||||
if st := k.Reconcile(context.Background(), m); st.GetShared() != SharedMissing ||
|
||||
!strings.Contains(st.GetError(), "not mounted from 10.0.0.11:/shared") {
|
||||
t.Errorf("status %v", st)
|
||||
}
|
||||
|
||||
// No shared path any more: the mount is taken away.
|
||||
h.outputs["systemctl list-units --all --plain --no-legend --type=mount"] = "shared.mount loaded active mounted llmgw\n"
|
||||
m.Shared = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
if !slices.Contains(h.ran, "systemctl disable --now shared.mount") || h.files["/etc/systemd/system/shared.mount"] != nil {
|
||||
t.Errorf("mount kept:\n%s", strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
|
||||
const pamStack = "@include common-auth\naccount required pam_nologin.so\n@include common-account\n@include common-session\n"
|
||||
|
||||
func TestUsersReachANodeOnlyThroughTheirJobs(t *testing.T) {
|
||||
h := newFake()
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
h.files[slurm.PAMSSHD] = []byte(pamStack)
|
||||
// The module is not there until its package is installed.
|
||||
for _, d := range slurm.ModuleDirs {
|
||||
h.fail["test -f "+d+"/"+slurm.AdoptModule] = true
|
||||
}
|
||||
m := nodeMsg(true)
|
||||
m.ManageUsers, m.Users = true, []*llmgwv1.SlurmUser{ayse()}
|
||||
k.Reconcile(context.Background(), m)
|
||||
if !slices.Contains(inst.got["deb"], "libpam-slurm-adopt") {
|
||||
t.Fatalf("not installed: %v", inst.got)
|
||||
}
|
||||
if got := string(h.files[slurm.PAMSSHD]); got != pamStack {
|
||||
t.Fatalf("lines put in with the module still missing:\n%s", got)
|
||||
}
|
||||
delete(h.fail, "test -f "+slurm.ModuleDirs[0]+"/"+slurm.AdoptModule)
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
got := string(h.files[slurm.PAMSSHD])
|
||||
if !strings.Contains(got, "pam_listfile.so item=user sense=allow file="+k.usersMarker()) ||
|
||||
!strings.Contains(got, "account required "+slurm.AdoptModule) {
|
||||
t.Fatalf("not held to their jobs (%s):\n%s", st.GetError(), got)
|
||||
}
|
||||
|
||||
// The controller is where they log in: not held there.
|
||||
m.Controller = true
|
||||
m.Files = map[string]string{"slurm.conf": "x"}
|
||||
k.Reconcile(context.Background(), m)
|
||||
if string(h.files[slurm.PAMSSHD]) != pamStack {
|
||||
t.Errorf("held on the controller:\n%s", h.files[slurm.PAMSSHD])
|
||||
}
|
||||
|
||||
// No users of llmgw's any more: the lines go.
|
||||
m.Controller, m.Files = false, nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
m.Users = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
if string(h.files[slurm.PAMSSHD]) != pamStack {
|
||||
t.Errorf("lines left with no users:\n%s", h.files[slurm.PAMSSHD])
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,733 @@
|
||||
// Package slurmnode keeps this machine's part in the Slurm cluster run beside
|
||||
// llmgw (BACKLOG P), as the leader says: slurmctld on the controller, slurmd
|
||||
// on a node, munge on both, installed from the machine's own package sources.
|
||||
//
|
||||
// What the leader sends is wanted state, kept by Keeper's loop: sent again,
|
||||
// or checked again a minute later, it changes nothing that is already so.
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// Host is what Keeper does to the machine; Local in the agent, a fake in
|
||||
// tests.
|
||||
type Host interface {
|
||||
Run(ctx context.Context, env []string, name string, args ...string) ([]byte, error)
|
||||
LookPath(name string) (string, error)
|
||||
ReadFile(path string) ([]byte, error)
|
||||
// WriteFile writes b at path with mode, owned by owner ("" for root),
|
||||
// making its directory.
|
||||
WriteFile(path string, b []byte, mode fs.FileMode, owner string) error
|
||||
// MkdirAll makes path, owned by owner.
|
||||
MkdirAll(path string, mode fs.FileMode, owner string) error
|
||||
Remove(path string) error
|
||||
// Download fetches url to path, kept only if its SHA-256 is sha; a file
|
||||
// already there with it is not fetched again.
|
||||
Download(ctx context.Context, url, path, sha string) error
|
||||
}
|
||||
|
||||
// Installer installs packages by family, "deb" and "rpm".
|
||||
type Installer interface {
|
||||
InstallPackages(ctx context.Context, byFamily map[string][]string) error
|
||||
}
|
||||
|
||||
// Reasons llmgw drains a node with; one drained for another reason, by a
|
||||
// Slurm administrator, is never resumed by llmgw.
|
||||
const (
|
||||
ReasonServing = "llmgw: serving models"
|
||||
reasonPrefix = "llmgw"
|
||||
)
|
||||
|
||||
// DropIn is the unit drop-in that starts slurmd configless.
|
||||
const DropIn = "/etc/systemd/system/slurmd.service.d/llmgw.conf"
|
||||
|
||||
// Keeper keeps the machine as the last Slurm message says.
|
||||
type Keeper struct {
|
||||
host Host
|
||||
inst Installer
|
||||
log *slog.Logger
|
||||
marker string
|
||||
|
||||
mu sync.Mutex
|
||||
want *llmgwv1.Slurm
|
||||
kick chan struct{}
|
||||
// resend has the next status sent even if unchanged: a new stream's
|
||||
// control plane may never have had it.
|
||||
resend bool
|
||||
// installed is true once the packages were found or put there, so a
|
||||
// minute's check does not look again.
|
||||
installed map[string]bool
|
||||
files string
|
||||
// acctDone is the digest of the password the database's user was last
|
||||
// given; clusterAdded the cluster last found in the accounting.
|
||||
acctDone string
|
||||
clusterAdded string
|
||||
}
|
||||
|
||||
// New is a Keeper; marker is a file it keeps while it runs Slurm here, so
|
||||
// an agent started again knows to stop what it started when told it has no
|
||||
// part any more.
|
||||
func New(host Host, inst Installer, marker string, log *slog.Logger) *Keeper {
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
return &Keeper{host: host, inst: inst, log: log, marker: marker,
|
||||
kick: make(chan struct{}, 1), installed: map[string]bool{}}
|
||||
}
|
||||
|
||||
// Apply is a new message from the leader: kept from now on.
|
||||
func (k *Keeper) Apply(m *llmgwv1.Slurm) {
|
||||
k.mu.Lock()
|
||||
k.want = proto.CloneOf(m)
|
||||
k.mu.Unlock()
|
||||
select {
|
||||
case k.kick <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
// HasPart is whether the last message gives this machine a part in Slurm.
|
||||
func (k *Keeper) HasPart() bool {
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
return k.want.GetController() || k.want.GetNode()
|
||||
}
|
||||
|
||||
// Resend has the status sent again on the next pass, which it starts now.
|
||||
func (k *Keeper) Resend() {
|
||||
k.mu.Lock()
|
||||
k.resend = true
|
||||
k.mu.Unlock()
|
||||
select {
|
||||
case k.kick <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
// Every is how often the machine is checked again without a new message.
|
||||
var Every = time.Minute
|
||||
|
||||
// resendAfter is how often an unchanged status is sent anyway.
|
||||
const resendAfter = 5 * time.Minute
|
||||
|
||||
// Run keeps the machine until ctx ends, sending its status when it changes
|
||||
// and every few minutes while it has a part.
|
||||
func (k *Keeper) Run(ctx context.Context, send func(*llmgwv1.SlurmStatus)) {
|
||||
t := time.NewTicker(Every)
|
||||
defer t.Stop()
|
||||
var last *llmgwv1.SlurmStatus
|
||||
var sent time.Time
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-k.kick:
|
||||
case <-t.C:
|
||||
}
|
||||
k.mu.Lock()
|
||||
want, again := k.want, k.resend
|
||||
k.resend = false
|
||||
k.mu.Unlock()
|
||||
if want == nil {
|
||||
continue
|
||||
}
|
||||
st := k.Reconcile(ctx, want)
|
||||
if st == nil {
|
||||
last = nil
|
||||
continue
|
||||
}
|
||||
if again || !proto.Equal(st, last) || time.Since(sent) > resendAfter {
|
||||
send(st)
|
||||
last, sent = st, time.Now()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Reconcile makes the machine what want says once, and answers its status;
|
||||
// nil when it has no part in Slurm.
|
||||
func (k *Keeper) Reconcile(ctx context.Context, want *llmgwv1.Slurm) *llmgwv1.SlurmStatus {
|
||||
ctx, cancel := context.WithTimeout(ctx, 20*time.Minute)
|
||||
defer cancel()
|
||||
if !want.GetController() && !want.GetNode() {
|
||||
k.stop(ctx)
|
||||
return nil
|
||||
}
|
||||
st := &llmgwv1.SlurmStatus{}
|
||||
fail := func(what string, err error) *llmgwv1.SlurmStatus {
|
||||
st.Error = what + ": " + err.Error()
|
||||
k.log.Warn("keeping Slurm on this node: "+what, "err", err)
|
||||
return st
|
||||
}
|
||||
if err := k.ensurePackages(ctx, want); err != nil {
|
||||
return fail("installing Slurm", err)
|
||||
}
|
||||
// From the daemon, not scontrol: a client command looks for its files
|
||||
// even to say its version, and a configless node has none before slurmd
|
||||
// has fetched them.
|
||||
daemon := "slurmd"
|
||||
if !want.GetNode() {
|
||||
daemon = "slurmctld"
|
||||
}
|
||||
out, err := k.host.Run(ctx, nil, daemon, "-V")
|
||||
if err != nil {
|
||||
return fail("reading Slurm's version", fmt.Errorf("%v: %s", err, lastLine(out)))
|
||||
}
|
||||
v, err := slurm.ParseVersion(string(out))
|
||||
if err != nil {
|
||||
return fail("reading Slurm's version", err)
|
||||
}
|
||||
st.Version = strings.TrimSpace(string(out))
|
||||
if i := strings.LastIndexByte(st.Version, ' '); i >= 0 {
|
||||
st.Version = st.Version[i+1:]
|
||||
}
|
||||
if slurm.TooOld(v) {
|
||||
return fail("Slurm", fmt.Errorf("%d.%02d is older than %d.%02d, the first to run jobs in cgroup v2; "+
|
||||
"a newer release of the distribution, or Slurm's own packages, has one", v[0], v[1],
|
||||
slurm.MinVersion[0], slurm.MinVersion[1]))
|
||||
}
|
||||
if err := k.host.WriteFile(k.marker, []byte("llmgw runs Slurm here (BACKLOG P)\n"), 0o644, ""); err != nil {
|
||||
return fail("keeping its marker", err)
|
||||
}
|
||||
if err := k.munge(ctx, want.GetMungeKey()); err != nil {
|
||||
return fail("munge", err)
|
||||
}
|
||||
// The shared files, then the users whose homes are on them. Neither
|
||||
// keeps Slurm from running: what goes wrong is said, and Slurm kept.
|
||||
var shErr, usErr error
|
||||
st.Shared, shErr = k.shared(ctx, want)
|
||||
if shErr != nil {
|
||||
k.log.Warn("keeping the Slurm cluster's shared files on this node", "err", shErr)
|
||||
}
|
||||
if shErr == nil || !want.GetMakeHomes() || want.GetShared().GetPath() == "" {
|
||||
st.Users, usErr = k.users(ctx, want)
|
||||
} else {
|
||||
// Homes made now would be made beside the shared files, not on them.
|
||||
usErr = errors.New("waiting for the shared files, where the homes are")
|
||||
}
|
||||
if usErr != nil {
|
||||
k.log.Warn("keeping the Slurm users on this node", "err", usErr)
|
||||
}
|
||||
// Jobs in containers; Slurm runs without them, and is kept anyway.
|
||||
var ctErr error
|
||||
if st.Containers, ctErr = k.containers(ctx, want); ctErr != nil {
|
||||
k.log.Warn("setting up jobs in containers on this node", "err", ctErr)
|
||||
}
|
||||
defer func() {
|
||||
switch {
|
||||
case st.Error != "":
|
||||
case shErr != nil:
|
||||
st.Error = "shared files: " + shErr.Error()
|
||||
case usErr != nil:
|
||||
st.Error = "users: " + usErr.Error()
|
||||
case ctErr != nil:
|
||||
st.Error = "containers: " + ctErr.Error()
|
||||
}
|
||||
}()
|
||||
if want.GetController() {
|
||||
// Accounting first: slurmctld reports to it from its first start.
|
||||
// Without it slurmctld still runs jobs, a pass of backfill late
|
||||
// each; so a failure here is said, and slurmctld is kept anyway.
|
||||
var acctErr error
|
||||
if acctErr = k.accounting(ctx, want.GetMungeKey(), want.GetFiles()); acctErr != nil {
|
||||
k.log.Warn("keeping Slurm's accounting on this node", "err", acctErr)
|
||||
}
|
||||
st.SlurmdbdRunning = k.active(ctx, "slurmdbd")
|
||||
defer func() {
|
||||
if acctErr != nil && st.Error == "" {
|
||||
st.Error = "slurmdbd: " + acctErr.Error()
|
||||
}
|
||||
}()
|
||||
sum, err := k.controller(ctx, want.GetFiles())
|
||||
st.FilesSha256 = sum
|
||||
if err != nil {
|
||||
return fail("slurmctld", err)
|
||||
}
|
||||
st.SlurmctldRunning = k.active(ctx, "slurmctld")
|
||||
}
|
||||
if want.GetNode() {
|
||||
if err := k.node(ctx, want, st); err != nil {
|
||||
return fail("slurmd", err)
|
||||
}
|
||||
}
|
||||
return st
|
||||
}
|
||||
|
||||
func (k *Keeper) ensurePackages(ctx context.Context, want *llmgwv1.Slurm) error {
|
||||
need := map[string][2]string{
|
||||
"munged": {"munge", "munge"},
|
||||
"scontrol": {"slurm-client", "slurm"},
|
||||
}
|
||||
if want.GetController() {
|
||||
need["slurmctld"] = [2]string{"slurmctld", "slurm-slurmctld"}
|
||||
need["slurmdbd"] = [2]string{"slurmdbd", "slurm-slurmdbd"}
|
||||
need["mariadbd"] = [2]string{"mariadb-server", "mariadb-server"}
|
||||
}
|
||||
if want.GetNode() {
|
||||
need["slurmd"] = [2]string{"slurmd", "slurm-slurmd"}
|
||||
}
|
||||
sharedPackages(want, need)
|
||||
pkgs := map[string][]string{}
|
||||
for bin, p := range need {
|
||||
if k.installed[bin] {
|
||||
continue
|
||||
}
|
||||
if _, err := k.host.LookPath(bin); err == nil {
|
||||
k.installed[bin] = true
|
||||
continue
|
||||
}
|
||||
pkgs["deb"] = append(pkgs["deb"], p[0])
|
||||
pkgs["rpm"] = append(pkgs["rpm"], p[1])
|
||||
}
|
||||
if len(pkgs) == 0 {
|
||||
return nil
|
||||
}
|
||||
slices.Sort(pkgs["deb"])
|
||||
slices.Sort(pkgs["rpm"])
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("%s missing, and this agent installs nothing", strings.Join(pkgs["deb"], ", "))
|
||||
}
|
||||
k.log.Info("installing Slurm, as the cluster's settings say", "packages", pkgs["deb"])
|
||||
if err := k.inst.InstallPackages(ctx, pkgs); err != nil {
|
||||
return err
|
||||
}
|
||||
if slices.Contains(pkgs["deb"], "mariadb-server") {
|
||||
// Installed by llmgw: its settings are llmgw's to keep. One that was
|
||||
// here before is another's, and is left as it is.
|
||||
if err := k.host.WriteFile(k.mariaDBMarker(), []byte("llmgw installed MariaDB here, for Slurm's accounting\n"),
|
||||
0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
for bin := range need {
|
||||
if _, err := k.host.LookPath(bin); err != nil {
|
||||
return fmt.Errorf("%s is still missing after installing %s", bin, strings.Join(pkgs["deb"], ", "))
|
||||
}
|
||||
k.installed[bin] = true
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// munge writes the cluster's key and keeps munged running on it.
|
||||
func (k *Keeper) munge(ctx context.Context, key []byte) error {
|
||||
if len(key) < 32 {
|
||||
return errors.New("the leader sent no key")
|
||||
}
|
||||
changed, err := k.write(slurm.MungeKeyPath, key, 0o400, "munge")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if changed {
|
||||
k.log.Info("munge key written; munge starts again on it")
|
||||
return k.systemctl(ctx, "enable", "--now", "munge", "&&restart")
|
||||
}
|
||||
if !k.active(ctx, "munge") {
|
||||
return k.systemctl(ctx, "enable", "--now", "munge")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// controller writes the files and keeps slurmctld running on them,
|
||||
// answering their digest.
|
||||
func (k *Keeper) controller(ctx context.Context, files map[string]string) (string, error) {
|
||||
if files["slurm.conf"] == "" {
|
||||
return "", errors.New("the leader sent no slurm.conf")
|
||||
}
|
||||
for _, d := range []string{slurm.StateDir, slurm.LogDir} {
|
||||
if err := k.host.MkdirAll(d, 0o755, "slurm"); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
names := make([]string, 0, len(files))
|
||||
for n := range files {
|
||||
names = append(names, n)
|
||||
}
|
||||
slices.Sort(names)
|
||||
h := sha256.New()
|
||||
changed := false
|
||||
for _, n := range names {
|
||||
if strings.ContainsAny(n, "/\\") || n == "" || n == "." || n == ".." {
|
||||
return "", fmt.Errorf("the leader sent a file named %q", n)
|
||||
}
|
||||
fmt.Fprintf(h, "%s\x00%s\x00", n, files[n])
|
||||
c, err := k.write(filepath.Join(slurm.ConfDir, n), []byte(files[n]), 0o644, "")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
changed = changed || c
|
||||
}
|
||||
sum := hex.EncodeToString(h.Sum(nil))
|
||||
if changed || sum != k.files {
|
||||
if err := k.systemctl(ctx, "enable", "slurmctld"); err != nil {
|
||||
return sum, err
|
||||
}
|
||||
// Nodes added or taken away need a restart before 23.11 and a
|
||||
// reconfigure after it; a restart does both, and running jobs live
|
||||
// through it.
|
||||
k.log.Info("slurmctld starts again on new files")
|
||||
if err := k.systemctl(ctx, "restart", "slurmctld"); err != nil {
|
||||
return sum, err
|
||||
}
|
||||
if err := k.waitPing(ctx); err != nil {
|
||||
return sum, err
|
||||
}
|
||||
// Hands the configless slurmd the new files.
|
||||
if out, err := k.host.Run(ctx, nil, "scontrol", "reconfigure"); err != nil {
|
||||
return sum, fmt.Errorf("scontrol reconfigure: %v: %s", err, lastLine(out))
|
||||
}
|
||||
k.files = sum
|
||||
return sum, nil
|
||||
}
|
||||
if !k.active(ctx, "slurmctld") {
|
||||
return sum, k.systemctl(ctx, "enable", "--now", "slurmctld")
|
||||
}
|
||||
return sum, nil
|
||||
}
|
||||
|
||||
func (k *Keeper) waitPing(ctx context.Context) error {
|
||||
var last []byte
|
||||
for range 30 {
|
||||
out, err := k.host.Run(ctx, nil, "scontrol", "ping")
|
||||
if err == nil && bytes.Contains(out, []byte("UP")) {
|
||||
return nil
|
||||
}
|
||||
last = out
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(pingWait):
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("slurmctld did not answer after starting: %s", lastLine(last))
|
||||
}
|
||||
|
||||
var pingWait = time.Second
|
||||
|
||||
// node keeps slurmd running configless and puts the node in or out of the
|
||||
// controller's hands as lent says.
|
||||
func (k *Keeper) node(ctx context.Context, want *llmgwv1.Slurm, st *llmgwv1.SlurmStatus) error {
|
||||
name, server := want.GetName(), want.GetConfServer()
|
||||
if name == "" || server == "" {
|
||||
return errors.New("the leader sent no name or controller")
|
||||
}
|
||||
if strings.ContainsAny(name+server, " \n\t\"'\\$`") {
|
||||
return fmt.Errorf("the leader sent a name or controller with spaces or quotes in it")
|
||||
}
|
||||
for _, d := range []string{slurm.SpoolDir, slurm.LogDir} {
|
||||
if err := k.host.MkdirAll(d, 0o755, "slurm"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
bin, err := k.host.LookPath("slurmd")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
unit := "# Written by llmgw (BACKLOG P): slurmd fetches its files from slurmctld.\n" +
|
||||
"[Unit]\nConditionPathExists=\nStartLimitIntervalSec=0\n" +
|
||||
// --systemd as the packaged unit has it (Type=notify, since 22.05):
|
||||
// without it systemd waits for a readiness that never comes.
|
||||
"[Service]\nExecStart=\nExecStart=" + bin + " --systemd --conf-server " + server + " -N " + name + "\n" +
|
||||
// A node new to the controller is refused until the leader has
|
||||
// written it into slurm.conf; tried again until it is.
|
||||
"Restart=on-failure\nRestartSec=15\n"
|
||||
changed, err := k.write(DropIn, []byte(unit), 0o644, "")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if out, err := k.host.Run(ctx, nil, "slurmd", "-C"); err == nil {
|
||||
st.Hardware = strings.TrimSpace(string(out))
|
||||
}
|
||||
if changed {
|
||||
if err := k.systemctl(ctx, "daemon-reload"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := k.systemctl(ctx, "enable", "slurmd"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := k.systemctl(ctx, "restart", "slurmd"); err != nil {
|
||||
return err
|
||||
}
|
||||
} else if !k.active(ctx, "slurmd") {
|
||||
if err := k.systemctl(ctx, "enable", "--now", "slurmd"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
st.SlurmdRunning = k.active(ctx, "slurmd")
|
||||
|
||||
st.Jobs = k.localJobs(ctx)
|
||||
env := k.clientEnv()
|
||||
out, err := k.host.Run(ctx, env, "scontrol", "show", "node", "-o", name)
|
||||
if err != nil {
|
||||
// Not in slurm.conf yet: the leader writes it once this node's
|
||||
// hardware is in.
|
||||
return nil
|
||||
}
|
||||
ns, err := slurm.ParseNodeState(string(out))
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
ours := ns.Reason == "" || strings.HasPrefix(ns.Reason, reasonPrefix)
|
||||
switch {
|
||||
case want.GetLent() && ns.Drained() && ours:
|
||||
k.log.Info("this node is lent to Slurm: resumed there")
|
||||
if out, err := k.host.Run(ctx, env, "scontrol", "update", "NodeName="+name, "State=RESUME"); err != nil {
|
||||
return fmt.Errorf("resuming the node: %v: %s", err, lastLine(out))
|
||||
}
|
||||
case !want.GetLent() && !strings.Contains(ns.State, "DRAIN"):
|
||||
k.log.Info("this node is llmgw's: drained in Slurm, its jobs left to finish")
|
||||
if out, err := k.host.Run(ctx, env, "scontrol", "update", "NodeName="+name, "State=DRAIN",
|
||||
"Reason="+ReasonServing); err != nil {
|
||||
return fmt.Errorf("draining the node: %v: %s", err, lastLine(out))
|
||||
}
|
||||
}
|
||||
if out, err := k.host.Run(ctx, env, "scontrol", "show", "node", "-o", name); err == nil {
|
||||
if ns2, err := slurm.ParseNodeState(string(out)); err == nil {
|
||||
ns = ns2
|
||||
}
|
||||
}
|
||||
st.State, st.Reason = ns.State, ns.Reason
|
||||
return nil
|
||||
}
|
||||
|
||||
// localJobs counts the job steps running here, from their slurmstepd, not
|
||||
// from the controller: a job outlives a controller that is down, and is
|
||||
// what decides whether llmgw may have the cards back. The one slurmstepd
|
||||
// that is no job's, "slurmstepd infinity" holding slurmd's cgroup v2 scope,
|
||||
// is not counted. -1 when it cannot tell.
|
||||
func (k *Keeper) localJobs(ctx context.Context) int32 {
|
||||
// pgrep exits 1 with no output when there is none.
|
||||
out, err := k.host.Run(ctx, nil, "pgrep", "-a", "-x", "slurmstepd")
|
||||
if err != nil && len(bytes.TrimSpace(out)) > 0 {
|
||||
return -1
|
||||
}
|
||||
n := int32(0)
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) < 2 || f[len(f)-1] == "infinity" {
|
||||
continue
|
||||
}
|
||||
n++
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// clientEnv points scontrol at slurmd's copy of the files on a node that is
|
||||
// not the controller, which has none of its own.
|
||||
func (k *Keeper) clientEnv() []string {
|
||||
if _, err := k.host.ReadFile(filepath.Join(slurm.ConfDir, "slurm.conf")); err == nil {
|
||||
return nil
|
||||
}
|
||||
return []string{"SLURM_CONF=" + filepath.Join(slurm.SpoolDir, "conf-cache", "slurm.conf")}
|
||||
}
|
||||
|
||||
// stop stops what llmgw started here, if it did.
|
||||
func (k *Keeper) stop(ctx context.Context) {
|
||||
if _, err := k.host.ReadFile(k.marker); err != nil {
|
||||
return
|
||||
}
|
||||
k.log.Info("this node has no part in Slurm any more: slurmd, slurmctld and slurmdbd stopped; the packages and MariaDB stay")
|
||||
// MariaDB stays: it may have been the machine's before, and holds the
|
||||
// jobs' records.
|
||||
for _, u := range []string{"slurmd", "slurmctld", "slurmdbd"} {
|
||||
_ = k.systemctl(ctx, "disable", "--now", u)
|
||||
}
|
||||
_ = k.host.Remove(DropIn)
|
||||
_ = k.systemctl(ctx, "daemon-reload")
|
||||
k.unshare(ctx, "", false, "")
|
||||
if err := k.removeUsers(ctx, k.madeUsers(), nil); err != nil {
|
||||
k.log.Warn("could not take away every Slurm user llmgw made", "err", err)
|
||||
}
|
||||
_ = k.host.Remove(k.marker)
|
||||
k.files, k.acctDone, k.clusterAdded = "", "", ""
|
||||
}
|
||||
|
||||
// write writes a file when it differs, saying whether it did.
|
||||
func (k *Keeper) write(path string, b []byte, mode fs.FileMode, owner string) (bool, error) {
|
||||
if old, err := k.host.ReadFile(path); err == nil && bytes.Equal(old, b) {
|
||||
return false, nil
|
||||
}
|
||||
return true, k.host.WriteFile(path, b, mode, owner)
|
||||
}
|
||||
|
||||
// systemctl runs systemctl; a last argument "&&restart" restarts the unit
|
||||
// after.
|
||||
func (k *Keeper) systemctl(ctx context.Context, args ...string) error {
|
||||
restart := false
|
||||
if n := len(args); n > 0 && args[n-1] == "&&restart" {
|
||||
args, restart = args[:n-1], true
|
||||
}
|
||||
if out, err := k.host.Run(ctx, nil, "systemctl", args...); err != nil {
|
||||
return fmt.Errorf("systemctl %s: %v: %s", strings.Join(args, " "), err, lastLine(out))
|
||||
}
|
||||
if restart {
|
||||
u := args[len(args)-1]
|
||||
if out, err := k.host.Run(ctx, nil, "systemctl", "restart", u); err != nil {
|
||||
return fmt.Errorf("systemctl restart %s: %v: %s", u, err, lastLine(out))
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (k *Keeper) active(ctx context.Context, unit string) bool {
|
||||
out, err := k.host.Run(ctx, nil, "systemctl", "is-active", unit)
|
||||
return err == nil && strings.TrimSpace(string(out)) == "active"
|
||||
}
|
||||
|
||||
func lastLine(b []byte) string {
|
||||
s := strings.TrimSpace(string(b))
|
||||
if i := strings.LastIndexByte(s, '\n'); i >= 0 {
|
||||
s = s[i+1:]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Local is the machine the agent runs on.
|
||||
type Local struct{}
|
||||
|
||||
// Run runs a command, with env added to the agent's.
|
||||
func (Local) Run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
|
||||
cmd := exec.CommandContext(ctx, name, args...)
|
||||
if len(env) > 0 {
|
||||
cmd.Env = append(os.Environ(), env...)
|
||||
}
|
||||
return cmd.CombinedOutput()
|
||||
}
|
||||
|
||||
// LookPath finds a command, in the sbin directories too: an agent started
|
||||
// by systemd has them, one started by hand may not.
|
||||
func (Local) LookPath(name string) (string, error) {
|
||||
if p, err := exec.LookPath(name); err == nil {
|
||||
return p, nil
|
||||
}
|
||||
// /usr/libexec: Red Hat's mariadbd.
|
||||
for _, d := range []string{"/usr/sbin", "/sbin", "/usr/local/sbin", "/usr/libexec"} {
|
||||
p := filepath.Join(d, name)
|
||||
if fi, err := os.Stat(p); err == nil && !fi.IsDir() {
|
||||
return p, nil
|
||||
}
|
||||
}
|
||||
return "", fmt.Errorf("%s is not installed", name)
|
||||
}
|
||||
|
||||
// ReadFile reads a file.
|
||||
func (Local) ReadFile(path string) ([]byte, error) { return os.ReadFile(path) } //nolint:gosec // our paths
|
||||
|
||||
// WriteFile writes a file whole, through a temporary one beside it.
|
||||
func (Local) WriteFile(path string, b []byte, mode fs.FileMode, owner string) error {
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { //nolint:gosec // system directories
|
||||
return err
|
||||
}
|
||||
tmp := path + ".llmgw-new"
|
||||
if err := os.WriteFile(tmp, b, mode); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.Chmod(tmp, mode); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := chown(tmp, owner); err != nil {
|
||||
_ = os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, path)
|
||||
}
|
||||
|
||||
// Download fetches url to path through a temporary file beside it, kept
|
||||
// only when its SHA-256 is sha.
|
||||
func (Local) Download(ctx context.Context, url, path, sha string) error {
|
||||
if f, err := os.Open(path); err == nil { //nolint:gosec // our paths
|
||||
h := sha256.New()
|
||||
_, cerr := io.Copy(h, f)
|
||||
_ = f.Close()
|
||||
if cerr == nil && hex.EncodeToString(h.Sum(nil)) == sha {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { //nolint:gosec // system directories
|
||||
return err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
|
||||
defer cancel()
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("fetching %s: %w", url, err)
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("fetching %s: %s", url, resp.Status)
|
||||
}
|
||||
tmp := path + ".llmgw-new"
|
||||
f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o644) //nolint:gosec // our paths
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
h := sha256.New()
|
||||
_, err = io.Copy(io.MultiWriter(f, h), io.LimitReader(resp.Body, 1<<30))
|
||||
if cerr := f.Close(); err == nil {
|
||||
err = cerr
|
||||
}
|
||||
if err != nil {
|
||||
_ = os.Remove(tmp)
|
||||
return fmt.Errorf("fetching %s: %w", url, err)
|
||||
}
|
||||
if got := hex.EncodeToString(h.Sum(nil)); got != sha {
|
||||
_ = os.Remove(tmp)
|
||||
return fmt.Errorf("%s has SHA-256 %s, not the %s llmgw was built with: not used", url, got, sha)
|
||||
}
|
||||
return os.Rename(tmp, path)
|
||||
}
|
||||
|
||||
// MkdirAll makes a directory owned by owner.
|
||||
func (Local) MkdirAll(path string, mode fs.FileMode, owner string) error {
|
||||
if err := os.MkdirAll(path, mode); err != nil {
|
||||
return err
|
||||
}
|
||||
return chown(path, owner)
|
||||
}
|
||||
|
||||
// Remove removes a file, and is content when there is none.
|
||||
func (Local) Remove(path string) error {
|
||||
if err := os.Remove(path); err != nil && !errors.Is(err, fs.ErrNotExist) {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func chown(path, owner string) error {
|
||||
if owner == "" {
|
||||
return nil
|
||||
}
|
||||
u, err := user.Lookup(owner)
|
||||
if err != nil {
|
||||
return fmt.Errorf("user %s: %w (installed with its package)", owner, err)
|
||||
}
|
||||
uid, _ := strconv.Atoi(u.Uid)
|
||||
gid, _ := strconv.Atoi(u.Gid)
|
||||
return os.Chown(path, uid, gid)
|
||||
}
|
||||
@@ -0,0 +1,367 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io/fs"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
type fakeHost struct {
|
||||
files map[string][]byte
|
||||
owners map[string]string
|
||||
bins map[string]bool
|
||||
active map[string]bool
|
||||
ran []string
|
||||
outputs map[string]string
|
||||
fail map[string]bool
|
||||
state string
|
||||
reason string
|
||||
}
|
||||
|
||||
func newFake() *fakeHost {
|
||||
return &fakeHost{files: map[string][]byte{}, owners: map[string]string{}, bins: map[string]bool{},
|
||||
active: map[string]bool{}, outputs: map[string]string{
|
||||
"slurmd -V": "slurm-wlm 23.11.4\n",
|
||||
"slurmctld -V": "slurm-wlm 23.11.4\n",
|
||||
"scontrol ping": "Slurmctld(primary) at node06 is UP\n",
|
||||
"slurmd -C": "NodeName=cuda01 CPUs=16 RealMemory=64000\nUpTime=1-00:00:00\n",
|
||||
"hostname -s": "node06\n",
|
||||
}, fail: map[string]bool{}, state: "IDLE+DRAIN", reason: slurm.ReasonNew}
|
||||
}
|
||||
|
||||
func (f *fakeHost) Run(_ context.Context, env []string, name string, args ...string) ([]byte, error) {
|
||||
c := strings.Join(append([]string{name}, args...), " ")
|
||||
f.ran = append(f.ran, c)
|
||||
if f.fail[c] {
|
||||
return []byte("it failed"), errors.New("exit status 1")
|
||||
}
|
||||
switch {
|
||||
case name == "systemctl" && len(args) >= 1:
|
||||
u := args[len(args)-1]
|
||||
switch args[0] {
|
||||
case "is-active":
|
||||
if f.active[u] {
|
||||
return []byte("active\n"), nil
|
||||
}
|
||||
return []byte("inactive\n"), errors.New("exit status 3")
|
||||
case "restart", "start":
|
||||
f.active[u] = true
|
||||
case "enable":
|
||||
if slices.Contains(args, "--now") {
|
||||
f.active[u] = true
|
||||
}
|
||||
case "disable":
|
||||
f.active[u] = false
|
||||
}
|
||||
return []byte(f.outputs[c]), nil
|
||||
case strings.HasPrefix(c, "scontrol show node"):
|
||||
return []byte("NodeName=cuda01 Arch=x86_64 State=" + f.state + " Partitions=gpu Reason=" + f.reason + " [root@now]\n"), nil
|
||||
case strings.HasPrefix(c, "scontrol update"):
|
||||
if slices.Contains(args, "State=RESUME") {
|
||||
f.state, f.reason = "IDLE", ""
|
||||
} else {
|
||||
f.state, f.reason = "IDLE+DRAIN", ReasonServing
|
||||
}
|
||||
return nil, nil
|
||||
case strings.HasPrefix(c, "pgrep"):
|
||||
out := "1800358 /usr/sbin/slurmstepd infinity\n" + f.outputs["jobs"]
|
||||
return []byte(out), nil
|
||||
}
|
||||
return []byte(f.outputs[c]), nil
|
||||
}
|
||||
|
||||
func (f *fakeHost) LookPath(name string) (string, error) {
|
||||
if f.bins[name] {
|
||||
return "/usr/sbin/" + name, nil
|
||||
}
|
||||
return "", errors.New("not found")
|
||||
}
|
||||
|
||||
func (f *fakeHost) ReadFile(path string) ([]byte, error) {
|
||||
b, ok := f.files[path]
|
||||
if !ok {
|
||||
return nil, fs.ErrNotExist
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
|
||||
func (f *fakeHost) WriteFile(path string, b []byte, _ fs.FileMode, owner string) error {
|
||||
f.files[path] = append([]byte(nil), b...)
|
||||
f.owners[path] = owner
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeHost) MkdirAll(string, fs.FileMode, string) error { return nil }
|
||||
|
||||
func (f *fakeHost) Remove(path string) error { delete(f.files, path); return nil }
|
||||
|
||||
func (f *fakeHost) Download(_ context.Context, url, path, sha string) error {
|
||||
c := "download " + url + " " + path + " " + sha
|
||||
f.ran = append(f.ran, c)
|
||||
if f.fail[c] || f.fail["download "+url] {
|
||||
return errors.New("fetch failed")
|
||||
}
|
||||
f.files[path] = []byte("fetched " + url)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeHost) count(prefix string) int {
|
||||
n := 0
|
||||
for _, c := range f.ran {
|
||||
if strings.HasPrefix(c, prefix) {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
type fakeInst struct {
|
||||
host *fakeHost
|
||||
got map[string][]string
|
||||
}
|
||||
|
||||
func (i *fakeInst) InstallPackages(_ context.Context, by map[string][]string) error {
|
||||
i.got = by
|
||||
for _, b := range []string{"munged", "scontrol", "slurmd", "slurmctld", "slurmdbd", "mariadbd", "mariadb", "exportfs", "mount.nfs"} {
|
||||
i.host.bins[b] = true
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
var key = []byte(strings.Repeat("k", 1024))
|
||||
|
||||
func nodeMsg(lent bool) *llmgwv1.Slurm {
|
||||
return &llmgwv1.Slurm{Node: true, Name: "cuda01", ConfServer: "10.0.0.146:6817", MungeKey: key, Lent: lent}
|
||||
}
|
||||
|
||||
func TestNodeInstallsAndStaysDrained(t *testing.T) {
|
||||
h := newFake()
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
st := k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if st.GetError() != "" {
|
||||
t.Fatal(st.GetError())
|
||||
}
|
||||
if !slices.Equal(inst.got["deb"], []string{"munge", "slurm-client", "slurmd"}) ||
|
||||
!slices.Equal(inst.got["rpm"], []string{"munge", "slurm", "slurm-slurmd"}) {
|
||||
t.Fatalf("installed %v", inst.got)
|
||||
}
|
||||
if string(h.files[slurm.MungeKeyPath]) != string(key) || h.owners[slurm.MungeKeyPath] != "munge" {
|
||||
t.Fatal("the munge key was not written for munge")
|
||||
}
|
||||
unit := string(h.files[DropIn])
|
||||
if !strings.Contains(unit, "ExecStart=/usr/sbin/slurmd --systemd --conf-server 10.0.0.146:6817 -N cuda01\n") ||
|
||||
!strings.Contains(unit, "ConditionPathExists=\n") {
|
||||
t.Fatalf("drop-in:\n%s", unit)
|
||||
}
|
||||
if !st.GetSlurmdRunning() || st.GetVersion() != "23.11.4" || !strings.HasPrefix(st.GetHardware(), "NodeName=cuda01 CPUs=16") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
// Not lent: left drained, with llmgw's first reason.
|
||||
if h.count("scontrol update") != 0 || st.GetState() != "IDLE+DRAIN" {
|
||||
t.Fatalf("a node not lent was changed: %v, %v", h.ran, st)
|
||||
}
|
||||
if _, ok := h.files["/var/lib/llmgw-agent/slurm"]; !ok {
|
||||
t.Fatal("no marker")
|
||||
}
|
||||
|
||||
// Again, nothing changed: no install, no restart.
|
||||
h.ran = nil
|
||||
inst.got = nil
|
||||
k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if inst.got != nil || h.count("systemctl restart") != 0 || h.count("systemctl enable") != 0 {
|
||||
t.Fatalf("a second pass did work: %v", h.ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLendAndReclaim(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
st := k.Reconcile(context.Background(), nodeMsg(true))
|
||||
if h.count("scontrol update NodeName=cuda01 State=RESUME") != 1 || st.GetState() != "IDLE" {
|
||||
t.Fatalf("a lent node was not resumed: %v %v", h.ran, st)
|
||||
}
|
||||
h.outputs["jobs"] = "2001 slurmstepd: [101.batch]\n2002 slurmstepd: [102.0]\n"
|
||||
st = k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if h.count("scontrol update NodeName=cuda01 State=DRAIN Reason="+ReasonServing) != 1 {
|
||||
t.Fatalf("a node taken back was not drained: %v", h.ran)
|
||||
}
|
||||
if st.GetJobs() != 2 || !strings.Contains(st.GetState(), "DRAIN") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
// Drained already: not drained again.
|
||||
h.ran = nil
|
||||
k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if h.count("scontrol update") != 0 {
|
||||
t.Fatalf("drained twice: %v", h.ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminsDrainIsLeft(t *testing.T) {
|
||||
h := newFake()
|
||||
h.reason = "bad DIMM, ticket 42"
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
k.Reconcile(context.Background(), nodeMsg(true))
|
||||
if h.count("scontrol update") != 0 {
|
||||
t.Fatalf("llmgw resumed a node a Slurm administrator drained: %v", h.ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClientEnvOnANodeWithoutFiles(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
if env := k.clientEnv(); len(env) != 1 || !strings.HasSuffix(env[0], "conf-cache/slurm.conf") {
|
||||
t.Fatalf("env %v", env)
|
||||
}
|
||||
h.files[filepath.Join(slurm.ConfDir, "slurm.conf")] = []byte("x")
|
||||
if env := k.clientEnv(); env != nil {
|
||||
t.Fatalf("env %v on the controller", env)
|
||||
}
|
||||
}
|
||||
|
||||
func TestController(t *testing.T) {
|
||||
h := newFake()
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/m", nil)
|
||||
files := map[string]string{"slurm.conf": "ClusterName=llmgw\n", "gres.conf": "#\n", "cgroup.conf": "#\n"}
|
||||
m := &llmgwv1.Slurm{Controller: true, MungeKey: key, Files: files}
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
if st.GetError() != "" {
|
||||
t.Fatal(st.GetError())
|
||||
}
|
||||
if !slices.Contains(inst.got["deb"], "slurmctld") || slices.Contains(inst.got["deb"], "slurmd") {
|
||||
t.Fatalf("installed %v", inst.got)
|
||||
}
|
||||
if string(h.files["/etc/slurm/slurm.conf"]) != "ClusterName=llmgw\n" {
|
||||
t.Fatal("slurm.conf not written")
|
||||
}
|
||||
if h.count("systemctl restart slurmctld") != 1 || h.count("scontrol reconfigure") != 1 {
|
||||
t.Fatalf("slurmctld not started on its files: %v", h.ran)
|
||||
}
|
||||
if !st.GetSlurmctldRunning() || len(st.GetFilesSha256()) != 64 {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
// Accounting: MariaDB on this machine alone, the database, slurmdbd on
|
||||
// it, the cluster added, all before slurmctld.
|
||||
if !slices.Contains(inst.got["deb"], "slurmdbd") || !slices.Contains(inst.got["deb"], "mariadb-server") {
|
||||
t.Fatalf("installed %v", inst.got)
|
||||
}
|
||||
if !strings.Contains(string(h.files["/etc/mysql/mariadb.conf.d/90-llmgw-slurm.cnf"]), "bind-address=127.0.0.1") {
|
||||
t.Fatal("MariaDB, installed by llmgw, not kept to this machine")
|
||||
}
|
||||
dbd := string(h.files[slurm.DbdConfPath])
|
||||
if !strings.Contains(dbd, "DbdHost=node06\n") || !strings.Contains(dbd, "StoragePass="+slurm.DbdPassword(key)+"\n") ||
|
||||
h.owners[slurm.DbdConfPath] != "slurm" {
|
||||
t.Fatalf("slurmdbd.conf for %q:\n%s", h.owners[slurm.DbdConfPath], dbd)
|
||||
}
|
||||
if _, ok := h.files[acctSQLPath]; ok {
|
||||
t.Fatal("the database's statements, password and all, were left on disk")
|
||||
}
|
||||
if strings.Contains(strings.Join(h.ran, "\n"), slurm.DbdPassword(key)) {
|
||||
t.Fatal("the password was on a command line")
|
||||
}
|
||||
order := strings.Join(h.ran, "\n")
|
||||
if !strings.Contains(order, "sacctmgr -i add cluster llmgw") ||
|
||||
strings.Index(order, "slurmdbd") > strings.Index(order, "systemctl restart slurmctld") {
|
||||
t.Fatalf("accounting not before slurmctld: %v", h.ran)
|
||||
}
|
||||
if !st.GetSlurmdbdRunning() {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
sum := st.GetFilesSha256()
|
||||
|
||||
h.ran = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
if h.count("systemctl restart") != 0 {
|
||||
t.Fatalf("restarted on the same files: %v", h.ran)
|
||||
}
|
||||
|
||||
files["gres.conf"] = "NodeName=cuda01 Name=gpu File=/dev/nvidia0\n"
|
||||
h.ran = nil
|
||||
st = k.Reconcile(context.Background(), m)
|
||||
if h.count("systemctl restart slurmctld") != 1 || st.GetFilesSha256() == sum {
|
||||
t.Fatalf("a changed file did not restart slurmctld: %v", h.ran)
|
||||
}
|
||||
|
||||
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Controller: true, MungeKey: key,
|
||||
Files: map[string]string{"slurm.conf": "x", "../evil": "y"}}); !strings.Contains(st.GetError(), "named") {
|
||||
t.Fatalf("a path in a file name was taken: %v", st)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTooOld(t *testing.T) {
|
||||
h := newFake()
|
||||
h.outputs["slurmd -V"] = "slurm-wlm 21.08.5\n"
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
st := k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if !strings.Contains(st.GetError(), "older than 22.05") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if _, ok := h.files[slurm.MungeKeyPath]; ok {
|
||||
t.Fatal("went on with a Slurm too old")
|
||||
}
|
||||
}
|
||||
|
||||
func TestStop(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
k.Reconcile(context.Background(), nodeMsg(false))
|
||||
h.ran = nil
|
||||
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{}); st != nil {
|
||||
t.Fatalf("status %v for no part", st)
|
||||
}
|
||||
if !slices.Contains(h.ran, "systemctl disable --now slurmd") {
|
||||
t.Fatalf("slurmd not stopped: %v", h.ran)
|
||||
}
|
||||
if _, ok := h.files[DropIn]; ok {
|
||||
t.Fatal("drop-in left")
|
||||
}
|
||||
// Never started here: nothing touched.
|
||||
h2 := newFake()
|
||||
k2 := New(h2, &fakeInst{host: h2}, "/m", nil)
|
||||
k2.Reconcile(context.Background(), &llmgwv1.Slurm{})
|
||||
if len(h2.ran) != 0 {
|
||||
t.Fatalf("a machine llmgw never ran Slurm on was touched: %v", h2.ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBadNames(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
m := nodeMsg(false)
|
||||
m.Name = "cuda01 -c /etc/evil"
|
||||
if st := k.Reconcile(context.Background(), m); !strings.Contains(st.GetError(), "spaces") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Node: true, Name: "a", ConfServer: "b"}); !strings.Contains(st.GetError(), "no key") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMariaDBThatWasThereKeepsItsSettings(t *testing.T) {
|
||||
h := newFake()
|
||||
for _, b := range []string{"munged", "scontrol", "slurmctld", "slurmdbd", "mariadbd", "mariadb", "exportfs", "mount.nfs"} {
|
||||
h.bins[b] = true
|
||||
}
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Controller: true, MungeKey: key,
|
||||
Files: map[string]string{"slurm.conf": "ClusterName=llmgw\n"}})
|
||||
if st.GetError() != "" {
|
||||
t.Fatal(st.GetError())
|
||||
}
|
||||
for p := range h.files {
|
||||
if strings.HasSuffix(p, ".cnf") {
|
||||
t.Fatalf("a MariaDB llmgw did not install had its settings changed: %s", p)
|
||||
}
|
||||
}
|
||||
if h.count("mariadb -e source") != 1 || !st.GetSlurmdbdRunning() {
|
||||
t.Fatalf("the database was not made in the MariaDB that was there: %v", h.ran)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,319 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// usersMarker lists, a line each, the logins llmgw made here: the ones it
|
||||
// may change and take away. A user that was here before is never touched.
|
||||
func (k *Keeper) usersMarker() string { return k.marker + "-users" }
|
||||
|
||||
// users keeps the Linux users want has, the same login and UID as on every
|
||||
// other machine, their keys where sshd reads them, and their homes when
|
||||
// this machine makes them; and takes away those llmgw made that are gone.
|
||||
// Answers how many it has. A user that cannot be made is said, the others
|
||||
// are made anyway.
|
||||
func (k *Keeper) users(ctx context.Context, want *llmgwv1.Slurm) (int32, error) {
|
||||
made := k.madeUsers()
|
||||
if !want.GetManageUsers() {
|
||||
return 0, k.removeUsers(ctx, made, nil)
|
||||
}
|
||||
var errs []error
|
||||
var keep []string
|
||||
n := int32(0)
|
||||
for _, u := range want.GetUsers() {
|
||||
err := k.user(ctx, u, slices.Contains(made, u.GetLogin()), want.GetMakeHomes())
|
||||
if errors.Is(err, errNotMade) {
|
||||
errs = append(errs, err)
|
||||
continue
|
||||
}
|
||||
keep = append(keep, u.GetLogin())
|
||||
if err != nil {
|
||||
errs = append(errs, err)
|
||||
continue
|
||||
}
|
||||
n++
|
||||
}
|
||||
if err := k.keepMade(keep); err != nil {
|
||||
return n, err
|
||||
}
|
||||
if err := k.removeUsers(ctx, made, keep); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
if err := k.sshConf(ctx, len(keep) > 0); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
// The controller is where they log in and submit from; a node only
|
||||
// while a job of theirs runs there.
|
||||
if err := k.nodeLogin(ctx, len(keep) > 0 && want.GetNode() && !want.GetController()); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
return n, errors.Join(errs...)
|
||||
}
|
||||
|
||||
// nodeLogin holds the users llmgw made here to their jobs when they log in
|
||||
// (pam_slurm_adopt), or no longer. Only they are: root and the site's
|
||||
// administrators log in as before, so a mistake here never locks out the
|
||||
// people who would put it right.
|
||||
func (k *Keeper) nodeLogin(ctx context.Context, on bool) error {
|
||||
old, err := k.host.ReadFile(slurm.PAMSSHD)
|
||||
if err != nil {
|
||||
if !on {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("reading %s: %w", slurm.PAMSSHD, err)
|
||||
}
|
||||
if on {
|
||||
if err := k.adoptModule(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
read := func(name string) (string, error) {
|
||||
b, err := k.host.ReadFile(filepath.Join(filepath.Dir(slurm.PAMSSHD), name))
|
||||
return string(b), err
|
||||
}
|
||||
stack, contained, err := slurm.PAMAdopt(string(old), k.usersMarker(), read, on)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if stack == string(old) {
|
||||
return nil
|
||||
}
|
||||
if on {
|
||||
k.log.Info("llmgw's Slurm users log in to this node only while a job of theirs runs here",
|
||||
"in_the_job", contained)
|
||||
if !contained {
|
||||
k.log.Warn("pam_systemd could not be skipped for them safely here: their ssh sessions are not held " +
|
||||
"to their job's cores and cards, only to having one")
|
||||
}
|
||||
} else {
|
||||
k.log.Info("llmgw's Slurm users are no longer held to their jobs when they log in here")
|
||||
}
|
||||
// Read by sshd for each login: nothing to reload.
|
||||
_, err = k.write(slurm.PAMSSHD, []byte(stack), 0o644, "")
|
||||
return err
|
||||
}
|
||||
|
||||
// adoptModule is there, or is installed.
|
||||
func (k *Keeper) adoptModule(ctx context.Context) error {
|
||||
if k.installed[slurm.AdoptModule] {
|
||||
return nil
|
||||
}
|
||||
find := func() bool {
|
||||
for _, d := range slurm.ModuleDirs {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-f", filepath.Join(d, slurm.AdoptModule)); err == nil {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
if !find() {
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("%s missing, and this agent installs nothing", slurm.AdoptModule)
|
||||
}
|
||||
k.log.Info("installing pam_slurm_adopt, which holds llmgw's Slurm users to their jobs here")
|
||||
if err := k.inst.InstallPackages(ctx, slurm.AdoptPackages); err != nil {
|
||||
return err
|
||||
}
|
||||
if !find() {
|
||||
return fmt.Errorf("%s is still missing after installing %s", slurm.AdoptModule,
|
||||
strings.Join(slurm.AdoptPackages["deb"], ", "))
|
||||
}
|
||||
}
|
||||
k.installed[slurm.AdoptModule] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// errNotMade is a user llmgw could not make here, nor took for its own.
|
||||
var errNotMade = errors.New("not made")
|
||||
|
||||
type notMade struct{ msg string }
|
||||
|
||||
func (e notMade) Error() string { return e.msg }
|
||||
func (e notMade) Is(target error) bool { return target == errNotMade }
|
||||
|
||||
// user keeps one user; ours is whether llmgw made it here before.
|
||||
func (k *Keeper) user(ctx context.Context, u *llmgwv1.SlurmUser, ours, makeHome bool) error {
|
||||
login, uid, home := u.GetLogin(), u.GetUid(), u.GetHome()
|
||||
if !slurm.ValidLogin(login) || uid < slurm.MinUID || !filepath.IsAbs(home) || strings.Contains(home, "..") {
|
||||
return notMade{fmt.Sprintf("the leader sent a user %q, UID %d, home %q", login, uid, home)}
|
||||
}
|
||||
id := strconv.FormatUint(uint64(uid), 10)
|
||||
gecos := strings.Map(func(r rune) rune {
|
||||
if r == ':' || r == ',' || r == '\n' || r < ' ' {
|
||||
return -1
|
||||
}
|
||||
return r
|
||||
}, u.GetName())
|
||||
byName, okName := k.passwd(ctx, login)
|
||||
byID, okID := k.passwd(ctx, id)
|
||||
switch {
|
||||
case okName && byName[2] != id:
|
||||
return notMade{fmt.Sprintf("%s is already a user here with UID %s, not llmgw's %s", login, byName[2], id)}
|
||||
case !okName && okID:
|
||||
return notMade{fmt.Sprintf("UID %s is already %s's here, so %s is not made", id, byID[0], login)}
|
||||
case !okName:
|
||||
if _, ok := k.group(ctx, login); !ok {
|
||||
if out, err := k.host.Run(ctx, nil, "groupadd", "-g", id, login); err != nil {
|
||||
return notMade{fmt.Sprintf("groupadd %s: %v: %s", login, err, lastLine(out))}
|
||||
}
|
||||
}
|
||||
k.log.Info("a Slurm user is made here, as llmgw's accounts say", "login", login, "uid", uid)
|
||||
if out, err := k.host.Run(ctx, nil, "useradd", "-u", id, "-g", id, "-M", "-d", home, "-s", "/bin/bash",
|
||||
"-c", gecos, login); err != nil {
|
||||
return notMade{fmt.Sprintf("useradd %s: %v: %s", login, err, lastLine(out))}
|
||||
}
|
||||
case ours && (byName[4] != gecos || byName[5] != home):
|
||||
if out, err := k.host.Run(ctx, nil, "usermod", "-c", gecos, "-d", home, login); err != nil {
|
||||
return fmt.Errorf("usermod %s: %v: %s", login, err, lastLine(out))
|
||||
}
|
||||
}
|
||||
keys := strings.Join(u.GetSshKeys(), "\n")
|
||||
file := filepath.Join(slurm.AuthorizedKeysDir, login)
|
||||
if keys == "" {
|
||||
_ = k.host.Remove(file)
|
||||
} else if _, err := k.write(file, []byte(keys+"\n"), 0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
if makeHome {
|
||||
return k.home(ctx, login, id, home)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// home makes a user's home directory, from /etc/skel, when it is not there.
|
||||
func (k *Keeper) home(ctx context.Context, login, id, home string) error {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-d", home); err == nil {
|
||||
return nil
|
||||
}
|
||||
if err := k.host.MkdirAll(filepath.Dir(home), 0o755, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
k.log.Info("a Slurm user's home is made", "login", login, "home", home)
|
||||
for _, c := range [][]string{
|
||||
{"install", "-d", "-m", "0700", "-o", id, "-g", id, home},
|
||||
{"cp", "-rT", "/etc/skel", home},
|
||||
{"chown", "-R", id + ":" + id, home},
|
||||
} {
|
||||
if out, err := k.host.Run(ctx, nil, c[0], c[1:]...); err != nil {
|
||||
return fmt.Errorf("making %s's home: %s: %v: %s", login, c[0], err, lastLine(out))
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// removeUsers takes away the users llmgw made here that are not in keep.
|
||||
// Their homes and files stay; their UID is never given to another.
|
||||
func (k *Keeper) removeUsers(ctx context.Context, made, keep []string) error {
|
||||
var errs []error
|
||||
left := slices.Clone(keep)
|
||||
for _, login := range made {
|
||||
if slices.Contains(keep, login) {
|
||||
continue
|
||||
}
|
||||
k.log.Info("a Slurm user llmgw made is taken away; its home stays", "login", login)
|
||||
_ = k.host.Remove(filepath.Join(slurm.AuthorizedKeysDir, login))
|
||||
if _, ok := k.passwd(ctx, login); ok {
|
||||
if out, err := k.host.Run(ctx, nil, "userdel", login); err != nil {
|
||||
// Still logged in, or running something: tried again.
|
||||
errs = append(errs, fmt.Errorf("userdel %s: %v: %s", login, err, lastLine(out)))
|
||||
left = append(left, login)
|
||||
continue
|
||||
}
|
||||
}
|
||||
}
|
||||
if !slices.Equal(left, made) {
|
||||
if err := k.keepMade(left); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
}
|
||||
if len(left) == 0 {
|
||||
if err := k.sshConf(ctx, false); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
if err := k.nodeLogin(ctx, false); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
}
|
||||
return errors.Join(errs...)
|
||||
}
|
||||
|
||||
func (k *Keeper) madeUsers() []string {
|
||||
b, err := k.host.ReadFile(k.usersMarker())
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return strings.Fields(string(b))
|
||||
}
|
||||
|
||||
func (k *Keeper) keepMade(logins []string) error {
|
||||
s := slices.Clone(logins)
|
||||
slices.Sort(s)
|
||||
s = slices.Compact(s)
|
||||
if len(s) == 0 {
|
||||
return k.host.Remove(k.usersMarker())
|
||||
}
|
||||
_, err := k.write(k.usersMarker(), []byte(strings.Join(s, "\n")+"\n"), 0o644, "")
|
||||
return err
|
||||
}
|
||||
|
||||
// sshConf has sshd read the users' keys from AuthorizedKeysDir, or no
|
||||
// longer.
|
||||
func (k *Keeper) sshConf(ctx context.Context, on bool) error {
|
||||
if !on {
|
||||
if _, err := k.host.ReadFile(slurm.SSHDropIn); err != nil {
|
||||
return nil
|
||||
}
|
||||
_ = k.host.Remove(slurm.SSHDropIn)
|
||||
return k.reloadSSH(ctx)
|
||||
}
|
||||
changed, err := k.write(slurm.SSHDropIn, []byte(slurm.SSHConf), 0o644, "")
|
||||
if err != nil || !changed {
|
||||
return err
|
||||
}
|
||||
if out, err := k.host.Run(ctx, nil, "sshd", "-t"); err != nil {
|
||||
_ = k.host.Remove(slurm.SSHDropIn)
|
||||
return fmt.Errorf("sshd refuses llmgw's setting for the users' keys, so it is not kept: %v: %s", err, lastLine(out))
|
||||
}
|
||||
k.log.Info("sshd reads the Slurm users' keys from llmgw too", "dir", slurm.AuthorizedKeysDir)
|
||||
return k.reloadSSH(ctx)
|
||||
}
|
||||
|
||||
// reloadSSH has sshd read its settings again, under Debian's name for it
|
||||
// or Red Hat's; one that is not running reads them when it starts.
|
||||
func (k *Keeper) reloadSSH(ctx context.Context) error {
|
||||
for _, u := range []string{"ssh", "sshd"} {
|
||||
if k.active(ctx, u) {
|
||||
return k.systemctl(ctx, "reload", u)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// passwd is getent's line for a user, by login or UID, in its fields.
|
||||
func (k *Keeper) passwd(ctx context.Context, key string) ([]string, bool) {
|
||||
out, err := k.host.Run(ctx, nil, "getent", "passwd", key)
|
||||
f := strings.Split(strings.TrimSpace(string(out)), ":")
|
||||
if err != nil || len(f) < 7 {
|
||||
return nil, false
|
||||
}
|
||||
return f, true
|
||||
}
|
||||
|
||||
func (k *Keeper) group(ctx context.Context, key string) ([]string, bool) {
|
||||
out, err := k.host.Run(ctx, nil, "getent", "group", key)
|
||||
f := strings.Split(strings.TrimSpace(string(out)), ":")
|
||||
if err != nil || len(f) < 3 {
|
||||
return nil, false
|
||||
}
|
||||
return f, true
|
||||
}
|
||||
@@ -33,6 +33,17 @@ func CachePathFor(dir, source string) string {
|
||||
return filepath.Join(dir, hex.EncodeToString(sum[:8])+"-"+name)
|
||||
}
|
||||
|
||||
// cacheSource is the source a cache name is computed from: a set's address
|
||||
// without its trailing slash, as its directory was named when it was fetched.
|
||||
// Computed from the slash-ended address, every set in the catalog looked like
|
||||
// nobody's, and a sweep offered 330 GB of models still in use for deletion.
|
||||
func cacheSource(source string) string {
|
||||
if IsSet(source) {
|
||||
return strings.TrimSuffix(source, "/")
|
||||
}
|
||||
return source
|
||||
}
|
||||
|
||||
// Owned reports whether a path is one this package created.
|
||||
//
|
||||
// The dividing line of the whole feature. A file without the cache name was
|
||||
@@ -65,7 +76,7 @@ func Orphans(have []string, wanted []string) []string {
|
||||
// By base name rather than full path: the control plane does not know
|
||||
// each node's cache directory, and the name is what identifies the
|
||||
// file wherever the node keeps it.
|
||||
keep[filepath.Base(CachePathFor("", src))] = struct{}{}
|
||||
keep[filepath.Base(CachePathFor("", cacheSource(src)))] = struct{}{}
|
||||
}
|
||||
|
||||
var out []string
|
||||
|
||||
@@ -51,7 +51,7 @@ type manifest struct {
|
||||
// as a broken model rather than failing to load, so a partial set is left
|
||||
// unfinished under its temporary name and the next attempt starts again.
|
||||
func (f *Fetcher) ensureSet(ctx context.Context, source string, allowPickle bool) (string, error) {
|
||||
dir, err := f.cachePath(strings.TrimSuffix(source, "/"))
|
||||
dir, err := f.cachePath(cacheSource(source))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
@@ -74,6 +74,23 @@ func TestASetIsFetchedIntoOneDirectory(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestASetInTheCatalogIsNotAnOrphan(t *testing.T) {
|
||||
// The catalog names a set by its address with the trailing slash; the
|
||||
// fetch names the directory without it. A sweep that computed the name
|
||||
// from the catalog's spelling offered every set in use for deletion.
|
||||
f, srv, _ := mirrorOf(t, map[string]string{"model.safetensors": "aaaa", "config.json": "{}"})
|
||||
dir, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := Orphans([]string{dir}, []string{srv.URL + "/m/v/"}); len(got) != 0 {
|
||||
t.Errorf("a set the catalog still names was called an orphan: %v", got)
|
||||
}
|
||||
if got := Orphans([]string{dir}, nil); len(got) != 1 {
|
||||
t.Errorf("a set nothing names was not found: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAFinishedSetIsNotFetchedAgain(t *testing.T) {
|
||||
f, srv, hits := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
|
||||
|
||||
|
||||
@@ -247,6 +247,8 @@ var liveSettings = []string{
|
||||
"presets",
|
||||
// Handed to the reconciler as they change.
|
||||
"on_demand",
|
||||
// Read by the Slurm service each pass, kicked when they change.
|
||||
"slurm",
|
||||
// Read by server init, node add and backups, not by the server.
|
||||
"backup_recipients", "backup_copy_to",
|
||||
}
|
||||
|
||||
@@ -189,6 +189,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newMCPCmd(client, &asJSON),
|
||||
newKBCmd(client, &asJSON),
|
||||
newTrainCmd(client, &asJSON),
|
||||
newSlurmCmd(client, &asJSON),
|
||||
newConfigCmd(client, &asJSON),
|
||||
newAuditCmd(client, &asJSON),
|
||||
newNotifyCmd(client, &asJSON),
|
||||
|
||||
@@ -0,0 +1,477 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/url"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"text/tabwriter"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
// newSlurmCmd is the Slurm cluster run beside llmgw (BACKLOG P): set up on
|
||||
// nodes llmgw already has, and nodes handed between the two.
|
||||
func newSlurmCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
cmd := &cobra.Command{
|
||||
Use: "slurm",
|
||||
Short: "A Slurm cluster beside llmgw, for training and other batch work",
|
||||
Long: "Slurm runs on llmgw's own nodes, installed and kept by their agents: slurmctld on one node,\n" +
|
||||
"slurmd on those named. A node is llmgw's, drained in Slurm, until it is lent; lent, its\n" +
|
||||
"replicas move off first, then Slurm may run jobs there. Taken back, Slurm drains it and\n" +
|
||||
"llmgw has it again when its last job ends. Jobs are submitted on the controller node\n" +
|
||||
"(sbatch, srun) by its own users, as on any Slurm cluster.",
|
||||
}
|
||||
cmd.AddCommand(newSlurmStatusCmd(client, asJSON), newSlurmSetupCmd(client), newSlurmOffCmd(client),
|
||||
newSlurmListCmd(client, "lend", "Let Slurm run jobs on nodes; llmgw's replicas move off first", true),
|
||||
newSlurmListCmd(client, "reclaim", "Take nodes back from Slurm; llmgw has them when their jobs end", false),
|
||||
newSlurmSharedCmd(client), newSlurmUsersCmd(client, asJSON), newSlurmContainersCmd(client))
|
||||
return cmd
|
||||
}
|
||||
|
||||
func newSlurmContainersCmd(client func() *ctlClient) *cobra.Command {
|
||||
cmd := &cobra.Command{
|
||||
Use: "containers",
|
||||
Short: "Jobs in containers: srun --container-image (pyxis and enroot)",
|
||||
Long: "On, the agents install enroot and pyxis on every machine of the Slurm cluster (from NVIDIA's\n" +
|
||||
"releases, at versions llmgw pins and checks) and, on GPU machines, nvidia-container-cli; once\n" +
|
||||
"every machine has them, Slurm loads pyxis and jobs may run in a container:\n\n" +
|
||||
" srun --gres=gpu:1 --container-image=nvcr.io#nvidia/pytorch:25.01-py3 python train.py\n\n" +
|
||||
"The shared path is in every container where it is on the host. Off, Slurm stops loading pyxis;\n" +
|
||||
"what was installed stays.",
|
||||
}
|
||||
var note string
|
||||
toggle := func(use, short string, on bool) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: use,
|
||||
Short: short,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
if !on {
|
||||
if doc.Node([]string{"slurm", "containers"}) != nil {
|
||||
doc.Unset([]string{"slurm", "containers"})
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return doc.SetYAML([]string{"slurm", "containers"}, "true")
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
cmd.AddCommand(toggle("on", "Install pyxis and enroot everywhere, and let jobs run in containers", true),
|
||||
toggle("off", "Slurm stops loading pyxis; what was installed stays", false))
|
||||
return cmd
|
||||
}
|
||||
|
||||
func newSlurmSharedCmd(client func() *ctlClient) *cobra.Command {
|
||||
var path, server, nfs, note string
|
||||
var off bool
|
||||
c := &cobra.Command{
|
||||
Use: "shared --path DIR (--server NODE | --nfs HOST:/EXPORT) | --off",
|
||||
Short: "Files every machine of the Slurm cluster sees at one path",
|
||||
Long: "Home directories, datasets, the jobs' output and checkpoints, at the same path on the controller\n" +
|
||||
"and every node. Three ways:\n" +
|
||||
" --nfs HOST:/EXPORT the site's own NFS export, mounted at --path on every machine;\n" +
|
||||
" --server NODE an llmgw node (the controller or a node) exports --path over NFS to the\n" +
|
||||
" others, for a site with no storage of its own: one machine, not highly\n" +
|
||||
" available, no faster than its disk and network;\n" +
|
||||
" neither the site mounts --path itself (Lustre, BeeGFS, GPFS); llmgw only looks\n" +
|
||||
" that it is there.\n" +
|
||||
"The export answers the Slurm machines alone, NFS's port the cluster's networks. A machine\n" +
|
||||
"that gives up its mount does so when no job holds files on it.",
|
||||
Example: " llmgw ctl slurm shared --path /shared --server node06\n" +
|
||||
" llmgw ctl slurm shared --path /shared --nfs filer.site:/export/hpc\n" +
|
||||
" llmgw ctl slurm shared --path /lustre/hpc\n" +
|
||||
" llmgw ctl slurm shared --off",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
if off == (path != "") {
|
||||
return fmt.Errorf("give --path, or --off")
|
||||
}
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
if off {
|
||||
if doc.Node([]string{"slurm", "shared"}) != nil {
|
||||
doc.Unset([]string{"slurm", "shared"})
|
||||
}
|
||||
return nil
|
||||
}
|
||||
v := map[string]string{"path": path}
|
||||
if server != "" {
|
||||
v["server"] = server
|
||||
}
|
||||
if nfs != "" {
|
||||
v["nfs"] = nfs
|
||||
}
|
||||
b, _ := json.Marshal(v)
|
||||
return doc.SetYAML([]string{"slurm", "shared"}, string(b))
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(&path, "path", "", "where every machine sees the files, as /shared")
|
||||
c.Flags().StringVar(&server, "server", "", "the llmgw node that exports them over NFS")
|
||||
c.Flags().StringVar(&nfs, "nfs", "", "the site's NFS export, HOST:/EXPORT")
|
||||
c.Flags().BoolVar(&off, "off", false, "no shared files; mounts and exports llmgw made are taken away")
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
|
||||
func newSlurmUsersCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
cmd := &cobra.Command{
|
||||
Use: "users",
|
||||
Short: "The Linux users that run jobs, made from llmgw's accounts",
|
||||
Long: "With slurm.users \"llmgw\" every active account is a Linux user on every machine of the Slurm\n" +
|
||||
"cluster: its login from its e-mail address, a UID of llmgw's from 200000 up (never given to\n" +
|
||||
"another, even when the account goes), its home under the shared path. It logs in over SSH\n" +
|
||||
"with the keys given in the portal (Profile) or here. An account suspended or deleted loses\n" +
|
||||
"its user; its home and files stay. Off, users are the site's: from its directory (sssd), or\n" +
|
||||
"made by hand.",
|
||||
}
|
||||
list := &cobra.Command{
|
||||
Use: "list",
|
||||
Short: "Every user, with its login, UID and how many keys it has",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var us []slurmsvc.User
|
||||
if err := client().get(cmd.Context(), "/api/v1/slurm/users", &us); err != nil {
|
||||
return err
|
||||
}
|
||||
w := cmd.OutOrStdout()
|
||||
if *asJSON {
|
||||
return json.NewEncoder(w).Encode(us)
|
||||
}
|
||||
tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
|
||||
fmt.Fprintln(tw, "LOGIN\tUID\tACCOUNT\tSSH KEYS")
|
||||
for _, u := range us {
|
||||
fmt.Fprintf(tw, "%s\t%d\t%s\t%d\n", u.Login, u.UID, u.Email, len(u.SSHKeys))
|
||||
}
|
||||
return tw.Flush()
|
||||
},
|
||||
}
|
||||
var note string
|
||||
toggle := func(use, short, value string) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: use,
|
||||
Short: short,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
if value == "" {
|
||||
if doc.Node([]string{"slurm", "users"}) != nil {
|
||||
doc.Unset([]string{"slurm", "users"})
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return doc.SetYAML([]string{"slurm", "users"}, yamlString(value))
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
keys := &cobra.Command{
|
||||
Use: "keys ACCOUNT FILE",
|
||||
Short: "Replace an account's SSH public keys, from a file (- for standard input)",
|
||||
Long: "FILE holds OpenSSH public keys, one a line, as ~/.ssh/id_ed25519.pub; an empty one takes them all away.",
|
||||
Example: " llmgw ctl slurm users keys ayse@site.example ayse.pub\n" +
|
||||
" ssh-add -L | llmgw ctl slurm users keys ayse@site.example -",
|
||||
Args: cobra.ExactArgs(2),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var b []byte
|
||||
var err error
|
||||
if args[1] == "-" {
|
||||
b, err = io.ReadAll(io.LimitReader(cmd.InOrStdin(), 64<<10))
|
||||
} else {
|
||||
b, err = os.ReadFile(args[1])
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
body, _ := json.Marshal(map[string]string{"keys": string(b)})
|
||||
var u slurmsvc.User
|
||||
if err := client().put(cmd.Context(), "/api/v1/slurm/users/"+url.PathEscape(args[0])+"/ssh-keys", body, &u); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s (UID %d) has %d SSH key(s); the machines have them within a minute\n",
|
||||
u.Login, u.UID, len(u.SSHKeys))
|
||||
return nil
|
||||
},
|
||||
}
|
||||
cmd.AddCommand(list,
|
||||
toggle("on", "Make a Linux user for every active account, on every machine of the Slurm cluster", "llmgw"),
|
||||
toggle("off", "Leave users to the site; those llmgw made are taken away, their homes left", ""),
|
||||
keys)
|
||||
return cmd
|
||||
}
|
||||
|
||||
func newSlurmStatusCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "status",
|
||||
Short: "The Slurm cluster, and where each of its nodes is",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var st slurmsvc.Status
|
||||
if err := client().get(cmd.Context(), "/api/v1/slurm", &st); err != nil {
|
||||
return err
|
||||
}
|
||||
w := cmd.OutOrStdout()
|
||||
if *asJSON {
|
||||
return json.NewEncoder(w).Encode(st)
|
||||
}
|
||||
if !st.Enabled {
|
||||
fmt.Fprintln(w, "Slurm is off. `llmgw ctl slurm setup --controller NODE --nodes NODE,...` turns it on.")
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintf(w, "cluster %s, partition %s, controller %s", st.Cluster, st.Partition, st.Controller)
|
||||
switch c := st.Ctld; {
|
||||
case c == nil:
|
||||
fmt.Fprintln(w, ": not heard from yet")
|
||||
case c.Error != "":
|
||||
fmt.Fprintf(w, ": %s\n", c.Error)
|
||||
case c.SlurmctldRunning && c.SlurmdbdRunning:
|
||||
fmt.Fprintf(w, ": slurmctld %s and slurmdbd running\n", c.Version)
|
||||
case c.SlurmctldRunning:
|
||||
fmt.Fprintf(w, ": slurmctld %s running, slurmdbd not (the jobs' records)\n", c.Version)
|
||||
default:
|
||||
fmt.Fprintln(w, ": slurmctld not running")
|
||||
}
|
||||
fmt.Fprintf(w, "shared files: %s\n", sharedLine(st.Shared))
|
||||
if st.Users != "" {
|
||||
fmt.Fprintf(w, "users: %d, made from llmgw's accounts\n", st.UserCount)
|
||||
}
|
||||
if line := containersLine(st); line != "" {
|
||||
fmt.Fprintf(w, "containers: %s\n", line)
|
||||
}
|
||||
tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
|
||||
fmt.Fprintln(tw, "NODE\tPHASE\tSLURM\tJOBS\tSLURMD\tSHARED\tNOTE")
|
||||
for _, n := range st.Nodes {
|
||||
state, jobs, slurmd, shared, note := "-", "-", "-", "-", n.Describe()
|
||||
if r := n.Status; r != nil {
|
||||
state = or(r.State, "-")
|
||||
jobs = fmt.Sprint(r.Jobs)
|
||||
slurmd = map[bool]string{true: "running", false: "stopped"}[r.SlurmdRunning]
|
||||
shared = or(r.Shared, "-")
|
||||
if r.Error != "" {
|
||||
note = r.Error
|
||||
}
|
||||
}
|
||||
if !n.Connected {
|
||||
note = "agent not connected; " + note
|
||||
}
|
||||
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n", n.Node, n.Phase, state, jobs, slurmd, shared, note)
|
||||
}
|
||||
if err := tw.Flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(st.Waiting) > 0 {
|
||||
fmt.Fprintf(w, "not in slurm.conf until they report their hardware: %s\n", strings.Join(st.Waiting, ", "))
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newSlurmSetupCmd(client func() *ctlClient) *cobra.Command {
|
||||
var controller, cluster, partition, note string
|
||||
var nodes []string
|
||||
var all bool
|
||||
c := &cobra.Command{
|
||||
Use: "setup --controller NODE --nodes NODE,... | --all-nodes",
|
||||
Short: "Run Slurm on llmgw's nodes: slurmctld on one, slurmd on those named",
|
||||
Long: "Turns slurm.enabled on and names the nodes, or with --all-nodes every node. Their agents install munge and Slurm (22.05 or\n" +
|
||||
"newer) from the machines' own package sources; the controller also slurmdbd, the jobs' records, in a\n" +
|
||||
"MariaDB on that machine alone (one that was there keeps its settings). Every node starts drained in Slurm and stays\n" +
|
||||
"llmgw's until `llmgw ctl slurm lend` gives it to Slurm.",
|
||||
Example: " llmgw ctl slurm setup --controller node06 --nodes cuda01,cuda02\n" +
|
||||
" llmgw ctl slurm setup --controller node06 --all-nodes",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
if controller == "" || (len(nodes) == 0 && !all) {
|
||||
return fmt.Errorf("give --controller, and --nodes or --all-nodes")
|
||||
}
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
if err := doc.SetYAML([]string{"slurm", "enabled"}, "true"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := doc.SetYAML([]string{"slurm", "controller"}, yamlString(controller)); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := setList(doc, []string{"slurm", "nodes"}, nodes); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := doc.SetYAML([]string{"slurm", "all_nodes"}, fmt.Sprint(all)); err != nil {
|
||||
return err
|
||||
}
|
||||
// Lent nodes no longer named are not lent.
|
||||
lent := slices.DeleteFunc(docList(doc, []string{"slurm", "lent"}),
|
||||
func(n string) bool { return !all && !slices.Contains(nodes, n) })
|
||||
if err := setList(doc, []string{"slurm", "lent"}, lent); err != nil {
|
||||
return err
|
||||
}
|
||||
for k, v := range map[string]string{"cluster_name": cluster, "partition": partition} {
|
||||
if v != "" {
|
||||
if err := doc.SetYAML([]string{"slurm", k}, yamlString(v)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(&controller, "controller", "", "the node that runs slurmctld")
|
||||
c.Flags().StringSliceVar(&nodes, "nodes", nil, "the nodes that run slurmd")
|
||||
c.Flags().BoolVar(&all, "all-nodes", false, "every node runs slurmd, those that join later too")
|
||||
c.Flags().StringVar(&cluster, "cluster-name", "", "Slurm's ClusterName (llmgw)")
|
||||
c.Flags().StringVar(&partition, "partition", "", "the partition every node is in (gpu)")
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
|
||||
func newSlurmOffCmd(client func() *ctlClient) *cobra.Command {
|
||||
var note string
|
||||
var force bool
|
||||
c := &cobra.Command{
|
||||
Use: "off",
|
||||
Short: "Stop running Slurm; every node is llmgw's again",
|
||||
Long: "slurmd and slurmctld are stopped where llmgw started them; the packages stay. A node still\n" +
|
||||
"lent is refused unless --force: take it back first, so its jobs end before the daemons do.",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
if lent := docList(doc, []string{"slurm", "lent"}); len(lent) > 0 && !force {
|
||||
return fmt.Errorf("%s still lent: `llmgw ctl slurm reclaim` them first, or --force",
|
||||
strings.Join(lent, ", "))
|
||||
}
|
||||
setList(doc, []string{"slurm", "lent"}, nil) //nolint:errcheck // an empty list
|
||||
return doc.SetYAML([]string{"slurm", "enabled"}, "false")
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().BoolVar(&force, "force", false, "even with nodes still lent")
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
|
||||
// containersLine says whether jobs may run in containers, and which
|
||||
// machines are not ready yet; empty when containers are off.
|
||||
func containersLine(st slurmsvc.Status) string {
|
||||
switch st.Containers {
|
||||
case "on":
|
||||
return "on (srun --container-image)"
|
||||
case "":
|
||||
return ""
|
||||
}
|
||||
var waiting []string
|
||||
for _, n := range st.Nodes {
|
||||
if n.Status == nil || n.Status.Containers == "" {
|
||||
waiting = append(waiting, n.Node)
|
||||
}
|
||||
}
|
||||
if st.Ctld == nil || st.Ctld.Containers == "" {
|
||||
if !slices.Contains(waiting, st.Controller) {
|
||||
waiting = append(waiting, st.Controller)
|
||||
}
|
||||
}
|
||||
return "installing pyxis and enroot; jobs take them once every machine has them (waiting: " +
|
||||
strings.Join(waiting, ", ") + ")"
|
||||
}
|
||||
|
||||
// sharedLine says where the shared files are and come from.
|
||||
func sharedLine(sh config.SharedStorage) string {
|
||||
switch {
|
||||
case sh.Path == "":
|
||||
return "none (`llmgw ctl slurm shared` adds them; a batch job's output stays on its node)"
|
||||
case sh.Server != "":
|
||||
return fmt.Sprintf("%s, exported over NFS by %s", sh.Path, sh.Server)
|
||||
case sh.NFS != "":
|
||||
return fmt.Sprintf("%s, mounted from %s", sh.Path, sh.NFS)
|
||||
}
|
||||
return sh.Path + ", mounted by the site"
|
||||
}
|
||||
|
||||
// newSlurmListCmd adds nodes to slurm.lent, or takes them out.
|
||||
func newSlurmListCmd(client func() *ctlClient, use, short string, add bool) *cobra.Command {
|
||||
var note string
|
||||
c := &cobra.Command{
|
||||
Use: use + " NODE...",
|
||||
Short: short,
|
||||
Args: cobra.MinimumNArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
return lendChange(doc, args, add)
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
|
||||
// lendChange adds nodes to slurm.lent, or with add false takes them out.
|
||||
func lendChange(doc *settings.Doc, names []string, add bool) error {
|
||||
nodes := docList(doc, []string{"slurm", "nodes"})
|
||||
all := docBool(doc, []string{"slurm", "all_nodes"})
|
||||
lent := docList(doc, []string{"slurm", "lent"})
|
||||
for _, n := range names {
|
||||
switch {
|
||||
case add && !all && !slices.Contains(nodes, n):
|
||||
return fmt.Errorf("%s does not run slurmd: add it with `llmgw ctl slurm setup --nodes`", n)
|
||||
case add && !slices.Contains(lent, n):
|
||||
lent = append(lent, n)
|
||||
case !add:
|
||||
lent = slices.DeleteFunc(lent, func(l string) bool { return l == n })
|
||||
}
|
||||
}
|
||||
return setList(doc, []string{"slurm", "lent"}, lent)
|
||||
}
|
||||
|
||||
// docList reads a list of strings from the settings, empty when unset.
|
||||
func docList(doc *settings.Doc, path []string) []string {
|
||||
n := doc.Node(path)
|
||||
if n == nil {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
if err := n.Decode(&out); err != nil {
|
||||
return nil
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// docBool reads a boolean from the settings, false when unset.
|
||||
func docBool(doc *settings.Doc, path []string) bool {
|
||||
var b bool
|
||||
if n := doc.Node(path); n != nil {
|
||||
_ = n.Decode(&b)
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func setList(doc *settings.Doc, path []string, list []string) error {
|
||||
if len(list) == 0 {
|
||||
if doc.Node(path) != nil {
|
||||
doc.Unset(path)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return doc.SetYAML(path, yamlList(list))
|
||||
}
|
||||
|
||||
func yamlString(s string) string {
|
||||
b, _ := json.Marshal(s)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func yamlList(list []string) string {
|
||||
b, _ := json.Marshal(list)
|
||||
return string(b)
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
func TestSlurmDocLists(t *testing.T) {
|
||||
doc, err := settings.FromBytes("settings", []byte("slurm:\n enabled: true\n nodes: [cuda01, site2]\n"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := docList(doc, []string{"slurm", "nodes"}); !slices.Equal(got, []string{"cuda01", "site2"}) {
|
||||
t.Fatalf("nodes %v", got)
|
||||
}
|
||||
if got := docList(doc, []string{"slurm", "lent"}); got != nil {
|
||||
t.Fatalf("lent %v", got)
|
||||
}
|
||||
if err := setList(doc, []string{"slurm", "lent"}, []string{"cuda01"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := docList(doc, []string{"slurm", "lent"}); !slices.Equal(got, []string{"cuda01"}) {
|
||||
t.Fatalf("lent %v", got)
|
||||
}
|
||||
if err := setList(doc, []string{"slurm", "lent"}, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := doc.Bytes()
|
||||
if strings.Contains(string(b), "lent") {
|
||||
t.Fatalf("an empty list left:\n%s", b)
|
||||
}
|
||||
}
|
||||
@@ -84,3 +84,19 @@ func TestTheSummaryIsWrittenWhereItIsToldNotToTheTerminal(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestStatusSaysWhenAlertsGoNowhere(t *testing.T) {
|
||||
render := func(s adminapi.Summary) string {
|
||||
cmd := &cobra.Command{}
|
||||
var b bytes.Buffer
|
||||
cmd.SetOut(&b)
|
||||
printSummary(cmd, s)
|
||||
return b.String()
|
||||
}
|
||||
if out := render(adminapi.Summary{NoAlerts: true}); !strings.Contains(out, "ALERTS nowhere") {
|
||||
t.Errorf("no webhook was not called out:\n%s", out)
|
||||
}
|
||||
if out := render(adminapi.Summary{}); strings.Contains(out, "ALERTS") {
|
||||
t.Errorf("an alerts line with webhooks set:\n%s", out)
|
||||
}
|
||||
}
|
||||
+19
-15
@@ -4,6 +4,7 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
@@ -87,13 +88,10 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var results []struct {
|
||||
Node string `json:"node"`
|
||||
Files int `json:"files"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
if err := c.post(cmd.Context(), "/api/v1/weights/collect", body, &results); err != nil {
|
||||
var results []adminapi.CollectResult
|
||||
// The control plane waits for the nodes to list their files without the
|
||||
// deleted ones: half a minute at most.
|
||||
if err := c.patient(time.Minute).post(cmd.Context(), "/api/v1/weights/collect", body, &results); err != nil {
|
||||
return err
|
||||
}
|
||||
if asJSON {
|
||||
@@ -103,7 +101,7 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "Nothing to reclaim.")
|
||||
return nil
|
||||
}
|
||||
var freed int64
|
||||
var freed, left int64
|
||||
for _, r := range results {
|
||||
if r.Error != "" {
|
||||
// Reported rather than fatal: the other nodes' space is still worth
|
||||
@@ -111,14 +109,20 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: %s\n", r.Node, r.Error)
|
||||
continue
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: asked to remove %d files, %s\n",
|
||||
r.Node, r.Files, fileSize(r.Bytes))
|
||||
freed += r.Bytes
|
||||
line := fmt.Sprintf("%s: removed %d files, %s", r.Node, r.Files, fileSize(r.Bytes))
|
||||
if r.Left > 0 {
|
||||
line += fmt.Sprintf("; %d still there, %s", r.Left, fileSize(r.LeftBytes))
|
||||
}
|
||||
fmt.Fprintln(cmd.OutOrStdout(), line)
|
||||
freed, left = freed+r.Bytes, left+r.LeftBytes
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "\n%s freed.\n", fileSize(freed))
|
||||
if left > 0 {
|
||||
// Each node decides: one that keeps a path it cannot prove it
|
||||
// fetched is the feature working. Its agent's log names each.
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s the nodes kept; each node's agent log says why "+
|
||||
"(journalctl -u llmgw-agent).\n", fileSize(left))
|
||||
}
|
||||
// "asked", not "removed": each node decides, and one that refuses a path it
|
||||
// cannot prove it fetched is the feature working rather than failing.
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "\n%s requested. The nodes verify each path before deleting it.\n",
|
||||
fileSize(freed))
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
func TestAPackageInstallThatIsNoEnginesIsNamed(t *testing.T) {
|
||||
if got := provisionLine("", "", "installing", "mariadb-server, munge, with apt-get"); got !=
|
||||
"system packages: installing: mariadb-server, munge, with apt-get" {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
if got := provisionLine("llamacpp", "b10991", "ready", ""); got != "llamacpp b10991: ready" {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoresAloneKeepTheMemory(t *testing.T) {
|
||||
m := catalog.Model{Name: "judge", Profiles: []catalog.Profile{{Name: "cpu", MinMemoryGB: 2}, {Name: "cpu-big", MinMemoryGB: 2}}}
|
||||
if got, err := keptMemory(m, ""); err != nil || got != 2 {
|
||||
t.Fatalf("got %d, %v", got, err)
|
||||
}
|
||||
m.Profiles[1].MinMemoryGB = 4
|
||||
if _, err := keptMemory(m, ""); err == nil {
|
||||
t.Fatal("profiles that differ were given one memory")
|
||||
}
|
||||
if got, err := keptMemory(m, "cpu-big"); err != nil || got != 4 {
|
||||
t.Fatalf("got %d, %v", got, err)
|
||||
}
|
||||
if _, err := keptMemory(m, "gpu"); err == nil {
|
||||
t.Fatal("a profile the model has not was taken")
|
||||
}
|
||||
}
|
||||
+34
-3
@@ -338,7 +338,7 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
|
||||
var memory, cores int
|
||||
var profile string
|
||||
c := &cobra.Command{
|
||||
Use: "resources <model> --memory GB [--cores N]",
|
||||
Use: "resources <model> [--memory GB] [--cores N]",
|
||||
Short: "Set what a model reserves of a node: memory, and CPU cores",
|
||||
Long: "--memory is the host RAM a replica needs: for a model with no GPU, its\n" +
|
||||
"weights, KV cache and engine together. Placement reserves it on the node,\n" +
|
||||
@@ -351,8 +351,20 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
|
||||
"--model M` moves them to the new figures.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if !cmd.Flags().Changed("memory") && !cmd.Flags().Changed("cores") {
|
||||
return fmt.Errorf("give --memory, --cores or both")
|
||||
}
|
||||
if !cmd.Flags().Changed("memory") {
|
||||
return fmt.Errorf("--memory is required")
|
||||
// Cores alone: the memory stays what it is.
|
||||
var cur catalog.Model
|
||||
if err := client().get(cmd.Context(), "/api/v1/models/"+args[0], &cur); err != nil {
|
||||
return err
|
||||
}
|
||||
kept, err := keptMemory(cur, profile)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
memory = kept
|
||||
}
|
||||
body, err := json.Marshal(adminapi.ResourcesRequest{Profile: profile, MemoryGB: memory, CPUCores: cores})
|
||||
if err != nil {
|
||||
@@ -369,12 +381,31 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().IntVar(&memory, "memory", 0, "host RAM a replica needs, in GB")
|
||||
c.Flags().IntVar(&memory, "memory", 0, "host RAM a replica needs, in GB; kept as it is when left out")
|
||||
c.Flags().IntVar(&cores, "cores", 0, "CPU cores a replica with no GPU computes on; 0 reserves none")
|
||||
c.Flags().StringVar(&profile, "profile", "", "one profile; every profile when left out")
|
||||
return c
|
||||
}
|
||||
|
||||
// keptMemory is the memory a model's profiles reserve now, for a change of
|
||||
// cores alone: one profile's, or every profile's when they agree.
|
||||
func keptMemory(m catalog.Model, profile string) (int, error) {
|
||||
memory := -1
|
||||
for _, p := range m.Profiles {
|
||||
if profile != "" && p.Name != profile {
|
||||
continue
|
||||
}
|
||||
if memory >= 0 && p.MinMemoryGB != memory {
|
||||
return 0, fmt.Errorf("%s's profiles reserve different memory: give --memory, or --profile", m.Name)
|
||||
}
|
||||
memory = p.MinMemoryGB
|
||||
}
|
||||
if memory < 0 {
|
||||
return 0, fmt.Errorf("%s has no profile %q", m.Name, profile)
|
||||
}
|
||||
return memory, nil
|
||||
}
|
||||
|
||||
func newClassCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "class <model> critical|standard|opportunistic|default",
|
||||
|
||||
+15
-4
@@ -114,7 +114,7 @@ engines are kept.`,
|
||||
}
|
||||
f := cmd.Flags()
|
||||
f.StringVar(&na.opts.Node, "node", "", "node id (default: the machine's hostname)")
|
||||
f.StringVar(&na.opts.User, "ssh-user", "", "user to log in as (default: the one before @ in the address, else whoever ran sudo, else you)")
|
||||
f.StringVar(&na.opts.User, "ssh-user", "", "user to log in as (default: the one before @ in the address, else the ssh config's for the machine, else whoever ran sudo, else you)")
|
||||
f.StringArrayVar(&na.servers, "server", nil, "control plane address the agent dials, https://host:7443; repeatable for failover (default: this machine)")
|
||||
f.StringArrayVar(&na.fabrics, "fabric", nil, "InfiniBand or RoCE fabric and its adapters, name:mlx5_0,mlx5_1; repeatable")
|
||||
f.StringVar(&na.opts.MgmtAddr, "mgmt-addr", "", "address engines are reached at (default: found by the agent)")
|
||||
@@ -228,8 +228,9 @@ func (na *nodeAdd) target(addr string) onboard.Options {
|
||||
if o.User == "" {
|
||||
// Under sudo, the person who ran it — their account is usually the
|
||||
// one that exists on the nodes too. root's own keys still do the
|
||||
// logging in, since that is who is running ssh.
|
||||
o.User = os.Getenv("SUDO_USER")
|
||||
// logging in, since that is who is running ssh. Only a fallback: a
|
||||
// User in the ssh config's block for the machine comes first.
|
||||
o.FallbackUser = os.Getenv("SUDO_USER")
|
||||
}
|
||||
return o
|
||||
}
|
||||
@@ -495,7 +496,17 @@ func engineEvent(engine, version, state, detail string) onboard.Event {
|
||||
level = onboard.Warn
|
||||
}
|
||||
return onboard.Event{Step: onboard.StepEngines, Level: level,
|
||||
Detail: fmt.Sprintf("%s %s: %s%s", engine, version, state, prefixed(": ", detail))}
|
||||
Detail: provisionLine(engine, version, state, detail)}
|
||||
}
|
||||
|
||||
// provisionLine is one line of what a node is installing. A package
|
||||
// install that is no engine's, Slurm's or MariaDB's, has no name or version.
|
||||
func provisionLine(engine, version, state, detail string) string {
|
||||
what := strings.TrimSpace(engine + " " + version)
|
||||
if what == "" {
|
||||
what = "system packages"
|
||||
}
|
||||
return fmt.Sprintf("%s: %s%s", what, state, prefixed(": ", detail))
|
||||
}
|
||||
|
||||
func prefixed(p, s string) string {
|
||||
|
||||
@@ -49,6 +49,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
@@ -59,6 +60,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
@@ -254,6 +256,7 @@ func newServerCmd() *cobra.Command {
|
||||
// The settings in force, for what reads them at each use.
|
||||
var current atomic.Pointer[config.Server]
|
||||
current.Store(&cfg)
|
||||
rt.SetSlurm(func() config.SlurmSettings { return current.Load().Slurm })
|
||||
var agentSvc *agentsvc.Service
|
||||
var agentHandler http.Handler
|
||||
var agentPath string
|
||||
@@ -261,6 +264,8 @@ func newServerCmd() *cobra.Command {
|
||||
// What the settings say of the firewall, for the mirror and the
|
||||
// store's members; nil without a control plane.
|
||||
var fwPub *firewallPublisher
|
||||
// The Slurm cluster beside llmgw, when there is a store (BACKLOG P).
|
||||
var slurmSvc *slurmsvc.Service
|
||||
updateMetrics := &modelserve.UpdateCollector{}
|
||||
// The store's cluster, when it is one; read by the metrics and
|
||||
// by `ctl status`.
|
||||
@@ -412,6 +417,7 @@ func newServerCmd() *cobra.Command {
|
||||
adminAPI = api
|
||||
api.SetReleases(cfg.Releases)
|
||||
api.SetNotifier(notifier)
|
||||
api.SetWebhooks(func() int { return len(current.Load().Notify.Webhooks) })
|
||||
api.SetReplayer(rt.Replay)
|
||||
// Quality suites, asked of a model the way a replay is.
|
||||
evals = newEvalService(cfg, rt, admin, log)
|
||||
@@ -436,6 +442,31 @@ func newServerCmd() *cobra.Command {
|
||||
agentSvc.SetTrain(trainer)
|
||||
go trainer.Run(ctx)
|
||||
api.SetTrain(&adminapi.TrainAdmin{Store: st, Service: trainer})
|
||||
// Slurm on the nodes the settings name, its munge key
|
||||
// made from a cluster secret.
|
||||
slurmSvc = slurmsvc.New(func() config.SlurmSettings { return current.Load().Slurm },
|
||||
admin.Inventory(), agentSvc, func(ctx context.Context) ([]byte, error) {
|
||||
secret, err := st.ClusterSecret(ctx, "slurm-munge")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return slurm.MungeKey(secret)
|
||||
}, log)
|
||||
// What slurm.conf held, for the leader that comes next.
|
||||
slurmSvc.SetMemory(slurmsvc.StoreMemory(st))
|
||||
// slurm.users "llmgw": a Linux user for each active account.
|
||||
slurmSvc.SetUsers(func(ctx context.Context) ([]slurmsvc.User, error) {
|
||||
us, err := st.SlurmUsers(ctx)
|
||||
out := make([]slurmsvc.User, 0, len(us))
|
||||
for _, u := range us {
|
||||
out = append(out, adminapi.SlurmUserView(u))
|
||||
}
|
||||
return out, err
|
||||
})
|
||||
api.SetSlurmUsers(st)
|
||||
agentSvc.SetSlurm(slurmSvc)
|
||||
go slurmSvc.Run(ctx)
|
||||
api.SetSlurm(slurmSvc)
|
||||
}
|
||||
// The cluster's settings, each change checked against this
|
||||
// machine's own keys before it is kept.
|
||||
@@ -841,6 +872,9 @@ func newServerCmd() *cobra.Command {
|
||||
fwPub.Kick()
|
||||
}
|
||||
}
|
||||
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "slurm") }) && slurmSvc != nil {
|
||||
slurmSvc.Kick()
|
||||
}
|
||||
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "on_demand") }) && admin != nil {
|
||||
admin.Reconciler().SetEvictionTimes(next.OnDemand.EvictAfterIdle, next.OnDemand.MinLifetime)
|
||||
}
|
||||
|
||||
@@ -443,7 +443,7 @@ func (sa *serverAdd) storeAccess(ctx context.Context) error {
|
||||
return err
|
||||
}
|
||||
}
|
||||
sa.done("nodes will be told of both control planes within half a minute")
|
||||
sa.done("nodes will be told of every control plane within half a minute")
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -124,6 +124,13 @@ type serverInit struct {
|
||||
// media has the nodes install ComfyUI and the leader add a picture model
|
||||
// and a 3D model once a card fits one.
|
||||
media bool
|
||||
// slurm runs Slurm beside llmgw: slurmctld on this machine's node,
|
||||
// slurmd on every node, each llmgw's until it is lent.
|
||||
slurm bool
|
||||
// slurmShared is where Slurm's shared files come from: "here" (this
|
||||
// machine exports /shared), the site's export as HOST:/EXPORT, or
|
||||
// "none". slurmUsers is "llmgw" (users from the accounts) or "site".
|
||||
slurmShared, slurmUsers string
|
||||
// evictAfterIdle and minLifetime are on_demand's; zero leaves its own.
|
||||
evictAfterIdle, minLifetime time.Duration
|
||||
// standbys makes each machine of the replicated store a control plane
|
||||
@@ -258,6 +265,12 @@ func newServerInitCmd() *cobra.Command {
|
||||
f.DurationVar(&in.maxKeyTTL, "max-key-ttl", defaultMaxKeyTTL, "the longest an API key lives")
|
||||
f.BoolVar(&in.media, "media", false,
|
||||
"pictures and 3D: the nodes install ComfyUI, and a picture model and a 3D model are added once a card fits one")
|
||||
f.BoolVar(&in.slurm, "slurm", false,
|
||||
"Slurm for training and batch jobs: slurmctld and slurmdbd (with a MariaDB) on this machine, slurmd on every node, each llmgw's until 'llmgw ctl slurm lend'")
|
||||
f.StringVar(&in.slurmShared, "slurm-shared", "here",
|
||||
"with --slurm, the jobs' shared files at /shared: here (this machine exports them over NFS), the site's HOST:/EXPORT, or none")
|
||||
f.StringVar(&in.slurmUsers, "slurm-users", config.UsersLlmgw,
|
||||
"with --slurm, who makes the Linux users that run jobs: llmgw (one for each account, SSH keys in the portal) or site")
|
||||
f.DurationVar(&in.evictAfterIdle, "evict-after-idle", 0,
|
||||
"how long a model loaded on demand goes unasked-for before another may have its card (0: 2m)")
|
||||
f.DurationVar(&in.minLifetime, "min-lifetime", 0,
|
||||
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
|
||||
)
|
||||
@@ -132,9 +133,25 @@ func (in *serverInit) checkChoices() error {
|
||||
return fmt.Errorf("max key ttl %s: an hour at least", in.maxKeyTTL)
|
||||
case in.evictAfterIdle < 0 || in.minLifetime < 0:
|
||||
return fmt.Errorf("on-demand times cannot be negative")
|
||||
case in.slurm && in.noNode:
|
||||
return fmt.Errorf("slurm runs slurmctld on this machine's node: it needs the node (no --no-node)")
|
||||
case in.vip != "" && !in.standbys:
|
||||
return fmt.Errorf("a floating address moves between control planes: it needs the standbys")
|
||||
}
|
||||
if in.slurm {
|
||||
// The controller is this machine's node, named by its hostname: a
|
||||
// name Slurm does not take would keep the server from starting.
|
||||
if err := (config.SlurmSettings{Enabled: true, Controller: localNodeID()}).Check(); err != nil {
|
||||
return fmt.Errorf("slurm: this machine's hostname: %w", err)
|
||||
}
|
||||
set := config.SlurmSettings{Enabled: true, Controller: localNodeID(), Shared: slurmShared(in.slurmShared)}
|
||||
if err := set.Check(); err != nil {
|
||||
return fmt.Errorf("--slurm-shared %q: here, none, or the site's HOST:/EXPORT: %w", in.slurmShared, err)
|
||||
}
|
||||
if u := in.slurmUsers; u != "" && u != config.UsersLlmgw && u != "site" {
|
||||
return fmt.Errorf("--slurm-users %q: llmgw or site", u)
|
||||
}
|
||||
}
|
||||
if in.vip != "" {
|
||||
ip, _, err := net.ParseCIDR(in.vip)
|
||||
if err != nil || ip.To4() == nil {
|
||||
@@ -318,6 +335,9 @@ func (in *serverInit) tuneConfig(body string) string {
|
||||
"# one ('llmgw ctl model presets auto off' stops it).\n"+
|
||||
"engines:\n comfyui: "+provision.ComfyUIVersion()+"\npresets:\n auto: true\n")
|
||||
}
|
||||
if in.slurm {
|
||||
extra = append(extra, slurmSection(localNodeID(), slurmShared(in.slurmShared), in.slurmUsers != "site"))
|
||||
}
|
||||
if in.evictAfterIdle > 0 || in.minLifetime > 0 {
|
||||
b := "# How models that load when asked for take turns on a card, chosen at\n# install.\non_demand:\n"
|
||||
if in.evictAfterIdle > 0 {
|
||||
@@ -334,6 +354,53 @@ func (in *serverInit) tuneConfig(body string) string {
|
||||
return body
|
||||
}
|
||||
|
||||
// slurmShared is --slurm-shared as settings: "here" is /shared exported
|
||||
// by this machine's node, "none" nothing, anything else the site's export.
|
||||
func slurmShared(v string) config.SharedStorage {
|
||||
switch v {
|
||||
case "", "none":
|
||||
return config.SharedStorage{}
|
||||
case "here":
|
||||
return config.SharedStorage{Path: "/shared", Server: localNodeID()}
|
||||
}
|
||||
return config.SharedStorage{Path: "/shared", NFS: v}
|
||||
}
|
||||
|
||||
// slurmSection is server.yaml's slurm block as an install chooses it:
|
||||
// slurmctld on this machine's node, slurmd on every node, the shared files
|
||||
// and the users as chosen.
|
||||
func slurmSection(controller string, shared config.SharedStorage, users bool) string {
|
||||
b := "# Slurm beside llmgw, chosen at install: slurmctld, and slurmdbd with a MariaDB\n" +
|
||||
"# for the jobs' records, on this machine; slurmd on\n" +
|
||||
"# every node, those that join later too. Each node is llmgw's, drained in\n" +
|
||||
"# Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm; 'llmgw ctl slurm\n" +
|
||||
"# reclaim NODE' takes it back once its jobs end.\n" +
|
||||
"slurm:\n enabled: true\n controller: " + yamlString(controller) + "\n all_nodes: true\n"
|
||||
switch {
|
||||
case shared.Server != "":
|
||||
b += " # /shared on every Slurm machine, exported over NFS from this one\n" +
|
||||
" # ('llmgw ctl slurm shared' points it at the site's storage instead).\n" +
|
||||
" shared:\n path: " + yamlString(shared.Path) + "\n server: " + yamlString(shared.Server) + "\n"
|
||||
case shared.NFS != "":
|
||||
b += " # /shared on every Slurm machine, mounted from the site's export.\n" +
|
||||
" shared:\n path: " + yamlString(shared.Path) + "\n nfs: " + yamlString(shared.NFS) + "\n"
|
||||
}
|
||||
if users {
|
||||
b += " # A Linux user for each llmgw account, the same UID everywhere; SSH keys\n" +
|
||||
" # in the portal's Profile. Empty leaves users to the site (sssd, a directory).\n" +
|
||||
" users: llmgw\n"
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// localNodeID is the node id this machine's agent takes: its hostname.
|
||||
func localNodeID() string {
|
||||
if h, err := os.Hostname(); err == nil && h != "" {
|
||||
return h
|
||||
}
|
||||
return "localhost"
|
||||
}
|
||||
|
||||
// durationYAML is a duration as a person writes it: 2h, 90m, 1m30s.
|
||||
func durationYAML(d time.Duration) string {
|
||||
switch {
|
||||
@@ -467,6 +534,15 @@ func (in *serverInit) initFlags() []string {
|
||||
if in.media {
|
||||
add("--media")
|
||||
}
|
||||
if in.slurm {
|
||||
add("--slurm")
|
||||
if in.slurmShared != "" && in.slurmShared != "here" {
|
||||
add("--slurm-shared", in.slurmShared)
|
||||
}
|
||||
if in.slurmUsers == "site" {
|
||||
add("--slurm-users", "site")
|
||||
}
|
||||
}
|
||||
if in.evictAfterIdle > 0 {
|
||||
add("--evict-after-idle", durationYAML(in.evictAfterIdle))
|
||||
}
|
||||
|
||||
@@ -17,7 +17,7 @@ import (
|
||||
func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
|
||||
in := &serverInit{
|
||||
language: "tr", signupOpen: true, defaultRPM: 120, defaultTPM: 50000, maxKeyTTL: 30 * 24 * time.Hour,
|
||||
media: true, evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
|
||||
media: true, slurm: true, slurmShared: "here", slurmUsers: "llmgw", evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
|
||||
judge: judgeSpec{Fail: "open", Categories: []string{"Violent", "PII"}, Replicas: 3, Budget: 3 * time.Second},
|
||||
}
|
||||
j := in.judgeFor()
|
||||
@@ -41,6 +41,10 @@ func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
|
||||
t.Errorf("limits %d %d %s", cfg.Auth.DefaultRPM, cfg.Auth.DefaultTPM, cfg.Auth.MaxKeyTTL)
|
||||
case !cfg.Presets.Auto || cfg.Engines["comfyui"] == "":
|
||||
t.Errorf("pictures and 3D: presets %v comfyui %q", cfg.Presets.Auto, cfg.Engines["comfyui"])
|
||||
case !cfg.Slurm.Enabled || !cfg.Slurm.AllNodes || cfg.Slurm.Controller != localNodeID():
|
||||
t.Errorf("slurm %+v", cfg.Slurm)
|
||||
case cfg.Slurm.Shared.Path != "/shared" || cfg.Slurm.Shared.Server != localNodeID() || cfg.Slurm.Users != "llmgw":
|
||||
t.Errorf("slurm's shared files and users %+v", cfg.Slurm)
|
||||
case cfg.OnDemand.EvictAfterIdle != time.Minute || cfg.OnDemand.MinLifetime != 90*time.Second:
|
||||
t.Errorf("on demand %+v", cfg.OnDemand)
|
||||
}
|
||||
@@ -95,14 +99,14 @@ func TestAStarterIsChosenByNameOrByItsFile(t *testing.T) {
|
||||
|
||||
func TestTheChoicesAreTheirFlagsAgain(t *testing.T) {
|
||||
in := &serverInit{admin: "ops@example.com", hosts: []string{"llm.example.com"}, postgres: true,
|
||||
mirrorPath: "/data/models", noNode: true, language: "tr", media: true,
|
||||
mirrorPath: "/data/models", language: "tr", media: true, slurm: true,
|
||||
webhooks: []string{"https://hooks.slack.com/services/T/B/x, json:https://a.example.com/in"},
|
||||
judge: judgeSpec{Categories: []string{"Violent", "Jailbreak"}}, maxKeyTTL: 7 * 24 * time.Hour}
|
||||
in.starter, _ = starterFrom("qwen3-4b")
|
||||
got := strings.Join(in.initFlags(), " ")
|
||||
want := "--admin ops@example.com --host llm.example.com --language tr --max-key-ttl 168h --postgres " +
|
||||
"--mirror-dir /data/models --no-node --starter qwen3-4b --judge-categories Violent,Jailbreak " +
|
||||
"--webhook https://hooks.slack.com/services/T/B/x --webhook json:https://a.example.com/in --media"
|
||||
"--mirror-dir /data/models --starter qwen3-4b --judge-categories Violent,Jailbreak " +
|
||||
"--webhook https://hooks.slack.com/services/T/B/x --webhook json:https://a.example.com/in --media --slurm"
|
||||
if got != want {
|
||||
t.Errorf("flags\n got %s\nwant %s", got, want)
|
||||
}
|
||||
@@ -124,6 +128,7 @@ func TestChoicesThatCannotBeInstalledAreRefused(t *testing.T) {
|
||||
{maxKeyTTL: time.Minute},
|
||||
{vip: "10.0.0.160/24"},
|
||||
{standbys: true, vip: "10.0.0.160"},
|
||||
{slurm: true, noNode: true},
|
||||
} {
|
||||
if err := in.checkChoices(); err == nil {
|
||||
t.Errorf("taken: %+v", in)
|
||||
|
||||
@@ -398,7 +398,9 @@ labels:
|
||||
// checks its own llama.cpp, the CUDA build when it has a GPU, and a new
|
||||
// machine is often busy with its own package updates first.
|
||||
client := in.adminAPI()
|
||||
last := ""
|
||||
// Each engine's line once, and again only when it changes: the engines'
|
||||
// states come back together on every look.
|
||||
seen := map[string]string{}
|
||||
err = waitFor(ctx, 20*time.Minute, "this node to be ready with an engine", func() (bool, error) {
|
||||
var nodes []adminapi.NodeView
|
||||
if err := client.get(ctx, "/api/v1/nodes", &nodes); err != nil {
|
||||
@@ -412,9 +414,9 @@ labels:
|
||||
return true, nil
|
||||
}
|
||||
for _, p := range n.Provision {
|
||||
if line := fmt.Sprintf("%s %s: %s %s", p.Engine, p.Version, p.State, p.Detail); line != last {
|
||||
fmt.Fprintf(in.out, " .. %s\n", strings.TrimSpace(line))
|
||||
last = line
|
||||
if line := provisionLine(p.Engine, p.Version, p.State, p.Detail); seen[p.Engine] != line {
|
||||
fmt.Fprintf(in.out, " .. %s\n", line)
|
||||
seen[p.Engine] = line
|
||||
}
|
||||
// Only llama.cpp is waited for: vLLM installs for minutes after,
|
||||
// and one this node cannot run is not a reason to stop here.
|
||||
|
||||
@@ -156,6 +156,10 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
|
||||
if s.Stopped {
|
||||
fmt.Fprintln(out, "SERVICE STOPPED: no model runs and requests are answered 503 (llmgw start)")
|
||||
}
|
||||
if s.NoAlerts {
|
||||
fmt.Fprintln(out, "ALERTS nowhere: a lost node or a judge down is told nobody "+
|
||||
"(llmgw ctl config set notify.webhooks, then llmgw ctl notify test)")
|
||||
}
|
||||
if s.GuardOff {
|
||||
fmt.Fprintln(out, "GUARD off: no request is checked, whatever the policies say (llmgw ctl guard on)")
|
||||
}
|
||||
|
||||
@@ -643,17 +643,30 @@ func (m consoleModel) nodeMenu(n adminapi.NodeView) *menuDialog {
|
||||
menuItem{"Delete model files nothing uses", func(m *consoleModel) tea.Cmd {
|
||||
return m.call("Unused files on "+n.ID, http.MethodPost, "/api/v1/weights/collect", map[string]string{"node": n.ID},
|
||||
func(raw json.RawMessage) string {
|
||||
var rs []struct {
|
||||
Files int `json:"files"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
}
|
||||
// What the node no longer lists, and what it kept.
|
||||
var rs []adminapi.CollectResult
|
||||
_ = json.Unmarshal(raw, &rs)
|
||||
var files int
|
||||
var b int64
|
||||
var files, left int
|
||||
var b, lb int64
|
||||
var errs []string
|
||||
for _, r := range rs {
|
||||
files, b = files+r.Files, b+r.Bytes
|
||||
files, b, left, lb = files+r.Files, b+r.Bytes, left+r.Left, lb+r.LeftBytes
|
||||
if r.Error != "" {
|
||||
errs = append(errs, r.Error)
|
||||
}
|
||||
}
|
||||
return fmt.Sprintf("%d files, %.1f GB freed.", files, float64(b)/(1<<30))
|
||||
if len(rs) == 0 {
|
||||
return "Nothing on " + n.ID + " that no model uses."
|
||||
}
|
||||
say := fmt.Sprintf("%d files removed, %.1f GB freed.", files, float64(b)/(1<<30))
|
||||
if left > 0 {
|
||||
say += fmt.Sprintf(" %d files (%.1f GB) are still there: the node kept them, and its agent's log "+
|
||||
"says why (journalctl -u llmgw-agent there).", left, float64(lb)/(1<<30))
|
||||
}
|
||||
if len(errs) > 0 {
|
||||
say += " " + strings.Join(errs, "; ")
|
||||
}
|
||||
return say
|
||||
})
|
||||
}},
|
||||
menuItem{"Remove from the cluster…", func(m *consoleModel) tea.Cmd {
|
||||
|
||||
@@ -57,7 +57,7 @@ type installForm struct {
|
||||
standby *installPart
|
||||
vipF *setupField
|
||||
|
||||
control, store, mirror, node, starter, media, judge, firewall, alerts, limits, turns, bundle *installPart
|
||||
control, store, mirror, node, starter, media, slurm, judge, firewall, alerts, limits, turns, bundle *installPart
|
||||
|
||||
admin, hosts, language, signup *setupField
|
||||
storeKind *setupField
|
||||
@@ -282,6 +282,31 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
|
||||
summary: func() string { return "ComfyUI; a picture model and a 3D model once a card fits" },
|
||||
}
|
||||
|
||||
// --- Slurm ---
|
||||
f.slurm = &installPart{
|
||||
title: "Slurm", on: in.slurm,
|
||||
about: "A Slurm cluster beside llmgw, for training and other batch work: slurmctld on this machine, " +
|
||||
"slurmd on every node, those that join later too. The agents install munge and Slurm from the " +
|
||||
"machines' own package sources, and on this machine slurmdbd, the jobs' records, with a MariaDB " +
|
||||
"that answers this machine alone (Slurm keeps its records in MySQL or MariaDB only, not in the " +
|
||||
"cluster's PostgreSQL). Its port 6819, with Slurm's others, answers the cluster's networks only.\n\n" +
|
||||
"Each node stays llmgw's, drained in Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm: " +
|
||||
"its models move off first. 'llmgw ctl slurm reclaim NODE' takes it back once its jobs end. Jobs " +
|
||||
"are submitted on this machine with sbatch and srun.\n\n" +
|
||||
"Every Slurm machine sees /shared, exported over NFS from this machine (homes, datasets, the jobs' " +
|
||||
"output); 'llmgw ctl slurm shared' points it at the site's own storage instead. Each llmgw account " +
|
||||
"is a Linux user there, the same UID on every machine, logging in with the SSH keys given in the " +
|
||||
"portal; 'llmgw ctl slurm users off' leaves users to the site's directory.",
|
||||
off: "no Slurm; 'llmgw ctl slurm setup' adds it later",
|
||||
summary: func() string { return "slurmctld here, slurmd on every node, /shared from here; none lent" },
|
||||
needs: func() string {
|
||||
if !f.node.on {
|
||||
return "slurmctld runs on this machine's node: turn the node on, or Slurm off"
|
||||
}
|
||||
return ""
|
||||
},
|
||||
}
|
||||
|
||||
// --- the guardrails' judge ---
|
||||
j := in.judgeFor()
|
||||
f.judgeFail = choiceField("When no judge answers in time", []string{"refuse", "serve flagged"},
|
||||
@@ -421,7 +446,7 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
|
||||
if ha {
|
||||
f.parts = append(f.parts, f.standby)
|
||||
}
|
||||
f.parts = append(f.parts, f.mirror, f.node, f.starter, f.media, f.judge, f.firewall,
|
||||
f.parts = append(f.parts, f.mirror, f.node, f.starter, f.media, f.slurm, f.judge, f.firewall,
|
||||
f.alerts, f.limits, f.turns, f.bundle)
|
||||
return f
|
||||
}
|
||||
@@ -653,6 +678,7 @@ func (f *installForm) apply(in *serverInit) error {
|
||||
in.noModel = !f.starter.on
|
||||
in.starter, in.starterCtx = starter, ctx
|
||||
in.media = f.media.on
|
||||
in.slurm = f.slurm.on
|
||||
in.noGuardModel = !f.judge.on
|
||||
in.judge = judge
|
||||
in.noFirewall = !f.firewall.on
|
||||
|
||||
@@ -24,6 +24,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/docgen"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
@@ -50,6 +51,7 @@ const (
|
||||
cFederation
|
||||
cMirror
|
||||
cJobs
|
||||
cSlurm
|
||||
cLogs
|
||||
cAccounts
|
||||
cKeys
|
||||
@@ -71,7 +73,7 @@ const (
|
||||
cCount
|
||||
)
|
||||
|
||||
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "MCP servers", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Sign-in", "Audit", "Manual",
|
||||
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "MCP servers", "Federation", "Mirror", "Jobs", "Slurm", "Logs", "Accounts", "API keys", "Organizations", "Sign-in", "Audit", "Manual",
|
||||
"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Serve the guardrails' judge", "Upgrade the cluster",
|
||||
"Restart all replicas", "Back up now", "Wipe this machine", "Leave"}
|
||||
|
||||
@@ -92,6 +94,8 @@ type consoleData struct {
|
||||
// federation is the other llmgw clusters whose models are offered
|
||||
// here; empty on a control plane without a store.
|
||||
federation adminapi.FederationView
|
||||
// slurm is the Slurm cluster beside llmgw; off when it has none.
|
||||
slurm slurmsvc.Status
|
||||
// presets is the models that come with llmgw as they stand here; nil
|
||||
// from a control plane that cannot say.
|
||||
presets *adminapi.PresetsView
|
||||
@@ -284,6 +288,7 @@ func (m consoleModel) fetch() tea.Cmd {
|
||||
_ = c.get(ctx, "/api/v1/jobs", &d.jobs)
|
||||
_ = c.get(ctx, "/api/v1/audit?since=7d&limit=300", &d.audit)
|
||||
_ = c.get(ctx, "/api/v1/federation", &d.federation)
|
||||
_ = c.get(ctx, "/api/v1/slurm", &d.slurm)
|
||||
var presets adminapi.PresetsView
|
||||
if c.get(ctx, "/api/v1/presets", &presets) == nil {
|
||||
d.presets = &presets
|
||||
@@ -588,6 +593,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
case cFederation:
|
||||
m.dialog = "error"
|
||||
m.dialogTitle, m.dialogText = peerDialog(m.data.federation.Peers[m.row])
|
||||
case cSlurm:
|
||||
m.menuD = m.slurmMenu(m.data.slurm.Nodes[m.row])
|
||||
case cServe:
|
||||
return m, m.fetchRepo(m.hf.models[m.row].ID)
|
||||
case cAccounts:
|
||||
@@ -1110,6 +1117,8 @@ func (m consoleModel) rowCount() int {
|
||||
return len(m.data.audit)
|
||||
case cFederation:
|
||||
return len(m.data.federation.Peers)
|
||||
case cSlurm:
|
||||
return len(m.data.slurm.Nodes)
|
||||
case cServe:
|
||||
return len(m.hf.models)
|
||||
case cAccounts:
|
||||
@@ -1369,6 +1378,8 @@ func (m consoleModel) panel() (title, body, keys string) {
|
||||
return m.auditPanel(pw)
|
||||
case cFederation:
|
||||
return m.federationPanel(pw)
|
||||
case cSlurm:
|
||||
return m.slurmPanel(pw)
|
||||
case cMonitor:
|
||||
return m.monitorPanel(pw)
|
||||
case cServe:
|
||||
|
||||
@@ -213,10 +213,17 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
|
||||
"before it is served.", modelplan.GB(p.KVBytesPerToken*1000))
|
||||
}
|
||||
note += "\nThe mirror fetches it and checks it, it joins the catalog, and a replica is started\nand asked to say hello."
|
||||
// The longest context the first way holds, up to what the model was
|
||||
// trained for: a choice of another way is checked again on submitting.
|
||||
ctxDefault := "4096"
|
||||
if a.Format == "safetensors" {
|
||||
ctxDefault = strconv.Itoa(p.Context)
|
||||
}
|
||||
if len(opts) > 0 {
|
||||
if most := opts[0].MostContext(p.KVBytesPerToken, p.Trained); most > 0 {
|
||||
ctxDefault = strconv.Itoa(most)
|
||||
}
|
||||
}
|
||||
if len(opts) > 0 && opts[0].Engine == "strata" {
|
||||
ctxDefault = "32768"
|
||||
}
|
||||
@@ -249,6 +256,11 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
|
||||
Description: v[3], Format: a.Format}
|
||||
if a.Format == "gguf" {
|
||||
req.File = a.Name
|
||||
// A context longer than the default pool is the pool: one
|
||||
// slot of it, where the server would refuse the request.
|
||||
if ctxLen > modelserve.DefaultCtxPool {
|
||||
req.CtxPool = ctxLen
|
||||
}
|
||||
}
|
||||
o := opts[slices.Index(where, v[4])]
|
||||
// Asked for a context other than planned: does it still fit?
|
||||
|
||||
@@ -221,11 +221,11 @@ func (m consoleModel) userMenu(u store.User) *menuDialog {
|
||||
items = append(items,
|
||||
menuItem{"Add to an organization, team or project…", func(m *consoleModel) tea.Cmd {
|
||||
m.openForm(&formDialog{title: "Membership for " + who,
|
||||
note: "A member of a team or organization may use every project in it.",
|
||||
note: "A member of a team or organization may use every project in it. On an organization, a manager also sees its page and an admin also changes it.",
|
||||
fields: []formField{
|
||||
choiceF("level", []string{"org", "team", "project"}, "team"),
|
||||
textF("which", "", "Acme, Acme/ml or Acme/ml/training"),
|
||||
choiceF("role", []string{store.RoleMember, "manager"}, store.RoleMember),
|
||||
choiceF("role", store.MembershipRoles, store.RoleMember),
|
||||
},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
if v[1] == "" {
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
// The console's Slurm view: the Slurm cluster run beside llmgw (BACKLOG P),
|
||||
// each of its nodes and whose it is now, as `llmgw ctl slurm status` shows
|
||||
// them. A node is lent to Slurm, or taken back, from its row. Turning Slurm
|
||||
// on, and choosing its nodes, is `ctl slurm setup` or server.yaml's slurm
|
||||
// part: a form for it would be server.yaml written again.
|
||||
|
||||
func (m consoleModel) slurmPanel(pw int) (string, string, string) {
|
||||
keys := "→ or Enter open · ↑↓ choose · Q leave"
|
||||
if m.inPanel {
|
||||
keys = "↑↓ node · Enter lend or take back · ← back · Q leave"
|
||||
}
|
||||
st := m.data.slurm
|
||||
head := "\n " + sDim.Render("Slurm beside llmgw, on its nodes: training and batch jobs.")
|
||||
if !st.Enabled {
|
||||
return "Slurm", head + "\n\n " + sDim.Render("Slurm is off. Turn it on with: "+
|
||||
"llmgw ctl slurm setup --controller NODE --nodes NODE,... (or --all-nodes)"), keys
|
||||
}
|
||||
ctl := sDim.Render("not heard from yet")
|
||||
switch c := st.Ctld; {
|
||||
case c == nil:
|
||||
case c.Error != "":
|
||||
ctl = sBad.Render(c.Error)
|
||||
case c.SlurmctldRunning && c.SlurmdbdRunning:
|
||||
ctl = sOK.Render("slurmctld " + c.Version + " and slurmdbd running")
|
||||
case c.SlurmctldRunning:
|
||||
ctl = sOK.Render("slurmctld "+c.Version+" running") + ", " + sWarn.Render("slurmdbd not")
|
||||
default:
|
||||
ctl = sBad.Render("slurmctld not running")
|
||||
}
|
||||
head += fmt.Sprintf("\n controller %s: %s", st.Controller, ctl)
|
||||
head += "\n " + sDim.Render(fmt.Sprintf("cluster %s, partition %s; sbatch and srun on %s", st.Cluster, st.Partition, st.Controller))
|
||||
if st.Shared.Path == "" {
|
||||
head += "\n shared files: " + sWarn.Render("none") + sDim.Render(" (llmgw ctl slurm shared; output stays on the job's node)")
|
||||
} else {
|
||||
head += "\n shared files: " + sharedLine(st.Shared)
|
||||
}
|
||||
if st.Users != "" {
|
||||
head += "\n " + fmt.Sprintf("users: %d from llmgw's accounts", st.UserCount) + sDim.Render(", SSH keys in the portal's Profile")
|
||||
} else {
|
||||
head += "\n " + sDim.Render("users: the site's (llmgw ctl slurm users on makes them from llmgw's accounts)")
|
||||
}
|
||||
if line := containersLine(st); line != "" {
|
||||
head += "\n containers: " + line
|
||||
}
|
||||
head += "\n " + sDim.Render("A node is llmgw's until lent; lent, its replicas move off first.")
|
||||
if len(st.Waiting) > 0 {
|
||||
head += "\n " + sWarn.Render(fmt.Sprintf("not in slurm.conf until they report their hardware: %v", st.Waiting))
|
||||
}
|
||||
head += "\n\n"
|
||||
if len(st.Nodes) == 0 {
|
||||
return "Slurm", head + " " + sDim.Render("No node runs slurmd."), keys
|
||||
}
|
||||
var rows [][]string
|
||||
for _, n := range st.Nodes {
|
||||
rows = append(rows, slurmRow(n))
|
||||
}
|
||||
return "Slurm", head + table([]string{"NODE", "WHOSE", "IN SLURM", "JOBS", "SLURMD", "SHARED", "NOTE"}, rows, m.row, pw, m.inPanel), keys
|
||||
}
|
||||
|
||||
func slurmRow(n slurmsvc.NodeStatus) []string {
|
||||
state, jobs, slurmd, shared, note := "-", "-", "-", "-", n.Describe()
|
||||
if r := n.Status; r != nil {
|
||||
state = or(r.State, "-")
|
||||
switch r.Shared {
|
||||
case "":
|
||||
case "missing":
|
||||
shared = sBad.Render(r.Shared)
|
||||
default:
|
||||
shared = sOK.Render(r.Shared)
|
||||
}
|
||||
jobs = fmt.Sprint(r.Jobs)
|
||||
slurmd = sBad.Render("stopped")
|
||||
if r.SlurmdRunning {
|
||||
slurmd = sOK.Render("running")
|
||||
}
|
||||
if r.Error != "" {
|
||||
note = sBad.Render(r.Error)
|
||||
}
|
||||
}
|
||||
if !n.Connected {
|
||||
note = sWarn.Render("agent not connected") + "; " + note
|
||||
}
|
||||
whose := string(n.Phase)
|
||||
switch n.Phase {
|
||||
case slurmsvc.Lent:
|
||||
whose = sOK.Render("Slurm's")
|
||||
case slurmsvc.Llmgw:
|
||||
whose = "llmgw's"
|
||||
case slurmsvc.Lending, slurmsvc.Reclaiming:
|
||||
whose = sWarn.Render(whose)
|
||||
}
|
||||
return []string{n.Node, whose, state, jobs, slurmd, shared, note}
|
||||
}
|
||||
|
||||
// slurmMenu is what can be done with one of Slurm's nodes.
|
||||
func (m consoleModel) slurmMenu(n slurmsvc.NodeStatus) *menuDialog {
|
||||
var items []menuItem
|
||||
if n.Phase == slurmsvc.Lent || n.Phase == slurmsvc.Lending {
|
||||
items = append(items, menuItem{"Take back from Slurm…", func(m *consoleModel) tea.Cmd {
|
||||
m.confirmD = &confirmDialog{title: "Take " + n.Node + " back",
|
||||
text: "Slurm starts no new job there. llmgw places replicas on it again once the jobs running there have ended.",
|
||||
yes: func(m *consoleModel) tea.Cmd { return m.slurmLend(n.Node, false) }}
|
||||
return nil
|
||||
}})
|
||||
} else {
|
||||
items = append(items, menuItem{"Lend to Slurm…", func(m *consoleModel) tea.Cmd {
|
||||
text := "Its replicas move off first, each replacement up before the original stops; then Slurm runs jobs there."
|
||||
if n.Replicas > 0 {
|
||||
text = fmt.Sprintf("Its %d replica(s) move off first, each replacement up before the original stops; "+
|
||||
"a model with nowhere else to go stops serving. Then Slurm runs jobs there.", n.Replicas)
|
||||
}
|
||||
m.confirmD = &confirmDialog{title: "Lend " + n.Node + " to Slurm", text: text,
|
||||
yes: func(m *consoleModel) tea.Cmd { return m.slurmLend(n.Node, true) }}
|
||||
return nil
|
||||
}})
|
||||
}
|
||||
return &menuDialog{title: n.Node, items: items}
|
||||
}
|
||||
|
||||
// slurmLend puts a node in slurm.lent, or takes it out, as `llmgw ctl slurm
|
||||
// lend|reclaim` does: the newest settings, changed and put back.
|
||||
func (m *consoleModel) slurmLend(node string, add bool) tea.Cmd {
|
||||
title, note, done := "Lend "+node, "lent to Slurm from the console", node+" is being lent to Slurm: its replicas move off first."
|
||||
if !add {
|
||||
title, note, done = "Take back "+node, "taken back from Slurm in the console",
|
||||
node+" is being taken back: llmgw has it once its Slurm jobs end."
|
||||
}
|
||||
m.busy = title
|
||||
c := m.client
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
var cur adminapi.SettingsView
|
||||
if err := c.get(ctx, "/api/v1/settings", &cur); err != nil {
|
||||
return actionDoneMsg{title: title, err: err}
|
||||
}
|
||||
doc, err := settings.FromBytes("settings", []byte(cur.Body))
|
||||
if err == nil {
|
||||
err = lendChange(doc, []string{node}, add)
|
||||
}
|
||||
var body []byte
|
||||
if err == nil {
|
||||
body, err = doc.Bytes()
|
||||
}
|
||||
if err != nil {
|
||||
return actionDoneMsg{title: title, err: err}
|
||||
}
|
||||
req, _ := json.Marshal(adminapi.SettingsRequest{Base: cur.Version, Body: string(body), Note: note})
|
||||
var v adminapi.SettingsView
|
||||
if err := c.put(ctx, "/api/v1/settings", req, &v); err != nil {
|
||||
return actionDoneMsg{title: title, err: err}
|
||||
}
|
||||
return actionDoneMsg{title: title, text: done}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
)
|
||||
|
||||
func TestTheConsoleLendsANodeToSlurmFromItsRow(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
var put adminapi.SettingsRequest
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v1/settings":
|
||||
_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 7,
|
||||
Body: "slurm:\n enabled: true\n controller: node06\n nodes: [node05, cuda01]\n"})
|
||||
case r.Method == http.MethodPut && r.URL.Path == "/api/v1/settings":
|
||||
b, _ := io.ReadAll(r.Body)
|
||||
mu.Lock()
|
||||
_ = json.Unmarshal(b, &put)
|
||||
mu.Unlock()
|
||||
_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 8})
|
||||
default:
|
||||
_, _ = w.Write([]byte("[]"))
|
||||
}
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
m := newConsoleModel(srv.URL, "/nonexistent/server.yaml")
|
||||
m.loaded, m.menu, m.inPanel = true, cSlurm, true
|
||||
m.data.slurm = slurmsvc.Status{Enabled: true, Cluster: "llmgw", Controller: "node06", Partition: "gpu",
|
||||
Ctld: &slurmsvc.Reported{SlurmctldRunning: true, SlurmdbdRunning: true, Version: "23.11.4"},
|
||||
Shared: config.SharedStorage{Path: "/shared", Server: "node06"}, Users: "llmgw", UserCount: 2,
|
||||
Nodes: []slurmsvc.NodeStatus{
|
||||
{Node: "node05", Phase: slurmsvc.Lent, Connected: true,
|
||||
Status: &slurmsvc.Reported{State: "MIXED", Jobs: 2, SlurmdRunning: true, Shared: "mounted"}},
|
||||
{Node: "cuda01", Phase: slurmsvc.Llmgw, Replicas: 3, Connected: true,
|
||||
Status: &slurmsvc.Reported{State: "IDLE+DRAIN", SlurmdRunning: true}},
|
||||
}}
|
||||
var tm tea.Model = m
|
||||
view := tm.View()
|
||||
for _, want := range []string{"slurmctld 23.11.4 and slurmdbd running", "/shared, exported over NFS by node06", "users: 2", "node05", "Slurm's", "MIXED", "cuda01", "llmgw's"} {
|
||||
if !strings.Contains(view, want) {
|
||||
t.Errorf("the Slurm view has no %q:\n%s", want, view)
|
||||
}
|
||||
}
|
||||
|
||||
// cuda01, the second row: lent, after saying its replicas move.
|
||||
tm = press(tm, "down", "enter")
|
||||
if !strings.Contains(tm.View(), "Lend to Slurm") {
|
||||
t.Fatalf("no lend in the menu:\n%s", tm.View())
|
||||
}
|
||||
tm = press(tm, "enter")
|
||||
if !strings.Contains(tm.View(), "3 replica(s) move off first") {
|
||||
t.Fatalf("the question does not say what moves:\n%s", tm.View())
|
||||
}
|
||||
tm = pressRun(tm, "y")
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if put.Base != 7 || !strings.Contains(put.Body, "lent:") || !strings.Contains(put.Body, "cuda01") {
|
||||
t.Fatalf("the settings put back: %+v", put)
|
||||
}
|
||||
if !strings.Contains(tm.View(), "cuda01 is being lent to Slurm") {
|
||||
t.Errorf("not said:\n%s", tm.View())
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheConsoleTakesALentNodeBack(t *testing.T) {
|
||||
m, _ := consoleWith(t, cSlurm)
|
||||
m.data.slurm = slurmsvc.Status{Enabled: true, Controller: "node06",
|
||||
Nodes: []slurmsvc.NodeStatus{{Node: "node05", Phase: slurmsvc.Lent, Connected: true}}}
|
||||
var tm tea.Model = m
|
||||
tm = press(tm, "enter")
|
||||
if v := tm.View(); !strings.Contains(v, "Take back from Slurm") || strings.Contains(v, "Lend to Slurm") {
|
||||
t.Fatalf("a lent node's menu:\n%s", v)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheConsoleSaysHowToTurnSlurmOn(t *testing.T) {
|
||||
m, _ := consoleWith(t, cSlurm)
|
||||
var tm tea.Model = m
|
||||
if !strings.Contains(tm.View(), "llmgw ctl slurm setup") {
|
||||
t.Fatalf("Slurm off says nothing of turning it on:\n%s", tm.View())
|
||||
}
|
||||
}
|
||||
@@ -171,7 +171,10 @@ func newUserStateCmd(open userOpener, verb string, state store.UserState, short
|
||||
if err := s.DeleteUserSessions(cmd.Context(), u.ID); err != nil {
|
||||
return err
|
||||
}
|
||||
cmd.PrintErrln("Their sessions were closed. Their API keys still work — revoke those separately.")
|
||||
// The keys are refused by their owner's state, which every
|
||||
// router reads again within auth.cache_ttl.
|
||||
cmd.PrintErrln("Their sessions were closed. Their API keys are refused within auth.cache_ttl " +
|
||||
"(30 s by default), and work again if they are reinstated.")
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s is now %s\n", u.Email, state)
|
||||
return nil
|
||||
@@ -391,7 +394,8 @@ func newUserJoinCmd(open userOpener) *cobra.Command {
|
||||
Long: "A person can be a member of any number of them. A member of a team or\n" +
|
||||
"organization may use every project in it.",
|
||||
Example: " llmgw users join ayse@example.com --team Acme/ml\n" +
|
||||
" llmgw users join ayse@example.com --project Scania/research/vision",
|
||||
" llmgw users join ayse@example.com --project Scania/research/vision\n" +
|
||||
" llmgw users join lars@example.com --org Scania --role admin",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
s, err := open(cmd)
|
||||
@@ -415,7 +419,8 @@ func newUserJoinCmd(open userOpener) *cobra.Command {
|
||||
},
|
||||
}
|
||||
where.bind(cmd)
|
||||
cmd.Flags().StringVar(&role, "role", store.RoleMember, "member, or manager: with --org, may see the organization's page")
|
||||
cmd.Flags().StringVar(&role, "role", store.RoleMember,
|
||||
"member; with --org also manager, who sees the organization's page, or admin, who also changes its teams, projects, people and keys")
|
||||
cmd.Flags().StringVar(&by, "by", "", "who added them, for the record")
|
||||
return cmd
|
||||
}
|
||||
|
||||
@@ -158,6 +158,10 @@ type Server struct {
|
||||
// Firewall turns llmgw's firewall on every machine on or off.
|
||||
Firewall FirewallSettings `koanf:"firewall"`
|
||||
|
||||
// Slurm is a Slurm cluster run beside llmgw on some of its nodes, and
|
||||
// which of them it may use now.
|
||||
Slurm SlurmSettings `koanf:"slurm"`
|
||||
|
||||
// Presets is how the models that come with llmgw reach the catalog.
|
||||
Presets PresetSettings `koanf:"presets"`
|
||||
|
||||
@@ -960,6 +964,9 @@ func (s Server) Validate() error {
|
||||
if _, err := s.Facility.Policy(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.Slurm.Check(); err != nil {
|
||||
return err
|
||||
}
|
||||
if s.Control.Timezone != "" {
|
||||
if _, err := time.LoadLocation(s.Control.Timezone); err != nil {
|
||||
return fmt.Errorf("control.timezone %q: %w", s.Control.Timezone, err)
|
||||
|
||||
@@ -0,0 +1,204 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// SlurmSettings is a Slurm cluster run beside llmgw on some of its nodes
|
||||
// (BACKLOG P), for training and other batch work llmgw does not schedule.
|
||||
// The leader tells each named node's agent what to run; the agents install
|
||||
// it from the machine's package sources and keep it going.
|
||||
type SlurmSettings struct {
|
||||
// Enabled runs Slurm on the nodes below. False stops llmgw keeping it;
|
||||
// what was installed is left installed, and every node is llmgw's.
|
||||
Enabled bool `koanf:"enabled"`
|
||||
|
||||
// ClusterName is Slurm's ClusterName. Empty is "llmgw".
|
||||
ClusterName string `koanf:"cluster_name"`
|
||||
|
||||
// Controller is the llmgw node whose agent runs slurmctld. It needs no
|
||||
// GPU, and need not be one of Nodes.
|
||||
Controller string `koanf:"controller"`
|
||||
|
||||
// Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained
|
||||
// in Slurm, until it is also in Lent.
|
||||
Nodes []string `koanf:"nodes"`
|
||||
|
||||
// AllNodes runs slurmd on every llmgw node, those that join later too,
|
||||
// besides any named in Nodes. Each is llmgw's until it is lent, as any.
|
||||
AllNodes bool `koanf:"all_nodes"`
|
||||
|
||||
// Lent are the nodes of Nodes Slurm may run jobs on now. A node put here
|
||||
// is drained of llmgw's replicas first, as `ctl drain` does, and resumed
|
||||
// in Slurm once none is left; a node taken out is drained in Slurm and
|
||||
// given back to llmgw when its last job ends.
|
||||
Lent []string `koanf:"lent"`
|
||||
|
||||
// Partition is the one partition every node is in. Empty is "gpu".
|
||||
Partition string `koanf:"partition"`
|
||||
|
||||
// Shared is the file system every machine of the Slurm cluster sees at
|
||||
// the same path: home directories, datasets, the jobs' output and
|
||||
// checkpoints. Empty path: none, and a batch job's output stays on the
|
||||
// node it ran on.
|
||||
Shared SharedStorage `koanf:"shared"`
|
||||
|
||||
// Users is who keeps the Linux users that run jobs, the same on every
|
||||
// machine of the Slurm cluster: "llmgw" makes one for each active llmgw
|
||||
// account (its login from the e-mail address, a UID of llmgw's from
|
||||
// 200000 up, its SSH keys from the portal), let in to the controller,
|
||||
// and to a node only while a job of theirs runs there (pam_slurm_adopt);
|
||||
// empty leaves users to the site (a directory through sssd, or by hand).
|
||||
Users string `koanf:"users"`
|
||||
|
||||
// Containers lets jobs run in containers: `srun --container-image
|
||||
// nvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),
|
||||
// the GPUs and the shared path inside. The agents install both on
|
||||
// every Slurm machine; jobs take it once every machine has them.
|
||||
Containers bool `koanf:"containers"`
|
||||
}
|
||||
|
||||
// SharedStorage is where the Slurm cluster's shared files come from: the
|
||||
// site's NFS export (NFS), an NFS export llmgw keeps on one of its nodes
|
||||
// (Server), or, with neither, a file system the site mounts itself
|
||||
// (Lustre, BeeGFS, GPFS), which llmgw only looks for.
|
||||
type SharedStorage struct {
|
||||
// Path is where every machine sees it, as /shared.
|
||||
Path string `koanf:"path"`
|
||||
// NFS is the site's export, host:/path, mounted at Path on every
|
||||
// machine of the Slurm cluster.
|
||||
NFS string `koanf:"nfs"`
|
||||
// Server is the llmgw node, the controller or one of the nodes, that
|
||||
// exports Path over NFS to the others: for a site with no storage of
|
||||
// its own. One machine: not highly available, and no faster than its
|
||||
// disk and network.
|
||||
Server string `koanf:"server"`
|
||||
}
|
||||
|
||||
// UsersLlmgw has llmgw make the Slurm cluster's users from its accounts.
|
||||
const UsersLlmgw = "llmgw"
|
||||
|
||||
// HomeDir is where a user's home directory is: under the shared path when
|
||||
// there is one, else on each machine's own disk.
|
||||
func (s SlurmSettings) HomeDir(login string) string {
|
||||
if s.Shared.Path != "" {
|
||||
return path.Join(s.Shared.Path, "home", login)
|
||||
}
|
||||
return path.Join("/home", login)
|
||||
}
|
||||
|
||||
// nfsExport is host:/path.
|
||||
var nfsExport = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.:\[\]-]*:/[A-Za-z0-9_./-]*$`)
|
||||
|
||||
// sharedPath is an absolute path of plain names, which a mount unit's name
|
||||
// is made from.
|
||||
var sharedPath = regexp.MustCompile(`^(/[A-Za-z0-9_.-]+)+$`)
|
||||
|
||||
func (s SlurmSettings) checkShared() error {
|
||||
sh := s.Shared
|
||||
if sh.Path == "" {
|
||||
if sh.NFS != "" || sh.Server != "" {
|
||||
return fmt.Errorf("slurm.shared.path is empty: say where every machine sees the shared files, as /shared")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if !sharedPath.MatchString(sh.Path) || strings.Contains(sh.Path, "/.") {
|
||||
return fmt.Errorf("slurm.shared.path %q is not an absolute path of plain names, as /shared", sh.Path)
|
||||
}
|
||||
for _, sys := range []string{"/", "/etc", "/usr", "/var", "/var/lib", "/home", "/root", "/boot", "/opt", "/srv", "/tmp", "/proc", "/sys", "/dev", "/run"} {
|
||||
if path.Clean(sh.Path) == sys {
|
||||
return fmt.Errorf("slurm.shared.path %s is the system's own directory: take one of its own, as /shared", sh.Path)
|
||||
}
|
||||
}
|
||||
if sh.NFS != "" && sh.Server != "" {
|
||||
return fmt.Errorf("slurm.shared names both the site's export (nfs) and an llmgw node to export it (server): one of them")
|
||||
}
|
||||
if sh.NFS != "" && !nfsExport.MatchString(sh.NFS) {
|
||||
return fmt.Errorf("slurm.shared.nfs %q is not host:/path", sh.NFS)
|
||||
}
|
||||
if sh.Server != "" && sh.Server != s.Controller && !s.AllNodes && !slices.Contains(s.Nodes, sh.Server) {
|
||||
return fmt.Errorf("slurm.shared.server %q is neither the controller nor one of slurm.nodes", sh.Server)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// slurmName is what Slurm takes as a node, cluster or partition name.
|
||||
var slurmName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]{0,62}$`)
|
||||
|
||||
// Cluster is ClusterName or its default.
|
||||
func (s SlurmSettings) Cluster() string {
|
||||
if s.ClusterName == "" {
|
||||
return "llmgw"
|
||||
}
|
||||
return s.ClusterName
|
||||
}
|
||||
|
||||
// PartitionName is Partition or its default.
|
||||
func (s SlurmSettings) PartitionName() string {
|
||||
if s.Partition == "" {
|
||||
return "gpu"
|
||||
}
|
||||
return s.Partition
|
||||
}
|
||||
|
||||
// IsLent is whether Slurm may use the node now.
|
||||
func (s SlurmSettings) IsLent(node string) bool {
|
||||
return s.Enabled && (s.AllNodes || slices.Contains(s.Nodes, node)) && slices.Contains(s.Lent, node)
|
||||
}
|
||||
|
||||
// WithNodes is the settings with every node of ids in Nodes when AllNodes:
|
||||
// the nodes that run slurmd, as the leader's inventory has them.
|
||||
func (s SlurmSettings) WithNodes(ids []string) SlurmSettings {
|
||||
if !s.AllNodes {
|
||||
return s
|
||||
}
|
||||
nodes := slices.Clone(s.Nodes)
|
||||
for _, id := range ids {
|
||||
if !slices.Contains(nodes, id) {
|
||||
nodes = append(nodes, id)
|
||||
}
|
||||
}
|
||||
s.Nodes = nodes
|
||||
return s
|
||||
}
|
||||
|
||||
// Check refuses settings Slurm could not run on.
|
||||
func (s SlurmSettings) Check() error {
|
||||
if !s.Enabled {
|
||||
return nil
|
||||
}
|
||||
if s.Controller == "" {
|
||||
return fmt.Errorf("slurm.controller is empty: name the node that runs slurmctld")
|
||||
}
|
||||
if !slurmName.MatchString(s.Controller) {
|
||||
return fmt.Errorf("slurm.controller %q is not a name Slurm takes", s.Controller)
|
||||
}
|
||||
for _, n := range []struct{ key, v string }{{"slurm.cluster_name", s.ClusterName}, {"slurm.partition", s.Partition}} {
|
||||
if n.v != "" && !slurmName.MatchString(n.v) {
|
||||
return fmt.Errorf("%s %q is not a name Slurm takes: letters, digits, _ . -", n.key, n.v)
|
||||
}
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for i, n := range s.Nodes {
|
||||
if !slurmName.MatchString(n) {
|
||||
return fmt.Errorf("slurm.nodes[%d] %q is not a name Slurm takes", i, n)
|
||||
}
|
||||
if seen[n] {
|
||||
return fmt.Errorf("slurm.nodes names %s twice", n)
|
||||
}
|
||||
seen[n] = true
|
||||
}
|
||||
for i, n := range s.Lent {
|
||||
if !seen[n] && !s.AllNodes {
|
||||
return fmt.Errorf("slurm.lent[%d] %q is not in slurm.nodes: only a node that runs slurmd can be lent", i, n)
|
||||
}
|
||||
}
|
||||
if s.Users != "" && s.Users != UsersLlmgw {
|
||||
return fmt.Errorf("slurm.users %q: %q has llmgw make them, empty leaves them to the site", s.Users, UsersLlmgw)
|
||||
}
|
||||
return s.checkShared()
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSlurmCheck(t *testing.T) {
|
||||
ok := SlurmSettings{Enabled: true, Controller: "node06", Nodes: []string{"cuda01", "site2"}, Lent: []string{"cuda01"}}
|
||||
if err := ok.Check(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !ok.IsLent("cuda01") || ok.IsLent("site2") || ok.IsLent("node06") {
|
||||
t.Error("IsLent is wrong")
|
||||
}
|
||||
if ok.Cluster() != "llmgw" || ok.PartitionName() != "gpu" {
|
||||
t.Error("defaults are wrong")
|
||||
}
|
||||
if (SlurmSettings{Lent: []string{"x"}}).Check() != nil {
|
||||
t.Error("settings that are off were checked")
|
||||
}
|
||||
for want, s := range map[string]SlurmSettings{
|
||||
"controller is empty": {Enabled: true},
|
||||
"not a name": {Enabled: true, Controller: "bad name"},
|
||||
"twice": {Enabled: true, Controller: "c", Nodes: []string{"a", "a"}},
|
||||
"not in slurm.nodes": {Enabled: true, Controller: "c", Nodes: []string{"a"}, Lent: []string{"b"}},
|
||||
"slurm.partition": {Enabled: true, Controller: "c", Partition: "a b"},
|
||||
} {
|
||||
if err := s.Check(); err == nil || !strings.Contains(err.Error(), want) {
|
||||
t.Errorf("%+v: got %v, want %q", s, err, want)
|
||||
}
|
||||
}
|
||||
if (SlurmSettings{Enabled: false, Lent: []string{"a"}}).IsLent("a") {
|
||||
t.Error("a node is lent with Slurm off")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlurmShared(t *testing.T) {
|
||||
base := SlurmSettings{Enabled: true, Controller: "c", Nodes: []string{"a"}}
|
||||
for _, sh := range []SharedStorage{{}, {Path: "/shared"}, {Path: "/shared", NFS: "filer.site:/export/hpc"},
|
||||
{Path: "/data/hpc", Server: "c"}, {Path: "/shared", Server: "a"}} {
|
||||
s := base
|
||||
s.Shared = sh
|
||||
if err := s.Check(); err != nil {
|
||||
t.Errorf("%+v refused: %v", sh, err)
|
||||
}
|
||||
}
|
||||
for want, sh := range map[string]SharedStorage{
|
||||
"path is empty": {NFS: "f:/x"},
|
||||
"plain names": {Path: "shared"},
|
||||
"system's own": {Path: "/home"},
|
||||
"one of them": {Path: "/shared", NFS: "f:/x", Server: "c"},
|
||||
"not host:/path": {Path: "/shared", NFS: "f/x"},
|
||||
"neither": {Path: "/shared", Server: "z"},
|
||||
} {
|
||||
s := base
|
||||
s.Shared = sh
|
||||
if err := s.Check(); err == nil || !strings.Contains(err.Error(), want) {
|
||||
t.Errorf("%+v: got %v, want %q", sh, err, want)
|
||||
}
|
||||
}
|
||||
s := base
|
||||
s.Shared = SharedStorage{Path: "/a/../etc"}
|
||||
if s.Check() == nil {
|
||||
t.Error("a path with .. taken")
|
||||
}
|
||||
s.Shared, s.Users = SharedStorage{}, "ldap"
|
||||
if s.Check() == nil {
|
||||
t.Error("slurm.users ldap taken")
|
||||
}
|
||||
s.Users, s.Shared = UsersLlmgw, SharedStorage{Path: "/shared"}
|
||||
if s.Check() != nil || s.HomeDir("ayse") != "/shared/home/ayse" {
|
||||
t.Errorf("home %s", s.HomeDir("ayse"))
|
||||
}
|
||||
s.Shared = SharedStorage{}
|
||||
if s.HomeDir("ayse") != "/home/ayse" {
|
||||
t.Errorf("home %s", s.HomeDir("ayse"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlurmAllNodes(t *testing.T) {
|
||||
s := SlurmSettings{Enabled: true, Controller: "c", AllNodes: true, Lent: []string{"joined"}}
|
||||
if err := s.Check(); err != nil {
|
||||
t.Fatalf("a node lent with all_nodes refused: %v", err)
|
||||
}
|
||||
if !s.IsLent("joined") {
|
||||
t.Fatal("not lent with all_nodes")
|
||||
}
|
||||
if got := s.WithNodes([]string{"a", "joined"}).Nodes; len(got) != 2 {
|
||||
t.Fatalf("nodes %v", got)
|
||||
}
|
||||
s.AllNodes = false
|
||||
if s.Check() == nil || len(s.WithNodes([]string{"a"}).Nodes) != 0 {
|
||||
t.Fatal("all_nodes off still takes the inventory's nodes")
|
||||
}
|
||||
}
|
||||
@@ -78,7 +78,9 @@ type API struct {
|
||||
// tenancy is organizations, keys and people; nil without a store.
|
||||
tenancy TenancyStore
|
||||
// train is adapter training; nil without a store and agents.
|
||||
train *TrainAdmin
|
||||
train *TrainAdmin
|
||||
// slurm is the Slurm cluster beside llmgw; nil without a store and agents.
|
||||
slurm SlurmService
|
||||
settings *Settings
|
||||
auth *AuthAdmin
|
||||
mcp MCPStore
|
||||
@@ -96,6 +98,10 @@ type API struct {
|
||||
usage Usage
|
||||
quotas Quotas
|
||||
ctl *control.Controller
|
||||
// slurmUsers are the Linux users made from the accounts; nil without a store.
|
||||
slurmUsers SlurmUsers
|
||||
// webhooks counts the settings' notify.webhooks; nil does not say.
|
||||
webhooks func() int
|
||||
}
|
||||
|
||||
// New builds the API over a controller.
|
||||
@@ -136,6 +142,7 @@ func (a *API) Register(mux Mux) {
|
||||
a.registerKB(mux)
|
||||
a.registerTenancy(mux)
|
||||
a.registerTrain(mux)
|
||||
a.registerSlurm(mux)
|
||||
a.registerGuard(mux)
|
||||
a.registerSettings(mux)
|
||||
a.registerAuth(mux)
|
||||
|
||||
@@ -6,6 +6,10 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
|
||||
)
|
||||
|
||||
// SetWebhooks has the summary say when the settings name no webhook:
|
||||
// count is how many they name now.
|
||||
func (a *API) SetWebhooks(count func() int) { a.webhooks = count }
|
||||
|
||||
// SetNotifier gives the API the webhooks `ctl notify test` reaches.
|
||||
func (a *API) SetNotifier(n *notify.Notifier) { a.notifier = n }
|
||||
|
||||
|
||||
@@ -216,7 +216,7 @@ func (a *API) purgeFromNodes(ctx context.Context, sources []string, others []cat
|
||||
names[filepath.Base(s)] = true
|
||||
}
|
||||
wanted := inUse(others)
|
||||
var out []CollectResult
|
||||
var rs []removal
|
||||
for _, n := range a.ctl.Inventory().Nodes() {
|
||||
size := map[string]int64{}
|
||||
have := make([]string, 0, len(n.Weights))
|
||||
@@ -235,15 +235,14 @@ func (a *API) purgeFromNodes(ctx context.Context, sources []string, others []cat
|
||||
if len(paths) == 0 {
|
||||
continue
|
||||
}
|
||||
res := CollectResult{Node: n.ID, Files: len(paths)}
|
||||
for _, p := range paths {
|
||||
res.Bytes += size[p]
|
||||
rs = append(rs, removal{node: n.ID, paths: paths, size: size})
|
||||
}
|
||||
out := a.removeConfirmed(ctx, rm, rs)
|
||||
for _, r := range out {
|
||||
if r.Error != "" || r.Left > 0 {
|
||||
slog.Warn("a purged model's copies were not all deleted on a node", "node", r.Node,
|
||||
"left", r.Left, "err", r.Error)
|
||||
}
|
||||
if err := rm.RemoveWeights(ctx, n.ID, paths); err != nil {
|
||||
res.Error, res.Files, res.Bytes = err.Error(), 0, 0
|
||||
slog.Warn("a purged model's copies could not be deleted on a node", "node", n.ID, "err", err)
|
||||
}
|
||||
out = append(out, res)
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/federation"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
@@ -84,6 +85,7 @@ var PathParams = map[string]string{
|
||||
"projects/{id}": "The project's ID.",
|
||||
"keys/{id}": "The key's ID, as listed (not the key itself).",
|
||||
"users/{id}": "The person's ID or email.",
|
||||
"users/{user}": "The person's ID or email.",
|
||||
"artifacts/{kind}": "What the job made: peft, a gzipped tar of the PEFT adapter, or gguf, the adapter for llama.cpp.",
|
||||
}
|
||||
|
||||
@@ -392,6 +394,20 @@ var Routes = []Route{
|
||||
Description: "Their keys in projects they no longer reach are revoked.",
|
||||
Request: MembershipRequest{}, Response: UserView{}},
|
||||
|
||||
{Method: "GET", Path: "/api/v1/slurm", Summary: "The Slurm cluster beside llmgw, and where each of its nodes is",
|
||||
Description: "From the cluster's settings (slurm.*) and what each node's agent last said. A node's phase is llmgw " +
|
||||
"(drained in Slurm, no job left), lending (its replicas moving off), lent, reclaiming (drained in Slurm, jobs " +
|
||||
"still running) or unknown (its agent has not said).",
|
||||
Response: slurmsvc.Status{}},
|
||||
{Method: "GET", Path: "/api/v1/slurm/users", Summary: "The Linux users made from llmgw's accounts for the Slurm cluster",
|
||||
Description: "Every active account, with its login, UID and SSH keys; an account is given its login and UID the first " +
|
||||
"time it is listed. The machines have them when slurm.users is \"llmgw\".",
|
||||
Response: []slurmsvc.User{}},
|
||||
{Method: "PUT", Path: "/api/v1/slurm/users/{user}/ssh-keys", Summary: "Replace an account's SSH public keys",
|
||||
Description: "{user} is an ID or e-mail. keys is OpenSSH public keys, one a line, without options; empty takes them " +
|
||||
"all away. The machines have them within a minute.",
|
||||
Request: SlurmKeysRequest{}, Response: slurmsvc.User{}},
|
||||
|
||||
{Method: "POST", Path: "/api/v1/train/jobs", Summary: "Queue a LoRA adapter's training on a node",
|
||||
Description: "The node trains when its agent is there, nothing else trains on it, and a card has gpu_memory_mib free; " +
|
||||
"the pairs are NDJSON of {\"q\", \"a\"}. An adapter learns how answers sound far better than what they say.",
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// SlurmService is the Slurm cluster beside llmgw (BACKLOG P), as the leader
|
||||
// keeps it.
|
||||
type SlurmService interface {
|
||||
Status() slurmsvc.Status
|
||||
UsersChanged()
|
||||
}
|
||||
|
||||
// SlurmUsers are the Linux users llmgw makes from its accounts.
|
||||
type SlurmUsers interface {
|
||||
SlurmUsers(ctx context.Context) ([]store.SlurmUser, error)
|
||||
SlurmUserByID(ctx context.Context, userID string) (store.SlurmUser, error)
|
||||
SetSlurmSSHKeys(ctx context.Context, userID string, keys []string) error
|
||||
}
|
||||
|
||||
// SetSlurmUsers enables the Slurm users' endpoints.
|
||||
func (a *API) SetSlurmUsers(u SlurmUsers) { a.slurmUsers = u }
|
||||
|
||||
// SlurmKeysRequest is a person's SSH public keys, one a line, as
|
||||
// authorized_keys has them; empty takes them all away.
|
||||
type SlurmKeysRequest struct {
|
||||
Keys string `json:"keys"`
|
||||
}
|
||||
|
||||
// SlurmUserView is the user from a store row.
|
||||
func SlurmUserView(u store.SlurmUser) slurmsvc.User {
|
||||
return slurmsvc.User{Login: u.Login, UID: u.UID, Name: u.Name, Email: u.Email, SSHKeys: u.SSHKeys}
|
||||
}
|
||||
|
||||
// SetSlurm gives the API the Slurm service.
|
||||
func (a *API) SetSlurm(s SlurmService) { a.slurm = s }
|
||||
|
||||
func (a *API) registerSlurm(mux Mux) {
|
||||
mux.HandleFunc("GET /api/v1/slurm", a.slurmStatus)
|
||||
mux.HandleFunc("GET /api/v1/slurm/users", a.slurmUserList)
|
||||
mux.HandleFunc("PUT /api/v1/slurm/users/{user}/ssh-keys", a.slurmUserKeys)
|
||||
}
|
||||
|
||||
func (a *API) slurmUserList(w http.ResponseWriter, r *http.Request) {
|
||||
if a.slurmUsers == nil {
|
||||
writeError(w, http.StatusNotImplemented, "the Slurm users need the store, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
us, err := a.slurmUsers.SlurmUsers(r.Context())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
out := make([]slurmsvc.User, 0, len(us))
|
||||
for _, u := range us {
|
||||
out = append(out, SlurmUserView(u))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
// slurmUserKeys replaces a person's SSH keys; {user} is an ID or e-mail.
|
||||
func (a *API) slurmUserKeys(w http.ResponseWriter, r *http.Request) {
|
||||
if a.slurmUsers == nil {
|
||||
writeError(w, http.StatusNotImplemented, "the Slurm users need the store, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
var in SlurmKeysRequest
|
||||
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 64<<10)).Decode(&in); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
keys, err := slurm.ParseSSHKeys(in.Keys)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
u, err := a.findUser(ctx, r.PathValue("user"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusNotFound, strings.TrimPrefix(err.Error(), store.ErrNotFound.Error()+": "))
|
||||
return
|
||||
}
|
||||
if err := a.slurmUsers.SetSlurmSSHKeys(ctx, u.ID, keys); err != nil {
|
||||
code := http.StatusInternalServerError
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
code = http.StatusNotFound
|
||||
}
|
||||
writeError(w, code, err.Error())
|
||||
return
|
||||
}
|
||||
if a.slurm != nil {
|
||||
a.slurm.UsersChanged()
|
||||
}
|
||||
su, err := a.slurmUsers.SlurmUserByID(ctx, u.ID)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, SlurmUserView(su))
|
||||
}
|
||||
|
||||
func (a *API) slurmStatus(w http.ResponseWriter, _ *http.Request) {
|
||||
if a.slurm == nil {
|
||||
writeError(w, http.StatusNotImplemented, "Slurm needs the store and the nodes' agents, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, a.slurm.Status())
|
||||
}
|
||||
@@ -50,6 +50,9 @@ type Summary struct {
|
||||
// GuardOff says the guardrails are turned off (guard.enabled: false):
|
||||
// no request is checked, whatever the policies say.
|
||||
GuardOff bool `json:"guard_off,omitempty"`
|
||||
// NoAlerts says the cluster's settings name no webhook: a model parked,
|
||||
// a node lost or a judge down is told nobody.
|
||||
NoAlerts bool `json:"no_alerts,omitempty"`
|
||||
}
|
||||
|
||||
// StoreMember is one member of a replicated store, as its Patroni reports it.
|
||||
@@ -160,6 +163,9 @@ func (a *API) Summary() Summary {
|
||||
if a.guard != nil && a.guard.Library != nil && !a.guard.Library().On() {
|
||||
s.GuardOff = true
|
||||
}
|
||||
if a.webhooks != nil && a.webhooks() == 0 {
|
||||
s.NoAlerts = true
|
||||
}
|
||||
for _, j := range a.Judges() {
|
||||
if j.Problem != "" {
|
||||
s.Judges = append(s.Judges, j)
|
||||
|
||||
@@ -153,3 +153,16 @@ func TestOneSharedCopyCountsAsMuchAsTwenty(t *testing.T) {
|
||||
t.Errorf("counted %d models; three nodes seeing one file is one model", f.Models)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheSummarySaysWhenNoWebhookIsSet(t *testing.T) {
|
||||
a, _ := setup(t, testModels(), gpuNode("n1", 8))
|
||||
n := 0
|
||||
a.SetWebhooks(func() int { return n })
|
||||
if !a.Summary().NoAlerts {
|
||||
t.Error("no webhook, not said")
|
||||
}
|
||||
n = 1
|
||||
if a.Summary().NoAlerts {
|
||||
t.Error("a webhook set, yet said there is none")
|
||||
}
|
||||
}
|
||||
@@ -634,8 +634,12 @@ func (a *API) membership(add bool) http.HandlerFunc {
|
||||
if role == "" {
|
||||
role = store.RoleMember
|
||||
}
|
||||
if role != store.RoleMember && role != store.RoleManager {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member or manager", role))
|
||||
if role != store.RoleMember && role != store.RoleManager && role != store.RoleOrgAdmin {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member, manager or admin", role))
|
||||
return
|
||||
}
|
||||
if role != store.RoleMember && scope != store.ScopeOrg {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("%s is a role on an organization", role))
|
||||
return
|
||||
}
|
||||
if err := a.tenancy.AddMembership(ctx, u.ID, scope, id, role, actorOf(r), time.Now()); err != nil {
|
||||
|
||||
@@ -1,11 +1,14 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
)
|
||||
|
||||
// OrphanView is cached weights on one node that no catalog model accounts for.
|
||||
@@ -71,12 +74,100 @@ type CollectRequest struct {
|
||||
Node string `json:"node,omitempty"`
|
||||
}
|
||||
|
||||
// CollectResult is what was deleted on one node, or why nothing was.
|
||||
// CollectResult is what was deleted on one node, or why nothing was: what
|
||||
// the node no longer lists once asked, not what it was asked to delete.
|
||||
type CollectResult struct {
|
||||
Node string `json:"node"`
|
||||
Files int `json:"files"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
Error string `json:"error,omitempty"`
|
||||
// Left and LeftBytes are what the node still lists: refused, or not
|
||||
// gone by the time the sweep stopped waiting. Its agent's log says why.
|
||||
Left int `json:"left,omitempty"`
|
||||
LeftBytes int64 `json:"left_bytes,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// confirmWait is how long a sweep waits for the nodes to list their files
|
||||
// without the ones deleted: they scan theirs every heartbeat, five seconds
|
||||
// by default, and a large set takes a moment to delete.
|
||||
var confirmWait = 30 * time.Second
|
||||
|
||||
// removal is what one node is asked to delete, with each file's size.
|
||||
type removal struct {
|
||||
node string
|
||||
paths []string
|
||||
size map[string]int64
|
||||
}
|
||||
|
||||
// removeConfirmed asks each node to delete its paths and says what each then
|
||||
// no longer lists. Counted as asked, a sweep said "25 GB freed" of files the
|
||||
// node had refused and kept.
|
||||
func (a *API) removeConfirmed(ctx context.Context, rm control.WeightRemover, rs []removal) []CollectResult {
|
||||
out := make([]CollectResult, len(rs))
|
||||
pending := map[int]bool{}
|
||||
for i, r := range rs {
|
||||
out[i].Node = r.node
|
||||
if err := rm.RemoveWeights(ctx, r.node, r.paths); err != nil {
|
||||
out[i].Error = err.Error()
|
||||
for _, p := range r.paths {
|
||||
out[i].Left++
|
||||
out[i].LeftBytes += r.size[p]
|
||||
}
|
||||
continue
|
||||
}
|
||||
pending[i] = true
|
||||
}
|
||||
deadline := time.Now().Add(confirmWait)
|
||||
for {
|
||||
for i := range pending {
|
||||
if left := a.stillListed(rs[i]); len(left) == 0 {
|
||||
delete(pending, i)
|
||||
}
|
||||
}
|
||||
if len(pending) == 0 || !time.Now().Before(deadline) {
|
||||
break
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
deadline = time.Now()
|
||||
case <-time.After(time.Second):
|
||||
}
|
||||
}
|
||||
for i, r := range rs {
|
||||
if out[i].Error != "" {
|
||||
continue
|
||||
}
|
||||
left := a.stillListed(r)
|
||||
for _, p := range r.paths {
|
||||
if left[p] {
|
||||
out[i].Left++
|
||||
out[i].LeftBytes += r.size[p]
|
||||
} else {
|
||||
out[i].Files++
|
||||
out[i].Bytes += r.size[p]
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// stillListed is which of a removal's paths its node still lists.
|
||||
func (a *API) stillListed(r removal) map[string]bool {
|
||||
left := map[string]bool{}
|
||||
n, ok := a.ctl.Inventory().Get(r.node)
|
||||
if !ok {
|
||||
return left
|
||||
}
|
||||
want := map[string]bool{}
|
||||
for _, p := range r.paths {
|
||||
want[p] = true
|
||||
}
|
||||
for _, w := range n.Weights {
|
||||
if want[w.Path] {
|
||||
left[w.Path] = true
|
||||
}
|
||||
}
|
||||
return left
|
||||
}
|
||||
|
||||
// collectWeights deletes them.
|
||||
@@ -97,20 +188,26 @@ func (a *API) collectWeights(w http.ResponseWriter, r *http.Request) {
|
||||
// An empty body is the "all nodes" case and must not be an error.
|
||||
_ = json.NewDecoder(r.Body).Decode(&body)
|
||||
|
||||
out := make([]CollectResult, 0)
|
||||
// A node that has gone since the list was built is reported rather than
|
||||
// failing the whole sweep: the other nodes' space is still worth
|
||||
// freeing, and this one will be offered again next time.
|
||||
var rs []removal
|
||||
for _, o := range a.Orphans() {
|
||||
if body.Node != "" && o.Node != body.Node {
|
||||
continue
|
||||
}
|
||||
res := CollectResult{Node: o.Node, Files: len(o.Paths), Bytes: o.Bytes}
|
||||
// A node that has gone since the list was built is reported rather than
|
||||
// failing the whole sweep: the other nodes' space is still worth
|
||||
// freeing, and this one will be offered again next time.
|
||||
if err := rm.RemoveWeights(r.Context(), o.Node, o.Paths); err != nil {
|
||||
res.Error = err.Error()
|
||||
res.Files, res.Bytes = 0, 0
|
||||
}
|
||||
out = append(out, res)
|
||||
rs = append(rs, removal{node: o.Node, paths: o.Paths, size: a.sizesOn(o.Node)})
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
writeJSON(w, http.StatusOK, a.removeConfirmed(r.Context(), rm, rs))
|
||||
}
|
||||
|
||||
// sizesOn is each file a node lists, by path, with its size.
|
||||
func (a *API) sizesOn(node string) map[string]int64 {
|
||||
out := map[string]int64{}
|
||||
if n, ok := a.ctl.Inventory().Get(node); ok {
|
||||
for _, w := range n.Weights {
|
||||
out[w.Path] = w.SizeBytes
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
// deletingDispatcher deletes what it is asked to as a node would, by its
|
||||
// files dropping out of what the node lists — but for refused, which it
|
||||
// keeps, as an agent keeps a file it cannot prove it fetched.
|
||||
type deletingDispatcher struct {
|
||||
nullDispatcher
|
||||
inv *inventory.Store
|
||||
refused string
|
||||
}
|
||||
|
||||
func (d *deletingDispatcher) RemoveWeights(_ context.Context, node string, paths []string) error {
|
||||
n, _ := d.inv.Get(node)
|
||||
n.Weights = slices.DeleteFunc(n.Weights, func(w discovery.LocalWeights) bool {
|
||||
return w.Path != d.refused && slices.Contains(paths, w.Path)
|
||||
})
|
||||
d.inv.Upsert(n)
|
||||
return nil
|
||||
}
|
||||
|
||||
// What a sweep says it freed is what the node no longer lists: a file it
|
||||
// kept is said to be left, not counted as freed.
|
||||
func TestASweepSaysWhatWasDeletedNotWhatWasAsked(t *testing.T) {
|
||||
confirmWait = 1500 * time.Millisecond
|
||||
t.Cleanup(func() { confirmWait = 30 * time.Second })
|
||||
|
||||
gone := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/old/v1/old.gguf")
|
||||
set := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/big/v1")
|
||||
kept := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/stuck/v1/stuck.gguf")
|
||||
cat, _ := catalog.New(nil)
|
||||
d := &deletingDispatcher{refused: kept}
|
||||
ctl := control.New(control.DefaultConfig(), catalog.NewLive(cat), d, nullSink{}, slog.New(slog.DiscardHandler))
|
||||
d.inv = ctl.Inventory()
|
||||
node := gpuNode("n1", 1)
|
||||
node.Weights = []discovery.LocalWeights{{Path: gone, SizeBytes: 1 << 30}, {Path: set, SizeBytes: 20 << 30},
|
||||
{Path: kept, SizeBytes: 4 << 30}}
|
||||
ctl.Inventory().Upsert(node)
|
||||
a := New(ctl)
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
a.collectWeights(rec, httptest.NewRequest(http.MethodPost, "/api/v1/weights/collect", strings.NewReader(`{"node":"n1"}`)))
|
||||
var got []CollectResult
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
|
||||
t.Fatalf("%v: %s", err, rec.Body.String())
|
||||
}
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("results %+v", got)
|
||||
}
|
||||
r := got[0]
|
||||
if r.Files != 2 || r.Bytes != 21<<30 || r.Left != 1 || r.LeftBytes != 4<<30 {
|
||||
t.Errorf("result %+v: want 2 files and 21 GB freed, 1 file of 4 GB left", r)
|
||||
}
|
||||
}
|
||||
@@ -84,6 +84,8 @@ type Service struct {
|
||||
agentSettings func(node string) string
|
||||
// train is told how training jobs go; see SetTrain.
|
||||
train TrainHandler
|
||||
// slurm is told of the nodes' part in Slurm; see SetSlurm.
|
||||
slurm SlurmHandler
|
||||
|
||||
// logWaits are the log requests awaiting a reply, by request id.
|
||||
logMu sync.Mutex
|
||||
@@ -308,6 +310,10 @@ func (s *Service) Connect(
|
||||
return err
|
||||
}
|
||||
}
|
||||
// Its part in Slurm, sent by the Slurm service's next pass.
|
||||
if h := s.slurmHandler(); h != nil {
|
||||
h.OnRegister(nodeID)
|
||||
}
|
||||
|
||||
// Recover replica state from what this node is already running, before any
|
||||
// reconciliation pass can conclude the cluster is empty and place
|
||||
@@ -515,6 +521,11 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
|
||||
h.OnTrainStatus(nodeID, m.TrainStatus)
|
||||
}
|
||||
|
||||
case *llmgwv1.AgentMessage_SlurmStatus:
|
||||
if h := s.slurmHandler(); h != nil {
|
||||
h.OnSlurmStatus(nodeID, m.SlurmStatus)
|
||||
}
|
||||
|
||||
case *llmgwv1.AgentMessage_TrainArtifact:
|
||||
if h := s.trainHandler(); h != nil {
|
||||
h.OnTrainArtifact(nodeID, m.TrainArtifact)
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
package agentsvc
|
||||
|
||||
import (
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
)
|
||||
|
||||
// SlurmHandler is told of the nodes' part in Slurm (BACKLOG P).
|
||||
type SlurmHandler interface {
|
||||
OnSlurmStatus(node string, st *llmgwv1.SlurmStatus)
|
||||
// OnRegister is a node's new stream, which is told again what it is.
|
||||
OnRegister(node string)
|
||||
}
|
||||
|
||||
// SetSlurm gives the service what Slurm's messages go to.
|
||||
func (s *Service) SetSlurm(h SlurmHandler) {
|
||||
s.mu.Lock()
|
||||
s.slurm = h
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func (s *Service) slurmHandler() SlurmHandler {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
return s.slurm
|
||||
}
|
||||
@@ -296,7 +296,7 @@ func (s *Server) Start(req Request) (Job, error) {
|
||||
}
|
||||
req.sized = req.CtxPool > 0 || req.ContextLength > 0
|
||||
if req.CtxPool <= 0 {
|
||||
req.CtxPool = 16384
|
||||
req.CtxPool = DefaultCtxPool
|
||||
}
|
||||
if req.ContextLength <= 0 {
|
||||
req.ContextLength = 4096
|
||||
@@ -731,6 +731,10 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
|
||||
return out
|
||||
}
|
||||
|
||||
// DefaultCtxPool is llama.cpp's pool when a request names none: a context
|
||||
// asked for longer than it needs a pool of its own.
|
||||
const DefaultCtxPool = 16384
|
||||
|
||||
// llamaArgs size llama.cpp's pool and its slots together. The pool is shared
|
||||
// by the slots, so slots of the context the catalog promises are as many as
|
||||
// the pool holds: the engine's default of four turned a 32768 pool into
|
||||
|
||||
@@ -43,6 +43,9 @@ type Plan struct {
|
||||
License string `json:"license,omitempty"`
|
||||
Params int64 `json:"params,omitempty"`
|
||||
Context int `json:"context"`
|
||||
// Trained is the context the model was trained for, the most a request
|
||||
// may use; 0 when neither its header nor its config says.
|
||||
Trained int `json:"trained,omitempty"`
|
||||
// KVBytesPerToken is one token of context, the whole model: what an
|
||||
// option needs for another context is worked out from it. 0 is unknown.
|
||||
KVBytesPerToken int64 `json:"kv_bytes_per_token,omitempty"`
|
||||
@@ -149,6 +152,7 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
|
||||
p.Servable, p.Reason = false, v.Reason
|
||||
}
|
||||
shape.KVBytesPerToken = v.Header.KVBytesPerToken()
|
||||
p.Trained = int(v.Header.ContextLength) //nolint:gosec // a header's context length
|
||||
shape.KVHeads = int(v.Header.HeadCountKV) //nolint:gosec // a header's head count
|
||||
shape.ActiveShare = v.Header.ActiveShare()
|
||||
if p.Arch == "" {
|
||||
@@ -167,6 +171,7 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
|
||||
m, err := modelplan.FetchModel(ctx, repo.ID)
|
||||
if err == nil {
|
||||
shape.KVBytesPerToken = m.KVBytesPerToken(modelplan.BF16)
|
||||
p.Trained = m.MaxContext
|
||||
shape.KVHeads = m.KVHeads
|
||||
shape.Experts = m.Experts
|
||||
if p.Params == 0 {
|
||||
|
||||
@@ -154,6 +154,9 @@ func (r Result) Explain() string {
|
||||
// Scheduler places replicas onto an inventory.
|
||||
type Scheduler struct {
|
||||
inv *inventory.Store
|
||||
// shun is where a replica of the model being placed failed to start;
|
||||
// set on a copy by Away, never on the scheduler the reconciler shares.
|
||||
shun []Spot
|
||||
// now is the clock power readings are judged fresh by; nil is the wall
|
||||
// clock.
|
||||
now func() time.Time
|
||||
@@ -162,6 +165,41 @@ type Scheduler struct {
|
||||
// New builds a scheduler over an inventory.
|
||||
func New(inv *inventory.Store) *Scheduler { return &Scheduler{inv: inv} }
|
||||
|
||||
// Spot is where a replica ran: a node, and the cards it held there (none
|
||||
// for one on the node's CPU).
|
||||
type Spot struct {
|
||||
Node string
|
||||
GPUs []int
|
||||
}
|
||||
|
||||
// Away is this scheduler placing nowhere in spots: not those cards, nor a
|
||||
// CPU replica on those nodes. A copy, so the one shared is not changed.
|
||||
//
|
||||
// For a model that failed to start somewhere: the same inputs pick the same
|
||||
// node and the same card every time, so without it every retry went back to
|
||||
// where the last one had failed until the model was parked, with another
|
||||
// card or node free all along.
|
||||
func (s *Scheduler) Away(spots []Spot) *Scheduler {
|
||||
c := *s
|
||||
c.shun = spots
|
||||
return &c
|
||||
}
|
||||
|
||||
// shunned is the cards of node n a model failed on, and whether it failed
|
||||
// on the node's CPU.
|
||||
func (s *Scheduler) shunned(node string) (gpus []int, cpu bool) {
|
||||
for _, sp := range s.shun {
|
||||
if sp.Node != node {
|
||||
continue
|
||||
}
|
||||
if len(sp.GPUs) == 0 {
|
||||
cpu = true
|
||||
}
|
||||
gpus = append(gpus, sp.GPUs...)
|
||||
}
|
||||
return gpus, cpu
|
||||
}
|
||||
|
||||
// Place picks the first profile, in priority order, that fits.
|
||||
//
|
||||
// avoid lists nodes already hosting this model, so replicas of one model spread
|
||||
@@ -686,7 +724,12 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
|
||||
}
|
||||
}
|
||||
|
||||
badGPUs, badCPU := s.shunned(n.ID)
|
||||
if p.CPUOnly() {
|
||||
if badCPU {
|
||||
reject("a replica of it failed to start here; trying elsewhere first")
|
||||
continue
|
||||
}
|
||||
// Every ready node can run it; there is no card to be free.
|
||||
c := candidate{node: n}
|
||||
if why := budget.check(n, s.shardWatts(p, c)); why != "" {
|
||||
@@ -703,8 +746,16 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
|
||||
if g.Class() != p.GPUClass || !slices.Contains(freeIdx, g.Index) {
|
||||
continue
|
||||
}
|
||||
if slices.Contains(badGPUs, g.Index) {
|
||||
continue
|
||||
}
|
||||
free = append(free, g)
|
||||
}
|
||||
if len(free) < p.GPUsPerNode && len(badGPUs) > 0 {
|
||||
reject(fmt.Sprintf("has %d free %s GPUs besides the ones a replica of it failed to start on, needs %d",
|
||||
len(free), p.GPUClass, p.GPUsPerNode))
|
||||
continue
|
||||
}
|
||||
if len(free) < p.GPUsPerNode {
|
||||
reject(fmt.Sprintf("has %d free %s GPUs, needs %d", len(free), p.GPUClass, p.GPUsPerNode))
|
||||
continue
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
// failAndRetry fails the last replica started, and returns where the next
|
||||
// one is started once the backoff is over.
|
||||
func failAndRetry(t *testing.T, h *harness) ShardCommand {
|
||||
t.Helper()
|
||||
started := h.disp.startedShards()
|
||||
if len(started) == 0 {
|
||||
t.Fatal("nothing was started")
|
||||
}
|
||||
s := started[len(started)-1]
|
||||
h.r.OnShardStatus(s.ReplicaID, s.ShardID, supervisor.Failed, "the engine exited")
|
||||
h.r.Reconcile(t.Context()) // reaps it
|
||||
h.disp.reset()
|
||||
h.advance(DefaultConfig().MaxBackoff)
|
||||
h.r.Reconcile(t.Context())
|
||||
again := h.disp.startedShards()
|
||||
if len(again) == 0 {
|
||||
t.Fatal("it was not tried again")
|
||||
}
|
||||
return again[0]
|
||||
}
|
||||
|
||||
func oneCard() catalog.Model {
|
||||
return testModel(1, catalog.Profile{Name: "one", GPUClass: "h100-80g", GPUsPerNode: 1})
|
||||
}
|
||||
|
||||
func TestAReplicaThatFailedIsTriedOnAnotherNode(t *testing.T) {
|
||||
// The same inputs give the same node every time: an on-demand 3D model
|
||||
// that failed to load was started on that node again, and again, until
|
||||
// it was parked, with another node free all along.
|
||||
h := newHarness(t, oneCard(), gpuNode("1", 1), gpuNode("2", 1))
|
||||
h.r.Reconcile(t.Context())
|
||||
first := h.disp.startedShards()[0].NodeID
|
||||
|
||||
if next := failAndRetry(t, h); next.NodeID == first {
|
||||
t.Errorf("tried on node %s again, where it had just failed", first)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReplicaThatFailedIsTriedOnAnotherCardOfTheNode(t *testing.T) {
|
||||
h := newHarness(t, oneCard(), gpuNode("1", 2))
|
||||
h.r.Reconcile(t.Context())
|
||||
first := h.disp.startedShards()[0].Spec.GPUIndices
|
||||
|
||||
next := failAndRetry(t, h)
|
||||
if len(next.Spec.GPUIndices) != 1 || len(first) != 1 || next.Spec.GPUIndices[0] == first[0] {
|
||||
t.Errorf("tried on card %v again, where it had just failed", next.Spec.GPUIndices)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithNowhereElseItIsTriedWhereItFailed(t *testing.T) {
|
||||
// A preference, not a rule: a model that has one place is not left
|
||||
// unplaced because it failed there once.
|
||||
h := newHarness(t, oneCard(), gpuNode("1", 1))
|
||||
h.r.Reconcile(t.Context())
|
||||
if next := failAndRetry(t, h); next.NodeID != "1" {
|
||||
t.Errorf("started on %q", next.NodeID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComingUpForgetsWhereItFailed(t *testing.T) {
|
||||
h := newHarness(t, oneCard(), gpuNode("1", 1), gpuNode("2", 1))
|
||||
h.r.Reconcile(t.Context())
|
||||
next := failAndRetry(t, h)
|
||||
h.r.OnShardStatus(next.ReplicaID, next.ShardID, supervisor.Ready, "")
|
||||
h.r.mu.Lock()
|
||||
left := len(h.r.failedAt["llama"])
|
||||
h.r.mu.Unlock()
|
||||
if left != 0 {
|
||||
t.Errorf("a model that came up still remembers %d failures' places", left)
|
||||
}
|
||||
}
|
||||
@@ -61,6 +61,7 @@ func (r *Reconciler) Retry(model string) bool {
|
||||
delete(r.parked, model)
|
||||
r.attempts[model] = 0
|
||||
delete(r.nextAttempt, model)
|
||||
delete(r.failedAt, model)
|
||||
r.mu.Unlock()
|
||||
r.Notify()
|
||||
return was
|
||||
|
||||
@@ -7,6 +7,8 @@ import (
|
||||
"log/slog"
|
||||
"math/rand/v2"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
@@ -251,6 +253,9 @@ type Reconciler struct {
|
||||
nextAttempt map[string]time.Time
|
||||
// attempts counts consecutive failures per model, for the backoff.
|
||||
attempts map[string]int
|
||||
// failedAt is where a model's replicas failed to start since it last
|
||||
// came up, most recent last: the next one is placed elsewhere if it can.
|
||||
failedAt map[string][]placement.Spot
|
||||
// lastExplain keeps the most recent placement failure per model, so
|
||||
// `ctl explain-placement` can answer without re-running placement.
|
||||
lastExplain map[string]placement.Result
|
||||
@@ -334,6 +339,7 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
|
||||
replicas: map[string]*Replica{},
|
||||
nextAttempt: map[string]time.Time{},
|
||||
attempts: map[string]int{},
|
||||
failedAt: map[string][]placement.Spot{},
|
||||
reservedPorts: map[int]struct{}{},
|
||||
wanted: map[string]time.Time{},
|
||||
coldStart: map[string]time.Duration{},
|
||||
@@ -726,6 +732,7 @@ func (r *Reconciler) reapFailed(ctx context.Context) {
|
||||
// replica is about to disappear, and the backoff has to survive it.
|
||||
r.attempts[rep.Model]++
|
||||
r.nextAttempt[rep.Model] = r.now().Add(r.backoff(r.attempts[rep.Model]))
|
||||
r.noteFailedAt(rep)
|
||||
if n := r.attempts[rep.Model]; n >= r.cfg.ParkAfter {
|
||||
if _, already := r.parked[rep.Model]; !already {
|
||||
m, _ := r.cat.Load().Resolve(rep.Model)
|
||||
@@ -875,6 +882,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
delete(r.parked, m.Name)
|
||||
r.attempts[m.Name] = 0
|
||||
delete(r.nextAttempt, m.Name)
|
||||
delete(r.failedAt, m.Name)
|
||||
isParked, blockedUntil = false, time.Time{}
|
||||
r.log.Info("model unparked: its definition changed", "model", m.Name)
|
||||
}
|
||||
@@ -1002,7 +1010,29 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
|
||||
return notTried
|
||||
}
|
||||
}
|
||||
res := r.sched.Place(profiles, avoid)
|
||||
// Away from where it last failed to start, when there is anywhere else:
|
||||
// the same inputs give the same node and card, and an on-demand model
|
||||
// that failed there was otherwise started there again until it was
|
||||
// parked. Only a preference too: nowhere else, and it is tried there.
|
||||
r.mu.Lock()
|
||||
failed := slices.Clone(r.failedAt[m.Name])
|
||||
r.mu.Unlock()
|
||||
var res placement.Result
|
||||
if len(failed) > 0 {
|
||||
away := r.sched.Away(failed)
|
||||
res = away.Place(profiles, avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
res = away.Place(profiles, nil)
|
||||
}
|
||||
pspan.SetAttributes(attribute.Bool("llmgw.away_from_failure", res.Placement != nil))
|
||||
if res.Placement != nil {
|
||||
r.log.Info("placing a model away from where it failed to start", "model", m.Name,
|
||||
"failed_on", spotsText(failed), "now_on", spotsText(spotsOf(res.Placement)))
|
||||
}
|
||||
}
|
||||
if res.Placement == nil {
|
||||
res = r.sched.Place(profiles, avoid)
|
||||
}
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
// Spreading across failure domains is a preference, not a rule. If it
|
||||
// is the only thing standing in the way, take the placement anyway:
|
||||
@@ -1054,6 +1084,51 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
|
||||
return placed
|
||||
}
|
||||
|
||||
// maxFailedSpots is how many places a model's failures are remembered at.
|
||||
const maxFailedSpots = 8
|
||||
|
||||
// noteFailedAt remembers where a failed replica ran. Called under r.mu.
|
||||
func (r *Reconciler) noteFailedAt(rep *Replica) {
|
||||
spots := r.failedAt[rep.Model]
|
||||
for _, sh := range rep.Shards {
|
||||
sp := placement.Spot{Node: sh.NodeID, GPUs: slices.Clone(sh.GPUIndices)}
|
||||
spots = slices.DeleteFunc(spots, func(o placement.Spot) bool {
|
||||
return o.Node == sp.Node && slices.Equal(o.GPUs, sp.GPUs)
|
||||
})
|
||||
spots = append(spots, sp)
|
||||
}
|
||||
if len(spots) > maxFailedSpots {
|
||||
spots = spots[len(spots)-maxFailedSpots:]
|
||||
}
|
||||
r.failedAt[rep.Model] = spots
|
||||
}
|
||||
|
||||
// spotsOf is where a placement puts its shards.
|
||||
func spotsOf(pl *placement.Placement) []placement.Spot {
|
||||
var out []placement.Spot
|
||||
for _, n := range pl.Nodes {
|
||||
out = append(out, placement.Spot{Node: n.NodeID, GPUs: n.GPUIndices})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// spotsText is spots as a log reads them: "node01 gpu 0,1; node02 cpu".
|
||||
func spotsText(spots []placement.Spot) string {
|
||||
parts := make([]string, 0, len(spots))
|
||||
for _, sp := range spots {
|
||||
if len(sp.GPUs) == 0 {
|
||||
parts = append(parts, sp.Node+" cpu")
|
||||
continue
|
||||
}
|
||||
gs := make([]string, len(sp.GPUs))
|
||||
for i, g := range sp.GPUs {
|
||||
gs[i] = strconv.Itoa(g)
|
||||
}
|
||||
parts = append(parts, sp.Node+" gpu "+strings.Join(gs, ","))
|
||||
}
|
||||
return strings.Join(parts, "; ")
|
||||
}
|
||||
|
||||
// scaleOutcome is what one scaleUp did.
|
||||
type scaleOutcome int
|
||||
|
||||
@@ -1433,6 +1508,7 @@ func (r *Reconciler) markShard(replicaID, shardID string, st supervisor.State, e
|
||||
// counting from scratch rather than from an old streak.
|
||||
r.attempts[rep.Model] = 0
|
||||
delete(r.nextAttempt, rep.Model)
|
||||
delete(r.failedAt, rep.Model)
|
||||
if !rep.CreatedAt.IsZero() {
|
||||
r.coldStart[rep.Model] = r.now().Sub(rep.CreatedAt)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
package slurmsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// fakeKV is the store's cluster state.
|
||||
type fakeKV struct {
|
||||
m map[string]string
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeKV) PutClusterState(_ context.Context, k, v string) (bool, error) {
|
||||
if f.err != nil {
|
||||
return false, f.err
|
||||
}
|
||||
f.m[k] = v
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func (f *fakeKV) ClusterState(_ context.Context, k string) (string, error) {
|
||||
if f.err != nil {
|
||||
return "", f.err
|
||||
}
|
||||
v, ok := f.m[k]
|
||||
if !ok {
|
||||
return "", store.ErrNotFound
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
// takeOver is the leader that comes next: the same settings and store, an
|
||||
// inventory with only the nodes connected to it so far.
|
||||
func takeOver(r *rig, kv *fakeKV, connected ...string) *rig {
|
||||
n := &rig{set: r.set, inv: &fakeInv{nodes: map[string]*inventory.Node{}, allocs: map[string]int{}},
|
||||
agents: &fakeAgents{conn: connected, got: map[string][]*llmgwv1.Slurm{}}}
|
||||
for _, c := range connected {
|
||||
node := *r.inv.nodes[c]
|
||||
node.State = inventory.Ready
|
||||
n.inv.nodes[c] = &node
|
||||
}
|
||||
n.svc = New(func() config.SlurmSettings { return n.set }, n.inv, n.agents,
|
||||
func(context.Context) ([]byte, error) { return []byte(strings.Repeat("k", 1024)), nil }, nil)
|
||||
n.svc.SetMemory(StoreMemory(kv))
|
||||
return n
|
||||
}
|
||||
|
||||
func TestANewLeaderKeepsTheNodesItHasNotHeardFromInTheControllersFiles(t *testing.T) {
|
||||
kv := &fakeKV{m: map[string]string{}}
|
||||
r := newRig()
|
||||
r.svc.SetMemory(StoreMemory(kv))
|
||||
r.set.Lent = []string{"cuda01"}
|
||||
r.inv.allocs = map[string]int{}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "MIXED", Jobs: 1})
|
||||
r.tick()
|
||||
want := r.agents.last("node06").GetFiles()["slurm.conf"]
|
||||
if !strings.Contains(want, "NodeName=cuda01") || !strings.Contains(want, "Gres=gpu:rtx3090:1") {
|
||||
t.Fatalf("slurm.conf:\n%s", want)
|
||||
}
|
||||
|
||||
// The leader goes; the next hears from the controller first.
|
||||
n := takeOver(r, kv, "node06")
|
||||
n.tick()
|
||||
if got := n.agents.last("node06").GetFiles()["slurm.conf"]; got != want {
|
||||
t.Fatalf("the new leader's slurm.conf differs; the controller would drop the node and its job:\n%s\nwas:\n%s",
|
||||
got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeIsToldNothingUntilItsControllerIsKnown(t *testing.T) {
|
||||
r := newRig()
|
||||
r.svc.SetMemory(StoreMemory(&fakeKV{m: map[string]string{}}))
|
||||
delete(r.inv.nodes, "node06")
|
||||
r.agents.conn = []string{"cuda01"}
|
||||
r.tick()
|
||||
if m := r.agents.last("cuda01"); m != nil {
|
||||
t.Fatalf("a node was told %v before its controller was known; an empty part stops its slurmd and its jobs", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANewLeaderTellsANodeItsControllerFromTheStore(t *testing.T) {
|
||||
kv := &fakeKV{m: map[string]string{}}
|
||||
r := newRig()
|
||||
r.svc.SetMemory(StoreMemory(kv))
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
|
||||
r.tick()
|
||||
|
||||
n := takeOver(r, kv, "cuda01")
|
||||
n.tick()
|
||||
if m := n.agents.last("cuda01"); !m.GetNode() || m.GetConfServer() != "10.0.0.146:6817" {
|
||||
t.Fatalf("the node was told %v", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithTheStoreUnreadTheControllerIsSentNothingThatLeavesANodeOut(t *testing.T) {
|
||||
kv := &fakeKV{m: map[string]string{}, err: errors.New("store down")}
|
||||
r := newRig()
|
||||
r.svc.SetMemory(StoreMemory(kv))
|
||||
r.tick()
|
||||
if m := r.agents.last("node06"); m != nil {
|
||||
t.Fatalf("the controller was sent %v without cuda01, with the store unread", m)
|
||||
}
|
||||
// Once every node has reported, nothing is left out: sent.
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
|
||||
r.tick()
|
||||
if !strings.Contains(r.agents.last("node06").GetFiles()["slurm.conf"], "NodeName=cuda01") {
|
||||
t.Fatal("with every node reported, the controller still got no files")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithNothingKeptANewLeaderWaitsForTheNodesAWhile(t *testing.T) {
|
||||
r := newRig()
|
||||
r.svc.SetMemory(StoreMemory(&fakeKV{m: map[string]string{}}))
|
||||
r.tick()
|
||||
if m := r.agents.last("node06"); m != nil {
|
||||
t.Fatalf("with nothing kept, the controller was sent %v before cuda01 reported", m)
|
||||
}
|
||||
defer func(d time.Duration) { HoldFor = d }(HoldFor)
|
||||
HoldFor = 0
|
||||
r.tick()
|
||||
if m := r.agents.last("node06"); !m.GetController() || strings.Contains(m.GetFiles()["slurm.conf"], "NodeName=cuda01") {
|
||||
t.Fatalf("after the wait, the controller was told %v", m)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
package slurmsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"slices"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Memory keeps the nodes slurm.conf last held, and the controller's
|
||||
// address, where a leader that takes over finds them. Without it a new
|
||||
// leader's first slurm.conf has none of the nodes it has not heard from
|
||||
// yet, and the controller, restarted on it, drops them and the jobs
|
||||
// running there.
|
||||
type Memory interface {
|
||||
Recall(ctx context.Context) (Kept, error)
|
||||
Keep(ctx context.Context, k Kept) error
|
||||
}
|
||||
|
||||
// Kept is what Memory holds.
|
||||
type Kept struct {
|
||||
Controller string `json:"controller,omitempty"`
|
||||
ControllerAddr string `json:"controller_addr,omitempty"`
|
||||
Nodes map[string]slurm.Node `json:"nodes,omitempty"`
|
||||
}
|
||||
|
||||
func (k Kept) equal(o Kept) bool {
|
||||
if k.Controller != o.Controller || k.ControllerAddr != o.ControllerAddr || len(k.Nodes) != len(o.Nodes) {
|
||||
return false
|
||||
}
|
||||
for name, n := range k.Nodes {
|
||||
m, ok := o.Nodes[name]
|
||||
if !ok || n.Addr != m.Addr || n.Hardware != m.Hardware || !slices.Equal(n.GPUs, m.GPUs) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// KV is the store's cluster state.
|
||||
type KV interface {
|
||||
PutClusterState(ctx context.Context, key, value string) (bool, error)
|
||||
ClusterState(ctx context.Context, key string) (string, error)
|
||||
}
|
||||
|
||||
// keptKey is the cluster state key it is kept under.
|
||||
const keptKey = "slurm-nodes"
|
||||
|
||||
// StoreMemory keeps it in the store's cluster state.
|
||||
func StoreMemory(kv KV) Memory { return kvMemory{kv} }
|
||||
|
||||
type kvMemory struct{ kv KV }
|
||||
|
||||
func (m kvMemory) Recall(ctx context.Context) (Kept, error) {
|
||||
var k Kept
|
||||
v, err := m.kv.ClusterState(ctx, keptKey)
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
return k, nil
|
||||
}
|
||||
if err != nil {
|
||||
return k, err
|
||||
}
|
||||
err = json.Unmarshal([]byte(v), &k)
|
||||
return k, err
|
||||
}
|
||||
|
||||
func (m kvMemory) Keep(ctx context.Context, k Kept) error {
|
||||
b, err := json.Marshal(k)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = m.kv.PutClusterState(ctx, keptKey, string(b))
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,742 @@
|
||||
// Package slurmsvc is the leader's side of the Slurm cluster run beside
|
||||
// llmgw (BACKLOG P): it tells each node's agent what it is, writes the
|
||||
// controller's files from what the nodes report, and hands nodes between
|
||||
// llmgw and Slurm as the cluster's settings say.
|
||||
//
|
||||
// A node in slurm.lent is drained of llmgw's replicas, as `ctl drain` does,
|
||||
// and lent to Slurm once none is left. A node taken out of slurm.lent is
|
||||
// drained in Slurm, and is llmgw's again when its last job has ended. Both
|
||||
// are kept in force every tick, from the settings and what the agents say,
|
||||
// so a leader that takes over carries on where the last one was.
|
||||
package slurmsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"net"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// Agents is how the service reaches the nodes.
|
||||
type Agents interface {
|
||||
Send(node string, msg *llmgwv1.ControlMessage) error
|
||||
ConnectedNodes() []string
|
||||
}
|
||||
|
||||
// Inventory is what the service reads and changes of llmgw's nodes.
|
||||
type Inventory interface {
|
||||
Get(id string) (inventory.Node, bool)
|
||||
SetState(id string, st inventory.NodeState) bool
|
||||
Allocations(nodeID string) []inventory.Allocation
|
||||
Nodes() []inventory.Node
|
||||
}
|
||||
|
||||
// Phase is where a node of slurm.nodes is between llmgw and Slurm.
|
||||
type Phase string
|
||||
|
||||
const (
|
||||
// Llmgw: drained in Slurm with no job left, llmgw's to place on.
|
||||
Llmgw Phase = "llmgw"
|
||||
// Lending: in slurm.lent, its replicas being moved off.
|
||||
Lending Phase = "lending"
|
||||
// Lent: Slurm's; llmgw places nothing there.
|
||||
Lent Phase = "lent"
|
||||
// Reclaiming: out of slurm.lent, drained in Slurm, jobs still running.
|
||||
Reclaiming Phase = "reclaiming"
|
||||
// Unknown: its agent has not said how it is in Slurm yet.
|
||||
Unknown Phase = "unknown"
|
||||
)
|
||||
|
||||
// NodeStatus is one node as the service sees it.
|
||||
type NodeStatus struct {
|
||||
Node string `json:"node"`
|
||||
Phase Phase `json:"phase"`
|
||||
Replicas int `json:"replicas"`
|
||||
// StopsBy is, while it is being lent, when a replica still on it is
|
||||
// stopped even with nowhere else to go: the drain's grace after the
|
||||
// lend began. Zero when not known (a leader that took over meanwhile).
|
||||
StopsBy time.Time `json:"stops_by,omitempty"`
|
||||
Status *Reported `json:"slurm,omitempty"`
|
||||
Connected bool `json:"connected"`
|
||||
}
|
||||
|
||||
// Reported is the agent's SlurmStatus, for the admin API.
|
||||
type Reported struct {
|
||||
Version string `json:"version,omitempty"`
|
||||
Hardware string `json:"hardware,omitempty"`
|
||||
SlurmdRunning bool `json:"slurmd_running"`
|
||||
SlurmctldRunning bool `json:"slurmctld_running"`
|
||||
SlurmdbdRunning bool `json:"slurmdbd_running"`
|
||||
// Shared is the shared path here: "mounted", "exported", "missing".
|
||||
Shared string `json:"shared,omitempty"`
|
||||
// Containers is what the machine has for jobs in containers, empty
|
||||
// while not both of pyxis and enroot.
|
||||
Containers string `json:"containers,omitempty"`
|
||||
// Users is how many of llmgw's users the machine has.
|
||||
Users int `json:"users"`
|
||||
State string `json:"state,omitempty"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
Jobs int `json:"jobs"`
|
||||
Error string `json:"error,omitempty"`
|
||||
FilesSHA256 string `json:"files_sha256,omitempty"`
|
||||
At time.Time `json:"at"`
|
||||
}
|
||||
|
||||
// Status is the whole Slurm cluster, for `ctl slurm status`.
|
||||
type Status struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
Cluster string `json:"cluster"`
|
||||
Controller string `json:"controller"`
|
||||
Partition string `json:"partition"`
|
||||
Ctld *Reported `json:"controller_status,omitempty"`
|
||||
Nodes []NodeStatus `json:"nodes"`
|
||||
// Waiting are nodes left out of slurm.conf until they report their
|
||||
// hardware.
|
||||
Waiting []string `json:"waiting,omitempty"`
|
||||
// Shared is where the shared files are and come from; Users who keeps
|
||||
// the users, and how many llmgw has.
|
||||
Shared config.SharedStorage `json:"shared"`
|
||||
Users string `json:"users,omitempty"`
|
||||
UserCount int `json:"user_count,omitempty"`
|
||||
// Containers is whether jobs may run in containers: "on" once every
|
||||
// machine has pyxis and enroot, "installing" until then, empty when
|
||||
// slurm.containers is off.
|
||||
Containers string `json:"containers,omitempty"`
|
||||
}
|
||||
|
||||
// User is a Linux user made from an llmgw account.
|
||||
type User struct {
|
||||
Login string `json:"login"`
|
||||
UID int `json:"uid"`
|
||||
Name string `json:"name"`
|
||||
Email string `json:"email"`
|
||||
SSHKeys []string `json:"ssh_keys,omitempty"`
|
||||
}
|
||||
|
||||
// Service keeps the Slurm cluster.
|
||||
type Service struct {
|
||||
settings func() config.SlurmSettings
|
||||
inv Inventory
|
||||
agents Agents
|
||||
key func(ctx context.Context) ([]byte, error)
|
||||
log *slog.Logger
|
||||
|
||||
mu sync.Mutex
|
||||
// status is each node's last SlurmStatus.
|
||||
status map[string]*Reported
|
||||
// drained are nodes this service put in Draining, which it alone makes
|
||||
// ready again.
|
||||
drained map[string]bool
|
||||
// lendingAt is when this leader began moving a node's replicas off.
|
||||
lendingAt map[string]time.Time
|
||||
// sent is what each node was last told, so it is told again only when
|
||||
// that changes or its stream is new.
|
||||
sent map[string]*llmgwv1.Slurm
|
||||
mungeKy []byte
|
||||
kick chan struct{}
|
||||
// mem keeps slurm.conf's nodes and the controller's address for the
|
||||
// next leader; known is what it held when read, and what was kept
|
||||
// since. loaded is false until it was read: until then the controller
|
||||
// is sent no files that leave out a node, which would drop that node
|
||||
// and its jobs.
|
||||
mem Memory
|
||||
known Kept
|
||||
loaded bool
|
||||
// started is the first pass, from which a node neither heard from nor
|
||||
// kept is waited for, for HoldFor.
|
||||
started time.Time
|
||||
// userList reads the users from the accounts; users is what it last
|
||||
// said, read at usersAt. Until it has been read once no node is told
|
||||
// anything of users: an empty list would take every user away.
|
||||
userList func(ctx context.Context) ([]User, error)
|
||||
users []User
|
||||
usersAt time.Time
|
||||
usersOK bool
|
||||
}
|
||||
|
||||
// UsersEvery is how often the users are read again without being asked.
|
||||
var UsersEvery = 30 * time.Second
|
||||
|
||||
// SetUsers has the users read with list, for slurm.users "llmgw".
|
||||
func (s *Service) SetUsers(list func(ctx context.Context) ([]User, error)) {
|
||||
s.mu.Lock()
|
||||
s.userList = list
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
// UsersChanged has the users read again now: an account or its keys
|
||||
// changed.
|
||||
func (s *Service) UsersChanged() {
|
||||
s.mu.Lock()
|
||||
s.usersAt = time.Time{}
|
||||
s.mu.Unlock()
|
||||
s.Kick()
|
||||
}
|
||||
|
||||
// readUsers reads the users when they are due. Under s.mu.
|
||||
func (s *Service) readUsers(ctx context.Context, set config.SlurmSettings) {
|
||||
if set.Users != config.UsersLlmgw || s.userList == nil || time.Since(s.usersAt) < UsersEvery {
|
||||
return
|
||||
}
|
||||
us, err := s.userList(ctx)
|
||||
s.usersAt = time.Now()
|
||||
if err != nil {
|
||||
s.log.Warn("could not read the Slurm users from the accounts; the nodes keep the last ones", "err", err)
|
||||
return
|
||||
}
|
||||
s.users, s.usersOK = us, true
|
||||
}
|
||||
|
||||
// HoldFor is how long a new leader waits for a node it neither has heard
|
||||
// from nor finds kept before it writes the controller's files without it:
|
||||
// on a cluster that kept nothing yet (one upgraded to this), the nodes
|
||||
// report within seconds of a new leader, and the controller keeps the
|
||||
// node and its jobs meanwhile.
|
||||
var HoldFor = 2 * time.Minute
|
||||
|
||||
// New is a Service; key gives the munge key, made once from a cluster
|
||||
// secret.
|
||||
func New(settings func() config.SlurmSettings, inv Inventory, agents Agents,
|
||||
key func(ctx context.Context) ([]byte, error), log *slog.Logger) *Service {
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
return &Service{settings: settings, inv: inv, agents: agents, key: key, log: log,
|
||||
status: map[string]*Reported{}, drained: map[string]bool{}, lendingAt: map[string]time.Time{}, sent: map[string]*llmgwv1.Slurm{},
|
||||
kick: make(chan struct{}, 1)}
|
||||
}
|
||||
|
||||
// SetMemory has slurm.conf's nodes kept in mem, and read from it first.
|
||||
func (s *Service) SetMemory(mem Memory) {
|
||||
s.mu.Lock()
|
||||
s.mem = mem
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
// current is the settings, slurm.all_nodes resolved to the nodes the
|
||||
// inventory has.
|
||||
func (s *Service) current() config.SlurmSettings {
|
||||
set := s.settings()
|
||||
if !set.AllNodes {
|
||||
return set
|
||||
}
|
||||
var ids []string
|
||||
for _, n := range s.inv.Nodes() {
|
||||
ids = append(ids, n.ID)
|
||||
}
|
||||
return set.WithNodes(ids)
|
||||
}
|
||||
|
||||
// Every is how often the service looks again without being asked.
|
||||
var Every = 10 * time.Second
|
||||
|
||||
// Run keeps the cluster until ctx ends.
|
||||
func (s *Service) Run(ctx context.Context) {
|
||||
t := time.NewTicker(Every)
|
||||
defer t.Stop()
|
||||
for {
|
||||
s.Tick(ctx)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
case <-s.kick:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Kick has the service look again now: the settings changed.
|
||||
func (s *Service) Kick() {
|
||||
select {
|
||||
case s.kick <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
// OnSlurmStatus records what a node's agent says.
|
||||
func (s *Service) OnSlurmStatus(node string, st *llmgwv1.SlurmStatus) {
|
||||
r := &Reported{Version: st.GetVersion(), Hardware: st.GetHardware(), SlurmdRunning: st.GetSlurmdRunning(),
|
||||
Shared: st.GetShared(), Users: int(st.GetUsers()), Containers: st.GetContainers(),
|
||||
SlurmctldRunning: st.GetSlurmctldRunning(), SlurmdbdRunning: st.GetSlurmdbdRunning(), State: st.GetState(), Reason: st.GetReason(),
|
||||
Jobs: int(st.GetJobs()), Error: st.GetError(), FilesSHA256: st.GetFilesSha256(), At: time.Now()}
|
||||
s.mu.Lock()
|
||||
prev := s.status[node]
|
||||
s.status[node] = r
|
||||
s.mu.Unlock()
|
||||
if r.Error != "" && (prev == nil || prev.Error != r.Error) {
|
||||
s.log.Warn("a node could not keep its part in Slurm", "node", node, "err", r.Error)
|
||||
}
|
||||
if prev == nil || prev.Hardware != r.Hardware || prev.State != r.State || prev.Jobs != r.Jobs {
|
||||
s.Kick()
|
||||
}
|
||||
}
|
||||
|
||||
// OnRegister is a node's new stream: it is told again what it is.
|
||||
func (s *Service) OnRegister(node string) {
|
||||
s.mu.Lock()
|
||||
delete(s.sent, node)
|
||||
s.mu.Unlock()
|
||||
s.Kick()
|
||||
}
|
||||
|
||||
// Tick does one pass: the nodes handed over as far as they can be, and
|
||||
// each told what it is.
|
||||
func (s *Service) Tick(ctx context.Context) {
|
||||
set := s.current()
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.recall(ctx)
|
||||
s.readUsers(ctx, set)
|
||||
if s.started.IsZero() {
|
||||
s.started = time.Now()
|
||||
}
|
||||
phases := s.handOver(set)
|
||||
var key []byte
|
||||
if set.Enabled {
|
||||
if s.mungeKy == nil {
|
||||
k, err := s.key(ctx)
|
||||
if err != nil {
|
||||
s.log.Warn("no munge key for Slurm; nothing is sent to the nodes", "err", err)
|
||||
return
|
||||
}
|
||||
s.mungeKy = k
|
||||
}
|
||||
key = s.mungeKy
|
||||
}
|
||||
files, waiting := s.files(set)
|
||||
// Not yet knowing what the last leader knew, a slurm.conf without some
|
||||
// node would take it, and its jobs, from the controller: kept back.
|
||||
hold := s.mem != nil && len(waiting) > 0 && (!s.loaded || time.Since(s.started) < HoldFor)
|
||||
for _, node := range s.agents.ConnectedNodes() {
|
||||
if hold && node == set.Controller {
|
||||
continue
|
||||
}
|
||||
m := s.message(set, node, phases[node] == Lent, key, files)
|
||||
if m == nil {
|
||||
continue
|
||||
}
|
||||
if old, ok := s.sent[node]; ok && proto.Equal(old, m) {
|
||||
continue
|
||||
}
|
||||
if err := s.agents.Send(node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_Slurm{Slurm: m}}); err != nil {
|
||||
s.log.Warn("could not tell a node its part in Slurm", "node", node, "err", err)
|
||||
continue
|
||||
}
|
||||
s.sent[node] = m
|
||||
}
|
||||
s.remember(ctx, set)
|
||||
}
|
||||
|
||||
// recall reads what the last leader kept, once. Under s.mu.
|
||||
func (s *Service) recall(ctx context.Context) {
|
||||
if s.mem == nil || s.loaded {
|
||||
return
|
||||
}
|
||||
k, err := s.mem.Recall(ctx)
|
||||
if err != nil {
|
||||
s.log.Warn("could not read Slurm's nodes from the store; the controller's files wait for every node",
|
||||
"err", err)
|
||||
return
|
||||
}
|
||||
s.known, s.loaded = k, true
|
||||
}
|
||||
|
||||
// remember keeps slurm.conf's nodes and the controller's address when they
|
||||
// changed. Under s.mu.
|
||||
func (s *Service) remember(ctx context.Context, set config.SlurmSettings) {
|
||||
if s.mem == nil || !s.loaded || !set.Enabled {
|
||||
return
|
||||
}
|
||||
c, _ := s.cluster(set)
|
||||
now := Kept{Controller: set.Controller, ControllerAddr: c.ControllerAddr, Nodes: map[string]slurm.Node{}}
|
||||
for _, n := range c.Nodes {
|
||||
now.Nodes[n.Name] = n
|
||||
}
|
||||
if now.equal(s.known) {
|
||||
return
|
||||
}
|
||||
if err := s.mem.Keep(ctx, now); err != nil {
|
||||
s.log.Warn("could not keep Slurm's nodes in the store", "err", err)
|
||||
return
|
||||
}
|
||||
s.known = now
|
||||
}
|
||||
|
||||
// controllerAddr is where the nodes reach slurmctld: the controller's
|
||||
// management address, else the one kept. Under s.mu.
|
||||
func (s *Service) controllerAddr(set config.SlurmSettings) string {
|
||||
if n, ok := s.inv.Get(set.Controller); ok && n.MgmtAddr != "" {
|
||||
return n.MgmtAddr
|
||||
}
|
||||
if s.known.Controller == set.Controller {
|
||||
return s.known.ControllerAddr
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// handOver moves each node of slurm.nodes a step between llmgw and Slurm,
|
||||
// answering where each is. Under s.mu.
|
||||
func (s *Service) handOver(set config.SlurmSettings) map[string]Phase {
|
||||
out := map[string]Phase{}
|
||||
if !set.Enabled {
|
||||
// Off: every node is llmgw's again. What Slurm still runs is left
|
||||
// to finish where it is; turning Slurm off with work lent is the
|
||||
// operator's choice.
|
||||
for node := range s.drained {
|
||||
s.ready(node)
|
||||
}
|
||||
return out
|
||||
}
|
||||
for _, node := range set.Nodes {
|
||||
out[node] = s.phase(set, node)
|
||||
}
|
||||
// A node no longer in slurm.nodes is llmgw's.
|
||||
for node := range s.drained {
|
||||
if !slices.Contains(set.Nodes, node) {
|
||||
s.ready(node)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (s *Service) phase(set config.SlurmSettings, node string) Phase {
|
||||
n, ok := s.inv.Get(node)
|
||||
st := s.status[node]
|
||||
if set.IsLent(node) {
|
||||
if ok && (n.State == inventory.Ready || n.State == inventory.Cordoned) {
|
||||
s.inv.SetState(node, inventory.Draining)
|
||||
s.drained[node] = true
|
||||
s.lendingAt[node] = time.Now()
|
||||
s.log.Info("lending a node to Slurm: its replicas are moved off first", "node", node,
|
||||
"stopped_by", time.Now().Add(drainGrace).Format(time.TimeOnly))
|
||||
}
|
||||
if !ok || len(s.inv.Allocations(node)) > 0 {
|
||||
return Lending
|
||||
}
|
||||
delete(s.lendingAt, node)
|
||||
return Lent
|
||||
}
|
||||
// Not lent. Until the agent says Slurm has drained it with no job left,
|
||||
// llmgw places nothing new on it: a new leader starts with every node
|
||||
// ready. Cordoned, not drained: llmgw has nothing there to move.
|
||||
if st == nil || slurmBusy(st) {
|
||||
if ok && n.State == inventory.Ready {
|
||||
s.inv.SetState(node, inventory.Cordoned)
|
||||
s.drained[node] = true
|
||||
if st != nil {
|
||||
s.log.Info("taking a node back from Slurm: llmgw places nothing there until its jobs end",
|
||||
"node", node, "jobs", st.Jobs)
|
||||
}
|
||||
}
|
||||
if st == nil {
|
||||
return Unknown
|
||||
}
|
||||
return Reclaiming
|
||||
}
|
||||
delete(s.lendingAt, node)
|
||||
if s.drained[node] {
|
||||
s.ready(node)
|
||||
}
|
||||
return Llmgw
|
||||
}
|
||||
|
||||
// drainGrace is how long a drained replica waits for its replacement
|
||||
// before it is stopped anyway: the reconciler's, which is not a setting.
|
||||
var drainGrace = reconciler.DefaultConfig().DrainGrace
|
||||
|
||||
// slurmBusy is whether Slurm may still run work on a node: a job step left
|
||||
// on it (or no telling), or a controller that would still give it one. A
|
||||
// node the controller does not know of, or cannot be asked about, gets no
|
||||
// job from it, so a Slurm that is broken never keeps a node from llmgw.
|
||||
func slurmBusy(st *Reported) bool {
|
||||
if st.Jobs != 0 {
|
||||
return true
|
||||
}
|
||||
if st.State == "" {
|
||||
return false
|
||||
}
|
||||
return !strings.Contains(st.State, "DRAIN") && !strings.Contains(st.State, "DOWN")
|
||||
}
|
||||
|
||||
func (s *Service) ready(node string) {
|
||||
if n, ok := s.inv.Get(node); ok && (n.State == inventory.Draining || n.State == inventory.Cordoned) {
|
||||
s.inv.SetState(node, inventory.Ready)
|
||||
s.log.Info("a node is llmgw's again: Slurm has no job there", "node", node)
|
||||
}
|
||||
delete(s.drained, node)
|
||||
}
|
||||
|
||||
// files are the controller's files, and the nodes left out of them for not
|
||||
// having reported their hardware. Under s.mu.
|
||||
func (s *Service) files(set config.SlurmSettings) (map[string]string, []string) {
|
||||
c, waiting := s.cluster(set)
|
||||
if c.ControllerAddr == "" {
|
||||
return nil, waiting
|
||||
}
|
||||
f := c.Files()
|
||||
if s.containersReady(set) {
|
||||
f["plugstack.conf"] = slurm.PlugstackConf
|
||||
}
|
||||
return f, waiting
|
||||
}
|
||||
|
||||
// containersReady is whether jobs may ask for containers: every machine of
|
||||
// the Slurm cluster says it has pyxis and enroot. Slurm fails a job on a
|
||||
// machine without the plugin it was told to load. Under s.mu.
|
||||
func (s *Service) containersReady(set config.SlurmSettings) bool {
|
||||
if !set.Containers {
|
||||
return false
|
||||
}
|
||||
for _, n := range s.machines(set) {
|
||||
if st := s.status[n]; st == nil || st.Containers == "" {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// cluster is what slurm.conf says: each node as it last reported, else as
|
||||
// kept from before, and the nodes left out for having neither. Under s.mu.
|
||||
func (s *Service) cluster(set config.SlurmSettings) (slurm.Cluster, []string) {
|
||||
c := slurm.Cluster{Name: set.Cluster(), Controller: set.Controller, Partition: set.PartitionName()}
|
||||
if !set.Enabled {
|
||||
return c, nil
|
||||
}
|
||||
c.ControllerAddr = s.controllerAddr(set)
|
||||
var waiting []string
|
||||
for _, node := range set.Nodes {
|
||||
sn, ok := s.reported(node)
|
||||
if !ok {
|
||||
sn, ok = s.known.Nodes[node]
|
||||
}
|
||||
if !ok {
|
||||
waiting = append(waiting, node)
|
||||
continue
|
||||
}
|
||||
sn.Lent = set.IsLent(node)
|
||||
c.Nodes = append(c.Nodes, sn)
|
||||
}
|
||||
return c, waiting
|
||||
}
|
||||
|
||||
// reported is a node as its agent and the inventory say now. Under s.mu.
|
||||
func (s *Service) reported(node string) (slurm.Node, bool) {
|
||||
n, ok := s.inv.Get(node)
|
||||
st := s.status[node]
|
||||
if !ok || n.MgmtAddr == "" || st == nil || st.Hardware == "" {
|
||||
return slurm.Node{}, false
|
||||
}
|
||||
hw, err := slurm.ParseHardware(st.Hardware)
|
||||
if err != nil {
|
||||
return slurm.Node{}, false
|
||||
}
|
||||
sn := slurm.Node{Name: node, Addr: n.MgmtAddr, Hardware: hw}
|
||||
for _, g := range n.GPUs {
|
||||
sn.GPUs = append(sn.GPUs, slurm.GPU{Minor: g.Index, Name: g.Name})
|
||||
}
|
||||
return sn, true
|
||||
}
|
||||
|
||||
// message is what a node is told. Under s.mu.
|
||||
func (s *Service) message(set config.SlurmSettings, node string, lent bool, key []byte,
|
||||
files map[string]string) *llmgwv1.Slurm {
|
||||
m := &llmgwv1.Slurm{}
|
||||
if !set.Enabled {
|
||||
return m
|
||||
}
|
||||
isNode := slices.Contains(set.Nodes, node)
|
||||
isCtl := set.Controller == node && files != nil
|
||||
if !isNode && !isCtl {
|
||||
return m
|
||||
}
|
||||
m.Name, m.MungeKey, m.Lent = node, key, lent
|
||||
m.Containers = set.Containers
|
||||
m.Node = isNode
|
||||
if addr := s.controllerAddr(set); addr != "" {
|
||||
m.ConfServer = net.JoinHostPort(addr, strconv.Itoa(slurm.CtldPort))
|
||||
}
|
||||
if isNode && m.ConfServer == "" {
|
||||
// Without its controller a node cannot start; it is told when the
|
||||
// controller registers. Told nothing until then, not "no part": a
|
||||
// node still running Slurm's jobs would stop them.
|
||||
return nil
|
||||
}
|
||||
if isCtl {
|
||||
m.Controller = true
|
||||
m.Files = maps.Clone(files)
|
||||
}
|
||||
if !s.sharedPart(set, node, m) || !s.usersPart(set, node, m) {
|
||||
return nil
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// sharedPart puts the shared files in m, false when where they come from
|
||||
// is not known yet: told nothing rather than no shared files, which would
|
||||
// unmount them under running jobs. Under s.mu.
|
||||
func (s *Service) sharedPart(set config.SlurmSettings, node string, m *llmgwv1.Slurm) bool {
|
||||
sh := set.Shared
|
||||
if sh.Path == "" {
|
||||
return true
|
||||
}
|
||||
m.Shared = &llmgwv1.SlurmShared{Path: sh.Path, Nfs: sh.NFS}
|
||||
switch {
|
||||
case sh.Server == "":
|
||||
case sh.Server == node:
|
||||
m.Shared.Serve = true
|
||||
for _, n := range s.machines(set) {
|
||||
if n == node {
|
||||
continue
|
||||
}
|
||||
if a := s.addr(set, n); a != "" {
|
||||
m.Shared.Clients = append(m.Shared.Clients, a)
|
||||
}
|
||||
}
|
||||
slices.Sort(m.Shared.Clients)
|
||||
default:
|
||||
a := s.addr(set, sh.Server)
|
||||
if a == "" {
|
||||
return false
|
||||
}
|
||||
if strings.Contains(a, ":") {
|
||||
a = "[" + a + "]"
|
||||
}
|
||||
m.Shared.Nfs = a + ":" + sh.Path
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// usersPart puts the users in m, false while they have not been read.
|
||||
// Under s.mu.
|
||||
func (s *Service) usersPart(set config.SlurmSettings, node string, m *llmgwv1.Slurm) bool {
|
||||
if set.Users != config.UsersLlmgw {
|
||||
return true
|
||||
}
|
||||
if !s.usersOK {
|
||||
return false
|
||||
}
|
||||
m.ManageUsers = true
|
||||
switch {
|
||||
case set.Shared.Path == "":
|
||||
m.MakeHomes = true // each machine's own /home
|
||||
case set.Shared.Server != "":
|
||||
m.MakeHomes = set.Shared.Server == node
|
||||
default:
|
||||
m.MakeHomes = set.Controller == node
|
||||
}
|
||||
for _, u := range s.users {
|
||||
m.Users = append(m.Users, &llmgwv1.SlurmUser{Login: u.Login, Uid: uint32(u.UID), Name: u.Name, //nolint:gosec // from FirstSlurmUID up
|
||||
Home: set.HomeDir(u.Login), SshKeys: slices.Clone(u.SSHKeys)})
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// machines are the controller and the nodes.
|
||||
func (s *Service) machines(set config.SlurmSettings) []string {
|
||||
out := slices.Clone(set.Nodes)
|
||||
if !slices.Contains(out, set.Controller) {
|
||||
out = append(out, set.Controller)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// addr is a machine's management address, from the inventory, else kept
|
||||
// from before. Under s.mu.
|
||||
func (s *Service) addr(set config.SlurmSettings, node string) string {
|
||||
if n, ok := s.inv.Get(node); ok && n.MgmtAddr != "" {
|
||||
return n.MgmtAddr
|
||||
}
|
||||
if node == set.Controller {
|
||||
return s.controllerAddr(set)
|
||||
}
|
||||
return s.known.Nodes[node].Addr
|
||||
}
|
||||
|
||||
// Status is the cluster as the service sees it now.
|
||||
func (s *Service) Status() Status {
|
||||
set := s.current()
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
out := Status{Enabled: set.Enabled, Cluster: set.Cluster(), Controller: set.Controller,
|
||||
Partition: set.PartitionName(), Ctld: s.status[set.Controller], Shared: set.Shared, Users: set.Users}
|
||||
if set.Users == config.UsersLlmgw {
|
||||
out.UserCount = len(s.users)
|
||||
}
|
||||
if !set.Enabled {
|
||||
return out
|
||||
}
|
||||
if set.Containers {
|
||||
out.Containers = "installing"
|
||||
if s.containersReady(set) {
|
||||
out.Containers = "on"
|
||||
}
|
||||
}
|
||||
_, out.Waiting = s.files(set)
|
||||
conn := s.agents.ConnectedNodes()
|
||||
for _, node := range set.Nodes {
|
||||
ns := NodeStatus{Node: node, Status: s.status[node], Replicas: len(s.inv.Allocations(node)),
|
||||
Connected: slices.Contains(conn, node)}
|
||||
ns.Phase = s.peek(set, node)
|
||||
if at, ok := s.lendingAt[node]; ok && ns.Phase == Lending {
|
||||
ns.StopsBy = at.Add(drainGrace)
|
||||
}
|
||||
out.Nodes = append(out.Nodes, ns)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// peek is phase without changing anything. Under s.mu.
|
||||
func (s *Service) peek(set config.SlurmSettings, node string) Phase {
|
||||
st := s.status[node]
|
||||
if set.IsLent(node) {
|
||||
if len(s.inv.Allocations(node)) > 0 {
|
||||
return Lending
|
||||
}
|
||||
return Lent
|
||||
}
|
||||
switch {
|
||||
case st == nil:
|
||||
return Unknown
|
||||
case slurmBusy(st):
|
||||
return Reclaiming
|
||||
}
|
||||
return Llmgw
|
||||
}
|
||||
|
||||
// Describe is one line for a log or a screen.
|
||||
func (n NodeStatus) Describe() string {
|
||||
switch n.Phase {
|
||||
case Lending:
|
||||
if !n.StopsBy.IsZero() {
|
||||
// A replica moves once its replacement serves; one with nowhere
|
||||
// else to go waits out the grace, and that wait is not a hang.
|
||||
return fmt.Sprintf("being lent: %d replica(s) still to move off; one with nowhere else to go is stopped at %s",
|
||||
n.Replicas, n.StopsBy.Local().Format(time.TimeOnly))
|
||||
}
|
||||
return fmt.Sprintf("being lent: %d replica(s) still to move off", n.Replicas)
|
||||
case Lent:
|
||||
return "lent to Slurm"
|
||||
case Reclaiming:
|
||||
if n.Status != nil {
|
||||
return fmt.Sprintf("being taken back: %d job(s) still running", n.Status.Jobs)
|
||||
}
|
||||
return "being taken back"
|
||||
case Llmgw:
|
||||
return "llmgw's"
|
||||
}
|
||||
return "not heard from yet"
|
||||
}
|
||||
@@ -0,0 +1,325 @@
|
||||
package slurmsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
type fakeInv struct {
|
||||
nodes map[string]*inventory.Node
|
||||
allocs map[string]int
|
||||
}
|
||||
|
||||
func (f *fakeInv) Get(id string) (inventory.Node, bool) {
|
||||
n, ok := f.nodes[id]
|
||||
if !ok {
|
||||
return inventory.Node{}, false
|
||||
}
|
||||
return *n, true
|
||||
}
|
||||
|
||||
func (f *fakeInv) SetState(id string, st inventory.NodeState) bool {
|
||||
n, ok := f.nodes[id]
|
||||
if ok {
|
||||
n.State = st
|
||||
}
|
||||
return ok
|
||||
}
|
||||
|
||||
func (f *fakeInv) Allocations(id string) []inventory.Allocation {
|
||||
return make([]inventory.Allocation, f.allocs[id])
|
||||
}
|
||||
|
||||
func (f *fakeInv) Nodes() []inventory.Node {
|
||||
var out []inventory.Node
|
||||
for _, n := range f.nodes {
|
||||
out = append(out, *n)
|
||||
}
|
||||
slices.SortFunc(out, func(a, b inventory.Node) int { return strings.Compare(a.ID, b.ID) })
|
||||
return out
|
||||
}
|
||||
|
||||
type fakeAgents struct {
|
||||
conn []string
|
||||
got map[string][]*llmgwv1.Slurm
|
||||
}
|
||||
|
||||
func (f *fakeAgents) Send(node string, m *llmgwv1.ControlMessage) error {
|
||||
f.got[node] = append(f.got[node], m.GetSlurm())
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeAgents) ConnectedNodes() []string { return f.conn }
|
||||
|
||||
func (f *fakeAgents) last(node string) *llmgwv1.Slurm {
|
||||
ms := f.got[node]
|
||||
if len(ms) == 0 {
|
||||
return nil
|
||||
}
|
||||
return ms[len(ms)-1]
|
||||
}
|
||||
|
||||
type rig struct {
|
||||
set config.SlurmSettings
|
||||
inv *fakeInv
|
||||
agents *fakeAgents
|
||||
svc *Service
|
||||
}
|
||||
|
||||
func newRig() *rig {
|
||||
r := &rig{
|
||||
set: config.SlurmSettings{Enabled: true, Controller: "node06", Nodes: []string{"cuda01"}},
|
||||
inv: &fakeInv{nodes: map[string]*inventory.Node{
|
||||
"node06": {ID: "node06", MgmtAddr: "10.0.0.146"},
|
||||
"cuda01": {ID: "cuda01", MgmtAddr: "10.0.0.147", GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA GeForce RTX 3090"}}},
|
||||
}, allocs: map[string]int{"cuda01": 2}},
|
||||
agents: &fakeAgents{conn: []string{"node06", "cuda01"}, got: map[string][]*llmgwv1.Slurm{}},
|
||||
}
|
||||
r.svc = New(func() config.SlurmSettings { return r.set }, r.inv, r.agents,
|
||||
func(context.Context) ([]byte, error) { return []byte(strings.Repeat("k", 1024)), nil }, nil)
|
||||
return r
|
||||
}
|
||||
|
||||
func (r *rig) report(node string, st *llmgwv1.SlurmStatus) { r.svc.OnSlurmStatus(node, st) }
|
||||
|
||||
func (r *rig) tick() { r.svc.Tick(context.Background()) }
|
||||
|
||||
const hw = "NodeName=cuda01 CPUs=16 RealMemory=64000"
|
||||
|
||||
func TestMessagesAndFiles(t *testing.T) {
|
||||
r := newRig()
|
||||
r.tick()
|
||||
ctl, node := r.agents.last("node06"), r.agents.last("cuda01")
|
||||
if !ctl.GetController() || ctl.GetNode() || len(ctl.GetMungeKey()) != 1024 {
|
||||
t.Fatalf("controller told %v", ctl)
|
||||
}
|
||||
if !node.GetNode() || node.GetController() || node.GetConfServer() != "10.0.0.146:6817" || node.GetLent() ||
|
||||
node.GetName() != "cuda01" || len(node.GetFiles()) != 0 {
|
||||
t.Fatalf("node told %v", node)
|
||||
}
|
||||
if strings.Contains(ctl.GetFiles()["slurm.conf"], "NodeName=cuda01") {
|
||||
t.Fatal("a node with no hardware reported is in slurm.conf")
|
||||
}
|
||||
if w := r.svc.Status().Waiting; len(w) != 1 || w[0] != "cuda01" {
|
||||
t.Fatalf("waiting %v", w)
|
||||
}
|
||||
// Same again: nobody is told twice.
|
||||
r.tick()
|
||||
if len(r.agents.got["node06"]) != 1 || len(r.agents.got["cuda01"]) != 1 {
|
||||
t.Fatal("told again with nothing changed")
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
|
||||
r.tick()
|
||||
conf := r.agents.last("node06").GetFiles()["slurm.conf"]
|
||||
if !strings.Contains(conf, "NodeName=cuda01 NodeAddr=10.0.0.147 CPUs=16 RealMemory=62080 Gres=gpu:rtx3090:1") {
|
||||
t.Fatalf("slurm.conf:\n%s", conf)
|
||||
}
|
||||
if !strings.Contains(r.agents.last("node06").GetFiles()["gres.conf"], "File=/dev/nvidia0") {
|
||||
t.Fatal("gres.conf has no card")
|
||||
}
|
||||
if len(r.agents.got["cuda01"]) != 1 {
|
||||
t.Fatal("a node was told again when only the controller's files changed")
|
||||
}
|
||||
// A new stream is told again.
|
||||
r.svc.OnRegister("cuda01")
|
||||
r.tick()
|
||||
if len(r.agents.got["cuda01"]) != 2 {
|
||||
t.Fatal("a new stream was not told")
|
||||
}
|
||||
if r.inv.nodes["cuda01"].State != inventory.Ready {
|
||||
t.Fatal("a node drained in Slurm with no job was taken from llmgw")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLendAndReclaim(t *testing.T) {
|
||||
r := newRig()
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
|
||||
r.set.Lent = []string{"cuda01"}
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State != inventory.Draining {
|
||||
t.Fatal("a lent node was not drained of its replicas")
|
||||
}
|
||||
if r.agents.last("cuda01").GetLent() {
|
||||
t.Fatal("lent to Slurm with replicas still on it")
|
||||
}
|
||||
if p := r.svc.Status().Nodes[0].Phase; p != Lending {
|
||||
t.Fatalf("phase %s", p)
|
||||
}
|
||||
r.inv.allocs["cuda01"] = 0
|
||||
r.tick()
|
||||
if !r.agents.last("cuda01").GetLent() || r.svc.Status().Nodes[0].Phase != Lent {
|
||||
t.Fatal("not lent once its replicas were gone")
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "ALLOCATED", Jobs: 3})
|
||||
|
||||
// Taken back: Slurm drains it; llmgw keeps off until its jobs end.
|
||||
r.set.Lent = nil
|
||||
r.tick()
|
||||
if r.agents.last("cuda01").GetLent() {
|
||||
t.Fatal("still lent")
|
||||
}
|
||||
if s := r.inv.nodes["cuda01"].State; s != inventory.Draining && s != inventory.Cordoned {
|
||||
t.Fatalf("llmgw may place on a node with Slurm jobs: %v", s)
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "ALLOCATED+DRAIN", Jobs: 1})
|
||||
r.tick()
|
||||
if r.svc.Status().Nodes[0].Phase != Reclaiming || r.inv.nodes["cuda01"].State == inventory.Ready {
|
||||
t.Fatal("given back with a job left")
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: serving models"})
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State != inventory.Ready || r.svc.Status().Nodes[0].Phase != Llmgw {
|
||||
t.Fatal("not llmgw's when its last job ended")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewLeaderKeepsOffUntilHeard(t *testing.T) {
|
||||
r := newRig()
|
||||
r.inv.allocs["cuda01"] = 0
|
||||
r.tick()
|
||||
// Nothing heard yet: cordoned, nothing moved.
|
||||
if r.inv.nodes["cuda01"].State != inventory.Cordoned {
|
||||
t.Fatalf("state %v before the agent said anything", r.inv.nodes["cuda01"].State)
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "MIXED", Jobs: 2})
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State == inventory.Ready {
|
||||
t.Fatal("ready with Slurm jobs on it")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBrokenSlurmLeavesTheNodeToLlmgw(t *testing.T) {
|
||||
r := newRig()
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Error: "installing Slurm: no package", Jobs: 0})
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State != inventory.Ready {
|
||||
t.Fatal("a Slurm that does not work took a node from llmgw")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOperatorsCordonIsLeft(t *testing.T) {
|
||||
r := newRig()
|
||||
r.inv.nodes["cuda01"].State = inventory.Cordoned
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN"})
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State != inventory.Cordoned {
|
||||
t.Fatal("llmgw uncordoned a node an operator cordoned")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOffAndOut(t *testing.T) {
|
||||
r := newRig()
|
||||
r.inv.allocs["cuda01"] = 0
|
||||
r.set.Lent = []string{"cuda01"}
|
||||
r.tick()
|
||||
r.set.Enabled = false
|
||||
r.tick()
|
||||
if m := r.agents.last("cuda01"); m.GetNode() || m.GetController() {
|
||||
t.Fatalf("told %v with Slurm off", m)
|
||||
}
|
||||
if r.inv.nodes["cuda01"].State != inventory.Ready {
|
||||
t.Fatal("not llmgw's with Slurm off")
|
||||
}
|
||||
|
||||
r = newRig()
|
||||
r.inv.allocs["cuda01"] = 0
|
||||
r.set.Lent = []string{"cuda01"}
|
||||
r.tick()
|
||||
r.set.Nodes, r.set.Lent = nil, nil
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State != inventory.Ready || r.agents.last("cuda01").GetNode() {
|
||||
t.Fatal("a node taken out of slurm.nodes is not llmgw's")
|
||||
}
|
||||
if !r.agents.last("node06").GetController() {
|
||||
t.Fatal("the controller stopped with no nodes")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoControllerYet(t *testing.T) {
|
||||
r := newRig()
|
||||
delete(r.inv.nodes, "node06")
|
||||
r.agents.conn = []string{"cuda01"}
|
||||
r.tick()
|
||||
if m := r.agents.last("cuda01"); m.GetNode() {
|
||||
t.Fatalf("a node was started with no controller to fetch its files from: %v", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAllNodes(t *testing.T) {
|
||||
r := newRig()
|
||||
r.set.Nodes, r.set.AllNodes = nil, true
|
||||
r.inv.nodes["cuda02"] = &inventory.Node{ID: "cuda02", MgmtAddr: "10.0.0.150"}
|
||||
r.agents.conn = append(r.agents.conn, "cuda02")
|
||||
r.tick()
|
||||
for _, n := range []string{"cuda01", "cuda02", "node06"} {
|
||||
if !r.agents.last(n).GetNode() {
|
||||
t.Fatalf("%s, a node that joined, does not run slurmd", n)
|
||||
}
|
||||
}
|
||||
if got := len(r.svc.Status().Nodes); got != 3 {
|
||||
t.Fatalf("%d nodes in the status", got)
|
||||
}
|
||||
r.set.Lent = []string{"cuda02"}
|
||||
r.tick()
|
||||
if p := r.svc.Status().Nodes[1]; p.Node != "cuda02" || p.Phase != Lent {
|
||||
t.Fatalf("cuda02 not lent: %+v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLendingSaysWhenAReplicaWithNowhereToGoStops(t *testing.T) {
|
||||
// Seen on the PoC: a GPU model with no other card waited out the drain's
|
||||
// grace, and five minutes of "1 replica(s) still to move off" read as a hang.
|
||||
at := time.Date(2026, 11, 14, 19, 38, 34, 0, time.Local)
|
||||
n := NodeStatus{Node: "cuda01", Phase: Lending, Replicas: 1, StopsBy: at}
|
||||
if d := n.Describe(); !strings.Contains(d, "nowhere else to go is stopped at 19:38:34") {
|
||||
t.Errorf("%q", d)
|
||||
}
|
||||
n.StopsBy = time.Time{}
|
||||
if d := n.Describe(); d != "being lent: 1 replica(s) still to move off" {
|
||||
t.Errorf("%q", d)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPyxisIsLoadedOnlyOnceEveryMachineHasIt(t *testing.T) {
|
||||
// "required" in plugstack.conf fails every job on a machine without the
|
||||
// plugin; sent early, it would stop Slurm's jobs everywhere it is missing.
|
||||
r := newRig()
|
||||
r.set.Containers = true
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw})
|
||||
r.tick()
|
||||
if !r.agents.last("cuda01").GetContainers() || !r.agents.last("node06").GetContainers() {
|
||||
t.Fatal("the machines were not told to set containers up")
|
||||
}
|
||||
if _, ok := r.agents.last("node06").GetFiles()["plugstack.conf"]; ok {
|
||||
t.Fatal("pyxis loaded before any machine has it")
|
||||
}
|
||||
if s := r.svc.Status().Containers; s != "installing" {
|
||||
t.Errorf("status %q", s)
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, Containers: "pyxis 0.24.0, enroot 4.2.1"})
|
||||
r.tick()
|
||||
if _, ok := r.agents.last("node06").GetFiles()["plugstack.conf"]; ok {
|
||||
t.Fatal("pyxis loaded with the controller still without it")
|
||||
}
|
||||
r.report("node06", &llmgwv1.SlurmStatus{Containers: "pyxis 0.24.0, enroot 4.2.1"})
|
||||
r.tick()
|
||||
if p := r.agents.last("node06").GetFiles()["plugstack.conf"]; !strings.Contains(p, "required /usr/local/lib/slurm/spank_pyxis.so") {
|
||||
t.Fatalf("plugstack.conf %q", p)
|
||||
}
|
||||
if s := r.svc.Status().Containers; s != "on" {
|
||||
t.Errorf("status %q", s)
|
||||
}
|
||||
r.set.Containers = false
|
||||
r.tick()
|
||||
if _, ok := r.agents.last("node06").GetFiles()["plugstack.conf"]; ok || r.svc.Status().Containers != "" {
|
||||
t.Fatal("pyxis still loaded with containers off")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package slurmsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"slices"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
)
|
||||
|
||||
func TestSharedFilesFromTheController(t *testing.T) {
|
||||
r := newRig()
|
||||
r.set.Shared = config.SharedStorage{Path: "/shared", Server: "node06"}
|
||||
r.tick()
|
||||
ctl, node := r.agents.last("node06").GetShared(), r.agents.last("cuda01").GetShared()
|
||||
if !ctl.GetServe() || !slices.Equal(ctl.GetClients(), []string{"10.0.0.147"}) || ctl.GetPath() != "/shared" {
|
||||
t.Fatalf("server told %v", ctl)
|
||||
}
|
||||
if node.GetServe() || node.GetNfs() != "10.0.0.146:/shared" {
|
||||
t.Fatalf("client told %v", node)
|
||||
}
|
||||
// The site's own export: every machine mounts it.
|
||||
r.set.Shared = config.SharedStorage{Path: "/shared", NFS: "filer:/hpc"}
|
||||
r.tick()
|
||||
if r.agents.last("node06").GetShared().GetNfs() != "filer:/hpc" || r.agents.last("cuda01").GetShared().GetServe() {
|
||||
t.Fatal("the site's export not sent")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSharedServerNotKnownTellsNothing(t *testing.T) {
|
||||
r := newRig()
|
||||
r.set.Shared = config.SharedStorage{Path: "/shared", Server: "node06"}
|
||||
delete(r.inv.nodes, "node06")
|
||||
r.svc.known.ControllerAddr, r.svc.known.Controller = "10.0.0.146", "node06"
|
||||
r.tick()
|
||||
if r.agents.last("cuda01").GetShared().GetNfs() != "10.0.0.146:/shared" {
|
||||
t.Fatalf("the kept address not used: %v", r.agents.last("cuda01"))
|
||||
}
|
||||
r2 := newRig()
|
||||
r2.set.Nodes = []string{"cuda01", "site2"}
|
||||
r2.agents.conn = append(r2.agents.conn, "site2")
|
||||
r2.set.Shared = config.SharedStorage{Path: "/shared", Server: "site2"}
|
||||
r2.tick()
|
||||
if r2.agents.last("cuda01") != nil {
|
||||
t.Fatal("told with no address for the server: its mount would go")
|
||||
}
|
||||
}
|
||||
|
||||
func TestUsersSentOnceRead(t *testing.T) {
|
||||
r := newRig()
|
||||
r.set.Users = config.UsersLlmgw
|
||||
r.set.Shared = config.SharedStorage{Path: "/shared", Server: "node06"}
|
||||
fail := true
|
||||
r.svc.SetUsers(func(context.Context) ([]User, error) {
|
||||
if fail {
|
||||
return nil, errors.New("store down")
|
||||
}
|
||||
return []User{{Login: "ayse", UID: 200000, Name: "Ayşe", SSHKeys: []string{"ssh-ed25519 AAAA"}}}, nil
|
||||
})
|
||||
r.tick()
|
||||
if len(r.agents.got["cuda01"]) != 0 {
|
||||
t.Fatal("told before the users were read: every user would go")
|
||||
}
|
||||
fail = false
|
||||
r.svc.UsersChanged()
|
||||
r.tick()
|
||||
ctl, node := r.agents.last("node06"), r.agents.last("cuda01")
|
||||
if !ctl.GetManageUsers() || !ctl.GetMakeHomes() || node.GetMakeHomes() || len(node.GetUsers()) != 1 {
|
||||
t.Fatalf("controller %v\nnode %v", ctl, node)
|
||||
}
|
||||
if u := node.GetUsers()[0]; u.GetHome() != "/shared/home/ayse" || u.GetUid() != 200000 || len(u.GetSshKeys()) != 1 {
|
||||
t.Fatalf("user %v", u)
|
||||
}
|
||||
// The store down later: the last users are kept.
|
||||
fail = true
|
||||
r.svc.UsersChanged()
|
||||
r.tick()
|
||||
if len(r.agents.last("cuda01").GetUsers()) != 1 {
|
||||
t.Fatal("users lost when the store could not be read")
|
||||
}
|
||||
if st := r.svc.Status(); st.UserCount != 1 || st.Shared.Server != "node06" {
|
||||
t.Fatalf("status %+v", st)
|
||||
}
|
||||
}
|
||||
@@ -94,7 +94,7 @@ type Store interface {
|
||||
SetStaffRole(ctx context.Context, id, role string) error
|
||||
SetUserState(ctx context.Context, id string, state store.UserState) error
|
||||
DeleteUserSessions(ctx context.Context, userID string) error
|
||||
AddMembership(ctx context.Context, user string, scope store.Scope, id, role, by string, at time.Time) error
|
||||
EnsureMembership(ctx context.Context, user string, scope store.Scope, id, by string, at time.Time) error
|
||||
RemoveMembership(ctx context.Context, userID string, scope store.Scope, scopeID string, at time.Time) (int, error)
|
||||
MembershipsOf(ctx context.Context, userID string) ([]store.Membership, error)
|
||||
AllOrgs(ctx context.Context) ([]store.Org, error)
|
||||
@@ -221,7 +221,7 @@ func (k *Keeper) apply(ctx context.Context, u *store.User, id Identity, remove b
|
||||
continue // a mapping to a place the tree lacks: said by Sync
|
||||
}
|
||||
want[string(scope)+"/"+sid] = scope
|
||||
if err := k.St.AddMembership(ctx, u.ID, scope, sid, store.RoleMember, k.Name, k.now()); err != nil {
|
||||
if err := k.St.EnsureMembership(ctx, u.ID, scope, sid, k.Name, k.now()); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
@@ -480,6 +480,7 @@ var Reference = []Block{
|
||||
{Key: "mail", Type: "mail.Config", Doc: "Mail is where the gateway's few emails go out from: a link to set a\nforgotten password. Unset, none is sent and people ask an operator.", Nested: "mail.Config"},
|
||||
{Key: "serving", Type: "ServingSettings", Doc: "Serving stops and starts the whole cluster's service (llmgw stop, llmgw\nstart).", Nested: "config.ServingSettings"},
|
||||
{Key: "firewall", Type: "FirewallSettings", Doc: "Firewall turns llmgw's firewall on every machine on or off.", Nested: "config.FirewallSettings"},
|
||||
{Key: "slurm", Type: "SlurmSettings", Doc: "Slurm is a Slurm cluster run beside llmgw on some of its nodes, and\nwhich of them it may use now.", Nested: "config.SlurmSettings"},
|
||||
{Key: "presets", Type: "PresetSettings", Doc: "Presets is how the models that come with llmgw reach the catalog.", Nested: "config.PresetSettings"},
|
||||
{Key: "on_demand", Type: "OnDemandSettings", Doc: "OnDemand is how models that load when asked for take turns on the\ncards.", Nested: "config.OnDemandSettings"},
|
||||
{Key: "agents", Type: "AgentSettings", Doc: "Agents are the nodes' agent settings: agent.yaml's keys but those an\nagent needs to start, for every node (default) and for one\n(nodes.<id>). Sent to the agents as they change; an agent whose\nsettings changed starts again on them, its engines serving\nthroughout.", Nested: "config.AgentSettings"},
|
||||
@@ -509,9 +510,26 @@ var Reference = []Block{
|
||||
{Package: "config", Name: "ServingSettings", Title: "ServingSettings", Doc: "", Fields: []Field{
|
||||
{Key: "stopped", Type: "bool", Doc: "Stopped takes every model down and answers every request for one with\n503 (system_stopped), leaving the control planes, agents and mirror\nrunning and every model's definition as it is: llmgw stop sets it,\nllmgw start clears it, and the models come back as their catalog\nentries ask. In force on every control plane within seconds.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "SharedStorage", Title: "SharedStorage", Doc: "", Fields: []Field{
|
||||
{Key: "path", Type: "string", Doc: "Path is where every machine sees it, as /shared.", Nested: ""},
|
||||
{Key: "nfs", Type: "string", Doc: "NFS is the site's export, host:/path, mounted at Path on every\nmachine of the Slurm cluster.", Nested: ""},
|
||||
{Key: "server", Type: "string", Doc: "Server is the llmgw node, the controller or one of the nodes, that\nexports Path over NFS to the others: for a site with no storage of\nits own. One machine: not highly available, and no faster than its\ndisk and network.", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "SimConfig", Title: "SimConfig", Doc: "", Fields: []Field{
|
||||
{Key: "bin", Type: "string", Doc: "Bin is the llmgw binary itself; the engine is one of its subcommands.\nEmpty means whatever started this agent, which is right almost always.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "SlurmSettings", Title: "SlurmSettings", Doc: "", Fields: []Field{
|
||||
{Key: "enabled", Type: "bool", Doc: "Enabled runs Slurm on the nodes below. False stops llmgw keeping it;\nwhat was installed is left installed, and every node is llmgw's.", Nested: ""},
|
||||
{Key: "cluster_name", Type: "string", Doc: "ClusterName is Slurm's ClusterName. Empty is \"llmgw\".", Nested: ""},
|
||||
{Key: "controller", Type: "string", Doc: "Controller is the llmgw node whose agent runs slurmctld. It needs no\nGPU, and need not be one of Nodes.", Nested: ""},
|
||||
{Key: "nodes", Type: "[]string", Doc: "Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained\nin Slurm, until it is also in Lent.", Nested: ""},
|
||||
{Key: "all_nodes", Type: "bool", Doc: "AllNodes runs slurmd on every llmgw node, those that join later too,\nbesides any named in Nodes. Each is llmgw's until it is lent, as any.", Nested: ""},
|
||||
{Key: "lent", Type: "[]string", Doc: "Lent are the nodes of Nodes Slurm may run jobs on now. A node put here\nis drained of llmgw's replicas first, as `ctl drain` does, and resumed\nin Slurm once none is left; a node taken out is drained in Slurm and\ngiven back to llmgw when its last job ends.", Nested: ""},
|
||||
{Key: "partition", Type: "string", Doc: "Partition is the one partition every node is in. Empty is \"gpu\".", Nested: ""},
|
||||
{Key: "shared", Type: "SharedStorage", Doc: "Shared is the file system every machine of the Slurm cluster sees at\nthe same path: home directories, datasets, the jobs' output and\ncheckpoints. Empty path: none, and a batch job's output stays on the\nnode it ran on.", Nested: "config.SharedStorage"},
|
||||
{Key: "users", Type: "string", Doc: "Users is who keeps the Linux users that run jobs, the same on every\nmachine of the Slurm cluster: \"llmgw\" makes one for each active llmgw\naccount (its login from the e-mail address, a UID of llmgw's from\n200000 up, its SSH keys from the portal), let in to the controller,\nand to a node only while a job of theirs runs there (pam_slurm_adopt);\nempty leaves users to the site (a directory through sssd, or by hand).", Nested: ""},
|
||||
{Key: "containers", Type: "bool", Doc: "Containers lets jobs run in containers: `srun --container-image\nnvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),\nthe GPUs and the shared path inside. The agents install both on\nevery Slurm machine; jobs take it once every machine has them.", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "StrataConfig", Title: "StrataConfig", Doc: "", Fields: []Field{
|
||||
{Key: "dir", Type: "string", Doc: "Dir is Strata's source directory, where setup.py and serve/ are.", Nested: ""},
|
||||
{Key: "python", Type: "string", Doc: "Python is the interpreter of its environment; Dir's .venv when unset.", Nested: ""},
|
||||
|
||||
@@ -41,6 +41,7 @@ var Sources = []string{
|
||||
"internal/config/serving.go",
|
||||
"internal/config/presets.go",
|
||||
"internal/config/ondemand.go",
|
||||
"internal/config/slurm.go",
|
||||
"internal/config/agents.go",
|
||||
"internal/mail/mail.go",
|
||||
"internal/notify/notify.go",
|
||||
|
||||
@@ -173,6 +173,29 @@ func (o Option) WithContext(kvPerToken int64, tokens int) Option {
|
||||
return o
|
||||
}
|
||||
|
||||
// MostContext is the longest context, up to trained, this option still holds
|
||||
// for one request, in whole thousands of tokens below trained: trained when
|
||||
// the KV cache's size is unknown, 0 when trained is, for Strata (its context
|
||||
// is its own), or when not even 1024 tokens fit.
|
||||
func (o Option) MostContext(kvPerToken int64, trained int) int {
|
||||
switch {
|
||||
case trained <= 0 || o.Engine == "strata":
|
||||
return 0
|
||||
case kvPerToken <= 0 || o.WithContext(kvPerToken, trained).Fits():
|
||||
return trained
|
||||
}
|
||||
lo, hi := 0, trained/1024+1 // lo fits (or is none), hi does not
|
||||
for hi-lo > 1 {
|
||||
mid := (lo + hi) / 2
|
||||
if o.WithContext(kvPerToken, mid*1024).Fits() {
|
||||
lo = mid
|
||||
} else {
|
||||
hi = mid
|
||||
}
|
||||
}
|
||||
return lo * 1024
|
||||
}
|
||||
|
||||
// MemoryText says what it needs, part by part.
|
||||
func (o Option) MemoryText() string {
|
||||
if o.Engine == "strata" {
|
||||
|
||||
@@ -193,3 +193,23 @@ func TestAnOptionSaysWhatItNeedsOfEachCardAndOfAll(t *testing.T) {
|
||||
t.Errorf("a 2M-token context fits four cards: %s", huge.MemoryText())
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheMostContextAnOptionHolds(t *testing.T) {
|
||||
// 4 GB of weights on a 24 GB card, a token's cache 1 MiB.
|
||||
o := Option{Engine: "llamacpp", Class: "6000-24g", GPUsPerNode: 1, Nodes: 1,
|
||||
WeightsGB: 4, EngineGB: 1, NeedGB: 5, HaveGB: 21.6}
|
||||
const kv = 1 << 20
|
||||
if got := o.MostContext(kv, 8192); got != 8192 {
|
||||
t.Errorf("a context that fits whole: %d, want what it was trained for", got)
|
||||
}
|
||||
got := o.MostContext(kv, 131072)
|
||||
if got <= 0 || got%1024 != 0 || !o.WithContext(kv, got).Fits() || o.WithContext(kv, got+1024).Fits() {
|
||||
t.Errorf("the most that fits is %d", got)
|
||||
}
|
||||
if got := o.MostContext(0, 32768); got != 32768 {
|
||||
t.Errorf("an unknown cache size: %d, want what it was trained for", got)
|
||||
}
|
||||
if got := (Option{Engine: "strata"}).MostContext(kv, 32768); got != 0 {
|
||||
t.Errorf("Strata's context is its own: %d", got)
|
||||
}
|
||||
}
|
||||
@@ -79,6 +79,9 @@ type Options struct {
|
||||
|
||||
// Addr is how to reach the machine over SSH, and User who to log in as.
|
||||
Addr, User string
|
||||
// FallbackUser logs in when User is empty and the ssh config names no
|
||||
// user for the machine.
|
||||
FallbackUser string
|
||||
// Node is its id in the cluster. Empty means its hostname, which is what
|
||||
// it should be: the node id, the hostname and the certificate's name
|
||||
// being one string saves an afternoon each time they would differ.
|
||||
@@ -193,7 +196,7 @@ type Runner struct {
|
||||
|
||||
// New makes a Runner that reaches the machine with the system's ssh.
|
||||
func New(o Options, emit func(Event)) *Runner {
|
||||
return &Runner{Opts: o, Remote: &SSH{Addr: o.Addr, User: o.User}, Emit: emit}
|
||||
return &Runner{Opts: o, Remote: &SSH{Addr: o.Addr, User: o.User, FallbackUser: o.FallbackUser}, Emit: emit}
|
||||
}
|
||||
|
||||
func (r *Runner) emit(step, level, detail, fix string) {
|
||||
@@ -302,6 +305,9 @@ func (r *Runner) connect(ctx context.Context, res *Result) error {
|
||||
}
|
||||
|
||||
func (r *Runner) target() string {
|
||||
if s, ok := r.Remote.(*SSH); ok {
|
||||
return s.Target(context.Background())
|
||||
}
|
||||
if r.Opts.User == "" {
|
||||
return r.Opts.Addr
|
||||
}
|
||||
|
||||
+33
-4
@@ -5,6 +5,7 @@ import (
|
||||
"context"
|
||||
"os/exec"
|
||||
"runtime"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// Remote runs a shell command on the machine being onboarded, with input.
|
||||
@@ -14,8 +15,39 @@ type Remote interface {
|
||||
|
||||
// SSH is Remote through the system's ssh, so the operator's keys, agent and
|
||||
// ~/.ssh/config apply exactly as they do when they log in by hand.
|
||||
//
|
||||
// Addr is looked up in the ssh config first: a Host block whose HostName is
|
||||
// Addr is connected through by its name, so its User, IdentityFile and
|
||||
// ProxyJump apply to a machine given by address, without the address
|
||||
// having to be added to its Host line.
|
||||
type SSH struct {
|
||||
Addr, User string
|
||||
// FallbackUser logs in when User is empty and the ssh config names no
|
||||
// user for the machine: under sudo, the person who ran it.
|
||||
FallbackUser string
|
||||
|
||||
once sync.Once
|
||||
target string
|
||||
}
|
||||
|
||||
// Target is what ssh is given: [user@]host, host being the ssh config's
|
||||
// name for Addr when it has one.
|
||||
func (s *SSH) Target(ctx context.Context) string {
|
||||
s.once.Do(func() {
|
||||
host := s.Addr
|
||||
if a := configAlias(ctx, s.Addr); a != "" {
|
||||
host = a
|
||||
}
|
||||
u := s.User
|
||||
if u == "" && configUser(ctx, host) == "" {
|
||||
u = s.FallbackUser
|
||||
}
|
||||
s.target = host
|
||||
if u != "" {
|
||||
s.target = u + "@" + host
|
||||
}
|
||||
})
|
||||
return s.target
|
||||
}
|
||||
|
||||
// sshOptions never prompt: a question nobody is there to answer would hang
|
||||
@@ -37,10 +69,7 @@ var sshOptions = []string{
|
||||
// Run runs script with sh on the machine. Files travel on standard input,
|
||||
// not scp, which a hardened sshd may not offer.
|
||||
func (s *SSH) Run(ctx context.Context, script string, stdin []byte) ([]byte, error) {
|
||||
target := s.Addr
|
||||
if s.User != "" {
|
||||
target = s.User + "@" + s.Addr
|
||||
}
|
||||
target := s.Target(ctx)
|
||||
// "--" before the target: an address beginning with "-" is not read as
|
||||
// an option.
|
||||
args := append(append([]string{}, sshOptions...), "--", target, "sh", "-c", shellQuote(script))
|
||||
|
||||
@@ -0,0 +1,223 @@
|
||||
package onboard
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Where ssh reads its configuration, and arguments every `ssh -G` here is
|
||||
// given; variables so a test can point them at a file of its own.
|
||||
var (
|
||||
userSSHConfig = func() string { return filepath.Join(sshHome(), ".ssh", "config") }
|
||||
systemSSHConfig = "/etc/ssh/ssh_config"
|
||||
sshExtraArgs []string
|
||||
)
|
||||
|
||||
// sshHome is the home ssh itself reads ~/.ssh from: the account's, from the
|
||||
// password database, not $HOME, which sudo may leave as the caller's.
|
||||
func sshHome() string {
|
||||
if u, err := user.Current(); err == nil && u.HomeDir != "" {
|
||||
return u.HomeDir
|
||||
}
|
||||
h, _ := os.UserHomeDir()
|
||||
return h
|
||||
}
|
||||
|
||||
// configAlias is the ssh config's name for addr: the first concrete Host
|
||||
// whose HostName is addr, or resolves to the same address. A machine is
|
||||
// usually written in the config under its name, `Host gpu01` with
|
||||
// `HostName 10.0.0.21` and the User, IdentityFile or ProxyJump it needs,
|
||||
// and given to node add by its address, which that block does not match.
|
||||
// Empty when addr is itself named on a Host line, or no block points at it.
|
||||
func configAlias(ctx context.Context, addr string) string {
|
||||
aliases := map[string]bool{}
|
||||
var order []string
|
||||
seen := map[string]bool{}
|
||||
collect := func(path, base string) {
|
||||
for _, h := range hostNames(path, base, seen, 0) {
|
||||
if !aliases[h] {
|
||||
aliases[h] = true
|
||||
order = append(order, h)
|
||||
}
|
||||
}
|
||||
}
|
||||
collect(userSSHConfig(), filepath.Join(sshHome(), ".ssh"))
|
||||
if systemSSHConfig != "" {
|
||||
collect(systemSSHConfig, filepath.Dir(systemSSHConfig))
|
||||
}
|
||||
for h := range aliases {
|
||||
if strings.EqualFold(h, addr) {
|
||||
return ""
|
||||
}
|
||||
}
|
||||
if len(order) > 256 {
|
||||
order = order[:256]
|
||||
}
|
||||
// Names are looked up only when no HostName is the address as written,
|
||||
// and within one budget: a config of many hosts must not hold up a
|
||||
// connection by seconds each.
|
||||
var pending []string
|
||||
hostOf := map[string]string{}
|
||||
for _, a := range order {
|
||||
h := sshGet(ctx, a)["hostname"]
|
||||
if strings.EqualFold(h, addr) {
|
||||
return a
|
||||
}
|
||||
if h != "" {
|
||||
hostOf[a] = h
|
||||
pending = append(pending, a)
|
||||
}
|
||||
}
|
||||
if len(pending) == 0 {
|
||||
return ""
|
||||
}
|
||||
lctx, cancel := context.WithTimeout(ctx, 3*time.Second)
|
||||
defer cancel()
|
||||
want := lookup(lctx, addr)
|
||||
if len(want) == 0 {
|
||||
return ""
|
||||
}
|
||||
for _, a := range pending {
|
||||
for ip := range lookup(lctx, hostOf[a]) {
|
||||
if want[ip] {
|
||||
return a
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// configUser is the user the ssh config logs in to host as, empty when it
|
||||
// names none. ssh -G reports the local account when nothing set one, so
|
||||
// that is taken as none: the config saying `User root` to a root caller is
|
||||
// the one case read wrongly, and it gets the fallback user instead.
|
||||
func configUser(ctx context.Context, host string) string {
|
||||
u := sshGet(ctx, host)["user"]
|
||||
if cur, err := user.Current(); err == nil && u == cur.Username {
|
||||
return ""
|
||||
}
|
||||
return u
|
||||
}
|
||||
|
||||
// sshGet is ssh's own reading of its configuration for host, keys
|
||||
// lowercased; empty when ssh cannot say.
|
||||
func sshGet(ctx context.Context, host string) map[string]string {
|
||||
args := append(append([]string{"-G"}, sshExtraArgs...), "--", host)
|
||||
out, err := exec.CommandContext(ctx, "ssh", args...).Output()
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
m := map[string]string{}
|
||||
sc := bufio.NewScanner(bytes.NewReader(out))
|
||||
for sc.Scan() {
|
||||
k, v, ok := strings.Cut(sc.Text(), " ")
|
||||
if k = strings.ToLower(k); ok {
|
||||
if _, dup := m[k]; !dup {
|
||||
m[k] = v
|
||||
}
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func lookup(ctx context.Context, host string) map[string]bool {
|
||||
if ip := net.ParseIP(host); ip != nil {
|
||||
return map[string]bool{ip.String(): true}
|
||||
}
|
||||
addrs, err := net.DefaultResolver.LookupHost(ctx, host)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
m := map[string]bool{}
|
||||
for _, a := range addrs {
|
||||
if ip := net.ParseIP(a); ip != nil {
|
||||
m[ip.String()] = true
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// hostNames are the concrete names on the Host lines of an ssh config file
|
||||
// and the files it includes — no patterns, no negations. base is where a
|
||||
// relative Include is looked for, as ssh does: ~/.ssh for the user's file,
|
||||
// /etc/ssh for the system's.
|
||||
func hostNames(path, base string, seen map[string]bool, depth int) []string {
|
||||
if depth > 16 || seen[path] {
|
||||
return nil
|
||||
}
|
||||
seen[path] = true
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var names []string
|
||||
for _, line := range strings.Split(string(data), "\n") {
|
||||
key, args := configLine(line)
|
||||
switch key {
|
||||
case "host":
|
||||
for _, a := range args {
|
||||
if a != "" && !strings.ContainsAny(a, "*?!") {
|
||||
names = append(names, a)
|
||||
}
|
||||
}
|
||||
case "include":
|
||||
for _, a := range args {
|
||||
if strings.HasPrefix(a, "~/") {
|
||||
a = filepath.Join(sshHome(), a[2:])
|
||||
} else if !filepath.IsAbs(a) {
|
||||
a = filepath.Join(base, a)
|
||||
}
|
||||
matches, _ := filepath.Glob(a)
|
||||
for _, m := range matches {
|
||||
names = append(names, hostNames(m, base, seen, depth+1)...)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return names
|
||||
}
|
||||
|
||||
// configLine splits one line of ssh config into its keyword, lowercased,
|
||||
// and arguments: "Host a b", "Host=a b" and quoted arguments alike.
|
||||
func configLine(line string) (string, []string) {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
return "", nil
|
||||
}
|
||||
i := strings.IndexAny(line, " \t=")
|
||||
if i < 0 {
|
||||
return strings.ToLower(line), nil
|
||||
}
|
||||
key := strings.ToLower(line[:i])
|
||||
rest := strings.TrimLeft(line[i:], " \t")
|
||||
rest = strings.TrimPrefix(rest, "=")
|
||||
var args []string
|
||||
for rest = strings.TrimSpace(rest); rest != ""; rest = strings.TrimSpace(rest) {
|
||||
if rest[0] == '"' {
|
||||
end := strings.IndexByte(rest[1:], '"')
|
||||
if end < 0 {
|
||||
args = append(args, rest[1:])
|
||||
break
|
||||
}
|
||||
args = append(args, rest[1:1+end])
|
||||
rest = rest[2+end:]
|
||||
continue
|
||||
}
|
||||
j := strings.IndexAny(rest, " \t")
|
||||
if j < 0 {
|
||||
args = append(args, rest)
|
||||
break
|
||||
}
|
||||
args = append(args, rest[:j])
|
||||
rest = rest[j:]
|
||||
}
|
||||
return key, args
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package onboard
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// withSSHConfig points configAlias and ssh -G at one file, and nothing else.
|
||||
func withSSHConfig(t *testing.T, config string, extra map[string]string) {
|
||||
t.Helper()
|
||||
if _, err := exec.LookPath("ssh"); err != nil {
|
||||
t.Skip("no ssh")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "config")
|
||||
if err := os.WriteFile(path, []byte(config), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for name, body := range extra {
|
||||
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
oldUser, oldSys, oldArgs := userSSHConfig, systemSSHConfig, sshExtraArgs
|
||||
userSSHConfig = func() string { return path }
|
||||
systemSSHConfig = ""
|
||||
sshExtraArgs = []string{"-F", path}
|
||||
t.Cleanup(func() { userSSHConfig, systemSSHConfig, sshExtraArgs = oldUser, oldSys, oldArgs })
|
||||
}
|
||||
|
||||
// A machine given by address is reached through the Host block whose
|
||||
// HostName it is, with that block's user — not the sudo caller's.
|
||||
func TestSSHTargetUsesConfigAlias(t *testing.T) {
|
||||
withSSHConfig(t, `
|
||||
Host *.example
|
||||
User nobody-wild
|
||||
Host gpu01 gpu01-alt
|
||||
HostName 10.9.8.7
|
||||
User ops
|
||||
Host gpu02
|
||||
HostName=10.9.8.8
|
||||
`, nil)
|
||||
ctx := context.Background()
|
||||
cases := []struct {
|
||||
s *SSH
|
||||
want string
|
||||
}{
|
||||
// The config's User applies by ssh itself: no fallback is added.
|
||||
{&SSH{Addr: "10.9.8.7", FallbackUser: "ilker"}, "gpu01"},
|
||||
{&SSH{Addr: "10.9.8.7", User: "root", FallbackUser: "ilker"}, "root@gpu01"},
|
||||
{&SSH{Addr: "10.9.8.8", FallbackUser: "ilker"}, "ilker@gpu02"},
|
||||
{&SSH{Addr: "gpu01", FallbackUser: "ilker"}, "gpu01"},
|
||||
{&SSH{Addr: "10.9.8.9", FallbackUser: "ilker"}, "ilker@10.9.8.9"},
|
||||
{&SSH{Addr: "10.9.8.9"}, "10.9.8.9"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := c.s.Target(ctx); got != c.want {
|
||||
t.Errorf("%+v: target %q, want %q", c.s.Addr, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A block in an included file counts, as it does for ssh.
|
||||
func TestSSHTargetFollowsInclude(t *testing.T) {
|
||||
withSSHConfig(t, "Include nodes.conf\n", map[string]string{
|
||||
"nodes.conf": "Host \"gpu03\"\n HostName 10.9.8.10\n",
|
||||
})
|
||||
// ssh resolves a relative Include against ~/.ssh, not -F's directory;
|
||||
// the scan does the same, so point base at the test's directory.
|
||||
names := hostNames(userSSHConfig(), filepath.Dir(userSSHConfig()), map[string]bool{}, 0)
|
||||
if len(names) != 1 || names[0] != "gpu03" {
|
||||
t.Fatalf("names %q", names)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigUserIgnoresLocalDefault(t *testing.T) {
|
||||
withSSHConfig(t, "Host x\n HostName 10.9.8.11\n", nil)
|
||||
if cur, err := user.Current(); err == nil {
|
||||
if u := configUser(context.Background(), "x"); u != "" {
|
||||
t.Errorf("config user %q for a block naming none (local %q)", u, cur.Username)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -253,6 +253,9 @@ func (r *Router) Handler() http.Handler {
|
||||
pages.SetPersonalDefaults(r.personalDefaults)
|
||||
if r.store != nil {
|
||||
pages.SetAudit(r.store)
|
||||
if r.slurmSettings != nil {
|
||||
pages.SetSlurm(r.store, r.slurmSettings)
|
||||
}
|
||||
// A forgotten password: a link mailed when the cluster's
|
||||
// settings say how, otherwise asked of an operator.
|
||||
pages.SetPasswordReset(r.store, r.mail, r.mail.BaseURL)
|
||||
|
||||
@@ -208,6 +208,9 @@ type Router struct {
|
||||
// settingsCheck and settingsLocal mount the portal's settings page.
|
||||
settingsCheck func([]byte) error
|
||||
settingsLocal []string
|
||||
// slurmSettings is the cluster's Slurm settings, for the portal; nil
|
||||
// offers nothing of Slurm.
|
||||
slurmSettings func() config.SlurmSettings
|
||||
}
|
||||
|
||||
// SetClusterView gives the router the data the /admin page renders.
|
||||
@@ -238,6 +241,10 @@ func (r *Router) SetSettingsCheck(check func([]byte) error, local []string) {
|
||||
r.settingsCheck, r.settingsLocal = check, local
|
||||
}
|
||||
|
||||
// SetSlurm has the portal offer the Slurm cluster's login and SSH keys as
|
||||
// settings say.
|
||||
func (r *Router) SetSlurm(s func() config.SlurmSettings) { r.slurmSettings = s }
|
||||
|
||||
// SetStopped refuses, or again serves, every request for a model.
|
||||
func (r *Router) SetStopped(stopped bool) { r.stopped.Store(stopped) }
|
||||
|
||||
|
||||
@@ -39,7 +39,7 @@ type Store interface {
|
||||
SetUserState(ctx context.Context, id string, state store.UserState) error
|
||||
DeleteUserSessions(ctx context.Context, userID string) error
|
||||
SetStaffRole(ctx context.Context, id, role string) error
|
||||
AddMembership(ctx context.Context, user string, scope store.Scope, id, role, by string, at time.Time) error
|
||||
EnsureMembership(ctx context.Context, user string, scope store.Scope, id, by string, at time.Time) error
|
||||
RemoveMembership(ctx context.Context, user string, scope store.Scope, id string, at time.Time) (int, error)
|
||||
AllOrgs(ctx context.Context) ([]store.Org, error)
|
||||
AllTeams(ctx context.Context) ([]store.TeamPath, error)
|
||||
@@ -548,7 +548,7 @@ func (s *Server) member(ctx context.Context, group, userID string, joined bool)
|
||||
return
|
||||
}
|
||||
if joined {
|
||||
err = s.st.AddMembership(ctx, u.ID, scope, id, store.RoleMember, "scim", time.Now())
|
||||
err = s.st.EnsureMembership(ctx, u.ID, scope, id, "scim", time.Now())
|
||||
} else {
|
||||
_, err = s.st.RemoveMembership(ctx, u.ID, scope, id, time.Now())
|
||||
}
|
||||
|
||||
@@ -58,7 +58,7 @@ func (m *mem) DeleteUserSessions(_ context.Context, id string) error {
|
||||
return nil
|
||||
}
|
||||
func (m *mem) SetStaffRole(_ context.Context, id, role string) error { m.roles[id] = role; return nil }
|
||||
func (m *mem) AddMembership(_ context.Context, u string, sc store.Scope, id, _, _ string, _ time.Time) error {
|
||||
func (m *mem) EnsureMembership(_ context.Context, u string, sc store.Scope, id, _ string, _ time.Time) error {
|
||||
m.members[u+"@"+string(sc)+":"+id] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
package slurm
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Jobs in containers: pyxis, Slurm's plugin that gives srun and sbatch
|
||||
// --container-image, and enroot under it, which turns an image into a
|
||||
// directory a job's user runs in without a daemon or root. Both are
|
||||
// NVIDIA's and in no distribution's archive, so each is fetched from its
|
||||
// release at a pinned version and checked against the digest here.
|
||||
const (
|
||||
EnrootVersion = "4.2.1"
|
||||
PyxisVersion = "0.24.0"
|
||||
|
||||
// PyxisSource is pyxis's source, built on each machine against its own
|
||||
// Slurm: a plugin built for another Slurm is refused by it.
|
||||
PyxisSource = "https://github.com/NVIDIA/pyxis/archive/refs/tags/v" + PyxisVersion + ".tar.gz"
|
||||
PyxisSHA256 = "9c4cdb79a67301d8ea05951aa4d2c205f40cf6145e338214b0191add8b36d6c4"
|
||||
|
||||
// PyxisPlugin is where it is installed (pyxis's own make install).
|
||||
PyxisPlugin = "/usr/local/lib/slurm/spank_pyxis.so"
|
||||
|
||||
// PlugstackConf has Slurm load pyxis. Written on the controller, from
|
||||
// which configless nodes fetch it, only once every machine has the
|
||||
// plugin: "required" fails every job on a machine without it, and
|
||||
// "optional" would run a job asking for a container on the host.
|
||||
PlugstackConf = "# Written by llmgw (BACKLOG P): jobs in containers, srun --container-image.\n" +
|
||||
"required " + PyxisPlugin + "\n"
|
||||
|
||||
// EnrootConfPath is enroot's settings.
|
||||
EnrootConfPath = "/etc/enroot/enroot.conf"
|
||||
// EnrootMounts has every container see the shared path.
|
||||
EnrootMounts = "/etc/enroot/mounts.d/50-llmgw-shared.fstab"
|
||||
// EnrootTmpfiles makes enroot's runtime directory at each boot.
|
||||
EnrootTmpfiles = "/etc/tmpfiles.d/llmgw-enroot.conf"
|
||||
// EnrootRun is where containers are unpacked while they run; EnrootLib
|
||||
// holds each user's image cache and named containers.
|
||||
EnrootRun = "/run/enroot"
|
||||
EnrootLib = "/var/lib/enroot"
|
||||
)
|
||||
|
||||
// EnrootPackage is one of enroot's release files.
|
||||
type EnrootPackage struct {
|
||||
Name, SHA256 string
|
||||
}
|
||||
|
||||
// URL is where the file is.
|
||||
func (p EnrootPackage) URL() string {
|
||||
return "https://github.com/NVIDIA/enroot/releases/download/v" + EnrootVersion + "/" +
|
||||
strings.ReplaceAll(p.Name, "+", "%2B")
|
||||
}
|
||||
|
||||
// EnrootPackages are the files for a family and architecture (Go's
|
||||
// names): enroot, and enroot+caps, which lets its helpers unpack images
|
||||
// for a user without root.
|
||||
func EnrootPackages(family, arch string) ([]EnrootPackage, error) {
|
||||
ps, ok := enrootPackages[family+"/"+arch]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("enroot %s has no package for %s on %s", EnrootVersion, family, arch)
|
||||
}
|
||||
return ps, nil
|
||||
}
|
||||
|
||||
var enrootPackages = map[string][]EnrootPackage{
|
||||
"deb/amd64": {
|
||||
{"enroot_4.2.1-1_amd64.deb", "5955c51e88df0c5ed538d87abb5264609d4bad55fa8570745ed5d4ec66c8ab91"},
|
||||
{"enroot+caps_4.2.1-1_amd64.deb", "ca22652dcbdd3ebb6e2c68df781ef4301647c79dd7a41ed62899c2fd0f692750"},
|
||||
},
|
||||
"deb/arm64": {
|
||||
{"enroot_4.2.1-1_arm64.deb", "267018b335815a6b32a562915de1f3fad579f7c9d6748e98c62e7369b40be421"},
|
||||
{"enroot+caps_4.2.1-1_arm64.deb", "12b811d0c95e879e42e4ac762fa85869a5f3cc9dbb06ab483c01e12e911284a1"},
|
||||
},
|
||||
"rpm/amd64": {
|
||||
{"enroot-4.2.1-1.el8.x86_64.rpm", "6749195e37e3a98071ec349884900b12de7677332599da0fd5d8afb5bed9233c"},
|
||||
{"enroot+caps-4.2.1-1.el8.x86_64.rpm", "56321a03a1f6c4e4273bf664917eb9a174714499d5506c498db4b64275a2b9c0"},
|
||||
},
|
||||
"rpm/arm64": {
|
||||
{"enroot-4.2.1-1.el8.aarch64.rpm", "6b6bc2526f0f0daa7d3cebe6dd33c0468b209e81eebd92def5777520042ec6fe"},
|
||||
{"enroot+caps-4.2.1-1.el8.aarch64.rpm", "38cf9ae9602a9a8c3e94637bc1b34e736e8176bda656ba71d371f5e1d1e1eed9"},
|
||||
},
|
||||
}
|
||||
|
||||
// PyxisBuildPackages build pyxis: a compiler and Slurm's headers.
|
||||
var PyxisBuildPackages = map[string][]string{
|
||||
"deb": {"gcc", "make", "libslurm-dev"},
|
||||
"rpm": {"gcc", "make", "slurm-devel"},
|
||||
}
|
||||
|
||||
// EnrootConf is enroot's settings: each user's containers, cache and
|
||||
// data apart, in directories that need no login session (a job has none,
|
||||
// and llmgw's users have no pam_systemd on a node).
|
||||
const EnrootConf = "# Written by llmgw (BACKLOG P): enroot, for Slurm's jobs in containers.\n" +
|
||||
"ENROOT_RUNTIME_PATH " + EnrootRun + "/user-$(id -u)\n" +
|
||||
"ENROOT_CACHE_PATH " + EnrootLib + "/cache/user-$(id -u)\n" +
|
||||
"ENROOT_DATA_PATH " + EnrootLib + "/data/user-$(id -u)\n" +
|
||||
"ENROOT_TEMP_PATH ${TMPDIR:-/tmp}\n" +
|
||||
// A container's root is the user's, as on the host: no fake root.
|
||||
"ENROOT_REMAP_ROOT no\n" +
|
||||
"ENROOT_ROOTFS_WRITABLE yes\n"
|
||||
|
||||
// EnrootTmpfilesConf makes EnrootRun, open to every user as /tmp is.
|
||||
const EnrootTmpfilesConf = "# Written by llmgw (BACKLOG P): enroot's runtime directory.\n" +
|
||||
"d " + EnrootRun + " 1777 root root -\n"
|
||||
|
||||
// EnrootSharedMount has a container see path where the host does.
|
||||
func EnrootSharedMount(path string) string {
|
||||
return "# Written by llmgw (BACKLOG P): the Slurm cluster's shared files, in every container.\n" +
|
||||
path + " " + path + " none x-create=dir,rbind,rw,nosuid\n"
|
||||
}
|
||||
|
||||
// The NVIDIA container library, for enroot's hook that gives a container
|
||||
// the job's GPUs: from NVIDIA's own repository, its key checked.
|
||||
const (
|
||||
NvidiaContainerKeyURL = "https://nvidia.github.io/libnvidia-container/gpgkey"
|
||||
NvidiaContainerKeySHA256 = "c880576d6cf75a48e5027a871bac70fd0421ab07d2b55f30877b21f1c87959c9"
|
||||
NvidiaContainerKeyPath = "/etc/apt/keyrings/llmgw-nvidia-container.asc"
|
||||
NvidiaContainerAptList = "/etc/apt/sources.list.d/llmgw-nvidia-container.list"
|
||||
NvidiaContainerYumRepo = "/etc/yum.repos.d/llmgw-nvidia-container.repo"
|
||||
)
|
||||
|
||||
// NvidiaContainerApt is the apt source.
|
||||
const NvidiaContainerApt = "# Written by llmgw (BACKLOG P): nvidia-container-cli, for GPUs in Slurm's containers.\n" +
|
||||
"deb [signed-by=" + NvidiaContainerKeyPath + "] https://nvidia.github.io/libnvidia-container/stable/deb/$(ARCH) /\n"
|
||||
|
||||
// NvidiaContainerYum is the yum/dnf repository.
|
||||
const NvidiaContainerYum = "# Written by llmgw (BACKLOG P): nvidia-container-cli, for GPUs in Slurm's containers.\n" +
|
||||
"[llmgw-nvidia-container]\nname=NVIDIA container library\n" +
|
||||
"baseurl=https://nvidia.github.io/libnvidia-container/stable/rpm/$basearch\n" +
|
||||
"repo_gpgcheck=1\ngpgcheck=0\nenabled=1\ngpgkey=" + NvidiaContainerKeyURL + "\n"
|
||||
|
||||
// NvidiaContainerPackages has nvidia-container-cli.
|
||||
var NvidiaContainerPackages = map[string][]string{"deb": {"libnvidia-container-tools"}, "rpm": {"libnvidia-container-tools"}}
|
||||
|
||||
// ContainersStatus is what a machine says it has.
|
||||
func ContainersStatus() string {
|
||||
return "pyxis " + PyxisVersion + ", enroot " + EnrootVersion
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
package slurm
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// Accounting: slurmdbd on the controller, keeping the jobs' records in a
|
||||
// MariaDB beside it that answers this machine alone. slurmdbd takes MySQL or
|
||||
// MariaDB only, so it is not the cluster's PostgreSQL. Since 23.11 Slurm
|
||||
// cannot run without it well: its main scheduler passes every job by as
|
||||
// InvalidAccount, and backfill starts it a pass later.
|
||||
const (
|
||||
// DbdPort is slurmdbd's, which slurmctld and sacct reach.
|
||||
DbdPort = 6819
|
||||
// DbdConfPath is slurmdbd's file, readable by the slurm user alone: it
|
||||
// holds the database's password.
|
||||
DbdConfPath = ConfDir + "/slurmdbd.conf"
|
||||
// AcctDB and AcctUser are the database and its user.
|
||||
AcctDB = "slurm_acct_db"
|
||||
AcctUser = "slurm"
|
||||
)
|
||||
|
||||
// DbdPassword is the database user's password, made from the munge key:
|
||||
// every controller the cluster has gets the same one, and nothing new is
|
||||
// kept anywhere.
|
||||
func DbdPassword(mungeKey []byte) string {
|
||||
h := sha256.New()
|
||||
h.Write([]byte("llmgw slurmdbd password\x00"))
|
||||
h.Write(mungeKey)
|
||||
return hex.EncodeToString(h.Sum(nil))[:40]
|
||||
}
|
||||
|
||||
// DbdConf is slurmdbd.conf for the controller host, named as `hostname -s`
|
||||
// says, which slurmdbd checks it runs on.
|
||||
func DbdConf(host, password string) string {
|
||||
return header + fmt.Sprintf(`AuthType=auth/munge
|
||||
DbdHost=%s
|
||||
DbdPort=%d
|
||||
SlurmUser=slurm
|
||||
LogFile=%s/slurmdbd.log
|
||||
PidFile=/run/slurmdbd.pid
|
||||
StorageType=accounting_storage/mysql
|
||||
StorageHost=localhost
|
||||
StorageUser=%s
|
||||
StoragePass=%s
|
||||
StorageLoc=%s
|
||||
# Kept a year; older records are purged, a month at a time.
|
||||
PurgeEventAfter=12months
|
||||
PurgeJobAfter=12months
|
||||
PurgeResvAfter=12months
|
||||
PurgeStepAfter=12months
|
||||
PurgeSuspendAfter=12months
|
||||
PurgeTXNAfter=12months
|
||||
PurgeUsageAfter=24months
|
||||
`, host, DbdPort, LogDir, AcctUser, password, AcctDB)
|
||||
}
|
||||
|
||||
// MariaDBConf is the database server's settings: this machine alone, and
|
||||
// what Slurm's documentation asks of InnoDB.
|
||||
const MariaDBConf = `# Written by llmgw (BACKLOG P): the database of Slurm's accounting
|
||||
# (slurmdbd), on this machine alone.
|
||||
[mysqld]
|
||||
bind-address=127.0.0.1
|
||||
skip-networking=0
|
||||
innodb_buffer_pool_size=256M
|
||||
innodb_lock_wait_timeout=900
|
||||
`
|
||||
|
||||
// AcctSQL makes the database and its user, and gives the user the password
|
||||
// again: run on every start of the controller's part, it changes nothing
|
||||
// that is already so.
|
||||
func AcctSQL(password string) string {
|
||||
return fmt.Sprintf(`CREATE DATABASE IF NOT EXISTS %[1]s;
|
||||
CREATE USER IF NOT EXISTS '%[2]s'@'localhost' IDENTIFIED BY '%[3]s';
|
||||
ALTER USER '%[2]s'@'localhost' IDENTIFIED BY '%[3]s';
|
||||
GRANT ALL ON %[1]s.* TO '%[2]s'@'localhost';
|
||||
FLUSH PRIVILEGES;
|
||||
`, AcctDB, AcctUser, password)
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
package slurm
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"path"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Logging in to a node: llmgw's users reach the controller, the machine
|
||||
// they submit from, whenever they like, and a node only while a job of
|
||||
// theirs runs there, their session put in that job (pam_slurm_adopt).
|
||||
// Without it a user could ssh to a node and use its cards outside Slurm,
|
||||
// on a node llmgw serves models from too.
|
||||
const (
|
||||
// PAMSSHD is sshd's PAM stack, where the lines go.
|
||||
PAMSSHD = "/etc/pam.d/sshd"
|
||||
// AdoptModule is the module's file name.
|
||||
AdoptModule = "pam_slurm_adopt.so"
|
||||
|
||||
pamBegin = "# llmgw (BACKLOG P): llmgw's Slurm users reach this node only while a job of theirs runs here."
|
||||
pamSession = "# llmgw (BACKLOG P): pam_systemd skipped for llmgw's Slurm users, so their session stays in the job pam_slurm_adopt put it in."
|
||||
pamWas = "# llmgw, in place of: "
|
||||
pamEnd = "# llmgw end"
|
||||
)
|
||||
|
||||
// AdoptPackages are the packages with AdoptModule, by family.
|
||||
var AdoptPackages = map[string][]string{"deb": {"libpam-slurm-adopt"}, "rpm": {"slurm-pam_slurm"}}
|
||||
|
||||
// ModuleDirs are where PAM's modules are, on Debian's and Red Hat's.
|
||||
var ModuleDirs = []string{
|
||||
"/usr/lib/x86_64-linux-gnu/security", "/lib/x86_64-linux-gnu/security",
|
||||
"/usr/lib/aarch64-linux-gnu/security", "/lib/aarch64-linux-gnu/security",
|
||||
"/usr/lib64/security", "/lib64/security", "/usr/lib/security", "/lib/security",
|
||||
}
|
||||
|
||||
// PAMAdopt is sshd's PAM stack with llmgw's lines in it (on) or taken out
|
||||
// and the stack as it was. Only the users listed in the file list, one a
|
||||
// line, are held to their jobs; anyone else, root and the site's
|
||||
// administrators, logs in as before.
|
||||
//
|
||||
// The account lines go after the last account line, so a user the
|
||||
// machine's own rules refuse is still refused. pam_systemd, which would
|
||||
// move the session out of the job into a scope of its own, is skipped for
|
||||
// the listed users: the file sshd includes it from (read through read) is
|
||||
// put in sshd's stack in its place, with the one line in front of
|
||||
// pam_systemd. contained is whether that was done; when it cannot be done
|
||||
// safely, the users are still held to their jobs, their sessions not to
|
||||
// the job's cards and cores.
|
||||
func PAMAdopt(stack, list string, read func(name string) (string, error), on bool) (out string, contained bool, err error) {
|
||||
lines, err := pamStrip(stack)
|
||||
if err != nil {
|
||||
return "", false, err
|
||||
}
|
||||
if !on {
|
||||
return join(lines), false, nil
|
||||
}
|
||||
last := -1
|
||||
for i, l := range lines {
|
||||
f := strings.Fields(l)
|
||||
if len(f) > 0 && (f[0] == "account" || (f[0] == "@include" && len(f) > 1 && strings.Contains(f[1], "account"))) {
|
||||
last = i
|
||||
}
|
||||
}
|
||||
if last < 0 {
|
||||
return "", false, errors.New(PAMSSHD + " has no account lines to put llmgw's after")
|
||||
}
|
||||
block := []string{
|
||||
pamBegin,
|
||||
// Listed: on to the next line; not listed, or no list: past it.
|
||||
"account [success=ignore default=1] pam_listfile.so item=user sense=allow file=" + list + " onerr=fail",
|
||||
"account required " + AdoptModule,
|
||||
pamEnd,
|
||||
}
|
||||
lines = append(lines[:last+1:last+1], append(block, lines[last+1:]...)...)
|
||||
lines, contained = pamSkipSystemd(lines, list, read)
|
||||
return join(lines), contained, nil
|
||||
}
|
||||
|
||||
func join(lines []string) string { return strings.Join(lines, "\n") + "\n" }
|
||||
|
||||
// pamStrip takes llmgw's blocks out, putting back the lines they stand in
|
||||
// place of.
|
||||
func pamStrip(stack string) ([]string, error) {
|
||||
var lines []string
|
||||
in := false
|
||||
for _, l := range strings.Split(strings.TrimSuffix(stack, "\n"), "\n") {
|
||||
switch {
|
||||
case !in && (l == pamBegin || l == pamSession):
|
||||
in = true
|
||||
case in && l == pamEnd:
|
||||
in = false
|
||||
case in && strings.HasPrefix(l, pamWas):
|
||||
lines = append(lines, strings.TrimPrefix(l, pamWas))
|
||||
case !in:
|
||||
lines = append(lines, l)
|
||||
}
|
||||
}
|
||||
if in {
|
||||
return nil, errors.New("llmgw's lines in " + PAMSSHD + " have no end; put them right by hand")
|
||||
}
|
||||
return lines, nil
|
||||
}
|
||||
|
||||
// pamLine is a PAM line's type, control and module, the control with its
|
||||
// brackets.
|
||||
var pamLine = regexp.MustCompile(`^\s*(-?[a-z]+)\s+(\[[^\]]*\]|\S+)\s+(\S+)`)
|
||||
|
||||
// pamJump is a jump of N in a bracketed control.
|
||||
var pamJump = regexp.MustCompile(`=(\d+)`)
|
||||
|
||||
func isSession(l string) bool {
|
||||
m := pamLine.FindStringSubmatch(l)
|
||||
return m != nil && strings.TrimPrefix(m[1], "-") == "session"
|
||||
}
|
||||
|
||||
// pamSkipSystemd puts the guard in front of pam_systemd, in the session
|
||||
// file sshd includes it from or in sshd's own lines; false, and lines as
|
||||
// they were, when it is in neither or cannot be done without changing
|
||||
// what any other jump in the stack lands on.
|
||||
func pamSkipSystemd(lines []string, list string, read func(string) (string, error)) ([]string, bool) {
|
||||
guard := "session [success=1 default=ignore] pam_listfile.so item=user sense=allow file=" + list + " onerr=fail"
|
||||
hasSystemd := func(l string) bool {
|
||||
m := pamLine.FindStringSubmatch(l)
|
||||
return m != nil && strings.TrimPrefix(m[1], "-") == "session" && path.Base(m[3]) == "pam_systemd.so"
|
||||
}
|
||||
for i, l := range lines {
|
||||
f := strings.Fields(l)
|
||||
var body []string
|
||||
switch {
|
||||
case hasSystemd(l):
|
||||
body = []string{l}
|
||||
case len(f) == 2 && f[0] == "@include" && strings.Contains(f[1], "session"),
|
||||
len(f) == 3 && f[0] == "session" && f[1] == "include":
|
||||
name := f[len(f)-1]
|
||||
if strings.Contains(name, "/") {
|
||||
continue
|
||||
}
|
||||
text, err := read(name)
|
||||
if err != nil {
|
||||
return lines, false
|
||||
}
|
||||
for _, il := range strings.Split(text, "\n") {
|
||||
t := strings.TrimSpace(il)
|
||||
if t == "" || strings.HasPrefix(t, "#") {
|
||||
continue
|
||||
}
|
||||
if f[0] == "session" && !isSession(t) {
|
||||
continue // include takes only the lines of its own type
|
||||
}
|
||||
body = append(body, t)
|
||||
}
|
||||
default:
|
||||
continue
|
||||
}
|
||||
at := -1
|
||||
for j, b := range body {
|
||||
if hasSystemd(b) {
|
||||
at = j
|
||||
}
|
||||
if g := strings.Fields(b); len(g) > 0 && (g[0] == "@include" || strings.Contains(b, "include") || strings.Contains(b, "substack")) {
|
||||
return lines, false // nested: not followed
|
||||
}
|
||||
}
|
||||
if at < 0 {
|
||||
continue
|
||||
}
|
||||
// A jump in sshd's own session lines may cross what is put in.
|
||||
for j, o := range lines {
|
||||
if j != i && isSession(o) && pamJump.MatchString(pamControl(o)) {
|
||||
return lines, false
|
||||
}
|
||||
}
|
||||
body, ok := pamInsert(body, at, guard)
|
||||
if !ok {
|
||||
return lines, false
|
||||
}
|
||||
block := append([]string{pamSession, pamWas + l}, body...)
|
||||
block = append(block, pamEnd)
|
||||
return append(lines[:i:i], append(block, lines[i+1:]...)...), true
|
||||
}
|
||||
return lines, false
|
||||
}
|
||||
|
||||
func pamControl(l string) string {
|
||||
if m := pamLine.FindStringSubmatch(l); m != nil {
|
||||
return m[2]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// pamInsert puts guard in front of body[at], a jump that crosses it made
|
||||
// one longer so it lands where it did.
|
||||
func pamInsert(body []string, at int, guard string) ([]string, bool) {
|
||||
// Each session line's place among the session lines.
|
||||
idx := make([]int, len(body))
|
||||
n := 0
|
||||
for j, b := range body {
|
||||
idx[j] = n
|
||||
if isSession(b) {
|
||||
n++
|
||||
}
|
||||
}
|
||||
out := make([]string, 0, len(body)+1)
|
||||
for j, b := range body {
|
||||
if j == at {
|
||||
out = append(out, guard)
|
||||
}
|
||||
ctl := pamControl(b)
|
||||
if isSession(b) && j < at && strings.HasPrefix(ctl, "[") {
|
||||
from, target := idx[j], idx[at]
|
||||
fixed := pamJump.ReplaceAllStringFunc(ctl, func(s string) string {
|
||||
k, err := strconv.Atoi(s[1:])
|
||||
if err != nil {
|
||||
return s
|
||||
}
|
||||
if from < target && target <= from+k {
|
||||
k++
|
||||
}
|
||||
return fmt.Sprintf("=%d", k)
|
||||
})
|
||||
b = strings.Replace(b, ctl, fixed, 1)
|
||||
}
|
||||
out = append(out, b)
|
||||
}
|
||||
return out, true
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
package slurm
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const ubuntuSSHD = `# PAM configuration for the Secure Shell service
|
||||
@include common-auth
|
||||
account required pam_nologin.so
|
||||
@include common-account
|
||||
session [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so close
|
||||
session required pam_loginuid.so
|
||||
@include common-session
|
||||
session required pam_limits.so
|
||||
@include common-password
|
||||
`
|
||||
|
||||
const ubuntuSession = `# here are the per-package modules (the "Primary" block)
|
||||
session [default=1] pam_permit.so
|
||||
session requisite pam_deny.so
|
||||
session required pam_permit.so
|
||||
session optional pam_umask.so
|
||||
session required pam_unix.so
|
||||
session optional pam_systemd.so
|
||||
# end of pam-auth-update config
|
||||
`
|
||||
|
||||
const redhatSSHD = `#%PAM-1.0
|
||||
auth substack password-auth
|
||||
account required pam_sepermit.so
|
||||
account required pam_nologin.so
|
||||
account include password-auth
|
||||
password include password-auth
|
||||
session required pam_loginuid.so
|
||||
session include password-auth
|
||||
session include postlogin
|
||||
`
|
||||
|
||||
const redhatPasswordAuth = `auth required pam_env.so
|
||||
account required pam_unix.so
|
||||
-session optional pam_keyinit.so revoke
|
||||
session required pam_limits.so
|
||||
-session optional pam_systemd.so
|
||||
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
|
||||
session required pam_unix.so
|
||||
`
|
||||
|
||||
func files(m map[string]string) func(string) (string, error) {
|
||||
return func(n string) (string, error) {
|
||||
if s, ok := m[n]; ok {
|
||||
return s, nil
|
||||
}
|
||||
return "", errors.New("no such file")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPAMAdopt(t *testing.T) {
|
||||
read := files(map[string]string{"common-session": ubuntuSession, "password-auth": redhatPasswordAuth, "postlogin": "session optional pam_lastlog.so\n"})
|
||||
for name, stack := range map[string]string{"ubuntu": ubuntuSSHD, "redhat": redhatSSHD} {
|
||||
on, contained, err := PAMAdopt(stack, "/var/lib/x-users", read, true)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
if !contained {
|
||||
t.Errorf("%s: pam_systemd not skipped:\n%s", name, on)
|
||||
}
|
||||
lines := strings.Split(on, "\n")
|
||||
if !strings.Contains(on, "account [success=ignore default=1] pam_listfile.so item=user sense=allow file=/var/lib/x-users onerr=fail") {
|
||||
t.Fatalf("%s: lines not put in:\n%s", name, on)
|
||||
}
|
||||
for j, l := range lines {
|
||||
if l == pamBegin && !strings.Contains(lines[j-1], "account") {
|
||||
t.Errorf("%s: put after %q", name, lines[j-1])
|
||||
}
|
||||
if strings.Contains(l, "pam_systemd.so") &&
|
||||
lines[j-1] != "session [success=1 default=ignore] pam_listfile.so item=user sense=allow file=/var/lib/x-users onerr=fail" {
|
||||
t.Errorf("%s: pam_systemd not guarded, after %q", name, lines[j-1])
|
||||
}
|
||||
}
|
||||
again, _, _ := PAMAdopt(on, "/var/lib/x-users", read, true)
|
||||
if again != on {
|
||||
t.Errorf("%s: put in twice:\n%s", name, again)
|
||||
}
|
||||
off, _, err := PAMAdopt(on, "", read, false)
|
||||
if err != nil || off != stack {
|
||||
t.Errorf("%s: taken out, not as it was (%v):\n%s", name, err, off)
|
||||
}
|
||||
}
|
||||
// The jump over pam_deny lands where it did; the one past crond's line too.
|
||||
on, _, _ := PAMAdopt(ubuntuSSHD, "/l", read, true)
|
||||
if !strings.Contains(on, "[default=1]\t\t\tpam_permit.so") {
|
||||
t.Errorf("a jump that does not cross the guard was changed:\n%s", on)
|
||||
}
|
||||
body := []string{"session [default=2] pam_permit.so", "session required pam_unix.so", "session optional pam_systemd.so", "session required pam_x.so"}
|
||||
got, _ := pamInsert(body, 2, "G")
|
||||
if got[0] != "session [default=3] pam_permit.so" || got[2] != "G" {
|
||||
t.Errorf("a jump across the guard not made longer: %q", got)
|
||||
}
|
||||
|
||||
if _, _, err := PAMAdopt("auth required pam_unix.so\n", "/x", read, true); err == nil {
|
||||
t.Error("a stack with no account lines taken")
|
||||
}
|
||||
if _, _, err := PAMAdopt(pamBegin+"\naccount required x\n", "/x", read, false); err == nil {
|
||||
t.Error("lines with no end taken")
|
||||
}
|
||||
// The session file unreadable: held to their jobs all the same.
|
||||
on, contained, err := PAMAdopt(ubuntuSSHD, "/l", files(nil), true)
|
||||
if err != nil || contained || !strings.Contains(on, AdoptModule) {
|
||||
t.Errorf("contained %v, err %v:\n%s", contained, err, on)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
package slurm
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"golang.org/x/crypto/ssh"
|
||||
)
|
||||
|
||||
// Shared storage: one path every machine of the Slurm cluster sees, from
|
||||
// the site's NFS export, from an llmgw node that exports it, or mounted by
|
||||
// the site itself.
|
||||
const (
|
||||
// NFSPort is NFSv4's one port, which is all a client needs.
|
||||
NFSPort = 2049
|
||||
// ExportsPath is the export llmgw keeps on the machine that serves.
|
||||
ExportsPath = "/etc/exports.d/llmgw-slurm.exports"
|
||||
// MountUnitDir is where the mount unit is written.
|
||||
MountUnitDir = "/etc/systemd/system"
|
||||
)
|
||||
|
||||
// Exports is the exports file for path, read-write to clients alone. Root
|
||||
// on a client is nobody here: a client's root cannot take another's files.
|
||||
func Exports(path string, clients []string) string {
|
||||
cs := slices.Clone(clients)
|
||||
slices.Sort(cs)
|
||||
cs = slices.Compact(cs)
|
||||
var b strings.Builder
|
||||
b.WriteString("# Written by llmgw (BACKLOG P): the Slurm cluster's shared files, to its machines alone.\n")
|
||||
if len(cs) == 0 {
|
||||
return b.String()
|
||||
}
|
||||
fmt.Fprintf(&b, "%q", path)
|
||||
for _, c := range cs {
|
||||
fmt.Fprintf(&b, " %s(rw,sync,no_subtree_check,root_squash)", c)
|
||||
}
|
||||
b.WriteString("\n")
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// MountUnitName is systemd's name for the mount unit of path, as
|
||||
// systemd-escape --path makes it.
|
||||
func MountUnitName(path string) string {
|
||||
p := strings.Trim(path, "/")
|
||||
var b strings.Builder
|
||||
for i := 0; i < len(p); i++ {
|
||||
c := p[i]
|
||||
switch {
|
||||
case c == '/':
|
||||
b.WriteByte('-')
|
||||
case c >= 'a' && c <= 'z', c >= 'A' && c <= 'Z', c >= '0' && c <= '9', c == '_', c == '.' && i > 0:
|
||||
b.WriteByte(c)
|
||||
default:
|
||||
fmt.Fprintf(&b, `\x%02x`, c)
|
||||
}
|
||||
}
|
||||
return b.String() + ".mount"
|
||||
}
|
||||
|
||||
// MountUnit mounts the export nfs at path. hard: a job waits for a server
|
||||
// that is gone rather than reading errors; nconnect spreads a client's
|
||||
// traffic over several connections. The version is the highest both
|
||||
// sides speak.
|
||||
func MountUnit(nfs, path string) string {
|
||||
return "# Written by llmgw (BACKLOG P): the Slurm cluster's shared files.\n" +
|
||||
"[Unit]\nDescription=llmgw: the Slurm cluster's shared files\n" +
|
||||
"After=network-online.target\nWants=network-online.target\n\n" +
|
||||
"[Mount]\nWhat=" + nfs + "\nWhere=" + path + "\nType=nfs\n" +
|
||||
"Options=hard,nconnect=4,_netdev\nTimeoutSec=60\n\n" +
|
||||
"[Install]\nWantedBy=remote-fs.target\n"
|
||||
}
|
||||
|
||||
// Users: Linux users llmgw makes from its accounts, the same login and UID
|
||||
// on every machine, logging in with the SSH keys given in the portal.
|
||||
const (
|
||||
// AuthorizedKeysDir holds each user's keys, root's to write, so a
|
||||
// user's own home (on a file system shared with others) need hold none.
|
||||
AuthorizedKeysDir = "/etc/llmgw/ssh-keys"
|
||||
// SSHDropIn has sshd look there too, after the user's own files.
|
||||
SSHDropIn = "/etc/ssh/sshd_config.d/50-llmgw-slurm.conf"
|
||||
// MinUID is the lowest UID llmgw gives.
|
||||
MinUID = 200000
|
||||
)
|
||||
|
||||
// SSHConf is SSHDropIn's settings.
|
||||
const SSHConf = "# Written by llmgw (BACKLOG P): the Slurm users' keys, from llmgw's portal.\n" +
|
||||
"AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 " + AuthorizedKeysDir + "/%u\n"
|
||||
|
||||
// login is what useradd takes, as llmgw gives it.
|
||||
var login = regexp.MustCompile(`^[a-z][a-z0-9_-]{0,31}$`)
|
||||
|
||||
// ValidLogin is whether a login is one llmgw gives.
|
||||
func ValidLogin(s string) bool { return login.MatchString(s) }
|
||||
|
||||
// ParseSSHKeys reads OpenSSH public keys, one a line, as authorized_keys
|
||||
// takes them, without options: what a person pastes into the portal.
|
||||
// Blank lines and # comments are skipped; each key comes back as its type,
|
||||
// its base64 and its comment.
|
||||
func ParseSSHKeys(text string) ([]string, error) {
|
||||
var out []string
|
||||
for i, line := range strings.Split(text, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
pk, comment, opts, _, err := ssh.ParseAuthorizedKey([]byte(line))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("line %d is not an SSH public key (as ~/.ssh/id_ed25519.pub has it)", i+1)
|
||||
}
|
||||
if len(opts) > 0 {
|
||||
return nil, fmt.Errorf("line %d has options before its key; only the key is taken", i+1)
|
||||
}
|
||||
k := strings.TrimSpace(string(ssh.MarshalAuthorizedKey(pk)))
|
||||
if comment = strings.Join(strings.Fields(comment), " "); comment != "" {
|
||||
k += " " + comment
|
||||
}
|
||||
if !slices.Contains(out, k) {
|
||||
out = append(out, k)
|
||||
}
|
||||
}
|
||||
if len(out) > 20 {
|
||||
return nil, fmt.Errorf("%d keys: 20 at most", len(out))
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
Loaded 100 of 145 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user