Compare commits
No files matched your search
+167
-6
@@ -49,7 +49,8 @@ Face, done 2026-10-30), then M5 (engines without the internet), M3, M4, M6.
|
||||
**Scale.** O: more than one control plane answering clients; O1 (measuring one router's ceiling) first.
|
||||
|
||||
**Governance.**
|
||||
7. **C1**: a tenant-admin role, and forms hidden from viewers.
|
||||
7. ~~**C1**~~: done 2026-11-08. An organization's admin (`--role admin` on an org membership) changes its teams,
|
||||
projects, people and keys from /org; viewers are drawn no forms; forms made compact.
|
||||
8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
|
||||
|
||||
**When there is room.**
|
||||
@@ -435,9 +436,9 @@ portal is not done — the page shows; the console and ctl change.
|
||||
API middleware and the /admin pages (with each change's result), `ctl
|
||||
audit`, the console's Audit view; staff roles viewer / operator /
|
||||
billing / admin. Left: before/after values (the request's fields are
|
||||
recorded, not the state they replaced); a tenant-admin role that changes
|
||||
its own organization (managers still only view it); hiding the forms a
|
||||
viewer cannot submit. **Audit log and roles (M).** Today "audit" is slog lines
|
||||
recorded, not the state they replaced). Done 2026-11-08: an organization's
|
||||
admin who changes its own organization from /org (managers still only view
|
||||
it), and the forms a viewer cannot submit no longer drawn. **Audit log and roles (M).** Today "audit" is slog lines
|
||||
(`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag
|
||||
plus an org "manager" who can view. Add `audit_events` (who, what,
|
||||
before/after, source) written by the `adminAction`/`tenancyAction`
|
||||
@@ -462,8 +463,9 @@ portal is not done — the page shows; the console and ctl change.
|
||||
5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node
|
||||
lost, budget spent, update available; signed JSON or Slack; retries,
|
||||
hourly de-duplication), `ctl notify test`, Alertmanager receiver
|
||||
templates. Left: `server init` and the console do not yet ask for a
|
||||
receiver — the templates are filled in by hand. Events are claimed in
|
||||
templates. `server init` and both setup screens ask for webhooks and test
|
||||
each (d1e9186, 2026-10-14); with none set, `ctl status` and the console's
|
||||
Overview say the alerts go nowhere (2026-11-12). Events are claimed in
|
||||
the store before they are sent (2026-10-10), so two control planes send
|
||||
each once.
|
||||
**Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
|
||||
@@ -1982,6 +1984,165 @@ one control plane's router can. In order:
|
||||
- O3. Only if O2 is not enough: stateless gateway machines (`llmgw server --role gateway`) in front of the leader's
|
||||
control plane, as many as the traffic needs.
|
||||
|
||||
## P. Training beyond one card (written 2026-11-11) — decision open
|
||||
|
||||
F4 trains a LoRA on one card of a node the caller names (`ctl train start --node`), one job per node, no resume.
|
||||
Larger work (full fine-tuning, a LoRA on a 70B, several GPUs or machines) needs more. Two ways:
|
||||
|
||||
**A. In llmgw.** Add, in order: a job that takes N whole GPUs on one node (torchrun, FSDP/DeepSpeed in the agent's
|
||||
venv); placement by the control plane instead of `--node`, freeing GPUs from replicas by the scaling schedule or by
|
||||
draining; checkpoints to shared storage and resume when an agent or node goes; datasets uploaded or read from
|
||||
storage, not only KB pairs; then gang placement of N nodes with a rendezvous on the first, the NCCL environment from
|
||||
the fabric discovery (IB/NVLink, `llmgw doctor`); a console screen with logs and loss. Size: L for one node, XL for
|
||||
several. For: one product, one install, one set of users and grants (KB rules carry over), GPUs shared with
|
||||
inference without a second scheduler. Against: it is Slurm's gang scheduling, requeue and accounting written again;
|
||||
multi-node NCCL cannot be tried on the PoC (one GPU node, no IB); every customer's training code would have to fit
|
||||
our job format.
|
||||
|
||||
**B. Slurm beside llmgw.** The TUI installs Slurm on the nodes the customer picks (munge, slurmctld on the control
|
||||
planes, slurmd, users from the same LDAP/AD), with pyxis/enroot for containers. A node is handed between the two:
|
||||
llmgw drains its replicas and `scontrol update state=resume`, and back. llmgw's part is what comes out: a checkpoint
|
||||
or adapter from a Slurm job published to the mirror (`ctl train import`), served through canary and the eval. Size:
|
||||
M. For: what research and university customers already run (`sbatch`, `srun torchrun`), any training code, mature
|
||||
multi-node and requeue; we write only the install and the handover. Against: a second system with its own failures,
|
||||
upgrades and users to keep in step; GPUs split by hand or by the handover, not by demand; access rules on training
|
||||
data are Slurm's and the file system's, not the KB's.
|
||||
|
||||
Both need shared storage (NFS, BeeGFS, Lustre) the customer provides; neither changes F4.
|
||||
|
||||
Recommendation: keep F4 in llmgw for small single-card adapters tied to KB grants; take B for anything larger;
|
||||
build only A's first step (N GPUs on one node, placed by the control plane, with resume) if customers ask for
|
||||
mid-size jobs without wanting Slurm.
|
||||
|
||||
**B taken (2026-11-11).** Design: the cluster's settings say it (`slurm: enabled, cluster_name, controller, nodes,
|
||||
lent, partition`); the leader tells each named node's agent (root) what to be, the agents install and run it.
|
||||
- The controller is an llmgw node (its agent runs slurmctld); `nodes` run slurmd configless
|
||||
(`--conf-server`), so slurm.conf, gres.conf and cgroup.conf live only on the controller.
|
||||
- The munge key is derived from the cluster secret (HKDF), so nothing new is stored or copied by hand.
|
||||
- Each node reports `slurmd -C`; the leader writes slurm.conf from those lines, its mgmt address and its GPUs
|
||||
(`Gres=gpu:<type>:N`, gres.conf with the device files), and sends it to the controller again when one changes.
|
||||
- A node in `nodes` belongs to llmgw until it is in `lent`: drained in Slurm (reason "llmgw: serving models").
|
||||
Lent: llmgw drains it as `ctl drain` does (kept in force each tick, as facility holds are, so a new leader
|
||||
keeps it) and, once no replica is left, its agent resumes it in Slurm. Taken out of `lent`: the agent drains
|
||||
it in Slurm, reports the jobs left, and when none are the leader makes it ready for llmgw again.
|
||||
- Phases: P1 the `slurm` package (conf, key, parsers) and settings; P2 the agent side (proto, install, munge,
|
||||
slurmd/slurmctld, status); P3 the leader side (conf from reports, lend/reclaim); P4 `ctl slurm`
|
||||
(status, lend, reclaim) and a TUI screen; P5 `ctl train import` of a Slurm job's output into the mirror.
|
||||
Later: a backup slurmctld on a second node, accounting (slurmdbd; it takes MySQL/MariaDB only, not the Patroni PG), users from LDAP/AD.
|
||||
- Tried on the PoC 2026-11-11 (dev build v0.16.3-17-g849760c-slurm on 140, 150, node05, node06; Ubuntu
|
||||
24.04, Slurm 23.11.4 from the archive): `ctl slurm setup --controller llmgw-node06 --nodes llmgw-node05`
|
||||
installed munge, slurmctld and slurmd, node05 entered drained with its slurmd -C hardware (8 CPUs,
|
||||
31132 MiB after the 3%). `lend`: its 2 replicas moved off, then IDLE in Slurm; srun and sbatch ran there.
|
||||
`reclaim` with a 90 s job running: MIXED+DRAIN, llmgw kept off, ready again within a minute of the job
|
||||
ending. Found and fixed on the way: a client command needs its files even for --version (version now
|
||||
from the daemon); the packaged unit is Type=notify (slurmd needs --systemd); slurmd's own "slurmstepd
|
||||
infinity" was counted as a job; without accounting storage 23.11's main scheduler marks batch jobs
|
||||
InvalidAccount and backfill starts them, one a pass (bf_interval=10: 6-10 s each). So many small jobs
|
||||
want slurmdbd. — Wrong cause, found 2026-11-11: a job asking no memory got the whole node's (ReqTRES
|
||||
mem=all), so a node ran one job at a time and each next one waited for backfill's pass. Fixed: the
|
||||
partition's DefMemPerCPU is the leanest node's share; slurmdbd is not needed for this. InvalidAccount is real too:
|
||||
23.11 dropped accounting_storage/none, and without slurmdbd the main scheduler passes every job by;
|
||||
backfill starts it within bf_interval (10 s). priority/basic did not change that (fair share needs
|
||||
slurmdbd anyway). Starting on submission needs slurmdbd, on MySQL/MariaDB.
|
||||
- slurmdbd, done 2026-11-12 (748f777): the controller's agent installs MariaDB (bound to 127.0.0.1; a
|
||||
MariaDB that was there keeps its settings), makes `slurm_acct_db` and its user (password from the munge
|
||||
key, through a 0600 file, never argv), runs slurmdbd (6819, in the firewall for the cluster) and adds
|
||||
the cluster, before slurmctld. slurm.conf: AccountingStorageType=slurmdbd, jobacct_gather/cgroup,
|
||||
AccountingStorageTRES=gres/gpu, priority/multifactor. Tried on the PoC: five jobs started on
|
||||
submission, `sacct -a` has them with their TRES. Not tried: a fresh `server init --slurm` (needs VMs);
|
||||
a Red Hat controller.
|
||||
- Fresh install tried 2026-11-12 on VMs (3 control planes with Patroni, 2 CPU nodes, `server init
|
||||
--slurm`): Slurm, slurmdbd and MariaDB came up with the install; two-node srun/sbatch on the lent nodes,
|
||||
`sacct` has them; leader failover kept serving. Open from it: `--slurm` needs the first machine to be a
|
||||
node (the controller is its node), so a site wanting control planes without nodes chooses the
|
||||
controller later with `ctl slurm setup`; batch output lands on the job's first node until there is
|
||||
shared storage; on shared CPUs the judge refuses the first request after a placement change (fail
|
||||
closed, ~10 s), and its advice repeats `--cores 4` to a judge that has 4. Tried again on the idle
|
||||
VMs: one judge restarted in place, then both at once, requests every second: 48 of 48 served; the
|
||||
first after a cold start took 7.6–8.9 s end to end against ~3 s, within the 5 s judge budget. So the
|
||||
refusal was the install's load on the CPUs; if it is seen again, warm a new judge with one judgment
|
||||
before it takes requests.
|
||||
- Shared files and users, built 2026-11-12: `slurm.shared` {path, nfs | server} — the site's export
|
||||
mounted (a systemd mount unit, hard, nconnect=4), an llmgw node exporting the path itself
|
||||
(/etc/exports.d, to the Slurm machines' addresses, root_squash; 2049 in the firewall), or the site's
|
||||
own mount only looked for; `slurm.users: llmgw` — every active account a Linux user everywhere
|
||||
(`slurm_users`: login from the e-mail, UID from 200000, never reused; homes made on the exporting
|
||||
machine, else the controller; keys in /etc/llmgw/ssh-keys/%u through an sshd drop-in), SSH keys
|
||||
from the portal's Profile or `ctl slurm users keys`. A user already on a machine with another UID
|
||||
is left alone and said. The site has AD (not yet live): users from it, through sssd, are the next
|
||||
step (`slurm.users` empty today leaves them to the site).
|
||||
- Logging in to a node, done 2026-11-14: llmgw's users reach the controller freely, another Slurm
|
||||
machine only while a job of theirs runs there (pam_slurm_adopt, installed by the agent; PrologFlags=
|
||||
Contain). Only the users listed in the agent's users file are held: root and the machine's own users
|
||||
log in as before, so a wrong line never locks out who would fix it. pam_systemd would move the session
|
||||
out of the job, so sshd's stack takes the session file in place of its include, with pam_systemd
|
||||
skipped for the listed users alone; where that cannot be done safely (nested includes, jumps across
|
||||
it) they are held to having a job, not to its cores and cards, and the agent says so. Tried on the
|
||||
VMs: ayse without a job refused ("no active jobs on this node"), with one in
|
||||
job_N/step_extern (nproc 1), cut off when it ended; mehmet refused; admin's session unchanged.
|
||||
Left: sites whose users come from AD (sssd) are not held yet; a group of theirs would be the list.
|
||||
- A GPU node, tried 2026-11-14 on the PoC (cuda01, Quadro RTX 6000, Slurm 23.11.4): gres.conf
|
||||
`Type=rtx6000 File=/dev/nvidia0`, slurm.conf `Gres=gpu:rtx6000:1`. `srun --gres=gpu:1`:
|
||||
CUDA_VISIBLE_DEVICES=0, the one card; torch found it and ran 14.1 TFLOPS (fp32 matmul). Without
|
||||
--gres: "No devices found", torch none (ConstrainDevices holds). `--gres=gpu:2`: refused. sacct has
|
||||
`gres/gpu=1` in AllocTRES. Found: lending a node whose GPU model has nowhere else to go waits out
|
||||
the drain grace (5 min) before the replica stops, and `ctl slurm status` says only "1 replica(s)
|
||||
still to move off"; it should say it waits for a replacement that cannot be placed, and until when.
|
||||
Done (c3a75dc): it says the time.
|
||||
- Jobs in containers, done 2026-11-14: `slurm.containers` (`ctl slurm containers on`). Each agent
|
||||
installs enroot 4.2.1 (NVIDIA's .deb/.rpm, sha256 pinned) and builds pyxis 0.24.0 (source tarball,
|
||||
sha256 pinned) against its own Slurm (gcc, make, libslurm-dev); on a machine with nvidia-smi,
|
||||
nvidia-container-cli from NVIDIA's repository, its key's digest pinned. enroot.conf keeps each
|
||||
user's runtime, cache and data under /run/enroot and /var/lib/enroot (no login session needed), the
|
||||
shared path bind-mounted in every container. The leader writes plugstack.conf ("required") only
|
||||
once every machine reports both, since a machine without the plugin would fail every job. Tried on
|
||||
the VMs: on in ~30 s; ayse's srun --container-image=ubuntu:24.04 on n1 ran as uid 200001 with
|
||||
/shared inside and nproc 1; a two-node sbatch ran a rank on each. Seen once: a node's pull timed out
|
||||
reaching auth.docker.io and the job failed (network, not pyxis). Not tried: GPUs in a container
|
||||
(cuda01's agent is an older build; needs the peer protocol), Red Hat.
|
||||
- AD end to end, tried 2026-11-14 (decided: one AD per install, the existing auth.ldap group mapping, no
|
||||
tree from AD, no sssd unless a site asks): the VMs' auth.ldap against Samba DC VM 165 (ldaps by IP,
|
||||
CA in the settings, ml-team -> team:Test/arge). ada signed in on the portal with her AD password: an
|
||||
account, in Test/arge, Slurm user ada UID 200003; her key added in the portal's Profile, ssh to the
|
||||
controller ~20 s later, sbatch ran on n1 as ada. Disabled in AD: suspended in 56 s, the Linux user
|
||||
gone in 89 s, ssh refused, home kept. Enabled: back in 32 s, same UID, key kept. Taken out of
|
||||
ml-team: out of the team in 11 s; put back: in again in 56 s.
|
||||
- P6, Slurm's jobs in llmgw's accounting (written 2026-11-12, not started). The tenancy tree to Slurm:
|
||||
each project an account (`sacctmgr add account`, parent its team and organization), each member a user
|
||||
of it, kept by the leader from the store; `sbatch --account=PROJECT`. The records back: the leader reads
|
||||
`sacct --json` for jobs ended since the last read (through the controller's agent) into a
|
||||
`batch_usage` table (job, user, project, start, end, GPU-seconds by class, CPU-seconds, memory), priced
|
||||
per GPU-hour and core-hour into cost_micros, summed with the requests' usage on the usage pages and in
|
||||
budgets. Budgets enforced: a project's remaining budget as GPU-minutes in its account's GrpTRESMins, so
|
||||
Slurm holds its new jobs once spent. Needs the same users on every node (open: LDAP/sssd or
|
||||
llmgw-made). Also found: slurm.conf
|
||||
wrote every node State=DRAIN, so each slurmctld start (on any change of the files) drained a lent node
|
||||
until its agent resumed it, up to a minute; a lent node is now written without it. Not tried yet: a GPU node (gres).
|
||||
- Failover while lent, tried 2026-11-11 (v0.16.3-23): the first try killed the job (NODE_FAIL). A new
|
||||
leader had heard from no node yet and sent the controller a slurm.conf without them; slurmctld,
|
||||
restarted on it, dropped node05 and its job. Fixed (141a1f4, 0774e90): slurm.conf's nodes and the
|
||||
controller's address are kept in cluster_state (`slurm-nodes`) and a new leader starts from them; with
|
||||
nothing kept (an upgraded cluster) or the store unread, the controller's files wait up to 2 min for the
|
||||
nodes; a node is told nothing until its controller's address is known (no part stops its slurmd). Tried
|
||||
again: a 5-minute job on lent node05, leader 140 stopped, 150 took over; slurmctld not restarted, the job
|
||||
ran to its end; reclaimed during it, node05 was llmgw's within seconds of its end. Also 3387f97: srun in
|
||||
SrunPortRange=60001-63000, and Slurm's ports in llmgw's firewall for the cluster's networks.
|
||||
- Built 2026-11-11: P1 (`internal/slurm`, `slurm.*` settings), P2
|
||||
(`internal/agent/slurmnode`, proto Slurm/SlurmStatus), P3 (`internal/control/slurmsvc`, wired in the
|
||||
leader; `GET /api/v1/slurm`), P4's commands (`ctl slurm status|setup|lend|reclaim|off`). Decisions on the
|
||||
way: a node not heard from or with Slurm jobs is cordoned, not drained (nothing of llmgw's to move, and a
|
||||
new leader must not move models off every Slurm node); jobs are counted from the node's own slurmstepd,
|
||||
so a controller that is down neither hides them nor, with none, keeps the node from llmgw; a drain whose
|
||||
reason does not start with "llmgw" is a Slurm administrator's and never resumed. Left: ~~the TUI screen~~
|
||||
(done 2026-11-11: console **Slurm**, lend/take back on a row), P5,
|
||||
a try on the PoC (needs agents built from this; slurm-wlm 23.11 on Ubuntu 24.04), GPU minor numbers are
|
||||
taken to be nvidia-smi's index.
|
||||
- In the first install too (2026-11-11): `llmgw server init --slurm`, and a Slurm part on the choices
|
||||
screen, write `slurm: {enabled, controller: <this hostname>, all_nodes: true}` into server.yaml.
|
||||
`slurm.all_nodes` (also `ctl slurm setup --all-nodes`) runs slurmd on every node in the leader's
|
||||
inventory, those that join later too; none is lent. Refused with `--no-node`, or a hostname Slurm does
|
||||
not take.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, behind the token in
|
||||
|
||||
+157
-1
@@ -20,12 +20,168 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
|
||||
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
|
||||
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node) |
|
||||
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node; v0.16.3: replicas per profile, a served model's settings on the mirror page, the CPU's speed before serving) |
|
||||
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
||||
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Added
|
||||
|
||||
- **An organization's admin.** Besides member and manager, a membership of
|
||||
an organization may now be `admin` (`llmgw users join … --org Acme --role
|
||||
admin`, the role chosen on a membership on the people page, the console,
|
||||
the admin API). An admin sees the organization's page as a manager does and
|
||||
also changes the organization from it: adds, renames and removes its teams
|
||||
and projects, adds people with an approved account by their address and
|
||||
takes them out, makes its members managers or admins, and revokes keys spent
|
||||
in its projects. Limits, approving and suspending accounts, the
|
||||
organization's name and the rest of the cluster stay with the cluster's
|
||||
administrators; an admin cannot take away their own role. Every change is in
|
||||
the audit log.
|
||||
- **Slurm beside llmgw, for training and batch work.** `llmgw ctl slurm setup
|
||||
--controller NODE --nodes NODE,...` (or `--all-nodes`, or `server init
|
||||
--slurm`) has the agents install munge and Slurm from the machines' package
|
||||
sources: slurmctld on one node, slurmd configless on the others, slurm.conf
|
||||
and gres.conf written from what each node reports. Every node stays llmgw's,
|
||||
drained in Slurm, until `ctl slurm lend NODE`: its replicas move off, then
|
||||
Slurm runs jobs there. `ctl slurm reclaim NODE` takes it back once its jobs
|
||||
end. `ctl slurm status` and the console's **Slurm** show each node. The
|
||||
controller also keeps the jobs' records: slurmdbd, in a MariaDB on that
|
||||
machine alone (Slurm takes MySQL or MariaDB only). Slurm's ports
|
||||
(6817–6819, srun's 60001–63000) answer the cluster's networks only, as
|
||||
llmgw's own do.
|
||||
- **Slurm's shared files and users.** `slurm.shared` (`ctl slurm shared`) is
|
||||
one path on every Slurm machine: the site's NFS export mounted, an llmgw
|
||||
node exporting it over NFS (what `server init --slurm` chooses, `/shared`
|
||||
from the first machine), or a file system the site mounts. `slurm.users:
|
||||
llmgw` (`ctl slurm users on`) makes each active account a Linux user on
|
||||
every Slurm machine, the same login and UID (from 200000, never reused)
|
||||
everywhere, its home on the shared path; people add SSH keys in the
|
||||
portal's Profile (`ctl slurm users keys` for an operator). They log in to
|
||||
the controller; to another Slurm machine only while a job of theirs runs
|
||||
there, their session put in the job (pam_slurm_adopt), so nobody uses a
|
||||
node's cards outside Slurm, nor on a node llmgw serves from. Root and the
|
||||
machines' own users are not held to it.
|
||||
- **Slurm's jobs in containers.** `llmgw ctl slurm containers on`
|
||||
(`slurm.containers`) installs enroot and pyxis on every Slurm machine, at
|
||||
pinned versions checked against their digests, and nvidia-container-cli on
|
||||
GPU machines; Slurm loads pyxis once every machine has it, and then `srun
|
||||
--container-image=...` runs a job in that image, as its user, with the
|
||||
shared path inside.
|
||||
|
||||
### Changed
|
||||
|
||||
- **Forms only for those who may send them.** A viewer sees the people,
|
||||
limits, mirror and cluster pages without their forms, an operator the
|
||||
mirror's and a node's engines but not the rest; before, they were drawn and
|
||||
answered "not found".
|
||||
- **Smaller forms.** A label sits close to its field, a field is as wide as
|
||||
what goes in it rather than the page, buttons are smaller, and the short
|
||||
forms — a new key, a password, connecting an editor, a capacity or a limit —
|
||||
sit on one line or two instead of a screen.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **The serve forms start at the longest context that fits.** The mirror
|
||||
page's form began every GGUF model at 4096 tokens, and the console's at
|
||||
4096 or the plan's figure, whatever the model was trained for. Both now
|
||||
start at what it was trained for, as long as its KV cache fits where the
|
||||
form places it beside the weights, else at the most that does, in whole
|
||||
thousands; the mirror page's llama.cpp slots are as many of that, up to
|
||||
four, as the room left holds, and its memory figure is for that pool. A
|
||||
plan says what the model was trained for (`trained`). Strata stays at
|
||||
its 32768; vLLM's form is unchanged.
|
||||
- **A model that failed to start was started in the same place again.**
|
||||
Placement given the same inputs chooses the same node and card, so an
|
||||
on-demand model whose replica failed to load was retried there, past each
|
||||
backoff, until it was parked, with another card or node free all along.
|
||||
The control plane now remembers where a model's replicas failed (node and
|
||||
cards, up to eight places) and places the next one elsewhere when anywhere
|
||||
else fits; with nowhere else it is tried there as before. Coming up, a
|
||||
change to its definition or `ctl retry` forgets the places. The log says
|
||||
`placing a model away from where it failed to start`, with both places.
|
||||
- **Deleted model files came back after a control plane restart.** An agent
|
||||
registers with the files its startup scan found, and sends the list again
|
||||
only when it changes; a file deleted after the agent started was listed
|
||||
again by a control plane that restarted, until the next change. A new
|
||||
stream now gets the list on its first heartbeat.
|
||||
- **`ctl weights` offered models in use for deletion.** A model of several
|
||||
files (a set, its address ending in `/`) is fetched into a directory named
|
||||
from the address without the slash; the sweep named it from the address
|
||||
with it, so every set in the catalog looked like nobody's. On the PoC that
|
||||
was 330 GB of six catalog models, the qwen3.8-flash-next Q8_0 among them;
|
||||
`--collect`, and `ctl model remove --purge` of a set, would have deleted
|
||||
them. Both now compute the name the fetch does.
|
||||
- **A sign-in took an organization's manager role away.** A group from single
|
||||
sign-on, SCIM or Active Directory that puts someone in an organization wrote
|
||||
the membership again at every sign-in or sync, as a plain member: a manager
|
||||
made one on the people page was one only until they next signed in. A
|
||||
membership the directory names is now added if missing and otherwise left
|
||||
as it is.
|
||||
- **The limits form on the people page's panel** was laid out on one line, its
|
||||
title squeezed into a column.
|
||||
- **Deleting the model files nothing uses said what it was asked to free,
|
||||
not what it freed.** The console's "Delete model files nothing uses" and
|
||||
`ctl weights --collect` added up every file the nodes were asked to
|
||||
delete as soon as the request was sent, and a model of several files —
|
||||
fetched into a directory of its own — was refused by the node as "not
|
||||
empty", so a sweep said tens of gigabytes freed and nothing went. The
|
||||
node now deletes such a directory whole (a link only as a link), and the
|
||||
sweep waits for the nodes to list their files again: freed is what they
|
||||
no longer list, and what they kept is said beside, with where to read
|
||||
why. Removing a model with its files reports the same way.
|
||||
|
||||
## v0.16.3 — 2026-11-03
|
||||
|
||||
Replicas counted per profile — seven on cards and four on CPUs under one name, the cards filled first — and every
|
||||
setting of a served model from the mirror page; how fast a model answers on the CPU, said before it is served, by
|
||||
the threads it is given rather than the whole machine's memory.
|
||||
|
||||
**Upgrading.**
|
||||
- No migrations. `llmgw ctl upgrade v0.16.3`: the control planes carry the replicas per profile, the model card and
|
||||
the speed in the serve form and in `ctl model plan`; the agents carry the measurement of one core's memory rate,
|
||||
without which the speed is reckoned the old way, from the whole machine's.
|
||||
|
||||
### Added
|
||||
|
||||
- **Replicas counted per profile: seven on cards and four on CPUs, under one
|
||||
name.** `llmgw ctl model replicas <model> gpu=7 cpu=4`, the console's
|
||||
Replicas per profile… and the mirror page's model card set how many of a
|
||||
model's replicas run on each of its profiles, rather than every replica
|
||||
on the best profile that fits. For a GGUF model with no profile on CPUs
|
||||
one is added (`--cpu-memory`), preferred below the cards, so the router
|
||||
fills the cards first and sends to the CPUs once they are full. A change
|
||||
starts the replicas short before stopping those over: the model never
|
||||
runs fewer. `ctl models` shows the counts; a scaling of another total
|
||||
ends them.
|
||||
- **Every setting of a served model from the mirror page.** A served
|
||||
model's row has a Settings link, and its card now changes the context
|
||||
and longest answer, the cards a replica takes, the replicas on each
|
||||
profile (with the CPU's speed beside), stops and starts it, restarts its
|
||||
replicas, and takes it out of the catalog with its files or without —
|
||||
beside the replicas, class, owner, memory and adapters it held before.
|
||||
Stopped on the row is now the model stopped and kept, started again as
|
||||
it was, rather than its scaling set to nothing.
|
||||
- **How fast a model answers on the CPU, before it is served.** The serve
|
||||
form says, as the cores are changed, about how many tokens a second one
|
||||
reply gets on the CPU and on one card of the fleet's fastest class, and
|
||||
warns when it is slow (under 15) or very slow (under 5, a reply of 500
|
||||
tokens taking minutes). Very slow on the CPU is refused until "run on the
|
||||
CPU anyway" is ticked, GPUs free or not. `llmgw ctl model plan` and the
|
||||
console's Hugging Face screen show the same figure for every way a model
|
||||
can run, and `ctl model serve --cpu` says it in its log.
|
||||
- **The speed follows the threads a replica is given.** Each agent now
|
||||
measures what one core reads beside what all of them read: a replica of
|
||||
four threads on a large server reads at about four cores' rate, not the
|
||||
server's, which the estimate used to assume and overstated several times.
|
||||
Calibrated on the PoC by llama-bench at 1, 2, 4 and 8 threads (within 15%),
|
||||
and a model of experts reads its experts' share, from the GGUF header.
|
||||
Agents need upgrading to report one core's rate; until then the old
|
||||
estimate is used.
|
||||
|
||||
## v0.16.2 — 2026-11-02
|
||||
|
||||
Standby control planes that answer clients themselves, with rate limits and budgets kept across them; FreeToken, an
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
# Functional Source License, Version 1.1, ALv2 Future License
|
||||
|
||||
## Abbreviation
|
||||
|
||||
FSL-1.1-ALv2
|
||||
|
||||
## Notice
|
||||
|
||||
Copyright 2026 Ilker Manap
|
||||
|
||||
## Terms and Conditions
|
||||
|
||||
### Licensor ("We")
|
||||
|
||||
The party offering the Software under these Terms and Conditions.
|
||||
|
||||
### The Software
|
||||
|
||||
The "Software" is each version of the software that we make available under
|
||||
these Terms and Conditions, as indicated by our inclusion of these Terms and
|
||||
Conditions with the Software.
|
||||
|
||||
### License Grant
|
||||
|
||||
Subject to your compliance with this License Grant and the Patents,
|
||||
Redistribution and Trademark clauses below, we hereby grant you the right to
|
||||
use, copy, modify, create derivative works, publicly perform, publicly display
|
||||
and redistribute the Software for any Permitted Purpose identified below.
|
||||
|
||||
### Permitted Purpose
|
||||
|
||||
A Permitted Purpose is any purpose other than a Competing Use. A Competing Use
|
||||
means making the Software available to others in a commercial product or
|
||||
service that:
|
||||
|
||||
1. substitutes for the Software;
|
||||
|
||||
2. substitutes for any other product or service we offer using the Software
|
||||
that exists as of the date we make the Software available; or
|
||||
|
||||
3. offers the same or substantially similar functionality as the Software.
|
||||
|
||||
Permitted Purposes specifically include using the Software:
|
||||
|
||||
1. for your internal use and access;
|
||||
|
||||
2. for non-commercial education;
|
||||
|
||||
3. for non-commercial research; and
|
||||
|
||||
4. in connection with professional services that you provide to a licensee
|
||||
using the Software in accordance with these Terms and Conditions.
|
||||
|
||||
### Patents
|
||||
|
||||
To the extent your use for a Permitted Purpose would necessarily infringe our
|
||||
patents, the license grant above includes a license under our patents. If you
|
||||
make a claim against any party that the Software infringes or contributes to
|
||||
the infringement of any patent, then your patent license to the Software ends
|
||||
immediately.
|
||||
|
||||
### Redistribution
|
||||
|
||||
The Terms and Conditions apply to all copies, modifications and derivatives of
|
||||
the Software.
|
||||
|
||||
If you redistribute any copies, modifications or derivatives of the Software,
|
||||
you must include a copy of or a link to these Terms and Conditions and not
|
||||
remove any copyright notices provided in or with the Software.
|
||||
|
||||
### Disclaimer
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTIES OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING WITHOUT LIMITATION WARRANTIES OF FITNESS FOR A PARTICULAR
|
||||
PURPOSE, MERCHANTABILITY, TITLE OR NON-INFRINGEMENT.
|
||||
|
||||
IN NO EVENT WILL WE HAVE ANY LIABILITY TO YOU ARISING OUT OF OR RELATED TO THE
|
||||
SOFTWARE, INCLUDING INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES,
|
||||
EVEN IF WE HAVE BEEN INFORMED OF THEIR POSSIBILITY IN ADVANCE.
|
||||
|
||||
### Trademarks
|
||||
|
||||
Except for displaying the License Details and identifying us as the origin of
|
||||
the Software, you have no right under these Terms and Conditions to use our
|
||||
trademarks, trade names, service marks or product names.
|
||||
|
||||
## Grant of Future License
|
||||
|
||||
We hereby irrevocably grant you an additional license to use the Software under
|
||||
the Apache License, Version 2.0 that is effective on the second anniversary of
|
||||
the date we make the Software available. On or after that date, you may use the
|
||||
Software under the Apache License, Version 2.0, in which case the following
|
||||
will apply:
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use
|
||||
this file except in compliance with the License.
|
||||
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed
|
||||
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
|
||||
CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
@@ -42,7 +42,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
|
||||
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
|
||||
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node) |
|
||||
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node; v0.16.3: replicas per profile, a served model's settings on the mirror page, the CPU's speed before serving) |
|
||||
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
||||
|
||||
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
|
||||
@@ -372,3 +372,10 @@ make rename NEW=git.example.com/team/llmgw
|
||||
## Language
|
||||
|
||||
Code, comments, tests, commit messages and documentation are in English.
|
||||
|
||||
## Licence
|
||||
|
||||
Functional Source License 1.1 with an Apache 2.0 future licence
|
||||
(FSL-1.1-ALv2); see [LICENSE](LICENSE). Each release may be used, read and
|
||||
changed for anything except offering it as a competing product or service,
|
||||
and becomes Apache 2.0 two years after it is published.
|
||||
+667
-81
@@ -108,6 +108,7 @@ type AgentMessage struct {
|
||||
// *AgentMessage_KvEvents
|
||||
// *AgentMessage_TrainStatus
|
||||
// *AgentMessage_TrainArtifact
|
||||
// *AgentMessage_SlurmStatus
|
||||
Msg isAgentMessage_Msg `protobuf_oneof:"msg"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
@@ -240,6 +241,15 @@ func (x *AgentMessage) GetTrainArtifact() *TrainArtifact {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *AgentMessage) GetSlurmStatus() *SlurmStatus {
|
||||
if x != nil {
|
||||
if x, ok := x.Msg.(*AgentMessage_SlurmStatus); ok {
|
||||
return x.SlurmStatus
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type isAgentMessage_Msg interface {
|
||||
isAgentMessage_Msg()
|
||||
}
|
||||
@@ -284,6 +294,10 @@ type AgentMessage_TrainArtifact struct {
|
||||
TrainArtifact *TrainArtifact `protobuf:"bytes,10,opt,name=train_artifact,json=trainArtifact,proto3,oneof"`
|
||||
}
|
||||
|
||||
type AgentMessage_SlurmStatus struct {
|
||||
SlurmStatus *SlurmStatus `protobuf:"bytes,11,opt,name=slurm_status,json=slurmStatus,proto3,oneof"`
|
||||
}
|
||||
|
||||
func (*AgentMessage_Register) isAgentMessage_Msg() {}
|
||||
|
||||
func (*AgentMessage_Heartbeat) isAgentMessage_Msg() {}
|
||||
@@ -304,6 +318,8 @@ func (*AgentMessage_TrainStatus) isAgentMessage_Msg() {}
|
||||
|
||||
func (*AgentMessage_TrainArtifact) isAgentMessage_Msg() {}
|
||||
|
||||
func (*AgentMessage_SlurmStatus) isAgentMessage_Msg() {}
|
||||
|
||||
// KVEvents is a change to what one vLLM replica's prefix cache holds, read
|
||||
// by the agent from the engine's KV cache events and sent on for the router's
|
||||
// precise index (plan §7.5 tier 2).
|
||||
@@ -426,6 +442,7 @@ type ControlMessage struct {
|
||||
// *ControlMessage_AgentSettings
|
||||
// *ControlMessage_TrainStart
|
||||
// *ControlMessage_TrainCancel
|
||||
// *ControlMessage_Slurm
|
||||
Msg isControlMessage_Msg `protobuf_oneof:"msg"`
|
||||
// trace is the W3C trace context (traceparent, and tracestate when there
|
||||
// is one) of the control plane's span that sent this command, so what the
|
||||
@@ -618,6 +635,15 @@ func (x *ControlMessage) GetTrainCancel() *TrainCancel {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *ControlMessage) GetSlurm() *Slurm {
|
||||
if x != nil {
|
||||
if x, ok := x.Msg.(*ControlMessage_Slurm); ok {
|
||||
return x.Slurm
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *ControlMessage) GetTrace() map[string]string {
|
||||
if x != nil {
|
||||
return x.Trace
|
||||
@@ -693,6 +719,10 @@ type ControlMessage_TrainCancel struct {
|
||||
TrainCancel *TrainCancel `protobuf:"bytes,17,opt,name=train_cancel,json=trainCancel,proto3,oneof"`
|
||||
}
|
||||
|
||||
type ControlMessage_Slurm struct {
|
||||
Slurm *Slurm `protobuf:"bytes,18,opt,name=slurm,proto3,oneof"`
|
||||
}
|
||||
|
||||
func (*ControlMessage_Start) isControlMessage_Msg() {}
|
||||
|
||||
func (*ControlMessage_Stop) isControlMessage_Msg() {}
|
||||
@@ -725,6 +755,8 @@ func (*ControlMessage_TrainStart) isControlMessage_Msg() {}
|
||||
|
||||
func (*ControlMessage_TrainCancel) isControlMessage_Msg() {}
|
||||
|
||||
func (*ControlMessage_Slurm) isControlMessage_Msg() {}
|
||||
|
||||
// RestartAgent has the agent start again, as llmgw restart asks of every
|
||||
// node: its engines keep serving and are taken over again.
|
||||
type RestartAgent struct {
|
||||
@@ -1210,8 +1242,12 @@ type Host struct {
|
||||
// mem_bandwidth_gbs is how fast the memory reads, in GB/s, measured when
|
||||
// the agent started: what decides a model's speed on the CPU. 0 unknown.
|
||||
MemBandwidthGbs float64 `protobuf:"fixed64,7,opt,name=mem_bandwidth_gbs,json=memBandwidthGbs,proto3" json:"mem_bandwidth_gbs,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
// mem_bandwidth_core_gbs is what one core reads alone, in GB/s: a replica
|
||||
// given a few threads of a large machine reads at a few times this, not at
|
||||
// the whole machine's rate. 0 unknown.
|
||||
MemBandwidthCoreGbs float64 `protobuf:"fixed64,8,opt,name=mem_bandwidth_core_gbs,json=memBandwidthCoreGbs,proto3" json:"mem_bandwidth_core_gbs,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *Host) Reset() {
|
||||
@@ -1293,6 +1329,13 @@ func (x *Host) GetMemBandwidthGbs() float64 {
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *Host) GetMemBandwidthCoreGbs() float64 {
|
||||
if x != nil {
|
||||
return x.MemBandwidthCoreGbs
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// Storage is one filesystem a node keeps weights on.
|
||||
//
|
||||
// Per filesystem rather than per configured directory: free space belongs to
|
||||
@@ -3049,8 +3092,10 @@ type Heartbeat struct {
|
||||
// the agent has measured it — in the background after it starts, so a
|
||||
// registration usually has none yet. 0 means not measured.
|
||||
MemBandwidthGbs float64 `protobuf:"fixed64,20,opt,name=mem_bandwidth_gbs,json=memBandwidthGbs,proto3" json:"mem_bandwidth_gbs,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
// mem_bandwidth_core_gbs is one core's read rate, as in Host.
|
||||
MemBandwidthCoreGbs float64 `protobuf:"fixed64,21,opt,name=mem_bandwidth_core_gbs,json=memBandwidthCoreGbs,proto3" json:"mem_bandwidth_core_gbs,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *Heartbeat) Reset() {
|
||||
@@ -3223,6 +3268,13 @@ func (x *Heartbeat) GetMemBandwidthGbs() float64 {
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *Heartbeat) GetMemBandwidthCoreGbs() float64 {
|
||||
if x != nil {
|
||||
return x.MemBandwidthCoreGbs
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// Provision is the software a node should have: which engines, at which
|
||||
// versions. Nothing more.
|
||||
//
|
||||
@@ -5231,11 +5283,477 @@ func (x *TrainArtifact) GetLast() bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// Slurm is what this node is in the Slurm cluster run beside llmgw
|
||||
// (BACKLOG P): its controller, one of its nodes, both or neither. Sent when
|
||||
// an agent registers and whenever it changes; the agent installs what is
|
||||
// missing and keeps the daemons running as it says.
|
||||
type Slurm struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
// controller runs slurmctld here, on files.
|
||||
Controller bool `protobuf:"varint,1,opt,name=controller,proto3" json:"controller,omitempty"`
|
||||
// node runs slurmd here, configless, fetching its files from conf_server.
|
||||
Node bool `protobuf:"varint,2,opt,name=node,proto3" json:"node,omitempty"`
|
||||
// name is this node's NodeName in slurm.conf.
|
||||
Name string `protobuf:"bytes,3,opt,name=name,proto3" json:"name,omitempty"`
|
||||
// conf_server is slurmctld's host:port.
|
||||
ConfServer string `protobuf:"bytes,4,opt,name=conf_server,json=confServer,proto3" json:"conf_server,omitempty"`
|
||||
// munge_key is the key every machine of the cluster shares.
|
||||
MungeKey []byte `protobuf:"bytes,5,opt,name=munge_key,json=mungeKey,proto3" json:"munge_key,omitempty"`
|
||||
// files are the controller's, by name in /etc/slurm; empty for a node
|
||||
// that is not the controller.
|
||||
Files map[string]string `protobuf:"bytes,6,rep,name=files,proto3" json:"files,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
|
||||
// lent says Slurm may run jobs here now: resume the node if llmgw drained
|
||||
// it. Otherwise it is drained, its jobs left to finish.
|
||||
Lent bool `protobuf:"varint,7,opt,name=lent,proto3" json:"lent,omitempty"`
|
||||
// shared is the file system every machine of the Slurm cluster sees at
|
||||
// the same path; unset or with no path, none.
|
||||
Shared *SlurmShared `protobuf:"bytes,8,opt,name=shared,proto3" json:"shared,omitempty"`
|
||||
// manage_users has llmgw keep the Linux users below on this machine,
|
||||
// and take away those it made that are no longer among them.
|
||||
ManageUsers bool `protobuf:"varint,9,opt,name=manage_users,json=manageUsers,proto3" json:"manage_users,omitempty"`
|
||||
Users []*SlurmUser `protobuf:"bytes,10,rep,name=users,proto3" json:"users,omitempty"`
|
||||
// make_homes has this machine make the users' home directories: the one
|
||||
// that exports the shared path, else the controller.
|
||||
MakeHomes bool `protobuf:"varint,11,opt,name=make_homes,json=makeHomes,proto3" json:"make_homes,omitempty"`
|
||||
// containers has the machine install pyxis and enroot, so jobs may run
|
||||
// in containers (srun --container-image), and set them up.
|
||||
Containers bool `protobuf:"varint,12,opt,name=containers,proto3" json:"containers,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *Slurm) Reset() {
|
||||
*x = Slurm{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[48]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *Slurm) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*Slurm) ProtoMessage() {}
|
||||
|
||||
func (x *Slurm) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[48]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use Slurm.ProtoReflect.Descriptor instead.
|
||||
func (*Slurm) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{48}
|
||||
}
|
||||
|
||||
func (x *Slurm) GetController() bool {
|
||||
if x != nil {
|
||||
return x.Controller
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Slurm) GetNode() bool {
|
||||
if x != nil {
|
||||
return x.Node
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Slurm) GetName() string {
|
||||
if x != nil {
|
||||
return x.Name
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *Slurm) GetConfServer() string {
|
||||
if x != nil {
|
||||
return x.ConfServer
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *Slurm) GetMungeKey() []byte {
|
||||
if x != nil {
|
||||
return x.MungeKey
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *Slurm) GetFiles() map[string]string {
|
||||
if x != nil {
|
||||
return x.Files
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *Slurm) GetLent() bool {
|
||||
if x != nil {
|
||||
return x.Lent
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Slurm) GetShared() *SlurmShared {
|
||||
if x != nil {
|
||||
return x.Shared
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *Slurm) GetManageUsers() bool {
|
||||
if x != nil {
|
||||
return x.ManageUsers
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Slurm) GetUsers() []*SlurmUser {
|
||||
if x != nil {
|
||||
return x.Users
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *Slurm) GetMakeHomes() bool {
|
||||
if x != nil {
|
||||
return x.MakeHomes
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Slurm) GetContainers() bool {
|
||||
if x != nil {
|
||||
return x.Containers
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// SlurmShared is where the Slurm cluster's shared files come from.
|
||||
type SlurmShared struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
// path is where every machine sees them, as /shared.
|
||||
Path string `protobuf:"bytes,1,opt,name=path,proto3" json:"path,omitempty"`
|
||||
// nfs is an export, host:/path, to mount at path; empty when this
|
||||
// machine exports path itself (serve) or the site mounts it.
|
||||
Nfs string `protobuf:"bytes,2,opt,name=nfs,proto3" json:"nfs,omitempty"`
|
||||
// serve has this machine export path over NFS to clients.
|
||||
Serve bool `protobuf:"varint,3,opt,name=serve,proto3" json:"serve,omitempty"`
|
||||
// clients are the addresses the export answers.
|
||||
Clients []string `protobuf:"bytes,4,rep,name=clients,proto3" json:"clients,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *SlurmShared) Reset() {
|
||||
*x = SlurmShared{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *SlurmShared) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*SlurmShared) ProtoMessage() {}
|
||||
|
||||
func (x *SlurmShared) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use SlurmShared.ProtoReflect.Descriptor instead.
|
||||
func (*SlurmShared) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{49}
|
||||
}
|
||||
|
||||
func (x *SlurmShared) GetPath() string {
|
||||
if x != nil {
|
||||
return x.Path
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmShared) GetNfs() string {
|
||||
if x != nil {
|
||||
return x.Nfs
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmShared) GetServe() bool {
|
||||
if x != nil {
|
||||
return x.Serve
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *SlurmShared) GetClients() []string {
|
||||
if x != nil {
|
||||
return x.Clients
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SlurmUser is a Linux user that runs jobs, the same on every machine.
|
||||
type SlurmUser struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Login string `protobuf:"bytes,1,opt,name=login,proto3" json:"login,omitempty"`
|
||||
// uid is also the user's own group's gid.
|
||||
Uid uint32 `protobuf:"varint,2,opt,name=uid,proto3" json:"uid,omitempty"`
|
||||
// name is the account's, for the GECOS field.
|
||||
Name string `protobuf:"bytes,3,opt,name=name,proto3" json:"name,omitempty"`
|
||||
Home string `protobuf:"bytes,4,opt,name=home,proto3" json:"home,omitempty"`
|
||||
// ssh_keys are OpenSSH public keys, one a line of authorized_keys.
|
||||
SshKeys []string `protobuf:"bytes,5,rep,name=ssh_keys,json=sshKeys,proto3" json:"ssh_keys,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *SlurmUser) Reset() {
|
||||
*x = SlurmUser{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[50]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *SlurmUser) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*SlurmUser) ProtoMessage() {}
|
||||
|
||||
func (x *SlurmUser) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[50]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use SlurmUser.ProtoReflect.Descriptor instead.
|
||||
func (*SlurmUser) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{50}
|
||||
}
|
||||
|
||||
func (x *SlurmUser) GetLogin() string {
|
||||
if x != nil {
|
||||
return x.Login
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmUser) GetUid() uint32 {
|
||||
if x != nil {
|
||||
return x.Uid
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *SlurmUser) GetName() string {
|
||||
if x != nil {
|
||||
return x.Name
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmUser) GetHome() string {
|
||||
if x != nil {
|
||||
return x.Home
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmUser) GetSshKeys() []string {
|
||||
if x != nil {
|
||||
return x.SshKeys
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SlurmStatus is what this node's Slurm daemons are doing, sent when it
|
||||
// changes and every few minutes while the node has a part in Slurm.
|
||||
type SlurmStatus struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
// version is Slurm's, as "23.11.4"; empty when it is not installed.
|
||||
Version string `protobuf:"bytes,1,opt,name=version,proto3" json:"version,omitempty"`
|
||||
// hardware is `slurmd -C` on a node, which slurm.conf must match.
|
||||
Hardware string `protobuf:"bytes,2,opt,name=hardware,proto3" json:"hardware,omitempty"`
|
||||
SlurmdRunning bool `protobuf:"varint,3,opt,name=slurmd_running,json=slurmdRunning,proto3" json:"slurmd_running,omitempty"`
|
||||
SlurmctldRunning bool `protobuf:"varint,4,opt,name=slurmctld_running,json=slurmctldRunning,proto3" json:"slurmctld_running,omitempty"`
|
||||
// state and reason are the controller's for this node, from scontrol.
|
||||
State string `protobuf:"bytes,5,opt,name=state,proto3" json:"state,omitempty"`
|
||||
Reason string `protobuf:"bytes,6,opt,name=reason,proto3" json:"reason,omitempty"`
|
||||
// jobs is how many job steps run on this node now, counted from their
|
||||
// slurmstepd so a controller that is down does not hide them; -1 when the
|
||||
// agent cannot tell.
|
||||
Jobs int32 `protobuf:"varint,7,opt,name=jobs,proto3" json:"jobs,omitempty"`
|
||||
// error is the last thing that went wrong keeping Slurm, empty when
|
||||
// nothing did.
|
||||
Error string `protobuf:"bytes,8,opt,name=error,proto3" json:"error,omitempty"`
|
||||
// files_sha256 is of the controller's files as written, so the leader
|
||||
// knows they arrived.
|
||||
FilesSha256 string `protobuf:"bytes,9,opt,name=files_sha256,json=filesSha256,proto3" json:"files_sha256,omitempty"`
|
||||
// slurmdbd_running is the accounting daemon on the controller, which keeps
|
||||
// the jobs' records in a MariaDB beside it.
|
||||
SlurmdbdRunning bool `protobuf:"varint,10,opt,name=slurmdbd_running,json=slurmdbdRunning,proto3" json:"slurmdbd_running,omitempty"`
|
||||
// shared is the shared path as this machine has it: "mounted",
|
||||
// "exported", "missing", or empty when there is none.
|
||||
Shared string `protobuf:"bytes,11,opt,name=shared,proto3" json:"shared,omitempty"`
|
||||
// users is how many of llmgw's users this machine has.
|
||||
Users int32 `protobuf:"varint,12,opt,name=users,proto3" json:"users,omitempty"`
|
||||
// containers is what this machine has for jobs in containers, as
|
||||
// "pyxis 0.24.0, enroot 4.2.1"; empty while it has not got both.
|
||||
Containers string `protobuf:"bytes,13,opt,name=containers,proto3" json:"containers,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) Reset() {
|
||||
*x = SlurmStatus{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[51]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*SlurmStatus) ProtoMessage() {}
|
||||
|
||||
func (x *SlurmStatus) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[51]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use SlurmStatus.ProtoReflect.Descriptor instead.
|
||||
func (*SlurmStatus) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{51}
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetVersion() string {
|
||||
if x != nil {
|
||||
return x.Version
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetHardware() string {
|
||||
if x != nil {
|
||||
return x.Hardware
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetSlurmdRunning() bool {
|
||||
if x != nil {
|
||||
return x.SlurmdRunning
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetSlurmctldRunning() bool {
|
||||
if x != nil {
|
||||
return x.SlurmctldRunning
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetState() string {
|
||||
if x != nil {
|
||||
return x.State
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetReason() string {
|
||||
if x != nil {
|
||||
return x.Reason
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetJobs() int32 {
|
||||
if x != nil {
|
||||
return x.Jobs
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetError() string {
|
||||
if x != nil {
|
||||
return x.Error
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetFilesSha256() string {
|
||||
if x != nil {
|
||||
return x.FilesSha256
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetSlurmdbdRunning() bool {
|
||||
if x != nil {
|
||||
return x.SlurmdbdRunning
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetShared() string {
|
||||
if x != nil {
|
||||
return x.Shared
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetUsers() int32 {
|
||||
if x != nil {
|
||||
return x.Users
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetContainers() string {
|
||||
if x != nil {
|
||||
return x.Containers
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
var File_llmgw_v1_agent_proto protoreflect.FileDescriptor
|
||||
|
||||
const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\n" +
|
||||
"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xb7\x04\n" +
|
||||
"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xf3\x04\n" +
|
||||
"\fAgentMessage\x120\n" +
|
||||
"\bregister\x18\x01 \x01(\v2\x12.llmgw.v1.RegisterH\x00R\bregister\x123\n" +
|
||||
"\theartbeat\x18\x02 \x01(\v2\x13.llmgw.v1.HeartbeatH\x00R\theartbeat\x12-\n" +
|
||||
@@ -5249,7 +5767,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\tkv_events\x18\b \x01(\v2\x12.llmgw.v1.KVEventsH\x00R\bkvEvents\x12:\n" +
|
||||
"\ftrain_status\x18\t \x01(\v2\x15.llmgw.v1.TrainStatusH\x00R\vtrainStatus\x12@\n" +
|
||||
"\x0etrain_artifact\x18\n" +
|
||||
" \x01(\v2\x17.llmgw.v1.TrainArtifactH\x00R\rtrainArtifactB\x05\n" +
|
||||
" \x01(\v2\x17.llmgw.v1.TrainArtifactH\x00R\rtrainArtifact\x12:\n" +
|
||||
"\fslurm_status\x18\v \x01(\v2\x15.llmgw.v1.SlurmStatusH\x00R\vslurmStatusB\x05\n" +
|
||||
"\x03msg\"\xb5\x01\n" +
|
||||
"\bKVEvents\x12\x1d\n" +
|
||||
"\n" +
|
||||
@@ -5260,7 +5779,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"clearFirst\x12\x16\n" +
|
||||
"\x06stored\x18\x04 \x03(\x06R\x06stored\x12\x18\n" +
|
||||
"\aremoved\x18\x05 \x03(\x06R\aremoved\x12\x18\n" +
|
||||
"\aproblem\x18\x06 \x01(\tR\aproblem\"\xf8\a\n" +
|
||||
"\aproblem\x18\x06 \x01(\tR\aproblem\"\xa1\b\n" +
|
||||
"\x0eControlMessage\x12,\n" +
|
||||
"\x05start\x18\x01 \x01(\v2\x14.llmgw.v1.StartShardH\x00R\x05start\x12)\n" +
|
||||
"\x04stop\x18\x02 \x01(\v2\x13.llmgw.v1.StopShardH\x00R\x04stop\x12,\n" +
|
||||
@@ -5281,7 +5800,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x0eagent_settings\x18\x0f \x01(\v2\x17.llmgw.v1.AgentSettingsH\x00R\ragentSettings\x127\n" +
|
||||
"\vtrain_start\x18\x10 \x01(\v2\x14.llmgw.v1.TrainStartH\x00R\n" +
|
||||
"trainStart\x12:\n" +
|
||||
"\ftrain_cancel\x18\x11 \x01(\v2\x15.llmgw.v1.TrainCancelH\x00R\vtrainCancel\x129\n" +
|
||||
"\ftrain_cancel\x18\x11 \x01(\v2\x15.llmgw.v1.TrainCancelH\x00R\vtrainCancel\x12'\n" +
|
||||
"\x05slurm\x18\x12 \x01(\v2\x0f.llmgw.v1.SlurmH\x00R\x05slurm\x129\n" +
|
||||
"\x05trace\x18\f \x03(\v2#.llmgw.v1.ControlMessage.TraceEntryR\x05trace\x1a8\n" +
|
||||
"\n" +
|
||||
"TraceEntry\x12\x10\n" +
|
||||
@@ -5324,7 +5844,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\x1aA\n" +
|
||||
"\x13EngineVersionsEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\xdd\x01\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\x92\x02\n" +
|
||||
"\x04Host\x12\x13\n" +
|
||||
"\x05os_id\x18\x01 \x01(\tR\x04osId\x12\x1d\n" +
|
||||
"\n" +
|
||||
@@ -5334,7 +5854,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x04arch\x18\x04 \x01(\tR\x04arch\x12\x1b\n" +
|
||||
"\tcpu_flags\x18\x05 \x03(\tR\bcpuFlags\x12#\n" +
|
||||
"\rnvidia_driver\x18\x06 \x01(\tR\fnvidiaDriver\x12*\n" +
|
||||
"\x11mem_bandwidth_gbs\x18\a \x01(\x01R\x0fmemBandwidthGbs\"u\n" +
|
||||
"\x11mem_bandwidth_gbs\x18\a \x01(\x01R\x0fmemBandwidthGbs\x123\n" +
|
||||
"\x16mem_bandwidth_core_gbs\x18\b \x01(\x01R\x13memBandwidthCoreGbs\"u\n" +
|
||||
"\aStorage\x12\x12\n" +
|
||||
"\x04path\x18\x01 \x01(\tR\x04path\x12\x1d\n" +
|
||||
"\n" +
|
||||
@@ -5520,7 +6041,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x06status\x18\x03 \x01(\tR\x06status\x12!\n" +
|
||||
"\fcluster_uuid\x18\x04 \x01(\tR\vclusterUuid\x12\x1b\n" +
|
||||
"\tclique_id\x18\x05 \x01(\tR\bcliqueId\x12\x1c\n" +
|
||||
"\tbandwidth\x18\x06 \x01(\tR\tbandwidth\"\xd5\a\n" +
|
||||
"\tbandwidth\x18\x06 \x01(\tR\tbandwidth\"\x8a\b\n" +
|
||||
"\tHeartbeat\x12\x17\n" +
|
||||
"\anode_id\x18\x01 \x01(\tR\x06nodeId\x12+\n" +
|
||||
"\x12gpu_memory_used_mb\x18\x02 \x03(\x05R\x0fgpuMemoryUsedMb\x12!\n" +
|
||||
@@ -5543,7 +6064,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x04imex\x18\x11 \x01(\v2\x0e.llmgw.v1.IMEXR\x04imex\x12\x1b\n" +
|
||||
"\tcpu_count\x18\x12 \x01(\x05R\bcpuCount\x121\n" +
|
||||
"\x15gpu_foreign_memory_mb\x18\x13 \x03(\x05R\x12gpuForeignMemoryMb\x12*\n" +
|
||||
"\x11mem_bandwidth_gbs\x18\x14 \x01(\x01R\x0fmemBandwidthGbs\x1aA\n" +
|
||||
"\x11mem_bandwidth_gbs\x18\x14 \x01(\x01R\x0fmemBandwidthGbs\x123\n" +
|
||||
"\x16mem_bandwidth_core_gbs\x18\x15 \x01(\x01R\x13memBandwidthCoreGbs\x1aA\n" +
|
||||
"\x13EngineVersionsEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\";\n" +
|
||||
@@ -5732,7 +6254,59 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x04kind\x18\x02 \x01(\tR\x04kind\x12\x10\n" +
|
||||
"\x03seq\x18\x03 \x01(\x05R\x03seq\x12\x12\n" +
|
||||
"\x04data\x18\x04 \x01(\fR\x04data\x12\x12\n" +
|
||||
"\x04last\x18\x05 \x01(\bR\x04last2O\n" +
|
||||
"\x04last\x18\x05 \x01(\bR\x04last\"\xc9\x03\n" +
|
||||
"\x05Slurm\x12\x1e\n" +
|
||||
"\n" +
|
||||
"controller\x18\x01 \x01(\bR\n" +
|
||||
"controller\x12\x12\n" +
|
||||
"\x04node\x18\x02 \x01(\bR\x04node\x12\x12\n" +
|
||||
"\x04name\x18\x03 \x01(\tR\x04name\x12\x1f\n" +
|
||||
"\vconf_server\x18\x04 \x01(\tR\n" +
|
||||
"confServer\x12\x1b\n" +
|
||||
"\tmunge_key\x18\x05 \x01(\fR\bmungeKey\x120\n" +
|
||||
"\x05files\x18\x06 \x03(\v2\x1a.llmgw.v1.Slurm.FilesEntryR\x05files\x12\x12\n" +
|
||||
"\x04lent\x18\a \x01(\bR\x04lent\x12-\n" +
|
||||
"\x06shared\x18\b \x01(\v2\x15.llmgw.v1.SlurmSharedR\x06shared\x12!\n" +
|
||||
"\fmanage_users\x18\t \x01(\bR\vmanageUsers\x12)\n" +
|
||||
"\x05users\x18\n" +
|
||||
" \x03(\v2\x13.llmgw.v1.SlurmUserR\x05users\x12\x1d\n" +
|
||||
"\n" +
|
||||
"make_homes\x18\v \x01(\bR\tmakeHomes\x12\x1e\n" +
|
||||
"\n" +
|
||||
"containers\x18\f \x01(\bR\n" +
|
||||
"containers\x1a8\n" +
|
||||
"\n" +
|
||||
"FilesEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"c\n" +
|
||||
"\vSlurmShared\x12\x12\n" +
|
||||
"\x04path\x18\x01 \x01(\tR\x04path\x12\x10\n" +
|
||||
"\x03nfs\x18\x02 \x01(\tR\x03nfs\x12\x14\n" +
|
||||
"\x05serve\x18\x03 \x01(\bR\x05serve\x12\x18\n" +
|
||||
"\aclients\x18\x04 \x03(\tR\aclients\"v\n" +
|
||||
"\tSlurmUser\x12\x14\n" +
|
||||
"\x05login\x18\x01 \x01(\tR\x05login\x12\x10\n" +
|
||||
"\x03uid\x18\x02 \x01(\rR\x03uid\x12\x12\n" +
|
||||
"\x04name\x18\x03 \x01(\tR\x04name\x12\x12\n" +
|
||||
"\x04home\x18\x04 \x01(\tR\x04home\x12\x19\n" +
|
||||
"\bssh_keys\x18\x05 \x03(\tR\asshKeys\"\x8b\x03\n" +
|
||||
"\vSlurmStatus\x12\x18\n" +
|
||||
"\aversion\x18\x01 \x01(\tR\aversion\x12\x1a\n" +
|
||||
"\bhardware\x18\x02 \x01(\tR\bhardware\x12%\n" +
|
||||
"\x0eslurmd_running\x18\x03 \x01(\bR\rslurmdRunning\x12+\n" +
|
||||
"\x11slurmctld_running\x18\x04 \x01(\bR\x10slurmctldRunning\x12\x14\n" +
|
||||
"\x05state\x18\x05 \x01(\tR\x05state\x12\x16\n" +
|
||||
"\x06reason\x18\x06 \x01(\tR\x06reason\x12\x12\n" +
|
||||
"\x04jobs\x18\a \x01(\x05R\x04jobs\x12\x14\n" +
|
||||
"\x05error\x18\b \x01(\tR\x05error\x12!\n" +
|
||||
"\ffiles_sha256\x18\t \x01(\tR\vfilesSha256\x12)\n" +
|
||||
"\x10slurmdbd_running\x18\n" +
|
||||
" \x01(\bR\x0fslurmdbdRunning\x12\x16\n" +
|
||||
"\x06shared\x18\v \x01(\tR\x06shared\x12\x14\n" +
|
||||
"\x05users\x18\f \x01(\x05R\x05users\x12\x1e\n" +
|
||||
"\n" +
|
||||
"containers\x18\r \x01(\tR\n" +
|
||||
"containers2O\n" +
|
||||
"\fAgentService\x12?\n" +
|
||||
"\aConnect\x12\x16.llmgw.v1.AgentMessage\x1a\x18.llmgw.v1.ControlMessage(\x010\x01B\x93\x01\n" +
|
||||
"\fcom.llmgw.v1B\n" +
|
||||
@@ -5751,7 +6325,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
|
||||
}
|
||||
|
||||
var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
|
||||
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 56)
|
||||
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 61)
|
||||
var file_llmgw_v1_agent_proto_goTypes = []any{
|
||||
(ShardStatus_State)(0), // 0: llmgw.v1.ShardStatus.State
|
||||
(*AgentMessage)(nil), // 1: llmgw.v1.AgentMessage
|
||||
@@ -5802,14 +6376,19 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
|
||||
(*TrainCancel)(nil), // 46: llmgw.v1.TrainCancel
|
||||
(*TrainStatus)(nil), // 47: llmgw.v1.TrainStatus
|
||||
(*TrainArtifact)(nil), // 48: llmgw.v1.TrainArtifact
|
||||
nil, // 49: llmgw.v1.ControlMessage.TraceEntry
|
||||
nil, // 50: llmgw.v1.Register.LabelsEntry
|
||||
nil, // 51: llmgw.v1.Register.EngineVersionsEntry
|
||||
nil, // 52: llmgw.v1.GPU.NicPathsEntry
|
||||
nil, // 53: llmgw.v1.IBPort.CountersEntry
|
||||
nil, // 54: llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
nil, // 55: llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
nil, // 56: llmgw.v1.EngineProfile.EnvEntry
|
||||
(*Slurm)(nil), // 49: llmgw.v1.Slurm
|
||||
(*SlurmShared)(nil), // 50: llmgw.v1.SlurmShared
|
||||
(*SlurmUser)(nil), // 51: llmgw.v1.SlurmUser
|
||||
(*SlurmStatus)(nil), // 52: llmgw.v1.SlurmStatus
|
||||
nil, // 53: llmgw.v1.ControlMessage.TraceEntry
|
||||
nil, // 54: llmgw.v1.Register.LabelsEntry
|
||||
nil, // 55: llmgw.v1.Register.EngineVersionsEntry
|
||||
nil, // 56: llmgw.v1.GPU.NicPathsEntry
|
||||
nil, // 57: llmgw.v1.IBPort.CountersEntry
|
||||
nil, // 58: llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
nil, // 59: llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
nil, // 60: llmgw.v1.EngineProfile.EnvEntry
|
||||
nil, // 61: llmgw.v1.Slurm.FilesEntry
|
||||
}
|
||||
var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
8, // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
|
||||
@@ -5822,64 +6401,69 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
2, // 7: llmgw.v1.AgentMessage.kv_events:type_name -> llmgw.v1.KVEvents
|
||||
47, // 8: llmgw.v1.AgentMessage.train_status:type_name -> llmgw.v1.TrainStatus
|
||||
48, // 9: llmgw.v1.AgentMessage.train_artifact:type_name -> llmgw.v1.TrainArtifact
|
||||
30, // 10: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
|
||||
33, // 11: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
|
||||
34, // 12: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
|
||||
35, // 13: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
|
||||
32, // 14: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
|
||||
24, // 15: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
|
||||
36, // 16: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
|
||||
38, // 17: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
|
||||
40, // 18: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
|
||||
42, // 19: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
|
||||
7, // 20: llmgw.v1.ControlMessage.control_planes:type_name -> llmgw.v1.ControlPlanes
|
||||
5, // 21: llmgw.v1.ControlMessage.firewall:type_name -> llmgw.v1.Firewall
|
||||
4, // 22: llmgw.v1.ControlMessage.restart_agent:type_name -> llmgw.v1.RestartAgent
|
||||
6, // 23: llmgw.v1.ControlMessage.agent_settings:type_name -> llmgw.v1.AgentSettings
|
||||
45, // 24: llmgw.v1.ControlMessage.train_start:type_name -> llmgw.v1.TrainStart
|
||||
46, // 25: llmgw.v1.ControlMessage.train_cancel:type_name -> llmgw.v1.TrainCancel
|
||||
49, // 26: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
|
||||
50, // 27: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
|
||||
12, // 28: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
|
||||
14, // 29: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
51, // 30: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
|
||||
11, // 31: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
|
||||
15, // 32: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
|
||||
10, // 33: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
|
||||
9, // 34: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
|
||||
21, // 35: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
|
||||
0, // 36: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
27, // 37: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
|
||||
52, // 38: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
|
||||
13, // 39: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
|
||||
18, // 40: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
|
||||
17, // 41: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
|
||||
53, // 42: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
|
||||
20, // 43: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
|
||||
10, // 44: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
|
||||
15, // 45: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
|
||||
54, // 46: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
26, // 47: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
|
||||
16, // 48: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
|
||||
19, // 49: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
|
||||
14, // 50: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
22, // 51: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
|
||||
21, // 52: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
|
||||
25, // 53: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
|
||||
55, // 54: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
0, // 55: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
29, // 56: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
|
||||
27, // 57: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
|
||||
31, // 58: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
|
||||
56, // 59: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
|
||||
44, // 60: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
|
||||
1, // 61: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
|
||||
3, // 62: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
|
||||
62, // [62:63] is the sub-list for method output_type
|
||||
61, // [61:62] is the sub-list for method input_type
|
||||
61, // [61:61] is the sub-list for extension type_name
|
||||
61, // [61:61] is the sub-list for extension extendee
|
||||
0, // [0:61] is the sub-list for field type_name
|
||||
52, // 10: llmgw.v1.AgentMessage.slurm_status:type_name -> llmgw.v1.SlurmStatus
|
||||
30, // 11: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
|
||||
33, // 12: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
|
||||
34, // 13: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
|
||||
35, // 14: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
|
||||
32, // 15: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
|
||||
24, // 16: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
|
||||
36, // 17: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
|
||||
38, // 18: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
|
||||
40, // 19: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
|
||||
42, // 20: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
|
||||
7, // 21: llmgw.v1.ControlMessage.control_planes:type_name -> llmgw.v1.ControlPlanes
|
||||
5, // 22: llmgw.v1.ControlMessage.firewall:type_name -> llmgw.v1.Firewall
|
||||
4, // 23: llmgw.v1.ControlMessage.restart_agent:type_name -> llmgw.v1.RestartAgent
|
||||
6, // 24: llmgw.v1.ControlMessage.agent_settings:type_name -> llmgw.v1.AgentSettings
|
||||
45, // 25: llmgw.v1.ControlMessage.train_start:type_name -> llmgw.v1.TrainStart
|
||||
46, // 26: llmgw.v1.ControlMessage.train_cancel:type_name -> llmgw.v1.TrainCancel
|
||||
49, // 27: llmgw.v1.ControlMessage.slurm:type_name -> llmgw.v1.Slurm
|
||||
53, // 28: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
|
||||
54, // 29: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
|
||||
12, // 30: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
|
||||
14, // 31: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
55, // 32: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
|
||||
11, // 33: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
|
||||
15, // 34: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
|
||||
10, // 35: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
|
||||
9, // 36: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
|
||||
21, // 37: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
|
||||
0, // 38: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
27, // 39: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
|
||||
56, // 40: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
|
||||
13, // 41: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
|
||||
18, // 42: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
|
||||
17, // 43: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
|
||||
57, // 44: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
|
||||
20, // 45: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
|
||||
10, // 46: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
|
||||
15, // 47: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
|
||||
58, // 48: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
26, // 49: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
|
||||
16, // 50: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
|
||||
19, // 51: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
|
||||
14, // 52: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
22, // 53: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
|
||||
21, // 54: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
|
||||
25, // 55: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
|
||||
59, // 56: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
0, // 57: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
29, // 58: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
|
||||
27, // 59: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
|
||||
31, // 60: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
|
||||
60, // 61: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
|
||||
44, // 62: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
|
||||
61, // 63: llmgw.v1.Slurm.files:type_name -> llmgw.v1.Slurm.FilesEntry
|
||||
50, // 64: llmgw.v1.Slurm.shared:type_name -> llmgw.v1.SlurmShared
|
||||
51, // 65: llmgw.v1.Slurm.users:type_name -> llmgw.v1.SlurmUser
|
||||
1, // 66: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
|
||||
3, // 67: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
|
||||
67, // [67:68] is the sub-list for method output_type
|
||||
66, // [66:67] is the sub-list for method input_type
|
||||
66, // [66:66] is the sub-list for extension type_name
|
||||
66, // [66:66] is the sub-list for extension extendee
|
||||
0, // [0:66] is the sub-list for field type_name
|
||||
}
|
||||
|
||||
func init() { file_llmgw_v1_agent_proto_init() }
|
||||
@@ -5898,6 +6482,7 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
(*AgentMessage_KvEvents)(nil),
|
||||
(*AgentMessage_TrainStatus)(nil),
|
||||
(*AgentMessage_TrainArtifact)(nil),
|
||||
(*AgentMessage_SlurmStatus)(nil),
|
||||
}
|
||||
file_llmgw_v1_agent_proto_msgTypes[2].OneofWrappers = []any{
|
||||
(*ControlMessage_Start)(nil),
|
||||
@@ -5916,6 +6501,7 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
(*ControlMessage_AgentSettings)(nil),
|
||||
(*ControlMessage_TrainStart)(nil),
|
||||
(*ControlMessage_TrainCancel)(nil),
|
||||
(*ControlMessage_Slurm)(nil),
|
||||
}
|
||||
type x struct{}
|
||||
out := protoimpl.TypeBuilder{
|
||||
@@ -5923,7 +6509,7 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
|
||||
RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
|
||||
NumEnums: 1,
|
||||
NumMessages: 56,
|
||||
NumMessages: 61,
|
||||
NumExtensions: 0,
|
||||
NumServices: 1,
|
||||
},
|
||||
|
||||
@@ -30,6 +30,7 @@ message AgentMessage {
|
||||
KVEvents kv_events = 8;
|
||||
TrainStatus train_status = 9;
|
||||
TrainArtifact train_artifact = 10;
|
||||
SlurmStatus slurm_status = 11;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -78,6 +79,7 @@ message ControlMessage {
|
||||
AgentSettings agent_settings = 15;
|
||||
TrainStart train_start = 16;
|
||||
TrainCancel train_cancel = 17;
|
||||
Slurm slurm = 18;
|
||||
}
|
||||
// trace is the W3C trace context (traceparent, and tracestate when there
|
||||
// is one) of the control plane's span that sent this command, so what the
|
||||
@@ -241,6 +243,10 @@ message Host {
|
||||
// mem_bandwidth_gbs is how fast the memory reads, in GB/s, measured when
|
||||
// the agent started: what decides a model's speed on the CPU. 0 unknown.
|
||||
double mem_bandwidth_gbs = 7;
|
||||
// mem_bandwidth_core_gbs is what one core reads alone, in GB/s: a replica
|
||||
// given a few threads of a large machine reads at a few times this, not at
|
||||
// the whole machine's rate. 0 unknown.
|
||||
double mem_bandwidth_core_gbs = 8;
|
||||
}
|
||||
|
||||
// Storage is one filesystem a node keeps weights on.
|
||||
@@ -618,6 +624,8 @@ message Heartbeat {
|
||||
// the agent has measured it — in the background after it starts, so a
|
||||
// registration usually has none yet. 0 means not measured.
|
||||
double mem_bandwidth_gbs = 20;
|
||||
// mem_bandwidth_core_gbs is one core's read rate, as in Host.
|
||||
double mem_bandwidth_core_gbs = 21;
|
||||
}
|
||||
|
||||
// Provision is the software a node should have: which engines, at which
|
||||
@@ -980,3 +988,99 @@ message TrainArtifact {
|
||||
bytes data = 4;
|
||||
bool last = 5;
|
||||
}
|
||||
|
||||
// Slurm is what this node is in the Slurm cluster run beside llmgw
|
||||
// (BACKLOG P): its controller, one of its nodes, both or neither. Sent when
|
||||
// an agent registers and whenever it changes; the agent installs what is
|
||||
// missing and keeps the daemons running as it says.
|
||||
message Slurm {
|
||||
// controller runs slurmctld here, on files.
|
||||
bool controller = 1;
|
||||
// node runs slurmd here, configless, fetching its files from conf_server.
|
||||
bool node = 2;
|
||||
// name is this node's NodeName in slurm.conf.
|
||||
string name = 3;
|
||||
// conf_server is slurmctld's host:port.
|
||||
string conf_server = 4;
|
||||
// munge_key is the key every machine of the cluster shares.
|
||||
bytes munge_key = 5;
|
||||
// files are the controller's, by name in /etc/slurm; empty for a node
|
||||
// that is not the controller.
|
||||
map<string, string> files = 6;
|
||||
// lent says Slurm may run jobs here now: resume the node if llmgw drained
|
||||
// it. Otherwise it is drained, its jobs left to finish.
|
||||
bool lent = 7;
|
||||
// shared is the file system every machine of the Slurm cluster sees at
|
||||
// the same path; unset or with no path, none.
|
||||
SlurmShared shared = 8;
|
||||
// manage_users has llmgw keep the Linux users below on this machine,
|
||||
// and take away those it made that are no longer among them.
|
||||
bool manage_users = 9;
|
||||
repeated SlurmUser users = 10;
|
||||
// make_homes has this machine make the users' home directories: the one
|
||||
// that exports the shared path, else the controller.
|
||||
bool make_homes = 11;
|
||||
// containers has the machine install pyxis and enroot, so jobs may run
|
||||
// in containers (srun --container-image), and set them up.
|
||||
bool containers = 12;
|
||||
}
|
||||
|
||||
// SlurmShared is where the Slurm cluster's shared files come from.
|
||||
message SlurmShared {
|
||||
// path is where every machine sees them, as /shared.
|
||||
string path = 1;
|
||||
// nfs is an export, host:/path, to mount at path; empty when this
|
||||
// machine exports path itself (serve) or the site mounts it.
|
||||
string nfs = 2;
|
||||
// serve has this machine export path over NFS to clients.
|
||||
bool serve = 3;
|
||||
// clients are the addresses the export answers.
|
||||
repeated string clients = 4;
|
||||
}
|
||||
|
||||
// SlurmUser is a Linux user that runs jobs, the same on every machine.
|
||||
message SlurmUser {
|
||||
string login = 1;
|
||||
// uid is also the user's own group's gid.
|
||||
uint32 uid = 2;
|
||||
// name is the account's, for the GECOS field.
|
||||
string name = 3;
|
||||
string home = 4;
|
||||
// ssh_keys are OpenSSH public keys, one a line of authorized_keys.
|
||||
repeated string ssh_keys = 5;
|
||||
}
|
||||
|
||||
// SlurmStatus is what this node's Slurm daemons are doing, sent when it
|
||||
// changes and every few minutes while the node has a part in Slurm.
|
||||
message SlurmStatus {
|
||||
// version is Slurm's, as "23.11.4"; empty when it is not installed.
|
||||
string version = 1;
|
||||
// hardware is `slurmd -C` on a node, which slurm.conf must match.
|
||||
string hardware = 2;
|
||||
bool slurmd_running = 3;
|
||||
bool slurmctld_running = 4;
|
||||
// state and reason are the controller's for this node, from scontrol.
|
||||
string state = 5;
|
||||
string reason = 6;
|
||||
// jobs is how many job steps run on this node now, counted from their
|
||||
// slurmstepd so a controller that is down does not hide them; -1 when the
|
||||
// agent cannot tell.
|
||||
int32 jobs = 7;
|
||||
// error is the last thing that went wrong keeping Slurm, empty when
|
||||
// nothing did.
|
||||
string error = 8;
|
||||
// files_sha256 is of the controller's files as written, so the leader
|
||||
// knows they arrived.
|
||||
string files_sha256 = 9;
|
||||
// slurmdbd_running is the accounting daemon on the controller, which keeps
|
||||
// the jobs' records in a MariaDB beside it.
|
||||
bool slurmdbd_running = 10;
|
||||
// shared is the shared path as this machine has it: "mounted",
|
||||
// "exported", "missing", or empty when there is none.
|
||||
string shared = 11;
|
||||
// users is how many of llmgw's users this machine has.
|
||||
int32 users = 12;
|
||||
// containers is what this machine has for jobs in containers, as
|
||||
// "pyxis 0.24.0, enroot 4.2.1"; empty while it has not got both.
|
||||
string containers = 13;
|
||||
}
|
||||
+409
-1
@@ -116,6 +116,9 @@
|
||||
{
|
||||
"name": "settings"
|
||||
},
|
||||
{
|
||||
"name": "slurm"
|
||||
},
|
||||
{
|
||||
"name": "summary"
|
||||
},
|
||||
@@ -3227,6 +3230,58 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/models/{name}/replicas": {
|
||||
"post": {
|
||||
"operationId": "postModelsNameReplicas",
|
||||
"summary": "How many replicas run on each profile",
|
||||
"description": "Seven on cards and four on CPUs, under the model's name. A profile not named keeps its count; cpu adds a profile on CPUs to a llama.cpp model that has none. Its scaling becomes the sum.",
|
||||
"tags": [
|
||||
"models"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "name",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "The model's name in the catalog.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/ProfileReplicasRequest"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/catalog.Model"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/models/{name}/resources": {
|
||||
"post": {
|
||||
"operationId": "postModelsNameResources",
|
||||
@@ -5026,6 +5081,117 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/slurm": {
|
||||
"get": {
|
||||
"operationId": "getSlurm",
|
||||
"summary": "The Slurm cluster beside llmgw, and where each of its nodes is",
|
||||
"description": "From the cluster's settings (slurm.*) and what each node's agent last said. A node's phase is llmgw (drained in Slurm, no job left), lending (its replicas moving off), lent, reclaiming (drained in Slurm, jobs still running) or unknown (its agent has not said).",
|
||||
"tags": [
|
||||
"slurm"
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/slurmsvc.Status"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/slurm/users": {
|
||||
"get": {
|
||||
"operationId": "getSlurmUsers",
|
||||
"summary": "The Linux users made from llmgw's accounts for the Slurm cluster",
|
||||
"description": "Every active account, with its login, UID and SSH keys; an account is given its login and UID the first time it is listed. The machines have them when slurm.users is \"llmgw\".",
|
||||
"tags": [
|
||||
"slurm"
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/slurmsvc.User"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/slurm/users/{user}/ssh-keys": {
|
||||
"put": {
|
||||
"operationId": "putSlurmUsersUserSshKeys",
|
||||
"summary": "Replace an account's SSH public keys",
|
||||
"description": "{user} is an ID or e-mail. keys is OpenSSH public keys, one a line, without options; empty takes them all away. The machines have them within a minute.",
|
||||
"tags": [
|
||||
"slurm"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "user",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "The person's ID or email.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/SlurmKeysRequest"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/slurmsvc.User"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/summary": {
|
||||
"get": {
|
||||
"operationId": "getSummary",
|
||||
@@ -6182,7 +6348,7 @@
|
||||
},
|
||||
"CollectResult": {
|
||||
"type": "object",
|
||||
"description": "CollectResult is what was deleted on one node, or why nothing was.",
|
||||
"description": "CollectResult is what was deleted on one node, or why nothing was: what the node no longer lists once asked, not what it was asked to delete.",
|
||||
"properties": {
|
||||
"node": {
|
||||
"type": "string"
|
||||
@@ -6194,6 +6360,14 @@
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"left": {
|
||||
"type": "integer",
|
||||
"description": "Left and LeftBytes are what the node still lists: refused, or not gone by the time the sweep stopped waiting. Its agent's log says why."
|
||||
},
|
||||
"left_bytes": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"error": {
|
||||
"type": "string"
|
||||
}
|
||||
@@ -8103,6 +8277,13 @@
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"profile_replicas": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "integer"
|
||||
},
|
||||
"description": "ProfileReplicas is how many replicas run on each profile, for a model whose replicas are counted per profile; absent otherwise."
|
||||
},
|
||||
"memory_gb": {
|
||||
"type": "integer",
|
||||
"description": "MemoryGB and CPUCores are what one replica reserves, from its first profile; MeasuredMemoryGB is the most one of its replicas uses now, 0 when none is measured."
|
||||
@@ -8305,6 +8486,10 @@
|
||||
"type": "number",
|
||||
"description": "MemBandwidthGBs is how fast its memory reads, in GB/s, measured when its agent started; 0 when not reported. A model's speed on the CPU."
|
||||
},
|
||||
"mem_bandwidth_core_gbs": {
|
||||
"type": "number",
|
||||
"description": "MemBandwidthCoreGBs is what one core reads alone, in GB/s; 0 when not reported. A CPU replica's threads read at about this each."
|
||||
},
|
||||
"engines": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
@@ -8661,6 +8846,21 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"ProfileReplicasRequest": {
|
||||
"type": "object",
|
||||
"description": "ProfileReplicasRequest counts a model's replicas per profile: profile name to replicas, and CPU, for a llama.cpp model with no profile on CPUs, the one to add.",
|
||||
"properties": {
|
||||
"profiles": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"cpu": {
|
||||
"$ref": "#/components/schemas/modelsvc.CPUProfile"
|
||||
}
|
||||
}
|
||||
},
|
||||
"ProjectView": {
|
||||
"type": "object",
|
||||
"description": "ProjectView is a project, with how many people and keys are in it.",
|
||||
@@ -9284,6 +9484,15 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"SlurmKeysRequest": {
|
||||
"type": "object",
|
||||
"description": "SlurmKeysRequest is a person's SSH public keys, one a line, as authorized_keys has them; empty takes them all away.",
|
||||
"properties": {
|
||||
"keys": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"StoreMember": {
|
||||
"type": "object",
|
||||
"description": "StoreMember is one member of a replicated store, as its Patroni reports it.",
|
||||
@@ -9386,6 +9595,10 @@
|
||||
"guard_off": {
|
||||
"type": "boolean",
|
||||
"description": "GuardOff says the guardrails are turned off (guard.enabled: false): no request is checked, whatever the policies say."
|
||||
},
|
||||
"no_alerts": {
|
||||
"type": "boolean",
|
||||
"description": "NoAlerts says the cluster's settings name no webhook: a model parked, a node lost or a judge down is told nobody."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -10506,6 +10719,10 @@
|
||||
"type": "integer",
|
||||
"description": "Priority orders profiles during placement; highest that fits wins."
|
||||
},
|
||||
"replicas": {
|
||||
"type": "integer",
|
||||
"description": "Replicas, set on any of a model's profiles, is how many of its replicas run on this one: seven on cards and four on CPUs, all answering under the model's name, rather than each replica on the best profile that fits. The model's scaling is then the sum, min and max alike; a profile left at zero runs none."
|
||||
},
|
||||
"startup_timeout": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
@@ -10664,6 +10881,24 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"config.SharedStorage": {
|
||||
"type": "object",
|
||||
"description": "SharedStorage is where the Slurm cluster's shared files come from: the site's NFS export (NFS), an NFS export llmgw keeps on one of its nodes (Server), or, with neither, a file system the site mounts itself (Lustre, BeeGFS, GPFS), which llmgw only looks for.",
|
||||
"properties": {
|
||||
"Path": {
|
||||
"type": "string",
|
||||
"description": "Path is where every machine sees it, as /shared."
|
||||
},
|
||||
"NFS": {
|
||||
"type": "string",
|
||||
"description": "NFS is the site's export, host:/path, mounted at Path on every machine of the Slurm cluster."
|
||||
},
|
||||
"Server": {
|
||||
"type": "string",
|
||||
"description": "Server is the llmgw node, the controller or one of the nodes, that exports Path over NFS to the others: for a site with no storage of its own. One machine: not highly available, and no faster than its disk and network."
|
||||
}
|
||||
}
|
||||
},
|
||||
"control.Activity": {
|
||||
"type": "object",
|
||||
"description": "Activity is what a standby's router reports back: when it last sent each replica a request, and the on-demand models it was asked for. The leader evicts and loads by these, and without them a replica busy behind a standby looks idle to the one machine that decides.",
|
||||
@@ -11406,6 +11641,10 @@
|
||||
},
|
||||
"strata_size": {
|
||||
"type": "string"
|
||||
},
|
||||
"tokens_per_second": {
|
||||
"type": "number",
|
||||
"description": "TokensPerSecond is about how many tokens a second one reply gets this way, on the fastest host it fits; 0 when not known. On the CPU with the threads a replica is given by default: half the node's."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -11651,6 +11890,10 @@
|
||||
"context": {
|
||||
"type": "integer"
|
||||
},
|
||||
"trained": {
|
||||
"type": "integer",
|
||||
"description": "Trained is the context the model was trained for, the most a request may use; 0 when neither its header nor its config says."
|
||||
},
|
||||
"kv_bytes_per_token": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
@@ -11898,6 +12141,21 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"modelsvc.CPUProfile": {
|
||||
"type": "object",
|
||||
"description": "CPUProfile is a profile on CPUs to add to a llama.cpp model, for replicas counted per profile: what a replica of it reserves.",
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"memory_gb": {
|
||||
"type": "integer"
|
||||
},
|
||||
"cpu_cores": {
|
||||
"type": "integer"
|
||||
}
|
||||
}
|
||||
},
|
||||
"notify.Result": {
|
||||
"type": "object",
|
||||
"description": "Result is how one webhook answered a test.",
|
||||
@@ -12065,6 +12323,156 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"slurmsvc.NodeStatus": {
|
||||
"type": "object",
|
||||
"description": "NodeStatus is one node as the service sees it.",
|
||||
"properties": {
|
||||
"node": {
|
||||
"type": "string"
|
||||
},
|
||||
"phase": {
|
||||
"type": "string"
|
||||
},
|
||||
"replicas": {
|
||||
"type": "integer"
|
||||
},
|
||||
"stops_by": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "StopsBy is, while it is being lent, when a replica still on it is stopped even with nowhere else to go: the drain's grace after the lend began. Zero when not known (a leader that took over meanwhile)."
|
||||
},
|
||||
"slurm": {
|
||||
"$ref": "#/components/schemas/slurmsvc.Reported"
|
||||
},
|
||||
"connected": {
|
||||
"type": "boolean"
|
||||
}
|
||||
}
|
||||
},
|
||||
"slurmsvc.Reported": {
|
||||
"type": "object",
|
||||
"description": "Reported is the agent's SlurmStatus, for the admin API.",
|
||||
"properties": {
|
||||
"version": {
|
||||
"type": "string"
|
||||
},
|
||||
"hardware": {
|
||||
"type": "string"
|
||||
},
|
||||
"slurmd_running": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"slurmctld_running": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"slurmdbd_running": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"shared": {
|
||||
"type": "string",
|
||||
"description": "Shared is the shared path here: \"mounted\", \"exported\", \"missing\"."
|
||||
},
|
||||
"containers": {
|
||||
"type": "string",
|
||||
"description": "Containers is what the machine has for jobs in containers, empty while not both of pyxis and enroot."
|
||||
},
|
||||
"users": {
|
||||
"type": "integer",
|
||||
"description": "Users is how many of llmgw's users the machine has."
|
||||
},
|
||||
"state": {
|
||||
"type": "string"
|
||||
},
|
||||
"reason": {
|
||||
"type": "string"
|
||||
},
|
||||
"jobs": {
|
||||
"type": "integer"
|
||||
},
|
||||
"error": {
|
||||
"type": "string"
|
||||
},
|
||||
"files_sha256": {
|
||||
"type": "string"
|
||||
},
|
||||
"at": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
}
|
||||
}
|
||||
},
|
||||
"slurmsvc.Status": {
|
||||
"type": "object",
|
||||
"description": "Status is the whole Slurm cluster, for `ctl slurm status`.",
|
||||
"properties": {
|
||||
"enabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"cluster": {
|
||||
"type": "string"
|
||||
},
|
||||
"controller": {
|
||||
"type": "string"
|
||||
},
|
||||
"partition": {
|
||||
"type": "string"
|
||||
},
|
||||
"controller_status": {
|
||||
"$ref": "#/components/schemas/slurmsvc.Reported"
|
||||
},
|
||||
"nodes": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/slurmsvc.NodeStatus"
|
||||
}
|
||||
},
|
||||
"waiting": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Waiting are nodes left out of slurm.conf until they report their hardware."
|
||||
},
|
||||
"shared": {
|
||||
"$ref": "#/components/schemas/config.SharedStorage",
|
||||
"description": "Shared is where the shared files are and come from; Users who keeps the users, and how many llmgw has."
|
||||
},
|
||||
"users": {
|
||||
"type": "string"
|
||||
},
|
||||
"user_count": {
|
||||
"type": "integer"
|
||||
},
|
||||
"containers": {
|
||||
"type": "string",
|
||||
"description": "Containers is whether jobs may run in containers: \"on\" once every machine has pyxis and enroot, \"installing\" until then, empty when slurm.containers is off."
|
||||
}
|
||||
}
|
||||
},
|
||||
"slurmsvc.User": {
|
||||
"type": "object",
|
||||
"description": "User is a Linux user made from an llmgw account.",
|
||||
"properties": {
|
||||
"login": {
|
||||
"type": "string"
|
||||
},
|
||||
"uid": {
|
||||
"type": "integer"
|
||||
},
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"email": {
|
||||
"type": "string"
|
||||
},
|
||||
"ssh_keys": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"store.AuditEvent": {
|
||||
"type": "object",
|
||||
"description": "AuditEvent is one change: who made it, through what, to what, and how it ended. Detail never holds a secret; the writer takes them out.",
|
||||
|
||||
+17
-2
@@ -67,7 +67,8 @@ or with the mirror unticked, is said in the form and nothing starts.
|
||||
## Watch
|
||||
|
||||
**Overview** is `llmgw ctl status`: nodes, GPUs by class, fabrics, replicas,
|
||||
models, the store's members, and anything parked, unplaced or idle.
|
||||
models, the store's members, and anything parked, unplaced or idle. With no
|
||||
webhook in the cluster's settings it says the alerts go nowhere.
|
||||
|
||||
**Monitor** draws the cluster the way btop draws a machine: CPU, memory,
|
||||
GPU busy and GPU memory, requests and tokens a second, time to first token,
|
||||
@@ -207,6 +208,19 @@ starting — and whether it ended done, failed, or waiting for room on busy
|
||||
cards. The entry shows how many are running. Enter on one shows its steps.
|
||||
`llmgw ctl jobs` is the same, and `llmgw ctl jobs ID` follows one.
|
||||
|
||||
**Slurm** is the Slurm cluster run beside llmgw, when there is one: its
|
||||
controller and whether slurmctld runs, and each node that runs slurmd —
|
||||
whose it is now (llmgw's, Slurm's, or on its way between them), its state in
|
||||
Slurm, the jobs running there. On a row, **Lend to Slurm…** moves the node's
|
||||
replicas off and then lets Slurm run jobs there; **Take back from Slurm…**
|
||||
starts no new job there and gives it back to llmgw when its jobs have ended.
|
||||
Both change `slurm.lent` in the cluster's settings, as `llmgw ctl slurm lend`
|
||||
and `reclaim` do. Turning Slurm on and choosing its nodes is `llmgw ctl slurm
|
||||
setup`, or the Slurm part of the first install. Above the nodes it says where
|
||||
the shared files come from and how many users llmgw made; the SHARED column
|
||||
is each node's (mounted, exported, missing). `llmgw ctl slurm shared` and
|
||||
`llmgw ctl slurm users` change them.
|
||||
|
||||
**Logs** shows a node's agent log, newest at the bottom, errors and warnings
|
||||
coloured.
|
||||
|
||||
@@ -220,7 +234,8 @@ and change it with the same commands as `llmgw users`, `llmgw keys` and
|
||||
dashboard, and what they belong to. **A** creates an account, already
|
||||
approved. On a row: approve one waiting, suspend or reinstate, give or take
|
||||
the cluster dashboard, add to or take out of an organization, team or
|
||||
project, their limits, a new password (their sessions close), delete, and
|
||||
project (on an organization as a member, its manager, who sees its page, or
|
||||
its admin, who also changes it), their limits, a new password (their sessions close), delete, and
|
||||
**Forget them…** for a request to be forgotten (`llmgw users forget`).
|
||||
|
||||
**API keys** — every key, its project, owner, state and last use. **A**
|
||||
|
||||
+63
-2
@@ -166,13 +166,74 @@ is a flag too, and the plan shows the command with them: `--starter qwen3-8b`
|
||||
`--judge-fail open`, `--judge-categories`, `--judge-replicas`,
|
||||
`--judge-budget`, `--language tr`, `--signup-open`, `--default-rpm`,
|
||||
`--default-tpm`, `--max-key-ttl`, `--media` (pictures and 3D: ComfyUI on the
|
||||
nodes, and `presets.auto`), `--evict-after-idle` and `--min-lifetime`. They
|
||||
nodes, and `presets.auto`), `--slurm` (see below), `--evict-after-idle` and `--min-lifetime`. They
|
||||
go into `server.yaml` as init writes it, and become the cluster's first
|
||||
settings when it starts; a `server.yaml` already there is never overwritten,
|
||||
so run again they change nothing — the settings page does. An NVIDIA card with no working
|
||||
driver is reported rather than silently ignored; install the driver and run
|
||||
init again to serve on it.
|
||||
|
||||
**Slurm, for training and batch jobs.** `--slurm`, or the Slurm part on the
|
||||
choices screen, runs a Slurm cluster beside llmgw: slurmctld on this
|
||||
machine's node, slurmd on every node, those that join later too
|
||||
(`slurm.all_nodes`). The agents install munge and Slurm (22.05 or newer)
|
||||
from the machines' own package sources, and on this machine slurmdbd, which
|
||||
keeps the jobs' records (`sacct`), in a MariaDB installed beside it that
|
||||
answers this machine alone. Slurm keeps its records in MySQL or MariaDB
|
||||
only, not in the cluster's PostgreSQL; the database's password is made from
|
||||
the cluster's munge key, so nothing new is stored. A MariaDB the machine
|
||||
already had keeps its settings and only gets the database `slurm_acct_db`
|
||||
and its user. Back it up with the machine (`mariadb-dump slurm_acct_db`);
|
||||
records older than a year are purged. No node is Slurm's at first: each is
|
||||
drained in Slurm and serves models until `llmgw ctl slurm lend NODE` gives it
|
||||
to Slurm — its models move off first — and `llmgw ctl slurm reclaim NODE`
|
||||
takes it back once its jobs end. Jobs are submitted on this machine with
|
||||
`sbatch` and `srun`; `llmgw ctl slurm status` shows where each node is. It
|
||||
needs this machine to be a node, and its hostname to be a name Slurm takes.
|
||||
On a cluster installed without it, `llmgw ctl slurm setup --controller NODE
|
||||
--all-nodes` does the same later. Slurm's ports (6817–6819, and srun's 60001–63000) answer the cluster's
|
||||
networks only, as llmgw's own do; a firewall of the site's own between the
|
||||
nodes must let them through.
|
||||
|
||||
**Slurm's shared files and users.** Jobs need the same files on every
|
||||
machine: homes, datasets, output, checkpoints. With `--slurm` this machine
|
||||
exports `/shared` over NFS to every Slurm machine and they mount it
|
||||
(`slurm.shared`); NFS's port 2049 answers the cluster's networks only, the
|
||||
export the Slurm machines alone. That is one machine, not highly available
|
||||
and no faster than its disk and network: enough for a few nodes. A site with
|
||||
storage of its own uses it instead — `--slurm-shared HOST:/EXPORT` at
|
||||
install, or `llmgw ctl slurm shared --path /shared --nfs HOST:/EXPORT`
|
||||
later; Lustre, BeeGFS or GPFS the site mounts itself, and `llmgw ctl slurm
|
||||
shared --path /lustre/hpc` only has llmgw look that it is there. Every
|
||||
active llmgw account is a Linux user on every Slurm machine
|
||||
(`slurm.users: llmgw`): its login from its e-mail address, a UID from
|
||||
200000 up that is never given to anyone else, its home under
|
||||
`/shared/home`. People add their SSH public keys in the portal's
|
||||
**Profile** and log in to this machine to run `sbatch`; an account suspended
|
||||
or deleted loses its user, its files stay. On the other Slurm machines they
|
||||
are let in only while a job of theirs runs there, and their ssh session is
|
||||
put in that job, held to its cores and cards (pam_slurm_adopt, which the
|
||||
agents install); root and the machines' own users log in as before. A site whose users come from its
|
||||
directory (sssd against Active Directory or LDAP) passes `--slurm-users
|
||||
site`, or later `llmgw ctl slurm users off`.
|
||||
|
||||
Jobs may run in containers: `llmgw ctl slurm containers on` has the agents
|
||||
install enroot and pyxis on every Slurm machine (NVIDIA's releases, at
|
||||
versions llmgw pins and checks; pyxis is built against each machine's own
|
||||
Slurm) and, on GPU machines, nvidia-container-cli from NVIDIA's repository.
|
||||
Once every machine has them Slurm loads pyxis, and `ctl slurm status` says
|
||||
`containers: on`:
|
||||
|
||||
```sh
|
||||
srun --gres=gpu:1 --container-image=nvcr.io#nvidia/pytorch:25.01-py3 python train.py
|
||||
```
|
||||
|
||||
The shared path is in every container where it is on the host, and the job
|
||||
runs as its user, held to its cores and cards. Each node pulls the image
|
||||
from its registry for each job; for a large image used often, import it
|
||||
once (`enroot import -o /shared/images/pytorch.sqsh docker://nvcr.io#...`)
|
||||
and give that file to `--container-image`.
|
||||
|
||||
**Secure by default.** What init writes is the hardened configuration, not a
|
||||
demonstration one:
|
||||
|
||||
@@ -3145,7 +3206,7 @@ loaded at boot by `llmgw-firewall.service`):
|
||||
| Machine | Ports, for the cluster only |
|
||||
| --- | --- |
|
||||
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
|
||||
| node | engine ports (`control.port_range`, 20000–20999), 9100 |
|
||||
| node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6819 (slurmctld, slurmd, slurmdbd) and 60001–63000 (srun's `SrunPortRange`); exporting Slurm's shared files, 2049 (NFS) |
|
||||
| store member | 5432, 8008, 2379, 2380, 9100 |
|
||||
| mirror | 8081, 8082, 9100 |
|
||||
|
||||
|
||||
Binary file not shown.
+32
-11
@@ -192,22 +192,41 @@ auth:
|
||||
A personal key has no project, so it keeps no conversations or agent notes;
|
||||
those are kept per project.
|
||||
|
||||
### Organization managers
|
||||
### Organization managers and admins
|
||||
|
||||
A member of an organization can be made its manager:
|
||||
A member of an organization can be made its manager, who sees it, or its
|
||||
admin, who also changes it:
|
||||
|
||||
```
|
||||
llmgw users join boss@acme.com --org Acme --role manager
|
||||
llmgw users join lead@acme.com --org Acme --role admin
|
||||
```
|
||||
|
||||
or with the star on their organization membership on `/admin/users`. A manager
|
||||
gets an **Organization** link and a page at `/org`: usage over time by team,
|
||||
or with the role chosen on their organization membership on `/admin/users`.
|
||||
Both get an **Organization** link and a page at `/org`: usage over time by team,
|
||||
project or person, the teams and projects with their limits and spend, and every
|
||||
member. Only what was spent in that organization's projects is on it; what its
|
||||
people do elsewhere is not. The page is read-only — a manager is not an
|
||||
administrator of the cluster, and limits and memberships stay with those who
|
||||
are. An administrator following `/org` lands on the same organization's panel
|
||||
on the people page instead, where all of that can be changed.
|
||||
people do elsewhere is not.
|
||||
|
||||
A manager's page is read-only. An organization's admin also changes the
|
||||
organization from it:
|
||||
|
||||
- adds and renames its teams and projects, and removes an empty one;
|
||||
- adds a person by their sign-in address to the organization, a team or a
|
||||
project, and takes them out (their keys there are revoked with it);
|
||||
- makes a member of the organization a manager or an admin, or neither;
|
||||
- revokes a key spent in its projects.
|
||||
|
||||
What reaches past the organization stays with the cluster's administrators:
|
||||
its limits and budgets, approving or suspending an account (someone not yet
|
||||
approved cannot be added), the organization's own name, and anything else on
|
||||
the cluster. An admin cannot take away their own role. A group from single
|
||||
sign-on or Active Directory that puts someone in the organization does not, by
|
||||
saying so again at the next sign-in, take a manager's or admin's role away.
|
||||
|
||||
An administrator of the cluster following `/org` lands on the same
|
||||
organization's panel on the people page instead, where all of that can be
|
||||
changed.
|
||||
|
||||
`llmgw keys init` still works for provisioning scripts and is idempotent, but it
|
||||
will no longer move a team or project that already exists under another parent;
|
||||
@@ -517,8 +536,10 @@ included: `llmgw ctl audit`, or **Audit** in the console.
|
||||
suspends accounts from it; see [install.md](install.md#single-sign-on).
|
||||
Without them accounts are local, with local passwords.
|
||||
- **Staff roles are coarse.** `viewer`, `operator`, `billing` and `admin`
|
||||
(`llmgw users role`) decide what someone may see and do under /admin; an
|
||||
organization's manager can view it but not change it. Every change is in
|
||||
the audit log (`llmgw ctl audit`).
|
||||
(`llmgw users role`) decide what someone may see and do under /admin, and a
|
||||
page draws only the forms that person may send: a viewer sees no forms. An
|
||||
organization's manager can view it; its admin also changes its teams,
|
||||
projects, people and keys. Every change is in the audit log (`llmgw ctl
|
||||
audit`).
|
||||
- **Rate limits are per gateway instance.** With two instances each counts its own,
|
||||
so the effective limit doubles. v1 is single-instance by design.
|
||||
@@ -295,6 +295,7 @@ rather than repeated.
|
||||
| `keep_within` | `string` | [Profile](#profile) |
|
||||
| `weight` | `float64` | [Profile](#profile) |
|
||||
| `priority` | `int` | [Profile](#profile) |
|
||||
| `replicas` | `int` | [Profile](#profile) |
|
||||
| `startup_timeout` | `time.Duration` | [Profile](#profile) |
|
||||
| `kv_events` | `bool` | [Profile](#profile) |
|
||||
| `gpu` | `*GPUNeed` | [Profile](#profile) |
|
||||
@@ -375,6 +376,7 @@ rather than repeated.
|
||||
| `mail` | `mail.Config` | [Server](#server) |
|
||||
| `serving` | `ServingSettings` | [Server](#server) |
|
||||
| `firewall` | `FirewallSettings` | [Server](#server) |
|
||||
| `slurm` | `SlurmSettings` | [Server](#server) |
|
||||
| `presets` | `PresetSettings` | [Server](#server) |
|
||||
| `on_demand` | `OnDemandSettings` | [Server](#server) |
|
||||
| `agents` | `AgentSettings` | [Server](#server) |
|
||||
@@ -399,7 +401,20 @@ rather than repeated.
|
||||
| `action` | `string` | [Service](#service) |
|
||||
| `timeout` | `time.Duration` | [Service](#service) |
|
||||
| `stopped` | `bool` | [ServingSettings](#servingsettings) |
|
||||
| `path` | `string` | [SharedStorage](#sharedstorage) |
|
||||
| `nfs` | `string` | [SharedStorage](#sharedstorage) |
|
||||
| `server` | `string` | [SharedStorage](#sharedstorage) |
|
||||
| `bin` | `string` | [SimConfig](#simconfig) |
|
||||
| `enabled` | `bool` | [SlurmSettings](#slurmsettings) |
|
||||
| `cluster_name` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `controller` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `nodes` | `[]string` | [SlurmSettings](#slurmsettings) |
|
||||
| `all_nodes` | `bool` | [SlurmSettings](#slurmsettings) |
|
||||
| `lent` | `[]string` | [SlurmSettings](#slurmsettings) |
|
||||
| `partition` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `shared` | `SharedStorage` | [SlurmSettings](#slurmsettings) |
|
||||
| `users` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `containers` | `bool` | [SlurmSettings](#slurmsettings) |
|
||||
| `dir` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `python` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `engine` | `string` | [StrataConfig](#strataconfig) |
|
||||
@@ -2611,6 +2626,16 @@ rather than a guess (plan §6.2.1).
|
||||
|
||||
Priority orders profiles during placement; highest that fits wins.
|
||||
|
||||
### `replicas`
|
||||
|
||||
`int`
|
||||
|
||||
Replicas, set on any of a model's profiles, is how many of its
|
||||
replicas run on this one: seven on cards and four on CPUs, all
|
||||
answering under the model's name, rather than each replica on the
|
||||
best profile that fits. The model's scaling is then the sum, min and
|
||||
max alike; a profile left at zero runs none.
|
||||
|
||||
### `startup_timeout`
|
||||
|
||||
`time.Duration`
|
||||
@@ -3268,6 +3293,15 @@ Firewall turns llmgw's firewall on every machine on or off.
|
||||
|
||||
Takes a [FirewallSettings](#firewallsettings) block.
|
||||
|
||||
### `slurm`
|
||||
|
||||
`SlurmSettings`
|
||||
|
||||
Slurm is a Slurm cluster run beside llmgw on some of its nodes, and
|
||||
which of them it may use now.
|
||||
|
||||
Takes a [SlurmSettings](#slurmsettings) block.
|
||||
|
||||
### `presets`
|
||||
|
||||
`PresetSettings`
|
||||
@@ -3460,6 +3494,30 @@ running and every model's definition as it is: llmgw stop sets it,
|
||||
llmgw start clears it, and the models come back as their catalog
|
||||
entries ask. In force on every control plane within seconds.
|
||||
|
||||
## SharedStorage
|
||||
|
||||
### `path`
|
||||
|
||||
`string`
|
||||
|
||||
Path is where every machine sees it, as /shared.
|
||||
|
||||
### `nfs`
|
||||
|
||||
`string`
|
||||
|
||||
NFS is the site's export, host:/path, mounted at Path on every
|
||||
machine of the Slurm cluster.
|
||||
|
||||
### `server`
|
||||
|
||||
`string`
|
||||
|
||||
Server is the llmgw node, the controller or one of the nodes, that
|
||||
exports Path over NFS to the others: for a site with no storage of
|
||||
its own. One machine: not highly available, and no faster than its
|
||||
disk and network.
|
||||
|
||||
## SimConfig
|
||||
|
||||
### `bin`
|
||||
@@ -3469,6 +3527,88 @@ entries ask. In force on every control plane within seconds.
|
||||
Bin is the llmgw binary itself; the engine is one of its subcommands.
|
||||
Empty means whatever started this agent, which is right almost always.
|
||||
|
||||
## SlurmSettings
|
||||
|
||||
### `enabled`
|
||||
|
||||
`bool`
|
||||
|
||||
Enabled runs Slurm on the nodes below. False stops llmgw keeping it;
|
||||
what was installed is left installed, and every node is llmgw's.
|
||||
|
||||
### `cluster_name`
|
||||
|
||||
`string`
|
||||
|
||||
ClusterName is Slurm's ClusterName. Empty is "llmgw".
|
||||
|
||||
### `controller`
|
||||
|
||||
`string`
|
||||
|
||||
Controller is the llmgw node whose agent runs slurmctld. It needs no
|
||||
GPU, and need not be one of Nodes.
|
||||
|
||||
### `nodes`
|
||||
|
||||
`[]string`
|
||||
|
||||
Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained
|
||||
in Slurm, until it is also in Lent.
|
||||
|
||||
### `all_nodes`
|
||||
|
||||
`bool`
|
||||
|
||||
AllNodes runs slurmd on every llmgw node, those that join later too,
|
||||
besides any named in Nodes. Each is llmgw's until it is lent, as any.
|
||||
|
||||
### `lent`
|
||||
|
||||
`[]string`
|
||||
|
||||
Lent are the nodes of Nodes Slurm may run jobs on now. A node put here
|
||||
is drained of llmgw's replicas first, as `ctl drain` does, and resumed
|
||||
in Slurm once none is left; a node taken out is drained in Slurm and
|
||||
given back to llmgw when its last job ends.
|
||||
|
||||
### `partition`
|
||||
|
||||
`string`
|
||||
|
||||
Partition is the one partition every node is in. Empty is "gpu".
|
||||
|
||||
### `shared`
|
||||
|
||||
`SharedStorage`
|
||||
|
||||
Shared is the file system every machine of the Slurm cluster sees at
|
||||
the same path: home directories, datasets, the jobs' output and
|
||||
checkpoints. Empty path: none, and a batch job's output stays on the
|
||||
node it ran on.
|
||||
|
||||
Takes a [SharedStorage](#sharedstorage) block.
|
||||
|
||||
### `users`
|
||||
|
||||
`string`
|
||||
|
||||
Users is who keeps the Linux users that run jobs, the same on every
|
||||
machine of the Slurm cluster: "llmgw" makes one for each active llmgw
|
||||
account (its login from the e-mail address, a UID of llmgw's from
|
||||
200000 up, its SSH keys from the portal), let in to the controller,
|
||||
and to a node only while a job of theirs runs there (pam_slurm_adopt);
|
||||
empty leaves users to the site (a directory through sssd, or by hand).
|
||||
|
||||
### `containers`
|
||||
|
||||
`bool`
|
||||
|
||||
Containers lets jobs run in containers: `srun --container-image
|
||||
nvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),
|
||||
the GPUs and the shared path inside. The agents install both on
|
||||
every Slurm machine; jobs take it once every machine has them.
|
||||
|
||||
## StrataConfig
|
||||
|
||||
### `dir`
|
||||
|
||||
+72
-16
@@ -35,6 +35,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/slurmnode"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
|
||||
@@ -97,7 +98,10 @@ type Agent struct {
|
||||
// (nil: this machine's shell).
|
||||
fwMu sync.Mutex
|
||||
fwRunner firewall.Runner
|
||||
inv discovery.Inventory
|
||||
// fwLast is the last firewall message, applied again when the node
|
||||
// is given a part in Slurm. Under fwMu.
|
||||
fwLast *llmgwv1.Firewall
|
||||
inv discovery.Inventory
|
||||
// shardReplica remembers which replica a shard belongs to, so a status
|
||||
// report can name both. The control plane keys on the replica.
|
||||
shardReplica map[string]string
|
||||
@@ -148,6 +152,10 @@ type Agent struct {
|
||||
|
||||
// trainer runs LoRA training jobs, one at a time (BACKLOG F4).
|
||||
trainer *train.Runner
|
||||
// slurm keeps the node's part in Slurm (BACKLOG P); slurmOut is where
|
||||
// its status goes, under mu.
|
||||
slurm *slurmnode.Keeper
|
||||
slurmOut chan<- *llmgwv1.AgentMessage
|
||||
}
|
||||
|
||||
// connectHTTPClient is the HTTP client connect uses; an interface so tests can
|
||||
@@ -206,6 +214,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
|
||||
}
|
||||
a.trainer = train.New(train.Env{EnginesDir: enginesDir, WorkDir: filepath.Join(cfg.Superv.WorkDir, "trainer"),
|
||||
RunAs: runAs, Log: log})
|
||||
a.slurm = slurmnode.New(slurmnode.Local{}, a.prov, slurmMarker, log)
|
||||
return a, nil
|
||||
}
|
||||
|
||||
@@ -232,6 +241,7 @@ func (a *Agent) Run(ctx context.Context) error {
|
||||
}
|
||||
go a.provisionLoop(ctx)
|
||||
go a.ensureDiags(ctx)
|
||||
go a.slurm.Run(ctx, a.sendSlurm)
|
||||
|
||||
attempt := 0
|
||||
next := 0
|
||||
@@ -519,7 +529,8 @@ func (a *Agent) scan(ctx context.Context) error {
|
||||
inv, err := discovery.Discover(ctx, opts, a.log)
|
||||
host := discovery.DiscoverHost()
|
||||
host.MinComputeCapability = discovery.MinComputeCapability(inv.GPUs)
|
||||
host.MemBandwidthGBs = a.memBandwidth()
|
||||
bw := a.memBandwidth()
|
||||
host.MemBandwidthGBs, host.MemBandwidthCoreGBs = bw.All, bw.Core
|
||||
a.mu.Lock()
|
||||
a.inv = inv
|
||||
a.prov.Host = host
|
||||
@@ -529,6 +540,18 @@ func (a *Agent) scan(ctx context.Context) error {
|
||||
return err
|
||||
}
|
||||
|
||||
// forgetSent is a new stream: a control plane that knows nothing of where
|
||||
// installs stand, nor which files are here beyond what registration says —
|
||||
// and registration says what the startup scan found. So the next heartbeat
|
||||
// tells it both again. Without it, files deleted after the agent started came
|
||||
// back on a control plane that restarted, and stayed until the next change.
|
||||
func (a *Agent) forgetSent() {
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
a.provSent = ""
|
||||
a.lastWeights = ""
|
||||
}
|
||||
|
||||
// session runs one connection until it breaks.
|
||||
func (a *Agent) session(ctx context.Context, server string) error {
|
||||
client := llmgwv1connect.NewAgentServiceClient(a.http, server)
|
||||
@@ -565,20 +588,19 @@ func (a *Agent) session(ctx context.Context, server string) error {
|
||||
reg.EngineVersions = a.engines(parent)
|
||||
reg.EnginesProbed = true
|
||||
reg.InstallableEngines = slices.DeleteFunc(provision.Installable(), a.disabled)
|
||||
reg.Features = []string{FeatureFabricTest}
|
||||
reg.Features = []string{FeatureFabricTest, FeatureSlurm}
|
||||
reg.RunningShards = a.runningShards()
|
||||
a.mu.Lock()
|
||||
h := a.prov.Host
|
||||
// A new stream is a control plane that knows nothing of where installs
|
||||
// stand, so the next heartbeat tells it again.
|
||||
a.provSent = ""
|
||||
a.mu.Unlock()
|
||||
a.forgetSent()
|
||||
reg.Host = &llmgwv1.Host{
|
||||
OsId: h.OSID, OsVersion: h.OSVersion, OsCodename: h.OSCodename,
|
||||
Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver,
|
||||
// What is measured by now, not what the startup scan found before
|
||||
// the measurement ended: a reconnect would otherwise say 0.
|
||||
MemBandwidthGbs: a.memBandwidth(),
|
||||
MemBandwidthGbs: a.memBandwidth().All,
|
||||
MemBandwidthCoreGbs: a.memBandwidth().Core,
|
||||
}
|
||||
|
||||
if err := stream.Send(&llmgwv1.AgentMessage{
|
||||
@@ -616,6 +638,8 @@ func (a *Agent) session(ctx context.Context, server string) error {
|
||||
go a.watchExits(ctx, out, unready(reg.RunningShards))
|
||||
go a.forwardKV(ctx, out)
|
||||
go a.reportCompute(ctx, out)
|
||||
a.setSlurmOut(out)
|
||||
defer a.setSlurmOut(nil)
|
||||
|
||||
for {
|
||||
msg, err := stream.Receive()
|
||||
@@ -676,7 +700,8 @@ func (a *Agent) heartbeat() *llmgwv1.AgentMessage {
|
||||
// without anybody restarting anything.
|
||||
hb.MemoryTotalMb = int64(discovery.MemoryTotalMB())
|
||||
hb.CpuCount = int32(runtime.NumCPU())
|
||||
hb.MemBandwidthGbs = a.memBandwidth()
|
||||
bw := a.memBandwidth()
|
||||
hb.MemBandwidthGbs, hb.MemBandwidthCoreGbs = bw.All, bw.Core
|
||||
|
||||
a.reprobeEngines()
|
||||
a.mu.Lock()
|
||||
@@ -881,7 +906,23 @@ func (a *Agent) removeWeights(paths []string) {
|
||||
a.log.Warn("refused to remove a file llmgw did not fetch", "path", p)
|
||||
continue
|
||||
}
|
||||
if err := os.Remove(abs); err != nil && !os.IsNotExist(err) {
|
||||
// A set — every file of one version, fetched into a directory of its
|
||||
// own — goes whole: os.Remove refused it as not empty, and nothing
|
||||
// was freed of what a model of several files left. Lstat, so a link
|
||||
// is taken away and never what it points at.
|
||||
info, err := os.Lstat(abs)
|
||||
switch {
|
||||
case os.IsNotExist(err):
|
||||
continue
|
||||
case err != nil:
|
||||
a.log.Warn("could not look at cached weights", "path", p, "err", err)
|
||||
continue
|
||||
case info.IsDir():
|
||||
err = os.RemoveAll(abs)
|
||||
default:
|
||||
err = os.Remove(abs)
|
||||
}
|
||||
if err != nil {
|
||||
a.log.Warn("could not remove cached weights", "path", p, "err", err)
|
||||
continue
|
||||
}
|
||||
@@ -922,6 +963,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
|
||||
go a.applySettings(m.AgentSettings)
|
||||
case *llmgwv1.ControlMessage_TrainStart:
|
||||
go a.trainJob(ctx, m.TrainStart, out)
|
||||
case *llmgwv1.ControlMessage_Slurm:
|
||||
a.slurmMessage(m.Slurm)
|
||||
case *llmgwv1.ControlMessage_TrainCancel:
|
||||
a.trainer.Cancel(m.TrainCancel.GetJobId())
|
||||
case *llmgwv1.ControlMessage_RestartAgent:
|
||||
@@ -1573,20 +1616,32 @@ func (a *Agent) reportCompute(ctx context.Context, out chan<- *llmgwv1.AgentMess
|
||||
// VMs on one host that all restart together measure each other, and read a
|
||||
// third less than any of them gets alone. Once a process rather than an
|
||||
// agent, so tests running many agents in one read it once.
|
||||
func (a *Agent) memBandwidth() float64 {
|
||||
func (a *Agent) memBandwidth() discovery.Bandwidth {
|
||||
bwOnce.Do(func() {
|
||||
go func() {
|
||||
time.Sleep(bandwidthDelay())
|
||||
for {
|
||||
if bw := measureBandwidth(); bw > math.Float64frombits(bwBits.Load()) {
|
||||
bwBits.Store(math.Float64bits(bw))
|
||||
a.log.Info("this node's memory", "read_gbs", math.Round(bw*10)/10)
|
||||
// The best of each seen so far: an hourly pass on a node
|
||||
// busy serving reads less, and that is not the memory.
|
||||
bw := measureBandwidth()
|
||||
cur := a.memBandwidthNow()
|
||||
if bw.All > cur.All || bw.Core > cur.Core {
|
||||
bwBits.Store(math.Float64bits(max(bw.All, cur.All)))
|
||||
coreBits.Store(math.Float64bits(max(bw.Core, cur.Core)))
|
||||
a.log.Info("this node's memory", "read_gbs", math.Round(bw.All*10)/10,
|
||||
"one_core_gbs", math.Round(bw.Core*10)/10)
|
||||
}
|
||||
time.Sleep(time.Hour)
|
||||
}
|
||||
}()
|
||||
})
|
||||
return math.Float64frombits(bwBits.Load())
|
||||
return a.memBandwidthNow()
|
||||
}
|
||||
|
||||
// memBandwidthNow is what has been measured so far, without starting a
|
||||
// measurement.
|
||||
func (a *Agent) memBandwidthNow() discovery.Bandwidth {
|
||||
return discovery.Bandwidth{All: math.Float64frombits(bwBits.Load()), Core: math.Float64frombits(coreBits.Load())}
|
||||
}
|
||||
|
||||
// bandwidthDelay is how long an agent waits before measuring, so agents
|
||||
@@ -1594,8 +1649,9 @@ func (a *Agent) memBandwidth() float64 {
|
||||
var bandwidthDelay = func() time.Duration { return 10*time.Second + rand.N(110*time.Second) }
|
||||
|
||||
var (
|
||||
bwOnce sync.Once
|
||||
bwBits atomic.Uint64
|
||||
bwOnce sync.Once
|
||||
bwBits atomic.Uint64
|
||||
coreBits atomic.Uint64
|
||||
// measureBandwidth is replaced in tests, which have no use for it.
|
||||
measureBandwidth = discovery.MeasureMemoryBandwidth
|
||||
)
|
||||
@@ -925,3 +925,23 @@ func TestABandwidthTestIsRunOnTheAdapterAskedFor(t *testing.T) {
|
||||
t.Errorf("reply %+v, ran %+v", r, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANewStreamIsToldTheFilesAgain(t *testing.T) {
|
||||
// Registration carries what the startup scan found; a file deleted since
|
||||
// came back on a control plane that restarted, because the next
|
||||
// heartbeat saw the list unchanged since it was last sent.
|
||||
dir := t.TempDir()
|
||||
h := start(t, func(c *Config) { c.ModelDirs = []string{dir} })
|
||||
if err := os.WriteFile(filepath.Join(dir, "0123456789abcdef-m.gguf"), []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.ag.heartbeat()
|
||||
if hb := h.ag.heartbeat().GetHeartbeat(); hb.GetWeightsChanged() {
|
||||
t.Fatal("an unchanged list should not be sent again on the same stream")
|
||||
}
|
||||
h.ag.forgetSent()
|
||||
hb := h.ag.heartbeat().GetHeartbeat()
|
||||
if !hb.GetWeightsChanged() || len(hb.GetWeights()) != 1 {
|
||||
t.Errorf("a new stream should be told the files: changed=%v %v", hb.GetWeightsChanged(), hb.GetWeights())
|
||||
}
|
||||
}
|
||||
@@ -35,6 +35,10 @@ type Host struct {
|
||||
// measured: what decides a model's speed on the CPU. Not read here but
|
||||
// measured by the agent when it starts (MeasureMemoryBandwidth).
|
||||
MemBandwidthGBs float64 `json:"mem_bandwidth_gbs,omitempty"`
|
||||
// MemBandwidthCoreGBs is what one core reads alone, in GB/s, 0 when not
|
||||
// measured: a replica given n threads reads at about n times this, up to
|
||||
// MemBandwidthGBs.
|
||||
MemBandwidthCoreGBs float64 `json:"mem_bandwidth_core_gbs,omitempty"`
|
||||
}
|
||||
|
||||
// Family is the packaging family: "deb", "rpm", or "" when neither is known.
|
||||
|
||||
@@ -28,9 +28,22 @@ const (
|
||||
minBuffer = 256 << 20
|
||||
)
|
||||
|
||||
// MeasureMemoryBandwidth is the memory's read rate in GB/s, the best of a
|
||||
// few passes; 0 when the machine has too little free memory to measure.
|
||||
func MeasureMemoryBandwidth() float64 {
|
||||
// Bandwidth is how fast this machine's memory reads, in GB/s: with every
|
||||
// CPU reading, and with one alone. A replica given a few threads of a large
|
||||
// machine reads at a few times the second, not at the first: on a server of
|
||||
// many memory channels one core cannot keep more than a fraction of them
|
||||
// busy.
|
||||
type Bandwidth struct {
|
||||
All, Core float64
|
||||
}
|
||||
|
||||
// coreBuffer is what the one-core passes read: past any cache, and short
|
||||
// enough that one core reads it in a fraction of a second.
|
||||
const coreBuffer = 512 << 20
|
||||
|
||||
// MeasureMemoryBandwidth is the memory's read rate, the best of a few passes
|
||||
// each way; zero when the machine has too little free memory to measure.
|
||||
func MeasureMemoryBandwidth() Bandwidth {
|
||||
size := int64(bandwidthBuffer)
|
||||
// Never more than a quarter of what is free: a node already holding
|
||||
// models must not be pushed into swap by a measurement. Without room
|
||||
@@ -40,7 +53,7 @@ func MeasureMemoryBandwidth() float64 {
|
||||
size = min(size, avail/4)
|
||||
}
|
||||
if size < max(minBuffer, 8*lastCacheBytes("/sys/devices/system/cpu")) {
|
||||
return 0
|
||||
return Bandwidth{}
|
||||
}
|
||||
buf := make([]uint64, size/8)
|
||||
for i := range buf {
|
||||
@@ -68,7 +81,17 @@ func MeasureMemoryBandwidth() float64 {
|
||||
best = max(best, float64(size)/s/1e9)
|
||||
}
|
||||
}
|
||||
return best
|
||||
// And one core alone, over a part of the same buffer.
|
||||
part := buf[:min(len(buf), coreBuffer/8)]
|
||||
core := 0.0
|
||||
for pass := 0; pass < 3; pass++ {
|
||||
start := time.Now()
|
||||
sink(sum(part))
|
||||
if s := time.Since(start).Seconds(); s > 0 {
|
||||
core = max(core, float64(len(part)*8)/s/1e9)
|
||||
}
|
||||
}
|
||||
return Bandwidth{All: best, Core: min(core, best)}
|
||||
}
|
||||
|
||||
// sum reads every word, eight at a time so the loop is not what limits it.
|
||||
|
||||
@@ -12,11 +12,11 @@ func TestMemoryBandwidthIsMeasured(t *testing.T) {
|
||||
}
|
||||
bw := MeasureMemoryBandwidth()
|
||||
// Any machine running this reads more than a gigabyte a second and less
|
||||
// than a few terabytes.
|
||||
if bw < 1 || bw > 5000 {
|
||||
t.Errorf("measured %.1f GB/s", bw)
|
||||
// than a few terabytes, and one core no more than all of them.
|
||||
if bw.All < 1 || bw.All > 5000 || bw.Core < 0.5 || bw.Core > bw.All {
|
||||
t.Errorf("measured %.1f GB/s, %.1f on one core", bw.All, bw.Core)
|
||||
}
|
||||
t.Logf("%.1f GB/s", bw)
|
||||
t.Logf("%.1f GB/s, %.1f on one core", bw.All, bw.Core)
|
||||
}
|
||||
|
||||
func TestTheLastCacheIsReadFromSys(t *testing.T) {
|
||||
|
||||
@@ -10,8 +10,20 @@ import (
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// slurmPorts are Slurm's on a node that runs any of it: slurmctld's and
|
||||
// slurmd's, and srun's range for the steps it starts.
|
||||
var slurmPorts = []firewall.Port{
|
||||
{From: slurm.CtldPort, To: slurm.DbdPort, What: "slurm: slurmctld, slurmd and slurmdbd"},
|
||||
{From: slurm.SrunPortsFrom, To: slurm.SrunPortsTo, What: "slurm: srun, for the steps it starts"},
|
||||
}
|
||||
|
||||
// nfsPort is NFSv4's, on the machine that exports the Slurm cluster's
|
||||
// shared files.
|
||||
var nfsPort = firewall.Port{From: slurm.NFSPort, To: slurm.NFSPort, What: "nfs: the Slurm cluster's shared files"}
|
||||
|
||||
// applyFirewall keeps llmgw's firewall on this node as the cluster's
|
||||
// settings say: the engines' ports and the node exporter's for the networks
|
||||
// the node reaches its control planes on, and those named, or no table at
|
||||
@@ -19,6 +31,7 @@ import (
|
||||
func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
|
||||
a.fwMu.Lock()
|
||||
defer a.fwMu.Unlock()
|
||||
a.fwLast = m
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
defer cancel()
|
||||
r := a.fwRunner
|
||||
@@ -55,6 +68,14 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
|
||||
if from > 0 && to >= from {
|
||||
ports = append(ports, firewall.Engines(from, to))
|
||||
}
|
||||
// Once there, they stay: the ruleset keeps every port it held, so a
|
||||
// node that leaves Slurm keeps them closed to everyone but the cluster.
|
||||
if a.slurm != nil && a.slurm.HasPart() {
|
||||
ports = append(ports, slurmPorts...)
|
||||
}
|
||||
if a.slurm != nil && a.slurm.Serves() {
|
||||
ports = append(ports, nfsPort)
|
||||
}
|
||||
changed, err := firewall.Install(ctx, r, firewall.Spec{Ports: ports, Allow: nets})
|
||||
switch {
|
||||
case err != nil:
|
||||
@@ -64,6 +85,23 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
|
||||
}
|
||||
}
|
||||
|
||||
// slurmMessage is the leader's word on this node's part in Slurm. A node
|
||||
// that gets one, or starts or stops exporting the shared files, has the
|
||||
// firewall applied again, with Slurm's ports and NFS's.
|
||||
func (a *Agent) slurmMessage(m *llmgwv1.Slurm) {
|
||||
had, served := a.slurm.HasPart(), a.slurm.Serves()
|
||||
a.slurm.Apply(m)
|
||||
if (had || !a.slurm.HasPart()) && served == a.slurm.Serves() {
|
||||
return
|
||||
}
|
||||
a.fwMu.Lock()
|
||||
last := a.fwLast
|
||||
a.fwMu.Unlock()
|
||||
if last != nil && last.GetEnabled() {
|
||||
go a.applyFirewall(last)
|
||||
}
|
||||
}
|
||||
|
||||
// controlPlaneIPs are the addresses of the control planes an agent dials.
|
||||
func controlPlaneIPs(ctx context.Context, servers []string) []string {
|
||||
var out []string
|
||||
|
||||
@@ -6,8 +6,10 @@ import (
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/slurmnode"
|
||||
)
|
||||
|
||||
// fakeShell answers the firewall package's scripts and records what was
|
||||
@@ -55,3 +57,28 @@ func TestANodeKeepsOrDropsItsFirewallAsTheClusterSays(t *testing.T) {
|
||||
t.Error("turned off, the table stayed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeGivenAPartInSlurmLetsSlurmInFromTheClusterOnly(t *testing.T) {
|
||||
sh := &fakeShell{}
|
||||
a := &Agent{log: slog.New(slog.DiscardHandler), fwRunner: sh, serverList: []string{"https://10.0.0.140:7443"},
|
||||
slurm: slurmnode.New(nil, nil, "", slog.New(slog.DiscardHandler))}
|
||||
a.applyFirewall(&llmgwv1.Firewall{Enabled: true, EnginePortsFrom: 20000, EnginePortsTo: 20999})
|
||||
if strings.Contains(sh.installed, "6817") {
|
||||
t.Fatalf("Slurm's ports before the node has a part in it:\n%s", sh.installed)
|
||||
}
|
||||
|
||||
a.slurmMessage(&llmgwv1.Slurm{Name: "node05", Node: true, ConfServer: "10.0.0.148:6817"})
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
a.fwMu.Lock()
|
||||
got := sh.installed
|
||||
a.fwMu.Unlock()
|
||||
if strings.Contains(got, "6817-6819") && strings.Contains(got, "60001-63000") {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("given a part in Slurm, the rules still lack its ports:\n%s", got)
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
@@ -4,12 +4,14 @@ import (
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
// Half a gigabyte read per test binary, and contending with the tests
|
||||
// for the CPUs: these never look at the figure.
|
||||
measureBandwidth = func() float64 { return 0 }
|
||||
measureBandwidth = func() discovery.Bandwidth { return discovery.Bandwidth{} }
|
||||
bandwidthDelay = func() time.Duration { return 0 }
|
||||
os.Exit(m.Run())
|
||||
}
|
||||
@@ -164,6 +164,13 @@ func (a *Agent) provisionLoop(ctx context.Context) {
|
||||
|
||||
// provisionStatus records a change in one engine's installation.
|
||||
func (a *Agent) provisionStatus(st provision.Status) {
|
||||
if st == (provision.Status{}) {
|
||||
// A package install that was no engine's has ended.
|
||||
a.mu.Lock()
|
||||
delete(a.provStatus, "")
|
||||
a.mu.Unlock()
|
||||
return
|
||||
}
|
||||
a.mu.Lock()
|
||||
prev, had := a.provStatus[st.Engine]
|
||||
a.provStatus[st.Engine] = st
|
||||
|
||||
@@ -483,7 +483,8 @@ func (in *Installer) packages(ctx context.Context, byFamily map[string][]string,
|
||||
return fmt.Errorf("%s is missing and no package manager was found to install it",
|
||||
strings.Join(missing, ", "))
|
||||
}
|
||||
in.progress(st, "installing %s with %s", strings.Join(missing, ", "), tool)
|
||||
// The state already says installing.
|
||||
in.progress(st, "%s, with %s", strings.Join(missing, ", "), tool)
|
||||
for _, c := range cmds {
|
||||
if err := in.runWaitingForLock(ctx, pm.env, c, st); err != nil {
|
||||
return err
|
||||
@@ -497,7 +498,13 @@ func (in *Installer) packages(ctx context.Context, byFamily map[string][]string,
|
||||
// byFamily is keyed by discovery.Host.Family: "deb" or "rpm".
|
||||
func (in *Installer) InstallPackages(ctx context.Context, byFamily map[string][]string) error {
|
||||
var st Status
|
||||
return in.packages(ctx, byFamily, &st)
|
||||
err := in.packages(ctx, byFamily, &st)
|
||||
if st.State == Installing {
|
||||
// Its progress was no engine's: an empty status takes it off the
|
||||
// node's page again, done or failed (the caller says which).
|
||||
in.report(Status{})
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// lockWait bounds how long a package command waits for another to finish.
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
package provision
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
// A package install that is no engine's says it is done, so its progress
|
||||
// does not stay on the node's page (found on the PoC installing Slurm).
|
||||
func TestInstallPackagesClearsItsProgress(t *testing.T) {
|
||||
var got []Status
|
||||
in := &Installer{Host: discovery.Host{OSID: "ubuntu"}, OnStatus: func(s Status) { got = append(got, s) },
|
||||
Run: func(_ context.Context, _ []string, name string, _ ...string) ([]byte, error) {
|
||||
if name == "sh" { // the query: not installed
|
||||
return nil, context.Canceled
|
||||
}
|
||||
return nil, nil
|
||||
}}
|
||||
if err := in.InstallPackages(context.Background(), map[string][]string{"deb": {"munge"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) < 2 || got[0].State != Installing || got[len(got)-1] != (Status{}) {
|
||||
t.Fatalf("statuses %+v", got)
|
||||
}
|
||||
|
||||
got = nil
|
||||
in.Run = func(context.Context, []string, string, ...string) ([]byte, error) { return nil, nil }
|
||||
if err := in.InstallPackages(context.Background(), map[string][]string{"deb": {"munge"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("installed already, yet said %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -94,3 +94,35 @@ func TestWithNoCacheConfiguredNothingIsDeleted(t *testing.T) {
|
||||
t.Errorf("an agent with no cache directory deleted something: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A model of several files is fetched into a directory of its own, and an
|
||||
// unused one goes whole: it was refused as not empty, and nothing was freed.
|
||||
func TestAnUnusedSetIsRemovedWhole(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
a := agentWithCache(t, dir)
|
||||
set := weights.CachePathFor(dir, "https://mirror/coder/v1")
|
||||
if err := os.MkdirAll(set, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
write(t, filepath.Join(set, "model-00001-of-00002.gguf"))
|
||||
write(t, filepath.Join(set, ".complete"))
|
||||
// A link in the cache is taken away, not what it points at.
|
||||
target := t.TempDir()
|
||||
write(t, filepath.Join(target, "keep.gguf"))
|
||||
link := weights.CachePathFor(dir, "https://mirror/linked/v1")
|
||||
if err := os.Symlink(target, link); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
a.removeWeights([]string{set, link})
|
||||
|
||||
if _, err := os.Stat(set); !os.IsNotExist(err) {
|
||||
t.Errorf("the set is still there: %v", err)
|
||||
}
|
||||
if _, err := os.Lstat(link); !os.IsNotExist(err) {
|
||||
t.Errorf("the link is still there: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(target, "keep.gguf")); err != nil {
|
||||
t.Errorf("what the link pointed at was deleted: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
)
|
||||
|
||||
// FeatureSlurm is the agent keeping its node's part in Slurm (BACKLOG P).
|
||||
const FeatureSlurm = "slurm"
|
||||
|
||||
// slurmMarker says llmgw started Slurm's daemons on this machine.
|
||||
var slurmMarker = "/var/lib/llmgw-agent/slurm"
|
||||
|
||||
// setSlurmOut is where Slurm's status goes now: this session's queue, or
|
||||
// nowhere between sessions.
|
||||
func (a *Agent) setSlurmOut(out chan<- *llmgwv1.AgentMessage) {
|
||||
a.mu.Lock()
|
||||
a.slurmOut = out
|
||||
a.mu.Unlock()
|
||||
if out != nil {
|
||||
a.slurm.Resend()
|
||||
}
|
||||
}
|
||||
|
||||
// sendSlurm queues a status without waiting: a full queue or no session
|
||||
// drops it, and the next session is sent it again.
|
||||
func (a *Agent) sendSlurm(st *llmgwv1.SlurmStatus) {
|
||||
a.mu.Lock()
|
||||
out := a.slurmOut
|
||||
a.mu.Unlock()
|
||||
if out == nil {
|
||||
return
|
||||
}
|
||||
select {
|
||||
case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_SlurmStatus{SlurmStatus: st}}:
|
||||
default:
|
||||
a.log.Warn("Slurm's status not sent: the queue to the control plane is full")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,198 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// cacheDir holds what is fetched for Slurm: enroot's packages, pyxis's
|
||||
// source and its build.
|
||||
const cacheDir = "/var/cache/llmgw-agent/slurm"
|
||||
|
||||
// pyxisMarker is the pyxis version built here, against this Slurm.
|
||||
func (k *Keeper) pyxisMarker() string { return k.marker + "-pyxis" }
|
||||
|
||||
// containers has enroot and pyxis here, set up, when want says so; what
|
||||
// is there is left when it no longer does (Slurm stops loading pyxis once
|
||||
// the controller's plugstack.conf goes). Answers the status line, empty
|
||||
// while either is missing.
|
||||
func (k *Keeper) containers(ctx context.Context, want *llmgwv1.Slurm) (string, error) {
|
||||
if !want.GetContainers() {
|
||||
return "", nil
|
||||
}
|
||||
family := k.family()
|
||||
if family == "" {
|
||||
return "", fmt.Errorf("neither dpkg nor rpm here: enroot and pyxis come as deb or rpm")
|
||||
}
|
||||
if err := k.enroot(ctx, family); err != nil {
|
||||
return "", fmt.Errorf("enroot: %w", err)
|
||||
}
|
||||
if err := k.enrootConf(ctx, want.GetShared().GetPath()); err != nil {
|
||||
return "", fmt.Errorf("enroot's settings: %w", err)
|
||||
}
|
||||
if _, err := k.host.LookPath("nvidia-smi"); err == nil {
|
||||
if err := k.nvidiaContainer(ctx, family); err != nil {
|
||||
return "", fmt.Errorf("the GPUs in containers (nvidia-container-cli): %w", err)
|
||||
}
|
||||
}
|
||||
if err := k.pyxis(ctx, family); err != nil {
|
||||
return "", fmt.Errorf("pyxis: %w", err)
|
||||
}
|
||||
return slurm.ContainersStatus(), nil
|
||||
}
|
||||
|
||||
func (k *Keeper) family() string {
|
||||
if _, err := k.host.LookPath("dpkg"); err == nil {
|
||||
return "deb"
|
||||
}
|
||||
if _, err := k.host.LookPath("rpm"); err == nil {
|
||||
return "rpm"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// enroot installs the pinned release when another, or none, is here.
|
||||
func (k *Keeper) enroot(ctx context.Context, family string) error {
|
||||
if k.installed["enroot"] {
|
||||
return nil
|
||||
}
|
||||
if out, err := k.host.Run(ctx, nil, "enroot", "version"); err == nil && strings.TrimSpace(string(out)) == slurm.EnrootVersion {
|
||||
k.installed["enroot"] = true
|
||||
return nil
|
||||
}
|
||||
pkgs, err := slurm.EnrootPackages(family, runtime.GOARCH)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("enroot missing, and this agent installs nothing")
|
||||
}
|
||||
var files []string
|
||||
for _, p := range pkgs {
|
||||
f := filepath.Join(cacheDir, p.Name)
|
||||
if err := k.host.Download(ctx, p.URL(), f, p.SHA256); err != nil {
|
||||
return err
|
||||
}
|
||||
files = append(files, f)
|
||||
}
|
||||
k.log.Info("installing enroot, for Slurm's jobs in containers", "version", slurm.EnrootVersion)
|
||||
if err := k.inst.InstallPackages(ctx, map[string][]string{family: files}); err != nil {
|
||||
return err
|
||||
}
|
||||
out, err := k.host.Run(ctx, nil, "enroot", "version")
|
||||
if v := strings.TrimSpace(string(out)); err != nil || v != slurm.EnrootVersion {
|
||||
return fmt.Errorf("enroot says %q after installing %s: %v", v, slurm.EnrootVersion, err)
|
||||
}
|
||||
k.installed["enroot"] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// enrootConf writes enroot's settings and makes its directories.
|
||||
func (k *Keeper) enrootConf(ctx context.Context, shared string) error {
|
||||
if _, err := k.write(slurm.EnrootConfPath, []byte(slurm.EnrootConf), 0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
changed, err := k.write(slurm.EnrootTmpfiles, []byte(slurm.EnrootTmpfilesConf), 0o644, "")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if changed {
|
||||
if out, err := k.host.Run(ctx, nil, "systemd-tmpfiles", "--create", slurm.EnrootTmpfiles); err != nil {
|
||||
return fmt.Errorf("systemd-tmpfiles: %v: %s", err, lastLine(out))
|
||||
}
|
||||
}
|
||||
for _, d := range []string{slurm.EnrootLib + "/cache", slurm.EnrootLib + "/data"} {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-d", d); err == nil {
|
||||
continue
|
||||
}
|
||||
// Each user makes their own beneath, as in /tmp.
|
||||
if out, err := k.host.Run(ctx, nil, "install", "-d", "-m", "1777", d); err != nil {
|
||||
return fmt.Errorf("making %s: %v: %s", d, err, lastLine(out))
|
||||
}
|
||||
}
|
||||
if shared == "" {
|
||||
return k.host.Remove(slurm.EnrootMounts)
|
||||
}
|
||||
_, err = k.write(slurm.EnrootMounts, []byte(slurm.EnrootSharedMount(shared)), 0o644, "")
|
||||
return err
|
||||
}
|
||||
|
||||
// nvidiaContainer installs nvidia-container-cli, which enroot's hook uses
|
||||
// to give a container the job's GPUs, from NVIDIA's repository.
|
||||
func (k *Keeper) nvidiaContainer(ctx context.Context, family string) error {
|
||||
if k.installed["nvidia-container-cli"] {
|
||||
return nil
|
||||
}
|
||||
if _, err := k.host.LookPath("nvidia-container-cli"); err == nil {
|
||||
k.installed["nvidia-container-cli"] = true
|
||||
return nil
|
||||
}
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("nvidia-container-cli missing, and this agent installs nothing")
|
||||
}
|
||||
switch family {
|
||||
case "deb":
|
||||
if err := k.host.Download(ctx, slurm.NvidiaContainerKeyURL, slurm.NvidiaContainerKeyPath,
|
||||
slurm.NvidiaContainerKeySHA256); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := k.write(slurm.NvidiaContainerAptList, []byte(slurm.NvidiaContainerApt), 0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
case "rpm":
|
||||
if _, err := k.write(slurm.NvidiaContainerYumRepo, []byte(slurm.NvidiaContainerYum), 0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
k.log.Info("installing nvidia-container-cli, so containers get the job's GPUs")
|
||||
if err := k.inst.InstallPackages(ctx, slurm.NvidiaContainerPackages); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := k.host.LookPath("nvidia-container-cli"); err != nil {
|
||||
return fmt.Errorf("still missing after installing libnvidia-container-tools")
|
||||
}
|
||||
k.installed["nvidia-container-cli"] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// pyxis builds the pinned pyxis against this machine's Slurm and installs
|
||||
// it, when it has not been, or was for another version.
|
||||
func (k *Keeper) pyxis(ctx context.Context, family string) error {
|
||||
if b, err := k.host.ReadFile(k.pyxisMarker()); err == nil && strings.TrimSpace(string(b)) == slurm.PyxisVersion {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-f", slurm.PyxisPlugin); err == nil {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("pyxis missing, and this agent installs nothing")
|
||||
}
|
||||
if err := k.inst.InstallPackages(ctx, map[string][]string{family: slurm.PyxisBuildPackages[family]}); err != nil {
|
||||
return fmt.Errorf("installing what builds it: %w", err)
|
||||
}
|
||||
tarball := filepath.Join(cacheDir, "pyxis-"+slurm.PyxisVersion+".tar.gz")
|
||||
if err := k.host.Download(ctx, slurm.PyxisSource, tarball, slurm.PyxisSHA256); err != nil {
|
||||
return err
|
||||
}
|
||||
src := filepath.Join(cacheDir, "pyxis-"+slurm.PyxisVersion)
|
||||
k.log.Info("building pyxis against this machine's Slurm", "version", slurm.PyxisVersion)
|
||||
for _, c := range [][]string{
|
||||
{"rm", "-rf", src},
|
||||
{"tar", "-xzf", tarball, "-C", cacheDir},
|
||||
{"make", "-C", src, "install"},
|
||||
} {
|
||||
if out, err := k.host.Run(ctx, nil, c[0], c[1:]...); err != nil {
|
||||
return fmt.Errorf("%s: %v: %s", strings.Join(c, " "), err, lastLine(out))
|
||||
}
|
||||
}
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-f", slurm.PyxisPlugin); err != nil {
|
||||
return fmt.Errorf("built, yet %s is not there", slurm.PyxisPlugin)
|
||||
}
|
||||
_, err := k.write(k.pyxisMarker(), []byte(slurm.PyxisVersion+"\n"), 0o644, "")
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"runtime"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
func TestContainersInstalledFromPinnedReleases(t *testing.T) {
|
||||
if _, err := slurm.EnrootPackages("deb", runtime.GOARCH); err != nil {
|
||||
t.Skip(err)
|
||||
}
|
||||
h := newFake()
|
||||
h.bins["dpkg"], h.bins["nvidia-smi"] = true, true
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
// enroot is not here until installed, and then says its version.
|
||||
h.fail["enroot version"] = true
|
||||
h.fail["test -d "+slurm.EnrootLib+"/cache"] = true
|
||||
h.fail["test -f "+slurm.PyxisPlugin] = true
|
||||
m := nodeMsg(true)
|
||||
m.Containers = true
|
||||
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Nfs: "10.0.0.11:/shared"}
|
||||
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
if st.GetContainers() != "" || !strings.Contains(st.GetError(), "containers: enroot") {
|
||||
t.Fatalf("ready with enroot saying nothing: %v", st)
|
||||
}
|
||||
pkgs, _ := slurm.EnrootPackages("deb", runtime.GOARCH)
|
||||
for _, p := range pkgs {
|
||||
if !slices.Contains(h.ran, "download "+p.URL()+" "+cacheDir+"/"+p.Name+" "+p.SHA256) {
|
||||
t.Errorf("%s not fetched with its digest:\n%s", p.Name, strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
if len(inst.got["deb"]) != 2 || !strings.HasPrefix(inst.got["deb"][0], cacheDir) {
|
||||
t.Errorf("installed %v", inst.got)
|
||||
}
|
||||
|
||||
delete(h.fail, "enroot version")
|
||||
h.outputs["enroot version"] = slurm.EnrootVersion + "\n"
|
||||
h.bins["nvidia-container-cli"] = false
|
||||
st = k.Reconcile(context.Background(), m)
|
||||
if !strings.Contains(st.GetError(), "nvidia-container-cli") {
|
||||
t.Errorf("a GPU machine without nvidia-container-cli is ready: %v", st)
|
||||
}
|
||||
if string(h.files[slurm.NvidiaContainerAptList]) != slurm.NvidiaContainerApt ||
|
||||
!slices.Contains(h.ran, "download "+slurm.NvidiaContainerKeyURL+" "+slurm.NvidiaContainerKeyPath+" "+slurm.NvidiaContainerKeySHA256) {
|
||||
t.Errorf("NVIDIA's repository not added with its key checked")
|
||||
}
|
||||
|
||||
h.bins["nvidia-container-cli"] = true
|
||||
st = k.Reconcile(context.Background(), m)
|
||||
// Built, but the plugin is still not there.
|
||||
if st.GetContainers() != "" || !strings.Contains(st.GetError(), "pyxis") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
for _, want := range []string{"tar -xzf " + cacheDir + "/pyxis-" + slurm.PyxisVersion + ".tar.gz -C " + cacheDir,
|
||||
"make -C " + cacheDir + "/pyxis-" + slurm.PyxisVersion + " install", "install -d -m 1777 " + slurm.EnrootLib + "/cache"} {
|
||||
if !slices.Contains(h.ran, want) {
|
||||
t.Errorf("did not run %q", want)
|
||||
}
|
||||
}
|
||||
delete(h.fail, "test -f "+slurm.PyxisPlugin)
|
||||
st = k.Reconcile(context.Background(), m)
|
||||
if st.GetContainers() != slurm.ContainersStatus() || st.GetError() != "" {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if string(h.files[slurm.EnrootConfPath]) != slurm.EnrootConf ||
|
||||
!strings.Contains(string(h.files[slurm.EnrootMounts]), "/shared /shared none") {
|
||||
t.Errorf("enroot's settings not written")
|
||||
}
|
||||
|
||||
// Done: nothing fetched or built again.
|
||||
h.ran = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
for _, c := range h.ran {
|
||||
if strings.HasPrefix(c, "download") || strings.HasPrefix(c, "make") {
|
||||
t.Errorf("ran %q again", c)
|
||||
}
|
||||
}
|
||||
|
||||
// Off: nothing said, nothing taken away.
|
||||
m.Containers = false
|
||||
if st := k.Reconcile(context.Background(), m); st.GetContainers() != "" {
|
||||
t.Errorf("status %v", st)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// mariaDBConfDirs are where each family's MariaDB reads its settings: Debian
|
||||
// and Ubuntu's, then Red Hat's.
|
||||
var mariaDBConfDirs = []string{"/etc/mysql/mariadb.conf.d", "/etc/my.cnf.d"}
|
||||
|
||||
const mariaDBConfName = "90-llmgw-slurm.cnf"
|
||||
|
||||
// acctSQLPath is where the database's statements are put for the client to
|
||||
// read, never on its command line: they hold the password.
|
||||
const acctSQLPath = "/run/llmgw-slurm-acct.sql"
|
||||
|
||||
// accounting keeps slurmdbd on the controller: MariaDB on this machine
|
||||
// alone, the database and its user, slurmdbd on them, and the cluster in it.
|
||||
// Before slurmctld, which reports to it from its first start.
|
||||
func (k *Keeper) accounting(ctx context.Context, key []byte, files map[string]string) error {
|
||||
password := slurm.DbdPassword(key)
|
||||
changed, err := k.mariaDBSettings(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch {
|
||||
case changed:
|
||||
k.log.Info("MariaDB, for Slurm's accounting, starts on this machine alone")
|
||||
if err := k.systemctl(ctx, "enable", "--now", "mariadb", "&&restart"); err != nil {
|
||||
return err
|
||||
}
|
||||
case !k.active(ctx, "mariadb"):
|
||||
if err := k.systemctl(ctx, "enable", "--now", "mariadb"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
client, err := k.mariaDBClient()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := k.waitFor(ctx, "MariaDB", client, "-e", "SELECT 1"); err != nil {
|
||||
return err
|
||||
}
|
||||
// The database and its user, once each time the agent starts or the
|
||||
// password is new: cheap, and a database restored from elsewhere gets
|
||||
// its user back.
|
||||
sum := sha256.Sum256([]byte(password))
|
||||
if mark := hex.EncodeToString(sum[:]); k.acctDone != mark {
|
||||
if err := k.host.WriteFile(acctSQLPath, []byte(slurm.AcctSQL(password)), 0o600, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
out, err := k.host.Run(ctx, nil, client, "-e", "source "+acctSQLPath)
|
||||
_ = k.host.Remove(acctSQLPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("making Slurm's database: %v: %s", err, lastLine(out))
|
||||
}
|
||||
k.acctDone = mark
|
||||
}
|
||||
|
||||
out, err := k.host.Run(ctx, nil, "hostname", "-s")
|
||||
host := strings.TrimSpace(string(out))
|
||||
if err != nil || host == "" {
|
||||
return fmt.Errorf("this machine's short hostname: %v: %s", err, lastLine(out))
|
||||
}
|
||||
changed, err = k.write(slurm.DbdConfPath, []byte(slurm.DbdConf(host, password)), 0o600, "slurm")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch {
|
||||
case changed:
|
||||
k.log.Info("slurmdbd starts again on new settings")
|
||||
if err := k.systemctl(ctx, "enable", "--now", "slurmdbd", "&&restart"); err != nil {
|
||||
return err
|
||||
}
|
||||
case !k.active(ctx, "slurmdbd"):
|
||||
if err := k.systemctl(ctx, "enable", "--now", "slurmdbd"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := k.waitFor(ctx, "slurmdbd", "sacctmgr", "-n", "-P", "show", "cluster", "format=cluster"); err != nil {
|
||||
return err
|
||||
}
|
||||
name := clusterName(files["slurm.conf"])
|
||||
if name == "" || k.clusterAdded == name {
|
||||
return nil
|
||||
}
|
||||
out, err = k.host.Run(ctx, nil, "sacctmgr", "-n", "-P", "show", "cluster", "format=cluster")
|
||||
if err != nil {
|
||||
return fmt.Errorf("sacctmgr show cluster: %v: %s", err, lastLine(out))
|
||||
}
|
||||
for _, l := range strings.Split(string(out), "\n") {
|
||||
if strings.TrimSpace(l) == name {
|
||||
k.clusterAdded = name
|
||||
return nil
|
||||
}
|
||||
}
|
||||
k.log.Info("the cluster is added to Slurm's accounting", "cluster", name)
|
||||
if out, err := k.host.Run(ctx, nil, "sacctmgr", "-i", "add", "cluster", name); err != nil {
|
||||
return fmt.Errorf("sacctmgr add cluster %s: %v: %s", name, err, lastLine(out))
|
||||
}
|
||||
k.clusterAdded = name
|
||||
return nil
|
||||
}
|
||||
|
||||
// mariaDBSettings keeps MariaDB to this machine, when llmgw installed it;
|
||||
// one that was here before llmgw keeps its own settings, and only gets the
|
||||
// database and its user.
|
||||
func (k *Keeper) mariaDBSettings(ctx context.Context) (bool, error) {
|
||||
if _, err := k.host.ReadFile(k.mariaDBMarker()); err != nil {
|
||||
return false, nil
|
||||
}
|
||||
for _, d := range mariaDBConfDirs {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-d", d); err == nil {
|
||||
return k.write(d+"/"+mariaDBConfName, []byte(slurm.MariaDBConf), 0o644, "")
|
||||
}
|
||||
}
|
||||
return false, fmt.Errorf("MariaDB has none of %s to put its settings in", strings.Join(mariaDBConfDirs, ", "))
|
||||
}
|
||||
|
||||
// mariaDBMarker is there when llmgw installed MariaDB on this machine.
|
||||
func (k *Keeper) mariaDBMarker() string { return k.marker + "-mariadb" }
|
||||
|
||||
// mariaDBClient is MariaDB's command-line client, under either of its
|
||||
// names.
|
||||
func (k *Keeper) mariaDBClient() (string, error) {
|
||||
for _, c := range []string{"mariadb", "mysql"} {
|
||||
if _, err := k.host.LookPath(c); err == nil {
|
||||
return c, nil
|
||||
}
|
||||
}
|
||||
return "", errors.New("MariaDB's client (mariadb or mysql) is not installed")
|
||||
}
|
||||
|
||||
// waitFor runs a command until it succeeds, half a minute at most.
|
||||
func (k *Keeper) waitFor(ctx context.Context, what, name string, args ...string) error {
|
||||
var last []byte
|
||||
for range 30 {
|
||||
out, err := k.host.Run(ctx, nil, name, args...)
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
last = out
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(pingWait):
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("%s did not answer after starting: %s", what, lastLine(last))
|
||||
}
|
||||
|
||||
// clusterName is ClusterName in slurm.conf.
|
||||
func clusterName(conf string) string {
|
||||
sc := bufio.NewScanner(strings.NewReader(conf))
|
||||
for sc.Scan() {
|
||||
if v, ok := strings.CutPrefix(strings.TrimSpace(sc.Text()), "ClusterName="); ok {
|
||||
return strings.TrimSpace(v)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,150 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// What a machine says of the shared path.
|
||||
const (
|
||||
SharedMounted = "mounted"
|
||||
SharedExported = "exported"
|
||||
SharedMissing = "missing"
|
||||
)
|
||||
|
||||
// shared keeps the shared path as want says: exported from here, mounted
|
||||
// from an export, or looked for where the site mounts it. Answers how it
|
||||
// is.
|
||||
func (k *Keeper) shared(ctx context.Context, want *llmgwv1.Slurm) (string, error) {
|
||||
sh := want.GetShared()
|
||||
path := sh.GetPath()
|
||||
k.unshare(ctx, path, sh.GetServe(), sh.GetNfs())
|
||||
if path == "" {
|
||||
return "", nil
|
||||
}
|
||||
if !strings.HasPrefix(path, "/") || strings.Contains(path, "..") || strings.ContainsAny(path, " \n\t\"'\\$`") {
|
||||
return SharedMissing, fmt.Errorf("the leader sent a path %q", path)
|
||||
}
|
||||
switch {
|
||||
case sh.GetServe():
|
||||
if err := k.host.MkdirAll(path, 0o755, ""); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
changed, err := k.write(slurm.ExportsPath, []byte(slurm.Exports(path, sh.GetClients())), 0o644, "")
|
||||
if err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
if !k.active(ctx, "nfs-server") {
|
||||
k.log.Info("NFS starts on this machine, exporting the Slurm cluster's shared files", "path", path)
|
||||
if err := k.systemctl(ctx, "enable", "--now", "nfs-server"); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
changed = true
|
||||
}
|
||||
if changed {
|
||||
if out, err := k.host.Run(ctx, nil, "exportfs", "-ra"); err != nil {
|
||||
return SharedMissing, fmt.Errorf("exportfs -ra: %v: %s", err, lastLine(out))
|
||||
}
|
||||
}
|
||||
return SharedExported, nil
|
||||
case sh.GetNfs() != "":
|
||||
nfs := sh.GetNfs()
|
||||
if strings.ContainsAny(nfs, " \n\t\"'\\$`") || !strings.Contains(nfs, ":/") {
|
||||
return SharedMissing, fmt.Errorf("the leader sent an export %q", nfs)
|
||||
}
|
||||
if err := k.host.MkdirAll(path, 0o755, ""); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
unit := slurm.MountUnitName(path)
|
||||
changed, err := k.write(filepath.Join(slurm.MountUnitDir, unit), []byte(slurm.MountUnit(nfs, path)), 0o644, "")
|
||||
if err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
if changed {
|
||||
k.log.Info("the Slurm cluster's shared files are mounted here", "from", nfs, "at", path)
|
||||
if err := k.systemctl(ctx, "daemon-reload"); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
if err := k.systemctl(ctx, "enable", unit); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
// From another export: a restart mounts the new one, and fails
|
||||
// while a job still has files open on the old.
|
||||
if err := k.systemctl(ctx, "restart", unit); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
} else if !k.active(ctx, unit) {
|
||||
if err := k.systemctl(ctx, "start", unit); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
}
|
||||
}
|
||||
if _, err := k.host.Run(ctx, nil, "mountpoint", "-q", path); err != nil {
|
||||
if sh.GetNfs() == "" {
|
||||
return SharedMissing, fmt.Errorf("%s is not mounted here: the site mounts it (slurm.shared names no export)", path)
|
||||
}
|
||||
return SharedMissing, fmt.Errorf("%s is not mounted from %s", path, sh.GetNfs())
|
||||
}
|
||||
return SharedMounted, nil
|
||||
}
|
||||
|
||||
// unshare takes away the export or the mount llmgw kept here that want no
|
||||
// longer has: path is where the files are wanted now, serve and nfs how.
|
||||
func (k *Keeper) unshare(ctx context.Context, path string, serve bool, nfs string) {
|
||||
if _, err := k.host.ReadFile(slurm.ExportsPath); err == nil && (!serve || path == "") {
|
||||
k.log.Info("this machine no longer exports the Slurm cluster's shared files")
|
||||
_ = k.host.Remove(slurm.ExportsPath)
|
||||
_, _ = k.host.Run(ctx, nil, "exportfs", "-ra")
|
||||
}
|
||||
out, err := k.host.Run(ctx, nil, "systemctl", "list-units", "--all", "--plain", "--no-legend", "--type=mount")
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
keep := ""
|
||||
if path != "" && nfs != "" && !serve {
|
||||
keep = slurm.MountUnitName(path)
|
||||
}
|
||||
for _, l := range strings.Split(string(out), "\n") {
|
||||
f := strings.Fields(l)
|
||||
if len(f) == 0 || f[0] == keep {
|
||||
continue
|
||||
}
|
||||
file := filepath.Join(slurm.MountUnitDir, f[0])
|
||||
b, err := k.host.ReadFile(file)
|
||||
if err != nil || !strings.HasPrefix(string(b), "# Written by llmgw (BACKLOG P): the Slurm cluster's shared files.") {
|
||||
continue
|
||||
}
|
||||
k.log.Info("the Slurm cluster's old shared mount is taken away", "unit", f[0])
|
||||
if err := k.systemctl(ctx, "disable", "--now", f[0]); err != nil {
|
||||
k.log.Warn("could not unmount the old shared files", "unit", f[0], "err", err)
|
||||
continue
|
||||
}
|
||||
_ = k.host.Remove(file)
|
||||
_ = k.systemctl(ctx, "daemon-reload")
|
||||
}
|
||||
}
|
||||
|
||||
// sharedPackages are what the shared path needs here, by the command that
|
||||
// shows each is installed.
|
||||
func sharedPackages(want *llmgwv1.Slurm, need map[string][2]string) {
|
||||
switch sh := want.GetShared(); {
|
||||
case sh.GetPath() == "":
|
||||
case sh.GetServe():
|
||||
need["exportfs"] = [2]string{"nfs-kernel-server", "nfs-utils"}
|
||||
case sh.GetNfs() != "":
|
||||
need["mount.nfs"] = [2]string{"nfs-common", "nfs-utils"}
|
||||
}
|
||||
}
|
||||
|
||||
// Serves is whether this machine exports the shared files, so NFS's port
|
||||
// is kept for the cluster's networks.
|
||||
func (k *Keeper) Serves() bool {
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
return k.want.GetShared().GetServe() && k.want.GetShared().GetPath() != ""
|
||||
}
|
||||
@@ -0,0 +1,171 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
func ayse() *llmgwv1.SlurmUser {
|
||||
return &llmgwv1.SlurmUser{Login: "ayse", Uid: 200000, Name: "Ayşe Y.", Home: "/shared/home/ayse",
|
||||
SshKeys: []string{"ssh-ed25519 AAAA ayse@laptop"}}
|
||||
}
|
||||
|
||||
func TestTheServerExportsAndMakesUsersWithHomes(t *testing.T) {
|
||||
h := newFake()
|
||||
h.active["ssh"] = true
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
m := nodeMsg(false)
|
||||
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Serve: true, Clients: []string{"10.0.0.22", "10.0.0.21"}}
|
||||
m.ManageUsers, m.MakeHomes, m.Users = true, true, []*llmgwv1.SlurmUser{ayse()}
|
||||
h.fail["test -d /shared/home/ayse"] = true
|
||||
h.files[slurm.PAMSSHD] = []byte(pamStack)
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
if st.GetError() != "" {
|
||||
t.Fatal(st.GetError())
|
||||
}
|
||||
if st.GetShared() != SharedExported || st.GetUsers() != 1 {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if !slices.Contains(inst.got["deb"], "nfs-kernel-server") {
|
||||
t.Errorf("packages %v", inst.got)
|
||||
}
|
||||
if e := string(h.files[slurm.ExportsPath]); !strings.Contains(e, `"/shared" 10.0.0.21(`) {
|
||||
t.Errorf("exports:\n%s", e)
|
||||
}
|
||||
for _, want := range []string{"systemctl enable --now nfs-server", "exportfs -ra", "groupadd -g 200000 ayse",
|
||||
"useradd -u 200000 -g 200000 -M -d /shared/home/ayse -s /bin/bash -c Ayşe Y. ayse",
|
||||
"install -d -m 0700 -o 200000 -g 200000 /shared/home/ayse", "sshd -t", "systemctl reload ssh"} {
|
||||
if !slices.Contains(h.ran, want) {
|
||||
t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
if string(h.files[slurm.AuthorizedKeysDir+"/ayse"]) != "ssh-ed25519 AAAA ayse@laptop\n" ||
|
||||
string(h.files[k.usersMarker()]) != "ayse\n" || h.files[slurm.SSHDropIn] == nil {
|
||||
t.Errorf("files: keys %q, marker %q", h.files[slurm.AuthorizedKeysDir+"/ayse"], h.files[k.usersMarker()])
|
||||
}
|
||||
|
||||
// Again, made: nothing is run again.
|
||||
h.outputs["getent passwd ayse"] = "ayse:x:200000:200000:Ayşe Y.:/shared/home/ayse:/bin/bash\n"
|
||||
delete(h.fail, "test -d /shared/home/ayse")
|
||||
h.ran = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
for _, c := range h.ran {
|
||||
if strings.HasPrefix(c, "useradd") || strings.HasPrefix(c, "usermod") || strings.HasPrefix(c, "exportfs") ||
|
||||
strings.HasPrefix(c, "install") || c == "systemctl reload ssh" {
|
||||
t.Errorf("ran %q again", c)
|
||||
}
|
||||
}
|
||||
|
||||
// Gone from the accounts: taken away, its home left.
|
||||
m.Users = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
if !slices.Contains(h.ran, "userdel ayse") || h.files[k.usersMarker()] != nil || h.files[slurm.SSHDropIn] != nil ||
|
||||
h.files[slurm.AuthorizedKeysDir+"/ayse"] != nil {
|
||||
t.Errorf("not taken away:\n%s", strings.Join(h.ran, "\n"))
|
||||
}
|
||||
if slices.ContainsFunc(h.ran, func(c string) bool { return strings.HasPrefix(c, "rm") || strings.Contains(c, "userdel -r") }) {
|
||||
t.Error("a home was removed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAClientMountsAndLeavesOthersUsersAlone(t *testing.T) {
|
||||
h := newFake()
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
m := nodeMsg(false)
|
||||
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Nfs: "10.0.0.11:/shared"}
|
||||
bob := &llmgwv1.SlurmUser{Login: "bob", Uid: 200001, Home: "/shared/home/bob"}
|
||||
m.ManageUsers, m.Users = true, []*llmgwv1.SlurmUser{ayse(), bob}
|
||||
// The site's own ayse, with its own UID.
|
||||
h.outputs["getent passwd ayse"] = "ayse:x:1001:1001::/home/ayse:/bin/bash\n"
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
if st.GetShared() != SharedMounted || st.GetUsers() != 1 {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if !strings.Contains(st.GetError(), "ayse is already a user here with UID 1001") {
|
||||
t.Errorf("error %q", st.GetError())
|
||||
}
|
||||
if u := string(h.files["/etc/systemd/system/shared.mount"]); !strings.Contains(u, "What=10.0.0.11:/shared") {
|
||||
t.Fatalf("unit:\n%s", u)
|
||||
}
|
||||
for _, want := range []string{"systemctl enable shared.mount", "systemctl restart shared.mount", "mountpoint -q /shared",
|
||||
"useradd -u 200001 -g 200001 -M -d /shared/home/bob -s /bin/bash -c bob"} {
|
||||
if !slices.Contains(h.ran, want) {
|
||||
t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
if slices.ContainsFunc(h.ran, func(c string) bool { return strings.Contains(c, "ayse") && !strings.HasPrefix(c, "getent") }) ||
|
||||
slices.ContainsFunc(h.ran, func(c string) bool { return strings.HasPrefix(c, "install -d") }) {
|
||||
t.Errorf("touched the site's ayse, or made a home:\n%s", strings.Join(h.ran, "\n"))
|
||||
}
|
||||
if string(h.files[k.usersMarker()]) != "bob\n" {
|
||||
t.Errorf("marker %q", h.files[k.usersMarker()])
|
||||
}
|
||||
|
||||
// Not mounted: said.
|
||||
h.fail["mountpoint -q /shared"] = true
|
||||
if st := k.Reconcile(context.Background(), m); st.GetShared() != SharedMissing ||
|
||||
!strings.Contains(st.GetError(), "not mounted from 10.0.0.11:/shared") {
|
||||
t.Errorf("status %v", st)
|
||||
}
|
||||
|
||||
// No shared path any more: the mount is taken away.
|
||||
h.outputs["systemctl list-units --all --plain --no-legend --type=mount"] = "shared.mount loaded active mounted llmgw\n"
|
||||
m.Shared = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
if !slices.Contains(h.ran, "systemctl disable --now shared.mount") || h.files["/etc/systemd/system/shared.mount"] != nil {
|
||||
t.Errorf("mount kept:\n%s", strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
|
||||
const pamStack = "@include common-auth\naccount required pam_nologin.so\n@include common-account\n@include common-session\n"
|
||||
|
||||
func TestUsersReachANodeOnlyThroughTheirJobs(t *testing.T) {
|
||||
h := newFake()
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
h.files[slurm.PAMSSHD] = []byte(pamStack)
|
||||
// The module is not there until its package is installed.
|
||||
for _, d := range slurm.ModuleDirs {
|
||||
h.fail["test -f "+d+"/"+slurm.AdoptModule] = true
|
||||
}
|
||||
m := nodeMsg(true)
|
||||
m.ManageUsers, m.Users = true, []*llmgwv1.SlurmUser{ayse()}
|
||||
k.Reconcile(context.Background(), m)
|
||||
if !slices.Contains(inst.got["deb"], "libpam-slurm-adopt") {
|
||||
t.Fatalf("not installed: %v", inst.got)
|
||||
}
|
||||
if got := string(h.files[slurm.PAMSSHD]); got != pamStack {
|
||||
t.Fatalf("lines put in with the module still missing:\n%s", got)
|
||||
}
|
||||
delete(h.fail, "test -f "+slurm.ModuleDirs[0]+"/"+slurm.AdoptModule)
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
got := string(h.files[slurm.PAMSSHD])
|
||||
if !strings.Contains(got, "pam_listfile.so item=user sense=allow file="+k.usersMarker()) ||
|
||||
!strings.Contains(got, "account required "+slurm.AdoptModule) {
|
||||
t.Fatalf("not held to their jobs (%s):\n%s", st.GetError(), got)
|
||||
}
|
||||
|
||||
// The controller is where they log in: not held there.
|
||||
m.Controller = true
|
||||
m.Files = map[string]string{"slurm.conf": "x"}
|
||||
k.Reconcile(context.Background(), m)
|
||||
if string(h.files[slurm.PAMSSHD]) != pamStack {
|
||||
t.Errorf("held on the controller:\n%s", h.files[slurm.PAMSSHD])
|
||||
}
|
||||
|
||||
// No users of llmgw's any more: the lines go.
|
||||
m.Controller, m.Files = false, nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
m.Users = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
if string(h.files[slurm.PAMSSHD]) != pamStack {
|
||||
t.Errorf("lines left with no users:\n%s", h.files[slurm.PAMSSHD])
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,733 @@
|
||||
// Package slurmnode keeps this machine's part in the Slurm cluster run beside
|
||||
// llmgw (BACKLOG P), as the leader says: slurmctld on the controller, slurmd
|
||||
// on a node, munge on both, installed from the machine's own package sources.
|
||||
//
|
||||
// What the leader sends is wanted state, kept by Keeper's loop: sent again,
|
||||
// or checked again a minute later, it changes nothing that is already so.
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// Host is what Keeper does to the machine; Local in the agent, a fake in
|
||||
// tests.
|
||||
type Host interface {
|
||||
Run(ctx context.Context, env []string, name string, args ...string) ([]byte, error)
|
||||
LookPath(name string) (string, error)
|
||||
ReadFile(path string) ([]byte, error)
|
||||
// WriteFile writes b at path with mode, owned by owner ("" for root),
|
||||
// making its directory.
|
||||
WriteFile(path string, b []byte, mode fs.FileMode, owner string) error
|
||||
// MkdirAll makes path, owned by owner.
|
||||
MkdirAll(path string, mode fs.FileMode, owner string) error
|
||||
Remove(path string) error
|
||||
// Download fetches url to path, kept only if its SHA-256 is sha; a file
|
||||
// already there with it is not fetched again.
|
||||
Download(ctx context.Context, url, path, sha string) error
|
||||
}
|
||||
|
||||
// Installer installs packages by family, "deb" and "rpm".
|
||||
type Installer interface {
|
||||
InstallPackages(ctx context.Context, byFamily map[string][]string) error
|
||||
}
|
||||
|
||||
// Reasons llmgw drains a node with; one drained for another reason, by a
|
||||
// Slurm administrator, is never resumed by llmgw.
|
||||
const (
|
||||
ReasonServing = "llmgw: serving models"
|
||||
reasonPrefix = "llmgw"
|
||||
)
|
||||
|
||||
// DropIn is the unit drop-in that starts slurmd configless.
|
||||
const DropIn = "/etc/systemd/system/slurmd.service.d/llmgw.conf"
|
||||
|
||||
// Keeper keeps the machine as the last Slurm message says.
|
||||
type Keeper struct {
|
||||
host Host
|
||||
inst Installer
|
||||
log *slog.Logger
|
||||
marker string
|
||||
|
||||
mu sync.Mutex
|
||||
want *llmgwv1.Slurm
|
||||
kick chan struct{}
|
||||
// resend has the next status sent even if unchanged: a new stream's
|
||||
// control plane may never have had it.
|
||||
resend bool
|
||||
// installed is true once the packages were found or put there, so a
|
||||
// minute's check does not look again.
|
||||
installed map[string]bool
|
||||
files string
|
||||
// acctDone is the digest of the password the database's user was last
|
||||
// given; clusterAdded the cluster last found in the accounting.
|
||||
acctDone string
|
||||
clusterAdded string
|
||||
}
|
||||
|
||||
// New is a Keeper; marker is a file it keeps while it runs Slurm here, so
|
||||
// an agent started again knows to stop what it started when told it has no
|
||||
// part any more.
|
||||
func New(host Host, inst Installer, marker string, log *slog.Logger) *Keeper {
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
return &Keeper{host: host, inst: inst, log: log, marker: marker,
|
||||
kick: make(chan struct{}, 1), installed: map[string]bool{}}
|
||||
}
|
||||
|
||||
// Apply is a new message from the leader: kept from now on.
|
||||
func (k *Keeper) Apply(m *llmgwv1.Slurm) {
|
||||
k.mu.Lock()
|
||||
k.want = proto.CloneOf(m)
|
||||
k.mu.Unlock()
|
||||
select {
|
||||
case k.kick <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
// HasPart is whether the last message gives this machine a part in Slurm.
|
||||
func (k *Keeper) HasPart() bool {
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
return k.want.GetController() || k.want.GetNode()
|
||||
}
|
||||
|
||||
// Resend has the status sent again on the next pass, which it starts now.
|
||||
func (k *Keeper) Resend() {
|
||||
k.mu.Lock()
|
||||
k.resend = true
|
||||
k.mu.Unlock()
|
||||
select {
|
||||
case k.kick <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
// Every is how often the machine is checked again without a new message.
|
||||
var Every = time.Minute
|
||||
|
||||
// resendAfter is how often an unchanged status is sent anyway.
|
||||
const resendAfter = 5 * time.Minute
|
||||
|
||||
// Run keeps the machine until ctx ends, sending its status when it changes
|
||||
// and every few minutes while it has a part.
|
||||
func (k *Keeper) Run(ctx context.Context, send func(*llmgwv1.SlurmStatus)) {
|
||||
t := time.NewTicker(Every)
|
||||
defer t.Stop()
|
||||
var last *llmgwv1.SlurmStatus
|
||||
var sent time.Time
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-k.kick:
|
||||
case <-t.C:
|
||||
}
|
||||
k.mu.Lock()
|
||||
want, again := k.want, k.resend
|
||||
k.resend = false
|
||||
k.mu.Unlock()
|
||||
if want == nil {
|
||||
continue
|
||||
}
|
||||
st := k.Reconcile(ctx, want)
|
||||
if st == nil {
|
||||
last = nil
|
||||
continue
|
||||
}
|
||||
if again || !proto.Equal(st, last) || time.Since(sent) > resendAfter {
|
||||
send(st)
|
||||
last, sent = st, time.Now()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Reconcile makes the machine what want says once, and answers its status;
|
||||
// nil when it has no part in Slurm.
|
||||
func (k *Keeper) Reconcile(ctx context.Context, want *llmgwv1.Slurm) *llmgwv1.SlurmStatus {
|
||||
ctx, cancel := context.WithTimeout(ctx, 20*time.Minute)
|
||||
defer cancel()
|
||||
if !want.GetController() && !want.GetNode() {
|
||||
k.stop(ctx)
|
||||
return nil
|
||||
}
|
||||
st := &llmgwv1.SlurmStatus{}
|
||||
fail := func(what string, err error) *llmgwv1.SlurmStatus {
|
||||
st.Error = what + ": " + err.Error()
|
||||
k.log.Warn("keeping Slurm on this node: "+what, "err", err)
|
||||
return st
|
||||
}
|
||||
if err := k.ensurePackages(ctx, want); err != nil {
|
||||
return fail("installing Slurm", err)
|
||||
}
|
||||
// From the daemon, not scontrol: a client command looks for its files
|
||||
// even to say its version, and a configless node has none before slurmd
|
||||
// has fetched them.
|
||||
daemon := "slurmd"
|
||||
if !want.GetNode() {
|
||||
daemon = "slurmctld"
|
||||
}
|
||||
out, err := k.host.Run(ctx, nil, daemon, "-V")
|
||||
if err != nil {
|
||||
return fail("reading Slurm's version", fmt.Errorf("%v: %s", err, lastLine(out)))
|
||||
}
|
||||
v, err := slurm.ParseVersion(string(out))
|
||||
if err != nil {
|
||||
return fail("reading Slurm's version", err)
|
||||
}
|
||||
st.Version = strings.TrimSpace(string(out))
|
||||
if i := strings.LastIndexByte(st.Version, ' '); i >= 0 {
|
||||
st.Version = st.Version[i+1:]
|
||||
}
|
||||
if slurm.TooOld(v) {
|
||||
return fail("Slurm", fmt.Errorf("%d.%02d is older than %d.%02d, the first to run jobs in cgroup v2; "+
|
||||
"a newer release of the distribution, or Slurm's own packages, has one", v[0], v[1],
|
||||
slurm.MinVersion[0], slurm.MinVersion[1]))
|
||||
}
|
||||
if err := k.host.WriteFile(k.marker, []byte("llmgw runs Slurm here (BACKLOG P)\n"), 0o644, ""); err != nil {
|
||||
return fail("keeping its marker", err)
|
||||
}
|
||||
if err := k.munge(ctx, want.GetMungeKey()); err != nil {
|
||||
return fail("munge", err)
|
||||
}
|
||||
// The shared files, then the users whose homes are on them. Neither
|
||||
// keeps Slurm from running: what goes wrong is said, and Slurm kept.
|
||||
var shErr, usErr error
|
||||
st.Shared, shErr = k.shared(ctx, want)
|
||||
if shErr != nil {
|
||||
k.log.Warn("keeping the Slurm cluster's shared files on this node", "err", shErr)
|
||||
}
|
||||
if shErr == nil || !want.GetMakeHomes() || want.GetShared().GetPath() == "" {
|
||||
st.Users, usErr = k.users(ctx, want)
|
||||
} else {
|
||||
// Homes made now would be made beside the shared files, not on them.
|
||||
usErr = errors.New("waiting for the shared files, where the homes are")
|
||||
}
|
||||
if usErr != nil {
|
||||
k.log.Warn("keeping the Slurm users on this node", "err", usErr)
|
||||
}
|
||||
// Jobs in containers; Slurm runs without them, and is kept anyway.
|
||||
var ctErr error
|
||||
if st.Containers, ctErr = k.containers(ctx, want); ctErr != nil {
|
||||
k.log.Warn("setting up jobs in containers on this node", "err", ctErr)
|
||||
}
|
||||
defer func() {
|
||||
switch {
|
||||
case st.Error != "":
|
||||
case shErr != nil:
|
||||
st.Error = "shared files: " + shErr.Error()
|
||||
case usErr != nil:
|
||||
st.Error = "users: " + usErr.Error()
|
||||
case ctErr != nil:
|
||||
st.Error = "containers: " + ctErr.Error()
|
||||
}
|
||||
}()
|
||||
if want.GetController() {
|
||||
// Accounting first: slurmctld reports to it from its first start.
|
||||
// Without it slurmctld still runs jobs, a pass of backfill late
|
||||
// each; so a failure here is said, and slurmctld is kept anyway.
|
||||
var acctErr error
|
||||
if acctErr = k.accounting(ctx, want.GetMungeKey(), want.GetFiles()); acctErr != nil {
|
||||
k.log.Warn("keeping Slurm's accounting on this node", "err", acctErr)
|
||||
}
|
||||
st.SlurmdbdRunning = k.active(ctx, "slurmdbd")
|
||||
defer func() {
|
||||
if acctErr != nil && st.Error == "" {
|
||||
st.Error = "slurmdbd: " + acctErr.Error()
|
||||
}
|
||||
}()
|
||||
sum, err := k.controller(ctx, want.GetFiles())
|
||||
st.FilesSha256 = sum
|
||||
if err != nil {
|
||||
return fail("slurmctld", err)
|
||||
}
|
||||
st.SlurmctldRunning = k.active(ctx, "slurmctld")
|
||||
}
|
||||
if want.GetNode() {
|
||||
if err := k.node(ctx, want, st); err != nil {
|
||||
return fail("slurmd", err)
|
||||
}
|
||||
}
|
||||
return st
|
||||
}
|
||||
|
||||
func (k *Keeper) ensurePackages(ctx context.Context, want *llmgwv1.Slurm) error {
|
||||
need := map[string][2]string{
|
||||
"munged": {"munge", "munge"},
|
||||
"scontrol": {"slurm-client", "slurm"},
|
||||
}
|
||||
if want.GetController() {
|
||||
need["slurmctld"] = [2]string{"slurmctld", "slurm-slurmctld"}
|
||||
need["slurmdbd"] = [2]string{"slurmdbd", "slurm-slurmdbd"}
|
||||
need["mariadbd"] = [2]string{"mariadb-server", "mariadb-server"}
|
||||
}
|
||||
if want.GetNode() {
|
||||
need["slurmd"] = [2]string{"slurmd", "slurm-slurmd"}
|
||||
}
|
||||
sharedPackages(want, need)
|
||||
pkgs := map[string][]string{}
|
||||
for bin, p := range need {
|
||||
if k.installed[bin] {
|
||||
continue
|
||||
}
|
||||
if _, err := k.host.LookPath(bin); err == nil {
|
||||
k.installed[bin] = true
|
||||
continue
|
||||
}
|
||||
pkgs["deb"] = append(pkgs["deb"], p[0])
|
||||
pkgs["rpm"] = append(pkgs["rpm"], p[1])
|
||||
}
|
||||
if len(pkgs) == 0 {
|
||||
return nil
|
||||
}
|
||||
slices.Sort(pkgs["deb"])
|
||||
slices.Sort(pkgs["rpm"])
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("%s missing, and this agent installs nothing", strings.Join(pkgs["deb"], ", "))
|
||||
}
|
||||
k.log.Info("installing Slurm, as the cluster's settings say", "packages", pkgs["deb"])
|
||||
if err := k.inst.InstallPackages(ctx, pkgs); err != nil {
|
||||
return err
|
||||
}
|
||||
if slices.Contains(pkgs["deb"], "mariadb-server") {
|
||||
// Installed by llmgw: its settings are llmgw's to keep. One that was
|
||||
// here before is another's, and is left as it is.
|
||||
if err := k.host.WriteFile(k.mariaDBMarker(), []byte("llmgw installed MariaDB here, for Slurm's accounting\n"),
|
||||
0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
for bin := range need {
|
||||
if _, err := k.host.LookPath(bin); err != nil {
|
||||
return fmt.Errorf("%s is still missing after installing %s", bin, strings.Join(pkgs["deb"], ", "))
|
||||
}
|
||||
k.installed[bin] = true
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// munge writes the cluster's key and keeps munged running on it.
|
||||
func (k *Keeper) munge(ctx context.Context, key []byte) error {
|
||||
if len(key) < 32 {
|
||||
return errors.New("the leader sent no key")
|
||||
}
|
||||
changed, err := k.write(slurm.MungeKeyPath, key, 0o400, "munge")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if changed {
|
||||
k.log.Info("munge key written; munge starts again on it")
|
||||
return k.systemctl(ctx, "enable", "--now", "munge", "&&restart")
|
||||
}
|
||||
if !k.active(ctx, "munge") {
|
||||
return k.systemctl(ctx, "enable", "--now", "munge")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// controller writes the files and keeps slurmctld running on them,
|
||||
// answering their digest.
|
||||
func (k *Keeper) controller(ctx context.Context, files map[string]string) (string, error) {
|
||||
if files["slurm.conf"] == "" {
|
||||
return "", errors.New("the leader sent no slurm.conf")
|
||||
}
|
||||
for _, d := range []string{slurm.StateDir, slurm.LogDir} {
|
||||
if err := k.host.MkdirAll(d, 0o755, "slurm"); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
names := make([]string, 0, len(files))
|
||||
for n := range files {
|
||||
names = append(names, n)
|
||||
}
|
||||
slices.Sort(names)
|
||||
h := sha256.New()
|
||||
changed := false
|
||||
for _, n := range names {
|
||||
if strings.ContainsAny(n, "/\\") || n == "" || n == "." || n == ".." {
|
||||
return "", fmt.Errorf("the leader sent a file named %q", n)
|
||||
}
|
||||
fmt.Fprintf(h, "%s\x00%s\x00", n, files[n])
|
||||
c, err := k.write(filepath.Join(slurm.ConfDir, n), []byte(files[n]), 0o644, "")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
changed = changed || c
|
||||
}
|
||||
sum := hex.EncodeToString(h.Sum(nil))
|
||||
if changed || sum != k.files {
|
||||
if err := k.systemctl(ctx, "enable", "slurmctld"); err != nil {
|
||||
return sum, err
|
||||
}
|
||||
// Nodes added or taken away need a restart before 23.11 and a
|
||||
// reconfigure after it; a restart does both, and running jobs live
|
||||
// through it.
|
||||
k.log.Info("slurmctld starts again on new files")
|
||||
if err := k.systemctl(ctx, "restart", "slurmctld"); err != nil {
|
||||
return sum, err
|
||||
}
|
||||
if err := k.waitPing(ctx); err != nil {
|
||||
return sum, err
|
||||
}
|
||||
// Hands the configless slurmd the new files.
|
||||
if out, err := k.host.Run(ctx, nil, "scontrol", "reconfigure"); err != nil {
|
||||
return sum, fmt.Errorf("scontrol reconfigure: %v: %s", err, lastLine(out))
|
||||
}
|
||||
k.files = sum
|
||||
return sum, nil
|
||||
}
|
||||
if !k.active(ctx, "slurmctld") {
|
||||
return sum, k.systemctl(ctx, "enable", "--now", "slurmctld")
|
||||
}
|
||||
return sum, nil
|
||||
}
|
||||
|
||||
func (k *Keeper) waitPing(ctx context.Context) error {
|
||||
var last []byte
|
||||
for range 30 {
|
||||
out, err := k.host.Run(ctx, nil, "scontrol", "ping")
|
||||
if err == nil && bytes.Contains(out, []byte("UP")) {
|
||||
return nil
|
||||
}
|
||||
last = out
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(pingWait):
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("slurmctld did not answer after starting: %s", lastLine(last))
|
||||
}
|
||||
|
||||
var pingWait = time.Second
|
||||
|
||||
// node keeps slurmd running configless and puts the node in or out of the
|
||||
// controller's hands as lent says.
|
||||
func (k *Keeper) node(ctx context.Context, want *llmgwv1.Slurm, st *llmgwv1.SlurmStatus) error {
|
||||
name, server := want.GetName(), want.GetConfServer()
|
||||
if name == "" || server == "" {
|
||||
return errors.New("the leader sent no name or controller")
|
||||
}
|
||||
if strings.ContainsAny(name+server, " \n\t\"'\\$`") {
|
||||
return fmt.Errorf("the leader sent a name or controller with spaces or quotes in it")
|
||||
}
|
||||
for _, d := range []string{slurm.SpoolDir, slurm.LogDir} {
|
||||
if err := k.host.MkdirAll(d, 0o755, "slurm"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
bin, err := k.host.LookPath("slurmd")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
unit := "# Written by llmgw (BACKLOG P): slurmd fetches its files from slurmctld.\n" +
|
||||
"[Unit]\nConditionPathExists=\nStartLimitIntervalSec=0\n" +
|
||||
// --systemd as the packaged unit has it (Type=notify, since 22.05):
|
||||
// without it systemd waits for a readiness that never comes.
|
||||
"[Service]\nExecStart=\nExecStart=" + bin + " --systemd --conf-server " + server + " -N " + name + "\n" +
|
||||
// A node new to the controller is refused until the leader has
|
||||
// written it into slurm.conf; tried again until it is.
|
||||
"Restart=on-failure\nRestartSec=15\n"
|
||||
changed, err := k.write(DropIn, []byte(unit), 0o644, "")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if out, err := k.host.Run(ctx, nil, "slurmd", "-C"); err == nil {
|
||||
st.Hardware = strings.TrimSpace(string(out))
|
||||
}
|
||||
if changed {
|
||||
if err := k.systemctl(ctx, "daemon-reload"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := k.systemctl(ctx, "enable", "slurmd"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := k.systemctl(ctx, "restart", "slurmd"); err != nil {
|
||||
return err
|
||||
}
|
||||
} else if !k.active(ctx, "slurmd") {
|
||||
if err := k.systemctl(ctx, "enable", "--now", "slurmd"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
st.SlurmdRunning = k.active(ctx, "slurmd")
|
||||
|
||||
st.Jobs = k.localJobs(ctx)
|
||||
env := k.clientEnv()
|
||||
out, err := k.host.Run(ctx, env, "scontrol", "show", "node", "-o", name)
|
||||
if err != nil {
|
||||
// Not in slurm.conf yet: the leader writes it once this node's
|
||||
// hardware is in.
|
||||
return nil
|
||||
}
|
||||
ns, err := slurm.ParseNodeState(string(out))
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
ours := ns.Reason == "" || strings.HasPrefix(ns.Reason, reasonPrefix)
|
||||
switch {
|
||||
case want.GetLent() && ns.Drained() && ours:
|
||||
k.log.Info("this node is lent to Slurm: resumed there")
|
||||
if out, err := k.host.Run(ctx, env, "scontrol", "update", "NodeName="+name, "State=RESUME"); err != nil {
|
||||
return fmt.Errorf("resuming the node: %v: %s", err, lastLine(out))
|
||||
}
|
||||
case !want.GetLent() && !strings.Contains(ns.State, "DRAIN"):
|
||||
k.log.Info("this node is llmgw's: drained in Slurm, its jobs left to finish")
|
||||
if out, err := k.host.Run(ctx, env, "scontrol", "update", "NodeName="+name, "State=DRAIN",
|
||||
"Reason="+ReasonServing); err != nil {
|
||||
return fmt.Errorf("draining the node: %v: %s", err, lastLine(out))
|
||||
}
|
||||
}
|
||||
if out, err := k.host.Run(ctx, env, "scontrol", "show", "node", "-o", name); err == nil {
|
||||
if ns2, err := slurm.ParseNodeState(string(out)); err == nil {
|
||||
ns = ns2
|
||||
}
|
||||
}
|
||||
st.State, st.Reason = ns.State, ns.Reason
|
||||
return nil
|
||||
}
|
||||
|
||||
// localJobs counts the job steps running here, from their slurmstepd, not
|
||||
// from the controller: a job outlives a controller that is down, and is
|
||||
// what decides whether llmgw may have the cards back. The one slurmstepd
|
||||
// that is no job's, "slurmstepd infinity" holding slurmd's cgroup v2 scope,
|
||||
// is not counted. -1 when it cannot tell.
|
||||
func (k *Keeper) localJobs(ctx context.Context) int32 {
|
||||
// pgrep exits 1 with no output when there is none.
|
||||
out, err := k.host.Run(ctx, nil, "pgrep", "-a", "-x", "slurmstepd")
|
||||
if err != nil && len(bytes.TrimSpace(out)) > 0 {
|
||||
return -1
|
||||
}
|
||||
n := int32(0)
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) < 2 || f[len(f)-1] == "infinity" {
|
||||
continue
|
||||
}
|
||||
n++
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// clientEnv points scontrol at slurmd's copy of the files on a node that is
|
||||
// not the controller, which has none of its own.
|
||||
func (k *Keeper) clientEnv() []string {
|
||||
if _, err := k.host.ReadFile(filepath.Join(slurm.ConfDir, "slurm.conf")); err == nil {
|
||||
return nil
|
||||
}
|
||||
return []string{"SLURM_CONF=" + filepath.Join(slurm.SpoolDir, "conf-cache", "slurm.conf")}
|
||||
}
|
||||
|
||||
// stop stops what llmgw started here, if it did.
|
||||
func (k *Keeper) stop(ctx context.Context) {
|
||||
if _, err := k.host.ReadFile(k.marker); err != nil {
|
||||
return
|
||||
}
|
||||
k.log.Info("this node has no part in Slurm any more: slurmd, slurmctld and slurmdbd stopped; the packages and MariaDB stay")
|
||||
// MariaDB stays: it may have been the machine's before, and holds the
|
||||
// jobs' records.
|
||||
for _, u := range []string{"slurmd", "slurmctld", "slurmdbd"} {
|
||||
_ = k.systemctl(ctx, "disable", "--now", u)
|
||||
}
|
||||
_ = k.host.Remove(DropIn)
|
||||
_ = k.systemctl(ctx, "daemon-reload")
|
||||
k.unshare(ctx, "", false, "")
|
||||
if err := k.removeUsers(ctx, k.madeUsers(), nil); err != nil {
|
||||
k.log.Warn("could not take away every Slurm user llmgw made", "err", err)
|
||||
}
|
||||
_ = k.host.Remove(k.marker)
|
||||
k.files, k.acctDone, k.clusterAdded = "", "", ""
|
||||
}
|
||||
|
||||
// write writes a file when it differs, saying whether it did.
|
||||
func (k *Keeper) write(path string, b []byte, mode fs.FileMode, owner string) (bool, error) {
|
||||
if old, err := k.host.ReadFile(path); err == nil && bytes.Equal(old, b) {
|
||||
return false, nil
|
||||
}
|
||||
return true, k.host.WriteFile(path, b, mode, owner)
|
||||
}
|
||||
|
||||
// systemctl runs systemctl; a last argument "&&restart" restarts the unit
|
||||
// after.
|
||||
func (k *Keeper) systemctl(ctx context.Context, args ...string) error {
|
||||
restart := false
|
||||
if n := len(args); n > 0 && args[n-1] == "&&restart" {
|
||||
args, restart = args[:n-1], true
|
||||
}
|
||||
if out, err := k.host.Run(ctx, nil, "systemctl", args...); err != nil {
|
||||
return fmt.Errorf("systemctl %s: %v: %s", strings.Join(args, " "), err, lastLine(out))
|
||||
}
|
||||
if restart {
|
||||
u := args[len(args)-1]
|
||||
if out, err := k.host.Run(ctx, nil, "systemctl", "restart", u); err != nil {
|
||||
return fmt.Errorf("systemctl restart %s: %v: %s", u, err, lastLine(out))
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (k *Keeper) active(ctx context.Context, unit string) bool {
|
||||
out, err := k.host.Run(ctx, nil, "systemctl", "is-active", unit)
|
||||
return err == nil && strings.TrimSpace(string(out)) == "active"
|
||||
}
|
||||
|
||||
func lastLine(b []byte) string {
|
||||
s := strings.TrimSpace(string(b))
|
||||
if i := strings.LastIndexByte(s, '\n'); i >= 0 {
|
||||
s = s[i+1:]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Local is the machine the agent runs on.
|
||||
type Local struct{}
|
||||
|
||||
// Run runs a command, with env added to the agent's.
|
||||
func (Local) Run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
|
||||
cmd := exec.CommandContext(ctx, name, args...)
|
||||
if len(env) > 0 {
|
||||
cmd.Env = append(os.Environ(), env...)
|
||||
}
|
||||
return cmd.CombinedOutput()
|
||||
}
|
||||
|
||||
// LookPath finds a command, in the sbin directories too: an agent started
|
||||
// by systemd has them, one started by hand may not.
|
||||
func (Local) LookPath(name string) (string, error) {
|
||||
if p, err := exec.LookPath(name); err == nil {
|
||||
return p, nil
|
||||
}
|
||||
// /usr/libexec: Red Hat's mariadbd.
|
||||
for _, d := range []string{"/usr/sbin", "/sbin", "/usr/local/sbin", "/usr/libexec"} {
|
||||
p := filepath.Join(d, name)
|
||||
if fi, err := os.Stat(p); err == nil && !fi.IsDir() {
|
||||
return p, nil
|
||||
}
|
||||
}
|
||||
return "", fmt.Errorf("%s is not installed", name)
|
||||
}
|
||||
|
||||
// ReadFile reads a file.
|
||||
func (Local) ReadFile(path string) ([]byte, error) { return os.ReadFile(path) } //nolint:gosec // our paths
|
||||
|
||||
// WriteFile writes a file whole, through a temporary one beside it.
|
||||
func (Local) WriteFile(path string, b []byte, mode fs.FileMode, owner string) error {
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { //nolint:gosec // system directories
|
||||
return err
|
||||
}
|
||||
tmp := path + ".llmgw-new"
|
||||
if err := os.WriteFile(tmp, b, mode); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.Chmod(tmp, mode); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := chown(tmp, owner); err != nil {
|
||||
_ = os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, path)
|
||||
}
|
||||
|
||||
// Download fetches url to path through a temporary file beside it, kept
|
||||
// only when its SHA-256 is sha.
|
||||
func (Local) Download(ctx context.Context, url, path, sha string) error {
|
||||
if f, err := os.Open(path); err == nil { //nolint:gosec // our paths
|
||||
h := sha256.New()
|
||||
_, cerr := io.Copy(h, f)
|
||||
_ = f.Close()
|
||||
if cerr == nil && hex.EncodeToString(h.Sum(nil)) == sha {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { //nolint:gosec // system directories
|
||||
return err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
|
||||
defer cancel()
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("fetching %s: %w", url, err)
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("fetching %s: %s", url, resp.Status)
|
||||
}
|
||||
tmp := path + ".llmgw-new"
|
||||
f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o644) //nolint:gosec // our paths
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
h := sha256.New()
|
||||
_, err = io.Copy(io.MultiWriter(f, h), io.LimitReader(resp.Body, 1<<30))
|
||||
if cerr := f.Close(); err == nil {
|
||||
err = cerr
|
||||
}
|
||||
if err != nil {
|
||||
_ = os.Remove(tmp)
|
||||
return fmt.Errorf("fetching %s: %w", url, err)
|
||||
}
|
||||
if got := hex.EncodeToString(h.Sum(nil)); got != sha {
|
||||
_ = os.Remove(tmp)
|
||||
return fmt.Errorf("%s has SHA-256 %s, not the %s llmgw was built with: not used", url, got, sha)
|
||||
}
|
||||
return os.Rename(tmp, path)
|
||||
}
|
||||
|
||||
// MkdirAll makes a directory owned by owner.
|
||||
func (Local) MkdirAll(path string, mode fs.FileMode, owner string) error {
|
||||
if err := os.MkdirAll(path, mode); err != nil {
|
||||
return err
|
||||
}
|
||||
return chown(path, owner)
|
||||
}
|
||||
|
||||
// Remove removes a file, and is content when there is none.
|
||||
func (Local) Remove(path string) error {
|
||||
if err := os.Remove(path); err != nil && !errors.Is(err, fs.ErrNotExist) {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func chown(path, owner string) error {
|
||||
if owner == "" {
|
||||
return nil
|
||||
}
|
||||
u, err := user.Lookup(owner)
|
||||
if err != nil {
|
||||
return fmt.Errorf("user %s: %w (installed with its package)", owner, err)
|
||||
}
|
||||
uid, _ := strconv.Atoi(u.Uid)
|
||||
gid, _ := strconv.Atoi(u.Gid)
|
||||
return os.Chown(path, uid, gid)
|
||||
}
|
||||
@@ -0,0 +1,367 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io/fs"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
type fakeHost struct {
|
||||
files map[string][]byte
|
||||
owners map[string]string
|
||||
bins map[string]bool
|
||||
active map[string]bool
|
||||
ran []string
|
||||
outputs map[string]string
|
||||
fail map[string]bool
|
||||
state string
|
||||
reason string
|
||||
}
|
||||
|
||||
func newFake() *fakeHost {
|
||||
return &fakeHost{files: map[string][]byte{}, owners: map[string]string{}, bins: map[string]bool{},
|
||||
active: map[string]bool{}, outputs: map[string]string{
|
||||
"slurmd -V": "slurm-wlm 23.11.4\n",
|
||||
"slurmctld -V": "slurm-wlm 23.11.4\n",
|
||||
"scontrol ping": "Slurmctld(primary) at node06 is UP\n",
|
||||
"slurmd -C": "NodeName=cuda01 CPUs=16 RealMemory=64000\nUpTime=1-00:00:00\n",
|
||||
"hostname -s": "node06\n",
|
||||
}, fail: map[string]bool{}, state: "IDLE+DRAIN", reason: slurm.ReasonNew}
|
||||
}
|
||||
|
||||
func (f *fakeHost) Run(_ context.Context, env []string, name string, args ...string) ([]byte, error) {
|
||||
c := strings.Join(append([]string{name}, args...), " ")
|
||||
f.ran = append(f.ran, c)
|
||||
if f.fail[c] {
|
||||
return []byte("it failed"), errors.New("exit status 1")
|
||||
}
|
||||
switch {
|
||||
case name == "systemctl" && len(args) >= 1:
|
||||
u := args[len(args)-1]
|
||||
switch args[0] {
|
||||
case "is-active":
|
||||
if f.active[u] {
|
||||
return []byte("active\n"), nil
|
||||
}
|
||||
return []byte("inactive\n"), errors.New("exit status 3")
|
||||
case "restart", "start":
|
||||
f.active[u] = true
|
||||
case "enable":
|
||||
if slices.Contains(args, "--now") {
|
||||
f.active[u] = true
|
||||
}
|
||||
case "disable":
|
||||
f.active[u] = false
|
||||
}
|
||||
return []byte(f.outputs[c]), nil
|
||||
case strings.HasPrefix(c, "scontrol show node"):
|
||||
return []byte("NodeName=cuda01 Arch=x86_64 State=" + f.state + " Partitions=gpu Reason=" + f.reason + " [root@now]\n"), nil
|
||||
case strings.HasPrefix(c, "scontrol update"):
|
||||
if slices.Contains(args, "State=RESUME") {
|
||||
f.state, f.reason = "IDLE", ""
|
||||
} else {
|
||||
f.state, f.reason = "IDLE+DRAIN", ReasonServing
|
||||
}
|
||||
return nil, nil
|
||||
case strings.HasPrefix(c, "pgrep"):
|
||||
out := "1800358 /usr/sbin/slurmstepd infinity\n" + f.outputs["jobs"]
|
||||
return []byte(out), nil
|
||||
}
|
||||
return []byte(f.outputs[c]), nil
|
||||
}
|
||||
|
||||
func (f *fakeHost) LookPath(name string) (string, error) {
|
||||
if f.bins[name] {
|
||||
return "/usr/sbin/" + name, nil
|
||||
}
|
||||
return "", errors.New("not found")
|
||||
}
|
||||
|
||||
func (f *fakeHost) ReadFile(path string) ([]byte, error) {
|
||||
b, ok := f.files[path]
|
||||
if !ok {
|
||||
return nil, fs.ErrNotExist
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
|
||||
func (f *fakeHost) WriteFile(path string, b []byte, _ fs.FileMode, owner string) error {
|
||||
f.files[path] = append([]byte(nil), b...)
|
||||
f.owners[path] = owner
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeHost) MkdirAll(string, fs.FileMode, string) error { return nil }
|
||||
|
||||
func (f *fakeHost) Remove(path string) error { delete(f.files, path); return nil }
|
||||
|
||||
func (f *fakeHost) Download(_ context.Context, url, path, sha string) error {
|
||||
c := "download " + url + " " + path + " " + sha
|
||||
f.ran = append(f.ran, c)
|
||||
if f.fail[c] || f.fail["download "+url] {
|
||||
return errors.New("fetch failed")
|
||||
}
|
||||
f.files[path] = []byte("fetched " + url)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeHost) count(prefix string) int {
|
||||
n := 0
|
||||
for _, c := range f.ran {
|
||||
if strings.HasPrefix(c, prefix) {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
type fakeInst struct {
|
||||
host *fakeHost
|
||||
got map[string][]string
|
||||
}
|
||||
|
||||
func (i *fakeInst) InstallPackages(_ context.Context, by map[string][]string) error {
|
||||
i.got = by
|
||||
for _, b := range []string{"munged", "scontrol", "slurmd", "slurmctld", "slurmdbd", "mariadbd", "mariadb", "exportfs", "mount.nfs"} {
|
||||
i.host.bins[b] = true
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
var key = []byte(strings.Repeat("k", 1024))
|
||||
|
||||
func nodeMsg(lent bool) *llmgwv1.Slurm {
|
||||
return &llmgwv1.Slurm{Node: true, Name: "cuda01", ConfServer: "10.0.0.146:6817", MungeKey: key, Lent: lent}
|
||||
}
|
||||
|
||||
func TestNodeInstallsAndStaysDrained(t *testing.T) {
|
||||
h := newFake()
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
st := k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if st.GetError() != "" {
|
||||
t.Fatal(st.GetError())
|
||||
}
|
||||
if !slices.Equal(inst.got["deb"], []string{"munge", "slurm-client", "slurmd"}) ||
|
||||
!slices.Equal(inst.got["rpm"], []string{"munge", "slurm", "slurm-slurmd"}) {
|
||||
t.Fatalf("installed %v", inst.got)
|
||||
}
|
||||
if string(h.files[slurm.MungeKeyPath]) != string(key) || h.owners[slurm.MungeKeyPath] != "munge" {
|
||||
t.Fatal("the munge key was not written for munge")
|
||||
}
|
||||
unit := string(h.files[DropIn])
|
||||
if !strings.Contains(unit, "ExecStart=/usr/sbin/slurmd --systemd --conf-server 10.0.0.146:6817 -N cuda01\n") ||
|
||||
!strings.Contains(unit, "ConditionPathExists=\n") {
|
||||
t.Fatalf("drop-in:\n%s", unit)
|
||||
}
|
||||
if !st.GetSlurmdRunning() || st.GetVersion() != "23.11.4" || !strings.HasPrefix(st.GetHardware(), "NodeName=cuda01 CPUs=16") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
// Not lent: left drained, with llmgw's first reason.
|
||||
if h.count("scontrol update") != 0 || st.GetState() != "IDLE+DRAIN" {
|
||||
t.Fatalf("a node not lent was changed: %v, %v", h.ran, st)
|
||||
}
|
||||
if _, ok := h.files["/var/lib/llmgw-agent/slurm"]; !ok {
|
||||
t.Fatal("no marker")
|
||||
}
|
||||
|
||||
// Again, nothing changed: no install, no restart.
|
||||
h.ran = nil
|
||||
inst.got = nil
|
||||
k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if inst.got != nil || h.count("systemctl restart") != 0 || h.count("systemctl enable") != 0 {
|
||||
t.Fatalf("a second pass did work: %v", h.ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLendAndReclaim(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
st := k.Reconcile(context.Background(), nodeMsg(true))
|
||||
if h.count("scontrol update NodeName=cuda01 State=RESUME") != 1 || st.GetState() != "IDLE" {
|
||||
t.Fatalf("a lent node was not resumed: %v %v", h.ran, st)
|
||||
}
|
||||
h.outputs["jobs"] = "2001 slurmstepd: [101.batch]\n2002 slurmstepd: [102.0]\n"
|
||||
st = k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if h.count("scontrol update NodeName=cuda01 State=DRAIN Reason="+ReasonServing) != 1 {
|
||||
t.Fatalf("a node taken back was not drained: %v", h.ran)
|
||||
}
|
||||
if st.GetJobs() != 2 || !strings.Contains(st.GetState(), "DRAIN") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
// Drained already: not drained again.
|
||||
h.ran = nil
|
||||
k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if h.count("scontrol update") != 0 {
|
||||
t.Fatalf("drained twice: %v", h.ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminsDrainIsLeft(t *testing.T) {
|
||||
h := newFake()
|
||||
h.reason = "bad DIMM, ticket 42"
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
k.Reconcile(context.Background(), nodeMsg(true))
|
||||
if h.count("scontrol update") != 0 {
|
||||
t.Fatalf("llmgw resumed a node a Slurm administrator drained: %v", h.ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClientEnvOnANodeWithoutFiles(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
if env := k.clientEnv(); len(env) != 1 || !strings.HasSuffix(env[0], "conf-cache/slurm.conf") {
|
||||
t.Fatalf("env %v", env)
|
||||
}
|
||||
h.files[filepath.Join(slurm.ConfDir, "slurm.conf")] = []byte("x")
|
||||
if env := k.clientEnv(); env != nil {
|
||||
t.Fatalf("env %v on the controller", env)
|
||||
}
|
||||
}
|
||||
|
||||
func TestController(t *testing.T) {
|
||||
h := newFake()
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/m", nil)
|
||||
files := map[string]string{"slurm.conf": "ClusterName=llmgw\n", "gres.conf": "#\n", "cgroup.conf": "#\n"}
|
||||
m := &llmgwv1.Slurm{Controller: true, MungeKey: key, Files: files}
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
if st.GetError() != "" {
|
||||
t.Fatal(st.GetError())
|
||||
}
|
||||
if !slices.Contains(inst.got["deb"], "slurmctld") || slices.Contains(inst.got["deb"], "slurmd") {
|
||||
t.Fatalf("installed %v", inst.got)
|
||||
}
|
||||
if string(h.files["/etc/slurm/slurm.conf"]) != "ClusterName=llmgw\n" {
|
||||
t.Fatal("slurm.conf not written")
|
||||
}
|
||||
if h.count("systemctl restart slurmctld") != 1 || h.count("scontrol reconfigure") != 1 {
|
||||
t.Fatalf("slurmctld not started on its files: %v", h.ran)
|
||||
}
|
||||
if !st.GetSlurmctldRunning() || len(st.GetFilesSha256()) != 64 {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
// Accounting: MariaDB on this machine alone, the database, slurmdbd on
|
||||
// it, the cluster added, all before slurmctld.
|
||||
if !slices.Contains(inst.got["deb"], "slurmdbd") || !slices.Contains(inst.got["deb"], "mariadb-server") {
|
||||
t.Fatalf("installed %v", inst.got)
|
||||
}
|
||||
if !strings.Contains(string(h.files["/etc/mysql/mariadb.conf.d/90-llmgw-slurm.cnf"]), "bind-address=127.0.0.1") {
|
||||
t.Fatal("MariaDB, installed by llmgw, not kept to this machine")
|
||||
}
|
||||
dbd := string(h.files[slurm.DbdConfPath])
|
||||
if !strings.Contains(dbd, "DbdHost=node06\n") || !strings.Contains(dbd, "StoragePass="+slurm.DbdPassword(key)+"\n") ||
|
||||
h.owners[slurm.DbdConfPath] != "slurm" {
|
||||
t.Fatalf("slurmdbd.conf for %q:\n%s", h.owners[slurm.DbdConfPath], dbd)
|
||||
}
|
||||
if _, ok := h.files[acctSQLPath]; ok {
|
||||
t.Fatal("the database's statements, password and all, were left on disk")
|
||||
}
|
||||
if strings.Contains(strings.Join(h.ran, "\n"), slurm.DbdPassword(key)) {
|
||||
t.Fatal("the password was on a command line")
|
||||
}
|
||||
order := strings.Join(h.ran, "\n")
|
||||
if !strings.Contains(order, "sacctmgr -i add cluster llmgw") ||
|
||||
strings.Index(order, "slurmdbd") > strings.Index(order, "systemctl restart slurmctld") {
|
||||
t.Fatalf("accounting not before slurmctld: %v", h.ran)
|
||||
}
|
||||
if !st.GetSlurmdbdRunning() {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
sum := st.GetFilesSha256()
|
||||
|
||||
h.ran = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
if h.count("systemctl restart") != 0 {
|
||||
t.Fatalf("restarted on the same files: %v", h.ran)
|
||||
}
|
||||
|
||||
files["gres.conf"] = "NodeName=cuda01 Name=gpu File=/dev/nvidia0\n"
|
||||
h.ran = nil
|
||||
st = k.Reconcile(context.Background(), m)
|
||||
if h.count("systemctl restart slurmctld") != 1 || st.GetFilesSha256() == sum {
|
||||
t.Fatalf("a changed file did not restart slurmctld: %v", h.ran)
|
||||
}
|
||||
|
||||
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Controller: true, MungeKey: key,
|
||||
Files: map[string]string{"slurm.conf": "x", "../evil": "y"}}); !strings.Contains(st.GetError(), "named") {
|
||||
t.Fatalf("a path in a file name was taken: %v", st)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTooOld(t *testing.T) {
|
||||
h := newFake()
|
||||
h.outputs["slurmd -V"] = "slurm-wlm 21.08.5\n"
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
st := k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if !strings.Contains(st.GetError(), "older than 22.05") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if _, ok := h.files[slurm.MungeKeyPath]; ok {
|
||||
t.Fatal("went on with a Slurm too old")
|
||||
}
|
||||
}
|
||||
|
||||
func TestStop(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
k.Reconcile(context.Background(), nodeMsg(false))
|
||||
h.ran = nil
|
||||
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{}); st != nil {
|
||||
t.Fatalf("status %v for no part", st)
|
||||
}
|
||||
if !slices.Contains(h.ran, "systemctl disable --now slurmd") {
|
||||
t.Fatalf("slurmd not stopped: %v", h.ran)
|
||||
}
|
||||
if _, ok := h.files[DropIn]; ok {
|
||||
t.Fatal("drop-in left")
|
||||
}
|
||||
// Never started here: nothing touched.
|
||||
h2 := newFake()
|
||||
k2 := New(h2, &fakeInst{host: h2}, "/m", nil)
|
||||
k2.Reconcile(context.Background(), &llmgwv1.Slurm{})
|
||||
if len(h2.ran) != 0 {
|
||||
t.Fatalf("a machine llmgw never ran Slurm on was touched: %v", h2.ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBadNames(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
m := nodeMsg(false)
|
||||
m.Name = "cuda01 -c /etc/evil"
|
||||
if st := k.Reconcile(context.Background(), m); !strings.Contains(st.GetError(), "spaces") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Node: true, Name: "a", ConfServer: "b"}); !strings.Contains(st.GetError(), "no key") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMariaDBThatWasThereKeepsItsSettings(t *testing.T) {
|
||||
h := newFake()
|
||||
for _, b := range []string{"munged", "scontrol", "slurmctld", "slurmdbd", "mariadbd", "mariadb", "exportfs", "mount.nfs"} {
|
||||
h.bins[b] = true
|
||||
}
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Controller: true, MungeKey: key,
|
||||
Files: map[string]string{"slurm.conf": "ClusterName=llmgw\n"}})
|
||||
if st.GetError() != "" {
|
||||
t.Fatal(st.GetError())
|
||||
}
|
||||
for p := range h.files {
|
||||
if strings.HasSuffix(p, ".cnf") {
|
||||
t.Fatalf("a MariaDB llmgw did not install had its settings changed: %s", p)
|
||||
}
|
||||
}
|
||||
if h.count("mariadb -e source") != 1 || !st.GetSlurmdbdRunning() {
|
||||
t.Fatalf("the database was not made in the MariaDB that was there: %v", h.ran)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,319 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// usersMarker lists, a line each, the logins llmgw made here: the ones it
|
||||
// may change and take away. A user that was here before is never touched.
|
||||
func (k *Keeper) usersMarker() string { return k.marker + "-users" }
|
||||
|
||||
// users keeps the Linux users want has, the same login and UID as on every
|
||||
// other machine, their keys where sshd reads them, and their homes when
|
||||
// this machine makes them; and takes away those llmgw made that are gone.
|
||||
// Answers how many it has. A user that cannot be made is said, the others
|
||||
// are made anyway.
|
||||
func (k *Keeper) users(ctx context.Context, want *llmgwv1.Slurm) (int32, error) {
|
||||
made := k.madeUsers()
|
||||
if !want.GetManageUsers() {
|
||||
return 0, k.removeUsers(ctx, made, nil)
|
||||
}
|
||||
var errs []error
|
||||
var keep []string
|
||||
n := int32(0)
|
||||
for _, u := range want.GetUsers() {
|
||||
err := k.user(ctx, u, slices.Contains(made, u.GetLogin()), want.GetMakeHomes())
|
||||
if errors.Is(err, errNotMade) {
|
||||
errs = append(errs, err)
|
||||
continue
|
||||
}
|
||||
keep = append(keep, u.GetLogin())
|
||||
if err != nil {
|
||||
errs = append(errs, err)
|
||||
continue
|
||||
}
|
||||
n++
|
||||
}
|
||||
if err := k.keepMade(keep); err != nil {
|
||||
return n, err
|
||||
}
|
||||
if err := k.removeUsers(ctx, made, keep); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
if err := k.sshConf(ctx, len(keep) > 0); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
// The controller is where they log in and submit from; a node only
|
||||
// while a job of theirs runs there.
|
||||
if err := k.nodeLogin(ctx, len(keep) > 0 && want.GetNode() && !want.GetController()); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
return n, errors.Join(errs...)
|
||||
}
|
||||
|
||||
// nodeLogin holds the users llmgw made here to their jobs when they log in
|
||||
// (pam_slurm_adopt), or no longer. Only they are: root and the site's
|
||||
// administrators log in as before, so a mistake here never locks out the
|
||||
// people who would put it right.
|
||||
func (k *Keeper) nodeLogin(ctx context.Context, on bool) error {
|
||||
old, err := k.host.ReadFile(slurm.PAMSSHD)
|
||||
if err != nil {
|
||||
if !on {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("reading %s: %w", slurm.PAMSSHD, err)
|
||||
}
|
||||
if on {
|
||||
if err := k.adoptModule(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
read := func(name string) (string, error) {
|
||||
b, err := k.host.ReadFile(filepath.Join(filepath.Dir(slurm.PAMSSHD), name))
|
||||
return string(b), err
|
||||
}
|
||||
stack, contained, err := slurm.PAMAdopt(string(old), k.usersMarker(), read, on)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if stack == string(old) {
|
||||
return nil
|
||||
}
|
||||
if on {
|
||||
k.log.Info("llmgw's Slurm users log in to this node only while a job of theirs runs here",
|
||||
"in_the_job", contained)
|
||||
if !contained {
|
||||
k.log.Warn("pam_systemd could not be skipped for them safely here: their ssh sessions are not held " +
|
||||
"to their job's cores and cards, only to having one")
|
||||
}
|
||||
} else {
|
||||
k.log.Info("llmgw's Slurm users are no longer held to their jobs when they log in here")
|
||||
}
|
||||
// Read by sshd for each login: nothing to reload.
|
||||
_, err = k.write(slurm.PAMSSHD, []byte(stack), 0o644, "")
|
||||
return err
|
||||
}
|
||||
|
||||
// adoptModule is there, or is installed.
|
||||
func (k *Keeper) adoptModule(ctx context.Context) error {
|
||||
if k.installed[slurm.AdoptModule] {
|
||||
return nil
|
||||
}
|
||||
find := func() bool {
|
||||
for _, d := range slurm.ModuleDirs {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-f", filepath.Join(d, slurm.AdoptModule)); err == nil {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
if !find() {
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("%s missing, and this agent installs nothing", slurm.AdoptModule)
|
||||
}
|
||||
k.log.Info("installing pam_slurm_adopt, which holds llmgw's Slurm users to their jobs here")
|
||||
if err := k.inst.InstallPackages(ctx, slurm.AdoptPackages); err != nil {
|
||||
return err
|
||||
}
|
||||
if !find() {
|
||||
return fmt.Errorf("%s is still missing after installing %s", slurm.AdoptModule,
|
||||
strings.Join(slurm.AdoptPackages["deb"], ", "))
|
||||
}
|
||||
}
|
||||
k.installed[slurm.AdoptModule] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// errNotMade is a user llmgw could not make here, nor took for its own.
|
||||
var errNotMade = errors.New("not made")
|
||||
|
||||
type notMade struct{ msg string }
|
||||
|
||||
func (e notMade) Error() string { return e.msg }
|
||||
func (e notMade) Is(target error) bool { return target == errNotMade }
|
||||
|
||||
// user keeps one user; ours is whether llmgw made it here before.
|
||||
func (k *Keeper) user(ctx context.Context, u *llmgwv1.SlurmUser, ours, makeHome bool) error {
|
||||
login, uid, home := u.GetLogin(), u.GetUid(), u.GetHome()
|
||||
if !slurm.ValidLogin(login) || uid < slurm.MinUID || !filepath.IsAbs(home) || strings.Contains(home, "..") {
|
||||
return notMade{fmt.Sprintf("the leader sent a user %q, UID %d, home %q", login, uid, home)}
|
||||
}
|
||||
id := strconv.FormatUint(uint64(uid), 10)
|
||||
gecos := strings.Map(func(r rune) rune {
|
||||
if r == ':' || r == ',' || r == '\n' || r < ' ' {
|
||||
return -1
|
||||
}
|
||||
return r
|
||||
}, u.GetName())
|
||||
byName, okName := k.passwd(ctx, login)
|
||||
byID, okID := k.passwd(ctx, id)
|
||||
switch {
|
||||
case okName && byName[2] != id:
|
||||
return notMade{fmt.Sprintf("%s is already a user here with UID %s, not llmgw's %s", login, byName[2], id)}
|
||||
case !okName && okID:
|
||||
return notMade{fmt.Sprintf("UID %s is already %s's here, so %s is not made", id, byID[0], login)}
|
||||
case !okName:
|
||||
if _, ok := k.group(ctx, login); !ok {
|
||||
if out, err := k.host.Run(ctx, nil, "groupadd", "-g", id, login); err != nil {
|
||||
return notMade{fmt.Sprintf("groupadd %s: %v: %s", login, err, lastLine(out))}
|
||||
}
|
||||
}
|
||||
k.log.Info("a Slurm user is made here, as llmgw's accounts say", "login", login, "uid", uid)
|
||||
if out, err := k.host.Run(ctx, nil, "useradd", "-u", id, "-g", id, "-M", "-d", home, "-s", "/bin/bash",
|
||||
"-c", gecos, login); err != nil {
|
||||
return notMade{fmt.Sprintf("useradd %s: %v: %s", login, err, lastLine(out))}
|
||||
}
|
||||
case ours && (byName[4] != gecos || byName[5] != home):
|
||||
if out, err := k.host.Run(ctx, nil, "usermod", "-c", gecos, "-d", home, login); err != nil {
|
||||
return fmt.Errorf("usermod %s: %v: %s", login, err, lastLine(out))
|
||||
}
|
||||
}
|
||||
keys := strings.Join(u.GetSshKeys(), "\n")
|
||||
file := filepath.Join(slurm.AuthorizedKeysDir, login)
|
||||
if keys == "" {
|
||||
_ = k.host.Remove(file)
|
||||
} else if _, err := k.write(file, []byte(keys+"\n"), 0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
if makeHome {
|
||||
return k.home(ctx, login, id, home)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// home makes a user's home directory, from /etc/skel, when it is not there.
|
||||
func (k *Keeper) home(ctx context.Context, login, id, home string) error {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-d", home); err == nil {
|
||||
return nil
|
||||
}
|
||||
if err := k.host.MkdirAll(filepath.Dir(home), 0o755, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
k.log.Info("a Slurm user's home is made", "login", login, "home", home)
|
||||
for _, c := range [][]string{
|
||||
{"install", "-d", "-m", "0700", "-o", id, "-g", id, home},
|
||||
{"cp", "-rT", "/etc/skel", home},
|
||||
{"chown", "-R", id + ":" + id, home},
|
||||
} {
|
||||
if out, err := k.host.Run(ctx, nil, c[0], c[1:]...); err != nil {
|
||||
return fmt.Errorf("making %s's home: %s: %v: %s", login, c[0], err, lastLine(out))
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// removeUsers takes away the users llmgw made here that are not in keep.
|
||||
// Their homes and files stay; their UID is never given to another.
|
||||
func (k *Keeper) removeUsers(ctx context.Context, made, keep []string) error {
|
||||
var errs []error
|
||||
left := slices.Clone(keep)
|
||||
for _, login := range made {
|
||||
if slices.Contains(keep, login) {
|
||||
continue
|
||||
}
|
||||
k.log.Info("a Slurm user llmgw made is taken away; its home stays", "login", login)
|
||||
_ = k.host.Remove(filepath.Join(slurm.AuthorizedKeysDir, login))
|
||||
if _, ok := k.passwd(ctx, login); ok {
|
||||
if out, err := k.host.Run(ctx, nil, "userdel", login); err != nil {
|
||||
// Still logged in, or running something: tried again.
|
||||
errs = append(errs, fmt.Errorf("userdel %s: %v: %s", login, err, lastLine(out)))
|
||||
left = append(left, login)
|
||||
continue
|
||||
}
|
||||
}
|
||||
}
|
||||
if !slices.Equal(left, made) {
|
||||
if err := k.keepMade(left); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
}
|
||||
if len(left) == 0 {
|
||||
if err := k.sshConf(ctx, false); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
if err := k.nodeLogin(ctx, false); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
}
|
||||
return errors.Join(errs...)
|
||||
}
|
||||
|
||||
func (k *Keeper) madeUsers() []string {
|
||||
b, err := k.host.ReadFile(k.usersMarker())
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return strings.Fields(string(b))
|
||||
}
|
||||
|
||||
func (k *Keeper) keepMade(logins []string) error {
|
||||
s := slices.Clone(logins)
|
||||
slices.Sort(s)
|
||||
s = slices.Compact(s)
|
||||
if len(s) == 0 {
|
||||
return k.host.Remove(k.usersMarker())
|
||||
}
|
||||
_, err := k.write(k.usersMarker(), []byte(strings.Join(s, "\n")+"\n"), 0o644, "")
|
||||
return err
|
||||
}
|
||||
|
||||
// sshConf has sshd read the users' keys from AuthorizedKeysDir, or no
|
||||
// longer.
|
||||
func (k *Keeper) sshConf(ctx context.Context, on bool) error {
|
||||
if !on {
|
||||
if _, err := k.host.ReadFile(slurm.SSHDropIn); err != nil {
|
||||
return nil
|
||||
}
|
||||
_ = k.host.Remove(slurm.SSHDropIn)
|
||||
return k.reloadSSH(ctx)
|
||||
}
|
||||
changed, err := k.write(slurm.SSHDropIn, []byte(slurm.SSHConf), 0o644, "")
|
||||
if err != nil || !changed {
|
||||
return err
|
||||
}
|
||||
if out, err := k.host.Run(ctx, nil, "sshd", "-t"); err != nil {
|
||||
_ = k.host.Remove(slurm.SSHDropIn)
|
||||
return fmt.Errorf("sshd refuses llmgw's setting for the users' keys, so it is not kept: %v: %s", err, lastLine(out))
|
||||
}
|
||||
k.log.Info("sshd reads the Slurm users' keys from llmgw too", "dir", slurm.AuthorizedKeysDir)
|
||||
return k.reloadSSH(ctx)
|
||||
}
|
||||
|
||||
// reloadSSH has sshd read its settings again, under Debian's name for it
|
||||
// or Red Hat's; one that is not running reads them when it starts.
|
||||
func (k *Keeper) reloadSSH(ctx context.Context) error {
|
||||
for _, u := range []string{"ssh", "sshd"} {
|
||||
if k.active(ctx, u) {
|
||||
return k.systemctl(ctx, "reload", u)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// passwd is getent's line for a user, by login or UID, in its fields.
|
||||
func (k *Keeper) passwd(ctx context.Context, key string) ([]string, bool) {
|
||||
out, err := k.host.Run(ctx, nil, "getent", "passwd", key)
|
||||
f := strings.Split(strings.TrimSpace(string(out)), ":")
|
||||
if err != nil || len(f) < 7 {
|
||||
return nil, false
|
||||
}
|
||||
return f, true
|
||||
}
|
||||
|
||||
func (k *Keeper) group(ctx context.Context, key string) ([]string, bool) {
|
||||
out, err := k.host.Run(ctx, nil, "getent", "group", key)
|
||||
f := strings.Split(strings.TrimSpace(string(out)), ":")
|
||||
if err != nil || len(f) < 3 {
|
||||
return nil, false
|
||||
}
|
||||
return f, true
|
||||
}
|
||||
@@ -33,6 +33,17 @@ func CachePathFor(dir, source string) string {
|
||||
return filepath.Join(dir, hex.EncodeToString(sum[:8])+"-"+name)
|
||||
}
|
||||
|
||||
// cacheSource is the source a cache name is computed from: a set's address
|
||||
// without its trailing slash, as its directory was named when it was fetched.
|
||||
// Computed from the slash-ended address, every set in the catalog looked like
|
||||
// nobody's, and a sweep offered 330 GB of models still in use for deletion.
|
||||
func cacheSource(source string) string {
|
||||
if IsSet(source) {
|
||||
return strings.TrimSuffix(source, "/")
|
||||
}
|
||||
return source
|
||||
}
|
||||
|
||||
// Owned reports whether a path is one this package created.
|
||||
//
|
||||
// The dividing line of the whole feature. A file without the cache name was
|
||||
@@ -65,7 +76,7 @@ func Orphans(have []string, wanted []string) []string {
|
||||
// By base name rather than full path: the control plane does not know
|
||||
// each node's cache directory, and the name is what identifies the
|
||||
// file wherever the node keeps it.
|
||||
keep[filepath.Base(CachePathFor("", src))] = struct{}{}
|
||||
keep[filepath.Base(CachePathFor("", cacheSource(src)))] = struct{}{}
|
||||
}
|
||||
|
||||
var out []string
|
||||
|
||||
@@ -51,7 +51,7 @@ type manifest struct {
|
||||
// as a broken model rather than failing to load, so a partial set is left
|
||||
// unfinished under its temporary name and the next attempt starts again.
|
||||
func (f *Fetcher) ensureSet(ctx context.Context, source string, allowPickle bool) (string, error) {
|
||||
dir, err := f.cachePath(strings.TrimSuffix(source, "/"))
|
||||
dir, err := f.cachePath(cacheSource(source))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
@@ -74,6 +74,23 @@ func TestASetIsFetchedIntoOneDirectory(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestASetInTheCatalogIsNotAnOrphan(t *testing.T) {
|
||||
// The catalog names a set by its address with the trailing slash; the
|
||||
// fetch names the directory without it. A sweep that computed the name
|
||||
// from the catalog's spelling offered every set in use for deletion.
|
||||
f, srv, _ := mirrorOf(t, map[string]string{"model.safetensors": "aaaa", "config.json": "{}"})
|
||||
dir, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := Orphans([]string{dir}, []string{srv.URL + "/m/v/"}); len(got) != 0 {
|
||||
t.Errorf("a set the catalog still names was called an orphan: %v", got)
|
||||
}
|
||||
if got := Orphans([]string{dir}, nil); len(got) != 1 {
|
||||
t.Errorf("a set nothing names was not found: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAFinishedSetIsNotFetchedAgain(t *testing.T) {
|
||||
f, srv, hits := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
|
||||
|
||||
|
||||
@@ -247,6 +247,8 @@ var liveSettings = []string{
|
||||
"presets",
|
||||
// Handed to the reconciler as they change.
|
||||
"on_demand",
|
||||
// Read by the Slurm service each pass, kicked when they change.
|
||||
"slurm",
|
||||
// Read by server init, node add and backups, not by the server.
|
||||
"backup_recipients", "backup_copy_to",
|
||||
}
|
||||
|
||||
+15
-1
@@ -189,6 +189,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newMCPCmd(client, &asJSON),
|
||||
newKBCmd(client, &asJSON),
|
||||
newTrainCmd(client, &asJSON),
|
||||
newSlurmCmd(client, &asJSON),
|
||||
newConfigCmd(client, &asJSON),
|
||||
newAuditCmd(client, &asJSON),
|
||||
newNotifyCmd(client, &asJSON),
|
||||
@@ -418,7 +419,7 @@ func newModelsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
}
|
||||
fmt.Fprintf(w, "%s\t%s\t%d/%d\t%s\t%s\t%s\t%s\t%s\t%s\n",
|
||||
name, serving, m.Ready, m.Min, modelStatus(m), m.Class, reserves(m),
|
||||
join(m.Aliases), join(m.Profiles), join(m.Fallbacks))
|
||||
join(m.Aliases), join(profilesWithCounts(m)), join(m.Fallbacks))
|
||||
for _, x := range m.Warnings {
|
||||
warnings = append(warnings, m.Name+": "+x)
|
||||
}
|
||||
@@ -696,3 +697,16 @@ func reserves(m adminapi.ModelView) string {
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// profilesWithCounts are a model's profiles, each with its replicas when
|
||||
// they are counted per profile: gpu=7, cpu=4.
|
||||
func profilesWithCounts(m adminapi.ModelView) []string {
|
||||
if len(m.ProfileReplicas) == 0 {
|
||||
return m.Profiles
|
||||
}
|
||||
out := make([]string, len(m.Profiles))
|
||||
for i, p := range m.Profiles {
|
||||
out[i] = fmt.Sprintf("%s=%d", p, m.ProfileReplicas[p])
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,477 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/url"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"text/tabwriter"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
// newSlurmCmd is the Slurm cluster run beside llmgw (BACKLOG P): set up on
|
||||
// nodes llmgw already has, and nodes handed between the two.
|
||||
func newSlurmCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
cmd := &cobra.Command{
|
||||
Use: "slurm",
|
||||
Short: "A Slurm cluster beside llmgw, for training and other batch work",
|
||||
Long: "Slurm runs on llmgw's own nodes, installed and kept by their agents: slurmctld on one node,\n" +
|
||||
"slurmd on those named. A node is llmgw's, drained in Slurm, until it is lent; lent, its\n" +
|
||||
"replicas move off first, then Slurm may run jobs there. Taken back, Slurm drains it and\n" +
|
||||
"llmgw has it again when its last job ends. Jobs are submitted on the controller node\n" +
|
||||
"(sbatch, srun) by its own users, as on any Slurm cluster.",
|
||||
}
|
||||
cmd.AddCommand(newSlurmStatusCmd(client, asJSON), newSlurmSetupCmd(client), newSlurmOffCmd(client),
|
||||
newSlurmListCmd(client, "lend", "Let Slurm run jobs on nodes; llmgw's replicas move off first", true),
|
||||
newSlurmListCmd(client, "reclaim", "Take nodes back from Slurm; llmgw has them when their jobs end", false),
|
||||
newSlurmSharedCmd(client), newSlurmUsersCmd(client, asJSON), newSlurmContainersCmd(client))
|
||||
return cmd
|
||||
}
|
||||
|
||||
func newSlurmContainersCmd(client func() *ctlClient) *cobra.Command {
|
||||
cmd := &cobra.Command{
|
||||
Use: "containers",
|
||||
Short: "Jobs in containers: srun --container-image (pyxis and enroot)",
|
||||
Long: "On, the agents install enroot and pyxis on every machine of the Slurm cluster (from NVIDIA's\n" +
|
||||
"releases, at versions llmgw pins and checks) and, on GPU machines, nvidia-container-cli; once\n" +
|
||||
"every machine has them, Slurm loads pyxis and jobs may run in a container:\n\n" +
|
||||
" srun --gres=gpu:1 --container-image=nvcr.io#nvidia/pytorch:25.01-py3 python train.py\n\n" +
|
||||
"The shared path is in every container where it is on the host. Off, Slurm stops loading pyxis;\n" +
|
||||
"what was installed stays.",
|
||||
}
|
||||
var note string
|
||||
toggle := func(use, short string, on bool) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: use,
|
||||
Short: short,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
if !on {
|
||||
if doc.Node([]string{"slurm", "containers"}) != nil {
|
||||
doc.Unset([]string{"slurm", "containers"})
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return doc.SetYAML([]string{"slurm", "containers"}, "true")
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
cmd.AddCommand(toggle("on", "Install pyxis and enroot everywhere, and let jobs run in containers", true),
|
||||
toggle("off", "Slurm stops loading pyxis; what was installed stays", false))
|
||||
return cmd
|
||||
}
|
||||
|
||||
func newSlurmSharedCmd(client func() *ctlClient) *cobra.Command {
|
||||
var path, server, nfs, note string
|
||||
var off bool
|
||||
c := &cobra.Command{
|
||||
Use: "shared --path DIR (--server NODE | --nfs HOST:/EXPORT) | --off",
|
||||
Short: "Files every machine of the Slurm cluster sees at one path",
|
||||
Long: "Home directories, datasets, the jobs' output and checkpoints, at the same path on the controller\n" +
|
||||
"and every node. Three ways:\n" +
|
||||
" --nfs HOST:/EXPORT the site's own NFS export, mounted at --path on every machine;\n" +
|
||||
" --server NODE an llmgw node (the controller or a node) exports --path over NFS to the\n" +
|
||||
" others, for a site with no storage of its own: one machine, not highly\n" +
|
||||
" available, no faster than its disk and network;\n" +
|
||||
" neither the site mounts --path itself (Lustre, BeeGFS, GPFS); llmgw only looks\n" +
|
||||
" that it is there.\n" +
|
||||
"The export answers the Slurm machines alone, NFS's port the cluster's networks. A machine\n" +
|
||||
"that gives up its mount does so when no job holds files on it.",
|
||||
Example: " llmgw ctl slurm shared --path /shared --server node06\n" +
|
||||
" llmgw ctl slurm shared --path /shared --nfs filer.site:/export/hpc\n" +
|
||||
" llmgw ctl slurm shared --path /lustre/hpc\n" +
|
||||
" llmgw ctl slurm shared --off",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
if off == (path != "") {
|
||||
return fmt.Errorf("give --path, or --off")
|
||||
}
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
if off {
|
||||
if doc.Node([]string{"slurm", "shared"}) != nil {
|
||||
doc.Unset([]string{"slurm", "shared"})
|
||||
}
|
||||
return nil
|
||||
}
|
||||
v := map[string]string{"path": path}
|
||||
if server != "" {
|
||||
v["server"] = server
|
||||
}
|
||||
if nfs != "" {
|
||||
v["nfs"] = nfs
|
||||
}
|
||||
b, _ := json.Marshal(v)
|
||||
return doc.SetYAML([]string{"slurm", "shared"}, string(b))
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(&path, "path", "", "where every machine sees the files, as /shared")
|
||||
c.Flags().StringVar(&server, "server", "", "the llmgw node that exports them over NFS")
|
||||
c.Flags().StringVar(&nfs, "nfs", "", "the site's NFS export, HOST:/EXPORT")
|
||||
c.Flags().BoolVar(&off, "off", false, "no shared files; mounts and exports llmgw made are taken away")
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
|
||||
func newSlurmUsersCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
cmd := &cobra.Command{
|
||||
Use: "users",
|
||||
Short: "The Linux users that run jobs, made from llmgw's accounts",
|
||||
Long: "With slurm.users \"llmgw\" every active account is a Linux user on every machine of the Slurm\n" +
|
||||
"cluster: its login from its e-mail address, a UID of llmgw's from 200000 up (never given to\n" +
|
||||
"another, even when the account goes), its home under the shared path. It logs in over SSH\n" +
|
||||
"with the keys given in the portal (Profile) or here. An account suspended or deleted loses\n" +
|
||||
"its user; its home and files stay. Off, users are the site's: from its directory (sssd), or\n" +
|
||||
"made by hand.",
|
||||
}
|
||||
list := &cobra.Command{
|
||||
Use: "list",
|
||||
Short: "Every user, with its login, UID and how many keys it has",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var us []slurmsvc.User
|
||||
if err := client().get(cmd.Context(), "/api/v1/slurm/users", &us); err != nil {
|
||||
return err
|
||||
}
|
||||
w := cmd.OutOrStdout()
|
||||
if *asJSON {
|
||||
return json.NewEncoder(w).Encode(us)
|
||||
}
|
||||
tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
|
||||
fmt.Fprintln(tw, "LOGIN\tUID\tACCOUNT\tSSH KEYS")
|
||||
for _, u := range us {
|
||||
fmt.Fprintf(tw, "%s\t%d\t%s\t%d\n", u.Login, u.UID, u.Email, len(u.SSHKeys))
|
||||
}
|
||||
return tw.Flush()
|
||||
},
|
||||
}
|
||||
var note string
|
||||
toggle := func(use, short, value string) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: use,
|
||||
Short: short,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
if value == "" {
|
||||
if doc.Node([]string{"slurm", "users"}) != nil {
|
||||
doc.Unset([]string{"slurm", "users"})
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return doc.SetYAML([]string{"slurm", "users"}, yamlString(value))
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
keys := &cobra.Command{
|
||||
Use: "keys ACCOUNT FILE",
|
||||
Short: "Replace an account's SSH public keys, from a file (- for standard input)",
|
||||
Long: "FILE holds OpenSSH public keys, one a line, as ~/.ssh/id_ed25519.pub; an empty one takes them all away.",
|
||||
Example: " llmgw ctl slurm users keys ayse@site.example ayse.pub\n" +
|
||||
" ssh-add -L | llmgw ctl slurm users keys ayse@site.example -",
|
||||
Args: cobra.ExactArgs(2),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var b []byte
|
||||
var err error
|
||||
if args[1] == "-" {
|
||||
b, err = io.ReadAll(io.LimitReader(cmd.InOrStdin(), 64<<10))
|
||||
} else {
|
||||
b, err = os.ReadFile(args[1])
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
body, _ := json.Marshal(map[string]string{"keys": string(b)})
|
||||
var u slurmsvc.User
|
||||
if err := client().put(cmd.Context(), "/api/v1/slurm/users/"+url.PathEscape(args[0])+"/ssh-keys", body, &u); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s (UID %d) has %d SSH key(s); the machines have them within a minute\n",
|
||||
u.Login, u.UID, len(u.SSHKeys))
|
||||
return nil
|
||||
},
|
||||
}
|
||||
cmd.AddCommand(list,
|
||||
toggle("on", "Make a Linux user for every active account, on every machine of the Slurm cluster", "llmgw"),
|
||||
toggle("off", "Leave users to the site; those llmgw made are taken away, their homes left", ""),
|
||||
keys)
|
||||
return cmd
|
||||
}
|
||||
|
||||
func newSlurmStatusCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "status",
|
||||
Short: "The Slurm cluster, and where each of its nodes is",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var st slurmsvc.Status
|
||||
if err := client().get(cmd.Context(), "/api/v1/slurm", &st); err != nil {
|
||||
return err
|
||||
}
|
||||
w := cmd.OutOrStdout()
|
||||
if *asJSON {
|
||||
return json.NewEncoder(w).Encode(st)
|
||||
}
|
||||
if !st.Enabled {
|
||||
fmt.Fprintln(w, "Slurm is off. `llmgw ctl slurm setup --controller NODE --nodes NODE,...` turns it on.")
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintf(w, "cluster %s, partition %s, controller %s", st.Cluster, st.Partition, st.Controller)
|
||||
switch c := st.Ctld; {
|
||||
case c == nil:
|
||||
fmt.Fprintln(w, ": not heard from yet")
|
||||
case c.Error != "":
|
||||
fmt.Fprintf(w, ": %s\n", c.Error)
|
||||
case c.SlurmctldRunning && c.SlurmdbdRunning:
|
||||
fmt.Fprintf(w, ": slurmctld %s and slurmdbd running\n", c.Version)
|
||||
case c.SlurmctldRunning:
|
||||
fmt.Fprintf(w, ": slurmctld %s running, slurmdbd not (the jobs' records)\n", c.Version)
|
||||
default:
|
||||
fmt.Fprintln(w, ": slurmctld not running")
|
||||
}
|
||||
fmt.Fprintf(w, "shared files: %s\n", sharedLine(st.Shared))
|
||||
if st.Users != "" {
|
||||
fmt.Fprintf(w, "users: %d, made from llmgw's accounts\n", st.UserCount)
|
||||
}
|
||||
if line := containersLine(st); line != "" {
|
||||
fmt.Fprintf(w, "containers: %s\n", line)
|
||||
}
|
||||
tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
|
||||
fmt.Fprintln(tw, "NODE\tPHASE\tSLURM\tJOBS\tSLURMD\tSHARED\tNOTE")
|
||||
for _, n := range st.Nodes {
|
||||
state, jobs, slurmd, shared, note := "-", "-", "-", "-", n.Describe()
|
||||
if r := n.Status; r != nil {
|
||||
state = or(r.State, "-")
|
||||
jobs = fmt.Sprint(r.Jobs)
|
||||
slurmd = map[bool]string{true: "running", false: "stopped"}[r.SlurmdRunning]
|
||||
shared = or(r.Shared, "-")
|
||||
if r.Error != "" {
|
||||
note = r.Error
|
||||
}
|
||||
}
|
||||
if !n.Connected {
|
||||
note = "agent not connected; " + note
|
||||
}
|
||||
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n", n.Node, n.Phase, state, jobs, slurmd, shared, note)
|
||||
}
|
||||
if err := tw.Flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(st.Waiting) > 0 {
|
||||
fmt.Fprintf(w, "not in slurm.conf until they report their hardware: %s\n", strings.Join(st.Waiting, ", "))
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newSlurmSetupCmd(client func() *ctlClient) *cobra.Command {
|
||||
var controller, cluster, partition, note string
|
||||
var nodes []string
|
||||
var all bool
|
||||
c := &cobra.Command{
|
||||
Use: "setup --controller NODE --nodes NODE,... | --all-nodes",
|
||||
Short: "Run Slurm on llmgw's nodes: slurmctld on one, slurmd on those named",
|
||||
Long: "Turns slurm.enabled on and names the nodes, or with --all-nodes every node. Their agents install munge and Slurm (22.05 or\n" +
|
||||
"newer) from the machines' own package sources; the controller also slurmdbd, the jobs' records, in a\n" +
|
||||
"MariaDB on that machine alone (one that was there keeps its settings). Every node starts drained in Slurm and stays\n" +
|
||||
"llmgw's until `llmgw ctl slurm lend` gives it to Slurm.",
|
||||
Example: " llmgw ctl slurm setup --controller node06 --nodes cuda01,cuda02\n" +
|
||||
" llmgw ctl slurm setup --controller node06 --all-nodes",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
if controller == "" || (len(nodes) == 0 && !all) {
|
||||
return fmt.Errorf("give --controller, and --nodes or --all-nodes")
|
||||
}
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
if err := doc.SetYAML([]string{"slurm", "enabled"}, "true"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := doc.SetYAML([]string{"slurm", "controller"}, yamlString(controller)); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := setList(doc, []string{"slurm", "nodes"}, nodes); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := doc.SetYAML([]string{"slurm", "all_nodes"}, fmt.Sprint(all)); err != nil {
|
||||
return err
|
||||
}
|
||||
// Lent nodes no longer named are not lent.
|
||||
lent := slices.DeleteFunc(docList(doc, []string{"slurm", "lent"}),
|
||||
func(n string) bool { return !all && !slices.Contains(nodes, n) })
|
||||
if err := setList(doc, []string{"slurm", "lent"}, lent); err != nil {
|
||||
return err
|
||||
}
|
||||
for k, v := range map[string]string{"cluster_name": cluster, "partition": partition} {
|
||||
if v != "" {
|
||||
if err := doc.SetYAML([]string{"slurm", k}, yamlString(v)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(&controller, "controller", "", "the node that runs slurmctld")
|
||||
c.Flags().StringSliceVar(&nodes, "nodes", nil, "the nodes that run slurmd")
|
||||
c.Flags().BoolVar(&all, "all-nodes", false, "every node runs slurmd, those that join later too")
|
||||
c.Flags().StringVar(&cluster, "cluster-name", "", "Slurm's ClusterName (llmgw)")
|
||||
c.Flags().StringVar(&partition, "partition", "", "the partition every node is in (gpu)")
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
|
||||
func newSlurmOffCmd(client func() *ctlClient) *cobra.Command {
|
||||
var note string
|
||||
var force bool
|
||||
c := &cobra.Command{
|
||||
Use: "off",
|
||||
Short: "Stop running Slurm; every node is llmgw's again",
|
||||
Long: "slurmd and slurmctld are stopped where llmgw started them; the packages stay. A node still\n" +
|
||||
"lent is refused unless --force: take it back first, so its jobs end before the daemons do.",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
if lent := docList(doc, []string{"slurm", "lent"}); len(lent) > 0 && !force {
|
||||
return fmt.Errorf("%s still lent: `llmgw ctl slurm reclaim` them first, or --force",
|
||||
strings.Join(lent, ", "))
|
||||
}
|
||||
setList(doc, []string{"slurm", "lent"}, nil) //nolint:errcheck // an empty list
|
||||
return doc.SetYAML([]string{"slurm", "enabled"}, "false")
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().BoolVar(&force, "force", false, "even with nodes still lent")
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
|
||||
// containersLine says whether jobs may run in containers, and which
|
||||
// machines are not ready yet; empty when containers are off.
|
||||
func containersLine(st slurmsvc.Status) string {
|
||||
switch st.Containers {
|
||||
case "on":
|
||||
return "on (srun --container-image)"
|
||||
case "":
|
||||
return ""
|
||||
}
|
||||
var waiting []string
|
||||
for _, n := range st.Nodes {
|
||||
if n.Status == nil || n.Status.Containers == "" {
|
||||
waiting = append(waiting, n.Node)
|
||||
}
|
||||
}
|
||||
if st.Ctld == nil || st.Ctld.Containers == "" {
|
||||
if !slices.Contains(waiting, st.Controller) {
|
||||
waiting = append(waiting, st.Controller)
|
||||
}
|
||||
}
|
||||
return "installing pyxis and enroot; jobs take them once every machine has them (waiting: " +
|
||||
strings.Join(waiting, ", ") + ")"
|
||||
}
|
||||
|
||||
// sharedLine says where the shared files are and come from.
|
||||
func sharedLine(sh config.SharedStorage) string {
|
||||
switch {
|
||||
case sh.Path == "":
|
||||
return "none (`llmgw ctl slurm shared` adds them; a batch job's output stays on its node)"
|
||||
case sh.Server != "":
|
||||
return fmt.Sprintf("%s, exported over NFS by %s", sh.Path, sh.Server)
|
||||
case sh.NFS != "":
|
||||
return fmt.Sprintf("%s, mounted from %s", sh.Path, sh.NFS)
|
||||
}
|
||||
return sh.Path + ", mounted by the site"
|
||||
}
|
||||
|
||||
// newSlurmListCmd adds nodes to slurm.lent, or takes them out.
|
||||
func newSlurmListCmd(client func() *ctlClient, use, short string, add bool) *cobra.Command {
|
||||
var note string
|
||||
c := &cobra.Command{
|
||||
Use: use + " NODE...",
|
||||
Short: short,
|
||||
Args: cobra.MinimumNArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
return lendChange(doc, args, add)
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
|
||||
// lendChange adds nodes to slurm.lent, or with add false takes them out.
|
||||
func lendChange(doc *settings.Doc, names []string, add bool) error {
|
||||
nodes := docList(doc, []string{"slurm", "nodes"})
|
||||
all := docBool(doc, []string{"slurm", "all_nodes"})
|
||||
lent := docList(doc, []string{"slurm", "lent"})
|
||||
for _, n := range names {
|
||||
switch {
|
||||
case add && !all && !slices.Contains(nodes, n):
|
||||
return fmt.Errorf("%s does not run slurmd: add it with `llmgw ctl slurm setup --nodes`", n)
|
||||
case add && !slices.Contains(lent, n):
|
||||
lent = append(lent, n)
|
||||
case !add:
|
||||
lent = slices.DeleteFunc(lent, func(l string) bool { return l == n })
|
||||
}
|
||||
}
|
||||
return setList(doc, []string{"slurm", "lent"}, lent)
|
||||
}
|
||||
|
||||
// docList reads a list of strings from the settings, empty when unset.
|
||||
func docList(doc *settings.Doc, path []string) []string {
|
||||
n := doc.Node(path)
|
||||
if n == nil {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
if err := n.Decode(&out); err != nil {
|
||||
return nil
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// docBool reads a boolean from the settings, false when unset.
|
||||
func docBool(doc *settings.Doc, path []string) bool {
|
||||
var b bool
|
||||
if n := doc.Node(path); n != nil {
|
||||
_ = n.Decode(&b)
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func setList(doc *settings.Doc, path []string, list []string) error {
|
||||
if len(list) == 0 {
|
||||
if doc.Node(path) != nil {
|
||||
doc.Unset(path)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return doc.SetYAML(path, yamlList(list))
|
||||
}
|
||||
|
||||
func yamlString(s string) string {
|
||||
b, _ := json.Marshal(s)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func yamlList(list []string) string {
|
||||
b, _ := json.Marshal(list)
|
||||
return string(b)
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
func TestSlurmDocLists(t *testing.T) {
|
||||
doc, err := settings.FromBytes("settings", []byte("slurm:\n enabled: true\n nodes: [cuda01, site2]\n"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := docList(doc, []string{"slurm", "nodes"}); !slices.Equal(got, []string{"cuda01", "site2"}) {
|
||||
t.Fatalf("nodes %v", got)
|
||||
}
|
||||
if got := docList(doc, []string{"slurm", "lent"}); got != nil {
|
||||
t.Fatalf("lent %v", got)
|
||||
}
|
||||
if err := setList(doc, []string{"slurm", "lent"}, []string{"cuda01"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := docList(doc, []string{"slurm", "lent"}); !slices.Equal(got, []string{"cuda01"}) {
|
||||
t.Fatalf("lent %v", got)
|
||||
}
|
||||
if err := setList(doc, []string{"slurm", "lent"}, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := doc.Bytes()
|
||||
if strings.Contains(string(b), "lent") {
|
||||
t.Fatalf("an empty list left:\n%s", b)
|
||||
}
|
||||
}
|
||||
@@ -84,3 +84,19 @@ func TestTheSummaryIsWrittenWhereItIsToldNotToTheTerminal(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestStatusSaysWhenAlertsGoNowhere(t *testing.T) {
|
||||
render := func(s adminapi.Summary) string {
|
||||
cmd := &cobra.Command{}
|
||||
var b bytes.Buffer
|
||||
cmd.SetOut(&b)
|
||||
printSummary(cmd, s)
|
||||
return b.String()
|
||||
}
|
||||
if out := render(adminapi.Summary{NoAlerts: true}); !strings.Contains(out, "ALERTS nowhere") {
|
||||
t.Errorf("no webhook was not called out:\n%s", out)
|
||||
}
|
||||
if out := render(adminapi.Summary{}); strings.Contains(out, "ALERTS") {
|
||||
t.Errorf("an alerts line with webhooks set:\n%s", out)
|
||||
}
|
||||
}
|
||||
+19
-15
@@ -4,6 +4,7 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
@@ -87,13 +88,10 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var results []struct {
|
||||
Node string `json:"node"`
|
||||
Files int `json:"files"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
if err := c.post(cmd.Context(), "/api/v1/weights/collect", body, &results); err != nil {
|
||||
var results []adminapi.CollectResult
|
||||
// The control plane waits for the nodes to list their files without the
|
||||
// deleted ones: half a minute at most.
|
||||
if err := c.patient(time.Minute).post(cmd.Context(), "/api/v1/weights/collect", body, &results); err != nil {
|
||||
return err
|
||||
}
|
||||
if asJSON {
|
||||
@@ -103,7 +101,7 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "Nothing to reclaim.")
|
||||
return nil
|
||||
}
|
||||
var freed int64
|
||||
var freed, left int64
|
||||
for _, r := range results {
|
||||
if r.Error != "" {
|
||||
// Reported rather than fatal: the other nodes' space is still worth
|
||||
@@ -111,14 +109,20 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: %s\n", r.Node, r.Error)
|
||||
continue
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: asked to remove %d files, %s\n",
|
||||
r.Node, r.Files, fileSize(r.Bytes))
|
||||
freed += r.Bytes
|
||||
line := fmt.Sprintf("%s: removed %d files, %s", r.Node, r.Files, fileSize(r.Bytes))
|
||||
if r.Left > 0 {
|
||||
line += fmt.Sprintf("; %d still there, %s", r.Left, fileSize(r.LeftBytes))
|
||||
}
|
||||
fmt.Fprintln(cmd.OutOrStdout(), line)
|
||||
freed, left = freed+r.Bytes, left+r.LeftBytes
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "\n%s freed.\n", fileSize(freed))
|
||||
if left > 0 {
|
||||
// Each node decides: one that keeps a path it cannot prove it
|
||||
// fetched is the feature working. Its agent's log names each.
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s the nodes kept; each node's agent log says why "+
|
||||
"(journalctl -u llmgw-agent).\n", fileSize(left))
|
||||
}
|
||||
// "asked", not "removed": each node decides, and one that refuses a path it
|
||||
// cannot prove it fetched is the feature working rather than failing.
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "\n%s requested. The nodes verify each path before deleting it.\n",
|
||||
fileSize(freed))
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
func TestAPackageInstallThatIsNoEnginesIsNamed(t *testing.T) {
|
||||
if got := provisionLine("", "", "installing", "mariadb-server, munge, with apt-get"); got !=
|
||||
"system packages: installing: mariadb-server, munge, with apt-get" {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
if got := provisionLine("llamacpp", "b10991", "ready", ""); got != "llamacpp b10991: ready" {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoresAloneKeepTheMemory(t *testing.T) {
|
||||
m := catalog.Model{Name: "judge", Profiles: []catalog.Profile{{Name: "cpu", MinMemoryGB: 2}, {Name: "cpu-big", MinMemoryGB: 2}}}
|
||||
if got, err := keptMemory(m, ""); err != nil || got != 2 {
|
||||
t.Fatalf("got %d, %v", got, err)
|
||||
}
|
||||
m.Profiles[1].MinMemoryGB = 4
|
||||
if _, err := keptMemory(m, ""); err == nil {
|
||||
t.Fatal("profiles that differ were given one memory")
|
||||
}
|
||||
if got, err := keptMemory(m, "cpu-big"); err != nil || got != 4 {
|
||||
t.Fatalf("got %d, %v", got, err)
|
||||
}
|
||||
if _, err := keptMemory(m, "gpu"); err == nil {
|
||||
t.Fatal("a profile the model has not was taken")
|
||||
}
|
||||
}
|
||||
+83
-4
@@ -19,6 +19,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
|
||||
)
|
||||
|
||||
@@ -34,7 +35,7 @@ func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
Use: "model",
|
||||
Short: "Add or remove models while the cluster is running",
|
||||
}
|
||||
c.AddCommand(newModelStopCmd(client), newModelStartCmd(client), newModelContextCmd(client), newModelGPUsCmd(client), newModelAddCmd(client), newModelPresetsCmd(client, asJSON), newModelRemoveCmd(client), newModelPlanCmd(client),
|
||||
c.AddCommand(newModelStopCmd(client), newModelStartCmd(client), newModelReplicasCmd(client), newModelContextCmd(client), newModelGPUsCmd(client), newModelAddCmd(client), newModelPresetsCmd(client, asJSON), newModelRemoveCmd(client), newModelPlanCmd(client),
|
||||
newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
|
||||
newTargetCmd(client, asJSON), newAdaptersCmd(client),
|
||||
newModelServeCmd(client), newModelUpdatesCmd(client, asJSON), newModelUpdateCmd(client),
|
||||
@@ -337,7 +338,7 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
|
||||
var memory, cores int
|
||||
var profile string
|
||||
c := &cobra.Command{
|
||||
Use: "resources <model> --memory GB [--cores N]",
|
||||
Use: "resources <model> [--memory GB] [--cores N]",
|
||||
Short: "Set what a model reserves of a node: memory, and CPU cores",
|
||||
Long: "--memory is the host RAM a replica needs: for a model with no GPU, its\n" +
|
||||
"weights, KV cache and engine together. Placement reserves it on the node,\n" +
|
||||
@@ -350,8 +351,20 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
|
||||
"--model M` moves them to the new figures.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if !cmd.Flags().Changed("memory") && !cmd.Flags().Changed("cores") {
|
||||
return fmt.Errorf("give --memory, --cores or both")
|
||||
}
|
||||
if !cmd.Flags().Changed("memory") {
|
||||
return fmt.Errorf("--memory is required")
|
||||
// Cores alone: the memory stays what it is.
|
||||
var cur catalog.Model
|
||||
if err := client().get(cmd.Context(), "/api/v1/models/"+args[0], &cur); err != nil {
|
||||
return err
|
||||
}
|
||||
kept, err := keptMemory(cur, profile)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
memory = kept
|
||||
}
|
||||
body, err := json.Marshal(adminapi.ResourcesRequest{Profile: profile, MemoryGB: memory, CPUCores: cores})
|
||||
if err != nil {
|
||||
@@ -368,12 +381,31 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().IntVar(&memory, "memory", 0, "host RAM a replica needs, in GB")
|
||||
c.Flags().IntVar(&memory, "memory", 0, "host RAM a replica needs, in GB; kept as it is when left out")
|
||||
c.Flags().IntVar(&cores, "cores", 0, "CPU cores a replica with no GPU computes on; 0 reserves none")
|
||||
c.Flags().StringVar(&profile, "profile", "", "one profile; every profile when left out")
|
||||
return c
|
||||
}
|
||||
|
||||
// keptMemory is the memory a model's profiles reserve now, for a change of
|
||||
// cores alone: one profile's, or every profile's when they agree.
|
||||
func keptMemory(m catalog.Model, profile string) (int, error) {
|
||||
memory := -1
|
||||
for _, p := range m.Profiles {
|
||||
if profile != "" && p.Name != profile {
|
||||
continue
|
||||
}
|
||||
if memory >= 0 && p.MinMemoryGB != memory {
|
||||
return 0, fmt.Errorf("%s's profiles reserve different memory: give --memory, or --profile", m.Name)
|
||||
}
|
||||
memory = p.MinMemoryGB
|
||||
}
|
||||
if memory < 0 {
|
||||
return 0, fmt.Errorf("%s has no profile %q", m.Name, profile)
|
||||
}
|
||||
return memory, nil
|
||||
}
|
||||
|
||||
func newClassCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "class <model> critical|standard|opportunistic|default",
|
||||
@@ -598,3 +630,50 @@ func newModelGPUsCmd(client func() *ctlClient) *cobra.Command {
|
||||
c.Flags().BoolVar(&inPlace, "in-place", false, "stop each replica before its replacement starts, when there is no room beside it")
|
||||
return c
|
||||
}
|
||||
|
||||
func newModelReplicasCmd(client func() *ctlClient) *cobra.Command {
|
||||
var cpuMemory, cpuCores int
|
||||
c := &cobra.Command{
|
||||
Use: "replicas <model> <profile>=<n>...",
|
||||
Short: "How many of a model's replicas run on each of its profiles: gpu=7 cpu=4",
|
||||
Long: "Each replica runs on the best profile that fits unless counted here: seven on\n" +
|
||||
"cards and four on CPUs, all answering under the model's name, the router filling\n" +
|
||||
"the cards before the CPUs. A profile not named keeps its count. For a GGUF model\n" +
|
||||
"with no profile on CPUs, --cpu-memory adds one named cpu. Its scaling becomes the\n" +
|
||||
"sum; llmgw ctl model scale with another total ends the count per profile.",
|
||||
Example: " llmgw ctl model replicas qwen-coder gpu-1xl40s-48g=7 cpu=4 --cpu-memory 24 --cpu-cores 8",
|
||||
Args: cobra.MinimumNArgs(2),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
req := adminapi.ProfileReplicasRequest{Profiles: map[string]int{}}
|
||||
for _, a := range args[1:] {
|
||||
name, v, ok := strings.Cut(a, "=")
|
||||
n, err := strconv.Atoi(v)
|
||||
if !ok || err != nil || name == "" {
|
||||
return fmt.Errorf("%q: a profile and a count, as gpu=7", a)
|
||||
}
|
||||
req.Profiles[name] = n
|
||||
}
|
||||
if cpuMemory > 0 {
|
||||
req.CPU = &modelsvc.CPUProfile{MemoryGB: cpuMemory, CPUCores: cpuCores}
|
||||
}
|
||||
body, err := json.Marshal(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var m catalog.Model
|
||||
if err := client().post(cmd.Context(), "/api/v1/models/"+args[0]+"/replicas", body, &m); err != nil {
|
||||
return err
|
||||
}
|
||||
var parts []string
|
||||
for _, p := range m.Profiles {
|
||||
parts = append(parts, fmt.Sprintf("%d on %s", p.Replicas, p.Name))
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s runs %s: %d in all. Those short start first, then those over stop.\n",
|
||||
m.Name, strings.Join(parts, ", "), m.SplitTotal())
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().IntVar(&cpuMemory, "cpu-memory", 0, "add a profile on CPUs named cpu, reserving this many GB a replica")
|
||||
c.Flags().IntVar(&cpuCores, "cpu-cores", 0, "the cores a replica on CPUs computes on (default: half the node's)")
|
||||
return c
|
||||
}
|
||||
@@ -321,7 +321,10 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
|
||||
fmt.Fprintln(out, "\nNo file fits this cluster as it is: a smaller context (--context), or more memory.")
|
||||
return errDoesNotFit
|
||||
}
|
||||
fmt.Fprintf(out, "\nBest: %s on %s: %s.\n", rec.Artifact, rec.Label(), rec.MemoryText())
|
||||
fmt.Fprintf(out, "\nBest: %s on %s: %s%s.\n", rec.Artifact, rec.Label(), rec.MemoryText(), speedNote(rec))
|
||||
if w := slowWarning(rec); w != "" {
|
||||
fmt.Fprintln(out, w)
|
||||
}
|
||||
if p.KVBytesPerToken > 0 {
|
||||
fmt.Fprintf(out, "The context costs %.2f GB per 1000 tokens, the whole model (--context to plan another).\n",
|
||||
modelplan.GB(p.KVBytesPerToken*1000))
|
||||
@@ -334,7 +337,8 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
|
||||
if o.FreeNow {
|
||||
now = "free"
|
||||
}
|
||||
fmt.Fprintf(w, " %s\t%s\t%s\t%d host(s)\t%s\n", o.Artifact, o.Label(), o.MemoryText(), o.Hosts, now)
|
||||
fmt.Fprintf(w, " %s\t%s\t%s\t%d host(s)\t%s\t%s\n", o.Artifact, o.Label(), o.MemoryText(), o.Hosts, now,
|
||||
speedCell(o.TokensPerSecond))
|
||||
}
|
||||
_ = w.Flush()
|
||||
}
|
||||
@@ -349,3 +353,37 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
|
||||
fmt.Fprintln(out, " --fallbacks")
|
||||
return nil
|
||||
}
|
||||
|
||||
// speedCell is a way's speed in a table: about how many tokens a second one
|
||||
// reply gets, and whether that is slow.
|
||||
func speedCell(tps float64) string {
|
||||
switch {
|
||||
case tps <= 0:
|
||||
return ""
|
||||
case tps < modelplan.CPUTooSlowTPS:
|
||||
return fmt.Sprintf("~%.1f tok/s, very slow", tps)
|
||||
case tps < modelplan.CPUSlowTPS:
|
||||
return fmt.Sprintf("~%.0f tok/s, slow", tps)
|
||||
}
|
||||
return fmt.Sprintf("~%.0f tok/s", tps)
|
||||
}
|
||||
|
||||
// speedNote is the same for a sentence, empty when not known.
|
||||
func speedNote(o modelplan.Option) string {
|
||||
if o.TokensPerSecond <= 0 {
|
||||
return ""
|
||||
}
|
||||
return fmt.Sprintf(", about %s tokens a second for one reply", strconv.FormatFloat(o.TokensPerSecond, 'f', 1, 64))
|
||||
}
|
||||
|
||||
// slowWarning says what a slow way means for somebody waiting on a reply.
|
||||
func slowWarning(o modelplan.Option) string {
|
||||
tps := o.TokensPerSecond
|
||||
switch {
|
||||
case tps <= 0 || tps >= modelplan.CPUSlowTPS:
|
||||
return ""
|
||||
case tps < modelplan.CPUTooSlowTPS:
|
||||
return fmt.Sprintf("Very slow: a reply of 500 tokens takes about %.0f minutes.", max(1, 500/tps/60))
|
||||
}
|
||||
return fmt.Sprintf("Slow: a reply of 500 tokens takes about %.0f seconds.", 500/tps)
|
||||
}
|
||||
+15
-4
@@ -114,7 +114,7 @@ engines are kept.`,
|
||||
}
|
||||
f := cmd.Flags()
|
||||
f.StringVar(&na.opts.Node, "node", "", "node id (default: the machine's hostname)")
|
||||
f.StringVar(&na.opts.User, "ssh-user", "", "user to log in as (default: the one before @ in the address, else whoever ran sudo, else you)")
|
||||
f.StringVar(&na.opts.User, "ssh-user", "", "user to log in as (default: the one before @ in the address, else the ssh config's for the machine, else whoever ran sudo, else you)")
|
||||
f.StringArrayVar(&na.servers, "server", nil, "control plane address the agent dials, https://host:7443; repeatable for failover (default: this machine)")
|
||||
f.StringArrayVar(&na.fabrics, "fabric", nil, "InfiniBand or RoCE fabric and its adapters, name:mlx5_0,mlx5_1; repeatable")
|
||||
f.StringVar(&na.opts.MgmtAddr, "mgmt-addr", "", "address engines are reached at (default: found by the agent)")
|
||||
@@ -228,8 +228,9 @@ func (na *nodeAdd) target(addr string) onboard.Options {
|
||||
if o.User == "" {
|
||||
// Under sudo, the person who ran it — their account is usually the
|
||||
// one that exists on the nodes too. root's own keys still do the
|
||||
// logging in, since that is who is running ssh.
|
||||
o.User = os.Getenv("SUDO_USER")
|
||||
// logging in, since that is who is running ssh. Only a fallback: a
|
||||
// User in the ssh config's block for the machine comes first.
|
||||
o.FallbackUser = os.Getenv("SUDO_USER")
|
||||
}
|
||||
return o
|
||||
}
|
||||
@@ -495,7 +496,17 @@ func engineEvent(engine, version, state, detail string) onboard.Event {
|
||||
level = onboard.Warn
|
||||
}
|
||||
return onboard.Event{Step: onboard.StepEngines, Level: level,
|
||||
Detail: fmt.Sprintf("%s %s: %s%s", engine, version, state, prefixed(": ", detail))}
|
||||
Detail: provisionLine(engine, version, state, detail)}
|
||||
}
|
||||
|
||||
// provisionLine is one line of what a node is installing. A package
|
||||
// install that is no engine's, Slurm's or MariaDB's, has no name or version.
|
||||
func provisionLine(engine, version, state, detail string) string {
|
||||
what := strings.TrimSpace(engine + " " + version)
|
||||
if what == "" {
|
||||
what = "system packages"
|
||||
}
|
||||
return fmt.Sprintf("%s: %s%s", what, state, prefixed(": ", detail))
|
||||
}
|
||||
|
||||
func prefixed(p, s string) string {
|
||||
|
||||
@@ -49,6 +49,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
@@ -59,6 +60,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
@@ -254,6 +256,7 @@ func newServerCmd() *cobra.Command {
|
||||
// The settings in force, for what reads them at each use.
|
||||
var current atomic.Pointer[config.Server]
|
||||
current.Store(&cfg)
|
||||
rt.SetSlurm(func() config.SlurmSettings { return current.Load().Slurm })
|
||||
var agentSvc *agentsvc.Service
|
||||
var agentHandler http.Handler
|
||||
var agentPath string
|
||||
@@ -261,6 +264,8 @@ func newServerCmd() *cobra.Command {
|
||||
// What the settings say of the firewall, for the mirror and the
|
||||
// store's members; nil without a control plane.
|
||||
var fwPub *firewallPublisher
|
||||
// The Slurm cluster beside llmgw, when there is a store (BACKLOG P).
|
||||
var slurmSvc *slurmsvc.Service
|
||||
updateMetrics := &modelserve.UpdateCollector{}
|
||||
// The store's cluster, when it is one; read by the metrics and
|
||||
// by `ctl status`.
|
||||
@@ -412,6 +417,7 @@ func newServerCmd() *cobra.Command {
|
||||
adminAPI = api
|
||||
api.SetReleases(cfg.Releases)
|
||||
api.SetNotifier(notifier)
|
||||
api.SetWebhooks(func() int { return len(current.Load().Notify.Webhooks) })
|
||||
api.SetReplayer(rt.Replay)
|
||||
// Quality suites, asked of a model the way a replay is.
|
||||
evals = newEvalService(cfg, rt, admin, log)
|
||||
@@ -436,6 +442,31 @@ func newServerCmd() *cobra.Command {
|
||||
agentSvc.SetTrain(trainer)
|
||||
go trainer.Run(ctx)
|
||||
api.SetTrain(&adminapi.TrainAdmin{Store: st, Service: trainer})
|
||||
// Slurm on the nodes the settings name, its munge key
|
||||
// made from a cluster secret.
|
||||
slurmSvc = slurmsvc.New(func() config.SlurmSettings { return current.Load().Slurm },
|
||||
admin.Inventory(), agentSvc, func(ctx context.Context) ([]byte, error) {
|
||||
secret, err := st.ClusterSecret(ctx, "slurm-munge")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return slurm.MungeKey(secret)
|
||||
}, log)
|
||||
// What slurm.conf held, for the leader that comes next.
|
||||
slurmSvc.SetMemory(slurmsvc.StoreMemory(st))
|
||||
// slurm.users "llmgw": a Linux user for each active account.
|
||||
slurmSvc.SetUsers(func(ctx context.Context) ([]slurmsvc.User, error) {
|
||||
us, err := st.SlurmUsers(ctx)
|
||||
out := make([]slurmsvc.User, 0, len(us))
|
||||
for _, u := range us {
|
||||
out = append(out, adminapi.SlurmUserView(u))
|
||||
}
|
||||
return out, err
|
||||
})
|
||||
api.SetSlurmUsers(st)
|
||||
agentSvc.SetSlurm(slurmSvc)
|
||||
go slurmSvc.Run(ctx)
|
||||
api.SetSlurm(slurmSvc)
|
||||
}
|
||||
// The cluster's settings, each change checked against this
|
||||
// machine's own keys before it is kept.
|
||||
@@ -841,6 +872,9 @@ func newServerCmd() *cobra.Command {
|
||||
fwPub.Kick()
|
||||
}
|
||||
}
|
||||
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "slurm") }) && slurmSvc != nil {
|
||||
slurmSvc.Kick()
|
||||
}
|
||||
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "on_demand") }) && admin != nil {
|
||||
admin.Reconciler().SetEvictionTimes(next.OnDemand.EvictAfterIdle, next.OnDemand.MinLifetime)
|
||||
}
|
||||
|
||||
@@ -443,7 +443,7 @@ func (sa *serverAdd) storeAccess(ctx context.Context) error {
|
||||
return err
|
||||
}
|
||||
}
|
||||
sa.done("nodes will be told of both control planes within half a minute")
|
||||
sa.done("nodes will be told of every control plane within half a minute")
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -124,6 +124,13 @@ type serverInit struct {
|
||||
// media has the nodes install ComfyUI and the leader add a picture model
|
||||
// and a 3D model once a card fits one.
|
||||
media bool
|
||||
// slurm runs Slurm beside llmgw: slurmctld on this machine's node,
|
||||
// slurmd on every node, each llmgw's until it is lent.
|
||||
slurm bool
|
||||
// slurmShared is where Slurm's shared files come from: "here" (this
|
||||
// machine exports /shared), the site's export as HOST:/EXPORT, or
|
||||
// "none". slurmUsers is "llmgw" (users from the accounts) or "site".
|
||||
slurmShared, slurmUsers string
|
||||
// evictAfterIdle and minLifetime are on_demand's; zero leaves its own.
|
||||
evictAfterIdle, minLifetime time.Duration
|
||||
// standbys makes each machine of the replicated store a control plane
|
||||
@@ -258,6 +265,12 @@ func newServerInitCmd() *cobra.Command {
|
||||
f.DurationVar(&in.maxKeyTTL, "max-key-ttl", defaultMaxKeyTTL, "the longest an API key lives")
|
||||
f.BoolVar(&in.media, "media", false,
|
||||
"pictures and 3D: the nodes install ComfyUI, and a picture model and a 3D model are added once a card fits one")
|
||||
f.BoolVar(&in.slurm, "slurm", false,
|
||||
"Slurm for training and batch jobs: slurmctld and slurmdbd (with a MariaDB) on this machine, slurmd on every node, each llmgw's until 'llmgw ctl slurm lend'")
|
||||
f.StringVar(&in.slurmShared, "slurm-shared", "here",
|
||||
"with --slurm, the jobs' shared files at /shared: here (this machine exports them over NFS), the site's HOST:/EXPORT, or none")
|
||||
f.StringVar(&in.slurmUsers, "slurm-users", config.UsersLlmgw,
|
||||
"with --slurm, who makes the Linux users that run jobs: llmgw (one for each account, SSH keys in the portal) or site")
|
||||
f.DurationVar(&in.evictAfterIdle, "evict-after-idle", 0,
|
||||
"how long a model loaded on demand goes unasked-for before another may have its card (0: 2m)")
|
||||
f.DurationVar(&in.minLifetime, "min-lifetime", 0,
|
||||
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
|
||||
)
|
||||
@@ -132,9 +133,25 @@ func (in *serverInit) checkChoices() error {
|
||||
return fmt.Errorf("max key ttl %s: an hour at least", in.maxKeyTTL)
|
||||
case in.evictAfterIdle < 0 || in.minLifetime < 0:
|
||||
return fmt.Errorf("on-demand times cannot be negative")
|
||||
case in.slurm && in.noNode:
|
||||
return fmt.Errorf("slurm runs slurmctld on this machine's node: it needs the node (no --no-node)")
|
||||
case in.vip != "" && !in.standbys:
|
||||
return fmt.Errorf("a floating address moves between control planes: it needs the standbys")
|
||||
}
|
||||
if in.slurm {
|
||||
// The controller is this machine's node, named by its hostname: a
|
||||
// name Slurm does not take would keep the server from starting.
|
||||
if err := (config.SlurmSettings{Enabled: true, Controller: localNodeID()}).Check(); err != nil {
|
||||
return fmt.Errorf("slurm: this machine's hostname: %w", err)
|
||||
}
|
||||
set := config.SlurmSettings{Enabled: true, Controller: localNodeID(), Shared: slurmShared(in.slurmShared)}
|
||||
if err := set.Check(); err != nil {
|
||||
return fmt.Errorf("--slurm-shared %q: here, none, or the site's HOST:/EXPORT: %w", in.slurmShared, err)
|
||||
}
|
||||
if u := in.slurmUsers; u != "" && u != config.UsersLlmgw && u != "site" {
|
||||
return fmt.Errorf("--slurm-users %q: llmgw or site", u)
|
||||
}
|
||||
}
|
||||
if in.vip != "" {
|
||||
ip, _, err := net.ParseCIDR(in.vip)
|
||||
if err != nil || ip.To4() == nil {
|
||||
@@ -318,6 +335,9 @@ func (in *serverInit) tuneConfig(body string) string {
|
||||
"# one ('llmgw ctl model presets auto off' stops it).\n"+
|
||||
"engines:\n comfyui: "+provision.ComfyUIVersion()+"\npresets:\n auto: true\n")
|
||||
}
|
||||
if in.slurm {
|
||||
extra = append(extra, slurmSection(localNodeID(), slurmShared(in.slurmShared), in.slurmUsers != "site"))
|
||||
}
|
||||
if in.evictAfterIdle > 0 || in.minLifetime > 0 {
|
||||
b := "# How models that load when asked for take turns on a card, chosen at\n# install.\non_demand:\n"
|
||||
if in.evictAfterIdle > 0 {
|
||||
@@ -334,6 +354,53 @@ func (in *serverInit) tuneConfig(body string) string {
|
||||
return body
|
||||
}
|
||||
|
||||
// slurmShared is --slurm-shared as settings: "here" is /shared exported
|
||||
// by this machine's node, "none" nothing, anything else the site's export.
|
||||
func slurmShared(v string) config.SharedStorage {
|
||||
switch v {
|
||||
case "", "none":
|
||||
return config.SharedStorage{}
|
||||
case "here":
|
||||
return config.SharedStorage{Path: "/shared", Server: localNodeID()}
|
||||
}
|
||||
return config.SharedStorage{Path: "/shared", NFS: v}
|
||||
}
|
||||
|
||||
// slurmSection is server.yaml's slurm block as an install chooses it:
|
||||
// slurmctld on this machine's node, slurmd on every node, the shared files
|
||||
// and the users as chosen.
|
||||
func slurmSection(controller string, shared config.SharedStorage, users bool) string {
|
||||
b := "# Slurm beside llmgw, chosen at install: slurmctld, and slurmdbd with a MariaDB\n" +
|
||||
"# for the jobs' records, on this machine; slurmd on\n" +
|
||||
"# every node, those that join later too. Each node is llmgw's, drained in\n" +
|
||||
"# Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm; 'llmgw ctl slurm\n" +
|
||||
"# reclaim NODE' takes it back once its jobs end.\n" +
|
||||
"slurm:\n enabled: true\n controller: " + yamlString(controller) + "\n all_nodes: true\n"
|
||||
switch {
|
||||
case shared.Server != "":
|
||||
b += " # /shared on every Slurm machine, exported over NFS from this one\n" +
|
||||
" # ('llmgw ctl slurm shared' points it at the site's storage instead).\n" +
|
||||
" shared:\n path: " + yamlString(shared.Path) + "\n server: " + yamlString(shared.Server) + "\n"
|
||||
case shared.NFS != "":
|
||||
b += " # /shared on every Slurm machine, mounted from the site's export.\n" +
|
||||
" shared:\n path: " + yamlString(shared.Path) + "\n nfs: " + yamlString(shared.NFS) + "\n"
|
||||
}
|
||||
if users {
|
||||
b += " # A Linux user for each llmgw account, the same UID everywhere; SSH keys\n" +
|
||||
" # in the portal's Profile. Empty leaves users to the site (sssd, a directory).\n" +
|
||||
" users: llmgw\n"
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// localNodeID is the node id this machine's agent takes: its hostname.
|
||||
func localNodeID() string {
|
||||
if h, err := os.Hostname(); err == nil && h != "" {
|
||||
return h
|
||||
}
|
||||
return "localhost"
|
||||
}
|
||||
|
||||
// durationYAML is a duration as a person writes it: 2h, 90m, 1m30s.
|
||||
func durationYAML(d time.Duration) string {
|
||||
switch {
|
||||
@@ -467,6 +534,15 @@ func (in *serverInit) initFlags() []string {
|
||||
if in.media {
|
||||
add("--media")
|
||||
}
|
||||
if in.slurm {
|
||||
add("--slurm")
|
||||
if in.slurmShared != "" && in.slurmShared != "here" {
|
||||
add("--slurm-shared", in.slurmShared)
|
||||
}
|
||||
if in.slurmUsers == "site" {
|
||||
add("--slurm-users", "site")
|
||||
}
|
||||
}
|
||||
if in.evictAfterIdle > 0 {
|
||||
add("--evict-after-idle", durationYAML(in.evictAfterIdle))
|
||||
}
|
||||
|
||||
@@ -17,7 +17,7 @@ import (
|
||||
func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
|
||||
in := &serverInit{
|
||||
language: "tr", signupOpen: true, defaultRPM: 120, defaultTPM: 50000, maxKeyTTL: 30 * 24 * time.Hour,
|
||||
media: true, evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
|
||||
media: true, slurm: true, slurmShared: "here", slurmUsers: "llmgw", evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
|
||||
judge: judgeSpec{Fail: "open", Categories: []string{"Violent", "PII"}, Replicas: 3, Budget: 3 * time.Second},
|
||||
}
|
||||
j := in.judgeFor()
|
||||
@@ -41,6 +41,10 @@ func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
|
||||
t.Errorf("limits %d %d %s", cfg.Auth.DefaultRPM, cfg.Auth.DefaultTPM, cfg.Auth.MaxKeyTTL)
|
||||
case !cfg.Presets.Auto || cfg.Engines["comfyui"] == "":
|
||||
t.Errorf("pictures and 3D: presets %v comfyui %q", cfg.Presets.Auto, cfg.Engines["comfyui"])
|
||||
case !cfg.Slurm.Enabled || !cfg.Slurm.AllNodes || cfg.Slurm.Controller != localNodeID():
|
||||
t.Errorf("slurm %+v", cfg.Slurm)
|
||||
case cfg.Slurm.Shared.Path != "/shared" || cfg.Slurm.Shared.Server != localNodeID() || cfg.Slurm.Users != "llmgw":
|
||||
t.Errorf("slurm's shared files and users %+v", cfg.Slurm)
|
||||
case cfg.OnDemand.EvictAfterIdle != time.Minute || cfg.OnDemand.MinLifetime != 90*time.Second:
|
||||
t.Errorf("on demand %+v", cfg.OnDemand)
|
||||
}
|
||||
@@ -95,14 +99,14 @@ func TestAStarterIsChosenByNameOrByItsFile(t *testing.T) {
|
||||
|
||||
func TestTheChoicesAreTheirFlagsAgain(t *testing.T) {
|
||||
in := &serverInit{admin: "ops@example.com", hosts: []string{"llm.example.com"}, postgres: true,
|
||||
mirrorPath: "/data/models", noNode: true, language: "tr", media: true,
|
||||
mirrorPath: "/data/models", language: "tr", media: true, slurm: true,
|
||||
webhooks: []string{"https://hooks.slack.com/services/T/B/x, json:https://a.example.com/in"},
|
||||
judge: judgeSpec{Categories: []string{"Violent", "Jailbreak"}}, maxKeyTTL: 7 * 24 * time.Hour}
|
||||
in.starter, _ = starterFrom("qwen3-4b")
|
||||
got := strings.Join(in.initFlags(), " ")
|
||||
want := "--admin ops@example.com --host llm.example.com --language tr --max-key-ttl 168h --postgres " +
|
||||
"--mirror-dir /data/models --no-node --starter qwen3-4b --judge-categories Violent,Jailbreak " +
|
||||
"--webhook https://hooks.slack.com/services/T/B/x --webhook json:https://a.example.com/in --media"
|
||||
"--mirror-dir /data/models --starter qwen3-4b --judge-categories Violent,Jailbreak " +
|
||||
"--webhook https://hooks.slack.com/services/T/B/x --webhook json:https://a.example.com/in --media --slurm"
|
||||
if got != want {
|
||||
t.Errorf("flags\n got %s\nwant %s", got, want)
|
||||
}
|
||||
@@ -124,6 +128,7 @@ func TestChoicesThatCannotBeInstalledAreRefused(t *testing.T) {
|
||||
{maxKeyTTL: time.Minute},
|
||||
{vip: "10.0.0.160/24"},
|
||||
{standbys: true, vip: "10.0.0.160"},
|
||||
{slurm: true, noNode: true},
|
||||
} {
|
||||
if err := in.checkChoices(); err == nil {
|
||||
t.Errorf("taken: %+v", in)
|
||||
|
||||
@@ -398,7 +398,9 @@ labels:
|
||||
// checks its own llama.cpp, the CUDA build when it has a GPU, and a new
|
||||
// machine is often busy with its own package updates first.
|
||||
client := in.adminAPI()
|
||||
last := ""
|
||||
// Each engine's line once, and again only when it changes: the engines'
|
||||
// states come back together on every look.
|
||||
seen := map[string]string{}
|
||||
err = waitFor(ctx, 20*time.Minute, "this node to be ready with an engine", func() (bool, error) {
|
||||
var nodes []adminapi.NodeView
|
||||
if err := client.get(ctx, "/api/v1/nodes", &nodes); err != nil {
|
||||
@@ -412,9 +414,9 @@ labels:
|
||||
return true, nil
|
||||
}
|
||||
for _, p := range n.Provision {
|
||||
if line := fmt.Sprintf("%s %s: %s %s", p.Engine, p.Version, p.State, p.Detail); line != last {
|
||||
fmt.Fprintf(in.out, " .. %s\n", strings.TrimSpace(line))
|
||||
last = line
|
||||
if line := provisionLine(p.Engine, p.Version, p.State, p.Detail); seen[p.Engine] != line {
|
||||
fmt.Fprintf(in.out, " .. %s\n", line)
|
||||
seen[p.Engine] = line
|
||||
}
|
||||
// Only llama.cpp is waited for: vLLM installs for minutes after,
|
||||
// and one this node cannot run is not a reason to stop here.
|
||||
|
||||
@@ -156,6 +156,10 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
|
||||
if s.Stopped {
|
||||
fmt.Fprintln(out, "SERVICE STOPPED: no model runs and requests are answered 503 (llmgw start)")
|
||||
}
|
||||
if s.NoAlerts {
|
||||
fmt.Fprintln(out, "ALERTS nowhere: a lost node or a judge down is told nobody "+
|
||||
"(llmgw ctl config set notify.webhooks, then llmgw ctl notify test)")
|
||||
}
|
||||
if s.GuardOff {
|
||||
fmt.Fprintln(out, "GUARD off: no request is checked, whatever the policies say (llmgw ctl guard on)")
|
||||
}
|
||||
|
||||
@@ -20,6 +20,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
)
|
||||
@@ -406,6 +407,48 @@ func (m consoleModel) modelMenu(md adminapi.ModelView) *menuDialog {
|
||||
}})
|
||||
return nil
|
||||
}},
|
||||
{"Replicas per profile…", func(m *consoleModel) tea.Cmd {
|
||||
var now []string
|
||||
for _, p := range md.Profiles {
|
||||
now = append(now, fmt.Sprintf("%s=%d", p, md.ProfileReplicas[p]))
|
||||
}
|
||||
m.openForm(&formDialog{title: "Replicas per profile of " + md.Name,
|
||||
note: "How many replicas run on each profile, all answering under the model's name: seven on cards and " +
|
||||
"four on CPUs, the router filling the cards first. A profile not named keeps its count; cpu=N with " +
|
||||
"the memory below adds a profile on CPUs to a GGUF model that has none. Those short start first, " +
|
||||
"then those over stop.",
|
||||
fields: []formField{
|
||||
textF("replicas", strings.Join(now, " "), "profile=count, separated by spaces"),
|
||||
textF("CPU memory GB", "", "only to add a profile on CPUs: what a replica reserves"),
|
||||
textF("CPU cores", "", "of a replica on CPUs; empty is half the node's"),
|
||||
},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
req := adminapi.ProfileReplicasRequest{Profiles: map[string]int{}}
|
||||
for _, f := range strings.Fields(v[0]) {
|
||||
name, c, ok := strings.Cut(f, "=")
|
||||
n, err := strconv.Atoi(c)
|
||||
if !ok || err != nil || n < 0 || name == "" {
|
||||
return nil, fmt.Errorf("%q: a profile and a count, as gpu=7", f)
|
||||
}
|
||||
req.Profiles[name] = n
|
||||
}
|
||||
if v[1] != "" {
|
||||
mem, err := strconv.Atoi(v[1])
|
||||
if err != nil || mem < 1 {
|
||||
return nil, fmt.Errorf("the CPU memory is a whole number of GB")
|
||||
}
|
||||
cores := 0
|
||||
if v[2] != "" {
|
||||
if cores, err = strconv.Atoi(v[2]); err != nil || cores < 1 {
|
||||
return nil, fmt.Errorf("the CPU cores are a whole number from 1")
|
||||
}
|
||||
}
|
||||
req.CPU = &modelsvc.CPUProfile{MemoryGB: mem, CPUCores: cores}
|
||||
}
|
||||
return m.call("Replicas of "+md.Name, http.MethodPost, path+"/replicas", req, nil), nil
|
||||
}})
|
||||
return nil
|
||||
}},
|
||||
{"Resources…", func(m *consoleModel) tea.Cmd {
|
||||
profiles := md.Profiles
|
||||
if len(profiles) == 0 {
|
||||
@@ -600,17 +643,30 @@ func (m consoleModel) nodeMenu(n adminapi.NodeView) *menuDialog {
|
||||
menuItem{"Delete model files nothing uses", func(m *consoleModel) tea.Cmd {
|
||||
return m.call("Unused files on "+n.ID, http.MethodPost, "/api/v1/weights/collect", map[string]string{"node": n.ID},
|
||||
func(raw json.RawMessage) string {
|
||||
var rs []struct {
|
||||
Files int `json:"files"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
}
|
||||
// What the node no longer lists, and what it kept.
|
||||
var rs []adminapi.CollectResult
|
||||
_ = json.Unmarshal(raw, &rs)
|
||||
var files int
|
||||
var b int64
|
||||
var files, left int
|
||||
var b, lb int64
|
||||
var errs []string
|
||||
for _, r := range rs {
|
||||
files, b = files+r.Files, b+r.Bytes
|
||||
files, b, left, lb = files+r.Files, b+r.Bytes, left+r.Left, lb+r.LeftBytes
|
||||
if r.Error != "" {
|
||||
errs = append(errs, r.Error)
|
||||
}
|
||||
}
|
||||
return fmt.Sprintf("%d files, %.1f GB freed.", files, float64(b)/(1<<30))
|
||||
if len(rs) == 0 {
|
||||
return "Nothing on " + n.ID + " that no model uses."
|
||||
}
|
||||
say := fmt.Sprintf("%d files removed, %.1f GB freed.", files, float64(b)/(1<<30))
|
||||
if left > 0 {
|
||||
say += fmt.Sprintf(" %d files (%.1f GB) are still there: the node kept them, and its agent's log "+
|
||||
"says why (journalctl -u llmgw-agent there).", left, float64(lb)/(1<<30))
|
||||
}
|
||||
if len(errs) > 0 {
|
||||
say += " " + strings.Join(errs, "; ")
|
||||
}
|
||||
return say
|
||||
})
|
||||
}},
|
||||
menuItem{"Remove from the cluster…", func(m *consoleModel) tea.Cmd {
|
||||
|
||||
@@ -57,7 +57,7 @@ type installForm struct {
|
||||
standby *installPart
|
||||
vipF *setupField
|
||||
|
||||
control, store, mirror, node, starter, media, judge, firewall, alerts, limits, turns, bundle *installPart
|
||||
control, store, mirror, node, starter, media, slurm, judge, firewall, alerts, limits, turns, bundle *installPart
|
||||
|
||||
admin, hosts, language, signup *setupField
|
||||
storeKind *setupField
|
||||
@@ -282,6 +282,31 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
|
||||
summary: func() string { return "ComfyUI; a picture model and a 3D model once a card fits" },
|
||||
}
|
||||
|
||||
// --- Slurm ---
|
||||
f.slurm = &installPart{
|
||||
title: "Slurm", on: in.slurm,
|
||||
about: "A Slurm cluster beside llmgw, for training and other batch work: slurmctld on this machine, " +
|
||||
"slurmd on every node, those that join later too. The agents install munge and Slurm from the " +
|
||||
"machines' own package sources, and on this machine slurmdbd, the jobs' records, with a MariaDB " +
|
||||
"that answers this machine alone (Slurm keeps its records in MySQL or MariaDB only, not in the " +
|
||||
"cluster's PostgreSQL). Its port 6819, with Slurm's others, answers the cluster's networks only.\n\n" +
|
||||
"Each node stays llmgw's, drained in Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm: " +
|
||||
"its models move off first. 'llmgw ctl slurm reclaim NODE' takes it back once its jobs end. Jobs " +
|
||||
"are submitted on this machine with sbatch and srun.\n\n" +
|
||||
"Every Slurm machine sees /shared, exported over NFS from this machine (homes, datasets, the jobs' " +
|
||||
"output); 'llmgw ctl slurm shared' points it at the site's own storage instead. Each llmgw account " +
|
||||
"is a Linux user there, the same UID on every machine, logging in with the SSH keys given in the " +
|
||||
"portal; 'llmgw ctl slurm users off' leaves users to the site's directory.",
|
||||
off: "no Slurm; 'llmgw ctl slurm setup' adds it later",
|
||||
summary: func() string { return "slurmctld here, slurmd on every node, /shared from here; none lent" },
|
||||
needs: func() string {
|
||||
if !f.node.on {
|
||||
return "slurmctld runs on this machine's node: turn the node on, or Slurm off"
|
||||
}
|
||||
return ""
|
||||
},
|
||||
}
|
||||
|
||||
// --- the guardrails' judge ---
|
||||
j := in.judgeFor()
|
||||
f.judgeFail = choiceField("When no judge answers in time", []string{"refuse", "serve flagged"},
|
||||
@@ -421,7 +446,7 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
|
||||
if ha {
|
||||
f.parts = append(f.parts, f.standby)
|
||||
}
|
||||
f.parts = append(f.parts, f.mirror, f.node, f.starter, f.media, f.judge, f.firewall,
|
||||
f.parts = append(f.parts, f.mirror, f.node, f.starter, f.media, f.slurm, f.judge, f.firewall,
|
||||
f.alerts, f.limits, f.turns, f.bundle)
|
||||
return f
|
||||
}
|
||||
@@ -653,6 +678,7 @@ func (f *installForm) apply(in *serverInit) error {
|
||||
in.noModel = !f.starter.on
|
||||
in.starter, in.starterCtx = starter, ctx
|
||||
in.media = f.media.on
|
||||
in.slurm = f.slurm.on
|
||||
in.noGuardModel = !f.judge.on
|
||||
in.judge = judge
|
||||
in.noFirewall = !f.firewall.on
|
||||
|
||||
@@ -24,6 +24,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/docgen"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
@@ -50,6 +51,7 @@ const (
|
||||
cFederation
|
||||
cMirror
|
||||
cJobs
|
||||
cSlurm
|
||||
cLogs
|
||||
cAccounts
|
||||
cKeys
|
||||
@@ -71,7 +73,7 @@ const (
|
||||
cCount
|
||||
)
|
||||
|
||||
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "MCP servers", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Sign-in", "Audit", "Manual",
|
||||
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "MCP servers", "Federation", "Mirror", "Jobs", "Slurm", "Logs", "Accounts", "API keys", "Organizations", "Sign-in", "Audit", "Manual",
|
||||
"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Serve the guardrails' judge", "Upgrade the cluster",
|
||||
"Restart all replicas", "Back up now", "Wipe this machine", "Leave"}
|
||||
|
||||
@@ -92,6 +94,8 @@ type consoleData struct {
|
||||
// federation is the other llmgw clusters whose models are offered
|
||||
// here; empty on a control plane without a store.
|
||||
federation adminapi.FederationView
|
||||
// slurm is the Slurm cluster beside llmgw; off when it has none.
|
||||
slurm slurmsvc.Status
|
||||
// presets is the models that come with llmgw as they stand here; nil
|
||||
// from a control plane that cannot say.
|
||||
presets *adminapi.PresetsView
|
||||
@@ -284,6 +288,7 @@ func (m consoleModel) fetch() tea.Cmd {
|
||||
_ = c.get(ctx, "/api/v1/jobs", &d.jobs)
|
||||
_ = c.get(ctx, "/api/v1/audit?since=7d&limit=300", &d.audit)
|
||||
_ = c.get(ctx, "/api/v1/federation", &d.federation)
|
||||
_ = c.get(ctx, "/api/v1/slurm", &d.slurm)
|
||||
var presets adminapi.PresetsView
|
||||
if c.get(ctx, "/api/v1/presets", &presets) == nil {
|
||||
d.presets = &presets
|
||||
@@ -588,6 +593,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
case cFederation:
|
||||
m.dialog = "error"
|
||||
m.dialogTitle, m.dialogText = peerDialog(m.data.federation.Peers[m.row])
|
||||
case cSlurm:
|
||||
m.menuD = m.slurmMenu(m.data.slurm.Nodes[m.row])
|
||||
case cServe:
|
||||
return m, m.fetchRepo(m.hf.models[m.row].ID)
|
||||
case cAccounts:
|
||||
@@ -1110,6 +1117,8 @@ func (m consoleModel) rowCount() int {
|
||||
return len(m.data.audit)
|
||||
case cFederation:
|
||||
return len(m.data.federation.Peers)
|
||||
case cSlurm:
|
||||
return len(m.data.slurm.Nodes)
|
||||
case cServe:
|
||||
return len(m.hf.models)
|
||||
case cAccounts:
|
||||
@@ -1369,6 +1378,8 @@ func (m consoleModel) panel() (title, body, keys string) {
|
||||
return m.auditPanel(pw)
|
||||
case cFederation:
|
||||
return m.federationPanel(pw)
|
||||
case cSlurm:
|
||||
return m.slurmPanel(pw)
|
||||
case cMonitor:
|
||||
return m.monitorPanel(pw)
|
||||
case cServe:
|
||||
|
||||
+24
-1
@@ -107,7 +107,15 @@ func optionLabel(o modelplan.Option) string {
|
||||
if o.FreeNow {
|
||||
now = sOK.Render("free")
|
||||
}
|
||||
return fmt.Sprintf("%s · %s · %s", o.Label(), o.MemoryText(), now)
|
||||
label := fmt.Sprintf("%s · %s · %s", o.Label(), o.MemoryText(), now)
|
||||
if c := speedCell(o.TokensPerSecond); c != "" {
|
||||
style := sDim
|
||||
if o.TokensPerSecond < modelplan.CPUTooSlowTPS {
|
||||
style = sWarn
|
||||
}
|
||||
label += " · " + style.Render(c)
|
||||
}
|
||||
return label
|
||||
}
|
||||
|
||||
// repoMenu offers a repository's files, each with the best way it runs here;
|
||||
@@ -196,16 +204,26 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
|
||||
note += fmt.Sprintf("\nMemory for a %d-token context, each way it runs:", p.Context)
|
||||
for _, o := range opts {
|
||||
note += "\n " + o.Label() + ": " + o.MemoryText()
|
||||
if w := slowWarning(o); w != "" && o.CPU {
|
||||
note += "\n " + sWarn.Render(w)
|
||||
}
|
||||
}
|
||||
if p.KVBytesPerToken > 0 {
|
||||
note += fmt.Sprintf("\nThe context costs %.2f GB per 1000 tokens, the whole model; a longer one is checked\n"+
|
||||
"before it is served.", modelplan.GB(p.KVBytesPerToken*1000))
|
||||
}
|
||||
note += "\nThe mirror fetches it and checks it, it joins the catalog, and a replica is started\nand asked to say hello."
|
||||
// The longest context the first way holds, up to what the model was
|
||||
// trained for: a choice of another way is checked again on submitting.
|
||||
ctxDefault := "4096"
|
||||
if a.Format == "safetensors" {
|
||||
ctxDefault = strconv.Itoa(p.Context)
|
||||
}
|
||||
if len(opts) > 0 {
|
||||
if most := opts[0].MostContext(p.KVBytesPerToken, p.Trained); most > 0 {
|
||||
ctxDefault = strconv.Itoa(most)
|
||||
}
|
||||
}
|
||||
if len(opts) > 0 && opts[0].Engine == "strata" {
|
||||
ctxDefault = "32768"
|
||||
}
|
||||
@@ -238,6 +256,11 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
|
||||
Description: v[3], Format: a.Format}
|
||||
if a.Format == "gguf" {
|
||||
req.File = a.Name
|
||||
// A context longer than the default pool is the pool: one
|
||||
// slot of it, where the server would refuse the request.
|
||||
if ctxLen > modelserve.DefaultCtxPool {
|
||||
req.CtxPool = ctxLen
|
||||
}
|
||||
}
|
||||
o := opts[slices.Index(where, v[4])]
|
||||
// Asked for a context other than planned: does it still fit?
|
||||
|
||||
@@ -221,11 +221,11 @@ func (m consoleModel) userMenu(u store.User) *menuDialog {
|
||||
items = append(items,
|
||||
menuItem{"Add to an organization, team or project…", func(m *consoleModel) tea.Cmd {
|
||||
m.openForm(&formDialog{title: "Membership for " + who,
|
||||
note: "A member of a team or organization may use every project in it.",
|
||||
note: "A member of a team or organization may use every project in it. On an organization, a manager also sees its page and an admin also changes it.",
|
||||
fields: []formField{
|
||||
choiceF("level", []string{"org", "team", "project"}, "team"),
|
||||
textF("which", "", "Acme, Acme/ml or Acme/ml/training"),
|
||||
choiceF("role", []string{store.RoleMember, "manager"}, store.RoleMember),
|
||||
choiceF("role", store.MembershipRoles, store.RoleMember),
|
||||
},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
if v[1] == "" {
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
// The console's Slurm view: the Slurm cluster run beside llmgw (BACKLOG P),
|
||||
// each of its nodes and whose it is now, as `llmgw ctl slurm status` shows
|
||||
// them. A node is lent to Slurm, or taken back, from its row. Turning Slurm
|
||||
// on, and choosing its nodes, is `ctl slurm setup` or server.yaml's slurm
|
||||
// part: a form for it would be server.yaml written again.
|
||||
|
||||
func (m consoleModel) slurmPanel(pw int) (string, string, string) {
|
||||
keys := "→ or Enter open · ↑↓ choose · Q leave"
|
||||
if m.inPanel {
|
||||
keys = "↑↓ node · Enter lend or take back · ← back · Q leave"
|
||||
}
|
||||
st := m.data.slurm
|
||||
head := "\n " + sDim.Render("Slurm beside llmgw, on its nodes: training and batch jobs.")
|
||||
if !st.Enabled {
|
||||
return "Slurm", head + "\n\n " + sDim.Render("Slurm is off. Turn it on with: "+
|
||||
"llmgw ctl slurm setup --controller NODE --nodes NODE,... (or --all-nodes)"), keys
|
||||
}
|
||||
ctl := sDim.Render("not heard from yet")
|
||||
switch c := st.Ctld; {
|
||||
case c == nil:
|
||||
case c.Error != "":
|
||||
ctl = sBad.Render(c.Error)
|
||||
case c.SlurmctldRunning && c.SlurmdbdRunning:
|
||||
ctl = sOK.Render("slurmctld " + c.Version + " and slurmdbd running")
|
||||
case c.SlurmctldRunning:
|
||||
ctl = sOK.Render("slurmctld "+c.Version+" running") + ", " + sWarn.Render("slurmdbd not")
|
||||
default:
|
||||
ctl = sBad.Render("slurmctld not running")
|
||||
}
|
||||
head += fmt.Sprintf("\n controller %s: %s", st.Controller, ctl)
|
||||
head += "\n " + sDim.Render(fmt.Sprintf("cluster %s, partition %s; sbatch and srun on %s", st.Cluster, st.Partition, st.Controller))
|
||||
if st.Shared.Path == "" {
|
||||
head += "\n shared files: " + sWarn.Render("none") + sDim.Render(" (llmgw ctl slurm shared; output stays on the job's node)")
|
||||
} else {
|
||||
head += "\n shared files: " + sharedLine(st.Shared)
|
||||
}
|
||||
if st.Users != "" {
|
||||
head += "\n " + fmt.Sprintf("users: %d from llmgw's accounts", st.UserCount) + sDim.Render(", SSH keys in the portal's Profile")
|
||||
} else {
|
||||
head += "\n " + sDim.Render("users: the site's (llmgw ctl slurm users on makes them from llmgw's accounts)")
|
||||
}
|
||||
if line := containersLine(st); line != "" {
|
||||
head += "\n containers: " + line
|
||||
}
|
||||
head += "\n " + sDim.Render("A node is llmgw's until lent; lent, its replicas move off first.")
|
||||
if len(st.Waiting) > 0 {
|
||||
head += "\n " + sWarn.Render(fmt.Sprintf("not in slurm.conf until they report their hardware: %v", st.Waiting))
|
||||
}
|
||||
head += "\n\n"
|
||||
if len(st.Nodes) == 0 {
|
||||
return "Slurm", head + " " + sDim.Render("No node runs slurmd."), keys
|
||||
}
|
||||
var rows [][]string
|
||||
for _, n := range st.Nodes {
|
||||
rows = append(rows, slurmRow(n))
|
||||
}
|
||||
return "Slurm", head + table([]string{"NODE", "WHOSE", "IN SLURM", "JOBS", "SLURMD", "SHARED", "NOTE"}, rows, m.row, pw, m.inPanel), keys
|
||||
}
|
||||
|
||||
func slurmRow(n slurmsvc.NodeStatus) []string {
|
||||
state, jobs, slurmd, shared, note := "-", "-", "-", "-", n.Describe()
|
||||
if r := n.Status; r != nil {
|
||||
state = or(r.State, "-")
|
||||
switch r.Shared {
|
||||
case "":
|
||||
case "missing":
|
||||
shared = sBad.Render(r.Shared)
|
||||
default:
|
||||
shared = sOK.Render(r.Shared)
|
||||
}
|
||||
jobs = fmt.Sprint(r.Jobs)
|
||||
slurmd = sBad.Render("stopped")
|
||||
if r.SlurmdRunning {
|
||||
slurmd = sOK.Render("running")
|
||||
}
|
||||
if r.Error != "" {
|
||||
note = sBad.Render(r.Error)
|
||||
}
|
||||
}
|
||||
if !n.Connected {
|
||||
note = sWarn.Render("agent not connected") + "; " + note
|
||||
}
|
||||
whose := string(n.Phase)
|
||||
switch n.Phase {
|
||||
case slurmsvc.Lent:
|
||||
whose = sOK.Render("Slurm's")
|
||||
case slurmsvc.Llmgw:
|
||||
whose = "llmgw's"
|
||||
case slurmsvc.Lending, slurmsvc.Reclaiming:
|
||||
whose = sWarn.Render(whose)
|
||||
}
|
||||
return []string{n.Node, whose, state, jobs, slurmd, shared, note}
|
||||
}
|
||||
|
||||
// slurmMenu is what can be done with one of Slurm's nodes.
|
||||
func (m consoleModel) slurmMenu(n slurmsvc.NodeStatus) *menuDialog {
|
||||
var items []menuItem
|
||||
if n.Phase == slurmsvc.Lent || n.Phase == slurmsvc.Lending {
|
||||
items = append(items, menuItem{"Take back from Slurm…", func(m *consoleModel) tea.Cmd {
|
||||
m.confirmD = &confirmDialog{title: "Take " + n.Node + " back",
|
||||
text: "Slurm starts no new job there. llmgw places replicas on it again once the jobs running there have ended.",
|
||||
yes: func(m *consoleModel) tea.Cmd { return m.slurmLend(n.Node, false) }}
|
||||
return nil
|
||||
}})
|
||||
} else {
|
||||
items = append(items, menuItem{"Lend to Slurm…", func(m *consoleModel) tea.Cmd {
|
||||
text := "Its replicas move off first, each replacement up before the original stops; then Slurm runs jobs there."
|
||||
if n.Replicas > 0 {
|
||||
text = fmt.Sprintf("Its %d replica(s) move off first, each replacement up before the original stops; "+
|
||||
"a model with nowhere else to go stops serving. Then Slurm runs jobs there.", n.Replicas)
|
||||
}
|
||||
m.confirmD = &confirmDialog{title: "Lend " + n.Node + " to Slurm", text: text,
|
||||
yes: func(m *consoleModel) tea.Cmd { return m.slurmLend(n.Node, true) }}
|
||||
return nil
|
||||
}})
|
||||
}
|
||||
return &menuDialog{title: n.Node, items: items}
|
||||
}
|
||||
|
||||
// slurmLend puts a node in slurm.lent, or takes it out, as `llmgw ctl slurm
|
||||
// lend|reclaim` does: the newest settings, changed and put back.
|
||||
func (m *consoleModel) slurmLend(node string, add bool) tea.Cmd {
|
||||
title, note, done := "Lend "+node, "lent to Slurm from the console", node+" is being lent to Slurm: its replicas move off first."
|
||||
if !add {
|
||||
title, note, done = "Take back "+node, "taken back from Slurm in the console",
|
||||
node+" is being taken back: llmgw has it once its Slurm jobs end."
|
||||
}
|
||||
m.busy = title
|
||||
c := m.client
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
var cur adminapi.SettingsView
|
||||
if err := c.get(ctx, "/api/v1/settings", &cur); err != nil {
|
||||
return actionDoneMsg{title: title, err: err}
|
||||
}
|
||||
doc, err := settings.FromBytes("settings", []byte(cur.Body))
|
||||
if err == nil {
|
||||
err = lendChange(doc, []string{node}, add)
|
||||
}
|
||||
var body []byte
|
||||
if err == nil {
|
||||
body, err = doc.Bytes()
|
||||
}
|
||||
if err != nil {
|
||||
return actionDoneMsg{title: title, err: err}
|
||||
}
|
||||
req, _ := json.Marshal(adminapi.SettingsRequest{Base: cur.Version, Body: string(body), Note: note})
|
||||
var v adminapi.SettingsView
|
||||
if err := c.put(ctx, "/api/v1/settings", req, &v); err != nil {
|
||||
return actionDoneMsg{title: title, err: err}
|
||||
}
|
||||
return actionDoneMsg{title: title, text: done}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
)
|
||||
|
||||
func TestTheConsoleLendsANodeToSlurmFromItsRow(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
var put adminapi.SettingsRequest
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v1/settings":
|
||||
_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 7,
|
||||
Body: "slurm:\n enabled: true\n controller: node06\n nodes: [node05, cuda01]\n"})
|
||||
case r.Method == http.MethodPut && r.URL.Path == "/api/v1/settings":
|
||||
b, _ := io.ReadAll(r.Body)
|
||||
mu.Lock()
|
||||
_ = json.Unmarshal(b, &put)
|
||||
mu.Unlock()
|
||||
_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 8})
|
||||
default:
|
||||
_, _ = w.Write([]byte("[]"))
|
||||
}
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
m := newConsoleModel(srv.URL, "/nonexistent/server.yaml")
|
||||
m.loaded, m.menu, m.inPanel = true, cSlurm, true
|
||||
m.data.slurm = slurmsvc.Status{Enabled: true, Cluster: "llmgw", Controller: "node06", Partition: "gpu",
|
||||
Ctld: &slurmsvc.Reported{SlurmctldRunning: true, SlurmdbdRunning: true, Version: "23.11.4"},
|
||||
Shared: config.SharedStorage{Path: "/shared", Server: "node06"}, Users: "llmgw", UserCount: 2,
|
||||
Nodes: []slurmsvc.NodeStatus{
|
||||
{Node: "node05", Phase: slurmsvc.Lent, Connected: true,
|
||||
Status: &slurmsvc.Reported{State: "MIXED", Jobs: 2, SlurmdRunning: true, Shared: "mounted"}},
|
||||
{Node: "cuda01", Phase: slurmsvc.Llmgw, Replicas: 3, Connected: true,
|
||||
Status: &slurmsvc.Reported{State: "IDLE+DRAIN", SlurmdRunning: true}},
|
||||
}}
|
||||
var tm tea.Model = m
|
||||
view := tm.View()
|
||||
for _, want := range []string{"slurmctld 23.11.4 and slurmdbd running", "/shared, exported over NFS by node06", "users: 2", "node05", "Slurm's", "MIXED", "cuda01", "llmgw's"} {
|
||||
if !strings.Contains(view, want) {
|
||||
t.Errorf("the Slurm view has no %q:\n%s", want, view)
|
||||
}
|
||||
}
|
||||
|
||||
// cuda01, the second row: lent, after saying its replicas move.
|
||||
tm = press(tm, "down", "enter")
|
||||
if !strings.Contains(tm.View(), "Lend to Slurm") {
|
||||
t.Fatalf("no lend in the menu:\n%s", tm.View())
|
||||
}
|
||||
tm = press(tm, "enter")
|
||||
if !strings.Contains(tm.View(), "3 replica(s) move off first") {
|
||||
t.Fatalf("the question does not say what moves:\n%s", tm.View())
|
||||
}
|
||||
tm = pressRun(tm, "y")
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if put.Base != 7 || !strings.Contains(put.Body, "lent:") || !strings.Contains(put.Body, "cuda01") {
|
||||
t.Fatalf("the settings put back: %+v", put)
|
||||
}
|
||||
if !strings.Contains(tm.View(), "cuda01 is being lent to Slurm") {
|
||||
t.Errorf("not said:\n%s", tm.View())
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheConsoleTakesALentNodeBack(t *testing.T) {
|
||||
m, _ := consoleWith(t, cSlurm)
|
||||
m.data.slurm = slurmsvc.Status{Enabled: true, Controller: "node06",
|
||||
Nodes: []slurmsvc.NodeStatus{{Node: "node05", Phase: slurmsvc.Lent, Connected: true}}}
|
||||
var tm tea.Model = m
|
||||
tm = press(tm, "enter")
|
||||
if v := tm.View(); !strings.Contains(v, "Take back from Slurm") || strings.Contains(v, "Lend to Slurm") {
|
||||
t.Fatalf("a lent node's menu:\n%s", v)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheConsoleSaysHowToTurnSlurmOn(t *testing.T) {
|
||||
m, _ := consoleWith(t, cSlurm)
|
||||
var tm tea.Model = m
|
||||
if !strings.Contains(tm.View(), "llmgw ctl slurm setup") {
|
||||
t.Fatalf("Slurm off says nothing of turning it on:\n%s", tm.View())
|
||||
}
|
||||
}
|
||||
@@ -171,7 +171,10 @@ func newUserStateCmd(open userOpener, verb string, state store.UserState, short
|
||||
if err := s.DeleteUserSessions(cmd.Context(), u.ID); err != nil {
|
||||
return err
|
||||
}
|
||||
cmd.PrintErrln("Their sessions were closed. Their API keys still work — revoke those separately.")
|
||||
// The keys are refused by their owner's state, which every
|
||||
// router reads again within auth.cache_ttl.
|
||||
cmd.PrintErrln("Their sessions were closed. Their API keys are refused within auth.cache_ttl " +
|
||||
"(30 s by default), and work again if they are reinstated.")
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s is now %s\n", u.Email, state)
|
||||
return nil
|
||||
@@ -391,7 +394,8 @@ func newUserJoinCmd(open userOpener) *cobra.Command {
|
||||
Long: "A person can be a member of any number of them. A member of a team or\n" +
|
||||
"organization may use every project in it.",
|
||||
Example: " llmgw users join ayse@example.com --team Acme/ml\n" +
|
||||
" llmgw users join ayse@example.com --project Scania/research/vision",
|
||||
" llmgw users join ayse@example.com --project Scania/research/vision\n" +
|
||||
" llmgw users join lars@example.com --org Scania --role admin",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
s, err := open(cmd)
|
||||
@@ -415,7 +419,8 @@ func newUserJoinCmd(open userOpener) *cobra.Command {
|
||||
},
|
||||
}
|
||||
where.bind(cmd)
|
||||
cmd.Flags().StringVar(&role, "role", store.RoleMember, "member, or manager: with --org, may see the organization's page")
|
||||
cmd.Flags().StringVar(&role, "role", store.RoleMember,
|
||||
"member; with --org also manager, who sees the organization's page, or admin, who also changes its teams, projects, people and keys")
|
||||
cmd.Flags().StringVar(&by, "by", "", "who added them, for the record")
|
||||
return cmd
|
||||
}
|
||||
|
||||
@@ -158,6 +158,10 @@ type Server struct {
|
||||
// Firewall turns llmgw's firewall on every machine on or off.
|
||||
Firewall FirewallSettings `koanf:"firewall"`
|
||||
|
||||
// Slurm is a Slurm cluster run beside llmgw on some of its nodes, and
|
||||
// which of them it may use now.
|
||||
Slurm SlurmSettings `koanf:"slurm"`
|
||||
|
||||
// Presets is how the models that come with llmgw reach the catalog.
|
||||
Presets PresetSettings `koanf:"presets"`
|
||||
|
||||
@@ -960,6 +964,9 @@ func (s Server) Validate() error {
|
||||
if _, err := s.Facility.Policy(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.Slurm.Check(); err != nil {
|
||||
return err
|
||||
}
|
||||
if s.Control.Timezone != "" {
|
||||
if _, err := time.LoadLocation(s.Control.Timezone); err != nil {
|
||||
return fmt.Errorf("control.timezone %q: %w", s.Control.Timezone, err)
|
||||
|
||||
@@ -0,0 +1,204 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// SlurmSettings is a Slurm cluster run beside llmgw on some of its nodes
|
||||
// (BACKLOG P), for training and other batch work llmgw does not schedule.
|
||||
// The leader tells each named node's agent what to run; the agents install
|
||||
// it from the machine's package sources and keep it going.
|
||||
type SlurmSettings struct {
|
||||
// Enabled runs Slurm on the nodes below. False stops llmgw keeping it;
|
||||
// what was installed is left installed, and every node is llmgw's.
|
||||
Enabled bool `koanf:"enabled"`
|
||||
|
||||
// ClusterName is Slurm's ClusterName. Empty is "llmgw".
|
||||
ClusterName string `koanf:"cluster_name"`
|
||||
|
||||
// Controller is the llmgw node whose agent runs slurmctld. It needs no
|
||||
// GPU, and need not be one of Nodes.
|
||||
Controller string `koanf:"controller"`
|
||||
|
||||
// Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained
|
||||
// in Slurm, until it is also in Lent.
|
||||
Nodes []string `koanf:"nodes"`
|
||||
|
||||
// AllNodes runs slurmd on every llmgw node, those that join later too,
|
||||
// besides any named in Nodes. Each is llmgw's until it is lent, as any.
|
||||
AllNodes bool `koanf:"all_nodes"`
|
||||
|
||||
// Lent are the nodes of Nodes Slurm may run jobs on now. A node put here
|
||||
// is drained of llmgw's replicas first, as `ctl drain` does, and resumed
|
||||
// in Slurm once none is left; a node taken out is drained in Slurm and
|
||||
// given back to llmgw when its last job ends.
|
||||
Lent []string `koanf:"lent"`
|
||||
|
||||
// Partition is the one partition every node is in. Empty is "gpu".
|
||||
Partition string `koanf:"partition"`
|
||||
|
||||
// Shared is the file system every machine of the Slurm cluster sees at
|
||||
// the same path: home directories, datasets, the jobs' output and
|
||||
// checkpoints. Empty path: none, and a batch job's output stays on the
|
||||
// node it ran on.
|
||||
Shared SharedStorage `koanf:"shared"`
|
||||
|
||||
// Users is who keeps the Linux users that run jobs, the same on every
|
||||
// machine of the Slurm cluster: "llmgw" makes one for each active llmgw
|
||||
// account (its login from the e-mail address, a UID of llmgw's from
|
||||
// 200000 up, its SSH keys from the portal), let in to the controller,
|
||||
// and to a node only while a job of theirs runs there (pam_slurm_adopt);
|
||||
// empty leaves users to the site (a directory through sssd, or by hand).
|
||||
Users string `koanf:"users"`
|
||||
|
||||
// Containers lets jobs run in containers: `srun --container-image
|
||||
// nvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),
|
||||
// the GPUs and the shared path inside. The agents install both on
|
||||
// every Slurm machine; jobs take it once every machine has them.
|
||||
Containers bool `koanf:"containers"`
|
||||
}
|
||||
|
||||
// SharedStorage is where the Slurm cluster's shared files come from: the
|
||||
// site's NFS export (NFS), an NFS export llmgw keeps on one of its nodes
|
||||
// (Server), or, with neither, a file system the site mounts itself
|
||||
// (Lustre, BeeGFS, GPFS), which llmgw only looks for.
|
||||
type SharedStorage struct {
|
||||
// Path is where every machine sees it, as /shared.
|
||||
Path string `koanf:"path"`
|
||||
// NFS is the site's export, host:/path, mounted at Path on every
|
||||
// machine of the Slurm cluster.
|
||||
NFS string `koanf:"nfs"`
|
||||
// Server is the llmgw node, the controller or one of the nodes, that
|
||||
// exports Path over NFS to the others: for a site with no storage of
|
||||
// its own. One machine: not highly available, and no faster than its
|
||||
// disk and network.
|
||||
Server string `koanf:"server"`
|
||||
}
|
||||
|
||||
// UsersLlmgw has llmgw make the Slurm cluster's users from its accounts.
|
||||
const UsersLlmgw = "llmgw"
|
||||
|
||||
// HomeDir is where a user's home directory is: under the shared path when
|
||||
// there is one, else on each machine's own disk.
|
||||
func (s SlurmSettings) HomeDir(login string) string {
|
||||
if s.Shared.Path != "" {
|
||||
return path.Join(s.Shared.Path, "home", login)
|
||||
}
|
||||
return path.Join("/home", login)
|
||||
}
|
||||
|
||||
// nfsExport is host:/path.
|
||||
var nfsExport = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.:\[\]-]*:/[A-Za-z0-9_./-]*$`)
|
||||
|
||||
// sharedPath is an absolute path of plain names, which a mount unit's name
|
||||
// is made from.
|
||||
var sharedPath = regexp.MustCompile(`^(/[A-Za-z0-9_.-]+)+$`)
|
||||
|
||||
func (s SlurmSettings) checkShared() error {
|
||||
sh := s.Shared
|
||||
if sh.Path == "" {
|
||||
if sh.NFS != "" || sh.Server != "" {
|
||||
return fmt.Errorf("slurm.shared.path is empty: say where every machine sees the shared files, as /shared")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if !sharedPath.MatchString(sh.Path) || strings.Contains(sh.Path, "/.") {
|
||||
return fmt.Errorf("slurm.shared.path %q is not an absolute path of plain names, as /shared", sh.Path)
|
||||
}
|
||||
for _, sys := range []string{"/", "/etc", "/usr", "/var", "/var/lib", "/home", "/root", "/boot", "/opt", "/srv", "/tmp", "/proc", "/sys", "/dev", "/run"} {
|
||||
if path.Clean(sh.Path) == sys {
|
||||
return fmt.Errorf("slurm.shared.path %s is the system's own directory: take one of its own, as /shared", sh.Path)
|
||||
}
|
||||
}
|
||||
if sh.NFS != "" && sh.Server != "" {
|
||||
return fmt.Errorf("slurm.shared names both the site's export (nfs) and an llmgw node to export it (server): one of them")
|
||||
}
|
||||
if sh.NFS != "" && !nfsExport.MatchString(sh.NFS) {
|
||||
return fmt.Errorf("slurm.shared.nfs %q is not host:/path", sh.NFS)
|
||||
}
|
||||
if sh.Server != "" && sh.Server != s.Controller && !s.AllNodes && !slices.Contains(s.Nodes, sh.Server) {
|
||||
return fmt.Errorf("slurm.shared.server %q is neither the controller nor one of slurm.nodes", sh.Server)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// slurmName is what Slurm takes as a node, cluster or partition name.
|
||||
var slurmName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]{0,62}$`)
|
||||
|
||||
// Cluster is ClusterName or its default.
|
||||
func (s SlurmSettings) Cluster() string {
|
||||
if s.ClusterName == "" {
|
||||
return "llmgw"
|
||||
}
|
||||
return s.ClusterName
|
||||
}
|
||||
|
||||
// PartitionName is Partition or its default.
|
||||
func (s SlurmSettings) PartitionName() string {
|
||||
if s.Partition == "" {
|
||||
return "gpu"
|
||||
}
|
||||
return s.Partition
|
||||
}
|
||||
|
||||
// IsLent is whether Slurm may use the node now.
|
||||
func (s SlurmSettings) IsLent(node string) bool {
|
||||
return s.Enabled && (s.AllNodes || slices.Contains(s.Nodes, node)) && slices.Contains(s.Lent, node)
|
||||
}
|
||||
|
||||
// WithNodes is the settings with every node of ids in Nodes when AllNodes:
|
||||
// the nodes that run slurmd, as the leader's inventory has them.
|
||||
func (s SlurmSettings) WithNodes(ids []string) SlurmSettings {
|
||||
if !s.AllNodes {
|
||||
return s
|
||||
}
|
||||
nodes := slices.Clone(s.Nodes)
|
||||
for _, id := range ids {
|
||||
if !slices.Contains(nodes, id) {
|
||||
nodes = append(nodes, id)
|
||||
}
|
||||
}
|
||||
s.Nodes = nodes
|
||||
return s
|
||||
}
|
||||
|
||||
// Check refuses settings Slurm could not run on.
|
||||
func (s SlurmSettings) Check() error {
|
||||
if !s.Enabled {
|
||||
return nil
|
||||
}
|
||||
if s.Controller == "" {
|
||||
return fmt.Errorf("slurm.controller is empty: name the node that runs slurmctld")
|
||||
}
|
||||
if !slurmName.MatchString(s.Controller) {
|
||||
return fmt.Errorf("slurm.controller %q is not a name Slurm takes", s.Controller)
|
||||
}
|
||||
for _, n := range []struct{ key, v string }{{"slurm.cluster_name", s.ClusterName}, {"slurm.partition", s.Partition}} {
|
||||
if n.v != "" && !slurmName.MatchString(n.v) {
|
||||
return fmt.Errorf("%s %q is not a name Slurm takes: letters, digits, _ . -", n.key, n.v)
|
||||
}
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for i, n := range s.Nodes {
|
||||
if !slurmName.MatchString(n) {
|
||||
return fmt.Errorf("slurm.nodes[%d] %q is not a name Slurm takes", i, n)
|
||||
}
|
||||
if seen[n] {
|
||||
return fmt.Errorf("slurm.nodes names %s twice", n)
|
||||
}
|
||||
seen[n] = true
|
||||
}
|
||||
for i, n := range s.Lent {
|
||||
if !seen[n] && !s.AllNodes {
|
||||
return fmt.Errorf("slurm.lent[%d] %q is not in slurm.nodes: only a node that runs slurmd can be lent", i, n)
|
||||
}
|
||||
}
|
||||
if s.Users != "" && s.Users != UsersLlmgw {
|
||||
return fmt.Errorf("slurm.users %q: %q has llmgw make them, empty leaves them to the site", s.Users, UsersLlmgw)
|
||||
}
|
||||
return s.checkShared()
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSlurmCheck(t *testing.T) {
|
||||
ok := SlurmSettings{Enabled: true, Controller: "node06", Nodes: []string{"cuda01", "site2"}, Lent: []string{"cuda01"}}
|
||||
if err := ok.Check(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !ok.IsLent("cuda01") || ok.IsLent("site2") || ok.IsLent("node06") {
|
||||
t.Error("IsLent is wrong")
|
||||
}
|
||||
if ok.Cluster() != "llmgw" || ok.PartitionName() != "gpu" {
|
||||
t.Error("defaults are wrong")
|
||||
}
|
||||
if (SlurmSettings{Lent: []string{"x"}}).Check() != nil {
|
||||
t.Error("settings that are off were checked")
|
||||
}
|
||||
for want, s := range map[string]SlurmSettings{
|
||||
"controller is empty": {Enabled: true},
|
||||
"not a name": {Enabled: true, Controller: "bad name"},
|
||||
"twice": {Enabled: true, Controller: "c", Nodes: []string{"a", "a"}},
|
||||
"not in slurm.nodes": {Enabled: true, Controller: "c", Nodes: []string{"a"}, Lent: []string{"b"}},
|
||||
"slurm.partition": {Enabled: true, Controller: "c", Partition: "a b"},
|
||||
} {
|
||||
if err := s.Check(); err == nil || !strings.Contains(err.Error(), want) {
|
||||
t.Errorf("%+v: got %v, want %q", s, err, want)
|
||||
}
|
||||
}
|
||||
if (SlurmSettings{Enabled: false, Lent: []string{"a"}}).IsLent("a") {
|
||||
t.Error("a node is lent with Slurm off")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlurmShared(t *testing.T) {
|
||||
base := SlurmSettings{Enabled: true, Controller: "c", Nodes: []string{"a"}}
|
||||
for _, sh := range []SharedStorage{{}, {Path: "/shared"}, {Path: "/shared", NFS: "filer.site:/export/hpc"},
|
||||
{Path: "/data/hpc", Server: "c"}, {Path: "/shared", Server: "a"}} {
|
||||
s := base
|
||||
s.Shared = sh
|
||||
if err := s.Check(); err != nil {
|
||||
t.Errorf("%+v refused: %v", sh, err)
|
||||
}
|
||||
}
|
||||
for want, sh := range map[string]SharedStorage{
|
||||
"path is empty": {NFS: "f:/x"},
|
||||
"plain names": {Path: "shared"},
|
||||
"system's own": {Path: "/home"},
|
||||
"one of them": {Path: "/shared", NFS: "f:/x", Server: "c"},
|
||||
"not host:/path": {Path: "/shared", NFS: "f/x"},
|
||||
"neither": {Path: "/shared", Server: "z"},
|
||||
} {
|
||||
s := base
|
||||
s.Shared = sh
|
||||
if err := s.Check(); err == nil || !strings.Contains(err.Error(), want) {
|
||||
t.Errorf("%+v: got %v, want %q", sh, err, want)
|
||||
}
|
||||
}
|
||||
s := base
|
||||
s.Shared = SharedStorage{Path: "/a/../etc"}
|
||||
if s.Check() == nil {
|
||||
t.Error("a path with .. taken")
|
||||
}
|
||||
s.Shared, s.Users = SharedStorage{}, "ldap"
|
||||
if s.Check() == nil {
|
||||
t.Error("slurm.users ldap taken")
|
||||
}
|
||||
s.Users, s.Shared = UsersLlmgw, SharedStorage{Path: "/shared"}
|
||||
if s.Check() != nil || s.HomeDir("ayse") != "/shared/home/ayse" {
|
||||
t.Errorf("home %s", s.HomeDir("ayse"))
|
||||
}
|
||||
s.Shared = SharedStorage{}
|
||||
if s.HomeDir("ayse") != "/home/ayse" {
|
||||
t.Errorf("home %s", s.HomeDir("ayse"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlurmAllNodes(t *testing.T) {
|
||||
s := SlurmSettings{Enabled: true, Controller: "c", AllNodes: true, Lent: []string{"joined"}}
|
||||
if err := s.Check(); err != nil {
|
||||
t.Fatalf("a node lent with all_nodes refused: %v", err)
|
||||
}
|
||||
if !s.IsLent("joined") {
|
||||
t.Fatal("not lent with all_nodes")
|
||||
}
|
||||
if got := s.WithNodes([]string{"a", "joined"}).Nodes; len(got) != 2 {
|
||||
t.Fatalf("nodes %v", got)
|
||||
}
|
||||
s.AllNodes = false
|
||||
if s.Check() == nil || len(s.WithNodes([]string{"a"}).Nodes) != 0 {
|
||||
t.Fatal("all_nodes off still takes the inventory's nodes")
|
||||
}
|
||||
}
|
||||
@@ -22,6 +22,7 @@ import (
|
||||
"unicode"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
@@ -77,7 +78,9 @@ type API struct {
|
||||
// tenancy is organizations, keys and people; nil without a store.
|
||||
tenancy TenancyStore
|
||||
// train is adapter training; nil without a store and agents.
|
||||
train *TrainAdmin
|
||||
train *TrainAdmin
|
||||
// slurm is the Slurm cluster beside llmgw; nil without a store and agents.
|
||||
slurm SlurmService
|
||||
settings *Settings
|
||||
auth *AuthAdmin
|
||||
mcp MCPStore
|
||||
@@ -95,6 +98,10 @@ type API struct {
|
||||
usage Usage
|
||||
quotas Quotas
|
||||
ctl *control.Controller
|
||||
// slurmUsers are the Linux users made from the accounts; nil without a store.
|
||||
slurmUsers SlurmUsers
|
||||
// webhooks counts the settings' notify.webhooks; nil does not say.
|
||||
webhooks func() int
|
||||
}
|
||||
|
||||
// New builds the API over a controller.
|
||||
@@ -130,10 +137,12 @@ func (a *API) Register(mux Mux) {
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/start", a.startModel)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/replicas", a.setProfileReplicas)
|
||||
a.registerQuotas(mux)
|
||||
a.registerKB(mux)
|
||||
a.registerTenancy(mux)
|
||||
a.registerTrain(mux)
|
||||
a.registerSlurm(mux)
|
||||
a.registerGuard(mux)
|
||||
a.registerSettings(mux)
|
||||
a.registerAuth(mux)
|
||||
@@ -259,6 +268,9 @@ type NodeView struct {
|
||||
// MemBandwidthGBs is how fast its memory reads, in GB/s, measured when
|
||||
// its agent started; 0 when not reported. A model's speed on the CPU.
|
||||
MemBandwidthGBs float64 `json:"mem_bandwidth_gbs,omitempty"`
|
||||
// MemBandwidthCoreGBs is what one core reads alone, in GB/s; 0 when not
|
||||
// reported. A CPU replica's threads read at about this each.
|
||||
MemBandwidthCoreGBs float64 `json:"mem_bandwidth_core_gbs,omitempty"`
|
||||
// Engines are the inference engines installed here, as the agent found
|
||||
// them. Absent means the agent is older than the probe and said nothing,
|
||||
// which is not the same as having none — placement treats it that way too,
|
||||
@@ -489,6 +501,9 @@ type ModelView struct {
|
||||
Unplaceable bool `json:"unplaceable,omitempty"`
|
||||
Fallbacks []string `json:"fallbacks,omitempty"`
|
||||
Profiles []string `json:"profiles"`
|
||||
// ProfileReplicas is how many replicas run on each profile, for a model
|
||||
// whose replicas are counted per profile; absent otherwise.
|
||||
ProfileReplicas map[string]int `json:"profile_replicas,omitempty"`
|
||||
// MemoryGB and CPUCores are what one replica reserves, from its first
|
||||
// profile; MeasuredMemoryGB is the most one of its replicas uses now, 0
|
||||
// when none is measured.
|
||||
@@ -723,7 +738,7 @@ func (a *API) Nodes() []NodeView {
|
||||
v.EngineBuilds[e] = b
|
||||
}
|
||||
}
|
||||
v.MemBandwidthGBs = n.Host.MemBandwidthGBs
|
||||
v.MemBandwidthGBs, v.MemBandwidthCoreGBs = n.Host.MemBandwidthGBs, n.Host.MemBandwidthCoreGBs
|
||||
for _, g := range n.GPUs {
|
||||
if g.MIG != nil {
|
||||
continue // said of the card it is carved from
|
||||
@@ -865,6 +880,12 @@ func (a *API) Models() []ModelView {
|
||||
Canary: m.Canary, CanaryOf: a.ctl.Catalog().CanaryOf(m.Name), Eval: m.Eval,
|
||||
Role: m.Role,
|
||||
}
|
||||
if m.Split() {
|
||||
v.ProfileReplicas = map[string]int{}
|
||||
for _, p := range m.Profiles {
|
||||
v.ProfileReplicas[p.Name] = p.Replicas
|
||||
}
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
v.Profiles = append(v.Profiles, p.Name)
|
||||
}
|
||||
@@ -962,6 +983,15 @@ type Models interface {
|
||||
SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
|
||||
SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
|
||||
SetAdapters(ctx context.Context, name string, adapters []catalog.Adapter, by string) (catalog.Model, error)
|
||||
SetProfileReplicas(ctx context.Context, name string, counts map[string]int, cpu *modelsvc.CPUProfile, by string) (catalog.Model, error)
|
||||
}
|
||||
|
||||
// ProfileReplicasRequest counts a model's replicas per profile: profile
|
||||
// name to replicas, and CPU, for a llama.cpp model with no profile on CPUs,
|
||||
// the one to add.
|
||||
type ProfileReplicasRequest struct {
|
||||
Profiles map[string]int `json:"profiles"`
|
||||
CPU *modelsvc.CPUProfile `json:"cpu,omitempty"`
|
||||
}
|
||||
|
||||
// ResourcesRequest sets what a model's profiles reserve of a node.
|
||||
@@ -1288,6 +1318,25 @@ func (a *API) setResources(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, http.StatusOK, m)
|
||||
}
|
||||
|
||||
// setProfileReplicas counts a model's replicas per profile.
|
||||
func (a *API) setProfileReplicas(w http.ResponseWriter, r *http.Request) {
|
||||
if a.models == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
return
|
||||
}
|
||||
var req ProfileReplicasRequest
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
m, err := a.models.SetProfileReplicas(r.Context(), r.PathValue("name"), req.Profiles, req.CPU, actorOf(r))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, m)
|
||||
}
|
||||
|
||||
func (a *API) removeModel(w http.ResponseWriter, r *http.Request) {
|
||||
if a.models == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store")
|
||||
@@ -1966,6 +2015,15 @@ func PoolNodesOf(nodes []NodeView) []modelplan.PoolNode {
|
||||
if l := n.Load; l != nil && l.MemTotal > 0 {
|
||||
p.HostFreeGB = int((l.MemTotal - l.MemUsed) >> 30)
|
||||
}
|
||||
p.CPUs, p.MemBWGBs, p.MemBWCoreGBs = n.CPUCount, n.MemBandwidthGBs, n.MemBandwidthCoreGBs
|
||||
for _, g := range n.GPUs {
|
||||
if bw := modelplan.GPUBandwidthGBs(g.Name); bw > 0 {
|
||||
if p.CardBWGBs == nil {
|
||||
p.CardBWGBs = map[string]float64{}
|
||||
}
|
||||
p.CardBWGBs[g.Class] = bw
|
||||
}
|
||||
}
|
||||
out = append(out, p)
|
||||
}
|
||||
return out
|
||||
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
@@ -263,6 +264,16 @@ func (f *fakeModels) SetAdapters(_ context.Context, name string, as []catalog.Ad
|
||||
return catalog.Model{Name: name, Adapters: as}, f.err
|
||||
}
|
||||
|
||||
func (f *fakeModels) SetProfileReplicas(_ context.Context, name string, counts map[string]int, _ *modelsvc.CPUProfile,
|
||||
_ string,
|
||||
) (catalog.Model, error) {
|
||||
m := catalog.Model{Name: name}
|
||||
for p, n := range counts {
|
||||
m.Profiles = append(m.Profiles, catalog.Profile{Name: p, Replicas: n})
|
||||
}
|
||||
return m, f.err
|
||||
}
|
||||
|
||||
func (f *fakeModels) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
|
||||
return catalog.Model{Name: name, Owner: owner}, f.err
|
||||
}
|
||||
|
||||
@@ -6,6 +6,10 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
|
||||
)
|
||||
|
||||
// SetWebhooks has the summary say when the settings name no webhook:
|
||||
// count is how many they name now.
|
||||
func (a *API) SetWebhooks(count func() int) { a.webhooks = count }
|
||||
|
||||
// SetNotifier gives the API the webhooks `ctl notify test` reaches.
|
||||
func (a *API) SetNotifier(n *notify.Notifier) { a.notifier = n }
|
||||
|
||||
|
||||
@@ -2,6 +2,8 @@ package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
@@ -48,10 +50,29 @@ type PurgeView struct {
|
||||
// purgeModel removes a model from the catalog and its files from the mirror
|
||||
// and the nodes.
|
||||
func (a *API) purgeModel(w http.ResponseWriter, r *http.Request, name string) {
|
||||
out, err := a.Purge(r.Context(), name, actorOf(r))
|
||||
switch {
|
||||
case errors.Is(err, errNoSuchModel):
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
case err != nil:
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
default:
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
}
|
||||
|
||||
// errNoSuchModel is a model the catalog does not have.
|
||||
var errNoSuchModel = errors.New("no such model in the catalog")
|
||||
|
||||
// Purge removes a model from the catalog and its files from the mirror and
|
||||
// the nodes, waiting a while for its replicas to stop before the nodes'.
|
||||
func (a *API) Purge(ctx context.Context, name, by string) (PurgeView, error) {
|
||||
if a.models == nil {
|
||||
return PurgeView{}, errors.New("this control plane has no store")
|
||||
}
|
||||
m, ok := a.ctl.Catalog().Resolve(name)
|
||||
if !ok {
|
||||
writeError(w, http.StatusNotFound, "no model "+name+" in the catalog")
|
||||
return
|
||||
return PurgeView{}, fmt.Errorf("%w: %s", errNoSuchModel, name)
|
||||
}
|
||||
name = m.Name
|
||||
others := []catalog.Model{}
|
||||
@@ -60,37 +81,35 @@ func (a *API) purgeModel(w http.ResponseWriter, r *http.Request, name string) {
|
||||
others = append(others, o)
|
||||
}
|
||||
}
|
||||
if err := a.models.Remove(r.Context(), name); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
if err := a.models.Remove(ctx, name); err != nil {
|
||||
return PurgeView{}, err
|
||||
}
|
||||
a.presetDeclined(r.Context(), name, actorOf(r))
|
||||
a.presetDeclined(ctx, name, by)
|
||||
out := PurgeView{Removed: name}
|
||||
|
||||
sources := purgeSources(m)
|
||||
if a.purgeMirror == nil {
|
||||
out.Note = "this control plane has no mirror; its files there, if any, are left"
|
||||
} else {
|
||||
out.Mirror, out.Kept, out.Note = purgeFromMirror(r.Context(), a.purgeMirror, sources, others)
|
||||
out.Mirror, out.Kept, out.Note = purgeFromMirror(ctx, a.purgeMirror, sources, others)
|
||||
}
|
||||
|
||||
// The nodes' copies, once nothing runs on them.
|
||||
deadline := time.Now().Add(purgeWait)
|
||||
for a.runs(name) && time.Now().Before(deadline) {
|
||||
select {
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
case <-ctx.Done():
|
||||
return out, ctx.Err()
|
||||
case <-time.After(time.Second):
|
||||
}
|
||||
}
|
||||
if a.runs(name) {
|
||||
out.Note = strings.TrimSpace(out.Note + " Its replicas had not stopped yet: its copies on the nodes are " +
|
||||
"left, and llmgw ctl weights --collect deletes them.")
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
return
|
||||
return out, nil
|
||||
}
|
||||
out.Nodes = a.purgeFromNodes(r.Context(), sources, others)
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
out.Nodes = a.purgeFromNodes(ctx, sources, others)
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// runs says whether any replica of a model is still on a node.
|
||||
@@ -197,7 +216,7 @@ func (a *API) purgeFromNodes(ctx context.Context, sources []string, others []cat
|
||||
names[filepath.Base(s)] = true
|
||||
}
|
||||
wanted := inUse(others)
|
||||
var out []CollectResult
|
||||
var rs []removal
|
||||
for _, n := range a.ctl.Inventory().Nodes() {
|
||||
size := map[string]int64{}
|
||||
have := make([]string, 0, len(n.Weights))
|
||||
@@ -216,15 +235,14 @@ func (a *API) purgeFromNodes(ctx context.Context, sources []string, others []cat
|
||||
if len(paths) == 0 {
|
||||
continue
|
||||
}
|
||||
res := CollectResult{Node: n.ID, Files: len(paths)}
|
||||
for _, p := range paths {
|
||||
res.Bytes += size[p]
|
||||
rs = append(rs, removal{node: n.ID, paths: paths, size: size})
|
||||
}
|
||||
out := a.removeConfirmed(ctx, rm, rs)
|
||||
for _, r := range out {
|
||||
if r.Error != "" || r.Left > 0 {
|
||||
slog.Warn("a purged model's copies were not all deleted on a node", "node", r.Node,
|
||||
"left", r.Left, "err", r.Error)
|
||||
}
|
||||
if err := rm.RemoveWeights(ctx, n.ID, paths); err != nil {
|
||||
res.Error, res.Files, res.Bytes = err.Error(), 0, 0
|
||||
slog.Warn("a purged model's copies could not be deleted on a node", "node", n.ID, "err", err)
|
||||
}
|
||||
out = append(out, res)
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/federation"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
@@ -84,6 +85,7 @@ var PathParams = map[string]string{
|
||||
"projects/{id}": "The project's ID.",
|
||||
"keys/{id}": "The key's ID, as listed (not the key itself).",
|
||||
"users/{id}": "The person's ID or email.",
|
||||
"users/{user}": "The person's ID or email.",
|
||||
"artifacts/{kind}": "What the job made: peft, a gzipped tar of the PEFT adapter, or gguf, the adapter for llama.cpp.",
|
||||
}
|
||||
|
||||
@@ -168,6 +170,10 @@ var Routes = []Route{
|
||||
Response: ScaleResponse{}},
|
||||
{Method: "POST", Path: "/api/v1/models/{name}/scale", Summary: "Set how many replicas a model runs",
|
||||
Request: ScaleRequest{}, Response: ScaleResponse{}},
|
||||
{Method: "POST", Path: "/api/v1/models/{name}/replicas", Summary: "How many replicas run on each profile",
|
||||
Description: "Seven on cards and four on CPUs, under the model's name. A profile not named keeps its count; " +
|
||||
"cpu adds a profile on CPUs to a llama.cpp model that has none. Its scaling becomes the sum.",
|
||||
Request: ProfileReplicasRequest{}, Response: catalog.Model{}},
|
||||
{Method: "POST", Path: "/api/v1/models/{name}/resources", Summary: "Set what a model's profiles reserve of a node",
|
||||
Request: ResourcesRequest{}, Response: catalog.Model{}},
|
||||
{Method: "POST", Path: "/api/v1/models/{name}/class", Summary: "Set a model's capacity class",
|
||||
@@ -388,6 +394,20 @@ var Routes = []Route{
|
||||
Description: "Their keys in projects they no longer reach are revoked.",
|
||||
Request: MembershipRequest{}, Response: UserView{}},
|
||||
|
||||
{Method: "GET", Path: "/api/v1/slurm", Summary: "The Slurm cluster beside llmgw, and where each of its nodes is",
|
||||
Description: "From the cluster's settings (slurm.*) and what each node's agent last said. A node's phase is llmgw " +
|
||||
"(drained in Slurm, no job left), lending (its replicas moving off), lent, reclaiming (drained in Slurm, jobs " +
|
||||
"still running) or unknown (its agent has not said).",
|
||||
Response: slurmsvc.Status{}},
|
||||
{Method: "GET", Path: "/api/v1/slurm/users", Summary: "The Linux users made from llmgw's accounts for the Slurm cluster",
|
||||
Description: "Every active account, with its login, UID and SSH keys; an account is given its login and UID the first " +
|
||||
"time it is listed. The machines have them when slurm.users is \"llmgw\".",
|
||||
Response: []slurmsvc.User{}},
|
||||
{Method: "PUT", Path: "/api/v1/slurm/users/{user}/ssh-keys", Summary: "Replace an account's SSH public keys",
|
||||
Description: "{user} is an ID or e-mail. keys is OpenSSH public keys, one a line, without options; empty takes them " +
|
||||
"all away. The machines have them within a minute.",
|
||||
Request: SlurmKeysRequest{}, Response: slurmsvc.User{}},
|
||||
|
||||
{Method: "POST", Path: "/api/v1/train/jobs", Summary: "Queue a LoRA adapter's training on a node",
|
||||
Description: "The node trains when its agent is there, nothing else trains on it, and a card has gpu_memory_mib free; " +
|
||||
"the pairs are NDJSON of {\"q\", \"a\"}. An adapter learns how answers sound far better than what they say.",
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// SlurmService is the Slurm cluster beside llmgw (BACKLOG P), as the leader
|
||||
// keeps it.
|
||||
type SlurmService interface {
|
||||
Status() slurmsvc.Status
|
||||
UsersChanged()
|
||||
}
|
||||
|
||||
// SlurmUsers are the Linux users llmgw makes from its accounts.
|
||||
type SlurmUsers interface {
|
||||
SlurmUsers(ctx context.Context) ([]store.SlurmUser, error)
|
||||
SlurmUserByID(ctx context.Context, userID string) (store.SlurmUser, error)
|
||||
SetSlurmSSHKeys(ctx context.Context, userID string, keys []string) error
|
||||
}
|
||||
|
||||
// SetSlurmUsers enables the Slurm users' endpoints.
|
||||
func (a *API) SetSlurmUsers(u SlurmUsers) { a.slurmUsers = u }
|
||||
|
||||
// SlurmKeysRequest is a person's SSH public keys, one a line, as
|
||||
// authorized_keys has them; empty takes them all away.
|
||||
type SlurmKeysRequest struct {
|
||||
Keys string `json:"keys"`
|
||||
}
|
||||
|
||||
// SlurmUserView is the user from a store row.
|
||||
func SlurmUserView(u store.SlurmUser) slurmsvc.User {
|
||||
return slurmsvc.User{Login: u.Login, UID: u.UID, Name: u.Name, Email: u.Email, SSHKeys: u.SSHKeys}
|
||||
}
|
||||
|
||||
// SetSlurm gives the API the Slurm service.
|
||||
func (a *API) SetSlurm(s SlurmService) { a.slurm = s }
|
||||
|
||||
func (a *API) registerSlurm(mux Mux) {
|
||||
mux.HandleFunc("GET /api/v1/slurm", a.slurmStatus)
|
||||
mux.HandleFunc("GET /api/v1/slurm/users", a.slurmUserList)
|
||||
mux.HandleFunc("PUT /api/v1/slurm/users/{user}/ssh-keys", a.slurmUserKeys)
|
||||
}
|
||||
|
||||
func (a *API) slurmUserList(w http.ResponseWriter, r *http.Request) {
|
||||
if a.slurmUsers == nil {
|
||||
writeError(w, http.StatusNotImplemented, "the Slurm users need the store, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
us, err := a.slurmUsers.SlurmUsers(r.Context())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
out := make([]slurmsvc.User, 0, len(us))
|
||||
for _, u := range us {
|
||||
out = append(out, SlurmUserView(u))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
// slurmUserKeys replaces a person's SSH keys; {user} is an ID or e-mail.
|
||||
func (a *API) slurmUserKeys(w http.ResponseWriter, r *http.Request) {
|
||||
if a.slurmUsers == nil {
|
||||
writeError(w, http.StatusNotImplemented, "the Slurm users need the store, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
var in SlurmKeysRequest
|
||||
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 64<<10)).Decode(&in); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
keys, err := slurm.ParseSSHKeys(in.Keys)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
u, err := a.findUser(ctx, r.PathValue("user"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusNotFound, strings.TrimPrefix(err.Error(), store.ErrNotFound.Error()+": "))
|
||||
return
|
||||
}
|
||||
if err := a.slurmUsers.SetSlurmSSHKeys(ctx, u.ID, keys); err != nil {
|
||||
code := http.StatusInternalServerError
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
code = http.StatusNotFound
|
||||
}
|
||||
writeError(w, code, err.Error())
|
||||
return
|
||||
}
|
||||
if a.slurm != nil {
|
||||
a.slurm.UsersChanged()
|
||||
}
|
||||
su, err := a.slurmUsers.SlurmUserByID(ctx, u.ID)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, SlurmUserView(su))
|
||||
}
|
||||
|
||||
func (a *API) slurmStatus(w http.ResponseWriter, _ *http.Request) {
|
||||
if a.slurm == nil {
|
||||
writeError(w, http.StatusNotImplemented, "Slurm needs the store and the nodes' agents, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, a.slurm.Status())
|
||||
}
|
||||
@@ -50,6 +50,9 @@ type Summary struct {
|
||||
// GuardOff says the guardrails are turned off (guard.enabled: false):
|
||||
// no request is checked, whatever the policies say.
|
||||
GuardOff bool `json:"guard_off,omitempty"`
|
||||
// NoAlerts says the cluster's settings name no webhook: a model parked,
|
||||
// a node lost or a judge down is told nobody.
|
||||
NoAlerts bool `json:"no_alerts,omitempty"`
|
||||
}
|
||||
|
||||
// StoreMember is one member of a replicated store, as its Patroni reports it.
|
||||
@@ -160,6 +163,9 @@ func (a *API) Summary() Summary {
|
||||
if a.guard != nil && a.guard.Library != nil && !a.guard.Library().On() {
|
||||
s.GuardOff = true
|
||||
}
|
||||
if a.webhooks != nil && a.webhooks() == 0 {
|
||||
s.NoAlerts = true
|
||||
}
|
||||
for _, j := range a.Judges() {
|
||||
if j.Problem != "" {
|
||||
s.Judges = append(s.Judges, j)
|
||||
|
||||
@@ -153,3 +153,16 @@ func TestOneSharedCopyCountsAsMuchAsTwenty(t *testing.T) {
|
||||
t.Errorf("counted %d models; three nodes seeing one file is one model", f.Models)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheSummarySaysWhenNoWebhookIsSet(t *testing.T) {
|
||||
a, _ := setup(t, testModels(), gpuNode("n1", 8))
|
||||
n := 0
|
||||
a.SetWebhooks(func() int { return n })
|
||||
if !a.Summary().NoAlerts {
|
||||
t.Error("no webhook, not said")
|
||||
}
|
||||
n = 1
|
||||
if a.Summary().NoAlerts {
|
||||
t.Error("a webhook set, yet said there is none")
|
||||
}
|
||||
}
|
||||
@@ -634,8 +634,12 @@ func (a *API) membership(add bool) http.HandlerFunc {
|
||||
if role == "" {
|
||||
role = store.RoleMember
|
||||
}
|
||||
if role != store.RoleMember && role != store.RoleManager {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member or manager", role))
|
||||
if role != store.RoleMember && role != store.RoleManager && role != store.RoleOrgAdmin {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member, manager or admin", role))
|
||||
return
|
||||
}
|
||||
if role != store.RoleMember && scope != store.ScopeOrg {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("%s is a role on an organization", role))
|
||||
return
|
||||
}
|
||||
if err := a.tenancy.AddMembership(ctx, u.ID, scope, id, role, actorOf(r), time.Now()); err != nil {
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
)
|
||||
|
||||
// What the portal changes of a served model beside what the catalog holds:
|
||||
// the same as the API's routes, called in the process.
|
||||
|
||||
// SetModelContext changes a served model's context and longest answer.
|
||||
func (a *API) SetModelContext(ctx context.Context, name string, req modelserve.ContextRequest, by string) (modelserve.ContextChange, error) {
|
||||
if a.serve == nil {
|
||||
return modelserve.ContextChange{}, errors.New("this control plane cannot change a model's context")
|
||||
}
|
||||
return a.serve.SetContext(ctx, name, req, by)
|
||||
}
|
||||
|
||||
// SetModelGPUs spreads a served model over more cards a replica, or fewer.
|
||||
func (a *API) SetModelGPUs(ctx context.Context, name string, req modelserve.GPUsRequest, by string) (modelserve.GPUsChange, error) {
|
||||
if a.serve == nil {
|
||||
return modelserve.GPUsChange{}, errors.New("this control plane cannot change a model's cards")
|
||||
}
|
||||
return a.serve.SetGPUs(ctx, name, req, by)
|
||||
}
|
||||
|
||||
// RestartModel starts each of a model's ready replicas again, one at a time,
|
||||
// and says how many.
|
||||
func (a *API) RestartModel(name string) (int, error) {
|
||||
var ids []string
|
||||
for _, rep := range a.ctl.Reconciler().ReadyReplicas() {
|
||||
if rep.Model == name {
|
||||
ids = append(ids, rep.ID)
|
||||
}
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
return 0, errors.New("none of its replicas is ready to restart")
|
||||
}
|
||||
return len(ids), a.ctl.Reconciler().RestartPaced(reconciler.RestartPace{}, false, ids...)
|
||||
}
|
||||
|
||||
// RemoveModel takes a model out of the catalog, with its files on the mirror
|
||||
// and the nodes when purge.
|
||||
func (a *API) RemoveModel(ctx context.Context, name string, purge bool, by string) (PurgeView, error) {
|
||||
if purge {
|
||||
return a.Purge(ctx, name, by)
|
||||
}
|
||||
if a.models == nil {
|
||||
return PurgeView{}, errors.New("this control plane has no store")
|
||||
}
|
||||
if err := a.models.Remove(ctx, name); err != nil {
|
||||
return PurgeView{}, err
|
||||
}
|
||||
a.presetDeclined(ctx, name, by)
|
||||
return PurgeView{Removed: name}, nil
|
||||
}
|
||||
@@ -1,11 +1,14 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
)
|
||||
|
||||
// OrphanView is cached weights on one node that no catalog model accounts for.
|
||||
@@ -71,12 +74,100 @@ type CollectRequest struct {
|
||||
Node string `json:"node,omitempty"`
|
||||
}
|
||||
|
||||
// CollectResult is what was deleted on one node, or why nothing was.
|
||||
// CollectResult is what was deleted on one node, or why nothing was: what
|
||||
// the node no longer lists once asked, not what it was asked to delete.
|
||||
type CollectResult struct {
|
||||
Node string `json:"node"`
|
||||
Files int `json:"files"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
Error string `json:"error,omitempty"`
|
||||
// Left and LeftBytes are what the node still lists: refused, or not
|
||||
// gone by the time the sweep stopped waiting. Its agent's log says why.
|
||||
Left int `json:"left,omitempty"`
|
||||
LeftBytes int64 `json:"left_bytes,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// confirmWait is how long a sweep waits for the nodes to list their files
|
||||
// without the ones deleted: they scan theirs every heartbeat, five seconds
|
||||
// by default, and a large set takes a moment to delete.
|
||||
var confirmWait = 30 * time.Second
|
||||
|
||||
// removal is what one node is asked to delete, with each file's size.
|
||||
type removal struct {
|
||||
node string
|
||||
paths []string
|
||||
size map[string]int64
|
||||
}
|
||||
|
||||
// removeConfirmed asks each node to delete its paths and says what each then
|
||||
// no longer lists. Counted as asked, a sweep said "25 GB freed" of files the
|
||||
// node had refused and kept.
|
||||
func (a *API) removeConfirmed(ctx context.Context, rm control.WeightRemover, rs []removal) []CollectResult {
|
||||
out := make([]CollectResult, len(rs))
|
||||
pending := map[int]bool{}
|
||||
for i, r := range rs {
|
||||
out[i].Node = r.node
|
||||
if err := rm.RemoveWeights(ctx, r.node, r.paths); err != nil {
|
||||
out[i].Error = err.Error()
|
||||
for _, p := range r.paths {
|
||||
out[i].Left++
|
||||
out[i].LeftBytes += r.size[p]
|
||||
}
|
||||
continue
|
||||
}
|
||||
pending[i] = true
|
||||
}
|
||||
deadline := time.Now().Add(confirmWait)
|
||||
for {
|
||||
for i := range pending {
|
||||
if left := a.stillListed(rs[i]); len(left) == 0 {
|
||||
delete(pending, i)
|
||||
}
|
||||
}
|
||||
if len(pending) == 0 || !time.Now().Before(deadline) {
|
||||
break
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
deadline = time.Now()
|
||||
case <-time.After(time.Second):
|
||||
}
|
||||
}
|
||||
for i, r := range rs {
|
||||
if out[i].Error != "" {
|
||||
continue
|
||||
}
|
||||
left := a.stillListed(r)
|
||||
for _, p := range r.paths {
|
||||
if left[p] {
|
||||
out[i].Left++
|
||||
out[i].LeftBytes += r.size[p]
|
||||
} else {
|
||||
out[i].Files++
|
||||
out[i].Bytes += r.size[p]
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// stillListed is which of a removal's paths its node still lists.
|
||||
func (a *API) stillListed(r removal) map[string]bool {
|
||||
left := map[string]bool{}
|
||||
n, ok := a.ctl.Inventory().Get(r.node)
|
||||
if !ok {
|
||||
return left
|
||||
}
|
||||
want := map[string]bool{}
|
||||
for _, p := range r.paths {
|
||||
want[p] = true
|
||||
}
|
||||
for _, w := range n.Weights {
|
||||
if want[w.Path] {
|
||||
left[w.Path] = true
|
||||
}
|
||||
}
|
||||
return left
|
||||
}
|
||||
|
||||
// collectWeights deletes them.
|
||||
@@ -97,20 +188,26 @@ func (a *API) collectWeights(w http.ResponseWriter, r *http.Request) {
|
||||
// An empty body is the "all nodes" case and must not be an error.
|
||||
_ = json.NewDecoder(r.Body).Decode(&body)
|
||||
|
||||
out := make([]CollectResult, 0)
|
||||
// A node that has gone since the list was built is reported rather than
|
||||
// failing the whole sweep: the other nodes' space is still worth
|
||||
// freeing, and this one will be offered again next time.
|
||||
var rs []removal
|
||||
for _, o := range a.Orphans() {
|
||||
if body.Node != "" && o.Node != body.Node {
|
||||
continue
|
||||
}
|
||||
res := CollectResult{Node: o.Node, Files: len(o.Paths), Bytes: o.Bytes}
|
||||
// A node that has gone since the list was built is reported rather than
|
||||
// failing the whole sweep: the other nodes' space is still worth
|
||||
// freeing, and this one will be offered again next time.
|
||||
if err := rm.RemoveWeights(r.Context(), o.Node, o.Paths); err != nil {
|
||||
res.Error = err.Error()
|
||||
res.Files, res.Bytes = 0, 0
|
||||
}
|
||||
out = append(out, res)
|
||||
rs = append(rs, removal{node: o.Node, paths: o.Paths, size: a.sizesOn(o.Node)})
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
writeJSON(w, http.StatusOK, a.removeConfirmed(r.Context(), rm, rs))
|
||||
}
|
||||
|
||||
// sizesOn is each file a node lists, by path, with its size.
|
||||
func (a *API) sizesOn(node string) map[string]int64 {
|
||||
out := map[string]int64{}
|
||||
if n, ok := a.ctl.Inventory().Get(node); ok {
|
||||
for _, w := range n.Weights {
|
||||
out[w.Path] = w.SizeBytes
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
// deletingDispatcher deletes what it is asked to as a node would, by its
|
||||
// files dropping out of what the node lists — but for refused, which it
|
||||
// keeps, as an agent keeps a file it cannot prove it fetched.
|
||||
type deletingDispatcher struct {
|
||||
nullDispatcher
|
||||
inv *inventory.Store
|
||||
refused string
|
||||
}
|
||||
|
||||
func (d *deletingDispatcher) RemoveWeights(_ context.Context, node string, paths []string) error {
|
||||
n, _ := d.inv.Get(node)
|
||||
n.Weights = slices.DeleteFunc(n.Weights, func(w discovery.LocalWeights) bool {
|
||||
return w.Path != d.refused && slices.Contains(paths, w.Path)
|
||||
})
|
||||
d.inv.Upsert(n)
|
||||
return nil
|
||||
}
|
||||
|
||||
// What a sweep says it freed is what the node no longer lists: a file it
|
||||
// kept is said to be left, not counted as freed.
|
||||
func TestASweepSaysWhatWasDeletedNotWhatWasAsked(t *testing.T) {
|
||||
confirmWait = 1500 * time.Millisecond
|
||||
t.Cleanup(func() { confirmWait = 30 * time.Second })
|
||||
|
||||
gone := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/old/v1/old.gguf")
|
||||
set := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/big/v1")
|
||||
kept := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/stuck/v1/stuck.gguf")
|
||||
cat, _ := catalog.New(nil)
|
||||
d := &deletingDispatcher{refused: kept}
|
||||
ctl := control.New(control.DefaultConfig(), catalog.NewLive(cat), d, nullSink{}, slog.New(slog.DiscardHandler))
|
||||
d.inv = ctl.Inventory()
|
||||
node := gpuNode("n1", 1)
|
||||
node.Weights = []discovery.LocalWeights{{Path: gone, SizeBytes: 1 << 30}, {Path: set, SizeBytes: 20 << 30},
|
||||
{Path: kept, SizeBytes: 4 << 30}}
|
||||
ctl.Inventory().Upsert(node)
|
||||
a := New(ctl)
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
a.collectWeights(rec, httptest.NewRequest(http.MethodPost, "/api/v1/weights/collect", strings.NewReader(`{"node":"n1"}`)))
|
||||
var got []CollectResult
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
|
||||
t.Fatalf("%v: %s", err, rec.Body.String())
|
||||
}
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("results %+v", got)
|
||||
}
|
||||
r := got[0]
|
||||
if r.Files != 2 || r.Bytes != 21<<30 || r.Left != 1 || r.LeftBytes != 4<<30 {
|
||||
t.Errorf("result %+v: want 2 files and 21 GB freed, 1 file of 4 GB left", r)
|
||||
}
|
||||
}
|
||||
@@ -36,7 +36,7 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
|
||||
n.Host = discovery.Host{
|
||||
OSID: h.GetOsId(), OSVersion: h.GetOsVersion(), OSCodename: h.GetOsCodename(),
|
||||
Arch: h.GetArch(), CPUFlags: h.GetCpuFlags(), NVIDIADriver: h.GetNvidiaDriver(),
|
||||
MemBandwidthGBs: h.GetMemBandwidthGbs(),
|
||||
MemBandwidthGBs: h.GetMemBandwidthGbs(), MemBandwidthCoreGBs: h.GetMemBandwidthCoreGbs(),
|
||||
}
|
||||
}
|
||||
for _, w := range r.GetWeights() {
|
||||
|
||||
@@ -84,6 +84,8 @@ type Service struct {
|
||||
agentSettings func(node string) string
|
||||
// train is told how training jobs go; see SetTrain.
|
||||
train TrainHandler
|
||||
// slurm is told of the nodes' part in Slurm; see SetSlurm.
|
||||
slurm SlurmHandler
|
||||
|
||||
// logWaits are the log requests awaiting a reply, by request id.
|
||||
logMu sync.Mutex
|
||||
@@ -308,6 +310,10 @@ func (s *Service) Connect(
|
||||
return err
|
||||
}
|
||||
}
|
||||
// Its part in Slurm, sent by the Slurm service's next pass.
|
||||
if h := s.slurmHandler(); h != nil {
|
||||
h.OnRegister(nodeID)
|
||||
}
|
||||
|
||||
// Recover replica state from what this node is already running, before any
|
||||
// reconciliation pass can conclude the cluster is empty and place
|
||||
@@ -515,6 +521,11 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
|
||||
h.OnTrainStatus(nodeID, m.TrainStatus)
|
||||
}
|
||||
|
||||
case *llmgwv1.AgentMessage_SlurmStatus:
|
||||
if h := s.slurmHandler(); h != nil {
|
||||
h.OnSlurmStatus(nodeID, m.SlurmStatus)
|
||||
}
|
||||
|
||||
case *llmgwv1.AgentMessage_TrainArtifact:
|
||||
if h := s.trainHandler(); h != nil {
|
||||
h.OnTrainArtifact(nodeID, m.TrainArtifact)
|
||||
@@ -547,7 +558,7 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
|
||||
storage := hb.GetStorage()
|
||||
memMB := int(hb.GetMemoryTotalMb())
|
||||
cpus := int(hb.GetCpuCount())
|
||||
bw := hb.GetMemBandwidthGbs()
|
||||
bw, coreBW := hb.GetMemBandwidthGbs(), hb.GetMemBandwidthCoreGbs()
|
||||
if len(used) == 0 && len(foreign) == 0 && len(problems) == 0 && len(storage) == 0 &&
|
||||
!hb.GetWeightsChanged() && memMB == 0 && cpus == 0 && !hb.GetEnginesChanged() &&
|
||||
!hb.GetProvisionChanged() && hb.GetLoad() == nil && len(hb.GetIbPorts()) == 0 &&
|
||||
@@ -626,6 +637,9 @@ func (s *Service) applyHeartbeat(nodeID string, hb *llmgwv1.Heartbeat) {
|
||||
if bw > 0 {
|
||||
n.Host.MemBandwidthGBs = bw
|
||||
}
|
||||
if coreBW > 0 {
|
||||
n.Host.MemBandwidthCoreGBs = coreBW
|
||||
}
|
||||
if cpus > 0 && cpus != n.CPUCount {
|
||||
if n.CPUCount > 0 {
|
||||
s.log.Info("node CPUs changed", "node", nodeID, "from", n.CPUCount, "to", cpus)
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
package agentsvc
|
||||
|
||||
import (
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
)
|
||||
|
||||
// SlurmHandler is told of the nodes' part in Slurm (BACKLOG P).
|
||||
type SlurmHandler interface {
|
||||
OnSlurmStatus(node string, st *llmgwv1.SlurmStatus)
|
||||
// OnRegister is a node's new stream, which is told again what it is.
|
||||
OnRegister(node string)
|
||||
}
|
||||
|
||||
// SetSlurm gives the service what Slurm's messages go to.
|
||||
func (s *Service) SetSlurm(h SlurmHandler) {
|
||||
s.mu.Lock()
|
||||
s.slurm = h
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func (s *Service) slurmHandler() SlurmHandler {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
return s.slurm
|
||||
}
|
||||
@@ -267,6 +267,12 @@ type Profile struct {
|
||||
Weight float64 `json:"weight,omitempty" koanf:"weight"`
|
||||
// Priority orders profiles during placement; highest that fits wins.
|
||||
Priority int `json:"priority,omitempty" koanf:"priority"`
|
||||
// Replicas, set on any of a model's profiles, is how many of its
|
||||
// replicas run on this one: seven on cards and four on CPUs, all
|
||||
// answering under the model's name, rather than each replica on the
|
||||
// best profile that fits. The model's scaling is then the sum, min and
|
||||
// max alike; a profile left at zero runs none.
|
||||
Replicas int `json:"replicas,omitempty" koanf:"replicas"`
|
||||
// StartupTimeout bounds model loading. A 70B model across two nodes takes
|
||||
// tens of minutes, so this is per profile rather than global.
|
||||
StartupTimeout time.Duration `json:"startup_timeout,omitempty" koanf:"startup_timeout"`
|
||||
@@ -1024,10 +1030,57 @@ func (m Model) normalize() Model {
|
||||
return m
|
||||
}
|
||||
|
||||
// Split says whether the model's replicas are counted per profile: any of
|
||||
// its profiles sets replicas.
|
||||
func (m Model) Split() bool {
|
||||
return slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Replicas > 0 })
|
||||
}
|
||||
|
||||
// SplitTotal is the replicas its profiles ask for together.
|
||||
func (m Model) SplitTotal() int {
|
||||
n := 0
|
||||
for _, p := range m.Profiles {
|
||||
n += max(p.Replicas, 0)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// ProfileBase is the catalog profile a placement profile stands for: one
|
||||
// named by the card it needs is placed as name@class.
|
||||
func ProfileBase(name string) string {
|
||||
base, _, _ := strings.Cut(name, "@")
|
||||
return base
|
||||
}
|
||||
|
||||
// validateSplit refuses replicas per profile that the model's scaling does
|
||||
// not add up to.
|
||||
func (m Model) validateSplit() error {
|
||||
for _, p := range m.Profiles {
|
||||
if p.Replicas < 0 {
|
||||
return fmt.Errorf("model %q profile %q: replicas %d is negative", m.Name, p.Name, p.Replicas)
|
||||
}
|
||||
}
|
||||
if !m.Split() || m.Stopped != nil {
|
||||
return nil
|
||||
}
|
||||
n := m.SplitTotal()
|
||||
if m.Scaling.Min != n || m.Scaling.Max != n {
|
||||
return fmt.Errorf("model %q: its profiles' replicas add up to %d, so its scaling is min %d, max %d, not %d and %d",
|
||||
m.Name, n, n, n, m.Scaling.Min, m.Scaling.Max)
|
||||
}
|
||||
if len(m.Scaling.Schedule) > 0 {
|
||||
return fmt.Errorf("model %q: replicas per profile and a schedule are one or the other", m.Name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m Model) validate() error {
|
||||
if m.Name == "" {
|
||||
return fmt.Errorf("a model has no name")
|
||||
}
|
||||
if err := m.validateSplit(); err != nil {
|
||||
return err
|
||||
}
|
||||
if m.Source == "" {
|
||||
return fmt.Errorf("model %q: source is required", m.Name)
|
||||
}
|
||||
@@ -1510,6 +1563,9 @@ func (m Model) Simulated() bool {
|
||||
// happened, and the answer to "why is this one on the wrong node" was in
|
||||
// somebody's memory of what they edited an hour ago.
|
||||
func (p Profile) Fingerprint() string {
|
||||
// How many replicas run on it is not how each runs: changing the count
|
||||
// must not mark the running ones stale.
|
||||
p.Replicas = 0
|
||||
b, err := json.Marshal(p)
|
||||
if err != nil {
|
||||
// A profile that will not marshal cannot be compared, and saying so by
|
||||
|
||||
@@ -296,7 +296,7 @@ func (s *Server) Start(req Request) (Job, error) {
|
||||
}
|
||||
req.sized = req.CtxPool > 0 || req.ContextLength > 0
|
||||
if req.CtxPool <= 0 {
|
||||
req.CtxPool = 16384
|
||||
req.CtxPool = DefaultCtxPool
|
||||
}
|
||||
if req.ContextLength <= 0 {
|
||||
req.ContextLength = 4096
|
||||
@@ -572,6 +572,14 @@ func (s *Server) run(j *Job) {
|
||||
// On one card, the weights and the context's cache have to fit it: a
|
||||
// 70B with a 65536-token pool on a 24 GB card loaded its weights and then
|
||||
// failed to make its context, every time, as did an 8B with 131072.
|
||||
// Asked for the CPU, and slow there: said, not refused — the request
|
||||
// named it — so the job's log says why its replies take minutes.
|
||||
if len(layout) > 0 && layout[0].CPU && layout[0].TokensPerSecond > 0 &&
|
||||
layout[0].TokensPerSecond < modelplan.CPUSlowTPS {
|
||||
tps := layout[0].TokensPerSecond
|
||||
s.step(j, "check", Running, fmt.Sprintf("on the CPU about %.1f tokens a second for one reply: "+
|
||||
"a reply of 500 tokens takes about %.0f seconds; a card would be many times faster", tps, 500/tps))
|
||||
}
|
||||
if len(layout) > 0 && layout[0].Class != "" && !layout[0].CPU {
|
||||
fitted, why, err := fitPoolToCard(req, header, set.Bytes, layout[0].Class, max(layout[0].GPUsPerNode, 1))
|
||||
if err != nil {
|
||||
@@ -723,6 +731,10 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
|
||||
return out
|
||||
}
|
||||
|
||||
// DefaultCtxPool is llama.cpp's pool when a request names none: a context
|
||||
// asked for longer than it needs a pool of its own.
|
||||
const DefaultCtxPool = 16384
|
||||
|
||||
// llamaArgs size llama.cpp's pool and its slots together. The pool is shared
|
||||
// by the slots, so slots of the context the catalog promises are as many as
|
||||
// the pool holds: the engine's default of four turned a 32768 pool into
|
||||
|
||||
@@ -43,6 +43,9 @@ type Plan struct {
|
||||
License string `json:"license,omitempty"`
|
||||
Params int64 `json:"params,omitempty"`
|
||||
Context int `json:"context"`
|
||||
// Trained is the context the model was trained for, the most a request
|
||||
// may use; 0 when neither its header nor its config says.
|
||||
Trained int `json:"trained,omitempty"`
|
||||
// KVBytesPerToken is one token of context, the whole model: what an
|
||||
// option needs for another context is worked out from it. 0 is unknown.
|
||||
KVBytesPerToken int64 `json:"kv_bytes_per_token,omitempty"`
|
||||
@@ -149,7 +152,9 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
|
||||
p.Servable, p.Reason = false, v.Reason
|
||||
}
|
||||
shape.KVBytesPerToken = v.Header.KVBytesPerToken()
|
||||
p.Trained = int(v.Header.ContextLength) //nolint:gosec // a header's context length
|
||||
shape.KVHeads = int(v.Header.HeadCountKV) //nolint:gosec // a header's head count
|
||||
shape.ActiveShare = v.Header.ActiveShare()
|
||||
if p.Arch == "" {
|
||||
p.Arch = v.Header.Architecture
|
||||
}
|
||||
@@ -166,6 +171,7 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
|
||||
m, err := modelplan.FetchModel(ctx, repo.ID)
|
||||
if err == nil {
|
||||
shape.KVBytesPerToken = m.KVBytesPerToken(modelplan.BF16)
|
||||
p.Trained = m.MaxContext
|
||||
shape.KVHeads = m.KVHeads
|
||||
shape.Experts = m.Experts
|
||||
if p.Params == 0 {
|
||||
@@ -181,6 +187,9 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
|
||||
p.Note = "the model's shape was not readable, so the context's memory is not counted"
|
||||
}
|
||||
shape.Params = p.Params
|
||||
if shape.ActiveShare == 0 || shape.ActiveShare == 1 {
|
||||
shape.ActiveShare = modelplan.ActiveShare(repo.ID, p.Params)
|
||||
}
|
||||
p.KVBytesPerToken, p.Experts = shape.KVBytesPerToken, shape.Experts
|
||||
p.Options = modelplan.Options(arts, shape, s.deps.Pool(), p.Context)
|
||||
if !p.Servable {
|
||||
|
||||
@@ -22,6 +22,7 @@ import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@@ -249,6 +250,13 @@ func (s *Service) Remove(ctx context.Context, name string) error {
|
||||
// Scale changes how many replicas a model added at runtime runs.
|
||||
func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by string) (catalog.Model, error) {
|
||||
final, err := s.update(ctx, name, by, "change its scaling", func(m *catalog.Model) error {
|
||||
// Replicas counted per profile stay so only for the same total, one
|
||||
// way: any other scaling is the model's again, on whatever fits.
|
||||
if m.Split() && (sc.Min != sc.Max || sc.Min != m.SplitTotal()) {
|
||||
for i := range m.Profiles {
|
||||
m.Profiles[i].Replicas = 0
|
||||
}
|
||||
}
|
||||
m.Scaling = sc
|
||||
// A number set is what it runs now: a stopped model is started by it.
|
||||
m.Stopped = nil
|
||||
@@ -261,6 +269,98 @@ func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by
|
||||
return final, err
|
||||
}
|
||||
|
||||
// CPUProfile is a profile on CPUs to add to a llama.cpp model, for replicas
|
||||
// counted per profile: what a replica of it reserves.
|
||||
type CPUProfile struct {
|
||||
Name string `json:"name,omitempty"`
|
||||
MemoryGB int `json:"memory_gb"`
|
||||
CPUCores int `json:"cpu_cores,omitempty"`
|
||||
}
|
||||
|
||||
// SetProfileReplicas sets how many of a model's replicas run on each of its
|
||||
// profiles: seven on cards and four on CPUs, under the one name. A profile
|
||||
// not named keeps its count. cpu, for a llama.cpp model with no profile on
|
||||
// CPUs, adds one — its card profile without the card, preferred below every
|
||||
// other, so the router fills the cards first and the CPUs after. Running
|
||||
// replicas carry on; those over a count stop once the ones short have
|
||||
// started.
|
||||
func (s *Service) SetProfileReplicas(ctx context.Context, name string, counts map[string]int, cpu *CPUProfile,
|
||||
by string,
|
||||
) (catalog.Model, error) {
|
||||
final, err := s.update(ctx, name, by, "count its replicas per profile", func(m *catalog.Model) error {
|
||||
if cpu != nil {
|
||||
if cpu.Name == "" {
|
||||
cpu.Name = "cpu"
|
||||
}
|
||||
if !slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Name == cpu.Name }) {
|
||||
p, err := cpuProfileOf(*m, *cpu)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
m.Profiles = append(m.Profiles, p)
|
||||
}
|
||||
}
|
||||
for prof, n := range counts {
|
||||
i := slices.IndexFunc(m.Profiles, func(p catalog.Profile) bool { return p.Name == prof })
|
||||
if i < 0 {
|
||||
return fmt.Errorf("model %q has no profile %q", name, prof)
|
||||
}
|
||||
if n < 0 {
|
||||
return fmt.Errorf("profile %q: %d replicas", prof, n)
|
||||
}
|
||||
m.Profiles[i].Replicas = n
|
||||
}
|
||||
n := m.SplitTotal()
|
||||
if n == 0 {
|
||||
return fmt.Errorf("none of %q's profiles would run a replica; llmgw ctl model stop %s keeps it without", name, name)
|
||||
}
|
||||
sc := catalog.Scaling{Min: n, Max: n, IdleTimeout: m.Scaling.IdleTimeout}
|
||||
if m.Stopped != nil {
|
||||
sc.IdleTimeout = m.Stopped.IdleTimeout
|
||||
m.Stopped = &sc
|
||||
} else {
|
||||
m.Scaling = sc
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err == nil {
|
||||
s.log.Info("replicas counted per profile", "model", name, "counts", counts, "by", by)
|
||||
}
|
||||
return final, err
|
||||
}
|
||||
|
||||
// cpuProfileOf is a model's llama.cpp profile on CPUs: the same engine and
|
||||
// arguments, without the card or what spreads layers over cards.
|
||||
func cpuProfileOf(m catalog.Model, cpu CPUProfile) (catalog.Profile, error) {
|
||||
i := slices.IndexFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" })
|
||||
if i < 0 {
|
||||
return catalog.Profile{}, fmt.Errorf("model %q is not served by llama.cpp; only a GGUF model runs on CPUs", m.Name)
|
||||
}
|
||||
if cpu.MemoryGB <= 0 {
|
||||
return catalog.Profile{}, errors.New("a profile on CPUs needs the memory a replica reserves")
|
||||
}
|
||||
p := m.Profiles[i]
|
||||
lowest := p.Priority
|
||||
for _, o := range m.Profiles {
|
||||
lowest = min(lowest, o.Priority)
|
||||
}
|
||||
out := catalog.Profile{Name: cpu.Name, Engine: p.Engine, EngineVersion: p.EngineVersion, Source: p.Source,
|
||||
Env: p.Env, NodeSelector: p.NodeSelector, StartupTimeout: p.StartupTimeout, Weight: p.Weight,
|
||||
Priority: lowest - 10, MinMemoryGB: cpu.MemoryGB, CPUCores: cpu.CPUCores}
|
||||
for j := 0; j < len(p.Args); j++ {
|
||||
switch a := p.Args[j]; {
|
||||
case slices.Contains([]string{"--n-gpu-layers", "-ngl", "--split-mode", "-sm", "--tensor-split", "-ts",
|
||||
"--main-gpu", "-mg"}, a):
|
||||
j++ // and its value
|
||||
case strings.HasPrefix(a, "--n-gpu-layers=") || strings.HasPrefix(a, "--split-mode=") ||
|
||||
strings.HasPrefix(a, "--tensor-split=") || strings.HasPrefix(a, "--main-gpu="):
|
||||
default:
|
||||
out.Args = append(out.Args, a)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Stop takes a model out of service and keeps it: none of its replicas run,
|
||||
// and what it was scaled to is kept for Start.
|
||||
func (s *Service) Stop(ctx context.Context, name, by string) (catalog.Model, error) {
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -458,3 +459,44 @@ func TestAStrataModelReservesTheMemoryItsExpertsTake(t *testing.T) {
|
||||
t.Errorf("an unknown size: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A GGUF model on a card gets a profile on CPUs, and seven replicas on cards
|
||||
// beside four on CPUs; a scaling of another total puts it back on whatever
|
||||
// fits.
|
||||
func TestReplicasCountedPerProfile(t *testing.T) {
|
||||
s, _, _ := newService(t)
|
||||
m := catalog.Model{Name: "g", Source: "/models/g.gguf", Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "gpu", Engine: "llamacpp", GPUClass: "l40s-48g", GPUsPerNode: 2,
|
||||
Priority: 100, Args: []string{"--ctx-size", "16384", "--n-gpu-layers", "999", "--split-mode=layer"}}}}
|
||||
if err := s.Add(t.Context(), m, "ilker"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.SetProfileReplicas(t.Context(), "g", map[string]int{"gpu": 7, "cpu": 4}, nil, "ilker"); err == nil {
|
||||
t.Fatal("a count for a profile it does not have was taken")
|
||||
}
|
||||
got, err := s.SetProfileReplicas(t.Context(), "g", map[string]int{"gpu": 7, "cpu": 4},
|
||||
&CPUProfile{MemoryGB: 24, CPUCores: 8}, "ilker")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got.Profiles) != 2 || got.Scaling.Min != 11 || got.Scaling.Max != 11 {
|
||||
t.Fatalf("profiles %+v, scaling %+v", got.Profiles, got.Scaling)
|
||||
}
|
||||
cpu := got.Profiles[1]
|
||||
if cpu.Name != "cpu" || cpu.GPUClass != "" || cpu.GPUsPerNode != 0 || cpu.Replicas != 4 || cpu.Priority >= 100 ||
|
||||
cpu.MinMemoryGB != 24 || cpu.CPUCores != 8 || !slices.Equal(cpu.Args, []string{"--ctx-size", "16384"}) {
|
||||
t.Errorf("the profile on CPUs: %+v", cpu)
|
||||
}
|
||||
// Stopped and started, it comes back split.
|
||||
_, _ = s.Stop(t.Context(), "g", "ilker")
|
||||
if got, _ = s.Start(t.Context(), "g", "ilker"); got.Scaling.Min != 11 || got.Profiles[0].Replicas != 7 {
|
||||
t.Errorf("started: %+v", got)
|
||||
}
|
||||
// The same total keeps the split; another one ends it.
|
||||
if got, _ = s.Scale(t.Context(), "g", catalog.Scaling{Min: 11, Max: 11}, "ilker"); !got.Split() {
|
||||
t.Error("the same total ended the split")
|
||||
}
|
||||
if got, _ = s.Scale(t.Context(), "g", catalog.Scaling{Min: 3, Max: 3}, "ilker"); got.Split() {
|
||||
t.Error("another total kept the split")
|
||||
}
|
||||
}
|
||||
@@ -154,6 +154,9 @@ func (r Result) Explain() string {
|
||||
// Scheduler places replicas onto an inventory.
|
||||
type Scheduler struct {
|
||||
inv *inventory.Store
|
||||
// shun is where a replica of the model being placed failed to start;
|
||||
// set on a copy by Away, never on the scheduler the reconciler shares.
|
||||
shun []Spot
|
||||
// now is the clock power readings are judged fresh by; nil is the wall
|
||||
// clock.
|
||||
now func() time.Time
|
||||
@@ -162,6 +165,41 @@ type Scheduler struct {
|
||||
// New builds a scheduler over an inventory.
|
||||
func New(inv *inventory.Store) *Scheduler { return &Scheduler{inv: inv} }
|
||||
|
||||
// Spot is where a replica ran: a node, and the cards it held there (none
|
||||
// for one on the node's CPU).
|
||||
type Spot struct {
|
||||
Node string
|
||||
GPUs []int
|
||||
}
|
||||
|
||||
// Away is this scheduler placing nowhere in spots: not those cards, nor a
|
||||
// CPU replica on those nodes. A copy, so the one shared is not changed.
|
||||
//
|
||||
// For a model that failed to start somewhere: the same inputs pick the same
|
||||
// node and the same card every time, so without it every retry went back to
|
||||
// where the last one had failed until the model was parked, with another
|
||||
// card or node free all along.
|
||||
func (s *Scheduler) Away(spots []Spot) *Scheduler {
|
||||
c := *s
|
||||
c.shun = spots
|
||||
return &c
|
||||
}
|
||||
|
||||
// shunned is the cards of node n a model failed on, and whether it failed
|
||||
// on the node's CPU.
|
||||
func (s *Scheduler) shunned(node string) (gpus []int, cpu bool) {
|
||||
for _, sp := range s.shun {
|
||||
if sp.Node != node {
|
||||
continue
|
||||
}
|
||||
if len(sp.GPUs) == 0 {
|
||||
cpu = true
|
||||
}
|
||||
gpus = append(gpus, sp.GPUs...)
|
||||
}
|
||||
return gpus, cpu
|
||||
}
|
||||
|
||||
// Place picks the first profile, in priority order, that fits.
|
||||
//
|
||||
// avoid lists nodes already hosting this model, so replicas of one model spread
|
||||
@@ -686,7 +724,12 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
|
||||
}
|
||||
}
|
||||
|
||||
badGPUs, badCPU := s.shunned(n.ID)
|
||||
if p.CPUOnly() {
|
||||
if badCPU {
|
||||
reject("a replica of it failed to start here; trying elsewhere first")
|
||||
continue
|
||||
}
|
||||
// Every ready node can run it; there is no card to be free.
|
||||
c := candidate{node: n}
|
||||
if why := budget.check(n, s.shardWatts(p, c)); why != "" {
|
||||
@@ -703,8 +746,16 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
|
||||
if g.Class() != p.GPUClass || !slices.Contains(freeIdx, g.Index) {
|
||||
continue
|
||||
}
|
||||
if slices.Contains(badGPUs, g.Index) {
|
||||
continue
|
||||
}
|
||||
free = append(free, g)
|
||||
}
|
||||
if len(free) < p.GPUsPerNode && len(badGPUs) > 0 {
|
||||
reject(fmt.Sprintf("has %d free %s GPUs besides the ones a replica of it failed to start on, needs %d",
|
||||
len(free), p.GPUClass, p.GPUsPerNode))
|
||||
continue
|
||||
}
|
||||
if len(free) < p.GPUsPerNode {
|
||||
reject(fmt.Sprintf("has %d free %s GPUs, needs %d", len(free), p.GPUClass, p.GPUsPerNode))
|
||||
continue
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
// failAndRetry fails the last replica started, and returns where the next
|
||||
// one is started once the backoff is over.
|
||||
func failAndRetry(t *testing.T, h *harness) ShardCommand {
|
||||
t.Helper()
|
||||
started := h.disp.startedShards()
|
||||
if len(started) == 0 {
|
||||
t.Fatal("nothing was started")
|
||||
}
|
||||
s := started[len(started)-1]
|
||||
h.r.OnShardStatus(s.ReplicaID, s.ShardID, supervisor.Failed, "the engine exited")
|
||||
h.r.Reconcile(t.Context()) // reaps it
|
||||
h.disp.reset()
|
||||
h.advance(DefaultConfig().MaxBackoff)
|
||||
h.r.Reconcile(t.Context())
|
||||
again := h.disp.startedShards()
|
||||
if len(again) == 0 {
|
||||
t.Fatal("it was not tried again")
|
||||
}
|
||||
return again[0]
|
||||
}
|
||||
|
||||
func oneCard() catalog.Model {
|
||||
return testModel(1, catalog.Profile{Name: "one", GPUClass: "h100-80g", GPUsPerNode: 1})
|
||||
}
|
||||
|
||||
func TestAReplicaThatFailedIsTriedOnAnotherNode(t *testing.T) {
|
||||
// The same inputs give the same node every time: an on-demand 3D model
|
||||
// that failed to load was started on that node again, and again, until
|
||||
// it was parked, with another node free all along.
|
||||
h := newHarness(t, oneCard(), gpuNode("1", 1), gpuNode("2", 1))
|
||||
h.r.Reconcile(t.Context())
|
||||
first := h.disp.startedShards()[0].NodeID
|
||||
|
||||
if next := failAndRetry(t, h); next.NodeID == first {
|
||||
t.Errorf("tried on node %s again, where it had just failed", first)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReplicaThatFailedIsTriedOnAnotherCardOfTheNode(t *testing.T) {
|
||||
h := newHarness(t, oneCard(), gpuNode("1", 2))
|
||||
h.r.Reconcile(t.Context())
|
||||
first := h.disp.startedShards()[0].Spec.GPUIndices
|
||||
|
||||
next := failAndRetry(t, h)
|
||||
if len(next.Spec.GPUIndices) != 1 || len(first) != 1 || next.Spec.GPUIndices[0] == first[0] {
|
||||
t.Errorf("tried on card %v again, where it had just failed", next.Spec.GPUIndices)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithNowhereElseItIsTriedWhereItFailed(t *testing.T) {
|
||||
// A preference, not a rule: a model that has one place is not left
|
||||
// unplaced because it failed there once.
|
||||
h := newHarness(t, oneCard(), gpuNode("1", 1))
|
||||
h.r.Reconcile(t.Context())
|
||||
if next := failAndRetry(t, h); next.NodeID != "1" {
|
||||
t.Errorf("started on %q", next.NodeID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComingUpForgetsWhereItFailed(t *testing.T) {
|
||||
h := newHarness(t, oneCard(), gpuNode("1", 1), gpuNode("2", 1))
|
||||
h.r.Reconcile(t.Context())
|
||||
next := failAndRetry(t, h)
|
||||
h.r.OnShardStatus(next.ReplicaID, next.ShardID, supervisor.Ready, "")
|
||||
h.r.mu.Lock()
|
||||
left := len(h.r.failedAt["llama"])
|
||||
h.r.mu.Unlock()
|
||||
if left != 0 {
|
||||
t.Errorf("a model that came up still remembers %d failures' places", left)
|
||||
}
|
||||
}
|
||||
@@ -61,6 +61,7 @@ func (r *Reconciler) Retry(model string) bool {
|
||||
delete(r.parked, model)
|
||||
r.attempts[model] = 0
|
||||
delete(r.nextAttempt, model)
|
||||
delete(r.failedAt, model)
|
||||
r.mu.Unlock()
|
||||
r.Notify()
|
||||
return was
|
||||
|
||||
@@ -7,6 +7,8 @@ import (
|
||||
"log/slog"
|
||||
"math/rand/v2"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
@@ -251,6 +253,9 @@ type Reconciler struct {
|
||||
nextAttempt map[string]time.Time
|
||||
// attempts counts consecutive failures per model, for the backoff.
|
||||
attempts map[string]int
|
||||
// failedAt is where a model's replicas failed to start since it last
|
||||
// came up, most recent last: the next one is placed elsewhere if it can.
|
||||
failedAt map[string][]placement.Spot
|
||||
// lastExplain keeps the most recent placement failure per model, so
|
||||
// `ctl explain-placement` can answer without re-running placement.
|
||||
lastExplain map[string]placement.Result
|
||||
@@ -334,6 +339,7 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
|
||||
replicas: map[string]*Replica{},
|
||||
nextAttempt: map[string]time.Time{},
|
||||
attempts: map[string]int{},
|
||||
failedAt: map[string][]placement.Spot{},
|
||||
reservedPorts: map[int]struct{}{},
|
||||
wanted: map[string]time.Time{},
|
||||
coldStart: map[string]time.Duration{},
|
||||
@@ -726,6 +732,7 @@ func (r *Reconciler) reapFailed(ctx context.Context) {
|
||||
// replica is about to disappear, and the backoff has to survive it.
|
||||
r.attempts[rep.Model]++
|
||||
r.nextAttempt[rep.Model] = r.now().Add(r.backoff(r.attempts[rep.Model]))
|
||||
r.noteFailedAt(rep)
|
||||
if n := r.attempts[rep.Model]; n >= r.cfg.ParkAfter {
|
||||
if _, already := r.parked[rep.Model]; !already {
|
||||
m, _ := r.cat.Load().Resolve(rep.Model)
|
||||
@@ -875,6 +882,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
delete(r.parked, m.Name)
|
||||
r.attempts[m.Name] = 0
|
||||
delete(r.nextAttempt, m.Name)
|
||||
delete(r.failedAt, m.Name)
|
||||
isParked, blockedUntil = false, time.Time{}
|
||||
r.log.Info("model unparked: its definition changed", "model", m.Name)
|
||||
}
|
||||
@@ -886,14 +894,25 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
}
|
||||
|
||||
desired := r.desiredFor(m)
|
||||
// Replicas counted per profile: one short on its profile is started even
|
||||
// with the total right — before one of a profile with too many is
|
||||
// stopped, which the total going over then does — so changing seven on
|
||||
// cards and four on CPUs to nine and two never runs fewer than eleven.
|
||||
var deficit map[string]bool
|
||||
var surplus []string
|
||||
split := m.Split() && desired == m.SplitTotal()
|
||||
if split {
|
||||
deficit, surplus = r.splitGaps(m)
|
||||
}
|
||||
needUp := len(liveIDs) < desired || split && len(deficit) > 0 && len(liveIDs) == desired
|
||||
outcome := notTried
|
||||
switch {
|
||||
case len(liveIDs) < desired && isParked:
|
||||
case needUp && isParked:
|
||||
// Not placed. Its failure is on every status the operator reads.
|
||||
case len(liveIDs) < desired && r.isHeld(m.Name):
|
||||
case needUp && r.isHeld(m.Name):
|
||||
// It gave up room to a model of a higher class, which is placed
|
||||
// first; taking it back now would undo the preemption.
|
||||
case len(liveIDs) < desired:
|
||||
case needUp:
|
||||
if r.now().Before(blockedUntil) {
|
||||
// Still in backoff. Rebuilding immediately would hot-loop on a
|
||||
// model that cannot start — a missing weights path, say — and bury
|
||||
@@ -912,7 +931,11 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
attribute.String("llmgw.model", m.Name),
|
||||
attribute.Int("llmgw.replicas", len(liveIDs)),
|
||||
attribute.Int("llmgw.desired", desired))
|
||||
r.scaleDown(sctx, liveIDs[desired:], "more replicas than desired")
|
||||
excess := liveIDs[desired:]
|
||||
if split {
|
||||
excess = pickSurplus(surplus, liveIDs, len(liveIDs)-desired)
|
||||
}
|
||||
r.scaleDown(sctx, excess, "more replicas than desired")
|
||||
span.End()
|
||||
}
|
||||
|
||||
@@ -976,7 +999,40 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
|
||||
attribute.String("llmgw.model", m.Name),
|
||||
attribute.StringSlice("llmgw.avoid", avoid))
|
||||
profiles := r.orderedProfiles(m)
|
||||
res := r.sched.Place(profiles, avoid)
|
||||
if m.Split() {
|
||||
// Only the profiles short of their replicas.
|
||||
deficit, _ := r.splitGaps(m)
|
||||
profiles = slices.DeleteFunc(profiles, func(p placement.Profile) bool {
|
||||
return !deficit[catalog.ProfileBase(p.Name)]
|
||||
})
|
||||
if len(profiles) == 0 {
|
||||
pspan.End()
|
||||
return notTried
|
||||
}
|
||||
}
|
||||
// Away from where it last failed to start, when there is anywhere else:
|
||||
// the same inputs give the same node and card, and an on-demand model
|
||||
// that failed there was otherwise started there again until it was
|
||||
// parked. Only a preference too: nowhere else, and it is tried there.
|
||||
r.mu.Lock()
|
||||
failed := slices.Clone(r.failedAt[m.Name])
|
||||
r.mu.Unlock()
|
||||
var res placement.Result
|
||||
if len(failed) > 0 {
|
||||
away := r.sched.Away(failed)
|
||||
res = away.Place(profiles, avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
res = away.Place(profiles, nil)
|
||||
}
|
||||
pspan.SetAttributes(attribute.Bool("llmgw.away_from_failure", res.Placement != nil))
|
||||
if res.Placement != nil {
|
||||
r.log.Info("placing a model away from where it failed to start", "model", m.Name,
|
||||
"failed_on", spotsText(failed), "now_on", spotsText(spotsOf(res.Placement)))
|
||||
}
|
||||
}
|
||||
if res.Placement == nil {
|
||||
res = r.sched.Place(profiles, avoid)
|
||||
}
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
// Spreading across failure domains is a preference, not a rule. If it
|
||||
// is the only thing standing in the way, take the placement anyway:
|
||||
@@ -1028,6 +1084,51 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
|
||||
return placed
|
||||
}
|
||||
|
||||
// maxFailedSpots is how many places a model's failures are remembered at.
|
||||
const maxFailedSpots = 8
|
||||
|
||||
// noteFailedAt remembers where a failed replica ran. Called under r.mu.
|
||||
func (r *Reconciler) noteFailedAt(rep *Replica) {
|
||||
spots := r.failedAt[rep.Model]
|
||||
for _, sh := range rep.Shards {
|
||||
sp := placement.Spot{Node: sh.NodeID, GPUs: slices.Clone(sh.GPUIndices)}
|
||||
spots = slices.DeleteFunc(spots, func(o placement.Spot) bool {
|
||||
return o.Node == sp.Node && slices.Equal(o.GPUs, sp.GPUs)
|
||||
})
|
||||
spots = append(spots, sp)
|
||||
}
|
||||
if len(spots) > maxFailedSpots {
|
||||
spots = spots[len(spots)-maxFailedSpots:]
|
||||
}
|
||||
r.failedAt[rep.Model] = spots
|
||||
}
|
||||
|
||||
// spotsOf is where a placement puts its shards.
|
||||
func spotsOf(pl *placement.Placement) []placement.Spot {
|
||||
var out []placement.Spot
|
||||
for _, n := range pl.Nodes {
|
||||
out = append(out, placement.Spot{Node: n.NodeID, GPUs: n.GPUIndices})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// spotsText is spots as a log reads them: "node01 gpu 0,1; node02 cpu".
|
||||
func spotsText(spots []placement.Spot) string {
|
||||
parts := make([]string, 0, len(spots))
|
||||
for _, sp := range spots {
|
||||
if len(sp.GPUs) == 0 {
|
||||
parts = append(parts, sp.Node+" cpu")
|
||||
continue
|
||||
}
|
||||
gs := make([]string, len(sp.GPUs))
|
||||
for i, g := range sp.GPUs {
|
||||
gs[i] = strconv.Itoa(g)
|
||||
}
|
||||
parts = append(parts, sp.Node+" gpu "+strings.Join(gs, ","))
|
||||
}
|
||||
return strings.Join(parts, "; ")
|
||||
}
|
||||
|
||||
// scaleOutcome is what one scaleUp did.
|
||||
type scaleOutcome int
|
||||
|
||||
@@ -1407,6 +1508,7 @@ func (r *Reconciler) markShard(replicaID, shardID string, st supervisor.State, e
|
||||
// counting from scratch rather than from an old streak.
|
||||
r.attempts[rep.Model] = 0
|
||||
delete(r.nextAttempt, rep.Model)
|
||||
delete(r.failedAt, rep.Model)
|
||||
if !rep.CreatedAt.IsZero() {
|
||||
r.coldStart[rep.Model] = r.now().Sub(rep.CreatedAt)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"slices"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
// splitGaps is, for a model whose replicas are counted per profile, the
|
||||
// profiles short of theirs and the replicas over: of a profile with more
|
||||
// than it asks, or of one it no longer has. Counted as reconcileModel
|
||||
// counts — a replica being retired is not one it has — so a replica
|
||||
// restarted is replaced on its own profile.
|
||||
func (r *Reconciler) splitGaps(m catalog.Model) (deficit map[string]bool, surplus []string) {
|
||||
live := map[string][]string{}
|
||||
r.mu.Lock()
|
||||
for _, rep := range r.replicas {
|
||||
if rep.Model == m.Name && rep.State.Counts() && rep.RetiringAt.IsZero() {
|
||||
b := catalog.ProfileBase(rep.Profile)
|
||||
live[b] = append(live[b], rep.ID)
|
||||
}
|
||||
}
|
||||
r.mu.Unlock()
|
||||
deficit = map[string]bool{}
|
||||
for _, p := range m.Profiles {
|
||||
ids := live[p.Name]
|
||||
delete(live, p.Name)
|
||||
slices.Sort(ids)
|
||||
switch {
|
||||
case len(ids) < p.Replicas:
|
||||
deficit[p.Name] = true
|
||||
case len(ids) > p.Replicas:
|
||||
surplus = append(surplus, ids[p.Replicas:]...)
|
||||
}
|
||||
}
|
||||
for _, ids := range live {
|
||||
surplus = append(surplus, ids...)
|
||||
}
|
||||
slices.Sort(surplus)
|
||||
return deficit, surplus
|
||||
}
|
||||
|
||||
// pickSurplus is n of the live replicas to stop: those over their
|
||||
// profile's count first, then any.
|
||||
func pickSurplus(surplus, live []string, n int) []string {
|
||||
out := make([]string, 0, n)
|
||||
for _, id := range surplus {
|
||||
if len(out) == n {
|
||||
return out
|
||||
}
|
||||
if slices.Contains(live, id) {
|
||||
out = append(out, id)
|
||||
}
|
||||
}
|
||||
for _, id := range live {
|
||||
if len(out) == n {
|
||||
break
|
||||
}
|
||||
if !slices.Contains(out, id) {
|
||||
out = append(out, id)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
// splitModel runs gpu replicas on cards and cpu on CPUs, under one name.
|
||||
func splitModel(gpu, cpu int) catalog.Model {
|
||||
return catalog.Model{Name: "m", Source: "/models/m.gguf",
|
||||
Scaling: catalog.Scaling{Min: gpu + cpu, Max: gpu + cpu},
|
||||
Profiles: []catalog.Profile{
|
||||
{Name: "gpu", Engine: "llamacpp", GPUClass: "h100-80g", GPUsPerNode: 1, Priority: 100, Replicas: gpu},
|
||||
{Name: "cpu", Engine: "llamacpp", Priority: 10, Replicas: cpu},
|
||||
}}
|
||||
}
|
||||
|
||||
// byProfile counts the model's live replicas on each profile.
|
||||
func byProfile(h *harness) map[string]int {
|
||||
out := map[string]int{}
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.State.Counts() && rep.RetiringAt.IsZero() {
|
||||
out[catalog.ProfileBase(rep.Profile)]++
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Seven on cards and four on CPUs is what runs, not eleven on whatever fits
|
||||
// best: with cards free for all eleven, four still go to the CPUs.
|
||||
func TestReplicasRunAsManyOnEachProfileAsItAsks(t *testing.T) {
|
||||
h := newHarness(t, splitModel(7, 4), gpuNode("1", 8), gpuNode("2", 8), cpuNode("3"), cpuNode("4"))
|
||||
for range 14 {
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
h.advance(time.Second)
|
||||
}
|
||||
if got := byProfile(h); got["gpu"] != 7 || got["cpu"] != 4 {
|
||||
t.Fatalf("running %v, want 7 on cards and 4 on CPUs", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Moving replicas from one profile to the other starts the new ones before
|
||||
// stopping the old: the model never runs fewer than it asks.
|
||||
func TestChangingTheSplitStartsBeforeItStops(t *testing.T) {
|
||||
h := newHarness(t, splitModel(2, 1), gpuNode("1", 8), cpuNode("3"), cpuNode("4"))
|
||||
for range 6 {
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
h.advance(time.Second)
|
||||
}
|
||||
if got := byProfile(h); got["gpu"] != 2 || got["cpu"] != 1 {
|
||||
t.Fatalf("running %v before the change", got)
|
||||
}
|
||||
h.setModels(t, splitModel(1, 2))
|
||||
for range 10 {
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
h.advance(h.r.cfg.DrainGrace + time.Second)
|
||||
if n := replicasOf(h, "m"); n < 3 {
|
||||
t.Fatalf("only %d replicas while the split changed", n)
|
||||
}
|
||||
}
|
||||
if got := byProfile(h); got["gpu"] != 1 || got["cpu"] != 2 {
|
||||
t.Errorf("running %v, want 1 on cards and 2 on CPUs", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The scaling has to be the profiles' sum, so a count in one place cannot
|
||||
// disagree with the other.
|
||||
func TestReplicasPerProfileAndScalingAgree(t *testing.T) {
|
||||
m := splitModel(2, 1)
|
||||
m.Scaling = catalog.Scaling{Min: 1, Max: 5}
|
||||
if _, err := catalog.New([]catalog.Model{m}); err == nil {
|
||||
t.Error("profiles asking for 3 with scaling 1..5 were accepted")
|
||||
}
|
||||
stopped := splitModel(2, 1)
|
||||
was := stopped.Scaling
|
||||
stopped.Stopped, stopped.Scaling = &was, catalog.Scaling{}
|
||||
if _, err := catalog.New([]catalog.Model{stopped}); err != nil {
|
||||
t.Errorf("a stopped model with replicas per profile was refused: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
package slurmsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// fakeKV is the store's cluster state.
|
||||
type fakeKV struct {
|
||||
m map[string]string
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeKV) PutClusterState(_ context.Context, k, v string) (bool, error) {
|
||||
if f.err != nil {
|
||||
return false, f.err
|
||||
}
|
||||
f.m[k] = v
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func (f *fakeKV) ClusterState(_ context.Context, k string) (string, error) {
|
||||
if f.err != nil {
|
||||
return "", f.err
|
||||
}
|
||||
v, ok := f.m[k]
|
||||
if !ok {
|
||||
return "", store.ErrNotFound
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
// takeOver is the leader that comes next: the same settings and store, an
|
||||
// inventory with only the nodes connected to it so far.
|
||||
func takeOver(r *rig, kv *fakeKV, connected ...string) *rig {
|
||||
n := &rig{set: r.set, inv: &fakeInv{nodes: map[string]*inventory.Node{}, allocs: map[string]int{}},
|
||||
agents: &fakeAgents{conn: connected, got: map[string][]*llmgwv1.Slurm{}}}
|
||||
for _, c := range connected {
|
||||
node := *r.inv.nodes[c]
|
||||
node.State = inventory.Ready
|
||||
n.inv.nodes[c] = &node
|
||||
}
|
||||
n.svc = New(func() config.SlurmSettings { return n.set }, n.inv, n.agents,
|
||||
func(context.Context) ([]byte, error) { return []byte(strings.Repeat("k", 1024)), nil }, nil)
|
||||
n.svc.SetMemory(StoreMemory(kv))
|
||||
return n
|
||||
}
|
||||
|
||||
func TestANewLeaderKeepsTheNodesItHasNotHeardFromInTheControllersFiles(t *testing.T) {
|
||||
kv := &fakeKV{m: map[string]string{}}
|
||||
r := newRig()
|
||||
r.svc.SetMemory(StoreMemory(kv))
|
||||
r.set.Lent = []string{"cuda01"}
|
||||
r.inv.allocs = map[string]int{}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "MIXED", Jobs: 1})
|
||||
r.tick()
|
||||
want := r.agents.last("node06").GetFiles()["slurm.conf"]
|
||||
if !strings.Contains(want, "NodeName=cuda01") || !strings.Contains(want, "Gres=gpu:rtx3090:1") {
|
||||
t.Fatalf("slurm.conf:\n%s", want)
|
||||
}
|
||||
|
||||
// The leader goes; the next hears from the controller first.
|
||||
n := takeOver(r, kv, "node06")
|
||||
n.tick()
|
||||
if got := n.agents.last("node06").GetFiles()["slurm.conf"]; got != want {
|
||||
t.Fatalf("the new leader's slurm.conf differs; the controller would drop the node and its job:\n%s\nwas:\n%s",
|
||||
got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeIsToldNothingUntilItsControllerIsKnown(t *testing.T) {
|
||||
r := newRig()
|
||||
r.svc.SetMemory(StoreMemory(&fakeKV{m: map[string]string{}}))
|
||||
delete(r.inv.nodes, "node06")
|
||||
r.agents.conn = []string{"cuda01"}
|
||||
r.tick()
|
||||
if m := r.agents.last("cuda01"); m != nil {
|
||||
t.Fatalf("a node was told %v before its controller was known; an empty part stops its slurmd and its jobs", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANewLeaderTellsANodeItsControllerFromTheStore(t *testing.T) {
|
||||
kv := &fakeKV{m: map[string]string{}}
|
||||
r := newRig()
|
||||
r.svc.SetMemory(StoreMemory(kv))
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
|
||||
r.tick()
|
||||
|
||||
n := takeOver(r, kv, "cuda01")
|
||||
n.tick()
|
||||
if m := n.agents.last("cuda01"); !m.GetNode() || m.GetConfServer() != "10.0.0.146:6817" {
|
||||
t.Fatalf("the node was told %v", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithTheStoreUnreadTheControllerIsSentNothingThatLeavesANodeOut(t *testing.T) {
|
||||
kv := &fakeKV{m: map[string]string{}, err: errors.New("store down")}
|
||||
r := newRig()
|
||||
r.svc.SetMemory(StoreMemory(kv))
|
||||
r.tick()
|
||||
if m := r.agents.last("node06"); m != nil {
|
||||
t.Fatalf("the controller was sent %v without cuda01, with the store unread", m)
|
||||
}
|
||||
// Once every node has reported, nothing is left out: sent.
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
|
||||
r.tick()
|
||||
if !strings.Contains(r.agents.last("node06").GetFiles()["slurm.conf"], "NodeName=cuda01") {
|
||||
t.Fatal("with every node reported, the controller still got no files")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithNothingKeptANewLeaderWaitsForTheNodesAWhile(t *testing.T) {
|
||||
r := newRig()
|
||||
r.svc.SetMemory(StoreMemory(&fakeKV{m: map[string]string{}}))
|
||||
r.tick()
|
||||
if m := r.agents.last("node06"); m != nil {
|
||||
t.Fatalf("with nothing kept, the controller was sent %v before cuda01 reported", m)
|
||||
}
|
||||
defer func(d time.Duration) { HoldFor = d }(HoldFor)
|
||||
HoldFor = 0
|
||||
r.tick()
|
||||
if m := r.agents.last("node06"); !m.GetController() || strings.Contains(m.GetFiles()["slurm.conf"], "NodeName=cuda01") {
|
||||
t.Fatalf("after the wait, the controller was told %v", m)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
package slurmsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"slices"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Memory keeps the nodes slurm.conf last held, and the controller's
|
||||
// address, where a leader that takes over finds them. Without it a new
|
||||
// leader's first slurm.conf has none of the nodes it has not heard from
|
||||
// yet, and the controller, restarted on it, drops them and the jobs
|
||||
// running there.
|
||||
type Memory interface {
|
||||
Recall(ctx context.Context) (Kept, error)
|
||||
Keep(ctx context.Context, k Kept) error
|
||||
}
|
||||
|
||||
// Kept is what Memory holds.
|
||||
type Kept struct {
|
||||
Controller string `json:"controller,omitempty"`
|
||||
ControllerAddr string `json:"controller_addr,omitempty"`
|
||||
Nodes map[string]slurm.Node `json:"nodes,omitempty"`
|
||||
}
|
||||
|
||||
func (k Kept) equal(o Kept) bool {
|
||||
if k.Controller != o.Controller || k.ControllerAddr != o.ControllerAddr || len(k.Nodes) != len(o.Nodes) {
|
||||
return false
|
||||
}
|
||||
for name, n := range k.Nodes {
|
||||
m, ok := o.Nodes[name]
|
||||
if !ok || n.Addr != m.Addr || n.Hardware != m.Hardware || !slices.Equal(n.GPUs, m.GPUs) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// KV is the store's cluster state.
|
||||
type KV interface {
|
||||
PutClusterState(ctx context.Context, key, value string) (bool, error)
|
||||
ClusterState(ctx context.Context, key string) (string, error)
|
||||
}
|
||||
|
||||
// keptKey is the cluster state key it is kept under.
|
||||
const keptKey = "slurm-nodes"
|
||||
|
||||
// StoreMemory keeps it in the store's cluster state.
|
||||
func StoreMemory(kv KV) Memory { return kvMemory{kv} }
|
||||
|
||||
type kvMemory struct{ kv KV }
|
||||
|
||||
func (m kvMemory) Recall(ctx context.Context) (Kept, error) {
|
||||
var k Kept
|
||||
v, err := m.kv.ClusterState(ctx, keptKey)
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
return k, nil
|
||||
}
|
||||
if err != nil {
|
||||
return k, err
|
||||
}
|
||||
err = json.Unmarshal([]byte(v), &k)
|
||||
return k, err
|
||||
}
|
||||
|
||||
func (m kvMemory) Keep(ctx context.Context, k Kept) error {
|
||||
b, err := json.Marshal(k)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = m.kv.PutClusterState(ctx, keptKey, string(b))
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,742 @@
|
||||
// Package slurmsvc is the leader's side of the Slurm cluster run beside
|
||||
// llmgw (BACKLOG P): it tells each node's agent what it is, writes the
|
||||
// controller's files from what the nodes report, and hands nodes between
|
||||
// llmgw and Slurm as the cluster's settings say.
|
||||
//
|
||||
// A node in slurm.lent is drained of llmgw's replicas, as `ctl drain` does,
|
||||
// and lent to Slurm once none is left. A node taken out of slurm.lent is
|
||||
// drained in Slurm, and is llmgw's again when its last job has ended. Both
|
||||
// are kept in force every tick, from the settings and what the agents say,
|
||||
// so a leader that takes over carries on where the last one was.
|
||||
package slurmsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"net"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// Agents is how the service reaches the nodes.
|
||||
type Agents interface {
|
||||
Send(node string, msg *llmgwv1.ControlMessage) error
|
||||
ConnectedNodes() []string
|
||||
}
|
||||
|
||||
// Inventory is what the service reads and changes of llmgw's nodes.
|
||||
type Inventory interface {
|
||||
Get(id string) (inventory.Node, bool)
|
||||
SetState(id string, st inventory.NodeState) bool
|
||||
Allocations(nodeID string) []inventory.Allocation
|
||||
Nodes() []inventory.Node
|
||||
}
|
||||
|
||||
// Phase is where a node of slurm.nodes is between llmgw and Slurm.
|
||||
type Phase string
|
||||
|
||||
const (
|
||||
// Llmgw: drained in Slurm with no job left, llmgw's to place on.
|
||||
Llmgw Phase = "llmgw"
|
||||
// Lending: in slurm.lent, its replicas being moved off.
|
||||
Lending Phase = "lending"
|
||||
// Lent: Slurm's; llmgw places nothing there.
|
||||
Lent Phase = "lent"
|
||||
// Reclaiming: out of slurm.lent, drained in Slurm, jobs still running.
|
||||
Reclaiming Phase = "reclaiming"
|
||||
// Unknown: its agent has not said how it is in Slurm yet.
|
||||
Unknown Phase = "unknown"
|
||||
)
|
||||
|
||||
// NodeStatus is one node as the service sees it.
|
||||
type NodeStatus struct {
|
||||
Node string `json:"node"`
|
||||
Phase Phase `json:"phase"`
|
||||
Replicas int `json:"replicas"`
|
||||
// StopsBy is, while it is being lent, when a replica still on it is
|
||||
// stopped even with nowhere else to go: the drain's grace after the
|
||||
// lend began. Zero when not known (a leader that took over meanwhile).
|
||||
StopsBy time.Time `json:"stops_by,omitempty"`
|
||||
Status *Reported `json:"slurm,omitempty"`
|
||||
Connected bool `json:"connected"`
|
||||
}
|
||||
|
||||
// Reported is the agent's SlurmStatus, for the admin API.
|
||||
type Reported struct {
|
||||
Version string `json:"version,omitempty"`
|
||||
Hardware string `json:"hardware,omitempty"`
|
||||
SlurmdRunning bool `json:"slurmd_running"`
|
||||
SlurmctldRunning bool `json:"slurmctld_running"`
|
||||
SlurmdbdRunning bool `json:"slurmdbd_running"`
|
||||
// Shared is the shared path here: "mounted", "exported", "missing".
|
||||
Shared string `json:"shared,omitempty"`
|
||||
// Containers is what the machine has for jobs in containers, empty
|
||||
// while not both of pyxis and enroot.
|
||||
Containers string `json:"containers,omitempty"`
|
||||
// Users is how many of llmgw's users the machine has.
|
||||
Users int `json:"users"`
|
||||
State string `json:"state,omitempty"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
Jobs int `json:"jobs"`
|
||||
Error string `json:"error,omitempty"`
|
||||
FilesSHA256 string `json:"files_sha256,omitempty"`
|
||||
At time.Time `json:"at"`
|
||||
}
|
||||
|
||||
// Status is the whole Slurm cluster, for `ctl slurm status`.
|
||||
type Status struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
Cluster string `json:"cluster"`
|
||||
Controller string `json:"controller"`
|
||||
Partition string `json:"partition"`
|
||||
Ctld *Reported `json:"controller_status,omitempty"`
|
||||
Nodes []NodeStatus `json:"nodes"`
|
||||
// Waiting are nodes left out of slurm.conf until they report their
|
||||
// hardware.
|
||||
Waiting []string `json:"waiting,omitempty"`
|
||||
// Shared is where the shared files are and come from; Users who keeps
|
||||
// the users, and how many llmgw has.
|
||||
Shared config.SharedStorage `json:"shared"`
|
||||
Users string `json:"users,omitempty"`
|
||||
UserCount int `json:"user_count,omitempty"`
|
||||
// Containers is whether jobs may run in containers: "on" once every
|
||||
// machine has pyxis and enroot, "installing" until then, empty when
|
||||
// slurm.containers is off.
|
||||
Containers string `json:"containers,omitempty"`
|
||||
}
|
||||
|
||||
// User is a Linux user made from an llmgw account.
|
||||
type User struct {
|
||||
Login string `json:"login"`
|
||||
UID int `json:"uid"`
|
||||
Name string `json:"name"`
|
||||
Email string `json:"email"`
|
||||
SSHKeys []string `json:"ssh_keys,omitempty"`
|
||||
}
|
||||
|
||||
// Service keeps the Slurm cluster.
|
||||
type Service struct {
|
||||
settings func() config.SlurmSettings
|
||||
inv Inventory
|
||||
agents Agents
|
||||
key func(ctx context.Context) ([]byte, error)
|
||||
log *slog.Logger
|
||||
|
||||
mu sync.Mutex
|
||||
// status is each node's last SlurmStatus.
|
||||
status map[string]*Reported
|
||||
// drained are nodes this service put in Draining, which it alone makes
|
||||
// ready again.
|
||||
drained map[string]bool
|
||||
// lendingAt is when this leader began moving a node's replicas off.
|
||||
lendingAt map[string]time.Time
|
||||
// sent is what each node was last told, so it is told again only when
|
||||
// that changes or its stream is new.
|
||||
sent map[string]*llmgwv1.Slurm
|
||||
mungeKy []byte
|
||||
kick chan struct{}
|
||||
// mem keeps slurm.conf's nodes and the controller's address for the
|
||||
// next leader; known is what it held when read, and what was kept
|
||||
// since. loaded is false until it was read: until then the controller
|
||||
// is sent no files that leave out a node, which would drop that node
|
||||
// and its jobs.
|
||||
mem Memory
|
||||
known Kept
|
||||
loaded bool
|
||||
// started is the first pass, from which a node neither heard from nor
|
||||
// kept is waited for, for HoldFor.
|
||||
started time.Time
|
||||
// userList reads the users from the accounts; users is what it last
|
||||
// said, read at usersAt. Until it has been read once no node is told
|
||||
// anything of users: an empty list would take every user away.
|
||||
userList func(ctx context.Context) ([]User, error)
|
||||
users []User
|
||||
usersAt time.Time
|
||||
usersOK bool
|
||||
}
|
||||
|
||||
// UsersEvery is how often the users are read again without being asked.
|
||||
var UsersEvery = 30 * time.Second
|
||||
|
||||
// SetUsers has the users read with list, for slurm.users "llmgw".
|
||||
func (s *Service) SetUsers(list func(ctx context.Context) ([]User, error)) {
|
||||
s.mu.Lock()
|
||||
s.userList = list
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
// UsersChanged has the users read again now: an account or its keys
|
||||
// changed.
|
||||
func (s *Service) UsersChanged() {
|
||||
s.mu.Lock()
|
||||
s.usersAt = time.Time{}
|
||||
s.mu.Unlock()
|
||||
s.Kick()
|
||||
}
|
||||
|
||||
// readUsers reads the users when they are due. Under s.mu.
|
||||
func (s *Service) readUsers(ctx context.Context, set config.SlurmSettings) {
|
||||
if set.Users != config.UsersLlmgw || s.userList == nil || time.Since(s.usersAt) < UsersEvery {
|
||||
return
|
||||
}
|
||||
us, err := s.userList(ctx)
|
||||
s.usersAt = time.Now()
|
||||
if err != nil {
|
||||
s.log.Warn("could not read the Slurm users from the accounts; the nodes keep the last ones", "err", err)
|
||||
return
|
||||
}
|
||||
s.users, s.usersOK = us, true
|
||||
}
|
||||
|
||||
// HoldFor is how long a new leader waits for a node it neither has heard
|
||||
// from nor finds kept before it writes the controller's files without it:
|
||||
// on a cluster that kept nothing yet (one upgraded to this), the nodes
|
||||
// report within seconds of a new leader, and the controller keeps the
|
||||
// node and its jobs meanwhile.
|
||||
var HoldFor = 2 * time.Minute
|
||||
|
||||
// New is a Service; key gives the munge key, made once from a cluster
|
||||
// secret.
|
||||
func New(settings func() config.SlurmSettings, inv Inventory, agents Agents,
|
||||
key func(ctx context.Context) ([]byte, error), log *slog.Logger) *Service {
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
return &Service{settings: settings, inv: inv, agents: agents, key: key, log: log,
|
||||
status: map[string]*Reported{}, drained: map[string]bool{}, lendingAt: map[string]time.Time{}, sent: map[string]*llmgwv1.Slurm{},
|
||||
kick: make(chan struct{}, 1)}
|
||||
}
|
||||
|
||||
// SetMemory has slurm.conf's nodes kept in mem, and read from it first.
|
||||
func (s *Service) SetMemory(mem Memory) {
|
||||
s.mu.Lock()
|
||||
s.mem = mem
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
// current is the settings, slurm.all_nodes resolved to the nodes the
|
||||
// inventory has.
|
||||
func (s *Service) current() config.SlurmSettings {
|
||||
set := s.settings()
|
||||
if !set.AllNodes {
|
||||
return set
|
||||
}
|
||||
var ids []string
|
||||
for _, n := range s.inv.Nodes() {
|
||||
ids = append(ids, n.ID)
|
||||
}
|
||||
return set.WithNodes(ids)
|
||||
}
|
||||
|
||||
// Every is how often the service looks again without being asked.
|
||||
var Every = 10 * time.Second
|
||||
|
||||
// Run keeps the cluster until ctx ends.
|
||||
func (s *Service) Run(ctx context.Context) {
|
||||
t := time.NewTicker(Every)
|
||||
defer t.Stop()
|
||||
for {
|
||||
s.Tick(ctx)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
case <-s.kick:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Kick has the service look again now: the settings changed.
|
||||
func (s *Service) Kick() {
|
||||
select {
|
||||
case s.kick <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
// OnSlurmStatus records what a node's agent says.
|
||||
func (s *Service) OnSlurmStatus(node string, st *llmgwv1.SlurmStatus) {
|
||||
r := &Reported{Version: st.GetVersion(), Hardware: st.GetHardware(), SlurmdRunning: st.GetSlurmdRunning(),
|
||||
Shared: st.GetShared(), Users: int(st.GetUsers()), Containers: st.GetContainers(),
|
||||
SlurmctldRunning: st.GetSlurmctldRunning(), SlurmdbdRunning: st.GetSlurmdbdRunning(), State: st.GetState(), Reason: st.GetReason(),
|
||||
Jobs: int(st.GetJobs()), Error: st.GetError(), FilesSHA256: st.GetFilesSha256(), At: time.Now()}
|
||||
s.mu.Lock()
|
||||
prev := s.status[node]
|
||||
s.status[node] = r
|
||||
s.mu.Unlock()
|
||||
if r.Error != "" && (prev == nil || prev.Error != r.Error) {
|
||||
s.log.Warn("a node could not keep its part in Slurm", "node", node, "err", r.Error)
|
||||
}
|
||||
if prev == nil || prev.Hardware != r.Hardware || prev.State != r.State || prev.Jobs != r.Jobs {
|
||||
s.Kick()
|
||||
}
|
||||
}
|
||||
|
||||
// OnRegister is a node's new stream: it is told again what it is.
|
||||
func (s *Service) OnRegister(node string) {
|
||||
s.mu.Lock()
|
||||
delete(s.sent, node)
|
||||
s.mu.Unlock()
|
||||
s.Kick()
|
||||
}
|
||||
|
||||
// Tick does one pass: the nodes handed over as far as they can be, and
|
||||
// each told what it is.
|
||||
func (s *Service) Tick(ctx context.Context) {
|
||||
set := s.current()
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.recall(ctx)
|
||||
s.readUsers(ctx, set)
|
||||
if s.started.IsZero() {
|
||||
s.started = time.Now()
|
||||
}
|
||||
phases := s.handOver(set)
|
||||
var key []byte
|
||||
if set.Enabled {
|
||||
if s.mungeKy == nil {
|
||||
k, err := s.key(ctx)
|
||||
if err != nil {
|
||||
s.log.Warn("no munge key for Slurm; nothing is sent to the nodes", "err", err)
|
||||
return
|
||||
}
|
||||
s.mungeKy = k
|
||||
}
|
||||
key = s.mungeKy
|
||||
}
|
||||
files, waiting := s.files(set)
|
||||
// Not yet knowing what the last leader knew, a slurm.conf without some
|
||||
// node would take it, and its jobs, from the controller: kept back.
|
||||
hold := s.mem != nil && len(waiting) > 0 && (!s.loaded || time.Since(s.started) < HoldFor)
|
||||
for _, node := range s.agents.ConnectedNodes() {
|
||||
if hold && node == set.Controller {
|
||||
continue
|
||||
}
|
||||
m := s.message(set, node, phases[node] == Lent, key, files)
|
||||
if m == nil {
|
||||
continue
|
||||
}
|
||||
if old, ok := s.sent[node]; ok && proto.Equal(old, m) {
|
||||
continue
|
||||
}
|
||||
if err := s.agents.Send(node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_Slurm{Slurm: m}}); err != nil {
|
||||
s.log.Warn("could not tell a node its part in Slurm", "node", node, "err", err)
|
||||
continue
|
||||
}
|
||||
s.sent[node] = m
|
||||
}
|
||||
s.remember(ctx, set)
|
||||
}
|
||||
|
||||
// recall reads what the last leader kept, once. Under s.mu.
|
||||
func (s *Service) recall(ctx context.Context) {
|
||||
if s.mem == nil || s.loaded {
|
||||
return
|
||||
}
|
||||
k, err := s.mem.Recall(ctx)
|
||||
if err != nil {
|
||||
s.log.Warn("could not read Slurm's nodes from the store; the controller's files wait for every node",
|
||||
"err", err)
|
||||
return
|
||||
}
|
||||
s.known, s.loaded = k, true
|
||||
}
|
||||
|
||||
// remember keeps slurm.conf's nodes and the controller's address when they
|
||||
// changed. Under s.mu.
|
||||
func (s *Service) remember(ctx context.Context, set config.SlurmSettings) {
|
||||
if s.mem == nil || !s.loaded || !set.Enabled {
|
||||
return
|
||||
}
|
||||
c, _ := s.cluster(set)
|
||||
now := Kept{Controller: set.Controller, ControllerAddr: c.ControllerAddr, Nodes: map[string]slurm.Node{}}
|
||||
for _, n := range c.Nodes {
|
||||
now.Nodes[n.Name] = n
|
||||
}
|
||||
if now.equal(s.known) {
|
||||
return
|
||||
}
|
||||
if err := s.mem.Keep(ctx, now); err != nil {
|
||||
s.log.Warn("could not keep Slurm's nodes in the store", "err", err)
|
||||
return
|
||||
}
|
||||
s.known = now
|
||||
}
|
||||
|
||||
// controllerAddr is where the nodes reach slurmctld: the controller's
|
||||
// management address, else the one kept. Under s.mu.
|
||||
func (s *Service) controllerAddr(set config.SlurmSettings) string {
|
||||
if n, ok := s.inv.Get(set.Controller); ok && n.MgmtAddr != "" {
|
||||
return n.MgmtAddr
|
||||
}
|
||||
if s.known.Controller == set.Controller {
|
||||
return s.known.ControllerAddr
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// handOver moves each node of slurm.nodes a step between llmgw and Slurm,
|
||||
// answering where each is. Under s.mu.
|
||||
func (s *Service) handOver(set config.SlurmSettings) map[string]Phase {
|
||||
out := map[string]Phase{}
|
||||
if !set.Enabled {
|
||||
// Off: every node is llmgw's again. What Slurm still runs is left
|
||||
// to finish where it is; turning Slurm off with work lent is the
|
||||
// operator's choice.
|
||||
for node := range s.drained {
|
||||
s.ready(node)
|
||||
}
|
||||
return out
|
||||
}
|
||||
for _, node := range set.Nodes {
|
||||
out[node] = s.phase(set, node)
|
||||
}
|
||||
// A node no longer in slurm.nodes is llmgw's.
|
||||
for node := range s.drained {
|
||||
if !slices.Contains(set.Nodes, node) {
|
||||
s.ready(node)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (s *Service) phase(set config.SlurmSettings, node string) Phase {
|
||||
n, ok := s.inv.Get(node)
|
||||
st := s.status[node]
|
||||
if set.IsLent(node) {
|
||||
if ok && (n.State == inventory.Ready || n.State == inventory.Cordoned) {
|
||||
s.inv.SetState(node, inventory.Draining)
|
||||
s.drained[node] = true
|
||||
s.lendingAt[node] = time.Now()
|
||||
s.log.Info("lending a node to Slurm: its replicas are moved off first", "node", node,
|
||||
"stopped_by", time.Now().Add(drainGrace).Format(time.TimeOnly))
|
||||
}
|
||||
if !ok || len(s.inv.Allocations(node)) > 0 {
|
||||
return Lending
|
||||
}
|
||||
delete(s.lendingAt, node)
|
||||
return Lent
|
||||
}
|
||||
// Not lent. Until the agent says Slurm has drained it with no job left,
|
||||
// llmgw places nothing new on it: a new leader starts with every node
|
||||
// ready. Cordoned, not drained: llmgw has nothing there to move.
|
||||
if st == nil || slurmBusy(st) {
|
||||
if ok && n.State == inventory.Ready {
|
||||
s.inv.SetState(node, inventory.Cordoned)
|
||||
s.drained[node] = true
|
||||
if st != nil {
|
||||
s.log.Info("taking a node back from Slurm: llmgw places nothing there until its jobs end",
|
||||
"node", node, "jobs", st.Jobs)
|
||||
}
|
||||
}
|
||||
if st == nil {
|
||||
return Unknown
|
||||
}
|
||||
return Reclaiming
|
||||
}
|
||||
delete(s.lendingAt, node)
|
||||
if s.drained[node] {
|
||||
s.ready(node)
|
||||
}
|
||||
return Llmgw
|
||||
}
|
||||
|
||||
// drainGrace is how long a drained replica waits for its replacement
|
||||
// before it is stopped anyway: the reconciler's, which is not a setting.
|
||||
var drainGrace = reconciler.DefaultConfig().DrainGrace
|
||||
|
||||
// slurmBusy is whether Slurm may still run work on a node: a job step left
|
||||
// on it (or no telling), or a controller that would still give it one. A
|
||||
// node the controller does not know of, or cannot be asked about, gets no
|
||||
// job from it, so a Slurm that is broken never keeps a node from llmgw.
|
||||
func slurmBusy(st *Reported) bool {
|
||||
if st.Jobs != 0 {
|
||||
return true
|
||||
}
|
||||
if st.State == "" {
|
||||
return false
|
||||
}
|
||||
return !strings.Contains(st.State, "DRAIN") && !strings.Contains(st.State, "DOWN")
|
||||
}
|
||||
|
||||
func (s *Service) ready(node string) {
|
||||
if n, ok := s.inv.Get(node); ok && (n.State == inventory.Draining || n.State == inventory.Cordoned) {
|
||||
s.inv.SetState(node, inventory.Ready)
|
||||
s.log.Info("a node is llmgw's again: Slurm has no job there", "node", node)
|
||||
}
|
||||
delete(s.drained, node)
|
||||
}
|
||||
|
||||
// files are the controller's files, and the nodes left out of them for not
|
||||
// having reported their hardware. Under s.mu.
|
||||
func (s *Service) files(set config.SlurmSettings) (map[string]string, []string) {
|
||||
c, waiting := s.cluster(set)
|
||||
if c.ControllerAddr == "" {
|
||||
return nil, waiting
|
||||
}
|
||||
f := c.Files()
|
||||
if s.containersReady(set) {
|
||||
f["plugstack.conf"] = slurm.PlugstackConf
|
||||
}
|
||||
return f, waiting
|
||||
}
|
||||
|
||||
// containersReady is whether jobs may ask for containers: every machine of
|
||||
// the Slurm cluster says it has pyxis and enroot. Slurm fails a job on a
|
||||
// machine without the plugin it was told to load. Under s.mu.
|
||||
func (s *Service) containersReady(set config.SlurmSettings) bool {
|
||||
if !set.Containers {
|
||||
return false
|
||||
}
|
||||
for _, n := range s.machines(set) {
|
||||
if st := s.status[n]; st == nil || st.Containers == "" {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// cluster is what slurm.conf says: each node as it last reported, else as
|
||||
// kept from before, and the nodes left out for having neither. Under s.mu.
|
||||
func (s *Service) cluster(set config.SlurmSettings) (slurm.Cluster, []string) {
|
||||
c := slurm.Cluster{Name: set.Cluster(), Controller: set.Controller, Partition: set.PartitionName()}
|
||||
if !set.Enabled {
|
||||
return c, nil
|
||||
}
|
||||
c.ControllerAddr = s.controllerAddr(set)
|
||||
var waiting []string
|
||||
for _, node := range set.Nodes {
|
||||
sn, ok := s.reported(node)
|
||||
if !ok {
|
||||
sn, ok = s.known.Nodes[node]
|
||||
}
|
||||
if !ok {
|
||||
waiting = append(waiting, node)
|
||||
continue
|
||||
}
|
||||
sn.Lent = set.IsLent(node)
|
||||
c.Nodes = append(c.Nodes, sn)
|
||||
}
|
||||
return c, waiting
|
||||
}
|
||||
|
||||
// reported is a node as its agent and the inventory say now. Under s.mu.
|
||||
func (s *Service) reported(node string) (slurm.Node, bool) {
|
||||
n, ok := s.inv.Get(node)
|
||||
st := s.status[node]
|
||||
if !ok || n.MgmtAddr == "" || st == nil || st.Hardware == "" {
|
||||
return slurm.Node{}, false
|
||||
}
|
||||
hw, err := slurm.ParseHardware(st.Hardware)
|
||||
if err != nil {
|
||||
return slurm.Node{}, false
|
||||
}
|
||||
sn := slurm.Node{Name: node, Addr: n.MgmtAddr, Hardware: hw}
|
||||
for _, g := range n.GPUs {
|
||||
sn.GPUs = append(sn.GPUs, slurm.GPU{Minor: g.Index, Name: g.Name})
|
||||
}
|
||||
return sn, true
|
||||
}
|
||||
|
||||
// message is what a node is told. Under s.mu.
|
||||
func (s *Service) message(set config.SlurmSettings, node string, lent bool, key []byte,
|
||||
files map[string]string) *llmgwv1.Slurm {
|
||||
m := &llmgwv1.Slurm{}
|
||||
if !set.Enabled {
|
||||
return m
|
||||
}
|
||||
isNode := slices.Contains(set.Nodes, node)
|
||||
isCtl := set.Controller == node && files != nil
|
||||
if !isNode && !isCtl {
|
||||
return m
|
||||
}
|
||||
m.Name, m.MungeKey, m.Lent = node, key, lent
|
||||
m.Containers = set.Containers
|
||||
m.Node = isNode
|
||||
if addr := s.controllerAddr(set); addr != "" {
|
||||
m.ConfServer = net.JoinHostPort(addr, strconv.Itoa(slurm.CtldPort))
|
||||
}
|
||||
if isNode && m.ConfServer == "" {
|
||||
// Without its controller a node cannot start; it is told when the
|
||||
// controller registers. Told nothing until then, not "no part": a
|
||||
// node still running Slurm's jobs would stop them.
|
||||
return nil
|
||||
}
|
||||
if isCtl {
|
||||
m.Controller = true
|
||||
m.Files = maps.Clone(files)
|
||||
}
|
||||
if !s.sharedPart(set, node, m) || !s.usersPart(set, node, m) {
|
||||
return nil
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// sharedPart puts the shared files in m, false when where they come from
|
||||
// is not known yet: told nothing rather than no shared files, which would
|
||||
// unmount them under running jobs. Under s.mu.
|
||||
func (s *Service) sharedPart(set config.SlurmSettings, node string, m *llmgwv1.Slurm) bool {
|
||||
sh := set.Shared
|
||||
if sh.Path == "" {
|
||||
return true
|
||||
}
|
||||
m.Shared = &llmgwv1.SlurmShared{Path: sh.Path, Nfs: sh.NFS}
|
||||
switch {
|
||||
case sh.Server == "":
|
||||
case sh.Server == node:
|
||||
m.Shared.Serve = true
|
||||
for _, n := range s.machines(set) {
|
||||
if n == node {
|
||||
continue
|
||||
}
|
||||
if a := s.addr(set, n); a != "" {
|
||||
m.Shared.Clients = append(m.Shared.Clients, a)
|
||||
}
|
||||
}
|
||||
slices.Sort(m.Shared.Clients)
|
||||
default:
|
||||
a := s.addr(set, sh.Server)
|
||||
if a == "" {
|
||||
return false
|
||||
}
|
||||
if strings.Contains(a, ":") {
|
||||
a = "[" + a + "]"
|
||||
}
|
||||
m.Shared.Nfs = a + ":" + sh.Path
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// usersPart puts the users in m, false while they have not been read.
|
||||
// Under s.mu.
|
||||
func (s *Service) usersPart(set config.SlurmSettings, node string, m *llmgwv1.Slurm) bool {
|
||||
if set.Users != config.UsersLlmgw {
|
||||
return true
|
||||
}
|
||||
if !s.usersOK {
|
||||
return false
|
||||
}
|
||||
m.ManageUsers = true
|
||||
switch {
|
||||
case set.Shared.Path == "":
|
||||
m.MakeHomes = true // each machine's own /home
|
||||
case set.Shared.Server != "":
|
||||
m.MakeHomes = set.Shared.Server == node
|
||||
default:
|
||||
m.MakeHomes = set.Controller == node
|
||||
}
|
||||
for _, u := range s.users {
|
||||
m.Users = append(m.Users, &llmgwv1.SlurmUser{Login: u.Login, Uid: uint32(u.UID), Name: u.Name, //nolint:gosec // from FirstSlurmUID up
|
||||
Home: set.HomeDir(u.Login), SshKeys: slices.Clone(u.SSHKeys)})
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// machines are the controller and the nodes.
|
||||
func (s *Service) machines(set config.SlurmSettings) []string {
|
||||
out := slices.Clone(set.Nodes)
|
||||
if !slices.Contains(out, set.Controller) {
|
||||
out = append(out, set.Controller)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// addr is a machine's management address, from the inventory, else kept
|
||||
// from before. Under s.mu.
|
||||
func (s *Service) addr(set config.SlurmSettings, node string) string {
|
||||
if n, ok := s.inv.Get(node); ok && n.MgmtAddr != "" {
|
||||
return n.MgmtAddr
|
||||
}
|
||||
if node == set.Controller {
|
||||
return s.controllerAddr(set)
|
||||
}
|
||||
return s.known.Nodes[node].Addr
|
||||
}
|
||||
|
||||
// Status is the cluster as the service sees it now.
|
||||
func (s *Service) Status() Status {
|
||||
set := s.current()
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
out := Status{Enabled: set.Enabled, Cluster: set.Cluster(), Controller: set.Controller,
|
||||
Partition: set.PartitionName(), Ctld: s.status[set.Controller], Shared: set.Shared, Users: set.Users}
|
||||
if set.Users == config.UsersLlmgw {
|
||||
out.UserCount = len(s.users)
|
||||
}
|
||||
if !set.Enabled {
|
||||
return out
|
||||
}
|
||||
if set.Containers {
|
||||
out.Containers = "installing"
|
||||
if s.containersReady(set) {
|
||||
out.Containers = "on"
|
||||
}
|
||||
}
|
||||
_, out.Waiting = s.files(set)
|
||||
conn := s.agents.ConnectedNodes()
|
||||
for _, node := range set.Nodes {
|
||||
ns := NodeStatus{Node: node, Status: s.status[node], Replicas: len(s.inv.Allocations(node)),
|
||||
Connected: slices.Contains(conn, node)}
|
||||
ns.Phase = s.peek(set, node)
|
||||
if at, ok := s.lendingAt[node]; ok && ns.Phase == Lending {
|
||||
ns.StopsBy = at.Add(drainGrace)
|
||||
}
|
||||
out.Nodes = append(out.Nodes, ns)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// peek is phase without changing anything. Under s.mu.
|
||||
func (s *Service) peek(set config.SlurmSettings, node string) Phase {
|
||||
st := s.status[node]
|
||||
if set.IsLent(node) {
|
||||
if len(s.inv.Allocations(node)) > 0 {
|
||||
return Lending
|
||||
}
|
||||
return Lent
|
||||
}
|
||||
switch {
|
||||
case st == nil:
|
||||
return Unknown
|
||||
case slurmBusy(st):
|
||||
return Reclaiming
|
||||
}
|
||||
return Llmgw
|
||||
}
|
||||
|
||||
// Describe is one line for a log or a screen.
|
||||
func (n NodeStatus) Describe() string {
|
||||
switch n.Phase {
|
||||
case Lending:
|
||||
if !n.StopsBy.IsZero() {
|
||||
// A replica moves once its replacement serves; one with nowhere
|
||||
// else to go waits out the grace, and that wait is not a hang.
|
||||
return fmt.Sprintf("being lent: %d replica(s) still to move off; one with nowhere else to go is stopped at %s",
|
||||
n.Replicas, n.StopsBy.Local().Format(time.TimeOnly))
|
||||
}
|
||||
return fmt.Sprintf("being lent: %d replica(s) still to move off", n.Replicas)
|
||||
case Lent:
|
||||
return "lent to Slurm"
|
||||
case Reclaiming:
|
||||
if n.Status != nil {
|
||||
return fmt.Sprintf("being taken back: %d job(s) still running", n.Status.Jobs)
|
||||
}
|
||||
return "being taken back"
|
||||
case Llmgw:
|
||||
return "llmgw's"
|
||||
}
|
||||
return "not heard from yet"
|
||||
}
|
||||
@@ -0,0 +1,325 @@
|
||||
package slurmsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
type fakeInv struct {
|
||||
nodes map[string]*inventory.Node
|
||||
allocs map[string]int
|
||||
}
|
||||
|
||||
func (f *fakeInv) Get(id string) (inventory.Node, bool) {
|
||||
n, ok := f.nodes[id]
|
||||
if !ok {
|
||||
return inventory.Node{}, false
|
||||
}
|
||||
return *n, true
|
||||
}
|
||||
|
||||
func (f *fakeInv) SetState(id string, st inventory.NodeState) bool {
|
||||
n, ok := f.nodes[id]
|
||||
if ok {
|
||||
n.State = st
|
||||
}
|
||||
return ok
|
||||
}
|
||||
|
||||
func (f *fakeInv) Allocations(id string) []inventory.Allocation {
|
||||
return make([]inventory.Allocation, f.allocs[id])
|
||||
}
|
||||
|
||||
func (f *fakeInv) Nodes() []inventory.Node {
|
||||
var out []inventory.Node
|
||||
for _, n := range f.nodes {
|
||||
out = append(out, *n)
|
||||
}
|
||||
slices.SortFunc(out, func(a, b inventory.Node) int { return strings.Compare(a.ID, b.ID) })
|
||||
return out
|
||||
}
|
||||
|
||||
type fakeAgents struct {
|
||||
conn []string
|
||||
got map[string][]*llmgwv1.Slurm
|
||||
}
|
||||
|
||||
func (f *fakeAgents) Send(node string, m *llmgwv1.ControlMessage) error {
|
||||
f.got[node] = append(f.got[node], m.GetSlurm())
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeAgents) ConnectedNodes() []string { return f.conn }
|
||||
|
||||
func (f *fakeAgents) last(node string) *llmgwv1.Slurm {
|
||||
ms := f.got[node]
|
||||
if len(ms) == 0 {
|
||||
return nil
|
||||
}
|
||||
return ms[len(ms)-1]
|
||||
}
|
||||
|
||||
type rig struct {
|
||||
set config.SlurmSettings
|
||||
inv *fakeInv
|
||||
agents *fakeAgents
|
||||
svc *Service
|
||||
}
|
||||
|
||||
func newRig() *rig {
|
||||
r := &rig{
|
||||
set: config.SlurmSettings{Enabled: true, Controller: "node06", Nodes: []string{"cuda01"}},
|
||||
inv: &fakeInv{nodes: map[string]*inventory.Node{
|
||||
"node06": {ID: "node06", MgmtAddr: "10.0.0.146"},
|
||||
"cuda01": {ID: "cuda01", MgmtAddr: "10.0.0.147", GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA GeForce RTX 3090"}}},
|
||||
}, allocs: map[string]int{"cuda01": 2}},
|
||||
agents: &fakeAgents{conn: []string{"node06", "cuda01"}, got: map[string][]*llmgwv1.Slurm{}},
|
||||
}
|
||||
r.svc = New(func() config.SlurmSettings { return r.set }, r.inv, r.agents,
|
||||
func(context.Context) ([]byte, error) { return []byte(strings.Repeat("k", 1024)), nil }, nil)
|
||||
return r
|
||||
}
|
||||
|
||||
func (r *rig) report(node string, st *llmgwv1.SlurmStatus) { r.svc.OnSlurmStatus(node, st) }
|
||||
|
||||
func (r *rig) tick() { r.svc.Tick(context.Background()) }
|
||||
|
||||
const hw = "NodeName=cuda01 CPUs=16 RealMemory=64000"
|
||||
|
||||
func TestMessagesAndFiles(t *testing.T) {
|
||||
r := newRig()
|
||||
r.tick()
|
||||
ctl, node := r.agents.last("node06"), r.agents.last("cuda01")
|
||||
if !ctl.GetController() || ctl.GetNode() || len(ctl.GetMungeKey()) != 1024 {
|
||||
t.Fatalf("controller told %v", ctl)
|
||||
}
|
||||
if !node.GetNode() || node.GetController() || node.GetConfServer() != "10.0.0.146:6817" || node.GetLent() ||
|
||||
node.GetName() != "cuda01" || len(node.GetFiles()) != 0 {
|
||||
t.Fatalf("node told %v", node)
|
||||
}
|
||||
if strings.Contains(ctl.GetFiles()["slurm.conf"], "NodeName=cuda01") {
|
||||
t.Fatal("a node with no hardware reported is in slurm.conf")
|
||||
}
|
||||
if w := r.svc.Status().Waiting; len(w) != 1 || w[0] != "cuda01" {
|
||||
t.Fatalf("waiting %v", w)
|
||||
}
|
||||
// Same again: nobody is told twice.
|
||||
r.tick()
|
||||
if len(r.agents.got["node06"]) != 1 || len(r.agents.got["cuda01"]) != 1 {
|
||||
t.Fatal("told again with nothing changed")
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
|
||||
r.tick()
|
||||
conf := r.agents.last("node06").GetFiles()["slurm.conf"]
|
||||
if !strings.Contains(conf, "NodeName=cuda01 NodeAddr=10.0.0.147 CPUs=16 RealMemory=62080 Gres=gpu:rtx3090:1") {
|
||||
t.Fatalf("slurm.conf:\n%s", conf)
|
||||
}
|
||||
if !strings.Contains(r.agents.last("node06").GetFiles()["gres.conf"], "File=/dev/nvidia0") {
|
||||
t.Fatal("gres.conf has no card")
|
||||
}
|
||||
if len(r.agents.got["cuda01"]) != 1 {
|
||||
t.Fatal("a node was told again when only the controller's files changed")
|
||||
}
|
||||
// A new stream is told again.
|
||||
r.svc.OnRegister("cuda01")
|
||||
r.tick()
|
||||
if len(r.agents.got["cuda01"]) != 2 {
|
||||
t.Fatal("a new stream was not told")
|
||||
}
|
||||
if r.inv.nodes["cuda01"].State != inventory.Ready {
|
||||
t.Fatal("a node drained in Slurm with no job was taken from llmgw")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLendAndReclaim(t *testing.T) {
|
||||
r := newRig()
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
|
||||
r.set.Lent = []string{"cuda01"}
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State != inventory.Draining {
|
||||
t.Fatal("a lent node was not drained of its replicas")
|
||||
}
|
||||
if r.agents.last("cuda01").GetLent() {
|
||||
t.Fatal("lent to Slurm with replicas still on it")
|
||||
}
|
||||
if p := r.svc.Status().Nodes[0].Phase; p != Lending {
|
||||
t.Fatalf("phase %s", p)
|
||||
}
|
||||
r.inv.allocs["cuda01"] = 0
|
||||
r.tick()
|
||||
if !r.agents.last("cuda01").GetLent() || r.svc.Status().Nodes[0].Phase != Lent {
|
||||
t.Fatal("not lent once its replicas were gone")
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "ALLOCATED", Jobs: 3})
|
||||
|
||||
// Taken back: Slurm drains it; llmgw keeps off until its jobs end.
|
||||
r.set.Lent = nil
|
||||
r.tick()
|
||||
if r.agents.last("cuda01").GetLent() {
|
||||
t.Fatal("still lent")
|
||||
}
|
||||
if s := r.inv.nodes["cuda01"].State; s != inventory.Draining && s != inventory.Cordoned {
|
||||
t.Fatalf("llmgw may place on a node with Slurm jobs: %v", s)
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "ALLOCATED+DRAIN", Jobs: 1})
|
||||
r.tick()
|
||||
if r.svc.Status().Nodes[0].Phase != Reclaiming || r.inv.nodes["cuda01"].State == inventory.Ready {
|
||||
t.Fatal("given back with a job left")
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: serving models"})
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State != inventory.Ready || r.svc.Status().Nodes[0].Phase != Llmgw {
|
||||
t.Fatal("not llmgw's when its last job ended")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewLeaderKeepsOffUntilHeard(t *testing.T) {
|
||||
r := newRig()
|
||||
r.inv.allocs["cuda01"] = 0
|
||||
r.tick()
|
||||
// Nothing heard yet: cordoned, nothing moved.
|
||||
if r.inv.nodes["cuda01"].State != inventory.Cordoned {
|
||||
t.Fatalf("state %v before the agent said anything", r.inv.nodes["cuda01"].State)
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "MIXED", Jobs: 2})
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State == inventory.Ready {
|
||||
t.Fatal("ready with Slurm jobs on it")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBrokenSlurmLeavesTheNodeToLlmgw(t *testing.T) {
|
||||
r := newRig()
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Error: "installing Slurm: no package", Jobs: 0})
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State != inventory.Ready {
|
||||
t.Fatal("a Slurm that does not work took a node from llmgw")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOperatorsCordonIsLeft(t *testing.T) {
|
||||
r := newRig()
|
||||
r.inv.nodes["cuda01"].State = inventory.Cordoned
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN"})
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State != inventory.Cordoned {
|
||||
t.Fatal("llmgw uncordoned a node an operator cordoned")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOffAndOut(t *testing.T) {
|
||||
r := newRig()
|
||||
r.inv.allocs["cuda01"] = 0
|
||||
r.set.Lent = []string{"cuda01"}
|
||||
r.tick()
|
||||
r.set.Enabled = false
|
||||
r.tick()
|
||||
if m := r.agents.last("cuda01"); m.GetNode() || m.GetController() {
|
||||
t.Fatalf("told %v with Slurm off", m)
|
||||
}
|
||||
if r.inv.nodes["cuda01"].State != inventory.Ready {
|
||||
t.Fatal("not llmgw's with Slurm off")
|
||||
}
|
||||
|
||||
r = newRig()
|
||||
r.inv.allocs["cuda01"] = 0
|
||||
r.set.Lent = []string{"cuda01"}
|
||||
r.tick()
|
||||
r.set.Nodes, r.set.Lent = nil, nil
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State != inventory.Ready || r.agents.last("cuda01").GetNode() {
|
||||
t.Fatal("a node taken out of slurm.nodes is not llmgw's")
|
||||
}
|
||||
if !r.agents.last("node06").GetController() {
|
||||
t.Fatal("the controller stopped with no nodes")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoControllerYet(t *testing.T) {
|
||||
r := newRig()
|
||||
delete(r.inv.nodes, "node06")
|
||||
r.agents.conn = []string{"cuda01"}
|
||||
r.tick()
|
||||
if m := r.agents.last("cuda01"); m.GetNode() {
|
||||
t.Fatalf("a node was started with no controller to fetch its files from: %v", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAllNodes(t *testing.T) {
|
||||
r := newRig()
|
||||
r.set.Nodes, r.set.AllNodes = nil, true
|
||||
r.inv.nodes["cuda02"] = &inventory.Node{ID: "cuda02", MgmtAddr: "10.0.0.150"}
|
||||
r.agents.conn = append(r.agents.conn, "cuda02")
|
||||
r.tick()
|
||||
for _, n := range []string{"cuda01", "cuda02", "node06"} {
|
||||
if !r.agents.last(n).GetNode() {
|
||||
t.Fatalf("%s, a node that joined, does not run slurmd", n)
|
||||
}
|
||||
}
|
||||
if got := len(r.svc.Status().Nodes); got != 3 {
|
||||
t.Fatalf("%d nodes in the status", got)
|
||||
}
|
||||
r.set.Lent = []string{"cuda02"}
|
||||
r.tick()
|
||||
if p := r.svc.Status().Nodes[1]; p.Node != "cuda02" || p.Phase != Lent {
|
||||
t.Fatalf("cuda02 not lent: %+v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLendingSaysWhenAReplicaWithNowhereToGoStops(t *testing.T) {
|
||||
// Seen on the PoC: a GPU model with no other card waited out the drain's
|
||||
// grace, and five minutes of "1 replica(s) still to move off" read as a hang.
|
||||
at := time.Date(2026, 11, 14, 19, 38, 34, 0, time.Local)
|
||||
n := NodeStatus{Node: "cuda01", Phase: Lending, Replicas: 1, StopsBy: at}
|
||||
if d := n.Describe(); !strings.Contains(d, "nowhere else to go is stopped at 19:38:34") {
|
||||
t.Errorf("%q", d)
|
||||
}
|
||||
n.StopsBy = time.Time{}
|
||||
if d := n.Describe(); d != "being lent: 1 replica(s) still to move off" {
|
||||
t.Errorf("%q", d)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPyxisIsLoadedOnlyOnceEveryMachineHasIt(t *testing.T) {
|
||||
// "required" in plugstack.conf fails every job on a machine without the
|
||||
// plugin; sent early, it would stop Slurm's jobs everywhere it is missing.
|
||||
r := newRig()
|
||||
r.set.Containers = true
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw})
|
||||
r.tick()
|
||||
if !r.agents.last("cuda01").GetContainers() || !r.agents.last("node06").GetContainers() {
|
||||
t.Fatal("the machines were not told to set containers up")
|
||||
}
|
||||
if _, ok := r.agents.last("node06").GetFiles()["plugstack.conf"]; ok {
|
||||
t.Fatal("pyxis loaded before any machine has it")
|
||||
}
|
||||
if s := r.svc.Status().Containers; s != "installing" {
|
||||
t.Errorf("status %q", s)
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, Containers: "pyxis 0.24.0, enroot 4.2.1"})
|
||||
r.tick()
|
||||
if _, ok := r.agents.last("node06").GetFiles()["plugstack.conf"]; ok {
|
||||
t.Fatal("pyxis loaded with the controller still without it")
|
||||
}
|
||||
r.report("node06", &llmgwv1.SlurmStatus{Containers: "pyxis 0.24.0, enroot 4.2.1"})
|
||||
r.tick()
|
||||
if p := r.agents.last("node06").GetFiles()["plugstack.conf"]; !strings.Contains(p, "required /usr/local/lib/slurm/spank_pyxis.so") {
|
||||
t.Fatalf("plugstack.conf %q", p)
|
||||
}
|
||||
if s := r.svc.Status().Containers; s != "on" {
|
||||
t.Errorf("status %q", s)
|
||||
}
|
||||
r.set.Containers = false
|
||||
r.tick()
|
||||
if _, ok := r.agents.last("node06").GetFiles()["plugstack.conf"]; ok || r.svc.Status().Containers != "" {
|
||||
t.Fatal("pyxis still loaded with containers off")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package slurmsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"slices"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
)
|
||||
|
||||
func TestSharedFilesFromTheController(t *testing.T) {
|
||||
r := newRig()
|
||||
r.set.Shared = config.SharedStorage{Path: "/shared", Server: "node06"}
|
||||
r.tick()
|
||||
ctl, node := r.agents.last("node06").GetShared(), r.agents.last("cuda01").GetShared()
|
||||
if !ctl.GetServe() || !slices.Equal(ctl.GetClients(), []string{"10.0.0.147"}) || ctl.GetPath() != "/shared" {
|
||||
t.Fatalf("server told %v", ctl)
|
||||
}
|
||||
if node.GetServe() || node.GetNfs() != "10.0.0.146:/shared" {
|
||||
t.Fatalf("client told %v", node)
|
||||
}
|
||||
// The site's own export: every machine mounts it.
|
||||
r.set.Shared = config.SharedStorage{Path: "/shared", NFS: "filer:/hpc"}
|
||||
r.tick()
|
||||
if r.agents.last("node06").GetShared().GetNfs() != "filer:/hpc" || r.agents.last("cuda01").GetShared().GetServe() {
|
||||
t.Fatal("the site's export not sent")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSharedServerNotKnownTellsNothing(t *testing.T) {
|
||||
r := newRig()
|
||||
r.set.Shared = config.SharedStorage{Path: "/shared", Server: "node06"}
|
||||
delete(r.inv.nodes, "node06")
|
||||
r.svc.known.ControllerAddr, r.svc.known.Controller = "10.0.0.146", "node06"
|
||||
r.tick()
|
||||
if r.agents.last("cuda01").GetShared().GetNfs() != "10.0.0.146:/shared" {
|
||||
t.Fatalf("the kept address not used: %v", r.agents.last("cuda01"))
|
||||
}
|
||||
r2 := newRig()
|
||||
r2.set.Nodes = []string{"cuda01", "site2"}
|
||||
r2.agents.conn = append(r2.agents.conn, "site2")
|
||||
r2.set.Shared = config.SharedStorage{Path: "/shared", Server: "site2"}
|
||||
r2.tick()
|
||||
if r2.agents.last("cuda01") != nil {
|
||||
t.Fatal("told with no address for the server: its mount would go")
|
||||
}
|
||||
}
|
||||
|
||||
func TestUsersSentOnceRead(t *testing.T) {
|
||||
r := newRig()
|
||||
r.set.Users = config.UsersLlmgw
|
||||
r.set.Shared = config.SharedStorage{Path: "/shared", Server: "node06"}
|
||||
fail := true
|
||||
r.svc.SetUsers(func(context.Context) ([]User, error) {
|
||||
if fail {
|
||||
return nil, errors.New("store down")
|
||||
}
|
||||
return []User{{Login: "ayse", UID: 200000, Name: "Ayşe", SSHKeys: []string{"ssh-ed25519 AAAA"}}}, nil
|
||||
})
|
||||
r.tick()
|
||||
if len(r.agents.got["cuda01"]) != 0 {
|
||||
t.Fatal("told before the users were read: every user would go")
|
||||
}
|
||||
fail = false
|
||||
r.svc.UsersChanged()
|
||||
r.tick()
|
||||
ctl, node := r.agents.last("node06"), r.agents.last("cuda01")
|
||||
if !ctl.GetManageUsers() || !ctl.GetMakeHomes() || node.GetMakeHomes() || len(node.GetUsers()) != 1 {
|
||||
t.Fatalf("controller %v\nnode %v", ctl, node)
|
||||
}
|
||||
if u := node.GetUsers()[0]; u.GetHome() != "/shared/home/ayse" || u.GetUid() != 200000 || len(u.GetSshKeys()) != 1 {
|
||||
t.Fatalf("user %v", u)
|
||||
}
|
||||
// The store down later: the last users are kept.
|
||||
fail = true
|
||||
r.svc.UsersChanged()
|
||||
r.tick()
|
||||
if len(r.agents.last("cuda01").GetUsers()) != 1 {
|
||||
t.Fatal("users lost when the store could not be read")
|
||||
}
|
||||
if st := r.svc.Status(); st.UserCount != 1 || st.Shared.Server != "node06" {
|
||||
t.Fatalf("status %+v", st)
|
||||
}
|
||||
}
|
||||
@@ -94,7 +94,7 @@ type Store interface {
|
||||
SetStaffRole(ctx context.Context, id, role string) error
|
||||
SetUserState(ctx context.Context, id string, state store.UserState) error
|
||||
DeleteUserSessions(ctx context.Context, userID string) error
|
||||
AddMembership(ctx context.Context, user string, scope store.Scope, id, role, by string, at time.Time) error
|
||||
EnsureMembership(ctx context.Context, user string, scope store.Scope, id, by string, at time.Time) error
|
||||
RemoveMembership(ctx context.Context, userID string, scope store.Scope, scopeID string, at time.Time) (int, error)
|
||||
MembershipsOf(ctx context.Context, userID string) ([]store.Membership, error)
|
||||
AllOrgs(ctx context.Context) ([]store.Org, error)
|
||||
@@ -221,7 +221,7 @@ func (k *Keeper) apply(ctx context.Context, u *store.User, id Identity, remove b
|
||||
continue // a mapping to a place the tree lacks: said by Sync
|
||||
}
|
||||
want[string(scope)+"/"+sid] = scope
|
||||
if err := k.St.AddMembership(ctx, u.ID, scope, sid, store.RoleMember, k.Name, k.now()); err != nil {
|
||||
if err := k.St.EnsureMembership(ctx, u.ID, scope, sid, k.Name, k.now()); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
@@ -377,6 +377,7 @@ var Reference = []Block{
|
||||
{Key: "keep_within", Type: "string", Doc: "KeepWithin keeps a multi-node replica's nodes inside one rack, row,\nroom or site, read from their llmgw.io/rack (and so on) labels: for a\nfabric whose leaves do not reach across racks, or cables that do not.\nNodes without the label are passed over for such a profile. Empty is\nno constraint.", Nested: ""},
|
||||
{Key: "weight", Type: "float64", Doc: "Weight is the router's capacity weight. It should come from ctl bench\nrather than a guess (plan §6.2.1).", Nested: ""},
|
||||
{Key: "priority", Type: "int", Doc: "Priority orders profiles during placement; highest that fits wins.", Nested: ""},
|
||||
{Key: "replicas", Type: "int", Doc: "Replicas, set on any of a model's profiles, is how many of its\nreplicas run on this one: seven on cards and four on CPUs, all\nanswering under the model's name, rather than each replica on the\nbest profile that fits. The model's scaling is then the sum, min and\nmax alike; a profile left at zero runs none.", Nested: ""},
|
||||
{Key: "startup_timeout", Type: "time.Duration", Doc: "StartupTimeout bounds model loading. A 70B model across two nodes takes\ntens of minutes, so this is per profile rather than global.", Nested: ""},
|
||||
{Key: "kv_events", Type: "bool", Doc: "KVEvents has vLLM publish what its prefix cache holds, and the router\nroute this model's requests by it: to the replica holding the most of\na prompt's leading blocks (plan §7.5 tier 2). vLLM only.\n\nOff by default, because it costs something on every request: the\nrouter asks a replica to tokenize the prompt before placing it, a\nround trip and the chat template run twice. It pays where prompts\nshare long prefixes that the approximate tree guesses wrong — many\nreplicas, conversations that move between them, evictions. The engine\nis started with sha256_cbor block hashing and a fixed PYTHONHASHSEED,\nwhich the router's hashing assumes; arguments overriding either leave\nthe index empty, and the agent says so in its log.", Nested: ""},
|
||||
{Key: "gpu", Type: "*GPUNeed", Doc: "GPU says what card the profile needs, in place of naming its class:\nit stands for every class in the cluster that meets it. A catalog\nentry written once then places sensibly on any site's cards.", Nested: "catalog.GPUNeed"},
|
||||
@@ -479,6 +480,7 @@ var Reference = []Block{
|
||||
{Key: "mail", Type: "mail.Config", Doc: "Mail is where the gateway's few emails go out from: a link to set a\nforgotten password. Unset, none is sent and people ask an operator.", Nested: "mail.Config"},
|
||||
{Key: "serving", Type: "ServingSettings", Doc: "Serving stops and starts the whole cluster's service (llmgw stop, llmgw\nstart).", Nested: "config.ServingSettings"},
|
||||
{Key: "firewall", Type: "FirewallSettings", Doc: "Firewall turns llmgw's firewall on every machine on or off.", Nested: "config.FirewallSettings"},
|
||||
{Key: "slurm", Type: "SlurmSettings", Doc: "Slurm is a Slurm cluster run beside llmgw on some of its nodes, and\nwhich of them it may use now.", Nested: "config.SlurmSettings"},
|
||||
{Key: "presets", Type: "PresetSettings", Doc: "Presets is how the models that come with llmgw reach the catalog.", Nested: "config.PresetSettings"},
|
||||
{Key: "on_demand", Type: "OnDemandSettings", Doc: "OnDemand is how models that load when asked for take turns on the\ncards.", Nested: "config.OnDemandSettings"},
|
||||
{Key: "agents", Type: "AgentSettings", Doc: "Agents are the nodes' agent settings: agent.yaml's keys but those an\nagent needs to start, for every node (default) and for one\n(nodes.<id>). Sent to the agents as they change; an agent whose\nsettings changed starts again on them, its engines serving\nthroughout.", Nested: "config.AgentSettings"},
|
||||
@@ -508,9 +510,26 @@ var Reference = []Block{
|
||||
{Package: "config", Name: "ServingSettings", Title: "ServingSettings", Doc: "", Fields: []Field{
|
||||
{Key: "stopped", Type: "bool", Doc: "Stopped takes every model down and answers every request for one with\n503 (system_stopped), leaving the control planes, agents and mirror\nrunning and every model's definition as it is: llmgw stop sets it,\nllmgw start clears it, and the models come back as their catalog\nentries ask. In force on every control plane within seconds.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "SharedStorage", Title: "SharedStorage", Doc: "", Fields: []Field{
|
||||
{Key: "path", Type: "string", Doc: "Path is where every machine sees it, as /shared.", Nested: ""},
|
||||
{Key: "nfs", Type: "string", Doc: "NFS is the site's export, host:/path, mounted at Path on every\nmachine of the Slurm cluster.", Nested: ""},
|
||||
{Key: "server", Type: "string", Doc: "Server is the llmgw node, the controller or one of the nodes, that\nexports Path over NFS to the others: for a site with no storage of\nits own. One machine: not highly available, and no faster than its\ndisk and network.", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "SimConfig", Title: "SimConfig", Doc: "", Fields: []Field{
|
||||
{Key: "bin", Type: "string", Doc: "Bin is the llmgw binary itself; the engine is one of its subcommands.\nEmpty means whatever started this agent, which is right almost always.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "SlurmSettings", Title: "SlurmSettings", Doc: "", Fields: []Field{
|
||||
{Key: "enabled", Type: "bool", Doc: "Enabled runs Slurm on the nodes below. False stops llmgw keeping it;\nwhat was installed is left installed, and every node is llmgw's.", Nested: ""},
|
||||
{Key: "cluster_name", Type: "string", Doc: "ClusterName is Slurm's ClusterName. Empty is \"llmgw\".", Nested: ""},
|
||||
{Key: "controller", Type: "string", Doc: "Controller is the llmgw node whose agent runs slurmctld. It needs no\nGPU, and need not be one of Nodes.", Nested: ""},
|
||||
{Key: "nodes", Type: "[]string", Doc: "Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained\nin Slurm, until it is also in Lent.", Nested: ""},
|
||||
{Key: "all_nodes", Type: "bool", Doc: "AllNodes runs slurmd on every llmgw node, those that join later too,\nbesides any named in Nodes. Each is llmgw's until it is lent, as any.", Nested: ""},
|
||||
{Key: "lent", Type: "[]string", Doc: "Lent are the nodes of Nodes Slurm may run jobs on now. A node put here\nis drained of llmgw's replicas first, as `ctl drain` does, and resumed\nin Slurm once none is left; a node taken out is drained in Slurm and\ngiven back to llmgw when its last job ends.", Nested: ""},
|
||||
{Key: "partition", Type: "string", Doc: "Partition is the one partition every node is in. Empty is \"gpu\".", Nested: ""},
|
||||
{Key: "shared", Type: "SharedStorage", Doc: "Shared is the file system every machine of the Slurm cluster sees at\nthe same path: home directories, datasets, the jobs' output and\ncheckpoints. Empty path: none, and a batch job's output stays on the\nnode it ran on.", Nested: "config.SharedStorage"},
|
||||
{Key: "users", Type: "string", Doc: "Users is who keeps the Linux users that run jobs, the same on every\nmachine of the Slurm cluster: \"llmgw\" makes one for each active llmgw\naccount (its login from the e-mail address, a UID of llmgw's from\n200000 up, its SSH keys from the portal), let in to the controller,\nand to a node only while a job of theirs runs there (pam_slurm_adopt);\nempty leaves users to the site (a directory through sssd, or by hand).", Nested: ""},
|
||||
{Key: "containers", Type: "bool", Doc: "Containers lets jobs run in containers: `srun --container-image\nnvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),\nthe GPUs and the shared path inside. The agents install both on\nevery Slurm machine; jobs take it once every machine has them.", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "StrataConfig", Title: "StrataConfig", Doc: "", Fields: []Field{
|
||||
{Key: "dir", Type: "string", Doc: "Dir is Strata's source directory, where setup.py and serve/ are.", Nested: ""},
|
||||
{Key: "python", Type: "string", Doc: "Python is the interpreter of its environment; Dir's .venv when unset.", Nested: ""},
|
||||
|
||||
@@ -41,6 +41,7 @@ var Sources = []string{
|
||||
"internal/config/serving.go",
|
||||
"internal/config/presets.go",
|
||||
"internal/config/ondemand.go",
|
||||
"internal/config/slurm.go",
|
||||
"internal/config/agents.go",
|
||||
"internal/mail/mail.go",
|
||||
"internal/notify/notify.go",
|
||||
|
||||
@@ -48,6 +48,22 @@ type Header struct {
|
||||
// (<arch>.pooling_type set and not none): an embedding model, which
|
||||
// llama.cpp serves with --embedding and does not chat.
|
||||
Embedding bool
|
||||
// ExpertCount and ExpertUsedCount are a model of experts' experts and
|
||||
// how many each token uses (<arch>.expert_count, expert_used_count);
|
||||
// zero for a dense model.
|
||||
ExpertCount, ExpertUsedCount uint64
|
||||
}
|
||||
|
||||
// ActiveShare is the part of the weights a token reads: for a model of
|
||||
// experts its experts' share plus a tenth for what every token reads
|
||||
// whatever the experts — attention, embeddings, routing, shared experts.
|
||||
// Qwen3-Coder-30B-A3B, eight of 128, read 0.16 of its weights a token on
|
||||
// cuda01's card, which is 0.0625 and the tenth. 1 for a dense model.
|
||||
func (h Header) ActiveShare() float64 {
|
||||
if h.ExpertCount == 0 || h.ExpertUsedCount == 0 || h.ExpertUsedCount >= h.ExpertCount {
|
||||
return 1
|
||||
}
|
||||
return min(1, float64(h.ExpertUsedCount)/float64(h.ExpertCount)+0.1)
|
||||
}
|
||||
|
||||
// KVBytesPerToken is what one token of context costs in llama.cpp's default
|
||||
@@ -276,6 +292,10 @@ func Read(r io.Reader) (Header, error) {
|
||||
h.Embedding = num > 0
|
||||
case "context_length":
|
||||
h.ContextLength = num
|
||||
case "expert_count":
|
||||
h.ExpertCount = num
|
||||
case "expert_used_count":
|
||||
h.ExpertUsedCount = num
|
||||
}
|
||||
}
|
||||
// The tokenizer's arrays — megabytes of vocabulary — come after the
|
||||
|
||||
@@ -210,3 +210,14 @@ func TestAPoolIsSizedToTheCard(t *testing.T) {
|
||||
t.Error("class memory misread")
|
||||
}
|
||||
}
|
||||
|
||||
// A model of experts reads its experts' share a token and what every token
|
||||
// reads besides; a dense one reads it all.
|
||||
func TestAModelOfExpertsReadsAShareOfItsWeights(t *testing.T) {
|
||||
if got := (Header{ExpertCount: 128, ExpertUsedCount: 8}).ActiveShare(); got < 0.16 || got > 0.17 {
|
||||
t.Errorf("Qwen3-30B-A3B's shape reads %.3f of its weights a token, measured 0.16", got)
|
||||
}
|
||||
if (Header{}).ActiveShare() != 1 || (Header{ExpertCount: 8, ExpertUsedCount: 8}).ActiveShare() != 1 {
|
||||
t.Error("a dense model reads all its weights")
|
||||
}
|
||||
}
|
||||
Loaded 100 of 170 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user