Compare commits
25
Commits
No files matched your search
@@ -11,3 +11,4 @@ coverage.out
|
||||
/terraform-provider-llmgw/examples/terraform.tfstate*
|
||||
# What a development server run from the checkout writes beside it.
|
||||
/secrets/
|
||||
linkedin/
|
||||
+114
-5
@@ -463,8 +463,9 @@ portal is not done — the page shows; the console and ctl change.
|
||||
5. **Done (branch `features-c`):** `notify.webhooks` (model parked, node
|
||||
lost, budget spent, update available; signed JSON or Slack; retries,
|
||||
hourly de-duplication), `ctl notify test`, Alertmanager receiver
|
||||
templates. Left: `server init` and the console do not yet ask for a
|
||||
receiver — the templates are filled in by hand. Events are claimed in
|
||||
templates. `server init` and both setup screens ask for webhooks and test
|
||||
each (d1e9186, 2026-10-14); with none set, `ctl status` and the console's
|
||||
Overview say the alerts go nowhere (2026-11-12). Events are claimed in
|
||||
the store before they are sent (2026-10-10), so two control planes send
|
||||
each once.
|
||||
**Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
|
||||
@@ -2027,7 +2028,7 @@ lent, partition`); the leader tells each named node's agent (root) what to be, t
|
||||
- Phases: P1 the `slurm` package (conf, key, parsers) and settings; P2 the agent side (proto, install, munge,
|
||||
slurmd/slurmctld, status); P3 the leader side (conf from reports, lend/reclaim); P4 `ctl slurm`
|
||||
(status, lend, reclaim) and a TUI screen; P5 `ctl train import` of a Slurm job's output into the mirror.
|
||||
Later: a backup slurmctld on a second node, accounting (slurmdbd on the Patroni PG), users from LDAP/AD.
|
||||
Later: a backup slurmctld on a second node, accounting (slurmdbd; it takes MySQL/MariaDB only, not the Patroni PG), users from LDAP/AD.
|
||||
- Tried on the PoC 2026-11-11 (dev build v0.16.3-17-g849760c-slurm on 140, 150, node05, node06; Ubuntu
|
||||
24.04, Slurm 23.11.4 from the archive): `ctl slurm setup --controller llmgw-node06 --nodes llmgw-node05`
|
||||
installed munge, slurmctld and slurmd, node05 entered drained with its slurmd -C hardware (8 CPUs,
|
||||
@@ -2037,14 +2038,122 @@ lent, partition`); the leader tells each named node's agent (root) what to be, t
|
||||
from the daemon); the packaged unit is Type=notify (slurmd needs --systemd); slurmd's own "slurmstepd
|
||||
infinity" was counted as a job; without accounting storage 23.11's main scheduler marks batch jobs
|
||||
InvalidAccount and backfill starts them, one a pass (bf_interval=10: 6-10 s each). So many small jobs
|
||||
want slurmdbd. Not tried yet: a GPU node (gres), a leader failover while lent.
|
||||
want slurmdbd. — Wrong cause, found 2026-11-11: a job asking no memory got the whole node's (ReqTRES
|
||||
mem=all), so a node ran one job at a time and each next one waited for backfill's pass. Fixed: the
|
||||
partition's DefMemPerCPU is the leanest node's share; slurmdbd is not needed for this. InvalidAccount is real too:
|
||||
23.11 dropped accounting_storage/none, and without slurmdbd the main scheduler passes every job by;
|
||||
backfill starts it within bf_interval (10 s). priority/basic did not change that (fair share needs
|
||||
slurmdbd anyway). Starting on submission needs slurmdbd, on MySQL/MariaDB.
|
||||
- slurmdbd, done 2026-11-12 (748f777): the controller's agent installs MariaDB (bound to 127.0.0.1; a
|
||||
MariaDB that was there keeps its settings), makes `slurm_acct_db` and its user (password from the munge
|
||||
key, through a 0600 file, never argv), runs slurmdbd (6819, in the firewall for the cluster) and adds
|
||||
the cluster, before slurmctld. slurm.conf: AccountingStorageType=slurmdbd, jobacct_gather/cgroup,
|
||||
AccountingStorageTRES=gres/gpu, priority/multifactor. Tried on the PoC: five jobs started on
|
||||
submission, `sacct -a` has them with their TRES. Not tried: a fresh `server init --slurm` (needs VMs);
|
||||
a Red Hat controller.
|
||||
- Fresh install tried 2026-11-12 on VMs (3 control planes with Patroni, 2 CPU nodes, `server init
|
||||
--slurm`): Slurm, slurmdbd and MariaDB came up with the install; two-node srun/sbatch on the lent nodes,
|
||||
`sacct` has them; leader failover kept serving. Open from it: `--slurm` needs the first machine to be a
|
||||
node (the controller is its node), so a site wanting control planes without nodes chooses the
|
||||
controller later with `ctl slurm setup`; batch output lands on the job's first node until there is
|
||||
shared storage; on shared CPUs the judge refuses the first request after a placement change (fail
|
||||
closed, ~10 s), and its advice repeats `--cores 4` to a judge that has 4. Tried again on the idle
|
||||
VMs: one judge restarted in place, then both at once, requests every second: 48 of 48 served; the
|
||||
first after a cold start took 7.6–8.9 s end to end against ~3 s, within the 5 s judge budget. So the
|
||||
refusal was the install's load on the CPUs; if it is seen again, warm a new judge with one judgment
|
||||
before it takes requests.
|
||||
- Shared files and users, built 2026-11-12: `slurm.shared` {path, nfs | server} — the site's export
|
||||
mounted (a systemd mount unit, hard, nconnect=4), an llmgw node exporting the path itself
|
||||
(/etc/exports.d, to the Slurm machines' addresses, root_squash; 2049 in the firewall), or the site's
|
||||
own mount only looked for; `slurm.users: llmgw` — every active account a Linux user everywhere
|
||||
(`slurm_users`: login from the e-mail, UID from 200000, never reused; homes made on the exporting
|
||||
machine, else the controller; keys in /etc/llmgw/ssh-keys/%u through an sshd drop-in), SSH keys
|
||||
from the portal's Profile or `ctl slurm users keys`. A user already on a machine with another UID
|
||||
is left alone and said. The site has AD (not yet live): users from it, through sssd, are the next
|
||||
step (`slurm.users` empty today leaves them to the site).
|
||||
- Logging in to a node, done 2026-11-14: llmgw's users reach the controller freely, another Slurm
|
||||
machine only while a job of theirs runs there (pam_slurm_adopt, installed by the agent; PrologFlags=
|
||||
Contain). Only the users listed in the agent's users file are held: root and the machine's own users
|
||||
log in as before, so a wrong line never locks out who would fix it. pam_systemd would move the session
|
||||
out of the job, so sshd's stack takes the session file in place of its include, with pam_systemd
|
||||
skipped for the listed users alone; where that cannot be done safely (nested includes, jumps across
|
||||
it) they are held to having a job, not to its cores and cards, and the agent says so. Tried on the
|
||||
VMs: ayse without a job refused ("no active jobs on this node"), with one in
|
||||
job_N/step_extern (nproc 1), cut off when it ended; mehmet refused; admin's session unchanged.
|
||||
Left: sites whose users come from AD (sssd) are not held yet; a group of theirs would be the list.
|
||||
- A GPU node, tried 2026-11-14 on the PoC (cuda01, Quadro RTX 6000, Slurm 23.11.4): gres.conf
|
||||
`Type=rtx6000 File=/dev/nvidia0`, slurm.conf `Gres=gpu:rtx6000:1`. `srun --gres=gpu:1`:
|
||||
CUDA_VISIBLE_DEVICES=0, the one card; torch found it and ran 14.1 TFLOPS (fp32 matmul). Without
|
||||
--gres: "No devices found", torch none (ConstrainDevices holds). `--gres=gpu:2`: refused. sacct has
|
||||
`gres/gpu=1` in AllocTRES. Found: lending a node whose GPU model has nowhere else to go waits out
|
||||
the drain grace (5 min) before the replica stops, and `ctl slurm status` says only "1 replica(s)
|
||||
still to move off"; it should say it waits for a replacement that cannot be placed, and until when.
|
||||
Done (c3a75dc): it says the time.
|
||||
- Jobs in containers, done 2026-11-14: `slurm.containers` (`ctl slurm containers on`). Each agent
|
||||
installs enroot 4.2.1 (NVIDIA's .deb/.rpm, sha256 pinned) and builds pyxis 0.24.0 (source tarball,
|
||||
sha256 pinned) against its own Slurm (gcc, make, libslurm-dev); on a machine with nvidia-smi,
|
||||
nvidia-container-cli from NVIDIA's repository, its key's digest pinned. enroot.conf keeps each
|
||||
user's runtime, cache and data under /run/enroot and /var/lib/enroot (no login session needed), the
|
||||
shared path bind-mounted in every container. The leader writes plugstack.conf ("required") only
|
||||
once every machine reports both, since a machine without the plugin would fail every job. Tried on
|
||||
the VMs: on in ~30 s; ayse's srun --container-image=ubuntu:24.04 on n1 ran as uid 200001 with
|
||||
/shared inside and nproc 1; a two-node sbatch ran a rank on each. Seen once: a node's pull timed out
|
||||
reaching auth.docker.io and the job failed (network, not pyxis). Not tried: GPUs in a container
|
||||
(cuda01's agent is an older build; needs the peer protocol), Red Hat.
|
||||
- AD end to end, tried 2026-11-14 (decided: one AD per install, the existing auth.ldap group mapping, no
|
||||
tree from AD, no sssd unless a site asks): the VMs' auth.ldap against Samba DC VM 165 (ldaps by IP,
|
||||
CA in the settings, ml-team -> team:Test/arge). ada signed in on the portal with her AD password: an
|
||||
account, in Test/arge, Slurm user ada UID 200003; her key added in the portal's Profile, ssh to the
|
||||
controller ~20 s later, sbatch ran on n1 as ada. Disabled in AD: suspended in 56 s, the Linux user
|
||||
gone in 89 s, ssh refused, home kept. Enabled: back in 32 s, same UID, key kept. Taken out of
|
||||
ml-team: out of the team in 11 s; put back: in again in 56 s.
|
||||
- P5, replaced 2026-11-17 by the portal (decided 2026-11-15: users of every level, build for the one who
|
||||
knows least first; any project member may start, no approval, one per project, the project's by default):
|
||||
**Customise a model** (`/train`): an xlsx/CSV/JSONL of questions and answers, checked and previewed; base
|
||||
weights from the model's origin (a GGUF repo's card base_model), memory from their size (2x + 1 GiB),
|
||||
the node with the most free; published by itself when done (llama.cpp replicas replaced to load it);
|
||||
shared me/project/team/org/everyone (wider than the project only confirmed, never wider than its
|
||||
knowledge bases), deleted from the page. Left: pairs generated from a knowledge base on the page, a
|
||||
base-vs-new comparison, the training's GPU time charged to the project, big jobs through Slurm, the
|
||||
server's own error messages in Turkish.
|
||||
- P6, done 2026-11-15 but for budgets: each project a Slurm account (org and team accounts above it,
|
||||
ids keyed by kind: the first install's org, team and project share the id "default"), its members and its
|
||||
team's and org's on it; AccountingStorageEnforce=associations; the controller's agent keeps them with
|
||||
sacctmgr (only what it made; a user's default account moved before an association goes, the user taken out
|
||||
when none is left) and reports the jobs that ended in the last day; the leader records each once
|
||||
(slurm_jobs, keyed cluster/id) with a usage row, model slurm/PARTITION, priced at gpu_hour_cost or
|
||||
core_hour_cost. Tried on the VMs: accounts made, ayse's job on her project's account ran and another
|
||||
project's was refused, both jobs in deneme's usage at 345 and 0 micro-units. Left: budgets (a job's cost
|
||||
is not counted against a budget, which binds keys; GrpTRESMins from a project's budget would hold its jobs).
|
||||
- P6 as written 2026-11-12: The tenancy tree to Slurm:
|
||||
each project an account (`sacctmgr add account`, parent its team and organization), each member a user
|
||||
of it, kept by the leader from the store; `sbatch --account=PROJECT`. The records back: the leader reads
|
||||
`sacct --json` for jobs ended since the last read (through the controller's agent) into a
|
||||
`batch_usage` table (job, user, project, start, end, GPU-seconds by class, CPU-seconds, memory), priced
|
||||
per GPU-hour and core-hour (auth.gpu_hour_cost, core_hour_cost: the same basis `ctl prices suggest`
|
||||
prices tokens from, 2026-11-15) into cost_micros, summed with the requests' usage on the usage pages and in
|
||||
budgets. Budgets enforced: a project's remaining budget as GPU-minutes in its account's GrpTRESMins, so
|
||||
Slurm holds its new jobs once spent. Needs the same users on every node (open: LDAP/sssd or
|
||||
llmgw-made). Also found: slurm.conf
|
||||
wrote every node State=DRAIN, so each slurmctld start (on any change of the files) drained a lent node
|
||||
until its agent resumed it, up to a minute; a lent node is now written without it. Not tried yet: a GPU node (gres).
|
||||
- Failover while lent, tried 2026-11-11 (v0.16.3-23): the first try killed the job (NODE_FAIL). A new
|
||||
leader had heard from no node yet and sent the controller a slurm.conf without them; slurmctld,
|
||||
restarted on it, dropped node05 and its job. Fixed (141a1f4, 0774e90): slurm.conf's nodes and the
|
||||
controller's address are kept in cluster_state (`slurm-nodes`) and a new leader starts from them; with
|
||||
nothing kept (an upgraded cluster) or the store unread, the controller's files wait up to 2 min for the
|
||||
nodes; a node is told nothing until its controller's address is known (no part stops its slurmd). Tried
|
||||
again: a 5-minute job on lent node05, leader 140 stopped, 150 took over; slurmctld not restarted, the job
|
||||
ran to its end; reclaimed during it, node05 was llmgw's within seconds of its end. Also 3387f97: srun in
|
||||
SrunPortRange=60001-63000, and Slurm's ports in llmgw's firewall for the cluster's networks.
|
||||
- Built 2026-11-11: P1 (`internal/slurm`, `slurm.*` settings), P2
|
||||
(`internal/agent/slurmnode`, proto Slurm/SlurmStatus), P3 (`internal/control/slurmsvc`, wired in the
|
||||
leader; `GET /api/v1/slurm`), P4's commands (`ctl slurm status|setup|lend|reclaim|off`). Decisions on the
|
||||
way: a node not heard from or with Slurm jobs is cordoned, not drained (nothing of llmgw's to move, and a
|
||||
new leader must not move models off every Slurm node); jobs are counted from the node's own slurmstepd,
|
||||
so a controller that is down neither hides them nor, with none, keeps the node from llmgw; a drain whose
|
||||
reason does not start with "llmgw" is a Slurm administrator's and never resumed. Left: the TUI screen, P5,
|
||||
reason does not start with "llmgw" is a Slurm administrator's and never resumed. Left: ~~the TUI screen~~
|
||||
(done 2026-11-11: console **Slurm**, lend/take back on a row), P5,
|
||||
a try on the PoC (needs agents built from this; slurm-wlm 23.11 on Ubuntu 24.04), GPU minor numbers are
|
||||
taken to be nvidia-smi's index.
|
||||
- In the first install too (2026-11-11): `llmgw server init --slurm`, and a Slurm part on the choices
|
||||
|
||||
+72
-4
@@ -20,16 +20,43 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
|
||||
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
|
||||
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node; v0.16.3: replicas per profile, a served model's settings on the mirror page, the CPU's speed before serving) |
|
||||
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node; v0.16.3: replicas per profile, a served model's settings on the mirror page, the CPU's speed before serving; v0.16.4: Slurm beside llmgw, a model customised from the portal, a GPU node without a driver reported) |
|
||||
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
||||
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
## v0.16.4 — 2026-11-19
|
||||
|
||||
Slurm beside llmgw for training and batch work — its users, shared files, containers, accounting priced in llmgw's
|
||||
books — a model customised from the portal with a file and a button, an organization's admin, and a GPU node whose
|
||||
driver was never installed told apart from a CPU node.
|
||||
|
||||
**Upgrading.**
|
||||
- Four migrations (0052–0055: Slurm's users and jobs, held capacity, the portal's training jobs), applied by the
|
||||
first control plane to start on this version; nothing to do by hand. `llmgw ctl upgrade v0.16.4`: the control
|
||||
planes carry Slurm, the training page, the admin role and the prices; the agents carry Slurm's node part, the
|
||||
containers, and the reading of the PCI bus for a card without a driver, without which such a node still joins
|
||||
as a CPU node. Built with Go 1.27, the newest patch at build time.
|
||||
|
||||
### Added
|
||||
|
||||
- **Customise a model from the portal.** A new page, **Customise a model**
|
||||
(`/train`), lets anyone in a project teach a model how to answer without a
|
||||
terminal: an Excel, CSV or JSONL file of questions and answers (columns
|
||||
headed Soru/Cevap or Question/Answer, else the first two), checked and
|
||||
shown before anything trains; a model; a name; who may use it; and a
|
||||
button. The weights to train on are found from where the model's came (a
|
||||
GGUF repository's base model on Hugging Face), the card's memory from their
|
||||
size, the node as the one with the most free; when it is done the adapter
|
||||
is published by itself and appears as a model of its own in the chat and
|
||||
the API, a llama.cpp model's replicas replaced one at a time to load it.
|
||||
It is the project's by default; its members may share it with the person,
|
||||
the team, the organization or everyone (wider than the project only once
|
||||
they confirm the file holds nothing that group must not see, and never
|
||||
wider than the knowledge bases its pairs came from), and delete it. One
|
||||
trains at a time per project. `llmgw ctl train start` needs only `--model`
|
||||
and `--data` now, and `--publish` publishes when done.
|
||||
- **An organization's admin.** Besides member and manager, a membership of
|
||||
an organization may now be `admin` (`llmgw users join … --org Acme --role
|
||||
admin`, the role chosen on a membership on the people page, the console,
|
||||
@@ -48,8 +75,41 @@ APIs change between minors without a deprecation period.
|
||||
and gres.conf written from what each node reports. Every node stays llmgw's,
|
||||
drained in Slurm, until `ctl slurm lend NODE`: its replicas move off, then
|
||||
Slurm runs jobs there. `ctl slurm reclaim NODE` takes it back once its jobs
|
||||
end. `ctl slurm status` shows each node. Slurm's ports (6817–6818, srun's
|
||||
60001–63000) answer the cluster's networks only, as llmgw's own do.
|
||||
end. `ctl slurm status` and the console's **Slurm** show each node. The
|
||||
controller also keeps the jobs' records: slurmdbd, in a MariaDB on that
|
||||
machine alone (Slurm takes MySQL or MariaDB only). Slurm's ports
|
||||
(6817–6819, srun's 60001–63000) answer the cluster's networks only, as
|
||||
llmgw's own do.
|
||||
- **Slurm's shared files and users.** `slurm.shared` (`ctl slurm shared`) is
|
||||
one path on every Slurm machine: the site's NFS export mounted, an llmgw
|
||||
node exporting it over NFS (what `server init --slurm` chooses, `/shared`
|
||||
from the first machine), or a file system the site mounts. `slurm.users:
|
||||
llmgw` (`ctl slurm users on`) makes each active account a Linux user on
|
||||
every Slurm machine, the same login and UID (from 200000, never reused)
|
||||
everywhere, its home on the shared path; people add SSH keys in the
|
||||
portal's Profile (`ctl slurm users keys` for an operator). They log in to
|
||||
the controller; to another Slurm machine only while a job of theirs runs
|
||||
there, their session put in the job (pam_slurm_adopt), so nobody uses a
|
||||
node's cards outside Slurm, nor on a node llmgw serves from. Root and the
|
||||
machines' own users are not held to it.
|
||||
- **Slurm's jobs in llmgw's accounting.** With `slurm.users: llmgw` each
|
||||
project is a Slurm account (organizations and teams above it), its members
|
||||
and its team's and organization's on it, and a job must run on one of its
|
||||
user's (`AccountingStorageEnforce=associations`). Each job that ends is
|
||||
recorded once, its GPU-hours (or core-hours) priced at `auth.gpu_hour_cost`
|
||||
into its project's usage as model `slurm/PARTITION`. Accounts a Slurm
|
||||
administrator made are not touched.
|
||||
- **Prices from what the cluster costs.** `auth.gpu_hour_cost` (and
|
||||
`core_hour_cost`) is what an hour of a GPU (a core) costs; the leader keeps
|
||||
the hours each model's replicas hold, and `llmgw ctl prices suggest` prices
|
||||
each model from them over the tokens it served, an output token four input
|
||||
ones and a cached one a tenth. `--apply` writes the prices.
|
||||
- **Slurm's jobs in containers.** `llmgw ctl slurm containers on`
|
||||
(`slurm.containers`) installs enroot and pyxis on every Slurm machine, at
|
||||
pinned versions checked against their digests, and nvidia-container-cli on
|
||||
GPU machines; Slurm loads pyxis once every machine has it, and then `srun
|
||||
--container-image=...` runs a job in that image, as its user, with the
|
||||
shared path inside.
|
||||
|
||||
### Changed
|
||||
|
||||
@@ -64,6 +124,14 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **A GPU node with no driver joined quietly as a CPU node.** The agent found
|
||||
GPUs through `nvidia-smi` alone, so a machine whose driver was never
|
||||
installed looked exactly like one with no card. The PCI bus is read as
|
||||
well: a card there with no driver fails the checks before the agent is
|
||||
installed, with the install and the reboot named (and a driver installed
|
||||
but not loaded — Secure Boot, nouveau — told apart from a missing one), and
|
||||
a node already in shows the warning in `ctl nodes` and the console until
|
||||
the driver is in and its agent restarted.
|
||||
- **The serve forms start at the longest context that fits.** The mirror
|
||||
page's form began every GGUF model at 4096 tokens, and the console's at
|
||||
4096 or the plan's figure, whatever the model was trained for. Both now
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
# Functional Source License, Version 1.1, ALv2 Future License
|
||||
|
||||
## Abbreviation
|
||||
|
||||
FSL-1.1-ALv2
|
||||
|
||||
## Notice
|
||||
|
||||
Copyright 2026 Ilker Manap
|
||||
|
||||
## Terms and Conditions
|
||||
|
||||
### Licensor ("We")
|
||||
|
||||
The party offering the Software under these Terms and Conditions.
|
||||
|
||||
### The Software
|
||||
|
||||
The "Software" is each version of the software that we make available under
|
||||
these Terms and Conditions, as indicated by our inclusion of these Terms and
|
||||
Conditions with the Software.
|
||||
|
||||
### License Grant
|
||||
|
||||
Subject to your compliance with this License Grant and the Patents,
|
||||
Redistribution and Trademark clauses below, we hereby grant you the right to
|
||||
use, copy, modify, create derivative works, publicly perform, publicly display
|
||||
and redistribute the Software for any Permitted Purpose identified below.
|
||||
|
||||
### Permitted Purpose
|
||||
|
||||
A Permitted Purpose is any purpose other than a Competing Use. A Competing Use
|
||||
means making the Software available to others in a commercial product or
|
||||
service that:
|
||||
|
||||
1. substitutes for the Software;
|
||||
|
||||
2. substitutes for any other product or service we offer using the Software
|
||||
that exists as of the date we make the Software available; or
|
||||
|
||||
3. offers the same or substantially similar functionality as the Software.
|
||||
|
||||
Permitted Purposes specifically include using the Software:
|
||||
|
||||
1. for your internal use and access;
|
||||
|
||||
2. for non-commercial education;
|
||||
|
||||
3. for non-commercial research; and
|
||||
|
||||
4. in connection with professional services that you provide to a licensee
|
||||
using the Software in accordance with these Terms and Conditions.
|
||||
|
||||
### Patents
|
||||
|
||||
To the extent your use for a Permitted Purpose would necessarily infringe our
|
||||
patents, the license grant above includes a license under our patents. If you
|
||||
make a claim against any party that the Software infringes or contributes to
|
||||
the infringement of any patent, then your patent license to the Software ends
|
||||
immediately.
|
||||
|
||||
### Redistribution
|
||||
|
||||
The Terms and Conditions apply to all copies, modifications and derivatives of
|
||||
the Software.
|
||||
|
||||
If you redistribute any copies, modifications or derivatives of the Software,
|
||||
you must include a copy of or a link to these Terms and Conditions and not
|
||||
remove any copyright notices provided in or with the Software.
|
||||
|
||||
### Disclaimer
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTIES OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING WITHOUT LIMITATION WARRANTIES OF FITNESS FOR A PARTICULAR
|
||||
PURPOSE, MERCHANTABILITY, TITLE OR NON-INFRINGEMENT.
|
||||
|
||||
IN NO EVENT WILL WE HAVE ANY LIABILITY TO YOU ARISING OUT OF OR RELATED TO THE
|
||||
SOFTWARE, INCLUDING INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES,
|
||||
EVEN IF WE HAVE BEEN INFORMED OF THEIR POSSIBILITY IN ADVANCE.
|
||||
|
||||
### Trademarks
|
||||
|
||||
Except for displaying the License Details and identifying us as the origin of
|
||||
the Software, you have no right under these Terms and Conditions to use our
|
||||
trademarks, trade names, service marks or product names.
|
||||
|
||||
## Grant of Future License
|
||||
|
||||
We hereby irrevocably grant you an additional license to use the Software under
|
||||
the Apache License, Version 2.0 that is effective on the second anniversary of
|
||||
the date we make the Software available. On or after that date, you may use the
|
||||
Software under the Apache License, Version 2.0, in which case the following
|
||||
will apply:
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use
|
||||
this file except in compliance with the License.
|
||||
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed
|
||||
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
|
||||
CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
@@ -42,7 +42,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
|
||||
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
|
||||
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node; v0.16.3: replicas per profile, a served model's settings on the mirror page, the CPU's speed before serving) |
|
||||
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node; v0.16.3: replicas per profile, a served model's settings on the mirror page, the CPU's speed before serving; v0.16.4: Slurm beside llmgw, a model customised from the portal, a GPU node without a driver reported) |
|
||||
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
||||
|
||||
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
|
||||
@@ -372,3 +372,10 @@ make rename NEW=git.example.com/team/llmgw
|
||||
## Language
|
||||
|
||||
Code, comments, tests, commit messages and documentation are in English.
|
||||
|
||||
## Licence
|
||||
|
||||
Functional Source License 1.1 with an Apache 2.0 future licence
|
||||
(FSL-1.1-ALv2); see [LICENSE](LICENSE). Each release may be used, read and
|
||||
changed for anything except offering it as a competing product or service,
|
||||
and becomes Apache 2.0 two years after it is published.
|
||||
+560
-44
@@ -1246,8 +1246,12 @@ type Host struct {
|
||||
// given a few threads of a large machine reads at a few times this, not at
|
||||
// the whole machine's rate. 0 unknown.
|
||||
MemBandwidthCoreGbs float64 `protobuf:"fixed64,8,opt,name=mem_bandwidth_core_gbs,json=memBandwidthCoreGbs,proto3" json:"mem_bandwidth_core_gbs,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
// nvidia_device is whether an NVIDIA display controller sits on the PCI
|
||||
// bus, driver or not. With nvidia_driver empty it is a card nothing can
|
||||
// use yet, which must be said rather than counted as no card at all.
|
||||
NvidiaDevice bool `protobuf:"varint,9,opt,name=nvidia_device,json=nvidiaDevice,proto3" json:"nvidia_device,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *Host) Reset() {
|
||||
@@ -1336,6 +1340,13 @@ func (x *Host) GetMemBandwidthCoreGbs() float64 {
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *Host) GetNvidiaDevice() bool {
|
||||
if x != nil {
|
||||
return x.NvidiaDevice
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Storage is one filesystem a node keeps weights on.
|
||||
//
|
||||
// Per filesystem rather than per configured directory: free space belongs to
|
||||
@@ -5304,9 +5315,27 @@ type Slurm struct {
|
||||
Files map[string]string `protobuf:"bytes,6,rep,name=files,proto3" json:"files,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
|
||||
// lent says Slurm may run jobs here now: resume the node if llmgw drained
|
||||
// it. Otherwise it is drained, its jobs left to finish.
|
||||
Lent bool `protobuf:"varint,7,opt,name=lent,proto3" json:"lent,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
Lent bool `protobuf:"varint,7,opt,name=lent,proto3" json:"lent,omitempty"`
|
||||
// shared is the file system every machine of the Slurm cluster sees at
|
||||
// the same path; unset or with no path, none.
|
||||
Shared *SlurmShared `protobuf:"bytes,8,opt,name=shared,proto3" json:"shared,omitempty"`
|
||||
// manage_users has llmgw keep the Linux users below on this machine,
|
||||
// and take away those it made that are no longer among them.
|
||||
ManageUsers bool `protobuf:"varint,9,opt,name=manage_users,json=manageUsers,proto3" json:"manage_users,omitempty"`
|
||||
Users []*SlurmUser `protobuf:"bytes,10,rep,name=users,proto3" json:"users,omitempty"`
|
||||
// make_homes has this machine make the users' home directories: the one
|
||||
// that exports the shared path, else the controller.
|
||||
MakeHomes bool `protobuf:"varint,11,opt,name=make_homes,json=makeHomes,proto3" json:"make_homes,omitempty"`
|
||||
// containers has the machine install pyxis and enroot, so jobs may run
|
||||
// in containers (srun --container-image), and set them up.
|
||||
Containers bool `protobuf:"varint,12,opt,name=containers,proto3" json:"containers,omitempty"`
|
||||
// manage_accounts has the controller keep Slurm's accounts as accounts
|
||||
// says: one for each organization, team and project, nested, the users
|
||||
// on each project's. Only those llmgw made are changed or taken away.
|
||||
ManageAccounts bool `protobuf:"varint,13,opt,name=manage_accounts,json=manageAccounts,proto3" json:"manage_accounts,omitempty"`
|
||||
Accounts []*SlurmAccount `protobuf:"bytes,14,rep,name=accounts,proto3" json:"accounts,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *Slurm) Reset() {
|
||||
@@ -5388,6 +5417,388 @@ func (x *Slurm) GetLent() bool {
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Slurm) GetShared() *SlurmShared {
|
||||
if x != nil {
|
||||
return x.Shared
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *Slurm) GetManageUsers() bool {
|
||||
if x != nil {
|
||||
return x.ManageUsers
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Slurm) GetUsers() []*SlurmUser {
|
||||
if x != nil {
|
||||
return x.Users
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *Slurm) GetMakeHomes() bool {
|
||||
if x != nil {
|
||||
return x.MakeHomes
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Slurm) GetContainers() bool {
|
||||
if x != nil {
|
||||
return x.Containers
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Slurm) GetManageAccounts() bool {
|
||||
if x != nil {
|
||||
return x.ManageAccounts
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Slurm) GetAccounts() []*SlurmAccount {
|
||||
if x != nil {
|
||||
return x.Accounts
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SlurmAccount is an account in Slurm's accounting, parents before their
|
||||
// children.
|
||||
type SlurmAccount struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
|
||||
Parent string `protobuf:"bytes,2,opt,name=parent,proto3" json:"parent,omitempty"`
|
||||
Users []string `protobuf:"bytes,3,rep,name=users,proto3" json:"users,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *SlurmAccount) Reset() {
|
||||
*x = SlurmAccount{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *SlurmAccount) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*SlurmAccount) ProtoMessage() {}
|
||||
|
||||
func (x *SlurmAccount) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use SlurmAccount.ProtoReflect.Descriptor instead.
|
||||
func (*SlurmAccount) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{49}
|
||||
}
|
||||
|
||||
func (x *SlurmAccount) GetName() string {
|
||||
if x != nil {
|
||||
return x.Name
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmAccount) GetParent() string {
|
||||
if x != nil {
|
||||
return x.Parent
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmAccount) GetUsers() []string {
|
||||
if x != nil {
|
||||
return x.Users
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SlurmJob is a job that ended, as sacct has it, for llmgw's accounting.
|
||||
type SlurmJob struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"`
|
||||
User string `protobuf:"bytes,2,opt,name=user,proto3" json:"user,omitempty"`
|
||||
Account string `protobuf:"bytes,3,opt,name=account,proto3" json:"account,omitempty"`
|
||||
Partition string `protobuf:"bytes,4,opt,name=partition,proto3" json:"partition,omitempty"`
|
||||
State string `protobuf:"bytes,5,opt,name=state,proto3" json:"state,omitempty"`
|
||||
StartUnix int64 `protobuf:"varint,6,opt,name=start_unix,json=startUnix,proto3" json:"start_unix,omitempty"`
|
||||
EndUnix int64 `protobuf:"varint,7,opt,name=end_unix,json=endUnix,proto3" json:"end_unix,omitempty"`
|
||||
ElapsedSeconds int64 `protobuf:"varint,8,opt,name=elapsed_seconds,json=elapsedSeconds,proto3" json:"elapsed_seconds,omitempty"`
|
||||
Cpus int32 `protobuf:"varint,9,opt,name=cpus,proto3" json:"cpus,omitempty"`
|
||||
Gpus int32 `protobuf:"varint,10,opt,name=gpus,proto3" json:"gpus,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *SlurmJob) Reset() {
|
||||
*x = SlurmJob{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[50]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *SlurmJob) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*SlurmJob) ProtoMessage() {}
|
||||
|
||||
func (x *SlurmJob) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[50]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use SlurmJob.ProtoReflect.Descriptor instead.
|
||||
func (*SlurmJob) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{50}
|
||||
}
|
||||
|
||||
func (x *SlurmJob) GetId() string {
|
||||
if x != nil {
|
||||
return x.Id
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmJob) GetUser() string {
|
||||
if x != nil {
|
||||
return x.User
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmJob) GetAccount() string {
|
||||
if x != nil {
|
||||
return x.Account
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmJob) GetPartition() string {
|
||||
if x != nil {
|
||||
return x.Partition
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmJob) GetState() string {
|
||||
if x != nil {
|
||||
return x.State
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmJob) GetStartUnix() int64 {
|
||||
if x != nil {
|
||||
return x.StartUnix
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *SlurmJob) GetEndUnix() int64 {
|
||||
if x != nil {
|
||||
return x.EndUnix
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *SlurmJob) GetElapsedSeconds() int64 {
|
||||
if x != nil {
|
||||
return x.ElapsedSeconds
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *SlurmJob) GetCpus() int32 {
|
||||
if x != nil {
|
||||
return x.Cpus
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *SlurmJob) GetGpus() int32 {
|
||||
if x != nil {
|
||||
return x.Gpus
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// SlurmShared is where the Slurm cluster's shared files come from.
|
||||
type SlurmShared struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
// path is where every machine sees them, as /shared.
|
||||
Path string `protobuf:"bytes,1,opt,name=path,proto3" json:"path,omitempty"`
|
||||
// nfs is an export, host:/path, to mount at path; empty when this
|
||||
// machine exports path itself (serve) or the site mounts it.
|
||||
Nfs string `protobuf:"bytes,2,opt,name=nfs,proto3" json:"nfs,omitempty"`
|
||||
// serve has this machine export path over NFS to clients.
|
||||
Serve bool `protobuf:"varint,3,opt,name=serve,proto3" json:"serve,omitempty"`
|
||||
// clients are the addresses the export answers.
|
||||
Clients []string `protobuf:"bytes,4,rep,name=clients,proto3" json:"clients,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *SlurmShared) Reset() {
|
||||
*x = SlurmShared{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[51]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *SlurmShared) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*SlurmShared) ProtoMessage() {}
|
||||
|
||||
func (x *SlurmShared) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[51]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use SlurmShared.ProtoReflect.Descriptor instead.
|
||||
func (*SlurmShared) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{51}
|
||||
}
|
||||
|
||||
func (x *SlurmShared) GetPath() string {
|
||||
if x != nil {
|
||||
return x.Path
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmShared) GetNfs() string {
|
||||
if x != nil {
|
||||
return x.Nfs
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmShared) GetServe() bool {
|
||||
if x != nil {
|
||||
return x.Serve
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *SlurmShared) GetClients() []string {
|
||||
if x != nil {
|
||||
return x.Clients
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SlurmUser is a Linux user that runs jobs, the same on every machine.
|
||||
type SlurmUser struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Login string `protobuf:"bytes,1,opt,name=login,proto3" json:"login,omitempty"`
|
||||
// uid is also the user's own group's gid.
|
||||
Uid uint32 `protobuf:"varint,2,opt,name=uid,proto3" json:"uid,omitempty"`
|
||||
// name is the account's, for the GECOS field.
|
||||
Name string `protobuf:"bytes,3,opt,name=name,proto3" json:"name,omitempty"`
|
||||
Home string `protobuf:"bytes,4,opt,name=home,proto3" json:"home,omitempty"`
|
||||
// ssh_keys are OpenSSH public keys, one a line of authorized_keys.
|
||||
SshKeys []string `protobuf:"bytes,5,rep,name=ssh_keys,json=sshKeys,proto3" json:"ssh_keys,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *SlurmUser) Reset() {
|
||||
*x = SlurmUser{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[52]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *SlurmUser) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*SlurmUser) ProtoMessage() {}
|
||||
|
||||
func (x *SlurmUser) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[52]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use SlurmUser.ProtoReflect.Descriptor instead.
|
||||
func (*SlurmUser) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{52}
|
||||
}
|
||||
|
||||
func (x *SlurmUser) GetLogin() string {
|
||||
if x != nil {
|
||||
return x.Login
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmUser) GetUid() uint32 {
|
||||
if x != nil {
|
||||
return x.Uid
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *SlurmUser) GetName() string {
|
||||
if x != nil {
|
||||
return x.Name
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmUser) GetHome() string {
|
||||
if x != nil {
|
||||
return x.Home
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmUser) GetSshKeys() []string {
|
||||
if x != nil {
|
||||
return x.SshKeys
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SlurmStatus is what this node's Slurm daemons are doing, sent when it
|
||||
// changes and every few minutes while the node has a part in Slurm.
|
||||
type SlurmStatus struct {
|
||||
@@ -5410,14 +5821,28 @@ type SlurmStatus struct {
|
||||
Error string `protobuf:"bytes,8,opt,name=error,proto3" json:"error,omitempty"`
|
||||
// files_sha256 is of the controller's files as written, so the leader
|
||||
// knows they arrived.
|
||||
FilesSha256 string `protobuf:"bytes,9,opt,name=files_sha256,json=filesSha256,proto3" json:"files_sha256,omitempty"`
|
||||
FilesSha256 string `protobuf:"bytes,9,opt,name=files_sha256,json=filesSha256,proto3" json:"files_sha256,omitempty"`
|
||||
// slurmdbd_running is the accounting daemon on the controller, which keeps
|
||||
// the jobs' records in a MariaDB beside it.
|
||||
SlurmdbdRunning bool `protobuf:"varint,10,opt,name=slurmdbd_running,json=slurmdbdRunning,proto3" json:"slurmdbd_running,omitempty"`
|
||||
// shared is the shared path as this machine has it: "mounted",
|
||||
// "exported", "missing", or empty when there is none.
|
||||
Shared string `protobuf:"bytes,11,opt,name=shared,proto3" json:"shared,omitempty"`
|
||||
// users is how many of llmgw's users this machine has.
|
||||
Users int32 `protobuf:"varint,12,opt,name=users,proto3" json:"users,omitempty"`
|
||||
// containers is what this machine has for jobs in containers, as
|
||||
// "pyxis 0.24.0, enroot 4.2.1"; empty while it has not got both.
|
||||
Containers string `protobuf:"bytes,13,opt,name=containers,proto3" json:"containers,omitempty"`
|
||||
// ended are, on the controller, the jobs that ended in the last day; the
|
||||
// leader records each once.
|
||||
Ended []*SlurmJob `protobuf:"bytes,14,rep,name=ended,proto3" json:"ended,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) Reset() {
|
||||
*x = SlurmStatus{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[53]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
@@ -5429,7 +5854,7 @@ func (x *SlurmStatus) String() string {
|
||||
func (*SlurmStatus) ProtoMessage() {}
|
||||
|
||||
func (x *SlurmStatus) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[53]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
@@ -5442,7 +5867,7 @@ func (x *SlurmStatus) ProtoReflect() protoreflect.Message {
|
||||
|
||||
// Deprecated: Use SlurmStatus.ProtoReflect.Descriptor instead.
|
||||
func (*SlurmStatus) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{49}
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{53}
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetVersion() string {
|
||||
@@ -5508,6 +5933,41 @@ func (x *SlurmStatus) GetFilesSha256() string {
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetSlurmdbdRunning() bool {
|
||||
if x != nil {
|
||||
return x.SlurmdbdRunning
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetShared() string {
|
||||
if x != nil {
|
||||
return x.Shared
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetUsers() int32 {
|
||||
if x != nil {
|
||||
return x.Users
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetContainers() string {
|
||||
if x != nil {
|
||||
return x.Containers
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetEnded() []*SlurmJob {
|
||||
if x != nil {
|
||||
return x.Ended
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
var File_llmgw_v1_agent_proto protoreflect.FileDescriptor
|
||||
|
||||
const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
@@ -5603,7 +6063,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\x1aA\n" +
|
||||
"\x13EngineVersionsEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\x92\x02\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\xb7\x02\n" +
|
||||
"\x04Host\x12\x13\n" +
|
||||
"\x05os_id\x18\x01 \x01(\tR\x04osId\x12\x1d\n" +
|
||||
"\n" +
|
||||
@@ -5614,7 +6074,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\tcpu_flags\x18\x05 \x03(\tR\bcpuFlags\x12#\n" +
|
||||
"\rnvidia_driver\x18\x06 \x01(\tR\fnvidiaDriver\x12*\n" +
|
||||
"\x11mem_bandwidth_gbs\x18\a \x01(\x01R\x0fmemBandwidthGbs\x123\n" +
|
||||
"\x16mem_bandwidth_core_gbs\x18\b \x01(\x01R\x13memBandwidthCoreGbs\"u\n" +
|
||||
"\x16mem_bandwidth_core_gbs\x18\b \x01(\x01R\x13memBandwidthCoreGbs\x12#\n" +
|
||||
"\rnvidia_device\x18\t \x01(\bR\fnvidiaDevice\"u\n" +
|
||||
"\aStorage\x12\x12\n" +
|
||||
"\x04path\x18\x01 \x01(\tR\x04path\x12\x1d\n" +
|
||||
"\n" +
|
||||
@@ -6013,7 +6474,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x04kind\x18\x02 \x01(\tR\x04kind\x12\x10\n" +
|
||||
"\x03seq\x18\x03 \x01(\x05R\x03seq\x12\x12\n" +
|
||||
"\x04data\x18\x04 \x01(\fR\x04data\x12\x12\n" +
|
||||
"\x04last\x18\x05 \x01(\bR\x04last\"\x8d\x02\n" +
|
||||
"\x04last\x18\x05 \x01(\bR\x04last\"\xa6\x04\n" +
|
||||
"\x05Slurm\x12\x1e\n" +
|
||||
"\n" +
|
||||
"controller\x18\x01 \x01(\bR\n" +
|
||||
@@ -6024,11 +6485,50 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"confServer\x12\x1b\n" +
|
||||
"\tmunge_key\x18\x05 \x01(\fR\bmungeKey\x120\n" +
|
||||
"\x05files\x18\x06 \x03(\v2\x1a.llmgw.v1.Slurm.FilesEntryR\x05files\x12\x12\n" +
|
||||
"\x04lent\x18\a \x01(\bR\x04lent\x1a8\n" +
|
||||
"\x04lent\x18\a \x01(\bR\x04lent\x12-\n" +
|
||||
"\x06shared\x18\b \x01(\v2\x15.llmgw.v1.SlurmSharedR\x06shared\x12!\n" +
|
||||
"\fmanage_users\x18\t \x01(\bR\vmanageUsers\x12)\n" +
|
||||
"\x05users\x18\n" +
|
||||
" \x03(\v2\x13.llmgw.v1.SlurmUserR\x05users\x12\x1d\n" +
|
||||
"\n" +
|
||||
"make_homes\x18\v \x01(\bR\tmakeHomes\x12\x1e\n" +
|
||||
"\n" +
|
||||
"containers\x18\f \x01(\bR\n" +
|
||||
"containers\x12'\n" +
|
||||
"\x0fmanage_accounts\x18\r \x01(\bR\x0emanageAccounts\x122\n" +
|
||||
"\baccounts\x18\x0e \x03(\v2\x16.llmgw.v1.SlurmAccountR\baccounts\x1a8\n" +
|
||||
"\n" +
|
||||
"FilesEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\x92\x02\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"P\n" +
|
||||
"\fSlurmAccount\x12\x12\n" +
|
||||
"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
|
||||
"\x06parent\x18\x02 \x01(\tR\x06parent\x12\x14\n" +
|
||||
"\x05users\x18\x03 \x03(\tR\x05users\"\x87\x02\n" +
|
||||
"\bSlurmJob\x12\x0e\n" +
|
||||
"\x02id\x18\x01 \x01(\tR\x02id\x12\x12\n" +
|
||||
"\x04user\x18\x02 \x01(\tR\x04user\x12\x18\n" +
|
||||
"\aaccount\x18\x03 \x01(\tR\aaccount\x12\x1c\n" +
|
||||
"\tpartition\x18\x04 \x01(\tR\tpartition\x12\x14\n" +
|
||||
"\x05state\x18\x05 \x01(\tR\x05state\x12\x1d\n" +
|
||||
"\n" +
|
||||
"start_unix\x18\x06 \x01(\x03R\tstartUnix\x12\x19\n" +
|
||||
"\bend_unix\x18\a \x01(\x03R\aendUnix\x12'\n" +
|
||||
"\x0felapsed_seconds\x18\b \x01(\x03R\x0eelapsedSeconds\x12\x12\n" +
|
||||
"\x04cpus\x18\t \x01(\x05R\x04cpus\x12\x12\n" +
|
||||
"\x04gpus\x18\n" +
|
||||
" \x01(\x05R\x04gpus\"c\n" +
|
||||
"\vSlurmShared\x12\x12\n" +
|
||||
"\x04path\x18\x01 \x01(\tR\x04path\x12\x10\n" +
|
||||
"\x03nfs\x18\x02 \x01(\tR\x03nfs\x12\x14\n" +
|
||||
"\x05serve\x18\x03 \x01(\bR\x05serve\x12\x18\n" +
|
||||
"\aclients\x18\x04 \x03(\tR\aclients\"v\n" +
|
||||
"\tSlurmUser\x12\x14\n" +
|
||||
"\x05login\x18\x01 \x01(\tR\x05login\x12\x10\n" +
|
||||
"\x03uid\x18\x02 \x01(\rR\x03uid\x12\x12\n" +
|
||||
"\x04name\x18\x03 \x01(\tR\x04name\x12\x12\n" +
|
||||
"\x04home\x18\x04 \x01(\tR\x04home\x12\x19\n" +
|
||||
"\bssh_keys\x18\x05 \x03(\tR\asshKeys\"\xb5\x03\n" +
|
||||
"\vSlurmStatus\x12\x18\n" +
|
||||
"\aversion\x18\x01 \x01(\tR\aversion\x12\x1a\n" +
|
||||
"\bhardware\x18\x02 \x01(\tR\bhardware\x12%\n" +
|
||||
@@ -6038,7 +6538,15 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x06reason\x18\x06 \x01(\tR\x06reason\x12\x12\n" +
|
||||
"\x04jobs\x18\a \x01(\x05R\x04jobs\x12\x14\n" +
|
||||
"\x05error\x18\b \x01(\tR\x05error\x12!\n" +
|
||||
"\ffiles_sha256\x18\t \x01(\tR\vfilesSha2562O\n" +
|
||||
"\ffiles_sha256\x18\t \x01(\tR\vfilesSha256\x12)\n" +
|
||||
"\x10slurmdbd_running\x18\n" +
|
||||
" \x01(\bR\x0fslurmdbdRunning\x12\x16\n" +
|
||||
"\x06shared\x18\v \x01(\tR\x06shared\x12\x14\n" +
|
||||
"\x05users\x18\f \x01(\x05R\x05users\x12\x1e\n" +
|
||||
"\n" +
|
||||
"containers\x18\r \x01(\tR\n" +
|
||||
"containers\x12(\n" +
|
||||
"\x05ended\x18\x0e \x03(\v2\x12.llmgw.v1.SlurmJobR\x05ended2O\n" +
|
||||
"\fAgentService\x12?\n" +
|
||||
"\aConnect\x12\x16.llmgw.v1.AgentMessage\x1a\x18.llmgw.v1.ControlMessage(\x010\x01B\x93\x01\n" +
|
||||
"\fcom.llmgw.v1B\n" +
|
||||
@@ -6057,7 +6565,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
|
||||
}
|
||||
|
||||
var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
|
||||
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 59)
|
||||
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 63)
|
||||
var file_llmgw_v1_agent_proto_goTypes = []any{
|
||||
(ShardStatus_State)(0), // 0: llmgw.v1.ShardStatus.State
|
||||
(*AgentMessage)(nil), // 1: llmgw.v1.AgentMessage
|
||||
@@ -6109,16 +6617,20 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
|
||||
(*TrainStatus)(nil), // 47: llmgw.v1.TrainStatus
|
||||
(*TrainArtifact)(nil), // 48: llmgw.v1.TrainArtifact
|
||||
(*Slurm)(nil), // 49: llmgw.v1.Slurm
|
||||
(*SlurmStatus)(nil), // 50: llmgw.v1.SlurmStatus
|
||||
nil, // 51: llmgw.v1.ControlMessage.TraceEntry
|
||||
nil, // 52: llmgw.v1.Register.LabelsEntry
|
||||
nil, // 53: llmgw.v1.Register.EngineVersionsEntry
|
||||
nil, // 54: llmgw.v1.GPU.NicPathsEntry
|
||||
nil, // 55: llmgw.v1.IBPort.CountersEntry
|
||||
nil, // 56: llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
nil, // 57: llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
nil, // 58: llmgw.v1.EngineProfile.EnvEntry
|
||||
nil, // 59: llmgw.v1.Slurm.FilesEntry
|
||||
(*SlurmAccount)(nil), // 50: llmgw.v1.SlurmAccount
|
||||
(*SlurmJob)(nil), // 51: llmgw.v1.SlurmJob
|
||||
(*SlurmShared)(nil), // 52: llmgw.v1.SlurmShared
|
||||
(*SlurmUser)(nil), // 53: llmgw.v1.SlurmUser
|
||||
(*SlurmStatus)(nil), // 54: llmgw.v1.SlurmStatus
|
||||
nil, // 55: llmgw.v1.ControlMessage.TraceEntry
|
||||
nil, // 56: llmgw.v1.Register.LabelsEntry
|
||||
nil, // 57: llmgw.v1.Register.EngineVersionsEntry
|
||||
nil, // 58: llmgw.v1.GPU.NicPathsEntry
|
||||
nil, // 59: llmgw.v1.IBPort.CountersEntry
|
||||
nil, // 60: llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
nil, // 61: llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
nil, // 62: llmgw.v1.EngineProfile.EnvEntry
|
||||
nil, // 63: llmgw.v1.Slurm.FilesEntry
|
||||
}
|
||||
var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
8, // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
|
||||
@@ -6131,7 +6643,7 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
2, // 7: llmgw.v1.AgentMessage.kv_events:type_name -> llmgw.v1.KVEvents
|
||||
47, // 8: llmgw.v1.AgentMessage.train_status:type_name -> llmgw.v1.TrainStatus
|
||||
48, // 9: llmgw.v1.AgentMessage.train_artifact:type_name -> llmgw.v1.TrainArtifact
|
||||
50, // 10: llmgw.v1.AgentMessage.slurm_status:type_name -> llmgw.v1.SlurmStatus
|
||||
54, // 10: llmgw.v1.AgentMessage.slurm_status:type_name -> llmgw.v1.SlurmStatus
|
||||
30, // 11: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
|
||||
33, // 12: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
|
||||
34, // 13: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
|
||||
@@ -6149,11 +6661,11 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
45, // 25: llmgw.v1.ControlMessage.train_start:type_name -> llmgw.v1.TrainStart
|
||||
46, // 26: llmgw.v1.ControlMessage.train_cancel:type_name -> llmgw.v1.TrainCancel
|
||||
49, // 27: llmgw.v1.ControlMessage.slurm:type_name -> llmgw.v1.Slurm
|
||||
51, // 28: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
|
||||
52, // 29: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
|
||||
55, // 28: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
|
||||
56, // 29: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
|
||||
12, // 30: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
|
||||
14, // 31: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
53, // 32: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
|
||||
57, // 32: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
|
||||
11, // 33: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
|
||||
15, // 34: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
|
||||
10, // 35: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
|
||||
@@ -6161,15 +6673,15 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
21, // 37: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
|
||||
0, // 38: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
27, // 39: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
|
||||
54, // 40: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
|
||||
58, // 40: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
|
||||
13, // 41: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
|
||||
18, // 42: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
|
||||
17, // 43: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
|
||||
55, // 44: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
|
||||
59, // 44: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
|
||||
20, // 45: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
|
||||
10, // 46: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
|
||||
15, // 47: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
|
||||
56, // 48: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
60, // 48: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
26, // 49: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
|
||||
16, // 50: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
|
||||
19, // 51: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
|
||||
@@ -6177,21 +6689,25 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
22, // 53: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
|
||||
21, // 54: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
|
||||
25, // 55: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
|
||||
57, // 56: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
61, // 56: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
0, // 57: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
29, // 58: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
|
||||
27, // 59: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
|
||||
31, // 60: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
|
||||
58, // 61: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
|
||||
62, // 61: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
|
||||
44, // 62: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
|
||||
59, // 63: llmgw.v1.Slurm.files:type_name -> llmgw.v1.Slurm.FilesEntry
|
||||
1, // 64: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
|
||||
3, // 65: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
|
||||
65, // [65:66] is the sub-list for method output_type
|
||||
64, // [64:65] is the sub-list for method input_type
|
||||
64, // [64:64] is the sub-list for extension type_name
|
||||
64, // [64:64] is the sub-list for extension extendee
|
||||
0, // [0:64] is the sub-list for field type_name
|
||||
63, // 63: llmgw.v1.Slurm.files:type_name -> llmgw.v1.Slurm.FilesEntry
|
||||
52, // 64: llmgw.v1.Slurm.shared:type_name -> llmgw.v1.SlurmShared
|
||||
53, // 65: llmgw.v1.Slurm.users:type_name -> llmgw.v1.SlurmUser
|
||||
50, // 66: llmgw.v1.Slurm.accounts:type_name -> llmgw.v1.SlurmAccount
|
||||
51, // 67: llmgw.v1.SlurmStatus.ended:type_name -> llmgw.v1.SlurmJob
|
||||
1, // 68: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
|
||||
3, // 69: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
|
||||
69, // [69:70] is the sub-list for method output_type
|
||||
68, // [68:69] is the sub-list for method input_type
|
||||
68, // [68:68] is the sub-list for extension type_name
|
||||
68, // [68:68] is the sub-list for extension extendee
|
||||
0, // [0:68] is the sub-list for field type_name
|
||||
}
|
||||
|
||||
func init() { file_llmgw_v1_agent_proto_init() }
|
||||
@@ -6237,7 +6753,7 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
|
||||
RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
|
||||
NumEnums: 1,
|
||||
NumMessages: 59,
|
||||
NumMessages: 63,
|
||||
NumExtensions: 0,
|
||||
NumServices: 1,
|
||||
},
|
||||
|
||||
@@ -247,6 +247,10 @@ message Host {
|
||||
// given a few threads of a large machine reads at a few times this, not at
|
||||
// the whole machine's rate. 0 unknown.
|
||||
double mem_bandwidth_core_gbs = 8;
|
||||
// nvidia_device is whether an NVIDIA display controller sits on the PCI
|
||||
// bus, driver or not. With nvidia_driver empty it is a card nothing can
|
||||
// use yet, which must be said rather than counted as no card at all.
|
||||
bool nvidia_device = 9;
|
||||
}
|
||||
|
||||
// Storage is one filesystem a node keeps weights on.
|
||||
@@ -1010,6 +1014,71 @@ message Slurm {
|
||||
// lent says Slurm may run jobs here now: resume the node if llmgw drained
|
||||
// it. Otherwise it is drained, its jobs left to finish.
|
||||
bool lent = 7;
|
||||
// shared is the file system every machine of the Slurm cluster sees at
|
||||
// the same path; unset or with no path, none.
|
||||
SlurmShared shared = 8;
|
||||
// manage_users has llmgw keep the Linux users below on this machine,
|
||||
// and take away those it made that are no longer among them.
|
||||
bool manage_users = 9;
|
||||
repeated SlurmUser users = 10;
|
||||
// make_homes has this machine make the users' home directories: the one
|
||||
// that exports the shared path, else the controller.
|
||||
bool make_homes = 11;
|
||||
// containers has the machine install pyxis and enroot, so jobs may run
|
||||
// in containers (srun --container-image), and set them up.
|
||||
bool containers = 12;
|
||||
// manage_accounts has the controller keep Slurm's accounts as accounts
|
||||
// says: one for each organization, team and project, nested, the users
|
||||
// on each project's. Only those llmgw made are changed or taken away.
|
||||
bool manage_accounts = 13;
|
||||
repeated SlurmAccount accounts = 14;
|
||||
}
|
||||
|
||||
// SlurmAccount is an account in Slurm's accounting, parents before their
|
||||
// children.
|
||||
message SlurmAccount {
|
||||
string name = 1;
|
||||
string parent = 2;
|
||||
repeated string users = 3;
|
||||
}
|
||||
|
||||
// SlurmJob is a job that ended, as sacct has it, for llmgw's accounting.
|
||||
message SlurmJob {
|
||||
string id = 1;
|
||||
string user = 2;
|
||||
string account = 3;
|
||||
string partition = 4;
|
||||
string state = 5;
|
||||
int64 start_unix = 6;
|
||||
int64 end_unix = 7;
|
||||
int64 elapsed_seconds = 8;
|
||||
int32 cpus = 9;
|
||||
int32 gpus = 10;
|
||||
}
|
||||
|
||||
// SlurmShared is where the Slurm cluster's shared files come from.
|
||||
message SlurmShared {
|
||||
// path is where every machine sees them, as /shared.
|
||||
string path = 1;
|
||||
// nfs is an export, host:/path, to mount at path; empty when this
|
||||
// machine exports path itself (serve) or the site mounts it.
|
||||
string nfs = 2;
|
||||
// serve has this machine export path over NFS to clients.
|
||||
bool serve = 3;
|
||||
// clients are the addresses the export answers.
|
||||
repeated string clients = 4;
|
||||
}
|
||||
|
||||
// SlurmUser is a Linux user that runs jobs, the same on every machine.
|
||||
message SlurmUser {
|
||||
string login = 1;
|
||||
// uid is also the user's own group's gid.
|
||||
uint32 uid = 2;
|
||||
// name is the account's, for the GECOS field.
|
||||
string name = 3;
|
||||
string home = 4;
|
||||
// ssh_keys are OpenSSH public keys, one a line of authorized_keys.
|
||||
repeated string ssh_keys = 5;
|
||||
}
|
||||
|
||||
// SlurmStatus is what this node's Slurm daemons are doing, sent when it
|
||||
@@ -1034,4 +1103,18 @@ message SlurmStatus {
|
||||
// files_sha256 is of the controller's files as written, so the leader
|
||||
// knows they arrived.
|
||||
string files_sha256 = 9;
|
||||
// slurmdbd_running is the accounting daemon on the controller, which keeps
|
||||
// the jobs' records in a MariaDB beside it.
|
||||
bool slurmdbd_running = 10;
|
||||
// shared is the shared path as this machine has it: "mounted",
|
||||
// "exported", "missing", or empty when there is none.
|
||||
string shared = 11;
|
||||
// users is how many of llmgw's users this machine has.
|
||||
int32 users = 12;
|
||||
// containers is what this machine has for jobs in containers, as
|
||||
// "pyxis 0.24.0, enroot 4.2.1"; empty while it has not got both.
|
||||
string containers = 13;
|
||||
// ended are, on the controller, the jobs that ended in the last day; the
|
||||
// leader records each once.
|
||||
repeated SlurmJob ended = 14;
|
||||
}
|
||||
@@ -16,7 +16,7 @@
|
||||
# machine, manages its GPUs and supervises processes there as root, which is
|
||||
# what a container is built to prevent: docs/kubernetes.md.
|
||||
|
||||
FROM golang:1.26 AS build
|
||||
FROM golang:1.27 AS build
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
|
||||
@@ -95,6 +95,9 @@
|
||||
{
|
||||
"name": "presets"
|
||||
},
|
||||
{
|
||||
"name": "prices"
|
||||
},
|
||||
{
|
||||
"name": "projects"
|
||||
},
|
||||
@@ -4396,6 +4399,34 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/prices/suggest": {
|
||||
"get": {
|
||||
"operationId": "getPricesSuggest",
|
||||
"summary": "Each model's price, from what it held of the cluster",
|
||||
"description": "days (30 if not given) is the window. A model's cost is the GPU- and core-hours its replicas held, idle ones too, at auth.gpu_hour_cost and auth.core_hour_cost; spread over its tokens, an output token four input ones and a cached prompt token a tenth of one, it is price, in micro-units a thousand tokens as auth.pricing takes them. current is the price in force. 409 with no cost set.",
|
||||
"tags": [
|
||||
"prices"
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/PriceSuggestions"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/projects": {
|
||||
"post": {
|
||||
"operationId": "postProjects",
|
||||
@@ -5109,6 +5140,89 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/slurm/users": {
|
||||
"get": {
|
||||
"operationId": "getSlurmUsers",
|
||||
"summary": "The Linux users made from llmgw's accounts for the Slurm cluster",
|
||||
"description": "Every active account, with its login, UID and SSH keys; an account is given its login and UID the first time it is listed. The machines have them when slurm.users is \"llmgw\".",
|
||||
"tags": [
|
||||
"slurm"
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/slurmsvc.User"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/slurm/users/{user}/ssh-keys": {
|
||||
"put": {
|
||||
"operationId": "putSlurmUsersUserSshKeys",
|
||||
"summary": "Replace an account's SSH public keys",
|
||||
"description": "{user} is an ID or e-mail. keys is OpenSSH public keys, one a line, without options; empty takes them all away. The machines have them within a minute.",
|
||||
"tags": [
|
||||
"slurm"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "user",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "The person's ID or email.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/SlurmKeysRequest"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/slurmsvc.User"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/summary": {
|
||||
"get": {
|
||||
"operationId": "getSummary",
|
||||
@@ -8763,6 +8877,73 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"PriceSuggestion": {
|
||||
"type": "object",
|
||||
"description": "PriceSuggestion is one model's.",
|
||||
"properties": {
|
||||
"model": {
|
||||
"type": "string"
|
||||
},
|
||||
"gpu_hours": {
|
||||
"type": "number"
|
||||
},
|
||||
"core_hours": {
|
||||
"type": "number"
|
||||
},
|
||||
"cost": {
|
||||
"type": "number",
|
||||
"description": "Cost is what it held cost, in budget units."
|
||||
},
|
||||
"prompt_tokens": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"completion_tokens": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"price": {
|
||||
"$ref": "#/components/schemas/accounting.Price",
|
||||
"description": "Price is the suggestion; zero with a Note saying why there is none."
|
||||
},
|
||||
"note": {
|
||||
"type": "string"
|
||||
},
|
||||
"current": {
|
||||
"$ref": "#/components/schemas/accounting.Price"
|
||||
}
|
||||
}
|
||||
},
|
||||
"PriceSuggestions": {
|
||||
"type": "object",
|
||||
"description": "PriceSuggestions is a model's price from its cost, for each model.",
|
||||
"properties": {
|
||||
"days": {
|
||||
"type": "integer",
|
||||
"description": "Days is the window; RecordedFrom the first hour held is known for, later than the window's start on a cluster that began keeping it since, which makes the window shorter than asked."
|
||||
},
|
||||
"since": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"recorded_from": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"gpu_hour_cost": {
|
||||
"type": "number"
|
||||
},
|
||||
"core_hour_cost": {
|
||||
"type": "number"
|
||||
},
|
||||
"models": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/PriceSuggestion"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"ProfileReplicasRequest": {
|
||||
"type": "object",
|
||||
"description": "ProfileReplicasRequest counts a model's replicas per profile: profile name to replicas, and CPU, for a llama.cpp model with no profile on CPUs, the one to add.",
|
||||
@@ -9401,6 +9582,15 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"SlurmKeysRequest": {
|
||||
"type": "object",
|
||||
"description": "SlurmKeysRequest is a person's SSH public keys, one a line, as authorized_keys has them; empty takes them all away.",
|
||||
"properties": {
|
||||
"keys": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"StoreMember": {
|
||||
"type": "object",
|
||||
"description": "StoreMember is one member of a replicated store, as its Patroni reports it.",
|
||||
@@ -9503,6 +9693,10 @@
|
||||
"guard_off": {
|
||||
"type": "boolean",
|
||||
"description": "GuardOff says the guardrails are turned off (guard.enabled: false): no request is checked, whatever the policies say."
|
||||
},
|
||||
"no_alerts": {
|
||||
"type": "boolean",
|
||||
"description": "NoAlerts says the cluster's settings name no webhook: a model parked, a node lost or a judge down is told nobody."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -9911,6 +10105,35 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"accounting.Price": {
|
||||
"type": "object",
|
||||
"description": "Price is what one model costs, in micro-units per thousand tokens.\n\nMicro-units rather than a float because spend is accumulated by repeated addition, and repeated addition of floats drifts. What a unit means is the operator's: self-hosted models have no per-token price, so it may be GPU-seconds, a notional rate, or 1000 everywhere — which turns a budget into a token quota.",
|
||||
"properties": {
|
||||
"PromptPer1k": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"CompletionPer1k": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"CachedPer1k": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
"description": "CachedPer1k is what a prompt token costs that the engine or provider read from its cache instead of computing: the part of prompt_tokens the answer counts as cached_tokens. Zero charges those at PromptPer1k, as before there was a rate for them."
|
||||
},
|
||||
"PerImage": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
"description": "PerImage is what one picture costs, for a model that makes them: a picture's cost is the card's time, which its prompt's few tokens do not measure."
|
||||
},
|
||||
"PerMesh": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
"description": "PerMesh is what one 3D model costs: minutes of a card, which no token count measures."
|
||||
}
|
||||
}
|
||||
},
|
||||
"accounting.Rates": {
|
||||
"type": "object",
|
||||
"description": "Rates are an accountant's two limiters' counts.",
|
||||
@@ -10785,6 +11008,24 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"config.SharedStorage": {
|
||||
"type": "object",
|
||||
"description": "SharedStorage is where the Slurm cluster's shared files come from: the site's NFS export (NFS), an NFS export llmgw keeps on one of its nodes (Server), or, with neither, a file system the site mounts itself (Lustre, BeeGFS, GPFS), which llmgw only looks for.",
|
||||
"properties": {
|
||||
"Path": {
|
||||
"type": "string",
|
||||
"description": "Path is where every machine sees it, as /shared."
|
||||
},
|
||||
"NFS": {
|
||||
"type": "string",
|
||||
"description": "NFS is the site's export, host:/path, mounted at Path on every machine of the Slurm cluster."
|
||||
},
|
||||
"Server": {
|
||||
"type": "string",
|
||||
"description": "Server is the llmgw node, the controller or one of the nodes, that exports Path over NFS to the others: for a site with no storage of its own. One machine: not highly available, and no faster than its disk and network."
|
||||
}
|
||||
}
|
||||
},
|
||||
"control.Activity": {
|
||||
"type": "object",
|
||||
"description": "Activity is what a standby's router reports back: when it last sent each replica a request, and the on-demand models it was asked for. The leader evicts and loads by these, and without them a replica busy behind a standby looks idle to the one machine that decides.",
|
||||
@@ -12222,6 +12463,11 @@
|
||||
"replicas": {
|
||||
"type": "integer"
|
||||
},
|
||||
"stops_by": {
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"description": "StopsBy is, while it is being lent, when a replica still on it is stopped even with nowhere else to go: the drain's grace after the lend began. Zero when not known (a leader that took over meanwhile)."
|
||||
},
|
||||
"slurm": {
|
||||
"$ref": "#/components/schemas/slurmsvc.Reported"
|
||||
},
|
||||
@@ -12246,6 +12492,21 @@
|
||||
"slurmctld_running": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"slurmdbd_running": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"shared": {
|
||||
"type": "string",
|
||||
"description": "Shared is the shared path here: \"mounted\", \"exported\", \"missing\"."
|
||||
},
|
||||
"containers": {
|
||||
"type": "string",
|
||||
"description": "Containers is what the machine has for jobs in containers, empty while not both of pyxis and enroot."
|
||||
},
|
||||
"users": {
|
||||
"type": "integer",
|
||||
"description": "Users is how many of llmgw's users the machine has."
|
||||
},
|
||||
"state": {
|
||||
"type": "string"
|
||||
},
|
||||
@@ -12298,6 +12559,48 @@
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Waiting are nodes left out of slurm.conf until they report their hardware."
|
||||
},
|
||||
"shared": {
|
||||
"$ref": "#/components/schemas/config.SharedStorage",
|
||||
"description": "Shared is where the shared files are and come from; Users who keeps the users, and how many llmgw has."
|
||||
},
|
||||
"users": {
|
||||
"type": "string"
|
||||
},
|
||||
"user_count": {
|
||||
"type": "integer"
|
||||
},
|
||||
"containers": {
|
||||
"type": "string",
|
||||
"description": "Containers is whether jobs may run in containers: \"on\" once every machine has pyxis and enroot, \"installing\" until then, empty when slurm.containers is off."
|
||||
},
|
||||
"accounts": {
|
||||
"type": "integer",
|
||||
"description": "Accounts is how many of Slurm's accounts are llmgw's projects'."
|
||||
}
|
||||
}
|
||||
},
|
||||
"slurmsvc.User": {
|
||||
"type": "object",
|
||||
"description": "User is a Linux user made from an llmgw account.",
|
||||
"properties": {
|
||||
"login": {
|
||||
"type": "string"
|
||||
},
|
||||
"uid": {
|
||||
"type": "integer"
|
||||
},
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"email": {
|
||||
"type": "string"
|
||||
},
|
||||
"ssh_keys": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
+15
-1
@@ -67,7 +67,8 @@ or with the mirror unticked, is said in the form and nothing starts.
|
||||
## Watch
|
||||
|
||||
**Overview** is `llmgw ctl status`: nodes, GPUs by class, fabrics, replicas,
|
||||
models, the store's members, and anything parked, unplaced or idle.
|
||||
models, the store's members, and anything parked, unplaced or idle. With no
|
||||
webhook in the cluster's settings it says the alerts go nowhere.
|
||||
|
||||
**Monitor** draws the cluster the way btop draws a machine: CPU, memory,
|
||||
GPU busy and GPU memory, requests and tokens a second, time to first token,
|
||||
@@ -207,6 +208,19 @@ starting — and whether it ended done, failed, or waiting for room on busy
|
||||
cards. The entry shows how many are running. Enter on one shows its steps.
|
||||
`llmgw ctl jobs` is the same, and `llmgw ctl jobs ID` follows one.
|
||||
|
||||
**Slurm** is the Slurm cluster run beside llmgw, when there is one: its
|
||||
controller and whether slurmctld runs, and each node that runs slurmd —
|
||||
whose it is now (llmgw's, Slurm's, or on its way between them), its state in
|
||||
Slurm, the jobs running there. On a row, **Lend to Slurm…** moves the node's
|
||||
replicas off and then lets Slurm run jobs there; **Take back from Slurm…**
|
||||
starts no new job there and gives it back to llmgw when its jobs have ended.
|
||||
Both change `slurm.lent` in the cluster's settings, as `llmgw ctl slurm lend`
|
||||
and `reclaim` do. Turning Slurm on and choosing its nodes is `llmgw ctl slurm
|
||||
setup`, or the Slurm part of the first install. Above the nodes it says where
|
||||
the shared files come from and how many users llmgw made; the SHARED column
|
||||
is each node's (mounted, exported, missing). `llmgw ctl slurm shared` and
|
||||
`llmgw ctl slurm users` change them.
|
||||
|
||||
**Logs** shows a node's agent log, newest at the bottom, errors and warnings
|
||||
coloured.
|
||||
|
||||
|
||||
+1
-1
@@ -27,7 +27,7 @@ make test
|
||||
./bin/llmgw --version
|
||||
```
|
||||
|
||||
`make build` writes `./bin/llmgw`. Go 1.26.6 or later is the only requirement (an older Go fetches it itself).
|
||||
`make build` writes `./bin/llmgw`. Go 1.27 or later is the only requirement (an older Go fetches it itself); the patch level is never pinned, so CI and the containers build with the newest 1.27.
|
||||
|
||||
## 2. Check the path with a fake engine
|
||||
|
||||
|
||||
+77
-4
@@ -177,17 +177,74 @@ init again to serve on it.
|
||||
choices screen, runs a Slurm cluster beside llmgw: slurmctld on this
|
||||
machine's node, slurmd on every node, those that join later too
|
||||
(`slurm.all_nodes`). The agents install munge and Slurm (22.05 or newer)
|
||||
from the machines' own package sources. No node is Slurm's at first: each is
|
||||
from the machines' own package sources, and on this machine slurmdbd, which
|
||||
keeps the jobs' records (`sacct`), in a MariaDB installed beside it that
|
||||
answers this machine alone. Slurm keeps its records in MySQL or MariaDB
|
||||
only, not in the cluster's PostgreSQL; the database's password is made from
|
||||
the cluster's munge key, so nothing new is stored. A MariaDB the machine
|
||||
already had keeps its settings and only gets the database `slurm_acct_db`
|
||||
and its user. Back it up with the machine (`mariadb-dump slurm_acct_db`);
|
||||
records older than a year are purged. No node is Slurm's at first: each is
|
||||
drained in Slurm and serves models until `llmgw ctl slurm lend NODE` gives it
|
||||
to Slurm — its models move off first — and `llmgw ctl slurm reclaim NODE`
|
||||
takes it back once its jobs end. Jobs are submitted on this machine with
|
||||
`sbatch` and `srun`; `llmgw ctl slurm status` shows where each node is. It
|
||||
needs this machine to be a node, and its hostname to be a name Slurm takes.
|
||||
On a cluster installed without it, `llmgw ctl slurm setup --controller NODE
|
||||
--all-nodes` does the same later. Slurm's ports (6817–6818, and srun's 60001–63000) answer the cluster's
|
||||
--all-nodes` does the same later. Slurm's ports (6817–6819, and srun's 60001–63000) answer the cluster's
|
||||
networks only, as llmgw's own do; a firewall of the site's own between the
|
||||
nodes must let them through.
|
||||
|
||||
**Slurm's shared files and users.** Jobs need the same files on every
|
||||
machine: homes, datasets, output, checkpoints. With `--slurm` this machine
|
||||
exports `/shared` over NFS to every Slurm machine and they mount it
|
||||
(`slurm.shared`); NFS's port 2049 answers the cluster's networks only, the
|
||||
export the Slurm machines alone. That is one machine, not highly available
|
||||
and no faster than its disk and network: enough for a few nodes. A site with
|
||||
storage of its own uses it instead — `--slurm-shared HOST:/EXPORT` at
|
||||
install, or `llmgw ctl slurm shared --path /shared --nfs HOST:/EXPORT`
|
||||
later; Lustre, BeeGFS or GPFS the site mounts itself, and `llmgw ctl slurm
|
||||
shared --path /lustre/hpc` only has llmgw look that it is there. Every
|
||||
active llmgw account is a Linux user on every Slurm machine
|
||||
(`slurm.users: llmgw`): its login from its e-mail address, a UID from
|
||||
200000 up that is never given to anyone else, its home under
|
||||
`/shared/home`. People add their SSH public keys in the portal's
|
||||
**Profile** and log in to this machine to run `sbatch`; an account suspended
|
||||
or deleted loses its user, its files stay. On the other Slurm machines they
|
||||
are let in only while a job of theirs runs there, and their ssh session is
|
||||
put in that job, held to its cores and cards (pam_slurm_adopt, which the
|
||||
agents install); root and the machines' own users log in as before. A site whose users come from its
|
||||
directory (sssd against Active Directory or LDAP) passes `--slurm-users
|
||||
site`, or later `llmgw ctl slurm users off`.
|
||||
|
||||
With llmgw's users, Slurm's accounts are llmgw's projects: an account for
|
||||
each organization, team and project, nested (`acme`, `acme-ml`,
|
||||
`acme-ml-training`), and on each project's account its members and its
|
||||
team's and organization's. A job runs on one of its user's accounts and no
|
||||
other (`sbatch --account=acme-ml-training`; without it, the user's first),
|
||||
and when it ends its cards' hours (a job without cards, its cores') are
|
||||
priced at `auth.gpu_hour_cost` (`core_hour_cost`) into the project's usage,
|
||||
where it shows as model `slurm/PARTITION` beside the project's requests.
|
||||
Accounts a Slurm administrator made are left alone. A job's cost does not
|
||||
yet count against a budget, which binds API keys only.
|
||||
|
||||
Jobs may run in containers: `llmgw ctl slurm containers on` has the agents
|
||||
install enroot and pyxis on every Slurm machine (NVIDIA's releases, at
|
||||
versions llmgw pins and checks; pyxis is built against each machine's own
|
||||
Slurm) and, on GPU machines, nvidia-container-cli from NVIDIA's repository.
|
||||
Once every machine has them Slurm loads pyxis, and `ctl slurm status` says
|
||||
`containers: on`:
|
||||
|
||||
```sh
|
||||
srun --gres=gpu:1 --container-image=nvcr.io#nvidia/pytorch:25.01-py3 python train.py
|
||||
```
|
||||
|
||||
The shared path is in every container where it is on the host, and the job
|
||||
runs as its user, held to its cores and cards. Each node pulls the image
|
||||
from its registry for each job; for a large image used often, import it
|
||||
once (`enroot import -o /shared/images/pytorch.sqsh docker://nvcr.io#...`)
|
||||
and give that file to `--container-image`.
|
||||
|
||||
**Secure by default.** What init writes is the hardened configuration, not a
|
||||
demonstration one:
|
||||
|
||||
@@ -411,6 +468,19 @@ by itself. Three things are worth knowing before you edit it:
|
||||
cache (the answer's `cached_tokens`) are charged at it, the rest at the
|
||||
prompt rate. A provider that bills cached reads at a tenth is
|
||||
`{prompt_per_1k: 2500, cached_per_1k: 250, completion_per_1k: 10000}`.
|
||||
- What to price a self-hosted model at follows from what the cluster costs.
|
||||
Set `auth.gpu_hour_cost`, what an hour of one GPU costs in the units budgets
|
||||
are in: the hardware written off over its years, its power and room, and the
|
||||
people who run it, over the GPUs and the 730 hours of a month (an 8-GPU
|
||||
server at 250,000 over four years, 10 kW, and some upkeep is about 1.35);
|
||||
`auth.core_hour_cost` likewise for models on CPUs. The leader keeps, by the
|
||||
hour, the GPU- and core-hours each model's replicas hold, and `llmgw ctl
|
||||
prices suggest` spreads them over the tokens the model served: an output
|
||||
token counted as four input ones (making one is that much dearer), a cached
|
||||
prompt token as a tenth of one. Idle hours are in the cost, so a model held
|
||||
all month and seldom used comes out dear, which is the point; one held and
|
||||
billed nothing (a guard's judge) is shown with what it held. `--apply`
|
||||
writes the prices into `auth.pricing`.
|
||||
- `control.enabled: true` is what makes this a control plane while its catalog
|
||||
is empty. Without it, a file with no models is refused as having nothing to
|
||||
serve.
|
||||
@@ -597,7 +667,10 @@ set `discovery.nvidia_smi` in the agent configuration rather than moving it.
|
||||
|
||||
A node with no GPU at all is fine and is not an error. It reports zero cards and
|
||||
can host CPU profiles, which is a good way to try llmgw before committing
|
||||
hardware to it.
|
||||
hardware to it. A node with an NVIDIA card and no driver is an error: adding it
|
||||
stops at the checks and says so, and a node already added shows the warning in
|
||||
`llmgw ctl nodes` and the console until the driver is installed and its agent
|
||||
restarted.
|
||||
|
||||
### NVLink, if your GPUs have it
|
||||
|
||||
@@ -3160,7 +3233,7 @@ loaded at boot by `llmgw-firewall.service`):
|
||||
| Machine | Ports, for the cluster only |
|
||||
| --- | --- |
|
||||
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
|
||||
| node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6818 (slurmctld, slurmd) and 60001–63000 (srun's `SrunPortRange`) |
|
||||
| node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6819 (slurmctld, slurmd, slurmdbd) and 60001–63000 (srun's `SrunPortRange`); exporting Slurm's shared files, 2049 (NFS) |
|
||||
| store member | 5432, 8008, 2379, 2380, 9100 |
|
||||
| mirror | 8081, 8082, 9100 |
|
||||
|
||||
|
||||
+12
-5
@@ -385,18 +385,25 @@ right and mixed facts up on the rest, while the same questions through the
|
||||
knowledge base got 9 of 9. Use both: the knowledge base for what answers
|
||||
say, an adapter for how they sound.
|
||||
|
||||
Anyone in a project can do it from the portal: **Customise a model**
|
||||
(`/train`) takes an Excel, CSV or JSONL file of questions and answers, a
|
||||
model, a name and who may use it, and publishes the adapter by itself when it
|
||||
is trained. The weights, the node and the card's memory are worked out. From
|
||||
the command line:
|
||||
|
||||
```sh
|
||||
# 1. Questions and answers from the collection, written by a model of the
|
||||
# cluster. Read the file through and correct it.
|
||||
llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl
|
||||
|
||||
# 2. Train on a node. --gpu-memory is what the job needs of a card; it
|
||||
# runs beside an engine when the engine leaves that much free.
|
||||
llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \
|
||||
--base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl
|
||||
# 2. Train. The weights to train on are found from where the model's came
|
||||
# (--base-repo names others), the card's memory from their size
|
||||
# (--gpu-memory), the node as the one whose card has the most free
|
||||
# (--node); it runs beside an engine when the engine leaves that much.
|
||||
llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct --data policies.jsonl
|
||||
llmgw ctl train show JOB --follow
|
||||
|
||||
# 3. Make it the model's adapter, kept on the mirror.
|
||||
# 3. Make it the model's adapter, kept on the mirror (or --publish in step 2).
|
||||
llmgw ctl train publish JOB --restart
|
||||
```
|
||||
|
||||
|
||||
@@ -181,6 +181,12 @@ Both default to the usual places. Containerized driver installs routinely put
|
||||
`nvidia-smi` somewhere else, and an agent that cannot find it registers with no
|
||||
GPUs — which looks exactly like a node that has none.
|
||||
|
||||
A node with no driver at all is told apart from one with no card: the PCI bus
|
||||
is read as well, and an NVIDIA card there with no driver loaded is a failure in
|
||||
the checks before the agent is installed (what to install, and that a reboot
|
||||
follows), and a warning on the node in `llmgw ctl nodes` and the console
|
||||
afterwards. It serves on CPU until the driver is in and its agent restarted.
|
||||
|
||||
## Removing a node
|
||||
|
||||
```
|
||||
|
||||
@@ -43,6 +43,8 @@ rather than repeated.
|
||||
| `personal` | `Personal` | [Auth](#auth) |
|
||||
| `pricing` | `map[string]Price` | [Auth](#auth) |
|
||||
| `default_price` | `Price` | [Auth](#auth) |
|
||||
| `gpu_hour_cost` | `float64` | [Auth](#auth) |
|
||||
| `core_hour_cost` | `float64` | [Auth](#auth) |
|
||||
| `flush_interval` | `time.Duration` | [Auth](#auth) |
|
||||
| `batch_size` | `int` | [Auth](#auth) |
|
||||
| `buffer_size` | `int` | [Auth](#auth) |
|
||||
@@ -401,6 +403,9 @@ rather than repeated.
|
||||
| `action` | `string` | [Service](#service) |
|
||||
| `timeout` | `time.Duration` | [Service](#service) |
|
||||
| `stopped` | `bool` | [ServingSettings](#servingsettings) |
|
||||
| `path` | `string` | [SharedStorage](#sharedstorage) |
|
||||
| `nfs` | `string` | [SharedStorage](#sharedstorage) |
|
||||
| `server` | `string` | [SharedStorage](#sharedstorage) |
|
||||
| `bin` | `string` | [SimConfig](#simconfig) |
|
||||
| `enabled` | `bool` | [SlurmSettings](#slurmsettings) |
|
||||
| `cluster_name` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
@@ -409,6 +414,9 @@ rather than repeated.
|
||||
| `all_nodes` | `bool` | [SlurmSettings](#slurmsettings) |
|
||||
| `lent` | `[]string` | [SlurmSettings](#slurmsettings) |
|
||||
| `partition` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `shared` | `SharedStorage` | [SlurmSettings](#slurmsettings) |
|
||||
| `users` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `containers` | `bool` | [SlurmSettings](#slurmsettings) |
|
||||
| `dir` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `python` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `engine` | `string` | [StrataConfig](#strataconfig) |
|
||||
@@ -716,6 +724,21 @@ table costs whatever DefaultPrice says, which is nothing unless set.
|
||||
|
||||
Takes a [config.Price](#configprice) block.
|
||||
|
||||
### `gpu_hour_cost`
|
||||
|
||||
`float64`
|
||||
|
||||
GPUHourCost is what an hour of one GPU costs the operator, in the
|
||||
units budgets are in: the hardware written off over its years, its
|
||||
power and room, and the people who run it, over the GPUs and the
|
||||
hours of a month. With CoreHourCost for a CPU core, it is what
|
||||
`llmgw ctl prices suggest` prices each model from: the hours its
|
||||
replicas held, idle ones too, over the tokens it served.
|
||||
|
||||
### `core_hour_cost`
|
||||
|
||||
`float64`
|
||||
|
||||
### `flush_interval`
|
||||
|
||||
`time.Duration`
|
||||
@@ -3488,6 +3511,30 @@ running and every model's definition as it is: llmgw stop sets it,
|
||||
llmgw start clears it, and the models come back as their catalog
|
||||
entries ask. In force on every control plane within seconds.
|
||||
|
||||
## SharedStorage
|
||||
|
||||
### `path`
|
||||
|
||||
`string`
|
||||
|
||||
Path is where every machine sees it, as /shared.
|
||||
|
||||
### `nfs`
|
||||
|
||||
`string`
|
||||
|
||||
NFS is the site's export, host:/path, mounted at Path on every
|
||||
machine of the Slurm cluster.
|
||||
|
||||
### `server`
|
||||
|
||||
`string`
|
||||
|
||||
Server is the llmgw node, the controller or one of the nodes, that
|
||||
exports Path over NFS to the others: for a site with no storage of
|
||||
its own. One machine: not highly available, and no faster than its
|
||||
disk and network.
|
||||
|
||||
## SimConfig
|
||||
|
||||
### `bin`
|
||||
@@ -3548,6 +3595,37 @@ given back to llmgw when its last job ends.
|
||||
|
||||
Partition is the one partition every node is in. Empty is "gpu".
|
||||
|
||||
### `shared`
|
||||
|
||||
`SharedStorage`
|
||||
|
||||
Shared is the file system every machine of the Slurm cluster sees at
|
||||
the same path: home directories, datasets, the jobs' output and
|
||||
checkpoints. Empty path: none, and a batch job's output stays on the
|
||||
node it ran on.
|
||||
|
||||
Takes a [SharedStorage](#sharedstorage) block.
|
||||
|
||||
### `users`
|
||||
|
||||
`string`
|
||||
|
||||
Users is who keeps the Linux users that run jobs, the same on every
|
||||
machine of the Slurm cluster: "llmgw" makes one for each active llmgw
|
||||
account (its login from the e-mail address, a UID of llmgw's from
|
||||
200000 up, its SSH keys from the portal), let in to the controller,
|
||||
and to a node only while a job of theirs runs there (pam_slurm_adopt);
|
||||
empty leaves users to the site (a directory through sssd, or by hand).
|
||||
|
||||
### `containers`
|
||||
|
||||
`bool`
|
||||
|
||||
Containers lets jobs run in containers: `srun --container-image
|
||||
nvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),
|
||||
the GPUs and the shared path inside. The agents install both on
|
||||
every Slurm machine; jobs take it once every machine has them.
|
||||
|
||||
## StrataConfig
|
||||
|
||||
### `dir`
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
module gitea.mynodes.xyz/ilker/llmgw
|
||||
|
||||
go 1.26.6
|
||||
go 1.27
|
||||
|
||||
require (
|
||||
connectrpc.com/connect v1.21.0
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
package accounting
|
||||
|
||||
import (
|
||||
"math"
|
||||
"slices"
|
||||
)
|
||||
|
||||
// Prices from what the cluster costs: what each model held of it (its
|
||||
// replicas' GPU- and core-hours) at what an hour of a GPU or a core costs
|
||||
// the operator, spread over the tokens it served. Idle time is in it: a
|
||||
// model held all month and used an hour costs what it held, not the hour.
|
||||
//
|
||||
// An output token is priced OutputPerInput input tokens, as producing one
|
||||
// is that much dearer (one step a token, against a prompt read in one
|
||||
// pass); a cached prompt token a tenth of an input one.
|
||||
const (
|
||||
OutputPerInput = 4
|
||||
CachedPerInput = 0.1
|
||||
)
|
||||
|
||||
// Basis is what an hour costs, in the units budgets are in.
|
||||
type Basis struct {
|
||||
GPUHour float64
|
||||
CoreHour float64
|
||||
}
|
||||
|
||||
// Held is what a model held, and Served what it answered with.
|
||||
type Held struct {
|
||||
GPUSeconds, CoreSeconds int64
|
||||
}
|
||||
|
||||
// Served is a model's tokens.
|
||||
type Served struct {
|
||||
Prompt, Completion int64
|
||||
}
|
||||
|
||||
// Suggestion is one model's price from its cost.
|
||||
type Suggestion struct {
|
||||
Model string `json:"model"`
|
||||
GPUHours float64 `json:"gpu_hours"`
|
||||
CoreHours float64 `json:"core_hours"`
|
||||
// Cost is what it held cost, in budget units.
|
||||
Cost float64 `json:"cost"`
|
||||
Prompt int64 `json:"prompt_tokens"`
|
||||
Completion int64 `json:"completion_tokens"`
|
||||
// Price is the suggestion; zero with a Note saying why there is none.
|
||||
Price Price `json:"price"`
|
||||
Note string `json:"note,omitempty"`
|
||||
}
|
||||
|
||||
// Suggest prices every model that held something or served something.
|
||||
func Suggest(held map[string]Held, served map[string]Served, b Basis) []Suggestion {
|
||||
var names []string
|
||||
for m := range held {
|
||||
names = append(names, m)
|
||||
}
|
||||
for m := range served {
|
||||
if _, ok := held[m]; !ok {
|
||||
names = append(names, m)
|
||||
}
|
||||
}
|
||||
slices.Sort(names)
|
||||
out := make([]Suggestion, 0, len(names))
|
||||
for _, m := range names {
|
||||
h, s := held[m], served[m]
|
||||
g := Suggestion{Model: m, GPUHours: float64(h.GPUSeconds) / 3600, CoreHours: float64(h.CoreSeconds) / 3600,
|
||||
Prompt: s.Prompt, Completion: s.Completion}
|
||||
g.Cost = g.GPUHours*b.GPUHour + g.CoreHours*b.CoreHour
|
||||
// Tokens counted as output ones: an input token is a fraction of one.
|
||||
units := float64(s.Completion) + float64(s.Prompt)/OutputPerInput
|
||||
switch {
|
||||
case g.Cost == 0 && units > 0:
|
||||
g.Note = "held nothing recorded here: an outside provider's, or served before the hours were kept"
|
||||
case g.Cost > 0 && units == 0:
|
||||
g.Note = "held the cluster and was billed no tokens (a guard's judge, or a model priced by the picture or 3D model): no one pays what it held"
|
||||
case g.Cost > 0:
|
||||
// Micro-units per thousand output tokens, rounded up so a
|
||||
// cheap model is never free.
|
||||
out1k := int64(math.Ceil(g.Cost / units * 1000 * 1e6))
|
||||
g.Price = Price{
|
||||
CompletionPer1k: out1k,
|
||||
PromptPer1k: ceilDiv(out1k, OutputPerInput),
|
||||
CachedPer1k: max(int64(math.Ceil(float64(ceilDiv(out1k, OutputPerInput))*CachedPerInput)), 1),
|
||||
}
|
||||
}
|
||||
out = append(out, g)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func ceilDiv(a, b int64) int64 { return (a + b - 1) / b }
|
||||
@@ -0,0 +1,45 @@
|
||||
package accounting
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSuggestSpreadsWhatAModelHeldOverItsTokens(t *testing.T) {
|
||||
// The worked example: two cards for 1000 hours at 1.35 an hour is 2700;
|
||||
// 3.0 billion output tokens and 4.0 billion input ones, counted as 1.0
|
||||
// billion output ones, are 4.0 billion: 0.675 a million output tokens.
|
||||
held := map[string]Held{
|
||||
"llama-70b": {GPUSeconds: 2 * 1000 * 3600},
|
||||
"idle": {GPUSeconds: 3600},
|
||||
"small-cpu": {CoreSeconds: 8 * 100 * 3600},
|
||||
}
|
||||
served := map[string]Served{
|
||||
"llama-70b": {Prompt: 4_000_000_000, Completion: 3_000_000_000},
|
||||
"small-cpu": {Prompt: 1_000_000, Completion: 1_000_000},
|
||||
"outside": {Prompt: 10, Completion: 10},
|
||||
}
|
||||
got := Suggest(held, served, Basis{GPUHour: 1.35, CoreHour: 0.01})
|
||||
by := map[string]Suggestion{}
|
||||
for _, s := range got {
|
||||
by[s.Model] = s
|
||||
}
|
||||
l := by["llama-70b"]
|
||||
if l.Cost != 2700 || l.Price.CompletionPer1k != 675 || l.Price.PromptPer1k != 169 || l.Price.CachedPer1k != 17 {
|
||||
t.Errorf("llama: %+v", l)
|
||||
}
|
||||
// What it costs back at that price is what it held, within the rounding.
|
||||
back := float64(l.Price.Cost(l.Prompt, 0, l.Completion, 0, 0)) / 1e6
|
||||
if back < l.Cost || back > l.Cost*1.01 {
|
||||
t.Errorf("charged %.2f for a cost of %.2f", back, l.Cost)
|
||||
}
|
||||
if s := by["small-cpu"]; s.Cost != 8 || s.Price.CompletionPer1k == 0 {
|
||||
t.Errorf("cpu: %+v", s)
|
||||
}
|
||||
if s := by["idle"]; !s.Price.Zero() || !strings.Contains(s.Note, "billed no tokens") {
|
||||
t.Errorf("idle: %+v", s)
|
||||
}
|
||||
if s := by["outside"]; !s.Price.Zero() || !strings.Contains(s.Note, "held nothing") {
|
||||
t.Errorf("outside: %+v", s)
|
||||
}
|
||||
}
|
||||
@@ -596,7 +596,7 @@ func (a *Agent) session(ctx context.Context, server string) error {
|
||||
a.forgetSent()
|
||||
reg.Host = &llmgwv1.Host{
|
||||
OsId: h.OSID, OsVersion: h.OSVersion, OsCodename: h.OSCodename,
|
||||
Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver,
|
||||
Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver, NvidiaDevice: h.NVIDIADevice,
|
||||
// What is measured by now, not what the startup scan found before
|
||||
// the measurement ended: a reconnect would otherwise say 0.
|
||||
MemBandwidthGbs: a.memBandwidth().All,
|
||||
|
||||
@@ -3,6 +3,7 @@ package discovery
|
||||
import (
|
||||
"bufio"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"runtime"
|
||||
"strconv"
|
||||
@@ -28,6 +29,10 @@ type Host struct {
|
||||
CPUFlags []string `json:"cpu_flags,omitempty"`
|
||||
// NVIDIADriver is the loaded driver's version, empty when there is none.
|
||||
NVIDIADriver string `json:"nvidia_driver,omitempty"`
|
||||
// NVIDIADevice is whether an NVIDIA display controller is on the PCI bus,
|
||||
// driver or not. True with NVIDIADriver empty is a card without a driver:
|
||||
// a machine that would otherwise serve on CPU with nothing saying why.
|
||||
NVIDIADevice bool `json:"nvidia_device,omitempty"`
|
||||
// MinComputeCapability is the oldest GPU's, 0 when there are none or it
|
||||
// is not known. Not read here but from the GPU inventory, by the agent.
|
||||
MinComputeCapability float64 `json:"min_compute_capability,omitempty"`
|
||||
@@ -75,9 +80,35 @@ func DiscoverHost() Host {
|
||||
h.Arch = debianArch(runtime.GOARCH)
|
||||
h.CPUFlags = cpuFlagsFrom("/proc/cpuinfo")
|
||||
h.NVIDIADriver = nvidiaDriverFrom("/proc/driver/nvidia/version")
|
||||
h.NVIDIADevice = HasNVIDIADevice()
|
||||
return h
|
||||
}
|
||||
|
||||
// CardWithoutDriver reports an NVIDIA card on the PCI bus with no driver
|
||||
// loaded: a node that looks like a CPU node to nvidia-smi and is not one.
|
||||
func (h Host) CardWithoutDriver() bool { return h.NVIDIADevice && h.NVIDIADriver == "" }
|
||||
|
||||
// HasNVIDIADevice reports an NVIDIA display or 3D controller on the PCI bus,
|
||||
// read from sysfs, which needs no driver.
|
||||
func HasNVIDIADevice() bool { return hasNVIDIADeviceIn("/sys/bus/pci/devices") }
|
||||
|
||||
func hasNVIDIADeviceIn(root string) bool {
|
||||
vendors, _ := filepath.Glob(filepath.Join(root, "*", "vendor"))
|
||||
for _, v := range vendors {
|
||||
b, err := os.ReadFile(v) //nolint:gosec // sysfs
|
||||
if err != nil || strings.TrimSpace(string(b)) != "0x10de" {
|
||||
continue
|
||||
}
|
||||
class, _ := os.ReadFile(filepath.Join(filepath.Dir(v), "class")) //nolint:gosec // sysfs
|
||||
// 0x03xxxx is a display or 3D controller; NVIDIA also makes bridges
|
||||
// and audio devices.
|
||||
if strings.HasPrefix(strings.TrimSpace(string(class)), "0x03") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func hostFromOSRelease(path string) Host {
|
||||
var h Host
|
||||
f, err := os.Open(path) //nolint:gosec // a fixed system path
|
||||
|
||||
@@ -23,3 +23,35 @@ func TestNVIDIADriverFrom(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestACardOnTheBusIsSeenWithoutADriver(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
dev := func(name, vendor, class string) {
|
||||
d := filepath.Join(root, name)
|
||||
if err := os.MkdirAll(d, 0o750); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(d, "vendor"), []byte(vendor+"\n"), 0o600)
|
||||
_ = os.WriteFile(filepath.Join(d, "class"), []byte(class+"\n"), 0o600)
|
||||
}
|
||||
dev("0000:00:1f.0", "0x8086", "0x060100") // the chipset
|
||||
if hasNVIDIADeviceIn(root) {
|
||||
t.Fatal("a machine with no NVIDIA device said it had one")
|
||||
}
|
||||
dev("0000:01:00.1", "0x10de", "0x040300") // the card's HDMI audio alone is not a GPU
|
||||
if hasNVIDIADeviceIn(root) {
|
||||
t.Fatal("NVIDIA's audio device was taken for a GPU")
|
||||
}
|
||||
dev("0000:01:00.0", "0x10de", "0x030000")
|
||||
if !hasNVIDIADeviceIn(root) {
|
||||
t.Fatal("a VGA controller of NVIDIA's was not seen")
|
||||
}
|
||||
h := Host{NVIDIADevice: true}
|
||||
if !h.CardWithoutDriver() {
|
||||
t.Error("a card with no driver version is a card without a driver")
|
||||
}
|
||||
h.NVIDIADriver = "580.82.07"
|
||||
if h.CardWithoutDriver() {
|
||||
t.Error("a card with a driver was reported as without one")
|
||||
}
|
||||
}
|
||||
@@ -16,10 +16,14 @@ import (
|
||||
// slurmPorts are Slurm's on a node that runs any of it: slurmctld's and
|
||||
// slurmd's, and srun's range for the steps it starts.
|
||||
var slurmPorts = []firewall.Port{
|
||||
{From: slurm.CtldPort, To: slurm.SlurmdPort, What: "slurm: slurmctld and slurmd"},
|
||||
{From: slurm.CtldPort, To: slurm.DbdPort, What: "slurm: slurmctld, slurmd and slurmdbd"},
|
||||
{From: slurm.SrunPortsFrom, To: slurm.SrunPortsTo, What: "slurm: srun, for the steps it starts"},
|
||||
}
|
||||
|
||||
// nfsPort is NFSv4's, on the machine that exports the Slurm cluster's
|
||||
// shared files.
|
||||
var nfsPort = firewall.Port{From: slurm.NFSPort, To: slurm.NFSPort, What: "nfs: the Slurm cluster's shared files"}
|
||||
|
||||
// applyFirewall keeps llmgw's firewall on this node as the cluster's
|
||||
// settings say: the engines' ports and the node exporter's for the networks
|
||||
// the node reaches its control planes on, and those named, or no table at
|
||||
@@ -69,6 +73,9 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
|
||||
if a.slurm != nil && a.slurm.HasPart() {
|
||||
ports = append(ports, slurmPorts...)
|
||||
}
|
||||
if a.slurm != nil && a.slurm.Serves() {
|
||||
ports = append(ports, nfsPort)
|
||||
}
|
||||
changed, err := firewall.Install(ctx, r, firewall.Spec{Ports: ports, Allow: nets})
|
||||
switch {
|
||||
case err != nil:
|
||||
@@ -79,11 +86,12 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
|
||||
}
|
||||
|
||||
// slurmMessage is the leader's word on this node's part in Slurm. A node
|
||||
// that gets one has the firewall applied again, with Slurm's ports.
|
||||
// that gets one, or starts or stops exporting the shared files, has the
|
||||
// firewall applied again, with Slurm's ports and NFS's.
|
||||
func (a *Agent) slurmMessage(m *llmgwv1.Slurm) {
|
||||
had := a.slurm.HasPart()
|
||||
had, served := a.slurm.HasPart(), a.slurm.Serves()
|
||||
a.slurm.Apply(m)
|
||||
if had || !a.slurm.HasPart() {
|
||||
if (had || !a.slurm.HasPart()) && served == a.slurm.Serves() {
|
||||
return
|
||||
}
|
||||
a.fwMu.Lock()
|
||||
|
||||
@@ -73,7 +73,7 @@ func TestANodeGivenAPartInSlurmLetsSlurmInFromTheClusterOnly(t *testing.T) {
|
||||
a.fwMu.Lock()
|
||||
got := sh.installed
|
||||
a.fwMu.Unlock()
|
||||
if strings.Contains(got, "6817-6818") && strings.Contains(got, "60001-63000") {
|
||||
if strings.Contains(got, "6817-6819") && strings.Contains(got, "60001-63000") {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
|
||||
@@ -483,7 +483,8 @@ func (in *Installer) packages(ctx context.Context, byFamily map[string][]string,
|
||||
return fmt.Errorf("%s is missing and no package manager was found to install it",
|
||||
strings.Join(missing, ", "))
|
||||
}
|
||||
in.progress(st, "installing %s with %s", strings.Join(missing, ", "), tool)
|
||||
// The state already says installing.
|
||||
in.progress(st, "%s, with %s", strings.Join(missing, ", "), tool)
|
||||
for _, c := range cmds {
|
||||
if err := in.runWaitingForLock(ctx, pm.env, c, st); err != nil {
|
||||
return err
|
||||
|
||||
@@ -0,0 +1,214 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// accountsMarker lists what llmgw made in Slurm's accounting, a line each:
|
||||
// "account NAME" and "user LOGIN NAME". Nothing else is changed or taken
|
||||
// away: an account a Slurm administrator made is theirs.
|
||||
func (k *Keeper) accountsMarker() string { return k.marker + "-accounts" }
|
||||
|
||||
// maxEnded is the most ended jobs sent at once: a day's on a busy cluster
|
||||
// would otherwise be a status message of megabytes.
|
||||
const maxEnded = 2000
|
||||
|
||||
// accounts keeps Slurm's accounts and their users as want says, on the
|
||||
// controller.
|
||||
func (k *Keeper) accounts(ctx context.Context, want *llmgwv1.Slurm) error {
|
||||
made := k.madeAccounts()
|
||||
if !want.GetManageAccounts() {
|
||||
if len(made) == 0 {
|
||||
return nil
|
||||
}
|
||||
return k.applyAccounts(ctx, nil, made)
|
||||
}
|
||||
return k.applyAccounts(ctx, want.GetAccounts(), made)
|
||||
}
|
||||
|
||||
func (k *Keeper) madeAccounts() []string {
|
||||
b, err := k.host.ReadFile(k.accountsMarker())
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
for _, l := range strings.Split(strings.TrimSpace(string(b)), "\n") {
|
||||
if l != "" {
|
||||
out = append(out, l)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (k *Keeper) sacctmgr(ctx context.Context, args ...string) error {
|
||||
out, err := k.host.Run(ctx, nil, "sacctmgr", append([]string{"-i"}, args...)...)
|
||||
if err != nil {
|
||||
return fmt.Errorf("sacctmgr %s: %v: %s", strings.Join(args, " "), err, lastLine(out))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (k *Keeper) applyAccounts(ctx context.Context, want []*llmgwv1.SlurmAccount, made []string) error {
|
||||
out, err := k.host.Run(ctx, nil, "sacctmgr", slurm.AssocFormat...)
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading Slurm's accounts: %v: %s", err, lastLine(out))
|
||||
}
|
||||
have := map[string]bool{} // "account NAME", "user LOGIN NAME"
|
||||
for _, a := range slurm.ParseAssociations(string(out)) {
|
||||
if a.User == "" {
|
||||
have["account "+a.Account] = true
|
||||
} else {
|
||||
have["user "+a.User+" "+a.Account] = true
|
||||
}
|
||||
}
|
||||
keep := map[string]bool{}
|
||||
var errs []error
|
||||
ours := slices.Clone(made)
|
||||
note := func(line string) {
|
||||
if !slices.Contains(ours, line) {
|
||||
ours = append(ours, line)
|
||||
}
|
||||
}
|
||||
for _, a := range want {
|
||||
name, parent := a.GetName(), a.GetParent()
|
||||
if !validAccount(name) || (parent != "root" && !validAccount(parent)) {
|
||||
errs = append(errs, fmt.Errorf("the leader sent an account %q under %q", name, parent))
|
||||
continue
|
||||
}
|
||||
line := "account " + name
|
||||
keep[line] = true
|
||||
if !have[line] {
|
||||
k.log.Info("a Slurm account is made, as llmgw's projects say", "account", name, "parent", parent)
|
||||
if err := k.sacctmgr(ctx, "add", "account", name, "parent="+parent, "description=llmgw"); err != nil {
|
||||
errs = append(errs, err)
|
||||
continue
|
||||
}
|
||||
note(line)
|
||||
}
|
||||
for _, u := range a.GetUsers() {
|
||||
if !slurm.ValidLogin(u) {
|
||||
continue
|
||||
}
|
||||
ul := "user " + u + " " + name
|
||||
keep[ul] = true
|
||||
if have[ul] {
|
||||
continue
|
||||
}
|
||||
if err := k.sacctmgr(ctx, "add", "user", u, "account="+name); err != nil {
|
||||
errs = append(errs, err)
|
||||
continue
|
||||
}
|
||||
note(ul)
|
||||
}
|
||||
}
|
||||
// Taken away: what llmgw made and no longer wants, users before their
|
||||
// accounts, children before their parents (made later, listed later).
|
||||
var left []string
|
||||
for i := len(ours) - 1; i >= 0; i-- {
|
||||
line := ours[i]
|
||||
if keep[line] || !strings.HasPrefix(line, "user ") {
|
||||
continue
|
||||
}
|
||||
f := strings.Fields(line)
|
||||
if !have[line] {
|
||||
continue
|
||||
}
|
||||
if err := k.removeAssoc(ctx, f[1], f[2], have, keep); err != nil {
|
||||
errs = append(errs, err)
|
||||
left = append(left, line)
|
||||
}
|
||||
}
|
||||
for i := len(ours) - 1; i >= 0; i-- {
|
||||
line := ours[i]
|
||||
if keep[line] || !strings.HasPrefix(line, "account ") {
|
||||
continue
|
||||
}
|
||||
if have[line] {
|
||||
k.log.Info("a Slurm account llmgw made is taken away; its jobs' records stay", "account", line[len("account "):])
|
||||
if err := k.sacctmgr(ctx, "remove", "account", line[len("account "):]); err != nil {
|
||||
errs = append(errs, err)
|
||||
left = append(left, line)
|
||||
}
|
||||
}
|
||||
}
|
||||
var lines []string
|
||||
for _, l := range ours {
|
||||
if keep[l] || slices.Contains(left, l) {
|
||||
lines = append(lines, l)
|
||||
}
|
||||
}
|
||||
if !slices.Equal(lines, made) {
|
||||
var err error
|
||||
if len(lines) == 0 {
|
||||
err = k.host.Remove(k.accountsMarker())
|
||||
} else {
|
||||
_, err = k.write(k.accountsMarker(), []byte(strings.Join(lines, "\n")+"\n"), 0o644, "")
|
||||
}
|
||||
errs = append(errs, err)
|
||||
}
|
||||
return errors.Join(errs...)
|
||||
}
|
||||
|
||||
func validAccount(s string) bool {
|
||||
return s != "" && len(s) <= 64 && strings.Trim(s, "abcdefghijklmnopqrstuvwxyz0123456789_-") == ""
|
||||
}
|
||||
|
||||
// ended are the jobs that ended in the last day, newest last.
|
||||
func (k *Keeper) ended(ctx context.Context) ([]*llmgwv1.SlurmJob, error) {
|
||||
out, err := k.host.Run(ctx, nil, "sacct", slurm.SacctArgs...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("sacct: %v: %s", err, lastLine(out))
|
||||
}
|
||||
jobs, err := slurm.ParseJobs(string(out))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(jobs) > maxEnded {
|
||||
jobs = jobs[len(jobs)-maxEnded:]
|
||||
}
|
||||
ps := make([]*llmgwv1.SlurmJob, 0, len(jobs))
|
||||
for _, j := range jobs {
|
||||
ps = append(ps, &llmgwv1.SlurmJob{Id: j.ID, User: j.User, Account: j.Account, Partition: j.Partition,
|
||||
State: j.State, StartUnix: j.Start.Unix(), EndUnix: j.End.Unix(), ElapsedSeconds: int64(j.Elapsed.Seconds()),
|
||||
Cpus: int32(j.CPUs), Gpus: int32(j.GPUs)}) //nolint:gosec // counts of a node's
|
||||
}
|
||||
return ps, nil
|
||||
}
|
||||
|
||||
// removeAssoc takes login off account. Slurm refuses to take away a
|
||||
// user's default account, so another of theirs is made the default first,
|
||||
// one llmgw keeps if there is one; with none left the user goes from the
|
||||
// accounting entirely.
|
||||
func (k *Keeper) removeAssoc(ctx context.Context, login, account string, have, keep map[string]bool) error {
|
||||
var others []string
|
||||
for line := range have {
|
||||
f := strings.Fields(line)
|
||||
if len(f) == 3 && f[0] == "user" && f[1] == login && f[2] != account {
|
||||
others = append(others, f[2])
|
||||
}
|
||||
}
|
||||
if len(others) == 0 {
|
||||
return k.sacctmgr(ctx, "remove", "user", login)
|
||||
}
|
||||
slices.SortFunc(others, func(a, b string) int {
|
||||
ka, kb := keep["user "+login+" "+a], keep["user "+login+" "+b]
|
||||
switch {
|
||||
case ka && !kb:
|
||||
return -1
|
||||
case kb && !ka:
|
||||
return 1
|
||||
}
|
||||
return strings.Compare(a, b)
|
||||
})
|
||||
if err := k.sacctmgr(ctx, "modify", "user", login, "set", "defaultaccount="+others[0]); err != nil {
|
||||
return err
|
||||
}
|
||||
return k.sacctmgr(ctx, "remove", "user", login, "account="+account)
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
func TestAccountsMadeAndTakenAwayOnlyIfLlmgwsOwn(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, nil, "/var/lib/llmgw-agent/slurm", nil)
|
||||
assoc := "sacctmgr " + strings.Join(slurm.AssocFormat, " ")
|
||||
// root, and an account a Slurm administrator made.
|
||||
h.outputs[assoc] = "root||\nroot|root|\nphysics||root\n"
|
||||
m := &llmgwv1.Slurm{ManageAccounts: true, Accounts: []*llmgwv1.SlurmAccount{
|
||||
{Name: "test", Parent: "root"}, {Name: "test-arge", Parent: "test"},
|
||||
{Name: "test-arge-deneme", Parent: "test-arge", Users: []string{"ada", "ayse"}},
|
||||
}}
|
||||
if err := k.accounts(context.Background(), m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, want := range []string{"sacctmgr -i add account test parent=root description=llmgw",
|
||||
"sacctmgr -i add account test-arge-deneme parent=test-arge description=llmgw",
|
||||
"sacctmgr -i add user ayse account=test-arge-deneme"} {
|
||||
if !slices.Contains(h.ran, want) {
|
||||
t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
if !strings.Contains(string(h.files[k.accountsMarker()]), "user ada test-arge-deneme") {
|
||||
t.Errorf("marker %q", h.files[k.accountsMarker()])
|
||||
}
|
||||
|
||||
// Now there; ayse leaves the project, and the team goes.
|
||||
h.outputs[assoc] = "root||\nphysics||root\ntest||root\ntest-arge||test\ntest-arge-deneme||test-arge\n" +
|
||||
"test-arge-deneme|ada|test-arge\ntest-arge-deneme|ayse|test-arge\n"
|
||||
h.ran = nil
|
||||
m.Accounts = []*llmgwv1.SlurmAccount{{Name: "test", Parent: "root"}}
|
||||
if err := k.accounts(context.Background(), m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Neither has another account: they go from the accounting.
|
||||
want := []string{"sacctmgr -i remove user ayse", "sacctmgr -i remove user ada",
|
||||
"sacctmgr -i remove account test-arge-deneme", "sacctmgr -i remove account test-arge"}
|
||||
var got []string
|
||||
for _, c := range h.ran {
|
||||
if strings.Contains(c, "remove") || strings.Contains(c, " add ") {
|
||||
got = append(got, c)
|
||||
}
|
||||
}
|
||||
if !slices.Equal(got, want) {
|
||||
t.Errorf("ran:\n%s\nwant:\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
|
||||
}
|
||||
if slices.ContainsFunc(h.ran, func(c string) bool { return strings.Contains(c, "physics") }) {
|
||||
t.Error("touched an account llmgw did not make")
|
||||
}
|
||||
|
||||
// One still on another account: that one is made the default first.
|
||||
h.outputs[assoc] = "root||\nphysics||root\nphysics|ada|\ntest||root\nx||test\nx|ada|\n"
|
||||
h.files[k.accountsMarker()] = []byte("account test\naccount x\nuser ada x\n")
|
||||
h.ran = nil
|
||||
m.Accounts = []*llmgwv1.SlurmAccount{{Name: "test", Parent: "root"}}
|
||||
if err := k.accounts(context.Background(), m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if i, j := slices.Index(h.ran, "sacctmgr -i modify user ada set defaultaccount=physics"),
|
||||
slices.Index(h.ran, "sacctmgr -i remove user ada account=x"); i < 0 || j < i {
|
||||
t.Errorf("ran:\n%s", strings.Join(h.ran, "\n"))
|
||||
}
|
||||
|
||||
// Off: what llmgw made goes.
|
||||
h.outputs[assoc] = "root||\nphysics||root\ntest||root\n"
|
||||
h.ran = nil
|
||||
if err := k.accounts(context.Background(), &llmgwv1.Slurm{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !slices.Contains(h.ran, "sacctmgr -i remove account test") || h.files[k.accountsMarker()] != nil {
|
||||
t.Errorf("not taken away:\n%s", strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,198 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// cacheDir holds what is fetched for Slurm: enroot's packages, pyxis's
|
||||
// source and its build.
|
||||
const cacheDir = "/var/cache/llmgw-agent/slurm"
|
||||
|
||||
// pyxisMarker is the pyxis version built here, against this Slurm.
|
||||
func (k *Keeper) pyxisMarker() string { return k.marker + "-pyxis" }
|
||||
|
||||
// containers has enroot and pyxis here, set up, when want says so; what
|
||||
// is there is left when it no longer does (Slurm stops loading pyxis once
|
||||
// the controller's plugstack.conf goes). Answers the status line, empty
|
||||
// while either is missing.
|
||||
func (k *Keeper) containers(ctx context.Context, want *llmgwv1.Slurm) (string, error) {
|
||||
if !want.GetContainers() {
|
||||
return "", nil
|
||||
}
|
||||
family := k.family()
|
||||
if family == "" {
|
||||
return "", fmt.Errorf("neither dpkg nor rpm here: enroot and pyxis come as deb or rpm")
|
||||
}
|
||||
if err := k.enroot(ctx, family); err != nil {
|
||||
return "", fmt.Errorf("enroot: %w", err)
|
||||
}
|
||||
if err := k.enrootConf(ctx, want.GetShared().GetPath()); err != nil {
|
||||
return "", fmt.Errorf("enroot's settings: %w", err)
|
||||
}
|
||||
if _, err := k.host.LookPath("nvidia-smi"); err == nil {
|
||||
if err := k.nvidiaContainer(ctx, family); err != nil {
|
||||
return "", fmt.Errorf("the GPUs in containers (nvidia-container-cli): %w", err)
|
||||
}
|
||||
}
|
||||
if err := k.pyxis(ctx, family); err != nil {
|
||||
return "", fmt.Errorf("pyxis: %w", err)
|
||||
}
|
||||
return slurm.ContainersStatus(), nil
|
||||
}
|
||||
|
||||
func (k *Keeper) family() string {
|
||||
if _, err := k.host.LookPath("dpkg"); err == nil {
|
||||
return "deb"
|
||||
}
|
||||
if _, err := k.host.LookPath("rpm"); err == nil {
|
||||
return "rpm"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// enroot installs the pinned release when another, or none, is here.
|
||||
func (k *Keeper) enroot(ctx context.Context, family string) error {
|
||||
if k.installed["enroot"] {
|
||||
return nil
|
||||
}
|
||||
if out, err := k.host.Run(ctx, nil, "enroot", "version"); err == nil && strings.TrimSpace(string(out)) == slurm.EnrootVersion {
|
||||
k.installed["enroot"] = true
|
||||
return nil
|
||||
}
|
||||
pkgs, err := slurm.EnrootPackages(family, runtime.GOARCH)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("enroot missing, and this agent installs nothing")
|
||||
}
|
||||
var files []string
|
||||
for _, p := range pkgs {
|
||||
f := filepath.Join(cacheDir, p.Name)
|
||||
if err := k.host.Download(ctx, p.URL(), f, p.SHA256); err != nil {
|
||||
return err
|
||||
}
|
||||
files = append(files, f)
|
||||
}
|
||||
k.log.Info("installing enroot, for Slurm's jobs in containers", "version", slurm.EnrootVersion)
|
||||
if err := k.inst.InstallPackages(ctx, map[string][]string{family: files}); err != nil {
|
||||
return err
|
||||
}
|
||||
out, err := k.host.Run(ctx, nil, "enroot", "version")
|
||||
if v := strings.TrimSpace(string(out)); err != nil || v != slurm.EnrootVersion {
|
||||
return fmt.Errorf("enroot says %q after installing %s: %v", v, slurm.EnrootVersion, err)
|
||||
}
|
||||
k.installed["enroot"] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// enrootConf writes enroot's settings and makes its directories.
|
||||
func (k *Keeper) enrootConf(ctx context.Context, shared string) error {
|
||||
if _, err := k.write(slurm.EnrootConfPath, []byte(slurm.EnrootConf), 0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
changed, err := k.write(slurm.EnrootTmpfiles, []byte(slurm.EnrootTmpfilesConf), 0o644, "")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if changed {
|
||||
if out, err := k.host.Run(ctx, nil, "systemd-tmpfiles", "--create", slurm.EnrootTmpfiles); err != nil {
|
||||
return fmt.Errorf("systemd-tmpfiles: %v: %s", err, lastLine(out))
|
||||
}
|
||||
}
|
||||
for _, d := range []string{slurm.EnrootLib + "/cache", slurm.EnrootLib + "/data"} {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-d", d); err == nil {
|
||||
continue
|
||||
}
|
||||
// Each user makes their own beneath, as in /tmp.
|
||||
if out, err := k.host.Run(ctx, nil, "install", "-d", "-m", "1777", d); err != nil {
|
||||
return fmt.Errorf("making %s: %v: %s", d, err, lastLine(out))
|
||||
}
|
||||
}
|
||||
if shared == "" {
|
||||
return k.host.Remove(slurm.EnrootMounts)
|
||||
}
|
||||
_, err = k.write(slurm.EnrootMounts, []byte(slurm.EnrootSharedMount(shared)), 0o644, "")
|
||||
return err
|
||||
}
|
||||
|
||||
// nvidiaContainer installs nvidia-container-cli, which enroot's hook uses
|
||||
// to give a container the job's GPUs, from NVIDIA's repository.
|
||||
func (k *Keeper) nvidiaContainer(ctx context.Context, family string) error {
|
||||
if k.installed["nvidia-container-cli"] {
|
||||
return nil
|
||||
}
|
||||
if _, err := k.host.LookPath("nvidia-container-cli"); err == nil {
|
||||
k.installed["nvidia-container-cli"] = true
|
||||
return nil
|
||||
}
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("nvidia-container-cli missing, and this agent installs nothing")
|
||||
}
|
||||
switch family {
|
||||
case "deb":
|
||||
if err := k.host.Download(ctx, slurm.NvidiaContainerKeyURL, slurm.NvidiaContainerKeyPath,
|
||||
slurm.NvidiaContainerKeySHA256); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := k.write(slurm.NvidiaContainerAptList, []byte(slurm.NvidiaContainerApt), 0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
case "rpm":
|
||||
if _, err := k.write(slurm.NvidiaContainerYumRepo, []byte(slurm.NvidiaContainerYum), 0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
k.log.Info("installing nvidia-container-cli, so containers get the job's GPUs")
|
||||
if err := k.inst.InstallPackages(ctx, slurm.NvidiaContainerPackages); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := k.host.LookPath("nvidia-container-cli"); err != nil {
|
||||
return fmt.Errorf("still missing after installing libnvidia-container-tools")
|
||||
}
|
||||
k.installed["nvidia-container-cli"] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// pyxis builds the pinned pyxis against this machine's Slurm and installs
|
||||
// it, when it has not been, or was for another version.
|
||||
func (k *Keeper) pyxis(ctx context.Context, family string) error {
|
||||
if b, err := k.host.ReadFile(k.pyxisMarker()); err == nil && strings.TrimSpace(string(b)) == slurm.PyxisVersion {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-f", slurm.PyxisPlugin); err == nil {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("pyxis missing, and this agent installs nothing")
|
||||
}
|
||||
if err := k.inst.InstallPackages(ctx, map[string][]string{family: slurm.PyxisBuildPackages[family]}); err != nil {
|
||||
return fmt.Errorf("installing what builds it: %w", err)
|
||||
}
|
||||
tarball := filepath.Join(cacheDir, "pyxis-"+slurm.PyxisVersion+".tar.gz")
|
||||
if err := k.host.Download(ctx, slurm.PyxisSource, tarball, slurm.PyxisSHA256); err != nil {
|
||||
return err
|
||||
}
|
||||
src := filepath.Join(cacheDir, "pyxis-"+slurm.PyxisVersion)
|
||||
k.log.Info("building pyxis against this machine's Slurm", "version", slurm.PyxisVersion)
|
||||
for _, c := range [][]string{
|
||||
{"rm", "-rf", src},
|
||||
{"tar", "-xzf", tarball, "-C", cacheDir},
|
||||
{"make", "-C", src, "install"},
|
||||
} {
|
||||
if out, err := k.host.Run(ctx, nil, c[0], c[1:]...); err != nil {
|
||||
return fmt.Errorf("%s: %v: %s", strings.Join(c, " "), err, lastLine(out))
|
||||
}
|
||||
}
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-f", slurm.PyxisPlugin); err != nil {
|
||||
return fmt.Errorf("built, yet %s is not there", slurm.PyxisPlugin)
|
||||
}
|
||||
_, err := k.write(k.pyxisMarker(), []byte(slurm.PyxisVersion+"\n"), 0o644, "")
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"runtime"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
func TestContainersInstalledFromPinnedReleases(t *testing.T) {
|
||||
if _, err := slurm.EnrootPackages("deb", runtime.GOARCH); err != nil {
|
||||
t.Skip(err)
|
||||
}
|
||||
h := newFake()
|
||||
h.bins["dpkg"], h.bins["nvidia-smi"] = true, true
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
// enroot is not here until installed, and then says its version.
|
||||
h.fail["enroot version"] = true
|
||||
h.fail["test -d "+slurm.EnrootLib+"/cache"] = true
|
||||
h.fail["test -f "+slurm.PyxisPlugin] = true
|
||||
m := nodeMsg(true)
|
||||
m.Containers = true
|
||||
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Nfs: "10.0.0.11:/shared"}
|
||||
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
if st.GetContainers() != "" || !strings.Contains(st.GetError(), "containers: enroot") {
|
||||
t.Fatalf("ready with enroot saying nothing: %v", st)
|
||||
}
|
||||
pkgs, _ := slurm.EnrootPackages("deb", runtime.GOARCH)
|
||||
for _, p := range pkgs {
|
||||
if !slices.Contains(h.ran, "download "+p.URL()+" "+cacheDir+"/"+p.Name+" "+p.SHA256) {
|
||||
t.Errorf("%s not fetched with its digest:\n%s", p.Name, strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
if len(inst.got["deb"]) != 2 || !strings.HasPrefix(inst.got["deb"][0], cacheDir) {
|
||||
t.Errorf("installed %v", inst.got)
|
||||
}
|
||||
|
||||
delete(h.fail, "enroot version")
|
||||
h.outputs["enroot version"] = slurm.EnrootVersion + "\n"
|
||||
h.bins["nvidia-container-cli"] = false
|
||||
st = k.Reconcile(context.Background(), m)
|
||||
if !strings.Contains(st.GetError(), "nvidia-container-cli") {
|
||||
t.Errorf("a GPU machine without nvidia-container-cli is ready: %v", st)
|
||||
}
|
||||
if string(h.files[slurm.NvidiaContainerAptList]) != slurm.NvidiaContainerApt ||
|
||||
!slices.Contains(h.ran, "download "+slurm.NvidiaContainerKeyURL+" "+slurm.NvidiaContainerKeyPath+" "+slurm.NvidiaContainerKeySHA256) {
|
||||
t.Errorf("NVIDIA's repository not added with its key checked")
|
||||
}
|
||||
|
||||
h.bins["nvidia-container-cli"] = true
|
||||
st = k.Reconcile(context.Background(), m)
|
||||
// Built, but the plugin is still not there.
|
||||
if st.GetContainers() != "" || !strings.Contains(st.GetError(), "pyxis") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
for _, want := range []string{"tar -xzf " + cacheDir + "/pyxis-" + slurm.PyxisVersion + ".tar.gz -C " + cacheDir,
|
||||
"make -C " + cacheDir + "/pyxis-" + slurm.PyxisVersion + " install", "install -d -m 1777 " + slurm.EnrootLib + "/cache"} {
|
||||
if !slices.Contains(h.ran, want) {
|
||||
t.Errorf("did not run %q", want)
|
||||
}
|
||||
}
|
||||
delete(h.fail, "test -f "+slurm.PyxisPlugin)
|
||||
st = k.Reconcile(context.Background(), m)
|
||||
if st.GetContainers() != slurm.ContainersStatus() || st.GetError() != "" {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if string(h.files[slurm.EnrootConfPath]) != slurm.EnrootConf ||
|
||||
!strings.Contains(string(h.files[slurm.EnrootMounts]), "/shared /shared none") {
|
||||
t.Errorf("enroot's settings not written")
|
||||
}
|
||||
|
||||
// Done: nothing fetched or built again.
|
||||
h.ran = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
for _, c := range h.ran {
|
||||
if strings.HasPrefix(c, "download") || strings.HasPrefix(c, "make") {
|
||||
t.Errorf("ran %q again", c)
|
||||
}
|
||||
}
|
||||
|
||||
// Off: nothing said, nothing taken away.
|
||||
m.Containers = false
|
||||
if st := k.Reconcile(context.Background(), m); st.GetContainers() != "" {
|
||||
t.Errorf("status %v", st)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// mariaDBConfDirs are where each family's MariaDB reads its settings: Debian
|
||||
// and Ubuntu's, then Red Hat's.
|
||||
var mariaDBConfDirs = []string{"/etc/mysql/mariadb.conf.d", "/etc/my.cnf.d"}
|
||||
|
||||
const mariaDBConfName = "90-llmgw-slurm.cnf"
|
||||
|
||||
// acctSQLPath is where the database's statements are put for the client to
|
||||
// read, never on its command line: they hold the password.
|
||||
const acctSQLPath = "/run/llmgw-slurm-acct.sql"
|
||||
|
||||
// accounting keeps slurmdbd on the controller: MariaDB on this machine
|
||||
// alone, the database and its user, slurmdbd on them, and the cluster in it.
|
||||
// Before slurmctld, which reports to it from its first start.
|
||||
func (k *Keeper) accounting(ctx context.Context, key []byte, files map[string]string) error {
|
||||
password := slurm.DbdPassword(key)
|
||||
changed, err := k.mariaDBSettings(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch {
|
||||
case changed:
|
||||
k.log.Info("MariaDB, for Slurm's accounting, starts on this machine alone")
|
||||
if err := k.systemctl(ctx, "enable", "--now", "mariadb", "&&restart"); err != nil {
|
||||
return err
|
||||
}
|
||||
case !k.active(ctx, "mariadb"):
|
||||
if err := k.systemctl(ctx, "enable", "--now", "mariadb"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
client, err := k.mariaDBClient()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := k.waitFor(ctx, "MariaDB", client, "-e", "SELECT 1"); err != nil {
|
||||
return err
|
||||
}
|
||||
// The database and its user, once each time the agent starts or the
|
||||
// password is new: cheap, and a database restored from elsewhere gets
|
||||
// its user back.
|
||||
sum := sha256.Sum256([]byte(password))
|
||||
if mark := hex.EncodeToString(sum[:]); k.acctDone != mark {
|
||||
if err := k.host.WriteFile(acctSQLPath, []byte(slurm.AcctSQL(password)), 0o600, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
out, err := k.host.Run(ctx, nil, client, "-e", "source "+acctSQLPath)
|
||||
_ = k.host.Remove(acctSQLPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("making Slurm's database: %v: %s", err, lastLine(out))
|
||||
}
|
||||
k.acctDone = mark
|
||||
}
|
||||
|
||||
out, err := k.host.Run(ctx, nil, "hostname", "-s")
|
||||
host := strings.TrimSpace(string(out))
|
||||
if err != nil || host == "" {
|
||||
return fmt.Errorf("this machine's short hostname: %v: %s", err, lastLine(out))
|
||||
}
|
||||
changed, err = k.write(slurm.DbdConfPath, []byte(slurm.DbdConf(host, password)), 0o600, "slurm")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch {
|
||||
case changed:
|
||||
k.log.Info("slurmdbd starts again on new settings")
|
||||
if err := k.systemctl(ctx, "enable", "--now", "slurmdbd", "&&restart"); err != nil {
|
||||
return err
|
||||
}
|
||||
case !k.active(ctx, "slurmdbd"):
|
||||
if err := k.systemctl(ctx, "enable", "--now", "slurmdbd"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := k.waitFor(ctx, "slurmdbd", "sacctmgr", "-n", "-P", "show", "cluster", "format=cluster"); err != nil {
|
||||
return err
|
||||
}
|
||||
name := clusterName(files["slurm.conf"])
|
||||
if name == "" || k.clusterAdded == name {
|
||||
return nil
|
||||
}
|
||||
out, err = k.host.Run(ctx, nil, "sacctmgr", "-n", "-P", "show", "cluster", "format=cluster")
|
||||
if err != nil {
|
||||
return fmt.Errorf("sacctmgr show cluster: %v: %s", err, lastLine(out))
|
||||
}
|
||||
for _, l := range strings.Split(string(out), "\n") {
|
||||
if strings.TrimSpace(l) == name {
|
||||
k.clusterAdded = name
|
||||
return nil
|
||||
}
|
||||
}
|
||||
k.log.Info("the cluster is added to Slurm's accounting", "cluster", name)
|
||||
if out, err := k.host.Run(ctx, nil, "sacctmgr", "-i", "add", "cluster", name); err != nil {
|
||||
return fmt.Errorf("sacctmgr add cluster %s: %v: %s", name, err, lastLine(out))
|
||||
}
|
||||
k.clusterAdded = name
|
||||
return nil
|
||||
}
|
||||
|
||||
// mariaDBSettings keeps MariaDB to this machine, when llmgw installed it;
|
||||
// one that was here before llmgw keeps its own settings, and only gets the
|
||||
// database and its user.
|
||||
func (k *Keeper) mariaDBSettings(ctx context.Context) (bool, error) {
|
||||
if _, err := k.host.ReadFile(k.mariaDBMarker()); err != nil {
|
||||
return false, nil
|
||||
}
|
||||
for _, d := range mariaDBConfDirs {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-d", d); err == nil {
|
||||
return k.write(d+"/"+mariaDBConfName, []byte(slurm.MariaDBConf), 0o644, "")
|
||||
}
|
||||
}
|
||||
return false, fmt.Errorf("MariaDB has none of %s to put its settings in", strings.Join(mariaDBConfDirs, ", "))
|
||||
}
|
||||
|
||||
// mariaDBMarker is there when llmgw installed MariaDB on this machine.
|
||||
func (k *Keeper) mariaDBMarker() string { return k.marker + "-mariadb" }
|
||||
|
||||
// mariaDBClient is MariaDB's command-line client, under either of its
|
||||
// names.
|
||||
func (k *Keeper) mariaDBClient() (string, error) {
|
||||
for _, c := range []string{"mariadb", "mysql"} {
|
||||
if _, err := k.host.LookPath(c); err == nil {
|
||||
return c, nil
|
||||
}
|
||||
}
|
||||
return "", errors.New("MariaDB's client (mariadb or mysql) is not installed")
|
||||
}
|
||||
|
||||
// waitFor runs a command until it succeeds, half a minute at most.
|
||||
func (k *Keeper) waitFor(ctx context.Context, what, name string, args ...string) error {
|
||||
var last []byte
|
||||
for range 30 {
|
||||
out, err := k.host.Run(ctx, nil, name, args...)
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
last = out
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(pingWait):
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("%s did not answer after starting: %s", what, lastLine(last))
|
||||
}
|
||||
|
||||
// clusterName is ClusterName in slurm.conf.
|
||||
func clusterName(conf string) string {
|
||||
sc := bufio.NewScanner(strings.NewReader(conf))
|
||||
for sc.Scan() {
|
||||
if v, ok := strings.CutPrefix(strings.TrimSpace(sc.Text()), "ClusterName="); ok {
|
||||
return strings.TrimSpace(v)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,150 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// What a machine says of the shared path.
|
||||
const (
|
||||
SharedMounted = "mounted"
|
||||
SharedExported = "exported"
|
||||
SharedMissing = "missing"
|
||||
)
|
||||
|
||||
// shared keeps the shared path as want says: exported from here, mounted
|
||||
// from an export, or looked for where the site mounts it. Answers how it
|
||||
// is.
|
||||
func (k *Keeper) shared(ctx context.Context, want *llmgwv1.Slurm) (string, error) {
|
||||
sh := want.GetShared()
|
||||
path := sh.GetPath()
|
||||
k.unshare(ctx, path, sh.GetServe(), sh.GetNfs())
|
||||
if path == "" {
|
||||
return "", nil
|
||||
}
|
||||
if !strings.HasPrefix(path, "/") || strings.Contains(path, "..") || strings.ContainsAny(path, " \n\t\"'\\$`") {
|
||||
return SharedMissing, fmt.Errorf("the leader sent a path %q", path)
|
||||
}
|
||||
switch {
|
||||
case sh.GetServe():
|
||||
if err := k.host.MkdirAll(path, 0o755, ""); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
changed, err := k.write(slurm.ExportsPath, []byte(slurm.Exports(path, sh.GetClients())), 0o644, "")
|
||||
if err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
if !k.active(ctx, "nfs-server") {
|
||||
k.log.Info("NFS starts on this machine, exporting the Slurm cluster's shared files", "path", path)
|
||||
if err := k.systemctl(ctx, "enable", "--now", "nfs-server"); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
changed = true
|
||||
}
|
||||
if changed {
|
||||
if out, err := k.host.Run(ctx, nil, "exportfs", "-ra"); err != nil {
|
||||
return SharedMissing, fmt.Errorf("exportfs -ra: %v: %s", err, lastLine(out))
|
||||
}
|
||||
}
|
||||
return SharedExported, nil
|
||||
case sh.GetNfs() != "":
|
||||
nfs := sh.GetNfs()
|
||||
if strings.ContainsAny(nfs, " \n\t\"'\\$`") || !strings.Contains(nfs, ":/") {
|
||||
return SharedMissing, fmt.Errorf("the leader sent an export %q", nfs)
|
||||
}
|
||||
if err := k.host.MkdirAll(path, 0o755, ""); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
unit := slurm.MountUnitName(path)
|
||||
changed, err := k.write(filepath.Join(slurm.MountUnitDir, unit), []byte(slurm.MountUnit(nfs, path)), 0o644, "")
|
||||
if err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
if changed {
|
||||
k.log.Info("the Slurm cluster's shared files are mounted here", "from", nfs, "at", path)
|
||||
if err := k.systemctl(ctx, "daemon-reload"); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
if err := k.systemctl(ctx, "enable", unit); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
// From another export: a restart mounts the new one, and fails
|
||||
// while a job still has files open on the old.
|
||||
if err := k.systemctl(ctx, "restart", unit); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
} else if !k.active(ctx, unit) {
|
||||
if err := k.systemctl(ctx, "start", unit); err != nil {
|
||||
return SharedMissing, err
|
||||
}
|
||||
}
|
||||
}
|
||||
if _, err := k.host.Run(ctx, nil, "mountpoint", "-q", path); err != nil {
|
||||
if sh.GetNfs() == "" {
|
||||
return SharedMissing, fmt.Errorf("%s is not mounted here: the site mounts it (slurm.shared names no export)", path)
|
||||
}
|
||||
return SharedMissing, fmt.Errorf("%s is not mounted from %s", path, sh.GetNfs())
|
||||
}
|
||||
return SharedMounted, nil
|
||||
}
|
||||
|
||||
// unshare takes away the export or the mount llmgw kept here that want no
|
||||
// longer has: path is where the files are wanted now, serve and nfs how.
|
||||
func (k *Keeper) unshare(ctx context.Context, path string, serve bool, nfs string) {
|
||||
if _, err := k.host.ReadFile(slurm.ExportsPath); err == nil && (!serve || path == "") {
|
||||
k.log.Info("this machine no longer exports the Slurm cluster's shared files")
|
||||
_ = k.host.Remove(slurm.ExportsPath)
|
||||
_, _ = k.host.Run(ctx, nil, "exportfs", "-ra")
|
||||
}
|
||||
out, err := k.host.Run(ctx, nil, "systemctl", "list-units", "--all", "--plain", "--no-legend", "--type=mount")
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
keep := ""
|
||||
if path != "" && nfs != "" && !serve {
|
||||
keep = slurm.MountUnitName(path)
|
||||
}
|
||||
for _, l := range strings.Split(string(out), "\n") {
|
||||
f := strings.Fields(l)
|
||||
if len(f) == 0 || f[0] == keep {
|
||||
continue
|
||||
}
|
||||
file := filepath.Join(slurm.MountUnitDir, f[0])
|
||||
b, err := k.host.ReadFile(file)
|
||||
if err != nil || !strings.HasPrefix(string(b), "# Written by llmgw (BACKLOG P): the Slurm cluster's shared files.") {
|
||||
continue
|
||||
}
|
||||
k.log.Info("the Slurm cluster's old shared mount is taken away", "unit", f[0])
|
||||
if err := k.systemctl(ctx, "disable", "--now", f[0]); err != nil {
|
||||
k.log.Warn("could not unmount the old shared files", "unit", f[0], "err", err)
|
||||
continue
|
||||
}
|
||||
_ = k.host.Remove(file)
|
||||
_ = k.systemctl(ctx, "daemon-reload")
|
||||
}
|
||||
}
|
||||
|
||||
// sharedPackages are what the shared path needs here, by the command that
|
||||
// shows each is installed.
|
||||
func sharedPackages(want *llmgwv1.Slurm, need map[string][2]string) {
|
||||
switch sh := want.GetShared(); {
|
||||
case sh.GetPath() == "":
|
||||
case sh.GetServe():
|
||||
need["exportfs"] = [2]string{"nfs-kernel-server", "nfs-utils"}
|
||||
case sh.GetNfs() != "":
|
||||
need["mount.nfs"] = [2]string{"nfs-common", "nfs-utils"}
|
||||
}
|
||||
}
|
||||
|
||||
// Serves is whether this machine exports the shared files, so NFS's port
|
||||
// is kept for the cluster's networks.
|
||||
func (k *Keeper) Serves() bool {
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
return k.want.GetShared().GetServe() && k.want.GetShared().GetPath() != ""
|
||||
}
|
||||
@@ -0,0 +1,171 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
func ayse() *llmgwv1.SlurmUser {
|
||||
return &llmgwv1.SlurmUser{Login: "ayse", Uid: 200000, Name: "Ayşe Y.", Home: "/shared/home/ayse",
|
||||
SshKeys: []string{"ssh-ed25519 AAAA ayse@laptop"}}
|
||||
}
|
||||
|
||||
func TestTheServerExportsAndMakesUsersWithHomes(t *testing.T) {
|
||||
h := newFake()
|
||||
h.active["ssh"] = true
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
m := nodeMsg(false)
|
||||
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Serve: true, Clients: []string{"10.0.0.22", "10.0.0.21"}}
|
||||
m.ManageUsers, m.MakeHomes, m.Users = true, true, []*llmgwv1.SlurmUser{ayse()}
|
||||
h.fail["test -d /shared/home/ayse"] = true
|
||||
h.files[slurm.PAMSSHD] = []byte(pamStack)
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
if st.GetError() != "" {
|
||||
t.Fatal(st.GetError())
|
||||
}
|
||||
if st.GetShared() != SharedExported || st.GetUsers() != 1 {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if !slices.Contains(inst.got["deb"], "nfs-kernel-server") {
|
||||
t.Errorf("packages %v", inst.got)
|
||||
}
|
||||
if e := string(h.files[slurm.ExportsPath]); !strings.Contains(e, `"/shared" 10.0.0.21(`) {
|
||||
t.Errorf("exports:\n%s", e)
|
||||
}
|
||||
for _, want := range []string{"systemctl enable --now nfs-server", "exportfs -ra", "groupadd -g 200000 ayse",
|
||||
"useradd -u 200000 -g 200000 -M -d /shared/home/ayse -s /bin/bash -c Ayşe Y. ayse",
|
||||
"install -d -m 0700 -o 200000 -g 200000 /shared/home/ayse", "sshd -t", "systemctl reload ssh"} {
|
||||
if !slices.Contains(h.ran, want) {
|
||||
t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
if string(h.files[slurm.AuthorizedKeysDir+"/ayse"]) != "ssh-ed25519 AAAA ayse@laptop\n" ||
|
||||
string(h.files[k.usersMarker()]) != "ayse\n" || h.files[slurm.SSHDropIn] == nil {
|
||||
t.Errorf("files: keys %q, marker %q", h.files[slurm.AuthorizedKeysDir+"/ayse"], h.files[k.usersMarker()])
|
||||
}
|
||||
|
||||
// Again, made: nothing is run again.
|
||||
h.outputs["getent passwd ayse"] = "ayse:x:200000:200000:Ayşe Y.:/shared/home/ayse:/bin/bash\n"
|
||||
delete(h.fail, "test -d /shared/home/ayse")
|
||||
h.ran = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
for _, c := range h.ran {
|
||||
if strings.HasPrefix(c, "useradd") || strings.HasPrefix(c, "usermod") || strings.HasPrefix(c, "exportfs") ||
|
||||
strings.HasPrefix(c, "install") || c == "systemctl reload ssh" {
|
||||
t.Errorf("ran %q again", c)
|
||||
}
|
||||
}
|
||||
|
||||
// Gone from the accounts: taken away, its home left.
|
||||
m.Users = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
if !slices.Contains(h.ran, "userdel ayse") || h.files[k.usersMarker()] != nil || h.files[slurm.SSHDropIn] != nil ||
|
||||
h.files[slurm.AuthorizedKeysDir+"/ayse"] != nil {
|
||||
t.Errorf("not taken away:\n%s", strings.Join(h.ran, "\n"))
|
||||
}
|
||||
if slices.ContainsFunc(h.ran, func(c string) bool { return strings.HasPrefix(c, "rm") || strings.Contains(c, "userdel -r") }) {
|
||||
t.Error("a home was removed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAClientMountsAndLeavesOthersUsersAlone(t *testing.T) {
|
||||
h := newFake()
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
m := nodeMsg(false)
|
||||
m.Shared = &llmgwv1.SlurmShared{Path: "/shared", Nfs: "10.0.0.11:/shared"}
|
||||
bob := &llmgwv1.SlurmUser{Login: "bob", Uid: 200001, Home: "/shared/home/bob"}
|
||||
m.ManageUsers, m.Users = true, []*llmgwv1.SlurmUser{ayse(), bob}
|
||||
// The site's own ayse, with its own UID.
|
||||
h.outputs["getent passwd ayse"] = "ayse:x:1001:1001::/home/ayse:/bin/bash\n"
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
if st.GetShared() != SharedMounted || st.GetUsers() != 1 {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if !strings.Contains(st.GetError(), "ayse is already a user here with UID 1001") {
|
||||
t.Errorf("error %q", st.GetError())
|
||||
}
|
||||
if u := string(h.files["/etc/systemd/system/shared.mount"]); !strings.Contains(u, "What=10.0.0.11:/shared") {
|
||||
t.Fatalf("unit:\n%s", u)
|
||||
}
|
||||
for _, want := range []string{"systemctl enable shared.mount", "systemctl restart shared.mount", "mountpoint -q /shared",
|
||||
"useradd -u 200001 -g 200001 -M -d /shared/home/bob -s /bin/bash -c bob"} {
|
||||
if !slices.Contains(h.ran, want) {
|
||||
t.Errorf("did not run %q:\n%s", want, strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
if slices.ContainsFunc(h.ran, func(c string) bool { return strings.Contains(c, "ayse") && !strings.HasPrefix(c, "getent") }) ||
|
||||
slices.ContainsFunc(h.ran, func(c string) bool { return strings.HasPrefix(c, "install -d") }) {
|
||||
t.Errorf("touched the site's ayse, or made a home:\n%s", strings.Join(h.ran, "\n"))
|
||||
}
|
||||
if string(h.files[k.usersMarker()]) != "bob\n" {
|
||||
t.Errorf("marker %q", h.files[k.usersMarker()])
|
||||
}
|
||||
|
||||
// Not mounted: said.
|
||||
h.fail["mountpoint -q /shared"] = true
|
||||
if st := k.Reconcile(context.Background(), m); st.GetShared() != SharedMissing ||
|
||||
!strings.Contains(st.GetError(), "not mounted from 10.0.0.11:/shared") {
|
||||
t.Errorf("status %v", st)
|
||||
}
|
||||
|
||||
// No shared path any more: the mount is taken away.
|
||||
h.outputs["systemctl list-units --all --plain --no-legend --type=mount"] = "shared.mount loaded active mounted llmgw\n"
|
||||
m.Shared = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
if !slices.Contains(h.ran, "systemctl disable --now shared.mount") || h.files["/etc/systemd/system/shared.mount"] != nil {
|
||||
t.Errorf("mount kept:\n%s", strings.Join(h.ran, "\n"))
|
||||
}
|
||||
}
|
||||
|
||||
const pamStack = "@include common-auth\naccount required pam_nologin.so\n@include common-account\n@include common-session\n"
|
||||
|
||||
func TestUsersReachANodeOnlyThroughTheirJobs(t *testing.T) {
|
||||
h := newFake()
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
h.files[slurm.PAMSSHD] = []byte(pamStack)
|
||||
// The module is not there until its package is installed.
|
||||
for _, d := range slurm.ModuleDirs {
|
||||
h.fail["test -f "+d+"/"+slurm.AdoptModule] = true
|
||||
}
|
||||
m := nodeMsg(true)
|
||||
m.ManageUsers, m.Users = true, []*llmgwv1.SlurmUser{ayse()}
|
||||
k.Reconcile(context.Background(), m)
|
||||
if !slices.Contains(inst.got["deb"], "libpam-slurm-adopt") {
|
||||
t.Fatalf("not installed: %v", inst.got)
|
||||
}
|
||||
if got := string(h.files[slurm.PAMSSHD]); got != pamStack {
|
||||
t.Fatalf("lines put in with the module still missing:\n%s", got)
|
||||
}
|
||||
delete(h.fail, "test -f "+slurm.ModuleDirs[0]+"/"+slurm.AdoptModule)
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
got := string(h.files[slurm.PAMSSHD])
|
||||
if !strings.Contains(got, "pam_listfile.so item=user sense=allow file="+k.usersMarker()) ||
|
||||
!strings.Contains(got, "account required "+slurm.AdoptModule) {
|
||||
t.Fatalf("not held to their jobs (%s):\n%s", st.GetError(), got)
|
||||
}
|
||||
|
||||
// The controller is where they log in: not held there.
|
||||
m.Controller = true
|
||||
m.Files = map[string]string{"slurm.conf": "x"}
|
||||
k.Reconcile(context.Background(), m)
|
||||
if string(h.files[slurm.PAMSSHD]) != pamStack {
|
||||
t.Errorf("held on the controller:\n%s", h.files[slurm.PAMSSHD])
|
||||
}
|
||||
|
||||
// No users of llmgw's any more: the lines go.
|
||||
m.Controller, m.Files = false, nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
m.Users = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
if string(h.files[slurm.PAMSSHD]) != pamStack {
|
||||
t.Errorf("lines left with no users:\n%s", h.files[slurm.PAMSSHD])
|
||||
}
|
||||
}
|
||||
@@ -13,8 +13,10 @@ import (
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
@@ -43,6 +45,9 @@ type Host interface {
|
||||
// MkdirAll makes path, owned by owner.
|
||||
MkdirAll(path string, mode fs.FileMode, owner string) error
|
||||
Remove(path string) error
|
||||
// Download fetches url to path, kept only if its SHA-256 is sha; a file
|
||||
// already there with it is not fetched again.
|
||||
Download(ctx context.Context, url, path, sha string) error
|
||||
}
|
||||
|
||||
// Installer installs packages by family, "deb" and "rpm".
|
||||
@@ -77,6 +82,10 @@ type Keeper struct {
|
||||
// minute's check does not look again.
|
||||
installed map[string]bool
|
||||
files string
|
||||
// acctDone is the digest of the password the database's user was last
|
||||
// given; clusterAdded the cluster last found in the accounting.
|
||||
acctDone string
|
||||
clusterAdded string
|
||||
}
|
||||
|
||||
// New is a Keeper; marker is a file it keeps while it runs Slurm here, so
|
||||
@@ -206,13 +215,72 @@ func (k *Keeper) Reconcile(ctx context.Context, want *llmgwv1.Slurm) *llmgwv1.Sl
|
||||
if err := k.munge(ctx, want.GetMungeKey()); err != nil {
|
||||
return fail("munge", err)
|
||||
}
|
||||
// The shared files, then the users whose homes are on them. Neither
|
||||
// keeps Slurm from running: what goes wrong is said, and Slurm kept.
|
||||
var shErr, usErr error
|
||||
st.Shared, shErr = k.shared(ctx, want)
|
||||
if shErr != nil {
|
||||
k.log.Warn("keeping the Slurm cluster's shared files on this node", "err", shErr)
|
||||
}
|
||||
if shErr == nil || !want.GetMakeHomes() || want.GetShared().GetPath() == "" {
|
||||
st.Users, usErr = k.users(ctx, want)
|
||||
} else {
|
||||
// Homes made now would be made beside the shared files, not on them.
|
||||
usErr = errors.New("waiting for the shared files, where the homes are")
|
||||
}
|
||||
if usErr != nil {
|
||||
k.log.Warn("keeping the Slurm users on this node", "err", usErr)
|
||||
}
|
||||
// Jobs in containers; Slurm runs without them, and is kept anyway.
|
||||
var ctErr error
|
||||
if st.Containers, ctErr = k.containers(ctx, want); ctErr != nil {
|
||||
k.log.Warn("setting up jobs in containers on this node", "err", ctErr)
|
||||
}
|
||||
defer func() {
|
||||
switch {
|
||||
case st.Error != "":
|
||||
case shErr != nil:
|
||||
st.Error = "shared files: " + shErr.Error()
|
||||
case usErr != nil:
|
||||
st.Error = "users: " + usErr.Error()
|
||||
case ctErr != nil:
|
||||
st.Error = "containers: " + ctErr.Error()
|
||||
}
|
||||
}()
|
||||
if want.GetController() {
|
||||
// Accounting first: slurmctld reports to it from its first start.
|
||||
// Without it slurmctld still runs jobs, a pass of backfill late
|
||||
// each; so a failure here is said, and slurmctld is kept anyway.
|
||||
var acctErr error
|
||||
if acctErr = k.accounting(ctx, want.GetMungeKey(), want.GetFiles()); acctErr != nil {
|
||||
k.log.Warn("keeping Slurm's accounting on this node", "err", acctErr)
|
||||
}
|
||||
st.SlurmdbdRunning = k.active(ctx, "slurmdbd")
|
||||
defer func() {
|
||||
if acctErr != nil && st.Error == "" {
|
||||
st.Error = "slurmdbd: " + acctErr.Error()
|
||||
}
|
||||
}()
|
||||
sum, err := k.controller(ctx, want.GetFiles())
|
||||
st.FilesSha256 = sum
|
||||
if err != nil {
|
||||
return fail("slurmctld", err)
|
||||
}
|
||||
st.SlurmctldRunning = k.active(ctx, "slurmctld")
|
||||
// The accounts, and the jobs that ended, need the accounting.
|
||||
if st.SlurmdbdRunning && st.SlurmctldRunning {
|
||||
if err := k.accounts(ctx, want); err != nil {
|
||||
k.log.Warn("keeping Slurm's accounts", "err", err)
|
||||
if acctErr == nil {
|
||||
acctErr = fmt.Errorf("accounts: %w", err)
|
||||
}
|
||||
}
|
||||
if ended, err := k.ended(ctx); err != nil {
|
||||
k.log.Warn("reading the jobs that ended", "err", err)
|
||||
} else {
|
||||
st.Ended = ended
|
||||
}
|
||||
}
|
||||
}
|
||||
if want.GetNode() {
|
||||
if err := k.node(ctx, want, st); err != nil {
|
||||
@@ -229,10 +297,13 @@ func (k *Keeper) ensurePackages(ctx context.Context, want *llmgwv1.Slurm) error
|
||||
}
|
||||
if want.GetController() {
|
||||
need["slurmctld"] = [2]string{"slurmctld", "slurm-slurmctld"}
|
||||
need["slurmdbd"] = [2]string{"slurmdbd", "slurm-slurmdbd"}
|
||||
need["mariadbd"] = [2]string{"mariadb-server", "mariadb-server"}
|
||||
}
|
||||
if want.GetNode() {
|
||||
need["slurmd"] = [2]string{"slurmd", "slurm-slurmd"}
|
||||
}
|
||||
sharedPackages(want, need)
|
||||
pkgs := map[string][]string{}
|
||||
for bin, p := range need {
|
||||
if k.installed[bin] {
|
||||
@@ -257,6 +328,14 @@ func (k *Keeper) ensurePackages(ctx context.Context, want *llmgwv1.Slurm) error
|
||||
if err := k.inst.InstallPackages(ctx, pkgs); err != nil {
|
||||
return err
|
||||
}
|
||||
if slices.Contains(pkgs["deb"], "mariadb-server") {
|
||||
// Installed by llmgw: its settings are llmgw's to keep. One that was
|
||||
// here before is another's, and is left as it is.
|
||||
if err := k.host.WriteFile(k.mariaDBMarker(), []byte("llmgw installed MariaDB here, for Slurm's accounting\n"),
|
||||
0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
for bin := range need {
|
||||
if _, err := k.host.LookPath(bin); err != nil {
|
||||
return fmt.Errorf("%s is still missing after installing %s", bin, strings.Join(pkgs["deb"], ", "))
|
||||
@@ -483,14 +562,20 @@ func (k *Keeper) stop(ctx context.Context) {
|
||||
if _, err := k.host.ReadFile(k.marker); err != nil {
|
||||
return
|
||||
}
|
||||
k.log.Info("this node has no part in Slurm any more: slurmd and slurmctld stopped; the packages stay")
|
||||
for _, u := range []string{"slurmd", "slurmctld"} {
|
||||
k.log.Info("this node has no part in Slurm any more: slurmd, slurmctld and slurmdbd stopped; the packages and MariaDB stay")
|
||||
// MariaDB stays: it may have been the machine's before, and holds the
|
||||
// jobs' records.
|
||||
for _, u := range []string{"slurmd", "slurmctld", "slurmdbd"} {
|
||||
_ = k.systemctl(ctx, "disable", "--now", u)
|
||||
}
|
||||
_ = k.host.Remove(DropIn)
|
||||
_ = k.systemctl(ctx, "daemon-reload")
|
||||
k.unshare(ctx, "", false, "")
|
||||
if err := k.removeUsers(ctx, k.madeUsers(), nil); err != nil {
|
||||
k.log.Warn("could not take away every Slurm user llmgw made", "err", err)
|
||||
}
|
||||
_ = k.host.Remove(k.marker)
|
||||
k.files = ""
|
||||
k.files, k.acctDone, k.clusterAdded = "", "", ""
|
||||
}
|
||||
|
||||
// write writes a file when it differs, saying whether it did.
|
||||
@@ -551,7 +636,8 @@ func (Local) LookPath(name string) (string, error) {
|
||||
if p, err := exec.LookPath(name); err == nil {
|
||||
return p, nil
|
||||
}
|
||||
for _, d := range []string{"/usr/sbin", "/sbin", "/usr/local/sbin"} {
|
||||
// /usr/libexec: Red Hat's mariadbd.
|
||||
for _, d := range []string{"/usr/sbin", "/sbin", "/usr/local/sbin", "/usr/libexec"} {
|
||||
p := filepath.Join(d, name)
|
||||
if fi, err := os.Stat(p); err == nil && !fi.IsDir() {
|
||||
return p, nil
|
||||
@@ -582,6 +668,55 @@ func (Local) WriteFile(path string, b []byte, mode fs.FileMode, owner string) er
|
||||
return os.Rename(tmp, path)
|
||||
}
|
||||
|
||||
// Download fetches url to path through a temporary file beside it, kept
|
||||
// only when its SHA-256 is sha.
|
||||
func (Local) Download(ctx context.Context, url, path, sha string) error {
|
||||
if f, err := os.Open(path); err == nil { //nolint:gosec // our paths
|
||||
h := sha256.New()
|
||||
_, cerr := io.Copy(h, f)
|
||||
_ = f.Close()
|
||||
if cerr == nil && hex.EncodeToString(h.Sum(nil)) == sha {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { //nolint:gosec // system directories
|
||||
return err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
|
||||
defer cancel()
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("fetching %s: %w", url, err)
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("fetching %s: %s", url, resp.Status)
|
||||
}
|
||||
tmp := path + ".llmgw-new"
|
||||
f, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o644) //nolint:gosec // our paths
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
h := sha256.New()
|
||||
_, err = io.Copy(io.MultiWriter(f, h), io.LimitReader(resp.Body, 1<<30))
|
||||
if cerr := f.Close(); err == nil {
|
||||
err = cerr
|
||||
}
|
||||
if err != nil {
|
||||
_ = os.Remove(tmp)
|
||||
return fmt.Errorf("fetching %s: %w", url, err)
|
||||
}
|
||||
if got := hex.EncodeToString(h.Sum(nil)); got != sha {
|
||||
_ = os.Remove(tmp)
|
||||
return fmt.Errorf("%s has SHA-256 %s, not the %s llmgw was built with: not used", url, got, sha)
|
||||
}
|
||||
return os.Rename(tmp, path)
|
||||
}
|
||||
|
||||
// MkdirAll makes a directory owned by owner.
|
||||
func (Local) MkdirAll(path string, mode fs.FileMode, owner string) error {
|
||||
if err := os.MkdirAll(path, mode); err != nil {
|
||||
|
||||
@@ -32,6 +32,7 @@ func newFake() *fakeHost {
|
||||
"slurmctld -V": "slurm-wlm 23.11.4\n",
|
||||
"scontrol ping": "Slurmctld(primary) at node06 is UP\n",
|
||||
"slurmd -C": "NodeName=cuda01 CPUs=16 RealMemory=64000\nUpTime=1-00:00:00\n",
|
||||
"hostname -s": "node06\n",
|
||||
}, fail: map[string]bool{}, state: "IDLE+DRAIN", reason: slurm.ReasonNew}
|
||||
}
|
||||
|
||||
@@ -59,7 +60,7 @@ func (f *fakeHost) Run(_ context.Context, env []string, name string, args ...str
|
||||
case "disable":
|
||||
f.active[u] = false
|
||||
}
|
||||
return nil, nil
|
||||
return []byte(f.outputs[c]), nil
|
||||
case strings.HasPrefix(c, "scontrol show node"):
|
||||
return []byte("NodeName=cuda01 Arch=x86_64 State=" + f.state + " Partitions=gpu Reason=" + f.reason + " [root@now]\n"), nil
|
||||
case strings.HasPrefix(c, "scontrol update"):
|
||||
@@ -101,6 +102,16 @@ func (f *fakeHost) MkdirAll(string, fs.FileMode, string) error { return nil }
|
||||
|
||||
func (f *fakeHost) Remove(path string) error { delete(f.files, path); return nil }
|
||||
|
||||
func (f *fakeHost) Download(_ context.Context, url, path, sha string) error {
|
||||
c := "download " + url + " " + path + " " + sha
|
||||
f.ran = append(f.ran, c)
|
||||
if f.fail[c] || f.fail["download "+url] {
|
||||
return errors.New("fetch failed")
|
||||
}
|
||||
f.files[path] = []byte("fetched " + url)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeHost) count(prefix string) int {
|
||||
n := 0
|
||||
for _, c := range f.ran {
|
||||
@@ -118,7 +129,7 @@ type fakeInst struct {
|
||||
|
||||
func (i *fakeInst) InstallPackages(_ context.Context, by map[string][]string) error {
|
||||
i.got = by
|
||||
for _, b := range []string{"munged", "scontrol", "slurmd", "slurmctld"} {
|
||||
for _, b := range []string{"munged", "scontrol", "slurmd", "slurmctld", "slurmdbd", "mariadbd", "mariadb", "exportfs", "mount.nfs"} {
|
||||
i.host.bins[b] = true
|
||||
}
|
||||
return nil
|
||||
@@ -237,6 +248,33 @@ func TestController(t *testing.T) {
|
||||
if !st.GetSlurmctldRunning() || len(st.GetFilesSha256()) != 64 {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
// Accounting: MariaDB on this machine alone, the database, slurmdbd on
|
||||
// it, the cluster added, all before slurmctld.
|
||||
if !slices.Contains(inst.got["deb"], "slurmdbd") || !slices.Contains(inst.got["deb"], "mariadb-server") {
|
||||
t.Fatalf("installed %v", inst.got)
|
||||
}
|
||||
if !strings.Contains(string(h.files["/etc/mysql/mariadb.conf.d/90-llmgw-slurm.cnf"]), "bind-address=127.0.0.1") {
|
||||
t.Fatal("MariaDB, installed by llmgw, not kept to this machine")
|
||||
}
|
||||
dbd := string(h.files[slurm.DbdConfPath])
|
||||
if !strings.Contains(dbd, "DbdHost=node06\n") || !strings.Contains(dbd, "StoragePass="+slurm.DbdPassword(key)+"\n") ||
|
||||
h.owners[slurm.DbdConfPath] != "slurm" {
|
||||
t.Fatalf("slurmdbd.conf for %q:\n%s", h.owners[slurm.DbdConfPath], dbd)
|
||||
}
|
||||
if _, ok := h.files[acctSQLPath]; ok {
|
||||
t.Fatal("the database's statements, password and all, were left on disk")
|
||||
}
|
||||
if strings.Contains(strings.Join(h.ran, "\n"), slurm.DbdPassword(key)) {
|
||||
t.Fatal("the password was on a command line")
|
||||
}
|
||||
order := strings.Join(h.ran, "\n")
|
||||
if !strings.Contains(order, "sacctmgr -i add cluster llmgw") ||
|
||||
strings.Index(order, "slurmdbd") > strings.Index(order, "systemctl restart slurmctld") {
|
||||
t.Fatalf("accounting not before slurmctld: %v", h.ran)
|
||||
}
|
||||
if !st.GetSlurmdbdRunning() {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
sum := st.GetFilesSha256()
|
||||
|
||||
h.ran = nil
|
||||
@@ -279,7 +317,7 @@ func TestStop(t *testing.T) {
|
||||
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{}); st != nil {
|
||||
t.Fatalf("status %v for no part", st)
|
||||
}
|
||||
if h.count("systemctl disable --now slurmd") != 1 {
|
||||
if !slices.Contains(h.ran, "systemctl disable --now slurmd") {
|
||||
t.Fatalf("slurmd not stopped: %v", h.ran)
|
||||
}
|
||||
if _, ok := h.files[DropIn]; ok {
|
||||
@@ -306,3 +344,24 @@ func TestBadNames(t *testing.T) {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMariaDBThatWasThereKeepsItsSettings(t *testing.T) {
|
||||
h := newFake()
|
||||
for _, b := range []string{"munged", "scontrol", "slurmctld", "slurmdbd", "mariadbd", "mariadb", "exportfs", "mount.nfs"} {
|
||||
h.bins[b] = true
|
||||
}
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Controller: true, MungeKey: key,
|
||||
Files: map[string]string{"slurm.conf": "ClusterName=llmgw\n"}})
|
||||
if st.GetError() != "" {
|
||||
t.Fatal(st.GetError())
|
||||
}
|
||||
for p := range h.files {
|
||||
if strings.HasSuffix(p, ".cnf") {
|
||||
t.Fatalf("a MariaDB llmgw did not install had its settings changed: %s", p)
|
||||
}
|
||||
}
|
||||
if h.count("mariadb -e source") != 1 || !st.GetSlurmdbdRunning() {
|
||||
t.Fatalf("the database was not made in the MariaDB that was there: %v", h.ran)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,319 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// usersMarker lists, a line each, the logins llmgw made here: the ones it
|
||||
// may change and take away. A user that was here before is never touched.
|
||||
func (k *Keeper) usersMarker() string { return k.marker + "-users" }
|
||||
|
||||
// users keeps the Linux users want has, the same login and UID as on every
|
||||
// other machine, their keys where sshd reads them, and their homes when
|
||||
// this machine makes them; and takes away those llmgw made that are gone.
|
||||
// Answers how many it has. A user that cannot be made is said, the others
|
||||
// are made anyway.
|
||||
func (k *Keeper) users(ctx context.Context, want *llmgwv1.Slurm) (int32, error) {
|
||||
made := k.madeUsers()
|
||||
if !want.GetManageUsers() {
|
||||
return 0, k.removeUsers(ctx, made, nil)
|
||||
}
|
||||
var errs []error
|
||||
var keep []string
|
||||
n := int32(0)
|
||||
for _, u := range want.GetUsers() {
|
||||
err := k.user(ctx, u, slices.Contains(made, u.GetLogin()), want.GetMakeHomes())
|
||||
if errors.Is(err, errNotMade) {
|
||||
errs = append(errs, err)
|
||||
continue
|
||||
}
|
||||
keep = append(keep, u.GetLogin())
|
||||
if err != nil {
|
||||
errs = append(errs, err)
|
||||
continue
|
||||
}
|
||||
n++
|
||||
}
|
||||
if err := k.keepMade(keep); err != nil {
|
||||
return n, err
|
||||
}
|
||||
if err := k.removeUsers(ctx, made, keep); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
if err := k.sshConf(ctx, len(keep) > 0); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
// The controller is where they log in and submit from; a node only
|
||||
// while a job of theirs runs there.
|
||||
if err := k.nodeLogin(ctx, len(keep) > 0 && want.GetNode() && !want.GetController()); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
return n, errors.Join(errs...)
|
||||
}
|
||||
|
||||
// nodeLogin holds the users llmgw made here to their jobs when they log in
|
||||
// (pam_slurm_adopt), or no longer. Only they are: root and the site's
|
||||
// administrators log in as before, so a mistake here never locks out the
|
||||
// people who would put it right.
|
||||
func (k *Keeper) nodeLogin(ctx context.Context, on bool) error {
|
||||
old, err := k.host.ReadFile(slurm.PAMSSHD)
|
||||
if err != nil {
|
||||
if !on {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("reading %s: %w", slurm.PAMSSHD, err)
|
||||
}
|
||||
if on {
|
||||
if err := k.adoptModule(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
read := func(name string) (string, error) {
|
||||
b, err := k.host.ReadFile(filepath.Join(filepath.Dir(slurm.PAMSSHD), name))
|
||||
return string(b), err
|
||||
}
|
||||
stack, contained, err := slurm.PAMAdopt(string(old), k.usersMarker(), read, on)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if stack == string(old) {
|
||||
return nil
|
||||
}
|
||||
if on {
|
||||
k.log.Info("llmgw's Slurm users log in to this node only while a job of theirs runs here",
|
||||
"in_the_job", contained)
|
||||
if !contained {
|
||||
k.log.Warn("pam_systemd could not be skipped for them safely here: their ssh sessions are not held " +
|
||||
"to their job's cores and cards, only to having one")
|
||||
}
|
||||
} else {
|
||||
k.log.Info("llmgw's Slurm users are no longer held to their jobs when they log in here")
|
||||
}
|
||||
// Read by sshd for each login: nothing to reload.
|
||||
_, err = k.write(slurm.PAMSSHD, []byte(stack), 0o644, "")
|
||||
return err
|
||||
}
|
||||
|
||||
// adoptModule is there, or is installed.
|
||||
func (k *Keeper) adoptModule(ctx context.Context) error {
|
||||
if k.installed[slurm.AdoptModule] {
|
||||
return nil
|
||||
}
|
||||
find := func() bool {
|
||||
for _, d := range slurm.ModuleDirs {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-f", filepath.Join(d, slurm.AdoptModule)); err == nil {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
if !find() {
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("%s missing, and this agent installs nothing", slurm.AdoptModule)
|
||||
}
|
||||
k.log.Info("installing pam_slurm_adopt, which holds llmgw's Slurm users to their jobs here")
|
||||
if err := k.inst.InstallPackages(ctx, slurm.AdoptPackages); err != nil {
|
||||
return err
|
||||
}
|
||||
if !find() {
|
||||
return fmt.Errorf("%s is still missing after installing %s", slurm.AdoptModule,
|
||||
strings.Join(slurm.AdoptPackages["deb"], ", "))
|
||||
}
|
||||
}
|
||||
k.installed[slurm.AdoptModule] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// errNotMade is a user llmgw could not make here, nor took for its own.
|
||||
var errNotMade = errors.New("not made")
|
||||
|
||||
type notMade struct{ msg string }
|
||||
|
||||
func (e notMade) Error() string { return e.msg }
|
||||
func (e notMade) Is(target error) bool { return target == errNotMade }
|
||||
|
||||
// user keeps one user; ours is whether llmgw made it here before.
|
||||
func (k *Keeper) user(ctx context.Context, u *llmgwv1.SlurmUser, ours, makeHome bool) error {
|
||||
login, uid, home := u.GetLogin(), u.GetUid(), u.GetHome()
|
||||
if !slurm.ValidLogin(login) || uid < slurm.MinUID || !filepath.IsAbs(home) || strings.Contains(home, "..") {
|
||||
return notMade{fmt.Sprintf("the leader sent a user %q, UID %d, home %q", login, uid, home)}
|
||||
}
|
||||
id := strconv.FormatUint(uint64(uid), 10)
|
||||
gecos := strings.Map(func(r rune) rune {
|
||||
if r == ':' || r == ',' || r == '\n' || r < ' ' {
|
||||
return -1
|
||||
}
|
||||
return r
|
||||
}, u.GetName())
|
||||
byName, okName := k.passwd(ctx, login)
|
||||
byID, okID := k.passwd(ctx, id)
|
||||
switch {
|
||||
case okName && byName[2] != id:
|
||||
return notMade{fmt.Sprintf("%s is already a user here with UID %s, not llmgw's %s", login, byName[2], id)}
|
||||
case !okName && okID:
|
||||
return notMade{fmt.Sprintf("UID %s is already %s's here, so %s is not made", id, byID[0], login)}
|
||||
case !okName:
|
||||
if _, ok := k.group(ctx, login); !ok {
|
||||
if out, err := k.host.Run(ctx, nil, "groupadd", "-g", id, login); err != nil {
|
||||
return notMade{fmt.Sprintf("groupadd %s: %v: %s", login, err, lastLine(out))}
|
||||
}
|
||||
}
|
||||
k.log.Info("a Slurm user is made here, as llmgw's accounts say", "login", login, "uid", uid)
|
||||
if out, err := k.host.Run(ctx, nil, "useradd", "-u", id, "-g", id, "-M", "-d", home, "-s", "/bin/bash",
|
||||
"-c", gecos, login); err != nil {
|
||||
return notMade{fmt.Sprintf("useradd %s: %v: %s", login, err, lastLine(out))}
|
||||
}
|
||||
case ours && (byName[4] != gecos || byName[5] != home):
|
||||
if out, err := k.host.Run(ctx, nil, "usermod", "-c", gecos, "-d", home, login); err != nil {
|
||||
return fmt.Errorf("usermod %s: %v: %s", login, err, lastLine(out))
|
||||
}
|
||||
}
|
||||
keys := strings.Join(u.GetSshKeys(), "\n")
|
||||
file := filepath.Join(slurm.AuthorizedKeysDir, login)
|
||||
if keys == "" {
|
||||
_ = k.host.Remove(file)
|
||||
} else if _, err := k.write(file, []byte(keys+"\n"), 0o644, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
if makeHome {
|
||||
return k.home(ctx, login, id, home)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// home makes a user's home directory, from /etc/skel, when it is not there.
|
||||
func (k *Keeper) home(ctx context.Context, login, id, home string) error {
|
||||
if _, err := k.host.Run(ctx, nil, "test", "-d", home); err == nil {
|
||||
return nil
|
||||
}
|
||||
if err := k.host.MkdirAll(filepath.Dir(home), 0o755, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
k.log.Info("a Slurm user's home is made", "login", login, "home", home)
|
||||
for _, c := range [][]string{
|
||||
{"install", "-d", "-m", "0700", "-o", id, "-g", id, home},
|
||||
{"cp", "-rT", "/etc/skel", home},
|
||||
{"chown", "-R", id + ":" + id, home},
|
||||
} {
|
||||
if out, err := k.host.Run(ctx, nil, c[0], c[1:]...); err != nil {
|
||||
return fmt.Errorf("making %s's home: %s: %v: %s", login, c[0], err, lastLine(out))
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// removeUsers takes away the users llmgw made here that are not in keep.
|
||||
// Their homes and files stay; their UID is never given to another.
|
||||
func (k *Keeper) removeUsers(ctx context.Context, made, keep []string) error {
|
||||
var errs []error
|
||||
left := slices.Clone(keep)
|
||||
for _, login := range made {
|
||||
if slices.Contains(keep, login) {
|
||||
continue
|
||||
}
|
||||
k.log.Info("a Slurm user llmgw made is taken away; its home stays", "login", login)
|
||||
_ = k.host.Remove(filepath.Join(slurm.AuthorizedKeysDir, login))
|
||||
if _, ok := k.passwd(ctx, login); ok {
|
||||
if out, err := k.host.Run(ctx, nil, "userdel", login); err != nil {
|
||||
// Still logged in, or running something: tried again.
|
||||
errs = append(errs, fmt.Errorf("userdel %s: %v: %s", login, err, lastLine(out)))
|
||||
left = append(left, login)
|
||||
continue
|
||||
}
|
||||
}
|
||||
}
|
||||
if !slices.Equal(left, made) {
|
||||
if err := k.keepMade(left); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
}
|
||||
if len(left) == 0 {
|
||||
if err := k.sshConf(ctx, false); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
if err := k.nodeLogin(ctx, false); err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
}
|
||||
return errors.Join(errs...)
|
||||
}
|
||||
|
||||
func (k *Keeper) madeUsers() []string {
|
||||
b, err := k.host.ReadFile(k.usersMarker())
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return strings.Fields(string(b))
|
||||
}
|
||||
|
||||
func (k *Keeper) keepMade(logins []string) error {
|
||||
s := slices.Clone(logins)
|
||||
slices.Sort(s)
|
||||
s = slices.Compact(s)
|
||||
if len(s) == 0 {
|
||||
return k.host.Remove(k.usersMarker())
|
||||
}
|
||||
_, err := k.write(k.usersMarker(), []byte(strings.Join(s, "\n")+"\n"), 0o644, "")
|
||||
return err
|
||||
}
|
||||
|
||||
// sshConf has sshd read the users' keys from AuthorizedKeysDir, or no
|
||||
// longer.
|
||||
func (k *Keeper) sshConf(ctx context.Context, on bool) error {
|
||||
if !on {
|
||||
if _, err := k.host.ReadFile(slurm.SSHDropIn); err != nil {
|
||||
return nil
|
||||
}
|
||||
_ = k.host.Remove(slurm.SSHDropIn)
|
||||
return k.reloadSSH(ctx)
|
||||
}
|
||||
changed, err := k.write(slurm.SSHDropIn, []byte(slurm.SSHConf), 0o644, "")
|
||||
if err != nil || !changed {
|
||||
return err
|
||||
}
|
||||
if out, err := k.host.Run(ctx, nil, "sshd", "-t"); err != nil {
|
||||
_ = k.host.Remove(slurm.SSHDropIn)
|
||||
return fmt.Errorf("sshd refuses llmgw's setting for the users' keys, so it is not kept: %v: %s", err, lastLine(out))
|
||||
}
|
||||
k.log.Info("sshd reads the Slurm users' keys from llmgw too", "dir", slurm.AuthorizedKeysDir)
|
||||
return k.reloadSSH(ctx)
|
||||
}
|
||||
|
||||
// reloadSSH has sshd read its settings again, under Debian's name for it
|
||||
// or Red Hat's; one that is not running reads them when it starts.
|
||||
func (k *Keeper) reloadSSH(ctx context.Context) error {
|
||||
for _, u := range []string{"ssh", "sshd"} {
|
||||
if k.active(ctx, u) {
|
||||
return k.systemctl(ctx, "reload", u)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// passwd is getent's line for a user, by login or UID, in its fields.
|
||||
func (k *Keeper) passwd(ctx context.Context, key string) ([]string, bool) {
|
||||
out, err := k.host.Run(ctx, nil, "getent", "passwd", key)
|
||||
f := strings.Split(strings.TrimSpace(string(out)), ":")
|
||||
if err != nil || len(f) < 7 {
|
||||
return nil, false
|
||||
}
|
||||
return f, true
|
||||
}
|
||||
|
||||
func (k *Keeper) group(ctx context.Context, key string) ([]string, bool) {
|
||||
out, err := k.host.Run(ctx, nil, "getent", "group", key)
|
||||
f := strings.Split(strings.TrimSpace(string(out)), ":")
|
||||
if err != nil || len(f) < 3 {
|
||||
return nil, false
|
||||
}
|
||||
return f, true
|
||||
}
|
||||
@@ -190,6 +190,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newKBCmd(client, &asJSON),
|
||||
newTrainCmd(client, &asJSON),
|
||||
newSlurmCmd(client, &asJSON),
|
||||
newPricesCmd(client, &asJSON),
|
||||
newConfigCmd(client, &asJSON),
|
||||
newAuditCmd(client, &asJSON),
|
||||
newNotifyCmd(client, &asJSON),
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"text/tabwriter"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
func newPricesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
cmd := &cobra.Command{
|
||||
Use: "prices",
|
||||
Short: "What each model's tokens cost, from what the cluster costs",
|
||||
}
|
||||
var days int
|
||||
var apply bool
|
||||
var note string
|
||||
suggest := &cobra.Command{
|
||||
Use: "suggest",
|
||||
Short: "Each model's price from the GPU- and core-hours it held, over the tokens it served",
|
||||
Long: "Set auth.gpu_hour_cost first: what an hour of one GPU costs, in the units budgets are in (the\n" +
|
||||
"hardware written off over its years, power and room, the people who run it, over the GPUs and\n" +
|
||||
"the hours of a month); auth.core_hour_cost likewise for models on CPUs. Each model's cost is\n" +
|
||||
"the hours its replicas held, idle ones too, kept by the hour on the leader; spread over its\n" +
|
||||
"tokens, an output token counted as four input ones and a cached prompt token as a tenth of one,\n" +
|
||||
"it is its price. A model held and not used shows what that cost. --apply writes the prices\n" +
|
||||
"into auth.pricing, one settings version, as `ctl config set` does.",
|
||||
Example: " llmgw ctl config set auth.gpu_hour_cost 1.35\n" +
|
||||
" llmgw ctl prices suggest --days 30\n" +
|
||||
" llmgw ctl prices suggest --apply --note \"November's costs\"",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var s adminapi.PriceSuggestions
|
||||
if err := client().get(cmd.Context(), "/api/v1/prices/suggest?days="+url.QueryEscape(strconv.Itoa(days)), &s); err != nil {
|
||||
return err
|
||||
}
|
||||
w := cmd.OutOrStdout()
|
||||
if *asJSON {
|
||||
return json.NewEncoder(w).Encode(s)
|
||||
}
|
||||
fmt.Fprintf(w, "cost: %g a GPU-hour, %g a core-hour; the last %d days", s.GPUHourCost, s.CoreHourCost, s.Days)
|
||||
if !s.RecordedFrom.IsZero() && s.RecordedFrom.After(s.Since) {
|
||||
fmt.Fprintf(w, " (held kept only since %s)", s.RecordedFrom.Local().Format("2006-01-02 15:04"))
|
||||
}
|
||||
fmt.Fprintln(w)
|
||||
fmt.Fprintln(w, "prices a million tokens: input / output / cached input, in budget units")
|
||||
tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
|
||||
fmt.Fprintln(tw, "MODEL\tGPU-H\tCORE-H\tCOST\tTOKENS IN/OUT\tSUGGESTED\tNOW\tNOTE")
|
||||
var changed []adminapi.PriceSuggestion
|
||||
for _, m := range s.Models {
|
||||
sug := "-"
|
||||
if !m.Price.Zero() {
|
||||
sug = perMillion(m.Price.PromptPer1k, m.Price.CompletionPer1k, m.Price.CachedPer1k)
|
||||
if m.Price.PromptPer1k != m.Current.PromptPer1k || m.Price.CompletionPer1k != m.Current.CompletionPer1k ||
|
||||
m.Price.CachedPer1k != m.Current.CachedPer1k {
|
||||
changed = append(changed, m)
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(tw, "%s\t%.1f\t%.1f\t%.2f\t%s/%s\t%s\t%s\t%s\n", m.Model, m.GPUHours, m.CoreHours, m.Cost,
|
||||
count(m.Prompt), count(m.Completion), sug,
|
||||
perMillion(m.Current.PromptPer1k, m.Current.CompletionPer1k, m.Current.CachedPer1k), m.Note)
|
||||
}
|
||||
if err := tw.Flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
if !apply {
|
||||
if len(changed) > 0 {
|
||||
fmt.Fprintf(w, "%d model(s) priced otherwise now; --apply writes the suggestions\n", len(changed))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if len(changed) == 0 {
|
||||
fmt.Fprintln(w, "nothing to change")
|
||||
return nil
|
||||
}
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
for _, m := range changed {
|
||||
for k, v := range map[string]int64{"prompt_per_1k": m.Price.PromptPer1k,
|
||||
"completion_per_1k": m.Price.CompletionPer1k, "cached_per_1k": m.Price.CachedPer1k} {
|
||||
if err := doc.SetYAML([]string{"auth", "pricing", m.Model, k}, strconv.FormatInt(v, 10)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
},
|
||||
}
|
||||
suggest.Flags().IntVar(&days, "days", 30, "the window, in days")
|
||||
suggest.Flags().BoolVar(&apply, "apply", false, "write the suggested prices into auth.pricing")
|
||||
suggest.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
cmd.AddCommand(suggest)
|
||||
return cmd
|
||||
}
|
||||
|
||||
// perMillion is micro-units a thousand tokens as budget units a million.
|
||||
func perMillion(in, out, cached int64) string {
|
||||
if in == 0 && out == 0 && cached == 0 {
|
||||
return "-"
|
||||
}
|
||||
f := func(v int64) string { return strconv.FormatFloat(float64(v)/1000, 'f', -1, 64) }
|
||||
return f(in) + " / " + f(out) + " / " + f(cached)
|
||||
}
|
||||
|
||||
func count(n int64) string {
|
||||
switch {
|
||||
case n >= 1e9:
|
||||
return fmt.Sprintf("%.1fB", float64(n)/1e9)
|
||||
case n >= 1e6:
|
||||
return fmt.Sprintf("%.1fM", float64(n)/1e6)
|
||||
case n >= 1e3:
|
||||
return fmt.Sprintf("%.1fk", float64(n)/1e3)
|
||||
}
|
||||
return strconv.FormatInt(n, 10)
|
||||
}
|
||||
+258
-20
@@ -3,12 +3,16 @@ package cli
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/url"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"text/tabwriter"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
@@ -27,7 +31,185 @@ func newSlurmCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
}
|
||||
cmd.AddCommand(newSlurmStatusCmd(client, asJSON), newSlurmSetupCmd(client), newSlurmOffCmd(client),
|
||||
newSlurmListCmd(client, "lend", "Let Slurm run jobs on nodes; llmgw's replicas move off first", true),
|
||||
newSlurmListCmd(client, "reclaim", "Take nodes back from Slurm; llmgw has them when their jobs end", false))
|
||||
newSlurmListCmd(client, "reclaim", "Take nodes back from Slurm; llmgw has them when their jobs end", false),
|
||||
newSlurmSharedCmd(client), newSlurmUsersCmd(client, asJSON), newSlurmContainersCmd(client))
|
||||
return cmd
|
||||
}
|
||||
|
||||
func newSlurmContainersCmd(client func() *ctlClient) *cobra.Command {
|
||||
cmd := &cobra.Command{
|
||||
Use: "containers",
|
||||
Short: "Jobs in containers: srun --container-image (pyxis and enroot)",
|
||||
Long: "On, the agents install enroot and pyxis on every machine of the Slurm cluster (from NVIDIA's\n" +
|
||||
"releases, at versions llmgw pins and checks) and, on GPU machines, nvidia-container-cli; once\n" +
|
||||
"every machine has them, Slurm loads pyxis and jobs may run in a container:\n\n" +
|
||||
" srun --gres=gpu:1 --container-image=nvcr.io#nvidia/pytorch:25.01-py3 python train.py\n\n" +
|
||||
"The shared path is in every container where it is on the host. Off, Slurm stops loading pyxis;\n" +
|
||||
"what was installed stays.",
|
||||
}
|
||||
var note string
|
||||
toggle := func(use, short string, on bool) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: use,
|
||||
Short: short,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
if !on {
|
||||
if doc.Node([]string{"slurm", "containers"}) != nil {
|
||||
doc.Unset([]string{"slurm", "containers"})
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return doc.SetYAML([]string{"slurm", "containers"}, "true")
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
cmd.AddCommand(toggle("on", "Install pyxis and enroot everywhere, and let jobs run in containers", true),
|
||||
toggle("off", "Slurm stops loading pyxis; what was installed stays", false))
|
||||
return cmd
|
||||
}
|
||||
|
||||
func newSlurmSharedCmd(client func() *ctlClient) *cobra.Command {
|
||||
var path, server, nfs, note string
|
||||
var off bool
|
||||
c := &cobra.Command{
|
||||
Use: "shared --path DIR (--server NODE | --nfs HOST:/EXPORT) | --off",
|
||||
Short: "Files every machine of the Slurm cluster sees at one path",
|
||||
Long: "Home directories, datasets, the jobs' output and checkpoints, at the same path on the controller\n" +
|
||||
"and every node. Three ways:\n" +
|
||||
" --nfs HOST:/EXPORT the site's own NFS export, mounted at --path on every machine;\n" +
|
||||
" --server NODE an llmgw node (the controller or a node) exports --path over NFS to the\n" +
|
||||
" others, for a site with no storage of its own: one machine, not highly\n" +
|
||||
" available, no faster than its disk and network;\n" +
|
||||
" neither the site mounts --path itself (Lustre, BeeGFS, GPFS); llmgw only looks\n" +
|
||||
" that it is there.\n" +
|
||||
"The export answers the Slurm machines alone, NFS's port the cluster's networks. A machine\n" +
|
||||
"that gives up its mount does so when no job holds files on it.",
|
||||
Example: " llmgw ctl slurm shared --path /shared --server node06\n" +
|
||||
" llmgw ctl slurm shared --path /shared --nfs filer.site:/export/hpc\n" +
|
||||
" llmgw ctl slurm shared --path /lustre/hpc\n" +
|
||||
" llmgw ctl slurm shared --off",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
if off == (path != "") {
|
||||
return fmt.Errorf("give --path, or --off")
|
||||
}
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
if off {
|
||||
if doc.Node([]string{"slurm", "shared"}) != nil {
|
||||
doc.Unset([]string{"slurm", "shared"})
|
||||
}
|
||||
return nil
|
||||
}
|
||||
v := map[string]string{"path": path}
|
||||
if server != "" {
|
||||
v["server"] = server
|
||||
}
|
||||
if nfs != "" {
|
||||
v["nfs"] = nfs
|
||||
}
|
||||
b, _ := json.Marshal(v)
|
||||
return doc.SetYAML([]string{"slurm", "shared"}, string(b))
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(&path, "path", "", "where every machine sees the files, as /shared")
|
||||
c.Flags().StringVar(&server, "server", "", "the llmgw node that exports them over NFS")
|
||||
c.Flags().StringVar(&nfs, "nfs", "", "the site's NFS export, HOST:/EXPORT")
|
||||
c.Flags().BoolVar(&off, "off", false, "no shared files; mounts and exports llmgw made are taken away")
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
|
||||
func newSlurmUsersCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
cmd := &cobra.Command{
|
||||
Use: "users",
|
||||
Short: "The Linux users that run jobs, made from llmgw's accounts",
|
||||
Long: "With slurm.users \"llmgw\" every active account is a Linux user on every machine of the Slurm\n" +
|
||||
"cluster: its login from its e-mail address, a UID of llmgw's from 200000 up (never given to\n" +
|
||||
"another, even when the account goes), its home under the shared path. It logs in over SSH\n" +
|
||||
"with the keys given in the portal (Profile) or here. An account suspended or deleted loses\n" +
|
||||
"its user; its home and files stay. Off, users are the site's: from its directory (sssd), or\n" +
|
||||
"made by hand.",
|
||||
}
|
||||
list := &cobra.Command{
|
||||
Use: "list",
|
||||
Short: "Every user, with its login, UID and how many keys it has",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var us []slurmsvc.User
|
||||
if err := client().get(cmd.Context(), "/api/v1/slurm/users", &us); err != nil {
|
||||
return err
|
||||
}
|
||||
w := cmd.OutOrStdout()
|
||||
if *asJSON {
|
||||
return json.NewEncoder(w).Encode(us)
|
||||
}
|
||||
tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
|
||||
fmt.Fprintln(tw, "LOGIN\tUID\tACCOUNT\tSSH KEYS")
|
||||
for _, u := range us {
|
||||
fmt.Fprintf(tw, "%s\t%d\t%s\t%d\n", u.Login, u.UID, u.Email, len(u.SSHKeys))
|
||||
}
|
||||
return tw.Flush()
|
||||
},
|
||||
}
|
||||
var note string
|
||||
toggle := func(use, short, value string) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: use,
|
||||
Short: short,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
if value == "" {
|
||||
if doc.Node([]string{"slurm", "users"}) != nil {
|
||||
doc.Unset([]string{"slurm", "users"})
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return doc.SetYAML([]string{"slurm", "users"}, yamlString(value))
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
keys := &cobra.Command{
|
||||
Use: "keys ACCOUNT FILE",
|
||||
Short: "Replace an account's SSH public keys, from a file (- for standard input)",
|
||||
Long: "FILE holds OpenSSH public keys, one a line, as ~/.ssh/id_ed25519.pub; an empty one takes them all away.",
|
||||
Example: " llmgw ctl slurm users keys ayse@site.example ayse.pub\n" +
|
||||
" ssh-add -L | llmgw ctl slurm users keys ayse@site.example -",
|
||||
Args: cobra.ExactArgs(2),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var b []byte
|
||||
var err error
|
||||
if args[1] == "-" {
|
||||
b, err = io.ReadAll(io.LimitReader(cmd.InOrStdin(), 64<<10))
|
||||
} else {
|
||||
b, err = os.ReadFile(args[1])
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
body, _ := json.Marshal(map[string]string{"keys": string(b)})
|
||||
var u slurmsvc.User
|
||||
if err := client().put(cmd.Context(), "/api/v1/slurm/users/"+url.PathEscape(args[0])+"/ssh-keys", body, &u); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s (UID %d) has %d SSH key(s); the machines have them within a minute\n",
|
||||
u.Login, u.UID, len(u.SSHKeys))
|
||||
return nil
|
||||
},
|
||||
}
|
||||
cmd.AddCommand(list,
|
||||
toggle("on", "Make a Linux user for every active account, on every machine of the Slurm cluster", "llmgw"),
|
||||
toggle("off", "Leave users to the site; those llmgw made are taken away, their homes left", ""),
|
||||
keys)
|
||||
return cmd
|
||||
}
|
||||
|
||||
@@ -55,19 +237,32 @@ func newSlurmStatusCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
fmt.Fprintln(w, ": not heard from yet")
|
||||
case c.Error != "":
|
||||
fmt.Fprintf(w, ": %s\n", c.Error)
|
||||
case c.SlurmctldRunning && c.SlurmdbdRunning:
|
||||
fmt.Fprintf(w, ": slurmctld %s and slurmdbd running\n", c.Version)
|
||||
case c.SlurmctldRunning:
|
||||
fmt.Fprintf(w, ": slurmctld %s running\n", c.Version)
|
||||
fmt.Fprintf(w, ": slurmctld %s running, slurmdbd not (the jobs' records)\n", c.Version)
|
||||
default:
|
||||
fmt.Fprintln(w, ": slurmctld not running")
|
||||
}
|
||||
fmt.Fprintf(w, "shared files: %s\n", sharedLine(st.Shared))
|
||||
if st.Users != "" {
|
||||
fmt.Fprintf(w, "users: %d, made from llmgw's accounts\n", st.UserCount)
|
||||
if st.Accounts > 0 {
|
||||
fmt.Fprintf(w, "accounts: %d, one for each project (sbatch --account=ORG-TEAM-PROJECT)\n", st.Accounts)
|
||||
}
|
||||
}
|
||||
if line := containersLine(st); line != "" {
|
||||
fmt.Fprintf(w, "containers: %s\n", line)
|
||||
}
|
||||
tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
|
||||
fmt.Fprintln(tw, "NODE\tPHASE\tSLURM\tJOBS\tSLURMD\tNOTE")
|
||||
fmt.Fprintln(tw, "NODE\tPHASE\tSLURM\tJOBS\tSLURMD\tSHARED\tNOTE")
|
||||
for _, n := range st.Nodes {
|
||||
state, jobs, slurmd, note := "-", "-", "-", n.Describe()
|
||||
state, jobs, slurmd, shared, note := "-", "-", "-", "-", n.Describe()
|
||||
if r := n.Status; r != nil {
|
||||
state = or(r.State, "-")
|
||||
jobs = fmt.Sprint(r.Jobs)
|
||||
slurmd = map[bool]string{true: "running", false: "stopped"}[r.SlurmdRunning]
|
||||
shared = or(r.Shared, "-")
|
||||
if r.Error != "" {
|
||||
note = r.Error
|
||||
}
|
||||
@@ -75,7 +270,7 @@ func newSlurmStatusCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
if !n.Connected {
|
||||
note = "agent not connected; " + note
|
||||
}
|
||||
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\t%s\n", n.Node, n.Phase, state, jobs, slurmd, note)
|
||||
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n", n.Node, n.Phase, state, jobs, slurmd, shared, note)
|
||||
}
|
||||
if err := tw.Flush(); err != nil {
|
||||
return err
|
||||
@@ -96,7 +291,8 @@ func newSlurmSetupCmd(client func() *ctlClient) *cobra.Command {
|
||||
Use: "setup --controller NODE --nodes NODE,... | --all-nodes",
|
||||
Short: "Run Slurm on llmgw's nodes: slurmctld on one, slurmd on those named",
|
||||
Long: "Turns slurm.enabled on and names the nodes, or with --all-nodes every node. Their agents install munge and Slurm (22.05 or\n" +
|
||||
"newer) from the machines' own package sources. Every node starts drained in Slurm and stays\n" +
|
||||
"newer) from the machines' own package sources; the controller also slurmdbd, the jobs' records, in a\n" +
|
||||
"MariaDB on that machine alone (one that was there keeps its settings). Every node starts drained in Slurm and stays\n" +
|
||||
"llmgw's until `llmgw ctl slurm lend` gives it to Slurm.",
|
||||
Example: " llmgw ctl slurm setup --controller node06 --nodes cuda01,cuda02\n" +
|
||||
" llmgw ctl slurm setup --controller node06 --all-nodes",
|
||||
@@ -169,6 +365,43 @@ func newSlurmOffCmd(client func() *ctlClient) *cobra.Command {
|
||||
return c
|
||||
}
|
||||
|
||||
// containersLine says whether jobs may run in containers, and which
|
||||
// machines are not ready yet; empty when containers are off.
|
||||
func containersLine(st slurmsvc.Status) string {
|
||||
switch st.Containers {
|
||||
case "on":
|
||||
return "on (srun --container-image)"
|
||||
case "":
|
||||
return ""
|
||||
}
|
||||
var waiting []string
|
||||
for _, n := range st.Nodes {
|
||||
if n.Status == nil || n.Status.Containers == "" {
|
||||
waiting = append(waiting, n.Node)
|
||||
}
|
||||
}
|
||||
if st.Ctld == nil || st.Ctld.Containers == "" {
|
||||
if !slices.Contains(waiting, st.Controller) {
|
||||
waiting = append(waiting, st.Controller)
|
||||
}
|
||||
}
|
||||
return "installing pyxis and enroot; jobs take them once every machine has them (waiting: " +
|
||||
strings.Join(waiting, ", ") + ")"
|
||||
}
|
||||
|
||||
// sharedLine says where the shared files are and come from.
|
||||
func sharedLine(sh config.SharedStorage) string {
|
||||
switch {
|
||||
case sh.Path == "":
|
||||
return "none (`llmgw ctl slurm shared` adds them; a batch job's output stays on its node)"
|
||||
case sh.Server != "":
|
||||
return fmt.Sprintf("%s, exported over NFS by %s", sh.Path, sh.Server)
|
||||
case sh.NFS != "":
|
||||
return fmt.Sprintf("%s, mounted from %s", sh.Path, sh.NFS)
|
||||
}
|
||||
return sh.Path + ", mounted by the site"
|
||||
}
|
||||
|
||||
// newSlurmListCmd adds nodes to slurm.lent, or takes them out.
|
||||
func newSlurmListCmd(client func() *ctlClient, use, short string, add bool) *cobra.Command {
|
||||
var note string
|
||||
@@ -178,20 +411,7 @@ func newSlurmListCmd(client func() *ctlClient, use, short string, add bool) *cob
|
||||
Args: cobra.MinimumNArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
nodes := docList(doc, []string{"slurm", "nodes"})
|
||||
all := docBool(doc, []string{"slurm", "all_nodes"})
|
||||
lent := docList(doc, []string{"slurm", "lent"})
|
||||
for _, n := range args {
|
||||
switch {
|
||||
case add && !all && !slices.Contains(nodes, n):
|
||||
return fmt.Errorf("%s does not run slurmd: add it with `llmgw ctl slurm setup --nodes`", n)
|
||||
case add && !slices.Contains(lent, n):
|
||||
lent = append(lent, n)
|
||||
case !add:
|
||||
lent = slices.DeleteFunc(lent, func(l string) bool { return l == n })
|
||||
}
|
||||
}
|
||||
return setList(doc, []string{"slurm", "lent"}, lent)
|
||||
return lendChange(doc, args, add)
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -199,6 +419,24 @@ func newSlurmListCmd(client func() *ctlClient, use, short string, add bool) *cob
|
||||
return c
|
||||
}
|
||||
|
||||
// lendChange adds nodes to slurm.lent, or with add false takes them out.
|
||||
func lendChange(doc *settings.Doc, names []string, add bool) error {
|
||||
nodes := docList(doc, []string{"slurm", "nodes"})
|
||||
all := docBool(doc, []string{"slurm", "all_nodes"})
|
||||
lent := docList(doc, []string{"slurm", "lent"})
|
||||
for _, n := range names {
|
||||
switch {
|
||||
case add && !all && !slices.Contains(nodes, n):
|
||||
return fmt.Errorf("%s does not run slurmd: add it with `llmgw ctl slurm setup --nodes`", n)
|
||||
case add && !slices.Contains(lent, n):
|
||||
lent = append(lent, n)
|
||||
case !add:
|
||||
lent = slices.DeleteFunc(lent, func(l string) bool { return l == n })
|
||||
}
|
||||
}
|
||||
return setList(doc, []string{"slurm", "lent"}, lent)
|
||||
}
|
||||
|
||||
// docList reads a list of strings from the settings, empty when unset.
|
||||
func docList(doc *settings.Doc, path []string) []string {
|
||||
n := doc.Node(path)
|
||||
|
||||
@@ -84,3 +84,19 @@ func TestTheSummaryIsWrittenWhereItIsToldNotToTheTerminal(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestStatusSaysWhenAlertsGoNowhere(t *testing.T) {
|
||||
render := func(s adminapi.Summary) string {
|
||||
cmd := &cobra.Command{}
|
||||
var b bytes.Buffer
|
||||
cmd.SetOut(&b)
|
||||
printSummary(cmd, s)
|
||||
return b.String()
|
||||
}
|
||||
if out := render(adminapi.Summary{NoAlerts: true}); !strings.Contains(out, "ALERTS nowhere") {
|
||||
t.Errorf("no webhook was not called out:\n%s", out)
|
||||
}
|
||||
if out := render(adminapi.Summary{}); strings.Contains(out, "ALERTS") {
|
||||
t.Errorf("an alerts line with webhooks set:\n%s", out)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
func TestAPackageInstallThatIsNoEnginesIsNamed(t *testing.T) {
|
||||
if got := provisionLine("", "", "installing", "mariadb-server, munge, with apt-get"); got !=
|
||||
"system packages: installing: mariadb-server, munge, with apt-get" {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
if got := provisionLine("llamacpp", "b10991", "ready", ""); got != "llamacpp b10991: ready" {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoresAloneKeepTheMemory(t *testing.T) {
|
||||
m := catalog.Model{Name: "judge", Profiles: []catalog.Profile{{Name: "cpu", MinMemoryGB: 2}, {Name: "cpu-big", MinMemoryGB: 2}}}
|
||||
if got, err := keptMemory(m, ""); err != nil || got != 2 {
|
||||
t.Fatalf("got %d, %v", got, err)
|
||||
}
|
||||
m.Profiles[1].MinMemoryGB = 4
|
||||
if _, err := keptMemory(m, ""); err == nil {
|
||||
t.Fatal("profiles that differ were given one memory")
|
||||
}
|
||||
if got, err := keptMemory(m, "cpu-big"); err != nil || got != 4 {
|
||||
t.Fatalf("got %d, %v", got, err)
|
||||
}
|
||||
if _, err := keptMemory(m, "gpu"); err == nil {
|
||||
t.Fatal("a profile the model has not was taken")
|
||||
}
|
||||
}
|
||||
+34
-3
@@ -338,7 +338,7 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
|
||||
var memory, cores int
|
||||
var profile string
|
||||
c := &cobra.Command{
|
||||
Use: "resources <model> --memory GB [--cores N]",
|
||||
Use: "resources <model> [--memory GB] [--cores N]",
|
||||
Short: "Set what a model reserves of a node: memory, and CPU cores",
|
||||
Long: "--memory is the host RAM a replica needs: for a model with no GPU, its\n" +
|
||||
"weights, KV cache and engine together. Placement reserves it on the node,\n" +
|
||||
@@ -351,8 +351,20 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
|
||||
"--model M` moves them to the new figures.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if !cmd.Flags().Changed("memory") && !cmd.Flags().Changed("cores") {
|
||||
return fmt.Errorf("give --memory, --cores or both")
|
||||
}
|
||||
if !cmd.Flags().Changed("memory") {
|
||||
return fmt.Errorf("--memory is required")
|
||||
// Cores alone: the memory stays what it is.
|
||||
var cur catalog.Model
|
||||
if err := client().get(cmd.Context(), "/api/v1/models/"+args[0], &cur); err != nil {
|
||||
return err
|
||||
}
|
||||
kept, err := keptMemory(cur, profile)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
memory = kept
|
||||
}
|
||||
body, err := json.Marshal(adminapi.ResourcesRequest{Profile: profile, MemoryGB: memory, CPUCores: cores})
|
||||
if err != nil {
|
||||
@@ -369,12 +381,31 @@ func newResourcesCmd(client func() *ctlClient) *cobra.Command {
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().IntVar(&memory, "memory", 0, "host RAM a replica needs, in GB")
|
||||
c.Flags().IntVar(&memory, "memory", 0, "host RAM a replica needs, in GB; kept as it is when left out")
|
||||
c.Flags().IntVar(&cores, "cores", 0, "CPU cores a replica with no GPU computes on; 0 reserves none")
|
||||
c.Flags().StringVar(&profile, "profile", "", "one profile; every profile when left out")
|
||||
return c
|
||||
}
|
||||
|
||||
// keptMemory is the memory a model's profiles reserve now, for a change of
|
||||
// cores alone: one profile's, or every profile's when they agree.
|
||||
func keptMemory(m catalog.Model, profile string) (int, error) {
|
||||
memory := -1
|
||||
for _, p := range m.Profiles {
|
||||
if profile != "" && p.Name != profile {
|
||||
continue
|
||||
}
|
||||
if memory >= 0 && p.MinMemoryGB != memory {
|
||||
return 0, fmt.Errorf("%s's profiles reserve different memory: give --memory, or --profile", m.Name)
|
||||
}
|
||||
memory = p.MinMemoryGB
|
||||
}
|
||||
if memory < 0 {
|
||||
return 0, fmt.Errorf("%s has no profile %q", m.Name, profile)
|
||||
}
|
||||
return memory, nil
|
||||
}
|
||||
|
||||
func newClassCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "class <model> critical|standard|opportunistic|default",
|
||||
|
||||
+15
-4
@@ -114,7 +114,7 @@ engines are kept.`,
|
||||
}
|
||||
f := cmd.Flags()
|
||||
f.StringVar(&na.opts.Node, "node", "", "node id (default: the machine's hostname)")
|
||||
f.StringVar(&na.opts.User, "ssh-user", "", "user to log in as (default: the one before @ in the address, else whoever ran sudo, else you)")
|
||||
f.StringVar(&na.opts.User, "ssh-user", "", "user to log in as (default: the one before @ in the address, else the ssh config's for the machine, else whoever ran sudo, else you)")
|
||||
f.StringArrayVar(&na.servers, "server", nil, "control plane address the agent dials, https://host:7443; repeatable for failover (default: this machine)")
|
||||
f.StringArrayVar(&na.fabrics, "fabric", nil, "InfiniBand or RoCE fabric and its adapters, name:mlx5_0,mlx5_1; repeatable")
|
||||
f.StringVar(&na.opts.MgmtAddr, "mgmt-addr", "", "address engines are reached at (default: found by the agent)")
|
||||
@@ -228,8 +228,9 @@ func (na *nodeAdd) target(addr string) onboard.Options {
|
||||
if o.User == "" {
|
||||
// Under sudo, the person who ran it — their account is usually the
|
||||
// one that exists on the nodes too. root's own keys still do the
|
||||
// logging in, since that is who is running ssh.
|
||||
o.User = os.Getenv("SUDO_USER")
|
||||
// logging in, since that is who is running ssh. Only a fallback: a
|
||||
// User in the ssh config's block for the machine comes first.
|
||||
o.FallbackUser = os.Getenv("SUDO_USER")
|
||||
}
|
||||
return o
|
||||
}
|
||||
@@ -495,7 +496,17 @@ func engineEvent(engine, version, state, detail string) onboard.Event {
|
||||
level = onboard.Warn
|
||||
}
|
||||
return onboard.Event{Step: onboard.StepEngines, Level: level,
|
||||
Detail: fmt.Sprintf("%s %s: %s%s", engine, version, state, prefixed(": ", detail))}
|
||||
Detail: provisionLine(engine, version, state, detail)}
|
||||
}
|
||||
|
||||
// provisionLine is one line of what a node is installing. A package
|
||||
// install that is no engine's, Slurm's or MariaDB's, has no name or version.
|
||||
func provisionLine(engine, version, state, detail string) string {
|
||||
what := strings.TrimSpace(engine + " " + version)
|
||||
if what == "" {
|
||||
what = "system packages"
|
||||
}
|
||||
return fmt.Sprintf("%s: %s%s", what, state, prefixed(": ", detail))
|
||||
}
|
||||
|
||||
func prefixed(p, s string) string {
|
||||
|
||||
+131
-2
@@ -10,6 +10,7 @@ import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"math"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
@@ -36,6 +37,7 @@ import (
|
||||
"golang.org/x/sync/errgroup"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/audit"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/bench/eval"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/bundle"
|
||||
@@ -47,6 +49,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/fabrictest"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/federation"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/held"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
@@ -256,6 +259,7 @@ func newServerCmd() *cobra.Command {
|
||||
// The settings in force, for what reads them at each use.
|
||||
var current atomic.Pointer[config.Server]
|
||||
current.Store(&cfg)
|
||||
rt.SetSlurm(func() config.SlurmSettings { return current.Load().Slurm })
|
||||
var agentSvc *agentsvc.Service
|
||||
var agentHandler http.Handler
|
||||
var agentPath string
|
||||
@@ -416,6 +420,7 @@ func newServerCmd() *cobra.Command {
|
||||
adminAPI = api
|
||||
api.SetReleases(cfg.Releases)
|
||||
api.SetNotifier(notifier)
|
||||
api.SetWebhooks(func() int { return len(current.Load().Notify.Webhooks) })
|
||||
api.SetReplayer(rt.Replay)
|
||||
// Quality suites, asked of a model the way a replay is.
|
||||
evals = newEvalService(cfg, rt, admin, log)
|
||||
@@ -431,6 +436,27 @@ func newServerCmd() *cobra.Command {
|
||||
api.SetAgentRestarter(agentSvc.RestartAgents)
|
||||
}
|
||||
api.SetAudit(st)
|
||||
// Prices from what the cluster costs: what each model
|
||||
// holds, kept by the hour on the leader.
|
||||
api.SetPrices(st, func() adminapi.PriceConfig {
|
||||
au := current.Load().Auth
|
||||
cur := make(map[string]accounting.Price, len(au.Pricing))
|
||||
for m, p := range au.Pricing {
|
||||
cur[m] = accountingPrice(p)
|
||||
}
|
||||
return adminapi.PriceConfig{Basis: accounting.Basis{GPUHour: au.GPUHourCost, CoreHour: au.CoreHourCost},
|
||||
Current: cur, Default: accountingPrice(au.DefaultPrice)}
|
||||
})
|
||||
go (&held.Recorder{Inv: admin.Inventory(), Store: st, Log: log,
|
||||
Models: func() map[string]string {
|
||||
m := map[string]string{}
|
||||
for _, rep := range admin.Reconciler().Replicas() {
|
||||
m[rep.ID] = rep.Model
|
||||
}
|
||||
return m
|
||||
},
|
||||
Leader: func() bool { return admin.Leadership() == nil || admin.Leadership().IsLeader() },
|
||||
}).Run(ctx)
|
||||
// Knowledge bases, with the cluster's own models to embed a
|
||||
// probe (a collection's dimension) and an operator's search.
|
||||
api.SetKB(&adminapi.KBAdmin{Store: st, Embed: embedder(rt.Replay)})
|
||||
@@ -439,7 +465,12 @@ func newServerCmd() *cobra.Command {
|
||||
trainer := train.New(st, agentSvc, log)
|
||||
agentSvc.SetTrain(trainer)
|
||||
go trainer.Run(ctx)
|
||||
api.SetTrain(&adminapi.TrainAdmin{Store: st, Service: trainer})
|
||||
api.SetTrain(&adminapi.TrainAdmin{Store: st, Service: trainer, Log: log})
|
||||
// From the portal: a node with a card free enough, and
|
||||
// what was made published when it is done.
|
||||
trainer.SetPicker(api.PickTrainNode)
|
||||
trainer.SetOnDone(api.TrainDone)
|
||||
rt.SetTraining(api)
|
||||
// Slurm on the nodes the settings name, its munge key
|
||||
// made from a cluster secret.
|
||||
slurmSvc = slurmsvc.New(func() config.SlurmSettings { return current.Load().Slurm },
|
||||
@@ -452,6 +483,25 @@ func newServerCmd() *cobra.Command {
|
||||
}, log)
|
||||
// What slurm.conf held, for the leader that comes next.
|
||||
slurmSvc.SetMemory(slurmsvc.StoreMemory(st))
|
||||
// slurm.users "llmgw": a Linux user for each active account.
|
||||
slurmSvc.SetUsers(func(ctx context.Context) ([]slurmsvc.User, error) {
|
||||
us, err := st.SlurmUsers(ctx)
|
||||
out := make([]slurmsvc.User, 0, len(us))
|
||||
for _, u := range us {
|
||||
out = append(out, adminapi.SlurmUserView(u))
|
||||
}
|
||||
return out, err
|
||||
})
|
||||
api.SetSlurmUsers(st)
|
||||
// Slurm's accounts from the projects, and the jobs
|
||||
// that ended priced as the cluster's hours are.
|
||||
slurmSvc.SetAccounts(func(ctx context.Context) ([]slurm.Account, error) {
|
||||
return slurmAccounts(ctx, st)
|
||||
})
|
||||
slurmSvc.SetJobs(func(ctx context.Context, jobs []slurmsvc.JobRecord) error {
|
||||
au := current.Load().Auth
|
||||
return recordSlurmJobs(ctx, st, current.Load().Slurm.Cluster(), au.GPUHourCost, au.CoreHourCost, jobs, log)
|
||||
})
|
||||
agentSvc.SetSlurm(slurmSvc)
|
||||
go slurmSvc.Run(ctx)
|
||||
api.SetSlurm(slurmSvc)
|
||||
@@ -616,7 +666,7 @@ func newServerCmd() *cobra.Command {
|
||||
// takes longer than the mirror's usual answers.
|
||||
if adminAPI != nil {
|
||||
if up, err := mirror.NewFailoverClient(mirrors, cfg.Mirror.TLS, 10*time.Minute); err == nil {
|
||||
adminAPI.SetTrainMirror(up)
|
||||
adminAPI.SetTrainMirror(up, hf.New(cfg.Mirror.Registry, cfg.Mirror.Timeout))
|
||||
}
|
||||
}
|
||||
// Browsing the registry is what turns "paste a resolve URL"
|
||||
@@ -1406,3 +1456,82 @@ func runPresetsPass(ctx context.Context, api *adminapi.API, lead *control.Leader
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// accountingPrice is a configured price as accounting has it.
|
||||
func accountingPrice(p config.Price) accounting.Price {
|
||||
return accounting.Price{PromptPer1k: p.PromptPer1k, CompletionPer1k: p.CompletionPer1k, CachedPer1k: p.CachedPer1k,
|
||||
PerImage: p.PerImage, PerMesh: p.PerMesh}
|
||||
}
|
||||
|
||||
// slurmAccounts are Slurm's accounts from the store's projects, members and
|
||||
// Slurm users.
|
||||
func slurmAccounts(ctx context.Context, st *storepkg.DB) ([]slurm.Account, error) {
|
||||
ps, err := st.SlurmProjects(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ms, err := st.AllMemberships(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
us, err := st.SlurmUsers(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
logins := make(map[string]string, len(us))
|
||||
for _, u := range us {
|
||||
logins[u.UserID] = u.Login
|
||||
}
|
||||
tps := make([]slurm.TenancyProject, 0, len(ps))
|
||||
for _, p := range ps {
|
||||
tps = append(tps, slurm.TenancyProject{ID: p.ID, Name: p.Name, TeamID: p.TeamID, TeamName: p.TeamName,
|
||||
OrgID: p.OrgID, OrgName: p.OrgName})
|
||||
}
|
||||
tms := make([]slurm.TenancyMember, 0, len(ms))
|
||||
for _, m := range ms {
|
||||
tms = append(tms, slurm.TenancyMember{UserID: m.UserID, Scope: string(m.Scope), ScopeID: m.ScopeID})
|
||||
}
|
||||
return slurm.Accounts(tps, tms, logins), nil
|
||||
}
|
||||
|
||||
// recordSlurmJobs records the jobs that ended, each with its project (from
|
||||
// its account) and person (from its login), priced as the hours the
|
||||
// cluster's models hold are: a GPU job its cards' hours, any other its
|
||||
// cores'.
|
||||
func recordSlurmJobs(ctx context.Context, st *storepkg.DB, cluster string, gpuHour, coreHour float64,
|
||||
jobs []slurmsvc.JobRecord, log *slog.Logger) error {
|
||||
as, err := slurmAccounts(ctx, st)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
project := map[string]string{}
|
||||
for _, a := range as {
|
||||
if a.Project != "" {
|
||||
project[a.Name] = a.Project
|
||||
}
|
||||
}
|
||||
us, err := st.SlurmUsers(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
person := map[string]string{}
|
||||
for _, u := range us {
|
||||
person[u.Login] = u.UserID
|
||||
}
|
||||
rows := make([]storepkg.SlurmJob, 0, len(jobs))
|
||||
for _, j := range jobs {
|
||||
secs := j.Elapsed.Seconds()
|
||||
cost := float64(j.CPUs) * secs / 3600 * coreHour
|
||||
if j.GPUs > 0 {
|
||||
cost = float64(j.GPUs) * secs / 3600 * gpuHour
|
||||
}
|
||||
rows = append(rows, storepkg.SlurmJob{Key: cluster + "/" + j.ID, UserID: person[j.Login], ProjectID: project[j.Account],
|
||||
Login: j.Login, Account: j.Account, Partition: j.Partition, State: j.State, Start: j.Start, End: j.End,
|
||||
ElapsedSeconds: int64(secs), CPUs: int64(j.CPUs), GPUs: int64(j.GPUs), CostMicros: int64(math.Ceil(cost * 1e6))})
|
||||
}
|
||||
n, err := st.RecordSlurmJobs(ctx, rows)
|
||||
if n > 0 {
|
||||
log.Info("Slurm jobs that ended recorded, with their cost", "jobs", n)
|
||||
}
|
||||
return err
|
||||
}
|
||||
@@ -443,7 +443,7 @@ func (sa *serverAdd) storeAccess(ctx context.Context) error {
|
||||
return err
|
||||
}
|
||||
}
|
||||
sa.done("nodes will be told of both control planes within half a minute")
|
||||
sa.done("nodes will be told of every control plane within half a minute")
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -127,6 +127,10 @@ type serverInit struct {
|
||||
// slurm runs Slurm beside llmgw: slurmctld on this machine's node,
|
||||
// slurmd on every node, each llmgw's until it is lent.
|
||||
slurm bool
|
||||
// slurmShared is where Slurm's shared files come from: "here" (this
|
||||
// machine exports /shared), the site's export as HOST:/EXPORT, or
|
||||
// "none". slurmUsers is "llmgw" (users from the accounts) or "site".
|
||||
slurmShared, slurmUsers string
|
||||
// evictAfterIdle and minLifetime are on_demand's; zero leaves its own.
|
||||
evictAfterIdle, minLifetime time.Duration
|
||||
// standbys makes each machine of the replicated store a control plane
|
||||
@@ -262,7 +266,11 @@ func newServerInitCmd() *cobra.Command {
|
||||
f.BoolVar(&in.media, "media", false,
|
||||
"pictures and 3D: the nodes install ComfyUI, and a picture model and a 3D model are added once a card fits one")
|
||||
f.BoolVar(&in.slurm, "slurm", false,
|
||||
"Slurm for training and batch jobs: slurmctld on this machine, slurmd on every node, each llmgw's until 'llmgw ctl slurm lend'")
|
||||
"Slurm for training and batch jobs: slurmctld and slurmdbd (with a MariaDB) on this machine, slurmd on every node, each llmgw's until 'llmgw ctl slurm lend'")
|
||||
f.StringVar(&in.slurmShared, "slurm-shared", "here",
|
||||
"with --slurm, the jobs' shared files at /shared: here (this machine exports them over NFS), the site's HOST:/EXPORT, or none")
|
||||
f.StringVar(&in.slurmUsers, "slurm-users", config.UsersLlmgw,
|
||||
"with --slurm, who makes the Linux users that run jobs: llmgw (one for each account, SSH keys in the portal) or site")
|
||||
f.DurationVar(&in.evictAfterIdle, "evict-after-idle", 0,
|
||||
"how long a model loaded on demand goes unasked-for before another may have its card (0: 2m)")
|
||||
f.DurationVar(&in.minLifetime, "min-lifetime", 0,
|
||||
|
||||
@@ -144,6 +144,13 @@ func (in *serverInit) checkChoices() error {
|
||||
if err := (config.SlurmSettings{Enabled: true, Controller: localNodeID()}).Check(); err != nil {
|
||||
return fmt.Errorf("slurm: this machine's hostname: %w", err)
|
||||
}
|
||||
set := config.SlurmSettings{Enabled: true, Controller: localNodeID(), Shared: slurmShared(in.slurmShared)}
|
||||
if err := set.Check(); err != nil {
|
||||
return fmt.Errorf("--slurm-shared %q: here, none, or the site's HOST:/EXPORT: %w", in.slurmShared, err)
|
||||
}
|
||||
if u := in.slurmUsers; u != "" && u != config.UsersLlmgw && u != "site" {
|
||||
return fmt.Errorf("--slurm-users %q: llmgw or site", u)
|
||||
}
|
||||
}
|
||||
if in.vip != "" {
|
||||
ip, _, err := net.ParseCIDR(in.vip)
|
||||
@@ -329,7 +336,7 @@ func (in *serverInit) tuneConfig(body string) string {
|
||||
"engines:\n comfyui: "+provision.ComfyUIVersion()+"\npresets:\n auto: true\n")
|
||||
}
|
||||
if in.slurm {
|
||||
extra = append(extra, slurmSection(localNodeID()))
|
||||
extra = append(extra, slurmSection(localNodeID(), slurmShared(in.slurmShared), in.slurmUsers != "site"))
|
||||
}
|
||||
if in.evictAfterIdle > 0 || in.minLifetime > 0 {
|
||||
b := "# How models that load when asked for take turns on a card, chosen at\n# install.\non_demand:\n"
|
||||
@@ -347,14 +354,43 @@ func (in *serverInit) tuneConfig(body string) string {
|
||||
return body
|
||||
}
|
||||
|
||||
// slurmShared is --slurm-shared as settings: "here" is /shared exported
|
||||
// by this machine's node, "none" nothing, anything else the site's export.
|
||||
func slurmShared(v string) config.SharedStorage {
|
||||
switch v {
|
||||
case "", "none":
|
||||
return config.SharedStorage{}
|
||||
case "here":
|
||||
return config.SharedStorage{Path: "/shared", Server: localNodeID()}
|
||||
}
|
||||
return config.SharedStorage{Path: "/shared", NFS: v}
|
||||
}
|
||||
|
||||
// slurmSection is server.yaml's slurm block as an install chooses it:
|
||||
// slurmctld on this machine's node, slurmd on every node.
|
||||
func slurmSection(controller string) string {
|
||||
return "# Slurm beside llmgw, chosen at install: slurmctld on this machine, slurmd on\n" +
|
||||
// slurmctld on this machine's node, slurmd on every node, the shared files
|
||||
// and the users as chosen.
|
||||
func slurmSection(controller string, shared config.SharedStorage, users bool) string {
|
||||
b := "# Slurm beside llmgw, chosen at install: slurmctld, and slurmdbd with a MariaDB\n" +
|
||||
"# for the jobs' records, on this machine; slurmd on\n" +
|
||||
"# every node, those that join later too. Each node is llmgw's, drained in\n" +
|
||||
"# Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm; 'llmgw ctl slurm\n" +
|
||||
"# reclaim NODE' takes it back once its jobs end.\n" +
|
||||
"slurm:\n enabled: true\n controller: " + yamlString(controller) + "\n all_nodes: true\n"
|
||||
switch {
|
||||
case shared.Server != "":
|
||||
b += " # /shared on every Slurm machine, exported over NFS from this one\n" +
|
||||
" # ('llmgw ctl slurm shared' points it at the site's storage instead).\n" +
|
||||
" shared:\n path: " + yamlString(shared.Path) + "\n server: " + yamlString(shared.Server) + "\n"
|
||||
case shared.NFS != "":
|
||||
b += " # /shared on every Slurm machine, mounted from the site's export.\n" +
|
||||
" shared:\n path: " + yamlString(shared.Path) + "\n nfs: " + yamlString(shared.NFS) + "\n"
|
||||
}
|
||||
if users {
|
||||
b += " # A Linux user for each llmgw account, the same UID everywhere; SSH keys\n" +
|
||||
" # in the portal's Profile. Empty leaves users to the site (sssd, a directory).\n" +
|
||||
" users: llmgw\n"
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// localNodeID is the node id this machine's agent takes: its hostname.
|
||||
@@ -500,6 +536,12 @@ func (in *serverInit) initFlags() []string {
|
||||
}
|
||||
if in.slurm {
|
||||
add("--slurm")
|
||||
if in.slurmShared != "" && in.slurmShared != "here" {
|
||||
add("--slurm-shared", in.slurmShared)
|
||||
}
|
||||
if in.slurmUsers == "site" {
|
||||
add("--slurm-users", "site")
|
||||
}
|
||||
}
|
||||
if in.evictAfterIdle > 0 {
|
||||
add("--evict-after-idle", durationYAML(in.evictAfterIdle))
|
||||
|
||||
@@ -17,7 +17,7 @@ import (
|
||||
func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
|
||||
in := &serverInit{
|
||||
language: "tr", signupOpen: true, defaultRPM: 120, defaultTPM: 50000, maxKeyTTL: 30 * 24 * time.Hour,
|
||||
media: true, slurm: true, evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
|
||||
media: true, slurm: true, slurmShared: "here", slurmUsers: "llmgw", evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
|
||||
judge: judgeSpec{Fail: "open", Categories: []string{"Violent", "PII"}, Replicas: 3, Budget: 3 * time.Second},
|
||||
}
|
||||
j := in.judgeFor()
|
||||
@@ -43,6 +43,8 @@ func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
|
||||
t.Errorf("pictures and 3D: presets %v comfyui %q", cfg.Presets.Auto, cfg.Engines["comfyui"])
|
||||
case !cfg.Slurm.Enabled || !cfg.Slurm.AllNodes || cfg.Slurm.Controller != localNodeID():
|
||||
t.Errorf("slurm %+v", cfg.Slurm)
|
||||
case cfg.Slurm.Shared.Path != "/shared" || cfg.Slurm.Shared.Server != localNodeID() || cfg.Slurm.Users != "llmgw":
|
||||
t.Errorf("slurm's shared files and users %+v", cfg.Slurm)
|
||||
case cfg.OnDemand.EvictAfterIdle != time.Minute || cfg.OnDemand.MinLifetime != 90*time.Second:
|
||||
t.Errorf("on demand %+v", cfg.OnDemand)
|
||||
}
|
||||
|
||||
@@ -329,7 +329,7 @@ func (in *serverInit) ensureNode(ctx context.Context) (string, error) {
|
||||
|
||||
if classes, n := gpuClasses(ctx); n > 0 {
|
||||
in.kept("%d NVIDIA GPU(s) here: %s — the node will use them", n, strings.Join(classes, ", "))
|
||||
} else if hasNVIDIADevice() {
|
||||
} else if discovery.HasNVIDIADevice() {
|
||||
fmt.Fprintf(in.out, " !! an NVIDIA card is on the PCI bus but nvidia-smi does not answer: "+
|
||||
"install the driver and run init again to serve on it; until then this node is CPU only\n")
|
||||
} else {
|
||||
@@ -398,7 +398,9 @@ labels:
|
||||
// checks its own llama.cpp, the CUDA build when it has a GPU, and a new
|
||||
// machine is often busy with its own package updates first.
|
||||
client := in.adminAPI()
|
||||
last := ""
|
||||
// Each engine's line once, and again only when it changes: the engines'
|
||||
// states come back together on every look.
|
||||
seen := map[string]string{}
|
||||
err = waitFor(ctx, 20*time.Minute, "this node to be ready with an engine", func() (bool, error) {
|
||||
var nodes []adminapi.NodeView
|
||||
if err := client.get(ctx, "/api/v1/nodes", &nodes); err != nil {
|
||||
@@ -412,9 +414,9 @@ labels:
|
||||
return true, nil
|
||||
}
|
||||
for _, p := range n.Provision {
|
||||
if line := fmt.Sprintf("%s %s: %s %s", p.Engine, p.Version, p.State, p.Detail); line != last {
|
||||
fmt.Fprintf(in.out, " .. %s\n", strings.TrimSpace(line))
|
||||
last = line
|
||||
if line := provisionLine(p.Engine, p.Version, p.State, p.Detail); seen[p.Engine] != line {
|
||||
fmt.Fprintf(in.out, " .. %s\n", line)
|
||||
seen[p.Engine] = line
|
||||
}
|
||||
// Only llama.cpp is waited for: vLLM installs for minutes after,
|
||||
// and one this node cannot run is not a reason to stop here.
|
||||
@@ -432,24 +434,6 @@ labels:
|
||||
return nodeID, nil
|
||||
}
|
||||
|
||||
// hasNVIDIADevice reports an NVIDIA card on the PCI bus, driver or not, so a
|
||||
// machine whose GPU is invisible for want of a driver says so instead of
|
||||
// quietly serving on CPU.
|
||||
func hasNVIDIADevice() bool {
|
||||
vendors, _ := filepath.Glob("/sys/bus/pci/devices/*/vendor")
|
||||
for _, v := range vendors {
|
||||
b, err := os.ReadFile(v)
|
||||
if err == nil && strings.TrimSpace(string(b)) == "0x10de" {
|
||||
class, _ := os.ReadFile(filepath.Join(filepath.Dir(v), "class"))
|
||||
// 0x03xxxx is a display or 3D controller; NVIDIA also makes bridges.
|
||||
if strings.HasPrefix(strings.TrimSpace(string(class)), "0x03") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// --- the starter model ---
|
||||
|
||||
// starterModel is the catalog entry: a GPU profile first when there is a card,
|
||||
|
||||
@@ -156,6 +156,10 @@ func printSummary(cmd *cobra.Command, s adminapi.Summary) {
|
||||
if s.Stopped {
|
||||
fmt.Fprintln(out, "SERVICE STOPPED: no model runs and requests are answered 503 (llmgw start)")
|
||||
}
|
||||
if s.NoAlerts {
|
||||
fmt.Fprintln(out, "ALERTS nowhere: a lost node or a judge down is told nobody "+
|
||||
"(llmgw ctl config set notify.webhooks, then llmgw ctl notify test)")
|
||||
}
|
||||
if s.GuardOff {
|
||||
fmt.Fprintln(out, "GUARD off: no request is checked, whatever the policies say (llmgw ctl guard on)")
|
||||
}
|
||||
|
||||
+23
-14
@@ -21,9 +21,13 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
Long: "A node trains an adapter for a catalog model on pairs (llmgw ctl kb dataset\n" +
|
||||
"writes them from a knowledge base; read them through first). It trains with\n" +
|
||||
"its installed vLLM's torch and a small overlay, as the engines' user, on a card\n" +
|
||||
"with --gpu-memory free — beside an engine, if it leaves that much. What it\n" +
|
||||
"makes is kept by the control plane: llmgw ctl train download, or publish it\n" +
|
||||
"as the model's adapter.\n\n" +
|
||||
"with --gpu-memory free — beside an engine, if it leaves that much. Without\n" +
|
||||
"--base-repo the weights are found from where the model's came, without\n" +
|
||||
"--gpu-memory from their size, and without --node it goes to the node whose\n" +
|
||||
"card has the most free. What it makes is kept by the control plane: llmgw ctl\n" +
|
||||
"train download, or publish it as the model's adapter (--publish does that by\n" +
|
||||
"itself when it is done). People in a project can do all this from the portal's\n" +
|
||||
"\"Customise a model\" page.\n\n" +
|
||||
"An adapter learns how answers sound — words, format, tone — far better than\n" +
|
||||
"what they say. Facts belong to the knowledge base.",
|
||||
}
|
||||
@@ -33,11 +37,12 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
var gpuMem int
|
||||
var epochs, rank int
|
||||
var lr float64
|
||||
var noGGUF bool
|
||||
var noGGUF, publishWhenDone bool
|
||||
start := &cobra.Command{
|
||||
Use: "start NAME --model MODEL --base-repo OWNER/NAME --node NODE --data FILE",
|
||||
Use: "start NAME --model MODEL --data FILE",
|
||||
Short: "Queue an adapter's training on a node",
|
||||
Example: " llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \\\n" +
|
||||
Example: " llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct --data policies.jsonl --public --publish\n" +
|
||||
" llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \\\n" +
|
||||
" --base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
@@ -46,7 +51,7 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
return err
|
||||
}
|
||||
req := adminapi.TrainRequest{Name: args[0], Model: model, BaseRepo: base, BaseRevision: rev, Node: node,
|
||||
GPUMemoryMiB: gpuMem, Data: string(b), Params: map[string]any{}, Owner: owner, Public: public}
|
||||
GPUMemoryMiB: gpuMem, Data: string(b), Params: map[string]any{}, Owner: owner, Public: public, Publish: publishWhenDone}
|
||||
if cmd.Flags().Changed("epochs") {
|
||||
req.Params["epochs"] = epochs
|
||||
}
|
||||
@@ -69,8 +74,8 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
if v.Owner != "" {
|
||||
who = v.Owner + " only"
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "queued %s: %s for %s on %s, %d pairs; it will be seen by %s\n",
|
||||
v.ID, v.Name, v.Model, v.Node, v.Pairs, who)
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "queued %s: %s for %s on %s, %d pairs from %s, %d MiB of a card; it will be seen by %s\n",
|
||||
v.ID, v.Name, v.Model, or(v.Node, "the node with a card free enough"), v.Pairs, v.BaseRepo, v.GPUMemoryMiB, who)
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "follow it with: llmgw ctl train show %s --follow\n", v.ID)
|
||||
return nil
|
||||
},
|
||||
@@ -79,8 +84,9 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
f.StringVar(&model, "model", "", "the catalog model the adapter is for")
|
||||
f.StringVar(&base, "base-repo", "", "the weights to train on, as safetensors on Hugging Face (owner/name)")
|
||||
f.StringVar(&rev, "base-revision", "", "a revision of them (default: the newest)")
|
||||
f.StringVar(&node, "node", "", "the node that trains")
|
||||
f.IntVar(&gpuMem, "gpu-memory", 4096, "MiB of a card the job needs free")
|
||||
f.StringVar(&node, "node", "", "the node that trains (default: the one whose card has the most free)")
|
||||
f.IntVar(&gpuMem, "gpu-memory", 0, "MiB of a card the job needs free (default: from the weights' size)")
|
||||
f.BoolVar(&publishWhenDone, "publish", false, "publish it as the model's adapter when it is done, restarting a llama.cpp model's replicas")
|
||||
f.StringVar(&data, "data", "", "the pairs, NDJSON of {\"q\", \"a\"}")
|
||||
f.IntVar(&epochs, "epochs", 6, "passes over the pairs")
|
||||
f.IntVar(&rank, "rank", 16, "the adapter's rank")
|
||||
@@ -89,7 +95,7 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
f.StringVar(&owner, "owner", "", "who the adapter belongs to — org:, team:, project: (path or ID), user: or key: — and only they see it "+
|
||||
"(default: the knowledge base's owner, when the pairs come from one)")
|
||||
f.BoolVar(&public, "public", false, "every caller of the model sees it; refused for pairs from a knowledge base")
|
||||
for _, req := range []string{"model", "base-repo", "node", "data"} {
|
||||
for _, req := range []string{"model", "data"} {
|
||||
_ = start.MarkFlagRequired(req)
|
||||
}
|
||||
c.AddCommand(start)
|
||||
@@ -138,8 +144,11 @@ func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
}
|
||||
line := fmt.Sprintf("%s %s %s loss %s peak %s %s", j.State, trainProgress(j), j.Node, trainLoss(j), trainPeak(j), j.Message)
|
||||
if !follow {
|
||||
fmt.Fprintf(out, "%s: adapter %s for %s, owned by %s, on %s (base %s), %d pairs\n", j.ID, j.Name, j.Model,
|
||||
or(j.Owner, "everyone"), j.Node, j.BaseRepo, j.Pairs)
|
||||
fmt.Fprintf(out, "%s: adapter %s for %s, owned by %s, on %s (base %s), %d pairs, %d MiB of a card\n", j.ID, j.Name, j.Model,
|
||||
or(j.Owner, "everyone"), or(j.Node, "the node whose card has the most free"), j.BaseRepo, j.Pairs, j.GPUMemoryMiB)
|
||||
if j.Project != "" {
|
||||
fmt.Fprintf(out, "started from the portal in project %s by %s; published when done: %t\n", j.Project, j.CreatedBy, j.Publish)
|
||||
}
|
||||
fmt.Fprintln(out, line)
|
||||
if len(j.Artifacts) > 0 {
|
||||
fmt.Fprintf(out, "made: %s (llmgw ctl train download %s --kind KIND)\n", strings.Join(j.Artifacts, ", "), j.ID)
|
||||
|
||||
@@ -287,12 +287,18 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
|
||||
title: "Slurm", on: in.slurm,
|
||||
about: "A Slurm cluster beside llmgw, for training and other batch work: slurmctld on this machine, " +
|
||||
"slurmd on every node, those that join later too. The agents install munge and Slurm from the " +
|
||||
"machines' own package sources.\n\n" +
|
||||
"machines' own package sources, and on this machine slurmdbd, the jobs' records, with a MariaDB " +
|
||||
"that answers this machine alone (Slurm keeps its records in MySQL or MariaDB only, not in the " +
|
||||
"cluster's PostgreSQL). Its port 6819, with Slurm's others, answers the cluster's networks only.\n\n" +
|
||||
"Each node stays llmgw's, drained in Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm: " +
|
||||
"its models move off first. 'llmgw ctl slurm reclaim NODE' takes it back once its jobs end. Jobs " +
|
||||
"are submitted on this machine with sbatch and srun.",
|
||||
"are submitted on this machine with sbatch and srun.\n\n" +
|
||||
"Every Slurm machine sees /shared, exported over NFS from this machine (homes, datasets, the jobs' " +
|
||||
"output); 'llmgw ctl slurm shared' points it at the site's own storage instead. Each llmgw account " +
|
||||
"is a Linux user there, the same UID on every machine, logging in with the SSH keys given in the " +
|
||||
"portal; 'llmgw ctl slurm users off' leaves users to the site's directory.",
|
||||
off: "no Slurm; 'llmgw ctl slurm setup' adds it later",
|
||||
summary: func() string { return "slurmctld here, slurmd on every node; none lent" },
|
||||
summary: func() string { return "slurmctld here, slurmd on every node, /shared from here; none lent" },
|
||||
needs: func() string {
|
||||
if !f.node.on {
|
||||
return "slurmctld runs on this machine's node: turn the node on, or Slurm off"
|
||||
|
||||
@@ -24,6 +24,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/docgen"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
@@ -50,6 +51,7 @@ const (
|
||||
cFederation
|
||||
cMirror
|
||||
cJobs
|
||||
cSlurm
|
||||
cLogs
|
||||
cAccounts
|
||||
cKeys
|
||||
@@ -71,7 +73,7 @@ const (
|
||||
cCount
|
||||
)
|
||||
|
||||
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "MCP servers", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Sign-in", "Audit", "Manual",
|
||||
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "MCP servers", "Federation", "Mirror", "Jobs", "Slurm", "Logs", "Accounts", "API keys", "Organizations", "Sign-in", "Audit", "Manual",
|
||||
"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Serve the guardrails' judge", "Upgrade the cluster",
|
||||
"Restart all replicas", "Back up now", "Wipe this machine", "Leave"}
|
||||
|
||||
@@ -92,6 +94,8 @@ type consoleData struct {
|
||||
// federation is the other llmgw clusters whose models are offered
|
||||
// here; empty on a control plane without a store.
|
||||
federation adminapi.FederationView
|
||||
// slurm is the Slurm cluster beside llmgw; off when it has none.
|
||||
slurm slurmsvc.Status
|
||||
// presets is the models that come with llmgw as they stand here; nil
|
||||
// from a control plane that cannot say.
|
||||
presets *adminapi.PresetsView
|
||||
@@ -284,6 +288,7 @@ func (m consoleModel) fetch() tea.Cmd {
|
||||
_ = c.get(ctx, "/api/v1/jobs", &d.jobs)
|
||||
_ = c.get(ctx, "/api/v1/audit?since=7d&limit=300", &d.audit)
|
||||
_ = c.get(ctx, "/api/v1/federation", &d.federation)
|
||||
_ = c.get(ctx, "/api/v1/slurm", &d.slurm)
|
||||
var presets adminapi.PresetsView
|
||||
if c.get(ctx, "/api/v1/presets", &presets) == nil {
|
||||
d.presets = &presets
|
||||
@@ -588,6 +593,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
case cFederation:
|
||||
m.dialog = "error"
|
||||
m.dialogTitle, m.dialogText = peerDialog(m.data.federation.Peers[m.row])
|
||||
case cSlurm:
|
||||
m.menuD = m.slurmMenu(m.data.slurm.Nodes[m.row])
|
||||
case cServe:
|
||||
return m, m.fetchRepo(m.hf.models[m.row].ID)
|
||||
case cAccounts:
|
||||
@@ -1110,6 +1117,8 @@ func (m consoleModel) rowCount() int {
|
||||
return len(m.data.audit)
|
||||
case cFederation:
|
||||
return len(m.data.federation.Peers)
|
||||
case cSlurm:
|
||||
return len(m.data.slurm.Nodes)
|
||||
case cServe:
|
||||
return len(m.hf.models)
|
||||
case cAccounts:
|
||||
@@ -1369,6 +1378,8 @@ func (m consoleModel) panel() (title, body, keys string) {
|
||||
return m.auditPanel(pw)
|
||||
case cFederation:
|
||||
return m.federationPanel(pw)
|
||||
case cSlurm:
|
||||
return m.slurmPanel(pw)
|
||||
case cMonitor:
|
||||
return m.monitorPanel(pw)
|
||||
case cServe:
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
// The console's Slurm view: the Slurm cluster run beside llmgw (BACKLOG P),
|
||||
// each of its nodes and whose it is now, as `llmgw ctl slurm status` shows
|
||||
// them. A node is lent to Slurm, or taken back, from its row. Turning Slurm
|
||||
// on, and choosing its nodes, is `ctl slurm setup` or server.yaml's slurm
|
||||
// part: a form for it would be server.yaml written again.
|
||||
|
||||
func (m consoleModel) slurmPanel(pw int) (string, string, string) {
|
||||
keys := "→ or Enter open · ↑↓ choose · Q leave"
|
||||
if m.inPanel {
|
||||
keys = "↑↓ node · Enter lend or take back · ← back · Q leave"
|
||||
}
|
||||
st := m.data.slurm
|
||||
head := "\n " + sDim.Render("Slurm beside llmgw, on its nodes: training and batch jobs.")
|
||||
if !st.Enabled {
|
||||
return "Slurm", head + "\n\n " + sDim.Render("Slurm is off. Turn it on with: "+
|
||||
"llmgw ctl slurm setup --controller NODE --nodes NODE,... (or --all-nodes)"), keys
|
||||
}
|
||||
ctl := sDim.Render("not heard from yet")
|
||||
switch c := st.Ctld; {
|
||||
case c == nil:
|
||||
case c.Error != "":
|
||||
ctl = sBad.Render(c.Error)
|
||||
case c.SlurmctldRunning && c.SlurmdbdRunning:
|
||||
ctl = sOK.Render("slurmctld " + c.Version + " and slurmdbd running")
|
||||
case c.SlurmctldRunning:
|
||||
ctl = sOK.Render("slurmctld "+c.Version+" running") + ", " + sWarn.Render("slurmdbd not")
|
||||
default:
|
||||
ctl = sBad.Render("slurmctld not running")
|
||||
}
|
||||
head += fmt.Sprintf("\n controller %s: %s", st.Controller, ctl)
|
||||
head += "\n " + sDim.Render(fmt.Sprintf("cluster %s, partition %s; sbatch and srun on %s", st.Cluster, st.Partition, st.Controller))
|
||||
if st.Shared.Path == "" {
|
||||
head += "\n shared files: " + sWarn.Render("none") + sDim.Render(" (llmgw ctl slurm shared; output stays on the job's node)")
|
||||
} else {
|
||||
head += "\n shared files: " + sharedLine(st.Shared)
|
||||
}
|
||||
if st.Users != "" {
|
||||
head += "\n " + fmt.Sprintf("users: %d from llmgw's accounts", st.UserCount) + sDim.Render(", SSH keys in the portal's Profile")
|
||||
} else {
|
||||
head += "\n " + sDim.Render("users: the site's (llmgw ctl slurm users on makes them from llmgw's accounts)")
|
||||
}
|
||||
if line := containersLine(st); line != "" {
|
||||
head += "\n containers: " + line
|
||||
}
|
||||
head += "\n " + sDim.Render("A node is llmgw's until lent; lent, its replicas move off first.")
|
||||
if len(st.Waiting) > 0 {
|
||||
head += "\n " + sWarn.Render(fmt.Sprintf("not in slurm.conf until they report their hardware: %v", st.Waiting))
|
||||
}
|
||||
head += "\n\n"
|
||||
if len(st.Nodes) == 0 {
|
||||
return "Slurm", head + " " + sDim.Render("No node runs slurmd."), keys
|
||||
}
|
||||
var rows [][]string
|
||||
for _, n := range st.Nodes {
|
||||
rows = append(rows, slurmRow(n))
|
||||
}
|
||||
return "Slurm", head + table([]string{"NODE", "WHOSE", "IN SLURM", "JOBS", "SLURMD", "SHARED", "NOTE"}, rows, m.row, pw, m.inPanel), keys
|
||||
}
|
||||
|
||||
func slurmRow(n slurmsvc.NodeStatus) []string {
|
||||
state, jobs, slurmd, shared, note := "-", "-", "-", "-", n.Describe()
|
||||
if r := n.Status; r != nil {
|
||||
state = or(r.State, "-")
|
||||
switch r.Shared {
|
||||
case "":
|
||||
case "missing":
|
||||
shared = sBad.Render(r.Shared)
|
||||
default:
|
||||
shared = sOK.Render(r.Shared)
|
||||
}
|
||||
jobs = fmt.Sprint(r.Jobs)
|
||||
slurmd = sBad.Render("stopped")
|
||||
if r.SlurmdRunning {
|
||||
slurmd = sOK.Render("running")
|
||||
}
|
||||
if r.Error != "" {
|
||||
note = sBad.Render(r.Error)
|
||||
}
|
||||
}
|
||||
if !n.Connected {
|
||||
note = sWarn.Render("agent not connected") + "; " + note
|
||||
}
|
||||
whose := string(n.Phase)
|
||||
switch n.Phase {
|
||||
case slurmsvc.Lent:
|
||||
whose = sOK.Render("Slurm's")
|
||||
case slurmsvc.Llmgw:
|
||||
whose = "llmgw's"
|
||||
case slurmsvc.Lending, slurmsvc.Reclaiming:
|
||||
whose = sWarn.Render(whose)
|
||||
}
|
||||
return []string{n.Node, whose, state, jobs, slurmd, shared, note}
|
||||
}
|
||||
|
||||
// slurmMenu is what can be done with one of Slurm's nodes.
|
||||
func (m consoleModel) slurmMenu(n slurmsvc.NodeStatus) *menuDialog {
|
||||
var items []menuItem
|
||||
if n.Phase == slurmsvc.Lent || n.Phase == slurmsvc.Lending {
|
||||
items = append(items, menuItem{"Take back from Slurm…", func(m *consoleModel) tea.Cmd {
|
||||
m.confirmD = &confirmDialog{title: "Take " + n.Node + " back",
|
||||
text: "Slurm starts no new job there. llmgw places replicas on it again once the jobs running there have ended.",
|
||||
yes: func(m *consoleModel) tea.Cmd { return m.slurmLend(n.Node, false) }}
|
||||
return nil
|
||||
}})
|
||||
} else {
|
||||
items = append(items, menuItem{"Lend to Slurm…", func(m *consoleModel) tea.Cmd {
|
||||
text := "Its replicas move off first, each replacement up before the original stops; then Slurm runs jobs there."
|
||||
if n.Replicas > 0 {
|
||||
text = fmt.Sprintf("Its %d replica(s) move off first, each replacement up before the original stops; "+
|
||||
"a model with nowhere else to go stops serving. Then Slurm runs jobs there.", n.Replicas)
|
||||
}
|
||||
m.confirmD = &confirmDialog{title: "Lend " + n.Node + " to Slurm", text: text,
|
||||
yes: func(m *consoleModel) tea.Cmd { return m.slurmLend(n.Node, true) }}
|
||||
return nil
|
||||
}})
|
||||
}
|
||||
return &menuDialog{title: n.Node, items: items}
|
||||
}
|
||||
|
||||
// slurmLend puts a node in slurm.lent, or takes it out, as `llmgw ctl slurm
|
||||
// lend|reclaim` does: the newest settings, changed and put back.
|
||||
func (m *consoleModel) slurmLend(node string, add bool) tea.Cmd {
|
||||
title, note, done := "Lend "+node, "lent to Slurm from the console", node+" is being lent to Slurm: its replicas move off first."
|
||||
if !add {
|
||||
title, note, done = "Take back "+node, "taken back from Slurm in the console",
|
||||
node+" is being taken back: llmgw has it once its Slurm jobs end."
|
||||
}
|
||||
m.busy = title
|
||||
c := m.client
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
var cur adminapi.SettingsView
|
||||
if err := c.get(ctx, "/api/v1/settings", &cur); err != nil {
|
||||
return actionDoneMsg{title: title, err: err}
|
||||
}
|
||||
doc, err := settings.FromBytes("settings", []byte(cur.Body))
|
||||
if err == nil {
|
||||
err = lendChange(doc, []string{node}, add)
|
||||
}
|
||||
var body []byte
|
||||
if err == nil {
|
||||
body, err = doc.Bytes()
|
||||
}
|
||||
if err != nil {
|
||||
return actionDoneMsg{title: title, err: err}
|
||||
}
|
||||
req, _ := json.Marshal(adminapi.SettingsRequest{Base: cur.Version, Body: string(body), Note: note})
|
||||
var v adminapi.SettingsView
|
||||
if err := c.put(ctx, "/api/v1/settings", req, &v); err != nil {
|
||||
return actionDoneMsg{title: title, err: err}
|
||||
}
|
||||
return actionDoneMsg{title: title, text: done}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
)
|
||||
|
||||
func TestTheConsoleLendsANodeToSlurmFromItsRow(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
var put adminapi.SettingsRequest
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v1/settings":
|
||||
_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 7,
|
||||
Body: "slurm:\n enabled: true\n controller: node06\n nodes: [node05, cuda01]\n"})
|
||||
case r.Method == http.MethodPut && r.URL.Path == "/api/v1/settings":
|
||||
b, _ := io.ReadAll(r.Body)
|
||||
mu.Lock()
|
||||
_ = json.Unmarshal(b, &put)
|
||||
mu.Unlock()
|
||||
_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 8})
|
||||
default:
|
||||
_, _ = w.Write([]byte("[]"))
|
||||
}
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
m := newConsoleModel(srv.URL, "/nonexistent/server.yaml")
|
||||
m.loaded, m.menu, m.inPanel = true, cSlurm, true
|
||||
m.data.slurm = slurmsvc.Status{Enabled: true, Cluster: "llmgw", Controller: "node06", Partition: "gpu",
|
||||
Ctld: &slurmsvc.Reported{SlurmctldRunning: true, SlurmdbdRunning: true, Version: "23.11.4"},
|
||||
Shared: config.SharedStorage{Path: "/shared", Server: "node06"}, Users: "llmgw", UserCount: 2,
|
||||
Nodes: []slurmsvc.NodeStatus{
|
||||
{Node: "node05", Phase: slurmsvc.Lent, Connected: true,
|
||||
Status: &slurmsvc.Reported{State: "MIXED", Jobs: 2, SlurmdRunning: true, Shared: "mounted"}},
|
||||
{Node: "cuda01", Phase: slurmsvc.Llmgw, Replicas: 3, Connected: true,
|
||||
Status: &slurmsvc.Reported{State: "IDLE+DRAIN", SlurmdRunning: true}},
|
||||
}}
|
||||
var tm tea.Model = m
|
||||
view := tm.View()
|
||||
for _, want := range []string{"slurmctld 23.11.4 and slurmdbd running", "/shared, exported over NFS by node06", "users: 2", "node05", "Slurm's", "MIXED", "cuda01", "llmgw's"} {
|
||||
if !strings.Contains(view, want) {
|
||||
t.Errorf("the Slurm view has no %q:\n%s", want, view)
|
||||
}
|
||||
}
|
||||
|
||||
// cuda01, the second row: lent, after saying its replicas move.
|
||||
tm = press(tm, "down", "enter")
|
||||
if !strings.Contains(tm.View(), "Lend to Slurm") {
|
||||
t.Fatalf("no lend in the menu:\n%s", tm.View())
|
||||
}
|
||||
tm = press(tm, "enter")
|
||||
if !strings.Contains(tm.View(), "3 replica(s) move off first") {
|
||||
t.Fatalf("the question does not say what moves:\n%s", tm.View())
|
||||
}
|
||||
tm = pressRun(tm, "y")
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if put.Base != 7 || !strings.Contains(put.Body, "lent:") || !strings.Contains(put.Body, "cuda01") {
|
||||
t.Fatalf("the settings put back: %+v", put)
|
||||
}
|
||||
if !strings.Contains(tm.View(), "cuda01 is being lent to Slurm") {
|
||||
t.Errorf("not said:\n%s", tm.View())
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheConsoleTakesALentNodeBack(t *testing.T) {
|
||||
m, _ := consoleWith(t, cSlurm)
|
||||
m.data.slurm = slurmsvc.Status{Enabled: true, Controller: "node06",
|
||||
Nodes: []slurmsvc.NodeStatus{{Node: "node05", Phase: slurmsvc.Lent, Connected: true}}}
|
||||
var tm tea.Model = m
|
||||
tm = press(tm, "enter")
|
||||
if v := tm.View(); !strings.Contains(v, "Take back from Slurm") || strings.Contains(v, "Lend to Slurm") {
|
||||
t.Fatalf("a lent node's menu:\n%s", v)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheConsoleSaysHowToTurnSlurmOn(t *testing.T) {
|
||||
m, _ := consoleWith(t, cSlurm)
|
||||
var tm tea.Model = m
|
||||
if !strings.Contains(tm.View(), "llmgw ctl slurm setup") {
|
||||
t.Fatalf("Slurm off says nothing of turning it on:\n%s", tm.View())
|
||||
}
|
||||
}
|
||||
@@ -171,7 +171,10 @@ func newUserStateCmd(open userOpener, verb string, state store.UserState, short
|
||||
if err := s.DeleteUserSessions(cmd.Context(), u.ID); err != nil {
|
||||
return err
|
||||
}
|
||||
cmd.PrintErrln("Their sessions were closed. Their API keys still work — revoke those separately.")
|
||||
// The keys are refused by their owner's state, which every
|
||||
// router reads again within auth.cache_ttl.
|
||||
cmd.PrintErrln("Their sessions were closed. Their API keys are refused within auth.cache_ttl " +
|
||||
"(30 s by default), and work again if they are reinstated.")
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s is now %s\n", u.Email, state)
|
||||
return nil
|
||||
|
||||
@@ -470,6 +470,15 @@ type Auth struct {
|
||||
Pricing map[string]Price `koanf:"pricing"`
|
||||
DefaultPrice Price `koanf:"default_price"`
|
||||
|
||||
// GPUHourCost is what an hour of one GPU costs the operator, in the
|
||||
// units budgets are in: the hardware written off over its years, its
|
||||
// power and room, and the people who run it, over the GPUs and the
|
||||
// hours of a month. With CoreHourCost for a CPU core, it is what
|
||||
// `llmgw ctl prices suggest` prices each model from: the hours its
|
||||
// replicas held, idle ones too, over the tokens it served.
|
||||
GPUHourCost float64 `koanf:"gpu_hour_cost"`
|
||||
CoreHourCost float64 `koanf:"core_hour_cost"`
|
||||
|
||||
// FlushInterval and BatchSize decide when usage records are written.
|
||||
FlushInterval time.Duration `koanf:"flush_interval"`
|
||||
BatchSize int `koanf:"batch_size"`
|
||||
|
||||
@@ -2,8 +2,10 @@ package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// SlurmSettings is a Slurm cluster run beside llmgw on some of its nodes
|
||||
@@ -38,6 +40,90 @@ type SlurmSettings struct {
|
||||
|
||||
// Partition is the one partition every node is in. Empty is "gpu".
|
||||
Partition string `koanf:"partition"`
|
||||
|
||||
// Shared is the file system every machine of the Slurm cluster sees at
|
||||
// the same path: home directories, datasets, the jobs' output and
|
||||
// checkpoints. Empty path: none, and a batch job's output stays on the
|
||||
// node it ran on.
|
||||
Shared SharedStorage `koanf:"shared"`
|
||||
|
||||
// Users is who keeps the Linux users that run jobs, the same on every
|
||||
// machine of the Slurm cluster: "llmgw" makes one for each active llmgw
|
||||
// account (its login from the e-mail address, a UID of llmgw's from
|
||||
// 200000 up, its SSH keys from the portal), let in to the controller,
|
||||
// and to a node only while a job of theirs runs there (pam_slurm_adopt);
|
||||
// empty leaves users to the site (a directory through sssd, or by hand).
|
||||
Users string `koanf:"users"`
|
||||
|
||||
// Containers lets jobs run in containers: `srun --container-image
|
||||
// nvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),
|
||||
// the GPUs and the shared path inside. The agents install both on
|
||||
// every Slurm machine; jobs take it once every machine has them.
|
||||
Containers bool `koanf:"containers"`
|
||||
}
|
||||
|
||||
// SharedStorage is where the Slurm cluster's shared files come from: the
|
||||
// site's NFS export (NFS), an NFS export llmgw keeps on one of its nodes
|
||||
// (Server), or, with neither, a file system the site mounts itself
|
||||
// (Lustre, BeeGFS, GPFS), which llmgw only looks for.
|
||||
type SharedStorage struct {
|
||||
// Path is where every machine sees it, as /shared.
|
||||
Path string `koanf:"path"`
|
||||
// NFS is the site's export, host:/path, mounted at Path on every
|
||||
// machine of the Slurm cluster.
|
||||
NFS string `koanf:"nfs"`
|
||||
// Server is the llmgw node, the controller or one of the nodes, that
|
||||
// exports Path over NFS to the others: for a site with no storage of
|
||||
// its own. One machine: not highly available, and no faster than its
|
||||
// disk and network.
|
||||
Server string `koanf:"server"`
|
||||
}
|
||||
|
||||
// UsersLlmgw has llmgw make the Slurm cluster's users from its accounts.
|
||||
const UsersLlmgw = "llmgw"
|
||||
|
||||
// HomeDir is where a user's home directory is: under the shared path when
|
||||
// there is one, else on each machine's own disk.
|
||||
func (s SlurmSettings) HomeDir(login string) string {
|
||||
if s.Shared.Path != "" {
|
||||
return path.Join(s.Shared.Path, "home", login)
|
||||
}
|
||||
return path.Join("/home", login)
|
||||
}
|
||||
|
||||
// nfsExport is host:/path.
|
||||
var nfsExport = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.:\[\]-]*:/[A-Za-z0-9_./-]*$`)
|
||||
|
||||
// sharedPath is an absolute path of plain names, which a mount unit's name
|
||||
// is made from.
|
||||
var sharedPath = regexp.MustCompile(`^(/[A-Za-z0-9_.-]+)+$`)
|
||||
|
||||
func (s SlurmSettings) checkShared() error {
|
||||
sh := s.Shared
|
||||
if sh.Path == "" {
|
||||
if sh.NFS != "" || sh.Server != "" {
|
||||
return fmt.Errorf("slurm.shared.path is empty: say where every machine sees the shared files, as /shared")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if !sharedPath.MatchString(sh.Path) || strings.Contains(sh.Path, "/.") {
|
||||
return fmt.Errorf("slurm.shared.path %q is not an absolute path of plain names, as /shared", sh.Path)
|
||||
}
|
||||
for _, sys := range []string{"/", "/etc", "/usr", "/var", "/var/lib", "/home", "/root", "/boot", "/opt", "/srv", "/tmp", "/proc", "/sys", "/dev", "/run"} {
|
||||
if path.Clean(sh.Path) == sys {
|
||||
return fmt.Errorf("slurm.shared.path %s is the system's own directory: take one of its own, as /shared", sh.Path)
|
||||
}
|
||||
}
|
||||
if sh.NFS != "" && sh.Server != "" {
|
||||
return fmt.Errorf("slurm.shared names both the site's export (nfs) and an llmgw node to export it (server): one of them")
|
||||
}
|
||||
if sh.NFS != "" && !nfsExport.MatchString(sh.NFS) {
|
||||
return fmt.Errorf("slurm.shared.nfs %q is not host:/path", sh.NFS)
|
||||
}
|
||||
if sh.Server != "" && sh.Server != s.Controller && !s.AllNodes && !slices.Contains(s.Nodes, sh.Server) {
|
||||
return fmt.Errorf("slurm.shared.server %q is neither the controller nor one of slurm.nodes", sh.Server)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// slurmName is what Slurm takes as a node, cluster or partition name.
|
||||
@@ -111,5 +197,8 @@ func (s SlurmSettings) Check() error {
|
||||
return fmt.Errorf("slurm.lent[%d] %q is not in slurm.nodes: only a node that runs slurmd can be lent", i, n)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
if s.Users != "" && s.Users != UsersLlmgw {
|
||||
return fmt.Errorf("slurm.users %q: %q has llmgw make them, empty leaves them to the site", s.Users, UsersLlmgw)
|
||||
}
|
||||
return s.checkShared()
|
||||
}
|
||||
@@ -35,6 +35,49 @@ func TestSlurmCheck(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlurmShared(t *testing.T) {
|
||||
base := SlurmSettings{Enabled: true, Controller: "c", Nodes: []string{"a"}}
|
||||
for _, sh := range []SharedStorage{{}, {Path: "/shared"}, {Path: "/shared", NFS: "filer.site:/export/hpc"},
|
||||
{Path: "/data/hpc", Server: "c"}, {Path: "/shared", Server: "a"}} {
|
||||
s := base
|
||||
s.Shared = sh
|
||||
if err := s.Check(); err != nil {
|
||||
t.Errorf("%+v refused: %v", sh, err)
|
||||
}
|
||||
}
|
||||
for want, sh := range map[string]SharedStorage{
|
||||
"path is empty": {NFS: "f:/x"},
|
||||
"plain names": {Path: "shared"},
|
||||
"system's own": {Path: "/home"},
|
||||
"one of them": {Path: "/shared", NFS: "f:/x", Server: "c"},
|
||||
"not host:/path": {Path: "/shared", NFS: "f/x"},
|
||||
"neither": {Path: "/shared", Server: "z"},
|
||||
} {
|
||||
s := base
|
||||
s.Shared = sh
|
||||
if err := s.Check(); err == nil || !strings.Contains(err.Error(), want) {
|
||||
t.Errorf("%+v: got %v, want %q", sh, err, want)
|
||||
}
|
||||
}
|
||||
s := base
|
||||
s.Shared = SharedStorage{Path: "/a/../etc"}
|
||||
if s.Check() == nil {
|
||||
t.Error("a path with .. taken")
|
||||
}
|
||||
s.Shared, s.Users = SharedStorage{}, "ldap"
|
||||
if s.Check() == nil {
|
||||
t.Error("slurm.users ldap taken")
|
||||
}
|
||||
s.Users, s.Shared = UsersLlmgw, SharedStorage{Path: "/shared"}
|
||||
if s.Check() != nil || s.HomeDir("ayse") != "/shared/home/ayse" {
|
||||
t.Errorf("home %s", s.HomeDir("ayse"))
|
||||
}
|
||||
s.Shared = SharedStorage{}
|
||||
if s.HomeDir("ayse") != "/home/ayse" {
|
||||
t.Errorf("home %s", s.HomeDir("ayse"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlurmAllNodes(t *testing.T) {
|
||||
s := SlurmSettings{Enabled: true, Controller: "c", AllNodes: true, Lent: []string{"joined"}}
|
||||
if err := s.Check(); err != nil {
|
||||
|
||||
@@ -98,6 +98,14 @@ type API struct {
|
||||
usage Usage
|
||||
quotas Quotas
|
||||
ctl *control.Controller
|
||||
// slurmUsers are the Linux users made from the accounts; nil without a store.
|
||||
slurmUsers SlurmUsers
|
||||
// prices are suggested from what each model held and served; nil
|
||||
// without a store.
|
||||
prices PriceStore
|
||||
priceCfg func() PriceConfig
|
||||
// webhooks counts the settings' notify.webhooks; nil does not say.
|
||||
webhooks func() int
|
||||
}
|
||||
|
||||
// New builds the API over a controller.
|
||||
@@ -139,6 +147,7 @@ func (a *API) Register(mux Mux) {
|
||||
a.registerTenancy(mux)
|
||||
a.registerTrain(mux)
|
||||
a.registerSlurm(mux)
|
||||
a.registerPrices(mux)
|
||||
a.registerGuard(mux)
|
||||
a.registerSettings(mux)
|
||||
a.registerAuth(mux)
|
||||
@@ -745,6 +754,10 @@ func (a *API) Nodes() []NodeView {
|
||||
}
|
||||
if h := n.Host; h.OSID != "" {
|
||||
v.OS = strings.TrimSpace(h.OSID + " " + h.OSVersion + " " + h.Arch)
|
||||
if h.CardWithoutDriver() {
|
||||
v.Warnings = append(v.Warnings, "an NVIDIA card is on the PCI bus but no driver is loaded: "+
|
||||
"this node serves on CPU only (install the driver, reboot, check nvidia-smi, restart the agent)")
|
||||
}
|
||||
if h.Arch == "amd64" && !h.HasCPUFlag("avx2") {
|
||||
v.Warnings = append(v.Warnings, "no AVX2: inference runs several times slower "+
|
||||
"(a VM needs CPU type \"host\")")
|
||||
|
||||
@@ -6,6 +6,10 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
|
||||
)
|
||||
|
||||
// SetWebhooks has the summary say when the settings name no webhook:
|
||||
// count is how many they name now.
|
||||
func (a *API) SetWebhooks(count func() int) { a.webhooks = count }
|
||||
|
||||
// SetNotifier gives the API the webhooks `ctl notify test` reaches.
|
||||
func (a *API) SetNotifier(n *notify.Notifier) { a.notifier = n }
|
||||
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// PriceStore is what a price is suggested from: what each model held of
|
||||
// the cluster, and the tokens it served.
|
||||
type PriceStore interface {
|
||||
HeldSince(ctx context.Context, since time.Time) ([]store.Held, time.Time, error)
|
||||
UsageByModel(ctx context.Context, since time.Time) ([]store.ModelUsage, error)
|
||||
}
|
||||
|
||||
// PriceConfig is the cost basis and the prices in force.
|
||||
type PriceConfig struct {
|
||||
Basis accounting.Basis
|
||||
Current map[string]accounting.Price
|
||||
Default accounting.Price
|
||||
}
|
||||
|
||||
// PriceSuggestions is a model's price from its cost, for each model.
|
||||
type PriceSuggestions struct {
|
||||
// Days is the window; RecordedFrom the first hour held is known for,
|
||||
// later than the window's start on a cluster that began keeping it
|
||||
// since, which makes the window shorter than asked.
|
||||
Days int `json:"days"`
|
||||
Since time.Time `json:"since"`
|
||||
RecordedFrom time.Time `json:"recorded_from,omitempty"`
|
||||
GPUHourCost float64 `json:"gpu_hour_cost"`
|
||||
CoreHourCost float64 `json:"core_hour_cost"`
|
||||
Models []PriceSuggestion `json:"models"`
|
||||
}
|
||||
|
||||
// PriceSuggestion is one model's.
|
||||
type PriceSuggestion struct {
|
||||
accounting.Suggestion
|
||||
Current accounting.Price `json:"current"`
|
||||
}
|
||||
|
||||
// SetPrices gives the API what prices are suggested from.
|
||||
func (a *API) SetPrices(st PriceStore, cfg func() PriceConfig) { a.prices, a.priceCfg = st, cfg }
|
||||
|
||||
func (a *API) registerPrices(mux Mux) {
|
||||
mux.HandleFunc("GET /api/v1/prices/suggest", a.suggestPrices)
|
||||
}
|
||||
|
||||
func (a *API) suggestPrices(w http.ResponseWriter, r *http.Request) {
|
||||
if a.prices == nil || a.priceCfg == nil {
|
||||
writeError(w, http.StatusNotImplemented, "prices are suggested from the store, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
days := 30
|
||||
if v := r.URL.Query().Get("days"); v != "" {
|
||||
n, err := strconv.Atoi(v)
|
||||
if err != nil || n < 1 || n > 366 {
|
||||
writeError(w, http.StatusBadRequest, "days is a whole number of days, 1 to 366")
|
||||
return
|
||||
}
|
||||
days = n
|
||||
}
|
||||
cfg := a.priceCfg()
|
||||
if cfg.Basis.GPUHour <= 0 && cfg.Basis.CoreHour <= 0 {
|
||||
writeError(w, http.StatusConflict, "no cost to price from: set auth.gpu_hour_cost (and auth.core_hour_cost for "+
|
||||
"models on CPUs), what an hour of one costs, in the units budgets are in")
|
||||
return
|
||||
}
|
||||
since := time.Now().Add(-time.Duration(days) * 24 * time.Hour)
|
||||
rows, first, err := a.prices.HeldSince(r.Context(), since)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
// Tokens from when holding was first recorded, so both sides cover the
|
||||
// same hours.
|
||||
from := since
|
||||
if first.After(from) {
|
||||
from = first
|
||||
}
|
||||
usage, err := a.prices.UsageByModel(r.Context(), from)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
held := map[string]accounting.Held{}
|
||||
for _, h := range rows {
|
||||
held[h.Model] = accounting.Held{GPUSeconds: h.GPUSeconds, CoreSeconds: h.CoreSeconds}
|
||||
}
|
||||
served := map[string]accounting.Served{}
|
||||
for _, u := range usage {
|
||||
served[u.Model] = accounting.Served{Prompt: u.Prompt, Completion: u.Completion}
|
||||
}
|
||||
out := PriceSuggestions{Days: days, Since: since.UTC(), RecordedFrom: first,
|
||||
GPUHourCost: cfg.Basis.GPUHour, CoreHourCost: cfg.Basis.CoreHour, Models: []PriceSuggestion{}}
|
||||
for _, s := range accounting.Suggest(held, served, cfg.Basis) {
|
||||
cur, ok := cfg.Current[s.Model]
|
||||
if !ok {
|
||||
cur = cfg.Default
|
||||
}
|
||||
out.Models = append(out.Models, PriceSuggestion{Suggestion: s, Current: cur})
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
@@ -85,6 +85,7 @@ var PathParams = map[string]string{
|
||||
"projects/{id}": "The project's ID.",
|
||||
"keys/{id}": "The key's ID, as listed (not the key itself).",
|
||||
"users/{id}": "The person's ID or email.",
|
||||
"users/{user}": "The person's ID or email.",
|
||||
"artifacts/{kind}": "What the job made: peft, a gzipped tar of the PEFT adapter, or gguf, the adapter for llama.cpp.",
|
||||
}
|
||||
|
||||
@@ -398,6 +399,21 @@ var Routes = []Route{
|
||||
"(drained in Slurm, no job left), lending (its replicas moving off), lent, reclaiming (drained in Slurm, jobs " +
|
||||
"still running) or unknown (its agent has not said).",
|
||||
Response: slurmsvc.Status{}},
|
||||
{Method: "GET", Path: "/api/v1/slurm/users", Summary: "The Linux users made from llmgw's accounts for the Slurm cluster",
|
||||
Description: "Every active account, with its login, UID and SSH keys; an account is given its login and UID the first " +
|
||||
"time it is listed. The machines have them when slurm.users is \"llmgw\".",
|
||||
Response: []slurmsvc.User{}},
|
||||
{Method: "PUT", Path: "/api/v1/slurm/users/{user}/ssh-keys", Summary: "Replace an account's SSH public keys",
|
||||
Description: "{user} is an ID or e-mail. keys is OpenSSH public keys, one a line, without options; empty takes them " +
|
||||
"all away. The machines have them within a minute.",
|
||||
Request: SlurmKeysRequest{}, Response: slurmsvc.User{}},
|
||||
|
||||
{Method: "GET", Path: "/api/v1/prices/suggest", Summary: "Each model's price, from what it held of the cluster",
|
||||
Description: "days (30 if not given) is the window. A model's cost is the GPU- and core-hours its replicas held, idle " +
|
||||
"ones too, at auth.gpu_hour_cost and auth.core_hour_cost; spread over its tokens, an output token four input " +
|
||||
"ones and a cached prompt token a tenth of one, it is price, in micro-units a thousand tokens as " +
|
||||
"auth.pricing takes them. current is the price in force. 409 with no cost set.",
|
||||
Response: PriceSuggestions{}},
|
||||
|
||||
{Method: "POST", Path: "/api/v1/train/jobs", Summary: "Queue a LoRA adapter's training on a node",
|
||||
Description: "The node trains when its agent is there, nothing else trains on it, and a card has gpu_memory_mib free; " +
|
||||
|
||||
@@ -1,15 +1,43 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// SlurmService is the Slurm cluster beside llmgw (BACKLOG P), as the leader
|
||||
// keeps it.
|
||||
type SlurmService interface {
|
||||
Status() slurmsvc.Status
|
||||
UsersChanged()
|
||||
}
|
||||
|
||||
// SlurmUsers are the Linux users llmgw makes from its accounts.
|
||||
type SlurmUsers interface {
|
||||
SlurmUsers(ctx context.Context) ([]store.SlurmUser, error)
|
||||
SlurmUserByID(ctx context.Context, userID string) (store.SlurmUser, error)
|
||||
SetSlurmSSHKeys(ctx context.Context, userID string, keys []string) error
|
||||
}
|
||||
|
||||
// SetSlurmUsers enables the Slurm users' endpoints.
|
||||
func (a *API) SetSlurmUsers(u SlurmUsers) { a.slurmUsers = u }
|
||||
|
||||
// SlurmKeysRequest is a person's SSH public keys, one a line, as
|
||||
// authorized_keys has them; empty takes them all away.
|
||||
type SlurmKeysRequest struct {
|
||||
Keys string `json:"keys"`
|
||||
}
|
||||
|
||||
// SlurmUserView is the user from a store row.
|
||||
func SlurmUserView(u store.SlurmUser) slurmsvc.User {
|
||||
return slurmsvc.User{Login: u.Login, UID: u.UID, Name: u.Name, Email: u.Email, SSHKeys: u.SSHKeys}
|
||||
}
|
||||
|
||||
// SetSlurm gives the API the Slurm service.
|
||||
@@ -17,6 +45,69 @@ func (a *API) SetSlurm(s SlurmService) { a.slurm = s }
|
||||
|
||||
func (a *API) registerSlurm(mux Mux) {
|
||||
mux.HandleFunc("GET /api/v1/slurm", a.slurmStatus)
|
||||
mux.HandleFunc("GET /api/v1/slurm/users", a.slurmUserList)
|
||||
mux.HandleFunc("PUT /api/v1/slurm/users/{user}/ssh-keys", a.slurmUserKeys)
|
||||
}
|
||||
|
||||
func (a *API) slurmUserList(w http.ResponseWriter, r *http.Request) {
|
||||
if a.slurmUsers == nil {
|
||||
writeError(w, http.StatusNotImplemented, "the Slurm users need the store, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
us, err := a.slurmUsers.SlurmUsers(r.Context())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
out := make([]slurmsvc.User, 0, len(us))
|
||||
for _, u := range us {
|
||||
out = append(out, SlurmUserView(u))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
// slurmUserKeys replaces a person's SSH keys; {user} is an ID or e-mail.
|
||||
func (a *API) slurmUserKeys(w http.ResponseWriter, r *http.Request) {
|
||||
if a.slurmUsers == nil {
|
||||
writeError(w, http.StatusNotImplemented, "the Slurm users need the store, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
var in SlurmKeysRequest
|
||||
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 64<<10)).Decode(&in); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
keys, err := slurm.ParseSSHKeys(in.Keys)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
u, err := a.findUser(ctx, r.PathValue("user"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusNotFound, strings.TrimPrefix(err.Error(), store.ErrNotFound.Error()+": "))
|
||||
return
|
||||
}
|
||||
if err := a.slurmUsers.SetSlurmSSHKeys(ctx, u.ID, keys); err != nil {
|
||||
code := http.StatusInternalServerError
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
code = http.StatusNotFound
|
||||
}
|
||||
writeError(w, code, err.Error())
|
||||
return
|
||||
}
|
||||
if a.slurm != nil {
|
||||
a.slurm.UsersChanged()
|
||||
}
|
||||
su, err := a.slurmUsers.SlurmUserByID(ctx, u.ID)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, SlurmUserView(su))
|
||||
}
|
||||
|
||||
func (a *API) slurmStatus(w http.ResponseWriter, _ *http.Request) {
|
||||
|
||||
@@ -50,6 +50,9 @@ type Summary struct {
|
||||
// GuardOff says the guardrails are turned off (guard.enabled: false):
|
||||
// no request is checked, whatever the policies say.
|
||||
GuardOff bool `json:"guard_off,omitempty"`
|
||||
// NoAlerts says the cluster's settings name no webhook: a model parked,
|
||||
// a node lost or a judge down is told nobody.
|
||||
NoAlerts bool `json:"no_alerts,omitempty"`
|
||||
}
|
||||
|
||||
// StoreMember is one member of a replicated store, as its Patroni reports it.
|
||||
@@ -160,6 +163,9 @@ func (a *API) Summary() Summary {
|
||||
if a.guard != nil && a.guard.Library != nil && !a.guard.Library().On() {
|
||||
s.GuardOff = true
|
||||
}
|
||||
if a.webhooks != nil && a.webhooks() == 0 {
|
||||
s.NoAlerts = true
|
||||
}
|
||||
for _, j := range a.Judges() {
|
||||
if j.Problem != "" {
|
||||
s.Judges = append(s.Judges, j)
|
||||
|
||||
@@ -153,3 +153,16 @@ func TestOneSharedCopyCountsAsMuchAsTwenty(t *testing.T) {
|
||||
t.Errorf("counted %d models; three nodes seeing one file is one model", f.Models)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheSummarySaysWhenNoWebhookIsSet(t *testing.T) {
|
||||
a, _ := setup(t, testModels(), gpuNode("n1", 8))
|
||||
n := 0
|
||||
a.SetWebhooks(func() int { return n })
|
||||
if !a.Summary().NoAlerts {
|
||||
t.Error("no webhook, not said")
|
||||
}
|
||||
n = 1
|
||||
if a.Summary().NoAlerts {
|
||||
t.Error("a webhook set, yet said there is none")
|
||||
}
|
||||
}
|
||||
@@ -10,15 +10,20 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
@@ -36,6 +41,11 @@ type TrainStore interface {
|
||||
TrainArtifact(ctx context.Context, job, kind string) ([]byte, error)
|
||||
TrainArtifactKinds(ctx context.Context, job string) ([]string, error)
|
||||
DeleteTrainJob(ctx context.Context, id string) error
|
||||
QueuedTrainJobs(ctx context.Context) ([]store.TrainJob, error)
|
||||
ActiveTrainJobs(ctx context.Context) ([]store.TrainJob, error)
|
||||
TrainJobsFor(ctx context.Context, projects []string, createdBy string, limit int) ([]store.TrainJob, error)
|
||||
SetTrainOwner(ctx context.Context, id, owner string) error
|
||||
SetTrainNote(ctx context.Context, id, message string) error
|
||||
}
|
||||
|
||||
// TrainAdmin is what the training endpoints need.
|
||||
@@ -45,6 +55,27 @@ type TrainAdmin struct {
|
||||
// Mirror keeps what a job made where the nodes fetch weights; nil
|
||||
// leaves publishing off.
|
||||
Mirror TrainMirror
|
||||
// Registry finds the weights to train a catalog model on, from where
|
||||
// its own came; nil has every job name them.
|
||||
Registry TrainRegistry
|
||||
// Log says what publishing by itself could not do.
|
||||
Log *slog.Logger
|
||||
|
||||
mu sync.Mutex
|
||||
bases map[string]trainBaseInfo // by the model's origin repository
|
||||
}
|
||||
|
||||
// TrainRegistry reads a repository of the model registry.
|
||||
type TrainRegistry interface {
|
||||
RepoAt(ctx context.Context, id, rev string) (hf.Repo, error)
|
||||
}
|
||||
|
||||
// trainBaseInfo is the safetensors weights a model is trained on.
|
||||
type trainBaseInfo struct {
|
||||
repo, revision string
|
||||
bytes int64
|
||||
err error
|
||||
at time.Time
|
||||
}
|
||||
|
||||
// TrainMirror is the model mirror, as publishing needs it.
|
||||
@@ -66,11 +97,11 @@ type TrainPublishView struct {
|
||||
// SetTrain enables the training endpoints.
|
||||
func (a *API) SetTrain(t *TrainAdmin) { a.train = t }
|
||||
|
||||
// SetTrainMirror gives publishing the mirror, which is set up after the
|
||||
// training endpoints are.
|
||||
func (a *API) SetTrainMirror(m TrainMirror) {
|
||||
// SetTrainMirror gives publishing the mirror, and finding a model's
|
||||
// weights the registry, which are set up after the training endpoints are.
|
||||
func (a *API) SetTrainMirror(m TrainMirror, reg TrainRegistry) {
|
||||
if a.train != nil {
|
||||
a.train.Mirror = m
|
||||
a.train.Mirror, a.train.Registry = m, reg
|
||||
}
|
||||
}
|
||||
|
||||
@@ -81,13 +112,23 @@ type TrainRequest struct {
|
||||
// Model is the catalog model the adapter is for.
|
||||
Model string `json:"model"`
|
||||
// BaseRepo and BaseRevision are the weights trained on: the model's own
|
||||
// on its registry, in safetensors (Hugging Face), not a GGUF.
|
||||
BaseRepo string `json:"base_repo"`
|
||||
// on its registry, in safetensors (Hugging Face), not a GGUF. Empty
|
||||
// finds them from where the model's came: the repository itself, or the
|
||||
// one its card says it was made from.
|
||||
BaseRepo string `json:"base_repo,omitempty"`
|
||||
BaseRevision string `json:"base_revision,omitempty"`
|
||||
// Node trains it; GPUMemoryMiB is what it needs of a card, which can be
|
||||
// what an engine there leaves free.
|
||||
Node string `json:"node"`
|
||||
GPUMemoryMiB int `json:"gpu_memory_mib"`
|
||||
// Node trains it; empty is the first with a card that has enough free.
|
||||
// GPUMemoryMiB is what it needs of a card, which can be what an engine
|
||||
// there leaves free; 0 is worked out from the weights' size.
|
||||
Node string `json:"node,omitempty"`
|
||||
GPUMemoryMiB int `json:"gpu_memory_mib,omitempty"`
|
||||
// Project is the project the job is started in: its members see and
|
||||
// manage it, and only one of its jobs trains at a time.
|
||||
Project string `json:"project,omitempty"`
|
||||
// Publish puts what the job makes on the mirror and makes it the
|
||||
// model's adapter by itself when the job is done, restarting a llama.cpp
|
||||
// model's replicas to load it.
|
||||
Publish bool `json:"publish,omitempty"`
|
||||
// Owner is who the adapter will belong to: org:, team:, project: (by
|
||||
// path or ID), user: or key:. Only callers there see it. When the
|
||||
// pairs come from one knowledge base, it defaults to that collection's
|
||||
@@ -129,6 +170,8 @@ type TrainJobView struct {
|
||||
StartedAt time.Time `json:"started_at,omitzero"`
|
||||
FinishedAt time.Time `json:"finished_at,omitzero"`
|
||||
CreatedBy string `json:"created_by,omitempty"`
|
||||
Project string `json:"project,omitempty"`
|
||||
Publish bool `json:"publish,omitempty"`
|
||||
// Artifacts are what it made: peft (a gzipped tar), gguf.
|
||||
Artifacts []string `json:"artifacts,omitempty"`
|
||||
}
|
||||
@@ -137,7 +180,8 @@ func trainView(j store.TrainJob) TrainJobView {
|
||||
return TrainJobView{ID: j.ID, Name: j.Name, Model: j.Model, BaseRepo: j.BaseRepo, BaseRevision: j.BaseRevision,
|
||||
Node: j.Node, Owner: j.Owner, GPUMemoryMiB: j.GPUMemoryMiB, Params: j.Params, GGUF: j.GGUF, Pairs: j.Pairs, State: j.State,
|
||||
Step: j.Step, Steps: j.Steps, Loss: j.Loss, Message: j.Message, PeakMiB: j.PeakMiB, CreatedAt: j.CreatedAt,
|
||||
StartedAt: j.StartedAt, FinishedAt: j.FinishedAt, CreatedBy: j.CreatedBy}
|
||||
StartedAt: j.StartedAt, FinishedAt: j.FinishedAt, CreatedBy: j.CreatedBy,
|
||||
Project: j.Project, Publish: j.Publish}
|
||||
}
|
||||
|
||||
func (a *API) registerTrain(mux Mux) {
|
||||
@@ -178,50 +222,122 @@ func (a *API) trainCreate(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
bad := func(format string, args ...any) {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf(format, args...))
|
||||
j, err := a.StartTraining(r.Context(), in, actorOf(r))
|
||||
if err != nil {
|
||||
writeTrainErr(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusCreated, trainView(j))
|
||||
}
|
||||
|
||||
// trainErr is a refusal, with the status it is answered with.
|
||||
type trainErr struct {
|
||||
code int
|
||||
msg string
|
||||
}
|
||||
|
||||
func (e *trainErr) Error() string { return e.msg }
|
||||
|
||||
func refused(code int, format string, args ...any) error {
|
||||
return &trainErr{code: code, msg: fmt.Sprintf(format, args...)}
|
||||
}
|
||||
|
||||
func writeTrainErr(w http.ResponseWriter, err error) {
|
||||
var te *trainErr
|
||||
if errors.As(err, &te) {
|
||||
writeError(w, te.code, te.msg)
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
}
|
||||
|
||||
// TrainRefusal is the reason a training request was refused, for someone
|
||||
// to read, and whether it was the request's fault rather than the
|
||||
// cluster's.
|
||||
func TrainRefusal(err error) (string, bool) {
|
||||
if err == nil {
|
||||
return "", false
|
||||
}
|
||||
var te *trainErr
|
||||
if errors.As(err, &te) {
|
||||
return te.msg, te.code < 500
|
||||
}
|
||||
return err.Error(), false
|
||||
}
|
||||
|
||||
// StartTraining checks a job and queues it, as actor.
|
||||
func (a *API) StartTraining(ctx context.Context, in TrainRequest, actor string) (store.TrainJob, error) {
|
||||
if a.train == nil {
|
||||
return store.TrainJob{}, refused(http.StatusNotImplemented,
|
||||
"training needs the store and the nodes' agents, which this control plane runs without")
|
||||
}
|
||||
bad := func(format string, args ...any) (store.TrainJob, error) {
|
||||
return store.TrainJob{}, refused(http.StatusBadRequest, format, args...)
|
||||
}
|
||||
if !adapterName.MatchString(in.Name) {
|
||||
bad("an adapter's name is letters, digits, . _ and -, up to 100: %q is not", in.Name)
|
||||
return
|
||||
return bad("an adapter's name is letters, digits, . _ and -, up to 100: %q is not", in.Name)
|
||||
}
|
||||
m, ok := a.ctl.Catalog().Resolve(in.Model)
|
||||
if !ok {
|
||||
bad("no model %s in the catalog", in.Model)
|
||||
return
|
||||
return bad("no model %s in the catalog", in.Model)
|
||||
}
|
||||
if _, _, _, taken := a.ctl.Catalog().ResolveAdapter(in.Name); taken || in.Name == m.Name {
|
||||
bad("%s is already a model's or an adapter's name", in.Name)
|
||||
return
|
||||
return bad("%s is already a model's or an adapter's name", in.Name)
|
||||
}
|
||||
if strings.Count(in.BaseRepo, "/") != 1 || strings.ContainsAny(in.BaseRepo, " :") {
|
||||
bad("base_repo is the weights' registry repository, owner/name: %q is not", in.BaseRepo)
|
||||
return
|
||||
if _, taken := a.ctl.Catalog().Resolve(in.Name); taken {
|
||||
return bad("%s is already a model's or an adapter's name", in.Name)
|
||||
}
|
||||
if _, ok := a.ctl.Inventory().Get(in.Node); !ok {
|
||||
bad("no node %s", in.Node)
|
||||
return
|
||||
queued, err := a.train.Store.QueuedTrainJobs(ctx)
|
||||
if err != nil {
|
||||
return store.TrainJob{}, err
|
||||
}
|
||||
active, err := a.train.Store.ActiveTrainJobs(ctx)
|
||||
if err != nil {
|
||||
return store.TrainJob{}, err
|
||||
}
|
||||
for _, j := range append(queued, active...) {
|
||||
if j.Name == in.Name {
|
||||
return bad("%s is already being trained (job %s)", in.Name, j.ID)
|
||||
}
|
||||
if in.Project != "" && j.Project == in.Project {
|
||||
return store.TrainJob{}, refused(http.StatusConflict,
|
||||
"this project already has a model training (%s): one at a time, so that one project does not hold every card", j.Name)
|
||||
}
|
||||
}
|
||||
var weights int64
|
||||
if in.BaseRepo == "" {
|
||||
b := a.trainBase(ctx, m)
|
||||
if b.err != nil {
|
||||
return store.TrainJob{}, b.err
|
||||
}
|
||||
in.BaseRepo, in.BaseRevision, weights = b.repo, b.revision, b.bytes
|
||||
} else if strings.Count(in.BaseRepo, "/") != 1 || strings.ContainsAny(in.BaseRepo, " :") {
|
||||
return bad("base_repo is the weights' registry repository, owner/name: %q is not", in.BaseRepo)
|
||||
}
|
||||
if in.Node != "" {
|
||||
if _, ok := a.ctl.Inventory().Get(in.Node); !ok {
|
||||
return bad("no node %s", in.Node)
|
||||
}
|
||||
}
|
||||
if in.GPUMemoryMiB <= 0 {
|
||||
bad("gpu_memory_mib is what the job needs of a card, in MiB")
|
||||
return
|
||||
if weights <= 0 {
|
||||
return bad("gpu_memory_mib is what the job needs of a card, in MiB")
|
||||
}
|
||||
in.GPUMemoryMiB = TrainMemoryMiB(weights)
|
||||
}
|
||||
pairs, colls, err := countPairs(in.Data)
|
||||
if err != nil {
|
||||
bad("%v", err)
|
||||
return
|
||||
return bad("%v", err)
|
||||
}
|
||||
owner, err := a.trainOwner(r.Context(), in, colls)
|
||||
owner, err := a.trainOwner(ctx, in, colls)
|
||||
if err != nil {
|
||||
bad("%v", err)
|
||||
return
|
||||
return bad("%v", err)
|
||||
}
|
||||
for k, v := range in.Params {
|
||||
lim, known := trainParams[k]
|
||||
f, isNum := v.(float64)
|
||||
if !known || !isNum || f < lim[0] || f > lim[1] {
|
||||
bad("params.%s: the trainer takes rank, alpha, dropout, epochs, lr, batch, max_len and seed, each a number in its range", k)
|
||||
return
|
||||
return bad("params.%s: the trainer takes rank, alpha, dropout, epochs, lr, batch, max_len and seed, each a number in its range", k)
|
||||
}
|
||||
}
|
||||
params := ""
|
||||
@@ -240,15 +356,132 @@ func (a *API) trainCreate(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
j := store.TrainJob{ID: store.NewID("train-"), Name: in.Name, Model: m.Name, BaseRepo: in.BaseRepo,
|
||||
BaseRevision: in.BaseRevision, Node: in.Node, Owner: owner, GPUMemoryMiB: in.GPUMemoryMiB, Params: params, GGUF: gguf,
|
||||
Data: []byte(in.Data), Pairs: pairs, CreatedBy: actorOf(r)}
|
||||
if err := a.train.Store.CreateTrainJob(r.Context(), j); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
Data: []byte(in.Data), Pairs: pairs, CreatedBy: actor, Project: in.Project, Publish: in.Publish}
|
||||
if err := a.train.Store.CreateTrainJob(ctx, j); err != nil {
|
||||
return store.TrainJob{}, err
|
||||
}
|
||||
// At once rather than at the next look, when the node is here.
|
||||
go a.train.Service.Dispatch(context.WithoutCancel(r.Context()))
|
||||
go a.train.Service.Dispatch(context.WithoutCancel(ctx))
|
||||
j.State = store.TrainQueued
|
||||
writeJSON(w, http.StatusCreated, trainView(j))
|
||||
return j, nil
|
||||
}
|
||||
|
||||
// TrainMemoryMiB is what training a LoRA adapter on weights of this size
|
||||
// takes of a card: the weights in 16 bits, as much again for what passes
|
||||
// through them, and the CUDA context. Measured at 2.3 GB for a 0.5B model.
|
||||
func TrainMemoryMiB(weightBytes int64) int {
|
||||
mib := int(weightBytes>>20)*2 + 1024
|
||||
return (mib + 255) / 256 * 256
|
||||
}
|
||||
|
||||
// trainBaseFor is how long what the registry said of a model's weights is
|
||||
// kept: it changes when the model is updated, which is rare.
|
||||
const trainBaseFor = time.Hour
|
||||
|
||||
// trainBase is the safetensors weights to train m on: its own origin's,
|
||||
// when they are safetensors, or those its card says it was made from — a
|
||||
// GGUF's base model.
|
||||
func (a *API) trainBase(ctx context.Context, m catalog.Model) trainBaseInfo {
|
||||
if m.Origin == nil || m.Origin.Repo == "" {
|
||||
return trainBaseInfo{err: refused(http.StatusBadRequest, "%s's weights did not come from the model registry, so "+
|
||||
"what to train it on is not known: an administrator can name them (llmgw ctl train start --base-repo)", m.Name)}
|
||||
}
|
||||
if a.train.Registry == nil {
|
||||
return trainBaseInfo{err: refused(http.StatusNotImplemented, "the model registry is not reachable from here "+
|
||||
"(mirror.registry): the weights to train %s on cannot be found", m.Name)}
|
||||
}
|
||||
a.train.mu.Lock()
|
||||
b, ok := a.train.bases[m.Origin.Repo]
|
||||
a.train.mu.Unlock()
|
||||
if ok && time.Since(b.at) < trainBaseFor {
|
||||
return b
|
||||
}
|
||||
b = a.findTrainBase(ctx, m)
|
||||
if b.err != nil {
|
||||
var te *trainErr
|
||||
if !errors.As(b.err, &te) {
|
||||
return b // the registry did not answer: asked again next time
|
||||
}
|
||||
}
|
||||
b.at = time.Now()
|
||||
a.train.mu.Lock()
|
||||
if a.train.bases == nil {
|
||||
a.train.bases = map[string]trainBaseInfo{}
|
||||
}
|
||||
a.train.bases[m.Origin.Repo] = b
|
||||
a.train.mu.Unlock()
|
||||
return b
|
||||
}
|
||||
|
||||
func (a *API) findTrainBase(ctx context.Context, m catalog.Model) trainBaseInfo {
|
||||
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
repo, err := a.train.Registry.RepoAt(ctx, m.Origin.Repo, "")
|
||||
if err != nil {
|
||||
return trainBaseInfo{err: fmt.Errorf("asking the model registry about %s: %w", m.Origin.Repo, err)}
|
||||
}
|
||||
candidates := []hf.Repo{repo}
|
||||
for _, id := range repo.BaseModel {
|
||||
if id == repo.ID || strings.Count(id, "/") != 1 {
|
||||
continue
|
||||
}
|
||||
r, err := a.train.Registry.RepoAt(ctx, id, "")
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
candidates = append(candidates, r)
|
||||
}
|
||||
gated := ""
|
||||
for _, r := range candidates {
|
||||
var n int64
|
||||
for _, f := range r.Files {
|
||||
if strings.HasSuffix(strings.ToLower(f.Path), ".safetensors") {
|
||||
n += f.Bytes
|
||||
}
|
||||
}
|
||||
if n == 0 {
|
||||
continue
|
||||
}
|
||||
if r.Gated.Is() {
|
||||
gated = r.ID
|
||||
continue
|
||||
}
|
||||
return trainBaseInfo{repo: r.ID, revision: r.Revision, bytes: n}
|
||||
}
|
||||
if gated != "" {
|
||||
return trainBaseInfo{err: refused(http.StatusBadRequest, "the weights to train %s on, %s, are gated on the registry: "+
|
||||
"the nodes cannot fetch them without an account that accepted their terms", m.Name, gated)}
|
||||
}
|
||||
return trainBaseInfo{err: refused(http.StatusBadRequest, "neither %s, where %s's weights came from, nor the model its card "+
|
||||
"names has safetensors weights to train on", m.Origin.Repo, m.Name)}
|
||||
}
|
||||
|
||||
// PickTrainNode is the node whose freest card has the most memory free,
|
||||
// if that is need, among those ready and not busy.
|
||||
func (a *API) PickTrainNode(need int, busy map[string]bool) string {
|
||||
best, bestFree := "", 0
|
||||
for _, n := range a.ctl.Inventory().Nodes() {
|
||||
if busy[n.ID] || n.State != inventory.Ready {
|
||||
continue
|
||||
}
|
||||
free := 0
|
||||
if n.Load != nil {
|
||||
for _, g := range n.Load.GPUs {
|
||||
if g.MemoryTotalMB > 0 {
|
||||
free = max(free, int(g.MemoryTotalMB-g.MemoryUsedMB))
|
||||
}
|
||||
}
|
||||
}
|
||||
if free == 0 {
|
||||
for _, g := range n.GPUs {
|
||||
free = max(free, g.MemoryTotalMB-g.MemoryUsedMB)
|
||||
}
|
||||
}
|
||||
if free >= need && free > bestFree {
|
||||
best, bestFree = n.ID, free
|
||||
}
|
||||
}
|
||||
return best
|
||||
}
|
||||
|
||||
// trainOwner is who a job's adapter will belong to, as scope:ID: one that
|
||||
@@ -453,73 +686,74 @@ func (a *API) trainPublish(w http.ResponseWriter, r *http.Request) {
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if a.train.Mirror == nil {
|
||||
writeError(w, http.StatusNotImplemented, "publishing needs the model mirror (mirror.api in the settings); "+
|
||||
"llmgw ctl train download, then llmgw mirror import there, does it by hand")
|
||||
v, err := a.publishTrained(r.Context(), j, actorOf(r))
|
||||
if err != nil {
|
||||
writeTrainErr(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
}
|
||||
|
||||
// publishTrained puts what a job made on the mirror and makes it the
|
||||
// model's adapter: the GGUF for a model llama.cpp serves, the PEFT
|
||||
// directory for one vLLM serves.
|
||||
func (a *API) publishTrained(ctx context.Context, j store.TrainJob, actor string) (TrainPublishView, error) {
|
||||
var none TrainPublishView
|
||||
if a.train.Mirror == nil {
|
||||
return none, refused(http.StatusNotImplemented, "publishing needs the model mirror (mirror.api in the settings); "+
|
||||
"llmgw ctl train download, then llmgw mirror import there, does it by hand")
|
||||
}
|
||||
if a.models == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store for the catalog")
|
||||
return
|
||||
return none, refused(http.StatusNotImplemented, "this control plane has no store for the catalog")
|
||||
}
|
||||
if j.State != store.TrainDone {
|
||||
writeError(w, http.StatusConflict, "the job has not finished: it is "+j.State)
|
||||
return
|
||||
return none, refused(http.StatusConflict, "%s", "the job has not finished: it is "+j.State)
|
||||
}
|
||||
m, found := a.ctl.Catalog().Resolve(j.Model)
|
||||
if !found {
|
||||
writeError(w, http.StatusConflict, "the model "+j.Model+" is no longer in the catalog")
|
||||
return
|
||||
return none, refused(http.StatusConflict, "%s", "the model "+j.Model+" is no longer in the catalog")
|
||||
}
|
||||
for _, ad := range m.Adapters {
|
||||
if ad.Name == j.Name {
|
||||
writeError(w, http.StatusConflict, j.Name+" is already "+m.Name+"'s adapter; train under another name")
|
||||
return
|
||||
return none, refused(http.StatusConflict, "%s", j.Name+" is already "+m.Name+"'s adapter; train under another name")
|
||||
}
|
||||
}
|
||||
llama := slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" })
|
||||
source := "trained by llmgw job " + j.ID + " on " + j.Node + " from " + j.BaseRepo
|
||||
var url string
|
||||
if llama {
|
||||
b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, "gguf")
|
||||
b, err := a.train.Store.TrainArtifact(ctx, j.ID, "gguf")
|
||||
if err != nil {
|
||||
writeError(w, http.StatusConflict, "the job made no GGUF, which llama.cpp needs; train again without --no-gguf")
|
||||
return
|
||||
return none, refused(http.StatusConflict, "the job made no GGUF, which llama.cpp needs; train again without --no-gguf")
|
||||
}
|
||||
res, err := a.train.Mirror.Upload(r.Context(), j.Name, j.ID, j.Name+".gguf", b, source)
|
||||
res, err := a.train.Mirror.Upload(ctx, j.Name, j.ID, j.Name+".gguf", b, source)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, "keeping it on the mirror: "+err.Error())
|
||||
return
|
||||
return none, refused(http.StatusBadGateway, "%s", "keeping it on the mirror: "+err.Error())
|
||||
}
|
||||
url = res.URL
|
||||
} else {
|
||||
b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, "peft")
|
||||
b, err := a.train.Store.TrainArtifact(ctx, j.ID, "peft")
|
||||
if err != nil {
|
||||
writeError(w, http.StatusConflict, "the job made no PEFT adapter")
|
||||
return
|
||||
return none, refused(http.StatusConflict, "the job made no PEFT adapter")
|
||||
}
|
||||
files, err := untarAll(b)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
return none, err
|
||||
}
|
||||
for name, data := range files {
|
||||
res, err := a.train.Mirror.Upload(r.Context(), j.Name, j.ID, name, data, source)
|
||||
res, err := a.train.Mirror.Upload(ctx, j.Name, j.ID, name, data, source)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, "keeping "+name+" on the mirror: "+err.Error())
|
||||
return
|
||||
return none, refused(http.StatusBadGateway, "%s", "keeping "+name+" on the mirror: "+err.Error())
|
||||
}
|
||||
url = res.URL[:strings.LastIndex(res.URL, "/")+1]
|
||||
}
|
||||
}
|
||||
ad := catalog.Adapter{Name: j.Name, Source: url, Owner: j.Owner}
|
||||
if err := checkAdapters(r.Context(), m, []catalog.Adapter{ad}); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
if err := checkAdapters(ctx, m, []catalog.Adapter{ad}); err != nil {
|
||||
return none, refused(http.StatusBadRequest, "%v", err)
|
||||
}
|
||||
if _, err := a.models.SetAdapters(r.Context(), m.Name, append(slices.Clone(m.Adapters), ad), actorOf(r)); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
if _, err := a.models.SetAdapters(ctx, m.Name, append(slices.Clone(m.Adapters), ad), actor); err != nil {
|
||||
return none, refused(http.StatusBadRequest, "%v", err)
|
||||
}
|
||||
v := TrainPublishView{Adapter: j.Name, Model: m.Name, Source: url, Owner: j.Owner}
|
||||
if llama {
|
||||
@@ -528,7 +762,48 @@ func (a *API) trainPublish(w http.ResponseWriter, r *http.Request) {
|
||||
} else {
|
||||
v.Note = "ask for model " + j.Name + "; a replica loads it on the first request"
|
||||
}
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
return v, nil
|
||||
}
|
||||
|
||||
func (a *API) trainLog() *slog.Logger {
|
||||
if a.train.Log == nil {
|
||||
return slog.New(slog.DiscardHandler)
|
||||
}
|
||||
return a.train.Log
|
||||
}
|
||||
|
||||
// TrainDone publishes what a job made, when it asked for that: on the
|
||||
// mirror, as its model's adapter, and for a llama.cpp model with the
|
||||
// model's replicas replaced, one at a time beside themselves, to load it.
|
||||
func (a *API) TrainDone(j store.TrainJob) {
|
||||
if !j.Publish || a.train == nil {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Minute)
|
||||
defer cancel()
|
||||
note := "published: ask for model " + j.Name
|
||||
v, err := a.publishTrained(ctx, j, "training:"+j.ID)
|
||||
if err != nil {
|
||||
note = "trained, but publishing it failed: " + err.Error()
|
||||
a.trainLog().Warn("a trained adapter could not be published", "job", j.ID, "adapter", j.Name, "err", err)
|
||||
} else if strings.HasPrefix(v.Note, "llama.cpp") {
|
||||
var ids []string
|
||||
for _, rep := range a.ctl.Reconciler().ReadyReplicas() {
|
||||
if rep.Model == v.Model {
|
||||
ids = append(ids, rep.ID)
|
||||
}
|
||||
}
|
||||
if len(ids) > 0 {
|
||||
if err := a.ctl.Reconciler().RestartPaced(reconciler.RestartPace{}, false, ids...); err != nil {
|
||||
note = "published, but " + v.Model + "'s replicas could not be restarted to load it: " + err.Error()
|
||||
} else {
|
||||
note = "published: " + v.Model + "'s replicas are restarting to load it; then ask for model " + j.Name
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := a.train.Store.SetTrainNote(ctx, j.ID, note); err != nil {
|
||||
a.trainLog().Warn("could not record a trained adapter's publishing", "job", j.ID, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// untarAll reads a gzipped tar of a PEFT directory's files.
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"slices"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Training from the portal's "customise a model" page: the page decides who
|
||||
// may do what (members of the job's project), and these do it.
|
||||
|
||||
// TrainableModels are the catalog's models an adapter can be trained for
|
||||
// from the portal: chat models llama.cpp or vLLM serve whose weights came
|
||||
// from the registry, so the weights to train on can be found.
|
||||
func (a *API) TrainableModels() []string {
|
||||
var out []string
|
||||
for _, m := range a.ctl.Catalog().Models() {
|
||||
if m.IsGuard() || m.IsExternal() || m.Capabilities.Embeddings || m.MakesImages() || m.MakesMeshes() ||
|
||||
m.Origin == nil || m.Origin.Repo == "" {
|
||||
continue
|
||||
}
|
||||
if slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" || p.Engine == "vllm" }) {
|
||||
out = append(out, m.Name)
|
||||
}
|
||||
}
|
||||
slices.Sort(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// Trainings are the jobs started in projects, or by createdBy outside any.
|
||||
func (a *API) Trainings(ctx context.Context, projects []string, createdBy string) ([]store.TrainJob, error) {
|
||||
if a.train == nil {
|
||||
return nil, nil
|
||||
}
|
||||
return a.train.Store.TrainJobsFor(ctx, projects, createdBy, 100)
|
||||
}
|
||||
|
||||
// Training is one job.
|
||||
func (a *API) Training(ctx context.Context, id string) (store.TrainJob, error) {
|
||||
if a.train == nil {
|
||||
return store.TrainJob{}, store.ErrNotFound
|
||||
}
|
||||
return a.train.Store.TrainJob(ctx, id)
|
||||
}
|
||||
|
||||
// Published is whether a job's adapter is its model's in the catalog.
|
||||
func (a *API) Published(j store.TrainJob) bool {
|
||||
m, ok := a.ctl.Catalog().Resolve(j.Model)
|
||||
return ok && slices.ContainsFunc(m.Adapters, func(ad catalog.Adapter) bool { return ad.Name == j.Name })
|
||||
}
|
||||
|
||||
// ShareTraining changes who a job's adapter belongs to — owner as scope:ID,
|
||||
// or "" for every caller its model allows — on the job and, once
|
||||
// published, in the catalog. Refused wider than the knowledge bases its
|
||||
// pairs came from may be read.
|
||||
func (a *API) ShareTraining(ctx context.Context, id, owner, actor string) error {
|
||||
j, err := a.Training(ctx, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, colls, err := countPairs(string(j.Data))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resolved, err := a.trainOwner(ctx, TrainRequest{Owner: owner, Public: owner == ""}, colls)
|
||||
if err != nil {
|
||||
return refused(http.StatusBadRequest, "%v", err)
|
||||
}
|
||||
if err := a.train.Store.SetTrainOwner(ctx, id, resolved); err != nil {
|
||||
return err
|
||||
}
|
||||
m, ok := a.ctl.Catalog().Resolve(j.Model)
|
||||
if !ok || a.models == nil {
|
||||
return nil
|
||||
}
|
||||
ads := slices.Clone(m.Adapters)
|
||||
i := slices.IndexFunc(ads, func(ad catalog.Adapter) bool { return ad.Name == j.Name })
|
||||
if i < 0 {
|
||||
return nil
|
||||
}
|
||||
ads[i].Owner = resolved
|
||||
_, err = a.models.SetAdapters(ctx, m.Name, ads, actor)
|
||||
return err
|
||||
}
|
||||
|
||||
// DeleteTraining stops a job if it trains, takes its adapter out of the
|
||||
// catalog if it was published, and forgets it.
|
||||
func (a *API) DeleteTraining(ctx context.Context, id, actor string) error {
|
||||
j, err := a.Training(ctx, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !store.TrainOver(j.State) {
|
||||
if err := a.train.Service.Cancel(ctx, id); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if m, ok := a.ctl.Catalog().Resolve(j.Model); ok && a.models != nil {
|
||||
ads := slices.DeleteFunc(slices.Clone(m.Adapters), func(ad catalog.Adapter) bool { return ad.Name == j.Name })
|
||||
if len(ads) != len(m.Adapters) {
|
||||
if _, err := a.models.SetAdapters(ctx, m.Name, ads, actor); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := a.train.Store.DeleteTrainJob(ctx, id); err != nil && !errors.Is(err, store.ErrNotFound) {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,137 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
type fakeRegistry struct {
|
||||
repos map[string]hf.Repo
|
||||
asked int
|
||||
}
|
||||
|
||||
func (f *fakeRegistry) RepoAt(_ context.Context, id, _ string) (hf.Repo, error) {
|
||||
f.asked++
|
||||
r, ok := f.repos[id]
|
||||
if !ok {
|
||||
return hf.Repo{}, errors.New("404")
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
func TestAPortalTrainingFindsItsWeightsAndMemoryAndOneTrainsPerProject(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
models := append(testModels(),
|
||||
catalog.Model{Name: "small", Source: "/s", Origin: &catalog.Origin{Repo: "bartowski/Small-GGUF"},
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}},
|
||||
catalog.Model{Name: "gated", Source: "/g", Origin: &catalog.Origin{Repo: "meta/Gated"},
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}},
|
||||
catalog.Model{Name: "local", Source: "/l", Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}})
|
||||
a, _ := setup(t, models, gpuNode("cuda01", 1))
|
||||
s, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "s.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.Close()
|
||||
reg := &fakeRegistry{repos: map[string]hf.Repo{
|
||||
"bartowski/Small-GGUF": {ID: "bartowski/Small-GGUF", Revision: "g1", BaseModel: []string{"Org/Small"},
|
||||
Files: []hf.File{{Path: "small-Q4_K_M.gguf", Bytes: 400 << 20}}},
|
||||
"Org/Small": {ID: "Org/Small", Revision: "abc", Files: []hf.File{{Path: "model.safetensors", Bytes: 1000 << 20},
|
||||
{Path: "config.json", Bytes: 1 << 10}}},
|
||||
"meta/Gated": {ID: "meta/Gated", Gated: hf.Gated{Reason: "manual"},
|
||||
Files: []hf.File{{Path: "model.safetensors", Bytes: 1 << 30}}},
|
||||
}}
|
||||
a.SetTrain(&TrainAdmin{Store: s, Service: train.New(s, noAgents{}, nil)})
|
||||
a.SetTrainMirror(nil, reg)
|
||||
|
||||
if got := a.TrainableModels(); strings.Join(got, ",") != "gated,small" {
|
||||
t.Errorf("trainable: %v", got)
|
||||
}
|
||||
data := train.NDJSON([]train.Pair{{Q: "a", A: "b"}})
|
||||
req := TrainRequest{Name: "tone", Model: "small", Data: data, Owner: "project:p1", Project: "p1", Publish: true}
|
||||
j, err := a.StartTraining(ctx, req, "ayse@x")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if j.BaseRepo != "Org/Small" || j.BaseRevision != "abc" || j.Node != "" || j.GPUMemoryMiB != TrainMemoryMiB(1000<<20) ||
|
||||
!j.Publish || j.Project != "p1" || j.CreatedBy != "ayse@x" || !j.GGUF {
|
||||
t.Errorf("queued %+v", j)
|
||||
}
|
||||
if j.GPUMemoryMiB != 3072 {
|
||||
t.Errorf("1000 MiB of weights take %d MiB to train", j.GPUMemoryMiB)
|
||||
}
|
||||
|
||||
req.Name = "tone2"
|
||||
if _, err := a.StartTraining(ctx, req, "mehmet@x"); err == nil || !strings.Contains(err.Error(), "already has a model training") {
|
||||
t.Errorf("a second in the project: %v", err)
|
||||
}
|
||||
req.Name, req.Project, req.Owner = "tone", "p2", "project:p2"
|
||||
if _, err := a.StartTraining(ctx, req, "x"); err == nil || !strings.Contains(err.Error(), "already being trained") {
|
||||
t.Errorf("the same name twice: %v", err)
|
||||
}
|
||||
req.Name = "tone3"
|
||||
asked := reg.asked
|
||||
if _, err := a.StartTraining(ctx, req, "x"); err != nil {
|
||||
t.Errorf("another project: %v", err)
|
||||
}
|
||||
if reg.asked != asked {
|
||||
t.Errorf("the registry was asked again: %d", reg.asked-asked)
|
||||
}
|
||||
for model, says := range map[string]string{"gated": "gated", "local": "did not come from the model registry"} {
|
||||
req.Model, req.Name, req.Project = model, "x-"+model, "p-"+model
|
||||
_, err := a.StartTraining(ctx, req, "x")
|
||||
if err == nil || !strings.Contains(err.Error(), says) {
|
||||
t.Errorf("%s: %v", model, err)
|
||||
}
|
||||
if msg, theirs := TrainRefusal(err); !theirs || msg == "" {
|
||||
t.Errorf("%s is not the request's fault: %q", model, msg)
|
||||
}
|
||||
}
|
||||
|
||||
jobs, err := a.Trainings(ctx, []string{"p1"}, "nobody")
|
||||
if err != nil || len(jobs) != 1 || jobs[0].Name != "tone" {
|
||||
t.Errorf("p1's: %+v %v", jobs, err)
|
||||
}
|
||||
if err := a.ShareTraining(ctx, j.ID, "", "ayse@x"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got, _ := a.Training(ctx, j.ID); got.Owner != "" {
|
||||
t.Errorf("shared with everyone: %q", got.Owner)
|
||||
}
|
||||
if err := a.DeleteTraining(ctx, j.ID, "ayse@x"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := a.Training(ctx, j.ID); !errors.Is(err, store.ErrNotFound) {
|
||||
t.Errorf("deleted: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestATrainingGoesToTheNodeWithTheMostFree(t *testing.T) {
|
||||
busy := gpuNode("busy", 1)
|
||||
busy.Load = &inventory.Load{GPUs: []inventory.GPUSensor{{Index: 0, MemoryTotalMB: 24000, MemoryUsedMB: 23000}}}
|
||||
roomy := gpuNode("roomy", 1)
|
||||
roomy.Load = &inventory.Load{GPUs: []inventory.GPUSensor{{Index: 0, MemoryTotalMB: 24000, MemoryUsedMB: 4000}}}
|
||||
cpu := inventory.Node{ID: "cpu", State: inventory.Ready}
|
||||
small := gpuNode("small", 1)
|
||||
small.GPUs = []discovery.GPU{{Index: 0, MemoryTotalMB: 8000}}
|
||||
a, _ := setup(t, testModels(), busy, roomy, cpu, small)
|
||||
if got := a.PickTrainNode(3000, nil); got != "roomy" {
|
||||
t.Errorf("picked %q", got)
|
||||
}
|
||||
if got := a.PickTrainNode(3000, map[string]bool{"roomy": true}); got != "small" {
|
||||
t.Errorf("with roomy busy, picked %q", got)
|
||||
}
|
||||
if got := a.PickTrainNode(30000, nil); got != "" {
|
||||
t.Errorf("none has 30 GB, picked %q", got)
|
||||
}
|
||||
}
|
||||
@@ -36,6 +36,7 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
|
||||
n.Host = discovery.Host{
|
||||
OSID: h.GetOsId(), OSVersion: h.GetOsVersion(), OSCodename: h.GetOsCodename(),
|
||||
Arch: h.GetArch(), CPUFlags: h.GetCpuFlags(), NVIDIADriver: h.GetNvidiaDriver(),
|
||||
NVIDIADevice: h.GetNvidiaDevice(),
|
||||
MemBandwidthGBs: h.GetMemBandwidthGbs(), MemBandwidthCoreGBs: h.GetMemBandwidthCoreGbs(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
// Package held records, on the leader, what each model holds of the
|
||||
// cluster: its replicas' GPUs and, for one on CPUs, its cores. Kept by the
|
||||
// hour, it is the cost side of a token's price (llmgw ctl prices suggest).
|
||||
package held
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Inventory is where the allocations are.
|
||||
type Inventory interface {
|
||||
Nodes() []inventory.Node
|
||||
Allocations(nodeID string) []inventory.Allocation
|
||||
}
|
||||
|
||||
// Store keeps what was held.
|
||||
type Store interface {
|
||||
AddHeld(ctx context.Context, at time.Time, held []store.Held) error
|
||||
}
|
||||
|
||||
// Recorder adds, every Every, what each model holds for that long.
|
||||
type Recorder struct {
|
||||
Inv Inventory
|
||||
// Models is each replica's model, by replica id.
|
||||
Models func() map[string]string
|
||||
Store Store
|
||||
Leader func() bool
|
||||
Every time.Duration
|
||||
Log *slog.Logger
|
||||
}
|
||||
|
||||
// Run records until ctx ends.
|
||||
func (r *Recorder) Run(ctx context.Context) {
|
||||
every := r.Every
|
||||
if every <= 0 {
|
||||
every = 5 * time.Minute
|
||||
}
|
||||
t := time.NewTicker(every)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case now := <-t.C:
|
||||
if r.Leader != nil && !r.Leader() {
|
||||
continue
|
||||
}
|
||||
held := Sample(r.Inv, r.Models(), every)
|
||||
if err := r.Store.AddHeld(ctx, now, held); err != nil && r.Log != nil {
|
||||
r.Log.Warn("could not record what the models held, for their prices", "err", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Sample is what each model holds now, for d: a GPU for each card a shard
|
||||
// was given, and the cores of a shard given cores and no GPU. A replica
|
||||
// whose model is not known (headroom kept for another, one just reaped) is
|
||||
// left out.
|
||||
func Sample(inv Inventory, models map[string]string, d time.Duration) []store.Held {
|
||||
by := map[string]*store.Held{}
|
||||
var order []string
|
||||
secs := int64(d.Seconds())
|
||||
for _, n := range inv.Nodes() {
|
||||
for _, a := range inv.Allocations(n.ID) {
|
||||
m, ok := models[a.ReplicaID]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
h := by[m]
|
||||
if h == nil {
|
||||
h = &store.Held{Model: m}
|
||||
by[m] = h
|
||||
order = append(order, m)
|
||||
}
|
||||
if g := len(a.GPUIndices); g > 0 {
|
||||
h.GPUSeconds += int64(g) * secs
|
||||
} else {
|
||||
h.CoreSeconds += int64(max(a.CPUCores, 1)) * secs
|
||||
}
|
||||
}
|
||||
}
|
||||
out := make([]store.Held, 0, len(order))
|
||||
for _, m := range order {
|
||||
out = append(out, *by[m])
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
package held
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
type inv map[string][]inventory.Allocation
|
||||
|
||||
func (i inv) Nodes() []inventory.Node {
|
||||
var out []inventory.Node
|
||||
for id := range i {
|
||||
out = append(out, inventory.Node{ID: id})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (i inv) Allocations(id string) []inventory.Allocation { return i[id] }
|
||||
|
||||
func TestSampleCountsCardsAndCores(t *testing.T) {
|
||||
in := inv{
|
||||
"gpu1": {{ReplicaID: "big-1", GPUIndices: []int{0, 1}}, {ReplicaID: "headroom", GPUIndices: []int{2}}},
|
||||
"gpu2": {{ReplicaID: "big-2", GPUIndices: []int{0, 1}}},
|
||||
"cpu1": {{ReplicaID: "small-1", CPUCores: 4}, {ReplicaID: "tiny-1"}},
|
||||
}
|
||||
models := map[string]string{"big-1": "big", "big-2": "big", "small-1": "small", "tiny-1": "tiny"}
|
||||
got := map[string]store.Held{}
|
||||
for _, h := range Sample(in, models, 5*time.Minute) {
|
||||
got[h.Model] = h
|
||||
}
|
||||
if got["big"].GPUSeconds != 4*300 || got["small"].CoreSeconds != 4*300 || got["tiny"].CoreSeconds != 300 {
|
||||
t.Errorf("held %+v", got)
|
||||
}
|
||||
if len(got) != 3 {
|
||||
t.Errorf("headroom counted: %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -27,6 +27,7 @@ import (
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
@@ -62,25 +63,37 @@ const (
|
||||
|
||||
// NodeStatus is one node as the service sees it.
|
||||
type NodeStatus struct {
|
||||
Node string `json:"node"`
|
||||
Phase Phase `json:"phase"`
|
||||
Replicas int `json:"replicas"`
|
||||
Node string `json:"node"`
|
||||
Phase Phase `json:"phase"`
|
||||
Replicas int `json:"replicas"`
|
||||
// StopsBy is, while it is being lent, when a replica still on it is
|
||||
// stopped even with nowhere else to go: the drain's grace after the
|
||||
// lend began. Zero when not known (a leader that took over meanwhile).
|
||||
StopsBy time.Time `json:"stops_by,omitempty"`
|
||||
Status *Reported `json:"slurm,omitempty"`
|
||||
Connected bool `json:"connected"`
|
||||
}
|
||||
|
||||
// Reported is the agent's SlurmStatus, for the admin API.
|
||||
type Reported struct {
|
||||
Version string `json:"version,omitempty"`
|
||||
Hardware string `json:"hardware,omitempty"`
|
||||
SlurmdRunning bool `json:"slurmd_running"`
|
||||
SlurmctldRunning bool `json:"slurmctld_running"`
|
||||
State string `json:"state,omitempty"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
Jobs int `json:"jobs"`
|
||||
Error string `json:"error,omitempty"`
|
||||
FilesSHA256 string `json:"files_sha256,omitempty"`
|
||||
At time.Time `json:"at"`
|
||||
Version string `json:"version,omitempty"`
|
||||
Hardware string `json:"hardware,omitempty"`
|
||||
SlurmdRunning bool `json:"slurmd_running"`
|
||||
SlurmctldRunning bool `json:"slurmctld_running"`
|
||||
SlurmdbdRunning bool `json:"slurmdbd_running"`
|
||||
// Shared is the shared path here: "mounted", "exported", "missing".
|
||||
Shared string `json:"shared,omitempty"`
|
||||
// Containers is what the machine has for jobs in containers, empty
|
||||
// while not both of pyxis and enroot.
|
||||
Containers string `json:"containers,omitempty"`
|
||||
// Users is how many of llmgw's users the machine has.
|
||||
Users int `json:"users"`
|
||||
State string `json:"state,omitempty"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
Jobs int `json:"jobs"`
|
||||
Error string `json:"error,omitempty"`
|
||||
FilesSHA256 string `json:"files_sha256,omitempty"`
|
||||
At time.Time `json:"at"`
|
||||
}
|
||||
|
||||
// Status is the whole Slurm cluster, for `ctl slurm status`.
|
||||
@@ -94,6 +107,26 @@ type Status struct {
|
||||
// Waiting are nodes left out of slurm.conf until they report their
|
||||
// hardware.
|
||||
Waiting []string `json:"waiting,omitempty"`
|
||||
// Shared is where the shared files are and come from; Users who keeps
|
||||
// the users, and how many llmgw has.
|
||||
Shared config.SharedStorage `json:"shared"`
|
||||
Users string `json:"users,omitempty"`
|
||||
UserCount int `json:"user_count,omitempty"`
|
||||
// Containers is whether jobs may run in containers: "on" once every
|
||||
// machine has pyxis and enroot, "installing" until then, empty when
|
||||
// slurm.containers is off.
|
||||
Containers string `json:"containers,omitempty"`
|
||||
// Accounts is how many of Slurm's accounts are llmgw's projects'.
|
||||
Accounts int `json:"accounts,omitempty"`
|
||||
}
|
||||
|
||||
// User is a Linux user made from an llmgw account.
|
||||
type User struct {
|
||||
Login string `json:"login"`
|
||||
UID int `json:"uid"`
|
||||
Name string `json:"name"`
|
||||
Email string `json:"email"`
|
||||
SSHKeys []string `json:"ssh_keys,omitempty"`
|
||||
}
|
||||
|
||||
// Service keeps the Slurm cluster.
|
||||
@@ -110,6 +143,8 @@ type Service struct {
|
||||
// drained are nodes this service put in Draining, which it alone makes
|
||||
// ready again.
|
||||
drained map[string]bool
|
||||
// lendingAt is when this leader began moving a node's replicas off.
|
||||
lendingAt map[string]time.Time
|
||||
// sent is what each node was last told, so it is told again only when
|
||||
// that changes or its stream is new.
|
||||
sent map[string]*llmgwv1.Slurm
|
||||
@@ -126,6 +161,151 @@ type Service struct {
|
||||
// started is the first pass, from which a node neither heard from nor
|
||||
// kept is waited for, for HoldFor.
|
||||
started time.Time
|
||||
// userList reads the users from the accounts; users is what it last
|
||||
// said, read at usersAt. Until it has been read once no node is told
|
||||
// anything of users: an empty list would take every user away.
|
||||
userList func(ctx context.Context) ([]User, error)
|
||||
users []User
|
||||
usersAt time.Time
|
||||
usersOK bool
|
||||
// accountList reads Slurm's accounts from the projects, with the users;
|
||||
// accountsOK is false until it has been read, and the controller is
|
||||
// told nothing of accounts until then: none would take every one away.
|
||||
accountList func(ctx context.Context) ([]slurm.Account, error)
|
||||
accounts []slurm.Account
|
||||
accountsOK bool
|
||||
// jobSink records the jobs that ended; recorded are those it took,
|
||||
// recording whether a recording is under way.
|
||||
jobSink func(ctx context.Context, jobs []JobRecord) error
|
||||
recorded map[string]time.Time
|
||||
recording bool
|
||||
}
|
||||
|
||||
// JobRecord is a job that ended, for llmgw's accounting.
|
||||
type JobRecord struct {
|
||||
ID, Login, Account, Partition, State string
|
||||
Start, End time.Time
|
||||
Elapsed time.Duration
|
||||
CPUs, GPUs int
|
||||
}
|
||||
|
||||
// SetAccounts has Slurm's accounts read with list, beside the users.
|
||||
func (s *Service) SetAccounts(list func(ctx context.Context) ([]slurm.Account, error)) {
|
||||
s.mu.Lock()
|
||||
s.accountList = list
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
// SetJobs has the jobs that ended recorded with record; each is given to it
|
||||
// once it took it without an error.
|
||||
func (s *Service) SetJobs(record func(ctx context.Context, jobs []JobRecord) error) {
|
||||
s.mu.Lock()
|
||||
s.jobSink = record
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
// AccountsOf are the accounts a login may run jobs on.
|
||||
func (s *Service) AccountsOf(login string) []string {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
var out []string
|
||||
for _, a := range s.accounts {
|
||||
if slices.Contains(a.Users, login) {
|
||||
out = append(out, a.Name)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// UsersEvery is how often the users are read again without being asked.
|
||||
var UsersEvery = 30 * time.Second
|
||||
|
||||
// SetUsers has the users read with list, for slurm.users "llmgw".
|
||||
func (s *Service) SetUsers(list func(ctx context.Context) ([]User, error)) {
|
||||
s.mu.Lock()
|
||||
s.userList = list
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
// UsersChanged has the users read again now: an account or its keys
|
||||
// changed.
|
||||
func (s *Service) UsersChanged() {
|
||||
s.mu.Lock()
|
||||
s.usersAt = time.Time{}
|
||||
s.mu.Unlock()
|
||||
s.Kick()
|
||||
}
|
||||
|
||||
// readUsers reads the users when they are due. Under s.mu.
|
||||
func (s *Service) readUsers(ctx context.Context, set config.SlurmSettings) {
|
||||
if set.Users != config.UsersLlmgw || s.userList == nil || time.Since(s.usersAt) < UsersEvery {
|
||||
return
|
||||
}
|
||||
us, err := s.userList(ctx)
|
||||
s.usersAt = time.Now()
|
||||
if err != nil {
|
||||
s.log.Warn("could not read the Slurm users from the accounts; the nodes keep the last ones", "err", err)
|
||||
return
|
||||
}
|
||||
s.users, s.usersOK = us, true
|
||||
if s.accountList == nil {
|
||||
return
|
||||
}
|
||||
as, err := s.accountList(ctx)
|
||||
if err != nil {
|
||||
s.log.Warn("could not read Slurm's accounts from the projects; the controller keeps the last ones", "err", err)
|
||||
return
|
||||
}
|
||||
s.accounts, s.accountsOK = as, true
|
||||
}
|
||||
|
||||
// takeJobs records the jobs a controller says ended that are not recorded
|
||||
// yet, away from the agent's stream.
|
||||
func (s *Service) takeJobs(ended []*llmgwv1.SlurmJob) {
|
||||
s.mu.Lock()
|
||||
if s.jobSink == nil || s.recording {
|
||||
s.mu.Unlock()
|
||||
return
|
||||
}
|
||||
if s.recorded == nil {
|
||||
s.recorded = map[string]time.Time{}
|
||||
}
|
||||
for id, at := range s.recorded {
|
||||
if time.Since(at) > 3*24*time.Hour {
|
||||
delete(s.recorded, id)
|
||||
}
|
||||
}
|
||||
var jobs []JobRecord
|
||||
for _, j := range ended {
|
||||
if _, ok := s.recorded[j.GetId()]; ok || j.GetId() == "" {
|
||||
continue
|
||||
}
|
||||
jobs = append(jobs, JobRecord{ID: j.GetId(), Login: j.GetUser(), Account: j.GetAccount(), Partition: j.GetPartition(),
|
||||
State: j.GetState(), Start: time.Unix(j.GetStartUnix(), 0), End: time.Unix(j.GetEndUnix(), 0),
|
||||
Elapsed: time.Duration(j.GetElapsedSeconds()) * time.Second, CPUs: int(j.GetCpus()), GPUs: int(j.GetGpus())})
|
||||
}
|
||||
if len(jobs) == 0 {
|
||||
s.mu.Unlock()
|
||||
return
|
||||
}
|
||||
sink := s.jobSink
|
||||
s.recording = true
|
||||
s.mu.Unlock()
|
||||
go func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
|
||||
defer cancel()
|
||||
err := sink(ctx, jobs)
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.recording = false
|
||||
if err != nil {
|
||||
s.log.Warn("could not record the Slurm jobs that ended; tried again with the next status", "jobs", len(jobs), "err", err)
|
||||
return
|
||||
}
|
||||
for _, j := range jobs {
|
||||
s.recorded[j.ID] = time.Now()
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// HoldFor is how long a new leader waits for a node it neither has heard
|
||||
@@ -143,7 +323,7 @@ func New(settings func() config.SlurmSettings, inv Inventory, agents Agents,
|
||||
log = slog.Default()
|
||||
}
|
||||
return &Service{settings: settings, inv: inv, agents: agents, key: key, log: log,
|
||||
status: map[string]*Reported{}, drained: map[string]bool{}, sent: map[string]*llmgwv1.Slurm{},
|
||||
status: map[string]*Reported{}, drained: map[string]bool{}, lendingAt: map[string]time.Time{}, sent: map[string]*llmgwv1.Slurm{},
|
||||
kick: make(chan struct{}, 1)}
|
||||
}
|
||||
|
||||
@@ -197,12 +377,17 @@ func (s *Service) Kick() {
|
||||
// OnSlurmStatus records what a node's agent says.
|
||||
func (s *Service) OnSlurmStatus(node string, st *llmgwv1.SlurmStatus) {
|
||||
r := &Reported{Version: st.GetVersion(), Hardware: st.GetHardware(), SlurmdRunning: st.GetSlurmdRunning(),
|
||||
SlurmctldRunning: st.GetSlurmctldRunning(), State: st.GetState(), Reason: st.GetReason(),
|
||||
Shared: st.GetShared(), Users: int(st.GetUsers()), Containers: st.GetContainers(),
|
||||
SlurmctldRunning: st.GetSlurmctldRunning(), SlurmdbdRunning: st.GetSlurmdbdRunning(), State: st.GetState(), Reason: st.GetReason(),
|
||||
Jobs: int(st.GetJobs()), Error: st.GetError(), FilesSHA256: st.GetFilesSha256(), At: time.Now()}
|
||||
s.mu.Lock()
|
||||
prev := s.status[node]
|
||||
s.status[node] = r
|
||||
isCtl := s.settings().Controller == node
|
||||
s.mu.Unlock()
|
||||
if isCtl && len(st.GetEnded()) > 0 {
|
||||
s.takeJobs(st.GetEnded())
|
||||
}
|
||||
if r.Error != "" && (prev == nil || prev.Error != r.Error) {
|
||||
s.log.Warn("a node could not keep its part in Slurm", "node", node, "err", r.Error)
|
||||
}
|
||||
@@ -226,6 +411,7 @@ func (s *Service) Tick(ctx context.Context) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.recall(ctx)
|
||||
s.readUsers(ctx, set)
|
||||
if s.started.IsZero() {
|
||||
s.started = time.Now()
|
||||
}
|
||||
@@ -345,11 +531,14 @@ func (s *Service) phase(set config.SlurmSettings, node string) Phase {
|
||||
if ok && (n.State == inventory.Ready || n.State == inventory.Cordoned) {
|
||||
s.inv.SetState(node, inventory.Draining)
|
||||
s.drained[node] = true
|
||||
s.log.Info("lending a node to Slurm: its replicas are moved off first", "node", node)
|
||||
s.lendingAt[node] = time.Now()
|
||||
s.log.Info("lending a node to Slurm: its replicas are moved off first", "node", node,
|
||||
"stopped_by", time.Now().Add(drainGrace).Format(time.TimeOnly))
|
||||
}
|
||||
if !ok || len(s.inv.Allocations(node)) > 0 {
|
||||
return Lending
|
||||
}
|
||||
delete(s.lendingAt, node)
|
||||
return Lent
|
||||
}
|
||||
// Not lent. Until the agent says Slurm has drained it with no job left,
|
||||
@@ -369,12 +558,17 @@ func (s *Service) phase(set config.SlurmSettings, node string) Phase {
|
||||
}
|
||||
return Reclaiming
|
||||
}
|
||||
delete(s.lendingAt, node)
|
||||
if s.drained[node] {
|
||||
s.ready(node)
|
||||
}
|
||||
return Llmgw
|
||||
}
|
||||
|
||||
// drainGrace is how long a drained replica waits for its replacement
|
||||
// before it is stopped anyway: the reconciler's, which is not a setting.
|
||||
var drainGrace = reconciler.DefaultConfig().DrainGrace
|
||||
|
||||
// slurmBusy is whether Slurm may still run work on a node: a job step left
|
||||
// on it (or no telling), or a controller that would still give it one. A
|
||||
// node the controller does not know of, or cannot be asked about, gets no
|
||||
@@ -404,7 +598,26 @@ func (s *Service) files(set config.SlurmSettings) (map[string]string, []string)
|
||||
if c.ControllerAddr == "" {
|
||||
return nil, waiting
|
||||
}
|
||||
return c.Files(), waiting
|
||||
f := c.Files()
|
||||
if s.containersReady(set) {
|
||||
f["plugstack.conf"] = slurm.PlugstackConf
|
||||
}
|
||||
return f, waiting
|
||||
}
|
||||
|
||||
// containersReady is whether jobs may ask for containers: every machine of
|
||||
// the Slurm cluster says it has pyxis and enroot. Slurm fails a job on a
|
||||
// machine without the plugin it was told to load. Under s.mu.
|
||||
func (s *Service) containersReady(set config.SlurmSettings) bool {
|
||||
if !set.Containers {
|
||||
return false
|
||||
}
|
||||
for _, n := range s.machines(set) {
|
||||
if st := s.status[n]; st == nil || st.Containers == "" {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// cluster is what slurm.conf says: each node as it last reported, else as
|
||||
@@ -415,15 +628,19 @@ func (s *Service) cluster(set config.SlurmSettings) (slurm.Cluster, []string) {
|
||||
return c, nil
|
||||
}
|
||||
c.ControllerAddr = s.controllerAddr(set)
|
||||
c.EnforceAccounts = set.Users == config.UsersLlmgw && s.accountList != nil
|
||||
var waiting []string
|
||||
for _, node := range set.Nodes {
|
||||
if sn, ok := s.reported(node); ok {
|
||||
c.Nodes = append(c.Nodes, sn)
|
||||
} else if sn, ok := s.known.Nodes[node]; ok {
|
||||
c.Nodes = append(c.Nodes, sn)
|
||||
} else {
|
||||
waiting = append(waiting, node)
|
||||
sn, ok := s.reported(node)
|
||||
if !ok {
|
||||
sn, ok = s.known.Nodes[node]
|
||||
}
|
||||
if !ok {
|
||||
waiting = append(waiting, node)
|
||||
continue
|
||||
}
|
||||
sn.Lent = set.IsLent(node)
|
||||
c.Nodes = append(c.Nodes, sn)
|
||||
}
|
||||
return c, waiting
|
||||
}
|
||||
@@ -459,6 +676,7 @@ func (s *Service) message(set config.SlurmSettings, node string, lent bool, key
|
||||
return m
|
||||
}
|
||||
m.Name, m.MungeKey, m.Lent = node, key, lent
|
||||
m.Containers = set.Containers
|
||||
m.Node = isNode
|
||||
if addr := s.controllerAddr(set); addr != "" {
|
||||
m.ConfServer = net.JoinHostPort(addr, strconv.Itoa(slurm.CtldPort))
|
||||
@@ -473,25 +691,135 @@ func (s *Service) message(set config.SlurmSettings, node string, lent bool, key
|
||||
m.Controller = true
|
||||
m.Files = maps.Clone(files)
|
||||
}
|
||||
if !s.sharedPart(set, node, m) || !s.usersPart(set, node, m) {
|
||||
return nil
|
||||
}
|
||||
if isCtl && set.Users == config.UsersLlmgw && s.accountList != nil {
|
||||
if !s.accountsOK {
|
||||
return nil
|
||||
}
|
||||
m.ManageAccounts = true
|
||||
for _, a := range s.accounts {
|
||||
m.Accounts = append(m.Accounts, &llmgwv1.SlurmAccount{Name: a.Name, Parent: a.Parent, Users: slices.Clone(a.Users)})
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// sharedPart puts the shared files in m, false when where they come from
|
||||
// is not known yet: told nothing rather than no shared files, which would
|
||||
// unmount them under running jobs. Under s.mu.
|
||||
func (s *Service) sharedPart(set config.SlurmSettings, node string, m *llmgwv1.Slurm) bool {
|
||||
sh := set.Shared
|
||||
if sh.Path == "" {
|
||||
return true
|
||||
}
|
||||
m.Shared = &llmgwv1.SlurmShared{Path: sh.Path, Nfs: sh.NFS}
|
||||
switch {
|
||||
case sh.Server == "":
|
||||
case sh.Server == node:
|
||||
m.Shared.Serve = true
|
||||
for _, n := range s.machines(set) {
|
||||
if n == node {
|
||||
continue
|
||||
}
|
||||
if a := s.addr(set, n); a != "" {
|
||||
m.Shared.Clients = append(m.Shared.Clients, a)
|
||||
}
|
||||
}
|
||||
slices.Sort(m.Shared.Clients)
|
||||
default:
|
||||
a := s.addr(set, sh.Server)
|
||||
if a == "" {
|
||||
return false
|
||||
}
|
||||
if strings.Contains(a, ":") {
|
||||
a = "[" + a + "]"
|
||||
}
|
||||
m.Shared.Nfs = a + ":" + sh.Path
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// usersPart puts the users in m, false while they have not been read.
|
||||
// Under s.mu.
|
||||
func (s *Service) usersPart(set config.SlurmSettings, node string, m *llmgwv1.Slurm) bool {
|
||||
if set.Users != config.UsersLlmgw {
|
||||
return true
|
||||
}
|
||||
if !s.usersOK {
|
||||
return false
|
||||
}
|
||||
m.ManageUsers = true
|
||||
switch {
|
||||
case set.Shared.Path == "":
|
||||
m.MakeHomes = true // each machine's own /home
|
||||
case set.Shared.Server != "":
|
||||
m.MakeHomes = set.Shared.Server == node
|
||||
default:
|
||||
m.MakeHomes = set.Controller == node
|
||||
}
|
||||
for _, u := range s.users {
|
||||
m.Users = append(m.Users, &llmgwv1.SlurmUser{Login: u.Login, Uid: uint32(u.UID), Name: u.Name, //nolint:gosec // from FirstSlurmUID up
|
||||
Home: set.HomeDir(u.Login), SshKeys: slices.Clone(u.SSHKeys)})
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// machines are the controller and the nodes.
|
||||
func (s *Service) machines(set config.SlurmSettings) []string {
|
||||
out := slices.Clone(set.Nodes)
|
||||
if !slices.Contains(out, set.Controller) {
|
||||
out = append(out, set.Controller)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// addr is a machine's management address, from the inventory, else kept
|
||||
// from before. Under s.mu.
|
||||
func (s *Service) addr(set config.SlurmSettings, node string) string {
|
||||
if n, ok := s.inv.Get(node); ok && n.MgmtAddr != "" {
|
||||
return n.MgmtAddr
|
||||
}
|
||||
if node == set.Controller {
|
||||
return s.controllerAddr(set)
|
||||
}
|
||||
return s.known.Nodes[node].Addr
|
||||
}
|
||||
|
||||
// Status is the cluster as the service sees it now.
|
||||
func (s *Service) Status() Status {
|
||||
set := s.current()
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
out := Status{Enabled: set.Enabled, Cluster: set.Cluster(), Controller: set.Controller,
|
||||
Partition: set.PartitionName(), Ctld: s.status[set.Controller]}
|
||||
Partition: set.PartitionName(), Ctld: s.status[set.Controller], Shared: set.Shared, Users: set.Users}
|
||||
if set.Users == config.UsersLlmgw {
|
||||
out.UserCount = len(s.users)
|
||||
for _, a := range s.accounts {
|
||||
if a.Project != "" {
|
||||
out.Accounts++
|
||||
}
|
||||
}
|
||||
}
|
||||
if !set.Enabled {
|
||||
return out
|
||||
}
|
||||
if set.Containers {
|
||||
out.Containers = "installing"
|
||||
if s.containersReady(set) {
|
||||
out.Containers = "on"
|
||||
}
|
||||
}
|
||||
_, out.Waiting = s.files(set)
|
||||
conn := s.agents.ConnectedNodes()
|
||||
for _, node := range set.Nodes {
|
||||
ns := NodeStatus{Node: node, Status: s.status[node], Replicas: len(s.inv.Allocations(node)),
|
||||
Connected: slices.Contains(conn, node)}
|
||||
ns.Phase = s.peek(set, node)
|
||||
if at, ok := s.lendingAt[node]; ok && ns.Phase == Lending {
|
||||
ns.StopsBy = at.Add(drainGrace)
|
||||
}
|
||||
out.Nodes = append(out.Nodes, ns)
|
||||
}
|
||||
return out
|
||||
@@ -519,6 +847,12 @@ func (s *Service) peek(set config.SlurmSettings, node string) Phase {
|
||||
func (n NodeStatus) Describe() string {
|
||||
switch n.Phase {
|
||||
case Lending:
|
||||
if !n.StopsBy.IsZero() {
|
||||
// A replica moves once its replacement serves; one with nowhere
|
||||
// else to go waits out the grace, and that wait is not a hang.
|
||||
return fmt.Sprintf("being lent: %d replica(s) still to move off; one with nowhere else to go is stopped at %s",
|
||||
n.Replicas, n.StopsBy.Local().Format(time.TimeOnly))
|
||||
}
|
||||
return fmt.Sprintf("being lent: %d replica(s) still to move off", n.Replicas)
|
||||
case Lent:
|
||||
return "lent to Slurm"
|
||||
|
||||
@@ -5,11 +5,13 @@ import (
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
type fakeInv struct {
|
||||
@@ -272,3 +274,109 @@ func TestAllNodes(t *testing.T) {
|
||||
t.Fatalf("cuda02 not lent: %+v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLendingSaysWhenAReplicaWithNowhereToGoStops(t *testing.T) {
|
||||
// Seen on the PoC: a GPU model with no other card waited out the drain's
|
||||
// grace, and five minutes of "1 replica(s) still to move off" read as a hang.
|
||||
at := time.Date(2026, 11, 14, 19, 38, 34, 0, time.Local)
|
||||
n := NodeStatus{Node: "cuda01", Phase: Lending, Replicas: 1, StopsBy: at}
|
||||
if d := n.Describe(); !strings.Contains(d, "nowhere else to go is stopped at 19:38:34") {
|
||||
t.Errorf("%q", d)
|
||||
}
|
||||
n.StopsBy = time.Time{}
|
||||
if d := n.Describe(); d != "being lent: 1 replica(s) still to move off" {
|
||||
t.Errorf("%q", d)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPyxisIsLoadedOnlyOnceEveryMachineHasIt(t *testing.T) {
|
||||
// "required" in plugstack.conf fails every job on a machine without the
|
||||
// plugin; sent early, it would stop Slurm's jobs everywhere it is missing.
|
||||
r := newRig()
|
||||
r.set.Containers = true
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw})
|
||||
r.tick()
|
||||
if !r.agents.last("cuda01").GetContainers() || !r.agents.last("node06").GetContainers() {
|
||||
t.Fatal("the machines were not told to set containers up")
|
||||
}
|
||||
if _, ok := r.agents.last("node06").GetFiles()["plugstack.conf"]; ok {
|
||||
t.Fatal("pyxis loaded before any machine has it")
|
||||
}
|
||||
if s := r.svc.Status().Containers; s != "installing" {
|
||||
t.Errorf("status %q", s)
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, Containers: "pyxis 0.24.0, enroot 4.2.1"})
|
||||
r.tick()
|
||||
if _, ok := r.agents.last("node06").GetFiles()["plugstack.conf"]; ok {
|
||||
t.Fatal("pyxis loaded with the controller still without it")
|
||||
}
|
||||
r.report("node06", &llmgwv1.SlurmStatus{Containers: "pyxis 0.24.0, enroot 4.2.1"})
|
||||
r.tick()
|
||||
if p := r.agents.last("node06").GetFiles()["plugstack.conf"]; !strings.Contains(p, "required /usr/local/lib/slurm/spank_pyxis.so") {
|
||||
t.Fatalf("plugstack.conf %q", p)
|
||||
}
|
||||
if s := r.svc.Status().Containers; s != "on" {
|
||||
t.Errorf("status %q", s)
|
||||
}
|
||||
r.set.Containers = false
|
||||
r.tick()
|
||||
if _, ok := r.agents.last("node06").GetFiles()["plugstack.conf"]; ok || r.svc.Status().Containers != "" {
|
||||
t.Fatal("pyxis still loaded with containers off")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheControllerKeepsTheProjectsAccountsAndJobsAreRecordedOnce(t *testing.T) {
|
||||
r := newRig()
|
||||
r.set.Users = config.UsersLlmgw
|
||||
r.svc.SetUsers(func(context.Context) ([]User, error) { return []User{{Login: "ayse", UID: 200001}}, nil })
|
||||
read := 0
|
||||
r.svc.SetAccounts(func(context.Context) ([]slurm.Account, error) {
|
||||
read++
|
||||
return []slurm.Account{{Name: "test", Parent: "root"},
|
||||
{Name: "test-arge-deneme", Parent: "test", Project: "p1", Users: []string{"ayse"}}}, nil
|
||||
})
|
||||
got := make(chan []JobRecord, 4)
|
||||
r.svc.SetJobs(func(_ context.Context, jobs []JobRecord) error { got <- jobs; return nil })
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw})
|
||||
r.tick()
|
||||
ctl := r.agents.last("node06")
|
||||
if !ctl.GetManageAccounts() || len(ctl.GetAccounts()) != 2 || ctl.GetAccounts()[1].GetUsers()[0] != "ayse" {
|
||||
t.Fatalf("controller told %v", ctl.GetAccounts())
|
||||
}
|
||||
if r.agents.last("cuda01").GetManageAccounts() {
|
||||
t.Error("a node was told the accounts")
|
||||
}
|
||||
if !strings.Contains(ctl.GetFiles()["slurm.conf"], "AccountingStorageEnforce=associations") {
|
||||
t.Error("jobs not held to accounts")
|
||||
}
|
||||
if r.svc.Status().Accounts != 1 || len(r.svc.AccountsOf("ayse")) != 1 {
|
||||
t.Errorf("status %d, ayse's %v", r.svc.Status().Accounts, r.svc.AccountsOf("ayse"))
|
||||
}
|
||||
ended := []*llmgwv1.SlurmJob{{Id: "41", User: "ayse", Account: "test-arge-deneme", ElapsedSeconds: 60, Gpus: 1}}
|
||||
r.report("node06", &llmgwv1.SlurmStatus{Ended: ended})
|
||||
select {
|
||||
case jobs := <-got:
|
||||
if len(jobs) != 1 || jobs[0].Login != "ayse" || jobs[0].GPUs != 1 {
|
||||
t.Fatalf("jobs %+v", jobs)
|
||||
}
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("not recorded")
|
||||
}
|
||||
// The next status says it again: not recorded twice.
|
||||
deadline := time.Now().Add(2 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
r.svc.mu.Lock()
|
||||
busy := r.svc.recording
|
||||
r.svc.mu.Unlock()
|
||||
if !busy {
|
||||
break
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
r.report("node06", &llmgwv1.SlurmStatus{Ended: ended})
|
||||
select {
|
||||
case jobs := <-got:
|
||||
t.Fatalf("recorded again: %+v", jobs)
|
||||
case <-time.After(200 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package slurmsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"slices"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
)
|
||||
|
||||
func TestSharedFilesFromTheController(t *testing.T) {
|
||||
r := newRig()
|
||||
r.set.Shared = config.SharedStorage{Path: "/shared", Server: "node06"}
|
||||
r.tick()
|
||||
ctl, node := r.agents.last("node06").GetShared(), r.agents.last("cuda01").GetShared()
|
||||
if !ctl.GetServe() || !slices.Equal(ctl.GetClients(), []string{"10.0.0.147"}) || ctl.GetPath() != "/shared" {
|
||||
t.Fatalf("server told %v", ctl)
|
||||
}
|
||||
if node.GetServe() || node.GetNfs() != "10.0.0.146:/shared" {
|
||||
t.Fatalf("client told %v", node)
|
||||
}
|
||||
// The site's own export: every machine mounts it.
|
||||
r.set.Shared = config.SharedStorage{Path: "/shared", NFS: "filer:/hpc"}
|
||||
r.tick()
|
||||
if r.agents.last("node06").GetShared().GetNfs() != "filer:/hpc" || r.agents.last("cuda01").GetShared().GetServe() {
|
||||
t.Fatal("the site's export not sent")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSharedServerNotKnownTellsNothing(t *testing.T) {
|
||||
r := newRig()
|
||||
r.set.Shared = config.SharedStorage{Path: "/shared", Server: "node06"}
|
||||
delete(r.inv.nodes, "node06")
|
||||
r.svc.known.ControllerAddr, r.svc.known.Controller = "10.0.0.146", "node06"
|
||||
r.tick()
|
||||
if r.agents.last("cuda01").GetShared().GetNfs() != "10.0.0.146:/shared" {
|
||||
t.Fatalf("the kept address not used: %v", r.agents.last("cuda01"))
|
||||
}
|
||||
r2 := newRig()
|
||||
r2.set.Nodes = []string{"cuda01", "site2"}
|
||||
r2.agents.conn = append(r2.agents.conn, "site2")
|
||||
r2.set.Shared = config.SharedStorage{Path: "/shared", Server: "site2"}
|
||||
r2.tick()
|
||||
if r2.agents.last("cuda01") != nil {
|
||||
t.Fatal("told with no address for the server: its mount would go")
|
||||
}
|
||||
}
|
||||
|
||||
func TestUsersSentOnceRead(t *testing.T) {
|
||||
r := newRig()
|
||||
r.set.Users = config.UsersLlmgw
|
||||
r.set.Shared = config.SharedStorage{Path: "/shared", Server: "node06"}
|
||||
fail := true
|
||||
r.svc.SetUsers(func(context.Context) ([]User, error) {
|
||||
if fail {
|
||||
return nil, errors.New("store down")
|
||||
}
|
||||
return []User{{Login: "ayse", UID: 200000, Name: "Ayşe", SSHKeys: []string{"ssh-ed25519 AAAA"}}}, nil
|
||||
})
|
||||
r.tick()
|
||||
if len(r.agents.got["cuda01"]) != 0 {
|
||||
t.Fatal("told before the users were read: every user would go")
|
||||
}
|
||||
fail = false
|
||||
r.svc.UsersChanged()
|
||||
r.tick()
|
||||
ctl, node := r.agents.last("node06"), r.agents.last("cuda01")
|
||||
if !ctl.GetManageUsers() || !ctl.GetMakeHomes() || node.GetMakeHomes() || len(node.GetUsers()) != 1 {
|
||||
t.Fatalf("controller %v\nnode %v", ctl, node)
|
||||
}
|
||||
if u := node.GetUsers()[0]; u.GetHome() != "/shared/home/ayse" || u.GetUid() != 200000 || len(u.GetSshKeys()) != 1 {
|
||||
t.Fatalf("user %v", u)
|
||||
}
|
||||
// The store down later: the last users are kept.
|
||||
fail = true
|
||||
r.svc.UsersChanged()
|
||||
r.tick()
|
||||
if len(r.agents.last("cuda01").GetUsers()) != 1 {
|
||||
t.Fatal("users lost when the store could not be read")
|
||||
}
|
||||
if st := r.svc.Status(); st.UserCount != 1 || st.Shared.Server != "node06" {
|
||||
t.Fatalf("status %+v", st)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,205 @@
|
||||
package train
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"encoding/csv"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"path"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/kb/extract"
|
||||
)
|
||||
|
||||
// Pairs from a file a person uploads: a spreadsheet of questions and
|
||||
// answers (Excel, CSV, tab-separated) or llmgw's own NDJSON of {"q", "a"}.
|
||||
// The columns are found by their header — Soru/Cevap, Question/Answer and
|
||||
// the like — or are the first two when the first row is not a header.
|
||||
|
||||
// Pair is one question and the answer to learn for it.
|
||||
type Pair struct {
|
||||
Q string `json:"q"`
|
||||
A string `json:"a"`
|
||||
}
|
||||
|
||||
// Read is what a file gave.
|
||||
type Read struct {
|
||||
Pairs []Pair
|
||||
// Skipped counts rows with a question and no answer, or the other way.
|
||||
Skipped int
|
||||
// Columns names the columns read, as the header has them; empty for
|
||||
// NDJSON and for a table with no header.
|
||||
Columns [2]string
|
||||
}
|
||||
|
||||
// MaxPairs bounds one file: past it a job trains for hours on a page that
|
||||
// was meant to take minutes.
|
||||
const MaxPairs = 20000
|
||||
|
||||
var (
|
||||
questionNames = []string{"soru", "sorular", "question", "questions", "q", "prompt", "instruction", "input", "girdi"}
|
||||
answerNames = []string{"cevap", "cevaplar", "yanıt", "yanit", "answer", "answers", "a", "response", "output", "completion", "çıktı", "cikti"}
|
||||
)
|
||||
|
||||
// ReadPairs reads a file's pairs; name says what kind it is.
|
||||
func ReadPairs(name string, b []byte) (Read, error) {
|
||||
var rows [][]string
|
||||
switch strings.ToLower(path.Ext(name)) {
|
||||
case ".xlsx", ".xlsm":
|
||||
r, err := extract.XLSXRows(b)
|
||||
if err != nil {
|
||||
return Read{}, fmt.Errorf("the Excel file could not be read: %w", err)
|
||||
}
|
||||
rows = r
|
||||
case ".jsonl", ".ndjson", ".json":
|
||||
return readNDJSON(b)
|
||||
case ".csv", ".tsv", ".txt":
|
||||
r, err := readDelimited(b)
|
||||
if err != nil {
|
||||
return Read{}, err
|
||||
}
|
||||
rows = r
|
||||
case ".xls":
|
||||
return Read{}, errors.New("an old Excel file (.xls): save it as .xlsx and choose it again")
|
||||
default:
|
||||
return Read{}, errors.New("choose an Excel (.xlsx), CSV or JSONL file")
|
||||
}
|
||||
return fromRows(rows)
|
||||
}
|
||||
|
||||
func readDelimited(b []byte) ([][]string, error) {
|
||||
b = bytes.TrimPrefix(b, []byte("\xef\xbb\xbf"))
|
||||
if !utf8.Valid(b) {
|
||||
return nil, errors.New("the file is not UTF-8 text: save it as \"CSV UTF-8\"")
|
||||
}
|
||||
first, _, _ := strings.Cut(string(b), "\n")
|
||||
// Whichever separator the header line has most of: Excel writes ";"
|
||||
// where the decimal mark is a comma, as it is in Turkish.
|
||||
sep, most := ',', strings.Count(first, ",")
|
||||
for _, c := range []rune{';', '\t'} {
|
||||
if n := strings.Count(first, string(c)); n > most {
|
||||
sep, most = c, n
|
||||
}
|
||||
}
|
||||
r := csv.NewReader(bytes.NewReader(b))
|
||||
r.Comma, r.FieldsPerRecord, r.LazyQuotes = sep, -1, true
|
||||
rows, err := r.ReadAll()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the file could not be read as a table: %w", err)
|
||||
}
|
||||
return rows, nil
|
||||
}
|
||||
|
||||
func fromRows(rows [][]string) (Read, error) {
|
||||
var out Read
|
||||
qc, ac := 0, 1
|
||||
if len(rows) > 0 {
|
||||
q, a := -1, -1
|
||||
for i, c := range rows[0] {
|
||||
n := strings.ToLower(strings.TrimSpace(c))
|
||||
switch {
|
||||
case q < 0 && contains(questionNames, n):
|
||||
q = i
|
||||
case a < 0 && contains(answerNames, n):
|
||||
a = i
|
||||
}
|
||||
}
|
||||
if q >= 0 && a >= 0 {
|
||||
qc, ac = q, a
|
||||
out.Columns = [2]string{strings.TrimSpace(rows[0][q]), strings.TrimSpace(rows[0][a])}
|
||||
rows = rows[1:]
|
||||
}
|
||||
}
|
||||
for _, r := range rows {
|
||||
var q, a string
|
||||
if qc < len(r) {
|
||||
q = strings.TrimSpace(r[qc])
|
||||
}
|
||||
if ac < len(r) {
|
||||
a = strings.TrimSpace(r[ac])
|
||||
}
|
||||
switch {
|
||||
case q == "" && a == "":
|
||||
case q == "" || a == "":
|
||||
out.Skipped++
|
||||
default:
|
||||
out.Pairs = append(out.Pairs, Pair{Q: q, A: a})
|
||||
}
|
||||
}
|
||||
return out, check(out)
|
||||
}
|
||||
|
||||
func contains(names []string, n string) bool {
|
||||
for _, x := range names {
|
||||
if x == n {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func readNDJSON(b []byte) (Read, error) {
|
||||
var out Read
|
||||
sc := bufio.NewScanner(bytes.NewReader(b))
|
||||
sc.Buffer(make([]byte, 64<<10), 1<<20)
|
||||
line := 0
|
||||
for sc.Scan() {
|
||||
line++
|
||||
l := bytes.TrimSpace(sc.Bytes())
|
||||
if len(l) == 0 {
|
||||
continue
|
||||
}
|
||||
var p struct {
|
||||
Q, A string
|
||||
Question, Answer string
|
||||
Prompt, Completion string
|
||||
}
|
||||
if err := json.Unmarshal(l, &p); err != nil {
|
||||
return Read{}, fmt.Errorf("line %d is not JSON: %v", line, err)
|
||||
}
|
||||
q, a := first(p.Q, p.Question, p.Prompt), first(p.A, p.Answer, p.Completion)
|
||||
if q == "" || a == "" {
|
||||
out.Skipped++
|
||||
continue
|
||||
}
|
||||
out.Pairs = append(out.Pairs, Pair{Q: q, A: a})
|
||||
}
|
||||
if err := sc.Err(); err != nil {
|
||||
return Read{}, err
|
||||
}
|
||||
return out, check(out)
|
||||
}
|
||||
|
||||
func first(s ...string) string {
|
||||
for _, x := range s {
|
||||
if x = strings.TrimSpace(x); x != "" {
|
||||
return x
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func check(r Read) error {
|
||||
switch {
|
||||
case len(r.Pairs) == 0:
|
||||
return errors.New("no question and answer found: the first column is the question and the second the answer, " +
|
||||
"or columns headed Soru and Cevap")
|
||||
case len(r.Pairs) > MaxPairs:
|
||||
return fmt.Errorf("%d pairs, more than the %d one training takes", len(r.Pairs), MaxPairs)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// NDJSON is the pairs as a job's data.
|
||||
func NDJSON(pairs []Pair) string {
|
||||
var b strings.Builder
|
||||
enc := json.NewEncoder(&b)
|
||||
enc.SetEscapeHTML(false)
|
||||
for _, p := range pairs {
|
||||
_ = enc.Encode(p)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
package train
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestPairsFromATableByItsHeaderOrItsFirstColumns(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
name, file string
|
||||
want []Pair
|
||||
skipped int
|
||||
cols [2]string
|
||||
}{
|
||||
{"headed.csv", "No;Cevap;Soru\n1;Yirmi gün.;İzin kaç gün?\n2;;Boş cevap\n3;Evet;Uzaktan çalışabilir miyim?\n",
|
||||
[]Pair{{"İzin kaç gün?", "Yirmi gün."}, {"Uzaktan çalışabilir miyim?", "Evet"}}, 1, [2]string{"Soru", "Cevap"}},
|
||||
{"plain.csv", "\ufeffİzin kaç gün?,\"Yirmi gün, yılda.\"\n\nMasraf?,Formla.\n",
|
||||
[]Pair{{"İzin kaç gün?", "Yirmi gün, yılda."}, {"Masraf?", "Formla."}}, 0, [2]string{}},
|
||||
{"tabs.tsv", "Question\tAnswer\nA?\tB.\n", []Pair{{"A?", "B."}}, 0, [2]string{"Question", "Answer"}},
|
||||
{"d.jsonl", `{"q":"A?","a":"B."}` + "\n" + `{"question":"C?","answer":"D."}` + "\n" + `{"q":"E?"}` + "\n",
|
||||
[]Pair{{"A?", "B."}, {"C?", "D."}}, 1, [2]string{}},
|
||||
} {
|
||||
got, err := ReadPairs(c.name, []byte(c.file))
|
||||
if err != nil {
|
||||
t.Errorf("%s: %v", c.name, err)
|
||||
continue
|
||||
}
|
||||
if len(got.Pairs) != len(c.want) || got.Skipped != c.skipped || got.Columns != c.cols {
|
||||
t.Errorf("%s: %+v", c.name, got)
|
||||
continue
|
||||
}
|
||||
for i := range c.want {
|
||||
if got.Pairs[i] != c.want[i] {
|
||||
t.Errorf("%s: pair %d = %+v, want %+v", c.name, i, got.Pairs[i], c.want[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPairsRefusedWithAReason(t *testing.T) {
|
||||
for name, file := range map[string]string{
|
||||
"a.csv": "only one column\n",
|
||||
"a.xls": "x",
|
||||
"a.pdf": "x",
|
||||
"a.json": "not json\n",
|
||||
} {
|
||||
if _, err := ReadPairs(name, []byte(file)); err == nil {
|
||||
t.Errorf("%s read", name)
|
||||
}
|
||||
}
|
||||
var b strings.Builder
|
||||
for range MaxPairs + 1 {
|
||||
b.WriteString("x,y\n")
|
||||
}
|
||||
if _, err := ReadPairs("big.csv", []byte(b.String())); err == nil || !strings.Contains(err.Error(), "more than") {
|
||||
t.Errorf("too many: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNDJSONIsWhatAJobReads(t *testing.T) {
|
||||
got := NDJSON([]Pair{{"<b>?", "a & b"}})
|
||||
if got != `{"q":"<b>?","a":"a & b"}`+"\n" {
|
||||
t.Errorf("%q", got)
|
||||
}
|
||||
}
|
||||
@@ -35,11 +35,26 @@ type Service struct {
|
||||
agents Agents
|
||||
log *slog.Logger
|
||||
|
||||
// pick chooses the node for a job that names none, among those not
|
||||
// busy; "" is none can take it now. Nil leaves such jobs queued.
|
||||
pick func(needMiB int, busy map[string]bool) string
|
||||
// done is told of a job that finished, after what it made is kept.
|
||||
done func(j store.TrainJob)
|
||||
|
||||
mu sync.Mutex
|
||||
// pieces are artifacts arriving, by job and kind.
|
||||
pieces map[[2]string][]byte
|
||||
}
|
||||
|
||||
// SetPicker has jobs that name no node sent where pick says.
|
||||
func (s *Service) SetPicker(pick func(needMiB int, busy map[string]bool) string) { s.pick = pick }
|
||||
|
||||
// SetOnDone has done told of each job that finishes.
|
||||
func (s *Service) SetOnDone(done func(j store.TrainJob)) { s.done = done }
|
||||
|
||||
// Waiting is what a job that names no node says while no node can take it.
|
||||
const Waiting = "waiting for a card with enough memory free"
|
||||
|
||||
// New is a service.
|
||||
func New(s Store, a Agents, log *slog.Logger) *Service {
|
||||
if log == nil {
|
||||
@@ -81,13 +96,27 @@ func (s *Service) Dispatch(ctx context.Context) {
|
||||
busy[j.Node] = true
|
||||
}
|
||||
for _, q := range queued {
|
||||
if busy[q.Node] || !s.agents.Connected(q.Node) {
|
||||
node := q.Node
|
||||
if node == "" {
|
||||
if s.pick != nil {
|
||||
node = s.pick(q.GPUMemoryMiB, busy)
|
||||
}
|
||||
if node == "" {
|
||||
if q.Message != Waiting {
|
||||
q.Message = Waiting
|
||||
_ = s.store.UpdateTrainJob(ctx, q)
|
||||
}
|
||||
continue
|
||||
}
|
||||
}
|
||||
if busy[node] || !s.agents.Connected(node) {
|
||||
continue
|
||||
}
|
||||
j, err := s.store.TrainJob(ctx, q.ID)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
j.Node = node
|
||||
err = s.agents.Send(j.Node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_TrainStart{TrainStart: &llmgwv1.TrainStart{
|
||||
JobId: j.ID, BaseRepo: j.BaseRepo, BaseRevision: j.BaseRevision, Data: j.Data, Params: j.Params,
|
||||
GpuMemoryMib: int32(j.GPUMemoryMiB), Gguf: j.GGUF}}}) //nolint:gosec // MiB
|
||||
@@ -135,6 +164,11 @@ func (s *Service) OnTrainStatus(node string, st *llmgwv1.TrainStatus) {
|
||||
}
|
||||
if err := s.store.UpdateTrainJob(ctx, j); err != nil {
|
||||
s.log.Warn("could not record a training job's progress", "job", j.ID, "err", err)
|
||||
return
|
||||
}
|
||||
// What it made came before it said it was done, on the same stream.
|
||||
if j.State == store.TrainDone && s.done != nil {
|
||||
go s.done(j)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -122,3 +122,41 @@ func TestCancellingAJob(t *testing.T) {
|
||||
t.Errorf("the node was not told: %+v", a.sent)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAJobNamingNoNodeGoesWhereThePickerSaysAndIsToldWhenDone(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
svc, s, a := setup(t)
|
||||
job(t, s, "j1", "")
|
||||
asked := 0
|
||||
free := ""
|
||||
svc.SetPicker(func(need int, busy map[string]bool) string {
|
||||
asked++
|
||||
if need != 2000 {
|
||||
t.Errorf("asked for %d MiB", need)
|
||||
}
|
||||
return free
|
||||
})
|
||||
done := make(chan store.TrainJob, 1)
|
||||
svc.SetOnDone(func(j store.TrainJob) { done <- j })
|
||||
|
||||
svc.Dispatch(ctx)
|
||||
if j, _ := s.TrainJob(ctx, "j1"); j.State != store.TrainQueued || j.Message != Waiting || len(a.sent) != 0 {
|
||||
t.Fatalf("with no card free: %+v, sent %d", j, len(a.sent))
|
||||
}
|
||||
free = "gpu2"
|
||||
a.connected["gpu2"] = true
|
||||
svc.Dispatch(ctx)
|
||||
j, _ := s.TrainJob(ctx, "j1")
|
||||
if len(a.sent) != 1 || j.Node != "gpu2" || j.State != store.TrainStarting {
|
||||
t.Fatalf("picked: %+v, sent %d", j, len(a.sent))
|
||||
}
|
||||
// Its node is the one it was sent to, which alone reports for it.
|
||||
svc.OnTrainStatus("cuda01", &llmgwv1.TrainStatus{JobId: "j1", State: "done"})
|
||||
if j, _ := s.TrainJob(ctx, "j1"); j.State == store.TrainDone {
|
||||
t.Fatal("another node finished it")
|
||||
}
|
||||
svc.OnTrainStatus("gpu2", &llmgwv1.TrainStatus{JobId: "j1", State: "done"})
|
||||
if got := <-done; got.ID != "j1" || got.Node != "gpu2" {
|
||||
t.Errorf("told of %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -39,6 +39,8 @@ var Reference = []Block{
|
||||
{Key: "personal", Type: "Personal", Doc: "Personal is what a person may do on their own, outside any project: make\npersonal keys and use the chat page personally, within these limits.", Nested: "config.Personal"},
|
||||
{Key: "pricing", Type: "map[string]Price", Doc: "Pricing is what each model costs, for budgets. A model absent from the\ntable costs whatever DefaultPrice says, which is nothing unless set.", Nested: ""},
|
||||
{Key: "default_price", Type: "Price", Doc: "", Nested: "config.Price"},
|
||||
{Key: "gpu_hour_cost", Type: "float64", Doc: "GPUHourCost is what an hour of one GPU costs the operator, in the\nunits budgets are in: the hardware written off over its years, its\npower and room, and the people who run it, over the GPUs and the\nhours of a month. With CoreHourCost for a CPU core, it is what\n`llmgw ctl prices suggest` prices each model from: the hours its\nreplicas held, idle ones too, over the tokens it served.", Nested: ""},
|
||||
{Key: "core_hour_cost", Type: "float64", Doc: "", Nested: ""},
|
||||
{Key: "flush_interval", Type: "time.Duration", Doc: "FlushInterval and BatchSize decide when usage records are written.", Nested: ""},
|
||||
{Key: "batch_size", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "buffer_size", Type: "int", Doc: "BufferSize bounds the queue. Beyond it records are dropped rather than\nblocking a request: losing accounting is bad, turning a stalled database\ninto an outage is worse.", Nested: ""},
|
||||
@@ -510,6 +512,11 @@ var Reference = []Block{
|
||||
{Package: "config", Name: "ServingSettings", Title: "ServingSettings", Doc: "", Fields: []Field{
|
||||
{Key: "stopped", Type: "bool", Doc: "Stopped takes every model down and answers every request for one with\n503 (system_stopped), leaving the control planes, agents and mirror\nrunning and every model's definition as it is: llmgw stop sets it,\nllmgw start clears it, and the models come back as their catalog\nentries ask. In force on every control plane within seconds.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "SharedStorage", Title: "SharedStorage", Doc: "", Fields: []Field{
|
||||
{Key: "path", Type: "string", Doc: "Path is where every machine sees it, as /shared.", Nested: ""},
|
||||
{Key: "nfs", Type: "string", Doc: "NFS is the site's export, host:/path, mounted at Path on every\nmachine of the Slurm cluster.", Nested: ""},
|
||||
{Key: "server", Type: "string", Doc: "Server is the llmgw node, the controller or one of the nodes, that\nexports Path over NFS to the others: for a site with no storage of\nits own. One machine: not highly available, and no faster than its\ndisk and network.", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "SimConfig", Title: "SimConfig", Doc: "", Fields: []Field{
|
||||
{Key: "bin", Type: "string", Doc: "Bin is the llmgw binary itself; the engine is one of its subcommands.\nEmpty means whatever started this agent, which is right almost always.", Nested: ""},
|
||||
}},
|
||||
@@ -521,6 +528,9 @@ var Reference = []Block{
|
||||
{Key: "all_nodes", Type: "bool", Doc: "AllNodes runs slurmd on every llmgw node, those that join later too,\nbesides any named in Nodes. Each is llmgw's until it is lent, as any.", Nested: ""},
|
||||
{Key: "lent", Type: "[]string", Doc: "Lent are the nodes of Nodes Slurm may run jobs on now. A node put here\nis drained of llmgw's replicas first, as `ctl drain` does, and resumed\nin Slurm once none is left; a node taken out is drained in Slurm and\ngiven back to llmgw when its last job ends.", Nested: ""},
|
||||
{Key: "partition", Type: "string", Doc: "Partition is the one partition every node is in. Empty is \"gpu\".", Nested: ""},
|
||||
{Key: "shared", Type: "SharedStorage", Doc: "Shared is the file system every machine of the Slurm cluster sees at\nthe same path: home directories, datasets, the jobs' output and\ncheckpoints. Empty path: none, and a batch job's output stays on the\nnode it ran on.", Nested: "config.SharedStorage"},
|
||||
{Key: "users", Type: "string", Doc: "Users is who keeps the Linux users that run jobs, the same on every\nmachine of the Slurm cluster: \"llmgw\" makes one for each active llmgw\naccount (its login from the e-mail address, a UID of llmgw's from\n200000 up, its SSH keys from the portal), let in to the controller,\nand to a node only while a job of theirs runs there (pam_slurm_adopt);\nempty leaves users to the site (a directory through sssd, or by hand).", Nested: ""},
|
||||
{Key: "containers", Type: "bool", Doc: "Containers lets jobs run in containers: `srun --container-image\nnvcr.io#nvidia/pytorch:25.01-py3` (pyxis, with enroot under it),\nthe GPUs and the shared path inside. The agents install both on\nevery Slurm machine; jobs take it once every machine has them.", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "StrataConfig", Title: "StrataConfig", Doc: "", Fields: []Field{
|
||||
{Key: "dir", Type: "string", Doc: "Dir is Strata's source directory, where setup.py and serve/ are.", Nested: ""},
|
||||
|
||||
@@ -122,6 +122,10 @@ type Options struct {
|
||||
}
|
||||
|
||||
// Collect reads the node.
|
||||
// driverFix is what to do about a card with no working driver.
|
||||
const driverFix = "install the NVIDIA driver the card needs (on Ubuntu: apt-get install nvidia-driver-580-open), reboot, " +
|
||||
"and nvidia-smi must list it; then add the node again, or restart its agent if it was added already"
|
||||
|
||||
func Collect(ctx context.Context, o Options) Facts {
|
||||
f := Facts{At: time.Now().UTC(), Modules: map[string]bool{}, Engines: map[string]string{}}
|
||||
f.Hostname, _ = os.Hostname()
|
||||
@@ -372,10 +376,19 @@ func Evaluate(f Facts) []Finding {
|
||||
|
||||
// --- GPUs ---
|
||||
switch {
|
||||
case !f.SMIFound && f.Host.NVIDIADevice:
|
||||
add(Fail, "gpu", "an NVIDIA card is on the PCI bus but no driver is installed: this node would serve on CPU only",
|
||||
driverFix)
|
||||
case !f.SMIFound:
|
||||
add(Info, "gpu", "no nvidia-smi: a CPU node, or the driver is not installed", "")
|
||||
add(Info, "gpu", "no nvidia-smi: a CPU node", "")
|
||||
case f.GPUError != "" && f.Host.CardWithoutDriver():
|
||||
add(Fail, "gpu", "an NVIDIA card is on the PCI bus but its driver is not loaded: "+f.GPUError,
|
||||
"reboot if the driver was just installed; with Secure Boot on, the module must be signed (mokutil --sb-state); "+
|
||||
"check that nouveau is not holding the card (lsmod | grep nouveau)")
|
||||
case f.GPUError != "":
|
||||
add(Fail, "gpu", "nvidia-smi fails: "+f.GPUError, "check the driver: run nvidia-smi by hand")
|
||||
case len(f.GPUs) == 0 && f.Host.NVIDIADevice:
|
||||
add(Fail, "gpu", "an NVIDIA card is on the PCI bus but nvidia-smi reports no GPUs", driverFix)
|
||||
case len(f.GPUs) == 0:
|
||||
add(Warn, "gpu", "nvidia-smi is installed but reports no GPUs", "")
|
||||
default:
|
||||
|
||||
@@ -276,3 +276,27 @@ func TestTheNodeAndItsFabricsAreReadySeparately(t *testing.T) {
|
||||
t.Errorf("%+v", r)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACardWithoutADriverIsAFailureNotACPUNode(t *testing.T) {
|
||||
// The mistake this guards: a GPU node whose driver was never installed
|
||||
// onboarded quietly as a CPU node, with nothing anywhere saying why.
|
||||
f := healthy()
|
||||
f.SMIFound, f.GPUs, f.GPUError = false, nil, ""
|
||||
f.Host.NVIDIADevice, f.Host.NVIDIADriver = true, ""
|
||||
fs := Evaluate(f)
|
||||
has(t, fs, Fail, "an NVIDIA card is on the PCI bus but no driver is installed")
|
||||
|
||||
f.Host.NVIDIADevice = false
|
||||
fs = Evaluate(f)
|
||||
if bad := levels(fs, Fail); len(bad) > 0 {
|
||||
t.Errorf("a plain CPU node was faulted: %v", bad)
|
||||
}
|
||||
has(t, fs, Info, "no nvidia-smi: a CPU node")
|
||||
|
||||
// nvidia-smi installed, the module not loaded: the message points at the
|
||||
// usual reasons rather than at a generic "run nvidia-smi".
|
||||
f.SMIFound, f.GPUError = true, "running nvidia-smi: exit status 9"
|
||||
f.Host.NVIDIADevice = true
|
||||
fs = Evaluate(f)
|
||||
has(t, fs, Fail, "its driver is not loaded")
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"os"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
@@ -321,3 +322,25 @@ func TestHelpersWhenInstalled(t *testing.T) {
|
||||
}
|
||||
wants(t, got, "LibreOffice yolu")
|
||||
}
|
||||
|
||||
func TestWorkbookRowsKeepTheirColumns(t *testing.T) {
|
||||
b := zipOf(t,
|
||||
"xl/workbook.xml", `<workbook xmlns:r="r"><sheets><sheet name="Sorular" sheetId="1" r:id="rId1"/></sheets></workbook>`,
|
||||
"xl/_rels/workbook.xml.rels", `<Relationships><Relationship Id="rId1" Target="worksheets/sheet1.xml"/></Relationships>`,
|
||||
"xl/sharedStrings.xml", `<sst><si><t>Soru</t></si><si><t>Cevap</t></si><si><t>İzin kaç gün?</t></si></sst>`,
|
||||
"xl/worksheets/sheet1.xml", `<worksheet><sheetData>`+
|
||||
`<row r="1"><c r="A1" t="s"><v>0</v></c><c r="B1" t="s"><v>1</v></c></row>`+
|
||||
`<row r="2"/>`+
|
||||
`<row r="3"><c r="A3" t="s"><v>2</v></c><c r="C3" t="inlineStr"><is><t>not</t></is></c></row>`+
|
||||
`<row r="4"><c r="B4"><v>20</v></c></row>`+
|
||||
`</sheetData></worksheet>`,
|
||||
)
|
||||
rows, err := XLSXRows(b)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := [][]string{{"Soru", "Cevap"}, {"İzin kaç gün?", "", "not"}, {"", "20"}}
|
||||
if !reflect.DeepEqual(rows, want) {
|
||||
t.Errorf("rows = %q, want %q", rows, want)
|
||||
}
|
||||
}
|
||||
+114
-23
@@ -9,6 +9,7 @@ import (
|
||||
"io"
|
||||
"path"
|
||||
"regexp"
|
||||
"slices"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
@@ -227,29 +228,7 @@ func xlsxText(b []byte) (string, error) {
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
var shared []string
|
||||
if ss, _ := member(zr, "xl/sharedStrings.xml"); ss != nil {
|
||||
var cur strings.Builder
|
||||
inT, inRPh := false, 0
|
||||
_ = walkXML(ss, func(e xmlEvent) {
|
||||
switch {
|
||||
case e.start && e.name == "si":
|
||||
cur.Reset()
|
||||
case e.end && e.name == "si":
|
||||
shared = append(shared, cur.String())
|
||||
case e.start && e.name == "rPh":
|
||||
inRPh++ // phonetic guides, not the cell's text
|
||||
case e.end && e.name == "rPh":
|
||||
inRPh--
|
||||
case e.start && e.name == "t":
|
||||
inT = true
|
||||
case e.end && e.name == "t":
|
||||
inT = false
|
||||
case !e.start && !e.end && inT && inRPh == 0:
|
||||
cur.WriteString(e.text)
|
||||
}
|
||||
})
|
||||
}
|
||||
shared := xlsxShared(zr)
|
||||
sheets, err := xlsxSheets(zr)
|
||||
if err != nil {
|
||||
return "", err
|
||||
@@ -307,6 +286,118 @@ func xlsxText(b []byte) (string, error) {
|
||||
return o.String(), nil
|
||||
}
|
||||
|
||||
// xlsxShared is a workbook's shared strings, which cells name by index.
|
||||
func xlsxShared(zr *zip.Reader) []string {
|
||||
var shared []string
|
||||
ss, _ := member(zr, "xl/sharedStrings.xml")
|
||||
if ss == nil {
|
||||
return nil
|
||||
}
|
||||
var cur strings.Builder
|
||||
inT, inRPh := false, 0
|
||||
_ = walkXML(ss, func(e xmlEvent) {
|
||||
switch {
|
||||
case e.start && e.name == "si":
|
||||
cur.Reset()
|
||||
case e.end && e.name == "si":
|
||||
shared = append(shared, cur.String())
|
||||
case e.start && e.name == "rPh":
|
||||
inRPh++ // phonetic guides, not the cell's text
|
||||
case e.end && e.name == "rPh":
|
||||
inRPh--
|
||||
case e.start && e.name == "t":
|
||||
inT = true
|
||||
case e.end && e.name == "t":
|
||||
inT = false
|
||||
case !e.start && !e.end && inT && inRPh == 0:
|
||||
cur.WriteString(e.text)
|
||||
}
|
||||
})
|
||||
return shared
|
||||
}
|
||||
|
||||
// XLSXRows reads a workbook's first sheet as rows of cells, each cell in
|
||||
// its column (an empty one is ""), for a table such as questions and
|
||||
// answers. Rows with nothing in them are left out.
|
||||
func XLSXRows(b []byte) ([][]string, error) {
|
||||
zr, err := openZip(b)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
shared := xlsxShared(zr)
|
||||
sheets, err := xlsxSheets(zr)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(sheets) == 0 {
|
||||
return nil, errors.New("the workbook has no sheets")
|
||||
}
|
||||
data, err := member(zr, sheets[0].path)
|
||||
if err != nil || data == nil {
|
||||
return nil, fmt.Errorf("the workbook's first sheet, %s, could not be read", sheets[0].name)
|
||||
}
|
||||
var rows [][]string
|
||||
var row []string
|
||||
var cell strings.Builder
|
||||
cellType, col, next := "", 0, 0
|
||||
inV, inIs := false, false
|
||||
_ = walkXML(data, func(e xmlEvent) {
|
||||
switch {
|
||||
case e.start && e.name == "row":
|
||||
row, next = nil, 0
|
||||
case e.end && e.name == "row":
|
||||
if slices.ContainsFunc(row, func(s string) bool { return s != "" }) {
|
||||
rows = append(rows, row)
|
||||
}
|
||||
case e.start && e.name == "c":
|
||||
cellType = e.get("t")
|
||||
col = next
|
||||
if c, ok := columnOf(e.get("r")); ok {
|
||||
col = c
|
||||
}
|
||||
next = col + 1
|
||||
cell.Reset()
|
||||
case e.end && e.name == "c":
|
||||
v := cell.String()
|
||||
if cellType == "s" {
|
||||
if i, err := strconv.Atoi(strings.TrimSpace(v)); err == nil && i >= 0 && i < len(shared) {
|
||||
v = shared[i]
|
||||
}
|
||||
}
|
||||
if col < 0 || col > 16383 {
|
||||
return
|
||||
}
|
||||
for len(row) <= col {
|
||||
row = append(row, "")
|
||||
}
|
||||
row[col] = strings.TrimSpace(v)
|
||||
case e.start && e.name == "v":
|
||||
inV = true
|
||||
case e.end && e.name == "v":
|
||||
inV = false
|
||||
case e.start && e.name == "is":
|
||||
inIs = true
|
||||
case e.end && e.name == "is":
|
||||
inIs = false
|
||||
case !e.start && !e.end && (inV || inIs):
|
||||
cell.WriteString(e.text)
|
||||
}
|
||||
})
|
||||
return rows, nil
|
||||
}
|
||||
|
||||
// columnOf is a cell reference's column, from 0: "B7" is 1.
|
||||
func columnOf(ref string) (int, bool) {
|
||||
n, i := 0, 0
|
||||
for ; i < len(ref) && ref[i] >= 'A' && ref[i] <= 'Z'; i++ {
|
||||
n = n*26 + int(ref[i]-'A'+1)
|
||||
}
|
||||
if i == 0 {
|
||||
return 0, false
|
||||
}
|
||||
return n - 1, true
|
||||
}
|
||||
|
||||
type sheetRef struct{ name, path string }
|
||||
|
||||
// xlsxSheets lists a workbook's sheets in their order, with the part each is
|
||||
|
||||
@@ -79,6 +79,9 @@ type Options struct {
|
||||
|
||||
// Addr is how to reach the machine over SSH, and User who to log in as.
|
||||
Addr, User string
|
||||
// FallbackUser logs in when User is empty and the ssh config names no
|
||||
// user for the machine.
|
||||
FallbackUser string
|
||||
// Node is its id in the cluster. Empty means its hostname, which is what
|
||||
// it should be: the node id, the hostname and the certificate's name
|
||||
// being one string saves an afternoon each time they would differ.
|
||||
@@ -193,7 +196,7 @@ type Runner struct {
|
||||
|
||||
// New makes a Runner that reaches the machine with the system's ssh.
|
||||
func New(o Options, emit func(Event)) *Runner {
|
||||
return &Runner{Opts: o, Remote: &SSH{Addr: o.Addr, User: o.User}, Emit: emit}
|
||||
return &Runner{Opts: o, Remote: &SSH{Addr: o.Addr, User: o.User, FallbackUser: o.FallbackUser}, Emit: emit}
|
||||
}
|
||||
|
||||
func (r *Runner) emit(step, level, detail, fix string) {
|
||||
@@ -302,6 +305,9 @@ func (r *Runner) connect(ctx context.Context, res *Result) error {
|
||||
}
|
||||
|
||||
func (r *Runner) target() string {
|
||||
if s, ok := r.Remote.(*SSH); ok {
|
||||
return s.Target(context.Background())
|
||||
}
|
||||
if r.Opts.User == "" {
|
||||
return r.Opts.Addr
|
||||
}
|
||||
|
||||
+33
-4
@@ -5,6 +5,7 @@ import (
|
||||
"context"
|
||||
"os/exec"
|
||||
"runtime"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// Remote runs a shell command on the machine being onboarded, with input.
|
||||
@@ -14,8 +15,39 @@ type Remote interface {
|
||||
|
||||
// SSH is Remote through the system's ssh, so the operator's keys, agent and
|
||||
// ~/.ssh/config apply exactly as they do when they log in by hand.
|
||||
//
|
||||
// Addr is looked up in the ssh config first: a Host block whose HostName is
|
||||
// Addr is connected through by its name, so its User, IdentityFile and
|
||||
// ProxyJump apply to a machine given by address, without the address
|
||||
// having to be added to its Host line.
|
||||
type SSH struct {
|
||||
Addr, User string
|
||||
// FallbackUser logs in when User is empty and the ssh config names no
|
||||
// user for the machine: under sudo, the person who ran it.
|
||||
FallbackUser string
|
||||
|
||||
once sync.Once
|
||||
target string
|
||||
}
|
||||
|
||||
// Target is what ssh is given: [user@]host, host being the ssh config's
|
||||
// name for Addr when it has one.
|
||||
func (s *SSH) Target(ctx context.Context) string {
|
||||
s.once.Do(func() {
|
||||
host := s.Addr
|
||||
if a := configAlias(ctx, s.Addr); a != "" {
|
||||
host = a
|
||||
}
|
||||
u := s.User
|
||||
if u == "" && configUser(ctx, host) == "" {
|
||||
u = s.FallbackUser
|
||||
}
|
||||
s.target = host
|
||||
if u != "" {
|
||||
s.target = u + "@" + host
|
||||
}
|
||||
})
|
||||
return s.target
|
||||
}
|
||||
|
||||
// sshOptions never prompt: a question nobody is there to answer would hang
|
||||
@@ -37,10 +69,7 @@ var sshOptions = []string{
|
||||
// Run runs script with sh on the machine. Files travel on standard input,
|
||||
// not scp, which a hardened sshd may not offer.
|
||||
func (s *SSH) Run(ctx context.Context, script string, stdin []byte) ([]byte, error) {
|
||||
target := s.Addr
|
||||
if s.User != "" {
|
||||
target = s.User + "@" + s.Addr
|
||||
}
|
||||
target := s.Target(ctx)
|
||||
// "--" before the target: an address beginning with "-" is not read as
|
||||
// an option.
|
||||
args := append(append([]string{}, sshOptions...), "--", target, "sh", "-c", shellQuote(script))
|
||||
|
||||
@@ -0,0 +1,223 @@
|
||||
package onboard
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Where ssh reads its configuration, and arguments every `ssh -G` here is
|
||||
// given; variables so a test can point them at a file of its own.
|
||||
var (
|
||||
userSSHConfig = func() string { return filepath.Join(sshHome(), ".ssh", "config") }
|
||||
systemSSHConfig = "/etc/ssh/ssh_config"
|
||||
sshExtraArgs []string
|
||||
)
|
||||
|
||||
// sshHome is the home ssh itself reads ~/.ssh from: the account's, from the
|
||||
// password database, not $HOME, which sudo may leave as the caller's.
|
||||
func sshHome() string {
|
||||
if u, err := user.Current(); err == nil && u.HomeDir != "" {
|
||||
return u.HomeDir
|
||||
}
|
||||
h, _ := os.UserHomeDir()
|
||||
return h
|
||||
}
|
||||
|
||||
// configAlias is the ssh config's name for addr: the first concrete Host
|
||||
// whose HostName is addr, or resolves to the same address. A machine is
|
||||
// usually written in the config under its name, `Host gpu01` with
|
||||
// `HostName 10.0.0.21` and the User, IdentityFile or ProxyJump it needs,
|
||||
// and given to node add by its address, which that block does not match.
|
||||
// Empty when addr is itself named on a Host line, or no block points at it.
|
||||
func configAlias(ctx context.Context, addr string) string {
|
||||
aliases := map[string]bool{}
|
||||
var order []string
|
||||
seen := map[string]bool{}
|
||||
collect := func(path, base string) {
|
||||
for _, h := range hostNames(path, base, seen, 0) {
|
||||
if !aliases[h] {
|
||||
aliases[h] = true
|
||||
order = append(order, h)
|
||||
}
|
||||
}
|
||||
}
|
||||
collect(userSSHConfig(), filepath.Join(sshHome(), ".ssh"))
|
||||
if systemSSHConfig != "" {
|
||||
collect(systemSSHConfig, filepath.Dir(systemSSHConfig))
|
||||
}
|
||||
for h := range aliases {
|
||||
if strings.EqualFold(h, addr) {
|
||||
return ""
|
||||
}
|
||||
}
|
||||
if len(order) > 256 {
|
||||
order = order[:256]
|
||||
}
|
||||
// Names are looked up only when no HostName is the address as written,
|
||||
// and within one budget: a config of many hosts must not hold up a
|
||||
// connection by seconds each.
|
||||
var pending []string
|
||||
hostOf := map[string]string{}
|
||||
for _, a := range order {
|
||||
h := sshGet(ctx, a)["hostname"]
|
||||
if strings.EqualFold(h, addr) {
|
||||
return a
|
||||
}
|
||||
if h != "" {
|
||||
hostOf[a] = h
|
||||
pending = append(pending, a)
|
||||
}
|
||||
}
|
||||
if len(pending) == 0 {
|
||||
return ""
|
||||
}
|
||||
lctx, cancel := context.WithTimeout(ctx, 3*time.Second)
|
||||
defer cancel()
|
||||
want := lookup(lctx, addr)
|
||||
if len(want) == 0 {
|
||||
return ""
|
||||
}
|
||||
for _, a := range pending {
|
||||
for ip := range lookup(lctx, hostOf[a]) {
|
||||
if want[ip] {
|
||||
return a
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// configUser is the user the ssh config logs in to host as, empty when it
|
||||
// names none. ssh -G reports the local account when nothing set one, so
|
||||
// that is taken as none: the config saying `User root` to a root caller is
|
||||
// the one case read wrongly, and it gets the fallback user instead.
|
||||
func configUser(ctx context.Context, host string) string {
|
||||
u := sshGet(ctx, host)["user"]
|
||||
if cur, err := user.Current(); err == nil && u == cur.Username {
|
||||
return ""
|
||||
}
|
||||
return u
|
||||
}
|
||||
|
||||
// sshGet is ssh's own reading of its configuration for host, keys
|
||||
// lowercased; empty when ssh cannot say.
|
||||
func sshGet(ctx context.Context, host string) map[string]string {
|
||||
args := append(append([]string{"-G"}, sshExtraArgs...), "--", host)
|
||||
out, err := exec.CommandContext(ctx, "ssh", args...).Output()
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
m := map[string]string{}
|
||||
sc := bufio.NewScanner(bytes.NewReader(out))
|
||||
for sc.Scan() {
|
||||
k, v, ok := strings.Cut(sc.Text(), " ")
|
||||
if k = strings.ToLower(k); ok {
|
||||
if _, dup := m[k]; !dup {
|
||||
m[k] = v
|
||||
}
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func lookup(ctx context.Context, host string) map[string]bool {
|
||||
if ip := net.ParseIP(host); ip != nil {
|
||||
return map[string]bool{ip.String(): true}
|
||||
}
|
||||
addrs, err := net.DefaultResolver.LookupHost(ctx, host)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
m := map[string]bool{}
|
||||
for _, a := range addrs {
|
||||
if ip := net.ParseIP(a); ip != nil {
|
||||
m[ip.String()] = true
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// hostNames are the concrete names on the Host lines of an ssh config file
|
||||
// and the files it includes — no patterns, no negations. base is where a
|
||||
// relative Include is looked for, as ssh does: ~/.ssh for the user's file,
|
||||
// /etc/ssh for the system's.
|
||||
func hostNames(path, base string, seen map[string]bool, depth int) []string {
|
||||
if depth > 16 || seen[path] {
|
||||
return nil
|
||||
}
|
||||
seen[path] = true
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var names []string
|
||||
for _, line := range strings.Split(string(data), "\n") {
|
||||
key, args := configLine(line)
|
||||
switch key {
|
||||
case "host":
|
||||
for _, a := range args {
|
||||
if a != "" && !strings.ContainsAny(a, "*?!") {
|
||||
names = append(names, a)
|
||||
}
|
||||
}
|
||||
case "include":
|
||||
for _, a := range args {
|
||||
if strings.HasPrefix(a, "~/") {
|
||||
a = filepath.Join(sshHome(), a[2:])
|
||||
} else if !filepath.IsAbs(a) {
|
||||
a = filepath.Join(base, a)
|
||||
}
|
||||
matches, _ := filepath.Glob(a)
|
||||
for _, m := range matches {
|
||||
names = append(names, hostNames(m, base, seen, depth+1)...)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return names
|
||||
}
|
||||
|
||||
// configLine splits one line of ssh config into its keyword, lowercased,
|
||||
// and arguments: "Host a b", "Host=a b" and quoted arguments alike.
|
||||
func configLine(line string) (string, []string) {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
return "", nil
|
||||
}
|
||||
i := strings.IndexAny(line, " \t=")
|
||||
if i < 0 {
|
||||
return strings.ToLower(line), nil
|
||||
}
|
||||
key := strings.ToLower(line[:i])
|
||||
rest := strings.TrimLeft(line[i:], " \t")
|
||||
rest = strings.TrimPrefix(rest, "=")
|
||||
var args []string
|
||||
for rest = strings.TrimSpace(rest); rest != ""; rest = strings.TrimSpace(rest) {
|
||||
if rest[0] == '"' {
|
||||
end := strings.IndexByte(rest[1:], '"')
|
||||
if end < 0 {
|
||||
args = append(args, rest[1:])
|
||||
break
|
||||
}
|
||||
args = append(args, rest[1:1+end])
|
||||
rest = rest[2+end:]
|
||||
continue
|
||||
}
|
||||
j := strings.IndexAny(rest, " \t")
|
||||
if j < 0 {
|
||||
args = append(args, rest)
|
||||
break
|
||||
}
|
||||
args = append(args, rest[:j])
|
||||
rest = rest[j:]
|
||||
}
|
||||
return key, args
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package onboard
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// withSSHConfig points configAlias and ssh -G at one file, and nothing else.
|
||||
func withSSHConfig(t *testing.T, config string, extra map[string]string) {
|
||||
t.Helper()
|
||||
if _, err := exec.LookPath("ssh"); err != nil {
|
||||
t.Skip("no ssh")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "config")
|
||||
if err := os.WriteFile(path, []byte(config), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for name, body := range extra {
|
||||
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
oldUser, oldSys, oldArgs := userSSHConfig, systemSSHConfig, sshExtraArgs
|
||||
userSSHConfig = func() string { return path }
|
||||
systemSSHConfig = ""
|
||||
sshExtraArgs = []string{"-F", path}
|
||||
t.Cleanup(func() { userSSHConfig, systemSSHConfig, sshExtraArgs = oldUser, oldSys, oldArgs })
|
||||
}
|
||||
|
||||
// A machine given by address is reached through the Host block whose
|
||||
// HostName it is, with that block's user — not the sudo caller's.
|
||||
func TestSSHTargetUsesConfigAlias(t *testing.T) {
|
||||
withSSHConfig(t, `
|
||||
Host *.example
|
||||
User nobody-wild
|
||||
Host gpu01 gpu01-alt
|
||||
HostName 10.9.8.7
|
||||
User ops
|
||||
Host gpu02
|
||||
HostName=10.9.8.8
|
||||
`, nil)
|
||||
ctx := context.Background()
|
||||
cases := []struct {
|
||||
s *SSH
|
||||
want string
|
||||
}{
|
||||
// The config's User applies by ssh itself: no fallback is added.
|
||||
{&SSH{Addr: "10.9.8.7", FallbackUser: "ilker"}, "gpu01"},
|
||||
{&SSH{Addr: "10.9.8.7", User: "root", FallbackUser: "ilker"}, "root@gpu01"},
|
||||
{&SSH{Addr: "10.9.8.8", FallbackUser: "ilker"}, "ilker@gpu02"},
|
||||
{&SSH{Addr: "gpu01", FallbackUser: "ilker"}, "gpu01"},
|
||||
{&SSH{Addr: "10.9.8.9", FallbackUser: "ilker"}, "ilker@10.9.8.9"},
|
||||
{&SSH{Addr: "10.9.8.9"}, "10.9.8.9"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := c.s.Target(ctx); got != c.want {
|
||||
t.Errorf("%+v: target %q, want %q", c.s.Addr, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A block in an included file counts, as it does for ssh.
|
||||
func TestSSHTargetFollowsInclude(t *testing.T) {
|
||||
withSSHConfig(t, "Include nodes.conf\n", map[string]string{
|
||||
"nodes.conf": "Host \"gpu03\"\n HostName 10.9.8.10\n",
|
||||
})
|
||||
// ssh resolves a relative Include against ~/.ssh, not -F's directory;
|
||||
// the scan does the same, so point base at the test's directory.
|
||||
names := hostNames(userSSHConfig(), filepath.Dir(userSSHConfig()), map[string]bool{}, 0)
|
||||
if len(names) != 1 || names[0] != "gpu03" {
|
||||
t.Fatalf("names %q", names)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigUserIgnoresLocalDefault(t *testing.T) {
|
||||
withSSHConfig(t, "Host x\n HostName 10.9.8.11\n", nil)
|
||||
if cur, err := user.Current(); err == nil {
|
||||
if u := configUser(context.Background(), "x"); u != "" {
|
||||
t.Errorf("config user %q for a block naming none (local %q)", u, cur.Username)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -58,6 +58,7 @@ func (r *Router) registerChat(mux *http.ServeMux, pages *web.AccountPages) {
|
||||
}))
|
||||
r.registerImagesPage(mux, pages)
|
||||
r.registerMeshesPage(mux, pages)
|
||||
r.registerTrainPage(mux, pages)
|
||||
if r.store != nil {
|
||||
pages.SetLicences(r.licenceChoices, r.store)
|
||||
}
|
||||
|
||||
@@ -253,6 +253,9 @@ func (r *Router) Handler() http.Handler {
|
||||
pages.SetPersonalDefaults(r.personalDefaults)
|
||||
if r.store != nil {
|
||||
pages.SetAudit(r.store)
|
||||
if r.slurmSettings != nil {
|
||||
pages.SetSlurm(r.store, r.slurmSettings)
|
||||
}
|
||||
// A forgotten password: a link mailed when the cluster's
|
||||
// settings say how, otherwise asked of an operator.
|
||||
pages.SetPasswordReset(r.store, r.mail, r.mail.BaseURL)
|
||||
@@ -381,7 +384,7 @@ func (r *Router) Handler() http.Handler {
|
||||
}
|
||||
exempt := []string{"/health", "/ready", "/leader", "/v1/openapi.json", "/", "/about", "/install", "/install/", "/_static/", "/_lang/", "/claim/",
|
||||
"/signup", "/login", "/login/", "/logout", "/profile", "/profile/", "/scim/",
|
||||
"/admin", "/admin/", "/chat", "/chat/", "/images", "/images/", "/3d", "/3d/", "/org", "/org/", "/setup/"}
|
||||
"/admin", "/admin/", "/chat", "/chat/", "/images", "/images/", "/3d", "/3d/", "/train", "/train/", "/org", "/org/", "/setup/"}
|
||||
// The docs are public unless the site said otherwise, and then guarded by
|
||||
// the session check in front of them — or, on a gateway with keys and no
|
||||
// accounts to sign in to, by the key like any other path.
|
||||
|
||||
@@ -17,7 +17,7 @@ func TestSessionPagesAreNotAskedForAnAPIKey(t *testing.T) {
|
||||
h := rt.Handler()
|
||||
// The /admin pages are mounted only with a cluster view, which this
|
||||
// router has none of; these are the session pages it does mount.
|
||||
for _, path := range []string{"/profile", "/chat", "/images", "/3d", "/org", "/org/org-x"} {
|
||||
for _, path := range []string{"/profile", "/chat", "/images", "/3d", "/train", "/org", "/org/org-x"} {
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, httptest.NewRequest(http.MethodGet, path, nil))
|
||||
if strings.Contains(w.Body.String(), "api key") {
|
||||
|
||||
@@ -192,6 +192,9 @@ type Router struct {
|
||||
// registry browses a model registry from the mirror page. Nil where the
|
||||
// deployment has no way out to the internet.
|
||||
registry web.Registry
|
||||
// training is the "customise a model" page's control plane; nil
|
||||
// leaves the page off.
|
||||
training atomic.Pointer[Training]
|
||||
// catalogAdmin puts a mirrored model into service. Nil without a store,
|
||||
// where a model added at runtime would not survive a restart.
|
||||
catalogAdmin web.CatalogAdmin
|
||||
@@ -208,6 +211,9 @@ type Router struct {
|
||||
// settingsCheck and settingsLocal mount the portal's settings page.
|
||||
settingsCheck func([]byte) error
|
||||
settingsLocal []string
|
||||
// slurmSettings is the cluster's Slurm settings, for the portal; nil
|
||||
// offers nothing of Slurm.
|
||||
slurmSettings func() config.SlurmSettings
|
||||
}
|
||||
|
||||
// SetClusterView gives the router the data the /admin page renders.
|
||||
@@ -238,6 +244,10 @@ func (r *Router) SetSettingsCheck(check func([]byte) error, local []string) {
|
||||
r.settingsCheck, r.settingsLocal = check, local
|
||||
}
|
||||
|
||||
// SetSlurm has the portal offer the Slurm cluster's login and SSH keys as
|
||||
// settings say.
|
||||
func (r *Router) SetSlurm(s func() config.SlurmSettings) { r.slurmSettings = s }
|
||||
|
||||
// SetStopped refuses, or again serves, every request for a model.
|
||||
func (r *Router) SetStopped(stopped bool) { r.stopped.Store(stopped) }
|
||||
|
||||
|
||||
@@ -0,0 +1,331 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/web"
|
||||
)
|
||||
|
||||
// The "customise a model" page: a person in a project uploads questions
|
||||
// and answers, picks a model, and presses a button. The node, the weights
|
||||
// to train on and how much of a card it takes are worked out; what it
|
||||
// makes is published as a model of its own, seen by the project unless it
|
||||
// is shared wider. Members of the project see its jobs and manage them;
|
||||
// one trains at a time per project.
|
||||
|
||||
// Training is what the page needs of the control plane.
|
||||
type Training interface {
|
||||
TrainableModels() []string
|
||||
StartTraining(ctx context.Context, in adminapi.TrainRequest, actor string) (store.TrainJob, error)
|
||||
Trainings(ctx context.Context, projects []string, createdBy string) ([]store.TrainJob, error)
|
||||
Training(ctx context.Context, id string) (store.TrainJob, error)
|
||||
Published(j store.TrainJob) bool
|
||||
ShareTraining(ctx context.Context, id, owner, actor string) error
|
||||
DeleteTraining(ctx context.Context, id, actor string) error
|
||||
}
|
||||
|
||||
// SetTraining mounts the page's actions on t.
|
||||
func (r *Router) SetTraining(t Training) { r.training.Store(&t) }
|
||||
|
||||
func (r *Router) trainer() Training {
|
||||
if t := r.training.Load(); t != nil {
|
||||
return *t
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// trainMaxFile bounds an uploaded file of pairs.
|
||||
const trainMaxFile = 32 << 20
|
||||
|
||||
// The widths an adapter can be shared at, narrowest first.
|
||||
var trainShares = []string{"me", "project", "team", "org", "all"}
|
||||
|
||||
func (r *Router) registerTrainPage(mux *http.ServeMux, pages *web.AccountPages) {
|
||||
pages.SetTrain(func() bool {
|
||||
t := r.trainer()
|
||||
return t != nil && r.store != nil && len(t.TrainableModels()) > 0
|
||||
})
|
||||
mux.HandleFunc("GET /train", pages.Guard(func(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
t := r.trainer()
|
||||
if t == nil || r.store == nil {
|
||||
http.NotFound(w, req)
|
||||
return
|
||||
}
|
||||
projects, _ := r.store.ProjectsFor(req.Context(), user.ID)
|
||||
var ps []web.ChatContext
|
||||
for _, p := range projects {
|
||||
ps = append(ps, web.ChatContext{Value: p.ID, Label: p.Path})
|
||||
}
|
||||
web.Train(user, t.TrainableModels(), ps, version.Version)(w, req)
|
||||
}))
|
||||
mux.HandleFunc("POST /train/preview", pages.Guard(r.trainPreview))
|
||||
mux.HandleFunc("POST /train/start", pages.Guard(r.trainStart))
|
||||
mux.HandleFunc("GET /train/jobs", pages.Guard(r.trainJobs))
|
||||
mux.HandleFunc("POST /train/jobs/{id}/share", pages.Guard(r.trainShare))
|
||||
mux.HandleFunc("POST /train/jobs/{id}/delete", pages.Guard(r.trainDelete))
|
||||
}
|
||||
|
||||
func trainFail(w http.ResponseWriter, code int, msg string) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(code)
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"error": msg})
|
||||
}
|
||||
|
||||
// trainFile reads the pairs of the uploaded file.
|
||||
func trainFile(w http.ResponseWriter, req *http.Request) (train.Read, bool) {
|
||||
req.Body = http.MaxBytesReader(w, req.Body, trainMaxFile+1<<20)
|
||||
f, h, err := req.FormFile("file")
|
||||
if err != nil {
|
||||
trainFail(w, http.StatusBadRequest, "choose a file of questions and answers")
|
||||
return train.Read{}, false
|
||||
}
|
||||
defer f.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(f, trainMaxFile+1))
|
||||
if err != nil || len(b) > trainMaxFile {
|
||||
trainFail(w, http.StatusBadRequest, "the file is over 32 MB")
|
||||
return train.Read{}, false
|
||||
}
|
||||
rd, err := train.ReadPairs(h.Filename, b)
|
||||
if err != nil {
|
||||
trainFail(w, http.StatusBadRequest, err.Error())
|
||||
return train.Read{}, false
|
||||
}
|
||||
return rd, true
|
||||
}
|
||||
|
||||
// trainPreview says what a file holds before anything is trained on it.
|
||||
func (r *Router) trainPreview(w http.ResponseWriter, req *http.Request, _ store.User) {
|
||||
rd, ok := trainFile(w, req)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
sample := rd.Pairs[:min(len(rd.Pairs), 5)]
|
||||
writeJSONBody(w, map[string]any{"pairs": len(rd.Pairs), "skipped": rd.Skipped, "columns": rd.Columns, "sample": sample})
|
||||
}
|
||||
|
||||
// trainPlace is where a project is in the tree.
|
||||
func (r *Router) trainPlace(ctx context.Context, user store.User, project string) (store.Effective, error) {
|
||||
projects, err := r.store.ProjectsFor(ctx, user.ID)
|
||||
if err != nil {
|
||||
return store.Effective{}, err
|
||||
}
|
||||
if !slices.ContainsFunc(projects, func(p store.ProjectPath) bool { return p.ID == project }) {
|
||||
return store.Effective{}, errors.New("you are not a member of that project")
|
||||
}
|
||||
return r.store.ResolveFor(ctx, "", "", project)
|
||||
}
|
||||
|
||||
// trainOwner is the adapter's owner for a share width, in a place.
|
||||
func trainOwner(share string, user store.User, e store.Effective) (string, bool) {
|
||||
switch share {
|
||||
case "me":
|
||||
return "user:" + user.ID, true
|
||||
case "project":
|
||||
return "project:" + e.ProjectID, true
|
||||
case "team":
|
||||
return "team:" + e.TeamID, true
|
||||
case "org":
|
||||
return "org:" + e.OrgID, true
|
||||
case "all":
|
||||
return "", true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// trainShareOf is the width an owner is, seen from the job's place.
|
||||
func trainShareOf(owner string, e store.Effective) string {
|
||||
switch {
|
||||
case owner == "":
|
||||
return "all"
|
||||
case strings.HasPrefix(owner, "user:"):
|
||||
return "me"
|
||||
case owner == "project:"+e.ProjectID:
|
||||
return "project"
|
||||
case owner == "team:"+e.TeamID:
|
||||
return "team"
|
||||
case owner == "org:"+e.OrgID:
|
||||
return "org"
|
||||
}
|
||||
return "other"
|
||||
}
|
||||
|
||||
func (r *Router) trainStart(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
t := r.trainer()
|
||||
if t == nil {
|
||||
trainFail(w, http.StatusNotFound, "training is not offered here")
|
||||
return
|
||||
}
|
||||
rd, ok := trainFile(w, req)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
e, err := r.trainPlace(req.Context(), user, req.FormValue("project"))
|
||||
if err != nil {
|
||||
trainFail(w, http.StatusForbidden, err.Error())
|
||||
return
|
||||
}
|
||||
share := req.FormValue("share")
|
||||
owner, ok := trainOwner(share, user, e)
|
||||
if !ok {
|
||||
trainFail(w, http.StatusBadRequest, "who may use it: me, the project, the team, the organization or everyone")
|
||||
return
|
||||
}
|
||||
if slices.Index(trainShares, share) > slices.Index(trainShares, "project") && req.FormValue("confirm") != "yes" {
|
||||
trainFail(w, http.StatusBadRequest, "confirm that the file holds nothing the wider group must not see")
|
||||
return
|
||||
}
|
||||
in := adminapi.TrainRequest{Name: strings.TrimSpace(req.FormValue("name")), Model: req.FormValue("model"),
|
||||
Owner: owner, Public: owner == "", Data: train.NDJSON(rd.Pairs), Project: e.ProjectID, Publish: true}
|
||||
if !slices.Contains(t.TrainableModels(), in.Model) {
|
||||
trainFail(w, http.StatusBadRequest, "choose one of the models offered")
|
||||
return
|
||||
}
|
||||
j, err := t.StartTraining(req.Context(), in, user.Email)
|
||||
if err != nil {
|
||||
msg, theirs := adminapi.TrainRefusal(err)
|
||||
code := http.StatusBadRequest
|
||||
if !theirs {
|
||||
code = http.StatusInternalServerError
|
||||
r.log.Warn("training could not be started from the portal", "user", user.Email, "err", err)
|
||||
}
|
||||
trainFail(w, code, msg)
|
||||
return
|
||||
}
|
||||
r.log.Info("training started from the portal", "job", j.ID, "adapter", j.Name, "model", j.Model,
|
||||
"project", j.Project, "user", user.Email, "pairs", j.Pairs)
|
||||
writeJSONBody(w, map[string]string{"id": j.ID})
|
||||
}
|
||||
|
||||
// trainRow is a job as the page lists it.
|
||||
type trainRow struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Model string `json:"model"`
|
||||
Project string `json:"project"`
|
||||
State string `json:"state"`
|
||||
Step int `json:"step"`
|
||||
Steps int `json:"steps"`
|
||||
Message string `json:"message,omitempty"`
|
||||
Pairs int `json:"pairs"`
|
||||
Share string `json:"share"`
|
||||
Published bool `json:"published"`
|
||||
CreatedBy string `json:"created_by"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
Minutes int `json:"minutes"`
|
||||
}
|
||||
|
||||
func (r *Router) trainJobs(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
t := r.trainer()
|
||||
if t == nil || r.store == nil {
|
||||
writeJSONBody(w, []trainRow{})
|
||||
return
|
||||
}
|
||||
projects, err := r.store.ProjectsFor(req.Context(), user.ID)
|
||||
if err != nil {
|
||||
trainFail(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
ids, names := []string{}, map[string]string{}
|
||||
for _, p := range projects {
|
||||
ids = append(ids, p.ID)
|
||||
names[p.ID] = p.Path
|
||||
}
|
||||
jobs, err := t.Trainings(req.Context(), ids, user.Email)
|
||||
if err != nil {
|
||||
trainFail(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
out := []trainRow{}
|
||||
for _, j := range jobs {
|
||||
e, _ := r.store.ResolveFor(req.Context(), "", "", j.Project)
|
||||
row := trainRow{ID: j.ID, Name: j.Name, Model: j.Model, Project: names[j.Project], State: j.State, Step: j.Step,
|
||||
Steps: j.Steps, Message: j.Message, Pairs: j.Pairs, Share: trainShareOf(j.Owner, e),
|
||||
Published: j.State == store.TrainDone && t.Published(j), CreatedBy: j.CreatedBy, CreatedAt: j.CreatedAt}
|
||||
if !j.StartedAt.IsZero() {
|
||||
end := j.FinishedAt
|
||||
if end.IsZero() {
|
||||
end = time.Now()
|
||||
}
|
||||
row.Minutes = int(end.Sub(j.StartedAt).Minutes())
|
||||
}
|
||||
out = append(out, row)
|
||||
}
|
||||
writeJSONBody(w, out)
|
||||
}
|
||||
|
||||
// trainMine is a job a person may manage: their project's, or their own.
|
||||
func (r *Router) trainMine(w http.ResponseWriter, req *http.Request, user store.User) (store.TrainJob, Training, bool) {
|
||||
t := r.trainer()
|
||||
if t == nil || r.store == nil {
|
||||
trainFail(w, http.StatusNotFound, "training is not offered here")
|
||||
return store.TrainJob{}, nil, false
|
||||
}
|
||||
j, err := t.Training(req.Context(), req.PathValue("id"))
|
||||
if err == nil {
|
||||
if j.Project == "" {
|
||||
if j.CreatedBy == user.Email {
|
||||
return j, t, true
|
||||
}
|
||||
} else if _, err := r.trainPlace(req.Context(), user, j.Project); err == nil {
|
||||
return j, t, true
|
||||
}
|
||||
}
|
||||
trainFail(w, http.StatusNotFound, "no such training")
|
||||
return store.TrainJob{}, nil, false
|
||||
}
|
||||
|
||||
func (r *Router) trainShare(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
j, t, ok := r.trainMine(w, req, user)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
e, _ := r.store.ResolveFor(req.Context(), "", "", j.Project)
|
||||
share := req.FormValue("share")
|
||||
owner, ok := trainOwner(share, user, e)
|
||||
if !ok || (j.Project == "" && share != "me" && share != "all") {
|
||||
trainFail(w, http.StatusBadRequest, "who may use it: me, the project, the team, the organization or everyone")
|
||||
return
|
||||
}
|
||||
if slices.Index(trainShares, share) > slices.Index(trainShares, "project") && req.FormValue("confirm") != "yes" {
|
||||
trainFail(w, http.StatusBadRequest, "confirm that the data holds nothing the wider group must not see")
|
||||
return
|
||||
}
|
||||
if err := t.ShareTraining(req.Context(), j.ID, owner, user.Email); err != nil {
|
||||
msg, _ := adminapi.TrainRefusal(err)
|
||||
trainFail(w, http.StatusBadRequest, msg)
|
||||
return
|
||||
}
|
||||
r.log.Info("a trained model's sharing changed from the portal", "job", j.ID, "adapter", j.Name, "share", share,
|
||||
"user", user.Email)
|
||||
writeJSONBody(w, map[string]string{"share": share})
|
||||
}
|
||||
|
||||
func (r *Router) trainDelete(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
j, t, ok := r.trainMine(w, req, user)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if err := t.DeleteTraining(req.Context(), j.ID, user.Email); err != nil {
|
||||
msg, _ := adminapi.TrainRefusal(err)
|
||||
trainFail(w, http.StatusInternalServerError, msg)
|
||||
return
|
||||
}
|
||||
r.log.Info("a trained model was deleted from the portal", "job", j.ID, "adapter", j.Name, "user", user.Email)
|
||||
writeJSONBody(w, map[string]bool{"deleted": true})
|
||||
}
|
||||
|
||||
func writeJSONBody(w http.ResponseWriter, v any) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(v)
|
||||
}
|
||||
@@ -0,0 +1,165 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
type fakeTraining struct {
|
||||
started []adminapi.TrainRequest
|
||||
jobs map[string]store.TrainJob
|
||||
shared map[string]string
|
||||
deleted []string
|
||||
}
|
||||
|
||||
func (f *fakeTraining) TrainableModels() []string { return []string{"qwen"} }
|
||||
func (f *fakeTraining) StartTraining(_ context.Context, in adminapi.TrainRequest, actor string) (store.TrainJob, error) {
|
||||
f.started = append(f.started, in)
|
||||
j := store.TrainJob{ID: "train-" + in.Name, Name: in.Name, Model: in.Model, Owner: in.Owner, Project: in.Project,
|
||||
CreatedBy: actor, State: store.TrainQueued, Pairs: strings.Count(in.Data, "\n"), CreatedAt: time.Now()}
|
||||
f.jobs[j.ID] = j
|
||||
return j, nil
|
||||
}
|
||||
func (f *fakeTraining) Trainings(_ context.Context, projects []string, by string) ([]store.TrainJob, error) {
|
||||
var out []store.TrainJob
|
||||
for _, j := range f.jobs {
|
||||
for _, p := range projects {
|
||||
if j.Project == p {
|
||||
out = append(out, j)
|
||||
}
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
func (f *fakeTraining) Training(_ context.Context, id string) (store.TrainJob, error) {
|
||||
j, ok := f.jobs[id]
|
||||
if !ok {
|
||||
return j, store.ErrNotFound
|
||||
}
|
||||
return j, nil
|
||||
}
|
||||
func (f *fakeTraining) Published(store.TrainJob) bool { return false }
|
||||
func (f *fakeTraining) ShareTraining(_ context.Context, id, owner, _ string) error {
|
||||
f.shared[id] = owner
|
||||
return nil
|
||||
}
|
||||
func (f *fakeTraining) DeleteTraining(_ context.Context, id, _ string) error {
|
||||
f.deleted = append(f.deleted, id)
|
||||
return nil
|
||||
}
|
||||
|
||||
func trainForm(t *testing.T, fields map[string]string, file string) *http.Request {
|
||||
t.Helper()
|
||||
var b bytes.Buffer
|
||||
mw := multipart.NewWriter(&b)
|
||||
for k, v := range fields {
|
||||
_ = mw.WriteField(k, v)
|
||||
}
|
||||
if file != "" {
|
||||
w, _ := mw.CreateFormFile("file", "ornekler.csv")
|
||||
_, _ = w.Write([]byte(file))
|
||||
}
|
||||
_ = mw.Close()
|
||||
req := httptest.NewRequest("POST", "/train/start", &b)
|
||||
req.Header.Set("Content-Type", mw.FormDataContentType())
|
||||
return req
|
||||
}
|
||||
|
||||
func TestThePortalTrainsForAProjectItsMembersManage(t *testing.T) {
|
||||
cfg := baseConfig()
|
||||
cfg.Auth.Enabled = true
|
||||
cfg.Auth.Store = filepath.Join(t.TempDir(), "store.db")
|
||||
rt := newTestRouter(t, cfg)
|
||||
ft := &fakeTraining{jobs: map[string]store.TrainJob{}, shared: map[string]string{}}
|
||||
rt.SetTraining(ft)
|
||||
ctx := t.Context()
|
||||
org, _ := rt.store.CreateOrg(ctx, "acme")
|
||||
team, _ := rt.store.CreateTeam(ctx, org, "support")
|
||||
project, _ := rt.store.CreateProject(ctx, team, "bot")
|
||||
ayse := store.User{ID: "usr-a", Email: "ayse@acme.example", State: store.UserActive, CreatedAt: time.Now()}
|
||||
bob := store.User{ID: "usr-b", Email: "bob@else.example", State: store.UserActive, CreatedAt: time.Now()}
|
||||
for _, u := range []store.User{ayse, bob} {
|
||||
if err := rt.store.CreateUser(ctx, u); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
_ = rt.store.AddMembership(ctx, ayse.ID, store.ScopeProject, project, "member", "test", time.Now())
|
||||
csv := "Soru,Cevap\nİzin kaç gün?,Yirmi gün.\nMasraf nasıl?,Formla.\n"
|
||||
start := func(u store.User, fields map[string]string) *httptest.ResponseRecorder {
|
||||
rec := httptest.NewRecorder()
|
||||
rt.trainStart(rec, trainForm(t, fields, csv), u)
|
||||
return rec
|
||||
}
|
||||
good := map[string]string{"name": "destek", "model": "qwen", "project": project, "share": "project"}
|
||||
|
||||
if rec := start(bob, good); rec.Code != http.StatusForbidden {
|
||||
t.Errorf("someone outside the project started one: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
wide := map[string]string{"name": "destek", "model": "qwen", "project": project, "share": "org"}
|
||||
if rec := start(ayse, wide); rec.Code != http.StatusBadRequest || !strings.Contains(rec.Body.String(), "confirm") {
|
||||
t.Errorf("shared with the organization unconfirmed: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
other := map[string]string{"name": "destek", "model": "llama-70b", "project": project, "share": "project"}
|
||||
if rec := start(ayse, other); rec.Code != http.StatusBadRequest {
|
||||
t.Errorf("a model not offered: %d", rec.Code)
|
||||
}
|
||||
if rec := start(ayse, good); rec.Code != http.StatusOK {
|
||||
t.Fatalf("%d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
in := ft.started[0]
|
||||
if in.Owner != "project:"+project || in.Project != project || !in.Publish || in.Public ||
|
||||
in.Data != `{"q":"İzin kaç gün?","a":"Yirmi gün."}`+"\n"+`{"q":"Masraf nasıl?","a":"Formla."}`+"\n" {
|
||||
t.Errorf("asked for %+v", in)
|
||||
}
|
||||
|
||||
list := func(u store.User) []trainRow {
|
||||
rec := httptest.NewRecorder()
|
||||
rt.trainJobs(rec, httptest.NewRequest("GET", "/train/jobs", nil), u)
|
||||
var rows []trainRow
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &rows)
|
||||
return rows
|
||||
}
|
||||
if rows := list(ayse); len(rows) != 1 || rows[0].Share != "project" || rows[0].Project != "acme / support / bot" ||
|
||||
rows[0].CreatedBy != ayse.Email {
|
||||
t.Errorf("ayse sees %+v", rows)
|
||||
}
|
||||
if rows := list(bob); len(rows) != 0 {
|
||||
t.Errorf("bob sees %+v", rows)
|
||||
}
|
||||
|
||||
act := func(u store.User, what string, fields map[string]string) int {
|
||||
req := trainForm(t, fields, "")
|
||||
req.SetPathValue("id", "train-destek")
|
||||
rec := httptest.NewRecorder()
|
||||
if what == "share" {
|
||||
rt.trainShare(rec, req, u)
|
||||
} else {
|
||||
rt.trainDelete(rec, req, u)
|
||||
}
|
||||
return rec.Code
|
||||
}
|
||||
if code := act(bob, "delete", nil); code != http.StatusNotFound || len(ft.deleted) != 0 {
|
||||
t.Errorf("bob deleted it: %d", code)
|
||||
}
|
||||
if code := act(ayse, "share", map[string]string{"share": "team", "confirm": "yes"}); code != http.StatusOK ||
|
||||
ft.shared["train-destek"] != "team:"+team {
|
||||
t.Errorf("shared with the team: %d %v", code, ft.shared)
|
||||
}
|
||||
if code := act(ayse, "share", map[string]string{"share": "all"}); code != http.StatusBadRequest {
|
||||
t.Errorf("shared with everyone unconfirmed: %d", code)
|
||||
}
|
||||
if code := act(ayse, "delete", nil); code != http.StatusOK || len(ft.deleted) != 1 {
|
||||
t.Errorf("ayse deleting: %d", code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,193 @@
|
||||
package slurm
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Slurm's accounts from llmgw's tree: an account for each organization,
|
||||
// team and project, nested as they are, and on each project's the users
|
||||
// that may use it, its members and its team's and organization's. A job
|
||||
// is charged to the project its --account names (`sbatch --account
|
||||
// acme-ml-training`), and what it used to that project's spend.
|
||||
|
||||
// Account is one Slurm account llmgw keeps.
|
||||
type Account struct {
|
||||
Name string
|
||||
Parent string // "root" at the top
|
||||
// Project is the llmgw project it is for; empty for an organization's
|
||||
// or a team's.
|
||||
Project string
|
||||
// Users are the logins that may run jobs on it.
|
||||
Users []string
|
||||
}
|
||||
|
||||
// TenancyProject is a project with where it is.
|
||||
type TenancyProject struct {
|
||||
ID, Name, TeamID, TeamName, OrgID, OrgName string
|
||||
}
|
||||
|
||||
// TenancyMember is a membership: a person in an organization, team or
|
||||
// project ("org", "team", "project").
|
||||
type TenancyMember struct {
|
||||
UserID, Scope, ScopeID string
|
||||
}
|
||||
|
||||
// Accounts are the accounts for projects, with logins by user id. Parents
|
||||
// come before their children, as sacctmgr needs them.
|
||||
func Accounts(projects []TenancyProject, members []TenancyMember, logins map[string]string) []Account {
|
||||
// By kind and id: an organization, its team and its project may share
|
||||
// an id (the first install's are all "default").
|
||||
names := map[string]string{} // kind/id -> name
|
||||
taken := map[string]string{} // name -> kind/id
|
||||
name := func(id string, parts ...string) string {
|
||||
if n, ok := names[id]; ok {
|
||||
return n
|
||||
}
|
||||
n := AccountName(parts...)
|
||||
if other, ok := taken[n]; ok && other != id {
|
||||
// Two places whose names come out the same: told apart by id.
|
||||
h := sha256.Sum256([]byte(id))
|
||||
n = AccountName(append(parts, hex.EncodeToString(h[:3]))...)
|
||||
}
|
||||
names[id], taken[n] = n, id
|
||||
return n
|
||||
}
|
||||
in := map[string][]string{} // kind/id -> logins
|
||||
for _, m := range members {
|
||||
if l, ok := logins[m.UserID]; ok {
|
||||
in[m.Scope+"/"+m.ScopeID] = append(in[m.Scope+"/"+m.ScopeID], l)
|
||||
}
|
||||
}
|
||||
ps := slices.Clone(projects)
|
||||
slices.SortFunc(ps, func(a, b TenancyProject) int {
|
||||
return strings.Compare(a.OrgName+"\x00"+a.TeamName+"\x00"+a.Name, b.OrgName+"\x00"+b.TeamName+"\x00"+b.Name)
|
||||
})
|
||||
var out []Account
|
||||
seen := map[string]bool{}
|
||||
for _, p := range ps {
|
||||
org := name("org/"+p.OrgID, p.OrgName)
|
||||
team := name("team/"+p.TeamID, p.OrgName, p.TeamName)
|
||||
proj := name("project/"+p.ID, p.OrgName, p.TeamName, p.Name)
|
||||
if !seen[org] {
|
||||
out = append(out, Account{Name: org, Parent: "root"})
|
||||
seen[org] = true
|
||||
}
|
||||
if !seen[team] {
|
||||
out = append(out, Account{Name: team, Parent: org})
|
||||
seen[team] = true
|
||||
}
|
||||
var users []string
|
||||
for _, id := range []string{"project/" + p.ID, "team/" + p.TeamID, "org/" + p.OrgID} {
|
||||
users = append(users, in[id]...)
|
||||
}
|
||||
slices.Sort(users)
|
||||
out = append(out, Account{Name: proj, Parent: team, Project: p.ID, Users: slices.Compact(users)})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// AccountName is a Slurm account name from names: lower case, ASCII,
|
||||
// joined with "-", at most 64 characters.
|
||||
func AccountName(parts ...string) string {
|
||||
var segs []string
|
||||
for _, p := range parts {
|
||||
var b strings.Builder
|
||||
for _, r := range strings.ToLower(p) {
|
||||
switch {
|
||||
case r >= 'a' && r <= 'z', r >= '0' && r <= '9', r == '_':
|
||||
b.WriteRune(r)
|
||||
case strings.ContainsRune("çğıöşüâîû", r):
|
||||
b.WriteString(map[rune]string{'ç': "c", 'ğ': "g", 'ı': "i", 'ö': "o", 'ş': "s", 'ü': "u",
|
||||
'â': "a", 'î': "i", 'û': "u"}[r])
|
||||
default:
|
||||
b.WriteByte('_')
|
||||
}
|
||||
}
|
||||
if s := strings.Trim(b.String(), "_"); s != "" {
|
||||
segs = append(segs, s)
|
||||
}
|
||||
}
|
||||
n := strings.Join(segs, "-")
|
||||
if n == "" {
|
||||
n = "x"
|
||||
}
|
||||
if len(n) > 64 {
|
||||
h := sha256.Sum256([]byte(n))
|
||||
n = n[:57] + "-" + hex.EncodeToString(h[:3])
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// Association is one line of `sacctmgr show assoc format=account,user,parentname`.
|
||||
type Association struct {
|
||||
Account, User, Parent string
|
||||
}
|
||||
|
||||
// AssocFormat is what ParseAssociations reads.
|
||||
var AssocFormat = []string{"show", "assoc", "format=account,user,parentname", "-n", "-P"}
|
||||
|
||||
// ParseAssociations reads sacctmgr's parsable lines.
|
||||
func ParseAssociations(out string) []Association {
|
||||
var as []Association
|
||||
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
|
||||
f := strings.Split(line, "|")
|
||||
if len(f) < 3 || f[0] == "" {
|
||||
continue
|
||||
}
|
||||
as = append(as, Association{Account: f[0], User: f[1], Parent: f[2]})
|
||||
}
|
||||
return as
|
||||
}
|
||||
|
||||
// Job is a finished job, as sacct has it.
|
||||
type Job struct {
|
||||
ID, User, Account, Partition, State string
|
||||
Start, End time.Time
|
||||
Elapsed time.Duration
|
||||
CPUs, GPUs int
|
||||
}
|
||||
|
||||
// SacctArgs are sacct's for the jobs that ended in the last day.
|
||||
var SacctArgs = []string{"-a", "-X", "-n", "-P", "--noconvert", "-S", "now-1days", "-E", "now",
|
||||
"-s", "CD,F,TO,CA,NF,OOM,PR,DL,BF",
|
||||
"-o", "JobIDRaw,User,Account,Partition,Start,End,ElapsedRaw,AllocTRES,State"}
|
||||
|
||||
// ParseJobs reads sacct's lines; a job that has not ended is left out.
|
||||
func ParseJobs(out string) ([]Job, error) {
|
||||
var jobs []Job
|
||||
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
f := strings.Split(line, "|")
|
||||
if len(f) < 9 {
|
||||
return nil, fmt.Errorf("sacct said %q, not 9 fields", line)
|
||||
}
|
||||
end, err := time.ParseInLocation("2006-01-02T15:04:05", f[5], time.Local)
|
||||
if err != nil {
|
||||
continue // Unknown: still running, or never started
|
||||
}
|
||||
start, _ := time.ParseInLocation("2006-01-02T15:04:05", f[4], time.Local)
|
||||
secs, _ := strconv.ParseInt(f[6], 10, 64)
|
||||
j := Job{ID: f[0], User: f[1], Account: f[2], Partition: f[3], Start: start, End: end,
|
||||
Elapsed: time.Duration(secs) * time.Second, State: strings.Fields(f[8] + " ")[0]}
|
||||
for _, t := range strings.Split(f[7], ",") {
|
||||
k, v, _ := strings.Cut(t, "=")
|
||||
n, _ := strconv.Atoi(v)
|
||||
switch k {
|
||||
case "cpu":
|
||||
j.CPUs = n
|
||||
case "gres/gpu":
|
||||
j.GPUs = n
|
||||
}
|
||||
}
|
||||
jobs = append(jobs, j)
|
||||
}
|
||||
return jobs, nil
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
package slurm
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestAccountsFollowTheTree(t *testing.T) {
|
||||
projects := []TenancyProject{
|
||||
{ID: "p1", Name: "deneme", TeamID: "t1", TeamName: "arge", OrgID: "o1", OrgName: "Test"},
|
||||
{ID: "p2", Name: "Görüntü İşleme", TeamID: "t1", TeamName: "arge", OrgID: "o1", OrgName: "Test"},
|
||||
{ID: "p3", Name: "x", TeamID: "t2", TeamName: "ops", OrgID: "o2", OrgName: "Other"},
|
||||
}
|
||||
members := []TenancyMember{
|
||||
{UserID: "u-ayse", Scope: "project", ScopeID: "p1"},
|
||||
{UserID: "u-ada", Scope: "team", ScopeID: "t1"},
|
||||
{UserID: "u-bob", Scope: "org", ScopeID: "o2"},
|
||||
{UserID: "u-nologin", Scope: "project", ScopeID: "p1"},
|
||||
}
|
||||
logins := map[string]string{"u-ayse": "ayse", "u-ada": "ada", "u-bob": "bob"}
|
||||
got := Accounts(projects, members, logins)
|
||||
var names []string
|
||||
by := map[string]Account{}
|
||||
for _, a := range got {
|
||||
names = append(names, a.Name)
|
||||
by[a.Name] = a
|
||||
}
|
||||
// Parents first, each before its children.
|
||||
want := []string{"other", "other-ops", "other-ops-x", "test", "test-arge", "test-arge-goruntu_isleme", "test-arge-deneme"}
|
||||
if !slices.Equal(names, want) {
|
||||
t.Fatalf("names %v", names)
|
||||
}
|
||||
if a := by["test-arge-deneme"]; a.Parent != "test-arge" || a.Project != "p1" || !slices.Equal(a.Users, []string{"ada", "ayse"}) {
|
||||
t.Errorf("deneme %+v", a)
|
||||
}
|
||||
if a := by["other-ops-x"]; !slices.Equal(a.Users, []string{"bob"}) {
|
||||
t.Errorf("x %+v", a)
|
||||
}
|
||||
if a := by["test"]; a.Parent != "root" || a.Project != "" || len(a.Users) != 0 {
|
||||
t.Errorf("org %+v", a)
|
||||
}
|
||||
if n := AccountName("Test", "arge", "Görüntü İşleme"); n != "test-arge-goruntu_isleme" {
|
||||
t.Errorf("name %q", n)
|
||||
}
|
||||
// The first install's tree: one id for all three.
|
||||
def := Accounts([]TenancyProject{{ID: "default", Name: "default", TeamID: "default", TeamName: "default",
|
||||
OrgID: "default", OrgName: "default"}}, []TenancyMember{{UserID: "u", Scope: "project", ScopeID: "default"}},
|
||||
map[string]string{"u": "admin2"})
|
||||
if len(def) != 3 || def[2].Name != "default-default-default" || def[2].Parent != "default-default" ||
|
||||
def[1].Parent != "default" || !slices.Equal(def[2].Users, []string{"admin2"}) {
|
||||
t.Errorf("default tree %+v", def)
|
||||
}
|
||||
// Two that come out the same are told apart.
|
||||
same := Accounts([]TenancyProject{
|
||||
{ID: "a", Name: "ml", TeamID: "t", TeamName: "x", OrgID: "o", OrgName: "A"},
|
||||
{ID: "b", Name: "ML", TeamID: "t", TeamName: "x", OrgID: "o", OrgName: "A"},
|
||||
}, nil, nil)
|
||||
if same[2].Name == same[3].Name {
|
||||
t.Errorf("same name twice: %+v", same)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseJobs(t *testing.T) {
|
||||
out := "41|ayse|test-arge-deneme|gpu|2026-11-15T10:00:00|2026-11-15T10:30:00|1800|billing=4,cpu=4,gres/gpu=1,mem=16G,node=1|COMPLETED\n" +
|
||||
"42|ada|test-arge-deneme|gpu|2026-11-15T10:00:00|Unknown|0|cpu=2|RUNNING\n" +
|
||||
"43|root|root|gpu|2026-11-15T09:00:00|2026-11-15T09:00:05|5|cpu=1,mem=1G|CANCELLED by 0\n"
|
||||
jobs, err := ParseJobs(out)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(jobs) != 2 {
|
||||
t.Fatalf("jobs %+v", jobs)
|
||||
}
|
||||
j := jobs[0]
|
||||
if j.ID != "41" || j.User != "ayse" || j.GPUs != 1 || j.CPUs != 4 || j.Elapsed != 30*time.Minute || j.State != "COMPLETED" {
|
||||
t.Errorf("job %+v", j)
|
||||
}
|
||||
if jobs[1].State != "CANCELLED" {
|
||||
t.Errorf("state %q", jobs[1].State)
|
||||
}
|
||||
if _, err := ParseJobs("short|line\n"); err == nil {
|
||||
t.Error("a short line taken")
|
||||
}
|
||||
as := ParseAssociations("root||\ntest|ayse|test-arge\n")
|
||||
if len(as) != 2 || as[1] != (Association{"test", "ayse", "test-arge"}) {
|
||||
t.Errorf("assoc %+v", as)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,139 @@
|
||||
package slurm
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Jobs in containers: pyxis, Slurm's plugin that gives srun and sbatch
|
||||
// --container-image, and enroot under it, which turns an image into a
|
||||
// directory a job's user runs in without a daemon or root. Both are
|
||||
// NVIDIA's and in no distribution's archive, so each is fetched from its
|
||||
// release at a pinned version and checked against the digest here.
|
||||
const (
|
||||
EnrootVersion = "4.2.1"
|
||||
PyxisVersion = "0.24.0"
|
||||
|
||||
// PyxisSource is pyxis's source, built on each machine against its own
|
||||
// Slurm: a plugin built for another Slurm is refused by it.
|
||||
PyxisSource = "https://github.com/NVIDIA/pyxis/archive/refs/tags/v" + PyxisVersion + ".tar.gz"
|
||||
PyxisSHA256 = "9c4cdb79a67301d8ea05951aa4d2c205f40cf6145e338214b0191add8b36d6c4"
|
||||
|
||||
// PyxisPlugin is where it is installed (pyxis's own make install).
|
||||
PyxisPlugin = "/usr/local/lib/slurm/spank_pyxis.so"
|
||||
|
||||
// PlugstackConf has Slurm load pyxis. Written on the controller, from
|
||||
// which configless nodes fetch it, only once every machine has the
|
||||
// plugin: "required" fails every job on a machine without it, and
|
||||
// "optional" would run a job asking for a container on the host.
|
||||
PlugstackConf = "# Written by llmgw (BACKLOG P): jobs in containers, srun --container-image.\n" +
|
||||
"required " + PyxisPlugin + "\n"
|
||||
|
||||
// EnrootConfPath is enroot's settings.
|
||||
EnrootConfPath = "/etc/enroot/enroot.conf"
|
||||
// EnrootMounts has every container see the shared path.
|
||||
EnrootMounts = "/etc/enroot/mounts.d/50-llmgw-shared.fstab"
|
||||
// EnrootTmpfiles makes enroot's runtime directory at each boot.
|
||||
EnrootTmpfiles = "/etc/tmpfiles.d/llmgw-enroot.conf"
|
||||
// EnrootRun is where containers are unpacked while they run; EnrootLib
|
||||
// holds each user's image cache and named containers.
|
||||
EnrootRun = "/run/enroot"
|
||||
EnrootLib = "/var/lib/enroot"
|
||||
)
|
||||
|
||||
// EnrootPackage is one of enroot's release files.
|
||||
type EnrootPackage struct {
|
||||
Name, SHA256 string
|
||||
}
|
||||
|
||||
// URL is where the file is.
|
||||
func (p EnrootPackage) URL() string {
|
||||
return "https://github.com/NVIDIA/enroot/releases/download/v" + EnrootVersion + "/" +
|
||||
strings.ReplaceAll(p.Name, "+", "%2B")
|
||||
}
|
||||
|
||||
// EnrootPackages are the files for a family and architecture (Go's
|
||||
// names): enroot, and enroot+caps, which lets its helpers unpack images
|
||||
// for a user without root.
|
||||
func EnrootPackages(family, arch string) ([]EnrootPackage, error) {
|
||||
ps, ok := enrootPackages[family+"/"+arch]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("enroot %s has no package for %s on %s", EnrootVersion, family, arch)
|
||||
}
|
||||
return ps, nil
|
||||
}
|
||||
|
||||
var enrootPackages = map[string][]EnrootPackage{
|
||||
"deb/amd64": {
|
||||
{"enroot_4.2.1-1_amd64.deb", "5955c51e88df0c5ed538d87abb5264609d4bad55fa8570745ed5d4ec66c8ab91"},
|
||||
{"enroot+caps_4.2.1-1_amd64.deb", "ca22652dcbdd3ebb6e2c68df781ef4301647c79dd7a41ed62899c2fd0f692750"},
|
||||
},
|
||||
"deb/arm64": {
|
||||
{"enroot_4.2.1-1_arm64.deb", "267018b335815a6b32a562915de1f3fad579f7c9d6748e98c62e7369b40be421"},
|
||||
{"enroot+caps_4.2.1-1_arm64.deb", "12b811d0c95e879e42e4ac762fa85869a5f3cc9dbb06ab483c01e12e911284a1"},
|
||||
},
|
||||
"rpm/amd64": {
|
||||
{"enroot-4.2.1-1.el8.x86_64.rpm", "6749195e37e3a98071ec349884900b12de7677332599da0fd5d8afb5bed9233c"},
|
||||
{"enroot+caps-4.2.1-1.el8.x86_64.rpm", "56321a03a1f6c4e4273bf664917eb9a174714499d5506c498db4b64275a2b9c0"},
|
||||
},
|
||||
"rpm/arm64": {
|
||||
{"enroot-4.2.1-1.el8.aarch64.rpm", "6b6bc2526f0f0daa7d3cebe6dd33c0468b209e81eebd92def5777520042ec6fe"},
|
||||
{"enroot+caps-4.2.1-1.el8.aarch64.rpm", "38cf9ae9602a9a8c3e94637bc1b34e736e8176bda656ba71d371f5e1d1e1eed9"},
|
||||
},
|
||||
}
|
||||
|
||||
// PyxisBuildPackages build pyxis: a compiler and Slurm's headers.
|
||||
var PyxisBuildPackages = map[string][]string{
|
||||
"deb": {"gcc", "make", "libslurm-dev"},
|
||||
"rpm": {"gcc", "make", "slurm-devel"},
|
||||
}
|
||||
|
||||
// EnrootConf is enroot's settings: each user's containers, cache and
|
||||
// data apart, in directories that need no login session (a job has none,
|
||||
// and llmgw's users have no pam_systemd on a node).
|
||||
const EnrootConf = "# Written by llmgw (BACKLOG P): enroot, for Slurm's jobs in containers.\n" +
|
||||
"ENROOT_RUNTIME_PATH " + EnrootRun + "/user-$(id -u)\n" +
|
||||
"ENROOT_CACHE_PATH " + EnrootLib + "/cache/user-$(id -u)\n" +
|
||||
"ENROOT_DATA_PATH " + EnrootLib + "/data/user-$(id -u)\n" +
|
||||
"ENROOT_TEMP_PATH ${TMPDIR:-/tmp}\n" +
|
||||
// A container's root is the user's, as on the host: no fake root.
|
||||
"ENROOT_REMAP_ROOT no\n" +
|
||||
"ENROOT_ROOTFS_WRITABLE yes\n"
|
||||
|
||||
// EnrootTmpfilesConf makes EnrootRun, open to every user as /tmp is.
|
||||
const EnrootTmpfilesConf = "# Written by llmgw (BACKLOG P): enroot's runtime directory.\n" +
|
||||
"d " + EnrootRun + " 1777 root root -\n"
|
||||
|
||||
// EnrootSharedMount has a container see path where the host does.
|
||||
func EnrootSharedMount(path string) string {
|
||||
return "# Written by llmgw (BACKLOG P): the Slurm cluster's shared files, in every container.\n" +
|
||||
path + " " + path + " none x-create=dir,rbind,rw,nosuid\n"
|
||||
}
|
||||
|
||||
// The NVIDIA container library, for enroot's hook that gives a container
|
||||
// the job's GPUs: from NVIDIA's own repository, its key checked.
|
||||
const (
|
||||
NvidiaContainerKeyURL = "https://nvidia.github.io/libnvidia-container/gpgkey"
|
||||
NvidiaContainerKeySHA256 = "c880576d6cf75a48e5027a871bac70fd0421ab07d2b55f30877b21f1c87959c9"
|
||||
NvidiaContainerKeyPath = "/etc/apt/keyrings/llmgw-nvidia-container.asc"
|
||||
NvidiaContainerAptList = "/etc/apt/sources.list.d/llmgw-nvidia-container.list"
|
||||
NvidiaContainerYumRepo = "/etc/yum.repos.d/llmgw-nvidia-container.repo"
|
||||
)
|
||||
|
||||
// NvidiaContainerApt is the apt source.
|
||||
const NvidiaContainerApt = "# Written by llmgw (BACKLOG P): nvidia-container-cli, for GPUs in Slurm's containers.\n" +
|
||||
"deb [signed-by=" + NvidiaContainerKeyPath + "] https://nvidia.github.io/libnvidia-container/stable/deb/$(ARCH) /\n"
|
||||
|
||||
// NvidiaContainerYum is the yum/dnf repository.
|
||||
const NvidiaContainerYum = "# Written by llmgw (BACKLOG P): nvidia-container-cli, for GPUs in Slurm's containers.\n" +
|
||||
"[llmgw-nvidia-container]\nname=NVIDIA container library\n" +
|
||||
"baseurl=https://nvidia.github.io/libnvidia-container/stable/rpm/$basearch\n" +
|
||||
"repo_gpgcheck=1\ngpgcheck=0\nenabled=1\ngpgkey=" + NvidiaContainerKeyURL + "\n"
|
||||
|
||||
// NvidiaContainerPackages has nvidia-container-cli.
|
||||
var NvidiaContainerPackages = map[string][]string{"deb": {"libnvidia-container-tools"}, "rpm": {"libnvidia-container-tools"}}
|
||||
|
||||
// ContainersStatus is what a machine says it has.
|
||||
func ContainersStatus() string {
|
||||
return "pyxis " + PyxisVersion + ", enroot " + EnrootVersion
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
package slurm
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// Accounting: slurmdbd on the controller, keeping the jobs' records in a
|
||||
// MariaDB beside it that answers this machine alone. slurmdbd takes MySQL or
|
||||
// MariaDB only, so it is not the cluster's PostgreSQL. Since 23.11 Slurm
|
||||
// cannot run without it well: its main scheduler passes every job by as
|
||||
// InvalidAccount, and backfill starts it a pass later.
|
||||
const (
|
||||
// DbdPort is slurmdbd's, which slurmctld and sacct reach.
|
||||
DbdPort = 6819
|
||||
// DbdConfPath is slurmdbd's file, readable by the slurm user alone: it
|
||||
// holds the database's password.
|
||||
DbdConfPath = ConfDir + "/slurmdbd.conf"
|
||||
// AcctDB and AcctUser are the database and its user.
|
||||
AcctDB = "slurm_acct_db"
|
||||
AcctUser = "slurm"
|
||||
)
|
||||
|
||||
// DbdPassword is the database user's password, made from the munge key:
|
||||
// every controller the cluster has gets the same one, and nothing new is
|
||||
// kept anywhere.
|
||||
func DbdPassword(mungeKey []byte) string {
|
||||
h := sha256.New()
|
||||
h.Write([]byte("llmgw slurmdbd password\x00"))
|
||||
h.Write(mungeKey)
|
||||
return hex.EncodeToString(h.Sum(nil))[:40]
|
||||
}
|
||||
|
||||
// DbdConf is slurmdbd.conf for the controller host, named as `hostname -s`
|
||||
// says, which slurmdbd checks it runs on.
|
||||
func DbdConf(host, password string) string {
|
||||
return header + fmt.Sprintf(`AuthType=auth/munge
|
||||
DbdHost=%s
|
||||
DbdPort=%d
|
||||
SlurmUser=slurm
|
||||
LogFile=%s/slurmdbd.log
|
||||
PidFile=/run/slurmdbd.pid
|
||||
StorageType=accounting_storage/mysql
|
||||
StorageHost=localhost
|
||||
StorageUser=%s
|
||||
StoragePass=%s
|
||||
StorageLoc=%s
|
||||
# Kept a year; older records are purged, a month at a time.
|
||||
PurgeEventAfter=12months
|
||||
PurgeJobAfter=12months
|
||||
PurgeResvAfter=12months
|
||||
PurgeStepAfter=12months
|
||||
PurgeSuspendAfter=12months
|
||||
PurgeTXNAfter=12months
|
||||
PurgeUsageAfter=24months
|
||||
`, host, DbdPort, LogDir, AcctUser, password, AcctDB)
|
||||
}
|
||||
|
||||
// MariaDBConf is the database server's settings: this machine alone, and
|
||||
// what Slurm's documentation asks of InnoDB.
|
||||
const MariaDBConf = `# Written by llmgw (BACKLOG P): the database of Slurm's accounting
|
||||
# (slurmdbd), on this machine alone.
|
||||
[mysqld]
|
||||
bind-address=127.0.0.1
|
||||
skip-networking=0
|
||||
innodb_buffer_pool_size=256M
|
||||
innodb_lock_wait_timeout=900
|
||||
`
|
||||
|
||||
// AcctSQL makes the database and its user, and gives the user the password
|
||||
// again: run on every start of the controller's part, it changes nothing
|
||||
// that is already so.
|
||||
func AcctSQL(password string) string {
|
||||
return fmt.Sprintf(`CREATE DATABASE IF NOT EXISTS %[1]s;
|
||||
CREATE USER IF NOT EXISTS '%[2]s'@'localhost' IDENTIFIED BY '%[3]s';
|
||||
ALTER USER '%[2]s'@'localhost' IDENTIFIED BY '%[3]s';
|
||||
GRANT ALL ON %[1]s.* TO '%[2]s'@'localhost';
|
||||
FLUSH PRIVILEGES;
|
||||
`, AcctDB, AcctUser, password)
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
package slurm
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"path"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Logging in to a node: llmgw's users reach the controller, the machine
|
||||
// they submit from, whenever they like, and a node only while a job of
|
||||
// theirs runs there, their session put in that job (pam_slurm_adopt).
|
||||
// Without it a user could ssh to a node and use its cards outside Slurm,
|
||||
// on a node llmgw serves models from too.
|
||||
const (
|
||||
// PAMSSHD is sshd's PAM stack, where the lines go.
|
||||
PAMSSHD = "/etc/pam.d/sshd"
|
||||
// AdoptModule is the module's file name.
|
||||
AdoptModule = "pam_slurm_adopt.so"
|
||||
|
||||
pamBegin = "# llmgw (BACKLOG P): llmgw's Slurm users reach this node only while a job of theirs runs here."
|
||||
pamSession = "# llmgw (BACKLOG P): pam_systemd skipped for llmgw's Slurm users, so their session stays in the job pam_slurm_adopt put it in."
|
||||
pamWas = "# llmgw, in place of: "
|
||||
pamEnd = "# llmgw end"
|
||||
)
|
||||
|
||||
// AdoptPackages are the packages with AdoptModule, by family.
|
||||
var AdoptPackages = map[string][]string{"deb": {"libpam-slurm-adopt"}, "rpm": {"slurm-pam_slurm"}}
|
||||
|
||||
// ModuleDirs are where PAM's modules are, on Debian's and Red Hat's.
|
||||
var ModuleDirs = []string{
|
||||
"/usr/lib/x86_64-linux-gnu/security", "/lib/x86_64-linux-gnu/security",
|
||||
"/usr/lib/aarch64-linux-gnu/security", "/lib/aarch64-linux-gnu/security",
|
||||
"/usr/lib64/security", "/lib64/security", "/usr/lib/security", "/lib/security",
|
||||
}
|
||||
|
||||
// PAMAdopt is sshd's PAM stack with llmgw's lines in it (on) or taken out
|
||||
// and the stack as it was. Only the users listed in the file list, one a
|
||||
// line, are held to their jobs; anyone else, root and the site's
|
||||
// administrators, logs in as before.
|
||||
//
|
||||
// The account lines go after the last account line, so a user the
|
||||
// machine's own rules refuse is still refused. pam_systemd, which would
|
||||
// move the session out of the job into a scope of its own, is skipped for
|
||||
// the listed users: the file sshd includes it from (read through read) is
|
||||
// put in sshd's stack in its place, with the one line in front of
|
||||
// pam_systemd. contained is whether that was done; when it cannot be done
|
||||
// safely, the users are still held to their jobs, their sessions not to
|
||||
// the job's cards and cores.
|
||||
func PAMAdopt(stack, list string, read func(name string) (string, error), on bool) (out string, contained bool, err error) {
|
||||
lines, err := pamStrip(stack)
|
||||
if err != nil {
|
||||
return "", false, err
|
||||
}
|
||||
if !on {
|
||||
return join(lines), false, nil
|
||||
}
|
||||
last := -1
|
||||
for i, l := range lines {
|
||||
f := strings.Fields(l)
|
||||
if len(f) > 0 && (f[0] == "account" || (f[0] == "@include" && len(f) > 1 && strings.Contains(f[1], "account"))) {
|
||||
last = i
|
||||
}
|
||||
}
|
||||
if last < 0 {
|
||||
return "", false, errors.New(PAMSSHD + " has no account lines to put llmgw's after")
|
||||
}
|
||||
block := []string{
|
||||
pamBegin,
|
||||
// Listed: on to the next line; not listed, or no list: past it.
|
||||
"account [success=ignore default=1] pam_listfile.so item=user sense=allow file=" + list + " onerr=fail",
|
||||
"account required " + AdoptModule,
|
||||
pamEnd,
|
||||
}
|
||||
lines = append(lines[:last+1:last+1], append(block, lines[last+1:]...)...)
|
||||
lines, contained = pamSkipSystemd(lines, list, read)
|
||||
return join(lines), contained, nil
|
||||
}
|
||||
|
||||
func join(lines []string) string { return strings.Join(lines, "\n") + "\n" }
|
||||
|
||||
// pamStrip takes llmgw's blocks out, putting back the lines they stand in
|
||||
// place of.
|
||||
func pamStrip(stack string) ([]string, error) {
|
||||
var lines []string
|
||||
in := false
|
||||
for _, l := range strings.Split(strings.TrimSuffix(stack, "\n"), "\n") {
|
||||
switch {
|
||||
case !in && (l == pamBegin || l == pamSession):
|
||||
in = true
|
||||
case in && l == pamEnd:
|
||||
in = false
|
||||
case in && strings.HasPrefix(l, pamWas):
|
||||
lines = append(lines, strings.TrimPrefix(l, pamWas))
|
||||
case !in:
|
||||
lines = append(lines, l)
|
||||
}
|
||||
}
|
||||
if in {
|
||||
return nil, errors.New("llmgw's lines in " + PAMSSHD + " have no end; put them right by hand")
|
||||
}
|
||||
return lines, nil
|
||||
}
|
||||
|
||||
// pamLine is a PAM line's type, control and module, the control with its
|
||||
// brackets.
|
||||
var pamLine = regexp.MustCompile(`^\s*(-?[a-z]+)\s+(\[[^\]]*\]|\S+)\s+(\S+)`)
|
||||
|
||||
// pamJump is a jump of N in a bracketed control.
|
||||
var pamJump = regexp.MustCompile(`=(\d+)`)
|
||||
|
||||
func isSession(l string) bool {
|
||||
m := pamLine.FindStringSubmatch(l)
|
||||
return m != nil && strings.TrimPrefix(m[1], "-") == "session"
|
||||
}
|
||||
|
||||
// pamSkipSystemd puts the guard in front of pam_systemd, in the session
|
||||
// file sshd includes it from or in sshd's own lines; false, and lines as
|
||||
// they were, when it is in neither or cannot be done without changing
|
||||
// what any other jump in the stack lands on.
|
||||
func pamSkipSystemd(lines []string, list string, read func(string) (string, error)) ([]string, bool) {
|
||||
guard := "session [success=1 default=ignore] pam_listfile.so item=user sense=allow file=" + list + " onerr=fail"
|
||||
hasSystemd := func(l string) bool {
|
||||
m := pamLine.FindStringSubmatch(l)
|
||||
return m != nil && strings.TrimPrefix(m[1], "-") == "session" && path.Base(m[3]) == "pam_systemd.so"
|
||||
}
|
||||
for i, l := range lines {
|
||||
f := strings.Fields(l)
|
||||
var body []string
|
||||
switch {
|
||||
case hasSystemd(l):
|
||||
body = []string{l}
|
||||
case len(f) == 2 && f[0] == "@include" && strings.Contains(f[1], "session"),
|
||||
len(f) == 3 && f[0] == "session" && f[1] == "include":
|
||||
name := f[len(f)-1]
|
||||
if strings.Contains(name, "/") {
|
||||
continue
|
||||
}
|
||||
text, err := read(name)
|
||||
if err != nil {
|
||||
return lines, false
|
||||
}
|
||||
for _, il := range strings.Split(text, "\n") {
|
||||
t := strings.TrimSpace(il)
|
||||
if t == "" || strings.HasPrefix(t, "#") {
|
||||
continue
|
||||
}
|
||||
if f[0] == "session" && !isSession(t) {
|
||||
continue // include takes only the lines of its own type
|
||||
}
|
||||
body = append(body, t)
|
||||
}
|
||||
default:
|
||||
continue
|
||||
}
|
||||
at := -1
|
||||
for j, b := range body {
|
||||
if hasSystemd(b) {
|
||||
at = j
|
||||
}
|
||||
if g := strings.Fields(b); len(g) > 0 && (g[0] == "@include" || strings.Contains(b, "include") || strings.Contains(b, "substack")) {
|
||||
return lines, false // nested: not followed
|
||||
}
|
||||
}
|
||||
if at < 0 {
|
||||
continue
|
||||
}
|
||||
// A jump in sshd's own session lines may cross what is put in.
|
||||
for j, o := range lines {
|
||||
if j != i && isSession(o) && pamJump.MatchString(pamControl(o)) {
|
||||
return lines, false
|
||||
}
|
||||
}
|
||||
body, ok := pamInsert(body, at, guard)
|
||||
if !ok {
|
||||
return lines, false
|
||||
}
|
||||
block := append([]string{pamSession, pamWas + l}, body...)
|
||||
block = append(block, pamEnd)
|
||||
return append(lines[:i:i], append(block, lines[i+1:]...)...), true
|
||||
}
|
||||
return lines, false
|
||||
}
|
||||
|
||||
func pamControl(l string) string {
|
||||
if m := pamLine.FindStringSubmatch(l); m != nil {
|
||||
return m[2]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// pamInsert puts guard in front of body[at], a jump that crosses it made
|
||||
// one longer so it lands where it did.
|
||||
func pamInsert(body []string, at int, guard string) ([]string, bool) {
|
||||
// Each session line's place among the session lines.
|
||||
idx := make([]int, len(body))
|
||||
n := 0
|
||||
for j, b := range body {
|
||||
idx[j] = n
|
||||
if isSession(b) {
|
||||
n++
|
||||
}
|
||||
}
|
||||
out := make([]string, 0, len(body)+1)
|
||||
for j, b := range body {
|
||||
if j == at {
|
||||
out = append(out, guard)
|
||||
}
|
||||
ctl := pamControl(b)
|
||||
if isSession(b) && j < at && strings.HasPrefix(ctl, "[") {
|
||||
from, target := idx[j], idx[at]
|
||||
fixed := pamJump.ReplaceAllStringFunc(ctl, func(s string) string {
|
||||
k, err := strconv.Atoi(s[1:])
|
||||
if err != nil {
|
||||
return s
|
||||
}
|
||||
if from < target && target <= from+k {
|
||||
k++
|
||||
}
|
||||
return fmt.Sprintf("=%d", k)
|
||||
})
|
||||
b = strings.Replace(b, ctl, fixed, 1)
|
||||
}
|
||||
out = append(out, b)
|
||||
}
|
||||
return out, true
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
package slurm
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const ubuntuSSHD = `# PAM configuration for the Secure Shell service
|
||||
@include common-auth
|
||||
account required pam_nologin.so
|
||||
@include common-account
|
||||
session [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so close
|
||||
session required pam_loginuid.so
|
||||
@include common-session
|
||||
session required pam_limits.so
|
||||
@include common-password
|
||||
`
|
||||
|
||||
const ubuntuSession = `# here are the per-package modules (the "Primary" block)
|
||||
session [default=1] pam_permit.so
|
||||
session requisite pam_deny.so
|
||||
session required pam_permit.so
|
||||
session optional pam_umask.so
|
||||
session required pam_unix.so
|
||||
session optional pam_systemd.so
|
||||
# end of pam-auth-update config
|
||||
`
|
||||
|
||||
const redhatSSHD = `#%PAM-1.0
|
||||
auth substack password-auth
|
||||
account required pam_sepermit.so
|
||||
account required pam_nologin.so
|
||||
account include password-auth
|
||||
password include password-auth
|
||||
session required pam_loginuid.so
|
||||
session include password-auth
|
||||
session include postlogin
|
||||
`
|
||||
|
||||
const redhatPasswordAuth = `auth required pam_env.so
|
||||
account required pam_unix.so
|
||||
-session optional pam_keyinit.so revoke
|
||||
session required pam_limits.so
|
||||
-session optional pam_systemd.so
|
||||
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
|
||||
session required pam_unix.so
|
||||
`
|
||||
|
||||
func files(m map[string]string) func(string) (string, error) {
|
||||
return func(n string) (string, error) {
|
||||
if s, ok := m[n]; ok {
|
||||
return s, nil
|
||||
}
|
||||
return "", errors.New("no such file")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPAMAdopt(t *testing.T) {
|
||||
read := files(map[string]string{"common-session": ubuntuSession, "password-auth": redhatPasswordAuth, "postlogin": "session optional pam_lastlog.so\n"})
|
||||
for name, stack := range map[string]string{"ubuntu": ubuntuSSHD, "redhat": redhatSSHD} {
|
||||
on, contained, err := PAMAdopt(stack, "/var/lib/x-users", read, true)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
if !contained {
|
||||
t.Errorf("%s: pam_systemd not skipped:\n%s", name, on)
|
||||
}
|
||||
lines := strings.Split(on, "\n")
|
||||
if !strings.Contains(on, "account [success=ignore default=1] pam_listfile.so item=user sense=allow file=/var/lib/x-users onerr=fail") {
|
||||
t.Fatalf("%s: lines not put in:\n%s", name, on)
|
||||
}
|
||||
for j, l := range lines {
|
||||
if l == pamBegin && !strings.Contains(lines[j-1], "account") {
|
||||
t.Errorf("%s: put after %q", name, lines[j-1])
|
||||
}
|
||||
if strings.Contains(l, "pam_systemd.so") &&
|
||||
lines[j-1] != "session [success=1 default=ignore] pam_listfile.so item=user sense=allow file=/var/lib/x-users onerr=fail" {
|
||||
t.Errorf("%s: pam_systemd not guarded, after %q", name, lines[j-1])
|
||||
}
|
||||
}
|
||||
again, _, _ := PAMAdopt(on, "/var/lib/x-users", read, true)
|
||||
if again != on {
|
||||
t.Errorf("%s: put in twice:\n%s", name, again)
|
||||
}
|
||||
off, _, err := PAMAdopt(on, "", read, false)
|
||||
if err != nil || off != stack {
|
||||
t.Errorf("%s: taken out, not as it was (%v):\n%s", name, err, off)
|
||||
}
|
||||
}
|
||||
// The jump over pam_deny lands where it did; the one past crond's line too.
|
||||
on, _, _ := PAMAdopt(ubuntuSSHD, "/l", read, true)
|
||||
if !strings.Contains(on, "[default=1]\t\t\tpam_permit.so") {
|
||||
t.Errorf("a jump that does not cross the guard was changed:\n%s", on)
|
||||
}
|
||||
body := []string{"session [default=2] pam_permit.so", "session required pam_unix.so", "session optional pam_systemd.so", "session required pam_x.so"}
|
||||
got, _ := pamInsert(body, 2, "G")
|
||||
if got[0] != "session [default=3] pam_permit.so" || got[2] != "G" {
|
||||
t.Errorf("a jump across the guard not made longer: %q", got)
|
||||
}
|
||||
|
||||
if _, _, err := PAMAdopt("auth required pam_unix.so\n", "/x", read, true); err == nil {
|
||||
t.Error("a stack with no account lines taken")
|
||||
}
|
||||
if _, _, err := PAMAdopt(pamBegin+"\naccount required x\n", "/x", read, false); err == nil {
|
||||
t.Error("lines with no end taken")
|
||||
}
|
||||
// The session file unreadable: held to their jobs all the same.
|
||||
on, contained, err := PAMAdopt(ubuntuSSHD, "/l", files(nil), true)
|
||||
if err != nil || contained || !strings.Contains(on, AdoptModule) {
|
||||
t.Errorf("contained %v, err %v:\n%s", contained, err, on)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
package slurm
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"golang.org/x/crypto/ssh"
|
||||
)
|
||||
|
||||
// Shared storage: one path every machine of the Slurm cluster sees, from
|
||||
// the site's NFS export, from an llmgw node that exports it, or mounted by
|
||||
// the site itself.
|
||||
const (
|
||||
// NFSPort is NFSv4's one port, which is all a client needs.
|
||||
NFSPort = 2049
|
||||
// ExportsPath is the export llmgw keeps on the machine that serves.
|
||||
ExportsPath = "/etc/exports.d/llmgw-slurm.exports"
|
||||
// MountUnitDir is where the mount unit is written.
|
||||
MountUnitDir = "/etc/systemd/system"
|
||||
)
|
||||
|
||||
// Exports is the exports file for path, read-write to clients alone. Root
|
||||
// on a client is nobody here: a client's root cannot take another's files.
|
||||
func Exports(path string, clients []string) string {
|
||||
cs := slices.Clone(clients)
|
||||
slices.Sort(cs)
|
||||
cs = slices.Compact(cs)
|
||||
var b strings.Builder
|
||||
b.WriteString("# Written by llmgw (BACKLOG P): the Slurm cluster's shared files, to its machines alone.\n")
|
||||
if len(cs) == 0 {
|
||||
return b.String()
|
||||
}
|
||||
fmt.Fprintf(&b, "%q", path)
|
||||
for _, c := range cs {
|
||||
fmt.Fprintf(&b, " %s(rw,sync,no_subtree_check,root_squash)", c)
|
||||
}
|
||||
b.WriteString("\n")
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// MountUnitName is systemd's name for the mount unit of path, as
|
||||
// systemd-escape --path makes it.
|
||||
func MountUnitName(path string) string {
|
||||
p := strings.Trim(path, "/")
|
||||
var b strings.Builder
|
||||
for i := 0; i < len(p); i++ {
|
||||
c := p[i]
|
||||
switch {
|
||||
case c == '/':
|
||||
b.WriteByte('-')
|
||||
case c >= 'a' && c <= 'z', c >= 'A' && c <= 'Z', c >= '0' && c <= '9', c == '_', c == '.' && i > 0:
|
||||
b.WriteByte(c)
|
||||
default:
|
||||
fmt.Fprintf(&b, `\x%02x`, c)
|
||||
}
|
||||
}
|
||||
return b.String() + ".mount"
|
||||
}
|
||||
|
||||
// MountUnit mounts the export nfs at path. hard: a job waits for a server
|
||||
// that is gone rather than reading errors; nconnect spreads a client's
|
||||
// traffic over several connections. The version is the highest both
|
||||
// sides speak.
|
||||
func MountUnit(nfs, path string) string {
|
||||
return "# Written by llmgw (BACKLOG P): the Slurm cluster's shared files.\n" +
|
||||
"[Unit]\nDescription=llmgw: the Slurm cluster's shared files\n" +
|
||||
"After=network-online.target\nWants=network-online.target\n\n" +
|
||||
"[Mount]\nWhat=" + nfs + "\nWhere=" + path + "\nType=nfs\n" +
|
||||
"Options=hard,nconnect=4,_netdev\nTimeoutSec=60\n\n" +
|
||||
"[Install]\nWantedBy=remote-fs.target\n"
|
||||
}
|
||||
|
||||
// Users: Linux users llmgw makes from its accounts, the same login and UID
|
||||
// on every machine, logging in with the SSH keys given in the portal.
|
||||
const (
|
||||
// AuthorizedKeysDir holds each user's keys, root's to write, so a
|
||||
// user's own home (on a file system shared with others) need hold none.
|
||||
AuthorizedKeysDir = "/etc/llmgw/ssh-keys"
|
||||
// SSHDropIn has sshd look there too, after the user's own files.
|
||||
SSHDropIn = "/etc/ssh/sshd_config.d/50-llmgw-slurm.conf"
|
||||
// MinUID is the lowest UID llmgw gives.
|
||||
MinUID = 200000
|
||||
)
|
||||
|
||||
// SSHConf is SSHDropIn's settings.
|
||||
const SSHConf = "# Written by llmgw (BACKLOG P): the Slurm users' keys, from llmgw's portal.\n" +
|
||||
"AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 " + AuthorizedKeysDir + "/%u\n"
|
||||
|
||||
// login is what useradd takes, as llmgw gives it.
|
||||
var login = regexp.MustCompile(`^[a-z][a-z0-9_-]{0,31}$`)
|
||||
|
||||
// ValidLogin is whether a login is one llmgw gives.
|
||||
func ValidLogin(s string) bool { return login.MatchString(s) }
|
||||
|
||||
// ParseSSHKeys reads OpenSSH public keys, one a line, as authorized_keys
|
||||
// takes them, without options: what a person pastes into the portal.
|
||||
// Blank lines and # comments are skipped; each key comes back as its type,
|
||||
// its base64 and its comment.
|
||||
func ParseSSHKeys(text string) ([]string, error) {
|
||||
var out []string
|
||||
for i, line := range strings.Split(text, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
pk, comment, opts, _, err := ssh.ParseAuthorizedKey([]byte(line))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("line %d is not an SSH public key (as ~/.ssh/id_ed25519.pub has it)", i+1)
|
||||
}
|
||||
if len(opts) > 0 {
|
||||
return nil, fmt.Errorf("line %d has options before its key; only the key is taken", i+1)
|
||||
}
|
||||
k := strings.TrimSpace(string(ssh.MarshalAuthorizedKey(pk)))
|
||||
if comment = strings.Join(strings.Fields(comment), " "); comment != "" {
|
||||
k += " " + comment
|
||||
}
|
||||
if !slices.Contains(out, k) {
|
||||
out = append(out, k)
|
||||
}
|
||||
}
|
||||
if len(out) > 20 {
|
||||
return nil, fmt.Errorf("%d keys: 20 at most", len(out))
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
package slurm
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestExportsAndMountUnit(t *testing.T) {
|
||||
e := Exports("/shared", []string{"10.0.0.22", "10.0.0.21", "10.0.0.22"})
|
||||
if !strings.Contains(e, `"/shared" 10.0.0.21(rw,sync,no_subtree_check,root_squash) 10.0.0.22(rw,`) ||
|
||||
strings.Count(e, "10.0.0.22") != 1 {
|
||||
t.Fatalf("exports:\n%s", e)
|
||||
}
|
||||
if strings.Contains(Exports("/shared", nil), "/shared\"") {
|
||||
t.Fatal("an export to nobody")
|
||||
}
|
||||
for path, want := range map[string]string{"/shared": "shared.mount", "/data/hpc": "data-hpc.mount",
|
||||
"/hpc-data": `hpc\x2ddata.mount`, "/.b": `\x2eb.mount`} {
|
||||
if got := MountUnitName(path); got != want {
|
||||
t.Errorf("%s: %s, want %s", path, got, want)
|
||||
}
|
||||
}
|
||||
if u := MountUnit("10.0.0.11:/shared", "/shared"); !strings.Contains(u, "What=10.0.0.11:/shared\nWhere=/shared\nType=nfs") {
|
||||
t.Fatalf("unit:\n%s", u)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseSSHKeys(t *testing.T) {
|
||||
key := "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILq3vMdLAKlaDdxwrSgCtP/j/qrF4nzxNBNGuJmK5vxh"
|
||||
got, err := ParseSSHKeys("# mine\n\n" + key + " ayse@laptop \n" + key + " ayse@laptop\n")
|
||||
if err != nil || len(got) != 1 || got[0] != key+" ayse@laptop" {
|
||||
t.Fatalf("%q %v", got, err)
|
||||
}
|
||||
for _, bad := range []string{"not a key", `command="rm -rf /" ` + key} {
|
||||
if _, err := ParseSSHKeys(bad); err == nil {
|
||||
t.Errorf("%q taken", bad)
|
||||
}
|
||||
}
|
||||
if !ValidLogin("ali_veli") || ValidLogin("9x") || ValidLogin("Ayse") {
|
||||
t.Error("ValidLogin")
|
||||
}
|
||||
}
|
||||
Loaded 100 of 134 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user