Compare commits

...
23 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 0774e901c0 Slurm: a new leader with nothing kept waits for the nodes a while
CI / test (push) Failing after 7m57s
On a cluster upgraded to the kept nodes, the first leader finds none and
wrote slurm.conf without the nodes it had not heard from yet, as before.
A node neither reported nor kept now holds the controller's files back
for slurmsvc.HoldFor (2 minutes) from the leader's first pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:37:52 +01:00
Ilker ManapandClaude Opus 5.5 141a1f46c5 Slurm: a new leader keeps the nodes it has not heard from yet
CI / test (push) Successful in 10m50s
Tried on the PoC: with a job on a lent node, the leader restarted and its
first slurm.conf had no node in it (none had reported to it yet), so
slurmctld dropped the node and the job ended in NODE_FAIL.

The leader keeps slurm.conf's nodes and the controller's address in the
store's cluster state (slurm-nodes); the next one starts from them and
writes the same files until the nodes report again. With the store
unread, the controller gets no files that leave a node out. A node is
told nothing, rather than that it has no part, until its controller's
address is known: no part stops slurmd, and its jobs with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:33:53 +01:00
Ilker ManapandClaude Opus 5.5 3387f97f6a Slurm's ports for the cluster only, and srun in a range of its own
CI / test (push) Successful in 9m40s
slurm.conf sets SrunPortRange=60001-63000, so a site's firewall can let
srun's step connections in. A node given a part in Slurm adds 6817-6818
and that range to llmgw's nftables table, open to the cluster's networks
and closed to everyone else, as the engines' ports are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:10:37 +01:00
Ilker ManapandClaude Opus 5.5 914985bf9f Slurm in the first install: server init --slurm, and slurm.all_nodes
CI / test (push) Successful in 9m30s
The choices screen has a Slurm part, and --slurm writes slurmctld on this
machine's node and slurmd on every node, those that join later too, into
server.yaml. slurm.all_nodes (ctl slurm setup --all-nodes) takes the nodes
from the leader's inventory. No node is lent at install.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 14:50:35 +01:00
Ilker ManapandClaude Opus 5.5 b15944aa1f A package install that is no engine's clears its progress; slurmd retried
CI / test (push) Successful in 10m15s
Found on the PoC: InstallPackages reported 'installing munge, slurm-client,
slurmd' under no engine and never cleared it, so the node's page kept
saying so (infiniband-diags and perftest did the same). slurmd started
before the leader has written its node into slurm.conf is refused; the
drop-in now has systemd try it again every 15 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:23:53 +01:00
Ilker ManapandClaude Opus 5.5 dcf4e256e6 BACKLOG P: tried on the PoC
CI / test (push) Failing after 7m26s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:20:32 +01:00
Ilker ManapandClaude Opus 5.5 849760c67d Slurm: no second restart; backfill every 10 s, MailProg set
The InvalidAccount seen on the PoC was not the reconfigure: 23.11.4 without
accounting storage has its main scheduler mark a batch job so, and backfill
starts it on its next pass (job 4 waited 15 s). bf_interval=10 keeps that
short; the extra restart is taken out again. MailProg=/bin/true quiets the
warning on every start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:18:35 +01:00
Ilker ManapandClaude Opus 5.5 087552136f Slurm: slurmctld started again after a reconfigure; no accounting gathered
Found on the PoC: Slurm 23.11.4 without accounting storage left batch jobs
pending InvalidAccount after scontrol reconfigure until slurmctld was
restarted. And jobacct_gather with nowhere to keep it logs an error per job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:16:31 +01:00
Ilker ManapandClaude Opus 5.5 d2c27e2c76 Slurm: slurmd started as its unit expects, and slurmd's own stepd not a job
Found on the PoC (Ubuntu 24.04, Slurm 23.11.4): the packaged unit is
Type=notify, so slurmd needs --systemd or systemd times it out; and slurmd
keeps a 'slurmstepd infinity' for its cgroup v2 scope, which was counted as
a job and kept the node from llmgw.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:11:41 +01:00
Ilker ManapandClaude Opus 5.5 8a5a04ed77 Slurm: the version from the daemon, not scontrol
Found on the PoC: scontrol looks for its files even for --version, and a
configless node has none before slurmd has fetched them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:06:42 +01:00
Ilker ManapandClaude Opus 5.5 410fe545ce Slurm beside llmgw, P3 and P4: the leader hands nodes over; ctl slurm
CI / test (push) Successful in 9m40s
internal/control/slurmsvc tells each node what it is, writes slurm.conf
and gres.conf from the nodes' reports and the inventory, and moves each node
of slurm.nodes between llmgw and Slurm: lent, drained of replicas first, then
resumed in Slurm; taken back, drained in Slurm and cordoned in llmgw until
its last job step ends. Kept every tick from the settings and the agents'
reports, so a new leader carries on. A broken Slurm never keeps a node from
llmgw. GET /api/v1/slurm; ctl slurm status, setup, lend, reclaim, off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:33:14 +01:00
Ilker ManapandClaude Opus 5.5 35de47444c Slurm beside llmgw, P2: the agent keeps its node's part
A Slurm message says whether the node runs slurmctld on the leader's files,
slurmd configless, both or neither, and whether it is lent. The agent
installs munge and Slurm from the machine's sources (22.05 or newer, for
cgroup v2), writes the munge key, starts slurmd through a drop-in with
--conf-server, resumes a lent node that llmgw drained and drains one that is
not, never touching a drain a Slurm administrator made, and reports its
version, slurmd -C, state and jobs in SlurmStatus. Told it has no part, it
stops what it started. New nodes enter slurm.conf drained.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:25:41 +01:00
Ilker ManapandClaude Opus 5.5 d439d507d7 Slurm beside llmgw, P1: its settings and the files a controller runs on
slurm.{enabled,cluster_name,controller,nodes,lent,partition} in the cluster's
settings, checked; internal/slurm writes slurm.conf (configless), gres.conf
and cgroup.conf from the nodes' slurmd -C lines and GPUs, derives the munge
key from a cluster secret, and reads scontrol and version output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:20:31 +01:00
Ilker ManapandClaude Opus 5.5 89767c8be1 BACKLOG P: training beyond one card, in llmgw or Slurm beside it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:12:07 +01:00
Ilker ManapandClaude Opus 5.5 d864438272 A model that failed to start is placed elsewhere next
CI / test (push) Successful in 9m31s
The same inputs place a replica on the same node and card, so an on-demand
model that failed to load there was started there again after every
backoff until it was parked. The reconciler remembers where a model's
replicas failed and places the next away from them when anything else
fits (Scheduler.Away); with nowhere else, there again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 09:22:11 +01:00
Ilker ManapandClaude Opus 5.5 6320dd5b13 A new stream is told the node's files again
CI / test (push) Successful in 9m31s
Registration carries what the agent's startup scan found, and a heartbeat
sends the list only when it changed since last sent; so a file deleted
after the agent started came back on a control plane that restarted. The
fingerprint is now forgotten with the provisioning one on every stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:14:52 +01:00
Ilker ManapandClaude Opus 5.5 5732a5ebda The serve forms start at the longest context that fits
CI / test (push) Successful in 9m26s
The mirror page's form began every GGUF model at 4096 tokens, the
console's at 4096 or the plan's figure. Both now start at what the model
was trained for, cut to the most whose KV cache fits beside the weights
where the form places it; the mirror page's slots are as many of that, up
to four, as the room left holds. Plan carries the trained context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 14:38:41 +01:00
Ilker ManapandClaude Opus 5.5 561a536477 The weight sweep names a set as its fetch did
CI / test (push) Successful in 9m33s
A set is fetched into a directory named from its address without the
trailing slash; Orphans named it from the catalog's address with it, so
every set in the catalog was offered for deletion: 330 GB of six models on
the PoC. One function, cacheSource, now gives both the name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 14:01:27 +01:00
Ilker ManapandClaude Opus 5.5 45fe7daaa0 The admin API document regenerated for CollectResult's left counts
CI / test (push) Successful in 9m32s
cba4897 added fields without regenerating it, and
TestTheAdminAPIDocumentIsNotStale failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 11:38:08 +01:00
Ilker ManapandClaude Opus 5.5 102ca9556f An organization's admin, and forms only for those who may send them
C1. A membership of an organization may be admin as well as member and
manager. An admin sees /org as a manager does and changes the
organization from it: its teams and projects, who of the approved
accounts is in them, its members' roles, and the keys spent in its
projects. Every id a form names is checked against the organization;
limits, accounts, the organization's name and the cluster stay with the
cluster's staff, and an admin cannot take their own role away.

A group from OIDC, SCIM or a directory no longer rewrites a membership
it names as a plain member at every sign-in, which took a manager's role
away (EnsureMembership).

Pages draw a form only for someone who may send it: Nav.May asks the
staff rule staffMay keeps, and a page whose forms a person may not send
gets no form token. A viewer sees the people, limits and mirror pages
without forms; an operator the mirror's.

Forms are compact: labels close to their fields, fields as wide as what
goes in them, smaller buttons, and the short forms on one line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 11:38:06 +01:00
Ilker ManapandClaude Opus 5.5 2cb1afda55 The introduction's comparison on one slide
CI / test (push) Failing after 6m38s
Four slides of products and cards become one table: llmgw beside API
gateways (LiteLLM, Kong AI Gateway, Agent Router) and serving stacks
(llm-d, NVIDIA Dynamo, vLLM production-stack), six rows. The PDF is
18 slides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 17:06:49 +01:00
Ilker ManapandClaude Opus 5.5 b6478e24b7 An introduction for decision makers, linked from the website
CI / test (push) Failing after 6m43s
docs/llmgw-intro.pdf is a 21-slide presentation: what llmgw adds on
premise, screens from a running cluster, and how it compares with
LiteLLM, Kong AI Gateway, Agent Router, llm-d, NVIDIA Dynamo and vLLM
production-stack from their own documentation. The /about page and the
published site link to it in the repository, so the binary does not
carry it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 14:36:47 +01:00
Ilker ManapandClaude Opus 5.5 cba4897987 Deleting unused model files says what went, and sets go whole
CI / test (push) Failing after 6m44s
A sweep counted every file the nodes were asked to delete as freed as soon
as it asked, and a node refused a set's directory as not empty, so tens of
gigabytes were reported freed and nothing was. The agent removes a set's
directory whole (a link only as a link), and the control plane waits for
each node to list its files again, saying what it no longer lists and what
it kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:29:16 +01:00
102 changed files with 5890 additions and 354 deletions

No files matched your search

+75 -4
View File
@@ -49,7 +49,8 @@ Face, done 2026-10-30), then M5 (engines without the internet), M3, M4, M6.
**Scale.** O: more than one control plane answering clients; O1 (measuring one router's ceiling) first.
**Governance.**
7. **C1**: a tenant-admin role, and forms hidden from viewers.
7. ~~**C1**~~: done 2026-11-08. An organization's admin (`--role admin` on an org membership) changes its teams,
projects, people and keys from /org; viewers are drawn no forms; forms made compact.
8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
**When there is room.**
@@ -435,9 +436,9 @@ portal is not done — the page shows; the console and ctl change.
API middleware and the /admin pages (with each change's result), `ctl
audit`, the console's Audit view; staff roles viewer / operator /
billing / admin. Left: before/after values (the request's fields are
recorded, not the state they replaced); a tenant-admin role that changes
its own organization (managers still only view it); hiding the forms a
viewer cannot submit. **Audit log and roles (M).** Today "audit" is slog lines
recorded, not the state they replaced). Done 2026-11-08: an organization's
admin who changes its own organization from /org (managers still only view
it), and the forms a viewer cannot submit no longer drawn. **Audit log and roles (M).** Today "audit" is slog lines
(`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag
plus an org "manager" who can view. Add `audit_events` (who, what,
before/after, source) written by the `adminAction`/`tenancyAction`
@@ -1982,6 +1983,76 @@ one control plane's router can. In order:
- O3. Only if O2 is not enough: stateless gateway machines (`llmgw server --role gateway`) in front of the leader's
control plane, as many as the traffic needs.
## P. Training beyond one card (written 2026-11-11) — decision open
F4 trains a LoRA on one card of a node the caller names (`ctl train start --node`), one job per node, no resume.
Larger work (full fine-tuning, a LoRA on a 70B, several GPUs or machines) needs more. Two ways:
**A. In llmgw.** Add, in order: a job that takes N whole GPUs on one node (torchrun, FSDP/DeepSpeed in the agent's
venv); placement by the control plane instead of `--node`, freeing GPUs from replicas by the scaling schedule or by
draining; checkpoints to shared storage and resume when an agent or node goes; datasets uploaded or read from
storage, not only KB pairs; then gang placement of N nodes with a rendezvous on the first, the NCCL environment from
the fabric discovery (IB/NVLink, `llmgw doctor`); a console screen with logs and loss. Size: L for one node, XL for
several. For: one product, one install, one set of users and grants (KB rules carry over), GPUs shared with
inference without a second scheduler. Against: it is Slurm's gang scheduling, requeue and accounting written again;
multi-node NCCL cannot be tried on the PoC (one GPU node, no IB); every customer's training code would have to fit
our job format.
**B. Slurm beside llmgw.** The TUI installs Slurm on the nodes the customer picks (munge, slurmctld on the control
planes, slurmd, users from the same LDAP/AD), with pyxis/enroot for containers. A node is handed between the two:
llmgw drains its replicas and `scontrol update state=resume`, and back. llmgw's part is what comes out: a checkpoint
or adapter from a Slurm job published to the mirror (`ctl train import`), served through canary and the eval. Size:
M. For: what research and university customers already run (`sbatch`, `srun torchrun`), any training code, mature
multi-node and requeue; we write only the install and the handover. Against: a second system with its own failures,
upgrades and users to keep in step; GPUs split by hand or by the handover, not by demand; access rules on training
data are Slurm's and the file system's, not the KB's.
Both need shared storage (NFS, BeeGFS, Lustre) the customer provides; neither changes F4.
Recommendation: keep F4 in llmgw for small single-card adapters tied to KB grants; take B for anything larger;
build only A's first step (N GPUs on one node, placed by the control plane, with resume) if customers ask for
mid-size jobs without wanting Slurm.
**B taken (2026-11-11).** Design: the cluster's settings say it (`slurm: enabled, cluster_name, controller, nodes,
lent, partition`); the leader tells each named node's agent (root) what to be, the agents install and run it.
- The controller is an llmgw node (its agent runs slurmctld); `nodes` run slurmd configless
(`--conf-server`), so slurm.conf, gres.conf and cgroup.conf live only on the controller.
- The munge key is derived from the cluster secret (HKDF), so nothing new is stored or copied by hand.
- Each node reports `slurmd -C`; the leader writes slurm.conf from those lines, its mgmt address and its GPUs
(`Gres=gpu:<type>:N`, gres.conf with the device files), and sends it to the controller again when one changes.
- A node in `nodes` belongs to llmgw until it is in `lent`: drained in Slurm (reason "llmgw: serving models").
Lent: llmgw drains it as `ctl drain` does (kept in force each tick, as facility holds are, so a new leader
keeps it) and, once no replica is left, its agent resumes it in Slurm. Taken out of `lent`: the agent drains
it in Slurm, reports the jobs left, and when none are the leader makes it ready for llmgw again.
- Phases: P1 the `slurm` package (conf, key, parsers) and settings; P2 the agent side (proto, install, munge,
slurmd/slurmctld, status); P3 the leader side (conf from reports, lend/reclaim); P4 `ctl slurm`
(status, lend, reclaim) and a TUI screen; P5 `ctl train import` of a Slurm job's output into the mirror.
Later: a backup slurmctld on a second node, accounting (slurmdbd on the Patroni PG), users from LDAP/AD.
- Tried on the PoC 2026-11-11 (dev build v0.16.3-17-g849760c-slurm on 140, 150, node05, node06; Ubuntu
24.04, Slurm 23.11.4 from the archive): `ctl slurm setup --controller llmgw-node06 --nodes llmgw-node05`
installed munge, slurmctld and slurmd, node05 entered drained with its slurmd -C hardware (8 CPUs,
31132 MiB after the 3%). `lend`: its 2 replicas moved off, then IDLE in Slurm; srun and sbatch ran there.
`reclaim` with a 90 s job running: MIXED+DRAIN, llmgw kept off, ready again within a minute of the job
ending. Found and fixed on the way: a client command needs its files even for --version (version now
from the daemon); the packaged unit is Type=notify (slurmd needs --systemd); slurmd's own "slurmstepd
infinity" was counted as a job; without accounting storage 23.11's main scheduler marks batch jobs
InvalidAccount and backfill starts them, one a pass (bf_interval=10: 6-10 s each). So many small jobs
want slurmdbd. Not tried yet: a GPU node (gres), a leader failover while lent.
- Built 2026-11-11: P1 (`internal/slurm`, `slurm.*` settings), P2
(`internal/agent/slurmnode`, proto Slurm/SlurmStatus), P3 (`internal/control/slurmsvc`, wired in the
leader; `GET /api/v1/slurm`), P4's commands (`ctl slurm status|setup|lend|reclaim|off`). Decisions on the
way: a node not heard from or with Slurm jobs is cordoned, not drained (nothing of llmgw's to move, and a
new leader must not move models off every Slurm node); jobs are counted from the node's own slurmstepd,
so a controller that is down neither hides them nor, with none, keeps the node from llmgw; a drain whose
reason does not start with "llmgw" is a Slurm administrator's and never resumed. Left: the TUI screen, P5,
a try on the PoC (needs agents built from this; slurm-wlm 23.11 on Ubuntu 24.04), GPU minor numbers are
taken to be nvidia-smi's index.
- In the first install too (2026-11-11): `llmgw server init --slurm`, and a Slurm part on the choices
screen, write `slurm: {enabled, controller: <this hostname>, all_nodes: true}` into server.yaml.
`slurm.all_nodes` (also `ctl slurm setup --all-nodes`) runs slurmd on every node in the leader's
inventory, those that join later too; none is lent. Refused with `--no-node`, or a hostname Slurm does
not take.
## D. Facts worth knowing before starting
- Admin API: `127.0.0.1:9090`, behind the token in
+87
View File
@@ -26,6 +26,93 @@ is only released once its **exit criterion** is met, not when its code lands.
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased
### Added
- **An organization's admin.** Besides member and manager, a membership of
an organization may now be `admin` (`llmgw users join … --org Acme --role
admin`, the role chosen on a membership on the people page, the console,
the admin API). An admin sees the organization's page as a manager does and
also changes the organization from it: adds, renames and removes its teams
and projects, adds people with an approved account by their address and
takes them out, makes its members managers or admins, and revokes keys spent
in its projects. Limits, approving and suspending accounts, the
organization's name and the rest of the cluster stay with the cluster's
administrators; an admin cannot take away their own role. Every change is in
the audit log.
- **Slurm beside llmgw, for training and batch work.** `llmgw ctl slurm setup
--controller NODE --nodes NODE,...` (or `--all-nodes`, or `server init
--slurm`) has the agents install munge and Slurm from the machines' package
sources: slurmctld on one node, slurmd configless on the others, slurm.conf
and gres.conf written from what each node reports. Every node stays llmgw's,
drained in Slurm, until `ctl slurm lend NODE`: its replicas move off, then
Slurm runs jobs there. `ctl slurm reclaim NODE` takes it back once its jobs
end. `ctl slurm status` shows each node. Slurm's ports (6817–6818, srun's
60001–63000) answer the cluster's networks only, as llmgw's own do.
### Changed
- **Forms only for those who may send them.** A viewer sees the people,
limits, mirror and cluster pages without their forms, an operator the
mirror's and a node's engines but not the rest; before, they were drawn and
answered "not found".
- **Smaller forms.** A label sits close to its field, a field is as wide as
what goes in it rather than the page, buttons are smaller, and the short
forms — a new key, a password, connecting an editor, a capacity or a limit —
sit on one line or two instead of a screen.
### Fixed
- **The serve forms start at the longest context that fits.** The mirror
page's form began every GGUF model at 4096 tokens, and the console's at
4096 or the plan's figure, whatever the model was trained for. Both now
start at what it was trained for, as long as its KV cache fits where the
form places it beside the weights, else at the most that does, in whole
thousands; the mirror page's llama.cpp slots are as many of that, up to
four, as the room left holds, and its memory figure is for that pool. A
plan says what the model was trained for (`trained`). Strata stays at
its 32768; vLLM's form is unchanged.
- **A model that failed to start was started in the same place again.**
Placement given the same inputs chooses the same node and card, so an
on-demand model whose replica failed to load was retried there, past each
backoff, until it was parked, with another card or node free all along.
The control plane now remembers where a model's replicas failed (node and
cards, up to eight places) and places the next one elsewhere when anywhere
else fits; with nowhere else it is tried there as before. Coming up, a
change to its definition or `ctl retry` forgets the places. The log says
`placing a model away from where it failed to start`, with both places.
- **Deleted model files came back after a control plane restart.** An agent
registers with the files its startup scan found, and sends the list again
only when it changes; a file deleted after the agent started was listed
again by a control plane that restarted, until the next change. A new
stream now gets the list on its first heartbeat.
- **`ctl weights` offered models in use for deletion.** A model of several
files (a set, its address ending in `/`) is fetched into a directory named
from the address without the slash; the sweep named it from the address
with it, so every set in the catalog looked like nobody's. On the PoC that
was 330 GB of six catalog models, the qwen3.8-flash-next Q8_0 among them;
`--collect`, and `ctl model remove --purge` of a set, would have deleted
them. Both now compute the name the fetch does.
- **A sign-in took an organization's manager role away.** A group from single
sign-on, SCIM or Active Directory that puts someone in an organization wrote
the membership again at every sign-in or sync, as a plain member: a manager
made one on the people page was one only until they next signed in. A
membership the directory names is now added if missing and otherwise left
as it is.
- **The limits form on the people page's panel** was laid out on one line, its
title squeezed into a column.
- **Deleting the model files nothing uses said what it was asked to free,
not what it freed.** The console's "Delete model files nothing uses" and
`ctl weights --collect` added up every file the nodes were asked to
delete as soon as the request was sent, and a model of several files —
fetched into a directory of its own — was refused by the node as "not
empty", so a sweep said tens of gigabytes freed and nothing went. The
node now deletes such a directory whole (a link only as a link), and the
sweep waits for the nodes to list their files again: freed is what they
no longer list, and what they kept is said beside, with where to read
why. Removing a model with its files reports the same way.
## v0.16.3 — 2026-11-03
Replicas counted per profile — seven on cards and four on CPUs under one name, the cards filled first — and every
+365 -73
View File
@@ -108,6 +108,7 @@ type AgentMessage struct {
// *AgentMessage_KvEvents
// *AgentMessage_TrainStatus
// *AgentMessage_TrainArtifact
// *AgentMessage_SlurmStatus
Msg isAgentMessage_Msg `protobuf_oneof:"msg"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
@@ -240,6 +241,15 @@ func (x *AgentMessage) GetTrainArtifact() *TrainArtifact {
return nil
}
func (x *AgentMessage) GetSlurmStatus() *SlurmStatus {
if x != nil {
if x, ok := x.Msg.(*AgentMessage_SlurmStatus); ok {
return x.SlurmStatus
}
}
return nil
}
type isAgentMessage_Msg interface {
isAgentMessage_Msg()
}
@@ -284,6 +294,10 @@ type AgentMessage_TrainArtifact struct {
TrainArtifact *TrainArtifact `protobuf:"bytes,10,opt,name=train_artifact,json=trainArtifact,proto3,oneof"`
}
type AgentMessage_SlurmStatus struct {
SlurmStatus *SlurmStatus `protobuf:"bytes,11,opt,name=slurm_status,json=slurmStatus,proto3,oneof"`
}
func (*AgentMessage_Register) isAgentMessage_Msg() {}
func (*AgentMessage_Heartbeat) isAgentMessage_Msg() {}
@@ -304,6 +318,8 @@ func (*AgentMessage_TrainStatus) isAgentMessage_Msg() {}
func (*AgentMessage_TrainArtifact) isAgentMessage_Msg() {}
func (*AgentMessage_SlurmStatus) isAgentMessage_Msg() {}
// KVEvents is a change to what one vLLM replica's prefix cache holds, read
// by the agent from the engine's KV cache events and sent on for the router's
// precise index (plan §7.5 tier 2).
@@ -426,6 +442,7 @@ type ControlMessage struct {
// *ControlMessage_AgentSettings
// *ControlMessage_TrainStart
// *ControlMessage_TrainCancel
// *ControlMessage_Slurm
Msg isControlMessage_Msg `protobuf_oneof:"msg"`
// trace is the W3C trace context (traceparent, and tracestate when there
// is one) of the control plane's span that sent this command, so what the
@@ -618,6 +635,15 @@ func (x *ControlMessage) GetTrainCancel() *TrainCancel {
return nil
}
func (x *ControlMessage) GetSlurm() *Slurm {
if x != nil {
if x, ok := x.Msg.(*ControlMessage_Slurm); ok {
return x.Slurm
}
}
return nil
}
func (x *ControlMessage) GetTrace() map[string]string {
if x != nil {
return x.Trace
@@ -693,6 +719,10 @@ type ControlMessage_TrainCancel struct {
TrainCancel *TrainCancel `protobuf:"bytes,17,opt,name=train_cancel,json=trainCancel,proto3,oneof"`
}
type ControlMessage_Slurm struct {
Slurm *Slurm `protobuf:"bytes,18,opt,name=slurm,proto3,oneof"`
}
func (*ControlMessage_Start) isControlMessage_Msg() {}
func (*ControlMessage_Stop) isControlMessage_Msg() {}
@@ -725,6 +755,8 @@ func (*ControlMessage_TrainStart) isControlMessage_Msg() {}
func (*ControlMessage_TrainCancel) isControlMessage_Msg() {}
func (*ControlMessage_Slurm) isControlMessage_Msg() {}
// RestartAgent has the agent start again, as llmgw restart asks of every
// node: its engines keep serving and are taken over again.
type RestartAgent struct {
@@ -5251,11 +5283,236 @@ func (x *TrainArtifact) GetLast() bool {
return false
}
// Slurm is what this node is in the Slurm cluster run beside llmgw
// (BACKLOG P): its controller, one of its nodes, both or neither. Sent when
// an agent registers and whenever it changes; the agent installs what is
// missing and keeps the daemons running as it says.
type Slurm struct {
state protoimpl.MessageState `protogen:"open.v1"`
// controller runs slurmctld here, on files.
Controller bool `protobuf:"varint,1,opt,name=controller,proto3" json:"controller,omitempty"`
// node runs slurmd here, configless, fetching its files from conf_server.
Node bool `protobuf:"varint,2,opt,name=node,proto3" json:"node,omitempty"`
// name is this node's NodeName in slurm.conf.
Name string `protobuf:"bytes,3,opt,name=name,proto3" json:"name,omitempty"`
// conf_server is slurmctld's host:port.
ConfServer string `protobuf:"bytes,4,opt,name=conf_server,json=confServer,proto3" json:"conf_server,omitempty"`
// munge_key is the key every machine of the cluster shares.
MungeKey []byte `protobuf:"bytes,5,opt,name=munge_key,json=mungeKey,proto3" json:"munge_key,omitempty"`
// files are the controller's, by name in /etc/slurm; empty for a node
// that is not the controller.
Files map[string]string `protobuf:"bytes,6,rep,name=files,proto3" json:"files,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
// lent says Slurm may run jobs here now: resume the node if llmgw drained
// it. Otherwise it is drained, its jobs left to finish.
Lent bool `protobuf:"varint,7,opt,name=lent,proto3" json:"lent,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *Slurm) Reset() {
*x = Slurm{}
mi := &file_llmgw_v1_agent_proto_msgTypes[48]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *Slurm) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*Slurm) ProtoMessage() {}
func (x *Slurm) ProtoReflect() protoreflect.Message {
mi := &file_llmgw_v1_agent_proto_msgTypes[48]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use Slurm.ProtoReflect.Descriptor instead.
func (*Slurm) Descriptor() ([]byte, []int) {
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{48}
}
func (x *Slurm) GetController() bool {
if x != nil {
return x.Controller
}
return false
}
func (x *Slurm) GetNode() bool {
if x != nil {
return x.Node
}
return false
}
func (x *Slurm) GetName() string {
if x != nil {
return x.Name
}
return ""
}
func (x *Slurm) GetConfServer() string {
if x != nil {
return x.ConfServer
}
return ""
}
func (x *Slurm) GetMungeKey() []byte {
if x != nil {
return x.MungeKey
}
return nil
}
func (x *Slurm) GetFiles() map[string]string {
if x != nil {
return x.Files
}
return nil
}
func (x *Slurm) GetLent() bool {
if x != nil {
return x.Lent
}
return false
}
// SlurmStatus is what this node's Slurm daemons are doing, sent when it
// changes and every few minutes while the node has a part in Slurm.
type SlurmStatus struct {
state protoimpl.MessageState `protogen:"open.v1"`
// version is Slurm's, as "23.11.4"; empty when it is not installed.
Version string `protobuf:"bytes,1,opt,name=version,proto3" json:"version,omitempty"`
// hardware is `slurmd -C` on a node, which slurm.conf must match.
Hardware string `protobuf:"bytes,2,opt,name=hardware,proto3" json:"hardware,omitempty"`
SlurmdRunning bool `protobuf:"varint,3,opt,name=slurmd_running,json=slurmdRunning,proto3" json:"slurmd_running,omitempty"`
SlurmctldRunning bool `protobuf:"varint,4,opt,name=slurmctld_running,json=slurmctldRunning,proto3" json:"slurmctld_running,omitempty"`
// state and reason are the controller's for this node, from scontrol.
State string `protobuf:"bytes,5,opt,name=state,proto3" json:"state,omitempty"`
Reason string `protobuf:"bytes,6,opt,name=reason,proto3" json:"reason,omitempty"`
// jobs is how many job steps run on this node now, counted from their
// slurmstepd so a controller that is down does not hide them; -1 when the
// agent cannot tell.
Jobs int32 `protobuf:"varint,7,opt,name=jobs,proto3" json:"jobs,omitempty"`
// error is the last thing that went wrong keeping Slurm, empty when
// nothing did.
Error string `protobuf:"bytes,8,opt,name=error,proto3" json:"error,omitempty"`
// files_sha256 is of the controller's files as written, so the leader
// knows they arrived.
FilesSha256 string `protobuf:"bytes,9,opt,name=files_sha256,json=filesSha256,proto3" json:"files_sha256,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *SlurmStatus) Reset() {
*x = SlurmStatus{}
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *SlurmStatus) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*SlurmStatus) ProtoMessage() {}
func (x *SlurmStatus) ProtoReflect() protoreflect.Message {
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use SlurmStatus.ProtoReflect.Descriptor instead.
func (*SlurmStatus) Descriptor() ([]byte, []int) {
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{49}
}
func (x *SlurmStatus) GetVersion() string {
if x != nil {
return x.Version
}
return ""
}
func (x *SlurmStatus) GetHardware() string {
if x != nil {
return x.Hardware
}
return ""
}
func (x *SlurmStatus) GetSlurmdRunning() bool {
if x != nil {
return x.SlurmdRunning
}
return false
}
func (x *SlurmStatus) GetSlurmctldRunning() bool {
if x != nil {
return x.SlurmctldRunning
}
return false
}
func (x *SlurmStatus) GetState() string {
if x != nil {
return x.State
}
return ""
}
func (x *SlurmStatus) GetReason() string {
if x != nil {
return x.Reason
}
return ""
}
func (x *SlurmStatus) GetJobs() int32 {
if x != nil {
return x.Jobs
}
return 0
}
func (x *SlurmStatus) GetError() string {
if x != nil {
return x.Error
}
return ""
}
func (x *SlurmStatus) GetFilesSha256() string {
if x != nil {
return x.FilesSha256
}
return ""
}
var File_llmgw_v1_agent_proto protoreflect.FileDescriptor
const file_llmgw_v1_agent_proto_rawDesc = "" +
"\n" +
"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xb7\x04\n" +
"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xf3\x04\n" +
"\fAgentMessage\x120\n" +
"\bregister\x18\x01 \x01(\v2\x12.llmgw.v1.RegisterH\x00R\bregister\x123\n" +
"\theartbeat\x18\x02 \x01(\v2\x13.llmgw.v1.HeartbeatH\x00R\theartbeat\x12-\n" +
@@ -5269,7 +5526,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\tkv_events\x18\b \x01(\v2\x12.llmgw.v1.KVEventsH\x00R\bkvEvents\x12:\n" +
"\ftrain_status\x18\t \x01(\v2\x15.llmgw.v1.TrainStatusH\x00R\vtrainStatus\x12@\n" +
"\x0etrain_artifact\x18\n" +
" \x01(\v2\x17.llmgw.v1.TrainArtifactH\x00R\rtrainArtifactB\x05\n" +
" \x01(\v2\x17.llmgw.v1.TrainArtifactH\x00R\rtrainArtifact\x12:\n" +
"\fslurm_status\x18\v \x01(\v2\x15.llmgw.v1.SlurmStatusH\x00R\vslurmStatusB\x05\n" +
"\x03msg\"\xb5\x01\n" +
"\bKVEvents\x12\x1d\n" +
"\n" +
@@ -5280,7 +5538,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"clearFirst\x12\x16\n" +
"\x06stored\x18\x04 \x03(\x06R\x06stored\x12\x18\n" +
"\aremoved\x18\x05 \x03(\x06R\aremoved\x12\x18\n" +
"\aproblem\x18\x06 \x01(\tR\aproblem\"\xf8\a\n" +
"\aproblem\x18\x06 \x01(\tR\aproblem\"\xa1\b\n" +
"\x0eControlMessage\x12,\n" +
"\x05start\x18\x01 \x01(\v2\x14.llmgw.v1.StartShardH\x00R\x05start\x12)\n" +
"\x04stop\x18\x02 \x01(\v2\x13.llmgw.v1.StopShardH\x00R\x04stop\x12,\n" +
@@ -5301,7 +5559,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\x0eagent_settings\x18\x0f \x01(\v2\x17.llmgw.v1.AgentSettingsH\x00R\ragentSettings\x127\n" +
"\vtrain_start\x18\x10 \x01(\v2\x14.llmgw.v1.TrainStartH\x00R\n" +
"trainStart\x12:\n" +
"\ftrain_cancel\x18\x11 \x01(\v2\x15.llmgw.v1.TrainCancelH\x00R\vtrainCancel\x129\n" +
"\ftrain_cancel\x18\x11 \x01(\v2\x15.llmgw.v1.TrainCancelH\x00R\vtrainCancel\x12'\n" +
"\x05slurm\x18\x12 \x01(\v2\x0f.llmgw.v1.SlurmH\x00R\x05slurm\x129\n" +
"\x05trace\x18\f \x03(\v2#.llmgw.v1.ControlMessage.TraceEntryR\x05trace\x1a8\n" +
"\n" +
"TraceEntry\x12\x10\n" +
@@ -5754,7 +6013,32 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\x04kind\x18\x02 \x01(\tR\x04kind\x12\x10\n" +
"\x03seq\x18\x03 \x01(\x05R\x03seq\x12\x12\n" +
"\x04data\x18\x04 \x01(\fR\x04data\x12\x12\n" +
"\x04last\x18\x05 \x01(\bR\x04last2O\n" +
"\x04last\x18\x05 \x01(\bR\x04last\"\x8d\x02\n" +
"\x05Slurm\x12\x1e\n" +
"\n" +
"controller\x18\x01 \x01(\bR\n" +
"controller\x12\x12\n" +
"\x04node\x18\x02 \x01(\bR\x04node\x12\x12\n" +
"\x04name\x18\x03 \x01(\tR\x04name\x12\x1f\n" +
"\vconf_server\x18\x04 \x01(\tR\n" +
"confServer\x12\x1b\n" +
"\tmunge_key\x18\x05 \x01(\fR\bmungeKey\x120\n" +
"\x05files\x18\x06 \x03(\v2\x1a.llmgw.v1.Slurm.FilesEntryR\x05files\x12\x12\n" +
"\x04lent\x18\a \x01(\bR\x04lent\x1a8\n" +
"\n" +
"FilesEntry\x12\x10\n" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\x92\x02\n" +
"\vSlurmStatus\x12\x18\n" +
"\aversion\x18\x01 \x01(\tR\aversion\x12\x1a\n" +
"\bhardware\x18\x02 \x01(\tR\bhardware\x12%\n" +
"\x0eslurmd_running\x18\x03 \x01(\bR\rslurmdRunning\x12+\n" +
"\x11slurmctld_running\x18\x04 \x01(\bR\x10slurmctldRunning\x12\x14\n" +
"\x05state\x18\x05 \x01(\tR\x05state\x12\x16\n" +
"\x06reason\x18\x06 \x01(\tR\x06reason\x12\x12\n" +
"\x04jobs\x18\a \x01(\x05R\x04jobs\x12\x14\n" +
"\x05error\x18\b \x01(\tR\x05error\x12!\n" +
"\ffiles_sha256\x18\t \x01(\tR\vfilesSha2562O\n" +
"\fAgentService\x12?\n" +
"\aConnect\x12\x16.llmgw.v1.AgentMessage\x1a\x18.llmgw.v1.ControlMessage(\x010\x01B\x93\x01\n" +
"\fcom.llmgw.v1B\n" +
@@ -5773,7 +6057,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
}
var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 56)
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 59)
var file_llmgw_v1_agent_proto_goTypes = []any{
(ShardStatus_State)(0), // 0: llmgw.v1.ShardStatus.State
(*AgentMessage)(nil), // 1: llmgw.v1.AgentMessage
@@ -5824,14 +6108,17 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
(*TrainCancel)(nil), // 46: llmgw.v1.TrainCancel
(*TrainStatus)(nil), // 47: llmgw.v1.TrainStatus
(*TrainArtifact)(nil), // 48: llmgw.v1.TrainArtifact
nil, // 49: llmgw.v1.ControlMessage.TraceEntry
nil, // 50: llmgw.v1.Register.LabelsEntry
nil, // 51: llmgw.v1.Register.EngineVersionsEntry
nil, // 52: llmgw.v1.GPU.NicPathsEntry
nil, // 53: llmgw.v1.IBPort.CountersEntry
nil, // 54: llmgw.v1.Heartbeat.EngineVersionsEntry
nil, // 55: llmgw.v1.EngineWant.OfflineSha256Entry
nil, // 56: llmgw.v1.EngineProfile.EnvEntry
(*Slurm)(nil), // 49: llmgw.v1.Slurm
(*SlurmStatus)(nil), // 50: llmgw.v1.SlurmStatus
nil, // 51: llmgw.v1.ControlMessage.TraceEntry
nil, // 52: llmgw.v1.Register.LabelsEntry
nil, // 53: llmgw.v1.Register.EngineVersionsEntry
nil, // 54: llmgw.v1.GPU.NicPathsEntry
nil, // 55: llmgw.v1.IBPort.CountersEntry
nil, // 56: llmgw.v1.Heartbeat.EngineVersionsEntry
nil, // 57: llmgw.v1.EngineWant.OfflineSha256Entry
nil, // 58: llmgw.v1.EngineProfile.EnvEntry
nil, // 59: llmgw.v1.Slurm.FilesEntry
}
var file_llmgw_v1_agent_proto_depIdxs = []int32{
8, // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
@@ -5844,64 +6131,67 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
2, // 7: llmgw.v1.AgentMessage.kv_events:type_name -> llmgw.v1.KVEvents
47, // 8: llmgw.v1.AgentMessage.train_status:type_name -> llmgw.v1.TrainStatus
48, // 9: llmgw.v1.AgentMessage.train_artifact:type_name -> llmgw.v1.TrainArtifact
30, // 10: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
33, // 11: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
34, // 12: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
35, // 13: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
32, // 14: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
24, // 15: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
36, // 16: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
38, // 17: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
40, // 18: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
42, // 19: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
7, // 20: llmgw.v1.ControlMessage.control_planes:type_name -> llmgw.v1.ControlPlanes
5, // 21: llmgw.v1.ControlMessage.firewall:type_name -> llmgw.v1.Firewall
4, // 22: llmgw.v1.ControlMessage.restart_agent:type_name -> llmgw.v1.RestartAgent
6, // 23: llmgw.v1.ControlMessage.agent_settings:type_name -> llmgw.v1.AgentSettings
45, // 24: llmgw.v1.ControlMessage.train_start:type_name -> llmgw.v1.TrainStart
46, // 25: llmgw.v1.ControlMessage.train_cancel:type_name -> llmgw.v1.TrainCancel
49, // 26: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
50, // 27: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
12, // 28: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
14, // 29: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
51, // 30: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
11, // 31: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
15, // 32: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
10, // 33: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
9, // 34: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
21, // 35: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
0, // 36: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
27, // 37: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
52, // 38: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
13, // 39: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
18, // 40: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
17, // 41: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
53, // 42: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
20, // 43: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
10, // 44: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
15, // 45: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
54, // 46: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
26, // 47: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
16, // 48: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
19, // 49: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
14, // 50: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
22, // 51: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
21, // 52: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
25, // 53: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
55, // 54: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
0, // 55: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
29, // 56: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
27, // 57: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
31, // 58: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
56, // 59: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
44, // 60: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
1, // 61: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
3, // 62: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
62, // [62:63] is the sub-list for method output_type
61, // [61:62] is the sub-list for method input_type
61, // [61:61] is the sub-list for extension type_name
61, // [61:61] is the sub-list for extension extendee
0, // [0:61] is the sub-list for field type_name
50, // 10: llmgw.v1.AgentMessage.slurm_status:type_name -> llmgw.v1.SlurmStatus
30, // 11: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
33, // 12: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
34, // 13: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
35, // 14: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
32, // 15: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
24, // 16: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
36, // 17: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
38, // 18: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
40, // 19: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
42, // 20: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
7, // 21: llmgw.v1.ControlMessage.control_planes:type_name -> llmgw.v1.ControlPlanes
5, // 22: llmgw.v1.ControlMessage.firewall:type_name -> llmgw.v1.Firewall
4, // 23: llmgw.v1.ControlMessage.restart_agent:type_name -> llmgw.v1.RestartAgent
6, // 24: llmgw.v1.ControlMessage.agent_settings:type_name -> llmgw.v1.AgentSettings
45, // 25: llmgw.v1.ControlMessage.train_start:type_name -> llmgw.v1.TrainStart
46, // 26: llmgw.v1.ControlMessage.train_cancel:type_name -> llmgw.v1.TrainCancel
49, // 27: llmgw.v1.ControlMessage.slurm:type_name -> llmgw.v1.Slurm
51, // 28: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
52, // 29: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
12, // 30: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
14, // 31: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
53, // 32: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
11, // 33: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
15, // 34: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
10, // 35: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
9, // 36: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
21, // 37: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
0, // 38: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
27, // 39: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
54, // 40: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
13, // 41: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
18, // 42: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
17, // 43: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
55, // 44: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
20, // 45: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
10, // 46: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
15, // 47: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
56, // 48: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
26, // 49: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
16, // 50: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
19, // 51: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
14, // 52: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
22, // 53: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
21, // 54: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
25, // 55: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
57, // 56: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
0, // 57: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
29, // 58: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
27, // 59: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
31, // 60: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
58, // 61: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
44, // 62: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
59, // 63: llmgw.v1.Slurm.files:type_name -> llmgw.v1.Slurm.FilesEntry
1, // 64: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
3, // 65: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
65, // [65:66] is the sub-list for method output_type
64, // [64:65] is the sub-list for method input_type
64, // [64:64] is the sub-list for extension type_name
64, // [64:64] is the sub-list for extension extendee
0, // [0:64] is the sub-list for field type_name
}
func init() { file_llmgw_v1_agent_proto_init() }
@@ -5920,6 +6210,7 @@ func file_llmgw_v1_agent_proto_init() {
(*AgentMessage_KvEvents)(nil),
(*AgentMessage_TrainStatus)(nil),
(*AgentMessage_TrainArtifact)(nil),
(*AgentMessage_SlurmStatus)(nil),
}
file_llmgw_v1_agent_proto_msgTypes[2].OneofWrappers = []any{
(*ControlMessage_Start)(nil),
@@ -5938,6 +6229,7 @@ func file_llmgw_v1_agent_proto_init() {
(*ControlMessage_AgentSettings)(nil),
(*ControlMessage_TrainStart)(nil),
(*ControlMessage_TrainCancel)(nil),
(*ControlMessage_Slurm)(nil),
}
type x struct{}
out := protoimpl.TypeBuilder{
@@ -5945,7 +6237,7 @@ func file_llmgw_v1_agent_proto_init() {
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
NumEnums: 1,
NumMessages: 56,
NumMessages: 59,
NumExtensions: 0,
NumServices: 1,
},
+49
View File
@@ -30,6 +30,7 @@ message AgentMessage {
KVEvents kv_events = 8;
TrainStatus train_status = 9;
TrainArtifact train_artifact = 10;
SlurmStatus slurm_status = 11;
}
}
@@ -78,6 +79,7 @@ message ControlMessage {
AgentSettings agent_settings = 15;
TrainStart train_start = 16;
TrainCancel train_cancel = 17;
Slurm slurm = 18;
}
// trace is the W3C trace context (traceparent, and tracestate when there
// is one) of the control plane's span that sent this command, so what the
@@ -986,3 +988,50 @@ message TrainArtifact {
bytes data = 4;
bool last = 5;
}
// Slurm is what this node is in the Slurm cluster run beside llmgw
// (BACKLOG P): its controller, one of its nodes, both or neither. Sent when
// an agent registers and whenever it changes; the agent installs what is
// missing and keeps the daemons running as it says.
message Slurm {
// controller runs slurmctld here, on files.
bool controller = 1;
// node runs slurmd here, configless, fetching its files from conf_server.
bool node = 2;
// name is this node's NodeName in slurm.conf.
string name = 3;
// conf_server is slurmctld's host:port.
string conf_server = 4;
// munge_key is the key every machine of the cluster shares.
bytes munge_key = 5;
// files are the controller's, by name in /etc/slurm; empty for a node
// that is not the controller.
map<string, string> files = 6;
// lent says Slurm may run jobs here now: resume the node if llmgw drained
// it. Otherwise it is drained, its jobs left to finish.
bool lent = 7;
}
// SlurmStatus is what this node's Slurm daemons are doing, sent when it
// changes and every few minutes while the node has a part in Slurm.
message SlurmStatus {
// version is Slurm's, as "23.11.4"; empty when it is not installed.
string version = 1;
// hardware is `slurmd -C` on a node, which slurm.conf must match.
string hardware = 2;
bool slurmd_running = 3;
bool slurmctld_running = 4;
// state and reason are the controller's for this node, from scontrol.
string state = 5;
string reason = 6;
// jobs is how many job steps run on this node now, counted from their
// slurmstepd so a controller that is down does not hide them; -1 when the
// agent cannot tell.
int32 jobs = 7;
// error is the last thing that went wrong keeping Slurm, empty when
// nothing did.
string error = 8;
// files_sha256 is of the controller's files as written, so the leader
// knows they arrived.
string files_sha256 = 9;
}
+136 -1
View File
@@ -116,6 +116,9 @@
{
"name": "settings"
},
{
"name": "slurm"
},
{
"name": "summary"
},
@@ -5078,6 +5081,34 @@
}
}
},
"/api/v1/slurm": {
"get": {
"operationId": "getSlurm",
"summary": "The Slurm cluster beside llmgw, and where each of its nodes is",
"description": "From the cluster's settings (slurm.*) and what each node's agent last said. A node's phase is llmgw (drained in Slurm, no job left), lending (its replicas moving off), lent, reclaiming (drained in Slurm, jobs still running) or unknown (its agent has not said).",
"tags": [
"slurm"
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/slurmsvc.Status"
}
}
}
},
"401": {
"$ref": "#/components/responses/Unauthorized"
},
"default": {
"$ref": "#/components/responses/Error"
}
}
}
},
"/api/v1/summary": {
"get": {
"operationId": "getSummary",
@@ -6234,7 +6265,7 @@
},
"CollectResult": {
"type": "object",
"description": "CollectResult is what was deleted on one node, or why nothing was.",
"description": "CollectResult is what was deleted on one node, or why nothing was: what the node no longer lists once asked, not what it was asked to delete.",
"properties": {
"node": {
"type": "string"
@@ -6246,6 +6277,14 @@
"type": "integer",
"format": "int64"
},
"left": {
"type": "integer",
"description": "Left and LeftBytes are what the node still lists: refused, or not gone by the time the sweep stopped waiting. Its agent's log says why."
},
"left_bytes": {
"type": "integer",
"format": "int64"
},
"error": {
"type": "string"
}
@@ -11737,6 +11776,10 @@
"context": {
"type": "integer"
},
"trained": {
"type": "integer",
"description": "Trained is the context the model was trained for, the most a request may use; 0 when neither its header nor its config says."
},
"kv_bytes_per_token": {
"type": "integer",
"format": "int64",
@@ -12166,6 +12209,98 @@
}
}
},
"slurmsvc.NodeStatus": {
"type": "object",
"description": "NodeStatus is one node as the service sees it.",
"properties": {
"node": {
"type": "string"
},
"phase": {
"type": "string"
},
"replicas": {
"type": "integer"
},
"slurm": {
"$ref": "#/components/schemas/slurmsvc.Reported"
},
"connected": {
"type": "boolean"
}
}
},
"slurmsvc.Reported": {
"type": "object",
"description": "Reported is the agent's SlurmStatus, for the admin API.",
"properties": {
"version": {
"type": "string"
},
"hardware": {
"type": "string"
},
"slurmd_running": {
"type": "boolean"
},
"slurmctld_running": {
"type": "boolean"
},
"state": {
"type": "string"
},
"reason": {
"type": "string"
},
"jobs": {
"type": "integer"
},
"error": {
"type": "string"
},
"files_sha256": {
"type": "string"
},
"at": {
"type": "string",
"format": "date-time"
}
}
},
"slurmsvc.Status": {
"type": "object",
"description": "Status is the whole Slurm cluster, for `ctl slurm status`.",
"properties": {
"enabled": {
"type": "boolean"
},
"cluster": {
"type": "string"
},
"controller": {
"type": "string"
},
"partition": {
"type": "string"
},
"controller_status": {
"$ref": "#/components/schemas/slurmsvc.Reported"
},
"nodes": {
"type": "array",
"items": {
"$ref": "#/components/schemas/slurmsvc.NodeStatus"
}
},
"waiting": {
"type": "array",
"items": {
"type": "string"
},
"description": "Waiting are nodes left out of slurm.conf until they report their hardware."
}
}
},
"store.AuditEvent": {
"type": "object",
"description": "AuditEvent is one change: who made it, through what, to what, and how it ended. Detail never holds a secret; the writer takes them out.",
+2 -1
View File
@@ -220,7 +220,8 @@ and change it with the same commands as `llmgw users`, `llmgw keys` and
dashboard, and what they belong to. **A** creates an account, already
approved. On a row: approve one waiting, suspend or reinstate, give or take
the cluster dashboard, add to or take out of an organization, team or
project, their limits, a new password (their sessions close), delete, and
project (on an organization as a member, its manager, who sees its page, or
its admin, who also changes it), their limits, a new password (their sessions close), delete, and
**Forget them…** for a request to be forgotten (`llmgw users forget`).
**API keys** — every key, its project, owner, state and last use. **A**
+17 -2
View File
@@ -166,13 +166,28 @@ is a flag too, and the plan shows the command with them: `--starter qwen3-8b`
`--judge-fail open`, `--judge-categories`, `--judge-replicas`,
`--judge-budget`, `--language tr`, `--signup-open`, `--default-rpm`,
`--default-tpm`, `--max-key-ttl`, `--media` (pictures and 3D: ComfyUI on the
nodes, and `presets.auto`), `--evict-after-idle` and `--min-lifetime`. They
nodes, and `presets.auto`), `--slurm` (see below), `--evict-after-idle` and `--min-lifetime`. They
go into `server.yaml` as init writes it, and become the cluster's first
settings when it starts; a `server.yaml` already there is never overwritten,
so run again they change nothing — the settings page does. An NVIDIA card with no working
driver is reported rather than silently ignored; install the driver and run
init again to serve on it.
**Slurm, for training and batch jobs.** `--slurm`, or the Slurm part on the
choices screen, runs a Slurm cluster beside llmgw: slurmctld on this
machine's node, slurmd on every node, those that join later too
(`slurm.all_nodes`). The agents install munge and Slurm (22.05 or newer)
from the machines' own package sources. No node is Slurm's at first: each is
drained in Slurm and serves models until `llmgw ctl slurm lend NODE` gives it
to Slurm — its models move off first — and `llmgw ctl slurm reclaim NODE`
takes it back once its jobs end. Jobs are submitted on this machine with
`sbatch` and `srun`; `llmgw ctl slurm status` shows where each node is. It
needs this machine to be a node, and its hostname to be a name Slurm takes.
On a cluster installed without it, `llmgw ctl slurm setup --controller NODE
--all-nodes` does the same later. Slurm's ports (6817–6818, and srun's 60001–63000) answer the cluster's
networks only, as llmgw's own do; a firewall of the site's own between the
nodes must let them through.
**Secure by default.** What init writes is the hardened configuration, not a
demonstration one:
@@ -3145,7 +3160,7 @@ loaded at boot by `llmgw-firewall.service`):
| Machine | Ports, for the cluster only |
| --- | --- |
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
| node | engine ports (`control.port_range`, 20000–20999), 9100 |
| node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6818 (slurmctld, slurmd) and 60001–63000 (srun's `SrunPortRange`) |
| store member | 5432, 8008, 2379, 2380, 9100 |
| mirror | 8081, 8082, 9100 |
Binary file not shown.
+32 -11
View File
@@ -192,22 +192,41 @@ auth:
A personal key has no project, so it keeps no conversations or agent notes;
those are kept per project.
### Organization managers
### Organization managers and admins
A member of an organization can be made its manager:
A member of an organization can be made its manager, who sees it, or its
admin, who also changes it:
```
llmgw users join boss@acme.com --org Acme --role manager
llmgw users join lead@acme.com --org Acme --role admin
```
or with the star on their organization membership on `/admin/users`. A manager
gets an **Organization** link and a page at `/org`: usage over time by team,
or with the role chosen on their organization membership on `/admin/users`.
Both get an **Organization** link and a page at `/org`: usage over time by team,
project or person, the teams and projects with their limits and spend, and every
member. Only what was spent in that organization's projects is on it; what its
people do elsewhere is not. The page is read-only — a manager is not an
administrator of the cluster, and limits and memberships stay with those who
are. An administrator following `/org` lands on the same organization's panel
on the people page instead, where all of that can be changed.
people do elsewhere is not.
A manager's page is read-only. An organization's admin also changes the
organization from it:
- adds and renames its teams and projects, and removes an empty one;
- adds a person by their sign-in address to the organization, a team or a
project, and takes them out (their keys there are revoked with it);
- makes a member of the organization a manager or an admin, or neither;
- revokes a key spent in its projects.
What reaches past the organization stays with the cluster's administrators:
its limits and budgets, approving or suspending an account (someone not yet
approved cannot be added), the organization's own name, and anything else on
the cluster. An admin cannot take away their own role. A group from single
sign-on or Active Directory that puts someone in the organization does not, by
saying so again at the next sign-in, take a manager's or admin's role away.
An administrator of the cluster following `/org` lands on the same
organization's panel on the people page instead, where all of that can be
changed.
`llmgw keys init` still works for provisioning scripts and is idempotent, but it
will no longer move a team or project that already exists under another parent;
@@ -517,8 +536,10 @@ included: `llmgw ctl audit`, or **Audit** in the console.
suspends accounts from it; see [install.md](install.md#single-sign-on).
Without them accounts are local, with local passwords.
- **Staff roles are coarse.** `viewer`, `operator`, `billing` and `admin`
(`llmgw users role`) decide what someone may see and do under /admin; an
organization's manager can view it but not change it. Every change is in
the audit log (`llmgw ctl audit`).
(`llmgw users role`) decide what someone may see and do under /admin, and a
page draws only the forms that person may send: a viewer sees no forms. An
organization's manager can view it; its admin also changes its teams,
projects, people and keys. Every change is in the audit log (`llmgw ctl
audit`).
- **Rate limits are per gateway instance.** With two instances each counts its own,
so the effective limit doubles. v1 is single-instance by design.
+68
View File
@@ -376,6 +376,7 @@ rather than repeated.
| `mail` | `mail.Config` | [Server](#server) |
| `serving` | `ServingSettings` | [Server](#server) |
| `firewall` | `FirewallSettings` | [Server](#server) |
| `slurm` | `SlurmSettings` | [Server](#server) |
| `presets` | `PresetSettings` | [Server](#server) |
| `on_demand` | `OnDemandSettings` | [Server](#server) |
| `agents` | `AgentSettings` | [Server](#server) |
@@ -401,6 +402,13 @@ rather than repeated.
| `timeout` | `time.Duration` | [Service](#service) |
| `stopped` | `bool` | [ServingSettings](#servingsettings) |
| `bin` | `string` | [SimConfig](#simconfig) |
| `enabled` | `bool` | [SlurmSettings](#slurmsettings) |
| `cluster_name` | `string` | [SlurmSettings](#slurmsettings) |
| `controller` | `string` | [SlurmSettings](#slurmsettings) |
| `nodes` | `[]string` | [SlurmSettings](#slurmsettings) |
| `all_nodes` | `bool` | [SlurmSettings](#slurmsettings) |
| `lent` | `[]string` | [SlurmSettings](#slurmsettings) |
| `partition` | `string` | [SlurmSettings](#slurmsettings) |
| `dir` | `string` | [StrataConfig](#strataconfig) |
| `python` | `string` | [StrataConfig](#strataconfig) |
| `engine` | `string` | [StrataConfig](#strataconfig) |
@@ -3279,6 +3287,15 @@ Firewall turns llmgw's firewall on every machine on or off.
Takes a [FirewallSettings](#firewallsettings) block.
### `slurm`
`SlurmSettings`
Slurm is a Slurm cluster run beside llmgw on some of its nodes, and
which of them it may use now.
Takes a [SlurmSettings](#slurmsettings) block.
### `presets`
`PresetSettings`
@@ -3480,6 +3497,57 @@ entries ask. In force on every control plane within seconds.
Bin is the llmgw binary itself; the engine is one of its subcommands.
Empty means whatever started this agent, which is right almost always.
## SlurmSettings
### `enabled`
`bool`
Enabled runs Slurm on the nodes below. False stops llmgw keeping it;
what was installed is left installed, and every node is llmgw's.
### `cluster_name`
`string`
ClusterName is Slurm's ClusterName. Empty is "llmgw".
### `controller`
`string`
Controller is the llmgw node whose agent runs slurmctld. It needs no
GPU, and need not be one of Nodes.
### `nodes`
`[]string`
Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained
in Slurm, until it is also in Lent.
### `all_nodes`
`bool`
AllNodes runs slurmd on every llmgw node, those that join later too,
besides any named in Nodes. Each is llmgw's until it is lent, as any.
### `lent`
`[]string`
Lent are the nodes of Nodes Slurm may run jobs on now. A node put here
is drained of llmgw's replicas first, as `ctl drain` does, and resumed
in Slurm once none is left; a node taken out is drained in Slurm and
given back to llmgw when its last job ends.
### `partition`
`string`
Partition is the one partition every node is in. Empty is "gpu".
## StrataConfig
### `dir`
+46 -6
View File
@@ -35,6 +35,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/slurmnode"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
@@ -97,7 +98,10 @@ type Agent struct {
// (nil: this machine's shell).
fwMu sync.Mutex
fwRunner firewall.Runner
inv discovery.Inventory
// fwLast is the last firewall message, applied again when the node
// is given a part in Slurm. Under fwMu.
fwLast *llmgwv1.Firewall
inv discovery.Inventory
// shardReplica remembers which replica a shard belongs to, so a status
// report can name both. The control plane keys on the replica.
shardReplica map[string]string
@@ -148,6 +152,10 @@ type Agent struct {
// trainer runs LoRA training jobs, one at a time (BACKLOG F4).
trainer *train.Runner
// slurm keeps the node's part in Slurm (BACKLOG P); slurmOut is where
// its status goes, under mu.
slurm *slurmnode.Keeper
slurmOut chan<- *llmgwv1.AgentMessage
}
// connectHTTPClient is the HTTP client connect uses; an interface so tests can
@@ -206,6 +214,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
}
a.trainer = train.New(train.Env{EnginesDir: enginesDir, WorkDir: filepath.Join(cfg.Superv.WorkDir, "trainer"),
RunAs: runAs, Log: log})
a.slurm = slurmnode.New(slurmnode.Local{}, a.prov, slurmMarker, log)
return a, nil
}
@@ -232,6 +241,7 @@ func (a *Agent) Run(ctx context.Context) error {
}
go a.provisionLoop(ctx)
go a.ensureDiags(ctx)
go a.slurm.Run(ctx, a.sendSlurm)
attempt := 0
next := 0
@@ -530,6 +540,18 @@ func (a *Agent) scan(ctx context.Context) error {
return err
}
// forgetSent is a new stream: a control plane that knows nothing of where
// installs stand, nor which files are here beyond what registration says —
// and registration says what the startup scan found. So the next heartbeat
// tells it both again. Without it, files deleted after the agent started came
// back on a control plane that restarted, and stayed until the next change.
func (a *Agent) forgetSent() {
a.mu.Lock()
defer a.mu.Unlock()
a.provSent = ""
a.lastWeights = ""
}
// session runs one connection until it breaks.
func (a *Agent) session(ctx context.Context, server string) error {
client := llmgwv1connect.NewAgentServiceClient(a.http, server)
@@ -566,14 +588,12 @@ func (a *Agent) session(ctx context.Context, server string) error {
reg.EngineVersions = a.engines(parent)
reg.EnginesProbed = true
reg.InstallableEngines = slices.DeleteFunc(provision.Installable(), a.disabled)
reg.Features = []string{FeatureFabricTest}
reg.Features = []string{FeatureFabricTest, FeatureSlurm}
reg.RunningShards = a.runningShards()
a.mu.Lock()
h := a.prov.Host
// A new stream is a control plane that knows nothing of where installs
// stand, so the next heartbeat tells it again.
a.provSent = ""
a.mu.Unlock()
a.forgetSent()
reg.Host = &llmgwv1.Host{
OsId: h.OSID, OsVersion: h.OSVersion, OsCodename: h.OSCodename,
Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver,
@@ -618,6 +638,8 @@ func (a *Agent) session(ctx context.Context, server string) error {
go a.watchExits(ctx, out, unready(reg.RunningShards))
go a.forwardKV(ctx, out)
go a.reportCompute(ctx, out)
a.setSlurmOut(out)
defer a.setSlurmOut(nil)
for {
msg, err := stream.Receive()
@@ -884,7 +906,23 @@ func (a *Agent) removeWeights(paths []string) {
a.log.Warn("refused to remove a file llmgw did not fetch", "path", p)
continue
}
if err := os.Remove(abs); err != nil && !os.IsNotExist(err) {
// A set — every file of one version, fetched into a directory of its
// own — goes whole: os.Remove refused it as not empty, and nothing
// was freed of what a model of several files left. Lstat, so a link
// is taken away and never what it points at.
info, err := os.Lstat(abs)
switch {
case os.IsNotExist(err):
continue
case err != nil:
a.log.Warn("could not look at cached weights", "path", p, "err", err)
continue
case info.IsDir():
err = os.RemoveAll(abs)
default:
err = os.Remove(abs)
}
if err != nil {
a.log.Warn("could not remove cached weights", "path", p, "err", err)
continue
}
@@ -925,6 +963,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
go a.applySettings(m.AgentSettings)
case *llmgwv1.ControlMessage_TrainStart:
go a.trainJob(ctx, m.TrainStart, out)
case *llmgwv1.ControlMessage_Slurm:
a.slurmMessage(m.Slurm)
case *llmgwv1.ControlMessage_TrainCancel:
a.trainer.Cancel(m.TrainCancel.GetJobId())
case *llmgwv1.ControlMessage_RestartAgent:
+20
View File
@@ -925,3 +925,23 @@ func TestABandwidthTestIsRunOnTheAdapterAskedFor(t *testing.T) {
t.Errorf("reply %+v, ran %+v", r, got)
}
}
func TestANewStreamIsToldTheFilesAgain(t *testing.T) {
// Registration carries what the startup scan found; a file deleted since
// came back on a control plane that restarted, because the next
// heartbeat saw the list unchanged since it was last sent.
dir := t.TempDir()
h := start(t, func(c *Config) { c.ModelDirs = []string{dir} })
if err := os.WriteFile(filepath.Join(dir, "0123456789abcdef-m.gguf"), []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
h.ag.heartbeat()
if hb := h.ag.heartbeat().GetHeartbeat(); hb.GetWeightsChanged() {
t.Fatal("an unchanged list should not be sent again on the same stream")
}
h.ag.forgetSent()
hb := h.ag.heartbeat().GetHeartbeat()
if !hb.GetWeightsChanged() || len(hb.GetWeights()) != 1 {
t.Errorf("a new stream should be told the files: changed=%v %v", hb.GetWeightsChanged(), hb.GetWeights())
}
}
+30
View File
@@ -10,8 +10,16 @@ import (
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
// slurmPorts are Slurm's on a node that runs any of it: slurmctld's and
// slurmd's, and srun's range for the steps it starts.
var slurmPorts = []firewall.Port{
{From: slurm.CtldPort, To: slurm.SlurmdPort, What: "slurm: slurmctld and slurmd"},
{From: slurm.SrunPortsFrom, To: slurm.SrunPortsTo, What: "slurm: srun, for the steps it starts"},
}
// applyFirewall keeps llmgw's firewall on this node as the cluster's
// settings say: the engines' ports and the node exporter's for the networks
// the node reaches its control planes on, and those named, or no table at
@@ -19,6 +27,7 @@ import (
func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
a.fwMu.Lock()
defer a.fwMu.Unlock()
a.fwLast = m
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()
r := a.fwRunner
@@ -55,6 +64,11 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
if from > 0 && to >= from {
ports = append(ports, firewall.Engines(from, to))
}
// Once there, they stay: the ruleset keeps every port it held, so a
// node that leaves Slurm keeps them closed to everyone but the cluster.
if a.slurm != nil && a.slurm.HasPart() {
ports = append(ports, slurmPorts...)
}
changed, err := firewall.Install(ctx, r, firewall.Spec{Ports: ports, Allow: nets})
switch {
case err != nil:
@@ -64,6 +78,22 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
}
}
// slurmMessage is the leader's word on this node's part in Slurm. A node
// that gets one has the firewall applied again, with Slurm's ports.
func (a *Agent) slurmMessage(m *llmgwv1.Slurm) {
had := a.slurm.HasPart()
a.slurm.Apply(m)
if had || !a.slurm.HasPart() {
return
}
a.fwMu.Lock()
last := a.fwLast
a.fwMu.Unlock()
if last != nil && last.GetEnabled() {
go a.applyFirewall(last)
}
}
// controlPlaneIPs are the addresses of the control planes an agent dials.
func controlPlaneIPs(ctx context.Context, servers []string) []string {
var out []string
+27
View File
@@ -6,8 +6,10 @@ import (
"log/slog"
"strings"
"testing"
"time"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/slurmnode"
)
// fakeShell answers the firewall package's scripts and records what was
@@ -55,3 +57,28 @@ func TestANodeKeepsOrDropsItsFirewallAsTheClusterSays(t *testing.T) {
t.Error("turned off, the table stayed")
}
}
func TestANodeGivenAPartInSlurmLetsSlurmInFromTheClusterOnly(t *testing.T) {
sh := &fakeShell{}
a := &Agent{log: slog.New(slog.DiscardHandler), fwRunner: sh, serverList: []string{"https://10.0.0.140:7443"},
slurm: slurmnode.New(nil, nil, "", slog.New(slog.DiscardHandler))}
a.applyFirewall(&llmgwv1.Firewall{Enabled: true, EnginePortsFrom: 20000, EnginePortsTo: 20999})
if strings.Contains(sh.installed, "6817") {
t.Fatalf("Slurm's ports before the node has a part in it:\n%s", sh.installed)
}
a.slurmMessage(&llmgwv1.Slurm{Name: "node05", Node: true, ConfServer: "10.0.0.148:6817"})
deadline := time.Now().Add(5 * time.Second)
for {
a.fwMu.Lock()
got := sh.installed
a.fwMu.Unlock()
if strings.Contains(got, "6817-6818") && strings.Contains(got, "60001-63000") {
break
}
if time.Now().After(deadline) {
t.Fatalf("given a part in Slurm, the rules still lack its ports:\n%s", got)
}
time.Sleep(10 * time.Millisecond)
}
}
+7
View File
@@ -164,6 +164,13 @@ func (a *Agent) provisionLoop(ctx context.Context) {
// provisionStatus records a change in one engine's installation.
func (a *Agent) provisionStatus(st provision.Status) {
if st == (provision.Status{}) {
// A package install that was no engine's has ended.
a.mu.Lock()
delete(a.provStatus, "")
a.mu.Unlock()
return
}
a.mu.Lock()
prev, had := a.provStatus[st.Engine]
a.provStatus[st.Engine] = st
+7 -1
View File
@@ -497,7 +497,13 @@ func (in *Installer) packages(ctx context.Context, byFamily map[string][]string,
// byFamily is keyed by discovery.Host.Family: "deb" or "rpm".
func (in *Installer) InstallPackages(ctx context.Context, byFamily map[string][]string) error {
var st Status
return in.packages(ctx, byFamily, &st)
err := in.packages(ctx, byFamily, &st)
if st.State == Installing {
// Its progress was no engine's: an empty status takes it off the
// node's page again, done or failed (the caller says which).
in.report(Status{})
}
return err
}
// lockWait bounds how long a package command waits for another to finish.
@@ -0,0 +1,36 @@
package provision
import (
"context"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
)
// A package install that is no engine's says it is done, so its progress
// does not stay on the node's page (found on the PoC installing Slurm).
func TestInstallPackagesClearsItsProgress(t *testing.T) {
var got []Status
in := &Installer{Host: discovery.Host{OSID: "ubuntu"}, OnStatus: func(s Status) { got = append(got, s) },
Run: func(_ context.Context, _ []string, name string, _ ...string) ([]byte, error) {
if name == "sh" { // the query: not installed
return nil, context.Canceled
}
return nil, nil
}}
if err := in.InstallPackages(context.Background(), map[string][]string{"deb": {"munge"}}); err != nil {
t.Fatal(err)
}
if len(got) < 2 || got[0].State != Installing || got[len(got)-1] != (Status{}) {
t.Fatalf("statuses %+v", got)
}
got = nil
in.Run = func(context.Context, []string, string, ...string) ([]byte, error) { return nil, nil }
if err := in.InstallPackages(context.Background(), map[string][]string{"deb": {"munge"}}); err != nil {
t.Fatal(err)
}
if len(got) != 0 {
t.Fatalf("installed already, yet said %+v", got)
}
}
+32
View File
@@ -94,3 +94,35 @@ func TestWithNoCacheConfiguredNothingIsDeleted(t *testing.T) {
t.Errorf("an agent with no cache directory deleted something: %v", err)
}
}
// A model of several files is fetched into a directory of its own, and an
// unused one goes whole: it was refused as not empty, and nothing was freed.
func TestAnUnusedSetIsRemovedWhole(t *testing.T) {
dir := t.TempDir()
a := agentWithCache(t, dir)
set := weights.CachePathFor(dir, "https://mirror/coder/v1")
if err := os.MkdirAll(set, 0o755); err != nil {
t.Fatal(err)
}
write(t, filepath.Join(set, "model-00001-of-00002.gguf"))
write(t, filepath.Join(set, ".complete"))
// A link in the cache is taken away, not what it points at.
target := t.TempDir()
write(t, filepath.Join(target, "keep.gguf"))
link := weights.CachePathFor(dir, "https://mirror/linked/v1")
if err := os.Symlink(target, link); err != nil {
t.Fatal(err)
}
a.removeWeights([]string{set, link})
if _, err := os.Stat(set); !os.IsNotExist(err) {
t.Errorf("the set is still there: %v", err)
}
if _, err := os.Lstat(link); !os.IsNotExist(err) {
t.Errorf("the link is still there: %v", err)
}
if _, err := os.Stat(filepath.Join(target, "keep.gguf")); err != nil {
t.Errorf("what the link pointed at was deleted: %v", err)
}
}
+38
View File
@@ -0,0 +1,38 @@
package agent
import (
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
)
// FeatureSlurm is the agent keeping its node's part in Slurm (BACKLOG P).
const FeatureSlurm = "slurm"
// slurmMarker says llmgw started Slurm's daemons on this machine.
var slurmMarker = "/var/lib/llmgw-agent/slurm"
// setSlurmOut is where Slurm's status goes now: this session's queue, or
// nowhere between sessions.
func (a *Agent) setSlurmOut(out chan<- *llmgwv1.AgentMessage) {
a.mu.Lock()
a.slurmOut = out
a.mu.Unlock()
if out != nil {
a.slurm.Resend()
}
}
// sendSlurm queues a status without waiting: a full queue or no session
// drops it, and the next session is sent it again.
func (a *Agent) sendSlurm(st *llmgwv1.SlurmStatus) {
a.mu.Lock()
out := a.slurmOut
a.mu.Unlock()
if out == nil {
return
}
select {
case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_SlurmStatus{SlurmStatus: st}}:
default:
a.log.Warn("Slurm's status not sent: the queue to the control plane is full")
}
}
+612
View File
@@ -0,0 +1,612 @@
// Package slurmnode keeps this machine's part in the Slurm cluster run beside
// llmgw (BACKLOG P), as the leader says: slurmctld on the controller, slurmd
// on a node, munge on both, installed from the machine's own package sources.
//
// What the leader sends is wanted state, kept by Keeper's loop: sent again,
// or checked again a minute later, it changes nothing that is already so.
package slurmnode
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io/fs"
"log/slog"
"os"
"os/exec"
"os/user"
"path/filepath"
"slices"
"strconv"
"strings"
"sync"
"time"
"google.golang.org/protobuf/proto"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
// Host is what Keeper does to the machine; Local in the agent, a fake in
// tests.
type Host interface {
Run(ctx context.Context, env []string, name string, args ...string) ([]byte, error)
LookPath(name string) (string, error)
ReadFile(path string) ([]byte, error)
// WriteFile writes b at path with mode, owned by owner ("" for root),
// making its directory.
WriteFile(path string, b []byte, mode fs.FileMode, owner string) error
// MkdirAll makes path, owned by owner.
MkdirAll(path string, mode fs.FileMode, owner string) error
Remove(path string) error
}
// Installer installs packages by family, "deb" and "rpm".
type Installer interface {
InstallPackages(ctx context.Context, byFamily map[string][]string) error
}
// Reasons llmgw drains a node with; one drained for another reason, by a
// Slurm administrator, is never resumed by llmgw.
const (
ReasonServing = "llmgw: serving models"
reasonPrefix = "llmgw"
)
// DropIn is the unit drop-in that starts slurmd configless.
const DropIn = "/etc/systemd/system/slurmd.service.d/llmgw.conf"
// Keeper keeps the machine as the last Slurm message says.
type Keeper struct {
host Host
inst Installer
log *slog.Logger
marker string
mu sync.Mutex
want *llmgwv1.Slurm
kick chan struct{}
// resend has the next status sent even if unchanged: a new stream's
// control plane may never have had it.
resend bool
// installed is true once the packages were found or put there, so a
// minute's check does not look again.
installed map[string]bool
files string
}
// New is a Keeper; marker is a file it keeps while it runs Slurm here, so
// an agent started again knows to stop what it started when told it has no
// part any more.
func New(host Host, inst Installer, marker string, log *slog.Logger) *Keeper {
if log == nil {
log = slog.Default()
}
return &Keeper{host: host, inst: inst, log: log, marker: marker,
kick: make(chan struct{}, 1), installed: map[string]bool{}}
}
// Apply is a new message from the leader: kept from now on.
func (k *Keeper) Apply(m *llmgwv1.Slurm) {
k.mu.Lock()
k.want = proto.CloneOf(m)
k.mu.Unlock()
select {
case k.kick <- struct{}{}:
default:
}
}
// HasPart is whether the last message gives this machine a part in Slurm.
func (k *Keeper) HasPart() bool {
k.mu.Lock()
defer k.mu.Unlock()
return k.want.GetController() || k.want.GetNode()
}
// Resend has the status sent again on the next pass, which it starts now.
func (k *Keeper) Resend() {
k.mu.Lock()
k.resend = true
k.mu.Unlock()
select {
case k.kick <- struct{}{}:
default:
}
}
// Every is how often the machine is checked again without a new message.
var Every = time.Minute
// resendAfter is how often an unchanged status is sent anyway.
const resendAfter = 5 * time.Minute
// Run keeps the machine until ctx ends, sending its status when it changes
// and every few minutes while it has a part.
func (k *Keeper) Run(ctx context.Context, send func(*llmgwv1.SlurmStatus)) {
t := time.NewTicker(Every)
defer t.Stop()
var last *llmgwv1.SlurmStatus
var sent time.Time
for {
select {
case <-ctx.Done():
return
case <-k.kick:
case <-t.C:
}
k.mu.Lock()
want, again := k.want, k.resend
k.resend = false
k.mu.Unlock()
if want == nil {
continue
}
st := k.Reconcile(ctx, want)
if st == nil {
last = nil
continue
}
if again || !proto.Equal(st, last) || time.Since(sent) > resendAfter {
send(st)
last, sent = st, time.Now()
}
}
}
// Reconcile makes the machine what want says once, and answers its status;
// nil when it has no part in Slurm.
func (k *Keeper) Reconcile(ctx context.Context, want *llmgwv1.Slurm) *llmgwv1.SlurmStatus {
ctx, cancel := context.WithTimeout(ctx, 20*time.Minute)
defer cancel()
if !want.GetController() && !want.GetNode() {
k.stop(ctx)
return nil
}
st := &llmgwv1.SlurmStatus{}
fail := func(what string, err error) *llmgwv1.SlurmStatus {
st.Error = what + ": " + err.Error()
k.log.Warn("keeping Slurm on this node: "+what, "err", err)
return st
}
if err := k.ensurePackages(ctx, want); err != nil {
return fail("installing Slurm", err)
}
// From the daemon, not scontrol: a client command looks for its files
// even to say its version, and a configless node has none before slurmd
// has fetched them.
daemon := "slurmd"
if !want.GetNode() {
daemon = "slurmctld"
}
out, err := k.host.Run(ctx, nil, daemon, "-V")
if err != nil {
return fail("reading Slurm's version", fmt.Errorf("%v: %s", err, lastLine(out)))
}
v, err := slurm.ParseVersion(string(out))
if err != nil {
return fail("reading Slurm's version", err)
}
st.Version = strings.TrimSpace(string(out))
if i := strings.LastIndexByte(st.Version, ' '); i >= 0 {
st.Version = st.Version[i+1:]
}
if slurm.TooOld(v) {
return fail("Slurm", fmt.Errorf("%d.%02d is older than %d.%02d, the first to run jobs in cgroup v2; "+
"a newer release of the distribution, or Slurm's own packages, has one", v[0], v[1],
slurm.MinVersion[0], slurm.MinVersion[1]))
}
if err := k.host.WriteFile(k.marker, []byte("llmgw runs Slurm here (BACKLOG P)\n"), 0o644, ""); err != nil {
return fail("keeping its marker", err)
}
if err := k.munge(ctx, want.GetMungeKey()); err != nil {
return fail("munge", err)
}
if want.GetController() {
sum, err := k.controller(ctx, want.GetFiles())
st.FilesSha256 = sum
if err != nil {
return fail("slurmctld", err)
}
st.SlurmctldRunning = k.active(ctx, "slurmctld")
}
if want.GetNode() {
if err := k.node(ctx, want, st); err != nil {
return fail("slurmd", err)
}
}
return st
}
func (k *Keeper) ensurePackages(ctx context.Context, want *llmgwv1.Slurm) error {
need := map[string][2]string{
"munged": {"munge", "munge"},
"scontrol": {"slurm-client", "slurm"},
}
if want.GetController() {
need["slurmctld"] = [2]string{"slurmctld", "slurm-slurmctld"}
}
if want.GetNode() {
need["slurmd"] = [2]string{"slurmd", "slurm-slurmd"}
}
pkgs := map[string][]string{}
for bin, p := range need {
if k.installed[bin] {
continue
}
if _, err := k.host.LookPath(bin); err == nil {
k.installed[bin] = true
continue
}
pkgs["deb"] = append(pkgs["deb"], p[0])
pkgs["rpm"] = append(pkgs["rpm"], p[1])
}
if len(pkgs) == 0 {
return nil
}
slices.Sort(pkgs["deb"])
slices.Sort(pkgs["rpm"])
if k.inst == nil {
return fmt.Errorf("%s missing, and this agent installs nothing", strings.Join(pkgs["deb"], ", "))
}
k.log.Info("installing Slurm, as the cluster's settings say", "packages", pkgs["deb"])
if err := k.inst.InstallPackages(ctx, pkgs); err != nil {
return err
}
for bin := range need {
if _, err := k.host.LookPath(bin); err != nil {
return fmt.Errorf("%s is still missing after installing %s", bin, strings.Join(pkgs["deb"], ", "))
}
k.installed[bin] = true
}
return nil
}
// munge writes the cluster's key and keeps munged running on it.
func (k *Keeper) munge(ctx context.Context, key []byte) error {
if len(key) < 32 {
return errors.New("the leader sent no key")
}
changed, err := k.write(slurm.MungeKeyPath, key, 0o400, "munge")
if err != nil {
return err
}
if changed {
k.log.Info("munge key written; munge starts again on it")
return k.systemctl(ctx, "enable", "--now", "munge", "&&restart")
}
if !k.active(ctx, "munge") {
return k.systemctl(ctx, "enable", "--now", "munge")
}
return nil
}
// controller writes the files and keeps slurmctld running on them,
// answering their digest.
func (k *Keeper) controller(ctx context.Context, files map[string]string) (string, error) {
if files["slurm.conf"] == "" {
return "", errors.New("the leader sent no slurm.conf")
}
for _, d := range []string{slurm.StateDir, slurm.LogDir} {
if err := k.host.MkdirAll(d, 0o755, "slurm"); err != nil {
return "", err
}
}
names := make([]string, 0, len(files))
for n := range files {
names = append(names, n)
}
slices.Sort(names)
h := sha256.New()
changed := false
for _, n := range names {
if strings.ContainsAny(n, "/\\") || n == "" || n == "." || n == ".." {
return "", fmt.Errorf("the leader sent a file named %q", n)
}
fmt.Fprintf(h, "%s\x00%s\x00", n, files[n])
c, err := k.write(filepath.Join(slurm.ConfDir, n), []byte(files[n]), 0o644, "")
if err != nil {
return "", err
}
changed = changed || c
}
sum := hex.EncodeToString(h.Sum(nil))
if changed || sum != k.files {
if err := k.systemctl(ctx, "enable", "slurmctld"); err != nil {
return sum, err
}
// Nodes added or taken away need a restart before 23.11 and a
// reconfigure after it; a restart does both, and running jobs live
// through it.
k.log.Info("slurmctld starts again on new files")
if err := k.systemctl(ctx, "restart", "slurmctld"); err != nil {
return sum, err
}
if err := k.waitPing(ctx); err != nil {
return sum, err
}
// Hands the configless slurmd the new files.
if out, err := k.host.Run(ctx, nil, "scontrol", "reconfigure"); err != nil {
return sum, fmt.Errorf("scontrol reconfigure: %v: %s", err, lastLine(out))
}
k.files = sum
return sum, nil
}
if !k.active(ctx, "slurmctld") {
return sum, k.systemctl(ctx, "enable", "--now", "slurmctld")
}
return sum, nil
}
func (k *Keeper) waitPing(ctx context.Context) error {
var last []byte
for range 30 {
out, err := k.host.Run(ctx, nil, "scontrol", "ping")
if err == nil && bytes.Contains(out, []byte("UP")) {
return nil
}
last = out
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(pingWait):
}
}
return fmt.Errorf("slurmctld did not answer after starting: %s", lastLine(last))
}
var pingWait = time.Second
// node keeps slurmd running configless and puts the node in or out of the
// controller's hands as lent says.
func (k *Keeper) node(ctx context.Context, want *llmgwv1.Slurm, st *llmgwv1.SlurmStatus) error {
name, server := want.GetName(), want.GetConfServer()
if name == "" || server == "" {
return errors.New("the leader sent no name or controller")
}
if strings.ContainsAny(name+server, " \n\t\"'\\$`") {
return fmt.Errorf("the leader sent a name or controller with spaces or quotes in it")
}
for _, d := range []string{slurm.SpoolDir, slurm.LogDir} {
if err := k.host.MkdirAll(d, 0o755, "slurm"); err != nil {
return err
}
}
bin, err := k.host.LookPath("slurmd")
if err != nil {
return err
}
unit := "# Written by llmgw (BACKLOG P): slurmd fetches its files from slurmctld.\n" +
"[Unit]\nConditionPathExists=\nStartLimitIntervalSec=0\n" +
// --systemd as the packaged unit has it (Type=notify, since 22.05):
// without it systemd waits for a readiness that never comes.
"[Service]\nExecStart=\nExecStart=" + bin + " --systemd --conf-server " + server + " -N " + name + "\n" +
// A node new to the controller is refused until the leader has
// written it into slurm.conf; tried again until it is.
"Restart=on-failure\nRestartSec=15\n"
changed, err := k.write(DropIn, []byte(unit), 0o644, "")
if err != nil {
return err
}
if out, err := k.host.Run(ctx, nil, "slurmd", "-C"); err == nil {
st.Hardware = strings.TrimSpace(string(out))
}
if changed {
if err := k.systemctl(ctx, "daemon-reload"); err != nil {
return err
}
if err := k.systemctl(ctx, "enable", "slurmd"); err != nil {
return err
}
if err := k.systemctl(ctx, "restart", "slurmd"); err != nil {
return err
}
} else if !k.active(ctx, "slurmd") {
if err := k.systemctl(ctx, "enable", "--now", "slurmd"); err != nil {
return err
}
}
st.SlurmdRunning = k.active(ctx, "slurmd")
st.Jobs = k.localJobs(ctx)
env := k.clientEnv()
out, err := k.host.Run(ctx, env, "scontrol", "show", "node", "-o", name)
if err != nil {
// Not in slurm.conf yet: the leader writes it once this node's
// hardware is in.
return nil
}
ns, err := slurm.ParseNodeState(string(out))
if err != nil {
return nil
}
ours := ns.Reason == "" || strings.HasPrefix(ns.Reason, reasonPrefix)
switch {
case want.GetLent() && ns.Drained() && ours:
k.log.Info("this node is lent to Slurm: resumed there")
if out, err := k.host.Run(ctx, env, "scontrol", "update", "NodeName="+name, "State=RESUME"); err != nil {
return fmt.Errorf("resuming the node: %v: %s", err, lastLine(out))
}
case !want.GetLent() && !strings.Contains(ns.State, "DRAIN"):
k.log.Info("this node is llmgw's: drained in Slurm, its jobs left to finish")
if out, err := k.host.Run(ctx, env, "scontrol", "update", "NodeName="+name, "State=DRAIN",
"Reason="+ReasonServing); err != nil {
return fmt.Errorf("draining the node: %v: %s", err, lastLine(out))
}
}
if out, err := k.host.Run(ctx, env, "scontrol", "show", "node", "-o", name); err == nil {
if ns2, err := slurm.ParseNodeState(string(out)); err == nil {
ns = ns2
}
}
st.State, st.Reason = ns.State, ns.Reason
return nil
}
// localJobs counts the job steps running here, from their slurmstepd, not
// from the controller: a job outlives a controller that is down, and is
// what decides whether llmgw may have the cards back. The one slurmstepd
// that is no job's, "slurmstepd infinity" holding slurmd's cgroup v2 scope,
// is not counted. -1 when it cannot tell.
func (k *Keeper) localJobs(ctx context.Context) int32 {
// pgrep exits 1 with no output when there is none.
out, err := k.host.Run(ctx, nil, "pgrep", "-a", "-x", "slurmstepd")
if err != nil && len(bytes.TrimSpace(out)) > 0 {
return -1
}
n := int32(0)
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
f := strings.Fields(line)
if len(f) < 2 || f[len(f)-1] == "infinity" {
continue
}
n++
}
return n
}
// clientEnv points scontrol at slurmd's copy of the files on a node that is
// not the controller, which has none of its own.
func (k *Keeper) clientEnv() []string {
if _, err := k.host.ReadFile(filepath.Join(slurm.ConfDir, "slurm.conf")); err == nil {
return nil
}
return []string{"SLURM_CONF=" + filepath.Join(slurm.SpoolDir, "conf-cache", "slurm.conf")}
}
// stop stops what llmgw started here, if it did.
func (k *Keeper) stop(ctx context.Context) {
if _, err := k.host.ReadFile(k.marker); err != nil {
return
}
k.log.Info("this node has no part in Slurm any more: slurmd and slurmctld stopped; the packages stay")
for _, u := range []string{"slurmd", "slurmctld"} {
_ = k.systemctl(ctx, "disable", "--now", u)
}
_ = k.host.Remove(DropIn)
_ = k.systemctl(ctx, "daemon-reload")
_ = k.host.Remove(k.marker)
k.files = ""
}
// write writes a file when it differs, saying whether it did.
func (k *Keeper) write(path string, b []byte, mode fs.FileMode, owner string) (bool, error) {
if old, err := k.host.ReadFile(path); err == nil && bytes.Equal(old, b) {
return false, nil
}
return true, k.host.WriteFile(path, b, mode, owner)
}
// systemctl runs systemctl; a last argument "&&restart" restarts the unit
// after.
func (k *Keeper) systemctl(ctx context.Context, args ...string) error {
restart := false
if n := len(args); n > 0 && args[n-1] == "&&restart" {
args, restart = args[:n-1], true
}
if out, err := k.host.Run(ctx, nil, "systemctl", args...); err != nil {
return fmt.Errorf("systemctl %s: %v: %s", strings.Join(args, " "), err, lastLine(out))
}
if restart {
u := args[len(args)-1]
if out, err := k.host.Run(ctx, nil, "systemctl", "restart", u); err != nil {
return fmt.Errorf("systemctl restart %s: %v: %s", u, err, lastLine(out))
}
}
return nil
}
func (k *Keeper) active(ctx context.Context, unit string) bool {
out, err := k.host.Run(ctx, nil, "systemctl", "is-active", unit)
return err == nil && strings.TrimSpace(string(out)) == "active"
}
func lastLine(b []byte) string {
s := strings.TrimSpace(string(b))
if i := strings.LastIndexByte(s, '\n'); i >= 0 {
s = s[i+1:]
}
return s
}
// Local is the machine the agent runs on.
type Local struct{}
// Run runs a command, with env added to the agent's.
func (Local) Run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
cmd := exec.CommandContext(ctx, name, args...)
if len(env) > 0 {
cmd.Env = append(os.Environ(), env...)
}
return cmd.CombinedOutput()
}
// LookPath finds a command, in the sbin directories too: an agent started
// by systemd has them, one started by hand may not.
func (Local) LookPath(name string) (string, error) {
if p, err := exec.LookPath(name); err == nil {
return p, nil
}
for _, d := range []string{"/usr/sbin", "/sbin", "/usr/local/sbin"} {
p := filepath.Join(d, name)
if fi, err := os.Stat(p); err == nil && !fi.IsDir() {
return p, nil
}
}
return "", fmt.Errorf("%s is not installed", name)
}
// ReadFile reads a file.
func (Local) ReadFile(path string) ([]byte, error) { return os.ReadFile(path) } //nolint:gosec // our paths
// WriteFile writes a file whole, through a temporary one beside it.
func (Local) WriteFile(path string, b []byte, mode fs.FileMode, owner string) error {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { //nolint:gosec // system directories
return err
}
tmp := path + ".llmgw-new"
if err := os.WriteFile(tmp, b, mode); err != nil {
return err
}
if err := os.Chmod(tmp, mode); err != nil {
return err
}
if err := chown(tmp, owner); err != nil {
_ = os.Remove(tmp)
return err
}
return os.Rename(tmp, path)
}
// MkdirAll makes a directory owned by owner.
func (Local) MkdirAll(path string, mode fs.FileMode, owner string) error {
if err := os.MkdirAll(path, mode); err != nil {
return err
}
return chown(path, owner)
}
// Remove removes a file, and is content when there is none.
func (Local) Remove(path string) error {
if err := os.Remove(path); err != nil && !errors.Is(err, fs.ErrNotExist) {
return err
}
return nil
}
func chown(path, owner string) error {
if owner == "" {
return nil
}
u, err := user.Lookup(owner)
if err != nil {
return fmt.Errorf("user %s: %w (installed with its package)", owner, err)
}
uid, _ := strconv.Atoi(u.Uid)
gid, _ := strconv.Atoi(u.Gid)
return os.Chown(path, uid, gid)
}
+308
View File
@@ -0,0 +1,308 @@
package slurmnode
import (
"context"
"errors"
"io/fs"
"path/filepath"
"slices"
"strings"
"testing"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
type fakeHost struct {
files map[string][]byte
owners map[string]string
bins map[string]bool
active map[string]bool
ran []string
outputs map[string]string
fail map[string]bool
state string
reason string
}
func newFake() *fakeHost {
return &fakeHost{files: map[string][]byte{}, owners: map[string]string{}, bins: map[string]bool{},
active: map[string]bool{}, outputs: map[string]string{
"slurmd -V": "slurm-wlm 23.11.4\n",
"slurmctld -V": "slurm-wlm 23.11.4\n",
"scontrol ping": "Slurmctld(primary) at node06 is UP\n",
"slurmd -C": "NodeName=cuda01 CPUs=16 RealMemory=64000\nUpTime=1-00:00:00\n",
}, fail: map[string]bool{}, state: "IDLE+DRAIN", reason: slurm.ReasonNew}
}
func (f *fakeHost) Run(_ context.Context, env []string, name string, args ...string) ([]byte, error) {
c := strings.Join(append([]string{name}, args...), " ")
f.ran = append(f.ran, c)
if f.fail[c] {
return []byte("it failed"), errors.New("exit status 1")
}
switch {
case name == "systemctl" && len(args) >= 1:
u := args[len(args)-1]
switch args[0] {
case "is-active":
if f.active[u] {
return []byte("active\n"), nil
}
return []byte("inactive\n"), errors.New("exit status 3")
case "restart", "start":
f.active[u] = true
case "enable":
if slices.Contains(args, "--now") {
f.active[u] = true
}
case "disable":
f.active[u] = false
}
return nil, nil
case strings.HasPrefix(c, "scontrol show node"):
return []byte("NodeName=cuda01 Arch=x86_64 State=" + f.state + " Partitions=gpu Reason=" + f.reason + " [root@now]\n"), nil
case strings.HasPrefix(c, "scontrol update"):
if slices.Contains(args, "State=RESUME") {
f.state, f.reason = "IDLE", ""
} else {
f.state, f.reason = "IDLE+DRAIN", ReasonServing
}
return nil, nil
case strings.HasPrefix(c, "pgrep"):
out := "1800358 /usr/sbin/slurmstepd infinity\n" + f.outputs["jobs"]
return []byte(out), nil
}
return []byte(f.outputs[c]), nil
}
func (f *fakeHost) LookPath(name string) (string, error) {
if f.bins[name] {
return "/usr/sbin/" + name, nil
}
return "", errors.New("not found")
}
func (f *fakeHost) ReadFile(path string) ([]byte, error) {
b, ok := f.files[path]
if !ok {
return nil, fs.ErrNotExist
}
return b, nil
}
func (f *fakeHost) WriteFile(path string, b []byte, _ fs.FileMode, owner string) error {
f.files[path] = append([]byte(nil), b...)
f.owners[path] = owner
return nil
}
func (f *fakeHost) MkdirAll(string, fs.FileMode, string) error { return nil }
func (f *fakeHost) Remove(path string) error { delete(f.files, path); return nil }
func (f *fakeHost) count(prefix string) int {
n := 0
for _, c := range f.ran {
if strings.HasPrefix(c, prefix) {
n++
}
}
return n
}
type fakeInst struct {
host *fakeHost
got map[string][]string
}
func (i *fakeInst) InstallPackages(_ context.Context, by map[string][]string) error {
i.got = by
for _, b := range []string{"munged", "scontrol", "slurmd", "slurmctld"} {
i.host.bins[b] = true
}
return nil
}
var key = []byte(strings.Repeat("k", 1024))
func nodeMsg(lent bool) *llmgwv1.Slurm {
return &llmgwv1.Slurm{Node: true, Name: "cuda01", ConfServer: "10.0.0.146:6817", MungeKey: key, Lent: lent}
}
func TestNodeInstallsAndStaysDrained(t *testing.T) {
h := newFake()
inst := &fakeInst{host: h}
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
st := k.Reconcile(context.Background(), nodeMsg(false))
if st.GetError() != "" {
t.Fatal(st.GetError())
}
if !slices.Equal(inst.got["deb"], []string{"munge", "slurm-client", "slurmd"}) ||
!slices.Equal(inst.got["rpm"], []string{"munge", "slurm", "slurm-slurmd"}) {
t.Fatalf("installed %v", inst.got)
}
if string(h.files[slurm.MungeKeyPath]) != string(key) || h.owners[slurm.MungeKeyPath] != "munge" {
t.Fatal("the munge key was not written for munge")
}
unit := string(h.files[DropIn])
if !strings.Contains(unit, "ExecStart=/usr/sbin/slurmd --systemd --conf-server 10.0.0.146:6817 -N cuda01\n") ||
!strings.Contains(unit, "ConditionPathExists=\n") {
t.Fatalf("drop-in:\n%s", unit)
}
if !st.GetSlurmdRunning() || st.GetVersion() != "23.11.4" || !strings.HasPrefix(st.GetHardware(), "NodeName=cuda01 CPUs=16") {
t.Fatalf("status %v", st)
}
// Not lent: left drained, with llmgw's first reason.
if h.count("scontrol update") != 0 || st.GetState() != "IDLE+DRAIN" {
t.Fatalf("a node not lent was changed: %v, %v", h.ran, st)
}
if _, ok := h.files["/var/lib/llmgw-agent/slurm"]; !ok {
t.Fatal("no marker")
}
// Again, nothing changed: no install, no restart.
h.ran = nil
inst.got = nil
k.Reconcile(context.Background(), nodeMsg(false))
if inst.got != nil || h.count("systemctl restart") != 0 || h.count("systemctl enable") != 0 {
t.Fatalf("a second pass did work: %v", h.ran)
}
}
func TestLendAndReclaim(t *testing.T) {
h := newFake()
k := New(h, &fakeInst{host: h}, "/m", nil)
st := k.Reconcile(context.Background(), nodeMsg(true))
if h.count("scontrol update NodeName=cuda01 State=RESUME") != 1 || st.GetState() != "IDLE" {
t.Fatalf("a lent node was not resumed: %v %v", h.ran, st)
}
h.outputs["jobs"] = "2001 slurmstepd: [101.batch]\n2002 slurmstepd: [102.0]\n"
st = k.Reconcile(context.Background(), nodeMsg(false))
if h.count("scontrol update NodeName=cuda01 State=DRAIN Reason="+ReasonServing) != 1 {
t.Fatalf("a node taken back was not drained: %v", h.ran)
}
if st.GetJobs() != 2 || !strings.Contains(st.GetState(), "DRAIN") {
t.Fatalf("status %v", st)
}
// Drained already: not drained again.
h.ran = nil
k.Reconcile(context.Background(), nodeMsg(false))
if h.count("scontrol update") != 0 {
t.Fatalf("drained twice: %v", h.ran)
}
}
func TestAdminsDrainIsLeft(t *testing.T) {
h := newFake()
h.reason = "bad DIMM, ticket 42"
k := New(h, &fakeInst{host: h}, "/m", nil)
k.Reconcile(context.Background(), nodeMsg(true))
if h.count("scontrol update") != 0 {
t.Fatalf("llmgw resumed a node a Slurm administrator drained: %v", h.ran)
}
}
func TestClientEnvOnANodeWithoutFiles(t *testing.T) {
h := newFake()
k := New(h, &fakeInst{host: h}, "/m", nil)
if env := k.clientEnv(); len(env) != 1 || !strings.HasSuffix(env[0], "conf-cache/slurm.conf") {
t.Fatalf("env %v", env)
}
h.files[filepath.Join(slurm.ConfDir, "slurm.conf")] = []byte("x")
if env := k.clientEnv(); env != nil {
t.Fatalf("env %v on the controller", env)
}
}
func TestController(t *testing.T) {
h := newFake()
inst := &fakeInst{host: h}
k := New(h, inst, "/m", nil)
files := map[string]string{"slurm.conf": "ClusterName=llmgw\n", "gres.conf": "#\n", "cgroup.conf": "#\n"}
m := &llmgwv1.Slurm{Controller: true, MungeKey: key, Files: files}
st := k.Reconcile(context.Background(), m)
if st.GetError() != "" {
t.Fatal(st.GetError())
}
if !slices.Contains(inst.got["deb"], "slurmctld") || slices.Contains(inst.got["deb"], "slurmd") {
t.Fatalf("installed %v", inst.got)
}
if string(h.files["/etc/slurm/slurm.conf"]) != "ClusterName=llmgw\n" {
t.Fatal("slurm.conf not written")
}
if h.count("systemctl restart slurmctld") != 1 || h.count("scontrol reconfigure") != 1 {
t.Fatalf("slurmctld not started on its files: %v", h.ran)
}
if !st.GetSlurmctldRunning() || len(st.GetFilesSha256()) != 64 {
t.Fatalf("status %v", st)
}
sum := st.GetFilesSha256()
h.ran = nil
k.Reconcile(context.Background(), m)
if h.count("systemctl restart") != 0 {
t.Fatalf("restarted on the same files: %v", h.ran)
}
files["gres.conf"] = "NodeName=cuda01 Name=gpu File=/dev/nvidia0\n"
h.ran = nil
st = k.Reconcile(context.Background(), m)
if h.count("systemctl restart slurmctld") != 1 || st.GetFilesSha256() == sum {
t.Fatalf("a changed file did not restart slurmctld: %v", h.ran)
}
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Controller: true, MungeKey: key,
Files: map[string]string{"slurm.conf": "x", "../evil": "y"}}); !strings.Contains(st.GetError(), "named") {
t.Fatalf("a path in a file name was taken: %v", st)
}
}
func TestTooOld(t *testing.T) {
h := newFake()
h.outputs["slurmd -V"] = "slurm-wlm 21.08.5\n"
k := New(h, &fakeInst{host: h}, "/m", nil)
st := k.Reconcile(context.Background(), nodeMsg(false))
if !strings.Contains(st.GetError(), "older than 22.05") {
t.Fatalf("status %v", st)
}
if _, ok := h.files[slurm.MungeKeyPath]; ok {
t.Fatal("went on with a Slurm too old")
}
}
func TestStop(t *testing.T) {
h := newFake()
k := New(h, &fakeInst{host: h}, "/m", nil)
k.Reconcile(context.Background(), nodeMsg(false))
h.ran = nil
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{}); st != nil {
t.Fatalf("status %v for no part", st)
}
if h.count("systemctl disable --now slurmd") != 1 {
t.Fatalf("slurmd not stopped: %v", h.ran)
}
if _, ok := h.files[DropIn]; ok {
t.Fatal("drop-in left")
}
// Never started here: nothing touched.
h2 := newFake()
k2 := New(h2, &fakeInst{host: h2}, "/m", nil)
k2.Reconcile(context.Background(), &llmgwv1.Slurm{})
if len(h2.ran) != 0 {
t.Fatalf("a machine llmgw never ran Slurm on was touched: %v", h2.ran)
}
}
func TestBadNames(t *testing.T) {
h := newFake()
k := New(h, &fakeInst{host: h}, "/m", nil)
m := nodeMsg(false)
m.Name = "cuda01 -c /etc/evil"
if st := k.Reconcile(context.Background(), m); !strings.Contains(st.GetError(), "spaces") {
t.Fatalf("status %v", st)
}
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Node: true, Name: "a", ConfServer: "b"}); !strings.Contains(st.GetError(), "no key") {
t.Fatalf("status %v", st)
}
}
+12 -1
View File
@@ -33,6 +33,17 @@ func CachePathFor(dir, source string) string {
return filepath.Join(dir, hex.EncodeToString(sum[:8])+"-"+name)
}
// cacheSource is the source a cache name is computed from: a set's address
// without its trailing slash, as its directory was named when it was fetched.
// Computed from the slash-ended address, every set in the catalog looked like
// nobody's, and a sweep offered 330 GB of models still in use for deletion.
func cacheSource(source string) string {
if IsSet(source) {
return strings.TrimSuffix(source, "/")
}
return source
}
// Owned reports whether a path is one this package created.
//
// The dividing line of the whole feature. A file without the cache name was
@@ -65,7 +76,7 @@ func Orphans(have []string, wanted []string) []string {
// By base name rather than full path: the control plane does not know
// each node's cache directory, and the name is what identifies the
// file wherever the node keeps it.
keep[filepath.Base(CachePathFor("", src))] = struct{}{}
keep[filepath.Base(CachePathFor("", cacheSource(src)))] = struct{}{}
}
var out []string
+1 -1
View File
@@ -51,7 +51,7 @@ type manifest struct {
// as a broken model rather than failing to load, so a partial set is left
// unfinished under its temporary name and the next attempt starts again.
func (f *Fetcher) ensureSet(ctx context.Context, source string, allowPickle bool) (string, error) {
dir, err := f.cachePath(strings.TrimSuffix(source, "/"))
dir, err := f.cachePath(cacheSource(source))
if err != nil {
return "", err
}
+17
View File
@@ -74,6 +74,23 @@ func TestASetIsFetchedIntoOneDirectory(t *testing.T) {
}
}
func TestASetInTheCatalogIsNotAnOrphan(t *testing.T) {
// The catalog names a set by its address with the trailing slash; the
// fetch names the directory without it. A sweep that computed the name
// from the catalog's spelling offered every set in use for deletion.
f, srv, _ := mirrorOf(t, map[string]string{"model.safetensors": "aaaa", "config.json": "{}"})
dir, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false)
if err != nil {
t.Fatal(err)
}
if got := Orphans([]string{dir}, []string{srv.URL + "/m/v/"}); len(got) != 0 {
t.Errorf("a set the catalog still names was called an orphan: %v", got)
}
if got := Orphans([]string{dir}, nil); len(got) != 1 {
t.Errorf("a set nothing names was not found: %v", got)
}
}
func TestAFinishedSetIsNotFetchedAgain(t *testing.T) {
f, srv, hits := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
+2
View File
@@ -247,6 +247,8 @@ var liveSettings = []string{
"presets",
// Handed to the reconciler as they change.
"on_demand",
// Read by the Slurm service each pass, kicked when they change.
"slurm",
// Read by server init, node add and backups, not by the server.
"backup_recipients", "backup_copy_to",
}
+1
View File
@@ -189,6 +189,7 @@ func newCtlCmd() *cobra.Command {
newMCPCmd(client, &asJSON),
newKBCmd(client, &asJSON),
newTrainCmd(client, &asJSON),
newSlurmCmd(client, &asJSON),
newConfigCmd(client, &asJSON),
newAuditCmd(client, &asJSON),
newNotifyCmd(client, &asJSON),
+242
View File
@@ -0,0 +1,242 @@
package cli
import (
"encoding/json"
"fmt"
"slices"
"strings"
"text/tabwriter"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
)
// newSlurmCmd is the Slurm cluster run beside llmgw (BACKLOG P): set up on
// nodes llmgw already has, and nodes handed between the two.
func newSlurmCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
cmd := &cobra.Command{
Use: "slurm",
Short: "A Slurm cluster beside llmgw, for training and other batch work",
Long: "Slurm runs on llmgw's own nodes, installed and kept by their agents: slurmctld on one node,\n" +
"slurmd on those named. A node is llmgw's, drained in Slurm, until it is lent; lent, its\n" +
"replicas move off first, then Slurm may run jobs there. Taken back, Slurm drains it and\n" +
"llmgw has it again when its last job ends. Jobs are submitted on the controller node\n" +
"(sbatch, srun) by its own users, as on any Slurm cluster.",
}
cmd.AddCommand(newSlurmStatusCmd(client, asJSON), newSlurmSetupCmd(client), newSlurmOffCmd(client),
newSlurmListCmd(client, "lend", "Let Slurm run jobs on nodes; llmgw's replicas move off first", true),
newSlurmListCmd(client, "reclaim", "Take nodes back from Slurm; llmgw has them when their jobs end", false))
return cmd
}
func newSlurmStatusCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
return &cobra.Command{
Use: "status",
Short: "The Slurm cluster, and where each of its nodes is",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
var st slurmsvc.Status
if err := client().get(cmd.Context(), "/api/v1/slurm", &st); err != nil {
return err
}
w := cmd.OutOrStdout()
if *asJSON {
return json.NewEncoder(w).Encode(st)
}
if !st.Enabled {
fmt.Fprintln(w, "Slurm is off. `llmgw ctl slurm setup --controller NODE --nodes NODE,...` turns it on.")
return nil
}
fmt.Fprintf(w, "cluster %s, partition %s, controller %s", st.Cluster, st.Partition, st.Controller)
switch c := st.Ctld; {
case c == nil:
fmt.Fprintln(w, ": not heard from yet")
case c.Error != "":
fmt.Fprintf(w, ": %s\n", c.Error)
case c.SlurmctldRunning:
fmt.Fprintf(w, ": slurmctld %s running\n", c.Version)
default:
fmt.Fprintln(w, ": slurmctld not running")
}
tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
fmt.Fprintln(tw, "NODE\tPHASE\tSLURM\tJOBS\tSLURMD\tNOTE")
for _, n := range st.Nodes {
state, jobs, slurmd, note := "-", "-", "-", n.Describe()
if r := n.Status; r != nil {
state = or(r.State, "-")
jobs = fmt.Sprint(r.Jobs)
slurmd = map[bool]string{true: "running", false: "stopped"}[r.SlurmdRunning]
if r.Error != "" {
note = r.Error
}
}
if !n.Connected {
note = "agent not connected; " + note
}
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\t%s\n", n.Node, n.Phase, state, jobs, slurmd, note)
}
if err := tw.Flush(); err != nil {
return err
}
if len(st.Waiting) > 0 {
fmt.Fprintf(w, "not in slurm.conf until they report their hardware: %s\n", strings.Join(st.Waiting, ", "))
}
return nil
},
}
}
func newSlurmSetupCmd(client func() *ctlClient) *cobra.Command {
var controller, cluster, partition, note string
var nodes []string
var all bool
c := &cobra.Command{
Use: "setup --controller NODE --nodes NODE,... | --all-nodes",
Short: "Run Slurm on llmgw's nodes: slurmctld on one, slurmd on those named",
Long: "Turns slurm.enabled on and names the nodes, or with --all-nodes every node. Their agents install munge and Slurm (22.05 or\n" +
"newer) from the machines' own package sources. Every node starts drained in Slurm and stays\n" +
"llmgw's until `llmgw ctl slurm lend` gives it to Slurm.",
Example: " llmgw ctl slurm setup --controller node06 --nodes cuda01,cuda02\n" +
" llmgw ctl slurm setup --controller node06 --all-nodes",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
if controller == "" || (len(nodes) == 0 && !all) {
return fmt.Errorf("give --controller, and --nodes or --all-nodes")
}
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
if err := doc.SetYAML([]string{"slurm", "enabled"}, "true"); err != nil {
return err
}
if err := doc.SetYAML([]string{"slurm", "controller"}, yamlString(controller)); err != nil {
return err
}
if err := setList(doc, []string{"slurm", "nodes"}, nodes); err != nil {
return err
}
if err := doc.SetYAML([]string{"slurm", "all_nodes"}, fmt.Sprint(all)); err != nil {
return err
}
// Lent nodes no longer named are not lent.
lent := slices.DeleteFunc(docList(doc, []string{"slurm", "lent"}),
func(n string) bool { return !all && !slices.Contains(nodes, n) })
if err := setList(doc, []string{"slurm", "lent"}, lent); err != nil {
return err
}
for k, v := range map[string]string{"cluster_name": cluster, "partition": partition} {
if v != "" {
if err := doc.SetYAML([]string{"slurm", k}, yamlString(v)); err != nil {
return err
}
}
}
return nil
})
},
}
c.Flags().StringVar(&controller, "controller", "", "the node that runs slurmctld")
c.Flags().StringSliceVar(&nodes, "nodes", nil, "the nodes that run slurmd")
c.Flags().BoolVar(&all, "all-nodes", false, "every node runs slurmd, those that join later too")
c.Flags().StringVar(&cluster, "cluster-name", "", "Slurm's ClusterName (llmgw)")
c.Flags().StringVar(&partition, "partition", "", "the partition every node is in (gpu)")
c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
return c
}
func newSlurmOffCmd(client func() *ctlClient) *cobra.Command {
var note string
var force bool
c := &cobra.Command{
Use: "off",
Short: "Stop running Slurm; every node is llmgw's again",
Long: "slurmd and slurmctld are stopped where llmgw started them; the packages stay. A node still\n" +
"lent is refused unless --force: take it back first, so its jobs end before the daemons do.",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
if lent := docList(doc, []string{"slurm", "lent"}); len(lent) > 0 && !force {
return fmt.Errorf("%s still lent: `llmgw ctl slurm reclaim` them first, or --force",
strings.Join(lent, ", "))
}
setList(doc, []string{"slurm", "lent"}, nil) //nolint:errcheck // an empty list
return doc.SetYAML([]string{"slurm", "enabled"}, "false")
})
},
}
c.Flags().BoolVar(&force, "force", false, "even with nodes still lent")
c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
return c
}
// newSlurmListCmd adds nodes to slurm.lent, or takes them out.
func newSlurmListCmd(client func() *ctlClient, use, short string, add bool) *cobra.Command {
var note string
c := &cobra.Command{
Use: use + " NODE...",
Short: short,
Args: cobra.MinimumNArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
nodes := docList(doc, []string{"slurm", "nodes"})
all := docBool(doc, []string{"slurm", "all_nodes"})
lent := docList(doc, []string{"slurm", "lent"})
for _, n := range args {
switch {
case add && !all && !slices.Contains(nodes, n):
return fmt.Errorf("%s does not run slurmd: add it with `llmgw ctl slurm setup --nodes`", n)
case add && !slices.Contains(lent, n):
lent = append(lent, n)
case !add:
lent = slices.DeleteFunc(lent, func(l string) bool { return l == n })
}
}
return setList(doc, []string{"slurm", "lent"}, lent)
})
},
}
c.Flags().StringVar(&note, "note", "", "why, kept with the settings' version")
return c
}
// docList reads a list of strings from the settings, empty when unset.
func docList(doc *settings.Doc, path []string) []string {
n := doc.Node(path)
if n == nil {
return nil
}
var out []string
if err := n.Decode(&out); err != nil {
return nil
}
return out
}
// docBool reads a boolean from the settings, false when unset.
func docBool(doc *settings.Doc, path []string) bool {
var b bool
if n := doc.Node(path); n != nil {
_ = n.Decode(&b)
}
return b
}
func setList(doc *settings.Doc, path []string, list []string) error {
if len(list) == 0 {
if doc.Node(path) != nil {
doc.Unset(path)
}
return nil
}
return doc.SetYAML(path, yamlList(list))
}
func yamlString(s string) string {
b, _ := json.Marshal(s)
return string(b)
}
func yamlList(list []string) string {
b, _ := json.Marshal(list)
return string(b)
}
+35
View File
@@ -0,0 +1,35 @@
package cli
import (
"slices"
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
)
func TestSlurmDocLists(t *testing.T) {
doc, err := settings.FromBytes("settings", []byte("slurm:\n enabled: true\n nodes: [cuda01, site2]\n"))
if err != nil {
t.Fatal(err)
}
if got := docList(doc, []string{"slurm", "nodes"}); !slices.Equal(got, []string{"cuda01", "site2"}) {
t.Fatalf("nodes %v", got)
}
if got := docList(doc, []string{"slurm", "lent"}); got != nil {
t.Fatalf("lent %v", got)
}
if err := setList(doc, []string{"slurm", "lent"}, []string{"cuda01"}); err != nil {
t.Fatal(err)
}
if got := docList(doc, []string{"slurm", "lent"}); !slices.Equal(got, []string{"cuda01"}) {
t.Fatalf("lent %v", got)
}
if err := setList(doc, []string{"slurm", "lent"}, nil); err != nil {
t.Fatal(err)
}
b, _ := doc.Bytes()
if strings.Contains(string(b), "lent") {
t.Fatalf("an empty list left:\n%s", b)
}
}
+19 -15
View File
@@ -4,6 +4,7 @@ import (
"encoding/json"
"fmt"
"strings"
"time"
"github.com/spf13/cobra"
@@ -87,13 +88,10 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
if err != nil {
return err
}
var results []struct {
Node string `json:"node"`
Files int `json:"files"`
Bytes int64 `json:"bytes"`
Error string `json:"error,omitempty"`
}
if err := c.post(cmd.Context(), "/api/v1/weights/collect", body, &results); err != nil {
var results []adminapi.CollectResult
// The control plane waits for the nodes to list their files without the
// deleted ones: half a minute at most.
if err := c.patient(time.Minute).post(cmd.Context(), "/api/v1/weights/collect", body, &results); err != nil {
return err
}
if asJSON {
@@ -103,7 +101,7 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
fmt.Fprintln(cmd.OutOrStdout(), "Nothing to reclaim.")
return nil
}
var freed int64
var freed, left int64
for _, r := range results {
if r.Error != "" {
// Reported rather than fatal: the other nodes' space is still worth
@@ -111,14 +109,20 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
fmt.Fprintf(cmd.OutOrStdout(), "%s: %s\n", r.Node, r.Error)
continue
}
fmt.Fprintf(cmd.OutOrStdout(), "%s: asked to remove %d files, %s\n",
r.Node, r.Files, fileSize(r.Bytes))
freed += r.Bytes
line := fmt.Sprintf("%s: removed %d files, %s", r.Node, r.Files, fileSize(r.Bytes))
if r.Left > 0 {
line += fmt.Sprintf("; %d still there, %s", r.Left, fileSize(r.LeftBytes))
}
fmt.Fprintln(cmd.OutOrStdout(), line)
freed, left = freed+r.Bytes, left+r.LeftBytes
}
fmt.Fprintf(cmd.OutOrStdout(), "\n%s freed.\n", fileSize(freed))
if left > 0 {
// Each node decides: one that keeps a path it cannot prove it
// fetched is the feature working. Its agent's log names each.
fmt.Fprintf(cmd.OutOrStdout(), "%s the nodes kept; each node's agent log says why "+
"(journalctl -u llmgw-agent).\n", fileSize(left))
}
// "asked", not "removed": each node decides, and one that refuses a path it
// cannot prove it fetched is the feature working rather than failing.
fmt.Fprintf(cmd.OutOrStdout(), "\n%s requested. The nodes verify each path before deleting it.\n",
fileSize(freed))
return nil
}
+22
View File
@@ -49,6 +49,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
@@ -59,6 +60,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
"gitea.mynodes.xyz/ilker/llmgw/internal/router"
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
@@ -261,6 +263,8 @@ func newServerCmd() *cobra.Command {
// What the settings say of the firewall, for the mirror and the
// store's members; nil without a control plane.
var fwPub *firewallPublisher
// The Slurm cluster beside llmgw, when there is a store (BACKLOG P).
var slurmSvc *slurmsvc.Service
updateMetrics := &modelserve.UpdateCollector{}
// The store's cluster, when it is one; read by the metrics and
// by `ctl status`.
@@ -436,6 +440,21 @@ func newServerCmd() *cobra.Command {
agentSvc.SetTrain(trainer)
go trainer.Run(ctx)
api.SetTrain(&adminapi.TrainAdmin{Store: st, Service: trainer})
// Slurm on the nodes the settings name, its munge key
// made from a cluster secret.
slurmSvc = slurmsvc.New(func() config.SlurmSettings { return current.Load().Slurm },
admin.Inventory(), agentSvc, func(ctx context.Context) ([]byte, error) {
secret, err := st.ClusterSecret(ctx, "slurm-munge")
if err != nil {
return nil, err
}
return slurm.MungeKey(secret)
}, log)
// What slurm.conf held, for the leader that comes next.
slurmSvc.SetMemory(slurmsvc.StoreMemory(st))
agentSvc.SetSlurm(slurmSvc)
go slurmSvc.Run(ctx)
api.SetSlurm(slurmSvc)
}
// The cluster's settings, each change checked against this
// machine's own keys before it is kept.
@@ -841,6 +860,9 @@ func newServerCmd() *cobra.Command {
fwPub.Kick()
}
}
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "slurm") }) && slurmSvc != nil {
slurmSvc.Kick()
}
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "on_demand") }) && admin != nil {
admin.Reconciler().SetEvictionTimes(next.OnDemand.EvictAfterIdle, next.OnDemand.MinLifetime)
}
+5
View File
@@ -124,6 +124,9 @@ type serverInit struct {
// media has the nodes install ComfyUI and the leader add a picture model
// and a 3D model once a card fits one.
media bool
// slurm runs Slurm beside llmgw: slurmctld on this machine's node,
// slurmd on every node, each llmgw's until it is lent.
slurm bool
// evictAfterIdle and minLifetime are on_demand's; zero leaves its own.
evictAfterIdle, minLifetime time.Duration
// standbys makes each machine of the replicated store a control plane
@@ -258,6 +261,8 @@ func newServerInitCmd() *cobra.Command {
f.DurationVar(&in.maxKeyTTL, "max-key-ttl", defaultMaxKeyTTL, "the longest an API key lives")
f.BoolVar(&in.media, "media", false,
"pictures and 3D: the nodes install ComfyUI, and a picture model and a 3D model are added once a card fits one")
f.BoolVar(&in.slurm, "slurm", false,
"Slurm for training and batch jobs: slurmctld on this machine, slurmd on every node, each llmgw's until 'llmgw ctl slurm lend'")
f.DurationVar(&in.evictAfterIdle, "evict-after-idle", 0,
"how long a model loaded on demand goes unasked-for before another may have its card (0: 2m)")
f.DurationVar(&in.minLifetime, "min-lifetime", 0,
+34
View File
@@ -13,6 +13,7 @@ import (
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
)
@@ -132,9 +133,18 @@ func (in *serverInit) checkChoices() error {
return fmt.Errorf("max key ttl %s: an hour at least", in.maxKeyTTL)
case in.evictAfterIdle < 0 || in.minLifetime < 0:
return fmt.Errorf("on-demand times cannot be negative")
case in.slurm && in.noNode:
return fmt.Errorf("slurm runs slurmctld on this machine's node: it needs the node (no --no-node)")
case in.vip != "" && !in.standbys:
return fmt.Errorf("a floating address moves between control planes: it needs the standbys")
}
if in.slurm {
// The controller is this machine's node, named by its hostname: a
// name Slurm does not take would keep the server from starting.
if err := (config.SlurmSettings{Enabled: true, Controller: localNodeID()}).Check(); err != nil {
return fmt.Errorf("slurm: this machine's hostname: %w", err)
}
}
if in.vip != "" {
ip, _, err := net.ParseCIDR(in.vip)
if err != nil || ip.To4() == nil {
@@ -318,6 +328,9 @@ func (in *serverInit) tuneConfig(body string) string {
"# one ('llmgw ctl model presets auto off' stops it).\n"+
"engines:\n comfyui: "+provision.ComfyUIVersion()+"\npresets:\n auto: true\n")
}
if in.slurm {
extra = append(extra, slurmSection(localNodeID()))
}
if in.evictAfterIdle > 0 || in.minLifetime > 0 {
b := "# How models that load when asked for take turns on a card, chosen at\n# install.\non_demand:\n"
if in.evictAfterIdle > 0 {
@@ -334,6 +347,24 @@ func (in *serverInit) tuneConfig(body string) string {
return body
}
// slurmSection is server.yaml's slurm block as an install chooses it:
// slurmctld on this machine's node, slurmd on every node.
func slurmSection(controller string) string {
return "# Slurm beside llmgw, chosen at install: slurmctld on this machine, slurmd on\n" +
"# every node, those that join later too. Each node is llmgw's, drained in\n" +
"# Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm; 'llmgw ctl slurm\n" +
"# reclaim NODE' takes it back once its jobs end.\n" +
"slurm:\n enabled: true\n controller: " + yamlString(controller) + "\n all_nodes: true\n"
}
// localNodeID is the node id this machine's agent takes: its hostname.
func localNodeID() string {
if h, err := os.Hostname(); err == nil && h != "" {
return h
}
return "localhost"
}
// durationYAML is a duration as a person writes it: 2h, 90m, 1m30s.
func durationYAML(d time.Duration) string {
switch {
@@ -467,6 +498,9 @@ func (in *serverInit) initFlags() []string {
if in.media {
add("--media")
}
if in.slurm {
add("--slurm")
}
if in.evictAfterIdle > 0 {
add("--evict-after-idle", durationYAML(in.evictAfterIdle))
}
+7 -4
View File
@@ -17,7 +17,7 @@ import (
func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
in := &serverInit{
language: "tr", signupOpen: true, defaultRPM: 120, defaultTPM: 50000, maxKeyTTL: 30 * 24 * time.Hour,
media: true, evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
media: true, slurm: true, evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
judge: judgeSpec{Fail: "open", Categories: []string{"Violent", "PII"}, Replicas: 3, Budget: 3 * time.Second},
}
j := in.judgeFor()
@@ -41,6 +41,8 @@ func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
t.Errorf("limits %d %d %s", cfg.Auth.DefaultRPM, cfg.Auth.DefaultTPM, cfg.Auth.MaxKeyTTL)
case !cfg.Presets.Auto || cfg.Engines["comfyui"] == "":
t.Errorf("pictures and 3D: presets %v comfyui %q", cfg.Presets.Auto, cfg.Engines["comfyui"])
case !cfg.Slurm.Enabled || !cfg.Slurm.AllNodes || cfg.Slurm.Controller != localNodeID():
t.Errorf("slurm %+v", cfg.Slurm)
case cfg.OnDemand.EvictAfterIdle != time.Minute || cfg.OnDemand.MinLifetime != 90*time.Second:
t.Errorf("on demand %+v", cfg.OnDemand)
}
@@ -95,14 +97,14 @@ func TestAStarterIsChosenByNameOrByItsFile(t *testing.T) {
func TestTheChoicesAreTheirFlagsAgain(t *testing.T) {
in := &serverInit{admin: "ops@example.com", hosts: []string{"llm.example.com"}, postgres: true,
mirrorPath: "/data/models", noNode: true, language: "tr", media: true,
mirrorPath: "/data/models", language: "tr", media: true, slurm: true,
webhooks: []string{"https://hooks.slack.com/services/T/B/x, json:https://a.example.com/in"},
judge: judgeSpec{Categories: []string{"Violent", "Jailbreak"}}, maxKeyTTL: 7 * 24 * time.Hour}
in.starter, _ = starterFrom("qwen3-4b")
got := strings.Join(in.initFlags(), " ")
want := "--admin ops@example.com --host llm.example.com --language tr --max-key-ttl 168h --postgres " +
"--mirror-dir /data/models --no-node --starter qwen3-4b --judge-categories Violent,Jailbreak " +
"--webhook https://hooks.slack.com/services/T/B/x --webhook json:https://a.example.com/in --media"
"--mirror-dir /data/models --starter qwen3-4b --judge-categories Violent,Jailbreak " +
"--webhook https://hooks.slack.com/services/T/B/x --webhook json:https://a.example.com/in --media --slurm"
if got != want {
t.Errorf("flags\n got %s\nwant %s", got, want)
}
@@ -124,6 +126,7 @@ func TestChoicesThatCannotBeInstalledAreRefused(t *testing.T) {
{maxKeyTTL: time.Minute},
{vip: "10.0.0.160/24"},
{standbys: true, vip: "10.0.0.160"},
{slurm: true, noNode: true},
} {
if err := in.checkChoices(); err == nil {
t.Errorf("taken: %+v", in)
+21 -8
View File
@@ -643,17 +643,30 @@ func (m consoleModel) nodeMenu(n adminapi.NodeView) *menuDialog {
menuItem{"Delete model files nothing uses", func(m *consoleModel) tea.Cmd {
return m.call("Unused files on "+n.ID, http.MethodPost, "/api/v1/weights/collect", map[string]string{"node": n.ID},
func(raw json.RawMessage) string {
var rs []struct {
Files int `json:"files"`
Bytes int64 `json:"bytes"`
}
// What the node no longer lists, and what it kept.
var rs []adminapi.CollectResult
_ = json.Unmarshal(raw, &rs)
var files int
var b int64
var files, left int
var b, lb int64
var errs []string
for _, r := range rs {
files, b = files+r.Files, b+r.Bytes
files, b, left, lb = files+r.Files, b+r.Bytes, left+r.Left, lb+r.LeftBytes
if r.Error != "" {
errs = append(errs, r.Error)
}
}
return fmt.Sprintf("%d files, %.1f GB freed.", files, float64(b)/(1<<30))
if len(rs) == 0 {
return "Nothing on " + n.ID + " that no model uses."
}
say := fmt.Sprintf("%d files removed, %.1f GB freed.", files, float64(b)/(1<<30))
if left > 0 {
say += fmt.Sprintf(" %d files (%.1f GB) are still there: the node kept them, and its agent's log "+
"says why (journalctl -u llmgw-agent there).", left, float64(lb)/(1<<30))
}
if len(errs) > 0 {
say += " " + strings.Join(errs, "; ")
}
return say
})
}},
menuItem{"Remove from the cluster…", func(m *consoleModel) tea.Cmd {
+22 -2
View File
@@ -57,7 +57,7 @@ type installForm struct {
standby *installPart
vipF *setupField
control, store, mirror, node, starter, media, judge, firewall, alerts, limits, turns, bundle *installPart
control, store, mirror, node, starter, media, slurm, judge, firewall, alerts, limits, turns, bundle *installPart
admin, hosts, language, signup *setupField
storeKind *setupField
@@ -282,6 +282,25 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
summary: func() string { return "ComfyUI; a picture model and a 3D model once a card fits" },
}
// --- Slurm ---
f.slurm = &installPart{
title: "Slurm", on: in.slurm,
about: "A Slurm cluster beside llmgw, for training and other batch work: slurmctld on this machine, " +
"slurmd on every node, those that join later too. The agents install munge and Slurm from the " +
"machines' own package sources.\n\n" +
"Each node stays llmgw's, drained in Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm: " +
"its models move off first. 'llmgw ctl slurm reclaim NODE' takes it back once its jobs end. Jobs " +
"are submitted on this machine with sbatch and srun.",
off: "no Slurm; 'llmgw ctl slurm setup' adds it later",
summary: func() string { return "slurmctld here, slurmd on every node; none lent" },
needs: func() string {
if !f.node.on {
return "slurmctld runs on this machine's node: turn the node on, or Slurm off"
}
return ""
},
}
// --- the guardrails' judge ---
j := in.judgeFor()
f.judgeFail = choiceField("When no judge answers in time", []string{"refuse", "serve flagged"},
@@ -421,7 +440,7 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
if ha {
f.parts = append(f.parts, f.standby)
}
f.parts = append(f.parts, f.mirror, f.node, f.starter, f.media, f.judge, f.firewall,
f.parts = append(f.parts, f.mirror, f.node, f.starter, f.media, f.slurm, f.judge, f.firewall,
f.alerts, f.limits, f.turns, f.bundle)
return f
}
@@ -653,6 +672,7 @@ func (f *installForm) apply(in *serverInit) error {
in.noModel = !f.starter.on
in.starter, in.starterCtx = starter, ctx
in.media = f.media.on
in.slurm = f.slurm.on
in.noGuardModel = !f.judge.on
in.judge = judge
in.noFirewall = !f.firewall.on
+12
View File
@@ -213,10 +213,17 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
"before it is served.", modelplan.GB(p.KVBytesPerToken*1000))
}
note += "\nThe mirror fetches it and checks it, it joins the catalog, and a replica is started\nand asked to say hello."
// The longest context the first way holds, up to what the model was
// trained for: a choice of another way is checked again on submitting.
ctxDefault := "4096"
if a.Format == "safetensors" {
ctxDefault = strconv.Itoa(p.Context)
}
if len(opts) > 0 {
if most := opts[0].MostContext(p.KVBytesPerToken, p.Trained); most > 0 {
ctxDefault = strconv.Itoa(most)
}
}
if len(opts) > 0 && opts[0].Engine == "strata" {
ctxDefault = "32768"
}
@@ -249,6 +256,11 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
Description: v[3], Format: a.Format}
if a.Format == "gguf" {
req.File = a.Name
// A context longer than the default pool is the pool: one
// slot of it, where the server would refuse the request.
if ctxLen > modelserve.DefaultCtxPool {
req.CtxPool = ctxLen
}
}
o := opts[slices.Index(where, v[4])]
// Asked for a context other than planned: does it still fit?
+2 -2
View File
@@ -221,11 +221,11 @@ func (m consoleModel) userMenu(u store.User) *menuDialog {
items = append(items,
menuItem{"Add to an organization, team or project…", func(m *consoleModel) tea.Cmd {
m.openForm(&formDialog{title: "Membership for " + who,
note: "A member of a team or organization may use every project in it.",
note: "A member of a team or organization may use every project in it. On an organization, a manager also sees its page and an admin also changes it.",
fields: []formField{
choiceF("level", []string{"org", "team", "project"}, "team"),
textF("which", "", "Acme, Acme/ml or Acme/ml/training"),
choiceF("role", []string{store.RoleMember, "manager"}, store.RoleMember),
choiceF("role", store.MembershipRoles, store.RoleMember),
},
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
if v[1] == "" {
+4 -2
View File
@@ -391,7 +391,8 @@ func newUserJoinCmd(open userOpener) *cobra.Command {
Long: "A person can be a member of any number of them. A member of a team or\n" +
"organization may use every project in it.",
Example: " llmgw users join ayse@example.com --team Acme/ml\n" +
" llmgw users join ayse@example.com --project Scania/research/vision",
" llmgw users join ayse@example.com --project Scania/research/vision\n" +
" llmgw users join lars@example.com --org Scania --role admin",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
s, err := open(cmd)
@@ -415,7 +416,8 @@ func newUserJoinCmd(open userOpener) *cobra.Command {
},
}
where.bind(cmd)
cmd.Flags().StringVar(&role, "role", store.RoleMember, "member, or manager: with --org, may see the organization's page")
cmd.Flags().StringVar(&role, "role", store.RoleMember,
"member; with --org also manager, who sees the organization's page, or admin, who also changes its teams, projects, people and keys")
cmd.Flags().StringVar(&by, "by", "", "who added them, for the record")
return cmd
}
+7
View File
@@ -158,6 +158,10 @@ type Server struct {
// Firewall turns llmgw's firewall on every machine on or off.
Firewall FirewallSettings `koanf:"firewall"`
// Slurm is a Slurm cluster run beside llmgw on some of its nodes, and
// which of them it may use now.
Slurm SlurmSettings `koanf:"slurm"`
// Presets is how the models that come with llmgw reach the catalog.
Presets PresetSettings `koanf:"presets"`
@@ -960,6 +964,9 @@ func (s Server) Validate() error {
if _, err := s.Facility.Policy(); err != nil {
return err
}
if err := s.Slurm.Check(); err != nil {
return err
}
if s.Control.Timezone != "" {
if _, err := time.LoadLocation(s.Control.Timezone); err != nil {
return fmt.Errorf("control.timezone %q: %w", s.Control.Timezone, err)
+115
View File
@@ -0,0 +1,115 @@
package config
import (
"fmt"
"regexp"
"slices"
)
// SlurmSettings is a Slurm cluster run beside llmgw on some of its nodes
// (BACKLOG P), for training and other batch work llmgw does not schedule.
// The leader tells each named node's agent what to run; the agents install
// it from the machine's package sources and keep it going.
type SlurmSettings struct {
// Enabled runs Slurm on the nodes below. False stops llmgw keeping it;
// what was installed is left installed, and every node is llmgw's.
Enabled bool `koanf:"enabled"`
// ClusterName is Slurm's ClusterName. Empty is "llmgw".
ClusterName string `koanf:"cluster_name"`
// Controller is the llmgw node whose agent runs slurmctld. It needs no
// GPU, and need not be one of Nodes.
Controller string `koanf:"controller"`
// Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained
// in Slurm, until it is also in Lent.
Nodes []string `koanf:"nodes"`
// AllNodes runs slurmd on every llmgw node, those that join later too,
// besides any named in Nodes. Each is llmgw's until it is lent, as any.
AllNodes bool `koanf:"all_nodes"`
// Lent are the nodes of Nodes Slurm may run jobs on now. A node put here
// is drained of llmgw's replicas first, as `ctl drain` does, and resumed
// in Slurm once none is left; a node taken out is drained in Slurm and
// given back to llmgw when its last job ends.
Lent []string `koanf:"lent"`
// Partition is the one partition every node is in. Empty is "gpu".
Partition string `koanf:"partition"`
}
// slurmName is what Slurm takes as a node, cluster or partition name.
var slurmName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]{0,62}$`)
// Cluster is ClusterName or its default.
func (s SlurmSettings) Cluster() string {
if s.ClusterName == "" {
return "llmgw"
}
return s.ClusterName
}
// PartitionName is Partition or its default.
func (s SlurmSettings) PartitionName() string {
if s.Partition == "" {
return "gpu"
}
return s.Partition
}
// IsLent is whether Slurm may use the node now.
func (s SlurmSettings) IsLent(node string) bool {
return s.Enabled && (s.AllNodes || slices.Contains(s.Nodes, node)) && slices.Contains(s.Lent, node)
}
// WithNodes is the settings with every node of ids in Nodes when AllNodes:
// the nodes that run slurmd, as the leader's inventory has them.
func (s SlurmSettings) WithNodes(ids []string) SlurmSettings {
if !s.AllNodes {
return s
}
nodes := slices.Clone(s.Nodes)
for _, id := range ids {
if !slices.Contains(nodes, id) {
nodes = append(nodes, id)
}
}
s.Nodes = nodes
return s
}
// Check refuses settings Slurm could not run on.
func (s SlurmSettings) Check() error {
if !s.Enabled {
return nil
}
if s.Controller == "" {
return fmt.Errorf("slurm.controller is empty: name the node that runs slurmctld")
}
if !slurmName.MatchString(s.Controller) {
return fmt.Errorf("slurm.controller %q is not a name Slurm takes", s.Controller)
}
for _, n := range []struct{ key, v string }{{"slurm.cluster_name", s.ClusterName}, {"slurm.partition", s.Partition}} {
if n.v != "" && !slurmName.MatchString(n.v) {
return fmt.Errorf("%s %q is not a name Slurm takes: letters, digits, _ . -", n.key, n.v)
}
}
seen := map[string]bool{}
for i, n := range s.Nodes {
if !slurmName.MatchString(n) {
return fmt.Errorf("slurm.nodes[%d] %q is not a name Slurm takes", i, n)
}
if seen[n] {
return fmt.Errorf("slurm.nodes names %s twice", n)
}
seen[n] = true
}
for i, n := range s.Lent {
if !seen[n] && !s.AllNodes {
return fmt.Errorf("slurm.lent[%d] %q is not in slurm.nodes: only a node that runs slurmd can be lent", i, n)
}
}
return nil
}
+53
View File
@@ -0,0 +1,53 @@
package config
import (
"strings"
"testing"
)
func TestSlurmCheck(t *testing.T) {
ok := SlurmSettings{Enabled: true, Controller: "node06", Nodes: []string{"cuda01", "site2"}, Lent: []string{"cuda01"}}
if err := ok.Check(); err != nil {
t.Fatal(err)
}
if !ok.IsLent("cuda01") || ok.IsLent("site2") || ok.IsLent("node06") {
t.Error("IsLent is wrong")
}
if ok.Cluster() != "llmgw" || ok.PartitionName() != "gpu" {
t.Error("defaults are wrong")
}
if (SlurmSettings{Lent: []string{"x"}}).Check() != nil {
t.Error("settings that are off were checked")
}
for want, s := range map[string]SlurmSettings{
"controller is empty": {Enabled: true},
"not a name": {Enabled: true, Controller: "bad name"},
"twice": {Enabled: true, Controller: "c", Nodes: []string{"a", "a"}},
"not in slurm.nodes": {Enabled: true, Controller: "c", Nodes: []string{"a"}, Lent: []string{"b"}},
"slurm.partition": {Enabled: true, Controller: "c", Partition: "a b"},
} {
if err := s.Check(); err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("%+v: got %v, want %q", s, err, want)
}
}
if (SlurmSettings{Enabled: false, Lent: []string{"a"}}).IsLent("a") {
t.Error("a node is lent with Slurm off")
}
}
func TestSlurmAllNodes(t *testing.T) {
s := SlurmSettings{Enabled: true, Controller: "c", AllNodes: true, Lent: []string{"joined"}}
if err := s.Check(); err != nil {
t.Fatalf("a node lent with all_nodes refused: %v", err)
}
if !s.IsLent("joined") {
t.Fatal("not lent with all_nodes")
}
if got := s.WithNodes([]string{"a", "joined"}).Nodes; len(got) != 2 {
t.Fatalf("nodes %v", got)
}
s.AllNodes = false
if s.Check() == nil || len(s.WithNodes([]string{"a"}).Nodes) != 0 {
t.Fatal("all_nodes off still takes the inventory's nodes")
}
}
+4 -1
View File
@@ -78,7 +78,9 @@ type API struct {
// tenancy is organizations, keys and people; nil without a store.
tenancy TenancyStore
// train is adapter training; nil without a store and agents.
train *TrainAdmin
train *TrainAdmin
// slurm is the Slurm cluster beside llmgw; nil without a store and agents.
slurm SlurmService
settings *Settings
auth *AuthAdmin
mcp MCPStore
@@ -136,6 +138,7 @@ func (a *API) Register(mux Mux) {
a.registerKB(mux)
a.registerTenancy(mux)
a.registerTrain(mux)
a.registerSlurm(mux)
a.registerGuard(mux)
a.registerSettings(mux)
a.registerAuth(mux)
+8 -9
View File
@@ -216,7 +216,7 @@ func (a *API) purgeFromNodes(ctx context.Context, sources []string, others []cat
names[filepath.Base(s)] = true
}
wanted := inUse(others)
var out []CollectResult
var rs []removal
for _, n := range a.ctl.Inventory().Nodes() {
size := map[string]int64{}
have := make([]string, 0, len(n.Weights))
@@ -235,15 +235,14 @@ func (a *API) purgeFromNodes(ctx context.Context, sources []string, others []cat
if len(paths) == 0 {
continue
}
res := CollectResult{Node: n.ID, Files: len(paths)}
for _, p := range paths {
res.Bytes += size[p]
rs = append(rs, removal{node: n.ID, paths: paths, size: size})
}
out := a.removeConfirmed(ctx, rm, rs)
for _, r := range out {
if r.Error != "" || r.Left > 0 {
slog.Warn("a purged model's copies were not all deleted on a node", "node", r.Node,
"left", r.Left, "err", r.Error)
}
if err := rm.RemoveWeights(ctx, n.ID, paths); err != nil {
res.Error, res.Files, res.Bytes = err.Error(), 0, 0
slog.Warn("a purged model's copies could not be deleted on a node", "node", n.ID, "err", err)
}
out = append(out, res)
}
return out
}
+7
View File
@@ -12,6 +12,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/federation"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
@@ -392,6 +393,12 @@ var Routes = []Route{
Description: "Their keys in projects they no longer reach are revoked.",
Request: MembershipRequest{}, Response: UserView{}},
{Method: "GET", Path: "/api/v1/slurm", Summary: "The Slurm cluster beside llmgw, and where each of its nodes is",
Description: "From the cluster's settings (slurm.*) and what each node's agent last said. A node's phase is llmgw " +
"(drained in Slurm, no job left), lending (its replicas moving off), lent, reclaiming (drained in Slurm, jobs " +
"still running) or unknown (its agent has not said).",
Response: slurmsvc.Status{}},
{Method: "POST", Path: "/api/v1/train/jobs", Summary: "Queue a LoRA adapter's training on a node",
Description: "The node trains when its agent is there, nothing else trains on it, and a card has gpu_memory_mib free; " +
"the pairs are NDJSON of {\"q\", \"a\"}. An adapter learns how answers sound far better than what they say.",
+28
View File
@@ -0,0 +1,28 @@
package adminapi
import (
"net/http"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
)
// SlurmService is the Slurm cluster beside llmgw (BACKLOG P), as the leader
// keeps it.
type SlurmService interface {
Status() slurmsvc.Status
}
// SetSlurm gives the API the Slurm service.
func (a *API) SetSlurm(s SlurmService) { a.slurm = s }
func (a *API) registerSlurm(mux Mux) {
mux.HandleFunc("GET /api/v1/slurm", a.slurmStatus)
}
func (a *API) slurmStatus(w http.ResponseWriter, _ *http.Request) {
if a.slurm == nil {
writeError(w, http.StatusNotImplemented, "Slurm needs the store and the nodes' agents, which this control plane runs without")
return
}
writeJSON(w, http.StatusOK, a.slurm.Status())
}
+6 -2
View File
@@ -634,8 +634,12 @@ func (a *API) membership(add bool) http.HandlerFunc {
if role == "" {
role = store.RoleMember
}
if role != store.RoleMember && role != store.RoleManager {
writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member or manager", role))
if role != store.RoleMember && role != store.RoleManager && role != store.RoleOrgAdmin {
writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member, manager or admin", role))
return
}
if role != store.RoleMember && scope != store.ScopeOrg {
writeError(w, http.StatusBadRequest, fmt.Sprintf("%s is a role on an organization", role))
return
}
if err := a.tenancy.AddMembership(ctx, u.ID, scope, id, role, actorOf(r), time.Now()); err != nil {
+110 -13
View File
@@ -1,11 +1,14 @@
package adminapi
import (
"context"
"encoding/json"
"net/http"
"sort"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
)
// OrphanView is cached weights on one node that no catalog model accounts for.
@@ -71,12 +74,100 @@ type CollectRequest struct {
Node string `json:"node,omitempty"`
}
// CollectResult is what was deleted on one node, or why nothing was.
// CollectResult is what was deleted on one node, or why nothing was: what
// the node no longer lists once asked, not what it was asked to delete.
type CollectResult struct {
Node string `json:"node"`
Files int `json:"files"`
Bytes int64 `json:"bytes"`
Error string `json:"error,omitempty"`
// Left and LeftBytes are what the node still lists: refused, or not
// gone by the time the sweep stopped waiting. Its agent's log says why.
Left int `json:"left,omitempty"`
LeftBytes int64 `json:"left_bytes,omitempty"`
Error string `json:"error,omitempty"`
}
// confirmWait is how long a sweep waits for the nodes to list their files
// without the ones deleted: they scan theirs every heartbeat, five seconds
// by default, and a large set takes a moment to delete.
var confirmWait = 30 * time.Second
// removal is what one node is asked to delete, with each file's size.
type removal struct {
node string
paths []string
size map[string]int64
}
// removeConfirmed asks each node to delete its paths and says what each then
// no longer lists. Counted as asked, a sweep said "25 GB freed" of files the
// node had refused and kept.
func (a *API) removeConfirmed(ctx context.Context, rm control.WeightRemover, rs []removal) []CollectResult {
out := make([]CollectResult, len(rs))
pending := map[int]bool{}
for i, r := range rs {
out[i].Node = r.node
if err := rm.RemoveWeights(ctx, r.node, r.paths); err != nil {
out[i].Error = err.Error()
for _, p := range r.paths {
out[i].Left++
out[i].LeftBytes += r.size[p]
}
continue
}
pending[i] = true
}
deadline := time.Now().Add(confirmWait)
for {
for i := range pending {
if left := a.stillListed(rs[i]); len(left) == 0 {
delete(pending, i)
}
}
if len(pending) == 0 || !time.Now().Before(deadline) {
break
}
select {
case <-ctx.Done():
deadline = time.Now()
case <-time.After(time.Second):
}
}
for i, r := range rs {
if out[i].Error != "" {
continue
}
left := a.stillListed(r)
for _, p := range r.paths {
if left[p] {
out[i].Left++
out[i].LeftBytes += r.size[p]
} else {
out[i].Files++
out[i].Bytes += r.size[p]
}
}
}
return out
}
// stillListed is which of a removal's paths its node still lists.
func (a *API) stillListed(r removal) map[string]bool {
left := map[string]bool{}
n, ok := a.ctl.Inventory().Get(r.node)
if !ok {
return left
}
want := map[string]bool{}
for _, p := range r.paths {
want[p] = true
}
for _, w := range n.Weights {
if want[w.Path] {
left[w.Path] = true
}
}
return left
}
// collectWeights deletes them.
@@ -97,20 +188,26 @@ func (a *API) collectWeights(w http.ResponseWriter, r *http.Request) {
// An empty body is the "all nodes" case and must not be an error.
_ = json.NewDecoder(r.Body).Decode(&body)
out := make([]CollectResult, 0)
// A node that has gone since the list was built is reported rather than
// failing the whole sweep: the other nodes' space is still worth
// freeing, and this one will be offered again next time.
var rs []removal
for _, o := range a.Orphans() {
if body.Node != "" && o.Node != body.Node {
continue
}
res := CollectResult{Node: o.Node, Files: len(o.Paths), Bytes: o.Bytes}
// A node that has gone since the list was built is reported rather than
// failing the whole sweep: the other nodes' space is still worth
// freeing, and this one will be offered again next time.
if err := rm.RemoveWeights(r.Context(), o.Node, o.Paths); err != nil {
res.Error = err.Error()
res.Files, res.Bytes = 0, 0
}
out = append(out, res)
rs = append(rs, removal{node: o.Node, paths: o.Paths, size: a.sizesOn(o.Node)})
}
writeJSON(w, http.StatusOK, out)
writeJSON(w, http.StatusOK, a.removeConfirmed(r.Context(), rm, rs))
}
// sizesOn is each file a node lists, by path, with its size.
func (a *API) sizesOn(node string) map[string]int64 {
out := map[string]int64{}
if n, ok := a.ctl.Inventory().Get(node); ok {
for _, w := range n.Weights {
out[w.Path] = w.SizeBytes
}
}
return out
}
+71
View File
@@ -0,0 +1,71 @@
package adminapi
import (
"context"
"encoding/json"
"log/slog"
"net/http"
"net/http/httptest"
"slices"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
)
// deletingDispatcher deletes what it is asked to as a node would, by its
// files dropping out of what the node lists — but for refused, which it
// keeps, as an agent keeps a file it cannot prove it fetched.
type deletingDispatcher struct {
nullDispatcher
inv *inventory.Store
refused string
}
func (d *deletingDispatcher) RemoveWeights(_ context.Context, node string, paths []string) error {
n, _ := d.inv.Get(node)
n.Weights = slices.DeleteFunc(n.Weights, func(w discovery.LocalWeights) bool {
return w.Path != d.refused && slices.Contains(paths, w.Path)
})
d.inv.Upsert(n)
return nil
}
// What a sweep says it freed is what the node no longer lists: a file it
// kept is said to be left, not counted as freed.
func TestASweepSaysWhatWasDeletedNotWhatWasAsked(t *testing.T) {
confirmWait = 1500 * time.Millisecond
t.Cleanup(func() { confirmWait = 30 * time.Second })
gone := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/old/v1/old.gguf")
set := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/big/v1")
kept := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/stuck/v1/stuck.gguf")
cat, _ := catalog.New(nil)
d := &deletingDispatcher{refused: kept}
ctl := control.New(control.DefaultConfig(), catalog.NewLive(cat), d, nullSink{}, slog.New(slog.DiscardHandler))
d.inv = ctl.Inventory()
node := gpuNode("n1", 1)
node.Weights = []discovery.LocalWeights{{Path: gone, SizeBytes: 1 << 30}, {Path: set, SizeBytes: 20 << 30},
{Path: kept, SizeBytes: 4 << 30}}
ctl.Inventory().Upsert(node)
a := New(ctl)
rec := httptest.NewRecorder()
a.collectWeights(rec, httptest.NewRequest(http.MethodPost, "/api/v1/weights/collect", strings.NewReader(`{"node":"n1"}`)))
var got []CollectResult
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatalf("%v: %s", err, rec.Body.String())
}
if len(got) != 1 {
t.Fatalf("results %+v", got)
}
r := got[0]
if r.Files != 2 || r.Bytes != 21<<30 || r.Left != 1 || r.LeftBytes != 4<<30 {
t.Errorf("result %+v: want 2 files and 21 GB freed, 1 file of 4 GB left", r)
}
}
+11
View File
@@ -84,6 +84,8 @@ type Service struct {
agentSettings func(node string) string
// train is told how training jobs go; see SetTrain.
train TrainHandler
// slurm is told of the nodes' part in Slurm; see SetSlurm.
slurm SlurmHandler
// logWaits are the log requests awaiting a reply, by request id.
logMu sync.Mutex
@@ -308,6 +310,10 @@ func (s *Service) Connect(
return err
}
}
// Its part in Slurm, sent by the Slurm service's next pass.
if h := s.slurmHandler(); h != nil {
h.OnRegister(nodeID)
}
// Recover replica state from what this node is already running, before any
// reconciliation pass can conclude the cluster is empty and place
@@ -515,6 +521,11 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
h.OnTrainStatus(nodeID, m.TrainStatus)
}
case *llmgwv1.AgentMessage_SlurmStatus:
if h := s.slurmHandler(); h != nil {
h.OnSlurmStatus(nodeID, m.SlurmStatus)
}
case *llmgwv1.AgentMessage_TrainArtifact:
if h := s.trainHandler(); h != nil {
h.OnTrainArtifact(nodeID, m.TrainArtifact)
+25
View File
@@ -0,0 +1,25 @@
package agentsvc
import (
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
)
// SlurmHandler is told of the nodes' part in Slurm (BACKLOG P).
type SlurmHandler interface {
OnSlurmStatus(node string, st *llmgwv1.SlurmStatus)
// OnRegister is a node's new stream, which is told again what it is.
OnRegister(node string)
}
// SetSlurm gives the service what Slurm's messages go to.
func (s *Service) SetSlurm(h SlurmHandler) {
s.mu.Lock()
s.slurm = h
s.mu.Unlock()
}
func (s *Service) slurmHandler() SlurmHandler {
s.mu.RLock()
defer s.mu.RUnlock()
return s.slurm
}
+5 -1
View File
@@ -296,7 +296,7 @@ func (s *Server) Start(req Request) (Job, error) {
}
req.sized = req.CtxPool > 0 || req.ContextLength > 0
if req.CtxPool <= 0 {
req.CtxPool = 16384
req.CtxPool = DefaultCtxPool
}
if req.ContextLength <= 0 {
req.ContextLength = 4096
@@ -731,6 +731,10 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
return out
}
// DefaultCtxPool is llama.cpp's pool when a request names none: a context
// asked for longer than it needs a pool of its own.
const DefaultCtxPool = 16384
// llamaArgs size llama.cpp's pool and its slots together. The pool is shared
// by the slots, so slots of the context the catalog promises are as many as
// the pool holds: the engine's default of four turned a 32768 pool into
+5
View File
@@ -43,6 +43,9 @@ type Plan struct {
License string `json:"license,omitempty"`
Params int64 `json:"params,omitempty"`
Context int `json:"context"`
// Trained is the context the model was trained for, the most a request
// may use; 0 when neither its header nor its config says.
Trained int `json:"trained,omitempty"`
// KVBytesPerToken is one token of context, the whole model: what an
// option needs for another context is worked out from it. 0 is unknown.
KVBytesPerToken int64 `json:"kv_bytes_per_token,omitempty"`
@@ -149,6 +152,7 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
p.Servable, p.Reason = false, v.Reason
}
shape.KVBytesPerToken = v.Header.KVBytesPerToken()
p.Trained = int(v.Header.ContextLength) //nolint:gosec // a header's context length
shape.KVHeads = int(v.Header.HeadCountKV) //nolint:gosec // a header's head count
shape.ActiveShare = v.Header.ActiveShare()
if p.Arch == "" {
@@ -167,6 +171,7 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
m, err := modelplan.FetchModel(ctx, repo.ID)
if err == nil {
shape.KVBytesPerToken = m.KVBytesPerToken(modelplan.BF16)
p.Trained = m.MaxContext
shape.KVHeads = m.KVHeads
shape.Experts = m.Experts
if p.Params == 0 {
+51
View File
@@ -154,6 +154,9 @@ func (r Result) Explain() string {
// Scheduler places replicas onto an inventory.
type Scheduler struct {
inv *inventory.Store
// shun is where a replica of the model being placed failed to start;
// set on a copy by Away, never on the scheduler the reconciler shares.
shun []Spot
// now is the clock power readings are judged fresh by; nil is the wall
// clock.
now func() time.Time
@@ -162,6 +165,41 @@ type Scheduler struct {
// New builds a scheduler over an inventory.
func New(inv *inventory.Store) *Scheduler { return &Scheduler{inv: inv} }
// Spot is where a replica ran: a node, and the cards it held there (none
// for one on the node's CPU).
type Spot struct {
Node string
GPUs []int
}
// Away is this scheduler placing nowhere in spots: not those cards, nor a
// CPU replica on those nodes. A copy, so the one shared is not changed.
//
// For a model that failed to start somewhere: the same inputs pick the same
// node and the same card every time, so without it every retry went back to
// where the last one had failed until the model was parked, with another
// card or node free all along.
func (s *Scheduler) Away(spots []Spot) *Scheduler {
c := *s
c.shun = spots
return &c
}
// shunned is the cards of node n a model failed on, and whether it failed
// on the node's CPU.
func (s *Scheduler) shunned(node string) (gpus []int, cpu bool) {
for _, sp := range s.shun {
if sp.Node != node {
continue
}
if len(sp.GPUs) == 0 {
cpu = true
}
gpus = append(gpus, sp.GPUs...)
}
return gpus, cpu
}
// Place picks the first profile, in priority order, that fits.
//
// avoid lists nodes already hosting this model, so replicas of one model spread
@@ -686,7 +724,12 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
}
}
badGPUs, badCPU := s.shunned(n.ID)
if p.CPUOnly() {
if badCPU {
reject("a replica of it failed to start here; trying elsewhere first")
continue
}
// Every ready node can run it; there is no card to be free.
c := candidate{node: n}
if why := budget.check(n, s.shardWatts(p, c)); why != "" {
@@ -703,8 +746,16 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
if g.Class() != p.GPUClass || !slices.Contains(freeIdx, g.Index) {
continue
}
if slices.Contains(badGPUs, g.Index) {
continue
}
free = append(free, g)
}
if len(free) < p.GPUsPerNode && len(badGPUs) > 0 {
reject(fmt.Sprintf("has %d free %s GPUs besides the ones a replica of it failed to start on, needs %d",
len(free), p.GPUClass, p.GPUsPerNode))
continue
}
if len(free) < p.GPUsPerNode {
reject(fmt.Sprintf("has %d free %s GPUs, needs %d", len(free), p.GPUClass, p.GPUsPerNode))
continue
+80
View File
@@ -0,0 +1,80 @@
package reconciler
import (
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
)
// failAndRetry fails the last replica started, and returns where the next
// one is started once the backoff is over.
func failAndRetry(t *testing.T, h *harness) ShardCommand {
t.Helper()
started := h.disp.startedShards()
if len(started) == 0 {
t.Fatal("nothing was started")
}
s := started[len(started)-1]
h.r.OnShardStatus(s.ReplicaID, s.ShardID, supervisor.Failed, "the engine exited")
h.r.Reconcile(t.Context()) // reaps it
h.disp.reset()
h.advance(DefaultConfig().MaxBackoff)
h.r.Reconcile(t.Context())
again := h.disp.startedShards()
if len(again) == 0 {
t.Fatal("it was not tried again")
}
return again[0]
}
func oneCard() catalog.Model {
return testModel(1, catalog.Profile{Name: "one", GPUClass: "h100-80g", GPUsPerNode: 1})
}
func TestAReplicaThatFailedIsTriedOnAnotherNode(t *testing.T) {
// The same inputs give the same node every time: an on-demand 3D model
// that failed to load was started on that node again, and again, until
// it was parked, with another node free all along.
h := newHarness(t, oneCard(), gpuNode("1", 1), gpuNode("2", 1))
h.r.Reconcile(t.Context())
first := h.disp.startedShards()[0].NodeID
if next := failAndRetry(t, h); next.NodeID == first {
t.Errorf("tried on node %s again, where it had just failed", first)
}
}
func TestAReplicaThatFailedIsTriedOnAnotherCardOfTheNode(t *testing.T) {
h := newHarness(t, oneCard(), gpuNode("1", 2))
h.r.Reconcile(t.Context())
first := h.disp.startedShards()[0].Spec.GPUIndices
next := failAndRetry(t, h)
if len(next.Spec.GPUIndices) != 1 || len(first) != 1 || next.Spec.GPUIndices[0] == first[0] {
t.Errorf("tried on card %v again, where it had just failed", next.Spec.GPUIndices)
}
}
func TestWithNowhereElseItIsTriedWhereItFailed(t *testing.T) {
// A preference, not a rule: a model that has one place is not left
// unplaced because it failed there once.
h := newHarness(t, oneCard(), gpuNode("1", 1))
h.r.Reconcile(t.Context())
if next := failAndRetry(t, h); next.NodeID != "1" {
t.Errorf("started on %q", next.NodeID)
}
}
func TestComingUpForgetsWhereItFailed(t *testing.T) {
h := newHarness(t, oneCard(), gpuNode("1", 1), gpuNode("2", 1))
h.r.Reconcile(t.Context())
next := failAndRetry(t, h)
h.r.OnShardStatus(next.ReplicaID, next.ShardID, supervisor.Ready, "")
h.r.mu.Lock()
left := len(h.r.failedAt["llama"])
h.r.mu.Unlock()
if left != 0 {
t.Errorf("a model that came up still remembers %d failures' places", left)
}
}
+1
View File
@@ -61,6 +61,7 @@ func (r *Reconciler) Retry(model string) bool {
delete(r.parked, model)
r.attempts[model] = 0
delete(r.nextAttempt, model)
delete(r.failedAt, model)
r.mu.Unlock()
r.Notify()
return was
+77 -1
View File
@@ -7,6 +7,8 @@ import (
"log/slog"
"math/rand/v2"
"slices"
"strconv"
"strings"
"sync"
"sync/atomic"
"time"
@@ -251,6 +253,9 @@ type Reconciler struct {
nextAttempt map[string]time.Time
// attempts counts consecutive failures per model, for the backoff.
attempts map[string]int
// failedAt is where a model's replicas failed to start since it last
// came up, most recent last: the next one is placed elsewhere if it can.
failedAt map[string][]placement.Spot
// lastExplain keeps the most recent placement failure per model, so
// `ctl explain-placement` can answer without re-running placement.
lastExplain map[string]placement.Result
@@ -334,6 +339,7 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
replicas: map[string]*Replica{},
nextAttempt: map[string]time.Time{},
attempts: map[string]int{},
failedAt: map[string][]placement.Spot{},
reservedPorts: map[int]struct{}{},
wanted: map[string]time.Time{},
coldStart: map[string]time.Duration{},
@@ -726,6 +732,7 @@ func (r *Reconciler) reapFailed(ctx context.Context) {
// replica is about to disappear, and the backoff has to survive it.
r.attempts[rep.Model]++
r.nextAttempt[rep.Model] = r.now().Add(r.backoff(r.attempts[rep.Model]))
r.noteFailedAt(rep)
if n := r.attempts[rep.Model]; n >= r.cfg.ParkAfter {
if _, already := r.parked[rep.Model]; !already {
m, _ := r.cat.Load().Resolve(rep.Model)
@@ -875,6 +882,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
delete(r.parked, m.Name)
r.attempts[m.Name] = 0
delete(r.nextAttempt, m.Name)
delete(r.failedAt, m.Name)
isParked, blockedUntil = false, time.Time{}
r.log.Info("model unparked: its definition changed", "model", m.Name)
}
@@ -1002,7 +1010,29 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
return notTried
}
}
res := r.sched.Place(profiles, avoid)
// Away from where it last failed to start, when there is anywhere else:
// the same inputs give the same node and card, and an on-demand model
// that failed there was otherwise started there again until it was
// parked. Only a preference too: nowhere else, and it is tried there.
r.mu.Lock()
failed := slices.Clone(r.failedAt[m.Name])
r.mu.Unlock()
var res placement.Result
if len(failed) > 0 {
away := r.sched.Away(failed)
res = away.Place(profiles, avoid)
if res.Placement == nil && len(avoid) > 0 {
res = away.Place(profiles, nil)
}
pspan.SetAttributes(attribute.Bool("llmgw.away_from_failure", res.Placement != nil))
if res.Placement != nil {
r.log.Info("placing a model away from where it failed to start", "model", m.Name,
"failed_on", spotsText(failed), "now_on", spotsText(spotsOf(res.Placement)))
}
}
if res.Placement == nil {
res = r.sched.Place(profiles, avoid)
}
if res.Placement == nil && len(avoid) > 0 {
// Spreading across failure domains is a preference, not a rule. If it
// is the only thing standing in the way, take the placement anyway:
@@ -1054,6 +1084,51 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
return placed
}
// maxFailedSpots is how many places a model's failures are remembered at.
const maxFailedSpots = 8
// noteFailedAt remembers where a failed replica ran. Called under r.mu.
func (r *Reconciler) noteFailedAt(rep *Replica) {
spots := r.failedAt[rep.Model]
for _, sh := range rep.Shards {
sp := placement.Spot{Node: sh.NodeID, GPUs: slices.Clone(sh.GPUIndices)}
spots = slices.DeleteFunc(spots, func(o placement.Spot) bool {
return o.Node == sp.Node && slices.Equal(o.GPUs, sp.GPUs)
})
spots = append(spots, sp)
}
if len(spots) > maxFailedSpots {
spots = spots[len(spots)-maxFailedSpots:]
}
r.failedAt[rep.Model] = spots
}
// spotsOf is where a placement puts its shards.
func spotsOf(pl *placement.Placement) []placement.Spot {
var out []placement.Spot
for _, n := range pl.Nodes {
out = append(out, placement.Spot{Node: n.NodeID, GPUs: n.GPUIndices})
}
return out
}
// spotsText is spots as a log reads them: "node01 gpu 0,1; node02 cpu".
func spotsText(spots []placement.Spot) string {
parts := make([]string, 0, len(spots))
for _, sp := range spots {
if len(sp.GPUs) == 0 {
parts = append(parts, sp.Node+" cpu")
continue
}
gs := make([]string, len(sp.GPUs))
for i, g := range sp.GPUs {
gs[i] = strconv.Itoa(g)
}
parts = append(parts, sp.Node+" gpu "+strings.Join(gs, ","))
}
return strings.Join(parts, "; ")
}
// scaleOutcome is what one scaleUp did.
type scaleOutcome int
@@ -1433,6 +1508,7 @@ func (r *Reconciler) markShard(replicaID, shardID string, st supervisor.State, e
// counting from scratch rather than from an old streak.
r.attempts[rep.Model] = 0
delete(r.nextAttempt, rep.Model)
delete(r.failedAt, rep.Model)
if !rep.CreatedAt.IsZero() {
r.coldStart[rep.Model] = r.now().Sub(rep.CreatedAt)
}
+133
View File
@@ -0,0 +1,133 @@
package slurmsvc
import (
"context"
"errors"
"strings"
"testing"
"time"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// fakeKV is the store's cluster state.
type fakeKV struct {
m map[string]string
err error
}
func (f *fakeKV) PutClusterState(_ context.Context, k, v string) (bool, error) {
if f.err != nil {
return false, f.err
}
f.m[k] = v
return true, nil
}
func (f *fakeKV) ClusterState(_ context.Context, k string) (string, error) {
if f.err != nil {
return "", f.err
}
v, ok := f.m[k]
if !ok {
return "", store.ErrNotFound
}
return v, nil
}
// takeOver is the leader that comes next: the same settings and store, an
// inventory with only the nodes connected to it so far.
func takeOver(r *rig, kv *fakeKV, connected ...string) *rig {
n := &rig{set: r.set, inv: &fakeInv{nodes: map[string]*inventory.Node{}, allocs: map[string]int{}},
agents: &fakeAgents{conn: connected, got: map[string][]*llmgwv1.Slurm{}}}
for _, c := range connected {
node := *r.inv.nodes[c]
node.State = inventory.Ready
n.inv.nodes[c] = &node
}
n.svc = New(func() config.SlurmSettings { return n.set }, n.inv, n.agents,
func(context.Context) ([]byte, error) { return []byte(strings.Repeat("k", 1024)), nil }, nil)
n.svc.SetMemory(StoreMemory(kv))
return n
}
func TestANewLeaderKeepsTheNodesItHasNotHeardFromInTheControllersFiles(t *testing.T) {
kv := &fakeKV{m: map[string]string{}}
r := newRig()
r.svc.SetMemory(StoreMemory(kv))
r.set.Lent = []string{"cuda01"}
r.inv.allocs = map[string]int{}
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "MIXED", Jobs: 1})
r.tick()
want := r.agents.last("node06").GetFiles()["slurm.conf"]
if !strings.Contains(want, "NodeName=cuda01") || !strings.Contains(want, "Gres=gpu:rtx3090:1") {
t.Fatalf("slurm.conf:\n%s", want)
}
// The leader goes; the next hears from the controller first.
n := takeOver(r, kv, "node06")
n.tick()
if got := n.agents.last("node06").GetFiles()["slurm.conf"]; got != want {
t.Fatalf("the new leader's slurm.conf differs; the controller would drop the node and its job:\n%s\nwas:\n%s",
got, want)
}
}
func TestANodeIsToldNothingUntilItsControllerIsKnown(t *testing.T) {
r := newRig()
r.svc.SetMemory(StoreMemory(&fakeKV{m: map[string]string{}}))
delete(r.inv.nodes, "node06")
r.agents.conn = []string{"cuda01"}
r.tick()
if m := r.agents.last("cuda01"); m != nil {
t.Fatalf("a node was told %v before its controller was known; an empty part stops its slurmd and its jobs", m)
}
}
func TestANewLeaderTellsANodeItsControllerFromTheStore(t *testing.T) {
kv := &fakeKV{m: map[string]string{}}
r := newRig()
r.svc.SetMemory(StoreMemory(kv))
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
r.tick()
n := takeOver(r, kv, "cuda01")
n.tick()
if m := n.agents.last("cuda01"); !m.GetNode() || m.GetConfServer() != "10.0.0.146:6817" {
t.Fatalf("the node was told %v", m)
}
}
func TestWithTheStoreUnreadTheControllerIsSentNothingThatLeavesANodeOut(t *testing.T) {
kv := &fakeKV{m: map[string]string{}, err: errors.New("store down")}
r := newRig()
r.svc.SetMemory(StoreMemory(kv))
r.tick()
if m := r.agents.last("node06"); m != nil {
t.Fatalf("the controller was sent %v without cuda01, with the store unread", m)
}
// Once every node has reported, nothing is left out: sent.
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
r.tick()
if !strings.Contains(r.agents.last("node06").GetFiles()["slurm.conf"], "NodeName=cuda01") {
t.Fatal("with every node reported, the controller still got no files")
}
}
func TestWithNothingKeptANewLeaderWaitsForTheNodesAWhile(t *testing.T) {
r := newRig()
r.svc.SetMemory(StoreMemory(&fakeKV{m: map[string]string{}}))
r.tick()
if m := r.agents.last("node06"); m != nil {
t.Fatalf("with nothing kept, the controller was sent %v before cuda01 reported", m)
}
defer func(d time.Duration) { HoldFor = d }(HoldFor)
HoldFor = 0
r.tick()
if m := r.agents.last("node06"); !m.GetController() || strings.Contains(m.GetFiles()["slurm.conf"], "NodeName=cuda01") {
t.Fatalf("after the wait, the controller was told %v", m)
}
}
+77
View File
@@ -0,0 +1,77 @@
package slurmsvc
import (
"context"
"encoding/json"
"errors"
"slices"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// Memory keeps the nodes slurm.conf last held, and the controller's
// address, where a leader that takes over finds them. Without it a new
// leader's first slurm.conf has none of the nodes it has not heard from
// yet, and the controller, restarted on it, drops them and the jobs
// running there.
type Memory interface {
Recall(ctx context.Context) (Kept, error)
Keep(ctx context.Context, k Kept) error
}
// Kept is what Memory holds.
type Kept struct {
Controller string `json:"controller,omitempty"`
ControllerAddr string `json:"controller_addr,omitempty"`
Nodes map[string]slurm.Node `json:"nodes,omitempty"`
}
func (k Kept) equal(o Kept) bool {
if k.Controller != o.Controller || k.ControllerAddr != o.ControllerAddr || len(k.Nodes) != len(o.Nodes) {
return false
}
for name, n := range k.Nodes {
m, ok := o.Nodes[name]
if !ok || n.Addr != m.Addr || n.Hardware != m.Hardware || !slices.Equal(n.GPUs, m.GPUs) {
return false
}
}
return true
}
// KV is the store's cluster state.
type KV interface {
PutClusterState(ctx context.Context, key, value string) (bool, error)
ClusterState(ctx context.Context, key string) (string, error)
}
// keptKey is the cluster state key it is kept under.
const keptKey = "slurm-nodes"
// StoreMemory keeps it in the store's cluster state.
func StoreMemory(kv KV) Memory { return kvMemory{kv} }
type kvMemory struct{ kv KV }
func (m kvMemory) Recall(ctx context.Context) (Kept, error) {
var k Kept
v, err := m.kv.ClusterState(ctx, keptKey)
if errors.Is(err, store.ErrNotFound) {
return k, nil
}
if err != nil {
return k, err
}
err = json.Unmarshal([]byte(v), &k)
return k, err
}
func (m kvMemory) Keep(ctx context.Context, k Kept) error {
b, err := json.Marshal(k)
if err != nil {
return err
}
_, err = m.kv.PutClusterState(ctx, keptKey, string(b))
return err
}
+534
View File
@@ -0,0 +1,534 @@
// Package slurmsvc is the leader's side of the Slurm cluster run beside
// llmgw (BACKLOG P): it tells each node's agent what it is, writes the
// controller's files from what the nodes report, and hands nodes between
// llmgw and Slurm as the cluster's settings say.
//
// A node in slurm.lent is drained of llmgw's replicas, as `ctl drain` does,
// and lent to Slurm once none is left. A node taken out of slurm.lent is
// drained in Slurm, and is llmgw's again when its last job has ended. Both
// are kept in force every tick, from the settings and what the agents say,
// so a leader that takes over carries on where the last one was.
package slurmsvc
import (
"context"
"fmt"
"log/slog"
"maps"
"net"
"slices"
"strconv"
"strings"
"sync"
"time"
"google.golang.org/protobuf/proto"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
)
// Agents is how the service reaches the nodes.
type Agents interface {
Send(node string, msg *llmgwv1.ControlMessage) error
ConnectedNodes() []string
}
// Inventory is what the service reads and changes of llmgw's nodes.
type Inventory interface {
Get(id string) (inventory.Node, bool)
SetState(id string, st inventory.NodeState) bool
Allocations(nodeID string) []inventory.Allocation
Nodes() []inventory.Node
}
// Phase is where a node of slurm.nodes is between llmgw and Slurm.
type Phase string
const (
// Llmgw: drained in Slurm with no job left, llmgw's to place on.
Llmgw Phase = "llmgw"
// Lending: in slurm.lent, its replicas being moved off.
Lending Phase = "lending"
// Lent: Slurm's; llmgw places nothing there.
Lent Phase = "lent"
// Reclaiming: out of slurm.lent, drained in Slurm, jobs still running.
Reclaiming Phase = "reclaiming"
// Unknown: its agent has not said how it is in Slurm yet.
Unknown Phase = "unknown"
)
// NodeStatus is one node as the service sees it.
type NodeStatus struct {
Node string `json:"node"`
Phase Phase `json:"phase"`
Replicas int `json:"replicas"`
Status *Reported `json:"slurm,omitempty"`
Connected bool `json:"connected"`
}
// Reported is the agent's SlurmStatus, for the admin API.
type Reported struct {
Version string `json:"version,omitempty"`
Hardware string `json:"hardware,omitempty"`
SlurmdRunning bool `json:"slurmd_running"`
SlurmctldRunning bool `json:"slurmctld_running"`
State string `json:"state,omitempty"`
Reason string `json:"reason,omitempty"`
Jobs int `json:"jobs"`
Error string `json:"error,omitempty"`
FilesSHA256 string `json:"files_sha256,omitempty"`
At time.Time `json:"at"`
}
// Status is the whole Slurm cluster, for `ctl slurm status`.
type Status struct {
Enabled bool `json:"enabled"`
Cluster string `json:"cluster"`
Controller string `json:"controller"`
Partition string `json:"partition"`
Ctld *Reported `json:"controller_status,omitempty"`
Nodes []NodeStatus `json:"nodes"`
// Waiting are nodes left out of slurm.conf until they report their
// hardware.
Waiting []string `json:"waiting,omitempty"`
}
// Service keeps the Slurm cluster.
type Service struct {
settings func() config.SlurmSettings
inv Inventory
agents Agents
key func(ctx context.Context) ([]byte, error)
log *slog.Logger
mu sync.Mutex
// status is each node's last SlurmStatus.
status map[string]*Reported
// drained are nodes this service put in Draining, which it alone makes
// ready again.
drained map[string]bool
// sent is what each node was last told, so it is told again only when
// that changes or its stream is new.
sent map[string]*llmgwv1.Slurm
mungeKy []byte
kick chan struct{}
// mem keeps slurm.conf's nodes and the controller's address for the
// next leader; known is what it held when read, and what was kept
// since. loaded is false until it was read: until then the controller
// is sent no files that leave out a node, which would drop that node
// and its jobs.
mem Memory
known Kept
loaded bool
// started is the first pass, from which a node neither heard from nor
// kept is waited for, for HoldFor.
started time.Time
}
// HoldFor is how long a new leader waits for a node it neither has heard
// from nor finds kept before it writes the controller's files without it:
// on a cluster that kept nothing yet (one upgraded to this), the nodes
// report within seconds of a new leader, and the controller keeps the
// node and its jobs meanwhile.
var HoldFor = 2 * time.Minute
// New is a Service; key gives the munge key, made once from a cluster
// secret.
func New(settings func() config.SlurmSettings, inv Inventory, agents Agents,
key func(ctx context.Context) ([]byte, error), log *slog.Logger) *Service {
if log == nil {
log = slog.Default()
}
return &Service{settings: settings, inv: inv, agents: agents, key: key, log: log,
status: map[string]*Reported{}, drained: map[string]bool{}, sent: map[string]*llmgwv1.Slurm{},
kick: make(chan struct{}, 1)}
}
// SetMemory has slurm.conf's nodes kept in mem, and read from it first.
func (s *Service) SetMemory(mem Memory) {
s.mu.Lock()
s.mem = mem
s.mu.Unlock()
}
// current is the settings, slurm.all_nodes resolved to the nodes the
// inventory has.
func (s *Service) current() config.SlurmSettings {
set := s.settings()
if !set.AllNodes {
return set
}
var ids []string
for _, n := range s.inv.Nodes() {
ids = append(ids, n.ID)
}
return set.WithNodes(ids)
}
// Every is how often the service looks again without being asked.
var Every = 10 * time.Second
// Run keeps the cluster until ctx ends.
func (s *Service) Run(ctx context.Context) {
t := time.NewTicker(Every)
defer t.Stop()
for {
s.Tick(ctx)
select {
case <-ctx.Done():
return
case <-t.C:
case <-s.kick:
}
}
}
// Kick has the service look again now: the settings changed.
func (s *Service) Kick() {
select {
case s.kick <- struct{}{}:
default:
}
}
// OnSlurmStatus records what a node's agent says.
func (s *Service) OnSlurmStatus(node string, st *llmgwv1.SlurmStatus) {
r := &Reported{Version: st.GetVersion(), Hardware: st.GetHardware(), SlurmdRunning: st.GetSlurmdRunning(),
SlurmctldRunning: st.GetSlurmctldRunning(), State: st.GetState(), Reason: st.GetReason(),
Jobs: int(st.GetJobs()), Error: st.GetError(), FilesSHA256: st.GetFilesSha256(), At: time.Now()}
s.mu.Lock()
prev := s.status[node]
s.status[node] = r
s.mu.Unlock()
if r.Error != "" && (prev == nil || prev.Error != r.Error) {
s.log.Warn("a node could not keep its part in Slurm", "node", node, "err", r.Error)
}
if prev == nil || prev.Hardware != r.Hardware || prev.State != r.State || prev.Jobs != r.Jobs {
s.Kick()
}
}
// OnRegister is a node's new stream: it is told again what it is.
func (s *Service) OnRegister(node string) {
s.mu.Lock()
delete(s.sent, node)
s.mu.Unlock()
s.Kick()
}
// Tick does one pass: the nodes handed over as far as they can be, and
// each told what it is.
func (s *Service) Tick(ctx context.Context) {
set := s.current()
s.mu.Lock()
defer s.mu.Unlock()
s.recall(ctx)
if s.started.IsZero() {
s.started = time.Now()
}
phases := s.handOver(set)
var key []byte
if set.Enabled {
if s.mungeKy == nil {
k, err := s.key(ctx)
if err != nil {
s.log.Warn("no munge key for Slurm; nothing is sent to the nodes", "err", err)
return
}
s.mungeKy = k
}
key = s.mungeKy
}
files, waiting := s.files(set)
// Not yet knowing what the last leader knew, a slurm.conf without some
// node would take it, and its jobs, from the controller: kept back.
hold := s.mem != nil && len(waiting) > 0 && (!s.loaded || time.Since(s.started) < HoldFor)
for _, node := range s.agents.ConnectedNodes() {
if hold && node == set.Controller {
continue
}
m := s.message(set, node, phases[node] == Lent, key, files)
if m == nil {
continue
}
if old, ok := s.sent[node]; ok && proto.Equal(old, m) {
continue
}
if err := s.agents.Send(node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_Slurm{Slurm: m}}); err != nil {
s.log.Warn("could not tell a node its part in Slurm", "node", node, "err", err)
continue
}
s.sent[node] = m
}
s.remember(ctx, set)
}
// recall reads what the last leader kept, once. Under s.mu.
func (s *Service) recall(ctx context.Context) {
if s.mem == nil || s.loaded {
return
}
k, err := s.mem.Recall(ctx)
if err != nil {
s.log.Warn("could not read Slurm's nodes from the store; the controller's files wait for every node",
"err", err)
return
}
s.known, s.loaded = k, true
}
// remember keeps slurm.conf's nodes and the controller's address when they
// changed. Under s.mu.
func (s *Service) remember(ctx context.Context, set config.SlurmSettings) {
if s.mem == nil || !s.loaded || !set.Enabled {
return
}
c, _ := s.cluster(set)
now := Kept{Controller: set.Controller, ControllerAddr: c.ControllerAddr, Nodes: map[string]slurm.Node{}}
for _, n := range c.Nodes {
now.Nodes[n.Name] = n
}
if now.equal(s.known) {
return
}
if err := s.mem.Keep(ctx, now); err != nil {
s.log.Warn("could not keep Slurm's nodes in the store", "err", err)
return
}
s.known = now
}
// controllerAddr is where the nodes reach slurmctld: the controller's
// management address, else the one kept. Under s.mu.
func (s *Service) controllerAddr(set config.SlurmSettings) string {
if n, ok := s.inv.Get(set.Controller); ok && n.MgmtAddr != "" {
return n.MgmtAddr
}
if s.known.Controller == set.Controller {
return s.known.ControllerAddr
}
return ""
}
// handOver moves each node of slurm.nodes a step between llmgw and Slurm,
// answering where each is. Under s.mu.
func (s *Service) handOver(set config.SlurmSettings) map[string]Phase {
out := map[string]Phase{}
if !set.Enabled {
// Off: every node is llmgw's again. What Slurm still runs is left
// to finish where it is; turning Slurm off with work lent is the
// operator's choice.
for node := range s.drained {
s.ready(node)
}
return out
}
for _, node := range set.Nodes {
out[node] = s.phase(set, node)
}
// A node no longer in slurm.nodes is llmgw's.
for node := range s.drained {
if !slices.Contains(set.Nodes, node) {
s.ready(node)
}
}
return out
}
func (s *Service) phase(set config.SlurmSettings, node string) Phase {
n, ok := s.inv.Get(node)
st := s.status[node]
if set.IsLent(node) {
if ok && (n.State == inventory.Ready || n.State == inventory.Cordoned) {
s.inv.SetState(node, inventory.Draining)
s.drained[node] = true
s.log.Info("lending a node to Slurm: its replicas are moved off first", "node", node)
}
if !ok || len(s.inv.Allocations(node)) > 0 {
return Lending
}
return Lent
}
// Not lent. Until the agent says Slurm has drained it with no job left,
// llmgw places nothing new on it: a new leader starts with every node
// ready. Cordoned, not drained: llmgw has nothing there to move.
if st == nil || slurmBusy(st) {
if ok && n.State == inventory.Ready {
s.inv.SetState(node, inventory.Cordoned)
s.drained[node] = true
if st != nil {
s.log.Info("taking a node back from Slurm: llmgw places nothing there until its jobs end",
"node", node, "jobs", st.Jobs)
}
}
if st == nil {
return Unknown
}
return Reclaiming
}
if s.drained[node] {
s.ready(node)
}
return Llmgw
}
// slurmBusy is whether Slurm may still run work on a node: a job step left
// on it (or no telling), or a controller that would still give it one. A
// node the controller does not know of, or cannot be asked about, gets no
// job from it, so a Slurm that is broken never keeps a node from llmgw.
func slurmBusy(st *Reported) bool {
if st.Jobs != 0 {
return true
}
if st.State == "" {
return false
}
return !strings.Contains(st.State, "DRAIN") && !strings.Contains(st.State, "DOWN")
}
func (s *Service) ready(node string) {
if n, ok := s.inv.Get(node); ok && (n.State == inventory.Draining || n.State == inventory.Cordoned) {
s.inv.SetState(node, inventory.Ready)
s.log.Info("a node is llmgw's again: Slurm has no job there", "node", node)
}
delete(s.drained, node)
}
// files are the controller's files, and the nodes left out of them for not
// having reported their hardware. Under s.mu.
func (s *Service) files(set config.SlurmSettings) (map[string]string, []string) {
c, waiting := s.cluster(set)
if c.ControllerAddr == "" {
return nil, waiting
}
return c.Files(), waiting
}
// cluster is what slurm.conf says: each node as it last reported, else as
// kept from before, and the nodes left out for having neither. Under s.mu.
func (s *Service) cluster(set config.SlurmSettings) (slurm.Cluster, []string) {
c := slurm.Cluster{Name: set.Cluster(), Controller: set.Controller, Partition: set.PartitionName()}
if !set.Enabled {
return c, nil
}
c.ControllerAddr = s.controllerAddr(set)
var waiting []string
for _, node := range set.Nodes {
if sn, ok := s.reported(node); ok {
c.Nodes = append(c.Nodes, sn)
} else if sn, ok := s.known.Nodes[node]; ok {
c.Nodes = append(c.Nodes, sn)
} else {
waiting = append(waiting, node)
}
}
return c, waiting
}
// reported is a node as its agent and the inventory say now. Under s.mu.
func (s *Service) reported(node string) (slurm.Node, bool) {
n, ok := s.inv.Get(node)
st := s.status[node]
if !ok || n.MgmtAddr == "" || st == nil || st.Hardware == "" {
return slurm.Node{}, false
}
hw, err := slurm.ParseHardware(st.Hardware)
if err != nil {
return slurm.Node{}, false
}
sn := slurm.Node{Name: node, Addr: n.MgmtAddr, Hardware: hw}
for _, g := range n.GPUs {
sn.GPUs = append(sn.GPUs, slurm.GPU{Minor: g.Index, Name: g.Name})
}
return sn, true
}
// message is what a node is told. Under s.mu.
func (s *Service) message(set config.SlurmSettings, node string, lent bool, key []byte,
files map[string]string) *llmgwv1.Slurm {
m := &llmgwv1.Slurm{}
if !set.Enabled {
return m
}
isNode := slices.Contains(set.Nodes, node)
isCtl := set.Controller == node && files != nil
if !isNode && !isCtl {
return m
}
m.Name, m.MungeKey, m.Lent = node, key, lent
m.Node = isNode
if addr := s.controllerAddr(set); addr != "" {
m.ConfServer = net.JoinHostPort(addr, strconv.Itoa(slurm.CtldPort))
}
if isNode && m.ConfServer == "" {
// Without its controller a node cannot start; it is told when the
// controller registers. Told nothing until then, not "no part": a
// node still running Slurm's jobs would stop them.
return nil
}
if isCtl {
m.Controller = true
m.Files = maps.Clone(files)
}
return m
}
// Status is the cluster as the service sees it now.
func (s *Service) Status() Status {
set := s.current()
s.mu.Lock()
defer s.mu.Unlock()
out := Status{Enabled: set.Enabled, Cluster: set.Cluster(), Controller: set.Controller,
Partition: set.PartitionName(), Ctld: s.status[set.Controller]}
if !set.Enabled {
return out
}
_, out.Waiting = s.files(set)
conn := s.agents.ConnectedNodes()
for _, node := range set.Nodes {
ns := NodeStatus{Node: node, Status: s.status[node], Replicas: len(s.inv.Allocations(node)),
Connected: slices.Contains(conn, node)}
ns.Phase = s.peek(set, node)
out.Nodes = append(out.Nodes, ns)
}
return out
}
// peek is phase without changing anything. Under s.mu.
func (s *Service) peek(set config.SlurmSettings, node string) Phase {
st := s.status[node]
if set.IsLent(node) {
if len(s.inv.Allocations(node)) > 0 {
return Lending
}
return Lent
}
switch {
case st == nil:
return Unknown
case slurmBusy(st):
return Reclaiming
}
return Llmgw
}
// Describe is one line for a log or a screen.
func (n NodeStatus) Describe() string {
switch n.Phase {
case Lending:
return fmt.Sprintf("being lent: %d replica(s) still to move off", n.Replicas)
case Lent:
return "lent to Slurm"
case Reclaiming:
if n.Status != nil {
return fmt.Sprintf("being taken back: %d job(s) still running", n.Status.Jobs)
}
return "being taken back"
case Llmgw:
return "llmgw's"
}
return "not heard from yet"
}
+274
View File
@@ -0,0 +1,274 @@
package slurmsvc
import (
"context"
"slices"
"strings"
"testing"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
)
type fakeInv struct {
nodes map[string]*inventory.Node
allocs map[string]int
}
func (f *fakeInv) Get(id string) (inventory.Node, bool) {
n, ok := f.nodes[id]
if !ok {
return inventory.Node{}, false
}
return *n, true
}
func (f *fakeInv) SetState(id string, st inventory.NodeState) bool {
n, ok := f.nodes[id]
if ok {
n.State = st
}
return ok
}
func (f *fakeInv) Allocations(id string) []inventory.Allocation {
return make([]inventory.Allocation, f.allocs[id])
}
func (f *fakeInv) Nodes() []inventory.Node {
var out []inventory.Node
for _, n := range f.nodes {
out = append(out, *n)
}
slices.SortFunc(out, func(a, b inventory.Node) int { return strings.Compare(a.ID, b.ID) })
return out
}
type fakeAgents struct {
conn []string
got map[string][]*llmgwv1.Slurm
}
func (f *fakeAgents) Send(node string, m *llmgwv1.ControlMessage) error {
f.got[node] = append(f.got[node], m.GetSlurm())
return nil
}
func (f *fakeAgents) ConnectedNodes() []string { return f.conn }
func (f *fakeAgents) last(node string) *llmgwv1.Slurm {
ms := f.got[node]
if len(ms) == 0 {
return nil
}
return ms[len(ms)-1]
}
type rig struct {
set config.SlurmSettings
inv *fakeInv
agents *fakeAgents
svc *Service
}
func newRig() *rig {
r := &rig{
set: config.SlurmSettings{Enabled: true, Controller: "node06", Nodes: []string{"cuda01"}},
inv: &fakeInv{nodes: map[string]*inventory.Node{
"node06": {ID: "node06", MgmtAddr: "10.0.0.146"},
"cuda01": {ID: "cuda01", MgmtAddr: "10.0.0.147", GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA GeForce RTX 3090"}}},
}, allocs: map[string]int{"cuda01": 2}},
agents: &fakeAgents{conn: []string{"node06", "cuda01"}, got: map[string][]*llmgwv1.Slurm{}},
}
r.svc = New(func() config.SlurmSettings { return r.set }, r.inv, r.agents,
func(context.Context) ([]byte, error) { return []byte(strings.Repeat("k", 1024)), nil }, nil)
return r
}
func (r *rig) report(node string, st *llmgwv1.SlurmStatus) { r.svc.OnSlurmStatus(node, st) }
func (r *rig) tick() { r.svc.Tick(context.Background()) }
const hw = "NodeName=cuda01 CPUs=16 RealMemory=64000"
func TestMessagesAndFiles(t *testing.T) {
r := newRig()
r.tick()
ctl, node := r.agents.last("node06"), r.agents.last("cuda01")
if !ctl.GetController() || ctl.GetNode() || len(ctl.GetMungeKey()) != 1024 {
t.Fatalf("controller told %v", ctl)
}
if !node.GetNode() || node.GetController() || node.GetConfServer() != "10.0.0.146:6817" || node.GetLent() ||
node.GetName() != "cuda01" || len(node.GetFiles()) != 0 {
t.Fatalf("node told %v", node)
}
if strings.Contains(ctl.GetFiles()["slurm.conf"], "NodeName=cuda01") {
t.Fatal("a node with no hardware reported is in slurm.conf")
}
if w := r.svc.Status().Waiting; len(w) != 1 || w[0] != "cuda01" {
t.Fatalf("waiting %v", w)
}
// Same again: nobody is told twice.
r.tick()
if len(r.agents.got["node06"]) != 1 || len(r.agents.got["cuda01"]) != 1 {
t.Fatal("told again with nothing changed")
}
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
r.tick()
conf := r.agents.last("node06").GetFiles()["slurm.conf"]
if !strings.Contains(conf, "NodeName=cuda01 NodeAddr=10.0.0.147 CPUs=16 RealMemory=62080 Gres=gpu:rtx3090:1") {
t.Fatalf("slurm.conf:\n%s", conf)
}
if !strings.Contains(r.agents.last("node06").GetFiles()["gres.conf"], "File=/dev/nvidia0") {
t.Fatal("gres.conf has no card")
}
if len(r.agents.got["cuda01"]) != 1 {
t.Fatal("a node was told again when only the controller's files changed")
}
// A new stream is told again.
r.svc.OnRegister("cuda01")
r.tick()
if len(r.agents.got["cuda01"]) != 2 {
t.Fatal("a new stream was not told")
}
if r.inv.nodes["cuda01"].State != inventory.Ready {
t.Fatal("a node drained in Slurm with no job was taken from llmgw")
}
}
func TestLendAndReclaim(t *testing.T) {
r := newRig()
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
r.set.Lent = []string{"cuda01"}
r.tick()
if r.inv.nodes["cuda01"].State != inventory.Draining {
t.Fatal("a lent node was not drained of its replicas")
}
if r.agents.last("cuda01").GetLent() {
t.Fatal("lent to Slurm with replicas still on it")
}
if p := r.svc.Status().Nodes[0].Phase; p != Lending {
t.Fatalf("phase %s", p)
}
r.inv.allocs["cuda01"] = 0
r.tick()
if !r.agents.last("cuda01").GetLent() || r.svc.Status().Nodes[0].Phase != Lent {
t.Fatal("not lent once its replicas were gone")
}
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "ALLOCATED", Jobs: 3})
// Taken back: Slurm drains it; llmgw keeps off until its jobs end.
r.set.Lent = nil
r.tick()
if r.agents.last("cuda01").GetLent() {
t.Fatal("still lent")
}
if s := r.inv.nodes["cuda01"].State; s != inventory.Draining && s != inventory.Cordoned {
t.Fatalf("llmgw may place on a node with Slurm jobs: %v", s)
}
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "ALLOCATED+DRAIN", Jobs: 1})
r.tick()
if r.svc.Status().Nodes[0].Phase != Reclaiming || r.inv.nodes["cuda01"].State == inventory.Ready {
t.Fatal("given back with a job left")
}
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: serving models"})
r.tick()
if r.inv.nodes["cuda01"].State != inventory.Ready || r.svc.Status().Nodes[0].Phase != Llmgw {
t.Fatal("not llmgw's when its last job ended")
}
}
func TestNewLeaderKeepsOffUntilHeard(t *testing.T) {
r := newRig()
r.inv.allocs["cuda01"] = 0
r.tick()
// Nothing heard yet: cordoned, nothing moved.
if r.inv.nodes["cuda01"].State != inventory.Cordoned {
t.Fatalf("state %v before the agent said anything", r.inv.nodes["cuda01"].State)
}
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "MIXED", Jobs: 2})
r.tick()
if r.inv.nodes["cuda01"].State == inventory.Ready {
t.Fatal("ready with Slurm jobs on it")
}
}
func TestBrokenSlurmLeavesTheNodeToLlmgw(t *testing.T) {
r := newRig()
r.report("cuda01", &llmgwv1.SlurmStatus{Error: "installing Slurm: no package", Jobs: 0})
r.tick()
if r.inv.nodes["cuda01"].State != inventory.Ready {
t.Fatal("a Slurm that does not work took a node from llmgw")
}
}
func TestOperatorsCordonIsLeft(t *testing.T) {
r := newRig()
r.inv.nodes["cuda01"].State = inventory.Cordoned
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN"})
r.tick()
if r.inv.nodes["cuda01"].State != inventory.Cordoned {
t.Fatal("llmgw uncordoned a node an operator cordoned")
}
}
func TestOffAndOut(t *testing.T) {
r := newRig()
r.inv.allocs["cuda01"] = 0
r.set.Lent = []string{"cuda01"}
r.tick()
r.set.Enabled = false
r.tick()
if m := r.agents.last("cuda01"); m.GetNode() || m.GetController() {
t.Fatalf("told %v with Slurm off", m)
}
if r.inv.nodes["cuda01"].State != inventory.Ready {
t.Fatal("not llmgw's with Slurm off")
}
r = newRig()
r.inv.allocs["cuda01"] = 0
r.set.Lent = []string{"cuda01"}
r.tick()
r.set.Nodes, r.set.Lent = nil, nil
r.tick()
if r.inv.nodes["cuda01"].State != inventory.Ready || r.agents.last("cuda01").GetNode() {
t.Fatal("a node taken out of slurm.nodes is not llmgw's")
}
if !r.agents.last("node06").GetController() {
t.Fatal("the controller stopped with no nodes")
}
}
func TestNoControllerYet(t *testing.T) {
r := newRig()
delete(r.inv.nodes, "node06")
r.agents.conn = []string{"cuda01"}
r.tick()
if m := r.agents.last("cuda01"); m.GetNode() {
t.Fatalf("a node was started with no controller to fetch its files from: %v", m)
}
}
func TestAllNodes(t *testing.T) {
r := newRig()
r.set.Nodes, r.set.AllNodes = nil, true
r.inv.nodes["cuda02"] = &inventory.Node{ID: "cuda02", MgmtAddr: "10.0.0.150"}
r.agents.conn = append(r.agents.conn, "cuda02")
r.tick()
for _, n := range []string{"cuda01", "cuda02", "node06"} {
if !r.agents.last(n).GetNode() {
t.Fatalf("%s, a node that joined, does not run slurmd", n)
}
}
if got := len(r.svc.Status().Nodes); got != 3 {
t.Fatalf("%d nodes in the status", got)
}
r.set.Lent = []string{"cuda02"}
r.tick()
if p := r.svc.Status().Nodes[1]; p.Node != "cuda02" || p.Phase != Lent {
t.Fatalf("cuda02 not lent: %+v", p)
}
}
+2 -2
View File
@@ -94,7 +94,7 @@ type Store interface {
SetStaffRole(ctx context.Context, id, role string) error
SetUserState(ctx context.Context, id string, state store.UserState) error
DeleteUserSessions(ctx context.Context, userID string) error
AddMembership(ctx context.Context, user string, scope store.Scope, id, role, by string, at time.Time) error
EnsureMembership(ctx context.Context, user string, scope store.Scope, id, by string, at time.Time) error
RemoveMembership(ctx context.Context, userID string, scope store.Scope, scopeID string, at time.Time) (int, error)
MembershipsOf(ctx context.Context, userID string) ([]store.Membership, error)
AllOrgs(ctx context.Context) ([]store.Org, error)
@@ -221,7 +221,7 @@ func (k *Keeper) apply(ctx context.Context, u *store.User, id Identity, remove b
continue // a mapping to a place the tree lacks: said by Sync
}
want[string(scope)+"/"+sid] = scope
if err := k.St.AddMembership(ctx, u.ID, scope, sid, store.RoleMember, k.Name, k.now()); err != nil {
if err := k.St.EnsureMembership(ctx, u.ID, scope, sid, k.Name, k.now()); err != nil {
return err
}
}
+10
View File
@@ -480,6 +480,7 @@ var Reference = []Block{
{Key: "mail", Type: "mail.Config", Doc: "Mail is where the gateway's few emails go out from: a link to set a\nforgotten password. Unset, none is sent and people ask an operator.", Nested: "mail.Config"},
{Key: "serving", Type: "ServingSettings", Doc: "Serving stops and starts the whole cluster's service (llmgw stop, llmgw\nstart).", Nested: "config.ServingSettings"},
{Key: "firewall", Type: "FirewallSettings", Doc: "Firewall turns llmgw's firewall on every machine on or off.", Nested: "config.FirewallSettings"},
{Key: "slurm", Type: "SlurmSettings", Doc: "Slurm is a Slurm cluster run beside llmgw on some of its nodes, and\nwhich of them it may use now.", Nested: "config.SlurmSettings"},
{Key: "presets", Type: "PresetSettings", Doc: "Presets is how the models that come with llmgw reach the catalog.", Nested: "config.PresetSettings"},
{Key: "on_demand", Type: "OnDemandSettings", Doc: "OnDemand is how models that load when asked for take turns on the\ncards.", Nested: "config.OnDemandSettings"},
{Key: "agents", Type: "AgentSettings", Doc: "Agents are the nodes' agent settings: agent.yaml's keys but those an\nagent needs to start, for every node (default) and for one\n(nodes.<id>). Sent to the agents as they change; an agent whose\nsettings changed starts again on them, its engines serving\nthroughout.", Nested: "config.AgentSettings"},
@@ -512,6 +513,15 @@ var Reference = []Block{
{Package: "engines", Name: "SimConfig", Title: "SimConfig", Doc: "", Fields: []Field{
{Key: "bin", Type: "string", Doc: "Bin is the llmgw binary itself; the engine is one of its subcommands.\nEmpty means whatever started this agent, which is right almost always.", Nested: ""},
}},
{Package: "config", Name: "SlurmSettings", Title: "SlurmSettings", Doc: "", Fields: []Field{
{Key: "enabled", Type: "bool", Doc: "Enabled runs Slurm on the nodes below. False stops llmgw keeping it;\nwhat was installed is left installed, and every node is llmgw's.", Nested: ""},
{Key: "cluster_name", Type: "string", Doc: "ClusterName is Slurm's ClusterName. Empty is \"llmgw\".", Nested: ""},
{Key: "controller", Type: "string", Doc: "Controller is the llmgw node whose agent runs slurmctld. It needs no\nGPU, and need not be one of Nodes.", Nested: ""},
{Key: "nodes", Type: "[]string", Doc: "Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained\nin Slurm, until it is also in Lent.", Nested: ""},
{Key: "all_nodes", Type: "bool", Doc: "AllNodes runs slurmd on every llmgw node, those that join later too,\nbesides any named in Nodes. Each is llmgw's until it is lent, as any.", Nested: ""},
{Key: "lent", Type: "[]string", Doc: "Lent are the nodes of Nodes Slurm may run jobs on now. A node put here\nis drained of llmgw's replicas first, as `ctl drain` does, and resumed\nin Slurm once none is left; a node taken out is drained in Slurm and\ngiven back to llmgw when its last job ends.", Nested: ""},
{Key: "partition", Type: "string", Doc: "Partition is the one partition every node is in. Empty is \"gpu\".", Nested: ""},
}},
{Package: "engines", Name: "StrataConfig", Title: "StrataConfig", Doc: "", Fields: []Field{
{Key: "dir", Type: "string", Doc: "Dir is Strata's source directory, where setup.py and serve/ are.", Nested: ""},
{Key: "python", Type: "string", Doc: "Python is the interpreter of its environment; Dir's .venv when unset.", Nested: ""},
+1
View File
@@ -41,6 +41,7 @@ var Sources = []string{
"internal/config/serving.go",
"internal/config/presets.go",
"internal/config/ondemand.go",
"internal/config/slurm.go",
"internal/config/agents.go",
"internal/mail/mail.go",
"internal/notify/notify.go",
+23
View File
@@ -173,6 +173,29 @@ func (o Option) WithContext(kvPerToken int64, tokens int) Option {
return o
}
// MostContext is the longest context, up to trained, this option still holds
// for one request, in whole thousands of tokens below trained: trained when
// the KV cache's size is unknown, 0 when trained is, for Strata (its context
// is its own), or when not even 1024 tokens fit.
func (o Option) MostContext(kvPerToken int64, trained int) int {
switch {
case trained <= 0 || o.Engine == "strata":
return 0
case kvPerToken <= 0 || o.WithContext(kvPerToken, trained).Fits():
return trained
}
lo, hi := 0, trained/1024+1 // lo fits (or is none), hi does not
for hi-lo > 1 {
mid := (lo + hi) / 2
if o.WithContext(kvPerToken, mid*1024).Fits() {
lo = mid
} else {
hi = mid
}
}
return lo * 1024
}
// MemoryText says what it needs, part by part.
func (o Option) MemoryText() string {
if o.Engine == "strata" {
+20
View File
@@ -193,3 +193,23 @@ func TestAnOptionSaysWhatItNeedsOfEachCardAndOfAll(t *testing.T) {
t.Errorf("a 2M-token context fits four cards: %s", huge.MemoryText())
}
}
func TestTheMostContextAnOptionHolds(t *testing.T) {
// 4 GB of weights on a 24 GB card, a token's cache 1 MiB.
o := Option{Engine: "llamacpp", Class: "6000-24g", GPUsPerNode: 1, Nodes: 1,
WeightsGB: 4, EngineGB: 1, NeedGB: 5, HaveGB: 21.6}
const kv = 1 << 20
if got := o.MostContext(kv, 8192); got != 8192 {
t.Errorf("a context that fits whole: %d, want what it was trained for", got)
}
got := o.MostContext(kv, 131072)
if got <= 0 || got%1024 != 0 || !o.WithContext(kv, got).Fits() || o.WithContext(kv, got+1024).Fits() {
t.Errorf("the most that fits is %d", got)
}
if got := o.MostContext(0, 32768); got != 32768 {
t.Errorf("an unknown cache size: %d, want what it was trained for", got)
}
if got := (Option{Engine: "strata"}).MostContext(kv, 32768); got != 0 {
t.Errorf("Strata's context is its own: %d", got)
}
}
+2 -2
View File
@@ -39,7 +39,7 @@ type Store interface {
SetUserState(ctx context.Context, id string, state store.UserState) error
DeleteUserSessions(ctx context.Context, userID string) error
SetStaffRole(ctx context.Context, id, role string) error
AddMembership(ctx context.Context, user string, scope store.Scope, id, role, by string, at time.Time) error
EnsureMembership(ctx context.Context, user string, scope store.Scope, id, by string, at time.Time) error
RemoveMembership(ctx context.Context, user string, scope store.Scope, id string, at time.Time) (int, error)
AllOrgs(ctx context.Context) ([]store.Org, error)
AllTeams(ctx context.Context) ([]store.TeamPath, error)
@@ -548,7 +548,7 @@ func (s *Server) member(ctx context.Context, group, userID string, joined bool)
return
}
if joined {
err = s.st.AddMembership(ctx, u.ID, scope, id, store.RoleMember, "scim", time.Now())
err = s.st.EnsureMembership(ctx, u.ID, scope, id, "scim", time.Now())
} else {
_, err = s.st.RemoveMembership(ctx, u.ID, scope, id, time.Now())
}
+1 -1
View File
@@ -58,7 +58,7 @@ func (m *mem) DeleteUserSessions(_ context.Context, id string) error {
return nil
}
func (m *mem) SetStaffRole(_ context.Context, id, role string) error { m.roles[id] = role; return nil }
func (m *mem) AddMembership(_ context.Context, u string, sc store.Scope, id, _, _ string, _ time.Time) error {
func (m *mem) EnsureMembership(_ context.Context, u string, sc store.Scope, id, _ string, _ time.Time) error {
m.members[u+"@"+string(sc)+":"+id] = true
return nil
}
+344
View File
@@ -0,0 +1,344 @@
// Package slurm is what llmgw knows of Slurm (BACKLOG P): the files a
// controller runs on, written from what llmgw knows of its nodes, the munge
// key every machine shares, and reading what slurmd and scontrol say.
//
// Nothing here runs anything. The agent installs and starts the daemons; the
// leader decides what each node is and writes the files with this package.
package slurm
import (
"crypto/hkdf"
"crypto/sha256"
"fmt"
"regexp"
"slices"
"strconv"
"strings"
)
// Paths are where the files are on every distribution llmgw installs on:
// Debian and Ubuntu since Slurm 20.11, and EPEL.
const (
ConfDir = "/etc/slurm"
MungeKeyPath = "/etc/munge/munge.key"
StateDir = "/var/spool/slurmctld"
SpoolDir = "/var/spool/slurmd"
LogDir = "/var/log/slurm"
// CtldPort is slurmctld's, which configless slurmd fetch their files
// from.
CtldPort = 6817
// SlurmdPort is slurmd's, which the controller and srun reach.
SlurmdPort = 6818
// SrunPortsFrom and SrunPortsTo are where srun listens for the steps it
// starts: a range of its own, so a firewall can let it in (Slurm's own
// example range). Without one srun takes any ephemeral port.
SrunPortsFrom = 60001
SrunPortsTo = 63000
)
// ReasonNew is why a node new to slurm.conf starts drained. Every reason
// llmgw drains with starts with "llmgw", and only those are resumed by it.
const ReasonNew = "llmgw: not lent yet"
// MinVersion is the oldest Slurm llmgw runs: the first to put jobs in
// cgroup v2, which every distribution llmgw supports now mounts alone.
var MinVersion = [2]int{22, 5}
// MungeKeyLen is the length munge's own tool writes.
const MungeKeyLen = 1024
// MungeKey is the key every machine of the Slurm cluster shares, derived
// from a cluster secret, so a control plane that takes over writes the same
// one and nobody copies it by hand.
func MungeKey(secret string) ([]byte, error) {
if len(secret) < 32 {
return nil, fmt.Errorf("the secret a munge key is made from is too short")
}
return hkdf.Key(sha256.New, []byte(secret), nil, "llmgw slurm munge key v1", MungeKeyLen)
}
// Hardware is a node's line from `slurmd -C`: how slurmd counts it, which is
// what slurm.conf must say or the controller refuses the node.
type Hardware struct {
CPUs int
Boards int
SocketsPerBoard int
CoresPerSocket int
ThreadsPerCore int
// RealMemory is in MiB.
RealMemory int64
}
// ParseHardware reads `slurmd -C`.
func ParseHardware(out string) (Hardware, error) {
var h Hardware
for _, line := range strings.Split(out, "\n") {
if !strings.HasPrefix(strings.TrimSpace(line), "NodeName=") {
continue
}
for _, f := range strings.Fields(line) {
k, v, _ := strings.Cut(f, "=")
n, err := strconv.ParseInt(v, 10, 64)
if err != nil {
continue
}
switch k {
case "CPUs":
h.CPUs = int(n)
case "Boards":
h.Boards = int(n)
case "SocketsPerBoard":
h.SocketsPerBoard = int(n)
case "CoresPerSocket":
h.CoresPerSocket = int(n)
case "ThreadsPerCore":
h.ThreadsPerCore = int(n)
case "RealMemory":
h.RealMemory = n
}
}
break
}
if h.CPUs <= 0 || h.RealMemory <= 0 {
return Hardware{}, fmt.Errorf("slurmd -C said no CPUs or memory: %q", strings.TrimSpace(out))
}
return h, nil
}
// GPU is one of a node's cards.
type GPU struct {
// Minor is N of /dev/nvidiaN.
Minor int
// Name is the driver's, as nvidia-smi gives it.
Name string
}
// Node is one machine slurmd runs on.
type Node struct {
Name string
// Addr is where slurmctld reaches it: its llmgw management address.
Addr string
Hardware
GPUs []GPU
}
// Cluster is everything slurm.conf says.
type Cluster struct {
Name string
Controller string
ControllerAddr string
Partition string
// Nodes without hardware (not reported yet) are left out of the files
// until they are.
Nodes []Node
}
// gpuModel is a word that names a card's model: A100, H100, V100, T4, L40S.
var gpuModel = regexp.MustCompile(`^[a-z]{1,2}\d{1,4}[a-z]?$`)
// GPUType is a card's name as a Slurm GRES type: "NVIDIA A100-SXM4-80GB" is
// a100, "NVIDIA GeForce RTX 3090" rtx3090, "Tesla T4" t4. A name it cannot
// read is "gpu".
func GPUType(name string) string {
words := strings.Fields(strings.ToLower(strings.NewReplacer("-", " ", "_", " ").Replace(name)))
for i, w := range words {
if (w == "rtx" || w == "gtx") && i+1 < len(words) {
return w + words[i+1]
}
if gpuModel.MatchString(w) {
return w
}
}
return "gpu"
}
// gpuGroups are a node's cards by type, the types sorted.
func gpuGroups(gs []GPU) ([]string, map[string][]int) {
by := map[string][]int{}
for _, g := range gs {
t := GPUType(g.Name)
by[t] = append(by[t], g.Minor)
}
types := make([]string, 0, len(by))
for t, ms := range by {
slices.Sort(ms)
types = append(types, t)
}
slices.Sort(types)
return types, by
}
// memoryKept is what is left out of a node's RealMemory: slurmd -C reads
// what the kernel says now, which is a little less after a reboot, and a
// node whose memory is below what slurm.conf says is refused.
const memoryKept = 0.97
func (c Cluster) ready() []Node {
var out []Node
for _, n := range c.Nodes {
if n.CPUs > 0 && n.RealMemory > 0 {
out = append(out, n)
}
}
slices.SortFunc(out, func(a, b Node) int { return strings.Compare(a.Name, b.Name) })
return out
}
const header = "# Written by llmgw (BACKLOG P), on the node that runs slurmctld; the nodes read it\n" +
"# from there. Change it through the cluster's settings (slurm.*), not here.\n"
// SlurmConf is slurm.conf.
func (c Cluster) SlurmConf() string {
var b strings.Builder
b.WriteString(header)
fmt.Fprintf(&b, "ClusterName=%s\n", c.Name)
fmt.Fprintf(&b, "SlurmctldHost=%s(%s)\n", c.Controller, c.ControllerAddr)
b.WriteString(`SlurmctldParameters=enable_configless
SlurmctldPort=6817
SlurmdPort=6818
SrunPortRange=60001-63000
AuthType=auth/munge
CredType=cred/munge
SlurmUser=slurm
SlurmdUser=root
StateSaveLocation=` + StateDir + `
SlurmdSpoolDir=` + SpoolDir + `
SlurmctldLogFile=` + LogDir + `/slurmctld.log
SlurmdLogFile=` + LogDir + `/slurmd.log
SlurmctldPidFile=/run/slurmctld.pid
SlurmdPidFile=/run/slurmd.pid
ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup,task/affinity
# Nothing gathered for accounting until there is somewhere to keep it
# (slurmdbd, later); gathered and not kept is an error in every log.
JobAcctGatherType=jobacct_gather/none
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory
SchedulerType=sched/backfill
GresTypes=gpu
MpiDefault=none
# No mail sent; without it every start says MailProg is invalid.
MailProg=/bin/true
# Without accounting storage, 23.11's main scheduler marks a batch job
# InvalidAccount; backfill starts it on its next pass, so it is kept short.
SchedulerParameters=bf_interval=10
# A node that went down and comes back is taken again; one llmgw drained
# stays drained until llmgw resumes it.
ReturnToService=2
SlurmctldTimeout=120
SlurmdTimeout=300
`)
nodes := c.ready()
names := make([]string, 0, len(nodes))
for _, n := range nodes {
names = append(names, n.Name)
fmt.Fprintf(&b, "NodeName=%s NodeAddr=%s CPUs=%d", n.Name, n.Addr, n.CPUs)
if n.Boards > 0 && n.SocketsPerBoard > 0 && n.CoresPerSocket > 0 && n.ThreadsPerCore > 0 {
fmt.Fprintf(&b, " Boards=%d SocketsPerBoard=%d CoresPerSocket=%d ThreadsPerCore=%d",
n.Boards, n.SocketsPerBoard, n.CoresPerSocket, n.ThreadsPerCore)
}
fmt.Fprintf(&b, " RealMemory=%d", int64(float64(n.RealMemory)*memoryKept))
if types, by := gpuGroups(n.GPUs); len(types) > 0 {
gres := make([]string, 0, len(types))
for _, t := range types {
gres = append(gres, fmt.Sprintf("gpu:%s:%d", t, len(by[t])))
}
fmt.Fprintf(&b, " Gres=%s", strings.Join(gres, ","))
}
// Drained until llmgw lends it: a node new to the controller would
// otherwise take jobs in the minute before its agent drains it.
fmt.Fprintf(&b, " State=DRAIN Reason=%q\n", ReasonNew)
}
if len(names) > 0 {
fmt.Fprintf(&b, "PartitionName=%s Nodes=%s Default=YES MaxTime=INFINITE State=UP\n",
c.Partition, strings.Join(names, ","))
}
return b.String()
}
// GresConf is gres.conf: each node's cards by device file, so a job is given
// the cards it asked for and kept off the others.
func (c Cluster) GresConf() string {
var b strings.Builder
b.WriteString(header)
for _, n := range c.ready() {
types, by := gpuGroups(n.GPUs)
for _, t := range types {
files := make([]string, 0, len(by[t]))
for _, m := range by[t] {
files = append(files, fmt.Sprintf("/dev/nvidia%d", m))
}
fmt.Fprintf(&b, "NodeName=%s Name=gpu Type=%s File=%s\n", n.Name, t, strings.Join(files, ","))
}
}
return b.String()
}
// CgroupConf is cgroup.conf: a job kept to its cores, memory and cards.
func (c Cluster) CgroupConf() string {
return header + `CgroupPlugin=autodetect
ConstrainCores=yes
ConstrainDevices=yes
ConstrainRAMSpace=yes
`
}
// Files are the controller's files, by name in ConfDir.
func (c Cluster) Files() map[string]string {
return map[string]string{"slurm.conf": c.SlurmConf(), "gres.conf": c.GresConf(), "cgroup.conf": c.CgroupConf()}
}
// NodeState is what `scontrol show node -o NAME` says of a node.
type NodeState struct {
// State is Slurm's, with its flags: IDLE, ALLOCATED, MIXED+DRAIN,
// DOWN+NOT_RESPONDING.
State string
Reason string
}
// Drained is whether the controller gives the node nothing new.
func (s NodeState) Drained() bool {
return strings.Contains(s.State, "DRAIN") || strings.Contains(s.State, "DOWN")
}
// ParseNodeState reads `scontrol show node -o NAME`, one line of key=value.
func ParseNodeState(out string) (NodeState, error) {
line := strings.TrimSpace(out)
if !strings.HasPrefix(line, "NodeName=") {
return NodeState{}, fmt.Errorf("scontrol said %q", line)
}
var s NodeState
if i := strings.Index(line, " State="); i >= 0 {
s.State, _, _ = strings.Cut(line[i+len(" State="):], " ")
}
// Reason is last but for its [user@time] and may have spaces.
if i := strings.Index(line, " Reason="); i >= 0 {
r := line[i+len(" Reason="):]
if j := strings.LastIndex(r, " ["); j >= 0 {
r = r[:j]
}
s.Reason = strings.TrimSpace(r)
}
return s, nil
}
// ParseVersion reads "slurm 23.11.4" or "slurm-wlm 21.08.5" as major, minor.
func ParseVersion(out string) ([2]int, error) {
for _, f := range strings.Fields(out) {
parts := strings.Split(f, ".")
if len(parts) < 2 {
continue
}
major, err1 := strconv.Atoi(parts[0])
minor, err2 := strconv.Atoi(parts[1])
if err1 == nil && err2 == nil {
return [2]int{major, minor}, nil
}
}
return [2]int{}, fmt.Errorf("no version in %q", strings.TrimSpace(out))
}
// TooOld is whether v is older than MinVersion.
func TooOld(v [2]int) bool {
return v[0] < MinVersion[0] || (v[0] == MinVersion[0] && v[1] < MinVersion[1])
}
+151
View File
@@ -0,0 +1,151 @@
package slurm
import (
"bytes"
"strings"
"testing"
)
func TestMungeKey(t *testing.T) {
s := strings.Repeat("ab", 32)
a, err := MungeKey(s)
if err != nil {
t.Fatal(err)
}
b, _ := MungeKey(s)
if len(a) != MungeKeyLen || !bytes.Equal(a, b) {
t.Fatalf("a key is %d bytes and the same each time: got %d, equal %v", MungeKeyLen, len(a), bytes.Equal(a, b))
}
c, _ := MungeKey(strings.Repeat("cd", 32))
if bytes.Equal(a, c) {
t.Fatal("two secrets made one key")
}
if _, err := MungeKey("short"); err == nil {
t.Fatal("a short secret was taken")
}
}
func TestParseHardware(t *testing.T) {
out := "NodeName=cuda01 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=64302\n" +
"UpTime=3-01:02:03\n"
h, err := ParseHardware(out)
if err != nil {
t.Fatal(err)
}
want := Hardware{CPUs: 16, Boards: 1, SocketsPerBoard: 2, CoresPerSocket: 4, ThreadsPerCore: 2, RealMemory: 64302}
if h != want {
t.Fatalf("got %+v", h)
}
if _, err := ParseHardware("slurmd: error: something\n"); err == nil {
t.Fatal("an error was read as hardware")
}
}
func TestGPUType(t *testing.T) {
for name, want := range map[string]string{
"NVIDIA A100-SXM4-80GB": "a100",
"NVIDIA H100 80GB HBM3": "h100",
"Tesla V100-SXM2-32GB": "v100",
"Tesla T4": "t4",
"NVIDIA GeForce RTX 3090": "rtx3090",
"NVIDIA RTX A6000": "rtxa6000",
"NVIDIA L40S": "l40s",
"Quadro RTX 8000": "rtx8000",
"NVIDIA GeForce GTX 1080": "gtx1080",
"": "gpu",
"Some Accelerator Unknown": "gpu",
} {
if got := GPUType(name); got != want {
t.Errorf("GPUType(%q) = %q, want %q", name, got, want)
}
}
}
func testCluster() Cluster {
return Cluster{Name: "llmgw", Controller: "node06", ControllerAddr: "10.0.0.146", Partition: "gpu",
Nodes: []Node{
{Name: "site2", Addr: "10.1.0.5",
Hardware: Hardware{CPUs: 32, Boards: 1, SocketsPerBoard: 2, CoresPerSocket: 8, ThreadsPerCore: 2, RealMemory: 100000},
GPUs: []GPU{{Minor: 2, Name: "Tesla T4"}, {Minor: 0, Name: "Tesla V100-SXM2-32GB"}, {Minor: 1, Name: "Tesla V100-SXM2-32GB"}}},
{Name: "cuda01", Addr: "10.0.0.147", Hardware: Hardware{CPUs: 16, RealMemory: 64000},
GPUs: []GPU{{Minor: 0, Name: "NVIDIA GeForce RTX 3090"}}},
{Name: "waiting", Addr: "10.0.0.9"},
}}
}
func TestSlurmConf(t *testing.T) {
conf := testCluster().SlurmConf()
for _, want := range []string{
"ClusterName=llmgw\n",
"SlurmctldHost=node06(10.0.0.146)\n",
"SlurmctldParameters=enable_configless\n",
"NodeName=cuda01 NodeAddr=10.0.0.147 CPUs=16 RealMemory=62080 Gres=gpu:rtx3090:1 State=DRAIN Reason=\"llmgw: not lent yet\"\n",
"NodeName=site2 NodeAddr=10.1.0.5 CPUs=32 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=97000 Gres=gpu:t4:1,gpu:v100:2 State=DRAIN Reason=\"llmgw: not lent yet\"\n",
"PartitionName=gpu Nodes=cuda01,site2 Default=YES",
} {
if !strings.Contains(conf, want) {
t.Errorf("slurm.conf has no %q:\n%s", want, conf)
}
}
if strings.Contains(conf, "waiting") {
t.Error("a node that has not reported its hardware is in slurm.conf")
}
if strings.Index(conf, "NodeName=cuda01") > strings.Index(conf, "NodeName=site2") {
t.Error("nodes are not in order")
}
}
func TestSlurmConfNoNodes(t *testing.T) {
c := testCluster()
c.Nodes = nil
if strings.Contains(c.SlurmConf(), "PartitionName") {
t.Error("a partition with no nodes would stop slurmctld starting")
}
}
func TestGresConf(t *testing.T) {
g := testCluster().GresConf()
for _, want := range []string{
"NodeName=cuda01 Name=gpu Type=rtx3090 File=/dev/nvidia0\n",
"NodeName=site2 Name=gpu Type=t4 File=/dev/nvidia2\n",
"NodeName=site2 Name=gpu Type=v100 File=/dev/nvidia0,/dev/nvidia1\n",
} {
if !strings.Contains(g, want) {
t.Errorf("gres.conf has no %q:\n%s", want, g)
}
}
}
func TestParseNodeState(t *testing.T) {
out := "NodeName=cuda01 Arch=x86_64 CoresPerSocket=8 CPUAlloc=0 State=IDLE+DRAIN ThreadsPerCore=2 " +
"Partitions=gpu Reason=llmgw: serving models [root@2026-11-11T10:00:00]\n"
s, err := ParseNodeState(out)
if err != nil {
t.Fatal(err)
}
if s.State != "IDLE+DRAIN" || s.Reason != "llmgw: serving models" || !s.Drained() {
t.Fatalf("got %+v", s)
}
s, _ = ParseNodeState("NodeName=cuda01 State=MIXED Partitions=gpu\n")
if s.Drained() || s.Reason != "" {
t.Fatalf("got %+v", s)
}
if _, err := ParseNodeState("Node cuda02 not found\n"); err == nil {
t.Fatal("an unknown node was read")
}
}
func TestParseVersion(t *testing.T) {
for out, want := range map[string][2]int{"slurm 23.11.4\n": {23, 11}, "slurm-wlm 21.08.5": {21, 8}, "slurm 22.05.9": {22, 5}} {
v, err := ParseVersion(out)
if err != nil || v != want {
t.Errorf("ParseVersion(%q) = %v, %v", out, v, err)
}
}
if !TooOld([2]int{21, 8}) || TooOld([2]int{22, 5}) || TooOld([2]int{23, 2}) {
t.Error("TooOld is wrong")
}
if _, err := ParseVersion("command not found"); err == nil {
t.Error("no version was read as one")
}
}
+54 -5
View File
@@ -14,10 +14,19 @@ var ErrNotMember = errors.New("not a member of that project")
// Roles a membership can carry.
const (
RoleMember = "member"
// RoleManager is reserved for delegated administration of a scope.
// RoleManager, on an organization, may look at that organization's page:
// its use, members and limits. It changes nothing.
RoleManager = "manager"
// RoleOrgAdmin, on an organization, also changes it: its teams and
// projects, who is in them, and their keys. Its limits, the accounts
// themselves and the rest of the cluster stay with the cluster's staff.
RoleOrgAdmin = "admin"
)
// MembershipRoles are the roles a membership may carry; manager and admin mean
// something only on an organization.
var MembershipRoles = []string{RoleMember, RoleManager, RoleOrgAdmin}
// Membership is a person's place at one level of the tree.
type Membership struct {
UserID string
@@ -43,6 +52,12 @@ func (s *DB) AddMembership(ctx context.Context, userID string, scope Scope, scop
if role == "" {
role = RoleMember
}
if role != RoleMember && role != RoleManager && role != RoleOrgAdmin {
return fmt.Errorf("a membership's role is member, manager or admin, not %q", role)
}
if role != RoleMember && scope != ScopeOrg {
return fmt.Errorf("%s is a role on an organization, not on a %s", role, scope)
}
table := map[Scope]string{ScopeOrg: "organizations", ScopeTeam: "teams", ScopeProject: "projects"}[scope]
if err := s.exists(ctx, string(scope), `SELECT COUNT(*) FROM `+table+` WHERE id = ?`, scopeID); err != nil {
return err
@@ -55,6 +70,27 @@ func (s *DB) AddMembership(ctx context.Context, userID string, scope Scope, scop
return err
}
// EnsureMembership puts a person in a scope as a member if they are not there
// yet, and leaves a membership that exists as it is. It is for what a
// directory or an identity provider says on every sign-in or sync: a group
// that places someone in an organization must not, by saying so again, take
// away the manager or admin role an administrator gave them there.
func (s *DB) EnsureMembership(ctx context.Context, userID string, scope Scope, scopeID, by string, at time.Time) error {
if !validMembershipScope(scope) {
return fmt.Errorf("a membership is in an org, team or project, not %q", scope)
}
table := map[Scope]string{ScopeOrg: "organizations", ScopeTeam: "teams", ScopeProject: "projects"}[scope]
if err := s.exists(ctx, string(scope), `SELECT COUNT(*) FROM `+table+` WHERE id = ?`, scopeID); err != nil {
return err
}
_, err := s.db.ExecContext(ctx, `
INSERT INTO memberships (user_id, scope_type, scope_id, role, added_at, added_by)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT (user_id, scope_type, scope_id) DO NOTHING`,
userID, string(scope), scopeID, RoleMember, millis(at), nullString(by))
return err
}
// RemoveMembership takes a person out of a scope and revokes the keys they
// made in any project they can no longer use, returning how many.
//
@@ -306,23 +342,36 @@ func (s *DB) resolveChain(ctx context.Context, e *Effective, keyLevel bool) erro
return nil
}
// ManagedOrgs lists the organizations a person manages.
// ManagedOrgs lists the organizations a person manages or administers: those
// whose page they may see.
func (s *DB) ManagedOrgs(ctx context.Context, userID string) ([]Org, error) {
return queryAll(ctx, s.db, `
SELECT o.id, o.name FROM memberships m JOIN organizations o ON o.id = m.scope_id
WHERE m.user_id = ? AND m.scope_type = 'org' AND m.role = 'manager'
WHERE m.user_id = ? AND m.scope_type = 'org' AND m.role IN ('manager', 'admin')
ORDER BY o.name`, []any{userID}, func(rows *sql.Rows) (Org, error) {
var o Org
return o, rows.Scan(&o.ID, &o.Name)
})
}
// ManagesOrg reports whether a person manages an organization.
// ManagesOrg reports whether a person manages or administers an
// organization: whether they may see its page.
func (s *DB) ManagesOrg(ctx context.Context, userID, orgID string) (bool, error) {
var n int
err := s.db.QueryRowContext(ctx, `
SELECT COUNT(*) FROM memberships
WHERE user_id = ? AND scope_type = 'org' AND scope_id = ? AND role = 'manager'`,
WHERE user_id = ? AND scope_type = 'org' AND scope_id = ? AND role IN ('manager', 'admin')`,
userID, orgID).Scan(&n)
return n > 0, err
}
// AdministersOrg reports whether a person administers an organization:
// whether they may change it.
func (s *DB) AdministersOrg(ctx context.Context, userID, orgID string) (bool, error) {
var n int
err := s.db.QueryRowContext(ctx, `
SELECT COUNT(*) FROM memberships
WHERE user_id = ? AND scope_type = 'org' AND scope_id = ? AND role = 'admin'`,
userID, orgID).Scan(&n)
return n > 0, err
}
+75
View File
@@ -307,3 +307,78 @@ func TestASuspendedPersonsKeysStopWorkingAndComeBackWithThem(t *testing.T) {
t.Errorf("reinstated, the key still refused: %v", err)
}
}
func TestAnOrganizationsAdminAlsoManagesIt(t *testing.T) {
f := newFixture(t)
ctx := t.Context()
if err := f.s.AddMembership(ctx, f.user, ScopeOrg, f.acme, RoleOrgAdmin, "t", time.Now()); err != nil {
t.Fatal(err)
}
if ok, _ := f.s.AdministersOrg(ctx, f.user, f.acme); !ok {
t.Error("an admin of Acme is not recognised as one")
}
if ok, _ := f.s.ManagesOrg(ctx, f.user, f.acme); !ok {
t.Error("an admin of Acme may not see its page")
}
if ok, _ := f.s.AdministersOrg(ctx, f.user, f.scania); ok {
t.Error("administering Acme made them an admin of Scania")
}
if orgs, _ := f.s.ManagedOrgs(ctx, f.user); len(orgs) != 1 || orgs[0].Name != "Acme" {
t.Errorf("ManagedOrgs = %+v", orgs)
}
// A manager sees the page and changes nothing.
if err := f.s.AddMembership(ctx, f.user, ScopeOrg, f.acme, RoleManager, "t", time.Now()); err != nil {
t.Fatal(err)
}
if ok, _ := f.s.AdministersOrg(ctx, f.user, f.acme); ok {
t.Error("a manager is taken for an admin")
}
}
func TestManagerAndAdminAreRolesOnAnOrganizationOnly(t *testing.T) {
f := newFixture(t)
ctx := t.Context()
for _, c := range []struct {
scope Scope
id string
role string
}{
{ScopeProject, f.gateway, RoleOrgAdmin},
{ScopeTeam, f.platform, RoleManager},
{ScopeOrg, f.acme, "owner"},
} {
if err := f.s.AddMembership(ctx, f.user, c.scope, c.id, c.role, "t", time.Now()); err == nil {
t.Errorf("%s on a %s was taken", c.role, c.scope)
}
}
}
func TestASyncedGroupLeavesARoleAsItIs(t *testing.T) {
// A directory group that places someone in Acme says so at every sign-in;
// saying it again must not take away the admin role they were given there.
f := newFixture(t)
ctx := t.Context()
if err := f.s.AddMembership(ctx, f.user, ScopeOrg, f.acme, RoleOrgAdmin, "t", time.Now()); err != nil {
t.Fatal(err)
}
if err := f.s.EnsureMembership(ctx, f.user, ScopeOrg, f.acme, "oidc", time.Now()); err != nil {
t.Fatal(err)
}
if ok, _ := f.s.AdministersOrg(ctx, f.user, f.acme); !ok {
t.Error("a sign-in's group took the admin role away")
}
// And one that is not there yet is added as a member.
if err := f.s.EnsureMembership(ctx, f.user, ScopeOrg, f.scania, "oidc", time.Now()); err != nil {
t.Fatal(err)
}
ms, _ := f.s.AllMemberships(ctx)
found := false
for _, m := range ms {
if m.UserID == f.user && m.Scope == ScopeOrg && m.ScopeID == f.scania {
found = m.Role == RoleMember
}
}
if !found {
t.Error("a new group did not make them a member of Scania")
}
}
+7
View File
@@ -50,6 +50,13 @@ func (d AboutData) InstallURL() string {
return "/install"
}
// DeckURL is the introduction for decision makers, as a PDF. It lives in the
// repository rather than in the binary, so every gateway does not carry a
// megabyte of slides; gateway and website link to the same file.
func (AboutData) DeckURL() string {
return repoURL + "/raw/branch/main/docs/llmgw-intro.pdf"
}
// Still is a screen of the install guide's recording, for the steps here.
func (d AboutData) Still(name string) template.HTML {
return InstallData{Site: d.Site}.Still(name)
+2
View File
@@ -123,6 +123,8 @@ type AccountPages struct {
personalDefaults *store.Limits
// orgs is set when /org is mounted, so the menu can offer it to managers.
orgs OrgViewer
// orgEditor changes an organization from its page, for its admin.
orgEditor OrgEditor
// llamaVersion is the llama.cpp build nodes install, for judging whether a
// GGUF can be served before it is fetched or catalogued.
llamaVersion string
+7 -2
View File
@@ -267,7 +267,7 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
http.NotFound(w, r)
return
}
d := adminMirrorData{CSRF: csrfFor(r), Inline: true, HasRegistry: true}
d := adminMirrorData{CSRF: csrfIfMay(r, "/admin/mirror/x"), Inline: true, HasRegistry: true}
p.openRepo(r, &d, repo)
if d.RegistryErr != "" {
// Read by the page's own script, which shows it beneath the
@@ -502,7 +502,7 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
Addr: m.Addr(),
Message: msg,
Error: errMsg,
CSRF: csrfFor(r),
CSRF: csrfIfMay(r, "/admin/mirror/x"),
Version: p.version,
Now: time.Now().UTC().Format(time.RFC3339),
}
@@ -736,6 +736,11 @@ func mirrorTabs(l *Lang, d adminMirrorData) []mirrorTab {
tabs = append(tabs, mirrorTab{Label: l.T("mirror.tab.strata"),
Href: "/admin/mirror?tab=strata", Active: d.Tab == tabStrata})
}
if d.CSRF == "" {
// Fetching is all that tab does: someone who may not fetch is not
// offered an empty page.
return tabs
}
return append(tabs, mirrorTab{Label: l.T("mirror.tab.fetch"),
Href: "/admin/mirror?tab=fetch", Active: d.Tab == tabFetch})
}
+1 -1
View File
@@ -149,7 +149,7 @@ func (p *AccountPages) detail(r *http.Request, sel string, u *adminUsersData) *p
}
ctx := r.Context()
l := langOf(r)
d := &peopleDetail{Kind: store.Scope(kind), ID: id, CSRF: csrfFor(r),
d := &peopleDetail{Kind: store.Scope(kind), ID: id, CSRF: csrfIfMay(r, "/admin/users"),
UsageWindow: l.T("usage.window.7d"), QuotaWindow: l.T("quotas.since")}
if !p.nameDetail(d, u) {
return nil
+1 -1
View File
@@ -353,7 +353,7 @@ func (p *AccountPages) buildQuotas(r *http.Request, s QuotaAdmin, msg, errMsg st
d := adminQuotasData{
Nav: navFrom(r, "quotas"),
Since: l.T("quotas.since"), Message: msg, Error: errMsg,
CSRF: csrfFor(r), Version: p.version,
CSRF: csrfIfMay(r, "/admin/quotas"), Version: p.version,
Now: time.Now().UTC().Format(time.RFC3339),
}
+46 -3
View File
@@ -224,7 +224,10 @@ func planServe(l *Lang, st mirror.Status, files []mirror.Status, base string,
if weights <= 0 {
weights = st.BytesDone
}
if b := header.MemoryBytes(weights, defaultCtxPool); b > 0 {
if c, par := mostContext(header, weights, f.Placements, f.GPUs, fleet); c > 0 {
f.Context, f.Parallel, f.CtxPool = c, par, c*par
}
if b := header.MemoryBytes(weights, f.CtxPool); b > 0 {
kvHeads := header.HeadCountKV
if kvHeads == 0 {
kvHeads = header.HeadCount
@@ -235,8 +238,8 @@ func planServe(l *Lang, st mirror.Status, files []mirror.Status, base string,
// flat "half again" could not see.
f.MinMemoryGB = int((b + 1<<30 - 1) >> 30)
f.MemoryBasis = l.T("serve.basis.shape",
l.Num(float64(weights)/(1<<30), 1), l.Num(float64(int64(defaultCtxPool)*header.KVBytesPerToken())/(1<<30), 1),
l.Int(defaultCtxPool), header.BlockCount, kvHeads,
l.Num(float64(weights)/(1<<30), 1), l.Num(float64(int64(f.CtxPool)*header.KVBytesPerToken())/(1<<30), 1),
l.Int(int64(f.CtxPool)), header.BlockCount, kvHeads,
l.Num(float64(int64(1<<30)+weights/20)/(1<<30), 1))
}
if fleet != nil {
@@ -552,6 +555,46 @@ func modeOf(sc catalog.Scaling) string {
// --parallel divides it among slots; it does not multiply it.
const defaultCtxPool = 16384
// mostContext is the longest context the serve form starts with: what the
// model was trained for, as long as its KV cache fits where the form places
// it beside the weights, else the most that does; and as many slots of it,
// up to defaultParallel, as the room left holds. 0 when the header does not
// say what it was trained for, or the weights alone do not fit.
//
// The longest, because a context is the one thing about a served model a
// client runs into without being told: a 4096 default on a model trained for
// 40960 cut documents short that the model could have read whole.
func mostContext(h gguf.Header, weights int64, places []placeChoice, gpus int, fleet Source) (tokens, slots int) {
trained := int(h.ContextLength) //nolint:gosec // a header's context length
if trained <= 0 {
return 0, 0
}
pool := 0
for _, pc := range places {
if !pc.Selected {
continue
}
if class, ok := strings.CutPrefix(pc.Value, "gpu:"); ok {
pool = h.PoolThatFits(weights, gguf.ClassMemGB(class), gpus)
} else {
pool = h.PoolThatFits(weights, biggestNode(fleet), 1)
}
}
switch {
case pool < 0:
return 0, 0
case pool == 0:
// Nothing to measure it against: what it was trained for, as one slot
// would hold it; the checks on submitting still see the card.
return trained, 1
}
tokens = min(trained, pool/1024*1024)
if tokens < 1024 {
return 0, 0
}
return tokens, max(1, min(defaultParallel, pool/tokens))
}
// defaultParallel is the requests a llama.cpp replica serves at once when
// the form is not changed: the agent's own default.
const defaultParallel = 4
+28
View File
@@ -670,3 +670,31 @@ func TestAVerySlowModelOnTheCPUIsAChoiceMadeOnPurpose(t *testing.T) {
t.Fatal("ticking the box should add it")
}
}
func TestTheServeFormStartsAtTheLongestContextThatFits(t *testing.T) {
// An 8B's shape: 36 layers, 8 KV heads of 128, so 144 KiB a token.
h := gguf.Header{BlockCount: 36, HeadCount: 32, HeadCountKV: 8, KeyLength: 128, ValueLength: 128,
ContextLength: 40960}
card := []placeChoice{{Value: "gpu:6000-24g", Selected: true}, {Value: "cpu"}}
cpu := []placeChoice{{Value: "gpu:6000-24g"}, {Value: "cpu", Selected: true}}
for _, c := range []struct {
name string
weights int64
places []placeChoice
header gguf.Header
tokens, slots int
}{
// All it was trained for, and two slots of it in the room left.
{"on a card with room", 5 << 30, card, h, 40960, 2},
// The weights take most of the card: the most that fits, one slot.
{"on a card nearly full", 18 << 30, card, h, 18432, 1},
{"on the CPU of a 64 GB node", 5 << 30, cpu, h, 40960, 4},
{"weights larger than the card", 30 << 30, card, h, 0, 0},
{"a header that does not say", 5 << 30, card, gguf.Header{BlockCount: 36, HeadCountKV: 8, HeadCount: 32, KeyLength: 128}, 0, 0},
} {
tokens, slots := mostContext(c.header, c.weights, c.places, 1, fakeFleet{memoryGB: 64})
if tokens != c.tokens || slots != c.slots {
t.Errorf("%s: %d tokens × %d slots, want %d × %d", c.name, tokens, slots, c.tokens, c.slots)
}
}
}
+24 -10
View File
@@ -4,6 +4,7 @@ import (
"context"
"errors"
"net/http"
"slices"
"sort"
"strings"
"time"
@@ -92,7 +93,10 @@ type adminUsersData struct {
// Open is the project whose section is shown expanded: the one just acted
// on, so the result of a change is in view without the reader hunting for
// it. Scripted pages keep every open section open; this is for the rest.
Open string
Open string
// Edit is whether this person may change what the page shows: a viewer
// is drawn the page without its forms.
Edit bool
AdminID string
Message string
Error string
@@ -173,23 +177,19 @@ func (p *AccountPages) RegisterAdminUsers(mux *http.ServeMux, s AccountAdmin) {
return say("users.msg.added_to", scopeLabel(ctx, s, store.Scope(scope), scopeID)), nil
}))
// Managing is a role on an organization membership: it lets the person see
// that organization's page, and nothing else changes.
// A role on an organization membership: a manager sees that organization's
// page, an admin also changes it; a member neither.
mux.HandleFunc("POST /admin/users/{id}/role", p.adminAction(s,
func(ctx context.Context, admin store.User, id string, r *http.Request) (notice, error) {
scope, scopeID, ok := strings.Cut(r.FormValue("scope"), ":")
role := r.FormValue("role")
if !ok || store.Scope(scope) != store.ScopeOrg || (role != store.RoleManager && role != store.RoleMember) {
if !ok || store.Scope(scope) != store.ScopeOrg || !slices.Contains(store.MembershipRoles, role) {
return notice{}, refuse("users.err.manager_role")
}
if err := s.AddMembership(ctx, id, store.ScopeOrg, scopeID, role, admin.Email, time.Now()); err != nil {
return notice{}, err
}
label := scopeLabel(ctx, s, store.ScopeOrg, scopeID)
if role == store.RoleManager {
return say("users.msg.now_manages", label), nil
}
return say("users.msg.no_longer_manages", label), nil
return roleNotice(role, scopeLabel(ctx, s, store.ScopeOrg, scopeID)), nil
}))
mux.HandleFunc("POST /admin/users/{id}/leave", p.adminAction(s,
@@ -351,6 +351,17 @@ func (p *AccountPages) RegisterAdminUsers(mux *http.ServeMux, s AccountAdmin) {
}))
}
// roleNotice says what a person may now do in an organization.
func roleNotice(role, org string) notice {
switch role {
case store.RoleManager:
return say("users.msg.now_manages", org)
case store.RoleOrgAdmin:
return say("users.msg.now_administers", org)
}
return say("users.msg.no_longer_manages", org)
}
// notice is what the page says about a change, kept as a catalogue key and
// its arguments rather than as text: the log and the audit record take it in
// English, which is what they are searched in, and the page in the language
@@ -394,7 +405,7 @@ func init() {
registerKeys(
"users.msg.approved_alone", "users.msg.approved_into", "users.msg.suspended",
"users.msg.reinstated", "users.msg.added_to", "users.msg.now_manages",
"users.msg.no_longer_manages", "users.msg.removed_from", "users.msg.removed_from_one_key",
"users.msg.no_longer_manages", "users.msg.now_administers", "users.msg.removed_from", "users.msg.removed_from_one_key",
"users.msg.removed_from_keys", "users.msg.removed_keys_revoked", "users.msg.org_created",
"users.msg.team_created", "users.msg.project_created", "users.msg.renamed",
"users.msg.team_moved", "users.msg.project_moved", "users.msg.removed",
@@ -408,6 +419,8 @@ func init() {
// state, which is also the row's class.
"users.state.active", "users.state.pending", "users.state.suspended",
"users.key_state.active", "users.key_state.revoked", "users.key_state.expired",
// A membership's role, looked up by the role.
"users.role.member", "users.role.manager", "users.role.admin",
)
}
@@ -573,6 +586,7 @@ func (p *AccountPages) buildUsers(r *http.Request, s AccountAdmin, admin store.U
Members: map[string][]userRow{},
Keys: map[string][]keyRow{},
}
d.Edit = d.Nav.May("/admin/users")
var err error
// logErr logs what could not be read in English and says so on the page,
// once, in the reader's language.
+3 -3
View File
@@ -134,13 +134,13 @@ func TestTheUsageReportWritesTurkishNumbers(t *testing.T) {
func TestTheOrganizationPageSpeaksTurkishWhenAsked(t *testing.T) {
body := inTurkish(t, orgPage(t, store.User{ID: "u1", Email: "boss@acme"}), "/org/org-a", nil)
for _, want := range []string{"2 kişi", "Kullanım", "Ekipler, projeler ve sınırlar", "Üyeler",
">yönetici</b>", "kuruluşun tamamı", "son 7 gün", `lang="tr"`} {
for _, want := range []string{"2 kişi", "Kullanım", "Sınırlar", "Üyeler",
">sorumlu</b>", "kuruluşun tamamı", "son 7 gün", `lang="tr"`} {
if !strings.Contains(body, want) {
t.Errorf("the organization page in Turkish lacks %q", want)
}
}
if strings.Contains(body, "Teams, projects and limits") {
if strings.Contains(body, ">Limits<") {
t.Error("the organization page in Turkish still has English headings")
}
}
+2
View File
@@ -7,6 +7,7 @@
"about.nav.install": "Install",
"about.nav.proxy": "Reverse proxy",
"about.nav.download": "Download",
"about.nav.deck": "Presentation",
"about.nav.source": "Source",
"about.nav.gateway": "This gateway",
"about.hero.eyebrow": "Self-hosted LLM gateway · %s",
@@ -14,6 +15,7 @@
"about.hero.lede": "llmgw runs vLLM, SGLang and llama.cpp across your servers and serves them behind one OpenAI-compatible API. It knows which GPUs share an NVLink domain or an InfiniBand fabric, and a model split across machines never straddles two.",
"about.hero.download": "Download",
"about.hero.guide": "Install guide",
"about.hero.deck": "Presentation (PDF)",
"about.hero.oneline_label": "Set up a control plane",
"about.copy": "Copy",
"about.copied": "Copied",
+42 -3
View File
@@ -13,14 +13,13 @@
"org.use_empty": "Nothing used in this window.",
"org.use_scope": "Only what was spent in this organization's projects. What its people do elsewhere — in another organization, or on their own — is not shown.",
"org.limits.title": "Teams, projects and limits",
"org.limits.title": "Limits",
"org.col.spent_30d": "Spent · 30 days",
"org.limits.note": "Limits are set by the cluster's administrators.",
"org.members.title": "Members",
"org.col.member_of": "Member of",
"org.col.state": "State",
"org.role.manager": "manager",
"org.state.pending": "pending",
"org.state.active": "active",
"org.state.suspended": "suspended",
@@ -30,7 +29,47 @@
"org.none.title": "Organizations",
"org.none.sub": "There are none on this cluster",
"org.none.about": "Everyone uses the gateway on their own, under their personal quota. An organization, once there is one, gets a page here for its managers: its use by team, project and person, its limits, and its members.",
"org.none.how": "<a href=\"/admin/users\">Create one on the people page</a>, then make someone its manager with the star on their membership.",
"org.none.how": "<a href=\"/admin/users\">Create one on the people page</a>, then choose manager or org admin on someone's membership of it.",
"org.col.role": "Role here",
"org.col.projects": "Projects",
"org.add.email": "their sign-in address",
"org.add.where": "Where",
"org.add.button": "Add",
"org.add.note": "Anyone with an approved account can be added. Someone new signs up first; the cluster's administrators approve them.",
"org.tree.title": "Teams and projects",
"org.tree.rename": "Rename",
"org.tree.remove": "Remove %s",
"org.tree.remove_confirm": "Remove %s? Only an empty one can be removed.",
"org.tree.remove_team": "Remove",
"org.tree.new_project": "new project",
"org.tree.new_project_in": "New project in %s",
"org.tree.new_team": "new team",
"org.tree.add_team": "Add team",
"org.tree.empty": "No teams yet.",
"org.tree.note": "Click a team's name to rename it. A team or project is removed only when nobody and no key is left in it.",
"org.keys.title": "Keys",
"org.keys.col.key": "Key",
"org.keys.col.owner": "Owner",
"org.keys.col.used": "Last used",
"org.keys.revoke": "Revoke",
"org.keys.revoke_confirm": "Revoke %s…? Whatever uses it stops working.",
"org.keys.empty": "No keys in this organization's projects.",
"org.msg.team_created": "Team %s added.",
"org.msg.project_created": "Project %s added.",
"org.msg.renamed": "Renamed to %s.",
"org.msg.removed": "%s removed.",
"org.msg.added": "%s added to %s.",
"org.msg.taken_out": "Taken out of %s.",
"org.msg.taken_out_keys": "Taken out of %s; %d of their keys there revoked.",
"org.msg.key_revoked": "Key %s… revoked.",
"org.err.not_here": "that is not in this organization",
"org.err.no_account": "there is no account for %s: they sign up first",
"org.err.not_approved": "%s waits for the cluster's administrators to approve them",
"org.err.suspended": "%s is suspended",
"org.err.self_role": "your own role here is changed by the cluster's administrators",
"org.err.self_leave": "you cannot take yourself out of the organization you administer",
"org.err.name": "a name is needed, without a slash",
"org.err.role": "a role is member, manager or org admin",
"org.err.read_tree": "could not read the organization"
}
+6 -5
View File
@@ -139,10 +139,10 @@
"users.state.pending": "pending",
"users.state.suspended": "suspended",
"users.role.manager": "manager",
"users.stop_managing_of": "Stop managing %s",
"users.make_manager_of": "Make manager of %s: can see its page",
"users.stop_managing": "Stop managing",
"users.make_manager": "Make manager",
"users.role.member": "member",
"users.role.admin": "org admin",
"users.role_in": "Role in %s",
"users.role_note": "A manager sees the organization's page; an admin also changes its teams, projects, people and keys.",
"users.take_out_confirm": "Take %s out of %s? Keys they made there are revoked.",
"users.take_out_of": "Take out of %s",
"users.on_own": "on their own",
@@ -164,6 +164,7 @@
"users.msg.added_to": "added to %s",
"users.msg.now_manages": "now manages %s",
"users.msg.no_longer_manages": "no longer manages %s",
"users.msg.now_administers": "now administers %s",
"users.msg.removed_from": "removed from %s",
"users.msg.removed_from_one_key": "removed from %s; 1 key they made there revoked",
"users.msg.removed_from_keys": "removed from %s; %d keys they made there revoked",
@@ -181,7 +182,7 @@
"users.err.about": "%s: %s",
"users.err.self_suspend": "you cannot suspend your own account",
"users.err.choose_where": "choose where they go",
"users.err.manager_role": "a manager is a role in an organization",
"users.err.manager_role": "manager and admin are roles in an organization",
"users.err.nothing_to_leave": "nothing to leave",
"users.err.self_remove": "you cannot remove your own account",
"users.err.choose_org": "choose the organization this team belongs to",
+2
View File
@@ -7,6 +7,7 @@
"about.nav.install": "Kurulum",
"about.nav.proxy": "Ters proxy",
"about.nav.download": "İndir",
"about.nav.deck": "Sunum",
"about.nav.source": "Kaynak kod",
"about.nav.gateway": "Bu ağ geçidi",
"about.hero.eyebrow": "Kendi sunucunuzda çalışan LLM ağ geçidi · %s",
@@ -14,6 +15,7 @@
"about.hero.lede": "llmgw; vLLM, SGLang ve llama.cpp'yi sunucularınızda çalıştırır ve hepsini OpenAI uyumlu tek bir API'nin arkasından sunar. Hangi GPU'ların aynı NVLink alanını ya da aynı InfiniBand fabric'ini paylaştığını bilir; makinelere bölünmüş bir model hiçbir zaman iki fabric'e birden yayılmaz.",
"about.hero.download": "İndir",
"about.hero.guide": "Kurulum kılavuzu",
"about.hero.deck": "Tanıtım sunumu (PDF)",
"about.hero.oneline_label": "Bir kontrol düzlemi kurun",
"about.copy": "Kopyala",
"about.copied": "Kopyalandı",
+42 -3
View File
@@ -12,14 +12,13 @@
"org.use_empty": "Bu dönemde hiçbir şey kullanılmadı.",
"org.use_scope": "Yalnızca bu kuruluşun projelerinde harcananlar. Kişilerinin başka yerlerde — başka bir kuruluşta ya da kendi başlarına — yaptıkları gösterilmez.",
"org.limits.title": "Ekipler, projeler ve sınırlar",
"org.limits.title": "Sınırlar",
"org.col.spent_30d": "Harcanan · 30 gün",
"org.limits.note": "Sınırları kümenin yöneticileri belirler.",
"org.members.title": "Üyeler",
"org.col.member_of": "Üyesi olduğu yer",
"org.col.state": "Durum",
"org.role.manager": "yönetici",
"org.state.pending": "onay bekliyor",
"org.state.active": "etkin",
"org.state.suspended": "askıya alındı",
@@ -29,7 +28,47 @@
"org.none.title": "Kuruluşlar",
"org.none.sub": "Bu kümede hiç kuruluş yok",
"org.none.about": "Herkes ağ geçidini kendi kişisel kotası altında, kendi başına kullanıyor. Bir kuruluş oluşturulduğunda, yöneticileri için burada bir sayfası olur: ekibe, projeye ve kişiye göre kullanımı, sınırları ve üyeleri.",
"org.none.how": "<a href=\"/admin/users\">Kişiler sayfasında bir kuruluş oluşturun</a>, ardından birinin üyeliğindeki yıldızla onu kuruluşun yöneticisi yapın.",
"org.none.how": "<a href=\"/admin/users\">Kişiler sayfasında bir kuruluş oluşturun</a>, ardından birinin oradaki üyeliğinde sorumlu ya da kuruluş yöneticisi rolünü seçin.",
"org.col.role": "Buradaki rolü",
"org.col.projects": "Projeler",
"org.add.email": "giriş adresi",
"org.add.where": "Nereye",
"org.add.button": "Ekle",
"org.add.note": "Hesabı onaylanmış herkes eklenebilir. Yeni biri önce kaydolur; kümenin yöneticileri onu onaylar.",
"org.tree.title": "Ekipler ve projeler",
"org.tree.rename": "Yeniden adlandır",
"org.tree.remove": "%s kaldır",
"org.tree.remove_confirm": "%s kaldırılsın mı? Yalnızca boş olan kaldırılabilir.",
"org.tree.remove_team": "Kaldır",
"org.tree.new_project": "yeni proje",
"org.tree.new_project_in": "%s içinde yeni proje",
"org.tree.new_team": "yeni ekip",
"org.tree.add_team": "Ekip ekle",
"org.tree.empty": "Henüz ekip yok.",
"org.tree.note": "Adını değiştirmek için ekibin adına tıklayın. Bir ekip ya da proje, içinde kimse ve hiçbir anahtar kalmadığında kaldırılabilir.",
"org.keys.title": "Anahtarlar",
"org.keys.col.key": "Anahtar",
"org.keys.col.owner": "Sahibi",
"org.keys.col.used": "Son kullanım",
"org.keys.revoke": "İptal et",
"org.keys.revoke_confirm": "%s… iptal edilsin mi? Onu kullanan her şey çalışmayı bırakır.",
"org.keys.empty": "Bu kuruluşun projelerinde anahtar yok.",
"org.msg.team_created": "%s ekibi eklendi.",
"org.msg.project_created": "%s projesi eklendi.",
"org.msg.renamed": "Yeni adı %s.",
"org.msg.removed": "%s kaldırıldı.",
"org.msg.added": "%s, %s içine eklendi.",
"org.msg.taken_out": "%s içinden çıkarıldı.",
"org.msg.taken_out_keys": "%s içinden çıkarıldı; oradaki %d anahtarı iptal edildi.",
"org.msg.key_revoked": "%s… anahtarı iptal edildi.",
"org.err.not_here": "bu, bu kuruluşta değil",
"org.err.no_account": "%s için bir hesap yok: önce kaydolması gerekiyor",
"org.err.not_approved": "%s, kümenin yöneticilerinin onayını bekliyor",
"org.err.suspended": "%s askıya alınmış",
"org.err.self_role": "buradaki kendi rolünüzü kümenin yöneticileri değiştirir",
"org.err.self_leave": "yönettiğiniz kuruluştan kendinizi çıkaramazsınız",
"org.err.name": "bir ad gerekli, eğik çizgi olmadan",
"org.err.role": "rol üye, sorumlu ya da kuruluş yöneticisidir",
"org.err.read_tree": "kuruluş okunamadı"
}
+6 -5
View File
@@ -135,10 +135,10 @@
"users.state.pending": "onay bekliyor",
"users.state.suspended": "askıda",
"users.role.manager": "sorumlu",
"users.stop_managing_of": "%s sorumluluğunu bırak",
"users.make_manager_of": "%s sorumlusu yap: sayfasını görebilir",
"users.stop_managing": "Sorumluluğu bırak",
"users.make_manager": "Sorumlu yap",
"users.role.member": "üye",
"users.role.admin": "kuruluş yöneticisi",
"users.role_in": "%s içindeki rolü",
"users.role_note": "Sorumlu, kuruluşun sayfasını görür; kuruluş yöneticisi ayrıca takımlarını, projelerini, kişilerini ve anahtarlarını değiştirir.",
"users.take_out_confirm": "%[1]s, %[2]s içinden çıkarılsın mı? Orada oluşturduğu anahtarlar iptal edilir.",
"users.take_out_of": "%s içinden çıkar",
"users.on_own": "tek başına",
@@ -160,6 +160,7 @@
"users.msg.added_to": "%s içine eklendi",
"users.msg.now_manages": "artık %s sorumlusu",
"users.msg.no_longer_manages": "artık %s sorumlusu değil",
"users.msg.now_administers": "artık %s kuruluş yöneticisi",
"users.msg.removed_from": "%s içinden çıkarıldı",
"users.msg.removed_from_one_key": "%s içinden çıkarıldı; orada oluşturduğu 1 anahtar iptal edildi",
"users.msg.removed_from_keys": "%s içinden çıkarıldı; orada oluşturduğu %d anahtar iptal edildi",
@@ -177,7 +178,7 @@
"users.err.about": "%s: %s",
"users.err.self_suspend": "kendi hesabınızı askıya alamazsınız",
"users.err.choose_where": "nereye gideceğini seçin",
"users.err.manager_role": "sorumluluk, bir kuruluştaki roldür",
"users.err.manager_role": "sorumlu ve kuruluş yöneticisi, bir kuruluştaki rollerdir",
"users.err.nothing_to_leave": "çıkılacak bir yer yok",
"users.err.self_remove": "kendi hesabınızı kaldıramazsınız",
"users.err.choose_org": "bu ekibin ait olduğu kuruluşu seçin",
+2 -2
View File
@@ -29,7 +29,7 @@ type OIDCStore interface {
UserByEmail(ctx context.Context, email string) (store.User, error)
CreateUser(ctx context.Context, u store.User) error
SetStaffRole(ctx context.Context, id, role string) error
AddMembership(ctx context.Context, user string, scope store.Scope, id, role, by string, at time.Time) error
EnsureMembership(ctx context.Context, user string, scope store.Scope, id, by string, at time.Time) error
AllOrgs(ctx context.Context) ([]store.Org, error)
AllTeams(ctx context.Context) ([]store.TeamPath, error)
AllProjects(ctx context.Context) ([]store.ProjectPath, error)
@@ -338,7 +338,7 @@ func (p *AccountPages) oidcUser(ctx context.Context, claims map[string]any) (sto
p.log.Warn("a group maps to a place the tree does not have", "group", g, "place", where, "err", err)
continue
}
if err := o.st.AddMembership(ctx, u.ID, scope, id, store.RoleMember, "oidc", time.Now()); err != nil {
if err := o.st.EnsureMembership(ctx, u.ID, scope, id, "oidc", time.Now()); err != nil {
p.log.Warn("could not add a membership from a group", "group", g, "err", err)
}
}
+1 -1
View File
@@ -89,7 +89,7 @@ func (m *oidcStoreMem) SetStaffRole(_ context.Context, id, role string) error {
m.roles[id] = role
return nil
}
func (m *oidcStoreMem) AddMembership(_ context.Context, user string, scope store.Scope, id, _, _ string, _ time.Time) error {
func (m *oidcStoreMem) EnsureMembership(_ context.Context, user string, scope store.Scope, id, _ string, _ time.Time) error {
m.joined = append(m.joined, user+"@"+string(scope)+":"+id)
return nil
}
+296
View File
@@ -0,0 +1,296 @@
package web
import (
"context"
"errors"
"net/http"
"slices"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// An organization's admin changes it from its own page.
//
// What they may change is what is theirs: the teams and projects in their
// organization, who of the cluster's people is in them and in what role, and
// the keys spent there. What stays with the cluster's staff is everything that
// reaches past the organization or into the cluster's capacity: its limits and
// budgets, approving or suspending an account, making anyone staff, and the
// organization's own name and existence. A person not yet approved is not
// added here either: approval says who may use the cluster at all.
// OrgEditor is what an organization's admin changes through.
type OrgEditor interface {
AdministersOrg(ctx context.Context, userID, orgID string) (bool, error)
UserByEmail(ctx context.Context, email string) (store.User, error)
CreateTeam(ctx context.Context, orgID, name string) (string, error)
CreateProject(ctx context.Context, teamID, name string) (string, error)
RenameTeam(ctx context.Context, id, name string) error
RenameProject(ctx context.Context, id, name string) error
DeleteTeam(ctx context.Context, id string) error
DeleteProject(ctx context.Context, id string) error
AddMembership(ctx context.Context, userID string, scope store.Scope, scopeID, role, by string, at time.Time) error
RemoveMembership(ctx context.Context, userID string, scope store.Scope, scopeID string, at time.Time) (int, error)
AllKeys(ctx context.Context) ([]store.KeyRow, error)
RevokeKey(ctx context.Context, id string, at time.Time) error
}
func init() {
registerKeys(
"org.msg.team_created", "org.msg.project_created", "org.msg.renamed", "org.msg.removed",
"org.msg.added", "org.msg.taken_out", "org.msg.taken_out_keys", "org.msg.key_revoked",
"org.err.not_here", "org.err.no_account", "org.err.not_approved", "org.err.suspended",
"org.err.self_role", "org.err.self_leave", "org.err.name", "org.err.role",
)
}
// orgChange is one change an organization's admin makes, given the
// organization as it is.
type orgChange func(ctx context.Context, r *http.Request, by store.User, org store.TreeOrg) (notice, error)
// registerOrgAdmin mounts the changes, when the store makes them.
func (p *AccountPages) registerOrgAdmin(mux *http.ServeMux, s OrgViewer, e OrgEditor, priced Priced) {
act := func(do orgChange) http.HandlerFunc { return p.orgAction(s, e, priced, do) }
mux.HandleFunc("POST /org/{id}/teams", act(func(ctx context.Context, r *http.Request, _ store.User, o store.TreeOrg) (notice, error) {
name, err := orgName(r)
if err != nil {
return notice{}, err
}
if _, err := e.CreateTeam(ctx, o.ID, name); err != nil {
return notice{}, err
}
return say("org.msg.team_created", name), nil
}))
mux.HandleFunc("POST /org/{id}/projects", act(func(ctx context.Context, r *http.Request, _ store.User, o store.TreeOrg) (notice, error) {
team := r.FormValue("team")
if !orgHas(o, store.ScopeTeam, team) {
return notice{}, refuse("org.err.not_here")
}
name, err := orgName(r)
if err != nil {
return notice{}, err
}
if _, err := e.CreateProject(ctx, team, name); err != nil {
return notice{}, err
}
return say("org.msg.project_created", name), nil
}))
mux.HandleFunc("POST /org/{id}/{kind}/{sid}/rename", act(func(ctx context.Context, r *http.Request, _ store.User, o store.TreeOrg) (notice, error) {
kind, sid := store.Scope(r.PathValue("kind")), r.PathValue("sid")
if (kind != store.ScopeTeam && kind != store.ScopeProject) || !orgHas(o, kind, sid) {
return notice{}, refuse("org.err.not_here")
}
name, err := orgName(r)
if err != nil {
return notice{}, err
}
if kind == store.ScopeTeam {
err = e.RenameTeam(ctx, sid, name)
} else {
err = e.RenameProject(ctx, sid, name)
}
if err != nil {
return notice{}, err
}
return say("org.msg.renamed", name), nil
}))
mux.HandleFunc("POST /org/{id}/{kind}/{sid}/delete", act(func(ctx context.Context, r *http.Request, _ store.User, o store.TreeOrg) (notice, error) {
kind, sid := store.Scope(r.PathValue("kind")), r.PathValue("sid")
if (kind != store.ScopeTeam && kind != store.ScopeProject) || !orgHas(o, kind, sid) {
return notice{}, refuse("org.err.not_here")
}
name := orgPlaceName(o, kind, sid)
var err error
if kind == store.ScopeTeam {
err = e.DeleteTeam(ctx, sid)
} else {
err = e.DeleteProject(ctx, sid)
}
if err != nil {
return notice{}, err
}
return say("org.msg.removed", name), nil
}))
// A person is added by their address: the page does not list the whole
// cluster's accounts to someone who administers one organization.
mux.HandleFunc("POST /org/{id}/members", act(func(ctx context.Context, r *http.Request, by store.User, o store.TreeOrg) (notice, error) {
scope, sid, ok := strings.Cut(r.FormValue("where"), ":")
if !ok || !orgHas(o, store.Scope(scope), sid) {
return notice{}, refuse("org.err.not_here")
}
email := strings.TrimSpace(r.FormValue("email"))
u, err := e.UserByEmail(ctx, email)
if errors.Is(err, store.ErrNotFound) {
return notice{}, refuse("org.err.no_account", email)
}
if err != nil {
return notice{}, err
}
switch u.State {
case store.UserPending:
return notice{}, refuse("org.err.not_approved", u.Email)
case store.UserSuspended:
return notice{}, refuse("org.err.suspended", u.Email)
}
if err := e.AddMembership(ctx, u.ID, store.Scope(scope), sid, store.RoleMember, by.Email, time.Now()); err != nil {
return notice{}, err
}
return say("org.msg.added", u.Email, orgPlaceName(o, store.Scope(scope), sid)), nil
}))
mux.HandleFunc("POST /org/{id}/members/{uid}/role", act(func(ctx context.Context, r *http.Request, by store.User, o store.TreeOrg) (notice, error) {
uid, role := r.PathValue("uid"), r.FormValue("role")
if !slices.Contains(store.MembershipRoles, role) {
return notice{}, refuse("org.err.role")
}
if uid == by.ID && !by.IsAdmin {
// Giving up one's own admin role by a misclick would leave
// nobody able to give it back but the cluster's staff.
return notice{}, refuse("org.err.self_role")
}
if err := e.AddMembership(ctx, uid, store.ScopeOrg, o.ID, role, by.Email, time.Now()); err != nil {
return notice{}, err
}
return roleNotice(role, o.Name), nil
}))
mux.HandleFunc("POST /org/{id}/members/{uid}/leave", act(func(ctx context.Context, r *http.Request, by store.User, o store.TreeOrg) (notice, error) {
uid := r.PathValue("uid")
scope, sid, ok := strings.Cut(r.FormValue("where"), ":")
if !ok || !orgHas(o, store.Scope(scope), sid) {
return notice{}, refuse("org.err.not_here")
}
if uid == by.ID && store.Scope(scope) == store.ScopeOrg && !by.IsAdmin {
return notice{}, refuse("org.err.self_leave")
}
where := orgPlaceName(o, store.Scope(scope), sid)
revoked, err := e.RemoveMembership(ctx, uid, store.Scope(scope), sid, time.Now())
if err != nil {
return notice{}, err
}
if revoked > 0 {
return say("org.msg.taken_out_keys", where, revoked), nil
}
return say("org.msg.taken_out", where), nil
}))
mux.HandleFunc("POST /org/{id}/keys/{kid}/revoke", act(func(ctx context.Context, r *http.Request, _ store.User, o store.TreeOrg) (notice, error) {
kid := r.PathValue("kid")
keys, err := e.AllKeys(ctx)
if err != nil {
return notice{}, err
}
i := slices.IndexFunc(keys, func(k store.KeyRow) bool { return k.ID == kid })
if i < 0 || !orgHas(o, store.ScopeProject, keys[i].ProjectID) {
return notice{}, refuse("org.err.not_here")
}
if err := e.RevokeKey(ctx, kid, time.Now()); err != nil {
return notice{}, err
}
return say("org.msg.key_revoked", keys[i].Prefix), nil
}))
}
// orgAction is the guard every change on an organization's page shares: a
// session, the organization's admin role (or the cluster's), the form's CSRF
// token, an audit record, and the page drawn again with what happened.
func (p *AccountPages) orgAction(s OrgViewer, e OrgEditor, priced Priced, do orgChange) http.HandlerFunc {
return p.orgGuard(func(w http.ResponseWriter, r *http.Request, user store.User) {
id := r.PathValue("id")
if !user.IsAdmin {
ok, err := e.AdministersOrg(r.Context(), user.ID, id)
if err != nil || !ok {
// 404, as for a page they may not see: a manager learns
// nothing from trying.
http.NotFound(w, r)
return
}
}
sess, _, _ := p.session(r)
if !p.accounts.CheckCSRF(sess, r.FormValue("csrf")) {
badForm(w, r)
return
}
p.audited(w, r, user, func(w http.ResponseWriter, r *http.Request) {
l := langOf(r)
org, ok := orgIn(r.Context(), s, id)
if !ok {
http.NotFound(w, r)
return
}
msg, err := do(r.Context(), r, user, org)
if err != nil {
err = plainError(err)
p.log.Warn("a change to an organization was refused", "by", user.Email, "org", org.Name,
"path", r.URL.Path, "err", err)
noteAudit(r, org.Name, "", err.Error())
p.renderOrg(w, r, s, user, id, priced, "", errorIn(l, err))
return
}
p.log.Info("an organization was changed", "by", user.Email, "org", org.Name,
"what", msg.in(english))
noteAudit(r, org.Name, msg.in(english), "")
p.renderOrg(w, r, s, user, id, priced, msg.in(l), "")
})
})
}
// orgIn finds an organization in the tree.
func orgIn(ctx context.Context, s OrgViewer, id string) (store.TreeOrg, bool) {
tree, err := s.Tree(ctx)
if err != nil {
return store.TreeOrg{}, false
}
for _, o := range tree {
if o.ID == id {
return o, true
}
}
return store.TreeOrg{}, false
}
// orgHas reports whether a place is the organization or lies in it. Every
// change checks it: an id in a form is the sender's to choose.
func orgHas(o store.TreeOrg, scope store.Scope, id string) bool {
return orgPlaceName(o, scope, id) != ""
}
// orgPlaceName is a place in the organization by its path, "" when it is not
// in it.
func orgPlaceName(o store.TreeOrg, scope store.Scope, id string) string {
if id == "" {
return ""
}
if scope == store.ScopeOrg {
if id == o.ID {
return o.Name
}
return ""
}
for _, t := range o.Teams {
if scope == store.ScopeTeam && t.ID == id {
return o.Name + " / " + t.Name
}
for _, pr := range t.Projects {
if scope == store.ScopeProject && pr.ID == id {
return o.Name + " / " + t.Name + " / " + pr.Name
}
}
}
return ""
}
func orgName(r *http.Request) (string, error) {
name := strings.TrimSpace(r.FormValue("name"))
if name == "" || strings.Contains(name, "/") {
return "", refuse("org.err.name")
}
return name, nil
}
+210
View File
@@ -0,0 +1,210 @@
package web
import (
"context"
"io"
"log/slog"
"net/http"
"net/url"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// fakeOrgEditor is fakeOrgs that also changes: u1 administers Acme, u4
// manages it, and what each change did is written down.
type fakeOrgEditor struct {
fakeOrgs
admins map[string]string // org -> admin user id
did []string
revoked []string
}
func (f *fakeOrgEditor) ManagesOrg(_ context.Context, user, org string) (bool, error) {
return f.admins[org] == user || f.managers[org] == user, nil
}
func (f *fakeOrgEditor) ManagedOrgs(ctx context.Context, user string) ([]store.Org, error) {
var out []store.Org
for _, o := range []store.Org{{ID: "org-a", Name: "Acme"}, {ID: "org-s", Name: "Scania"}} {
if ok, _ := f.ManagesOrg(ctx, user, o.ID); ok {
out = append(out, o)
}
}
return out, nil
}
func (f *fakeOrgEditor) AdministersOrg(_ context.Context, user, org string) (bool, error) {
return f.admins[org] == user, nil
}
func (f *fakeOrgEditor) UserByEmail(_ context.Context, email string) (store.User, error) {
switch email {
case "new@acme":
return store.User{ID: "u5", Email: email, State: store.UserActive}, nil
case "waiting@acme":
return store.User{ID: "u6", Email: email, State: store.UserPending}, nil
}
return store.User{}, store.ErrNotFound
}
func (f *fakeOrgEditor) CreateTeam(_ context.Context, org, name string) (string, error) {
f.did = append(f.did, "team "+org+" "+name)
return "t9", nil
}
func (f *fakeOrgEditor) CreateProject(_ context.Context, team, name string) (string, error) {
f.did = append(f.did, "project "+team+" "+name)
return "p9", nil
}
func (f *fakeOrgEditor) RenameTeam(_ context.Context, id, name string) error {
f.did = append(f.did, "rename team "+id+" "+name)
return nil
}
func (f *fakeOrgEditor) RenameProject(_ context.Context, id, name string) error {
f.did = append(f.did, "rename project "+id+" "+name)
return nil
}
func (f *fakeOrgEditor) DeleteTeam(_ context.Context, id string) error {
f.did = append(f.did, "delete team "+id)
return nil
}
func (f *fakeOrgEditor) DeleteProject(_ context.Context, id string) error {
f.did = append(f.did, "delete project "+id)
return nil
}
func (f *fakeOrgEditor) AddMembership(_ context.Context, user string, scope store.Scope, id, role, _ string, _ time.Time) error {
f.did = append(f.did, "join "+user+" "+string(scope)+":"+id+" "+role)
return nil
}
func (f *fakeOrgEditor) RemoveMembership(_ context.Context, user string, scope store.Scope, id string, _ time.Time) (int, error) {
f.did = append(f.did, "leave "+user+" "+string(scope)+":"+id)
return 0, nil
}
func (f *fakeOrgEditor) AllKeys(context.Context) ([]store.KeyRow, error) {
return []store.KeyRow{
{Key: store.Key{ID: "k-acme", Prefix: "sk-a", ProjectID: "p1"}, ProjectPath: "Acme / ml / training"},
{Key: store.Key{ID: "k-scania", Prefix: "sk-s", ProjectID: "p-s"}},
}, nil
}
func (f *fakeOrgEditor) RevokeKey(_ context.Context, id string, _ time.Time) error {
f.revoked = append(f.revoked, id)
return nil
}
func orgAdminPage(t *testing.T, user store.User) (*http.ServeMux, *fakeOrgEditor) {
t.Helper()
f := &fakeOrgEditor{
fakeOrgs: fakeOrgs{managers: map[string]string{"org-a": "u4"}},
admins: map[string]string{"org-a": "u1"},
}
p := NewAccountPages(fakeAccounts{user: user}, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
mux := http.NewServeMux()
p.RegisterOrgPages(mux, f, nil)
return mux, f
}
func form(kv ...string) url.Values {
v := url.Values{"csrf": {"tok"}}
for i := 0; i+1 < len(kv); i += 2 {
v.Set(kv[i], kv[i+1])
}
return v
}
func TestAnOrganizationsAdminIsGivenTheFormsAndAManagerIsNot(t *testing.T) {
admin, _ := orgAdminPage(t, store.User{ID: "u1", Email: "boss@acme"})
body := mirrorGet(t, admin, "/org/org-a").Body.String()
for _, want := range []string{`action="/org/org-a/teams"`, `action="/org/org-a/members"`,
`action="/org/org-a/keys/k-acme/revoke"`, "sk-a"} {
if !strings.Contains(body, want) {
t.Errorf("the admin's page lacks %q", want)
}
}
if strings.Contains(body, "k-scania") {
t.Error("a key of another organization is on the page")
}
manager, f := orgAdminPage(t, store.User{ID: "u4", Email: "watch@acme"})
w := mirrorGet(t, manager, "/org/org-a")
if w.Code != http.StatusOK {
t.Fatalf("the manager lost their page: %d", w.Code)
}
if strings.Contains(w.Body.String(), "<form method=\"post\" action=\"/org/") {
t.Error("a manager is shown forms they cannot send")
}
if w := mirrorPost(t, manager, "/org/org-a/teams", form("name", "x")); w.Code != http.StatusNotFound {
t.Errorf("a manager changed the organization: %d", w.Code)
}
if len(f.did) != 0 {
t.Errorf("a manager's change reached the store: %v", f.did)
}
}
func TestAnOrganizationsAdminChangesOnlyTheirOwn(t *testing.T) {
mux, f := orgAdminPage(t, store.User{ID: "u1", Email: "boss@acme"})
if w := mirrorPost(t, mux, "/org/org-a/teams", form("name", "vision")); w.Code != http.StatusOK ||
!strings.Contains(w.Body.String(), "Team vision added") {
t.Errorf("adding a team: %d", w.Code)
}
if w := mirrorPost(t, mux, "/org/org-s/teams", form("name", "x")); w.Code != http.StatusNotFound {
t.Errorf("an admin of Acme added a team to Scania: %d", w.Code)
}
// Ids in a form are the sender's to choose: a team of another
// organization, named on Acme's page, is refused.
mirrorPost(t, mux, "/org/org-a/projects", form("team", "t-scania", "name", "x"))
mirrorPost(t, mux, "/org/org-a/members", form("email", "new@acme", "where", "org:org-s"))
mirrorPost(t, mux, "/org/org-a/keys/k-scania/revoke", form())
mirrorPost(t, mux, "/org/org-a/project/p-scania/delete", form())
if len(f.revoked) != 0 {
t.Errorf("a key of another organization was revoked: %v", f.revoked)
}
mirrorPost(t, mux, "/org/org-a/projects", form("team", "t1", "name", "eval"))
mirrorPost(t, mux, "/org/org-a/members", form("email", "new@acme", "where", "project:p1"))
mirrorPost(t, mux, "/org/org-a/members/u2/role", form("role", "manager"))
mirrorPost(t, mux, "/org/org-a/keys/k-acme/revoke", form())
want := []string{"team org-a vision", "project t1 eval", "join u5 project:p1 member", "join u2 org:org-a manager"}
if strings.Join(f.did, "|") != strings.Join(want, "|") {
t.Errorf("the store was asked\n %v\nwant\n %v", f.did, want)
}
if len(f.revoked) != 1 || f.revoked[0] != "k-acme" {
t.Errorf("revoked %v", f.revoked)
}
}
func TestAnOrganizationsAdminAddsOnlyApprovedPeople(t *testing.T) {
mux, f := orgAdminPage(t, store.User{ID: "u1", Email: "boss@acme"})
for email, says := range map[string]string{
"nobody@acme": "there is no account for nobody@acme",
"waiting@acme": "waits for the cluster",
} {
w := mirrorPost(t, mux, "/org/org-a/members", form("email", email, "where", "org:org-a"))
if !strings.Contains(w.Body.String(), says) {
t.Errorf("%s: the page does not say %q", email, says)
}
}
if len(f.did) != 0 {
t.Errorf("someone not approved was added: %v", f.did)
}
}
func TestAnOrganizationsAdminCannotLockThemselvesOut(t *testing.T) {
mux, f := orgAdminPage(t, store.User{ID: "u1", Email: "boss@acme"})
mirrorPost(t, mux, "/org/org-a/members/u1/role", form("role", "member"))
mirrorPost(t, mux, "/org/org-a/members/u1/leave", form("where", "org:org-a"))
if len(f.did) != 0 {
t.Errorf("an admin took their own role away: %v", f.did)
}
}
func TestAViewerIsShownNoFormsOnThePeoplePage(t *testing.T) {
for role, wantForms := range map[string]bool{store.RoleViewer: false, store.RoleAdmin: true} {
u := store.User{ID: "s", Email: "s@x", Role: role, IsAdmin: role == store.RoleAdmin}
p := NewAccountPages(fakeAccounts{user: u}, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
mux := http.NewServeMux()
p.RegisterAdminUsers(mux, &fakeAdmin{})
body := mirrorGet(t, mux, "/admin/users").Body.String()
if got := strings.Contains(body, `method="post" action="/admin`); got != wantForms {
t.Errorf("%s: forms on the page = %v, want %v", role, got, wantForms)
}
}
}
+72 -10
View File
@@ -10,7 +10,8 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// OrgViewer is what an organization's page reads. Nothing on it writes.
// OrgViewer is what an organization's page reads. What its admin changes
// goes through OrgEditor, in org_admin.go.
type OrgViewer interface {
ManagesOrg(ctx context.Context, userID, orgID string) (bool, error)
ManagedOrgs(ctx context.Context, userID string) ([]store.Org, error)
@@ -27,12 +28,18 @@ type OrgViewer interface {
// orgMember is a person as an organization's page lists them: once, with
// every place in the organization they are a member of.
type orgMember struct {
ID string
Email string
State store.UserState
Places []orgPlace
// OrgRole is their role on the organization itself, "" when they are
// only in its teams or projects.
OrgRole string
}
type orgPlace struct{ Where, Role string }
// orgPlace is one place a member is in: its path within the organization,
// the role there, and the "scope:id" a form names it by.
type orgPlace struct{ Where, Role, Value string }
// A member's state is a word the page shows and a class it styles by.
func init() {
@@ -58,6 +65,16 @@ type orgPageData struct {
Members []orgMember
People int
// Edit is whether this person may change the organization; the rest is
// what the forms for that need.
Edit bool
Me string // the person looking, whose own role is not theirs to change
CSRF string
Teams []store.TreeTeam
Places []scopeOption
Keys []keyRow
Message string
Error string
Version string
Now string
@@ -66,12 +83,16 @@ type orgPageData struct {
// RegisterOrgPages mounts /org: an organization's own view of itself, for the
// people who manage it.
//
// Read-only on purpose. A manager sees their organization's use, members and
// limits — what they are accountable for — without being made an administrator
// of the whole cluster, and without anything on the page that changes who may
// do what. That stays with the cluster's administrators.
// A manager sees their organization's use, members and limits — what they are
// accountable for — without being made an administrator of the whole cluster.
// An organization's admin also changes it here (org_admin.go); its limits stay
// with the cluster's administrators.
func (p *AccountPages) RegisterOrgPages(mux *http.ServeMux, s OrgViewer, priced Priced) {
p.orgs = s
if e, ok := s.(OrgEditor); ok {
p.orgEditor = e
p.registerOrgAdmin(mux, s, e, priced)
}
mux.HandleFunc("GET /org", p.orgGuard(func(w http.ResponseWriter, r *http.Request, user store.User) {
if p.toPeople(w, r, user, "") {
return
@@ -107,7 +128,7 @@ func (p *AccountPages) RegisterOrgPages(mux *http.ServeMux, s OrgViewer, priced
return
}
}
p.renderOrg(w, r, s, user, id, priced)
p.renderOrg(w, r, s, user, id, priced, "", "")
}))
}
@@ -149,7 +170,7 @@ func (p *AccountPages) orgGuard(next func(http.ResponseWriter, *http.Request, st
}
func (p *AccountPages) renderOrg(w http.ResponseWriter, r *http.Request, s OrgViewer,
user store.User, orgID string, priced Priced,
user store.User, orgID string, priced Priced, msg, errMsg string,
) {
ctx := r.Context()
by := store.Grouping(r.URL.Query().Get("by"))
@@ -162,6 +183,7 @@ func (p *AccountPages) renderOrg(w http.ResponseWriter, r *http.Request, s OrgVi
Nav: navFrom(r, "org"),
By: string(by), Window: win.Key,
Priced: priced != nil && priced.AnythingPriced(),
Message: msg, Error: errMsg,
Version: p.version,
Now: time.Now().UTC().Format(time.RFC3339),
}
@@ -217,10 +239,13 @@ func (p *AccountPages) renderOrg(w http.ResponseWriter, r *http.Request, s OrgVi
om := byPerson[m.UserID]
if om == nil {
u := byID[m.UserID]
om = &orgMember{Email: u.Email, State: u.State}
om = &orgMember{ID: u.ID, Email: u.Email, State: u.State}
byPerson[m.UserID] = om
}
om.Places = append(om.Places, orgPlace{Where: where, Role: m.Role})
om.Places = append(om.Places, orgPlace{Where: where, Role: m.Role, Value: string(m.Scope) + ":" + m.ScopeID})
if m.Scope == store.ScopeOrg {
om.OrgRole = m.Role
}
}
for _, om := range byPerson {
d.Members = append(d.Members, *om)
@@ -228,6 +253,8 @@ func (p *AccountPages) renderOrg(w http.ResponseWriter, r *http.Request, s OrgVi
sort.Slice(d.Members, func(i, j int) bool { return d.Members[i].Email < d.Members[j].Email })
d.People = len(byPerson)
p.orgEditing(r, &d, user, mine)
rows, err := s.UsageBucketsIn(ctx, by, orgID, time.Now().Add(-win.Span), win.Width)
if err != nil {
p.log.Error("could not read an organization's usage", "org", orgID, "err", err)
@@ -236,3 +263,38 @@ func (p *AccountPages) renderOrg(w http.ResponseWriter, r *http.Request, s OrgVi
d.Requests, d.Tokens, d.Spend, d.Rows = usageChartsIn(l, rows)
render(w, r, "org", d)
}
// orgEditing fills what the page's forms need, when this person may change
// the organization.
func (p *AccountPages) orgEditing(r *http.Request, d *orgPageData, user store.User, mine []store.TreeOrg) {
ctx, e := r.Context(), p.orgEditor
if e == nil || len(mine) != 1 {
return
}
o := mine[0]
if !user.IsAdmin {
if ok, err := e.AdministersOrg(ctx, user.ID, o.ID); err != nil || !ok {
return
}
}
d.Edit, d.Teams, d.Me = true, o.Teams, user.ID
d.CSRF = csrfFor(r)
d.Places = append(d.Places, scopeOption{Value: "org:" + o.ID, Label: o.Name})
for _, t := range o.Teams {
d.Places = append(d.Places, scopeOption{Value: "team:" + t.ID, Label: o.Name + " / " + t.Name})
for _, pr := range t.Projects {
d.Places = append(d.Places, scopeOption{Value: "project:" + pr.ID, Label: o.Name + " / " + t.Name + " / " + pr.Name})
}
}
keys, err := e.AllKeys(ctx)
if err != nil {
p.log.Error("could not read the keys", "err", err)
return
}
now := time.Now()
for _, k := range keys {
if orgHas(o, store.ScopeProject, k.ProjectID) && k.RevokedAt.IsZero() {
d.Keys = append(d.Keys, keyRow{KeyRow: k, State: keyState(k.Key, now)})
}
}
}
+34 -2
View File
@@ -24,10 +24,42 @@ func staffMay(u store.User, r *http.Request) bool {
case store.RoleViewer:
return read
case store.RoleOperator:
return read || strings.HasPrefix(path, "/admin/mirror/") ||
strings.HasPrefix(path, "/admin/nodes/") && strings.HasSuffix(path, "/engines")
return read || staffMayChange(u.Role, path)
case store.RoleBilling:
return read && (path == "/admin/usage" || path == "/admin/quotas")
}
return false
}
// staffMayChange reports whether a staff role may send a change to an
// administration path. It is staffMay's rule for everything but reading, and
// what the pages ask before they draw a form, so nobody is shown a form whose
// answer would be "not found".
func staffMayChange(role, path string) bool {
switch role {
case store.RoleAdmin:
return true
case store.RoleOperator:
return strings.HasPrefix(path, "/admin/mirror/") ||
strings.HasPrefix(path, "/admin/nodes/") && strings.HasSuffix(path, "/engines")
}
return false
}
// csrfIfMay is the form token for a page whose forms send to path, or ""
// when this person may not send them: a page draws a form only where it has a
// token for it, so a viewer is drawn the page without its forms.
func csrfIfMay(r *http.Request, path string) string {
if !navFrom(r, "").May(path) {
return ""
}
return csrfFor(r)
}
// May reports whether the person this menu is for may send a change to an
// administration path: a template asks it before drawing a form, with the
// form's action ("/admin/mirror/fetch"); a part of the path that varies may be
// anything ("/admin/nodes/x/engines").
func (n Nav) May(path string) bool {
return n.Admin && staffMayChange(n.Role, path)
}
+36 -10
View File
@@ -52,19 +52,30 @@ footer { margin-top: 3rem; padding-top: 1rem; border-top: 1px solid var(--line);
button {
font: inherit; font-weight: 600; cursor: pointer;
background: var(--accent); color: var(--bg); border: none;
border-radius: 6px; padding: .55rem 1.1rem;
border-radius: 6px; padding: .4rem .9rem; font-size: .875rem;
}
button:hover { filter: brightness(1.1); }
ul { padding-left: 1.2rem; }
li { margin: .25rem 0; }
a { color: var(--accent); }
.wrap.narrow { max-width: 30rem; }
label { display: block; margin: .9rem 0; font-size: .9rem; font-weight: 600; }
/* Forms are compact: a label sits close to its field, a field is as wide as
what goes in it rather than the page, and a short form fits on one line
(form.row). The simplest form used to take a screen's height. */
label { display: block; margin: .5rem 0; font-size: .85rem; font-weight: 600; }
input[type=text], input[type=email], input[type=password], input[type=search], input[type=number] {
display: block; width: 100%; margin-top: .3rem; font: inherit; font-weight: 400;
padding: .5rem .6rem; border: 1px solid var(--line); border-radius: 6px;
display: block; width: 100%; max-width: 30rem; margin-top: .2rem; font: inherit; font-size: .9rem;
font-weight: 400; padding: .32rem .5rem; border: 1px solid var(--line); border-radius: 6px;
background: var(--card); color: var(--fg);
}
input[type=number] { max-width: 10rem; }
form.row { display: flex; flex-wrap: wrap; gap: .4rem .6rem; align-items: flex-end; margin: .5rem 0; }
form.row label { margin: 0; }
form.row input[type=text], form.row input[type=email], form.row input[type=password],
form.row input[type=number], form.row select { width: auto; min-width: 12rem; margin: 0; }
form.row input[type=number] { min-width: 6rem; width: 7rem; }
form.row > button, form.row label > button { margin: 0; }
form.row .grow { flex: 1 1 16rem; }
input:focus { outline: 2px solid var(--accent); outline-offset: 1px; }
.error { color: var(--bad); border: 1px solid var(--bad); border-radius: 6px;
padding: .6rem .8rem; background: var(--card); }
@@ -74,7 +85,7 @@ button.link { background: none; color: var(--accent); padding: 0;
button.link.danger { color: var(--bad); }
.ok { color: var(--ok); border: 1px solid var(--ok); border-radius: 6px;
padding: .6rem .8rem; background: var(--card); }
select { font: inherit; padding: .35rem .4rem; border: 1px solid var(--line);
select { font: inherit; font-size: .9rem; padding: .28rem .4rem; border: 1px solid var(--line);
border-radius: 6px; background: var(--card); color: var(--fg); }
.chart-wrap { position: relative; margin: .5rem 0 1.5rem; }
.chart { width: 100%; height: 120px; display: block;
@@ -277,8 +288,11 @@ tr:has(+ .url-row) td, tr:has(+ .error-row) td { border-bottom: none; }
/* The fetch form's three short fields sit on one line on a wide screen and
stack on a narrow one, because two of them are a word long and a full-width
input for "2026-09" reads as though more is wanted. */
.field-row { display: flex; gap: 1rem; flex-wrap: wrap; }
.field-row label { flex: 1 1 12rem; }
.field-row { display: flex; gap: .4rem .75rem; flex-wrap: wrap; align-items: flex-end; }
.field-row label { flex: 0 1 12rem; margin: .3rem 0; }
.field-row label.short { flex: 0 0 7.5rem; }
.field-row label.grow { flex: 1 1 14rem; }
.field-row .field-gap { flex: 0 0 1rem; }
.opt { font-weight: 400; color: var(--muted); font-size: .8rem; }
/* The row whose quota form is open. Tinted rather than outlined: a border
@@ -334,6 +348,8 @@ details.edit > summary::-webkit-details-marker { display: none; }
details.edit[open] > summary { color: var(--muted); }
details.edit form { display: flex; gap: .4rem; align-items: center; margin: .35rem 0; }
details.edit input[type=text] { width: 14rem; }
details.edit form.q-form { display: block; }
details.edit form.q-form input[type=text] { width: 100%; }
details.project { margin: .45rem 0; border: 1px solid var(--line); border-radius: 6px;
background: var(--bg); }
details.project > summary { cursor: pointer; padding: .45rem .7rem; display: flex;
@@ -418,10 +434,11 @@ table.quota-tree tr.q-depth-0 .q-name { font-weight: 600; }
.q-meter.hot i { background: var(--bad); }
.q-pct { font-size: .78rem; color: var(--muted); }
.q-act { text-align: right; white-space: nowrap; }
.q-form { max-width: 46rem; }
.q-form { max-width: 62rem; }
.q-form .field-row { align-items: flex-end; }
.q-form .field-row label { flex: 1 1 8rem; }
.q-form .field-row label { flex: 0 1 8rem; }
.q-form .field-row label.q-period { flex: 0 0 auto; }
.q-form .field-row label.grow { flex: 1 1 14rem; }
.q-form-title { margin: .2rem 0 .4rem; font-size: .88rem; }
.q-form-actions { display: flex; gap: 1rem; align-items: center; margin-top: .6rem; }
.q-form-actions .danger { margin-left: auto; }
@@ -757,6 +774,7 @@ table.gpu-list { width: auto; flex: 1 1 22rem; }
form.add-to { display: flex; gap: .35rem; align-items: center; flex-wrap: nowrap; }
form.add-to select { width: auto; max-width: 13rem; margin: 0; padding: .25rem .4rem; font-size: .8rem; }
form.add-to button { padding: .25rem .6rem; font-size: .78rem; }
form.add-to input[type=text] { width: 11rem; margin: 0; padding: .22rem .45rem; font-size: .8rem; }
details.row-menu { position: relative; display: inline-block; }
details.row-menu > summary { list-style: none; cursor: pointer; padding: .1rem .55rem; border-radius: 6px;
color: var(--muted); font-weight: 700; letter-spacing: .05em; user-select: none; }
@@ -813,7 +831,8 @@ details.chat-settings output { font-variant-numeric: tabular-nums; font-weight:
.copyable { position: relative; }
.copyable pre { margin-bottom: .25rem; overflow-x: auto; }
.copyable button { margin-right: .75rem; font-size: .8rem; }
.connect-form .field-row label { flex: 1 1 12rem; }
.connect-form .field-row label { flex: 0 1 auto; }
.connect-form .field-row > button { margin: .3rem 0; }
table.connect-fields th { white-space: nowrap; }
/* The card opened beneath a held version on the mirror page. */
@@ -941,3 +960,10 @@ a.sel-link.small { font-size: .78rem; color: var(--muted); margin-left: .4rem; }
.meshes-picture input { display: block; margin-top: .3rem; font-weight: 400; }
.meshes-preview { max-width: min(100%, 16rem); max-height: 16rem; border: 1px solid var(--line); border-radius: 6px; }
.meshes-viewer { display: block; width: 100%; height: 22rem; background: var(--card); }
/* A role chosen inside a membership chip. */
select.chip-sel { font-size: .75rem; padding: 0 .2rem; margin-left: .3rem; border-radius: 4px; }
fieldset { border: 1px solid var(--line); border-radius: 6px; padding: .3rem .8rem .5rem; margin: .5rem 0; }
legend { font-size: .82rem; font-weight: 600; padding: 0 .3rem; }
fieldset label.check { margin: .2rem 0; font-weight: 400; font-size: .85rem; }
+2
View File
@@ -21,6 +21,7 @@
<a href="#install">{{t "about.nav.install"}}</a>
<a href="#proxy">{{t "about.nav.proxy"}}</a>
<a href="#download">{{t "about.nav.download"}}</a>
<a href="{{.DeckURL}}" type="application/pdf">{{t "about.nav.deck"}}</a>
<a href="https://gitea.mynodes.xyz/ilker/llmgw">{{t "about.nav.source"}}</a>
{{if not .Site}}<a href="/">{{t "about.nav.gateway"}}</a>{{end}}
</nav>
@@ -34,6 +35,7 @@
<div class="cta">
<a class="btn" href="#download">{{t "about.hero.download"}}{{with .Release}} {{.}}{{end}}</a>
<a class="btn ghost" href="{{.InstallURL}}">{{t "about.hero.guide"}}</a>
<a class="btn ghost" href="{{.DeckURL}}" type="application/pdf">{{t "about.hero.deck"}}</a>
</div>
<div class="oneline" aria-label="{{t "about.hero.oneline_label"}}">
<code><span class="p">$ </span>sudo llmgw server init --admin you@example.com</code>
+3 -3
View File
@@ -119,8 +119,8 @@
{{else}}<input type="text" name="model" placeholder="{{t "connect.model_placeholder"}}" required>{{end}}</label>
{{if .Contexts}}<label>{{t "connect.spends_against"}}
<select name="context">{{range .Contexts}}<option value="{{.Value}}">{{.Label}}</option>{{end}}</select></label>{{end}}
<button type="submit">{{t "connect.submit"}}</button>
</div>
<button type="submit">{{t "connect.submit"}}</button>
</form>
<script>
// Copy and download for what the connect panel shows, done in the page: the
@@ -145,7 +145,7 @@ document.addEventListener("click", (e) => {
</script>
<h2>{{t "profile.create_key"}}</h2>
<form method="post" action="/profile/keys" class="narrow">
<form method="post" action="/profile/keys" class="row">
<input type="hidden" name="csrf" value="{{.CSRF}}">
<label>{{t "profile.key_for"}}<input type="text" name="name" placeholder="{{t "profile.key_for_placeholder"}}" required></label>
{{if .Contexts}}
@@ -235,7 +235,7 @@ document.addEventListener("click", (e) => {
<h2>{{t "profile.password"}}</h2>
{{if .PasswordChanged}}<p>{{t "profile.password_changed"}}</p>{{end}}
{{if .PasswordError}}<p class="error">{{.PasswordError}}</p>{{end}}
<form method="post" action="/profile/password" autocomplete="on" class="narrow">
<form method="post" action="/profile/password" autocomplete="on" class="row">
<input type="hidden" name="csrf" value="{{.CSRF}}">
<label>{{t "profile.current_password"}}<input type="password" name="current" required
autocomplete="current-password"></label>
+33 -33
View File
@@ -71,7 +71,7 @@
{{else}}
{{/* Submitted on change: the choice is the action, and a second click
to confirm it would only be a way to forget to. */}}
<form method="post" action="/admin/mirror/mode" class="inline">
{{if $.CSRF}}<form method="post" action="/admin/mirror/mode" class="inline">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="model" value="{{.Name}}">
<input type="hidden" name="replicas" value="{{.Replicas}}">
@@ -81,7 +81,7 @@
<option value="stopped"{{if eq .Mode "stopped"}} selected{{end}}>{{t "mirror.mode.stopped"}}</option>
</select>
{{/* The button is in the catalogue: what a noscript holds is read as text, not markup. */}}<noscript>{{th "mirror.use.set"}}</noscript>
</form>
</form>{{end}}
{{end}}
{{if .Fleet}}<span class="runs-now{{if or .Parked .Unplaceable}} bad{{else if lt .Ready .Wanted}} short{{end}}" title="{{if .Starting}}{{t "mirror.use.runs_title_starting"}}{{else}}{{t "mirror.use.runs_title"}}{{end}}">
{{- if .Parked}}{{t "mirror.use.parked"}}
@@ -99,24 +99,24 @@
whose only outcome is a confusing error. Starting the same download
again is what somebody looking at a failed row wants, and the mirror
allows it: only a download still running is refused. */}}
<form method="post" action="/admin/mirror/fetch" class="inline">
{{if $.CSRF}}<form method="post" action="/admin/mirror/fetch" class="inline">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="source" value="{{.Source}}">
<input type="hidden" name="model" value="{{.Model}}">
<input type="hidden" name="version" value="{{.Version}}">
<input type="hidden" name="file" value="{{.File}}">
<button type="submit" class="link">{{t "mirror.retry"}}</button>
</form>
</form>{{end}}
{{else if eq (printf "%s" .State) "present"}}
<button type="button" class="link copy-url" data-url="{{.URL}}" title="{{.URL}}">{{t "mirror.copy_address"}}</button>
{{if and .UsedBy (not .Detail)}}<a href="/admin/mirror?tab=held&amp;open={{.Open}}">{{t "mirror.settings.open"}}</a>{{end}}
{{if and $.HasCatalog (not .UsedBy)}}{{if .Serve}}<a href="/admin/mirror">{{t "mirror.close"}}</a>
{{else}}<a href="/admin/mirror?serve={{.Model}}:{{.Version}}">{{t "mirror.serve"}}</a>{{end}}{{end}}
<form method="post" action="/admin/mirror/{{.Model}}/{{.Version}}/remove"
{{if $.CSRF}}<form method="post" action="/admin/mirror/{{.Model}}/{{.Version}}/remove"
class="inline" data-confirm="{{if .UsedBy}}{{t "mirror.remove_confirm_in_use" .Model .Version}}{{else}}{{t "mirror.remove_confirm" .Model .Version}}{{end}}">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<button type="submit" class="link danger">{{t "mirror.remove"}}</button>
</form>
</form>{{end}}
{{end}}
</td>
</tr>
@@ -130,7 +130,7 @@
<p class="error">{{t "serve.refusal" .Refusal}}</p>
{{else}}
{{if .Warning}}<p class="note error-row">{{.Warning}}.</p>{{end}}
<form method="post" action="/admin/mirror/serve" class="serve-form" data-engine="{{.Engine}}" data-free-gpus="{{.FreeGPUs}}">
{{if $.CSRF}}<form method="post" action="/admin/mirror/serve" class="serve-form" data-engine="{{.Engine}}" data-free-gpus="{{.FreeGPUs}}">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="source" value="{{.Source}}">
<input type="hidden" name="engine" value="{{.Engine}}">
@@ -191,7 +191,7 @@
</div>
<p class="sub serve-summary" hidden></p>
<button type="submit">{{t "serve.add"}}</button>
</form>
</form>{{end}}
<script>
// What it will run as, in one line, kept up to date as the form changes:
// engine, where, how many cards, memory, context and how it is shared. The
@@ -473,18 +473,18 @@ document.querySelector("table.explore").addEventListener("submit", async functio
<td>{{.DownloadGB}} GB</td>
<td>{{.RAMGB}} GB</td>
<td>{{if .Fits}}{{.Runs}}{{else}}<span class="opt">{{.Runs}}</span>{{end}}</td>
<td>{{if and $fam.FetchesItself .Fits}}<form method="post" action="/admin/mirror/strata" class="inline">
<td>{{if and $fam.FetchesItself .Fits}}{{if $.CSRF}}<form method="post" action="/admin/mirror/strata" class="inline">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="tab" value="strata">
<input type="hidden" name="action" value="serve">
<input type="hidden" name="family" value="{{$fam.Name}}">
<input type="hidden" name="size" value="{{.Size}}">
<button type="submit">{{t "strata.serve"}}</button></form>{{end}}{{if and $fam.Fetchable .Fits}}<form method="post" action="/admin/mirror/strata" class="inline">
<button type="submit">{{t "strata.serve"}}</button></form>{{end}}{{end}}{{if and $fam.Fetchable .Fits}}{{if $.CSRF}}<form method="post" action="/admin/mirror/strata" class="inline">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="tab" value="strata">
<input type="hidden" name="family" value="{{$fam.Name}}">
<input type="hidden" name="size" value="{{.Size}}">
<button type="submit">{{t "strata.fetch"}}</button></form>{{end}}</td>
<button type="submit">{{t "strata.fetch"}}</button></form>{{end}}{{end}}</td>
</tr>
{{end}}
</table>
@@ -493,7 +493,7 @@ document.querySelector("table.explore").addEventListener("submit", async functio
{{if eq .Tab "fetch"}}
<p class="note">{{t "mirror.fetch.intro"}}</p>
<form method="post" action="/admin/mirror/fetch">
{{if .CSRF}}<form method="post" action="/admin/mirror/fetch">
<input type="hidden" name="csrf" value="{{.CSRF}}">
<input type="hidden" name="tab" value="fetch">
<label>{{t "mirror.fetch.source"}}
@@ -508,7 +508,7 @@ document.querySelector("table.explore").addEventListener("submit", async functio
<label class="inline-check"><input type="checkbox" name="force" value="1">
{{t "mirror.fetch.force"}} <span class="opt">{{t "mirror.fetch.force_hint"}}</span></label>
<button type="submit">{{t "mirror.fetch.start"}}</button>
</form>
</form>{{end}}
<p class="note">{{t "mirror.fetch.note"}}</p>
{{end}}
@@ -552,7 +552,7 @@ document.querySelector("table.explore").addEventListener("submit", async functio
<span class="clip-inline" title="{{.Verdict.Reason}}">{{.Verdict.Reason}}</span>
{{else}}
{{if .Verdict.Architecture}}<span class="opt" title="{{.Verdict.Reason}}">{{.Verdict.Architecture}}{{if .Verdict.Checked}} ✓{{end}}</span>{{end}}
<form method="post" action="/admin/mirror/fetch" class="inline">
{{if $.CSRF}}<form method="post" action="/admin/mirror/fetch" class="inline">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="tab" value="explore">
{{range .Sources}}<input type="hidden" name="source" value="{{.URL}}">
@@ -561,7 +561,7 @@ document.querySelector("table.explore").addEventListener("submit", async functio
<input type="text" name="model" value="{{.Model}}" size="24">
<input type="text" name="version" value="{{.Version}}" size="10">
<button type="submit">{{t "card.fetch"}}</button>
</form>
</form>{{end}}
{{end}}
</td>
</tr>{{end}}
@@ -628,17 +628,17 @@ document.querySelector("table.explore").addEventListener("submit", async functio
<td><code>{{.Name}}</code></td>
<td class="clip" title="{{.Source}}">{{.Source}}</td>
<td class="num">{{.EffectiveScale}}</td>
<td class="actions"><form method="post" action="/admin/mirror/adapters" class="inline">
<td class="actions">{{if $csrf}}<form method="post" action="/admin/mirror/adapters" class="inline">
<input type="hidden" name="csrf" value="{{$csrf}}">
<input type="hidden" name="model" value="{{$mname}}">
<input type="hidden" name="open" value="{{$row.Open}}">
<input type="hidden" name="op" value="remove">
<input type="hidden" name="name" value="{{$a.Name}}">
<button type="submit" class="link danger">{{t "mirror.remove"}}</button></form></td>
<button type="submit" class="link danger">{{t "mirror.remove"}}</button></form>{{end}}</td>
</tr>{{end}}</table>
{{else}}<p class="note">{{t "mirror.adapters.none"}}</p>{{end}}
{{if .AdapterChoices}}{{$m := .}}
<form method="post" action="/admin/mirror/adapters" class="adapter-form">
{{if $csrf}}<form method="post" action="/admin/mirror/adapters" class="adapter-form">
<input type="hidden" name="csrf" value="{{$csrf}}">
<input type="hidden" name="model" value="{{.Name}}">
<input type="hidden" name="open" value="{{$row.Open}}">
@@ -651,14 +651,14 @@ document.querySelector("table.explore").addEventListener("submit", async functio
<label>{{t "mirror.adapters.scale"}} <input type="text" name="scale" value="1" size="4"></label>
<button type="submit">{{t "mirror.adapters.attach"}}</button>
</div>
</form>{{else}}<p class="note">{{t "mirror.adapters.no_choice"}}</p>{{end}}
</form>{{end}}{{else}}<p class="note">{{t "mirror.adapters.no_choice"}}</p>{{end}}
</div>
{{end}}
{{if .Declared}}
<p class="note">{{t "mirror.detail.declared"}}</p>
{{else}}
<form method="post" action="/admin/mirror/replicas" class="replicas-form">
{{if $csrf}}<form method="post" action="/admin/mirror/replicas" class="replicas-form">
<input type="hidden" name="csrf" value="{{$csrf}}">
<input type="hidden" name="model" value="{{.Name}}">
<input type="hidden" name="open" value="{{$row.Open}}">
@@ -688,7 +688,7 @@ document.querySelector("table.explore").addEventListener("submit", async functio
<button type="submit">{{t "common.save"}}</button>
</div>
<p class="note">{{if .Split}}{{t "mirror.split.scaling_note"}} {{end}}{{if .MostNow}}{{if and .Headroom .Headroom.ByPreempting}}{{t "mirror.form.most_now_preempting" .MostNow}}{{else}}{{t "mirror.form.most_now" .MostNow}}{{end}} {{end}}{{t "mirror.form.note"}}</p>
</form>
</form>{{end}}
{{template "mirror_model_settings" dict "M" . "Row" $row "CSRF" $csrf}}
{{end}}
</section>
@@ -703,7 +703,7 @@ document.querySelector("table.explore").addEventListener("submit", async functio
<details class="model-settings" open>
<summary><h4>{{t "mirror.settings.heading"}}</h4></summary>
<form method="post" action="/admin/mirror/context" class="settings-form">
{{if $csrf}}<form method="post" action="/admin/mirror/context" class="settings-form">
<input type="hidden" name="csrf" value="{{$csrf}}">
<input type="hidden" name="model" value="{{$m.Name}}">
<input type="hidden" name="open" value="{{$row.Open}}">
@@ -713,10 +713,10 @@ document.querySelector("table.explore").addEventListener("submit", async functio
<button type="submit">{{t "mirror.settings.context"}}</button>
</div>
<p class="note">{{t "mirror.settings.context_note"}}</p>
</form>
</form>{{end}}
{{if $m.GPUProfile}}
<form method="post" action="/admin/mirror/gpus" class="settings-form">
{{if $csrf}}<form method="post" action="/admin/mirror/gpus" class="settings-form">
<input type="hidden" name="csrf" value="{{$csrf}}">
<input type="hidden" name="model" value="{{$m.Name}}">
<input type="hidden" name="profile" value="{{$m.GPUProfile}}">
@@ -726,11 +726,11 @@ document.querySelector("table.explore").addEventListener("submit", async functio
<button type="submit">{{t "mirror.settings.gpus_apply"}}</button>
</div>
<p class="note">{{t "mirror.settings.gpus_note"}}</p>
</form>
</form>{{end}}
{{end}}
{{if or (gt (len $m.Profiles) 1) $m.CanCPU}}
<form method="post" action="/admin/mirror/split" class="settings-form">
{{if $csrf}}<form method="post" action="/admin/mirror/split" class="settings-form">
<input type="hidden" name="csrf" value="{{$csrf}}">
<input type="hidden" name="model" value="{{$m.Name}}">
<input type="hidden" name="open" value="{{$row.Open}}">
@@ -746,29 +746,29 @@ document.querySelector("table.explore").addEventListener("submit", async functio
</div>
{{if and $m.CanCPU $m.CPUSpeed}}<p class="note{{if lt $m.CPUSpeed 5.0}} error{{end}}">{{t "mirror.split.cpu_speed" (num $m.CPUSpeed 1) $m.CPUSpeedNode}}</p>{{end}}
<p class="note">{{t "mirror.split.note"}}</p>
</form>
</form>{{end}}
{{end}}
<div class="settings-actions">
<form method="post" action="/admin/mirror/state" class="inline">
{{if $csrf}}<form method="post" action="/admin/mirror/state" class="inline">
<input type="hidden" name="csrf" value="{{$csrf}}">
<input type="hidden" name="model" value="{{$m.Name}}">
<input type="hidden" name="open" value="{{$row.Open}}">
{{if $m.Stopped}}<button type="submit" name="action" value="start">{{t "mirror.settings.start"}}</button>
{{else}}<button type="submit" name="action" value="stop" data-confirm="{{t "mirror.settings.stop_confirm" $m.Name}}">{{t "mirror.settings.stop"}}</button>
<button type="submit" name="action" value="restart" data-confirm="{{t "mirror.settings.restart_confirm" $m.Name}}">{{t "mirror.settings.restart"}}</button>{{end}}
</form>
<form method="post" action="/admin/mirror/unserve" class="inline" data-confirm="{{t "mirror.settings.unserve_confirm" $m.Name}}">
</form>{{end}}
{{if $csrf}}<form method="post" action="/admin/mirror/unserve" class="inline" data-confirm="{{t "mirror.settings.unserve_confirm" $m.Name}}">
<input type="hidden" name="csrf" value="{{$csrf}}">
<input type="hidden" name="model" value="{{$m.Name}}">
<button type="submit" class="link danger">{{t "mirror.settings.unserve"}}</button>
</form>
<form method="post" action="/admin/mirror/unserve" class="inline" data-confirm="{{t "mirror.settings.purge_confirm" $m.Name}}">
</form>{{end}}
{{if $csrf}}<form method="post" action="/admin/mirror/unserve" class="inline" data-confirm="{{t "mirror.settings.purge_confirm" $m.Name}}">
<input type="hidden" name="csrf" value="{{$csrf}}">
<input type="hidden" name="model" value="{{$m.Name}}">
<input type="hidden" name="purge" value="yes">
<button type="submit" class="link danger">{{t "mirror.settings.purge"}}</button>
</form>
</form>{{end}}
</div>
</details>
{{end}}
+7 -6
View File
@@ -8,6 +8,7 @@
<title>{{t "people.title"}} · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
<meta name="htmx-config" content='{"scrollIntoViewOnBoost":false,"defaultFocusScroll":false,"allowEval":false}'>
<script src="/_static/htmx-2.0.11.min.js" defer></script>
<script defer src="{{asset "pages.js"}}"></script>
</head><body><div class="wrap">
<header>
<h1>{{t "people.title"}}</h1>
@@ -94,17 +95,17 @@
{{if .Member}}<dt>{{t "people.each_member"}}</dt><dd>{{.Member}}</dd>{{end}}
</dl>
{{if .Shadowed}}<p class="q-sub q-warn">{{.Shadowed}}</p>{{end}}
<details class="edit"><summary>{{if .HasAny}}{{t "people.change_limits"}}{{else}}{{t "people.set_limits"}}{{end}}</summary>
{{if $d.CSRF}}<details class="edit"><summary>{{if .HasAny}}{{t "people.change_limits"}}{{else}}{{t "people.set_limits"}}{{end}}</summary>
{{template "quota-form" (dict "Scope" (printf "%s" .Scope) "ID" .ID "CSRF" $d.CSRF "Sel" $sel
"F" (dict "RPM" .RPM "TPM" .TPM "Budget" .Budget "Period" .Period "Models" .Models "HasAny" .HasAny)
"Title" (t "people.quota_for" .Crumb)
"Note" (t "people.note.shared"))}}
</details>
<details class="edit"><summary>{{t "people.limit_each_member"}}</summary>
</details>{{end}}
{{if $d.CSRF}}<details class="edit"><summary>{{t "people.limit_each_member"}}</summary>
{{template "quota-form" (dict "Scope" (printf "%s-member" .Scope) "ID" .ID "F" .MemberRaw "CSRF" $d.CSRF "Sel" $sel
"Title" (t "people.each_member_of" .Crumb)
"Note" (t "people.note.each_member"))}}
</details>
</details>{{end}}
{{end}}
{{with .Person}}
<dl class="pd-limits">
@@ -114,12 +115,12 @@
<dt>{{t "people.spent" $d.QuotaWindow}}</dt><dd><span class="q-spend">{{.Spend}}</span>{{if .HasPct}} <span class="q-meter{{if ge .Percent 90}} hot{{end}}"><i style="width:{{pct .Percent 100}}%"></i></span> <span class="q-pct">{{t "quotas.pct" .Percent}}</span>{{end}}</dd>
<dt>{{t "people.models"}}</dt><dd>{{if .Models}}{{th "people.models_only" .Models}}{{else}}{{t "people.models_every"}}{{end}}</dd>
</dl>
<details class="edit"><summary>{{if .HasAny}}{{t "people.change_limits"}}{{else}}{{t "people.set_limits"}}{{end}}</summary>
{{if $d.CSRF}}<details class="edit"><summary>{{if .HasAny}}{{t "people.change_limits"}}{{else}}{{t "people.set_limits"}}{{end}}</summary>
{{template "quota-form" (dict "Scope" "user" "ID" .ID "CSRF" $d.CSRF "Sel" $sel
"F" (dict "RPM" .RPM "TPM" .TPM "Budget" .Budget "Period" .Period "Models" .Models "HasAny" .HasAny)
"Title" (t "people.everywhere" .Email)
"Note" (t "people.note.person"))}}
</details>
</details>{{end}}
{{end}}
{{end}}
+25 -24
View File
@@ -41,12 +41,12 @@
{{if .HasPct}}<span class="q-meter{{if ge .Percent 90}} hot{{end}}" title="{{t "quotas.pct_of_budget" .Percent}}"><i style="width:{{pct .Percent 100}}%"></i></span>
<span class="q-pct">{{t "quotas.pct" .Percent}}</span>{{end}}
</td>
<td class="q-act">
<td class="q-act">{{if $.CSRF}}
{{if .Editing}}<a href="/admin/quotas">{{t "quotas.close"}}</a>
{{else}}<a href="/admin/quotas?edit={{.Scope}}:{{.ID}}">{{if .HasAny}}{{t "quotas.change"}}{{else}}{{t "quotas.set"}}{{end}}</a>{{end}}
{{if .MemberEditing}}<a href="/admin/quotas">{{t "quotas.close"}}</a>
{{else}}<a href="/admin/quotas?edit={{.Scope}}-member:{{.ID}}" title="{{t "quotas.per_member_title"}}">{{t "quotas.per_member"}}</a>{{end}}
</td>
{{end}}</td>
</tr>
{{if .MemberEditing}}
<tr class="editing"><td colspan="6">
@@ -58,7 +58,7 @@
{{if .Editing}}
<tr class="editing"><td colspan="6">
<form method="post" action="/admin/quotas" class="q-form">
{{if $.CSRF}}<form method="post" action="/admin/quotas" class="q-form">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="scope" value="{{.Scope}}">
<input type="hidden" name="id" value="{{.ID}}">
@@ -84,7 +84,7 @@
<button type="submit" name="clear" value="1" class="link danger"
{{if not .HasAny}}disabled{{end}}>{{t "quotas.form.remove_all"}}</button>
</div>
</form>
</form>{{end}}
</td></tr>
{{end}}
{{end}}
@@ -110,8 +110,8 @@
<td class="q-spent"><span class="q-spend">{{.Spend}}</span>
{{if .HasPct}}<span class="q-meter{{if ge .Percent 90}} hot{{end}}"><i style="width:{{pct .Percent 100}}%"></i></span>
<span class="q-pct">{{t "quotas.pct" .Percent}}</span>{{end}}</td>
<td class="q-act">{{if .Editing}}<a href="/admin/quotas">{{t "quotas.close"}}</a>
{{else}}<a href="/admin/quotas?edit=user:{{.ID}}">{{if .HasAny}}{{t "quotas.change"}}{{else}}{{t "quotas.set"}}{{end}}</a>{{end}}</td>
<td class="q-act">{{if $.CSRF}}{{if .Editing}}<a href="/admin/quotas">{{t "quotas.close"}}</a>
{{else}}<a href="/admin/quotas?edit=user:{{.ID}}">{{if .HasAny}}{{t "quotas.change"}}{{else}}{{t "quotas.set"}}{{end}}</a>{{end}}{{end}}</td>
</tr>
{{if .Editing}}
<tr class="editing"><td colspan="6">
@@ -138,38 +138,39 @@
<td class="num">{{or .Memory "—"}}</td>
<td class="note">{{.UpdatedBy}}</td>
<td class="q-act"><a href="/admin/quotas?cap={{.Scope}}#capacity">{{t "quotas.change"}}</a>
<form method="post" action="/admin/quotas/capacity" class="inline">
{{if $.CSRF}}<form method="post" action="/admin/quotas/capacity" class="inline">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="scope" value="{{.Scope}}">
<button type="submit" name="clear" value="1" class="link danger">{{t "capacity.remove"}}</button>
</form></td>
</form>{{end}}</td>
</tr>{{end}}
</table>
<p class="q-legend">{{t "capacity.legend"}}</p>
{{else}}<p class="empty">{{t "capacity.empty"}}</p>{{end}}
{{if .Scopes}}
<form method="post" action="/admin/quotas/capacity" class="q-form capacity-form">
{{if .CSRF}}<form method="post" action="/admin/quotas/capacity" class="q-form capacity-form">
<input type="hidden" name="csrf" value="{{.CSRF}}">
<p class="q-form-title">{{if .CapForm.Scope}}{{th "capacity.form.title_change"}}{{else}}{{th "capacity.form.title_set"}}{{end}}</p>
<div class="field-row">
<label>{{t "quotas.col.where"}}
<select name="scope" required>
<option value="">{{t "capacity.form.choose"}}</option>
{{range .Scopes}}<option value="{{.Key}}"{{if eq .Key $.CapForm.Scope}} selected{{end}}>{{.Name}}</option>{{end}}
</select></label>
<div class="field-row">
<label>{{t "capacity.form.g_gpus"}} <input type="text" name="gpus" value="{{.CapForm.GPUs}}" placeholder="—"></label>
<label>{{t "capacity.col.cores"}} <input type="text" name="cores" value="{{.CapForm.Cores}}" placeholder="—"></label>
<label>{{t "capacity.col.memory"}} <input type="text" name="memory" value="{{.CapForm.Memory}}" placeholder="—"></label>
</div>
<div class="field-row">
<label>{{t "capacity.form.l_gpus"}} <input type="text" name="limit_gpus" value="{{.CapForm.LGPUs}}" placeholder="—"></label>
<label>{{t "capacity.col.cores"}} <input type="text" name="limit_cores" value="{{.CapForm.LCores}}" placeholder="—"></label>
<label>{{t "capacity.col.memory"}} <input type="text" name="limit_memory" value="{{.CapForm.LMem}}" placeholder="—"></label>
<label class="short">{{t "capacity.form.g_gpus"}} <input type="text" name="gpus" value="{{.CapForm.GPUs}}" placeholder="—"></label>
<label class="short">{{t "capacity.col.cores"}} <input type="text" name="cores" value="{{.CapForm.Cores}}" placeholder="—"></label>
<label class="short">{{t "capacity.col.memory"}} <input type="text" name="memory" value="{{.CapForm.Memory}}" placeholder="—"></label>
<span class="field-gap"></span>
<label class="short">{{t "capacity.form.l_gpus"}} <input type="text" name="limit_gpus" value="{{.CapForm.LGPUs}}" placeholder="—"></label>
<label class="short">{{t "capacity.col.cores"}} <input type="text" name="limit_cores" value="{{.CapForm.LCores}}" placeholder="—"></label>
<label class="short">{{t "capacity.col.memory"}} <input type="text" name="limit_memory" value="{{.CapForm.LMem}}" placeholder="—"></label>
</div>
<div class="q-form-actions"><button type="submit">{{t "common.save"}}</button>
{{if .CapForm.Scope}}<a href="/admin/quotas#capacity">{{t "common.cancel"}}</a>{{end}}</div>
</form>
</form>{{end}}
{{else}}<p class="empty">{{th "capacity.empty_tree"}}</p>{{end}}
{{end}}
@@ -185,7 +186,7 @@
held to from above, marked as inherited and saying from where. */}}
{{/* The form for a limit that is not a branch's own: per member, or a person. */}}
{{define "quota-form"}}
<form method="post" action="/admin/quotas" class="q-form">
{{if .CSRF}}<form method="post" action="/admin/quotas" class="q-form">
<input type="hidden" name="csrf" value="{{.CSRF}}">
<input type="hidden" name="scope" value="{{.Scope}}">
<input type="hidden" name="id" value="{{.ID}}">
@@ -193,24 +194,24 @@
{{with .Sel}}<input type="hidden" name="tab" value="people"><input type="hidden" name="sel" value="{{.}}">{{end}}
<p class="q-form-title">{{th "quotas.qform.title" .Title .Note}}</p>
<div class="field-row">
<label>{{t "quotas.form.rpm"}} <input type="text" name="rpm" value="{{.F.RPM}}" placeholder="{{t "common.none"}}"></label>
<label>{{t "quotas.form.tpm"}} <input type="text" name="tpm" value="{{.F.TPM}}" placeholder="{{t "common.none"}}"></label>
<label>{{t "quotas.col.budget"}} <input type="text" name="budget" value="{{.F.Budget}}" placeholder="{{t "common.none"}}"></label>
<label class="short">{{t "quotas.form.rpm"}} <input type="text" name="rpm" value="{{.F.RPM}}" placeholder="{{t "common.none"}}"></label>
<label class="short">{{t "quotas.form.tpm"}} <input type="text" name="tpm" value="{{.F.TPM}}" placeholder="{{t "common.none"}}"></label>
<label class="short">{{t "quotas.col.budget"}} <input type="text" name="budget" value="{{.F.Budget}}" placeholder="{{t "common.none"}}"></label>
<label class="q-period">{{t "quotas.form.per"}}
<select name="period">
<option value="month"{{if eq .F.Period "month"}} selected{{end}}>{{t "quotas.per.month"}}</option>
<option value="day"{{if eq .F.Period "day"}} selected{{end}}>{{t "quotas.per.day"}}</option>
</select></label>
<label class="grow">{{t "quotas.form.models"}} <span class="opt">{{t "quotas.form.models_hint"}}</span>
<input type="text" name="models" value="{{.F.Models}}" placeholder="{{t "quotas.form.every_model"}}"></label>
</div>
<label>{{t "quotas.form.models"}} <span class="opt">{{t "quotas.form.models_hint"}}</span>
<input type="text" name="models" value="{{.F.Models}}" placeholder="{{t "quotas.form.every_model"}}"></label>
<div class="q-form-actions">
<button type="submit">{{t "common.save"}}</button>
{{if not .Sel}}<a href="/admin/quotas">{{t "common.cancel"}}</a>{{end}}
<button type="submit" name="clear" value="1" class="link danger"
{{if not .F.HasAny}}disabled{{end}}>{{t "quotas.qform.remove"}}</button>
</div>
</form>
</form>{{end}}
{{end}}
{{define "quota-value"}}{{if .Own}}{{.Own}}{{else if .Up}}<span class="q-up" title="{{t "quotas.set_on" .From}}">{{.Up}} ↑</span>{{else}}<span class="q-none">—</span>{{end}}{{end}}
+48 -48
View File
@@ -10,21 +10,21 @@
{{range .Pending}}<tr>
<td>{{.Email}}</td><td>{{.Name}}</td><td>{{ago .CreatedAt}}</td>
<td>
<form method="post" action="/admin/users/{{.ID}}/approve" class="inline">
{{if $.Edit}}<form method="post" action="/admin/users/{{.ID}}/approve" class="inline">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<select name="project" aria-label="{{t "users.project_for" .Email}}">
<option value="">{{t "users.no_project"}}</option>
{{range $.Projects}}<option value="{{.ID}}">{{.Path}}</option>{{end}}
</select>
<button type="submit">{{t "users.approve"}}</button>
</form>
</form>{{end}}
</td>
<td>
<form method="post" action="/admin/users/{{.ID}}/delete" class="inline"
{{if $.Edit}}<form method="post" action="/admin/users/{{.ID}}/delete" class="inline"
hx-confirm="{{t "users.reject_confirm" .Email}}">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<button type="submit" class="link danger">{{t "users.reject"}}</button>
</form>
</form>{{end}}
</td>
</tr>{{end}}</table>
{{end}}
@@ -42,19 +42,19 @@
<strong class="node-name"><a href="/admin/users?sel=org:{{$org.ID}}" class="sel-link{{if eq $.Sel (printf "org:%s" $org.ID)}} on{{end}}">{{$org.Name}}</a></strong>
<span class="node-meta">{{tn "users.n_teams" (len $org.Teams)}}</span>
{{template "scope-members" (dict "Kind" "org" "ID" $org.ID "Name" $org.Name "D" $)}}
<details class="edit"><summary>{{t "users.rename"}}</summary>
{{if $.Edit}}<details class="edit"><summary>{{t "users.rename"}}</summary>
<form method="post" action="/admin/tenancy/org/{{$org.ID}}/rename">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="text" name="name" value="{{$org.Name}}" required aria-label="{{t "users.new_name"}}">
<button type="submit">{{t "common.save"}}</button>
</form>
</details>
</details>{{end}}
{{if not $org.Teams}}
<form method="post" action="/admin/tenancy/org/{{$org.ID}}/delete" class="inline"
{{if $.Edit}}<form method="post" action="/admin/tenancy/org/{{$org.ID}}/delete" class="inline"
hx-confirm="{{t "users.remove_org_confirm" $org.Name}}">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<button type="submit" class="link danger">{{t "users.remove"}}</button>
</form>
</form>{{end}}
{{end}}
</div>
@@ -64,15 +64,15 @@
<span class="node-name"><a href="/admin/users?sel=team:{{$team.ID}}" class="sel-link{{if eq $.Sel (printf "team:%s" $team.ID)}} on{{end}}">{{$team.Name}}</a></span>
<span class="node-meta">{{tn "users.n_projects" (len $team.Projects)}}</span>
{{template "scope-members" (dict "Kind" "team" "ID" $team.ID "Name" $team.Name "D" $)}}
<details class="edit"><summary>{{t "users.rename"}}</summary>
{{if $.Edit}}<details class="edit"><summary>{{t "users.rename"}}</summary>
<form method="post" action="/admin/tenancy/team/{{$team.ID}}/rename">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="text" name="name" value="{{$team.Name}}" required aria-label="{{t "users.new_name"}}">
<button type="submit">{{t "common.save"}}</button>
</form>
</details>
</details>{{end}}
{{if gt (len $.Orgs) 1}}
<details class="edit"><summary>{{t "users.move"}}</summary>
{{if $.Edit}}<details class="edit"><summary>{{t "users.move"}}</summary>
<form method="post" action="/admin/tenancy/team/{{$team.ID}}/move">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<select name="to" aria-label="{{t "users.move_team_to" $team.Name}}">
@@ -80,14 +80,14 @@
</select>
<button type="submit">{{t "users.move_button"}}</button>
</form>
</details>
</details>{{end}}
{{end}}
{{if not $team.Projects}}
<form method="post" action="/admin/tenancy/team/{{$team.ID}}/delete" class="inline"
{{if $.Edit}}<form method="post" action="/admin/tenancy/team/{{$team.ID}}/delete" class="inline"
hx-confirm="{{t "users.remove_team_confirm" $team.Name}}">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<button type="submit" class="link danger">{{t "users.remove"}}</button>
</form>
</form>{{end}}
{{end}}
</div>
@@ -101,16 +101,16 @@
</summary>
<div class="project-body">
<div class="node-head">
<details class="edit"><summary>{{t "users.rename"}}</summary>
{{if $.Edit}}<details class="edit"><summary>{{t "users.rename"}}</summary>
<form method="post" action="/admin/tenancy/project/{{$p.ID}}/rename">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="open" value="{{$p.ID}}">
<input type="text" name="name" value="{{$p.Name}}" required aria-label="{{t "users.new_name"}}">
<button type="submit">{{t "common.save"}}</button>
</form>
</details>
</details>{{end}}
{{if gt (len $.Teams) 1}}
<details class="edit"><summary>{{t "users.move_project"}}</summary>
{{if $.Edit}}<details class="edit"><summary>{{t "users.move_project"}}</summary>
<form method="post" action="/admin/tenancy/project/{{$p.ID}}/move">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="open" value="{{$p.ID}}">
@@ -119,14 +119,14 @@
</select>
<button type="submit">{{t "users.move_button"}}</button>
</form>
</details>
</details>{{end}}
{{end}}
{{if and (eq $p.Users 0) (eq $p.Keys 0)}}
<form method="post" action="/admin/tenancy/project/{{$p.ID}}/delete" class="inline"
{{if $.Edit}}<form method="post" action="/admin/tenancy/project/{{$p.ID}}/delete" class="inline"
hx-confirm="{{t "users.remove_project_confirm" $p.Name}}">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<button type="submit" class="link danger">{{t "users.remove"}}</button>
</form>
</form>{{end}}
{{else}}<span class="note">{{t "users.not_removable"}}</span>{{end}}
</div>
@@ -145,12 +145,12 @@
<td class="state state-{{.State}}">{{t (print "users.key_state." .State)}}</td>
<td>{{ago .LastUsedAt}}</td>
<td>{{if eq .State "active"}}
<form method="post" action="/admin/keys/{{.ID}}/revoke" class="inline"
{{if $.Edit}}<form method="post" action="/admin/keys/{{.ID}}/revoke" class="inline"
hx-confirm="{{t "users.revoke_confirm" .Name}}">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="open" value="{{$p.ID}}">
<button type="submit" class="link danger">{{t "users.revoke"}}</button>
</form>{{end}}</td>
</form>{{end}}{{end}}</td>
</tr>{{end}}
</table>
{{end}}
@@ -160,7 +160,7 @@
{{/* Folded until wanted: a box open under every team was most of the
tree's height and read as something to fill in. */}}
<details class="add-inline">
{{if $.Edit}}<details class="add-inline">
<summary>{{t "users.add_project"}}</summary>
<form method="post" action="/admin/tenancy/project" class="add">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
@@ -168,11 +168,11 @@
<input type="text" name="name" required placeholder="{{t "users.new_project_in" $team.Name}}" aria-label="{{t "users.new_project_in_label" $team.Name}}">
<button type="submit">{{t "common.add"}}</button>
</form>
</details>
</details>{{end}}
</div>
{{end}}
<details class="add-inline team-add">
{{if $.Edit}}<details class="add-inline team-add">
<summary>{{t "users.add_team"}}</summary>
<form method="post" action="/admin/tenancy/team" class="add">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
@@ -180,15 +180,15 @@
<input type="text" name="name" required placeholder="{{t "users.new_team_in" $org.Name}}" aria-label="{{t "users.new_team_in_label" $org.Name}}">
<button type="submit">{{t "common.add"}}</button>
</form>
</details>
</details>{{end}}
</section>
{{end}}
<form method="post" action="/admin/tenancy/org" class="add org-add">
{{if $.Edit}}<form method="post" action="/admin/tenancy/org" class="add org-add">
<input type="hidden" name="csrf" value="{{.CSRF}}">
<input type="text" name="name" required placeholder="{{t "users.new_org"}}" aria-label="{{t "users.new_org_label"}}">
<button type="submit">{{t "users.add_org"}}</button>
</form>
</form>{{end}}
<h2 id="accounts">{{t "users.accounts"}}</h2>
{{if .Accounts}}
@@ -211,24 +211,24 @@
<summary aria-label="{{t "users.more_for" $u.Email}}" title="{{t "users.more"}}">⋯</summary>
<div class="menu">
{{if eq (printf "%s" $u.State) "active"}}
<form method="post" action="/admin/users/{{$u.ID}}/suspend" class="inline">
{{if $d.Edit}}<form method="post" action="/admin/users/{{$u.ID}}/suspend" class="inline">
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
<input type="hidden" name="open" value="{{.Open}}">
<button type="submit" class="link danger">{{t "users.suspend"}}</button>
</form>
</form>{{end}}
{{else}}
<form method="post" action="/admin/users/{{$u.ID}}/reinstate" class="inline">
{{if $d.Edit}}<form method="post" action="/admin/users/{{$u.ID}}/reinstate" class="inline">
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
<input type="hidden" name="open" value="{{.Open}}">
<button type="submit" class="link">{{t "users.reinstate"}}</button>
</form>
</form>{{end}}
{{end}}
<form method="post" action="/admin/users/{{$u.ID}}/delete" class="inline"
{{if $d.Edit}}<form method="post" action="/admin/users/{{$u.ID}}/delete" class="inline"
hx-confirm="{{t "users.remove_confirm" $u.Email}}">
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
<input type="hidden" name="open" value="{{.Open}}">
<button type="submit" class="link danger">{{t "users.remove"}}</button>
</form>
</form>{{end}}
</div>
</details>
{{else}}<span class="note">{{t "users.you"}}</span>{{end}}
@@ -241,22 +241,22 @@
<td><a href="/admin/users?sel=user:{{$u.ID}}" class="sel-link{{if eq $d.Sel (printf "user:%s" $u.ID)}} on{{end}}">{{$u.Email}}</a>{{if $u.IsAdmin}} <span class="tag">{{t "users.tag.admin"}}</span>{{end}}</td>
<td class="state state-{{$u.State}}">{{t (print "users.state." $u.State)}}</td>
<td class="where">
{{range $i, $m := $u.Memberships}}{{if lt $i 2}}{{with $m}}<span class="chip-m">{{.Path}}{{if eq .Role "manager"}} <b class="chip-role">{{t "users.role.manager"}}</b>{{end}}
{{if eq (printf "%s" .Scope) "org"}}
{{range $i, $m := $u.Memberships}}{{if lt $i 2}}{{with $m}}<span class="chip-m">{{.Path}}
{{if and (eq (printf "%s" .Scope) "org") $d.Edit}}
<form method="post" action="/admin/users/{{$u.ID}}/role" class="inline">
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
<input type="hidden" name="scope" value="org:{{.ScopeID}}">
<input type="hidden" name="role" value="{{if eq .Role "manager"}}member{{else}}manager{{end}}">
<button type="submit" class="chip-x" title="{{if eq .Role "manager"}}{{t "users.stop_managing_of" .Path}}{{else}}{{t "users.make_manager_of" .Path}}{{end}}"
aria-label="{{if eq .Role "manager"}}{{t "users.stop_managing"}}{{else}}{{t "users.make_manager"}}{{end}}">{{if eq .Role "manager"}}☆{{else}}★{{end}}</button>
<select name="role" class="chip-sel" data-autosubmit aria-label="{{t "users.role_in" .Path}}" title="{{t "users.role_note"}}">
{{$r := .Role}}{{range $role := roles}}<option value="{{$role}}"{{if eq $role $r}} selected{{end}}>{{t (print "users.role." $role)}}</option>{{end}}
</select>
</form>
{{end}}
<form method="post" action="/admin/users/{{$u.ID}}/leave" class="inline"
{{else if ne .Role "member"}} <b class="chip-role">{{t (print "users.role." .Role)}}</b>{{end}}
{{if $d.Edit}}<form method="post" action="/admin/users/{{$u.ID}}/leave" class="inline"
hx-confirm="{{t "users.take_out_confirm" $u.Email .Path}}">
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
<input type="hidden" name="scope" value="{{.Scope}}:{{.ScopeID}}">
<button type="submit" class="chip-x" aria-label="{{t "users.take_out_of" .Path}}" title="{{t "users.take_out_of" .Path}}">×</button>
</form></span>{{end}}{{end}}
</form>{{end}}</span>{{end}}{{end}}
{{else}}<span class="opt">{{t "users.on_own"}}</span>{{end}}
{{/* Two, then a count: a person in six teams was six chips wrapping the
row into a paragraph. The rest are named on hover and in their own
@@ -265,14 +265,14 @@
</td>
<td>{{ago $u.LastLoginAt}}</td>
<td>
<form method="post" action="/admin/users/{{$u.ID}}/join" class="inline add-to">
{{if $d.Edit}}<form method="post" action="/admin/users/{{$u.ID}}/join" class="inline add-to">
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
<select name="scope" aria-label="{{t "users.add_to_label" $u.Email}}" required>
<option value="">{{t "users.choose"}}</option>
{{range $d.Scopes}}<option value="{{.Value}}">{{.Label}}</option>{{end}}
</select>
<button type="submit" class="small">{{t "common.add"}}</button>
</form>
</form>{{end}}
</td>
<td class="actions">{{template "state-actions" (dict "U" $u "D" $d "Open" "")}}</td>
</tr>{{end}}
@@ -285,19 +285,19 @@
<td class="state state-{{$u.State}}">{{t (print "users.state." $u.State)}}</td>
<td>{{ago $u.LastLoginAt}}</td>
<td class="actions">
<form method="post" action="/admin/users/{{$u.ID}}/leave" class="inline"
{{if $d.Edit}}<form method="post" action="/admin/users/{{$u.ID}}/leave" class="inline"
hx-confirm="{{t "users.take_out_here_confirm" $u.Email}}">
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
<input type="hidden" name="scope" value="{{.Scope}}">
<input type="hidden" name="open" value="{{.Open}}">
<button type="submit" class="link danger">{{t "users.take_out"}}</button>
</form>
</form>{{end}}
</td>
</tr>{{end}}
{{/* Adding a person to a scope, from the scope. */}}
{{define "add-member"}}{{$d := .D}}
<form method="post" action="/admin/tenancy/{{.Kind}}/{{.ID}}/add" class="add">
{{if $d.Edit}}<form method="post" action="/admin/tenancy/{{.Kind}}/{{.ID}}/add" class="add">
<input type="hidden" name="csrf" value="{{$d.CSRF}}">
<input type="hidden" name="open" value="{{.Open}}">
<select name="user" aria-label="{{t "users.person_to_add"}}" required>
@@ -305,7 +305,7 @@
{{range $d.Accounts}}<option value="{{.ID}}">{{.Email}}</option>{{end}}
</select>
<button type="submit">{{t "users.add_member"}}</button>
</form>{{end}}
</form>{{end}}{{end}}
{{/* An org's or team's own members, folded away until asked for. */}}
{{define "scope-members"}}{{$d := .D}}{{$k := printf "%s:%s" .Kind .ID}}
+3 -3
View File
@@ -109,7 +109,7 @@
administrator, on the page reached through a session. */}}
{{if $.AnyGPUs}}<td class="clip">{{if .GPUTotal}}{{$node := .}}
{{range .Provision}}{{if eq .Engine "vllm"}}<span class="note" title="{{.Detail}}">{{.Version}} {{with .Variant}}{{buildLabel .}} {{end}}{{t (print "dash.state." .State)}}</span>{{end}}{{end}}
{{if and $.Menu.CSRF .EngineBuilds}}
{{if and ($.Menu.May "/admin/nodes/x/engines") .EngineBuilds}}
<form method="post" action="/admin/nodes/{{.ID}}/engines" class="inline build-choice">
<input type="hidden" name="csrf" value="{{$.Menu.CSRF}}">
<input type="hidden" name="engine" value="vllm">
@@ -177,7 +177,7 @@ document.querySelectorAll("form.build-choice select").forEach(function (s) {
<td>{{or .Fits (t "dash.presets.fits_none")}}</td>
{{/* Why the leader does not add it itself is the control plane's, in English. */}}
<td{{if .Declined}} class="note"{{end}}>{{or .State "—"}}{{with .Blocked}} <span class="note">({{.}})</span>{{end}}</td>
<td>{{if and $.Menu.CSRF (not .InCatalog)}}
<td>{{if and ($.Menu.May "/admin/presets") (not .InCatalog)}}
<form method="post" action="/admin/presets/{{.Name}}" class="inline">
<input type="hidden" name="csrf" value="{{$.Menu.CSRF}}">
<button type="submit"{{if .AddsComfyUI}} title="{{t "dash.presets.adds_comfyui"}}"{{end}}>{{if .Declined}}{{t "dash.presets.add_again"}}{{else}}{{t "dash.presets.add"}}{{end}}</button>
@@ -185,7 +185,7 @@ document.querySelectorAll("form.build-choice select").forEach(function (s) {
</tr>
{{end}}</table>
<p>{{if .Auto}}{{t "dash.presets.auto_on"}}{{else}}{{t "dash.presets.auto_off"}}{{end}}
{{if $.Menu.CSRF}}
{{if $.Menu.May "/admin/presets"}}
<form method="post" action="/admin/presets/auto" class="inline">
<input type="hidden" name="csrf" value="{{$.Menu.CSRF}}">
<input type="hidden" name="on" value="{{if .Auto}}false{{else}}true{{end}}">
Loaded 100 of 102 files, more files were not shown because too many files have changed in this diff. Show more