Compare commits
25
Commits
v0.16.3
...
356f7e0ced
No files matched your search
+85
-4
@@ -49,7 +49,8 @@ Face, done 2026-10-30), then M5 (engines without the internet), M3, M4, M6.
|
||||
**Scale.** O: more than one control plane answering clients; O1 (measuring one router's ceiling) first.
|
||||
|
||||
**Governance.**
|
||||
7. **C1**: a tenant-admin role, and forms hidden from viewers.
|
||||
7. ~~**C1**~~: done 2026-11-08. An organization's admin (`--role admin` on an org membership) changes its teams,
|
||||
projects, people and keys from /org; viewers are drawn no forms; forms made compact.
|
||||
8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
|
||||
|
||||
**When there is room.**
|
||||
@@ -435,9 +436,9 @@ portal is not done — the page shows; the console and ctl change.
|
||||
API middleware and the /admin pages (with each change's result), `ctl
|
||||
audit`, the console's Audit view; staff roles viewer / operator /
|
||||
billing / admin. Left: before/after values (the request's fields are
|
||||
recorded, not the state they replaced); a tenant-admin role that changes
|
||||
its own organization (managers still only view it); hiding the forms a
|
||||
viewer cannot submit. **Audit log and roles (M).** Today "audit" is slog lines
|
||||
recorded, not the state they replaced). Done 2026-11-08: an organization's
|
||||
admin who changes its own organization from /org (managers still only view
|
||||
it), and the forms a viewer cannot submit no longer drawn. **Audit log and roles (M).** Today "audit" is slog lines
|
||||
(`internal/web/admin_users.go:403-480`) and roles are the `IsAdmin` flag
|
||||
plus an org "manager" who can view. Add `audit_events` (who, what,
|
||||
before/after, source) written by the `adminAction`/`tenancyAction`
|
||||
@@ -1982,6 +1983,86 @@ one control plane's router can. In order:
|
||||
- O3. Only if O2 is not enough: stateless gateway machines (`llmgw server --role gateway`) in front of the leader's
|
||||
control plane, as many as the traffic needs.
|
||||
|
||||
## P. Training beyond one card (written 2026-11-11) — decision open
|
||||
|
||||
F4 trains a LoRA on one card of a node the caller names (`ctl train start --node`), one job per node, no resume.
|
||||
Larger work (full fine-tuning, a LoRA on a 70B, several GPUs or machines) needs more. Two ways:
|
||||
|
||||
**A. In llmgw.** Add, in order: a job that takes N whole GPUs on one node (torchrun, FSDP/DeepSpeed in the agent's
|
||||
venv); placement by the control plane instead of `--node`, freeing GPUs from replicas by the scaling schedule or by
|
||||
draining; checkpoints to shared storage and resume when an agent or node goes; datasets uploaded or read from
|
||||
storage, not only KB pairs; then gang placement of N nodes with a rendezvous on the first, the NCCL environment from
|
||||
the fabric discovery (IB/NVLink, `llmgw doctor`); a console screen with logs and loss. Size: L for one node, XL for
|
||||
several. For: one product, one install, one set of users and grants (KB rules carry over), GPUs shared with
|
||||
inference without a second scheduler. Against: it is Slurm's gang scheduling, requeue and accounting written again;
|
||||
multi-node NCCL cannot be tried on the PoC (one GPU node, no IB); every customer's training code would have to fit
|
||||
our job format.
|
||||
|
||||
**B. Slurm beside llmgw.** The TUI installs Slurm on the nodes the customer picks (munge, slurmctld on the control
|
||||
planes, slurmd, users from the same LDAP/AD), with pyxis/enroot for containers. A node is handed between the two:
|
||||
llmgw drains its replicas and `scontrol update state=resume`, and back. llmgw's part is what comes out: a checkpoint
|
||||
or adapter from a Slurm job published to the mirror (`ctl train import`), served through canary and the eval. Size:
|
||||
M. For: what research and university customers already run (`sbatch`, `srun torchrun`), any training code, mature
|
||||
multi-node and requeue; we write only the install and the handover. Against: a second system with its own failures,
|
||||
upgrades and users to keep in step; GPUs split by hand or by the handover, not by demand; access rules on training
|
||||
data are Slurm's and the file system's, not the KB's.
|
||||
|
||||
Both need shared storage (NFS, BeeGFS, Lustre) the customer provides; neither changes F4.
|
||||
|
||||
Recommendation: keep F4 in llmgw for small single-card adapters tied to KB grants; take B for anything larger;
|
||||
build only A's first step (N GPUs on one node, placed by the control plane, with resume) if customers ask for
|
||||
mid-size jobs without wanting Slurm.
|
||||
|
||||
**B taken (2026-11-11).** Design: the cluster's settings say it (`slurm: enabled, cluster_name, controller, nodes,
|
||||
lent, partition`); the leader tells each named node's agent (root) what to be, the agents install and run it.
|
||||
- The controller is an llmgw node (its agent runs slurmctld); `nodes` run slurmd configless
|
||||
(`--conf-server`), so slurm.conf, gres.conf and cgroup.conf live only on the controller.
|
||||
- The munge key is derived from the cluster secret (HKDF), so nothing new is stored or copied by hand.
|
||||
- Each node reports `slurmd -C`; the leader writes slurm.conf from those lines, its mgmt address and its GPUs
|
||||
(`Gres=gpu:<type>:N`, gres.conf with the device files), and sends it to the controller again when one changes.
|
||||
- A node in `nodes` belongs to llmgw until it is in `lent`: drained in Slurm (reason "llmgw: serving models").
|
||||
Lent: llmgw drains it as `ctl drain` does (kept in force each tick, as facility holds are, so a new leader
|
||||
keeps it) and, once no replica is left, its agent resumes it in Slurm. Taken out of `lent`: the agent drains
|
||||
it in Slurm, reports the jobs left, and when none are the leader makes it ready for llmgw again.
|
||||
- Phases: P1 the `slurm` package (conf, key, parsers) and settings; P2 the agent side (proto, install, munge,
|
||||
slurmd/slurmctld, status); P3 the leader side (conf from reports, lend/reclaim); P4 `ctl slurm`
|
||||
(status, lend, reclaim) and a TUI screen; P5 `ctl train import` of a Slurm job's output into the mirror.
|
||||
Later: a backup slurmctld on a second node, accounting (slurmdbd on the Patroni PG), users from LDAP/AD.
|
||||
- Tried on the PoC 2026-11-11 (dev build v0.16.3-17-g849760c-slurm on 140, 150, node05, node06; Ubuntu
|
||||
24.04, Slurm 23.11.4 from the archive): `ctl slurm setup --controller llmgw-node06 --nodes llmgw-node05`
|
||||
installed munge, slurmctld and slurmd, node05 entered drained with its slurmd -C hardware (8 CPUs,
|
||||
31132 MiB after the 3%). `lend`: its 2 replicas moved off, then IDLE in Slurm; srun and sbatch ran there.
|
||||
`reclaim` with a 90 s job running: MIXED+DRAIN, llmgw kept off, ready again within a minute of the job
|
||||
ending. Found and fixed on the way: a client command needs its files even for --version (version now
|
||||
from the daemon); the packaged unit is Type=notify (slurmd needs --systemd); slurmd's own "slurmstepd
|
||||
infinity" was counted as a job; without accounting storage 23.11's main scheduler marks batch jobs
|
||||
InvalidAccount and backfill starts them, one a pass (bf_interval=10: 6-10 s each). So many small jobs
|
||||
want slurmdbd. Not tried yet: a GPU node (gres).
|
||||
- Failover while lent, tried 2026-11-11 (v0.16.3-23): the first try killed the job (NODE_FAIL). A new
|
||||
leader had heard from no node yet and sent the controller a slurm.conf without them; slurmctld,
|
||||
restarted on it, dropped node05 and its job. Fixed (141a1f4, 0774e90): slurm.conf's nodes and the
|
||||
controller's address are kept in cluster_state (`slurm-nodes`) and a new leader starts from them; with
|
||||
nothing kept (an upgraded cluster) or the store unread, the controller's files wait up to 2 min for the
|
||||
nodes; a node is told nothing until its controller's address is known (no part stops its slurmd). Tried
|
||||
again: a 5-minute job on lent node05, leader 140 stopped, 150 took over; slurmctld not restarted, the job
|
||||
ran to its end; reclaimed during it, node05 was llmgw's within seconds of its end. Also 3387f97: srun in
|
||||
SrunPortRange=60001-63000, and Slurm's ports in llmgw's firewall for the cluster's networks.
|
||||
- Built 2026-11-11: P1 (`internal/slurm`, `slurm.*` settings), P2
|
||||
(`internal/agent/slurmnode`, proto Slurm/SlurmStatus), P3 (`internal/control/slurmsvc`, wired in the
|
||||
leader; `GET /api/v1/slurm`), P4's commands (`ctl slurm status|setup|lend|reclaim|off`). Decisions on the
|
||||
way: a node not heard from or with Slurm jobs is cordoned, not drained (nothing of llmgw's to move, and a
|
||||
new leader must not move models off every Slurm node); jobs are counted from the node's own slurmstepd,
|
||||
so a controller that is down neither hides them nor, with none, keeps the node from llmgw; a drain whose
|
||||
reason does not start with "llmgw" is a Slurm administrator's and never resumed. Left: ~~the TUI screen~~
|
||||
(done 2026-11-11: console **Slurm**, lend/take back on a row), P5,
|
||||
a try on the PoC (needs agents built from this; slurm-wlm 23.11 on Ubuntu 24.04), GPU minor numbers are
|
||||
taken to be nvidia-smi's index.
|
||||
- In the first install too (2026-11-11): `llmgw server init --slurm`, and a Slurm part on the choices
|
||||
screen, write `slurm: {enabled, controller: <this hostname>, all_nodes: true}` into server.yaml.
|
||||
`slurm.all_nodes` (also `ctl slurm setup --all-nodes`) runs slurmd on every node in the leader's
|
||||
inventory, those that join later too; none is lent. Refused with `--no-node`, or a hostname Slurm does
|
||||
not take.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, behind the token in
|
||||
|
||||
@@ -26,6 +26,93 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Added
|
||||
|
||||
- **An organization's admin.** Besides member and manager, a membership of
|
||||
an organization may now be `admin` (`llmgw users join … --org Acme --role
|
||||
admin`, the role chosen on a membership on the people page, the console,
|
||||
the admin API). An admin sees the organization's page as a manager does and
|
||||
also changes the organization from it: adds, renames and removes its teams
|
||||
and projects, adds people with an approved account by their address and
|
||||
takes them out, makes its members managers or admins, and revokes keys spent
|
||||
in its projects. Limits, approving and suspending accounts, the
|
||||
organization's name and the rest of the cluster stay with the cluster's
|
||||
administrators; an admin cannot take away their own role. Every change is in
|
||||
the audit log.
|
||||
- **Slurm beside llmgw, for training and batch work.** `llmgw ctl slurm setup
|
||||
--controller NODE --nodes NODE,...` (or `--all-nodes`, or `server init
|
||||
--slurm`) has the agents install munge and Slurm from the machines' package
|
||||
sources: slurmctld on one node, slurmd configless on the others, slurm.conf
|
||||
and gres.conf written from what each node reports. Every node stays llmgw's,
|
||||
drained in Slurm, until `ctl slurm lend NODE`: its replicas move off, then
|
||||
Slurm runs jobs there. `ctl slurm reclaim NODE` takes it back once its jobs
|
||||
end. `ctl slurm status` shows each node. Slurm's ports (6817–6818, srun's
|
||||
60001–63000) answer the cluster's networks only, as llmgw's own do.
|
||||
|
||||
### Changed
|
||||
|
||||
- **Forms only for those who may send them.** A viewer sees the people,
|
||||
limits, mirror and cluster pages without their forms, an operator the
|
||||
mirror's and a node's engines but not the rest; before, they were drawn and
|
||||
answered "not found".
|
||||
- **Smaller forms.** A label sits close to its field, a field is as wide as
|
||||
what goes in it rather than the page, buttons are smaller, and the short
|
||||
forms — a new key, a password, connecting an editor, a capacity or a limit —
|
||||
sit on one line or two instead of a screen.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **The serve forms start at the longest context that fits.** The mirror
|
||||
page's form began every GGUF model at 4096 tokens, and the console's at
|
||||
4096 or the plan's figure, whatever the model was trained for. Both now
|
||||
start at what it was trained for, as long as its KV cache fits where the
|
||||
form places it beside the weights, else at the most that does, in whole
|
||||
thousands; the mirror page's llama.cpp slots are as many of that, up to
|
||||
four, as the room left holds, and its memory figure is for that pool. A
|
||||
plan says what the model was trained for (`trained`). Strata stays at
|
||||
its 32768; vLLM's form is unchanged.
|
||||
- **A model that failed to start was started in the same place again.**
|
||||
Placement given the same inputs chooses the same node and card, so an
|
||||
on-demand model whose replica failed to load was retried there, past each
|
||||
backoff, until it was parked, with another card or node free all along.
|
||||
The control plane now remembers where a model's replicas failed (node and
|
||||
cards, up to eight places) and places the next one elsewhere when anywhere
|
||||
else fits; with nowhere else it is tried there as before. Coming up, a
|
||||
change to its definition or `ctl retry` forgets the places. The log says
|
||||
`placing a model away from where it failed to start`, with both places.
|
||||
- **Deleted model files came back after a control plane restart.** An agent
|
||||
registers with the files its startup scan found, and sends the list again
|
||||
only when it changes; a file deleted after the agent started was listed
|
||||
again by a control plane that restarted, until the next change. A new
|
||||
stream now gets the list on its first heartbeat.
|
||||
- **`ctl weights` offered models in use for deletion.** A model of several
|
||||
files (a set, its address ending in `/`) is fetched into a directory named
|
||||
from the address without the slash; the sweep named it from the address
|
||||
with it, so every set in the catalog looked like nobody's. On the PoC that
|
||||
was 330 GB of six catalog models, the qwen3.8-flash-next Q8_0 among them;
|
||||
`--collect`, and `ctl model remove --purge` of a set, would have deleted
|
||||
them. Both now compute the name the fetch does.
|
||||
- **A sign-in took an organization's manager role away.** A group from single
|
||||
sign-on, SCIM or Active Directory that puts someone in an organization wrote
|
||||
the membership again at every sign-in or sync, as a plain member: a manager
|
||||
made one on the people page was one only until they next signed in. A
|
||||
membership the directory names is now added if missing and otherwise left
|
||||
as it is.
|
||||
- **The limits form on the people page's panel** was laid out on one line, its
|
||||
title squeezed into a column.
|
||||
- **Deleting the model files nothing uses said what it was asked to free,
|
||||
not what it freed.** The console's "Delete model files nothing uses" and
|
||||
`ctl weights --collect` added up every file the nodes were asked to
|
||||
delete as soon as the request was sent, and a model of several files —
|
||||
fetched into a directory of its own — was refused by the node as "not
|
||||
empty", so a sweep said tens of gigabytes freed and nothing went. The
|
||||
node now deletes such a directory whole (a link only as a link), and the
|
||||
sweep waits for the nodes to list their files again: freed is what they
|
||||
no longer list, and what they kept is said beside, with where to read
|
||||
why. Removing a model with its files reports the same way.
|
||||
|
||||
## v0.16.3 — 2026-11-03
|
||||
|
||||
Replicas counted per profile — seven on cards and four on CPUs under one name, the cards filled first — and every
|
||||
|
||||
+365
-73
@@ -108,6 +108,7 @@ type AgentMessage struct {
|
||||
// *AgentMessage_KvEvents
|
||||
// *AgentMessage_TrainStatus
|
||||
// *AgentMessage_TrainArtifact
|
||||
// *AgentMessage_SlurmStatus
|
||||
Msg isAgentMessage_Msg `protobuf_oneof:"msg"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
@@ -240,6 +241,15 @@ func (x *AgentMessage) GetTrainArtifact() *TrainArtifact {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *AgentMessage) GetSlurmStatus() *SlurmStatus {
|
||||
if x != nil {
|
||||
if x, ok := x.Msg.(*AgentMessage_SlurmStatus); ok {
|
||||
return x.SlurmStatus
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type isAgentMessage_Msg interface {
|
||||
isAgentMessage_Msg()
|
||||
}
|
||||
@@ -284,6 +294,10 @@ type AgentMessage_TrainArtifact struct {
|
||||
TrainArtifact *TrainArtifact `protobuf:"bytes,10,opt,name=train_artifact,json=trainArtifact,proto3,oneof"`
|
||||
}
|
||||
|
||||
type AgentMessage_SlurmStatus struct {
|
||||
SlurmStatus *SlurmStatus `protobuf:"bytes,11,opt,name=slurm_status,json=slurmStatus,proto3,oneof"`
|
||||
}
|
||||
|
||||
func (*AgentMessage_Register) isAgentMessage_Msg() {}
|
||||
|
||||
func (*AgentMessage_Heartbeat) isAgentMessage_Msg() {}
|
||||
@@ -304,6 +318,8 @@ func (*AgentMessage_TrainStatus) isAgentMessage_Msg() {}
|
||||
|
||||
func (*AgentMessage_TrainArtifact) isAgentMessage_Msg() {}
|
||||
|
||||
func (*AgentMessage_SlurmStatus) isAgentMessage_Msg() {}
|
||||
|
||||
// KVEvents is a change to what one vLLM replica's prefix cache holds, read
|
||||
// by the agent from the engine's KV cache events and sent on for the router's
|
||||
// precise index (plan §7.5 tier 2).
|
||||
@@ -426,6 +442,7 @@ type ControlMessage struct {
|
||||
// *ControlMessage_AgentSettings
|
||||
// *ControlMessage_TrainStart
|
||||
// *ControlMessage_TrainCancel
|
||||
// *ControlMessage_Slurm
|
||||
Msg isControlMessage_Msg `protobuf_oneof:"msg"`
|
||||
// trace is the W3C trace context (traceparent, and tracestate when there
|
||||
// is one) of the control plane's span that sent this command, so what the
|
||||
@@ -618,6 +635,15 @@ func (x *ControlMessage) GetTrainCancel() *TrainCancel {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *ControlMessage) GetSlurm() *Slurm {
|
||||
if x != nil {
|
||||
if x, ok := x.Msg.(*ControlMessage_Slurm); ok {
|
||||
return x.Slurm
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *ControlMessage) GetTrace() map[string]string {
|
||||
if x != nil {
|
||||
return x.Trace
|
||||
@@ -693,6 +719,10 @@ type ControlMessage_TrainCancel struct {
|
||||
TrainCancel *TrainCancel `protobuf:"bytes,17,opt,name=train_cancel,json=trainCancel,proto3,oneof"`
|
||||
}
|
||||
|
||||
type ControlMessage_Slurm struct {
|
||||
Slurm *Slurm `protobuf:"bytes,18,opt,name=slurm,proto3,oneof"`
|
||||
}
|
||||
|
||||
func (*ControlMessage_Start) isControlMessage_Msg() {}
|
||||
|
||||
func (*ControlMessage_Stop) isControlMessage_Msg() {}
|
||||
@@ -725,6 +755,8 @@ func (*ControlMessage_TrainStart) isControlMessage_Msg() {}
|
||||
|
||||
func (*ControlMessage_TrainCancel) isControlMessage_Msg() {}
|
||||
|
||||
func (*ControlMessage_Slurm) isControlMessage_Msg() {}
|
||||
|
||||
// RestartAgent has the agent start again, as llmgw restart asks of every
|
||||
// node: its engines keep serving and are taken over again.
|
||||
type RestartAgent struct {
|
||||
@@ -5251,11 +5283,236 @@ func (x *TrainArtifact) GetLast() bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// Slurm is what this node is in the Slurm cluster run beside llmgw
|
||||
// (BACKLOG P): its controller, one of its nodes, both or neither. Sent when
|
||||
// an agent registers and whenever it changes; the agent installs what is
|
||||
// missing and keeps the daemons running as it says.
|
||||
type Slurm struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
// controller runs slurmctld here, on files.
|
||||
Controller bool `protobuf:"varint,1,opt,name=controller,proto3" json:"controller,omitempty"`
|
||||
// node runs slurmd here, configless, fetching its files from conf_server.
|
||||
Node bool `protobuf:"varint,2,opt,name=node,proto3" json:"node,omitempty"`
|
||||
// name is this node's NodeName in slurm.conf.
|
||||
Name string `protobuf:"bytes,3,opt,name=name,proto3" json:"name,omitempty"`
|
||||
// conf_server is slurmctld's host:port.
|
||||
ConfServer string `protobuf:"bytes,4,opt,name=conf_server,json=confServer,proto3" json:"conf_server,omitempty"`
|
||||
// munge_key is the key every machine of the cluster shares.
|
||||
MungeKey []byte `protobuf:"bytes,5,opt,name=munge_key,json=mungeKey,proto3" json:"munge_key,omitempty"`
|
||||
// files are the controller's, by name in /etc/slurm; empty for a node
|
||||
// that is not the controller.
|
||||
Files map[string]string `protobuf:"bytes,6,rep,name=files,proto3" json:"files,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
|
||||
// lent says Slurm may run jobs here now: resume the node if llmgw drained
|
||||
// it. Otherwise it is drained, its jobs left to finish.
|
||||
Lent bool `protobuf:"varint,7,opt,name=lent,proto3" json:"lent,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *Slurm) Reset() {
|
||||
*x = Slurm{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[48]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *Slurm) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*Slurm) ProtoMessage() {}
|
||||
|
||||
func (x *Slurm) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[48]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use Slurm.ProtoReflect.Descriptor instead.
|
||||
func (*Slurm) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{48}
|
||||
}
|
||||
|
||||
func (x *Slurm) GetController() bool {
|
||||
if x != nil {
|
||||
return x.Controller
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Slurm) GetNode() bool {
|
||||
if x != nil {
|
||||
return x.Node
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *Slurm) GetName() string {
|
||||
if x != nil {
|
||||
return x.Name
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *Slurm) GetConfServer() string {
|
||||
if x != nil {
|
||||
return x.ConfServer
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *Slurm) GetMungeKey() []byte {
|
||||
if x != nil {
|
||||
return x.MungeKey
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *Slurm) GetFiles() map[string]string {
|
||||
if x != nil {
|
||||
return x.Files
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *Slurm) GetLent() bool {
|
||||
if x != nil {
|
||||
return x.Lent
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// SlurmStatus is what this node's Slurm daemons are doing, sent when it
|
||||
// changes and every few minutes while the node has a part in Slurm.
|
||||
type SlurmStatus struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
// version is Slurm's, as "23.11.4"; empty when it is not installed.
|
||||
Version string `protobuf:"bytes,1,opt,name=version,proto3" json:"version,omitempty"`
|
||||
// hardware is `slurmd -C` on a node, which slurm.conf must match.
|
||||
Hardware string `protobuf:"bytes,2,opt,name=hardware,proto3" json:"hardware,omitempty"`
|
||||
SlurmdRunning bool `protobuf:"varint,3,opt,name=slurmd_running,json=slurmdRunning,proto3" json:"slurmd_running,omitempty"`
|
||||
SlurmctldRunning bool `protobuf:"varint,4,opt,name=slurmctld_running,json=slurmctldRunning,proto3" json:"slurmctld_running,omitempty"`
|
||||
// state and reason are the controller's for this node, from scontrol.
|
||||
State string `protobuf:"bytes,5,opt,name=state,proto3" json:"state,omitempty"`
|
||||
Reason string `protobuf:"bytes,6,opt,name=reason,proto3" json:"reason,omitempty"`
|
||||
// jobs is how many job steps run on this node now, counted from their
|
||||
// slurmstepd so a controller that is down does not hide them; -1 when the
|
||||
// agent cannot tell.
|
||||
Jobs int32 `protobuf:"varint,7,opt,name=jobs,proto3" json:"jobs,omitempty"`
|
||||
// error is the last thing that went wrong keeping Slurm, empty when
|
||||
// nothing did.
|
||||
Error string `protobuf:"bytes,8,opt,name=error,proto3" json:"error,omitempty"`
|
||||
// files_sha256 is of the controller's files as written, so the leader
|
||||
// knows they arrived.
|
||||
FilesSha256 string `protobuf:"bytes,9,opt,name=files_sha256,json=filesSha256,proto3" json:"files_sha256,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) Reset() {
|
||||
*x = SlurmStatus{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*SlurmStatus) ProtoMessage() {}
|
||||
|
||||
func (x *SlurmStatus) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[49]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use SlurmStatus.ProtoReflect.Descriptor instead.
|
||||
func (*SlurmStatus) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{49}
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetVersion() string {
|
||||
if x != nil {
|
||||
return x.Version
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetHardware() string {
|
||||
if x != nil {
|
||||
return x.Hardware
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetSlurmdRunning() bool {
|
||||
if x != nil {
|
||||
return x.SlurmdRunning
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetSlurmctldRunning() bool {
|
||||
if x != nil {
|
||||
return x.SlurmctldRunning
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetState() string {
|
||||
if x != nil {
|
||||
return x.State
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetReason() string {
|
||||
if x != nil {
|
||||
return x.Reason
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetJobs() int32 {
|
||||
if x != nil {
|
||||
return x.Jobs
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetError() string {
|
||||
if x != nil {
|
||||
return x.Error
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *SlurmStatus) GetFilesSha256() string {
|
||||
if x != nil {
|
||||
return x.FilesSha256
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
var File_llmgw_v1_agent_proto protoreflect.FileDescriptor
|
||||
|
||||
const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\n" +
|
||||
"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xb7\x04\n" +
|
||||
"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xf3\x04\n" +
|
||||
"\fAgentMessage\x120\n" +
|
||||
"\bregister\x18\x01 \x01(\v2\x12.llmgw.v1.RegisterH\x00R\bregister\x123\n" +
|
||||
"\theartbeat\x18\x02 \x01(\v2\x13.llmgw.v1.HeartbeatH\x00R\theartbeat\x12-\n" +
|
||||
@@ -5269,7 +5526,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\tkv_events\x18\b \x01(\v2\x12.llmgw.v1.KVEventsH\x00R\bkvEvents\x12:\n" +
|
||||
"\ftrain_status\x18\t \x01(\v2\x15.llmgw.v1.TrainStatusH\x00R\vtrainStatus\x12@\n" +
|
||||
"\x0etrain_artifact\x18\n" +
|
||||
" \x01(\v2\x17.llmgw.v1.TrainArtifactH\x00R\rtrainArtifactB\x05\n" +
|
||||
" \x01(\v2\x17.llmgw.v1.TrainArtifactH\x00R\rtrainArtifact\x12:\n" +
|
||||
"\fslurm_status\x18\v \x01(\v2\x15.llmgw.v1.SlurmStatusH\x00R\vslurmStatusB\x05\n" +
|
||||
"\x03msg\"\xb5\x01\n" +
|
||||
"\bKVEvents\x12\x1d\n" +
|
||||
"\n" +
|
||||
@@ -5280,7 +5538,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"clearFirst\x12\x16\n" +
|
||||
"\x06stored\x18\x04 \x03(\x06R\x06stored\x12\x18\n" +
|
||||
"\aremoved\x18\x05 \x03(\x06R\aremoved\x12\x18\n" +
|
||||
"\aproblem\x18\x06 \x01(\tR\aproblem\"\xf8\a\n" +
|
||||
"\aproblem\x18\x06 \x01(\tR\aproblem\"\xa1\b\n" +
|
||||
"\x0eControlMessage\x12,\n" +
|
||||
"\x05start\x18\x01 \x01(\v2\x14.llmgw.v1.StartShardH\x00R\x05start\x12)\n" +
|
||||
"\x04stop\x18\x02 \x01(\v2\x13.llmgw.v1.StopShardH\x00R\x04stop\x12,\n" +
|
||||
@@ -5301,7 +5559,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x0eagent_settings\x18\x0f \x01(\v2\x17.llmgw.v1.AgentSettingsH\x00R\ragentSettings\x127\n" +
|
||||
"\vtrain_start\x18\x10 \x01(\v2\x14.llmgw.v1.TrainStartH\x00R\n" +
|
||||
"trainStart\x12:\n" +
|
||||
"\ftrain_cancel\x18\x11 \x01(\v2\x15.llmgw.v1.TrainCancelH\x00R\vtrainCancel\x129\n" +
|
||||
"\ftrain_cancel\x18\x11 \x01(\v2\x15.llmgw.v1.TrainCancelH\x00R\vtrainCancel\x12'\n" +
|
||||
"\x05slurm\x18\x12 \x01(\v2\x0f.llmgw.v1.SlurmH\x00R\x05slurm\x129\n" +
|
||||
"\x05trace\x18\f \x03(\v2#.llmgw.v1.ControlMessage.TraceEntryR\x05trace\x1a8\n" +
|
||||
"\n" +
|
||||
"TraceEntry\x12\x10\n" +
|
||||
@@ -5754,7 +6013,32 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x04kind\x18\x02 \x01(\tR\x04kind\x12\x10\n" +
|
||||
"\x03seq\x18\x03 \x01(\x05R\x03seq\x12\x12\n" +
|
||||
"\x04data\x18\x04 \x01(\fR\x04data\x12\x12\n" +
|
||||
"\x04last\x18\x05 \x01(\bR\x04last2O\n" +
|
||||
"\x04last\x18\x05 \x01(\bR\x04last\"\x8d\x02\n" +
|
||||
"\x05Slurm\x12\x1e\n" +
|
||||
"\n" +
|
||||
"controller\x18\x01 \x01(\bR\n" +
|
||||
"controller\x12\x12\n" +
|
||||
"\x04node\x18\x02 \x01(\bR\x04node\x12\x12\n" +
|
||||
"\x04name\x18\x03 \x01(\tR\x04name\x12\x1f\n" +
|
||||
"\vconf_server\x18\x04 \x01(\tR\n" +
|
||||
"confServer\x12\x1b\n" +
|
||||
"\tmunge_key\x18\x05 \x01(\fR\bmungeKey\x120\n" +
|
||||
"\x05files\x18\x06 \x03(\v2\x1a.llmgw.v1.Slurm.FilesEntryR\x05files\x12\x12\n" +
|
||||
"\x04lent\x18\a \x01(\bR\x04lent\x1a8\n" +
|
||||
"\n" +
|
||||
"FilesEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\x92\x02\n" +
|
||||
"\vSlurmStatus\x12\x18\n" +
|
||||
"\aversion\x18\x01 \x01(\tR\aversion\x12\x1a\n" +
|
||||
"\bhardware\x18\x02 \x01(\tR\bhardware\x12%\n" +
|
||||
"\x0eslurmd_running\x18\x03 \x01(\bR\rslurmdRunning\x12+\n" +
|
||||
"\x11slurmctld_running\x18\x04 \x01(\bR\x10slurmctldRunning\x12\x14\n" +
|
||||
"\x05state\x18\x05 \x01(\tR\x05state\x12\x16\n" +
|
||||
"\x06reason\x18\x06 \x01(\tR\x06reason\x12\x12\n" +
|
||||
"\x04jobs\x18\a \x01(\x05R\x04jobs\x12\x14\n" +
|
||||
"\x05error\x18\b \x01(\tR\x05error\x12!\n" +
|
||||
"\ffiles_sha256\x18\t \x01(\tR\vfilesSha2562O\n" +
|
||||
"\fAgentService\x12?\n" +
|
||||
"\aConnect\x12\x16.llmgw.v1.AgentMessage\x1a\x18.llmgw.v1.ControlMessage(\x010\x01B\x93\x01\n" +
|
||||
"\fcom.llmgw.v1B\n" +
|
||||
@@ -5773,7 +6057,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
|
||||
}
|
||||
|
||||
var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
|
||||
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 56)
|
||||
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 59)
|
||||
var file_llmgw_v1_agent_proto_goTypes = []any{
|
||||
(ShardStatus_State)(0), // 0: llmgw.v1.ShardStatus.State
|
||||
(*AgentMessage)(nil), // 1: llmgw.v1.AgentMessage
|
||||
@@ -5824,14 +6108,17 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
|
||||
(*TrainCancel)(nil), // 46: llmgw.v1.TrainCancel
|
||||
(*TrainStatus)(nil), // 47: llmgw.v1.TrainStatus
|
||||
(*TrainArtifact)(nil), // 48: llmgw.v1.TrainArtifact
|
||||
nil, // 49: llmgw.v1.ControlMessage.TraceEntry
|
||||
nil, // 50: llmgw.v1.Register.LabelsEntry
|
||||
nil, // 51: llmgw.v1.Register.EngineVersionsEntry
|
||||
nil, // 52: llmgw.v1.GPU.NicPathsEntry
|
||||
nil, // 53: llmgw.v1.IBPort.CountersEntry
|
||||
nil, // 54: llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
nil, // 55: llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
nil, // 56: llmgw.v1.EngineProfile.EnvEntry
|
||||
(*Slurm)(nil), // 49: llmgw.v1.Slurm
|
||||
(*SlurmStatus)(nil), // 50: llmgw.v1.SlurmStatus
|
||||
nil, // 51: llmgw.v1.ControlMessage.TraceEntry
|
||||
nil, // 52: llmgw.v1.Register.LabelsEntry
|
||||
nil, // 53: llmgw.v1.Register.EngineVersionsEntry
|
||||
nil, // 54: llmgw.v1.GPU.NicPathsEntry
|
||||
nil, // 55: llmgw.v1.IBPort.CountersEntry
|
||||
nil, // 56: llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
nil, // 57: llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
nil, // 58: llmgw.v1.EngineProfile.EnvEntry
|
||||
nil, // 59: llmgw.v1.Slurm.FilesEntry
|
||||
}
|
||||
var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
8, // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
|
||||
@@ -5844,64 +6131,67 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
2, // 7: llmgw.v1.AgentMessage.kv_events:type_name -> llmgw.v1.KVEvents
|
||||
47, // 8: llmgw.v1.AgentMessage.train_status:type_name -> llmgw.v1.TrainStatus
|
||||
48, // 9: llmgw.v1.AgentMessage.train_artifact:type_name -> llmgw.v1.TrainArtifact
|
||||
30, // 10: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
|
||||
33, // 11: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
|
||||
34, // 12: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
|
||||
35, // 13: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
|
||||
32, // 14: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
|
||||
24, // 15: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
|
||||
36, // 16: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
|
||||
38, // 17: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
|
||||
40, // 18: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
|
||||
42, // 19: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
|
||||
7, // 20: llmgw.v1.ControlMessage.control_planes:type_name -> llmgw.v1.ControlPlanes
|
||||
5, // 21: llmgw.v1.ControlMessage.firewall:type_name -> llmgw.v1.Firewall
|
||||
4, // 22: llmgw.v1.ControlMessage.restart_agent:type_name -> llmgw.v1.RestartAgent
|
||||
6, // 23: llmgw.v1.ControlMessage.agent_settings:type_name -> llmgw.v1.AgentSettings
|
||||
45, // 24: llmgw.v1.ControlMessage.train_start:type_name -> llmgw.v1.TrainStart
|
||||
46, // 25: llmgw.v1.ControlMessage.train_cancel:type_name -> llmgw.v1.TrainCancel
|
||||
49, // 26: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
|
||||
50, // 27: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
|
||||
12, // 28: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
|
||||
14, // 29: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
51, // 30: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
|
||||
11, // 31: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
|
||||
15, // 32: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
|
||||
10, // 33: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
|
||||
9, // 34: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
|
||||
21, // 35: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
|
||||
0, // 36: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
27, // 37: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
|
||||
52, // 38: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
|
||||
13, // 39: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
|
||||
18, // 40: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
|
||||
17, // 41: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
|
||||
53, // 42: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
|
||||
20, // 43: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
|
||||
10, // 44: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
|
||||
15, // 45: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
|
||||
54, // 46: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
26, // 47: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
|
||||
16, // 48: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
|
||||
19, // 49: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
|
||||
14, // 50: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
22, // 51: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
|
||||
21, // 52: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
|
||||
25, // 53: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
|
||||
55, // 54: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
0, // 55: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
29, // 56: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
|
||||
27, // 57: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
|
||||
31, // 58: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
|
||||
56, // 59: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
|
||||
44, // 60: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
|
||||
1, // 61: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
|
||||
3, // 62: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
|
||||
62, // [62:63] is the sub-list for method output_type
|
||||
61, // [61:62] is the sub-list for method input_type
|
||||
61, // [61:61] is the sub-list for extension type_name
|
||||
61, // [61:61] is the sub-list for extension extendee
|
||||
0, // [0:61] is the sub-list for field type_name
|
||||
50, // 10: llmgw.v1.AgentMessage.slurm_status:type_name -> llmgw.v1.SlurmStatus
|
||||
30, // 11: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
|
||||
33, // 12: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
|
||||
34, // 13: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
|
||||
35, // 14: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
|
||||
32, // 15: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
|
||||
24, // 16: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
|
||||
36, // 17: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
|
||||
38, // 18: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
|
||||
40, // 19: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
|
||||
42, // 20: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
|
||||
7, // 21: llmgw.v1.ControlMessage.control_planes:type_name -> llmgw.v1.ControlPlanes
|
||||
5, // 22: llmgw.v1.ControlMessage.firewall:type_name -> llmgw.v1.Firewall
|
||||
4, // 23: llmgw.v1.ControlMessage.restart_agent:type_name -> llmgw.v1.RestartAgent
|
||||
6, // 24: llmgw.v1.ControlMessage.agent_settings:type_name -> llmgw.v1.AgentSettings
|
||||
45, // 25: llmgw.v1.ControlMessage.train_start:type_name -> llmgw.v1.TrainStart
|
||||
46, // 26: llmgw.v1.ControlMessage.train_cancel:type_name -> llmgw.v1.TrainCancel
|
||||
49, // 27: llmgw.v1.ControlMessage.slurm:type_name -> llmgw.v1.Slurm
|
||||
51, // 28: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
|
||||
52, // 29: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
|
||||
12, // 30: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
|
||||
14, // 31: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
53, // 32: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
|
||||
11, // 33: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
|
||||
15, // 34: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
|
||||
10, // 35: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
|
||||
9, // 36: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
|
||||
21, // 37: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
|
||||
0, // 38: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
27, // 39: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
|
||||
54, // 40: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
|
||||
13, // 41: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
|
||||
18, // 42: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
|
||||
17, // 43: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
|
||||
55, // 44: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
|
||||
20, // 45: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
|
||||
10, // 46: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
|
||||
15, // 47: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
|
||||
56, // 48: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
26, // 49: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
|
||||
16, // 50: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
|
||||
19, // 51: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
|
||||
14, // 52: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
22, // 53: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
|
||||
21, // 54: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
|
||||
25, // 55: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
|
||||
57, // 56: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
0, // 57: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
29, // 58: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
|
||||
27, // 59: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
|
||||
31, // 60: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
|
||||
58, // 61: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
|
||||
44, // 62: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
|
||||
59, // 63: llmgw.v1.Slurm.files:type_name -> llmgw.v1.Slurm.FilesEntry
|
||||
1, // 64: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
|
||||
3, // 65: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
|
||||
65, // [65:66] is the sub-list for method output_type
|
||||
64, // [64:65] is the sub-list for method input_type
|
||||
64, // [64:64] is the sub-list for extension type_name
|
||||
64, // [64:64] is the sub-list for extension extendee
|
||||
0, // [0:64] is the sub-list for field type_name
|
||||
}
|
||||
|
||||
func init() { file_llmgw_v1_agent_proto_init() }
|
||||
@@ -5920,6 +6210,7 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
(*AgentMessage_KvEvents)(nil),
|
||||
(*AgentMessage_TrainStatus)(nil),
|
||||
(*AgentMessage_TrainArtifact)(nil),
|
||||
(*AgentMessage_SlurmStatus)(nil),
|
||||
}
|
||||
file_llmgw_v1_agent_proto_msgTypes[2].OneofWrappers = []any{
|
||||
(*ControlMessage_Start)(nil),
|
||||
@@ -5938,6 +6229,7 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
(*ControlMessage_AgentSettings)(nil),
|
||||
(*ControlMessage_TrainStart)(nil),
|
||||
(*ControlMessage_TrainCancel)(nil),
|
||||
(*ControlMessage_Slurm)(nil),
|
||||
}
|
||||
type x struct{}
|
||||
out := protoimpl.TypeBuilder{
|
||||
@@ -5945,7 +6237,7 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
|
||||
RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
|
||||
NumEnums: 1,
|
||||
NumMessages: 56,
|
||||
NumMessages: 59,
|
||||
NumExtensions: 0,
|
||||
NumServices: 1,
|
||||
},
|
||||
|
||||
@@ -30,6 +30,7 @@ message AgentMessage {
|
||||
KVEvents kv_events = 8;
|
||||
TrainStatus train_status = 9;
|
||||
TrainArtifact train_artifact = 10;
|
||||
SlurmStatus slurm_status = 11;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -78,6 +79,7 @@ message ControlMessage {
|
||||
AgentSettings agent_settings = 15;
|
||||
TrainStart train_start = 16;
|
||||
TrainCancel train_cancel = 17;
|
||||
Slurm slurm = 18;
|
||||
}
|
||||
// trace is the W3C trace context (traceparent, and tracestate when there
|
||||
// is one) of the control plane's span that sent this command, so what the
|
||||
@@ -986,3 +988,50 @@ message TrainArtifact {
|
||||
bytes data = 4;
|
||||
bool last = 5;
|
||||
}
|
||||
|
||||
// Slurm is what this node is in the Slurm cluster run beside llmgw
|
||||
// (BACKLOG P): its controller, one of its nodes, both or neither. Sent when
|
||||
// an agent registers and whenever it changes; the agent installs what is
|
||||
// missing and keeps the daemons running as it says.
|
||||
message Slurm {
|
||||
// controller runs slurmctld here, on files.
|
||||
bool controller = 1;
|
||||
// node runs slurmd here, configless, fetching its files from conf_server.
|
||||
bool node = 2;
|
||||
// name is this node's NodeName in slurm.conf.
|
||||
string name = 3;
|
||||
// conf_server is slurmctld's host:port.
|
||||
string conf_server = 4;
|
||||
// munge_key is the key every machine of the cluster shares.
|
||||
bytes munge_key = 5;
|
||||
// files are the controller's, by name in /etc/slurm; empty for a node
|
||||
// that is not the controller.
|
||||
map<string, string> files = 6;
|
||||
// lent says Slurm may run jobs here now: resume the node if llmgw drained
|
||||
// it. Otherwise it is drained, its jobs left to finish.
|
||||
bool lent = 7;
|
||||
}
|
||||
|
||||
// SlurmStatus is what this node's Slurm daemons are doing, sent when it
|
||||
// changes and every few minutes while the node has a part in Slurm.
|
||||
message SlurmStatus {
|
||||
// version is Slurm's, as "23.11.4"; empty when it is not installed.
|
||||
string version = 1;
|
||||
// hardware is `slurmd -C` on a node, which slurm.conf must match.
|
||||
string hardware = 2;
|
||||
bool slurmd_running = 3;
|
||||
bool slurmctld_running = 4;
|
||||
// state and reason are the controller's for this node, from scontrol.
|
||||
string state = 5;
|
||||
string reason = 6;
|
||||
// jobs is how many job steps run on this node now, counted from their
|
||||
// slurmstepd so a controller that is down does not hide them; -1 when the
|
||||
// agent cannot tell.
|
||||
int32 jobs = 7;
|
||||
// error is the last thing that went wrong keeping Slurm, empty when
|
||||
// nothing did.
|
||||
string error = 8;
|
||||
// files_sha256 is of the controller's files as written, so the leader
|
||||
// knows they arrived.
|
||||
string files_sha256 = 9;
|
||||
}
|
||||
+136
-1
@@ -116,6 +116,9 @@
|
||||
{
|
||||
"name": "settings"
|
||||
},
|
||||
{
|
||||
"name": "slurm"
|
||||
},
|
||||
{
|
||||
"name": "summary"
|
||||
},
|
||||
@@ -5078,6 +5081,34 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/slurm": {
|
||||
"get": {
|
||||
"operationId": "getSlurm",
|
||||
"summary": "The Slurm cluster beside llmgw, and where each of its nodes is",
|
||||
"description": "From the cluster's settings (slurm.*) and what each node's agent last said. A node's phase is llmgw (drained in Slurm, no job left), lending (its replicas moving off), lent, reclaiming (drained in Slurm, jobs still running) or unknown (its agent has not said).",
|
||||
"tags": [
|
||||
"slurm"
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/slurmsvc.Status"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/summary": {
|
||||
"get": {
|
||||
"operationId": "getSummary",
|
||||
@@ -6234,7 +6265,7 @@
|
||||
},
|
||||
"CollectResult": {
|
||||
"type": "object",
|
||||
"description": "CollectResult is what was deleted on one node, or why nothing was.",
|
||||
"description": "CollectResult is what was deleted on one node, or why nothing was: what the node no longer lists once asked, not what it was asked to delete.",
|
||||
"properties": {
|
||||
"node": {
|
||||
"type": "string"
|
||||
@@ -6246,6 +6277,14 @@
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"left": {
|
||||
"type": "integer",
|
||||
"description": "Left and LeftBytes are what the node still lists: refused, or not gone by the time the sweep stopped waiting. Its agent's log says why."
|
||||
},
|
||||
"left_bytes": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"error": {
|
||||
"type": "string"
|
||||
}
|
||||
@@ -11737,6 +11776,10 @@
|
||||
"context": {
|
||||
"type": "integer"
|
||||
},
|
||||
"trained": {
|
||||
"type": "integer",
|
||||
"description": "Trained is the context the model was trained for, the most a request may use; 0 when neither its header nor its config says."
|
||||
},
|
||||
"kv_bytes_per_token": {
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
@@ -12166,6 +12209,98 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"slurmsvc.NodeStatus": {
|
||||
"type": "object",
|
||||
"description": "NodeStatus is one node as the service sees it.",
|
||||
"properties": {
|
||||
"node": {
|
||||
"type": "string"
|
||||
},
|
||||
"phase": {
|
||||
"type": "string"
|
||||
},
|
||||
"replicas": {
|
||||
"type": "integer"
|
||||
},
|
||||
"slurm": {
|
||||
"$ref": "#/components/schemas/slurmsvc.Reported"
|
||||
},
|
||||
"connected": {
|
||||
"type": "boolean"
|
||||
}
|
||||
}
|
||||
},
|
||||
"slurmsvc.Reported": {
|
||||
"type": "object",
|
||||
"description": "Reported is the agent's SlurmStatus, for the admin API.",
|
||||
"properties": {
|
||||
"version": {
|
||||
"type": "string"
|
||||
},
|
||||
"hardware": {
|
||||
"type": "string"
|
||||
},
|
||||
"slurmd_running": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"slurmctld_running": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"state": {
|
||||
"type": "string"
|
||||
},
|
||||
"reason": {
|
||||
"type": "string"
|
||||
},
|
||||
"jobs": {
|
||||
"type": "integer"
|
||||
},
|
||||
"error": {
|
||||
"type": "string"
|
||||
},
|
||||
"files_sha256": {
|
||||
"type": "string"
|
||||
},
|
||||
"at": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
}
|
||||
}
|
||||
},
|
||||
"slurmsvc.Status": {
|
||||
"type": "object",
|
||||
"description": "Status is the whole Slurm cluster, for `ctl slurm status`.",
|
||||
"properties": {
|
||||
"enabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"cluster": {
|
||||
"type": "string"
|
||||
},
|
||||
"controller": {
|
||||
"type": "string"
|
||||
},
|
||||
"partition": {
|
||||
"type": "string"
|
||||
},
|
||||
"controller_status": {
|
||||
"$ref": "#/components/schemas/slurmsvc.Reported"
|
||||
},
|
||||
"nodes": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/slurmsvc.NodeStatus"
|
||||
}
|
||||
},
|
||||
"waiting": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Waiting are nodes left out of slurm.conf until they report their hardware."
|
||||
}
|
||||
}
|
||||
},
|
||||
"store.AuditEvent": {
|
||||
"type": "object",
|
||||
"description": "AuditEvent is one change: who made it, through what, to what, and how it ended. Detail never holds a secret; the writer takes them out.",
|
||||
|
||||
+12
-1
@@ -207,6 +207,16 @@ starting — and whether it ended done, failed, or waiting for room on busy
|
||||
cards. The entry shows how many are running. Enter on one shows its steps.
|
||||
`llmgw ctl jobs` is the same, and `llmgw ctl jobs ID` follows one.
|
||||
|
||||
**Slurm** is the Slurm cluster run beside llmgw, when there is one: its
|
||||
controller and whether slurmctld runs, and each node that runs slurmd —
|
||||
whose it is now (llmgw's, Slurm's, or on its way between them), its state in
|
||||
Slurm, the jobs running there. On a row, **Lend to Slurm…** moves the node's
|
||||
replicas off and then lets Slurm run jobs there; **Take back from Slurm…**
|
||||
starts no new job there and gives it back to llmgw when its jobs have ended.
|
||||
Both change `slurm.lent` in the cluster's settings, as `llmgw ctl slurm lend`
|
||||
and `reclaim` do. Turning Slurm on and choosing its nodes is `llmgw ctl slurm
|
||||
setup`, or the Slurm part of the first install.
|
||||
|
||||
**Logs** shows a node's agent log, newest at the bottom, errors and warnings
|
||||
coloured.
|
||||
|
||||
@@ -220,7 +230,8 @@ and change it with the same commands as `llmgw users`, `llmgw keys` and
|
||||
dashboard, and what they belong to. **A** creates an account, already
|
||||
approved. On a row: approve one waiting, suspend or reinstate, give or take
|
||||
the cluster dashboard, add to or take out of an organization, team or
|
||||
project, their limits, a new password (their sessions close), delete, and
|
||||
project (on an organization as a member, its manager, who sees its page, or
|
||||
its admin, who also changes it), their limits, a new password (their sessions close), delete, and
|
||||
**Forget them…** for a request to be forgotten (`llmgw users forget`).
|
||||
|
||||
**API keys** — every key, its project, owner, state and last use. **A**
|
||||
|
||||
+17
-2
@@ -166,13 +166,28 @@ is a flag too, and the plan shows the command with them: `--starter qwen3-8b`
|
||||
`--judge-fail open`, `--judge-categories`, `--judge-replicas`,
|
||||
`--judge-budget`, `--language tr`, `--signup-open`, `--default-rpm`,
|
||||
`--default-tpm`, `--max-key-ttl`, `--media` (pictures and 3D: ComfyUI on the
|
||||
nodes, and `presets.auto`), `--evict-after-idle` and `--min-lifetime`. They
|
||||
nodes, and `presets.auto`), `--slurm` (see below), `--evict-after-idle` and `--min-lifetime`. They
|
||||
go into `server.yaml` as init writes it, and become the cluster's first
|
||||
settings when it starts; a `server.yaml` already there is never overwritten,
|
||||
so run again they change nothing — the settings page does. An NVIDIA card with no working
|
||||
driver is reported rather than silently ignored; install the driver and run
|
||||
init again to serve on it.
|
||||
|
||||
**Slurm, for training and batch jobs.** `--slurm`, or the Slurm part on the
|
||||
choices screen, runs a Slurm cluster beside llmgw: slurmctld on this
|
||||
machine's node, slurmd on every node, those that join later too
|
||||
(`slurm.all_nodes`). The agents install munge and Slurm (22.05 or newer)
|
||||
from the machines' own package sources. No node is Slurm's at first: each is
|
||||
drained in Slurm and serves models until `llmgw ctl slurm lend NODE` gives it
|
||||
to Slurm — its models move off first — and `llmgw ctl slurm reclaim NODE`
|
||||
takes it back once its jobs end. Jobs are submitted on this machine with
|
||||
`sbatch` and `srun`; `llmgw ctl slurm status` shows where each node is. It
|
||||
needs this machine to be a node, and its hostname to be a name Slurm takes.
|
||||
On a cluster installed without it, `llmgw ctl slurm setup --controller NODE
|
||||
--all-nodes` does the same later. Slurm's ports (6817–6818, and srun's 60001–63000) answer the cluster's
|
||||
networks only, as llmgw's own do; a firewall of the site's own between the
|
||||
nodes must let them through.
|
||||
|
||||
**Secure by default.** What init writes is the hardened configuration, not a
|
||||
demonstration one:
|
||||
|
||||
@@ -3145,7 +3160,7 @@ loaded at boot by `llmgw-firewall.service`):
|
||||
| Machine | Ports, for the cluster only |
|
||||
| --- | --- |
|
||||
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
|
||||
| node | engine ports (`control.port_range`, 20000–20999), 9100 |
|
||||
| node | engine ports (`control.port_range`, 20000–20999), 9100; with a part in Slurm also 6817–6818 (slurmctld, slurmd) and 60001–63000 (srun's `SrunPortRange`) |
|
||||
| store member | 5432, 8008, 2379, 2380, 9100 |
|
||||
| mirror | 8081, 8082, 9100 |
|
||||
|
||||
|
||||
Binary file not shown.
+32
-11
@@ -192,22 +192,41 @@ auth:
|
||||
A personal key has no project, so it keeps no conversations or agent notes;
|
||||
those are kept per project.
|
||||
|
||||
### Organization managers
|
||||
### Organization managers and admins
|
||||
|
||||
A member of an organization can be made its manager:
|
||||
A member of an organization can be made its manager, who sees it, or its
|
||||
admin, who also changes it:
|
||||
|
||||
```
|
||||
llmgw users join boss@acme.com --org Acme --role manager
|
||||
llmgw users join lead@acme.com --org Acme --role admin
|
||||
```
|
||||
|
||||
or with the star on their organization membership on `/admin/users`. A manager
|
||||
gets an **Organization** link and a page at `/org`: usage over time by team,
|
||||
or with the role chosen on their organization membership on `/admin/users`.
|
||||
Both get an **Organization** link and a page at `/org`: usage over time by team,
|
||||
project or person, the teams and projects with their limits and spend, and every
|
||||
member. Only what was spent in that organization's projects is on it; what its
|
||||
people do elsewhere is not. The page is read-only — a manager is not an
|
||||
administrator of the cluster, and limits and memberships stay with those who
|
||||
are. An administrator following `/org` lands on the same organization's panel
|
||||
on the people page instead, where all of that can be changed.
|
||||
people do elsewhere is not.
|
||||
|
||||
A manager's page is read-only. An organization's admin also changes the
|
||||
organization from it:
|
||||
|
||||
- adds and renames its teams and projects, and removes an empty one;
|
||||
- adds a person by their sign-in address to the organization, a team or a
|
||||
project, and takes them out (their keys there are revoked with it);
|
||||
- makes a member of the organization a manager or an admin, or neither;
|
||||
- revokes a key spent in its projects.
|
||||
|
||||
What reaches past the organization stays with the cluster's administrators:
|
||||
its limits and budgets, approving or suspending an account (someone not yet
|
||||
approved cannot be added), the organization's own name, and anything else on
|
||||
the cluster. An admin cannot take away their own role. A group from single
|
||||
sign-on or Active Directory that puts someone in the organization does not, by
|
||||
saying so again at the next sign-in, take a manager's or admin's role away.
|
||||
|
||||
An administrator of the cluster following `/org` lands on the same
|
||||
organization's panel on the people page instead, where all of that can be
|
||||
changed.
|
||||
|
||||
`llmgw keys init` still works for provisioning scripts and is idempotent, but it
|
||||
will no longer move a team or project that already exists under another parent;
|
||||
@@ -517,8 +536,10 @@ included: `llmgw ctl audit`, or **Audit** in the console.
|
||||
suspends accounts from it; see [install.md](install.md#single-sign-on).
|
||||
Without them accounts are local, with local passwords.
|
||||
- **Staff roles are coarse.** `viewer`, `operator`, `billing` and `admin`
|
||||
(`llmgw users role`) decide what someone may see and do under /admin; an
|
||||
organization's manager can view it but not change it. Every change is in
|
||||
the audit log (`llmgw ctl audit`).
|
||||
(`llmgw users role`) decide what someone may see and do under /admin, and a
|
||||
page draws only the forms that person may send: a viewer sees no forms. An
|
||||
organization's manager can view it; its admin also changes its teams,
|
||||
projects, people and keys. Every change is in the audit log (`llmgw ctl
|
||||
audit`).
|
||||
- **Rate limits are per gateway instance.** With two instances each counts its own,
|
||||
so the effective limit doubles. v1 is single-instance by design.
|
||||
@@ -376,6 +376,7 @@ rather than repeated.
|
||||
| `mail` | `mail.Config` | [Server](#server) |
|
||||
| `serving` | `ServingSettings` | [Server](#server) |
|
||||
| `firewall` | `FirewallSettings` | [Server](#server) |
|
||||
| `slurm` | `SlurmSettings` | [Server](#server) |
|
||||
| `presets` | `PresetSettings` | [Server](#server) |
|
||||
| `on_demand` | `OnDemandSettings` | [Server](#server) |
|
||||
| `agents` | `AgentSettings` | [Server](#server) |
|
||||
@@ -401,6 +402,13 @@ rather than repeated.
|
||||
| `timeout` | `time.Duration` | [Service](#service) |
|
||||
| `stopped` | `bool` | [ServingSettings](#servingsettings) |
|
||||
| `bin` | `string` | [SimConfig](#simconfig) |
|
||||
| `enabled` | `bool` | [SlurmSettings](#slurmsettings) |
|
||||
| `cluster_name` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `controller` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `nodes` | `[]string` | [SlurmSettings](#slurmsettings) |
|
||||
| `all_nodes` | `bool` | [SlurmSettings](#slurmsettings) |
|
||||
| `lent` | `[]string` | [SlurmSettings](#slurmsettings) |
|
||||
| `partition` | `string` | [SlurmSettings](#slurmsettings) |
|
||||
| `dir` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `python` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `engine` | `string` | [StrataConfig](#strataconfig) |
|
||||
@@ -3279,6 +3287,15 @@ Firewall turns llmgw's firewall on every machine on or off.
|
||||
|
||||
Takes a [FirewallSettings](#firewallsettings) block.
|
||||
|
||||
### `slurm`
|
||||
|
||||
`SlurmSettings`
|
||||
|
||||
Slurm is a Slurm cluster run beside llmgw on some of its nodes, and
|
||||
which of them it may use now.
|
||||
|
||||
Takes a [SlurmSettings](#slurmsettings) block.
|
||||
|
||||
### `presets`
|
||||
|
||||
`PresetSettings`
|
||||
@@ -3480,6 +3497,57 @@ entries ask. In force on every control plane within seconds.
|
||||
Bin is the llmgw binary itself; the engine is one of its subcommands.
|
||||
Empty means whatever started this agent, which is right almost always.
|
||||
|
||||
## SlurmSettings
|
||||
|
||||
### `enabled`
|
||||
|
||||
`bool`
|
||||
|
||||
Enabled runs Slurm on the nodes below. False stops llmgw keeping it;
|
||||
what was installed is left installed, and every node is llmgw's.
|
||||
|
||||
### `cluster_name`
|
||||
|
||||
`string`
|
||||
|
||||
ClusterName is Slurm's ClusterName. Empty is "llmgw".
|
||||
|
||||
### `controller`
|
||||
|
||||
`string`
|
||||
|
||||
Controller is the llmgw node whose agent runs slurmctld. It needs no
|
||||
GPU, and need not be one of Nodes.
|
||||
|
||||
### `nodes`
|
||||
|
||||
`[]string`
|
||||
|
||||
Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained
|
||||
in Slurm, until it is also in Lent.
|
||||
|
||||
### `all_nodes`
|
||||
|
||||
`bool`
|
||||
|
||||
AllNodes runs slurmd on every llmgw node, those that join later too,
|
||||
besides any named in Nodes. Each is llmgw's until it is lent, as any.
|
||||
|
||||
### `lent`
|
||||
|
||||
`[]string`
|
||||
|
||||
Lent are the nodes of Nodes Slurm may run jobs on now. A node put here
|
||||
is drained of llmgw's replicas first, as `ctl drain` does, and resumed
|
||||
in Slurm once none is left; a node taken out is drained in Slurm and
|
||||
given back to llmgw when its last job ends.
|
||||
|
||||
### `partition`
|
||||
|
||||
`string`
|
||||
|
||||
Partition is the one partition every node is in. Empty is "gpu".
|
||||
|
||||
## StrataConfig
|
||||
|
||||
### `dir`
|
||||
|
||||
+46
-6
@@ -35,6 +35,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/slurmnode"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
|
||||
@@ -97,7 +98,10 @@ type Agent struct {
|
||||
// (nil: this machine's shell).
|
||||
fwMu sync.Mutex
|
||||
fwRunner firewall.Runner
|
||||
inv discovery.Inventory
|
||||
// fwLast is the last firewall message, applied again when the node
|
||||
// is given a part in Slurm. Under fwMu.
|
||||
fwLast *llmgwv1.Firewall
|
||||
inv discovery.Inventory
|
||||
// shardReplica remembers which replica a shard belongs to, so a status
|
||||
// report can name both. The control plane keys on the replica.
|
||||
shardReplica map[string]string
|
||||
@@ -148,6 +152,10 @@ type Agent struct {
|
||||
|
||||
// trainer runs LoRA training jobs, one at a time (BACKLOG F4).
|
||||
trainer *train.Runner
|
||||
// slurm keeps the node's part in Slurm (BACKLOG P); slurmOut is where
|
||||
// its status goes, under mu.
|
||||
slurm *slurmnode.Keeper
|
||||
slurmOut chan<- *llmgwv1.AgentMessage
|
||||
}
|
||||
|
||||
// connectHTTPClient is the HTTP client connect uses; an interface so tests can
|
||||
@@ -206,6 +214,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
|
||||
}
|
||||
a.trainer = train.New(train.Env{EnginesDir: enginesDir, WorkDir: filepath.Join(cfg.Superv.WorkDir, "trainer"),
|
||||
RunAs: runAs, Log: log})
|
||||
a.slurm = slurmnode.New(slurmnode.Local{}, a.prov, slurmMarker, log)
|
||||
return a, nil
|
||||
}
|
||||
|
||||
@@ -232,6 +241,7 @@ func (a *Agent) Run(ctx context.Context) error {
|
||||
}
|
||||
go a.provisionLoop(ctx)
|
||||
go a.ensureDiags(ctx)
|
||||
go a.slurm.Run(ctx, a.sendSlurm)
|
||||
|
||||
attempt := 0
|
||||
next := 0
|
||||
@@ -530,6 +540,18 @@ func (a *Agent) scan(ctx context.Context) error {
|
||||
return err
|
||||
}
|
||||
|
||||
// forgetSent is a new stream: a control plane that knows nothing of where
|
||||
// installs stand, nor which files are here beyond what registration says —
|
||||
// and registration says what the startup scan found. So the next heartbeat
|
||||
// tells it both again. Without it, files deleted after the agent started came
|
||||
// back on a control plane that restarted, and stayed until the next change.
|
||||
func (a *Agent) forgetSent() {
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
a.provSent = ""
|
||||
a.lastWeights = ""
|
||||
}
|
||||
|
||||
// session runs one connection until it breaks.
|
||||
func (a *Agent) session(ctx context.Context, server string) error {
|
||||
client := llmgwv1connect.NewAgentServiceClient(a.http, server)
|
||||
@@ -566,14 +588,12 @@ func (a *Agent) session(ctx context.Context, server string) error {
|
||||
reg.EngineVersions = a.engines(parent)
|
||||
reg.EnginesProbed = true
|
||||
reg.InstallableEngines = slices.DeleteFunc(provision.Installable(), a.disabled)
|
||||
reg.Features = []string{FeatureFabricTest}
|
||||
reg.Features = []string{FeatureFabricTest, FeatureSlurm}
|
||||
reg.RunningShards = a.runningShards()
|
||||
a.mu.Lock()
|
||||
h := a.prov.Host
|
||||
// A new stream is a control plane that knows nothing of where installs
|
||||
// stand, so the next heartbeat tells it again.
|
||||
a.provSent = ""
|
||||
a.mu.Unlock()
|
||||
a.forgetSent()
|
||||
reg.Host = &llmgwv1.Host{
|
||||
OsId: h.OSID, OsVersion: h.OSVersion, OsCodename: h.OSCodename,
|
||||
Arch: h.Arch, CpuFlags: h.CPUFlags, NvidiaDriver: h.NVIDIADriver,
|
||||
@@ -618,6 +638,8 @@ func (a *Agent) session(ctx context.Context, server string) error {
|
||||
go a.watchExits(ctx, out, unready(reg.RunningShards))
|
||||
go a.forwardKV(ctx, out)
|
||||
go a.reportCompute(ctx, out)
|
||||
a.setSlurmOut(out)
|
||||
defer a.setSlurmOut(nil)
|
||||
|
||||
for {
|
||||
msg, err := stream.Receive()
|
||||
@@ -884,7 +906,23 @@ func (a *Agent) removeWeights(paths []string) {
|
||||
a.log.Warn("refused to remove a file llmgw did not fetch", "path", p)
|
||||
continue
|
||||
}
|
||||
if err := os.Remove(abs); err != nil && !os.IsNotExist(err) {
|
||||
// A set — every file of one version, fetched into a directory of its
|
||||
// own — goes whole: os.Remove refused it as not empty, and nothing
|
||||
// was freed of what a model of several files left. Lstat, so a link
|
||||
// is taken away and never what it points at.
|
||||
info, err := os.Lstat(abs)
|
||||
switch {
|
||||
case os.IsNotExist(err):
|
||||
continue
|
||||
case err != nil:
|
||||
a.log.Warn("could not look at cached weights", "path", p, "err", err)
|
||||
continue
|
||||
case info.IsDir():
|
||||
err = os.RemoveAll(abs)
|
||||
default:
|
||||
err = os.Remove(abs)
|
||||
}
|
||||
if err != nil {
|
||||
a.log.Warn("could not remove cached weights", "path", p, "err", err)
|
||||
continue
|
||||
}
|
||||
@@ -925,6 +963,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
|
||||
go a.applySettings(m.AgentSettings)
|
||||
case *llmgwv1.ControlMessage_TrainStart:
|
||||
go a.trainJob(ctx, m.TrainStart, out)
|
||||
case *llmgwv1.ControlMessage_Slurm:
|
||||
a.slurmMessage(m.Slurm)
|
||||
case *llmgwv1.ControlMessage_TrainCancel:
|
||||
a.trainer.Cancel(m.TrainCancel.GetJobId())
|
||||
case *llmgwv1.ControlMessage_RestartAgent:
|
||||
|
||||
@@ -925,3 +925,23 @@ func TestABandwidthTestIsRunOnTheAdapterAskedFor(t *testing.T) {
|
||||
t.Errorf("reply %+v, ran %+v", r, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANewStreamIsToldTheFilesAgain(t *testing.T) {
|
||||
// Registration carries what the startup scan found; a file deleted since
|
||||
// came back on a control plane that restarted, because the next
|
||||
// heartbeat saw the list unchanged since it was last sent.
|
||||
dir := t.TempDir()
|
||||
h := start(t, func(c *Config) { c.ModelDirs = []string{dir} })
|
||||
if err := os.WriteFile(filepath.Join(dir, "0123456789abcdef-m.gguf"), []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.ag.heartbeat()
|
||||
if hb := h.ag.heartbeat().GetHeartbeat(); hb.GetWeightsChanged() {
|
||||
t.Fatal("an unchanged list should not be sent again on the same stream")
|
||||
}
|
||||
h.ag.forgetSent()
|
||||
hb := h.ag.heartbeat().GetHeartbeat()
|
||||
if !hb.GetWeightsChanged() || len(hb.GetWeights()) != 1 {
|
||||
t.Errorf("a new stream should be told the files: changed=%v %v", hb.GetWeightsChanged(), hb.GetWeights())
|
||||
}
|
||||
}
|
||||
@@ -10,8 +10,16 @@ import (
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// slurmPorts are Slurm's on a node that runs any of it: slurmctld's and
|
||||
// slurmd's, and srun's range for the steps it starts.
|
||||
var slurmPorts = []firewall.Port{
|
||||
{From: slurm.CtldPort, To: slurm.SlurmdPort, What: "slurm: slurmctld and slurmd"},
|
||||
{From: slurm.SrunPortsFrom, To: slurm.SrunPortsTo, What: "slurm: srun, for the steps it starts"},
|
||||
}
|
||||
|
||||
// applyFirewall keeps llmgw's firewall on this node as the cluster's
|
||||
// settings say: the engines' ports and the node exporter's for the networks
|
||||
// the node reaches its control planes on, and those named, or no table at
|
||||
@@ -19,6 +27,7 @@ import (
|
||||
func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
|
||||
a.fwMu.Lock()
|
||||
defer a.fwMu.Unlock()
|
||||
a.fwLast = m
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
defer cancel()
|
||||
r := a.fwRunner
|
||||
@@ -55,6 +64,11 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
|
||||
if from > 0 && to >= from {
|
||||
ports = append(ports, firewall.Engines(from, to))
|
||||
}
|
||||
// Once there, they stay: the ruleset keeps every port it held, so a
|
||||
// node that leaves Slurm keeps them closed to everyone but the cluster.
|
||||
if a.slurm != nil && a.slurm.HasPart() {
|
||||
ports = append(ports, slurmPorts...)
|
||||
}
|
||||
changed, err := firewall.Install(ctx, r, firewall.Spec{Ports: ports, Allow: nets})
|
||||
switch {
|
||||
case err != nil:
|
||||
@@ -64,6 +78,22 @@ func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
|
||||
}
|
||||
}
|
||||
|
||||
// slurmMessage is the leader's word on this node's part in Slurm. A node
|
||||
// that gets one has the firewall applied again, with Slurm's ports.
|
||||
func (a *Agent) slurmMessage(m *llmgwv1.Slurm) {
|
||||
had := a.slurm.HasPart()
|
||||
a.slurm.Apply(m)
|
||||
if had || !a.slurm.HasPart() {
|
||||
return
|
||||
}
|
||||
a.fwMu.Lock()
|
||||
last := a.fwLast
|
||||
a.fwMu.Unlock()
|
||||
if last != nil && last.GetEnabled() {
|
||||
go a.applyFirewall(last)
|
||||
}
|
||||
}
|
||||
|
||||
// controlPlaneIPs are the addresses of the control planes an agent dials.
|
||||
func controlPlaneIPs(ctx context.Context, servers []string) []string {
|
||||
var out []string
|
||||
|
||||
@@ -6,8 +6,10 @@ import (
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/slurmnode"
|
||||
)
|
||||
|
||||
// fakeShell answers the firewall package's scripts and records what was
|
||||
@@ -55,3 +57,28 @@ func TestANodeKeepsOrDropsItsFirewallAsTheClusterSays(t *testing.T) {
|
||||
t.Error("turned off, the table stayed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeGivenAPartInSlurmLetsSlurmInFromTheClusterOnly(t *testing.T) {
|
||||
sh := &fakeShell{}
|
||||
a := &Agent{log: slog.New(slog.DiscardHandler), fwRunner: sh, serverList: []string{"https://10.0.0.140:7443"},
|
||||
slurm: slurmnode.New(nil, nil, "", slog.New(slog.DiscardHandler))}
|
||||
a.applyFirewall(&llmgwv1.Firewall{Enabled: true, EnginePortsFrom: 20000, EnginePortsTo: 20999})
|
||||
if strings.Contains(sh.installed, "6817") {
|
||||
t.Fatalf("Slurm's ports before the node has a part in it:\n%s", sh.installed)
|
||||
}
|
||||
|
||||
a.slurmMessage(&llmgwv1.Slurm{Name: "node05", Node: true, ConfServer: "10.0.0.148:6817"})
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
a.fwMu.Lock()
|
||||
got := sh.installed
|
||||
a.fwMu.Unlock()
|
||||
if strings.Contains(got, "6817-6818") && strings.Contains(got, "60001-63000") {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("given a part in Slurm, the rules still lack its ports:\n%s", got)
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
@@ -164,6 +164,13 @@ func (a *Agent) provisionLoop(ctx context.Context) {
|
||||
|
||||
// provisionStatus records a change in one engine's installation.
|
||||
func (a *Agent) provisionStatus(st provision.Status) {
|
||||
if st == (provision.Status{}) {
|
||||
// A package install that was no engine's has ended.
|
||||
a.mu.Lock()
|
||||
delete(a.provStatus, "")
|
||||
a.mu.Unlock()
|
||||
return
|
||||
}
|
||||
a.mu.Lock()
|
||||
prev, had := a.provStatus[st.Engine]
|
||||
a.provStatus[st.Engine] = st
|
||||
|
||||
@@ -497,7 +497,13 @@ func (in *Installer) packages(ctx context.Context, byFamily map[string][]string,
|
||||
// byFamily is keyed by discovery.Host.Family: "deb" or "rpm".
|
||||
func (in *Installer) InstallPackages(ctx context.Context, byFamily map[string][]string) error {
|
||||
var st Status
|
||||
return in.packages(ctx, byFamily, &st)
|
||||
err := in.packages(ctx, byFamily, &st)
|
||||
if st.State == Installing {
|
||||
// Its progress was no engine's: an empty status takes it off the
|
||||
// node's page again, done or failed (the caller says which).
|
||||
in.report(Status{})
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// lockWait bounds how long a package command waits for another to finish.
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
package provision
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
// A package install that is no engine's says it is done, so its progress
|
||||
// does not stay on the node's page (found on the PoC installing Slurm).
|
||||
func TestInstallPackagesClearsItsProgress(t *testing.T) {
|
||||
var got []Status
|
||||
in := &Installer{Host: discovery.Host{OSID: "ubuntu"}, OnStatus: func(s Status) { got = append(got, s) },
|
||||
Run: func(_ context.Context, _ []string, name string, _ ...string) ([]byte, error) {
|
||||
if name == "sh" { // the query: not installed
|
||||
return nil, context.Canceled
|
||||
}
|
||||
return nil, nil
|
||||
}}
|
||||
if err := in.InstallPackages(context.Background(), map[string][]string{"deb": {"munge"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) < 2 || got[0].State != Installing || got[len(got)-1] != (Status{}) {
|
||||
t.Fatalf("statuses %+v", got)
|
||||
}
|
||||
|
||||
got = nil
|
||||
in.Run = func(context.Context, []string, string, ...string) ([]byte, error) { return nil, nil }
|
||||
if err := in.InstallPackages(context.Background(), map[string][]string{"deb": {"munge"}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("installed already, yet said %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -94,3 +94,35 @@ func TestWithNoCacheConfiguredNothingIsDeleted(t *testing.T) {
|
||||
t.Errorf("an agent with no cache directory deleted something: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A model of several files is fetched into a directory of its own, and an
|
||||
// unused one goes whole: it was refused as not empty, and nothing was freed.
|
||||
func TestAnUnusedSetIsRemovedWhole(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
a := agentWithCache(t, dir)
|
||||
set := weights.CachePathFor(dir, "https://mirror/coder/v1")
|
||||
if err := os.MkdirAll(set, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
write(t, filepath.Join(set, "model-00001-of-00002.gguf"))
|
||||
write(t, filepath.Join(set, ".complete"))
|
||||
// A link in the cache is taken away, not what it points at.
|
||||
target := t.TempDir()
|
||||
write(t, filepath.Join(target, "keep.gguf"))
|
||||
link := weights.CachePathFor(dir, "https://mirror/linked/v1")
|
||||
if err := os.Symlink(target, link); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
a.removeWeights([]string{set, link})
|
||||
|
||||
if _, err := os.Stat(set); !os.IsNotExist(err) {
|
||||
t.Errorf("the set is still there: %v", err)
|
||||
}
|
||||
if _, err := os.Lstat(link); !os.IsNotExist(err) {
|
||||
t.Errorf("the link is still there: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(target, "keep.gguf")); err != nil {
|
||||
t.Errorf("what the link pointed at was deleted: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
)
|
||||
|
||||
// FeatureSlurm is the agent keeping its node's part in Slurm (BACKLOG P).
|
||||
const FeatureSlurm = "slurm"
|
||||
|
||||
// slurmMarker says llmgw started Slurm's daemons on this machine.
|
||||
var slurmMarker = "/var/lib/llmgw-agent/slurm"
|
||||
|
||||
// setSlurmOut is where Slurm's status goes now: this session's queue, or
|
||||
// nowhere between sessions.
|
||||
func (a *Agent) setSlurmOut(out chan<- *llmgwv1.AgentMessage) {
|
||||
a.mu.Lock()
|
||||
a.slurmOut = out
|
||||
a.mu.Unlock()
|
||||
if out != nil {
|
||||
a.slurm.Resend()
|
||||
}
|
||||
}
|
||||
|
||||
// sendSlurm queues a status without waiting: a full queue or no session
|
||||
// drops it, and the next session is sent it again.
|
||||
func (a *Agent) sendSlurm(st *llmgwv1.SlurmStatus) {
|
||||
a.mu.Lock()
|
||||
out := a.slurmOut
|
||||
a.mu.Unlock()
|
||||
if out == nil {
|
||||
return
|
||||
}
|
||||
select {
|
||||
case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_SlurmStatus{SlurmStatus: st}}:
|
||||
default:
|
||||
a.log.Warn("Slurm's status not sent: the queue to the control plane is full")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,612 @@
|
||||
// Package slurmnode keeps this machine's part in the Slurm cluster run beside
|
||||
// llmgw (BACKLOG P), as the leader says: slurmctld on the controller, slurmd
|
||||
// on a node, munge on both, installed from the machine's own package sources.
|
||||
//
|
||||
// What the leader sends is wanted state, kept by Keeper's loop: sent again,
|
||||
// or checked again a minute later, it changes nothing that is already so.
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"log/slog"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// Host is what Keeper does to the machine; Local in the agent, a fake in
|
||||
// tests.
|
||||
type Host interface {
|
||||
Run(ctx context.Context, env []string, name string, args ...string) ([]byte, error)
|
||||
LookPath(name string) (string, error)
|
||||
ReadFile(path string) ([]byte, error)
|
||||
// WriteFile writes b at path with mode, owned by owner ("" for root),
|
||||
// making its directory.
|
||||
WriteFile(path string, b []byte, mode fs.FileMode, owner string) error
|
||||
// MkdirAll makes path, owned by owner.
|
||||
MkdirAll(path string, mode fs.FileMode, owner string) error
|
||||
Remove(path string) error
|
||||
}
|
||||
|
||||
// Installer installs packages by family, "deb" and "rpm".
|
||||
type Installer interface {
|
||||
InstallPackages(ctx context.Context, byFamily map[string][]string) error
|
||||
}
|
||||
|
||||
// Reasons llmgw drains a node with; one drained for another reason, by a
|
||||
// Slurm administrator, is never resumed by llmgw.
|
||||
const (
|
||||
ReasonServing = "llmgw: serving models"
|
||||
reasonPrefix = "llmgw"
|
||||
)
|
||||
|
||||
// DropIn is the unit drop-in that starts slurmd configless.
|
||||
const DropIn = "/etc/systemd/system/slurmd.service.d/llmgw.conf"
|
||||
|
||||
// Keeper keeps the machine as the last Slurm message says.
|
||||
type Keeper struct {
|
||||
host Host
|
||||
inst Installer
|
||||
log *slog.Logger
|
||||
marker string
|
||||
|
||||
mu sync.Mutex
|
||||
want *llmgwv1.Slurm
|
||||
kick chan struct{}
|
||||
// resend has the next status sent even if unchanged: a new stream's
|
||||
// control plane may never have had it.
|
||||
resend bool
|
||||
// installed is true once the packages were found or put there, so a
|
||||
// minute's check does not look again.
|
||||
installed map[string]bool
|
||||
files string
|
||||
}
|
||||
|
||||
// New is a Keeper; marker is a file it keeps while it runs Slurm here, so
|
||||
// an agent started again knows to stop what it started when told it has no
|
||||
// part any more.
|
||||
func New(host Host, inst Installer, marker string, log *slog.Logger) *Keeper {
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
return &Keeper{host: host, inst: inst, log: log, marker: marker,
|
||||
kick: make(chan struct{}, 1), installed: map[string]bool{}}
|
||||
}
|
||||
|
||||
// Apply is a new message from the leader: kept from now on.
|
||||
func (k *Keeper) Apply(m *llmgwv1.Slurm) {
|
||||
k.mu.Lock()
|
||||
k.want = proto.CloneOf(m)
|
||||
k.mu.Unlock()
|
||||
select {
|
||||
case k.kick <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
// HasPart is whether the last message gives this machine a part in Slurm.
|
||||
func (k *Keeper) HasPart() bool {
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
return k.want.GetController() || k.want.GetNode()
|
||||
}
|
||||
|
||||
// Resend has the status sent again on the next pass, which it starts now.
|
||||
func (k *Keeper) Resend() {
|
||||
k.mu.Lock()
|
||||
k.resend = true
|
||||
k.mu.Unlock()
|
||||
select {
|
||||
case k.kick <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
// Every is how often the machine is checked again without a new message.
|
||||
var Every = time.Minute
|
||||
|
||||
// resendAfter is how often an unchanged status is sent anyway.
|
||||
const resendAfter = 5 * time.Minute
|
||||
|
||||
// Run keeps the machine until ctx ends, sending its status when it changes
|
||||
// and every few minutes while it has a part.
|
||||
func (k *Keeper) Run(ctx context.Context, send func(*llmgwv1.SlurmStatus)) {
|
||||
t := time.NewTicker(Every)
|
||||
defer t.Stop()
|
||||
var last *llmgwv1.SlurmStatus
|
||||
var sent time.Time
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-k.kick:
|
||||
case <-t.C:
|
||||
}
|
||||
k.mu.Lock()
|
||||
want, again := k.want, k.resend
|
||||
k.resend = false
|
||||
k.mu.Unlock()
|
||||
if want == nil {
|
||||
continue
|
||||
}
|
||||
st := k.Reconcile(ctx, want)
|
||||
if st == nil {
|
||||
last = nil
|
||||
continue
|
||||
}
|
||||
if again || !proto.Equal(st, last) || time.Since(sent) > resendAfter {
|
||||
send(st)
|
||||
last, sent = st, time.Now()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Reconcile makes the machine what want says once, and answers its status;
|
||||
// nil when it has no part in Slurm.
|
||||
func (k *Keeper) Reconcile(ctx context.Context, want *llmgwv1.Slurm) *llmgwv1.SlurmStatus {
|
||||
ctx, cancel := context.WithTimeout(ctx, 20*time.Minute)
|
||||
defer cancel()
|
||||
if !want.GetController() && !want.GetNode() {
|
||||
k.stop(ctx)
|
||||
return nil
|
||||
}
|
||||
st := &llmgwv1.SlurmStatus{}
|
||||
fail := func(what string, err error) *llmgwv1.SlurmStatus {
|
||||
st.Error = what + ": " + err.Error()
|
||||
k.log.Warn("keeping Slurm on this node: "+what, "err", err)
|
||||
return st
|
||||
}
|
||||
if err := k.ensurePackages(ctx, want); err != nil {
|
||||
return fail("installing Slurm", err)
|
||||
}
|
||||
// From the daemon, not scontrol: a client command looks for its files
|
||||
// even to say its version, and a configless node has none before slurmd
|
||||
// has fetched them.
|
||||
daemon := "slurmd"
|
||||
if !want.GetNode() {
|
||||
daemon = "slurmctld"
|
||||
}
|
||||
out, err := k.host.Run(ctx, nil, daemon, "-V")
|
||||
if err != nil {
|
||||
return fail("reading Slurm's version", fmt.Errorf("%v: %s", err, lastLine(out)))
|
||||
}
|
||||
v, err := slurm.ParseVersion(string(out))
|
||||
if err != nil {
|
||||
return fail("reading Slurm's version", err)
|
||||
}
|
||||
st.Version = strings.TrimSpace(string(out))
|
||||
if i := strings.LastIndexByte(st.Version, ' '); i >= 0 {
|
||||
st.Version = st.Version[i+1:]
|
||||
}
|
||||
if slurm.TooOld(v) {
|
||||
return fail("Slurm", fmt.Errorf("%d.%02d is older than %d.%02d, the first to run jobs in cgroup v2; "+
|
||||
"a newer release of the distribution, or Slurm's own packages, has one", v[0], v[1],
|
||||
slurm.MinVersion[0], slurm.MinVersion[1]))
|
||||
}
|
||||
if err := k.host.WriteFile(k.marker, []byte("llmgw runs Slurm here (BACKLOG P)\n"), 0o644, ""); err != nil {
|
||||
return fail("keeping its marker", err)
|
||||
}
|
||||
if err := k.munge(ctx, want.GetMungeKey()); err != nil {
|
||||
return fail("munge", err)
|
||||
}
|
||||
if want.GetController() {
|
||||
sum, err := k.controller(ctx, want.GetFiles())
|
||||
st.FilesSha256 = sum
|
||||
if err != nil {
|
||||
return fail("slurmctld", err)
|
||||
}
|
||||
st.SlurmctldRunning = k.active(ctx, "slurmctld")
|
||||
}
|
||||
if want.GetNode() {
|
||||
if err := k.node(ctx, want, st); err != nil {
|
||||
return fail("slurmd", err)
|
||||
}
|
||||
}
|
||||
return st
|
||||
}
|
||||
|
||||
func (k *Keeper) ensurePackages(ctx context.Context, want *llmgwv1.Slurm) error {
|
||||
need := map[string][2]string{
|
||||
"munged": {"munge", "munge"},
|
||||
"scontrol": {"slurm-client", "slurm"},
|
||||
}
|
||||
if want.GetController() {
|
||||
need["slurmctld"] = [2]string{"slurmctld", "slurm-slurmctld"}
|
||||
}
|
||||
if want.GetNode() {
|
||||
need["slurmd"] = [2]string{"slurmd", "slurm-slurmd"}
|
||||
}
|
||||
pkgs := map[string][]string{}
|
||||
for bin, p := range need {
|
||||
if k.installed[bin] {
|
||||
continue
|
||||
}
|
||||
if _, err := k.host.LookPath(bin); err == nil {
|
||||
k.installed[bin] = true
|
||||
continue
|
||||
}
|
||||
pkgs["deb"] = append(pkgs["deb"], p[0])
|
||||
pkgs["rpm"] = append(pkgs["rpm"], p[1])
|
||||
}
|
||||
if len(pkgs) == 0 {
|
||||
return nil
|
||||
}
|
||||
slices.Sort(pkgs["deb"])
|
||||
slices.Sort(pkgs["rpm"])
|
||||
if k.inst == nil {
|
||||
return fmt.Errorf("%s missing, and this agent installs nothing", strings.Join(pkgs["deb"], ", "))
|
||||
}
|
||||
k.log.Info("installing Slurm, as the cluster's settings say", "packages", pkgs["deb"])
|
||||
if err := k.inst.InstallPackages(ctx, pkgs); err != nil {
|
||||
return err
|
||||
}
|
||||
for bin := range need {
|
||||
if _, err := k.host.LookPath(bin); err != nil {
|
||||
return fmt.Errorf("%s is still missing after installing %s", bin, strings.Join(pkgs["deb"], ", "))
|
||||
}
|
||||
k.installed[bin] = true
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// munge writes the cluster's key and keeps munged running on it.
|
||||
func (k *Keeper) munge(ctx context.Context, key []byte) error {
|
||||
if len(key) < 32 {
|
||||
return errors.New("the leader sent no key")
|
||||
}
|
||||
changed, err := k.write(slurm.MungeKeyPath, key, 0o400, "munge")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if changed {
|
||||
k.log.Info("munge key written; munge starts again on it")
|
||||
return k.systemctl(ctx, "enable", "--now", "munge", "&&restart")
|
||||
}
|
||||
if !k.active(ctx, "munge") {
|
||||
return k.systemctl(ctx, "enable", "--now", "munge")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// controller writes the files and keeps slurmctld running on them,
|
||||
// answering their digest.
|
||||
func (k *Keeper) controller(ctx context.Context, files map[string]string) (string, error) {
|
||||
if files["slurm.conf"] == "" {
|
||||
return "", errors.New("the leader sent no slurm.conf")
|
||||
}
|
||||
for _, d := range []string{slurm.StateDir, slurm.LogDir} {
|
||||
if err := k.host.MkdirAll(d, 0o755, "slurm"); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
names := make([]string, 0, len(files))
|
||||
for n := range files {
|
||||
names = append(names, n)
|
||||
}
|
||||
slices.Sort(names)
|
||||
h := sha256.New()
|
||||
changed := false
|
||||
for _, n := range names {
|
||||
if strings.ContainsAny(n, "/\\") || n == "" || n == "." || n == ".." {
|
||||
return "", fmt.Errorf("the leader sent a file named %q", n)
|
||||
}
|
||||
fmt.Fprintf(h, "%s\x00%s\x00", n, files[n])
|
||||
c, err := k.write(filepath.Join(slurm.ConfDir, n), []byte(files[n]), 0o644, "")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
changed = changed || c
|
||||
}
|
||||
sum := hex.EncodeToString(h.Sum(nil))
|
||||
if changed || sum != k.files {
|
||||
if err := k.systemctl(ctx, "enable", "slurmctld"); err != nil {
|
||||
return sum, err
|
||||
}
|
||||
// Nodes added or taken away need a restart before 23.11 and a
|
||||
// reconfigure after it; a restart does both, and running jobs live
|
||||
// through it.
|
||||
k.log.Info("slurmctld starts again on new files")
|
||||
if err := k.systemctl(ctx, "restart", "slurmctld"); err != nil {
|
||||
return sum, err
|
||||
}
|
||||
if err := k.waitPing(ctx); err != nil {
|
||||
return sum, err
|
||||
}
|
||||
// Hands the configless slurmd the new files.
|
||||
if out, err := k.host.Run(ctx, nil, "scontrol", "reconfigure"); err != nil {
|
||||
return sum, fmt.Errorf("scontrol reconfigure: %v: %s", err, lastLine(out))
|
||||
}
|
||||
k.files = sum
|
||||
return sum, nil
|
||||
}
|
||||
if !k.active(ctx, "slurmctld") {
|
||||
return sum, k.systemctl(ctx, "enable", "--now", "slurmctld")
|
||||
}
|
||||
return sum, nil
|
||||
}
|
||||
|
||||
func (k *Keeper) waitPing(ctx context.Context) error {
|
||||
var last []byte
|
||||
for range 30 {
|
||||
out, err := k.host.Run(ctx, nil, "scontrol", "ping")
|
||||
if err == nil && bytes.Contains(out, []byte("UP")) {
|
||||
return nil
|
||||
}
|
||||
last = out
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(pingWait):
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("slurmctld did not answer after starting: %s", lastLine(last))
|
||||
}
|
||||
|
||||
var pingWait = time.Second
|
||||
|
||||
// node keeps slurmd running configless and puts the node in or out of the
|
||||
// controller's hands as lent says.
|
||||
func (k *Keeper) node(ctx context.Context, want *llmgwv1.Slurm, st *llmgwv1.SlurmStatus) error {
|
||||
name, server := want.GetName(), want.GetConfServer()
|
||||
if name == "" || server == "" {
|
||||
return errors.New("the leader sent no name or controller")
|
||||
}
|
||||
if strings.ContainsAny(name+server, " \n\t\"'\\$`") {
|
||||
return fmt.Errorf("the leader sent a name or controller with spaces or quotes in it")
|
||||
}
|
||||
for _, d := range []string{slurm.SpoolDir, slurm.LogDir} {
|
||||
if err := k.host.MkdirAll(d, 0o755, "slurm"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
bin, err := k.host.LookPath("slurmd")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
unit := "# Written by llmgw (BACKLOG P): slurmd fetches its files from slurmctld.\n" +
|
||||
"[Unit]\nConditionPathExists=\nStartLimitIntervalSec=0\n" +
|
||||
// --systemd as the packaged unit has it (Type=notify, since 22.05):
|
||||
// without it systemd waits for a readiness that never comes.
|
||||
"[Service]\nExecStart=\nExecStart=" + bin + " --systemd --conf-server " + server + " -N " + name + "\n" +
|
||||
// A node new to the controller is refused until the leader has
|
||||
// written it into slurm.conf; tried again until it is.
|
||||
"Restart=on-failure\nRestartSec=15\n"
|
||||
changed, err := k.write(DropIn, []byte(unit), 0o644, "")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if out, err := k.host.Run(ctx, nil, "slurmd", "-C"); err == nil {
|
||||
st.Hardware = strings.TrimSpace(string(out))
|
||||
}
|
||||
if changed {
|
||||
if err := k.systemctl(ctx, "daemon-reload"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := k.systemctl(ctx, "enable", "slurmd"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := k.systemctl(ctx, "restart", "slurmd"); err != nil {
|
||||
return err
|
||||
}
|
||||
} else if !k.active(ctx, "slurmd") {
|
||||
if err := k.systemctl(ctx, "enable", "--now", "slurmd"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
st.SlurmdRunning = k.active(ctx, "slurmd")
|
||||
|
||||
st.Jobs = k.localJobs(ctx)
|
||||
env := k.clientEnv()
|
||||
out, err := k.host.Run(ctx, env, "scontrol", "show", "node", "-o", name)
|
||||
if err != nil {
|
||||
// Not in slurm.conf yet: the leader writes it once this node's
|
||||
// hardware is in.
|
||||
return nil
|
||||
}
|
||||
ns, err := slurm.ParseNodeState(string(out))
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
ours := ns.Reason == "" || strings.HasPrefix(ns.Reason, reasonPrefix)
|
||||
switch {
|
||||
case want.GetLent() && ns.Drained() && ours:
|
||||
k.log.Info("this node is lent to Slurm: resumed there")
|
||||
if out, err := k.host.Run(ctx, env, "scontrol", "update", "NodeName="+name, "State=RESUME"); err != nil {
|
||||
return fmt.Errorf("resuming the node: %v: %s", err, lastLine(out))
|
||||
}
|
||||
case !want.GetLent() && !strings.Contains(ns.State, "DRAIN"):
|
||||
k.log.Info("this node is llmgw's: drained in Slurm, its jobs left to finish")
|
||||
if out, err := k.host.Run(ctx, env, "scontrol", "update", "NodeName="+name, "State=DRAIN",
|
||||
"Reason="+ReasonServing); err != nil {
|
||||
return fmt.Errorf("draining the node: %v: %s", err, lastLine(out))
|
||||
}
|
||||
}
|
||||
if out, err := k.host.Run(ctx, env, "scontrol", "show", "node", "-o", name); err == nil {
|
||||
if ns2, err := slurm.ParseNodeState(string(out)); err == nil {
|
||||
ns = ns2
|
||||
}
|
||||
}
|
||||
st.State, st.Reason = ns.State, ns.Reason
|
||||
return nil
|
||||
}
|
||||
|
||||
// localJobs counts the job steps running here, from their slurmstepd, not
|
||||
// from the controller: a job outlives a controller that is down, and is
|
||||
// what decides whether llmgw may have the cards back. The one slurmstepd
|
||||
// that is no job's, "slurmstepd infinity" holding slurmd's cgroup v2 scope,
|
||||
// is not counted. -1 when it cannot tell.
|
||||
func (k *Keeper) localJobs(ctx context.Context) int32 {
|
||||
// pgrep exits 1 with no output when there is none.
|
||||
out, err := k.host.Run(ctx, nil, "pgrep", "-a", "-x", "slurmstepd")
|
||||
if err != nil && len(bytes.TrimSpace(out)) > 0 {
|
||||
return -1
|
||||
}
|
||||
n := int32(0)
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) < 2 || f[len(f)-1] == "infinity" {
|
||||
continue
|
||||
}
|
||||
n++
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// clientEnv points scontrol at slurmd's copy of the files on a node that is
|
||||
// not the controller, which has none of its own.
|
||||
func (k *Keeper) clientEnv() []string {
|
||||
if _, err := k.host.ReadFile(filepath.Join(slurm.ConfDir, "slurm.conf")); err == nil {
|
||||
return nil
|
||||
}
|
||||
return []string{"SLURM_CONF=" + filepath.Join(slurm.SpoolDir, "conf-cache", "slurm.conf")}
|
||||
}
|
||||
|
||||
// stop stops what llmgw started here, if it did.
|
||||
func (k *Keeper) stop(ctx context.Context) {
|
||||
if _, err := k.host.ReadFile(k.marker); err != nil {
|
||||
return
|
||||
}
|
||||
k.log.Info("this node has no part in Slurm any more: slurmd and slurmctld stopped; the packages stay")
|
||||
for _, u := range []string{"slurmd", "slurmctld"} {
|
||||
_ = k.systemctl(ctx, "disable", "--now", u)
|
||||
}
|
||||
_ = k.host.Remove(DropIn)
|
||||
_ = k.systemctl(ctx, "daemon-reload")
|
||||
_ = k.host.Remove(k.marker)
|
||||
k.files = ""
|
||||
}
|
||||
|
||||
// write writes a file when it differs, saying whether it did.
|
||||
func (k *Keeper) write(path string, b []byte, mode fs.FileMode, owner string) (bool, error) {
|
||||
if old, err := k.host.ReadFile(path); err == nil && bytes.Equal(old, b) {
|
||||
return false, nil
|
||||
}
|
||||
return true, k.host.WriteFile(path, b, mode, owner)
|
||||
}
|
||||
|
||||
// systemctl runs systemctl; a last argument "&&restart" restarts the unit
|
||||
// after.
|
||||
func (k *Keeper) systemctl(ctx context.Context, args ...string) error {
|
||||
restart := false
|
||||
if n := len(args); n > 0 && args[n-1] == "&&restart" {
|
||||
args, restart = args[:n-1], true
|
||||
}
|
||||
if out, err := k.host.Run(ctx, nil, "systemctl", args...); err != nil {
|
||||
return fmt.Errorf("systemctl %s: %v: %s", strings.Join(args, " "), err, lastLine(out))
|
||||
}
|
||||
if restart {
|
||||
u := args[len(args)-1]
|
||||
if out, err := k.host.Run(ctx, nil, "systemctl", "restart", u); err != nil {
|
||||
return fmt.Errorf("systemctl restart %s: %v: %s", u, err, lastLine(out))
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (k *Keeper) active(ctx context.Context, unit string) bool {
|
||||
out, err := k.host.Run(ctx, nil, "systemctl", "is-active", unit)
|
||||
return err == nil && strings.TrimSpace(string(out)) == "active"
|
||||
}
|
||||
|
||||
func lastLine(b []byte) string {
|
||||
s := strings.TrimSpace(string(b))
|
||||
if i := strings.LastIndexByte(s, '\n'); i >= 0 {
|
||||
s = s[i+1:]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// Local is the machine the agent runs on.
|
||||
type Local struct{}
|
||||
|
||||
// Run runs a command, with env added to the agent's.
|
||||
func (Local) Run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
|
||||
cmd := exec.CommandContext(ctx, name, args...)
|
||||
if len(env) > 0 {
|
||||
cmd.Env = append(os.Environ(), env...)
|
||||
}
|
||||
return cmd.CombinedOutput()
|
||||
}
|
||||
|
||||
// LookPath finds a command, in the sbin directories too: an agent started
|
||||
// by systemd has them, one started by hand may not.
|
||||
func (Local) LookPath(name string) (string, error) {
|
||||
if p, err := exec.LookPath(name); err == nil {
|
||||
return p, nil
|
||||
}
|
||||
for _, d := range []string{"/usr/sbin", "/sbin", "/usr/local/sbin"} {
|
||||
p := filepath.Join(d, name)
|
||||
if fi, err := os.Stat(p); err == nil && !fi.IsDir() {
|
||||
return p, nil
|
||||
}
|
||||
}
|
||||
return "", fmt.Errorf("%s is not installed", name)
|
||||
}
|
||||
|
||||
// ReadFile reads a file.
|
||||
func (Local) ReadFile(path string) ([]byte, error) { return os.ReadFile(path) } //nolint:gosec // our paths
|
||||
|
||||
// WriteFile writes a file whole, through a temporary one beside it.
|
||||
func (Local) WriteFile(path string, b []byte, mode fs.FileMode, owner string) error {
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { //nolint:gosec // system directories
|
||||
return err
|
||||
}
|
||||
tmp := path + ".llmgw-new"
|
||||
if err := os.WriteFile(tmp, b, mode); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.Chmod(tmp, mode); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := chown(tmp, owner); err != nil {
|
||||
_ = os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, path)
|
||||
}
|
||||
|
||||
// MkdirAll makes a directory owned by owner.
|
||||
func (Local) MkdirAll(path string, mode fs.FileMode, owner string) error {
|
||||
if err := os.MkdirAll(path, mode); err != nil {
|
||||
return err
|
||||
}
|
||||
return chown(path, owner)
|
||||
}
|
||||
|
||||
// Remove removes a file, and is content when there is none.
|
||||
func (Local) Remove(path string) error {
|
||||
if err := os.Remove(path); err != nil && !errors.Is(err, fs.ErrNotExist) {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func chown(path, owner string) error {
|
||||
if owner == "" {
|
||||
return nil
|
||||
}
|
||||
u, err := user.Lookup(owner)
|
||||
if err != nil {
|
||||
return fmt.Errorf("user %s: %w (installed with its package)", owner, err)
|
||||
}
|
||||
uid, _ := strconv.Atoi(u.Uid)
|
||||
gid, _ := strconv.Atoi(u.Gid)
|
||||
return os.Chown(path, uid, gid)
|
||||
}
|
||||
@@ -0,0 +1,308 @@
|
||||
package slurmnode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io/fs"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
type fakeHost struct {
|
||||
files map[string][]byte
|
||||
owners map[string]string
|
||||
bins map[string]bool
|
||||
active map[string]bool
|
||||
ran []string
|
||||
outputs map[string]string
|
||||
fail map[string]bool
|
||||
state string
|
||||
reason string
|
||||
}
|
||||
|
||||
func newFake() *fakeHost {
|
||||
return &fakeHost{files: map[string][]byte{}, owners: map[string]string{}, bins: map[string]bool{},
|
||||
active: map[string]bool{}, outputs: map[string]string{
|
||||
"slurmd -V": "slurm-wlm 23.11.4\n",
|
||||
"slurmctld -V": "slurm-wlm 23.11.4\n",
|
||||
"scontrol ping": "Slurmctld(primary) at node06 is UP\n",
|
||||
"slurmd -C": "NodeName=cuda01 CPUs=16 RealMemory=64000\nUpTime=1-00:00:00\n",
|
||||
}, fail: map[string]bool{}, state: "IDLE+DRAIN", reason: slurm.ReasonNew}
|
||||
}
|
||||
|
||||
func (f *fakeHost) Run(_ context.Context, env []string, name string, args ...string) ([]byte, error) {
|
||||
c := strings.Join(append([]string{name}, args...), " ")
|
||||
f.ran = append(f.ran, c)
|
||||
if f.fail[c] {
|
||||
return []byte("it failed"), errors.New("exit status 1")
|
||||
}
|
||||
switch {
|
||||
case name == "systemctl" && len(args) >= 1:
|
||||
u := args[len(args)-1]
|
||||
switch args[0] {
|
||||
case "is-active":
|
||||
if f.active[u] {
|
||||
return []byte("active\n"), nil
|
||||
}
|
||||
return []byte("inactive\n"), errors.New("exit status 3")
|
||||
case "restart", "start":
|
||||
f.active[u] = true
|
||||
case "enable":
|
||||
if slices.Contains(args, "--now") {
|
||||
f.active[u] = true
|
||||
}
|
||||
case "disable":
|
||||
f.active[u] = false
|
||||
}
|
||||
return nil, nil
|
||||
case strings.HasPrefix(c, "scontrol show node"):
|
||||
return []byte("NodeName=cuda01 Arch=x86_64 State=" + f.state + " Partitions=gpu Reason=" + f.reason + " [root@now]\n"), nil
|
||||
case strings.HasPrefix(c, "scontrol update"):
|
||||
if slices.Contains(args, "State=RESUME") {
|
||||
f.state, f.reason = "IDLE", ""
|
||||
} else {
|
||||
f.state, f.reason = "IDLE+DRAIN", ReasonServing
|
||||
}
|
||||
return nil, nil
|
||||
case strings.HasPrefix(c, "pgrep"):
|
||||
out := "1800358 /usr/sbin/slurmstepd infinity\n" + f.outputs["jobs"]
|
||||
return []byte(out), nil
|
||||
}
|
||||
return []byte(f.outputs[c]), nil
|
||||
}
|
||||
|
||||
func (f *fakeHost) LookPath(name string) (string, error) {
|
||||
if f.bins[name] {
|
||||
return "/usr/sbin/" + name, nil
|
||||
}
|
||||
return "", errors.New("not found")
|
||||
}
|
||||
|
||||
func (f *fakeHost) ReadFile(path string) ([]byte, error) {
|
||||
b, ok := f.files[path]
|
||||
if !ok {
|
||||
return nil, fs.ErrNotExist
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
|
||||
func (f *fakeHost) WriteFile(path string, b []byte, _ fs.FileMode, owner string) error {
|
||||
f.files[path] = append([]byte(nil), b...)
|
||||
f.owners[path] = owner
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeHost) MkdirAll(string, fs.FileMode, string) error { return nil }
|
||||
|
||||
func (f *fakeHost) Remove(path string) error { delete(f.files, path); return nil }
|
||||
|
||||
func (f *fakeHost) count(prefix string) int {
|
||||
n := 0
|
||||
for _, c := range f.ran {
|
||||
if strings.HasPrefix(c, prefix) {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
type fakeInst struct {
|
||||
host *fakeHost
|
||||
got map[string][]string
|
||||
}
|
||||
|
||||
func (i *fakeInst) InstallPackages(_ context.Context, by map[string][]string) error {
|
||||
i.got = by
|
||||
for _, b := range []string{"munged", "scontrol", "slurmd", "slurmctld"} {
|
||||
i.host.bins[b] = true
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
var key = []byte(strings.Repeat("k", 1024))
|
||||
|
||||
func nodeMsg(lent bool) *llmgwv1.Slurm {
|
||||
return &llmgwv1.Slurm{Node: true, Name: "cuda01", ConfServer: "10.0.0.146:6817", MungeKey: key, Lent: lent}
|
||||
}
|
||||
|
||||
func TestNodeInstallsAndStaysDrained(t *testing.T) {
|
||||
h := newFake()
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/var/lib/llmgw-agent/slurm", nil)
|
||||
st := k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if st.GetError() != "" {
|
||||
t.Fatal(st.GetError())
|
||||
}
|
||||
if !slices.Equal(inst.got["deb"], []string{"munge", "slurm-client", "slurmd"}) ||
|
||||
!slices.Equal(inst.got["rpm"], []string{"munge", "slurm", "slurm-slurmd"}) {
|
||||
t.Fatalf("installed %v", inst.got)
|
||||
}
|
||||
if string(h.files[slurm.MungeKeyPath]) != string(key) || h.owners[slurm.MungeKeyPath] != "munge" {
|
||||
t.Fatal("the munge key was not written for munge")
|
||||
}
|
||||
unit := string(h.files[DropIn])
|
||||
if !strings.Contains(unit, "ExecStart=/usr/sbin/slurmd --systemd --conf-server 10.0.0.146:6817 -N cuda01\n") ||
|
||||
!strings.Contains(unit, "ConditionPathExists=\n") {
|
||||
t.Fatalf("drop-in:\n%s", unit)
|
||||
}
|
||||
if !st.GetSlurmdRunning() || st.GetVersion() != "23.11.4" || !strings.HasPrefix(st.GetHardware(), "NodeName=cuda01 CPUs=16") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
// Not lent: left drained, with llmgw's first reason.
|
||||
if h.count("scontrol update") != 0 || st.GetState() != "IDLE+DRAIN" {
|
||||
t.Fatalf("a node not lent was changed: %v, %v", h.ran, st)
|
||||
}
|
||||
if _, ok := h.files["/var/lib/llmgw-agent/slurm"]; !ok {
|
||||
t.Fatal("no marker")
|
||||
}
|
||||
|
||||
// Again, nothing changed: no install, no restart.
|
||||
h.ran = nil
|
||||
inst.got = nil
|
||||
k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if inst.got != nil || h.count("systemctl restart") != 0 || h.count("systemctl enable") != 0 {
|
||||
t.Fatalf("a second pass did work: %v", h.ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLendAndReclaim(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
st := k.Reconcile(context.Background(), nodeMsg(true))
|
||||
if h.count("scontrol update NodeName=cuda01 State=RESUME") != 1 || st.GetState() != "IDLE" {
|
||||
t.Fatalf("a lent node was not resumed: %v %v", h.ran, st)
|
||||
}
|
||||
h.outputs["jobs"] = "2001 slurmstepd: [101.batch]\n2002 slurmstepd: [102.0]\n"
|
||||
st = k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if h.count("scontrol update NodeName=cuda01 State=DRAIN Reason="+ReasonServing) != 1 {
|
||||
t.Fatalf("a node taken back was not drained: %v", h.ran)
|
||||
}
|
||||
if st.GetJobs() != 2 || !strings.Contains(st.GetState(), "DRAIN") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
// Drained already: not drained again.
|
||||
h.ran = nil
|
||||
k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if h.count("scontrol update") != 0 {
|
||||
t.Fatalf("drained twice: %v", h.ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminsDrainIsLeft(t *testing.T) {
|
||||
h := newFake()
|
||||
h.reason = "bad DIMM, ticket 42"
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
k.Reconcile(context.Background(), nodeMsg(true))
|
||||
if h.count("scontrol update") != 0 {
|
||||
t.Fatalf("llmgw resumed a node a Slurm administrator drained: %v", h.ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClientEnvOnANodeWithoutFiles(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
if env := k.clientEnv(); len(env) != 1 || !strings.HasSuffix(env[0], "conf-cache/slurm.conf") {
|
||||
t.Fatalf("env %v", env)
|
||||
}
|
||||
h.files[filepath.Join(slurm.ConfDir, "slurm.conf")] = []byte("x")
|
||||
if env := k.clientEnv(); env != nil {
|
||||
t.Fatalf("env %v on the controller", env)
|
||||
}
|
||||
}
|
||||
|
||||
func TestController(t *testing.T) {
|
||||
h := newFake()
|
||||
inst := &fakeInst{host: h}
|
||||
k := New(h, inst, "/m", nil)
|
||||
files := map[string]string{"slurm.conf": "ClusterName=llmgw\n", "gres.conf": "#\n", "cgroup.conf": "#\n"}
|
||||
m := &llmgwv1.Slurm{Controller: true, MungeKey: key, Files: files}
|
||||
st := k.Reconcile(context.Background(), m)
|
||||
if st.GetError() != "" {
|
||||
t.Fatal(st.GetError())
|
||||
}
|
||||
if !slices.Contains(inst.got["deb"], "slurmctld") || slices.Contains(inst.got["deb"], "slurmd") {
|
||||
t.Fatalf("installed %v", inst.got)
|
||||
}
|
||||
if string(h.files["/etc/slurm/slurm.conf"]) != "ClusterName=llmgw\n" {
|
||||
t.Fatal("slurm.conf not written")
|
||||
}
|
||||
if h.count("systemctl restart slurmctld") != 1 || h.count("scontrol reconfigure") != 1 {
|
||||
t.Fatalf("slurmctld not started on its files: %v", h.ran)
|
||||
}
|
||||
if !st.GetSlurmctldRunning() || len(st.GetFilesSha256()) != 64 {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
sum := st.GetFilesSha256()
|
||||
|
||||
h.ran = nil
|
||||
k.Reconcile(context.Background(), m)
|
||||
if h.count("systemctl restart") != 0 {
|
||||
t.Fatalf("restarted on the same files: %v", h.ran)
|
||||
}
|
||||
|
||||
files["gres.conf"] = "NodeName=cuda01 Name=gpu File=/dev/nvidia0\n"
|
||||
h.ran = nil
|
||||
st = k.Reconcile(context.Background(), m)
|
||||
if h.count("systemctl restart slurmctld") != 1 || st.GetFilesSha256() == sum {
|
||||
t.Fatalf("a changed file did not restart slurmctld: %v", h.ran)
|
||||
}
|
||||
|
||||
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Controller: true, MungeKey: key,
|
||||
Files: map[string]string{"slurm.conf": "x", "../evil": "y"}}); !strings.Contains(st.GetError(), "named") {
|
||||
t.Fatalf("a path in a file name was taken: %v", st)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTooOld(t *testing.T) {
|
||||
h := newFake()
|
||||
h.outputs["slurmd -V"] = "slurm-wlm 21.08.5\n"
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
st := k.Reconcile(context.Background(), nodeMsg(false))
|
||||
if !strings.Contains(st.GetError(), "older than 22.05") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if _, ok := h.files[slurm.MungeKeyPath]; ok {
|
||||
t.Fatal("went on with a Slurm too old")
|
||||
}
|
||||
}
|
||||
|
||||
func TestStop(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
k.Reconcile(context.Background(), nodeMsg(false))
|
||||
h.ran = nil
|
||||
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{}); st != nil {
|
||||
t.Fatalf("status %v for no part", st)
|
||||
}
|
||||
if h.count("systemctl disable --now slurmd") != 1 {
|
||||
t.Fatalf("slurmd not stopped: %v", h.ran)
|
||||
}
|
||||
if _, ok := h.files[DropIn]; ok {
|
||||
t.Fatal("drop-in left")
|
||||
}
|
||||
// Never started here: nothing touched.
|
||||
h2 := newFake()
|
||||
k2 := New(h2, &fakeInst{host: h2}, "/m", nil)
|
||||
k2.Reconcile(context.Background(), &llmgwv1.Slurm{})
|
||||
if len(h2.ran) != 0 {
|
||||
t.Fatalf("a machine llmgw never ran Slurm on was touched: %v", h2.ran)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBadNames(t *testing.T) {
|
||||
h := newFake()
|
||||
k := New(h, &fakeInst{host: h}, "/m", nil)
|
||||
m := nodeMsg(false)
|
||||
m.Name = "cuda01 -c /etc/evil"
|
||||
if st := k.Reconcile(context.Background(), m); !strings.Contains(st.GetError(), "spaces") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
if st := k.Reconcile(context.Background(), &llmgwv1.Slurm{Node: true, Name: "a", ConfServer: "b"}); !strings.Contains(st.GetError(), "no key") {
|
||||
t.Fatalf("status %v", st)
|
||||
}
|
||||
}
|
||||
@@ -33,6 +33,17 @@ func CachePathFor(dir, source string) string {
|
||||
return filepath.Join(dir, hex.EncodeToString(sum[:8])+"-"+name)
|
||||
}
|
||||
|
||||
// cacheSource is the source a cache name is computed from: a set's address
|
||||
// without its trailing slash, as its directory was named when it was fetched.
|
||||
// Computed from the slash-ended address, every set in the catalog looked like
|
||||
// nobody's, and a sweep offered 330 GB of models still in use for deletion.
|
||||
func cacheSource(source string) string {
|
||||
if IsSet(source) {
|
||||
return strings.TrimSuffix(source, "/")
|
||||
}
|
||||
return source
|
||||
}
|
||||
|
||||
// Owned reports whether a path is one this package created.
|
||||
//
|
||||
// The dividing line of the whole feature. A file without the cache name was
|
||||
@@ -65,7 +76,7 @@ func Orphans(have []string, wanted []string) []string {
|
||||
// By base name rather than full path: the control plane does not know
|
||||
// each node's cache directory, and the name is what identifies the
|
||||
// file wherever the node keeps it.
|
||||
keep[filepath.Base(CachePathFor("", src))] = struct{}{}
|
||||
keep[filepath.Base(CachePathFor("", cacheSource(src)))] = struct{}{}
|
||||
}
|
||||
|
||||
var out []string
|
||||
|
||||
@@ -51,7 +51,7 @@ type manifest struct {
|
||||
// as a broken model rather than failing to load, so a partial set is left
|
||||
// unfinished under its temporary name and the next attempt starts again.
|
||||
func (f *Fetcher) ensureSet(ctx context.Context, source string, allowPickle bool) (string, error) {
|
||||
dir, err := f.cachePath(strings.TrimSuffix(source, "/"))
|
||||
dir, err := f.cachePath(cacheSource(source))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
@@ -74,6 +74,23 @@ func TestASetIsFetchedIntoOneDirectory(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestASetInTheCatalogIsNotAnOrphan(t *testing.T) {
|
||||
// The catalog names a set by its address with the trailing slash; the
|
||||
// fetch names the directory without it. A sweep that computed the name
|
||||
// from the catalog's spelling offered every set in use for deletion.
|
||||
f, srv, _ := mirrorOf(t, map[string]string{"model.safetensors": "aaaa", "config.json": "{}"})
|
||||
dir, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := Orphans([]string{dir}, []string{srv.URL + "/m/v/"}); len(got) != 0 {
|
||||
t.Errorf("a set the catalog still names was called an orphan: %v", got)
|
||||
}
|
||||
if got := Orphans([]string{dir}, nil); len(got) != 1 {
|
||||
t.Errorf("a set nothing names was not found: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAFinishedSetIsNotFetchedAgain(t *testing.T) {
|
||||
f, srv, hits := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"})
|
||||
|
||||
|
||||
@@ -247,6 +247,8 @@ var liveSettings = []string{
|
||||
"presets",
|
||||
// Handed to the reconciler as they change.
|
||||
"on_demand",
|
||||
// Read by the Slurm service each pass, kicked when they change.
|
||||
"slurm",
|
||||
// Read by server init, node add and backups, not by the server.
|
||||
"backup_recipients", "backup_copy_to",
|
||||
}
|
||||
|
||||
@@ -189,6 +189,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newMCPCmd(client, &asJSON),
|
||||
newKBCmd(client, &asJSON),
|
||||
newTrainCmd(client, &asJSON),
|
||||
newSlurmCmd(client, &asJSON),
|
||||
newConfigCmd(client, &asJSON),
|
||||
newAuditCmd(client, &asJSON),
|
||||
newNotifyCmd(client, &asJSON),
|
||||
|
||||
@@ -0,0 +1,247 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
"text/tabwriter"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
// newSlurmCmd is the Slurm cluster run beside llmgw (BACKLOG P): set up on
|
||||
// nodes llmgw already has, and nodes handed between the two.
|
||||
func newSlurmCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
cmd := &cobra.Command{
|
||||
Use: "slurm",
|
||||
Short: "A Slurm cluster beside llmgw, for training and other batch work",
|
||||
Long: "Slurm runs on llmgw's own nodes, installed and kept by their agents: slurmctld on one node,\n" +
|
||||
"slurmd on those named. A node is llmgw's, drained in Slurm, until it is lent; lent, its\n" +
|
||||
"replicas move off first, then Slurm may run jobs there. Taken back, Slurm drains it and\n" +
|
||||
"llmgw has it again when its last job ends. Jobs are submitted on the controller node\n" +
|
||||
"(sbatch, srun) by its own users, as on any Slurm cluster.",
|
||||
}
|
||||
cmd.AddCommand(newSlurmStatusCmd(client, asJSON), newSlurmSetupCmd(client), newSlurmOffCmd(client),
|
||||
newSlurmListCmd(client, "lend", "Let Slurm run jobs on nodes; llmgw's replicas move off first", true),
|
||||
newSlurmListCmd(client, "reclaim", "Take nodes back from Slurm; llmgw has them when their jobs end", false))
|
||||
return cmd
|
||||
}
|
||||
|
||||
func newSlurmStatusCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "status",
|
||||
Short: "The Slurm cluster, and where each of its nodes is",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var st slurmsvc.Status
|
||||
if err := client().get(cmd.Context(), "/api/v1/slurm", &st); err != nil {
|
||||
return err
|
||||
}
|
||||
w := cmd.OutOrStdout()
|
||||
if *asJSON {
|
||||
return json.NewEncoder(w).Encode(st)
|
||||
}
|
||||
if !st.Enabled {
|
||||
fmt.Fprintln(w, "Slurm is off. `llmgw ctl slurm setup --controller NODE --nodes NODE,...` turns it on.")
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintf(w, "cluster %s, partition %s, controller %s", st.Cluster, st.Partition, st.Controller)
|
||||
switch c := st.Ctld; {
|
||||
case c == nil:
|
||||
fmt.Fprintln(w, ": not heard from yet")
|
||||
case c.Error != "":
|
||||
fmt.Fprintf(w, ": %s\n", c.Error)
|
||||
case c.SlurmctldRunning:
|
||||
fmt.Fprintf(w, ": slurmctld %s running\n", c.Version)
|
||||
default:
|
||||
fmt.Fprintln(w, ": slurmctld not running")
|
||||
}
|
||||
tw := tabwriter.NewWriter(w, 0, 4, 2, ' ', 0)
|
||||
fmt.Fprintln(tw, "NODE\tPHASE\tSLURM\tJOBS\tSLURMD\tNOTE")
|
||||
for _, n := range st.Nodes {
|
||||
state, jobs, slurmd, note := "-", "-", "-", n.Describe()
|
||||
if r := n.Status; r != nil {
|
||||
state = or(r.State, "-")
|
||||
jobs = fmt.Sprint(r.Jobs)
|
||||
slurmd = map[bool]string{true: "running", false: "stopped"}[r.SlurmdRunning]
|
||||
if r.Error != "" {
|
||||
note = r.Error
|
||||
}
|
||||
}
|
||||
if !n.Connected {
|
||||
note = "agent not connected; " + note
|
||||
}
|
||||
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\t%s\n", n.Node, n.Phase, state, jobs, slurmd, note)
|
||||
}
|
||||
if err := tw.Flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(st.Waiting) > 0 {
|
||||
fmt.Fprintf(w, "not in slurm.conf until they report their hardware: %s\n", strings.Join(st.Waiting, ", "))
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newSlurmSetupCmd(client func() *ctlClient) *cobra.Command {
|
||||
var controller, cluster, partition, note string
|
||||
var nodes []string
|
||||
var all bool
|
||||
c := &cobra.Command{
|
||||
Use: "setup --controller NODE --nodes NODE,... | --all-nodes",
|
||||
Short: "Run Slurm on llmgw's nodes: slurmctld on one, slurmd on those named",
|
||||
Long: "Turns slurm.enabled on and names the nodes, or with --all-nodes every node. Their agents install munge and Slurm (22.05 or\n" +
|
||||
"newer) from the machines' own package sources. Every node starts drained in Slurm and stays\n" +
|
||||
"llmgw's until `llmgw ctl slurm lend` gives it to Slurm.",
|
||||
Example: " llmgw ctl slurm setup --controller node06 --nodes cuda01,cuda02\n" +
|
||||
" llmgw ctl slurm setup --controller node06 --all-nodes",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
if controller == "" || (len(nodes) == 0 && !all) {
|
||||
return fmt.Errorf("give --controller, and --nodes or --all-nodes")
|
||||
}
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
if err := doc.SetYAML([]string{"slurm", "enabled"}, "true"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := doc.SetYAML([]string{"slurm", "controller"}, yamlString(controller)); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := setList(doc, []string{"slurm", "nodes"}, nodes); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := doc.SetYAML([]string{"slurm", "all_nodes"}, fmt.Sprint(all)); err != nil {
|
||||
return err
|
||||
}
|
||||
// Lent nodes no longer named are not lent.
|
||||
lent := slices.DeleteFunc(docList(doc, []string{"slurm", "lent"}),
|
||||
func(n string) bool { return !all && !slices.Contains(nodes, n) })
|
||||
if err := setList(doc, []string{"slurm", "lent"}, lent); err != nil {
|
||||
return err
|
||||
}
|
||||
for k, v := range map[string]string{"cluster_name": cluster, "partition": partition} {
|
||||
if v != "" {
|
||||
if err := doc.SetYAML([]string{"slurm", k}, yamlString(v)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(&controller, "controller", "", "the node that runs slurmctld")
|
||||
c.Flags().StringSliceVar(&nodes, "nodes", nil, "the nodes that run slurmd")
|
||||
c.Flags().BoolVar(&all, "all-nodes", false, "every node runs slurmd, those that join later too")
|
||||
c.Flags().StringVar(&cluster, "cluster-name", "", "Slurm's ClusterName (llmgw)")
|
||||
c.Flags().StringVar(&partition, "partition", "", "the partition every node is in (gpu)")
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
|
||||
func newSlurmOffCmd(client func() *ctlClient) *cobra.Command {
|
||||
var note string
|
||||
var force bool
|
||||
c := &cobra.Command{
|
||||
Use: "off",
|
||||
Short: "Stop running Slurm; every node is llmgw's again",
|
||||
Long: "slurmd and slurmctld are stopped where llmgw started them; the packages stay. A node still\n" +
|
||||
"lent is refused unless --force: take it back first, so its jobs end before the daemons do.",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
if lent := docList(doc, []string{"slurm", "lent"}); len(lent) > 0 && !force {
|
||||
return fmt.Errorf("%s still lent: `llmgw ctl slurm reclaim` them first, or --force",
|
||||
strings.Join(lent, ", "))
|
||||
}
|
||||
setList(doc, []string{"slurm", "lent"}, nil) //nolint:errcheck // an empty list
|
||||
return doc.SetYAML([]string{"slurm", "enabled"}, "false")
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().BoolVar(&force, "force", false, "even with nodes still lent")
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
|
||||
// newSlurmListCmd adds nodes to slurm.lent, or takes them out.
|
||||
func newSlurmListCmd(client func() *ctlClient, use, short string, add bool) *cobra.Command {
|
||||
var note string
|
||||
c := &cobra.Command{
|
||||
Use: use + " NODE...",
|
||||
Short: short,
|
||||
Args: cobra.MinimumNArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
return changeSettings(cmd, client, note, func(doc *settings.Doc) error {
|
||||
return lendChange(doc, args, add)
|
||||
})
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(¬e, "note", "", "why, kept with the settings' version")
|
||||
return c
|
||||
}
|
||||
|
||||
// lendChange adds nodes to slurm.lent, or with add false takes them out.
|
||||
func lendChange(doc *settings.Doc, names []string, add bool) error {
|
||||
nodes := docList(doc, []string{"slurm", "nodes"})
|
||||
all := docBool(doc, []string{"slurm", "all_nodes"})
|
||||
lent := docList(doc, []string{"slurm", "lent"})
|
||||
for _, n := range names {
|
||||
switch {
|
||||
case add && !all && !slices.Contains(nodes, n):
|
||||
return fmt.Errorf("%s does not run slurmd: add it with `llmgw ctl slurm setup --nodes`", n)
|
||||
case add && !slices.Contains(lent, n):
|
||||
lent = append(lent, n)
|
||||
case !add:
|
||||
lent = slices.DeleteFunc(lent, func(l string) bool { return l == n })
|
||||
}
|
||||
}
|
||||
return setList(doc, []string{"slurm", "lent"}, lent)
|
||||
}
|
||||
|
||||
// docList reads a list of strings from the settings, empty when unset.
|
||||
func docList(doc *settings.Doc, path []string) []string {
|
||||
n := doc.Node(path)
|
||||
if n == nil {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
if err := n.Decode(&out); err != nil {
|
||||
return nil
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// docBool reads a boolean from the settings, false when unset.
|
||||
func docBool(doc *settings.Doc, path []string) bool {
|
||||
var b bool
|
||||
if n := doc.Node(path); n != nil {
|
||||
_ = n.Decode(&b)
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func setList(doc *settings.Doc, path []string, list []string) error {
|
||||
if len(list) == 0 {
|
||||
if doc.Node(path) != nil {
|
||||
doc.Unset(path)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return doc.SetYAML(path, yamlList(list))
|
||||
}
|
||||
|
||||
func yamlString(s string) string {
|
||||
b, _ := json.Marshal(s)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func yamlList(list []string) string {
|
||||
b, _ := json.Marshal(list)
|
||||
return string(b)
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
func TestSlurmDocLists(t *testing.T) {
|
||||
doc, err := settings.FromBytes("settings", []byte("slurm:\n enabled: true\n nodes: [cuda01, site2]\n"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := docList(doc, []string{"slurm", "nodes"}); !slices.Equal(got, []string{"cuda01", "site2"}) {
|
||||
t.Fatalf("nodes %v", got)
|
||||
}
|
||||
if got := docList(doc, []string{"slurm", "lent"}); got != nil {
|
||||
t.Fatalf("lent %v", got)
|
||||
}
|
||||
if err := setList(doc, []string{"slurm", "lent"}, []string{"cuda01"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := docList(doc, []string{"slurm", "lent"}); !slices.Equal(got, []string{"cuda01"}) {
|
||||
t.Fatalf("lent %v", got)
|
||||
}
|
||||
if err := setList(doc, []string{"slurm", "lent"}, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := doc.Bytes()
|
||||
if strings.Contains(string(b), "lent") {
|
||||
t.Fatalf("an empty list left:\n%s", b)
|
||||
}
|
||||
}
|
||||
+19
-15
@@ -4,6 +4,7 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
@@ -87,13 +88,10 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var results []struct {
|
||||
Node string `json:"node"`
|
||||
Files int `json:"files"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
if err := c.post(cmd.Context(), "/api/v1/weights/collect", body, &results); err != nil {
|
||||
var results []adminapi.CollectResult
|
||||
// The control plane waits for the nodes to list their files without the
|
||||
// deleted ones: half a minute at most.
|
||||
if err := c.patient(time.Minute).post(cmd.Context(), "/api/v1/weights/collect", body, &results); err != nil {
|
||||
return err
|
||||
}
|
||||
if asJSON {
|
||||
@@ -103,7 +101,7 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "Nothing to reclaim.")
|
||||
return nil
|
||||
}
|
||||
var freed int64
|
||||
var freed, left int64
|
||||
for _, r := range results {
|
||||
if r.Error != "" {
|
||||
// Reported rather than fatal: the other nodes' space is still worth
|
||||
@@ -111,14 +109,20 @@ func collectOrphans(cmd *cobra.Command, c *ctlClient, node string, asJSON bool)
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: %s\n", r.Node, r.Error)
|
||||
continue
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: asked to remove %d files, %s\n",
|
||||
r.Node, r.Files, fileSize(r.Bytes))
|
||||
freed += r.Bytes
|
||||
line := fmt.Sprintf("%s: removed %d files, %s", r.Node, r.Files, fileSize(r.Bytes))
|
||||
if r.Left > 0 {
|
||||
line += fmt.Sprintf("; %d still there, %s", r.Left, fileSize(r.LeftBytes))
|
||||
}
|
||||
fmt.Fprintln(cmd.OutOrStdout(), line)
|
||||
freed, left = freed+r.Bytes, left+r.LeftBytes
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "\n%s freed.\n", fileSize(freed))
|
||||
if left > 0 {
|
||||
// Each node decides: one that keeps a path it cannot prove it
|
||||
// fetched is the feature working. Its agent's log names each.
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s the nodes kept; each node's agent log says why "+
|
||||
"(journalctl -u llmgw-agent).\n", fileSize(left))
|
||||
}
|
||||
// "asked", not "removed": each node decides, and one that refuses a path it
|
||||
// cannot prove it fetched is the feature working rather than failing.
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "\n%s requested. The nodes verify each path before deleting it.\n",
|
||||
fileSize(freed))
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -49,6 +49,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
@@ -59,6 +60,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/secrets"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
@@ -261,6 +263,8 @@ func newServerCmd() *cobra.Command {
|
||||
// What the settings say of the firewall, for the mirror and the
|
||||
// store's members; nil without a control plane.
|
||||
var fwPub *firewallPublisher
|
||||
// The Slurm cluster beside llmgw, when there is a store (BACKLOG P).
|
||||
var slurmSvc *slurmsvc.Service
|
||||
updateMetrics := &modelserve.UpdateCollector{}
|
||||
// The store's cluster, when it is one; read by the metrics and
|
||||
// by `ctl status`.
|
||||
@@ -436,6 +440,21 @@ func newServerCmd() *cobra.Command {
|
||||
agentSvc.SetTrain(trainer)
|
||||
go trainer.Run(ctx)
|
||||
api.SetTrain(&adminapi.TrainAdmin{Store: st, Service: trainer})
|
||||
// Slurm on the nodes the settings name, its munge key
|
||||
// made from a cluster secret.
|
||||
slurmSvc = slurmsvc.New(func() config.SlurmSettings { return current.Load().Slurm },
|
||||
admin.Inventory(), agentSvc, func(ctx context.Context) ([]byte, error) {
|
||||
secret, err := st.ClusterSecret(ctx, "slurm-munge")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return slurm.MungeKey(secret)
|
||||
}, log)
|
||||
// What slurm.conf held, for the leader that comes next.
|
||||
slurmSvc.SetMemory(slurmsvc.StoreMemory(st))
|
||||
agentSvc.SetSlurm(slurmSvc)
|
||||
go slurmSvc.Run(ctx)
|
||||
api.SetSlurm(slurmSvc)
|
||||
}
|
||||
// The cluster's settings, each change checked against this
|
||||
// machine's own keys before it is kept.
|
||||
@@ -841,6 +860,9 @@ func newServerCmd() *cobra.Command {
|
||||
fwPub.Kick()
|
||||
}
|
||||
}
|
||||
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "slurm") }) && slurmSvc != nil {
|
||||
slurmSvc.Kick()
|
||||
}
|
||||
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "on_demand") }) && admin != nil {
|
||||
admin.Reconciler().SetEvictionTimes(next.OnDemand.EvictAfterIdle, next.OnDemand.MinLifetime)
|
||||
}
|
||||
|
||||
@@ -124,6 +124,9 @@ type serverInit struct {
|
||||
// media has the nodes install ComfyUI and the leader add a picture model
|
||||
// and a 3D model once a card fits one.
|
||||
media bool
|
||||
// slurm runs Slurm beside llmgw: slurmctld on this machine's node,
|
||||
// slurmd on every node, each llmgw's until it is lent.
|
||||
slurm bool
|
||||
// evictAfterIdle and minLifetime are on_demand's; zero leaves its own.
|
||||
evictAfterIdle, minLifetime time.Duration
|
||||
// standbys makes each machine of the replicated store a control plane
|
||||
@@ -258,6 +261,8 @@ func newServerInitCmd() *cobra.Command {
|
||||
f.DurationVar(&in.maxKeyTTL, "max-key-ttl", defaultMaxKeyTTL, "the longest an API key lives")
|
||||
f.BoolVar(&in.media, "media", false,
|
||||
"pictures and 3D: the nodes install ComfyUI, and a picture model and a 3D model are added once a card fits one")
|
||||
f.BoolVar(&in.slurm, "slurm", false,
|
||||
"Slurm for training and batch jobs: slurmctld on this machine, slurmd on every node, each llmgw's until 'llmgw ctl slurm lend'")
|
||||
f.DurationVar(&in.evictAfterIdle, "evict-after-idle", 0,
|
||||
"how long a model loaded on demand goes unasked-for before another may have its card (0: 2m)")
|
||||
f.DurationVar(&in.minLifetime, "min-lifetime", 0,
|
||||
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/preflight"
|
||||
)
|
||||
@@ -132,9 +133,18 @@ func (in *serverInit) checkChoices() error {
|
||||
return fmt.Errorf("max key ttl %s: an hour at least", in.maxKeyTTL)
|
||||
case in.evictAfterIdle < 0 || in.minLifetime < 0:
|
||||
return fmt.Errorf("on-demand times cannot be negative")
|
||||
case in.slurm && in.noNode:
|
||||
return fmt.Errorf("slurm runs slurmctld on this machine's node: it needs the node (no --no-node)")
|
||||
case in.vip != "" && !in.standbys:
|
||||
return fmt.Errorf("a floating address moves between control planes: it needs the standbys")
|
||||
}
|
||||
if in.slurm {
|
||||
// The controller is this machine's node, named by its hostname: a
|
||||
// name Slurm does not take would keep the server from starting.
|
||||
if err := (config.SlurmSettings{Enabled: true, Controller: localNodeID()}).Check(); err != nil {
|
||||
return fmt.Errorf("slurm: this machine's hostname: %w", err)
|
||||
}
|
||||
}
|
||||
if in.vip != "" {
|
||||
ip, _, err := net.ParseCIDR(in.vip)
|
||||
if err != nil || ip.To4() == nil {
|
||||
@@ -318,6 +328,9 @@ func (in *serverInit) tuneConfig(body string) string {
|
||||
"# one ('llmgw ctl model presets auto off' stops it).\n"+
|
||||
"engines:\n comfyui: "+provision.ComfyUIVersion()+"\npresets:\n auto: true\n")
|
||||
}
|
||||
if in.slurm {
|
||||
extra = append(extra, slurmSection(localNodeID()))
|
||||
}
|
||||
if in.evictAfterIdle > 0 || in.minLifetime > 0 {
|
||||
b := "# How models that load when asked for take turns on a card, chosen at\n# install.\non_demand:\n"
|
||||
if in.evictAfterIdle > 0 {
|
||||
@@ -334,6 +347,24 @@ func (in *serverInit) tuneConfig(body string) string {
|
||||
return body
|
||||
}
|
||||
|
||||
// slurmSection is server.yaml's slurm block as an install chooses it:
|
||||
// slurmctld on this machine's node, slurmd on every node.
|
||||
func slurmSection(controller string) string {
|
||||
return "# Slurm beside llmgw, chosen at install: slurmctld on this machine, slurmd on\n" +
|
||||
"# every node, those that join later too. Each node is llmgw's, drained in\n" +
|
||||
"# Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm; 'llmgw ctl slurm\n" +
|
||||
"# reclaim NODE' takes it back once its jobs end.\n" +
|
||||
"slurm:\n enabled: true\n controller: " + yamlString(controller) + "\n all_nodes: true\n"
|
||||
}
|
||||
|
||||
// localNodeID is the node id this machine's agent takes: its hostname.
|
||||
func localNodeID() string {
|
||||
if h, err := os.Hostname(); err == nil && h != "" {
|
||||
return h
|
||||
}
|
||||
return "localhost"
|
||||
}
|
||||
|
||||
// durationYAML is a duration as a person writes it: 2h, 90m, 1m30s.
|
||||
func durationYAML(d time.Duration) string {
|
||||
switch {
|
||||
@@ -467,6 +498,9 @@ func (in *serverInit) initFlags() []string {
|
||||
if in.media {
|
||||
add("--media")
|
||||
}
|
||||
if in.slurm {
|
||||
add("--slurm")
|
||||
}
|
||||
if in.evictAfterIdle > 0 {
|
||||
add("--evict-after-idle", durationYAML(in.evictAfterIdle))
|
||||
}
|
||||
|
||||
@@ -17,7 +17,7 @@ import (
|
||||
func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
|
||||
in := &serverInit{
|
||||
language: "tr", signupOpen: true, defaultRPM: 120, defaultTPM: 50000, maxKeyTTL: 30 * 24 * time.Hour,
|
||||
media: true, evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
|
||||
media: true, slurm: true, evictAfterIdle: time.Minute, minLifetime: 90 * time.Second,
|
||||
judge: judgeSpec{Fail: "open", Categories: []string{"Violent", "PII"}, Replicas: 3, Budget: 3 * time.Second},
|
||||
}
|
||||
j := in.judgeFor()
|
||||
@@ -41,6 +41,8 @@ func TestEveryChoiceIsInTheConfigurationWrittenAndItLoads(t *testing.T) {
|
||||
t.Errorf("limits %d %d %s", cfg.Auth.DefaultRPM, cfg.Auth.DefaultTPM, cfg.Auth.MaxKeyTTL)
|
||||
case !cfg.Presets.Auto || cfg.Engines["comfyui"] == "":
|
||||
t.Errorf("pictures and 3D: presets %v comfyui %q", cfg.Presets.Auto, cfg.Engines["comfyui"])
|
||||
case !cfg.Slurm.Enabled || !cfg.Slurm.AllNodes || cfg.Slurm.Controller != localNodeID():
|
||||
t.Errorf("slurm %+v", cfg.Slurm)
|
||||
case cfg.OnDemand.EvictAfterIdle != time.Minute || cfg.OnDemand.MinLifetime != 90*time.Second:
|
||||
t.Errorf("on demand %+v", cfg.OnDemand)
|
||||
}
|
||||
@@ -95,14 +97,14 @@ func TestAStarterIsChosenByNameOrByItsFile(t *testing.T) {
|
||||
|
||||
func TestTheChoicesAreTheirFlagsAgain(t *testing.T) {
|
||||
in := &serverInit{admin: "ops@example.com", hosts: []string{"llm.example.com"}, postgres: true,
|
||||
mirrorPath: "/data/models", noNode: true, language: "tr", media: true,
|
||||
mirrorPath: "/data/models", language: "tr", media: true, slurm: true,
|
||||
webhooks: []string{"https://hooks.slack.com/services/T/B/x, json:https://a.example.com/in"},
|
||||
judge: judgeSpec{Categories: []string{"Violent", "Jailbreak"}}, maxKeyTTL: 7 * 24 * time.Hour}
|
||||
in.starter, _ = starterFrom("qwen3-4b")
|
||||
got := strings.Join(in.initFlags(), " ")
|
||||
want := "--admin ops@example.com --host llm.example.com --language tr --max-key-ttl 168h --postgres " +
|
||||
"--mirror-dir /data/models --no-node --starter qwen3-4b --judge-categories Violent,Jailbreak " +
|
||||
"--webhook https://hooks.slack.com/services/T/B/x --webhook json:https://a.example.com/in --media"
|
||||
"--mirror-dir /data/models --starter qwen3-4b --judge-categories Violent,Jailbreak " +
|
||||
"--webhook https://hooks.slack.com/services/T/B/x --webhook json:https://a.example.com/in --media --slurm"
|
||||
if got != want {
|
||||
t.Errorf("flags\n got %s\nwant %s", got, want)
|
||||
}
|
||||
@@ -124,6 +126,7 @@ func TestChoicesThatCannotBeInstalledAreRefused(t *testing.T) {
|
||||
{maxKeyTTL: time.Minute},
|
||||
{vip: "10.0.0.160/24"},
|
||||
{standbys: true, vip: "10.0.0.160"},
|
||||
{slurm: true, noNode: true},
|
||||
} {
|
||||
if err := in.checkChoices(); err == nil {
|
||||
t.Errorf("taken: %+v", in)
|
||||
|
||||
@@ -643,17 +643,30 @@ func (m consoleModel) nodeMenu(n adminapi.NodeView) *menuDialog {
|
||||
menuItem{"Delete model files nothing uses", func(m *consoleModel) tea.Cmd {
|
||||
return m.call("Unused files on "+n.ID, http.MethodPost, "/api/v1/weights/collect", map[string]string{"node": n.ID},
|
||||
func(raw json.RawMessage) string {
|
||||
var rs []struct {
|
||||
Files int `json:"files"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
}
|
||||
// What the node no longer lists, and what it kept.
|
||||
var rs []adminapi.CollectResult
|
||||
_ = json.Unmarshal(raw, &rs)
|
||||
var files int
|
||||
var b int64
|
||||
var files, left int
|
||||
var b, lb int64
|
||||
var errs []string
|
||||
for _, r := range rs {
|
||||
files, b = files+r.Files, b+r.Bytes
|
||||
files, b, left, lb = files+r.Files, b+r.Bytes, left+r.Left, lb+r.LeftBytes
|
||||
if r.Error != "" {
|
||||
errs = append(errs, r.Error)
|
||||
}
|
||||
}
|
||||
return fmt.Sprintf("%d files, %.1f GB freed.", files, float64(b)/(1<<30))
|
||||
if len(rs) == 0 {
|
||||
return "Nothing on " + n.ID + " that no model uses."
|
||||
}
|
||||
say := fmt.Sprintf("%d files removed, %.1f GB freed.", files, float64(b)/(1<<30))
|
||||
if left > 0 {
|
||||
say += fmt.Sprintf(" %d files (%.1f GB) are still there: the node kept them, and its agent's log "+
|
||||
"says why (journalctl -u llmgw-agent there).", left, float64(lb)/(1<<30))
|
||||
}
|
||||
if len(errs) > 0 {
|
||||
say += " " + strings.Join(errs, "; ")
|
||||
}
|
||||
return say
|
||||
})
|
||||
}},
|
||||
menuItem{"Remove from the cluster…", func(m *consoleModel) tea.Cmd {
|
||||
|
||||
@@ -57,7 +57,7 @@ type installForm struct {
|
||||
standby *installPart
|
||||
vipF *setupField
|
||||
|
||||
control, store, mirror, node, starter, media, judge, firewall, alerts, limits, turns, bundle *installPart
|
||||
control, store, mirror, node, starter, media, slurm, judge, firewall, alerts, limits, turns, bundle *installPart
|
||||
|
||||
admin, hosts, language, signup *setupField
|
||||
storeKind *setupField
|
||||
@@ -282,6 +282,25 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
|
||||
summary: func() string { return "ComfyUI; a picture model and a 3D model once a card fits" },
|
||||
}
|
||||
|
||||
// --- Slurm ---
|
||||
f.slurm = &installPart{
|
||||
title: "Slurm", on: in.slurm,
|
||||
about: "A Slurm cluster beside llmgw, for training and other batch work: slurmctld on this machine, " +
|
||||
"slurmd on every node, those that join later too. The agents install munge and Slurm from the " +
|
||||
"machines' own package sources.\n\n" +
|
||||
"Each node stays llmgw's, drained in Slurm, until 'llmgw ctl slurm lend NODE' gives it to Slurm: " +
|
||||
"its models move off first. 'llmgw ctl slurm reclaim NODE' takes it back once its jobs end. Jobs " +
|
||||
"are submitted on this machine with sbatch and srun.",
|
||||
off: "no Slurm; 'llmgw ctl slurm setup' adds it later",
|
||||
summary: func() string { return "slurmctld here, slurmd on every node; none lent" },
|
||||
needs: func() string {
|
||||
if !f.node.on {
|
||||
return "slurmctld runs on this machine's node: turn the node on, or Slurm off"
|
||||
}
|
||||
return ""
|
||||
},
|
||||
}
|
||||
|
||||
// --- the guardrails' judge ---
|
||||
j := in.judgeFor()
|
||||
f.judgeFail = choiceField("When no judge answers in time", []string{"refuse", "serve flagged"},
|
||||
@@ -421,7 +440,7 @@ func newInstallForm(in *serverInit, host string, found []string, ha bool) *insta
|
||||
if ha {
|
||||
f.parts = append(f.parts, f.standby)
|
||||
}
|
||||
f.parts = append(f.parts, f.mirror, f.node, f.starter, f.media, f.judge, f.firewall,
|
||||
f.parts = append(f.parts, f.mirror, f.node, f.starter, f.media, f.slurm, f.judge, f.firewall,
|
||||
f.alerts, f.limits, f.turns, f.bundle)
|
||||
return f
|
||||
}
|
||||
@@ -653,6 +672,7 @@ func (f *installForm) apply(in *serverInit) error {
|
||||
in.noModel = !f.starter.on
|
||||
in.starter, in.starterCtx = starter, ctx
|
||||
in.media = f.media.on
|
||||
in.slurm = f.slurm.on
|
||||
in.noGuardModel = !f.judge.on
|
||||
in.judge = judge
|
||||
in.noFirewall = !f.firewall.on
|
||||
|
||||
@@ -24,6 +24,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/docgen"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
@@ -50,6 +51,7 @@ const (
|
||||
cFederation
|
||||
cMirror
|
||||
cJobs
|
||||
cSlurm
|
||||
cLogs
|
||||
cAccounts
|
||||
cKeys
|
||||
@@ -71,7 +73,7 @@ const (
|
||||
cCount
|
||||
)
|
||||
|
||||
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "MCP servers", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Sign-in", "Audit", "Manual",
|
||||
var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Knowledge", "MCP servers", "Federation", "Mirror", "Jobs", "Slurm", "Logs", "Accounts", "API keys", "Organizations", "Sign-in", "Audit", "Manual",
|
||||
"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Serve the guardrails' judge", "Upgrade the cluster",
|
||||
"Restart all replicas", "Back up now", "Wipe this machine", "Leave"}
|
||||
|
||||
@@ -92,6 +94,8 @@ type consoleData struct {
|
||||
// federation is the other llmgw clusters whose models are offered
|
||||
// here; empty on a control plane without a store.
|
||||
federation adminapi.FederationView
|
||||
// slurm is the Slurm cluster beside llmgw; off when it has none.
|
||||
slurm slurmsvc.Status
|
||||
// presets is the models that come with llmgw as they stand here; nil
|
||||
// from a control plane that cannot say.
|
||||
presets *adminapi.PresetsView
|
||||
@@ -284,6 +288,7 @@ func (m consoleModel) fetch() tea.Cmd {
|
||||
_ = c.get(ctx, "/api/v1/jobs", &d.jobs)
|
||||
_ = c.get(ctx, "/api/v1/audit?since=7d&limit=300", &d.audit)
|
||||
_ = c.get(ctx, "/api/v1/federation", &d.federation)
|
||||
_ = c.get(ctx, "/api/v1/slurm", &d.slurm)
|
||||
var presets adminapi.PresetsView
|
||||
if c.get(ctx, "/api/v1/presets", &presets) == nil {
|
||||
d.presets = &presets
|
||||
@@ -588,6 +593,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
|
||||
case cFederation:
|
||||
m.dialog = "error"
|
||||
m.dialogTitle, m.dialogText = peerDialog(m.data.federation.Peers[m.row])
|
||||
case cSlurm:
|
||||
m.menuD = m.slurmMenu(m.data.slurm.Nodes[m.row])
|
||||
case cServe:
|
||||
return m, m.fetchRepo(m.hf.models[m.row].ID)
|
||||
case cAccounts:
|
||||
@@ -1110,6 +1117,8 @@ func (m consoleModel) rowCount() int {
|
||||
return len(m.data.audit)
|
||||
case cFederation:
|
||||
return len(m.data.federation.Peers)
|
||||
case cSlurm:
|
||||
return len(m.data.slurm.Nodes)
|
||||
case cServe:
|
||||
return len(m.hf.models)
|
||||
case cAccounts:
|
||||
@@ -1369,6 +1378,8 @@ func (m consoleModel) panel() (title, body, keys string) {
|
||||
return m.auditPanel(pw)
|
||||
case cFederation:
|
||||
return m.federationPanel(pw)
|
||||
case cSlurm:
|
||||
return m.slurmPanel(pw)
|
||||
case cMonitor:
|
||||
return m.monitorPanel(pw)
|
||||
case cServe:
|
||||
|
||||
@@ -213,10 +213,17 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
|
||||
"before it is served.", modelplan.GB(p.KVBytesPerToken*1000))
|
||||
}
|
||||
note += "\nThe mirror fetches it and checks it, it joins the catalog, and a replica is started\nand asked to say hello."
|
||||
// The longest context the first way holds, up to what the model was
|
||||
// trained for: a choice of another way is checked again on submitting.
|
||||
ctxDefault := "4096"
|
||||
if a.Format == "safetensors" {
|
||||
ctxDefault = strconv.Itoa(p.Context)
|
||||
}
|
||||
if len(opts) > 0 {
|
||||
if most := opts[0].MostContext(p.KVBytesPerToken, p.Trained); most > 0 {
|
||||
ctxDefault = strconv.Itoa(most)
|
||||
}
|
||||
}
|
||||
if len(opts) > 0 && opts[0].Engine == "strata" {
|
||||
ctxDefault = "32768"
|
||||
}
|
||||
@@ -249,6 +256,11 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
|
||||
Description: v[3], Format: a.Format}
|
||||
if a.Format == "gguf" {
|
||||
req.File = a.Name
|
||||
// A context longer than the default pool is the pool: one
|
||||
// slot of it, where the server would refuse the request.
|
||||
if ctxLen > modelserve.DefaultCtxPool {
|
||||
req.CtxPool = ctxLen
|
||||
}
|
||||
}
|
||||
o := opts[slices.Index(where, v[4])]
|
||||
// Asked for a context other than planned: does it still fit?
|
||||
|
||||
@@ -221,11 +221,11 @@ func (m consoleModel) userMenu(u store.User) *menuDialog {
|
||||
items = append(items,
|
||||
menuItem{"Add to an organization, team or project…", func(m *consoleModel) tea.Cmd {
|
||||
m.openForm(&formDialog{title: "Membership for " + who,
|
||||
note: "A member of a team or organization may use every project in it.",
|
||||
note: "A member of a team or organization may use every project in it. On an organization, a manager also sees its page and an admin also changes it.",
|
||||
fields: []formField{
|
||||
choiceF("level", []string{"org", "team", "project"}, "team"),
|
||||
textF("which", "", "Acme, Acme/ml or Acme/ml/training"),
|
||||
choiceF("role", []string{store.RoleMember, "manager"}, store.RoleMember),
|
||||
choiceF("role", store.MembershipRoles, store.RoleMember),
|
||||
},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
if v[1] == "" {
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
// The console's Slurm view: the Slurm cluster run beside llmgw (BACKLOG P),
|
||||
// each of its nodes and whose it is now, as `llmgw ctl slurm status` shows
|
||||
// them. A node is lent to Slurm, or taken back, from its row. Turning Slurm
|
||||
// on, and choosing its nodes, is `ctl slurm setup` or server.yaml's slurm
|
||||
// part: a form for it would be server.yaml written again.
|
||||
|
||||
func (m consoleModel) slurmPanel(pw int) (string, string, string) {
|
||||
keys := "→ or Enter open · ↑↓ choose · Q leave"
|
||||
if m.inPanel {
|
||||
keys = "↑↓ node · Enter lend or take back · ← back · Q leave"
|
||||
}
|
||||
st := m.data.slurm
|
||||
head := "\n " + sDim.Render("Slurm beside llmgw, on its nodes: training and batch jobs.")
|
||||
if !st.Enabled {
|
||||
return "Slurm", head + "\n\n " + sDim.Render("Slurm is off. Turn it on with: "+
|
||||
"llmgw ctl slurm setup --controller NODE --nodes NODE,... (or --all-nodes)"), keys
|
||||
}
|
||||
ctl := sDim.Render("not heard from yet")
|
||||
switch c := st.Ctld; {
|
||||
case c == nil:
|
||||
case c.Error != "":
|
||||
ctl = sBad.Render(c.Error)
|
||||
case c.SlurmctldRunning:
|
||||
ctl = sOK.Render("slurmctld " + c.Version + " running")
|
||||
default:
|
||||
ctl = sBad.Render("slurmctld not running")
|
||||
}
|
||||
head += fmt.Sprintf("\n controller %s: %s", st.Controller, ctl)
|
||||
head += "\n " + sDim.Render(fmt.Sprintf("cluster %s, partition %s; sbatch and srun on %s", st.Cluster, st.Partition, st.Controller))
|
||||
head += "\n " + sDim.Render("A node is llmgw's until lent; lent, its replicas move off first.")
|
||||
if len(st.Waiting) > 0 {
|
||||
head += "\n " + sWarn.Render(fmt.Sprintf("not in slurm.conf until they report their hardware: %v", st.Waiting))
|
||||
}
|
||||
head += "\n\n"
|
||||
if len(st.Nodes) == 0 {
|
||||
return "Slurm", head + " " + sDim.Render("No node runs slurmd."), keys
|
||||
}
|
||||
var rows [][]string
|
||||
for _, n := range st.Nodes {
|
||||
rows = append(rows, slurmRow(n))
|
||||
}
|
||||
return "Slurm", head + table([]string{"NODE", "WHOSE", "IN SLURM", "JOBS", "SLURMD", "NOTE"}, rows, m.row, pw, m.inPanel), keys
|
||||
}
|
||||
|
||||
func slurmRow(n slurmsvc.NodeStatus) []string {
|
||||
state, jobs, slurmd, note := "-", "-", "-", n.Describe()
|
||||
if r := n.Status; r != nil {
|
||||
state = or(r.State, "-")
|
||||
jobs = fmt.Sprint(r.Jobs)
|
||||
slurmd = sBad.Render("stopped")
|
||||
if r.SlurmdRunning {
|
||||
slurmd = sOK.Render("running")
|
||||
}
|
||||
if r.Error != "" {
|
||||
note = sBad.Render(r.Error)
|
||||
}
|
||||
}
|
||||
if !n.Connected {
|
||||
note = sWarn.Render("agent not connected") + "; " + note
|
||||
}
|
||||
whose := string(n.Phase)
|
||||
switch n.Phase {
|
||||
case slurmsvc.Lent:
|
||||
whose = sOK.Render("Slurm's")
|
||||
case slurmsvc.Llmgw:
|
||||
whose = "llmgw's"
|
||||
case slurmsvc.Lending, slurmsvc.Reclaiming:
|
||||
whose = sWarn.Render(whose)
|
||||
}
|
||||
return []string{n.Node, whose, state, jobs, slurmd, note}
|
||||
}
|
||||
|
||||
// slurmMenu is what can be done with one of Slurm's nodes.
|
||||
func (m consoleModel) slurmMenu(n slurmsvc.NodeStatus) *menuDialog {
|
||||
var items []menuItem
|
||||
if n.Phase == slurmsvc.Lent || n.Phase == slurmsvc.Lending {
|
||||
items = append(items, menuItem{"Take back from Slurm…", func(m *consoleModel) tea.Cmd {
|
||||
m.confirmD = &confirmDialog{title: "Take " + n.Node + " back",
|
||||
text: "Slurm starts no new job there. llmgw places replicas on it again once the jobs running there have ended.",
|
||||
yes: func(m *consoleModel) tea.Cmd { return m.slurmLend(n.Node, false) }}
|
||||
return nil
|
||||
}})
|
||||
} else {
|
||||
items = append(items, menuItem{"Lend to Slurm…", func(m *consoleModel) tea.Cmd {
|
||||
text := "Its replicas move off first, each replacement up before the original stops; then Slurm runs jobs there."
|
||||
if n.Replicas > 0 {
|
||||
text = fmt.Sprintf("Its %d replica(s) move off first, each replacement up before the original stops; "+
|
||||
"a model with nowhere else to go stops serving. Then Slurm runs jobs there.", n.Replicas)
|
||||
}
|
||||
m.confirmD = &confirmDialog{title: "Lend " + n.Node + " to Slurm", text: text,
|
||||
yes: func(m *consoleModel) tea.Cmd { return m.slurmLend(n.Node, true) }}
|
||||
return nil
|
||||
}})
|
||||
}
|
||||
return &menuDialog{title: n.Node, items: items}
|
||||
}
|
||||
|
||||
// slurmLend puts a node in slurm.lent, or takes it out, as `llmgw ctl slurm
|
||||
// lend|reclaim` does: the newest settings, changed and put back.
|
||||
func (m *consoleModel) slurmLend(node string, add bool) tea.Cmd {
|
||||
title, note, done := "Lend "+node, "lent to Slurm from the console", node+" is being lent to Slurm: its replicas move off first."
|
||||
if !add {
|
||||
title, note, done = "Take back "+node, "taken back from Slurm in the console",
|
||||
node+" is being taken back: llmgw has it once its Slurm jobs end."
|
||||
}
|
||||
m.busy = title
|
||||
c := m.client
|
||||
return func() tea.Msg {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
var cur adminapi.SettingsView
|
||||
if err := c.get(ctx, "/api/v1/settings", &cur); err != nil {
|
||||
return actionDoneMsg{title: title, err: err}
|
||||
}
|
||||
doc, err := settings.FromBytes("settings", []byte(cur.Body))
|
||||
if err == nil {
|
||||
err = lendChange(doc, []string{node}, add)
|
||||
}
|
||||
var body []byte
|
||||
if err == nil {
|
||||
body, err = doc.Bytes()
|
||||
}
|
||||
if err != nil {
|
||||
return actionDoneMsg{title: title, err: err}
|
||||
}
|
||||
req, _ := json.Marshal(adminapi.SettingsRequest{Base: cur.Version, Body: string(body), Note: note})
|
||||
var v adminapi.SettingsView
|
||||
if err := c.put(ctx, "/api/v1/settings", req, &v); err != nil {
|
||||
return actionDoneMsg{title: title, err: err}
|
||||
}
|
||||
return actionDoneMsg{title: title, text: done}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
)
|
||||
|
||||
func TestTheConsoleLendsANodeToSlurmFromItsRow(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
var put adminapi.SettingsRequest
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v1/settings":
|
||||
_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 7,
|
||||
Body: "slurm:\n enabled: true\n controller: node06\n nodes: [node05, cuda01]\n"})
|
||||
case r.Method == http.MethodPut && r.URL.Path == "/api/v1/settings":
|
||||
b, _ := io.ReadAll(r.Body)
|
||||
mu.Lock()
|
||||
_ = json.Unmarshal(b, &put)
|
||||
mu.Unlock()
|
||||
_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 8})
|
||||
default:
|
||||
_, _ = w.Write([]byte("[]"))
|
||||
}
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
m := newConsoleModel(srv.URL, "/nonexistent/server.yaml")
|
||||
m.loaded, m.menu, m.inPanel = true, cSlurm, true
|
||||
m.data.slurm = slurmsvc.Status{Enabled: true, Cluster: "llmgw", Controller: "node06", Partition: "gpu",
|
||||
Ctld: &slurmsvc.Reported{SlurmctldRunning: true, Version: "23.11.4"},
|
||||
Nodes: []slurmsvc.NodeStatus{
|
||||
{Node: "node05", Phase: slurmsvc.Lent, Connected: true,
|
||||
Status: &slurmsvc.Reported{State: "MIXED", Jobs: 2, SlurmdRunning: true}},
|
||||
{Node: "cuda01", Phase: slurmsvc.Llmgw, Replicas: 3, Connected: true,
|
||||
Status: &slurmsvc.Reported{State: "IDLE+DRAIN", SlurmdRunning: true}},
|
||||
}}
|
||||
var tm tea.Model = m
|
||||
view := tm.View()
|
||||
for _, want := range []string{"slurmctld 23.11.4 running", "node05", "Slurm's", "MIXED", "cuda01", "llmgw's"} {
|
||||
if !strings.Contains(view, want) {
|
||||
t.Errorf("the Slurm view has no %q:\n%s", want, view)
|
||||
}
|
||||
}
|
||||
|
||||
// cuda01, the second row: lent, after saying its replicas move.
|
||||
tm = press(tm, "down", "enter")
|
||||
if !strings.Contains(tm.View(), "Lend to Slurm") {
|
||||
t.Fatalf("no lend in the menu:\n%s", tm.View())
|
||||
}
|
||||
tm = press(tm, "enter")
|
||||
if !strings.Contains(tm.View(), "3 replica(s) move off first") {
|
||||
t.Fatalf("the question does not say what moves:\n%s", tm.View())
|
||||
}
|
||||
tm = pressRun(tm, "y")
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if put.Base != 7 || !strings.Contains(put.Body, "lent:") || !strings.Contains(put.Body, "cuda01") {
|
||||
t.Fatalf("the settings put back: %+v", put)
|
||||
}
|
||||
if !strings.Contains(tm.View(), "cuda01 is being lent to Slurm") {
|
||||
t.Errorf("not said:\n%s", tm.View())
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheConsoleTakesALentNodeBack(t *testing.T) {
|
||||
m, _ := consoleWith(t, cSlurm)
|
||||
m.data.slurm = slurmsvc.Status{Enabled: true, Controller: "node06",
|
||||
Nodes: []slurmsvc.NodeStatus{{Node: "node05", Phase: slurmsvc.Lent, Connected: true}}}
|
||||
var tm tea.Model = m
|
||||
tm = press(tm, "enter")
|
||||
if v := tm.View(); !strings.Contains(v, "Take back from Slurm") || strings.Contains(v, "Lend to Slurm") {
|
||||
t.Fatalf("a lent node's menu:\n%s", v)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheConsoleSaysHowToTurnSlurmOn(t *testing.T) {
|
||||
m, _ := consoleWith(t, cSlurm)
|
||||
var tm tea.Model = m
|
||||
if !strings.Contains(tm.View(), "llmgw ctl slurm setup") {
|
||||
t.Fatalf("Slurm off says nothing of turning it on:\n%s", tm.View())
|
||||
}
|
||||
}
|
||||
@@ -391,7 +391,8 @@ func newUserJoinCmd(open userOpener) *cobra.Command {
|
||||
Long: "A person can be a member of any number of them. A member of a team or\n" +
|
||||
"organization may use every project in it.",
|
||||
Example: " llmgw users join ayse@example.com --team Acme/ml\n" +
|
||||
" llmgw users join ayse@example.com --project Scania/research/vision",
|
||||
" llmgw users join ayse@example.com --project Scania/research/vision\n" +
|
||||
" llmgw users join lars@example.com --org Scania --role admin",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
s, err := open(cmd)
|
||||
@@ -415,7 +416,8 @@ func newUserJoinCmd(open userOpener) *cobra.Command {
|
||||
},
|
||||
}
|
||||
where.bind(cmd)
|
||||
cmd.Flags().StringVar(&role, "role", store.RoleMember, "member, or manager: with --org, may see the organization's page")
|
||||
cmd.Flags().StringVar(&role, "role", store.RoleMember,
|
||||
"member; with --org also manager, who sees the organization's page, or admin, who also changes its teams, projects, people and keys")
|
||||
cmd.Flags().StringVar(&by, "by", "", "who added them, for the record")
|
||||
return cmd
|
||||
}
|
||||
|
||||
@@ -158,6 +158,10 @@ type Server struct {
|
||||
// Firewall turns llmgw's firewall on every machine on or off.
|
||||
Firewall FirewallSettings `koanf:"firewall"`
|
||||
|
||||
// Slurm is a Slurm cluster run beside llmgw on some of its nodes, and
|
||||
// which of them it may use now.
|
||||
Slurm SlurmSettings `koanf:"slurm"`
|
||||
|
||||
// Presets is how the models that come with llmgw reach the catalog.
|
||||
Presets PresetSettings `koanf:"presets"`
|
||||
|
||||
@@ -960,6 +964,9 @@ func (s Server) Validate() error {
|
||||
if _, err := s.Facility.Policy(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.Slurm.Check(); err != nil {
|
||||
return err
|
||||
}
|
||||
if s.Control.Timezone != "" {
|
||||
if _, err := time.LoadLocation(s.Control.Timezone); err != nil {
|
||||
return fmt.Errorf("control.timezone %q: %w", s.Control.Timezone, err)
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"regexp"
|
||||
"slices"
|
||||
)
|
||||
|
||||
// SlurmSettings is a Slurm cluster run beside llmgw on some of its nodes
|
||||
// (BACKLOG P), for training and other batch work llmgw does not schedule.
|
||||
// The leader tells each named node's agent what to run; the agents install
|
||||
// it from the machine's package sources and keep it going.
|
||||
type SlurmSettings struct {
|
||||
// Enabled runs Slurm on the nodes below. False stops llmgw keeping it;
|
||||
// what was installed is left installed, and every node is llmgw's.
|
||||
Enabled bool `koanf:"enabled"`
|
||||
|
||||
// ClusterName is Slurm's ClusterName. Empty is "llmgw".
|
||||
ClusterName string `koanf:"cluster_name"`
|
||||
|
||||
// Controller is the llmgw node whose agent runs slurmctld. It needs no
|
||||
// GPU, and need not be one of Nodes.
|
||||
Controller string `koanf:"controller"`
|
||||
|
||||
// Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained
|
||||
// in Slurm, until it is also in Lent.
|
||||
Nodes []string `koanf:"nodes"`
|
||||
|
||||
// AllNodes runs slurmd on every llmgw node, those that join later too,
|
||||
// besides any named in Nodes. Each is llmgw's until it is lent, as any.
|
||||
AllNodes bool `koanf:"all_nodes"`
|
||||
|
||||
// Lent are the nodes of Nodes Slurm may run jobs on now. A node put here
|
||||
// is drained of llmgw's replicas first, as `ctl drain` does, and resumed
|
||||
// in Slurm once none is left; a node taken out is drained in Slurm and
|
||||
// given back to llmgw when its last job ends.
|
||||
Lent []string `koanf:"lent"`
|
||||
|
||||
// Partition is the one partition every node is in. Empty is "gpu".
|
||||
Partition string `koanf:"partition"`
|
||||
}
|
||||
|
||||
// slurmName is what Slurm takes as a node, cluster or partition name.
|
||||
var slurmName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]{0,62}$`)
|
||||
|
||||
// Cluster is ClusterName or its default.
|
||||
func (s SlurmSettings) Cluster() string {
|
||||
if s.ClusterName == "" {
|
||||
return "llmgw"
|
||||
}
|
||||
return s.ClusterName
|
||||
}
|
||||
|
||||
// PartitionName is Partition or its default.
|
||||
func (s SlurmSettings) PartitionName() string {
|
||||
if s.Partition == "" {
|
||||
return "gpu"
|
||||
}
|
||||
return s.Partition
|
||||
}
|
||||
|
||||
// IsLent is whether Slurm may use the node now.
|
||||
func (s SlurmSettings) IsLent(node string) bool {
|
||||
return s.Enabled && (s.AllNodes || slices.Contains(s.Nodes, node)) && slices.Contains(s.Lent, node)
|
||||
}
|
||||
|
||||
// WithNodes is the settings with every node of ids in Nodes when AllNodes:
|
||||
// the nodes that run slurmd, as the leader's inventory has them.
|
||||
func (s SlurmSettings) WithNodes(ids []string) SlurmSettings {
|
||||
if !s.AllNodes {
|
||||
return s
|
||||
}
|
||||
nodes := slices.Clone(s.Nodes)
|
||||
for _, id := range ids {
|
||||
if !slices.Contains(nodes, id) {
|
||||
nodes = append(nodes, id)
|
||||
}
|
||||
}
|
||||
s.Nodes = nodes
|
||||
return s
|
||||
}
|
||||
|
||||
// Check refuses settings Slurm could not run on.
|
||||
func (s SlurmSettings) Check() error {
|
||||
if !s.Enabled {
|
||||
return nil
|
||||
}
|
||||
if s.Controller == "" {
|
||||
return fmt.Errorf("slurm.controller is empty: name the node that runs slurmctld")
|
||||
}
|
||||
if !slurmName.MatchString(s.Controller) {
|
||||
return fmt.Errorf("slurm.controller %q is not a name Slurm takes", s.Controller)
|
||||
}
|
||||
for _, n := range []struct{ key, v string }{{"slurm.cluster_name", s.ClusterName}, {"slurm.partition", s.Partition}} {
|
||||
if n.v != "" && !slurmName.MatchString(n.v) {
|
||||
return fmt.Errorf("%s %q is not a name Slurm takes: letters, digits, _ . -", n.key, n.v)
|
||||
}
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for i, n := range s.Nodes {
|
||||
if !slurmName.MatchString(n) {
|
||||
return fmt.Errorf("slurm.nodes[%d] %q is not a name Slurm takes", i, n)
|
||||
}
|
||||
if seen[n] {
|
||||
return fmt.Errorf("slurm.nodes names %s twice", n)
|
||||
}
|
||||
seen[n] = true
|
||||
}
|
||||
for i, n := range s.Lent {
|
||||
if !seen[n] && !s.AllNodes {
|
||||
return fmt.Errorf("slurm.lent[%d] %q is not in slurm.nodes: only a node that runs slurmd can be lent", i, n)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSlurmCheck(t *testing.T) {
|
||||
ok := SlurmSettings{Enabled: true, Controller: "node06", Nodes: []string{"cuda01", "site2"}, Lent: []string{"cuda01"}}
|
||||
if err := ok.Check(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !ok.IsLent("cuda01") || ok.IsLent("site2") || ok.IsLent("node06") {
|
||||
t.Error("IsLent is wrong")
|
||||
}
|
||||
if ok.Cluster() != "llmgw" || ok.PartitionName() != "gpu" {
|
||||
t.Error("defaults are wrong")
|
||||
}
|
||||
if (SlurmSettings{Lent: []string{"x"}}).Check() != nil {
|
||||
t.Error("settings that are off were checked")
|
||||
}
|
||||
for want, s := range map[string]SlurmSettings{
|
||||
"controller is empty": {Enabled: true},
|
||||
"not a name": {Enabled: true, Controller: "bad name"},
|
||||
"twice": {Enabled: true, Controller: "c", Nodes: []string{"a", "a"}},
|
||||
"not in slurm.nodes": {Enabled: true, Controller: "c", Nodes: []string{"a"}, Lent: []string{"b"}},
|
||||
"slurm.partition": {Enabled: true, Controller: "c", Partition: "a b"},
|
||||
} {
|
||||
if err := s.Check(); err == nil || !strings.Contains(err.Error(), want) {
|
||||
t.Errorf("%+v: got %v, want %q", s, err, want)
|
||||
}
|
||||
}
|
||||
if (SlurmSettings{Enabled: false, Lent: []string{"a"}}).IsLent("a") {
|
||||
t.Error("a node is lent with Slurm off")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlurmAllNodes(t *testing.T) {
|
||||
s := SlurmSettings{Enabled: true, Controller: "c", AllNodes: true, Lent: []string{"joined"}}
|
||||
if err := s.Check(); err != nil {
|
||||
t.Fatalf("a node lent with all_nodes refused: %v", err)
|
||||
}
|
||||
if !s.IsLent("joined") {
|
||||
t.Fatal("not lent with all_nodes")
|
||||
}
|
||||
if got := s.WithNodes([]string{"a", "joined"}).Nodes; len(got) != 2 {
|
||||
t.Fatalf("nodes %v", got)
|
||||
}
|
||||
s.AllNodes = false
|
||||
if s.Check() == nil || len(s.WithNodes([]string{"a"}).Nodes) != 0 {
|
||||
t.Fatal("all_nodes off still takes the inventory's nodes")
|
||||
}
|
||||
}
|
||||
@@ -78,7 +78,9 @@ type API struct {
|
||||
// tenancy is organizations, keys and people; nil without a store.
|
||||
tenancy TenancyStore
|
||||
// train is adapter training; nil without a store and agents.
|
||||
train *TrainAdmin
|
||||
train *TrainAdmin
|
||||
// slurm is the Slurm cluster beside llmgw; nil without a store and agents.
|
||||
slurm SlurmService
|
||||
settings *Settings
|
||||
auth *AuthAdmin
|
||||
mcp MCPStore
|
||||
@@ -136,6 +138,7 @@ func (a *API) Register(mux Mux) {
|
||||
a.registerKB(mux)
|
||||
a.registerTenancy(mux)
|
||||
a.registerTrain(mux)
|
||||
a.registerSlurm(mux)
|
||||
a.registerGuard(mux)
|
||||
a.registerSettings(mux)
|
||||
a.registerAuth(mux)
|
||||
|
||||
@@ -216,7 +216,7 @@ func (a *API) purgeFromNodes(ctx context.Context, sources []string, others []cat
|
||||
names[filepath.Base(s)] = true
|
||||
}
|
||||
wanted := inUse(others)
|
||||
var out []CollectResult
|
||||
var rs []removal
|
||||
for _, n := range a.ctl.Inventory().Nodes() {
|
||||
size := map[string]int64{}
|
||||
have := make([]string, 0, len(n.Weights))
|
||||
@@ -235,15 +235,14 @@ func (a *API) purgeFromNodes(ctx context.Context, sources []string, others []cat
|
||||
if len(paths) == 0 {
|
||||
continue
|
||||
}
|
||||
res := CollectResult{Node: n.ID, Files: len(paths)}
|
||||
for _, p := range paths {
|
||||
res.Bytes += size[p]
|
||||
rs = append(rs, removal{node: n.ID, paths: paths, size: size})
|
||||
}
|
||||
out := a.removeConfirmed(ctx, rm, rs)
|
||||
for _, r := range out {
|
||||
if r.Error != "" || r.Left > 0 {
|
||||
slog.Warn("a purged model's copies were not all deleted on a node", "node", r.Node,
|
||||
"left", r.Left, "err", r.Error)
|
||||
}
|
||||
if err := rm.RemoveWeights(ctx, n.ID, paths); err != nil {
|
||||
res.Error, res.Files, res.Bytes = err.Error(), 0, 0
|
||||
slog.Warn("a purged model's copies could not be deleted on a node", "node", n.ID, "err", err)
|
||||
}
|
||||
out = append(out, res)
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/federation"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
@@ -392,6 +393,12 @@ var Routes = []Route{
|
||||
Description: "Their keys in projects they no longer reach are revoked.",
|
||||
Request: MembershipRequest{}, Response: UserView{}},
|
||||
|
||||
{Method: "GET", Path: "/api/v1/slurm", Summary: "The Slurm cluster beside llmgw, and where each of its nodes is",
|
||||
Description: "From the cluster's settings (slurm.*) and what each node's agent last said. A node's phase is llmgw " +
|
||||
"(drained in Slurm, no job left), lending (its replicas moving off), lent, reclaiming (drained in Slurm, jobs " +
|
||||
"still running) or unknown (its agent has not said).",
|
||||
Response: slurmsvc.Status{}},
|
||||
|
||||
{Method: "POST", Path: "/api/v1/train/jobs", Summary: "Queue a LoRA adapter's training on a node",
|
||||
Description: "The node trains when its agent is there, nothing else trains on it, and a card has gpu_memory_mib free; " +
|
||||
"the pairs are NDJSON of {\"q\", \"a\"}. An adapter learns how answers sound far better than what they say.",
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/slurmsvc"
|
||||
)
|
||||
|
||||
// SlurmService is the Slurm cluster beside llmgw (BACKLOG P), as the leader
|
||||
// keeps it.
|
||||
type SlurmService interface {
|
||||
Status() slurmsvc.Status
|
||||
}
|
||||
|
||||
// SetSlurm gives the API the Slurm service.
|
||||
func (a *API) SetSlurm(s SlurmService) { a.slurm = s }
|
||||
|
||||
func (a *API) registerSlurm(mux Mux) {
|
||||
mux.HandleFunc("GET /api/v1/slurm", a.slurmStatus)
|
||||
}
|
||||
|
||||
func (a *API) slurmStatus(w http.ResponseWriter, _ *http.Request) {
|
||||
if a.slurm == nil {
|
||||
writeError(w, http.StatusNotImplemented, "Slurm needs the store and the nodes' agents, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, a.slurm.Status())
|
||||
}
|
||||
@@ -634,8 +634,12 @@ func (a *API) membership(add bool) http.HandlerFunc {
|
||||
if role == "" {
|
||||
role = store.RoleMember
|
||||
}
|
||||
if role != store.RoleMember && role != store.RoleManager {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member or manager", role))
|
||||
if role != store.RoleMember && role != store.RoleManager && role != store.RoleOrgAdmin {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member, manager or admin", role))
|
||||
return
|
||||
}
|
||||
if role != store.RoleMember && scope != store.ScopeOrg {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("%s is a role on an organization", role))
|
||||
return
|
||||
}
|
||||
if err := a.tenancy.AddMembership(ctx, u.ID, scope, id, role, actorOf(r), time.Now()); err != nil {
|
||||
|
||||
@@ -1,11 +1,14 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
)
|
||||
|
||||
// OrphanView is cached weights on one node that no catalog model accounts for.
|
||||
@@ -71,12 +74,100 @@ type CollectRequest struct {
|
||||
Node string `json:"node,omitempty"`
|
||||
}
|
||||
|
||||
// CollectResult is what was deleted on one node, or why nothing was.
|
||||
// CollectResult is what was deleted on one node, or why nothing was: what
|
||||
// the node no longer lists once asked, not what it was asked to delete.
|
||||
type CollectResult struct {
|
||||
Node string `json:"node"`
|
||||
Files int `json:"files"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
Error string `json:"error,omitempty"`
|
||||
// Left and LeftBytes are what the node still lists: refused, or not
|
||||
// gone by the time the sweep stopped waiting. Its agent's log says why.
|
||||
Left int `json:"left,omitempty"`
|
||||
LeftBytes int64 `json:"left_bytes,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// confirmWait is how long a sweep waits for the nodes to list their files
|
||||
// without the ones deleted: they scan theirs every heartbeat, five seconds
|
||||
// by default, and a large set takes a moment to delete.
|
||||
var confirmWait = 30 * time.Second
|
||||
|
||||
// removal is what one node is asked to delete, with each file's size.
|
||||
type removal struct {
|
||||
node string
|
||||
paths []string
|
||||
size map[string]int64
|
||||
}
|
||||
|
||||
// removeConfirmed asks each node to delete its paths and says what each then
|
||||
// no longer lists. Counted as asked, a sweep said "25 GB freed" of files the
|
||||
// node had refused and kept.
|
||||
func (a *API) removeConfirmed(ctx context.Context, rm control.WeightRemover, rs []removal) []CollectResult {
|
||||
out := make([]CollectResult, len(rs))
|
||||
pending := map[int]bool{}
|
||||
for i, r := range rs {
|
||||
out[i].Node = r.node
|
||||
if err := rm.RemoveWeights(ctx, r.node, r.paths); err != nil {
|
||||
out[i].Error = err.Error()
|
||||
for _, p := range r.paths {
|
||||
out[i].Left++
|
||||
out[i].LeftBytes += r.size[p]
|
||||
}
|
||||
continue
|
||||
}
|
||||
pending[i] = true
|
||||
}
|
||||
deadline := time.Now().Add(confirmWait)
|
||||
for {
|
||||
for i := range pending {
|
||||
if left := a.stillListed(rs[i]); len(left) == 0 {
|
||||
delete(pending, i)
|
||||
}
|
||||
}
|
||||
if len(pending) == 0 || !time.Now().Before(deadline) {
|
||||
break
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
deadline = time.Now()
|
||||
case <-time.After(time.Second):
|
||||
}
|
||||
}
|
||||
for i, r := range rs {
|
||||
if out[i].Error != "" {
|
||||
continue
|
||||
}
|
||||
left := a.stillListed(r)
|
||||
for _, p := range r.paths {
|
||||
if left[p] {
|
||||
out[i].Left++
|
||||
out[i].LeftBytes += r.size[p]
|
||||
} else {
|
||||
out[i].Files++
|
||||
out[i].Bytes += r.size[p]
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// stillListed is which of a removal's paths its node still lists.
|
||||
func (a *API) stillListed(r removal) map[string]bool {
|
||||
left := map[string]bool{}
|
||||
n, ok := a.ctl.Inventory().Get(r.node)
|
||||
if !ok {
|
||||
return left
|
||||
}
|
||||
want := map[string]bool{}
|
||||
for _, p := range r.paths {
|
||||
want[p] = true
|
||||
}
|
||||
for _, w := range n.Weights {
|
||||
if want[w.Path] {
|
||||
left[w.Path] = true
|
||||
}
|
||||
}
|
||||
return left
|
||||
}
|
||||
|
||||
// collectWeights deletes them.
|
||||
@@ -97,20 +188,26 @@ func (a *API) collectWeights(w http.ResponseWriter, r *http.Request) {
|
||||
// An empty body is the "all nodes" case and must not be an error.
|
||||
_ = json.NewDecoder(r.Body).Decode(&body)
|
||||
|
||||
out := make([]CollectResult, 0)
|
||||
// A node that has gone since the list was built is reported rather than
|
||||
// failing the whole sweep: the other nodes' space is still worth
|
||||
// freeing, and this one will be offered again next time.
|
||||
var rs []removal
|
||||
for _, o := range a.Orphans() {
|
||||
if body.Node != "" && o.Node != body.Node {
|
||||
continue
|
||||
}
|
||||
res := CollectResult{Node: o.Node, Files: len(o.Paths), Bytes: o.Bytes}
|
||||
// A node that has gone since the list was built is reported rather than
|
||||
// failing the whole sweep: the other nodes' space is still worth
|
||||
// freeing, and this one will be offered again next time.
|
||||
if err := rm.RemoveWeights(r.Context(), o.Node, o.Paths); err != nil {
|
||||
res.Error = err.Error()
|
||||
res.Files, res.Bytes = 0, 0
|
||||
}
|
||||
out = append(out, res)
|
||||
rs = append(rs, removal{node: o.Node, paths: o.Paths, size: a.sizesOn(o.Node)})
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
writeJSON(w, http.StatusOK, a.removeConfirmed(r.Context(), rm, rs))
|
||||
}
|
||||
|
||||
// sizesOn is each file a node lists, by path, with its size.
|
||||
func (a *API) sizesOn(node string) map[string]int64 {
|
||||
out := map[string]int64{}
|
||||
if n, ok := a.ctl.Inventory().Get(node); ok {
|
||||
for _, w := range n.Weights {
|
||||
out[w.Path] = w.SizeBytes
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
// deletingDispatcher deletes what it is asked to as a node would, by its
|
||||
// files dropping out of what the node lists — but for refused, which it
|
||||
// keeps, as an agent keeps a file it cannot prove it fetched.
|
||||
type deletingDispatcher struct {
|
||||
nullDispatcher
|
||||
inv *inventory.Store
|
||||
refused string
|
||||
}
|
||||
|
||||
func (d *deletingDispatcher) RemoveWeights(_ context.Context, node string, paths []string) error {
|
||||
n, _ := d.inv.Get(node)
|
||||
n.Weights = slices.DeleteFunc(n.Weights, func(w discovery.LocalWeights) bool {
|
||||
return w.Path != d.refused && slices.Contains(paths, w.Path)
|
||||
})
|
||||
d.inv.Upsert(n)
|
||||
return nil
|
||||
}
|
||||
|
||||
// What a sweep says it freed is what the node no longer lists: a file it
|
||||
// kept is said to be left, not counted as freed.
|
||||
func TestASweepSaysWhatWasDeletedNotWhatWasAsked(t *testing.T) {
|
||||
confirmWait = 1500 * time.Millisecond
|
||||
t.Cleanup(func() { confirmWait = 30 * time.Second })
|
||||
|
||||
gone := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/old/v1/old.gguf")
|
||||
set := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/big/v1")
|
||||
kept := weights.CachePathFor("/var/lib/llmgw-weights", "http://mirror/stuck/v1/stuck.gguf")
|
||||
cat, _ := catalog.New(nil)
|
||||
d := &deletingDispatcher{refused: kept}
|
||||
ctl := control.New(control.DefaultConfig(), catalog.NewLive(cat), d, nullSink{}, slog.New(slog.DiscardHandler))
|
||||
d.inv = ctl.Inventory()
|
||||
node := gpuNode("n1", 1)
|
||||
node.Weights = []discovery.LocalWeights{{Path: gone, SizeBytes: 1 << 30}, {Path: set, SizeBytes: 20 << 30},
|
||||
{Path: kept, SizeBytes: 4 << 30}}
|
||||
ctl.Inventory().Upsert(node)
|
||||
a := New(ctl)
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
a.collectWeights(rec, httptest.NewRequest(http.MethodPost, "/api/v1/weights/collect", strings.NewReader(`{"node":"n1"}`)))
|
||||
var got []CollectResult
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
|
||||
t.Fatalf("%v: %s", err, rec.Body.String())
|
||||
}
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("results %+v", got)
|
||||
}
|
||||
r := got[0]
|
||||
if r.Files != 2 || r.Bytes != 21<<30 || r.Left != 1 || r.LeftBytes != 4<<30 {
|
||||
t.Errorf("result %+v: want 2 files and 21 GB freed, 1 file of 4 GB left", r)
|
||||
}
|
||||
}
|
||||
@@ -84,6 +84,8 @@ type Service struct {
|
||||
agentSettings func(node string) string
|
||||
// train is told how training jobs go; see SetTrain.
|
||||
train TrainHandler
|
||||
// slurm is told of the nodes' part in Slurm; see SetSlurm.
|
||||
slurm SlurmHandler
|
||||
|
||||
// logWaits are the log requests awaiting a reply, by request id.
|
||||
logMu sync.Mutex
|
||||
@@ -308,6 +310,10 @@ func (s *Service) Connect(
|
||||
return err
|
||||
}
|
||||
}
|
||||
// Its part in Slurm, sent by the Slurm service's next pass.
|
||||
if h := s.slurmHandler(); h != nil {
|
||||
h.OnRegister(nodeID)
|
||||
}
|
||||
|
||||
// Recover replica state from what this node is already running, before any
|
||||
// reconciliation pass can conclude the cluster is empty and place
|
||||
@@ -515,6 +521,11 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
|
||||
h.OnTrainStatus(nodeID, m.TrainStatus)
|
||||
}
|
||||
|
||||
case *llmgwv1.AgentMessage_SlurmStatus:
|
||||
if h := s.slurmHandler(); h != nil {
|
||||
h.OnSlurmStatus(nodeID, m.SlurmStatus)
|
||||
}
|
||||
|
||||
case *llmgwv1.AgentMessage_TrainArtifact:
|
||||
if h := s.trainHandler(); h != nil {
|
||||
h.OnTrainArtifact(nodeID, m.TrainArtifact)
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
package agentsvc
|
||||
|
||||
import (
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
)
|
||||
|
||||
// SlurmHandler is told of the nodes' part in Slurm (BACKLOG P).
|
||||
type SlurmHandler interface {
|
||||
OnSlurmStatus(node string, st *llmgwv1.SlurmStatus)
|
||||
// OnRegister is a node's new stream, which is told again what it is.
|
||||
OnRegister(node string)
|
||||
}
|
||||
|
||||
// SetSlurm gives the service what Slurm's messages go to.
|
||||
func (s *Service) SetSlurm(h SlurmHandler) {
|
||||
s.mu.Lock()
|
||||
s.slurm = h
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func (s *Service) slurmHandler() SlurmHandler {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
return s.slurm
|
||||
}
|
||||
@@ -296,7 +296,7 @@ func (s *Server) Start(req Request) (Job, error) {
|
||||
}
|
||||
req.sized = req.CtxPool > 0 || req.ContextLength > 0
|
||||
if req.CtxPool <= 0 {
|
||||
req.CtxPool = 16384
|
||||
req.CtxPool = DefaultCtxPool
|
||||
}
|
||||
if req.ContextLength <= 0 {
|
||||
req.ContextLength = 4096
|
||||
@@ -731,6 +731,10 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
|
||||
return out
|
||||
}
|
||||
|
||||
// DefaultCtxPool is llama.cpp's pool when a request names none: a context
|
||||
// asked for longer than it needs a pool of its own.
|
||||
const DefaultCtxPool = 16384
|
||||
|
||||
// llamaArgs size llama.cpp's pool and its slots together. The pool is shared
|
||||
// by the slots, so slots of the context the catalog promises are as many as
|
||||
// the pool holds: the engine's default of four turned a 32768 pool into
|
||||
|
||||
@@ -43,6 +43,9 @@ type Plan struct {
|
||||
License string `json:"license,omitempty"`
|
||||
Params int64 `json:"params,omitempty"`
|
||||
Context int `json:"context"`
|
||||
// Trained is the context the model was trained for, the most a request
|
||||
// may use; 0 when neither its header nor its config says.
|
||||
Trained int `json:"trained,omitempty"`
|
||||
// KVBytesPerToken is one token of context, the whole model: what an
|
||||
// option needs for another context is worked out from it. 0 is unknown.
|
||||
KVBytesPerToken int64 `json:"kv_bytes_per_token,omitempty"`
|
||||
@@ -149,6 +152,7 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
|
||||
p.Servable, p.Reason = false, v.Reason
|
||||
}
|
||||
shape.KVBytesPerToken = v.Header.KVBytesPerToken()
|
||||
p.Trained = int(v.Header.ContextLength) //nolint:gosec // a header's context length
|
||||
shape.KVHeads = int(v.Header.HeadCountKV) //nolint:gosec // a header's head count
|
||||
shape.ActiveShare = v.Header.ActiveShare()
|
||||
if p.Arch == "" {
|
||||
@@ -167,6 +171,7 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
|
||||
m, err := modelplan.FetchModel(ctx, repo.ID)
|
||||
if err == nil {
|
||||
shape.KVBytesPerToken = m.KVBytesPerToken(modelplan.BF16)
|
||||
p.Trained = m.MaxContext
|
||||
shape.KVHeads = m.KVHeads
|
||||
shape.Experts = m.Experts
|
||||
if p.Params == 0 {
|
||||
|
||||
@@ -154,6 +154,9 @@ func (r Result) Explain() string {
|
||||
// Scheduler places replicas onto an inventory.
|
||||
type Scheduler struct {
|
||||
inv *inventory.Store
|
||||
// shun is where a replica of the model being placed failed to start;
|
||||
// set on a copy by Away, never on the scheduler the reconciler shares.
|
||||
shun []Spot
|
||||
// now is the clock power readings are judged fresh by; nil is the wall
|
||||
// clock.
|
||||
now func() time.Time
|
||||
@@ -162,6 +165,41 @@ type Scheduler struct {
|
||||
// New builds a scheduler over an inventory.
|
||||
func New(inv *inventory.Store) *Scheduler { return &Scheduler{inv: inv} }
|
||||
|
||||
// Spot is where a replica ran: a node, and the cards it held there (none
|
||||
// for one on the node's CPU).
|
||||
type Spot struct {
|
||||
Node string
|
||||
GPUs []int
|
||||
}
|
||||
|
||||
// Away is this scheduler placing nowhere in spots: not those cards, nor a
|
||||
// CPU replica on those nodes. A copy, so the one shared is not changed.
|
||||
//
|
||||
// For a model that failed to start somewhere: the same inputs pick the same
|
||||
// node and the same card every time, so without it every retry went back to
|
||||
// where the last one had failed until the model was parked, with another
|
||||
// card or node free all along.
|
||||
func (s *Scheduler) Away(spots []Spot) *Scheduler {
|
||||
c := *s
|
||||
c.shun = spots
|
||||
return &c
|
||||
}
|
||||
|
||||
// shunned is the cards of node n a model failed on, and whether it failed
|
||||
// on the node's CPU.
|
||||
func (s *Scheduler) shunned(node string) (gpus []int, cpu bool) {
|
||||
for _, sp := range s.shun {
|
||||
if sp.Node != node {
|
||||
continue
|
||||
}
|
||||
if len(sp.GPUs) == 0 {
|
||||
cpu = true
|
||||
}
|
||||
gpus = append(gpus, sp.GPUs...)
|
||||
}
|
||||
return gpus, cpu
|
||||
}
|
||||
|
||||
// Place picks the first profile, in priority order, that fits.
|
||||
//
|
||||
// avoid lists nodes already hosting this model, so replicas of one model spread
|
||||
@@ -686,7 +724,12 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
|
||||
}
|
||||
}
|
||||
|
||||
badGPUs, badCPU := s.shunned(n.ID)
|
||||
if p.CPUOnly() {
|
||||
if badCPU {
|
||||
reject("a replica of it failed to start here; trying elsewhere first")
|
||||
continue
|
||||
}
|
||||
// Every ready node can run it; there is no card to be free.
|
||||
c := candidate{node: n}
|
||||
if why := budget.check(n, s.shardWatts(p, c)); why != "" {
|
||||
@@ -703,8 +746,16 @@ func (s *Scheduler) eligibleNodes(p Profile, fabric string, avoid []string, reje
|
||||
if g.Class() != p.GPUClass || !slices.Contains(freeIdx, g.Index) {
|
||||
continue
|
||||
}
|
||||
if slices.Contains(badGPUs, g.Index) {
|
||||
continue
|
||||
}
|
||||
free = append(free, g)
|
||||
}
|
||||
if len(free) < p.GPUsPerNode && len(badGPUs) > 0 {
|
||||
reject(fmt.Sprintf("has %d free %s GPUs besides the ones a replica of it failed to start on, needs %d",
|
||||
len(free), p.GPUClass, p.GPUsPerNode))
|
||||
continue
|
||||
}
|
||||
if len(free) < p.GPUsPerNode {
|
||||
reject(fmt.Sprintf("has %d free %s GPUs, needs %d", len(free), p.GPUClass, p.GPUsPerNode))
|
||||
continue
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
// failAndRetry fails the last replica started, and returns where the next
|
||||
// one is started once the backoff is over.
|
||||
func failAndRetry(t *testing.T, h *harness) ShardCommand {
|
||||
t.Helper()
|
||||
started := h.disp.startedShards()
|
||||
if len(started) == 0 {
|
||||
t.Fatal("nothing was started")
|
||||
}
|
||||
s := started[len(started)-1]
|
||||
h.r.OnShardStatus(s.ReplicaID, s.ShardID, supervisor.Failed, "the engine exited")
|
||||
h.r.Reconcile(t.Context()) // reaps it
|
||||
h.disp.reset()
|
||||
h.advance(DefaultConfig().MaxBackoff)
|
||||
h.r.Reconcile(t.Context())
|
||||
again := h.disp.startedShards()
|
||||
if len(again) == 0 {
|
||||
t.Fatal("it was not tried again")
|
||||
}
|
||||
return again[0]
|
||||
}
|
||||
|
||||
func oneCard() catalog.Model {
|
||||
return testModel(1, catalog.Profile{Name: "one", GPUClass: "h100-80g", GPUsPerNode: 1})
|
||||
}
|
||||
|
||||
func TestAReplicaThatFailedIsTriedOnAnotherNode(t *testing.T) {
|
||||
// The same inputs give the same node every time: an on-demand 3D model
|
||||
// that failed to load was started on that node again, and again, until
|
||||
// it was parked, with another node free all along.
|
||||
h := newHarness(t, oneCard(), gpuNode("1", 1), gpuNode("2", 1))
|
||||
h.r.Reconcile(t.Context())
|
||||
first := h.disp.startedShards()[0].NodeID
|
||||
|
||||
if next := failAndRetry(t, h); next.NodeID == first {
|
||||
t.Errorf("tried on node %s again, where it had just failed", first)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReplicaThatFailedIsTriedOnAnotherCardOfTheNode(t *testing.T) {
|
||||
h := newHarness(t, oneCard(), gpuNode("1", 2))
|
||||
h.r.Reconcile(t.Context())
|
||||
first := h.disp.startedShards()[0].Spec.GPUIndices
|
||||
|
||||
next := failAndRetry(t, h)
|
||||
if len(next.Spec.GPUIndices) != 1 || len(first) != 1 || next.Spec.GPUIndices[0] == first[0] {
|
||||
t.Errorf("tried on card %v again, where it had just failed", next.Spec.GPUIndices)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithNowhereElseItIsTriedWhereItFailed(t *testing.T) {
|
||||
// A preference, not a rule: a model that has one place is not left
|
||||
// unplaced because it failed there once.
|
||||
h := newHarness(t, oneCard(), gpuNode("1", 1))
|
||||
h.r.Reconcile(t.Context())
|
||||
if next := failAndRetry(t, h); next.NodeID != "1" {
|
||||
t.Errorf("started on %q", next.NodeID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComingUpForgetsWhereItFailed(t *testing.T) {
|
||||
h := newHarness(t, oneCard(), gpuNode("1", 1), gpuNode("2", 1))
|
||||
h.r.Reconcile(t.Context())
|
||||
next := failAndRetry(t, h)
|
||||
h.r.OnShardStatus(next.ReplicaID, next.ShardID, supervisor.Ready, "")
|
||||
h.r.mu.Lock()
|
||||
left := len(h.r.failedAt["llama"])
|
||||
h.r.mu.Unlock()
|
||||
if left != 0 {
|
||||
t.Errorf("a model that came up still remembers %d failures' places", left)
|
||||
}
|
||||
}
|
||||
@@ -61,6 +61,7 @@ func (r *Reconciler) Retry(model string) bool {
|
||||
delete(r.parked, model)
|
||||
r.attempts[model] = 0
|
||||
delete(r.nextAttempt, model)
|
||||
delete(r.failedAt, model)
|
||||
r.mu.Unlock()
|
||||
r.Notify()
|
||||
return was
|
||||
|
||||
@@ -7,6 +7,8 @@ import (
|
||||
"log/slog"
|
||||
"math/rand/v2"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
@@ -251,6 +253,9 @@ type Reconciler struct {
|
||||
nextAttempt map[string]time.Time
|
||||
// attempts counts consecutive failures per model, for the backoff.
|
||||
attempts map[string]int
|
||||
// failedAt is where a model's replicas failed to start since it last
|
||||
// came up, most recent last: the next one is placed elsewhere if it can.
|
||||
failedAt map[string][]placement.Spot
|
||||
// lastExplain keeps the most recent placement failure per model, so
|
||||
// `ctl explain-placement` can answer without re-running placement.
|
||||
lastExplain map[string]placement.Result
|
||||
@@ -334,6 +339,7 @@ func New(cfg Config, cat *catalog.Live, inv *inventory.Store, disp Dispatcher, l
|
||||
replicas: map[string]*Replica{},
|
||||
nextAttempt: map[string]time.Time{},
|
||||
attempts: map[string]int{},
|
||||
failedAt: map[string][]placement.Spot{},
|
||||
reservedPorts: map[int]struct{}{},
|
||||
wanted: map[string]time.Time{},
|
||||
coldStart: map[string]time.Duration{},
|
||||
@@ -726,6 +732,7 @@ func (r *Reconciler) reapFailed(ctx context.Context) {
|
||||
// replica is about to disappear, and the backoff has to survive it.
|
||||
r.attempts[rep.Model]++
|
||||
r.nextAttempt[rep.Model] = r.now().Add(r.backoff(r.attempts[rep.Model]))
|
||||
r.noteFailedAt(rep)
|
||||
if n := r.attempts[rep.Model]; n >= r.cfg.ParkAfter {
|
||||
if _, already := r.parked[rep.Model]; !already {
|
||||
m, _ := r.cat.Load().Resolve(rep.Model)
|
||||
@@ -875,6 +882,7 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
delete(r.parked, m.Name)
|
||||
r.attempts[m.Name] = 0
|
||||
delete(r.nextAttempt, m.Name)
|
||||
delete(r.failedAt, m.Name)
|
||||
isParked, blockedUntil = false, time.Time{}
|
||||
r.log.Info("model unparked: its definition changed", "model", m.Name)
|
||||
}
|
||||
@@ -1002,7 +1010,29 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
|
||||
return notTried
|
||||
}
|
||||
}
|
||||
res := r.sched.Place(profiles, avoid)
|
||||
// Away from where it last failed to start, when there is anywhere else:
|
||||
// the same inputs give the same node and card, and an on-demand model
|
||||
// that failed there was otherwise started there again until it was
|
||||
// parked. Only a preference too: nowhere else, and it is tried there.
|
||||
r.mu.Lock()
|
||||
failed := slices.Clone(r.failedAt[m.Name])
|
||||
r.mu.Unlock()
|
||||
var res placement.Result
|
||||
if len(failed) > 0 {
|
||||
away := r.sched.Away(failed)
|
||||
res = away.Place(profiles, avoid)
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
res = away.Place(profiles, nil)
|
||||
}
|
||||
pspan.SetAttributes(attribute.Bool("llmgw.away_from_failure", res.Placement != nil))
|
||||
if res.Placement != nil {
|
||||
r.log.Info("placing a model away from where it failed to start", "model", m.Name,
|
||||
"failed_on", spotsText(failed), "now_on", spotsText(spotsOf(res.Placement)))
|
||||
}
|
||||
}
|
||||
if res.Placement == nil {
|
||||
res = r.sched.Place(profiles, avoid)
|
||||
}
|
||||
if res.Placement == nil && len(avoid) > 0 {
|
||||
// Spreading across failure domains is a preference, not a rule. If it
|
||||
// is the only thing standing in the way, take the placement anyway:
|
||||
@@ -1054,6 +1084,51 @@ func (r *Reconciler) scaleUp(ctx context.Context, m catalog.Model, avoid []strin
|
||||
return placed
|
||||
}
|
||||
|
||||
// maxFailedSpots is how many places a model's failures are remembered at.
|
||||
const maxFailedSpots = 8
|
||||
|
||||
// noteFailedAt remembers where a failed replica ran. Called under r.mu.
|
||||
func (r *Reconciler) noteFailedAt(rep *Replica) {
|
||||
spots := r.failedAt[rep.Model]
|
||||
for _, sh := range rep.Shards {
|
||||
sp := placement.Spot{Node: sh.NodeID, GPUs: slices.Clone(sh.GPUIndices)}
|
||||
spots = slices.DeleteFunc(spots, func(o placement.Spot) bool {
|
||||
return o.Node == sp.Node && slices.Equal(o.GPUs, sp.GPUs)
|
||||
})
|
||||
spots = append(spots, sp)
|
||||
}
|
||||
if len(spots) > maxFailedSpots {
|
||||
spots = spots[len(spots)-maxFailedSpots:]
|
||||
}
|
||||
r.failedAt[rep.Model] = spots
|
||||
}
|
||||
|
||||
// spotsOf is where a placement puts its shards.
|
||||
func spotsOf(pl *placement.Placement) []placement.Spot {
|
||||
var out []placement.Spot
|
||||
for _, n := range pl.Nodes {
|
||||
out = append(out, placement.Spot{Node: n.NodeID, GPUs: n.GPUIndices})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// spotsText is spots as a log reads them: "node01 gpu 0,1; node02 cpu".
|
||||
func spotsText(spots []placement.Spot) string {
|
||||
parts := make([]string, 0, len(spots))
|
||||
for _, sp := range spots {
|
||||
if len(sp.GPUs) == 0 {
|
||||
parts = append(parts, sp.Node+" cpu")
|
||||
continue
|
||||
}
|
||||
gs := make([]string, len(sp.GPUs))
|
||||
for i, g := range sp.GPUs {
|
||||
gs[i] = strconv.Itoa(g)
|
||||
}
|
||||
parts = append(parts, sp.Node+" gpu "+strings.Join(gs, ","))
|
||||
}
|
||||
return strings.Join(parts, "; ")
|
||||
}
|
||||
|
||||
// scaleOutcome is what one scaleUp did.
|
||||
type scaleOutcome int
|
||||
|
||||
@@ -1433,6 +1508,7 @@ func (r *Reconciler) markShard(replicaID, shardID string, st supervisor.State, e
|
||||
// counting from scratch rather than from an old streak.
|
||||
r.attempts[rep.Model] = 0
|
||||
delete(r.nextAttempt, rep.Model)
|
||||
delete(r.failedAt, rep.Model)
|
||||
if !rep.CreatedAt.IsZero() {
|
||||
r.coldStart[rep.Model] = r.now().Sub(rep.CreatedAt)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
package slurmsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// fakeKV is the store's cluster state.
|
||||
type fakeKV struct {
|
||||
m map[string]string
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeKV) PutClusterState(_ context.Context, k, v string) (bool, error) {
|
||||
if f.err != nil {
|
||||
return false, f.err
|
||||
}
|
||||
f.m[k] = v
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func (f *fakeKV) ClusterState(_ context.Context, k string) (string, error) {
|
||||
if f.err != nil {
|
||||
return "", f.err
|
||||
}
|
||||
v, ok := f.m[k]
|
||||
if !ok {
|
||||
return "", store.ErrNotFound
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
// takeOver is the leader that comes next: the same settings and store, an
|
||||
// inventory with only the nodes connected to it so far.
|
||||
func takeOver(r *rig, kv *fakeKV, connected ...string) *rig {
|
||||
n := &rig{set: r.set, inv: &fakeInv{nodes: map[string]*inventory.Node{}, allocs: map[string]int{}},
|
||||
agents: &fakeAgents{conn: connected, got: map[string][]*llmgwv1.Slurm{}}}
|
||||
for _, c := range connected {
|
||||
node := *r.inv.nodes[c]
|
||||
node.State = inventory.Ready
|
||||
n.inv.nodes[c] = &node
|
||||
}
|
||||
n.svc = New(func() config.SlurmSettings { return n.set }, n.inv, n.agents,
|
||||
func(context.Context) ([]byte, error) { return []byte(strings.Repeat("k", 1024)), nil }, nil)
|
||||
n.svc.SetMemory(StoreMemory(kv))
|
||||
return n
|
||||
}
|
||||
|
||||
func TestANewLeaderKeepsTheNodesItHasNotHeardFromInTheControllersFiles(t *testing.T) {
|
||||
kv := &fakeKV{m: map[string]string{}}
|
||||
r := newRig()
|
||||
r.svc.SetMemory(StoreMemory(kv))
|
||||
r.set.Lent = []string{"cuda01"}
|
||||
r.inv.allocs = map[string]int{}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "MIXED", Jobs: 1})
|
||||
r.tick()
|
||||
want := r.agents.last("node06").GetFiles()["slurm.conf"]
|
||||
if !strings.Contains(want, "NodeName=cuda01") || !strings.Contains(want, "Gres=gpu:rtx3090:1") {
|
||||
t.Fatalf("slurm.conf:\n%s", want)
|
||||
}
|
||||
|
||||
// The leader goes; the next hears from the controller first.
|
||||
n := takeOver(r, kv, "node06")
|
||||
n.tick()
|
||||
if got := n.agents.last("node06").GetFiles()["slurm.conf"]; got != want {
|
||||
t.Fatalf("the new leader's slurm.conf differs; the controller would drop the node and its job:\n%s\nwas:\n%s",
|
||||
got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeIsToldNothingUntilItsControllerIsKnown(t *testing.T) {
|
||||
r := newRig()
|
||||
r.svc.SetMemory(StoreMemory(&fakeKV{m: map[string]string{}}))
|
||||
delete(r.inv.nodes, "node06")
|
||||
r.agents.conn = []string{"cuda01"}
|
||||
r.tick()
|
||||
if m := r.agents.last("cuda01"); m != nil {
|
||||
t.Fatalf("a node was told %v before its controller was known; an empty part stops its slurmd and its jobs", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANewLeaderTellsANodeItsControllerFromTheStore(t *testing.T) {
|
||||
kv := &fakeKV{m: map[string]string{}}
|
||||
r := newRig()
|
||||
r.svc.SetMemory(StoreMemory(kv))
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
|
||||
r.tick()
|
||||
|
||||
n := takeOver(r, kv, "cuda01")
|
||||
n.tick()
|
||||
if m := n.agents.last("cuda01"); !m.GetNode() || m.GetConfServer() != "10.0.0.146:6817" {
|
||||
t.Fatalf("the node was told %v", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithTheStoreUnreadTheControllerIsSentNothingThatLeavesANodeOut(t *testing.T) {
|
||||
kv := &fakeKV{m: map[string]string{}, err: errors.New("store down")}
|
||||
r := newRig()
|
||||
r.svc.SetMemory(StoreMemory(kv))
|
||||
r.tick()
|
||||
if m := r.agents.last("node06"); m != nil {
|
||||
t.Fatalf("the controller was sent %v without cuda01, with the store unread", m)
|
||||
}
|
||||
// Once every node has reported, nothing is left out: sent.
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
|
||||
r.tick()
|
||||
if !strings.Contains(r.agents.last("node06").GetFiles()["slurm.conf"], "NodeName=cuda01") {
|
||||
t.Fatal("with every node reported, the controller still got no files")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithNothingKeptANewLeaderWaitsForTheNodesAWhile(t *testing.T) {
|
||||
r := newRig()
|
||||
r.svc.SetMemory(StoreMemory(&fakeKV{m: map[string]string{}}))
|
||||
r.tick()
|
||||
if m := r.agents.last("node06"); m != nil {
|
||||
t.Fatalf("with nothing kept, the controller was sent %v before cuda01 reported", m)
|
||||
}
|
||||
defer func(d time.Duration) { HoldFor = d }(HoldFor)
|
||||
HoldFor = 0
|
||||
r.tick()
|
||||
if m := r.agents.last("node06"); !m.GetController() || strings.Contains(m.GetFiles()["slurm.conf"], "NodeName=cuda01") {
|
||||
t.Fatalf("after the wait, the controller was told %v", m)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
package slurmsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"slices"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Memory keeps the nodes slurm.conf last held, and the controller's
|
||||
// address, where a leader that takes over finds them. Without it a new
|
||||
// leader's first slurm.conf has none of the nodes it has not heard from
|
||||
// yet, and the controller, restarted on it, drops them and the jobs
|
||||
// running there.
|
||||
type Memory interface {
|
||||
Recall(ctx context.Context) (Kept, error)
|
||||
Keep(ctx context.Context, k Kept) error
|
||||
}
|
||||
|
||||
// Kept is what Memory holds.
|
||||
type Kept struct {
|
||||
Controller string `json:"controller,omitempty"`
|
||||
ControllerAddr string `json:"controller_addr,omitempty"`
|
||||
Nodes map[string]slurm.Node `json:"nodes,omitempty"`
|
||||
}
|
||||
|
||||
func (k Kept) equal(o Kept) bool {
|
||||
if k.Controller != o.Controller || k.ControllerAddr != o.ControllerAddr || len(k.Nodes) != len(o.Nodes) {
|
||||
return false
|
||||
}
|
||||
for name, n := range k.Nodes {
|
||||
m, ok := o.Nodes[name]
|
||||
if !ok || n.Addr != m.Addr || n.Hardware != m.Hardware || !slices.Equal(n.GPUs, m.GPUs) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// KV is the store's cluster state.
|
||||
type KV interface {
|
||||
PutClusterState(ctx context.Context, key, value string) (bool, error)
|
||||
ClusterState(ctx context.Context, key string) (string, error)
|
||||
}
|
||||
|
||||
// keptKey is the cluster state key it is kept under.
|
||||
const keptKey = "slurm-nodes"
|
||||
|
||||
// StoreMemory keeps it in the store's cluster state.
|
||||
func StoreMemory(kv KV) Memory { return kvMemory{kv} }
|
||||
|
||||
type kvMemory struct{ kv KV }
|
||||
|
||||
func (m kvMemory) Recall(ctx context.Context) (Kept, error) {
|
||||
var k Kept
|
||||
v, err := m.kv.ClusterState(ctx, keptKey)
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
return k, nil
|
||||
}
|
||||
if err != nil {
|
||||
return k, err
|
||||
}
|
||||
err = json.Unmarshal([]byte(v), &k)
|
||||
return k, err
|
||||
}
|
||||
|
||||
func (m kvMemory) Keep(ctx context.Context, k Kept) error {
|
||||
b, err := json.Marshal(k)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = m.kv.PutClusterState(ctx, keptKey, string(b))
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,534 @@
|
||||
// Package slurmsvc is the leader's side of the Slurm cluster run beside
|
||||
// llmgw (BACKLOG P): it tells each node's agent what it is, writes the
|
||||
// controller's files from what the nodes report, and hands nodes between
|
||||
// llmgw and Slurm as the cluster's settings say.
|
||||
//
|
||||
// A node in slurm.lent is drained of llmgw's replicas, as `ctl drain` does,
|
||||
// and lent to Slurm once none is left. A node taken out of slurm.lent is
|
||||
// drained in Slurm, and is llmgw's again when its last job has ended. Both
|
||||
// are kept in force every tick, from the settings and what the agents say,
|
||||
// so a leader that takes over carries on where the last one was.
|
||||
package slurmsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"net"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/slurm"
|
||||
)
|
||||
|
||||
// Agents is how the service reaches the nodes.
|
||||
type Agents interface {
|
||||
Send(node string, msg *llmgwv1.ControlMessage) error
|
||||
ConnectedNodes() []string
|
||||
}
|
||||
|
||||
// Inventory is what the service reads and changes of llmgw's nodes.
|
||||
type Inventory interface {
|
||||
Get(id string) (inventory.Node, bool)
|
||||
SetState(id string, st inventory.NodeState) bool
|
||||
Allocations(nodeID string) []inventory.Allocation
|
||||
Nodes() []inventory.Node
|
||||
}
|
||||
|
||||
// Phase is where a node of slurm.nodes is between llmgw and Slurm.
|
||||
type Phase string
|
||||
|
||||
const (
|
||||
// Llmgw: drained in Slurm with no job left, llmgw's to place on.
|
||||
Llmgw Phase = "llmgw"
|
||||
// Lending: in slurm.lent, its replicas being moved off.
|
||||
Lending Phase = "lending"
|
||||
// Lent: Slurm's; llmgw places nothing there.
|
||||
Lent Phase = "lent"
|
||||
// Reclaiming: out of slurm.lent, drained in Slurm, jobs still running.
|
||||
Reclaiming Phase = "reclaiming"
|
||||
// Unknown: its agent has not said how it is in Slurm yet.
|
||||
Unknown Phase = "unknown"
|
||||
)
|
||||
|
||||
// NodeStatus is one node as the service sees it.
|
||||
type NodeStatus struct {
|
||||
Node string `json:"node"`
|
||||
Phase Phase `json:"phase"`
|
||||
Replicas int `json:"replicas"`
|
||||
Status *Reported `json:"slurm,omitempty"`
|
||||
Connected bool `json:"connected"`
|
||||
}
|
||||
|
||||
// Reported is the agent's SlurmStatus, for the admin API.
|
||||
type Reported struct {
|
||||
Version string `json:"version,omitempty"`
|
||||
Hardware string `json:"hardware,omitempty"`
|
||||
SlurmdRunning bool `json:"slurmd_running"`
|
||||
SlurmctldRunning bool `json:"slurmctld_running"`
|
||||
State string `json:"state,omitempty"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
Jobs int `json:"jobs"`
|
||||
Error string `json:"error,omitempty"`
|
||||
FilesSHA256 string `json:"files_sha256,omitempty"`
|
||||
At time.Time `json:"at"`
|
||||
}
|
||||
|
||||
// Status is the whole Slurm cluster, for `ctl slurm status`.
|
||||
type Status struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
Cluster string `json:"cluster"`
|
||||
Controller string `json:"controller"`
|
||||
Partition string `json:"partition"`
|
||||
Ctld *Reported `json:"controller_status,omitempty"`
|
||||
Nodes []NodeStatus `json:"nodes"`
|
||||
// Waiting are nodes left out of slurm.conf until they report their
|
||||
// hardware.
|
||||
Waiting []string `json:"waiting,omitempty"`
|
||||
}
|
||||
|
||||
// Service keeps the Slurm cluster.
|
||||
type Service struct {
|
||||
settings func() config.SlurmSettings
|
||||
inv Inventory
|
||||
agents Agents
|
||||
key func(ctx context.Context) ([]byte, error)
|
||||
log *slog.Logger
|
||||
|
||||
mu sync.Mutex
|
||||
// status is each node's last SlurmStatus.
|
||||
status map[string]*Reported
|
||||
// drained are nodes this service put in Draining, which it alone makes
|
||||
// ready again.
|
||||
drained map[string]bool
|
||||
// sent is what each node was last told, so it is told again only when
|
||||
// that changes or its stream is new.
|
||||
sent map[string]*llmgwv1.Slurm
|
||||
mungeKy []byte
|
||||
kick chan struct{}
|
||||
// mem keeps slurm.conf's nodes and the controller's address for the
|
||||
// next leader; known is what it held when read, and what was kept
|
||||
// since. loaded is false until it was read: until then the controller
|
||||
// is sent no files that leave out a node, which would drop that node
|
||||
// and its jobs.
|
||||
mem Memory
|
||||
known Kept
|
||||
loaded bool
|
||||
// started is the first pass, from which a node neither heard from nor
|
||||
// kept is waited for, for HoldFor.
|
||||
started time.Time
|
||||
}
|
||||
|
||||
// HoldFor is how long a new leader waits for a node it neither has heard
|
||||
// from nor finds kept before it writes the controller's files without it:
|
||||
// on a cluster that kept nothing yet (one upgraded to this), the nodes
|
||||
// report within seconds of a new leader, and the controller keeps the
|
||||
// node and its jobs meanwhile.
|
||||
var HoldFor = 2 * time.Minute
|
||||
|
||||
// New is a Service; key gives the munge key, made once from a cluster
|
||||
// secret.
|
||||
func New(settings func() config.SlurmSettings, inv Inventory, agents Agents,
|
||||
key func(ctx context.Context) ([]byte, error), log *slog.Logger) *Service {
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
return &Service{settings: settings, inv: inv, agents: agents, key: key, log: log,
|
||||
status: map[string]*Reported{}, drained: map[string]bool{}, sent: map[string]*llmgwv1.Slurm{},
|
||||
kick: make(chan struct{}, 1)}
|
||||
}
|
||||
|
||||
// SetMemory has slurm.conf's nodes kept in mem, and read from it first.
|
||||
func (s *Service) SetMemory(mem Memory) {
|
||||
s.mu.Lock()
|
||||
s.mem = mem
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
// current is the settings, slurm.all_nodes resolved to the nodes the
|
||||
// inventory has.
|
||||
func (s *Service) current() config.SlurmSettings {
|
||||
set := s.settings()
|
||||
if !set.AllNodes {
|
||||
return set
|
||||
}
|
||||
var ids []string
|
||||
for _, n := range s.inv.Nodes() {
|
||||
ids = append(ids, n.ID)
|
||||
}
|
||||
return set.WithNodes(ids)
|
||||
}
|
||||
|
||||
// Every is how often the service looks again without being asked.
|
||||
var Every = 10 * time.Second
|
||||
|
||||
// Run keeps the cluster until ctx ends.
|
||||
func (s *Service) Run(ctx context.Context) {
|
||||
t := time.NewTicker(Every)
|
||||
defer t.Stop()
|
||||
for {
|
||||
s.Tick(ctx)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
case <-s.kick:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Kick has the service look again now: the settings changed.
|
||||
func (s *Service) Kick() {
|
||||
select {
|
||||
case s.kick <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
// OnSlurmStatus records what a node's agent says.
|
||||
func (s *Service) OnSlurmStatus(node string, st *llmgwv1.SlurmStatus) {
|
||||
r := &Reported{Version: st.GetVersion(), Hardware: st.GetHardware(), SlurmdRunning: st.GetSlurmdRunning(),
|
||||
SlurmctldRunning: st.GetSlurmctldRunning(), State: st.GetState(), Reason: st.GetReason(),
|
||||
Jobs: int(st.GetJobs()), Error: st.GetError(), FilesSHA256: st.GetFilesSha256(), At: time.Now()}
|
||||
s.mu.Lock()
|
||||
prev := s.status[node]
|
||||
s.status[node] = r
|
||||
s.mu.Unlock()
|
||||
if r.Error != "" && (prev == nil || prev.Error != r.Error) {
|
||||
s.log.Warn("a node could not keep its part in Slurm", "node", node, "err", r.Error)
|
||||
}
|
||||
if prev == nil || prev.Hardware != r.Hardware || prev.State != r.State || prev.Jobs != r.Jobs {
|
||||
s.Kick()
|
||||
}
|
||||
}
|
||||
|
||||
// OnRegister is a node's new stream: it is told again what it is.
|
||||
func (s *Service) OnRegister(node string) {
|
||||
s.mu.Lock()
|
||||
delete(s.sent, node)
|
||||
s.mu.Unlock()
|
||||
s.Kick()
|
||||
}
|
||||
|
||||
// Tick does one pass: the nodes handed over as far as they can be, and
|
||||
// each told what it is.
|
||||
func (s *Service) Tick(ctx context.Context) {
|
||||
set := s.current()
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.recall(ctx)
|
||||
if s.started.IsZero() {
|
||||
s.started = time.Now()
|
||||
}
|
||||
phases := s.handOver(set)
|
||||
var key []byte
|
||||
if set.Enabled {
|
||||
if s.mungeKy == nil {
|
||||
k, err := s.key(ctx)
|
||||
if err != nil {
|
||||
s.log.Warn("no munge key for Slurm; nothing is sent to the nodes", "err", err)
|
||||
return
|
||||
}
|
||||
s.mungeKy = k
|
||||
}
|
||||
key = s.mungeKy
|
||||
}
|
||||
files, waiting := s.files(set)
|
||||
// Not yet knowing what the last leader knew, a slurm.conf without some
|
||||
// node would take it, and its jobs, from the controller: kept back.
|
||||
hold := s.mem != nil && len(waiting) > 0 && (!s.loaded || time.Since(s.started) < HoldFor)
|
||||
for _, node := range s.agents.ConnectedNodes() {
|
||||
if hold && node == set.Controller {
|
||||
continue
|
||||
}
|
||||
m := s.message(set, node, phases[node] == Lent, key, files)
|
||||
if m == nil {
|
||||
continue
|
||||
}
|
||||
if old, ok := s.sent[node]; ok && proto.Equal(old, m) {
|
||||
continue
|
||||
}
|
||||
if err := s.agents.Send(node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_Slurm{Slurm: m}}); err != nil {
|
||||
s.log.Warn("could not tell a node its part in Slurm", "node", node, "err", err)
|
||||
continue
|
||||
}
|
||||
s.sent[node] = m
|
||||
}
|
||||
s.remember(ctx, set)
|
||||
}
|
||||
|
||||
// recall reads what the last leader kept, once. Under s.mu.
|
||||
func (s *Service) recall(ctx context.Context) {
|
||||
if s.mem == nil || s.loaded {
|
||||
return
|
||||
}
|
||||
k, err := s.mem.Recall(ctx)
|
||||
if err != nil {
|
||||
s.log.Warn("could not read Slurm's nodes from the store; the controller's files wait for every node",
|
||||
"err", err)
|
||||
return
|
||||
}
|
||||
s.known, s.loaded = k, true
|
||||
}
|
||||
|
||||
// remember keeps slurm.conf's nodes and the controller's address when they
|
||||
// changed. Under s.mu.
|
||||
func (s *Service) remember(ctx context.Context, set config.SlurmSettings) {
|
||||
if s.mem == nil || !s.loaded || !set.Enabled {
|
||||
return
|
||||
}
|
||||
c, _ := s.cluster(set)
|
||||
now := Kept{Controller: set.Controller, ControllerAddr: c.ControllerAddr, Nodes: map[string]slurm.Node{}}
|
||||
for _, n := range c.Nodes {
|
||||
now.Nodes[n.Name] = n
|
||||
}
|
||||
if now.equal(s.known) {
|
||||
return
|
||||
}
|
||||
if err := s.mem.Keep(ctx, now); err != nil {
|
||||
s.log.Warn("could not keep Slurm's nodes in the store", "err", err)
|
||||
return
|
||||
}
|
||||
s.known = now
|
||||
}
|
||||
|
||||
// controllerAddr is where the nodes reach slurmctld: the controller's
|
||||
// management address, else the one kept. Under s.mu.
|
||||
func (s *Service) controllerAddr(set config.SlurmSettings) string {
|
||||
if n, ok := s.inv.Get(set.Controller); ok && n.MgmtAddr != "" {
|
||||
return n.MgmtAddr
|
||||
}
|
||||
if s.known.Controller == set.Controller {
|
||||
return s.known.ControllerAddr
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// handOver moves each node of slurm.nodes a step between llmgw and Slurm,
|
||||
// answering where each is. Under s.mu.
|
||||
func (s *Service) handOver(set config.SlurmSettings) map[string]Phase {
|
||||
out := map[string]Phase{}
|
||||
if !set.Enabled {
|
||||
// Off: every node is llmgw's again. What Slurm still runs is left
|
||||
// to finish where it is; turning Slurm off with work lent is the
|
||||
// operator's choice.
|
||||
for node := range s.drained {
|
||||
s.ready(node)
|
||||
}
|
||||
return out
|
||||
}
|
||||
for _, node := range set.Nodes {
|
||||
out[node] = s.phase(set, node)
|
||||
}
|
||||
// A node no longer in slurm.nodes is llmgw's.
|
||||
for node := range s.drained {
|
||||
if !slices.Contains(set.Nodes, node) {
|
||||
s.ready(node)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (s *Service) phase(set config.SlurmSettings, node string) Phase {
|
||||
n, ok := s.inv.Get(node)
|
||||
st := s.status[node]
|
||||
if set.IsLent(node) {
|
||||
if ok && (n.State == inventory.Ready || n.State == inventory.Cordoned) {
|
||||
s.inv.SetState(node, inventory.Draining)
|
||||
s.drained[node] = true
|
||||
s.log.Info("lending a node to Slurm: its replicas are moved off first", "node", node)
|
||||
}
|
||||
if !ok || len(s.inv.Allocations(node)) > 0 {
|
||||
return Lending
|
||||
}
|
||||
return Lent
|
||||
}
|
||||
// Not lent. Until the agent says Slurm has drained it with no job left,
|
||||
// llmgw places nothing new on it: a new leader starts with every node
|
||||
// ready. Cordoned, not drained: llmgw has nothing there to move.
|
||||
if st == nil || slurmBusy(st) {
|
||||
if ok && n.State == inventory.Ready {
|
||||
s.inv.SetState(node, inventory.Cordoned)
|
||||
s.drained[node] = true
|
||||
if st != nil {
|
||||
s.log.Info("taking a node back from Slurm: llmgw places nothing there until its jobs end",
|
||||
"node", node, "jobs", st.Jobs)
|
||||
}
|
||||
}
|
||||
if st == nil {
|
||||
return Unknown
|
||||
}
|
||||
return Reclaiming
|
||||
}
|
||||
if s.drained[node] {
|
||||
s.ready(node)
|
||||
}
|
||||
return Llmgw
|
||||
}
|
||||
|
||||
// slurmBusy is whether Slurm may still run work on a node: a job step left
|
||||
// on it (or no telling), or a controller that would still give it one. A
|
||||
// node the controller does not know of, or cannot be asked about, gets no
|
||||
// job from it, so a Slurm that is broken never keeps a node from llmgw.
|
||||
func slurmBusy(st *Reported) bool {
|
||||
if st.Jobs != 0 {
|
||||
return true
|
||||
}
|
||||
if st.State == "" {
|
||||
return false
|
||||
}
|
||||
return !strings.Contains(st.State, "DRAIN") && !strings.Contains(st.State, "DOWN")
|
||||
}
|
||||
|
||||
func (s *Service) ready(node string) {
|
||||
if n, ok := s.inv.Get(node); ok && (n.State == inventory.Draining || n.State == inventory.Cordoned) {
|
||||
s.inv.SetState(node, inventory.Ready)
|
||||
s.log.Info("a node is llmgw's again: Slurm has no job there", "node", node)
|
||||
}
|
||||
delete(s.drained, node)
|
||||
}
|
||||
|
||||
// files are the controller's files, and the nodes left out of them for not
|
||||
// having reported their hardware. Under s.mu.
|
||||
func (s *Service) files(set config.SlurmSettings) (map[string]string, []string) {
|
||||
c, waiting := s.cluster(set)
|
||||
if c.ControllerAddr == "" {
|
||||
return nil, waiting
|
||||
}
|
||||
return c.Files(), waiting
|
||||
}
|
||||
|
||||
// cluster is what slurm.conf says: each node as it last reported, else as
|
||||
// kept from before, and the nodes left out for having neither. Under s.mu.
|
||||
func (s *Service) cluster(set config.SlurmSettings) (slurm.Cluster, []string) {
|
||||
c := slurm.Cluster{Name: set.Cluster(), Controller: set.Controller, Partition: set.PartitionName()}
|
||||
if !set.Enabled {
|
||||
return c, nil
|
||||
}
|
||||
c.ControllerAddr = s.controllerAddr(set)
|
||||
var waiting []string
|
||||
for _, node := range set.Nodes {
|
||||
if sn, ok := s.reported(node); ok {
|
||||
c.Nodes = append(c.Nodes, sn)
|
||||
} else if sn, ok := s.known.Nodes[node]; ok {
|
||||
c.Nodes = append(c.Nodes, sn)
|
||||
} else {
|
||||
waiting = append(waiting, node)
|
||||
}
|
||||
}
|
||||
return c, waiting
|
||||
}
|
||||
|
||||
// reported is a node as its agent and the inventory say now. Under s.mu.
|
||||
func (s *Service) reported(node string) (slurm.Node, bool) {
|
||||
n, ok := s.inv.Get(node)
|
||||
st := s.status[node]
|
||||
if !ok || n.MgmtAddr == "" || st == nil || st.Hardware == "" {
|
||||
return slurm.Node{}, false
|
||||
}
|
||||
hw, err := slurm.ParseHardware(st.Hardware)
|
||||
if err != nil {
|
||||
return slurm.Node{}, false
|
||||
}
|
||||
sn := slurm.Node{Name: node, Addr: n.MgmtAddr, Hardware: hw}
|
||||
for _, g := range n.GPUs {
|
||||
sn.GPUs = append(sn.GPUs, slurm.GPU{Minor: g.Index, Name: g.Name})
|
||||
}
|
||||
return sn, true
|
||||
}
|
||||
|
||||
// message is what a node is told. Under s.mu.
|
||||
func (s *Service) message(set config.SlurmSettings, node string, lent bool, key []byte,
|
||||
files map[string]string) *llmgwv1.Slurm {
|
||||
m := &llmgwv1.Slurm{}
|
||||
if !set.Enabled {
|
||||
return m
|
||||
}
|
||||
isNode := slices.Contains(set.Nodes, node)
|
||||
isCtl := set.Controller == node && files != nil
|
||||
if !isNode && !isCtl {
|
||||
return m
|
||||
}
|
||||
m.Name, m.MungeKey, m.Lent = node, key, lent
|
||||
m.Node = isNode
|
||||
if addr := s.controllerAddr(set); addr != "" {
|
||||
m.ConfServer = net.JoinHostPort(addr, strconv.Itoa(slurm.CtldPort))
|
||||
}
|
||||
if isNode && m.ConfServer == "" {
|
||||
// Without its controller a node cannot start; it is told when the
|
||||
// controller registers. Told nothing until then, not "no part": a
|
||||
// node still running Slurm's jobs would stop them.
|
||||
return nil
|
||||
}
|
||||
if isCtl {
|
||||
m.Controller = true
|
||||
m.Files = maps.Clone(files)
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// Status is the cluster as the service sees it now.
|
||||
func (s *Service) Status() Status {
|
||||
set := s.current()
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
out := Status{Enabled: set.Enabled, Cluster: set.Cluster(), Controller: set.Controller,
|
||||
Partition: set.PartitionName(), Ctld: s.status[set.Controller]}
|
||||
if !set.Enabled {
|
||||
return out
|
||||
}
|
||||
_, out.Waiting = s.files(set)
|
||||
conn := s.agents.ConnectedNodes()
|
||||
for _, node := range set.Nodes {
|
||||
ns := NodeStatus{Node: node, Status: s.status[node], Replicas: len(s.inv.Allocations(node)),
|
||||
Connected: slices.Contains(conn, node)}
|
||||
ns.Phase = s.peek(set, node)
|
||||
out.Nodes = append(out.Nodes, ns)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// peek is phase without changing anything. Under s.mu.
|
||||
func (s *Service) peek(set config.SlurmSettings, node string) Phase {
|
||||
st := s.status[node]
|
||||
if set.IsLent(node) {
|
||||
if len(s.inv.Allocations(node)) > 0 {
|
||||
return Lending
|
||||
}
|
||||
return Lent
|
||||
}
|
||||
switch {
|
||||
case st == nil:
|
||||
return Unknown
|
||||
case slurmBusy(st):
|
||||
return Reclaiming
|
||||
}
|
||||
return Llmgw
|
||||
}
|
||||
|
||||
// Describe is one line for a log or a screen.
|
||||
func (n NodeStatus) Describe() string {
|
||||
switch n.Phase {
|
||||
case Lending:
|
||||
return fmt.Sprintf("being lent: %d replica(s) still to move off", n.Replicas)
|
||||
case Lent:
|
||||
return "lent to Slurm"
|
||||
case Reclaiming:
|
||||
if n.Status != nil {
|
||||
return fmt.Sprintf("being taken back: %d job(s) still running", n.Status.Jobs)
|
||||
}
|
||||
return "being taken back"
|
||||
case Llmgw:
|
||||
return "llmgw's"
|
||||
}
|
||||
return "not heard from yet"
|
||||
}
|
||||
@@ -0,0 +1,274 @@
|
||||
package slurmsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
type fakeInv struct {
|
||||
nodes map[string]*inventory.Node
|
||||
allocs map[string]int
|
||||
}
|
||||
|
||||
func (f *fakeInv) Get(id string) (inventory.Node, bool) {
|
||||
n, ok := f.nodes[id]
|
||||
if !ok {
|
||||
return inventory.Node{}, false
|
||||
}
|
||||
return *n, true
|
||||
}
|
||||
|
||||
func (f *fakeInv) SetState(id string, st inventory.NodeState) bool {
|
||||
n, ok := f.nodes[id]
|
||||
if ok {
|
||||
n.State = st
|
||||
}
|
||||
return ok
|
||||
}
|
||||
|
||||
func (f *fakeInv) Allocations(id string) []inventory.Allocation {
|
||||
return make([]inventory.Allocation, f.allocs[id])
|
||||
}
|
||||
|
||||
func (f *fakeInv) Nodes() []inventory.Node {
|
||||
var out []inventory.Node
|
||||
for _, n := range f.nodes {
|
||||
out = append(out, *n)
|
||||
}
|
||||
slices.SortFunc(out, func(a, b inventory.Node) int { return strings.Compare(a.ID, b.ID) })
|
||||
return out
|
||||
}
|
||||
|
||||
type fakeAgents struct {
|
||||
conn []string
|
||||
got map[string][]*llmgwv1.Slurm
|
||||
}
|
||||
|
||||
func (f *fakeAgents) Send(node string, m *llmgwv1.ControlMessage) error {
|
||||
f.got[node] = append(f.got[node], m.GetSlurm())
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeAgents) ConnectedNodes() []string { return f.conn }
|
||||
|
||||
func (f *fakeAgents) last(node string) *llmgwv1.Slurm {
|
||||
ms := f.got[node]
|
||||
if len(ms) == 0 {
|
||||
return nil
|
||||
}
|
||||
return ms[len(ms)-1]
|
||||
}
|
||||
|
||||
type rig struct {
|
||||
set config.SlurmSettings
|
||||
inv *fakeInv
|
||||
agents *fakeAgents
|
||||
svc *Service
|
||||
}
|
||||
|
||||
func newRig() *rig {
|
||||
r := &rig{
|
||||
set: config.SlurmSettings{Enabled: true, Controller: "node06", Nodes: []string{"cuda01"}},
|
||||
inv: &fakeInv{nodes: map[string]*inventory.Node{
|
||||
"node06": {ID: "node06", MgmtAddr: "10.0.0.146"},
|
||||
"cuda01": {ID: "cuda01", MgmtAddr: "10.0.0.147", GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA GeForce RTX 3090"}}},
|
||||
}, allocs: map[string]int{"cuda01": 2}},
|
||||
agents: &fakeAgents{conn: []string{"node06", "cuda01"}, got: map[string][]*llmgwv1.Slurm{}},
|
||||
}
|
||||
r.svc = New(func() config.SlurmSettings { return r.set }, r.inv, r.agents,
|
||||
func(context.Context) ([]byte, error) { return []byte(strings.Repeat("k", 1024)), nil }, nil)
|
||||
return r
|
||||
}
|
||||
|
||||
func (r *rig) report(node string, st *llmgwv1.SlurmStatus) { r.svc.OnSlurmStatus(node, st) }
|
||||
|
||||
func (r *rig) tick() { r.svc.Tick(context.Background()) }
|
||||
|
||||
const hw = "NodeName=cuda01 CPUs=16 RealMemory=64000"
|
||||
|
||||
func TestMessagesAndFiles(t *testing.T) {
|
||||
r := newRig()
|
||||
r.tick()
|
||||
ctl, node := r.agents.last("node06"), r.agents.last("cuda01")
|
||||
if !ctl.GetController() || ctl.GetNode() || len(ctl.GetMungeKey()) != 1024 {
|
||||
t.Fatalf("controller told %v", ctl)
|
||||
}
|
||||
if !node.GetNode() || node.GetController() || node.GetConfServer() != "10.0.0.146:6817" || node.GetLent() ||
|
||||
node.GetName() != "cuda01" || len(node.GetFiles()) != 0 {
|
||||
t.Fatalf("node told %v", node)
|
||||
}
|
||||
if strings.Contains(ctl.GetFiles()["slurm.conf"], "NodeName=cuda01") {
|
||||
t.Fatal("a node with no hardware reported is in slurm.conf")
|
||||
}
|
||||
if w := r.svc.Status().Waiting; len(w) != 1 || w[0] != "cuda01" {
|
||||
t.Fatalf("waiting %v", w)
|
||||
}
|
||||
// Same again: nobody is told twice.
|
||||
r.tick()
|
||||
if len(r.agents.got["node06"]) != 1 || len(r.agents.got["cuda01"]) != 1 {
|
||||
t.Fatal("told again with nothing changed")
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
|
||||
r.tick()
|
||||
conf := r.agents.last("node06").GetFiles()["slurm.conf"]
|
||||
if !strings.Contains(conf, "NodeName=cuda01 NodeAddr=10.0.0.147 CPUs=16 RealMemory=62080 Gres=gpu:rtx3090:1") {
|
||||
t.Fatalf("slurm.conf:\n%s", conf)
|
||||
}
|
||||
if !strings.Contains(r.agents.last("node06").GetFiles()["gres.conf"], "File=/dev/nvidia0") {
|
||||
t.Fatal("gres.conf has no card")
|
||||
}
|
||||
if len(r.agents.got["cuda01"]) != 1 {
|
||||
t.Fatal("a node was told again when only the controller's files changed")
|
||||
}
|
||||
// A new stream is told again.
|
||||
r.svc.OnRegister("cuda01")
|
||||
r.tick()
|
||||
if len(r.agents.got["cuda01"]) != 2 {
|
||||
t.Fatal("a new stream was not told")
|
||||
}
|
||||
if r.inv.nodes["cuda01"].State != inventory.Ready {
|
||||
t.Fatal("a node drained in Slurm with no job was taken from llmgw")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLendAndReclaim(t *testing.T) {
|
||||
r := newRig()
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: not lent yet"})
|
||||
r.set.Lent = []string{"cuda01"}
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State != inventory.Draining {
|
||||
t.Fatal("a lent node was not drained of its replicas")
|
||||
}
|
||||
if r.agents.last("cuda01").GetLent() {
|
||||
t.Fatal("lent to Slurm with replicas still on it")
|
||||
}
|
||||
if p := r.svc.Status().Nodes[0].Phase; p != Lending {
|
||||
t.Fatalf("phase %s", p)
|
||||
}
|
||||
r.inv.allocs["cuda01"] = 0
|
||||
r.tick()
|
||||
if !r.agents.last("cuda01").GetLent() || r.svc.Status().Nodes[0].Phase != Lent {
|
||||
t.Fatal("not lent once its replicas were gone")
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "ALLOCATED", Jobs: 3})
|
||||
|
||||
// Taken back: Slurm drains it; llmgw keeps off until its jobs end.
|
||||
r.set.Lent = nil
|
||||
r.tick()
|
||||
if r.agents.last("cuda01").GetLent() {
|
||||
t.Fatal("still lent")
|
||||
}
|
||||
if s := r.inv.nodes["cuda01"].State; s != inventory.Draining && s != inventory.Cordoned {
|
||||
t.Fatalf("llmgw may place on a node with Slurm jobs: %v", s)
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "ALLOCATED+DRAIN", Jobs: 1})
|
||||
r.tick()
|
||||
if r.svc.Status().Nodes[0].Phase != Reclaiming || r.inv.nodes["cuda01"].State == inventory.Ready {
|
||||
t.Fatal("given back with a job left")
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN", Reason: "llmgw: serving models"})
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State != inventory.Ready || r.svc.Status().Nodes[0].Phase != Llmgw {
|
||||
t.Fatal("not llmgw's when its last job ended")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewLeaderKeepsOffUntilHeard(t *testing.T) {
|
||||
r := newRig()
|
||||
r.inv.allocs["cuda01"] = 0
|
||||
r.tick()
|
||||
// Nothing heard yet: cordoned, nothing moved.
|
||||
if r.inv.nodes["cuda01"].State != inventory.Cordoned {
|
||||
t.Fatalf("state %v before the agent said anything", r.inv.nodes["cuda01"].State)
|
||||
}
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "MIXED", Jobs: 2})
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State == inventory.Ready {
|
||||
t.Fatal("ready with Slurm jobs on it")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBrokenSlurmLeavesTheNodeToLlmgw(t *testing.T) {
|
||||
r := newRig()
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Error: "installing Slurm: no package", Jobs: 0})
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State != inventory.Ready {
|
||||
t.Fatal("a Slurm that does not work took a node from llmgw")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOperatorsCordonIsLeft(t *testing.T) {
|
||||
r := newRig()
|
||||
r.inv.nodes["cuda01"].State = inventory.Cordoned
|
||||
r.report("cuda01", &llmgwv1.SlurmStatus{Hardware: hw, State: "IDLE+DRAIN"})
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State != inventory.Cordoned {
|
||||
t.Fatal("llmgw uncordoned a node an operator cordoned")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOffAndOut(t *testing.T) {
|
||||
r := newRig()
|
||||
r.inv.allocs["cuda01"] = 0
|
||||
r.set.Lent = []string{"cuda01"}
|
||||
r.tick()
|
||||
r.set.Enabled = false
|
||||
r.tick()
|
||||
if m := r.agents.last("cuda01"); m.GetNode() || m.GetController() {
|
||||
t.Fatalf("told %v with Slurm off", m)
|
||||
}
|
||||
if r.inv.nodes["cuda01"].State != inventory.Ready {
|
||||
t.Fatal("not llmgw's with Slurm off")
|
||||
}
|
||||
|
||||
r = newRig()
|
||||
r.inv.allocs["cuda01"] = 0
|
||||
r.set.Lent = []string{"cuda01"}
|
||||
r.tick()
|
||||
r.set.Nodes, r.set.Lent = nil, nil
|
||||
r.tick()
|
||||
if r.inv.nodes["cuda01"].State != inventory.Ready || r.agents.last("cuda01").GetNode() {
|
||||
t.Fatal("a node taken out of slurm.nodes is not llmgw's")
|
||||
}
|
||||
if !r.agents.last("node06").GetController() {
|
||||
t.Fatal("the controller stopped with no nodes")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoControllerYet(t *testing.T) {
|
||||
r := newRig()
|
||||
delete(r.inv.nodes, "node06")
|
||||
r.agents.conn = []string{"cuda01"}
|
||||
r.tick()
|
||||
if m := r.agents.last("cuda01"); m.GetNode() {
|
||||
t.Fatalf("a node was started with no controller to fetch its files from: %v", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAllNodes(t *testing.T) {
|
||||
r := newRig()
|
||||
r.set.Nodes, r.set.AllNodes = nil, true
|
||||
r.inv.nodes["cuda02"] = &inventory.Node{ID: "cuda02", MgmtAddr: "10.0.0.150"}
|
||||
r.agents.conn = append(r.agents.conn, "cuda02")
|
||||
r.tick()
|
||||
for _, n := range []string{"cuda01", "cuda02", "node06"} {
|
||||
if !r.agents.last(n).GetNode() {
|
||||
t.Fatalf("%s, a node that joined, does not run slurmd", n)
|
||||
}
|
||||
}
|
||||
if got := len(r.svc.Status().Nodes); got != 3 {
|
||||
t.Fatalf("%d nodes in the status", got)
|
||||
}
|
||||
r.set.Lent = []string{"cuda02"}
|
||||
r.tick()
|
||||
if p := r.svc.Status().Nodes[1]; p.Node != "cuda02" || p.Phase != Lent {
|
||||
t.Fatalf("cuda02 not lent: %+v", p)
|
||||
}
|
||||
}
|
||||
@@ -94,7 +94,7 @@ type Store interface {
|
||||
SetStaffRole(ctx context.Context, id, role string) error
|
||||
SetUserState(ctx context.Context, id string, state store.UserState) error
|
||||
DeleteUserSessions(ctx context.Context, userID string) error
|
||||
AddMembership(ctx context.Context, user string, scope store.Scope, id, role, by string, at time.Time) error
|
||||
EnsureMembership(ctx context.Context, user string, scope store.Scope, id, by string, at time.Time) error
|
||||
RemoveMembership(ctx context.Context, userID string, scope store.Scope, scopeID string, at time.Time) (int, error)
|
||||
MembershipsOf(ctx context.Context, userID string) ([]store.Membership, error)
|
||||
AllOrgs(ctx context.Context) ([]store.Org, error)
|
||||
@@ -221,7 +221,7 @@ func (k *Keeper) apply(ctx context.Context, u *store.User, id Identity, remove b
|
||||
continue // a mapping to a place the tree lacks: said by Sync
|
||||
}
|
||||
want[string(scope)+"/"+sid] = scope
|
||||
if err := k.St.AddMembership(ctx, u.ID, scope, sid, store.RoleMember, k.Name, k.now()); err != nil {
|
||||
if err := k.St.EnsureMembership(ctx, u.ID, scope, sid, k.Name, k.now()); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
@@ -480,6 +480,7 @@ var Reference = []Block{
|
||||
{Key: "mail", Type: "mail.Config", Doc: "Mail is where the gateway's few emails go out from: a link to set a\nforgotten password. Unset, none is sent and people ask an operator.", Nested: "mail.Config"},
|
||||
{Key: "serving", Type: "ServingSettings", Doc: "Serving stops and starts the whole cluster's service (llmgw stop, llmgw\nstart).", Nested: "config.ServingSettings"},
|
||||
{Key: "firewall", Type: "FirewallSettings", Doc: "Firewall turns llmgw's firewall on every machine on or off.", Nested: "config.FirewallSettings"},
|
||||
{Key: "slurm", Type: "SlurmSettings", Doc: "Slurm is a Slurm cluster run beside llmgw on some of its nodes, and\nwhich of them it may use now.", Nested: "config.SlurmSettings"},
|
||||
{Key: "presets", Type: "PresetSettings", Doc: "Presets is how the models that come with llmgw reach the catalog.", Nested: "config.PresetSettings"},
|
||||
{Key: "on_demand", Type: "OnDemandSettings", Doc: "OnDemand is how models that load when asked for take turns on the\ncards.", Nested: "config.OnDemandSettings"},
|
||||
{Key: "agents", Type: "AgentSettings", Doc: "Agents are the nodes' agent settings: agent.yaml's keys but those an\nagent needs to start, for every node (default) and for one\n(nodes.<id>). Sent to the agents as they change; an agent whose\nsettings changed starts again on them, its engines serving\nthroughout.", Nested: "config.AgentSettings"},
|
||||
@@ -512,6 +513,15 @@ var Reference = []Block{
|
||||
{Package: "engines", Name: "SimConfig", Title: "SimConfig", Doc: "", Fields: []Field{
|
||||
{Key: "bin", Type: "string", Doc: "Bin is the llmgw binary itself; the engine is one of its subcommands.\nEmpty means whatever started this agent, which is right almost always.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "SlurmSettings", Title: "SlurmSettings", Doc: "", Fields: []Field{
|
||||
{Key: "enabled", Type: "bool", Doc: "Enabled runs Slurm on the nodes below. False stops llmgw keeping it;\nwhat was installed is left installed, and every node is llmgw's.", Nested: ""},
|
||||
{Key: "cluster_name", Type: "string", Doc: "ClusterName is Slurm's ClusterName. Empty is \"llmgw\".", Nested: ""},
|
||||
{Key: "controller", Type: "string", Doc: "Controller is the llmgw node whose agent runs slurmctld. It needs no\nGPU, and need not be one of Nodes.", Nested: ""},
|
||||
{Key: "nodes", Type: "[]string", Doc: "Nodes are the llmgw nodes that run slurmd. Each stays llmgw's, drained\nin Slurm, until it is also in Lent.", Nested: ""},
|
||||
{Key: "all_nodes", Type: "bool", Doc: "AllNodes runs slurmd on every llmgw node, those that join later too,\nbesides any named in Nodes. Each is llmgw's until it is lent, as any.", Nested: ""},
|
||||
{Key: "lent", Type: "[]string", Doc: "Lent are the nodes of Nodes Slurm may run jobs on now. A node put here\nis drained of llmgw's replicas first, as `ctl drain` does, and resumed\nin Slurm once none is left; a node taken out is drained in Slurm and\ngiven back to llmgw when its last job ends.", Nested: ""},
|
||||
{Key: "partition", Type: "string", Doc: "Partition is the one partition every node is in. Empty is \"gpu\".", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "StrataConfig", Title: "StrataConfig", Doc: "", Fields: []Field{
|
||||
{Key: "dir", Type: "string", Doc: "Dir is Strata's source directory, where setup.py and serve/ are.", Nested: ""},
|
||||
{Key: "python", Type: "string", Doc: "Python is the interpreter of its environment; Dir's .venv when unset.", Nested: ""},
|
||||
|
||||
@@ -41,6 +41,7 @@ var Sources = []string{
|
||||
"internal/config/serving.go",
|
||||
"internal/config/presets.go",
|
||||
"internal/config/ondemand.go",
|
||||
"internal/config/slurm.go",
|
||||
"internal/config/agents.go",
|
||||
"internal/mail/mail.go",
|
||||
"internal/notify/notify.go",
|
||||
|
||||
@@ -173,6 +173,29 @@ func (o Option) WithContext(kvPerToken int64, tokens int) Option {
|
||||
return o
|
||||
}
|
||||
|
||||
// MostContext is the longest context, up to trained, this option still holds
|
||||
// for one request, in whole thousands of tokens below trained: trained when
|
||||
// the KV cache's size is unknown, 0 when trained is, for Strata (its context
|
||||
// is its own), or when not even 1024 tokens fit.
|
||||
func (o Option) MostContext(kvPerToken int64, trained int) int {
|
||||
switch {
|
||||
case trained <= 0 || o.Engine == "strata":
|
||||
return 0
|
||||
case kvPerToken <= 0 || o.WithContext(kvPerToken, trained).Fits():
|
||||
return trained
|
||||
}
|
||||
lo, hi := 0, trained/1024+1 // lo fits (or is none), hi does not
|
||||
for hi-lo > 1 {
|
||||
mid := (lo + hi) / 2
|
||||
if o.WithContext(kvPerToken, mid*1024).Fits() {
|
||||
lo = mid
|
||||
} else {
|
||||
hi = mid
|
||||
}
|
||||
}
|
||||
return lo * 1024
|
||||
}
|
||||
|
||||
// MemoryText says what it needs, part by part.
|
||||
func (o Option) MemoryText() string {
|
||||
if o.Engine == "strata" {
|
||||
|
||||
@@ -193,3 +193,23 @@ func TestAnOptionSaysWhatItNeedsOfEachCardAndOfAll(t *testing.T) {
|
||||
t.Errorf("a 2M-token context fits four cards: %s", huge.MemoryText())
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheMostContextAnOptionHolds(t *testing.T) {
|
||||
// 4 GB of weights on a 24 GB card, a token's cache 1 MiB.
|
||||
o := Option{Engine: "llamacpp", Class: "6000-24g", GPUsPerNode: 1, Nodes: 1,
|
||||
WeightsGB: 4, EngineGB: 1, NeedGB: 5, HaveGB: 21.6}
|
||||
const kv = 1 << 20
|
||||
if got := o.MostContext(kv, 8192); got != 8192 {
|
||||
t.Errorf("a context that fits whole: %d, want what it was trained for", got)
|
||||
}
|
||||
got := o.MostContext(kv, 131072)
|
||||
if got <= 0 || got%1024 != 0 || !o.WithContext(kv, got).Fits() || o.WithContext(kv, got+1024).Fits() {
|
||||
t.Errorf("the most that fits is %d", got)
|
||||
}
|
||||
if got := o.MostContext(0, 32768); got != 32768 {
|
||||
t.Errorf("an unknown cache size: %d, want what it was trained for", got)
|
||||
}
|
||||
if got := (Option{Engine: "strata"}).MostContext(kv, 32768); got != 0 {
|
||||
t.Errorf("Strata's context is its own: %d", got)
|
||||
}
|
||||
}
|
||||
@@ -39,7 +39,7 @@ type Store interface {
|
||||
SetUserState(ctx context.Context, id string, state store.UserState) error
|
||||
DeleteUserSessions(ctx context.Context, userID string) error
|
||||
SetStaffRole(ctx context.Context, id, role string) error
|
||||
AddMembership(ctx context.Context, user string, scope store.Scope, id, role, by string, at time.Time) error
|
||||
EnsureMembership(ctx context.Context, user string, scope store.Scope, id, by string, at time.Time) error
|
||||
RemoveMembership(ctx context.Context, user string, scope store.Scope, id string, at time.Time) (int, error)
|
||||
AllOrgs(ctx context.Context) ([]store.Org, error)
|
||||
AllTeams(ctx context.Context) ([]store.TeamPath, error)
|
||||
@@ -548,7 +548,7 @@ func (s *Server) member(ctx context.Context, group, userID string, joined bool)
|
||||
return
|
||||
}
|
||||
if joined {
|
||||
err = s.st.AddMembership(ctx, u.ID, scope, id, store.RoleMember, "scim", time.Now())
|
||||
err = s.st.EnsureMembership(ctx, u.ID, scope, id, "scim", time.Now())
|
||||
} else {
|
||||
_, err = s.st.RemoveMembership(ctx, u.ID, scope, id, time.Now())
|
||||
}
|
||||
|
||||
@@ -58,7 +58,7 @@ func (m *mem) DeleteUserSessions(_ context.Context, id string) error {
|
||||
return nil
|
||||
}
|
||||
func (m *mem) SetStaffRole(_ context.Context, id, role string) error { m.roles[id] = role; return nil }
|
||||
func (m *mem) AddMembership(_ context.Context, u string, sc store.Scope, id, _, _ string, _ time.Time) error {
|
||||
func (m *mem) EnsureMembership(_ context.Context, u string, sc store.Scope, id, _ string, _ time.Time) error {
|
||||
m.members[u+"@"+string(sc)+":"+id] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,344 @@
|
||||
// Package slurm is what llmgw knows of Slurm (BACKLOG P): the files a
|
||||
// controller runs on, written from what llmgw knows of its nodes, the munge
|
||||
// key every machine shares, and reading what slurmd and scontrol say.
|
||||
//
|
||||
// Nothing here runs anything. The agent installs and starts the daemons; the
|
||||
// leader decides what each node is and writes the files with this package.
|
||||
package slurm
|
||||
|
||||
import (
|
||||
"crypto/hkdf"
|
||||
"crypto/sha256"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Paths are where the files are on every distribution llmgw installs on:
|
||||
// Debian and Ubuntu since Slurm 20.11, and EPEL.
|
||||
const (
|
||||
ConfDir = "/etc/slurm"
|
||||
MungeKeyPath = "/etc/munge/munge.key"
|
||||
StateDir = "/var/spool/slurmctld"
|
||||
SpoolDir = "/var/spool/slurmd"
|
||||
LogDir = "/var/log/slurm"
|
||||
// CtldPort is slurmctld's, which configless slurmd fetch their files
|
||||
// from.
|
||||
CtldPort = 6817
|
||||
// SlurmdPort is slurmd's, which the controller and srun reach.
|
||||
SlurmdPort = 6818
|
||||
// SrunPortsFrom and SrunPortsTo are where srun listens for the steps it
|
||||
// starts: a range of its own, so a firewall can let it in (Slurm's own
|
||||
// example range). Without one srun takes any ephemeral port.
|
||||
SrunPortsFrom = 60001
|
||||
SrunPortsTo = 63000
|
||||
)
|
||||
|
||||
// ReasonNew is why a node new to slurm.conf starts drained. Every reason
|
||||
// llmgw drains with starts with "llmgw", and only those are resumed by it.
|
||||
const ReasonNew = "llmgw: not lent yet"
|
||||
|
||||
// MinVersion is the oldest Slurm llmgw runs: the first to put jobs in
|
||||
// cgroup v2, which every distribution llmgw supports now mounts alone.
|
||||
var MinVersion = [2]int{22, 5}
|
||||
|
||||
// MungeKeyLen is the length munge's own tool writes.
|
||||
const MungeKeyLen = 1024
|
||||
|
||||
// MungeKey is the key every machine of the Slurm cluster shares, derived
|
||||
// from a cluster secret, so a control plane that takes over writes the same
|
||||
// one and nobody copies it by hand.
|
||||
func MungeKey(secret string) ([]byte, error) {
|
||||
if len(secret) < 32 {
|
||||
return nil, fmt.Errorf("the secret a munge key is made from is too short")
|
||||
}
|
||||
return hkdf.Key(sha256.New, []byte(secret), nil, "llmgw slurm munge key v1", MungeKeyLen)
|
||||
}
|
||||
|
||||
// Hardware is a node's line from `slurmd -C`: how slurmd counts it, which is
|
||||
// what slurm.conf must say or the controller refuses the node.
|
||||
type Hardware struct {
|
||||
CPUs int
|
||||
Boards int
|
||||
SocketsPerBoard int
|
||||
CoresPerSocket int
|
||||
ThreadsPerCore int
|
||||
// RealMemory is in MiB.
|
||||
RealMemory int64
|
||||
}
|
||||
|
||||
// ParseHardware reads `slurmd -C`.
|
||||
func ParseHardware(out string) (Hardware, error) {
|
||||
var h Hardware
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if !strings.HasPrefix(strings.TrimSpace(line), "NodeName=") {
|
||||
continue
|
||||
}
|
||||
for _, f := range strings.Fields(line) {
|
||||
k, v, _ := strings.Cut(f, "=")
|
||||
n, err := strconv.ParseInt(v, 10, 64)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
switch k {
|
||||
case "CPUs":
|
||||
h.CPUs = int(n)
|
||||
case "Boards":
|
||||
h.Boards = int(n)
|
||||
case "SocketsPerBoard":
|
||||
h.SocketsPerBoard = int(n)
|
||||
case "CoresPerSocket":
|
||||
h.CoresPerSocket = int(n)
|
||||
case "ThreadsPerCore":
|
||||
h.ThreadsPerCore = int(n)
|
||||
case "RealMemory":
|
||||
h.RealMemory = n
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
if h.CPUs <= 0 || h.RealMemory <= 0 {
|
||||
return Hardware{}, fmt.Errorf("slurmd -C said no CPUs or memory: %q", strings.TrimSpace(out))
|
||||
}
|
||||
return h, nil
|
||||
}
|
||||
|
||||
// GPU is one of a node's cards.
|
||||
type GPU struct {
|
||||
// Minor is N of /dev/nvidiaN.
|
||||
Minor int
|
||||
// Name is the driver's, as nvidia-smi gives it.
|
||||
Name string
|
||||
}
|
||||
|
||||
// Node is one machine slurmd runs on.
|
||||
type Node struct {
|
||||
Name string
|
||||
// Addr is where slurmctld reaches it: its llmgw management address.
|
||||
Addr string
|
||||
Hardware
|
||||
GPUs []GPU
|
||||
}
|
||||
|
||||
// Cluster is everything slurm.conf says.
|
||||
type Cluster struct {
|
||||
Name string
|
||||
Controller string
|
||||
ControllerAddr string
|
||||
Partition string
|
||||
// Nodes without hardware (not reported yet) are left out of the files
|
||||
// until they are.
|
||||
Nodes []Node
|
||||
}
|
||||
|
||||
// gpuModel is a word that names a card's model: A100, H100, V100, T4, L40S.
|
||||
var gpuModel = regexp.MustCompile(`^[a-z]{1,2}\d{1,4}[a-z]?$`)
|
||||
|
||||
// GPUType is a card's name as a Slurm GRES type: "NVIDIA A100-SXM4-80GB" is
|
||||
// a100, "NVIDIA GeForce RTX 3090" rtx3090, "Tesla T4" t4. A name it cannot
|
||||
// read is "gpu".
|
||||
func GPUType(name string) string {
|
||||
words := strings.Fields(strings.ToLower(strings.NewReplacer("-", " ", "_", " ").Replace(name)))
|
||||
for i, w := range words {
|
||||
if (w == "rtx" || w == "gtx") && i+1 < len(words) {
|
||||
return w + words[i+1]
|
||||
}
|
||||
if gpuModel.MatchString(w) {
|
||||
return w
|
||||
}
|
||||
}
|
||||
return "gpu"
|
||||
}
|
||||
|
||||
// gpuGroups are a node's cards by type, the types sorted.
|
||||
func gpuGroups(gs []GPU) ([]string, map[string][]int) {
|
||||
by := map[string][]int{}
|
||||
for _, g := range gs {
|
||||
t := GPUType(g.Name)
|
||||
by[t] = append(by[t], g.Minor)
|
||||
}
|
||||
types := make([]string, 0, len(by))
|
||||
for t, ms := range by {
|
||||
slices.Sort(ms)
|
||||
types = append(types, t)
|
||||
}
|
||||
slices.Sort(types)
|
||||
return types, by
|
||||
}
|
||||
|
||||
// memoryKept is what is left out of a node's RealMemory: slurmd -C reads
|
||||
// what the kernel says now, which is a little less after a reboot, and a
|
||||
// node whose memory is below what slurm.conf says is refused.
|
||||
const memoryKept = 0.97
|
||||
|
||||
func (c Cluster) ready() []Node {
|
||||
var out []Node
|
||||
for _, n := range c.Nodes {
|
||||
if n.CPUs > 0 && n.RealMemory > 0 {
|
||||
out = append(out, n)
|
||||
}
|
||||
}
|
||||
slices.SortFunc(out, func(a, b Node) int { return strings.Compare(a.Name, b.Name) })
|
||||
return out
|
||||
}
|
||||
|
||||
const header = "# Written by llmgw (BACKLOG P), on the node that runs slurmctld; the nodes read it\n" +
|
||||
"# from there. Change it through the cluster's settings (slurm.*), not here.\n"
|
||||
|
||||
// SlurmConf is slurm.conf.
|
||||
func (c Cluster) SlurmConf() string {
|
||||
var b strings.Builder
|
||||
b.WriteString(header)
|
||||
fmt.Fprintf(&b, "ClusterName=%s\n", c.Name)
|
||||
fmt.Fprintf(&b, "SlurmctldHost=%s(%s)\n", c.Controller, c.ControllerAddr)
|
||||
b.WriteString(`SlurmctldParameters=enable_configless
|
||||
SlurmctldPort=6817
|
||||
SlurmdPort=6818
|
||||
SrunPortRange=60001-63000
|
||||
AuthType=auth/munge
|
||||
CredType=cred/munge
|
||||
SlurmUser=slurm
|
||||
SlurmdUser=root
|
||||
StateSaveLocation=` + StateDir + `
|
||||
SlurmdSpoolDir=` + SpoolDir + `
|
||||
SlurmctldLogFile=` + LogDir + `/slurmctld.log
|
||||
SlurmdLogFile=` + LogDir + `/slurmd.log
|
||||
SlurmctldPidFile=/run/slurmctld.pid
|
||||
SlurmdPidFile=/run/slurmd.pid
|
||||
ProctrackType=proctrack/cgroup
|
||||
TaskPlugin=task/cgroup,task/affinity
|
||||
# Nothing gathered for accounting until there is somewhere to keep it
|
||||
# (slurmdbd, later); gathered and not kept is an error in every log.
|
||||
JobAcctGatherType=jobacct_gather/none
|
||||
SelectType=select/cons_tres
|
||||
SelectTypeParameters=CR_Core_Memory
|
||||
SchedulerType=sched/backfill
|
||||
GresTypes=gpu
|
||||
MpiDefault=none
|
||||
# No mail sent; without it every start says MailProg is invalid.
|
||||
MailProg=/bin/true
|
||||
# Without accounting storage, 23.11's main scheduler marks a batch job
|
||||
# InvalidAccount; backfill starts it on its next pass, so it is kept short.
|
||||
SchedulerParameters=bf_interval=10
|
||||
# A node that went down and comes back is taken again; one llmgw drained
|
||||
# stays drained until llmgw resumes it.
|
||||
ReturnToService=2
|
||||
SlurmctldTimeout=120
|
||||
SlurmdTimeout=300
|
||||
`)
|
||||
nodes := c.ready()
|
||||
names := make([]string, 0, len(nodes))
|
||||
for _, n := range nodes {
|
||||
names = append(names, n.Name)
|
||||
fmt.Fprintf(&b, "NodeName=%s NodeAddr=%s CPUs=%d", n.Name, n.Addr, n.CPUs)
|
||||
if n.Boards > 0 && n.SocketsPerBoard > 0 && n.CoresPerSocket > 0 && n.ThreadsPerCore > 0 {
|
||||
fmt.Fprintf(&b, " Boards=%d SocketsPerBoard=%d CoresPerSocket=%d ThreadsPerCore=%d",
|
||||
n.Boards, n.SocketsPerBoard, n.CoresPerSocket, n.ThreadsPerCore)
|
||||
}
|
||||
fmt.Fprintf(&b, " RealMemory=%d", int64(float64(n.RealMemory)*memoryKept))
|
||||
if types, by := gpuGroups(n.GPUs); len(types) > 0 {
|
||||
gres := make([]string, 0, len(types))
|
||||
for _, t := range types {
|
||||
gres = append(gres, fmt.Sprintf("gpu:%s:%d", t, len(by[t])))
|
||||
}
|
||||
fmt.Fprintf(&b, " Gres=%s", strings.Join(gres, ","))
|
||||
}
|
||||
// Drained until llmgw lends it: a node new to the controller would
|
||||
// otherwise take jobs in the minute before its agent drains it.
|
||||
fmt.Fprintf(&b, " State=DRAIN Reason=%q\n", ReasonNew)
|
||||
}
|
||||
if len(names) > 0 {
|
||||
fmt.Fprintf(&b, "PartitionName=%s Nodes=%s Default=YES MaxTime=INFINITE State=UP\n",
|
||||
c.Partition, strings.Join(names, ","))
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// GresConf is gres.conf: each node's cards by device file, so a job is given
|
||||
// the cards it asked for and kept off the others.
|
||||
func (c Cluster) GresConf() string {
|
||||
var b strings.Builder
|
||||
b.WriteString(header)
|
||||
for _, n := range c.ready() {
|
||||
types, by := gpuGroups(n.GPUs)
|
||||
for _, t := range types {
|
||||
files := make([]string, 0, len(by[t]))
|
||||
for _, m := range by[t] {
|
||||
files = append(files, fmt.Sprintf("/dev/nvidia%d", m))
|
||||
}
|
||||
fmt.Fprintf(&b, "NodeName=%s Name=gpu Type=%s File=%s\n", n.Name, t, strings.Join(files, ","))
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// CgroupConf is cgroup.conf: a job kept to its cores, memory and cards.
|
||||
func (c Cluster) CgroupConf() string {
|
||||
return header + `CgroupPlugin=autodetect
|
||||
ConstrainCores=yes
|
||||
ConstrainDevices=yes
|
||||
ConstrainRAMSpace=yes
|
||||
`
|
||||
}
|
||||
|
||||
// Files are the controller's files, by name in ConfDir.
|
||||
func (c Cluster) Files() map[string]string {
|
||||
return map[string]string{"slurm.conf": c.SlurmConf(), "gres.conf": c.GresConf(), "cgroup.conf": c.CgroupConf()}
|
||||
}
|
||||
|
||||
// NodeState is what `scontrol show node -o NAME` says of a node.
|
||||
type NodeState struct {
|
||||
// State is Slurm's, with its flags: IDLE, ALLOCATED, MIXED+DRAIN,
|
||||
// DOWN+NOT_RESPONDING.
|
||||
State string
|
||||
Reason string
|
||||
}
|
||||
|
||||
// Drained is whether the controller gives the node nothing new.
|
||||
func (s NodeState) Drained() bool {
|
||||
return strings.Contains(s.State, "DRAIN") || strings.Contains(s.State, "DOWN")
|
||||
}
|
||||
|
||||
// ParseNodeState reads `scontrol show node -o NAME`, one line of key=value.
|
||||
func ParseNodeState(out string) (NodeState, error) {
|
||||
line := strings.TrimSpace(out)
|
||||
if !strings.HasPrefix(line, "NodeName=") {
|
||||
return NodeState{}, fmt.Errorf("scontrol said %q", line)
|
||||
}
|
||||
var s NodeState
|
||||
if i := strings.Index(line, " State="); i >= 0 {
|
||||
s.State, _, _ = strings.Cut(line[i+len(" State="):], " ")
|
||||
}
|
||||
// Reason is last but for its [user@time] and may have spaces.
|
||||
if i := strings.Index(line, " Reason="); i >= 0 {
|
||||
r := line[i+len(" Reason="):]
|
||||
if j := strings.LastIndex(r, " ["); j >= 0 {
|
||||
r = r[:j]
|
||||
}
|
||||
s.Reason = strings.TrimSpace(r)
|
||||
}
|
||||
return s, nil
|
||||
}
|
||||
|
||||
// ParseVersion reads "slurm 23.11.4" or "slurm-wlm 21.08.5" as major, minor.
|
||||
func ParseVersion(out string) ([2]int, error) {
|
||||
for _, f := range strings.Fields(out) {
|
||||
parts := strings.Split(f, ".")
|
||||
if len(parts) < 2 {
|
||||
continue
|
||||
}
|
||||
major, err1 := strconv.Atoi(parts[0])
|
||||
minor, err2 := strconv.Atoi(parts[1])
|
||||
if err1 == nil && err2 == nil {
|
||||
return [2]int{major, minor}, nil
|
||||
}
|
||||
}
|
||||
return [2]int{}, fmt.Errorf("no version in %q", strings.TrimSpace(out))
|
||||
}
|
||||
|
||||
// TooOld is whether v is older than MinVersion.
|
||||
func TooOld(v [2]int) bool {
|
||||
return v[0] < MinVersion[0] || (v[0] == MinVersion[0] && v[1] < MinVersion[1])
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
package slurm
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestMungeKey(t *testing.T) {
|
||||
s := strings.Repeat("ab", 32)
|
||||
a, err := MungeKey(s)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := MungeKey(s)
|
||||
if len(a) != MungeKeyLen || !bytes.Equal(a, b) {
|
||||
t.Fatalf("a key is %d bytes and the same each time: got %d, equal %v", MungeKeyLen, len(a), bytes.Equal(a, b))
|
||||
}
|
||||
c, _ := MungeKey(strings.Repeat("cd", 32))
|
||||
if bytes.Equal(a, c) {
|
||||
t.Fatal("two secrets made one key")
|
||||
}
|
||||
if _, err := MungeKey("short"); err == nil {
|
||||
t.Fatal("a short secret was taken")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseHardware(t *testing.T) {
|
||||
out := "NodeName=cuda01 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=64302\n" +
|
||||
"UpTime=3-01:02:03\n"
|
||||
h, err := ParseHardware(out)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := Hardware{CPUs: 16, Boards: 1, SocketsPerBoard: 2, CoresPerSocket: 4, ThreadsPerCore: 2, RealMemory: 64302}
|
||||
if h != want {
|
||||
t.Fatalf("got %+v", h)
|
||||
}
|
||||
if _, err := ParseHardware("slurmd: error: something\n"); err == nil {
|
||||
t.Fatal("an error was read as hardware")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGPUType(t *testing.T) {
|
||||
for name, want := range map[string]string{
|
||||
"NVIDIA A100-SXM4-80GB": "a100",
|
||||
"NVIDIA H100 80GB HBM3": "h100",
|
||||
"Tesla V100-SXM2-32GB": "v100",
|
||||
"Tesla T4": "t4",
|
||||
"NVIDIA GeForce RTX 3090": "rtx3090",
|
||||
"NVIDIA RTX A6000": "rtxa6000",
|
||||
"NVIDIA L40S": "l40s",
|
||||
"Quadro RTX 8000": "rtx8000",
|
||||
"NVIDIA GeForce GTX 1080": "gtx1080",
|
||||
"": "gpu",
|
||||
"Some Accelerator Unknown": "gpu",
|
||||
} {
|
||||
if got := GPUType(name); got != want {
|
||||
t.Errorf("GPUType(%q) = %q, want %q", name, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func testCluster() Cluster {
|
||||
return Cluster{Name: "llmgw", Controller: "node06", ControllerAddr: "10.0.0.146", Partition: "gpu",
|
||||
Nodes: []Node{
|
||||
{Name: "site2", Addr: "10.1.0.5",
|
||||
Hardware: Hardware{CPUs: 32, Boards: 1, SocketsPerBoard: 2, CoresPerSocket: 8, ThreadsPerCore: 2, RealMemory: 100000},
|
||||
GPUs: []GPU{{Minor: 2, Name: "Tesla T4"}, {Minor: 0, Name: "Tesla V100-SXM2-32GB"}, {Minor: 1, Name: "Tesla V100-SXM2-32GB"}}},
|
||||
{Name: "cuda01", Addr: "10.0.0.147", Hardware: Hardware{CPUs: 16, RealMemory: 64000},
|
||||
GPUs: []GPU{{Minor: 0, Name: "NVIDIA GeForce RTX 3090"}}},
|
||||
{Name: "waiting", Addr: "10.0.0.9"},
|
||||
}}
|
||||
}
|
||||
|
||||
func TestSlurmConf(t *testing.T) {
|
||||
conf := testCluster().SlurmConf()
|
||||
for _, want := range []string{
|
||||
"ClusterName=llmgw\n",
|
||||
"SlurmctldHost=node06(10.0.0.146)\n",
|
||||
"SlurmctldParameters=enable_configless\n",
|
||||
"NodeName=cuda01 NodeAddr=10.0.0.147 CPUs=16 RealMemory=62080 Gres=gpu:rtx3090:1 State=DRAIN Reason=\"llmgw: not lent yet\"\n",
|
||||
"NodeName=site2 NodeAddr=10.1.0.5 CPUs=32 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=97000 Gres=gpu:t4:1,gpu:v100:2 State=DRAIN Reason=\"llmgw: not lent yet\"\n",
|
||||
"PartitionName=gpu Nodes=cuda01,site2 Default=YES",
|
||||
} {
|
||||
if !strings.Contains(conf, want) {
|
||||
t.Errorf("slurm.conf has no %q:\n%s", want, conf)
|
||||
}
|
||||
}
|
||||
if strings.Contains(conf, "waiting") {
|
||||
t.Error("a node that has not reported its hardware is in slurm.conf")
|
||||
}
|
||||
if strings.Index(conf, "NodeName=cuda01") > strings.Index(conf, "NodeName=site2") {
|
||||
t.Error("nodes are not in order")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlurmConfNoNodes(t *testing.T) {
|
||||
c := testCluster()
|
||||
c.Nodes = nil
|
||||
if strings.Contains(c.SlurmConf(), "PartitionName") {
|
||||
t.Error("a partition with no nodes would stop slurmctld starting")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGresConf(t *testing.T) {
|
||||
g := testCluster().GresConf()
|
||||
for _, want := range []string{
|
||||
"NodeName=cuda01 Name=gpu Type=rtx3090 File=/dev/nvidia0\n",
|
||||
"NodeName=site2 Name=gpu Type=t4 File=/dev/nvidia2\n",
|
||||
"NodeName=site2 Name=gpu Type=v100 File=/dev/nvidia0,/dev/nvidia1\n",
|
||||
} {
|
||||
if !strings.Contains(g, want) {
|
||||
t.Errorf("gres.conf has no %q:\n%s", want, g)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseNodeState(t *testing.T) {
|
||||
out := "NodeName=cuda01 Arch=x86_64 CoresPerSocket=8 CPUAlloc=0 State=IDLE+DRAIN ThreadsPerCore=2 " +
|
||||
"Partitions=gpu Reason=llmgw: serving models [root@2026-11-11T10:00:00]\n"
|
||||
s, err := ParseNodeState(out)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if s.State != "IDLE+DRAIN" || s.Reason != "llmgw: serving models" || !s.Drained() {
|
||||
t.Fatalf("got %+v", s)
|
||||
}
|
||||
s, _ = ParseNodeState("NodeName=cuda01 State=MIXED Partitions=gpu\n")
|
||||
if s.Drained() || s.Reason != "" {
|
||||
t.Fatalf("got %+v", s)
|
||||
}
|
||||
if _, err := ParseNodeState("Node cuda02 not found\n"); err == nil {
|
||||
t.Fatal("an unknown node was read")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseVersion(t *testing.T) {
|
||||
for out, want := range map[string][2]int{"slurm 23.11.4\n": {23, 11}, "slurm-wlm 21.08.5": {21, 8}, "slurm 22.05.9": {22, 5}} {
|
||||
v, err := ParseVersion(out)
|
||||
if err != nil || v != want {
|
||||
t.Errorf("ParseVersion(%q) = %v, %v", out, v, err)
|
||||
}
|
||||
}
|
||||
if !TooOld([2]int{21, 8}) || TooOld([2]int{22, 5}) || TooOld([2]int{23, 2}) {
|
||||
t.Error("TooOld is wrong")
|
||||
}
|
||||
if _, err := ParseVersion("command not found"); err == nil {
|
||||
t.Error("no version was read as one")
|
||||
}
|
||||
}
|
||||
@@ -14,10 +14,19 @@ var ErrNotMember = errors.New("not a member of that project")
|
||||
// Roles a membership can carry.
|
||||
const (
|
||||
RoleMember = "member"
|
||||
// RoleManager is reserved for delegated administration of a scope.
|
||||
// RoleManager, on an organization, may look at that organization's page:
|
||||
// its use, members and limits. It changes nothing.
|
||||
RoleManager = "manager"
|
||||
// RoleOrgAdmin, on an organization, also changes it: its teams and
|
||||
// projects, who is in them, and their keys. Its limits, the accounts
|
||||
// themselves and the rest of the cluster stay with the cluster's staff.
|
||||
RoleOrgAdmin = "admin"
|
||||
)
|
||||
|
||||
// MembershipRoles are the roles a membership may carry; manager and admin mean
|
||||
// something only on an organization.
|
||||
var MembershipRoles = []string{RoleMember, RoleManager, RoleOrgAdmin}
|
||||
|
||||
// Membership is a person's place at one level of the tree.
|
||||
type Membership struct {
|
||||
UserID string
|
||||
@@ -43,6 +52,12 @@ func (s *DB) AddMembership(ctx context.Context, userID string, scope Scope, scop
|
||||
if role == "" {
|
||||
role = RoleMember
|
||||
}
|
||||
if role != RoleMember && role != RoleManager && role != RoleOrgAdmin {
|
||||
return fmt.Errorf("a membership's role is member, manager or admin, not %q", role)
|
||||
}
|
||||
if role != RoleMember && scope != ScopeOrg {
|
||||
return fmt.Errorf("%s is a role on an organization, not on a %s", role, scope)
|
||||
}
|
||||
table := map[Scope]string{ScopeOrg: "organizations", ScopeTeam: "teams", ScopeProject: "projects"}[scope]
|
||||
if err := s.exists(ctx, string(scope), `SELECT COUNT(*) FROM `+table+` WHERE id = ?`, scopeID); err != nil {
|
||||
return err
|
||||
@@ -55,6 +70,27 @@ func (s *DB) AddMembership(ctx context.Context, userID string, scope Scope, scop
|
||||
return err
|
||||
}
|
||||
|
||||
// EnsureMembership puts a person in a scope as a member if they are not there
|
||||
// yet, and leaves a membership that exists as it is. It is for what a
|
||||
// directory or an identity provider says on every sign-in or sync: a group
|
||||
// that places someone in an organization must not, by saying so again, take
|
||||
// away the manager or admin role an administrator gave them there.
|
||||
func (s *DB) EnsureMembership(ctx context.Context, userID string, scope Scope, scopeID, by string, at time.Time) error {
|
||||
if !validMembershipScope(scope) {
|
||||
return fmt.Errorf("a membership is in an org, team or project, not %q", scope)
|
||||
}
|
||||
table := map[Scope]string{ScopeOrg: "organizations", ScopeTeam: "teams", ScopeProject: "projects"}[scope]
|
||||
if err := s.exists(ctx, string(scope), `SELECT COUNT(*) FROM `+table+` WHERE id = ?`, scopeID); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err := s.db.ExecContext(ctx, `
|
||||
INSERT INTO memberships (user_id, scope_type, scope_id, role, added_at, added_by)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT (user_id, scope_type, scope_id) DO NOTHING`,
|
||||
userID, string(scope), scopeID, RoleMember, millis(at), nullString(by))
|
||||
return err
|
||||
}
|
||||
|
||||
// RemoveMembership takes a person out of a scope and revokes the keys they
|
||||
// made in any project they can no longer use, returning how many.
|
||||
//
|
||||
@@ -306,23 +342,36 @@ func (s *DB) resolveChain(ctx context.Context, e *Effective, keyLevel bool) erro
|
||||
return nil
|
||||
}
|
||||
|
||||
// ManagedOrgs lists the organizations a person manages.
|
||||
// ManagedOrgs lists the organizations a person manages or administers: those
|
||||
// whose page they may see.
|
||||
func (s *DB) ManagedOrgs(ctx context.Context, userID string) ([]Org, error) {
|
||||
return queryAll(ctx, s.db, `
|
||||
SELECT o.id, o.name FROM memberships m JOIN organizations o ON o.id = m.scope_id
|
||||
WHERE m.user_id = ? AND m.scope_type = 'org' AND m.role = 'manager'
|
||||
WHERE m.user_id = ? AND m.scope_type = 'org' AND m.role IN ('manager', 'admin')
|
||||
ORDER BY o.name`, []any{userID}, func(rows *sql.Rows) (Org, error) {
|
||||
var o Org
|
||||
return o, rows.Scan(&o.ID, &o.Name)
|
||||
})
|
||||
}
|
||||
|
||||
// ManagesOrg reports whether a person manages an organization.
|
||||
// ManagesOrg reports whether a person manages or administers an
|
||||
// organization: whether they may see its page.
|
||||
func (s *DB) ManagesOrg(ctx context.Context, userID, orgID string) (bool, error) {
|
||||
var n int
|
||||
err := s.db.QueryRowContext(ctx, `
|
||||
SELECT COUNT(*) FROM memberships
|
||||
WHERE user_id = ? AND scope_type = 'org' AND scope_id = ? AND role = 'manager'`,
|
||||
WHERE user_id = ? AND scope_type = 'org' AND scope_id = ? AND role IN ('manager', 'admin')`,
|
||||
userID, orgID).Scan(&n)
|
||||
return n > 0, err
|
||||
}
|
||||
|
||||
// AdministersOrg reports whether a person administers an organization:
|
||||
// whether they may change it.
|
||||
func (s *DB) AdministersOrg(ctx context.Context, userID, orgID string) (bool, error) {
|
||||
var n int
|
||||
err := s.db.QueryRowContext(ctx, `
|
||||
SELECT COUNT(*) FROM memberships
|
||||
WHERE user_id = ? AND scope_type = 'org' AND scope_id = ? AND role = 'admin'`,
|
||||
userID, orgID).Scan(&n)
|
||||
return n > 0, err
|
||||
}
|
||||
@@ -307,3 +307,78 @@ func TestASuspendedPersonsKeysStopWorkingAndComeBackWithThem(t *testing.T) {
|
||||
t.Errorf("reinstated, the key still refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOrganizationsAdminAlsoManagesIt(t *testing.T) {
|
||||
f := newFixture(t)
|
||||
ctx := t.Context()
|
||||
if err := f.s.AddMembership(ctx, f.user, ScopeOrg, f.acme, RoleOrgAdmin, "t", time.Now()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ok, _ := f.s.AdministersOrg(ctx, f.user, f.acme); !ok {
|
||||
t.Error("an admin of Acme is not recognised as one")
|
||||
}
|
||||
if ok, _ := f.s.ManagesOrg(ctx, f.user, f.acme); !ok {
|
||||
t.Error("an admin of Acme may not see its page")
|
||||
}
|
||||
if ok, _ := f.s.AdministersOrg(ctx, f.user, f.scania); ok {
|
||||
t.Error("administering Acme made them an admin of Scania")
|
||||
}
|
||||
if orgs, _ := f.s.ManagedOrgs(ctx, f.user); len(orgs) != 1 || orgs[0].Name != "Acme" {
|
||||
t.Errorf("ManagedOrgs = %+v", orgs)
|
||||
}
|
||||
// A manager sees the page and changes nothing.
|
||||
if err := f.s.AddMembership(ctx, f.user, ScopeOrg, f.acme, RoleManager, "t", time.Now()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ok, _ := f.s.AdministersOrg(ctx, f.user, f.acme); ok {
|
||||
t.Error("a manager is taken for an admin")
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagerAndAdminAreRolesOnAnOrganizationOnly(t *testing.T) {
|
||||
f := newFixture(t)
|
||||
ctx := t.Context()
|
||||
for _, c := range []struct {
|
||||
scope Scope
|
||||
id string
|
||||
role string
|
||||
}{
|
||||
{ScopeProject, f.gateway, RoleOrgAdmin},
|
||||
{ScopeTeam, f.platform, RoleManager},
|
||||
{ScopeOrg, f.acme, "owner"},
|
||||
} {
|
||||
if err := f.s.AddMembership(ctx, f.user, c.scope, c.id, c.role, "t", time.Now()); err == nil {
|
||||
t.Errorf("%s on a %s was taken", c.role, c.scope)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestASyncedGroupLeavesARoleAsItIs(t *testing.T) {
|
||||
// A directory group that places someone in Acme says so at every sign-in;
|
||||
// saying it again must not take away the admin role they were given there.
|
||||
f := newFixture(t)
|
||||
ctx := t.Context()
|
||||
if err := f.s.AddMembership(ctx, f.user, ScopeOrg, f.acme, RoleOrgAdmin, "t", time.Now()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.s.EnsureMembership(ctx, f.user, ScopeOrg, f.acme, "oidc", time.Now()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ok, _ := f.s.AdministersOrg(ctx, f.user, f.acme); !ok {
|
||||
t.Error("a sign-in's group took the admin role away")
|
||||
}
|
||||
// And one that is not there yet is added as a member.
|
||||
if err := f.s.EnsureMembership(ctx, f.user, ScopeOrg, f.scania, "oidc", time.Now()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ms, _ := f.s.AllMemberships(ctx)
|
||||
found := false
|
||||
for _, m := range ms {
|
||||
if m.UserID == f.user && m.Scope == ScopeOrg && m.ScopeID == f.scania {
|
||||
found = m.Role == RoleMember
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Error("a new group did not make them a member of Scania")
|
||||
}
|
||||
}
|
||||
@@ -50,6 +50,13 @@ func (d AboutData) InstallURL() string {
|
||||
return "/install"
|
||||
}
|
||||
|
||||
// DeckURL is the introduction for decision makers, as a PDF. It lives in the
|
||||
// repository rather than in the binary, so every gateway does not carry a
|
||||
// megabyte of slides; gateway and website link to the same file.
|
||||
func (AboutData) DeckURL() string {
|
||||
return repoURL + "/raw/branch/main/docs/llmgw-intro.pdf"
|
||||
}
|
||||
|
||||
// Still is a screen of the install guide's recording, for the steps here.
|
||||
func (d AboutData) Still(name string) template.HTML {
|
||||
return InstallData{Site: d.Site}.Still(name)
|
||||
|
||||
@@ -123,6 +123,8 @@ type AccountPages struct {
|
||||
personalDefaults *store.Limits
|
||||
// orgs is set when /org is mounted, so the menu can offer it to managers.
|
||||
orgs OrgViewer
|
||||
// orgEditor changes an organization from its page, for its admin.
|
||||
orgEditor OrgEditor
|
||||
// llamaVersion is the llama.cpp build nodes install, for judging whether a
|
||||
// GGUF can be served before it is fetched or catalogued.
|
||||
llamaVersion string
|
||||
|
||||
@@ -267,7 +267,7 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
d := adminMirrorData{CSRF: csrfFor(r), Inline: true, HasRegistry: true}
|
||||
d := adminMirrorData{CSRF: csrfIfMay(r, "/admin/mirror/x"), Inline: true, HasRegistry: true}
|
||||
p.openRepo(r, &d, repo)
|
||||
if d.RegistryErr != "" {
|
||||
// Read by the page's own script, which shows it beneath the
|
||||
@@ -502,7 +502,7 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
|
||||
Addr: m.Addr(),
|
||||
Message: msg,
|
||||
Error: errMsg,
|
||||
CSRF: csrfFor(r),
|
||||
CSRF: csrfIfMay(r, "/admin/mirror/x"),
|
||||
Version: p.version,
|
||||
Now: time.Now().UTC().Format(time.RFC3339),
|
||||
}
|
||||
@@ -736,6 +736,11 @@ func mirrorTabs(l *Lang, d adminMirrorData) []mirrorTab {
|
||||
tabs = append(tabs, mirrorTab{Label: l.T("mirror.tab.strata"),
|
||||
Href: "/admin/mirror?tab=strata", Active: d.Tab == tabStrata})
|
||||
}
|
||||
if d.CSRF == "" {
|
||||
// Fetching is all that tab does: someone who may not fetch is not
|
||||
// offered an empty page.
|
||||
return tabs
|
||||
}
|
||||
return append(tabs, mirrorTab{Label: l.T("mirror.tab.fetch"),
|
||||
Href: "/admin/mirror?tab=fetch", Active: d.Tab == tabFetch})
|
||||
}
|
||||
|
||||
@@ -149,7 +149,7 @@ func (p *AccountPages) detail(r *http.Request, sel string, u *adminUsersData) *p
|
||||
}
|
||||
ctx := r.Context()
|
||||
l := langOf(r)
|
||||
d := &peopleDetail{Kind: store.Scope(kind), ID: id, CSRF: csrfFor(r),
|
||||
d := &peopleDetail{Kind: store.Scope(kind), ID: id, CSRF: csrfIfMay(r, "/admin/users"),
|
||||
UsageWindow: l.T("usage.window.7d"), QuotaWindow: l.T("quotas.since")}
|
||||
if !p.nameDetail(d, u) {
|
||||
return nil
|
||||
|
||||
@@ -353,7 +353,7 @@ func (p *AccountPages) buildQuotas(r *http.Request, s QuotaAdmin, msg, errMsg st
|
||||
d := adminQuotasData{
|
||||
Nav: navFrom(r, "quotas"),
|
||||
Since: l.T("quotas.since"), Message: msg, Error: errMsg,
|
||||
CSRF: csrfFor(r), Version: p.version,
|
||||
CSRF: csrfIfMay(r, "/admin/quotas"), Version: p.version,
|
||||
Now: time.Now().UTC().Format(time.RFC3339),
|
||||
}
|
||||
|
||||
|
||||
@@ -224,7 +224,10 @@ func planServe(l *Lang, st mirror.Status, files []mirror.Status, base string,
|
||||
if weights <= 0 {
|
||||
weights = st.BytesDone
|
||||
}
|
||||
if b := header.MemoryBytes(weights, defaultCtxPool); b > 0 {
|
||||
if c, par := mostContext(header, weights, f.Placements, f.GPUs, fleet); c > 0 {
|
||||
f.Context, f.Parallel, f.CtxPool = c, par, c*par
|
||||
}
|
||||
if b := header.MemoryBytes(weights, f.CtxPool); b > 0 {
|
||||
kvHeads := header.HeadCountKV
|
||||
if kvHeads == 0 {
|
||||
kvHeads = header.HeadCount
|
||||
@@ -235,8 +238,8 @@ func planServe(l *Lang, st mirror.Status, files []mirror.Status, base string,
|
||||
// flat "half again" could not see.
|
||||
f.MinMemoryGB = int((b + 1<<30 - 1) >> 30)
|
||||
f.MemoryBasis = l.T("serve.basis.shape",
|
||||
l.Num(float64(weights)/(1<<30), 1), l.Num(float64(int64(defaultCtxPool)*header.KVBytesPerToken())/(1<<30), 1),
|
||||
l.Int(defaultCtxPool), header.BlockCount, kvHeads,
|
||||
l.Num(float64(weights)/(1<<30), 1), l.Num(float64(int64(f.CtxPool)*header.KVBytesPerToken())/(1<<30), 1),
|
||||
l.Int(int64(f.CtxPool)), header.BlockCount, kvHeads,
|
||||
l.Num(float64(int64(1<<30)+weights/20)/(1<<30), 1))
|
||||
}
|
||||
if fleet != nil {
|
||||
@@ -552,6 +555,46 @@ func modeOf(sc catalog.Scaling) string {
|
||||
// --parallel divides it among slots; it does not multiply it.
|
||||
const defaultCtxPool = 16384
|
||||
|
||||
// mostContext is the longest context the serve form starts with: what the
|
||||
// model was trained for, as long as its KV cache fits where the form places
|
||||
// it beside the weights, else the most that does; and as many slots of it,
|
||||
// up to defaultParallel, as the room left holds. 0 when the header does not
|
||||
// say what it was trained for, or the weights alone do not fit.
|
||||
//
|
||||
// The longest, because a context is the one thing about a served model a
|
||||
// client runs into without being told: a 4096 default on a model trained for
|
||||
// 40960 cut documents short that the model could have read whole.
|
||||
func mostContext(h gguf.Header, weights int64, places []placeChoice, gpus int, fleet Source) (tokens, slots int) {
|
||||
trained := int(h.ContextLength) //nolint:gosec // a header's context length
|
||||
if trained <= 0 {
|
||||
return 0, 0
|
||||
}
|
||||
pool := 0
|
||||
for _, pc := range places {
|
||||
if !pc.Selected {
|
||||
continue
|
||||
}
|
||||
if class, ok := strings.CutPrefix(pc.Value, "gpu:"); ok {
|
||||
pool = h.PoolThatFits(weights, gguf.ClassMemGB(class), gpus)
|
||||
} else {
|
||||
pool = h.PoolThatFits(weights, biggestNode(fleet), 1)
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case pool < 0:
|
||||
return 0, 0
|
||||
case pool == 0:
|
||||
// Nothing to measure it against: what it was trained for, as one slot
|
||||
// would hold it; the checks on submitting still see the card.
|
||||
return trained, 1
|
||||
}
|
||||
tokens = min(trained, pool/1024*1024)
|
||||
if tokens < 1024 {
|
||||
return 0, 0
|
||||
}
|
||||
return tokens, max(1, min(defaultParallel, pool/tokens))
|
||||
}
|
||||
|
||||
// defaultParallel is the requests a llama.cpp replica serves at once when
|
||||
// the form is not changed: the agent's own default.
|
||||
const defaultParallel = 4
|
||||
|
||||
@@ -670,3 +670,31 @@ func TestAVerySlowModelOnTheCPUIsAChoiceMadeOnPurpose(t *testing.T) {
|
||||
t.Fatal("ticking the box should add it")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheServeFormStartsAtTheLongestContextThatFits(t *testing.T) {
|
||||
// An 8B's shape: 36 layers, 8 KV heads of 128, so 144 KiB a token.
|
||||
h := gguf.Header{BlockCount: 36, HeadCount: 32, HeadCountKV: 8, KeyLength: 128, ValueLength: 128,
|
||||
ContextLength: 40960}
|
||||
card := []placeChoice{{Value: "gpu:6000-24g", Selected: true}, {Value: "cpu"}}
|
||||
cpu := []placeChoice{{Value: "gpu:6000-24g"}, {Value: "cpu", Selected: true}}
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
weights int64
|
||||
places []placeChoice
|
||||
header gguf.Header
|
||||
tokens, slots int
|
||||
}{
|
||||
// All it was trained for, and two slots of it in the room left.
|
||||
{"on a card with room", 5 << 30, card, h, 40960, 2},
|
||||
// The weights take most of the card: the most that fits, one slot.
|
||||
{"on a card nearly full", 18 << 30, card, h, 18432, 1},
|
||||
{"on the CPU of a 64 GB node", 5 << 30, cpu, h, 40960, 4},
|
||||
{"weights larger than the card", 30 << 30, card, h, 0, 0},
|
||||
{"a header that does not say", 5 << 30, card, gguf.Header{BlockCount: 36, HeadCountKV: 8, HeadCount: 32, KeyLength: 128}, 0, 0},
|
||||
} {
|
||||
tokens, slots := mostContext(c.header, c.weights, c.places, 1, fakeFleet{memoryGB: 64})
|
||||
if tokens != c.tokens || slots != c.slots {
|
||||
t.Errorf("%s: %d tokens × %d slots, want %d × %d", c.name, tokens, slots, c.tokens, c.slots)
|
||||
}
|
||||
}
|
||||
}
|
||||
+24
-10
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -92,7 +93,10 @@ type adminUsersData struct {
|
||||
// Open is the project whose section is shown expanded: the one just acted
|
||||
// on, so the result of a change is in view without the reader hunting for
|
||||
// it. Scripted pages keep every open section open; this is for the rest.
|
||||
Open string
|
||||
Open string
|
||||
// Edit is whether this person may change what the page shows: a viewer
|
||||
// is drawn the page without its forms.
|
||||
Edit bool
|
||||
AdminID string
|
||||
Message string
|
||||
Error string
|
||||
@@ -173,23 +177,19 @@ func (p *AccountPages) RegisterAdminUsers(mux *http.ServeMux, s AccountAdmin) {
|
||||
return say("users.msg.added_to", scopeLabel(ctx, s, store.Scope(scope), scopeID)), nil
|
||||
}))
|
||||
|
||||
// Managing is a role on an organization membership: it lets the person see
|
||||
// that organization's page, and nothing else changes.
|
||||
// A role on an organization membership: a manager sees that organization's
|
||||
// page, an admin also changes it; a member neither.
|
||||
mux.HandleFunc("POST /admin/users/{id}/role", p.adminAction(s,
|
||||
func(ctx context.Context, admin store.User, id string, r *http.Request) (notice, error) {
|
||||
scope, scopeID, ok := strings.Cut(r.FormValue("scope"), ":")
|
||||
role := r.FormValue("role")
|
||||
if !ok || store.Scope(scope) != store.ScopeOrg || (role != store.RoleManager && role != store.RoleMember) {
|
||||
if !ok || store.Scope(scope) != store.ScopeOrg || !slices.Contains(store.MembershipRoles, role) {
|
||||
return notice{}, refuse("users.err.manager_role")
|
||||
}
|
||||
if err := s.AddMembership(ctx, id, store.ScopeOrg, scopeID, role, admin.Email, time.Now()); err != nil {
|
||||
return notice{}, err
|
||||
}
|
||||
label := scopeLabel(ctx, s, store.ScopeOrg, scopeID)
|
||||
if role == store.RoleManager {
|
||||
return say("users.msg.now_manages", label), nil
|
||||
}
|
||||
return say("users.msg.no_longer_manages", label), nil
|
||||
return roleNotice(role, scopeLabel(ctx, s, store.ScopeOrg, scopeID)), nil
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /admin/users/{id}/leave", p.adminAction(s,
|
||||
@@ -351,6 +351,17 @@ func (p *AccountPages) RegisterAdminUsers(mux *http.ServeMux, s AccountAdmin) {
|
||||
}))
|
||||
}
|
||||
|
||||
// roleNotice says what a person may now do in an organization.
|
||||
func roleNotice(role, org string) notice {
|
||||
switch role {
|
||||
case store.RoleManager:
|
||||
return say("users.msg.now_manages", org)
|
||||
case store.RoleOrgAdmin:
|
||||
return say("users.msg.now_administers", org)
|
||||
}
|
||||
return say("users.msg.no_longer_manages", org)
|
||||
}
|
||||
|
||||
// notice is what the page says about a change, kept as a catalogue key and
|
||||
// its arguments rather than as text: the log and the audit record take it in
|
||||
// English, which is what they are searched in, and the page in the language
|
||||
@@ -394,7 +405,7 @@ func init() {
|
||||
registerKeys(
|
||||
"users.msg.approved_alone", "users.msg.approved_into", "users.msg.suspended",
|
||||
"users.msg.reinstated", "users.msg.added_to", "users.msg.now_manages",
|
||||
"users.msg.no_longer_manages", "users.msg.removed_from", "users.msg.removed_from_one_key",
|
||||
"users.msg.no_longer_manages", "users.msg.now_administers", "users.msg.removed_from", "users.msg.removed_from_one_key",
|
||||
"users.msg.removed_from_keys", "users.msg.removed_keys_revoked", "users.msg.org_created",
|
||||
"users.msg.team_created", "users.msg.project_created", "users.msg.renamed",
|
||||
"users.msg.team_moved", "users.msg.project_moved", "users.msg.removed",
|
||||
@@ -408,6 +419,8 @@ func init() {
|
||||
// state, which is also the row's class.
|
||||
"users.state.active", "users.state.pending", "users.state.suspended",
|
||||
"users.key_state.active", "users.key_state.revoked", "users.key_state.expired",
|
||||
// A membership's role, looked up by the role.
|
||||
"users.role.member", "users.role.manager", "users.role.admin",
|
||||
)
|
||||
}
|
||||
|
||||
@@ -573,6 +586,7 @@ func (p *AccountPages) buildUsers(r *http.Request, s AccountAdmin, admin store.U
|
||||
Members: map[string][]userRow{},
|
||||
Keys: map[string][]keyRow{},
|
||||
}
|
||||
d.Edit = d.Nav.May("/admin/users")
|
||||
var err error
|
||||
// logErr logs what could not be read in English and says so on the page,
|
||||
// once, in the reader's language.
|
||||
|
||||
@@ -134,13 +134,13 @@ func TestTheUsageReportWritesTurkishNumbers(t *testing.T) {
|
||||
|
||||
func TestTheOrganizationPageSpeaksTurkishWhenAsked(t *testing.T) {
|
||||
body := inTurkish(t, orgPage(t, store.User{ID: "u1", Email: "boss@acme"}), "/org/org-a", nil)
|
||||
for _, want := range []string{"2 kişi", "Kullanım", "Ekipler, projeler ve sınırlar", "Üyeler",
|
||||
">yönetici</b>", "kuruluşun tamamı", "son 7 gün", `lang="tr"`} {
|
||||
for _, want := range []string{"2 kişi", "Kullanım", "Sınırlar", "Üyeler",
|
||||
">sorumlu</b>", "kuruluşun tamamı", "son 7 gün", `lang="tr"`} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the organization page in Turkish lacks %q", want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(body, "Teams, projects and limits") {
|
||||
if strings.Contains(body, ">Limits<") {
|
||||
t.Error("the organization page in Turkish still has English headings")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,6 +7,7 @@
|
||||
"about.nav.install": "Install",
|
||||
"about.nav.proxy": "Reverse proxy",
|
||||
"about.nav.download": "Download",
|
||||
"about.nav.deck": "Presentation",
|
||||
"about.nav.source": "Source",
|
||||
"about.nav.gateway": "This gateway",
|
||||
"about.hero.eyebrow": "Self-hosted LLM gateway · %s",
|
||||
@@ -14,6 +15,7 @@
|
||||
"about.hero.lede": "llmgw runs vLLM, SGLang and llama.cpp across your servers and serves them behind one OpenAI-compatible API. It knows which GPUs share an NVLink domain or an InfiniBand fabric, and a model split across machines never straddles two.",
|
||||
"about.hero.download": "Download",
|
||||
"about.hero.guide": "Install guide",
|
||||
"about.hero.deck": "Presentation (PDF)",
|
||||
"about.hero.oneline_label": "Set up a control plane",
|
||||
"about.copy": "Copy",
|
||||
"about.copied": "Copied",
|
||||
|
||||
@@ -13,14 +13,13 @@
|
||||
"org.use_empty": "Nothing used in this window.",
|
||||
"org.use_scope": "Only what was spent in this organization's projects. What its people do elsewhere — in another organization, or on their own — is not shown.",
|
||||
|
||||
"org.limits.title": "Teams, projects and limits",
|
||||
"org.limits.title": "Limits",
|
||||
"org.col.spent_30d": "Spent · 30 days",
|
||||
"org.limits.note": "Limits are set by the cluster's administrators.",
|
||||
|
||||
"org.members.title": "Members",
|
||||
"org.col.member_of": "Member of",
|
||||
"org.col.state": "State",
|
||||
"org.role.manager": "manager",
|
||||
"org.state.pending": "pending",
|
||||
"org.state.active": "active",
|
||||
"org.state.suspended": "suspended",
|
||||
@@ -30,7 +29,47 @@
|
||||
"org.none.title": "Organizations",
|
||||
"org.none.sub": "There are none on this cluster",
|
||||
"org.none.about": "Everyone uses the gateway on their own, under their personal quota. An organization, once there is one, gets a page here for its managers: its use by team, project and person, its limits, and its members.",
|
||||
"org.none.how": "<a href=\"/admin/users\">Create one on the people page</a>, then make someone its manager with the star on their membership.",
|
||||
"org.none.how": "<a href=\"/admin/users\">Create one on the people page</a>, then choose manager or org admin on someone's membership of it.",
|
||||
|
||||
"org.col.role": "Role here",
|
||||
"org.col.projects": "Projects",
|
||||
"org.add.email": "their sign-in address",
|
||||
"org.add.where": "Where",
|
||||
"org.add.button": "Add",
|
||||
"org.add.note": "Anyone with an approved account can be added. Someone new signs up first; the cluster's administrators approve them.",
|
||||
"org.tree.title": "Teams and projects",
|
||||
"org.tree.rename": "Rename",
|
||||
"org.tree.remove": "Remove %s",
|
||||
"org.tree.remove_confirm": "Remove %s? Only an empty one can be removed.",
|
||||
"org.tree.remove_team": "Remove",
|
||||
"org.tree.new_project": "new project",
|
||||
"org.tree.new_project_in": "New project in %s",
|
||||
"org.tree.new_team": "new team",
|
||||
"org.tree.add_team": "Add team",
|
||||
"org.tree.empty": "No teams yet.",
|
||||
"org.tree.note": "Click a team's name to rename it. A team or project is removed only when nobody and no key is left in it.",
|
||||
"org.keys.title": "Keys",
|
||||
"org.keys.col.key": "Key",
|
||||
"org.keys.col.owner": "Owner",
|
||||
"org.keys.col.used": "Last used",
|
||||
"org.keys.revoke": "Revoke",
|
||||
"org.keys.revoke_confirm": "Revoke %s…? Whatever uses it stops working.",
|
||||
"org.keys.empty": "No keys in this organization's projects.",
|
||||
"org.msg.team_created": "Team %s added.",
|
||||
"org.msg.project_created": "Project %s added.",
|
||||
"org.msg.renamed": "Renamed to %s.",
|
||||
"org.msg.removed": "%s removed.",
|
||||
"org.msg.added": "%s added to %s.",
|
||||
"org.msg.taken_out": "Taken out of %s.",
|
||||
"org.msg.taken_out_keys": "Taken out of %s; %d of their keys there revoked.",
|
||||
"org.msg.key_revoked": "Key %s… revoked.",
|
||||
"org.err.not_here": "that is not in this organization",
|
||||
"org.err.no_account": "there is no account for %s: they sign up first",
|
||||
"org.err.not_approved": "%s waits for the cluster's administrators to approve them",
|
||||
"org.err.suspended": "%s is suspended",
|
||||
"org.err.self_role": "your own role here is changed by the cluster's administrators",
|
||||
"org.err.self_leave": "you cannot take yourself out of the organization you administer",
|
||||
"org.err.name": "a name is needed, without a slash",
|
||||
"org.err.role": "a role is member, manager or org admin",
|
||||
"org.err.read_tree": "could not read the organization"
|
||||
}
|
||||
@@ -139,10 +139,10 @@
|
||||
"users.state.pending": "pending",
|
||||
"users.state.suspended": "suspended",
|
||||
"users.role.manager": "manager",
|
||||
"users.stop_managing_of": "Stop managing %s",
|
||||
"users.make_manager_of": "Make manager of %s: can see its page",
|
||||
"users.stop_managing": "Stop managing",
|
||||
"users.make_manager": "Make manager",
|
||||
"users.role.member": "member",
|
||||
"users.role.admin": "org admin",
|
||||
"users.role_in": "Role in %s",
|
||||
"users.role_note": "A manager sees the organization's page; an admin also changes its teams, projects, people and keys.",
|
||||
"users.take_out_confirm": "Take %s out of %s? Keys they made there are revoked.",
|
||||
"users.take_out_of": "Take out of %s",
|
||||
"users.on_own": "on their own",
|
||||
@@ -164,6 +164,7 @@
|
||||
"users.msg.added_to": "added to %s",
|
||||
"users.msg.now_manages": "now manages %s",
|
||||
"users.msg.no_longer_manages": "no longer manages %s",
|
||||
"users.msg.now_administers": "now administers %s",
|
||||
"users.msg.removed_from": "removed from %s",
|
||||
"users.msg.removed_from_one_key": "removed from %s; 1 key they made there revoked",
|
||||
"users.msg.removed_from_keys": "removed from %s; %d keys they made there revoked",
|
||||
@@ -181,7 +182,7 @@
|
||||
"users.err.about": "%s: %s",
|
||||
"users.err.self_suspend": "you cannot suspend your own account",
|
||||
"users.err.choose_where": "choose where they go",
|
||||
"users.err.manager_role": "a manager is a role in an organization",
|
||||
"users.err.manager_role": "manager and admin are roles in an organization",
|
||||
"users.err.nothing_to_leave": "nothing to leave",
|
||||
"users.err.self_remove": "you cannot remove your own account",
|
||||
"users.err.choose_org": "choose the organization this team belongs to",
|
||||
|
||||
@@ -7,6 +7,7 @@
|
||||
"about.nav.install": "Kurulum",
|
||||
"about.nav.proxy": "Ters proxy",
|
||||
"about.nav.download": "İndir",
|
||||
"about.nav.deck": "Sunum",
|
||||
"about.nav.source": "Kaynak kod",
|
||||
"about.nav.gateway": "Bu ağ geçidi",
|
||||
"about.hero.eyebrow": "Kendi sunucunuzda çalışan LLM ağ geçidi · %s",
|
||||
@@ -14,6 +15,7 @@
|
||||
"about.hero.lede": "llmgw; vLLM, SGLang ve llama.cpp'yi sunucularınızda çalıştırır ve hepsini OpenAI uyumlu tek bir API'nin arkasından sunar. Hangi GPU'ların aynı NVLink alanını ya da aynı InfiniBand fabric'ini paylaştığını bilir; makinelere bölünmüş bir model hiçbir zaman iki fabric'e birden yayılmaz.",
|
||||
"about.hero.download": "İndir",
|
||||
"about.hero.guide": "Kurulum kılavuzu",
|
||||
"about.hero.deck": "Tanıtım sunumu (PDF)",
|
||||
"about.hero.oneline_label": "Bir kontrol düzlemi kurun",
|
||||
"about.copy": "Kopyala",
|
||||
"about.copied": "Kopyalandı",
|
||||
|
||||
@@ -12,14 +12,13 @@
|
||||
"org.use_empty": "Bu dönemde hiçbir şey kullanılmadı.",
|
||||
"org.use_scope": "Yalnızca bu kuruluşun projelerinde harcananlar. Kişilerinin başka yerlerde — başka bir kuruluşta ya da kendi başlarına — yaptıkları gösterilmez.",
|
||||
|
||||
"org.limits.title": "Ekipler, projeler ve sınırlar",
|
||||
"org.limits.title": "Sınırlar",
|
||||
"org.col.spent_30d": "Harcanan · 30 gün",
|
||||
"org.limits.note": "Sınırları kümenin yöneticileri belirler.",
|
||||
|
||||
"org.members.title": "Üyeler",
|
||||
"org.col.member_of": "Üyesi olduğu yer",
|
||||
"org.col.state": "Durum",
|
||||
"org.role.manager": "yönetici",
|
||||
"org.state.pending": "onay bekliyor",
|
||||
"org.state.active": "etkin",
|
||||
"org.state.suspended": "askıya alındı",
|
||||
@@ -29,7 +28,47 @@
|
||||
"org.none.title": "Kuruluşlar",
|
||||
"org.none.sub": "Bu kümede hiç kuruluş yok",
|
||||
"org.none.about": "Herkes ağ geçidini kendi kişisel kotası altında, kendi başına kullanıyor. Bir kuruluş oluşturulduğunda, yöneticileri için burada bir sayfası olur: ekibe, projeye ve kişiye göre kullanımı, sınırları ve üyeleri.",
|
||||
"org.none.how": "<a href=\"/admin/users\">Kişiler sayfasında bir kuruluş oluşturun</a>, ardından birinin üyeliğindeki yıldızla onu kuruluşun yöneticisi yapın.",
|
||||
"org.none.how": "<a href=\"/admin/users\">Kişiler sayfasında bir kuruluş oluşturun</a>, ardından birinin oradaki üyeliğinde sorumlu ya da kuruluş yöneticisi rolünü seçin.",
|
||||
|
||||
"org.col.role": "Buradaki rolü",
|
||||
"org.col.projects": "Projeler",
|
||||
"org.add.email": "giriş adresi",
|
||||
"org.add.where": "Nereye",
|
||||
"org.add.button": "Ekle",
|
||||
"org.add.note": "Hesabı onaylanmış herkes eklenebilir. Yeni biri önce kaydolur; kümenin yöneticileri onu onaylar.",
|
||||
"org.tree.title": "Ekipler ve projeler",
|
||||
"org.tree.rename": "Yeniden adlandır",
|
||||
"org.tree.remove": "%s kaldır",
|
||||
"org.tree.remove_confirm": "%s kaldırılsın mı? Yalnızca boş olan kaldırılabilir.",
|
||||
"org.tree.remove_team": "Kaldır",
|
||||
"org.tree.new_project": "yeni proje",
|
||||
"org.tree.new_project_in": "%s içinde yeni proje",
|
||||
"org.tree.new_team": "yeni ekip",
|
||||
"org.tree.add_team": "Ekip ekle",
|
||||
"org.tree.empty": "Henüz ekip yok.",
|
||||
"org.tree.note": "Adını değiştirmek için ekibin adına tıklayın. Bir ekip ya da proje, içinde kimse ve hiçbir anahtar kalmadığında kaldırılabilir.",
|
||||
"org.keys.title": "Anahtarlar",
|
||||
"org.keys.col.key": "Anahtar",
|
||||
"org.keys.col.owner": "Sahibi",
|
||||
"org.keys.col.used": "Son kullanım",
|
||||
"org.keys.revoke": "İptal et",
|
||||
"org.keys.revoke_confirm": "%s… iptal edilsin mi? Onu kullanan her şey çalışmayı bırakır.",
|
||||
"org.keys.empty": "Bu kuruluşun projelerinde anahtar yok.",
|
||||
"org.msg.team_created": "%s ekibi eklendi.",
|
||||
"org.msg.project_created": "%s projesi eklendi.",
|
||||
"org.msg.renamed": "Yeni adı %s.",
|
||||
"org.msg.removed": "%s kaldırıldı.",
|
||||
"org.msg.added": "%s, %s içine eklendi.",
|
||||
"org.msg.taken_out": "%s içinden çıkarıldı.",
|
||||
"org.msg.taken_out_keys": "%s içinden çıkarıldı; oradaki %d anahtarı iptal edildi.",
|
||||
"org.msg.key_revoked": "%s… anahtarı iptal edildi.",
|
||||
"org.err.not_here": "bu, bu kuruluşta değil",
|
||||
"org.err.no_account": "%s için bir hesap yok: önce kaydolması gerekiyor",
|
||||
"org.err.not_approved": "%s, kümenin yöneticilerinin onayını bekliyor",
|
||||
"org.err.suspended": "%s askıya alınmış",
|
||||
"org.err.self_role": "buradaki kendi rolünüzü kümenin yöneticileri değiştirir",
|
||||
"org.err.self_leave": "yönettiğiniz kuruluştan kendinizi çıkaramazsınız",
|
||||
"org.err.name": "bir ad gerekli, eğik çizgi olmadan",
|
||||
"org.err.role": "rol üye, sorumlu ya da kuruluş yöneticisidir",
|
||||
"org.err.read_tree": "kuruluş okunamadı"
|
||||
}
|
||||
@@ -135,10 +135,10 @@
|
||||
"users.state.pending": "onay bekliyor",
|
||||
"users.state.suspended": "askıda",
|
||||
"users.role.manager": "sorumlu",
|
||||
"users.stop_managing_of": "%s sorumluluğunu bırak",
|
||||
"users.make_manager_of": "%s sorumlusu yap: sayfasını görebilir",
|
||||
"users.stop_managing": "Sorumluluğu bırak",
|
||||
"users.make_manager": "Sorumlu yap",
|
||||
"users.role.member": "üye",
|
||||
"users.role.admin": "kuruluş yöneticisi",
|
||||
"users.role_in": "%s içindeki rolü",
|
||||
"users.role_note": "Sorumlu, kuruluşun sayfasını görür; kuruluş yöneticisi ayrıca takımlarını, projelerini, kişilerini ve anahtarlarını değiştirir.",
|
||||
"users.take_out_confirm": "%[1]s, %[2]s içinden çıkarılsın mı? Orada oluşturduğu anahtarlar iptal edilir.",
|
||||
"users.take_out_of": "%s içinden çıkar",
|
||||
"users.on_own": "tek başına",
|
||||
@@ -160,6 +160,7 @@
|
||||
"users.msg.added_to": "%s içine eklendi",
|
||||
"users.msg.now_manages": "artık %s sorumlusu",
|
||||
"users.msg.no_longer_manages": "artık %s sorumlusu değil",
|
||||
"users.msg.now_administers": "artık %s kuruluş yöneticisi",
|
||||
"users.msg.removed_from": "%s içinden çıkarıldı",
|
||||
"users.msg.removed_from_one_key": "%s içinden çıkarıldı; orada oluşturduğu 1 anahtar iptal edildi",
|
||||
"users.msg.removed_from_keys": "%s içinden çıkarıldı; orada oluşturduğu %d anahtar iptal edildi",
|
||||
@@ -177,7 +178,7 @@
|
||||
"users.err.about": "%s: %s",
|
||||
"users.err.self_suspend": "kendi hesabınızı askıya alamazsınız",
|
||||
"users.err.choose_where": "nereye gideceğini seçin",
|
||||
"users.err.manager_role": "sorumluluk, bir kuruluştaki roldür",
|
||||
"users.err.manager_role": "sorumlu ve kuruluş yöneticisi, bir kuruluştaki rollerdir",
|
||||
"users.err.nothing_to_leave": "çıkılacak bir yer yok",
|
||||
"users.err.self_remove": "kendi hesabınızı kaldıramazsınız",
|
||||
"users.err.choose_org": "bu ekibin ait olduğu kuruluşu seçin",
|
||||
|
||||
@@ -29,7 +29,7 @@ type OIDCStore interface {
|
||||
UserByEmail(ctx context.Context, email string) (store.User, error)
|
||||
CreateUser(ctx context.Context, u store.User) error
|
||||
SetStaffRole(ctx context.Context, id, role string) error
|
||||
AddMembership(ctx context.Context, user string, scope store.Scope, id, role, by string, at time.Time) error
|
||||
EnsureMembership(ctx context.Context, user string, scope store.Scope, id, by string, at time.Time) error
|
||||
AllOrgs(ctx context.Context) ([]store.Org, error)
|
||||
AllTeams(ctx context.Context) ([]store.TeamPath, error)
|
||||
AllProjects(ctx context.Context) ([]store.ProjectPath, error)
|
||||
@@ -338,7 +338,7 @@ func (p *AccountPages) oidcUser(ctx context.Context, claims map[string]any) (sto
|
||||
p.log.Warn("a group maps to a place the tree does not have", "group", g, "place", where, "err", err)
|
||||
continue
|
||||
}
|
||||
if err := o.st.AddMembership(ctx, u.ID, scope, id, store.RoleMember, "oidc", time.Now()); err != nil {
|
||||
if err := o.st.EnsureMembership(ctx, u.ID, scope, id, "oidc", time.Now()); err != nil {
|
||||
p.log.Warn("could not add a membership from a group", "group", g, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -89,7 +89,7 @@ func (m *oidcStoreMem) SetStaffRole(_ context.Context, id, role string) error {
|
||||
m.roles[id] = role
|
||||
return nil
|
||||
}
|
||||
func (m *oidcStoreMem) AddMembership(_ context.Context, user string, scope store.Scope, id, _, _ string, _ time.Time) error {
|
||||
func (m *oidcStoreMem) EnsureMembership(_ context.Context, user string, scope store.Scope, id, _ string, _ time.Time) error {
|
||||
m.joined = append(m.joined, user+"@"+string(scope)+":"+id)
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,296 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// An organization's admin changes it from its own page.
|
||||
//
|
||||
// What they may change is what is theirs: the teams and projects in their
|
||||
// organization, who of the cluster's people is in them and in what role, and
|
||||
// the keys spent there. What stays with the cluster's staff is everything that
|
||||
// reaches past the organization or into the cluster's capacity: its limits and
|
||||
// budgets, approving or suspending an account, making anyone staff, and the
|
||||
// organization's own name and existence. A person not yet approved is not
|
||||
// added here either: approval says who may use the cluster at all.
|
||||
|
||||
// OrgEditor is what an organization's admin changes through.
|
||||
type OrgEditor interface {
|
||||
AdministersOrg(ctx context.Context, userID, orgID string) (bool, error)
|
||||
UserByEmail(ctx context.Context, email string) (store.User, error)
|
||||
CreateTeam(ctx context.Context, orgID, name string) (string, error)
|
||||
CreateProject(ctx context.Context, teamID, name string) (string, error)
|
||||
RenameTeam(ctx context.Context, id, name string) error
|
||||
RenameProject(ctx context.Context, id, name string) error
|
||||
DeleteTeam(ctx context.Context, id string) error
|
||||
DeleteProject(ctx context.Context, id string) error
|
||||
AddMembership(ctx context.Context, userID string, scope store.Scope, scopeID, role, by string, at time.Time) error
|
||||
RemoveMembership(ctx context.Context, userID string, scope store.Scope, scopeID string, at time.Time) (int, error)
|
||||
AllKeys(ctx context.Context) ([]store.KeyRow, error)
|
||||
RevokeKey(ctx context.Context, id string, at time.Time) error
|
||||
}
|
||||
|
||||
func init() {
|
||||
registerKeys(
|
||||
"org.msg.team_created", "org.msg.project_created", "org.msg.renamed", "org.msg.removed",
|
||||
"org.msg.added", "org.msg.taken_out", "org.msg.taken_out_keys", "org.msg.key_revoked",
|
||||
"org.err.not_here", "org.err.no_account", "org.err.not_approved", "org.err.suspended",
|
||||
"org.err.self_role", "org.err.self_leave", "org.err.name", "org.err.role",
|
||||
)
|
||||
}
|
||||
|
||||
// orgChange is one change an organization's admin makes, given the
|
||||
// organization as it is.
|
||||
type orgChange func(ctx context.Context, r *http.Request, by store.User, org store.TreeOrg) (notice, error)
|
||||
|
||||
// registerOrgAdmin mounts the changes, when the store makes them.
|
||||
func (p *AccountPages) registerOrgAdmin(mux *http.ServeMux, s OrgViewer, e OrgEditor, priced Priced) {
|
||||
act := func(do orgChange) http.HandlerFunc { return p.orgAction(s, e, priced, do) }
|
||||
|
||||
mux.HandleFunc("POST /org/{id}/teams", act(func(ctx context.Context, r *http.Request, _ store.User, o store.TreeOrg) (notice, error) {
|
||||
name, err := orgName(r)
|
||||
if err != nil {
|
||||
return notice{}, err
|
||||
}
|
||||
if _, err := e.CreateTeam(ctx, o.ID, name); err != nil {
|
||||
return notice{}, err
|
||||
}
|
||||
return say("org.msg.team_created", name), nil
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /org/{id}/projects", act(func(ctx context.Context, r *http.Request, _ store.User, o store.TreeOrg) (notice, error) {
|
||||
team := r.FormValue("team")
|
||||
if !orgHas(o, store.ScopeTeam, team) {
|
||||
return notice{}, refuse("org.err.not_here")
|
||||
}
|
||||
name, err := orgName(r)
|
||||
if err != nil {
|
||||
return notice{}, err
|
||||
}
|
||||
if _, err := e.CreateProject(ctx, team, name); err != nil {
|
||||
return notice{}, err
|
||||
}
|
||||
return say("org.msg.project_created", name), nil
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /org/{id}/{kind}/{sid}/rename", act(func(ctx context.Context, r *http.Request, _ store.User, o store.TreeOrg) (notice, error) {
|
||||
kind, sid := store.Scope(r.PathValue("kind")), r.PathValue("sid")
|
||||
if (kind != store.ScopeTeam && kind != store.ScopeProject) || !orgHas(o, kind, sid) {
|
||||
return notice{}, refuse("org.err.not_here")
|
||||
}
|
||||
name, err := orgName(r)
|
||||
if err != nil {
|
||||
return notice{}, err
|
||||
}
|
||||
if kind == store.ScopeTeam {
|
||||
err = e.RenameTeam(ctx, sid, name)
|
||||
} else {
|
||||
err = e.RenameProject(ctx, sid, name)
|
||||
}
|
||||
if err != nil {
|
||||
return notice{}, err
|
||||
}
|
||||
return say("org.msg.renamed", name), nil
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /org/{id}/{kind}/{sid}/delete", act(func(ctx context.Context, r *http.Request, _ store.User, o store.TreeOrg) (notice, error) {
|
||||
kind, sid := store.Scope(r.PathValue("kind")), r.PathValue("sid")
|
||||
if (kind != store.ScopeTeam && kind != store.ScopeProject) || !orgHas(o, kind, sid) {
|
||||
return notice{}, refuse("org.err.not_here")
|
||||
}
|
||||
name := orgPlaceName(o, kind, sid)
|
||||
var err error
|
||||
if kind == store.ScopeTeam {
|
||||
err = e.DeleteTeam(ctx, sid)
|
||||
} else {
|
||||
err = e.DeleteProject(ctx, sid)
|
||||
}
|
||||
if err != nil {
|
||||
return notice{}, err
|
||||
}
|
||||
return say("org.msg.removed", name), nil
|
||||
}))
|
||||
|
||||
// A person is added by their address: the page does not list the whole
|
||||
// cluster's accounts to someone who administers one organization.
|
||||
mux.HandleFunc("POST /org/{id}/members", act(func(ctx context.Context, r *http.Request, by store.User, o store.TreeOrg) (notice, error) {
|
||||
scope, sid, ok := strings.Cut(r.FormValue("where"), ":")
|
||||
if !ok || !orgHas(o, store.Scope(scope), sid) {
|
||||
return notice{}, refuse("org.err.not_here")
|
||||
}
|
||||
email := strings.TrimSpace(r.FormValue("email"))
|
||||
u, err := e.UserByEmail(ctx, email)
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
return notice{}, refuse("org.err.no_account", email)
|
||||
}
|
||||
if err != nil {
|
||||
return notice{}, err
|
||||
}
|
||||
switch u.State {
|
||||
case store.UserPending:
|
||||
return notice{}, refuse("org.err.not_approved", u.Email)
|
||||
case store.UserSuspended:
|
||||
return notice{}, refuse("org.err.suspended", u.Email)
|
||||
}
|
||||
if err := e.AddMembership(ctx, u.ID, store.Scope(scope), sid, store.RoleMember, by.Email, time.Now()); err != nil {
|
||||
return notice{}, err
|
||||
}
|
||||
return say("org.msg.added", u.Email, orgPlaceName(o, store.Scope(scope), sid)), nil
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /org/{id}/members/{uid}/role", act(func(ctx context.Context, r *http.Request, by store.User, o store.TreeOrg) (notice, error) {
|
||||
uid, role := r.PathValue("uid"), r.FormValue("role")
|
||||
if !slices.Contains(store.MembershipRoles, role) {
|
||||
return notice{}, refuse("org.err.role")
|
||||
}
|
||||
if uid == by.ID && !by.IsAdmin {
|
||||
// Giving up one's own admin role by a misclick would leave
|
||||
// nobody able to give it back but the cluster's staff.
|
||||
return notice{}, refuse("org.err.self_role")
|
||||
}
|
||||
if err := e.AddMembership(ctx, uid, store.ScopeOrg, o.ID, role, by.Email, time.Now()); err != nil {
|
||||
return notice{}, err
|
||||
}
|
||||
return roleNotice(role, o.Name), nil
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /org/{id}/members/{uid}/leave", act(func(ctx context.Context, r *http.Request, by store.User, o store.TreeOrg) (notice, error) {
|
||||
uid := r.PathValue("uid")
|
||||
scope, sid, ok := strings.Cut(r.FormValue("where"), ":")
|
||||
if !ok || !orgHas(o, store.Scope(scope), sid) {
|
||||
return notice{}, refuse("org.err.not_here")
|
||||
}
|
||||
if uid == by.ID && store.Scope(scope) == store.ScopeOrg && !by.IsAdmin {
|
||||
return notice{}, refuse("org.err.self_leave")
|
||||
}
|
||||
where := orgPlaceName(o, store.Scope(scope), sid)
|
||||
revoked, err := e.RemoveMembership(ctx, uid, store.Scope(scope), sid, time.Now())
|
||||
if err != nil {
|
||||
return notice{}, err
|
||||
}
|
||||
if revoked > 0 {
|
||||
return say("org.msg.taken_out_keys", where, revoked), nil
|
||||
}
|
||||
return say("org.msg.taken_out", where), nil
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /org/{id}/keys/{kid}/revoke", act(func(ctx context.Context, r *http.Request, _ store.User, o store.TreeOrg) (notice, error) {
|
||||
kid := r.PathValue("kid")
|
||||
keys, err := e.AllKeys(ctx)
|
||||
if err != nil {
|
||||
return notice{}, err
|
||||
}
|
||||
i := slices.IndexFunc(keys, func(k store.KeyRow) bool { return k.ID == kid })
|
||||
if i < 0 || !orgHas(o, store.ScopeProject, keys[i].ProjectID) {
|
||||
return notice{}, refuse("org.err.not_here")
|
||||
}
|
||||
if err := e.RevokeKey(ctx, kid, time.Now()); err != nil {
|
||||
return notice{}, err
|
||||
}
|
||||
return say("org.msg.key_revoked", keys[i].Prefix), nil
|
||||
}))
|
||||
}
|
||||
|
||||
// orgAction is the guard every change on an organization's page shares: a
|
||||
// session, the organization's admin role (or the cluster's), the form's CSRF
|
||||
// token, an audit record, and the page drawn again with what happened.
|
||||
func (p *AccountPages) orgAction(s OrgViewer, e OrgEditor, priced Priced, do orgChange) http.HandlerFunc {
|
||||
return p.orgGuard(func(w http.ResponseWriter, r *http.Request, user store.User) {
|
||||
id := r.PathValue("id")
|
||||
if !user.IsAdmin {
|
||||
ok, err := e.AdministersOrg(r.Context(), user.ID, id)
|
||||
if err != nil || !ok {
|
||||
// 404, as for a page they may not see: a manager learns
|
||||
// nothing from trying.
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
}
|
||||
sess, _, _ := p.session(r)
|
||||
if !p.accounts.CheckCSRF(sess, r.FormValue("csrf")) {
|
||||
badForm(w, r)
|
||||
return
|
||||
}
|
||||
p.audited(w, r, user, func(w http.ResponseWriter, r *http.Request) {
|
||||
l := langOf(r)
|
||||
org, ok := orgIn(r.Context(), s, id)
|
||||
if !ok {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
msg, err := do(r.Context(), r, user, org)
|
||||
if err != nil {
|
||||
err = plainError(err)
|
||||
p.log.Warn("a change to an organization was refused", "by", user.Email, "org", org.Name,
|
||||
"path", r.URL.Path, "err", err)
|
||||
noteAudit(r, org.Name, "", err.Error())
|
||||
p.renderOrg(w, r, s, user, id, priced, "", errorIn(l, err))
|
||||
return
|
||||
}
|
||||
p.log.Info("an organization was changed", "by", user.Email, "org", org.Name,
|
||||
"what", msg.in(english))
|
||||
noteAudit(r, org.Name, msg.in(english), "")
|
||||
p.renderOrg(w, r, s, user, id, priced, msg.in(l), "")
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
// orgIn finds an organization in the tree.
|
||||
func orgIn(ctx context.Context, s OrgViewer, id string) (store.TreeOrg, bool) {
|
||||
tree, err := s.Tree(ctx)
|
||||
if err != nil {
|
||||
return store.TreeOrg{}, false
|
||||
}
|
||||
for _, o := range tree {
|
||||
if o.ID == id {
|
||||
return o, true
|
||||
}
|
||||
}
|
||||
return store.TreeOrg{}, false
|
||||
}
|
||||
|
||||
// orgHas reports whether a place is the organization or lies in it. Every
|
||||
// change checks it: an id in a form is the sender's to choose.
|
||||
func orgHas(o store.TreeOrg, scope store.Scope, id string) bool {
|
||||
return orgPlaceName(o, scope, id) != ""
|
||||
}
|
||||
|
||||
// orgPlaceName is a place in the organization by its path, "" when it is not
|
||||
// in it.
|
||||
func orgPlaceName(o store.TreeOrg, scope store.Scope, id string) string {
|
||||
if id == "" {
|
||||
return ""
|
||||
}
|
||||
if scope == store.ScopeOrg {
|
||||
if id == o.ID {
|
||||
return o.Name
|
||||
}
|
||||
return ""
|
||||
}
|
||||
for _, t := range o.Teams {
|
||||
if scope == store.ScopeTeam && t.ID == id {
|
||||
return o.Name + " / " + t.Name
|
||||
}
|
||||
for _, pr := range t.Projects {
|
||||
if scope == store.ScopeProject && pr.ID == id {
|
||||
return o.Name + " / " + t.Name + " / " + pr.Name
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func orgName(r *http.Request) (string, error) {
|
||||
name := strings.TrimSpace(r.FormValue("name"))
|
||||
if name == "" || strings.Contains(name, "/") {
|
||||
return "", refuse("org.err.name")
|
||||
}
|
||||
return name, nil
|
||||
}
|
||||
@@ -0,0 +1,210 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// fakeOrgEditor is fakeOrgs that also changes: u1 administers Acme, u4
|
||||
// manages it, and what each change did is written down.
|
||||
type fakeOrgEditor struct {
|
||||
fakeOrgs
|
||||
admins map[string]string // org -> admin user id
|
||||
did []string
|
||||
revoked []string
|
||||
}
|
||||
|
||||
func (f *fakeOrgEditor) ManagesOrg(_ context.Context, user, org string) (bool, error) {
|
||||
return f.admins[org] == user || f.managers[org] == user, nil
|
||||
}
|
||||
func (f *fakeOrgEditor) ManagedOrgs(ctx context.Context, user string) ([]store.Org, error) {
|
||||
var out []store.Org
|
||||
for _, o := range []store.Org{{ID: "org-a", Name: "Acme"}, {ID: "org-s", Name: "Scania"}} {
|
||||
if ok, _ := f.ManagesOrg(ctx, user, o.ID); ok {
|
||||
out = append(out, o)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
func (f *fakeOrgEditor) AdministersOrg(_ context.Context, user, org string) (bool, error) {
|
||||
return f.admins[org] == user, nil
|
||||
}
|
||||
func (f *fakeOrgEditor) UserByEmail(_ context.Context, email string) (store.User, error) {
|
||||
switch email {
|
||||
case "new@acme":
|
||||
return store.User{ID: "u5", Email: email, State: store.UserActive}, nil
|
||||
case "waiting@acme":
|
||||
return store.User{ID: "u6", Email: email, State: store.UserPending}, nil
|
||||
}
|
||||
return store.User{}, store.ErrNotFound
|
||||
}
|
||||
func (f *fakeOrgEditor) CreateTeam(_ context.Context, org, name string) (string, error) {
|
||||
f.did = append(f.did, "team "+org+" "+name)
|
||||
return "t9", nil
|
||||
}
|
||||
func (f *fakeOrgEditor) CreateProject(_ context.Context, team, name string) (string, error) {
|
||||
f.did = append(f.did, "project "+team+" "+name)
|
||||
return "p9", nil
|
||||
}
|
||||
func (f *fakeOrgEditor) RenameTeam(_ context.Context, id, name string) error {
|
||||
f.did = append(f.did, "rename team "+id+" "+name)
|
||||
return nil
|
||||
}
|
||||
func (f *fakeOrgEditor) RenameProject(_ context.Context, id, name string) error {
|
||||
f.did = append(f.did, "rename project "+id+" "+name)
|
||||
return nil
|
||||
}
|
||||
func (f *fakeOrgEditor) DeleteTeam(_ context.Context, id string) error {
|
||||
f.did = append(f.did, "delete team "+id)
|
||||
return nil
|
||||
}
|
||||
func (f *fakeOrgEditor) DeleteProject(_ context.Context, id string) error {
|
||||
f.did = append(f.did, "delete project "+id)
|
||||
return nil
|
||||
}
|
||||
func (f *fakeOrgEditor) AddMembership(_ context.Context, user string, scope store.Scope, id, role, _ string, _ time.Time) error {
|
||||
f.did = append(f.did, "join "+user+" "+string(scope)+":"+id+" "+role)
|
||||
return nil
|
||||
}
|
||||
func (f *fakeOrgEditor) RemoveMembership(_ context.Context, user string, scope store.Scope, id string, _ time.Time) (int, error) {
|
||||
f.did = append(f.did, "leave "+user+" "+string(scope)+":"+id)
|
||||
return 0, nil
|
||||
}
|
||||
func (f *fakeOrgEditor) AllKeys(context.Context) ([]store.KeyRow, error) {
|
||||
return []store.KeyRow{
|
||||
{Key: store.Key{ID: "k-acme", Prefix: "sk-a", ProjectID: "p1"}, ProjectPath: "Acme / ml / training"},
|
||||
{Key: store.Key{ID: "k-scania", Prefix: "sk-s", ProjectID: "p-s"}},
|
||||
}, nil
|
||||
}
|
||||
func (f *fakeOrgEditor) RevokeKey(_ context.Context, id string, _ time.Time) error {
|
||||
f.revoked = append(f.revoked, id)
|
||||
return nil
|
||||
}
|
||||
|
||||
func orgAdminPage(t *testing.T, user store.User) (*http.ServeMux, *fakeOrgEditor) {
|
||||
t.Helper()
|
||||
f := &fakeOrgEditor{
|
||||
fakeOrgs: fakeOrgs{managers: map[string]string{"org-a": "u4"}},
|
||||
admins: map[string]string{"org-a": "u1"},
|
||||
}
|
||||
p := NewAccountPages(fakeAccounts{user: user}, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
mux := http.NewServeMux()
|
||||
p.RegisterOrgPages(mux, f, nil)
|
||||
return mux, f
|
||||
}
|
||||
|
||||
func form(kv ...string) url.Values {
|
||||
v := url.Values{"csrf": {"tok"}}
|
||||
for i := 0; i+1 < len(kv); i += 2 {
|
||||
v.Set(kv[i], kv[i+1])
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func TestAnOrganizationsAdminIsGivenTheFormsAndAManagerIsNot(t *testing.T) {
|
||||
admin, _ := orgAdminPage(t, store.User{ID: "u1", Email: "boss@acme"})
|
||||
body := mirrorGet(t, admin, "/org/org-a").Body.String()
|
||||
for _, want := range []string{`action="/org/org-a/teams"`, `action="/org/org-a/members"`,
|
||||
`action="/org/org-a/keys/k-acme/revoke"`, "sk-a"} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the admin's page lacks %q", want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(body, "k-scania") {
|
||||
t.Error("a key of another organization is on the page")
|
||||
}
|
||||
|
||||
manager, f := orgAdminPage(t, store.User{ID: "u4", Email: "watch@acme"})
|
||||
w := mirrorGet(t, manager, "/org/org-a")
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("the manager lost their page: %d", w.Code)
|
||||
}
|
||||
if strings.Contains(w.Body.String(), "<form method=\"post\" action=\"/org/") {
|
||||
t.Error("a manager is shown forms they cannot send")
|
||||
}
|
||||
if w := mirrorPost(t, manager, "/org/org-a/teams", form("name", "x")); w.Code != http.StatusNotFound {
|
||||
t.Errorf("a manager changed the organization: %d", w.Code)
|
||||
}
|
||||
if len(f.did) != 0 {
|
||||
t.Errorf("a manager's change reached the store: %v", f.did)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOrganizationsAdminChangesOnlyTheirOwn(t *testing.T) {
|
||||
mux, f := orgAdminPage(t, store.User{ID: "u1", Email: "boss@acme"})
|
||||
|
||||
if w := mirrorPost(t, mux, "/org/org-a/teams", form("name", "vision")); w.Code != http.StatusOK ||
|
||||
!strings.Contains(w.Body.String(), "Team vision added") {
|
||||
t.Errorf("adding a team: %d", w.Code)
|
||||
}
|
||||
if w := mirrorPost(t, mux, "/org/org-s/teams", form("name", "x")); w.Code != http.StatusNotFound {
|
||||
t.Errorf("an admin of Acme added a team to Scania: %d", w.Code)
|
||||
}
|
||||
// Ids in a form are the sender's to choose: a team of another
|
||||
// organization, named on Acme's page, is refused.
|
||||
mirrorPost(t, mux, "/org/org-a/projects", form("team", "t-scania", "name", "x"))
|
||||
mirrorPost(t, mux, "/org/org-a/members", form("email", "new@acme", "where", "org:org-s"))
|
||||
mirrorPost(t, mux, "/org/org-a/keys/k-scania/revoke", form())
|
||||
mirrorPost(t, mux, "/org/org-a/project/p-scania/delete", form())
|
||||
if len(f.revoked) != 0 {
|
||||
t.Errorf("a key of another organization was revoked: %v", f.revoked)
|
||||
}
|
||||
|
||||
mirrorPost(t, mux, "/org/org-a/projects", form("team", "t1", "name", "eval"))
|
||||
mirrorPost(t, mux, "/org/org-a/members", form("email", "new@acme", "where", "project:p1"))
|
||||
mirrorPost(t, mux, "/org/org-a/members/u2/role", form("role", "manager"))
|
||||
mirrorPost(t, mux, "/org/org-a/keys/k-acme/revoke", form())
|
||||
want := []string{"team org-a vision", "project t1 eval", "join u5 project:p1 member", "join u2 org:org-a manager"}
|
||||
if strings.Join(f.did, "|") != strings.Join(want, "|") {
|
||||
t.Errorf("the store was asked\n %v\nwant\n %v", f.did, want)
|
||||
}
|
||||
if len(f.revoked) != 1 || f.revoked[0] != "k-acme" {
|
||||
t.Errorf("revoked %v", f.revoked)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOrganizationsAdminAddsOnlyApprovedPeople(t *testing.T) {
|
||||
mux, f := orgAdminPage(t, store.User{ID: "u1", Email: "boss@acme"})
|
||||
for email, says := range map[string]string{
|
||||
"nobody@acme": "there is no account for nobody@acme",
|
||||
"waiting@acme": "waits for the cluster",
|
||||
} {
|
||||
w := mirrorPost(t, mux, "/org/org-a/members", form("email", email, "where", "org:org-a"))
|
||||
if !strings.Contains(w.Body.String(), says) {
|
||||
t.Errorf("%s: the page does not say %q", email, says)
|
||||
}
|
||||
}
|
||||
if len(f.did) != 0 {
|
||||
t.Errorf("someone not approved was added: %v", f.did)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOrganizationsAdminCannotLockThemselvesOut(t *testing.T) {
|
||||
mux, f := orgAdminPage(t, store.User{ID: "u1", Email: "boss@acme"})
|
||||
mirrorPost(t, mux, "/org/org-a/members/u1/role", form("role", "member"))
|
||||
mirrorPost(t, mux, "/org/org-a/members/u1/leave", form("where", "org:org-a"))
|
||||
if len(f.did) != 0 {
|
||||
t.Errorf("an admin took their own role away: %v", f.did)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAViewerIsShownNoFormsOnThePeoplePage(t *testing.T) {
|
||||
for role, wantForms := range map[string]bool{store.RoleViewer: false, store.RoleAdmin: true} {
|
||||
u := store.User{ID: "s", Email: "s@x", Role: role, IsAdmin: role == store.RoleAdmin}
|
||||
p := NewAccountPages(fakeAccounts{user: u}, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
mux := http.NewServeMux()
|
||||
p.RegisterAdminUsers(mux, &fakeAdmin{})
|
||||
body := mirrorGet(t, mux, "/admin/users").Body.String()
|
||||
if got := strings.Contains(body, `method="post" action="/admin`); got != wantForms {
|
||||
t.Errorf("%s: forms on the page = %v, want %v", role, got, wantForms)
|
||||
}
|
||||
}
|
||||
}
|
||||
+72
-10
@@ -10,7 +10,8 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// OrgViewer is what an organization's page reads. Nothing on it writes.
|
||||
// OrgViewer is what an organization's page reads. What its admin changes
|
||||
// goes through OrgEditor, in org_admin.go.
|
||||
type OrgViewer interface {
|
||||
ManagesOrg(ctx context.Context, userID, orgID string) (bool, error)
|
||||
ManagedOrgs(ctx context.Context, userID string) ([]store.Org, error)
|
||||
@@ -27,12 +28,18 @@ type OrgViewer interface {
|
||||
// orgMember is a person as an organization's page lists them: once, with
|
||||
// every place in the organization they are a member of.
|
||||
type orgMember struct {
|
||||
ID string
|
||||
Email string
|
||||
State store.UserState
|
||||
Places []orgPlace
|
||||
// OrgRole is their role on the organization itself, "" when they are
|
||||
// only in its teams or projects.
|
||||
OrgRole string
|
||||
}
|
||||
|
||||
type orgPlace struct{ Where, Role string }
|
||||
// orgPlace is one place a member is in: its path within the organization,
|
||||
// the role there, and the "scope:id" a form names it by.
|
||||
type orgPlace struct{ Where, Role, Value string }
|
||||
|
||||
// A member's state is a word the page shows and a class it styles by.
|
||||
func init() {
|
||||
@@ -58,6 +65,16 @@ type orgPageData struct {
|
||||
Members []orgMember
|
||||
People int
|
||||
|
||||
// Edit is whether this person may change the organization; the rest is
|
||||
// what the forms for that need.
|
||||
Edit bool
|
||||
Me string // the person looking, whose own role is not theirs to change
|
||||
CSRF string
|
||||
Teams []store.TreeTeam
|
||||
Places []scopeOption
|
||||
Keys []keyRow
|
||||
Message string
|
||||
|
||||
Error string
|
||||
Version string
|
||||
Now string
|
||||
@@ -66,12 +83,16 @@ type orgPageData struct {
|
||||
// RegisterOrgPages mounts /org: an organization's own view of itself, for the
|
||||
// people who manage it.
|
||||
//
|
||||
// Read-only on purpose. A manager sees their organization's use, members and
|
||||
// limits — what they are accountable for — without being made an administrator
|
||||
// of the whole cluster, and without anything on the page that changes who may
|
||||
// do what. That stays with the cluster's administrators.
|
||||
// A manager sees their organization's use, members and limits — what they are
|
||||
// accountable for — without being made an administrator of the whole cluster.
|
||||
// An organization's admin also changes it here (org_admin.go); its limits stay
|
||||
// with the cluster's administrators.
|
||||
func (p *AccountPages) RegisterOrgPages(mux *http.ServeMux, s OrgViewer, priced Priced) {
|
||||
p.orgs = s
|
||||
if e, ok := s.(OrgEditor); ok {
|
||||
p.orgEditor = e
|
||||
p.registerOrgAdmin(mux, s, e, priced)
|
||||
}
|
||||
mux.HandleFunc("GET /org", p.orgGuard(func(w http.ResponseWriter, r *http.Request, user store.User) {
|
||||
if p.toPeople(w, r, user, "") {
|
||||
return
|
||||
@@ -107,7 +128,7 @@ func (p *AccountPages) RegisterOrgPages(mux *http.ServeMux, s OrgViewer, priced
|
||||
return
|
||||
}
|
||||
}
|
||||
p.renderOrg(w, r, s, user, id, priced)
|
||||
p.renderOrg(w, r, s, user, id, priced, "", "")
|
||||
}))
|
||||
}
|
||||
|
||||
@@ -149,7 +170,7 @@ func (p *AccountPages) orgGuard(next func(http.ResponseWriter, *http.Request, st
|
||||
}
|
||||
|
||||
func (p *AccountPages) renderOrg(w http.ResponseWriter, r *http.Request, s OrgViewer,
|
||||
user store.User, orgID string, priced Priced,
|
||||
user store.User, orgID string, priced Priced, msg, errMsg string,
|
||||
) {
|
||||
ctx := r.Context()
|
||||
by := store.Grouping(r.URL.Query().Get("by"))
|
||||
@@ -162,6 +183,7 @@ func (p *AccountPages) renderOrg(w http.ResponseWriter, r *http.Request, s OrgVi
|
||||
Nav: navFrom(r, "org"),
|
||||
By: string(by), Window: win.Key,
|
||||
Priced: priced != nil && priced.AnythingPriced(),
|
||||
Message: msg, Error: errMsg,
|
||||
Version: p.version,
|
||||
Now: time.Now().UTC().Format(time.RFC3339),
|
||||
}
|
||||
@@ -217,10 +239,13 @@ func (p *AccountPages) renderOrg(w http.ResponseWriter, r *http.Request, s OrgVi
|
||||
om := byPerson[m.UserID]
|
||||
if om == nil {
|
||||
u := byID[m.UserID]
|
||||
om = &orgMember{Email: u.Email, State: u.State}
|
||||
om = &orgMember{ID: u.ID, Email: u.Email, State: u.State}
|
||||
byPerson[m.UserID] = om
|
||||
}
|
||||
om.Places = append(om.Places, orgPlace{Where: where, Role: m.Role})
|
||||
om.Places = append(om.Places, orgPlace{Where: where, Role: m.Role, Value: string(m.Scope) + ":" + m.ScopeID})
|
||||
if m.Scope == store.ScopeOrg {
|
||||
om.OrgRole = m.Role
|
||||
}
|
||||
}
|
||||
for _, om := range byPerson {
|
||||
d.Members = append(d.Members, *om)
|
||||
@@ -228,6 +253,8 @@ func (p *AccountPages) renderOrg(w http.ResponseWriter, r *http.Request, s OrgVi
|
||||
sort.Slice(d.Members, func(i, j int) bool { return d.Members[i].Email < d.Members[j].Email })
|
||||
d.People = len(byPerson)
|
||||
|
||||
p.orgEditing(r, &d, user, mine)
|
||||
|
||||
rows, err := s.UsageBucketsIn(ctx, by, orgID, time.Now().Add(-win.Span), win.Width)
|
||||
if err != nil {
|
||||
p.log.Error("could not read an organization's usage", "org", orgID, "err", err)
|
||||
@@ -236,3 +263,38 @@ func (p *AccountPages) renderOrg(w http.ResponseWriter, r *http.Request, s OrgVi
|
||||
d.Requests, d.Tokens, d.Spend, d.Rows = usageChartsIn(l, rows)
|
||||
render(w, r, "org", d)
|
||||
}
|
||||
|
||||
// orgEditing fills what the page's forms need, when this person may change
|
||||
// the organization.
|
||||
func (p *AccountPages) orgEditing(r *http.Request, d *orgPageData, user store.User, mine []store.TreeOrg) {
|
||||
ctx, e := r.Context(), p.orgEditor
|
||||
if e == nil || len(mine) != 1 {
|
||||
return
|
||||
}
|
||||
o := mine[0]
|
||||
if !user.IsAdmin {
|
||||
if ok, err := e.AdministersOrg(ctx, user.ID, o.ID); err != nil || !ok {
|
||||
return
|
||||
}
|
||||
}
|
||||
d.Edit, d.Teams, d.Me = true, o.Teams, user.ID
|
||||
d.CSRF = csrfFor(r)
|
||||
d.Places = append(d.Places, scopeOption{Value: "org:" + o.ID, Label: o.Name})
|
||||
for _, t := range o.Teams {
|
||||
d.Places = append(d.Places, scopeOption{Value: "team:" + t.ID, Label: o.Name + " / " + t.Name})
|
||||
for _, pr := range t.Projects {
|
||||
d.Places = append(d.Places, scopeOption{Value: "project:" + pr.ID, Label: o.Name + " / " + t.Name + " / " + pr.Name})
|
||||
}
|
||||
}
|
||||
keys, err := e.AllKeys(ctx)
|
||||
if err != nil {
|
||||
p.log.Error("could not read the keys", "err", err)
|
||||
return
|
||||
}
|
||||
now := time.Now()
|
||||
for _, k := range keys {
|
||||
if orgHas(o, store.ScopeProject, k.ProjectID) && k.RevokedAt.IsZero() {
|
||||
d.Keys = append(d.Keys, keyRow{KeyRow: k, State: keyState(k.Key, now)})
|
||||
}
|
||||
}
|
||||
}
|
||||
+34
-2
@@ -24,10 +24,42 @@ func staffMay(u store.User, r *http.Request) bool {
|
||||
case store.RoleViewer:
|
||||
return read
|
||||
case store.RoleOperator:
|
||||
return read || strings.HasPrefix(path, "/admin/mirror/") ||
|
||||
strings.HasPrefix(path, "/admin/nodes/") && strings.HasSuffix(path, "/engines")
|
||||
return read || staffMayChange(u.Role, path)
|
||||
case store.RoleBilling:
|
||||
return read && (path == "/admin/usage" || path == "/admin/quotas")
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// staffMayChange reports whether a staff role may send a change to an
|
||||
// administration path. It is staffMay's rule for everything but reading, and
|
||||
// what the pages ask before they draw a form, so nobody is shown a form whose
|
||||
// answer would be "not found".
|
||||
func staffMayChange(role, path string) bool {
|
||||
switch role {
|
||||
case store.RoleAdmin:
|
||||
return true
|
||||
case store.RoleOperator:
|
||||
return strings.HasPrefix(path, "/admin/mirror/") ||
|
||||
strings.HasPrefix(path, "/admin/nodes/") && strings.HasSuffix(path, "/engines")
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// csrfIfMay is the form token for a page whose forms send to path, or ""
|
||||
// when this person may not send them: a page draws a form only where it has a
|
||||
// token for it, so a viewer is drawn the page without its forms.
|
||||
func csrfIfMay(r *http.Request, path string) string {
|
||||
if !navFrom(r, "").May(path) {
|
||||
return ""
|
||||
}
|
||||
return csrfFor(r)
|
||||
}
|
||||
|
||||
// May reports whether the person this menu is for may send a change to an
|
||||
// administration path: a template asks it before drawing a form, with the
|
||||
// form's action ("/admin/mirror/fetch"); a part of the path that varies may be
|
||||
// anything ("/admin/nodes/x/engines").
|
||||
func (n Nav) May(path string) bool {
|
||||
return n.Admin && staffMayChange(n.Role, path)
|
||||
}
|
||||
@@ -52,19 +52,30 @@ footer { margin-top: 3rem; padding-top: 1rem; border-top: 1px solid var(--line);
|
||||
button {
|
||||
font: inherit; font-weight: 600; cursor: pointer;
|
||||
background: var(--accent); color: var(--bg); border: none;
|
||||
border-radius: 6px; padding: .55rem 1.1rem;
|
||||
border-radius: 6px; padding: .4rem .9rem; font-size: .875rem;
|
||||
}
|
||||
button:hover { filter: brightness(1.1); }
|
||||
ul { padding-left: 1.2rem; }
|
||||
li { margin: .25rem 0; }
|
||||
a { color: var(--accent); }
|
||||
.wrap.narrow { max-width: 30rem; }
|
||||
label { display: block; margin: .9rem 0; font-size: .9rem; font-weight: 600; }
|
||||
/* Forms are compact: a label sits close to its field, a field is as wide as
|
||||
what goes in it rather than the page, and a short form fits on one line
|
||||
(form.row). The simplest form used to take a screen's height. */
|
||||
label { display: block; margin: .5rem 0; font-size: .85rem; font-weight: 600; }
|
||||
input[type=text], input[type=email], input[type=password], input[type=search], input[type=number] {
|
||||
display: block; width: 100%; margin-top: .3rem; font: inherit; font-weight: 400;
|
||||
padding: .5rem .6rem; border: 1px solid var(--line); border-radius: 6px;
|
||||
display: block; width: 100%; max-width: 30rem; margin-top: .2rem; font: inherit; font-size: .9rem;
|
||||
font-weight: 400; padding: .32rem .5rem; border: 1px solid var(--line); border-radius: 6px;
|
||||
background: var(--card); color: var(--fg);
|
||||
}
|
||||
input[type=number] { max-width: 10rem; }
|
||||
form.row { display: flex; flex-wrap: wrap; gap: .4rem .6rem; align-items: flex-end; margin: .5rem 0; }
|
||||
form.row label { margin: 0; }
|
||||
form.row input[type=text], form.row input[type=email], form.row input[type=password],
|
||||
form.row input[type=number], form.row select { width: auto; min-width: 12rem; margin: 0; }
|
||||
form.row input[type=number] { min-width: 6rem; width: 7rem; }
|
||||
form.row > button, form.row label > button { margin: 0; }
|
||||
form.row .grow { flex: 1 1 16rem; }
|
||||
input:focus { outline: 2px solid var(--accent); outline-offset: 1px; }
|
||||
.error { color: var(--bad); border: 1px solid var(--bad); border-radius: 6px;
|
||||
padding: .6rem .8rem; background: var(--card); }
|
||||
@@ -74,7 +85,7 @@ button.link { background: none; color: var(--accent); padding: 0;
|
||||
button.link.danger { color: var(--bad); }
|
||||
.ok { color: var(--ok); border: 1px solid var(--ok); border-radius: 6px;
|
||||
padding: .6rem .8rem; background: var(--card); }
|
||||
select { font: inherit; padding: .35rem .4rem; border: 1px solid var(--line);
|
||||
select { font: inherit; font-size: .9rem; padding: .28rem .4rem; border: 1px solid var(--line);
|
||||
border-radius: 6px; background: var(--card); color: var(--fg); }
|
||||
.chart-wrap { position: relative; margin: .5rem 0 1.5rem; }
|
||||
.chart { width: 100%; height: 120px; display: block;
|
||||
@@ -277,8 +288,11 @@ tr:has(+ .url-row) td, tr:has(+ .error-row) td { border-bottom: none; }
|
||||
/* The fetch form's three short fields sit on one line on a wide screen and
|
||||
stack on a narrow one, because two of them are a word long and a full-width
|
||||
input for "2026-09" reads as though more is wanted. */
|
||||
.field-row { display: flex; gap: 1rem; flex-wrap: wrap; }
|
||||
.field-row label { flex: 1 1 12rem; }
|
||||
.field-row { display: flex; gap: .4rem .75rem; flex-wrap: wrap; align-items: flex-end; }
|
||||
.field-row label { flex: 0 1 12rem; margin: .3rem 0; }
|
||||
.field-row label.short { flex: 0 0 7.5rem; }
|
||||
.field-row label.grow { flex: 1 1 14rem; }
|
||||
.field-row .field-gap { flex: 0 0 1rem; }
|
||||
.opt { font-weight: 400; color: var(--muted); font-size: .8rem; }
|
||||
|
||||
/* The row whose quota form is open. Tinted rather than outlined: a border
|
||||
@@ -334,6 +348,8 @@ details.edit > summary::-webkit-details-marker { display: none; }
|
||||
details.edit[open] > summary { color: var(--muted); }
|
||||
details.edit form { display: flex; gap: .4rem; align-items: center; margin: .35rem 0; }
|
||||
details.edit input[type=text] { width: 14rem; }
|
||||
details.edit form.q-form { display: block; }
|
||||
details.edit form.q-form input[type=text] { width: 100%; }
|
||||
details.project { margin: .45rem 0; border: 1px solid var(--line); border-radius: 6px;
|
||||
background: var(--bg); }
|
||||
details.project > summary { cursor: pointer; padding: .45rem .7rem; display: flex;
|
||||
@@ -418,10 +434,11 @@ table.quota-tree tr.q-depth-0 .q-name { font-weight: 600; }
|
||||
.q-meter.hot i { background: var(--bad); }
|
||||
.q-pct { font-size: .78rem; color: var(--muted); }
|
||||
.q-act { text-align: right; white-space: nowrap; }
|
||||
.q-form { max-width: 46rem; }
|
||||
.q-form { max-width: 62rem; }
|
||||
.q-form .field-row { align-items: flex-end; }
|
||||
.q-form .field-row label { flex: 1 1 8rem; }
|
||||
.q-form .field-row label { flex: 0 1 8rem; }
|
||||
.q-form .field-row label.q-period { flex: 0 0 auto; }
|
||||
.q-form .field-row label.grow { flex: 1 1 14rem; }
|
||||
.q-form-title { margin: .2rem 0 .4rem; font-size: .88rem; }
|
||||
.q-form-actions { display: flex; gap: 1rem; align-items: center; margin-top: .6rem; }
|
||||
.q-form-actions .danger { margin-left: auto; }
|
||||
@@ -757,6 +774,7 @@ table.gpu-list { width: auto; flex: 1 1 22rem; }
|
||||
form.add-to { display: flex; gap: .35rem; align-items: center; flex-wrap: nowrap; }
|
||||
form.add-to select { width: auto; max-width: 13rem; margin: 0; padding: .25rem .4rem; font-size: .8rem; }
|
||||
form.add-to button { padding: .25rem .6rem; font-size: .78rem; }
|
||||
form.add-to input[type=text] { width: 11rem; margin: 0; padding: .22rem .45rem; font-size: .8rem; }
|
||||
details.row-menu { position: relative; display: inline-block; }
|
||||
details.row-menu > summary { list-style: none; cursor: pointer; padding: .1rem .55rem; border-radius: 6px;
|
||||
color: var(--muted); font-weight: 700; letter-spacing: .05em; user-select: none; }
|
||||
@@ -813,7 +831,8 @@ details.chat-settings output { font-variant-numeric: tabular-nums; font-weight:
|
||||
.copyable { position: relative; }
|
||||
.copyable pre { margin-bottom: .25rem; overflow-x: auto; }
|
||||
.copyable button { margin-right: .75rem; font-size: .8rem; }
|
||||
.connect-form .field-row label { flex: 1 1 12rem; }
|
||||
.connect-form .field-row label { flex: 0 1 auto; }
|
||||
.connect-form .field-row > button { margin: .3rem 0; }
|
||||
table.connect-fields th { white-space: nowrap; }
|
||||
|
||||
/* The card opened beneath a held version on the mirror page. */
|
||||
@@ -941,3 +960,10 @@ a.sel-link.small { font-size: .78rem; color: var(--muted); margin-left: .4rem; }
|
||||
.meshes-picture input { display: block; margin-top: .3rem; font-weight: 400; }
|
||||
.meshes-preview { max-width: min(100%, 16rem); max-height: 16rem; border: 1px solid var(--line); border-radius: 6px; }
|
||||
.meshes-viewer { display: block; width: 100%; height: 22rem; background: var(--card); }
|
||||
|
||||
/* A role chosen inside a membership chip. */
|
||||
select.chip-sel { font-size: .75rem; padding: 0 .2rem; margin-left: .3rem; border-radius: 4px; }
|
||||
fieldset { border: 1px solid var(--line); border-radius: 6px; padding: .3rem .8rem .5rem; margin: .5rem 0; }
|
||||
legend { font-size: .82rem; font-weight: 600; padding: 0 .3rem; }
|
||||
fieldset label.check { margin: .2rem 0; font-weight: 400; font-size: .85rem; }
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
<a href="#install">{{t "about.nav.install"}}</a>
|
||||
<a href="#proxy">{{t "about.nav.proxy"}}</a>
|
||||
<a href="#download">{{t "about.nav.download"}}</a>
|
||||
<a href="{{.DeckURL}}" type="application/pdf">{{t "about.nav.deck"}}</a>
|
||||
<a href="https://gitea.mynodes.xyz/ilker/llmgw">{{t "about.nav.source"}}</a>
|
||||
{{if not .Site}}<a href="/">{{t "about.nav.gateway"}}</a>{{end}}
|
||||
</nav>
|
||||
@@ -34,6 +35,7 @@
|
||||
<div class="cta">
|
||||
<a class="btn" href="#download">{{t "about.hero.download"}}{{with .Release}} {{.}}{{end}}</a>
|
||||
<a class="btn ghost" href="{{.InstallURL}}">{{t "about.hero.guide"}}</a>
|
||||
<a class="btn ghost" href="{{.DeckURL}}" type="application/pdf">{{t "about.hero.deck"}}</a>
|
||||
</div>
|
||||
<div class="oneline" aria-label="{{t "about.hero.oneline_label"}}">
|
||||
<code><span class="p">$ </span>sudo llmgw server init --admin you@example.com</code>
|
||||
|
||||
@@ -119,8 +119,8 @@
|
||||
{{else}}<input type="text" name="model" placeholder="{{t "connect.model_placeholder"}}" required>{{end}}</label>
|
||||
{{if .Contexts}}<label>{{t "connect.spends_against"}}
|
||||
<select name="context">{{range .Contexts}}<option value="{{.Value}}">{{.Label}}</option>{{end}}</select></label>{{end}}
|
||||
<button type="submit">{{t "connect.submit"}}</button>
|
||||
</div>
|
||||
<button type="submit">{{t "connect.submit"}}</button>
|
||||
</form>
|
||||
<script>
|
||||
// Copy and download for what the connect panel shows, done in the page: the
|
||||
@@ -145,7 +145,7 @@ document.addEventListener("click", (e) => {
|
||||
</script>
|
||||
|
||||
<h2>{{t "profile.create_key"}}</h2>
|
||||
<form method="post" action="/profile/keys" class="narrow">
|
||||
<form method="post" action="/profile/keys" class="row">
|
||||
<input type="hidden" name="csrf" value="{{.CSRF}}">
|
||||
<label>{{t "profile.key_for"}}<input type="text" name="name" placeholder="{{t "profile.key_for_placeholder"}}" required></label>
|
||||
{{if .Contexts}}
|
||||
@@ -235,7 +235,7 @@ document.addEventListener("click", (e) => {
|
||||
<h2>{{t "profile.password"}}</h2>
|
||||
{{if .PasswordChanged}}<p>{{t "profile.password_changed"}}</p>{{end}}
|
||||
{{if .PasswordError}}<p class="error">{{.PasswordError}}</p>{{end}}
|
||||
<form method="post" action="/profile/password" autocomplete="on" class="narrow">
|
||||
<form method="post" action="/profile/password" autocomplete="on" class="row">
|
||||
<input type="hidden" name="csrf" value="{{.CSRF}}">
|
||||
<label>{{t "profile.current_password"}}<input type="password" name="current" required
|
||||
autocomplete="current-password"></label>
|
||||
|
||||
@@ -71,7 +71,7 @@
|
||||
{{else}}
|
||||
{{/* Submitted on change: the choice is the action, and a second click
|
||||
to confirm it would only be a way to forget to. */}}
|
||||
<form method="post" action="/admin/mirror/mode" class="inline">
|
||||
{{if $.CSRF}}<form method="post" action="/admin/mirror/mode" class="inline">
|
||||
<input type="hidden" name="csrf" value="{{$.CSRF}}">
|
||||
<input type="hidden" name="model" value="{{.Name}}">
|
||||
<input type="hidden" name="replicas" value="{{.Replicas}}">
|
||||
@@ -81,7 +81,7 @@
|
||||
<option value="stopped"{{if eq .Mode "stopped"}} selected{{end}}>{{t "mirror.mode.stopped"}}</option>
|
||||
</select>
|
||||
{{/* The button is in the catalogue: what a noscript holds is read as text, not markup. */}}<noscript>{{th "mirror.use.set"}}</noscript>
|
||||
</form>
|
||||
</form>{{end}}
|
||||
{{end}}
|
||||
{{if .Fleet}}<span class="runs-now{{if or .Parked .Unplaceable}} bad{{else if lt .Ready .Wanted}} short{{end}}" title="{{if .Starting}}{{t "mirror.use.runs_title_starting"}}{{else}}{{t "mirror.use.runs_title"}}{{end}}">
|
||||
{{- if .Parked}}{{t "mirror.use.parked"}}
|
||||
@@ -99,24 +99,24 @@
|
||||
whose only outcome is a confusing error. Starting the same download
|
||||
again is what somebody looking at a failed row wants, and the mirror
|
||||
allows it: only a download still running is refused. */}}
|
||||
<form method="post" action="/admin/mirror/fetch" class="inline">
|
||||
{{if $.CSRF}}<form method="post" action="/admin/mirror/fetch" class="inline">
|
||||
<input type="hidden" name="csrf" value="{{$.CSRF}}">
|
||||
<input type="hidden" name="source" value="{{.Source}}">
|
||||
<input type="hidden" name="model" value="{{.Model}}">
|
||||
<input type="hidden" name="version" value="{{.Version}}">
|
||||
<input type="hidden" name="file" value="{{.File}}">
|
||||
<button type="submit" class="link">{{t "mirror.retry"}}</button>
|
||||
</form>
|
||||
</form>{{end}}
|
||||
{{else if eq (printf "%s" .State) "present"}}
|
||||
<button type="button" class="link copy-url" data-url="{{.URL}}" title="{{.URL}}">{{t "mirror.copy_address"}}</button>
|
||||
{{if and .UsedBy (not .Detail)}}<a href="/admin/mirror?tab=held&open={{.Open}}">{{t "mirror.settings.open"}}</a>{{end}}
|
||||
{{if and $.HasCatalog (not .UsedBy)}}{{if .Serve}}<a href="/admin/mirror">{{t "mirror.close"}}</a>
|
||||
{{else}}<a href="/admin/mirror?serve={{.Model}}:{{.Version}}">{{t "mirror.serve"}}</a>{{end}}{{end}}
|
||||
<form method="post" action="/admin/mirror/{{.Model}}/{{.Version}}/remove"
|
||||
{{if $.CSRF}}<form method="post" action="/admin/mirror/{{.Model}}/{{.Version}}/remove"
|
||||
class="inline" data-confirm="{{if .UsedBy}}{{t "mirror.remove_confirm_in_use" .Model .Version}}{{else}}{{t "mirror.remove_confirm" .Model .Version}}{{end}}">
|
||||
<input type="hidden" name="csrf" value="{{$.CSRF}}">
|
||||
<button type="submit" class="link danger">{{t "mirror.remove"}}</button>
|
||||
</form>
|
||||
</form>{{end}}
|
||||
{{end}}
|
||||
</td>
|
||||
</tr>
|
||||
@@ -130,7 +130,7 @@
|
||||
<p class="error">{{t "serve.refusal" .Refusal}}</p>
|
||||
{{else}}
|
||||
{{if .Warning}}<p class="note error-row">{{.Warning}}.</p>{{end}}
|
||||
<form method="post" action="/admin/mirror/serve" class="serve-form" data-engine="{{.Engine}}" data-free-gpus="{{.FreeGPUs}}">
|
||||
{{if $.CSRF}}<form method="post" action="/admin/mirror/serve" class="serve-form" data-engine="{{.Engine}}" data-free-gpus="{{.FreeGPUs}}">
|
||||
<input type="hidden" name="csrf" value="{{$.CSRF}}">
|
||||
<input type="hidden" name="source" value="{{.Source}}">
|
||||
<input type="hidden" name="engine" value="{{.Engine}}">
|
||||
@@ -191,7 +191,7 @@
|
||||
</div>
|
||||
<p class="sub serve-summary" hidden></p>
|
||||
<button type="submit">{{t "serve.add"}}</button>
|
||||
</form>
|
||||
</form>{{end}}
|
||||
<script>
|
||||
// What it will run as, in one line, kept up to date as the form changes:
|
||||
// engine, where, how many cards, memory, context and how it is shared. The
|
||||
@@ -473,18 +473,18 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
<td>{{.DownloadGB}} GB</td>
|
||||
<td>{{.RAMGB}} GB</td>
|
||||
<td>{{if .Fits}}{{.Runs}}{{else}}<span class="opt">{{.Runs}}</span>{{end}}</td>
|
||||
<td>{{if and $fam.FetchesItself .Fits}}<form method="post" action="/admin/mirror/strata" class="inline">
|
||||
<td>{{if and $fam.FetchesItself .Fits}}{{if $.CSRF}}<form method="post" action="/admin/mirror/strata" class="inline">
|
||||
<input type="hidden" name="csrf" value="{{$.CSRF}}">
|
||||
<input type="hidden" name="tab" value="strata">
|
||||
<input type="hidden" name="action" value="serve">
|
||||
<input type="hidden" name="family" value="{{$fam.Name}}">
|
||||
<input type="hidden" name="size" value="{{.Size}}">
|
||||
<button type="submit">{{t "strata.serve"}}</button></form>{{end}}{{if and $fam.Fetchable .Fits}}<form method="post" action="/admin/mirror/strata" class="inline">
|
||||
<button type="submit">{{t "strata.serve"}}</button></form>{{end}}{{end}}{{if and $fam.Fetchable .Fits}}{{if $.CSRF}}<form method="post" action="/admin/mirror/strata" class="inline">
|
||||
<input type="hidden" name="csrf" value="{{$.CSRF}}">
|
||||
<input type="hidden" name="tab" value="strata">
|
||||
<input type="hidden" name="family" value="{{$fam.Name}}">
|
||||
<input type="hidden" name="size" value="{{.Size}}">
|
||||
<button type="submit">{{t "strata.fetch"}}</button></form>{{end}}</td>
|
||||
<button type="submit">{{t "strata.fetch"}}</button></form>{{end}}{{end}}</td>
|
||||
</tr>
|
||||
{{end}}
|
||||
</table>
|
||||
@@ -493,7 +493,7 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
|
||||
{{if eq .Tab "fetch"}}
|
||||
<p class="note">{{t "mirror.fetch.intro"}}</p>
|
||||
<form method="post" action="/admin/mirror/fetch">
|
||||
{{if .CSRF}}<form method="post" action="/admin/mirror/fetch">
|
||||
<input type="hidden" name="csrf" value="{{.CSRF}}">
|
||||
<input type="hidden" name="tab" value="fetch">
|
||||
<label>{{t "mirror.fetch.source"}}
|
||||
@@ -508,7 +508,7 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
<label class="inline-check"><input type="checkbox" name="force" value="1">
|
||||
{{t "mirror.fetch.force"}} <span class="opt">{{t "mirror.fetch.force_hint"}}</span></label>
|
||||
<button type="submit">{{t "mirror.fetch.start"}}</button>
|
||||
</form>
|
||||
</form>{{end}}
|
||||
<p class="note">{{t "mirror.fetch.note"}}</p>
|
||||
{{end}}
|
||||
|
||||
@@ -552,7 +552,7 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
<span class="clip-inline" title="{{.Verdict.Reason}}">{{.Verdict.Reason}}</span>
|
||||
{{else}}
|
||||
{{if .Verdict.Architecture}}<span class="opt" title="{{.Verdict.Reason}}">{{.Verdict.Architecture}}{{if .Verdict.Checked}} ✓{{end}}</span>{{end}}
|
||||
<form method="post" action="/admin/mirror/fetch" class="inline">
|
||||
{{if $.CSRF}}<form method="post" action="/admin/mirror/fetch" class="inline">
|
||||
<input type="hidden" name="csrf" value="{{$.CSRF}}">
|
||||
<input type="hidden" name="tab" value="explore">
|
||||
{{range .Sources}}<input type="hidden" name="source" value="{{.URL}}">
|
||||
@@ -561,7 +561,7 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
<input type="text" name="model" value="{{.Model}}" size="24">
|
||||
<input type="text" name="version" value="{{.Version}}" size="10">
|
||||
<button type="submit">{{t "card.fetch"}}</button>
|
||||
</form>
|
||||
</form>{{end}}
|
||||
{{end}}
|
||||
</td>
|
||||
</tr>{{end}}
|
||||
@@ -628,17 +628,17 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
<td><code>{{.Name}}</code></td>
|
||||
<td class="clip" title="{{.Source}}">{{.Source}}</td>
|
||||
<td class="num">{{.EffectiveScale}}</td>
|
||||
<td class="actions"><form method="post" action="/admin/mirror/adapters" class="inline">
|
||||
<td class="actions">{{if $csrf}}<form method="post" action="/admin/mirror/adapters" class="inline">
|
||||
<input type="hidden" name="csrf" value="{{$csrf}}">
|
||||
<input type="hidden" name="model" value="{{$mname}}">
|
||||
<input type="hidden" name="open" value="{{$row.Open}}">
|
||||
<input type="hidden" name="op" value="remove">
|
||||
<input type="hidden" name="name" value="{{$a.Name}}">
|
||||
<button type="submit" class="link danger">{{t "mirror.remove"}}</button></form></td>
|
||||
<button type="submit" class="link danger">{{t "mirror.remove"}}</button></form>{{end}}</td>
|
||||
</tr>{{end}}</table>
|
||||
{{else}}<p class="note">{{t "mirror.adapters.none"}}</p>{{end}}
|
||||
{{if .AdapterChoices}}{{$m := .}}
|
||||
<form method="post" action="/admin/mirror/adapters" class="adapter-form">
|
||||
{{if $csrf}}<form method="post" action="/admin/mirror/adapters" class="adapter-form">
|
||||
<input type="hidden" name="csrf" value="{{$csrf}}">
|
||||
<input type="hidden" name="model" value="{{.Name}}">
|
||||
<input type="hidden" name="open" value="{{$row.Open}}">
|
||||
@@ -651,14 +651,14 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
<label>{{t "mirror.adapters.scale"}} <input type="text" name="scale" value="1" size="4"></label>
|
||||
<button type="submit">{{t "mirror.adapters.attach"}}</button>
|
||||
</div>
|
||||
</form>{{else}}<p class="note">{{t "mirror.adapters.no_choice"}}</p>{{end}}
|
||||
</form>{{end}}{{else}}<p class="note">{{t "mirror.adapters.no_choice"}}</p>{{end}}
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
{{if .Declared}}
|
||||
<p class="note">{{t "mirror.detail.declared"}}</p>
|
||||
{{else}}
|
||||
<form method="post" action="/admin/mirror/replicas" class="replicas-form">
|
||||
{{if $csrf}}<form method="post" action="/admin/mirror/replicas" class="replicas-form">
|
||||
<input type="hidden" name="csrf" value="{{$csrf}}">
|
||||
<input type="hidden" name="model" value="{{.Name}}">
|
||||
<input type="hidden" name="open" value="{{$row.Open}}">
|
||||
@@ -688,7 +688,7 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
<button type="submit">{{t "common.save"}}</button>
|
||||
</div>
|
||||
<p class="note">{{if .Split}}{{t "mirror.split.scaling_note"}} {{end}}{{if .MostNow}}{{if and .Headroom .Headroom.ByPreempting}}{{t "mirror.form.most_now_preempting" .MostNow}}{{else}}{{t "mirror.form.most_now" .MostNow}}{{end}} {{end}}{{t "mirror.form.note"}}</p>
|
||||
</form>
|
||||
</form>{{end}}
|
||||
{{template "mirror_model_settings" dict "M" . "Row" $row "CSRF" $csrf}}
|
||||
{{end}}
|
||||
</section>
|
||||
@@ -703,7 +703,7 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
<details class="model-settings" open>
|
||||
<summary><h4>{{t "mirror.settings.heading"}}</h4></summary>
|
||||
|
||||
<form method="post" action="/admin/mirror/context" class="settings-form">
|
||||
{{if $csrf}}<form method="post" action="/admin/mirror/context" class="settings-form">
|
||||
<input type="hidden" name="csrf" value="{{$csrf}}">
|
||||
<input type="hidden" name="model" value="{{$m.Name}}">
|
||||
<input type="hidden" name="open" value="{{$row.Open}}">
|
||||
@@ -713,10 +713,10 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
<button type="submit">{{t "mirror.settings.context"}}</button>
|
||||
</div>
|
||||
<p class="note">{{t "mirror.settings.context_note"}}</p>
|
||||
</form>
|
||||
</form>{{end}}
|
||||
|
||||
{{if $m.GPUProfile}}
|
||||
<form method="post" action="/admin/mirror/gpus" class="settings-form">
|
||||
{{if $csrf}}<form method="post" action="/admin/mirror/gpus" class="settings-form">
|
||||
<input type="hidden" name="csrf" value="{{$csrf}}">
|
||||
<input type="hidden" name="model" value="{{$m.Name}}">
|
||||
<input type="hidden" name="profile" value="{{$m.GPUProfile}}">
|
||||
@@ -726,11 +726,11 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
<button type="submit">{{t "mirror.settings.gpus_apply"}}</button>
|
||||
</div>
|
||||
<p class="note">{{t "mirror.settings.gpus_note"}}</p>
|
||||
</form>
|
||||
</form>{{end}}
|
||||
{{end}}
|
||||
|
||||
{{if or (gt (len $m.Profiles) 1) $m.CanCPU}}
|
||||
<form method="post" action="/admin/mirror/split" class="settings-form">
|
||||
{{if $csrf}}<form method="post" action="/admin/mirror/split" class="settings-form">
|
||||
<input type="hidden" name="csrf" value="{{$csrf}}">
|
||||
<input type="hidden" name="model" value="{{$m.Name}}">
|
||||
<input type="hidden" name="open" value="{{$row.Open}}">
|
||||
@@ -746,29 +746,29 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
</div>
|
||||
{{if and $m.CanCPU $m.CPUSpeed}}<p class="note{{if lt $m.CPUSpeed 5.0}} error{{end}}">{{t "mirror.split.cpu_speed" (num $m.CPUSpeed 1) $m.CPUSpeedNode}}</p>{{end}}
|
||||
<p class="note">{{t "mirror.split.note"}}</p>
|
||||
</form>
|
||||
</form>{{end}}
|
||||
{{end}}
|
||||
|
||||
<div class="settings-actions">
|
||||
<form method="post" action="/admin/mirror/state" class="inline">
|
||||
{{if $csrf}}<form method="post" action="/admin/mirror/state" class="inline">
|
||||
<input type="hidden" name="csrf" value="{{$csrf}}">
|
||||
<input type="hidden" name="model" value="{{$m.Name}}">
|
||||
<input type="hidden" name="open" value="{{$row.Open}}">
|
||||
{{if $m.Stopped}}<button type="submit" name="action" value="start">{{t "mirror.settings.start"}}</button>
|
||||
{{else}}<button type="submit" name="action" value="stop" data-confirm="{{t "mirror.settings.stop_confirm" $m.Name}}">{{t "mirror.settings.stop"}}</button>
|
||||
<button type="submit" name="action" value="restart" data-confirm="{{t "mirror.settings.restart_confirm" $m.Name}}">{{t "mirror.settings.restart"}}</button>{{end}}
|
||||
</form>
|
||||
<form method="post" action="/admin/mirror/unserve" class="inline" data-confirm="{{t "mirror.settings.unserve_confirm" $m.Name}}">
|
||||
</form>{{end}}
|
||||
{{if $csrf}}<form method="post" action="/admin/mirror/unserve" class="inline" data-confirm="{{t "mirror.settings.unserve_confirm" $m.Name}}">
|
||||
<input type="hidden" name="csrf" value="{{$csrf}}">
|
||||
<input type="hidden" name="model" value="{{$m.Name}}">
|
||||
<button type="submit" class="link danger">{{t "mirror.settings.unserve"}}</button>
|
||||
</form>
|
||||
<form method="post" action="/admin/mirror/unserve" class="inline" data-confirm="{{t "mirror.settings.purge_confirm" $m.Name}}">
|
||||
</form>{{end}}
|
||||
{{if $csrf}}<form method="post" action="/admin/mirror/unserve" class="inline" data-confirm="{{t "mirror.settings.purge_confirm" $m.Name}}">
|
||||
<input type="hidden" name="csrf" value="{{$csrf}}">
|
||||
<input type="hidden" name="model" value="{{$m.Name}}">
|
||||
<input type="hidden" name="purge" value="yes">
|
||||
<button type="submit" class="link danger">{{t "mirror.settings.purge"}}</button>
|
||||
</form>
|
||||
</form>{{end}}
|
||||
</div>
|
||||
</details>
|
||||
{{end}}
|
||||
@@ -8,6 +8,7 @@
|
||||
<title>{{t "people.title"}} · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
|
||||
<meta name="htmx-config" content='{"scrollIntoViewOnBoost":false,"defaultFocusScroll":false,"allowEval":false}'>
|
||||
<script src="/_static/htmx-2.0.11.min.js" defer></script>
|
||||
<script defer src="{{asset "pages.js"}}"></script>
|
||||
</head><body><div class="wrap">
|
||||
<header>
|
||||
<h1>{{t "people.title"}}</h1>
|
||||
@@ -94,17 +95,17 @@
|
||||
{{if .Member}}<dt>{{t "people.each_member"}}</dt><dd>{{.Member}}</dd>{{end}}
|
||||
</dl>
|
||||
{{if .Shadowed}}<p class="q-sub q-warn">{{.Shadowed}}</p>{{end}}
|
||||
<details class="edit"><summary>{{if .HasAny}}{{t "people.change_limits"}}{{else}}{{t "people.set_limits"}}{{end}}</summary>
|
||||
{{if $d.CSRF}}<details class="edit"><summary>{{if .HasAny}}{{t "people.change_limits"}}{{else}}{{t "people.set_limits"}}{{end}}</summary>
|
||||
{{template "quota-form" (dict "Scope" (printf "%s" .Scope) "ID" .ID "CSRF" $d.CSRF "Sel" $sel
|
||||
"F" (dict "RPM" .RPM "TPM" .TPM "Budget" .Budget "Period" .Period "Models" .Models "HasAny" .HasAny)
|
||||
"Title" (t "people.quota_for" .Crumb)
|
||||
"Note" (t "people.note.shared"))}}
|
||||
</details>
|
||||
<details class="edit"><summary>{{t "people.limit_each_member"}}</summary>
|
||||
</details>{{end}}
|
||||
{{if $d.CSRF}}<details class="edit"><summary>{{t "people.limit_each_member"}}</summary>
|
||||
{{template "quota-form" (dict "Scope" (printf "%s-member" .Scope) "ID" .ID "F" .MemberRaw "CSRF" $d.CSRF "Sel" $sel
|
||||
"Title" (t "people.each_member_of" .Crumb)
|
||||
"Note" (t "people.note.each_member"))}}
|
||||
</details>
|
||||
</details>{{end}}
|
||||
{{end}}
|
||||
{{with .Person}}
|
||||
<dl class="pd-limits">
|
||||
@@ -114,12 +115,12 @@
|
||||
<dt>{{t "people.spent" $d.QuotaWindow}}</dt><dd><span class="q-spend">{{.Spend}}</span>{{if .HasPct}} <span class="q-meter{{if ge .Percent 90}} hot{{end}}"><i style="width:{{pct .Percent 100}}%"></i></span> <span class="q-pct">{{t "quotas.pct" .Percent}}</span>{{end}}</dd>
|
||||
<dt>{{t "people.models"}}</dt><dd>{{if .Models}}{{th "people.models_only" .Models}}{{else}}{{t "people.models_every"}}{{end}}</dd>
|
||||
</dl>
|
||||
<details class="edit"><summary>{{if .HasAny}}{{t "people.change_limits"}}{{else}}{{t "people.set_limits"}}{{end}}</summary>
|
||||
{{if $d.CSRF}}<details class="edit"><summary>{{if .HasAny}}{{t "people.change_limits"}}{{else}}{{t "people.set_limits"}}{{end}}</summary>
|
||||
{{template "quota-form" (dict "Scope" "user" "ID" .ID "CSRF" $d.CSRF "Sel" $sel
|
||||
"F" (dict "RPM" .RPM "TPM" .TPM "Budget" .Budget "Period" .Period "Models" .Models "HasAny" .HasAny)
|
||||
"Title" (t "people.everywhere" .Email)
|
||||
"Note" (t "people.note.person"))}}
|
||||
</details>
|
||||
</details>{{end}}
|
||||
{{end}}
|
||||
{{end}}
|
||||
|
||||
|
||||
Loaded 100 of 105 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user