Commit Graph
370 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 a53b2f4be9 Customise a model from the portal: a file, a model, a button
A page for anyone in a project, built for the one who knows least: an
Excel, CSV or JSONL file of questions and answers, previewed; a model; a
name; who may use it; and a button. The weights to train on are found
from where the model's came (a GGUF repository's base model), the card's
memory from their size, the node as the one with the most free. When it
is done the adapter is published by itself (a llama.cpp model's replicas
replaced to load it) and is a model of its own in the chat and the API.

The project's by default; its members share it wider (confirmed, and
never wider than the knowledge bases its pairs came from) or delete it.
One trains at a time per project. ctl train start needs only --model and
--data now, and --publish publishes when done.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 11:31:12 +01:00
Ilker ManapandClaude Opus 5.5 fcda4d3f85 Slurm's jobs in llmgw's accounting: projects as accounts, jobs priced
CI / test (push) Failing after 7m24s
With slurm.users llmgw, each project is a Slurm account, its organization's
and team's above it, with the project's members and its team's and
organization's on it; slurm.conf holds a job to its user's accounts. The
controller's agent keeps them with sacctmgr, only what it made, and reports
the jobs that ended in the last day; the leader records each once, its
GPU-hours (core-hours without cards) priced at auth.gpu_hour_cost into its
project's usage as model slurm/PARTITION.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 11:54:01 +01:00
Ilker ManapandClaude Opus 5.5 8547528fb7 Prices from what the cluster costs: ctl prices suggest
CI / test (push) Successful in 9m51s
auth.gpu_hour_cost (and core_hour_cost) is what an hour of a GPU (a core)
costs the operator. The leader keeps, by the hour, the GPU- and core-hours
each model's replicas hold (held_1h); llmgw ctl prices suggest spreads a
model's over the tokens it served, an output token counted as four input
ones and a cached prompt token as a tenth of one, and --apply writes the
prices into auth.pricing. Idle hours are in the cost, so a model held and
little used comes out dear; one held and billed nothing is shown with what
it held.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 06:54:55 +01:00
Ilker ManapandClaude Opus 5.5 57f0fd2ed0 Slurm's jobs in containers: pyxis and enroot, installed by the agents
CI / test (push) Failing after 6m53s
slurm.containers (ctl slurm containers on) has every Slurm machine's agent
install enroot from NVIDIA's release and build pyxis against its own
Slurm, both pinned and checked against their digests, and on GPU machines
nvidia-container-cli from NVIDIA's repository. enroot keeps each user's
containers under /run/enroot and /var/lib/enroot, needing no login
session, with the shared path in every container. The leader has Slurm
load pyxis only once every machine reports it: plugstack.conf's
"required" fails every job on a machine without the plugin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:31:20 +01:00
Ilker ManapandClaude Opus 5.5 4b4306bb6a Slurm users reach a node only through a job of theirs
CI / test (push) Successful in 9m41s
llmgw's Slurm users log in to the controller freely and to another Slurm
machine only while a job of theirs runs there: the agent installs
pam_slurm_adopt and puts it in sshd's account stack for the users it made
alone, so root and the machine's own users are never locked out. Their
session is put in the job's extern step (PrologFlags=Contain), and
pam_systemd, which would move it out, is skipped for them by taking the
session file into sshd's stack with a guard in front of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:30:49 +01:00
Ilker ManapandClaude Opus 5.5 d8ce573bdb Slurm's shared files and users: one path everywhere, accounts as Linux users
CI / test (push) Failing after 7m1s
slurm.shared mounts the site's NFS export, has an llmgw node export the path
over NFS to the Slurm machines alone (2049 in the firewall), or looks for a
file system the site mounts. slurm.users "llmgw" makes each active account a
Linux user on every Slurm machine: login from its e-mail, a UID from 200000
kept in slurm_users and never reused, its home on the shared path, its SSH
keys from the portal's Profile (or ctl slurm users keys) read by sshd from
/etc/llmgw/ssh-keys. server init --slurm chooses /shared from the first
machine and llmgw's users; --slurm-shared and --slurm-users change that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:34:34 +01:00
Ilker ManapandClaude Opus 5.5 748f777a56 Slurm's accounting: slurmdbd and a MariaDB on the controller
CI / test (push) Successful in 10m57s
Since 23.11 Slurm runs badly without accounting storage: the main
scheduler passes every job by as InvalidAccount. slurmdbd takes MySQL or
MariaDB only, so the controller's agent installs MariaDB beside it,
answering that machine alone, makes the database and its user (the
password made from the munge key, never on a command line), runs
slurmdbd and adds the cluster, all before slurmctld. A MariaDB that was
on the machine before keeps its settings. slurm.conf accounts the jobs
there, with what each used from its cgroup and the cards, and the
priority is multifactor again. Port 6819 joins Slurm's in the firewall.
ctl slurm status and the console say whether slurmdbd runs; the install
screen, server init --slurm and the guide say what is put where.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:59:08 +01:00
Ilker ManapandClaude Opus 5.5 3387f97f6a Slurm's ports for the cluster only, and srun in a range of its own
CI / test (push) Successful in 9m40s
slurm.conf sets SrunPortRange=60001-63000, so a site's firewall can let
srun's step connections in. A node given a part in Slurm adds 6817-6818
and that range to llmgw's nftables table, open to the cluster's networks
and closed to everyone else, as the engines' ports are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:10:37 +01:00
Ilker ManapandClaude Opus 5.5 d864438272 A model that failed to start is placed elsewhere next
CI / test (push) Successful in 9m31s
The same inputs place a replica on the same node and card, so an on-demand
model that failed to load there was started there again after every
backoff until it was parked. The reconciler remembers where a model's
replicas failed and places the next away from them when anything else
fits (Scheduler.Away); with nowhere else, there again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 09:22:11 +01:00
Ilker ManapandClaude Opus 5.5 6320dd5b13 A new stream is told the node's files again
CI / test (push) Successful in 9m31s
Registration carries what the agent's startup scan found, and a heartbeat
sends the list only when it changed since last sent; so a file deleted
after the agent started came back on a control plane that restarted. The
fingerprint is now forgotten with the provisioning one on every stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:14:52 +01:00
Ilker ManapandClaude Opus 5.5 5732a5ebda The serve forms start at the longest context that fits
CI / test (push) Successful in 9m26s
The mirror page's form began every GGUF model at 4096 tokens, the
console's at 4096 or the plan's figure. Both now start at what the model
was trained for, cut to the most whose KV cache fits beside the weights
where the form places it; the mirror page's slots are as many of that, up
to four, as the room left holds. Plan carries the trained context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 14:38:41 +01:00
Ilker ManapandClaude Opus 5.5 561a536477 The weight sweep names a set as its fetch did
CI / test (push) Successful in 9m33s
A set is fetched into a directory named from its address without the
trailing slash; Orphans named it from the catalog's address with it, so
every set in the catalog was offered for deletion: 330 GB of six models on
the PoC. One function, cacheSource, now gives both the name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 14:01:27 +01:00
Ilker ManapandClaude Opus 5.5 102ca9556f An organization's admin, and forms only for those who may send them
C1. A membership of an organization may be admin as well as member and
manager. An admin sees /org as a manager does and changes the
organization from it: its teams and projects, who of the approved
accounts is in them, its members' roles, and the keys spent in its
projects. Every id a form names is checked against the organization;
limits, accounts, the organization's name and the cluster stay with the
cluster's staff, and an admin cannot take their own role away.

A group from OIDC, SCIM or a directory no longer rewrites a membership
it names as a plain member at every sign-in, which took a manager's role
away (EnsureMembership).

Pages draw a form only for someone who may send it: Nav.May asks the
staff rule staffMay keeps, and a page whose forms a person may not send
gets no form token. A viewer sees the people, limits and mirror pages
without forms; an operator the mirror's.

Forms are compact: labels close to their fields, fields as wide as what
goes in them, smaller buttons, and the short forms on one line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 11:38:06 +01:00
Ilker ManapandClaude Opus 5.5 cba4897987 Deleting unused model files says what went, and sets go whole
CI / test (push) Failing after 6m44s
A sweep counted every file the nodes were asked to delete as freed as soon
as it asked, and a node refused a set's directory as not empty, so tens of
gigabytes were reported freed and nothing was. The agent removes a set's
directory whole (a link only as a link), and the control plane waits for
each node to list its files again, saying what it no longer lists and what
it kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:29:16 +01:00
Ilker ManapandClaude Opus 5.5 3bb52742a4 v0.16.3: changelog and README
Release / build (push) Successful in 5m11s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 10m10s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:43:40 +01:00
Ilker ManapandClaude Opus 5.5 1a3f385be4 Every setting of a served model from the mirror page, and its replicas per profile
CI / test (push) Successful in 10m51s
The row of a served model leads to its card, which now changes its context
and longest answer, its cards, its replicas on each profile with a profile
on CPUs to add and its speed beside, stops, starts and restarts it, and
removes it with its files or without. Stopped on the row stops the model
rather than setting its scaling to nothing. The console gains Replicas per
profile, and ctl models shows the counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:40:06 +01:00
Ilker ManapandClaude Opus 5.5 6403e61f4b How fast a model answers on the CPU, before it is served
CI / test (push) Successful in 9m31s
Each agent measures one core's memory rate beside the whole machine's; the
estimate takes a replica's threads at about one core's rate each, up to the
machine's, calibrated by llama-bench on the PoC. The serve form shows it
beside a card's, warns when slow, and asks for the CPU on purpose when a
reply would take minutes; ctl model plan, the console and serve jobs say
it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 08:52:15 +01:00
Ilker ManapandClaude Opus 5.5 220110ed0d v0.16.2: changelog and README
Release / build (push) Successful in 5m43s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 11m11s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 08:32:34 +01:00
Ilker ManapandClaude Opus 5.5 a88624747e A set wanted by several replicas on one node is fetched once
CI / test (push) Failing after 7m23s
Single files were shared between the shards and replicas asking for them
at the same time; sets were not, and four replicas on one node's four cards
wrote the same .part files over one another until all but one failed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-02 08:29:25 +01:00
Ilker ManapandClaude Opus 5.5 540b518fde Rate limits counted across the control planes with router.active_active
CI / test (push) Successful in 9m30s
Each router's per-counter request and token counts, this minute's and the
last, ride the standby's two-second exchange with the leader. The leader
answers each standby with everyone else's sum, its own included, and puts
the standbys' sum in force on its own router; each limiter adds the
others' counts to its sliding window.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 17:30:08 +01:00
Ilker ManapandClaude Opus 5.5 8b7eefef16 Standbys that answer the client API themselves, behind router.active_active
CI / test (push) Successful in 9m35s
A standby follows the leader's Ready replicas over the agent port, every
two seconds, and reports back the replicas its router used and the
on-demand models it was asked for. No engine keys travel: every control
plane derives them from the cluster secret. Budgets are raised to the
store's every 30 s. Unheard from the leader for a minute, a standby
passes /v1 on as before. Off by default.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 13:47:47 +01:00
Ilker ManapandClaude Opus 5.5 31f440ab6e A served model's longest answer changed in place, and Strata's context
CI / test (push) Successful in 9m29s
max_output on POST /api/v1/models/{name}/context, --max-output on ctl model
context, and a field in the console's Context length… form. Alone it changes
only the catalog's limit, which the router holds max_tokens to, and starts no
replica again; with a context it is set with it, never past it.

Strata's context is now changed like the others', through its setup's
--context, which writes the engine's start script at each start. Neither
needs a model removed and added again any more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 00:15:38 +01:00
Ilker ManapandClaude Opus 5.5 f4b34dbf11 FreeToken, an engine for models larger than the card
CI / test (push) Successful in 9m37s
FreeToken (FlashML-org/FreeToken) serves mixture-of-experts models several
times a card's size from one card, experts in RAM, from Hugging Face
safetensors checkpoints. Installed like vLLM, from PyPI by uv, on every node
it runs on: compute capability 8.0+, driver 580+, x86-64, and the CUDA 13
toolkit it compiles kernels with. Older cards are passed over: on cuda01's
Quadro RTX 6000 (7.5) 0.2.1 had no kernel image for the first prefill, and
in fp16 hit an illegal memory access on its first request.

ft serve gets the model, address, card and served name; the key goes in
FREETOKEN_API_KEY. Flags that read files or fetch are refused, as are
FREETOKEN_ variables naming paths or the agent's own settings. ctl model
context sets --max-seq-len-override.

Readiness now also waits for an engine whose /health answers 200 while its
body says "loading", as FreeToken's does from the moment its HTTP server is up.

FreeToken is followed release by release: a weekly workflow fails when PyPI
has a version newer than DefaultEngines installs (scripts/engine_releases.py).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 22:10:05 +01:00
Ilker ManapandClaude Opus 5.5 c866b3a8ee How much of each GPU works, beside whether anything does
CI / test (push) Failing after 6m42s
The busy figure is nvidia-smi's utilization.gpu: the share of time any
kernel ran, which a kernel on one SM fills. On cuda01 decode read 92% busy
while the SMs had work 69% of the time and issued on 18% of their cycles.

Busy is now said to be that, and beside it: each card's draw against its
power limit (efficiency table, fleet and node charts, the console's GPU
work chart), and where the profiling counters can be read, SMs active, SM
occupancy, tensor cores and DRAM (llmgw_gpu_*_active_percent), from NVML's
GPM on Hopper and later or a dcgm-exporter on the node
(discovery.dcgm_exporter). The GPU map colours by work by default and keeps
busy as a mode. A MIG instance is never published with its card's profile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 20:11:26 +01:00
Ilker ManapandClaude Opus 5.5 7bfe70c8df BACKLOG: the router profiled, the write per token removed, what is left
CI / test (push) Successful in 9m22s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-31 15:50:15 +01:00
Ilker ManapandClaude Opus 5.5 9ff49cad8e Requests missing the cache for the same key at once share one read and one allowance
CI / test (push) Successful in 9m51s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-31 09:30:18 +01:00
Ilker ManapandClaude Opus 5.5 524f029603 External providers left out of admission's saturation; the scale-out section in the backlog
CI / test (push) Successful in 9m25s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 19:13:15 +01:00
Ilker ManapandClaude Opus 5.5 be54f1617b A mirror download cut off carries on from where it got to, and is tried again
CI / test (push) Successful in 10m36s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 17:47:32 +01:00
Ilker ManapandClaude Opus 5.5 76746715ec A set's files checked four at once, the check said as it goes and not taken for a stuck download
CI / test (push) Failing after 6m51s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 14:58:44 +01:00
Ilker ManapandClaude Opus 5.5 d5dcf78836 A set's download reported as the whole set's, not the part being fetched
CI / test (push) Successful in 9m15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 14:48:45 +01:00
Ilker ManapandClaude Opus 5.5 a5fefbdaaa v0.16.1: changelog and README
CI / test (push) Successful in 12m33s
Release / build (push) Successful in 6m32s
Release / notes (push) Successful in 4s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 13:14:25 +01:00
Ilker ManapandClaude Opus 5.5 f41c96d30f Strata's DeepSeek not offered on cards its engine was not built for, and refused there by the agent
CI / test (push) Successful in 9m42s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 13:00:21 +01:00
Ilker ManapandClaude Opus 5.5 104b60fbf8 A repository's files read from every directory, not only its top
CI / test (push) Successful in 11m40s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 12:16:11 +01:00
Ilker ManapandClaude Opus 5.5 845de8eef4 The mirror page's Strata tab; Strata's ready-made packs served with Strata fetching them, strata-ple named
CI / test (push) Successful in 10m35s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 12:06:46 +01:00
Ilker ManapandClaude Opus 5.5 4ee9ba54ef v0.16.0: changelog and README
Release / build (push) Successful in 6m6s
Release / notes (push) Successful in 5s
CI / test (push) Successful in 11m14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 11:21:32 +01:00
Ilker ManapandClaude Opus 5.5 12464ce0f6 Cluster upgrades: nodes told together with progress said, done when back on the version, the download kept past a broken stream
CI / test (push) Successful in 10m2s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 11:14:02 +01:00
Ilker Manap d6f8f433f4 CHANGELOG: one list
CI / test (push) Successful in 9m18s
2026-10-30 10:39:30 +01:00
Ilker ManapandClaude Opus 5.5 b8ed368e31 Weights from Hugging Face pinned to a commit and checked against the sha256 it published
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 07:33:15 +01:00
Ilker ManapandClaude Opus 5.5 29d7f54cfd A path escaped as it came reaches the leader through a standby; F5 in the backlog
CI / test (push) Successful in 9m30s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 22:54:27 +01:00
Ilker ManapandClaude Opus 5.5 cacaa35c3f The gateway embeds the documents llmgw-kb sends as text
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 21:56:05 +01:00
Ilker ManapandClaude Opus 5.5 01ad8c9207 v0.15.3: changelog and README
CI / test (push) Successful in 11m44s
Release / build (push) Successful in 5m43s
Release / notes (push) Successful in 4s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 20:54:53 +01:00
Ilker ManapandClaude Opus 5.5 59187cdae5 A model can be stopped and kept, or removed with its files from the mirror and the nodes
CI / test (push) Successful in 10m37s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 20:45:48 +01:00
Ilker ManapandClaude Opus 5.5 435ffe1171 A file removed outside the running mirror is no longer reported, and its removal reaches the peers
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 19:25:00 +01:00
Ilker ManapandClaude Opus 5.5 785e06ecbf A mirror with peers records the checksums of files that had none, so they are copied too
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 18:36:59 +01:00
Ilker ManapandClaude Opus 5.5 0033a530c7 mirror add-peer on a mirror machine of its own says what to run on a control plane
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 18:30:30 +01:00
Ilker ManapandClaude Opus 5.5 8cd376615b v0.15.2: changelog and README
Release / build (push) Successful in 4m44s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 9m18s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 16:46:15 +01:00
Ilker ManapandClaude Opus 5.5 e1cff0d4e3 A served model can be moved to more cards, or fewer
CI / test (push) Successful in 10m28s
llmgw ctl model gpus MODEL N [--profile P] [--in-place],
POST /api/v1/models/{name}/gpus and the console's GPUs…: how many cards of
one node each replica uses. Tensor parallel for vLLM, SGLang and
TensorRT-LLM, held to the model's KV heads (modelplan.SplitsHeads, now one
rule for the planner and this); the layers split for llama.cpp, its old
--tensor-split dropped. Refused for a count no node has, for cards that
would not hold the weights and the context, and for engines on one card.
The replicas start again beside the old ones; with no room beside them the
command says so, and --in-place stops each first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 16:40:49 +01:00
Ilker ManapandClaude Opus 5.5 51aeadae3e v0.15.1: changelog and README
Release / build (push) Successful in 6m4s
Release / notes (push) Successful in 5s
CI / test (push) Successful in 11m42s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 15:59:12 +01:00
Ilker ManapandClaude Opus 5.5 986981ac81 A served model's context can be changed, engine and memory with it
CI / test (push) Successful in 12m0s
ctl model context MODEL TOKENS [--slots N], POST /api/v1/models/{name}/context
and the console's Context length…: the catalog, the engine's arguments and
the KV memory change together, refused past the trained context or what the
cards hold, then the replicas restart one by one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 15:50:04 +01:00
Ilker ManapandClaude Opus 5.5 1c16f2337d A second mirror: llmgw mirror add-peer, and on a store machine at setup
CI / test (push) Successful in 9m29s
mirror add-peer, run on the mirror's machine, sets up a mirror on
another over ssh, kept in step with this one: llmgw there if it has
none, mirror init --peer, this mirror told of it and allowed to be read
by it, and mirror.peers in the cluster's settings. The setup screens
offer one on either database machine, as server init --second-mirror.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 13:43:27 +01:00