Commit Graph
62 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 adb1b3cf2a Add replicas under load, up to a model's max, and take them back when quiet
A model's max never did anything: desiredFor returned the minimum, or one
for an on-demand model someone was asking for, whatever the queue. Six small
CPU nodes run out of slots quickly, and a model with room to grow stayed at
its minimum while requests waited.

The router already scrapes each engine's running and waiting requests. The
control plane now reads them per model on every tick, alongside when each
replica was last used, and the reconciler keeps how many replicas load has
added above a model's base. One or more requests waiting per ready replica,
held for a minute, adds one; ten minutes with nothing waiting removes one of
those again. It never goes above max or below the base, moves one step at a
time, and does not add while the last addition is still placing or starting
— on this fleet a replica is minutes of weights and gigabytes of memory, and
a spike should not leave a pile of replicas behind it. Decisions are logged
with the load behind them.

Nothing live changes until a model's max is raised above its min.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 03:18:07 +02:00
Ilker ManapandClaude Opus 5.5 bd12b0909a Serve a model from the registry with one command
Adding a model took four steps and a YAML file — plan it, fetch it into the
mirror, write its catalog entry with a memory figure somebody estimated,
scale it — and whether it would load was found out at the end.

`llmgw ctl model serve OWNER/REPO` does it as one job in the control plane,
where the registry, the mirror, the catalog and the fleet already are:

- the registry: the GGUF file set matching --quant (Q4_K_M by default) or
  --file, leaving out vision projectors and importance matrices, and naming
  what the repository does have when nothing matches; gated repositories are
  refused;
- the mirror: fetched at the repository's revision, with progress, and each
  file's sha256 compared with what the registry published — the mirror is
  the one place a corrupted download is caught once rather than loaded on
  every node;
- the check: whether the nodes' llama.cpp loads the architecture, and the
  memory it needs from its own header;
- the catalog, then a wait for a replica that stops early, with the reason,
  when the model is parked or fits on no node;
- a smoke test: one short request to the replica, with its engine key, and
  the answer and tokens per second.

ctl starts the job and prints each step as it changes; a download of hours
does not depend on the terminal. --on-demand adds the model without starting
a replica. Jobs are kept in memory until the control plane restarts.

The catalog name a repository suggests moved into the registry package, so
the mirror page and this agree on it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 03:05:13 +02:00
Ilker ManapandClaude Opus 5.5 116e75f8cb Upgrade the cluster with one command, over the agents' own streams
Every upgrade so far has been a script of mine copying a binary to each
machine over ssh, restarting it and polling until it came back — which is
the job, but not something the product could do, and which needed ssh into
every node.

`sudo llmgw ctl upgrade VERSION` on the control plane does it. This machine
first: the release for its platform, checked against the release's
checksums, the old binary kept beside it, llmgw-server restarted, and a wait
until every node that was ready has reconnected. Then each node in turn: the
control plane tells its agent the version over the stream it already holds;
the agent fetches and checks the release itself, swaps it in with a rename,
answers, and exits with status 75 so systemd starts the new binary. Engines
keep running, as for any agent restart, and are adopted again.

A node is done when it is back on the new version, ready, with as many ready
replicas as before; one that is not within the timeout stops the run before
the next node, so a bad release reaches one machine. Nodes already on the
version are skipped, so running it again resumes.

The fetch-and-verify is one package now, shared with onboarding's idea of
what a release version looks like.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 02:54:36 +02:00
Ilker ManapandClaude Opus 5.5 3bf4b7b709 Tell clients how much of their rate limit is left
SDKs and editors pace themselves from OpenAI's x-ratelimit-* headers, and
the gateway sent none: only Retry-After, and only on the refusal they would
have avoided. A limit can be set at several levels of the tenancy tree, each
counted in its own pool, so the headers report the one with least left —
the one that will refuse first — after this request's own charge, on
refusals and successes alike. The windows slide, so the reset figure is the
next minute boundary, when a whole window has moved on. A limit that is not
set is not advertised.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 02:42:21 +02:00
Ilker ManapandClaude Opus 5.5 91fe24a105 Close what a second security review found still open
A review of v0.10.4-rc.4, and a look at the live network, found:

- Alertmanager listening on every interface: from any node its API took
  silences, with no login. Bound to loopback on the monitoring host, and the
  monitoring README now installs it that way; Prometheus beside it is its only
  client, and one Alertmanager needs no cluster port.
- Login and signup unbounded in memory. Each attempt runs argon2 with 64 MiB,
  and nothing limited how many ran at once; the per-address throttle kept one
  entry per address ever tried. Hashing now takes one of a few slots and
  waits for one otherwise. The throttle expires entries whose window has
  passed when it fills, and if it is still full it refuses new addresses:
  evicting live ones instead would let a flood of made-up addresses reset the
  throttle on the address actually being guessed.
- Metric labels taken from the request. A refused request counted the model
  name it asked for, so any key holder could mint a series per request.
  Requests are counted under the catalog's name, or "unknown".
- Two in code written this week. A SQLite backup made its snapshot with the
  default mode in the listable backup directory, a full copy of the store
  readable while the backup ran; it is now made in a private directory, and
  the unit sets UMask=0077. And archive links were checked by name only, so
  "a -> ." then "a/a/b -> ../.." passed and a file written through them landed
  above the install directory, as root. Archives now unpack through os.Root,
  and a link may only point at a file beside it — which every llama.cpp
  b10991 archive, CPU and CUDA, was checked to follow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 02:25:07 +02:00
Ilker ManapandClaude Opus 5.5 1da4a2e1ef Back up the store and the certificate authority, and alert when it stops
There were no backups: docs/install.md gave a pg_dump command, and nothing
ran it. The store holds every key, person, quota and usage record, the models
added at runtime, and now the secret each engine's key is derived from; the
certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk
lost all of it.

`llmgw server backup` writes both to one archive: pg_dump's custom format, or
a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's
key, so it is 0600; the directory is listable, so the control plane — not
root — can see when the newest was written. pg_dump is given the connection
through its environment, since a DSN carries the password and a process's
arguments are readable by every user. Written beside its name and renamed,
so an interrupted backup is never the newest; the newest 14 are kept, and
nothing the command did not write is ever removed.

llmgw-backup.timer runs it nightly and catches up after downtime. The control
plane publishes the newest archive's time, and BackupTooOld fires past 26
hours — or when there has never been one. The install guide now covers
restoring, and says to copy archives off the machine, since they share its
disk.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:44:32 +02:00
Ilker ManapandClaude Opus 5.5 37a80a9b44 Park a model that fails every time it starts, and say so everywhere
On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not
know — was started 155 times in eight hours, failing the same way each time,
and phi-4, which fitted on no node, logged "no placement" 250 times in two
days. Neither showed up anywhere an operator looked: the backoff never gave
up, no metric said a model was failing or unplaceable, and the one alert for
a model with nothing serving could not fire, because its series was published
only while a replica was ready.

After five failures in a row a model is parked: not placed again until its
definition changes — a new source or profile — or `llmgw ctl retry` is run.
Status, the model list and the admin API show it with its last error. The
placement warning is logged when its reason changes or every ten minutes.

A request for a parked or unplaceable model is answered at once with why,
rather than "loading; retry shortly", which a client retries forever. Each
request still clears the previous placement result and triggers a fresh
attempt, so a model that fits once room is made is not refused on a stale
answer; the router waits for that attempt, not for its full activation
window.

Metrics now cover model health (desired, ready at 0 for what should run,
consecutive start failures, parked, unplaceable), engine installs per node,
leadership, and certificate expiry read from disk at each scrape, with alerts
for each. `ctl status` counts a model against what it should have now, so an
on-demand model that was asked for and fitted nowhere no longer reads as
satisfied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:21:57 +02:00
Ilker ManapandClaude Opus 5.5 5e1b6ecea0 Read a queued change as a success in ctl
ctl accepted only 200, so the first real `ctl restart --all` printed
"Error: 202 Accepted" over a request that had queued nine restarts, all of
which then happened. Any 2xx is now a success, and its body is read.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:14:12 +02:00
Ilker ManapandClaude Opus 5.5 5dac1854c2 Publish small fixes as release candidates of the next version
Every fix so far became its own patch release, and the natural alternative —
v0.10.3-a — sorts before v0.10.3 by SemVer's rules: a hyphen marks a
pre-release, so every tool that orders versions would take the fix for the
older one. A release candidate of the next version, v0.10.4-rc.1, sorts
after v0.10.3 and before v0.10.4, which is what it is.

CI already builds any v* tag. Its release notes now come from a
"## Unreleased" section in CHANGELOG.md, where changes gather until the
version they belong to is released, and the release is marked as a
pre-release. Onboarding, in Go and in the shell script, counts a candidate
as a release and downloads it rather than copying the operator's binary.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:00:11 +02:00
Ilker ManapandClaude Opus 5.5 611fddd236 Describe v0.10.3
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 00:38:55 +02:00
Ilker ManapandClaude Opus 5.5 bb90a927aa Print only what must outlive init's screen, and drop bubbles
Once server init's screen closed, the whole run was printed again under the
frame that had just shown it. Now the password and the next steps are, the
whole run goes to /var/lib/llmgw/init.log (0600: it holds the password), and
it is printed in full only when init fails. Seen in a draft of the install
video, where it scrolled the screen away.

The spinner and the text field are written here instead of taken from
bubbles, whose packages query the terminal's colours as they load. bubbletea
does the same deliberately, so the query stays; the dependency goes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 22:37:28 +02:00
Ilker ManapandClaude Opus 5.5 49c446271c Describe v0.10.1
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 22:17:33 +02:00
Ilker ManapandClaude Opus 5.5 b5f412f9df Put the mirror on its own machine in two commands
mirror init sets a machine up as the model mirror — the user, /srv/models,
and a unit serving files on :8081 and the management API on :8082 — and
server init --mirror points the control plane at it instead of running one.
Before, that took a unit written by hand and an edit to server.yaml.

The management API can fetch and delete models and answered anyone who
could reach it. mirror serve --api-allow limits it to the addresses given
and the machine itself; mirror init passes the control plane's. The unit
text is shared with the one server init writes for a mirror beside it.

With --no-node the starter model is still fetched and catalogued, unpinned,
so it starts on the first node added, and the init form shows the mirror in
use instead of offering to run one.

Tried on three empty VMs: the mirror answered the control plane and refused
the node, and the starter model came up on the node when it was added.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 22:17:24 +02:00
Ilker ManapandClaude Opus 5.5 a1b5daceb8 Let server init ask for what it needs, and node add take several nodes
In a terminal and without --admin, server init shows a short form: the
administrator's email, the other names nodes will use (with the addresses
it found), SQLite or PostgreSQL, and the mirror, node and starter model.
Its own lines then fill in as a list — read from the writer init already
uses, so nothing in init changed — with downloads updating in place and the
password in a box until the end. The whole run, password included, is
printed again when the screen closes, so it is in the terminal's history.
The last screen says how to add a node and to back up the authority's key.

node add takes several addresses and adds them in turn, with a summary.
Suggested fabrics carry what the adapters report them to be, so a name
given for one node is used for the next on the same fabric without asking.

Tried on empty VMs: init from the form through to the starter model
serving, and two nodes added to that cluster in one command. The first
attempt at the form lost its keystrokes to the terminal colour query in a
pseudo-terminal that never answers it; a real terminal does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 21:54:18 +02:00
Ilker ManapandClaude Opus 5.5 7bb360fc78 Measure two nodes' fabric rail by rail: llmgw ctl fabric-test
doctor checks a node alone, and every check can pass while two nodes still
cannot reach each other. ctl fabric-test A B asks B's agent for an
ib_write_bw server and A's for the client, adapter by adapter as NCCL pairs
them, one rail at a time, and grades each against its link's rate. It goes
through the agents' streams, so there is no SSH between nodes; the agent
installs perftest the first time, and names the RoCE v2 GID itself.

The result is read from perftest's table by its heading, not a column
number. Agents now list the requests they answer beyond the usual
(Register.features), and the control plane refuses a test on one too old to
answer instead of waiting minutes for it. The forwarding dispatcher carries
the new capability, and its test says so: a missing forward would have made
the command quietly unavailable in production.

Not yet run over real InfiniBand or RoCE; this cluster has neither.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 21:29:23 +02:00
Ilker ManapandClaude Opus 5.5 af6f8e023e Describe v0.10.0
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 21:06:42 +02:00
Ilker ManapandClaude Opus 5.5 590e5b9f3e Ask a node only for engines it can install; doctor says when AVX2 is hidden
A control plane of this release asks every node for vLLM. An agent of the
last one does not know how to install it, and answered "unsupported", so a
cluster half way through its upgrade showed every node as failing. Agents
now say which engines they can install (Register.installable_engines), and
are asked for no others; an agent that says nothing is asked for llama.cpp
alone, as it always was.

doctor reports the CPU's vector extensions, and warns when AVX2 is missing,
with the fix for a VM: the test node for node add was one, on QEMU's default
CPU model, where llama.cpp runs several times slower and nothing else looks
wrong. node add shows it, since it runs doctor.

The changelog's unreleased section was two releases' worth of sections one
after the other, with two lists of fixes and the upgrade notes in the
middle. It is one entry now: largest changes first, the fixes together, and
every upgrade note at the end, with vLLM's default install and how to keep
your own.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 21:03:03 +02:00
Ilker ManapandClaude Opus 5.5 2a16c26575 Add a node with one command: llmgw node add, with a terminal screen
Adding a node was deploy/bootstrap/onboard-node.sh, which is not installed
anywhere and wanted the node's name, the control plane's address and a
dozen flags. It is now in the binary, run on the control plane:

    sudo llmgw node add ubuntu@10.0.0.21

The node id is the machine's hostname, and the address its agent dials is
this machine's on the route to it, checked against the server certificate.
It installs llmgw there (the release for its architecture, fetched and
checked by the node; uname rather than dpkg, so RHEL works), runs doctor
there and stops on a problem before changing anything, issues and installs
the certificate, writes agent.yaml and the unit, starts the agent, and waits
for it to join and install llama.cpp. The authority is read first, so a
missing one is said before the node is touched.

The steps are events. In a terminal they draw a checklist (bubbletea) that
asks what a script cannot: which fabric each group of active adapters is on,
grouped by the subnet manager or RoCE network they report, and whether to go
on past a problem. Elsewhere, one line per step.

Found by running it on an empty VM: ssh's "Permanently added" warning on
stderr was read as the machine's hostname, so standard error is now kept
apart; and a node added again while cordoned never counted as joined, so
joining now means a heartbeat since the restart.

llmgw node remove forgets a lost node, which otherwise stays listed until
the control plane restarts. ctl nodes no longer lists vLLM as skipped on
every CPU node. The quick install, install and onboarding docs use it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 19:33:47 +02:00
Ilker ManapandClaude Opus 5.5 4c5a2fd6da Install vLLM on every GPU node that can run it, as llama.cpp is
vLLM was installed by hand and named in each agent's configuration. The
agent now installs it: a Python 3.12 environment built by uv, which is
itself a GitHub release checked against its checksum, from PyPI resolved as
it stood a day after that vLLM was published and installed with
--require-hashes. The resolved list is kept in the environment. The
environment is built relocatable under a scratch name and renamed into place
once it runs, like an archive.

The default is 0.30.0, whose flags were checked against the ones llmgw
builds. It is built for CUDA 13, so it needs driver 580 and a Turing GPU or
newer, and 25 GB free while it installs; a node lacking one says which, and a
node with no GPU skips it without being shown as a fault. gcc comes with it
because Triton compiles its launcher at the first model load. SGLang
installs the same way when the control plane names a version. llmgw doctor
reports whether vLLM can be installed on the node.

Checked by installing 0.30.0 this way on a test node: 7.7 GB, torch 2.13 for
CUDA 13.0, importing cleanly; the node has no GPU, so vLLM was not started.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 13:15:54 +02:00
Ilker ManapandClaude Opus 5.5 e751c89937 Serve the project's website at /about, with its fonts in the binary
The website was a file built by hand outside the repository. It is now a
template in internal/web: every gateway serves it at /about, without a key,
linked from the landing page's footer, and the published site is rendered
from the same template.

The fonts are vendored (Latin subsets, OFL) so the page sends nobody's
browser to Google. The proxy configurations it shows are copies of
deploy/proxy, kept equal by a test. A development build says so instead of
linking to release files that do not exist.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 10:44:45 +02:00
Ilker ManapandClaude Opus 5.5 80e0dd252f Document what changed since v0.9.2, and fabrics for the sites that have them
The changelog's unreleased section had stopped at refusing unloadable
weights; it now covers the rest: telling independent fabrics apart,
their subnet managers and maps, NVLink on every build and across nodes,
the multi-node launch fixes and vLLM's new default, doctor, node load,
editors and the Responses API, the chat and mirror pages, a person's own
use, the admin pages, the new metrics, and how to upgrade.

docs/fabrics.md is new, for a site with InfiniBand or NVLink: what
llmgw does with them, what each node and fabric needs first, declaring
fabrics, checking a node with doctor, what the pages and metrics show
and which to alert on, what an engine is given, and where to look when
a replica spanning nodes will not run. It says plainly that none of it
has yet run on such hardware.

install.md, node-onboarding.md, onboarding.md, the README and the
example agent configuration no longer say the portable binary cannot
see NVLink or that a fabric's name cannot be checked, and describe the
account page and connecting an editor.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 10:14:58 +02:00
Ilker ManapandClaude Opus 5.5 f13b18149a Refuse weights the nodes' llama.cpp cannot load, before fetching them
kev-0.5b declared architecture "ggmlc", which llama.cpp does not know.
It was fetched, catalogued and placed, and failed on every node — while
the answer was in the first kilobytes of the file.

internal/gguf reads a GGUF header (v1-v3) up to general.architecture,
over HTTP with a Range request, and judges it against the architectures
llama.cpp b10991 lists in LLM_ARCH_NAMES (152, copied from the source at
that tag); a vision projector is not a model on its own. An unknown
build is not refused, only not vouched for, and a header that cannot be
fetched does not refuse anything.

Checked in three places: the registry listing (from the architecture
Hugging Face reports with expand[]=gguf, so one request per repository)
shows it and withholds the fetch button; a fetch reads the header and
refuses unless "fetch anyway"; and the model service refuses to add
such a model, which covers the serve form and ctl model add.

Tried on real files: the mirror's qwen2.5-1.5b (qwen2) and bge-m3
(bert), and Qwen2.5-0.5B on Hugging Face through its redirect.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 20:44:36 +02:00
Ilker ManapandClaude Opus 5.5 bf94cd1667 Set how a model runs from the mirror page
Every mirrored version was in use on the cluster, so the serve link — hidden for a version already served — appeared nowhere, and changing a model between always on and on demand needed the CLI. Each model using a file now shows how it runs and offers always on, on demand (15 minutes idle) or stopped, through the model service's Scale; models declared in server.yaml show their mode and where to change it. The serve form asks the same question.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 20:34:41 +02:00
Ilker ManapandClaude Opus 5.5 e493b2f53d Give an organization's managers a page of their own
A member of an organization can be its manager (users join --org X
--role manager, or the star on the people page). /org shows a manager
their organization: usage over time by team, project or person, its
teams and projects with limits and 30-day spend, and its members, each
once with every place they belong. Read-only; administrators see every
organization. Only what was spent in the organization's projects is
shown — UsageBucketsIn filters by where a request was spent.

Found on the way:
- Usage by org, team and project, and the quotas page's spend, joined
  through api_keys and so dropped every chat-page request, which has no
  key row. They now read usage.project_id; keys no longer move between
  projects, so the two agree.
- /org was not exempt from API-key auth and answered browsers with a
  JSON error. A router test now drives the session pages through the
  middleware; it fails without the fix.

Membership queries, org-scoped usage and the reworked joins pass the
PostgreSQL store tests on 16.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 19:23:12 +02:00
Ilker ManapandClaude Opus 5.5 098886e0e4 Let a person belong to many places, or to none
A person was in exactly one project, chosen at approval, and could not
use the gateway before that. People belong to several teams and to
none, so:

- memberships: a person is a member of any number of organizations,
  teams and projects, or of none; a member of a team or organization
  may use every project in it. users join/leave, and the people page
  adds and takes out from either side. Approval no longer needs a
  project.
- personal keys: a key may have an owner and no project. /profile asks
  what a new key spends against; keys create --for does the same.
  auth.personal sets default limits for personal use or turns it off.
- limits for people: --scope user holds a person across every key,
  project and the chat page; --per-member on an org, team or project
  gives each member their own share there. Resolution folds the tree,
  each per-member limit for the key's owner, and the owner's own, each
  in its own pool; the quotas page shows and sets both.
- the chat page asks what to spend against and checks it per message.
- usage records who spent it, so a person's budget survives a restart,
  personal use and chat included; restore no longer drops rows with no
  project.

Leaving a project revokes the keys made there unless another membership
still reaches it. A personal key keeps no threads or notes, which are
per project, and is refused rather than dropped into the anonymous pool.

Migration 0010 turns each person's project into a membership, adds
usage.user_id and backfills it, and makes api_keys.project_id nullable.
SQLite cannot drop NOT NULL, so the table is rebuilt; that needs foreign
keys off, which only works outside a transaction, so migrations marked
for it run on one connection with foreign_key_check before commit.
Tested on a populated SQLite store with claims, and on a copy of the
cluster's PostgreSQL store: 5 people became members, 17 keys and 3
claims intact, 145 of 577 usage rows and all 69 chat rows attributed.
The PostgreSQL store tests, including new membership cases, pass on 16.

users move is gone; join and leave replace it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 19:03:46 +02:00
Ilker ManapandClaude Opus 5.5 429c05a319 Show on the quotas page the limits that actually apply
A branch with no limit of its own read "—" in every column while its
organization held it to sixty requests a minute. Each cell now shows
the strictest value from above, muted and marked with an arrow, naming
its source on hover and in the form's placeholders. A larger value set
under a stricter one says what caps it, for rate and token limits as
well as budgets; only budgets were checked before.

The table follows the people page: indented tree, tinted organizations,
spend with a meter that turns red past 90%, a form that opens and saves
without reloading (htmx, falling back to plain links), and the
explanations folded under "How limits combine".

Rows are built from the store's Tree, so a project sits under its team
by id. Matching the start of its path put projects under the wrong team
when two organizations shared a name.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 18:19:49 +02:00
Ilker ManapandClaude Opus 5.5 c9cd25efc2 Draw one menu on every page
Each page wrote its own nav: the same links in a different order, some
missing, the current page sometimes listed and sometimes not. A shared
"nav" template now draws it everywhere — Cluster, People, Quotas, Usage,
Mirror for an administrator, then Chat, Your keys and Sign out — with
the current page marked by aria-current. Pages rendered outside the
account pages (the dashboard, node logs) get it through the request
context their admin wrapper sets. The admin listener's own page keeps
its metrics/pprof/api links.

The chat and key pages said "project proj-3kq7…"; they now show the
project's path, and the key page no longer repeats the email.

A test renders several admin pages and requires the same menu on each.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 17:55:36 +02:00
Ilker ManapandClaude Opus 5.5 c22c3a309b Describe v0.9.2
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 16:50:12 +02:00
Ilker ManapandClaude Opus 5.5 42fa602247 Make a new control plane serve a model by itself
server init now also:
- runs a model mirror beside the control plane (/srv/models, bytes on
  :8081 at an address nodes reach, management API on loopback so it
  needs no certificate) and adds it to server.yaml;
- makes the machine a node of its own control plane: issues its agent
  certificate, writes agent.yaml labelled as the control plane, starts
  llmgw-agent, and waits for it to register and install llama.cpp;
- checks for NVIDIA GPUs with nvidia-smi, and reports a card on the PCI
  bus whose driver does not answer rather than ignoring it;
- fetches Qwen2.5 0.5B (Q4_K_M) into the mirror and adds it to the
  catalog pinned to this node: a llama.cpp profile per GPU class found,
  with a CPU profile as the fallback, then waits for it to serve.

--no-mirror, --no-node and --no-model leave these out. Every step looks
before it acts, so a rerun finishes what an interrupted one started.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 16:49:55 +02:00
Ilker ManapandClaude Opus 5.5 5501f77805 Describe v0.9.1
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 16:37:43 +02:00
Ilker ManapandClaude Opus 5.5 75d69a6b9c Show the mirror one line per model version
Each held file took two table rows — seven columns and its address
beneath — so one sharded model filled the screen. A version is now one
line: model and version, the file or "N files", total size, and the
catalog models that use it. The address is behind a "copy address"
button (a set's is its directory, trailing slash included) and the
checksum is in the file's tooltip.

The model service reports which models name each source, so a file in
use shows who uses it and is not offered for serving a second time;
removing one warns that the catalog uses it. Downloads and failures
stay one line per file, since each has its own progress or retry. The
header counts versions, not files, and the nav calls /admin/users
"people" everywhere.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 16:36:36 +02:00
Ilker ManapandClaude Opus 5.5 beae60a76e Describe v0.9.0
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 16:27:54 +02:00
Ilker ManapandClaude Opus 5.5 3104f4b706 Manage people and the tenancy tree, from the web and the CLI
Nothing in the tree could be renamed, moved or removed, and an approved
person stayed in the project they were first put in — from the web or
the CLI. Now:

- /admin/users is the tree, each project opening onto its people and
  keys, with approve, move, suspend, reinstate and remove for people,
  revoke for keys, and add, rename, move and remove at every level,
  plus a search over every account. Forms post and get the page back,
  so it works with no script; htmx 2.0.11 (0BSD, embedded, checked
  against npm's integrity hash) swaps only the changed part, keeping
  open projects open.
- llmgw org tree/add/rename/move/remove, users move/delete, and
  keys list --all.
- Projects, teams and organizations are named by id, path
  (Acme/platform/gateway) or unique name; users by id or email. With a
  single project --project can be left out, instead of defaulting to
  "default", which a tree built from the web does not have.

Moving a person moves the keys they made; operator keys stay with the
project. Only empty things are removed, and never a project that had a
key, since usage history refers to it.

Also: keys init no longer reparents an existing team or project;
reinstating an account with no project is refused (a rejected signup
used to come back active and fail at its first key); rejecting a
signup removes it; keys limits --budget takes whole units like the
quotas page.

Checked in headless Chrome: a move from an open project updates the
page in place, keeps the project open, and the search still filters.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 16:26:39 +02:00
Ilker ManapandClaude Opus 5.5 5141e4b138 Describe v0.8.8
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 14:47:55 +02:00
Ilker ManapandClaude Opus 5.5 2e0465f10a Describe v0.8.7
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 14:35:28 +02:00
Ilker ManapandClaude Opus 5.5 40c3d83110 Describe v0.8.6
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 12:55:46 +02:00
Ilker ManapandClaude Opus 5.5 4865a96ec2 Describe v0.8.5
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 12:44:32 +02:00
Ilker ManapandClaude Opus 5.5 746d4852af Describe v0.8.4
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 12:27:47 +02:00
Ilker ManapandClaude Opus 5.5 a821a05840 Describe v0.8.3
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 12:14:22 +02:00
Ilker ManapandClaude Opus 5.5 8097a5616d Describe v0.8.2
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 11:50:12 +02:00
Ilker ManapandClaude Opus 5.5 f90daa1ee9 Describe v0.8.1
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 10:22:23 +02:00
Ilker ManapandClaude Opus 5 3a4260c8c5 Cut v0.8.0
The store stops being a file on one machine, which is what stood between this
and more than one control plane.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 07:26:54 +02:00
Ilker ManapandClaude Opus 5 b9c2be6cc2 Record what failover actually did
Three seconds to move leadership, six for every agent to arrive, and every
replica adopted rather than restarted. Also what it did not do: the public
address returned 502 while the standby served correctly on its own port,
because nothing in front of the two moved.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-25 19:39:55 +02:00
Ilker ManapandClaude Opus 5 fa60b81f37 Document moving to PostgreSQL, and when it is worth it
The section leads with why SQLite is still the default rather than with how to
leave it: one file, no server, and a request path that barely touches it. Two
things change that, and both are recent — conversations and notes read and
write on every turn, and a second control plane is impossible while the store
is a file on one machine.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-25 16:29:42 +02:00
Ilker ManapandClaude Opus 5 7dd1de6537 Cut v0.7.0
Seventeen commits since v0.6.0, and not a phase of the roadmap — which is why
the table leaves the phase column empty rather than inventing one. What landed
is what a cluster needs to be lived with rather than demonstrated: somewhere
to keep a conversation, somewhere to keep a note, and an honest answer about
the disk underneath both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-25 14:20:43 +02:00
deniz 2eb79f3400 agenta memory eklendi 2026-09-24 04:04:03 +03:00
Ilker ManapandClaude Opus 5 21a502035f Document conversations, starting with what is not kept
The section opens on the default rather than on the feature, because the
default is the part someone deploying this needs to be sure of: nothing anyone
types is stored until a project asks, and the command to ask is an act rather
than a setting.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 11:58:53 +02:00
Ilker ManapandClaude Opus 5 1b7e11b91e Document reclaiming disk, and what it deliberately will not reclaim
A command that deletes data needs its limits written down more than its
syntax. The section says plainly that llmgw only takes back files it fetched
itself, and then says the uncomfortable half out loud: that this means it will
not reclaim the storage most likely to be filling your disk, because weights
you placed yourself are yours to remove.

Also records the dash in the DISK column as "not asked" rather than "no
space", which is the reading that matters when a node will not schedule.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 09:59:16 +02:00
Ilker ManapandClaude Opus 5 a2b3f63a0d Say what v0.6.0 is, and let the release actually publish it
The changelog had two "Unreleased" headings. The second sat below v0.5.0 and
was the detail behind it — phases 2 and 3, written while they were in progress
and never folded in when the release was cut — so it claimed as unreleased
work that had been shipped for two days. It is now a section of v0.5.0. The
status table said v0.5.0 was planned, which its own release notes disprove.

The first "Unreleased" becomes v0.6.0: eighty commits, of which phase 4 is the
headline and the rest is what a cluster needs before anyone but its operator
is let near it.

Two things would have stopped the release workflow publishing anything:

Gitea gives every job a built-in GITEA_TOKEN, but what it may do depends on
the instance's default mode, and under the restricted default it is read-only
on releases. The job now asks for `releases: write`, which is correct under
either default. An upload that is refused now fails the job — without `-f`,
curl treats a 403 body as a successful transfer, so the release would end up
with some of its files, or none, and a green tick either way.

`workflow_dispatch` is offered but the upload assumed a tag. Fired from a
branch it would have created a release named "main".

README's status paragraph still said authentication, budgets and the control
plane were missing. It now says what is actually left, including the part
worth saying out loud: the InfiniBand, NVML and NCCL paths have never run on
a machine that has any of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 00:29:15 +02:00
Ilker ManapandClaude Opus 5 f84e3b642a Record the llama.cpp work in the plan and changelog
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:19:55 +02:00