Commit Graph
90 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 cba3e329fc Let the mirror's serve form place a model on a GPU
The form offered no placement and named every GGUF profile cpu. It now
lists the fleet's GPU classes (the freest chosen) and the CPU, a GPU
count and parallel requests, names the profile from the placement, shows
what the model will run as, and refuses the CPU while GPUs are free
unless that is ticked.

Refs #2 (item 1).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 11:59:22 +02:00
Ilker ManapandClaude Opus 5.5 cbd5521f4f Place llama.cpp across several GPUs of a node with gpus_per_node
tp on a llama.cpp profile was a width nothing placed. gpus_per_node is
now the count: placement reserves that many cards on one node and the
engine splits layers across them. tp, pp and nodes are refused with
what to write instead.

Refs #2 (item 8).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 11:56:34 +02:00
Ilker ManapandClaude Opus 5.5 057be16a92 Do not call a GPU shard ready while it computes on the CPU
llama.cpp whose CUDA backend cannot load runs on the CPU without an
error. The CUDA build is now asked for its devices when installed, a
shard on GPUs must hold memory on each before it is ready, and
engine-check reports the devices. Shards get CUDA_DEVICE_ORDER=PCI_BUS_ID
so CUDA numbers GPUs as placement does.

Refs #2 (item 2).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 11:54:39 +02:00
Ilker ManapandClaude Opus 5.5 0c7934ba96 v0.10.6
Release / build (push) Successful in 1m11s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m51s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 08:52:01 +02:00
Ilker ManapandClaude Opus 5.5 e40b9ac202 Load LoRA adapters at scale zero
Release / build (push) Successful in 1m17s
CI / test (push) Successful in 4m0s
Release / notes (push) Successful in 5s
--lora with --lora-init-without-apply leaves an adapter's scale at 1,
and llama-server applies it at that scale to any request without a
"lora" list of its own: every plain request for the model was served
with its adapters on. Found live on qwen2.5-coder-7b-instruct. Each
adapter is now loaded with --lora-scaled FILE:0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 07:39:12 +02:00
Ilker ManapandClaude Opus 5.5 323d172538 Attach LoRA adapters from the command line
Release / build (push) Successful in 1m14s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m52s
llmgw ctl model adapters lists a model's adapters and adds or removes
them through POST /api/v1/models/{name}/adapters, which reads each new
file first and refuses one that is not an adapter for the model's
architecture, as the mirror page's card does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 07:30:13 +02:00
Ilker ManapandClaude Opus 5.5 663770226e Find LoRA adapters on Hugging Face and attach them from the mirror page
CI / test (push) Failing after 33s
Release / build (push) Successful in 1m3s
Release / notes (push) Successful in 3s
An adapters view in the explorer, filtered by the model a held model was
made from; GGUF files told apart as adapter or model by their first bytes;
held adapter files marked; and a model's card attaches and detaches held
adapters of its architecture, refusing models and mismatched architectures.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 07:19:08 +02:00
Ilker ManapandClaude Opus 5.5 cf53904cc5 Serve LoRA adapters on llama.cpp models, asked for by name
Release / notes (push) Successful in 3s
Release / build (push) Successful in 1m11s
CI / test (push) Successful in 3m47s
A model's adapters are fetched by each node with its weights and loaded
unapplied at start; a request naming an adapter reaches the model's
replicas with llama.cpp's per-request lora field. Adapter names share the
model namespace, are listed, and follow allowlists on their own. The
router's config seed and pushed catalog now build the name map in one place.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:40:47 +02:00
Ilker ManapandClaude Opus 5.5 94eec557ad Serve a model from an external OpenAI-compatible provider
CI / test (push) Successful in 3m47s
Release / build (push) Successful in 1m11s
Release / notes (push) Successful in 3s
A catalog model with external {url, model, api_key_file} and no profiles
is published as an endpoint by the control plane, the key read from its
file each time; the router rewrites the request's model to the provider's
name and sends its key. It is never scraped nor stale, and nothing is
placed for it. Named in fallbacks, it serves when the cluster has no room.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:29:01 +02:00
Ilker ManapandClaude Opus 5.5 d5f02c0a4b Spill a saturated pool's requests to the next pool
CI / test (push) Successful in 3m40s
The pool-priority filter kept the highest-priority pool however full it
was. It now takes the first pool, by priority, that is not saturated by
admission's own measure, and the preferred one when all are.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:23:32 +02:00
Ilker ManapandClaude Opus 5.5 3d3c96e422 Displace the newest lowest-band request when the gateway queue is full
Release / notes (push) Successful in 3s
Release / build (push) Successful in 1m9s
CI / test (push) Successful in 3m50s
A total bound across bands (admission.queue_capacity_total, default half of
all bands' capacity together); reached, a request is not refused while a
lower band has requests queued, and the newest of the lowest band gets a 503
with Retry-After instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:07:53 +02:00
Ilker ManapandClaude Opus 5.5 623cdb5ba3 Sign releases, and install only signed ones once a key is trusted
CI / test (push) Successful in 4m6s
ed25519 over checksums.txt: llmgw release keygen makes the pair, CI signs
with the RELEASE_SIGNING_KEY secret when it is set, and release.Install
refuses an unsigned or wrongly signed release when the build embeds a
trusted key (internal/release/trusted.pub, empty for now) or
LLMGW_RELEASE_KEY names one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 23:03:48 +02:00
Ilker ManapandClaude Opus 5.5 cf0178bf05 Test that a restart borrows a lower class's room and returns it
CI / test (push) Successful in 3m41s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 22:47:12 +02:00
Ilker ManapandClaude Opus 5.5 f288f3c423 Set capacity quotas, class, owner and reservation from the admin pages
Release / build (push) Successful in 1m9s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m49s
The quotas page gains a capacity section over the same API methods the CLI
uses; a model's card sets its class, owner, memory and cores, applying only
what changed. Headroom counts the replicas a model could get by preempting
what its class outranks, and the card and ctl model target use it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 22:40:45 +02:00
Ilker ManapandClaude Opus 5.5 5cf0003e74 Release v0.10.5
Release / notes (push) Successful in 3s
Release / build (push) Successful in 1m12s
CI / test (push) Successful in 3m53s
Collects rc.1 to rc.14: the mirror page in tabs with a card per model,
one line per record in every admin table, reserved CPU cores and required
memory, capacity classes and preemption, tenant capacity quotas, sizing by
measured throughput, and node sensor readings. Adds `llmgw upgrade` for the
mirror host, which `ctl upgrade` does not reach, and says so when it ends.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 22:26:19 +02:00
Ilker ManapandClaude Opus 5.5 cd10e086c8 Read nodes' heat and power: GPUs, CPUs, board inlet, RAPL
Release / build (push) Successful in 1m10s
CI / test (push) Successful in 3m51s
Release / notes (push) Successful in 3s
Every agent samples, at most every ten seconds: nvidia-smi's temperature,
HBM temperature, power draw, enforced limit and throttle reasons per GPU;
hwmon's CPU package and board inlet temperatures; and CPU package power
from RAPL's energy counters. Sent with heartbeats, published as metrics
and shown on the node page; unmeasured sensors are absent, never zero.
Observe-only, step 0 of docs/facility-aware-placement.md, which now
describes the agent source and what these readings can and cannot say.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 19:49:08 +02:00
Ilker ManapandClaude Opus 5.5 d0e9813ce9 Of equally idle victims, try the smallest first
Release / build (push) Successful in 1m11s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m55s
Idle times are unknown after a control plane restart, which left the choice
between a 1-core and a 4-core replica to map order.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 18:23:08 +02:00
Ilker ManapandClaude Opus 5.5 0e73f37128 Stop only the replicas that make the room
Release / build (push) Successful in 1m4s
CI / test (push) Successful in 4m32s
Release / notes (push) Successful in 3s
Candidates were added in order until the model fitted, and every one added
was stopped, including those tried first that freed room on the wrong node.
A set that works is now pruned to the replicas it needs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 18:21:58 +02:00
Ilker ManapandClaude Opus 5.5 78e41460f9 Scale and place by what a replica was measured to serve
Release / build (push) Successful in 1m8s
CI / test (push) Successful in 4m24s
Release / notes (push) Successful in 3s
Stored ctl bench results reach the reconciler: requests per replica past
the measured SLO concurrency scale a model up, a replica is taken back only
if the rest stay under 70% of theirs, and among profiles of equal priority
the most measured tokens per GPU or core is tried first. ctl model target
sizes a model for a throughput against the room there is.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 18:21:08 +02:00
Ilker ManapandClaude Opus 5.5 9dc1b62fdc Guarantee and limit capacity per part of the tenancy tree
Release / build (push) Successful in 1m8s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m51s
A model has an owner (org, team or project). Quotas guarantee a scope
GPUs, cores and memory and may limit it; what a scope holds is summed from
its models' replicas and those beneath it. A replica within its owner's
guarantee is not taken for another scope; what is held beyond a guarantee
is borrowed and taken back for a scope within its own. A limit stops a
model before it starts, and headroom stops there too. ctl quota and ctl
model owner set them; metrics and a CapacityAtLimit alert report them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 18:15:38 +02:00
Ilker ManapandClaude Opus 5.5 ac24e204a6 Add capacity classes and preempt lower ones to place a model
Release / build (push) Successful in 1m7s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m46s
A model's class (critical, standard, opportunistic; unset follows its
scaling) decides what may be stopped to make room for it: a lower class
for a higher, and within a class only what another grew into under load,
for a model below its minimum. Victims are tried on a copy of the
inventory and stopped only if they make the room. Preempted replicas are
drained with a short grace and their model is held back until the room is
used. ctl model resources and class move under ctl model.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 18:02:59 +02:00
Ilker ManapandClaude Opus 5.5 990d82d801 Reserve CPU cores and require memory for CPU models
Release / build (push) Successful in 1m5s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m47s
A profile's cpu_cores is reserved out of its node's CPUs by placement, as
memory is, and becomes the engine's thread count. Agents report their CPU
count in registration and heartbeats. New CPU models must declare memory;
model views show reserved against measured memory with warnings, and
ctl model resources sets both for a stored model.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 17:43:13 +02:00
Ilker ManapandClaude Opus 5.5 235bf33dcd Keep every admin table to one line per record
CI / test (push) Failing after 31s
Release / notes (push) Successful in 4s
Release / build (push) Successful in 1m4s
Headers never wrap, free text is clipped with the whole on hover, full-width
rows are only notes and forms. The cluster page hides columns empty on every
row and shortens node names; People shows two memberships and a count.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 11:02:18 +02:00
Ilker ManapandClaude Opus 5.5 7099043c75 Keep each held row on one line
Release / build (push) Successful in 1m8s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m53s
The replica count, its nodes and the mode selector sit side by side instead
of stacking in a fixed-width cell, and node names drop the prefix every node
shares, with the full name on hover.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 10:50:02 +02:00
Ilker ManapandClaude Opus 5.5 a32ea0e5ab Say on each held row how its model runs and where
Release / build (push) Successful in 1m8s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 4m0s
Ready of wanted, how many are starting, and the nodes, from the fleet data
the card already reads, so the list answers it without a click.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 10:38:49 +02:00
Ilker ManapandClaude Opus 5.5 d7f1c1d545 Open a held model to its replicas, nodes and room for more
Release / build (push) Successful in 1m6s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m48s
The Held tab's model name opens a card: each replica with its node, state,
age, CPU and memory, the room for more found by dry-running the scheduler
on a copy of the inventory, and a form for the always-running and
under-load replica counts. An always-running count placement cannot meet
is refused with the reason. Headroom is also on the admin API.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 10:19:31 +02:00
Ilker ManapandClaude Opus 5.5 2ae12ae1fe Split the mirror page into Held, Explore and Fetch tabs
CI / test (push) Successful in 3m45s
Release / build (push) Successful in 58s
Release / notes (push) Successful in 3s
One long page made every visit wait on Hugging Face for a list it might not
show. Each tab now has its own address and does only its own work; only
Held reloads while a download runs, and a refused fetch returns to the form
it came from.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 08:31:28 +02:00
Ilker ManapandClaude Opus 5.5 62657f4c2a Release v0.10.4: CPU threads, tracing, and the rest of the review
Release / notes (push) Successful in 3s
CI / test (push) Successful in 4m57s
Release / build (push) Successful in 4m11s
CPU replicas start on half the machine's CPUs, not all of them: two busy
llama.cpp servers each spinning on all 8 vCPUs fell to 0.8 tok/s apiece,
where at 4 threads they keep 5.5. llamacpp.threads sets it per node.

Inference requests can be traced over OTLP/HTTP (tracing.endpoint), with
admission, schedule and upstream spans and traceparent sent to the engine.

Security: onboarding no longer puts the cluster root in the system trust
store and removes it where it was; the mirror's mutual-TLS API can be
limited to named certificates (--api-client); init.log no longer keeps the
administrator's password; a warning when auth is off on a public listen;
the recorded actor is sanitised and sudo-aware; Ray's dashboard is off, and
the remaining engine gaps are documented.

reconciler.go and agent.go are split by topic, the status mapping of each
failure is tested, and the plan's checklist is brought up to date.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 04:00:35 +02:00
Ilker ManapandClaude Opus 5.5 adb1b3cf2a Add replicas under load, up to a model's max, and take them back when quiet
Release / build (push) Successful in 1m2s
Release / notes (push) Successful in 2s
CI / test (push) Successful in 3m48s
A model's max never did anything: desiredFor returned the minimum, or one
for an on-demand model someone was asking for, whatever the queue. Six small
CPU nodes run out of slots quickly, and a model with room to grow stayed at
its minimum while requests waited.

The router already scrapes each engine's running and waiting requests. The
control plane now reads them per model on every tick, alongside when each
replica was last used, and the reconciler keeps how many replicas load has
added above a model's base. One or more requests waiting per ready replica,
held for a minute, adds one; ten minutes with nothing waiting removes one of
those again. It never goes above max or below the base, moves one step at a
time, and does not add while the last addition is still placing or starting
— on this fleet a replica is minutes of weights and gigabytes of memory, and
a spike should not leave a pile of replicas behind it. Decisions are logged
with the load behind them.

Nothing live changes until a model's max is raised above its min.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 03:18:07 +02:00
Ilker ManapandClaude Opus 5.5 bd12b0909a Serve a model from the registry with one command
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m50s
Release / build (push) Successful in 1m8s
Adding a model took four steps and a YAML file — plan it, fetch it into the
mirror, write its catalog entry with a memory figure somebody estimated,
scale it — and whether it would load was found out at the end.

`llmgw ctl model serve OWNER/REPO` does it as one job in the control plane,
where the registry, the mirror, the catalog and the fleet already are:

- the registry: the GGUF file set matching --quant (Q4_K_M by default) or
  --file, leaving out vision projectors and importance matrices, and naming
  what the repository does have when nothing matches; gated repositories are
  refused;
- the mirror: fetched at the repository's revision, with progress, and each
  file's sha256 compared with what the registry published — the mirror is
  the one place a corrupted download is caught once rather than loaded on
  every node;
- the check: whether the nodes' llama.cpp loads the architecture, and the
  memory it needs from its own header;
- the catalog, then a wait for a replica that stops early, with the reason,
  when the model is parked or fits on no node;
- a smoke test: one short request to the replica, with its engine key, and
  the answer and tokens per second.

ctl starts the job and prints each step as it changes; a download of hours
does not depend on the terminal. --on-demand adds the model without starting
a replica. Jobs are kept in memory until the control plane restarts.

The catalog name a repository suggests moved into the registry package, so
the mirror page and this agree on it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 03:05:13 +02:00
Ilker ManapandClaude Opus 5.5 116e75f8cb Upgrade the cluster with one command, over the agents' own streams
Release / build (push) Successful in 1m5s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m42s
Every upgrade so far has been a script of mine copying a binary to each
machine over ssh, restarting it and polling until it came back — which is
the job, but not something the product could do, and which needed ssh into
every node.

`sudo llmgw ctl upgrade VERSION` on the control plane does it. This machine
first: the release for its platform, checked against the release's
checksums, the old binary kept beside it, llmgw-server restarted, and a wait
until every node that was ready has reconnected. Then each node in turn: the
control plane tells its agent the version over the stream it already holds;
the agent fetches and checks the release itself, swaps it in with a rename,
answers, and exits with status 75 so systemd starts the new binary. Engines
keep running, as for any agent restart, and are adopted again.

A node is done when it is back on the new version, ready, with as many ready
replicas as before; one that is not within the timeout stops the run before
the next node, so a bad release reaches one machine. Nodes already on the
version are skipped, so running it again resumes.

The fetch-and-verify is one package now, shared with onboarding's idea of
what a release version looks like.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 02:54:36 +02:00
Ilker ManapandClaude Opus 5.5 3bf4b7b709 Tell clients how much of their rate limit is left
CI / test (push) Successful in 3m49s
Release / build (push) Successful in 1m4s
Release / notes (push) Successful in 3s
SDKs and editors pace themselves from OpenAI's x-ratelimit-* headers, and
the gateway sent none: only Retry-After, and only on the refusal they would
have avoided. A limit can be set at several levels of the tenancy tree, each
counted in its own pool, so the headers report the one with least left —
the one that will refuse first — after this request's own charge, on
refusals and successes alike. The windows slide, so the reset figure is the
next minute boundary, when a whole window has moved on. A limit that is not
set is not advertised.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 02:42:21 +02:00
Ilker ManapandClaude Opus 5.5 91fe24a105 Close what a second security review found still open
Release / build (push) Successful in 1m8s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m50s
A review of v0.10.4-rc.4, and a look at the live network, found:

- Alertmanager listening on every interface: from any node its API took
  silences, with no login. Bound to loopback on the monitoring host, and the
  monitoring README now installs it that way; Prometheus beside it is its only
  client, and one Alertmanager needs no cluster port.
- Login and signup unbounded in memory. Each attempt runs argon2 with 64 MiB,
  and nothing limited how many ran at once; the per-address throttle kept one
  entry per address ever tried. Hashing now takes one of a few slots and
  waits for one otherwise. The throttle expires entries whose window has
  passed when it fills, and if it is still full it refuses new addresses:
  evicting live ones instead would let a flood of made-up addresses reset the
  throttle on the address actually being guessed.
- Metric labels taken from the request. A refused request counted the model
  name it asked for, so any key holder could mint a series per request.
  Requests are counted under the catalog's name, or "unknown".
- Two in code written this week. A SQLite backup made its snapshot with the
  default mode in the listable backup directory, a full copy of the store
  readable while the backup ran; it is now made in a private directory, and
  the unit sets UMask=0077. And archive links were checked by name only, so
  "a -> ." then "a/a/b -> ../.." passed and a file written through them landed
  above the install directory, as root. Archives now unpack through os.Root,
  and a link may only point at a file beside it — which every llama.cpp
  b10991 archive, CPU and CUDA, was checked to follow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 02:25:07 +02:00
Ilker ManapandClaude Opus 5.5 1da4a2e1ef Back up the store and the certificate authority, and alert when it stops
Release / build (push) Successful in 1m1s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m32s
There were no backups: docs/install.md gave a pg_dump command, and nothing
ran it. The store holds every key, person, quota and usage record, the models
added at runtime, and now the secret each engine's key is derived from; the
certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk
lost all of it.

`llmgw server backup` writes both to one archive: pg_dump's custom format, or
a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's
key, so it is 0600; the directory is listable, so the control plane — not
root — can see when the newest was written. pg_dump is given the connection
through its environment, since a DSN carries the password and a process's
arguments are readable by every user. Written beside its name and renamed,
so an interrupted backup is never the newest; the newest 14 are kept, and
nothing the command did not write is ever removed.

llmgw-backup.timer runs it nightly and catches up after downtime. The control
plane publishes the newest archive's time, and BackupTooOld fires past 26
hours — or when there has never been one. The install guide now covers
restoring, and says to copy archives off the machine, since they share its
disk.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:44:32 +02:00
Ilker ManapandClaude Opus 5.5 37a80a9b44 Park a model that fails every time it starts, and say so everywhere
Release / build (push) Successful in 1m3s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m34s
On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not
know — was started 155 times in eight hours, failing the same way each time,
and phi-4, which fitted on no node, logged "no placement" 250 times in two
days. Neither showed up anywhere an operator looked: the backoff never gave
up, no metric said a model was failing or unplaceable, and the one alert for
a model with nothing serving could not fire, because its series was published
only while a replica was ready.

After five failures in a row a model is parked: not placed again until its
definition changes — a new source or profile — or `llmgw ctl retry` is run.
Status, the model list and the admin API show it with its last error. The
placement warning is logged when its reason changes or every ten minutes.

A request for a parked or unplaceable model is answered at once with why,
rather than "loading; retry shortly", which a client retries forever. Each
request still clears the previous placement result and triggers a fresh
attempt, so a model that fits once room is made is not refused on a stale
answer; the router waits for that attempt, not for its full activation
window.

Metrics now cover model health (desired, ready at 0 for what should run,
consecutive start failures, parked, unplaceable), engine installs per node,
leadership, and certificate expiry read from disk at each scrape, with alerts
for each. `ctl status` counts a model against what it should have now, so an
on-demand model that was asked for and fitted nowhere no longer reads as
satisfied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:21:57 +02:00
Ilker ManapandClaude Opus 5.5 5e1b6ecea0 Read a queued change as a success in ctl
ctl accepted only 200, so the first real `ctl restart --all` printed
"Error: 202 Accepted" over a request that had queued nine restarts, all of
which then happened. Any 2xx is now a success, and its body is read.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:14:12 +02:00
Ilker ManapandClaude Opus 5.5 5dac1854c2 Publish small fixes as release candidates of the next version
Release / build (push) Successful in 1m1s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m30s
Every fix so far became its own patch release, and the natural alternative —
v0.10.3-a — sorts before v0.10.3 by SemVer's rules: a hyphen marks a
pre-release, so every tool that orders versions would take the fix for the
older one. A release candidate of the next version, v0.10.4-rc.1, sorts
after v0.10.3 and before v0.10.4, which is what it is.

CI already builds any v* tag. Its release notes now come from a
"## Unreleased" section in CHANGELOG.md, where changes gather until the
version they belong to is released, and the release is marked as a
pre-release. Onboarding, in Go and in the shell script, counts a candidate
as a release and downloads it rather than copying the operator's binary.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:00:11 +02:00
Ilker ManapandClaude Opus 5.5 611fddd236 Describe v0.10.3
Release / build (push) Successful in 1m4s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m39s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 00:38:55 +02:00
Ilker ManapandClaude Opus 5.5 bb90a927aa Print only what must outlive init's screen, and drop bubbles
Once server init's screen closed, the whole run was printed again under the
frame that had just shown it. Now the password and the next steps are, the
whole run goes to /var/lib/llmgw/init.log (0600: it holds the password), and
it is printed in full only when init fails. Seen in a draft of the install
video, where it scrolled the screen away.

The spinner and the text field are written here instead of taken from
bubbles, whose packages query the terminal's colours as they load. bubbletea
does the same deliberately, so the query stays; the dependency goes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 22:37:28 +02:00
Ilker ManapandClaude Opus 5.5 49c446271c Describe v0.10.1
Release / build (push) Successful in 1m1s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m39s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 22:17:33 +02:00
Ilker ManapandClaude Opus 5.5 b5f412f9df Put the mirror on its own machine in two commands
mirror init sets a machine up as the model mirror — the user, /srv/models,
and a unit serving files on :8081 and the management API on :8082 — and
server init --mirror points the control plane at it instead of running one.
Before, that took a unit written by hand and an edit to server.yaml.

The management API can fetch and delete models and answered anyone who
could reach it. mirror serve --api-allow limits it to the addresses given
and the machine itself; mirror init passes the control plane's. The unit
text is shared with the one server init writes for a mirror beside it.

With --no-node the starter model is still fetched and catalogued, unpinned,
so it starts on the first node added, and the init form shows the mirror in
use instead of offering to run one.

Tried on three empty VMs: the mirror answered the control plane and refused
the node, and the starter model came up on the node when it was added.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 22:17:24 +02:00
Ilker ManapandClaude Opus 5.5 a1b5daceb8 Let server init ask for what it needs, and node add take several nodes
In a terminal and without --admin, server init shows a short form: the
administrator's email, the other names nodes will use (with the addresses
it found), SQLite or PostgreSQL, and the mirror, node and starter model.
Its own lines then fill in as a list — read from the writer init already
uses, so nothing in init changed — with downloads updating in place and the
password in a box until the end. The whole run, password included, is
printed again when the screen closes, so it is in the terminal's history.
The last screen says how to add a node and to back up the authority's key.

node add takes several addresses and adds them in turn, with a summary.
Suggested fabrics carry what the adapters report them to be, so a name
given for one node is used for the next on the same fabric without asking.

Tried on empty VMs: init from the form through to the starter model
serving, and two nodes added to that cluster in one command. The first
attempt at the form lost its keystrokes to the terminal colour query in a
pseudo-terminal that never answers it; a real terminal does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 21:54:18 +02:00
Ilker ManapandClaude Opus 5.5 7bb360fc78 Measure two nodes' fabric rail by rail: llmgw ctl fabric-test
doctor checks a node alone, and every check can pass while two nodes still
cannot reach each other. ctl fabric-test A B asks B's agent for an
ib_write_bw server and A's for the client, adapter by adapter as NCCL pairs
them, one rail at a time, and grades each against its link's rate. It goes
through the agents' streams, so there is no SSH between nodes; the agent
installs perftest the first time, and names the RoCE v2 GID itself.

The result is read from perftest's table by its heading, not a column
number. Agents now list the requests they answer beyond the usual
(Register.features), and the control plane refuses a test on one too old to
answer instead of waiting minutes for it. The forwarding dispatcher carries
the new capability, and its test says so: a missing forward would have made
the command quietly unavailable in production.

Not yet run over real InfiniBand or RoCE; this cluster has neither.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 21:29:23 +02:00
Ilker ManapandClaude Opus 5.5 af6f8e023e Describe v0.10.0
Release / build (push) Successful in 3m41s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 4m24s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 21:06:42 +02:00
Ilker ManapandClaude Opus 5.5 590e5b9f3e Ask a node only for engines it can install; doctor says when AVX2 is hidden
A control plane of this release asks every node for vLLM. An agent of the
last one does not know how to install it, and answered "unsupported", so a
cluster half way through its upgrade showed every node as failing. Agents
now say which engines they can install (Register.installable_engines), and
are asked for no others; an agent that says nothing is asked for llama.cpp
alone, as it always was.

doctor reports the CPU's vector extensions, and warns when AVX2 is missing,
with the fix for a VM: the test node for node add was one, on QEMU's default
CPU model, where llama.cpp runs several times slower and nothing else looks
wrong. node add shows it, since it runs doctor.

The changelog's unreleased section was two releases' worth of sections one
after the other, with two lists of fixes and the upgrade notes in the
middle. It is one entry now: largest changes first, the fixes together, and
every upgrade note at the end, with vLLM's default install and how to keep
your own.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 21:03:03 +02:00
Ilker ManapandClaude Opus 5.5 2a16c26575 Add a node with one command: llmgw node add, with a terminal screen
Adding a node was deploy/bootstrap/onboard-node.sh, which is not installed
anywhere and wanted the node's name, the control plane's address and a
dozen flags. It is now in the binary, run on the control plane:

    sudo llmgw node add ubuntu@10.0.0.21

The node id is the machine's hostname, and the address its agent dials is
this machine's on the route to it, checked against the server certificate.
It installs llmgw there (the release for its architecture, fetched and
checked by the node; uname rather than dpkg, so RHEL works), runs doctor
there and stops on a problem before changing anything, issues and installs
the certificate, writes agent.yaml and the unit, starts the agent, and waits
for it to join and install llama.cpp. The authority is read first, so a
missing one is said before the node is touched.

The steps are events. In a terminal they draw a checklist (bubbletea) that
asks what a script cannot: which fabric each group of active adapters is on,
grouped by the subnet manager or RoCE network they report, and whether to go
on past a problem. Elsewhere, one line per step.

Found by running it on an empty VM: ssh's "Permanently added" warning on
stderr was read as the machine's hostname, so standard error is now kept
apart; and a node added again while cordoned never counted as joined, so
joining now means a heartbeat since the restart.

llmgw node remove forgets a lost node, which otherwise stays listed until
the control plane restarts. ctl nodes no longer lists vLLM as skipped on
every CPU node. The quick install, install and onboarding docs use it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 19:33:47 +02:00
Ilker ManapandClaude Opus 5.5 4c5a2fd6da Install vLLM on every GPU node that can run it, as llama.cpp is
vLLM was installed by hand and named in each agent's configuration. The
agent now installs it: a Python 3.12 environment built by uv, which is
itself a GitHub release checked against its checksum, from PyPI resolved as
it stood a day after that vLLM was published and installed with
--require-hashes. The resolved list is kept in the environment. The
environment is built relocatable under a scratch name and renamed into place
once it runs, like an archive.

The default is 0.30.0, whose flags were checked against the ones llmgw
builds. It is built for CUDA 13, so it needs driver 580 and a Turing GPU or
newer, and 25 GB free while it installs; a node lacking one says which, and a
node with no GPU skips it without being shown as a fault. gcc comes with it
because Triton compiles its launcher at the first model load. SGLang
installs the same way when the control plane names a version. llmgw doctor
reports whether vLLM can be installed on the node.

Checked by installing 0.30.0 this way on a test node: 7.7 GB, torch 2.13 for
CUDA 13.0, importing cleanly; the node has no GPU, so vLLM was not started.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 13:15:54 +02:00
Ilker ManapandClaude Opus 5.5 e751c89937 Serve the project's website at /about, with its fonts in the binary
The website was a file built by hand outside the repository. It is now a
template in internal/web: every gateway serves it at /about, without a key,
linked from the landing page's footer, and the published site is rendered
from the same template.

The fonts are vendored (Latin subsets, OFL) so the page sends nobody's
browser to Google. The proxy configurations it shows are copies of
deploy/proxy, kept equal by a test. A development build says so instead of
linking to release files that do not exist.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 10:44:45 +02:00
Ilker ManapandClaude Opus 5.5 80e0dd252f Document what changed since v0.9.2, and fabrics for the sites that have them
The changelog's unreleased section had stopped at refusing unloadable
weights; it now covers the rest: telling independent fabrics apart,
their subnet managers and maps, NVLink on every build and across nodes,
the multi-node launch fixes and vLLM's new default, doctor, node load,
editors and the Responses API, the chat and mirror pages, a person's own
use, the admin pages, the new metrics, and how to upgrade.

docs/fabrics.md is new, for a site with InfiniBand or NVLink: what
llmgw does with them, what each node and fabric needs first, declaring
fabrics, checking a node with doctor, what the pages and metrics show
and which to alert on, what an engine is given, and where to look when
a replica spanning nodes will not run. It says plainly that none of it
has yet run on such hardware.

install.md, node-onboarding.md, onboarding.md, the README and the
example agent configuration no longer say the portable binary cannot
see NVLink or that a fabric's name cannot be checked, and describe the
account page and connecting an editor.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 10:14:58 +02:00
Ilker ManapandClaude Opus 5.5 f13b18149a Refuse weights the nodes' llama.cpp cannot load, before fetching them
kev-0.5b declared architecture "ggmlc", which llama.cpp does not know.
It was fetched, catalogued and placed, and failed on every node — while
the answer was in the first kilobytes of the file.

internal/gguf reads a GGUF header (v1-v3) up to general.architecture,
over HTTP with a Range request, and judges it against the architectures
llama.cpp b10991 lists in LLM_ARCH_NAMES (152, copied from the source at
that tag); a vision projector is not a model on its own. An unknown
build is not refused, only not vouched for, and a header that cannot be
fetched does not refuse anything.

Checked in three places: the registry listing (from the architecture
Hugging Face reports with expand[]=gguf, so one request per repository)
shows it and withholds the fetch button; a fetch reads the header and
refuses unless "fetch anyway"; and the model service refuses to add
such a model, which covers the serve form and ctl model add.

Tried on real files: the mirror's qwen2.5-1.5b (qwen2) and bge-m3
(bert), and Qwen2.5-0.5B on Hugging Face through its redirect.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 20:44:36 +02:00