Commit Graph
246 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 cfc636e328 A mesh's base colour can be brought to the picture it was made from
CI / test (push) Failing after 6m28s
Stable Fast 3D takes the light out of the picture it sees, and a dark
basalt (median 0.17) came back a light stone dimly lit (0.67). The save
node takes albedo_match, the picture and its mask, and scales the base
colour texture in linear light by the ratio of the object's medians, so
marks keep their contrast; texels nothing maps to are left alone. On the
game team's rock it is 0.18.

A workflow's {{name|false}} default now fills a BOOLEAN input with a
boolean rather than the word.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 00:51:06 +02:00
Ilker ManapandClaude Opus 5.5 848db08469 The cluster page and the console say which picture and 3D models are here, and add them
CI / test (push) Failing after 6m27s
A site whose catalog had no flux-schnell or triposr showed no Images or 3D
page, and nothing said they were one click away. The cluster page now has
a Pictures and 3D models card under the models: each preset, what it
makes, the classes of card here it fits, whether and by whom it was
added, and why the leader has not added it by itself; an Add button, which
says first when it would install ComfyUI on the GPU nodes and restart the
control planes; and the switch for presets.auto. The console's models
screen shows the same under its table: A adds one, asking first in the
same case, and P turns automatic adding on or off.

`ctl model add PRESET` and `ctl model presets` go through the presets API,
so the control plane keeps who added one, and `ctl model presets auto
on|off` flips the setting; against a control plane from before, add sends
the entry as v0.14.4 did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 20:48:40 +02:00
Ilker ManapandClaude Opus 5.5 f42af27da5 The leader adds the picture and 3D presets a node fits, unless told not to
A customer's portal had no Images or 3D pages: nobody had written a
definition for either, on a cluster whose V100 runs both. The leading
control plane now adds flux-schnell and triposr, on demand, once a node
has a card that fits one, the cluster installs ComfyUI, and the catalog
makes nothing of that kind. presets.auto (a live setting, on by default)
turns it off; a preset somebody removes is recorded as declined and not
added back until added again by hand. Each addition is in the audit log.

GET /api/v1/presets says how each preset stands, the ComfyUI the cluster
installs, and why one is not added; POST /api/v1/presets/{name} adds one
(setting engines.comfyui where there is none); PUT /api/v1/presets/auto
switches it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 20:48:40 +02:00
Ilker ManapandClaude Opus 5.5 d817558121 v0.14.4: changelog and README
Release / build (push) Successful in 2m4s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 9m28s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 10:37:05 +02:00
Ilker ManapandClaude Opus 5.5 295186a318 CHANGELOG: presets, ComfyUI on a V100, the open driver's version
CI / test (push) Successful in 10m11s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 10:21:51 +02:00
Ilker ManapandClaude Opus 5.5 7781b6a308 CHANGELOG: the gated repository a node loads by name
CI / test (push) Successful in 12m24s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 10:20:07 +02:00
Ilker ManapandClaude Opus 5.5 fa6e68c8e8 A model put first is not evicted, idle, for a lower class
CI / test (push) Failing after 7m59s
The VLM is to come first on the one card. Its class makes it take the card
from flux, triposr or sf3d when asked for, but idle eviction ignored class:
a 3D job two minutes into a lull would have taken the card from it. Idle
eviction offers only replicas of no higher class than the model wanting
room, and the wait such a model is told runs to the holder's idle timeout.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 10:15:14 +02:00
Ilker ManapandClaude Opus 5.5 3858114ada A picture in a prompt counts as what a model makes of it, not its base64
CI / test (push) Successful in 10m43s
A 1280x720 PNG sent to qwen3-vl-8b was estimated at 138 thousand prompt
tokens and refused against its 8k context before reaching the engine, which
makes about a thousand of it. Parts are counted by kind: text by its text,
a picture or a sound as a thousand tokens.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-11 20:45:06 +02:00
Ilker ManapandClaude Opus 5.5 ac4360b3fd A custom node's compiled packages are built for the card that runs them
CI / test (push) Failing after 6m23s
Stable Fast 3D's texture_baker and uv_unwrapper are C++/CUDA packages no
index has; on the PoC they were built and installed into ComfyUI's
environment by hand. nodes[].build names such directories, and the gate
builds each as a wheel with ComfyUI's Python, for the compute capability
torch reports for the replica's own card, keeps it on the node by the
node's commit, the architecture and the directory, and installs it
beside the replica with the node's requirements. A copy in ComfyUI's
environment does not pin it. A node with no CUDA toolkit says so.

comfyui.huggingface_repo shows the replica's weights to a node that
loads them by repository name, in the engines' Hugging Face cache, so a
gated repository needs no token on the node.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-11 17:23:05 +02:00
Ilker ManapandClaude Opus 5.5 4bbac0833d A person leaves licences out, and no key of theirs reaches a model under one
CI / test (push) Successful in 11m28s
A game sold in the EU cannot ship what a licence excluding the EU made. On
the profile page a person leaves out licences one by one, or by rule:
every licence forbidding commercial use, every licence granting nothing in
a region. The rules cover models added later. The choice reaches every
principal through the resolved limits, so permits() leaves such models out
of /v1/models, requests, jobs, fallbacks and the portal's pickers, and a
request for one is refused 403 licence_excluded saying which choice did.

Kept in user_excluded_licences (migration 0046), copied with the store and
gone with the person.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-11 16:10:22 +02:00
Ilker ManapandClaude Opus 5.5 777a9223a7 A profile may name the card it needs rather than its class, and fit any site
CI / test (push) Failing after 6m19s
A catalog entry meant for many sites cannot know their GPU class names:
the PoC has a Quadro RTX 6000, a customer an A100, another an L40S. A
profile may now say gpu: {min_compute_capability, min_memory_gb} in
place of gpu_class, and stands for every class in the cluster that meets
it, named <profile>@<class>, at its own priority; placement still sees
exact classes. ProfileByName and EngineProfile resolve such a name to
the profile on that class, so every caller that maps a replica back to
its profile works unchanged. A ComfyUI profile may carry its own
workflow, checkpoint and nodes, and with its own source an fp8 workflow
and weights go to cards with fp8 and fp16 ones to Turing.

A profile by need is left out of placement until the cluster's cards are
known: with no class it would have read as a CPU profile.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-11 16:02:32 +02:00
Ilker ManapandClaude Opus 5.5 b6bfc33ef7 Any ComfyUI workflow runs as a job, from the inputs its catalog entry declares
CI / test (push) Successful in 9m7s
A model's catalog entry now says what its workflow takes and makes:
comfyui.inputs, each a placeholder of its name with a type (text, int,
float, bool, choice, image, mask, audio, video, file), bounds, choices
and a default; and comfyui.output, which gains video, audio and files
beside pictures and meshes. POST /v1/workflows/runs with {model, inputs}
checks a request against them before a job is written — a required input
left out, an unknown one, a number out of bounds, a song sent as a
picture, another owner's file id are each refused with what was wrong —
and the job worker replays it to a replica, where the files go into
ComfyUI's input directory, the workflow is filled and run, and the first
file of the model's output kind is the job's result. /v1/models lists
each ComfyUI model's workflow, so a client can build its form.

A PNG result carries the licence in an iTXt chunk, as a GLB does in
asset.extras. A file whose server named no type is typed by its name
before its bytes, so a WAV is audio/wav, not audio/wave.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-11 14:28:15 +02:00
Ilker ManapandClaude Opus 5.5 0639edfd3b An adopted on-demand replica counts as used when adopted
CI / test (push) Failing after 6m25s
After a control plane restart flux, adopted and asked for by nobody, had no
last use and so was never idle enough to evict: a 3D job waited out its
whole fifteen minutes with no estimate. It counts as used at adoption, as it
is already counted as wanted then; a replica in use is reported by the
router within seconds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-11 08:55:33 +02:00
Ilker ManapandClaude Opus 5.5 95c61daf1b Jobs are served by the leader, and a moment of fitting nowhere is waited out
CI / test (push) Failing after 8m47s
A 3D job leased by the standby control plane, whose router has no endpoints
and which a job replayed in the process never leaves, was told "loading"
for seven minutes while the leader served the jobs behind it. Jobs and
batches are claimed only while leading.

Five jobs asked for while cuda01's agent restarted failed for good as
"cannot be placed": re-registering, the node had no engines for a few
seconds. A job asks again for five minutes before that is its end.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 23:53:33 +02:00
Ilker ManapandClaude Opus 5.5 b1490dbf30 A job waiting for the card says what holds it and about how long
CI / test (push) Failing after 6m16s
A 3D job waiting for the picture model to give up the one card had an empty
progress; its client could not tell waiting from stuck. The control plane
says which model holds the cards and when it lets go, plus the waiting
model's last start; the 503 carries it as headers, and a job keeps it as
progress {state: waiting_gpu, held_by, eta_seconds}.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 22:25:27 +02:00
Ilker ManapandClaude Opus 5.5 fd3c7bda32 The client API described in OpenAPI, and both APIs in Swagger UI
CI / test (push) Failing after 6m12s
/v1/openapi.json is the API clients use — OpenAI's endpoints, Anthropic's,
and what llmgw adds — in OpenAPI 3.1, generated from internal/clientapi's
route table by the generator the admin API's document comes from, now
told each document's title, server, token and error. A test reads the
router's mux.HandleFunc calls and refuses a route described and not
mounted, or mounted and not described. Asked for before there is a key,
so it needs none.

/docs/api shows it in Swagger UI 5.33.1 (Apache 2.0, served from the
gateway as the page's policy requires): Authorize takes a key, Try it out
sends to this gateway. Staff are offered the admin API's document too,
from /docs/api/admin.json, which nobody else is given.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 20:22:48 +02:00
Ilker ManapandClaude Opus 5.5 cf41adbf25 v1.0.0 in the roadmap is what phase 5 has left, not what it has done
CI / test (push) Failing after 8m29s
Tenancy, prefix affinity and benchmarking were built in v0.9 and after.
What phase 5 of llmgw-plan.md has open is precise KV routing proven on a
real vLLM, the block cost picker, defragmentation across NVLink domains
and prefill/decode disaggregation. Control-plane HA by Raft is left out:
HA is there already, by leader election on PostgreSQL.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 19:33:10 +02:00
Ilker ManapandClaude Opus 5.5 7f320c5f65 v0.14.3: changelog, README, docs for 3D models; a finished job says done
Release / notes (push) Successful in 5s
CI / test (push) Failing after 6m50s
Release / build (push) Successful in 2m6s
A finished job kept the last word heard from ComfyUI, the node running;
it is now said done before the job is marked completed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 19:22:58 +02:00
Ilker ManapandClaude Opus 5.5 ec233d3586 Merge admission-bulk into jobs-3d
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 17:35:39 +02:00
Ilker ManapandClaude Opus 5.5 63198f2753 A profile may load weights of its own
The catalog had one source a model, so an embedding model served by vLLM
from its repository on a card and by llama.cpp from a GGUF on CPUs had its
entry swapped by hand halfway through the metallurgy import. A profile's
source is what its engine loads and placement looks for; the model's is
the default, and what updates and canaries move. ctl model add checks
every source, the GGUF check reads each llama.cpp profile's, and ctl
weights collects none a profile loads. An empty source leaves a profile's
fingerprint as it was, so running replicas do not read as stale.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 17:15:38 +02:00
Ilker ManapandClaude Opus 5.5 497e4b24a1 A band may wait longer than the rest; a busy refusal says when to return
A knowledge-base import sends far more than the cluster can take at once,
and every band waited the same 30 s. A band now has its own queue_ttl, so
one for bulk work waits behind people instead of being turned away.

And a request turned away because the cluster stayed busy — the queue
full (503) or waited as long as its band may (504) — now carries
Retry-After: 10. Without it llmgw-kb, which retries only what says when
to, counted each such passage as failed; that is what turned parts of the
metallurgy import away.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 16:11:09 +02:00
Ilker ManapandClaude Fable 5.1 b51cc3cf92 A 3D model is a job: queued with an id, asked after, its result a file
POST /v1/3d/generations answers 202 with a job; GET says its status,
progress and result file; cancel stops it in the engine's queue too. The
request is kept as sent and replayed by whichever gateway holds the lease
through the path a client's request takes, so it is charged and judged
like any request (tagged job=<id>), and a model still loading is waited
out for as long as the job has. Migration 0044. Nothing makes 3D models
yet; the door refuses every model until the ComfyUI adapter learns meshes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-10 16:04:11 +02:00
Ilker ManapandClaude Opus 5.5 39bb8ee557 Two control planes send a notification once
Each kept its own hour of quiet per event, so a budget spent on a team
both control planes served was announced by each. An event is now claimed
in the store before it is sent (notify_sent, migration 0043; one statement,
so two asking at once cannot both win), and only the claimer sends it. A
store that cannot be asked sends it anyway.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 13:36:08 +02:00
Ilker ManapandClaude Opus 5.5 8d7b96621e A sign-in through the identity provider may end on either control plane
The cookie that carries a sign-in across the provider was signed with a
key each process made for itself, so with two control planes behind a
balancer a callback reaching the other one was refused as tampered — half
of all sign-ins. The key is now the store's oidc-state secret, sealed like
the others and the same on every control plane.

Changelog: the fixes since v0.14.2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 11:33:41 +02:00
Ilker ManapandClaude Opus 5.5 9e7618c073 v0.14.2: changelog and README
Release / notes (push) Successful in 4s
Release / build (push) Successful in 2m3s
CI / test (push) Successful in 8m44s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 02:12:08 +02:00
Ilker ManapandClaude Opus 5.5 37ad1657ee v0.14.1: changelog and README
Release / build (push) Successful in 1m59s
Release / notes (push) Successful in 5s
CI / test (push) Successful in 8m37s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 10:11:10 +02:00
Ilker ManapandClaude Opus 5.5 8df78a4d41 The firewall setting reaches the mirror machine and the store's members
The leading control plane tells the mirror over its API (PUT /v1/firewall,
kept as .firewall.json beside the models and applied by the same path unit
control planes use), and keeps the state in the store's cluster_state for
members that run no llmgw, which a root timer reads from the member's own
copy. The mirror also lets in the network of every node and control plane
the leader knows. Tried on the PoC: off, on and a network added, on every
kind of machine.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 08:53:17 +02:00
Ilker ManapandClaude Opus 5.5 15a209e652 Terraform: organizations, teams, projects, keys and memberships
llmgw_org, llmgw_team and llmgw_project rename and move in place;
llmgw_key holds the key in a sensitive secret and is made again when
revoked or expired outside Terraform; llmgw_membership puts a person in
the tree. Tried with OpenTofu against the fake admin API, drift and
import included.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 06:42:07 +02:00
Ilker ManapandClaude Opus 5.5 90e0942481 Organizations, teams, projects, keys and memberships in the admin API
What llmgw org, keys and users did against the store on a control plane
can now be done remotely and by automation: the tree read, added to,
renamed, moved and emptied; keys issued (the key in the answer, once),
listed and revoked, with the scopes' lifetime limits; people put in and
taken out of the tree, their keys in projects they no longer reach
revoked with it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 22:42:43 +02:00
Ilker ManapandClaude Opus 5.5 9c675e7293 Knowledge bases at scale: halfvec, a text index per collection, ef_search raised to the candidates wanted
A search asking for more than 10 passages got at most 40 candidates from
HNSW, pgvector's default ef_search. The full-text arm scored every chunk
holding a word of the question before limiting, over the whole
organization's GIN index; it now takes candidates by ts_rank from the
collection's own partial index and scores those. New collections index
halfvec on pgvector 0.7 and later (up to 4000 dimensions).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 22:08:29 +02:00
Ilker ManapandClaude Opus 5.5 42d3b7ea22 The store's PostgreSQL sized to its machine's memory
Each Patroni member gets a share of its own RAM in its own config.yml:
shared_buffers a quarter (15% beside a control plane or agent),
effective_cache_size, maintenance_work_mem and work_mem scaled to it, all
clamped. It is read again on every server init --patroni, so a machine
given more memory gets the use of it. A single-machine PostgreSQL init
installs is sized the same way, as a shared machine, instead of a fixed
96MB.

The step itself went in with 85025df by accident, in an early form that
restarted PostgreSQL before Patroni had taken the reload, leaving the old
shared_buffers with a restart pending. It now waits for Patroni to report
the restart pending, restarts with restart_pending (replicas first, the
leader last), and restarts a member it finds pending from an earlier run.

Applied to the PoC: pg01 978MB, ctl02 (control plane beside it) 587MB,
both from 128MB; streaming, lag 0, gateways unaffected.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 21:48:53 +02:00
Ilker ManapandClaude Opus 5.5 85025dfb03 Trained adapters published, private to their owners like the knowledge bases they come from
An adapter now has an owner (org, team, project, user or key): only callers
there see it, and to anyone else it is a model that does not exist, absent
from /v1/models too. A training job's owner defaults to the one collection
its pairs come from, must be able to read every collection they come from,
and pairs from a knowledge base are never made public. ctl train publish
keeps the GGUF or PEFT files on the mirror, which accepts uploads now, and
sets the owned adapter after the usual architecture check.

The knowledge bases' cache_salt follows the collections searched rather
than the organization, so two teams of one organization no longer share
cached passages.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 21:30:50 +02:00
Ilker ManapandClaude Opus 5.5 dc243f8840 The nodes' agent settings kept with the cluster's, agent.yaml down to what an agent needs to start
agents.default and agents.nodes.<id> in the cluster's settings hold
everything agent.yaml can say but the agent's id, control planes, address and
certificate, which stay in the file and are refused there. The control plane
sends each node its settings on register and, a second apart, when they
change; an agent whose settings changed checks they load beside its file,
keeps them and starts again on them, engines serving throughout. An edited
agent.yaml that loads restarts the agent too. History, audit, ctl config, the
console and the portal's settings page cover them as the rest of the
settings.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 18:06:36 +02:00
Ilker ManapandClaude Opus 5.5 659f20dc9a The cluster's settings edited on the portal, by administrators
/admin/settings edits the document llmgw ctl config edit opens: checked as a
control plane would load it, refused when made from a version since replaced
(the edit kept), its history viewed and rolled back as a new version, every
change audited by size and never by contents, since it can hold webhook
secrets. Administrators only; in English and Turkish.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 17:23:13 +02:00
Ilker ManapandClaude Opus 5.5 54570066e3 v0.14.0
Release / build (push) Successful in 6m21s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 9m34s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 16:22:07 +02:00
Ilker ManapandClaude Opus 5.5 496625824b Chat requests answered from knowledge bases
A request naming collections in X-Llmgw-Knowledge has its last question
embedded on the caller's account, the best passages found and added to the
system message, numbered, with an instruction to cite them and to say when
they do not hold the answer; x-llmgw-sources says which were given. Done
before the guardrails judge the request, never served from the answer cache,
and a collection the key may not read is answered as one that does not
exist. Metrics llmgw_kb_retrievals_total and llmgw_kb_retrieval_seconds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 15:27:50 +02:00
Ilker ManapandClaude Opus 5.5 eec0d92d18 The console's Knowledge screen, and the manual's page on knowledge bases
Collections listed with owner, model, documents and grants; A makes one; on a
row its documents, grant, revoke, a search to try, delete. docs/knowledge.md
says what administrators do and how people use llmgw-kb on Windows, macOS
and Linux: what it reads, what leaves their machine, where its ledger is.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 13:58:12 +02:00
Ilker ManapandClaude Opus 5.5 d35630defe Knowledge bases in the store: collections, grants and chunks kept apart per tenant
Collections owned by an organization, team, project or person; grants that
add up down the tree and never cross an organization; chunks in PostgreSQL
with pgvector, partitioned by organization, an HNSW index per collection, and
row-level security as a second line behind each query's filter. Search fuses
nearest-neighbour and full-text ranks. Off on SQLite and without pgvector,
saying why. server init and the HA database setup install and enable it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 12:47:57 +02:00
Ilker ManapandClaude Opus 5.5 0a7fb821d8 v0.13.0
CI / test (push) Failing after 4m47s
Release / build (push) Successful in 4m54s
Release / notes (push) Successful in 3s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 11:55:34 +02:00
Ilker ManapandClaude Opus 5.5 1fabb05fb2 A replica restarted in place keeps the room it frees for its replacement
The replica was stopped and the pass went on to the other models before
placing its replacement, so a model waiting for room took the freed cards.
On the PoC a GPU judge waiting for the only card took it from
qwen3.8-35b-a3b-distill, restarted there after its definition changed,
until the judge was stopped by hand. With the waiting model first in the
catalog it happened one pass later. The replacement is now placed in the
same step that stops the old replica.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 11:42:55 +02:00
Ilker ManapandClaude Opus 5.5 b0273ffb5f Qwen3Guard is the judge, and judges on a GPU and on CPUs are asked in turn
Llama Guard 3 1B could not tell ordinary coding requests from harmful ones:
on the PoC it scored "continue" 0.51, "continue the code above" 0.89 and
"how do I kill all child processes in bash" 0.92 unsafe, against 0.97 to
1.00 for harmful requests, at Q4_K_M and Q8_0 alike, with its own template.
A site had "continue" refused. Qwen3Guard-Gen 0.6B gave the ordinary ones
0.15 at the most and the harmful ones 1.00, in 0.8 GB on CPUs.

- format qwen3guard: its prompt written out as its template writes it, for
  requests and answers, and its Safe / Unsafe / Controversial verdict read;
  controversial counts as the policy's judge.controversial says, safe by
  default.
- judge.models: judges asked in turn, the first that answers deciding; a
  failure is recorded against the judge that failed. guard judge use sets
  the order, replacing categories none of the judges names.
- known judges (Qwen3Guard 0.6B/4B/8B, Llama Guard 1B/8B): guard judge
  enable serves any of them, --gpu the 4B on the roomiest GPU class as a
  standard model; the leader fetches and serves one a policy names that the
  catalog lacks.
- server init installs qwen3guard-0.6b with the baseline's categories.
- a judge reads a long text by its start and end, which fit its context,
  rather than overflowing it and refusing the request under fail: closed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 10:22:47 +02:00
Ilker ManapandClaude Opus 5.5 2d14918b99 An adopted engine that turns ready after registration is reported
The registration carries each shard's state as it is then. An adopted
engine is health-checked after that, and nothing reported the ready that
followed, so when the agent registered first the control plane kept the
replica starting while it served. After llmgw restart on the PoC two
replicas stayed so, one the only replica of qwen2.5-coder-14b. The
session's watcher now reports every shard the registration called not
ready once it is.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 19:43:13 +02:00
Ilker ManapandClaude Opus 5.5 7245dbe1dd llmgw stop, start and restart act on the whole system
Issue #3 asked for commands to stop and start the system, and 05aa21c gave
only one machine's services. Now:

- llmgw stop sets serving.stopped in the cluster's settings, live: the
  reconciler wants every model at no replicas, freeing every GPU, and the
  gateway answers every POST under /v1/ with 503 system_stopped. The control
  planes, agents and mirror keep running, and no model's definition changes.
- llmgw start clears it; every model is placed again as it asks.
- llmgw restart has every agent restart over its stream (RestartAgent), then
  the other control planes over ssh, standbys before a leader, then this
  one; engines keep serving.
- llmgw status is the cluster's summary, saying when it is stopped.

One machine's services moved to llmgw service status|start|stop|restart.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 18:01:45 +02:00
Ilker ManapandClaude Opus 5.5 05aa21c4d4 llmgw status, start, stop and restart for this machine's services
CI / test (push) Successful in 3m44s
There was no command to stop or start llmgw on a machine (issue #3). These
act on the llmgw units the machine has - mirror, server, agent - or those
named, in dependency order, saying what each did. stop leaves a node's
engines serving unless --engines.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 14:52:52 +02:00
Ilker ManapandClaude Opus 5.5 b5d650af16 What a replica computes on comes from nvidia-smi; llama.cpp says its layers only at -lv 4
Checked against the installed llama.cpp (b10991): at its default verbosity
it prints neither the devices it uses nor the layers it offloads, so the
start-up output said nothing and every replica's ON column stayed empty.
The agent now takes compute from nvidia-smi - the GPUs the shard's processes
hold memory on, and how much - or CPU, for every engine, falling back to it
when llama.cpp's output says nothing; adopted engines get it too, and each
session reports it once per shard as a compute-only status. engine-check
runs its GPU token at -lv 4, where llama.cpp does say where its layers went.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 14:04:08 +02:00
Ilker ManapandClaude Opus 5.5 0add189e02 A standby that takes over waits for the fleet before placing
The settling window ran from process start, so a standby that took over
after a minute placed at once, before the agents reconnected and reported
what they run: it started second copies of replicas it did not know of and
drained the healthy ones when they were adopted. Seen on the PoC: a leader
restart replaced llama-guard-3-1b and a qwen2.5-1.5b replica. Reconcile now
reopens the window on the first pass after standing by.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 13:40:28 +02:00
Ilker ManapandClaude Opus 5.5 66f702d5c1 A forgotten password is set through a one-time link
/forgot mails a link to set a new password when the cluster's mail settings
say how - SMTP relay, sender, and the public base_url the link points at,
never the request's Host - and says the same whether or not the address has
an account. /reset sets it once, closing every session. Links are kept by
their hash in password_resets, good for an hour, a newer one replacing an
older. Without mail, /forgot says to ask an operator, and llmgw users
reset-link makes the link to hand over. Single sign-on accounts get none.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 13:20:11 +02:00
Ilker ManapandClaude Opus 5.5 f285aa0548 v0.12.2
Release / build (push) Successful in 4m41s
CI / test (push) Successful in 4m43s
Release / notes (push) Successful in 3s
Guardrails off and on; a judge's refusal names the hazard; ctl guard judge ask.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 13:35:16 +02:00
Ilker ManapandClaude Opus 5.5 249a91aeec Guardrails turn off and on; a judge's refusal says which hazard; ctl guard judge ask
guard.enabled (unset is on) makes every policy check nothing while false,
live: llmgw ctl guard off/on set it in the cluster's settings, and ctl
status, ctl guard list and the guardrails page say when they are off.

A refusal by a judge names the hazard it saw - judge: S6,Specialized
Advice - and the verdict is kept with the request in a new usage column,
guard_rule, shown by ctl request, the request log and the guardrails page.

llmgw ctl guard judge ask has a judge read a text, past the cache, and shows
its raw answer and how it is read; the endpoint is kept out of the audit so
no text is recorded.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 12:41:08 +02:00
Ilker ManapandClaude Opus 5.5 3f5a809373 v0.12.1
CI / test (push) Failing after 4m20s
Release / build (push) Successful in 4m36s
Release / notes (push) Successful in 3s
The guardrails' judge served and watched without server init.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 10:52:24 +02:00