Commit Graph
112 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 6bef138c23 The judge check gives each judgment a text of its own
The same text again was answered from what the engine kept of the last
reading, in a tenth of the time: the check read 4.9 s, then 0.5 and 0.5.
Each judgment now begins with a line of its own.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 22:45:58 +02:00
Ilker ManapandClaude Opus 5.5 85f135311e How fast the judge is, measured, and said at install
ctl guard judge check, and POST /api/v1/guard/judges/check, time three
judgments of a 280-token prompt, then a burst at once, against the
policies' wait: ok, tight or slow, about how many requests a second can
be judged, and what to do. server init runs it once the judge is up and
says a machine too slow before the next steps; the wait is not changed
without asking. BACKLOG reordered by what goes wrong where installed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 22:35:19 +02:00
Ilker ManapandClaude Opus 5.5 4eb49d0e4c The cluster as a read-only MCP server, llmgw:admin
Its tools say what llmgw ctl does: cluster_status, list_nodes,
list_models, list_replicas, explain_placement, node_logs; none changes
anything. The admin API serves it at POST /api/v1/mcp, and the gateway
answers a server registered as llmgw:admin in its own process, so no
token is kept and each control plane serves it. ctl mcp add takes
--bearer-file for a token sent as Authorization: Bearer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 17:18:28 +02:00
Ilker ManapandClaude Opus 5.5 4453f469e1 Company MCP servers are registered with ctl mcp, and tools granted with keys limits --tools
llmgw ctl mcp add|list|remove|test and /api/v1/mcp/servers keep the
servers; a token given with --secret-file is kept in the store, sealed,
and named by reference. keys limits --tools, and the console's limits
form, grant tools as patterns. Documented in install.md and clients.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 12:50:55 +02:00
Ilker ManapandClaude Opus 5.5 ac25a9136a Sign-in is set up from the console, and a directory tried before it is kept
People → Sign-in shows single sign-on, Active Directory and SCIM, and
the directory's last sync. E sets up Entra ID from the tenant and client
IDs; D sets up Active Directory, tried from the control plane first:
each server bound, people who may sign in, members of each mapped group.
The secrets go into the store as store:auth/..., sealed by the master
key, and the CA into the settings, so no control plane needs a file.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 12:55:27 +02:00
Ilker ManapandClaude Opus 5.5 fc60481e40 Every control plane lists the others alike to Prometheus
140 named 150 a store member, 150 named itself a control plane: Prometheus
asking both scraped 150 twice and counted its every series twice, and asked
of 140 alone while 150 led it lost every node. Each control plane now lists
all of them as control planes before the store's members, so two answers
agree and a target is scraped once. The shipped prometheus.yml says to
scrape and ask every control plane.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 07:53:22 +02:00
Ilker ManapandClaude Opus 5.5 f42af27da5 The leader adds the picture and 3D presets a node fits, unless told not to
A customer's portal had no Images or 3D pages: nobody had written a
definition for either, on a cluster whose V100 runs both. The leading
control plane now adds flux-schnell and triposr, on demand, once a node
has a card that fits one, the cluster installs ComfyUI, and the catalog
makes nothing of that kind. presets.auto (a live setting, on by default)
turns it off; a preset somebody removes is recorded as declined and not
added back until added again by hand. Each addition is in the audit log.

GET /api/v1/presets says how each preset stands, the ComfyUI the cluster
installs, and why one is not added; POST /api/v1/presets/{name} adds one
(setting engines.comfyui where there is none); PUT /api/v1/presets/auto
switches it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 20:48:40 +02:00
Ilker ManapandClaude Opus 5.5 fd3c7bda32 The client API described in OpenAPI, and both APIs in Swagger UI
/v1/openapi.json is the API clients use — OpenAI's endpoints, Anthropic's,
and what llmgw adds — in OpenAPI 3.1, generated from internal/clientapi's
route table by the generator the admin API's document comes from, now
told each document's title, server, token and error. A test reads the
router's mux.HandleFunc calls and refuses a route described and not
mounted, or mounted and not described. Asked for before there is a key,
so it needs none.

/docs/api shows it in Swagger UI 5.33.1 (Apache 2.0, served from the
gateway as the page's policy requires): Authorize takes a key, Try it out
sends to this gateway. Staff are offered the admin API's document too,
from /docs/api/admin.json, which nobody else is given.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 20:22:48 +02:00
Ilker ManapandClaude Opus 5.5 2f54780802 G4: a page that makes 3D models, and ctl 3d
/3d beside /images, while some model makes meshes: a picture chosen, the
job asked for through the same path an API client's takes, its progress
shown (waiting for the card, n ahead; making it, the node and percent),
and the mesh in a viewer — model-viewer 4.4.0 (Apache 2.0, with three.js
and lit), served from the gateway since the page's policy loads scripts
from nowhere else — with a download and the model's licence. The mesh is
served to its owner from /3d/jobs/{id}/file; another project's person is
told there is no such job.

A job made from the portal acted for a key that does not exist: the
worker now rebuilds a session's principal from the account that made it,
while it is active and still in the project the job is charged to.

ctl 3d list|get|cancel, over /api/v1/client-jobs (/api/v1/jobs is the
control plane's own work, which ctl jobs shows).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 19:16:20 +02:00
Ilker ManapandClaude Opus 5.5 c87e413360 A ComfyUI model may make meshes: from a picture, with its licence inside
G3, the part that needs no model: a ComfyUI model's comfyui.output (or
external.output) says pictures or meshes. A request for a 3D model — a
job at /v1/3d/generations — carries the picture as base64; at the replica
it is uploaded to ComfyUI's input directory (the gate lets /upload/image
through), the workflow is filled in with its name, the seed and the
resolution, and the mesh the workflow saves comes back with its type,
which is the file the job keeps. comfy.Client reads every list of files an
output names, not "images" only, and types a file by its extension where
ComfyUI sends octet-stream.

A model's license (name, url, notice) is said in /v1/models and in the
job, and written into a GLB's asset.extras, since the file travels without
the API; a mesh it cannot be written into is refused. per_mesh in
auth.pricing, usage.meshes in the rows, the sums and the export
(migration 0045). A picture asked of a mesh model, or a mesh of a picture
model, is refused saying which it makes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 18:30:52 +02:00
Ilker ManapandClaude Opus 5.5 63198f2753 A profile may load weights of its own
The catalog had one source a model, so an embedding model served by vLLM
from its repository on a card and by llama.cpp from a GGUF on CPUs had its
entry swapped by hand halfway through the metallurgy import. A profile's
source is what its engine loads and placement looks for; the model's is
the default, and what updates and canaries move. ctl model add checks
every source, the GGUF check reads each llama.cpp profile's, and ctl
weights collects none a profile loads. An empty source leaves a profile's
fingerprint as it was, so running replicas do not read as stale.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 17:15:38 +02:00
Ilker ManapandClaude Opus 5.5 eaea671f23 The usage sums and exports count pictures
The five-minute, hourly and daily sums gain images (migration 0042), and
the CSV and JSON exports a column for them: once request rows were past
their retention, an export said what pictures cost but not how many there
were. The pictures made before are added to the sums from their rows.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 10:36:19 +02:00
Ilker ManapandClaude Opus 5.5 4a4fe8fa16 Pictures are priced by the picture
auth.pricing takes per_image, charged on top of a prompt's tokens; a model
priced only per picture is not free, so a spent budget still stops it.
Each usage row keeps how many pictures were made (migration 0041), and
ctl request and the admin API's request view show them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 21:48:47 +02:00
Ilker ManapandClaude Opus 5.5 6b7750a0dd An embedding model's query_prefix, put before questions and never before documents
Qwen3-Embedding and models like it are trained with an instruction on the
query side only. The prefix goes before the question a knowledge-base
search embeds, at answer time and in the admin API's search; documents
reach /v1/embeddings from llmgw-kb as they are.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 18:03:00 +02:00
Ilker ManapandClaude Opus 5.5 5be3da505b Keys' project paths written as the API takes them, Acme/ml/search
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-09 06:46:12 +02:00
Ilker ManapandClaude Opus 5.5 90e0942481 Organizations, teams, projects, keys and memberships in the admin API
What llmgw org, keys and users did against the store on a control plane
can now be done remotely and by automation: the tree read, added to,
renamed, moved and emptied; keys issued (the key in the answer, once),
listed and revoked, with the scopes' lifetime limits; people put in and
taken out of the tree, their keys in projects they no longer reach
revoked with it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 22:42:43 +02:00
Ilker ManapandClaude Opus 5.5 85025dfb03 Trained adapters published, private to their owners like the knowledge bases they come from
An adapter now has an owner (org, team, project, user or key): only callers
there see it, and to anyone else it is a model that does not exist, absent
from /v1/models too. A training job's owner defaults to the one collection
its pairs come from, must be able to read every collection they come from,
and pairs from a knowledge base are never made public. ctl train publish
keeps the GGUF or PEFT files on the mirror, which accepts uploads now, and
sets the owned adapter after the usual architecture check.

The knowledge bases' cache_salt follows the collections searched rather
than the organization, so two teams of one organization no longer share
cached passages.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 21:30:50 +02:00
Ilker ManapandClaude Opus 5.5 79b3e9f887 LoRA adapters trained on the nodes: llmgw ctl train
A job names the catalog model the adapter is for, the safetensors weights to
train on, the node and the MiB of a card it needs, and question/answer pairs.
The control plane sends it when the node's agent is there and nothing else
trains on it; the agent trains with its installed vLLM's torch and a pinned
overlay of peft and accelerate (no second environment), as the engines'
user, on the card with that much free — beside an engine, if it leaves that
much — converts the adapter to GGUF with the installed llama.cpp's own
conversion code when the model is served by llama.cpp, and sends what it
made back in pieces, kept in the store to download. Progress, loss and peak
card memory are reported as it goes; a node that goes fails its job.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 20:25:57 +02:00
Ilker ManapandClaude Opus 5.5 ae95ac8261 Training data from a collection: llmgw ctl kb dataset
A chat model of the cluster reads each chunk and writes questions a person
might ask of it with the answers it holds, streamed as NDJSON a chunk at a
time, for people to correct before an adapter is trained on it. A chunk that
gave no pairs says so on its own line.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 20:04:37 +02:00
Ilker ManapandClaude Opus 5.5 0093cc3d45 Collections a part of the tree answers from when a request names none
On an organization, team, project or key, the narrowest winning: llmgw ctl
kb default, POST /api/v1/kb/defaults, and the console's Knowledge screen. A
request's own X-Llmgw-Knowledge overrides it and none turns it off. A default
never refuses a request: a collection the caller cannot read is left out, and
a failed search answers without passages. Removing a collection takes it out
of every default.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 15:52:43 +02:00
Ilker ManapandClaude Opus 5.5 bc5ee9f3aa Knowledge search ranks by how near a chunk's meaning is, not by its place in two lists
Fused by rank, a full-text match on a question's everyday words outvoted a
far nearer meaning: on the PoC, "how much leave do fathers get" found the
annual-leave page before the paternity one (cosine distance 0.44 against
0.25). Now a chunk scores its similarity to the question plus up to 0.3 for
a full-text match, each search brings four times k candidates, the text
search drops Turkish and English stop words, and an operator's search says
when the embedding model did not answer rather than quietly searching text.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 15:05:18 +02:00
Ilker ManapandClaude Opus 5.5 6466f37d50 Knowledge bases administered: llmgw ctl kb and /api/v1/kb
Collections made for an organization, team, project or person, their
dimension asked of the embedding model; granted and revoked by path, never
outside their organization; their documents listed and removed; and a search
an operator can try. In the OpenAPI document.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 13:49:44 +02:00
Ilker ManapandClaude Opus 5.5 7f7e5a470e guard judge says controversial, and names a judge no policy asks as one
judge ask showed Qwen3Guard's controversial verdict as "safe (Violent)"; it
now says controversial, which a policy's judge.controversial counts. The
stray-judge note called a judge no policy asks any more "served as a model
clients use"; a model with role: guard is now said to be a judge no policy
asks.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 11:42:55 +02:00
Ilker ManapandClaude Opus 5.5 b0273ffb5f Qwen3Guard is the judge, and judges on a GPU and on CPUs are asked in turn
Llama Guard 3 1B could not tell ordinary coding requests from harmful ones:
on the PoC it scored "continue" 0.51, "continue the code above" 0.89 and
"how do I kill all child processes in bash" 0.92 unsafe, against 0.97 to
1.00 for harmful requests, at Q4_K_M and Q8_0 alike, with its own template.
A site had "continue" refused. Qwen3Guard-Gen 0.6B gave the ordinary ones
0.15 at the most and the harmful ones 1.00, in 0.8 GB on CPUs.

- format qwen3guard: its prompt written out as its template writes it, for
  requests and answers, and its Safe / Unsafe / Controversial verdict read;
  controversial counts as the policy's judge.controversial says, safe by
  default.
- judge.models: judges asked in turn, the first that answers deciding; a
  failure is recorded against the judge that failed. guard judge use sets
  the order, replacing categories none of the judges names.
- known judges (Qwen3Guard 0.6B/4B/8B, Llama Guard 1B/8B): guard judge
  enable serves any of them, --gpu the 4B on the roomiest GPU class as a
  standard model; the leader fetches and serves one a policy names that the
  catalog lacks.
- server init installs qwen3guard-0.6b with the baseline's categories.
- a judge reads a long text by its start and end, which fit its context,
  rather than overflowing it and refusing the request under fail: closed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 10:22:47 +02:00
Ilker ManapandClaude Opus 5.5 7245dbe1dd llmgw stop, start and restart act on the whole system
Issue #3 asked for commands to stop and start the system, and 05aa21c gave
only one machine's services. Now:

- llmgw stop sets serving.stopped in the cluster's settings, live: the
  reconciler wants every model at no replicas, freeing every GPU, and the
  gateway answers every POST under /v1/ with 503 system_stopped. The control
  planes, agents and mirror keep running, and no model's definition changes.
- llmgw start clears it; every model is placed again as it asks.
- llmgw restart has every agent restart over its stream (RestartAgent), then
  the other control planes over ssh, standbys before a leader, then this
  one; engines keep serving.
- llmgw status is the cluster's summary, saying when it is stopped.

One machine's services moved to llmgw service status|start|stop|restart.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 18:01:45 +02:00
Ilker ManapandClaude Opus 5.5 f2b2e24e1f A node's own readiness and its multi-node readiness are kept apart
doctor sums findings up as two readinesses - the node's, which decides
whether it joins, and multi-node, its fabrics for replicas spanning nodes -
prints fabric findings in a section of their own, and gives both in --json.
node add says the multi-node one. NodeView.multi_node says whether a node's
declared fabrics are up, naming adapters that are down, and ctl nodes shows
it beside the fabric.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 13:11:16 +02:00
Ilker ManapandClaude Opus 5.5 5ae870e1b2 ctl restart --max-surge and --max-unavailable
A model's replicas restarted one at a time, always. RestartPace says how
many restart at once: MaxSurge beside themselves (replacement first),
MaxUnavailable stopped first (no room needed, out while loading); the
reconciler begins that many per model per pass. The admin API takes
max_surge and max_unavailable on POST /api/v1/restart.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 13:09:10 +02:00
Ilker ManapandClaude Opus 5.5 619dd751ab Nodes show their engines' versions and GPUs' capability; a profile can ask for an engine version
NodeView carries engine_versions and compute_capabilities; ctl nodes shows
llamacpp b10991 and 6000-24g (7.5). A profile's engine_version restricts it
to nodes whose engine version starts with it, and explain-placement says
which version a node has, or which engines it has when it lacks the one
asked for.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 13:06:57 +02:00
Ilker ManapandClaude Opus 5.5 0bd0d881c5 A replica says what its engine computes on, and llama.cpp's word on the CPU refuses a GPU placement
llama.cpp's start-up output is read as a shard becomes ready: the devices it
used, its backend, and the layers it offloaded. That goes up with READY in a
new ShardStatus.compute, is kept on the shard, and is shown by the admin
API, ctl replicas (ON) and the console (COMPUTE). A shard given GPUs whose
engine says it found no usable GPU or offloaded no layer is stopped and
failed, even where nvidia-smi cannot be asked.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 13:02:17 +02:00
Ilker ManapandClaude Opus 5.5 249a91aeec Guardrails turn off and on; a judge's refusal says which hazard; ctl guard judge ask
guard.enabled (unset is on) makes every policy check nothing while false,
live: llmgw ctl guard off/on set it in the cluster's settings, and ctl
status, ctl guard list and the guardrails page say when they are off.

A refusal by a judge names the hazard it saw - judge: S6,Specialized
Advice - and the verdict is kept with the request in a new usage column,
guard_rule, shown by ctl request, the request log and the guardrails page.

llmgw ctl guard judge ask has a judge read a text, past the cache, and shows
its raw answer and how it is read; the endpoint is kept out of the audit so
no text is recorded.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 12:41:08 +02:00
Ilker ManapandClaude Opus 5.5 a8bb417618 The guardrails' judge has its own command, and its trouble is shown
llmgw ctl guard judge status lists each judge model the policies ask:
in the catalog, as a judge, replicas ready, its last failure to answer, and
what is wrong in words. enable serves the missing one through model serve -
Llama Guard 3 1B on CPUs, role guard, under the name the policies give it -
with --replace for a model of that name that is not a judge; the console has
it under Do. GET /api/v1/guard/judges, and the summary's judges, put the
problem on ctl status and the console's Overview.

A Llama Guard file served from the mirror page is added as a judge, under
the name the policies ask, rather than as a model clients use the policies
never find. ctl says plainly when the API answers with a page.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 10:47:37 +02:00
Ilker ManapandClaude Opus 5.5 6289b01ed6 The cluster's settings live in the store, and every change takes effect
Everything in server.yaml that is not one machine's own is now a versioned
document in the store (cluster_settings), the same on every control plane.
The first start against a store imports the file's cluster keys, with the
baseline guard policy when there is none; after that those keys in a file
are ignored, with a warning.

Each control plane polls the document's version and its own server.yaml's
hash. A change is validated and applied: guard live (Gate.Replace), anything
else by a graceful restart that systemd brings back, standbys first and the
leader 30s later. A version that does not load on a machine is not applied;
at start the newest one that loads is used, so a bad edit cannot keep every
control plane down.

llmgw ctl config show/get/set/unset/edit/history/rollback, the admin API
under /api/v1/settings, and the console's Settings screen, which layers the
store's document over the machine's file, write it. Commands beside the
control plane that read cluster keys (init, node add, server add, backup,
upgrade) read the layered configuration; init --bundle writes offline to
the store once it holds the settings.

A standby that becomes the leader re-reads models, engine choices and
facility holds from the store.

Retires appendGuardIfMissing: the import does it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 15:25:23 +02:00
Ilker ManapandClaude Opus 5.5 4ff62c5dd2 ctl upgrade asks the server what it runs, and waits for a node as long as it does
A binary installed by hand made ctl upgrade say this machine already ran
the release and leave the old server running. And a node had 15 seconds
to answer an upgrade the control plane gives five minutes, so a node that
was upgrading stopped the run.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 12:43:25 +02:00
Ilker ManapandClaude Opus 5.5 c5b34978f5 Guardrails in the console
The console never mentioned them. A Guardrails view shows each policy with
its mode, fail behaviour and checks, whether its judge models have replicas
ready, where policies are attached (attach and detach from it), and what
they did in a window. The admin API gains GET /api/v1/guard/events, and GET
/api/v1/guard says what each policy does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 03:36:17 +02:00
Ilker ManapandClaude Opus 5.5 1d1409faad A standby's admin API is the leader's
The console and ctl on a standby showed its own empty view: nodes and
replicas live in the leader's memory. A standby now passes /api/ to the
leader over the agent port, with the cluster's certificates both ways; the
leader answers only a certificate naming a control plane the store lists,
and only while it leads. Control plane certificates are issued for client
use as well, and server add reissues older ones.

An agent's reconnect backoff starts again after a session that worked, so
each failover it lives through no longer makes the next one slower.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 02:27:11 +02:00
Ilker ManapandClaude Opus 5.5 ec1918b23b Two control planes: a failover in seconds, and the console to set one up
A control plane being stopped held every agent's stream for the thirty
seconds of a graceful shutdown, so the standby led within four seconds and
had no nodes for thirty. The agent server is now closed, not drained. A
control plane that loses its lease while it lives on lets its agents go,
rather than keeping nodes the new leader cannot reach.

ctl status and the console's Overview say which control plane leads, which
stand by, and whether the floating address is on the leader. The console
adds a control plane (Do → Add a control plane), asking for the machine and
the floating address. server add says when the mirror would refuse the new
control plane.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 02:17:51 +02:00
Ilker ManapandClaude Opus 5.5 33334c8f33 Merge C24 federation: another llmgw cluster as a provider
Conflicts with facility-aware placement (config, control, admin API routes,
store test tables) all keep both; the generated reference and OpenAPI
document regenerated.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:29:35 +02:00
Ilker ManapandClaude Opus 5.5 40e9fd2ab1 Federation: another llmgw cluster as a provider, its models as PEER/MODEL
ctl federation add/list/remove/sync keeps peers in the store (migration
0029); their /v1/models are offered here as external models named
PEER/MODEL and kept in step, a model still named by a fallback kept
withdrawn. Peers are checked at /ready, timed, and taken out of routing
while down. Requests to a peer carry x-llmgw-hop and are refused (508) when
they come back to a cluster they passed through; residency and no_store
keep requests off peers; X-Request-Id and, with --send-tags, attribution
tags go along. ctl model fallbacks sets overflow to a peer. Console view,
metrics, admin API routes and docs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:21:30 +02:00
Ilker ManapandClaude Opus 5.5 3ec90ded72 Merge C24 facility-aware placement
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:18:20 +02:00
Ilker ManapandClaude Opus 5.5 0d3eb82b9b Facility-aware placement: racks, PDUs, holds and power caps
Nodes say where they are with llmgw.io/site, room, row, rack and pdu labels,
from agent.yaml or `llmgw ctl label` (kept in the store, over the agent's).
Placement spreads a model's replicas across racks and PDUs, keeps a
multi-node replica within one domain when its profile says keep_within,
places nothing in a domain held with `llmgw ctl facility hold` (optionally
draining it), and refuses a shard that would take a domain past its power
cap from facility.budgets, counting declared, derived and measured draw.
Explain-placement names the domain, and keeps the rejections of a placed
replica; `ctl nodes` gains LOCATION and --domain; `ctl facility` lists the
domains; the console's Nodes view shows and sets location and holds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:17:49 +02:00
Ilker ManapandClaude Opus 5.5 2a1e0e6c4c LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters
A vLLM model can carry PEFT adapters, registered at start, or with
max_loaded_adapters held at most N per replica: the router loads an adapter
through vLLM's runtime LoRA API when a request first wants it, sends the
adapter's requests to a replica that holds it while that one is not
saturated, loads each cold adapter once, and unloads the least recently used
idle one to make room. llama.cpp keeps loading every adapter at start.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:14:38 +02:00
Ilker ManapandClaude Opus 5.5 d41104a452 Merge C21 quality suites and gating
Conflicts: docs.go (manual, OpenAPI and suites all kept), modelserve Job
(trace and eval both kept). The eval routes are added to the OpenAPI route
table, update and promote documented with their optional body, and the
document regenerated.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:08:33 +02:00
Ilker ManapandClaude Opus 5.5 9dc82f73d4 Merge C19 OpenAPI for the admin API and a Terraform provider
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:06:34 +02:00
Ilker ManapandClaude Opus 5.5 f0477f140d Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate
A suite is a set of cases in YAML or JSON lines, each scored by exact,
contains, regex, a number within a tolerance, a JSON Schema subset, or a
judge model with a rubric; weighed into one score against a threshold.
internal/bench/eval runs one on the control plane through Router.Replay, as
the gateway's own requests, and keeps every run and its cases in the store
(migration 0028: eval_runs, eval_cases).

- llmgw ctl eval MODEL --suite FILE|NAME, --history, --run, --suites; exits
  non-zero below the score. Suites by name from control.eval_dir, or the
  two built in from deploy/eval (arithmetic, json-format).
- model update and promote take --eval SUITE --min-score N, or a model's
  standing gate (ctl model gate, eval: in the catalog): a promote asks the
  canary first and leaves it below the score; an update tries the new
  weights as a canary given no share, then promotes or removes it.
- The console's model menu: ask a suite with its progress, its runs, its
  gate.
- The audit middleware no longer hands a handler a body cut off at 64 KiB.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:04:43 +02:00
Ilker ManapandClaude Opus 5.5 330a421118 Merge C22 MIG
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:01:51 +02:00
Ilker ManapandClaude Opus 5.5 0fd55a862b MIG: a card in MIG mode is placed as its instances, each a GPU of class <card>-mig-<profile>
Discovery reports a card in MIG mode as its instances instead of the whole
card, on both paths: nvidia-smi (-L names them, -q -x gives their memory,
ids and processes) and NVML (GetMigMode, GetMaxMigDeviceCount,
GetMigDeviceHandleByIndex, behind an interface so it is tested with a fake).
Each instance has its MIG UUID, its own memory, and a class from the card's
and the profile: a100-80g-mig-3g.40gb. Placement numbers instances
1000 + 100 x card + instance and every view shows them as card/migN.

A replica on MIG runs on exactly one instance: a process opens one MIG
device and instances have no NVLink between them, so tp, pp, gpus or nodes
above one on a MIG class are refused in catalog validation and again in
placement, and the planner offers one instance or none. The agent gives
the engine CUDA_VISIBLE_DEVICES=MIG-<uuid> (no Vulkan device variable on
MIG). Memory and processes are per instance; heat, power and wear the
card's; utilisation is left out rather than reported as zero. A card in MIG
mode with nothing carved stays out of placement with the reason, and
llmgw doctor reports each card's mode and instances and warns of the states
in which nothing can run. ctl nodes, ctl status, the console and the plan
count devices by class. docs/install.md has a MIG section.

Written against fakes of nvidia-smi and NVML output; not yet run on MIG
hardware.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:01:14 +02:00
Ilker ManapandClaude Opus 5.5 44f3a9e31d Reusing answers: a response cache per project, exact match, opt-in with a TTL
llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on
the console's project menu. The same request — project, endpoint, canonical
model and the body without stream, stream_options, user, metadata, store and
service_tier, keys sorted — is answered from the gateway's memory for the
TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank,
score) unless --any; only a whole 200 from the model asked for, not a
fallback's, a canary's, or one a guardrail changed.

Looked up after the limits and the input guardrails; the output policy
judges a reused answer; no_store and stored conversations bypass it.
Streamed answers are kept put back together and streamed back as chunks.
x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or
x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost,
marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded
by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total,
llmgw_cache_bytes, llmgw_cache_entries.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 23:50:53 +02:00
Ilker ManapandClaude Opus 5.5 a95c519bc5 The admin API described in OpenAPI 3.1: generated from a route table by docgen, served at /api/v1/openapi.json
adminapi.Routes lists every endpoint with its query parameters and Go values
of the bodies it reads and answers with; internal/docgen/openapi turns that
into docs/admin-api.openapi.json by reflection, the way encoding/json encodes
the types, with the fields' comments as descriptions. A test fails when an
endpoint is registered and not in the table, another when the committed
document is stale. The binary embeds it and serves it with the token.

Handlers that wrote map literals and decoded anonymous structs now use named
types, so the document can see them. GET /api/v1/models/{name} answers with a
model's whole definition as it is in force, which a tool managing it needs.
docs/admin-api.md: reaching the API, the token, what a request must carry.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 23:47:59 +02:00
Ilker ManapandClaude Opus 5.5 32ff1dbd6f Requests kept and replayed: keys requests per project with retention and a share, ctl request --body and --replay
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 22:51:21 +02:00
Ilker ManapandClaude Opus 5.5 a1dfd4df1d Canary of a model's newer weights: ctl model canary, promote and canaries, rolled back by itself when it serves worse
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 20:31:26 +02:00