Commit Graph
613 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 ed0d1ec0ad A cached prompt costs the cached rate
CI / test (push) Successful in 8m45s
cached_per_1k in a price charges the prompt tokens the answer counts as
cached_tokens at their own rate; zero keeps charging them as prompt tokens.
The count reaches the cost and not a column: CostMicros holds what it
changed. Cached beyond the prompt is clamped, an engine's miscount rather
than a refund. BACKLOG L5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 17:15:48 +01:00
Ilker ManapandClaude Opus 5.5 73e1bb6733 Chat goes to the replica that has been answering soonest
CI / test (push) Failing after 7m18s
The router keeps each endpoint's time to first token, smoothed over its last
streamed answers, and the chat profile scores by it beside session, queue,
KV and inflight: the fastest endpoint 1, one twice as slow 0.5, one not yet
measured 1 so a new replica is tried. Load still sees a slow drain from the
engine's own counters; this sees what answers actually took. BACKLOG L4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 17:01:30 +01:00
Ilker ManapandClaude Opus 5.5 9e7de57816 The reference describes a guardrail policy's services
CI / test (push) Successful in 11m42s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 16:50:58 +01:00
Ilker ManapandClaude Opus 5.5 783dc9e9f6 A guardrail policy can ask someone else's service
CI / test (push) Failing after 7m39s
A site may be required to use a particular guardrail: Azure Content Safety,
Lakera, NeMo, a classifier of its own. A policy's services asks one over
HTTP — a body template given the text, header values from files read each
call, dotted paths to where the answer says unsafe and why — on input,
output or both, to block or flag. On input it reads the latest user turn
before the judge; on output the whole answer once at its end. A service
that does not answer is what the policy's fail says. It is told the text
only, never who sent it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 16:49:56 +01:00
Ilker ManapandClaude Opus 5.5 81685ecd75 A request's redacted values are put back in its answer
CI / test (push) Successful in 8m49s
With restore: true on a guardrail policy's input, what redact rewrote as
placeholders comes back in the answer: the model reads "[EMAIL_1]", the
client reads the address it sent. A streamed placeholder split across
events is held until it is whole, and what is held is sent at the end.
The swap runs before the output filter, so the output rules judge the
answer as the client reads it and a placeholder they make of a value the
model invented is never taken for the request's own. The map lives in the
Verdict for the request only; nothing stores, logs or traces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 16:41:53 +01:00
Ilker ManapandClaude Opus 5.5 d110968243 A machine too small for the models it would serve is said so first
CI / test (push) Successful in 8m42s
A control plane that is also a node, serving the starter model and two
judges, needs about 10.5 GB: 3.5 for itself, 2 for the starter, 2 for each
judge, and one more because the scheduler counts whole gigabytes. On 8 GB
the install did every step and then had nowhere to run the starter. The
setup's checks now fail this machine's memory and name what it would serve,
and server init refuses before it installs anything, naming the flags that
leave those models off it. quick-install.md said 8 GB; it says 12.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 16:19:00 +01:00
Ilker ManapandClaude Opus 5.5 06baa38858 Traces say model calls in the OpenTelemetry GenAI conventions
CI / test (push) Successful in 8m55s
An inference request and each attempt at an engine carry gen_ai.*: the
operation, the requested and answering model, max_tokens, temperature and
top_p when the client set them, the session as the conversation id, the
input, output and cached tokens, and on an attempt the engine that answered.
A customer's Grafana, Jaeger or Langfuse then shows llmgw's calls as model
calls. No prompt or answer text is ever put in a span, and a test says so.

BACKLOG gains section L: what is worth taking from Kong's AI Gateway, of
which this is the first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 12:59:20 +01:00
Ilker ManapandClaude Opus 5.5 38dfe268c9 VS Code gets the gateway's models from an extension the gateway serves
CI / test (push) Successful in 8m43s
llmgw's own extension for VS Code, plain JavaScript with no dependencies,
packed into a .vsix by the binary at /setup/vscode/llmgw.vsix with the
serving gateway's address in it, so it needs no build, no Node and no
marketplace. It registers llmgw as a language-model provider: the chat's
model picker, agent mode and any extension that asks VS Code for a model
get every chat model the key may use, sized and with tool and vision
capabilities as the catalog says, streamed, tool calls both ways.

VS Code is a tool on the account page's Connect your editor: a key, the
install command, and an Open in VS Code link that hands the key to the
extension's secret storage. The extension's Sign In opens the page with a
state of its own; a link without it is confirmed, the gateway named, since
it decides where code is sent.

Tested in VS Code 1.138 against the PoC: 27 models listed, an answer from
qwen2.5-0.5b-instruct, and qwen3-coder-30b calling a tool and using its
result (internal/vscodeext/e2e).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 01:39:50 +02:00
Ilker ManapandClaude Opus 5.5 fea5774fe5 The install guide shows every screen of a real install
/install takes a reader from three bare Ubuntu machines to a first answer:
what each machine needs and the commands that check it, the release fetched
and its checksum checked, the eight setup screens, the system's state and
the console, and a first request with a key from the account page. Every
screen is from an install of v0.14.9 on three fresh VMs, recorded with tmux
and kept as terminal frames: the page shows them as text, and plays a
session back when asked, fetching it compressed from /install/cast/. The
administrator's password and the API key were masked before the recording
was kept, and a test fails if either shape reaches the page.

/about gains what llmgw now does beyond routing, and its install steps are
today's: the setup screens and node add, not onboard-node.sh and a model
file, with a link into the guide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 01:35:08 +02:00
Ilker ManapandClaude Opus 5.5 f8d18efaef Turkish pages are Turkish to the end
CI / test (push) Successful in 5m35s
The errors other packages give a page (store, catalog, mirror) are
shown in its language; a percentage is written as the language writes
it (%93 in Turkish); the footer's time in its date form; and a stale
form is refused with a sentence rather than "bad request".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-21 09:22:42 +02:00
Ilker ManapandClaude Opus 5.5 3a4488b41e The company's MCP tools under the guardrails, a rate and a breaker
CI / test (push) Successful in 5m35s
A tool's answer is read by the caller's policies as the prompt it
becomes, so an instruction planted in a ticket is withheld and personal
data redacted; its arguments are read before they leave. ctl mcp add
--rate bounds each caller's calls a minute to a server (migration 0050),
and five failures in a row take a server out for a minute, its tools
off the list.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 23:00:22 +02:00
Ilker ManapandClaude Opus 5.5 6bef138c23 The judge check gives each judgment a text of its own
CI / test (push) Successful in 5m32s
The same text again was answered from what the engine kept of the last
reading, in a tenth of the time: the check read 4.9 s, then 0.5 and 0.5.
Each judgment now begins with a line of its own.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 22:45:58 +02:00
Ilker ManapandClaude Opus 5.5 85f135311e How fast the judge is, measured, and said at install
CI / test (push) Successful in 5m29s
ctl guard judge check, and POST /api/v1/guard/judges/check, time three
judgments of a 280-token prompt, then a burst at once, against the
policies' wait: ok, tight or slow, about how many requests a second can
be judged, and what to do. server init runs it once the judge is up and
says a machine too slow before the next steps; the wait is not changed
without asking. BACKLOG reordered by what goes wrong where installed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 22:35:19 +02:00
Ilker ManapandClaude Opus 5.5 d1bf26528b keys limits changes only the limits it is given
CI / test (push) Successful in 5m25s
Setting one limit at a scope wrote the whole row, so a tool grant given
alone cleared the scope's rate, budget and models. --replace writes it
whole, as the console's form, which shows every field, now asks for.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 17:32:43 +02:00
Ilker ManapandClaude Opus 5.5 4eb49d0e4c The cluster as a read-only MCP server, llmgw:admin
CI / test (push) Successful in 5m26s
Its tools say what llmgw ctl does: cluster_status, list_nodes,
list_models, list_replicas, explain_placement, node_logs; none changes
anything. The admin API serves it at POST /api/v1/mcp, and the gateway
answers a server registered as llmgw:admin in its own process, so no
token is kept and each control plane serves it. ctl mcp add takes
--bearer-file for a token sent as Authorization: Bearer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 17:18:28 +02:00
Ilker ManapandClaude Opus 5.5 4453f469e1 Company MCP servers are registered with ctl mcp, and tools granted with keys limits --tools
CI / test (push) Failing after 5m4s
llmgw ctl mcp add|list|remove|test and /api/v1/mcp/servers keep the
servers; a token given with --secret-file is kept in the store, sealed,
and named by reference. keys limits --tools, and the console's limits
form, grant tools as patterns. Documented in install.md and clients.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 12:50:55 +02:00
Ilker ManapandClaude Opus 5.5 ca77637a71 /mcp serves the company's own MCP servers' tools to keys granted them
Each registered server's tools are listed as server.tool beside llmgw's
own, to a key whose tool grants match, and called with the server's
credentials, read from their secret references at each call; the
server is told the key, person and project calling. A call is bounded
in time and size, its progress passed on to a streaming client, and
recorded in the audit log with its arguments' size and digest, never
their text. A key granted nothing sees none of them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 10:53:24 +02:00
Ilker ManapandClaude Opus 5.5 765b6f5603 An MCP client for the company's tool servers
internal/mcpclient speaks streamable HTTP: initializes once and keeps
the session the server gives, beginning it again once if the server
forgot it; lists tools page by page; calls one with headers of its own,
taking the answer as JSON or as events with progress before it; and
bounds what it reads. mcptest is a server for tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 10:43:35 +02:00
Ilker ManapandClaude Opus 5.5 6dfdfbf2fe The store keeps the company's MCP servers, and which tools a scope may call
mcp_servers holds each server: its address, headers, and secret
references for the headers that carry credentials. limits gains
allowed_tools, patterns such as jira.* resolved down the tenancy tree;
unlike models, a tool no level grants is not granted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 10:36:08 +02:00
Ilker ManapandClaude Opus 5.5 8ffde2b80c A big model's download no longer runs out its startup timeout
CI / test (push) Successful in 5m18s
The startup timeout runs from when a node's download last moved, so a
model of hundreds of gigabytes on a slow mirror is not failed while it
arrives. ctl model serve --env sets engine variables (NCCL_DEBUG=INFO)
on every node, and a layout given by hand keeps expert parallelism for a
model of experts. docs/large-models.md: how a model is split, what each
way needs of a card, and a checklist for bringing one up on a fabric.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 09:45:31 +02:00
Ilker ManapandClaude Opus 5.5 4fe9b46451 A plan for a big model: experts spread, any number of nodes, memory said
A mixture-of-experts model on more than one card is served with vLLM's
--enable-expert-parallel; config.json's experts are read by any of their
names, and a model that also reads pictures is sized by its text_config.
Tensor parallelism past the KV heads is allowed when it is a multiple of
them (Qwen3-235B, four KV heads, runs on eight cards), which the plan
had refused. Several nodes are offered in any number, not only powers of
two. Each way says what it needs of a card, part by part, and of all its
cards; the serve form checks a context longer than planned still fits.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 08:45:33 +02:00
Ilker ManapandClaude Opus 5.5 314cc5e6a3 The judge init installs is no longer replaced by the leader's own
CI / test (push) Successful in 5m27s
Found on a fresh HA install with a judge machine: the leader saw the
policy's judge missing while init was still fetching it, served one
itself, and its entry replaced init's, so a second judge ran off the
judge machine. The keeper now waits ten minutes after it starts, and a
serve job it starts gives way to a model added meanwhile (AddNew).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 20:37:10 +02:00
Ilker ManapandClaude Opus 5.5 34764f5ef0 A node given a role takes only what asks for it
CI / test (push) Successful in 5m24s
The judges' machine was selected by the judge's profile, but any other
model could still be placed beside it. A node labelled llmgw.role now
takes only profiles whose selector asks for that role.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 17:52:30 +02:00
Ilker ManapandClaude Opus 5.5 51ca09287f A three-machine install can give the judges machines of their own
CI / test (push) Successful in 5m10s
The setup screen asks for up to two judge machines after the store's
two, and server init takes --judge-node for each. Each is added as a
node labelled llmgw.role: judge, and the judge's profile selects that
label, with cores by init's rule over those machines. A judge beside a
busy control plane refused 38 of 60 requests under load; alone, 2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 17:45:00 +02:00
Ilker ManapandClaude Opus 5.5 8b122aae09 A judge the leader serves by itself is sized as init's is
CI / test (push) Successful in 5m13s
A judge a policy names and the catalog lacked was served with 3 GB, no
cores and llama.cpp's prompt cache; on a busy node its judgments missed
their budget and fail: closed refused the requests. It now gets init's
2 GB and cores, and every judge modelserve serves runs with
--cache-ram 0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 16:32:04 +02:00
Ilker ManapandClaude Opus 5.5 cfc38d35cc A node's console opens on what the machine is doing
CI / test (push) Successful in 5m12s
sudo llmgw on a node showed only agent.yaml, the manual, wipe and leave.
It now opens on an Overview read on the node itself: the agent's state and
version, the control planes it dials and whether they answer, each GPU with
the models on it, the engines from the supervisor's state files, the free
space for weights and engines, and the agent's warnings of the last hour.
No admin token is needed, every command is bounded by two seconds, and a
missing tool is said in a line.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 12:55:27 +02:00
Ilker ManapandClaude Opus 5.5 ac25a9136a Sign-in is set up from the console, and a directory tried before it is kept
People → Sign-in shows single sign-on, Active Directory and SCIM, and
the directory's last sync. E sets up Entra ID from the tenant and client
IDs; D sets up Active Directory, tried from the control plane first:
each server bound, people who may sign in, members of each mapped group.
The secrets go into the store as store:auth/..., sealed by the master
key, and the CA into the settings, so no control plane needs a file.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 12:55:27 +02:00
Ilker ManapandClaude Opus 5.5 4b6b3eeddb A node's page says which llmgw it runs and which model is on each GPU
CI / test (push) Successful in 4m13s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 11:31:52 +02:00
Ilker ManapandClaude Opus 5.5 178b168bfd A broken directory setting, or a second server, no longer takes the cluster down
CI / test (push) Successful in 4m15s
Found trying auth.ldap against a Samba DC on the PoC. A CA file a
control plane could not use made every control plane exit, and no
ctl config could then put the setting right: directory sign-in is now
turned off there, and said. `llmgw server check` ran a whole server
that, its ports taken, held the leader's lease for three hours: server
refuses words it does not know and takes its ports before anything else.
The guides now say auth is set with ctl config, not server.yaml.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 11:10:43 +02:00
Ilker ManapandClaude Opus 5.5 f27d722111 People sign in with their Active Directory account
auth.ldap checks a username and password against the domain controllers
over LDAPS or StartTLS: found with a service account, checked by binding
as the person. Groups, nested ones included, give staff roles and places
in the tree. Accounts are tied to the objectGUID; the leader reads the
directory every 15 minutes, suspends whom it disabled or removed and
reinstates them after, and refuses a sync that would suspend most.

Entra ID sends no groups for someone in more than 200, only where to
fetch them; that was read as none and took an administrator's role away.
It is now left as it was, and logged. An account made by SCIM or OIDC
can be reinstated again.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-18 10:59:50 +02:00
Ilker ManapandClaude Opus 5.5 fe310576e7 Several keys for one outside provider
CI / test (push) Successful in 8m52s
An external model takes api_key_files, each a file and an optional weight,
in place of api_key_file. Each key is an endpoint of its own
(external:MODEL#1, #2), so the router's scoring, retries and breakers work
per key. A key answering 429 is passed over while its breaker is open. A
key answering 401 is passed over too, and external_key_refused is sent,
naming the endpoint and never the key; with a single key the 401 is still
the client's. A key whose file cannot be read is left out, and usage
records which key answered by its endpoint.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-17 08:56:30 +02:00
Ilker ManapandClaude Opus 5.5 688b205604 What llmgw adds to a request, measured: ctl bench overhead
CI / test (push) Successful in 8m56s
An engine that answers at once and the gateway in front of it, in one
process on loopback, given the same load straight and through the gateway:
bare, with authentication and accounting, and with the guardrails' pattern
checks. It reports p50 and p99, what the gateway added, and a rough CPU
per thousand requests.

On an Apple M5 at 100 to 2,000 requests a second (10 s a step, 72,000
requests, none failed), the gateway added 0.05–0.42 ms at p50 and at most
0.49 ms at p99. That is now in the README, with how it was measured and
why the CPU figure is rough. CI runs a one-second version.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-17 00:23:26 +02:00
Ilker ManapandClaude Opus 5.5 35ccabac28 modelserve: a job's request is set under the lock Get reads it with
CI / test (push) Successful in 8m49s
CI run 394, on a commit that changed only BACKLOG.md, failed with a data
race in TestAVisionModelIsServedWithItsProjectorBesideIt. The serve job's
goroutine wrote j.Request without the lock (since efffdb6, which sized
embedding slots), while Get's snapshot read the job under it. Every other
write to a running job holds the lock. It did not reproduce in 30 runs
here; the race detector's trace names this write and that read.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 22:40:09 +02:00
Ilker ManapandClaude Opus 5.5 2545fca174 An upgrade refits the judge an earlier server init installed
CI / test (push) Successful in 8m53s
A v0.14.6 install on fewer than eight cores refused every request until
its judge was given cores by hand, and upgrading did not change that. The
judge keeper, on the leader, now finds a judge whose entry is exactly what
init before v0.14.7 wrote: one llama.cpp CPU profile, an 8192 context and
nothing else, 2 or 4 GB, none to two cores. It gives the judge 2 GB,
--cache-ram 0 and cores by init's rule through modelsvc.SetProfiles, then
restarts its ready replicas. The replica count is left as it is. A judge
whose entry differs is left alone, and the log says so once.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 22:06:56 +02:00
Ilker ManapandClaude Opus 5.5 b8274d6278 A restart is not refused for room another restart is giving back
CI / test (push) Successful in 8m51s
On a 4-core single-machine install, `ctl model resources` was run on the
judge (to two cores) and on the starter (to none), then both restarted.
The judge's restart was refused: its replacement fit on no node even with
its old replica stopped, because the starter still held its two cores. A
restart now waits for other restarts under way (queued, retiring or
draining) when their going would make room, up to DrainGrace.

A drained replica was also kept until its model had enough live replicas,
and a replica being restarted that still answers did not count. So with
replicas restarted one after another, each drained one waited out the
five-minute grace, holding its room.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 21:33:20 +02:00
Ilker ManapandClaude Opus 5.5 3e7dd20bad One machine runs one judge, on up to four cores
CI / test (push) Successful in 9m1s
A second judge replica on the same machine shares its cores and is lost
with it, so it bought nothing but half the speed of each. On a 4-core,
8 GB single-machine install, one judge on four cores passed 20 of 20
requests at about 3.5 s each, judged in and out. Two judges on two cores
each had passed 19 of 20.

server init now serves one judge when this machine is the only node, and
two, on two cores each and spread over the nodes, where there will be
others (HA setups, or a control plane that is not a node).
--judge-replicas and the setup's Copies still choose. install.md gives
the measured times in place of "half a second".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 19:35:42 +02:00
Ilker ManapandClaude Opus 5.5 f8206c3540 A judge reserves two cores and 2 GB, with no prompt cache
CI / test (push) Successful in 9m1s
Checked by a single-machine install of v0.14.6 on a 4-core, 8 GB VM. The
second judge fitted nowhere: v0.14.6 reserved 4 GB a judge, a figure from
one replica that grew to 3.2 GB through llama.cpp's prompt cache (up to
8 GB by default), which a judge never reuses. Once it fitted, every
judgment timed out. v0.14.6 gave a judge one core below 8 CPUs, and a
judgment of about 280 tokens takes 8.0 s on one core, 4.2 s on two and
2.3 s on four, against a 5 s budget: 0 of 20 requests went through.

A judge now runs with --cache-ram 0, uses about 1.6 GB and reserves 2,
and reserves two cores (one only below four CPUs). On the same VM, 19 of
20 went through.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 19:22:33 +02:00
Ilker ManapandClaude Opus 5.5 63b2c6d9a4 The judges and the starter divide a machine's CPUs instead of each taking all
CI / test (push) Failing after 30m7s
Measured on the three-VM cluster with ha1's CPUs loaded: 38 of 60 requests
refused with both judges and the starter on ha1, each engine on all four
cores. With the judges moved to a 4-core, 32 GB node at 2 cores each, 2 of
60 were refused, both the first after the judges started.

init now reserves cores by this machine's count: one per judge replica
under 8 CPUs and two from 8, and the rest, up to 8, for the starter on CPU.
The starter reserves nothing when nothing is left, so it still fits. Judges
bound for nodes not yet known get two each. A judge reserves 4 GB; it uses
about 3.2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 13:00:39 +02:00
Ilker ManapandClaude Opus 5.5 0b984a92bc A judge is asked again only on another machine
CI / test (push) Failing after 30m3s
Measured under CPU load on the three-VM cluster, with both judge replicas on
ha1: 37 of 60 requests failed with the hedge, 38 of 60 without it. Two
replicas on one saturated machine are slow together, and the second
question doubled the work there. The hedge now goes only to a replica whose
address names another host.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 10:27:50 +02:00
Ilker ManapandClaude Opus 5.5 bc4c5d5ac9 A judge replica that hangs is overtaken by another at half the budget
On the three-VM cluster about one answer in twenty-five came back as
content_filter: both judge replicas timed out within the same millisecond.
The tries shared the policy's 5 s budget, so the first replica, busy on an
oversubscribed machine, used it all and the retry on the second had nothing
left. A judge request now goes to a second replica too when the first has
not answered by half the budget, and the first answer is taken. The first
is not cut short.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 08:35:28 +02:00
Ilker ManapandClaude Opus 5.5 b09922b29b Leadership stays put while the store's primary is replaced
CI / test (push) Failing after 30m3s
Measured again after the store fixes. Cutting the store primary's power no
longer hung requests, but leadership moved anyway: the lease had lapsed for
everyone while Patroni promoted the standby, and a standby asked first. The
new leader then spent about 25 s without its nodes, and so without its
judges.

A standby that saw the store out of reach now leaves the lease alone for one
renew round after the store returns. The default lease is 10 s, and a node
detects a control plane that died silently in 8 s (5 s idle, 3 s ping)
rather than 15.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 23:17:58 +02:00
Ilker ManapandClaude Opus 5.5 5a4b022dcd A lost store machine is given up on in seconds, and a new leader waits for its judge
CI / test (push) Failing after 30m6s
Measured on three VMs: cutting the store primary's power cost about 45 s of
hanging requests. New connections tried the dead member first with no
connect timeout, and Patroni kept its 30 s leader key. Cutting the leader of
both cost a further 17 s of 503s while the new leader knew no judge.

The store dials each member for 3 s and closes connections to a machine that
is gone within about 10 s (keepalive, TCP_USER_TIMEOUT). Patroni runs with
ttl 20, loop_wait 5 and retry_timeout 5, applied in place to existing
clusters through its API. Judges that all fail at once are asked again
every 250 ms while the policy's budget lasts, and reported once.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 20:33:13 +02:00
Ilker ManapandClaude Opus 5.5 eeaffd764d A floating address keeps to its own network, and Caddy follows the leader
CI / test (push) Failing after 30m3s
Found by a three-machine install on fresh VMs. The second standby's run
asked for the floating address's interface with `ip route get <vip>`.
The machine already held the address, so that answered lo, and keepalived
would not start. The interface now comes from the route to the address's
network. Every cluster's VRRP router id was 51, so the PoC's two control
planes and the new three on one LAN each logged the other's adverts as an
invalid password; the id now comes from the address.

The Caddyfile, deploy/ha and install.md send clients to the control plane
whose /leader is 200, not /ready: /ready is 503 on a leader with no model
yet and would take the portal away with it. Setup no longer hides a key
file's path as if it were a password, and the starter's text no longer
says it runs elsewhere when this machine is a node.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 17:35:10 +02:00
Ilker ManapandClaude Opus 5.5 94401e21e6 Setup lets every part be chosen, each with its settings; three machines each run the gateway
CI / test (push) Failing after 30m4s
The setup screens list the parts of an install: control plane, store, model
mirror, this machine as a node, starter model, pictures and 3D, the judge,
firewall, notifications, limits, on-demand turns and an offline bundle. Each
says what it is or what is missing without it. Space turns a part on or off;
Enter opens its settings in a dialog that explains it. A plan page lists what
is installed and what is left out, and the same install as a command. Every
setting is a server init flag too, written into server.yaml when init writes
it.

With three machines, the two database machines become control planes too,
standing by (server add's steps), with an optional floating address. The
mirror's management API answers all three control planes, so a standby that
comes to lead can still fetch models.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 11:22:36 +02:00
Ilker Manap 733e20924c Merge remote-tracking branch 'origin/mcp-wait' into sd-dedupe
CI / test (push) Failing after 30m3s
2026-10-15 07:53:22 +02:00
Ilker ManapandClaude Opus 5.5 fc60481e40 Every control plane lists the others alike to Prometheus
140 named 150 a store member, 150 named itself a control plane: Prometheus
asking both scraped 150 twice and counted its every series twice, and asked
of 140 alone while 150 led it lost every node. Each control plane now lists
all of them as control planes before the store's members, so two answers
agree and a target is scraped once. The shipped prometheus.yml says to
scrape and ask every control plane.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 07:53:22 +02:00
Ilker ManapandClaude Opus 5.5 edeeb9f8e1 MCP takes pictures of real size, by id too, and waits for a cold model
The game's bridge found three things on the PoC. /mcp held its body to
64 KiB — the edit meant for it had raised the memory notes' limit instead —
so a 1024 px concept sent as base64 failed as "expected one JSON-RPC
request object". The notes are back to 64 KiB, /mcp takes 64 MB, and a
call past it is a 413 saying so.

/v1/files took batch files only, so a picture could not be sent once and
named after. purpose vision and user_data are kept too, as long as job
results, and generate_mesh hands a workflow model the id as it is.

generate_image and review_images failed at once while their model was
cold. They wait out an answer that says when to come back — loading,
another model holding the GPU, a full queue — for as long as the call may,
saying who holds the GPU and about how long; a refusal that says no
Retry-After, a spent budget, is said at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 07:24:59 +02:00
Ilker ManapandClaude Opus 5.5 c22528c26e A model that does not fit its card is not added to fail; weights in use are not removed
CI / test (push) Failing after 30m3s
Apertus 70B with a 65536-token pool, and then 8B with 131072, on a 24 GB
card loaded their weights and failed to make their context on every start,
the 8B taking the card from the 70B each time only to fail. ctl model serve
now shrinks a pool the card cannot hold beside the weights (KV cache from
the GGUF's shape, 90% of the card, a gigabyte for the engine), saying so,
and refuses weights that do not fit at all; the portal's serve form refuses
such a pool and says the largest that fits.

The mirror page removed the 70B's weights while the catalog still had the
model, whose replicas went on loading the node's copy with nothing said.
Removing weights a catalog model loads is refused, naming the model.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 00:22:01 +02:00
Ilker Manap bda988b78f Merge remote-tracking branch 'origin/setup-webhooks' into shared-replica
CI / test (push) Successful in 9m54s
# Conflicts:
#	CHANGELOG.md
2026-10-14 14:26:27 +02:00
Ilker Manap a7cb3aac3c Merge remote-tracking branch 'origin/mcp-media' into shared-replica
# Conflicts:
#	CHANGELOG.md
2026-10-14 14:01:24 +02:00