Commit Graph
608 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 d110968243 A machine too small for the models it would serve is said so first
CI / test (push) Successful in 8m42s
A control plane that is also a node, serving the starter model and two
judges, needs about 10.5 GB: 3.5 for itself, 2 for the starter, 2 for each
judge, and one more because the scheduler counts whole gigabytes. On 8 GB
the install did every step and then had nowhere to run the starter. The
setup's checks now fail this machine's memory and name what it would serve,
and server init refuses before it installs anything, naming the flags that
leave those models off it. quick-install.md said 8 GB; it says 12.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 16:19:00 +01:00
Ilker ManapandClaude Opus 5.5 06baa38858 Traces say model calls in the OpenTelemetry GenAI conventions
CI / test (push) Successful in 8m55s
An inference request and each attempt at an engine carry gen_ai.*: the
operation, the requested and answering model, max_tokens, temperature and
top_p when the client set them, the session as the conversation id, the
input, output and cached tokens, and on an attempt the engine that answered.
A customer's Grafana, Jaeger or Langfuse then shows llmgw's calls as model
calls. No prompt or answer text is ever put in a span, and a test says so.

BACKLOG gains section L: what is worth taking from Kong's AI Gateway, of
which this is the first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 12:59:20 +01:00
Ilker ManapandClaude Opus 5.5 38dfe268c9 VS Code gets the gateway's models from an extension the gateway serves
CI / test (push) Successful in 8m43s
llmgw's own extension for VS Code, plain JavaScript with no dependencies,
packed into a .vsix by the binary at /setup/vscode/llmgw.vsix with the
serving gateway's address in it, so it needs no build, no Node and no
marketplace. It registers llmgw as a language-model provider: the chat's
model picker, agent mode and any extension that asks VS Code for a model
get every chat model the key may use, sized and with tool and vision
capabilities as the catalog says, streamed, tool calls both ways.

VS Code is a tool on the account page's Connect your editor: a key, the
install command, and an Open in VS Code link that hands the key to the
extension's secret storage. The extension's Sign In opens the page with a
state of its own; a link without it is confirmed, the gateway named, since
it decides where code is sent.

Tested in VS Code 1.138 against the PoC: 27 models listed, an answer from
qwen2.5-0.5b-instruct, and qwen3-coder-30b calling a tool and using its
result (internal/vscodeext/e2e).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 01:39:50 +02:00
Ilker ManapandClaude Opus 5.5 fea5774fe5 The install guide shows every screen of a real install
/install takes a reader from three bare Ubuntu machines to a first answer:
what each machine needs and the commands that check it, the release fetched
and its checksum checked, the eight setup screens, the system's state and
the console, and a first request with a key from the account page. Every
screen is from an install of v0.14.9 on three fresh VMs, recorded with tmux
and kept as terminal frames: the page shows them as text, and plays a
session back when asked, fetching it compressed from /install/cast/. The
administrator's password and the API key were masked before the recording
was kept, and a test fails if either shape reaches the page.

/about gains what llmgw now does beyond routing, and its install steps are
today's: the setup screens and node add, not onboard-node.sh and a model
file, with a link into the guide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 01:35:08 +02:00
Ilker ManapandClaude Opus 5.5 f8d18efaef Turkish pages are Turkish to the end
CI / test (push) Successful in 5m35s
The errors other packages give a page (store, catalog, mirror) are
shown in its language; a percentage is written as the language writes
it (%93 in Turkish); the footer's time in its date form; and a stale
form is refused with a sentence rather than "bad request".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-21 09:22:42 +02:00
Ilker ManapandClaude Opus 5.5 3a4488b41e The company's MCP tools under the guardrails, a rate and a breaker
CI / test (push) Successful in 5m35s
A tool's answer is read by the caller's policies as the prompt it
becomes, so an instruction planted in a ticket is withheld and personal
data redacted; its arguments are read before they leave. ctl mcp add
--rate bounds each caller's calls a minute to a server (migration 0050),
and five failures in a row take a server out for a minute, its tools
off the list.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 23:00:22 +02:00
Ilker ManapandClaude Opus 5.5 6bef138c23 The judge check gives each judgment a text of its own
CI / test (push) Successful in 5m32s
The same text again was answered from what the engine kept of the last
reading, in a tenth of the time: the check read 4.9 s, then 0.5 and 0.5.
Each judgment now begins with a line of its own.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 22:45:58 +02:00
Ilker ManapandClaude Opus 5.5 85f135311e How fast the judge is, measured, and said at install
CI / test (push) Successful in 5m29s
ctl guard judge check, and POST /api/v1/guard/judges/check, time three
judgments of a 280-token prompt, then a burst at once, against the
policies' wait: ok, tight or slow, about how many requests a second can
be judged, and what to do. server init runs it once the judge is up and
says a machine too slow before the next steps; the wait is not changed
without asking. BACKLOG reordered by what goes wrong where installed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 22:35:19 +02:00
Ilker ManapandClaude Opus 5.5 d1bf26528b keys limits changes only the limits it is given
CI / test (push) Successful in 5m25s
Setting one limit at a scope wrote the whole row, so a tool grant given
alone cleared the scope's rate, budget and models. --replace writes it
whole, as the console's form, which shows every field, now asks for.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 17:32:43 +02:00
Ilker ManapandClaude Opus 5.5 4eb49d0e4c The cluster as a read-only MCP server, llmgw:admin
CI / test (push) Successful in 5m26s
Its tools say what llmgw ctl does: cluster_status, list_nodes,
list_models, list_replicas, explain_placement, node_logs; none changes
anything. The admin API serves it at POST /api/v1/mcp, and the gateway
answers a server registered as llmgw:admin in its own process, so no
token is kept and each control plane serves it. ctl mcp add takes
--bearer-file for a token sent as Authorization: Bearer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 17:18:28 +02:00
Ilker ManapandClaude Opus 5.5 4453f469e1 Company MCP servers are registered with ctl mcp, and tools granted with keys limits --tools
CI / test (push) Failing after 5m4s
llmgw ctl mcp add|list|remove|test and /api/v1/mcp/servers keep the
servers; a token given with --secret-file is kept in the store, sealed,
and named by reference. keys limits --tools, and the console's limits
form, grant tools as patterns. Documented in install.md and clients.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 12:50:55 +02:00
Ilker ManapandClaude Opus 5.5 ca77637a71 /mcp serves the company's own MCP servers' tools to keys granted them
Each registered server's tools are listed as server.tool beside llmgw's
own, to a key whose tool grants match, and called with the server's
credentials, read from their secret references at each call; the
server is told the key, person and project calling. A call is bounded
in time and size, its progress passed on to a streaming client, and
recorded in the audit log with its arguments' size and digest, never
their text. A key granted nothing sees none of them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 10:53:24 +02:00
Ilker ManapandClaude Opus 5.5 765b6f5603 An MCP client for the company's tool servers
internal/mcpclient speaks streamable HTTP: initializes once and keeps
the session the server gives, beginning it again once if the server
forgot it; lists tools page by page; calls one with headers of its own,
taking the answer as JSON or as events with progress before it; and
bounds what it reads. mcptest is a server for tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 10:43:35 +02:00
Ilker ManapandClaude Opus 5.5 6dfdfbf2fe The store keeps the company's MCP servers, and which tools a scope may call
mcp_servers holds each server: its address, headers, and secret
references for the headers that carry credentials. limits gains
allowed_tools, patterns such as jira.* resolved down the tenancy tree;
unlike models, a tool no level grants is not granted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 10:36:08 +02:00
Ilker ManapandClaude Opus 5.5 8ffde2b80c A big model's download no longer runs out its startup timeout
CI / test (push) Successful in 5m18s
The startup timeout runs from when a node's download last moved, so a
model of hundreds of gigabytes on a slow mirror is not failed while it
arrives. ctl model serve --env sets engine variables (NCCL_DEBUG=INFO)
on every node, and a layout given by hand keeps expert parallelism for a
model of experts. docs/large-models.md: how a model is split, what each
way needs of a card, and a checklist for bringing one up on a fabric.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 09:45:31 +02:00
Ilker ManapandClaude Opus 5.5 4fe9b46451 A plan for a big model: experts spread, any number of nodes, memory said
A mixture-of-experts model on more than one card is served with vLLM's
--enable-expert-parallel; config.json's experts are read by any of their
names, and a model that also reads pictures is sized by its text_config.
Tensor parallelism past the KV heads is allowed when it is a multiple of
them (Qwen3-235B, four KV heads, runs on eight cards), which the plan
had refused. Several nodes are offered in any number, not only powers of
two. Each way says what it needs of a card, part by part, and of all its
cards; the serve form checks a context longer than planned still fits.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 08:45:33 +02:00
Ilker ManapandClaude Opus 5.5 314cc5e6a3 The judge init installs is no longer replaced by the leader's own
CI / test (push) Successful in 5m27s
Found on a fresh HA install with a judge machine: the leader saw the
policy's judge missing while init was still fetching it, served one
itself, and its entry replaced init's, so a second judge ran off the
judge machine. The keeper now waits ten minutes after it starts, and a
serve job it starts gives way to a model added meanwhile (AddNew).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 20:37:10 +02:00
Ilker ManapandClaude Opus 5.5 34764f5ef0 A node given a role takes only what asks for it
CI / test (push) Successful in 5m24s
The judges' machine was selected by the judge's profile, but any other
model could still be placed beside it. A node labelled llmgw.role now
takes only profiles whose selector asks for that role.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 17:52:30 +02:00
Ilker ManapandClaude Opus 5.5 51ca09287f A three-machine install can give the judges machines of their own
CI / test (push) Successful in 5m10s
The setup screen asks for up to two judge machines after the store's
two, and server init takes --judge-node for each. Each is added as a
node labelled llmgw.role: judge, and the judge's profile selects that
label, with cores by init's rule over those machines. A judge beside a
busy control plane refused 38 of 60 requests under load; alone, 2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 17:45:00 +02:00
Ilker ManapandClaude Opus 5.5 8b122aae09 A judge the leader serves by itself is sized as init's is
CI / test (push) Successful in 5m13s
A judge a policy names and the catalog lacked was served with 3 GB, no
cores and llama.cpp's prompt cache; on a busy node its judgments missed
their budget and fail: closed refused the requests. It now gets init's
2 GB and cores, and every judge modelserve serves runs with
--cache-ram 0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 16:32:04 +02:00
Ilker ManapandClaude Opus 5.5 cfc38d35cc A node's console opens on what the machine is doing
CI / test (push) Successful in 5m12s
sudo llmgw on a node showed only agent.yaml, the manual, wipe and leave.
It now opens on an Overview read on the node itself: the agent's state and
version, the control planes it dials and whether they answer, each GPU with
the models on it, the engines from the supervisor's state files, the free
space for weights and engines, and the agent's warnings of the last hour.
No admin token is needed, every command is bounded by two seconds, and a
missing tool is said in a line.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 12:55:27 +02:00
Ilker ManapandClaude Opus 5.5 ac25a9136a Sign-in is set up from the console, and a directory tried before it is kept
People → Sign-in shows single sign-on, Active Directory and SCIM, and
the directory's last sync. E sets up Entra ID from the tenant and client
IDs; D sets up Active Directory, tried from the control plane first:
each server bound, people who may sign in, members of each mapped group.
The secrets go into the store as store:auth/..., sealed by the master
key, and the CA into the settings, so no control plane needs a file.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 12:55:27 +02:00
Ilker ManapandClaude Opus 5.5 4b6b3eeddb A node's page says which llmgw it runs and which model is on each GPU
CI / test (push) Successful in 4m13s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 11:31:52 +02:00
Ilker ManapandClaude Opus 5.5 178b168bfd A broken directory setting, or a second server, no longer takes the cluster down
CI / test (push) Successful in 4m15s
Found trying auth.ldap against a Samba DC on the PoC. A CA file a
control plane could not use made every control plane exit, and no
ctl config could then put the setting right: directory sign-in is now
turned off there, and said. `llmgw server check` ran a whole server
that, its ports taken, held the leader's lease for three hours: server
refuses words it does not know and takes its ports before anything else.
The guides now say auth is set with ctl config, not server.yaml.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 11:10:43 +02:00
Ilker ManapandClaude Opus 5.5 f27d722111 People sign in with their Active Directory account
auth.ldap checks a username and password against the domain controllers
over LDAPS or StartTLS: found with a service account, checked by binding
as the person. Groups, nested ones included, give staff roles and places
in the tree. Accounts are tied to the objectGUID; the leader reads the
directory every 15 minutes, suspends whom it disabled or removed and
reinstates them after, and refuses a sync that would suspend most.

Entra ID sends no groups for someone in more than 200, only where to
fetch them; that was read as none and took an administrator's role away.
It is now left as it was, and logged. An account made by SCIM or OIDC
can be reinstated again.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-18 10:59:50 +02:00
Ilker ManapandClaude Opus 5.5 fe310576e7 Several keys for one outside provider
CI / test (push) Successful in 8m52s
An external model takes api_key_files, each a file and an optional weight,
in place of api_key_file. Each key is an endpoint of its own
(external:MODEL#1, #2), so the router's scoring, retries and breakers work
per key. A key answering 429 is passed over while its breaker is open. A
key answering 401 is passed over too, and external_key_refused is sent,
naming the endpoint and never the key; with a single key the 401 is still
the client's. A key whose file cannot be read is left out, and usage
records which key answered by its endpoint.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-17 08:56:30 +02:00
Ilker ManapandClaude Opus 5.5 688b205604 What llmgw adds to a request, measured: ctl bench overhead
CI / test (push) Successful in 8m56s
An engine that answers at once and the gateway in front of it, in one
process on loopback, given the same load straight and through the gateway:
bare, with authentication and accounting, and with the guardrails' pattern
checks. It reports p50 and p99, what the gateway added, and a rough CPU
per thousand requests.

On an Apple M5 at 100 to 2,000 requests a second (10 s a step, 72,000
requests, none failed), the gateway added 0.05–0.42 ms at p50 and at most
0.49 ms at p99. That is now in the README, with how it was measured and
why the CPU figure is rough. CI runs a one-second version.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-17 00:23:26 +02:00
Ilker ManapandClaude Opus 5.5 35ccabac28 modelserve: a job's request is set under the lock Get reads it with
CI / test (push) Successful in 8m49s
CI run 394, on a commit that changed only BACKLOG.md, failed with a data
race in TestAVisionModelIsServedWithItsProjectorBesideIt. The serve job's
goroutine wrote j.Request without the lock (since efffdb6, which sized
embedding slots), while Get's snapshot read the job under it. Every other
write to a running job holds the lock. It did not reproduce in 30 runs
here; the race detector's trace names this write and that read.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 22:40:09 +02:00
Ilker ManapandClaude Opus 5.5 2545fca174 An upgrade refits the judge an earlier server init installed
CI / test (push) Successful in 8m53s
A v0.14.6 install on fewer than eight cores refused every request until
its judge was given cores by hand, and upgrading did not change that. The
judge keeper, on the leader, now finds a judge whose entry is exactly what
init before v0.14.7 wrote: one llama.cpp CPU profile, an 8192 context and
nothing else, 2 or 4 GB, none to two cores. It gives the judge 2 GB,
--cache-ram 0 and cores by init's rule through modelsvc.SetProfiles, then
restarts its ready replicas. The replica count is left as it is. A judge
whose entry differs is left alone, and the log says so once.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 22:06:56 +02:00
Ilker ManapandClaude Opus 5.5 b8274d6278 A restart is not refused for room another restart is giving back
CI / test (push) Successful in 8m51s
On a 4-core single-machine install, `ctl model resources` was run on the
judge (to two cores) and on the starter (to none), then both restarted.
The judge's restart was refused: its replacement fit on no node even with
its old replica stopped, because the starter still held its two cores. A
restart now waits for other restarts under way (queued, retiring or
draining) when their going would make room, up to DrainGrace.

A drained replica was also kept until its model had enough live replicas,
and a replica being restarted that still answers did not count. So with
replicas restarted one after another, each drained one waited out the
five-minute grace, holding its room.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 21:33:20 +02:00
Ilker ManapandClaude Opus 5.5 3e7dd20bad One machine runs one judge, on up to four cores
CI / test (push) Successful in 9m1s
A second judge replica on the same machine shares its cores and is lost
with it, so it bought nothing but half the speed of each. On a 4-core,
8 GB single-machine install, one judge on four cores passed 20 of 20
requests at about 3.5 s each, judged in and out. Two judges on two cores
each had passed 19 of 20.

server init now serves one judge when this machine is the only node, and
two, on two cores each and spread over the nodes, where there will be
others (HA setups, or a control plane that is not a node).
--judge-replicas and the setup's Copies still choose. install.md gives
the measured times in place of "half a second".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 19:35:42 +02:00
Ilker ManapandClaude Opus 5.5 f8206c3540 A judge reserves two cores and 2 GB, with no prompt cache
CI / test (push) Successful in 9m1s
Checked by a single-machine install of v0.14.6 on a 4-core, 8 GB VM. The
second judge fitted nowhere: v0.14.6 reserved 4 GB a judge, a figure from
one replica that grew to 3.2 GB through llama.cpp's prompt cache (up to
8 GB by default), which a judge never reuses. Once it fitted, every
judgment timed out. v0.14.6 gave a judge one core below 8 CPUs, and a
judgment of about 280 tokens takes 8.0 s on one core, 4.2 s on two and
2.3 s on four, against a 5 s budget: 0 of 20 requests went through.

A judge now runs with --cache-ram 0, uses about 1.6 GB and reserves 2,
and reserves two cores (one only below four CPUs). On the same VM, 19 of
20 went through.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 19:22:33 +02:00
Ilker ManapandClaude Opus 5.5 63b2c6d9a4 The judges and the starter divide a machine's CPUs instead of each taking all
CI / test (push) Failing after 30m7s
Measured on the three-VM cluster with ha1's CPUs loaded: 38 of 60 requests
refused with both judges and the starter on ha1, each engine on all four
cores. With the judges moved to a 4-core, 32 GB node at 2 cores each, 2 of
60 were refused, both the first after the judges started.

init now reserves cores by this machine's count: one per judge replica
under 8 CPUs and two from 8, and the rest, up to 8, for the starter on CPU.
The starter reserves nothing when nothing is left, so it still fits. Judges
bound for nodes not yet known get two each. A judge reserves 4 GB; it uses
about 3.2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 13:00:39 +02:00
Ilker ManapandClaude Opus 5.5 0b984a92bc A judge is asked again only on another machine
CI / test (push) Failing after 30m3s
Measured under CPU load on the three-VM cluster, with both judge replicas on
ha1: 37 of 60 requests failed with the hedge, 38 of 60 without it. Two
replicas on one saturated machine are slow together, and the second
question doubled the work there. The hedge now goes only to a replica whose
address names another host.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 10:27:50 +02:00
Ilker ManapandClaude Opus 5.5 bc4c5d5ac9 A judge replica that hangs is overtaken by another at half the budget
On the three-VM cluster about one answer in twenty-five came back as
content_filter: both judge replicas timed out within the same millisecond.
The tries shared the policy's 5 s budget, so the first replica, busy on an
oversubscribed machine, used it all and the retry on the second had nothing
left. A judge request now goes to a second replica too when the first has
not answered by half the budget, and the first answer is taken. The first
is not cut short.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 08:35:28 +02:00
Ilker ManapandClaude Opus 5.5 b09922b29b Leadership stays put while the store's primary is replaced
CI / test (push) Failing after 30m3s
Measured again after the store fixes. Cutting the store primary's power no
longer hung requests, but leadership moved anyway: the lease had lapsed for
everyone while Patroni promoted the standby, and a standby asked first. The
new leader then spent about 25 s without its nodes, and so without its
judges.

A standby that saw the store out of reach now leaves the lease alone for one
renew round after the store returns. The default lease is 10 s, and a node
detects a control plane that died silently in 8 s (5 s idle, 3 s ping)
rather than 15.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 23:17:58 +02:00
Ilker ManapandClaude Opus 5.5 5a4b022dcd A lost store machine is given up on in seconds, and a new leader waits for its judge
CI / test (push) Failing after 30m6s
Measured on three VMs: cutting the store primary's power cost about 45 s of
hanging requests. New connections tried the dead member first with no
connect timeout, and Patroni kept its 30 s leader key. Cutting the leader of
both cost a further 17 s of 503s while the new leader knew no judge.

The store dials each member for 3 s and closes connections to a machine that
is gone within about 10 s (keepalive, TCP_USER_TIMEOUT). Patroni runs with
ttl 20, loop_wait 5 and retry_timeout 5, applied in place to existing
clusters through its API. Judges that all fail at once are asked again
every 250 ms while the policy's budget lasts, and reported once.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 20:33:13 +02:00
Ilker ManapandClaude Opus 5.5 eeaffd764d A floating address keeps to its own network, and Caddy follows the leader
CI / test (push) Failing after 30m3s
Found by a three-machine install on fresh VMs. The second standby's run
asked for the floating address's interface with `ip route get <vip>`.
The machine already held the address, so that answered lo, and keepalived
would not start. The interface now comes from the route to the address's
network. Every cluster's VRRP router id was 51, so the PoC's two control
planes and the new three on one LAN each logged the other's adverts as an
invalid password; the id now comes from the address.

The Caddyfile, deploy/ha and install.md send clients to the control plane
whose /leader is 200, not /ready: /ready is 503 on a leader with no model
yet and would take the portal away with it. Setup no longer hides a key
file's path as if it were a password, and the starter's text no longer
says it runs elsewhere when this machine is a node.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 17:35:10 +02:00
Ilker ManapandClaude Opus 5.5 94401e21e6 Setup lets every part be chosen, each with its settings; three machines each run the gateway
CI / test (push) Failing after 30m4s
The setup screens list the parts of an install: control plane, store, model
mirror, this machine as a node, starter model, pictures and 3D, the judge,
firewall, notifications, limits, on-demand turns and an offline bundle. Each
says what it is or what is missing without it. Space turns a part on or off;
Enter opens its settings in a dialog that explains it. A plan page lists what
is installed and what is left out, and the same install as a command. Every
setting is a server init flag too, written into server.yaml when init writes
it.

With three machines, the two database machines become control planes too,
standing by (server add's steps), with an optional floating address. The
mirror's management API answers all three control planes, so a standby that
comes to lead can still fetch models.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 11:22:36 +02:00
Ilker Manap 733e20924c Merge remote-tracking branch 'origin/mcp-wait' into sd-dedupe
CI / test (push) Failing after 30m3s
2026-10-15 07:53:22 +02:00
Ilker ManapandClaude Opus 5.5 fc60481e40 Every control plane lists the others alike to Prometheus
140 named 150 a store member, 150 named itself a control plane: Prometheus
asking both scraped 150 twice and counted its every series twice, and asked
of 140 alone while 150 led it lost every node. Each control plane now lists
all of them as control planes before the store's members, so two answers
agree and a target is scraped once. The shipped prometheus.yml says to
scrape and ask every control plane.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 07:53:22 +02:00
Ilker ManapandClaude Opus 5.5 edeeb9f8e1 MCP takes pictures of real size, by id too, and waits for a cold model
The game's bridge found three things on the PoC. /mcp held its body to
64 KiB — the edit meant for it had raised the memory notes' limit instead —
so a 1024 px concept sent as base64 failed as "expected one JSON-RPC
request object". The notes are back to 64 KiB, /mcp takes 64 MB, and a
call past it is a 413 saying so.

/v1/files took batch files only, so a picture could not be sent once and
named after. purpose vision and user_data are kept too, as long as job
results, and generate_mesh hands a workflow model the id as it is.

generate_image and review_images failed at once while their model was
cold. They wait out an answer that says when to come back — loading,
another model holding the GPU, a full queue — for as long as the call may,
saying who holds the GPU and about how long; a refusal that says no
Retry-After, a spent budget, is said at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 07:24:59 +02:00
Ilker ManapandClaude Opus 5.5 c22528c26e A model that does not fit its card is not added to fail; weights in use are not removed
CI / test (push) Failing after 30m3s
Apertus 70B with a 65536-token pool, and then 8B with 131072, on a 24 GB
card loaded their weights and failed to make their context on every start,
the 8B taking the card from the 70B each time only to fail. ctl model serve
now shrinks a pool the card cannot hold beside the weights (KV cache from
the GGUF's shape, 90% of the card, a gigabyte for the engine), saying so,
and refuses weights that do not fit at all; the portal's serve form refuses
such a pool and says the largest that fits.

The mirror page removed the 70B's weights while the catalog still had the
model, whose replicas went on loading the node's copy with nothing said.
Removing weights a catalog model loads is refused, naming the model.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 00:22:01 +02:00
Ilker Manap bda988b78f Merge remote-tracking branch 'origin/setup-webhooks' into shared-replica
CI / test (push) Successful in 9m54s
# Conflicts:
#	CHANGELOG.md
2026-10-14 14:26:27 +02:00
Ilker Manap a7cb3aac3c Merge remote-tracking branch 'origin/mcp-media' into shared-replica
# Conflicts:
#	CHANGELOG.md
2026-10-14 14:01:24 +02:00
Ilker ManapandClaude Opus 5.5 d1e91868ee Setup asks where alerts go, and sends each a test
notify.webhooks existed and setup never asked for one, so a new site
learnt of a parked model or a lost node from its users. `llmgw server
init --webhook URL` (repeatable) and a Notification webhooks field on
both setup screens take them now. Slack's address, Mattermost's and
Rocket.Chat's /hooks/ and Discord's …/slack are sent Slack's text, any
other the event as JSON; slack: or json: in front says which.

Once the control plane runs, they are added to the cluster's settings
beside the webhooks already there, none twice, and setup waits for the
control planes to start again on the change before sending each a test,
saying which answered. A webhook's path is its secret, so it is shown by
host only. A failure is said, not fatal: the cluster serves without its
alerts, and ctl notify test tries again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:55:48 +02:00
Ilker ManapandClaude Opus 5.5 aa531b1053 ComfyUI models of one environment share one replica
flux and sf3d on one card handed it to each other, each starting ComfyUI
and loading its weights from nothing (BACKLOG H3, placement side). An
environment of two or more models now gets a hidden catalog entry,
comfyui-env-<name>: on its first member's profiles with the most memory
any member needs, on demand for the longest idle timeout, carrying every
member's weights, repositories and the environment's custom nodes. The
members get no replicas of their own; asking for one wants the entry.

The agent fetches each member's weights; the gate links all of them,
refusing two weights files of one name, links each member's Hugging Face
repository, and answers /v1/models for every member. The control plane
offers the replica under the members' names with each one's workflow and
checkpoint, which the router sends by the model asked for. A member's
sibling never evicts the replica serving it, and the members' load counts
as the entry's.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 13:55:34 +02:00
Ilker ManapandClaude Opus 5.5 55d5e13534 Pictures, 3D models, a vision model and workflows as tools at /mcp
An agent driving Blender made its concepts, meshes and render reviews
through a tool server of the game's own, which called this gateway; the
gateway's MCP server offered only the memory notes. It now offers
generate_image, generate_mesh, review_images, run_workflow, job_status and
gateway_status beside them, each only where a model for it is here.

Each tool makes its request of the client API as the caller, through the
handler Handler built, key check and all: allowlists, limits, licences
left out, budgets and accounting apply as to any request, and a job is
the caller's. What a tool makes is kept as the caller's file and answered
by id and download URL, since a server over HTTP cannot write to the
caller's disk; a picture comes with a 512 px preview, and whole where no
file can be kept. A picture to work from is a file id, a data URL or
base64; one sent to the vision model is shrunk to 1280 px.

A mesh takes minutes. A client that takes a stream is answered as
server-sent events, with notifications/progress for its token while the
job is queued, waiting for the GPU and running, and waits up to ten
minutes; one that does not waits 50 seconds and gets the job id, which
job_status asks after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:11:09 +02:00
Ilker ManapandClaude Opus 5.5 7c53380e68 Extended thinking on /v1/messages: a model's reasoning as thinking blocks
CI / test (push) Successful in 9m12s
Claude Code and the Anthropic SDKs asking for extended thinking got the
answer alone: a reasoning model's reasoning_content was dropped. Asked for,
it comes back as a thinking block before the text — streamed as
thinking_delta events and closed with a signature before the text begins —
and the model is told to think through its chat template
(enable_thinking); disabled tells it not to. Not asked for, it is left out
as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 11:05:02 +02:00
Ilker ManapandClaude Opus 5.5 0460f52dd2 Jobs run side by side, one per model at a time
CI / test (push) Successful in 10m28s
One job at a time, a 3D job waiting for the card held up another model's
job that could have run (seen on the PoC, BACKLOG H7). The leader's worker
claims jobs while fewer than eight run, passing over the models that have
one running, and starts each in its own goroutine; one ending wakes it for
the next. Lease and leader-only rules are as they were.

BACKLOG: H5–H7 marked, H8 (presets) and H9 (what the PoC taught) written.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 10:06:20 +02:00