Commit Graph
219 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 540b518fde Rate limits counted across the control planes with router.active_active
CI / test (push) Successful in 9m30s
Each router's per-counter request and token counts, this minute's and the
last, ride the standby's two-second exchange with the leader. The leader
answers each standby with everyone else's sum, its own included, and puts
the standbys' sum in force on its own router; each limiter adds the
others' counts to its sliding window.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 17:30:08 +01:00
Ilker ManapandClaude Opus 5.5 8b7eefef16 Standbys that answer the client API themselves, behind router.active_active
CI / test (push) Successful in 9m35s
A standby follows the leader's Ready replicas over the agent port, every
two seconds, and reports back the replicas its router used and the
on-demand models it was asked for. No engine keys travel: every control
plane derives them from the cluster secret. Budgets are raised to the
store's every 30 s. Unheard from the leader for a minute, a standby
passes /v1 on as before. Off by default.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 13:47:47 +01:00
Ilker ManapandClaude Opus 5.5 5550fefaad Stream events already in hand go to the client in one write, not one each
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-31 15:30:19 +01:00
Ilker ManapandClaude Opus 5.5 524f029603 External providers left out of admission's saturation; the scale-out section in the backlog
CI / test (push) Successful in 9m25s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 19:13:15 +01:00
Ilker ManapandClaude Opus 5.5 76746715ec A set's files checked four at once, the check said as it goes and not taken for a stuck download
CI / test (push) Failing after 6m51s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 14:58:44 +01:00
Ilker ManapandClaude Opus 5.5 cacaa35c3f The gateway embeds the documents llmgw-kb sends as text
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 21:56:05 +01:00
Ilker ManapandClaude Opus 5.5 59187cdae5 A model can be stopped and kept, or removed with its files from the mirror and the nodes
CI / test (push) Successful in 10m37s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 20:45:48 +01:00
Ilker ManapandClaude Opus 5.5 1908bfb4b2 Strata on cuda01 end to end: its engine is the shard's, its metrics read
CI / test (push) Failing after 7m31s
Found running it through the cluster:

- Strata's server starts its engine in a process group of its own, so the
  agent saw a GPU replica on the CPU and its 22 GB as somebody else's
  memory. A process the shard started, at any remove, is now the shard's.
- An adopted shard's kept compute was sent at once and never again, so a
  wrong one read again on adoption never reached the control plane: what
  is sent is sent again when it changes, and a GPU shard kept as the CPU
  is read again on adoption.
- /metrics/prometheus is llama.cpp's names without the KV cache; vLLM's,
  with it, are /metrics?format=prometheus. The router now sees running 1,
  waiting 2 under load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 15:48:15 +01:00
Ilker ManapandClaude Opus 5.5 c0d5194ea0 Strata runs as an engine: a 125B MoE on one card
Strata (github.com/Niko1221/Strata) keeps a mixture-of-experts model's
experts in RAM and the ones used most in VRAM; on cuda01's RTX 6000 it
serves Qwen3.8-Flash-Next 125B at 70-87 tokens a second to one client.

- engines/strata.go: the adapter. The agent starts llmgw strata-serve,
  which runs Strata's setup the way its desktop app does
  (STRATA_CONTROL_INSTALL: installs nothing, writes only under the
  replica's and the node's directories) on the fetched GGUF files, points
  the configuration it writes at the installed engine, and execs Strata's
  server. The key goes in STRATA_API_KEY, never on a command line.
- A profile's args go to setup: --family, --size (setup's --model, which
  is the source everywhere else), --context, --parallel, ... Flags that
  move files or pick the engine or GPU are refused.
- provision: the source pinned by digest in a Python environment, and the
  release's prebuilt engine unpacked beside it, checked against the digest
  GitHub publishes: CUDA 13 (7.5+, driver 580) or CUDA 12.9 (6.0-8.9,
  driver 575). Python builds can now carry release assets.
- Metrics are vLLM's names under /metrics/prometheus.
- The catalog wants one card and a family, and a directory, not one .gguf.

Opt-in: not in DefaultEngines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 15:09:13 +01:00
Ilker ManapandClaude Opus 5.5 c09d5ad4ee Outside services read answers redacted and whole; a card's own lanes
CI / test (push) Successful in 9m16s
From the second review.
- An output service reads the answer with the values restore put back
  taken out again (guard.Subject.Restored), as an input service reads the
  request redacted.
- With n above 1, the service reads the first choice once it is whole,
  not at whichever choice finished first.
- Restored values go into an answer asked for as JSON (response_format)
  escaped for it, as into tool-call arguments.
- The embedding failure is logged once a minute through an outage, not
  once.
- A card's own lane count and fastest generation come from the kernel
  (max_link_width, max_link_speed): the driver's width.max is what card
  and slot allow together, so an x16 card in an x8 slot read x8 of x8 and
  was never said.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 13:22:39 +01:00
Ilker ManapandClaude Opus 5.5 5eac62d556 Guardrail services merged the stricter way; the request log keeps no restored value
CI / test (push) Failing after 7m44s
From review.
- A service two policies name is asked the stricter way of the two, both
  directions and block over flag, rather than as the first one says.
- A service answering null where it says unsafe gave no answer, not safe.
- With restore: true, the kept answer had the request's redacted values in
  it beside the redacted request; they are taken out again before it is
  kept.
- guardrails.md says what fail: open lets through, and that of n answers a
  service reads the first.
- The similar cache's embedding failures are logged once a minute, not per
  request; the page's test reads the gateway's local MCP handler under its
  lock; org/name:Q4_K_M is not taken for a Hugging Face id to read as
  safetensors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 11:03:57 +01:00
Ilker ManapandClaude Opus 5.5 54c6b66cb9 Fixes from review: the similar cache, restore, outside services, latency
CI / test (push) Successful in 9m48s
- The similar-question cache embeds a question as its project would: a
  project kept in the cluster has external fallbacks refused, and an
  embedding model outside the cluster is not asked at all. The vectors are
  compared outside the cache's lock. The question's bound is in characters.
- The restorer puts values back in streamed tool calls' arguments (escaped
  for the JSON they are), completions' text and reasoning, each held across
  events on its own, and releases a held tail with the finish event. A
  whole answer is walked as JSON, so a value with a quote no longer breaks
  tool-call arguments.
- An outside guardrail service reads the user's turn as redacted, and a
  streamed answer once rather than at the finish chunk and again at the
  end. guardrails.md says it can stop a stream only under buffer_all.
- A replica's smoothed time to first token is forgotten after five minutes
  without an answer: one found slow and then given nothing was never tried
  again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 08:15:43 +01:00
Ilker ManapandClaude Opus 5.5 fd94265e08 Pickles refused however they come; a string temperature passes again
CI / test (push) Failing after 7m17s
From review. A mirror set already on the node is looked at again, not
trusted for having been fetched once; Pickled cuts a query or a fragment
from any source, hf: too; .pt2 is a pickle. The flag check reads a flag as
the engine does: underscores as dashes and any beginning of a refused flag
as that flag, since argparse takes --trust-remote for --trust-remote-code;
SGLang refuses --load-format, which would undo the agent's safetensors.

TripoSR's preset says allow_pickle: without it the preset could not be
added and the presets pass failed on it each time. A shared ComfyUI
environment's hidden entry takes its first member's allow_pickle, as it
takes its source.

The GenAI trace attributes made temperature and top_p typed fields of the
request's parse, so "0.7" as a string was a 400; they are read loosely,
as they are read only for the trace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 22:19:06 +01:00
Ilker ManapandClaude Opus 5.5 7a04248148 A reused answer never carries another caller's values
CI / test (push) Successful in 9m50s
With restore: true and the answer cache on, alice's "write to
alice@example.com" and bob's "write to bob@example.com" redact to the
same request, and bob was given the answer kept for alice, her address put
back in it. A request with restore values is now never looked up in the
cache nor kept in it.

The similar-question cache compared the last message's text and blanked
its parts: two pictures asked about in the same words were one question.
A last message with any part that is not text is now reused only when the
request is the same.

Found in review; both tests fail without the fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 22:03:32 +01:00
Ilker ManapandClaude Opus 5.5 03d3bcfd2d The company's MCP servers on a web page
CI / test (push) Successful in 9m8s
/admin/mcp, for administrators only, since a server's credential reaches
whatever it guards: each server's state, who has been granted its tools,
its rate, the headers it is sent by name with the sealed ones marked, and
its address; adding one with a bearer token or another header kept in the
store; testing it with its credential and listing its tools; granting them
to a team, project, org, person or key, written into that level's limits
beside its other grants and its rate and budget; revoking; changing;
replacing the credential; turning off and on; removing; and how a person
points an agent at the gateway's /mcp with a key of their own. English and
Turkish. The secret's field is named so the audit line redacts it.

The admin API's server merge and test are package functions now,
adminapi.MCPServerFrom and adminapi.TestMCPServer, shared by both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 19:26:51 +01:00
Ilker ManapandClaude Opus 5.5 3a3db8cd20 Answers to similar questions, not only the same one
CI / test (push) Successful in 8m52s
llmgw keys cache --similar 0.95 also gives a chat question the answer to
one like it: the last user message embedded by router.cache.embedding_model
(asked on the gateway's account, as a judge is, within 2 s), compared by
cosine with the questions answered in the same conversation — system
prompt, earlier turns, tools and sampling the same. The exact cache is
asked first; an exact hit embeds nothing. Said as x-llmgw-cache: similar
with x-llmgw-cache-similarity, charged as a hit.

The vectors live with their answers in the gateway's memory, counted in
max_bytes and evicted with them, not in a pgvector table as BACKLOG I1
planned: the cache keeps answer text out of the store on purpose. A
question over 2,000 characters is reused only when the same. Migration
0051 holds the setting. BACKLOG I1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 17:47:11 +01:00
Ilker ManapandClaude Opus 5.5 ed0d1ec0ad A cached prompt costs the cached rate
CI / test (push) Successful in 8m45s
cached_per_1k in a price charges the prompt tokens the answer counts as
cached_tokens at their own rate; zero keeps charging them as prompt tokens.
The count reaches the cost and not a column: CostMicros holds what it
changed. Cached beyond the prompt is clamped, an engine's miscount rather
than a refund. BACKLOG L5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 17:15:48 +01:00
Ilker ManapandClaude Opus 5.5 73e1bb6733 Chat goes to the replica that has been answering soonest
CI / test (push) Failing after 7m18s
The router keeps each endpoint's time to first token, smoothed over its last
streamed answers, and the chat profile scores by it beside session, queue,
KV and inflight: the fastest endpoint 1, one twice as slow 0.5, one not yet
measured 1 so a new replica is tried. Load still sees a slow drain from the
engine's own counters; this sees what answers actually took. BACKLOG L4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 17:01:30 +01:00
Ilker ManapandClaude Opus 5.5 81685ecd75 A request's redacted values are put back in its answer
CI / test (push) Successful in 8m49s
With restore: true on a guardrail policy's input, what redact rewrote as
placeholders comes back in the answer: the model reads "[EMAIL_1]", the
client reads the address it sent. A streamed placeholder split across
events is held until it is whole, and what is held is sent at the end.
The swap runs before the output filter, so the output rules judge the
answer as the client reads it and a placeholder they make of a value the
model invented is never taken for the request's own. The map lives in the
Verdict for the request only; nothing stores, logs or traces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 16:41:53 +01:00
Ilker ManapandClaude Opus 5.5 06baa38858 Traces say model calls in the OpenTelemetry GenAI conventions
CI / test (push) Successful in 8m55s
An inference request and each attempt at an engine carry gen_ai.*: the
operation, the requested and answering model, max_tokens, temperature and
top_p when the client set them, the session as the conversation id, the
input, output and cached tokens, and on an attempt the engine that answered.
A customer's Grafana, Jaeger or Langfuse then shows llmgw's calls as model
calls. No prompt or answer text is ever put in a span, and a test says so.

BACKLOG gains section L: what is worth taking from Kong's AI Gateway, of
which this is the first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 12:59:20 +01:00
Ilker ManapandClaude Opus 5.5 fea5774fe5 The install guide shows every screen of a real install
/install takes a reader from three bare Ubuntu machines to a first answer:
what each machine needs and the commands that check it, the release fetched
and its checksum checked, the eight setup screens, the system's state and
the console, and a first request with a key from the account page. Every
screen is from an install of v0.14.9 on three fresh VMs, recorded with tmux
and kept as terminal frames: the page shows them as text, and plays a
session back when asked, fetching it compressed from /install/cast/. The
administrator's password and the API key were masked before the recording
was kept, and a test fails if either shape reaches the page.

/about gains what llmgw now does beyond routing, and its install steps are
today's: the setup screens and node add, not onboard-node.sh and a model
file, with a link into the guide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 01:35:08 +02:00
Ilker ManapandClaude Opus 5.5 3a4488b41e The company's MCP tools under the guardrails, a rate and a breaker
CI / test (push) Successful in 5m35s
A tool's answer is read by the caller's policies as the prompt it
becomes, so an instruction planted in a ticket is withheld and personal
data redacted; its arguments are read before they leave. ctl mcp add
--rate bounds each caller's calls a minute to a server (migration 0050),
and five failures in a row take a server out for a minute, its tools
off the list.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 23:00:22 +02:00
Ilker ManapandClaude Opus 5.5 4eb49d0e4c The cluster as a read-only MCP server, llmgw:admin
CI / test (push) Successful in 5m26s
Its tools say what llmgw ctl does: cluster_status, list_nodes,
list_models, list_replicas, explain_placement, node_logs; none changes
anything. The admin API serves it at POST /api/v1/mcp, and the gateway
answers a server registered as llmgw:admin in its own process, so no
token is kept and each control plane serves it. ctl mcp add takes
--bearer-file for a token sent as Authorization: Bearer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 17:18:28 +02:00
Ilker ManapandClaude Opus 5.5 4453f469e1 Company MCP servers are registered with ctl mcp, and tools granted with keys limits --tools
CI / test (push) Failing after 5m4s
llmgw ctl mcp add|list|remove|test and /api/v1/mcp/servers keep the
servers; a token given with --secret-file is kept in the store, sealed,
and named by reference. keys limits --tools, and the console's limits
form, grant tools as patterns. Documented in install.md and clients.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 12:50:55 +02:00
Ilker ManapandClaude Opus 5.5 ca77637a71 /mcp serves the company's own MCP servers' tools to keys granted them
Each registered server's tools are listed as server.tool beside llmgw's
own, to a key whose tool grants match, and called with the server's
credentials, read from their secret references at each call; the
server is told the key, person and project calling. A call is bounded
in time and size, its progress passed on to a streaming client, and
recorded in the audit log with its arguments' size and digest, never
their text. A key granted nothing sees none of them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 10:53:24 +02:00
Ilker ManapandClaude Opus 5.5 ac25a9136a Sign-in is set up from the console, and a directory tried before it is kept
People → Sign-in shows single sign-on, Active Directory and SCIM, and
the directory's last sync. E sets up Entra ID from the tenant and client
IDs; D sets up Active Directory, tried from the control plane first:
each server bound, people who may sign in, members of each mapped group.
The secrets go into the store as store:auth/..., sealed by the master
key, and the CA into the settings, so no control plane needs a file.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 12:55:27 +02:00
Ilker ManapandClaude Opus 5.5 178b168bfd A broken directory setting, or a second server, no longer takes the cluster down
CI / test (push) Successful in 4m15s
Found trying auth.ldap against a Samba DC on the PoC. A CA file a
control plane could not use made every control plane exit, and no
ctl config could then put the setting right: directory sign-in is now
turned off there, and said. `llmgw server check` ran a whole server
that, its ports taken, held the leader's lease for three hours: server
refuses words it does not know and takes its ports before anything else.
The guides now say auth is set with ctl config, not server.yaml.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 11:10:43 +02:00
Ilker ManapandClaude Opus 5.5 f27d722111 People sign in with their Active Directory account
auth.ldap checks a username and password against the domain controllers
over LDAPS or StartTLS: found with a service account, checked by binding
as the person. Groups, nested ones included, give staff roles and places
in the tree. Accounts are tied to the objectGUID; the leader reads the
directory every 15 minutes, suspends whom it disabled or removed and
reinstates them after, and refuses a sync that would suspend most.

Entra ID sends no groups for someone in more than 200, only where to
fetch them; that was read as none and took an administrator's role away.
It is now left as it was, and logged. An account made by SCIM or OIDC
can be reinstated again.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-18 10:59:50 +02:00
Ilker ManapandClaude Opus 5.5 fe310576e7 Several keys for one outside provider
CI / test (push) Successful in 8m52s
An external model takes api_key_files, each a file and an optional weight,
in place of api_key_file. Each key is an endpoint of its own
(external:MODEL#1, #2), so the router's scoring, retries and breakers work
per key. A key answering 429 is passed over while its breaker is open. A
key answering 401 is passed over too, and external_key_refused is sent,
naming the endpoint and never the key; with a single key the 401 is still
the client's. A key whose file cannot be read is left out, and usage
records which key answered by its endpoint.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-17 08:56:30 +02:00
Ilker ManapandClaude Opus 5.5 0b984a92bc A judge is asked again only on another machine
CI / test (push) Failing after 30m3s
Measured under CPU load on the three-VM cluster, with both judge replicas on
ha1: 37 of 60 requests failed with the hedge, 38 of 60 without it. Two
replicas on one saturated machine are slow together, and the second
question doubled the work there. The hedge now goes only to a replica whose
address names another host.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 10:27:50 +02:00
Ilker ManapandClaude Opus 5.5 bc4c5d5ac9 A judge replica that hangs is overtaken by another at half the budget
On the three-VM cluster about one answer in twenty-five came back as
content_filter: both judge replicas timed out within the same millisecond.
The tries shared the policy's 5 s budget, so the first replica, busy on an
oversubscribed machine, used it all and the retry on the second had nothing
left. A judge request now goes to a second replica too when the first has
not answered by half the budget, and the first answer is taken. The first
is not cut short.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 08:35:28 +02:00
Ilker ManapandClaude Opus 5.5 edeeb9f8e1 MCP takes pictures of real size, by id too, and waits for a cold model
The game's bridge found three things on the PoC. /mcp held its body to
64 KiB — the edit meant for it had raised the memory notes' limit instead —
so a 1024 px concept sent as base64 failed as "expected one JSON-RPC
request object". The notes are back to 64 KiB, /mcp takes 64 MB, and a
call past it is a 413 saying so.

/v1/files took batch files only, so a picture could not be sent once and
named after. purpose vision and user_data are kept too, as long as job
results, and generate_mesh hands a workflow model the id as it is.

generate_image and review_images failed at once while their model was
cold. They wait out an answer that says when to come back — loading,
another model holding the GPU, a full queue — for as long as the call may,
saying who holds the GPU and about how long; a refusal that says no
Retry-After, a spent budget, is said at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 07:24:59 +02:00
Ilker Manap a7cb3aac3c Merge remote-tracking branch 'origin/mcp-media' into shared-replica
# Conflicts:
#	CHANGELOG.md
2026-10-14 14:01:24 +02:00
Ilker ManapandClaude Opus 5.5 aa531b1053 ComfyUI models of one environment share one replica
flux and sf3d on one card handed it to each other, each starting ComfyUI
and loading its weights from nothing (BACKLOG H3, placement side). An
environment of two or more models now gets a hidden catalog entry,
comfyui-env-<name>: on its first member's profiles with the most memory
any member needs, on demand for the longest idle timeout, carrying every
member's weights, repositories and the environment's custom nodes. The
members get no replicas of their own; asking for one wants the entry.

The agent fetches each member's weights; the gate links all of them,
refusing two weights files of one name, links each member's Hugging Face
repository, and answers /v1/models for every member. The control plane
offers the replica under the members' names with each one's workflow and
checkpoint, which the router sends by the model asked for. A member's
sibling never evicts the replica serving it, and the members' load counts
as the entry's.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 13:55:34 +02:00
Ilker ManapandClaude Opus 5.5 55d5e13534 Pictures, 3D models, a vision model and workflows as tools at /mcp
An agent driving Blender made its concepts, meshes and render reviews
through a tool server of the game's own, which called this gateway; the
gateway's MCP server offered only the memory notes. It now offers
generate_image, generate_mesh, review_images, run_workflow, job_status and
gateway_status beside them, each only where a model for it is here.

Each tool makes its request of the client API as the caller, through the
handler Handler built, key check and all: allowlists, limits, licences
left out, budgets and accounting apply as to any request, and a job is
the caller's. What a tool makes is kept as the caller's file and answered
by id and download URL, since a server over HTTP cannot write to the
caller's disk; a picture comes with a 512 px preview, and whole where no
file can be kept. A picture to work from is a file id, a data URL or
base64; one sent to the vision model is shrunk to 1280 px.

A mesh takes minutes. A client that takes a stream is answered as
server-sent events, with notifications/progress for its token while the
job is queued, waiting for the GPU and running, and waits up to ten
minutes; one that does not waits 50 seconds and gets the job id, which
job_status asks after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:11:09 +02:00
Ilker ManapandClaude Opus 5.5 7c53380e68 Extended thinking on /v1/messages: a model's reasoning as thinking blocks
CI / test (push) Successful in 9m12s
Claude Code and the Anthropic SDKs asking for extended thinking got the
answer alone: a reasoning model's reasoning_content was dropped. Asked for,
it comes back as a thinking block before the text — streamed as
thinking_delta events and closed with a signature before the text begins —
and the model is told to think through its chat template
(enable_thinking); disabled tells it not to. Not asked for, it is left out
as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 11:05:02 +02:00
Ilker ManapandClaude Opus 5.5 0460f52dd2 Jobs run side by side, one per model at a time
CI / test (push) Successful in 10m28s
One job at a time, a 3D job waiting for the card held up another model's
job that could have run (seen on the PoC, BACKLOG H7). The leader's worker
claims jobs while fewer than eight run, passing over the models that have
one running, and starts each in its own goroutine; one ending wakes it for
the next. Lease and leader-only rules are as they were.

BACKLOG: H5–H7 marked, H8 (presets) and H9 (what the PoC taught) written.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 10:06:20 +02:00
Ilker ManapandClaude Opus 5.5 2eb2a6d8fe A mesh carries its whole licence, not only its name
CI / test (push) Failing after 6m25s
GLBs carried the licence's name, URL and notice in asset.extras.license.
A pipeline that sorts assets by licence needs what a machine matches on:
the id, whether commercial use is allowed, the regions it excludes, the
revenue cap and the attribution. The whole licence is written, as
/v1/models gives it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 01:12:47 +02:00
Ilker ManapandClaude Opus 5.5 848db08469 The cluster page and the console say which picture and 3D models are here, and add them
CI / test (push) Failing after 6m27s
A site whose catalog had no flux-schnell or triposr showed no Images or 3D
page, and nothing said they were one click away. The cluster page now has
a Pictures and 3D models card under the models: each preset, what it
makes, the classes of card here it fits, whether and by whom it was
added, and why the leader has not added it by itself; an Add button, which
says first when it would install ComfyUI on the GPU nodes and restart the
control planes; and the switch for presets.auto. The console's models
screen shows the same under its table: A adds one, asking first in the
same case, and P turns automatic adding on or off.

`ctl model add PRESET` and `ctl model presets` go through the presets API,
so the control plane keeps who added one, and `ctl model presets auto
on|off` flips the setting; against a control plane from before, add sends
the entry as v0.14.4 did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 20:48:40 +02:00
Ilker ManapandClaude Opus 5.5 3858114ada A picture in a prompt counts as what a model makes of it, not its base64
CI / test (push) Successful in 10m43s
A 1280x720 PNG sent to qwen3-vl-8b was estimated at 138 thousand prompt
tokens and refused against its 8k context before reaching the engine, which
makes about a thousand of it. Parts are counted by kind: text by its text,
a picture or a sound as a thousand tokens.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-11 20:45:06 +02:00
Ilker ManapandClaude Opus 5.5 587d8d5059 A licence left out by its name stays out once its entry gives an id
CI / test (push) Failing after 6m30s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-11 16:28:40 +02:00
Ilker ManapandClaude Opus 5.5 4bbac0833d A person leaves licences out, and no key of theirs reaches a model under one
CI / test (push) Successful in 11m28s
A game sold in the EU cannot ship what a licence excluding the EU made. On
the profile page a person leaves out licences one by one, or by rule:
every licence forbidding commercial use, every licence granting nothing in
a region. The rules cover models added later. The choice reaches every
principal through the resolved limits, so permits() leaves such models out
of /v1/models, requests, jobs, fallbacks and the portal's pickers, and a
request for one is refused 403 licence_excluded saying which choice did.

Kept in user_excluded_licences (migration 0046), copied with the store and
gone with the person.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-11 16:10:22 +02:00
Ilker ManapandClaude Opus 5.5 b6bfc33ef7 Any ComfyUI workflow runs as a job, from the inputs its catalog entry declares
CI / test (push) Successful in 9m7s
A model's catalog entry now says what its workflow takes and makes:
comfyui.inputs, each a placeholder of its name with a type (text, int,
float, bool, choice, image, mask, audio, video, file), bounds, choices
and a default; and comfyui.output, which gains video, audio and files
beside pictures and meshes. POST /v1/workflows/runs with {model, inputs}
checks a request against them before a job is written — a required input
left out, an unknown one, a number out of bounds, a song sent as a
picture, another owner's file id are each refused with what was wrong —
and the job worker replays it to a replica, where the files go into
ComfyUI's input directory, the workflow is filled and run, and the first
file of the model's output kind is the job's result. /v1/models lists
each ComfyUI model's workflow, so a client can build its form.

A PNG result carries the licence in an iTXt chunk, as a GLB does in
asset.extras. A file whose server named no type is typed by its name
before its bytes, so a WAV is audio/wav, not audio/wave.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-11 14:28:15 +02:00
Ilker ManapandClaude Opus 5.5 95c61daf1b Jobs are served by the leader, and a moment of fitting nowhere is waited out
CI / test (push) Failing after 8m47s
A 3D job leased by the standby control plane, whose router has no endpoints
and which a job replayed in the process never leaves, was told "loading"
for seven minutes while the leader served the jobs behind it. Jobs and
batches are claimed only while leading.

Five jobs asked for while cuda01's agent restarted failed for good as
"cannot be placed": re-registering, the node had no engines for a few
seconds. A job asks again for five minutes before that is its end.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 23:53:33 +02:00
Ilker ManapandClaude Opus 5.5 b1490dbf30 A job waiting for the card says what holds it and about how long
CI / test (push) Failing after 6m16s
A 3D job waiting for the picture model to give up the one card had an empty
progress; its client could not tell waiting from stuck. The control plane
says which model holds the cards and when it lets go, plus the waiting
model's last start; the 503 carries it as headers, and a job keeps it as
progress {state: waiting_gpu, held_by, eta_seconds}.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 22:25:27 +02:00
Ilker ManapandClaude Opus 5.5 b743b11c54 A picture is made ready for a 3D model; a model asleep is listed asleep
LLMGWPrepareForMesh, llmgw's own ComfyUI node, written by the gate into
every replica's custom_nodes from the binary: TripoSR's own preparation
(tsr/utils.py) — the background taken away, by the picture's alpha where
it has one and by rembg (MIT; u2net, Apache 2.0) where it does not, the
object cropped to its bounds, set square at a ratio of the side (0.85),
on grey. A picture sent as it was had a white or black shell round the
object, whatever was behind it made into the mesh.

/v1/models listed only models with a replica up, so one that loads when
asked — triposr, flux-schnell — was invisible to the clients that would
have asked for it. Those are listed now, with loaded: false; every entry
says loaded.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 20:43:20 +02:00
Ilker ManapandClaude Opus 5.5 fd3c7bda32 The client API described in OpenAPI, and both APIs in Swagger UI
CI / test (push) Failing after 6m12s
/v1/openapi.json is the API clients use — OpenAI's endpoints, Anthropic's,
and what llmgw adds — in OpenAPI 3.1, generated from internal/clientapi's
route table by the generator the admin API's document comes from, now
told each document's title, server, token and error. A test reads the
router's mux.HandleFunc calls and refuses a route described and not
mounted, or mounted and not described. Asked for before there is a key,
so it needs none.

/docs/api shows it in Swagger UI 5.33.1 (Apache 2.0, served from the
gateway as the page's policy requires): Authorize takes a key, Try it out
sends to this gateway. Staff are offered the admin API's document too,
from /docs/api/admin.json, which nobody else is given.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 20:22:48 +02:00
Ilker ManapandClaude Opus 5.5 7f320c5f65 v0.14.3: changelog, README, docs for 3D models; a finished job says done
Release / notes (push) Successful in 5s
CI / test (push) Failing after 6m50s
Release / build (push) Successful in 2m6s
A finished job kept the last word heard from ComfyUI, the node running;
it is now said done before the job is marked completed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 19:22:58 +02:00
Ilker ManapandClaude Opus 5.5 2f54780802 G4: a page that makes 3D models, and ctl 3d
/3d beside /images, while some model makes meshes: a picture chosen, the
job asked for through the same path an API client's takes, its progress
shown (waiting for the card, n ahead; making it, the node and percent),
and the mesh in a viewer — model-viewer 4.4.0 (Apache 2.0, with three.js
and lit), served from the gateway since the page's policy loads scripts
from nowhere else — with a download and the model's licence. The mesh is
served to its owner from /3d/jobs/{id}/file; another project's person is
told there is no such job.

A job made from the portal acted for a key that does not exist: the
worker now rebuilds a session's principal from the account that made it,
while it is active and still in the project the job is charged to.

ctl 3d list|get|cancel, over /api/v1/client-jobs (/api/v1/jobs is the
control plane's own work, which ctl jobs shows).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 19:16:20 +02:00
Ilker ManapandClaude Opus 5.5 4b68328e3e A mesh's resolution is the model's own: a multiple of 32 from 64 to 2048
TripoSR takes 256 by default; 512, 1024 and 1536 were TRELLIS.2's alone.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 18:47:52 +02:00