Ilker ManapandClaude Opus 5.5 bc4c5d5ac9 A judge replica that hangs is overtaken by another at half the budget ...

On the three-VM cluster about one answer in twenty-five came back as
content_filter: both judge replicas timed out within the same millisecond.
The tries shared the policy's 5 s budget, so the first replica, busy on an
oversubscribed machine, used it all and the retry on the second had nothing
left. A judge request now goes to a second replica too when the first has
not answered by half the budget, and the first answer is taken. The first
is not cut short.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-16 08:35:28 +02:00

..

admission

A band may wait longer than the rest; a busy refusal says when to return

2026-10-10 16:11:09 +02:00

breaker

Phase 1 (1/2): scheduler pipeline, endpoint registry, metric scraping

2026-09-16 01:15:21 +02:00

endpoint

ComfyUI models of one environment share one replica

2026-10-14 13:55:34 +02:00

kvindex

Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held

2026-10-05 01:21:11 +02:00

prefix

Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held

2026-10-05 01:21:11 +02:00

proxy

A ComfyUI model may make meshes: from a picture, with its licence inside

2026-10-10 18:30:52 +02:00

scheduler

Merge C24 KV-index routing from vLLM's cache events

2026-10-05 01:24:25 +02:00

scrape

Pictures from ComfyUI at /v1/images/generations

2026-10-09 21:15:50 +02:00

activate.go

A job waiting for the card says what holds it and about how long

2026-10-10 22:25:27 +02:00

adapter_test.go

LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

2026-10-05 01:14:38 +02:00

adapterowner_test.go

Trained adapters published, private to their owners like the knowledge bases they come from

2026-10-08 21:30:50 +02:00

admin_test.go

Give metrics a door that does not also open onto pprof

2026-09-17 18:12:10 +02:00

admin.go

Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held

2026-10-05 01:21:11 +02:00

adminauth_test.go

Guard the admin API and pprof with a token only root and the server can read

2026-10-04 10:02:44 +02:00

adminauth.go

Guard the admin API and pprof with a token only root and the server can read

2026-10-04 10:02:44 +02:00

adminguard_test.go

Stop a browser page, or a model definition, from running code on the nodes

2026-09-30 00:29:52 +02:00

adminguard.go

Stop a browser page, or a model definition, from running code on the nodes

2026-09-30 00:29:52 +02:00

audio.go

More of OpenAI's API: Responses structured output, images and previous_response_id; rerank, score and audio passed through; moderations by the judge

2026-10-04 22:56:25 +02:00

batches_test.go

Batches: OpenAI's Files and Batch APIs, served at the lowest priority under a store lease, each line a request of the key that made it

2026-10-04 21:00:01 +02:00

batches.go

Batches: OpenAI's Files and Batch APIs, served at the lowest priority under a store lease, each line a request of the key that made it

2026-10-04 21:00:01 +02:00

batchworker.go

Jobs are served by the leader, and a moment of fitting nowhere is waited out

2026-10-10 23:53:33 +02:00

budget_fallback.go

A batch of embeddings is checked text by text against the model's context

2026-10-09 11:43:21 +02:00

cache_test.go

Reusing answers: a response cache per project, exact match, opt-in with a TTL

2026-10-04 23:50:53 +02:00

cache.go

Collections a part of the tree answers from when a request names none

2026-10-08 15:52:43 +02:00

cachestream.go

Reusing answers: a response cache per project, exact match, opt-in with a TTL

2026-10-04 23:50:53 +02:00

canary_test.go

Canary routing: a model's canary takes its share of requests, a session stays on one side, x-llmgw-variant pins one, and the two are compared

2026-10-04 19:56:16 +02:00

canary.go

ComfyUI models of one environment share one replica

2026-10-14 13:55:34 +02:00

chat_load_test.go

Show a model loading in the chat, and send once it is up

2026-09-28 20:52:31 +02:00

chat_load.go

Show a model loading in the chat, and send once it is up

2026-09-28 20:52:31 +02:00

chat_models_test.go

The portal's chat answers from the documents a person picks

2026-10-08 15:59:23 +02:00

chat.go

A person leaves licences out, and no key of theirs reaches a model under one

2026-10-11 16:10:22 +02:00

compat_test.go

Moderations name a judge's categories once, by name

2026-10-04 23:03:35 +02:00

context_limit_test.go

A picture in a prompt counts as what a model makes of it, not its base64

2026-10-11 20:45:06 +02:00

context_limit.go

Know how big a model's context is, and what it can do

2026-09-16 23:07:49 +02:00

docs_test.go

A docs portal at /docs and a client developer's guide

2026-10-04 23:46:42 +02:00

external_test.go

Serve a model from an external OpenAI-compatible provider

2026-09-30 23:29:01 +02:00

federation_test.go

Federation: another llmgw cluster as a provider, its models as PEER/MODEL

2026-10-05 01:21:30 +02:00

federation.go

Federation: another llmgw cluster as a provider, its models as PEER/MODEL

2026-10-05 01:21:30 +02:00

files.go

MCP takes pictures of real size, by id too, and waits for a cold model

2026-10-15 07:24:59 +02:00

finish_test.go

A band may wait longer than the rest; a busy refusal says when to return

2026-10-10 16:11:09 +02:00

guard_test.go

A judge replica that hangs is overtaken by another at half the budget

2026-10-16 08:35:28 +02:00

guard.go

Guardrails turn off and on; a judge's refusal says which hazard; ctl guard judge ask

2026-10-06 12:41:08 +02:00

guardmetrics.go

Guardrail metrics and alerts: checks, latency, judge failures, fail-open, redactions, stopped streams, tenant blocks

2026-10-04 15:11:50 +02:00

handlers.go

A judge replica that hangs is overtaken by another at half the budget

2026-10-16 08:35:28 +02:00

hardware_leader_test.go

Add a standby control plane with server add, and draw the screens as windows

2026-10-03 22:29:12 +02:00

hardware_test.go

Say on the chat page whether a GPU or a CPU answered

2026-10-03 18:57:50 +02:00

images_test.go

G4: a page that makes 3D models, and ctl 3d

2026-10-10 19:16:20 +02:00

images.go

ComfyUI models of one environment share one replica

2026-10-14 13:55:34 +02:00

jobs_test.go

Jobs run side by side, one per model at a time

2026-10-14 10:06:20 +02:00

jobs.go

A person leaves licences out, and no key of theirs reaches a model under one

2026-10-11 16:10:22 +02:00

jobworker.go

Jobs run side by side, one per model at a time

2026-10-14 10:06:20 +02:00

judge.go

A judge replica that hangs is overtaken by another at half the budget

2026-10-16 08:35:28 +02:00

kb_e2e_test.go

llmgw-kb: sync a person's documents into a knowledge base from their own machine

2026-10-08 13:43:47 +02:00

kb_test.go

llmgw-kb collections names each collection's owner as people know it

2026-10-08 15:42:40 +02:00

kb.go

llmgw-kb collections names each collection's owner as people know it

2026-10-08 15:42:40 +02:00

kvroute_test.go

KV routing tests wait for both replicas to be scraped: a replica not yet heard from is passed over, and the test was racing that

2026-10-05 01:27:14 +02:00

kvroute.go

Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held

2026-10-05 01:21:11 +02:00

language_test.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

licences_test.go

A licence left out by its name stays out once its entry gives an id

2026-10-11 16:28:40 +02:00

licences.go

A licence left out by its name stays out once its entry gives an id

2026-10-11 16:28:40 +02:00

lora_test.go

LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

2026-10-05 01:14:38 +02:00

lora.go

LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

2026-10-05 01:14:38 +02:00

mcp_media_test.go

MCP takes pictures of real size, by id too, and waits for a cold model

2026-10-15 07:24:59 +02:00

mcp_media.go

MCP takes pictures of real size, by id too, and waits for a cold model

2026-10-15 07:24:59 +02:00

memory.go

MCP takes pictures of real size, by id too, and waits for a cold model

2026-10-15 07:24:59 +02:00

meshes_test.go

A mesh carries its whole licence, not only its name

2026-10-14 01:12:47 +02:00

meshes.go

ComfyUI models of one environment share one replica

2026-10-14 13:55:34 +02:00

messages_test.go

Extended thinking on /v1/messages: a model's reasoning as thinking blocks

2026-10-14 11:05:02 +02:00

messages.go

Extended thinking on /v1/messages: a model's reasoning as thinking blocks

2026-10-14 11:05:02 +02:00

metrics.go

Chat requests answered from knowledge bases

2026-10-08 15:27:50 +02:00

moderations.go

Moderations name a judge's categories once, by name

2026-10-04 23:03:35 +02:00

objglb.go

A job says how far along it is; a ComfyUI model brings its custom nodes

2026-10-10 18:46:44 +02:00

pages_auth_test.go

The client API described in OpenAPI, and both APIs in Swagger UI

2026-10-10 20:22:48 +02:00

pricing_test.go

Publish per-tenant request and budget series, and alert on them

2026-10-03 19:55:24 +02:00

quality_test.go

Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

2026-10-05 00:04:43 +02:00

rag_test.go

An embedding model's query_prefix, put before questions and never before documents

2026-10-09 18:03:00 +02:00

rag.go

A picture in a prompt counts as what a model makes of it, not its base64

2026-10-11 20:45:06 +02:00

requestlog_test.go

Requests kept and replayed: keys requests per project with retention and a share, ctl request --body and --replay

2026-10-04 22:51:21 +02:00

requestlog.go

Reusing answers: a response cache per project, exact match, opt-in with a TTL

2026-10-04 23:50:53 +02:00

residency_test.go

Forgetting a person and keeping requests in the cluster: users forget, keys residency --in-cluster

2026-10-04 23:27:31 +02:00

responses_test.go

Hook a guard into the router: judge requests before dispatch and filter answers as they are relayed

2026-10-04 12:20:24 +02:00

responses.go

More of OpenAI's API: Responses structured output, images and previous_response_id; rerank, score and audio passed through; moderations by the judge

2026-10-04 22:56:25 +02:00

router_listen_test.go

Release v0.10.4: CPU threads, tracing, and the rest of the review

2026-09-30 04:00:35 +02:00

router_test.go

A judge replica that hangs is overtaken by another at half the budget

2026-10-16 08:35:28 +02:00

router.go

A judge replica that hangs is overtaken by another at half the budget

2026-10-16 08:35:28 +02:00

tags_test.go

Requests can be tagged: metadata, x-llmgw-tags and the user field on the usage record, and counted by the tags router.tag_labels names

2026-10-04 19:45:37 +02:00

tags.go

A job waiting for the card says what holds it and about how long

2026-10-10 22:25:27 +02:00

tenantmetrics.go

Requests can be tagged: metadata, x-llmgw-tags and the user field on the usage record, and counted by the tags router.tag_labels names

2026-10-04 19:45:37 +02:00

threadid_test.go

Guard a GPU-less node's problems, and bound thread ids

2026-10-04 10:49:10 +02:00

threads_test.go

Stop the text-search comment naming one language

2026-09-27 08:09:49 +02:00

threads.go

A picture in a prompt counts as what a model makes of it, not its base64

2026-10-11 20:45:06 +02:00

tokenmetrics_test.go

Token-level timing and admission by SLO: TPOT, inter-token and tokens a second by model and tenant; a model past its SLO is saturated for the lower bands

2026-10-04 23:13:18 +02:00

tokenmetrics.go

Token-level timing and admission by SLO: TPOT, inter-token and tokens a second by model and tenant; a model past its SLO is saturated for the lower bands

2026-10-04 23:13:18 +02:00

trace_test.go

Release v0.10.4: CPU threads, tracing, and the rest of the review

2026-09-30 04:00:35 +02:00

trace.go

Release v0.10.4: CPU threads, tracing, and the rest of the review

2026-09-30 04:00:35 +02:00

workflows_test.go

Any ComfyUI workflow runs as a job, from the inputs its catalog entry declares

2026-10-11 14:28:15 +02:00

workflows.go

ComfyUI models of one environment share one replica

2026-10-14 13:55:34 +02:00