Ilker ManapandClaude Opus 5.5 0b984a92bc A judge is asked again only on another machine ...

Measured under CPU load on the three-VM cluster, with both judge replicas on
ha1: 37 of 60 requests failed with the hedge, 38 of 60 without it. Two
replicas on one saturated machine are slow together, and the second
question doubled the work there. The hedge now goes only to a replica whose
address names another host.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-16 10:27:50 +02:00

..

admission

A band may wait longer than the rest; a busy refusal says when to return

2026-10-10 16:11:09 +02:00

breaker

Phase 1 (1/2): scheduler pipeline, endpoint registry, metric scraping

2026-09-16 01:15:21 +02:00

endpoint

ComfyUI models of one environment share one replica

2026-10-14 13:55:34 +02:00

kvindex

Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held

2026-10-05 01:21:11 +02:00

prefix

Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held

2026-10-05 01:21:11 +02:00

proxy

A ComfyUI model may make meshes: from a picture, with its licence inside

2026-10-10 18:30:52 +02:00

scheduler

Merge C24 KV-index routing from vLLM's cache events

2026-10-05 01:24:25 +02:00

scrape

Pictures from ComfyUI at /v1/images/generations

2026-10-09 21:15:50 +02:00

activate.go

A job waiting for the card says what holds it and about how long

2026-10-10 22:25:27 +02:00

adapter_test.go

LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

2026-10-05 01:14:38 +02:00

adapterowner_test.go

Trained adapters published, private to their owners like the knowledge bases they come from

2026-10-08 21:30:50 +02:00

admin_test.go

Give metrics a door that does not also open onto pprof

2026-09-17 18:12:10 +02:00

admin.go

Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held

2026-10-05 01:21:11 +02:00

adminauth_test.go

Guard the admin API and pprof with a token only root and the server can read

2026-10-04 10:02:44 +02:00

adminauth.go

Guard the admin API and pprof with a token only root and the server can read

2026-10-04 10:02:44 +02:00

adminguard_test.go

Stop a browser page, or a model definition, from running code on the nodes

2026-09-30 00:29:52 +02:00

adminguard.go

Stop a browser page, or a model definition, from running code on the nodes

2026-09-30 00:29:52 +02:00

audio.go

More of OpenAI's API: Responses structured output, images and previous_response_id; rerank, score and audio passed through; moderations by the judge

2026-10-04 22:56:25 +02:00

batches_test.go

Batches: OpenAI's Files and Batch APIs, served at the lowest priority under a store lease, each line a request of the key that made it

2026-10-04 21:00:01 +02:00

batches.go

Batches: OpenAI's Files and Batch APIs, served at the lowest priority under a store lease, each line a request of the key that made it

2026-10-04 21:00:01 +02:00

batchworker.go

Jobs are served by the leader, and a moment of fitting nowhere is waited out

2026-10-10 23:53:33 +02:00

budget_fallback.go

A batch of embeddings is checked text by text against the model's context

2026-10-09 11:43:21 +02:00

cache_test.go

Reusing answers: a response cache per project, exact match, opt-in with a TTL

2026-10-04 23:50:53 +02:00

cache.go

Collections a part of the tree answers from when a request names none

2026-10-08 15:52:43 +02:00

cachestream.go

Reusing answers: a response cache per project, exact match, opt-in with a TTL

2026-10-04 23:50:53 +02:00

canary_test.go

Canary routing: a model's canary takes its share of requests, a session stays on one side, x-llmgw-variant pins one, and the two are compared

2026-10-04 19:56:16 +02:00

canary.go

ComfyUI models of one environment share one replica

2026-10-14 13:55:34 +02:00

chat_load_test.go

Show a model loading in the chat, and send once it is up

2026-09-28 20:52:31 +02:00

chat_load.go

Show a model loading in the chat, and send once it is up

2026-09-28 20:52:31 +02:00

chat_models_test.go

The portal's chat answers from the documents a person picks

2026-10-08 15:59:23 +02:00

chat.go

A person leaves licences out, and no key of theirs reaches a model under one

2026-10-11 16:10:22 +02:00

compat_test.go

Moderations name a judge's categories once, by name

2026-10-04 23:03:35 +02:00

context_limit_test.go

A picture in a prompt counts as what a model makes of it, not its base64

2026-10-11 20:45:06 +02:00

context_limit.go

Know how big a model's context is, and what it can do

2026-09-16 23:07:49 +02:00

docs_test.go

A docs portal at /docs and a client developer's guide

2026-10-04 23:46:42 +02:00

external_test.go

Serve a model from an external OpenAI-compatible provider

2026-09-30 23:29:01 +02:00

federation_test.go

Federation: another llmgw cluster as a provider, its models as PEER/MODEL

2026-10-05 01:21:30 +02:00

federation.go

Federation: another llmgw cluster as a provider, its models as PEER/MODEL

2026-10-05 01:21:30 +02:00

files.go

MCP takes pictures of real size, by id too, and waits for a cold model

2026-10-15 07:24:59 +02:00

finish_test.go

A band may wait longer than the rest; a busy refusal says when to return

2026-10-10 16:11:09 +02:00

guard_test.go

A judge is asked again only on another machine

2026-10-16 10:27:50 +02:00

guard.go

Guardrails turn off and on; a judge's refusal says which hazard; ctl guard judge ask

2026-10-06 12:41:08 +02:00

guardmetrics.go

Guardrail metrics and alerts: checks, latency, judge failures, fail-open, redactions, stopped streams, tenant blocks

2026-10-04 15:11:50 +02:00

handlers.go

A judge replica that hangs is overtaken by another at half the budget

2026-10-16 08:35:28 +02:00

hardware_leader_test.go

Add a standby control plane with server add, and draw the screens as windows

2026-10-03 22:29:12 +02:00

hardware_test.go

Say on the chat page whether a GPU or a CPU answered

2026-10-03 18:57:50 +02:00

images_test.go

G4: a page that makes 3D models, and ctl 3d

2026-10-10 19:16:20 +02:00

images.go

ComfyUI models of one environment share one replica

2026-10-14 13:55:34 +02:00

jobs_test.go

Jobs run side by side, one per model at a time

2026-10-14 10:06:20 +02:00

jobs.go

A person leaves licences out, and no key of theirs reaches a model under one

2026-10-11 16:10:22 +02:00

jobworker.go

Jobs run side by side, one per model at a time

2026-10-14 10:06:20 +02:00

judge.go

A judge is asked again only on another machine

2026-10-16 10:27:50 +02:00

kb_e2e_test.go

llmgw-kb: sync a person's documents into a knowledge base from their own machine

2026-10-08 13:43:47 +02:00

kb_test.go

llmgw-kb collections names each collection's owner as people know it

2026-10-08 15:42:40 +02:00

kb.go

llmgw-kb collections names each collection's owner as people know it

2026-10-08 15:42:40 +02:00

kvroute_test.go

KV routing tests wait for both replicas to be scraped: a replica not yet heard from is passed over, and the test was racing that

2026-10-05 01:27:14 +02:00

kvroute.go

Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held

2026-10-05 01:21:11 +02:00

language_test.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

licences_test.go

A licence left out by its name stays out once its entry gives an id

2026-10-11 16:28:40 +02:00

licences.go

A licence left out by its name stays out once its entry gives an id

2026-10-11 16:28:40 +02:00

lora_test.go

LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

2026-10-05 01:14:38 +02:00

lora.go

LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

2026-10-05 01:14:38 +02:00

mcp_media_test.go

MCP takes pictures of real size, by id too, and waits for a cold model

2026-10-15 07:24:59 +02:00

mcp_media.go

MCP takes pictures of real size, by id too, and waits for a cold model

2026-10-15 07:24:59 +02:00

memory.go

MCP takes pictures of real size, by id too, and waits for a cold model

2026-10-15 07:24:59 +02:00

meshes_test.go

A mesh carries its whole licence, not only its name

2026-10-14 01:12:47 +02:00

meshes.go

ComfyUI models of one environment share one replica

2026-10-14 13:55:34 +02:00

messages_test.go

Extended thinking on /v1/messages: a model's reasoning as thinking blocks

2026-10-14 11:05:02 +02:00

messages.go

Extended thinking on /v1/messages: a model's reasoning as thinking blocks

2026-10-14 11:05:02 +02:00

metrics.go

Chat requests answered from knowledge bases

2026-10-08 15:27:50 +02:00

moderations.go

Moderations name a judge's categories once, by name

2026-10-04 23:03:35 +02:00

objglb.go

A job says how far along it is; a ComfyUI model brings its custom nodes

2026-10-10 18:46:44 +02:00

pages_auth_test.go

The client API described in OpenAPI, and both APIs in Swagger UI

2026-10-10 20:22:48 +02:00

pricing_test.go

Publish per-tenant request and budget series, and alert on them

2026-10-03 19:55:24 +02:00

quality_test.go

Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

2026-10-05 00:04:43 +02:00

rag_test.go

An embedding model's query_prefix, put before questions and never before documents

2026-10-09 18:03:00 +02:00

rag.go

A picture in a prompt counts as what a model makes of it, not its base64

2026-10-11 20:45:06 +02:00

requestlog_test.go

Requests kept and replayed: keys requests per project with retention and a share, ctl request --body and --replay

2026-10-04 22:51:21 +02:00

requestlog.go

Reusing answers: a response cache per project, exact match, opt-in with a TTL

2026-10-04 23:50:53 +02:00

residency_test.go

Forgetting a person and keeping requests in the cluster: users forget, keys residency --in-cluster

2026-10-04 23:27:31 +02:00

responses_test.go

Hook a guard into the router: judge requests before dispatch and filter answers as they are relayed

2026-10-04 12:20:24 +02:00

responses.go

More of OpenAI's API: Responses structured output, images and previous_response_id; rerank, score and audio passed through; moderations by the judge

2026-10-04 22:56:25 +02:00

router_listen_test.go

Release v0.10.4: CPU threads, tracing, and the rest of the review

2026-09-30 04:00:35 +02:00

router_test.go

A judge replica that hangs is overtaken by another at half the budget

2026-10-16 08:35:28 +02:00

router.go

A judge replica that hangs is overtaken by another at half the budget

2026-10-16 08:35:28 +02:00

tags_test.go

Requests can be tagged: metadata, x-llmgw-tags and the user field on the usage record, and counted by the tags router.tag_labels names

2026-10-04 19:45:37 +02:00

tags.go

A job waiting for the card says what holds it and about how long

2026-10-10 22:25:27 +02:00

tenantmetrics.go

Requests can be tagged: metadata, x-llmgw-tags and the user field on the usage record, and counted by the tags router.tag_labels names

2026-10-04 19:45:37 +02:00

threadid_test.go

Guard a GPU-less node's problems, and bound thread ids

2026-10-04 10:49:10 +02:00

threads_test.go

Stop the text-search comment naming one language

2026-09-27 08:09:49 +02:00

threads.go

A picture in a prompt counts as what a model makes of it, not its base64

2026-10-11 20:45:06 +02:00

tokenmetrics_test.go

Token-level timing and admission by SLO: TPOT, inter-token and tokens a second by model and tenant; a model past its SLO is saturated for the lower bands

2026-10-04 23:13:18 +02:00

tokenmetrics.go

Token-level timing and admission by SLO: TPOT, inter-token and tokens a second by model and tenant; a model past its SLO is saturated for the lower bands

2026-10-04 23:13:18 +02:00

trace_test.go

Release v0.10.4: CPU threads, tracing, and the rest of the review

2026-09-30 04:00:35 +02:00

trace.go

Release v0.10.4: CPU threads, tracing, and the rest of the review

2026-09-30 04:00:35 +02:00

workflows_test.go

Any ComfyUI workflow runs as a job, from the inputs its catalog entry declares

2026-10-11 14:28:15 +02:00

workflows.go

ComfyUI models of one environment share one replica

2026-10-14 13:55:34 +02:00