Ilker ManapandClaude Opus 5.5 73e1bb6733 Chat goes to the replica that has been answering soonest ...

The router keeps each endpoint's time to first token, smoothed over its last
streamed answers, and the chat profile scores by it beside session, queue,
KV and inflight: the fastest endpoint 1, one twice as slow 0.5, one not yet
measured 1 so a new replica is tried. Load still sees a slow drain from the
engine's own counters; this sees what answers actually took. BACKLOG L4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-25 17:01:30 +01:00

..

account

People sign in with their Active Directory account

2026-10-18 10:59:50 +02:00

accounting

A ComfyUI model may make meshes: from a picture, with its licence inside

2026-10-10 18:30:52 +02:00

agent

A node's console opens on what the machine is doing

2026-10-19 12:55:27 +02:00

audit

Guardrails turn off and on; a judge's refusal says which hazard; ctl guard judge ask

2026-10-06 12:41:08 +02:00

auth

Anthropic's Messages API: /v1/messages translated to chat completions and back, streamed or not, with tools and images; keys in x-api-key

2026-10-04 22:59:55 +02:00

bench

ctl bench reads a guarded engine's metrics with the engine's key, or says why it cannot

2026-10-05 00:20:25 +02:00

bundle

Offline sites and Kubernetes: llmgw bundle create, import and trust, offline.mirror, server init --bundle; a Helm chart for the control plane

2026-10-05 00:13:39 +02:00

canary

Canary routing: a model's canary takes its share of requests, a session stays on one side, x-llmgw-variant pins one, and the two are compared

2026-10-04 19:56:16 +02:00

cli

A machine too small for the models it would serve is said so first

2026-10-25 16:19:00 +01:00

clientapi

Pictures, 3D models, a vision model and workflows as tools at /mcp

2026-10-14 13:11:09 +02:00

clientip

Believe X-Forwarded-* only from router.trusted_proxies

2026-10-04 10:29:02 +02:00

comfy

ComfyUI models of one environment share one replica

2026-10-14 13:55:34 +02:00

config

Sign-in is set up from the console, and a directory tried before it is kept

2026-10-19 12:55:27 +02:00

control

The company's MCP tools under the guardrails, a rate and a breaker

2026-10-20 23:00:22 +02:00

directory

Sign-in is set up from the console, and a directory tried before it is kept

2026-10-19 12:55:27 +02:00

docgen

The reference describes a guardrail policy's services

2026-10-25 16:50:58 +01:00

doctor

A node's own readiness and its multi-node readiness are kept apart

2026-10-07 13:11:16 +02:00

firewall

The firewall setting reaches the mirror machine and the store's members

2026-10-09 08:53:17 +02:00

gguf

A model that does not fit its card is not added to fail; weights in use are not removed

2026-10-15 00:22:01 +02:00

guard

A guardrail policy can ask someone else's service

2026-10-25 16:49:56 +01:00

hf

Plan a repository's files against the fleet, and serve the one that suits it the way it fits

2026-10-04 01:03:48 +02:00

join

Bound idle time, header size and request reading on every listener, and connections on the client port

2026-10-04 10:21:15 +02:00

kb

llmgw-kb collections names each collection's owner as people know it

2026-10-08 15:42:40 +02:00

kvevents

Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held

2026-10-05 01:21:11 +02:00

mail

A forgotten password is set through a one-time link

2026-10-07 13:20:11 +02:00

mcpclient

An MCP client for the company's tool servers

2026-10-20 10:43:35 +02:00

mirror

The firewall setting reaches the mirror machine and the store's members

2026-10-09 08:53:17 +02:00

modelplan

A plan for a big model: experts spread, any number of nodes, memory said

2026-10-20 08:45:33 +02:00

notify

Several keys for one outside provider

2026-10-17 08:56:30 +02:00

obs

Tracing beyond the router: reconcile passes, placement and a replica's start traced down to the node

2026-10-05 00:03:17 +02:00

onboard

A node's own readiness and its multi-node readiness are kept apart

2026-10-07 13:11:16 +02:00

pgha

A lost store machine is given up on in seconds, and a new leader waits for its judge

2026-10-15 20:33:13 +02:00

preflight

A machine too small for the models it would serve is said so first

2026-10-25 16:19:00 +01:00

promq

Show usage over time, per organization, team, project and person

2026-09-28 08:11:31 +02:00

release

Offline sites and Kubernetes: llmgw bundle create, import and trust, offline.mirror, server init --bundle; a Helm chart for the control plane

2026-10-05 00:13:39 +02:00

router

Chat goes to the replica that has been answering soonest

2026-10-25 17:01:30 +01:00

safedial

gofmt what the security work left unformatted

2026-10-04 12:22:51 +02:00

scim

Match a mapped project however its path is spaced: the store writes "Acme / ml / train"

2026-10-04 16:45:47 +02:00

secrets

Sign-in is set up from the console, and a directory tried before it is kept

2026-10-19 12:55:27 +02:00

settings

The cluster's settings live in the store, and every change takes effect

2026-10-05 15:25:23 +02:00

sim

Onboard a node with a script, and find two examples that never worked

2026-09-16 12:28:36 +02:00

sourcecheck

Keep the control plane's fetches off link-local addresses, and see through NAT64, 6to4 and proxies

2026-10-04 10:48:01 +02:00

store

The company's MCP tools under the guardrails, a rate and a breaker

2026-10-20 23:00:22 +02:00

tlsconf

Put etcd and Patroni's API on TLS, and move a running cluster there in maintenance mode

2026-10-04 12:07:17 +02:00

version

Switch the project language to English

2026-09-16 01:15:21 +02:00

vscodeext

VS Code gets the gateway's models from an extension the gateway serves

2026-10-24 01:39:50 +02:00

web

A machine too small for the models it would serve is said so first

2026-10-25 16:19:00 +01:00