llmgw

admission

Displace the newest lowest-band request when the gateway queue is full

2026-09-30 23:07:53 +02:00

breaker

Phase 1 (1/2): scheduler pipeline, endpoint registry, metric scraping

2026-09-16 01:15:21 +02:00

endpoint

Serve a model from an external OpenAI-compatible provider

2026-09-30 23:29:01 +02:00

prefix

Add approximate prefix routing, and find out it did nothing

2026-09-16 12:13:19 +02:00

proxy

Token-level timing and admission by SLO: TPOT, inter-token and tokens a second by model and tenant; a model past its SLO is saturated for the lower bands

2026-10-04 23:13:18 +02:00

scheduler

Spill a saturated pool's requests to the next pool

2026-09-30 23:23:32 +02:00

scrape

TensorRT-LLM as an engine, and models from Ollama into the mirror

2026-10-05 00:02:01 +02:00

activate.go

Notifications: webhooks for a parked model, a lost node, a spent budget and new weights; Alertmanager receiver templates

2026-10-04 16:58:11 +02:00

adapter_test.go

Serve LoRA adapters on llama.cpp models, asked for by name

2026-09-30 23:40:47 +02:00

admin_test.go

Give metrics a door that does not also open onto pprof

2026-09-17 18:12:10 +02:00

admin.go

Give metrics a door that does not also open onto pprof

2026-09-17 18:12:10 +02:00

adminauth_test.go

Guard the admin API and pprof with a token only root and the server can read

2026-10-04 10:02:44 +02:00

adminauth.go

Guard the admin API and pprof with a token only root and the server can read

2026-10-04 10:02:44 +02:00

adminguard_test.go

Stop a browser page, or a model definition, from running code on the nodes

2026-09-30 00:29:52 +02:00

adminguard.go

Stop a browser page, or a model definition, from running code on the nodes

2026-09-30 00:29:52 +02:00

audio.go

More of OpenAI's API: Responses structured output, images and previous_response_id; rerank, score and audio passed through; moderations by the judge

2026-10-04 22:56:25 +02:00

batches_test.go

Batches: OpenAI's Files and Batch APIs, served at the lowest priority under a store lease, each line a request of the key that made it

2026-10-04 21:00:01 +02:00

batches.go

Batches: OpenAI's Files and Batch APIs, served at the lowest priority under a store lease, each line a request of the key that made it

2026-10-04 21:00:01 +02:00

batchworker.go

Batches: OpenAI's Files and Batch APIs, served at the lowest priority under a store lease, each line a request of the key that made it

2026-10-04 21:00:01 +02:00

budget_fallback.go

Price a request by the model that serves it

2026-09-28 00:15:40 +02:00

cache_test.go

Reusing answers: a response cache per project, exact match, opt-in with a TTL

2026-10-04 23:50:53 +02:00

cache.go

Reusing answers: a response cache per project, exact match, opt-in with a TTL

2026-10-04 23:50:53 +02:00

cachestream.go

Reusing answers: a response cache per project, exact match, opt-in with a TTL

2026-10-04 23:50:53 +02:00

canary_test.go

Canary routing: a model's canary takes its share of requests, a session stays on one side, x-llmgw-variant pins one, and the two are compared

2026-10-04 19:56:16 +02:00

canary.go

Canary routing: a model's canary takes its share of requests, a session stays on one side, x-llmgw-variant pins one, and the two are compared

2026-10-04 19:56:16 +02:00

chat_load_test.go

Show a model loading in the chat, and send once it is up

2026-09-28 20:52:31 +02:00

chat_load.go

Show a model loading in the chat, and send once it is up

2026-09-28 20:52:31 +02:00

chat_models_test.go

Let the chat be stopped, timed, tuned and kept across a reload

2026-09-29 09:54:23 +02:00

chat.go

Say on the chat page whether a GPU or a CPU answered

2026-10-03 18:57:50 +02:00

compat_test.go

Moderations name a judge's categories once, by name

2026-10-04 23:03:35 +02:00

context_limit_test.go

Know how big a model's context is, and what it can do

2026-09-16 23:07:49 +02:00

context_limit.go

Know how big a model's context is, and what it can do

2026-09-16 23:07:49 +02:00

docs_test.go

A docs portal at /docs and a client developer's guide

2026-10-04 23:46:42 +02:00

external_test.go

Serve a model from an external OpenAI-compatible provider

2026-09-30 23:29:01 +02:00

files.go

Batches: OpenAI's Files and Batch APIs, served at the lowest priority under a store lease, each line a request of the key that made it

2026-10-04 21:00:01 +02:00

finish_test.go

Displace the newest lowest-band request when the gateway queue is full

2026-09-30 23:07:53 +02:00

guard_test.go

Let /login/oidc through without an API key: /login alone was exempt

2026-10-04 16:37:24 +02:00

guard.go

More of OpenAI's API: Responses structured output, images and previous_response_id; rerank, score and audio passed through; moderations by the judge

2026-10-04 22:56:25 +02:00

guardmetrics.go

Guardrail metrics and alerts: checks, latency, judge failures, fail-open, redactions, stopped streams, tenant blocks

2026-10-04 15:11:50 +02:00

handlers.go

Merge features-c2: the docs portal and its links in English and Turkish

2026-10-05 00:15:25 +02:00

hardware_leader_test.go

Add a standby control plane with server add, and draw the screens as windows

2026-10-03 22:29:12 +02:00

hardware_test.go

Say on the chat page whether a GPU or a CPU answered

2026-10-03 18:57:50 +02:00

judge.go

Judge an answer with Llama Guard 3's own prompt, as a completion, not a conversation ending on the answer

2026-10-04 15:35:57 +02:00

language_test.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

memory.go

no_store: refuse conversations, notes and models outside the cluster under a policy that keeps nothing

2026-10-04 12:36:02 +02:00

messages_test.go

Anthropic's Messages API: /v1/messages translated to chat completions and back, streamed or not, with tools and images; keys in x-api-key

2026-10-04 22:59:55 +02:00

messages.go

Anthropic's Messages API: /v1/messages translated to chat completions and back, streamed or not, with tools and images; keys in x-api-key

2026-10-04 22:59:55 +02:00

metrics.go

Reusing answers: a response cache per project, exact match, opt-in with a TTL

2026-10-04 23:50:53 +02:00

moderations.go

Moderations name a judge's categories once, by name

2026-10-04 23:03:35 +02:00

pages_auth_test.go

Connect an editor from the account page, and serve the Responses API

2026-09-29 10:04:17 +02:00

pricing_test.go

Publish per-tenant request and budget series, and alert on them

2026-10-03 19:55:24 +02:00

quality_test.go

Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

2026-10-05 00:04:43 +02:00

requestlog_test.go

Requests kept and replayed: keys requests per project with retention and a share, ctl request --body and --replay

2026-10-04 22:51:21 +02:00

requestlog.go

Reusing answers: a response cache per project, exact match, opt-in with a TTL

2026-10-04 23:50:53 +02:00

residency_test.go

Forgetting a person and keeping requests in the cluster: users forget, keys residency --in-cluster

2026-10-04 23:27:31 +02:00

responses_test.go

Hook a guard into the router: judge requests before dispatch and filter answers as they are relayed

2026-10-04 12:20:24 +02:00

responses.go

More of OpenAI's API: Responses structured output, images and previous_response_id; rerank, score and audio passed through; moderations by the judge

2026-10-04 22:56:25 +02:00

router_listen_test.go

Release v0.10.4: CPU threads, tracing, and the rest of the review

2026-09-30 04:00:35 +02:00

router_test.go

List in /v1/models only the models the key may use

2026-10-04 10:49:54 +02:00

router.go

Merge features-c2: the docs portal and its links in English and Turkish

2026-10-05 00:15:25 +02:00

tags_test.go

Requests can be tagged: metadata, x-llmgw-tags and the user field on the usage record, and counted by the tags router.tag_labels names

2026-10-04 19:45:37 +02:00

tags.go

Requests can be tagged: metadata, x-llmgw-tags and the user field on the usage record, and counted by the tags router.tag_labels names

2026-10-04 19:45:37 +02:00

tenantmetrics.go

Requests can be tagged: metadata, x-llmgw-tags and the user field on the usage record, and counted by the tags router.tag_labels names

2026-10-04 19:45:37 +02:00

threadid_test.go

Guard a GPU-less node's problems, and bound thread ids

2026-10-04 10:49:10 +02:00

threads_test.go

Stop the text-search comment naming one language

2026-09-27 08:09:49 +02:00

threads.go

Requests kept and replayed: keys requests per project with retention and a share, ctl request --body and --replay

2026-10-04 22:51:21 +02:00

tokenmetrics_test.go

Token-level timing and admission by SLO: TPOT, inter-token and tokens a second by model and tenant; a model past its SLO is saturated for the lower bands

2026-10-04 23:13:18 +02:00

tokenmetrics.go

Token-level timing and admission by SLO: TPOT, inter-token and tokens a second by model and tenant; a model past its SLO is saturated for the lower bands

2026-10-04 23:13:18 +02:00

trace_test.go

Release v0.10.4: CPU threads, tracing, and the rest of the review

2026-09-30 04:00:35 +02:00

trace.go

Release v0.10.4: CPU threads, tracing, and the rest of the review

2026-09-30 04:00:35 +02:00