Ilker ManapandClaude Opus 5.5 688b205604 What llmgw adds to a request, measured: ctl bench overhead ...

An engine that answers at once and the gateway in front of it, in one
process on loopback, given the same load straight and through the gateway:
bare, with authentication and accounting, and with the guardrails' pattern
checks. It reports p50 and p99, what the gateway added, and a rough CPU
per thousand requests.

On an Apple M5 at 100 to 2,000 requests a second (10 s a step, 72,000
requests, none failed), the gateway added 0.05–0.42 ms at p50 and at most
0.49 ms at p99. That is now in the README, with how it was measured and
why the CPU figure is rough. CI runs a one-second version.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-17 00:23:26 +02:00

..

admintoken_test.go

Guard the admin API and pprof with a token only root and the server can read

2026-10-04 10:02:44 +02:00

admintoken.go

Guard the admin API and pprof with a token only root and the server can read

2026-10-04 10:02:44 +02:00

agent.go

The nodes' agent settings kept with the cluster's, agent.yaml down to what an agent needs to start

2026-10-08 18:06:36 +02:00

answercache_test.go

Reusing answers: a response cache per project, exact match, opt-in with a TTL

2026-10-04 23:50:53 +02:00

answercache.go

Reusing answers: a response cache per project, exact match, opt-in with a TTL

2026-10-04 23:50:53 +02:00

audit.go

Audit log: every change through the admin API and the administration pages, with ctl audit

2026-10-04 16:25:56 +02:00

backup_test.go

Backups reach a replicated store: pg_dump is given the cluster's authority with verify-full

2026-10-08 21:04:46 +02:00

backup.go

Backups reach a replicated store: pg_dump is given the cluster's authority with verify-full

2026-10-08 21:04:46 +02:00

backupcopy.go

server restore: an archive checked, dry-run, and put back — configuration with modes and owners, the store in SQLite or PostgreSQL; backup_copy_to copies archives off the machine

2026-10-04 23:09:59 +02:00

bench_overhead_test.go

What llmgw adds to a request, measured: ctl bench overhead

2026-10-17 00:23:26 +02:00

bench_overhead.go

What llmgw adds to a request, measured: ctl bench overhead

2026-10-17 00:23:26 +02:00

bench_test.go

ctl bench reads a guarded engine's metrics with the engine's key, or says why it cannot

2026-10-05 00:20:25 +02:00

bench.go

What llmgw adds to a request, measured: ctl bench overhead

2026-10-17 00:23:26 +02:00

bundle.go

The cluster's settings live in the store, and every change takes effect

2026-10-05 15:25:23 +02:00

capacity.go

Keep what a benchmark found out, instead of leaving it in a terminal

2026-09-27 11:51:58 +02:00

cert.go

Add a private CA and mutual TLS for the agent stream

2026-09-16 07:28:56 +02:00

clustersettings_test.go

The cluster's settings live in the store, and every change takes effect

2026-10-05 15:25:23 +02:00

clustersettings.go

How long an on-demand model holds a card is a live setting

2026-10-14 08:58:38 +02:00

comfyuigate.go

ComfyUI as an engine of the cluster's own

2026-10-10 00:39:10 +02:00

controlplanes_test.go

The probe reads the floating address from its file each time

2026-10-05 02:56:23 +02:00

controlplanes.go

The probe reads the floating address from its file each time

2026-10-05 02:56:23 +02:00

ctl_3d.go

G4: a page that makes 3D models, and ctl 3d

2026-10-10 19:16:20 +02:00

ctl_config_test.go

The guardrails' judge has its own command, and its trouble is shown

2026-10-06 10:47:37 +02:00

ctl_config.go

The cluster's settings live in the store, and every change takes effect

2026-10-05 15:25:23 +02:00

ctl_eval_test.go

Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

2026-10-05 00:04:43 +02:00

ctl_eval.go

Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

2026-10-05 00:04:43 +02:00

ctl_fabrictest.go

Measure two nodes' fabric rail by rail: llmgw ctl fabric-test

2026-09-29 21:29:23 +02:00

ctl_logs.go

Show a node's logs on the admin page

2026-09-28 12:27:47 +02:00

ctl_request.go

A ComfyUI model may make meshes: from a picture, with its licence inside

2026-10-10 18:30:52 +02:00

ctl_restart.go

ctl restart --max-surge and --max-unavailable

2026-10-07 13:09:10 +02:00

ctl_status_test.go

Draw the console's overview in its panel, not over the screen

2026-10-03 23:09:40 +02:00

ctl_upgrade_test.go

An upgrade that leaves nodes behind says which

2026-10-14 09:01:09 +02:00

ctl_upgrade.go

An upgrade that leaves nodes behind says which

2026-10-14 09:01:09 +02:00

ctl.go

G4: a page that makes 3D models, and ctl 3d

2026-10-10 19:16:20 +02:00

ctlweights.go

Carry a download's progress to where somebody is looking at it

2026-09-27 07:49:08 +02:00

dispatcherref_test.go

Measure two nodes' fabric rail by rail: llmgw ctl fabric-test

2026-09-29 21:29:23 +02:00

doctor.go

A node's own readiness and its multi-node readiness are kept apart

2026-10-07 13:11:16 +02:00

enginebuild.go

Choose each engine's build from the node's hardware, for every GPU generation

2026-10-01 14:35:58 +02:00

enginecheck_gpu_test.go

An adopted engine's state is written without its command; engine-check finds the installed build

2026-10-07 14:26:28 +02:00

enginecheck_test.go

SGLang read in its own metric names, started serving them, and asked for its own help

2026-10-05 00:20:25 +02:00

enginecheck.go

An adopted engine's state is written without its command; engine-check finds the installed build

2026-10-07 14:26:28 +02:00

facility_test.go

Facility-aware placement: racks, PDUs, holds and power caps

2026-10-05 01:17:49 +02:00

facility.go

Facility-aware placement: racks, PDUs, holds and power caps

2026-10-05 01:17:49 +02:00

federation.go

Federation: another llmgw cluster as a provider, its models as PEER/MODEL

2026-10-05 01:21:30 +02:00

filter_test.go

Let ctl ask a question instead of printing the whole fleet

2026-09-27 07:30:51 +02:00

filter.go

Facility-aware placement: racks, PDUs, holds and power caps

2026-10-05 01:17:49 +02:00

firewallcmd_test.go

The firewall setting reaches the mirror machine and the store's members

2026-10-09 08:53:17 +02:00

firewallcmd.go

The firewall setting reaches the mirror machine and the store's members

2026-10-09 08:53:17 +02:00

firewallpub_test.go

The firewall setting reaches the mirror machine and the store's members

2026-10-09 08:53:17 +02:00

firewallpub.go

The firewall setting reaches the mirror machine and the store's members

2026-10-09 08:53:17 +02:00

guard_judge.go

guard judge says controversial, and names a judge no policy asks as one

2026-10-08 11:42:55 +02:00

guard.go

Guardrails turn off and on; a judge's refusal says which hazard; ctl guard judge ask

2026-10-06 12:41:08 +02:00

join.go

Offline sites and Kubernetes: llmgw bundle create, import and trust, offline.mirror, server init --bundle; a Helm chart for the control plane

2026-10-05 00:13:39 +02:00

judgekeeper_test.go

An upgrade refits the judge an earlier server init installed

2026-10-16 22:06:56 +02:00

judgekeeper.go

An upgrade refits the judge an earlier server init installed

2026-10-16 22:06:56 +02:00

judges_known.go

Qwen3Guard is the judge, and judges on a GPU and on CPUs are asked in turn

2026-10-08 10:22:47 +02:00

kb.go

Training data from a collection: llmgw ctl kb dataset

2026-10-08 20:04:37 +02:00

keys_lifecycle_test.go

Key lifecycle: lifetime limits by scope, rotation with overlap, kinds and expiry in the listing

2026-10-04 16:45:05 +02:00

keys.go

Reusing answers: a response cache per project, exact match, opt-in with a TTL

2026-10-04 23:50:53 +02:00

leaderforward_test.go

A standby's portal is the leader's; the changelog as the release notes read it

2026-10-05 06:34:57 +02:00

leaderforward.go

A standby's portal is the leader's; the changelog as the release notes read it

2026-10-05 06:34:57 +02:00

memory.go

Manage people and the tenancy tree, from the web and the CLI

2026-09-28 16:26:39 +02:00

mig_test.go

MIG: a card in MIG mode is placed as its instances, each a GPU of class <card>-mig-<profile>

2026-10-05 00:01:14 +02:00

mirror_gate_test.go

Release v0.10.4: CPU threads, tracing, and the rest of the review

2026-09-30 04:00:35 +02:00

mirror_import_test.go

TensorRT-LLM as an engine, and models from Ollama into the mirror

2026-10-05 00:02:01 +02:00

mirror_import.go

TensorRT-LLM as an engine, and models from Ollama into the mirror

2026-10-05 00:02:01 +02:00

mirror_test.go

Hide the mirror's bookkeeping, bound GGUF array nesting, and forget old serve jobs and downloads

2026-10-04 10:56:16 +02:00

mirror.go

TensorRT-LLM as an engine, and models from Ollama into the mirror

2026-10-05 00:02:01 +02:00

mirrorinit.go

The firewall setting reaches the mirror machine and the store's members

2026-10-09 08:53:17 +02:00

model_canary.go

Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

2026-10-05 00:04:43 +02:00

model_presets_test.go

The cluster page and the console say which picture and 3D models are here, and add them

2026-10-13 20:48:40 +02:00

model_presets.go

The cluster page and the console say which picture and 3D models are here, and add them

2026-10-13 20:48:40 +02:00

model_serve.go

An embedding model is served for embedding, with many slots

2026-10-14 09:19:57 +02:00

model_update.go

Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

2026-10-05 00:04:43 +02:00

model.go

The cluster page and the console say which picture and 3D models are here, and add them

2026-10-13 20:48:40 +02:00

modelplan.go

MIG: a card in MIG mode is placed as its instances, each a GPU of class <card>-mig-<profile>

2026-10-05 00:01:14 +02:00

movestore.go

Move a store to another one without reissuing every key

2026-09-25 16:28:38 +02:00

node.go

The mirror's address is one the nodes reach, and node add checks it

2026-10-06 09:36:09 +02:00

notify.go

Notifications: webhooks for a parked model, a lost node, a spent budget and new weights; Alertmanager receiver templates

2026-10-04 16:58:11 +02:00

org_test.go

Let a person belong to many places, or to none

2026-09-28 19:03:46 +02:00

org.go

Let a person belong to many places, or to none

2026-09-28 19:03:46 +02:00

quota.go

Set capacity quotas, class, owner and reservation from the admin pages

2026-09-30 22:40:45 +02:00

release.go

Sign releases, and install only signed ones once a key is trusted

2026-09-30 23:03:48 +02:00

requestlog.go

Forgetting a person and keeping requests in the cluster: users forget, keys residency --in-cluster

2026-10-04 23:27:31 +02:00

resolve.go

Manage people and the tenancy tree, from the web and the CLI

2026-09-28 16:26:39 +02:00

restore_test.go

server restore: an archive checked, dry-run, and put back — configuration with modes and owners, the store in SQLite or PostgreSQL; backup_copy_to copies archives off the machine

2026-10-04 23:09:59 +02:00

restore.go

server restore: an archive checked, dry-run, and put back — configuration with modes and owners, the store in SQLite or PostgreSQL; backup_copy_to copies archives off the machine

2026-10-04 23:09:59 +02:00

root.go

ComfyUI as an engine of the cluster's own

2026-10-10 00:39:10 +02:00

selfupgrade.go

An upgrade that leaves nodes behind says which

2026-10-14 09:01:09 +02:00

servable_test.go

Refuse weights the nodes' llama.cpp cannot load, before fetching them

2026-09-28 20:44:36 +02:00

server_agent_test.go

Take agent_tls.insecure as consent to an agent port without TLS too

2026-10-04 10:11:40 +02:00

server.go

An upgrade refits the judge an earlier server init installed

2026-10-16 22:06:56 +02:00

serveradd.go

A floating address keeps to its own network, and Caddy follows the leader

2026-10-15 17:35:10 +02:00

serverinit_bundle_test.go

Offline sites and Kubernetes: llmgw bundle create, import and trust, offline.mirror, server init --bundle; a Helm chart for the control plane

2026-10-05 00:13:39 +02:00

serverinit_bundle.go

The cluster's settings live in the store, and every change takes effect

2026-10-05 15:25:23 +02:00

serverinit_choices_test.go

One machine runs one judge, on up to four cores

2026-10-16 19:35:42 +02:00

serverinit_choices.go

One machine runs one judge, on up to four cores

2026-10-16 19:35:42 +02:00

serverinit_cluster_test.go

The judges and the starter divide a machine's CPUs instead of each taking all

2026-10-16 13:00:39 +02:00

serverinit_cluster.go

One machine runs one judge, on up to four cores

2026-10-16 19:35:42 +02:00

serverinit_firewall.go

The firewall setting reaches the mirror machine and the store's members

2026-10-09 08:53:17 +02:00

serverinit_notify_test.go

Setup asks where alerts go, and sends each a test

2026-10-14 13:55:48 +02:00

serverinit_notify.go

Setup asks where alerts go, and sends each a test

2026-10-14 13:55:48 +02:00

serverinit_patroni.go

Setup lets every part be chosen, each with its settings; three machines each run the gateway

2026-10-15 11:22:36 +02:00

serverinit_postgres_test.go

Find the PostgreSQL a machine has, rather than assume its names

2026-09-28 14:47:55 +02:00

serverinit_postgres.go

The store's PostgreSQL sized to its machine's memory

2026-10-08 21:48:53 +02:00

serverinit_standby.go

Setup lets every part be chosen, each with its settings; three machines each run the gateway

2026-10-15 11:22:36 +02:00

serverinit_test.go

The mirror's address is one the nodes reach, and node add checks it

2026-10-06 09:36:09 +02:00

serverinit.go

The judges and the starter divide a machine's CPUs instead of each taking all

2026-10-16 13:00:39 +02:00

service_test.go

llmgw start --wait waits for the models, not for the setting

2026-10-07 21:43:45 +02:00

service.go

llmgw stop, start and restart act on the whole system

2026-10-07 18:01:45 +02:00

simengine.go

Onboard a node with a script, and find two examples that never worked

2026-09-16 12:28:36 +02:00

sourcenote_test.go

Say when a replica is running an edited profile, and check a source from the web too

2026-09-27 22:42:26 +02:00

sourcenote.go

Say when a replica is running an edited profile, and check a source from the web too

2026-09-27 22:42:26 +02:00

status_test.go

An on-demand model that is up reads as ready, not as not placed

2026-10-10 01:01:55 +02:00

status.go

llmgw stop, start and restart act on the whole system

2026-10-07 18:01:45 +02:00

storeflag_test.go

Stop calling the PostgreSQL store SQLite, and stop guessing where it is

2026-09-27 12:20:40 +02:00

storeflag.go

Stop calling the PostgreSQL store SQLite, and stop guessing where it is

2026-09-27 12:20:40 +02:00

system.go

llmgw start --wait waits for the models, not for the setting

2026-10-07 21:43:45 +02:00

threads.go

Manage people and the tenancy tree, from the web and the CLI

2026-09-28 16:26:39 +02:00

toolcheck_test.go

llmgw ctl tool-check asks with the engine's key

2026-10-08 11:24:37 +02:00

toolcheck.go

llmgw ctl tool-check asks with the engine's key

2026-10-08 11:24:37 +02:00

train.go

Trained adapters published, private to their owners like the knowledge bases they come from

2026-10-08 21:30:50 +02:00

tui_actions_test.go

Canary on the console's model menu; docs

2026-10-04 20:33:59 +02:00

tui_actions.go

The guardrails' judge has its own command, and its trouble is shown

2026-10-06 10:47:37 +02:00

tui_audit.go

Usage export as CSV: by period and by place, person, key and model, from the rollups

2026-10-04 16:49:04 +02:00

tui_chart.go

Place chart points by their time, so a window's samples fill the chart's width

2026-10-04 08:55:19 +02:00

tui_choices.go

One machine runs one judge, on up to four cores

2026-10-16 19:35:42 +02:00

tui_console.go

The cluster page and the console say which picture and 3D models are here, and add them

2026-10-13 20:48:40 +02:00

tui_docs.go

A docs portal at /docs and a client developer's guide

2026-10-04 23:46:42 +02:00

tui_federation_test.go

Federation: another llmgw cluster as a provider, its models as PEER/MODEL

2026-10-05 01:21:30 +02:00

tui_federation.go

Federation: another llmgw cluster as a provider, its models as PEER/MODEL

2026-10-05 01:21:30 +02:00

tui_form.go

Setup lets every part be chosen, each with its settings; three machines each run the gateway

2026-10-15 11:22:36 +02:00

tui_frame.go

Show the version top right on every screen

2026-10-04 01:18:45 +02:00

tui_guard.go

Guardrails in the console

2026-10-05 03:36:17 +02:00

tui_hf.go

The cluster's settings live in the store, and every change takes effect

2026-10-05 15:25:23 +02:00

tui_home.go

Watch the cluster and edit every setting from the console

2026-10-03 22:54:09 +02:00

tui_init_test.go

Setup lets every part be chosen, each with its settings; three machines each run the gateway

2026-10-15 11:22:36 +02:00

tui_init.go

Setup lets every part be chosen, each with its settings; three machines each run the gateway

2026-10-15 11:22:36 +02:00

tui_jobs.go

Serve and update models in the background from the console, and follow them under Jobs

2026-10-04 01:27:02 +02:00

tui_kb_test.go

The console's Knowledge screen, and the manual's page on knowledge bases

2026-10-08 13:58:12 +02:00

tui_kb.go

Collections a part of the tree answers from when a request names none

2026-10-08 15:52:43 +02:00

tui_mirror.go

The cluster page and the console say which picture and 3D models are here, and add them

2026-10-13 20:48:40 +02:00

tui_monitor.go

Console: rotate a key, cap key lifetimes, test the webhooks; README, onboarding and console docs caught up

2026-10-04 17:28:24 +02:00

tui_node_test.go

Let server init ask for what it needs, and node add take several nodes

2026-09-29 21:54:18 +02:00

tui_node.go

Add a standby control plane with server add, and draw the screens as windows

2026-10-03 22:29:12 +02:00

tui_people.go

The cluster's settings live in the store, and every change takes effect

2026-10-05 15:25:23 +02:00

tui_presets_test.go

The cluster page and the console say which picture and 3D models are here, and add them

2026-10-13 20:48:40 +02:00

tui_presets.go

The cluster page and the console say which picture and 3D models are here, and add them

2026-10-13 20:48:40 +02:00

tui_quality_test.go

Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

2026-10-05 00:04:43 +02:00

tui_quality.go

Quality suites, and new weights held to one: ctl eval, model update/promote --eval, model gate

2026-10-05 00:04:43 +02:00

tui_setup_test.go

Setup lets every part be chosen, each with its settings; three machines each run the gateway

2026-10-15 11:22:36 +02:00

tui_setup.go

One machine runs one judge, on up to four cores

2026-10-16 19:35:42 +02:00

tui_widgets.go

Open setup screens or a console when llmgw runs bare in a terminal

2026-10-03 21:33:59 +02:00

usage.go

Usage export as CSV: by period and by place, person, key and model, from the rollups

2026-10-04 16:49:04 +02:00

users.go

A forgotten password is set through a one-time link

2026-10-07 13:20:11 +02:00

wipe_test.go

The firewall turns off and on with the cluster's settings; llmgw wipe

2026-10-06 08:00:06 +02:00

wipe.go

The firewall turns off and on with the cluster's settings; llmgw wipe

2026-10-06 08:00:06 +02:00