Ilker ManapandClaude Opus 5.5 f2c6f12cfb How fast a model would answer on the cards, as this cluster would lay it out ...

For each layout modelplan.Options offers here — one card, tensor parallel
across a node's cards, llama.cpp's layer split, a pipeline across nodes on
a fabric — modelplan.GPUTokensPerSecond gives one reply's speed: the bytes
a token reads over each card's bandwidth, by its model name (a table, SXM
and PCIe apart), plus two all-reduces a layer for tensor parallel (8 us on
NVLink, 25 us and up on PCIe), turns between cards for a layer split, and
a hop per node over InfiniBand or RoCE for a pipeline. The explorer and a
repository's files show the fastest, with the layout; mixed classes are
judged class by class, FP8 on 8.9 cards only.

Calibrated on cuda01's Quadro RTX 6000 (672 GB/s): qwen3-vl-8b Q8, 8.7 GB,
62.1 tok/s measured, 59 estimated; Qwen3-Coder-30B-A3B Q4_K_M, 18.6 GB,
144 measured, which says a model of experts reads 1.6 times its active
share (ActiveShare now does, on the CPU too). Tensor parallel and fabrics
are not measured here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-28 00:19:36 +01:00

..

guide

The install guide shows every screen of a real install

2026-10-24 01:35:08 +02:00

locales

How fast a model would answer on the cards, as this cluster would lay it out

2026-10-28 00:19:36 +01:00

proxy

A floating address keeps to its own network, and Caddy follows the leader

2026-10-15 17:35:10 +02:00

static

The install guide shows every screen of a real install

2026-10-24 01:35:08 +02:00

templates

How fast a model would answer on the cards, as this cluster would lay it out

2026-10-28 00:19:36 +01:00

about_test.go

Describe v0.10.0

2026-09-29 21:06:42 +02:00

about.go

The install guide shows every screen of a real install

2026-10-24 01:35:08 +02:00

account_test.go

Stop shipping a placeholder that looks like a value, and say what is served

2026-09-16 18:23:19 +02:00

account.go

VS Code gets the gateway's models from an extension the gateway serves

2026-10-24 01:39:50 +02:00

admin_capacity_test.go

Set capacity quotas, class, owner and reservation from the admin pages

2026-09-30 22:40:45 +02:00

admin_capacity.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

admin_guard_test.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

admin_guard.go

Guardrails turn off and on; a judge's refusal says which hazard; ctl guard judge ask

2026-10-06 12:41:08 +02:00

admin_mirror_test.go

The explorer lists only what a node loads

2026-10-26 13:00:28 +01:00

admin_mirror.go

How fast a model would answer on the cards, as this cluster would lay it out

2026-10-28 00:19:36 +01:00

admin_nodeengine_test.go

Choose each engine's build from the node's hardware, for every GPU generation

2026-10-01 14:35:58 +02:00

admin_nodeengine.go

Turkish pages are Turkish to the end

2026-10-21 09:22:42 +02:00

admin_page.go

Staff roles (viewer, operator, billing) and the console's Audit view

2026-10-04 16:30:56 +02:00

admin_people_test.go

Show each team's first-token and latency percentiles, errors and refusals

2026-10-03 19:50:16 +02:00

admin_people.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

admin_presets_test.go

The cluster page and the console say which picture and 3D models are here, and add them

2026-10-13 20:48:40 +02:00

admin_presets.go

Turkish pages are Turkish to the end

2026-10-21 09:22:42 +02:00

admin_quotas_test.go

Let a person belong to many places, or to none

2026-09-28 19:03:46 +02:00

admin_quotas.go

Turkish pages are Turkish to the end

2026-10-21 09:22:42 +02:00

admin_serve_test.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

admin_serve.go

Qwen3Guard is the judge, and judges on a GPU and on CPUs are asked in turn

2026-10-08 10:22:47 +02:00

admin_settings_test.go

The cluster's settings edited on the portal, by administrators

2026-10-08 17:23:13 +02:00

admin_settings.go

Turkish pages are Turkish to the end

2026-10-21 09:22:42 +02:00

admin_tenancy_test.go

Audit log: every change through the admin API and the administration pages, with ctl audit

2026-10-04 16:25:56 +02:00

admin_usage_test.go

Show each team's first-token and latency percentiles, errors and refusals

2026-10-03 19:50:16 +02:00

admin_usage.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

admin_users.go

Turkish pages are Turkish to the end

2026-10-21 09:22:42 +02:00

apidocs.go

The client API described in OpenAPI, and both APIs in Swagger UI

2026-10-10 20:22:48 +02:00

audit.go

Audit log: every change through the admin API and the administration pages, with ctl audit

2026-10-04 16:25:56 +02:00

chart_test.go

Show usage over time, per organization, team, project and person

2026-09-28 08:11:31 +02:00

chart.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

charts2_test.go

Write the bar chart's latency the way its table does

2026-09-28 18:11:27 +02:00

charts2.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

chat_test.go

The chat's kept conversation belongs to the account that kept it

2026-10-09 15:44:47 +02:00

chat.go

Turkish pages are Turkish to the end

2026-10-21 09:22:42 +02:00

chatrender_test.go

Render a finished chat reply as markdown

2026-09-28 22:44:18 +02:00

chatrender.go

Render a finished chat reply as markdown

2026-09-28 22:44:18 +02:00

claim_test.go

Believe X-Forwarded-* only from router.trusted_proxies

2026-10-04 10:29:02 +02:00

claim.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

codeblocks_test.go

Read the filename from a path comment inside the block

2026-09-16 21:12:00 +02:00

codeblocks.go

Read the filename from a path comment inside the block

2026-09-16 21:12:00 +02:00

connect_test.go

VS Code gets the gateway's models from an extension the gateway serves

2026-10-24 01:39:50 +02:00

connect.go

VS Code gets the gateway's models from an extension the gateway serves

2026-10-24 01:39:50 +02:00

contact.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

csp_test.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

csp.go

G4: a page that makes 3D models, and ctl 3d

2026-10-10 19:16:20 +02:00

dashboard.go

The cluster page and the console say which picture and 3D models are here, and add them

2026-10-13 20:48:40 +02:00

dashcharts.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

docs_test.go

A docs portal at /docs and a client developer's guide

2026-10-04 23:46:42 +02:00

docs.go

Merge features-c2: the docs portal and its links in English and Turkish

2026-10-05 00:15:25 +02:00

download_test.go

Let a reply full of code be downloaded as a zip

2026-09-16 21:06:07 +02:00

download.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

efficiency.go

Show what each model gets from its GPUs, per card and per watt-hour

2026-10-01 15:56:42 +02:00

errors_i18n_test.go

Turkish pages are Turkish to the end

2026-10-21 09:22:42 +02:00

errors_i18n.go

Turkish pages are Turkish to the end

2026-10-21 09:22:42 +02:00

explore_fit_test.go

How fast a model would answer on the cards, as this cluster would lay it out

2026-10-28 00:19:36 +01:00

explore_fit.go

How fast a model would answer on the cards, as this cluster would lay it out

2026-10-28 00:19:36 +01:00

fabric_page.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

formguard.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

gpumap_test.go

Read what wears a GPU out: XIDs, remapped rows, PCIe and NVLink errors

2026-10-01 16:02:07 +02:00

gpumap.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

i18n_chat_test.go

The portal's chat answers from the documents a person picks

2026-10-08 15:59:23 +02:00

i18n_cluster_test.go

Turkish pages are Turkish to the end

2026-10-21 09:22:42 +02:00

i18n_mirror_test.go

The explorer lists only what a node loads

2026-10-26 13:00:28 +01:00

i18n_people_test.go

The cluster's settings live in the store, and every change takes effect

2026-10-05 15:25:23 +02:00

i18n_public_test.go

The Continue configuration says what the catalog says of the model

2026-10-05 09:16:34 +02:00

i18n_quotas_test.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

i18n_test.go

Merge features-c2: the docs portal and its links in English and Turkish

2026-10-05 00:15:25 +02:00

i18n.go

Turkish pages are Turkish to the end

2026-10-21 09:22:42 +02:00

images_test.go

The client API described in OpenAPI, and both APIs in Swagger UI

2026-10-10 20:22:48 +02:00

images.go

G4: a page that makes 3D models, and ctl 3d

2026-10-10 19:16:20 +02:00

install_test.go

The install guide shows every screen of a real install

2026-10-24 01:35:08 +02:00

install.go

The install guide shows every screen of a real install

2026-10-24 01:35:08 +02:00

landing.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

licences_test.go

A person leaves licences out, and no key of theirs reaches a model under one

2026-10-11 16:10:22 +02:00

licences.go

Turkish pages are Turkish to the end

2026-10-21 09:22:42 +02:00

mirror_detail_test.go

A form being filled in is not refreshed away

2026-10-14 08:13:08 +02:00

mirror_detail.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

mirror_explore_test.go

The explorer lists only what a node loads

2026-10-26 13:00:28 +01:00

mirror_explore.go

How fast a model would answer on the cards, as this cluster would lay it out

2026-10-28 00:19:36 +01:00

modelcard_test.go

Show a Hugging Face repository's model card on the mirror page

2026-09-28 22:03:19 +02:00

modelcard.go

Render a finished chat reply as markdown

2026-09-28 22:44:18 +02:00

nav_test.go

Make people, limits and usage one page, with a panel per branch

2026-10-03 19:10:48 +02:00

nav.go

G4: a page that makes 3D models, and ctl 3d

2026-10-10 19:16:20 +02:00

node_logs_test.go

Show a node's logs on the admin page

2026-09-28 12:27:47 +02:00

node_logs.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

node_page_test.go

A node's page says which llmgw it runs and which model is on each GPU

2026-10-19 11:31:52 +02:00

node_page.go

A node's page says which llmgw it runs and which model is on each GPU

2026-10-19 11:31:52 +02:00

nvlink_page.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

oidc_test.go

People sign in with their Active Directory account

2026-10-18 10:59:50 +02:00

oidc.go

People sign in with their Active Directory account

2026-10-18 10:59:50 +02:00

org_page_test.go

Send an administrator from /org to the people page's panel

2026-10-03 19:15:25 +02:00

org_page.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

reset_test.go

A forgotten password is set through a one-time link

2026-10-07 13:20:11 +02:00

reset.go

A forgotten password is set through a one-time link

2026-10-07 13:20:11 +02:00

roles_test.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

roles.go

Staff roles (viewer, operator, billing) and the console's Audit view

2026-10-04 16:30:56 +02:00

rows_test.go

Keep every admin table to one line per record

2026-09-30 11:02:18 +02:00

selfusage_test.go

Show a person their own use and what is left of their budgets

2026-09-28 22:51:13 +02:00

selfusage.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

servable_test.go

Refuse weights the nodes' llama.cpp cannot load, before fetching them

2026-09-28 20:44:36 +02:00

signup_test.go

A forgotten password is set through a one-time link

2026-10-07 13:20:11 +02:00

timeseries_test.go

Label chart axes in binary sizes and small percents, and windows in words

2026-09-28 23:30:01 +02:00

timeseries.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00

web_test.go

Believe X-Forwarded-* only from router.trusted_proxies

2026-10-04 10:29:02 +02:00

web.go

The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates

2026-10-05 00:12:53 +02:00