llmgw

v0.10.7-rc.9 1b89d5cdea

v0.10.7-rc.9

Release / build (push) Successful in 1m12s

Details

Release / notes (push) Successful in 3s

Details

CI / test (push) Successful in 3m54s

Details

Pre-Release

gitea-actions released this 2026-10-03 18:57:50 +02:00 | 523 commits to main since this release

From the field report in issue #2, after a deployment on GPU nodes.

GPU metrics that move, and charts of them

A GPU's utilisation and memory in use were read once, when the agent
started, and published from then on: a card busy since looked as idle as it
was at registration, and one idle since as busy. They are now sampled with
the temperatures and power every heartbeat (ten seconds at most), with the
memory bandwidth in use, the SM clock and uncorrected ECC errors besides.
An agent too old to send them still has its registration figures published,
and a figure the driver does not give is left out rather than shown as zero.

New series: llmgw_gpu_memory_utilization_percent,
llmgw_gpu_sm_clock_mhz; llmgw_gpu_utilization_percent,
llmgw_gpu_memory_used_bytes and llmgw_gpu_ecc_errors now from the
sample.
The node page shows each card's utilisation, memory, clock and ECC
errors in its table, and charts of utilisation, memory, temperature and
power, one line per card.
The cluster page charts the fleet's GPUs by class: average utilisation
and memory in use.
Grafana (deploy/monitoring/grafana/llmgw-overview.json) has a GPU row:
utilisation, memory, memory bandwidth, temperature, power against the
limit, clock, and what holds cards back.
Alerts: GPUThermalThrottling (ten minutes slowed by heat),
GPUUncorrectedECCErrors, GPUHot (above 87 °C for ten minutes).
The cluster page's GPU map: every card a square, a row per node,
coloured by how busy, how full or how hot it is (?gpumap=util|memory|temp),
with what holds it and its figures under the pointer; free cards dashed.
Cards held and doing nothing: the replicas whose every card reads under
5% busy, with when they were last asked — the first place to find room.
What each model gets from its cards, on the cluster page and in
Grafana: output tokens a second per GPU and per watt-hour, with the cards'
draw and average utilisation, over the last quarter hour.
What wears a card out, read every minute: XID errors from the kernel's
log (counted from when the agent started), memory rows remapped or pages
retired and whether either waits for a reset, a card out of spare rows,
PCIe link retries and NVLink errors. Published as llmgw_gpu_xid_errors,
llmgw_gpu_last_xid, llmgw_gpu_remapped_rows, llmgw_gpu_remap_pending,
llmgw_gpu_remap_failure, llmgw_gpu_retired_pages,
llmgw_gpu_retirement_pending, llmgw_gpu_pcie_replays,
llmgw_gpu_nvlink_errors; said in words on the node page; a red outline
on the GPU map; a Grafana panel; and alerts — GPUXidError,
GPUOutOfSpareRows, GPUNeedsReset, GPUPCIeRetriesClimbing,
GPUNVLinkErrorsClimbing.
GPU time by owner on the usage page: the cards each owner's models
held over the window, charted and summed into GPU-hours, whether or not the
models were asked anything — the time a GPU quota is about. From the
metrics store, as far back as it keeps history.
llmgw_gpu_assigned{node,gpu,model,replica,owner}: which replica holds
each card and whose quota it counts against, for joining the GPU series to
models and tenants.

The chat page says whether a GPU or a CPU answered

A model can have replicas on a card and on CPUs at once, and the two answer
at very different speeds; the chat page gave no way to tell which one a reply
came from. The picker now says where each model runs — GPU, CPU or
GPU + CPU — and the line under each reply names what answered it: GPU 6000-24g or CPU, beside its tokens a second.

Every answer through the gateway carries the same as x-llmgw-hardware:
cpu, or the GPU class. An endpoint configured by hand, with no pool, sends
none.

A card shared with something else is judged every heartbeat

A card with another process on it when its agent started was kept out of
placement, and stayed out — "5014 MiB already allocated by another process" —
after that process let go of it, until the agent was restarted. Found on a
node shared with ComfyUI. The other way round, a tenant arriving after
registration was never seen, and a card the agent's own engine held when the
agent restarted read as someone else's.

The agent now reads every card's memory per process on each heartbeat,
leaves out what its own engines hold, and sends the rest; the control plane
keeps a card out of placement on that reading, and lets it back in when it
falls under 512 MiB. llmgw_gpu_memory_used_bytes reads the same figure. An
agent too old to send it keeps the reading from registration.

A GPU placement no longer runs on the CPU unnoticed

llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
where the loader looks), came up on the CPU, answered at CPU speed, and was
reported ready while its GPU sat reserved and idle. Three checks now stop it:

Installing the CUDA build asks it which devices it sees
(llama-server --list-devices). None, on a node with GPUs, is a failed
install, with what ldd says libggml-cuda.so is missing. The binary's
own ldd could not say it: the backend is opened at run time.
After the upgrade, engines the agent finds already running are held to
the same check when it adopts them: one serving a GPU placement on the CPU
is stopped, and its replacement starts through the check. The installed
CUDA build is checked again within ten minutes, and installed afresh if it
sees no GPU.
Starting a shard on GPUs is ready only when its process holds memory on
each of them (nvidia-smi --query-compute-apps, its process group
included). One that does not is stopped and reported failed: "placed on GPU
0 but computing on the CPU", with the devices the engine lists. This covers
a hand-installed binary (engines.llamacpp.bin), which nothing else checks.
llmgw ctl engine-check llamacpp, run on a GPU node, now also fails
when the engine sees no GPU, and otherwise prints the devices it computes on.

Shards are also given CUDA_DEVICE_ORDER=PCI_BUS_ID: placement numbers GPUs
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
node with two kinds of GPU, a shard could get a different card from the one it
was given.

A model of many files, mirrored whole for vLLM

vLLM loads a directory — shards, config.json, the tokenizer — and does not
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
shards alone, a node could be handed a version still downloading or missing a
shard, and a repository's page given as a model's source was fetched as one
file of HTML.

The explorer fetches a safetensors set with what the engine reads beside
it: config.json, generation_config.json, the tokenizer files, the shard
index, the quantisation config. Never code: a .py would be the engine
running what the repository ships, which is to be chosen on purpose.
A version is offered to nodes only whole. The mirror refuses its
manifest (409, with what is missing) while a file is still downloading, or
while a shard its model.safetensors.index.json names, or a part of a
split (-00002-of-00004), is absent. A node reports "not complete on the
mirror yet" and is tried again.
Checksums go with the files. The mirror keeps each file's sha256 beside
it and the manifest carries it; a node checks every file of a set against
it before using the set, and fetches one that differs again.
llmgw mirror add takes several URLs into one version.
A Hugging Face repository page given as a source is read as the
repository id for a model vLLM or SGLang runs, which the engine downloads
itself (the node then needs to reach Hugging Face), and refused for one
llama.cpp runs, which needs a file.

With a vLLM profile of tp: 4, placement reserves four GPUs of one node for
such a model, as before.

Signing up with an address that has an account

On PostgreSQL, signing up with an address already registered showed
"something went wrong; try again": the store recognised a duplicate by the
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
code 23505. It now knows both, and the address is answered as on SQLite —
with the same page as a new sign-up, so the page cannot be used to find out
who has an account. That page now also says what to do if the address had
one: sign in, or ask an operator to reset the password.

An error the page does not explain says the account was not created and
gives a reference (err-…), logged beside the error, for an operator to find
it; the database's own words stay in the log.

An adapter no fabric uses does not keep a node out

llmgw doctor failed a node for a RoCE port with no RoCE v2 GID entry, or for
ib_uverbs not being loaded, whether or not any fabric used them; on a node
added for single-node inference, with no fabric given, that refused it for
RDMA nobody had asked for.

Those are failures only for ports a declared fabric uses, and only when a
fabric is declared. Otherwise they are warnings saying nothing uses them yet.
Which fabric an undeclared InfiniBand port is on, and whether it has a
subnet manager LID, are notes rather than warnings; node add still warns
once that adapters are up with no fabric given.
node add --fabric hands its fabrics to doctor (doctor --fabric name:mlx5_0,…), so they are checked as declared even though agent.yaml is
written afterwards.

A restart with no room beside it

ctl restart places each replacement beside the replica it replaces. With
no room for it — one GPU, every card that fits held by the replica itself —
the old replica was stopped only after the drain grace period, five minutes
of "restart replacing" with nothing saying why, which read as a restart that
never happened; scaling to zero and back was the way out. And a replacement
that fit nowhere even with the old replica gone was stopped for all the same,
leaving the model with nothing. Now:

The replica is the only thing in the way, found by placing the
replacement on a copy of the inventory without it: stopped at once, the
replacement taking its place.
The replacement fits nowhere even then: the restart is given up and the
replica keeps serving. ctl replicas shows restart refused: and why.
Room being made — a lower class preempted, an idle on-demand replica
stopping — is waited for, as before.
ctl restart --in-place stops each replica first, whatever room there is.

Every GPU node gets engine builds it can run, by itself

Which build of an engine a node installs follows its hardware — the CPU
architecture, the driver, and the oldest GPU on it — from what each project
publishes and how it was compiled. Each engine has a list of builds, best
first; the node takes the first its hardware meets, and the first of those
that installs and is seen to compute on the GPU. A build the table is wrong
about costs its download, and the next one is tried.

	llama.cpp	vLLM
Driver 580+, Turing and newer	CUDA 13.3	the version asked for, CUDA 13 (PyPI)
Driver 525–579	CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570	the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch)
Volta (V100)	CUDA 12.8 on driver 570+, else Vulkan	0.19.1, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels
Pascal, Maxwell	CUDA 12.8 on 570+, else Vulkan	none: no vLLM is built for them
arm64 GPU (GH200, Grace)	CUDA 13.3, else Vulkan — never the CPU build	CUDA 13 or 12.9, 8.0 and newer
No NVIDIA GPU	the CPU build	none

Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
starts at 7.5.

A node running an older vLLM says so: the cluster page's vLLM build
column shows the version installed, and the installation's detail says why
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
vLLM is not placed on a V100 by accident.
The install computes on the GPU before it counts: a small tensor
operation for vLLM, the devices llama.cpp lists for its GPU builds.
The build can still be chosen per node on the cluster page (auto, CUDA
13, CUDA 12.9, none), with llmgw ctl engine-build NODE vllm BUILD, or
POST /api/v1/nodes/{id}/engines; kept in the store and sent to the node
at once.
Whether an engine computes on its GPUs is read from every process
nvidia-smi lists, not compute processes alone: the Vulkan build is a
graphics one. A llama.cpp shard is given GGML_VK_VISIBLE_DEVICES beside
CUDA_VISIBLE_DEVICES.

Engines per node, and nodes that do not run them all

A cluster with older GPUs could not add them cleanly: node add refused a
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
and --ignore-checks passed over every other problem with it.

What vLLM cannot do is a warning, in llmgw doctor and node add: the
node joins without it and serves what it does run. node add checks the
vLLM the control plane installs, not the binary's own default.
node add --disable-engine vllm keeps an engine off a node for good:
written as engines.disabled in its agent.yaml, never installed or
advertised there, and not checked by doctor.
engine_overrides in server.yaml give nodes picked by GPU class, id
or labels a different version of an engine, or none — for exceptions; the
build each GPU generation needs is chosen without it (above).
The requirements follow the version, as the table above sets out; the
threshold had been CUDA 13's for every version.
An engine binary named in agent.yaml is used. The agent had installed
its own beside it, which was then used first.
The node page shows each GPU's compute capability.

Serving from the mirror says where a model will run

The serve form put every GGUF model on the CPU: it offered no choice, and its
"Profile name" field, pre-filled "cpu", read as though renaming it would move
the model to a GPU. Placement is decided by gpu_class, which the form never
set. Now:

Runs on lists the fleet's GPU classes with how many cards are free, and
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
cards is chosen.
GPUs per replica sets how many cards a replica takes on one node
(gpus_per_node for llama.cpp, tp for vLLM), and parallel requests
how many a llama.cpp replica serves at once.
The profile is named for where it runs (t4, t4-x4, cpu) rather than
typed.
A line under the form says what the model will run as — engine, CPU or how
many of which GPU, memory, context pool and how it is divided — and turns
red when the CPU is chosen with GPUs free. The message after adding repeats
it.
Choosing the CPU while GPUs are free is refused unless run on the CPU
anyway is ticked.

llama.cpp on several GPUs of one node

A llama.cpp profile takes several of a node's GPUs with gpus_per_node: N:
placement reserves N cards of its gpu_class on one node, the engine sees
those N (CUDA_VISIBLE_DEVICES) and spreads the layers over them
(--split-mode layer, in proportion to each card's free memory, which for
whole cards is even). tp or pp on a llama.cpp profile was a width nothing
placed — the replica stayed unplaced or ran on one card. It is now read as
gpus_per_node, so a profile written that way is placed on that many cards
after the upgrade, rather than dropped; only both given and disagreeing, or
nodes above one, is refused.

Downloads

Caddyfile
1.5 KiB
2026-10-03 18:58:57 +02:00
checksums.txt
645 B
2026-10-03 18:58:57 +02:00
llmgw-v0.10.7-rc.9-darwin-amd64
33 MiB
2026-10-03 18:58:59 +02:00
llmgw-v0.10.7-rc.9-darwin-arm64
31 MiB
2026-10-03 18:59:00 +02:00
llmgw-v0.10.7-rc.9-linux-amd64
33 MiB
2026-10-03 18:59:01 +02:00
llmgw-v0.10.7-rc.9-linux-amd64-nvml
34 MiB
2026-10-03 18:59:02 +02:00
llmgw-v0.10.7-rc.9-linux-arm64
31 MiB
2026-10-03 18:59:03 +02:00
nginx.conf
2.4 KiB
2026-10-03 18:59:03 +02:00
Source Code (ZIP)
Source Code (TAR.GZ)

v0.10.7-rc.8 477d0bf19d

v0.10.7-rc.8

Release / build (push) Successful in 1m14s

Details

Release / notes (push) Successful in 4s

Details

CI / test (push) Successful in 3m50s

Details

Pre-Release

gitea-actions released this 2026-10-03 18:46:23 +02:00 | 524 commits to main since this release

From the field report in issue #2, after a deployment on GPU nodes.

GPU metrics that move, and charts of them

A GPU's utilisation and memory in use were read once, when the agent
started, and published from then on: a card busy since looked as idle as it
was at registration, and one idle since as busy. They are now sampled with
the temperatures and power every heartbeat (ten seconds at most), with the
memory bandwidth in use, the SM clock and uncorrected ECC errors besides.
An agent too old to send them still has its registration figures published,
and a figure the driver does not give is left out rather than shown as zero.

New series: llmgw_gpu_memory_utilization_percent,
llmgw_gpu_sm_clock_mhz; llmgw_gpu_utilization_percent,
llmgw_gpu_memory_used_bytes and llmgw_gpu_ecc_errors now from the
sample.
The node page shows each card's utilisation, memory, clock and ECC
errors in its table, and charts of utilisation, memory, temperature and
power, one line per card.
The cluster page charts the fleet's GPUs by class: average utilisation
and memory in use.
Grafana (deploy/monitoring/grafana/llmgw-overview.json) has a GPU row:
utilisation, memory, memory bandwidth, temperature, power against the
limit, clock, and what holds cards back.
Alerts: GPUThermalThrottling (ten minutes slowed by heat),
GPUUncorrectedECCErrors, GPUHot (above 87 °C for ten minutes).
The cluster page's GPU map: every card a square, a row per node,
coloured by how busy, how full or how hot it is (?gpumap=util|memory|temp),
with what holds it and its figures under the pointer; free cards dashed.
Cards held and doing nothing: the replicas whose every card reads under
5% busy, with when they were last asked — the first place to find room.
What each model gets from its cards, on the cluster page and in
Grafana: output tokens a second per GPU and per watt-hour, with the cards'
draw and average utilisation, over the last quarter hour.
What wears a card out, read every minute: XID errors from the kernel's
log (counted from when the agent started), memory rows remapped or pages
retired and whether either waits for a reset, a card out of spare rows,
PCIe link retries and NVLink errors. Published as llmgw_gpu_xid_errors,
llmgw_gpu_last_xid, llmgw_gpu_remapped_rows, llmgw_gpu_remap_pending,
llmgw_gpu_remap_failure, llmgw_gpu_retired_pages,
llmgw_gpu_retirement_pending, llmgw_gpu_pcie_replays,
llmgw_gpu_nvlink_errors; said in words on the node page; a red outline
on the GPU map; a Grafana panel; and alerts — GPUXidError,
GPUOutOfSpareRows, GPUNeedsReset, GPUPCIeRetriesClimbing,
GPUNVLinkErrorsClimbing.
GPU time by owner on the usage page: the cards each owner's models
held over the window, charted and summed into GPU-hours, whether or not the
models were asked anything — the time a GPU quota is about. From the
metrics store, as far back as it keeps history.
llmgw_gpu_assigned{node,gpu,model,replica,owner}: which replica holds
each card and whose quota it counts against, for joining the GPU series to
models and tenants.

A card shared with something else is judged every heartbeat

A card with another process on it when its agent started was kept out of
placement, and stayed out — "5014 MiB already allocated by another process" —
after that process let go of it, until the agent was restarted. Found on a
node shared with ComfyUI. The other way round, a tenant arriving after
registration was never seen, and a card the agent's own engine held when the
agent restarted read as someone else's.

The agent now reads every card's memory per process on each heartbeat,
leaves out what its own engines hold, and sends the rest; the control plane
keeps a card out of placement on that reading, and lets it back in when it
falls under 512 MiB. llmgw_gpu_memory_used_bytes reads the same figure. An
agent too old to send it keeps the reading from registration.

A GPU placement no longer runs on the CPU unnoticed

llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
where the loader looks), came up on the CPU, answered at CPU speed, and was
reported ready while its GPU sat reserved and idle. Three checks now stop it:

Installing the CUDA build asks it which devices it sees
(llama-server --list-devices). None, on a node with GPUs, is a failed
install, with what ldd says libggml-cuda.so is missing. The binary's
own ldd could not say it: the backend is opened at run time.
After the upgrade, engines the agent finds already running are held to
the same check when it adopts them: one serving a GPU placement on the CPU
is stopped, and its replacement starts through the check. The installed
CUDA build is checked again within ten minutes, and installed afresh if it
sees no GPU.
Starting a shard on GPUs is ready only when its process holds memory on
each of them (nvidia-smi --query-compute-apps, its process group
included). One that does not is stopped and reported failed: "placed on GPU
0 but computing on the CPU", with the devices the engine lists. This covers
a hand-installed binary (engines.llamacpp.bin), which nothing else checks.
llmgw ctl engine-check llamacpp, run on a GPU node, now also fails
when the engine sees no GPU, and otherwise prints the devices it computes on.

Shards are also given CUDA_DEVICE_ORDER=PCI_BUS_ID: placement numbers GPUs
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
node with two kinds of GPU, a shard could get a different card from the one it
was given.

A model of many files, mirrored whole for vLLM

vLLM loads a directory — shards, config.json, the tokenizer — and does not
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
shards alone, a node could be handed a version still downloading or missing a
shard, and a repository's page given as a model's source was fetched as one
file of HTML.

The explorer fetches a safetensors set with what the engine reads beside
it: config.json, generation_config.json, the tokenizer files, the shard
index, the quantisation config. Never code: a .py would be the engine
running what the repository ships, which is to be chosen on purpose.
A version is offered to nodes only whole. The mirror refuses its
manifest (409, with what is missing) while a file is still downloading, or
while a shard its model.safetensors.index.json names, or a part of a
split (-00002-of-00004), is absent. A node reports "not complete on the
mirror yet" and is tried again.
Checksums go with the files. The mirror keeps each file's sha256 beside
it and the manifest carries it; a node checks every file of a set against
it before using the set, and fetches one that differs again.
llmgw mirror add takes several URLs into one version.
A Hugging Face repository page given as a source is read as the
repository id for a model vLLM or SGLang runs, which the engine downloads
itself (the node then needs to reach Hugging Face), and refused for one
llama.cpp runs, which needs a file.

With a vLLM profile of tp: 4, placement reserves four GPUs of one node for
such a model, as before.

Signing up with an address that has an account

On PostgreSQL, signing up with an address already registered showed
"something went wrong; try again": the store recognised a duplicate by the
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
code 23505. It now knows both, and the address is answered as on SQLite —
with the same page as a new sign-up, so the page cannot be used to find out
who has an account. That page now also says what to do if the address had
one: sign in, or ask an operator to reset the password.

An error the page does not explain says the account was not created and
gives a reference (err-…), logged beside the error, for an operator to find
it; the database's own words stay in the log.

An adapter no fabric uses does not keep a node out

llmgw doctor failed a node for a RoCE port with no RoCE v2 GID entry, or for
ib_uverbs not being loaded, whether or not any fabric used them; on a node
added for single-node inference, with no fabric given, that refused it for
RDMA nobody had asked for.

Those are failures only for ports a declared fabric uses, and only when a
fabric is declared. Otherwise they are warnings saying nothing uses them yet.
Which fabric an undeclared InfiniBand port is on, and whether it has a
subnet manager LID, are notes rather than warnings; node add still warns
once that adapters are up with no fabric given.
node add --fabric hands its fabrics to doctor (doctor --fabric name:mlx5_0,…), so they are checked as declared even though agent.yaml is
written afterwards.

A restart with no room beside it

ctl restart places each replacement beside the replica it replaces. With
no room for it — one GPU, every card that fits held by the replica itself —
the old replica was stopped only after the drain grace period, five minutes
of "restart replacing" with nothing saying why, which read as a restart that
never happened; scaling to zero and back was the way out. And a replacement
that fit nowhere even with the old replica gone was stopped for all the same,
leaving the model with nothing. Now:

The replica is the only thing in the way, found by placing the
replacement on a copy of the inventory without it: stopped at once, the
replacement taking its place.
The replacement fits nowhere even then: the restart is given up and the
replica keeps serving. ctl replicas shows restart refused: and why.
Room being made — a lower class preempted, an idle on-demand replica
stopping — is waited for, as before.
ctl restart --in-place stops each replica first, whatever room there is.

Every GPU node gets engine builds it can run, by itself

Which build of an engine a node installs follows its hardware — the CPU
architecture, the driver, and the oldest GPU on it — from what each project
publishes and how it was compiled. Each engine has a list of builds, best
first; the node takes the first its hardware meets, and the first of those
that installs and is seen to compute on the GPU. A build the table is wrong
about costs its download, and the next one is tried.

	llama.cpp	vLLM
Driver 580+, Turing and newer	CUDA 13.3	the version asked for, CUDA 13 (PyPI)
Driver 525–579	CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570	the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch)
Volta (V100)	CUDA 12.8 on driver 570+, else Vulkan	0.19.1, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels
Pascal, Maxwell	CUDA 12.8 on 570+, else Vulkan	none: no vLLM is built for them
arm64 GPU (GH200, Grace)	CUDA 13.3, else Vulkan — never the CPU build	CUDA 13 or 12.9, 8.0 and newer
No NVIDIA GPU	the CPU build	none

Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
starts at 7.5.

A node running an older vLLM says so: the cluster page's vLLM build
column shows the version installed, and the installation's detail says why
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
vLLM is not placed on a V100 by accident.
The install computes on the GPU before it counts: a small tensor
operation for vLLM, the devices llama.cpp lists for its GPU builds.
The build can still be chosen per node on the cluster page (auto, CUDA
13, CUDA 12.9, none), with llmgw ctl engine-build NODE vllm BUILD, or
POST /api/v1/nodes/{id}/engines; kept in the store and sent to the node
at once.
Whether an engine computes on its GPUs is read from every process
nvidia-smi lists, not compute processes alone: the Vulkan build is a
graphics one. A llama.cpp shard is given GGML_VK_VISIBLE_DEVICES beside
CUDA_VISIBLE_DEVICES.

Engines per node, and nodes that do not run them all

A cluster with older GPUs could not add them cleanly: node add refused a
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
and --ignore-checks passed over every other problem with it.

What vLLM cannot do is a warning, in llmgw doctor and node add: the
node joins without it and serves what it does run. node add checks the
vLLM the control plane installs, not the binary's own default.
node add --disable-engine vllm keeps an engine off a node for good:
written as engines.disabled in its agent.yaml, never installed or
advertised there, and not checked by doctor.
engine_overrides in server.yaml give nodes picked by GPU class, id
or labels a different version of an engine, or none — for exceptions; the
build each GPU generation needs is chosen without it (above).
The requirements follow the version, as the table above sets out; the
threshold had been CUDA 13's for every version.
An engine binary named in agent.yaml is used. The agent had installed
its own beside it, which was then used first.
The node page shows each GPU's compute capability.

Serving from the mirror says where a model will run

The serve form put every GGUF model on the CPU: it offered no choice, and its
"Profile name" field, pre-filled "cpu", read as though renaming it would move
the model to a GPU. Placement is decided by gpu_class, which the form never
set. Now:

Runs on lists the fleet's GPU classes with how many cards are free, and
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
cards is chosen.
GPUs per replica sets how many cards a replica takes on one node
(gpus_per_node for llama.cpp, tp for vLLM), and parallel requests
how many a llama.cpp replica serves at once.
The profile is named for where it runs (t4, t4-x4, cpu) rather than
typed.
A line under the form says what the model will run as — engine, CPU or how
many of which GPU, memory, context pool and how it is divided — and turns
red when the CPU is chosen with GPUs free. The message after adding repeats
it.
Choosing the CPU while GPUs are free is refused unless run on the CPU
anyway is ticked.

llama.cpp on several GPUs of one node

A llama.cpp profile takes several of a node's GPUs with gpus_per_node: N:
placement reserves N cards of its gpu_class on one node, the engine sees
those N (CUDA_VISIBLE_DEVICES) and spreads the layers over them
(--split-mode layer, in proportion to each card's free memory, which for
whole cards is even). tp or pp on a llama.cpp profile was a width nothing
placed — the replica stayed unplaced or ran on one card. It is now read as
gpus_per_node, so a profile written that way is placed on that many cards
after the upgrade, rather than dropped; only both given and disagreeing, or
nodes above one, is refused.

Downloads

Caddyfile
1.5 KiB
2026-10-03 18:47:29 +02:00
checksums.txt
645 B
2026-10-03 18:47:29 +02:00
llmgw-v0.10.7-rc.8-darwin-amd64
33 MiB
2026-10-03 18:47:31 +02:00
llmgw-v0.10.7-rc.8-darwin-arm64
31 MiB
2026-10-03 18:47:32 +02:00
llmgw-v0.10.7-rc.8-linux-amd64
33 MiB
2026-10-03 18:47:33 +02:00
llmgw-v0.10.7-rc.8-linux-amd64-nvml
34 MiB
2026-10-03 18:47:34 +02:00
llmgw-v0.10.7-rc.8-linux-arm64
31 MiB
2026-10-03 18:47:36 +02:00
nginx.conf
2.4 KiB
2026-10-03 18:47:36 +02:00
Source Code (ZIP)
Source Code (TAR.GZ)

v0.10.7-rc.7 372ef64889

v0.10.7-rc.7

Release / build (push) Successful in 1m11s

Details

Release / notes (push) Successful in 3s

Details

CI / test (push) Successful in 3m48s

Details

Pre-Release

gitea-actions released this 2026-10-01 16:07:59 +02:00 | 525 commits to main since this release

From the field report in issue #2, after a deployment on GPU nodes.

GPU metrics that move, and charts of them

A GPU's utilisation and memory in use were read once, when the agent
started, and published from then on: a card busy since looked as idle as it
was at registration, and one idle since as busy. They are now sampled with
the temperatures and power every heartbeat (ten seconds at most), with the
memory bandwidth in use, the SM clock and uncorrected ECC errors besides.
An agent too old to send them still has its registration figures published,
and a figure the driver does not give is left out rather than shown as zero.

New series: llmgw_gpu_memory_utilization_percent,
llmgw_gpu_sm_clock_mhz; llmgw_gpu_utilization_percent,
llmgw_gpu_memory_used_bytes and llmgw_gpu_ecc_errors now from the
sample.
The node page shows each card's utilisation, memory, clock and ECC
errors in its table, and charts of utilisation, memory, temperature and
power, one line per card.
The cluster page charts the fleet's GPUs by class: average utilisation
and memory in use.
Grafana (deploy/monitoring/grafana/llmgw-overview.json) has a GPU row:
utilisation, memory, memory bandwidth, temperature, power against the
limit, clock, and what holds cards back.
Alerts: GPUThermalThrottling (ten minutes slowed by heat),
GPUUncorrectedECCErrors, GPUHot (above 87 °C for ten minutes).
The cluster page's GPU map: every card a square, a row per node,
coloured by how busy, how full or how hot it is (?gpumap=util|memory|temp),
with what holds it and its figures under the pointer; free cards dashed.
Cards held and doing nothing: the replicas whose every card reads under
5% busy, with when they were last asked — the first place to find room.
What each model gets from its cards, on the cluster page and in
Grafana: output tokens a second per GPU and per watt-hour, with the cards'
draw and average utilisation, over the last quarter hour.
What wears a card out, read every minute: XID errors from the kernel's
log (counted from when the agent started), memory rows remapped or pages
retired and whether either waits for a reset, a card out of spare rows,
PCIe link retries and NVLink errors. Published as llmgw_gpu_xid_errors,
llmgw_gpu_last_xid, llmgw_gpu_remapped_rows, llmgw_gpu_remap_pending,
llmgw_gpu_remap_failure, llmgw_gpu_retired_pages,
llmgw_gpu_retirement_pending, llmgw_gpu_pcie_replays,
llmgw_gpu_nvlink_errors; said in words on the node page; a red outline
on the GPU map; a Grafana panel; and alerts — GPUXidError,
GPUOutOfSpareRows, GPUNeedsReset, GPUPCIeRetriesClimbing,
GPUNVLinkErrorsClimbing.
GPU time by owner on the usage page: the cards each owner's models
held over the window, charted and summed into GPU-hours, whether or not the
models were asked anything — the time a GPU quota is about. From the
metrics store, as far back as it keeps history.
llmgw_gpu_assigned{node,gpu,model,replica,owner}: which replica holds
each card and whose quota it counts against, for joining the GPU series to
models and tenants.

A GPU placement no longer runs on the CPU unnoticed

llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
where the loader looks), came up on the CPU, answered at CPU speed, and was
reported ready while its GPU sat reserved and idle. Three checks now stop it:

Installing the CUDA build asks it which devices it sees
(llama-server --list-devices). None, on a node with GPUs, is a failed
install, with what ldd says libggml-cuda.so is missing. The binary's
own ldd could not say it: the backend is opened at run time.
After the upgrade, engines the agent finds already running are held to
the same check when it adopts them: one serving a GPU placement on the CPU
is stopped, and its replacement starts through the check. The installed
CUDA build is checked again within ten minutes, and installed afresh if it
sees no GPU.
Starting a shard on GPUs is ready only when its process holds memory on
each of them (nvidia-smi --query-compute-apps, its process group
included). One that does not is stopped and reported failed: "placed on GPU
0 but computing on the CPU", with the devices the engine lists. This covers
a hand-installed binary (engines.llamacpp.bin), which nothing else checks.
llmgw ctl engine-check llamacpp, run on a GPU node, now also fails
when the engine sees no GPU, and otherwise prints the devices it computes on.

Shards are also given CUDA_DEVICE_ORDER=PCI_BUS_ID: placement numbers GPUs
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
node with two kinds of GPU, a shard could get a different card from the one it
was given.

A model of many files, mirrored whole for vLLM

vLLM loads a directory — shards, config.json, the tokenizer — and does not
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
shards alone, a node could be handed a version still downloading or missing a
shard, and a repository's page given as a model's source was fetched as one
file of HTML.

The explorer fetches a safetensors set with what the engine reads beside
it: config.json, generation_config.json, the tokenizer files, the shard
index, the quantisation config. Never code: a .py would be the engine
running what the repository ships, which is to be chosen on purpose.
A version is offered to nodes only whole. The mirror refuses its
manifest (409, with what is missing) while a file is still downloading, or
while a shard its model.safetensors.index.json names, or a part of a
split (-00002-of-00004), is absent. A node reports "not complete on the
mirror yet" and is tried again.
Checksums go with the files. The mirror keeps each file's sha256 beside
it and the manifest carries it; a node checks every file of a set against
it before using the set, and fetches one that differs again.
llmgw mirror add takes several URLs into one version.
A Hugging Face repository page given as a source is read as the
repository id for a model vLLM or SGLang runs, which the engine downloads
itself (the node then needs to reach Hugging Face), and refused for one
llama.cpp runs, which needs a file.

With a vLLM profile of tp: 4, placement reserves four GPUs of one node for
such a model, as before.

Signing up with an address that has an account

On PostgreSQL, signing up with an address already registered showed
"something went wrong; try again": the store recognised a duplicate by the
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
code 23505. It now knows both, and the address is answered as on SQLite —
with the same page as a new sign-up, so the page cannot be used to find out
who has an account. That page now also says what to do if the address had
one: sign in, or ask an operator to reset the password.

An error the page does not explain says the account was not created and
gives a reference (err-…), logged beside the error, for an operator to find
it; the database's own words stay in the log.

An adapter no fabric uses does not keep a node out

llmgw doctor failed a node for a RoCE port with no RoCE v2 GID entry, or for
ib_uverbs not being loaded, whether or not any fabric used them; on a node
added for single-node inference, with no fabric given, that refused it for
RDMA nobody had asked for.

Those are failures only for ports a declared fabric uses, and only when a
fabric is declared. Otherwise they are warnings saying nothing uses them yet.
Which fabric an undeclared InfiniBand port is on, and whether it has a
subnet manager LID, are notes rather than warnings; node add still warns
once that adapters are up with no fabric given.
node add --fabric hands its fabrics to doctor (doctor --fabric name:mlx5_0,…), so they are checked as declared even though agent.yaml is
written afterwards.

A restart with no room beside it

ctl restart places each replacement beside the replica it replaces. With
no room for it — one GPU, every card that fits held by the replica itself —
the old replica was stopped only after the drain grace period, five minutes
of "restart replacing" with nothing saying why, which read as a restart that
never happened; scaling to zero and back was the way out. And a replacement
that fit nowhere even with the old replica gone was stopped for all the same,
leaving the model with nothing. Now:

The replica is the only thing in the way, found by placing the
replacement on a copy of the inventory without it: stopped at once, the
replacement taking its place.
The replacement fits nowhere even then: the restart is given up and the
replica keeps serving. ctl replicas shows restart refused: and why.
Room being made — a lower class preempted, an idle on-demand replica
stopping — is waited for, as before.
ctl restart --in-place stops each replica first, whatever room there is.

Every GPU node gets engine builds it can run, by itself

Which build of an engine a node installs follows its hardware — the CPU
architecture, the driver, and the oldest GPU on it — from what each project
publishes and how it was compiled. Each engine has a list of builds, best
first; the node takes the first its hardware meets, and the first of those
that installs and is seen to compute on the GPU. A build the table is wrong
about costs its download, and the next one is tried.

	llama.cpp	vLLM
Driver 580+, Turing and newer	CUDA 13.3	the version asked for, CUDA 13 (PyPI)
Driver 525–579	CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570	the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch)
Volta (V100)	CUDA 12.8 on driver 570+, else Vulkan	0.19.1, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels
Pascal, Maxwell	CUDA 12.8 on 570+, else Vulkan	none: no vLLM is built for them
arm64 GPU (GH200, Grace)	CUDA 13.3, else Vulkan — never the CPU build	CUDA 13 or 12.9, 8.0 and newer
No NVIDIA GPU	the CPU build	none

Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
starts at 7.5.

A node running an older vLLM says so: the cluster page's vLLM build
column shows the version installed, and the installation's detail says why
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
vLLM is not placed on a V100 by accident.
The install computes on the GPU before it counts: a small tensor
operation for vLLM, the devices llama.cpp lists for its GPU builds.
The build can still be chosen per node on the cluster page (auto, CUDA
13, CUDA 12.9, none), with llmgw ctl engine-build NODE vllm BUILD, or
POST /api/v1/nodes/{id}/engines; kept in the store and sent to the node
at once.
Whether an engine computes on its GPUs is read from every process
nvidia-smi lists, not compute processes alone: the Vulkan build is a
graphics one. A llama.cpp shard is given GGML_VK_VISIBLE_DEVICES beside
CUDA_VISIBLE_DEVICES.

Engines per node, and nodes that do not run them all

A cluster with older GPUs could not add them cleanly: node add refused a
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
and --ignore-checks passed over every other problem with it.

What vLLM cannot do is a warning, in llmgw doctor and node add: the
node joins without it and serves what it does run. node add checks the
vLLM the control plane installs, not the binary's own default.
node add --disable-engine vllm keeps an engine off a node for good:
written as engines.disabled in its agent.yaml, never installed or
advertised there, and not checked by doctor.
engine_overrides in server.yaml give nodes picked by GPU class, id
or labels a different version of an engine, or none — for exceptions; the
build each GPU generation needs is chosen without it (above).
The requirements follow the version, as the table above sets out; the
threshold had been CUDA 13's for every version.
An engine binary named in agent.yaml is used. The agent had installed
its own beside it, which was then used first.
The node page shows each GPU's compute capability.

Serving from the mirror says where a model will run

The serve form put every GGUF model on the CPU: it offered no choice, and its
"Profile name" field, pre-filled "cpu", read as though renaming it would move
the model to a GPU. Placement is decided by gpu_class, which the form never
set. Now:

Runs on lists the fleet's GPU classes with how many cards are free, and
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
cards is chosen.
GPUs per replica sets how many cards a replica takes on one node
(gpus_per_node for llama.cpp, tp for vLLM), and parallel requests
how many a llama.cpp replica serves at once.
The profile is named for where it runs (t4, t4-x4, cpu) rather than
typed.
A line under the form says what the model will run as — engine, CPU or how
many of which GPU, memory, context pool and how it is divided — and turns
red when the CPU is chosen with GPUs free. The message after adding repeats
it.
Choosing the CPU while GPUs are free is refused unless run on the CPU
anyway is ticked.

llama.cpp on several GPUs of one node

A llama.cpp profile takes several of a node's GPUs with gpus_per_node: N:
placement reserves N cards of its gpu_class on one node, the engine sees
those N (CUDA_VISIBLE_DEVICES) and spreads the layers over them
(--split-mode layer, in proportion to each card's free memory, which for
whole cards is even). tp or pp on a llama.cpp profile was a width nothing
placed — the replica stayed unplaced or ran on one card. It is now read as
gpus_per_node, so a profile written that way is placed on that many cards
after the upgrade, rather than dropped; only both given and disagreeing, or
nodes above one, is refused.

Downloads

Caddyfile
1.5 KiB
2026-10-01 16:09:06 +02:00
checksums.txt
645 B
2026-10-01 16:09:06 +02:00
llmgw-v0.10.7-rc.7-darwin-amd64
33 MiB
2026-10-01 16:09:07 +02:00
llmgw-v0.10.7-rc.7-darwin-arm64
31 MiB
2026-10-01 16:09:08 +02:00
llmgw-v0.10.7-rc.7-linux-amd64
33 MiB
2026-10-01 16:09:10 +02:00
llmgw-v0.10.7-rc.7-linux-amd64-nvml
34 MiB
2026-10-01 16:09:11 +02:00
llmgw-v0.10.7-rc.7-linux-arm64
31 MiB
2026-10-01 16:09:12 +02:00
nginx.conf
2.4 KiB
2026-10-01 16:09:12 +02:00
Source Code (ZIP)
Source Code (TAR.GZ)

v0.10.7-rc.6 bd36798b15

v0.10.7-rc.6

Release / build (push) Successful in 1m12s

Details

Release / notes (push) Successful in 4s

Details

CI / test (push) Successful in 3m52s

Details

Pre-Release

gitea-actions released this 2026-10-01 15:42:17 +02:00 | 529 commits to main since this release

From the field report in issue #2, after a deployment on GPU nodes.

GPU metrics that move, and charts of them

A GPU's utilisation and memory in use were read once, when the agent
started, and published from then on: a card busy since looked as idle as it
was at registration, and one idle since as busy. They are now sampled with
the temperatures and power every heartbeat (ten seconds at most), with the
memory bandwidth in use, the SM clock and uncorrected ECC errors besides.
An agent too old to send them still has its registration figures published,
and a figure the driver does not give is left out rather than shown as zero.

New series: llmgw_gpu_memory_utilization_percent,
llmgw_gpu_sm_clock_mhz; llmgw_gpu_utilization_percent,
llmgw_gpu_memory_used_bytes and llmgw_gpu_ecc_errors now from the
sample.
The node page shows each card's utilisation, memory, clock and ECC
errors in its table, and charts of utilisation, memory, temperature and
power, one line per card.
The cluster page charts the fleet's GPUs by class: average utilisation
and memory in use.
Grafana (deploy/monitoring/grafana/llmgw-overview.json) has a GPU row:
utilisation, memory, memory bandwidth, temperature, power against the
limit, clock, and what holds cards back.
Alerts: GPUThermalThrottling (ten minutes slowed by heat),
GPUUncorrectedECCErrors, GPUHot (above 87 °C for ten minutes).

A GPU placement no longer runs on the CPU unnoticed

llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
where the loader looks), came up on the CPU, answered at CPU speed, and was
reported ready while its GPU sat reserved and idle. Three checks now stop it:

Installing the CUDA build asks it which devices it sees
(llama-server --list-devices). None, on a node with GPUs, is a failed
install, with what ldd says libggml-cuda.so is missing. The binary's
own ldd could not say it: the backend is opened at run time.
After the upgrade, engines the agent finds already running are held to
the same check when it adopts them: one serving a GPU placement on the CPU
is stopped, and its replacement starts through the check. The installed
CUDA build is checked again within ten minutes, and installed afresh if it
sees no GPU.
Starting a shard on GPUs is ready only when its process holds memory on
each of them (nvidia-smi --query-compute-apps, its process group
included). One that does not is stopped and reported failed: "placed on GPU
0 but computing on the CPU", with the devices the engine lists. This covers
a hand-installed binary (engines.llamacpp.bin), which nothing else checks.
llmgw ctl engine-check llamacpp, run on a GPU node, now also fails
when the engine sees no GPU, and otherwise prints the devices it computes on.

Shards are also given CUDA_DEVICE_ORDER=PCI_BUS_ID: placement numbers GPUs
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
node with two kinds of GPU, a shard could get a different card from the one it
was given.

A model of many files, mirrored whole for vLLM

vLLM loads a directory — shards, config.json, the tokenizer — and does not
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
shards alone, a node could be handed a version still downloading or missing a
shard, and a repository's page given as a model's source was fetched as one
file of HTML.

The explorer fetches a safetensors set with what the engine reads beside
it: config.json, generation_config.json, the tokenizer files, the shard
index, the quantisation config. Never code: a .py would be the engine
running what the repository ships, which is to be chosen on purpose.
A version is offered to nodes only whole. The mirror refuses its
manifest (409, with what is missing) while a file is still downloading, or
while a shard its model.safetensors.index.json names, or a part of a
split (-00002-of-00004), is absent. A node reports "not complete on the
mirror yet" and is tried again.
Checksums go with the files. The mirror keeps each file's sha256 beside
it and the manifest carries it; a node checks every file of a set against
it before using the set, and fetches one that differs again.
llmgw mirror add takes several URLs into one version.
A Hugging Face repository page given as a source is read as the
repository id for a model vLLM or SGLang runs, which the engine downloads
itself (the node then needs to reach Hugging Face), and refused for one
llama.cpp runs, which needs a file.

With a vLLM profile of tp: 4, placement reserves four GPUs of one node for
such a model, as before.

Signing up with an address that has an account

On PostgreSQL, signing up with an address already registered showed
"something went wrong; try again": the store recognised a duplicate by the
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
code 23505. It now knows both, and the address is answered as on SQLite —
with the same page as a new sign-up, so the page cannot be used to find out
who has an account. That page now also says what to do if the address had
one: sign in, or ask an operator to reset the password.

An error the page does not explain says the account was not created and
gives a reference (err-…), logged beside the error, for an operator to find
it; the database's own words stay in the log.

An adapter no fabric uses does not keep a node out

llmgw doctor failed a node for a RoCE port with no RoCE v2 GID entry, or for
ib_uverbs not being loaded, whether or not any fabric used them; on a node
added for single-node inference, with no fabric given, that refused it for
RDMA nobody had asked for.

Those are failures only for ports a declared fabric uses, and only when a
fabric is declared. Otherwise they are warnings saying nothing uses them yet.
Which fabric an undeclared InfiniBand port is on, and whether it has a
subnet manager LID, are notes rather than warnings; node add still warns
once that adapters are up with no fabric given.
node add --fabric hands its fabrics to doctor (doctor --fabric name:mlx5_0,…), so they are checked as declared even though agent.yaml is
written afterwards.

A restart with no room beside it

ctl restart places each replacement beside the replica it replaces. With
no room for it — one GPU, every card that fits held by the replica itself —
the old replica was stopped only after the drain grace period, five minutes
of "restart replacing" with nothing saying why, which read as a restart that
never happened; scaling to zero and back was the way out. And a replacement
that fit nowhere even with the old replica gone was stopped for all the same,
leaving the model with nothing. Now:

The replica is the only thing in the way, found by placing the
replacement on a copy of the inventory without it: stopped at once, the
replacement taking its place.
The replacement fits nowhere even then: the restart is given up and the
replica keeps serving. ctl replicas shows restart refused: and why.
Room being made — a lower class preempted, an idle on-demand replica
stopping — is waited for, as before.
ctl restart --in-place stops each replica first, whatever room there is.

Every GPU node gets engine builds it can run, by itself

Which build of an engine a node installs follows its hardware — the CPU
architecture, the driver, and the oldest GPU on it — from what each project
publishes and how it was compiled. Each engine has a list of builds, best
first; the node takes the first its hardware meets, and the first of those
that installs and is seen to compute on the GPU. A build the table is wrong
about costs its download, and the next one is tried.

	llama.cpp	vLLM
Driver 580+, Turing and newer	CUDA 13.3	the version asked for, CUDA 13 (PyPI)
Driver 525–579	CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570	the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch)
Volta (V100)	CUDA 12.8 on driver 570+, else Vulkan	0.19.1, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels
Pascal, Maxwell	CUDA 12.8 on 570+, else Vulkan	none: no vLLM is built for them
arm64 GPU (GH200, Grace)	CUDA 13.3, else Vulkan — never the CPU build	CUDA 13 or 12.9, 8.0 and newer
No NVIDIA GPU	the CPU build	none

Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
starts at 7.5.

A node running an older vLLM says so: the cluster page's vLLM build
column shows the version installed, and the installation's detail says why
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
vLLM is not placed on a V100 by accident.
The install computes on the GPU before it counts: a small tensor
operation for vLLM, the devices llama.cpp lists for its GPU builds.
The build can still be chosen per node on the cluster page (auto, CUDA
13, CUDA 12.9, none), with llmgw ctl engine-build NODE vllm BUILD, or
POST /api/v1/nodes/{id}/engines; kept in the store and sent to the node
at once.
Whether an engine computes on its GPUs is read from every process
nvidia-smi lists, not compute processes alone: the Vulkan build is a
graphics one. A llama.cpp shard is given GGML_VK_VISIBLE_DEVICES beside
CUDA_VISIBLE_DEVICES.

Engines per node, and nodes that do not run them all

A cluster with older GPUs could not add them cleanly: node add refused a
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
and --ignore-checks passed over every other problem with it.

What vLLM cannot do is a warning, in llmgw doctor and node add: the
node joins without it and serves what it does run. node add checks the
vLLM the control plane installs, not the binary's own default.
node add --disable-engine vllm keeps an engine off a node for good:
written as engines.disabled in its agent.yaml, never installed or
advertised there, and not checked by doctor.
engine_overrides in server.yaml give nodes picked by GPU class, id
or labels a different version of an engine, or none — for exceptions; the
build each GPU generation needs is chosen without it (above).
The requirements follow the version, as the table above sets out; the
threshold had been CUDA 13's for every version.
An engine binary named in agent.yaml is used. The agent had installed
its own beside it, which was then used first.
The node page shows each GPU's compute capability.

Serving from the mirror says where a model will run

The serve form put every GGUF model on the CPU: it offered no choice, and its
"Profile name" field, pre-filled "cpu", read as though renaming it would move
the model to a GPU. Placement is decided by gpu_class, which the form never
set. Now:

Runs on lists the fleet's GPU classes with how many cards are free, and
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
cards is chosen.
GPUs per replica sets how many cards a replica takes on one node
(gpus_per_node for llama.cpp, tp for vLLM), and parallel requests
how many a llama.cpp replica serves at once.
The profile is named for where it runs (t4, t4-x4, cpu) rather than
typed.
A line under the form says what the model will run as — engine, CPU or how
many of which GPU, memory, context pool and how it is divided — and turns
red when the CPU is chosen with GPUs free. The message after adding repeats
it.
Choosing the CPU while GPUs are free is refused unless run on the CPU
anyway is ticked.

llama.cpp on several GPUs of one node

A llama.cpp profile takes several of a node's GPUs with gpus_per_node: N:
placement reserves N cards of its gpu_class on one node, the engine sees
those N (CUDA_VISIBLE_DEVICES) and spreads the layers over them
(--split-mode layer, in proportion to each card's free memory, which for
whole cards is even). tp or pp on a llama.cpp profile was a width nothing
placed — the replica stayed unplaced or ran on one card. It is now read as
gpus_per_node, so a profile written that way is placed on that many cards
after the upgrade, rather than dropped; only both given and disagreeing, or
nodes above one, is refused.

Downloads

Caddyfile
1.5 KiB
2026-10-01 15:43:26 +02:00
checksums.txt
645 B
2026-10-01 15:43:26 +02:00
llmgw-v0.10.7-rc.6-darwin-amd64
33 MiB
2026-10-01 15:43:27 +02:00
llmgw-v0.10.7-rc.6-darwin-arm64
31 MiB
2026-10-01 15:43:28 +02:00
llmgw-v0.10.7-rc.6-linux-amd64
33 MiB
2026-10-01 15:43:30 +02:00
llmgw-v0.10.7-rc.6-linux-amd64-nvml
34 MiB
2026-10-01 15:43:31 +02:00
llmgw-v0.10.7-rc.6-linux-arm64
31 MiB
2026-10-01 15:43:32 +02:00
nginx.conf
2.4 KiB
2026-10-01 15:43:32 +02:00
Source Code (ZIP)
Source Code (TAR.GZ)

v0.10.7-rc.5 dc483d8c68

v0.10.7-rc.5

Release / build (push) Successful in 1m12s

Details

CI / test (push) Successful in 3m52s

Details

Release / notes (push) Successful in 3s

Details

Pre-Release

gitea-actions released this 2026-10-01 14:35:58 +02:00 | 530 commits to main since this release

From the field report in issue #2, after a deployment on GPU nodes.

A GPU placement no longer runs on the CPU unnoticed

llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
where the loader looks), came up on the CPU, answered at CPU speed, and was
reported ready while its GPU sat reserved and idle. Three checks now stop it:

Installing the CUDA build asks it which devices it sees
(llama-server --list-devices). None, on a node with GPUs, is a failed
install, with what ldd says libggml-cuda.so is missing. The binary's
own ldd could not say it: the backend is opened at run time.
After the upgrade, engines the agent finds already running are held to
the same check when it adopts them: one serving a GPU placement on the CPU
is stopped, and its replacement starts through the check. The installed
CUDA build is checked again within ten minutes, and installed afresh if it
sees no GPU.
Starting a shard on GPUs is ready only when its process holds memory on
each of them (nvidia-smi --query-compute-apps, its process group
included). One that does not is stopped and reported failed: "placed on GPU
0 but computing on the CPU", with the devices the engine lists. This covers
a hand-installed binary (engines.llamacpp.bin), which nothing else checks.
llmgw ctl engine-check llamacpp, run on a GPU node, now also fails
when the engine sees no GPU, and otherwise prints the devices it computes on.

Shards are also given CUDA_DEVICE_ORDER=PCI_BUS_ID: placement numbers GPUs
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
node with two kinds of GPU, a shard could get a different card from the one it
was given.

A model of many files, mirrored whole for vLLM

vLLM loads a directory — shards, config.json, the tokenizer — and does not
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
shards alone, a node could be handed a version still downloading or missing a
shard, and a repository's page given as a model's source was fetched as one
file of HTML.

The explorer fetches a safetensors set with what the engine reads beside
it: config.json, generation_config.json, the tokenizer files, the shard
index, the quantisation config. Never code: a .py would be the engine
running what the repository ships, which is to be chosen on purpose.
A version is offered to nodes only whole. The mirror refuses its
manifest (409, with what is missing) while a file is still downloading, or
while a shard its model.safetensors.index.json names, or a part of a
split (-00002-of-00004), is absent. A node reports "not complete on the
mirror yet" and is tried again.
Checksums go with the files. The mirror keeps each file's sha256 beside
it and the manifest carries it; a node checks every file of a set against
it before using the set, and fetches one that differs again.
llmgw mirror add takes several URLs into one version.
A Hugging Face repository page given as a source is read as the
repository id for a model vLLM or SGLang runs, which the engine downloads
itself (the node then needs to reach Hugging Face), and refused for one
llama.cpp runs, which needs a file.

With a vLLM profile of tp: 4, placement reserves four GPUs of one node for
such a model, as before.

Signing up with an address that has an account

On PostgreSQL, signing up with an address already registered showed
"something went wrong; try again": the store recognised a duplicate by the
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
code 23505. It now knows both, and the address is answered as on SQLite —
with the same page as a new sign-up, so the page cannot be used to find out
who has an account. That page now also says what to do if the address had
one: sign in, or ask an operator to reset the password.

An error the page does not explain says the account was not created and
gives a reference (err-…), logged beside the error, for an operator to find
it; the database's own words stay in the log.

An adapter no fabric uses does not keep a node out

llmgw doctor failed a node for a RoCE port with no RoCE v2 GID entry, or for
ib_uverbs not being loaded, whether or not any fabric used them; on a node
added for single-node inference, with no fabric given, that refused it for
RDMA nobody had asked for.

Those are failures only for ports a declared fabric uses, and only when a
fabric is declared. Otherwise they are warnings saying nothing uses them yet.
Which fabric an undeclared InfiniBand port is on, and whether it has a
subnet manager LID, are notes rather than warnings; node add still warns
once that adapters are up with no fabric given.
node add --fabric hands its fabrics to doctor (doctor --fabric name:mlx5_0,…), so they are checked as declared even though agent.yaml is
written afterwards.

A restart with no room beside it

ctl restart places each replacement beside the replica it replaces. With
no room for it — one GPU, every card that fits held by the replica itself —
the old replica was stopped only after the drain grace period, five minutes
of "restart replacing" with nothing saying why, which read as a restart that
never happened; scaling to zero and back was the way out. And a replacement
that fit nowhere even with the old replica gone was stopped for all the same,
leaving the model with nothing. Now:

The replica is the only thing in the way, found by placing the
replacement on a copy of the inventory without it: stopped at once, the
replacement taking its place.
The replacement fits nowhere even then: the restart is given up and the
replica keeps serving. ctl replicas shows restart refused: and why.
Room being made — a lower class preempted, an idle on-demand replica
stopping — is waited for, as before.
ctl restart --in-place stops each replica first, whatever room there is.

Every GPU node gets engine builds it can run, by itself

Which build of an engine a node installs follows its hardware — the CPU
architecture, the driver, and the oldest GPU on it — from what each project
publishes and how it was compiled. Each engine has a list of builds, best
first; the node takes the first its hardware meets, and the first of those
that installs and is seen to compute on the GPU. A build the table is wrong
about costs its download, and the next one is tried.

	llama.cpp	vLLM
Driver 580+, Turing and newer	CUDA 13.3	the version asked for, CUDA 13 (PyPI)
Driver 525–579	CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570	the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch)
Volta (V100)	CUDA 12.8 on driver 570+, else Vulkan	0.19.1, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels
Pascal, Maxwell	CUDA 12.8 on 570+, else Vulkan	none: no vLLM is built for them
arm64 GPU (GH200, Grace)	CUDA 13.3, else Vulkan — never the CPU build	CUDA 13 or 12.9, 8.0 and newer
No NVIDIA GPU	the CPU build	none

Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
starts at 7.5.

A node running an older vLLM says so: the cluster page's vLLM build
column shows the version installed, and the installation's detail says why
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
vLLM is not placed on a V100 by accident.
The install computes on the GPU before it counts: a small tensor
operation for vLLM, the devices llama.cpp lists for its GPU builds.
The build can still be chosen per node on the cluster page (auto, CUDA
13, CUDA 12.9, none), with llmgw ctl engine-build NODE vllm BUILD, or
POST /api/v1/nodes/{id}/engines; kept in the store and sent to the node
at once.
Whether an engine computes on its GPUs is read from every process
nvidia-smi lists, not compute processes alone: the Vulkan build is a
graphics one. A llama.cpp shard is given GGML_VK_VISIBLE_DEVICES beside
CUDA_VISIBLE_DEVICES.

Engines per node, and nodes that do not run them all

A cluster with older GPUs could not add them cleanly: node add refused a
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
and --ignore-checks passed over every other problem with it.

What vLLM cannot do is a warning, in llmgw doctor and node add: the
node joins without it and serves what it does run. node add checks the
vLLM the control plane installs, not the binary's own default.
node add --disable-engine vllm keeps an engine off a node for good:
written as engines.disabled in its agent.yaml, never installed or
advertised there, and not checked by doctor.
engine_overrides in server.yaml give nodes picked by GPU class, id
or labels a different version of an engine, or none — for exceptions; the
build each GPU generation needs is chosen without it (above).
The requirements follow the version, as the table above sets out; the
threshold had been CUDA 13's for every version.
An engine binary named in agent.yaml is used. The agent had installed
its own beside it, which was then used first.
The node page shows each GPU's compute capability.

Serving from the mirror says where a model will run

The serve form put every GGUF model on the CPU: it offered no choice, and its
"Profile name" field, pre-filled "cpu", read as though renaming it would move
the model to a GPU. Placement is decided by gpu_class, which the form never
set. Now:

Runs on lists the fleet's GPU classes with how many cards are free, and
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
cards is chosen.
GPUs per replica sets how many cards a replica takes on one node
(gpus_per_node for llama.cpp, tp for vLLM), and parallel requests
how many a llama.cpp replica serves at once.
The profile is named for where it runs (t4, t4-x4, cpu) rather than
typed.
A line under the form says what the model will run as — engine, CPU or how
many of which GPU, memory, context pool and how it is divided — and turns
red when the CPU is chosen with GPUs free. The message after adding repeats
it.
Choosing the CPU while GPUs are free is refused unless run on the CPU
anyway is ticked.

llama.cpp on several GPUs of one node

A llama.cpp profile takes several of a node's GPUs with gpus_per_node: N:
placement reserves N cards of its gpu_class on one node, the engine sees
those N (CUDA_VISIBLE_DEVICES) and spreads the layers over them
(--split-mode layer, in proportion to each card's free memory, which for
whole cards is even). tp or pp on a llama.cpp profile was a width nothing
placed — the replica stayed unplaced or ran on one card. It is now read as
gpus_per_node, so a profile written that way is placed on that many cards
after the upgrade, rather than dropped; only both given and disagreeing, or
nodes above one, is refused.

Downloads

Caddyfile
1.5 KiB
2026-10-01 14:37:07 +02:00
checksums.txt
645 B
2026-10-01 14:37:07 +02:00
llmgw-v0.10.7-rc.5-darwin-amd64
33 MiB
2026-10-01 14:37:08 +02:00
llmgw-v0.10.7-rc.5-darwin-arm64
31 MiB
2026-10-01 14:37:10 +02:00
llmgw-v0.10.7-rc.5-linux-amd64
33 MiB
2026-10-01 14:37:11 +02:00
llmgw-v0.10.7-rc.5-linux-amd64-nvml
34 MiB
2026-10-01 14:37:12 +02:00
llmgw-v0.10.7-rc.5-linux-arm64
31 MiB
2026-10-01 14:37:13 +02:00
nginx.conf
2.4 KiB
2026-10-01 14:37:13 +02:00
Source Code (ZIP)
Source Code (TAR.GZ)

v0.10.7-rc.4 43d7c4eac1

v0.10.7-rc.4

Release / build (push) Successful in 1m11s

Details

Release / notes (push) Successful in 3s

Details

CI / test (push) Successful in 3m51s

Details

Pre-Release

gitea-actions released this 2026-10-01 14:20:11 +02:00 | 531 commits to main since this release

From the field report in issue #2, after a deployment on GPU nodes.

A GPU placement no longer runs on the CPU unnoticed

llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
where the loader looks), came up on the CPU, answered at CPU speed, and was
reported ready while its GPU sat reserved and idle. Three checks now stop it:

Installing the CUDA build asks it which devices it sees
(llama-server --list-devices). None, on a node with GPUs, is a failed
install, with what ldd says libggml-cuda.so is missing. The binary's
own ldd could not say it: the backend is opened at run time.
After the upgrade, engines the agent finds already running are held to
the same check when it adopts them: one serving a GPU placement on the CPU
is stopped, and its replacement starts through the check. The installed
CUDA build is checked again within ten minutes, and installed afresh if it
sees no GPU.
Starting a shard on GPUs is ready only when its process holds memory on
each of them (nvidia-smi --query-compute-apps, its process group
included). One that does not is stopped and reported failed: "placed on GPU
0 but computing on the CPU", with the devices the engine lists. This covers
a hand-installed binary (engines.llamacpp.bin), which nothing else checks.
llmgw ctl engine-check llamacpp, run on a GPU node, now also fails
when the engine sees no GPU, and otherwise prints the devices it computes on.

Shards are also given CUDA_DEVICE_ORDER=PCI_BUS_ID: placement numbers GPUs
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
node with two kinds of GPU, a shard could get a different card from the one it
was given.

A model of many files, mirrored whole for vLLM

vLLM loads a directory — shards, config.json, the tokenizer — and does not
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
shards alone, a node could be handed a version still downloading or missing a
shard, and a repository's page given as a model's source was fetched as one
file of HTML.

The explorer fetches a safetensors set with what the engine reads beside
it: config.json, generation_config.json, the tokenizer files, the shard
index, the quantisation config. Never code: a .py would be the engine
running what the repository ships, which is to be chosen on purpose.
A version is offered to nodes only whole. The mirror refuses its
manifest (409, with what is missing) while a file is still downloading, or
while a shard its model.safetensors.index.json names, or a part of a
split (-00002-of-00004), is absent. A node reports "not complete on the
mirror yet" and is tried again.
Checksums go with the files. The mirror keeps each file's sha256 beside
it and the manifest carries it; a node checks every file of a set against
it before using the set, and fetches one that differs again.
llmgw mirror add takes several URLs into one version.
A Hugging Face repository page given as a source is read as the
repository id for a model vLLM or SGLang runs, which the engine downloads
itself (the node then needs to reach Hugging Face), and refused for one
llama.cpp runs, which needs a file.

With a vLLM profile of tp: 4, placement reserves four GPUs of one node for
such a model, as before.

Signing up with an address that has an account

On PostgreSQL, signing up with an address already registered showed
"something went wrong; try again": the store recognised a duplicate by the
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
code 23505. It now knows both, and the address is answered as on SQLite —
with the same page as a new sign-up, so the page cannot be used to find out
who has an account. That page now also says what to do if the address had
one: sign in, or ask an operator to reset the password.

An error the page does not explain says the account was not created and
gives a reference (err-…), logged beside the error, for an operator to find
it; the database's own words stay in the log.

An adapter no fabric uses does not keep a node out

llmgw doctor failed a node for a RoCE port with no RoCE v2 GID entry, or for
ib_uverbs not being loaded, whether or not any fabric used them; on a node
added for single-node inference, with no fabric given, that refused it for
RDMA nobody had asked for.

Those are failures only for ports a declared fabric uses, and only when a
fabric is declared. Otherwise they are warnings saying nothing uses them yet.
Which fabric an undeclared InfiniBand port is on, and whether it has a
subnet manager LID, are notes rather than warnings; node add still warns
once that adapters are up with no fabric given.
node add --fabric hands its fabrics to doctor (doctor --fabric name:mlx5_0,…), so they are checked as declared even though agent.yaml is
written afterwards.

A restart with no room beside it

ctl restart places each replacement beside the replica it replaces. With
no room for it — one GPU, every card that fits held by the replica itself —
the old replica was stopped only after the drain grace period, five minutes
of "restart replacing" with nothing saying why, which read as a restart that
never happened; scaling to zero and back was the way out. And a replacement
that fit nowhere even with the old replica gone was stopped for all the same,
leaving the model with nothing. Now:

The replica is the only thing in the way, found by placing the
replacement on a copy of the inventory without it: stopped at once, the
replacement taking its place.
The replacement fits nowhere even then: the restart is given up and the
replica keeps serving. ctl replicas shows restart refused: and why.
Room being made — a lower class preempted, an idle on-demand replica
stopping — is waited for, as before.
ctl restart --in-place stops each replica first, whatever room there is.

Every GPU node gets a vLLM it can run, chosen on the cluster page

The vLLM a node installs is the build its hardware runs: CUDA 13 from PyPI
where the driver (580+) allows, and otherwise the same version's CUDA 12.9
build from vLLM's GitHub release, pinned to GitHub's digest, with PyTorch's
CUDA 12.9 torch (driver 525+). One version across the cluster, so the flags
llmgw builds — checked against that version — hold on every node.

Volta (V100) is not covered by either build. Read from vLLM's release
pipeline: every x86 build from 0.20.0 on — CUDA 13 and CUDA 12.9 alike — has
kernels for compute capability 7.5 and up (aarch64: 8.0 and up). The last
vLLM built for 7.0 is 0.19.1, whose PyPI wheel is CUDA 12 on PyPI's torch 2.10
(also built for 7.0). A V100 node asked for a newer vLLM is refused, saying
so; give such nodes 0.19.1 with engine_overrides by gpu_class. llmgw's
launch flags are checked against 0.30.0, so run llmgw ctl engine-check vllm
on such a node once it is installed.

The install computes on the GPU before it counts: a small tensor
operation in the new environment. A build without kernels for the card
imports and prints its version, and failed only on its first model; now it
fails to install, saying so.
The build is chosen per node on the cluster page (vLLM build: auto,
CUDA 13, CUDA 12.9, none), with llmgw ctl engine-build NODE vllm BUILD, or
POST /api/v1/nodes/{id}/engines. The choice is kept in the store and sent
to the node at once.

Engines per node, and nodes that do not run them all

A cluster with older GPUs could not add them cleanly: node add refused a
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
and --ignore-checks passed over every other problem with it.

What vLLM cannot do is a warning, in llmgw doctor and node add: the
node joins without it and serves what it does run. node add checks the
vLLM the control plane installs, not the binary's own default.
node add --disable-engine vllm keeps an engine off a node for good:
written as engines.disabled in its agent.yaml, never installed or
advertised there, and not checked by doctor.
engine_overrides in server.yaml give nodes picked by GPU class, id
or labels a different version of an engine, or none: vLLM 0.19.1, the last
built for Volta, on V100s beside the current one on newer cards.
The requirements follow the version. vLLM before 0.20 is CUDA 12 on
PyPI (driver 525+), and only up to 0.19.1 has Volta kernels; the threshold
had been CUDA 13's for every version, so the doctor's advice to set an older
version could not work. A refusal now says which version would.
An engine binary named in agent.yaml is used. The agent had installed
its own beside it, which was then used first.
The node page shows each GPU's compute capability.

Serving from the mirror says where a model will run

The serve form put every GGUF model on the CPU: it offered no choice, and its
"Profile name" field, pre-filled "cpu", read as though renaming it would move
the model to a GPU. Placement is decided by gpu_class, which the form never
set. Now:

Runs on lists the fleet's GPU classes with how many cards are free, and
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
cards is chosen.
GPUs per replica sets how many cards a replica takes on one node
(gpus_per_node for llama.cpp, tp for vLLM), and parallel requests
how many a llama.cpp replica serves at once.
The profile is named for where it runs (t4, t4-x4, cpu) rather than
typed.
A line under the form says what the model will run as — engine, CPU or how
many of which GPU, memory, context pool and how it is divided — and turns
red when the CPU is chosen with GPUs free. The message after adding repeats
it.
Choosing the CPU while GPUs are free is refused unless run on the CPU
anyway is ticked.

llama.cpp on several GPUs of one node

A llama.cpp profile takes several of a node's GPUs with gpus_per_node: N:
placement reserves N cards of its gpu_class on one node, the engine sees
those N (CUDA_VISIBLE_DEVICES) and spreads the layers over them
(--split-mode layer, in proportion to each card's free memory, which for
whole cards is even). tp or pp on a llama.cpp profile was a width nothing
placed — the replica stayed unplaced or ran on one card. It is now read as
gpus_per_node, so a profile written that way is placed on that many cards
after the upgrade, rather than dropped; only both given and disagreeing, or
nodes above one, is refused.

Downloads

Caddyfile
1.5 KiB
2026-10-01 14:21:16 +02:00
checksums.txt
645 B
2026-10-01 14:21:16 +02:00
llmgw-v0.10.7-rc.4-darwin-amd64
33 MiB
2026-10-01 14:21:18 +02:00
llmgw-v0.10.7-rc.4-darwin-arm64
31 MiB
2026-10-01 14:21:19 +02:00
llmgw-v0.10.7-rc.4-linux-amd64
33 MiB
2026-10-01 14:21:20 +02:00
llmgw-v0.10.7-rc.4-linux-amd64-nvml
34 MiB
2026-10-01 14:21:22 +02:00
llmgw-v0.10.7-rc.4-linux-arm64
31 MiB
2026-10-01 14:21:23 +02:00
nginx.conf
2.4 KiB
2026-10-01 14:21:23 +02:00
Source Code (ZIP)
Source Code (TAR.GZ)

v0.10.7-rc.3 86e9a1cb83

v0.10.7-rc.3

Release / build (push) Successful in 1m13s

Details

Release / notes (push) Successful in 4s

Details

CI / test (push) Successful in 3m47s

Details

Pre-Release

gitea-actions released this 2026-10-01 14:04:45 +02:00 | 532 commits to main since this release

From the field report in issue #2, after a deployment on GPU nodes.

A GPU placement no longer runs on the CPU unnoticed

llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
where the loader looks), came up on the CPU, answered at CPU speed, and was
reported ready while its GPU sat reserved and idle. Three checks now stop it:

Installing the CUDA build asks it which devices it sees
(llama-server --list-devices). None, on a node with GPUs, is a failed
install, with what ldd says libggml-cuda.so is missing. The binary's
own ldd could not say it: the backend is opened at run time.
Starting a shard on GPUs is ready only when its process holds memory on
each of them (nvidia-smi --query-compute-apps, its process group
included). One that does not is stopped and reported failed: "placed on GPU
0 but computing on the CPU", with the devices the engine lists. This covers
a hand-installed binary (engines.llamacpp.bin), which nothing else checks.
llmgw ctl engine-check llamacpp, run on a GPU node, now also fails
when the engine sees no GPU, and otherwise prints the devices it computes on.

Shards are also given CUDA_DEVICE_ORDER=PCI_BUS_ID: placement numbers GPUs
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
node with two kinds of GPU, a shard could get a different card from the one it
was given.

A model of many files, mirrored whole for vLLM

vLLM loads a directory — shards, config.json, the tokenizer — and does not
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
shards alone, a node could be handed a version still downloading or missing a
shard, and a repository's page given as a model's source was fetched as one
file of HTML.

The explorer fetches a safetensors set with what the engine reads beside
it: config.json, generation_config.json, the tokenizer files, the shard
index, the quantisation config. Never code: a .py would be the engine
running what the repository ships, which is to be chosen on purpose.
A version is offered to nodes only whole. The mirror refuses its
manifest (409, with what is missing) while a file is still downloading, or
while a shard its model.safetensors.index.json names, or a part of a
split (-00002-of-00004), is absent. A node reports "not complete on the
mirror yet" and is tried again.
Checksums go with the files. The mirror keeps each file's sha256 beside
it and the manifest carries it; a node checks every file of a set against
it before using the set, and fetches one that differs again.
llmgw mirror add takes several URLs into one version.
A Hugging Face repository page is refused as a source, saying to mirror
it or to give the repository id for vLLM to fetch itself.

With a vLLM profile of tp: 4, placement reserves four GPUs of one node for
such a model, as before.

Signing up with an address that has an account

On PostgreSQL, signing up with an address already registered showed
"something went wrong; try again": the store recognised a duplicate by the
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
code 23505. It now knows both, and the address is answered as on SQLite —
with the same page as a new sign-up, so the page cannot be used to find out
who has an account. That page now also says what to do if the address had
one: sign in, or ask an operator to reset the password.

An error the page does not explain says the account was not created and
gives a reference (err-…), logged beside the error, for an operator to find
it; the database's own words stay in the log.

An adapter no fabric uses does not keep a node out

llmgw doctor failed a node for a RoCE port with no RoCE v2 GID entry, or for
ib_uverbs not being loaded, whether or not any fabric used them; on a node
added for single-node inference, with no fabric given, that refused it for
RDMA nobody had asked for.

Those are failures only for ports a declared fabric uses, and only when a
fabric is declared. Otherwise they are warnings saying nothing uses them yet.
Which fabric an undeclared InfiniBand port is on, and whether it has a
subnet manager LID, are notes rather than warnings; node add still warns
once that adapters are up with no fabric given.
node add --fabric hands its fabrics to doctor (doctor --fabric name:mlx5_0,…), so they are checked as declared even though agent.yaml is
written afterwards.

A restart with no room beside it

ctl restart places each replacement beside the replica it replaces. With
no room for it — one GPU, every card that fits held by the replica itself —
the old replica was stopped only after the drain grace period, five minutes
of "restart replacing" with nothing saying why, which read as a restart that
never happened; scaling to zero and back was the way out. And a replacement
that fit nowhere even with the old replica gone was stopped for all the same,
leaving the model with nothing. Now:

The replica is the only thing in the way, found by placing the
replacement on a copy of the inventory without it: stopped at once, the
replacement taking its place.
The replacement fits nowhere even then: the restart is given up and the
replica keeps serving. ctl replicas shows restart refused: and why.
Room being made — a lower class preempted, an idle on-demand replica
stopping — is waited for, as before.
ctl restart --in-place stops each replica first, whatever room there is.

Every GPU node gets a vLLM it can run, chosen on the cluster page

The vLLM a node installs is the build its hardware runs: CUDA 13 from PyPI
where the driver (580+) allows, and otherwise the same version's CUDA 12.9
build from vLLM's GitHub release, pinned to GitHub's digest, with PyTorch's
CUDA 12.9 torch (driver 525+). One version across the cluster, so the flags
llmgw builds — checked against that version — hold on every node.

Volta (V100) is not covered by either build. Read from vLLM's release
pipeline: every x86 build from 0.20.0 on — CUDA 13 and CUDA 12.9 alike — has
kernels for compute capability 7.5 and up (aarch64: 8.0 and up). The last
vLLM built for 7.0 is 0.19.1, whose PyPI wheel is CUDA 12 on PyPI's torch 2.10
(also built for 7.0). A V100 node asked for a newer vLLM is refused, saying
so; give such nodes 0.19.1 with engine_overrides by gpu_class. llmgw's
launch flags are checked against 0.30.0, so run llmgw ctl engine-check vllm
on such a node once it is installed.

The install computes on the GPU before it counts: a small tensor
operation in the new environment. A build without kernels for the card
imports and prints its version, and failed only on its first model; now it
fails to install, saying so.
The build is chosen per node on the cluster page (vLLM build: auto,
CUDA 13, CUDA 12.9, none), with llmgw ctl engine-build NODE vllm BUILD, or
POST /api/v1/nodes/{id}/engines. The choice is kept in the store and sent
to the node at once.

Engines per node, and nodes that do not run them all

A cluster with older GPUs could not add them cleanly: node add refused a
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
and --ignore-checks passed over every other problem with it.

What vLLM cannot do is a warning, in llmgw doctor and node add: the
node joins without it and serves what it does run. node add checks the
vLLM the control plane installs, not the binary's own default.
node add --disable-engine vllm keeps an engine off a node for good:
written as engines.disabled in its agent.yaml, never installed or
advertised there, and not checked by doctor.
engine_overrides in server.yaml give nodes picked by GPU class, id
or labels a different version of an engine, or none: vLLM 0.19.1, the last
built for Volta, on V100s beside the current one on newer cards.
The requirements follow the version. vLLM before 0.20 is CUDA 12 on
PyPI (driver 525+), and only up to 0.19.1 has Volta kernels; the threshold
had been CUDA 13's for every version, so the doctor's advice to set an older
version could not work. A refusal now says which version would.
An engine binary named in agent.yaml is used. The agent had installed
its own beside it, which was then used first.
The node page shows each GPU's compute capability.

Serving from the mirror says where a model will run

The serve form put every GGUF model on the CPU: it offered no choice, and its
"Profile name" field, pre-filled "cpu", read as though renaming it would move
the model to a GPU. Placement is decided by gpu_class, which the form never
set. Now:

Runs on lists the fleet's GPU classes with how many cards are free, and
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
cards is chosen.
GPUs per replica sets how many cards a replica takes on one node
(gpus_per_node for llama.cpp, tp for vLLM), and parallel requests
how many a llama.cpp replica serves at once.
The profile is named for where it runs (t4, t4-x4, cpu) rather than
typed.
A line under the form says what the model will run as — engine, CPU or how
many of which GPU, memory, context pool and how it is divided — and turns
red when the CPU is chosen with GPUs free. The message after adding repeats
it.
Choosing the CPU while GPUs are free is refused unless run on the CPU
anyway is ticked.

llama.cpp on several GPUs of one node

A llama.cpp profile takes several of a node's GPUs with gpus_per_node: N:
placement reserves N cards of its gpu_class on one node, the engine sees
those N (CUDA_VISIBLE_DEVICES) and spreads the layers over them
(--split-mode layer, in proportion to each card's free memory, which for
whole cards is even). tp or pp on a llama.cpp profile was a width nothing
placed — the replica stayed unplaced or ran on one card — and is now refused
when the model is added, saying to write gpus_per_node instead; so is
nodes above one.

Downloads

Caddyfile
1.5 KiB
2026-10-01 14:05:52 +02:00
checksums.txt
645 B
2026-10-01 14:05:52 +02:00
llmgw-v0.10.7-rc.3-darwin-amd64
33 MiB
2026-10-01 14:05:53 +02:00
llmgw-v0.10.7-rc.3-darwin-arm64
31 MiB
2026-10-01 14:05:54 +02:00
llmgw-v0.10.7-rc.3-linux-amd64
33 MiB
2026-10-01 14:05:56 +02:00
llmgw-v0.10.7-rc.3-linux-amd64-nvml
34 MiB
2026-10-01 14:05:57 +02:00
llmgw-v0.10.7-rc.3-linux-arm64
31 MiB
2026-10-01 14:05:58 +02:00
nginx.conf
2.4 KiB
2026-10-01 14:05:58 +02:00
Source Code (ZIP)
Source Code (TAR.GZ)

v0.10.7-rc.2 4f280a3546

v0.10.7-rc.2

Release / build (push) Successful in 1m12s

Details

Release / notes (push) Successful in 3s

Details

CI / test (push) Successful in 3m55s

Details

Pre-Release

gitea-actions released this 2026-10-01 12:39:14 +02:00 | 533 commits to main since this release

From the field report in issue #2, after a deployment on GPU nodes.

A GPU placement no longer runs on the CPU unnoticed

llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
where the loader looks), came up on the CPU, answered at CPU speed, and was
reported ready while its GPU sat reserved and idle. Three checks now stop it:

Installing the CUDA build asks it which devices it sees
(llama-server --list-devices). None, on a node with GPUs, is a failed
install, with what ldd says libggml-cuda.so is missing. The binary's
own ldd could not say it: the backend is opened at run time.
Starting a shard on GPUs is ready only when its process holds memory on
each of them (nvidia-smi --query-compute-apps, its process group
included). One that does not is stopped and reported failed: "placed on GPU
0 but computing on the CPU", with the devices the engine lists. This covers
a hand-installed binary (engines.llamacpp.bin), which nothing else checks.
llmgw ctl engine-check llamacpp, run on a GPU node, now also fails
when the engine sees no GPU, and otherwise prints the devices it computes on.

Shards are also given CUDA_DEVICE_ORDER=PCI_BUS_ID: placement numbers GPUs
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
node with two kinds of GPU, a shard could get a different card from the one it
was given.

A model of many files, mirrored whole for vLLM

vLLM loads a directory — shards, config.json, the tokenizer — and does not
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
shards alone, a node could be handed a version still downloading or missing a
shard, and a repository's page given as a model's source was fetched as one
file of HTML.

The explorer fetches a safetensors set with what the engine reads beside
it: config.json, generation_config.json, the tokenizer files, the shard
index, the quantisation config. Never code: a .py would be the engine
running what the repository ships, which is to be chosen on purpose.
A version is offered to nodes only whole. The mirror refuses its
manifest (409, with what is missing) while a file is still downloading, or
while a shard its model.safetensors.index.json names, or a part of a
split (-00002-of-00004), is absent. A node reports "not complete on the
mirror yet" and is tried again.
Checksums go with the files. The mirror keeps each file's sha256 beside
it and the manifest carries it; a node checks every file of a set against
it before using the set, and fetches one that differs again.
llmgw mirror add takes several URLs into one version.
A Hugging Face repository page is refused as a source, saying to mirror
it or to give the repository id for vLLM to fetch itself.

With a vLLM profile of tp: 4, placement reserves four GPUs of one node for
such a model, as before.

Signing up with an address that has an account

On PostgreSQL, signing up with an address already registered showed
"something went wrong; try again": the store recognised a duplicate by the
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
code 23505. It now knows both, and the address is answered as on SQLite —
with the same page as a new sign-up, so the page cannot be used to find out
who has an account. That page now also says what to do if the address had
one: sign in, or ask an operator to reset the password.

An error the page does not explain says the account was not created and
gives a reference (err-…), logged beside the error, for an operator to find
it; the database's own words stay in the log.

An adapter no fabric uses does not keep a node out

llmgw doctor failed a node for a RoCE port with no RoCE v2 GID entry, or for
ib_uverbs not being loaded, whether or not any fabric used them; on a node
added for single-node inference, with no fabric given, that refused it for
RDMA nobody had asked for.

Those are failures only for ports a declared fabric uses, and only when a
fabric is declared. Otherwise they are warnings saying nothing uses them yet.
Which fabric an undeclared InfiniBand port is on, and whether it has a
subnet manager LID, are notes rather than warnings; node add still warns
once that adapters are up with no fabric given.
node add --fabric hands its fabrics to doctor (doctor --fabric name:mlx5_0,…), so they are checked as declared even though agent.yaml is
written afterwards.

A restart with no room beside it

ctl restart places each replacement beside the replica it replaces. With
no room for it — one GPU, every card that fits held by the replica itself —
the old replica was stopped only after the drain grace period, five minutes
of "restart replacing" with nothing saying why, which read as a restart that
never happened; scaling to zero and back was the way out. And a replacement
that fit nowhere even with the old replica gone was stopped for all the same,
leaving the model with nothing. Now:

The replica is the only thing in the way, found by placing the
replacement on a copy of the inventory without it: stopped at once, the
replacement taking its place.
The replacement fits nowhere even then: the restart is given up and the
replica keeps serving. ctl replicas shows restart refused: and why.
Room being made — a lower class preempted, an idle on-demand replica
stopping — is waited for, as before.
ctl restart --in-place stops each replica first, whatever room there is.

Every GPU node gets a vLLM it can run, chosen on the cluster page

The vLLM a node installs is the build its hardware runs: CUDA 13 from PyPI
where the driver (580+) and the GPUs (Turing+) allow, and otherwise the same
version's CUDA 12.9 build from vLLM's GitHub release, pinned to GitHub's
digest, with PyTorch's CUDA 12.9 torch (driver 525+, Volta+). One version
across the cluster, so the flags llmgw builds — checked against that version
— hold on every node.

The install computes on the GPU before it counts: a small tensor
operation in the new environment. A build without kernels for the card
imports and prints its version, and failed only on its first model; now it
fails to install, saying so.
The build is chosen per node on the cluster page (vLLM build: auto,
CUDA 13, CUDA 12.9, none), with llmgw ctl engine-build NODE vllm BUILD, or
POST /api/v1/nodes/{id}/engines. The choice is kept in the store and sent
to the node at once.

Engines per node, and nodes that do not run them all

A cluster with older GPUs could not add them cleanly: node add refused a
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
and --ignore-checks passed over every other problem with it.

What vLLM cannot do is a warning, in llmgw doctor and node add: the
node joins without it and serves what it does run. node add checks the
vLLM the control plane installs, not the binary's own default.
node add --disable-engine vllm keeps an engine off a node for good:
written as engines.disabled in its agent.yaml, never installed or
advertised there, and not checked by doctor.
engine_overrides in server.yaml give nodes picked by GPU class, id
or labels a different version of an engine, or none: a CUDA 12 vLLM on
Volta beside a CUDA 13 one on newer cards.
The CUDA requirement follows the version. vLLM before 0.26 and SGLang
before 0.5 are built for CUDA 12 and are installed on driver 525 and Volta;
the threshold had been CUDA 13's for every version, so the doctor's advice
to set an older version could not work. A refusal now says which versions
would.
An engine binary named in agent.yaml is used. The agent had installed
its own beside it, which was then used first.
The node page shows each GPU's compute capability.

Serving from the mirror says where a model will run

The serve form put every GGUF model on the CPU: it offered no choice, and its
"Profile name" field, pre-filled "cpu", read as though renaming it would move
the model to a GPU. Placement is decided by gpu_class, which the form never
set. Now:

Runs on lists the fleet's GPU classes with how many cards are free, and
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
cards is chosen.
GPUs per replica sets how many cards a replica takes on one node
(gpus_per_node for llama.cpp, tp for vLLM), and parallel requests
how many a llama.cpp replica serves at once.
The profile is named for where it runs (t4, t4-x4, cpu) rather than
typed.
A line under the form says what the model will run as — engine, CPU or how
many of which GPU, memory, context pool and how it is divided — and turns
red when the CPU is chosen with GPUs free. The message after adding repeats
it.
Choosing the CPU while GPUs are free is refused unless run on the CPU
anyway is ticked.

llama.cpp on several GPUs of one node

A llama.cpp profile takes several of a node's GPUs with gpus_per_node: N:
placement reserves N cards of its gpu_class on one node, the engine sees
those N (CUDA_VISIBLE_DEVICES) and spreads the layers over them
(--split-mode layer, in proportion to each card's free memory, which for
whole cards is even). tp or pp on a llama.cpp profile was a width nothing
placed — the replica stayed unplaced or ran on one card — and is now refused
when the model is added, saying to write gpus_per_node instead; so is
nodes above one.

Downloads

Caddyfile
1.5 KiB
2026-10-01 12:40:19 +02:00
checksums.txt
645 B
2026-10-01 12:40:19 +02:00
llmgw-v0.10.7-rc.2-darwin-amd64
33 MiB
2026-10-01 12:40:20 +02:00
llmgw-v0.10.7-rc.2-darwin-arm64
31 MiB
2026-10-01 12:40:21 +02:00
llmgw-v0.10.7-rc.2-linux-amd64
33 MiB
2026-10-01 12:40:22 +02:00
llmgw-v0.10.7-rc.2-linux-amd64-nvml
34 MiB
2026-10-01 12:40:24 +02:00
llmgw-v0.10.7-rc.2-linux-arm64
31 MiB
2026-10-01 12:40:25 +02:00
nginx.conf
2.4 KiB
2026-10-01 12:40:25 +02:00
Source Code (ZIP)
Source Code (TAR.GZ)

v0.10.7-rc.1 2735c5a318

v0.10.7-rc.1

CI / test (push) Successful in 3m58s

Details

Release / build (push) Successful in 1m12s

Details

Release / notes (push) Successful in 3s

Details

Pre-Release

gitea-actions released this 2026-10-01 12:21:04 +02:00 | 534 commits to main since this release

From the field report in issue #2, after a deployment on GPU nodes.

A GPU placement no longer runs on the CPU unnoticed

llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
where the loader looks), came up on the CPU, answered at CPU speed, and was
reported ready while its GPU sat reserved and idle. Three checks now stop it:

Installing the CUDA build asks it which devices it sees
(llama-server --list-devices). None, on a node with GPUs, is a failed
install, with what ldd says libggml-cuda.so is missing. The binary's
own ldd could not say it: the backend is opened at run time.
Starting a shard on GPUs is ready only when its process holds memory on
each of them (nvidia-smi --query-compute-apps, its process group
included). One that does not is stopped and reported failed: "placed on GPU
0 but computing on the CPU", with the devices the engine lists. This covers
a hand-installed binary (engines.llamacpp.bin), which nothing else checks.
llmgw ctl engine-check llamacpp, run on a GPU node, now also fails
when the engine sees no GPU, and otherwise prints the devices it computes on.

Shards are also given CUDA_DEVICE_ORDER=PCI_BUS_ID: placement numbers GPUs
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
node with two kinds of GPU, a shard could get a different card from the one it
was given.

A model of many files, mirrored whole for vLLM

vLLM loads a directory — shards, config.json, the tokenizer — and does not
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
shards alone, a node could be handed a version still downloading or missing a
shard, and a repository's page given as a model's source was fetched as one
file of HTML.

The explorer fetches a safetensors set with what the engine reads beside
it: config.json, generation_config.json, the tokenizer files, the shard
index, the quantisation config. Never code: a .py would be the engine
running what the repository ships, which is to be chosen on purpose.
A version is offered to nodes only whole. The mirror refuses its
manifest (409, with what is missing) while a file is still downloading, or
while a shard its model.safetensors.index.json names, or a part of a
split (-00002-of-00004), is absent. A node reports "not complete on the
mirror yet" and is tried again.
Checksums go with the files. The mirror keeps each file's sha256 beside
it and the manifest carries it; a node checks every file of a set against
it before using the set, and fetches one that differs again.
llmgw mirror add takes several URLs into one version.
A Hugging Face repository page is refused as a source, saying to mirror
it or to give the repository id for vLLM to fetch itself.

With a vLLM profile of tp: 4, placement reserves four GPUs of one node for
such a model, as before.

Signing up with an address that has an account

On PostgreSQL, signing up with an address already registered showed
"something went wrong; try again": the store recognised a duplicate by the
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
code 23505. It now knows both, and the address is answered as on SQLite —
with the same page as a new sign-up, so the page cannot be used to find out
who has an account. That page now also says what to do if the address had
one: sign in, or ask an operator to reset the password.

An error the page does not explain says the account was not created and
gives a reference (err-…), logged beside the error, for an operator to find
it; the database's own words stay in the log.

An adapter no fabric uses does not keep a node out

llmgw doctor failed a node for a RoCE port with no RoCE v2 GID entry, or for
ib_uverbs not being loaded, whether or not any fabric used them; on a node
added for single-node inference, with no fabric given, that refused it for
RDMA nobody had asked for.

Those are failures only for ports a declared fabric uses, and only when a
fabric is declared. Otherwise they are warnings saying nothing uses them yet.
Which fabric an undeclared InfiniBand port is on, and whether it has a
subnet manager LID, are notes rather than warnings; node add still warns
once that adapters are up with no fabric given.
node add --fabric hands its fabrics to doctor (doctor --fabric name:mlx5_0,…), so they are checked as declared even though agent.yaml is
written afterwards.

A restart with no room beside it

ctl restart places each replacement beside the replica it replaces. With
no room for it — one GPU, every card that fits held by the replica itself —
the old replica was stopped only after the drain grace period, five minutes
of "restart replacing" with nothing saying why, which read as a restart that
never happened; scaling to zero and back was the way out. And a replacement
that fit nowhere even with the old replica gone was stopped for all the same,
leaving the model with nothing. Now:

The replica is the only thing in the way, found by placing the
replacement on a copy of the inventory without it: stopped at once, the
replacement taking its place.
The replacement fits nowhere even then: the restart is given up and the
replica keeps serving. ctl replicas shows restart refused: and why.
Room being made — a lower class preempted, an idle on-demand replica
stopping — is waited for, as before.
ctl restart --in-place stops each replica first, whatever room there is.

Engines per node, and nodes that do not run them all

A cluster with older GPUs could not add them cleanly: node add refused a
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
and --ignore-checks passed over every other problem with it.

What vLLM cannot do is a warning, in llmgw doctor and node add: the
node joins without it and serves what it does run. node add checks the
vLLM the control plane installs, not the binary's own default.
node add --disable-engine vllm keeps an engine off a node for good:
written as engines.disabled in its agent.yaml, never installed or
advertised there, and not checked by doctor.
engine_overrides in server.yaml give nodes picked by GPU class, id
or labels a different version of an engine, or none: a CUDA 12 vLLM on
Volta beside a CUDA 13 one on newer cards.
The CUDA requirement follows the version. vLLM before 0.26 and SGLang
before 0.5 are built for CUDA 12 and are installed on driver 525 and Volta;
the threshold had been CUDA 13's for every version, so the doctor's advice
to set an older version could not work. A refusal now says which versions
would.
An engine binary named in agent.yaml is used. The agent had installed
its own beside it, which was then used first.
The node page shows each GPU's compute capability.

Serving from the mirror says where a model will run

The serve form put every GGUF model on the CPU: it offered no choice, and its
"Profile name" field, pre-filled "cpu", read as though renaming it would move
the model to a GPU. Placement is decided by gpu_class, which the form never
set. Now:

Runs on lists the fleet's GPU classes with how many cards are free, and
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
cards is chosen.
GPUs per replica sets how many cards a replica takes on one node
(gpus_per_node for llama.cpp, tp for vLLM), and parallel requests
how many a llama.cpp replica serves at once.
The profile is named for where it runs (t4, t4-x4, cpu) rather than
typed.
A line under the form says what the model will run as — engine, CPU or how
many of which GPU, memory, context pool and how it is divided — and turns
red when the CPU is chosen with GPUs free. The message after adding repeats
it.
Choosing the CPU while GPUs are free is refused unless run on the CPU
anyway is ticked.

llama.cpp on several GPUs of one node

A llama.cpp profile takes several of a node's GPUs with gpus_per_node: N:
placement reserves N cards of its gpu_class on one node, the engine sees
those N (CUDA_VISIBLE_DEVICES) and spreads the layers over them
(--split-mode layer, in proportion to each card's free memory, which for
whole cards is even). tp or pp on a llama.cpp profile was a width nothing
placed — the replica stayed unplaced or ran on one card — and is now refused
when the model is added, saying to write gpus_per_node instead; so is
nodes above one.

Downloads

Caddyfile
1.5 KiB
2026-10-01 12:22:24 +02:00
checksums.txt
645 B
2026-10-01 12:22:24 +02:00
llmgw-v0.10.7-rc.1-darwin-amd64
33 MiB
2026-10-01 12:22:25 +02:00
llmgw-v0.10.7-rc.1-darwin-arm64
31 MiB
2026-10-01 12:22:27 +02:00
llmgw-v0.10.7-rc.1-linux-amd64
33 MiB
2026-10-01 12:22:28 +02:00
llmgw-v0.10.7-rc.1-linux-amd64-nvml
34 MiB
2026-10-01 12:22:29 +02:00
llmgw-v0.10.7-rc.1-linux-arm64
31 MiB
2026-10-01 12:22:30 +02:00
nginx.conf
2.4 KiB
2026-10-01 12:22:30 +02:00
Source Code (ZIP)
Source Code (TAR.GZ)

v0.10.6 0c7934ba96

v0.10.6

Release / build (push) Successful in 1m11s

Details

Release / notes (push) Successful in 4s

Details

CI / test (push) Successful in 3m51s

Details

Stable

gitea-actions released this 2026-10-01 08:52:01 +02:00 | 543 commits to main since this release

Published first as release candidates v0.10.6-rc.1 to rc.8; rc.5 failed its
format check and has no binaries.

Tried on the PoC cluster before release: LoRA adapters asked for by name
through the gateway, two on one model, each answering as trained and the model
itself unchanged; a saturated pool spilling to the next (8 of 12 concurrent
requests on the preferred replica, 4 on the other); and a bronze request
displaced from a full queue by a gold one, with 503 and Retry-After 5. Not tried
live: a model served from an external provider, which needs a provider key.

Capacity from the admin pages

What v0.10.5 made settable from the command line is now on the pages too:

Quotas has a Capacity section: every scope's guaranteed and most GPUs,
cores and memory beside what it holds, with a form to set, change or remove
one. It goes through the same checks as llmgw ctl quota.
A model's card on the mirror page sets its class, its owner (when there
are quotas to count against), and the memory and cores a replica reserves,
beside its replica counts. Only what changed is applied, and each change is
said.
Room counts what a class may take. The card, llmgw ctl model target and
the headroom API count the replicas a model could get by stopping those its
class outranks, separately from free room, and name whose they would be. A
critical model's card no longer refuses a replica that preemption would make
room for.

LoRA adapters

A llama.cpp model can carry adapters, each a name and a GGUF LoRA file. The
nodes fetch them with the model's weights and llama.cpp loads them all at start,
applied to nothing (--lora-scaled FILE:0); a request that names an
adapter goes to the model's replicas with "lora": [{"id": i, "scale": s}],
so one replica serves the model and all its adapters. Adapter names share the
namespace of models, are listed in /v1/models, and follow key allowlists on
their own — a key allowed an adapter does not reach the model under it. Adapter
files are kept from weight cleanup like the model's own. Not yet for vLLM or
SGLang profiles, whose adapters are loaded differently.

Finding and attaching adapters from the mirror page

Explore has a LoRA adapters view: adapters for any model, or — from a
held model's card, find adapters on Hugging Face — those made for the model
its repository was made from, since a GGUF repository is a quantisation and
adapters are made for the original.
Opening a repository reads the first bytes of each GGUF file and marks it
a LoRA adapter or a model. Repositories tagged "lora" hold a model with
the adapter merged in about as often as an adapter, and only the file says.
Held adapter files are marked in the list, and a model's card lists its
adapters, offers the held adapter files of its architecture to attach under a
name and scale, and removes them. A file that is not an adapter, or is for
another architecture, is refused before it reaches the catalog.
llmgw ctl model adapters MODEL [--add NAME=URL[@SCALE]] [--remove NAME]
lists and changes them from the command line, through
POST /api/v1/models/{name}/adapters, with the same checks.

Models served outside the cluster

A catalog model can be an OpenAI-compatible API elsewhere: external: {url, model, api_key_file} and no profiles. Nothing is placed for it; the router
sends its requests to the provider, rewritten to the provider's name for the
model and with its key, read from the file on every publish so a rotated key is
picked up. Named in another model's fallbacks, it is used only when the
cluster has no capacity left. It is never scraped for metrics, and never reads
as stale. Prices, budgets, limits, usage and key allowlists apply as to any
model — and since its prompts leave the cluster, the allowlist is how to decide
who may send them (docs/install.md). ctl models and the cluster page mark it
external.

Pool spillover

A model served from pools of different priority — H100 before A100, say — was
routed to the preferred pool however full it was. Now a pool whose replicas are
saturated, by the same measure admission uses (queue past its healthy depth, or
KV cache past its ceiling), passes requests to the next pool that is not. When
every pool is saturated the preferred one is kept: a pool that is just as full
gains nothing.

Displacement: a higher band is not refused while a lower one waits

Each band's queue was bounded, and nothing bounded them together, so a full
gateway turned an interactive request away while batch requests sat queued
behind it. admission.queue_capacity_total now bounds everything queued —
by default half of every band's capacity together. Reached, a request is not
refused while a lower band has any queued: the newest request of the lowest
band is turned away instead, with a 503 and Retry-After: 5, so it backs off
and returns. The newest, because the oldest have waited longest and are
nearest their TTL. Counted as displaced in the admission rejection metric.

Releases can be signed

llmgw release keygen makes an ed25519 key pair. With the private key in the
CI secret RELEASE_SIGNING_KEY, releases carry checksums.txt.sig; with the
public key in internal/release/trusted.pub, every binary built after refuses
a release not signed with it, wherever it installs one. Off until then, and
docs/install.md says in which order to turn it on.

Restarts borrow a lower class's room

A restart places the replacement beside the replica it replaces, and where
there is no free room it now makes some the way any placement does: by
stopping a replica its class outranks, which gets its room back once the old
replica has gone. The in-place restart — the model unserved while its
replacement loads — is left for when nothing of a lower class is running. On a
cluster where every model is standard, that is every full one: mark the models
that can wait opportunistic (llmgw ctl model class, or the model's card).

Downloads

Caddyfile
1.5 KiB
2026-10-01 08:53:09 +02:00
checksums.txt
620 B
2026-10-01 08:53:09 +02:00
llmgw-v0.10.6-darwin-amd64
33 MiB
2026-10-01 08:53:10 +02:00
llmgw-v0.10.6-darwin-arm64
31 MiB
2026-10-01 08:53:11 +02:00
llmgw-v0.10.6-linux-amd64
33 MiB
2026-10-01 08:53:13 +02:00
llmgw-v0.10.6-linux-amd64-nvml
34 MiB
2026-10-01 08:53:14 +02:00
llmgw-v0.10.6-linux-arm64
31 MiB
2026-10-01 08:53:15 +02:00
nginx.conf
2.4 KiB
2026-10-01 08:53:15 +02:00
Source Code (ZIP)
Source Code (TAR.GZ)

95 Releases 95 Tags

v0.10.7-rc.9 Release / build (push) Successful in 1m12s Details Release / notes (push) Successful in 3s Details CI / test (push) Successful in 3m54s Details Pre-Release

GPU metrics that move, and charts of them

The chat page says whether a GPU or a CPU answered

A card shared with something else is judged every heartbeat

A GPU placement no longer runs on the CPU unnoticed

A model of many files, mirrored whole for vLLM

Signing up with an address that has an account

An adapter no fabric uses does not keep a node out

A restart with no room beside it

Every GPU node gets engine builds it can run, by itself

Engines per node, and nodes that do not run them all

Serving from the mirror says where a model will run

llama.cpp on several GPUs of one node

v0.10.7-rc.8 Release / build (push) Successful in 1m14s Details Release / notes (push) Successful in 4s Details CI / test (push) Successful in 3m50s Details Pre-Release

GPU metrics that move, and charts of them

A card shared with something else is judged every heartbeat

A GPU placement no longer runs on the CPU unnoticed

A model of many files, mirrored whole for vLLM

Signing up with an address that has an account

An adapter no fabric uses does not keep a node out

A restart with no room beside it

Every GPU node gets engine builds it can run, by itself

Engines per node, and nodes that do not run them all

Serving from the mirror says where a model will run

llama.cpp on several GPUs of one node

v0.10.7-rc.7 Release / build (push) Successful in 1m11s Details Release / notes (push) Successful in 3s Details CI / test (push) Successful in 3m48s Details Pre-Release

GPU metrics that move, and charts of them

A GPU placement no longer runs on the CPU unnoticed

A model of many files, mirrored whole for vLLM

Signing up with an address that has an account

An adapter no fabric uses does not keep a node out

A restart with no room beside it

Every GPU node gets engine builds it can run, by itself

Engines per node, and nodes that do not run them all

Serving from the mirror says where a model will run

llama.cpp on several GPUs of one node

v0.10.7-rc.6 Release / build (push) Successful in 1m12s Details Release / notes (push) Successful in 4s Details CI / test (push) Successful in 3m52s Details Pre-Release

GPU metrics that move, and charts of them

A GPU placement no longer runs on the CPU unnoticed

A model of many files, mirrored whole for vLLM

Signing up with an address that has an account

An adapter no fabric uses does not keep a node out

A restart with no room beside it

Every GPU node gets engine builds it can run, by itself

Engines per node, and nodes that do not run them all

Serving from the mirror says where a model will run

llama.cpp on several GPUs of one node

v0.10.7-rc.5 Release / build (push) Successful in 1m12s Details CI / test (push) Successful in 3m52s Details Release / notes (push) Successful in 3s Details Pre-Release

A GPU placement no longer runs on the CPU unnoticed

A model of many files, mirrored whole for vLLM

Signing up with an address that has an account

An adapter no fabric uses does not keep a node out

A restart with no room beside it

Every GPU node gets engine builds it can run, by itself

Engines per node, and nodes that do not run them all

Serving from the mirror says where a model will run

llama.cpp on several GPUs of one node

v0.10.7-rc.4 Release / build (push) Successful in 1m11s Details Release / notes (push) Successful in 3s Details CI / test (push) Successful in 3m51s Details Pre-Release

A GPU placement no longer runs on the CPU unnoticed

A model of many files, mirrored whole for vLLM

Signing up with an address that has an account

An adapter no fabric uses does not keep a node out

A restart with no room beside it

Every GPU node gets a vLLM it can run, chosen on the cluster page

Engines per node, and nodes that do not run them all

Serving from the mirror says where a model will run

llama.cpp on several GPUs of one node

v0.10.7-rc.3 Release / build (push) Successful in 1m13s Details Release / notes (push) Successful in 4s Details CI / test (push) Successful in 3m47s Details Pre-Release

A GPU placement no longer runs on the CPU unnoticed

A model of many files, mirrored whole for vLLM

Signing up with an address that has an account

An adapter no fabric uses does not keep a node out

A restart with no room beside it

Every GPU node gets a vLLM it can run, chosen on the cluster page

Engines per node, and nodes that do not run them all

Serving from the mirror says where a model will run

llama.cpp on several GPUs of one node

v0.10.7-rc.2 Release / build (push) Successful in 1m12s Details Release / notes (push) Successful in 3s Details CI / test (push) Successful in 3m55s Details Pre-Release

A GPU placement no longer runs on the CPU unnoticed

v0.10.7-rc.9

Release / build (push) Successful in 1m12s

Details

Release / notes (push) Successful in 3s

Details

CI / test (push) Successful in 3m54s

Details

Pre-Release

v0.10.7-rc.8

Release / build (push) Successful in 1m14s

Details

Release / notes (push) Successful in 4s

Details

CI / test (push) Successful in 3m50s

Details

Pre-Release

v0.10.7-rc.7

Release / build (push) Successful in 1m11s

Details

Release / notes (push) Successful in 3s

Details

CI / test (push) Successful in 3m48s

Details

Pre-Release

v0.10.7-rc.6

Release / build (push) Successful in 1m12s

Details

Release / notes (push) Successful in 4s

Details

CI / test (push) Successful in 3m52s

Details

Pre-Release

v0.10.7-rc.5

Release / build (push) Successful in 1m12s

Details

CI / test (push) Successful in 3m52s

Details

Release / notes (push) Successful in 3s

Details

Pre-Release

v0.10.7-rc.4

Release / build (push) Successful in 1m11s

Details

Release / notes (push) Successful in 3s

Details

CI / test (push) Successful in 3m51s

Details

Pre-Release

v0.10.7-rc.3

Release / build (push) Successful in 1m13s

Details

Release / notes (push) Successful in 4s

Details

CI / test (push) Successful in 3m47s

Details

Pre-Release

v0.10.7-rc.2

Release / build (push) Successful in 1m12s

Details

Release / notes (push) Successful in 3s

Details

CI / test (push) Successful in 3m55s

Details

Pre-Release

v0.10.7-rc.1

CI / test (push) Successful in 3m58s

Details

Release / build (push) Successful in 1m12s

Details

Release / notes (push) Successful in 3s

Details

Pre-Release

v0.10.6

Release / build (push) Successful in 1m11s

Details

Release / notes (push) Successful in 4s

Details

CI / test (push) Successful in 3m51s

Details

Stable