-
v0.12.1
Stable
gitea-actions
released this
2026-10-06 10:52:24 +02:00 | 319 commits to main since this releaseThe guardrails' judge, served and watched without server init: a judge
missing, served as the wrong kind of model, or not answering is said on
ctl statusand the console, andllmgw ctl guard judge enableputs it
right. Nothing to do to upgrade.Added
llmgw ctl guard judge statusandenable: each judge model the
policies ask, whether it is in the catalog, as a judge, with replicas
ready and answering — and serving the missing one, Llama Guard 3 1B on
CPUs under the name the policies give it, with--replacefor a model of
that name that is not a judge. No moreserver initto put a judge back.
Also Do → Serve the guardrails' judge in the console.- A missing or broken judge is said where it is looked for:
llmgw ctl statusand the console's Overview list each judge with a problem — not in
the catalog, not a judge, no replica ready, or failing to answer — and
what that does to requests (refused or unjudged).
Fixed
- A Llama Guard file served from the mirror page is the judge: it was
added as a model clients use, under whatever name the form had, so the
policies never found it and refused everything they judged
(guard_unavailable). It is now added as a judge, under the name the
policies ask. ctlsays so when the API answers with a page: a command a control
plane does not know — the leader on an older release, through a standby —
failed with "invalid character '<'"; it now says which and why.
Downloads
-
v0.12.0
Stable
gitea-actions
released this
2026-10-06 09:39:54 +02:00 | 321 commits to main since this releaseA minor version, because where the configuration lives changes: everything in
server.yaml that is not one machine's own is now the cluster's settings, kept
in the store, the same on every control plane and in force within seconds of
a change. Nothing needs doing by hand to upgrade. The first control plane to
start on v0.12.0 brings its server.yaml's cluster keys into the store, with
the baseline guard policy when the file had none. From then on, change them
withllmgw ctl configor the console's Settings, not in server.yaml, whose
cluster keys are ignored, with a warning. Upgrade every control plane (llmgw ctl upgradedoes, the leader last): until the leader runs v0.12.0,
llmgw ctl configthrough a standby is refused. Nodes need v0.12.0 to follow
the firewall switch.Changed
- The cluster's settings live in the store, and every change is in force
within seconds. Everything in server.yaml that is not one machine's own
(guardrails, limits and prices, routing, engines, notifications, the
mirror's policy, declared models, offline, federation, facility budgets) is
a versioned document in the store, the same on every control plane.
llmgw ctl configshows, sets, edits, lists and rolls it back, and the
console's Settings screen edits it. A change is checked, kept with who made
it and why, and applied: guardrails live, anything else by every control
plane starting again on it, standbys first and the leader last. A change to
a machine's own keys in its server.yaml is noticed and applied the same way.
No "restart now?" any more, and no second copy to keep in step. The first
start against a store brings in the file's cluster keys, with the baseline
guard policy when the file had none, so an upgraded cluster gets its
guardrails; after that, those keys in a server.yaml are ignored, with a
warning.
Added
- llmgw's firewall can be turned off and on for the whole cluster:
llmgw ctl config set firewall.enabled false. Every node's agent takes its
table out within seconds, and each control plane's new
llmgw-firewall-sync.pathdoes the same there;trueputs the rules back
as they were.firewall_networkschanges reach the nodes the same way. llmgw wiperemoves llmgw from a machine — services, firewall table,
configuration, the authority, secrets, state, users — after listing it and
asking for the machine's name;--models,--storeand--alltake
weights, the store's database here, backups and the binary too. Also
Do → Wipe this machine in the console.
Fixed
- The mirror is given an address the nodes reach:
server initand
mirror inittook the machine's first address for the mirror's--base,
which on a machine with Docker could be 172.17.0.1 — a node then asked its
own bridge for every model and was refused. They now take the source of
the default route, else the first address not on a container's or a
bridge's interface. Andnode addhas the node reach the mirror's files
before it is added: one that cannot fails the checks, naming the address
and what to change. - A standby that becomes the leader reads the store again: models added
or removed, engine builds chosen and domains held through the leader before
it, rather than placing on what it read when it started. - A node on another network joins:
node addset up the node's firewall
but left the control plane's to the networks it was installed with, so a node
on another subnet was installed and then never heard from (lost). It now lets
the node in on the control plane it runs on — its subnet, or its address
when a router is between them — and says so; a network already let in
changes nothing.
Downloads
- The cluster's settings live in the store, and every change is in force
-
v0.11.4
Stable
gitea-actions
released this
2026-10-05 12:29:05 +02:00 | 329 commits to main since this releaseTwo control planes, run in anger on the reference cluster.
Added
-
The console sets up a second control plane: Do → Add a control plane
asks for the machine and a floating address and runsserver add. -
Who leads, everywhere.
ctl statusand the console's Overview have a
CONTROLline: which control plane leads, which stand by, one that does not
answer, and whether the floating address is on the leader. -
A standby's console is the leader's. Its admin API passes
/api/to the
leader over the agent port, with the cluster's certificates both ways, so
sudo llmgwandllmgw ctlshow the same cluster on either machine. Control
plane certificates are now issued for client use too;server addreissues
older ones. -
ctl upgradein waves. Nodes are upgraded one, then two, four… up to
--parallel(20) at once, so hundreds take minutes; a wave that does not
come back stops it. The other control plane is upgraded first, over ssh. -
server addsays when the mirror's--api-allowwould refuse the new
control plane. -
Guardrails in the console: a view of the policies and what each checks,
whether their judge models are serving, where they are attached (attach
and detach from it), and what they did in a window.GET /api/v1/guard/eventsanswers the same, andGET /api/v1/guardnow says
what each policy does. -
A standby's portal is the leader's. Its client port passes everything
but/health,/readyand/leaderto the leader, and its admin port the
dashboard as well as the API; a proxy left pointing at a standby showed an
empty cluster. The leader trusts the control planes'X-Forwarded-For. -
ctl model serve --toolssays in the catalog that a model makes tool
calls, for agents and editors.
Fixed
- Continue's agent did nothing. The configuration the portal writes told
Continue every model uses tools, and nothing of its context: a model whose
calls came back as text stalled the agent, and prompts over a CPU
replica's 4096 tokens were refused. It now saystool_useonly for a
model whose catalog says tools, and the context and longest answer the
gateway allows. ctl model servegave llama.cpp four slots whatever was asked: a
--context-length 32768 --ctx-pool 32768model had slots of 8192 and
refused what the gateway admitted. Slots are now what the pool holds of
the context. The longest answer grows with the context (a quarter, from
1024 to 8192) rather than being 1024 for every model.ctl upgradeupgrades the leader last wherever it is run; run on the
standby, it upgraded the leader first and the leadership moved twice.- Failover took 40 seconds; it takes a few. A stopping control plane held
every agent's stream for the thirty seconds of a graceful shutdown; it now
gives up its lease first and lets the agents go at once. - A control plane that lost its lease while running kept its agents, leaving
the new leader with no nodes. It now lets them go. - An agent's reconnect backoff carried over from one outage to the next, so
each failover made the next one slower; it starts again after a session that
worked. - The
CONTROLline named no floating address.
Downloads
-
-
v0.11.3-rc.2
Pre-Release
gitea-actions
released this
2026-10-05 06:39:51 +02:00 | 333 commits to main since this releaseTwo control planes, run in anger on the reference cluster.
Added
-
The console sets up a second control plane: Do → Add a control plane
asks for the machine and a floating address and runsserver add. -
Who leads, everywhere.
ctl statusand the console's Overview have a
CONTROLline: which control plane leads, which stand by, one that does not
answer, and whether the floating address is on the leader. -
A standby's console is the leader's. Its admin API passes
/api/to the
leader over the agent port, with the cluster's certificates both ways, so
sudo llmgwandllmgw ctlshow the same cluster on either machine. Control
plane certificates are now issued for client use too;server addreissues
older ones. -
ctl upgradein waves. Nodes are upgraded one, then two, four… up to
--parallel(20) at once, so hundreds take minutes; a wave that does not
come back stops it. The other control plane is upgraded first, over ssh. -
server addsays when the mirror's--api-allowwould refuse the new
control plane. -
Guardrails in the console: a view of the policies and what each checks,
whether their judge models are serving, where they are attached (attach
and detach from it), and what they did in a window.GET /api/v1/guard/eventsanswers the same, andGET /api/v1/guardnow says
what each policy does. -
A standby's portal is the leader's. Its client port passes everything
but/health,/readyand/leaderto the leader, and its admin port the
dashboard as well as the API; a proxy left pointing at a standby showed an
empty cluster. The leader trusts the control planes'X-Forwarded-For.
Fixed
- Failover took 40 seconds; it takes a few. A stopping control plane held
every agent's stream for the thirty seconds of a graceful shutdown; it now
gives up its lease first and lets the agents go at once. - A control plane that lost its lease while running kept its agents, leaving
the new leader with no nodes. It now lets them go. - An agent's reconnect backoff carried over from one outage to the next, so
each failover made the next one slower; it starts again after a session that
worked. - The
CONTROLline named no floating address.
Downloads
-
-
v0.11.3-rc.1
Stable
gitea-actions
released this
2026-10-05 02:44:55 +02:00 | 336 commits to main since this releaseDownloads
-
v0.11.2
Stable
gitea-actions
released this
2026-10-05 01:37:28 +02:00 | 343 commits to main since this releaseRouting by what each vLLM replica's cache holds
- A vLLM profile can set
kv_events: true. The agent then starts the engine
publishing its KV cache events (--kv-events-config) to a port on
127.0.0.1 the agent listens on, with--prefix-caching-hash-algo sha256_cborandPYTHONHASHSEED=1729, so its block hashes can be
computed outside Python. The events never leave the node. - The agent mirrors each engine's set of cached blocks and sends the net
changes up its existing stream to the control plane four times a second
(newKVEventsagent message), starting every new stream with a full
copy. Lost events (a gap in vLLM's sequence numbers), an engine restart or
AllBlocksClearedstart a replica's index again; blocks offloaded to CPU,
remote ones and a hybrid model's second cache group are not indexed. - The control plane passes them to the router's index, only for replicas
the sending node runs, and empties what a stream described when it closes. - For a model with
kv_events, the router asks one of its replicas to
tokenize the request (vLLM's/tokenize, with messages and tools, so the
chat template is applied exactly), hashes the tokens into blocks as vLLM
does, and scores replicas by the share of the prompt's leading blocks they
hold. The imbalance guard still applies. With no answer — no events yet,
an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls
back to the approximate tree (router.prefix) or to load. - The agent checks first blocks against llmgw's hashing; an engine that
hashes another way (a profile overriding the algorithm, a changed vLLM) is
reported in the agent's log andllmgw_kv_index_problem, and routed as
though it published nothing. - New settings
router.kv_index.tokenize_timeout(250ms),min_blocks(4)
andmax_blocks(4096). New metricsllmgw_kv_index_decisions_total
(kv_hit,kv_miss_min_load,load_balance_override,
fallback_no_index,fallback_unsupported,fallback_tokenize_failed),
llmgw_kv_index_tokenize_seconds,llmgw_kv_index_blocks. - Experimental: tested against vLLM's event format and hashing as its source
defines them, and against libzmq and msgspec publishing — not yet against a
running vLLM.
LoRA adapters on vLLM, loaded on demand
- vLLM models can carry adapters. A PEFT adapter directory (a mirror
version ending in/, a path on the nodes, or a Hugging Face repository
id) is registered at start with--enable-lora --lora-modules, and a
request names it as its model, as on llama.cpp.scaleapplies on
llama.cpp only. SGLang and TensorRT-LLM profiles still refuse adapters. max_loaded_adapters: Non a model has each replica hold at most N of
its adapters. Replicas start holding none (vLLM with
VLLM_ALLOW_RUNTIME_LORA_UPDATING=Trueand--max-loras N, unless the
profile's args set their own). The router loads an adapter on a replica
through/v1/load_lora_adapterwhen a request first wants it there, and
unloads the least recently used one no request is running on to make room.
vLLM profiles only; llama.cpp keeps loading every adapter at start, since
llama-server can rescale an adapter it started with but not load another.- Adapter affinity: a request for an adapter goes to a replica that has
it, or is loading it, while that replica is not saturated; otherwise to one
with room for another adapter; otherwise where load sends it — which is how
a busy adapter spreads to a second replica. Requests arriving during a load
wait for it instead of loading the adapter again. - Self-correcting: each replica's
/v1/modelsis read when the router
first meets it and every 30 seconds, so adapters loaded before a restart or
by another router are known; a 404 for an adapter the router thought loaded
loads it again and retries. An adapter vLLM refuses (not found, rank above
--max-lora-rank) fails the request with vLLM's reason, without trying it
on every replica. - Metrics:
llmgw_adapter_affinity_total{model,result=hit|joined|loaded},
llmgw_adapter_loads_total{model,outcome},
llmgw_adapter_evictions_total{model,outcome}. - The admin API and
ctl model adaptersread the file as GGUF only for a
model with llama.cpp profiles.
Federating clusters: another llmgw cluster as a provider
llmgw ctl federation add NAME --url URL --key-file FILE [--send-tags]
makes another llmgw gateway a peer. It is checked first (its/ready,
timed, and its/v1/modelswith the key) and kept in the store (migration
0029,federation_peers) with the models it offers.- A peer's models join the catalog as
NAME/MODEL, always prefixed, and
follow the peer's own list everyfederation.interval(10s). A model the
peer drops goes, unless a model here still falls back to it: then it is
kept withdrawn (in the catalog, sent nothing) until the fallback changes.
The peer's aliases and its own peers' models are not taken. - A peer whose
/readyfails, which refuses the key, or which does not answer
is taken out of routing until it answers again./readynow counts only a
gateway's own endpoints, not its peers'. llmgw ctl model fallbacks MODEL [FALLBACK...](andPUT /api/v1/models/{name}/fallbacks) sets a runtime model's fallbacks, so
overflow can go to a peer's model.- Requests to a peer carry
x-llmgw-hopwith the ids of the clusters they
have passed through (each cluster's id is kept in the store). A request that
comes back to a cluster is refused with 508federation_loop, and a peer a
request came from is not sent it. - The peer sees one key, this cluster's, and the request's
X-Request-Id.
With--send-tags, alsox-llmgw-tags: federated_from=<federation.name>, org=,team=,project=(ids); off by default. - In-cluster residency and
no_storekeep requests off peers, by name and as
fallbacks. Usage records the peer's model as served, andpeer:NAME/MODEL
as the replica. ctl federation list|remove|sync, a Federation view in the console,
llmgw_federation_peer_up,_latency_secondsand_modelsmetrics, admin
API/api/v1/federation, and a "Federating clusters" section in
docs/install.md. Newfederation:block in server.yaml (name,
interval).- A response header the gateway already set (X-Request-Id,
x-llmgw-served-model) is no longer repeated from the upstream's.
Facility-aware placement: racks, PDUs, holds and power caps
- Where a node is, from five labels:
llmgw.io/site,room,row,
rackandpdu(a list, for a dual-fed rack), inagent.yamlor set from
the control plane withllmgw ctl label NODE key=value… [key-…], which
is kept in the store (migration 0031,node_labels), wins over the agent's
labels, works before the node joins, and refuses a misspeltllmgw.io/key.
A node with no location places as before. - Replicas spread across racks and PDUs. A model's next replica goes to a
node sharing no rack or PDU with its others when there is one: an ordering
above weights already on disk and bin packing, never a filter. keep_within: rack(or row, room, site) on a profile keeps a
multi-node replica's nodes in one such domain.- Holds.
llmgw ctl facility hold rack:r12-07 --reason … [--drain]
places nothing new in a rack, row, room, site or PDU;--drainalso drains
its nodes.llmgw ctl facility releaseundoes both. Kept across
restarts (migration 0031,facility_holds). - Power caps.
facility.budgetsinserver.yamlgives a domain's
cap_wattsandother_watts, with amargin; a replica's draw is the
profile'spower_watts, or its cards at the power limit they report
(gpu_wattsotherwise) and its cores atcpu_core_watts, withnode_watts
per node. Use is the larger of reserved and measured, node by node; a shard
that would take a domain past its cap less the margin is refused, and a
multi-node replica's shards are counted together. Unknown draw never blocks. - Seen everywhere it matters.
ctl explain-placementnames the held
domain or the kilowatts, and keeps the rejections of a replica that was
placed, so one in an unexpected rack says why.ctl nodeshas LOCATION,
ready, held and--domain.llmgw ctl facilitylists every domain
with nodes, use, limit, reserved, measured and holds. The console's Nodes
view has WHERE; a node's menu has Location… and Hold rack…/Release.
Admin API:GET /api/v1/facility,POST /api/v1/facility/holds,
DELETE /api/v1/facility/holds/{domain},POST /api/v1/nodes/{id}/labels. - Docs:
docs/facility-aware-placement.mdsays what is built and where it
departs from the design;docs/install.mdanddocs/console.md.
SGLang measured, checked and benchmarked as itself
- Fixed: the router read SGLang replicas with vLLM's metric names, and
SGLang was started without--enable-metrics, so it served no/metrics
at all: every scrape failed as though the replica were unreachable, quietly,
and requests were routed to it blind. SGLang is now started with
--enable-metricsand read in its own names —sglang:num_running_reqs,
sglang:num_queue_reqs,sglang:token_usage(the KV pool's fill), the
prompt and generation token counters andsglang:num_requests_total, as
0.5.21 publishes them. - Fixed:
llmgw ctl engine-check sglangasked the interpreter for
--helpand checked SGLang's flags against Python's. It now asks
python -m sglang.launch_server --help, in the environment SGLang was
installed into (or--bin). - Fixed: SGLang was given
--tp, which argparse accepts only as an
abbreviation of--tp-size; it is given--tp-size. - Fixed:
llmgw ctl bench --metricsread an engine's metrics with the
key given for--target— a client's key when the target is the gateway,
or none — so llama.cpp and TensorRT-LLM, which guard their metrics with the
engine's key, refused every read, and the run reported a queue depth of
zero and suggested no thresholds without saying why. The engine key is now
asked of the admin API for the replica at--metrics' address (or given
with the new--metrics-key); metrics that cannot be read — refused, or in
another engine's names than--engine— fail the run before any load,
saying why; and a step whose metrics went unread shows-and the reason
rather than zeros.
Reusing answers
- A response cache, per project, off until asked for.
llmgw keys cache --project P --ttl 1hhas the same request — same project, endpoint,
model by its catalog name, and body withstream,stream_options,
user,metadata,storeandservice_tierleft out and its keys in
any order — answered from the gateway's memory instead of the model, for
the TTL.--offstops it; without either it says how it stands. The
console has it as Cache answers… on a project. - Only answers that come out the same are reused: chat and completions
asked at temperature 0 or with a seed, and embeddings, rerank and score.
--anyreuses sampled answers too. Only a whole 200 from the model asked
for is kept — not an error, a fallback's or a canary's answer, one a
guardrail changed or stopped, or one overrouter.cache.max_entry_bytes. - Streams both ways: a streamed answer is kept put back together into
one completion (tool calls and reasoning included), and a kept answer is
streamed as chunks — with the usage chunk if the client asked — to a
client that streams./v1/messagesand/v1/responsesget it too. - Headers:
x-llmgw-cache: hit|miss|bypasson every answer of a
project that reuses answers,Ageon a hit.Cache-Control: no-cache
orno-store, orx-llmgw-cache: no, asks the model anew. - Policies still apply: limits and the input guardrails run before the
cache is looked in, the output policy judges a reused answer as it would
a new one, and a request underno_storeor continuing a stored
conversation is never served from it nor kept in it. A project's answers
are never looked up by another project's keys. - A hit costs nothing: recorded as a 200 with no tokens and no cost,
markedcache_hiton the usage record (migration 0027), and shown by
llmgw ctl request; it counts towards the requests-per-minute limit only. - In memory only:
router.cache.max_bytes(128 MB) for all projects
together in each gateway process, least recently used dropped first;
nothing is written to the store.llmgw_cache_requests_total{model,result},
llmgw_cache_bytesandllmgw_cache_entries.
Offline sites and Kubernetes
- Offline bundles.
llmgw bundle create(on any machine with internet)
collects what a cluster would otherwise download — llmgw's release for
each--archwith its signed checksums, every llama.cpp build for those
architectures (or the--variantones) with the checksums GitHub
publishes, the starter model and guardrails' judge, and any--model name/version=hf:owner/repo/filepinned to the registry's commit and its
checksum — each checked as it arrives, into a directory or one.tar
(--tar), with a manifest whose sha256 it prints to carry apart from the
bundle. Run again into the same directory, it downloads only what is
missing.llmgw bundle show --verifyreads one back. llmgw bundle importchecks every file against the manifest and
llmgw's release against the project's signing key, and only then puts them
into the mirror — releases and engines under_llmgw/, models where the
mirror keeps models, with their checksums — so a damaged bundle leaves the
mirror as it was; a different file under a version already there is
refused. On a control plane it also records the engines' checksums in
offline.checksums(/etc/llmgw/offline.sha256);llmgw bundle trust
does that alone, for a mirror on another machine.offline.mirrorin server.yaml makes the cluster reach nothing
outside: nodes install llama.cpp from the mirror, each file held to the
checksum the control plane sends them over the agent's authenticated
connection — never one from the mirror — andreleasesdefaults to the
mirror's copy, sonode add,ctl upgradeand agents upgrading fetch
from it; no registry is asked about newer weights. A release may now come
over plain HTTP when the build trusts a signing key, since the signature
holds whatever carries it. vLLM and SGLang come from PyPI and are said to
be unsupported on an offline node rather than attempted.llmgw server init --bundle PATH --bundle-sha256 …, and the same two
fields on the setup form: installs a control plane offline — the bundle
into the mirror, the checksums trusted,offline:written to server.yaml,
the starter model and judge served from the bundle or skipped when it does
not carry them.docs/data.mdno longer says only weights andctl planreach outside:
it lists everything that does, and what an offline cluster changes.- A Helm chart for the control plane (
deploy/helm/llmgw): one replica
in a StatefulSet, SQLite on a claim or an external PostgreSQL, the agent
port's certificate and the admin token from Secrets, the API and metrics
Service, a LoadBalancer for the agents' mutual TLS, an optional Ingress
and ServiceMonitor,configlaid into server.yaml, and the offline
checksums from a ConfigMap; read-only root, non-root, ready on/leader.
deploy/container/Dockerfilebuilds the image it runs. GPU nodes stay
outside Kubernetes and join withnode add(docs/kubernetes.md says why).
The admin API described, and a Terraform provider
- The admin API in OpenAPI 3.1. Every endpoint under
/api/v1, its path
and query parameters, and the schema of each request and answer with the
meaning of each field, in
docs/admin-api.openapi.json and served by
every control plane atGET /api/v1/openapi.json(with the token, like the
rest). The bearer token, theX-Llmgw-Actorheader and the error shape are
in it. Generated bygo generate ./internal/docgenfrom a route table in
internal/control/adminapi/routes.goand the Go types the handlers use, with
the fields' comments as descriptions; a test fails when an endpoint is
served and not in the table, another when the committed document is stale.
Checked with openapi-spec-validator and openapi-generator-cli's validate. - docs/admin-api.md: how to reach the API (loopback,
an SSH tunnel,admin_remote), the token at/var/lib/llmgw/admin-token,
what a request must carry — an address inHost,application/jsonon
every change, even aDELETE— and some calls withcurl. GET /api/v1/models/{name}: one model's whole definition as it is in
force, with the defaults the catalog filled in. The listing has only its
state and its profiles' names.- A Terraform provider, in
terraform-provider-llmgw/, a module of its
own so the gateway does not carry its dependencies.llmgw_model(a model
added at runtime: scaling, class, owner and adapters change in place, the
rest replaces it),llmgw_quota,llmgw_guard_attachment, and the
llmgw_modelsandllmgw_nodesdata sources. Its client is checked
against the OpenAPI document, endpoint by endpoint and field by field, so a
change to the admin API that breaks it fails its tests; CI runs them.
Built and installed by hand for now: see its
README.
Tracing beyond the router
- The control plane traces what it does to the fleet. A reconcile pass
that does something is areconciletrace — passes with nothing to do are
not traced at all — with what it did beneath it:scale upwithplace
(the profile, engine, GPU class, pool, fabric and nodes chosen, or every
candidate turned down and why, asctl explain-placementwould print it),
scale down,drain replicaandstop replicaeach with its reason,
evict,preemptwith its victims,reap replicawith the error and
when the model is tried again,restart replica,restart refused,
node unreachableandnode lost. - A replica's start is one trace, down to the node.
replica start
stays open from placement until the replica serves, fails or is stopped,
with each shard's progress as an event and how long it took. The command
to each node is asend startspan whose context travels in the command
(a newtracefield on the agent protocol'sControlMessage), and the
agent continues the same trace:start shard, withfetch weights
(local, cached, downloaded, or downloaded by another shard; adownload
with its bytes and where a resume began, the source's credentials and
query left out),launch engine(andwait for rendezvouson a worker
of a multi-node replica), andwait for enginewith how often the engine
was asked and its last answer. "Why did qwen take four minutes to start"
is that one trace. - Stops, drains and adoptions on a node are in the control plane's
trace too:stop shardandstop engine, saying whether the engine had
to be killed;drain shard; andadoptwhen a registering node reports
engines already running, with how many nobody claimed. An engine install
is aninstall enginetrace on the node. - Model jobs are traces.
ctl model serveandctl model updateare
serve modelandupdate model, with a span per step — registry,
mirror, check, catalog, placement, smoke test, replicas — and from the
placement step a link to the replica's own start. - The agent has a
tracingblock inagent.yaml, the same as
server.yaml's (endpoint,sample_ratio;OTEL_EXPORTER_OTLP_ENDPOINT
when unset). Point both at one collector. A node follows the control
plane's sampling for every start it is sent. Traces from a node carry its
id asservice.instance.id, and a control plane's its host name. - With no endpoint nothing changes: every span is the no-op one, and a
command carries no trace context.
Quality suites, and new weights held to one
llmgw ctl eval MODEL --suite FILE|NAMEasks a model a quality suite
and scores its answers: per case, then the cases weighed into one score, 0
to 1, against the suite'sthreshold(or--min-score); it exits non-zero
below it.--concurrency,--judge,--detach,--json.- Suites are YAML or JSON lines: a prompt or a conversation, a weight,
and one way to score it —exact,contains,regex,numberwithin
atolerance(the last number in the answer),json_schema(the answer
read as JSON, in a code fence or a sentence or not, held to a schema), or
judge: a rubric another model on the cluster grades 0 to 1. A
reasoning model's<think>is not part of its answer. - Run on the control plane, through the router as its own requests,
as a replay is: past admission, quotas, accounting and the guardrails.
--suiteis a file sent whole, or a suite's name incontrol.eval_dir
(/etc/llmgw/eval) or built in:arithmeticandjson-format, in
deploy/eval/, sized for small models.--suiteslists them. - Kept in the store (migration 0028): every run with its model, the
repository and revision of the weights that answered, the suite's name
and hash, its score and each case's answer and reason.
llmgw ctl eval --history [MODEL]lists them,--run IDshows one.
- Suites are YAML or JSON lines: a prompt or a conversation, a weight,
- Updates and promotions held to a suite.
llmgw ctl model updateand
model promotetake--eval SUITE --min-score N, or a model carries one
(llmgw ctl model gate MODEL --eval SUITE,eval:in its definition).
A promotion asks the canary first and leaves it in place below the score;
an update runs the new weights first as a canary given none of the
model's requests, asks it, and promotes it or takes it out again, so the
old weights serve until the new have passed. The job'sevalstep says
the score;--skip-evalmoves once without the model's suite. - The console has Ask it a quality suite… (the score counted in as it
comes), Its quality runs and Hold its updates to a quality suite…
on a model's menu. - Fixed: a change to the admin API with a body over 64 KiB reached its
handler cut off at 64 KiB, by the audit log's reading of it.
MIG instances as GPUs of their own class
- A card in MIG mode is placed as its instances. Discovery reports a card
in MIG mode as its MIG instances instead of the whole card, each a device of
its own with its UUID (MIG-…), its own memory, and a class named from the
card's and the profile: two3g.40gbinstances of an A100 80GB are two
devices of classa100-80g-mig-3g.40gb. A profile names that class like
any other, and placement puts a replica on a free instance of it. Both
discovery paths do it: nvidia-smi (instances fromnvidia-smi -L, their
memory and ids fromnvidia-smi -q -x) and NVML (GetMigMode,
GetMaxMigDeviceCount,GetMigDeviceHandleByIndex). llmgw does not create
instances;docs/install.mdhas a MIG section on making them. - One instance a replica. A process opens only one MIG device, and
instances have no NVLink or peer-to-peer between them, so a profile on a
MIG class asking for more than one GPU, tp or pp above 1, or more than one
node is refused when the model is added, with that reason; placement
refuses it again.ctl model planandctl model serveplan an instance
for a model one holds and never several together. - Engines are given the instance by UUID. Placement numbers card 0's
instance 2 as device 1002 (1000 + 100 × card + instance); the agent turns
that intoCUDA_VISIBLE_DEVICES=MIG-<uuid>for vLLM and llama.cpp, and
leavesGGML_VK_VISIBLE_DEVICESunset (Vulkan cannot see MIG). Every view
shows the device as0/mig2. - Memory per instance, no made-up utilisation. Each instance's memory,
and the processes holding it, are its own — read per GPU and compute
instance from nvidia-smi's report, so a foreign process on one instance
keeps that instance out of placement and not its neighbours. Temperature,
power and wear are the card's. No driver reports utilisation per instance,
so none is published for one: the console saysn/a on MIG, and
llmgw_gpu_utilization_percenthas no series for it rather than a zero. - A card in MIG mode with nothing carved is kept out of placement, with
the reason inctl nodes, since CUDA will not open the whole card while MIG
is on. llmgw doctorreports MIG: each card's mode and instances, and warns
of a card in MIG mode with no instances, of GPU instances with no compute
instance, and of a mode change waiting for a reset, with thenvidia-smi
command for each.ctl nodescounts a node's devices by class when it has MIG instances or
more than one class (a100-80g-mig-3g.40gb×2,a100-80g×1); the console's
GPU list shows instances under their card with their own memory and the
card's heat and power, and a node's page lists its classes with counts.
ctl statusand the plan count each device under its own class, where a
node's cards were all put under its first class before.- The agent protocol's
GPUgainsmig(parent card, its memory, profile,
GPU and compute instance ids); the node API's GPUs gainlabel,class,
card,mig_profileand memory, and nodes gaingpus_by_class.
Engines and the mirror
- TensorRT-LLM as an engine (
engine: trtllm), on nodes where you
installed it: name its interpreter in the node'sagent.yaml
(engines.trtllm.python) and the node offers it while that interpreter
has thetensorrt_llmpackage. llmgw does not install it. The agent starts
trtllm-serve servewith the replica's host, port,--tp_size,
--pp_sizeand served name, through a few lines of its own that make it
demand the gateway's key on every path but/health(trtllm-serve has no
key option), turn on the Prometheus endpoint and iteration statistics, and
keep usage reports to NVIDIA off. The router reads it at
/prometheus/metrics—trtllm_num_requests_running,
trtllm_num_requests_waiting,trtllm_kv_cache_utilization, tokens and
successes. Needs TensorRT-LLM 1.3 (rc13 or later) and an Ampere or newer
GPU; one node per replica. Profiles may not give it an options file,
--trust_remote_code,--grpc,--num_serve_frontendsor the flags the
agent sets;--set PATH=VALUEchanges a setting instead.llmgw ctl engine-check trtllm --bin <python>checks an installed one. Written
against TensorRT-LLM's documented command line, server and metrics and
tested with stand-ins for them; not yet run on a datacenter GPU. llmgw mirror import-ollama [NAME[:TAG]]puts a model you already
have in Ollama into the mirror for llama.cpp: its GGUF, and a vision
model's projector beside it, each checked against the sha256 Ollama named
its file by. With no name it lists Ollama's models; it finds Ollama's
directory itself (OLLAMA_MODELS,~/.ollama/models, Ollama's Linux
service) or takes--ollama-dir. The version is the tag and the start of
the digest, since a tag moves in Ollama. It prints a model definition to
add, with Ollama'snum_ctxas--ctx-size. The console's Mirror
view does the same with I, for the mirror beside the control plane.llmgw mirror import FILE... --model NAMEcopies files already on the
mirror machine into it the way a download is kept: whole or not at all, the
checksum beside it, a version never replaced, a.ggufchecked to be one,
and--sha256checked when given.
The portal in Turkish
- The web portal speaks English and Turkish — the landing page, sign-up
and sign-in, the account page and its editor setup, the claim page, chat,
the about page, and every administrators' page (cluster, nodes, NVLink and
fabrics, people, limits, usage, organizations, mirror, guardrails), and
the docs portal around its documents, which stay English. Flash
messages and form errors built by the server are translated too. - A switcher at the foot of every page (English · Türkçe) remembers the
choice in a cookie; without one the browser's Accept-Language decides, and
without that the newlanding.language(enortr, empty is English).
An unknown code stops the server starting.<html lang>and
Content-Language follow the page. - Figures and dates in the page's language: a Turkish page writes
1.234,5, 04.10.2026 and "3 saat önce"; chart tooltips name the day in
Turkish. Form values, CSS and SVG coordinates are unchanged. - Ready for more languages: the catalogue is JSON under
internal/web/locales/<code>/, and a language is a directory. Tests refuse
a key a page uses that some language lacks, a translation with different
arguments from English, and any template text left in English — so a new
page cannot quietly ship English-only. - Messages that come from the control plane and other packages (node
warnings, engine errors, store refusals) are still shown as written, in
English; the API, CLI and console are English.
Documentation
- A docs portal at /docs. The manual the binary carries — the pages
llmgw docsprints and the console's Manual shows, the configuration
reference among them — rendered on the client port in the portal's own
look: a sidebar of the documents in reading order, the open one's sections
under it, a link on every heading, and links between documents kept inside
the portal (links to files that are not in the binary, such asdeploy/,
go to the repository). No script on the page, so the Content-Security-Policy
has nothing to allow. Public by default, like the landing page; set
landing.docs_sign_in: trueto send visitors to sign in first. Signed in,
the page carries the usual menu, which now has a Docs link. - A client developer's guide,
docs/clients.md. How to use llmgw from
the official OpenAI and Anthropic SDKs — there is no llmgw SDK, and none is
needed — in Python, TypeScript, Go and curl: the base URL and key,
streaming, tools, embeddings, the Responses API, batches, the Anthropic SDK
against/v1/messages, and what llmgw adds as headers (x-llmgw-thread,
x-session-id, metadata andx-llmgw-tags,x-llmgw-variant,
/v1/memory,/mcp), with the statuses a client meets. The portal's index
leads with it, and the landing page and the account page link to it. - The console's Manual and the portal read one list of documents in one
order (llmgw.Manual), so the two cannot drift.
Administration
- An audit log. Every change made through the admin API (
llmgw ctl,
the console) or the pages under /admin is recorded: who, when, through
what, the fields it was given with passwords, keys and tokens taken out,
and how it ended — refusals included, in words. So are a person's own new
keys, revoked keys and password changes. Read it withllmgw ctl audit
(--since,--actor,--contains,--detail),GET /api/v1/audit, or
the console's Audit view. It moves with the store. - Staff roles besides administrator:
viewerlooks at every
administration page and changes nothing,operatoralso runs the cluster
from them,billingsees usage and quotas. Set withllmgw users roleor
the console's Staff role…. - Single sign-on through OpenID Connect (
auth.oidc). A "Sign in
with …" button sends people to the identity provider and back, with PKCE,
state and nonce. The ID token is verified, and the account is found by
the verified email address, or made whencreate_accountsis on. Groups
give staff roles (roles, set at every sign-in) and places in the tree
(memberships).no_passwordsleaves the provider as the only way in,
except for administrators. - Provisioning through SCIM 2.0 (
auth.scim.token_file,/scim/v2).- Users: the identity provider makes, suspends and reinstates
accounts — the PATCH forms Entra ID and Okta send are both read.
Deleting a user suspends the account. - Groups: joining and leaving the groups
auth.oidcmaps gives and
takes away staff roles and places in the tree.
- Users: the identity provider makes, suspends and reinstates
- Key lifecycle.
- Lifetime limits:
auth.max_key_ttllimits how long any key may
live, andllmgw keys ttl team:Acme/ml 720hsets shorter limits for an
organization, team or project. A key is made within the shortest that
applies, and one asked for with no expiry is given that longest — from
the CLI, the profile page and the editor connect page alike. - Rotation:
llmgw keys rotate <id> --overlap 48hissues a
replacement where the old key was, and the old one stops after the
overlap. - Listing:
keys list --allshows each key's kind — service (a
project's), member, personal — with service keys first, and when it
expires.
- Lifetime limits:
- Usage export.
llmgw ctl usage export --from 2026-09-01 --to 2026-10-01 --by project,modelwrites CSV by hour, day or month and any of
organization, team, project, user, key and model.- Data: read from the hourly and daily sums, so it reaches back past
the request records' retention. - Format: cost is given in micro-units and units, and names that
look like a spreadsheet formula are quoted. - Other ways in:
GET /api/v1/usage/export(CSV, orformat=json)
and E on the console's Usage view.
- Data: read from the hourly and daily sums, so it reaches back past
Forgetting a person, and requests that stay in the cluster
llmgw users forgeterases a person for a request to be forgotten:
their usage kept under a random pseudonym so totals stay true, their kept
requests, personal files, batches and limits deleted, their name taken out
of the audit log, the account removed.llmgw keys residency --project P --in-cluster: no model served by an
outside provider for the project's keys, asked for, as a fallback or as a
canary.- Both are on the console's menus.
Secrets in Vault
- Secrets from Vault or the environment. Every setting naming a secret's
file also takesenv:VARIABLEorvault:mount/path#field(HashiCorp
Vault or OpenBao, KV v1 or v2; a token, AppRole, or VAULT_TOKEN). - The store's own secrets sealed under
secrets.master_key: Vault's
transit engine (vault-transit:KEY), or a 32-byte key read from a
reference. A store, or a backup of it, then does not hold what engine keys
are derived from. Values from before are sealed as they are next read.
Scaling by the clock
scaling.schedule: windows of the week with their ownminand
max— more replicas through working hours, an on-demand model kept
loaded while people are at work — incontrol.timezone.scaling.prewarmopens a window early, so its replicas have loaded
when it starts.
Token-level timing, and admission by SLO
- Per-token metrics:
llmgw_tpot_seconds,llmgw_inter_token_seconds
andllmgw_output_tokens_per_secondby model and pool, and TPOT and
tokens a second by tenant. - A model's
slocounts in admission: past itsttft_p99or
tpot_p99(the recent p99), it is treated as saturated, so the lower bands
wait.llmgw_model_slo_secondsand thellmgw-sloalerts make the
promise visible in Prometheus.
Restoring a backup, and copies off the machine
llmgw server restore ARCHIVEputs a backup back: checks it first
(whole, opens with--identity, not newer than this llmgw, where the store
goes), says so with--dry-run, then moves/etc/llmgwaside, restores it
with modes and owners, restores the store (SQLite, or PostgreSQL with
pg_restore --clean), and starts the server. A drill is in
docs/install.md.backup_copy_tocopies each archive to a directory or over ssh as it
is written, keeping as many there as here; refused without
backup_recipients.- Archives now record each file's owner by name, so a restore on another
machine gives the llmgw user its files back.
Anthropic's Messages API
/v1/messages(and/v1/messages/count_tokens) translated to chat
completions and back, streamed or not: text, images, tools and their
results, the system prompt, stop sequences. Claude Code and Anthropic's
SDKs reach the models here; the key may come inx-api-key. Errors come
back in Anthropic's shape.
More of OpenAI's API
- Responses:
text.formatasks for structured output,input_image
sends images, andprevious_response_idcontinues a conversation the
gateway keeps as a thread (for a project that keeps conversations). - Passed through:
/v1/rerankand/v1/scorefor vLLM's rerankers, and
/v1/audio/transcriptionsand/translationsas the multipart form came,
for an engine serving Whisper. /v1/moderationsis answered by the guardrails' judge model.
Requests kept, and replayed
- A project can keep its requests and answers (
llmgw keys requests --project P --retain-hours N [--sample 0.05], or Keep requests… in the
console): what the model was sent, after redaction, and what the client
got. Off until asked; never underno_store; pruned hourly. llmgw ctl request ID --bodyshows one;--replay MODELsends it
to another model and shows both answers.
Batches
- OpenAI's Batch and Files APIs.
POST /v1/files(purpose batch),
POST /v1/batches, and reading, listing, cancelling and deleting them, so
OpenAI's clients send batches here unchanged.- Serving: each line goes as a request of the key that made the batch,
in the lowest admission band, a few at a time; rate limits are waited
out. Answers and failures are written to files kept 30 days. - Accounting: each line is an ordinary usage record tagged
batch=<id>. - Restarts: answered lines are kept as they are answered, and any
gateway on the store carries a batch on under a lease. - Limits:
router.batchbounds the file, the lines and the
concurrency. Refused to a project under ano_storepolicy.
- Serving: each line goes as a request of the key that made the batch,
New weights tried on a share of the traffic first
- Canaries.
llmgw ctl model canary MODEL --percent 10fetches the newer
weights of a model's repository and runs them beside it asMODEL-canary,
given that share of its requests.- Routing: a session stays on one side;
x-llmgw-variantpins a
request tocanaryorstableand says which served it. Clients cannot
name the canary. - Rolled back by itself when, over 50 of its requests or more, its
error rate is five points above the model's or its time to the first
token half again as long;canary_rolled_backis sent to the
notification webhooks. - Then:
--percentchanges its share,llmgw ctl model promote
moves the model to it,--aborttakes it out, andllmgw ctl model canariescompares the two sides. All of it is also on the console's
model menu.
- Routing: a session stays on one side;
Requests can be tagged
- Tags and end users on every usage record. OpenAI's
metadataand the
x-llmgw-tagsheader (feature=search, env=prod) tag a request; OpenAI's
userfield names the client's own end user. Both are kept with the
request's usage and shown byllmgw ctl request. Tags are not sent to the
engine. A malformed one is refused with 400. - Counted by tag, for the tags named.
router.tag_labelslists the tags
counted inllmgw_tagged_requests_totalandllmgw_tagged_tokens_total,
the first 100 values of each apart and the rest asother.
A hardened install by default
server initand the setup screens write a hardened configuration.- Guardrails: a
baselinepolicy on every request. It refuses
secrets in prompts and answers, masks card numbers and records injection
attempts. - The judge: Llama Guard 3 1B, two replicas, fetched and run on CPUs
(about half a second for a short text). It refuses violence, crime,
weapons, child abuse, privacy violations, hate, self-harm and sexual
content. The policy fails closed: a request it cannot read within 5
seconds is refused (503), andjudge_unavailablegoes to the
notification webhooks. - Keys: they live at most 90 days.
- Opting out:
--no-guard-model(or unticking it on the setup screen)
leaves the judge out and keeps the pattern checks. Aserver.yaml
written earlier is never changed. - What is left to you: the closing text now names the two things
init cannot do — a certificate for the client port, and the keys
backups are encrypted to.
- Guardrails: a
Notifications
- Webhooks for what needs a person (
notify.webhooks): a model parked,
a node lost, a budget spent, new weights in a model's repository — sent by
the gateway itself, with no Prometheus in between.- Formats: JSON signed with HMAC-SHA256 (
X-Llmgw-Signature) when
secret_fileis set, or a Slack-compatible{"text": …}. - Delivery: queued, never holding up what raised it; tried three
times; the same event about the same thing at most once an hour, and new
weights once per revision. - Checking:
llmgw ctl notify test. - A judge not answering (
judge_unavailable) is one of the events.
- Formats: JSON signed with HMAC-SHA256 (
- Alertmanager receiver templates for Slack, e-mail, PagerDuty and a
webhook indeploy/monitoring/alertmanager.yml, commented out until
chosen.
Guardrails
- Hooks for guardrails on the request path (nothing judges yet; B2 adds
the policies). A guard, when one is set, judges a request once it is
authorised, within its limits and has its stored conversation spliced
in, and before it is dispatched; and filters the answer, buffered or
streamed, before the client sees it. A blocked request is refused with 400
andcode: "content_policy_violation"— the first error with a code — and
costs nothing; a blocked answer ends withfinish_reason: "content_filter",
and a stopped stream with that chunk and[DONE], the engine cancelled.
The Responses API reports either asincompletewith
incomplete_details.reason: "content_filter".x-llmgw-guardand
x-llmgw-guard-policysay what was done; the usage row records the
action, policy, category and time taken — never the text. - Guardrail policies (
guard:in server.yaml; see docs/guardrails.md).- What is found: personal data, each kind validated before it counts
(email, phone, IBAN by mod-97, card by Luhn, TCKN by its check digits,
IP); secrets (API keys, AWS, GitHub, Slack, Google, JWT, PEM); prompt
injection seen through disguises (look-alike letters, invisible
characters, base64/hex/URL encoding); deny patterns. - What is done: flag, redact as
[EMAIL_1], or block. A request can
be shaped too: a pinned system prompt, the caller's own system messages
dropped, an output limit, tools stripped or refused. - Answers: judged buffered, or a sentence at a time when streamed,
never releasing half a value; tool calls are checked against allow
lists, a count and the request's own schemas. - Modes:
shadowrecords what it would do;no_storerefuses
conversations, notes and models outside the cluster. - Where they apply:
guard.defaulteverywhere; others attached with
llmgw ctl guard attachto an organization, team, project, user or
key, and narrowed like limits.
- What is found: personal data, each kind validated before it counts
- A judge model on the cluster (
role: guardin the catalog,judge:
in a guard policy). It reads what patterns cannot — Llama Guard 3,
ShieldGemma or a JSON-answering model — and its verdict flags or blocks
input, output or both.- Calls: past admission and accounting, bounded by a latency budget
(250ms by default), after which the policy'sfaildecides. - Caching and streams: answers are cached for five minutes, and a
streamed answer is read every 600 characters. - Isolation: a judge model is hidden from clients and refused to them.
It is critical by default, and cannot be loaded on demand.
- Calls: past admission and accounting, bounded by a latency budget
- Seeing what the guardrails do.
- Metrics:
llmgw_guard_checks_total(stage, policy, action,
category, mode), latency for checks and for the judge, judge errors,
fail-open, redactions by kind, stopped streams, and blocks per tenant. - Alerts: judged traffic served unjudged, a slow or unreachable
judge, a tenant's blocks spiking, a secret in an answer. - Pages and commands:
/admin/guardshows counts by place, policy
and category and the recent requests by id;ctl requestshows the
verdict; the profile page lists the policies in force where the person
works; the console's organization rows have Guardrails… and a
column. - The chat page says when a request was refused or an answer stopped
by policy, instead of showing an error.
- Metrics:
Security
- Built with Go 1.26.6. Eleven standard-library vulnerabilities reachable
from this code at 1.26.3 (net/http, crypto/x509, os.Root, net/textproto)
are fixed by it.make vulnruns govulncheck;make checkruns it with the
tests and vet. - Releases are signed, and only signed releases are installed. The build
now carries the release signing key (internal/release/trusted.pub); the
control plane upgrading itself, an agent told to upgrade and a node being
added all refuse a release whose checksums are not signed with it. The CI
release job stops rather than publish an unsigned release. Adding a node
no longer has the node fetch the release and compare it with a checksum
from the same server: the control plane fetches it for the node's
architecture, checks the signature and the checksum, and hands it over.
Releases are fetched over https only (loopback excepted), and
LLMGW_RELEASE_KEYreplaces the trusted key in development builds only. - The agent port is not opened without TLS by accident. A control plane
with noagent_tlsrefused nothing: its agent port was plaintext, on every
interface, and any machine reaching it could register as any node — be
given replicas and so be sent prompts. It now refuses to start unless
agent_insecure: trueis set, and then listens on loopback unless
agent_listennames an address.server initwritesagent_tls, as
before. - A model definition cannot run code on its nodes. A profile's
envand
argsreach a process the agent runs as root. The environment was checked
against a short denylist that missed NCCL's and CUDA's plugin and
injection libraries; arguments were not checked at all, so
--trust-remote-code, llama.cpp's--log-fileor--slot-save-path, or
vLLM's--tool-parser-pluginwere taken.envis now an allowlist of the
families engines are tuned with (CUDA_, NCCL_, VLLM_, OMP_, HF_…) less any
name for a path, plugin, library, address or credential, and llama.cpp's
LLAMA_ARG_*are checked as the flags they are;argsrefuse what reads,
writes or loads files, loads code, or moves what the agent sets (host,
port, key, model). Both are refused where a model is defined and again on
the node before the engine starts. - Engines no longer run as root. The agent starts them as the system
userllmgw-engine(in the video and render groups for the GPUs), with a
home and caches of its own under the agent's work directory.llmgw node addmakes the user; a node without it keeps running engines as root and
the agent says so in its log.supervisor.run_asin agent.yaml names
another user, orroot. - The admin API wants a token. It listened on loopback with no
authentication, so any process or user on the control plane could add a
model, upgrade the fleet, read the engines' keys or dump the heap through
pprof. Its/apiand/debugpaths now demand the token the server
writes, 0600, to its state directory (/var/lib/llmgw/admin-token) on
first start;llmgw ctl, the console andnode addread it, so they run
with sudo, andLLMGW_ADMIN_TOKENorctl --token-filegive it from
elsewhere. pprof is off unlessdebug_pprof: true.admin_listenoff
loopback is refused unlessadmin_remote: true. The read-only dashboard
pages,/metricsand/healthare unchanged. - The replicated store asks for a password and speaks TLS. Patroni's
REST API took changes from anyone who reached port 8008 — rewriting the
access rules totrust, or moving the leader — and PostgreSQL traffic
between the machines was plaintext (sslmode=disable). Patroni now asks
for a password before a change (kept with the cluster's others in
/root/llmgw-pg, and on the control plane in
/etc/llmgw/secrets/patroni-restforserver add); each member has a
certificate from the cluster's authority, PostgreSQL hassslon, every
remote access rule ishostssl, and the control planes connect with
sslmode=verify-fullagainst the authority's certificate. A new cluster is
set up this way; an existing one is changed in place, by reload, the
cluster serving throughout, by running Replicate the store in the
console again — orsudo --preserve-env=SSH_AUTH_SOCK llmgw server init --patroni <user>@<a> --patroni <user>@<b> --no-mirror --no-node --no-model— on the control plane that holds the authority, then on each
other control plane so it reconnects by TLS. etcd is not yet given TLS:
keep ports 2379 and 2380 to the cluster's machines. - A capacity fallback respects the key's allowed models, and is charged
at its own price. When the model asked for had no capacity, the next in
itsfallbacksserved the request even if the key was not allowed it, and
the request was priced as the model asked for. Fallbacks a key may not use
are now skipped, and a request is charged what the model that answered
costs. - Suspending an account stops its keys. Suspension closed the person's
sessions and left every API key they had made working. A key whose owner's
account is not active — suspended, pending, or gone — is now refused as a
revoked key is, withinauth.cache_ttl, and works again when the account
is reinstated. - A join code is a credential only for its hour, and only on its
address. The setup checked a code's expiry on the joining side alone, so
a code stayed good for as long as the setup ran; it now refuses an expired
one itself. Its listener binds the address the codes name rather than
every interface, and a joined machine is known by the address it connected
from, not the one it claimed. - The setup's sudo grant cannot outlive it. Given a sudo password, the
setup writes a temporaryNOPASSWDgrant and removes it when it
finishes; a setup that crashed or was interrupted left it for good. The
grant now schedules its own removal within the hour as it is written, and
the password is dropped once the grant stands in for it. An address given
to the system ssh can no longer be read as one of its options. - Made-up keys cannot fill memory or the database. Every well-formed key
presented, real or not, was cached for good and cost a database read the
first time. The cache is now bounded, and keys not in it are looked up at
most 20 a second per client address (50 at once); beyond that the request
is answered 429 without a read. - Slow or idle connections cannot pile up. The servers bounded only how
long headers took. Every listener now also bounds how long a connection
may sit idle and how large its headers may be, and reading a request
(5 minutes on the client port, a minute or less elsewhere — never on the
agents' long-lived streams); the client port holds at most
max_connections(10,000) at once, and a message from an agent at most
16 MB. - The sign-in and signup forms resist scripts. Ten wrong passwords from
anywhere locked an address's owner out for the window; attempts are now
counted per address and client, so somebody guessing elsewhere cannot lock
the owner out, with a looser count per address alone so guesses spread
over many clients are still bounded. The sign-in form carries a token its
cookie matches, so another site cannot sign a visitor into an account of
its choosing. Posting either form is rationed per client address (sign-in
30 a minute, signup 5), and a password hash that cannot get a turn within
ten seconds answers "busy" instead of holding the request.server init
now writessignup_open: false. - Forwarding headers are believed only from the proxies named.
X-Forwarded-Hostand-Protowere taken from anyone, and decide the
gateway address written into the/setupscripts clients pipe to a
shell; behind a caching proxy one request could poison them for others.
They — andX-Forwarded-For, now used for the per-address limits — are
believed only fromrouter.trusted_proxies(addresses or CIDR prefixes).
Behind a reverse proxy, set it, or the pages and scripts name the
address the proxy reached and every client counts as the proxy./setup
scripts are servedno-store. - Removing a node revokes its certificate.
ctl node removeonly
dropped the node from the inventory; its certificate, valid for a year,
still connected and was given work. Now the removal is recorded in the
store and the agent port refuses any certificate issued for that node
before it — onboarding the node again issues one that works. A mirror
serving its management API with--tls-canow requires--api-client
(the control plane's name), since the same authority signs every node. - Engines listen on the management address only, the one the control
plane reaches them on, instead of every address the node has — a public
one included. (When that address is a name, or not one of the node's,
they still listen on all.) SGLang's key is off its command line,
where every user on the node could read it: it reaches the engine in its
environment, and a short launcher hands it to SGLang inside the process.
Both take effect as engines restart under an upgraded agent. - Backups can be encrypted.
backup_recipientsin server.yaml (age
recipients or SSH public keys) hasllmgw server backupencrypt each
archive to them with age, as.tar.gz.age: an archive holds the CA's key
and every secret in the store, and is meant to be copied off the machine.
Open one withage -d -i <key>. - PyTorch's index gives torch and nothing else. vLLM's CUDA 12.9 build
was resolved with PyTorch's index beside PyPI andunsafe-best-match, so
either index could supply any package — how torchtriton was hijacked in
2022. Now that index is explicit, named only for torch, torchvision and
torchaudio; every other package comes from PyPI alone, and the lock is
PEP 751's (pylock.llmgw.toml), pinning each file's address as well as
its hash. Checked against vLLM 0.30.0: the same 195 packages resolve. - The control plane's own fetches stay off link-local addresses.
Checking a model's source and reading a GGUF's header went wherever the
URL said, cloud instance metadata (169.254.169.254) included. They now
refuse link-local, multicast and unspecified addresses, while still
reaching the cluster's network, where the mirror is. The mirror's fetches,
which refuse private addresses too, now also see through IPv6 addresses
carrying an IPv4 one (NAT64, 6to4, IPv4-compatible), and through a proxy:
the target's addresses are checked before the request is handed to it. - A CPU-only node reporting a problem no longer panics the agent
handler, and thread ids are bounded to 128 letters, digits and
- _ . :. A thread is readable by every key of its project that knows
the id; the docs now say to use unguessable ids. /v1/modelslists only what the key may use, rather than every
model the gateway serves to anyone.- Portal sessions end after a day unused (still at most 14 days in
all), and expired or idle sessions and used or expired key claims are
deleted hourly — they were refused, but never removed. - Every page carries a Content-Security-Policy: scripts only from the
gateway itself and the page's own inline ones, each named by its hash —
no inline handlers, no eval, nothing a model's reply or a model card
could slip into a page. The fewonchange/onsubmitattributes moved
to/_static/pages.js, htmx runs withallowEval: false, and the live
pages' "only while visible" refresh no longer needs an evaluated filter.
Pages now carry their headers even when answered with an error status. - KaTeX 0.16.47 (from 0.16.11), past the advisories fixed in 0.16.21.
- Smaller hardening. The mirror no longer serves its bookkeeping files
(a file's.sourceURL and.sha256) on:8081. A GGUF header nesting
arrays more than 4 deep is refused rather than recursed into. Serve and
update jobs keep the last 100 finished, and the mirror forgets finished
downloads after a week, instead of remembering every one until restart. - A firewall for llmgw's internal ports.
server init(and the store
members it sets up with--patroni),node add,server addand
mirror initnow keep the machine's internal ports — the agents' stream,
engines, the mirror, PostgreSQL, Patroni, etcd, node exporter — to the
cluster's own networks, in an nftables table of llmgw's own loaded by
llmgw-firewall.service. Only those ports are touched: the table accepts
everything else, so other services on the machine and any firewall it
already has are left as they were. Extra networks go in
firewall_networks;--no-firewallskips it;llmgw doctorsays when a
node lacks it. - etcd and Patroni's API speak TLS. etcd's peers and clients use
certificates from the cluster's authority, and a client must present one;
Patroni reaches it with its member's certificate and serves its API over
TLS.server init --patroni, run again, moves an existing cluster: with
Patroni in maintenance mode, so PostgreSQL serves on and nothing fails
over, etcd is rebuilt on TLS and Patroni's keys carried into it. The
control plane asks the API over HTTPS, verified, and over HTTP only where a
member has not moved yet. - A cluster keeping its access rules per member got no
hostssl. The
TLS step looked only at the shared configuration, and on a cluster set up
before the rules moved there said the rules werehostsslwhile
plaintext was still accepted. Each member's own rules are now made
hostssltoo.
Fixed
- An engine that died after an agent restart went unnoticed. The watcher
that reports a ready engine dying was tied to the stream it was started
on, so after a reconnect nothing reported it, and an engine adopted from
the previous agent had no watcher at all: its replica stayed "ready" and
was routed to. Every connection now watches every ready shard, started or
adopted. Found on the PoC: a killed engine on node06 stayed ready for
minutes. server initrewriting the store's address no longer leaves
/etc/llmgw/secretsunreadable to the service when it was made 0700 for
the service user, which kept the gateway from starting.
Everything that can be changed on a running cluster, from the console
sudo llmgwonly watched the cluster and edited the configuration files;
changing a model's replicas still meantllmgw ctl. Now Enter on a row opens
what can be done to it, as forms and questions:- Models: replicas (min, max, idle timeout), class, owner, resources, LoRA
adapters, restart, retry when parked, why it is placed where it is, remove;
A adds one from a definition file. - Nodes: cordon, uncordon, drain, the vLLM and SGLang builds, upgrade its
agent, delete model files nothing uses, remove. - Replicas: restart, replaced or in place.
- Quotas (new): capacity quotas, set and removed.
- Mirror (new): what it holds and which models use it; download into it,
remove from it. - People (new): accounts — approve, suspend, reinstate, the cluster
dashboard, memberships, limits, a new password, delete, create; API keys —
issue (shown once), limits, revoke; organizations, teams and projects — add,
rename, move, remove, limits for the level and for each member, capacity
quotas, how long conversations and agent notes are kept. - Serve a model (new view): GGUF models on Hugging Face — trending, most
downloaded, most liked or newest, or searched with / — then one of the
chosen repository's files, a name and a replica count, and it is fetched,
checked, added and started. The registry is mirror.registry's when set. - Do: upgrade the cluster, restart every replica, back up now.
The people side runs the same commands as
llmgw users,keysandorg,
against the same store, so their checks and messages are the ones shown. A
sidebar taller than the terminal now scrolls with the current entry.New weights in a model's repository, found and moved to
A repository on Hugging Face can publish new weights under the same file
names — a fixed chat template, a better quantisation — and nothing said so. A
model served from the registry now records its origin: the repository, the
commit, and each file's checksum there; a model served before is worked out
from the mirror's record of where it fetched each file. Every
mirror.update_check(six hours by default; 0 turns it off) the control
plane asks whether those files changed. A new commit that leaves them alone
is not an update.llmgw ctl model updates [--check]lists what each repository has.llmgw ctl model update MODELfetches the new files into the mirror beside
the old ones, checks them against the registry's checksums and the nodes'
llama.cpp, points the catalog at them, replaces the replicas one at a
time and asks one to say hello.llmgw ctl model rollback MODELgoes back to the files the update
replaced, which stay on the mirror.- The console shows UPDATE on Models and offers all three on a model's row.
A model from
server.yamlis reported but not changed: its source is edited
there.The monitor: the cluster drawn like btop
The console has a Monitor under Watch: CPU, memory, GPU busy and memory,
requests, tokens, time to first token, in flight and queued, network and
disk, as Braille charts filled and coloured by height, over 5 minutes, an
hour, a day or a week (1–4 or R); then each node's CPU, memory and network,
each card's busy, memory, heat and power, and each model's requests, as
sparklines. A node's and a model's own pages carry their charts too.The history is read from the Prometheus in
monitoring.prometheus, through
the admin API (POST /api/v1/metrics/range); without one, the charts draw
what the console has seen since it opened.The control plane now lists the cluster's machines for Prometheus' service
discovery, at/prometheus/targetson its metrics port — nodes, itself and
the store's members, each labelled with its llmgw id — andllmgw node add
installsprometheus-node-exporteron the machine (--no-node-exporter
leaves it off).deploy/monitoring/prometheus.ymlscrapes them as the
nodesjob.Serving and updating run in the background
Serving a model or moving one to new weights from the console starts a job
on the control plane and leaves the console free at once, instead of
closing it to follow a download that can take an hour. A new Jobs view
shows each job's progress and steps, and the sidebar how many are running.
llmgw ctl model serve --detachandllmgw ctl model update --detachdo the
same;llmgw ctl jobslists the jobs andllmgw ctl jobs IDfollows one.The file that suits the hardware, and how to spread it
llmgw ctl model plan REPOnow plans the files the repository actually has
against the fleet as it is, before anything is downloaded: the model's shape
comes from one GGUF file's header, read from the registry, or from its
config.json. Every file is listed with the best way it runs here — one card;
several of a node's cards (llama.cpp splits by layer, vLLM by tensor
parallelism, dividing the KV heads); several nodes' cards over one fabric,
pipeline-parallel between them (vLLM); or a CPU node's memory — what each
card or node would hold and whether that hardware is free now. Best first:
on a GPU, the fewest cards, then the better quantisation up to eight bits;
whether the cards are free now only breaks a tie. Busy is not unfit: a model
for cards that are in use is added and starts when there is room, the job
saying so rather than failing.--precisionkeeps the old plan in the abstract.llmgw ctl model servewithout--quanttakes the plan's file, and without
--cpuor--gpu-class/--gpus/--nodesthe plan's way of running it —
a GGUF model on GPUs is now given a GPU profile rather than always the CPU.
--fallbackswrites the next best ways too, lower in priority, so placement
uses one of them when the first is busy. The console's Serve a model shows
the plan for each file, marks the best, and asks for the way and the
fallback.Models of several files: vision projectors, original weights
- A vision model's projector (
mmproj) is fetched beside its GGUF weights
and loaded with them:llmgw ctl model serve REPO --projector FILE, or the
choice offered by the console. The model is marked as taking images. Nodes
load it once their agent is this version or later. --format safetensorsserves a repository's original weights, with their
config.json, tokenizer and shard index, on vLLM across--tpGPUs of
--gpu-class(the fleet's one class when there is one). In the console,
F on Serve a model switches to them.- The mirror keeps where it fetched each file beside it, and reports that
and the checksum after it restarts. The console's Mirror view shows one
row per version of a model, with its files on Enter. - An update watches every file of a model, the projector and the tokenizer
included; one whose files the mirror cannot trace is found on the
registry by name, commit and size, or given withllmgw ctl model origin. llmgw_model_update_availableis published, andModelUpdateAvailable
alerts on it.
The manual, in the binary
llmgw docsreads every document — installation, operation, the
configuration reference, the README, the changes — and a page for every
command, built from the commands' own help, likeinfo: topics on the left,
T for a page's contents, / to find, n for the next match. Piped, it prints
the page. In the console it is Help › Manual, and ? opens the new
page about the console itself from any screen.Downloads
- A vLLM profile can set
-
v0.11.1
Stable
gitea-actions
released this
2026-10-03 23:10:20 +02:00 | 507 commits to main since this releaseThe console's overview no longer spills over the screen
On a cluster with a GPU,
sudo llmgwprinted the overview's GPU table onto the
terminal under the console every three seconds, scrolling across the full
screen. The summary's tables were written to standard output whatever they
were given; they are now written where the summary is, which for the console
is its panel. The test cluster the console was tried on had no GPU, which is
why it was not seen there.Downloads
-
v0.11.0
Stable
gitea-actions
released this
2026-10-03 22:59:55 +02:00 | 509 commits to main since this releaseA minor version: the store can be replicated and the control plane doubled,
both set up by llmgw itself;sudo llmgwopens setup screens and a console;
usage is read from sums kept as it is written. Released without candidates.Upgrading. Migrations 0015 (usage sums, filled from the request table
already recorded — once, at the first start) and 0016 (control planes) run by
themselves. An agent older than this ignores the list of control planes and
keeps its configured one. Nothing inserver.yamloragent.yamlchanges
meaning;auth.usage_retentionandrouter.tenant_labelare new and optional.The GPU and fabric work below began from the field report in issue #2, after
a deployment on GPU nodes.GPU metrics that move, and charts of them
A GPU's utilisation and memory in use were read once, when the agent
started, and published from then on: a card busy since looked as idle as it
was at registration, and one idle since as busy. They are now sampled with
the temperatures and power every heartbeat (ten seconds at most), with the
memory bandwidth in use, the SM clock and uncorrected ECC errors besides.
An agent too old to send them still has its registration figures published,
and a figure the driver does not give is left out rather than shown as zero.- New series:
llmgw_gpu_memory_utilization_percent,
llmgw_gpu_sm_clock_mhz;llmgw_gpu_utilization_percent,
llmgw_gpu_memory_used_bytesandllmgw_gpu_ecc_errorsnow from the
sample. - The node page shows each card's utilisation, memory, clock and ECC
errors in its table, and charts of utilisation, memory, temperature and
power, one line per card. - The cluster page charts the fleet's GPUs by class: average utilisation
and memory in use. - Grafana (
deploy/monitoring/grafana/llmgw-overview.json) has a GPU row:
utilisation, memory, memory bandwidth, temperature, power against the
limit, clock, and what holds cards back. - Alerts:
GPUThermalThrottling(ten minutes slowed by heat),
GPUUncorrectedECCErrors,GPUHot(above 87 °C for ten minutes). - The cluster page's GPU map: every card a square, a row per node,
coloured by how busy, how full or how hot it is (?gpumap=util|memory|temp),
with what holds it and its figures under the pointer; free cards dashed. - Cards held and doing nothing: the replicas whose every card reads under
5% busy, with when they were last asked — the first place to find room. - What each model gets from its cards, on the cluster page and in
Grafana: output tokens a second per GPU and per watt-hour, with the cards'
draw and average utilisation, over the last quarter hour. - What wears a card out, read every minute: XID errors from the kernel's
log (counted from when the agent started), memory rows remapped or pages
retired and whether either waits for a reset, a card out of spare rows,
PCIe link retries and NVLink errors. Published asllmgw_gpu_xid_errors,
llmgw_gpu_last_xid,llmgw_gpu_remapped_rows,llmgw_gpu_remap_pending,
llmgw_gpu_remap_failure,llmgw_gpu_retired_pages,
llmgw_gpu_retirement_pending,llmgw_gpu_pcie_replays,
llmgw_gpu_nvlink_errors; said in words on the node page; a red outline
on the GPU map; a Grafana panel; and alerts —GPUXidError,
GPUOutOfSpareRows,GPUNeedsReset,GPUPCIeRetriesClimbing,
GPUNVLinkErrorsClimbing. - GPU time by owner on the usage page: the cards each owner's models
held over the window, charted and summed into GPU-hours, whether or not the
models were asked anything — the time a GPU quota is about. From the
metrics store, as far back as it keeps history. llmgw_gpu_assigned{node,gpu,model,replica,owner}: which replica holds
each card and whose quota it counts against, for joining the GPU series to
models and tenants.
Usage is read from sums kept as it is written
Every chart and total — the usage tab, quotas, the dashboard, a person's own
page, and budgets restored at start — summed the request table again on each
view: a row per request, scanned whole. Usage is now also summed as it is
written, in the same transaction, into five-minute slices, hours and days per
key, person, project and model (usage_5m,usage_1h,usage_1d; migration
0015 fills them from what is already recorded).- Charts read the coarsest of the three their slices are made of: a day in
20-minute slices from the five-minute table, a week or a month from hours. - Totals and budgets read the rows up to the next five-minute slice, then
slices, hours and days — exact, not rounded to a slice. On three million
requests over thirty days, a month's total per person went from 705 ms to
18 ms on PostgreSQL. - Latency and time to first token are kept as histograms in the sums, so a
percentile per team or model can be read after the rows are gone. auth.usage_retention(e.g.2160h) deletes request records older than
that, hourly. Totals, charts and budgets are unchanged by it — they read the
sums — butctl requestfinds nothing older. Unset keeps every record, as
before.
Alerts per tenant and per budget
llmgw_tenant_requests_total{tenant,outcome},llmgw_tenant_tokens_total
andllmgw_tenant_ttft_seconds: requests by outcome (ok, error, limited),
tokens and time to first token per tenant. The tenant is the level
router.tenant_labelnames —teamby default, ororg,project, or
none— named by its path, "acme/platform". One level rather than every
project, so a cluster of thousands of projects does not become a Prometheus
of millions of series.llmgw_budget_spent_ratio{scope,id,name,period}: this period's spend
over the budget for every organization, team, project and person given one,
refreshed every minute.- Alerts in
deploy/monitoring/llmgw-rules.yml:BudgetMostlySpent(80%),
BudgetSpent,TenantHittingLimits(a tenth of requests refused),
TenantErrors,TenantTrafficSpike(three times the weekly average) and
TenantSlowFirstToken. - A Grafana dashboard per tenant,
deploy/monitoring/grafana/llmgw-tenants.json
(uidllmgw-tenants): requests and tokens by tenant, the share failing and
refused by a limit, first-token p95, every budget's share spent, and who used
most in the last day — with a tenant chooser at the top.
How each team's requests fared, not only what they cost
- Percentiles from the sums. First token and latency, p50, p95 and p99,
per organization, team, project, person or model, read from the histograms
kept with the usage sums: on the usage tab's table, and in a branch's or
person's panel on the people page, beside the error rate and requests
refused by a limit. - Refused requests are counted against their owner. A request a rate,
token or budget limit refused was counted only in Prometheus, by model; a
team running into its limit looked like a team that had gone quiet. It is
now recorded as a 429 against its key, person and project — no tokens, no
cost — and shown as "Limited". - Usage by model is a grouping of the usage tab, by the model that
answered.
The console watches the cluster and holds every setting
sudo llmgwon a control plane is a console in three groups down the left:- Watch, refreshed every three seconds: the overview; nodes, with CPU,
memory, GPUs free, disk and agent version, and a node's detail — load,
engines, each card's busy, memory, temperature and power; every GPU in the
fleet and which model holds it; models, with a model's profiles, warnings
and replicas; replicas and where they run; the last day's use by team,
project or organization; and a node's agent log. - Settings: every key of
server.yaml, block by block, with what applies
when the file sets nothing and what each key is for, from its comment in
the code. Enter changes a key or opens a block; lists and maps of blocks
(endpoints, models, prices) take entries added and removed. A save is read
by the same loader the service uses first — nothing is written that would
not load — keeps the file's comments, leaves the old one beside it, and
offers to restart the service. - Do: add a node, replicate the store, leave.
On a node,
sudo llmgwopens the same settings for itsagent.yaml.The keys come from the configuration structs themselves, through the
generator that writesdocs/reference.md, so a setting added to the code is on
the screen without anyone listing it, and a test fails if the two drift.
That generator told blocks apart by name alone, sotlsconf.Config,
engines.Configandsupervisor.Configwere each taken for the agent's
Configand dropped: the reference had no TLS, engine or supervisor settings
at all. It now has 37 blocks, six more than before.A second control plane with
llmgw server add, and the screens as windowssudo llmgw server add [user@]host [--vip 10.0.0.160/24], run on a
control plane, makes another machine a standby: this llmgw copied over, a
certificate from the cluster's authority, the same configuration and store
address, the machine let into the store (for a store under Patroni, one
change through Patroni's API), and the service started. Every node is told
of it within half a minute and fails over to it untouched.--vipkeeps a
floating address on whichever control plane leads, with keepalived on both.
Tried on new machines: switching the leader off, the standby led after 14
seconds and the floating address answered from it after 23.- Nodes learn every control plane from the leader (a new
ControlPlanes
message, kept in/var/lib/llmgw-agent/control-planes.json), after the
ones in their configuration. - A node notices a dead control plane within a quarter of a minute. A
leader whose machine stopped dead left its nodes' streams open until the
kernel gave up retransmitting, about fifteen minutes; the agent now pings
the stream after ten quiet seconds, and bounds dialling too. GET /leader: 200 on the control plane that leads, 503 on a standby,
open like/ready— what keepalived follows.- The store's access rules live in Patroni's shared configuration for a
cluster set up byserver init --patroni, so another control plane is let
in without touching each member. - The screens are windows: full screen, a title bar, the steps or actions
down the left, the content in a bordered panel, the keys along the bottom,
questions and errors as dialogs over it — the setup, the console, adding a
node and one-machine init alike. Keyboard only.
Machines nothing may log into:
llmgw joinFor sites that allow no ssh into a machine, the setup screens offer a fourth
way to reach one: a join code. The setup listens on port 7444 while it waits
and shows a code for each such machine, with the commands that fetch llmgw
onto it; there,sudo llmgw join <code>dials out — nothing dials in —
checks by the certificate fingerprint in the code that it reached the setup
that made it, admits itself with the code's secret, and runs what the setup
sends, printing each step, until the setup says it is done. A code is good
for one machine and an hour, and is under 90 characters so it can be copied
whole from one screen into another.To the setup a joined machine is one more way of running a script there:
the checks, PostgreSQL, Patroni and etcd go over it unchanged. Tried with one
machine joining by code and another logged into with a password, in one
setup.Setting up from nothing in screens:
sudo llmgwRun with nothing after it in a terminal, as root,
llmgwopens full-screen
setup on a machine with nothing installed, and a console on a control plane.
In a pipe or a script it prints its help, as before.- Setup asks one machine or three. For three — this one as the control
plane and etcd's third vote, two more for the store under Patroni — it asks
how to reach the other two: the ssh agent, a key file, or a password, and a
sudo password where sudo asks for one. Passwords stay in memory for the run;
a sudo password grants the user passwordless sudo until the install
finishes, and the grant is removed after, failed or not. - Every machine is checked before anything is installed: login, system
(Ubuntu 22.04/24.04, Debian 12), architecture, CPUs, memory and disk for its
role, clock against this one, NTP, the ports its role needs, internet, and
whether it reaches the other machines. Failures stop it, with what to do;
warnings let it go on. - Then the plan, and the command line that does the same from a script,
then the install with its progress, and the administrator's password. - The console shows
ctl status, refreshed every five seconds, and offers
to add a node or, for a store on one machine, to replicate it. - Machines are reached with Go's ssh client rather than the system's, so a
password can be used; a host key is recorded on first sight and refused if it
changes.
Tried end to end on three new machines whose user logs in and sudoes only with
a password.A store that survives losing its machine
deploy/ha/now covers the store as well as the control planes: PostgreSQL
under Patroni on two machines, its leader lock in a three-member etcd, and
llmgw given every member withtarget_session_attrs=read-write. Measured: a
planned switchover cost one failed lease renewal; switching off the primary's
machine promoted the standby in 24 seconds with the gateway serving
throughout. Example configurations are beside it.llmgw server init --patroni user@host --patroni user@hostsets it up:
PostgreSQL, Patroni and etcd on those machines over ssh — this one as etcd's
third vote — the role and database, and the store already in use moved in.
Every step looks first; run against a cluster that is up, it changes nothing.
Tried from scratch on three new machines (two minutes, then a failover by
switching the primary off) and against the reference cluster (nothing
touched).Watching it: for a store of several hosts the control plane publishes
llmgw_store_member,llmgw_store_replica_lag_bytesand
llmgw_store_patroni_upfrom each member's Patroni, a member that stopped
showing asmissingrather than vanishing;llmgw ctl statushas a STORE
line; and four alerts —StoreHasNoLeader,StorePatroniUnreachable,
StoreReplicaNotStreaming,StoreReplicaBehind.Making an administrator on PostgreSQL (
init --admin,users promote)
failed: the flag was sent as a bool to an integer column, which SQLite takes
and PostgreSQL refuses. Every existing PostgreSQL store got its administrator
by being copied from SQLite, which is why nobody had met it.llmgw server backupreads a store address of several hosts. It took the
list for one host name and reached the database only by falling through to
the second; it now hands libpq the list, and dumps from a standby when there
is one.People, limits and usage are one page
Managing a team meant its members on the people page, its quota on the quotas
page and its spend on the usage page, each linking to the others. They are now
one page in three tabs — Organizations & people, Limits, Usage —
each at the address it had before, so links and bookmarks still land. The menu
has one entry for them, People.Choosing an organization, team, project or person in the tree, or a person in
the account list, opens a panel beside it with everything about that one
thing: its own limits and what it is held to from above, spend against its
budget, its per-member limit, its capacity quota, and its requests, tokens
and spend over the last week, drawn. Its limits can be set or changed there,
and the answer comes back to the same panel.The administrator's menu no longer has an Organizations entry beside it:
/org
and/org/<id>send an administrator to that organization's panel here. The
page stays as it was for an organization's managers, who cannot see this one.The chat page says whether a GPU or a CPU answered
A model can have replicas on a card and on CPUs at once, and the two answer
at very different speeds; the chat page gave no way to tell which one a reply
came from. The picker now says where each model runs —GPU,CPUor
GPU + CPU— and the line under each reply names what answered it:GPU 6000-24gorCPU, beside its tokens a second.Every answer through the gateway carries the same as
x-llmgw-hardware:
cpu, or the GPU class. An endpoint configured by hand, with no pool, sends
none.A card shared with something else is judged every heartbeat
A card with another process on it when its agent started was kept out of
placement, and stayed out — "5014 MiB already allocated by another process" —
after that process let go of it, until the agent was restarted. Found on a
node shared with ComfyUI. The other way round, a tenant arriving after
registration was never seen, and a card the agent's own engine held when the
agent restarted read as someone else's.The agent now reads every card's memory per process on each heartbeat,
leaves out what its own engines hold, and sends the rest; the control plane
keeps a card out of placement on that reading, and lets it back in when it
falls under 512 MiB.llmgw_gpu_memory_used_bytesreads the same figure. An
agent too old to send it keeps the reading from registration.A GPU placement no longer runs on the CPU unnoticed
llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
where the loader looks), came up on the CPU, answered at CPU speed, and was
reported ready while its GPU sat reserved and idle. Three checks now stop it:- Installing the CUDA build asks it which devices it sees
(llama-server --list-devices). None, on a node with GPUs, is a failed
install, with whatlddsayslibggml-cuda.sois missing. The binary's
ownlddcould not say it: the backend is opened at run time. - After the upgrade, engines the agent finds already running are held to
the same check when it adopts them: one serving a GPU placement on the CPU
is stopped, and its replacement starts through the check. The installed
CUDA build is checked again within ten minutes, and installed afresh if it
sees no GPU. - Starting a shard on GPUs is ready only when its process holds memory on
each of them (nvidia-smi --query-compute-apps, its process group
included). One that does not is stopped and reported failed: "placed on GPU
0 but computing on the CPU", with the devices the engine lists. This covers
a hand-installed binary (engines.llamacpp.bin), which nothing else checks. llmgw ctl engine-check llamacpp, run on a GPU node, now also fails
when the engine sees no GPU, and otherwise prints the devices it computes on.
Shards are also given
CUDA_DEVICE_ORDER=PCI_BUS_ID: placement numbers GPUs
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
node with two kinds of GPU, a shard could get a different card from the one it
was given.A model of many files, mirrored whole for vLLM
vLLM loads a directory — shards, config.json, the tokenizer — and does not
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
shards alone, a node could be handed a version still downloading or missing a
shard, and a repository's page given as a model's source was fetched as one
file of HTML.- The explorer fetches a safetensors set with what the engine reads beside
it: config.json, generation_config.json, the tokenizer files, the shard
index, the quantisation config. Never code: a.pywould be the engine
running what the repository ships, which is to be chosen on purpose. - A version is offered to nodes only whole. The mirror refuses its
manifest (409, with what is missing) while a file is still downloading, or
while a shard itsmodel.safetensors.index.jsonnames, or a part of a
split (-00002-of-00004), is absent. A node reports "not complete on the
mirror yet" and is tried again. - Checksums go with the files. The mirror keeps each file's sha256 beside
it and the manifest carries it; a node checks every file of a set against
it before using the set, and fetches one that differs again. llmgw mirror addtakes several URLs into one version.- A Hugging Face repository page given as a source is read as the
repository id for a model vLLM or SGLang runs, which the engine downloads
itself (the node then needs to reach Hugging Face), and refused for one
llama.cpp runs, which needs a file.
With a vLLM profile of
tp: 4, placement reserves four GPUs of one node for
such a model, as before.Signing up with an address that has an account
On PostgreSQL, signing up with an address already registered showed
"something went wrong; try again": the store recognised a duplicate by the
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
code 23505. It now knows both, and the address is answered as on SQLite —
with the same page as a new sign-up, so the page cannot be used to find out
who has an account. That page now also says what to do if the address had
one: sign in, or ask an operator to reset the password.An error the page does not explain says the account was not created and
gives a reference (err-…), logged beside the error, for an operator to find
it; the database's own words stay in the log.An adapter no fabric uses does not keep a node out
llmgw doctorfailed a node for a RoCE port with no RoCE v2 GID entry, or for
ib_uverbsnot being loaded, whether or not any fabric used them; on a node
added for single-node inference, with no fabric given, that refused it for
RDMA nobody had asked for.- Those are failures only for ports a declared fabric uses, and only when a
fabric is declared. Otherwise they are warnings saying nothing uses them yet. - Which fabric an undeclared InfiniBand port is on, and whether it has a
subnet manager LID, are notes rather than warnings;node addstill warns
once that adapters are up with no fabric given. node add --fabrichands its fabrics to doctor (doctor --fabric name:mlx5_0,…), so they are checked as declared even though agent.yaml is
written afterwards.
A restart with no room beside it
ctl restartplaces each replacement beside the replica it replaces. With
no room for it — one GPU, every card that fits held by the replica itself —
the old replica was stopped only after the drain grace period, five minutes
of "restart replacing" with nothing saying why, which read as a restart that
never happened; scaling to zero and back was the way out. And a replacement
that fit nowhere even with the old replica gone was stopped for all the same,
leaving the model with nothing. Now:- The replica is the only thing in the way, found by placing the
replacement on a copy of the inventory without it: stopped at once, the
replacement taking its place. - The replacement fits nowhere even then: the restart is given up and the
replica keeps serving.ctl replicasshowsrestart refused:and why. - Room being made — a lower class preempted, an idle on-demand replica
stopping — is waited for, as before. ctl restart --in-placestops each replica first, whatever room there is.
Every GPU node gets engine builds it can run, by itself
Which build of an engine a node installs follows its hardware — the CPU
architecture, the driver, and the oldest GPU on it — from what each project
publishes and how it was compiled. Each engine has a list of builds, best
first; the node takes the first its hardware meets, and the first of those
that installs and is seen to compute on the GPU. A build the table is wrong
about costs its download, and the next one is tried.llama.cpp vLLM Driver 580+, Turing and newer CUDA 13.3 the version asked for, CUDA 13 (PyPI) Driver 525–579 CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570 the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch) Volta (V100) CUDA 12.8 on driver 570+, else Vulkan 0.19.1, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels Pascal, Maxwell CUDA 12.8 on 570+, else Vulkan none: no vLLM is built for them arm64 GPU (GH200, Grace) CUDA 13.3, else Vulkan — never the CPU build CUDA 13 or 12.9, 8.0 and newer No NVIDIA GPU the CPU build none Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
starts at 7.5.- A node running an older vLLM says so: the cluster page's vLLM build
column shows the version installed, and the installation's detail says why
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
vLLM is not placed on a V100 by accident. - The install computes on the GPU before it counts: a small tensor
operation for vLLM, the devices llama.cpp lists for its GPU builds. - The build can still be chosen per node on the cluster page (auto, CUDA
13, CUDA 12.9, none), withllmgw ctl engine-build NODE vllm BUILD, or
POST /api/v1/nodes/{id}/engines; kept in the store and sent to the node
at once. - Whether an engine computes on its GPUs is read from every process
nvidia-smi lists, not compute processes alone: the Vulkan build is a
graphics one. A llama.cpp shard is givenGGML_VK_VISIBLE_DEVICESbeside
CUDA_VISIBLE_DEVICES.
Engines per node, and nodes that do not run them all
A cluster with older GPUs could not add them cleanly:
node addrefused a
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
and--ignore-checkspassed over every other problem with it.- What vLLM cannot do is a warning, in
llmgw doctorandnode add: the
node joins without it and serves what it does run.node addchecks the
vLLM the control plane installs, not the binary's own default. node add --disable-engine vllmkeeps an engine off a node for good:
written asengines.disabledin itsagent.yaml, never installed or
advertised there, and not checked by doctor.engine_overridesinserver.yamlgive nodes picked by GPU class, id
or labels a different version of an engine, or none — for exceptions; the
build each GPU generation needs is chosen without it (above).- The requirements follow the version, as the table above sets out; the
threshold had been CUDA 13's for every version. - An engine binary named in
agent.yamlis used. The agent had installed
its own beside it, which was then used first. - The node page shows each GPU's compute capability.
Serving from the mirror says where a model will run
The serve form put every GGUF model on the CPU: it offered no choice, and its
"Profile name" field, pre-filled "cpu", read as though renaming it would move
the model to a GPU. Placement is decided bygpu_class, which the form never
set. Now:- Runs on lists the fleet's GPU classes with how many cards are free, and
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
cards is chosen. - GPUs per replica sets how many cards a replica takes on one node
(gpus_per_nodefor llama.cpp,tpfor vLLM), and parallel requests
how many a llama.cpp replica serves at once. - The profile is named for where it runs (
t4,t4-x4,cpu) rather than
typed. - A line under the form says what the model will run as — engine, CPU or how
many of which GPU, memory, context pool and how it is divided — and turns
red when the CPU is chosen with GPUs free. The message after adding repeats
it. - Choosing the CPU while GPUs are free is refused unless run on the CPU
anyway is ticked.
llama.cpp on several GPUs of one node
A llama.cpp profile takes several of a node's GPUs with
gpus_per_node: N:
placement reserves N cards of itsgpu_classon one node, the engine sees
those N (CUDA_VISIBLE_DEVICES) and spreads the layers over them
(--split-mode layer, in proportion to each card's free memory, which for
whole cards is even).tporppon a llama.cpp profile was a width nothing
placed — the replica stayed unplaced or ran on one card. It is now read as
gpus_per_node, so a profile written that way is placed on that many cards
after the upgrade, rather than dropped; only both given and disagreeing, or
nodesabove one, is refused.Downloads
- New series:
-
v0.10.7-rc.11
Pre-Release
gitea-actions
released this
2026-10-03 19:15:25 +02:00 | 521 commits to main since this releaseFrom the field report in issue #2, after a deployment on GPU nodes.
GPU metrics that move, and charts of them
A GPU's utilisation and memory in use were read once, when the agent
started, and published from then on: a card busy since looked as idle as it
was at registration, and one idle since as busy. They are now sampled with
the temperatures and power every heartbeat (ten seconds at most), with the
memory bandwidth in use, the SM clock and uncorrected ECC errors besides.
An agent too old to send them still has its registration figures published,
and a figure the driver does not give is left out rather than shown as zero.- New series:
llmgw_gpu_memory_utilization_percent,
llmgw_gpu_sm_clock_mhz;llmgw_gpu_utilization_percent,
llmgw_gpu_memory_used_bytesandllmgw_gpu_ecc_errorsnow from the
sample. - The node page shows each card's utilisation, memory, clock and ECC
errors in its table, and charts of utilisation, memory, temperature and
power, one line per card. - The cluster page charts the fleet's GPUs by class: average utilisation
and memory in use. - Grafana (
deploy/monitoring/grafana/llmgw-overview.json) has a GPU row:
utilisation, memory, memory bandwidth, temperature, power against the
limit, clock, and what holds cards back. - Alerts:
GPUThermalThrottling(ten minutes slowed by heat),
GPUUncorrectedECCErrors,GPUHot(above 87 °C for ten minutes). - The cluster page's GPU map: every card a square, a row per node,
coloured by how busy, how full or how hot it is (?gpumap=util|memory|temp),
with what holds it and its figures under the pointer; free cards dashed. - Cards held and doing nothing: the replicas whose every card reads under
5% busy, with when they were last asked — the first place to find room. - What each model gets from its cards, on the cluster page and in
Grafana: output tokens a second per GPU and per watt-hour, with the cards'
draw and average utilisation, over the last quarter hour. - What wears a card out, read every minute: XID errors from the kernel's
log (counted from when the agent started), memory rows remapped or pages
retired and whether either waits for a reset, a card out of spare rows,
PCIe link retries and NVLink errors. Published asllmgw_gpu_xid_errors,
llmgw_gpu_last_xid,llmgw_gpu_remapped_rows,llmgw_gpu_remap_pending,
llmgw_gpu_remap_failure,llmgw_gpu_retired_pages,
llmgw_gpu_retirement_pending,llmgw_gpu_pcie_replays,
llmgw_gpu_nvlink_errors; said in words on the node page; a red outline
on the GPU map; a Grafana panel; and alerts —GPUXidError,
GPUOutOfSpareRows,GPUNeedsReset,GPUPCIeRetriesClimbing,
GPUNVLinkErrorsClimbing. - GPU time by owner on the usage page: the cards each owner's models
held over the window, charted and summed into GPU-hours, whether or not the
models were asked anything — the time a GPU quota is about. From the
metrics store, as far back as it keeps history. llmgw_gpu_assigned{node,gpu,model,replica,owner}: which replica holds
each card and whose quota it counts against, for joining the GPU series to
models and tenants.
People, limits and usage are one page
Managing a team meant its members on the people page, its quota on the quotas
page and its spend on the usage page, each linking to the others. They are now
one page in three tabs — Organizations & people, Limits, Usage —
each at the address it had before, so links and bookmarks still land. The menu
has one entry for them, People.Choosing an organization, team, project or person in the tree, or a person in
the account list, opens a panel beside it with everything about that one
thing: its own limits and what it is held to from above, spend against its
budget, its per-member limit, its capacity quota, and its requests, tokens
and spend over the last week, drawn. Its limits can be set or changed there,
and the answer comes back to the same panel.The administrator's menu no longer has an Organizations entry beside it:
/org
and/org/<id>send an administrator to that organization's panel here. The
page stays as it was for an organization's managers, who cannot see this one.The chat page says whether a GPU or a CPU answered
A model can have replicas on a card and on CPUs at once, and the two answer
at very different speeds; the chat page gave no way to tell which one a reply
came from. The picker now says where each model runs —GPU,CPUor
GPU + CPU— and the line under each reply names what answered it:GPU 6000-24gorCPU, beside its tokens a second.Every answer through the gateway carries the same as
x-llmgw-hardware:
cpu, or the GPU class. An endpoint configured by hand, with no pool, sends
none.A card shared with something else is judged every heartbeat
A card with another process on it when its agent started was kept out of
placement, and stayed out — "5014 MiB already allocated by another process" —
after that process let go of it, until the agent was restarted. Found on a
node shared with ComfyUI. The other way round, a tenant arriving after
registration was never seen, and a card the agent's own engine held when the
agent restarted read as someone else's.The agent now reads every card's memory per process on each heartbeat,
leaves out what its own engines hold, and sends the rest; the control plane
keeps a card out of placement on that reading, and lets it back in when it
falls under 512 MiB.llmgw_gpu_memory_used_bytesreads the same figure. An
agent too old to send it keeps the reading from registration.A GPU placement no longer runs on the CPU unnoticed
llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
where the loader looks), came up on the CPU, answered at CPU speed, and was
reported ready while its GPU sat reserved and idle. Three checks now stop it:- Installing the CUDA build asks it which devices it sees
(llama-server --list-devices). None, on a node with GPUs, is a failed
install, with whatlddsayslibggml-cuda.sois missing. The binary's
ownlddcould not say it: the backend is opened at run time. - After the upgrade, engines the agent finds already running are held to
the same check when it adopts them: one serving a GPU placement on the CPU
is stopped, and its replacement starts through the check. The installed
CUDA build is checked again within ten minutes, and installed afresh if it
sees no GPU. - Starting a shard on GPUs is ready only when its process holds memory on
each of them (nvidia-smi --query-compute-apps, its process group
included). One that does not is stopped and reported failed: "placed on GPU
0 but computing on the CPU", with the devices the engine lists. This covers
a hand-installed binary (engines.llamacpp.bin), which nothing else checks. llmgw ctl engine-check llamacpp, run on a GPU node, now also fails
when the engine sees no GPU, and otherwise prints the devices it computes on.
Shards are also given
CUDA_DEVICE_ORDER=PCI_BUS_ID: placement numbers GPUs
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
node with two kinds of GPU, a shard could get a different card from the one it
was given.A model of many files, mirrored whole for vLLM
vLLM loads a directory — shards, config.json, the tokenizer — and does not
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
shards alone, a node could be handed a version still downloading or missing a
shard, and a repository's page given as a model's source was fetched as one
file of HTML.- The explorer fetches a safetensors set with what the engine reads beside
it: config.json, generation_config.json, the tokenizer files, the shard
index, the quantisation config. Never code: a.pywould be the engine
running what the repository ships, which is to be chosen on purpose. - A version is offered to nodes only whole. The mirror refuses its
manifest (409, with what is missing) while a file is still downloading, or
while a shard itsmodel.safetensors.index.jsonnames, or a part of a
split (-00002-of-00004), is absent. A node reports "not complete on the
mirror yet" and is tried again. - Checksums go with the files. The mirror keeps each file's sha256 beside
it and the manifest carries it; a node checks every file of a set against
it before using the set, and fetches one that differs again. llmgw mirror addtakes several URLs into one version.- A Hugging Face repository page given as a source is read as the
repository id for a model vLLM or SGLang runs, which the engine downloads
itself (the node then needs to reach Hugging Face), and refused for one
llama.cpp runs, which needs a file.
With a vLLM profile of
tp: 4, placement reserves four GPUs of one node for
such a model, as before.Signing up with an address that has an account
On PostgreSQL, signing up with an address already registered showed
"something went wrong; try again": the store recognised a duplicate by the
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
code 23505. It now knows both, and the address is answered as on SQLite —
with the same page as a new sign-up, so the page cannot be used to find out
who has an account. That page now also says what to do if the address had
one: sign in, or ask an operator to reset the password.An error the page does not explain says the account was not created and
gives a reference (err-…), logged beside the error, for an operator to find
it; the database's own words stay in the log.An adapter no fabric uses does not keep a node out
llmgw doctorfailed a node for a RoCE port with no RoCE v2 GID entry, or for
ib_uverbsnot being loaded, whether or not any fabric used them; on a node
added for single-node inference, with no fabric given, that refused it for
RDMA nobody had asked for.- Those are failures only for ports a declared fabric uses, and only when a
fabric is declared. Otherwise they are warnings saying nothing uses them yet. - Which fabric an undeclared InfiniBand port is on, and whether it has a
subnet manager LID, are notes rather than warnings;node addstill warns
once that adapters are up with no fabric given. node add --fabrichands its fabrics to doctor (doctor --fabric name:mlx5_0,…), so they are checked as declared even though agent.yaml is
written afterwards.
A restart with no room beside it
ctl restartplaces each replacement beside the replica it replaces. With
no room for it — one GPU, every card that fits held by the replica itself —
the old replica was stopped only after the drain grace period, five minutes
of "restart replacing" with nothing saying why, which read as a restart that
never happened; scaling to zero and back was the way out. And a replacement
that fit nowhere even with the old replica gone was stopped for all the same,
leaving the model with nothing. Now:- The replica is the only thing in the way, found by placing the
replacement on a copy of the inventory without it: stopped at once, the
replacement taking its place. - The replacement fits nowhere even then: the restart is given up and the
replica keeps serving.ctl replicasshowsrestart refused:and why. - Room being made — a lower class preempted, an idle on-demand replica
stopping — is waited for, as before. ctl restart --in-placestops each replica first, whatever room there is.
Every GPU node gets engine builds it can run, by itself
Which build of an engine a node installs follows its hardware — the CPU
architecture, the driver, and the oldest GPU on it — from what each project
publishes and how it was compiled. Each engine has a list of builds, best
first; the node takes the first its hardware meets, and the first of those
that installs and is seen to compute on the GPU. A build the table is wrong
about costs its download, and the next one is tried.llama.cpp vLLM Driver 580+, Turing and newer CUDA 13.3 the version asked for, CUDA 13 (PyPI) Driver 525–579 CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570 the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch) Volta (V100) CUDA 12.8 on driver 570+, else Vulkan 0.19.1, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels Pascal, Maxwell CUDA 12.8 on 570+, else Vulkan none: no vLLM is built for them arm64 GPU (GH200, Grace) CUDA 13.3, else Vulkan — never the CPU build CUDA 13 or 12.9, 8.0 and newer No NVIDIA GPU the CPU build none Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
starts at 7.5.- A node running an older vLLM says so: the cluster page's vLLM build
column shows the version installed, and the installation's detail says why
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
vLLM is not placed on a V100 by accident. - The install computes on the GPU before it counts: a small tensor
operation for vLLM, the devices llama.cpp lists for its GPU builds. - The build can still be chosen per node on the cluster page (auto, CUDA
13, CUDA 12.9, none), withllmgw ctl engine-build NODE vllm BUILD, or
POST /api/v1/nodes/{id}/engines; kept in the store and sent to the node
at once. - Whether an engine computes on its GPUs is read from every process
nvidia-smi lists, not compute processes alone: the Vulkan build is a
graphics one. A llama.cpp shard is givenGGML_VK_VISIBLE_DEVICESbeside
CUDA_VISIBLE_DEVICES.
Engines per node, and nodes that do not run them all
A cluster with older GPUs could not add them cleanly:
node addrefused a
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
and--ignore-checkspassed over every other problem with it.- What vLLM cannot do is a warning, in
llmgw doctorandnode add: the
node joins without it and serves what it does run.node addchecks the
vLLM the control plane installs, not the binary's own default. node add --disable-engine vllmkeeps an engine off a node for good:
written asengines.disabledin itsagent.yaml, never installed or
advertised there, and not checked by doctor.engine_overridesinserver.yamlgive nodes picked by GPU class, id
or labels a different version of an engine, or none — for exceptions; the
build each GPU generation needs is chosen without it (above).- The requirements follow the version, as the table above sets out; the
threshold had been CUDA 13's for every version. - An engine binary named in
agent.yamlis used. The agent had installed
its own beside it, which was then used first. - The node page shows each GPU's compute capability.
Serving from the mirror says where a model will run
The serve form put every GGUF model on the CPU: it offered no choice, and its
"Profile name" field, pre-filled "cpu", read as though renaming it would move
the model to a GPU. Placement is decided bygpu_class, which the form never
set. Now:- Runs on lists the fleet's GPU classes with how many cards are free, and
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
cards is chosen. - GPUs per replica sets how many cards a replica takes on one node
(gpus_per_nodefor llama.cpp,tpfor vLLM), and parallel requests
how many a llama.cpp replica serves at once. - The profile is named for where it runs (
t4,t4-x4,cpu) rather than
typed. - A line under the form says what the model will run as — engine, CPU or how
many of which GPU, memory, context pool and how it is divided — and turns
red when the CPU is chosen with GPUs free. The message after adding repeats
it. - Choosing the CPU while GPUs are free is refused unless run on the CPU
anyway is ticked.
llama.cpp on several GPUs of one node
A llama.cpp profile takes several of a node's GPUs with
gpus_per_node: N:
placement reserves N cards of itsgpu_classon one node, the engine sees
those N (CUDA_VISIBLE_DEVICES) and spreads the layers over them
(--split-mode layer, in proportion to each card's free memory, which for
whole cards is even).tporppon a llama.cpp profile was a width nothing
placed — the replica stayed unplaced or ran on one card. It is now read as
gpus_per_node, so a profile written that way is placed on that many cards
after the upgrade, rather than dropped; only both given and disagreeing, or
nodesabove one, is refused.Downloads
- New series:
-
v0.10.7-rc.10
Pre-Release
gitea-actions
released this
2026-10-03 19:10:48 +02:00 | 522 commits to main since this releaseFrom the field report in issue #2, after a deployment on GPU nodes.
GPU metrics that move, and charts of them
A GPU's utilisation and memory in use were read once, when the agent
started, and published from then on: a card busy since looked as idle as it
was at registration, and one idle since as busy. They are now sampled with
the temperatures and power every heartbeat (ten seconds at most), with the
memory bandwidth in use, the SM clock and uncorrected ECC errors besides.
An agent too old to send them still has its registration figures published,
and a figure the driver does not give is left out rather than shown as zero.- New series:
llmgw_gpu_memory_utilization_percent,
llmgw_gpu_sm_clock_mhz;llmgw_gpu_utilization_percent,
llmgw_gpu_memory_used_bytesandllmgw_gpu_ecc_errorsnow from the
sample. - The node page shows each card's utilisation, memory, clock and ECC
errors in its table, and charts of utilisation, memory, temperature and
power, one line per card. - The cluster page charts the fleet's GPUs by class: average utilisation
and memory in use. - Grafana (
deploy/monitoring/grafana/llmgw-overview.json) has a GPU row:
utilisation, memory, memory bandwidth, temperature, power against the
limit, clock, and what holds cards back. - Alerts:
GPUThermalThrottling(ten minutes slowed by heat),
GPUUncorrectedECCErrors,GPUHot(above 87 °C for ten minutes). - The cluster page's GPU map: every card a square, a row per node,
coloured by how busy, how full or how hot it is (?gpumap=util|memory|temp),
with what holds it and its figures under the pointer; free cards dashed. - Cards held and doing nothing: the replicas whose every card reads under
5% busy, with when they were last asked — the first place to find room. - What each model gets from its cards, on the cluster page and in
Grafana: output tokens a second per GPU and per watt-hour, with the cards'
draw and average utilisation, over the last quarter hour. - What wears a card out, read every minute: XID errors from the kernel's
log (counted from when the agent started), memory rows remapped or pages
retired and whether either waits for a reset, a card out of spare rows,
PCIe link retries and NVLink errors. Published asllmgw_gpu_xid_errors,
llmgw_gpu_last_xid,llmgw_gpu_remapped_rows,llmgw_gpu_remap_pending,
llmgw_gpu_remap_failure,llmgw_gpu_retired_pages,
llmgw_gpu_retirement_pending,llmgw_gpu_pcie_replays,
llmgw_gpu_nvlink_errors; said in words on the node page; a red outline
on the GPU map; a Grafana panel; and alerts —GPUXidError,
GPUOutOfSpareRows,GPUNeedsReset,GPUPCIeRetriesClimbing,
GPUNVLinkErrorsClimbing. - GPU time by owner on the usage page: the cards each owner's models
held over the window, charted and summed into GPU-hours, whether or not the
models were asked anything — the time a GPU quota is about. From the
metrics store, as far back as it keeps history. llmgw_gpu_assigned{node,gpu,model,replica,owner}: which replica holds
each card and whose quota it counts against, for joining the GPU series to
models and tenants.
People, limits and usage are one page
Managing a team meant its members on the people page, its quota on the quotas
page and its spend on the usage page, each linking to the others. They are now
one page in three tabs — Organizations & people, Limits, Usage —
each at the address it had before, so links and bookmarks still land. The menu
has one entry for them, People.Choosing an organization, team, project or person in the tree, or a person in
the account list, opens a panel beside it with everything about that one
thing: its own limits and what it is held to from above, spend against its
budget, its per-member limit, its capacity quota, and its requests, tokens
and spend over the last week, drawn. Its limits can be set or changed there,
and the answer comes back to the same panel.The chat page says whether a GPU or a CPU answered
A model can have replicas on a card and on CPUs at once, and the two answer
at very different speeds; the chat page gave no way to tell which one a reply
came from. The picker now says where each model runs —GPU,CPUor
GPU + CPU— and the line under each reply names what answered it:GPU 6000-24gorCPU, beside its tokens a second.Every answer through the gateway carries the same as
x-llmgw-hardware:
cpu, or the GPU class. An endpoint configured by hand, with no pool, sends
none.A card shared with something else is judged every heartbeat
A card with another process on it when its agent started was kept out of
placement, and stayed out — "5014 MiB already allocated by another process" —
after that process let go of it, until the agent was restarted. Found on a
node shared with ComfyUI. The other way round, a tenant arriving after
registration was never seen, and a card the agent's own engine held when the
agent restarted read as someone else's.The agent now reads every card's memory per process on each heartbeat,
leaves out what its own engines hold, and sends the rest; the control plane
keeps a card out of placement on that reading, and lets it back in when it
falls under 512 MiB.llmgw_gpu_memory_used_bytesreads the same figure. An
agent too old to send it keeps the reading from registration.A GPU placement no longer runs on the CPU unnoticed
llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
where the loader looks), came up on the CPU, answered at CPU speed, and was
reported ready while its GPU sat reserved and idle. Three checks now stop it:- Installing the CUDA build asks it which devices it sees
(llama-server --list-devices). None, on a node with GPUs, is a failed
install, with whatlddsayslibggml-cuda.sois missing. The binary's
ownlddcould not say it: the backend is opened at run time. - After the upgrade, engines the agent finds already running are held to
the same check when it adopts them: one serving a GPU placement on the CPU
is stopped, and its replacement starts through the check. The installed
CUDA build is checked again within ten minutes, and installed afresh if it
sees no GPU. - Starting a shard on GPUs is ready only when its process holds memory on
each of them (nvidia-smi --query-compute-apps, its process group
included). One that does not is stopped and reported failed: "placed on GPU
0 but computing on the CPU", with the devices the engine lists. This covers
a hand-installed binary (engines.llamacpp.bin), which nothing else checks. llmgw ctl engine-check llamacpp, run on a GPU node, now also fails
when the engine sees no GPU, and otherwise prints the devices it computes on.
Shards are also given
CUDA_DEVICE_ORDER=PCI_BUS_ID: placement numbers GPUs
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
node with two kinds of GPU, a shard could get a different card from the one it
was given.A model of many files, mirrored whole for vLLM
vLLM loads a directory — shards, config.json, the tokenizer — and does not
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
shards alone, a node could be handed a version still downloading or missing a
shard, and a repository's page given as a model's source was fetched as one
file of HTML.- The explorer fetches a safetensors set with what the engine reads beside
it: config.json, generation_config.json, the tokenizer files, the shard
index, the quantisation config. Never code: a.pywould be the engine
running what the repository ships, which is to be chosen on purpose. - A version is offered to nodes only whole. The mirror refuses its
manifest (409, with what is missing) while a file is still downloading, or
while a shard itsmodel.safetensors.index.jsonnames, or a part of a
split (-00002-of-00004), is absent. A node reports "not complete on the
mirror yet" and is tried again. - Checksums go with the files. The mirror keeps each file's sha256 beside
it and the manifest carries it; a node checks every file of a set against
it before using the set, and fetches one that differs again. llmgw mirror addtakes several URLs into one version.- A Hugging Face repository page given as a source is read as the
repository id for a model vLLM or SGLang runs, which the engine downloads
itself (the node then needs to reach Hugging Face), and refused for one
llama.cpp runs, which needs a file.
With a vLLM profile of
tp: 4, placement reserves four GPUs of one node for
such a model, as before.Signing up with an address that has an account
On PostgreSQL, signing up with an address already registered showed
"something went wrong; try again": the store recognised a duplicate by the
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
code 23505. It now knows both, and the address is answered as on SQLite —
with the same page as a new sign-up, so the page cannot be used to find out
who has an account. That page now also says what to do if the address had
one: sign in, or ask an operator to reset the password.An error the page does not explain says the account was not created and
gives a reference (err-…), logged beside the error, for an operator to find
it; the database's own words stay in the log.An adapter no fabric uses does not keep a node out
llmgw doctorfailed a node for a RoCE port with no RoCE v2 GID entry, or for
ib_uverbsnot being loaded, whether or not any fabric used them; on a node
added for single-node inference, with no fabric given, that refused it for
RDMA nobody had asked for.- Those are failures only for ports a declared fabric uses, and only when a
fabric is declared. Otherwise they are warnings saying nothing uses them yet. - Which fabric an undeclared InfiniBand port is on, and whether it has a
subnet manager LID, are notes rather than warnings;node addstill warns
once that adapters are up with no fabric given. node add --fabrichands its fabrics to doctor (doctor --fabric name:mlx5_0,…), so they are checked as declared even though agent.yaml is
written afterwards.
A restart with no room beside it
ctl restartplaces each replacement beside the replica it replaces. With
no room for it — one GPU, every card that fits held by the replica itself —
the old replica was stopped only after the drain grace period, five minutes
of "restart replacing" with nothing saying why, which read as a restart that
never happened; scaling to zero and back was the way out. And a replacement
that fit nowhere even with the old replica gone was stopped for all the same,
leaving the model with nothing. Now:- The replica is the only thing in the way, found by placing the
replacement on a copy of the inventory without it: stopped at once, the
replacement taking its place. - The replacement fits nowhere even then: the restart is given up and the
replica keeps serving.ctl replicasshowsrestart refused:and why. - Room being made — a lower class preempted, an idle on-demand replica
stopping — is waited for, as before. ctl restart --in-placestops each replica first, whatever room there is.
Every GPU node gets engine builds it can run, by itself
Which build of an engine a node installs follows its hardware — the CPU
architecture, the driver, and the oldest GPU on it — from what each project
publishes and how it was compiled. Each engine has a list of builds, best
first; the node takes the first its hardware meets, and the first of those
that installs and is seen to compute on the GPU. A build the table is wrong
about costs its download, and the next one is tried.llama.cpp vLLM Driver 580+, Turing and newer CUDA 13.3 the version asked for, CUDA 13 (PyPI) Driver 525–579 CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570 the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch) Volta (V100) CUDA 12.8 on driver 570+, else Vulkan 0.19.1, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels Pascal, Maxwell CUDA 12.8 on 570+, else Vulkan none: no vLLM is built for them arm64 GPU (GH200, Grace) CUDA 13.3, else Vulkan — never the CPU build CUDA 13 or 12.9, 8.0 and newer No NVIDIA GPU the CPU build none Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
starts at 7.5.- A node running an older vLLM says so: the cluster page's vLLM build
column shows the version installed, and the installation's detail says why
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
vLLM is not placed on a V100 by accident. - The install computes on the GPU before it counts: a small tensor
operation for vLLM, the devices llama.cpp lists for its GPU builds. - The build can still be chosen per node on the cluster page (auto, CUDA
13, CUDA 12.9, none), withllmgw ctl engine-build NODE vllm BUILD, or
POST /api/v1/nodes/{id}/engines; kept in the store and sent to the node
at once. - Whether an engine computes on its GPUs is read from every process
nvidia-smi lists, not compute processes alone: the Vulkan build is a
graphics one. A llama.cpp shard is givenGGML_VK_VISIBLE_DEVICESbeside
CUDA_VISIBLE_DEVICES.
Engines per node, and nodes that do not run them all
A cluster with older GPUs could not add them cleanly:
node addrefused a
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
and--ignore-checkspassed over every other problem with it.- What vLLM cannot do is a warning, in
llmgw doctorandnode add: the
node joins without it and serves what it does run.node addchecks the
vLLM the control plane installs, not the binary's own default. node add --disable-engine vllmkeeps an engine off a node for good:
written asengines.disabledin itsagent.yaml, never installed or
advertised there, and not checked by doctor.engine_overridesinserver.yamlgive nodes picked by GPU class, id
or labels a different version of an engine, or none — for exceptions; the
build each GPU generation needs is chosen without it (above).- The requirements follow the version, as the table above sets out; the
threshold had been CUDA 13's for every version. - An engine binary named in
agent.yamlis used. The agent had installed
its own beside it, which was then used first. - The node page shows each GPU's compute capability.
Serving from the mirror says where a model will run
The serve form put every GGUF model on the CPU: it offered no choice, and its
"Profile name" field, pre-filled "cpu", read as though renaming it would move
the model to a GPU. Placement is decided bygpu_class, which the form never
set. Now:- Runs on lists the fleet's GPU classes with how many cards are free, and
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
cards is chosen. - GPUs per replica sets how many cards a replica takes on one node
(gpus_per_nodefor llama.cpp,tpfor vLLM), and parallel requests
how many a llama.cpp replica serves at once. - The profile is named for where it runs (
t4,t4-x4,cpu) rather than
typed. - A line under the form says what the model will run as — engine, CPU or how
many of which GPU, memory, context pool and how it is divided — and turns
red when the CPU is chosen with GPUs free. The message after adding repeats
it. - Choosing the CPU while GPUs are free is refused unless run on the CPU
anyway is ticked.
llama.cpp on several GPUs of one node
A llama.cpp profile takes several of a node's GPUs with
gpus_per_node: N:
placement reserves N cards of itsgpu_classon one node, the engine sees
those N (CUDA_VISIBLE_DEVICES) and spreads the layers over them
(--split-mode layer, in proportion to each card's free memory, which for
whole cards is even).tporppon a llama.cpp profile was a width nothing
placed — the replica stayed unplaced or ran on one card. It is now read as
gpus_per_node, so a profile written that way is placed on that many cards
after the upgrade, rather than dropped; only both given and disagreeing, or
nodesabove one, is refused.Downloads
- New series: