-
v0.16.2
Stable
gitea-actions
released this
2026-11-02 08:42:29 +01:00 | 45 commits to main since this releaseStandby control planes that answer clients themselves, with rate limits and budgets kept across them; FreeToken, an
engine for models larger than the card; a served model's longest answer changed in place; and fixes found putting
the router under load and fetching large models: a multi-file model fetched once per node however many replicas
start there, downloads that carry on where they were cut off, and a router that does twice the work on two cores.Upgrading.
- No migrations.
llmgw ctl upgrade v0.16.2: the control planes carry active-active routing (off until
llmgw ctl config set router.active_active true) and the router fixes; the agents carry the set fetched once
per node and the set's progress and checking; the mirror carries the resumed downloads.
Added
- Standbys that answer the client API themselves. With
router.active_active: truein the cluster's settings, a standby control
plane routes/v1to the leader's replicas instead of passing every
request to the leader, so more machines carry the traffic. It hears the
replicas from the leader every two seconds over the agent port (no engine
keys travel) and reports back what it used, so the leader does not evict
a replica busy behind a standby, and the on-demand models it was asked
for. Rate limits are the cluster's: each router's request and token
counts reach the others through the leader every two seconds, so a key
spread over several control planes gets its limit, not that many times
it. Budgets are brought up to the store's every 30 seconds. Off by
default; the portal,/adminand/mcpstill go to the leader. - A served model's longest answer, changed in place.
llmgw ctl model context <model> --max-output N(andmax_outputonPOST /api/v1/models/{name}/context, and the console's Context length… form)
sets the longest answer on its own: the catalog's limit, in force at once,
with no replica started again. With a context it is set with it. Strata's
context can now be changed too (its setup's--context), so neither
needs the model removed and added again. - FreeToken, an engine for models larger than the card. Mixture-of-experts
models several times a card's size, from one card, with the experts in RAM
and streamed over PCIe — read from Hugging Face safetensors checkpoints as
published (DeepSeek V4 Flash, GLM-5.x, Qwen3.6/3.8, gpt-oss, Gemma 4,
MiniMax), with batching. Installed like vLLM on every node it runs on: a
card of compute capability 8.0 or newer, driver 580+, x86-64, and the CUDA 13
toolkit it compiles kernels with; older cards are passed over (tried on a
Turing card, where 0.2.1 has no kernels). Profiles are one card
(engine: freetoken); the key goes in its environment, and flags that read
files or fetch are refused. Readiness now also waits for an engine whose
/healthanswers 200 while sayingloadingin its body. FreeToken is
followed release by release: a weekly workflow fails when PyPI has a version
newer thanDefaultEnginesinstalls. - How much of each GPU works, beside whether anything does. The busy
figure is nvidia-smi'sutilization.gpu— the share of time any kernel
ran, which a kernel on one SM fills — and it is now said to be that: on
the dashboard, the node page, the metric's help and Grafana. Beside it,
each card's draw against its power limit (the efficiency table, the fleet
and node charts, the console's GPU work chart), and where the card's
profiling counters can be read, its SMs, SM occupancy, tensor cores and
memory interface:llmgw_gpu_sm_active_percent,
llmgw_gpu_sm_occupancy_percent,llmgw_gpu_tensor_active_percent,
llmgw_gpu_dram_active_percent. The agent reads them from NVML's GPM on
Hopper and later, and from a dcgm-exporter on the node elsewhere
(discovery.dcgm_exporter, defaulthttp://127.0.0.1:9400/metrics,off
for none). The GPU map colours by this "work" by default — SMs active
where read, power against the limit otherwise — and keeps busy as a mode.
On cuda01 decode read 92% busy with the SMs at work 69% of the time.
Fixed
- Several replicas of a multi-file model on one node fetched it once
each, over one another. Four replicas placed on a node's four cards
wanted the same safetensors set at the same moment, each wrote the same
.partfiles, and all but one failed to start ("rename … no such file")
until a retry found the set finished. A set is now fetched once per node
and every replica waiting on it gets it, as single files always were. - A model of several files showed one file's download as the whole. A node fetching a set reported the part it
was on — "59% of 46.1 GB" for a 188 GB model — instead of the set: it now reports the set's bytes against its
whole size. - A set's files checked with nothing said, one at a time. After the last byte arrived a node checked each file
against the mirror's checksum, 195 GB on one core for eight minutes, while the replica still read "fetching 100%".
It now checks four at once and says so:checking 40% of 195 GBinctl replicas, "Checking the weights" in the
chat page; and a check that takes long is not taken for a stuck download. - A mirror download the registry cut off started again from nothing, or not at all. Hugging Face's CDN ended a
60 GB transfer at 23 GB (stream error … CANCEL), the mirror deleted what had arrived, andctl model serve
failed. The mirror now tries again up to six times, carrying on from where it got to (Range), keeps what arrived
when it gives up, and carries on from it after a restart too. - A model from an outside provider took three or four requests at once. Admission read every request in flight to
an endpoint it does not measure as a queue, so an external model (OpenAI, another llmgw cluster) was held to what a
saturation of 0.85 allows of "in flight / 4"; the rest waited and timed out with 504. An outside provider queues for
itself and says 429 when full, which its breaker already hears: it no longer counts towards saturation. - A valid key told to slow down under load, every thirty seconds. When a busy key's cached entry expired, every
request in flight missed the cache together and each took one of the address's allowance for unknown keys: past a
hundred or so streams at once, a valid key got 429 "too many unrecognised api keys". Requests missing the same key
at once now share one read and one allowance. - A streamed answer was one write to the client a token. Each event was flushed alone, a system call each: a
third of a busy router's CPU. Events already received from the engine now go out in one write, with no token
waiting for another; on the PoC's two-core leader the ceiling rose from ~575 to ~700 streams' requests a second,
and the first token at 128 at once from 91 to 45 ms.
Downloads
- No migrations.