-
v0.16.3
Stable
gitea-actions
released this
2026-11-03 09:56:29 +01:00 | 41 commits to main since this releaseReplicas counted per profile — seven on cards and four on CPUs under one name, the cards filled first — and every
setting of a served model from the mirror page; how fast a model answers on the CPU, said before it is served, by
the threads it is given rather than the whole machine's memory.Upgrading.
- No migrations.
llmgw ctl upgrade v0.16.3: the control planes carry the replicas per profile, the model card and
the speed in the serve form and inctl model plan; the agents carry the measurement of one core's memory rate,
without which the speed is reckoned the old way, from the whole machine's.
Added
- Replicas counted per profile: seven on cards and four on CPUs, under one
name.llmgw ctl model replicas <model> gpu=7 cpu=4, the console's
Replicas per profile… and the mirror page's model card set how many of a
model's replicas run on each of its profiles, rather than every replica
on the best profile that fits. For a GGUF model with no profile on CPUs
one is added (--cpu-memory), preferred below the cards, so the router
fills the cards first and sends to the CPUs once they are full. A change
starts the replicas short before stopping those over: the model never
runs fewer.ctl modelsshows the counts; a scaling of another total
ends them. - Every setting of a served model from the mirror page. A served
model's row has a Settings link, and its card now changes the context
and longest answer, the cards a replica takes, the replicas on each
profile (with the CPU's speed beside), stops and starts it, restarts its
replicas, and takes it out of the catalog with its files or without —
beside the replicas, class, owner, memory and adapters it held before.
Stopped on the row is now the model stopped and kept, started again as
it was, rather than its scaling set to nothing. - How fast a model answers on the CPU, before it is served. The serve
form says, as the cores are changed, about how many tokens a second one
reply gets on the CPU and on one card of the fleet's fastest class, and
warns when it is slow (under 15) or very slow (under 5, a reply of 500
tokens taking minutes). Very slow on the CPU is refused until "run on the
CPU anyway" is ticked, GPUs free or not.llmgw ctl model planand the
console's Hugging Face screen show the same figure for every way a model
can run, andctl model serve --cpusays it in its log. - The speed follows the threads a replica is given. Each agent now
measures what one core reads beside what all of them read: a replica of
four threads on a large server reads at about four cores' rate, not the
server's, which the estimate used to assume and overstated several times.
Calibrated on the PoC by llama-bench at 1, 2, 4 and 8 threads (within 15%),
and a model of experts reads its experts' share, from the GGUF header.
Agents need upgrading to report one core's rate; until then the old
estimate is used.
Downloads
- No migrations.