How fast a model answers on the CPU, before it is served
CI / test (push) Successful in 9m31s

Each agent measures one core's memory rate beside the whole machine's; the
estimate takes a replica's threads at about one core's rate each, up to the
machine's, calibrated by llama-bench on the PoC. The serve form shows it
beside a card's, warns when slow, and asks for the CPU on purpose when a
reply would take minutes; ctl model plan, the console and serve jobs say
it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-11-03 08:52:15 +01:00
1 parent 220110ed0d
commit 6403e61f4b
28 files changed
+657 -62

No files matched your search

+8
View File
@@ -8305,6 +8305,10 @@
"type": "number",
"description": "MemBandwidthGBs is how fast its memory reads, in GB/s, measured when its agent started; 0 when not reported. A model's speed on the CPU."
},
"mem_bandwidth_core_gbs": {
"type": "number",
"description": "MemBandwidthCoreGBs is what one core reads alone, in GB/s; 0 when not reported. A CPU replica's threads read at about this each."
},
"engines": {
"type": "array",
"items": {
@@ -11406,6 +11410,10 @@
},
"strata_size": {
"type": "string"
},
"tokens_per_second": {
"type": "number",
"description": "TokensPerSecond is about how many tokens a second one reply gets this way, on the fastest host it fits; 0 when not known. On the CPU with the threads a replica is given by default: half the node's."
}
}
},