Provision nodes with the host CPU model
CI / test (push) Successful in 1m26s

Proxmox defaults a VM to kvm64, which hides AVX, AVX2, FMA and even SSE4.2
from the guest. llama.cpp then loads libggml-cpu-x64.so — its scalar
baseline — instead of the backend built for the host's instruction set, and
an inference node is not slightly slower but unusable.

Measured on node05, an 8-core VM on a Xeon E5-2680 v4, same weights, same
thread count, the only change being --cpu host:

  Qwen2.5-Coder     kvm64   host
  1.5B q4_K_M        5.80  39.09  tok/s
  3B   q4_K_M        2.90  20.51
  7B   q4_K_M        1.33   9.88

Prompt processing went from 7.6 to 143.8, a factor of nineteen.

This cost an afternoon because nothing looks broken. The node registers, the
engine starts, the replies are correct; every number is just small, and the
conclusion that suggests itself is that CPU inference is hopeless rather
than that the VM cannot multiply efficiently. A 14B model at 1.7 tokens a
second was nearly written off on that basis.

The trade is live migration, which a VM pinned to the host CPU cannot do.
For a node whose job is arithmetic that is the right way round, and --cpu
overrides it where it is not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-16 20:42:48 +02:00
1 parent 8a894af1ca
commit 634f87ba15
3 files changed
+45 -3

No files matched your search

+26
View File
@@ -90,6 +90,32 @@ it accepted a moment earlier.
It also checks that a configured engine actually runs, which is cheap here and
expensive later — see below.
## The CPU model matters, and its default is wrong
Nodes are provisioned with `--cpu host`. Proxmox's default, `kvm64`, hides
AVX, AVX2, FMA and even SSE4.2 from the guest, and llama.cpp then loads its
scalar backend instead of the one built for the host's instruction set.
Measured on node05, an 8-core VM on a Xeon E5-2680 v4, same weights, same
threads, the only change being the CPU model:
| Qwen2.5-Coder | kvm64 tok/s | host tok/s |
| -------------- | ----------: | ---------: |
| 1.5B q4_K_M | 5.80 | 39.09 |
| 3B q4_K_M | 2.90 | 20.51 |
| 7B q4_K_M | 1.33 | 9.88 |
Prompt processing is worse still: 7.6 tokens a second against 143.8, a factor
of nineteen.
It is worth calling out because nothing looks broken. The node registers, the
engine starts, replies are correct — every number is simply small, and the
obvious conclusion is that CPU inference is hopeless rather than that the VM
cannot multiply efficiently.
The trade is live migration: a VM pinned to the host CPU cannot move to a
machine with a different one. For a node whose job is arithmetic that is the
right way round, and `--cpu` overrides it where it is not.
## What onboarding does not do
It gives a machine an identity and an agent. It does **not** install inference