Proxmox defaults a VM to kvm64, which hides AVX, AVX2, FMA and even SSE4.2
from the guest. llama.cpp then loads libggml-cpu-x64.so — its scalar
baseline — instead of the backend built for the host's instruction set, and
an inference node is not slightly slower but unusable.
Measured on node05, an 8-core VM on a Xeon E5-2680 v4, same weights, same
thread count, the only change being --cpu host:
Qwen2.5-Coder kvm64 host
1.5B q4_K_M 5.80 39.09 tok/s
3B q4_K_M 2.90 20.51
7B q4_K_M 1.33 9.88
Prompt processing went from 7.6 to 143.8, a factor of nineteen.
This cost an afternoon because nothing looks broken. The node registers, the
engine starts, the replies are correct; every number is just small, and the
conclusion that suggests itself is that CPU inference is hopeless rather
than that the VM cannot multiply efficiently. A 14B model at 1.7 tokens a
second was nearly written off on that basis.
The trade is live migration, which a VM pinned to the host CPU cannot do.
For a node whose job is arithmetic that is the right way round, and --cpu
overrides it where it is not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
