Provision nodes with the host CPU model

Proxmox defaults a VM to kvm64, which hides AVX, AVX2, FMA and even SSE4.2
from the guest. llama.cpp then loads libggml-cpu-x64.so — its scalar
baseline — instead of the backend built for the host's instruction set, and
an inference node is not slightly slower but unusable.

Measured on node05, an 8-core VM on a Xeon E5-2680 v4, same weights, same
thread count, the only change being --cpu host:

  Qwen2.5-Coder     kvm64   host
  1.5B q4_K_M        5.80  39.09  tok/s
  3B   q4_K_M        2.90  20.51
  7B   q4_K_M        1.33   9.88

Prompt processing went from 7.6 to 143.8, a factor of nineteen.

This cost an afternoon because nothing looks broken. The node registers, the
engine starts, the replies are correct; every number is just small, and the
conclusion that suggests itself is that CPU inference is hopeless rather
than that the VM cannot multiply efficiently. A 14B model at 1.7 tokens a
second was nearly written off on that basis.

The trade is live migration, which a VM pinned to the host CPU cannot do.
For a node whose job is arithmetic that is the right way round, and --cpu
overrides it where it is not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-16 20:42:48 +02:00
1 parent 8a894af1ca
commit 634f87ba15
3 files changed
+45 -3

No files matched your search

+5 -2
View File
@@ -23,6 +23,7 @@ usage: add-node.sh --name NODE --ip ADDR/CIDR --server URL [options]
--vmid N Proxmox VMID (default: the last octet of --ip)
--cores N (default 4)
--cpu TYPE QEMU CPU model (default host; see provision-node.sh)
--memory MB (default 8192)
--disk GB (default 20)
--fabric NAME InfiniBand fabric; repeatable as NAME:hca1,hca2.
@@ -39,7 +40,7 @@ USAGE
exit 2
}
NAME= IP= SERVER= VMID= CORES=4 MEMORY=8192 DISK=20
NAME= IP= SERVER= VMID= CORES=4 MEMORY=8192 DISK=20 CPU=
BINARY= PKI=/etc/llmgw/pki SSH_USER=ilker
PROVISION=1 ONBOARD=1
set -- "$@"
@@ -51,6 +52,7 @@ while [ $# -gt 0 ]; do
--server) SERVER="$2"; shift 2;;
--vmid) VMID="$2"; shift 2;;
--cores) CORES="$2"; shift 2;;
--cpu) CPU="$2"; shift 2;;
--memory) MEMORY="$2"; shift 2;;
--disk) DISK="$2"; shift 2;;
--binary) BINARY="$2"; shift 2;;
@@ -82,8 +84,9 @@ ONBOARD_SH="$(here)/onboard-node.sh"
if [ "$PROVISION" = 1 ]; then
[ -x "$PROVISION_SH" ] || { echo "cannot find provision-node.sh" >&2; exit 1; }
echo "== 1/2 building $NAME as VM $VMID"
# shellcheck disable=SC2086
"$PROVISION_SH" --vmid "$VMID" --name "$NAME" --ip "$IP" \
--cores "$CORES" --memory "$MEMORY" --disk "$DISK"
--cores "$CORES" --memory "$MEMORY" --disk "$DISK" ${CPU:+--cpu "$CPU"}
fi
if [ "$ONBOARD" = 0 ]; then