Provision nodes with the host CPU model
CI / test (push) Successful in 1m26s

Proxmox defaults a VM to kvm64, which hides AVX, AVX2, FMA and even SSE4.2
from the guest. llama.cpp then loads libggml-cpu-x64.so — its scalar
baseline — instead of the backend built for the host's instruction set, and
an inference node is not slightly slower but unusable.

Measured on node05, an 8-core VM on a Xeon E5-2680 v4, same weights, same
thread count, the only change being --cpu host:

  Qwen2.5-Coder     kvm64   host
  1.5B q4_K_M        5.80  39.09  tok/s
  3B   q4_K_M        2.90  20.51
  7B   q4_K_M        1.33   9.88

Prompt processing went from 7.6 to 143.8, a factor of nineteen.

This cost an afternoon because nothing looks broken. The node registers, the
engine starts, the replies are correct; every number is just small, and the
conclusion that suggests itself is that CPU inference is hopeless rather
than that the VM cannot multiply efficiently. A 14B model at 1.7 tokens a
second was nearly written off on that basis.

The trade is live migration, which a VM pinned to the host CPU cannot do.
For a node whose job is arithmetic that is the right way round, and --cpu
overrides it where it is not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-16 20:42:48 +02:00
1 parent 8a894af1ca
commit 634f87ba15
3 files changed
+45 -3

No files matched your search

+14 -1
View File
@@ -21,6 +21,7 @@ usage: $0 --vmid N --name NAME --ip ADDR [options]
--ip ADDR address with prefix, e.g. 10.0.0.144/24
--gw ADDR gateway (default 10.0.0.1)
--cores N (default 4)
--cpu TYPE QEMU CPU model (default host)
--memory MB (default 8192)
--disk GB (default 20)
--bridge NAME (default vmbr0)
@@ -33,6 +34,17 @@ USAGE
}
VMID= NAME= IP= GW=10.0.0.1 CORES=4 MEMORY=8192 DISK=20
# host, not Proxmox's kvm64 default. kvm64 hides AVX, AVX2, FMA and even
# SSE4.2 from the guest, and an inference node without vector instructions is
# not slightly slower, it is unusable: llama.cpp falls back to its scalar
# backend and a 1.5B model dropped from 39 tokens a second to 5.8, with prompt
# processing 19x worse. It cost an afternoon to find because nothing is broken
# — every number is simply small.
#
# The trade is live migration: a VM pinned to the host CPU cannot move to a
# machine with a different one. For a node whose job is arithmetic, that is
# the right way round.
CPU=host
BRIDGE=vmbr0 STORAGE=local-lvm USER=ilker
IMAGE=/var/lib/vz/template/iso/noble-server-cloudimg-amd64.img
SSHKEYS=/root/.ssh/authorized_keys
@@ -44,6 +56,7 @@ while [ $# -gt 0 ]; do
--ip) IP="$2"; shift 2;;
--gw) GW="$2"; shift 2;;
--cores) CORES="$2"; shift 2;;
--cpu) CPU="$2"; shift 2;;
--memory) MEMORY="$2"; shift 2;;
--disk) DISK="$2"; shift 2;;
--bridge) BRIDGE="$2"; shift 2;;
@@ -64,7 +77,7 @@ fi
echo "creating $NAME as VM $VMID at $IP"
qm create "$VMID" \
--name "$NAME" --cores "$CORES" --memory "$MEMORY" \
--name "$NAME" --cores "$CORES" --memory "$MEMORY" --cpu "$CPU" \
--net0 "virtio,bridge=$BRIDGE" --ostype l26 --agent enabled=1
# Imported rather than installed: a cloud image plus cloud-init is the whole