Provision nodes with the host CPU model

Proxmox defaults a VM to kvm64, which hides AVX, AVX2, FMA and even SSE4.2
from the guest. llama.cpp then loads libggml-cpu-x64.so — its scalar
baseline — instead of the backend built for the host's instruction set, and
an inference node is not slightly slower but unusable.

Measured on node05, an 8-core VM on a Xeon E5-2680 v4, same weights, same
thread count, the only change being --cpu host:

  Qwen2.5-Coder     kvm64   host
  1.5B q4_K_M        5.80  39.09  tok/s
  3B   q4_K_M        2.90  20.51
  7B   q4_K_M        1.33   9.88

Prompt processing went from 7.6 to 143.8, a factor of nineteen.

This cost an afternoon because nothing looks broken. The node registers, the
engine starts, the replies are correct; every number is just small, and the
conclusion that suggests itself is that CPU inference is hopeless rather
than that the VM cannot multiply efficiently. A 14B model at 1.7 tokens a
second was nearly written off on that basis.

The trade is live migration, which a VM pinned to the host CPU cannot do.
For a node whose job is arithmetic that is the right way round, and --cpu
overrides it where it is not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-16 20:42:48 +02:00
1 parent 8a894af1ca
commit 634f87ba15
3 files changed
+45 -3

No files matched your search

+5 -2
View File
@@ -23,6 +23,7 @@ usage: add-node.sh --name NODE --ip ADDR/CIDR --server URL [options]
--vmid N Proxmox VMID (default: the last octet of --ip)
--cores N (default 4)
--cpu TYPE QEMU CPU model (default host; see provision-node.sh)
--memory MB (default 8192)
--disk GB (default 20)
--fabric NAME InfiniBand fabric; repeatable as NAME:hca1,hca2.
@@ -39,7 +40,7 @@ USAGE
exit 2
}
NAME= IP= SERVER= VMID= CORES=4 MEMORY=8192 DISK=20
NAME= IP= SERVER= VMID= CORES=4 MEMORY=8192 DISK=20 CPU=
BINARY= PKI=/etc/llmgw/pki SSH_USER=ilker
PROVISION=1 ONBOARD=1
set -- "$@"
@@ -51,6 +52,7 @@ while [ $# -gt 0 ]; do
--server) SERVER="$2"; shift 2;;
--vmid) VMID="$2"; shift 2;;
--cores) CORES="$2"; shift 2;;
--cpu) CPU="$2"; shift 2;;
--memory) MEMORY="$2"; shift 2;;
--disk) DISK="$2"; shift 2;;
--binary) BINARY="$2"; shift 2;;
@@ -82,8 +84,9 @@ ONBOARD_SH="$(here)/onboard-node.sh"
if [ "$PROVISION" = 1 ]; then
[ -x "$PROVISION_SH" ] || { echo "cannot find provision-node.sh" >&2; exit 1; }
echo "== 1/2 building $NAME as VM $VMID"
# shellcheck disable=SC2086
"$PROVISION_SH" --vmid "$VMID" --name "$NAME" --ip "$IP" \
--cores "$CORES" --memory "$MEMORY" --disk "$DISK"
--cores "$CORES" --memory "$MEMORY" --disk "$DISK" ${CPU:+--cpu "$CPU"}
fi
if [ "$ONBOARD" = 0 ]; then
+14 -1
View File
@@ -21,6 +21,7 @@ usage: $0 --vmid N --name NAME --ip ADDR [options]
--ip ADDR address with prefix, e.g. 10.0.0.144/24
--gw ADDR gateway (default 10.0.0.1)
--cores N (default 4)
--cpu TYPE QEMU CPU model (default host)
--memory MB (default 8192)
--disk GB (default 20)
--bridge NAME (default vmbr0)
@@ -33,6 +34,17 @@ USAGE
}
VMID= NAME= IP= GW=10.0.0.1 CORES=4 MEMORY=8192 DISK=20
# host, not Proxmox's kvm64 default. kvm64 hides AVX, AVX2, FMA and even
# SSE4.2 from the guest, and an inference node without vector instructions is
# not slightly slower, it is unusable: llama.cpp falls back to its scalar
# backend and a 1.5B model dropped from 39 tokens a second to 5.8, with prompt
# processing 19x worse. It cost an afternoon to find because nothing is broken
# — every number is simply small.
#
# The trade is live migration: a VM pinned to the host CPU cannot move to a
# machine with a different one. For a node whose job is arithmetic, that is
# the right way round.
CPU=host
BRIDGE=vmbr0 STORAGE=local-lvm USER=ilker
IMAGE=/var/lib/vz/template/iso/noble-server-cloudimg-amd64.img
SSHKEYS=/root/.ssh/authorized_keys
@@ -44,6 +56,7 @@ while [ $# -gt 0 ]; do
--ip) IP="$2"; shift 2;;
--gw) GW="$2"; shift 2;;
--cores) CORES="$2"; shift 2;;
--cpu) CPU="$2"; shift 2;;
--memory) MEMORY="$2"; shift 2;;
--disk) DISK="$2"; shift 2;;
--bridge) BRIDGE="$2"; shift 2;;
@@ -64,7 +77,7 @@ fi
echo "creating $NAME as VM $VMID at $IP"
qm create "$VMID" \
--name "$NAME" --cores "$CORES" --memory "$MEMORY" \
--name "$NAME" --cores "$CORES" --memory "$MEMORY" --cpu "$CPU" \
--net0 "virtio,bridge=$BRIDGE" --ostype l26 --agent enabled=1
# Imported rather than installed: a cloud image plus cloud-init is the whole
+26
View File
@@ -90,6 +90,32 @@ it accepted a moment earlier.
It also checks that a configured engine actually runs, which is cheap here and
expensive later — see below.
## The CPU model matters, and its default is wrong
Nodes are provisioned with `--cpu host`. Proxmox's default, `kvm64`, hides
AVX, AVX2, FMA and even SSE4.2 from the guest, and llama.cpp then loads its
scalar backend instead of the one built for the host's instruction set.
Measured on node05, an 8-core VM on a Xeon E5-2680 v4, same weights, same
threads, the only change being the CPU model:
| Qwen2.5-Coder | kvm64 tok/s | host tok/s |
| -------------- | ----------: | ---------: |
| 1.5B q4_K_M | 5.80 | 39.09 |
| 3B q4_K_M | 2.90 | 20.51 |
| 7B q4_K_M | 1.33 | 9.88 |
Prompt processing is worse still: 7.6 tokens a second against 143.8, a factor
of nineteen.
It is worth calling out because nothing looks broken. The node registers, the
engine starts, replies are correct — every number is simply small, and the
obvious conclusion is that CPU inference is hopeless rather than that the VM
cannot multiply efficiently.
The trade is live migration: a VM pinned to the host CPU cannot move to a
machine with a different one. For a node whose job is arithmetic that is the
right way round, and `--cpu` overrides it where it is not.
## What onboarding does not do
It gives a machine an identity and an agent. It does **not** install inference