Ilker ManapandClaude Opus 5
634f87ba15
Proxmox defaults a VM to kvm64, which hides AVX, AVX2, FMA and even SSE4.2 from the guest. llama.cpp then loads libggml-cpu-x64.so — its scalar baseline — instead of the backend built for the host's instruction set, and an inference node is not slightly slower but unusable. Measured on node05, an 8-core VM on a Xeon E5-2680 v4, same weights, same thread count, the only change being --cpu host: Qwen2.5-Coder kvm64 host 1.5B q4_K_M 5.80 39.09 tok/s 3B q4_K_M 2.90 20.51 7B q4_K_M 1.33 9.88 Prompt processing went from 7.6 to 143.8, a factor of nineteen. This cost an afternoon because nothing looks broken. The node registers, the engine starts, the replies are correct; every number is just small, and the conclusion that suggests itself is that CPU inference is hopeless rather than that the VM cannot multiply efficiently. A 14B model at 1.7 tokens a second was nearly written off on that basis. The trade is live migration, which a VM pinned to the host CPU cannot do. For a node whose job is arithmetic that is the right way round, and --cpu overrides it where it is not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
97 lines
3.7 KiB
Bash
Executable File
97 lines
3.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# Create a VM on Proxmox for a new llmgw node.
|
|
#
|
|
# Run on the Proxmox host. It does the half of onboarding that is about
|
|
# hardware; onboard-node.sh does the half that is about identity, and the two
|
|
# are separate because they are done by different people with different access.
|
|
#
|
|
# Idempotent only in the sense that it refuses to touch an existing VMID. A node
|
|
# that already exists is not something a provisioning script should have an
|
|
# opinion about.
|
|
set -euo pipefail
|
|
|
|
usage() {
|
|
cat >&2 <<USAGE
|
|
usage: $0 --vmid N --name NAME --ip ADDR [options]
|
|
|
|
--vmid N Proxmox VM id; must be free
|
|
--name NAME VM name; use the node id, so the machine, the hostname and
|
|
the certificate common name are the same string
|
|
--ip ADDR address with prefix, e.g. 10.0.0.144/24
|
|
--gw ADDR gateway (default 10.0.0.1)
|
|
--cores N (default 4)
|
|
--cpu TYPE QEMU CPU model (default host)
|
|
--memory MB (default 8192)
|
|
--disk GB (default 20)
|
|
--bridge NAME (default vmbr0)
|
|
--storage NAME (default local-lvm)
|
|
--image PATH cloud image (default Ubuntu 24.04 in the ISO store)
|
|
--user NAME cloud-init user (default ilker)
|
|
--sshkeys PATH authorized_keys to install (default /root/.ssh/authorized_keys)
|
|
USAGE
|
|
exit 2
|
|
}
|
|
|
|
VMID= NAME= IP= GW=10.0.0.1 CORES=4 MEMORY=8192 DISK=20
|
|
# host, not Proxmox's kvm64 default. kvm64 hides AVX, AVX2, FMA and even
|
|
# SSE4.2 from the guest, and an inference node without vector instructions is
|
|
# not slightly slower, it is unusable: llama.cpp falls back to its scalar
|
|
# backend and a 1.5B model dropped from 39 tokens a second to 5.8, with prompt
|
|
# processing 19x worse. It cost an afternoon to find because nothing is broken
|
|
# — every number is simply small.
|
|
#
|
|
# The trade is live migration: a VM pinned to the host CPU cannot move to a
|
|
# machine with a different one. For a node whose job is arithmetic, that is
|
|
# the right way round.
|
|
CPU=host
|
|
BRIDGE=vmbr0 STORAGE=local-lvm USER=ilker
|
|
IMAGE=/var/lib/vz/template/iso/noble-server-cloudimg-amd64.img
|
|
SSHKEYS=/root/.ssh/authorized_keys
|
|
|
|
while [ $# -gt 0 ]; do
|
|
case "$1" in
|
|
--vmid) VMID="$2"; shift 2;;
|
|
--name) NAME="$2"; shift 2;;
|
|
--ip) IP="$2"; shift 2;;
|
|
--gw) GW="$2"; shift 2;;
|
|
--cores) CORES="$2"; shift 2;;
|
|
--cpu) CPU="$2"; shift 2;;
|
|
--memory) MEMORY="$2"; shift 2;;
|
|
--disk) DISK="$2"; shift 2;;
|
|
--bridge) BRIDGE="$2"; shift 2;;
|
|
--storage) STORAGE="$2"; shift 2;;
|
|
--image) IMAGE="$2"; shift 2;;
|
|
--user) USER="$2"; shift 2;;
|
|
--sshkeys) SSHKEYS="$2"; shift 2;;
|
|
*) usage;;
|
|
esac
|
|
done
|
|
[ -n "$VMID" ] && [ -n "$NAME" ] && [ -n "$IP" ] || usage
|
|
|
|
if qm status "$VMID" >/dev/null 2>&1; then
|
|
echo "VM $VMID already exists. Pick a free id; this script will not adopt one." >&2
|
|
exit 1
|
|
fi
|
|
[ -f "$IMAGE" ] || { echo "no cloud image at $IMAGE" >&2; exit 1; }
|
|
|
|
echo "creating $NAME as VM $VMID at $IP"
|
|
qm create "$VMID" \
|
|
--name "$NAME" --cores "$CORES" --memory "$MEMORY" --cpu "$CPU" \
|
|
--net0 "virtio,bridge=$BRIDGE" --ostype l26 --agent enabled=1
|
|
|
|
# Imported rather than installed: a cloud image plus cloud-init is the whole
|
|
# reason a node takes a minute to build instead of an afternoon.
|
|
qm importdisk "$VMID" "$IMAGE" "$STORAGE" >/dev/null
|
|
qm set "$VMID" --scsihw virtio-scsi-pci --scsi0 "$STORAGE:vm-$VMID-disk-0,discard=on,ssd=1" >/dev/null
|
|
qm disk resize "$VMID" scsi0 "${DISK}G" >/dev/null
|
|
qm set "$VMID" --ide2 "$STORAGE:cloudinit" --boot order=scsi0 >/dev/null
|
|
qm set "$VMID" \
|
|
--ipconfig0 "ip=$IP,gw=$GW" \
|
|
--nameserver "$GW 8.8.8.8" \
|
|
--ciuser "$USER" \
|
|
--sshkeys "$SSHKEYS" >/dev/null
|
|
|
|
qm start "$VMID"
|
|
echo "started. cloud-init takes a minute; then run onboard-node.sh from the CA host."
|