Files
werewulf/scripts/00-proxmox-provision.sh
T
ilker 29bd39c135 Warewulf HPC test cluster: provisioning, identity, docs and overlay UI
A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.

Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
  hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
  Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
  server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
  fabric so Kerberos and NFS never traverse head01's masquerade.

Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.

Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.

Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.

Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.

No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
2026-08-29 09:07:51 +02:00

70 lines
3.0 KiB
Bash
Executable File

#!/bin/bash
# Builds the Warewulf test cluster infrastructure on Proxmox (hv01).
# Run as root on the proxmox host: ssh proxmox 'bash -s' < 00-proxmox-provision.sh
set -euo pipefail
BRIDGE=vmbr6
HEAD_VMID=300
NODE_VMIDS="301 302 303 304"
STORAGE=zfspool
IMG_URL=https://dl.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2
IMG=/var/lib/vz/template/iso/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2
PUBKEY=/root/ilker_ed25519.pub # must already be staged here
# --- 1) Isolated cluster bridge ----------------------------------------------
# NO IP: pure L2. head01 (VM300) acts as DHCP/TFTP/DNS/gateway for the subnet.
if ! grep -q "^auto ${BRIDGE}$" /etc/network/interfaces; then
cat >> /etc/network/interfaces <<EOF
auto ${BRIDGE}
iface ${BRIDGE} inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
#Warewulf HPC cluster subnet 10.10.10.0/24 - L2 only. Head node (VM${HEAD_VMID}) = DHCP/TFTP/DNS/gateway.
EOF
ifup ${BRIDGE}
fi
ip -br a show ${BRIDGE}
# --- 2) Rocky 9 cloud image --------------------------------------------------
[ -f "${IMG}" ] || wget -q "${IMG_URL}" -O "${IMG}"
# --- 3) Head node (VM300) ----------------------------------------------------
# net0 = vmbr0 management (10.0.0.30), net1 = vmbr6 cluster (10.10.10.1)
if ! qm config ${HEAD_VMID} >/dev/null 2>&1; then
qm create ${HEAD_VMID} --name head01 --memory 8192 --cores 4 --sockets 1 --cpu host \
--net0 virtio=BC:24:11:E0:00:FF,bridge=vmbr0 \
--net1 virtio=BC:24:11:E0:0A:01,bridge=${BRIDGE} \
--scsihw virtio-scsi-single --ostype l26 --agent 1 --onboot 1 \
--serial0 socket --tags warewulf,hpc \
--description "Warewulf 4 head node (Rocky Linux 9). net0=vmbr0 mgmt 10.0.0.30, net1=${BRIDGE} cluster 10.10.10.1 (DHCP/TFTP/DNS/NFS/slurmctld/NAT gw)."
qm importdisk ${HEAD_VMID} "${IMG}" ${STORAGE}
qm set ${HEAD_VMID} --scsi0 ${STORAGE}:vm-${HEAD_VMID}-disk-0,discard=on,iothread=1,ssd=1
qm resize ${HEAD_VMID} scsi0 100G
qm set ${HEAD_VMID} --ide2 ${STORAGE}:cloudinit --boot order=scsi0
qm set ${HEAD_VMID} --ciuser ilker --sshkeys ${PUBKEY} \
--ipconfig0 ip=10.0.0.30/24,gw=10.0.0.1 \
--ipconfig1 ip=10.10.10.1/24 \
--nameserver "1.1.1.1 8.8.8.8" --ciupgrade 0
fi
# --- 4) Compute nodes (VM301-304) -------------------------------------------
# Diskless, PXE only. 8G RAM: the rootfs (~1.8G) is unpacked into RAM as an
# initramfs while the compressed image (~1.2G) is still held there. 4G IS NOT ENOUGH.
i=1
for vmid in ${NODE_VMIDS}; do
if ! qm config ${vmid} >/dev/null 2>&1; then
qm create ${vmid} --name wwnode${i} --memory 8192 --cores 2 --sockets 1 --cpu host \
--net0 virtio=BC:24:11:E0:0A:1${i},bridge=${BRIDGE} \
--scsihw virtio-scsi-single --ostype l26 --onboot 0 --balloon 0 \
--serial0 socket --vga serial0 --boot order=net0 \
--tags warewulf,hpc,compute \
--description "Warewulf stateless compute node ${i}. Diskless - PXE-provisioned by head01 over ${BRIDGE}."
fi
i=$((i+1))
done
qm list | grep -E "head01|wwnode"
echo "PROXMOX PROVISION OK"