Files
werewulf/docs/10-lab-installation.md
ilkerandClaude Opus 5.5 5b6e1ed3dd docs: section 02 installs hpcod from scratch; the lab build moves to 10
02 now walks an empty host through `hpcod setup`: what it needs, what the
TUI asks and does, the simulated Proxmox, reaching the portal, verify and
selftest, with timings measured on one run. The hand-built lab's script
order moves unchanged to 10-lab-installation.md. README, 01, 09 and the
handbook (contents, chapter 2 with two screenshots, new chapter 10) follow;
the PDF is regenerated, now 25 pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 19:08:32 +01:00

5.5 KiB
Raw Permalink Blame History

10. Building the reference lab by hand

The reference lab of sections 01–09, built with the numbered scripts in scripts/: one cluster, five VMs, the identity tier added after the cluster works. This is where the design was proven. It is not how the HPC-on-Demand control plane is installed; that is section 02.

10.1 Prerequisites

On the laptop: a working proxmox entry in ~/.ssh/config (via the jumphost) and ~/.ssh/id_ed25519.

On Proxmox: the public key staged where the provisioning scripts expect it:

scp ~/.ssh/id_ed25519.pub proxmox:/root/ilker_ed25519.pub

Resource footprint: 40 GB RAM, 200 GB disk (zfspool) and 14 vCPU across 5 VMs. hv01 has 503 GB RAM and 28 cores, so this fits comfortably.

10.2 Order of operations

Scripts are numbered. 0*–9* build the cluster tier; A* build the identity tier. The identity tier is installed after the cluster is already working, so each layer can be verified independently.

cd ~/src/werewulf
./scripts/run-all.sh          # 00 → 70, cluster tier, ends with an MPI test

Then the identity tier:

ssh proxmox 'bash -s' < scripts/A00-proxmox-ipa-vm.sh
scp scripts/A10-ipa-server.sh   idm1:/tmp/ && ssh idm1 'sudo bash /tmp/A10-ipa-server.sh'
scp scripts/A20-ipa-nfs-home.sh idm1:/tmp/ && ssh idm1 'sudo bash /tmp/A20-ipa-nfs-home.sh'

# migrate hpcuser's home from head01 to idm1 (see 10.4)

PW=$(ssh idm1 'sudo grep IPA_ADMIN_PASSWORD /root/.ipa-credentials | cut -d= -f2')
scp scripts/A30-head01-ipa.sh head01:/tmp/ && ssh head01 "sudo bash /tmp/A30-head01-ipa.sh '$PW'"

scp scripts/A40-ipa-node-hosts.sh idm1:/tmp/ && ssh idm1 'sudo bash /tmp/A40-ipa-node-hosts.sh'
# transfer keytabs from idm1 to head01 (see 10.5)
scp scripts/A50-node-image-ipa.sh    head01:/tmp/ && ssh head01 'sudo bash /tmp/A50-node-image-ipa.sh'
scp scripts/A60-node-overlays-ipa.sh head01:/tmp/ && ssh head01 'sudo bash /tmp/A60-node-overlays-ipa.sh'

ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset
./scripts/A99-verify.sh

10.3 What each script does

Cluster tier

Script Runs on Purpose
00-proxmox-provision.sh proxmox vmbr6 bridge, Rocky 9 image, VMs 300-304
10-head-base.sh head01 hostname/hosts, packages, firewalld zones, IP forwarding, SELinux permissive
20-warewulf-install.sh head01 Warewulf 4.7.0 RPM (official el9 build)
30-warewulf-configure.sh head01 warewulf.conf → 10.10.10.0/24, wwctl configure --all
40-node-image.sh head01 Node image: kernel, wwinit dracut, OpenHPC compute, slurmd spool
50-nodes-register.sh head01 default profile + 4 nodes (MAC/IP mapping)
60-slurm-openhpc.sh head01 OpenHPC, slurmctld, munge, OpenMPI, hpc overlay, NFS
70-mpi-test.sh head01 Compile + sbatch + collect output

Identity tier

Script Runs on Purpose
A00-proxmox-ipa-vm.sh proxmox idm1 VM (305), dual-homed
A10-ipa-server.sh idm1 FreeIPA server, integrated DNS, realm HPC.LOCAL
A20-ipa-nfs-home.sh idm1 /export/home export, cluster DNS records, users/groups
A30-head01-ipa.sh head01 Enrols head01 as an IPA client, switches /home to NFS
A40-ipa-node-hosts.sh idm1 Node host entries + per-node keytabs
A50-node-image-ipa.sh head01 Adds SSSD/Kerberos/NFS client to the node image
A60-node-overlays-ipa.sh head01 ipa + ipa.wwnodeN overlays, node DNS, /home fstab

Helpers

Script Runs on Purpose
99-cluster-ctl.sh proxmox start/stop/stopall/reset/status/console N
A85-slurm-accounting.sh head01 slurmdbd + MariaDB, sacct/sreport
A99-verify.sh laptop End-to-end verification
run-all.sh laptop Builds the whole cluster tier

10.4 Manual step: home migration

Before A30 runs, hpcuser's home must be moved from head01 to idm1 — otherwise the existing MPI files vanish the moment /home becomes an NFS mount. This is a one-time operation, so it was deliberately left out of the scripts:

ssh head01 'sudo tar czf /tmp/h.tgz -C /home hpcuser'
ssh head01 'sudo cat /tmp/h.tgz' > /tmp/h.tgz
ssh idm1   'cat > /tmp/h.tgz' < /tmp/h.tgz
ssh idm1   'sudo bash -c "
  . /root/.ipa-credentials
  echo \$IPA_ADMIN_PASSWORD | kinit admin
  U=\$(ipa user-show hpcuser --all --raw | awk \"/uidnumber:/ {print \\\$2}\")
  G=\$(ipa user-show hpcuser --all --raw | awk \"/gidnumber:/ {print \\\$2}\")
  tar xzf /tmp/h.tgz -C /tmp
  cp -a /tmp/hpcuser/. /export/home/hpcuser/
  chown -R \$U:\$G /export/home/hpcuser"'

Note the files are chowned to the IPA uid/gid (721800005), not the old local uid on head01 (1001).

10.5 Manual step: keytab transfer

A40 produces keytabs under idm1:/root/node-keytabs/. A60 expects them at head01:/root/node-keytabs/:

ssh idm1 'sudo tar cz -C /root/node-keytabs .' > /tmp/kt.tgz
ssh head01 'sudo mkdir -p /root/node-keytabs && sudo tar xz -C /root/node-keytabs' < /tmp/kt.tgz

The transfer goes through the laptop because the two servers have no direct SSH trust. Keytabs are secret material — delete /tmp/kt.tgz afterwards.

10.6 Timings

Reference figures on hv01 with cold package caches:

Stage Duration
VM creation + Rocky image download ~3 min
10–30 (head prep + Warewulf) ~5 min
40 (node image + OpenHPC) ~12 min
60 (OpenHPC head + OpenMPI) ~10 min
A10 (FreeIPA installation) ~8 min
Node boot (4 nodes in parallel, 1.2 GB image) ~5 min
Total ~45 min