02 now walks an empty host through `hpcod setup`: what it needs, what the TUI asks and does, the simulated Proxmox, reaching the portal, verify and selftest, with timings measured on one run. The hand-built lab's script order moves unchanged to 10-lab-installation.md. README, 01, 09 and the handbook (contents, chapter 2 with two screenshots, new chapter 10) follow; the PDF is regenerated, now 25 pages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5.5 KiB
10. Building the reference lab by hand
The reference lab of sections 01–09, built with the numbered scripts in
scripts/: one cluster, five VMs, the identity tier added after the cluster
works. This is where the design was proven. It is not how the HPC-on-Demand
control plane is installed; that is section 02.
10.1 Prerequisites
On the laptop: a working proxmox entry in ~/.ssh/config (via the
jumphost) and ~/.ssh/id_ed25519.
On Proxmox: the public key staged where the provisioning scripts expect it:
scp ~/.ssh/id_ed25519.pub proxmox:/root/ilker_ed25519.pub
Resource footprint: 40 GB RAM, 200 GB disk (zfspool) and 14 vCPU across
5 VMs. hv01 has 503 GB RAM and 28 cores, so this fits comfortably.
10.2 Order of operations
Scripts are numbered. 0*–9* build the cluster tier; A* build the identity
tier. The identity tier is installed after the cluster is already working,
so each layer can be verified independently.
cd ~/src/werewulf
./scripts/run-all.sh # 00 → 70, cluster tier, ends with an MPI test
Then the identity tier:
ssh proxmox 'bash -s' < scripts/A00-proxmox-ipa-vm.sh
scp scripts/A10-ipa-server.sh idm1:/tmp/ && ssh idm1 'sudo bash /tmp/A10-ipa-server.sh'
scp scripts/A20-ipa-nfs-home.sh idm1:/tmp/ && ssh idm1 'sudo bash /tmp/A20-ipa-nfs-home.sh'
# migrate hpcuser's home from head01 to idm1 (see 10.4)
PW=$(ssh idm1 'sudo grep IPA_ADMIN_PASSWORD /root/.ipa-credentials | cut -d= -f2')
scp scripts/A30-head01-ipa.sh head01:/tmp/ && ssh head01 "sudo bash /tmp/A30-head01-ipa.sh '$PW'"
scp scripts/A40-ipa-node-hosts.sh idm1:/tmp/ && ssh idm1 'sudo bash /tmp/A40-ipa-node-hosts.sh'
# transfer keytabs from idm1 to head01 (see 10.5)
scp scripts/A50-node-image-ipa.sh head01:/tmp/ && ssh head01 'sudo bash /tmp/A50-node-image-ipa.sh'
scp scripts/A60-node-overlays-ipa.sh head01:/tmp/ && ssh head01 'sudo bash /tmp/A60-node-overlays-ipa.sh'
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset
./scripts/A99-verify.sh
10.3 What each script does
Cluster tier
| Script | Runs on | Purpose |
|---|---|---|
00-proxmox-provision.sh |
proxmox | vmbr6 bridge, Rocky 9 image, VMs 300-304 |
10-head-base.sh |
head01 | hostname/hosts, packages, firewalld zones, IP forwarding, SELinux permissive |
20-warewulf-install.sh |
head01 | Warewulf 4.7.0 RPM (official el9 build) |
30-warewulf-configure.sh |
head01 | warewulf.conf → 10.10.10.0/24, wwctl configure --all |
40-node-image.sh |
head01 | Node image: kernel, wwinit dracut, OpenHPC compute, slurmd spool |
50-nodes-register.sh |
head01 | default profile + 4 nodes (MAC/IP mapping) |
60-slurm-openhpc.sh |
head01 | OpenHPC, slurmctld, munge, OpenMPI, hpc overlay, NFS |
70-mpi-test.sh |
head01 | Compile + sbatch + collect output |
Identity tier
| Script | Runs on | Purpose |
|---|---|---|
A00-proxmox-ipa-vm.sh |
proxmox | idm1 VM (305), dual-homed |
A10-ipa-server.sh |
idm1 | FreeIPA server, integrated DNS, realm HPC.LOCAL |
A20-ipa-nfs-home.sh |
idm1 | /export/home export, cluster DNS records, users/groups |
A30-head01-ipa.sh |
head01 | Enrols head01 as an IPA client, switches /home to NFS |
A40-ipa-node-hosts.sh |
idm1 | Node host entries + per-node keytabs |
A50-node-image-ipa.sh |
head01 | Adds SSSD/Kerberos/NFS client to the node image |
A60-node-overlays-ipa.sh |
head01 | ipa + ipa.wwnodeN overlays, node DNS, /home fstab |
Helpers
| Script | Runs on | Purpose |
|---|---|---|
99-cluster-ctl.sh |
proxmox | start/stop/stopall/reset/status/console N |
A85-slurm-accounting.sh |
head01 | slurmdbd + MariaDB, sacct/sreport |
A99-verify.sh |
laptop | End-to-end verification |
run-all.sh |
laptop | Builds the whole cluster tier |
10.4 Manual step: home migration
Before A30 runs, hpcuser's home must be moved from head01 to idm1 —
otherwise the existing MPI files vanish the moment /home becomes an NFS
mount. This is a one-time operation, so it was deliberately left out of the
scripts:
ssh head01 'sudo tar czf /tmp/h.tgz -C /home hpcuser'
ssh head01 'sudo cat /tmp/h.tgz' > /tmp/h.tgz
ssh idm1 'cat > /tmp/h.tgz' < /tmp/h.tgz
ssh idm1 'sudo bash -c "
. /root/.ipa-credentials
echo \$IPA_ADMIN_PASSWORD | kinit admin
U=\$(ipa user-show hpcuser --all --raw | awk \"/uidnumber:/ {print \\\$2}\")
G=\$(ipa user-show hpcuser --all --raw | awk \"/gidnumber:/ {print \\\$2}\")
tar xzf /tmp/h.tgz -C /tmp
cp -a /tmp/hpcuser/. /export/home/hpcuser/
chown -R \$U:\$G /export/home/hpcuser"'
Note the files are chowned to the IPA uid/gid (721800005), not the old local uid on head01 (1001).
10.5 Manual step: keytab transfer
A40 produces keytabs under idm1:/root/node-keytabs/. A60 expects them at
head01:/root/node-keytabs/:
ssh idm1 'sudo tar cz -C /root/node-keytabs .' > /tmp/kt.tgz
ssh head01 'sudo mkdir -p /root/node-keytabs && sudo tar xz -C /root/node-keytabs' < /tmp/kt.tgz
The transfer goes through the laptop because the two servers have no direct SSH
trust. Keytabs are secret material — delete /tmp/kt.tgz afterwards.
10.6 Timings
Reference figures on hv01 with cold package caches:
| Stage | Duration |
|---|---|
| VM creation + Rocky image download | ~3 min |
10–30 (head prep + Warewulf) |
~5 min |
40 (node image + OpenHPC) |
~12 min |
60 (OpenHPC head + OpenMPI) |
~10 min |
A10 (FreeIPA installation) |
~8 min |
| Node boot (4 nodes in parallel, 1.2 GB image) | ~5 min |
| Total | ~45 min |