Files
werewulf/docs/10-lab-installation.md
ilkerandClaude Opus 5.5 5b6e1ed3dd docs: section 02 installs hpcod from scratch; the lab build moves to 10
02 now walks an empty host through `hpcod setup`: what it needs, what the
TUI asks and does, the simulated Proxmox, reaching the portal, verify and
selftest, with timings measured on one run. The hand-built lab's script
order moves unchanged to 10-lab-installation.md. README, 01, 09 and the
handbook (contents, chapter 2 with two screenshots, new chapter 10) follow;
the PDF is regenerated, now 25 pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 19:08:32 +01:00

140 lines
5.5 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 10. Building the reference lab by hand
The reference lab of sections 01–09, built with the numbered scripts in
`scripts/`: one cluster, five VMs, the identity tier added after the cluster
works. This is where the design was proven. It is not how the HPC-on-Demand
control plane is installed; that is section [02](02-installation.md).
## 10.1 Prerequisites
**On the laptop:** a working `proxmox` entry in `~/.ssh/config` (via the
jumphost) and `~/.ssh/id_ed25519`.
**On Proxmox:** the public key staged where the provisioning scripts expect it:
```bash
scp ~/.ssh/id_ed25519.pub proxmox:/root/ilker_ed25519.pub
```
Resource footprint: **40 GB RAM**, 200 GB disk (`zfspool`) and 14 vCPU across
5 VMs. `hv01` has 503 GB RAM and 28 cores, so this fits comfortably.
## 10.2 Order of operations
Scripts are numbered. `0*`–`9*` build the cluster tier; `A*` build the identity
tier. **The identity tier is installed after the cluster is already working**,
so each layer can be verified independently.
```bash
cd ~/src/werewulf
./scripts/run-all.sh # 00 → 70, cluster tier, ends with an MPI test
```
Then the identity tier:
```bash
ssh proxmox 'bash -s' < scripts/A00-proxmox-ipa-vm.sh
scp scripts/A10-ipa-server.sh idm1:/tmp/ && ssh idm1 'sudo bash /tmp/A10-ipa-server.sh'
scp scripts/A20-ipa-nfs-home.sh idm1:/tmp/ && ssh idm1 'sudo bash /tmp/A20-ipa-nfs-home.sh'
# migrate hpcuser's home from head01 to idm1 (see 10.4)
PW=$(ssh idm1 'sudo grep IPA_ADMIN_PASSWORD /root/.ipa-credentials | cut -d= -f2')
scp scripts/A30-head01-ipa.sh head01:/tmp/ && ssh head01 "sudo bash /tmp/A30-head01-ipa.sh '$PW'"
scp scripts/A40-ipa-node-hosts.sh idm1:/tmp/ && ssh idm1 'sudo bash /tmp/A40-ipa-node-hosts.sh'
# transfer keytabs from idm1 to head01 (see 10.5)
scp scripts/A50-node-image-ipa.sh head01:/tmp/ && ssh head01 'sudo bash /tmp/A50-node-image-ipa.sh'
scp scripts/A60-node-overlays-ipa.sh head01:/tmp/ && ssh head01 'sudo bash /tmp/A60-node-overlays-ipa.sh'
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset
./scripts/A99-verify.sh
```
## 10.3 What each script does
### Cluster tier
| Script | Runs on | Purpose |
|---|---|---|
| `00-proxmox-provision.sh` | proxmox | `vmbr6` bridge, Rocky 9 image, VMs 300-304 |
| `10-head-base.sh` | head01 | hostname/hosts, packages, firewalld zones, IP forwarding, SELinux permissive |
| `20-warewulf-install.sh` | head01 | Warewulf 4.7.0 RPM (official el9 build) |
| `30-warewulf-configure.sh` | head01 | `warewulf.conf` → 10.10.10.0/24, `wwctl configure --all` |
| `40-node-image.sh` | head01 | Node image: kernel, wwinit dracut, OpenHPC compute, slurmd spool |
| `50-nodes-register.sh` | head01 | `default` profile + 4 nodes (MAC/IP mapping) |
| `60-slurm-openhpc.sh` | head01 | OpenHPC, slurmctld, munge, OpenMPI, `hpc` overlay, NFS |
| `70-mpi-test.sh` | head01 | Compile + `sbatch` + collect output |
### Identity tier
| Script | Runs on | Purpose |
|---|---|---|
| `A00-proxmox-ipa-vm.sh` | proxmox | idm1 VM (305), dual-homed |
| `A10-ipa-server.sh` | idm1 | FreeIPA server, integrated DNS, realm `HPC.LOCAL` |
| `A20-ipa-nfs-home.sh` | idm1 | `/export/home` export, cluster DNS records, users/groups |
| `A30-head01-ipa.sh` | head01 | Enrols head01 as an IPA client, switches `/home` to NFS |
| `A40-ipa-node-hosts.sh` | idm1 | Node host entries + **per-node keytabs** |
| `A50-node-image-ipa.sh` | head01 | Adds SSSD/Kerberos/NFS client to the node image |
| `A60-node-overlays-ipa.sh` | head01 | `ipa` + `ipa.wwnodeN` overlays, node DNS, `/home` fstab |
### Helpers
| Script | Runs on | Purpose |
|---|---|---|
| `99-cluster-ctl.sh` | proxmox | `start`/`stop`/`stopall`/`reset`/`status`/`console N` |
| `A85-slurm-accounting.sh` | head01 | slurmdbd + MariaDB, `sacct`/`sreport` |
| `A99-verify.sh` | laptop | End-to-end verification |
| `run-all.sh` | laptop | Builds the whole cluster tier |
## 10.4 Manual step: home migration
Before `A30` runs, `hpcuser`'s home must be moved from head01 to idm1 —
otherwise the existing MPI files vanish the moment `/home` becomes an NFS
mount. This is a one-time operation, so it was deliberately left out of the
scripts:
```bash
ssh head01 'sudo tar czf /tmp/h.tgz -C /home hpcuser'
ssh head01 'sudo cat /tmp/h.tgz' > /tmp/h.tgz
ssh idm1 'cat > /tmp/h.tgz' < /tmp/h.tgz
ssh idm1 'sudo bash -c "
. /root/.ipa-credentials
echo \$IPA_ADMIN_PASSWORD | kinit admin
U=\$(ipa user-show hpcuser --all --raw | awk \"/uidnumber:/ {print \\\$2}\")
G=\$(ipa user-show hpcuser --all --raw | awk \"/gidnumber:/ {print \\\$2}\")
tar xzf /tmp/h.tgz -C /tmp
cp -a /tmp/hpcuser/. /export/home/hpcuser/
chown -R \$U:\$G /export/home/hpcuser"'
```
Note the files are chowned to the **IPA uid/gid** (721800005), not the old
local uid on head01 (1001).
## 10.5 Manual step: keytab transfer
`A40` produces keytabs under `idm1:/root/node-keytabs/`. `A60` expects them at
`head01:/root/node-keytabs/`:
```bash
ssh idm1 'sudo tar cz -C /root/node-keytabs .' > /tmp/kt.tgz
ssh head01 'sudo mkdir -p /root/node-keytabs && sudo tar xz -C /root/node-keytabs' < /tmp/kt.tgz
```
The transfer goes through the laptop because the two servers have no direct SSH
trust. Keytabs are **secret material** — delete `/tmp/kt.tgz` afterwards.
## 10.6 Timings
Reference figures on `hv01` with cold package caches:
| Stage | Duration |
|---|---|
| VM creation + Rocky image download | ~3 min |
| `10`–`30` (head prep + Warewulf) | ~5 min |
| `40` (node image + OpenHPC) | ~12 min |
| `60` (OpenHPC head + OpenMPI) | ~10 min |
| `A10` (FreeIPA installation) | ~8 min |
| Node boot (4 nodes in parallel, 1.2 GB image) | ~5 min |
| **Total** | **~45 min** |