02 now walks an empty host through `hpcod setup`: what it needs, what the TUI asks and does, the simulated Proxmox, reaching the portal, verify and selftest, with timings measured on one run. The hand-built lab's script order moves unchanged to 10-lab-installation.md. README, 01, 09 and the handbook (contents, chapter 2 with two screenshots, new chapter 10) follow; the PDF is regenerated, now 25 pages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
140 lines
5.5 KiB
Markdown
140 lines
5.5 KiB
Markdown
# 10. Building the reference lab by hand
|
||
|
||
The reference lab of sections 01–09, built with the numbered scripts in
|
||
`scripts/`: one cluster, five VMs, the identity tier added after the cluster
|
||
works. This is where the design was proven. It is not how the HPC-on-Demand
|
||
control plane is installed; that is section [02](02-installation.md).
|
||
|
||
## 10.1 Prerequisites
|
||
|
||
**On the laptop:** a working `proxmox` entry in `~/.ssh/config` (via the
|
||
jumphost) and `~/.ssh/id_ed25519`.
|
||
|
||
**On Proxmox:** the public key staged where the provisioning scripts expect it:
|
||
|
||
```bash
|
||
scp ~/.ssh/id_ed25519.pub proxmox:/root/ilker_ed25519.pub
|
||
```
|
||
|
||
Resource footprint: **40 GB RAM**, 200 GB disk (`zfspool`) and 14 vCPU across
|
||
5 VMs. `hv01` has 503 GB RAM and 28 cores, so this fits comfortably.
|
||
|
||
## 10.2 Order of operations
|
||
|
||
Scripts are numbered. `0*`–`9*` build the cluster tier; `A*` build the identity
|
||
tier. **The identity tier is installed after the cluster is already working**,
|
||
so each layer can be verified independently.
|
||
|
||
```bash
|
||
cd ~/src/werewulf
|
||
./scripts/run-all.sh # 00 → 70, cluster tier, ends with an MPI test
|
||
```
|
||
|
||
Then the identity tier:
|
||
|
||
```bash
|
||
ssh proxmox 'bash -s' < scripts/A00-proxmox-ipa-vm.sh
|
||
scp scripts/A10-ipa-server.sh idm1:/tmp/ && ssh idm1 'sudo bash /tmp/A10-ipa-server.sh'
|
||
scp scripts/A20-ipa-nfs-home.sh idm1:/tmp/ && ssh idm1 'sudo bash /tmp/A20-ipa-nfs-home.sh'
|
||
|
||
# migrate hpcuser's home from head01 to idm1 (see 10.4)
|
||
|
||
PW=$(ssh idm1 'sudo grep IPA_ADMIN_PASSWORD /root/.ipa-credentials | cut -d= -f2')
|
||
scp scripts/A30-head01-ipa.sh head01:/tmp/ && ssh head01 "sudo bash /tmp/A30-head01-ipa.sh '$PW'"
|
||
|
||
scp scripts/A40-ipa-node-hosts.sh idm1:/tmp/ && ssh idm1 'sudo bash /tmp/A40-ipa-node-hosts.sh'
|
||
# transfer keytabs from idm1 to head01 (see 10.5)
|
||
scp scripts/A50-node-image-ipa.sh head01:/tmp/ && ssh head01 'sudo bash /tmp/A50-node-image-ipa.sh'
|
||
scp scripts/A60-node-overlays-ipa.sh head01:/tmp/ && ssh head01 'sudo bash /tmp/A60-node-overlays-ipa.sh'
|
||
|
||
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset
|
||
./scripts/A99-verify.sh
|
||
```
|
||
|
||
## 10.3 What each script does
|
||
|
||
### Cluster tier
|
||
|
||
| Script | Runs on | Purpose |
|
||
|---|---|---|
|
||
| `00-proxmox-provision.sh` | proxmox | `vmbr6` bridge, Rocky 9 image, VMs 300-304 |
|
||
| `10-head-base.sh` | head01 | hostname/hosts, packages, firewalld zones, IP forwarding, SELinux permissive |
|
||
| `20-warewulf-install.sh` | head01 | Warewulf 4.7.0 RPM (official el9 build) |
|
||
| `30-warewulf-configure.sh` | head01 | `warewulf.conf` → 10.10.10.0/24, `wwctl configure --all` |
|
||
| `40-node-image.sh` | head01 | Node image: kernel, wwinit dracut, OpenHPC compute, slurmd spool |
|
||
| `50-nodes-register.sh` | head01 | `default` profile + 4 nodes (MAC/IP mapping) |
|
||
| `60-slurm-openhpc.sh` | head01 | OpenHPC, slurmctld, munge, OpenMPI, `hpc` overlay, NFS |
|
||
| `70-mpi-test.sh` | head01 | Compile + `sbatch` + collect output |
|
||
|
||
### Identity tier
|
||
|
||
| Script | Runs on | Purpose |
|
||
|---|---|---|
|
||
| `A00-proxmox-ipa-vm.sh` | proxmox | idm1 VM (305), dual-homed |
|
||
| `A10-ipa-server.sh` | idm1 | FreeIPA server, integrated DNS, realm `HPC.LOCAL` |
|
||
| `A20-ipa-nfs-home.sh` | idm1 | `/export/home` export, cluster DNS records, users/groups |
|
||
| `A30-head01-ipa.sh` | head01 | Enrols head01 as an IPA client, switches `/home` to NFS |
|
||
| `A40-ipa-node-hosts.sh` | idm1 | Node host entries + **per-node keytabs** |
|
||
| `A50-node-image-ipa.sh` | head01 | Adds SSSD/Kerberos/NFS client to the node image |
|
||
| `A60-node-overlays-ipa.sh` | head01 | `ipa` + `ipa.wwnodeN` overlays, node DNS, `/home` fstab |
|
||
|
||
### Helpers
|
||
|
||
| Script | Runs on | Purpose |
|
||
|---|---|---|
|
||
| `99-cluster-ctl.sh` | proxmox | `start`/`stop`/`stopall`/`reset`/`status`/`console N` |
|
||
| `A85-slurm-accounting.sh` | head01 | slurmdbd + MariaDB, `sacct`/`sreport` |
|
||
| `A99-verify.sh` | laptop | End-to-end verification |
|
||
| `run-all.sh` | laptop | Builds the whole cluster tier |
|
||
|
||
## 10.4 Manual step: home migration
|
||
|
||
Before `A30` runs, `hpcuser`'s home must be moved from head01 to idm1 —
|
||
otherwise the existing MPI files vanish the moment `/home` becomes an NFS
|
||
mount. This is a one-time operation, so it was deliberately left out of the
|
||
scripts:
|
||
|
||
```bash
|
||
ssh head01 'sudo tar czf /tmp/h.tgz -C /home hpcuser'
|
||
ssh head01 'sudo cat /tmp/h.tgz' > /tmp/h.tgz
|
||
ssh idm1 'cat > /tmp/h.tgz' < /tmp/h.tgz
|
||
ssh idm1 'sudo bash -c "
|
||
. /root/.ipa-credentials
|
||
echo \$IPA_ADMIN_PASSWORD | kinit admin
|
||
U=\$(ipa user-show hpcuser --all --raw | awk \"/uidnumber:/ {print \\\$2}\")
|
||
G=\$(ipa user-show hpcuser --all --raw | awk \"/gidnumber:/ {print \\\$2}\")
|
||
tar xzf /tmp/h.tgz -C /tmp
|
||
cp -a /tmp/hpcuser/. /export/home/hpcuser/
|
||
chown -R \$U:\$G /export/home/hpcuser"'
|
||
```
|
||
|
||
Note the files are chowned to the **IPA uid/gid** (721800005), not the old
|
||
local uid on head01 (1001).
|
||
|
||
## 10.5 Manual step: keytab transfer
|
||
|
||
`A40` produces keytabs under `idm1:/root/node-keytabs/`. `A60` expects them at
|
||
`head01:/root/node-keytabs/`:
|
||
|
||
```bash
|
||
ssh idm1 'sudo tar cz -C /root/node-keytabs .' > /tmp/kt.tgz
|
||
ssh head01 'sudo mkdir -p /root/node-keytabs && sudo tar xz -C /root/node-keytabs' < /tmp/kt.tgz
|
||
```
|
||
|
||
The transfer goes through the laptop because the two servers have no direct SSH
|
||
trust. Keytabs are **secret material** — delete `/tmp/kt.tgz` afterwards.
|
||
|
||
## 10.6 Timings
|
||
|
||
Reference figures on `hv01` with cold package caches:
|
||
|
||
| Stage | Duration |
|
||
|---|---|
|
||
| VM creation + Rocky image download | ~3 min |
|
||
| `10`–`30` (head prep + Warewulf) | ~5 min |
|
||
| `40` (node image + OpenHPC) | ~12 min |
|
||
| `60` (OpenHPC head + OpenMPI) | ~10 min |
|
||
| `A10` (FreeIPA installation) | ~8 min |
|
||
| Node boot (4 nodes in parallel, 1.2 GB image) | ~5 min |
|
||
| **Total** | **~45 min** |
|