A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.
Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
fabric so Kerberos and NFS never traverse head01's masquerade.
Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.
Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.
Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.
Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.
No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
166 lines
5.0 KiB
Markdown
166 lines
5.0 KiB
Markdown
# 7. Operations Runbook
|
|
|
|
## 7.1 Access
|
|
|
|
```bash
|
|
ssh head01 # head node 10.0.0.30, user ilker, sudo NOPASSWD
|
|
ssh idm1 # identity 10.0.0.31, user ilker, sudo NOPASSWD
|
|
ssh proxmox # hypervisor 10.0.0.10, root
|
|
```
|
|
|
|
Compute nodes sit on the isolated subnet and are reached **through** the head
|
|
node:
|
|
|
|
```bash
|
|
ssh head01
|
|
sudo ssh root@wwnode1 # or root@10.10.10.11
|
|
```
|
|
|
|
Root SSH from head01 to the nodes uses `/root/.ssh/cluster`, a dedicated key
|
|
pair distributed by the `ssh.authorized_keys` overlay.
|
|
|
|
Serial console (useful when a node will not finish booting):
|
|
|
|
```bash
|
|
ssh proxmox 'qm terminal 301' # wwnode1 = 301, wwnode2 = 302, …
|
|
# exit with Ctrl-O
|
|
```
|
|
|
|
## 7.2 Power management
|
|
|
|
```bash
|
|
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh status
|
|
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh start # idm1, head01, nodes
|
|
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset # reboot nodes only
|
|
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh stop # stop nodes only
|
|
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh stopall # everything
|
|
```
|
|
|
|
**Start order matters.** `idm1` must be up before nodes boot — they need DNS
|
|
from it during the overlay stage and will otherwise come up without working
|
|
identity. `head01` must be up before nodes for PXE. `99-cluster-ctl.sh start`
|
|
enforces this order.
|
|
|
|
`idm1` and `head01` have `onboot=1` and start with the hypervisor. Nodes have
|
|
`onboot=0` and are started deliberately.
|
|
|
|
## 7.3 Health check
|
|
|
|
```bash
|
|
./scripts/A99-verify.sh
|
|
```
|
|
|
|
Checks, in order: IPA services (expect 9 running), NFS exports, IPA user count,
|
|
head01 hostname/DNS/`/home`/services/`sinfo`, then per node `sssd`, `slurmd`,
|
|
`munge`, `/home` mount and a real `kinit -k` against the KDC — finally
|
|
submitting an MPI job and printing its output.
|
|
|
|
`krb5=OK` means the node actually obtained a ticket with its host keytab, not
|
|
merely that the file exists.
|
|
|
|
## 7.4 Common tasks
|
|
|
|
### Add a user
|
|
|
|
```bash
|
|
ssh idm1
|
|
sudo bash -c '. /root/.ipa-credentials
|
|
echo $IPA_ADMIN_PASSWORD | kinit admin
|
|
ipa user-add ayse --first=Ayse --last=Yilmaz \
|
|
--homedir=/home/ayse --shell=/bin/bash --password
|
|
ipa group-add-member hpcusers --users=ayse'
|
|
```
|
|
|
|
Home directory is created on first login. No node reboot needed — SSSD picks up
|
|
new users immediately.
|
|
|
|
### Change something in the node image
|
|
|
|
```bash
|
|
ssh head01
|
|
sudo wwctl image exec rocky-9 -- /bin/bash
|
|
dnf install -y <package>
|
|
exit
|
|
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset
|
|
```
|
|
|
|
Costs a full reboot cycle (~5 min) since nodes re-download the image.
|
|
|
|
### Change configuration without touching the image
|
|
|
|
```bash
|
|
ssh head01
|
|
sudo wwctl overlay import -p -o hpc /etc/slurm/slurm.conf /etc/slurm/slurm.conf
|
|
sudo wwctl overlay chmod hpc /etc 0755 # always re-check after import -p
|
|
sudo wwctl overlay build
|
|
```
|
|
|
|
- **System overlay** change → node reboot required.
|
|
- **Runtime overlay** change → picked up within 60 s automatically.
|
|
|
|
### Add a compute node
|
|
|
|
1. Create the VM on Proxmox (copy the `00-proxmox-provision.sh` node block,
|
|
next free VMID and MAC `BC:24:11:E0:0A:1<n>`).
|
|
2. Register it:
|
|
```bash
|
|
sudo wwctl node add wwnode5 --ipaddr 10.10.10.15 \
|
|
--hwaddr BC:24:11:E0:0A:15 --netdev eth0 --discoverable=false
|
|
```
|
|
3. Create its host entry and keytab on `idm1` (`A40` pattern), copy the keytab
|
|
to head01, build an `ipa.wwnode5` overlay, assign it:
|
|
```bash
|
|
sudo wwctl node set --yes wwnode5 -O ipa.wwnode5
|
|
```
|
|
4. Add DNS A and PTR records on `idm1`.
|
|
5. Extend `slurm.conf` to `NodeName=wwnode[1-5]`, re-import into the `hpc`
|
|
overlay, `scontrol reconfigure`.
|
|
6. `wwctl overlay build`, then boot the node.
|
|
|
|
### Restart services
|
|
|
|
```bash
|
|
# head
|
|
sudo systemctl restart warewulfd dhcpd slurmctld
|
|
|
|
# identity
|
|
ssh idm1 'sudo ipactl restart'
|
|
|
|
# a node
|
|
ssh head01 'sudo ssh root@wwnode1 systemctl restart sssd'
|
|
```
|
|
|
|
## 7.5 Logs
|
|
|
|
| What | Where |
|
|
|---|---|
|
|
| Warewulf provisioning | `journalctl -u warewulfd` on head01 |
|
|
| DHCP / PXE | `journalctl -u dhcpd` on head01 |
|
|
| Slurm controller | `/var/log/slurmctld.log` on head01 |
|
|
| Slurm node daemon | `/var/log/slurmd.log` on the node |
|
|
| FreeIPA (all) | `idm1:/var/log/` — `dirsrv/`, `krb5kdc.log`, `httpd/` |
|
|
| SSSD | `journalctl -u sssd` on the client |
|
|
| Node boot | serial console (`qm terminal <vmid>`) |
|
|
|
|
For a node that will not boot, capture the serial output to a file:
|
|
|
|
```bash
|
|
ssh proxmox 'systemd-run --unit=wwnode1-serial --collect \
|
|
socat -u UNIX-CONNECT:/var/run/qemu-server/301.serial0 CREATE:/tmp/wwnode1.log'
|
|
ssh proxmox 'qm reset 301; tail -f /tmp/wwnode1.log'
|
|
```
|
|
|
|
This is how the original 4 GB RAM failure was diagnosed — the node was dying
|
|
silently with nothing in any server-side log.
|
|
|
|
## 7.6 Shutting the lab down cleanly
|
|
|
|
```bash
|
|
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh stop # nodes first
|
|
ssh head01 'sudo umount /home' # release NFS
|
|
ssh proxmox 'qm stop 300; qm stop 305'
|
|
```
|
|
|
|
Stopping `idm1` before unmounting `/home` on head01 leaves a hung NFS mount and
|
|
makes the head node's shutdown hang.
|