Files
werewulf/scripts/A30-head01-ipa.sh
T
ilker 29bd39c135 Warewulf HPC test cluster: provisioning, identity, docs and overlay UI
A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.

Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
  hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
  Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
  server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
  fabric so Kerberos and NFS never traverse head01's masquerade.

Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.

Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.

Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.

Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.

No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
2026-08-29 09:07:51 +02:00

104 lines
3.6 KiB
Bash
Executable File

#!/bin/bash
# Enrol head01 as an IPA client and mount /home from idm1 over NFS
# PREREQUISITE: /home/hpcuser must already be migrated to idm1:/export/home/hpcuser.
set -euo pipefail
DOMAIN=hpc.local
REALM=HPC.LOCAL
IPA_SRV=idm1.hpc.local
IPA_IP=10.10.10.5
ADMIN_PW="${1:?usage: A30-head01-ipa.sh <ipa-admin-password>}"
echo "==> hostname -> head01.${DOMAIN}"
hostnamectl set-hostname head01.${DOMAIN}
cat > /etc/hosts <<EOF
127.0.0.1 localhost localhost.localdomain
10.10.10.1 head01.${DOMAIN} head01
10.0.0.30 head01-mgmt.${DOMAIN}
${IPA_IP} ${IPA_SRV} idm1
10.10.10.11 wwnode1.${DOMAIN} wwnode1
10.10.10.12 wwnode2.${DOMAIN} wwnode2
10.10.10.13 wwnode3.${DOMAIN} wwnode3
10.10.10.14 wwnode4.${DOMAIN} wwnode4
::1 localhost localhost.localdomain
EOF
echo "==> DNS -> ${IPA_IP} (IPA forwards external queries)"
# cloud-init pins DNS in two places:
# 1) it writes a static /etc/resolv.conf
# 2) it sets "dns = none" for NetworkManager, so NM never touches resolv.conf
# Both must be relaxed, or the nmcli settings below have no effect whatsoever.
mkdir -p /etc/cloud/cloud.cfg.d
printf 'manage_resolv_conf: false\n' > /etc/cloud/cloud.cfg.d/99-disable-resolv.cfg
cat > /etc/NetworkManager/conf.d/99-zz-ipa-dns.conf <<'INI'
[main]
dns = default
INI
rm -f /etc/resolv.conf
E0=$(nmcli -g GENERAL.CONNECTION device show eth0)
E1=$(nmcli -g GENERAL.CONNECTION device show eth1)
nmcli con mod "$E1" ipv4.dns "${IPA_IP}" ipv4.dns-search "${DOMAIN}" ipv4.ignore-auto-dns yes
nmcli con mod "$E0" ipv4.dns "" ipv4.dns-search "" ipv4.ignore-auto-dns yes
systemctl restart NetworkManager
sleep 4
cat /etc/resolv.conf
dig +short ${IPA_SRV} | head -1
echo "==> packages"
dnf install -y -q ipa-client sssd sssd-tools oddjob-mkhomedir nfs-utils autofs
echo "==> IPA client enrolment"
if ! id hpcuser@${DOMAIN} >/dev/null 2>&1 && [ ! -f /etc/ipa/default.conf ]; then
ipa-client-install --unattended \
--domain="${DOMAIN}" --realm="${REALM}" --server="${IPA_SRV}" \
--principal=admin --password="${ADMIN_PW}" \
--mkhomedir --enable-dns-updates --no-ntp --force-join
fi
echo "==> NFSv4 idmap domain"
sed -i "s/^#\?Domain =.*/Domain = ${DOMAIN}/" /etc/idmapd.conf
grep -q "^Domain" /etc/idmapd.conf || echo "Domain = ${DOMAIN}" >> /etc/idmapd.conf
echo "==> removing local hpcuser (the IPA account takes over)"
if id hpcuser >/dev/null 2>&1 && [ "$(id -u hpcuser)" -lt 10000 ]; then
userdel hpcuser 2>/dev/null || true
rm -rf /home/hpcuser
fi
echo "==> move the local admin home out of /home (so the NFS mount doesn't hide it)"
# usermod -m refuses while the user is logged in (we are connected as ilker),
# so copy the directory by hand and edit /etc/passwd directly.
if [ -d /home/ilker ] && ! mountpoint -q /home; then
mkdir -p /var/home
[ -d /var/home/ilker ] || cp -a /home/ilker /var/home/ilker
sed -i 's#^\(ilker:[^:]*:[^:]*:[^:]*:[^:]*\):/home/ilker:#\1:/var/home/ilker:#' /etc/passwd
getent passwd ilker
fi
echo "==> head01 NO LONGER exports /home (idm1 serves it)"
python3 - <<'PY'
import re
p = "/etc/warewulf/warewulf.conf"
s = open(p).read()
# drop /home from the nfs export list; keep /opt (OpenHPC modules)
s = re.sub(r'\n\s*- path: /home\n\s*export options:[^\n]*', '', s)
open(p, "w").write(s)
PY
grep -A6 '^nfs:' /etc/warewulf/warewulf.conf
wwctl configure nfs
exportfs -ra
echo "==> /home -> ${IPA_SRV}:/export/home"
if ! grep -q "/export/home" /etc/fstab; then
echo "${IPA_SRV}:/export/home /home nfs4 defaults,_netdev,nofail 0 0" >> /etc/fstab
fi
mount -a
mountpoint -q /home && echo "/home mounted over NFS" || { echo "ERROR: /home failed to mount"; exit 1; }
ls -la /home
echo "==> verification"
id hpcuser
getent passwd testuser
echo "HEAD01 IPA OK"