Files
ilker 29bd39c135 Warewulf HPC test cluster: provisioning, identity, docs and overlay UI
A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.

Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
  hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
  Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
  server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
  fabric so Kerberos and NFS never traverse head01's masquerade.

Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.

Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.

Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.

Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.

No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
2026-08-29 09:07:51 +02:00

6.8 KiB
Raw Permalink Blame History

4. Identity — FreeIPA and SSSD

4.1 FreeIPA is not Active Directory

The original request was for Active Directory. What was built is FreeIPA, chosen deliberately after weighing the options:

Option Verdict
Samba 4 AD DC on Rocky 9 Red Hat does not support Samba in AD DC mode. Only an experimental MIT-KDC build exists. Not viable.
Samba 4 AD DC on Debian Fully supported AD DC (Heimdal KDC), genuine AD — but a second distro in an otherwise all-Rocky lab.
FreeIPA on Rocky 9 Chosen. The RHEL-native identity server: LDAP + Kerberos + CA + DNS.

For Linux clients FreeIPA is functionally equivalent to AD: Kerberos realm, LDAP directory, POSIX attributes, host principals, centralised sudo/HBAC.

What it is not: it does not implement the AD schema, does not serve Group Policy and Windows machines cannot join it natively. If those are needed, the options are a Samba AD DC or establishing a trust between FreeIPA and a real AD forest — FreeIPA supports the latter, and it is the usual path in mixed environments.

4.2 Server layout

Realm            HPC.LOCAL
Domain           hpc.local
Server FQDN      idm1.hpc.local  →  10.10.10.5   (cluster fabric)
Management       idm1-mgmt.hpc.local → 10.0.0.31
CA               Dogtag, self-signed root, valid until 2046
DNS              integrated (bind-dyndb-ldap), forwards to 1.1.1.1
Reverse zone     10.10.10.in-addr.arpa

Components: ipa-server 4.13.1, 389-ds-base 2.8.0, krb5-server 1.21.1, bind-dyndb-ldap 11.11.

Nine services must be running (ipactl status): Directory Service, krb5kdc, kadmin, named, httpd, ipa-custodia, pki-tomcatd, ipa-otpd, ipa-dnskeysyncd.

DNS records created for the cluster

Name Address
idm1 10.10.10.5
idm1-mgmt 10.0.0.31
head01 10.10.10.1
head01-mgmt 10.0.0.30
wwnode1…wwnode4 10.10.10.11–14

Matching PTR records exist in 10.10.10.in-addr.arpa. Reverse DNS matters: Kerberos service-ticket validation uses it.

4.3 The hard part — enrolling stateless nodes

Compute nodes are diskless; the rootfs is rebuilt in RAM on every boot. ipa-client-install therefore cannot be used: it would mint a fresh host record on every reboot and fill the directory with garbage.

The solution is to enrol once and distribute the result through Warewulf overlays:

What Where Why
krb5.conf, ipa/ca.crt ipa overlay (shared) Identical on every node
sssd.conf, ipa/default.conf ipa overlay, Warewulf template (.ww) {{.Id}} gives each node its own FQDN
krb5.keytab ipa.wwnodeN overlay (per node) Binary — cannot be templated

Why per-node keytabs

A single shared host principal would have been less work. It was rejected because it would let wwnode1 impersonate wwnode2 — any node could request service tickets as any other. Per-node keytabs preserve the security property that host identity is actually distinct.

The cost is four extra overlays and a keytab-generation step. In a larger cluster this would be scripted over the node list.

The ipa_hostname trick

The rendered sssd.conf contains:

ipa_hostname = {{.Id}}.hpc.local     →  ipa_hostname = wwnode1.hpc.local

This lets the node's system hostname stay short (wwnode1) while SSSD still finds the right Kerberos principal (host/wwnode1.hpc.local@HPC.LOCAL). That matters because Slurm's NodeName values are short — changing the system hostname to an FQDN would have meant reworking slurm.conf.

Static DNS, no dynamic updates

Nodes run with dyndns_update = False. Their DNS records are defined statically in IPA; there is no reason for a stateless node to rewrite DNS on every boot, and doing so would generate constant churn.

4.4 Rendered node configuration

/etc/sssd/sssd.conf on wwnode1 (mode 0600, or SSSD refuses to start):

[domain/hpc.local]
id_provider = ipa
auth_provider = ipa
chpass_provider = ipa
access_provider = ipa
ipa_domain = hpc.local
ipa_server = idm1.hpc.local
ipa_hostname = wwnode1.hpc.local
ldap_tls_cacert = /etc/ipa/ca.crt
cache_credentials = True
krb5_store_password_if_offline = True
dyndns_update = False

[sssd]
services = nss, pam, ssh, sudo
domains = hpc.local

[nss]
homedir_substring = /home

/etc/krb5.conf is a hand-written minimal version. Note what is absent: the pkinit_anchors and pkinit_pool lines that ipa-client-install normally adds. Those point at /var/lib/ipa-client/pki/…, which only exists after a real client install — leaving them in would break Kerberos on the nodes. auto_fast_armor = true is kept and works, because it uses the host keytab.

4.5 Users and groups

Object ID Notes
hpcusers (group) 721800003 All cluster users
hpcuser 721800005 Runs the MPI test jobs
testuser 721800006 Login verification
admin 721800000 IPA administrator

Why there is no IPA user named ilker

cloud-init creates a local ilker (uid 1000) on every VM. nsswitch.conf resolves files before sss, so an IPA user with the same name would resolve to uid 1000 on head01 and idm1 but to 721800xxx on the nodes — the same username with two different uids depending on where you land, which silently corrupts NFS home ownership.

For the same reason the local hpcuser was deleted from both head01 and the node image (wwctl image syncuser had copied it there), so the IPA account is the only one.

The local ilker home was moved from /home/ilker to /var/home/ilker so that the NFS /home mount does not hide it.

Adding a user

ssh idm1
sudo bash -c '. /root/.ipa-credentials
  echo $IPA_ADMIN_PASSWORD | kinit admin
  ipa user-add ayse --first=Ayse --last=Yilmaz \
      --homedir=/home/ayse --shell=/bin/bash --password
  ipa group-add-member hpcusers --users=ayse'

The home directory is created automatically on first login by pam_oddjob_mkhomedir, on the NFS share, on both head and nodes. No pre-creation is needed — this was verified explicitly.

4.6 Credentials

Everything lives in idm1:/root/.ipa-credentials, mode 0600: the IPA admin password, the Directory Manager password, and the generated test-user passwords. No password is stored in this repository.

ssh idm1
sudo cat /root/.ipa-credentials
sudo bash -c '. /root/.ipa-credentials; echo $IPA_ADMIN_PASSWORD | kinit admin'
sudo ipa user-find

4.7 Web UI access

The certificate is issued for idm1.hpc.local, and that name resolves to 10.10.10.5 — unreachable from the laptop. Use an SSH tunnel:

ssh -L 8443:10.10.10.5:443 head01
# add "127.0.0.1 idm1.hpc.local" to /etc/hosts, then browse:
#   https://idm1.hpc.local:8443

Going straight to https://10.0.0.31 reaches the server but produces a certificate name mismatch, and IPA's web UI redirects to its canonical hostname regardless.