A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.
Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
fabric so Kerberos and NFS never traverse head01's masquerade.
Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.
Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.
Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.
Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.
No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
6.8 KiB
4. Identity — FreeIPA and SSSD
4.1 FreeIPA is not Active Directory
The original request was for Active Directory. What was built is FreeIPA, chosen deliberately after weighing the options:
| Option | Verdict |
|---|---|
| Samba 4 AD DC on Rocky 9 | Red Hat does not support Samba in AD DC mode. Only an experimental MIT-KDC build exists. Not viable. |
| Samba 4 AD DC on Debian | Fully supported AD DC (Heimdal KDC), genuine AD — but a second distro in an otherwise all-Rocky lab. |
| FreeIPA on Rocky 9 | Chosen. The RHEL-native identity server: LDAP + Kerberos + CA + DNS. |
For Linux clients FreeIPA is functionally equivalent to AD: Kerberos realm, LDAP directory, POSIX attributes, host principals, centralised sudo/HBAC.
What it is not: it does not implement the AD schema, does not serve Group Policy and Windows machines cannot join it natively. If those are needed, the options are a Samba AD DC or establishing a trust between FreeIPA and a real AD forest — FreeIPA supports the latter, and it is the usual path in mixed environments.
4.2 Server layout
Realm HPC.LOCAL
Domain hpc.local
Server FQDN idm1.hpc.local → 10.10.10.5 (cluster fabric)
Management idm1-mgmt.hpc.local → 10.0.0.31
CA Dogtag, self-signed root, valid until 2046
DNS integrated (bind-dyndb-ldap), forwards to 1.1.1.1
Reverse zone 10.10.10.in-addr.arpa
Components: ipa-server 4.13.1, 389-ds-base 2.8.0, krb5-server 1.21.1,
bind-dyndb-ldap 11.11.
Nine services must be running (ipactl status): Directory Service, krb5kdc,
kadmin, named, httpd, ipa-custodia, pki-tomcatd, ipa-otpd, ipa-dnskeysyncd.
DNS records created for the cluster
| Name | Address |
|---|---|
idm1 |
10.10.10.5 |
idm1-mgmt |
10.0.0.31 |
head01 |
10.10.10.1 |
head01-mgmt |
10.0.0.30 |
wwnode1…wwnode4 |
10.10.10.11–14 |
Matching PTR records exist in 10.10.10.in-addr.arpa. Reverse DNS matters:
Kerberos service-ticket validation uses it.
4.3 The hard part — enrolling stateless nodes
Compute nodes are diskless; the rootfs is rebuilt in RAM on every boot.
ipa-client-install therefore cannot be used: it would mint a fresh host
record on every reboot and fill the directory with garbage.
The solution is to enrol once and distribute the result through Warewulf overlays:
| What | Where | Why |
|---|---|---|
krb5.conf, ipa/ca.crt |
ipa overlay (shared) |
Identical on every node |
sssd.conf, ipa/default.conf |
ipa overlay, Warewulf template (.ww) |
{{.Id}} gives each node its own FQDN |
krb5.keytab |
ipa.wwnodeN overlay (per node) |
Binary — cannot be templated |
Why per-node keytabs
A single shared host principal would have been less work. It was rejected
because it would let wwnode1 impersonate wwnode2 — any node could request
service tickets as any other. Per-node keytabs preserve the security property
that host identity is actually distinct.
The cost is four extra overlays and a keytab-generation step. In a larger cluster this would be scripted over the node list.
The ipa_hostname trick
The rendered sssd.conf contains:
ipa_hostname = {{.Id}}.hpc.local → ipa_hostname = wwnode1.hpc.local
This lets the node's system hostname stay short (wwnode1) while SSSD
still finds the right Kerberos principal (host/wwnode1.hpc.local@HPC.LOCAL).
That matters because Slurm's NodeName values are short — changing the system
hostname to an FQDN would have meant reworking slurm.conf.
Static DNS, no dynamic updates
Nodes run with dyndns_update = False. Their DNS records are defined
statically in IPA; there is no reason for a stateless node to rewrite DNS on
every boot, and doing so would generate constant churn.
4.4 Rendered node configuration
/etc/sssd/sssd.conf on wwnode1 (mode 0600, or SSSD refuses to start):
[domain/hpc.local]
id_provider = ipa
auth_provider = ipa
chpass_provider = ipa
access_provider = ipa
ipa_domain = hpc.local
ipa_server = idm1.hpc.local
ipa_hostname = wwnode1.hpc.local
ldap_tls_cacert = /etc/ipa/ca.crt
cache_credentials = True
krb5_store_password_if_offline = True
dyndns_update = False
[sssd]
services = nss, pam, ssh, sudo
domains = hpc.local
[nss]
homedir_substring = /home
/etc/krb5.conf is a hand-written minimal version. Note what is absent:
the pkinit_anchors and pkinit_pool lines that ipa-client-install normally
adds. Those point at /var/lib/ipa-client/pki/…, which only exists after a
real client install — leaving them in would break Kerberos on the nodes.
auto_fast_armor = true is kept and works, because it uses the host keytab.
4.5 Users and groups
| Object | ID | Notes |
|---|---|---|
hpcusers (group) |
721800003 | All cluster users |
hpcuser |
721800005 | Runs the MPI test jobs |
testuser |
721800006 | Login verification |
admin |
721800000 | IPA administrator |
Why there is no IPA user named ilker
cloud-init creates a local ilker (uid 1000) on every VM. nsswitch.conf
resolves files before sss, so an IPA user with the same name would resolve
to uid 1000 on head01 and idm1 but to 721800xxx on the nodes — the same
username with two different uids depending on where you land, which silently
corrupts NFS home ownership.
For the same reason the local hpcuser was deleted from both head01 and
the node image (wwctl image syncuser had copied it there), so the IPA account
is the only one.
The local ilker home was moved from /home/ilker to /var/home/ilker
so that the NFS /home mount does not hide it.
Adding a user
ssh idm1
sudo bash -c '. /root/.ipa-credentials
echo $IPA_ADMIN_PASSWORD | kinit admin
ipa user-add ayse --first=Ayse --last=Yilmaz \
--homedir=/home/ayse --shell=/bin/bash --password
ipa group-add-member hpcusers --users=ayse'
The home directory is created automatically on first login by
pam_oddjob_mkhomedir, on the NFS share, on both head and nodes. No
pre-creation is needed — this was verified explicitly.
4.6 Credentials
Everything lives in idm1:/root/.ipa-credentials, mode 0600: the IPA admin
password, the Directory Manager password, and the generated test-user
passwords. No password is stored in this repository.
ssh idm1
sudo cat /root/.ipa-credentials
sudo bash -c '. /root/.ipa-credentials; echo $IPA_ADMIN_PASSWORD | kinit admin'
sudo ipa user-find
4.7 Web UI access
The certificate is issued for idm1.hpc.local, and that name resolves to
10.10.10.5 — unreachable from the laptop. Use an SSH tunnel:
ssh -L 8443:10.10.10.5:443 head01
# add "127.0.0.1 idm1.hpc.local" to /etc/hosts, then browse:
# https://idm1.hpc.local:8443
Going straight to https://10.0.0.31 reaches the server but produces a
certificate name mismatch, and IPA's web UI redirects to its canonical
hostname regardless.