Files
werewulf/docs/02-installation.md
ilkerandClaude Opus 5.5 3880311e36
ci / derle (push) Successful in 22s
hpcod: changelog for v0.2.2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 20:22:09 +01:00

9.0 KiB
Raw Permalink Blame History

2. Installation from Scratch

From an empty host to a control plane that has built and torn down a real cluster. Everything comes from one file, the hpcod binary: it carries the Ansible roles, the Terraform module, the database schema and the application catalogue, and it installs itself. There are no scripts to copy.

Allow about an hour. Most of it is spent waiting for the head node template to build.

The reference lab of sections 01 and 03–09 was built by hand, with scripts; section 10 records how. You do not need it to run hpcod. hpcod/DEPLOY.md is the reference for everything below, with unattended installs, updates, backups and the known limits.

2.1 What you need

You need Why
A Debian 13 or Ubuntu 24.04 host with root, 2 vCPU, 4 GB RAM, 20 GB disk the control plane
Proxmox VE 8 or later, and root's password (or two API tokens, 2.4) where clusters are built
A /24 management network shared by the control plane and Proxmox head nodes get addresses on it
About 40 free addresses on it, outside DHCP, plus one more one per live cluster, one for the template build
A name for the portal and a domain for clusters each cluster's portal is c<NNN>.<CLUSTER_DOMAIN>
Internet access from the control plane and from head nodes packages, OpenTofu providers, images

Proxmox is one node of a Proxmox cluster? Setup lists the other nodes and asks which ones clusters may use; you do not add them one by one (2.3). No Proxmox to hand? Setup can run one for you, simulated in QEMU on the control plane host (2.5). No public domain or certificate? hpcod can serve HTTPS and DNS itself (2.6). The node, desktop and application images come from a public registry and pull anonymously; you build none of them.

2.2 Get the binary and run setup

On the control plane host:

V=v0.2.2 A=linux-amd64        # A=linux-arm64 on an arm64 host
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/hpcod-$V-$A
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/SHA256SUMS
sha256sum -c --ignore-missing SHA256SUMS
install -m 0755 hpcod-$V-$A hpcod
sudo ./hpcod setup

A release binary arrives without its execute bit, which install sets. Setup copies itself to /usr/local/bin/hpcod; from then on it is sudo hpcod …. To build the binary instead, make linux in hpcod/.

Setup is for administrators. It refuses to run as anyone but root, runs only in a terminal on the control plane host, and the portal never exposes it.

It is a full-screen program. The stages of the install run down the left and tick off as you go; the keys that work are along the bottom. Arrows move, enter goes on, esc goes back.

The setup welcome screen

On a fresh host the first screen offers one thing, Install hpcod on this host. On an installed one it offers to check the installation, reconfigure and install again, build a test cluster, or rebuild the head node template.

2.3 What setup asks

  1. Where clusters are built: a Proxmox VE you already have, or a simulated one in QEMU on this host.

    Choosing where clusters are built

  2. Proxmox. The API address, its certificate, and how to sign in: root@pam's password, used once to create two API tokens and to let this host in over SSH and never stored; or two tokens you made yourself (2.4). Setup signs in at once, so a wrong password, an untrusted certificate or an address nothing answers on is reported here, not twenty minutes later. For a simulated Proxmox this screen asks only for its cores, memory and disk.

  3. The site, on one form: the Proxmox node and datastores, picked from what Proxmox returned; in a Proxmox cluster, which of its nodes compute nodes may use (heads stay on the node with the template, and more than one node makes the SDN zone vxlan, see DEPLOY.md); the portal and cluster domains; how users reach the portal (behind your reverse proxy, or hpcod serving HTTPS and DNS itself); the management network and the head node window. Each field is checked as you type, including against the others.

  4. Review. Everything that will happen, before any of it does.

    The review screen

2.4 What setup does

The steps tick off in two columns, with the current step's output below:

site       packages   account    binary     keys       ca
database   tree       schema     catalogue  config     unit
start      pvesim     proxmox    template
  • site … unit install PostgreSQL, Ansible and OpenTofu, create the hpcod account, install the binary as /usr/local/bin/hpcod, unpack the roles, module and schema into /var/lib/hpcod, apply the schema, and write /etc/hpcod and the systemd unit.
  • start starts the portal and waits for /healthz. It comes before the Proxmox steps, so an install that stops later still leaves a portal running.
  • pvesim installs Proxmox VE into the simulated VM, when there is one.
  • proxmox sets up SSH access, both tokens, the SDN zone, snippets and import on the datastores, and the cloud image.
  • template builds the head node template, the longest step (2.8).

Each step reads back what it changed, and every step is safe to run again. The full log is /var/log/hpcod-setup.log. The first portal login is printed once and kept in /root/.hpcod-portal-login.

If you would rather not give setup root's password, make the tokens yourself and choose "API tokens I made myself":

pveum user add hpcod@pve
pveum acl modify / --users hpcod@pve --roles PVEAuditor
pveum user token add hpcod@pve discovery --privsep 0       # discovery token
pveum user token add root@pam hpcod-tf --privsep 0         # Terraform token

The Terraform token is root's because Proxmox allows raw PCI passthrough, which is how a GPU reaches a compute node, to root alone. With your own tokens you also add this host's SSH key to root on Proxmox; setup shows it.

2.5 No Proxmox yet: the simulated one

"A simulated Proxmox, in QEMU on this host" runs Proxmox VE in a VM on the control plane host, on a private bridge (10.99.0.0/24) with NAT, so it never touches your network. Setup installs Proxmox VE into it, fills in every Proxmox and network value, and carries on as with a real one.

The host needs /dev/kvm, nested virtualization (clusters run inside the simulation), and room: 8 cores, 24 GiB and 150 GiB by default. On a VM, give it CPU type host. It is for trying hpcod, not for tenants.

2.6 Reaching the portal

Behind your own reverse proxy. Copy the two blocks from hpcod/deploy/Caddyfile.example into your Caddyfile, set your domains and a wildcard certificate for *.CLUSTER_DOMAIN (DNS-01), and reload. Setup does not touch the proxy.

On a closed network. Choose "hpcod serves HTTPS itself" and "hpcod answers DNS for them". hpcod then listens on 443 at PUBLIC_IP with a certificate from its own CA, which can sign only for your domains, and answers DNS on port 53 for the portal and every cluster name, forwarding the rest to NAMESERVERS. Users point their resolver at it (or forward the zone to it) and trust http://<portal>/ca.crt once. Setup prints exactly this at the end.

2.7 Check it, then prove it

Setup ends by checking the installation. The same check runs on its own:

sudo hpcod verify

It checks each fact by using it: the tokens against the API, the template by asking Proxmox, the database by connecting as the portal does. It ends with READY or NOT READY, and each failed line names the step that fixes it.

The installation check

Then setup offers to build a test cluster. The same test runs on its own:

sudo hpcod selftest                # 1 compute node; -compute 2 also runs a job across two

It takes the path a user's "create" takes: discovery, placement, allocation, then the portal's own builder. It waits for Slurm to see the nodes, runs srun -N <n> hostname, checks that the cluster's portal is published, and takes the cluster down again. About 25 minutes. A failed test leaves the cluster up for inspection and prints the command that removes it.

A passing run ends like this:

==> placed on pvesim
==> allocated c001-1 (serial 1); it shows on the portal's cluster list
==> published: https://c001.hpc.lab
==> waiting for 1 compute node(s) to be idle in Slurm
==> PASSED: c001-1 built, published, and ran a job on 1 node(s)
==> destroying c001-1
==> destroyed and released

From here, sign in to the portal with the login setup printed and create a cluster. Section 20 explains what happens when you do.

2.8 Timings

One run, on the simulated Proxmox, itself nested inside a Proxmox VM:

Stage Duration
A full setup run with a template build 16½ min
of which the template playbook 11 min
Test cluster, built and destroyed (setup's own estimate) ~25 min

Installing Proxmox VE into the simulated VM comes on top, once; it was not timed separately.