Files
werewulf/hpcod/DEPLOY.md
T
ilkerandClaude Opus 5.5 8556a50e94
ci / derle (push) Failing after 34s
hpcod: DEPLOY.md, and the changelog for v0.2.0
How someone else installs hpcod: the binary, sudo hpcod setup, what it
asks and what each step does, the simulated Proxmox, a closed network
with its own CA and DNS, verify, selftest, and what to back up. The
changelog says what was proven and what was not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:20:10 +01:00

10 KiB

Deploying hpcod

How to install the HPC-on-Demand control plane against your own Proxmox VE. You need one file, the hpcod binary, and about an hour, most of it spent waiting for the head node template to build.

The binary carries everything a control plane needs: the Ansible roles, the Terraform module, the database schema and the application catalogue. It installs itself. There are no scripts to copy alongside it.

What you end up with

  browser ──https──▶ Caddy ──▶ hpcod (127.0.0.1:9880) ──▶ PostgreSQL
                       │            │
       c001.<domain> ──┘            ├──tofu────▶ Proxmox API: head + compute VMs, SDN vnet
                                    └──ssh─────▶ head nodes (archive /home on release)
  • Control plane host: a Debian 13 or Ubuntu 24.04 VM or machine. It runs hpcod, PostgreSQL, OpenTofu and Ansible, behind a reverse proxy.
  • Proxmox VE 8 or 9: where the clusters are built. Each cluster gets a head node cloned from a template, compute nodes that PXE boot from it, and an SDN vnet of its own.
  • Per-cluster portals: each cluster runs Open OnDemand at c<NNN>.<CLUSTER_DOMAIN>, reached through the control plane.

Before you start

You need Why
A Debian 13 or Ubuntu 24.04 host with root, 2 vCPU, 4 GB RAM, 20 GB disk the control plane
Proxmox VE 8 or later, and root's password (or two API tokens, see below) setup creates the tokens and lets this host in over SSH
A /24 management network shared by the control plane and Proxmox head nodes get addresses on it
About 40 free addresses on that network, outside DHCP, plus one more one per live cluster, and one for the template build
DNS: an A record for PORTAL_DOMAIN and a wildcard *.CLUSTER_DOMAIN, both pointing at the proxy cluster portals are subdomains
A wildcard TLS certificate for *.CLUSTER_DOMAIN needs DNS-01; see deploy/Caddyfile.example
Internet access from the control plane and from head nodes packages, OpenTofu providers, and the images

The node, desktop and application images come from a public registry by default and pull anonymously. You do not need to build any images.

Install

Get the binary for the host: a release from https://gitea.mynodes.xyz/ilker/werewulf/releases, or make linux in hpcod/ (LINUX_ARCH=arm64 for an arm64 host). Copy it to the host, then:

sudo ./hpcod setup

setup is the installer. It is for administrators: it refuses to run as anyone but root, it runs only in a terminal on the control plane host, and the portal never exposes it.

It asks for the following, in order:

  1. Proxmox. The API address, and how to sign in:

    • root@pam's password. Setup uses it once, to create two API tokens and to put this host's SSH key on Proxmox. It is not stored.
    • Or two tokens you made yourself (see below). In that case you also add this host's SSH key to root on Proxmox yourself; setup shows the key.

    Setup signs in before it asks anything else. A wrong password, an untrusted certificate or an address nothing answers on is reported at this point, not twenty minutes later.

  2. Proxmox resources. The node and the datastores, chosen from the lists Proxmox returned. Also the cloud image, the SDN zone, and the SSH address.

  3. Names. The portal domain and the cluster domain.

  4. Management network. Bridge, gateway, the head node address window, a build address, and DNS servers. Each field is checked as you enter it, including against the others: for example, the build address may not fall inside the head node window.

  5. Head nodes and images. The defaults work.

Then it runs every step and shows the output as it goes:

Step What it does
site records the answers in /etc/hpcod/site.env
packages PostgreSQL, Ansible, and OpenTofu from its own apt repository
account the hpcod service account and its directories
binary installs itself as /usr/local/bin/hpcod
keys the provisioner's SSH key and the client certificate every cluster pins
database a local role and database; /etc/hpcod/db.env
tree the roles, Terraform module and schema, from inside the binary, into /var/lib/hpcod
schema applies every migration, then checks the portal's role can read every table
catalogue the application catalogue, into /etc/hpcod/apps.yaml
config sites.yaml, the Terraform variables, and the first portal login
unit the systemd unit, then reads it back through systemd
start starts the portal and waits for /healthz
proxmox SSH access, both tokens, the SDN zone, snippets and import on the datastores, the cloud image
template builds the head node template if there is none (about 15 minutes)

Each step reads back what it changed, and every step is safe to run again. The full log goes to /var/log/hpcod-setup.log. The first portal login is printed once and saved in /root/.hpcod-portal-login.

At the end, setup checks the installation (see Verify) and offers to build a test cluster.

Making the tokens yourself

If you would rather not give setup root's password, create the two tokens and pick "API tokens I made myself":

pveum user add hpcod@pve
pveum acl modify / --users hpcod@pve --roles PVEAuditor
pveum user token add hpcod@pve discovery --privsep 0       # discovery token
pveum user token add root@pam hpcod-tf --privsep 0         # Terraform token

The Terraform token is root's because Proxmox allows raw PCI passthrough (how a GPU reaches a compute node) to root and nobody else.

No Proxmox yet: the simulated one

Setup's first question is where clusters are built. "A simulated Proxmox, in QEMU on this host" runs Proxmox VE in a VM on the control plane host. It uses a private bridge (10.99.0.0/24) with NAT, so it never touches your network. Setup installs Proxmox VE into it, fills in every Proxmox and network value, and continues as with a real one.

The host needs:

  • /dev/kvm;
  • nested virtualization, because clusters run inside the simulation;
  • room for it: 8 cores, 24 GiB and 150 GiB by default.

On a VM, give it CPU type host. It is for trying hpcod, not for tenants.

No public domain: its own CA and DNS

On a closed network, choose the following when setup asks how users reach the portal:

  • "hpcod serves HTTPS itself, with its own certificate authority". hpcod listens on 443 at PUBLIC_IP. Its CA can sign only for your domains.
  • "hpcod answers DNS for them". hpcod answers for the portal and for every cluster name on port 53 at PUBLIC_IP, and forwards other names to NAMESERVERS.

At the end, setup prints what to tell users: use it as a resolver (or forward the zone to it), and trust http://<portal>/ca.crt once. You don't need a proxy in this mode.

Put the proxy in front

Copy the two blocks from deploy/Caddyfile.example into your Caddyfile, set your domains and wildcard certificate, and reload Caddy. Setup does not touch the proxy; it usually serves other sites too.

Verify

sudo hpcod verify

Verify checks each fact by using it: the tokens against the API, the template by asking Proxmox, the database by connecting as the portal does. It ends with READY or NOT READY, and every failed line names the step that fixes it. The same check is in the setup menu.

Test cluster

sudo hpcod selftest                # 1 compute node; -compute 2 also tests jobs across nodes

This builds the smallest real cluster, by the same path a user's "create" takes: discovery, placement, allocation, then the portal's own builder.

  1. It waits for Slurm to see the nodes, runs srun -N <n> hostname, and checks that the cluster's portal is published.
  2. It then takes the cluster down again. -keep leaves it up.
  3. If the test fails, the cluster is left up for inspection, and selftest prints the command that removes it (hpcod selftest -destroy <serial>).

It takes about 25 minutes. The cluster appears on the portal's cluster list while it exists, under the tenant hpcod-selftest.

Unattended install and updates

From a workstation with the repository, make deploy builds the binary, copies it and deploy/site.env to DEPLOY_HOST, and runs hpcod setup -yes there:

cd hpcod
cp deploy/site.env.example deploy/site.env    # fill it in
make deploy                                   # LINUX_ARCH=arm64 for an arm64 host
make verify
make selftest

Unattended, setup reads the Proxmox credentials from the host's environment, never from the command line. They are needed only until the tokens exist:

HPCOD_PVE_PASSWORD           root@pam's password, or
HPCOD_PVE_DISCOVERY_TOKEN    the two tokens
HPCOD_PVE_TERRAFORM_TOKEN

On a first install without them, every step up to the portal start completes. The Proxmox step then stops and says what it needs. Running sudo hpcod setup on the host finishes the install.

When Do
After git pull make deploy
Binary-only change make deploy-portal
The roles changed and setup says the template is older than them make deploy SETUP_ARGS=-rebuild-template, or the menu entry
A value in site.env changed make deploy; if it was the domain, the images or CI_USER, also rebuild the template
Add a second Proxmox site edit /etc/hpcod/sites.yaml; setup writes it once and never again
Keep your own application catalogue create /etc/hpcod/apps.local; setup then leaves apps.yaml alone

Back up these three things:

  • the database: sudo -u postgres pg_dump hpcod
  • /etc/hpcod: tokens, the portal's users, and the client certificate every cluster pins
  • /var/lib/hpcod/terraform/envs: one Terraform state per cluster, and the only record of which VMs belong to which cluster

Known limits

  • Only a /24 management network. The head node prefix is fixed at 24.
  • One default Proxmox node per installation. PVE_NODE is where snippets and the template live.
  • Only Debian and Ubuntu control plane hosts. Setup installs with apt. On other distributions, install PostgreSQL, Ansible and OpenTofu yourself and run hpcod setup -yes -skip packages.
  • No TLS automation. The wildcard certificate is yours to obtain and renew.
  • Host keys are not checked. Setup does not check SSH host keys on the hypervisor or on freshly cloned heads. The Terraform provider and the builds already behaved this way.