Files
ilkerandClaude Opus 5.5 3880311e36
ci / derle (push) Successful in 22s
hpcod: changelog for v0.2.2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 20:22:09 +01:00

15 KiB

Deploying hpcod

How to install the HPC-on-Demand control plane: on your own Proxmox VE, or, to try it, on a Proxmox that setup simulates on the same host. You need one file, the hpcod binary, and about an hour, most of it spent waiting for the head node template to build.

The binary carries everything a control plane needs: the Ansible roles, the Terraform module, the database schema and the application catalogue. It installs itself. There are no scripts to copy alongside it.

What you end up with

  browser ──https──▶ Caddy ──▶ hpcod (127.0.0.1:9880) ──▶ PostgreSQL
                       │            │
       c001.<domain> ──┘            ├──tofu────▶ Proxmox API: head + compute VMs, SDN vnet
                                    └──ssh─────▶ head nodes (archive /home on release)
  • Control plane host: a Debian 13 or Ubuntu 24.04 VM or machine. It runs hpcod, PostgreSQL, OpenTofu and Ansible. It sits behind your reverse proxy, or serves HTTPS and DNS itself on a closed network; in that case Caddy drops out of the picture above.
  • Proxmox VE 8 or 9: where the clusters are built. Each cluster gets a head node cloned from a template, compute nodes that PXE boot from it, and an SDN vnet of its own.
  • Per-cluster portals: each cluster runs Open OnDemand at c<NNN>.<CLUSTER_DOMAIN>, reached through the control plane.

Before you start

You need Why
A Debian 13 or Ubuntu 24.04 host with root, 2 vCPU, 4 GB RAM, 20 GB disk the control plane
Proxmox VE 8 or later, and root's password (or two API tokens, see below) setup creates the tokens and lets this host in over SSH. Or none, to try it: a simulated Proxmox needs /dev/kvm and nested virtualization instead
A /24 management network shared by the control plane and Proxmox head nodes get addresses on it (a simulated Proxmox brings its own)
About 40 free addresses on that network, outside DHCP, plus one more one per live cluster, and one for the template build
Names for the portal and for *.CLUSTER_DOMAIN cluster portals are subdomains. Either public DNS and a wildcard certificate (DNS-01; deploy/Caddyfile.example), or hpcod's own DNS and CA on a closed network
Internet access from the control plane and from head nodes packages, OpenTofu providers, and the images

The node, desktop and application images come from a public registry by default and pull anonymously. You do not need to build any images.

Install

On the control plane host, as an ordinary user with sudo:

V=v0.2.2 A=linux-amd64        # A=linux-arm64 on an arm64 host
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/hpcod-$V-$A
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/SHA256SUMS
sha256sum -c --ignore-missing SHA256SUMS
install -m 0755 hpcod-$V-$A hpcod
sudo ./hpcod setup

A release binary arrives without its execute bit and under its release name, which is what install fixes. Setup copies itself to /usr/local/bin/hpcod, so after the first run it is sudo hpcod …. To build the binary yourself instead: make linux in hpcod/ (LINUX_ARCH=arm64 for arm64), and copy hpcod-linux-amd64 to the host.

setup is the installer. It is for administrators: it refuses to run as anyone but root, it runs only in a terminal on the control plane host, and the portal never exposes it.

It is a full-screen program. The stages of an install run down the left and tick off as you go. The screen in hand is on the right, and the keys that work on it are along the bottom: arrows to move, enter to go on, esc to go back.

The setup welcome screen

On a host that is already installed, the first screen offers four things instead of an install:

  • check the installation;
  • reconfigure and install again;
  • build a test cluster;
  • rebuild the head node template.

What it asks

  1. Where clusters are built. A Proxmox VE you already have, or a simulated one in QEMU on this host (see below).

    Choosing where clusters are built

  2. Proxmox. The API address, its certificate, and how to sign in:

    • root@pam's password. Setup uses it once, to create two API tokens and to put this host's SSH key on Proxmox. It is not stored.
    • Or two tokens you made yourself (see below). In that case you also add this host's SSH key to root on Proxmox yourself; setup shows the key.

    Setup signs in before it asks anything else. A wrong password, an untrusted certificate or an address nothing answers on is reported at this point, not twenty minutes later.

    With a simulated Proxmox this screen is replaced by three numbers: the cores, memory and disk to give it.

  3. The site. On one form, in four groups:

    • Proxmox resources: the node and datastores, picked from the lists Proxmox returned; also the cloud image, the SDN zone and the SSH address.
    • Names: the portal domain and the cluster domain.
    • How users reach the portal: behind your own reverse proxy, or served by hpcod itself with its own CA and DNS (see below).
    • Management network and head nodes and images. The defaults for images work.

    Each field is checked as you type, including against the others. For example, the build address may not fall inside the head node window. With a simulated Proxmox, the network group is left out, because its network is its own.

  4. Review. Everything that will happen, before any of it does. On a reinstall it also asks whether to rebuild the head node template.

    The review screen

What it does

Then it runs every step. The steps tick off in two columns, with the current step's output below them:

Step What it does
site records the answers in /etc/hpcod/site.env
packages PostgreSQL, Ansible, and OpenTofu from its own apt repository
account the hpcod service account and its directories
binary installs itself as /usr/local/bin/hpcod
keys the provisioner's SSH key and the client certificate every cluster pins
ca the name-constrained CA, when hpcod serves HTTPS itself
database a local role and database; /etc/hpcod/db.env
tree the roles, Terraform module and schema, from inside the binary, into /var/lib/hpcod
schema applies every migration, then checks the portal's role can read every table
catalogue the application catalogue, into /etc/hpcod/apps.yaml
config sites.yaml, the Terraform variables, and the first portal login
unit the systemd unit, then reads it back through systemd
start starts the portal and waits for /healthz
pvesim the simulated Proxmox, when there is one: its network, its disk, Proxmox VE inside it
proxmox SSH access, both tokens, the SDN zone, snippets and import on the datastores, the cloud image
template builds the head node template if there is none (15 minutes on hardware, longer nested)

Each step reads back what it changed, and every step is safe to run again. The portal starts before Proxmox is prepared. If an install stops there, for example because Proxmox cannot be reached yet, the portal is still running. The full log goes to /var/log/hpcod-setup.log. The first portal login is printed once and saved in /root/.hpcod-portal-login.

At the end, setup checks the installation (see Verify), says what to tell the people who will use the portal, and offers to build a test cluster.

The installation check

Making the tokens yourself

If you would rather not give setup root's password, create the two tokens and pick "API tokens I made myself":

pveum user add hpcod@pve
pveum acl modify / --users hpcod@pve --roles PVEAuditor
pveum user token add hpcod@pve discovery --privsep 0       # discovery token
pveum user token add root@pam hpcod-tf --privsep 0         # Terraform token

The Terraform token is root's because Proxmox allows raw PCI passthrough (how a GPU reaches a compute node) to root and nobody else.

No Proxmox yet: the simulated one

Setup's first question is where clusters are built. "A simulated Proxmox, in QEMU on this host" runs Proxmox VE in a VM on the control plane host. It uses a private bridge (10.99.0.0/24) with NAT, so it never touches your network. Setup installs Proxmox VE into it, fills in every Proxmox and network value, and continues as with a real one.

The host needs:

  • /dev/kvm;
  • nested virtualization, because clusters run inside the simulation;
  • room for it: 8 cores, 24 GiB and 150 GiB by default.

On a VM, give it CPU type host. It is for trying hpcod, not for tenants.

A Proxmox cluster

If the Proxmox VE is one node of a Proxmox cluster, you do not add the other nodes one by one. Discovery reads the whole cluster through one API, and the tokens setup makes are valid on every node. Setup lists the nodes and asks which of them clusters may use. Nodes left unticked are never touched, so a Proxmox cluster shared with other work stays shared. Unattended, the same choice is PVE_NODES in site.env.

  • Heads stay on PVE_NODE. The head template lives there, and Proxmox cannot clone a template onto another node unless its disk is on shared storage. The head is the only VM with a disk, so this asks for no shared storage at all. The snippets stay there too, for the same reason.
  • Compute nodes go on any ticked node. They are diskless and PXE boot from their head, so they need nothing on the node but room, and a GPU if they asked for one.
  • The SDN zone becomes vxlan. A simple zone gives every node its own bridge, so a compute node on another node could never reach its head. With more than one node ticked, setup makes the zone vxlan, with every ticked node as a peer and an MTU of 1450 (VXLAN takes 50 bytes of every frame). Guests are given the bridge's MTU through virtio. The nodes must reach each other on UDP 4789.

A zone's type cannot be changed in place. Setup deletes and remakes it only while no cluster has a vnet in it; otherwise it stops and names the clusters to release first. Adding or removing a node later only changes the peers, in place. hpcod verify checks that every node is online and that the zone is the type, and has the peers, that the nodes need.

PVE_SDN_ZONE_TYPE=vxlan makes a vxlan zone on a single node as well, to be ready for more nodes without releasing every cluster.

No public domain: its own CA and DNS

On a closed network, choose the following when setup asks how users reach the portal:

  • "hpcod serves HTTPS itself, with its own certificate authority". hpcod listens on 443 at PUBLIC_IP. Its CA can sign only for your domains.
  • "hpcod answers DNS for them". hpcod answers for the portal and for every cluster name on port 53 at PUBLIC_IP, and forwards other names to NAMESERVERS.

At the end, setup prints what to tell users: use it as a resolver (or forward the zone to it), and trust http://<portal>/ca.crt once. You don't need a proxy in this mode.

Put the proxy in front

Only when you chose "behind my own reverse proxy". Copy the two blocks from deploy/Caddyfile.example into your Caddyfile, set your domains and wildcard certificate, and reload Caddy. Setup does not touch the proxy; it usually serves other sites too.

Verify

sudo hpcod verify

Verify checks each fact by using it: the tokens against the API, the template by asking Proxmox, the database by connecting as the portal does. It ends with READY or NOT READY, and every failed line names the step that fixes it. The same check is in the setup menu.

Test cluster

sudo hpcod selftest                # 1 compute node; -compute 2 also tests jobs across nodes

This builds the smallest real cluster, by the same path a user's "create" takes: discovery, placement, allocation, then the portal's own builder.

  1. It waits for Slurm to see the nodes, runs srun -N <n> hostname, and checks that the cluster's portal is published.
  2. It then takes the cluster down again. -keep leaves it up.
  3. If the test fails, the cluster is left up for inspection, and selftest prints the command that removes it (hpcod selftest -destroy <serial>).

It takes about 25 minutes. The cluster appears on the portal's cluster list while it exists, under the tenant hpcod-selftest.

Unattended install and updates

From a workstation with the repository, make deploy builds the binary, copies it and deploy/site.env to DEPLOY_HOST, and runs hpcod setup -yes there:

cd hpcod
cp deploy/site.env.example deploy/site.env    # fill it in
make deploy                                   # LINUX_ARCH=arm64 for an arm64 host
make verify
make selftest

Unattended, setup reads the Proxmox credentials from the host's environment, never from the command line. They are needed only until the tokens exist:

HPCOD_PVE_PASSWORD           root@pam's password, or
HPCOD_PVE_DISCOVERY_TOKEN    the two tokens
HPCOD_PVE_TERRAFORM_TOKEN

On a first install without them, every step up to the portal start completes. The Proxmox step then stops and says what it needs. Running sudo hpcod setup on the host finishes the install.

When Do
After git pull make deploy
Binary-only change make deploy-portal
The roles changed and setup says the template is older than them make deploy SETUP_ARGS=-rebuild-template, or the menu entry
A value in site.env changed make deploy; if it was the domain, the images or CI_USER, also rebuild the template
Use more nodes of the same Proxmox cluster sudo hpcod setup, reconfigure, tick them; or PVE_NODES and make deploy
Add a second Proxmox site edit /etc/hpcod/sites.yaml; setup writes it once, then keeps only its own site's nodes and head_node
Keep your own application catalogue create /etc/hpcod/apps.local; setup then leaves apps.yaml alone

Back up these three things:

  • the database: sudo -u postgres pg_dump hpcod
  • /etc/hpcod: tokens, the portal's users, and the client certificate every cluster pins
  • /var/lib/hpcod/terraform/envs: one Terraform state per cluster, and the only record of which VMs belong to which cluster

Known limits

  • Only a /24 management network. The head node prefix is fixed at 24.
  • Heads run on one node. PVE_NODE holds the template, the snippets and every head; only compute nodes spread over a Proxmox cluster.
  • Only Debian and Ubuntu control plane hosts. Setup installs with apt. On other distributions, install PostgreSQL, Ansible and OpenTofu yourself and run hpcod setup -yes -skip packages.
  • No TLS automation. The wildcard certificate is yours to obtain and renew.
  • Host keys are not checked. Setup does not check SSH host keys on the hypervisor or on freshly cloned heads. The Terraform provider and the builds already behaved this way.