Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
15 KiB
Deploying hpcod
How to install the HPC-on-Demand control plane: on your own Proxmox VE, or,
to try it, on a Proxmox that setup simulates on the same host. You need one
file, the hpcod binary, and about an hour, most of it spent waiting for
the head node template to build.
The binary carries everything a control plane needs: the Ansible roles, the Terraform module, the database schema and the application catalogue. It installs itself. There are no scripts to copy alongside it.
What you end up with
browser ──https──▶ Caddy ──▶ hpcod (127.0.0.1:9880) ──▶ PostgreSQL
│ │
c001.<domain> ──┘ ├──tofu────▶ Proxmox API: head + compute VMs, SDN vnet
└──ssh─────▶ head nodes (archive /home on release)
- Control plane host: a Debian 13 or Ubuntu 24.04 VM or machine. It runs hpcod, PostgreSQL, OpenTofu and Ansible. It sits behind your reverse proxy, or serves HTTPS and DNS itself on a closed network; in that case Caddy drops out of the picture above.
- Proxmox VE 8 or 9: where the clusters are built. Each cluster gets a head node cloned from a template, compute nodes that PXE boot from it, and an SDN vnet of its own.
- Per-cluster portals: each cluster runs Open OnDemand at
c<NNN>.<CLUSTER_DOMAIN>, reached through the control plane.
Before you start
| You need | Why |
|---|---|
| A Debian 13 or Ubuntu 24.04 host with root, 2 vCPU, 4 GB RAM, 20 GB disk | the control plane |
| Proxmox VE 8 or later, and root's password (or two API tokens, see below) | setup creates the tokens and lets this host in over SSH. Or none, to try it: a simulated Proxmox needs /dev/kvm and nested virtualization instead |
| A /24 management network shared by the control plane and Proxmox | head nodes get addresses on it (a simulated Proxmox brings its own) |
| About 40 free addresses on that network, outside DHCP, plus one more | one per live cluster, and one for the template build |
Names for the portal and for *.CLUSTER_DOMAIN |
cluster portals are subdomains. Either public DNS and a wildcard certificate (DNS-01; deploy/Caddyfile.example), or hpcod's own DNS and CA on a closed network |
| Internet access from the control plane and from head nodes | packages, OpenTofu providers, and the images |
The node, desktop and application images come from a public registry by default and pull anonymously. You do not need to build any images.
Install
On the control plane host, as an ordinary user with sudo:
V=v0.2.2 A=linux-amd64 # A=linux-arm64 on an arm64 host
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/hpcod-$V-$A
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/SHA256SUMS
sha256sum -c --ignore-missing SHA256SUMS
install -m 0755 hpcod-$V-$A hpcod
sudo ./hpcod setup
A release binary arrives without its execute bit and under its release
name, which is what install fixes. Setup copies itself to
/usr/local/bin/hpcod, so after the first run it is sudo hpcod …. To
build the binary yourself instead: make linux in hpcod/
(LINUX_ARCH=arm64 for arm64), and copy hpcod-linux-amd64 to the host.
setup is the installer. It is for administrators: it refuses to run as
anyone but root, it runs only in a terminal on the control plane host, and
the portal never exposes it.
It is a full-screen program. The stages of an install run down the left and tick off as you go. The screen in hand is on the right, and the keys that work on it are along the bottom: arrows to move, enter to go on, esc to go back.
On a host that is already installed, the first screen offers four things instead of an install:
- check the installation;
- reconfigure and install again;
- build a test cluster;
- rebuild the head node template.
What it asks
-
Where clusters are built. A Proxmox VE you already have, or a simulated one in QEMU on this host (see below).
-
Proxmox. The API address, its certificate, and how to sign in:
- root@pam's password. Setup uses it once, to create two API tokens and to put this host's SSH key on Proxmox. It is not stored.
- Or two tokens you made yourself (see below). In that case you also add this host's SSH key to root on Proxmox yourself; setup shows the key.
Setup signs in before it asks anything else. A wrong password, an untrusted certificate or an address nothing answers on is reported at this point, not twenty minutes later.
With a simulated Proxmox this screen is replaced by three numbers: the cores, memory and disk to give it.
-
The site. On one form, in four groups:
- Proxmox resources: the node and datastores, picked from the lists Proxmox returned; also the cloud image, the SDN zone and the SSH address.
- Names: the portal domain and the cluster domain.
- How users reach the portal: behind your own reverse proxy, or served by hpcod itself with its own CA and DNS (see below).
- Management network and head nodes and images. The defaults for images work.
Each field is checked as you type, including against the others. For example, the build address may not fall inside the head node window. With a simulated Proxmox, the network group is left out, because its network is its own.
-
Review. Everything that will happen, before any of it does. On a reinstall it also asks whether to rebuild the head node template.
What it does
Then it runs every step. The steps tick off in two columns, with the current step's output below them:
| Step | What it does |
|---|---|
| site | records the answers in /etc/hpcod/site.env |
| packages | PostgreSQL, Ansible, and OpenTofu from its own apt repository |
| account | the hpcod service account and its directories |
| binary | installs itself as /usr/local/bin/hpcod |
| keys | the provisioner's SSH key and the client certificate every cluster pins |
| ca | the name-constrained CA, when hpcod serves HTTPS itself |
| database | a local role and database; /etc/hpcod/db.env |
| tree | the roles, Terraform module and schema, from inside the binary, into /var/lib/hpcod |
| schema | applies every migration, then checks the portal's role can read every table |
| catalogue | the application catalogue, into /etc/hpcod/apps.yaml |
| config | sites.yaml, the Terraform variables, and the first portal login |
| unit | the systemd unit, then reads it back through systemd |
| start | starts the portal and waits for /healthz |
| pvesim | the simulated Proxmox, when there is one: its network, its disk, Proxmox VE inside it |
| proxmox | SSH access, both tokens, the SDN zone, snippets and import on the datastores, the cloud image |
| template | builds the head node template if there is none (15 minutes on hardware, longer nested) |
Each step reads back what it changed, and every step is safe to run again.
The portal starts before Proxmox is prepared. If an install stops there,
for example because Proxmox cannot be reached yet, the portal is still
running. The full log goes to /var/log/hpcod-setup.log. The first portal
login is printed once and saved in /root/.hpcod-portal-login.
At the end, setup checks the installation (see Verify), says what to tell the people who will use the portal, and offers to build a test cluster.
Making the tokens yourself
If you would rather not give setup root's password, create the two tokens and pick "API tokens I made myself":
pveum user add hpcod@pve
pveum acl modify / --users hpcod@pve --roles PVEAuditor
pveum user token add hpcod@pve discovery --privsep 0 # discovery token
pveum user token add root@pam hpcod-tf --privsep 0 # Terraform token
The Terraform token is root's because Proxmox allows raw PCI passthrough (how a GPU reaches a compute node) to root and nobody else.
No Proxmox yet: the simulated one
Setup's first question is where clusters are built. "A simulated Proxmox,
in QEMU on this host" runs Proxmox VE in a VM on the control plane host.
It uses a private bridge (10.99.0.0/24) with NAT, so it never touches
your network. Setup installs Proxmox VE into it, fills in every Proxmox and
network value, and continues as with a real one.
The host needs:
/dev/kvm;- nested virtualization, because clusters run inside the simulation;
- room for it: 8 cores, 24 GiB and 150 GiB by default.
On a VM, give it CPU type host. It is for trying hpcod, not for tenants.
A Proxmox cluster
If the Proxmox VE is one node of a Proxmox cluster, you do not add the
other nodes one by one. Discovery reads the whole cluster through one API,
and the tokens setup makes are valid on every node. Setup lists the nodes
and asks which of them clusters may use. Nodes left unticked are never
touched, so a Proxmox cluster shared with other work stays shared.
Unattended, the same choice is PVE_NODES in site.env.
- Heads stay on
PVE_NODE. The head template lives there, and Proxmox cannot clone a template onto another node unless its disk is on shared storage. The head is the only VM with a disk, so this asks for no shared storage at all. The snippets stay there too, for the same reason. - Compute nodes go on any ticked node. They are diskless and PXE boot from their head, so they need nothing on the node but room, and a GPU if they asked for one.
- The SDN zone becomes vxlan. A
simplezone gives every node its own bridge, so a compute node on another node could never reach its head. With more than one node ticked, setup makes the zonevxlan, with every ticked node as a peer and an MTU of 1450 (VXLAN takes 50 bytes of every frame). Guests are given the bridge's MTU through virtio. The nodes must reach each other on UDP 4789.
A zone's type cannot be changed in place. Setup deletes and remakes it only
while no cluster has a vnet in it; otherwise it stops and names the
clusters to release first. Adding or removing a node later only changes the
peers, in place. hpcod verify checks that every node is online and that
the zone is the type, and has the peers, that the nodes need.
PVE_SDN_ZONE_TYPE=vxlan makes a vxlan zone on a single node as well, to
be ready for more nodes without releasing every cluster.
No public domain: its own CA and DNS
On a closed network, choose the following when setup asks how users reach the portal:
- "hpcod serves HTTPS itself, with its own certificate authority".
hpcod listens on 443 at
PUBLIC_IP. Its CA can sign only for your domains. - "hpcod answers DNS for them". hpcod answers for the portal and for
every cluster name on port 53 at
PUBLIC_IP, and forwards other names toNAMESERVERS.
At the end, setup prints what to tell users: use it as a resolver (or
forward the zone to it), and trust http://<portal>/ca.crt once. You don't
need a proxy in this mode.
Put the proxy in front
Only when you chose "behind my own reverse proxy". Copy the two blocks from
deploy/Caddyfile.example into your Caddyfile, set your domains and
wildcard certificate, and reload Caddy. Setup does not touch the proxy; it
usually serves other sites too.
Verify
sudo hpcod verify
Verify checks each fact by using it: the tokens against the API, the
template by asking Proxmox, the database by connecting as the portal does.
It ends with READY or NOT READY, and every failed line names the step
that fixes it. The same check is in the setup menu.
Test cluster
sudo hpcod selftest # 1 compute node; -compute 2 also tests jobs across nodes
This builds the smallest real cluster, by the same path a user's "create" takes: discovery, placement, allocation, then the portal's own builder.
- It waits for Slurm to see the nodes, runs
srun -N <n> hostname, and checks that the cluster's portal is published. - It then takes the cluster down again.
-keepleaves it up. - If the test fails, the cluster is left up for inspection, and selftest
prints the command that removes it (
hpcod selftest -destroy <serial>).
It takes about 25 minutes. The cluster appears on the portal's cluster list
while it exists, under the tenant hpcod-selftest.
Unattended install and updates
From a workstation with the repository, make deploy builds the binary,
copies it and deploy/site.env to DEPLOY_HOST, and runs
hpcod setup -yes there:
cd hpcod
cp deploy/site.env.example deploy/site.env # fill it in
make deploy # LINUX_ARCH=arm64 for an arm64 host
make verify
make selftest
Unattended, setup reads the Proxmox credentials from the host's environment, never from the command line. They are needed only until the tokens exist:
HPCOD_PVE_PASSWORD root@pam's password, or
HPCOD_PVE_DISCOVERY_TOKEN the two tokens
HPCOD_PVE_TERRAFORM_TOKEN
On a first install without them, every step up to the portal start
completes. The Proxmox step then stops and says what it needs. Running
sudo hpcod setup on the host finishes the install.
| When | Do |
|---|---|
After git pull |
make deploy |
| Binary-only change | make deploy-portal |
| The roles changed and setup says the template is older than them | make deploy SETUP_ARGS=-rebuild-template, or the menu entry |
A value in site.env changed |
make deploy; if it was the domain, the images or CI_USER, also rebuild the template |
| Use more nodes of the same Proxmox cluster | sudo hpcod setup, reconfigure, tick them; or PVE_NODES and make deploy |
| Add a second Proxmox site | edit /etc/hpcod/sites.yaml; setup writes it once, then keeps only its own site's nodes and head_node |
| Keep your own application catalogue | create /etc/hpcod/apps.local; setup then leaves apps.yaml alone |
Back up these three things:
- the database:
sudo -u postgres pg_dump hpcod /etc/hpcod: tokens, the portal's users, and the client certificate every cluster pins/var/lib/hpcod/terraform/envs: one Terraform state per cluster, and the only record of which VMs belong to which cluster
Known limits
- Only a /24 management network. The head node prefix is fixed at 24.
- Heads run on one node.
PVE_NODEholds the template, the snippets and every head; only compute nodes spread over a Proxmox cluster. - Only Debian and Ubuntu control plane hosts. Setup installs with apt.
On other distributions, install PostgreSQL, Ansible and OpenTofu yourself
and run
hpcod setup -yes -skip packages. - No TLS automation. The wildcard certificate is yours to obtain and renew.
- Host keys are not checked. Setup does not check SSH host keys on the hypervisor or on freshly cloned heads. The Terraform provider and the builds already behaved this way.



