How someone else installs hpcod: the binary, sudo hpcod setup, what it asks and what each step does, the simulated Proxmox, a closed network with its own CA and DNS, verify, selftest, and what to back up. The changelog says what was proven and what was not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
10 KiB
Deploying hpcod
How to install the HPC-on-Demand control plane against your own Proxmox VE.
You need one file, the hpcod binary, and about an hour, most of it spent
waiting for the head node template to build.
The binary carries everything a control plane needs: the Ansible roles, the Terraform module, the database schema and the application catalogue. It installs itself. There are no scripts to copy alongside it.
What you end up with
browser ──https──▶ Caddy ──▶ hpcod (127.0.0.1:9880) ──▶ PostgreSQL
│ │
c001.<domain> ──┘ ├──tofu────▶ Proxmox API: head + compute VMs, SDN vnet
└──ssh─────▶ head nodes (archive /home on release)
- Control plane host: a Debian 13 or Ubuntu 24.04 VM or machine. It runs hpcod, PostgreSQL, OpenTofu and Ansible, behind a reverse proxy.
- Proxmox VE 8 or 9: where the clusters are built. Each cluster gets a head node cloned from a template, compute nodes that PXE boot from it, and an SDN vnet of its own.
- Per-cluster portals: each cluster runs Open OnDemand at
c<NNN>.<CLUSTER_DOMAIN>, reached through the control plane.
Before you start
| You need | Why |
|---|---|
| A Debian 13 or Ubuntu 24.04 host with root, 2 vCPU, 4 GB RAM, 20 GB disk | the control plane |
| Proxmox VE 8 or later, and root's password (or two API tokens, see below) | setup creates the tokens and lets this host in over SSH |
| A /24 management network shared by the control plane and Proxmox | head nodes get addresses on it |
| About 40 free addresses on that network, outside DHCP, plus one more | one per live cluster, and one for the template build |
DNS: an A record for PORTAL_DOMAIN and a wildcard *.CLUSTER_DOMAIN, both pointing at the proxy |
cluster portals are subdomains |
A wildcard TLS certificate for *.CLUSTER_DOMAIN |
needs DNS-01; see deploy/Caddyfile.example |
| Internet access from the control plane and from head nodes | packages, OpenTofu providers, and the images |
The node, desktop and application images come from a public registry by default and pull anonymously. You do not need to build any images.
Install
Get the binary for the host: a release from
https://gitea.mynodes.xyz/ilker/werewulf/releases, or make linux in
hpcod/ (LINUX_ARCH=arm64 for an arm64 host). Copy it to the host, then:
sudo ./hpcod setup
setup is the installer. It is for administrators: it refuses to run as
anyone but root, it runs only in a terminal on the control plane host, and
the portal never exposes it.
It asks for the following, in order:
-
Proxmox. The API address, and how to sign in:
- root@pam's password. Setup uses it once, to create two API tokens and to put this host's SSH key on Proxmox. It is not stored.
- Or two tokens you made yourself (see below). In that case you also add this host's SSH key to root on Proxmox yourself; setup shows the key.
Setup signs in before it asks anything else. A wrong password, an untrusted certificate or an address nothing answers on is reported at this point, not twenty minutes later.
-
Proxmox resources. The node and the datastores, chosen from the lists Proxmox returned. Also the cloud image, the SDN zone, and the SSH address.
-
Names. The portal domain and the cluster domain.
-
Management network. Bridge, gateway, the head node address window, a build address, and DNS servers. Each field is checked as you enter it, including against the others: for example, the build address may not fall inside the head node window.
-
Head nodes and images. The defaults work.
Then it runs every step and shows the output as it goes:
| Step | What it does |
|---|---|
| site | records the answers in /etc/hpcod/site.env |
| packages | PostgreSQL, Ansible, and OpenTofu from its own apt repository |
| account | the hpcod service account and its directories |
| binary | installs itself as /usr/local/bin/hpcod |
| keys | the provisioner's SSH key and the client certificate every cluster pins |
| database | a local role and database; /etc/hpcod/db.env |
| tree | the roles, Terraform module and schema, from inside the binary, into /var/lib/hpcod |
| schema | applies every migration, then checks the portal's role can read every table |
| catalogue | the application catalogue, into /etc/hpcod/apps.yaml |
| config | sites.yaml, the Terraform variables, and the first portal login |
| unit | the systemd unit, then reads it back through systemd |
| start | starts the portal and waits for /healthz |
| proxmox | SSH access, both tokens, the SDN zone, snippets and import on the datastores, the cloud image |
| template | builds the head node template if there is none (about 15 minutes) |
Each step reads back what it changed, and every step is safe to run again.
The full log goes to /var/log/hpcod-setup.log. The first portal login is
printed once and saved in /root/.hpcod-portal-login.
At the end, setup checks the installation (see Verify) and offers to build a test cluster.
Making the tokens yourself
If you would rather not give setup root's password, create the two tokens and pick "API tokens I made myself":
pveum user add hpcod@pve
pveum acl modify / --users hpcod@pve --roles PVEAuditor
pveum user token add hpcod@pve discovery --privsep 0 # discovery token
pveum user token add root@pam hpcod-tf --privsep 0 # Terraform token
The Terraform token is root's because Proxmox allows raw PCI passthrough (how a GPU reaches a compute node) to root and nobody else.
No Proxmox yet: the simulated one
Setup's first question is where clusters are built. "A simulated Proxmox,
in QEMU on this host" runs Proxmox VE in a VM on the control plane host.
It uses a private bridge (10.99.0.0/24) with NAT, so it never touches
your network. Setup installs Proxmox VE into it, fills in every Proxmox and
network value, and continues as with a real one.
The host needs:
/dev/kvm;- nested virtualization, because clusters run inside the simulation;
- room for it: 8 cores, 24 GiB and 150 GiB by default.
On a VM, give it CPU type host. It is for trying hpcod, not for tenants.
No public domain: its own CA and DNS
On a closed network, choose the following when setup asks how users reach the portal:
- "hpcod serves HTTPS itself, with its own certificate authority".
hpcod listens on 443 at
PUBLIC_IP. Its CA can sign only for your domains. - "hpcod answers DNS for them". hpcod answers for the portal and for
every cluster name on port 53 at
PUBLIC_IP, and forwards other names toNAMESERVERS.
At the end, setup prints what to tell users: use it as a resolver (or
forward the zone to it), and trust http://<portal>/ca.crt once. You don't
need a proxy in this mode.
Put the proxy in front
Copy the two blocks from deploy/Caddyfile.example into your Caddyfile, set
your domains and wildcard certificate, and reload Caddy. Setup does not touch
the proxy; it usually serves other sites too.
Verify
sudo hpcod verify
Verify checks each fact by using it: the tokens against the API, the
template by asking Proxmox, the database by connecting as the portal does.
It ends with READY or NOT READY, and every failed line names the step
that fixes it. The same check is in the setup menu.
Test cluster
sudo hpcod selftest # 1 compute node; -compute 2 also tests jobs across nodes
This builds the smallest real cluster, by the same path a user's "create" takes: discovery, placement, allocation, then the portal's own builder.
- It waits for Slurm to see the nodes, runs
srun -N <n> hostname, and checks that the cluster's portal is published. - It then takes the cluster down again.
-keepleaves it up. - If the test fails, the cluster is left up for inspection, and selftest
prints the command that removes it (
hpcod selftest -destroy <serial>).
It takes about 25 minutes. The cluster appears on the portal's cluster list
while it exists, under the tenant hpcod-selftest.
Unattended install and updates
From a workstation with the repository, make deploy builds the binary,
copies it and deploy/site.env to DEPLOY_HOST, and runs
hpcod setup -yes there:
cd hpcod
cp deploy/site.env.example deploy/site.env # fill it in
make deploy # LINUX_ARCH=arm64 for an arm64 host
make verify
make selftest
Unattended, setup reads the Proxmox credentials from the host's environment, never from the command line. They are needed only until the tokens exist:
HPCOD_PVE_PASSWORD root@pam's password, or
HPCOD_PVE_DISCOVERY_TOKEN the two tokens
HPCOD_PVE_TERRAFORM_TOKEN
On a first install without them, every step up to the portal start
completes. The Proxmox step then stops and says what it needs. Running
sudo hpcod setup on the host finishes the install.
| When | Do |
|---|---|
After git pull |
make deploy |
| Binary-only change | make deploy-portal |
| The roles changed and setup says the template is older than them | make deploy SETUP_ARGS=-rebuild-template, or the menu entry |
A value in site.env changed |
make deploy; if it was the domain, the images or CI_USER, also rebuild the template |
| Add a second Proxmox site | edit /etc/hpcod/sites.yaml; setup writes it once and never again |
| Keep your own application catalogue | create /etc/hpcod/apps.local; setup then leaves apps.yaml alone |
Back up these three things:
- the database:
sudo -u postgres pg_dump hpcod /etc/hpcod: tokens, the portal's users, and the client certificate every cluster pins/var/lib/hpcod/terraform/envs: one Terraform state per cluster, and the only record of which VMs belong to which cluster
Known limits
- Only a /24 management network. The head node prefix is fixed at 24.
- One default Proxmox node per installation.
PVE_NODEis where snippets and the template live. - Only Debian and Ubuntu control plane hosts. Setup installs with apt.
On other distributions, install PostgreSQL, Ansible and OpenTofu yourself
and run
hpcod setup -yes -skip packages. - No TLS automation. The wildcard certificate is yours to obtain and renew.
- Host keys are not checked. Setup does not check SSH host keys on the hypervisor or on freshly cloned heads. The Terraform provider and the builds already behaved this way.