Files
ilkerandClaude Opus 5.5 3880311e36
ci / derle (push) Successful in 22s
hpcod: changelog for v0.2.2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 20:22:09 +01:00

11 KiB
Raw Permalink Blame History

Changelog

v0.2.2 — 2026-11-16

Clusters across the nodes of a Proxmox cluster

A Proxmox VE that is one node of a Proxmox cluster no longer means one node. Setup lists the cluster's nodes and asks which ones clusters may use (PVE_NODES); the rest are never touched. Compute nodes are spread over the chosen nodes. Heads stay on PVE_NODE, where the template is, so no shared storage is needed. With more than one node the SDN zone becomes vxlan, with every node a peer and an MTU of 1450 that guests are given through virtio; setup remakes a simple zone only while no cluster uses it, and hpcod verify checks the nodes and the zone. Before this, discovery already saw every node and placement could pick any of them, while the template, the snippets and a simple zone only worked on one: a cluster could land where it could not be built.

Run on the simulated Proxmox with the zone forced to vxlan: a two-node test cluster built and ran a job, both compute NICs came up at 1450, and a full 1450-byte frame passed between them while 1451 did not. Traffic between two real Proxmox nodes has not been tested; the lab has one.

Also

  • Setup's step list keeps to its columns on narrow terminals, and the check says "1 warning", not "1 warnings".
  • The documented install works on a fresh host: it now downloads the release, checks SHA256SUMS, sets the execute bit and runs sudo ./hpcod setup. Before, it said sudo hpcod setup, which is not found until setup has installed itself.
  • The documentation was brought in line with what is built: section 02 installs hpcod from scratch, and the hand-built lab moved to section 10.

v0.2.1 — 2026-11-16

A rebuild of v0.2.0 with Go 1.27.4, for GO-2026-6859: a race in net/http's MaxBytesReader that can crash an HTTP/1 server, which the portal is. The v0.2.0 binaries were built with 1.27.3. Nothing else changes for anyone running hpcod; x/crypto moves to v0.59.0 (GO-2026-5932, which hpcod does not call), and CI takes the latest Go patch instead of a cached one.

v0.2.0 — 2026-11-16

One binary installs the control plane

Until now an installation was this one. The deploy scripts and the Ansible defaults carried one installation's domain, hypervisor address, proxy host and login account. The Proxmox side, the database and the Terraform variables had been set up by hand. Deploying meant copying shell scripts and a tree of roles beside the binary, and keeping all of them in step.

Now the binary is the whole of it:

  • hpcod setup is the installer, for administrators only: root, in a terminal on the control plane host. It is a full-screen TUI with the stages of an install down the side. It asks first where clusters are built, then signs in to Proxmox before asking anything else, with root's password (used once, never stored) or tokens you made yourself. Nodes and datastores are picked from what Proxmox has. Every field is checked as it is typed, including against the others. Then it installs, showing each step's output live, checks the result, and offers a test cluster.
  • hpcod setup -yes runs the same steps unattended, from a site file. make deploy copies the binary and deploy/site.env to the host and runs it there. Nothing else is sent.
  • The binary carries the tree. The Ansible roles, the Terraform module, the schema and the application catalogue are embedded (embed.go). A host always runs the roles and schema of the binary it runs. A test checks that nothing local leaks into the embedded tree.
  • hpcod verify checks each fact by using it: the tokens against the API, the template by asking Proxmox, the database as the portal connects, HTTPS against the installation's CA, DNS as a user's resolver would ask. It reports READY, or which step fixes what failed.
  • hpcod selftest builds the smallest real cluster by the portal's own path (placement, allocation, the portal's builder). It waits for Slurm, runs srun across the nodes, checks the portal is published, and takes the cluster down. A failed test leaves the cluster up and prints how to remove it.

Closed networks

For an installation with no public domain, DNS or certificate:

  • Its own CA. SERVE_MODE=direct makes hpcod serve HTTPS itself on 443, with a CA setup creates once. The CA is name-constrained to the installation's domains, so trusting it trusts nothing else. The portal issues its own server certificate from it in memory and renews it before it expires. Users fetch the CA at http://<portal>/ca.crt.
  • Its own DNS. SERVE_DNS=true answers for the portal and every cluster's name, and forwards everything else. With no forwarders it refuses, so it is never an open resolver.

A simulated Proxmox, to try it

PVE_MODE=sim runs Proxmox VE in QEMU/KVM on the control plane host itself, on a private bridge with NAT. Setup installs Proxmox VE into a Debian 13 cloud image and fills in every Proxmox and network value. Clusters run inside it, so the host needs nested virtualization. It is for trying hpcod, not for tenants.

What was proven, and what was not

  • End to end, on a simulated Proxmox (a control plane VM on a real Proxmox host, with Proxmox VE 9 inside it):
    • the install, from the single binary;
    • the Proxmox step by the root-password path: SSH key, both tokens, the SDN zone, the content types, the cloud image;
    • the head template (47 tasks, none failed);
    • hpcod verify READY, with HTTPS from its own CA and its own DNS;
    • hpcod selftest: c001 built in about ten minutes, its compute node idle in Slurm, srun hostname answered from it, then destroyed and released.
  • On a clean Debian 13 host: the install twice in a row, with nothing changed the second time. The generated password signs in, and the TUI was driven through a terminal.
  • Not yet run:
    • setup against a bare-metal Proxmox host;
    • the token path, which was tested only against a fake Proxmox API;
    • a fresh simulated Proxmox with the last two fixes to it (resolv.conf, and a clean power-down). Both were verified on the running simulation only.

Changes an existing installation should know about

  • The Proxmox token moves from site.auto.tfvars to secrets.auto.tfvars. Setup moves it the first time and keeps a backup.
  • ci_user and the portal's -ssh-user no longer default to a personal account. Setup writes both from CI_USER.
  • ansible/group_vars/all.yml is now group_vars/all/main.yml. The site's values are in a generated site.yml beside it, and the playbooks refuse to run without it. The head template only picks them up when it is rebuilt (-rebuild-template). Until then, setup and verify say it is stale.
  • The catalogue now comes from inside the binary. Publishing an application means rebuilding and deploying the binary. A site that keeps its own catalogue creates /etc/hpcod/apps.local.
  • Removed: scripts/B10, B20 and C20. Their work is the setup steps. /var/lib/hpcod/scripts on an existing host is no longer used.

v0.1.0 — 2026-09-21

First tagged release. The repository holds two things: the Warewulf HPC lab documented in sections 01–09, and HPC-on-Demand (hpcod/), a control plane that provisions clusters of that kind on demand. This release is about the second.

The chain that works end to end

Someone signs up on the public site, an administrator approves them, and from there the portal takes a request for a cluster through to a running desktop with a job on it. Every step below was exercised on hv01, not inferred:

  1. Sign up, and wait. New accounts land in pending; nobody gets in until an administrator approves them. Disabling an account takes effect on the next request, not when the session happens to expire.
  2. Ask for hardware, not for a flavour. A request names capabilities — GPU, InfiniBand, RoCE, NVMe, cores, memory — or just names the software it wants and lets the application catalogue imply the hardware.
  3. Placement picks a site and a node. Devices already handed to another cluster are not offered again, and neither are devices reserved for a cluster that has been allocated but not yet built. Oversubscription is a policy an operator sets from the portal, with separate CPU and memory ratios; turning it off does not lose them.
  4. Allocation is atomic. Cluster ids, management addresses and PCI devices come from PostgreSQL under an advisory lock, so two requests racing for the last GPU produce one cluster and one clear refusal.
  5. Terraform builds it. An SDN vnet for the cluster fabric, a head node and compute VMs, with the placement carried straight through from the record. Teardown removes them in an order that actually works.
  6. Ansible finishes it. Warewulf 4.7, OpenHPC 3, Slurm 25.11, MUNGE, Apptainer, shared /opt/apps, and Open OnDemand on the head node.
  7. The cluster publishes itself. It appears at c<NNN>.<domain>, behind the control plane, with the portal's certificate pinned at the moment it was published.
  8. One sign-in, not two. The cluster portal trusts the control plane over mTLS, so a session that may use a cluster reaches it without another password. A session that may not gets nothing, and a hostname for a cluster that does not exist says so instead of showing the front page.
  9. A desktop, a terminal, and a job. XFCE over TurboVNC in a container, a shell on the head node, and srun from inside either.

Measured

  • Cold provision of a tenant cluster: ~134 s.
  • Node image build: ~12 min, done once and shared, not per tenant.

Known limits

These are absent by decision or by circumstance, not by oversight:

  • Leases are recorded but not enforced. store.Expired returns what has run out; nothing acts on it yet. No reaper.
  • One Proxmox node. Placement discriminates between sites and between hosts, and the Terraform module takes a per-VM host — but hv01 is a single node, so multi-site placement is tested against fixtures rather than metal.
  • No central Slurm accounting. There is no slurmdbd, so jobs carry no cluster name and sacct history is per-cluster.
  • Whole-device GPU passthrough only. MIG and vGPU are out of scope.
  • /home only. No scratch, no parallel filesystem.
  • Nothing bills. The allocation record is the raw material for it and is written; no code reads it for that purpose.
  • The API in §20.9 is partly implemented. The portal and CLI cover what is built; the REST surface does not yet.

Operating notes

  • hpcod version now reports the release tag alongside the revision, e.g. v0.1.0 (f6ea30dc1234). A binary built outside the Makefile prints the revision alone rather than guessing.
  • The control plane needs $HPCOD_DB_DSN or -db; the CLI and the service resolve it the same way.
  • make deploy-portal builds for Linux and installs it on the Caddy host.