The repository had no license, so it was public but not open source: nobody could use, change or redistribute it. LICENSE is the text from apache.org, unmodified. NOTICE says that the IBM Plex fonts embedded in the print handbook are under the SIL Open Font License, not Apache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
10 KiB
hpcod — HPC-on-Demand control plane
Phase 1 of the architecture: discovery. It reads the backing clouds, translates their inventory into a provider-neutral vocabulary, and reports what could actually be scheduled.
Nothing here writes to a site.
Why the provider abstraction comes first
The portal is meant to sit over more than Proxmox — OpenStack is expected next.
So internal/provider defines the interface, internal/model defines the
vocabulary everything above it reasons in, and Proxmox is simply the first
implementation. Adding a provider means writing one package and registering it;
nothing above internal/provider changes.
cmd/hpcod CLI
internal/model provider-neutral inventory types
internal/provider Provider interface, registry, site config
internal/provider/proxmox first implementation
internal/capability PCI identity -> scheduling vocabulary
internal/inventory catalogue loading, concurrent discovery
The thing this code exists to get right
A PCI class code cannot identify InfiniBand.
A ConnectX-4 Lx reports class 0x0200 (Ethernet) because that variant is
Ethernet-only. A ConnectX-6 reports 0x0207 in IB mode and 0x0200 in
Ethernet mode. So class-based detection produces false negatives and false
positives, and a portal that trusts it will eventually hand a tenant a cluster
whose fabric is not up.
Detection is therefore layered, and every claim carries how it was made:
| Confidence | Source | Trusted to schedule on |
|---|---|---|
class |
PCI class code | GPU, NVMe — yes. Fabrics — no. |
map |
curated vendor:device table |
GPU, NVMe — yes. Fabrics — no. |
agent |
live check on the node | yes |
model.Capability.Trusted enforces it: a fabric capability is never
schedulable until an agent has confirmed the link. Unconfirmed devices are
still reported, because that difference is what an operator needs to fix.
Presence is not enough either. A GPU is offerable only if its IOMMU group
holds nothing else that matters, so Passthrough is derived from group
membership — sibling functions of the same card are fine, an unrelated device
in the group is not.
Use
cp config/sites.yaml.example config/sites.yaml # then edit
export HV01_TOKEN='hpcod@pve!discovery=<uuid>'
go run ./cmd/hpcod sites # configured, reachable?
go run ./cmd/hpcod discover # full inventory
go run ./cmd/hpcod capacity # what could be scheduled now
go run ./cmd/hpcod capmap # the curated table, and why each row exists
go run ./cmd/hpcod discover -json | jq .
Secrets are never written in the catalogue: any setting may be env:NAME and
is read from the environment at startup.
The Proxmox token needs read access only — create it against PVEAuditor:
pveum user add hpcod@pve --comment "HPC-on-Demand discovery"
pveum acl modify / --users hpcod@pve --roles PVEAuditor
pveum user token add hpcod@pve discovery --privsep 0
Sample output
=== hv01 (Lab cluster, proxmox)
node hv01 (online) 28 cores (0 free), 504 GiB RAM (52 GiB free)
cpu: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
gpu 0000:01:00.0 SCHEDULABLE via class GM107GL [Quadro K620]
roce 0000:07:00.0 unconfirmed via map MT27710 Family [ConnectX-4 Lx]
roce 0000:07:00.1 unconfirmed via map MT27710 Family [ConnectX-4 Lx]
The Mellanox cards are reported as roce, not infiniband, and withheld from
the schedulable set — which is the whole point.
Capacity accounting
Free capacity is total minus what running guests have already been allocated,
not raw hardware totals. Physical cores are the schedulable total by default
(cpu_overcommit: "1.0"): an HPC tenant handed oversubscribed cores simply
runs slowly. Raise it per site only where that is acceptable.
Phase 2 — placement
placement.Plan(sites, request, weights) is a pure function: no I/O, no
cloud. That is what makes the decision reproducible, unit-testable against
synthetic multi-site fixtures, and usable as a dry-run endpoint later.
hpcod plan -f examples/gpu-request.yaml # against live sites
hpcod plan -f examples/gpu-request.yaml -inventory snap.json # against a snapshot
hpcod discover -json > snap.json # take a snapshot
Two phases, in this order. Filter eliminates the impossible and records why per site; score ranks the survivors.
score = 3.0 * capability_fit // penalise consuming scarce hardware the
// request did not ask for
+ 1.0 * free_capacity // prefer emptier sites
+ 0.8 * packing // fewer hypervisors per tenant
- 1.2 * fragmentation // avoid unusable remainders
capability_fit dominates deliberately. Scarce hardware must be kept for
requests that need it: a plain CPU job placed on the only InfiniBand site is
a scheduling failure even though it fits.
What the refusals look like
The point of recording reasons is that "no capacity" is useless while this is actionable:
all candidates:
site-a feasible, score 4.40
site-b rejected
- gpu: none present (need 4)
site-c rejected
- infiniband: 1 present but unconfirmed - the fabric has not been
verified by a node agent, so it is not schedulable (need 4, confirmed 0)
site-d rejected
- site is unreachable: dial tcp 10.0.0.40:8006: no route to host
Three different problems, three different fixes: buy hardware, run the agent, fix the network.
Memory is not what the user typed
A compute node unpacks its root filesystem into RAM, so memory_gb in a
request is usable memory and the portal adds RootfsReserveGB on top. A
tenant that asked for 32 GiB and was allocated exactly 32 GiB would OOM under
its own job.
Installing it
On a Debian 13 or Ubuntu 24.04 host:
V=v0.2.2 A=linux-amd64 # A=linux-arm64 on an arm64 host
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/hpcod-$V-$A
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/SHA256SUMS
sha256sum -c --ignore-missing SHA256SUMS
install -m 0755 hpcod-$V-$A hpcod
sudo ./hpcod setup
That runs the installer, a full-screen TUI for administrators. It installs
the binary as /usr/local/bin/hpcod, so afterwards:
sudo hpcod verify # can this installation build a cluster?
sudo hpcod selftest # build a real cluster, run a job on it, take it down
The binary carries the Ansible roles, the Terraform module, the schema and the application catalogue; setup installs everything else. It installs against a Proxmox VE you have, or against one it simulates in QEMU on the same host. See DEPLOY.md.
The portal
hpcod serve is the portal; setup installs it as the hpcod-portal
service. It serves:
- sites and their devices;
- the plan and create form;
- the cluster list, with build logs as they happen;
- projects and usage;
- the curated device map;
- account approval;
- each cluster's own portal at
c<NNN>.<domain>; - a JSON API (
/v1/sites,/v1/capacity,/v1/dry-run,/v1/clusters).
It listens on 127.0.0.1:9880, and is reached in one of two ways:
- Behind your reverse proxy. The proxy is then the only thing that can reach it, and it refuses to bind off loopback without a user file.
- On a closed network, serving HTTPS on 443 itself. Its certificate comes from the installation's own name-constrained CA. It can also answer DNS for its names.
Discovery runs on a timer rather than per request. Polling every backing cloud on each page load would hammer their APIs and make the portal as slow as the slowest site, so pages show how old the inventory is instead.
Phase 3 — Terraform
terraform/modules/tenant builds one tenant: an isolated bridge, a head node
with a management and a cluster NIC, and N diskless compute nodes that PXE boot
from it. Applied and destroyed cleanly against a real Proxmox.
cd terraform/envs/lab
cp terraform.tfvars.example terraform.tfvars # then edit
tofu init && tofu plan && tofu apply
MACs are derived, not generated: BC:24:11:E1:<tenant>:<index>. Warewulf
matches nodes by MAC, so a MAC that changed on re-apply would orphan the node's
registration and it would never boot again.
The E1 prefix is not arbitrary. The hand-built cluster on this hypervisor
already uses BC:24:11:E0: with 00 and 0A in the next byte, so a tenant
with id 10 would have produced a duplicate MAC on the same L2 segment and
broken the running cluster. Tenants get their own prefix so the two estates
cannot collide:
| hand-built lab | hpcod tenants | |
|---|---|---|
| bridge | vmbr6 |
vmbrT001–vmbrT254 |
| subnet | 10.10.10.0/24 |
10.64.<tenant>.0/24 |
| VMID | 300–304 | 9001+ |
| MAC | BC:24:11:E0:* |
BC:24:11:E1:* |
The bridge is not the provider's resource
proxmox_network_linux_bridge writes the stanza and then reads the live
interface back before anything has applied it, so creation fails. Proxmox
applies pending network changes with ifreload -a, which reloads every
interface — on a hypervisor carrying other people's guests that is a bad
trade, and on any host with one unrelated broken stanza it returns non-zero
and takes the new bridge down with it.
So the bridge is managed with an explicit remote-exec that appends the
stanza, runs ifquery -a to prove the file still parses, and brings up only
the new interface with ifup. It is idempotent, so an interrupted run
recovers by re-running, and the destroy path verifies the interface is really
gone — a leaked bridge is a leaked subnet.
Two things this needs from Proxmox
- The API token's role needs
SDN.Use, or attaching a VM to the new bridge fails with a 403 that does not mention SDN anywhere obvious. - The cloud image must live in a datastore with the
importcontent type (/var/lib/vz/import/, referenced aslocal:import/...). An image sitting in the ISO directory is rejected with "unable to parse directory volume name".
Not built yet
Phase 4 onward: Ansible roles ported from scripts/, the API server, leases
and the reconciler. See
docs/20 §11.
License
Apache License 2.0. The fonts embedded in the print handbook are under the SIL Open Font License; see NOTICE.