Files
werewulf/hpcod/README.md
T
ilkerandClaude Opus 5.5 3880311e36
ci / derle (push) Successful in 22s
hpcod: changelog for v0.2.2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 20:22:09 +01:00

10 KiB
Raw Permalink Blame History

hpcod — HPC-on-Demand control plane

Phase 1 of the architecture: discovery. It reads the backing clouds, translates their inventory into a provider-neutral vocabulary, and reports what could actually be scheduled.

Nothing here writes to a site.

Why the provider abstraction comes first

The portal is meant to sit over more than Proxmox — OpenStack is expected next. So internal/provider defines the interface, internal/model defines the vocabulary everything above it reasons in, and Proxmox is simply the first implementation. Adding a provider means writing one package and registering it; nothing above internal/provider changes.

cmd/hpcod                CLI
internal/model           provider-neutral inventory types
internal/provider        Provider interface, registry, site config
internal/provider/proxmox   first implementation
internal/capability      PCI identity -> scheduling vocabulary
internal/inventory       catalogue loading, concurrent discovery

The thing this code exists to get right

A PCI class code cannot identify InfiniBand.

A ConnectX-4 Lx reports class 0x0200 (Ethernet) because that variant is Ethernet-only. A ConnectX-6 reports 0x0207 in IB mode and 0x0200 in Ethernet mode. So class-based detection produces false negatives and false positives, and a portal that trusts it will eventually hand a tenant a cluster whose fabric is not up.

Detection is therefore layered, and every claim carries how it was made:

Confidence Source Trusted to schedule on
class PCI class code GPU, NVMe — yes. Fabrics — no.
map curated vendor:device table GPU, NVMe — yes. Fabrics — no.
agent live check on the node yes

model.Capability.Trusted enforces it: a fabric capability is never schedulable until an agent has confirmed the link. Unconfirmed devices are still reported, because that difference is what an operator needs to fix.

Presence is not enough either. A GPU is offerable only if its IOMMU group holds nothing else that matters, so Passthrough is derived from group membership — sibling functions of the same card are fine, an unrelated device in the group is not.

Use

cp config/sites.yaml.example config/sites.yaml   # then edit
export HV01_TOKEN='hpcod@pve!discovery=<uuid>'

go run ./cmd/hpcod sites       # configured, reachable?
go run ./cmd/hpcod discover    # full inventory
go run ./cmd/hpcod capacity    # what could be scheduled now
go run ./cmd/hpcod capmap      # the curated table, and why each row exists
go run ./cmd/hpcod discover -json | jq .

Secrets are never written in the catalogue: any setting may be env:NAME and is read from the environment at startup.

The Proxmox token needs read access only — create it against PVEAuditor:

pveum user add hpcod@pve --comment "HPC-on-Demand discovery"
pveum acl modify / --users hpcod@pve --roles PVEAuditor
pveum user token add hpcod@pve discovery --privsep 0

Sample output

=== hv01 (Lab cluster, proxmox)
  node hv01 (online)  28 cores (0 free), 504 GiB RAM (52 GiB free)
       cpu: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
       gpu         0000:01:00.0 SCHEDULABLE via class  GM107GL [Quadro K620]
       roce        0000:07:00.0 unconfirmed via map    MT27710 Family [ConnectX-4 Lx]
       roce        0000:07:00.1 unconfirmed via map    MT27710 Family [ConnectX-4 Lx]

The Mellanox cards are reported as roce, not infiniband, and withheld from the schedulable set — which is the whole point.

Capacity accounting

Free capacity is total minus what running guests have already been allocated, not raw hardware totals. Physical cores are the schedulable total by default (cpu_overcommit: "1.0"): an HPC tenant handed oversubscribed cores simply runs slowly. Raise it per site only where that is acceptable.

Phase 2 — placement

placement.Plan(sites, request, weights) is a pure function: no I/O, no cloud. That is what makes the decision reproducible, unit-testable against synthetic multi-site fixtures, and usable as a dry-run endpoint later.

hpcod plan -f examples/gpu-request.yaml                       # against live sites
hpcod plan -f examples/gpu-request.yaml -inventory snap.json  # against a snapshot
hpcod discover -json > snap.json                              # take a snapshot

Two phases, in this order. Filter eliminates the impossible and records why per site; score ranks the survivors.

score = 3.0 * capability_fit    // penalise consuming scarce hardware the
                                //   request did not ask for
      + 1.0 * free_capacity     // prefer emptier sites
      + 0.8 * packing           // fewer hypervisors per tenant
      - 1.2 * fragmentation     // avoid unusable remainders

capability_fit dominates deliberately. Scarce hardware must be kept for requests that need it: a plain CPU job placed on the only InfiniBand site is a scheduling failure even though it fits.

What the refusals look like

The point of recording reasons is that "no capacity" is useless while this is actionable:

all candidates:
  site-a       feasible, score 4.40
  site-b       rejected
       - gpu: none present (need 4)
  site-c       rejected
       - infiniband: 1 present but unconfirmed - the fabric has not been
         verified by a node agent, so it is not schedulable (need 4, confirmed 0)
  site-d       rejected
       - site is unreachable: dial tcp 10.0.0.40:8006: no route to host

Three different problems, three different fixes: buy hardware, run the agent, fix the network.

Memory is not what the user typed

A compute node unpacks its root filesystem into RAM, so memory_gb in a request is usable memory and the portal adds RootfsReserveGB on top. A tenant that asked for 32 GiB and was allocated exactly 32 GiB would OOM under its own job.

Installing it

On a Debian 13 or Ubuntu 24.04 host:

V=v0.2.2 A=linux-amd64        # A=linux-arm64 on an arm64 host
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/hpcod-$V-$A
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/SHA256SUMS
sha256sum -c --ignore-missing SHA256SUMS
install -m 0755 hpcod-$V-$A hpcod
sudo ./hpcod setup

That runs the installer, a full-screen TUI for administrators. It installs the binary as /usr/local/bin/hpcod, so afterwards:

sudo hpcod verify         # can this installation build a cluster?
sudo hpcod selftest       # build a real cluster, run a job on it, take it down

The binary carries the Ansible roles, the Terraform module, the schema and the application catalogue; setup installs everything else. It installs against a Proxmox VE you have, or against one it simulates in QEMU on the same host. See DEPLOY.md.

The portal

hpcod serve is the portal; setup installs it as the hpcod-portal service. It serves:

  • sites and their devices;
  • the plan and create form;
  • the cluster list, with build logs as they happen;
  • projects and usage;
  • the curated device map;
  • account approval;
  • each cluster's own portal at c<NNN>.<domain>;
  • a JSON API (/v1/sites, /v1/capacity, /v1/dry-run, /v1/clusters).

It listens on 127.0.0.1:9880, and is reached in one of two ways:

  • Behind your reverse proxy. The proxy is then the only thing that can reach it, and it refuses to bind off loopback without a user file.
  • On a closed network, serving HTTPS on 443 itself. Its certificate comes from the installation's own name-constrained CA. It can also answer DNS for its names.

Discovery runs on a timer rather than per request. Polling every backing cloud on each page load would hammer their APIs and make the portal as slow as the slowest site, so pages show how old the inventory is instead.

Phase 3 — Terraform

terraform/modules/tenant builds one tenant: an isolated bridge, a head node with a management and a cluster NIC, and N diskless compute nodes that PXE boot from it. Applied and destroyed cleanly against a real Proxmox.

cd terraform/envs/lab
cp terraform.tfvars.example terraform.tfvars   # then edit
tofu init && tofu plan && tofu apply

MACs are derived, not generated: BC:24:11:E1:<tenant>:<index>. Warewulf matches nodes by MAC, so a MAC that changed on re-apply would orphan the node's registration and it would never boot again.

The E1 prefix is not arbitrary. The hand-built cluster on this hypervisor already uses BC:24:11:E0: with 00 and 0A in the next byte, so a tenant with id 10 would have produced a duplicate MAC on the same L2 segment and broken the running cluster. Tenants get their own prefix so the two estates cannot collide:

hand-built lab hpcod tenants
bridge vmbr6 vmbrT001–vmbrT254
subnet 10.10.10.0/24 10.64.<tenant>.0/24
VMID 300–304 9001+
MAC BC:24:11:E0:* BC:24:11:E1:*

The bridge is not the provider's resource

proxmox_network_linux_bridge writes the stanza and then reads the live interface back before anything has applied it, so creation fails. Proxmox applies pending network changes with ifreload -a, which reloads every interface — on a hypervisor carrying other people's guests that is a bad trade, and on any host with one unrelated broken stanza it returns non-zero and takes the new bridge down with it.

So the bridge is managed with an explicit remote-exec that appends the stanza, runs ifquery -a to prove the file still parses, and brings up only the new interface with ifup. It is idempotent, so an interrupted run recovers by re-running, and the destroy path verifies the interface is really gone — a leaked bridge is a leaked subnet.

Two things this needs from Proxmox

  • The API token's role needs SDN.Use, or attaching a VM to the new bridge fails with a 403 that does not mention SDN anywhere obvious.
  • The cloud image must live in a datastore with the import content type (/var/lib/vz/import/, referenced as local:import/...). An image sitting in the ISO directory is rejected with "unable to parse directory volume name".

Not built yet

Phase 4 onward: Ansible roles ported from scripts/, the API server, leases and the reconciler. See docs/20 §11.