Yalnizca hpcod/ icin (paths: hpcod/**, .gitea/**). Tag hpcod-v1.2.3 -> deploy-al surumu v1.2.3; surum ikiliye main.releaseVersion olarak yazilir. Paket yalnizca /usr/local/bin/hpcod. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hpcod — HPC-on-Demand control plane
Phase 1 of the architecture: discovery. It reads the backing clouds, translates their inventory into a provider-neutral vocabulary, and reports what could actually be scheduled.
Nothing here writes to a site.
Why the provider abstraction comes first
The portal is meant to sit over more than Proxmox — OpenStack is expected next.
So internal/provider defines the interface, internal/model defines the
vocabulary everything above it reasons in, and Proxmox is simply the first
implementation. Adding a provider means writing one package and registering it;
nothing above internal/provider changes.
cmd/hpcod CLI
internal/model provider-neutral inventory types
internal/provider Provider interface, registry, site config
internal/provider/proxmox first implementation
internal/capability PCI identity -> scheduling vocabulary
internal/inventory catalogue loading, concurrent discovery
The thing this code exists to get right
A PCI class code cannot identify InfiniBand.
A ConnectX-4 Lx reports class 0x0200 (Ethernet) because that variant is
Ethernet-only. A ConnectX-6 reports 0x0207 in IB mode and 0x0200 in
Ethernet mode. So class-based detection produces false negatives and false
positives, and a portal that trusts it will eventually hand a tenant a cluster
whose fabric is not up.
Detection is therefore layered, and every claim carries how it was made:
| Confidence | Source | Trusted to schedule on |
|---|---|---|
class |
PCI class code | GPU, NVMe — yes. Fabrics — no. |
map |
curated vendor:device table |
GPU, NVMe — yes. Fabrics — no. |
agent |
live check on the node | yes |
model.Capability.Trusted enforces it: a fabric capability is never
schedulable until an agent has confirmed the link. Unconfirmed devices are
still reported, because that difference is what an operator needs to fix.
Presence is not enough either. A GPU is offerable only if its IOMMU group
holds nothing else that matters, so Passthrough is derived from group
membership — sibling functions of the same card are fine, an unrelated device
in the group is not.
Use
cp config/sites.yaml.example config/sites.yaml # then edit
export HV01_TOKEN='hpcod@pve!discovery=<uuid>'
go run ./cmd/hpcod sites # configured, reachable?
go run ./cmd/hpcod discover # full inventory
go run ./cmd/hpcod capacity # what could be scheduled now
go run ./cmd/hpcod capmap # the curated table, and why each row exists
go run ./cmd/hpcod discover -json | jq .
Secrets are never written in the catalogue: any setting may be env:NAME and
is read from the environment at startup.
The Proxmox token needs read access only — create it against PVEAuditor:
pveum user add hpcod@pve --comment "HPC-on-Demand discovery"
pveum acl modify / --users hpcod@pve --roles PVEAuditor
pveum user token add hpcod@pve discovery --privsep 0
Sample output
=== hv01 (Lab cluster, proxmox)
node hv01 (online) 28 cores (0 free), 504 GiB RAM (52 GiB free)
cpu: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
gpu 0000:01:00.0 SCHEDULABLE via class GM107GL [Quadro K620]
roce 0000:07:00.0 unconfirmed via map MT27710 Family [ConnectX-4 Lx]
roce 0000:07:00.1 unconfirmed via map MT27710 Family [ConnectX-4 Lx]
The Mellanox cards are reported as roce, not infiniband, and withheld from
the schedulable set — which is the whole point.
Capacity accounting
Free capacity is total minus what running guests have already been allocated,
not raw hardware totals. Physical cores are the schedulable total by default
(cpu_overcommit: "1.0"): an HPC tenant handed oversubscribed cores simply
runs slowly. Raise it per site only where that is acceptable.
Phase 2 — placement
placement.Plan(sites, request, weights) is a pure function: no I/O, no
cloud. That is what makes the decision reproducible, unit-testable against
synthetic multi-site fixtures, and usable as a dry-run endpoint later.
hpcod plan -f examples/gpu-request.yaml # against live sites
hpcod plan -f examples/gpu-request.yaml -inventory snap.json # against a snapshot
hpcod discover -json > snap.json # take a snapshot
Two phases, in this order. Filter eliminates the impossible and records why per site; score ranks the survivors.
score = 3.0 * capability_fit // penalise consuming scarce hardware the
// request did not ask for
+ 1.0 * free_capacity // prefer emptier sites
+ 0.8 * packing // fewer hypervisors per tenant
- 1.2 * fragmentation // avoid unusable remainders
capability_fit dominates deliberately. Scarce hardware must be kept for
requests that need it: a plain CPU job placed on the only InfiniBand site is
a scheduling failure even though it fits.
What the refusals look like
The point of recording reasons is that "no capacity" is useless while this is actionable:
all candidates:
site-a feasible, score 4.40
site-b rejected
- gpu: none present (need 4)
site-c rejected
- infiniband: 1 present but unconfirmed - the fabric has not been
verified by a node agent, so it is not schedulable (need 4, confirmed 0)
site-d rejected
- site is unreachable: dial tcp 10.0.0.40:8006: no route to host
Three different problems, three different fixes: buy hardware, run the agent, fix the network.
Memory is not what the user typed
A compute node unpacks its root filesystem into RAM, so memory_gb in a
request is usable memory and the portal adds RootfsReserveGB on top. A
tenant that asked for 32 GiB and was allocated exactly 32 GiB would OOM under
its own job.
The portal
hpcod serve exposes phases 1 and 2 over HTTP at
https://hpc.example.org — sites and their devices, a dry-run form, and
the curated device map, plus a JSON API (/v1/sites, /v1/capacity,
/v1/dry-run).
It runs on the Caddy host bound to 127.0.0.1:9880, so the reverse proxy is
the only thing that can reach it, and it refuses to bind off loopback without
a user file. Discovery runs on a timer rather than per request: polling every
backing cloud on each page load would hammer their APIs and make the portal as
slow as the slowest site, so pages show how old the inventory is instead.
Everything it serves is read-only. It provisions nothing.
scripts/B10-deploy-hpcod-portal.sh # on the Caddy host
Phase 3 — Terraform
terraform/modules/tenant builds one tenant: an isolated bridge, a head node
with a management and a cluster NIC, and N diskless compute nodes that PXE boot
from it. Applied and destroyed cleanly against a real Proxmox.
cd terraform/envs/lab
cp terraform.tfvars.example terraform.tfvars # then edit
tofu init && tofu plan && tofu apply
MACs are derived, not generated: BC:24:11:E1:<tenant>:<index>. Warewulf
matches nodes by MAC, so a MAC that changed on re-apply would orphan the node's
registration and it would never boot again.
The E1 prefix is not arbitrary. The hand-built cluster on this hypervisor
already uses BC:24:11:E0: with 00 and 0A in the next byte, so a tenant
with id 10 would have produced a duplicate MAC on the same L2 segment and
broken the running cluster. Tenants get their own prefix so the two estates
cannot collide:
| hand-built lab | hpcod tenants | |
|---|---|---|
| bridge | vmbr6 |
vmbrT001–vmbrT254 |
| subnet | 10.10.10.0/24 |
10.64.<tenant>.0/24 |
| VMID | 300–304 | 9001+ |
| MAC | BC:24:11:E0:* |
BC:24:11:E1:* |
The bridge is not the provider's resource
proxmox_network_linux_bridge writes the stanza and then reads the live
interface back before anything has applied it, so creation fails. Proxmox
applies pending network changes with ifreload -a, which reloads every
interface — on a hypervisor carrying other people's guests that is a bad
trade, and on any host with one unrelated broken stanza it returns non-zero
and takes the new bridge down with it.
So the bridge is managed with an explicit remote-exec that appends the
stanza, runs ifquery -a to prove the file still parses, and brings up only
the new interface with ifup. It is idempotent, so an interrupted run
recovers by re-running, and the destroy path verifies the interface is really
gone — a leaked bridge is a leaked subnet.
Two things this needs from Proxmox
- The API token's role needs
SDN.Use, or attaching a VM to the new bridge fails with a 403 that does not mention SDN anywhere obvious. - The cloud image must live in a datastore with the
importcontent type (/var/lib/vz/import/, referenced aslocal:import/...). An image sitting in the ISO directory is rejected with "unable to parse directory volume name".
Not built yet
Phase 4 onward: Ansible roles ported from scripts/, the API server, leases
and the reconciler. See
docs/20 §11.