ci / derle (push) Successful in 22s
The repository had no license, so it was public but not open source: nobody could use, change or redistribute it. LICENSE is the text from apache.org, unmodified. NOTICE says that the IBM Plex fonts embedded in the print handbook are under the SIL Open Font License, not Apache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
267 lines
10 KiB
Markdown
267 lines
10 KiB
Markdown
# hpcod — HPC-on-Demand control plane
|
||
|
||
Phase 1 of [the architecture](../docs/20-hpc-on-demand-architecture.md):
|
||
**discovery**. It reads the backing clouds, translates their inventory into a
|
||
provider-neutral vocabulary, and reports what could actually be scheduled.
|
||
|
||
Nothing here writes to a site.
|
||
|
||
## Why the provider abstraction comes first
|
||
|
||
The portal is meant to sit over more than Proxmox — OpenStack is expected next.
|
||
So `internal/provider` defines the interface, `internal/model` defines the
|
||
vocabulary everything above it reasons in, and Proxmox is simply the first
|
||
implementation. Adding a provider means writing one package and registering it;
|
||
nothing above `internal/provider` changes.
|
||
|
||
```
|
||
cmd/hpcod CLI
|
||
internal/model provider-neutral inventory types
|
||
internal/provider Provider interface, registry, site config
|
||
internal/provider/proxmox first implementation
|
||
internal/capability PCI identity -> scheduling vocabulary
|
||
internal/inventory catalogue loading, concurrent discovery
|
||
```
|
||
|
||
## The thing this code exists to get right
|
||
|
||
**A PCI class code cannot identify InfiniBand.**
|
||
|
||
A ConnectX-4 Lx reports class `0x0200` (Ethernet) because that variant is
|
||
Ethernet-only. A ConnectX-6 reports `0x0207` in IB mode and `0x0200` in
|
||
Ethernet mode. So class-based detection produces false negatives *and* false
|
||
positives, and a portal that trusts it will eventually hand a tenant a cluster
|
||
whose fabric is not up.
|
||
|
||
Detection is therefore layered, and every claim carries how it was made:
|
||
|
||
| Confidence | Source | Trusted to schedule on |
|
||
|---|---|---|
|
||
| `class` | PCI class code | GPU, NVMe — yes. Fabrics — no. |
|
||
| `map` | curated `vendor:device` table | GPU, NVMe — yes. Fabrics — no. |
|
||
| `agent` | live check on the node | yes |
|
||
|
||
`model.Capability.Trusted` enforces it: a fabric capability is never
|
||
schedulable until an agent has confirmed the link. Unconfirmed devices are
|
||
still reported, because that difference is what an operator needs to fix.
|
||
|
||
Presence is not enough either. A GPU is offerable only if its IOMMU group
|
||
holds nothing else that matters, so `Passthrough` is derived from group
|
||
membership — sibling functions of the same card are fine, an unrelated device
|
||
in the group is not.
|
||
|
||
## Use
|
||
|
||
```bash
|
||
cp config/sites.yaml.example config/sites.yaml # then edit
|
||
export HV01_TOKEN='hpcod@pve!discovery=<uuid>'
|
||
|
||
go run ./cmd/hpcod sites # configured, reachable?
|
||
go run ./cmd/hpcod discover # full inventory
|
||
go run ./cmd/hpcod capacity # what could be scheduled now
|
||
go run ./cmd/hpcod capmap # the curated table, and why each row exists
|
||
go run ./cmd/hpcod discover -json | jq .
|
||
```
|
||
|
||
Secrets are never written in the catalogue: any setting may be `env:NAME` and
|
||
is read from the environment at startup.
|
||
|
||
The Proxmox token needs read access only — create it against `PVEAuditor`:
|
||
|
||
```bash
|
||
pveum user add hpcod@pve --comment "HPC-on-Demand discovery"
|
||
pveum acl modify / --users hpcod@pve --roles PVEAuditor
|
||
pveum user token add hpcod@pve discovery --privsep 0
|
||
```
|
||
|
||
## Sample output
|
||
|
||
```
|
||
=== hv01 (Lab cluster, proxmox)
|
||
node hv01 (online) 28 cores (0 free), 504 GiB RAM (52 GiB free)
|
||
cpu: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
|
||
gpu 0000:01:00.0 SCHEDULABLE via class GM107GL [Quadro K620]
|
||
roce 0000:07:00.0 unconfirmed via map MT27710 Family [ConnectX-4 Lx]
|
||
roce 0000:07:00.1 unconfirmed via map MT27710 Family [ConnectX-4 Lx]
|
||
```
|
||
|
||
The Mellanox cards are reported as `roce`, not `infiniband`, and withheld from
|
||
the schedulable set — which is the whole point.
|
||
|
||
## Capacity accounting
|
||
|
||
Free capacity is total minus what running guests have already been allocated,
|
||
not raw hardware totals. Physical cores are the schedulable total by default
|
||
(`cpu_overcommit: "1.0"`): an HPC tenant handed oversubscribed cores simply
|
||
runs slowly. Raise it per site only where that is acceptable.
|
||
|
||
## Phase 2 — placement
|
||
|
||
`placement.Plan(sites, request, weights)` is a **pure function**: no I/O, no
|
||
cloud. That is what makes the decision reproducible, unit-testable against
|
||
synthetic multi-site fixtures, and usable as a dry-run endpoint later.
|
||
|
||
```bash
|
||
hpcod plan -f examples/gpu-request.yaml # against live sites
|
||
hpcod plan -f examples/gpu-request.yaml -inventory snap.json # against a snapshot
|
||
hpcod discover -json > snap.json # take a snapshot
|
||
```
|
||
|
||
Two phases, in this order. **Filter** eliminates the impossible and records why
|
||
per site; **score** ranks the survivors.
|
||
|
||
```
|
||
score = 3.0 * capability_fit // penalise consuming scarce hardware the
|
||
// request did not ask for
|
||
+ 1.0 * free_capacity // prefer emptier sites
|
||
+ 0.8 * packing // fewer hypervisors per tenant
|
||
- 1.2 * fragmentation // avoid unusable remainders
|
||
```
|
||
|
||
`capability_fit` dominates deliberately. Scarce hardware must be kept for
|
||
requests that need it: **a plain CPU job placed on the only InfiniBand site is
|
||
a scheduling failure even though it fits.**
|
||
|
||
### What the refusals look like
|
||
|
||
The point of recording reasons is that "no capacity" is useless while this is
|
||
actionable:
|
||
|
||
```
|
||
all candidates:
|
||
site-a feasible, score 4.40
|
||
site-b rejected
|
||
- gpu: none present (need 4)
|
||
site-c rejected
|
||
- infiniband: 1 present but unconfirmed - the fabric has not been
|
||
verified by a node agent, so it is not schedulable (need 4, confirmed 0)
|
||
site-d rejected
|
||
- site is unreachable: dial tcp 10.0.0.40:8006: no route to host
|
||
```
|
||
|
||
Three different problems, three different fixes: buy hardware, run the agent,
|
||
fix the network.
|
||
|
||
### Memory is not what the user typed
|
||
|
||
A compute node unpacks its root filesystem into RAM, so `memory_gb` in a
|
||
request is **usable** memory and the portal adds `RootfsReserveGB` on top. A
|
||
tenant that asked for 32 GiB and was allocated exactly 32 GiB would OOM under
|
||
its own job.
|
||
|
||
## Installing it
|
||
|
||
On a Debian 13 or Ubuntu 24.04 host:
|
||
|
||
```bash
|
||
V=v0.2.2 A=linux-amd64 # A=linux-arm64 on an arm64 host
|
||
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/hpcod-$V-$A
|
||
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/SHA256SUMS
|
||
sha256sum -c --ignore-missing SHA256SUMS
|
||
install -m 0755 hpcod-$V-$A hpcod
|
||
sudo ./hpcod setup
|
||
```
|
||
|
||
That runs the installer, a full-screen TUI for administrators. It installs
|
||
the binary as `/usr/local/bin/hpcod`, so afterwards:
|
||
|
||
```bash
|
||
sudo hpcod verify # can this installation build a cluster?
|
||
sudo hpcod selftest # build a real cluster, run a job on it, take it down
|
||
```
|
||
|
||
The binary carries the Ansible roles, the Terraform module, the schema and
|
||
the application catalogue; setup installs everything else. It installs
|
||
against a Proxmox VE you have, or against one it simulates in QEMU on the
|
||
same host. See **[DEPLOY.md](DEPLOY.md)**.
|
||
|
||
## The portal
|
||
|
||
`hpcod serve` is the portal; setup installs it as the `hpcod-portal`
|
||
service. It serves:
|
||
|
||
- sites and their devices;
|
||
- the plan and create form;
|
||
- the cluster list, with build logs as they happen;
|
||
- projects and usage;
|
||
- the curated device map;
|
||
- account approval;
|
||
- each cluster's own portal at `c<NNN>.<domain>`;
|
||
- a JSON API (`/v1/sites`, `/v1/capacity`, `/v1/dry-run`, `/v1/clusters`).
|
||
|
||
It listens on `127.0.0.1:9880`, and is reached in one of two ways:
|
||
|
||
- **Behind your reverse proxy.** The proxy is then the only thing that can
|
||
reach it, and it refuses to bind off loopback without a user file.
|
||
- **On a closed network, serving HTTPS on 443 itself.** Its certificate
|
||
comes from the installation's own name-constrained CA. It can also answer
|
||
DNS for its names.
|
||
|
||
Discovery runs on a timer rather than per request. Polling every backing
|
||
cloud on each page load would hammer their APIs and make the portal as slow
|
||
as the slowest site, so pages show how old the inventory is instead.
|
||
|
||
## Phase 3 — Terraform
|
||
|
||
`terraform/modules/tenant` builds one tenant: an isolated bridge, a head node
|
||
with a management and a cluster NIC, and N diskless compute nodes that PXE boot
|
||
from it. Applied and destroyed cleanly against a real Proxmox.
|
||
|
||
```bash
|
||
cd terraform/envs/lab
|
||
cp terraform.tfvars.example terraform.tfvars # then edit
|
||
tofu init && tofu plan && tofu apply
|
||
```
|
||
|
||
MACs are **derived, not generated**: `BC:24:11:E1:<tenant>:<index>`. Warewulf
|
||
matches nodes by MAC, so a MAC that changed on re-apply would orphan the node's
|
||
registration and it would never boot again.
|
||
|
||
The `E1` prefix is not arbitrary. The hand-built cluster on this hypervisor
|
||
already uses `BC:24:11:E0:` with `00` and `0A` in the next byte, so a tenant
|
||
with id 10 would have produced a **duplicate MAC on the same L2 segment** and
|
||
broken the running cluster. Tenants get their own prefix so the two estates
|
||
cannot collide:
|
||
|
||
| | hand-built lab | hpcod tenants |
|
||
|---|---|---|
|
||
| bridge | `vmbr6` | `vmbrT001`–`vmbrT254` |
|
||
| subnet | `10.10.10.0/24` | `10.64.<tenant>.0/24` |
|
||
| VMID | 300–304 | 9001+ |
|
||
| MAC | `BC:24:11:E0:*` | `BC:24:11:E1:*` |
|
||
|
||
### The bridge is not the provider's resource
|
||
|
||
`proxmox_network_linux_bridge` writes the stanza and then reads the live
|
||
interface back before anything has applied it, so creation fails. Proxmox
|
||
applies pending network changes with `ifreload -a`, which reloads **every**
|
||
interface — on a hypervisor carrying other people's guests that is a bad
|
||
trade, and on any host with one unrelated broken stanza it returns non-zero
|
||
and takes the new bridge down with it.
|
||
|
||
So the bridge is managed with an explicit `remote-exec` that appends the
|
||
stanza, runs `ifquery -a` to prove the file still parses, and brings up **only
|
||
the new interface** with `ifup`. It is idempotent, so an interrupted run
|
||
recovers by re-running, and the destroy path verifies the interface is really
|
||
gone — a leaked bridge is a leaked subnet.
|
||
|
||
### Two things this needs from Proxmox
|
||
|
||
- The API token's role needs **`SDN.Use`**, or attaching a VM to the new
|
||
bridge fails with a 403 that does not mention SDN anywhere obvious.
|
||
- The cloud image must live in a datastore with the **`import`** content type
|
||
(`/var/lib/vz/import/`, referenced as `local:import/...`). An image sitting
|
||
in the ISO directory is rejected with *"unable to parse directory volume
|
||
name"*.
|
||
|
||
## Not built yet
|
||
|
||
Phase 4 onward: Ansible roles ported from `scripts/`, the API server, leases
|
||
and the reconciler. See
|
||
[docs/20 §11](../docs/20-hpc-on-demand-architecture.md).
|
||
|
||
## License
|
||
|
||
[Apache License 2.0](../LICENSE). The fonts embedded in the print handbook
|
||
are under the SIL Open Font License; see [NOTICE](../NOTICE).
|