ci / derle (push) Successful in 22s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
209 lines
9.0 KiB
Markdown
209 lines
9.0 KiB
Markdown
# 2. Installation from Scratch
|
||
|
||
From an empty host to a control plane that has built and torn down a real
|
||
cluster. Everything comes from one file, the `hpcod` binary: it carries the
|
||
Ansible roles, the Terraform module, the database schema and the
|
||
application catalogue, and it installs itself. There are no scripts to copy.
|
||
|
||
Allow about an hour. Most of it is spent waiting for the head node template
|
||
to build.
|
||
|
||
The reference lab of sections 01 and 03–09 was built by hand, with scripts;
|
||
section [10](10-lab-installation.md) records how. You do not need it to run
|
||
hpcod. [hpcod/DEPLOY.md](../hpcod/DEPLOY.md) is the reference for everything
|
||
below, with unattended installs, updates, backups and the known limits.
|
||
|
||
## 2.1 What you need
|
||
|
||
| You need | Why |
|
||
|---|---|
|
||
| A Debian 13 or Ubuntu 24.04 host with root, 2 vCPU, 4 GB RAM, 20 GB disk | the control plane |
|
||
| Proxmox VE 8 or later, and root's password (or two API tokens, 2.4) | where clusters are built |
|
||
| A /24 management network shared by the control plane and Proxmox | head nodes get addresses on it |
|
||
| About 40 free addresses on it, outside DHCP, plus one more | one per live cluster, one for the template build |
|
||
| A name for the portal and a domain for clusters | each cluster's portal is `c<NNN>.<CLUSTER_DOMAIN>` |
|
||
| Internet access from the control plane and from head nodes | packages, OpenTofu providers, images |
|
||
|
||
Proxmox is one node of a Proxmox cluster? Setup lists the other nodes and
|
||
asks which ones clusters may use; you do not add them one by one (2.3).
|
||
No Proxmox to hand? Setup can run one for you, simulated in QEMU on the
|
||
control plane host (2.5). No public domain or certificate? hpcod can serve
|
||
HTTPS and DNS itself (2.6). The node, desktop and application images come
|
||
from a public registry and pull anonymously; you build none of them.
|
||
|
||
## 2.2 Get the binary and run setup
|
||
|
||
On the control plane host:
|
||
|
||
```bash
|
||
V=v0.2.2 A=linux-amd64 # A=linux-arm64 on an arm64 host
|
||
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/hpcod-$V-$A
|
||
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/SHA256SUMS
|
||
sha256sum -c --ignore-missing SHA256SUMS
|
||
install -m 0755 hpcod-$V-$A hpcod
|
||
sudo ./hpcod setup
|
||
```
|
||
|
||
A release binary arrives without its execute bit, which `install` sets.
|
||
Setup copies itself to `/usr/local/bin/hpcod`; from then on it is
|
||
`sudo hpcod …`. To build the binary instead, `make linux` in `hpcod/`.
|
||
|
||
Setup is for administrators. It refuses to run as anyone but root, runs only
|
||
in a terminal on the control plane host, and the portal never exposes it.
|
||
|
||
It is a full-screen program. The stages of the install run down the left and
|
||
tick off as you go; the keys that work are along the bottom. Arrows move,
|
||
enter goes on, esc goes back.
|
||
|
||

|
||
|
||
On a fresh host the first screen offers one thing, **Install hpcod on this
|
||
host**. On an installed one it offers to check the installation, reconfigure
|
||
and install again, build a test cluster, or rebuild the head node template.
|
||
|
||
## 2.3 What setup asks
|
||
|
||
1. **Where clusters are built:** a Proxmox VE you already have, or a
|
||
simulated one in QEMU on this host.
|
||
|
||

|
||
|
||
2. **Proxmox.** The API address, its certificate, and how to sign in:
|
||
root@pam's password, used once to create two API tokens and to let this
|
||
host in over SSH and never stored; or two tokens you made yourself (2.4).
|
||
Setup signs in at once, so a wrong password, an untrusted certificate or
|
||
an address nothing answers on is reported here, not twenty minutes
|
||
later. For a simulated Proxmox this screen asks only for its cores,
|
||
memory and disk.
|
||
3. **The site,** on one form: the Proxmox node and datastores, picked from
|
||
what Proxmox returned; in a Proxmox cluster, which of its nodes compute
|
||
nodes may use (heads stay on the node with the template, and more than
|
||
one node makes the SDN zone vxlan, see DEPLOY.md); the portal and cluster domains; how users reach
|
||
the portal (behind your reverse proxy, or hpcod serving HTTPS and DNS
|
||
itself); the management network and the head node window. Each field is
|
||
checked as you type, including against the others.
|
||
4. **Review.** Everything that will happen, before any of it does.
|
||
|
||

|
||
|
||
## 2.4 What setup does
|
||
|
||
The steps tick off in two columns, with the current step's output below:
|
||
|
||
```
|
||
site packages account binary keys ca
|
||
database tree schema catalogue config unit
|
||
start pvesim proxmox template
|
||
```
|
||
|
||
- **site … unit** install PostgreSQL, Ansible and OpenTofu, create the
|
||
`hpcod` account, install the binary as `/usr/local/bin/hpcod`, unpack the
|
||
roles, module and schema into `/var/lib/hpcod`, apply the schema, and
|
||
write `/etc/hpcod` and the systemd unit.
|
||
- **start** starts the portal and waits for `/healthz`. It comes before the
|
||
Proxmox steps, so an install that stops later still leaves a portal
|
||
running.
|
||
- **pvesim** installs Proxmox VE into the simulated VM, when there is one.
|
||
- **proxmox** sets up SSH access, both tokens, the SDN zone, snippets and
|
||
import on the datastores, and the cloud image.
|
||
- **template** builds the head node template, the longest step (2.8).
|
||
|
||
Each step reads back what it changed, and every step is safe to run again.
|
||
The full log is `/var/log/hpcod-setup.log`. The first portal login is
|
||
printed once and kept in `/root/.hpcod-portal-login`.
|
||
|
||
If you would rather not give setup root's password, make the tokens
|
||
yourself and choose "API tokens I made myself":
|
||
|
||
```bash
|
||
pveum user add hpcod@pve
|
||
pveum acl modify / --users hpcod@pve --roles PVEAuditor
|
||
pveum user token add hpcod@pve discovery --privsep 0 # discovery token
|
||
pveum user token add root@pam hpcod-tf --privsep 0 # Terraform token
|
||
```
|
||
|
||
The Terraform token is root's because Proxmox allows raw PCI passthrough,
|
||
which is how a GPU reaches a compute node, to root alone. With your own
|
||
tokens you also add this host's SSH key to root on Proxmox; setup shows it.
|
||
|
||
## 2.5 No Proxmox yet: the simulated one
|
||
|
||
"A simulated Proxmox, in QEMU on this host" runs Proxmox VE in a VM on the
|
||
control plane host, on a private bridge (`10.99.0.0/24`) with NAT, so it
|
||
never touches your network. Setup installs Proxmox VE into it, fills in
|
||
every Proxmox and network value, and carries on as with a real one.
|
||
|
||
The host needs `/dev/kvm`, nested virtualization (clusters run inside the
|
||
simulation), and room: 8 cores, 24 GiB and 150 GiB by default. On a VM, give
|
||
it CPU type `host`. It is for trying hpcod, not for tenants.
|
||
|
||
## 2.6 Reaching the portal
|
||
|
||
**Behind your own reverse proxy.** Copy the two blocks from
|
||
`hpcod/deploy/Caddyfile.example` into your Caddyfile, set your domains and
|
||
a wildcard certificate for `*.CLUSTER_DOMAIN` (DNS-01), and reload. Setup
|
||
does not touch the proxy.
|
||
|
||
**On a closed network.** Choose "hpcod serves HTTPS itself" and "hpcod
|
||
answers DNS for them". hpcod then listens on 443 at `PUBLIC_IP` with a
|
||
certificate from its own CA, which can sign only for your domains, and
|
||
answers DNS on port 53 for the portal and every cluster name, forwarding
|
||
the rest to `NAMESERVERS`. Users point their resolver at it (or forward the
|
||
zone to it) and trust `http://<portal>/ca.crt` once. Setup prints exactly
|
||
this at the end.
|
||
|
||
## 2.7 Check it, then prove it
|
||
|
||
Setup ends by checking the installation. The same check runs on its own:
|
||
|
||
```bash
|
||
sudo hpcod verify
|
||
```
|
||
|
||
It checks each fact by using it: the tokens against the API, the template by
|
||
asking Proxmox, the database by connecting as the portal does. It ends with
|
||
`READY` or `NOT READY`, and each failed line names the step that fixes it.
|
||
|
||

|
||
|
||
Then setup offers to build a test cluster. The same test runs on its own:
|
||
|
||
```bash
|
||
sudo hpcod selftest # 1 compute node; -compute 2 also runs a job across two
|
||
```
|
||
|
||
It takes the path a user's "create" takes: discovery, placement, allocation,
|
||
then the portal's own builder. It waits for Slurm to see the nodes, runs
|
||
`srun -N <n> hostname`, checks that the cluster's portal is published, and
|
||
takes the cluster down again. About 25 minutes. A failed test leaves the
|
||
cluster up for inspection and prints the command that removes it.
|
||
|
||
A passing run ends like this:
|
||
|
||
```
|
||
==> placed on pvesim
|
||
==> allocated c001-1 (serial 1); it shows on the portal's cluster list
|
||
==> published: https://c001.hpc.lab
|
||
==> waiting for 1 compute node(s) to be idle in Slurm
|
||
==> PASSED: c001-1 built, published, and ran a job on 1 node(s)
|
||
==> destroying c001-1
|
||
==> destroyed and released
|
||
```
|
||
|
||
From here, sign in to the portal with the login setup printed and create a
|
||
cluster. Section [20](20-hpc-on-demand-architecture.md) explains what
|
||
happens when you do.
|
||
|
||
## 2.8 Timings
|
||
|
||
One run, on the simulated Proxmox, itself nested inside a Proxmox VM:
|
||
|
||
| Stage | Duration |
|
||
|---|---|
|
||
| A full `setup` run with a template build | 16½ min |
|
||
| of which the template playbook | 11 min |
|
||
| Test cluster, built and destroyed (setup's own estimate) | ~25 min |
|
||
|
||
Installing Proxmox VE into the simulated VM comes on top, once; it was not
|
||
timed separately.
|