Files
werewulf/docs/02-installation.md
ilkerandClaude Opus 5.5 3880311e36
ci / derle (push) Successful in 22s
hpcod: changelog for v0.2.2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 20:22:09 +01:00

209 lines
9.0 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 2. Installation from Scratch
From an empty host to a control plane that has built and torn down a real
cluster. Everything comes from one file, the `hpcod` binary: it carries the
Ansible roles, the Terraform module, the database schema and the
application catalogue, and it installs itself. There are no scripts to copy.
Allow about an hour. Most of it is spent waiting for the head node template
to build.
The reference lab of sections 01 and 03–09 was built by hand, with scripts;
section [10](10-lab-installation.md) records how. You do not need it to run
hpcod. [hpcod/DEPLOY.md](../hpcod/DEPLOY.md) is the reference for everything
below, with unattended installs, updates, backups and the known limits.
## 2.1 What you need
| You need | Why |
|---|---|
| A Debian 13 or Ubuntu 24.04 host with root, 2 vCPU, 4 GB RAM, 20 GB disk | the control plane |
| Proxmox VE 8 or later, and root's password (or two API tokens, 2.4) | where clusters are built |
| A /24 management network shared by the control plane and Proxmox | head nodes get addresses on it |
| About 40 free addresses on it, outside DHCP, plus one more | one per live cluster, one for the template build |
| A name for the portal and a domain for clusters | each cluster's portal is `c<NNN>.<CLUSTER_DOMAIN>` |
| Internet access from the control plane and from head nodes | packages, OpenTofu providers, images |
Proxmox is one node of a Proxmox cluster? Setup lists the other nodes and
asks which ones clusters may use; you do not add them one by one (2.3).
No Proxmox to hand? Setup can run one for you, simulated in QEMU on the
control plane host (2.5). No public domain or certificate? hpcod can serve
HTTPS and DNS itself (2.6). The node, desktop and application images come
from a public registry and pull anonymously; you build none of them.
## 2.2 Get the binary and run setup
On the control plane host:
```bash
V=v0.2.2 A=linux-amd64 # A=linux-arm64 on an arm64 host
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/hpcod-$V-$A
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/SHA256SUMS
sha256sum -c --ignore-missing SHA256SUMS
install -m 0755 hpcod-$V-$A hpcod
sudo ./hpcod setup
```
A release binary arrives without its execute bit, which `install` sets.
Setup copies itself to `/usr/local/bin/hpcod`; from then on it is
`sudo hpcod …`. To build the binary instead, `make linux` in `hpcod/`.
Setup is for administrators. It refuses to run as anyone but root, runs only
in a terminal on the control plane host, and the portal never exposes it.
It is a full-screen program. The stages of the install run down the left and
tick off as you go; the keys that work are along the bottom. Arrows move,
enter goes on, esc goes back.
![The setup welcome screen](../hpcod/docs/setup-1-welcome.png)
On a fresh host the first screen offers one thing, **Install hpcod on this
host**. On an installed one it offers to check the installation, reconfigure
and install again, build a test cluster, or rebuild the head node template.
## 2.3 What setup asks
1. **Where clusters are built:** a Proxmox VE you already have, or a
simulated one in QEMU on this host.
![Choosing where clusters are built](../hpcod/docs/setup-2-where.png)
2. **Proxmox.** The API address, its certificate, and how to sign in:
root@pam's password, used once to create two API tokens and to let this
host in over SSH and never stored; or two tokens you made yourself (2.4).
Setup signs in at once, so a wrong password, an untrusted certificate or
an address nothing answers on is reported here, not twenty minutes
later. For a simulated Proxmox this screen asks only for its cores,
memory and disk.
3. **The site,** on one form: the Proxmox node and datastores, picked from
what Proxmox returned; in a Proxmox cluster, which of its nodes compute
nodes may use (heads stay on the node with the template, and more than
one node makes the SDN zone vxlan, see DEPLOY.md); the portal and cluster domains; how users reach
the portal (behind your reverse proxy, or hpcod serving HTTPS and DNS
itself); the management network and the head node window. Each field is
checked as you type, including against the others.
4. **Review.** Everything that will happen, before any of it does.
![The review screen](../hpcod/docs/setup-3-review.png)
## 2.4 What setup does
The steps tick off in two columns, with the current step's output below:
```
site packages account binary keys ca
database tree schema catalogue config unit
start pvesim proxmox template
```
- **site … unit** install PostgreSQL, Ansible and OpenTofu, create the
`hpcod` account, install the binary as `/usr/local/bin/hpcod`, unpack the
roles, module and schema into `/var/lib/hpcod`, apply the schema, and
write `/etc/hpcod` and the systemd unit.
- **start** starts the portal and waits for `/healthz`. It comes before the
Proxmox steps, so an install that stops later still leaves a portal
running.
- **pvesim** installs Proxmox VE into the simulated VM, when there is one.
- **proxmox** sets up SSH access, both tokens, the SDN zone, snippets and
import on the datastores, and the cloud image.
- **template** builds the head node template, the longest step (2.8).
Each step reads back what it changed, and every step is safe to run again.
The full log is `/var/log/hpcod-setup.log`. The first portal login is
printed once and kept in `/root/.hpcod-portal-login`.
If you would rather not give setup root's password, make the tokens
yourself and choose "API tokens I made myself":
```bash
pveum user add hpcod@pve
pveum acl modify / --users hpcod@pve --roles PVEAuditor
pveum user token add hpcod@pve discovery --privsep 0 # discovery token
pveum user token add root@pam hpcod-tf --privsep 0 # Terraform token
```
The Terraform token is root's because Proxmox allows raw PCI passthrough,
which is how a GPU reaches a compute node, to root alone. With your own
tokens you also add this host's SSH key to root on Proxmox; setup shows it.
## 2.5 No Proxmox yet: the simulated one
"A simulated Proxmox, in QEMU on this host" runs Proxmox VE in a VM on the
control plane host, on a private bridge (`10.99.0.0/24`) with NAT, so it
never touches your network. Setup installs Proxmox VE into it, fills in
every Proxmox and network value, and carries on as with a real one.
The host needs `/dev/kvm`, nested virtualization (clusters run inside the
simulation), and room: 8 cores, 24 GiB and 150 GiB by default. On a VM, give
it CPU type `host`. It is for trying hpcod, not for tenants.
## 2.6 Reaching the portal
**Behind your own reverse proxy.** Copy the two blocks from
`hpcod/deploy/Caddyfile.example` into your Caddyfile, set your domains and
a wildcard certificate for `*.CLUSTER_DOMAIN` (DNS-01), and reload. Setup
does not touch the proxy.
**On a closed network.** Choose "hpcod serves HTTPS itself" and "hpcod
answers DNS for them". hpcod then listens on 443 at `PUBLIC_IP` with a
certificate from its own CA, which can sign only for your domains, and
answers DNS on port 53 for the portal and every cluster name, forwarding
the rest to `NAMESERVERS`. Users point their resolver at it (or forward the
zone to it) and trust `http://<portal>/ca.crt` once. Setup prints exactly
this at the end.
## 2.7 Check it, then prove it
Setup ends by checking the installation. The same check runs on its own:
```bash
sudo hpcod verify
```
It checks each fact by using it: the tokens against the API, the template by
asking Proxmox, the database by connecting as the portal does. It ends with
`READY` or `NOT READY`, and each failed line names the step that fixes it.
![The installation check](../hpcod/docs/setup-6-check.png)
Then setup offers to build a test cluster. The same test runs on its own:
```bash
sudo hpcod selftest # 1 compute node; -compute 2 also runs a job across two
```
It takes the path a user's "create" takes: discovery, placement, allocation,
then the portal's own builder. It waits for Slurm to see the nodes, runs
`srun -N <n> hostname`, checks that the cluster's portal is published, and
takes the cluster down again. About 25 minutes. A failed test leaves the
cluster up for inspection and prints the command that removes it.
A passing run ends like this:
```
==> placed on pvesim
==> allocated c001-1 (serial 1); it shows on the portal's cluster list
==> published: https://c001.hpc.lab
==> waiting for 1 compute node(s) to be idle in Slurm
==> PASSED: c001-1 built, published, and ran a job on 1 node(s)
==> destroying c001-1
==> destroyed and released
```
From here, sign in to the portal with the login setup printed and create a
cluster. Section [20](20-hpc-on-demand-architecture.md) explains what
happens when you do.
## 2.8 Timings
One run, on the simulated Proxmox, itself nested inside a Proxmox VM:
| Stage | Duration |
|---|---|
| A full `setup` run with a template build | 16½ min |
| of which the template playbook | 11 min |
| Test cluster, built and destroyed (setup's own estimate) | ~25 min |
Installing Proxmox VE into the simulated VM comes on top, once; it was not
timed separately.