Files
werewulf/scripts/B40-publish-cluster.py
T
ilkerandClaude Opus 5.5 772579648c hpcod: one binary installs, checks and tests the control plane
An installation used to be this one: the deploy scripts and the Ansible
defaults carried one site's domain, hypervisor and login, the Proxmox side
and the database were set up by hand, and deploying meant copying scripts
and a tree of roles beside the binary and keeping them in step with it.

The binary now carries the roles, the Terraform module, the schema and the
catalogue (embed.go), and installs itself:

- hpcod setup: a full-screen installer for administrators. It asks where
  clusters are built, signs in to Proxmox before asking anything else
  (root's password once, or tokens), offers nodes and datastores from what
  Proxmox has, checks every field as it is typed, runs every step with its
  output live, checks the result and offers a test cluster. -yes runs the
  same steps from a site file; make deploy uses that.
- hpcod verify: each fact checked by using it.
- hpcod selftest: the smallest real cluster, by the portal's own path, a
  job run on it, then taken down.
- Closed networks: hpcod serves HTTPS itself from a CA name-constrained to
  the site's domains, issuing and renewing its own certificate, and
  answers DNS for the portal and every cluster name.
- A simulated Proxmox: Proxmox VE in QEMU on the control plane host, on a
  private bridge with NAT, for trying it without one.

Every step reads back what it changed and is safe to run again. The
site's values live in one file; group_vars/all/site.yml is generated from
it, and the playbooks refuse to run without it. ci_user and -ssh-user no
longer default to a personal account. scripts/B10, B20 and C20 are gone:
their work is the setup steps.

Proven end to end on a simulated Proxmox (Proxmox VE 9 in a VM on a real
host): the install, the Proxmox step by the password path, the head
template, verify READY, and a test cluster that ran srun on its compute
node and was destroyed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:20:04 +01:00

152 lines
5.3 KiB
Python
Executable File

#!/usr/bin/env python3
"""Publish a cluster's portal at c<NNN>.<CLUSTER_DOMAIN>, or withdraw it.
The control plane routes cluster portals by subdomain, resolving the label
through a registry file (see hpcod/internal/httpapi/tenantproxy.go). This
script is what writes that file, because the head node cannot: it has no
access to the control plane's configuration, and should not have.
Clusters, not tenants: one tenant may run several at once, so the id here
is the cluster's.
Publishing copies the cluster's own certificate to the control plane, which
then pins it as the sole trust anchor for that one cluster. Re-running after
the cluster regenerates its certificate refreshes the copy - necessary,
because the proxy will refuse a certificate it does not hold.
B40-publish-cluster.py 2 10.0.0.62
B40-publish-cluster.py --remove 2
"""
import argparse
import os
import subprocess
import sys
import tempfile
import yaml
# The portal writes both of these, and it runs with ProtectSystem=strict.
# They were under /etc until a build reported "built, but not published:
# read-only file system" - correct, and the wrong place to fix it, since
# /etc/hpcod also holds the user file, the database password and the
# proxy's private key.
REGISTRY = "/var/lib/hpcod/clusters.yaml"
CERT_DIR = "/var/lib/hpcod/clusters"
SSH_KEY = "/var/lib/hpcod/.ssh/id_ed25519"
SITE = os.environ.get("SITE", "/etc/hpcod/site.env")
CLUSTER_CERT = "/etc/pki/tls/certs/ood-portal.crt"
def site() -> dict:
"""The site's values, from the file `hpcod setup` writes. KEY=value lines, with
optional double quotes - the subset of shell that site.env is written in.
"""
vals = {}
try:
with open(SITE) as fh:
for line in fh:
line = line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
k, _, v = line.partition("=")
vals[k.strip()] = v.strip().strip('"')
except FileNotFoundError:
raise SystemExit(f"no {SITE}; run `hpcod setup` first")
for k in ("CI_USER", "CLUSTER_DOMAIN"):
if not vals.get(k):
raise SystemExit(f"{SITE} does not set {k}")
return vals
def fetch_cert(head: str) -> bytes:
"""Read the certificate the cluster's portal presents, over SSH."""
out = subprocess.run(
[
"ssh", "-o", "BatchMode=yes", "-o", "StrictHostKeyChecking=no",
"-o", "UserKnownHostsFile=/dev/null", "-o", "ConnectTimeout=10",
"-i", SSH_KEY, f"{site()['CI_USER']}@{head}", f"sudo cat {CLUSTER_CERT}",
],
capture_output=True, check=True,
)
if b"BEGIN CERTIFICATE" not in out.stdout:
raise SystemExit(f"{head} did not return a certificate")
return out.stdout
def load() -> dict:
try:
with open(REGISTRY) as fh:
doc = yaml.safe_load(fh) or {}
except FileNotFoundError:
doc = {}
doc.setdefault("clusters", [])
return doc
def save(doc: dict) -> None:
# Written whole and moved into place: the control plane re-reads this
# file whenever its mtime moves, and a half-written one would take every
# published cluster offline until the next write.
d = os.path.dirname(REGISTRY)
with tempfile.NamedTemporaryFile("w", dir=d, delete=False) as tmp:
tmp.write("# Written by B40-publish-cluster.py - do not edit by hand.\n")
yaml.safe_dump(doc, tmp, default_flow_style=False, sort_keys=False)
name = tmp.name
os.chmod(name, 0o644)
os.replace(name, REGISTRY)
def publish(cluster_id: int, head: str) -> None:
os.makedirs(CERT_DIR, mode=0o755, exist_ok=True)
cert_path = os.path.join(CERT_DIR, f"c{cluster_id:03d}.crt")
with open(cert_path, "wb") as fh:
fh.write(fetch_cert(head))
os.chmod(cert_path, 0o644)
doc = load()
doc["clusters"] = [t for t in doc["clusters"] if t.get("id") != cluster_id]
doc["clusters"].append(
{"id": cluster_id, "portal": f"{head}:443", "cert": cert_path}
)
doc["clusters"].sort(key=lambda t: t.get("id", 0))
save(doc)
print(f"published c{cluster_id:03d} -> {head}:443")
print(f" https://c{cluster_id:03d}.{site()['CLUSTER_DOMAIN']}")
def remove(cluster_id: int) -> None:
doc = load()
before = len(doc["clusters"])
doc["clusters"] = [t for t in doc["clusters"] if t.get("id") != cluster_id]
save(doc)
# The certificate goes too. Leaving it would let a later cluster reusing
# this id be trusted against its predecessor's key.
cert_path = os.path.join(CERT_DIR, f"c{cluster_id:03d}.crt")
if os.path.exists(cert_path):
os.remove(cert_path)
print(f"withdrew c{cluster_id:03d}" if before != len(doc["clusters"])
else f"c{cluster_id:03d} was not published")
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("cluster_id", type=int)
ap.add_argument("head", nargs="?", help="management address of the head node")
ap.add_argument("--remove", action="store_true")
args = ap.parse_args()
if not 1 <= args.cluster_id <= 254:
raise SystemExit("cluster id must be 1-254")
if args.remove:
remove(args.cluster_id)
elif not args.head:
raise SystemExit("publishing needs the head node's address")
else:
publish(args.cluster_id, args.head)
return 0
if __name__ == "__main__":
sys.exit(main())