Files
werewulf/scripts/B20-build-head-template.sh
T
ilkerandClaude Opus 5 04791c159e hpcod: deploy the tree the portal builds clusters with, in one step
`make deploy-portal` ships the binary, which is the smaller half. A
cluster is built by the Terraform module and configured by the
Ansible roles, and those live on the control plane as their own
copies. Copying them by hand, a file at a time, is how they came to
disagree - twice in one evening.

First the roles on the host had the single sign-on work and the copy
baked into the head template did not, so the first cluster the portal
built came up with no mTLS and answered its own tenant with 401. Then
the environment template was updated to pass control_plane_cert while
the module that receives it was not, and the next build died at plan
with "An argument named control_plane_cert is not expected here".

`make deploy-tree` sends the three together, and `make deploy` does
that and the binary. It leaves site.auto.tfvars alone - that is the
site's own, and it holds the Proxmox token - and never touches
terraform/envs, which is one working directory per cluster holding
its state.

It reads back what landed: the files hashed on the host and compared
here, not "scp exited 0". That caught its own first run - macOS tar
writes an AppleDouble twin beside every file with an extended
attribute, and GNU tar unpacked 37 of them as real files.

And the drift itself is now visible. The deploy stamps a REVISION
into the roles, which rides into the head template with them, so
/opt/hpcod/ansible/REVISION on any head says what configured it. B20
records what it baked, and the deploy compares: the repository being
current, and the control plane's copy being current, said nothing
about what clusters are actually built from.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 07:28:52 +02:00

110 lines
4.1 KiB
Bash
Executable File

#!/bin/bash
# Build the golden head node and convert it to a Proxmox template.
#
# Run on the control plane host. This is a build-time job, like the container
# image: everything it installs is paid for once here instead of by every
# tenant while a user waits.
#
# ./B20-build-head-template.sh
# ./B20-build-head-template.sh -v 9101 -t hpcod-head-v2
#
# The template deliberately carries no MUNGE key, SSH host keys or machine id.
# A shared MUNGE key would let any tenant authenticate to any other tenant's
# Slurm; the tenant playbook regenerates all three on first boot.
set -euo pipefail
SSH_KEY=${SSH_KEY:-/var/lib/hpcod/.ssh/id_ed25519}
PVE_HOST=${PVE_HOST:-10.0.0.10}
SSH_OPTS="-o BatchMode=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${SSH_KEY}"
PVE_SSH="ssh ${SSH_OPTS} root@${PVE_HOST}"
VMID=9100
NAME=hpcod-head
BUILD_IP=${BUILD_IP:-10.0.0.61}
GATEWAY=${GATEWAY:-10.0.0.1}
IMAGE=${IMAGE:-local:import/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2}
STORAGE=${STORAGE:-zfspool}
ANSIBLE_DIR=${ANSIBLE_DIR:-/var/lib/hpcod/ansible}
while getopts "v:t:i:" o; do
case $o in
v) VMID=$OPTARG ;;
t) NAME=$OPTARG ;;
i) BUILD_IP=$OPTARG ;;
*) echo "usage: $0 [-v vmid] [-t name] [-i build-ip]"; exit 1 ;;
esac
done
PUBKEY=$(cat "${SSH_KEY}.pub")
echo "==> removing any previous build VM"
$PVE_SSH "qm stop ${VMID} >/dev/null 2>&1 || true; qm destroy ${VMID} --purge >/dev/null 2>&1 || true"
echo "==> creating build VM ${VMID} at ${BUILD_IP}"
$PVE_SSH "set -e
printf '%s\n' '${PUBKEY}' > /tmp/hpcod-golden.pub
qm create ${VMID} --name ${NAME}-build --memory 4096 --cores 2 --cpu host \
--net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-single --ostype l26 \
--agent 1 --serial0 socket
qm set ${VMID} --scsi0 ${STORAGE}:0,import-from=${IMAGE},discard=on,ssd=1 >/dev/null
qm resize ${VMID} scsi0 40G >/dev/null
qm set ${VMID} --ide2 ${STORAGE}:cloudinit --boot order=scsi0 >/dev/null
qm set ${VMID} --ciuser ilker --sshkeys /tmp/hpcod-golden.pub \
--ipconfig0 ip=${BUILD_IP}/24,gw=${GATEWAY} --nameserver '1.1.1.1 8.8.8.8' --ciupgrade 0 >/dev/null
rm -f /tmp/hpcod-golden.pub
qm start ${VMID}"
echo "==> waiting for SSH"
for _ in $(seq 1 60); do
ssh ${SSH_OPTS} -o ConnectTimeout=5 "ilker@${BUILD_IP}" true 2>/dev/null && break
sleep 5
done
echo "==> installing the stack"
# A private mktemp file, not a fixed path in the shared /tmp: on a host with
# other users a predictable name is a symlink target, and fs.protected_regular
# blocks the write outright once someone else owns the leftover file.
INV_DIR=$(mktemp -d "${STATE_DIR:-/var/lib/hpcod}/.golden-inv.XXXXXX")
trap 'rm -rf "${INV_DIR}"' EXIT
INV="${INV_DIR}/inventory.yml"
cat > "${INV}" <<EOF
all:
children:
golden:
hosts:
${NAME}-build:
ansible_host: ${BUILD_IP}
ansible_user: ilker
ansible_become: true
ansible_ssh_private_key_file: ${SSH_KEY}
EOF
# --tags install selects only what belongs in a template; the cleanup play
# is tagged "always" so it still runs.
( cd "${ANSIBLE_DIR}" && ansible-playbook -i "${INV}" golden.yml --tags install )
echo "==> shutting down and converting to a template"
$PVE_SSH "set -e
qm shutdown ${VMID} --timeout 120 || qm stop ${VMID}
sleep 3
qm set ${VMID} --name ${NAME} >/dev/null
qm template ${VMID}"
# Record which revision of the roles this template baked.
#
# The template carries a copy of the automation - selfconfig puts it in
# /opt/hpcod/ansible and the head configures itself from that on first boot.
# So a role change reaches new clusters only after this script runs again,
# and until then nothing says otherwise: the repository is current, the
# control plane's copy is current, and the clusters are built from neither.
# That gap shipped a cluster with no mTLS and cost an evening.
if [ -f "${ANSIBLE_DIR}/REVISION" ]; then
cp "${ANSIBLE_DIR}/REVISION" /var/lib/hpcod/head-template-revision
else
echo "unknown $(date -u +%Y-%m-%dT%H:%M:%SZ)" > /var/lib/hpcod/head-template-revision
fi
echo
echo "TEMPLATE READY: ${VMID} (${NAME})"
echo " built from: $(cat /var/lib/hpcod/head-template-revision)"
echo "Point the Terraform module at it: head_template_vmid = ${VMID}"