`make deploy-portal` ships the binary, which is the smaller half. A cluster is built by the Terraform module and configured by the Ansible roles, and those live on the control plane as their own copies. Copying them by hand, a file at a time, is how they came to disagree - twice in one evening. First the roles on the host had the single sign-on work and the copy baked into the head template did not, so the first cluster the portal built came up with no mTLS and answered its own tenant with 401. Then the environment template was updated to pass control_plane_cert while the module that receives it was not, and the next build died at plan with "An argument named control_plane_cert is not expected here". `make deploy-tree` sends the three together, and `make deploy` does that and the binary. It leaves site.auto.tfvars alone - that is the site's own, and it holds the Proxmox token - and never touches terraform/envs, which is one working directory per cluster holding its state. It reads back what landed: the files hashed on the host and compared here, not "scp exited 0". That caught its own first run - macOS tar writes an AppleDouble twin beside every file with an extended attribute, and GNU tar unpacked 37 of them as real files. And the drift itself is now visible. The deploy stamps a REVISION into the roles, which rides into the head template with them, so /opt/hpcod/ansible/REVISION on any head says what configured it. B20 records what it baked, and the deploy compares: the repository being current, and the control plane's copy being current, said nothing about what clusters are actually built from. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
110 lines
4.1 KiB
Bash
Executable File
110 lines
4.1 KiB
Bash
Executable File
#!/bin/bash
|
|
# Build the golden head node and convert it to a Proxmox template.
|
|
#
|
|
# Run on the control plane host. This is a build-time job, like the container
|
|
# image: everything it installs is paid for once here instead of by every
|
|
# tenant while a user waits.
|
|
#
|
|
# ./B20-build-head-template.sh
|
|
# ./B20-build-head-template.sh -v 9101 -t hpcod-head-v2
|
|
#
|
|
# The template deliberately carries no MUNGE key, SSH host keys or machine id.
|
|
# A shared MUNGE key would let any tenant authenticate to any other tenant's
|
|
# Slurm; the tenant playbook regenerates all three on first boot.
|
|
set -euo pipefail
|
|
|
|
SSH_KEY=${SSH_KEY:-/var/lib/hpcod/.ssh/id_ed25519}
|
|
PVE_HOST=${PVE_HOST:-10.0.0.10}
|
|
SSH_OPTS="-o BatchMode=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${SSH_KEY}"
|
|
PVE_SSH="ssh ${SSH_OPTS} root@${PVE_HOST}"
|
|
|
|
VMID=9100
|
|
NAME=hpcod-head
|
|
BUILD_IP=${BUILD_IP:-10.0.0.61}
|
|
GATEWAY=${GATEWAY:-10.0.0.1}
|
|
IMAGE=${IMAGE:-local:import/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2}
|
|
STORAGE=${STORAGE:-zfspool}
|
|
ANSIBLE_DIR=${ANSIBLE_DIR:-/var/lib/hpcod/ansible}
|
|
|
|
while getopts "v:t:i:" o; do
|
|
case $o in
|
|
v) VMID=$OPTARG ;;
|
|
t) NAME=$OPTARG ;;
|
|
i) BUILD_IP=$OPTARG ;;
|
|
*) echo "usage: $0 [-v vmid] [-t name] [-i build-ip]"; exit 1 ;;
|
|
esac
|
|
done
|
|
|
|
PUBKEY=$(cat "${SSH_KEY}.pub")
|
|
|
|
echo "==> removing any previous build VM"
|
|
$PVE_SSH "qm stop ${VMID} >/dev/null 2>&1 || true; qm destroy ${VMID} --purge >/dev/null 2>&1 || true"
|
|
|
|
echo "==> creating build VM ${VMID} at ${BUILD_IP}"
|
|
$PVE_SSH "set -e
|
|
printf '%s\n' '${PUBKEY}' > /tmp/hpcod-golden.pub
|
|
qm create ${VMID} --name ${NAME}-build --memory 4096 --cores 2 --cpu host \
|
|
--net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-single --ostype l26 \
|
|
--agent 1 --serial0 socket
|
|
qm set ${VMID} --scsi0 ${STORAGE}:0,import-from=${IMAGE},discard=on,ssd=1 >/dev/null
|
|
qm resize ${VMID} scsi0 40G >/dev/null
|
|
qm set ${VMID} --ide2 ${STORAGE}:cloudinit --boot order=scsi0 >/dev/null
|
|
qm set ${VMID} --ciuser ilker --sshkeys /tmp/hpcod-golden.pub \
|
|
--ipconfig0 ip=${BUILD_IP}/24,gw=${GATEWAY} --nameserver '1.1.1.1 8.8.8.8' --ciupgrade 0 >/dev/null
|
|
rm -f /tmp/hpcod-golden.pub
|
|
qm start ${VMID}"
|
|
|
|
echo "==> waiting for SSH"
|
|
for _ in $(seq 1 60); do
|
|
ssh ${SSH_OPTS} -o ConnectTimeout=5 "ilker@${BUILD_IP}" true 2>/dev/null && break
|
|
sleep 5
|
|
done
|
|
|
|
echo "==> installing the stack"
|
|
# A private mktemp file, not a fixed path in the shared /tmp: on a host with
|
|
# other users a predictable name is a symlink target, and fs.protected_regular
|
|
# blocks the write outright once someone else owns the leftover file.
|
|
INV_DIR=$(mktemp -d "${STATE_DIR:-/var/lib/hpcod}/.golden-inv.XXXXXX")
|
|
trap 'rm -rf "${INV_DIR}"' EXIT
|
|
INV="${INV_DIR}/inventory.yml"
|
|
cat > "${INV}" <<EOF
|
|
all:
|
|
children:
|
|
golden:
|
|
hosts:
|
|
${NAME}-build:
|
|
ansible_host: ${BUILD_IP}
|
|
ansible_user: ilker
|
|
ansible_become: true
|
|
ansible_ssh_private_key_file: ${SSH_KEY}
|
|
EOF
|
|
# --tags install selects only what belongs in a template; the cleanup play
|
|
# is tagged "always" so it still runs.
|
|
( cd "${ANSIBLE_DIR}" && ansible-playbook -i "${INV}" golden.yml --tags install )
|
|
|
|
echo "==> shutting down and converting to a template"
|
|
$PVE_SSH "set -e
|
|
qm shutdown ${VMID} --timeout 120 || qm stop ${VMID}
|
|
sleep 3
|
|
qm set ${VMID} --name ${NAME} >/dev/null
|
|
qm template ${VMID}"
|
|
|
|
# Record which revision of the roles this template baked.
|
|
#
|
|
# The template carries a copy of the automation - selfconfig puts it in
|
|
# /opt/hpcod/ansible and the head configures itself from that on first boot.
|
|
# So a role change reaches new clusters only after this script runs again,
|
|
# and until then nothing says otherwise: the repository is current, the
|
|
# control plane's copy is current, and the clusters are built from neither.
|
|
# That gap shipped a cluster with no mTLS and cost an evening.
|
|
if [ -f "${ANSIBLE_DIR}/REVISION" ]; then
|
|
cp "${ANSIBLE_DIR}/REVISION" /var/lib/hpcod/head-template-revision
|
|
else
|
|
echo "unknown $(date -u +%Y-%m-%dT%H:%M:%SZ)" > /var/lib/hpcod/head-template-revision
|
|
fi
|
|
|
|
echo
|
|
echo "TEMPLATE READY: ${VMID} (${NAME})"
|
|
echo " built from: $(cat /var/lib/hpcod/head-template-revision)"
|
|
echo "Point the Terraform module at it: head_template_vmid = ${VMID}"
|