Files
werewulf/scripts/C20-deploy-tree.sh
T
ilkerandClaude Opus 5 6dc2826c17 hpcod: the life of an application, carried past the moment it is published
Publishing worked; everything after it did not. Four things were wrong
and each was silent in its own way.

Versions were compared as text, so asking for "gromacs" with 2024.9 and
2024.10 published returned 2024.9. The whole meaning of an unversioned
reference is "the newest one", and it was a wrong answer rather than a
missing one - invisible until somebody noticed they had spent a week on
last year's build.

A recipe could hold one version while the catalogue could hold many, so
an update either destroyed what it replaced or grew a parallel
directory sharing nothing with the first. Recipes now carry a list, and
the single-version form is read into the same shape, so nothing
downstream knows which was written.

An image was named by a mutable tag. Whoever repushes gromacs:2024.4
changed what every rebuilt cluster received, silently, and the record
meant to freeze the cluster froze only its opinion of it. Two hashes
are now recorded because they are checkable in different places: the
manifest digest answers "has this moved" from the control plane, and
the SIF hash is the only one a head can check with no network and no
credential - which matters, because the head is where a wrong image
does harm. Pulling by digest is deliberately not done: whether this
Apptainer accepts an @sha256: reference is not established, and a
provision failing at its last task over untested syntax would be worse
than the problem.

Module and binaries had been in the catalogue since it was written and
were read by nothing - Module was referenced nowhere in the program. So
a tenant was handed /opt/apps/gromacs.sif and left to work out
Apptainer. They now become a wrapper per binary and an Lmod modulefile,
and the answer is `module load gromacs`. Whether to pass --nv is
decided on the node at run time: passing it without a driver fails
outright, and omitting it where there is one leaves the job on the CPU
saying nothing at all, which is the dangerous one.

Four states replace `published:`. A withdrawn entry stays in the
catalogue file, because dropping it would make it indistinguishable
from one that never existed - and that difference is what an operator
needs when a cluster is found still holding it. A pinned reference to
one is refused with the recorded reason instead of "no application",
which would send the user hunting for a typo in a correct name.

The portal re-reads the catalogue when it changes, so publishing an
application no longer needs a restart - the step that is invisible when
forgotten. C20 ships the catalogue, which nothing did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 10:25:54 +02:00

208 lines
9.5 KiB
Bash
Executable File

#!/bin/bash
# Deploys what the control plane builds clusters *with*: the Ansible roles,
# the Terraform module, and the environment template.
#
# ./scripts/C20-deploy-tree.sh # to the usual host
# DEPLOY_HOST=other ./scripts/C20-deploy-tree.sh
#
# `make deploy-portal` ships the binary. That is the smaller half: a cluster
# is built by the Terraform module and configured by the Ansible roles, and
# those live on the host as their own copies. Copying them by hand, a file at
# a time, is how they came to disagree - twice.
#
# The first time, the roles on the host had the single sign-on work and the
# copy baked into the head template did not, so the first cluster the portal
# built came up with no mTLS and answered its own tenant with 401. The second
# time, the environment template was updated to pass control_plane_cert while
# the module that receives it was not, and the build died at plan with
# "An argument named control_plane_cert is not expected here".
#
# Both were partial deployments. This sends the three together or not at all,
# and reads back what landed.
#
# The scripts go too. B20 bakes the roles into the head template and B40
# publishes a cluster by hand; both live on the host as their own copies,
# and the copy is what runs. Sending B20 by scp just before running it is
# the same mistake one step earlier.
#
# The application catalogue goes too, to /etc/hpcod/apps.yaml. It is
# generated by image/apps/build-push.sh --catalog and is not site-specific -
# it is the platform team's published list, which is exactly the kind of
# thing this script exists to keep from drifting. The portal re-reads it when
# it changes, so this needs no restart.
#
# What it deliberately does not touch:
# site.auto.tfvars the site's own variables, including the Proxmox token
# terraform/envs/* one working directory per cluster, holding its state
# sites.yaml, users, env, db.env the site's own configuration; B10's
set -euo pipefail
HERE=$(cd "$(dirname "$0")/.." && pwd)
SRC=${SRC:-${HERE}/hpcod}
DEPLOY_HOST=${DEPLOY_HOST:-web-frontend}
ROOT=${ROOT:-/var/lib/hpcod}
ETC=${ETC:-/etc/hpcod}
OWNER=${OWNER:-hpcod:hpcod}
APPS=${APPS:-${HERE}/hpcod/config/apps.yaml}
REV=$(cd "$HERE" && git describe --tags --always --dirty 2>/dev/null || echo unknown)
# The revision that last changed the roles, which is the only one the head
# template can be out of date with respect to. Comparing the overall
# revision instead would call the template stale after any commit at all -
# a warning that is always on is one nobody reads.
ROLES_REV=$(cd "$HERE" && git log -1 --format=%h -- hpcod/ansible 2>/dev/null || echo unknown)
case "$REV" in *-dirty) ROLES_REV="$ROLES_REV+dirty" ;; esac
STAMP=$(printf 'revision: %s\nroles: %s\nbuilt: %s\n' \
"$REV" "$ROLES_REV" "$(date -u +%Y-%m-%dT%H:%M:%SZ)")
echo "==> ${DEPLOY_HOST}:${ROOT} <- ${REV}"
# --- 1) Pack ---------------------------------------------------------------
# One archive, so a failure between two copies cannot leave half of it there.
TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT
# macOS tar writes an AppleDouble "._name" beside every file carrying an
# extended attribute, and GNU tar on the far side unpacks those as real
# files. The read-back below caught a tree with a ._ twin for almost every
# entry in it. Harmless to Ansible, but it is not what was sent.
export COPYFILE_DISABLE=1
mkdir -p "$TMP/ansible" "$TMP/modules/tenant" "$TMP/template" "$TMP/scripts"
tar cf - --exclude "._*" -C "$SRC/ansible" . | tar xf - -C "$TMP/ansible"
tar cf - --exclude "._*" -C "$SRC/terraform/modules/tenant" . | tar xf - -C "$TMP/modules/tenant"
# Only the two files the template is made of. Everything else in envs/lab is
# a local working copy: state, and a tfvars holding this developer's token.
cp "$SRC/terraform/envs/lab/main.tf" "$SRC/terraform/envs/lab/variables.tf" "$TMP/template/"
# The B and C scripts: the ones the control plane runs. Matched without an
# extension, because they are a mix of shell and Python and a glob on .sh
# quietly left B30 behind. The numbered scripts below B are the ones that
# built the original lab by hand, and have no business here.
for f in "$HERE"/scripts/[BC][0-9]*; do
[ -f "$f" ] && cp "$f" "$TMP/scripts/"
done
# The stamp rides into the head template with the roles, so a head node can
# always say which revision configured it: /opt/hpcod/ansible/REVISION.
printf '%s\n' "$STAMP" > "$TMP/ansible/REVISION"
printf '%s\n' "$STAMP" > "$TMP/template/REVISION"
tar czf "$TMP/tree.tgz" --exclude "._*" -C "$TMP" ansible modules template scripts
# What we expect to find afterwards, to compare against what is there.
( cd "$TMP" && find ansible modules template scripts -type f ! -name '*.tgz' -print0 \
| sort -z | xargs -0 shasum -a 256 ) > "$TMP/want.sums"
echo " $(wc -l < "$TMP/want.sums" | tr -d ' ') files"
# --- 2) Ship and swap ------------------------------------------------------
scp -q "$TMP/tree.tgz" "${DEPLOY_HOST}:/tmp/hpcod-tree.tgz"
ssh "$DEPLOY_HOST" "sudo ROOT='$ROOT' OWNER='$OWNER' bash -s" <<'REMOTE'
set -euo pipefail
S=/tmp/hpcod-unpack
rm -rf "$S" && mkdir -p "$S"
tar xzf /tmp/hpcod-tree.tgz -C "$S"
stamp=$(date +%Y%m%d-%H%M%S)
swap() { # swap <unpacked> <live>
[ -d "$2" ] && mv "$2" "$2.bak-$stamp"
mkdir -p "$(dirname "$2")"
mv "$1" "$2"
}
swap "$S/ansible" "$ROOT/ansible"
swap "$S/modules/tenant" "$ROOT/terraform/modules/tenant"
swap "$S/scripts" "$ROOT/scripts"
# The template directory is not replaced: site.auto.tfvars lives in it and is
# the site's own, not ours. Only the files we sent are overwritten.
mkdir -p "$ROOT/terraform/template"
for f in "$S"/template/*; do
cp -a "$f" "$ROOT/terraform/template/$(basename "$f")"
done
chmod 0755 "$ROOT"/scripts/*
chown -R "$OWNER" "$ROOT/ansible" "$ROOT/terraform/modules/tenant" "$ROOT/terraform/template" "$ROOT/scripts"
# Keep the last three of our own backups and drop the rest. Every run makes
# one, so without this the directory fills with copies of a tree that is in
# git anyway - the backups are there to get back to this morning, not to
# last month.
for base in "$ROOT/ansible" "$ROOT/terraform/modules/tenant" "$ROOT/scripts"; do
ls -1dt "$base".bak-* 2>/dev/null | tail -n +4 | while read -r old; do rm -rf "$old"; done
done
rm -rf "$S" /tmp/hpcod-tree.tgz
REMOTE
# --- 2b) The application catalogue ------------------------------------------
# Separate from the tarball because it lands outside $ROOT, beside the site's
# own configuration rather than inside the tree the clusters are built from.
if [ -f "$APPS" ]; then
echo "==> ${DEPLOY_HOST}:${ETC}/apps.yaml"
scp -q "$APPS" "${DEPLOY_HOST}:/tmp/hpcod-apps.yaml"
ssh "$DEPLOY_HOST" "sudo ETC='$ETC' OWNER='$OWNER' bash -s" <<'REMOTE_APPS'
set -euo pipefail
install -d -m 0750 "$ETC"
# Written to a temporary name and renamed, so a portal reading the file
# during the copy sees the old one or the new one and never half of one.
install -m 0640 -o root -g "${OWNER#*:}" /tmp/hpcod-apps.yaml "$ETC/.apps.yaml.new"
mv "$ETC/.apps.yaml.new" "$ETC/apps.yaml"
rm -f /tmp/hpcod-apps.yaml
REMOTE_APPS
# Read back the bytes, not the exit code. scp reporting success and the
# portal offering the previous list is the failure this is here to catch.
want=$(shasum -a 256 "$APPS" | cut -d' ' -f1)
got=$(ssh "$DEPLOY_HOST" "sudo sha256sum $ETC/apps.yaml 2>/dev/null | cut -d' ' -f1" || true)
if [ "$want" != "$got" ]; then
echo " CATALOGUE DID NOT LAND (local $want, remote ${got:-nothing})"
exit 1
fi
n=$(grep -c '^ - name:\|^- name:' "$APPS" || true)
echo " ${n} entries, same bytes; no restart needed - the portal re-reads it"
else
echo "==> no application catalogue at $APPS; not deploying one"
fi
# --- 3) Read it back -------------------------------------------------------
# Not "scp said 0". The files on the host, hashed there and compared here.
echo "==> geri okuma"
ssh "$DEPLOY_HOST" "sudo bash -c '
cd \"$ROOT\" 2>/dev/null || exit 1
{ find ansible -type f -printf \"%p\0\";
find terraform/modules/tenant -type f -printf \"%p\0\";
find scripts -type f -printf \"%p\0\";
for f in terraform/template/main.tf terraform/template/variables.tf terraform/template/REVISION; do
[ -f \"\$f\" ] && printf \"%s\0\" \"\$f\"; done; } | sort -z | xargs -0 sha256sum
'" 2>/dev/null \
| sed -e 's# terraform/modules/tenant/# modules/tenant/#' \
-e 's# terraform/template/# template/#' \
| sort -k2 > /tmp/got.sums
sort -k2 "$TMP/want.sums" > /tmp/want.sorted
if diff -q /tmp/want.sorted /tmp/got.sums >/dev/null; then
echo " $(wc -l < /tmp/got.sums | tr -d ' ') dosya, hepsi ayni"
else
echo " DEPLOY DID NOT LAND - what differs:"
diff /tmp/want.sorted /tmp/got.sums | head -20
exit 1
fi
# --- 4) Say what is now stale ---------------------------------------------
# The head template bakes a copy of the roles at build time, so deploying
# them here does nothing for clusters until the template is rebuilt. Saying
# so is the whole reason the stamp exists.
baked=$(ssh "$DEPLOY_HOST" "sudo awk '/^roles:/{print \$2}' $ROOT/head-template-revision 2>/dev/null" || true)
echo "==> roller: $ROLES_REV (surum $REV)"
if [ -z "$baked" ]; then
echo " bas dugum sablonu: bilinmiyor (B20 bunu kaydetmeden once derlenmis)"
echo " ./scripts/B20-build-head-template.sh - yoksa yeni kumeler eski rolleri alir"
elif [ "$baked" != "$ROLES_REV" ]; then
echo " bas dugum sablonu roller $baked ile derlenmis"
echo " ./scripts/B20-build-head-template.sh - sablon bu rollerden eski"
else
echo " bas dugum sablonu ayni rollerden derlenmis"
fi