The catalogue published one image. Filling it is not nine more Containerfiles: it is a question of who compiles the software, against which MPI, and how a second cluster gets it without paying for the first one's build. image/apps holds the answer to the first of those - one driver over one directory per application, with three recipes (somebody else's image, one we assemble, or Spack against our own MPI) and the rule for choosing between them. The catalogue is generated from the recipes rather than kept beside them, because an entry in it is a promise that a head node can pull the image, and a promise nobody checks is how the first generated catalogue came to point busybox at a tag that had never existed - guessed from the upstream version instead of read from the registry. The generator now asks the registry, and an application whose image is not there is reported rather than offered. Recipes are intent and published.json is measurement, kept apart so that recording the hash of a push does not rewrite the reasoning in the recipe that produced it. Two entries to start: busybox, which is the cheapest way to prove the whole delivery path, and stress-ng, which is what somebody actually runs first on a cluster they have just been handed. docs/22 is the Spack half of the question, from reading Chapar: what it would give us, the three levels of integration, and the PMIx version to settle on a real head before a single image is built against it. Two things about the build host, both found by trying: Apptainer is not installed, and Docker is, which sets FORWARD DROP and leaves podman unable to build anything that needs the network. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4.8 KiB
Working rules for this repository
Verify the effect, never the intent
Every mistake that has cost real time in this repository has the same shape: an action was taken, the action reported success, the action had not actually done anything, and a measurement or a conclusion was built on top of it.
The exit code tells you the command ran. It does not tell you the state changed. These are different questions and only the second one matters.
After changing something, read the changed thing back. Not the command's output — the state itself.
# Wrong: the edit is assumed to have landed
sed -i 's/foo/bar/' file && next_step
# Right: the new state is confirmed before anything depends on it
sed -i 's/foo/bar/' file
grep -q bar file || { echo "edit did not land"; exit 1; }
This has happened here with sed (pattern did not match because the file
used different spacing), with a Python str.replace (same cause, and the
assertion checked only that the value was absent beforehand, not that it was
present after), and with wwctl image import, which prints
ERROR: image name exists and leaves the old image in place. In the last
case a full boot measurement was taken against the wrong image and the
resulting number was reported as real.
When writing a guard, assert the postcondition, not the precondition.
Assert what a measurement depends on, before measuring
A measurement of the wrong thing is worse than no measurement, because it gets reported and believed. Before timing anything, check that the change under test is actually in place, and abort loudly if it is not.
WW=$(ssh $HEAD "lsinitrd .../initramfs-*.img | grep -c wwinit")
[ "$WW" = "0" ] && { echo "ABORT: new image is not in place"; exit 1; }
The measurement chain here is expensive — a template build is ~4 minutes and a provision ~3 — so a wasted cycle is not free, and a wrong number wastes more than a cycle.
A completion check must not match itself
pgrep -f B20-build run inside bash -c '... pgrep -f B20-build ...'
matches its own shell. It always reports "running", so the loop never exits
and the work looks unfinished forever.
Match on something that cannot be the checker: a pid file, a marker the job writes on exit, or a pattern that excludes the checking process.
Completion checks must cover failure, not just success
Grepping a log for the success marker is silent when the job crashed, hung, or was killed — and silence is indistinguishable from "still working". Cover every terminal state:
grep -E "TEMPLATE READY|failed=[1-9]|ERROR!|fatal:" "$LOG"
Prefer a single command that exits when the condition is true (run in the background, one notification on exit) over a hand-rolled polling loop. Where a loop is unavoidable, remember that a dropped SSH connection yields an empty result, which is neither success nor failure — decide explicitly what that means instead of letting it fall through.
Batch changes into one verification cycle
One rebuild that validates four fixes beats four rebuilds. When a fix is found mid-cycle, stop the cycle, fold the fix in, and restart once.
Tools in this stack that fail quietly
wwctl profile set --nettagaddexits 0 and changes nothing. Read the profile back withwwctl profile list -aafter writing it.wwctl image importrefuses an existing name and exits non-zero, but the surrounding pipeline may swallow it. Pass--forcewhen replacing.wwctlneeds$HOME. systemd units do not set one, so a playbook that works over SSH fails as a service untilEnvironment=HOME=/rootis set.- Ansible finds
ansible.cfgrelative to the working directory. A playbook run from/silently uses stock defaults. SetANSIBLE_CONFIGexplicitly. - Playbook-adjacent
group_vars/all.ymloutranks an inventory's own group vars. A value left there overrides the inventory rather than defaulting behind it. podman build --squashkeeps the base image layers, so files deleted in later layers still ship. Use--squash-allwhen size is the point.- Podman on a host that also runs Docker builds nothing that needs the
network. Docker sets
iptables -P FORWARD DROPand whitelists only its own bridges; netavark accepts podman's subnet in its own nftables table, but a DROP in any table is final, so the packets die.podman pullstill works, because podman does that itself over the host's network - so the base image arrives and the firstRUNthat fetches anything fails with a DNS or fetch error that names neither Docker nor the firewall.podman build --network=hostis the one-line way past it; fixing it properly means an accept forpodman0in the legacy FORWARD chain, which does not survive a reboot unless it is saved. - Apptainer is not installed on the control plane host.
image/apps/build-push.shneeds it to turn an OCI image into a SIF and refuses to start without it.