Files
ilkerandClaude Opus 5 a6d453ae7b hpcod: application images, and a catalogue generated from them
The catalogue published one image. Filling it is not nine more
Containerfiles: it is a question of who compiles the software, against
which MPI, and how a second cluster gets it without paying for the
first one's build. image/apps holds the answer to the first of those -
one driver over one directory per application, with three recipes
(somebody else's image, one we assemble, or Spack against our own MPI)
and the rule for choosing between them.

The catalogue is generated from the recipes rather than kept beside
them, because an entry in it is a promise that a head node can pull the
image, and a promise nobody checks is how the first generated catalogue
came to point busybox at a tag that had never existed - guessed from
the upstream version instead of read from the registry. The generator
now asks the registry, and an application whose image is not there is
reported rather than offered.

Recipes are intent and published.json is measurement, kept apart so
that recording the hash of a push does not rewrite the reasoning in the
recipe that produced it.

Two entries to start: busybox, which is the cheapest way to prove the
whole delivery path, and stress-ng, which is what somebody actually
runs first on a cluster they have just been handed.

docs/22 is the Spack half of the question, from reading Chapar: what it
would give us, the three levels of integration, and the PMIx version to
settle on a real head before a single image is built against it.

Two things about the build host, both found by trying: Apptainer is not
installed, and Docker is, which sets FORWARD DROP and leaves podman
unable to build anything that needs the network.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 10:25:37 +02:00

4.8 KiB

Working rules for this repository

Verify the effect, never the intent

Every mistake that has cost real time in this repository has the same shape: an action was taken, the action reported success, the action had not actually done anything, and a measurement or a conclusion was built on top of it.

The exit code tells you the command ran. It does not tell you the state changed. These are different questions and only the second one matters.

After changing something, read the changed thing back. Not the command's output — the state itself.

# Wrong: the edit is assumed to have landed
sed -i 's/foo/bar/' file && next_step

# Right: the new state is confirmed before anything depends on it
sed -i 's/foo/bar/' file
grep -q bar file || { echo "edit did not land"; exit 1; }

This has happened here with sed (pattern did not match because the file used different spacing), with a Python str.replace (same cause, and the assertion checked only that the value was absent beforehand, not that it was present after), and with wwctl image import, which prints ERROR: image name exists and leaves the old image in place. In the last case a full boot measurement was taken against the wrong image and the resulting number was reported as real.

When writing a guard, assert the postcondition, not the precondition.

Assert what a measurement depends on, before measuring

A measurement of the wrong thing is worse than no measurement, because it gets reported and believed. Before timing anything, check that the change under test is actually in place, and abort loudly if it is not.

WW=$(ssh $HEAD "lsinitrd .../initramfs-*.img | grep -c wwinit")
[ "$WW" = "0" ] && { echo "ABORT: new image is not in place"; exit 1; }

The measurement chain here is expensive — a template build is ~4 minutes and a provision ~3 — so a wasted cycle is not free, and a wrong number wastes more than a cycle.

A completion check must not match itself

pgrep -f B20-build run inside bash -c '... pgrep -f B20-build ...' matches its own shell. It always reports "running", so the loop never exits and the work looks unfinished forever.

Match on something that cannot be the checker: a pid file, a marker the job writes on exit, or a pattern that excludes the checking process.

Completion checks must cover failure, not just success

Grepping a log for the success marker is silent when the job crashed, hung, or was killed — and silence is indistinguishable from "still working". Cover every terminal state:

grep -E "TEMPLATE READY|failed=[1-9]|ERROR!|fatal:" "$LOG"

Prefer a single command that exits when the condition is true (run in the background, one notification on exit) over a hand-rolled polling loop. Where a loop is unavoidable, remember that a dropped SSH connection yields an empty result, which is neither success nor failure — decide explicitly what that means instead of letting it fall through.

Batch changes into one verification cycle

One rebuild that validates four fixes beats four rebuilds. When a fix is found mid-cycle, stop the cycle, fold the fix in, and restart once.

Tools in this stack that fail quietly

  • wwctl profile set --nettagadd exits 0 and changes nothing. Read the profile back with wwctl profile list -a after writing it.
  • wwctl image import refuses an existing name and exits non-zero, but the surrounding pipeline may swallow it. Pass --force when replacing.
  • wwctl needs $HOME. systemd units do not set one, so a playbook that works over SSH fails as a service until Environment=HOME=/root is set.
  • Ansible finds ansible.cfg relative to the working directory. A playbook run from / silently uses stock defaults. Set ANSIBLE_CONFIG explicitly.
  • Playbook-adjacent group_vars/all.yml outranks an inventory's own group vars. A value left there overrides the inventory rather than defaulting behind it.
  • podman build --squash keeps the base image layers, so files deleted in later layers still ship. Use --squash-all when size is the point.
  • Podman on a host that also runs Docker builds nothing that needs the network. Docker sets iptables -P FORWARD DROP and whitelists only its own bridges; netavark accepts podman's subnet in its own nftables table, but a DROP in any table is final, so the packets die. podman pull still works, because podman does that itself over the host's network - so the base image arrives and the first RUN that fetches anything fails with a DNS or fetch error that names neither Docker nor the firewall. podman build --network=host is the one-line way past it; fixing it properly means an accept for podman0 in the legacy FORWARD chain, which does not survive a reboot unless it is saved.
  • Apptainer is not installed on the control plane host. image/apps/build-push.sh needs it to turn an OCI image into a SIF and refuses to start without it.