Files
werewulf/docs/23-application-lifecycle.md
T
ilkerandClaude Opus 5 b94c27667d hpcod: hpcod apps, which asks the catalogue and the record together
The portal publishes what can be asked for and the database records
what each cluster was given. Those two were connected exactly once, at
allocation, and never looked at together again - so the questions that
matter after an application is published had no way to be asked:

  which live clusters are running this
  has the tag moved since they were allocated
  which tags can the registry not delete

All three are answerable from data already being written. cluster_app
has had an index on name since it was created and nothing ever queried
it. This is the thing that asks.

prunable is the retention rule made operable. Two things protect a tag
and they guard different mistakes: the catalogue, because a new cluster
may still ask for it, and a live cluster's record, because that cluster
may be rebuilt - and a rebuild that cannot find its image does not fail
loudly, it builds a cluster running different software while every
record still says the same version. Not a retention window: a
constraint, in one query. Run against the real registry it protected
desktop:xfce, which no catalogue entry mentions and a live cluster
holds.

The registry client is here rather than in the build script although
that script asks the same question, because the other half of the
answer comes from the database - and assembled by hand from two
printouts it would eventually be assembled wrong. A repository that
cannot be listed makes the whole answer incomplete and says so, because
a partial list of "safe to delete" is worse than none.

The first version of `apps list` swallowed the error from its own usage
query and printed "-" in the CLUSTERS column, which reads as "nothing
holds this" - the answer an operator would act on when deciding a tag
is safe to delete. It printed that the first time it was run, against a
database one migration behind. The column now prints "?" when the
question could not be asked, because that and "0" are different facts.

docs/23 is the plan the five items came from, and what each turned out
to involve.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 10:26:07 +02:00

10 KiB

23 — The life of an application

Written for whoever maintains hpcod/. It assumes §22 and the code in internal/catalog, internal/store and image/apps.

Publishing an application now works: a recipe is built, verified against its own binaries, pushed, checked against the registry, and the portal picks up the new catalogue without a restart. That is the first minute of an application's life. This is about the next two years of it.

The one idea

An application and a cluster have different lifetimes, and every hard question here lives at the seam between them.

changes lives in
a catalogue entry monthly config/apps.yaml, from the recipes
what a cluster holds never, once allocated cluster_app, in PostgreSQL
a cluster fourteen days Proxmox

Those two records are connected exactly once, at allocation, and never looked at together again. That is correct as far as it goes — a cluster must not silently change underneath a running job — but it means every question worth asking about an application after it is published is currently unanswerable:

  • A CVE lands in GROMACS. Which live clusters are running it?
  • We published 2025.1. Who is still on 2024.4, and does anyone tell them?
  • The registry is at 400 GB. Which tags can be deleted without breaking a rebuild?

The schema already holds the answers. cluster_app has an index on name for exactly this, and nothing queries it. The whole of this plan is: make that seam visible, then act on it.

Four states, and what each transition owes

Today an entry is on or off — published: in app.yaml, overridden by whether the registry really holds the tag. That is two states where four are needed.

state offered? pullable? means
draft no no the recipe exists, the image does not
available yes yes the normal state
deprecated yes, with a warning yes a newer version exists; still supported
withdrawn no yes, for now do not start anything new on this

The states are cheap. What matters is what each transition is obliged to do, and this is where the work is:

add — already done. Build, verify binaries, push, read back, regenerate.

update — publish a new version; the old one becomes deprecated automatically, because a newer one exists. Running clusters do not move and must not. But the tenant of a cluster pinned to a deprecated version should be told once, on their cluster page, rather than finding out from a colleague.

deprecate — a statement, not an action. Needs a date and a reason, which are metadata and belong in the recipe.

withdraw — the interesting one. Not offered to new clusters, and the operator is handed the list of live clusters that hold it. A withdrawal that cannot name who is affected is a note to oneself.

delete from the registry — and here the database gives us a rule rather than a guess: a tag referenced by any live cluster's cluster_app row cannot be deleted. Not a retention window, not "keep the last three" — a correctness constraint, checkable in one query. A cluster that is rebuilt must get the bytes it was allocated with.

What has to be fixed before any of that

Four things found by looking, not by reading. Each is small; together they are the floor the rest stands on.

Versions are compared as text

Catalog.Lookup sorts candidates with a.Version < b.Version on strings, so asking for gromacs with 2024.9 and 2024.10 published returns 2024.9. Measured:

2024.9 and 2024.10 -> "2024.9"      wrong
1.9    and 1.10    -> "1.9"         wrong
0.9.0  and 0.18.07 -> "0.9.0"       wrong
2024.4 and 2025.1  -> "2025.1"      right, by luck

The whole meaning of an unversioned reference is "the newest one", and it is wrong the first time a minor version reaches ten. Everything about updating rests on this, so it is first.

A recipe holds one version; the catalogue holds many

Catalog already supports several entries under one name and picks the highest. image/apps/<name>/ cannot express that: one directory, one app.yaml, one version:. Updating GROMACS therefore means either losing 2024.4 or creating a parallel gromacs-2025/ directory that shares nothing with the first.

This is the layout decision the plan turns on. The recipe should carry a list of versions with their own build arguments, and the generator should emit one catalogue entry per version. Without it, "deprecated" has nowhere to live, because the old version has already been overwritten.

Images are named by a mutable tag

cluster_app.image records oras://…/gromacs:2024.4. Anyone who repushes that tag changes what every rebuilt cluster gets, silently, and the record that was supposed to freeze the cluster freezes nothing. The registry hands out a digest at push time and build-push.sh already pulls the image back to compare it — recording that digest and pulling by it costs almost nothing and turns the record into the guarantee it is meant to be.

Three catalogue fields are decoration

Module is read nowhere at all. SizeGB and Signed are read once each, to be copied into a JSON response, and no decision is taken from either.

Module is the worst of the three, because it marks the real gap: a tenant is handed /opt/apps/gromacs.sif and no way to run it. binaries: is already carried through the recipe into the catalogue and stops there. The apps role should turn those two fields into a wrapper per binary and an Lmod modulefile, so the answer to "how do I run this" is module load gromacs rather than an Apptainer incantation the user has to be taught.

Until that exists we ship images, not software.

The plan

Stage 1 — make what exists usable and true. Nothing new to reason about. All five are now done; what follows is what each turned out to involve.

  1. Numeric version comparison in Lookup, with the table above as the test.
  2. Multi-version recipes, so an update does not destroy its predecessor.
  3. Digest recorded at publish, carried into cluster_app, pulled by digest.
  4. Modulefiles and wrappers from module: and binaries:.
  5. hpcod apps — list, users <name>, verify. It reads data that is already written and is the only reason the cluster_app name index exists.

Stage 2 — the lifecycle proper. Done. Three things it settled that the plan above left open:

  • published: became status:. Two spellings of one fact in one file is worse than one awkward spelling, so the nine recipes were migrated rather than both being accepted. An unrecognised status is a hard error: a typo that silently meant "draft" would quietly unpublish an application.
  • A withdrawn entry stays in the catalogue file. Dropping it would make it indistinguishable from one that never existed, and that difference is exactly what an operator needs when a cluster is found still holding it. The portal is what stops it being offered, and Resolve refuses a pinned reference to it with the recorded reason rather than "no such application"
    • which would send the user hunting for a typo in a correct name.
  • Deprecated is offered, and says so. It means a newer version exists, not that this one has stopped working; refusing it would break the pin of everyone who named it deliberately. hpcod apps verify reports it and does not exit non-zero for it, so the command is usable from cron.

hpcod apps prunable is the retention rule made operable. Two things protect a tag and they guard different mistakes: the catalogue, because a new cluster may still ask for it, and a live cluster's record, because that cluster may be rebuilt. Run against the real registry it correctly protected desktop:xfce, which no catalogue entry mentions at all and a live cluster holds. A repository that cannot be listed makes the whole answer incomplete and is reported as such rather than skipped, because a partial list of "safe to delete" is worse than none.

Stage 3 — governance. Independent of both, and each item is its own decision rather than a consequence of this design:

  • Provenance: who published, when, from which revision, at which digest.
  • Signing, and dropping --allow-unsigned from the apps role. The Signed field is a promise nothing keeps.
  • A CVE gate on publication — see §22, where Chapar's checker is discussed.
  • Private and licensed applications — see §22. VASP and ANSYS cannot live in a shared registry, and that is a visibility model, not another recipe.

Two things were found only by running them, and both were the failure this repository keeps having: an action that reported success and had not changed what it claimed to. template does not create intermediate directories, so the modulefile task failed mid-provision with the machines already up. And hpcod apps list swallowed the error from its own usage query and printed - in the CLUSTERS column - which reads as "nothing holds this", the answer an operator would act on when deciding a tag is safe to delete. It printed that the first time it was run, against a database one migration behind. That column now prints ? when the question could not be asked.

Stage 1 also left one limit in place on purpose. The SIF path is still /opt/apps/<name>.sif because the ondemand role hard-codes /opt/apps/desktop.sif, so a cluster can hold only one version of an application and the control plane refuses a request naming two. The catalogue supports several; the head does not. Versioned SIF paths are what would lift it, and that is its own change with its own blast radius.

Recommended first: item 4, the modulefiles. It is the difference between delivering a file and delivering software, the fields for it already exist and are dead, and it is the only item a tenant would notice this week. Then item 1, because it is a wrong answer rather than a missing one.

Deliberately not planned

  • Moving a running cluster to a new version. A fourteen-day cluster should end on the software it started with. Tell the tenant; do not reach in.
  • Rollback of a published application. Publishing a lower version is already the whole of it, once versions compare numerically.
  • A live registry query from the portal. The catalogue is a file that only the platform team writes, deliberately (see internal/catalog). The generator asks the registry; the portal reads the file. Making the portal depend on the registry being up would trade a clear failure for a vague one.