The portal publishes what can be asked for and the database records what each cluster was given. Those two were connected exactly once, at allocation, and never looked at together again - so the questions that matter after an application is published had no way to be asked: which live clusters are running this has the tag moved since they were allocated which tags can the registry not delete All three are answerable from data already being written. cluster_app has had an index on name since it was created and nothing ever queried it. This is the thing that asks. prunable is the retention rule made operable. Two things protect a tag and they guard different mistakes: the catalogue, because a new cluster may still ask for it, and a live cluster's record, because that cluster may be rebuilt - and a rebuild that cannot find its image does not fail loudly, it builds a cluster running different software while every record still says the same version. Not a retention window: a constraint, in one query. Run against the real registry it protected desktop:xfce, which no catalogue entry mentions and a live cluster holds. The registry client is here rather than in the build script although that script asks the same question, because the other half of the answer comes from the database - and assembled by hand from two printouts it would eventually be assembled wrong. A repository that cannot be listed makes the whole answer incomplete and says so, because a partial list of "safe to delete" is worse than none. The first version of `apps list` swallowed the error from its own usage query and printed "-" in the CLUSTERS column, which reads as "nothing holds this" - the answer an operator would act on when deciding a tag is safe to delete. It printed that the first time it was run, against a database one migration behind. The column now prints "?" when the question could not be asked, because that and "0" are different facts. docs/23 is the plan the five items came from, and what each turned out to involve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
10 KiB
23 — The life of an application
Written for whoever maintains hpcod/. It assumes §22 and the code in
internal/catalog, internal/store and image/apps.
Publishing an application now works: a recipe is built, verified against its own binaries, pushed, checked against the registry, and the portal picks up the new catalogue without a restart. That is the first minute of an application's life. This is about the next two years of it.
The one idea
An application and a cluster have different lifetimes, and every hard question here lives at the seam between them.
| changes | lives in | |
|---|---|---|
| a catalogue entry | monthly | config/apps.yaml, from the recipes |
| what a cluster holds | never, once allocated | cluster_app, in PostgreSQL |
| a cluster | fourteen days | Proxmox |
Those two records are connected exactly once, at allocation, and never looked at together again. That is correct as far as it goes — a cluster must not silently change underneath a running job — but it means every question worth asking about an application after it is published is currently unanswerable:
- A CVE lands in GROMACS. Which live clusters are running it?
- We published 2025.1. Who is still on 2024.4, and does anyone tell them?
- The registry is at 400 GB. Which tags can be deleted without breaking a rebuild?
The schema already holds the answers. cluster_app has an index on name
for exactly this, and nothing queries it. The whole of this plan is: make
that seam visible, then act on it.
Four states, and what each transition owes
Today an entry is on or off — published: in app.yaml, overridden by
whether the registry really holds the tag. That is two states where four are
needed.
| state | offered? | pullable? | means |
|---|---|---|---|
draft |
no | no | the recipe exists, the image does not |
available |
yes | yes | the normal state |
deprecated |
yes, with a warning | yes | a newer version exists; still supported |
withdrawn |
no | yes, for now | do not start anything new on this |
The states are cheap. What matters is what each transition is obliged to do, and this is where the work is:
add — already done. Build, verify binaries, push, read back, regenerate.
update — publish a new version; the old one becomes deprecated
automatically, because a newer one exists. Running clusters do not move and
must not. But the tenant of a cluster pinned to a deprecated version should
be told once, on their cluster page, rather than finding out from a
colleague.
deprecate — a statement, not an action. Needs a date and a reason, which are metadata and belong in the recipe.
withdraw — the interesting one. Not offered to new clusters, and the operator is handed the list of live clusters that hold it. A withdrawal that cannot name who is affected is a note to oneself.
delete from the registry — and here the database gives us a rule rather
than a guess: a tag referenced by any live cluster's cluster_app row
cannot be deleted. Not a retention window, not "keep the last three" — a
correctness constraint, checkable in one query. A cluster that is rebuilt
must get the bytes it was allocated with.
What has to be fixed before any of that
Four things found by looking, not by reading. Each is small; together they are the floor the rest stands on.
Versions are compared as text
Catalog.Lookup sorts candidates with a.Version < b.Version on strings, so
asking for gromacs with 2024.9 and 2024.10 published returns 2024.9.
Measured:
2024.9 and 2024.10 -> "2024.9" wrong
1.9 and 1.10 -> "1.9" wrong
0.9.0 and 0.18.07 -> "0.9.0" wrong
2024.4 and 2025.1 -> "2025.1" right, by luck
The whole meaning of an unversioned reference is "the newest one", and it is wrong the first time a minor version reaches ten. Everything about updating rests on this, so it is first.
A recipe holds one version; the catalogue holds many
Catalog already supports several entries under one name and picks the
highest. image/apps/<name>/ cannot express that: one directory, one
app.yaml, one version:. Updating GROMACS therefore means either losing
2024.4 or creating a parallel gromacs-2025/ directory that shares nothing
with the first.
This is the layout decision the plan turns on. The recipe should carry a list of versions with their own build arguments, and the generator should emit one catalogue entry per version. Without it, "deprecated" has nowhere to live, because the old version has already been overwritten.
Images are named by a mutable tag
cluster_app.image records oras://…/gromacs:2024.4. Anyone who repushes
that tag changes what every rebuilt cluster gets, silently, and the record
that was supposed to freeze the cluster freezes nothing. The registry hands
out a digest at push time and build-push.sh already pulls the image back to
compare it — recording that digest and pulling by it costs almost nothing and
turns the record into the guarantee it is meant to be.
Three catalogue fields are decoration
Module is read nowhere at all. SizeGB and Signed are read once each, to
be copied into a JSON response, and no decision is taken from either.
Module is the worst of the three, because it marks the real gap: a tenant
is handed /opt/apps/gromacs.sif and no way to run it. binaries: is
already carried through the recipe into the catalogue and stops there. The
apps role should turn those two fields into a wrapper per binary and an
Lmod modulefile, so the answer to "how do I run this" is module load gromacs rather than an Apptainer incantation the user has to be taught.
Until that exists we ship images, not software.
The plan
Stage 1 — make what exists usable and true. Nothing new to reason about. All five are now done; what follows is what each turned out to involve.
- Numeric version comparison in
Lookup, with the table above as the test. - Multi-version recipes, so an update does not destroy its predecessor.
- Digest recorded at publish, carried into
cluster_app, pulled by digest. - Modulefiles and wrappers from
module:andbinaries:. hpcod apps—list,users <name>,verify. It reads data that is already written and is the only reason thecluster_appname index exists.
Stage 2 — the lifecycle proper. Done. Three things it settled that the plan above left open:
published:becamestatus:. Two spellings of one fact in one file is worse than one awkward spelling, so the nine recipes were migrated rather than both being accepted. An unrecognised status is a hard error: a typo that silently meant "draft" would quietly unpublish an application.- A withdrawn entry stays in the catalogue file. Dropping it would make
it indistinguishable from one that never existed, and that difference is
exactly what an operator needs when a cluster is found still holding it.
The portal is what stops it being offered, and
Resolverefuses a pinned reference to it with the recorded reason rather than "no such application"- which would send the user hunting for a typo in a correct name.
- Deprecated is offered, and says so. It means a newer version exists,
not that this one has stopped working; refusing it would break the pin of
everyone who named it deliberately.
hpcod apps verifyreports it and does not exit non-zero for it, so the command is usable from cron.
hpcod apps prunable is the retention rule made operable. Two things protect
a tag and they guard different mistakes: the catalogue, because a new cluster
may still ask for it, and a live cluster's record, because that cluster may
be rebuilt. Run against the real registry it correctly protected
desktop:xfce, which no catalogue entry mentions at all and a live cluster
holds. A repository that cannot be listed makes the whole answer incomplete
and is reported as such rather than skipped, because a partial list of "safe
to delete" is worse than none.
Stage 3 — governance. Independent of both, and each item is its own decision rather than a consequence of this design:
- Provenance: who published, when, from which revision, at which digest.
- Signing, and dropping
--allow-unsignedfrom theappsrole. TheSignedfield is a promise nothing keeps. - A CVE gate on publication — see §22, where Chapar's checker is discussed.
- Private and licensed applications — see §22. VASP and ANSYS cannot live in a shared registry, and that is a visibility model, not another recipe.
Two things were found only by running them, and both were the failure this
repository keeps having: an action that reported success and had not changed
what it claimed to. template does not create intermediate directories, so
the modulefile task failed mid-provision with the machines already up. And
hpcod apps list swallowed the error from its own usage query and printed
- in the CLUSTERS column - which reads as "nothing holds this", the answer
an operator would act on when deciding a tag is safe to delete. It printed
that the first time it was run, against a database one migration behind.
That column now prints ? when the question could not be asked.
Stage 1 also left one limit in place on purpose. The SIF path is still
/opt/apps/<name>.sif because the ondemand role hard-codes
/opt/apps/desktop.sif, so a cluster can hold only one version of an
application and the control plane refuses a request naming two. The
catalogue supports several; the head does not. Versioned SIF paths are what
would lift it, and that is its own change with its own blast radius.
Recommended first: item 4, the modulefiles. It is the difference between delivering a file and delivering software, the fields for it already exist and are dead, and it is the only item a tenant would notice this week. Then item 1, because it is a wrong answer rather than a missing one.
Deliberately not planned
- Moving a running cluster to a new version. A fourteen-day cluster should end on the software it started with. Tell the tenant; do not reach in.
- Rollback of a published application. Publishing a lower version is already the whole of it, once versions compare numerically.
- A live registry query from the portal. The catalogue is a file that
only the platform team writes, deliberately (see
internal/catalog). The generator asks the registry; the portal reads the file. Making the portal depend on the registry being up would trade a clear failure for a vague one.