Files
werewulf/docs/23-application-lifecycle.md
ilkerandClaude Opus 5.5 861aceb283 docs: bring sections 01-23 in line with hpcod setup and what is built
- 01, 02: say the lab is built by hand and point to hpcod/DEPLOY.md for
  installing the control plane.
- 09: repository layout now includes hpcod/, docs 20-23 and the B/C
  scripts; ports for hpcod's own HTTPS and DNS; Slurm accounting is
  recorded, not absent (matches 6.6). Handbook PDF regenerated.
- 20: /v1/clusters exists; first-boot selfconfig instead of a playbook run
  from the control plane; Terraform state per cluster, not in Postgres;
  no reaper yet; usage per project.
- 21: c<NNN> portals, publishing wired into provisioning, closed-network
  mode.
- 22: three published images; chosen apps now reach the cluster.
- 23: setup installs the catalogue; apps.local keeps a site's own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:50:33 +01:00

217 lines
11 KiB
Markdown

# 23 — The life of an application
Written for whoever maintains `hpcod/`. It assumes §22 and the code in
`internal/catalog`, `internal/store` and `image/apps`.
Publishing an application now works: a recipe is built, verified against its
own binaries, pushed, checked against the registry, and the portal picks up
the new catalogue without a restart. The binary carries a copy and `hpcod
setup` installs it; a site that keeps its own creates `/etc/hpcod/apps.local`,
and setup then leaves the file alone. That is the first minute of an
application's life. This is about the next two years of it.
## The one idea
An application and a cluster have different lifetimes, and every hard
question here lives at the seam between them.
| | changes | lives in |
|---|---|---|
| a catalogue entry | monthly | `config/apps.yaml`, from the recipes; setup installs it as `/etc/hpcod/apps.yaml` |
| what a cluster holds | never, once allocated | `cluster_app`, in PostgreSQL |
| a cluster | fourteen days | Proxmox |
Those two records are connected exactly once, at allocation, and never
looked at together again. That is correct as far as it goes — a cluster must
not silently change underneath a running job — but it means every question
worth asking about an application after it is published is currently
unanswerable:
- A CVE lands in GROMACS. **Which live clusters are running it?**
- We published 2025.1. **Who is still on 2024.4, and does anyone tell them?**
- The registry is at 400 GB. **Which tags can be deleted without breaking a
rebuild?**
The schema already holds the answers. `cluster_app` has an index on `name`
for exactly this, and nothing queries it. The whole of this plan is: make
that seam visible, then act on it.
## Four states, and what each transition owes
Today an entry is on or off — `published:` in `app.yaml`, overridden by
whether the registry really holds the tag. That is two states where four are
needed.
| state | offered? | pullable? | means |
|---|---|---|---|
| `draft` | no | no | the recipe exists, the image does not |
| `available` | yes | yes | the normal state |
| `deprecated` | yes, with a warning | yes | a newer version exists; still supported |
| `withdrawn` | no | yes, for now | do not start anything new on this |
The states are cheap. What matters is what each transition is *obliged* to
do, and this is where the work is:
**add** — already done. Build, verify binaries, push, read back, regenerate.
**update** — publish a new version; the old one becomes `deprecated`
automatically, because a newer one exists. Running clusters do not move and
must not. But the tenant of a cluster pinned to a deprecated version should
be told once, on their cluster page, rather than finding out from a
colleague.
**deprecate** — a statement, not an action. Needs a date and a reason, which
are metadata and belong in the recipe.
**withdraw** — the interesting one. Not offered to new clusters, *and* the
operator is handed the list of live clusters that hold it. A withdrawal that
cannot name who is affected is a note to oneself.
**delete from the registry** — and here the database gives us a rule rather
than a guess: **a tag referenced by any live cluster's `cluster_app` row
cannot be deleted.** Not a retention window, not "keep the last three" — a
correctness constraint, checkable in one query. A cluster that is rebuilt
must get the bytes it was allocated with.
## What has to be fixed before any of that
Four things found by looking, not by reading. Each is small; together they
are the floor the rest stands on.
### Versions are compared as text
`Catalog.Lookup` sorts candidates with `a.Version < b.Version` on strings, so
asking for `gromacs` with 2024.9 and 2024.10 published returns **2024.9**.
Measured:
```
2024.9 and 2024.10 -> "2024.9" wrong
1.9 and 1.10 -> "1.9" wrong
0.9.0 and 0.18.07 -> "0.9.0" wrong
2024.4 and 2025.1 -> "2025.1" right, by luck
```
The whole meaning of an unversioned reference is "the newest one", and it is
wrong the first time a minor version reaches ten. Everything about updating
rests on this, so it is first.
### A recipe holds one version; the catalogue holds many
`Catalog` already supports several entries under one name and picks the
highest. `image/apps/<name>/` cannot express that: one directory, one
`app.yaml`, one `version:`. Updating GROMACS therefore means either losing
2024.4 or creating a parallel `gromacs-2025/` directory that shares nothing
with the first.
This is the layout decision the plan turns on. The recipe should carry a list
of versions with their own build arguments, and the generator should emit one
catalogue entry per version. Without it, "deprecated" has nowhere to live,
because the old version has already been overwritten.
### Images are named by a mutable tag
`cluster_app.image` records `oras://…/gromacs:2024.4`. Anyone who repushes
that tag changes what every *rebuilt* cluster gets, silently, and the record
that was supposed to freeze the cluster freezes nothing. The registry hands
out a digest at push time and `build-push.sh` already pulls the image back to
compare it — recording that digest and pulling by it costs almost nothing and
turns the record into the guarantee it is meant to be.
### Three catalogue fields are decoration
`Module` is read nowhere at all. `SizeGB` and `Signed` are read once each, to
be copied into a JSON response, and no decision is taken from either.
`Module` is the worst of the three, because it marks the real gap: a tenant
is handed `/opt/apps/gromacs.sif` and no way to run it. `binaries:` is
already carried through the recipe into the catalogue and stops there. The
`apps` role should turn those two fields into a wrapper per binary and an
Lmod modulefile, so the answer to "how do I run this" is `module load
gromacs` rather than an Apptainer incantation the user has to be taught.
Until that exists we ship images, not software.
## The plan
**Stage 1 — make what exists usable and true.** Nothing new to reason about.
All five are now done; what follows is what each turned out to involve.
1. Numeric version comparison in `Lookup`, with the table above as the test.
2. Multi-version recipes, so an update does not destroy its predecessor.
3. Digest recorded at publish, carried into `cluster_app`, pulled by digest.
4. Modulefiles and wrappers from `module:` and `binaries:`.
5. `hpcod apps` — `list`, `users <name>`, `verify`. It reads data that is
already written and is the only reason the `cluster_app` name index
exists.
**Stage 2 — the lifecycle proper.** Done. Three things it settled that the
plan above left open:
- **`published:` became `status:`.** Two spellings of one fact in one file is
worse than one awkward spelling, so the nine recipes were migrated rather
than both being accepted. An unrecognised status is a hard error: a typo
that silently meant "draft" would quietly unpublish an application.
- **A withdrawn entry stays in the catalogue file.** Dropping it would make
it indistinguishable from one that never existed, and that difference is
exactly what an operator needs when a cluster is found still holding it.
The portal is what stops it being offered, and `Resolve` refuses a pinned
reference to it with the recorded reason rather than "no such application"
- which would send the user hunting for a typo in a correct name.
- **Deprecated is offered, and says so.** It means a newer version exists,
not that this one has stopped working; refusing it would break the pin of
everyone who named it deliberately. `hpcod apps verify` reports it and
does not exit non-zero for it, so the command is usable from cron.
`hpcod apps prunable` is the retention rule made operable. Two things protect
a tag and they guard different mistakes: the catalogue, because a new cluster
may still ask for it, and a live cluster's record, because that cluster may
be rebuilt. Run against the real registry it correctly protected
`desktop:xfce`, which no catalogue entry mentions at all and a live cluster
holds. A repository that cannot be listed makes the whole answer incomplete
and is reported as such rather than skipped, because a partial list of "safe
to delete" is worse than none.
**Stage 3 — governance.** Independent of both, and each item is its own
decision rather than a consequence of this design:
- Provenance: who published, when, from which revision, at which digest.
- Signing, and dropping `--allow-unsigned` from the `apps` role. The
`Signed` field is a promise nothing keeps.
- A CVE gate on publication — see §22, where Chapar's checker is discussed.
- Private and licensed applications — see §22. VASP and ANSYS cannot live in
a shared registry, and that is a visibility model, not another recipe.
Two things were found only by running them, and both were the failure this
repository keeps having: an action that reported success and had not changed
what it claimed to. `template` does not create intermediate directories, so
the modulefile task failed mid-provision with the machines already up. And
`hpcod apps list` swallowed the error from its own usage query and printed
`-` in the CLUSTERS column - which reads as "nothing holds this", the answer
an operator would act on when deciding a tag is safe to delete. It printed
that the first time it was run, against a database one migration behind.
That column now prints `?` when the question could not be asked.
Stage 1 also left one limit in place on purpose. The SIF path is still
`/opt/apps/<name>.sif` because the `ondemand` role hard-codes
`/opt/apps/desktop.sif`, so a cluster can hold only one version of an
application and the control plane refuses a request naming two. The
catalogue supports several; the head does not. Versioned SIF paths are what
would lift it, and that is its own change with its own blast radius.
**Recommended first:** item 4, the modulefiles. It is the difference between
delivering a file and delivering software, the fields for it already exist
and are dead, and it is the only item a tenant would notice this week. Then
item 1, because it is a wrong answer rather than a missing one.
## Deliberately not planned
- **Moving a running cluster to a new version.** A fourteen-day cluster
should end on the software it started with. Tell the tenant; do not
reach in.
- **Rollback of a published application.** Publishing a lower version is
already the whole of it, once versions compare numerically.
- **A live registry query from the portal.** The catalogue is a file that
only the platform team writes, deliberately (see `internal/catalog`). The
generator asks the registry; the portal reads the file. Making the portal
depend on the registry being up would trade a clear failure for a vague one.