- 01, 02: say the lab is built by hand and point to hpcod/DEPLOY.md for installing the control plane. - 09: repository layout now includes hpcod/, docs 20-23 and the B/C scripts; ports for hpcod's own HTTPS and DNS; Slurm accounting is recorded, not absent (matches 6.6). Handbook PDF regenerated. - 20: /v1/clusters exists; first-boot selfconfig instead of a playbook run from the control plane; Terraform state per cluster, not in Postgres; no reaper yet; usage per project. - 21: c<NNN> portals, publishing wired into provisioning, closed-network mode. - 22: three published images; chosen apps now reach the cluster. - 23: setup installs the catalogue; apps.local keeps a site's own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
217 lines
11 KiB
Markdown
217 lines
11 KiB
Markdown
# 23 — The life of an application
|
|
|
|
Written for whoever maintains `hpcod/`. It assumes §22 and the code in
|
|
`internal/catalog`, `internal/store` and `image/apps`.
|
|
|
|
Publishing an application now works: a recipe is built, verified against its
|
|
own binaries, pushed, checked against the registry, and the portal picks up
|
|
the new catalogue without a restart. The binary carries a copy and `hpcod
|
|
setup` installs it; a site that keeps its own creates `/etc/hpcod/apps.local`,
|
|
and setup then leaves the file alone. That is the first minute of an
|
|
application's life. This is about the next two years of it.
|
|
|
|
## The one idea
|
|
|
|
An application and a cluster have different lifetimes, and every hard
|
|
question here lives at the seam between them.
|
|
|
|
| | changes | lives in |
|
|
|---|---|---|
|
|
| a catalogue entry | monthly | `config/apps.yaml`, from the recipes; setup installs it as `/etc/hpcod/apps.yaml` |
|
|
| what a cluster holds | never, once allocated | `cluster_app`, in PostgreSQL |
|
|
| a cluster | fourteen days | Proxmox |
|
|
|
|
Those two records are connected exactly once, at allocation, and never
|
|
looked at together again. That is correct as far as it goes — a cluster must
|
|
not silently change underneath a running job — but it means every question
|
|
worth asking about an application after it is published is currently
|
|
unanswerable:
|
|
|
|
- A CVE lands in GROMACS. **Which live clusters are running it?**
|
|
- We published 2025.1. **Who is still on 2024.4, and does anyone tell them?**
|
|
- The registry is at 400 GB. **Which tags can be deleted without breaking a
|
|
rebuild?**
|
|
|
|
The schema already holds the answers. `cluster_app` has an index on `name`
|
|
for exactly this, and nothing queries it. The whole of this plan is: make
|
|
that seam visible, then act on it.
|
|
|
|
## Four states, and what each transition owes
|
|
|
|
Today an entry is on or off — `published:` in `app.yaml`, overridden by
|
|
whether the registry really holds the tag. That is two states where four are
|
|
needed.
|
|
|
|
| state | offered? | pullable? | means |
|
|
|---|---|---|---|
|
|
| `draft` | no | no | the recipe exists, the image does not |
|
|
| `available` | yes | yes | the normal state |
|
|
| `deprecated` | yes, with a warning | yes | a newer version exists; still supported |
|
|
| `withdrawn` | no | yes, for now | do not start anything new on this |
|
|
|
|
The states are cheap. What matters is what each transition is *obliged* to
|
|
do, and this is where the work is:
|
|
|
|
**add** — already done. Build, verify binaries, push, read back, regenerate.
|
|
|
|
**update** — publish a new version; the old one becomes `deprecated`
|
|
automatically, because a newer one exists. Running clusters do not move and
|
|
must not. But the tenant of a cluster pinned to a deprecated version should
|
|
be told once, on their cluster page, rather than finding out from a
|
|
colleague.
|
|
|
|
**deprecate** — a statement, not an action. Needs a date and a reason, which
|
|
are metadata and belong in the recipe.
|
|
|
|
**withdraw** — the interesting one. Not offered to new clusters, *and* the
|
|
operator is handed the list of live clusters that hold it. A withdrawal that
|
|
cannot name who is affected is a note to oneself.
|
|
|
|
**delete from the registry** — and here the database gives us a rule rather
|
|
than a guess: **a tag referenced by any live cluster's `cluster_app` row
|
|
cannot be deleted.** Not a retention window, not "keep the last three" — a
|
|
correctness constraint, checkable in one query. A cluster that is rebuilt
|
|
must get the bytes it was allocated with.
|
|
|
|
## What has to be fixed before any of that
|
|
|
|
Four things found by looking, not by reading. Each is small; together they
|
|
are the floor the rest stands on.
|
|
|
|
### Versions are compared as text
|
|
|
|
`Catalog.Lookup` sorts candidates with `a.Version < b.Version` on strings, so
|
|
asking for `gromacs` with 2024.9 and 2024.10 published returns **2024.9**.
|
|
Measured:
|
|
|
|
```
|
|
2024.9 and 2024.10 -> "2024.9" wrong
|
|
1.9 and 1.10 -> "1.9" wrong
|
|
0.9.0 and 0.18.07 -> "0.9.0" wrong
|
|
2024.4 and 2025.1 -> "2025.1" right, by luck
|
|
```
|
|
|
|
The whole meaning of an unversioned reference is "the newest one", and it is
|
|
wrong the first time a minor version reaches ten. Everything about updating
|
|
rests on this, so it is first.
|
|
|
|
### A recipe holds one version; the catalogue holds many
|
|
|
|
`Catalog` already supports several entries under one name and picks the
|
|
highest. `image/apps/<name>/` cannot express that: one directory, one
|
|
`app.yaml`, one `version:`. Updating GROMACS therefore means either losing
|
|
2024.4 or creating a parallel `gromacs-2025/` directory that shares nothing
|
|
with the first.
|
|
|
|
This is the layout decision the plan turns on. The recipe should carry a list
|
|
of versions with their own build arguments, and the generator should emit one
|
|
catalogue entry per version. Without it, "deprecated" has nowhere to live,
|
|
because the old version has already been overwritten.
|
|
|
|
### Images are named by a mutable tag
|
|
|
|
`cluster_app.image` records `oras://…/gromacs:2024.4`. Anyone who repushes
|
|
that tag changes what every *rebuilt* cluster gets, silently, and the record
|
|
that was supposed to freeze the cluster freezes nothing. The registry hands
|
|
out a digest at push time and `build-push.sh` already pulls the image back to
|
|
compare it — recording that digest and pulling by it costs almost nothing and
|
|
turns the record into the guarantee it is meant to be.
|
|
|
|
### Three catalogue fields are decoration
|
|
|
|
`Module` is read nowhere at all. `SizeGB` and `Signed` are read once each, to
|
|
be copied into a JSON response, and no decision is taken from either.
|
|
|
|
`Module` is the worst of the three, because it marks the real gap: a tenant
|
|
is handed `/opt/apps/gromacs.sif` and no way to run it. `binaries:` is
|
|
already carried through the recipe into the catalogue and stops there. The
|
|
`apps` role should turn those two fields into a wrapper per binary and an
|
|
Lmod modulefile, so the answer to "how do I run this" is `module load
|
|
gromacs` rather than an Apptainer incantation the user has to be taught.
|
|
|
|
Until that exists we ship images, not software.
|
|
|
|
## The plan
|
|
|
|
**Stage 1 — make what exists usable and true.** Nothing new to reason about.
|
|
All five are now done; what follows is what each turned out to involve.
|
|
|
|
1. Numeric version comparison in `Lookup`, with the table above as the test.
|
|
2. Multi-version recipes, so an update does not destroy its predecessor.
|
|
3. Digest recorded at publish, carried into `cluster_app`, pulled by digest.
|
|
4. Modulefiles and wrappers from `module:` and `binaries:`.
|
|
5. `hpcod apps` — `list`, `users <name>`, `verify`. It reads data that is
|
|
already written and is the only reason the `cluster_app` name index
|
|
exists.
|
|
|
|
**Stage 2 — the lifecycle proper.** Done. Three things it settled that the
|
|
plan above left open:
|
|
|
|
- **`published:` became `status:`.** Two spellings of one fact in one file is
|
|
worse than one awkward spelling, so the nine recipes were migrated rather
|
|
than both being accepted. An unrecognised status is a hard error: a typo
|
|
that silently meant "draft" would quietly unpublish an application.
|
|
- **A withdrawn entry stays in the catalogue file.** Dropping it would make
|
|
it indistinguishable from one that never existed, and that difference is
|
|
exactly what an operator needs when a cluster is found still holding it.
|
|
The portal is what stops it being offered, and `Resolve` refuses a pinned
|
|
reference to it with the recorded reason rather than "no such application"
|
|
- which would send the user hunting for a typo in a correct name.
|
|
- **Deprecated is offered, and says so.** It means a newer version exists,
|
|
not that this one has stopped working; refusing it would break the pin of
|
|
everyone who named it deliberately. `hpcod apps verify` reports it and
|
|
does not exit non-zero for it, so the command is usable from cron.
|
|
|
|
`hpcod apps prunable` is the retention rule made operable. Two things protect
|
|
a tag and they guard different mistakes: the catalogue, because a new cluster
|
|
may still ask for it, and a live cluster's record, because that cluster may
|
|
be rebuilt. Run against the real registry it correctly protected
|
|
`desktop:xfce`, which no catalogue entry mentions at all and a live cluster
|
|
holds. A repository that cannot be listed makes the whole answer incomplete
|
|
and is reported as such rather than skipped, because a partial list of "safe
|
|
to delete" is worse than none.
|
|
|
|
**Stage 3 — governance.** Independent of both, and each item is its own
|
|
decision rather than a consequence of this design:
|
|
|
|
- Provenance: who published, when, from which revision, at which digest.
|
|
- Signing, and dropping `--allow-unsigned` from the `apps` role. The
|
|
`Signed` field is a promise nothing keeps.
|
|
- A CVE gate on publication — see §22, where Chapar's checker is discussed.
|
|
- Private and licensed applications — see §22. VASP and ANSYS cannot live in
|
|
a shared registry, and that is a visibility model, not another recipe.
|
|
|
|
Two things were found only by running them, and both were the failure this
|
|
repository keeps having: an action that reported success and had not changed
|
|
what it claimed to. `template` does not create intermediate directories, so
|
|
the modulefile task failed mid-provision with the machines already up. And
|
|
`hpcod apps list` swallowed the error from its own usage query and printed
|
|
`-` in the CLUSTERS column - which reads as "nothing holds this", the answer
|
|
an operator would act on when deciding a tag is safe to delete. It printed
|
|
that the first time it was run, against a database one migration behind.
|
|
That column now prints `?` when the question could not be asked.
|
|
|
|
Stage 1 also left one limit in place on purpose. The SIF path is still
|
|
`/opt/apps/<name>.sif` because the `ondemand` role hard-codes
|
|
`/opt/apps/desktop.sif`, so a cluster can hold only one version of an
|
|
application and the control plane refuses a request naming two. The
|
|
catalogue supports several; the head does not. Versioned SIF paths are what
|
|
would lift it, and that is its own change with its own blast radius.
|
|
|
|
**Recommended first:** item 4, the modulefiles. It is the difference between
|
|
delivering a file and delivering software, the fields for it already exist
|
|
and are dead, and it is the only item a tenant would notice this week. Then
|
|
item 1, because it is a wrong answer rather than a missing one.
|
|
|
|
## Deliberately not planned
|
|
|
|
- **Moving a running cluster to a new version.** A fourteen-day cluster
|
|
should end on the software it started with. Tell the tenant; do not
|
|
reach in.
|
|
- **Rollback of a published application.** Publishing a lower version is
|
|
already the whole of it, once versions compare numerically.
|
|
- **A live registry query from the portal.** The catalogue is a file that
|
|
only the platform team writes, deliberately (see `internal/catalog`). The
|
|
generator asks the registry; the portal reads the file. Making the portal
|
|
depend on the registry being up would trade a clear failure for a vague one.
|