OS deployment: image catalog with upstream sync, deploy/release jobs, agent deploy mode, UI and CLI

- rollcall os list|sync|import|promote|rm: <os_dir>/<name>/<version>/{disk.raw.zst,meta.json},
  current symlink, vendor checksum verification (GNU/BSD formats, sha256/sha512), qemu-img -> raw,
  zstd in-process, keep-N pruning, nightly rollcall-os-sync.timer, deploy/os-sources.json
  (Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9, CentOS Stream 9)
- jobs table + API: deploy (disk pick, hostname/IP allocation, job token, per-MAC deploy PXE
  entry), release (wipe), progress, cancel, cloud-init phone_home
- agent/rollcall-deploy.py: streamed write with sha256 + pv progress, NoCloud seed, SELinux
  autorelabel, efibootmgr on UEFI; commissioning image gains zstd/parted/xfsprogs/...
- web UI: deploy/release controls, deployment card with progress, OS catalog table
- README: deployment section, image freshness policy

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Fable 5.1 committed 2026-10-27 10:10:39 +01:00
1 parent 1452b79218
commit fad4e1c8bd
24 files changed
+2194 -26

No files matched your search

+73 -4
View File
@@ -44,7 +44,8 @@ unknown server ──PXE──▶ dnsmasq (DHCP+TFTP) ──▶ pxelinux default
|---|---|
| `cmd/rollcall/` | Go server + CLI + embedded web UI (`web/`) |
| `agent/` | `rollcall-agent.py` (runs inside the live image), its systemd unit, `build-image.sh` (live-build recipe) |
| `deploy/` | `install.sh`, `rollcall.service`, `config.example.json`, `pxelinux-default` |
| `deploy/` | `install.sh`, `rollcall.service`, `config.example.json`, `pxelinux-default`, `os-sources.json`, sync/dnsmasq/controller units, `overlays/` |
| `images/` | stateless root image recipes (`build-rootfs.sh`, `compute-debian13/`) |
| `lab/` | Proxmox test lab: `pve-vbmc` (virtual IPMI BMC for VMs backed by the Proxmox API), cloud-init for the PXE server VM |
| `site/` | Project website (static) |
| `.gitea/workflows/` | CI (fmt/vet/build/tests), release on tags, deploy on push to `main` |
@@ -97,6 +98,15 @@ Reads `ROLLCALL_URL` / `ROLLCALL_TOKEN`, or the config file when readable.
| POST | `/api/v1/machines/{id}/bootdev` | admin | `{"device":"pxe"}` |
| POST | `/api/v1/machines/{id}/recommission[?now=1]` | admin | boot the image again on next PXE (`now=1`: via BMC immediately) |
| DELETE | `/api/v1/machines/{id}` | admin | forget the machine |
| GET/POST/DELETE | `/api/v1/profiles[/{name}]` | admin | stateless profiles |
| POST | `/api/v1/machines/{id}/assign` `/unassign` | admin | stateless node assignment |
| GET | `/api/v1/os` | admin | OS image catalog |
| POST | `/api/v1/machines/{id}/deploy` | admin | `{"os":"rocky-9","disk":"/dev/sda","hostname":"","package_upgrade":false,"boot":true}` |
| POST | `/api/v1/machines/{id}/release` | admin | wipe job |
| GET / POST | `/api/v1/machines/{id}/jobs[/cancel]` | admin | job history / cancel |
| GET | `/api/v1/jobs/self`, POST `…/progress` | job token (`X-Rollcall-Job`) | deploy agent |
| POST | `/api/v1/phone-home/{token}` | job token in path | cloud-init phone_home |
| GET | `/os/*` | none | OS disk images |
| GET | `/rollcall/*` | none | image files (kernel, initrd, squashfs) |
| GET | `/` | token in browser | web UI |
@@ -157,6 +167,64 @@ dress it with per-node files, instead of installing an OS on disk.
States: `ready → stateless → booted`; `rollcall unassign` releases name and IPs.
## OS deployment (install to disk)
For nodes that should run an operating system from their own disk, Rollcall
writes a vendor cloud image onto the disk and lets cloud-init do first-boot
configuration. No installer, no kickstart/preseed: a deployment is a streamed
disk write and takes about as long as the image download.
```
rollcall deploy <id> rocky-9 -boot -watch
│ server: picks the disk (smallest that fits, or -disk), allocates hostname + IP,
│ creates a job with a one-time token, writes pxelinux.cfg/01-<mac>
│ ("deploy" entry = commissioning image with rollcall.mode=deploy), power-cycles via BMC
▼
commissioning image, deploy mode (agent/rollcall-deploy.py)
GET /api/v1/jobs/self → image URL, sha256, disk, rendered cloud-init seed
curl | tee sha256 | pv | zstd -d | dd of=/dev/sda (progress every 3 s)
mount root fs → /var/lib/cloud/seed/nocloud/{meta-data,user-data}, datasource_list: [NoCloud, None]
(SELinux images: /.autorelabel; UEFI hosts: efibootmgr entry for the ESP loader)
POST progress state=done → server: state deployed, PXE entry = local disk, reboot
▼
first boot: cloud-init sets hostname/fqdn, SSH keys (root + default user), optional
package_upgrade, then phone_home → POST /api/v1/phone-home/<token> (shown as "phoned home")
```
### Image catalog and freshness
```
os_dir/<name>/<version>/disk.raw.zst raw disk image, zstd
os_dir/<name>/<version>/meta.json sizes, sha256, upstream checksum, import time
os_dir/<name>/current -> <version> what new deployments use
```
`os_sources` (`deploy/os-sources.json`) lists upstream "latest" URLs with the
vendor checksum file for each: Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9,
CentOS Stream 9. `rollcall os sync` fetches the checksum file, and only when the
vendor checksum differs from every imported version does it download the image,
verify it, convert to raw (`qemu-img`), compress, write a new version directory
and flip `current` atomically. `rollcall-os-sync.timer` runs it nightly;
`keep` versions are retained per source, nothing younger than a day is pruned,
and a job always pins the exact version it started with. Sources with
`"track": "pinned"` still download new versions but keep `current` until
`rollcall os promote <name> <version>`. Local files go in with
`rollcall os import <name> <file.qcow2>`. Image freshness is monthly at best
upstream; the deploy option `-upgrade` (`package_upgrade`) pulls the rest on
first boot.
```sh
rollcall os list | sync [NAME...] [-force] | import NAME FILE [-family debian|rhel] | promote NAME VERSION | rm NAME VERSION
rollcall deploy ID OS [-version V] [-disk /dev/nvme0n1] [-hostname H] [-upgrade] [-packages a,b] [-user-data FILE] [-boot] [-watch]
rollcall release ID [-wipe quick|full] [-boot] [-watch] # wipe the disk, back to ready
rollcall jobs ID [-watch] | job cancel ID
```
States: `ready → deploying → deployed` (`deploy-failed` on error; the agent stays
up for ssh). A deployed machine's PXE entry defaults to local disk, so boot order
can stay "network first" on BIOS hosts; on UEFI hosts the agent adds a boot entry
for the installed loader. `rollcall recommission` or `release` take it back.
## Lab
`lab/` documents a Proxmox SDN test setup: an isolated vnet, a PXE server VM
@@ -166,6 +234,7 @@ IPMI-over-LAN BMC emulator that drives VMs through the Proxmox API so that
## Status
Working end to end in the lab (virtual machines, out-of-band BMC hints). The
in-band BMC path is implemented but has not yet been exercised on real
hardware.
Commissioning, stateless nodes and disk deployment work end to end in the lab
(virtual machines with 30 GB disks, SeaBIOS, out-of-band BMC hints). The
in-band BMC path and the UEFI boot-entry step are implemented but have not yet
been exercised on real hardware.