Ilker ManapandClaude Fable 5.1
fad4e1c8bd
- rollcall os list|sync|import|promote|rm: <os_dir>/<name>/<version>/{disk.raw.zst,meta.json},
current symlink, vendor checksum verification (GNU/BSD formats, sha256/sha512), qemu-img -> raw,
zstd in-process, keep-N pruning, nightly rollcall-os-sync.timer, deploy/os-sources.json
(Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9, CentOS Stream 9)
- jobs table + API: deploy (disk pick, hostname/IP allocation, job token, per-MAC deploy PXE
entry), release (wipe), progress, cancel, cloud-init phone_home
- agent/rollcall-deploy.py: streamed write with sha256 + pv progress, NoCloud seed, SELinux
autorelabel, efibootmgr on UEFI; commissioning image gains zstd/parted/xfsprogs/...
- web UI: deploy/release controls, deployment card with progress, OS catalog table
- README: deployment section, image freshness policy
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
241 lines
13 KiB
Markdown
241 lines
13 KiB
Markdown
# Rollcall
|
|
|
|
Rollcall is a small, self-contained commissioning service for bare-metal servers,
|
|
in the spirit of MAAS enlistment and commissioning, without the rest of MAAS.
|
|
|
|
A server that PXE-boots for the first time gets an ephemeral Debian live image.
|
|
The image takes a full hardware inventory (DMI, DIMMs, CPUs, disks, NVMe, NICs,
|
|
InfiniBand HCAs, GPUs, PCI, LLDP neighbours, TPM), creates an administrator
|
|
account on the server's BMC through in-band IPMI, reports everything to the
|
|
Rollcall server and powers the machine off. From then on the server is known:
|
|
it shows up in the web UI and the CLI, its BMC can be driven through the API,
|
|
and it will not boot the commissioning image again unless you ask for it.
|
|
|
|
Everything server-side is **one static Go binary**: HTTP API, embedded web UI,
|
|
CLI, PostgreSQL schema management and the PXE bookkeeping.
|
|
|
|
## How it works
|
|
|
|
```
|
|
unknown server ──PXE──▶ dnsmasq (DHCP+TFTP) ──▶ pxelinux default menu
|
|
│ 8 s timeout
|
|
▼
|
|
Debian live image over HTTP (kernel+initrd via TFTP,
|
|
squashfs via HTTP, runs entirely from RAM)
|
|
│
|
|
rollcall-agent: inventory + BMC user + POST /api/v1/commission
|
|
│
|
|
rollcall server: PostgreSQL ◀─┘──▶ writes pxelinux.cfg/01-<mac>
|
|
(next boot: "commissioned → power off")
|
|
```
|
|
|
|
* **Enlist**: a machine is identified by its DMI system UUID (fallback: serial,
|
|
then boot MAC). First contact creates it in state `new`.
|
|
* **Commission**: the inventory upload moves it to `ready`. The BMC password
|
|
the agent generated is sealed with AES-256-GCM (key in
|
|
`/etc/rollcall/master.key`) before it touches the database. It never leaves
|
|
the server again: power and boot-device operations run `ipmitool` server-side.
|
|
* **Re-commission**: removes the per-MAC PXE override; optionally sets
|
|
bootdev=pxe and power-cycles the machine through its BMC.
|
|
|
|
## Repository layout
|
|
|
|
| Path | What |
|
|
|---|---|
|
|
| `cmd/rollcall/` | Go server + CLI + embedded web UI (`web/`) |
|
|
| `agent/` | `rollcall-agent.py` (runs inside the live image), its systemd unit, `build-image.sh` (live-build recipe) |
|
|
| `deploy/` | `install.sh`, `rollcall.service`, `config.example.json`, `pxelinux-default`, `os-sources.json`, sync/dnsmasq/controller units, `overlays/` |
|
|
| `images/` | stateless root image recipes (`build-rootfs.sh`, `compute-debian13/`) |
|
|
| `lab/` | Proxmox test lab: `pve-vbmc` (virtual IPMI BMC for VMs backed by the Proxmox API), cloud-init for the PXE server VM |
|
|
| `site/` | Project website (static) |
|
|
| `.gitea/workflows/` | CI (fmt/vet/build/tests), release on tags, deploy on push to `main` |
|
|
|
|
## Server
|
|
|
|
```sh
|
|
# build
|
|
cd cmd/rollcall && CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o rollcall .
|
|
|
|
# PostgreSQL role + database
|
|
sudo -u postgres psql -c "CREATE ROLE rollcall LOGIN PASSWORD '...';" -c "CREATE DATABASE rollcall OWNER rollcall;"
|
|
|
|
# config (see deploy/config.example.json), tokens from `rollcall keygen`
|
|
sudo install -d -m 0750 /etc/rollcall && sudo install -m 0640 config.json /etc/rollcall/config.json
|
|
|
|
# install binary, systemd unit, pxelinux menu, tftp symlinks
|
|
cd deploy && sudo ./install.sh
|
|
```
|
|
|
|
`config.json` keys: `listen`, `image_dir` (served under `/rollcall/`), `tftp_root`,
|
|
`database_url`, `master_key_file`, `admin_token` (web UI + management API),
|
|
`agent_token` (what the live image presents), `commission_append` (kernel
|
|
command line baked into generated PXE files), `bmc_hints` (out-of-band BMC
|
|
mapping for machines without in-band IPMI, e.g. VMs).
|
|
|
|
### CLI (same binary)
|
|
|
|
```sh
|
|
rollcall list
|
|
rollcall show m-2ef1dab828c8 [-json]
|
|
rollcall events m-2ef1dab828c8
|
|
rollcall power m-2ef1dab828c8 status|on|off|soft|cycle|reset
|
|
rollcall bootdev m-2ef1dab828c8 pxe|disk|cdrom|bios|none
|
|
rollcall recommission m-2ef1dab828c8 [-now]
|
|
rollcall delete m-2ef1dab828c8
|
|
```
|
|
|
|
Reads `ROLLCALL_URL` / `ROLLCALL_TOKEN`, or the config file when readable.
|
|
|
|
### API
|
|
|
|
| Method | Path | Auth | Purpose |
|
|
|---|---|---|---|
|
|
| POST | `/api/v1/commission` | agent token | inventory upload from the live image |
|
|
| GET | `/api/v1/bmc-hint?macs=` | agent token | out-of-band BMC hint for VMs |
|
|
| GET | `/api/v1/machines` | admin | list (summaries) |
|
|
| GET | `/api/v1/machines/{id}` | admin | full record: inventory, BMC (no secret), events |
|
|
| POST | `/api/v1/machines/{id}/power` | admin | `{"action":"on"}` etc., runs ipmitool server-side |
|
|
| POST | `/api/v1/machines/{id}/bootdev` | admin | `{"device":"pxe"}` |
|
|
| POST | `/api/v1/machines/{id}/recommission[?now=1]` | admin | boot the image again on next PXE (`now=1`: via BMC immediately) |
|
|
| DELETE | `/api/v1/machines/{id}` | admin | forget the machine |
|
|
| GET/POST/DELETE | `/api/v1/profiles[/{name}]` | admin | stateless profiles |
|
|
| POST | `/api/v1/machines/{id}/assign` `/unassign` | admin | stateless node assignment |
|
|
| GET | `/api/v1/os` | admin | OS image catalog |
|
|
| POST | `/api/v1/machines/{id}/deploy` | admin | `{"os":"rocky-9","disk":"/dev/sda","hostname":"","package_upgrade":false,"boot":true}` |
|
|
| POST | `/api/v1/machines/{id}/release` | admin | wipe job |
|
|
| GET / POST | `/api/v1/machines/{id}/jobs[/cancel]` | admin | job history / cancel |
|
|
| GET | `/api/v1/jobs/self`, POST `…/progress` | job token (`X-Rollcall-Job`) | deploy agent |
|
|
| POST | `/api/v1/phone-home/{token}` | job token in path | cloud-init phone_home |
|
|
| GET | `/os/*` | none | OS disk images |
|
|
| GET | `/rollcall/*` | none | image files (kernel, initrd, squashfs) |
|
|
| GET | `/` | token in browser | web UI |
|
|
|
|
Admin auth is `Authorization: Bearer <admin_token>`; the agent sends
|
|
`X-Rollcall-Token`.
|
|
|
|
## Commissioning image
|
|
|
|
`agent/build-image.sh` builds a Debian 13 live image with live-build (about
|
|
8 minutes, ~780 MB squashfs). It contains the full Debian kernel with non-free
|
|
firmware, `ipmitool`, `freeipmi`, `rdma-core`, `infiniband-diags`, `mstflint`,
|
|
`nvme-cli`, `smartmontools`, `lldpd`, `lshw` and an SSH server (root, key only)
|
|
for debugging. Output goes to `image_dir`.
|
|
|
|
Kernel parameters understood by the agent: `rollcall.server=URL` (required),
|
|
`rollcall.token=…`, `rollcall.bmcuser=NAME` (default `commissioner`),
|
|
`rollcall.nopoweroff` (stay up after reporting).
|
|
|
|
### BMC account
|
|
|
|
In-band (`/dev/ipmi0`): find the LAN channel, pick the existing `commissioner`
|
|
slot or the first empty user ID ≥ 3, set name/password, ADMINISTRATOR privilege,
|
|
channel access, enable, SOL payload. The password is random (16 chars) and is
|
|
reported once to the server. If no in-band IPMI exists the agent asks the server
|
|
for a hint (`bmc_hints`) and verifies it out-of-band.
|
|
|
|
Vendor quirks (iDRAC/iLO password policies, missing empty slots) are reported in
|
|
`bmc.errors` and visible in the UI.
|
|
|
|
## Stateless nodes (Warewulf-style)
|
|
|
|
For HPC compute nodes Rollcall can boot a root image straight into RAM and
|
|
dress it with per-node files, instead of installing an OS on disk.
|
|
|
|
* **Images** live in `images_dir/<name>/{vmlinuz,initrd.img,filesystem.squashfs}`
|
|
and are built with `images/build-rootfs.sh <name>` from
|
|
`images/<name>/{packages.list,hook.chroot,includes/}` (live-build, same
|
|
toolchain as the commissioning image). `images/compute-debian13` is a
|
|
Debian 13 compute node: slurmd, munge, rdma-core, NFS client, firmware.
|
|
* **Profiles** bind an image to overlays, extra kernel parameters, a node name
|
|
format (`node%03d`) and variables: `rollcall profile add compute -image compute-debian13 -overlays compute`.
|
|
* **Assigning** a commissioned machine to a profile allocates a node name and
|
|
IPs from `node_pools`, writes a per-MAC PXE entry that boots the image with
|
|
`hostname=`, `rollcall.server=` and a per-node token, and exports static
|
|
DHCP leases plus host names for dnsmasq (`dnsmasq_dir`, re-read on HUP via
|
|
`rollcall-dnsmasq.path`). `rollcall assign <id> compute -boot` also
|
|
power-cycles the machine through its BMC.
|
|
* **Overlays** are directory trees under `overlays_dir/<overlay>/{system,runtime}`;
|
|
files ending in `.tmpl` are Go templates rendered with the node, all nodes of
|
|
the cluster, the profile and `cluster_vars`. The `compute` overlay generates
|
|
`/etc/hosts`, `slurm.conf` (node lines from the commissioning inventory:
|
|
sockets, cores, threads, memory, GPUs), `gres.conf`, a persistent SSH host
|
|
key per node (sealed in the database), root `authorized_keys` and the munge
|
|
key. Inside the image `rollcall-overlay.service` applies the system overlay
|
|
before munge/slurmd/ssh start; a timer refreshes the runtime overlay every
|
|
five minutes, which doubles as the node heartbeat (`booted` state, last seen).
|
|
* The controller renders the same files: `rollcall render compute etc/slurm/slurm.conf > /etc/slurm/slurm.conf`.
|
|
|
|
States: `ready → stateless → booted`; `rollcall unassign` releases name and IPs.
|
|
|
|
## OS deployment (install to disk)
|
|
|
|
For nodes that should run an operating system from their own disk, Rollcall
|
|
writes a vendor cloud image onto the disk and lets cloud-init do first-boot
|
|
configuration. No installer, no kickstart/preseed: a deployment is a streamed
|
|
disk write and takes about as long as the image download.
|
|
|
|
```
|
|
rollcall deploy <id> rocky-9 -boot -watch
|
|
│ server: picks the disk (smallest that fits, or -disk), allocates hostname + IP,
|
|
│ creates a job with a one-time token, writes pxelinux.cfg/01-<mac>
|
|
│ ("deploy" entry = commissioning image with rollcall.mode=deploy), power-cycles via BMC
|
|
▼
|
|
commissioning image, deploy mode (agent/rollcall-deploy.py)
|
|
GET /api/v1/jobs/self → image URL, sha256, disk, rendered cloud-init seed
|
|
curl | tee sha256 | pv | zstd -d | dd of=/dev/sda (progress every 3 s)
|
|
mount root fs → /var/lib/cloud/seed/nocloud/{meta-data,user-data}, datasource_list: [NoCloud, None]
|
|
(SELinux images: /.autorelabel; UEFI hosts: efibootmgr entry for the ESP loader)
|
|
POST progress state=done → server: state deployed, PXE entry = local disk, reboot
|
|
▼
|
|
first boot: cloud-init sets hostname/fqdn, SSH keys (root + default user), optional
|
|
package_upgrade, then phone_home → POST /api/v1/phone-home/<token> (shown as "phoned home")
|
|
```
|
|
|
|
### Image catalog and freshness
|
|
|
|
```
|
|
os_dir/<name>/<version>/disk.raw.zst raw disk image, zstd
|
|
os_dir/<name>/<version>/meta.json sizes, sha256, upstream checksum, import time
|
|
os_dir/<name>/current -> <version> what new deployments use
|
|
```
|
|
|
|
`os_sources` (`deploy/os-sources.json`) lists upstream "latest" URLs with the
|
|
vendor checksum file for each: Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9,
|
|
CentOS Stream 9. `rollcall os sync` fetches the checksum file, and only when the
|
|
vendor checksum differs from every imported version does it download the image,
|
|
verify it, convert to raw (`qemu-img`), compress, write a new version directory
|
|
and flip `current` atomically. `rollcall-os-sync.timer` runs it nightly;
|
|
`keep` versions are retained per source, nothing younger than a day is pruned,
|
|
and a job always pins the exact version it started with. Sources with
|
|
`"track": "pinned"` still download new versions but keep `current` until
|
|
`rollcall os promote <name> <version>`. Local files go in with
|
|
`rollcall os import <name> <file.qcow2>`. Image freshness is monthly at best
|
|
upstream; the deploy option `-upgrade` (`package_upgrade`) pulls the rest on
|
|
first boot.
|
|
|
|
```sh
|
|
rollcall os list | sync [NAME...] [-force] | import NAME FILE [-family debian|rhel] | promote NAME VERSION | rm NAME VERSION
|
|
rollcall deploy ID OS [-version V] [-disk /dev/nvme0n1] [-hostname H] [-upgrade] [-packages a,b] [-user-data FILE] [-boot] [-watch]
|
|
rollcall release ID [-wipe quick|full] [-boot] [-watch] # wipe the disk, back to ready
|
|
rollcall jobs ID [-watch] | job cancel ID
|
|
```
|
|
|
|
States: `ready → deploying → deployed` (`deploy-failed` on error; the agent stays
|
|
up for ssh). A deployed machine's PXE entry defaults to local disk, so boot order
|
|
can stay "network first" on BIOS hosts; on UEFI hosts the agent adds a boot entry
|
|
for the installed loader. `rollcall recommission` or `release` take it back.
|
|
|
|
## Lab
|
|
|
|
`lab/` documents a Proxmox SDN test setup: an isolated vnet, a PXE server VM
|
|
running dnsmasq + Rollcall, two diskless VMs, and `pve-vbmc`, a tiny
|
|
IPMI-over-LAN BMC emulator that drives VMs through the Proxmox API so that
|
|
`ipmitool power on` and `chassis bootdev pxe` work against virtual machines.
|
|
|
|
## Status
|
|
|
|
Commissioning, stateless nodes and disk deployment work end to end in the lab
|
|
(virtual machines with 30 GB disks, SeaBIOS, out-of-band BMC hints). The
|
|
in-band BMC path and the UEFI boot-entry step are implemented but have not yet
|
|
been exercised on real hardware.
|