rollcall/README.md

# Rollcall

Rollcall is a small, self-contained commissioning service for bare-metal servers,
in the spirit of MAAS enlistment and commissioning, without the rest of MAAS.

A server that PXE-boots for the first time gets an ephemeral Debian live image.
The image takes a full hardware inventory (DMI, DIMMs, CPUs, disks, NVMe, NICs,
InfiniBand HCAs, GPUs, PCI, LLDP neighbours, TPM), creates an administrator
account on the server's BMC through in-band IPMI, reports everything to the
Rollcall server and powers the machine off. From then on the server is known:
it shows up in the web UI and the CLI, its BMC can be driven through the API,
and it will not boot the commissioning image again unless you ask for it.

Everything server-side is **one static Go binary**: HTTP API, embedded web UI,
CLI, PostgreSQL schema management and the PXE bookkeeping.

## How it works

```
unknown server ──PXE──▶ dnsmasq (DHCP+TFTP) ──▶ pxelinux default menu
                                                  │ 8 s timeout
                                                  ▼
                               Debian live image over HTTP (kernel+initrd via TFTP,
                               squashfs via HTTP, runs entirely from RAM)
                                                  │
                      rollcall-agent: inventory + BMC user + POST /api/v1/commission
                                                  │
                     rollcall server: PostgreSQL ◀─┘──▶ writes pxelinux.cfg/01-<mac>
                                                       (next boot: "commissioned → power off")
```

* **Enlist**: a machine is identified by its DMI system UUID (fallback: serial,
  then boot MAC). First contact creates it in state `new`.
* **Commission**: the inventory upload moves it to `ready`. The BMC password
  the agent generated is sealed with AES-256-GCM (key in
  `/etc/rollcall/master.key`) before it touches the database. It never leaves
  the server again: power and boot-device operations run `ipmitool` server-side.
* **Re-commission**: removes the per-MAC PXE override; optionally sets
  bootdev=pxe and power-cycles the machine through its BMC.

## Repository layout

| Path | What |
|---|---|
| `cmd/rollcall/` | Go server + CLI + embedded web UI (`web/`) |
| `agent/` | `rollcall-agent.py` (runs inside the live image), its systemd unit, `build-image.sh` (live-build recipe) |
| `deploy/` | `install.sh`, `rollcall.service`, `config.example.json`, `pxelinux-default`, `os-sources.json`, sync/dnsmasq/controller units, `overlays/` |
| `images/` | stateless root image recipes (`build-rootfs.sh`, `compute-debian13/`) |
| `lab/` | Proxmox test lab: `pve-vbmc` (virtual IPMI BMC for VMs backed by the Proxmox API), cloud-init for the PXE server VM |
| `site/` | Project website (static) |
| `.gitea/workflows/` | CI (fmt/vet/build/tests), release on tags, deploy on push to `main` |

## Server

```sh
# build
cd cmd/rollcall && CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o rollcall .

# PostgreSQL role + database
sudo -u postgres psql -c "CREATE ROLE rollcall LOGIN PASSWORD '...';" -c "CREATE DATABASE rollcall OWNER rollcall;"

# config (see deploy/config.example.json), tokens from `rollcall keygen`
sudo install -d -m 0750 /etc/rollcall && sudo install -m 0640 config.json /etc/rollcall/config.json

# install binary, systemd unit, pxelinux menu, tftp symlinks
cd deploy && sudo ./install.sh
```

`config.json` keys: `listen`, `image_dir` (served under `/rollcall/`), `tftp_root`,
`database_url`, `master_key_file`, `admin_token` (web UI + management API),
`agent_token` (what the live image presents), `commission_append` (kernel
command line baked into generated PXE files), `bmc_hints` (out-of-band BMC
mapping for machines without in-band IPMI, e.g. VMs).

### CLI (same binary)

```sh
rollcall list
rollcall show m-2ef1dab828c8 [-json]
rollcall events m-2ef1dab828c8
rollcall power m-2ef1dab828c8 status|on|off|soft|cycle|reset
rollcall bootdev m-2ef1dab828c8 pxe|disk|cdrom|bios|none
rollcall recommission m-2ef1dab828c8 [-now]
rollcall delete m-2ef1dab828c8
```

Reads `ROLLCALL_URL` / `ROLLCALL_TOKEN`, or the config file when readable.

### API

| Method | Path | Auth | Purpose |
|---|---|---|---|
| POST | `/api/v1/commission` | agent token | inventory upload from the live image |
| GET | `/api/v1/bmc-hint?macs=` | agent token | out-of-band BMC hint for VMs |
| GET | `/api/v1/machines` | admin | list (summaries) |
| GET | `/api/v1/machines/{id}` | admin | full record: inventory, BMC (no secret), events |
| POST | `/api/v1/machines/{id}/power` | admin | `{"action":"on"}` etc., runs ipmitool server-side |
| POST | `/api/v1/machines/{id}/bootdev` | admin | `{"device":"pxe"}` |
| POST | `/api/v1/machines/{id}/recommission[?now=1]` | admin | boot the image again on next PXE (`now=1`: via BMC immediately) |
| DELETE | `/api/v1/machines/{id}` | admin | forget the machine |
| GET/POST/DELETE | `/api/v1/profiles[/{name}]` | admin | stateless profiles |
| POST | `/api/v1/machines/{id}/assign` `/unassign` | admin | stateless node assignment |
| GET | `/api/v1/os` | admin | OS image catalog |
| POST | `/api/v1/machines/{id}/deploy` | admin | `{"os":"rocky-9","disk":"/dev/sda","hostname":"","package_upgrade":false,"boot":true}` |
| POST | `/api/v1/machines/{id}/release` | admin | wipe job |
| GET / POST | `/api/v1/machines/{id}/jobs[/cancel]` | admin | job history / cancel |
| GET | `/api/v1/jobs/self`, POST `…/progress` | job token (`X-Rollcall-Job`) | deploy agent |
| POST | `/api/v1/phone-home/{token}` | job token in path | cloud-init phone_home |
| GET | `/os/*` | none | OS disk images |
| GET | `/rollcall/*` | none | image files (kernel, initrd, squashfs) |
| GET | `/` | token in browser | web UI |

Admin auth is `Authorization: Bearer <admin_token>`; the agent sends
`X-Rollcall-Token`.

## Commissioning image

`agent/build-image.sh` builds a Debian 13 live image with live-build (about
8 minutes, ~780 MB squashfs). It contains the full Debian kernel with non-free
firmware, `ipmitool`, `freeipmi`, `rdma-core`, `infiniband-diags`, `mstflint`,
`nvme-cli`, `smartmontools`, `lldpd`, `lshw` and an SSH server (root, key only)
for debugging. Output goes to `image_dir`.

Kernel parameters understood by the agent: `rollcall.server=URL` (required),
`rollcall.token=…`, `rollcall.bmcuser=NAME` (default `commissioner`),
`rollcall.nopoweroff` (stay up after reporting).

### BMC account

In-band (`/dev/ipmi0`): find the LAN channel, pick the existing `commissioner`
slot or the first empty user ID ≥ 3, set name/password, ADMINISTRATOR privilege,
channel access, enable, SOL payload. The password is random (16 chars) and is
reported once to the server. If no in-band IPMI exists the agent asks the server
for a hint (`bmc_hints`) and verifies it out-of-band.

Vendor quirks (iDRAC/iLO password policies, missing empty slots) are reported in
`bmc.errors` and visible in the UI.

## Stateless nodes (Warewulf-style)

For HPC compute nodes Rollcall can boot a root image straight into RAM and
dress it with per-node files, instead of installing an OS on disk.

* **Images** live in `images_dir/<name>/{vmlinuz,initrd.img,filesystem.squashfs}`
  and are built with `images/build-rootfs.sh <name>` from
  `images/<name>/{packages.list,hook.chroot,includes/}` (live-build, same
  toolchain as the commissioning image). `images/compute-debian13` is a
  Debian 13 compute node: slurmd, munge, rdma-core, NFS client, firmware.
* **Profiles** bind an image to overlays, extra kernel parameters, a node name
  format (`node%03d`) and variables: `rollcall profile add compute -image compute-debian13 -overlays compute`.
* **Assigning** a commissioned machine to a profile allocates a node name and
  IPs from `node_pools`, writes a per-MAC PXE entry that boots the image with
  `hostname=`, `rollcall.server=` and a per-node token, and exports static
  DHCP leases plus host names for dnsmasq (`dnsmasq_dir`, re-read on HUP via
  `rollcall-dnsmasq.path`). `rollcall assign <id> compute -boot` also
  power-cycles the machine through its BMC.
* **Overlays** are directory trees under `overlays_dir/<overlay>/{system,runtime}`;
  files ending in `.tmpl` are Go templates rendered with the node, all nodes of
  the cluster, the profile and `cluster_vars`. The `compute` overlay generates
  `/etc/hosts`, `slurm.conf` (node lines from the commissioning inventory:
  sockets, cores, threads, memory, GPUs), `gres.conf`, a persistent SSH host
  key per node (sealed in the database), root `authorized_keys` and the munge
  key. Inside the image `rollcall-overlay.service` applies the system overlay
  before munge/slurmd/ssh start; a timer refreshes the runtime overlay every
  five minutes, which doubles as the node heartbeat (`booted` state, last seen).
* The controller renders the same files: `rollcall render compute etc/slurm/slurm.conf > /etc/slurm/slurm.conf`.

States: `ready → stateless → booted`; `rollcall unassign` releases name and IPs.

## OS deployment (install to disk)

For nodes that should run an operating system from their own disk, Rollcall
writes a vendor cloud image onto the disk and lets cloud-init do first-boot
configuration. No installer, no kickstart/preseed: a deployment is a streamed
disk write and takes about as long as the image download.

```
rollcall deploy <id> rocky-9 -boot -watch
   │ server: picks the disk (smallest that fits, or -disk), allocates hostname + IP,
   │         creates a job with a one-time token, writes pxelinux.cfg/01-<mac>
   │         ("deploy" entry = commissioning image with rollcall.mode=deploy), power-cycles via BMC
   ▼
commissioning image, deploy mode (agent/rollcall-deploy.py)
   GET /api/v1/jobs/self        → image URL, sha256, disk, rendered cloud-init seed
   curl | tee sha256 | pv | zstd -d | dd of=/dev/sda      (progress every 3 s)
   mount root fs → /var/lib/cloud/seed/nocloud/{meta-data,user-data}, datasource_list: [NoCloud, None]
   (SELinux images: /.autorelabel; UEFI hosts: efibootmgr entry for the ESP loader)
   POST progress state=done  → server: state deployed, PXE entry = local disk, reboot
   ▼
first boot: cloud-init sets hostname/fqdn, SSH keys (root + default user), optional
package_upgrade, then phone_home → POST /api/v1/phone-home/<token> (shown as "phoned home")
```

### Image catalog and freshness

```
os_dir/<name>/<version>/disk.raw.zst     raw disk image, zstd
os_dir/<name>/<version>/meta.json        sizes, sha256, upstream checksum, import time
os_dir/<name>/current -> <version>       what new deployments use
```

`os_sources` (`deploy/os-sources.json`) lists upstream "latest" URLs with the
vendor checksum file for each: Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9,
CentOS Stream 9. `rollcall os sync` fetches the checksum file, and only when the
vendor checksum differs from every imported version does it download the image,
verify it, convert to raw (`qemu-img`), compress, write a new version directory
and flip `current` atomically. `rollcall-os-sync.timer` runs it nightly;
`keep` versions are retained per source, nothing younger than a day is pruned,
and a job always pins the exact version it started with. Sources with
`"track": "pinned"` still download new versions but keep `current` until
`rollcall os promote <name> <version>`. Local files go in with
`rollcall os import <name> <file.qcow2>`. Image freshness is monthly at best
upstream; the deploy option `-upgrade` (`package_upgrade`) pulls the rest on
first boot.

```sh
rollcall os list | sync [NAME...] [-force] | import NAME FILE [-family debian|rhel] | promote NAME VERSION | rm NAME VERSION
rollcall deploy ID OS [-version V] [-disk /dev/nvme0n1] [-hostname H] [-upgrade] [-packages a,b] [-user-data FILE] [-boot] [-watch]
rollcall release ID [-wipe quick|full] [-boot] [-watch]     # wipe the disk, back to ready
rollcall jobs ID [-watch] | job cancel ID
```

States: `ready → deploying → deployed` (`deploy-failed` on error; the agent stays
up for ssh). A deployed machine's PXE entry defaults to local disk, so boot order
can stay "network first" on BIOS hosts; on UEFI hosts the agent adds a boot entry
for the installed loader. `rollcall recommission` or `release` take it back.

## Tags

Tags single out machines for special treatment, as in MAAS. A tag has a name
and optionally a **match rule**, a **cloud-config fragment** and **overlays**.

```sh
rollcall tags add gpu -match '{{ gt .Node.GPUs 0 }}' -cloud-config deploy/tags/gpu-nvidia.yaml -comment "NVIDIA nodes"
rollcall tags add rack-3 -comment "rack 3, row B"        # manual only
rollcall tag m-2ef1dab828c8 +rack-3 +burn-in -staging     # manual tags on one machine
rollcall list -tag gpu
rollcall deploy tag:gpu rocky-9 -boot                    # every gpu machine
rollcall power tag:rack-3 off
```

* **Auto tags** come from the match rule, a Go template over `.Node` (GPUs,
  IB GUIDs, CPUs, memory, disks, NVMe, PCI vendor ids, manufacturer, product,
  OS, existing tags). They are recomputed at every commissioning and whenever a
  tag definition is saved; manual tags are never touched by rules.
* **At deployment** the fragments of the machine's tags are appended to the
  generated user-data as MIME parts (`packages`, `runcmd`, `write_files` merge
  additively; a `#!` fragment runs as a script). `deploy/tags/gpu-nvidia.yaml`
  installs the NVIDIA driver on Debian, Ubuntu and the RHEL family and reboots.
* **For stateless nodes** a tag's overlays are applied after the profile's, so a
  `gpu` tag can add `gres.conf` lines or driver modules without a second profile.

## Physical location

Every machine can carry building / room / rack / rack unit and an asset tag.

```sh
rollcall location m-2ef1dab828c8 -building DC1 -room "Hall A" -rack R12 -unit 24 -asset A-0042
rollcall location tag:gpu -building DC1 -room "Hall B"             # bulk by tag
rollcall location import racks.csv -dry                             # from a DCIM / asset export
rollcall location racks | rack R12 -building DC1 -room "Hall A" -height 48
rollcall list -rack R12
```

The CSV needs a header; rows are matched by `id`, `serial`, `mac` or `node_name`
and may carry `building, room, rack, unit, asset_tag`. Racks are created on
first use (42U) and can be resized. The web UI shows a location column with
building › room › rack chips, a **Racks** page with elevation views (machines
drawn at their U, coloured by state) and bulk "Set location…". Templates and
tag rules see `.Node.Building`, `.Node.Room`, `.Node.Rack`, `.Node.Unit`, so a
rack can drive a tag (`{{ eq .Node.Rack "R12" }}`) or a Slurm `topology.conf`
overlay.

## Lab

`lab/` documents a Proxmox SDN test setup: an isolated vnet, a PXE server VM
running dnsmasq + Rollcall, two diskless VMs, and `pve-vbmc`, a tiny
IPMI-over-LAN BMC emulator that drives VMs through the Proxmox API so that
`ipmitool power on` and `chassis bootdev pxe` work against virtual machines.

## Status

Commissioning, stateless nodes and disk deployment work end to end in the lab
(virtual machines with 30 GB disks, SeaBIOS, out-of-band BMC hints). The
in-band BMC path and the UEFI boot-entry step are implemented but have not yet
been exercised on real hardware.