rollcall/README.md

Rollcall

Rollcall is a small, self-contained commissioning service for bare-metal servers, in the spirit of MAAS enlistment and commissioning, without the rest of MAAS.

A server that PXE-boots for the first time gets an ephemeral Debian live image. The image takes a full hardware inventory (DMI, DIMMs, CPUs, disks, NVMe, NICs, InfiniBand HCAs, GPUs, PCI, LLDP neighbours, TPM), creates an administrator account on the server's BMC through in-band IPMI, reports everything to the Rollcall server and powers the machine off. From then on the server is known: it shows up in the web UI and the CLI, its BMC can be driven through the API, and it will not boot the commissioning image again unless you ask for it.

Everything server-side is one static Go binary: HTTP API, embedded web UI, CLI, PostgreSQL schema management and the PXE bookkeeping.

How it works

unknown server ──PXE──▶ dnsmasq (DHCP+TFTP) ──▶ pxelinux default menu
                                                  │ 8 s timeout
                                                  ▼
                               Debian live image over HTTP (kernel+initrd via TFTP,
                               squashfs via HTTP, runs entirely from RAM)
                                                  │
                      rollcall-agent: inventory + BMC user + POST /api/v1/commission
                                                  │
                     rollcall server: PostgreSQL ◀─┘──▶ writes pxelinux.cfg/01-<mac>
                                                       (next boot: "commissioned → power off")

• Enlist: a machine is identified by its DMI system UUID (fallback: serial, then boot MAC). First contact creates it in state new.
• Commission: the inventory upload moves it to ready. The BMC password the agent generated is sealed with AES-256-GCM (key in /etc/rollcall/master.key) before it touches the database. It never leaves the server again: power and boot-device operations run ipmitool server-side.
• Re-commission: removes the per-MAC PXE override; optionally sets bootdev=pxe and power-cycles the machine through its BMC.

Repository layout

Path	What
cmd/rollcall/	Go server + CLI + embedded web UI (web/)
agent/	rollcall-agent.py (runs inside the live image), its systemd unit, build-image.sh (live-build recipe)
deploy/	install.sh (region), install-rack.sh + rack.example.json (rack controller), units, config.example.json, os-sources.json, tags/, overlays/
images/	stateless root image recipes (build-rootfs.sh, compute-debian13/)
lab/	Proxmox test lab: pve-vbmc (virtual IPMI BMC for VMs backed by the Proxmox API), cloud-init for the PXE server VM
site/	Project website (static)
.gitea/workflows/	CI (fmt/vet/build/tests), release on tags, deploy on push to main

Server

# build
cd cmd/rollcall && CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o rollcall .

# PostgreSQL role + database
sudo -u postgres psql -c "CREATE ROLE rollcall LOGIN PASSWORD '...';" -c "CREATE DATABASE rollcall OWNER rollcall;"

# config (see deploy/config.example.json), tokens from `rollcall keygen`
sudo install -d -m 0750 /etc/rollcall && sudo install -m 0640 config.json /etc/rollcall/config.json

# install binary, systemd unit, pxelinux menu, tftp symlinks
cd deploy && sudo ./install.sh

config.json keys: listen, image_dir (served under /rollcall/), tftp_root, database_url, master_key_file, admin_token (web UI + management API), agent_token (what the live image presents), commission_append (kernel command line baked into generated PXE files), bmc_hints (out-of-band BMC mapping for machines without in-band IPMI, e.g. VMs).

Console (same binary)

A bare rollcall in a terminal opens a full-screen console, built like the llmgw one (Bubble Tea): the fleet down the left (Overview, Machines, Racks, Jobs, Events) and provisioning (Tags, Profiles, OS images), the chosen page on the right, keys along the bottom. It is keyboard only, so it works on serial consoles and BMC KVMs.

• Machines: / filter (name, serial, MAC, IP, rack, tag), s cycles the state filter, Space selects, * selects everything shown, Enter opens a machine, a opens the actions for the selection or the current row: power, boot device, deploy OS, assign/unassign profile, tags, location, re-commission, release, cancel job, delete. Destructive ones ask first.
• Racks: Enter shows the elevation; Enter on a unit opens that machine.
• Jobs and events open the machine they belong to; tags, profiles and OS images show their definition. r refreshes (it also refreshes on its own, every 3 s while a job runs), ? lists the keys, q quits.

On the server as root it reads /etc/rollcall/config.json. Anywhere else it asks for the server URL and admin token once and keeps them in ~/.config/rollcall/client.json (mode 0600), which the CLI then uses as well. In a pipe or a script a bare rollcall prints its help as before. rollcall tui -dump 140x40 -page machines -plain renders one screen to stdout.

CLI (same binary)

rollcall list
rollcall show m-2ef1dab828c8 [-json]
rollcall events m-2ef1dab828c8
rollcall power m-2ef1dab828c8 status|on|off|soft|cycle|reset
rollcall bootdev m-2ef1dab828c8 pxe|disk|cdrom|bios|none
rollcall recommission m-2ef1dab828c8 [-now]
rollcall delete m-2ef1dab828c8

Reads ROLLCALL_URL / ROLLCALL_TOKEN, or the config file when readable.

API

Method	Path	Auth	Purpose
POST	/api/v1/commission	agent token	inventory upload from the live image
GET	/api/v1/bmc-hint?macs=	agent token	out-of-band BMC hint for VMs
GET	/api/v1/machines	admin	list (summaries)
GET	/api/v1/machines/{id}	admin	full record: inventory, BMC (no secret), events
POST	/api/v1/machines/{id}/power	admin	{"action":"on"} etc., runs ipmitool server-side
POST	/api/v1/machines/{id}/bootdev	admin	{"device":"pxe"}
POST	/api/v1/machines/{id}/recommission[?now=1]	admin	boot the image again on next PXE (now=1: via BMC immediately)
DELETE	/api/v1/machines/{id}	admin	forget the machine
GET/POST/DELETE	/api/v1/profiles[/{name}]	admin	stateless profiles
POST	/api/v1/machines/{id}/assign /unassign	admin	stateless node assignment
GET	/api/v1/os	admin	OS image catalog
POST	/api/v1/machines/{id}/deploy	admin	{"os":"rocky-9","disk":"/dev/sda","hostname":"","package_upgrade":false,"boot":true}
POST	/api/v1/machines/{id}/release	admin	wipe job
GET / POST	/api/v1/machines/{id}/jobs[/cancel]	admin	job history / cancel
GET	/api/v1/jobs/self, POST …/progress	job token (X-Rollcall-Job)	deploy agent
POST	/api/v1/phone-home/{token}	job token in path	cloud-init phone_home
GET	/os/*	none	OS disk images
GET	/rollcall/*	none	image files (kernel, initrd, squashfs)
GET	/	token in browser	web UI

Admin auth is Authorization: Bearer <admin_token>; the agent sends X-Rollcall-Token.

Commissioning image

agent/build-image.sh builds a Debian 13 live image with live-build (about 8 minutes, ~780 MB squashfs). It contains the full Debian kernel with non-free firmware, ipmitool, freeipmi, rdma-core, infiniband-diags, mstflint, nvme-cli, smartmontools, lldpd, lshw and an SSH server (root, key only) for debugging. Output goes to image_dir.

Kernel parameters understood by the agent: rollcall.server=URL (required), rollcall.token=…, rollcall.bmcuser=NAME (default commissioner), rollcall.nopoweroff (stay up after reporting).

BMC account

In-band (/dev/ipmi0): find the LAN channel, pick the existing commissioner slot or the first empty user ID ≥ 3, set name/password, ADMINISTRATOR privilege, channel access, enable, SOL payload. The password is random (16 chars) and is reported once to the server. If no in-band IPMI exists the agent asks the server for a hint (bmc_hints) and verifies it out-of-band.

Vendor quirks (iDRAC/iLO password policies, missing empty slots) are reported in bmc.errors and visible in the UI.

Stateless nodes (Warewulf-style)

For HPC compute nodes Rollcall can boot a root image straight into RAM and dress it with per-node files, instead of installing an OS on disk.

• Images live in images_dir/<name>/{vmlinuz,initrd.img,filesystem.squashfs} and are built with images/build-rootfs.sh <name> from images/<name>/{packages.list,hook.chroot,includes/} (live-build, same toolchain as the commissioning image). images/compute-debian13 is a Debian 13 compute node: slurmd, munge, rdma-core, NFS client, firmware.
• Profiles bind an image to overlays, extra kernel parameters, a node name format (node%03d) and variables: rollcall profile add compute -image compute-debian13 -overlays compute.
• Assigning a commissioned machine to a profile allocates a node name and IPs from node_pools, writes a per-MAC PXE entry that boots the image with hostname=, rollcall.server= and a per-node token, and exports static DHCP leases plus host names for dnsmasq (dnsmasq_dir, re-read on HUP via rollcall-dnsmasq.path). rollcall assign <id> compute -boot also power-cycles the machine through its BMC.
• Overlays are directory trees under overlays_dir/<overlay>/{system,runtime}; files ending in .tmpl are Go templates rendered with the node, all nodes of the cluster, the profile and cluster_vars. The compute overlay generates /etc/hosts, slurm.conf (node lines from the commissioning inventory: sockets, cores, threads, memory, GPUs), gres.conf, a persistent SSH host key per node (sealed in the database), root authorized_keys and the munge key. Inside the image rollcall-overlay.service applies the system overlay before munge/slurmd/ssh start; a timer refreshes the runtime overlay every five minutes, which doubles as the node heartbeat (booted state, last seen).
• The controller renders the same files: rollcall render compute etc/slurm/slurm.conf > /etc/slurm/slurm.conf.

States: ready → stateless → booted; rollcall unassign releases name and IPs.

OS deployment (install to disk)

For nodes that should run an operating system from their own disk, Rollcall writes a vendor cloud image onto the disk and lets cloud-init do first-boot configuration. No installer, no kickstart/preseed: a deployment is a streamed disk write and takes about as long as the image download.

rollcall deploy <id> rocky-9 -boot -watch
   │ server: picks the disk (smallest that fits, or -disk), allocates hostname + IP,
   │         creates a job with a one-time token, writes pxelinux.cfg/01-<mac>
   │         ("deploy" entry = commissioning image with rollcall.mode=deploy), power-cycles via BMC
   ▼
commissioning image, deploy mode (agent/rollcall-deploy.py)
   GET /api/v1/jobs/self        → image URL, sha256, disk, rendered cloud-init seed
   curl | tee sha256 | pv | zstd -d | dd of=/dev/sda      (progress every 3 s)
   mount root fs → /var/lib/cloud/seed/nocloud/{meta-data,user-data}, datasource_list: [NoCloud, None]
   (SELinux images: /.autorelabel; UEFI hosts: efibootmgr entry for the ESP loader)
   POST progress state=done  → server: state deployed, PXE entry = local disk, reboot
   ▼
first boot: cloud-init sets hostname/fqdn, SSH keys (root + default user), optional
package_upgrade, then phone_home → POST /api/v1/phone-home/<token> (shown as "phoned home")

Image catalog and freshness

os_dir/<name>/<version>/disk.raw.zst     raw disk image, zstd
os_dir/<name>/<version>/meta.json        sizes, sha256, upstream checksum, import time
os_dir/<name>/current -> <version>       what new deployments use

os_sources (deploy/os-sources.json) lists upstream "latest" URLs with the vendor checksum file for each: Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9, CentOS Stream 9. rollcall os sync fetches the checksum file, and only when the vendor checksum differs from every imported version does it download the image, verify it, convert to raw (qemu-img), compress, write a new version directory and flip current atomically. rollcall-os-sync.timer runs it nightly; keep versions are retained per source, nothing younger than a day is pruned, and a job always pins the exact version it started with. Sources with "track": "pinned" still download new versions but keep current until rollcall os promote <name> <version>. Local files go in with rollcall os import <name> <file.qcow2>. Image freshness is monthly at best upstream; the deploy option -upgrade (package_upgrade) pulls the rest on first boot.

rollcall os list | sync [NAME...] [-force] | import NAME FILE [-family debian|rhel] | promote NAME VERSION | rm NAME VERSION
rollcall deploy ID OS [-version V] [-disk /dev/nvme0n1] [-hostname H] [-upgrade] [-packages a,b] [-user-data FILE] [-boot] [-watch]
rollcall release ID [-wipe quick|full] [-boot] [-watch]     # wipe the disk, back to ready
rollcall jobs ID [-watch] | job cancel ID

States: ready → deploying → deployed (deploy-failed on error; the agent stays up for ssh). A deployed machine's PXE entry defaults to local disk, so boot order can stay "network first" on BIOS hosts; on UEFI hosts the agent adds a boot entry for the installed loader. rollcall recommission or release take it back.

Tags

Tags single out machines for special treatment, as in MAAS. A tag has a name and optionally a match rule, a cloud-config fragment and overlays.

rollcall tags add gpu -match '{{ gt .Node.GPUs 0 }}' -cloud-config deploy/tags/gpu-nvidia.yaml -comment "NVIDIA nodes"
rollcall tags add rack-3 -comment "rack 3, row B"        # manual only
rollcall tag m-2ef1dab828c8 +rack-3 +burn-in -staging     # manual tags on one machine
rollcall list -tag gpu
rollcall deploy tag:gpu rocky-9 -boot                    # every gpu machine
rollcall power tag:rack-3 off

• Auto tags come from the match rule, a Go template over .Node (GPUs, IB GUIDs, CPUs, memory, disks, NVMe, PCI vendor ids, manufacturer, product, OS, existing tags). They are recomputed at every commissioning and whenever a tag definition is saved; manual tags are never touched by rules.
• At deployment the fragments of the machine's tags are appended to the generated user-data as MIME parts (packages, runcmd, write_files merge additively; a #! fragment runs as a script). deploy/tags/gpu-nvidia.yaml installs the NVIDIA driver on Debian, Ubuntu and the RHEL family and reboots.
• For stateless nodes a tag's overlays are applied after the profile's, so a gpu tag can add gres.conf lines or driver modules without a second profile.

Physical location

Every machine can carry building / room / rack / rack unit and an asset tag.

rollcall location m-2ef1dab828c8 -building DC1 -room "Hall A" -rack R12 -unit 24 -asset A-0042
rollcall location tag:gpu -building DC1 -room "Hall B"             # bulk by tag
rollcall location import racks.csv -dry                             # from a DCIM / asset export
rollcall location racks | rack R12 -building DC1 -room "Hall A" -height 48
rollcall list -rack R12

The CSV needs a header; rows are matched by id, serial, mac or node_name and may carry building, room, rack, unit, asset_tag. Racks are created on first use (42U) and can be resized. The web UI shows a location column with building › room › rack chips, a Racks page with elevation views (machines drawn at their U, coloured by state) and bulk "Set location…". Templates and tag rules see .Node.Building, .Node.Room, .Node.Rack, .Node.Unit, so a rack can drive a tag ({{ eq .Node.Rack "R12" }}) or a Slurm topology.conf overlay.

Network and rack controllers

Model

As in MAAS: fabric → VLAN → subnet → IP ranges.

• A fabric is a switched layer-2 domain (ethernet or infiniband).
• A VLAN has a VID (0 = untagged; a partition key on IB), an MTU, and says whether rollcall serves DHCP/PXE on it and which rack controller does (a second one splits the dynamic pool with it).
• A subnet has a purpose (provisioning, mgmt, bmc, storage, ib, other), a gateway, DNS servers and a domain.
• Ranges inside a subnet are dynamic (the DHCP pool PXE and commissioning use), reserved (never handed out: routers, switches, controllers) and static (where machines get their addresses; without one, anything outside dynamic/reserved).
• IPAM records every address handed out (rollcall ip list), pinned ones (rollcall net set ID -static ens18=10.0.0.60) and reserved single addresses (rollcall ip reserve 10.0.0.5 -name switch-1). Addresses a machine already had are adopted when its subnet is defined.

rollcall fabric add hall-a
rollcall subnet add 10.50.0.0/24 -fabric hall-a -vid 0 -purpose provisioning -gateway 10.50.0.1 -dns 10.50.0.1 -domain dc.example
rollcall range add 10.50.0.0/24 reserved 10.50.0.1 10.50.0.20
rollcall range add 10.50.0.0/24 static   10.50.0.50 10.50.0.199
rollcall range add 10.50.0.0/24 dynamic  10.50.0.200 10.50.0.250
rollcall vlan add hall-a 0 -dhcp -rack hall-a-rack1      # this rack controller serves DHCP/PXE here
rollcall subnet add 10.60.0.0/24 -fabric hall-a -vid 200 -purpose storage
rollcall dhcp hall-a-rack1                                 # the dnsmasq configuration it generates

Machine interfaces

A layout describes interfaces in terms that fit many machines: NICs are chosen by selectors (boot, name:X, mac:X, fastest:N, ib:N, all), not by MAC. Interfaces are physical, bond (802.3ad by default) or vlan; each link names a subnet (CIDR, name, or boot) and a mode: auto (an address from the subnet), static (a given one), dhcp or link_up.

{"interfaces": [
  {"type": "physical", "select": "boot", "links": [{"subnet": "boot", "mode": "auto", "default_gateway": true}]},
  {"type": "bond", "name": "bond0", "members": ["fastest:2"], "mtu": 9000, "links": [{"mode": "link_up"}]},
  {"type": "vlan", "parent": "bond0", "vid": 200, "links": [{"subnet": "10.60.0.0/24", "mode": "auto"}]},
  {"type": "physical", "select": "ib:0", "links": [{"subnet": "ib0", "mode": "auto"}]}
]}

rollcall layout add gpu-bond layout.json, then either per machine (rollcall net set ID -layout gpu-bond) or per tag (rollcall tags add gpu … -layout gpu-bond). Without a layout the boot NIC gets an auto address on its subnet; without any subnet model, deployed machines use DHCP as before.

At deployment the layout is resolved against the machine's inventory, addresses are allocated, and two files go into the image next to the cloud-init seed: network-config (cloud-init v2; cloud-init renders it for netplan, NetworkManager or ifupdown) and a udev rule that pins each NIC's name to its MAC, because RHEL-family cloud images boot with net.ifnames=0. Release returns the allocated addresses. rollcall net show ID previews it all.

Verified in the lab: Debian 13 and Ubuntu 24.04 (netplan) and Rocky 9 (NetworkManager), with a static boot NIC and with an active-backup bond carrying VLAN 200.

Region and rack controllers

rollcall serve is the region: database, API, web UI, IPAM, image store. Rack controllers sit next to the machines, one or two per network segment (hall, building, site):

• they serve DHCP and TFTP with dnsmasq on the VLANs assigned to them; the configuration, per-MAC PXE files and host files are computed by the region;
• they run BMC commands (BMC networks are rarely routed to the region);
• they serve boot and OS images from a local cache, filled from the region on first use (kernels and initrds are prefetched, TFTP cannot wait);
• they pass the node API (commissioning, deploy jobs, overlays, phone-home) through to the region and tell it which rack a machine came through.

A rack controller only needs to reach the region: it long-polls its desired state and a command queue. The region host runs an embedded rack controller (rack_name, default its hostname), which is all a single-server installation needs. Remote ones:

# region: config.json gets "rack_secret": "<rollcall keygen>"
# rack host (Debian/Ubuntu), from deploy/ with the binary next to it:
cat > /etc/rollcall/rack.json <<EOF
{"name": "hall-b", "region": "http://10.0.0.234", "secret": "<rack_secret>",
 "url": "http://10.213.78.1", "listen": "10.213.78.1:80"}
EOF
sh install-rack.sh            # dnsmasq, syslinux, cache dirs, rollcall-rack.service
rollcall controllers          # on the region: it shows up online
rollcall vlan add hall-b 0 -dhcp -rack hall-b

dnsmasq is restarted only when its generated configuration changes, and only if it comes back; otherwise the last working configuration is restored (a hand-written DHCP configuration for the same interface is the usual cause: install.sh points it out). A machine that moved to another segment gets rollcall net set ID -rack NAME.

The lab runs both: pxe-server (embedded, 10.213.77.0/24) and pxe-rack2 (VM 213, remote, 10.213.78.0/24 on SDN vnet pxenet2), with node002 behind pxe-rack2 and its virtual BMC on pxe-rack2. Rack ↔ region traffic is plain HTTP in the lab; BMC passwords travel on it for queued commands, so put the region behind TLS on real networks.

Lab

lab/ documents a Proxmox SDN test setup: an isolated vnet, a PXE server VM running dnsmasq + Rollcall, two diskless VMs, and pve-vbmc, a tiny IPMI-over-LAN BMC emulator that drives VMs through the Proxmox API so that ipmitool power on and chassis bootdev pxe work against virtual machines.

Status

Commissioning, stateless nodes and disk deployment work end to end in the lab (virtual machines with 30 GB disks, SeaBIOS, out-of-band BMC hints). The in-band BMC path and the UEFI boot-entry step are implemented but have not yet been exercised on real hardware.