rollcall/README.md

# Rollcall

Rollcall is a small, self-contained commissioning service for bare-metal servers,
in the spirit of MAAS enlistment and commissioning, without the rest of MAAS.

A server that PXE-boots for the first time gets an ephemeral Debian live image.
The image takes a full hardware inventory (DMI, DIMMs, CPUs, disks, NVMe, NICs,
InfiniBand HCAs, GPUs, PCI, LLDP neighbours, TPM), creates an administrator
account on the server's BMC through in-band IPMI, reports everything to the
Rollcall server and powers the machine off. From then on the server is known:
it shows up in the web UI and the CLI, its BMC can be driven through the API,
and it will not boot the commissioning image again unless you ask for it.

Everything server-side is **one static Go binary**: HTTP API, embedded web UI,
CLI, PostgreSQL schema management and the PXE bookkeeping.

## How it works

```
unknown server ──PXE──▶ dnsmasq (DHCP+TFTP) ──▶ pxelinux default menu
                                                  │ 8 s timeout
                                                  ▼
                               Debian live image over HTTP (kernel+initrd via TFTP,
                               squashfs via HTTP, runs entirely from RAM)
                                                  │
                      rollcall-agent: inventory + BMC user + POST /api/v1/commission
                                                  │
                     rollcall server: PostgreSQL ◀─┘──▶ writes pxelinux.cfg/01-<mac>
                                                       (next boot: "commissioned → power off")
```

* **Enlist**: a machine is identified by its DMI system UUID (fallback: serial,
  then boot MAC). First contact creates it in state `new`.
* **Commission**: the inventory upload moves it to `ready`. The BMC password
  the agent generated is sealed with AES-256-GCM (key in
  `/etc/rollcall/master.key`) before it touches the database. It never leaves
  the server again: power and boot-device operations run `ipmitool` server-side.
* **Re-commission**: removes the per-MAC PXE override; optionally sets
  bootdev=pxe and power-cycles the machine through its BMC.

## Repository layout

| Path | What |
|---|---|
| `cmd/rollcall/` | Go server + CLI + embedded web UI (`web/`) |
| `agent/` | `rollcall-agent.py` (runs inside the live image), its systemd unit, `build-image.sh` (live-build recipe) |
| `deploy/` | `install.sh` (region), `install-rack.sh` + `rack.example.json` (rack controller), units, `config.example.json`, `os-sources.json`, `tags/`, `overlays/` |
| `images/` | stateless root image recipes (`build-rootfs.sh`, `compute-debian13/`) |
| `lab/` | Proxmox test lab: `pve-vbmc` (virtual IPMI BMC for VMs backed by the Proxmox API), cloud-init for the PXE server VM |
| `site/` | Project website (static) |
| `.gitea/workflows/` | CI (fmt/vet/build/tests), release on tags, deploy on push to `main` |

## Server

New site: **[INSTALL.md](INSTALL.md)** (`deploy/bootstrap.sh`, from day 0 without a network plan to the first deployed machine). The steps below are the manual equivalent.

```sh
# build
cd cmd/rollcall && CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o rollcall .

# PostgreSQL role + database
sudo -u postgres psql -c "CREATE ROLE rollcall LOGIN PASSWORD '...';" -c "CREATE DATABASE rollcall OWNER rollcall;"

# config (see deploy/config.example.json), tokens from `rollcall keygen`
sudo install -d -m 0750 /etc/rollcall && sudo install -m 0640 config.json /etc/rollcall/config.json

# install binary, systemd unit, pxelinux menu, tftp symlinks
cd deploy && sudo ./install.sh
```

`config.json` keys: `listen`, `image_dir` (served under `/rollcall/`), `tftp_root`,
`database_url`, `master_key_file`, `admin_token` (web UI + management API),
`agent_token` (what the live image presents), `commission_append` (kernel
command line baked into generated PXE files), `bmc_hints` (out-of-band BMC
mapping for machines without in-band IPMI, e.g. VMs).

### Console (same binary)

A bare `rollcall` in a terminal opens a full-screen console, built like the
llmgw one (Bubble Tea): the fleet down the left (Overview, Machines, Racks,
Jobs, Events) and provisioning (Tags, Profiles, OS images), the chosen page on
the right, keys along the bottom. It is keyboard only, so it works on serial
consoles and BMC KVMs.

* Machines: `/` filter (name, serial, MAC, IP, rack, tag), `s` cycles the state
  filter, `Space` selects, `*` selects everything shown, `Enter` opens a
  machine, `a` opens the actions for the selection or the current row: power,
  boot device, deploy OS, assign/unassign profile, tags, location,
  re-commission, release, cancel job, delete. Destructive ones ask first.
* Racks: `Enter` shows the elevation; `Enter` on a unit opens that machine.
* Jobs and events open the machine they belong to; tags, profiles and OS
  images show their definition. `r` refreshes (it also refreshes on its own,
  every 3 s while a job runs), `?` lists the keys, `q` quits.

On the server as root it reads `/etc/rollcall/config.json`. Anywhere else it
asks for the server URL and admin token once and keeps them in
`~/.config/rollcall/client.json` (mode 0600), which the CLI then uses as well.
In a pipe or a script a bare `rollcall` prints its help as before.
`rollcall tui -dump 140x40 -page machines -plain` renders one screen to stdout.

### CLI (same binary)

```sh
rollcall list
rollcall show m-2ef1dab828c8 [-json]
rollcall events m-2ef1dab828c8
rollcall power m-2ef1dab828c8 status|on|off|soft|cycle|reset
rollcall bootdev m-2ef1dab828c8 pxe|disk|cdrom|bios|none
rollcall recommission m-2ef1dab828c8 [-now]
rollcall delete m-2ef1dab828c8
```

Reads `ROLLCALL_URL` / `ROLLCALL_TOKEN`, or the config file when readable.

### API

| Method | Path | Auth | Purpose |
|---|---|---|---|
| POST | `/api/v1/commission` | agent token | inventory upload from the live image |
| GET | `/api/v1/bmc-hint?macs=` | agent token | out-of-band BMC hint for VMs |
| GET | `/api/v1/machines` | admin | list (summaries) |
| GET | `/api/v1/machines/{id}` | admin | full record: inventory, BMC (no secret), events |
| POST | `/api/v1/machines/{id}/power` | admin | `{"action":"on"}` etc., runs ipmitool server-side |
| POST | `/api/v1/machines/{id}/bootdev` | admin | `{"device":"pxe"}` |
| POST | `/api/v1/machines/{id}/recommission[?now=1]` | admin | boot the image again on next PXE (`now=1`: via BMC immediately) |
| DELETE | `/api/v1/machines/{id}` | admin | forget the machine |
| GET/POST/DELETE | `/api/v1/profiles[/{name}]` | admin | stateless profiles |
| POST | `/api/v1/machines/{id}/assign` `/unassign` | admin | stateless node assignment |
| GET | `/api/v1/os` | admin | OS image catalog |
| POST | `/api/v1/machines/{id}/deploy` | admin | `{"os":"rocky-9","disk":"/dev/sda","hostname":"","package_upgrade":false,"boot":true}` |
| POST | `/api/v1/machines/{id}/release` | admin | wipe job |
| GET / POST | `/api/v1/machines/{id}/jobs[/cancel]` | admin | job history / cancel |
| GET | `/api/v1/jobs/self`, POST `…/progress` | job token (`X-Rollcall-Job`) | deploy agent |
| POST | `/api/v1/phone-home/{token}` | job token in path | cloud-init phone_home |
| GET | `/os/*` | none | OS disk images |
| GET | `/rollcall/*` | none | image files (kernel, initrd, squashfs) |
| GET | `/` | token in browser | web UI |

Admin auth is `Authorization: Bearer <admin_token>`; the agent sends
`X-Rollcall-Token`.

## Commissioning image

`agent/build-image.sh` builds a Debian 13 live image with live-build (about
8 minutes, ~780 MB squashfs). It contains the full Debian kernel with non-free
firmware, `ipmitool`, `freeipmi`, `rdma-core`, `infiniband-diags`, `mstflint`,
`nvme-cli`, `smartmontools`, `lldpd`, `lshw` and an SSH server (root, key only)
for debugging. Output goes to `image_dir`.

Kernel parameters understood by the agent: `rollcall.server=URL` (required),
`rollcall.token=…`, `rollcall.bmcuser=NAME` (default `commissioner`),
`rollcall.nopoweroff` (stay up after reporting).

### BMC account

In-band (`/dev/ipmi0`): find the LAN channel, pick the existing `commissioner`
slot or the first empty user ID ≥ 3, set name/password, ADMINISTRATOR privilege,
channel access, enable, SOL payload. The password is random (16 chars) and is
reported once to the server. If no in-band IPMI exists the agent asks the server
for a hint (`bmc_hints`) and verifies it out-of-band.

Vendor quirks (iDRAC/iLO password policies, missing empty slots) are reported in
`bmc.errors` and visible in the UI.

## Stateless nodes (Warewulf-style)

For HPC compute nodes Rollcall can boot a root image straight into RAM and
dress it with per-node files, instead of installing an OS on disk.

* **Images** live in `images_dir/<name>/{vmlinuz,initrd.img,filesystem.squashfs}`
  and are built with `images/build-rootfs.sh <name>` from
  `images/<name>/{packages.list,hook.chroot,includes/}` (live-build, same
  toolchain as the commissioning image). `images/compute-debian13` is a
  Debian 13 compute node: slurmd, munge, rdma-core, NFS client, firmware.
* **Profiles** bind an image to overlays, extra kernel parameters, a node name
  format (`node%03d`) and variables: `rollcall profile add compute -image compute-debian13 -overlays compute`.
* **Assigning** a commissioned machine to a profile allocates a node name and
  IPs from `node_pools`, writes a per-MAC PXE entry that boots the image with
  `hostname=`, `rollcall.server=` and a per-node token, and exports static
  DHCP leases plus host names for dnsmasq (`dnsmasq_dir`, re-read on HUP via
  `rollcall-dnsmasq.path`). `rollcall assign <id> compute -boot` also
  power-cycles the machine through its BMC.
* **Overlays** are directory trees under `overlays_dir/<overlay>/{system,runtime}`;
  files ending in `.tmpl` are Go templates rendered with the node, all nodes of
  the cluster, the profile and `cluster_vars`. The `compute` overlay generates
  `/etc/hosts`, `slurm.conf` (node lines from the commissioning inventory:
  sockets, cores, threads, memory, GPUs), `gres.conf`, a persistent SSH host
  key per node (sealed in the database), root `authorized_keys` and the munge
  key. Inside the image `rollcall-overlay.service` applies the system overlay
  before munge/slurmd/ssh start; a timer refreshes the runtime overlay every
  five minutes, which doubles as the node heartbeat (`booted` state, last seen).
* The controller renders the same files: `rollcall render compute etc/slurm/slurm.conf > /etc/slurm/slurm.conf`.

States: `ready → stateless → booted`; `rollcall unassign` releases name and IPs.

## OS deployment (install to disk)

For nodes that should run an operating system from their own disk, Rollcall
writes a vendor cloud image onto the disk and lets cloud-init do first-boot
configuration. No installer, no kickstart/preseed: a deployment is a streamed
disk write and takes about as long as the image download.

```
rollcall deploy <id> rocky-9 -boot -watch
   │ server: picks the disk (smallest that fits, or -disk), allocates hostname + IP,
   │         creates a job with a one-time token, writes pxelinux.cfg/01-<mac>
   │         ("deploy" entry = commissioning image with rollcall.mode=deploy), power-cycles via BMC
   ▼
commissioning image, deploy mode (agent/rollcall-deploy.py)
   GET /api/v1/jobs/self        → image URL, sha256, disk, rendered cloud-init seed
   curl | tee sha256 | pv | zstd -d | dd of=/dev/sda      (progress every 3 s)
   mount root fs → /var/lib/cloud/seed/nocloud/{meta-data,user-data}, datasource_list: [NoCloud, None]
   (SELinux images: /.autorelabel; UEFI hosts: efibootmgr entry for the ESP loader)
   POST progress state=done  → server: state deployed, PXE entry = local disk, reboot
   ▼
first boot: cloud-init sets hostname/fqdn, SSH keys (root + default user), optional
package_upgrade, then phone_home → POST /api/v1/phone-home/<token> (shown as "phoned home")
```

### Image catalog and freshness

```
os_dir/<name>/<version>/disk.raw.zst     raw disk image, zstd
os_dir/<name>/<version>/meta.json        sizes, sha256, upstream checksum, import time
os_dir/<name>/current -> <version>       what new deployments use
```

`os_sources` (`deploy/os-sources.json`) lists upstream "latest" URLs with the
vendor checksum file for each: Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9,
CentOS Stream 9. `rollcall os sync` fetches the checksum file, and only when the
vendor checksum differs from every imported version does it download the image,
verify it, convert to raw (`qemu-img`), compress, write a new version directory
and flip `current` atomically. `rollcall-os-sync.timer` runs it nightly;
`keep` versions are retained per source, nothing younger than a day is pruned,
and a job always pins the exact version it started with. Sources with
`"track": "pinned"` still download new versions but keep `current` until
`rollcall os promote <name> <version>`. Local files go in with
`rollcall os import <name> <file.qcow2>`. Image freshness is monthly at best
upstream; the deploy option `-upgrade` (`package_upgrade`) pulls the rest on
first boot.

```sh
rollcall os list | sync [NAME...] [-force] | import NAME FILE [-family debian|rhel] | promote NAME VERSION | rm NAME VERSION
rollcall deploy ID OS [-version V] [-disk /dev/nvme0n1] [-hostname H] [-upgrade] [-packages a,b] [-user-data FILE] [-boot] [-watch]
rollcall release ID [-wipe quick|full] [-boot] [-watch]     # wipe the disk, back to ready
rollcall jobs ID [-watch] | job cancel ID
```

States: `ready → deploying → deployed` (`deploy-failed` on error; the agent stays
up for ssh). A deployed machine's PXE entry defaults to local disk, so boot order
can stay "network first" on BIOS hosts; on UEFI hosts the agent adds a boot entry
for the installed loader. `rollcall recommission` or `release` take it back.

## Tags

Tags single out machines for special treatment, as in MAAS. A tag has a name
and optionally a **match rule**, a **cloud-config fragment** and **overlays**.

```sh
rollcall tags add gpu -match '{{ gt .Node.GPUs 0 }}' -cloud-config deploy/tags/gpu-nvidia.yaml -comment "NVIDIA nodes"
rollcall tags add rack-3 -comment "rack 3, row B"        # manual only
rollcall tag m-2ef1dab828c8 +rack-3 +burn-in -staging     # manual tags on one machine
rollcall list -tag gpu
rollcall deploy tag:gpu rocky-9 -boot                    # every gpu machine
rollcall power tag:rack-3 off
```

* **Auto tags** come from the match rule, a Go template over `.Node` (GPUs,
  IB GUIDs, CPUs, memory, disks, NVMe, PCI vendor ids, manufacturer, product,
  OS, existing tags). They are recomputed at every commissioning and whenever a
  tag definition is saved; manual tags are never touched by rules.
* **At deployment** the fragments of the machine's tags are appended to the
  generated user-data as MIME parts (`packages`, `runcmd`, `write_files` merge
  additively; a `#!` fragment runs as a script). `deploy/tags/gpu-nvidia.yaml`
  installs the NVIDIA driver on Debian, Ubuntu and the RHEL family and reboots.
* **For stateless nodes** a tag's overlays are applied after the profile's, so a
  `gpu` tag can add `gres.conf` lines or driver modules without a second profile.

## Physical location

Every machine can carry building / room / rack / rack unit and an asset tag.

```sh
rollcall location m-2ef1dab828c8 -building DC1 -room "Hall A" -rack R12 -unit 24 -asset A-0042
rollcall location tag:gpu -building DC1 -room "Hall B"             # bulk by tag
rollcall location import racks.csv -dry                             # from a DCIM / asset export
rollcall location racks | rack R12 -building DC1 -room "Hall A" -height 48
rollcall list -rack R12
```

The CSV needs a header; rows are matched by `id`, `serial`, `mac` or `node_name`
and may carry `building, room, rack, unit, asset_tag`. Racks are created on
first use (42U) and can be resized. The web UI shows a location column with
building › room › rack chips, a **Racks** page with elevation views (machines
drawn at their U, coloured by state) and bulk "Set location…". Templates and
tag rules see `.Node.Building`, `.Node.Room`, `.Node.Rack`, `.Node.Unit`, so a
rack can drive a tag (`{{ eq .Node.Rack "R12" }}`) or a Slurm `topology.conf`
overlay.

## Network and rack controllers

### Model

As in MAAS: **fabric → VLAN → subnet → IP ranges**.

* A **fabric** is a switched layer-2 domain (`ethernet` or `infiniband`).
* A **VLAN** has a VID (0 = untagged; a partition key on IB), an MTU, and
  says whether rollcall serves DHCP/PXE on it and which **rack controller**
  does (a second one splits the dynamic pool with it).
* A **subnet** has a purpose (provisioning, mgmt, bmc, storage, ib, other), a
  gateway, DNS servers and a domain.
* **Ranges** inside a subnet are `dynamic` (the DHCP pool PXE and
  commissioning use), `reserved` (never handed out: routers, switches,
  controllers) and `static` (where machines get their addresses; without one,
  anything outside dynamic/reserved).
* **IPAM** records every address handed out (`rollcall ip list`), pinned ones
  (`rollcall net set ID -static ens18=10.0.0.60`) and reserved single
  addresses (`rollcall ip reserve 10.0.0.5 -name switch-1`). Addresses a machine
  already had are adopted when its subnet is defined.

```sh
rollcall fabric add hall-a
rollcall subnet add 10.50.0.0/24 -fabric hall-a -vid 0 -purpose provisioning -gateway 10.50.0.1 -dns 10.50.0.1 -domain dc.example
rollcall range add 10.50.0.0/24 reserved 10.50.0.1 10.50.0.20
rollcall range add 10.50.0.0/24 static   10.50.0.50 10.50.0.199
rollcall range add 10.50.0.0/24 dynamic  10.50.0.200 10.50.0.250
rollcall vlan add hall-a 0 -dhcp -rack hall-a-rack1      # this rack controller serves DHCP/PXE here
rollcall subnet add 10.60.0.0/24 -fabric hall-a -vid 200 -purpose storage
rollcall dhcp hall-a-rack1                                 # the dnsmasq configuration it generates
```

### Machine interfaces

A **layout** describes interfaces in terms that fit many machines: NICs are
chosen by selectors (`boot`, `name:X`, `mac:X`, `fastest:N`, `ib:N`, `all`),
not by MAC. Interfaces are `physical`, `bond` (802.3ad by default) or `vlan`;
each link names a subnet (CIDR, name, or `boot`) and a mode: `auto` (an address
from the subnet), `static` (a given one), `dhcp` or `link_up`.

```json
{"interfaces": [
  {"type": "physical", "select": "boot", "links": [{"subnet": "boot", "mode": "auto", "default_gateway": true}]},
  {"type": "bond", "name": "bond0", "members": ["fastest:2"], "mtu": 9000, "links": [{"mode": "link_up"}]},
  {"type": "vlan", "parent": "bond0", "vid": 200, "links": [{"subnet": "10.60.0.0/24", "mode": "auto"}]},
  {"type": "physical", "select": "ib:0", "links": [{"subnet": "ib0", "mode": "auto"}]}
]}
```

`rollcall layout add gpu-bond layout.json`, then either per machine
(`rollcall net set ID -layout gpu-bond`) or per tag (`rollcall tags add gpu
… -layout gpu-bond`). Without a layout the boot NIC gets an `auto` address on
its subnet; without any subnet model, deployed machines use DHCP as before.

At deployment the layout is resolved against the machine's inventory,
addresses are allocated, and two files go into the image next to the
cloud-init seed: `network-config` (cloud-init v2; cloud-init renders it for
netplan, NetworkManager or ifupdown) and a udev rule that pins each NIC's name
to its MAC, because RHEL-family cloud images boot with `net.ifnames=0`.
Release returns the allocated addresses. `rollcall net show ID` previews it all.

Verified in the lab: Debian 13 and Ubuntu 24.04 (netplan) and Rocky 9
(NetworkManager), with a static boot NIC and with an active-backup bond
carrying VLAN 200.

### Region and rack controllers

`rollcall serve` is the **region**: database, API, web UI, IPAM, image store.
**Rack controllers** sit next to the machines, one or two per network segment
(hall, building, site):

* they serve DHCP and TFTP with dnsmasq on the VLANs assigned to them; the
  configuration, per-MAC PXE files and host files are computed by the region;
* they run BMC commands (BMC networks are rarely routed to the region);
* they serve boot and OS images from a local cache, filled from the region on
  first use (kernels and initrds are prefetched, TFTP cannot wait);
* they pass the node API (commissioning, deploy jobs, overlays, phone-home)
  through to the region and tell it which rack a machine came through.

A rack controller only needs to reach the region: it long-polls its desired
state and a command queue. The region host runs an **embedded** rack
controller (`rack_name`, default its hostname), which is all a single-server
installation needs. Remote ones:

```sh
rollcall controller join-code -name hall-b -addr 10.0.0.234:443     # on the region: one word, one use, 60 min
# on the rack host (Debian/Ubuntu), from deploy/ with the binary next to it:
./rollcall rack join rollcall-join1-... -url http://10.213.78.1     # writes /etc/rollcall/rack.json + certificate
sh install-rack.sh                                                  # dnsmasq, syslinux, cache dirs, rollcall-rack.service
rollcall controllers                                                # on the region: it shows up online
rollcall vlan add hall-b 0 -dhcp -rack hall-b
```

dnsmasq is restarted only when its generated configuration changes, and only
if it comes back; otherwise the last working configuration is restored (a
hand-written DHCP configuration for the same interface is the usual cause:
`install.sh` points it out). A machine that moved to another segment gets
`rollcall net set ID -rack NAME`.

The lab runs both: `pxe-server` (embedded, 10.213.77.0/24) and `pxe-rack2`
(VM 213, remote, 10.213.78.0/24 on SDN vnet `pxenet2`), with node002 behind
pxe-rack2 and its virtual BMC on pxe-rack2. Rack ↔ region traffic is mutual
TLS (next section), so the BMC passwords of queued commands are encrypted.

### DHCP already on the network: proxy DHCP

Where the site's DHCP server keeps handing out addresses (a corporate DHCP,
Infoblox, a router), a VLAN can be served in **proxy** mode: the rack
controller does not hand out addresses, it only answers PXE clients with where
to boot from (dnsmasq proxy DHCP), and TFTP and HTTP stay on the rack.

```
rollcall vlan add FABRIC VID -proxy-dhcp -rack RACK      # web UI: subnet, DHCP / PXE: proxy
deploy/bootstrap.sh --iface eth1 --proxy-dhcp           # the same at install time
```

- The subnet needs no ranges. Without a static range, deployed machines keep
  getting their addresses from the site's server (`dhcp` links); set a static
  range aside, outside the site's pool, to give them fixed ones.
- No `dhcp-host` lines are written for proxy subnets, and the router and DNS
  options come from the site's server.
- `rollcall doctor` expects the foreign DHCP server here, and fails if none answers.
- dnsmasq (2.91, still in 2.93) loops forever in its proxy code when a proxy
  range carries a `set:` tag: Rollcall's generated configuration leaves the tag
  out and limits the PXE services by interface instead.

Verified in the lab with a separate DHCP server on the provisioning network:
commissioning and three deployments in a row through the proxy.

## Certificate authority and TLS

The region is the installation's certificate authority, as the llmgw control
plane is: `ca_dir` (default `/etc/rollcall/ca`) holds `ca.pem` and
`ca-key.pem` (ECDSA P-256, ten years), created on first start.

* **HTTPS.** The region serves API, UI and images on `tls_listen` (default
  `:443`) with a certificate for every name and address it has plus
  `tls_names`; it is reissued when they change or a month before it ends.
  `admin_http` decides what plain HTTP may still do with the admin API:
  `allow` (default, for compatibility), `loopback` (the CLI on the region
  host) or `deny`. Nodes keep booting over HTTP: firmware and the live image
  cannot check a certificate before they have one.
* **Rack controllers** join with a code from `rollcall controller join-code`:
  one word carrying the first half of the CA's SHA-256, a one-time secret, an
  expiry and the region's address. `rollcall rack join` checks the region
  against the fingerprint, sends a CSR with the secret and gets a
  client+server certificate (`rack:<name>`, a year). From then on every rack
  call is mutual TLS and the region identifies the rack by its certificate;
  `rollcall controller rm` revokes it at once. Racks renew a month before
  expiry. The old `rack_secret` is still accepted when configured.
* **Updates.** Racks follow the region's version: the region names the
  binary (version, architecture, SHA-256) in each rack's state and serves it
  over the rack's mutual-TLS channel. The rack stages it in
  `/var/lib/rollcall/update` after checking the checksum and that it reports
  the expected version; `rollcall-rack-update.path` (root) installs it,
  restarts the rack and puts the previous binary back if the rack does not
  register again within a minute (`last-result` says what happened). Racks on
  another architecture get `rack_binaries/rollcall-linux-<arch>`;
  `"auto_update": false` in `rack.json` opts out. `rollcall controllers`
  shows each rack's version.
* **Admin certificates.** Instead of the admin token, a person or laptop can
  hold a client certificate: `rollcall admin-cert request NAME` (with the
  token once) makes a key locally, sends only a CSR and gets
  `admin:NAME` back; the CLI and console use it from `client.json`, and the
  `.p12` it writes goes into the browser's keychain, after which the web UI
  signs in by itself. `rollcall admin-cert list | revoke NAME` (revocation
  takes effect within 30 seconds). Certificates are checked against the CA,
  the client-auth usage and the serials the region recorded.
* **Clients.** On the region host the CLI reads the CA from `ca_dir`;
  elsewhere `ROLLCALL_CA=ca.pem`, or the console shows the CA fingerprint on
  first connect and keeps the CA in `~/.config/rollcall/client.json` once you
  confirm it (`rollcall ca show` prints it on the region). `/ca.pem` serves
  the certificate; the web login links to it.
* **Machines** trust it: deployments write the CA into the trust anchors of
  both families (`/usr/local/share/ca-certificates`, `/etc/pki/ca-trust/source/anchors`)
  and rebuild the store at every boot (`update-ca-certificates` or
  `update-ca-trust`; cloud-init's `ca_certs` fails on RHEL-family systems);
  stateless overlays get it as `.CA`. `rollcall ca issue NAME
  -host h1,h2 [-client]` issues leaves for anything else (Slurm REST, a
  registry, a web front end).
* **Delegation.** The root may sign one level of intermediate CAs, each
  limited by name constraints to the domains and address ranges it serves:
  `rollcall ca intermediate caddy -permit int.example.com,10.213.0.0/16
  -csr caddy.csr` signs a CSR (the key stays on the host that made it). An ACME
  server under that intermediate (Caddy's `acme_server`, step-ca) then issues
  short-lived certificates to nodes with certbot or acme.sh; everything that
  trusts the Rollcall CA trusts them, and nothing outside the permitted names
  can be issued. Roots created before this (path length 0) cannot delegate:
  `rollcall ca rotate -yes` replaces the root, keeping the old one as a
  backup; rack controllers then join again and clients trust the new
  fingerprint. `deploy/tags/acme-cert.yaml` is a tag that fetches a node
  certificate at first boot and renews it.

## Power drivers

Every machine's BMC record names a driver; commands run on the rack controller
that reaches the BMC network.

| Driver | For |
|---|---|
| `ipmi` | IPMI 2.0 over LAN (ipmitool lanplus), any vendor; settings `cipher` (17 on newer BMCs), `privilege`, `interface` |
| `redfish` | DMTF Redfish, any vendor: system discovered, ResetType chosen from what the BMC allows, boot override once (with If-Match where the BMC wants it) |
| `idrac` | Dell iDRAC 8/9 (Redfish, `System.Embedded.1`) |
| `ilo` | HPE iLO 4 (2.30+), 5, 6 (Redfish; no PowerCycle, so ForceRestart; PushPowerButton for soft off on older firmware) |
| `supermicro`, `xcc`, `openbmc` | Supermicro, Lenovo XClarity, OpenBMC (Redfish) |
| `proxmox` | Proxmox VE VMs: API token (user = token id, password = secret), settings `node`, `vmid` |
| `manual` | no power control; actions say so |

After commissioning, a BMC still on IPMI is asked whether it speaks Redfish
with the same account; if it does, the driver for its vendor takes over
(event `bmc driver`). An operator's choice is never overridden.

```sh
rollcall bmc drivers
rollcall bmc m-1234 -driver idrac -address 10.0.5.21 -user root -password - -verify < pw
rollcall bmc m-1234 -driver redfish -set system=/redfish/v1/Systems/1 -set verify_tls=true
rollcall bmc m-1234 -probe            # detect Redfish again
```

The lab has `lab/pve-vbmc/pve_redfish.py`, a Redfish service per Proxmox VM
with iDRAC, iLO and generic profiles, next to the IPMI emulator.

## IPv6

Subnets can be IPv6 (`rollcall subnet add fd00:50::/64 ...`). IPAM hands out
addresses from static ranges (walked from their start), deployments get them
in `network-config` with a separate default route per family, and the default
layout links every subnet of the boot NIC's VLAN, so an IPv6 prefix next to
the IPv4 one makes machines dual-stack. Interfaces with planned addresses
ignore router advertisements. dnsmasq serves router advertisements and
DHCPv6 on IPv6 subnets with dhcp on; the rack offers itself as default router
only when it is the subnet's gateway. Host files carry AAAA records, rack
controllers report their IPv6 addresses, the region certificate covers them,
and region, racks and clients work over IPv6 (`https://[fd00::1]`).

PXE stays IPv4: syslinux cannot netboot over IPv6, so the provisioning VLAN
needs an IPv4 subnet for commissioning and deployment.

Verified in the lab: Debian 13 (with a bond and VLAN), Rocky 9 and AlmaLinux 9
dual-stack on two segments, AAAA lookups through dnsmasq, the admin CLI over
`https://[fdc3:...]`.

## External IPAM (NetBox; Infoblox-style)

Rollcall works on its own, but at a site with an established network plan the
address authority is somewhere else. With `ipam` in `config.json` every
address Rollcall hands out is registered there first:

```json
"ipam": {
  "provider": "netbox",
  "url": "https://netbox.int.example.com",
  "token_file": "/etc/rollcall/netbox.token",
  "ca_file": "/etc/rollcall/netbox-ca.pem",
  "dns_zone": "int.example.com",
  "sync_interval": "5m"
}
```

- **Allocation asks the authority.** An address that exists in NetBox and is
  not Rollcall's is skipped and the next candidate is tried, so Rollcall never
  collides with a printer, a VIP or another team's reservation. If NetBox is
  unreachable, allocation fails instead of guessing.
- **Ownership is explicit.** Rollcall's records carry the tag `rollcall` and the
  machine ID in the custom field `rollcall_machine` (create both in NetBox;
  on Infoblox these are an extensible attribute). Records without both are
  never changed or deleted, only reported.
- **Releases reconcile.** Releasing, unassigning or deleting a machine frees
  addresses locally and triggers a reconcile, which also runs every
  `sync_interval`: missing records are created, names and owners fixed, and
  Rollcall-owned records that are no longer in use deleted.
- **DNS follows the record.** `dns_name` is `<host>.<zone>` for a machine's
  provisioning/mgmt address and `<host>-<iface>.<zone>` elsewhere (a subnet's
  `domain` replaces the zone when it lies inside it; `pxe.local` stays local). A NetBox-to-DNS sync (PowerDNS here, a host
  record on Infoblox) publishes it; Rollcall writes one object per address.
- **Prefixes are delegated in NetBox.** Tag a prefix `rollcall`;
  `rollcall ipam prefixes` lists them and `rollcall ipam import` creates the
  missing subnets (VLAN and rack are then set as usual).

```
rollcall ipam                 # provider, last reconcile
rollcall ipam sync -dry       # what would change
rollcall ipam sync
rollcall ipam prefixes | import
```

The provider is an interface (reserve, owned records, update, delete,
delegated prefixes); the NetBox driver is the first. An Infoblox WAPI driver
maps them to `record:host`/`fixedaddress` with extensible attributes.

## DNS provider (PowerDNS, RFC 2136)

Without an IPAM authority whose DNS sync publishes names, Rollcall can write
its machines' A/AAAA records into the site's DNS itself. Names follow the
IPAM rule above (`<host>.<zone>`, `<host>-<iface>.<zone>`).

```json
"dns": {
  "provider": "powerdns",
  "zone": "int.example.com",
  "url": "http://dns.int.example.com:8081",
  "api_key_file": "/etc/rollcall/pdns.key"
}
```

```json
"dns": {
  "provider": "rfc2136",
  "zone": "int.example.com",
  "server": "ns1.int.example.com:53",
  "tsig_name": "rollcall-key",
  "tsig_algorithm": "hmac-sha256",
  "tsig_secret_file": "/etc/rollcall/tsig.secret"
}
```

- **powerdns** uses the HTTP API. Rollcall's RRsets carry a comment with
  account `rollcall`; RRsets without it are never changed or deleted.
- **rfc2136** is a TSIG-signed dynamic update, for BIND, Knot, PowerDNS,
  Windows DNS and Infoblox. Next to each name Rollcall writes a marker
  `_rollcall.<name> TXT "heritage=rollcall"`. A change to an existing record
  needs the marker as a prerequisite; a new record needs the name to be
  free. The server checks both atomically, so a record someone else owns is
  never overwritten. What Rollcall wrote is kept in the database, and each
  sync queries those records. A record changed behind Rollcall's back is
  repaired when the marker is still there. When someone else's record has
  taken the name, it is reported as a conflict.
  For BIND, grant the key the zone: `update-policy { grant rollcall-key zonesub ANY; };`.

A sync runs on every allocation and release and every `sync_interval`
(default 5m), with `ttl` defaulting to 300.

```
rollcall dns                  # provider, last sync, conflicts
rollcall dns sync -dry        # what would change
rollcall dns sync
```

Conflicts and errors appear there, in the log and in `rollcall doctor`.

## Lab

`lab/` documents a Proxmox SDN test setup: an isolated vnet, a PXE server VM
running dnsmasq + Rollcall, two diskless VMs, and `pve-vbmc`, a tiny
IPMI-over-LAN BMC emulator that drives VMs through the Proxmox API so that
`ipmitool power on` and `chassis bootdev pxe` work against virtual machines.

## Status

Commissioning, stateless nodes and disk deployment work end to end in the lab
(virtual machines with 30 GB disks, SeaBIOS, out-of-band BMC hints). The
in-band BMC path and the UEFI boot-entry step are implemented but have not yet
been exercised on real hardware.