rollcall/CHANGELOG.md

Changelog

All notable changes to Rollcall. The format follows Keep a Changelog; the release workflow uses the section of the pushed tag as the release notes and refuses to publish a tag that has no section here.

[Unreleased]

Added

• Rooms in 3D (three.js shipped in the binary, loaded on first use): racks at their places and height, servers drawn from their model's front (drive bays, ears, vents; no photos or logos) with the state on the ear, multi-node sleds side by side, backs with power supplies and the network cards' ports. Drag to turn, double click to go to a rack; selection is shared with the floor plan.
• Cabling from LLDP: GET /api/v1/cabling (each machine's ports and the switch port they lead to), switches placed from NetBox or by hand (POST/DELETE /api/v1/switches), cables drawn in the 3D room from port to switch port, coloured per switch, and listed per rack in the side panel. Where LLDP sees nothing, cables recorded in NetBox (through patch panels) or entered by hand (POST/DELETE /api/v1/cables); the BMC's port to the out-of-band switch that way, OOB switches marked; NetBox cables that disagree with LLDP are shown.
• NetBox's web UI through Rollcall at its BASE_PATH (admins only), and links to NetBox from the web UI (sidebar, sites, rooms, racks, switches, machine devices); ipam.ui_url for direct links. With ipam.ui_login_secret_file the admin certificate signs in to NetBox too (NetBox's remote user header, guarded by a shared secret the web server in front of NetBox checks).
• Sites page: sites on an OpenStreetMap map (Leaflet shipped in the binary), a site's rooms, and a room's floor plan with racks on 600 mm tiles coloured by their machines; a rack opens its front. Racks are placed by dragging and rotated in an edit mode; unplaced ones are laid out automatically. With NetBox, sites (with coordinates), locations and racks are read from it; otherwise, or without coordinates there, sites are placed on the map. /api/v1/sites, /api/v1/rooms, POST /api/v1/racks/positions. The web UI's CSP allows OSM tile images.
• Server heights without typing them in: SMBIOS chassis height from commissioning, else a model table checked against vendor documentation (Dell, HPE, Lenovo, xFusion, Fujitsu, Inspur/KAYTUS, GIGABYTE; not Supermicro, whose model numbers do not carry it), else rack_units per model in config.json. Multi-node sleds share their chassis. The rack elevation spans servers over their height, groups sleds, marks unknown heights and warns about overlaps; NetBox device types get the height (0 for sleds, placed without a position). The machine API has size.
• Network page for large sites: a problems strip (offline rack controllers and the VLANs left without DHCP/PXE, provisioning VLANs without it, subnets over 80%, IPAM errors), filters by fabric, purpose and rack controller, a search that shows a machine's path (interface, address, VLAN, rack controller, region), and a VLAN x rack-controller matrix, the default once a site is large. The topology groups machines more coarsely when boxes would not fit and scrolls instead of shrinking; the overview shows one row per fabric on a large site. Filters can be put on the URL (?fabric=…&view=matrix&q=…) to share a view.
• Root rotation without re-joining (rollcall ca rotate prepare | switch | retire, status without an argument). The trust bundle (all current roots, /ca-bundle.pem) reaches rack controllers signed by the active root and is installed by their root unit only when that signature verifies; racks renew client and server certificates from the new root by themselves; switch and retire wait for every rack. Deployed machines get one file per root and a daily rollcall-ca-refresh timer that fetches the bundle from their rack over HTTPS. Upgrading: rerun deploy/install-rack.sh on rack hosts (the update unit installs bundles now).

Changed

• rollcall ca rotate -yes (replace the root, racks join again) is gone; the three steps replace it.

Fixed

• A rack typed in from a machine's location and the same rack from NetBox in another case showed twice; the layout sync folds it into NetBox's spelling (place, comment and machines follow).
• NetBox paging under a BASE_PATH doubled the path on the next page.

[0.3.0] - 2026-10-30

Third release: working with a site's existing network services (DNS provider, fixed leases on Kea and OpenWrt, NetBox device sync, Infoblox), Redfish across vendors with firmware updates, UEFI Secure Boot, agents over HTTPS, and a security review of the whole code base. Upgrading: rerun deploy/install.sh on the region and deploy/install-rack.sh on rack hosts.

Added

• Proxy DHCP: a VLAN in proxy mode (rollcall vlan add ... -proxy-dhcp, web UI, bootstrap.sh --proxy-dhcp) leaves addresses to the site's DHCP server and only answers PXE; machines keep DHCP unless a static range is set aside; doctor and discover know the mode.
• DNS provider: dns in config.json writes machines' A/AAAA records into the site's DNS, through the PowerDNS API (ownership by RRset comment) or RFC 2136 dynamic update with TSIG (ownership by a _rollcall.<name> TXT marker and update prerequisites, so foreign records are never overwritten; drift is detected by querying). rollcall dns, rollcall dns sync [-dry], a doctor check.
• NetBox device sync (ipam.device_sync): machines become dcim devices with serial, device type, role, status, interfaces with MAC objects, assigned addresses and primary IP. Site, location and rack are matched by name from the machine's location and never created; ownership by tag and rollcall_machine as for addresses, foreign devices of the same name are conflicts.
• Infoblox WAPI driver for the IPAM authority (ipam.provider: infoblox): one host record per address (DNS-published in dns_view, or IPAM-only), ownership by the rollcall_machine extensible attribute (kept with extattrs+), leases, fixed addresses and ranges count as taken, delegated networks by the rollcall attribute. Not tested against a real grid yet.
• The release workflow builds the tag's commissioning image on the PXE server and publishes it as commissioning-* assets.
• Lab NetBox (lab/netbox/install.sh): a container on the lab segment for the NetBox drivers.
• Fixed leases on the site's DHCP server for proxy-mode VLANs (dhcp_reservations), for Kea with host_cmds (Kea 3.0+) and OpenWrt (rpcd). Rollcall's own leases are reconciled; foreign ones are reported. rollcall reservations.
• Firmware updates over Redfish. A package repository on the region (rollcall firmware import from a file or URL, served to BMCs through rack controllers) and each machine's firmware inventory.
  • Update jobs: hand-over, task watch through BMC restarts, optional power cycle, version check against the inventory. They resume after a region restart.
  • Vendor plans from documentation or vendor tools: Dell, HPE (.fwpkg unpacked), Lenovo, Supermicro, GIGABYTE/AMI/ASRock Rack, OpenBMC, xFusion, KAYTUS, Fujitsu iRMC. Cisco CIMC is refused (SCP only).
  • fw_method, fw_params and firmware_base on the BMC record.
  • CLI rollcall firmware, a Firmware tab and repository in the web UI.
• Redfish vendor profiles: one standard driver plus per-vendor behaviour from public documentation, for Dell iDRAC, HPE iLO, Lenovo XCC, Supermicro, xFusion/Huawei iBMC, KAYTUS/Inspur, Fujitsu iRMC, Cisco CIMC, GIGABYTE, ASRock Rack, OpenBMC and AMI MegaRAC.
  • Boot mode kept, omitted or tried per vendor.
  • If-Match with 412/428 handling; future-state (/SD) and @Redfish.Settings targets.
  • Busy retries and task following.
  • Chassis reset fallback and an emulated power cycle.
  • Session login (auth=session).
  • Probe results (reset types, boot targets, boot mode, profile) are kept on the BMC and shown in the UI, and the UI's driver list comes from the server.

Security

A review of the whole code base; the fixes below. Upgrading: rerun deploy/install.sh on the region and deploy/install-rack.sh on rack hosts (new dnsmasq layout, signed rack updates).

• Root SSH keys in published images. agent/build-image.sh took the build user's authorized_keys unless told otherwise, so the images published with v0.2.0 and v0.2.1 let three lab keys (the maintainer's laptop, the CI deploy key, the lab hypervisor) log in as root to machines while they commission. Keys are now opt-in (AUTHKEYS=, the lab deploy sets it), and the release workflow builds the image of the tag itself and refuses one with authorized_keys.
• Region file read (critical). A rack controller could read any file on the region (CA key, config) through /api/v1/rack/binary?arch=../... arch is now a known architecture.
• Path traversal. In /api/v1/render/... (admin) as well.
• Machine-reported MACs. A MAC could be a path, letting a PXE-booting host make the rack write outside the TFTP directory. Only real hardware addresses are accepted. Rack controllers write only 01-xx-.. file names.
• Delegated intermediates. An ACME intermediate could mint admin or rack certificates (name constraints do not cover the CN), and IP addresses when only a domain was permitted.
  • Admin and rack identities must now be signed by the root itself.
  • Intermediates carry a ServerAuth-only EKU, and the unconstrained name type is closed.
• Rack certificates. These were ServerAuth certificates for any address the rack reported, so one rack could pose as the region.
  • They are now client-only with no names.
  • Racks accept only the region's own certificate.
  • rack join checks the returned CA against the fingerprint in the code.
  • An unbound join code cannot take over an existing rack's name.
  • Job results are accepted only from the rack that took the job.
• Rack self-update. The root unit ran a binary the unprivileged rack user staged.
  • The region now signs each binary with its CA key.
  • The root unit verifies the signature against the CA from rack join, in a root-owned directory, without following links.
• dnsmasq as root. dnsmasq (root) read configuration the rollcall user writes, and the reload script copied files through links in rollcall's directory.
  • It now reads a checked copy in /var/lib/rollcall-dnsmasq.
  • The copy may hold only the options Rollcall generates: nothing that runs a program or includes a file.
• Machine record takeover. Any host on the provisioning network could overwrite a deployed machine's record (and BMC) by reporting its UUID.
  • Known machines are re-commissioned only when ready, recommission or failed.
  • A MAC another machine owns keeps its boot file.
  • The rack controller of a ready machine is not changed by whoever reports it.
• Injection into dnsmasq, cloud-init, PXE menus, udev rules and DNS updates. Subnet domain and gateway, host names, NIC names, reservation names and machine-reported identity strings are now validated or reduced to one line. Machine IDs are letters and digits only.
• Web UI.
  • Escaping is fixed for machine-reported inventory fields (stored XSS).
  • A Content-Security-Policy, frame-ancestors 'none', nosniff and no-referrer are set.
  • Cross-origin state-changing requests are refused (http.CrossOriginProtection), so a page elsewhere cannot use a browser's admin certificate.
  • ?token= is no longer logged.
• Deploy agent.
  • The image URL no longer goes into a shell command; the checksum is taken over the bytes written.
  • Seed files cannot leave the target root.
  • The BMC password is not kept in the fallback inventory file.
• Redfish. Requests stay on the BMC and under /redfish/: paths the BMC returns cannot send the credentials elsewhere, and scheme:// addresses are reduced to scheme and host.
• Machine tokens. Job and node tokens are readable from the TFTP pxelinux files. They are now accepted only from the machine's own address (its static lease); through a rack, the rack's X-Forwarded-For entry counts. A machine without a fixed address binds a job to its first user. agent_source_check: false turns this off.
• Agents over HTTPS. Rack controllers get a server certificate for their own host and serve the agents' endpoints and images on 443. Agents accept the CA by the fingerprint on the PXE command line and do not fall back to HTTP. Old agents keep working over HTTP. The commissioning image has to be rebuilt to get the new agents.
  • The stateless overlay client works the same way (rebuild stateless images).
• UEFI and Secure Boot. UEFI machines PXE-boot through Debian's signed shim and GRUB, so machines with Secure Boot on can boot at all; before, the firmware refused the unsigned syslinux.efi.
  • GRUB menus are made from the pxelinux ones.
  • The local disk is booted by chainloading the installed OS's signed loader.
  • Secure Boot verifies the kernel; the initrd and squashfs are not covered.
• BMC certificates. HTTPS BMCs (Redfish, Proxmox) are pinned on first use. The certificate is read in a handshake that carries no credentials, and a changed certificate gets none (tls_fingerprint; cleared on an address change or a BMC firmware update by Rollcall).
• BMC hints. bmc-hint no longer returns the password. The region attaches the configured credentials itself and checks the BMC from the rack controller.
• Rack certificate renewal. Renewal could not overwrite the root-owned files from rack join. Renewed certificates now go atomically to /var/lib/rollcall/rack-cert, the previous serial is accepted for 7 days after a renewal, and rollcall rack renew renews on demand.
• Defaults and limits.
  • admin_http defaults to loopback.
  • Request bodies are capped (commissioning 4 MB, all 16 MB), with read and idle timeouts.
  • bootstrap.sh and publish-image.sh keep the database password and API tokens off command lines.

Fixed

• IPAM: a reconcile running during an allocation could delete the record just reserved in NetBox.
• IPAM and DNS sync: one failing record no longer stops the rest of the pass, deletes included. Reservations are stored in one spelling.
• The job watchdog could fail a job that had just started.
• Release without -disk wiped the smallest disk, not the one the OS was deployed to. Automatic disk choice skips SAN LUNs (iSCSI, FC, SRP).
• RFC 2136 lookups follow the request context. The marker is not removed when the database cannot say whether other records need it.
• Smaller fixes:
  • seq in templates is capped.
  • Layout selector ib:-1 no longer panics.
  • Redfish detection no longer runs on a machine that cannot be read.
• dnsmasq busy loop in proxy mode (a dnsmasq bug with tagged proxy ranges, found with gdb): proxy ranges are untagged, PXE services limited by interface; dhcp-boot/dhcp-match only for served subnets, no dhcp-host lines for proxied ones.

[0.2.1] - 2026-10-28

Fixed

• bootstrap.sh --image-url with a release URL took the release's SHA256SUMS (the binaries') instead of commissioning-SHA256SUMS; it now prefers the image's and says when a private repository needs IMAGE_TOKEN.

[0.2.0] - 2026-10-28

Second release: install from scratch, network model and region/rack split, external IPAM (NetBox), a certificate authority with delegation, many power drivers, IPv6, and a web UI and console for fleets.

Added

• Machine tags (MAAS-style): manual tags per machine, auto tags from match rules over the inventory ({{ gt .Node.GPUs 0 }}), re-evaluated at commissioning and when a tag changes.
• Tags carry a cloud-init fragment merged into deployment user-data (deploy/tags/gpu-nvidia.yaml installs the NVIDIA driver on GPU nodes) and extra overlays for stateless nodes.
• rollcall tag, rollcall tags, rollcall list -tag, deploy tag:NAME / power tag:NAME fan-out; tags column, editor and definitions panel in the web UI; ?tag= filter on the API.
• Web UI redesigned for fleets: sidebar navigation (Overview, Machines, Jobs, Tags, Profiles, OS images), state/tag filter chips, search, sorting and paging, multi-select with bulk power/deploy/assign/tag/re-commission/release, machine detail with tabs, dialog-driven actions, fleet-wide /api/v1/events and /api/v1/jobs, ?token= sign-in for bookmarks.
• Physical location per machine (building, room, rack, unit, asset tag): rollcall location, CSV import matched by id/serial/MAC/node name with dry run, rack definitions with height, list -rack/-room/-building, location column and chips in the UI, Racks page with elevation views, bulk "Set location…", .Node.Rack etc. in templates and tag rules.
• Terminal console: a bare rollcall in a terminal opens a full-screen UI in the llmgw style (Bubble Tea): overview, machines with filter/selection/bulk actions, machine detail, racks with elevation, jobs, events, tags, profiles, OS images; connect screen that saves ~/.config/rollcall/client.json, which the CLI also reads; rollcall tui -dump renders one screen. First Go tests (console rendering and keys).
• Network model after MAAS: fabrics, VLANs, subnets, IP ranges (dynamic/reserved/static) and IPAM; addresses already in use are adopted. rollcall fabric|vlan|subnet|range|ip.
• Interface layouts (physical, bond, VLAN; NICs by selector) per machine or per tag; deployment writes cloud-init network-config and udev name rules into the image; release frees addresses. rollcall layout, rollcall net show|set.
• Region/rack split: rollcall rack (dnsmasq state, per-MAC PXE files, BMC commands, image cache, node API proxy) long-polls the region; the region runs an embedded rack controller. deploy/install-rack.sh, rollcall controllers, rollcall dhcp.
• dnsmasq configuration generated from the model, restarted only when it changes and rolled back if dnsmasq does not come back.
• Web UI: Network page (rack controllers, subnets with ranges and usage, layouts), network tab per machine, layout action; console: Network page, network section, layout action.
• Certificate authority on the region (llmgw-style): HTTPS listener with an auto-renewed region certificate, admin_http policy, /ca.pem; rack controllers enroll with one-time join codes (rollcall controller join-code, rollcall rack join) and use mutual TLS, renewing their certificate themselves; removal revokes; deployed machines and overlays trust the CA; rollcall ca show|issue; the console trusts a CA on first connect after showing its fingerprint.
• Power drivers: ipmi, redfish, idrac, ilo, supermicro, xcc, openbmc, proxmox, manual, run on the rack controller; Redfish auto-detection after commissioning; rollcall bmc, BMC editing in the web UI and console. Lab Redfish emulator with iDRAC/iLO/generic profiles.
• IPv6: subnets, IPAM, dual-stack default layout, per-family default routes, dnsmasq RA/DHCPv6, AAAA host records, IPv6 listen addresses and URLs, IPv6 in the commissioning inventory.
• External IPAM authority (NetBox driver): allocation registers each address in NetBox first and skips addresses others hold there; ownership by tag + rollcall_machine custom field; releases and a periodic reconcile create, fix and delete only Rollcall-owned records; dns_name per address for DNS sync; delegated prefixes (rollcall ipam prefixes|import); rollcall ipam [sync -dry], /api/v1/ipam.
• Web UI: network topology drawing (region, external IPAM, rack controllers, one bus per VLAN with its subnets, machines wired to every VLAN they have an address on; large groups collapse) on the Network page and, compact, on the Overview. Overview redesigned: fleet state bar, per-rack counts, attention list incl. IPAM sync, capacity and free pool, OS, hardware, BMC drivers, locations, tags, recent deployments.
• CA delegation: new roots allow one level of intermediates; rollcall ca intermediate NAME -permit DOMAIN,CIDR [-csr FILE] signs a name-constrained intermediate (for an ACME server such as Caddy's acme_server); rollcall ca rotate -yes replaces a root that cannot delegate; tag fragment deploy/tags/acme-cert.yaml fetches and renews a node certificate with certbot.
• Rack controllers update themselves to the region's version: checksum- and version-checked download over mutual TLS, installed by a root path unit with automatic rollback; rack_binaries for other architectures, auto_update to opt out; versions in rollcall controllers and the web UI.
• Addresses on secondary subnets with a gateway get their own routing table and a source rule in network-config, so replies from a storage or tenant VLAN leave through that VLAN's gateway.
• With admin_http other than allow, browsers that open the web UI over HTTP are redirected to HTTPS (nodes keep HTTP for images and the agent).
• Admin client certificates: rollcall admin-cert request NAME (key stays on the client, CSR only), list, revoke; the region accepts them as admin over HTTPS; CLI/console use them from client.json, the web UI works without a token when the browser presents one (.p12 written).
• deploy/bootstrap.sh: a region on a fresh Debian 13 host. Day 0 needs nothing about the network: packages, PostgreSQL, TFTP tree, config, services and a verified commissioning image, the region reachable on its management address with no DHCP or subnets. Run again with --iface once the provisioning network is known: server address updated, subnet with ranges, DHCP, optional NAT.
• rollcall net discover: per interface link, speed, addresses and implied subnets, switch/port/VLAN from LLDP, DHCP servers that answer, neighbours, and what to do with it.
• commission_append follows server_url when left empty.
• rollcall doctor: checks configuration, database, listeners, certificates, TFTP files, image checksums, dnsmasq, DNS as machines see it, clock, foreign DHCP servers on the provisioning network (DHCPDISCOVER probe), rack controllers, OS images and IPAM, with what to do.
• rollcall version; CD-built binary carries git describe.
• Release notes from CHANGELOG.md.

Fixed

• RHEL-family deployments (Rocky, Alma, CentOS) did not trust the Rollcall CA: cloud-init's ca_certs ran update-ca-certificates there and failed. The CA now goes into the image's trust anchors and a bootcmd rebuilds the store with the distribution's tool.
• A deploy whose power command was lost (rack or region restarting) stayed pending forever and the event looked like a success: power errors are now part of the event (pxe boot failed), and a job the machine has not started within job_start_timeout (default 30m, room for servers that take 4-5 minutes to POST) fails with a hint.
• CentOS Stream 9 image URL (the 502 was a missing file name).
• Rack controllers' dnsmasq had no upstream DNS on Debian (it looks for a resolvconf file that is usually missing) and refused every name outside the cluster domain; the generated configuration now forwards to systemd-resolved's stub (keeping split DNS) or the host's resolv.conf servers.
• Virtual BMC addresses are configured in netplan on the lab PXE server (they were lost on a network reconfigure, which made every BMC call fail).
• Commissioning console no longer shows a failing autologin (noautologin).

[0.1.0] - 2026-10-27

First tagged release.

Added

• Commissioning: Debian 13 live image with firmware, IB and NVMe tooling; hardware inventory (DMI, DIMMs, CPUs, disks, NVMe, NICs, InfiniBand, GPUs, PCI, LLDP, TPM); in-band BMC account creation with out-of-band hints for VMs; secrets sealed with AES-256-GCM; ipmitool server-side.
• Single static Go binary: HTTP API, embedded web UI, CLI, PostgreSQL schema.
• Stateless (Warewulf-style) nodes: profiles, node names and IP pools, rendered overlays, dnsmasq export, Slurm compute image and controller sync.
• OS deployment: image catalog with nightly upstream sync and vendor checksum verification (Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9, CentOS Stream 9), streamed zstd | dd writes with sha256 and live progress, cloud-init NoCloud seed, SELinux relabel, UEFI boot entry, phone-home, release/wipe jobs.
• Gitea Actions: CI, deploy to the PXE server, website publish, release on tags.
• Lab: Proxmox SDN subnet, pve-vbmc IPMI emulator for VMs.