rollcall/CHANGELOG.md

# Changelog

All notable changes to Rollcall. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
the release workflow uses the section of the pushed tag as the release notes and refuses to
publish a tag that has no section here.

## [Unreleased]

### Added
- Rooms in 3D (three.js shipped in the binary, loaded on first use): racks at their places and
  height, servers drawn from their model's front (drive bays, ears, vents; no photos or logos) with
  the state on the ear, multi-node sleds side by side, backs with power supplies and the network
  cards' ports. Drag to turn, double click to go to a rack; selection is shared with the floor plan.
- Cabling from LLDP: `GET /api/v1/cabling` (each machine's ports and the switch port they lead to),
  switches placed from NetBox or by hand (`POST`/`DELETE /api/v1/switches`), cables drawn in the 3D
  room from port to switch port, coloured per switch, and listed per rack in the side panel.
  Where LLDP sees nothing, cables recorded in NetBox (through patch panels) or entered by hand
  (`POST`/`DELETE /api/v1/cables`); the BMC's port to the out-of-band switch that way, OOB
  switches marked; NetBox cables that disagree with LLDP are shown.
- NetBox's web UI through Rollcall at its `BASE_PATH` (admins only), and links to NetBox from the
  web UI (sidebar, sites, rooms, racks, switches, machine devices); `ipam.ui_url` for direct links.
  With `ipam.ui_login_secret_file` the admin certificate signs in to NetBox too (NetBox's remote
  user header, guarded by a shared secret the web server in front of NetBox checks).
- Sites page: sites on an OpenStreetMap map (Leaflet shipped in the binary), a site's rooms, and a
  room's floor plan with racks on 600 mm tiles coloured by their machines; a rack opens its front.
  Racks are placed by dragging and rotated in an edit mode; unplaced ones are laid out
  automatically. With NetBox, sites (with coordinates), locations and racks are read from it;
  otherwise, or without coordinates there, sites are placed on the map. `/api/v1/sites`,
  `/api/v1/rooms`, `POST /api/v1/racks/positions`. The web UI's CSP allows OSM tile images.
- Server heights without typing them in: SMBIOS chassis height from commissioning, else a model
  table checked against vendor documentation (Dell, HPE, Lenovo, xFusion, Fujitsu, Inspur/KAYTUS,
  GIGABYTE; not Supermicro, whose model numbers do not carry it), else `rack_units` per model in
  `config.json`. Multi-node sleds share their chassis. The rack elevation spans servers over their
  height, groups sleds, marks unknown heights and warns about overlaps; NetBox device types get the
  height (0 for sleds, placed without a position). The machine API has `size`.
- Network page for large sites: a problems strip (offline rack controllers and the VLANs left
  without DHCP/PXE, provisioning VLANs without it, subnets over 80%, IPAM errors), filters by fabric,
  purpose and rack controller, a search that shows a machine's path (interface, address, VLAN,
  rack controller, region), and a VLAN x rack-controller matrix, the default once a site is large.
  The topology groups machines more coarsely when boxes would not fit and scrolls instead of
  shrinking; the overview shows one row per fabric on a large site. Filters can be put on the URL
  (`?fabric=…&view=matrix&q=…`) to share a view.
- Root rotation without re-joining (`rollcall ca rotate prepare | switch | retire`, status without
  an argument). The trust bundle (all current roots, `/ca-bundle.pem`) reaches rack controllers
  signed by the active root and is installed by their root unit only when that signature verifies;
  racks renew client and server certificates from the new root by themselves; `switch` and
  `retire` wait for every rack. Deployed machines get one file per root and a daily
  `rollcall-ca-refresh` timer that fetches the bundle from their rack over HTTPS.
  Upgrading: rerun `deploy/install-rack.sh` on rack hosts (the update unit installs bundles now).

### Changed
- `rollcall ca rotate -yes` (replace the root, racks join again) is gone; the three steps replace it.

### Fixed
- A rack typed in from a machine's location and the same rack from NetBox in another case showed
  twice; the layout sync folds it into NetBox's spelling (place, comment and machines follow).
- NetBox paging under a `BASE_PATH` doubled the path on the next page.

## [0.3.0] - 2026-10-30

Third release: working with a site's existing network services (DNS provider, fixed leases on
Kea and OpenWrt, NetBox device sync, Infoblox), Redfish across vendors with firmware updates,
UEFI Secure Boot, agents over HTTPS, and a security review of the whole code base. Upgrading:
rerun `deploy/install.sh` on the region and `deploy/install-rack.sh` on rack hosts.

### Added
- Proxy DHCP: a VLAN in `proxy` mode (`rollcall vlan add ... -proxy-dhcp`, web UI, `bootstrap.sh
  --proxy-dhcp`) leaves addresses to the site's DHCP server and only answers PXE; machines keep DHCP
  unless a static range is set aside; doctor and discover know the mode.
- DNS provider: `dns` in `config.json` writes machines' A/AAAA records into the site's DNS, through
  the PowerDNS API (ownership by RRset comment) or RFC 2136 dynamic update with TSIG (ownership by a
  `_rollcall.<name>` TXT marker and update prerequisites, so foreign records are never overwritten;
  drift is detected by querying). `rollcall dns`, `rollcall dns sync [-dry]`, a doctor check.
- NetBox device sync (`ipam.device_sync`): machines become dcim devices with serial, device type,
  role, status, interfaces with MAC objects, assigned addresses and primary IP. Site, location and
  rack are matched by name from the machine's location and never created; ownership by tag and
  `rollcall_machine` as for addresses, foreign devices of the same name are conflicts.
- Infoblox WAPI driver for the IPAM authority (`ipam.provider: infoblox`): one host record per
  address (DNS-published in `dns_view`, or IPAM-only), ownership by the `rollcall_machine` extensible
  attribute (kept with `extattrs+`), leases, fixed addresses and ranges count as taken, delegated
  networks by the `rollcall` attribute. Not tested against a real grid yet.
- The release workflow builds the tag's commissioning image on the PXE server and publishes it as
  `commissioning-*` assets.
- Lab NetBox (`lab/netbox/install.sh`): a container on the lab segment for the NetBox drivers.
- Fixed leases on the site's DHCP server for proxy-mode VLANs (`dhcp_reservations`), for Kea with
  `host_cmds` (Kea 3.0+) and OpenWrt (rpcd). Rollcall's own leases are reconciled; foreign ones are
  reported. `rollcall reservations`.
- Firmware updates over Redfish. A package repository on the region (`rollcall firmware import`
  from a file or URL, served to BMCs through rack controllers) and each machine's firmware inventory.
  - Update jobs: hand-over, task watch through BMC restarts, optional power cycle, version check
    against the inventory. They resume after a region restart.
  - Vendor plans from documentation or vendor tools: Dell, HPE (`.fwpkg` unpacked), Lenovo,
    Supermicro, GIGABYTE/AMI/ASRock Rack, OpenBMC, xFusion, KAYTUS, Fujitsu iRMC. Cisco CIMC is
    refused (SCP only).
  - `fw_method`, `fw_params` and `firmware_base` on the BMC record.
  - CLI `rollcall firmware`, a Firmware tab and repository in the web UI.
- Redfish vendor profiles: one standard driver plus per-vendor behaviour from public documentation,
  for Dell iDRAC, HPE iLO, Lenovo XCC, Supermicro, xFusion/Huawei iBMC, KAYTUS/Inspur, Fujitsu iRMC,
  Cisco CIMC, GIGABYTE, ASRock Rack, OpenBMC and AMI MegaRAC.
  - Boot mode kept, omitted or tried per vendor.
  - If-Match with 412/428 handling; future-state (`/SD`) and `@Redfish.Settings` targets.
  - Busy retries and task following.
  - Chassis reset fallback and an emulated power cycle.
  - Session login (`auth=session`).
  - Probe results (reset types, boot targets, boot mode, profile) are kept on the BMC and shown in
    the UI, and the UI's driver list comes from the server.

### Security
A review of the whole code base; the fixes below. Upgrading: rerun `deploy/install.sh` on the
region and `deploy/install-rack.sh` on rack hosts (new dnsmasq layout, signed rack updates).
- **Root SSH keys in published images.** `agent/build-image.sh` took
  the build user's `authorized_keys` unless told otherwise, so the images published with v0.2.0 and
  v0.2.1 let three lab keys (the maintainer's laptop, the CI deploy key, the lab hypervisor) log in as
  root to machines while they commission. Keys are now opt-in (`AUTHKEYS=`, the lab deploy sets it),
  and the release workflow builds the image of the tag itself and refuses one with `authorized_keys`.
- **Region file read (critical).** A rack controller could read any file on the region (CA key,
  config) through `/api/v1/rack/binary?arch=../..`. `arch` is now a known architecture.
- **Path traversal.** In `/api/v1/render/...` (admin) as well.
- **Machine-reported MACs.** A MAC could be a path, letting a PXE-booting host make the rack write
  outside the TFTP directory. Only real hardware addresses are accepted. Rack controllers write only
  `01-xx-..` file names.
- **Delegated intermediates.** An ACME intermediate could mint admin or rack certificates (name
  constraints do not cover the CN), and IP addresses when only a domain was permitted.
  - Admin and rack identities must now be signed by the root itself.
  - Intermediates carry a ServerAuth-only EKU, and the unconstrained name type is closed.
- **Rack certificates.** These were ServerAuth certificates for any address the rack reported, so
  one rack could pose as the region.
  - They are now client-only with no names.
  - Racks accept only the region's own certificate.
  - `rack join` checks the returned CA against the fingerprint in the code.
  - An unbound join code cannot take over an existing rack's name.
  - Job results are accepted only from the rack that took the job.
- **Rack self-update.** The root unit ran a binary the unprivileged rack user staged.
  - The region now signs each binary with its CA key.
  - The root unit verifies the signature against the CA from `rack join`, in a root-owned directory,
    without following links.
- **dnsmasq as root.** dnsmasq (root) read configuration the rollcall user writes, and the reload
  script copied files through links in rollcall's directory.
  - It now reads a checked copy in `/var/lib/rollcall-dnsmasq`.
  - The copy may hold only the options Rollcall generates: nothing that runs a program or includes
    a file.
- **Machine record takeover.** Any host on the provisioning network could overwrite a deployed
  machine's record (and BMC) by reporting its UUID.
  - Known machines are re-commissioned only when `ready`, `recommission` or failed.
  - A MAC another machine owns keeps its boot file.
  - The rack controller of a ready machine is not changed by whoever reports it.
- **Injection into dnsmasq, cloud-init, PXE menus, udev rules and DNS updates.** Subnet domain and
  gateway, host names, NIC names, reservation names and machine-reported identity strings are now
  validated or reduced to one line. Machine IDs are letters and digits only.
- **Web UI.**
  - Escaping is fixed for machine-reported inventory fields (stored XSS).
  - A Content-Security-Policy, `frame-ancestors 'none'`, nosniff and no-referrer are set.
  - Cross-origin state-changing requests are refused (`http.CrossOriginProtection`), so a page
    elsewhere cannot use a browser's admin certificate.
  - `?token=` is no longer logged.
- **Deploy agent.**
  - The image URL no longer goes into a shell command; the checksum is taken over the bytes written.
  - Seed files cannot leave the target root.
  - The BMC password is not kept in the fallback inventory file.
- **Redfish.** Requests stay on the BMC and under `/redfish/`: paths the BMC returns cannot send the
  credentials elsewhere, and `scheme://` addresses are reduced to scheme and host.
- **Machine tokens.** Job and node tokens are readable from the TFTP pxelinux files. They are now
  accepted only from the machine's own address (its static lease); through a rack, the rack's
  `X-Forwarded-For` entry counts. A machine without a fixed address binds a job to its first user.
  `agent_source_check: false` turns this off.
- **Agents over HTTPS.** Rack controllers get a server certificate for their own host and serve
  the agents' endpoints and images on 443. Agents accept the CA by the fingerprint on the PXE
  command line and do not fall back to HTTP. Old agents keep working over HTTP. The commissioning
  image has to be rebuilt to get the new agents.
  - The stateless overlay client works the same way (rebuild stateless images).
- **UEFI and Secure Boot.** UEFI machines PXE-boot through Debian's signed shim and GRUB, so machines
  with Secure Boot on can boot at all; before, the firmware refused the unsigned `syslinux.efi`.
  - GRUB menus are made from the pxelinux ones.
  - The local disk is booted by chainloading the installed OS's signed loader.
  - Secure Boot verifies the kernel; the initrd and squashfs are not covered.
- **BMC certificates.** HTTPS BMCs (Redfish, Proxmox) are pinned on first use. The certificate
  is read in a handshake that carries no credentials, and a changed certificate gets none
  (`tls_fingerprint`; cleared on an address change or a BMC firmware update by Rollcall).
- **BMC hints.** `bmc-hint` no longer returns the password. The region attaches the configured
  credentials itself and checks the BMC from the rack controller.
- **Rack certificate renewal.** Renewal could not overwrite the root-owned files from `rack join`.
  Renewed certificates now go atomically to `/var/lib/rollcall/rack-cert`, the previous serial is
  accepted for 7 days after a renewal, and `rollcall rack renew` renews on demand.
- **Defaults and limits.**
  - `admin_http` defaults to `loopback`.
  - Request bodies are capped (commissioning 4 MB, all 16 MB), with read and idle timeouts.
  - `bootstrap.sh` and `publish-image.sh` keep the database password and API tokens off command
    lines.

### Fixed
- IPAM: a reconcile running during an allocation could delete the record just reserved in NetBox.
- IPAM and DNS sync: one failing record no longer stops the rest of the pass, deletes included.
  Reservations are stored in one spelling.
- The job watchdog could fail a job that had just started.
- Release without `-disk` wiped the smallest disk, not the one the OS was deployed to. Automatic
  disk choice skips SAN LUNs (iSCSI, FC, SRP).
- RFC 2136 lookups follow the request context. The marker is not removed when the database cannot
  say whether other records need it.
- Smaller fixes:
  - `seq` in templates is capped.
  - Layout selector `ib:-1` no longer panics.
  - Redfish detection no longer runs on a machine that cannot be read.
- dnsmasq busy loop in proxy mode (a dnsmasq bug with tagged proxy ranges, found with gdb): proxy
  ranges are untagged, PXE services limited by interface; `dhcp-boot`/`dhcp-match` only for served
  subnets, no `dhcp-host` lines for proxied ones.

## [0.2.1] - 2026-10-28

### Fixed
- `bootstrap.sh --image-url` with a release URL took the release's `SHA256SUMS` (the binaries') instead
  of `commissioning-SHA256SUMS`; it now prefers the image's and says when a private repository
  needs `IMAGE_TOKEN`.

## [0.2.0] - 2026-10-28

Second release: install from scratch, network model and region/rack split, external IPAM
(NetBox), a certificate authority with delegation, many power drivers, IPv6, and a web UI and
console for fleets.

### Added
- Machine tags (MAAS-style): manual tags per machine, auto tags from match rules over the
  inventory (`{{ gt .Node.GPUs 0 }}`), re-evaluated at commissioning and when a tag changes.
- Tags carry a cloud-init fragment merged into deployment user-data (`deploy/tags/gpu-nvidia.yaml`
  installs the NVIDIA driver on GPU nodes) and extra overlays for stateless nodes.
- `rollcall tag`, `rollcall tags`, `rollcall list -tag`, `deploy tag:NAME` / `power tag:NAME`
  fan-out; tags column, editor and definitions panel in the web UI; `?tag=` filter on the API.
- Web UI redesigned for fleets: sidebar navigation (Overview, Machines, Jobs, Tags, Profiles,
  OS images), state/tag filter chips, search, sorting and paging, multi-select with bulk
  power/deploy/assign/tag/re-commission/release, machine detail with tabs, dialog-driven
  actions, fleet-wide `/api/v1/events` and `/api/v1/jobs`, `?token=` sign-in for bookmarks.
- Physical location per machine (building, room, rack, unit, asset tag): `rollcall location`,
  CSV import matched by id/serial/MAC/node name with dry run, rack definitions with height,
  `list -rack/-room/-building`, location column and chips in the UI, Racks page with
  elevation views, bulk "Set location…", `.Node.Rack` etc. in templates and tag rules.
- Terminal console: a bare `rollcall` in a terminal opens a full-screen UI in the llmgw style
  (Bubble Tea): overview, machines with filter/selection/bulk actions, machine detail, racks
  with elevation, jobs, events, tags, profiles, OS images; connect screen that saves
  `~/.config/rollcall/client.json`, which the CLI also reads; `rollcall tui -dump` renders one
  screen. First Go tests (console rendering and keys).
- Network model after MAAS: fabrics, VLANs, subnets, IP ranges (dynamic/reserved/static) and
  IPAM; addresses already in use are adopted. `rollcall fabric|vlan|subnet|range|ip`.
- Interface layouts (physical, bond, VLAN; NICs by selector) per machine or per tag; deployment
  writes cloud-init `network-config` and udev name rules into the image; release frees addresses.
  `rollcall layout`, `rollcall net show|set`.
- Region/rack split: `rollcall rack` (dnsmasq state, per-MAC PXE files, BMC commands, image cache,
  node API proxy) long-polls the region; the region runs an embedded rack controller.
  `deploy/install-rack.sh`, `rollcall controllers`, `rollcall dhcp`.
- dnsmasq configuration generated from the model, restarted only when it changes and rolled back
  if dnsmasq does not come back.
- Web UI: Network page (rack controllers, subnets with ranges and usage, layouts), network tab per
  machine, layout action; console: Network page, network section, layout action.
- Certificate authority on the region (llmgw-style): HTTPS listener with an auto-renewed region
  certificate, `admin_http` policy, `/ca.pem`; rack controllers enroll with one-time join codes
  (`rollcall controller join-code`, `rollcall rack join`) and use mutual TLS, renewing their
  certificate themselves; removal revokes; deployed machines and overlays trust the CA;
  `rollcall ca show|issue`; the console trusts a CA on first connect after showing its fingerprint.
- Power drivers: ipmi, redfish, idrac, ilo, supermicro, xcc, openbmc, proxmox, manual, run on the
  rack controller; Redfish auto-detection after commissioning; `rollcall bmc`, BMC editing in the
  web UI and console. Lab Redfish emulator with iDRAC/iLO/generic profiles.
- IPv6: subnets, IPAM, dual-stack default layout, per-family default routes, dnsmasq RA/DHCPv6,
  AAAA host records, IPv6 listen addresses and URLs, IPv6 in the commissioning inventory.
- External IPAM authority (NetBox driver): allocation registers each address in NetBox first and
  skips addresses others hold there; ownership by tag + `rollcall_machine` custom field; releases
  and a periodic reconcile create, fix and delete only Rollcall-owned records; `dns_name` per address
  for DNS sync; delegated prefixes (`rollcall ipam prefixes|import`); `rollcall ipam [sync -dry]`,
  `/api/v1/ipam`.
- Web UI: network topology drawing (region, external IPAM, rack controllers, one bus per VLAN with
  its subnets, machines wired to every VLAN they have an address on; large groups collapse) on the
  Network page and, compact, on the Overview. Overview redesigned: fleet state bar, per-rack counts,
  attention list incl. IPAM sync, capacity and free pool, OS, hardware, BMC drivers, locations, tags,
  recent deployments.
- CA delegation: new roots allow one level of intermediates; `rollcall ca intermediate NAME -permit
  DOMAIN,CIDR [-csr FILE]` signs a name-constrained intermediate (for an ACME server such as Caddy's
  `acme_server`); `rollcall ca rotate -yes` replaces a root that cannot delegate; tag fragment
  `deploy/tags/acme-cert.yaml` fetches and renews a node certificate with certbot.
- Rack controllers update themselves to the region's version: checksum- and version-checked
  download over mutual TLS, installed by a root path unit with automatic rollback;
  `rack_binaries` for other architectures, `auto_update` to opt out; versions in
  `rollcall controllers` and the web UI.
- Addresses on secondary subnets with a gateway get their own routing table and a source rule in
  `network-config`, so replies from a storage or tenant VLAN leave through that VLAN's gateway.
- With `admin_http` other than `allow`, browsers that open the web UI over HTTP are redirected to
  HTTPS (nodes keep HTTP for images and the agent).
- Admin client certificates: `rollcall admin-cert request NAME` (key stays on the client, CSR only),
  `list`, `revoke`; the region accepts them as admin over HTTPS; CLI/console use them from
  `client.json`, the web UI works without a token when the browser presents one (`.p12` written).
- `deploy/bootstrap.sh`: a region on a fresh Debian 13 host. Day 0 needs nothing about the network:
  packages, PostgreSQL, TFTP tree, config, services and a verified commissioning image, the region
  reachable on its management address with no DHCP or subnets. Run again with `--iface` once the
  provisioning network is known: server address updated, subnet with ranges, DHCP, optional NAT.
- `rollcall net discover`: per interface link, speed, addresses and implied subnets, switch/port/VLAN
  from LLDP, DHCP servers that answer, neighbours, and what to do with it.
- `commission_append` follows `server_url` when left empty.
- `rollcall doctor`: checks configuration, database, listeners, certificates, TFTP files, image
  checksums, dnsmasq, DNS as machines see it, clock, foreign DHCP servers on the provisioning
  network (DHCPDISCOVER probe), rack controllers, OS images and IPAM, with what to do.
- `rollcall version`; CD-built binary carries `git describe`.
- Release notes from `CHANGELOG.md`.

### Fixed
- RHEL-family deployments (Rocky, Alma, CentOS) did not trust the Rollcall CA: cloud-init's
  `ca_certs` ran `update-ca-certificates` there and failed. The CA now goes into the image's trust
  anchors and a `bootcmd` rebuilds the store with the distribution's tool.
- A deploy whose power command was lost (rack or region restarting) stayed pending forever and the
  event looked like a success: power errors are now part of the event (`pxe boot failed`), and a
  job the machine has not started within `job_start_timeout` (default 30m, room for servers that
  take 4-5 minutes to POST) fails with a hint.
- CentOS Stream 9 image URL (the 502 was a missing file name).
- Rack controllers' dnsmasq had no upstream DNS on Debian (it looks for a resolvconf file that is
  usually missing) and refused every name outside the cluster domain; the generated configuration
  now forwards to systemd-resolved's stub (keeping split DNS) or the host's resolv.conf servers.
- Virtual BMC addresses are configured in netplan on the lab PXE server (they were lost on a
  network reconfigure, which made every BMC call fail).
- Commissioning console no longer shows a failing autologin (`noautologin`).

## [0.1.0] - 2026-10-27

First tagged release.

### Added
- Commissioning: Debian 13 live image with firmware, IB and NVMe tooling; hardware inventory
  (DMI, DIMMs, CPUs, disks, NVMe, NICs, InfiniBand, GPUs, PCI, LLDP, TPM); in-band BMC account
  creation with out-of-band hints for VMs; secrets sealed with AES-256-GCM; ipmitool server-side.
- Single static Go binary: HTTP API, embedded web UI, CLI, PostgreSQL schema.
- Stateless (Warewulf-style) nodes: profiles, node names and IP pools, rendered overlays, dnsmasq
  export, Slurm compute image and controller sync.
- OS deployment: image catalog with nightly upstream sync and vendor checksum verification
  (Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9, CentOS Stream 9), streamed `zstd | dd` writes
  with sha256 and live progress, cloud-init NoCloud seed, SELinux relabel, UEFI boot entry,
  phone-home, release/wipe jobs.
- Gitea Actions: CI, deploy to the PXE server, website publish, release on tags.
- Lab: Proxmox SDN subnet, `pve-vbmc` IPMI emulator for VMs.