rollcall/CHANGELOG.md

# Changelog

All notable changes to Rollcall. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
the release workflow uses the section of the pushed tag as the release notes and refuses to
publish a tag that has no section here.

## [Unreleased]

### Added
- Proxy DHCP: a VLAN in `proxy` mode (`rollcall vlan add ... -proxy-dhcp`, web UI, `bootstrap.sh
  --proxy-dhcp`) leaves addresses to the site's DHCP server and only answers PXE; machines keep DHCP
  unless a static range is set aside; doctor and discover know the mode.
- DNS provider: `dns` in `config.json` writes machines' A/AAAA records into the site's DNS, through
  the PowerDNS API (ownership by RRset comment) or RFC 2136 dynamic update with TSIG (ownership by a
  `_rollcall.<name>` TXT marker and update prerequisites, so foreign records are never overwritten;
  drift is detected by querying). `rollcall dns`, `rollcall dns sync [-dry]`, a doctor check.
- NetBox device sync (`ipam.device_sync`): machines become dcim devices with serial, device type,
  role, status, interfaces with MAC objects, assigned addresses and primary IP. Site, location and
  rack are matched by name from the machine's location and never created; ownership by tag and
  `rollcall_machine` as for addresses, foreign devices of the same name are conflicts.
- Lab NetBox (`lab/netbox/install.sh`): a container on the lab segment for the NetBox drivers.
- Fixed leases on the site's DHCP server for proxy-mode VLANs (`dhcp_reservations`), for Kea with
  `host_cmds` (Kea 3.0+) and OpenWrt (rpcd). Rollcall's own leases are reconciled; foreign ones are
  reported. `rollcall reservations`.
- Firmware updates over Redfish. A package repository on the region (`rollcall firmware import`
  from a file or URL, served to BMCs through rack controllers) and each machine's firmware inventory.
  - Update jobs: hand-over, task watch through BMC restarts, optional power cycle, version check
    against the inventory. They resume after a region restart.
  - Vendor plans from documentation or vendor tools: Dell, HPE (`.fwpkg` unpacked), Lenovo,
    Supermicro, GIGABYTE/AMI/ASRock Rack, OpenBMC, xFusion, KAYTUS, Fujitsu iRMC. Cisco CIMC is
    refused (SCP only).
  - `fw_method`, `fw_params` and `firmware_base` on the BMC record.
  - CLI `rollcall firmware`, a Firmware tab and repository in the web UI.
- Redfish vendor profiles: one standard driver plus per-vendor behaviour from public documentation,
  for Dell iDRAC, HPE iLO, Lenovo XCC, Supermicro, xFusion/Huawei iBMC, KAYTUS/Inspur, Fujitsu iRMC,
  Cisco CIMC, GIGABYTE, ASRock Rack, OpenBMC and AMI MegaRAC.
  - Boot mode kept, omitted or tried per vendor.
  - If-Match with 412/428 handling; future-state (`/SD`) and `@Redfish.Settings` targets.
  - Busy retries and task following.
  - Chassis reset fallback and an emulated power cycle.
  - Session login (`auth=session`).
  - Probe results (reset types, boot targets, boot mode, profile) are kept on the BMC and shown in
    the UI, and the UI's driver list comes from the server.

### Security
A review of the whole code base; the fixes below. Upgrading: rerun `deploy/install.sh` on the
region and `deploy/install-rack.sh` on rack hosts (new dnsmasq layout, signed rack updates).
- **Region file read (critical).** A rack controller could read any file on the region (CA key,
  config) through `/api/v1/rack/binary?arch=../..`. `arch` is now a known architecture.
- **Path traversal.** In `/api/v1/render/...` (admin) as well.
- **Machine-reported MACs.** A MAC could be a path, letting a PXE-booting host make the rack write
  outside the TFTP directory. Only real hardware addresses are accepted. Rack controllers write only
  `01-xx-..` file names.
- **Delegated intermediates.** An ACME intermediate could mint admin or rack certificates (name
  constraints do not cover the CN), and IP addresses when only a domain was permitted.
  - Admin and rack identities must now be signed by the root itself.
  - Intermediates carry a ServerAuth-only EKU, and the unconstrained name type is closed.
- **Rack certificates.** These were ServerAuth certificates for any address the rack reported, so
  one rack could pose as the region.
  - They are now client-only with no names.
  - Racks accept only the region's own certificate.
  - `rack join` checks the returned CA against the fingerprint in the code.
  - An unbound join code cannot take over an existing rack's name.
  - Job results are accepted only from the rack that took the job.
- **Rack self-update.** The root unit ran a binary the unprivileged rack user staged.
  - The region now signs each binary with its CA key.
  - The root unit verifies the signature against the CA from `rack join`, in a root-owned directory,
    without following links.
- **dnsmasq as root.** dnsmasq (root) read configuration the rollcall user writes, and the reload
  script copied files through links in rollcall's directory.
  - It now reads a checked copy in `/var/lib/rollcall-dnsmasq`.
  - The copy may hold only the options Rollcall generates: nothing that runs a program or includes
    a file.
- **Machine record takeover.** Any host on the provisioning network could overwrite a deployed
  machine's record (and BMC) by reporting its UUID.
  - Known machines are re-commissioned only when `ready`, `recommission` or failed.
  - A MAC another machine owns keeps its boot file.
  - The rack controller of a ready machine is not changed by whoever reports it.
- **Injection into dnsmasq, cloud-init, PXE menus, udev rules and DNS updates.** Subnet domain and
  gateway, host names, NIC names, reservation names and machine-reported identity strings are now
  validated or reduced to one line. Machine IDs are letters and digits only.
- **Web UI.**
  - Escaping is fixed for machine-reported inventory fields (stored XSS).
  - A Content-Security-Policy, `frame-ancestors 'none'`, nosniff and no-referrer are set.
  - Cross-origin state-changing requests are refused (`http.CrossOriginProtection`), so a page
    elsewhere cannot use a browser's admin certificate.
  - `?token=` is no longer logged.
- **Deploy agent.**
  - The image URL no longer goes into a shell command; the checksum is taken over the bytes written.
  - Seed files cannot leave the target root.
  - The BMC password is not kept in the fallback inventory file.
- **Redfish.** Requests stay on the BMC and under `/redfish/`: paths the BMC returns cannot send the
  credentials elsewhere, and `scheme://` addresses are reduced to scheme and host.
- **Machine tokens.** Job and node tokens are readable from the TFTP pxelinux files. They are now
  accepted only from the machine's own address (its static lease); through a rack, the rack's
  `X-Forwarded-For` entry counts. A machine without a fixed address binds a job to its first user.
  `agent_source_check: false` turns this off.
- **Agents over HTTPS.** Rack controllers get a server certificate for their own host and serve
  the agents' endpoints and images on 443. Agents accept the CA by the fingerprint on the PXE
  command line and do not fall back to HTTP. Old agents keep working over HTTP. The commissioning
  image has to be rebuilt to get the new agents.
  - The stateless overlay client works the same way (rebuild stateless images).
- **UEFI and Secure Boot.** UEFI machines PXE-boot through Debian's signed shim and GRUB, so machines
  with Secure Boot on can boot at all; before, the firmware refused the unsigned `syslinux.efi`.
  - GRUB menus are made from the pxelinux ones.
  - The local disk is booted by chainloading the installed OS's signed loader.
  - Secure Boot verifies the kernel; the initrd and squashfs are not covered.
- **BMC certificates.** HTTPS BMCs (Redfish, Proxmox) are pinned on first use. The certificate
  is read in a handshake that carries no credentials, and a changed certificate gets none
  (`tls_fingerprint`; cleared on an address change or a BMC firmware update by Rollcall).
- **BMC hints.** `bmc-hint` no longer returns the password. The region attaches the configured
  credentials itself and checks the BMC from the rack controller.
- **Rack certificate renewal.** Renewal could not overwrite the root-owned files from `rack join`.
  Renewed certificates now go atomically to `/var/lib/rollcall/rack-cert`, the previous serial is
  accepted for 7 days after a renewal, and `rollcall rack renew` renews on demand.
- **Defaults and limits.**
  - `admin_http` defaults to `loopback`.
  - Request bodies are capped (commissioning 4 MB, all 16 MB), with read and idle timeouts.
  - `bootstrap.sh` and `publish-image.sh` keep the database password and API tokens off command
    lines.

### Fixed
- IPAM: a reconcile running during an allocation could delete the record just reserved in NetBox.
- IPAM and DNS sync: one failing record no longer stops the rest of the pass, deletes included.
  Reservations are stored in one spelling.
- The job watchdog could fail a job that had just started.
- Release without `-disk` wiped the smallest disk, not the one the OS was deployed to. Automatic
  disk choice skips SAN LUNs (iSCSI, FC, SRP).
- RFC 2136 lookups follow the request context. The marker is not removed when the database cannot
  say whether other records need it.
- Smaller fixes:
  - `seq` in templates is capped.
  - Layout selector `ib:-1` no longer panics.
  - Redfish detection no longer runs on a machine that cannot be read.
- dnsmasq busy loop in proxy mode (a dnsmasq bug with tagged proxy ranges, found with gdb): proxy
  ranges are untagged, PXE services limited by interface; `dhcp-boot`/`dhcp-match` only for served
  subnets, no `dhcp-host` lines for proxied ones.

## [0.2.1] - 2026-10-28

### Fixed
- `bootstrap.sh --image-url` with a release URL took the release's `SHA256SUMS` (the binaries') instead
  of `commissioning-SHA256SUMS`; it now prefers the image's and says when a private repository
  needs `IMAGE_TOKEN`.

## [0.2.0] - 2026-10-28

Second release: install from scratch, network model and region/rack split, external IPAM
(NetBox), a certificate authority with delegation, many power drivers, IPv6, and a web UI and
console for fleets.

### Added
- Machine tags (MAAS-style): manual tags per machine, auto tags from match rules over the
  inventory (`{{ gt .Node.GPUs 0 }}`), re-evaluated at commissioning and when a tag changes.
- Tags carry a cloud-init fragment merged into deployment user-data (`deploy/tags/gpu-nvidia.yaml`
  installs the NVIDIA driver on GPU nodes) and extra overlays for stateless nodes.
- `rollcall tag`, `rollcall tags`, `rollcall list -tag`, `deploy tag:NAME` / `power tag:NAME`
  fan-out; tags column, editor and definitions panel in the web UI; `?tag=` filter on the API.
- Web UI redesigned for fleets: sidebar navigation (Overview, Machines, Jobs, Tags, Profiles,
  OS images), state/tag filter chips, search, sorting and paging, multi-select with bulk
  power/deploy/assign/tag/re-commission/release, machine detail with tabs, dialog-driven
  actions, fleet-wide `/api/v1/events` and `/api/v1/jobs`, `?token=` sign-in for bookmarks.
- Physical location per machine (building, room, rack, unit, asset tag): `rollcall location`,
  CSV import matched by id/serial/MAC/node name with dry run, rack definitions with height,
  `list -rack/-room/-building`, location column and chips in the UI, Racks page with
  elevation views, bulk "Set location…", `.Node.Rack` etc. in templates and tag rules.
- Terminal console: a bare `rollcall` in a terminal opens a full-screen UI in the llmgw style
  (Bubble Tea): overview, machines with filter/selection/bulk actions, machine detail, racks
  with elevation, jobs, events, tags, profiles, OS images; connect screen that saves
  `~/.config/rollcall/client.json`, which the CLI also reads; `rollcall tui -dump` renders one
  screen. First Go tests (console rendering and keys).
- Network model after MAAS: fabrics, VLANs, subnets, IP ranges (dynamic/reserved/static) and
  IPAM; addresses already in use are adopted. `rollcall fabric|vlan|subnet|range|ip`.
- Interface layouts (physical, bond, VLAN; NICs by selector) per machine or per tag; deployment
  writes cloud-init `network-config` and udev name rules into the image; release frees addresses.
  `rollcall layout`, `rollcall net show|set`.
- Region/rack split: `rollcall rack` (dnsmasq state, per-MAC PXE files, BMC commands, image cache,
  node API proxy) long-polls the region; the region runs an embedded rack controller.
  `deploy/install-rack.sh`, `rollcall controllers`, `rollcall dhcp`.
- dnsmasq configuration generated from the model, restarted only when it changes and rolled back
  if dnsmasq does not come back.
- Web UI: Network page (rack controllers, subnets with ranges and usage, layouts), network tab per
  machine, layout action; console: Network page, network section, layout action.
- Certificate authority on the region (llmgw-style): HTTPS listener with an auto-renewed region
  certificate, `admin_http` policy, `/ca.pem`; rack controllers enroll with one-time join codes
  (`rollcall controller join-code`, `rollcall rack join`) and use mutual TLS, renewing their
  certificate themselves; removal revokes; deployed machines and overlays trust the CA;
  `rollcall ca show|issue`; the console trusts a CA on first connect after showing its fingerprint.
- Power drivers: ipmi, redfish, idrac, ilo, supermicro, xcc, openbmc, proxmox, manual, run on the
  rack controller; Redfish auto-detection after commissioning; `rollcall bmc`, BMC editing in the
  web UI and console. Lab Redfish emulator with iDRAC/iLO/generic profiles.
- IPv6: subnets, IPAM, dual-stack default layout, per-family default routes, dnsmasq RA/DHCPv6,
  AAAA host records, IPv6 listen addresses and URLs, IPv6 in the commissioning inventory.
- External IPAM authority (NetBox driver): allocation registers each address in NetBox first and
  skips addresses others hold there; ownership by tag + `rollcall_machine` custom field; releases
  and a periodic reconcile create, fix and delete only Rollcall-owned records; `dns_name` per address
  for DNS sync; delegated prefixes (`rollcall ipam prefixes|import`); `rollcall ipam [sync -dry]`,
  `/api/v1/ipam`.
- Web UI: network topology drawing (region, external IPAM, rack controllers, one bus per VLAN with
  its subnets, machines wired to every VLAN they have an address on; large groups collapse) on the
  Network page and, compact, on the Overview. Overview redesigned: fleet state bar, per-rack counts,
  attention list incl. IPAM sync, capacity and free pool, OS, hardware, BMC drivers, locations, tags,
  recent deployments.
- CA delegation: new roots allow one level of intermediates; `rollcall ca intermediate NAME -permit
  DOMAIN,CIDR [-csr FILE]` signs a name-constrained intermediate (for an ACME server such as Caddy's
  `acme_server`); `rollcall ca rotate -yes` replaces a root that cannot delegate; tag fragment
  `deploy/tags/acme-cert.yaml` fetches and renews a node certificate with certbot.
- Rack controllers update themselves to the region's version: checksum- and version-checked
  download over mutual TLS, installed by a root path unit with automatic rollback;
  `rack_binaries` for other architectures, `auto_update` to opt out; versions in
  `rollcall controllers` and the web UI.
- Addresses on secondary subnets with a gateway get their own routing table and a source rule in
  `network-config`, so replies from a storage or tenant VLAN leave through that VLAN's gateway.
- With `admin_http` other than `allow`, browsers that open the web UI over HTTP are redirected to
  HTTPS (nodes keep HTTP for images and the agent).
- Admin client certificates: `rollcall admin-cert request NAME` (key stays on the client, CSR only),
  `list`, `revoke`; the region accepts them as admin over HTTPS; CLI/console use them from
  `client.json`, the web UI works without a token when the browser presents one (`.p12` written).
- `deploy/bootstrap.sh`: a region on a fresh Debian 13 host. Day 0 needs nothing about the network:
  packages, PostgreSQL, TFTP tree, config, services and a verified commissioning image, the region
  reachable on its management address with no DHCP or subnets. Run again with `--iface` once the
  provisioning network is known: server address updated, subnet with ranges, DHCP, optional NAT.
- `rollcall net discover`: per interface link, speed, addresses and implied subnets, switch/port/VLAN
  from LLDP, DHCP servers that answer, neighbours, and what to do with it.
- `commission_append` follows `server_url` when left empty.
- `rollcall doctor`: checks configuration, database, listeners, certificates, TFTP files, image
  checksums, dnsmasq, DNS as machines see it, clock, foreign DHCP servers on the provisioning
  network (DHCPDISCOVER probe), rack controllers, OS images and IPAM, with what to do.
- `rollcall version`; CD-built binary carries `git describe`.
- Release notes from `CHANGELOG.md`.

### Fixed
- RHEL-family deployments (Rocky, Alma, CentOS) did not trust the Rollcall CA: cloud-init's
  `ca_certs` ran `update-ca-certificates` there and failed. The CA now goes into the image's trust
  anchors and a `bootcmd` rebuilds the store with the distribution's tool.
- A deploy whose power command was lost (rack or region restarting) stayed pending forever and the
  event looked like a success: power errors are now part of the event (`pxe boot failed`), and a
  job the machine has not started within `job_start_timeout` (default 30m, room for servers that
  take 4-5 minutes to POST) fails with a hint.
- CentOS Stream 9 image URL (the 502 was a missing file name).
- Rack controllers' dnsmasq had no upstream DNS on Debian (it looks for a resolvconf file that is
  usually missing) and refused every name outside the cluster domain; the generated configuration
  now forwards to systemd-resolved's stub (keeping split DNS) or the host's resolv.conf servers.
- Virtual BMC addresses are configured in netplan on the lab PXE server (they were lost on a
  network reconfigure, which made every BMC call fail).
- Commissioning console no longer shows a failing autologin (`noautologin`).

## [0.1.0] - 2026-10-27

First tagged release.

### Added
- Commissioning: Debian 13 live image with firmware, IB and NVMe tooling; hardware inventory
  (DMI, DIMMs, CPUs, disks, NVMe, NICs, InfiniBand, GPUs, PCI, LLDP, TPM); in-band BMC account
  creation with out-of-band hints for VMs; secrets sealed with AES-256-GCM; ipmitool server-side.
- Single static Go binary: HTTP API, embedded web UI, CLI, PostgreSQL schema.
- Stateless (Warewulf-style) nodes: profiles, node names and IP pools, rendered overlays, dnsmasq
  export, Slurm compute image and controller sync.
- OS deployment: image catalog with nightly upstream sync and vendor checksum verification
  (Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9, CentOS Stream 9), streamed `zstd | dd` writes
  with sha256 and live progress, cloud-init NoCloud seed, SELinux relabel, UEFI boot entry,
  phone-home, release/wipe jobs.
- Gitea Actions: CI, deploy to the PXE server, website publish, release on tags.
- Lab: Proxmox SDN subnet, `pve-vbmc` IPMI emulator for VMs.