Findings from a full review (auth/API, CA and rack channel, config generation, integrations, UI/agents/scripts), each verified in code: - rack binary endpoint read any region file via arch=../.. (CA key) - render endpoint traversal; machine-reported MACs used as file paths - delegated intermediates could mint admin/rack identities and IPs; identities now require root-issued certs, intermediates are ServerAuth-only with both name types constrained - rack certs were ServerAuth for rack-chosen SANs: now client-only; racks accept only the region's own cert; join checks the CA fingerprint; unbound join codes cannot take an existing name; job results bound to the rack that took them - rack self-update: region signs binaries with the CA key, the root unit verifies against the join-time CA in a root-owned dir - dnsmasq reads a checked, root-owned copy limited to generated options - commissioning could overwrite deployed machines; MAC takeover of PXE - injection into dnsmasq/cloud-init/PXE/udev/DNS via names and machine-reported strings; stored XSS in the UI; CSP and cross-origin protection; deploy agent shell injection and seed traversal - Redfish requests confined to the BMC's /redfish/ tree - admin_http defaults to loopback; body limits and timeouts; secrets off command lines Also: IPAM claim/reconcile race, sync stopping on one error, watchdog race, release wiping the wrong disk, SAN LUNs as auto targets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
216 lines
15 KiB
Markdown
216 lines
15 KiB
Markdown
# Changelog
|
|
|
|
All notable changes to Rollcall. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
|
|
the release workflow uses the section of the pushed tag as the release notes and refuses to
|
|
publish a tag that has no section here.
|
|
|
|
## [Unreleased]
|
|
|
|
### Added
|
|
- Proxy DHCP: a VLAN in `proxy` mode (`rollcall vlan add ... -proxy-dhcp`, web UI, `bootstrap.sh
|
|
--proxy-dhcp`) leaves addresses to the site's DHCP server and only answers PXE; machines keep DHCP
|
|
unless a static range is set aside; doctor and discover know the mode.
|
|
- DNS provider: `dns` in `config.json` writes machines' A/AAAA records into the site's DNS, through
|
|
the PowerDNS API (ownership by RRset comment) or RFC 2136 dynamic update with TSIG (ownership by a
|
|
`_rollcall.<name>` TXT marker and update prerequisites, so foreign records are never overwritten;
|
|
drift is detected by querying). `rollcall dns`, `rollcall dns sync [-dry]`, a doctor check.
|
|
|
|
### Security
|
|
A review of the whole code base; the fixes below. Upgrading: rerun `deploy/install.sh` on the
|
|
region and `deploy/install-rack.sh` on rack hosts (new dnsmasq layout, signed rack updates).
|
|
- **Region file read (critical).** A rack controller could read any file on the region (CA key,
|
|
config) through `/api/v1/rack/binary?arch=../..`. `arch` is now a known architecture.
|
|
- **Path traversal.** In `/api/v1/render/...` (admin) as well.
|
|
- **Machine-reported MACs.** A MAC could be a path, letting a PXE-booting host make the rack write
|
|
outside the TFTP directory. Only real hardware addresses are accepted. Rack controllers write only
|
|
`01-xx-..` file names.
|
|
- **Delegated intermediates.** An ACME intermediate could mint admin or rack certificates (name
|
|
constraints do not cover the CN), and IP addresses when only a domain was permitted.
|
|
- Admin and rack identities must now be signed by the root itself.
|
|
- Intermediates carry a ServerAuth-only EKU, and the unconstrained name type is closed.
|
|
- **Rack certificates.** These were ServerAuth certificates for any address the rack reported, so
|
|
one rack could pose as the region.
|
|
- They are now client-only with no names.
|
|
- Racks accept only the region's own certificate.
|
|
- `rack join` checks the returned CA against the fingerprint in the code.
|
|
- An unbound join code cannot take over an existing rack's name.
|
|
- Job results are accepted only from the rack that took the job.
|
|
- **Rack self-update.** The root unit ran a binary the unprivileged rack user staged.
|
|
- The region now signs each binary with its CA key.
|
|
- The root unit verifies the signature against the CA from `rack join`, in a root-owned directory,
|
|
without following links.
|
|
- **dnsmasq as root.** dnsmasq (root) read configuration the rollcall user writes, and the reload
|
|
script copied files through links in rollcall's directory.
|
|
- It now reads a checked copy in `/var/lib/rollcall-dnsmasq`.
|
|
- The copy may hold only the options Rollcall generates: nothing that runs a program or includes
|
|
a file.
|
|
- **Machine record takeover.** Any host on the provisioning network could overwrite a deployed
|
|
machine's record (and BMC) by reporting its UUID.
|
|
- Known machines are re-commissioned only when `ready`, `recommission` or failed.
|
|
- A MAC another machine owns keeps its boot file.
|
|
- The rack controller of a ready machine is not changed by whoever reports it.
|
|
- **Injection into dnsmasq, cloud-init, PXE menus, udev rules and DNS updates.** Subnet domain and
|
|
gateway, host names, NIC names, reservation names and machine-reported identity strings are now
|
|
validated or reduced to one line. Machine IDs are letters and digits only.
|
|
- **Web UI.**
|
|
- Escaping is fixed for machine-reported inventory fields (stored XSS).
|
|
- A Content-Security-Policy, `frame-ancestors 'none'`, nosniff and no-referrer are set.
|
|
- Cross-origin state-changing requests are refused (`http.CrossOriginProtection`), so a page
|
|
elsewhere cannot use a browser's admin certificate.
|
|
- `?token=` is no longer logged.
|
|
- **Deploy agent.**
|
|
- The image URL no longer goes into a shell command; the checksum is taken over the bytes written.
|
|
- Seed files cannot leave the target root.
|
|
- The BMC password is not kept in the fallback inventory file.
|
|
- **Redfish.** Requests stay on the BMC and under `/redfish/`: paths the BMC returns cannot send the
|
|
credentials elsewhere, and `scheme://` addresses are reduced to scheme and host.
|
|
- **Defaults and limits.**
|
|
- `admin_http` defaults to `loopback`.
|
|
- Request bodies are capped (commissioning 4 MB, all 16 MB), with read and idle timeouts.
|
|
- `bootstrap.sh` and `publish-image.sh` keep the database password and API tokens off command
|
|
lines.
|
|
|
|
### Fixed
|
|
- IPAM: a reconcile running during an allocation could delete the record just reserved in NetBox.
|
|
- IPAM and DNS sync: one failing record no longer stops the rest of the pass, deletes included.
|
|
Reservations are stored in one spelling.
|
|
- The job watchdog could fail a job that had just started.
|
|
- Release without `-disk` wiped the smallest disk, not the one the OS was deployed to. Automatic
|
|
disk choice skips SAN LUNs (iSCSI, FC, SRP).
|
|
- RFC 2136 lookups follow the request context. The marker is not removed when the database cannot
|
|
say whether other records need it.
|
|
- Smaller fixes:
|
|
- `seq` in templates is capped.
|
|
- Layout selector `ib:-1` no longer panics.
|
|
- Redfish detection no longer runs on a machine that cannot be read.
|
|
- dnsmasq busy loop in proxy mode (a dnsmasq bug with tagged proxy ranges, found with gdb): proxy
|
|
ranges are untagged, PXE services limited by interface; `dhcp-boot`/`dhcp-match` only for served
|
|
subnets, no `dhcp-host` lines for proxied ones.
|
|
|
|
## [0.2.1] - 2026-10-28
|
|
|
|
### Fixed
|
|
- `bootstrap.sh --image-url` with a release URL took the release's `SHA256SUMS` (the binaries') instead
|
|
of `commissioning-SHA256SUMS`; it now prefers the image's and says when a private repository
|
|
needs `IMAGE_TOKEN`.
|
|
|
|
## [0.2.0] - 2026-10-28
|
|
|
|
Second release: install from scratch, network model and region/rack split, external IPAM
|
|
(NetBox), a certificate authority with delegation, many power drivers, IPv6, and a web UI and
|
|
console for fleets.
|
|
|
|
### Added
|
|
- Machine tags (MAAS-style): manual tags per machine, auto tags from match rules over the
|
|
inventory (`{{ gt .Node.GPUs 0 }}`), re-evaluated at commissioning and when a tag changes.
|
|
- Tags carry a cloud-init fragment merged into deployment user-data (`deploy/tags/gpu-nvidia.yaml`
|
|
installs the NVIDIA driver on GPU nodes) and extra overlays for stateless nodes.
|
|
- `rollcall tag`, `rollcall tags`, `rollcall list -tag`, `deploy tag:NAME` / `power tag:NAME`
|
|
fan-out; tags column, editor and definitions panel in the web UI; `?tag=` filter on the API.
|
|
- Web UI redesigned for fleets: sidebar navigation (Overview, Machines, Jobs, Tags, Profiles,
|
|
OS images), state/tag filter chips, search, sorting and paging, multi-select with bulk
|
|
power/deploy/assign/tag/re-commission/release, machine detail with tabs, dialog-driven
|
|
actions, fleet-wide `/api/v1/events` and `/api/v1/jobs`, `?token=` sign-in for bookmarks.
|
|
- Physical location per machine (building, room, rack, unit, asset tag): `rollcall location`,
|
|
CSV import matched by id/serial/MAC/node name with dry run, rack definitions with height,
|
|
`list -rack/-room/-building`, location column and chips in the UI, Racks page with
|
|
elevation views, bulk "Set location…", `.Node.Rack` etc. in templates and tag rules.
|
|
- Terminal console: a bare `rollcall` in a terminal opens a full-screen UI in the llmgw style
|
|
(Bubble Tea): overview, machines with filter/selection/bulk actions, machine detail, racks
|
|
with elevation, jobs, events, tags, profiles, OS images; connect screen that saves
|
|
`~/.config/rollcall/client.json`, which the CLI also reads; `rollcall tui -dump` renders one
|
|
screen. First Go tests (console rendering and keys).
|
|
- Network model after MAAS: fabrics, VLANs, subnets, IP ranges (dynamic/reserved/static) and
|
|
IPAM; addresses already in use are adopted. `rollcall fabric|vlan|subnet|range|ip`.
|
|
- Interface layouts (physical, bond, VLAN; NICs by selector) per machine or per tag; deployment
|
|
writes cloud-init `network-config` and udev name rules into the image; release frees addresses.
|
|
`rollcall layout`, `rollcall net show|set`.
|
|
- Region/rack split: `rollcall rack` (dnsmasq state, per-MAC PXE files, BMC commands, image cache,
|
|
node API proxy) long-polls the region; the region runs an embedded rack controller.
|
|
`deploy/install-rack.sh`, `rollcall controllers`, `rollcall dhcp`.
|
|
- dnsmasq configuration generated from the model, restarted only when it changes and rolled back
|
|
if dnsmasq does not come back.
|
|
- Web UI: Network page (rack controllers, subnets with ranges and usage, layouts), network tab per
|
|
machine, layout action; console: Network page, network section, layout action.
|
|
- Certificate authority on the region (llmgw-style): HTTPS listener with an auto-renewed region
|
|
certificate, `admin_http` policy, `/ca.pem`; rack controllers enroll with one-time join codes
|
|
(`rollcall controller join-code`, `rollcall rack join`) and use mutual TLS, renewing their
|
|
certificate themselves; removal revokes; deployed machines and overlays trust the CA;
|
|
`rollcall ca show|issue`; the console trusts a CA on first connect after showing its fingerprint.
|
|
- Power drivers: ipmi, redfish, idrac, ilo, supermicro, xcc, openbmc, proxmox, manual, run on the
|
|
rack controller; Redfish auto-detection after commissioning; `rollcall bmc`, BMC editing in the
|
|
web UI and console. Lab Redfish emulator with iDRAC/iLO/generic profiles.
|
|
- IPv6: subnets, IPAM, dual-stack default layout, per-family default routes, dnsmasq RA/DHCPv6,
|
|
AAAA host records, IPv6 listen addresses and URLs, IPv6 in the commissioning inventory.
|
|
- External IPAM authority (NetBox driver): allocation registers each address in NetBox first and
|
|
skips addresses others hold there; ownership by tag + `rollcall_machine` custom field; releases
|
|
and a periodic reconcile create, fix and delete only Rollcall-owned records; `dns_name` per address
|
|
for DNS sync; delegated prefixes (`rollcall ipam prefixes|import`); `rollcall ipam [sync -dry]`,
|
|
`/api/v1/ipam`.
|
|
- Web UI: network topology drawing (region, external IPAM, rack controllers, one bus per VLAN with
|
|
its subnets, machines wired to every VLAN they have an address on; large groups collapse) on the
|
|
Network page and, compact, on the Overview. Overview redesigned: fleet state bar, per-rack counts,
|
|
attention list incl. IPAM sync, capacity and free pool, OS, hardware, BMC drivers, locations, tags,
|
|
recent deployments.
|
|
- CA delegation: new roots allow one level of intermediates; `rollcall ca intermediate NAME -permit
|
|
DOMAIN,CIDR [-csr FILE]` signs a name-constrained intermediate (for an ACME server such as Caddy's
|
|
`acme_server`); `rollcall ca rotate -yes` replaces a root that cannot delegate; tag fragment
|
|
`deploy/tags/acme-cert.yaml` fetches and renews a node certificate with certbot.
|
|
- Rack controllers update themselves to the region's version: checksum- and version-checked
|
|
download over mutual TLS, installed by a root path unit with automatic rollback;
|
|
`rack_binaries` for other architectures, `auto_update` to opt out; versions in
|
|
`rollcall controllers` and the web UI.
|
|
- Addresses on secondary subnets with a gateway get their own routing table and a source rule in
|
|
`network-config`, so replies from a storage or tenant VLAN leave through that VLAN's gateway.
|
|
- With `admin_http` other than `allow`, browsers that open the web UI over HTTP are redirected to
|
|
HTTPS (nodes keep HTTP for images and the agent).
|
|
- Admin client certificates: `rollcall admin-cert request NAME` (key stays on the client, CSR only),
|
|
`list`, `revoke`; the region accepts them as admin over HTTPS; CLI/console use them from
|
|
`client.json`, the web UI works without a token when the browser presents one (`.p12` written).
|
|
- `deploy/bootstrap.sh`: a region on a fresh Debian 13 host. Day 0 needs nothing about the network:
|
|
packages, PostgreSQL, TFTP tree, config, services and a verified commissioning image, the region
|
|
reachable on its management address with no DHCP or subnets. Run again with `--iface` once the
|
|
provisioning network is known: server address updated, subnet with ranges, DHCP, optional NAT.
|
|
- `rollcall net discover`: per interface link, speed, addresses and implied subnets, switch/port/VLAN
|
|
from LLDP, DHCP servers that answer, neighbours, and what to do with it.
|
|
- `commission_append` follows `server_url` when left empty.
|
|
- `rollcall doctor`: checks configuration, database, listeners, certificates, TFTP files, image
|
|
checksums, dnsmasq, DNS as machines see it, clock, foreign DHCP servers on the provisioning
|
|
network (DHCPDISCOVER probe), rack controllers, OS images and IPAM, with what to do.
|
|
- `rollcall version`; CD-built binary carries `git describe`.
|
|
- Release notes from `CHANGELOG.md`.
|
|
|
|
### Fixed
|
|
- RHEL-family deployments (Rocky, Alma, CentOS) did not trust the Rollcall CA: cloud-init's
|
|
`ca_certs` ran `update-ca-certificates` there and failed. The CA now goes into the image's trust
|
|
anchors and a `bootcmd` rebuilds the store with the distribution's tool.
|
|
- A deploy whose power command was lost (rack or region restarting) stayed pending forever and the
|
|
event looked like a success: power errors are now part of the event (`pxe boot failed`), and a
|
|
job the machine has not started within `job_start_timeout` (default 30m, room for servers that
|
|
take 4-5 minutes to POST) fails with a hint.
|
|
- CentOS Stream 9 image URL (the 502 was a missing file name).
|
|
- Rack controllers' dnsmasq had no upstream DNS on Debian (it looks for a resolvconf file that is
|
|
usually missing) and refused every name outside the cluster domain; the generated configuration
|
|
now forwards to systemd-resolved's stub (keeping split DNS) or the host's resolv.conf servers.
|
|
- Virtual BMC addresses are configured in netplan on the lab PXE server (they were lost on a
|
|
network reconfigure, which made every BMC call fail).
|
|
- Commissioning console no longer shows a failing autologin (`noautologin`).
|
|
|
|
## [0.1.0] - 2026-10-27
|
|
|
|
First tagged release.
|
|
|
|
### Added
|
|
- Commissioning: Debian 13 live image with firmware, IB and NVMe tooling; hardware inventory
|
|
(DMI, DIMMs, CPUs, disks, NVMe, NICs, InfiniBand, GPUs, PCI, LLDP, TPM); in-band BMC account
|
|
creation with out-of-band hints for VMs; secrets sealed with AES-256-GCM; ipmitool server-side.
|
|
- Single static Go binary: HTTP API, embedded web UI, CLI, PostgreSQL schema.
|
|
- Stateless (Warewulf-style) nodes: profiles, node names and IP pools, rendered overlays, dnsmasq
|
|
export, Slurm compute image and controller sync.
|
|
- OS deployment: image catalog with nightly upstream sync and vendor checksum verification
|
|
(Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9, CentOS Stream 9), streamed `zstd | dd` writes
|
|
with sha256 and live progress, cloud-init NoCloud seed, SELinux relabel, UEFI boot entry,
|
|
phone-home, release/wipe jobs.
|
|
- Gitea Actions: CI, deploy to the PXE server, website publish, release on tags.
|
|
- Lab: Proxmox SDN subnet, `pve-vbmc` IPMI emulator for VMs.
|