ipam.provider infoblox: one record:host per address, named by its DNS
name and configured for DNS in dns_view ("-" = IPAM only); ownership is
the rollcall_machine extensible attribute, changed with extattrs+ so a
host's other attributes stay. Anything else on the address (a host, a
lease, a fixed address, a DHCP or reserved range, network/broadcast) makes
it taken. Delegated networks carry the rollcall attribute. WAPI paging.
The reconcile no longer compares fields an authority does not keep
(Infoblox has no prefix length or status on a host). Built from the WAPI
reference and tested against a fake WAPI; no real grid yet.
Closes #11
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
269 lines
19 KiB
Markdown
269 lines
19 KiB
Markdown
# Changelog
|
|
|
|
All notable changes to Rollcall. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
|
|
the release workflow uses the section of the pushed tag as the release notes and refuses to
|
|
publish a tag that has no section here.
|
|
|
|
## [Unreleased]
|
|
|
|
### Added
|
|
- Proxy DHCP: a VLAN in `proxy` mode (`rollcall vlan add ... -proxy-dhcp`, web UI, `bootstrap.sh
|
|
--proxy-dhcp`) leaves addresses to the site's DHCP server and only answers PXE; machines keep DHCP
|
|
unless a static range is set aside; doctor and discover know the mode.
|
|
- DNS provider: `dns` in `config.json` writes machines' A/AAAA records into the site's DNS, through
|
|
the PowerDNS API (ownership by RRset comment) or RFC 2136 dynamic update with TSIG (ownership by a
|
|
`_rollcall.<name>` TXT marker and update prerequisites, so foreign records are never overwritten;
|
|
drift is detected by querying). `rollcall dns`, `rollcall dns sync [-dry]`, a doctor check.
|
|
- NetBox device sync (`ipam.device_sync`): machines become dcim devices with serial, device type,
|
|
role, status, interfaces with MAC objects, assigned addresses and primary IP. Site, location and
|
|
rack are matched by name from the machine's location and never created; ownership by tag and
|
|
`rollcall_machine` as for addresses, foreign devices of the same name are conflicts.
|
|
- Infoblox WAPI driver for the IPAM authority (`ipam.provider: infoblox`): one host record per
|
|
address (DNS-published in `dns_view`, or IPAM-only), ownership by the `rollcall_machine` extensible
|
|
attribute (kept with `extattrs+`), leases, fixed addresses and ranges count as taken, delegated
|
|
networks by the `rollcall` attribute. Not tested against a real grid yet.
|
|
- Lab NetBox (`lab/netbox/install.sh`): a container on the lab segment for the NetBox drivers.
|
|
- Fixed leases on the site's DHCP server for proxy-mode VLANs (`dhcp_reservations`), for Kea with
|
|
`host_cmds` (Kea 3.0+) and OpenWrt (rpcd). Rollcall's own leases are reconciled; foreign ones are
|
|
reported. `rollcall reservations`.
|
|
- Firmware updates over Redfish. A package repository on the region (`rollcall firmware import`
|
|
from a file or URL, served to BMCs through rack controllers) and each machine's firmware inventory.
|
|
- Update jobs: hand-over, task watch through BMC restarts, optional power cycle, version check
|
|
against the inventory. They resume after a region restart.
|
|
- Vendor plans from documentation or vendor tools: Dell, HPE (`.fwpkg` unpacked), Lenovo,
|
|
Supermicro, GIGABYTE/AMI/ASRock Rack, OpenBMC, xFusion, KAYTUS, Fujitsu iRMC. Cisco CIMC is
|
|
refused (SCP only).
|
|
- `fw_method`, `fw_params` and `firmware_base` on the BMC record.
|
|
- CLI `rollcall firmware`, a Firmware tab and repository in the web UI.
|
|
- Redfish vendor profiles: one standard driver plus per-vendor behaviour from public documentation,
|
|
for Dell iDRAC, HPE iLO, Lenovo XCC, Supermicro, xFusion/Huawei iBMC, KAYTUS/Inspur, Fujitsu iRMC,
|
|
Cisco CIMC, GIGABYTE, ASRock Rack, OpenBMC and AMI MegaRAC.
|
|
- Boot mode kept, omitted or tried per vendor.
|
|
- If-Match with 412/428 handling; future-state (`/SD`) and `@Redfish.Settings` targets.
|
|
- Busy retries and task following.
|
|
- Chassis reset fallback and an emulated power cycle.
|
|
- Session login (`auth=session`).
|
|
- Probe results (reset types, boot targets, boot mode, profile) are kept on the BMC and shown in
|
|
the UI, and the UI's driver list comes from the server.
|
|
|
|
### Security
|
|
A review of the whole code base; the fixes below. Upgrading: rerun `deploy/install.sh` on the
|
|
region and `deploy/install-rack.sh` on rack hosts (new dnsmasq layout, signed rack updates).
|
|
- **Region file read (critical).** A rack controller could read any file on the region (CA key,
|
|
config) through `/api/v1/rack/binary?arch=../..`. `arch` is now a known architecture.
|
|
- **Path traversal.** In `/api/v1/render/...` (admin) as well.
|
|
- **Machine-reported MACs.** A MAC could be a path, letting a PXE-booting host make the rack write
|
|
outside the TFTP directory. Only real hardware addresses are accepted. Rack controllers write only
|
|
`01-xx-..` file names.
|
|
- **Delegated intermediates.** An ACME intermediate could mint admin or rack certificates (name
|
|
constraints do not cover the CN), and IP addresses when only a domain was permitted.
|
|
- Admin and rack identities must now be signed by the root itself.
|
|
- Intermediates carry a ServerAuth-only EKU, and the unconstrained name type is closed.
|
|
- **Rack certificates.** These were ServerAuth certificates for any address the rack reported, so
|
|
one rack could pose as the region.
|
|
- They are now client-only with no names.
|
|
- Racks accept only the region's own certificate.
|
|
- `rack join` checks the returned CA against the fingerprint in the code.
|
|
- An unbound join code cannot take over an existing rack's name.
|
|
- Job results are accepted only from the rack that took the job.
|
|
- **Rack self-update.** The root unit ran a binary the unprivileged rack user staged.
|
|
- The region now signs each binary with its CA key.
|
|
- The root unit verifies the signature against the CA from `rack join`, in a root-owned directory,
|
|
without following links.
|
|
- **dnsmasq as root.** dnsmasq (root) read configuration the rollcall user writes, and the reload
|
|
script copied files through links in rollcall's directory.
|
|
- It now reads a checked copy in `/var/lib/rollcall-dnsmasq`.
|
|
- The copy may hold only the options Rollcall generates: nothing that runs a program or includes
|
|
a file.
|
|
- **Machine record takeover.** Any host on the provisioning network could overwrite a deployed
|
|
machine's record (and BMC) by reporting its UUID.
|
|
- Known machines are re-commissioned only when `ready`, `recommission` or failed.
|
|
- A MAC another machine owns keeps its boot file.
|
|
- The rack controller of a ready machine is not changed by whoever reports it.
|
|
- **Injection into dnsmasq, cloud-init, PXE menus, udev rules and DNS updates.** Subnet domain and
|
|
gateway, host names, NIC names, reservation names and machine-reported identity strings are now
|
|
validated or reduced to one line. Machine IDs are letters and digits only.
|
|
- **Web UI.**
|
|
- Escaping is fixed for machine-reported inventory fields (stored XSS).
|
|
- A Content-Security-Policy, `frame-ancestors 'none'`, nosniff and no-referrer are set.
|
|
- Cross-origin state-changing requests are refused (`http.CrossOriginProtection`), so a page
|
|
elsewhere cannot use a browser's admin certificate.
|
|
- `?token=` is no longer logged.
|
|
- **Deploy agent.**
|
|
- The image URL no longer goes into a shell command; the checksum is taken over the bytes written.
|
|
- Seed files cannot leave the target root.
|
|
- The BMC password is not kept in the fallback inventory file.
|
|
- **Redfish.** Requests stay on the BMC and under `/redfish/`: paths the BMC returns cannot send the
|
|
credentials elsewhere, and `scheme://` addresses are reduced to scheme and host.
|
|
- **Machine tokens.** Job and node tokens are readable from the TFTP pxelinux files. They are now
|
|
accepted only from the machine's own address (its static lease); through a rack, the rack's
|
|
`X-Forwarded-For` entry counts. A machine without a fixed address binds a job to its first user.
|
|
`agent_source_check: false` turns this off.
|
|
- **Agents over HTTPS.** Rack controllers get a server certificate for their own host and serve
|
|
the agents' endpoints and images on 443. Agents accept the CA by the fingerprint on the PXE
|
|
command line and do not fall back to HTTP. Old agents keep working over HTTP. The commissioning
|
|
image has to be rebuilt to get the new agents.
|
|
- The stateless overlay client works the same way (rebuild stateless images).
|
|
- **UEFI and Secure Boot.** UEFI machines PXE-boot through Debian's signed shim and GRUB, so machines
|
|
with Secure Boot on can boot at all; before, the firmware refused the unsigned `syslinux.efi`.
|
|
- GRUB menus are made from the pxelinux ones.
|
|
- The local disk is booted by chainloading the installed OS's signed loader.
|
|
- Secure Boot verifies the kernel; the initrd and squashfs are not covered.
|
|
- **BMC certificates.** HTTPS BMCs (Redfish, Proxmox) are pinned on first use. The certificate
|
|
is read in a handshake that carries no credentials, and a changed certificate gets none
|
|
(`tls_fingerprint`; cleared on an address change or a BMC firmware update by Rollcall).
|
|
- **BMC hints.** `bmc-hint` no longer returns the password. The region attaches the configured
|
|
credentials itself and checks the BMC from the rack controller.
|
|
- **Rack certificate renewal.** Renewal could not overwrite the root-owned files from `rack join`.
|
|
Renewed certificates now go atomically to `/var/lib/rollcall/rack-cert`, the previous serial is
|
|
accepted for 7 days after a renewal, and `rollcall rack renew` renews on demand.
|
|
- **Defaults and limits.**
|
|
- `admin_http` defaults to `loopback`.
|
|
- Request bodies are capped (commissioning 4 MB, all 16 MB), with read and idle timeouts.
|
|
- `bootstrap.sh` and `publish-image.sh` keep the database password and API tokens off command
|
|
lines.
|
|
|
|
### Fixed
|
|
- IPAM: a reconcile running during an allocation could delete the record just reserved in NetBox.
|
|
- IPAM and DNS sync: one failing record no longer stops the rest of the pass, deletes included.
|
|
Reservations are stored in one spelling.
|
|
- The job watchdog could fail a job that had just started.
|
|
- Release without `-disk` wiped the smallest disk, not the one the OS was deployed to. Automatic
|
|
disk choice skips SAN LUNs (iSCSI, FC, SRP).
|
|
- RFC 2136 lookups follow the request context. The marker is not removed when the database cannot
|
|
say whether other records need it.
|
|
- Smaller fixes:
|
|
- `seq` in templates is capped.
|
|
- Layout selector `ib:-1` no longer panics.
|
|
- Redfish detection no longer runs on a machine that cannot be read.
|
|
- dnsmasq busy loop in proxy mode (a dnsmasq bug with tagged proxy ranges, found with gdb): proxy
|
|
ranges are untagged, PXE services limited by interface; `dhcp-boot`/`dhcp-match` only for served
|
|
subnets, no `dhcp-host` lines for proxied ones.
|
|
|
|
## [0.2.1] - 2026-10-28
|
|
|
|
### Fixed
|
|
- `bootstrap.sh --image-url` with a release URL took the release's `SHA256SUMS` (the binaries') instead
|
|
of `commissioning-SHA256SUMS`; it now prefers the image's and says when a private repository
|
|
needs `IMAGE_TOKEN`.
|
|
|
|
## [0.2.0] - 2026-10-28
|
|
|
|
Second release: install from scratch, network model and region/rack split, external IPAM
|
|
(NetBox), a certificate authority with delegation, many power drivers, IPv6, and a web UI and
|
|
console for fleets.
|
|
|
|
### Added
|
|
- Machine tags (MAAS-style): manual tags per machine, auto tags from match rules over the
|
|
inventory (`{{ gt .Node.GPUs 0 }}`), re-evaluated at commissioning and when a tag changes.
|
|
- Tags carry a cloud-init fragment merged into deployment user-data (`deploy/tags/gpu-nvidia.yaml`
|
|
installs the NVIDIA driver on GPU nodes) and extra overlays for stateless nodes.
|
|
- `rollcall tag`, `rollcall tags`, `rollcall list -tag`, `deploy tag:NAME` / `power tag:NAME`
|
|
fan-out; tags column, editor and definitions panel in the web UI; `?tag=` filter on the API.
|
|
- Web UI redesigned for fleets: sidebar navigation (Overview, Machines, Jobs, Tags, Profiles,
|
|
OS images), state/tag filter chips, search, sorting and paging, multi-select with bulk
|
|
power/deploy/assign/tag/re-commission/release, machine detail with tabs, dialog-driven
|
|
actions, fleet-wide `/api/v1/events` and `/api/v1/jobs`, `?token=` sign-in for bookmarks.
|
|
- Physical location per machine (building, room, rack, unit, asset tag): `rollcall location`,
|
|
CSV import matched by id/serial/MAC/node name with dry run, rack definitions with height,
|
|
`list -rack/-room/-building`, location column and chips in the UI, Racks page with
|
|
elevation views, bulk "Set location…", `.Node.Rack` etc. in templates and tag rules.
|
|
- Terminal console: a bare `rollcall` in a terminal opens a full-screen UI in the llmgw style
|
|
(Bubble Tea): overview, machines with filter/selection/bulk actions, machine detail, racks
|
|
with elevation, jobs, events, tags, profiles, OS images; connect screen that saves
|
|
`~/.config/rollcall/client.json`, which the CLI also reads; `rollcall tui -dump` renders one
|
|
screen. First Go tests (console rendering and keys).
|
|
- Network model after MAAS: fabrics, VLANs, subnets, IP ranges (dynamic/reserved/static) and
|
|
IPAM; addresses already in use are adopted. `rollcall fabric|vlan|subnet|range|ip`.
|
|
- Interface layouts (physical, bond, VLAN; NICs by selector) per machine or per tag; deployment
|
|
writes cloud-init `network-config` and udev name rules into the image; release frees addresses.
|
|
`rollcall layout`, `rollcall net show|set`.
|
|
- Region/rack split: `rollcall rack` (dnsmasq state, per-MAC PXE files, BMC commands, image cache,
|
|
node API proxy) long-polls the region; the region runs an embedded rack controller.
|
|
`deploy/install-rack.sh`, `rollcall controllers`, `rollcall dhcp`.
|
|
- dnsmasq configuration generated from the model, restarted only when it changes and rolled back
|
|
if dnsmasq does not come back.
|
|
- Web UI: Network page (rack controllers, subnets with ranges and usage, layouts), network tab per
|
|
machine, layout action; console: Network page, network section, layout action.
|
|
- Certificate authority on the region (llmgw-style): HTTPS listener with an auto-renewed region
|
|
certificate, `admin_http` policy, `/ca.pem`; rack controllers enroll with one-time join codes
|
|
(`rollcall controller join-code`, `rollcall rack join`) and use mutual TLS, renewing their
|
|
certificate themselves; removal revokes; deployed machines and overlays trust the CA;
|
|
`rollcall ca show|issue`; the console trusts a CA on first connect after showing its fingerprint.
|
|
- Power drivers: ipmi, redfish, idrac, ilo, supermicro, xcc, openbmc, proxmox, manual, run on the
|
|
rack controller; Redfish auto-detection after commissioning; `rollcall bmc`, BMC editing in the
|
|
web UI and console. Lab Redfish emulator with iDRAC/iLO/generic profiles.
|
|
- IPv6: subnets, IPAM, dual-stack default layout, per-family default routes, dnsmasq RA/DHCPv6,
|
|
AAAA host records, IPv6 listen addresses and URLs, IPv6 in the commissioning inventory.
|
|
- External IPAM authority (NetBox driver): allocation registers each address in NetBox first and
|
|
skips addresses others hold there; ownership by tag + `rollcall_machine` custom field; releases
|
|
and a periodic reconcile create, fix and delete only Rollcall-owned records; `dns_name` per address
|
|
for DNS sync; delegated prefixes (`rollcall ipam prefixes|import`); `rollcall ipam [sync -dry]`,
|
|
`/api/v1/ipam`.
|
|
- Web UI: network topology drawing (region, external IPAM, rack controllers, one bus per VLAN with
|
|
its subnets, machines wired to every VLAN they have an address on; large groups collapse) on the
|
|
Network page and, compact, on the Overview. Overview redesigned: fleet state bar, per-rack counts,
|
|
attention list incl. IPAM sync, capacity and free pool, OS, hardware, BMC drivers, locations, tags,
|
|
recent deployments.
|
|
- CA delegation: new roots allow one level of intermediates; `rollcall ca intermediate NAME -permit
|
|
DOMAIN,CIDR [-csr FILE]` signs a name-constrained intermediate (for an ACME server such as Caddy's
|
|
`acme_server`); `rollcall ca rotate -yes` replaces a root that cannot delegate; tag fragment
|
|
`deploy/tags/acme-cert.yaml` fetches and renews a node certificate with certbot.
|
|
- Rack controllers update themselves to the region's version: checksum- and version-checked
|
|
download over mutual TLS, installed by a root path unit with automatic rollback;
|
|
`rack_binaries` for other architectures, `auto_update` to opt out; versions in
|
|
`rollcall controllers` and the web UI.
|
|
- Addresses on secondary subnets with a gateway get their own routing table and a source rule in
|
|
`network-config`, so replies from a storage or tenant VLAN leave through that VLAN's gateway.
|
|
- With `admin_http` other than `allow`, browsers that open the web UI over HTTP are redirected to
|
|
HTTPS (nodes keep HTTP for images and the agent).
|
|
- Admin client certificates: `rollcall admin-cert request NAME` (key stays on the client, CSR only),
|
|
`list`, `revoke`; the region accepts them as admin over HTTPS; CLI/console use them from
|
|
`client.json`, the web UI works without a token when the browser presents one (`.p12` written).
|
|
- `deploy/bootstrap.sh`: a region on a fresh Debian 13 host. Day 0 needs nothing about the network:
|
|
packages, PostgreSQL, TFTP tree, config, services and a verified commissioning image, the region
|
|
reachable on its management address with no DHCP or subnets. Run again with `--iface` once the
|
|
provisioning network is known: server address updated, subnet with ranges, DHCP, optional NAT.
|
|
- `rollcall net discover`: per interface link, speed, addresses and implied subnets, switch/port/VLAN
|
|
from LLDP, DHCP servers that answer, neighbours, and what to do with it.
|
|
- `commission_append` follows `server_url` when left empty.
|
|
- `rollcall doctor`: checks configuration, database, listeners, certificates, TFTP files, image
|
|
checksums, dnsmasq, DNS as machines see it, clock, foreign DHCP servers on the provisioning
|
|
network (DHCPDISCOVER probe), rack controllers, OS images and IPAM, with what to do.
|
|
- `rollcall version`; CD-built binary carries `git describe`.
|
|
- Release notes from `CHANGELOG.md`.
|
|
|
|
### Fixed
|
|
- RHEL-family deployments (Rocky, Alma, CentOS) did not trust the Rollcall CA: cloud-init's
|
|
`ca_certs` ran `update-ca-certificates` there and failed. The CA now goes into the image's trust
|
|
anchors and a `bootcmd` rebuilds the store with the distribution's tool.
|
|
- A deploy whose power command was lost (rack or region restarting) stayed pending forever and the
|
|
event looked like a success: power errors are now part of the event (`pxe boot failed`), and a
|
|
job the machine has not started within `job_start_timeout` (default 30m, room for servers that
|
|
take 4-5 minutes to POST) fails with a hint.
|
|
- CentOS Stream 9 image URL (the 502 was a missing file name).
|
|
- Rack controllers' dnsmasq had no upstream DNS on Debian (it looks for a resolvconf file that is
|
|
usually missing) and refused every name outside the cluster domain; the generated configuration
|
|
now forwards to systemd-resolved's stub (keeping split DNS) or the host's resolv.conf servers.
|
|
- Virtual BMC addresses are configured in netplan on the lab PXE server (they were lost on a
|
|
network reconfigure, which made every BMC call fail).
|
|
- Commissioning console no longer shows a failing autologin (`noautologin`).
|
|
|
|
## [0.1.0] - 2026-10-27
|
|
|
|
First tagged release.
|
|
|
|
### Added
|
|
- Commissioning: Debian 13 live image with firmware, IB and NVMe tooling; hardware inventory
|
|
(DMI, DIMMs, CPUs, disks, NVMe, NICs, InfiniBand, GPUs, PCI, LLDP, TPM); in-band BMC account
|
|
creation with out-of-band hints for VMs; secrets sealed with AES-256-GCM; ipmitool server-side.
|
|
- Single static Go binary: HTTP API, embedded web UI, CLI, PostgreSQL schema.
|
|
- Stateless (Warewulf-style) nodes: profiles, node names and IP pools, rendered overlays, dnsmasq
|
|
export, Slurm compute image and controller sync.
|
|
- OS deployment: image catalog with nightly upstream sync and vendor checksum verification
|
|
(Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9, CentOS Stream 9), streamed `zstd | dd` writes
|
|
with sha256 and live progress, cloud-init NoCloud seed, SELinux relabel, UEFI boot entry,
|
|
phone-home, release/wipe jobs.
|
|
- Gitea Actions: CI, deploy to the PXE server, website publish, release on tags.
|
|
- Lab: Proxmox SDN subnet, `pve-vbmc` IPMI emulator for VMs.
|