A firmware repository on the region (firmware_dir, served under
/firmware/<id>/<file>, cached by rack controllers for BMCs to pull),
each machine's firmware inventory (FirmwareInventory, classified into
bios/bmc/nic/raid/cpld/psu/drive), and update jobs (kind "firmware")
that the region drives with short rack commands: fw-start hands the
package over, fw-poll watches the task (tolerating a BMC that restarts),
fw-finish releases what was claimed, fw-inventory verifies the version.
Running updates resume after a region restart; they cannot be cancelled
once the BMC has the package.
Standard methods (multipart, SimpleUpdate pull, HttpPushUri) with known
Content-Length, plus vendor plans from their documentation or their own
tools: Dell MultipartUpload (OnReset), HPE SimpleUpdate with .fwpkg
extracted at import, Lenovo /mfwupdate with HttpPushUriTargetsBusy,
Supermicro /upload with targets and preserve options, GIGABYTE/AMI/
ASRock Rack /upload with OemParameters ImageType, OpenBMC
update-multipart, xFusion upload to /tmp/web then SimpleUpdate, KAYTUS
upload then OEM SimpleUpdate with fixed tasks, Fujitsu iRMC OEM actions;
Cisco CIMC is refused clearly (SCP only). fw_params and fw_method on
the BMC record override a plan.
CLI rollcall firmware list|import|rm|show|update, API, a Firmware tab
and a repository card in the UI. The lab Redfish emulator gains an
UpdateService (BMC applies at once, BIOS at the next reset).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Findings from a full review (auth/API, CA and rack channel, config
generation, integrations, UI/agents/scripts), each verified in code:
- rack binary endpoint read any region file via arch=../.. (CA key)
- render endpoint traversal; machine-reported MACs used as file paths
- delegated intermediates could mint admin/rack identities and IPs;
identities now require root-issued certs, intermediates are
ServerAuth-only with both name types constrained
- rack certs were ServerAuth for rack-chosen SANs: now client-only;
racks accept only the region's own cert; join checks the CA
fingerprint; unbound join codes cannot take an existing name; job
results bound to the rack that took them
- rack self-update: region signs binaries with the CA key, the root
unit verifies against the join-time CA in a root-owned dir
- dnsmasq reads a checked, root-owned copy limited to generated options
- commissioning could overwrite deployed machines; MAC takeover of PXE
- injection into dnsmasq/cloud-init/PXE/udev/DNS via names and
machine-reported strings; stored XSS in the UI; CSP and cross-origin
protection; deploy agent shell injection and seed traversal
- Redfish requests confined to the BMC's /redfish/ tree
- admin_http defaults to loopback; body limits and timeouts; secrets
off command lines
Also: IPAM claim/reconcile race, sync stopping on one error, watchdog
race, release wiping the wrong disk, SAN LUNs as auto targets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cloud-init's ca_certs ran update-ca-certificates on Rocky and failed, so
those machines never trusted the Rollcall CA. The CA is now written into
both trust-anchor directories and a bootcmd rebuilds the store with
update-ca-trust or update-ca-certificates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pxeBoot dropped the errors of its BMC calls, so a command lost while the
rack or region restarted produced an empty, success-looking event and the
job waited forever. Errors now go into the event (pxe boot failed), and a
watchdog fails jobs the machine has not started within job_start_timeout
(default 30m: real servers take 4-5 minutes to POST, longer in bulk).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every address Rollcall hands out is registered in the configured authority
first; addresses others hold there are skipped. Rollcall's records carry the
tag `rollcall` and the machine ID in the `rollcall_machine` custom field;
only those are changed or deleted. Releases trigger a reconcile that also
runs periodically. dns_name per address feeds the site's DNS sync. Delegated
prefixes can be listed and imported as subnets. `rollcall ipam`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CA (llmgw-style)
- region creates its CA (ECDSA P-256) and serves API/UI/images over HTTPS on tls_listen with a
certificate for every name/address it has (reissued on change or a month before expiry);
admin_http policy (allow|loopback|deny) for the admin API over plain HTTP; /ca.pem
- rack controllers enroll with one-time join codes (CA fingerprint + secret + expiry + address):
rollcall controller join-code, rollcall rack join; mutual TLS from then on, identity from the
certificate (CN rack:NAME, serial on record), removal revokes, renewal a month before expiry;
the shared rack_secret stays as a legacy fallback
- clients: CA from the server config, ROLLCALL_CA, or trusted on first connect in the console after
showing the fingerprint (kept in client.json); deployed machines trust it via cloud-init ca_certs
(allowed on RHEL-family images), overlays get .CA; rollcall ca show|issue
Power drivers
- ipmi (cipher/privilege/interface), redfish (system discovery, ResetType from AllowableValues or
ActionInfo, boot override with If-Match), idrac, ilo, supermicro, xcc, openbmc, proxmox (API
token), manual; executed on the rack controller; Redfish auto-detection after commissioning;
operator settings survive recommissioning; rollcall bmc, BMC editing in web UI and console
- lab: pve_redfish.py Redfish emulator per Proxmox VM (iDRAC, iLO, generic profiles)
IPv6
- IPv6 subnets and IPAM (netip), dual-stack default layout (all subnets of the boot VLAN),
per-family default routes, accept-ra off where addresses are planned, dnsmasq RA/DHCPv6 with
router lifetime 0 unless the rack is the gateway, AAAA host records from IPAM, IPv6 listen
addresses and URLs, rack interfaces and region certificate with IPv6, agent reports IPv6
Tests for CA/join/pinning/CSR signing, Redfish (iDRAC, iLO fallbacks, probe), Proxmox, manual,
IPv6 candidates, dual-stack plans and dnsmasq IPv6.
Verified in the lab: rack2 enrolled and running on mutual TLS only, revocation and single-use
codes; iLO auto-detected through rack2, iDRAC and Proxmox drivers, wrong password not saved;
Debian 13 (bond+VLAN), Rocky 9 and AlmaLinux 9 dual-stack with Redfish-driven deployments; CA
trusted on Debian and AlmaLinux; admin CLI over HTTPS on IPv4 and IPv6.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- fabrics, VLANs, subnets, IP ranges (dynamic/reserved/static), IPAM with adoption of
addresses machines already have; rollcall fabric|vlan|subnet|range|ip
- interface layouts (physical/bond/vlan, NICs by selector: boot, name, mac, fastest:N, ib:N)
per machine or per tag; deployment allocates addresses and writes cloud-init network-config
plus udev name rules (RHEL cloud images boot with net.ifnames=0); release frees them;
rollcall layout, rollcall net show|set
- region/rack: rollcall rack long-polls its desired state (per-MAC PXE files kept in the
database, dnsmasq configuration and host files) and a BMC command queue, caches boot and OS
images (kernels/initrds prefetched for TFTP), proxies the node API and tags requests with its
name; the region runs an embedded rack controller; install-rack.sh, rollcall-rack.service
- dnsmasq configuration generated per rack from the VLANs it serves (split pool for two racks);
reload script restarts only on change and restores the last working config if dnsmasq does
not come back (drilled in the lab)
- web: Network page, per-machine network tab, layout action, tag layout; console: Network page,
network section in the machine view, layout action
- tests for candidates, plans (default, bond+VLAN, pinned, DHCP fallback), network-config,
udev rules and dnsmasq rendering
- lab: pve-vbmc address script finds the interface per BMC address
Verified in the lab: embedded rack (10.213.77.0/24) and a remote rack controller on a second
SDN segment (10.213.78.0/24): enlistment, commissioning, BMC through the rack's queue, image
cache, deployment and phone-home through the rack; Debian 13, Ubuntu 24.04 and Rocky 9 with a
static boot NIC and with an active-backup bond carrying VLAN 200.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- tags table + machines.tags/auto_tags; match rules as Go templates over .Node
(GPUs, IB, CPUs, memory, disks, NVMe, PCI vendors, manufacturer/product, OS)
re-evaluated at commissioning and on tag changes
- deployment: tag cloud-config fragments appended as MIME parts with additive merge;
deploy/tags/gpu-nvidia.yaml installs the NVIDIA driver per distro family
- stateless: tag overlays appended to the profile's overlays
- CLI: tag, tags, list -tag, deploy/power tag:NAME fan-out; UI: tags column, per-machine
editor, tag definitions panel; API: /api/v1/tags, /machines/{id}/tags, ?tag=
- CHANGELOG.md; release workflow publishes the tag's section as release notes and
refuses tags without one
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>