The region keeps every intermediate it signs: its name constraints,
certificate, the root that signed it and a renewal token (stored hashed,
printed once). `rollcall ca intermediate` goes through the API now (so it
works from any admin client) and writes NAME.token beside NAME.pem;
`rollcall ca intermediates` lists them.
The host with the intermediate's key runs deploy/rollcall-intermediate-
renew.sh daily (rollcall-intermediate-renew@NAME.timer): it posts a CSR
from the same key with the token to /api/v1/ca/intermediates/NAME/renew.
The region signs again in the last third of the certificate's life, or
at once when another root signs since a rotation's switch, always with
the constraints it was first given; otherwise 204. The script checks the
answer (a CA certificate for its key, from a root it holds), keeps the
old file, keeps the region's roots and the active root current from
/ca-bundle.pem, and reloads (RELOAD).
Attention: an intermediate under 30 days from its end, whose last renewal
failed or whose host has not asked for three days shows in rollcall
doctor, on the web UI's overview, and one from the old root in the
rotation status.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The CA directory holds the active root and, during a rotation, the next
or the previous one; everything that trusts the region gets the trust
bundle (all of them). Rack controllers receive it in their state, signed
by the active root; the root update unit installs it only when the
signature verifies against the roots it already trusts, and binary
signatures verify against any of them. After switch racks renew their
client and server certificates from the new root by themselves and
report what they run on; switch and retire wait for every rack (-force).
/ca.pem stays one certificate, the root of that server's certificate, and
the PXE command line names that root, so agents in older images keep
working; /ca-bundle.pem is the bundle. Deployed machines get one file per
root and a daily timer that refreshes them from their rack over HTTPS.
Refs #12
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The image job built as root and uploaded as the deploy user, so
publish-image.sh could not write SHA256SUMS (v0.3.0's image job failed).
The job hands the directory over after the build, and publish-image.sh
checks the SHA256SUMS build-image.sh wrote instead of rewriting it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UEFI machines were given syslinux.efi, which is unsigned: with Secure
Boot on (the default on much server hardware) the firmware refuses it
("Access Denied", seen on an OVMF lab VM), so such a machine could not
PXE-boot at all. They now get Debian's signed shim (bootx64.efi), which
loads the signed network GRUB (grubx64.efi); GRUB reads
grub/grub.cfg-<mac>, else grub/grub.cfg-default, and loads kernels
only as shim verifies them. Rack controllers write the GRUB menus from
the pxelinux menus they already get (same entries: kernel and initrd,
local disk via exit, power off via halt), so both always agree; BIOS
machines keep pxelinux. install.sh, install-rack.sh and bootstrap.sh
install shim-signed and grub-efi-amd64-signed and lay out the files.
Secure Boot covers the kernel; the initrd and the squashfs it fetches
are not covered by it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Renewal could not work: rack join writes the certificate and key as
root, and the rack (user rollcall) failed to overwrite them. A lost
renewal answer also locked the rack out, since the region accepted only
the new serial.
- the renewed certificate and key go into one file in
/var/lib/rollcall/rack-cert (cert_state_dir), written atomically, and
are preferred at start when they are this rack's, from the CA, newer
- after a renewal the region accepts the previous serial for 7 days; a
new join does not keep it
- rollcall rack renew renews now (run as the service user)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found by the first signed update on pxe-rack2: the signature verified,
then the copy in /run could not be executed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Findings from a full review (auth/API, CA and rack channel, config
generation, integrations, UI/agents/scripts), each verified in code:
- rack binary endpoint read any region file via arch=../.. (CA key)
- render endpoint traversal; machine-reported MACs used as file paths
- delegated intermediates could mint admin/rack identities and IPs;
identities now require root-issued certs, intermediates are
ServerAuth-only with both name types constrained
- rack certs were ServerAuth for rack-chosen SANs: now client-only;
racks accept only the region's own cert; join checks the CA
fingerprint; unbound join codes cannot take an existing name; job
results bound to the rack that took them
- rack self-update: region signs binaries with the CA key, the root
unit verifies against the join-time CA in a root-owned dir
- dnsmasq reads a checked, root-owned copy limited to generated options
- commissioning could overwrite deployed machines; MAC takeover of PXE
- injection into dnsmasq/cloud-init/PXE/udev/DNS via names and
machine-reported strings; stored XSS in the UI; CSP and cross-origin
protection; deploy agent shell injection and seed traversal
- Redfish requests confined to the BMC's /redfish/ tree
- admin_http defaults to loopback; body limits and timeouts; secrets
off command lines
Also: IPAM claim/reconcile race, sync stopping on one error, watchdog
race, release wiping the wrong disk, SAN LUNs as auto targets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A VLAN in proxy mode leaves addresses to the site's DHCP server; the rack
only answers PXE (dnsmasq proxy range + pxe-service), machines keep DHCP
unless a static range is set aside. CLI -proxy-dhcp, web UI mode select,
bootstrap --proxy-dhcp, doctor/discover aware.
dnsmasq 2.91 (and 2.93) loops forever in its proxy path when the proxy
range carries a set: tag: option_filter appends a request's tags to the
context's persistent list, so the next request's stack interface tag
points at itself (found with gdb on the lab rack). Proxy ranges are
untagged and PXE services limited by interface; dhcp-boot/dhcp-match only
for served subnets; no dhcp-host lines for proxied ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
INSTALL.md follows a clean run from day 0 to a deployed machine.
Release assets now include rollcall-<tag>-linux-<arch>.tar.gz (binary,
deploy/, agent/). deploy/publish-image.sh uploads a region's commissioning
image to a release (commissioning-*); bootstrap.sh downloads either layout,
with IMAGE_TOKEN for private repositories. bootstrap: --ssh-key (default:
the operator's keys), keeps server_url on reruns without --iface.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bootstrap.sh no longer needs the provisioning network: it installs and
starts the region on the management address; --iface later models the
subnet, serves DHCP and moves server_url. rollcall net discover shows what
each interface sees (LLDP, DHCP servers via a parallel probe, neighbours)
and suggests the next step. commission_append follows server_url. Doctor
treats a host that serves no DHCP yet as day 0. Fixes found on a clean VM:
os-release clobbered the rack name, install.sh not executable, slow DHCP
offers missed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bootstrap.sh sets up a region on a fresh Debian 13 host (packages,
PostgreSQL, TFTP tree, config, services, verified commissioning image,
provisioning subnet with ranges/DHCP, optional NAT, first OS image).
rollcall doctor checks an installation and says what to fix, including a
DHCPDISCOVER probe for foreign DHCP servers. pxelinux-default no longer
hard-codes the lab address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The region offers its binary (version, arch, sha256) in each rack's state and
serves it over mutual TLS; the rack stages it after checking checksum and
version; a root path unit installs it, restarts the rack and rolls back if it
does not register within a minute.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New roots carry path length 1. `rollcall ca intermediate` signs a CSR (or a
generated key) as an intermediate limited to the permitted domains and
address ranges; a root with path length 0 refuses and `rollcall ca rotate`
replaces it. deploy/tags/acme-cert.yaml gets a node certificate from an
ACME server under such an intermediate and renews it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every address Rollcall hands out is registered in the configured authority
first; addresses others hold there are skipped. Rollcall's records carry the
tag `rollcall` and the machine ID in the `rollcall_machine` custom field;
only those are changed or deleted. Releases trigger a reconcile that also
runs periodically. dns_name per address feeds the site's DNS sync. Delegated
prefixes can be listed and imported as subnets. `rollcall ipam`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CA (llmgw-style)
- region creates its CA (ECDSA P-256) and serves API/UI/images over HTTPS on tls_listen with a
certificate for every name/address it has (reissued on change or a month before expiry);
admin_http policy (allow|loopback|deny) for the admin API over plain HTTP; /ca.pem
- rack controllers enroll with one-time join codes (CA fingerprint + secret + expiry + address):
rollcall controller join-code, rollcall rack join; mutual TLS from then on, identity from the
certificate (CN rack:NAME, serial on record), removal revokes, renewal a month before expiry;
the shared rack_secret stays as a legacy fallback
- clients: CA from the server config, ROLLCALL_CA, or trusted on first connect in the console after
showing the fingerprint (kept in client.json); deployed machines trust it via cloud-init ca_certs
(allowed on RHEL-family images), overlays get .CA; rollcall ca show|issue
Power drivers
- ipmi (cipher/privilege/interface), redfish (system discovery, ResetType from AllowableValues or
ActionInfo, boot override with If-Match), idrac, ilo, supermicro, xcc, openbmc, proxmox (API
token), manual; executed on the rack controller; Redfish auto-detection after commissioning;
operator settings survive recommissioning; rollcall bmc, BMC editing in web UI and console
- lab: pve_redfish.py Redfish emulator per Proxmox VM (iDRAC, iLO, generic profiles)
IPv6
- IPv6 subnets and IPAM (netip), dual-stack default layout (all subnets of the boot VLAN),
per-family default routes, accept-ra off where addresses are planned, dnsmasq RA/DHCPv6 with
router lifetime 0 unless the rack is the gateway, AAAA host records from IPAM, IPv6 listen
addresses and URLs, rack interfaces and region certificate with IPv6, agent reports IPv6
Tests for CA/join/pinning/CSR signing, Redfish (iDRAC, iLO fallbacks, probe), Proxmox, manual,
IPv6 candidates, dual-stack plans and dnsmasq IPv6.
Verified in the lab: rack2 enrolled and running on mutual TLS only, revocation and single-use
codes; iLO auto-detected through rack2, iDRAC and Proxmox drivers, wrong password not saved;
Debian 13 (bond+VLAN), Rocky 9 and AlmaLinux 9 dual-stack with Redfish-driven deployments; CA
trusted on Debian and AlmaLinux; admin CLI over HTTPS on IPv4 and IPv6.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- fabrics, VLANs, subnets, IP ranges (dynamic/reserved/static), IPAM with adoption of
addresses machines already have; rollcall fabric|vlan|subnet|range|ip
- interface layouts (physical/bond/vlan, NICs by selector: boot, name, mac, fastest:N, ib:N)
per machine or per tag; deployment allocates addresses and writes cloud-init network-config
plus udev name rules (RHEL cloud images boot with net.ifnames=0); release frees them;
rollcall layout, rollcall net show|set
- region/rack: rollcall rack long-polls its desired state (per-MAC PXE files kept in the
database, dnsmasq configuration and host files) and a BMC command queue, caches boot and OS
images (kernels/initrds prefetched for TFTP), proxies the node API and tags requests with its
name; the region runs an embedded rack controller; install-rack.sh, rollcall-rack.service
- dnsmasq configuration generated per rack from the VLANs it serves (split pool for two racks);
reload script restarts only on change and restores the last working config if dnsmasq does
not come back (drilled in the lab)
- web: Network page, per-machine network tab, layout action, tag layout; console: Network page,
network section in the machine view, layout action
- tests for candidates, plans (default, bond+VLAN, pinned, DHCP fallback), network-config,
udev rules and dnsmasq rendering
- lab: pve-vbmc address script finds the interface per BMC address
Verified in the lab: embedded rack (10.213.77.0/24) and a remote rack controller on a second
SDN segment (10.213.78.0/24): enlistment, commissioning, BMC through the rack's queue, image
cache, deployment and phone-home through the rack; Debian 13, Ubuntu 24.04 and Rocky 9 with a
static boot NIC and with an active-backup bond carrying VLAN 200.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- tags table + machines.tags/auto_tags; match rules as Go templates over .Node
(GPUs, IB, CPUs, memory, disks, NVMe, PCI vendors, manufacturer/product, OS)
re-evaluated at commissioning and on tag changes
- deployment: tag cloud-config fragments appended as MIME parts with additive merge;
deploy/tags/gpu-nvidia.yaml installs the NVIDIA driver per distro family
- stateless: tag overlays appended to the profile's overlays
- CLI: tag, tags, list -tag, deploy/power tag:NAME fan-out; UI: tags column, per-machine
editor, tag definitions panel; API: /api/v1/tags, /machines/{id}/tags, ?tag=
- CHANGELOG.md; release workflow publishes the tag's section as release notes and
refuses tags without one
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Profiles bind a RAM-booted root image to overlay templates; assigning a
machine allocates a node name and management IP, writes the per-MAC PXE
entry with a per-node token and exports static leases for dnsmasq. Nodes
pull rendered overlays (hosts, slurm.conf from inventory, gres.conf, SSH
host key, munge key) through rollcall-overlay.service; the runtime refresh
timer is the heartbeat. Adds the compute-debian13 image definition and a
generic live-build root image script, CLI and web UI for profiles/assign.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Go server (API, embedded web UI, CLI) backed by PostgreSQL with AES-GCM
sealed BMC secrets; Debian live commissioning agent and live-build recipe;
Proxmox lab with pve-vbmc virtual BMCs; Gitea workflows for CI, releases
and deployment; project website.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
