Commit Graph
12 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 b01d8e2b4d Firmware updates over Redfish, per vendor
CI / agent (push) Successful in 3s
CI / site (push) Successful in 2s
CI / go (push) Successful in 39s
A firmware repository on the region (firmware_dir, served under
/firmware/<id>/<file>, cached by rack controllers for BMCs to pull),
each machine's firmware inventory (FirmwareInventory, classified into
bios/bmc/nic/raid/cpld/psu/drive), and update jobs (kind "firmware")
that the region drives with short rack commands: fw-start hands the
package over, fw-poll watches the task (tolerating a BMC that restarts),
fw-finish releases what was claimed, fw-inventory verifies the version.
Running updates resume after a region restart; they cannot be cancelled
once the BMC has the package.

Standard methods (multipart, SimpleUpdate pull, HttpPushUri) with known
Content-Length, plus vendor plans from their documentation or their own
tools: Dell MultipartUpload (OnReset), HPE SimpleUpdate with .fwpkg
extracted at import, Lenovo /mfwupdate with HttpPushUriTargetsBusy,
Supermicro /upload with targets and preserve options, GIGABYTE/AMI/
ASRock Rack /upload with OemParameters ImageType, OpenBMC
update-multipart, xFusion upload to /tmp/web then SimpleUpdate, KAYTUS
upload then OEM SimpleUpdate with fixed tasks, Fujitsu iRMC OEM actions;
Cisco CIMC is refused clearly (SCP only). fw_params and fw_method on
the BMC record override a plan.

CLI rollcall firmware list|import|rm|show|update, API, a Firmware tab
and a repository card in the UI. The lab Redfish emulator gains an
UpdateService (BMC applies at once, BIOS at the next reset).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 19:22:01 +01:00
Ilker ManapandClaude Opus 5.5 a549f01ef5 Security review: fix traversal, identity, injection and takeover paths
CI / agent (push) Successful in 3s
CI / site (push) Successful in 2s
CI / go (push) Successful in 34s
Findings from a full review (auth/API, CA and rack channel, config
generation, integrations, UI/agents/scripts), each verified in code:

- rack binary endpoint read any region file via arch=../.. (CA key)
- render endpoint traversal; machine-reported MACs used as file paths
- delegated intermediates could mint admin/rack identities and IPs;
  identities now require root-issued certs, intermediates are
  ServerAuth-only with both name types constrained
- rack certs were ServerAuth for rack-chosen SANs: now client-only;
  racks accept only the region's own cert; join checks the CA
  fingerprint; unbound join codes cannot take an existing name; job
  results bound to the rack that took them
- rack self-update: region signs binaries with the CA key, the root
  unit verifies against the join-time CA in a root-owned dir
- dnsmasq reads a checked, root-owned copy limited to generated options
- commissioning could overwrite deployed machines; MAC takeover of PXE
- injection into dnsmasq/cloud-init/PXE/udev/DNS via names and
  machine-reported strings; stored XSS in the UI; CSP and cross-origin
  protection; deploy agent shell injection and seed traversal
- Redfish requests confined to the BMC's /redfish/ tree
- admin_http defaults to loopback; body limits and timeouts; secrets
  off command lines

Also: IPAM claim/reconcile race, sync stopping on one error, watchdog
race, release wiping the wrong disk, SAN LUNs as auto targets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 17:58:42 +01:00
Ilker ManapandClaude Opus 5.5 0c6f6557db cloud-init final_message: no $hostname (not expanded there; warned on every deploy)
CI / agent (push) Successful in 3s
CI / site (push) Successful in 2s
CI / go (push) Successful in 36s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 20:01:11 +01:00
Ilker ManapandClaude Opus 5.5 e2102e1741 Deployments trust the CA on RHEL-family systems too
CI / agent (push) Successful in 3s
CI / site (push) Successful in 2s
CI / go (push) Successful in 36s
cloud-init's ca_certs ran update-ca-certificates on Rocky and failed, so
those machines never trusted the Rollcall CA. The CA is now written into
both trust-anchor directories and a bootcmd rebuilds the store with
update-ca-trust or update-ca-certificates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 19:38:46 +01:00
Ilker ManapandClaude Opus 5.5 996ecbd89d Lost power commands are visible; jobs not started in job_start_timeout fail
CI / agent (push) Successful in 3s
CI / site (push) Successful in 2s
CI / go (push) Successful in 37s
pxeBoot dropped the errors of its BMC calls, so a command lost while the
rack or region restarted produced an empty, success-looking event and the
job waited forever. Errors now go into the event (pxe boot failed), and a
watchdog fails jobs the machine has not started within job_start_timeout
(default 30m: real servers take 4-5 minutes to POST, longer in bulk).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 19:23:38 +01:00
Ilker ManapandClaude Opus 5.5 2706fd65e7 External IPAM authority: NetBox driver, ownership, reconcile
CI / agent (push) Successful in 2s
CI / site (push) Successful in 2s
CI / go (push) Successful in 35s
Every address Rollcall hands out is registered in the configured authority
first; addresses others hold there are skipped. Rollcall's records carry the
tag `rollcall` and the machine ID in the `rollcall_machine` custom field;
only those are changed or deleted. Releases trigger a reconcile that also
runs periodically. dns_name per address feeds the site's DNS sync. Delegated
prefixes can be listed and imported as subnets. `rollcall ipam`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 09:35:21 +01:00
Ilker ManapandClaude Opus 5.5 c8465c235a Region as certificate authority, BMC power drivers, IPv6
CI / agent (push) Successful in 2s
CI / site (push) Successful in 3s
CI / go (push) Successful in 34s
CA (llmgw-style)
- region creates its CA (ECDSA P-256) and serves API/UI/images over HTTPS on tls_listen with a
  certificate for every name/address it has (reissued on change or a month before expiry);
  admin_http policy (allow|loopback|deny) for the admin API over plain HTTP; /ca.pem
- rack controllers enroll with one-time join codes (CA fingerprint + secret + expiry + address):
  rollcall controller join-code, rollcall rack join; mutual TLS from then on, identity from the
  certificate (CN rack:NAME, serial on record), removal revokes, renewal a month before expiry;
  the shared rack_secret stays as a legacy fallback
- clients: CA from the server config, ROLLCALL_CA, or trusted on first connect in the console after
  showing the fingerprint (kept in client.json); deployed machines trust it via cloud-init ca_certs
  (allowed on RHEL-family images), overlays get .CA; rollcall ca show|issue

Power drivers
- ipmi (cipher/privilege/interface), redfish (system discovery, ResetType from AllowableValues or
  ActionInfo, boot override with If-Match), idrac, ilo, supermicro, xcc, openbmc, proxmox (API
  token), manual; executed on the rack controller; Redfish auto-detection after commissioning;
  operator settings survive recommissioning; rollcall bmc, BMC editing in web UI and console
- lab: pve_redfish.py Redfish emulator per Proxmox VM (iDRAC, iLO, generic profiles)

IPv6
- IPv6 subnets and IPAM (netip), dual-stack default layout (all subnets of the boot VLAN),
  per-family default routes, accept-ra off where addresses are planned, dnsmasq RA/DHCPv6 with
  router lifetime 0 unless the rack is the gateway, AAAA host records from IPAM, IPv6 listen
  addresses and URLs, rack interfaces and region certificate with IPv6, agent reports IPv6

Tests for CA/join/pinning/CSR signing, Redfish (iDRAC, iLO fallbacks, probe), Proxmox, manual,
IPv6 candidates, dual-stack plans and dnsmasq IPv6.

Verified in the lab: rack2 enrolled and running on mutual TLS only, revocation and single-use
codes; iLO auto-detected through rack2, iDRAC and Proxmox drivers, wrong password not saved;
Debian 13 (bond+VLAN), Rocky 9 and AlmaLinux 9 dual-stack with Redfish-driven deployments; CA
trusted on Debian and AlmaLinux; admin CLI over HTTPS on IPv4 and IPv6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 07:44:13 +01:00
Ilker ManapandClaude Opus 5.5 736a7e38a3 Network model, IPAM, interface layouts and the region/rack split (MAAS-style)
CI / agent (push) Successful in 3s
CI / go (push) Successful in 32s
CI / site (push) Successful in 2s
- fabrics, VLANs, subnets, IP ranges (dynamic/reserved/static), IPAM with adoption of
  addresses machines already have; rollcall fabric|vlan|subnet|range|ip
- interface layouts (physical/bond/vlan, NICs by selector: boot, name, mac, fastest:N, ib:N)
  per machine or per tag; deployment allocates addresses and writes cloud-init network-config
  plus udev name rules (RHEL cloud images boot with net.ifnames=0); release frees them;
  rollcall layout, rollcall net show|set
- region/rack: rollcall rack long-polls its desired state (per-MAC PXE files kept in the
  database, dnsmasq configuration and host files) and a BMC command queue, caches boot and OS
  images (kernels/initrds prefetched for TFTP), proxies the node API and tags requests with its
  name; the region runs an embedded rack controller; install-rack.sh, rollcall-rack.service
- dnsmasq configuration generated per rack from the VLANs it serves (split pool for two racks);
  reload script restarts only on change and restores the last working config if dnsmasq does
  not come back (drilled in the lab)
- web: Network page, per-machine network tab, layout action, tag layout; console: Network page,
  network section in the machine view, layout action
- tests for candidates, plans (default, bond+VLAN, pinned, DHCP fallback), network-config,
  udev rules and dnsmasq rendering
- lab: pve-vbmc address script finds the interface per BMC address

Verified in the lab: embedded rack (10.213.77.0/24) and a remote rack controller on a second
SDN segment (10.213.78.0/24): enlistment, commissioning, BMC through the rack's queue, image
cache, deployment and phone-home through the rack; Debian 13, Ubuntu 24.04 and Rocky 9 with a
static boot NIC and with an active-backup bond carrying VLAN 200.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 23:58:40 +01:00
Ilker ManapandClaude Fable 5.1 0333c1e486 Web UI redesigned for datacenter fleets
CI / agent (push) Successful in 3s
CI / site (push) Successful in 2s
CI / go (push) Successful in 27s
Sidebar navigation (Overview, Machines, Jobs, Tags, Profiles, OS images); machine list with
state/tag chips, search, sort, paging and multi-select bulk actions (power, deploy, assign,
tag, re-commission, release, boot device); machine detail with tabs (overview, hardware,
network, BMC, jobs, events, raw) and dialog-driven actions; fleet-wide /api/v1/events and
/api/v1/jobs; ?token= sign-in; GPU count in summaries counts real accelerators only;
OS catalog ignores dot-directories.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 14:48:29 +01:00
Ilker ManapandClaude Fable 5.1 0f798d8797 Machine tags (MAAS-style), changelog-driven release notes
CI / agent (push) Successful in 3s
CI / site (push) Successful in 2s
CI / go (push) Successful in 26s
- tags table + machines.tags/auto_tags; match rules as Go templates over .Node
  (GPUs, IB, CPUs, memory, disks, NVMe, PCI vendors, manufacturer/product, OS)
  re-evaluated at commissioning and on tag changes
- deployment: tag cloud-config fragments appended as MIME parts with additive merge;
  deploy/tags/gpu-nvidia.yaml installs the NVIDIA driver per distro family
- stateless: tag overlays appended to the profile's overlays
- CLI: tag, tags, list -tag, deploy/power tag:NAME fan-out; UI: tags column, per-machine
  editor, tag definitions panel; API: /api/v1/tags, /machines/{id}/tags, ?tag=
- CHANGELOG.md; release workflow publishes the tag's section as release notes and
  refuses tags without one

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 11:32:02 +01:00
Ilker ManapandClaude Fable 5.1 ee3a1a4787 deploy: cancelled job restores the previous machine state; site: deployment features
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 10:13:16 +01:00
Ilker ManapandClaude Fable 5.1 fad4e1c8bd OS deployment: image catalog with upstream sync, deploy/release jobs, agent deploy mode, UI and CLI
- rollcall os list|sync|import|promote|rm: <os_dir>/<name>/<version>/{disk.raw.zst,meta.json},
  current symlink, vendor checksum verification (GNU/BSD formats, sha256/sha512), qemu-img -> raw,
  zstd in-process, keep-N pruning, nightly rollcall-os-sync.timer, deploy/os-sources.json
  (Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9, CentOS Stream 9)
- jobs table + API: deploy (disk pick, hostname/IP allocation, job token, per-MAC deploy PXE
  entry), release (wipe), progress, cancel, cloud-init phone_home
- agent/rollcall-deploy.py: streamed write with sha256 + pv progress, NoCloud seed, SELinux
  autorelabel, efibootmgr on UEFI; commissioning image gains zstd/parted/xfsprogs/...
- web UI: deploy/release controls, deployment card with progress, OS catalog table
- README: deployment section, image freshness policy

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 10:10:39 +01:00