Network model, IPAM, interface layouts and the region/rack split (MAAS-style)
- fabrics, VLANs, subnets, IP ranges (dynamic/reserved/static), IPAM with adoption of addresses machines already have; rollcall fabric|vlan|subnet|range|ip - interface layouts (physical/bond/vlan, NICs by selector: boot, name, mac, fastest:N, ib:N) per machine or per tag; deployment allocates addresses and writes cloud-init network-config plus udev name rules (RHEL cloud images boot with net.ifnames=0); release frees them; rollcall layout, rollcall net show|set - region/rack: rollcall rack long-polls its desired state (per-MAC PXE files kept in the database, dnsmasq configuration and host files) and a BMC command queue, caches boot and OS images (kernels/initrds prefetched for TFTP), proxies the node API and tags requests with its name; the region runs an embedded rack controller; install-rack.sh, rollcall-rack.service - dnsmasq configuration generated per rack from the VLANs it serves (split pool for two racks); reload script restarts only on change and restores the last working config if dnsmasq does not come back (drilled in the lab) - web: Network page, per-machine network tab, layout action, tag layout; console: Network page, network section in the machine view, layout action - tests for candidates, plans (default, bond+VLAN, pinned, DHCP fallback), network-config, udev rules and dnsmasq rendering - lab: pve-vbmc address script finds the interface per BMC address Verified in the lab: embedded rack (10.213.77.0/24) and a remote rack controller on a second SDN segment (10.213.78.0/24): enlistment, commissioning, BMC through the rack's queue, image cache, deployment and phone-home through the rack; Debian 13, Ubuntu 24.04 and Rocky 9 with a static boot NIC and with an active-backup bond carrying VLAN 200. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
e3d9373201
commit
736a7e38a3
31 files changed
+4357
-176
No files matched your search
@@ -44,7 +44,7 @@ unknown server ──PXE──▶ dnsmasq (DHCP+TFTP) ──▶ pxelinux default
|
||||
|---|---|
|
||||
| `cmd/rollcall/` | Go server + CLI + embedded web UI (`web/`) |
|
||||
| `agent/` | `rollcall-agent.py` (runs inside the live image), its systemd unit, `build-image.sh` (live-build recipe) |
|
||||
| `deploy/` | `install.sh`, `rollcall.service`, `config.example.json`, `pxelinux-default`, `os-sources.json`, sync/dnsmasq/controller units, `overlays/` |
|
||||
| `deploy/` | `install.sh` (region), `install-rack.sh` + `rack.example.json` (rack controller), units, `config.example.json`, `os-sources.json`, `tags/`, `overlays/` |
|
||||
| `images/` | stateless root image recipes (`build-rootfs.sh`, `compute-debian13/`) |
|
||||
| `lab/` | Proxmox test lab: `pve-vbmc` (virtual IPMI BMC for VMs backed by the Proxmox API), cloud-init for the PXE server VM |
|
||||
| `site/` | Project website (static) |
|
||||
@@ -295,6 +295,114 @@ tag rules see `.Node.Building`, `.Node.Room`, `.Node.Rack`, `.Node.Unit`, so a
|
||||
rack can drive a tag (`{{ eq .Node.Rack "R12" }}`) or a Slurm `topology.conf`
|
||||
overlay.
|
||||
|
||||
## Network and rack controllers
|
||||
|
||||
### Model
|
||||
|
||||
As in MAAS: **fabric → VLAN → subnet → IP ranges**.
|
||||
|
||||
* A **fabric** is a switched layer-2 domain (`ethernet` or `infiniband`).
|
||||
* A **VLAN** has a VID (0 = untagged; a partition key on IB), an MTU, and
|
||||
says whether rollcall serves DHCP/PXE on it and which **rack controller**
|
||||
does (a second one splits the dynamic pool with it).
|
||||
* A **subnet** has a purpose (provisioning, mgmt, bmc, storage, ib, other), a
|
||||
gateway, DNS servers and a domain.
|
||||
* **Ranges** inside a subnet are `dynamic` (the DHCP pool PXE and
|
||||
commissioning use), `reserved` (never handed out: routers, switches,
|
||||
controllers) and `static` (where machines get their addresses; without one,
|
||||
anything outside dynamic/reserved).
|
||||
* **IPAM** records every address handed out (`rollcall ip list`), pinned ones
|
||||
(`rollcall net set ID -static ens18=10.0.0.60`) and reserved single
|
||||
addresses (`rollcall ip reserve 10.0.0.5 -name switch-1`). Addresses a machine
|
||||
already had are adopted when its subnet is defined.
|
||||
|
||||
```sh
|
||||
rollcall fabric add hall-a
|
||||
rollcall subnet add 10.50.0.0/24 -fabric hall-a -vid 0 -purpose provisioning -gateway 10.50.0.1 -dns 10.50.0.1 -domain dc.example
|
||||
rollcall range add 10.50.0.0/24 reserved 10.50.0.1 10.50.0.20
|
||||
rollcall range add 10.50.0.0/24 static 10.50.0.50 10.50.0.199
|
||||
rollcall range add 10.50.0.0/24 dynamic 10.50.0.200 10.50.0.250
|
||||
rollcall vlan add hall-a 0 -dhcp -rack hall-a-rack1 # this rack controller serves DHCP/PXE here
|
||||
rollcall subnet add 10.60.0.0/24 -fabric hall-a -vid 200 -purpose storage
|
||||
rollcall dhcp hall-a-rack1 # the dnsmasq configuration it generates
|
||||
```
|
||||
|
||||
### Machine interfaces
|
||||
|
||||
A **layout** describes interfaces in terms that fit many machines: NICs are
|
||||
chosen by selectors (`boot`, `name:X`, `mac:X`, `fastest:N`, `ib:N`, `all`),
|
||||
not by MAC. Interfaces are `physical`, `bond` (802.3ad by default) or `vlan`;
|
||||
each link names a subnet (CIDR, name, or `boot`) and a mode: `auto` (an address
|
||||
from the subnet), `static` (a given one), `dhcp` or `link_up`.
|
||||
|
||||
```json
|
||||
{"interfaces": [
|
||||
{"type": "physical", "select": "boot", "links": [{"subnet": "boot", "mode": "auto", "default_gateway": true}]},
|
||||
{"type": "bond", "name": "bond0", "members": ["fastest:2"], "mtu": 9000, "links": [{"mode": "link_up"}]},
|
||||
{"type": "vlan", "parent": "bond0", "vid": 200, "links": [{"subnet": "10.60.0.0/24", "mode": "auto"}]},
|
||||
{"type": "physical", "select": "ib:0", "links": [{"subnet": "ib0", "mode": "auto"}]}
|
||||
]}
|
||||
```
|
||||
|
||||
`rollcall layout add gpu-bond layout.json`, then either per machine
|
||||
(`rollcall net set ID -layout gpu-bond`) or per tag (`rollcall tags add gpu
|
||||
… -layout gpu-bond`). Without a layout the boot NIC gets an `auto` address on
|
||||
its subnet; without any subnet model, deployed machines use DHCP as before.
|
||||
|
||||
At deployment the layout is resolved against the machine's inventory,
|
||||
addresses are allocated, and two files go into the image next to the
|
||||
cloud-init seed: `network-config` (cloud-init v2; cloud-init renders it for
|
||||
netplan, NetworkManager or ifupdown) and a udev rule that pins each NIC's name
|
||||
to its MAC, because RHEL-family cloud images boot with `net.ifnames=0`.
|
||||
Release returns the allocated addresses. `rollcall net show ID` previews it all.
|
||||
|
||||
Verified in the lab: Debian 13 and Ubuntu 24.04 (netplan) and Rocky 9
|
||||
(NetworkManager), with a static boot NIC and with an active-backup bond
|
||||
carrying VLAN 200.
|
||||
|
||||
### Region and rack controllers
|
||||
|
||||
`rollcall serve` is the **region**: database, API, web UI, IPAM, image store.
|
||||
**Rack controllers** sit next to the machines, one or two per network segment
|
||||
(hall, building, site):
|
||||
|
||||
* they serve DHCP and TFTP with dnsmasq on the VLANs assigned to them; the
|
||||
configuration, per-MAC PXE files and host files are computed by the region;
|
||||
* they run BMC commands (BMC networks are rarely routed to the region);
|
||||
* they serve boot and OS images from a local cache, filled from the region on
|
||||
first use (kernels and initrds are prefetched, TFTP cannot wait);
|
||||
* they pass the node API (commissioning, deploy jobs, overlays, phone-home)
|
||||
through to the region and tell it which rack a machine came through.
|
||||
|
||||
A rack controller only needs to reach the region: it long-polls its desired
|
||||
state and a command queue. The region host runs an **embedded** rack
|
||||
controller (`rack_name`, default its hostname), which is all a single-server
|
||||
installation needs. Remote ones:
|
||||
|
||||
```sh
|
||||
# region: config.json gets "rack_secret": "<rollcall keygen>"
|
||||
# rack host (Debian/Ubuntu), from deploy/ with the binary next to it:
|
||||
cat > /etc/rollcall/rack.json <<EOF
|
||||
{"name": "hall-b", "region": "http://10.0.0.234", "secret": "<rack_secret>",
|
||||
"url": "http://10.213.78.1", "listen": "10.213.78.1:80"}
|
||||
EOF
|
||||
sh install-rack.sh # dnsmasq, syslinux, cache dirs, rollcall-rack.service
|
||||
rollcall controllers # on the region: it shows up online
|
||||
rollcall vlan add hall-b 0 -dhcp -rack hall-b
|
||||
```
|
||||
|
||||
dnsmasq is restarted only when its generated configuration changes, and only
|
||||
if it comes back; otherwise the last working configuration is restored (a
|
||||
hand-written DHCP configuration for the same interface is the usual cause:
|
||||
`install.sh` points it out). A machine that moved to another segment gets
|
||||
`rollcall net set ID -rack NAME`.
|
||||
|
||||
The lab runs both: `pxe-server` (embedded, 10.213.77.0/24) and `pxe-rack2`
|
||||
(VM 213, remote, 10.213.78.0/24 on SDN vnet `pxenet2`), with node002 behind
|
||||
pxe-rack2 and its virtual BMC on pxe-rack2. Rack ↔ region traffic is plain
|
||||
HTTP in the lab; BMC passwords travel on it for queued commands, so put the
|
||||
region behind TLS on real networks.
|
||||
|
||||
## Lab
|
||||
|
||||
`lab/` documents a Proxmox SDN test setup: an isolated vnet, a PXE server VM
|
||||
|
||||
Reference in new issue
Block a user