OS deployment: image catalog with upstream sync, deploy/release jobs, agent deploy mode, UI and CLI

- rollcall os list|sync|import|promote|rm: <os_dir>/<name>/<version>/{disk.raw.zst,meta.json},
  current symlink, vendor checksum verification (GNU/BSD formats, sha256/sha512), qemu-img -> raw,
  zstd in-process, keep-N pruning, nightly rollcall-os-sync.timer, deploy/os-sources.json
  (Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9, CentOS Stream 9)
- jobs table + API: deploy (disk pick, hostname/IP allocation, job token, per-MAC deploy PXE
  entry), release (wipe), progress, cancel, cloud-init phone_home
- agent/rollcall-deploy.py: streamed write with sha256 + pv progress, NoCloud seed, SELinux
  autorelabel, efibootmgr on UEFI; commissioning image gains zstd/parted/xfsprogs/...
- web UI: deploy/release controls, deployment card with progress, OS catalog table
- README: deployment section, image freshness policy

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Fable 5.1 committed 2026-10-27 10:10:39 +01:00
1 parent 1452b79218
commit fad4e1c8bd
24 files changed
+2194 -26

No files matched your search

+73 -4
View File
@@ -44,7 +44,8 @@ unknown server ──PXE──▶ dnsmasq (DHCP+TFTP) ──▶ pxelinux default
|---|---| |---|---|
| `cmd/rollcall/` | Go server + CLI + embedded web UI (`web/`) | | `cmd/rollcall/` | Go server + CLI + embedded web UI (`web/`) |
| `agent/` | `rollcall-agent.py` (runs inside the live image), its systemd unit, `build-image.sh` (live-build recipe) | | `agent/` | `rollcall-agent.py` (runs inside the live image), its systemd unit, `build-image.sh` (live-build recipe) |
| `deploy/` | `install.sh`, `rollcall.service`, `config.example.json`, `pxelinux-default` | | `deploy/` | `install.sh`, `rollcall.service`, `config.example.json`, `pxelinux-default`, `os-sources.json`, sync/dnsmasq/controller units, `overlays/` |
| `images/` | stateless root image recipes (`build-rootfs.sh`, `compute-debian13/`) |
| `lab/` | Proxmox test lab: `pve-vbmc` (virtual IPMI BMC for VMs backed by the Proxmox API), cloud-init for the PXE server VM | | `lab/` | Proxmox test lab: `pve-vbmc` (virtual IPMI BMC for VMs backed by the Proxmox API), cloud-init for the PXE server VM |
| `site/` | Project website (static) | | `site/` | Project website (static) |
| `.gitea/workflows/` | CI (fmt/vet/build/tests), release on tags, deploy on push to `main` | | `.gitea/workflows/` | CI (fmt/vet/build/tests), release on tags, deploy on push to `main` |
@@ -97,6 +98,15 @@ Reads `ROLLCALL_URL` / `ROLLCALL_TOKEN`, or the config file when readable.
| POST | `/api/v1/machines/{id}/bootdev` | admin | `{"device":"pxe"}` | | POST | `/api/v1/machines/{id}/bootdev` | admin | `{"device":"pxe"}` |
| POST | `/api/v1/machines/{id}/recommission[?now=1]` | admin | boot the image again on next PXE (`now=1`: via BMC immediately) | | POST | `/api/v1/machines/{id}/recommission[?now=1]` | admin | boot the image again on next PXE (`now=1`: via BMC immediately) |
| DELETE | `/api/v1/machines/{id}` | admin | forget the machine | | DELETE | `/api/v1/machines/{id}` | admin | forget the machine |
| GET/POST/DELETE | `/api/v1/profiles[/{name}]` | admin | stateless profiles |
| POST | `/api/v1/machines/{id}/assign` `/unassign` | admin | stateless node assignment |
| GET | `/api/v1/os` | admin | OS image catalog |
| POST | `/api/v1/machines/{id}/deploy` | admin | `{"os":"rocky-9","disk":"/dev/sda","hostname":"","package_upgrade":false,"boot":true}` |
| POST | `/api/v1/machines/{id}/release` | admin | wipe job |
| GET / POST | `/api/v1/machines/{id}/jobs[/cancel]` | admin | job history / cancel |
| GET | `/api/v1/jobs/self`, POST `…/progress` | job token (`X-Rollcall-Job`) | deploy agent |
| POST | `/api/v1/phone-home/{token}` | job token in path | cloud-init phone_home |
| GET | `/os/*` | none | OS disk images |
| GET | `/rollcall/*` | none | image files (kernel, initrd, squashfs) | | GET | `/rollcall/*` | none | image files (kernel, initrd, squashfs) |
| GET | `/` | token in browser | web UI | | GET | `/` | token in browser | web UI |
@@ -157,6 +167,64 @@ dress it with per-node files, instead of installing an OS on disk.
States: `ready → stateless → booted`; `rollcall unassign` releases name and IPs. States: `ready → stateless → booted`; `rollcall unassign` releases name and IPs.
## OS deployment (install to disk)
For nodes that should run an operating system from their own disk, Rollcall
writes a vendor cloud image onto the disk and lets cloud-init do first-boot
configuration. No installer, no kickstart/preseed: a deployment is a streamed
disk write and takes about as long as the image download.
```
rollcall deploy <id> rocky-9 -boot -watch
│ server: picks the disk (smallest that fits, or -disk), allocates hostname + IP,
│ creates a job with a one-time token, writes pxelinux.cfg/01-<mac>
│ ("deploy" entry = commissioning image with rollcall.mode=deploy), power-cycles via BMC
▼
commissioning image, deploy mode (agent/rollcall-deploy.py)
GET /api/v1/jobs/self → image URL, sha256, disk, rendered cloud-init seed
curl | tee sha256 | pv | zstd -d | dd of=/dev/sda (progress every 3 s)
mount root fs → /var/lib/cloud/seed/nocloud/{meta-data,user-data}, datasource_list: [NoCloud, None]
(SELinux images: /.autorelabel; UEFI hosts: efibootmgr entry for the ESP loader)
POST progress state=done → server: state deployed, PXE entry = local disk, reboot
▼
first boot: cloud-init sets hostname/fqdn, SSH keys (root + default user), optional
package_upgrade, then phone_home → POST /api/v1/phone-home/<token> (shown as "phoned home")
```
### Image catalog and freshness
```
os_dir/<name>/<version>/disk.raw.zst raw disk image, zstd
os_dir/<name>/<version>/meta.json sizes, sha256, upstream checksum, import time
os_dir/<name>/current -> <version> what new deployments use
```
`os_sources` (`deploy/os-sources.json`) lists upstream "latest" URLs with the
vendor checksum file for each: Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9,
CentOS Stream 9. `rollcall os sync` fetches the checksum file, and only when the
vendor checksum differs from every imported version does it download the image,
verify it, convert to raw (`qemu-img`), compress, write a new version directory
and flip `current` atomically. `rollcall-os-sync.timer` runs it nightly;
`keep` versions are retained per source, nothing younger than a day is pruned,
and a job always pins the exact version it started with. Sources with
`"track": "pinned"` still download new versions but keep `current` until
`rollcall os promote <name> <version>`. Local files go in with
`rollcall os import <name> <file.qcow2>`. Image freshness is monthly at best
upstream; the deploy option `-upgrade` (`package_upgrade`) pulls the rest on
first boot.
```sh
rollcall os list | sync [NAME...] [-force] | import NAME FILE [-family debian|rhel] | promote NAME VERSION | rm NAME VERSION
rollcall deploy ID OS [-version V] [-disk /dev/nvme0n1] [-hostname H] [-upgrade] [-packages a,b] [-user-data FILE] [-boot] [-watch]
rollcall release ID [-wipe quick|full] [-boot] [-watch] # wipe the disk, back to ready
rollcall jobs ID [-watch] | job cancel ID
```
States: `ready → deploying → deployed` (`deploy-failed` on error; the agent stays
up for ssh). A deployed machine's PXE entry defaults to local disk, so boot order
can stay "network first" on BIOS hosts; on UEFI hosts the agent adds a boot entry
for the installed loader. `rollcall recommission` or `release` take it back.
## Lab ## Lab
`lab/` documents a Proxmox SDN test setup: an isolated vnet, a PXE server VM `lab/` documents a Proxmox SDN test setup: an isolated vnet, a PXE server VM
@@ -166,6 +234,7 @@ IPMI-over-LAN BMC emulator that drives VMs through the Proxmox API so that
## Status ## Status
Working end to end in the lab (virtual machines, out-of-band BMC hints). The Commissioning, stateless nodes and disk deployment work end to end in the lab
in-band BMC path is implemented but has not yet been exercised on real (virtual machines with 30 GB disks, SeaBIOS, out-of-band BMC hints). The
hardware. in-band BMC path and the UEFI boot-entry step are implemented but have not yet
been exercised on real hardware.
Binary file not shown.
Binary file not shown.
+9 -1
View File
@@ -68,8 +68,15 @@ pv
jq jq
mdadm mdadm
lvm2 lvm2
nvme-cli
efibootmgr efibootmgr
zstd
gdisk
parted
xfsprogs
e2fsprogs
btrfs-progs
dosfstools
bash
firmware-linux firmware-linux
firmware-linux-nonfree firmware-linux-nonfree
firmware-misc-nonfree firmware-misc-nonfree
@@ -88,6 +95,7 @@ mkdir -p config/includes.chroot/usr/local/bin config/includes.chroot/etc/systemd
config/includes.chroot/root/.ssh config/includes.chroot/etc/modules-load.d \ config/includes.chroot/root/.ssh config/includes.chroot/etc/modules-load.d \
config/hooks/live config/hooks/live
install -m 0755 "$SRC/rollcall-agent.py" config/includes.chroot/usr/local/bin/rollcall-agent.py install -m 0755 "$SRC/rollcall-agent.py" config/includes.chroot/usr/local/bin/rollcall-agent.py
install -m 0755 "$SRC/rollcall-deploy.py" config/includes.chroot/usr/local/bin/rollcall-deploy.py
install -m 0644 "$SRC/rollcall-agent.service" config/includes.chroot/etc/systemd/system/rollcall-agent.service install -m 0644 "$SRC/rollcall-agent.service" config/includes.chroot/etc/systemd/system/rollcall-agent.service
printf 'ipmi_msghandler\nipmi_devintf\nipmi_si\nipmi_ssif\n' > config/includes.chroot/etc/modules-load.d/ipmi.conf printf 'ipmi_msghandler\nipmi_devintf\nipmi_si\nipmi_ssif\n' > config/includes.chroot/etc/modules-load.d/ipmi.conf
if [ -r "$AUTHKEYS" ]; then if [ -r "$AUTHKEYS" ]; then
+4
View File
@@ -10,6 +10,7 @@ Kernel cmdline options:
rollcall.nopoweroff stay up (debugging) rollcall.nopoweroff stay up (debugging)
rollcall.bmcuser=commissioner BMC account name to create rollcall.bmcuser=commissioner BMC account name to create
rollcall.token=... agent token (sent as X-Rollcall-Token) rollcall.token=... agent token (sent as X-Rollcall-Token)
rollcall.mode=deploy rollcall.job=.. run rollcall-deploy.py instead (OS deployment / release job)
""" """
import glob import glob
import json import json
@@ -441,6 +442,9 @@ def post(server, payload, tries=30):
def main(): def main():
opts = cmdline() opts = cmdline()
if opts.get("mode") == "deploy":
# same image, different job: hand over to the deploy/release agent
os.execv("/usr/bin/python3", ["python3", "/usr/local/bin/rollcall-deploy.py"])
server = opts.get("server") server = opts.get("server")
if not server: if not server:
log("rollcall.server= missing on kernel cmdline; nothing to do") log("rollcall.server= missing on kernel cmdline; nothing to do")
+385
View File
@@ -0,0 +1,385 @@
#!/usr/bin/env python3
"""Deploy/release agent: runs inside the commissioning live image when the
kernel cmdline carries rollcall.mode=deploy rollcall.job=<token>.
deploy : stream <os>/<version>/disk.raw.zst onto the target disk (sha256
verified), seed cloud-init NoCloud files into the written root fs,
add a UEFI boot entry when booted via UEFI, report, reboot.
release: wipe the target disk (quick: signatures + first/last 64 MiB), report,
power off.
Everything the agent needs (image URL, checksum, disk, seed files) comes from
GET /api/v1/jobs/self; it never decides policy itself.
"""
import glob
import json
import os
import re
import subprocess
import sys
import threading
import time
import urllib.error
import urllib.request
LOG = open("/dev/console", "w", buffering=1) if os.path.exists("/dev/console") else sys.stdout
SERVER = ""
TOKEN = ""
EFI_GUID = "c12a7328-f81e-11d2-ba4b-00a0c93ec93b"
ROOT_FS = ("ext4", "xfs", "btrfs", "ext3")
def log(msg):
line = f"[deploy] {msg}"
print(line, file=LOG)
if LOG is not sys.stdout:
print(line, flush=True)
def run(cmd, timeout=600, check=False, **kw):
p = subprocess.run(cmd, shell=isinstance(cmd, str), capture_output=True, text=True, timeout=timeout, **kw)
if check and p.returncode != 0:
raise RuntimeError(f"{cmd if isinstance(cmd, str) else ' '.join(cmd)}: rc={p.returncode} {p.stderr.strip()[-400:]}")
return p
def readf(path, default=""):
try:
with open(path) as f:
return f.read().strip()
except OSError:
return default
def cmdline():
opts = {}
for tok in readf("/proc/cmdline").split():
if tok.startswith("rollcall."):
k, _, v = tok[len("rollcall."):].partition("=")
opts[k] = v if v else True
return opts
# --------------------------------------------------------------------------- server
def api(method, path, body=None, tries=20):
data = json.dumps(body).encode() if body is not None else None
req = urllib.request.Request(SERVER + path, data=data, method=method,
headers={"Content-Type": "application/json", "X-Rollcall-Job": TOKEN})
last = None
for i in range(tries):
try:
with urllib.request.urlopen(req, timeout=30) as r:
return json.load(r)
except urllib.error.HTTPError as e:
if 400 <= e.code < 500:
raise RuntimeError(f"{path}: HTTP {e.code} {e.read().decode(errors='replace')[:200]}")
last = e
except (urllib.error.URLError, OSError) as e:
last = e
time.sleep(3)
raise RuntimeError(f"{path}: {last}")
class Progress:
def __init__(self):
self.phase, self.pct, self.msg = "starting", 0, ""
self.lock = threading.Lock()
self.stop = threading.Event()
self.thread = threading.Thread(target=self._loop, daemon=True)
self.thread.start()
def set(self, phase=None, pct=None, msg=None, now=False):
with self.lock:
if phase is not None:
self.phase = phase
if pct is not None:
self.pct = pct
if msg is not None:
self.msg = msg
if now:
self._post()
def _post(self):
with self.lock:
body = {"phase": self.phase, "pct": round(self.pct, 1), "msg": self.msg}
try:
api("POST", "/api/v1/jobs/self/progress", body, tries=1)
except Exception as e: # noqa: BLE001
log(f"progress post failed: {e}")
def _loop(self):
while not self.stop.wait(3):
self._post()
def finish(self, state, msg="", error="", details=None):
self.stop.set()
body = {"state": state, "phase": state, "pct": 100 if state == "done" else self.pct,
"msg": msg, "error": error, "details": details or {}}
return api("POST", "/api/v1/jobs/self/progress", body)
# --------------------------------------------------------------------------- disks
def lsblk(dev=None):
cmd = ["lsblk", "-J", "-b", "-o", "NAME,PATH,TYPE,SIZE,SERIAL,RM,ROTA,TRAN,FSTYPE,PARTTYPE,PARTN,PKNAME,MOUNTPOINT"]
if dev:
cmd.append(dev)
p = run(cmd)
if p.returncode != 0:
return []
out = []
def walk(items):
for d in items:
out.append(d)
walk(d.get("children") or [])
walk(json.loads(p.stdout).get("blockdevices", []))
return out
def find_disk(spec):
want, serial, need = spec.get("disk") or "", spec.get("disk_serial") or "", int(spec.get("raw_size") or 0)
disks = [d for d in lsblk() if d.get("type") == "disk" and not d["name"].startswith(("loop", "ram", "zram", "sr"))]
if want:
for d in disks:
if d["path"] == want or d["name"] == want.replace("/dev/", ""):
if serial and d.get("serial") and d["serial"] != serial:
# device names move around; trust the serial over the name
for e in disks:
if e.get("serial") == serial:
log(f"{want} has serial {d['serial']}, expected {serial}; using {e['path']} instead")
return e
raise RuntimeError(f"{want} has serial {d['serial']}, expected {serial}; refusing to write")
return d
if serial:
for d in disks:
if d.get("serial") == serial:
log(f"{want} not found; serial {serial} is {d['path']}")
return d
raise RuntimeError(f"target disk {want} not present")
cands = [d for d in disks if not d.get("rm") and d.get("tran") != "usb" and int(d["size"]) >= need]
if not cands:
raise RuntimeError(f"no disk >= {need} bytes")
cands.sort(key=lambda d: (int(d["size"]), d["name"]))
return cands[0]
def release_disk(path):
"""Make sure nothing holds the disk: swap, mounts, md, lvm."""
for d in lsblk(path):
if d.get("mountpoint"):
run(["umount", "-f", d["path"]])
run("swapoff -a")
run(["vgchange", "-an"])
run(["mdadm", "--stop", "--scan"])
for d in lsblk(path):
if d.get("type") == "part":
run(["wipefs", "-a", d["path"]])
run(["wipefs", "-a", path])
run(["blockdev", "--rereadpt", path])
run(["udevadm", "settle"])
# --------------------------------------------------------------------------- deploy
def write_image(spec, disk, prog):
url, want, size = spec["image_url"], spec["sha256"], int(spec.get("size") or 0)
sumf, pvf = "/run/rollcall.sum", "/run/rollcall.pv"
for f in (sumf, pvf):
if os.path.exists(f):
os.remove(f)
pipeline = (f"set -o pipefail; curl -fsSL --retry 5 --retry-connrefused '{url}' "
f"| tee >(sha256sum > {sumf}) "
f"| pv -n -f -i 2 -s {size} 2>{pvf} "
f"| zstd -dc -T0 "
f"| dd of='{disk}' bs=4M iflag=fullblock oflag=direct conv=fsync status=none")
log(f"writing {url} -> {disk}")
proc = subprocess.Popen(["bash", "-c", pipeline], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
pct = 0
while proc.poll() is None:
time.sleep(2)
try:
with open(pvf) as f:
lines = f.read().split()
if lines:
pct = int(lines[-1])
except (OSError, ValueError):
pass
prog.set("writing", min(pct, 99), f"{pct}% of {size // (1 << 20)} MiB written to {disk}")
if proc.returncode != 0:
raise RuntimeError(f"image write failed rc={proc.returncode}: {proc.stderr.read().strip()[-400:]}")
got = readf(sumf).split()[0] if readf(sumf) else ""
if got != want:
raise RuntimeError(f"sha256 mismatch: got {got} want {want}")
run(["sync"])
run(["blockdev", "--rereadpt", disk])
run(["partprobe", disk])
run(["udevadm", "settle"])
time.sleep(2)
def find_parts(disk):
parts = [d for d in lsblk(disk) if d.get("type") == "part"]
root = esp = None
for p in sorted(parts, key=lambda d: -int(d["size"])):
if p.get("fstype") in ROOT_FS and root is None:
root = p
if (p.get("parttype") or "").lower() == EFI_GUID and p.get("fstype") == "vfat" and esp is None:
esp = p
return root, esp, parts
def mount(dev, where, opts=None):
os.makedirs(where, exist_ok=True)
cmd = ["mount"] + (["-o", opts] if opts else []) + [dev, where]
run(cmd, check=True)
def seed(root, files):
mnt = "/mnt/rollcall-root"
# xfs root written from an image can carry a duplicate UUID with nothing else; nouuid is harmless
try:
mount(root["path"], mnt)
except RuntimeError:
mount(root["path"], mnt, "nouuid")
try:
if not os.path.isdir(f"{mnt}/etc"):
raise RuntimeError(f"{root['path']} has no /etc; not a root filesystem")
for f in files:
dst = os.path.join(mnt, f["path"].lstrip("/"))
os.makedirs(os.path.dirname(dst), exist_ok=True)
with open(dst, "w") as fh:
fh.write(f["content"])
os.chmod(dst, int(f.get("mode", "0644"), 8))
log(f"wrote {f['path']}")
# cloud-init's stale state would stop it from re-running on a reused image
run(f"rm -rf {mnt}/var/lib/cloud/instance {mnt}/var/lib/cloud/instances/* {mnt}/var/lib/cloud/data/* 2>/dev/null")
sel = readf(f"{mnt}/etc/selinux/config")
if re.search(r"^SELINUX=(enforcing|permissive)", sel, re.M):
open(f"{mnt}/.autorelabel", "w").close()
log("SELinux enabled in image: scheduled autorelabel for first boot")
os_release = readf(f"{mnt}/etc/os-release")
m = re.search(r'^PRETTY_NAME="?([^"\n]*)', os_release, re.M)
run(["sync"])
return m.group(1) if m else ""
finally:
run(["umount", mnt])
def efi_entry(disk, esp, label):
if not os.path.isdir("/sys/firmware/efi") or esp is None:
return None
mnt = "/mnt/rollcall-esp"
try:
mount(esp["path"], mnt, "ro")
except RuntimeError as e:
return f"esp mount failed: {e}"
try:
loader = None
for name in ("shimx64.efi", "grubx64.efi"):
for d in ("debian", "ubuntu", "rocky", "almalinux", "centos", "redhat", "fedora"):
if os.path.exists(f"{mnt}/EFI/{d}/{name}"):
loader = f"\\EFI\\{d}\\{name}"
break
if loader:
break
if not loader:
hits = glob.glob(f"{mnt}/EFI/*/shimx64.efi") + glob.glob(f"{mnt}/EFI/*/grubx64.efi") + glob.glob(f"{mnt}/EFI/BOOT/BOOTX64.EFI")
if hits:
loader = "\\" + os.path.relpath(hits[0], mnt).replace("/", "\\")
if not loader:
return "no EFI loader found on ESP"
finally:
run(["umount", mnt])
# drop stale entries with our label, then create a new one first in BootOrder
cur = run(["efibootmgr"]).stdout
for m in re.finditer(r"^Boot([0-9A-Fa-f]{4})\*?\s+(.*)$", cur, re.M):
if m.group(2).strip().startswith(label):
run(["efibootmgr", "-q", "-b", m.group(1), "-B"])
p = run(["efibootmgr", "-c", "-d", disk, "-p", str(esp.get("partn")), "-L", label, "-l", loader])
if p.returncode != 0:
return f"efibootmgr failed: {p.stderr.strip()[-200:]}"
return f"{label} -> {loader} (part {esp.get('partn')})"
def do_deploy(job, prog):
spec, files = job["spec"], job.get("files") or []
started = time.time()
prog.set("preparing", 0, "selecting disk", now=True)
disk = find_disk(spec)
size = int(disk["size"])
if int(spec.get("raw_size") or 0) > size:
raise RuntimeError(f"{disk['path']} ({size} bytes) smaller than image ({spec['raw_size']} bytes)")
log(f"target {disk['path']} serial={disk.get('serial')} size={size}")
prog.set("preparing", 1, f"wiping {disk['path']}", now=True)
release_disk(disk["path"])
write_image(spec, disk["path"], prog)
prog.set("seeding", 99, "writing cloud-init seed", now=True)
root, esp, parts = find_parts(disk["path"])
if root is None:
raise RuntimeError("no root filesystem found after writing image: " + ", ".join(f"{p['name']}:{p.get('fstype')}" for p in parts))
pretty = seed(root, files)
efi = efi_entry(disk["path"], esp, f"rollcall {spec.get('os', '')}")
if efi:
log(f"efi: {efi}")
details = {"disk": disk["path"], "serial": disk.get("serial"), "root": root["path"], "root_fs": root.get("fstype"),
"esp": esp["path"] if esp else None, "efi": efi, "os_release": pretty, "firmware": "uefi" if os.path.isdir("/sys/firmware/efi") else "bios",
"seconds": round(time.time() - started, 1)}
return details
def do_release(job, prog):
spec = job["spec"]
prog.set("preparing", 0, "selecting disk", now=True)
disk = find_disk(spec)
path, size = disk["path"], int(disk["size"])
prog.set("wiping", 5, f"wiping {path}", now=True)
release_disk(path)
mib = 64
run(f"dd if=/dev/zero of={path} bs=1M count={mib} conv=fsync status=none")
run(f"dd if=/dev/zero of={path} bs=1M count={mib} seek={max(0, size // (1 << 20) - mib)} conv=fsync status=none")
if spec.get("wipe") == "full":
p = run(["blkdiscard", "-f", path], timeout=3600)
if p.returncode != 0:
prog.set("wiping", 10, "full zero fill", now=True)
run(f"dd if=/dev/zero of={path} bs=16M oflag=direct status=none", timeout=24 * 3600)
run(["blockdev", "--rereadpt", path])
return {"disk": path, "serial": disk.get("serial"), "wipe": spec.get("wipe")}
def main():
global SERVER, TOKEN
opts = cmdline()
SERVER = (opts.get("server") or "").rstrip("/")
TOKEN = opts.get("job") if isinstance(opts.get("job"), str) else ""
if not SERVER or not TOKEN:
log("rollcall.server= and rollcall.job= required")
return 2
log(f"fetching job from {SERVER}")
try:
job = api("GET", "/api/v1/jobs/self")
except Exception as e: # noqa: BLE001
log(f"cannot fetch job: {e}; staying up")
return 1
log(f"job {job['id']} kind={job['kind']} spec={json.dumps({k: v for k, v in job['spec'].items() if k not in ('ssh_keys', 'user_data')})}")
prog = Progress()
try:
details = do_deploy(job, prog) if job["kind"] == "deploy" else do_release(job, prog)
resp = prog.finish("done", "finished", details=details)
except Exception as e: # noqa: BLE001
log(f"FAILED: {e!r}")
try:
resp = prog.finish("failed", str(e), error=str(e))
except Exception as e2: # noqa: BLE001
log(f"could not report failure: {e2}")
resp = {"next": "stay"}
nxt = resp.get("next", "stay")
log(f"server says: {nxt}")
if opts.get("nopoweroff") or nxt == "stay":
log("staying up for inspection (ssh root@<ip>)")
return 0 if nxt != "stay" else 1
time.sleep(3)
run(["systemctl", "reboot" if nxt == "reboot" else "poweroff"])
return 0
if __name__ == "__main__":
sys.exit(main())
+143 -3
View File
@@ -99,9 +99,19 @@ func runClient(cmd string, args []string) error {
if err := c.do("GET", "/api/v1/machines", nil, &ms); err != nil { if err := c.do("GET", "/api/v1/machines", nil, &ms); err != nil {
return err return err
} }
f := "%-16s %-12s %-10s %-10s %-14s %-24s %4s %7s %3s %7s %3s %2s %3s %s\n" f := "%-16s %-14s %-10s %-16s %-14s %-24s %4s %7s %3s %7s %3s %2s %3s %s\n"
fmt.Printf(f, "ID", "STATE", "NODE", "PROFILE", "MGMT-IP", "MODEL", "CPU", "RAM", "DSK", "STORAGE", "NIC", "IB", "GPU", "BMC") fmt.Printf(f, "ID", "STATE", "NODE", "PROFILE/OS", "MGMT-IP", "MODEL", "CPU", "RAM", "DSK", "STORAGE", "NIC", "IB", "GPU", "BMC")
for _, m := range ms { for _, m := range ms {
state := m.State
if m.Job != nil {
var p obj
_ = json.Unmarshal(m.Job.Progress, &p)
state = fmt.Sprintf("%s %.0f%%", m.Job.Kind, num(p["pct"]))
}
what := deref(m.Profile)
if what == "" && m.OS != nil {
what = *m.OS
}
bmc := "-" bmc := "-"
if m.BMC != nil { if m.BMC != nil {
b := *m.BMC b := *m.BMC
@@ -112,7 +122,7 @@ func runClient(cmd string, args []string) error {
bmc = fmt.Sprintf("%s (%s,%s)", str(b["address"]), str(b["source"]), ok) bmc = fmt.Sprintf("%s (%s,%s)", str(b["address"]), str(b["source"]), ok)
} }
model := strings.TrimSpace(deref(m.Manufacturer) + " " + deref(m.Product)) model := strings.TrimSpace(deref(m.Manufacturer) + " " + deref(m.Product))
fmt.Printf(f, m.ID, m.State, deref(m.NodeName), deref(m.Profile), deref(m.MgmtIP), trunc(model, 24), m.CPUs, human(m.MemoryBytes), fmt.Printf(f, m.ID, state, deref(m.NodeName), trunc(what, 16), deref(m.MgmtIP), trunc(model, 24), m.CPUs, human(m.MemoryBytes),
fmt.Sprint(int(m.Disks)), human(m.StorageBytes), fmt.Sprint(m.NICs), fmt.Sprint(m.IBHCAs), fmt.Sprint(m.GPUs), bmc) fmt.Sprint(int(m.Disks)), human(m.StorageBytes), fmt.Sprint(m.NICs), fmt.Sprint(m.IBHCAs), fmt.Sprint(m.GPUs), bmc)
} }
case "show": case "show":
@@ -177,12 +187,142 @@ func runClient(cmd string, args []string) error {
return err return err
} }
fmt.Println("deleted", str(d["deleted"])) fmt.Println("deleted", str(d["deleted"]))
case "deploy", "release", "jobs", "job":
return c.deployCmd(cmd, args)
default: default:
return c.statelessCmd(cmd, args) return c.statelessCmd(cmd, args)
} }
return nil return nil
} }
func (c *client) deployCmd(cmd string, args []string) error {
switch cmd {
case "deploy":
if len(args) < 2 {
usage()
}
fs := newFlags("deploy")
ver := fs.String("version", "", "image version (default: current)")
disk := fs.String("disk", "", "target disk, e.g. /dev/sda (default: smallest disk that fits)")
host := fs.String("hostname", "", "hostname (default: existing node name or next free)")
upgrade := fs.Bool("upgrade", false, "cloud-init package_upgrade on first boot")
pkgs := fs.String("packages", "", "comma-separated packages to install on first boot")
ud := fs.String("user-data", "", "file with a complete #cloud-config to use instead of the generated one")
boot := fs.Bool("boot", false, "set bootdev pxe and power-cycle via BMC")
watch := fs.Bool("watch", false, "follow job progress")
_ = fs.Parse(args[2:])
req := obj{"os": args[1], "version": *ver, "disk": *disk, "hostname": *host, "package_upgrade": *upgrade, "boot": *boot}
if *pkgs != "" {
req["packages"] = strings.Split(*pkgs, ",")
}
if *ud != "" {
b, err := os.ReadFile(*ud)
if err != nil {
return err
}
req["user_data"] = string(b)
}
var d obj
if err := c.do("POST", "/api/v1/machines/"+args[0]+"/deploy", req, &d); err != nil {
return err
}
fmt.Printf("%s: job %s %s/%s -> %s hostname=%s mgmt=%s\n", str(d["id"]), str(d["job"]), str(d["os"]), str(d["version"]), str(d["disk"]), str(d["hostname"]), str(d["mgmt_ip"]))
if d["ipmi"] != nil {
fmt.Println(str(d["ipmi"]))
}
if *watch {
return c.watchJob(args[0])
}
case "release":
if len(args) < 1 {
usage()
}
fs := newFlags("release")
boot := fs.Bool("boot", false, "set bootdev pxe and power-cycle via BMC")
wipe := fs.String("wipe", "quick", "quick (signatures + first/last 64 MiB) or full")
watch := fs.Bool("watch", false, "follow job progress")
_ = fs.Parse(args[1:])
var d obj
if err := c.do("POST", "/api/v1/machines/"+args[0]+"/release", obj{"boot": *boot, "wipe": *wipe}, &d); err != nil {
return err
}
fmt.Printf("%s: job %s wipe %s\n", str(d["id"]), str(d["job"]), str(d["disk"]))
if d["ipmi"] != nil {
fmt.Println(str(d["ipmi"]))
}
if *watch {
return c.watchJob(args[0])
}
case "jobs":
if len(args) < 1 {
usage()
}
if len(args) > 1 && args[1] == "-watch" {
return c.watchJob(args[0])
}
var jobs []Job
if err := c.do("GET", "/api/v1/machines/"+args[0]+"/jobs", nil, &jobs); err != nil {
return err
}
fmt.Printf("%-10s %-8s %-10s %-20s %-20s %s\n", "JOB", "KIND", "STATE", "CREATED", "FINISHED", "OS / PROGRESS / ERROR")
for _, j := range jobs {
sp := j.spec()
var p obj
_ = json.Unmarshal(j.Progress, &p)
fin := ""
if j.Finished != nil {
fin = j.Finished.Local().Format("2006-01-02 15:04:05")
}
extra := sp.OS + "/" + sp.Version + " " + sp.Disk
if j.State == "running" {
extra += fmt.Sprintf(" %s %.0f%% %s", str(p["phase"]), num(p["pct"]), str(p["msg"]))
}
if j.Error != nil {
extra += " " + *j.Error
}
fmt.Printf("%-10s %-8s %-10s %-20s %-20s %s\n", j.ID, j.Kind, j.State, j.Created.Local().Format("2006-01-02 15:04:05"), fin, extra)
}
case "job":
if len(args) < 2 || args[0] != "cancel" {
usage()
}
var d obj
if err := c.do("POST", "/api/v1/machines/"+args[1]+"/jobs/cancel", nil, &d); err != nil {
return err
}
fmt.Printf("job %s cancelled\n", str(d["job"]))
}
return nil
}
func (c *client) watchJob(id string) error {
last := ""
for {
var jobs []Job
if err := c.do("GET", "/api/v1/machines/"+id+"/jobs", nil, &jobs); err != nil {
return err
}
if len(jobs) == 0 {
return fmt.Errorf("no jobs")
}
j := jobs[0]
var p obj
_ = json.Unmarshal(j.Progress, &p)
line := fmt.Sprintf("%s %s %-9s %-12s %3.0f%% %s", time.Now().Format("15:04:05"), j.ID, j.State, str(p["phase"]), num(p["pct"]), str(p["msg"]))
if line[9:] != last {
fmt.Println(line)
last = line[9:]
}
if j.State != "pending" && j.State != "running" {
if j.Error != nil {
return fmt.Errorf("%s", *j.Error)
}
return nil
}
time.Sleep(3 * time.Second)
}
}
func (c *client) statelessCmd(cmd string, args []string) error { func (c *client) statelessCmd(cmd string, args []string) error {
switch cmd { switch cmd {
case "images": case "images":
+10
View File
@@ -28,6 +28,10 @@ type Config struct {
DnsmasqDir string `json:"dnsmasq_dir"` // hosts.dhcp + hosts written here for dnsmasq (dhcp-hostsfile / addn-hosts); "" disables DnsmasqDir string `json:"dnsmasq_dir"` // hosts.dhcp + hosts written here for dnsmasq (dhcp-hostsfile / addn-hosts); "" disables
NodePools map[string]string `json:"node_pools"` // {"mgmt": "10.0.0.50-10.0.0.99", "ib": "10.1.0.10-10.1.0.250"} NodePools map[string]string `json:"node_pools"` // {"mgmt": "10.0.0.50-10.0.0.99", "ib": "10.1.0.10-10.1.0.250"}
ClusterVars map[string]any `json:"cluster_vars"` // available to overlay templates as .Vars (profile vars override) ClusterVars map[string]any `json:"cluster_vars"` // available to overlay templates as .Vars (profile vars override)
// disk-based deployment
OSDir string `json:"os_dir"` // <os_dir>/<name>/<version>/{disk.raw.zst,meta.json}, served under /os/
OSSources string `json:"os_sources"` // JSON list of upstream cloud images for `rollcall os sync`
} }
// BMCHint maps a machine MAC to a known BMC (virtual lab nodes). // BMCHint maps a machine MAC to a known BMC (virtual lab nodes).
@@ -68,5 +72,11 @@ func loadConfig(path string) (*Config, error) {
if c.OverlaysDir == "" { if c.OverlaysDir == "" {
c.OverlaysDir = "/etc/rollcall/overlays" c.OverlaysDir = "/etc/rollcall/overlays"
} }
if c.OSDir == "" {
c.OSDir = filepath.Join(filepath.Dir(c.ImageDir), "os")
}
if c.OSSources == "" {
c.OSSources = "/etc/rollcall/os-sources.json"
}
return c, nil return c, nil
} }
+643
View File
@@ -0,0 +1,643 @@
package main
// Disk-based OS deployment: a job boots the commissioning image in deploy
// mode; the agent streams <os>/<version>/disk.raw.zst onto the chosen disk,
// seeds cloud-init (NoCloud) in the written root filesystem and reports
// progress. On success the machine boots from disk and cloud-init phones home.
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"log"
"net/http"
"sort"
"strings"
"time"
)
type Job struct {
ID string `json:"id"`
MachineID string `json:"machine_id"`
Kind string `json:"kind"` // deploy | release
State string `json:"state"`
Spec json.RawMessage `json:"spec"`
Progress json.RawMessage `json:"progress"`
Created time.Time `json:"created"`
Started *time.Time `json:"started"`
Finished *time.Time `json:"finished"`
Error *string `json:"error"`
}
// DeploySpec is what the agent receives (plus the rendered seed files).
type DeploySpec struct {
OS string `json:"os"`
Version string `json:"version"`
ImageURL string `json:"image_url"`
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
RawSize int64 `json:"raw_size"`
Disk string `json:"disk"`
DiskSerial string `json:"disk_serial,omitempty"`
Hostname string `json:"hostname"`
Domain string `json:"domain,omitempty"`
SSHKeys []string `json:"ssh_keys"`
PackageUpgrade bool `json:"package_upgrade"`
Packages []string `json:"packages,omitempty"`
UserData string `json:"user_data,omitempty"` // full #cloud-config override
Wipe string `json:"wipe,omitempty"` // release: quick | full
}
type seedFile struct {
Path string `json:"path"`
Mode string `json:"mode"`
Content string `json:"content"`
}
func (j *Job) spec() DeploySpec {
var s DeploySpec
_ = json.Unmarshal(j.Spec, &s)
return s
}
// ---------------------------------------------------------------- store
const jobCols = `id, machine_id, kind, state, spec, progress, created, started, finished, error`
func scanJob(row interface{ Scan(...any) error }) (*Job, error) {
j := &Job{}
err := row.Scan(&j.ID, &j.MachineID, &j.Kind, &j.State, &j.Spec, &j.Progress, &j.Created, &j.Started, &j.Finished, &j.Error)
if err != nil {
if strings.Contains(err.Error(), "no rows") {
return nil, errNotFound
}
return nil, err
}
return j, nil
}
func (s *Store) CreateJob(ctx context.Context, j *Job, tokenHash string, sealed []byte) error {
_, err := s.pool.Exec(ctx, `INSERT INTO jobs(id, machine_id, kind, state, spec, token_hash, token) VALUES ($1,$2,$3,'pending',$4,$5,$6)`,
j.ID, j.MachineID, j.Kind, j.Spec, tokenHash, sealed)
return err
}
func (s *Store) JobByToken(ctx context.Context, hash string) (*Job, error) {
return scanJob(s.pool.QueryRow(ctx, `SELECT `+jobCols+` FROM jobs WHERE token_hash=$1`, hash))
}
func (s *Store) Job(ctx context.Context, id string) (*Job, error) {
return scanJob(s.pool.QueryRow(ctx, `SELECT `+jobCols+` FROM jobs WHERE id=$1`, id))
}
func (s *Store) ActiveJob(ctx context.Context, machineID string) (*Job, error) {
return scanJob(s.pool.QueryRow(ctx, `SELECT `+jobCols+` FROM jobs WHERE machine_id=$1 AND state IN ('pending','running') ORDER BY created DESC LIMIT 1`, machineID))
}
func (s *Store) ActiveJobs(ctx context.Context) (map[string]*Job, error) {
rows, err := s.pool.Query(ctx, `SELECT `+jobCols+` FROM jobs WHERE state IN ('pending','running') ORDER BY created`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]*Job{}
for rows.Next() {
j, err := scanJob(rows)
if err != nil {
return nil, err
}
out[j.MachineID] = j
}
return out, rows.Err()
}
func (s *Store) Jobs(ctx context.Context, machineID string, limit int) ([]*Job, error) {
rows, err := s.pool.Query(ctx, `SELECT `+jobCols+` FROM jobs WHERE machine_id=$1 ORDER BY created DESC LIMIT $2`, machineID, limit)
if err != nil {
return nil, err
}
defer rows.Close()
out := []*Job{}
for rows.Next() {
j, err := scanJob(rows)
if err != nil {
return nil, err
}
out = append(out, j)
}
return out, rows.Err()
}
func (s *Store) JobProgress(ctx context.Context, id string, progress []byte) error {
_, err := s.pool.Exec(ctx, `UPDATE jobs SET state='running', started=coalesce(started, now()), progress=$2 WHERE id=$1 AND state IN ('pending','running')`, id, progress)
return err
}
func (s *Store) FinishJob(ctx context.Context, id, state, errMsg string) error {
_, err := s.pool.Exec(ctx, `UPDATE jobs SET state=$2, finished=now(), started=coalesce(started, now()), error=nullif($3,'') WHERE id=$1`, id, state, errMsg)
return err
}
func (s *Store) SetDeployed(ctx context.Context, id, os, version string) error {
_, err := s.pool.Exec(ctx, `UPDATE machines SET state='deployed', os=$2, os_version=$3, deployed_at=now(), booted_at=NULL, last_seen=now() WHERE id=$1`, id, os, version)
return err
}
func (s *Store) SetReleased(ctx context.Context, id string) error {
_, err := s.pool.Exec(ctx, `UPDATE machines SET state='ready', os=NULL, os_version=NULL, deployed_at=NULL, booted_at=NULL, last_seen=now() WHERE id=$1`, id)
return err
}
func (s *Store) SetNodeAddr(ctx context.Context, id, name, mgmt, ib string) error {
_, err := s.pool.Exec(ctx, `UPDATE machines SET node_name=$2, mgmt_ip=$3, ib_ip=$4 WHERE id=$1`, id, name, nilIfEmpty(mgmt), nilIfEmpty(ib))
return err
}
func (s *Store) MarkPhoneHome(ctx context.Context, id string) error {
_, err := s.pool.Exec(ctx, `UPDATE machines SET booted_at=now(), last_seen=now() WHERE id=$1`, id)
return err
}
// ---------------------------------------------------------------- disk selection
type invDisk struct {
Name, Path, Serial, Tran, Model string
Size int64
Rota, Removable bool
}
func inventoryDisks(m *Machine) []invDisk {
var inv obj
_ = json.Unmarshal(m.Inventory, &inv)
var out []invDisk
for _, dv := range toList(dig(inv, "storage", "disks")) {
d, _ := dv.(obj)
rm, _ := d["rm"].(bool)
rota, _ := d["rota"].(bool)
id := invDisk{Name: str(d["name"]), Path: str(d["path"]), Serial: str(d["serial"]), Tran: str(d["tran"]),
Model: str(d["model"]), Size: int64(num(d["size"])), Rota: rota, Removable: rm}
if id.Path == "" {
id.Path = "/dev/" + id.Name
}
out = append(out, id)
}
return out
}
// pickDisk: the smallest non-removable, non-USB disk that fits the image (boot/OS disks are
// usually the small ones on servers; data disks stay untouched). Explicit choice wins.
func pickDisk(disks []invDisk, want string, need int64) (*invDisk, error) {
if want != "" {
for i := range disks {
if disks[i].Path == want || disks[i].Name == want || "/dev/"+disks[i].Name == want {
if disks[i].Size > 0 && disks[i].Size < need {
return nil, fmt.Errorf("disk %s (%s) is smaller than the image (%s)", want, human(float64(disks[i].Size)), human(float64(need)))
}
return &disks[i], nil
}
}
if len(disks) == 0 {
return &invDisk{Path: want, Name: strings.TrimPrefix(want, "/dev/")}, nil
}
return nil, fmt.Errorf("disk %s not in inventory", want)
}
var cands []invDisk
for _, d := range disks {
if d.Removable || d.Tran == "usb" || d.Size < need {
continue
}
cands = append(cands, d)
}
if len(cands) == 0 {
return nil, fmt.Errorf("no disk >= %s in inventory (pass disk explicitly)", human(float64(need)))
}
sort.Slice(cands, func(i, j int) bool {
if cands[i].Size != cands[j].Size {
return cands[i].Size < cands[j].Size
}
return cands[i].Name < cands[j].Name
})
return &cands[0], nil
}
// ---------------------------------------------------------------- cloud-init seed
func yamlStr(s string) string { b, _ := json.Marshal(s); return string(b) }
func yamlList(l []string) string {
if l == nil {
l = []string{}
}
b, _ := json.Marshal(l)
return string(b)
}
func (a *App) renderSeed(j *Job, sp DeploySpec, token string) []seedFile {
meta := fmt.Sprintf("instance-id: %s\nlocal-hostname: %s\n", j.ID, sp.Hostname)
ud := sp.UserData
if ud == "" {
var b strings.Builder
b.WriteString("#cloud-config\n# generated by rollcall for " + j.MachineID + " (" + j.ID + ")\n")
fmt.Fprintf(&b, "hostname: %s\n", yamlStr(sp.Hostname))
if sp.Domain != "" {
fmt.Fprintf(&b, "fqdn: %s\n", yamlStr(sp.Hostname+"."+sp.Domain))
}
b.WriteString("manage_etc_hosts: true\npreserve_hostname: false\ndisable_root: false\nssh_pwauth: false\n")
fmt.Fprintf(&b, "ssh_authorized_keys: %s\n", yamlList(sp.SSHKeys))
fmt.Fprintf(&b, "users:\n - default\n - name: root\n ssh_authorized_keys: %s\n", yamlList(sp.SSHKeys))
if sp.PackageUpgrade || len(sp.Packages) > 0 {
b.WriteString("package_update: true\n")
}
if sp.PackageUpgrade {
b.WriteString("package_upgrade: true\npackage_reboot_if_required: true\n")
}
if len(sp.Packages) > 0 {
fmt.Fprintf(&b, "packages: %s\n", yamlList(sp.Packages))
}
fmt.Fprintf(&b, "phone_home:\n url: %s/api/v1/phone-home/%s\n post: [instance_id, hostname, fqdn, pub_key_ed25519]\n tries: 10\n", a.cfg.ServerURL, token)
b.WriteString("final_message: \"rollcall: deployed $hostname up after $UPTIME s\"\n")
ud = b.String()
}
return []seedFile{
{Path: "var/lib/cloud/seed/nocloud/meta-data", Mode: "0600", Content: meta},
{Path: "var/lib/cloud/seed/nocloud/user-data", Mode: "0600", Content: ud},
{Path: "etc/cloud/cloud.cfg.d/99-rollcall.cfg", Mode: "0644", Content: "# written by rollcall at deploy time\ndatasource_list: [ NoCloud, None ]\n"},
}
}
// ---------------------------------------------------------------- PXE
const deployTmpl = `# generated by rollcall %s
# machine %s job %s %s %s/%s -> %s
DEFAULT menu.c32
PROMPT 0
TIMEOUT 30
MENU TITLE %s - %s %s (job %s)
LABEL deploy
MENU LABEL Run deployment job (default in 3s)
MENU DEFAULT
KERNEL rollcall/vmlinuz
INITRD rollcall/initrd.img
APPEND %s
LABEL recommission
MENU LABEL Re-run commissioning
KERNEL rollcall/vmlinuz
INITRD rollcall/initrd.img
APPEND %s
LABEL local
MENU LABEL Boot from local disk
LOCALBOOT 0
`
const deployedTmpl = `# generated by rollcall %s
# machine %s deployed %s/%s
DEFAULT menu.c32
PROMPT 0
TIMEOUT 30
MENU TITLE %s - deployed %s (%s)
LABEL local
MENU LABEL Boot %s from local disk (default in 3s)
MENU DEFAULT
LOCALBOOT 0
LABEL recommission
MENU LABEL Re-run commissioning
KERNEL rollcall/vmlinuz
INITRD rollcall/initrd.img
APPEND %s
`
// deployAppend derives the deploy-mode cmdline from commission_append (same image, no agent token).
func (a *App) deployAppend(token string) string {
var parts []string
for _, p := range strings.Fields(a.cfg.CommissionAppend) {
if strings.HasPrefix(p, "rollcall.token=") || strings.HasPrefix(p, "rollcall.nopoweroff") {
continue
}
parts = append(parts, p)
}
parts = append(parts, "rollcall.mode=deploy", "rollcall.job="+token)
return strings.Join(parts, " ")
}
func (a *App) writePXEDeploy(m *Machine, j *Job, sp DeploySpec, token string) error {
body := fmt.Sprintf(deployTmpl, time.Now().Format(time.RFC3339), m.ID, j.ID, j.Kind, sp.OS, sp.Version, sp.Disk,
m.ID, j.Kind, sp.OS, j.ID, a.deployAppend(token), a.cfg.CommissionAppend)
return a.writePXEBody(m, body)
}
func (a *App) writePXEDeployed(m *Machine) error {
body := fmt.Sprintf(deployedTmpl, time.Now().Format(time.RFC3339), m.ID, deref(m.OS), deref(m.OSVersion),
deref(m.NodeName), deref(m.OS), m.ID, deref(m.OS), a.cfg.CommissionAppend)
return a.writePXEBody(m, body)
}
// ---------------------------------------------------------------- job creation
type deployRequest struct {
OS string `json:"os"`
Version string `json:"version"`
Disk string `json:"disk"`
Hostname string `json:"hostname"`
SSHKeys []string `json:"ssh_keys"`
PackageUpgrade bool `json:"package_upgrade"`
Packages []string `json:"packages"`
UserData string `json:"user_data"`
Boot bool `json:"boot"`
Wipe string `json:"wipe"`
}
func (a *App) configSSHKeys() []string {
var keys []string
for _, k := range toList(a.cfg.ClusterVars["ssh_keys"]) {
if s := str(k); s != "" {
keys = append(keys, s)
}
}
return keys
}
// startJob creates the job row, PXE entry and (optionally) power-cycles the machine.
func (a *App) startJob(ctx context.Context, m *Machine, kind string, sp DeploySpec, boot bool) (obj, error) {
if m.Profile != nil {
return nil, fmt.Errorf("machine is a stateless node (profile %s); unassign it first", *m.Profile)
}
if j, err := a.store.ActiveJob(ctx, m.ID); err == nil {
return nil, fmt.Errorf("job %s (%s, %s) is still active; cancel it first", j.ID, j.Kind, j.State)
}
token := randomToken(16)
sum := sha256.Sum256([]byte(token))
sealed, err := a.sealer.Seal(token, m.ID)
if err != nil {
return nil, err
}
specB, _ := json.Marshal(sp)
j := &Job{ID: "j-" + randomToken(5), MachineID: m.ID, Kind: kind, State: "pending", Spec: specB}
if err := a.store.CreateJob(ctx, j, hex.EncodeToString(sum[:]), sealed); err != nil {
return nil, err
}
state := "deploying"
if kind == "release" {
state = "releasing"
}
_ = a.store.SetState(ctx, m.ID, state)
if err := a.writePXEDeploy(m, j, sp, token); err != nil {
return nil, err
}
_ = a.store.AddEvent(ctx, m.ID, kind+" requested", fmt.Sprintf("job=%s os=%s/%s disk=%s hostname=%s", j.ID, sp.OS, sp.Version, sp.Disk, sp.Hostname))
out := obj{"id": m.ID, "job": j.ID, "state": state, "os": sp.OS, "version": sp.Version, "disk": sp.Disk, "hostname": sp.Hostname}
if boot {
out["ipmi"], out["ipmi_ok"] = a.pxeBoot(ctx, m)
}
log.Printf("%s %s: job %s os=%s/%s disk=%s", kind, m.ID, j.ID, sp.OS, sp.Version, sp.Disk)
return out, nil
}
func (a *App) handleDeploy(w http.ResponseWriter, r *http.Request) {
m, err := a.store.Get(r.Context(), r.PathValue("id"))
if err != nil {
fail(w, 500, err)
return
}
var req deployRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil || req.OS == "" {
fail(w, 400, fmt.Errorf("os is required"))
return
}
meta, err := osResolve(a.cfg.OSDir, req.OS, req.Version)
if err != nil {
fail(w, 404, err)
return
}
disk, err := pickDisk(inventoryDisks(m), req.Disk, meta.RawSize)
if err != nil {
fail(w, 409, err)
return
}
// node name + management IP: keep an existing one, otherwise allocate like a stateless node
name, mgmt, ib := deref(m.NodeName), deref(m.MgmtIP), deref(m.IBIP)
if req.Hostname != "" && req.Hostname != name {
name = ""
}
if name == "" {
if name, mgmt, ib, err = a.allocate(r.Context(), str(a.cfg.ClusterVars["name_format"]), req.Hostname); err != nil {
fail(w, 409, err)
return
}
if err := a.store.SetNodeAddr(r.Context(), m.ID, name, mgmt, ib); err != nil {
fail(w, 500, err)
return
}
m, _ = a.store.Get(r.Context(), m.ID)
if err := a.exportDnsmasq(r.Context()); err != nil {
log.Printf("dnsmasq export: %v", err)
}
}
keys := req.SSHKeys
if keys == nil {
keys = a.configSSHKeys()
}
sp := DeploySpec{OS: meta.Name, Version: meta.Version,
ImageURL: a.cfg.ServerURL + "/os/" + meta.Name + "/" + meta.Version + "/disk.raw.zst",
SHA256: meta.SHA256, Size: meta.Size, RawSize: meta.RawSize, Disk: disk.Path, DiskSerial: disk.Serial,
Hostname: name, Domain: str(a.cfg.ClusterVars["domain"]), SSHKeys: keys, PackageUpgrade: req.PackageUpgrade,
Packages: req.Packages, UserData: req.UserData}
out, err := a.startJob(r.Context(), m, "deploy", sp, req.Boot)
if err != nil {
fail(w, 409, err)
return
}
out["mgmt_ip"] = mgmt
writeJSON(w, 200, out)
}
func (a *App) handleRelease(w http.ResponseWriter, r *http.Request) {
m, err := a.store.Get(r.Context(), r.PathValue("id"))
if err != nil {
fail(w, 500, err)
return
}
var req deployRequest
_ = json.NewDecoder(r.Body).Decode(&req)
if req.Wipe == "" {
req.Wipe = "quick"
}
disk, err := pickDisk(inventoryDisks(m), req.Disk, 0)
if err != nil {
fail(w, 409, err)
return
}
sp := DeploySpec{OS: deref(m.OS), Version: deref(m.OSVersion), Disk: disk.Path, DiskSerial: disk.Serial, Wipe: req.Wipe, Hostname: deref(m.NodeName)}
out, err := a.startJob(r.Context(), m, "release", sp, req.Boot)
if err != nil {
fail(w, 409, err)
return
}
writeJSON(w, 200, out)
}
func (a *App) handleJobs(w http.ResponseWriter, r *http.Request) {
m, err := a.store.Get(r.Context(), r.PathValue("id"))
if err != nil {
fail(w, 500, err)
return
}
jobs, err := a.store.Jobs(r.Context(), m.ID, 20)
if err != nil {
fail(w, 500, err)
return
}
writeJSON(w, 200, jobs)
}
func (a *App) handleJobCancel(w http.ResponseWriter, r *http.Request) {
m, err := a.store.Get(r.Context(), r.PathValue("id"))
if err != nil {
fail(w, 500, err)
return
}
j, err := a.store.ActiveJob(r.Context(), m.ID)
if err != nil {
fail(w, 404, fmt.Errorf("no active job"))
return
}
_ = a.store.FinishJob(r.Context(), j.ID, "cancelled", "cancelled by operator")
a.afterJob(r.Context(), m, j, "cancelled", "cancelled by operator")
writeJSON(w, 200, obj{"id": m.ID, "job": j.ID, "state": "cancelled"})
}
// afterJob settles machine state and the PXE entry once a job stopped (done/failed/cancelled).
func (a *App) afterJob(ctx context.Context, m *Machine, j *Job, state, errMsg string) {
sp := j.spec()
switch {
case state == "done" && j.Kind == "deploy":
_ = a.store.SetDeployed(ctx, m.ID, sp.OS, sp.Version)
m, _ = a.store.Get(ctx, m.ID)
if err := a.writePXEDeployed(m); err != nil {
log.Printf("pxe write %s: %v", m.ID, err)
}
_ = a.store.AddEvent(ctx, m.ID, "deployed", fmt.Sprintf("job=%s os=%s/%s disk=%s", j.ID, sp.OS, sp.Version, sp.Disk))
case state == "done" && j.Kind == "release":
_ = a.store.SetReleased(ctx, m.ID)
m, _ = a.store.Get(ctx, m.ID)
_ = a.writePXE(m)
_ = a.store.AddEvent(ctx, m.ID, "released", fmt.Sprintf("job=%s disk=%s wipe=%s", j.ID, sp.Disk, sp.Wipe))
default:
_ = a.store.SetState(ctx, m.ID, j.Kind+"-failed")
m, _ = a.store.Get(ctx, m.ID)
_ = a.writePXE(m) // next PXE boot: commissioned menu (power off), no retry loop
_ = a.store.AddEvent(ctx, m.ID, j.Kind+" "+state, "job="+j.ID+" "+errMsg)
}
log.Printf("job %s (%s %s): %s %s", j.ID, j.Kind, m.ID, state, errMsg)
}
// ---------------------------------------------------------------- agent side (job token)
func (a *App) jobFromRequest(w http.ResponseWriter, r *http.Request) (*Job, string) {
tok := r.Header.Get("X-Rollcall-Job")
if tok == "" {
fail(w, 401, errors.New("X-Rollcall-Job required"))
return nil, ""
}
sum := sha256.Sum256([]byte(tok))
j, err := a.store.JobByToken(r.Context(), hex.EncodeToString(sum[:]))
if err != nil {
fail(w, 401, errors.New("unknown job token"))
return nil, ""
}
return j, tok
}
func (a *App) handleJobSelf(w http.ResponseWriter, r *http.Request) {
j, tok := a.jobFromRequest(w, r)
if j == nil {
return
}
if j.State != "pending" && j.State != "running" {
fail(w, 409, fmt.Errorf("job %s is %s", j.ID, j.State))
return
}
sp := j.spec()
out := obj{"id": j.ID, "machine_id": j.MachineID, "kind": j.Kind, "spec": sp}
if j.Kind == "deploy" {
out["files"] = a.renderSeed(j, sp, tok)
}
if j.State == "pending" {
_ = a.store.JobProgress(r.Context(), j.ID, []byte(`{"phase":"started","pct":0}`))
_ = a.store.AddEvent(r.Context(), j.MachineID, j.Kind+" started", "job="+j.ID+" from "+r.RemoteAddr)
}
writeJSON(w, 200, out)
}
func (a *App) handleJobProgress(w http.ResponseWriter, r *http.Request) {
j, _ := a.jobFromRequest(w, r)
if j == nil {
return
}
var p struct {
State string `json:"state"` // "" (progress), done, failed
Phase string `json:"phase"`
Pct float64 `json:"pct"`
Msg string `json:"msg"`
Error string `json:"error"`
Details map[string]any `json:"details"`
}
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&p); err != nil {
fail(w, 400, err)
return
}
if j.State != "pending" && j.State != "running" {
writeJSON(w, 200, obj{"job": j.ID, "state": j.State, "next": "poweroff"})
return
}
prog, _ := json.Marshal(obj{"phase": p.Phase, "pct": p.Pct, "msg": p.Msg, "details": p.Details, "updated": time.Now()})
_ = a.store.JobProgress(r.Context(), j.ID, prog)
m, err := a.store.Get(r.Context(), j.MachineID)
if err != nil {
fail(w, 500, err)
return
}
next := "continue"
switch p.State {
case "done":
_ = a.store.FinishJob(r.Context(), j.ID, "done", "")
a.afterJob(r.Context(), m, j, "done", "")
next = "reboot"
if j.Kind == "release" {
next = "poweroff"
}
case "failed":
if p.Error == "" {
p.Error = p.Msg
}
_ = a.store.FinishJob(r.Context(), j.ID, "failed", p.Error)
a.afterJob(r.Context(), m, j, "failed", p.Error)
next = "stay"
}
writeJSON(w, 200, obj{"job": j.ID, "state": p.State, "next": next})
}
// handlePhoneHome is cloud-init's phone_home from the freshly deployed OS (form-encoded POST).
func (a *App) handlePhoneHome(w http.ResponseWriter, r *http.Request) {
sum := sha256.Sum256([]byte(r.PathValue("token")))
j, err := a.store.JobByToken(r.Context(), hex.EncodeToString(sum[:]))
if err != nil || j.Kind != "deploy" {
fail(w, 404, errNotFound)
return
}
_ = r.ParseForm()
_ = a.store.MarkPhoneHome(r.Context(), j.MachineID)
detail := fmt.Sprintf("job=%s from %s hostname=%s fqdn=%s", j.ID, r.RemoteAddr, r.Form.Get("hostname"), r.Form.Get("fqdn"))
if k := strings.TrimSpace(r.Form.Get("pub_key_ed25519")); k != "" {
detail += " hostkey=" + k
}
_ = a.store.AddEvent(r.Context(), j.MachineID, "phone-home", detail)
log.Printf("phone-home %s: %s", j.MachineID, detail)
writeJSON(w, 200, obj{"ok": true})
}
+1
View File
@@ -11,6 +11,7 @@ require (
github.com/jackc/pgpassfile v1.0.0 // indirect github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/klauspost/compress v1.18.0 // indirect
golang.org/x/sync v0.13.0 // indirect golang.org/x/sync v0.13.0 // indirect
golang.org/x/sys v0.32.0 // indirect golang.org/x/sys v0.32.0 // indirect
golang.org/x/text v0.24.0 // indirect golang.org/x/text v0.24.0 // indirect
+2
View File
@@ -9,6 +9,8 @@ github.com/jackc/pgx/v5 v5.7.5 h1:JHGfMnQY+IEtGM63d+NGMjoRpysB2JBwDr5fsngwmJs=
github.com/jackc/pgx/v5 v5.7.5/go.mod h1:aruU7o91Tc2q2cFp5h4uP3f6ztExVpyVv88Xl/8Vl8M= github.com/jackc/pgx/v5 v5.7.5/go.mod h1:aruU7o91Tc2q2cFp5h4uP3f6ztExVpyVv88Xl/8Vl8M=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo= github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+13 -1
View File
@@ -36,10 +36,16 @@ func usage() {
rollcall unassign ID rollcall unassign ID
rollcall overlay ID [-stage system|runtime] [-extract DIR] rollcall overlay ID [-stage system|runtime] [-extract DIR]
rollcall render PROFILE PATH [-stage system|runtime] rollcall render PROFILE PATH [-stage system|runtime]
rollcall os list | sync [NAME...] [-force] | import NAME FILE [-family F] [-version V] | promote NAME VERSION | rm NAME VERSION
rollcall deploy ID OS [-version V] [-disk /dev/sdX] [-hostname H] [-upgrade] [-user-data FILE] [-boot] [-watch]
rollcall release ID [-boot] [-watch]
rollcall jobs ID [-watch] | job cancel ID
env: ROLLCALL_URL (default from config listen), ROLLCALL_TOKEN, ROLLCALL_CONFIG`) env: ROLLCALL_URL (default from config listen), ROLLCALL_TOKEN, ROLLCALL_CONFIG`)
os.Exit(2) os.Exit(2)
} }
func newFlags(name string) *flag.FlagSet { return flag.NewFlagSet(name, flag.ExitOnError) }
func main() { func main() {
log.SetFlags(log.Ldate | log.Ltime | log.Lmicroseconds) log.SetFlags(log.Ldate | log.Ltime | log.Lmicroseconds)
if len(os.Args) < 2 { if len(os.Args) < 2 {
@@ -57,11 +63,17 @@ func main() {
case "keygen": case "keygen":
fmt.Println(randomToken(24)) fmt.Println(randomToken(24))
case "list", "show", "events", "power", "bootdev", "recommission", "delete", case "list", "show", "events", "power", "bootdev", "recommission", "delete",
"images", "profile", "assign", "unassign", "overlay", "render": "images", "profile", "assign", "unassign", "overlay", "render",
"deploy", "release", "jobs", "job":
if err := runClient(cmd, args); err != nil { if err := runClient(cmd, args); err != nil {
fmt.Fprintln(os.Stderr, "error:", err) fmt.Fprintln(os.Stderr, "error:", err)
os.Exit(1) os.Exit(1)
} }
case "os":
if err := osCmd(args); err != nil {
fmt.Fprintln(os.Stderr, "error:", err)
os.Exit(1)
}
default: default:
usage() usage()
} }
+694
View File
@@ -0,0 +1,694 @@
package main
// OS image catalog for disk-based deployment.
//
// <os_dir>/<name>/<version>/disk.raw.zst zstd-compressed raw disk image
// <os_dir>/<name>/<version>/meta.json OSMeta
// <os_dir>/<name>/current -> <version> symlink flipped atomically by sync/promote
//
// Sources (upstream "latest" URLs + vendor checksum files) live in
// os_sources (default /etc/rollcall/os-sources.json). `rollcall os sync`
// downloads only when the vendor checksum changed, verifies it, converts
// to raw with qemu-img, compresses with zstd and prunes old versions.
import (
"bufio"
"context"
"crypto/sha256"
"crypto/sha512"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"os/exec"
"path"
"path/filepath"
"regexp"
"runtime"
"sort"
"strconv"
"strings"
"time"
"github.com/klauspost/compress/zstd"
)
type OSSource struct {
Name string `json:"name"`
URL string `json:"url"`
ChecksumURL string `json:"checksum_url"`
Checksum string `json:"checksum"` // sha256 | sha512 (type of the vendor checksum file)
Family string `json:"family"` // debian | rhel (informational)
Description string `json:"description"`
Track string `json:"track"` // latest (default): new versions become current; pinned: only on promote
Keep int `json:"keep"` // versions to keep (default 2)
}
type OSMeta struct {
Name string `json:"name"`
Version string `json:"version"`
Family string `json:"family"`
Description string `json:"description"`
SourceURL string `json:"source_url"`
SourceSHA256 string `json:"source_sha256"`
SourceSHA512 string `json:"source_sha512"`
SourceModified string `json:"source_modified"`
SourceFormat string `json:"source_format"`
SHA256 string `json:"sha256"` // of disk.raw.zst
Size int64 `json:"size"` // compressed bytes
RawSize int64 `json:"raw_size"` // virtual disk size the image needs
Imported time.Time `json:"imported"`
Current bool `json:"current,omitempty"`
}
type OSImage struct {
Name string `json:"name"`
Current *OSMeta `json:"current"`
Versions []OSMeta `json:"versions"`
Source *OSSource `json:"source"`
Warning string `json:"warning,omitempty"`
}
func loadSources(p string) ([]OSSource, error) {
b, err := os.ReadFile(p)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return nil, nil
}
return nil, err
}
var src []OSSource
if err := json.Unmarshal(b, &src); err != nil {
return nil, fmt.Errorf("%s: %w", p, err)
}
for i := range src {
if src[i].Track == "" {
src[i].Track = "latest"
}
if src[i].Keep <= 0 {
src[i].Keep = 2
}
if src[i].Checksum == "" {
src[i].Checksum = "sha256"
}
if strings.ContainsAny(src[i].Name, "/ \t") || src[i].Name == "" {
return nil, fmt.Errorf("%s: bad source name %q", p, src[i].Name)
}
}
return src, nil
}
var versionRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*$`)
func osVersions(dir, name string) ([]OSMeta, string) {
ents, err := os.ReadDir(filepath.Join(dir, name))
if err != nil {
return nil, ""
}
cur, _ := os.Readlink(filepath.Join(dir, name, "current"))
cur = path.Base(cur)
var out []OSMeta
for _, e := range ents {
if !e.IsDir() || !versionRe.MatchString(e.Name()) || strings.HasSuffix(e.Name(), ".tmp") {
continue
}
b, err := os.ReadFile(filepath.Join(dir, name, e.Name(), "meta.json"))
if err != nil {
continue
}
var m OSMeta
if json.Unmarshal(b, &m) != nil {
continue
}
m.Name, m.Version = name, e.Name()
m.Current = e.Name() == cur
out = append(out, m)
}
sort.Slice(out, func(i, j int) bool { return out[i].Imported.After(out[j].Imported) })
return out, cur
}
func osCatalog(dir, sourcesFile string) ([]OSImage, error) {
src, err := loadSources(sourcesFile)
if err != nil {
return nil, err
}
byName := map[string]*OSSource{}
for i := range src {
byName[src[i].Name] = &src[i]
}
names := map[string]bool{}
for n := range byName {
names[n] = true
}
if ents, err := os.ReadDir(dir); err == nil {
for _, e := range ents {
if e.IsDir() {
names[e.Name()] = true
}
}
}
out := []OSImage{}
for n := range names {
img := OSImage{Name: n, Source: byName[n]}
vs, cur := osVersions(dir, n)
img.Versions = vs
for i := range vs {
if vs[i].Version == cur {
img.Current = &vs[i]
}
}
switch {
case img.Current == nil && len(vs) == 0:
img.Warning = "not imported"
case img.Current == nil:
img.Warning = "no current version (promote one)"
case time.Since(img.Current.Imported) > 45*24*time.Hour:
img.Warning = fmt.Sprintf("current version is %d days old", int(time.Since(img.Current.Imported).Hours()/24))
}
out = append(out, img)
}
sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
return out, nil
}
// osResolve returns the metadata of name/version ("" or "current" = current).
func osResolve(dir, name, version string) (*OSMeta, error) {
if strings.ContainsAny(name, "/ \t") || name == "" {
return nil, fmt.Errorf("bad os name %q", name)
}
vs, cur := osVersions(dir, name)
if version == "" || version == "current" {
version = cur
if version == "" {
return nil, fmt.Errorf("os %s has no current version", name)
}
}
for i := range vs {
if vs[i].Version == version {
if _, err := os.Stat(filepath.Join(dir, name, version, "disk.raw.zst")); err != nil {
return nil, err
}
return &vs[i], nil
}
}
return nil, fmt.Errorf("os %s version %s: %w", name, version, errNotFound)
}
func osSetCurrent(dir, name, version string) error {
if _, err := os.Stat(filepath.Join(dir, name, version, "disk.raw.zst")); err != nil {
return err
}
link := filepath.Join(dir, name, "current")
tmp := link + ".tmp"
_ = os.Remove(tmp)
if err := os.Symlink(version, tmp); err != nil {
return err
}
return os.Rename(tmp, link)
}
func osRemove(dir, name, version string) error {
if name == "" || version == "" || !versionRe.MatchString(version) {
return fmt.Errorf("name and version required")
}
_, cur := osVersions(dir, name)
if cur == version {
return fmt.Errorf("%s/%s is the current version; promote another one first", name, version)
}
return os.RemoveAll(filepath.Join(dir, name, version))
}
// ---------------------------------------------------------------- import pipeline
type importOpts struct {
Dir string
Name string
Version string // "" = derive
Family string
Description string
SourceURL string
Modified string
SHA256 string // of the source file, if already known
SHA512 string
Log io.Writer
}
func logf(w io.Writer, f string, a ...any) {
if w != nil {
fmt.Fprintf(w, f+"\n", a...)
}
}
// osImportFile converts src (qcow2/raw/vmdk...) into <dir>/<name>/<version>/ and returns the meta.
func osImportFile(ctx context.Context, src string, o importOpts) (*OSMeta, error) {
if !versionRe.MatchString(o.Name) {
return nil, fmt.Errorf("bad name %q", o.Name)
}
info, err := qemuInfo(ctx, src)
if err != nil {
return nil, err
}
if o.SHA256 == "" || o.SHA512 == "" {
logf(o.Log, "hashing %s", src)
if o.SHA256, o.SHA512, err = hashFile(src); err != nil {
return nil, err
}
}
if o.Version == "" {
d := o.Modified
if d == "" {
st, _ := os.Stat(src)
d = st.ModTime().UTC().Format("20060102")
}
o.Version = d + "-" + o.SHA256[:6]
}
if !versionRe.MatchString(o.Version) {
return nil, fmt.Errorf("bad version %q", o.Version)
}
final := filepath.Join(o.Dir, o.Name, o.Version)
if _, err := os.Stat(filepath.Join(final, "meta.json")); err == nil {
return nil, fmt.Errorf("%s/%s already exists", o.Name, o.Version)
}
work := final + ".tmp"
_ = os.RemoveAll(work)
if err := os.MkdirAll(work, 0o755); err != nil {
return nil, err
}
defer os.RemoveAll(work)
raw := src
if info.Format != "raw" {
raw = filepath.Join(work, "disk.raw")
logf(o.Log, "converting %s (%s, %s virtual) to raw", filepath.Base(src), info.Format, human(float64(info.VirtualSize)))
cmd := exec.CommandContext(ctx, "qemu-img", "convert", "-O", "raw", "-S", "4k", src, raw)
if out, err := cmd.CombinedOutput(); err != nil {
return nil, fmt.Errorf("qemu-img convert: %v: %s", err, strings.TrimSpace(string(out)))
}
}
logf(o.Log, "compressing to disk.raw.zst")
sum, size, err := compressZstd(raw, filepath.Join(work, "disk.raw.zst"))
if err != nil {
return nil, err
}
if raw != src {
_ = os.Remove(raw)
}
m := &OSMeta{Name: o.Name, Version: o.Version, Family: o.Family, Description: o.Description, SourceURL: o.SourceURL,
SourceSHA256: o.SHA256, SourceSHA512: o.SHA512, SourceModified: o.Modified, SourceFormat: info.Format,
SHA256: sum, Size: size, RawSize: info.VirtualSize, Imported: time.Now().UTC()}
mb, _ := json.MarshalIndent(m, "", " ")
if err := os.WriteFile(filepath.Join(work, "meta.json"), mb, 0o644); err != nil {
return nil, err
}
if err := os.Rename(work, final); err != nil {
return nil, err
}
logf(o.Log, "imported %s/%s (%s compressed, %s raw)", o.Name, o.Version, human(float64(size)), human(float64(info.VirtualSize)))
return m, nil
}
type qemuImgInfo struct {
Format string `json:"format"`
VirtualSize int64 `json:"virtual-size"`
}
func qemuInfo(ctx context.Context, p string) (*qemuImgInfo, error) {
out, err := exec.CommandContext(ctx, "qemu-img", "info", "--output=json", p).Output()
if err != nil {
return nil, fmt.Errorf("qemu-img info %s: %w", p, err)
}
var i qemuImgInfo
if err := json.Unmarshal(out, &i); err != nil {
return nil, err
}
return &i, nil
}
func hashFile(p string) (s256, s512 string, err error) {
f, err := os.Open(p)
if err != nil {
return "", "", err
}
defer f.Close()
h1, h2 := sha256.New(), sha512.New()
if _, err := io.Copy(io.MultiWriter(h1, h2), bufio.NewReaderSize(f, 4<<20)); err != nil {
return "", "", err
}
return hex.EncodeToString(h1.Sum(nil)), hex.EncodeToString(h2.Sum(nil)), nil
}
func compressZstd(src, dst string) (sum string, size int64, err error) {
in, err := os.Open(src)
if err != nil {
return "", 0, err
}
defer in.Close()
out, err := os.Create(dst)
if err != nil {
return "", 0, err
}
defer out.Close()
h := sha256.New()
cw := &countWriter{w: io.MultiWriter(out, h)}
enc, err := zstd.NewWriter(cw, zstd.WithEncoderLevel(zstd.SpeedDefault), zstd.WithEncoderConcurrency(runtime.NumCPU()))
if err != nil {
return "", 0, err
}
if _, err := io.Copy(enc, bufio.NewReaderSize(in, 4<<20)); err != nil {
enc.Close()
return "", 0, err
}
if err := enc.Close(); err != nil {
return "", 0, err
}
if err := out.Sync(); err != nil {
return "", 0, err
}
return hex.EncodeToString(h.Sum(nil)), cw.n, nil
}
type countWriter struct {
w io.Writer
n int64
}
func (c *countWriter) Write(p []byte) (int, error) {
n, err := c.w.Write(p)
c.n += int64(n)
return n, err
}
// ---------------------------------------------------------------- sync
// parseChecksums understands "HASH file", "HASH *file" and "ALGO (file) = HASH".
func parseChecksums(txt string) map[string]string {
out := map[string]string{}
bsd := regexp.MustCompile(`^\s*[A-Za-z0-9]+\s*\((.+)\)\s*=\s*([0-9a-fA-F]+)\s*$`)
gnu := regexp.MustCompile(`^\s*([0-9a-fA-F]{32,128})\s+\*?(.+?)\s*$`)
for _, line := range strings.Split(txt, "\n") {
if m := bsd.FindStringSubmatch(line); m != nil {
out[path.Base(m[1])] = strings.ToLower(m[2])
} else if m := gnu.FindStringSubmatch(line); m != nil {
out[path.Base(m[2])] = strings.ToLower(m[1])
}
}
return out
}
func httpGetText(ctx context.Context, url string) (string, error) {
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode != 200 {
return "", fmt.Errorf("%s: %s", url, resp.Status)
}
b, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
return string(b), err
}
// upstreamChecksum fetches the vendor checksum for the source's file.
func upstreamChecksum(ctx context.Context, s OSSource) (string, error) {
txt, err := httpGetText(ctx, s.ChecksumURL)
if err != nil {
return "", err
}
sums := parseChecksums(txt)
if v, ok := sums[path.Base(s.URL)]; ok {
return v, nil
}
if len(sums) == 1 {
for _, v := range sums {
return v, nil
}
}
return "", fmt.Errorf("%s: no entry for %s", s.ChecksumURL, path.Base(s.URL))
}
func metaChecksum(m *OSMeta, typ string) string {
if typ == "sha512" {
return m.SourceSHA512
}
return m.SourceSHA256
}
// download streams url to dst while hashing; verifies against want (hex, type typ).
func download(ctx context.Context, url, dst, typ, want string, log io.Writer) (s256, s512, modified string, err error) {
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return "", "", "", err
}
defer resp.Body.Close()
if resp.StatusCode != 200 {
return "", "", "", fmt.Errorf("%s: %s", url, resp.Status)
}
if t, err := http.ParseTime(resp.Header.Get("Last-Modified")); err == nil {
modified = t.UTC().Format("20060102")
}
f, err := os.Create(dst)
if err != nil {
return "", "", "", err
}
defer f.Close()
h1, h2 := sha256.New(), sha512.New()
pr := &progressReader{r: resp.Body, total: resp.ContentLength, log: log, last: time.Now()}
if _, err := io.Copy(io.MultiWriter(f, h1, h2), pr); err != nil {
return "", "", "", err
}
s256, s512 = hex.EncodeToString(h1.Sum(nil)), hex.EncodeToString(h2.Sum(nil))
got := s256
if typ == "sha512" {
got = s512
}
if want != "" && got != want {
return "", "", "", fmt.Errorf("%s: %s mismatch: vendor %s, downloaded %s", url, typ, want, got)
}
return s256, s512, modified, nil
}
type progressReader struct {
r io.Reader
n int64
total int64
log io.Writer
last time.Time
}
func (p *progressReader) Read(b []byte) (int, error) {
n, err := p.r.Read(b)
p.n += int64(n)
if p.log != nil && time.Since(p.last) > 10*time.Second {
p.last = time.Now()
if p.total > 0 {
logf(p.log, " %s / %s (%d%%)", human(float64(p.n)), human(float64(p.total)), p.n*100/p.total)
} else {
logf(p.log, " %s", human(float64(p.n)))
}
}
return n, err
}
// osSync refreshes the named sources (all when names is empty).
func osSync(ctx context.Context, cfg *Config, names []string, force bool, log io.Writer) error {
src, err := loadSources(cfg.OSSources)
if err != nil {
return err
}
if len(src) == 0 {
return fmt.Errorf("no sources in %s", cfg.OSSources)
}
want := map[string]bool{}
for _, n := range names {
want[n] = true
}
var failed []string
for _, s := range src {
if len(want) > 0 && !want[s.Name] {
continue
}
if err := syncOne(ctx, cfg, s, force, log); err != nil {
logf(log, "%s: FAILED: %v", s.Name, err)
failed = append(failed, s.Name)
}
}
if len(failed) > 0 {
return fmt.Errorf("failed: %s", strings.Join(failed, ", "))
}
return nil
}
func syncOne(ctx context.Context, cfg *Config, s OSSource, force bool, log io.Writer) error {
vs, cur := osVersions(cfg.OSDir, s.Name)
up, err := upstreamChecksum(ctx, s)
if err != nil {
return err
}
for i := range vs {
if metaChecksum(&vs[i], s.Checksum) == up {
if !force {
logf(log, "%s: up to date (%s, upstream %s unchanged)", s.Name, vs[i].Version, s.Checksum)
if cur == "" && s.Track == "latest" {
return osSetCurrent(cfg.OSDir, s.Name, vs[i].Version)
}
return osPrune(cfg.OSDir, s, log)
}
break
}
}
tmpDir := filepath.Join(cfg.OSDir, ".tmp")
if err := os.MkdirAll(tmpDir, 0o755); err != nil {
return err
}
dl := filepath.Join(tmpDir, s.Name+"-"+path.Base(s.URL))
defer os.Remove(dl)
logf(log, "%s: downloading %s", s.Name, s.URL)
s256, s512, mod, err := download(ctx, s.URL, dl, s.Checksum, up, log)
if err != nil {
return err
}
logf(log, "%s: vendor %s verified", s.Name, s.Checksum)
m, err := osImportFile(ctx, dl, importOpts{Dir: cfg.OSDir, Name: s.Name, Family: s.Family, Description: s.Description,
SourceURL: s.URL, Modified: mod, SHA256: s256, SHA512: s512, Log: log})
if err != nil {
return err
}
if s.Track == "latest" || cur == "" {
if err := osSetCurrent(cfg.OSDir, s.Name, m.Version); err != nil {
return err
}
logf(log, "%s: current -> %s", s.Name, m.Version)
} else {
logf(log, "%s: %s imported but pinned to %s (rollcall os promote %s %s)", s.Name, m.Version, cur, s.Name, m.Version)
}
return osPrune(cfg.OSDir, s, log)
}
// osPrune keeps the newest `keep` versions plus current; nothing younger than a day is removed
// so a deploy job that pinned a version keeps its file.
func osPrune(dir string, s OSSource, log io.Writer) error {
vs, cur := osVersions(dir, s.Name)
kept := 0
for _, v := range vs {
if v.Version == cur || kept < s.Keep {
kept++
continue
}
if time.Since(v.Imported) < 24*time.Hour {
continue
}
logf(log, "%s: pruning %s", s.Name, v.Version)
if err := os.RemoveAll(filepath.Join(dir, s.Name, v.Version)); err != nil {
return err
}
}
return nil
}
// ---------------------------------------------------------------- CLI (local, filesystem side)
func osCmd(args []string) error {
if len(args) < 1 {
usage()
}
cfg, err := loadConfig(envOr("ROLLCALL_CONFIG", "/etc/rollcall/config.json"))
if err != nil && args[0] != "list" {
return fmt.Errorf("os %s needs the server config: %w", args[0], err)
}
ctx := context.Background()
switch args[0] {
case "list":
if c, cerr := newClient(); cerr == nil {
var imgs []OSImage
if err := c.do("GET", "/api/v1/os", nil, &imgs); err != nil {
return err
}
printOSList(imgs)
return nil
}
imgs, err := osCatalog(cfg.OSDir, cfg.OSSources)
if err != nil {
return err
}
printOSList(imgs)
case "sync":
fs := newFlags("os sync")
force := fs.Bool("force", false, "re-import even when the upstream checksum is unchanged")
_ = fs.Parse(args[1:])
return osSync(ctx, cfg, fs.Args(), *force, os.Stdout)
case "import":
if len(args) < 3 {
usage()
}
fs := newFlags("os import")
fam := fs.String("family", "", "debian | rhel")
desc := fs.String("description", "", "description")
ver := fs.String("version", "", "version label (default <date>-<sha256 prefix>)")
cur := fs.Bool("current", true, "make it the current version")
_ = fs.Parse(args[3:])
m, err := osImportFile(ctx, args[2], importOpts{Dir: cfg.OSDir, Name: args[1], Version: *ver, Family: *fam, Description: *desc,
SourceURL: "file://" + args[2], Log: os.Stdout})
if err != nil {
return err
}
if *cur {
return osSetCurrent(cfg.OSDir, m.Name, m.Version)
}
case "promote":
if len(args) < 3 {
usage()
}
return osSetCurrent(cfg.OSDir, args[1], args[2])
case "rm":
if len(args) < 3 {
usage()
}
return osRemove(cfg.OSDir, args[1], args[2])
default:
usage()
}
return nil
}
func printOSList(imgs []OSImage) {
fmt.Printf("%-18s %-20s %-7s %8s %8s %-6s %-19s %s\n", "OS", "CURRENT", "FAMILY", "ZST", "RAW", "TRACK", "IMPORTED", "VERSIONS / WARNING")
for _, i := range imgs {
cur, fam, zst, raw, imp, track := "-", "", "", "", "", ""
if i.Source != nil {
track, fam = i.Source.Track, i.Source.Family
}
if i.Current != nil {
cur, zst, raw = i.Current.Version, human(float64(i.Current.Size)), human(float64(i.Current.RawSize))
imp = i.Current.Imported.Local().Format("2006-01-02 15:04")
if fam == "" {
fam = i.Current.Family
}
}
var vs []string
for _, v := range i.Versions {
vs = append(vs, v.Version)
}
extra := strconv.Itoa(len(i.Versions)) + " (" + strings.Join(vs, ", ") + ")"
if i.Warning != "" {
extra += " ! " + i.Warning
}
fmt.Printf("%-18s %-20s %-7s %8s %8s %-6s %-19s %s\n", i.Name, cur, fam, zst, raw, track, imp, extra)
}
}
// ---------------------------------------------------------------- HTTP
func (a *App) handleOS(w http.ResponseWriter, r *http.Request) {
imgs, err := osCatalog(a.cfg.OSDir, a.cfg.OSSources)
if err != nil {
fail(w, 500, err)
return
}
writeJSON(w, 200, imgs)
}
+28 -5
View File
@@ -81,7 +81,7 @@ func logging(next http.Handler) http.Handler {
start := time.Now() start := time.Now()
rw := &statusWriter{ResponseWriter: w, code: 200} rw := &statusWriter{ResponseWriter: w, code: 200}
next.ServeHTTP(rw, r) next.ServeHTTP(rw, r)
if (!strings.HasPrefix(r.URL.Path, "/rollcall/") && !strings.HasPrefix(r.URL.Path, "/images/")) || rw.code >= 400 { if (!strings.HasPrefix(r.URL.Path, "/rollcall/") && !strings.HasPrefix(r.URL.Path, "/images/") && !strings.HasPrefix(r.URL.Path, "/os/")) || rw.code >= 400 {
log.Printf("%s %s %s %d %s", r.RemoteAddr, r.Method, r.URL.RequestURI(), rw.code, time.Since(start).Round(time.Millisecond)) log.Printf("%s %s %s %d %s", r.RemoteAddr, r.Method, r.URL.RequestURI(), rw.code, time.Since(start).Round(time.Millisecond))
} }
}) })
@@ -120,10 +120,20 @@ func (a *App) routes() http.Handler {
mux.HandleFunc("GET /api/v1/machines/{id}/overlay", a.adminAuth(a.handleAdminOverlay)) mux.HandleFunc("GET /api/v1/machines/{id}/overlay", a.adminAuth(a.handleAdminOverlay))
mux.HandleFunc("GET /api/v1/render/{profile}/{path...}", a.adminAuth(a.handleRender)) mux.HandleFunc("GET /api/v1/render/{profile}/{path...}", a.adminAuth(a.handleRender))
mux.HandleFunc("GET /api/v1/nodes/self/overlay", a.handleNodeOverlay) // node token auth inside mux.HandleFunc("GET /api/v1/nodes/self/overlay", a.handleNodeOverlay) // node token auth inside
// disk-based deployment
mux.HandleFunc("GET /api/v1/os", a.adminAuth(a.handleOS))
mux.HandleFunc("POST /api/v1/machines/{id}/deploy", a.adminAuth(a.handleDeploy))
mux.HandleFunc("POST /api/v1/machines/{id}/release", a.adminAuth(a.handleRelease))
mux.HandleFunc("GET /api/v1/machines/{id}/jobs", a.adminAuth(a.handleJobs))
mux.HandleFunc("POST /api/v1/machines/{id}/jobs/cancel", a.adminAuth(a.handleJobCancel))
mux.HandleFunc("GET /api/v1/jobs/self", a.handleJobSelf) // job token auth inside
mux.HandleFunc("POST /api/v1/jobs/self/progress", a.handleJobProgress) // job token auth inside
mux.HandleFunc("POST /api/v1/phone-home/{token}", a.handlePhoneHome)
// image files for the live boot (no auth: the kernel/initrd/squashfs must be fetchable) // image files for the live boot (no auth: the kernel/initrd/squashfs must be fetchable)
imgDir := a.cfg.ImageDir imgDir := a.cfg.ImageDir
mux.Handle("GET /rollcall/", http.StripPrefix("/rollcall/", http.FileServer(http.Dir(imgDir)))) mux.Handle("GET /rollcall/", http.StripPrefix("/rollcall/", http.FileServer(http.Dir(imgDir))))
mux.Handle("GET /images/", http.StripPrefix("/images/", http.FileServer(http.Dir(a.cfg.ImagesDir)))) mux.Handle("GET /images/", http.StripPrefix("/images/", http.FileServer(http.Dir(a.cfg.ImagesDir))))
mux.Handle("GET /os/", http.StripPrefix("/os/", http.FileServer(http.Dir(a.cfg.OSDir))))
// embedded web UI // embedded web UI
sub, _ := fs.Sub(webFS, "web") sub, _ := fs.Sub(webFS, "web")
mux.Handle("GET /", http.FileServer(http.FS(sub))) mux.Handle("GET /", http.FileServer(http.FS(sub)))
@@ -225,6 +235,10 @@ type Summary struct {
MgmtIP *string `json:"mgmt_ip"` MgmtIP *string `json:"mgmt_ip"`
IBIP *string `json:"ib_ip"` IBIP *string `json:"ib_ip"`
BootedAt *time.Time `json:"booted_at"` BootedAt *time.Time `json:"booted_at"`
OS *string `json:"os"`
OSVersion *string `json:"os_version"`
DeployedAt *time.Time `json:"deployed_at"`
Job *Job `json:"job,omitempty"` // active deployment job, if any
} }
func num(v any) float64 { func num(v any) float64 {
@@ -241,7 +255,8 @@ func summarize(m *Machine) Summary {
MemoryBytes: num(dig(inv, "memory", "total_bytes")), Disks: num(dig(inv, "storage", "disk_count")), MemoryBytes: num(dig(inv, "memory", "total_bytes")), Disks: num(dig(inv, "storage", "disk_count")),
StorageBytes: num(dig(inv, "storage", "total_bytes")), NICs: listLen(inv["network"]), StorageBytes: num(dig(inv, "storage", "total_bytes")), NICs: listLen(inv["network"]),
IBHCAs: listLen(dig(inv, "infiniband", "hcas")), GPUs: listLen(inv["gpus"]), IBHCAs: listLen(dig(inv, "infiniband", "hcas")), GPUs: listLen(inv["gpus"]),
Profile: m.Profile, NodeName: m.NodeName, MgmtIP: m.MgmtIP, IBIP: m.IBIP, BootedAt: m.BootedAt} Profile: m.Profile, NodeName: m.NodeName, MgmtIP: m.MgmtIP, IBIP: m.IBIP, BootedAt: m.BootedAt,
OS: m.OS, OSVersion: m.OSVersion, DeployedAt: m.DeployedAt}
if m.BMC != nil { if m.BMC != nil {
var b obj var b obj
if json.Unmarshal(m.BMC, &b) == nil { if json.Unmarshal(m.BMC, &b) == nil {
@@ -415,9 +430,12 @@ func (a *App) handleList(w http.ResponseWriter, r *http.Request) {
fail(w, 500, err) fail(w, 500, err)
return return
} }
active, _ := a.store.ActiveJobs(r.Context())
out := make([]Summary, 0, len(ms)) out := make([]Summary, 0, len(ms))
for _, m := range ms { for _, m := range ms {
out = append(out, summarize(m)) s := summarize(m)
s.Job = active[m.ID]
out = append(out, s)
} }
writeJSON(w, 200, out) writeJSON(w, 200, out)
} }
@@ -429,11 +447,16 @@ func (a *App) handleGet(w http.ResponseWriter, r *http.Request) {
return return
} }
evs, _ := a.store.Events(r.Context(), m.ID) evs, _ := a.store.Events(r.Context(), m.ID)
jobs, _ := a.store.Jobs(r.Context(), m.ID, 10)
s := summarize(m)
if j, err := a.store.ActiveJob(r.Context(), m.ID); err == nil {
s.Job = j
}
var bmc, inv any var bmc, inv any
_ = json.Unmarshal(m.BMC, &bmc) _ = json.Unmarshal(m.BMC, &bmc)
_ = json.Unmarshal(m.Inventory, &inv) _ = json.Unmarshal(m.Inventory, &inv)
writeJSON(w, 200, obj{"summary": summarize(m), "bmc": bmc, "has_bmc_secret": m.bmcSecret != nil, writeJSON(w, 200, obj{"summary": s, "bmc": bmc, "has_bmc_secret": m.bmcSecret != nil,
"inventory": inv, "events": evs}) "inventory": inv, "events": evs, "jobs": jobs})
} }
func (a *App) handleDelete(w http.ResponseWriter, r *http.Request) { func (a *App) handleDelete(w http.ResponseWriter, r *http.Request) {
+6 -3
View File
@@ -185,7 +185,7 @@ func parsePool(spec string) ([]string, error) {
return out, nil return out, nil
} }
func (a *App) allocate(ctx context.Context, p *Profile, wantName string) (name, mgmt, ib string, err error) { func (a *App) allocate(ctx context.Context, nameFormat, wantName string) (name, mgmt, ib string, err error) {
ms, err := a.store.List(ctx) ms, err := a.store.List(ctx)
if err != nil { if err != nil {
return return
@@ -204,7 +204,7 @@ func (a *App) allocate(ctx context.Context, p *Profile, wantName string) (name,
} }
name = wantName name = wantName
if name == "" { if name == "" {
f := p.NameFormat f := nameFormat
if f == "" { if f == "" {
f = "node%03d" f = "node%03d"
} }
@@ -601,6 +601,9 @@ func (a *App) writePXEAny(ctx context.Context, m *Machine) error {
} }
return a.writePXEStateless(m, p, token) return a.writePXEStateless(m, p, token)
} }
if m.State == "deployed" {
return a.writePXEDeployed(m)
}
return a.writePXE(m) return a.writePXE(m)
} }
@@ -722,7 +725,7 @@ func (a *App) handleAssign(w http.ResponseWriter, r *http.Request) {
// keep the existing name/IPs when re-assigning an already named node // keep the existing name/IPs when re-assigning an already named node
name, mgmt, ib := deref(m.NodeName), deref(m.MgmtIP), deref(m.IBIP) name, mgmt, ib := deref(m.NodeName), deref(m.MgmtIP), deref(m.IBIP)
if name == "" || (req.Name != "" && req.Name != name) { if name == "" || (req.Name != "" && req.Name != name) {
if name, mgmt, ib, err = a.allocate(r.Context(), p, req.Name); err != nil { if name, mgmt, ib, err = a.allocate(r.Context(), p.NameFormat, req.Name); err != nil {
fail(w, 409, err) fail(w, 409, err)
return return
} }
+29 -2
View File
@@ -56,6 +56,25 @@ ALTER TABLE machines ADD COLUMN IF NOT EXISTS node_token bytea;
ALTER TABLE machines ADD COLUMN IF NOT EXISTS host_key bytea; ALTER TABLE machines ADD COLUMN IF NOT EXISTS host_key bytea;
ALTER TABLE machines ADD COLUMN IF NOT EXISTS booted_at timestamptz; ALTER TABLE machines ADD COLUMN IF NOT EXISTS booted_at timestamptz;
CREATE INDEX IF NOT EXISTS machines_node_token ON machines(node_token_hash); CREATE INDEX IF NOT EXISTS machines_node_token ON machines(node_token_hash);
ALTER TABLE machines ADD COLUMN IF NOT EXISTS os text;
ALTER TABLE machines ADD COLUMN IF NOT EXISTS os_version text;
ALTER TABLE machines ADD COLUMN IF NOT EXISTS deployed_at timestamptz;
CREATE TABLE IF NOT EXISTS jobs (
id text PRIMARY KEY,
machine_id text NOT NULL,
kind text NOT NULL,
state text NOT NULL DEFAULT 'pending',
spec jsonb NOT NULL,
progress jsonb,
token_hash text,
token bytea,
created timestamptz NOT NULL DEFAULT now(),
started timestamptz,
finished timestamptz,
error text
);
CREATE INDEX IF NOT EXISTS jobs_machine ON jobs(machine_id, created);
CREATE INDEX IF NOT EXISTS jobs_token ON jobs(token_hash);
` `
var errNotFound = errors.New("not found") var errNotFound = errors.New("not found")
@@ -81,6 +100,9 @@ type Machine struct {
MgmtIP *string `json:"mgmt_ip"` MgmtIP *string `json:"mgmt_ip"`
IBIP *string `json:"ib_ip"` IBIP *string `json:"ib_ip"`
BootedAt *time.Time `json:"booted_at"` BootedAt *time.Time `json:"booted_at"`
OS *string `json:"os"`
OSVersion *string `json:"os_version"`
DeployedAt *time.Time `json:"deployed_at"`
bmcSecret []byte bmcSecret []byte
nodeToken []byte nodeToken []byte
hostKey []byte hostKey []byte
@@ -104,14 +126,16 @@ func openStore(ctx context.Context, url string) (*Store, error) {
} }
const machineCols = `id, uuid, serial, hostname, manufacturer, product, state, first_seen, last_seen, const machineCols = `id, uuid, serial, hostname, manufacturer, product, state, first_seen, last_seen,
commissioned_at, macs, bmc, bmc_secret, inventory, profile, node_name, mgmt_ip, ib_ip, node_token, host_key, booted_at` commissioned_at, macs, bmc, bmc_secret, inventory, profile, node_name, mgmt_ip, ib_ip, node_token, host_key, booted_at,
os, os_version, deployed_at`
func scanMachine(row pgx.Row, full bool) (*Machine, error) { func scanMachine(row pgx.Row, full bool) (*Machine, error) {
m := &Machine{} m := &Machine{}
var macs []byte var macs []byte
err := row.Scan(&m.ID, &m.UUID, &m.Serial, &m.Hostname, &m.Manufacturer, &m.Product, &m.State, err := row.Scan(&m.ID, &m.UUID, &m.Serial, &m.Hostname, &m.Manufacturer, &m.Product, &m.State,
&m.FirstSeen, &m.LastSeen, &m.CommissionedAt, &macs, &m.BMC, &m.bmcSecret, &m.Inventory, &m.FirstSeen, &m.LastSeen, &m.CommissionedAt, &macs, &m.BMC, &m.bmcSecret, &m.Inventory,
&m.Profile, &m.NodeName, &m.MgmtIP, &m.IBIP, &m.nodeToken, &m.hostKey, &m.BootedAt) &m.Profile, &m.NodeName, &m.MgmtIP, &m.IBIP, &m.nodeToken, &m.hostKey, &m.BootedAt,
&m.OS, &m.OSVersion, &m.DeployedAt)
if errors.Is(err, pgx.ErrNoRows) { if errors.Is(err, pgx.ErrNoRows) {
return nil, errNotFound return nil, errNotFound
} }
@@ -220,6 +244,9 @@ func (s *Store) Delete(ctx context.Context, id string) error {
if _, err := tx.Exec(ctx, `DELETE FROM events WHERE machine_id=$1`, id); err != nil { if _, err := tx.Exec(ctx, `DELETE FROM events WHERE machine_id=$1`, id); err != nil {
return err return err
} }
if _, err := tx.Exec(ctx, `DELETE FROM jobs WHERE machine_id=$1`, id); err != nil {
return err
}
tag, err := tx.Exec(ctx, `DELETE FROM machines WHERE id=$1`, id) tag, err := tx.Exec(ctx, `DELETE FROM machines WHERE id=$1`, id)
if err != nil { if err != nil {
return err return err
+52 -3
View File
@@ -36,6 +36,7 @@ $('#back').onclick = () => { current = null; $('#detail').hidden = true; $('#lis
function setStatus(msg, bad) { const el = $('#status'); el.textContent = msg; el.className = bad ? 'bad' : 'muted'; } function setStatus(msg, bad) { const el = $('#status'); el.textContent = msg; el.className = bad ? 'bad' : 'muted'; }
let profiles = []; let profiles = [];
let osImages = [];
async function loadList() { async function loadList() {
if (!token) return askToken(); if (!token) return askToken();
try { try {
@@ -56,6 +57,12 @@ async function loadProfiles() {
const sel = $('#assign-profile'); const cur = sel.value; const sel = $('#assign-profile'); const cur = sel.value;
sel.innerHTML = '<option value="">profile…</option>' + profiles.map(p => `<option value="${esc(p.name)}">${esc(p.name)} (${esc(p.image)})</option>`).join(''); sel.innerHTML = '<option value="">profile…</option>' + profiles.map(p => `<option value="${esc(p.name)}">${esc(p.name)} (${esc(p.image)})</option>`).join('');
sel.value = cur; sel.value = cur;
osImages = await api('GET', '/api/v1/os');
$('#osimages').innerHTML = table(['OS', 'Current version', 'Family', 'Compressed', 'Disk needed', 'Imported', 'Track', 'Versions', 'Status'],
osImages.map(i => { const c = i.current || {}; const s = i.source || {}; return [esc(i.name), `<span class="mono">${esc(c.version || '-')}</span>`, esc(s.family || c.family), c.size ? human(c.size) : '', c.raw_size ? human(c.raw_size) : '', c.imported ? esc(new Date(c.imported).toLocaleString()) : '', esc(s.track || ''), (i.versions || []).length, i.warning ? `<span class="bad">${esc(i.warning)}</span>` : '<span class="ok">ok</span>']; }));
const os = $('#deploy-os'); const curOS = os.value;
os.innerHTML = '<option value="">OS image…</option>' + osImages.filter(i => i.current).map(i => `<option value="${esc(i.name)}">${esc(i.name)} (${esc(i.current.version)}, ${human(i.current.raw_size)})</option>`).join('');
os.value = curOS;
} catch (e) { /* non-fatal */ } } catch (e) { /* non-fatal */ }
} }
$('#profile-form').onsubmit = async ev => { $('#profile-form').onsubmit = async ev => {
@@ -75,9 +82,12 @@ function renderList() {
$('#machines tbody').innerHTML = rows.map(m => { $('#machines tbody').innerHTML = rows.map(m => {
const b = m.bmc; const b = m.bmc;
const bmc = b ? `<span class="${b.configured ? 'ok' : 'bad'}">${esc(b.address || '-')}</span> <span class="muted">${esc(b.source)}</span>` : '<span class="muted">-</span>'; const bmc = b ? `<span class="${b.configured ? 'ok' : 'bad'}">${esc(b.address || '-')}</span> <span class="muted">${esc(b.source)}</span>` : '<span class="muted">-</span>';
const prog = m.job && m.job.progress ? m.job.progress : null;
const state = m.job ? `${esc(m.state)} ${prog ? Math.round(prog.pct || 0) + '%' : ''}` : esc(m.state);
const what = m.profile ? esc(m.profile) : m.os ? `${esc(m.os)}<br><span class="muted mono">${esc(m.os_version || '')}</span>` : '';
return `<tr data-id="${esc(m.id)}"> return `<tr data-id="${esc(m.id)}">
<td class="mono">${esc(m.id)}</td><td><span class="pill ${esc(m.state)}">${esc(m.state)}</span></td> <td class="mono">${esc(m.id)}</td><td><span class="pill ${esc(m.state)}">${state}</span></td>
<td>${esc(m.node_name || '')}${m.mgmt_ip ? `<br><span class="muted mono">${esc(m.mgmt_ip)}</span>` : ''}</td><td>${esc(m.profile || '')}</td> <td>${esc(m.node_name || '')}${m.mgmt_ip ? `<br><span class="muted mono">${esc(m.mgmt_ip)}</span>` : ''}</td><td>${what}</td>
<td>${esc((m.manufacturer || '') + ' ' + (m.product || ''))}</td><td class="mono">${esc(m.serial || '')}</td> <td>${esc((m.manufacturer || '') + ' ' + (m.product || ''))}</td><td class="mono">${esc(m.serial || '')}</td>
<td class="num">${esc(m.cpus)}</td><td class="num">${human(m.memory_bytes)}</td><td class="num">${m.disks | 0}</td> <td class="num">${esc(m.cpus)}</td><td class="num">${human(m.memory_bytes)}</td><td class="num">${m.disks | 0}</td>
<td class="num">${human(m.storage_bytes)}</td><td class="num">${m.nics}</td><td class="num">${m.ib_hcas}</td><td class="num">${m.gpus}</td> <td class="num">${human(m.storage_bytes)}</td><td class="num">${m.nics}</td><td class="num">${m.ib_hcas}</td><td class="num">${m.gpus}</td>
@@ -106,7 +116,27 @@ function renderDetail(d) {
$('#action-out').textContent = ''; $('#action-out').textContent = '';
if (s.profile) $('#assign-profile').value = s.profile; if (s.profile) $('#assign-profile').value = s.profile;
$('#assign-name').value = s.node_name || ''; $('#assign-name').value = s.node_name || '';
const diskSel = $('#deploy-disk'); const curDisk = diskSel.value;
diskSel.innerHTML = '<option value="">disk: auto (smallest that fits)</option>' + ((st.disks || []).map(x => `<option value="${esc(x.path || '/dev/' + x.name)}">${esc(x.name)} ${human(x.size)} ${esc(x.model || '')}</option>`).join(''));
diskSel.value = curDisk;
$('#job-cancel').hidden = !s.job;
const cards = []; const cards = [];
const job = s.job || (d.jobs || [])[0];
if (s.os || job) {
const p = (job && job.progress) || {}; const sp = (job && job.spec) || {};
const active = job && (job.state === 'pending' || job.state === 'running');
cards.push(card('Deployment', kv([
['OS', s.os ? `${esc(s.os)} <span class="muted mono">${esc(s.os_version || '')}</span>` : ''],
['Deployed', esc(s.deployed_at ? new Date(s.deployed_at).toLocaleString() : '')],
['Phoned home', s.deployed_at ? (s.booted_at ? `<span class="ok">${esc(new Date(s.booted_at).toLocaleString())}</span>` : '<span class="muted">not yet</span>') : ''],
['Hostname', esc(s.node_name)], ['Management IP', s.mgmt_ip ? `<span class="mono">${esc(s.mgmt_ip)}</span>` : ''],
['Job', job ? `<span class="mono">${esc(job.id)}</span> ${esc(job.kind)} <span class="pill ${esc(job.state)}">${esc(job.state)}</span> ${esc(ago(job.created))}` : ''],
['Target', sp.disk ? `<span class="mono">${esc(sp.disk)}</span> ← ${esc(sp.os || '')}/${esc(sp.version || '')}` : ''],
['Progress', active ? `<div class="bar"><span style="width:${Math.round(p.pct || 0)}%"></span></div><span class="muted">${esc(p.phase || '')} ${Math.round(p.pct || 0)}% ${esc(p.msg || '')}</span>` : ''],
['Result', p.details && Object.keys(p.details).length ? `<span class="mono">${esc(JSON.stringify(p.details))}</span>` : ''],
['Error', job && job.error ? `<span class="bad">${esc(job.error)}</span>` : ''],
]) + ((d.jobs || []).length > 1 ? '<p class="muted">History</p>' + table(['Job', 'Kind', 'State', 'Created', 'OS', 'Error'], d.jobs.map(j => [`<span class="mono">${esc(j.id)}</span>`, esc(j.kind), `<span class="pill ${esc(j.state)}">${esc(j.state)}</span>`, esc(new Date(j.created).toLocaleString()), esc(((j.spec || {}).os || '') + '/' + ((j.spec || {}).version || '')), esc(j.error || '')])) : ''), true));
}
if (s.profile) cards.push(card('Stateless node', kv([ if (s.profile) cards.push(card('Stateless node', kv([
['Node name', esc(s.node_name)], ['Profile', esc(s.profile)], ['Management IP', `<span class="mono">${esc(s.mgmt_ip)}</span>`], ['Node name', esc(s.node_name)], ['Profile', esc(s.profile)], ['Management IP', `<span class="mono">${esc(s.mgmt_ip)}</span>`],
['InfiniBand IP', s.ib_ip ? `<span class="mono">${esc(s.ib_ip)}</span>` : ''], ['Booted', esc(s.booted_at ? new Date(s.booted_at).toLocaleString() : 'not yet')], ['InfiniBand IP', s.ib_ip ? `<span class="mono">${esc(s.ib_ip)}</span>` : ''], ['Booted', esc(s.booted_at ? new Date(s.booted_at).toLocaleString() : 'not yet')],
@@ -188,10 +218,29 @@ const doAssign = boot => action(async () => {
$('#assign').onclick = () => doAssign(false); $('#assign').onclick = () => doAssign(false);
$('#assign-boot').onclick = () => { if (confirm('Assign and power-cycle this machine into the stateless image?')) doAssign(true); }; $('#assign-boot').onclick = () => { if (confirm('Assign and power-cycle this machine into the stateless image?')) doAssign(true); };
$('#unassign').onclick = () => { if (confirm('Remove the profile? The node name and IP are released.')) action(async () => { const r = await api('POST', `/api/v1/machines/${current}/unassign`); await loadDetail(current); return 'state ' + r.state; }, 'unassign'); }; $('#unassign').onclick = () => { if (confirm('Remove the profile? The node name and IP are released.')) action(async () => { const r = await api('POST', `/api/v1/machines/${current}/unassign`); await loadDetail(current); return 'state ' + r.state; }, 'unassign'); };
$('#deploy').onclick = () => {
const os = $('#deploy-os').value; if (!os) { $('#action-out').textContent = 'pick an OS image'; return; }
const disk = $('#deploy-disk').value;
if (!confirm(`Deploy ${os} to ${disk || 'the auto-selected disk'} of ${current}?\nThe disk is wiped and the machine is power-cycled.`)) return;
action(async () => {
const r = await api('POST', `/api/v1/machines/${current}/deploy`, { os, disk, hostname: $('#deploy-host').value.trim(), package_upgrade: $('#deploy-upgrade').checked, boot: true });
await loadDetail(current); return `job ${r.job}: ${r.os}/${r.version} → ${r.disk} as ${r.hostname}${r.ipmi ? ' · ' + r.ipmi : ''}`;
}, 'deploy');
};
$('#release').onclick = () => { if (confirm(`Release ${current}? The OS disk is wiped (signatures + first/last 64 MiB) and the machine returns to ready.`)) action(async () => { const r = await api('POST', `/api/v1/machines/${current}/release`, { boot: true }); await loadDetail(current); return `job ${r.job} (wipe ${r.disk})${r.ipmi ? ' · ' + r.ipmi : ''}`; }, 'release'); };
$('#job-cancel').onclick = () => { if (confirm('Cancel the active job? A running write is not interrupted on the machine; it just stops being tracked.')) action(async () => { const r = await api('POST', `/api/v1/machines/${current}/jobs/cancel`); await loadDetail(current); return `job ${r.job} cancelled`; }, 'cancel job'); };
$('#recommission').onclick = () => action(async () => { const r = await api('POST', `/api/v1/machines/${current}/recommission`); await loadDetail(current); return `state ${r.state}: next PXE boot runs commissioning`; }, 'recommission'); $('#recommission').onclick = () => action(async () => { const r = await api('POST', `/api/v1/machines/${current}/recommission`); await loadDetail(current); return `state ${r.state}: next PXE boot runs commissioning`; }, 'recommission');
$('#recommission-now').onclick = () => { if (confirm('Remove PXE override, set bootdev=pxe and power-cycle this machine via its BMC?')) action(async () => { const r = await api('POST', `/api/v1/machines/${current}/recommission?now=1`); await loadDetail(current); return r.ipmi || r.state; }, 'recommission now'); }; $('#recommission-now').onclick = () => { if (confirm('Remove PXE override, set bootdev=pxe and power-cycle this machine via its BMC?')) action(async () => { const r = await api('POST', `/api/v1/machines/${current}/recommission?now=1`); await loadDetail(current); return r.ipmi || r.state; }, 'recommission now'); };
$('#delete').onclick = () => { if (confirm(`Delete ${current}? It will be commissioned again as a new machine on next PXE boot.`)) action(async () => { await api('DELETE', `/api/v1/machines/${current}`); $('#back').click(); return 'deleted'; }, 'delete'); }; $('#delete').onclick = () => { if (confirm(`Delete ${current}? It will be commissioned again as a new machine on next PXE boot.`)) action(async () => { await api('DELETE', `/api/v1/machines/${current}`); $('#back').click(); return 'deleted'; }, 'delete'); };
function schedule() { clearInterval(timer); timer = setInterval(() => { if (!current && document.visibilityState === 'visible') loadList(); }, 10000); } function schedule() {
clearInterval(timer);
timer = setInterval(() => {
if (document.visibilityState !== 'visible') return;
if (!current) return loadList();
const m = machines.find(x => x.id === current);
if (m && (m.job || m.state === 'deploying' || m.state === 'releasing')) { loadDetail(current); loadList(); }
}, 4000);
}
schedule(); schedule();
loadList(); loadList();
+13 -3
View File
@@ -33,7 +33,7 @@
</div> </div>
<table id="machines"> <table id="machines">
<thead><tr> <thead><tr>
<th>ID</th><th>State</th><th>Node</th><th>Profile</th><th>Model</th><th>Serial</th><th class="num">CPU</th><th class="num">RAM</th> <th>ID</th><th>State</th><th>Node</th><th>Profile / OS</th><th>Model</th><th>Serial</th><th class="num">CPU</th><th class="num">RAM</th>
<th class="num">Disks</th><th class="num">Storage</th><th class="num">NIC</th><th class="num">IB</th><th class="num">GPU</th> <th class="num">Disks</th><th class="num">Storage</th><th class="num">NIC</th><th class="num">IB</th><th class="num">GPU</th>
<th>BMC</th><th>Last seen</th> <th>BMC</th><th>Last seen</th>
</tr></thead> </tr></thead>
@@ -42,10 +42,11 @@
<p id="empty" class="muted" hidden>No machines yet. PXE-boot an unknown server: it commissions itself and appears here.</p> <p id="empty" class="muted" hidden>No machines yet. PXE-boot an unknown server: it commissions itself and appears here.</p>
<details id="profiles-box" style="margin-top:18px"> <details id="profiles-box" style="margin-top:18px">
<summary>Profiles &amp; images (stateless provisioning)</summary> <summary>Provisioning: OS images (deployment), profiles &amp; root images (stateless)</summary>
<div class="grid" style="margin-top:10px"> <div class="grid" style="margin-top:10px">
<div class="card wide"><h3>OS images for deployment <span class="muted" style="text-transform:none">· refreshed nightly by <code>rollcall os sync</code></span></h3><div id="osimages"></div></div>
<div class="card"><h3>Profiles</h3><div id="profiles"></div></div> <div class="card"><h3>Profiles</h3><div id="profiles"></div></div>
<div class="card"><h3>Images</h3><div id="images"></div></div> <div class="card"><h3>Stateless root images</h3><div id="images"></div></div>
<div class="card wide"><h3>Add / update profile</h3> <div class="card wide"><h3>Add / update profile</h3>
<form id="profile-form" class="kv" style="grid-template-columns:max-content 1fr;align-items:center"> <form id="profile-form" class="kv" style="grid-template-columns:max-content 1fr;align-items:center">
<label>Name</label><input name="name" required placeholder="compute"> <label>Name</label><input name="name" required placeholder="compute">
@@ -87,6 +88,15 @@
<button id="assign-boot" class="btn" title="assign, bootdev pxe, power-cycle">Assign + boot</button> <button id="assign-boot" class="btn" title="assign, bootdev pxe, power-cycle">Assign + boot</button>
<button id="unassign" class="btn warn">Unassign</button> <button id="unassign" class="btn warn">Unassign</button>
<span class="sep"></span> <span class="sep"></span>
<span class="muted">Deploy:</span>
<select id="deploy-os"><option value="">OS image…</option></select>
<select id="deploy-disk"><option value="">disk: auto (smallest that fits)</option></select>
<input id="deploy-host" type="text" placeholder="hostname (auto)" style="width:140px">
<label><input id="deploy-upgrade" type="checkbox"> upgrade on first boot</label>
<button id="deploy" class="btn primary" title="write the OS image to the disk, seed cloud-init, bootdev pxe, power-cycle">Deploy + boot</button>
<button id="release" class="btn warn" title="wipe the disk signatures and return the machine to ready">Release</button>
<button id="job-cancel" class="btn" hidden>Cancel job</button>
<span class="sep"></span>
<button id="recommission" class="btn warn">Re-commission</button> <button id="recommission" class="btn warn">Re-commission</button>
<button id="recommission-now" class="btn warn" title="remove per-MAC PXE config, set bootdev pxe and power-cycle via BMC">Re-commission now</button> <button id="recommission-now" class="btn warn" title="remove per-MAC PXE config, set bootdev pxe and power-cycle via BMC">Re-commission now</button>
<button id="delete" class="btn danger">Delete</button> <button id="delete" class="btn danger">Delete</button>
+6 -1
View File
@@ -33,7 +33,12 @@ th { font-weight: 600; color: var(--muted); font-size: 12px; text-transform: upp
th.num, td.num { text-align: right; font-variant-numeric: tabular-nums; } th.num, td.num { text-align: right; font-variant-numeric: tabular-nums; }
tbody tr { cursor: pointer; } tbody tr:hover { background: var(--hover); } tbody tr { cursor: pointer; } tbody tr:hover { background: var(--hover); }
.pill { display: inline-block; padding: 2px 8px; border-radius: 999px; background: var(--pill); font-size: 12px; } .pill { display: inline-block; padding: 2px 8px; border-radius: 999px; background: var(--pill); font-size: 12px; }
.pill.ready, .pill.booted { color: var(--ok); } .pill.recommission, .pill.new, .pill.stateless { color: var(--warn); } .pill.ready, .pill.booted, .pill.deployed { color: var(--ok); } .pill.recommission, .pill.new, .pill.stateless, .pill.deploying, .pill.releasing { color: var(--warn); }
.pill.deploy-failed, .pill.release-failed { color: var(--danger); }
.bar { position: relative; height: 10px; background: var(--pill); border-radius: 999px; overflow: hidden; margin: 6px 0; }
.bar > span { position: absolute; inset: 0 auto 0 0; background: var(--accent); transition: width .5s; }
.actions select, .actions input[type=text] { padding: 5px 8px; border: 1px solid var(--line); border-radius: 6px; background: var(--panel); color: var(--fg); font: inherit; }
.actions label { display: inline-flex; gap: 4px; align-items: center; color: var(--muted); font-size: 13px; }
.card form input { padding: 5px 8px; border: 1px solid var(--line); border-radius: 6px; background: var(--panel); color: var(--fg); font: inherit; min-width: 0; width: 100%; } .card form input { padding: 5px 8px; border: 1px solid var(--line); border-radius: 6px; background: var(--panel); color: var(--fg); font: inherit; min-width: 0; width: 100%; }
#assign-profile { padding: 5px 8px; border: 1px solid var(--line); border-radius: 6px; background: var(--panel); color: var(--fg); font: inherit; } #assign-profile { padding: 5px 8px; border: 1px solid var(--line); border-radius: 6px; background: var(--panel); color: var(--fg); font: inherit; }
.ok { color: var(--ok); } .bad { color: var(--danger); } .ok { color: var(--ok); } .bad { color: var(--danger); }
+2
View File
@@ -12,6 +12,8 @@
"images_dir": "/srv/rollcall/images", "images_dir": "/srv/rollcall/images",
"overlays_dir": "/etc/rollcall/overlays", "overlays_dir": "/etc/rollcall/overlays",
"dnsmasq_dir": "/var/lib/rollcall/dnsmasq", "dnsmasq_dir": "/var/lib/rollcall/dnsmasq",
"os_dir": "/srv/rollcall/os",
"os_sources": "/etc/rollcall/os-sources.json",
"node_pools": {"mgmt": "10.213.77.50-10.213.77.99"}, "node_pools": {"mgmt": "10.213.77.50-10.213.77.99"},
"cluster_vars": { "cluster_vars": {
"domain": "pxe.local", "domain": "pxe.local",
+6
View File
@@ -27,6 +27,11 @@ install -d /srv/rollcall/images
install -d -m 0750 -o root -g rollcall /etc/rollcall/overlays install -d -m 0750 -o root -g rollcall /etc/rollcall/overlays
cp -a overlays/. /etc/rollcall/overlays/ cp -a overlays/. /etc/rollcall/overlays/
chgrp -R rollcall /etc/rollcall/overlays chgrp -R rollcall /etc/rollcall/overlays
# OS deployment images: <os_dir>/<name>/<version>/disk.raw.zst, written by `rollcall os sync` (user rollcall)
install -d -m 0755 -o rollcall -g rollcall /srv/rollcall/os
[ -e /etc/rollcall/os-sources.json ] || install -m 0644 os-sources.json /etc/rollcall/os-sources.json
install -m 0644 rollcall-os-sync.service rollcall-os-sync.timer /etc/systemd/system/
command -v qemu-img >/dev/null || echo "WARNING: qemu-img missing (apt install qemu-utils) - rollcall os sync/import will fail" >&2
# dnsmasq host export + reload trigger # dnsmasq host export + reload trigger
install -d -m 0755 /var/lib/rollcall install -d -m 0755 /var/lib/rollcall
install -d -m 0755 -o rollcall -g rollcall /var/lib/rollcall/dnsmasq install -d -m 0755 -o rollcall -g rollcall /var/lib/rollcall/dnsmasq
@@ -36,6 +41,7 @@ install -m 0644 rollcall-controller-sync.service rollcall-controller-sync.timer
install -m 0755 rollcall-controller-sync.sh /usr/local/bin/rollcall-controller-sync.sh install -m 0755 rollcall-controller-sync.sh /usr/local/bin/rollcall-controller-sync.sh
systemctl daemon-reload systemctl daemon-reload
systemctl enable --now rollcall-dnsmasq.path >/dev/null 2>&1 || true systemctl enable --now rollcall-dnsmasq.path >/dev/null 2>&1 || true
systemctl enable --now rollcall-os-sync.timer >/dev/null 2>&1 || true
if systemctl list-unit-files slurmctld.service >/dev/null 2>&1 && [ -d /etc/slurm ]; then if systemctl list-unit-files slurmctld.service >/dev/null 2>&1 && [ -d /etc/slurm ]; then
systemctl enable --now rollcall-controller-sync.timer >/dev/null 2>&1 || true systemctl enable --now rollcall-controller-sync.timer >/dev/null 2>&1 || true
fi fi
+52
View File
@@ -0,0 +1,52 @@
[
{
"name": "debian-13",
"description": "Debian 13 (trixie) generic cloud image",
"family": "debian",
"url": "https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2",
"checksum_url": "https://cloud.debian.org/images/cloud/trixie/latest/SHA512SUMS",
"checksum": "sha512",
"track": "latest",
"keep": 2
},
{
"name": "ubuntu-24.04",
"description": "Ubuntu 24.04 LTS (noble) server cloud image",
"family": "debian",
"url": "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img",
"checksum_url": "https://cloud-images.ubuntu.com/noble/current/SHA256SUMS",
"checksum": "sha256",
"track": "latest",
"keep": 2
},
{
"name": "rocky-9",
"description": "Rocky Linux 9 GenericCloud (Base)",
"family": "rhel",
"url": "https://dl.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2",
"checksum_url": "https://dl.rockylinux.org/pub/rocky/9/images/x86_64/CHECKSUM",
"checksum": "sha256",
"track": "latest",
"keep": 2
},
{
"name": "almalinux-9",
"description": "AlmaLinux 9 GenericCloud",
"family": "rhel",
"url": "https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-latest.x86_64.qcow2",
"checksum_url": "https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/CHECKSUM",
"checksum": "sha256",
"track": "latest",
"keep": 2
},
{
"name": "centos-stream-9",
"description": "CentOS Stream 9 GenericCloud",
"family": "rhel",
"url": "https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-x86_64-9-latest.x86_64.qcow2",
"checksum_url": "https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-x86_64-9-latest.x86_64.qcow2.SHA256SUM",
"checksum": "sha256",
"track": "latest",
"keep": 2
}
]
+13
View File
@@ -0,0 +1,13 @@
[Unit]
Description=rollcall: refresh OS deployment images from upstream (downloads only when the vendor checksum changed)
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User=rollcall
Group=rollcall
ExecStart=/usr/local/bin/rollcall os sync
Nice=10
IOSchedulingClass=idle
TimeoutStartSec=6h
+10
View File
@@ -0,0 +1,10 @@
[Unit]
Description=rollcall: nightly OS image refresh
[Timer]
OnCalendar=*-*-* 03:30:00
RandomizedDelaySec=30m
Persistent=true
[Install]
WantedBy=timers.target