OS deployment: image catalog with upstream sync, deploy/release jobs, agent deploy mode, UI and CLI
- rollcall os list|sync|import|promote|rm: <os_dir>/<name>/<version>/{disk.raw.zst,meta.json},
current symlink, vendor checksum verification (GNU/BSD formats, sha256/sha512), qemu-img -> raw,
zstd in-process, keep-N pruning, nightly rollcall-os-sync.timer, deploy/os-sources.json
(Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9, CentOS Stream 9)
- jobs table + API: deploy (disk pick, hostname/IP allocation, job token, per-MAC deploy PXE
entry), release (wipe), progress, cancel, cloud-init phone_home
- agent/rollcall-deploy.py: streamed write with sha256 + pv progress, NoCloud seed, SELinux
autorelabel, efibootmgr on UEFI; commissioning image gains zstd/parted/xfsprogs/...
- web UI: deploy/release controls, deployment card with progress, OS catalog table
- README: deployment section, image freshness policy
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Fable 5.1 committed
1 parent
1452b79218
commit
fad4e1c8bd
24 files changed
+2194
-26
No files matched your search
@@ -44,7 +44,8 @@ unknown server ──PXE──▶ dnsmasq (DHCP+TFTP) ──▶ pxelinux default
|
||||
|---|---|
|
||||
| `cmd/rollcall/` | Go server + CLI + embedded web UI (`web/`) |
|
||||
| `agent/` | `rollcall-agent.py` (runs inside the live image), its systemd unit, `build-image.sh` (live-build recipe) |
|
||||
| `deploy/` | `install.sh`, `rollcall.service`, `config.example.json`, `pxelinux-default` |
|
||||
| `deploy/` | `install.sh`, `rollcall.service`, `config.example.json`, `pxelinux-default`, `os-sources.json`, sync/dnsmasq/controller units, `overlays/` |
|
||||
| `images/` | stateless root image recipes (`build-rootfs.sh`, `compute-debian13/`) |
|
||||
| `lab/` | Proxmox test lab: `pve-vbmc` (virtual IPMI BMC for VMs backed by the Proxmox API), cloud-init for the PXE server VM |
|
||||
| `site/` | Project website (static) |
|
||||
| `.gitea/workflows/` | CI (fmt/vet/build/tests), release on tags, deploy on push to `main` |
|
||||
@@ -97,6 +98,15 @@ Reads `ROLLCALL_URL` / `ROLLCALL_TOKEN`, or the config file when readable.
|
||||
| POST | `/api/v1/machines/{id}/bootdev` | admin | `{"device":"pxe"}` |
|
||||
| POST | `/api/v1/machines/{id}/recommission[?now=1]` | admin | boot the image again on next PXE (`now=1`: via BMC immediately) |
|
||||
| DELETE | `/api/v1/machines/{id}` | admin | forget the machine |
|
||||
| GET/POST/DELETE | `/api/v1/profiles[/{name}]` | admin | stateless profiles |
|
||||
| POST | `/api/v1/machines/{id}/assign` `/unassign` | admin | stateless node assignment |
|
||||
| GET | `/api/v1/os` | admin | OS image catalog |
|
||||
| POST | `/api/v1/machines/{id}/deploy` | admin | `{"os":"rocky-9","disk":"/dev/sda","hostname":"","package_upgrade":false,"boot":true}` |
|
||||
| POST | `/api/v1/machines/{id}/release` | admin | wipe job |
|
||||
| GET / POST | `/api/v1/machines/{id}/jobs[/cancel]` | admin | job history / cancel |
|
||||
| GET | `/api/v1/jobs/self`, POST `…/progress` | job token (`X-Rollcall-Job`) | deploy agent |
|
||||
| POST | `/api/v1/phone-home/{token}` | job token in path | cloud-init phone_home |
|
||||
| GET | `/os/*` | none | OS disk images |
|
||||
| GET | `/rollcall/*` | none | image files (kernel, initrd, squashfs) |
|
||||
| GET | `/` | token in browser | web UI |
|
||||
|
||||
@@ -157,6 +167,64 @@ dress it with per-node files, instead of installing an OS on disk.
|
||||
|
||||
States: `ready → stateless → booted`; `rollcall unassign` releases name and IPs.
|
||||
|
||||
## OS deployment (install to disk)
|
||||
|
||||
For nodes that should run an operating system from their own disk, Rollcall
|
||||
writes a vendor cloud image onto the disk and lets cloud-init do first-boot
|
||||
configuration. No installer, no kickstart/preseed: a deployment is a streamed
|
||||
disk write and takes about as long as the image download.
|
||||
|
||||
```
|
||||
rollcall deploy <id> rocky-9 -boot -watch
|
||||
│ server: picks the disk (smallest that fits, or -disk), allocates hostname + IP,
|
||||
│ creates a job with a one-time token, writes pxelinux.cfg/01-<mac>
|
||||
│ ("deploy" entry = commissioning image with rollcall.mode=deploy), power-cycles via BMC
|
||||
▼
|
||||
commissioning image, deploy mode (agent/rollcall-deploy.py)
|
||||
GET /api/v1/jobs/self → image URL, sha256, disk, rendered cloud-init seed
|
||||
curl | tee sha256 | pv | zstd -d | dd of=/dev/sda (progress every 3 s)
|
||||
mount root fs → /var/lib/cloud/seed/nocloud/{meta-data,user-data}, datasource_list: [NoCloud, None]
|
||||
(SELinux images: /.autorelabel; UEFI hosts: efibootmgr entry for the ESP loader)
|
||||
POST progress state=done → server: state deployed, PXE entry = local disk, reboot
|
||||
▼
|
||||
first boot: cloud-init sets hostname/fqdn, SSH keys (root + default user), optional
|
||||
package_upgrade, then phone_home → POST /api/v1/phone-home/<token> (shown as "phoned home")
|
||||
```
|
||||
|
||||
### Image catalog and freshness
|
||||
|
||||
```
|
||||
os_dir/<name>/<version>/disk.raw.zst raw disk image, zstd
|
||||
os_dir/<name>/<version>/meta.json sizes, sha256, upstream checksum, import time
|
||||
os_dir/<name>/current -> <version> what new deployments use
|
||||
```
|
||||
|
||||
`os_sources` (`deploy/os-sources.json`) lists upstream "latest" URLs with the
|
||||
vendor checksum file for each: Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9,
|
||||
CentOS Stream 9. `rollcall os sync` fetches the checksum file, and only when the
|
||||
vendor checksum differs from every imported version does it download the image,
|
||||
verify it, convert to raw (`qemu-img`), compress, write a new version directory
|
||||
and flip `current` atomically. `rollcall-os-sync.timer` runs it nightly;
|
||||
`keep` versions are retained per source, nothing younger than a day is pruned,
|
||||
and a job always pins the exact version it started with. Sources with
|
||||
`"track": "pinned"` still download new versions but keep `current` until
|
||||
`rollcall os promote <name> <version>`. Local files go in with
|
||||
`rollcall os import <name> <file.qcow2>`. Image freshness is monthly at best
|
||||
upstream; the deploy option `-upgrade` (`package_upgrade`) pulls the rest on
|
||||
first boot.
|
||||
|
||||
```sh
|
||||
rollcall os list | sync [NAME...] [-force] | import NAME FILE [-family debian|rhel] | promote NAME VERSION | rm NAME VERSION
|
||||
rollcall deploy ID OS [-version V] [-disk /dev/nvme0n1] [-hostname H] [-upgrade] [-packages a,b] [-user-data FILE] [-boot] [-watch]
|
||||
rollcall release ID [-wipe quick|full] [-boot] [-watch] # wipe the disk, back to ready
|
||||
rollcall jobs ID [-watch] | job cancel ID
|
||||
```
|
||||
|
||||
States: `ready → deploying → deployed` (`deploy-failed` on error; the agent stays
|
||||
up for ssh). A deployed machine's PXE entry defaults to local disk, so boot order
|
||||
can stay "network first" on BIOS hosts; on UEFI hosts the agent adds a boot entry
|
||||
for the installed loader. `rollcall recommission` or `release` take it back.
|
||||
|
||||
## Lab
|
||||
|
||||
`lab/` documents a Proxmox SDN test setup: an isolated vnet, a PXE server VM
|
||||
@@ -166,6 +234,7 @@ IPMI-over-LAN BMC emulator that drives VMs through the Proxmox API so that
|
||||
|
||||
## Status
|
||||
|
||||
Working end to end in the lab (virtual machines, out-of-band BMC hints). The
|
||||
in-band BMC path is implemented but has not yet been exercised on real
|
||||
hardware.
|
||||
Commissioning, stateless nodes and disk deployment work end to end in the lab
|
||||
(virtual machines with 30 GB disks, SeaBIOS, out-of-band BMC hints). The
|
||||
in-band BMC path and the UEFI boot-entry step are implemented but have not yet
|
||||
been exercised on real hardware.
|
||||
Binary file not shown.
Binary file not shown.
@@ -68,8 +68,15 @@ pv
|
||||
jq
|
||||
mdadm
|
||||
lvm2
|
||||
nvme-cli
|
||||
efibootmgr
|
||||
zstd
|
||||
gdisk
|
||||
parted
|
||||
xfsprogs
|
||||
e2fsprogs
|
||||
btrfs-progs
|
||||
dosfstools
|
||||
bash
|
||||
firmware-linux
|
||||
firmware-linux-nonfree
|
||||
firmware-misc-nonfree
|
||||
@@ -88,6 +95,7 @@ mkdir -p config/includes.chroot/usr/local/bin config/includes.chroot/etc/systemd
|
||||
config/includes.chroot/root/.ssh config/includes.chroot/etc/modules-load.d \
|
||||
config/hooks/live
|
||||
install -m 0755 "$SRC/rollcall-agent.py" config/includes.chroot/usr/local/bin/rollcall-agent.py
|
||||
install -m 0755 "$SRC/rollcall-deploy.py" config/includes.chroot/usr/local/bin/rollcall-deploy.py
|
||||
install -m 0644 "$SRC/rollcall-agent.service" config/includes.chroot/etc/systemd/system/rollcall-agent.service
|
||||
printf 'ipmi_msghandler\nipmi_devintf\nipmi_si\nipmi_ssif\n' > config/includes.chroot/etc/modules-load.d/ipmi.conf
|
||||
if [ -r "$AUTHKEYS" ]; then
|
||||
|
||||
@@ -10,6 +10,7 @@ Kernel cmdline options:
|
||||
rollcall.nopoweroff stay up (debugging)
|
||||
rollcall.bmcuser=commissioner BMC account name to create
|
||||
rollcall.token=... agent token (sent as X-Rollcall-Token)
|
||||
rollcall.mode=deploy rollcall.job=.. run rollcall-deploy.py instead (OS deployment / release job)
|
||||
"""
|
||||
import glob
|
||||
import json
|
||||
@@ -441,6 +442,9 @@ def post(server, payload, tries=30):
|
||||
|
||||
def main():
|
||||
opts = cmdline()
|
||||
if opts.get("mode") == "deploy":
|
||||
# same image, different job: hand over to the deploy/release agent
|
||||
os.execv("/usr/bin/python3", ["python3", "/usr/local/bin/rollcall-deploy.py"])
|
||||
server = opts.get("server")
|
||||
if not server:
|
||||
log("rollcall.server= missing on kernel cmdline; nothing to do")
|
||||
|
||||
@@ -0,0 +1,385 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Deploy/release agent: runs inside the commissioning live image when the
|
||||
kernel cmdline carries rollcall.mode=deploy rollcall.job=<token>.
|
||||
|
||||
deploy : stream <os>/<version>/disk.raw.zst onto the target disk (sha256
|
||||
verified), seed cloud-init NoCloud files into the written root fs,
|
||||
add a UEFI boot entry when booted via UEFI, report, reboot.
|
||||
release: wipe the target disk (quick: signatures + first/last 64 MiB), report,
|
||||
power off.
|
||||
|
||||
Everything the agent needs (image URL, checksum, disk, seed files) comes from
|
||||
GET /api/v1/jobs/self; it never decides policy itself.
|
||||
"""
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
LOG = open("/dev/console", "w", buffering=1) if os.path.exists("/dev/console") else sys.stdout
|
||||
SERVER = ""
|
||||
TOKEN = ""
|
||||
EFI_GUID = "c12a7328-f81e-11d2-ba4b-00a0c93ec93b"
|
||||
ROOT_FS = ("ext4", "xfs", "btrfs", "ext3")
|
||||
|
||||
|
||||
def log(msg):
|
||||
line = f"[deploy] {msg}"
|
||||
print(line, file=LOG)
|
||||
if LOG is not sys.stdout:
|
||||
print(line, flush=True)
|
||||
|
||||
|
||||
def run(cmd, timeout=600, check=False, **kw):
|
||||
p = subprocess.run(cmd, shell=isinstance(cmd, str), capture_output=True, text=True, timeout=timeout, **kw)
|
||||
if check and p.returncode != 0:
|
||||
raise RuntimeError(f"{cmd if isinstance(cmd, str) else ' '.join(cmd)}: rc={p.returncode} {p.stderr.strip()[-400:]}")
|
||||
return p
|
||||
|
||||
|
||||
def readf(path, default=""):
|
||||
try:
|
||||
with open(path) as f:
|
||||
return f.read().strip()
|
||||
except OSError:
|
||||
return default
|
||||
|
||||
|
||||
def cmdline():
|
||||
opts = {}
|
||||
for tok in readf("/proc/cmdline").split():
|
||||
if tok.startswith("rollcall."):
|
||||
k, _, v = tok[len("rollcall."):].partition("=")
|
||||
opts[k] = v if v else True
|
||||
return opts
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- server
|
||||
def api(method, path, body=None, tries=20):
|
||||
data = json.dumps(body).encode() if body is not None else None
|
||||
req = urllib.request.Request(SERVER + path, data=data, method=method,
|
||||
headers={"Content-Type": "application/json", "X-Rollcall-Job": TOKEN})
|
||||
last = None
|
||||
for i in range(tries):
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as r:
|
||||
return json.load(r)
|
||||
except urllib.error.HTTPError as e:
|
||||
if 400 <= e.code < 500:
|
||||
raise RuntimeError(f"{path}: HTTP {e.code} {e.read().decode(errors='replace')[:200]}")
|
||||
last = e
|
||||
except (urllib.error.URLError, OSError) as e:
|
||||
last = e
|
||||
time.sleep(3)
|
||||
raise RuntimeError(f"{path}: {last}")
|
||||
|
||||
|
||||
class Progress:
|
||||
def __init__(self):
|
||||
self.phase, self.pct, self.msg = "starting", 0, ""
|
||||
self.lock = threading.Lock()
|
||||
self.stop = threading.Event()
|
||||
self.thread = threading.Thread(target=self._loop, daemon=True)
|
||||
self.thread.start()
|
||||
|
||||
def set(self, phase=None, pct=None, msg=None, now=False):
|
||||
with self.lock:
|
||||
if phase is not None:
|
||||
self.phase = phase
|
||||
if pct is not None:
|
||||
self.pct = pct
|
||||
if msg is not None:
|
||||
self.msg = msg
|
||||
if now:
|
||||
self._post()
|
||||
|
||||
def _post(self):
|
||||
with self.lock:
|
||||
body = {"phase": self.phase, "pct": round(self.pct, 1), "msg": self.msg}
|
||||
try:
|
||||
api("POST", "/api/v1/jobs/self/progress", body, tries=1)
|
||||
except Exception as e: # noqa: BLE001
|
||||
log(f"progress post failed: {e}")
|
||||
|
||||
def _loop(self):
|
||||
while not self.stop.wait(3):
|
||||
self._post()
|
||||
|
||||
def finish(self, state, msg="", error="", details=None):
|
||||
self.stop.set()
|
||||
body = {"state": state, "phase": state, "pct": 100 if state == "done" else self.pct,
|
||||
"msg": msg, "error": error, "details": details or {}}
|
||||
return api("POST", "/api/v1/jobs/self/progress", body)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- disks
|
||||
def lsblk(dev=None):
|
||||
cmd = ["lsblk", "-J", "-b", "-o", "NAME,PATH,TYPE,SIZE,SERIAL,RM,ROTA,TRAN,FSTYPE,PARTTYPE,PARTN,PKNAME,MOUNTPOINT"]
|
||||
if dev:
|
||||
cmd.append(dev)
|
||||
p = run(cmd)
|
||||
if p.returncode != 0:
|
||||
return []
|
||||
out = []
|
||||
|
||||
def walk(items):
|
||||
for d in items:
|
||||
out.append(d)
|
||||
walk(d.get("children") or [])
|
||||
walk(json.loads(p.stdout).get("blockdevices", []))
|
||||
return out
|
||||
|
||||
|
||||
def find_disk(spec):
|
||||
want, serial, need = spec.get("disk") or "", spec.get("disk_serial") or "", int(spec.get("raw_size") or 0)
|
||||
disks = [d for d in lsblk() if d.get("type") == "disk" and not d["name"].startswith(("loop", "ram", "zram", "sr"))]
|
||||
if want:
|
||||
for d in disks:
|
||||
if d["path"] == want or d["name"] == want.replace("/dev/", ""):
|
||||
if serial and d.get("serial") and d["serial"] != serial:
|
||||
# device names move around; trust the serial over the name
|
||||
for e in disks:
|
||||
if e.get("serial") == serial:
|
||||
log(f"{want} has serial {d['serial']}, expected {serial}; using {e['path']} instead")
|
||||
return e
|
||||
raise RuntimeError(f"{want} has serial {d['serial']}, expected {serial}; refusing to write")
|
||||
return d
|
||||
if serial:
|
||||
for d in disks:
|
||||
if d.get("serial") == serial:
|
||||
log(f"{want} not found; serial {serial} is {d['path']}")
|
||||
return d
|
||||
raise RuntimeError(f"target disk {want} not present")
|
||||
cands = [d for d in disks if not d.get("rm") and d.get("tran") != "usb" and int(d["size"]) >= need]
|
||||
if not cands:
|
||||
raise RuntimeError(f"no disk >= {need} bytes")
|
||||
cands.sort(key=lambda d: (int(d["size"]), d["name"]))
|
||||
return cands[0]
|
||||
|
||||
|
||||
def release_disk(path):
|
||||
"""Make sure nothing holds the disk: swap, mounts, md, lvm."""
|
||||
for d in lsblk(path):
|
||||
if d.get("mountpoint"):
|
||||
run(["umount", "-f", d["path"]])
|
||||
run("swapoff -a")
|
||||
run(["vgchange", "-an"])
|
||||
run(["mdadm", "--stop", "--scan"])
|
||||
for d in lsblk(path):
|
||||
if d.get("type") == "part":
|
||||
run(["wipefs", "-a", d["path"]])
|
||||
run(["wipefs", "-a", path])
|
||||
run(["blockdev", "--rereadpt", path])
|
||||
run(["udevadm", "settle"])
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- deploy
|
||||
def write_image(spec, disk, prog):
|
||||
url, want, size = spec["image_url"], spec["sha256"], int(spec.get("size") or 0)
|
||||
sumf, pvf = "/run/rollcall.sum", "/run/rollcall.pv"
|
||||
for f in (sumf, pvf):
|
||||
if os.path.exists(f):
|
||||
os.remove(f)
|
||||
pipeline = (f"set -o pipefail; curl -fsSL --retry 5 --retry-connrefused '{url}' "
|
||||
f"| tee >(sha256sum > {sumf}) "
|
||||
f"| pv -n -f -i 2 -s {size} 2>{pvf} "
|
||||
f"| zstd -dc -T0 "
|
||||
f"| dd of='{disk}' bs=4M iflag=fullblock oflag=direct conv=fsync status=none")
|
||||
log(f"writing {url} -> {disk}")
|
||||
proc = subprocess.Popen(["bash", "-c", pipeline], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
pct = 0
|
||||
while proc.poll() is None:
|
||||
time.sleep(2)
|
||||
try:
|
||||
with open(pvf) as f:
|
||||
lines = f.read().split()
|
||||
if lines:
|
||||
pct = int(lines[-1])
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
prog.set("writing", min(pct, 99), f"{pct}% of {size // (1 << 20)} MiB written to {disk}")
|
||||
if proc.returncode != 0:
|
||||
raise RuntimeError(f"image write failed rc={proc.returncode}: {proc.stderr.read().strip()[-400:]}")
|
||||
got = readf(sumf).split()[0] if readf(sumf) else ""
|
||||
if got != want:
|
||||
raise RuntimeError(f"sha256 mismatch: got {got} want {want}")
|
||||
run(["sync"])
|
||||
run(["blockdev", "--rereadpt", disk])
|
||||
run(["partprobe", disk])
|
||||
run(["udevadm", "settle"])
|
||||
time.sleep(2)
|
||||
|
||||
|
||||
def find_parts(disk):
|
||||
parts = [d for d in lsblk(disk) if d.get("type") == "part"]
|
||||
root = esp = None
|
||||
for p in sorted(parts, key=lambda d: -int(d["size"])):
|
||||
if p.get("fstype") in ROOT_FS and root is None:
|
||||
root = p
|
||||
if (p.get("parttype") or "").lower() == EFI_GUID and p.get("fstype") == "vfat" and esp is None:
|
||||
esp = p
|
||||
return root, esp, parts
|
||||
|
||||
|
||||
def mount(dev, where, opts=None):
|
||||
os.makedirs(where, exist_ok=True)
|
||||
cmd = ["mount"] + (["-o", opts] if opts else []) + [dev, where]
|
||||
run(cmd, check=True)
|
||||
|
||||
|
||||
def seed(root, files):
|
||||
mnt = "/mnt/rollcall-root"
|
||||
# xfs root written from an image can carry a duplicate UUID with nothing else; nouuid is harmless
|
||||
try:
|
||||
mount(root["path"], mnt)
|
||||
except RuntimeError:
|
||||
mount(root["path"], mnt, "nouuid")
|
||||
try:
|
||||
if not os.path.isdir(f"{mnt}/etc"):
|
||||
raise RuntimeError(f"{root['path']} has no /etc; not a root filesystem")
|
||||
for f in files:
|
||||
dst = os.path.join(mnt, f["path"].lstrip("/"))
|
||||
os.makedirs(os.path.dirname(dst), exist_ok=True)
|
||||
with open(dst, "w") as fh:
|
||||
fh.write(f["content"])
|
||||
os.chmod(dst, int(f.get("mode", "0644"), 8))
|
||||
log(f"wrote {f['path']}")
|
||||
# cloud-init's stale state would stop it from re-running on a reused image
|
||||
run(f"rm -rf {mnt}/var/lib/cloud/instance {mnt}/var/lib/cloud/instances/* {mnt}/var/lib/cloud/data/* 2>/dev/null")
|
||||
sel = readf(f"{mnt}/etc/selinux/config")
|
||||
if re.search(r"^SELINUX=(enforcing|permissive)", sel, re.M):
|
||||
open(f"{mnt}/.autorelabel", "w").close()
|
||||
log("SELinux enabled in image: scheduled autorelabel for first boot")
|
||||
os_release = readf(f"{mnt}/etc/os-release")
|
||||
m = re.search(r'^PRETTY_NAME="?([^"\n]*)', os_release, re.M)
|
||||
run(["sync"])
|
||||
return m.group(1) if m else ""
|
||||
finally:
|
||||
run(["umount", mnt])
|
||||
|
||||
|
||||
def efi_entry(disk, esp, label):
|
||||
if not os.path.isdir("/sys/firmware/efi") or esp is None:
|
||||
return None
|
||||
mnt = "/mnt/rollcall-esp"
|
||||
try:
|
||||
mount(esp["path"], mnt, "ro")
|
||||
except RuntimeError as e:
|
||||
return f"esp mount failed: {e}"
|
||||
try:
|
||||
loader = None
|
||||
for name in ("shimx64.efi", "grubx64.efi"):
|
||||
for d in ("debian", "ubuntu", "rocky", "almalinux", "centos", "redhat", "fedora"):
|
||||
if os.path.exists(f"{mnt}/EFI/{d}/{name}"):
|
||||
loader = f"\\EFI\\{d}\\{name}"
|
||||
break
|
||||
if loader:
|
||||
break
|
||||
if not loader:
|
||||
hits = glob.glob(f"{mnt}/EFI/*/shimx64.efi") + glob.glob(f"{mnt}/EFI/*/grubx64.efi") + glob.glob(f"{mnt}/EFI/BOOT/BOOTX64.EFI")
|
||||
if hits:
|
||||
loader = "\\" + os.path.relpath(hits[0], mnt).replace("/", "\\")
|
||||
if not loader:
|
||||
return "no EFI loader found on ESP"
|
||||
finally:
|
||||
run(["umount", mnt])
|
||||
# drop stale entries with our label, then create a new one first in BootOrder
|
||||
cur = run(["efibootmgr"]).stdout
|
||||
for m in re.finditer(r"^Boot([0-9A-Fa-f]{4})\*?\s+(.*)$", cur, re.M):
|
||||
if m.group(2).strip().startswith(label):
|
||||
run(["efibootmgr", "-q", "-b", m.group(1), "-B"])
|
||||
p = run(["efibootmgr", "-c", "-d", disk, "-p", str(esp.get("partn")), "-L", label, "-l", loader])
|
||||
if p.returncode != 0:
|
||||
return f"efibootmgr failed: {p.stderr.strip()[-200:]}"
|
||||
return f"{label} -> {loader} (part {esp.get('partn')})"
|
||||
|
||||
|
||||
def do_deploy(job, prog):
|
||||
spec, files = job["spec"], job.get("files") or []
|
||||
started = time.time()
|
||||
prog.set("preparing", 0, "selecting disk", now=True)
|
||||
disk = find_disk(spec)
|
||||
size = int(disk["size"])
|
||||
if int(spec.get("raw_size") or 0) > size:
|
||||
raise RuntimeError(f"{disk['path']} ({size} bytes) smaller than image ({spec['raw_size']} bytes)")
|
||||
log(f"target {disk['path']} serial={disk.get('serial')} size={size}")
|
||||
prog.set("preparing", 1, f"wiping {disk['path']}", now=True)
|
||||
release_disk(disk["path"])
|
||||
write_image(spec, disk["path"], prog)
|
||||
prog.set("seeding", 99, "writing cloud-init seed", now=True)
|
||||
root, esp, parts = find_parts(disk["path"])
|
||||
if root is None:
|
||||
raise RuntimeError("no root filesystem found after writing image: " + ", ".join(f"{p['name']}:{p.get('fstype')}" for p in parts))
|
||||
pretty = seed(root, files)
|
||||
efi = efi_entry(disk["path"], esp, f"rollcall {spec.get('os', '')}")
|
||||
if efi:
|
||||
log(f"efi: {efi}")
|
||||
details = {"disk": disk["path"], "serial": disk.get("serial"), "root": root["path"], "root_fs": root.get("fstype"),
|
||||
"esp": esp["path"] if esp else None, "efi": efi, "os_release": pretty, "firmware": "uefi" if os.path.isdir("/sys/firmware/efi") else "bios",
|
||||
"seconds": round(time.time() - started, 1)}
|
||||
return details
|
||||
|
||||
|
||||
def do_release(job, prog):
|
||||
spec = job["spec"]
|
||||
prog.set("preparing", 0, "selecting disk", now=True)
|
||||
disk = find_disk(spec)
|
||||
path, size = disk["path"], int(disk["size"])
|
||||
prog.set("wiping", 5, f"wiping {path}", now=True)
|
||||
release_disk(path)
|
||||
mib = 64
|
||||
run(f"dd if=/dev/zero of={path} bs=1M count={mib} conv=fsync status=none")
|
||||
run(f"dd if=/dev/zero of={path} bs=1M count={mib} seek={max(0, size // (1 << 20) - mib)} conv=fsync status=none")
|
||||
if spec.get("wipe") == "full":
|
||||
p = run(["blkdiscard", "-f", path], timeout=3600)
|
||||
if p.returncode != 0:
|
||||
prog.set("wiping", 10, "full zero fill", now=True)
|
||||
run(f"dd if=/dev/zero of={path} bs=16M oflag=direct status=none", timeout=24 * 3600)
|
||||
run(["blockdev", "--rereadpt", path])
|
||||
return {"disk": path, "serial": disk.get("serial"), "wipe": spec.get("wipe")}
|
||||
|
||||
|
||||
def main():
|
||||
global SERVER, TOKEN
|
||||
opts = cmdline()
|
||||
SERVER = (opts.get("server") or "").rstrip("/")
|
||||
TOKEN = opts.get("job") if isinstance(opts.get("job"), str) else ""
|
||||
if not SERVER or not TOKEN:
|
||||
log("rollcall.server= and rollcall.job= required")
|
||||
return 2
|
||||
log(f"fetching job from {SERVER}")
|
||||
try:
|
||||
job = api("GET", "/api/v1/jobs/self")
|
||||
except Exception as e: # noqa: BLE001
|
||||
log(f"cannot fetch job: {e}; staying up")
|
||||
return 1
|
||||
log(f"job {job['id']} kind={job['kind']} spec={json.dumps({k: v for k, v in job['spec'].items() if k not in ('ssh_keys', 'user_data')})}")
|
||||
prog = Progress()
|
||||
try:
|
||||
details = do_deploy(job, prog) if job["kind"] == "deploy" else do_release(job, prog)
|
||||
resp = prog.finish("done", "finished", details=details)
|
||||
except Exception as e: # noqa: BLE001
|
||||
log(f"FAILED: {e!r}")
|
||||
try:
|
||||
resp = prog.finish("failed", str(e), error=str(e))
|
||||
except Exception as e2: # noqa: BLE001
|
||||
log(f"could not report failure: {e2}")
|
||||
resp = {"next": "stay"}
|
||||
nxt = resp.get("next", "stay")
|
||||
log(f"server says: {nxt}")
|
||||
if opts.get("nopoweroff") or nxt == "stay":
|
||||
log("staying up for inspection (ssh root@<ip>)")
|
||||
return 0 if nxt != "stay" else 1
|
||||
time.sleep(3)
|
||||
run(["systemctl", "reboot" if nxt == "reboot" else "poweroff"])
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+143
-3
@@ -99,9 +99,19 @@ func runClient(cmd string, args []string) error {
|
||||
if err := c.do("GET", "/api/v1/machines", nil, &ms); err != nil {
|
||||
return err
|
||||
}
|
||||
f := "%-16s %-12s %-10s %-10s %-14s %-24s %4s %7s %3s %7s %3s %2s %3s %s\n"
|
||||
fmt.Printf(f, "ID", "STATE", "NODE", "PROFILE", "MGMT-IP", "MODEL", "CPU", "RAM", "DSK", "STORAGE", "NIC", "IB", "GPU", "BMC")
|
||||
f := "%-16s %-14s %-10s %-16s %-14s %-24s %4s %7s %3s %7s %3s %2s %3s %s\n"
|
||||
fmt.Printf(f, "ID", "STATE", "NODE", "PROFILE/OS", "MGMT-IP", "MODEL", "CPU", "RAM", "DSK", "STORAGE", "NIC", "IB", "GPU", "BMC")
|
||||
for _, m := range ms {
|
||||
state := m.State
|
||||
if m.Job != nil {
|
||||
var p obj
|
||||
_ = json.Unmarshal(m.Job.Progress, &p)
|
||||
state = fmt.Sprintf("%s %.0f%%", m.Job.Kind, num(p["pct"]))
|
||||
}
|
||||
what := deref(m.Profile)
|
||||
if what == "" && m.OS != nil {
|
||||
what = *m.OS
|
||||
}
|
||||
bmc := "-"
|
||||
if m.BMC != nil {
|
||||
b := *m.BMC
|
||||
@@ -112,7 +122,7 @@ func runClient(cmd string, args []string) error {
|
||||
bmc = fmt.Sprintf("%s (%s,%s)", str(b["address"]), str(b["source"]), ok)
|
||||
}
|
||||
model := strings.TrimSpace(deref(m.Manufacturer) + " " + deref(m.Product))
|
||||
fmt.Printf(f, m.ID, m.State, deref(m.NodeName), deref(m.Profile), deref(m.MgmtIP), trunc(model, 24), m.CPUs, human(m.MemoryBytes),
|
||||
fmt.Printf(f, m.ID, state, deref(m.NodeName), trunc(what, 16), deref(m.MgmtIP), trunc(model, 24), m.CPUs, human(m.MemoryBytes),
|
||||
fmt.Sprint(int(m.Disks)), human(m.StorageBytes), fmt.Sprint(m.NICs), fmt.Sprint(m.IBHCAs), fmt.Sprint(m.GPUs), bmc)
|
||||
}
|
||||
case "show":
|
||||
@@ -177,12 +187,142 @@ func runClient(cmd string, args []string) error {
|
||||
return err
|
||||
}
|
||||
fmt.Println("deleted", str(d["deleted"]))
|
||||
case "deploy", "release", "jobs", "job":
|
||||
return c.deployCmd(cmd, args)
|
||||
default:
|
||||
return c.statelessCmd(cmd, args)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *client) deployCmd(cmd string, args []string) error {
|
||||
switch cmd {
|
||||
case "deploy":
|
||||
if len(args) < 2 {
|
||||
usage()
|
||||
}
|
||||
fs := newFlags("deploy")
|
||||
ver := fs.String("version", "", "image version (default: current)")
|
||||
disk := fs.String("disk", "", "target disk, e.g. /dev/sda (default: smallest disk that fits)")
|
||||
host := fs.String("hostname", "", "hostname (default: existing node name or next free)")
|
||||
upgrade := fs.Bool("upgrade", false, "cloud-init package_upgrade on first boot")
|
||||
pkgs := fs.String("packages", "", "comma-separated packages to install on first boot")
|
||||
ud := fs.String("user-data", "", "file with a complete #cloud-config to use instead of the generated one")
|
||||
boot := fs.Bool("boot", false, "set bootdev pxe and power-cycle via BMC")
|
||||
watch := fs.Bool("watch", false, "follow job progress")
|
||||
_ = fs.Parse(args[2:])
|
||||
req := obj{"os": args[1], "version": *ver, "disk": *disk, "hostname": *host, "package_upgrade": *upgrade, "boot": *boot}
|
||||
if *pkgs != "" {
|
||||
req["packages"] = strings.Split(*pkgs, ",")
|
||||
}
|
||||
if *ud != "" {
|
||||
b, err := os.ReadFile(*ud)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req["user_data"] = string(b)
|
||||
}
|
||||
var d obj
|
||||
if err := c.do("POST", "/api/v1/machines/"+args[0]+"/deploy", req, &d); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("%s: job %s %s/%s -> %s hostname=%s mgmt=%s\n", str(d["id"]), str(d["job"]), str(d["os"]), str(d["version"]), str(d["disk"]), str(d["hostname"]), str(d["mgmt_ip"]))
|
||||
if d["ipmi"] != nil {
|
||||
fmt.Println(str(d["ipmi"]))
|
||||
}
|
||||
if *watch {
|
||||
return c.watchJob(args[0])
|
||||
}
|
||||
case "release":
|
||||
if len(args) < 1 {
|
||||
usage()
|
||||
}
|
||||
fs := newFlags("release")
|
||||
boot := fs.Bool("boot", false, "set bootdev pxe and power-cycle via BMC")
|
||||
wipe := fs.String("wipe", "quick", "quick (signatures + first/last 64 MiB) or full")
|
||||
watch := fs.Bool("watch", false, "follow job progress")
|
||||
_ = fs.Parse(args[1:])
|
||||
var d obj
|
||||
if err := c.do("POST", "/api/v1/machines/"+args[0]+"/release", obj{"boot": *boot, "wipe": *wipe}, &d); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("%s: job %s wipe %s\n", str(d["id"]), str(d["job"]), str(d["disk"]))
|
||||
if d["ipmi"] != nil {
|
||||
fmt.Println(str(d["ipmi"]))
|
||||
}
|
||||
if *watch {
|
||||
return c.watchJob(args[0])
|
||||
}
|
||||
case "jobs":
|
||||
if len(args) < 1 {
|
||||
usage()
|
||||
}
|
||||
if len(args) > 1 && args[1] == "-watch" {
|
||||
return c.watchJob(args[0])
|
||||
}
|
||||
var jobs []Job
|
||||
if err := c.do("GET", "/api/v1/machines/"+args[0]+"/jobs", nil, &jobs); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("%-10s %-8s %-10s %-20s %-20s %s\n", "JOB", "KIND", "STATE", "CREATED", "FINISHED", "OS / PROGRESS / ERROR")
|
||||
for _, j := range jobs {
|
||||
sp := j.spec()
|
||||
var p obj
|
||||
_ = json.Unmarshal(j.Progress, &p)
|
||||
fin := ""
|
||||
if j.Finished != nil {
|
||||
fin = j.Finished.Local().Format("2006-01-02 15:04:05")
|
||||
}
|
||||
extra := sp.OS + "/" + sp.Version + " " + sp.Disk
|
||||
if j.State == "running" {
|
||||
extra += fmt.Sprintf(" %s %.0f%% %s", str(p["phase"]), num(p["pct"]), str(p["msg"]))
|
||||
}
|
||||
if j.Error != nil {
|
||||
extra += " " + *j.Error
|
||||
}
|
||||
fmt.Printf("%-10s %-8s %-10s %-20s %-20s %s\n", j.ID, j.Kind, j.State, j.Created.Local().Format("2006-01-02 15:04:05"), fin, extra)
|
||||
}
|
||||
case "job":
|
||||
if len(args) < 2 || args[0] != "cancel" {
|
||||
usage()
|
||||
}
|
||||
var d obj
|
||||
if err := c.do("POST", "/api/v1/machines/"+args[1]+"/jobs/cancel", nil, &d); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("job %s cancelled\n", str(d["job"]))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *client) watchJob(id string) error {
|
||||
last := ""
|
||||
for {
|
||||
var jobs []Job
|
||||
if err := c.do("GET", "/api/v1/machines/"+id+"/jobs", nil, &jobs); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(jobs) == 0 {
|
||||
return fmt.Errorf("no jobs")
|
||||
}
|
||||
j := jobs[0]
|
||||
var p obj
|
||||
_ = json.Unmarshal(j.Progress, &p)
|
||||
line := fmt.Sprintf("%s %s %-9s %-12s %3.0f%% %s", time.Now().Format("15:04:05"), j.ID, j.State, str(p["phase"]), num(p["pct"]), str(p["msg"]))
|
||||
if line[9:] != last {
|
||||
fmt.Println(line)
|
||||
last = line[9:]
|
||||
}
|
||||
if j.State != "pending" && j.State != "running" {
|
||||
if j.Error != nil {
|
||||
return fmt.Errorf("%s", *j.Error)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
time.Sleep(3 * time.Second)
|
||||
}
|
||||
}
|
||||
|
||||
func (c *client) statelessCmd(cmd string, args []string) error {
|
||||
switch cmd {
|
||||
case "images":
|
||||
|
||||
@@ -28,6 +28,10 @@ type Config struct {
|
||||
DnsmasqDir string `json:"dnsmasq_dir"` // hosts.dhcp + hosts written here for dnsmasq (dhcp-hostsfile / addn-hosts); "" disables
|
||||
NodePools map[string]string `json:"node_pools"` // {"mgmt": "10.0.0.50-10.0.0.99", "ib": "10.1.0.10-10.1.0.250"}
|
||||
ClusterVars map[string]any `json:"cluster_vars"` // available to overlay templates as .Vars (profile vars override)
|
||||
|
||||
// disk-based deployment
|
||||
OSDir string `json:"os_dir"` // <os_dir>/<name>/<version>/{disk.raw.zst,meta.json}, served under /os/
|
||||
OSSources string `json:"os_sources"` // JSON list of upstream cloud images for `rollcall os sync`
|
||||
}
|
||||
|
||||
// BMCHint maps a machine MAC to a known BMC (virtual lab nodes).
|
||||
@@ -68,5 +72,11 @@ func loadConfig(path string) (*Config, error) {
|
||||
if c.OverlaysDir == "" {
|
||||
c.OverlaysDir = "/etc/rollcall/overlays"
|
||||
}
|
||||
if c.OSDir == "" {
|
||||
c.OSDir = filepath.Join(filepath.Dir(c.ImageDir), "os")
|
||||
}
|
||||
if c.OSSources == "" {
|
||||
c.OSSources = "/etc/rollcall/os-sources.json"
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
@@ -0,0 +1,643 @@
|
||||
package main
|
||||
|
||||
// Disk-based OS deployment: a job boots the commissioning image in deploy
|
||||
// mode; the agent streams <os>/<version>/disk.raw.zst onto the chosen disk,
|
||||
// seeds cloud-init (NoCloud) in the written root filesystem and reports
|
||||
// progress. On success the machine boots from disk and cloud-init phones home.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Job struct {
|
||||
ID string `json:"id"`
|
||||
MachineID string `json:"machine_id"`
|
||||
Kind string `json:"kind"` // deploy | release
|
||||
State string `json:"state"`
|
||||
Spec json.RawMessage `json:"spec"`
|
||||
Progress json.RawMessage `json:"progress"`
|
||||
Created time.Time `json:"created"`
|
||||
Started *time.Time `json:"started"`
|
||||
Finished *time.Time `json:"finished"`
|
||||
Error *string `json:"error"`
|
||||
}
|
||||
|
||||
// DeploySpec is what the agent receives (plus the rendered seed files).
|
||||
type DeploySpec struct {
|
||||
OS string `json:"os"`
|
||||
Version string `json:"version"`
|
||||
ImageURL string `json:"image_url"`
|
||||
SHA256 string `json:"sha256"`
|
||||
Size int64 `json:"size"`
|
||||
RawSize int64 `json:"raw_size"`
|
||||
Disk string `json:"disk"`
|
||||
DiskSerial string `json:"disk_serial,omitempty"`
|
||||
Hostname string `json:"hostname"`
|
||||
Domain string `json:"domain,omitempty"`
|
||||
SSHKeys []string `json:"ssh_keys"`
|
||||
PackageUpgrade bool `json:"package_upgrade"`
|
||||
Packages []string `json:"packages,omitempty"`
|
||||
UserData string `json:"user_data,omitempty"` // full #cloud-config override
|
||||
Wipe string `json:"wipe,omitempty"` // release: quick | full
|
||||
}
|
||||
|
||||
type seedFile struct {
|
||||
Path string `json:"path"`
|
||||
Mode string `json:"mode"`
|
||||
Content string `json:"content"`
|
||||
}
|
||||
|
||||
func (j *Job) spec() DeploySpec {
|
||||
var s DeploySpec
|
||||
_ = json.Unmarshal(j.Spec, &s)
|
||||
return s
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- store
|
||||
|
||||
const jobCols = `id, machine_id, kind, state, spec, progress, created, started, finished, error`
|
||||
|
||||
func scanJob(row interface{ Scan(...any) error }) (*Job, error) {
|
||||
j := &Job{}
|
||||
err := row.Scan(&j.ID, &j.MachineID, &j.Kind, &j.State, &j.Spec, &j.Progress, &j.Created, &j.Started, &j.Finished, &j.Error)
|
||||
if err != nil {
|
||||
if strings.Contains(err.Error(), "no rows") {
|
||||
return nil, errNotFound
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
return j, nil
|
||||
}
|
||||
|
||||
func (s *Store) CreateJob(ctx context.Context, j *Job, tokenHash string, sealed []byte) error {
|
||||
_, err := s.pool.Exec(ctx, `INSERT INTO jobs(id, machine_id, kind, state, spec, token_hash, token) VALUES ($1,$2,$3,'pending',$4,$5,$6)`,
|
||||
j.ID, j.MachineID, j.Kind, j.Spec, tokenHash, sealed)
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) JobByToken(ctx context.Context, hash string) (*Job, error) {
|
||||
return scanJob(s.pool.QueryRow(ctx, `SELECT `+jobCols+` FROM jobs WHERE token_hash=$1`, hash))
|
||||
}
|
||||
|
||||
func (s *Store) Job(ctx context.Context, id string) (*Job, error) {
|
||||
return scanJob(s.pool.QueryRow(ctx, `SELECT `+jobCols+` FROM jobs WHERE id=$1`, id))
|
||||
}
|
||||
|
||||
func (s *Store) ActiveJob(ctx context.Context, machineID string) (*Job, error) {
|
||||
return scanJob(s.pool.QueryRow(ctx, `SELECT `+jobCols+` FROM jobs WHERE machine_id=$1 AND state IN ('pending','running') ORDER BY created DESC LIMIT 1`, machineID))
|
||||
}
|
||||
|
||||
func (s *Store) ActiveJobs(ctx context.Context) (map[string]*Job, error) {
|
||||
rows, err := s.pool.Query(ctx, `SELECT `+jobCols+` FROM jobs WHERE state IN ('pending','running') ORDER BY created`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
out := map[string]*Job{}
|
||||
for rows.Next() {
|
||||
j, err := scanJob(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out[j.MachineID] = j
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
func (s *Store) Jobs(ctx context.Context, machineID string, limit int) ([]*Job, error) {
|
||||
rows, err := s.pool.Query(ctx, `SELECT `+jobCols+` FROM jobs WHERE machine_id=$1 ORDER BY created DESC LIMIT $2`, machineID, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
out := []*Job{}
|
||||
for rows.Next() {
|
||||
j, err := scanJob(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, j)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
func (s *Store) JobProgress(ctx context.Context, id string, progress []byte) error {
|
||||
_, err := s.pool.Exec(ctx, `UPDATE jobs SET state='running', started=coalesce(started, now()), progress=$2 WHERE id=$1 AND state IN ('pending','running')`, id, progress)
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) FinishJob(ctx context.Context, id, state, errMsg string) error {
|
||||
_, err := s.pool.Exec(ctx, `UPDATE jobs SET state=$2, finished=now(), started=coalesce(started, now()), error=nullif($3,'') WHERE id=$1`, id, state, errMsg)
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) SetDeployed(ctx context.Context, id, os, version string) error {
|
||||
_, err := s.pool.Exec(ctx, `UPDATE machines SET state='deployed', os=$2, os_version=$3, deployed_at=now(), booted_at=NULL, last_seen=now() WHERE id=$1`, id, os, version)
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) SetReleased(ctx context.Context, id string) error {
|
||||
_, err := s.pool.Exec(ctx, `UPDATE machines SET state='ready', os=NULL, os_version=NULL, deployed_at=NULL, booted_at=NULL, last_seen=now() WHERE id=$1`, id)
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) SetNodeAddr(ctx context.Context, id, name, mgmt, ib string) error {
|
||||
_, err := s.pool.Exec(ctx, `UPDATE machines SET node_name=$2, mgmt_ip=$3, ib_ip=$4 WHERE id=$1`, id, name, nilIfEmpty(mgmt), nilIfEmpty(ib))
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) MarkPhoneHome(ctx context.Context, id string) error {
|
||||
_, err := s.pool.Exec(ctx, `UPDATE machines SET booted_at=now(), last_seen=now() WHERE id=$1`, id)
|
||||
return err
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- disk selection
|
||||
|
||||
type invDisk struct {
|
||||
Name, Path, Serial, Tran, Model string
|
||||
Size int64
|
||||
Rota, Removable bool
|
||||
}
|
||||
|
||||
func inventoryDisks(m *Machine) []invDisk {
|
||||
var inv obj
|
||||
_ = json.Unmarshal(m.Inventory, &inv)
|
||||
var out []invDisk
|
||||
for _, dv := range toList(dig(inv, "storage", "disks")) {
|
||||
d, _ := dv.(obj)
|
||||
rm, _ := d["rm"].(bool)
|
||||
rota, _ := d["rota"].(bool)
|
||||
id := invDisk{Name: str(d["name"]), Path: str(d["path"]), Serial: str(d["serial"]), Tran: str(d["tran"]),
|
||||
Model: str(d["model"]), Size: int64(num(d["size"])), Rota: rota, Removable: rm}
|
||||
if id.Path == "" {
|
||||
id.Path = "/dev/" + id.Name
|
||||
}
|
||||
out = append(out, id)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// pickDisk: the smallest non-removable, non-USB disk that fits the image (boot/OS disks are
|
||||
// usually the small ones on servers; data disks stay untouched). Explicit choice wins.
|
||||
func pickDisk(disks []invDisk, want string, need int64) (*invDisk, error) {
|
||||
if want != "" {
|
||||
for i := range disks {
|
||||
if disks[i].Path == want || disks[i].Name == want || "/dev/"+disks[i].Name == want {
|
||||
if disks[i].Size > 0 && disks[i].Size < need {
|
||||
return nil, fmt.Errorf("disk %s (%s) is smaller than the image (%s)", want, human(float64(disks[i].Size)), human(float64(need)))
|
||||
}
|
||||
return &disks[i], nil
|
||||
}
|
||||
}
|
||||
if len(disks) == 0 {
|
||||
return &invDisk{Path: want, Name: strings.TrimPrefix(want, "/dev/")}, nil
|
||||
}
|
||||
return nil, fmt.Errorf("disk %s not in inventory", want)
|
||||
}
|
||||
var cands []invDisk
|
||||
for _, d := range disks {
|
||||
if d.Removable || d.Tran == "usb" || d.Size < need {
|
||||
continue
|
||||
}
|
||||
cands = append(cands, d)
|
||||
}
|
||||
if len(cands) == 0 {
|
||||
return nil, fmt.Errorf("no disk >= %s in inventory (pass disk explicitly)", human(float64(need)))
|
||||
}
|
||||
sort.Slice(cands, func(i, j int) bool {
|
||||
if cands[i].Size != cands[j].Size {
|
||||
return cands[i].Size < cands[j].Size
|
||||
}
|
||||
return cands[i].Name < cands[j].Name
|
||||
})
|
||||
return &cands[0], nil
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- cloud-init seed
|
||||
|
||||
func yamlStr(s string) string { b, _ := json.Marshal(s); return string(b) }
|
||||
func yamlList(l []string) string {
|
||||
if l == nil {
|
||||
l = []string{}
|
||||
}
|
||||
b, _ := json.Marshal(l)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func (a *App) renderSeed(j *Job, sp DeploySpec, token string) []seedFile {
|
||||
meta := fmt.Sprintf("instance-id: %s\nlocal-hostname: %s\n", j.ID, sp.Hostname)
|
||||
ud := sp.UserData
|
||||
if ud == "" {
|
||||
var b strings.Builder
|
||||
b.WriteString("#cloud-config\n# generated by rollcall for " + j.MachineID + " (" + j.ID + ")\n")
|
||||
fmt.Fprintf(&b, "hostname: %s\n", yamlStr(sp.Hostname))
|
||||
if sp.Domain != "" {
|
||||
fmt.Fprintf(&b, "fqdn: %s\n", yamlStr(sp.Hostname+"."+sp.Domain))
|
||||
}
|
||||
b.WriteString("manage_etc_hosts: true\npreserve_hostname: false\ndisable_root: false\nssh_pwauth: false\n")
|
||||
fmt.Fprintf(&b, "ssh_authorized_keys: %s\n", yamlList(sp.SSHKeys))
|
||||
fmt.Fprintf(&b, "users:\n - default\n - name: root\n ssh_authorized_keys: %s\n", yamlList(sp.SSHKeys))
|
||||
if sp.PackageUpgrade || len(sp.Packages) > 0 {
|
||||
b.WriteString("package_update: true\n")
|
||||
}
|
||||
if sp.PackageUpgrade {
|
||||
b.WriteString("package_upgrade: true\npackage_reboot_if_required: true\n")
|
||||
}
|
||||
if len(sp.Packages) > 0 {
|
||||
fmt.Fprintf(&b, "packages: %s\n", yamlList(sp.Packages))
|
||||
}
|
||||
fmt.Fprintf(&b, "phone_home:\n url: %s/api/v1/phone-home/%s\n post: [instance_id, hostname, fqdn, pub_key_ed25519]\n tries: 10\n", a.cfg.ServerURL, token)
|
||||
b.WriteString("final_message: \"rollcall: deployed $hostname up after $UPTIME s\"\n")
|
||||
ud = b.String()
|
||||
}
|
||||
return []seedFile{
|
||||
{Path: "var/lib/cloud/seed/nocloud/meta-data", Mode: "0600", Content: meta},
|
||||
{Path: "var/lib/cloud/seed/nocloud/user-data", Mode: "0600", Content: ud},
|
||||
{Path: "etc/cloud/cloud.cfg.d/99-rollcall.cfg", Mode: "0644", Content: "# written by rollcall at deploy time\ndatasource_list: [ NoCloud, None ]\n"},
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- PXE
|
||||
|
||||
const deployTmpl = `# generated by rollcall %s
|
||||
# machine %s job %s %s %s/%s -> %s
|
||||
DEFAULT menu.c32
|
||||
PROMPT 0
|
||||
TIMEOUT 30
|
||||
MENU TITLE %s - %s %s (job %s)
|
||||
|
||||
LABEL deploy
|
||||
MENU LABEL Run deployment job (default in 3s)
|
||||
MENU DEFAULT
|
||||
KERNEL rollcall/vmlinuz
|
||||
INITRD rollcall/initrd.img
|
||||
APPEND %s
|
||||
|
||||
LABEL recommission
|
||||
MENU LABEL Re-run commissioning
|
||||
KERNEL rollcall/vmlinuz
|
||||
INITRD rollcall/initrd.img
|
||||
APPEND %s
|
||||
|
||||
LABEL local
|
||||
MENU LABEL Boot from local disk
|
||||
LOCALBOOT 0
|
||||
`
|
||||
|
||||
const deployedTmpl = `# generated by rollcall %s
|
||||
# machine %s deployed %s/%s
|
||||
DEFAULT menu.c32
|
||||
PROMPT 0
|
||||
TIMEOUT 30
|
||||
MENU TITLE %s - deployed %s (%s)
|
||||
|
||||
LABEL local
|
||||
MENU LABEL Boot %s from local disk (default in 3s)
|
||||
MENU DEFAULT
|
||||
LOCALBOOT 0
|
||||
|
||||
LABEL recommission
|
||||
MENU LABEL Re-run commissioning
|
||||
KERNEL rollcall/vmlinuz
|
||||
INITRD rollcall/initrd.img
|
||||
APPEND %s
|
||||
`
|
||||
|
||||
// deployAppend derives the deploy-mode cmdline from commission_append (same image, no agent token).
|
||||
func (a *App) deployAppend(token string) string {
|
||||
var parts []string
|
||||
for _, p := range strings.Fields(a.cfg.CommissionAppend) {
|
||||
if strings.HasPrefix(p, "rollcall.token=") || strings.HasPrefix(p, "rollcall.nopoweroff") {
|
||||
continue
|
||||
}
|
||||
parts = append(parts, p)
|
||||
}
|
||||
parts = append(parts, "rollcall.mode=deploy", "rollcall.job="+token)
|
||||
return strings.Join(parts, " ")
|
||||
}
|
||||
|
||||
func (a *App) writePXEDeploy(m *Machine, j *Job, sp DeploySpec, token string) error {
|
||||
body := fmt.Sprintf(deployTmpl, time.Now().Format(time.RFC3339), m.ID, j.ID, j.Kind, sp.OS, sp.Version, sp.Disk,
|
||||
m.ID, j.Kind, sp.OS, j.ID, a.deployAppend(token), a.cfg.CommissionAppend)
|
||||
return a.writePXEBody(m, body)
|
||||
}
|
||||
|
||||
func (a *App) writePXEDeployed(m *Machine) error {
|
||||
body := fmt.Sprintf(deployedTmpl, time.Now().Format(time.RFC3339), m.ID, deref(m.OS), deref(m.OSVersion),
|
||||
deref(m.NodeName), deref(m.OS), m.ID, deref(m.OS), a.cfg.CommissionAppend)
|
||||
return a.writePXEBody(m, body)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- job creation
|
||||
|
||||
type deployRequest struct {
|
||||
OS string `json:"os"`
|
||||
Version string `json:"version"`
|
||||
Disk string `json:"disk"`
|
||||
Hostname string `json:"hostname"`
|
||||
SSHKeys []string `json:"ssh_keys"`
|
||||
PackageUpgrade bool `json:"package_upgrade"`
|
||||
Packages []string `json:"packages"`
|
||||
UserData string `json:"user_data"`
|
||||
Boot bool `json:"boot"`
|
||||
Wipe string `json:"wipe"`
|
||||
}
|
||||
|
||||
func (a *App) configSSHKeys() []string {
|
||||
var keys []string
|
||||
for _, k := range toList(a.cfg.ClusterVars["ssh_keys"]) {
|
||||
if s := str(k); s != "" {
|
||||
keys = append(keys, s)
|
||||
}
|
||||
}
|
||||
return keys
|
||||
}
|
||||
|
||||
// startJob creates the job row, PXE entry and (optionally) power-cycles the machine.
|
||||
func (a *App) startJob(ctx context.Context, m *Machine, kind string, sp DeploySpec, boot bool) (obj, error) {
|
||||
if m.Profile != nil {
|
||||
return nil, fmt.Errorf("machine is a stateless node (profile %s); unassign it first", *m.Profile)
|
||||
}
|
||||
if j, err := a.store.ActiveJob(ctx, m.ID); err == nil {
|
||||
return nil, fmt.Errorf("job %s (%s, %s) is still active; cancel it first", j.ID, j.Kind, j.State)
|
||||
}
|
||||
token := randomToken(16)
|
||||
sum := sha256.Sum256([]byte(token))
|
||||
sealed, err := a.sealer.Seal(token, m.ID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
specB, _ := json.Marshal(sp)
|
||||
j := &Job{ID: "j-" + randomToken(5), MachineID: m.ID, Kind: kind, State: "pending", Spec: specB}
|
||||
if err := a.store.CreateJob(ctx, j, hex.EncodeToString(sum[:]), sealed); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
state := "deploying"
|
||||
if kind == "release" {
|
||||
state = "releasing"
|
||||
}
|
||||
_ = a.store.SetState(ctx, m.ID, state)
|
||||
if err := a.writePXEDeploy(m, j, sp, token); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
_ = a.store.AddEvent(ctx, m.ID, kind+" requested", fmt.Sprintf("job=%s os=%s/%s disk=%s hostname=%s", j.ID, sp.OS, sp.Version, sp.Disk, sp.Hostname))
|
||||
out := obj{"id": m.ID, "job": j.ID, "state": state, "os": sp.OS, "version": sp.Version, "disk": sp.Disk, "hostname": sp.Hostname}
|
||||
if boot {
|
||||
out["ipmi"], out["ipmi_ok"] = a.pxeBoot(ctx, m)
|
||||
}
|
||||
log.Printf("%s %s: job %s os=%s/%s disk=%s", kind, m.ID, j.ID, sp.OS, sp.Version, sp.Disk)
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (a *App) handleDeploy(w http.ResponseWriter, r *http.Request) {
|
||||
m, err := a.store.Get(r.Context(), r.PathValue("id"))
|
||||
if err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
var req deployRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil || req.OS == "" {
|
||||
fail(w, 400, fmt.Errorf("os is required"))
|
||||
return
|
||||
}
|
||||
meta, err := osResolve(a.cfg.OSDir, req.OS, req.Version)
|
||||
if err != nil {
|
||||
fail(w, 404, err)
|
||||
return
|
||||
}
|
||||
disk, err := pickDisk(inventoryDisks(m), req.Disk, meta.RawSize)
|
||||
if err != nil {
|
||||
fail(w, 409, err)
|
||||
return
|
||||
}
|
||||
// node name + management IP: keep an existing one, otherwise allocate like a stateless node
|
||||
name, mgmt, ib := deref(m.NodeName), deref(m.MgmtIP), deref(m.IBIP)
|
||||
if req.Hostname != "" && req.Hostname != name {
|
||||
name = ""
|
||||
}
|
||||
if name == "" {
|
||||
if name, mgmt, ib, err = a.allocate(r.Context(), str(a.cfg.ClusterVars["name_format"]), req.Hostname); err != nil {
|
||||
fail(w, 409, err)
|
||||
return
|
||||
}
|
||||
if err := a.store.SetNodeAddr(r.Context(), m.ID, name, mgmt, ib); err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
m, _ = a.store.Get(r.Context(), m.ID)
|
||||
if err := a.exportDnsmasq(r.Context()); err != nil {
|
||||
log.Printf("dnsmasq export: %v", err)
|
||||
}
|
||||
}
|
||||
keys := req.SSHKeys
|
||||
if keys == nil {
|
||||
keys = a.configSSHKeys()
|
||||
}
|
||||
sp := DeploySpec{OS: meta.Name, Version: meta.Version,
|
||||
ImageURL: a.cfg.ServerURL + "/os/" + meta.Name + "/" + meta.Version + "/disk.raw.zst",
|
||||
SHA256: meta.SHA256, Size: meta.Size, RawSize: meta.RawSize, Disk: disk.Path, DiskSerial: disk.Serial,
|
||||
Hostname: name, Domain: str(a.cfg.ClusterVars["domain"]), SSHKeys: keys, PackageUpgrade: req.PackageUpgrade,
|
||||
Packages: req.Packages, UserData: req.UserData}
|
||||
out, err := a.startJob(r.Context(), m, "deploy", sp, req.Boot)
|
||||
if err != nil {
|
||||
fail(w, 409, err)
|
||||
return
|
||||
}
|
||||
out["mgmt_ip"] = mgmt
|
||||
writeJSON(w, 200, out)
|
||||
}
|
||||
|
||||
func (a *App) handleRelease(w http.ResponseWriter, r *http.Request) {
|
||||
m, err := a.store.Get(r.Context(), r.PathValue("id"))
|
||||
if err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
var req deployRequest
|
||||
_ = json.NewDecoder(r.Body).Decode(&req)
|
||||
if req.Wipe == "" {
|
||||
req.Wipe = "quick"
|
||||
}
|
||||
disk, err := pickDisk(inventoryDisks(m), req.Disk, 0)
|
||||
if err != nil {
|
||||
fail(w, 409, err)
|
||||
return
|
||||
}
|
||||
sp := DeploySpec{OS: deref(m.OS), Version: deref(m.OSVersion), Disk: disk.Path, DiskSerial: disk.Serial, Wipe: req.Wipe, Hostname: deref(m.NodeName)}
|
||||
out, err := a.startJob(r.Context(), m, "release", sp, req.Boot)
|
||||
if err != nil {
|
||||
fail(w, 409, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, 200, out)
|
||||
}
|
||||
|
||||
func (a *App) handleJobs(w http.ResponseWriter, r *http.Request) {
|
||||
m, err := a.store.Get(r.Context(), r.PathValue("id"))
|
||||
if err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
jobs, err := a.store.Jobs(r.Context(), m.ID, 20)
|
||||
if err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, 200, jobs)
|
||||
}
|
||||
|
||||
func (a *App) handleJobCancel(w http.ResponseWriter, r *http.Request) {
|
||||
m, err := a.store.Get(r.Context(), r.PathValue("id"))
|
||||
if err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
j, err := a.store.ActiveJob(r.Context(), m.ID)
|
||||
if err != nil {
|
||||
fail(w, 404, fmt.Errorf("no active job"))
|
||||
return
|
||||
}
|
||||
_ = a.store.FinishJob(r.Context(), j.ID, "cancelled", "cancelled by operator")
|
||||
a.afterJob(r.Context(), m, j, "cancelled", "cancelled by operator")
|
||||
writeJSON(w, 200, obj{"id": m.ID, "job": j.ID, "state": "cancelled"})
|
||||
}
|
||||
|
||||
// afterJob settles machine state and the PXE entry once a job stopped (done/failed/cancelled).
|
||||
func (a *App) afterJob(ctx context.Context, m *Machine, j *Job, state, errMsg string) {
|
||||
sp := j.spec()
|
||||
switch {
|
||||
case state == "done" && j.Kind == "deploy":
|
||||
_ = a.store.SetDeployed(ctx, m.ID, sp.OS, sp.Version)
|
||||
m, _ = a.store.Get(ctx, m.ID)
|
||||
if err := a.writePXEDeployed(m); err != nil {
|
||||
log.Printf("pxe write %s: %v", m.ID, err)
|
||||
}
|
||||
_ = a.store.AddEvent(ctx, m.ID, "deployed", fmt.Sprintf("job=%s os=%s/%s disk=%s", j.ID, sp.OS, sp.Version, sp.Disk))
|
||||
case state == "done" && j.Kind == "release":
|
||||
_ = a.store.SetReleased(ctx, m.ID)
|
||||
m, _ = a.store.Get(ctx, m.ID)
|
||||
_ = a.writePXE(m)
|
||||
_ = a.store.AddEvent(ctx, m.ID, "released", fmt.Sprintf("job=%s disk=%s wipe=%s", j.ID, sp.Disk, sp.Wipe))
|
||||
default:
|
||||
_ = a.store.SetState(ctx, m.ID, j.Kind+"-failed")
|
||||
m, _ = a.store.Get(ctx, m.ID)
|
||||
_ = a.writePXE(m) // next PXE boot: commissioned menu (power off), no retry loop
|
||||
_ = a.store.AddEvent(ctx, m.ID, j.Kind+" "+state, "job="+j.ID+" "+errMsg)
|
||||
}
|
||||
log.Printf("job %s (%s %s): %s %s", j.ID, j.Kind, m.ID, state, errMsg)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- agent side (job token)
|
||||
|
||||
func (a *App) jobFromRequest(w http.ResponseWriter, r *http.Request) (*Job, string) {
|
||||
tok := r.Header.Get("X-Rollcall-Job")
|
||||
if tok == "" {
|
||||
fail(w, 401, errors.New("X-Rollcall-Job required"))
|
||||
return nil, ""
|
||||
}
|
||||
sum := sha256.Sum256([]byte(tok))
|
||||
j, err := a.store.JobByToken(r.Context(), hex.EncodeToString(sum[:]))
|
||||
if err != nil {
|
||||
fail(w, 401, errors.New("unknown job token"))
|
||||
return nil, ""
|
||||
}
|
||||
return j, tok
|
||||
}
|
||||
|
||||
func (a *App) handleJobSelf(w http.ResponseWriter, r *http.Request) {
|
||||
j, tok := a.jobFromRequest(w, r)
|
||||
if j == nil {
|
||||
return
|
||||
}
|
||||
if j.State != "pending" && j.State != "running" {
|
||||
fail(w, 409, fmt.Errorf("job %s is %s", j.ID, j.State))
|
||||
return
|
||||
}
|
||||
sp := j.spec()
|
||||
out := obj{"id": j.ID, "machine_id": j.MachineID, "kind": j.Kind, "spec": sp}
|
||||
if j.Kind == "deploy" {
|
||||
out["files"] = a.renderSeed(j, sp, tok)
|
||||
}
|
||||
if j.State == "pending" {
|
||||
_ = a.store.JobProgress(r.Context(), j.ID, []byte(`{"phase":"started","pct":0}`))
|
||||
_ = a.store.AddEvent(r.Context(), j.MachineID, j.Kind+" started", "job="+j.ID+" from "+r.RemoteAddr)
|
||||
}
|
||||
writeJSON(w, 200, out)
|
||||
}
|
||||
|
||||
func (a *App) handleJobProgress(w http.ResponseWriter, r *http.Request) {
|
||||
j, _ := a.jobFromRequest(w, r)
|
||||
if j == nil {
|
||||
return
|
||||
}
|
||||
var p struct {
|
||||
State string `json:"state"` // "" (progress), done, failed
|
||||
Phase string `json:"phase"`
|
||||
Pct float64 `json:"pct"`
|
||||
Msg string `json:"msg"`
|
||||
Error string `json:"error"`
|
||||
Details map[string]any `json:"details"`
|
||||
}
|
||||
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&p); err != nil {
|
||||
fail(w, 400, err)
|
||||
return
|
||||
}
|
||||
if j.State != "pending" && j.State != "running" {
|
||||
writeJSON(w, 200, obj{"job": j.ID, "state": j.State, "next": "poweroff"})
|
||||
return
|
||||
}
|
||||
prog, _ := json.Marshal(obj{"phase": p.Phase, "pct": p.Pct, "msg": p.Msg, "details": p.Details, "updated": time.Now()})
|
||||
_ = a.store.JobProgress(r.Context(), j.ID, prog)
|
||||
m, err := a.store.Get(r.Context(), j.MachineID)
|
||||
if err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
next := "continue"
|
||||
switch p.State {
|
||||
case "done":
|
||||
_ = a.store.FinishJob(r.Context(), j.ID, "done", "")
|
||||
a.afterJob(r.Context(), m, j, "done", "")
|
||||
next = "reboot"
|
||||
if j.Kind == "release" {
|
||||
next = "poweroff"
|
||||
}
|
||||
case "failed":
|
||||
if p.Error == "" {
|
||||
p.Error = p.Msg
|
||||
}
|
||||
_ = a.store.FinishJob(r.Context(), j.ID, "failed", p.Error)
|
||||
a.afterJob(r.Context(), m, j, "failed", p.Error)
|
||||
next = "stay"
|
||||
}
|
||||
writeJSON(w, 200, obj{"job": j.ID, "state": p.State, "next": next})
|
||||
}
|
||||
|
||||
// handlePhoneHome is cloud-init's phone_home from the freshly deployed OS (form-encoded POST).
|
||||
func (a *App) handlePhoneHome(w http.ResponseWriter, r *http.Request) {
|
||||
sum := sha256.Sum256([]byte(r.PathValue("token")))
|
||||
j, err := a.store.JobByToken(r.Context(), hex.EncodeToString(sum[:]))
|
||||
if err != nil || j.Kind != "deploy" {
|
||||
fail(w, 404, errNotFound)
|
||||
return
|
||||
}
|
||||
_ = r.ParseForm()
|
||||
_ = a.store.MarkPhoneHome(r.Context(), j.MachineID)
|
||||
detail := fmt.Sprintf("job=%s from %s hostname=%s fqdn=%s", j.ID, r.RemoteAddr, r.Form.Get("hostname"), r.Form.Get("fqdn"))
|
||||
if k := strings.TrimSpace(r.Form.Get("pub_key_ed25519")); k != "" {
|
||||
detail += " hostkey=" + k
|
||||
}
|
||||
_ = a.store.AddEvent(r.Context(), j.MachineID, "phone-home", detail)
|
||||
log.Printf("phone-home %s: %s", j.MachineID, detail)
|
||||
writeJSON(w, 200, obj{"ok": true})
|
||||
}
|
||||
@@ -11,6 +11,7 @@ require (
|
||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||
github.com/klauspost/compress v1.18.0 // indirect
|
||||
golang.org/x/sync v0.13.0 // indirect
|
||||
golang.org/x/sys v0.32.0 // indirect
|
||||
golang.org/x/text v0.24.0 // indirect
|
||||
|
||||
@@ -9,6 +9,8 @@ github.com/jackc/pgx/v5 v5.7.5 h1:JHGfMnQY+IEtGM63d+NGMjoRpysB2JBwDr5fsngwmJs=
|
||||
github.com/jackc/pgx/v5 v5.7.5/go.mod h1:aruU7o91Tc2q2cFp5h4uP3f6ztExVpyVv88Xl/8Vl8M=
|
||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
|
||||
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
|
||||
+13
-1
@@ -36,10 +36,16 @@ func usage() {
|
||||
rollcall unassign ID
|
||||
rollcall overlay ID [-stage system|runtime] [-extract DIR]
|
||||
rollcall render PROFILE PATH [-stage system|runtime]
|
||||
rollcall os list | sync [NAME...] [-force] | import NAME FILE [-family F] [-version V] | promote NAME VERSION | rm NAME VERSION
|
||||
rollcall deploy ID OS [-version V] [-disk /dev/sdX] [-hostname H] [-upgrade] [-user-data FILE] [-boot] [-watch]
|
||||
rollcall release ID [-boot] [-watch]
|
||||
rollcall jobs ID [-watch] | job cancel ID
|
||||
env: ROLLCALL_URL (default from config listen), ROLLCALL_TOKEN, ROLLCALL_CONFIG`)
|
||||
os.Exit(2)
|
||||
}
|
||||
|
||||
func newFlags(name string) *flag.FlagSet { return flag.NewFlagSet(name, flag.ExitOnError) }
|
||||
|
||||
func main() {
|
||||
log.SetFlags(log.Ldate | log.Ltime | log.Lmicroseconds)
|
||||
if len(os.Args) < 2 {
|
||||
@@ -57,11 +63,17 @@ func main() {
|
||||
case "keygen":
|
||||
fmt.Println(randomToken(24))
|
||||
case "list", "show", "events", "power", "bootdev", "recommission", "delete",
|
||||
"images", "profile", "assign", "unassign", "overlay", "render":
|
||||
"images", "profile", "assign", "unassign", "overlay", "render",
|
||||
"deploy", "release", "jobs", "job":
|
||||
if err := runClient(cmd, args); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "error:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
case "os":
|
||||
if err := osCmd(args); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "error:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
default:
|
||||
usage()
|
||||
}
|
||||
|
||||
@@ -0,0 +1,694 @@
|
||||
package main
|
||||
|
||||
// OS image catalog for disk-based deployment.
|
||||
//
|
||||
// <os_dir>/<name>/<version>/disk.raw.zst zstd-compressed raw disk image
|
||||
// <os_dir>/<name>/<version>/meta.json OSMeta
|
||||
// <os_dir>/<name>/current -> <version> symlink flipped atomically by sync/promote
|
||||
//
|
||||
// Sources (upstream "latest" URLs + vendor checksum files) live in
|
||||
// os_sources (default /etc/rollcall/os-sources.json). `rollcall os sync`
|
||||
// downloads only when the vendor checksum changed, verifies it, converts
|
||||
// to raw with qemu-img, compresses with zstd and prunes old versions.
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"crypto/sha512"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"runtime"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/klauspost/compress/zstd"
|
||||
)
|
||||
|
||||
type OSSource struct {
|
||||
Name string `json:"name"`
|
||||
URL string `json:"url"`
|
||||
ChecksumURL string `json:"checksum_url"`
|
||||
Checksum string `json:"checksum"` // sha256 | sha512 (type of the vendor checksum file)
|
||||
Family string `json:"family"` // debian | rhel (informational)
|
||||
Description string `json:"description"`
|
||||
Track string `json:"track"` // latest (default): new versions become current; pinned: only on promote
|
||||
Keep int `json:"keep"` // versions to keep (default 2)
|
||||
}
|
||||
|
||||
type OSMeta struct {
|
||||
Name string `json:"name"`
|
||||
Version string `json:"version"`
|
||||
Family string `json:"family"`
|
||||
Description string `json:"description"`
|
||||
SourceURL string `json:"source_url"`
|
||||
SourceSHA256 string `json:"source_sha256"`
|
||||
SourceSHA512 string `json:"source_sha512"`
|
||||
SourceModified string `json:"source_modified"`
|
||||
SourceFormat string `json:"source_format"`
|
||||
SHA256 string `json:"sha256"` // of disk.raw.zst
|
||||
Size int64 `json:"size"` // compressed bytes
|
||||
RawSize int64 `json:"raw_size"` // virtual disk size the image needs
|
||||
Imported time.Time `json:"imported"`
|
||||
Current bool `json:"current,omitempty"`
|
||||
}
|
||||
|
||||
type OSImage struct {
|
||||
Name string `json:"name"`
|
||||
Current *OSMeta `json:"current"`
|
||||
Versions []OSMeta `json:"versions"`
|
||||
Source *OSSource `json:"source"`
|
||||
Warning string `json:"warning,omitempty"`
|
||||
}
|
||||
|
||||
func loadSources(p string) ([]OSSource, error) {
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
var src []OSSource
|
||||
if err := json.Unmarshal(b, &src); err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", p, err)
|
||||
}
|
||||
for i := range src {
|
||||
if src[i].Track == "" {
|
||||
src[i].Track = "latest"
|
||||
}
|
||||
if src[i].Keep <= 0 {
|
||||
src[i].Keep = 2
|
||||
}
|
||||
if src[i].Checksum == "" {
|
||||
src[i].Checksum = "sha256"
|
||||
}
|
||||
if strings.ContainsAny(src[i].Name, "/ \t") || src[i].Name == "" {
|
||||
return nil, fmt.Errorf("%s: bad source name %q", p, src[i].Name)
|
||||
}
|
||||
}
|
||||
return src, nil
|
||||
}
|
||||
|
||||
var versionRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*$`)
|
||||
|
||||
func osVersions(dir, name string) ([]OSMeta, string) {
|
||||
ents, err := os.ReadDir(filepath.Join(dir, name))
|
||||
if err != nil {
|
||||
return nil, ""
|
||||
}
|
||||
cur, _ := os.Readlink(filepath.Join(dir, name, "current"))
|
||||
cur = path.Base(cur)
|
||||
var out []OSMeta
|
||||
for _, e := range ents {
|
||||
if !e.IsDir() || !versionRe.MatchString(e.Name()) || strings.HasSuffix(e.Name(), ".tmp") {
|
||||
continue
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, name, e.Name(), "meta.json"))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
var m OSMeta
|
||||
if json.Unmarshal(b, &m) != nil {
|
||||
continue
|
||||
}
|
||||
m.Name, m.Version = name, e.Name()
|
||||
m.Current = e.Name() == cur
|
||||
out = append(out, m)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Imported.After(out[j].Imported) })
|
||||
return out, cur
|
||||
}
|
||||
|
||||
func osCatalog(dir, sourcesFile string) ([]OSImage, error) {
|
||||
src, err := loadSources(sourcesFile)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
byName := map[string]*OSSource{}
|
||||
for i := range src {
|
||||
byName[src[i].Name] = &src[i]
|
||||
}
|
||||
names := map[string]bool{}
|
||||
for n := range byName {
|
||||
names[n] = true
|
||||
}
|
||||
if ents, err := os.ReadDir(dir); err == nil {
|
||||
for _, e := range ents {
|
||||
if e.IsDir() {
|
||||
names[e.Name()] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
out := []OSImage{}
|
||||
for n := range names {
|
||||
img := OSImage{Name: n, Source: byName[n]}
|
||||
vs, cur := osVersions(dir, n)
|
||||
img.Versions = vs
|
||||
for i := range vs {
|
||||
if vs[i].Version == cur {
|
||||
img.Current = &vs[i]
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case img.Current == nil && len(vs) == 0:
|
||||
img.Warning = "not imported"
|
||||
case img.Current == nil:
|
||||
img.Warning = "no current version (promote one)"
|
||||
case time.Since(img.Current.Imported) > 45*24*time.Hour:
|
||||
img.Warning = fmt.Sprintf("current version is %d days old", int(time.Since(img.Current.Imported).Hours()/24))
|
||||
}
|
||||
out = append(out, img)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// osResolve returns the metadata of name/version ("" or "current" = current).
|
||||
func osResolve(dir, name, version string) (*OSMeta, error) {
|
||||
if strings.ContainsAny(name, "/ \t") || name == "" {
|
||||
return nil, fmt.Errorf("bad os name %q", name)
|
||||
}
|
||||
vs, cur := osVersions(dir, name)
|
||||
if version == "" || version == "current" {
|
||||
version = cur
|
||||
if version == "" {
|
||||
return nil, fmt.Errorf("os %s has no current version", name)
|
||||
}
|
||||
}
|
||||
for i := range vs {
|
||||
if vs[i].Version == version {
|
||||
if _, err := os.Stat(filepath.Join(dir, name, version, "disk.raw.zst")); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &vs[i], nil
|
||||
}
|
||||
}
|
||||
return nil, fmt.Errorf("os %s version %s: %w", name, version, errNotFound)
|
||||
}
|
||||
|
||||
func osSetCurrent(dir, name, version string) error {
|
||||
if _, err := os.Stat(filepath.Join(dir, name, version, "disk.raw.zst")); err != nil {
|
||||
return err
|
||||
}
|
||||
link := filepath.Join(dir, name, "current")
|
||||
tmp := link + ".tmp"
|
||||
_ = os.Remove(tmp)
|
||||
if err := os.Symlink(version, tmp); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, link)
|
||||
}
|
||||
|
||||
func osRemove(dir, name, version string) error {
|
||||
if name == "" || version == "" || !versionRe.MatchString(version) {
|
||||
return fmt.Errorf("name and version required")
|
||||
}
|
||||
_, cur := osVersions(dir, name)
|
||||
if cur == version {
|
||||
return fmt.Errorf("%s/%s is the current version; promote another one first", name, version)
|
||||
}
|
||||
return os.RemoveAll(filepath.Join(dir, name, version))
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- import pipeline
|
||||
|
||||
type importOpts struct {
|
||||
Dir string
|
||||
Name string
|
||||
Version string // "" = derive
|
||||
Family string
|
||||
Description string
|
||||
SourceURL string
|
||||
Modified string
|
||||
SHA256 string // of the source file, if already known
|
||||
SHA512 string
|
||||
Log io.Writer
|
||||
}
|
||||
|
||||
func logf(w io.Writer, f string, a ...any) {
|
||||
if w != nil {
|
||||
fmt.Fprintf(w, f+"\n", a...)
|
||||
}
|
||||
}
|
||||
|
||||
// osImportFile converts src (qcow2/raw/vmdk...) into <dir>/<name>/<version>/ and returns the meta.
|
||||
func osImportFile(ctx context.Context, src string, o importOpts) (*OSMeta, error) {
|
||||
if !versionRe.MatchString(o.Name) {
|
||||
return nil, fmt.Errorf("bad name %q", o.Name)
|
||||
}
|
||||
info, err := qemuInfo(ctx, src)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if o.SHA256 == "" || o.SHA512 == "" {
|
||||
logf(o.Log, "hashing %s", src)
|
||||
if o.SHA256, o.SHA512, err = hashFile(src); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
if o.Version == "" {
|
||||
d := o.Modified
|
||||
if d == "" {
|
||||
st, _ := os.Stat(src)
|
||||
d = st.ModTime().UTC().Format("20060102")
|
||||
}
|
||||
o.Version = d + "-" + o.SHA256[:6]
|
||||
}
|
||||
if !versionRe.MatchString(o.Version) {
|
||||
return nil, fmt.Errorf("bad version %q", o.Version)
|
||||
}
|
||||
final := filepath.Join(o.Dir, o.Name, o.Version)
|
||||
if _, err := os.Stat(filepath.Join(final, "meta.json")); err == nil {
|
||||
return nil, fmt.Errorf("%s/%s already exists", o.Name, o.Version)
|
||||
}
|
||||
work := final + ".tmp"
|
||||
_ = os.RemoveAll(work)
|
||||
if err := os.MkdirAll(work, 0o755); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer os.RemoveAll(work)
|
||||
|
||||
raw := src
|
||||
if info.Format != "raw" {
|
||||
raw = filepath.Join(work, "disk.raw")
|
||||
logf(o.Log, "converting %s (%s, %s virtual) to raw", filepath.Base(src), info.Format, human(float64(info.VirtualSize)))
|
||||
cmd := exec.CommandContext(ctx, "qemu-img", "convert", "-O", "raw", "-S", "4k", src, raw)
|
||||
if out, err := cmd.CombinedOutput(); err != nil {
|
||||
return nil, fmt.Errorf("qemu-img convert: %v: %s", err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
}
|
||||
logf(o.Log, "compressing to disk.raw.zst")
|
||||
sum, size, err := compressZstd(raw, filepath.Join(work, "disk.raw.zst"))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if raw != src {
|
||||
_ = os.Remove(raw)
|
||||
}
|
||||
m := &OSMeta{Name: o.Name, Version: o.Version, Family: o.Family, Description: o.Description, SourceURL: o.SourceURL,
|
||||
SourceSHA256: o.SHA256, SourceSHA512: o.SHA512, SourceModified: o.Modified, SourceFormat: info.Format,
|
||||
SHA256: sum, Size: size, RawSize: info.VirtualSize, Imported: time.Now().UTC()}
|
||||
mb, _ := json.MarshalIndent(m, "", " ")
|
||||
if err := os.WriteFile(filepath.Join(work, "meta.json"), mb, 0o644); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := os.Rename(work, final); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
logf(o.Log, "imported %s/%s (%s compressed, %s raw)", o.Name, o.Version, human(float64(size)), human(float64(info.VirtualSize)))
|
||||
return m, nil
|
||||
}
|
||||
|
||||
type qemuImgInfo struct {
|
||||
Format string `json:"format"`
|
||||
VirtualSize int64 `json:"virtual-size"`
|
||||
}
|
||||
|
||||
func qemuInfo(ctx context.Context, p string) (*qemuImgInfo, error) {
|
||||
out, err := exec.CommandContext(ctx, "qemu-img", "info", "--output=json", p).Output()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("qemu-img info %s: %w", p, err)
|
||||
}
|
||||
var i qemuImgInfo
|
||||
if err := json.Unmarshal(out, &i); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &i, nil
|
||||
}
|
||||
|
||||
func hashFile(p string) (s256, s512 string, err error) {
|
||||
f, err := os.Open(p)
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
defer f.Close()
|
||||
h1, h2 := sha256.New(), sha512.New()
|
||||
if _, err := io.Copy(io.MultiWriter(h1, h2), bufio.NewReaderSize(f, 4<<20)); err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
return hex.EncodeToString(h1.Sum(nil)), hex.EncodeToString(h2.Sum(nil)), nil
|
||||
}
|
||||
|
||||
func compressZstd(src, dst string) (sum string, size int64, err error) {
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
defer in.Close()
|
||||
out, err := os.Create(dst)
|
||||
if err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
defer out.Close()
|
||||
h := sha256.New()
|
||||
cw := &countWriter{w: io.MultiWriter(out, h)}
|
||||
enc, err := zstd.NewWriter(cw, zstd.WithEncoderLevel(zstd.SpeedDefault), zstd.WithEncoderConcurrency(runtime.NumCPU()))
|
||||
if err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
if _, err := io.Copy(enc, bufio.NewReaderSize(in, 4<<20)); err != nil {
|
||||
enc.Close()
|
||||
return "", 0, err
|
||||
}
|
||||
if err := enc.Close(); err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
if err := out.Sync(); err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
return hex.EncodeToString(h.Sum(nil)), cw.n, nil
|
||||
}
|
||||
|
||||
type countWriter struct {
|
||||
w io.Writer
|
||||
n int64
|
||||
}
|
||||
|
||||
func (c *countWriter) Write(p []byte) (int, error) {
|
||||
n, err := c.w.Write(p)
|
||||
c.n += int64(n)
|
||||
return n, err
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- sync
|
||||
|
||||
// parseChecksums understands "HASH file", "HASH *file" and "ALGO (file) = HASH".
|
||||
func parseChecksums(txt string) map[string]string {
|
||||
out := map[string]string{}
|
||||
bsd := regexp.MustCompile(`^\s*[A-Za-z0-9]+\s*\((.+)\)\s*=\s*([0-9a-fA-F]+)\s*$`)
|
||||
gnu := regexp.MustCompile(`^\s*([0-9a-fA-F]{32,128})\s+\*?(.+?)\s*$`)
|
||||
for _, line := range strings.Split(txt, "\n") {
|
||||
if m := bsd.FindStringSubmatch(line); m != nil {
|
||||
out[path.Base(m[1])] = strings.ToLower(m[2])
|
||||
} else if m := gnu.FindStringSubmatch(line); m != nil {
|
||||
out[path.Base(m[2])] = strings.ToLower(m[1])
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func httpGetText(ctx context.Context, url string) (string, error) {
|
||||
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != 200 {
|
||||
return "", fmt.Errorf("%s: %s", url, resp.Status)
|
||||
}
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||
return string(b), err
|
||||
}
|
||||
|
||||
// upstreamChecksum fetches the vendor checksum for the source's file.
|
||||
func upstreamChecksum(ctx context.Context, s OSSource) (string, error) {
|
||||
txt, err := httpGetText(ctx, s.ChecksumURL)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
sums := parseChecksums(txt)
|
||||
if v, ok := sums[path.Base(s.URL)]; ok {
|
||||
return v, nil
|
||||
}
|
||||
if len(sums) == 1 {
|
||||
for _, v := range sums {
|
||||
return v, nil
|
||||
}
|
||||
}
|
||||
return "", fmt.Errorf("%s: no entry for %s", s.ChecksumURL, path.Base(s.URL))
|
||||
}
|
||||
|
||||
func metaChecksum(m *OSMeta, typ string) string {
|
||||
if typ == "sha512" {
|
||||
return m.SourceSHA512
|
||||
}
|
||||
return m.SourceSHA256
|
||||
}
|
||||
|
||||
// download streams url to dst while hashing; verifies against want (hex, type typ).
|
||||
func download(ctx context.Context, url, dst, typ, want string, log io.Writer) (s256, s512, modified string, err error) {
|
||||
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
return "", "", "", err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != 200 {
|
||||
return "", "", "", fmt.Errorf("%s: %s", url, resp.Status)
|
||||
}
|
||||
if t, err := http.ParseTime(resp.Header.Get("Last-Modified")); err == nil {
|
||||
modified = t.UTC().Format("20060102")
|
||||
}
|
||||
f, err := os.Create(dst)
|
||||
if err != nil {
|
||||
return "", "", "", err
|
||||
}
|
||||
defer f.Close()
|
||||
h1, h2 := sha256.New(), sha512.New()
|
||||
pr := &progressReader{r: resp.Body, total: resp.ContentLength, log: log, last: time.Now()}
|
||||
if _, err := io.Copy(io.MultiWriter(f, h1, h2), pr); err != nil {
|
||||
return "", "", "", err
|
||||
}
|
||||
s256, s512 = hex.EncodeToString(h1.Sum(nil)), hex.EncodeToString(h2.Sum(nil))
|
||||
got := s256
|
||||
if typ == "sha512" {
|
||||
got = s512
|
||||
}
|
||||
if want != "" && got != want {
|
||||
return "", "", "", fmt.Errorf("%s: %s mismatch: vendor %s, downloaded %s", url, typ, want, got)
|
||||
}
|
||||
return s256, s512, modified, nil
|
||||
}
|
||||
|
||||
type progressReader struct {
|
||||
r io.Reader
|
||||
n int64
|
||||
total int64
|
||||
log io.Writer
|
||||
last time.Time
|
||||
}
|
||||
|
||||
func (p *progressReader) Read(b []byte) (int, error) {
|
||||
n, err := p.r.Read(b)
|
||||
p.n += int64(n)
|
||||
if p.log != nil && time.Since(p.last) > 10*time.Second {
|
||||
p.last = time.Now()
|
||||
if p.total > 0 {
|
||||
logf(p.log, " %s / %s (%d%%)", human(float64(p.n)), human(float64(p.total)), p.n*100/p.total)
|
||||
} else {
|
||||
logf(p.log, " %s", human(float64(p.n)))
|
||||
}
|
||||
}
|
||||
return n, err
|
||||
}
|
||||
|
||||
// osSync refreshes the named sources (all when names is empty).
|
||||
func osSync(ctx context.Context, cfg *Config, names []string, force bool, log io.Writer) error {
|
||||
src, err := loadSources(cfg.OSSources)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(src) == 0 {
|
||||
return fmt.Errorf("no sources in %s", cfg.OSSources)
|
||||
}
|
||||
want := map[string]bool{}
|
||||
for _, n := range names {
|
||||
want[n] = true
|
||||
}
|
||||
var failed []string
|
||||
for _, s := range src {
|
||||
if len(want) > 0 && !want[s.Name] {
|
||||
continue
|
||||
}
|
||||
if err := syncOne(ctx, cfg, s, force, log); err != nil {
|
||||
logf(log, "%s: FAILED: %v", s.Name, err)
|
||||
failed = append(failed, s.Name)
|
||||
}
|
||||
}
|
||||
if len(failed) > 0 {
|
||||
return fmt.Errorf("failed: %s", strings.Join(failed, ", "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func syncOne(ctx context.Context, cfg *Config, s OSSource, force bool, log io.Writer) error {
|
||||
vs, cur := osVersions(cfg.OSDir, s.Name)
|
||||
up, err := upstreamChecksum(ctx, s)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for i := range vs {
|
||||
if metaChecksum(&vs[i], s.Checksum) == up {
|
||||
if !force {
|
||||
logf(log, "%s: up to date (%s, upstream %s unchanged)", s.Name, vs[i].Version, s.Checksum)
|
||||
if cur == "" && s.Track == "latest" {
|
||||
return osSetCurrent(cfg.OSDir, s.Name, vs[i].Version)
|
||||
}
|
||||
return osPrune(cfg.OSDir, s, log)
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
tmpDir := filepath.Join(cfg.OSDir, ".tmp")
|
||||
if err := os.MkdirAll(tmpDir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
dl := filepath.Join(tmpDir, s.Name+"-"+path.Base(s.URL))
|
||||
defer os.Remove(dl)
|
||||
logf(log, "%s: downloading %s", s.Name, s.URL)
|
||||
s256, s512, mod, err := download(ctx, s.URL, dl, s.Checksum, up, log)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
logf(log, "%s: vendor %s verified", s.Name, s.Checksum)
|
||||
m, err := osImportFile(ctx, dl, importOpts{Dir: cfg.OSDir, Name: s.Name, Family: s.Family, Description: s.Description,
|
||||
SourceURL: s.URL, Modified: mod, SHA256: s256, SHA512: s512, Log: log})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if s.Track == "latest" || cur == "" {
|
||||
if err := osSetCurrent(cfg.OSDir, s.Name, m.Version); err != nil {
|
||||
return err
|
||||
}
|
||||
logf(log, "%s: current -> %s", s.Name, m.Version)
|
||||
} else {
|
||||
logf(log, "%s: %s imported but pinned to %s (rollcall os promote %s %s)", s.Name, m.Version, cur, s.Name, m.Version)
|
||||
}
|
||||
return osPrune(cfg.OSDir, s, log)
|
||||
}
|
||||
|
||||
// osPrune keeps the newest `keep` versions plus current; nothing younger than a day is removed
|
||||
// so a deploy job that pinned a version keeps its file.
|
||||
func osPrune(dir string, s OSSource, log io.Writer) error {
|
||||
vs, cur := osVersions(dir, s.Name)
|
||||
kept := 0
|
||||
for _, v := range vs {
|
||||
if v.Version == cur || kept < s.Keep {
|
||||
kept++
|
||||
continue
|
||||
}
|
||||
if time.Since(v.Imported) < 24*time.Hour {
|
||||
continue
|
||||
}
|
||||
logf(log, "%s: pruning %s", s.Name, v.Version)
|
||||
if err := os.RemoveAll(filepath.Join(dir, s.Name, v.Version)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- CLI (local, filesystem side)
|
||||
|
||||
func osCmd(args []string) error {
|
||||
if len(args) < 1 {
|
||||
usage()
|
||||
}
|
||||
cfg, err := loadConfig(envOr("ROLLCALL_CONFIG", "/etc/rollcall/config.json"))
|
||||
if err != nil && args[0] != "list" {
|
||||
return fmt.Errorf("os %s needs the server config: %w", args[0], err)
|
||||
}
|
||||
ctx := context.Background()
|
||||
switch args[0] {
|
||||
case "list":
|
||||
if c, cerr := newClient(); cerr == nil {
|
||||
var imgs []OSImage
|
||||
if err := c.do("GET", "/api/v1/os", nil, &imgs); err != nil {
|
||||
return err
|
||||
}
|
||||
printOSList(imgs)
|
||||
return nil
|
||||
}
|
||||
imgs, err := osCatalog(cfg.OSDir, cfg.OSSources)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
printOSList(imgs)
|
||||
case "sync":
|
||||
fs := newFlags("os sync")
|
||||
force := fs.Bool("force", false, "re-import even when the upstream checksum is unchanged")
|
||||
_ = fs.Parse(args[1:])
|
||||
return osSync(ctx, cfg, fs.Args(), *force, os.Stdout)
|
||||
case "import":
|
||||
if len(args) < 3 {
|
||||
usage()
|
||||
}
|
||||
fs := newFlags("os import")
|
||||
fam := fs.String("family", "", "debian | rhel")
|
||||
desc := fs.String("description", "", "description")
|
||||
ver := fs.String("version", "", "version label (default <date>-<sha256 prefix>)")
|
||||
cur := fs.Bool("current", true, "make it the current version")
|
||||
_ = fs.Parse(args[3:])
|
||||
m, err := osImportFile(ctx, args[2], importOpts{Dir: cfg.OSDir, Name: args[1], Version: *ver, Family: *fam, Description: *desc,
|
||||
SourceURL: "file://" + args[2], Log: os.Stdout})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if *cur {
|
||||
return osSetCurrent(cfg.OSDir, m.Name, m.Version)
|
||||
}
|
||||
case "promote":
|
||||
if len(args) < 3 {
|
||||
usage()
|
||||
}
|
||||
return osSetCurrent(cfg.OSDir, args[1], args[2])
|
||||
case "rm":
|
||||
if len(args) < 3 {
|
||||
usage()
|
||||
}
|
||||
return osRemove(cfg.OSDir, args[1], args[2])
|
||||
default:
|
||||
usage()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func printOSList(imgs []OSImage) {
|
||||
fmt.Printf("%-18s %-20s %-7s %8s %8s %-6s %-19s %s\n", "OS", "CURRENT", "FAMILY", "ZST", "RAW", "TRACK", "IMPORTED", "VERSIONS / WARNING")
|
||||
for _, i := range imgs {
|
||||
cur, fam, zst, raw, imp, track := "-", "", "", "", "", ""
|
||||
if i.Source != nil {
|
||||
track, fam = i.Source.Track, i.Source.Family
|
||||
}
|
||||
if i.Current != nil {
|
||||
cur, zst, raw = i.Current.Version, human(float64(i.Current.Size)), human(float64(i.Current.RawSize))
|
||||
imp = i.Current.Imported.Local().Format("2006-01-02 15:04")
|
||||
if fam == "" {
|
||||
fam = i.Current.Family
|
||||
}
|
||||
}
|
||||
var vs []string
|
||||
for _, v := range i.Versions {
|
||||
vs = append(vs, v.Version)
|
||||
}
|
||||
extra := strconv.Itoa(len(i.Versions)) + " (" + strings.Join(vs, ", ") + ")"
|
||||
if i.Warning != "" {
|
||||
extra += " ! " + i.Warning
|
||||
}
|
||||
fmt.Printf("%-18s %-20s %-7s %8s %8s %-6s %-19s %s\n", i.Name, cur, fam, zst, raw, track, imp, extra)
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- HTTP
|
||||
|
||||
func (a *App) handleOS(w http.ResponseWriter, r *http.Request) {
|
||||
imgs, err := osCatalog(a.cfg.OSDir, a.cfg.OSSources)
|
||||
if err != nil {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, 200, imgs)
|
||||
}
|
||||
+28
-5
@@ -81,7 +81,7 @@ func logging(next http.Handler) http.Handler {
|
||||
start := time.Now()
|
||||
rw := &statusWriter{ResponseWriter: w, code: 200}
|
||||
next.ServeHTTP(rw, r)
|
||||
if (!strings.HasPrefix(r.URL.Path, "/rollcall/") && !strings.HasPrefix(r.URL.Path, "/images/")) || rw.code >= 400 {
|
||||
if (!strings.HasPrefix(r.URL.Path, "/rollcall/") && !strings.HasPrefix(r.URL.Path, "/images/") && !strings.HasPrefix(r.URL.Path, "/os/")) || rw.code >= 400 {
|
||||
log.Printf("%s %s %s %d %s", r.RemoteAddr, r.Method, r.URL.RequestURI(), rw.code, time.Since(start).Round(time.Millisecond))
|
||||
}
|
||||
})
|
||||
@@ -120,10 +120,20 @@ func (a *App) routes() http.Handler {
|
||||
mux.HandleFunc("GET /api/v1/machines/{id}/overlay", a.adminAuth(a.handleAdminOverlay))
|
||||
mux.HandleFunc("GET /api/v1/render/{profile}/{path...}", a.adminAuth(a.handleRender))
|
||||
mux.HandleFunc("GET /api/v1/nodes/self/overlay", a.handleNodeOverlay) // node token auth inside
|
||||
// disk-based deployment
|
||||
mux.HandleFunc("GET /api/v1/os", a.adminAuth(a.handleOS))
|
||||
mux.HandleFunc("POST /api/v1/machines/{id}/deploy", a.adminAuth(a.handleDeploy))
|
||||
mux.HandleFunc("POST /api/v1/machines/{id}/release", a.adminAuth(a.handleRelease))
|
||||
mux.HandleFunc("GET /api/v1/machines/{id}/jobs", a.adminAuth(a.handleJobs))
|
||||
mux.HandleFunc("POST /api/v1/machines/{id}/jobs/cancel", a.adminAuth(a.handleJobCancel))
|
||||
mux.HandleFunc("GET /api/v1/jobs/self", a.handleJobSelf) // job token auth inside
|
||||
mux.HandleFunc("POST /api/v1/jobs/self/progress", a.handleJobProgress) // job token auth inside
|
||||
mux.HandleFunc("POST /api/v1/phone-home/{token}", a.handlePhoneHome)
|
||||
// image files for the live boot (no auth: the kernel/initrd/squashfs must be fetchable)
|
||||
imgDir := a.cfg.ImageDir
|
||||
mux.Handle("GET /rollcall/", http.StripPrefix("/rollcall/", http.FileServer(http.Dir(imgDir))))
|
||||
mux.Handle("GET /images/", http.StripPrefix("/images/", http.FileServer(http.Dir(a.cfg.ImagesDir))))
|
||||
mux.Handle("GET /os/", http.StripPrefix("/os/", http.FileServer(http.Dir(a.cfg.OSDir))))
|
||||
// embedded web UI
|
||||
sub, _ := fs.Sub(webFS, "web")
|
||||
mux.Handle("GET /", http.FileServer(http.FS(sub)))
|
||||
@@ -225,6 +235,10 @@ type Summary struct {
|
||||
MgmtIP *string `json:"mgmt_ip"`
|
||||
IBIP *string `json:"ib_ip"`
|
||||
BootedAt *time.Time `json:"booted_at"`
|
||||
OS *string `json:"os"`
|
||||
OSVersion *string `json:"os_version"`
|
||||
DeployedAt *time.Time `json:"deployed_at"`
|
||||
Job *Job `json:"job,omitempty"` // active deployment job, if any
|
||||
}
|
||||
|
||||
func num(v any) float64 {
|
||||
@@ -241,7 +255,8 @@ func summarize(m *Machine) Summary {
|
||||
MemoryBytes: num(dig(inv, "memory", "total_bytes")), Disks: num(dig(inv, "storage", "disk_count")),
|
||||
StorageBytes: num(dig(inv, "storage", "total_bytes")), NICs: listLen(inv["network"]),
|
||||
IBHCAs: listLen(dig(inv, "infiniband", "hcas")), GPUs: listLen(inv["gpus"]),
|
||||
Profile: m.Profile, NodeName: m.NodeName, MgmtIP: m.MgmtIP, IBIP: m.IBIP, BootedAt: m.BootedAt}
|
||||
Profile: m.Profile, NodeName: m.NodeName, MgmtIP: m.MgmtIP, IBIP: m.IBIP, BootedAt: m.BootedAt,
|
||||
OS: m.OS, OSVersion: m.OSVersion, DeployedAt: m.DeployedAt}
|
||||
if m.BMC != nil {
|
||||
var b obj
|
||||
if json.Unmarshal(m.BMC, &b) == nil {
|
||||
@@ -415,9 +430,12 @@ func (a *App) handleList(w http.ResponseWriter, r *http.Request) {
|
||||
fail(w, 500, err)
|
||||
return
|
||||
}
|
||||
active, _ := a.store.ActiveJobs(r.Context())
|
||||
out := make([]Summary, 0, len(ms))
|
||||
for _, m := range ms {
|
||||
out = append(out, summarize(m))
|
||||
s := summarize(m)
|
||||
s.Job = active[m.ID]
|
||||
out = append(out, s)
|
||||
}
|
||||
writeJSON(w, 200, out)
|
||||
}
|
||||
@@ -429,11 +447,16 @@ func (a *App) handleGet(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
evs, _ := a.store.Events(r.Context(), m.ID)
|
||||
jobs, _ := a.store.Jobs(r.Context(), m.ID, 10)
|
||||
s := summarize(m)
|
||||
if j, err := a.store.ActiveJob(r.Context(), m.ID); err == nil {
|
||||
s.Job = j
|
||||
}
|
||||
var bmc, inv any
|
||||
_ = json.Unmarshal(m.BMC, &bmc)
|
||||
_ = json.Unmarshal(m.Inventory, &inv)
|
||||
writeJSON(w, 200, obj{"summary": summarize(m), "bmc": bmc, "has_bmc_secret": m.bmcSecret != nil,
|
||||
"inventory": inv, "events": evs})
|
||||
writeJSON(w, 200, obj{"summary": s, "bmc": bmc, "has_bmc_secret": m.bmcSecret != nil,
|
||||
"inventory": inv, "events": evs, "jobs": jobs})
|
||||
}
|
||||
|
||||
func (a *App) handleDelete(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
@@ -185,7 +185,7 @@ func parsePool(spec string) ([]string, error) {
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (a *App) allocate(ctx context.Context, p *Profile, wantName string) (name, mgmt, ib string, err error) {
|
||||
func (a *App) allocate(ctx context.Context, nameFormat, wantName string) (name, mgmt, ib string, err error) {
|
||||
ms, err := a.store.List(ctx)
|
||||
if err != nil {
|
||||
return
|
||||
@@ -204,7 +204,7 @@ func (a *App) allocate(ctx context.Context, p *Profile, wantName string) (name,
|
||||
}
|
||||
name = wantName
|
||||
if name == "" {
|
||||
f := p.NameFormat
|
||||
f := nameFormat
|
||||
if f == "" {
|
||||
f = "node%03d"
|
||||
}
|
||||
@@ -601,6 +601,9 @@ func (a *App) writePXEAny(ctx context.Context, m *Machine) error {
|
||||
}
|
||||
return a.writePXEStateless(m, p, token)
|
||||
}
|
||||
if m.State == "deployed" {
|
||||
return a.writePXEDeployed(m)
|
||||
}
|
||||
return a.writePXE(m)
|
||||
}
|
||||
|
||||
@@ -722,7 +725,7 @@ func (a *App) handleAssign(w http.ResponseWriter, r *http.Request) {
|
||||
// keep the existing name/IPs when re-assigning an already named node
|
||||
name, mgmt, ib := deref(m.NodeName), deref(m.MgmtIP), deref(m.IBIP)
|
||||
if name == "" || (req.Name != "" && req.Name != name) {
|
||||
if name, mgmt, ib, err = a.allocate(r.Context(), p, req.Name); err != nil {
|
||||
if name, mgmt, ib, err = a.allocate(r.Context(), p.NameFormat, req.Name); err != nil {
|
||||
fail(w, 409, err)
|
||||
return
|
||||
}
|
||||
|
||||
+29
-2
@@ -56,6 +56,25 @@ ALTER TABLE machines ADD COLUMN IF NOT EXISTS node_token bytea;
|
||||
ALTER TABLE machines ADD COLUMN IF NOT EXISTS host_key bytea;
|
||||
ALTER TABLE machines ADD COLUMN IF NOT EXISTS booted_at timestamptz;
|
||||
CREATE INDEX IF NOT EXISTS machines_node_token ON machines(node_token_hash);
|
||||
ALTER TABLE machines ADD COLUMN IF NOT EXISTS os text;
|
||||
ALTER TABLE machines ADD COLUMN IF NOT EXISTS os_version text;
|
||||
ALTER TABLE machines ADD COLUMN IF NOT EXISTS deployed_at timestamptz;
|
||||
CREATE TABLE IF NOT EXISTS jobs (
|
||||
id text PRIMARY KEY,
|
||||
machine_id text NOT NULL,
|
||||
kind text NOT NULL,
|
||||
state text NOT NULL DEFAULT 'pending',
|
||||
spec jsonb NOT NULL,
|
||||
progress jsonb,
|
||||
token_hash text,
|
||||
token bytea,
|
||||
created timestamptz NOT NULL DEFAULT now(),
|
||||
started timestamptz,
|
||||
finished timestamptz,
|
||||
error text
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS jobs_machine ON jobs(machine_id, created);
|
||||
CREATE INDEX IF NOT EXISTS jobs_token ON jobs(token_hash);
|
||||
`
|
||||
|
||||
var errNotFound = errors.New("not found")
|
||||
@@ -81,6 +100,9 @@ type Machine struct {
|
||||
MgmtIP *string `json:"mgmt_ip"`
|
||||
IBIP *string `json:"ib_ip"`
|
||||
BootedAt *time.Time `json:"booted_at"`
|
||||
OS *string `json:"os"`
|
||||
OSVersion *string `json:"os_version"`
|
||||
DeployedAt *time.Time `json:"deployed_at"`
|
||||
bmcSecret []byte
|
||||
nodeToken []byte
|
||||
hostKey []byte
|
||||
@@ -104,14 +126,16 @@ func openStore(ctx context.Context, url string) (*Store, error) {
|
||||
}
|
||||
|
||||
const machineCols = `id, uuid, serial, hostname, manufacturer, product, state, first_seen, last_seen,
|
||||
commissioned_at, macs, bmc, bmc_secret, inventory, profile, node_name, mgmt_ip, ib_ip, node_token, host_key, booted_at`
|
||||
commissioned_at, macs, bmc, bmc_secret, inventory, profile, node_name, mgmt_ip, ib_ip, node_token, host_key, booted_at,
|
||||
os, os_version, deployed_at`
|
||||
|
||||
func scanMachine(row pgx.Row, full bool) (*Machine, error) {
|
||||
m := &Machine{}
|
||||
var macs []byte
|
||||
err := row.Scan(&m.ID, &m.UUID, &m.Serial, &m.Hostname, &m.Manufacturer, &m.Product, &m.State,
|
||||
&m.FirstSeen, &m.LastSeen, &m.CommissionedAt, &macs, &m.BMC, &m.bmcSecret, &m.Inventory,
|
||||
&m.Profile, &m.NodeName, &m.MgmtIP, &m.IBIP, &m.nodeToken, &m.hostKey, &m.BootedAt)
|
||||
&m.Profile, &m.NodeName, &m.MgmtIP, &m.IBIP, &m.nodeToken, &m.hostKey, &m.BootedAt,
|
||||
&m.OS, &m.OSVersion, &m.DeployedAt)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil, errNotFound
|
||||
}
|
||||
@@ -220,6 +244,9 @@ func (s *Store) Delete(ctx context.Context, id string) error {
|
||||
if _, err := tx.Exec(ctx, `DELETE FROM events WHERE machine_id=$1`, id); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `DELETE FROM jobs WHERE machine_id=$1`, id); err != nil {
|
||||
return err
|
||||
}
|
||||
tag, err := tx.Exec(ctx, `DELETE FROM machines WHERE id=$1`, id)
|
||||
if err != nil {
|
||||
return err
|
||||
|
||||
+52
-3
@@ -36,6 +36,7 @@ $('#back').onclick = () => { current = null; $('#detail').hidden = true; $('#lis
|
||||
function setStatus(msg, bad) { const el = $('#status'); el.textContent = msg; el.className = bad ? 'bad' : 'muted'; }
|
||||
|
||||
let profiles = [];
|
||||
let osImages = [];
|
||||
async function loadList() {
|
||||
if (!token) return askToken();
|
||||
try {
|
||||
@@ -56,6 +57,12 @@ async function loadProfiles() {
|
||||
const sel = $('#assign-profile'); const cur = sel.value;
|
||||
sel.innerHTML = '<option value="">profile…</option>' + profiles.map(p => `<option value="${esc(p.name)}">${esc(p.name)} (${esc(p.image)})</option>`).join('');
|
||||
sel.value = cur;
|
||||
osImages = await api('GET', '/api/v1/os');
|
||||
$('#osimages').innerHTML = table(['OS', 'Current version', 'Family', 'Compressed', 'Disk needed', 'Imported', 'Track', 'Versions', 'Status'],
|
||||
osImages.map(i => { const c = i.current || {}; const s = i.source || {}; return [esc(i.name), `<span class="mono">${esc(c.version || '-')}</span>`, esc(s.family || c.family), c.size ? human(c.size) : '', c.raw_size ? human(c.raw_size) : '', c.imported ? esc(new Date(c.imported).toLocaleString()) : '', esc(s.track || ''), (i.versions || []).length, i.warning ? `<span class="bad">${esc(i.warning)}</span>` : '<span class="ok">ok</span>']; }));
|
||||
const os = $('#deploy-os'); const curOS = os.value;
|
||||
os.innerHTML = '<option value="">OS image…</option>' + osImages.filter(i => i.current).map(i => `<option value="${esc(i.name)}">${esc(i.name)} (${esc(i.current.version)}, ${human(i.current.raw_size)})</option>`).join('');
|
||||
os.value = curOS;
|
||||
} catch (e) { /* non-fatal */ }
|
||||
}
|
||||
$('#profile-form').onsubmit = async ev => {
|
||||
@@ -75,9 +82,12 @@ function renderList() {
|
||||
$('#machines tbody').innerHTML = rows.map(m => {
|
||||
const b = m.bmc;
|
||||
const bmc = b ? `<span class="${b.configured ? 'ok' : 'bad'}">${esc(b.address || '-')}</span> <span class="muted">${esc(b.source)}</span>` : '<span class="muted">-</span>';
|
||||
const prog = m.job && m.job.progress ? m.job.progress : null;
|
||||
const state = m.job ? `${esc(m.state)} ${prog ? Math.round(prog.pct || 0) + '%' : ''}` : esc(m.state);
|
||||
const what = m.profile ? esc(m.profile) : m.os ? `${esc(m.os)}<br><span class="muted mono">${esc(m.os_version || '')}</span>` : '';
|
||||
return `<tr data-id="${esc(m.id)}">
|
||||
<td class="mono">${esc(m.id)}</td><td><span class="pill ${esc(m.state)}">${esc(m.state)}</span></td>
|
||||
<td>${esc(m.node_name || '')}${m.mgmt_ip ? `<br><span class="muted mono">${esc(m.mgmt_ip)}</span>` : ''}</td><td>${esc(m.profile || '')}</td>
|
||||
<td class="mono">${esc(m.id)}</td><td><span class="pill ${esc(m.state)}">${state}</span></td>
|
||||
<td>${esc(m.node_name || '')}${m.mgmt_ip ? `<br><span class="muted mono">${esc(m.mgmt_ip)}</span>` : ''}</td><td>${what}</td>
|
||||
<td>${esc((m.manufacturer || '') + ' ' + (m.product || ''))}</td><td class="mono">${esc(m.serial || '')}</td>
|
||||
<td class="num">${esc(m.cpus)}</td><td class="num">${human(m.memory_bytes)}</td><td class="num">${m.disks | 0}</td>
|
||||
<td class="num">${human(m.storage_bytes)}</td><td class="num">${m.nics}</td><td class="num">${m.ib_hcas}</td><td class="num">${m.gpus}</td>
|
||||
@@ -106,7 +116,27 @@ function renderDetail(d) {
|
||||
$('#action-out').textContent = '';
|
||||
if (s.profile) $('#assign-profile').value = s.profile;
|
||||
$('#assign-name').value = s.node_name || '';
|
||||
const diskSel = $('#deploy-disk'); const curDisk = diskSel.value;
|
||||
diskSel.innerHTML = '<option value="">disk: auto (smallest that fits)</option>' + ((st.disks || []).map(x => `<option value="${esc(x.path || '/dev/' + x.name)}">${esc(x.name)} ${human(x.size)} ${esc(x.model || '')}</option>`).join(''));
|
||||
diskSel.value = curDisk;
|
||||
$('#job-cancel').hidden = !s.job;
|
||||
const cards = [];
|
||||
const job = s.job || (d.jobs || [])[0];
|
||||
if (s.os || job) {
|
||||
const p = (job && job.progress) || {}; const sp = (job && job.spec) || {};
|
||||
const active = job && (job.state === 'pending' || job.state === 'running');
|
||||
cards.push(card('Deployment', kv([
|
||||
['OS', s.os ? `${esc(s.os)} <span class="muted mono">${esc(s.os_version || '')}</span>` : ''],
|
||||
['Deployed', esc(s.deployed_at ? new Date(s.deployed_at).toLocaleString() : '')],
|
||||
['Phoned home', s.deployed_at ? (s.booted_at ? `<span class="ok">${esc(new Date(s.booted_at).toLocaleString())}</span>` : '<span class="muted">not yet</span>') : ''],
|
||||
['Hostname', esc(s.node_name)], ['Management IP', s.mgmt_ip ? `<span class="mono">${esc(s.mgmt_ip)}</span>` : ''],
|
||||
['Job', job ? `<span class="mono">${esc(job.id)}</span> ${esc(job.kind)} <span class="pill ${esc(job.state)}">${esc(job.state)}</span> ${esc(ago(job.created))}` : ''],
|
||||
['Target', sp.disk ? `<span class="mono">${esc(sp.disk)}</span> ← ${esc(sp.os || '')}/${esc(sp.version || '')}` : ''],
|
||||
['Progress', active ? `<div class="bar"><span style="width:${Math.round(p.pct || 0)}%"></span></div><span class="muted">${esc(p.phase || '')} ${Math.round(p.pct || 0)}% ${esc(p.msg || '')}</span>` : ''],
|
||||
['Result', p.details && Object.keys(p.details).length ? `<span class="mono">${esc(JSON.stringify(p.details))}</span>` : ''],
|
||||
['Error', job && job.error ? `<span class="bad">${esc(job.error)}</span>` : ''],
|
||||
]) + ((d.jobs || []).length > 1 ? '<p class="muted">History</p>' + table(['Job', 'Kind', 'State', 'Created', 'OS', 'Error'], d.jobs.map(j => [`<span class="mono">${esc(j.id)}</span>`, esc(j.kind), `<span class="pill ${esc(j.state)}">${esc(j.state)}</span>`, esc(new Date(j.created).toLocaleString()), esc(((j.spec || {}).os || '') + '/' + ((j.spec || {}).version || '')), esc(j.error || '')])) : ''), true));
|
||||
}
|
||||
if (s.profile) cards.push(card('Stateless node', kv([
|
||||
['Node name', esc(s.node_name)], ['Profile', esc(s.profile)], ['Management IP', `<span class="mono">${esc(s.mgmt_ip)}</span>`],
|
||||
['InfiniBand IP', s.ib_ip ? `<span class="mono">${esc(s.ib_ip)}</span>` : ''], ['Booted', esc(s.booted_at ? new Date(s.booted_at).toLocaleString() : 'not yet')],
|
||||
@@ -188,10 +218,29 @@ const doAssign = boot => action(async () => {
|
||||
$('#assign').onclick = () => doAssign(false);
|
||||
$('#assign-boot').onclick = () => { if (confirm('Assign and power-cycle this machine into the stateless image?')) doAssign(true); };
|
||||
$('#unassign').onclick = () => { if (confirm('Remove the profile? The node name and IP are released.')) action(async () => { const r = await api('POST', `/api/v1/machines/${current}/unassign`); await loadDetail(current); return 'state ' + r.state; }, 'unassign'); };
|
||||
$('#deploy').onclick = () => {
|
||||
const os = $('#deploy-os').value; if (!os) { $('#action-out').textContent = 'pick an OS image'; return; }
|
||||
const disk = $('#deploy-disk').value;
|
||||
if (!confirm(`Deploy ${os} to ${disk || 'the auto-selected disk'} of ${current}?\nThe disk is wiped and the machine is power-cycled.`)) return;
|
||||
action(async () => {
|
||||
const r = await api('POST', `/api/v1/machines/${current}/deploy`, { os, disk, hostname: $('#deploy-host').value.trim(), package_upgrade: $('#deploy-upgrade').checked, boot: true });
|
||||
await loadDetail(current); return `job ${r.job}: ${r.os}/${r.version} → ${r.disk} as ${r.hostname}${r.ipmi ? ' · ' + r.ipmi : ''}`;
|
||||
}, 'deploy');
|
||||
};
|
||||
$('#release').onclick = () => { if (confirm(`Release ${current}? The OS disk is wiped (signatures + first/last 64 MiB) and the machine returns to ready.`)) action(async () => { const r = await api('POST', `/api/v1/machines/${current}/release`, { boot: true }); await loadDetail(current); return `job ${r.job} (wipe ${r.disk})${r.ipmi ? ' · ' + r.ipmi : ''}`; }, 'release'); };
|
||||
$('#job-cancel').onclick = () => { if (confirm('Cancel the active job? A running write is not interrupted on the machine; it just stops being tracked.')) action(async () => { const r = await api('POST', `/api/v1/machines/${current}/jobs/cancel`); await loadDetail(current); return `job ${r.job} cancelled`; }, 'cancel job'); };
|
||||
$('#recommission').onclick = () => action(async () => { const r = await api('POST', `/api/v1/machines/${current}/recommission`); await loadDetail(current); return `state ${r.state}: next PXE boot runs commissioning`; }, 'recommission');
|
||||
$('#recommission-now').onclick = () => { if (confirm('Remove PXE override, set bootdev=pxe and power-cycle this machine via its BMC?')) action(async () => { const r = await api('POST', `/api/v1/machines/${current}/recommission?now=1`); await loadDetail(current); return r.ipmi || r.state; }, 'recommission now'); };
|
||||
$('#delete').onclick = () => { if (confirm(`Delete ${current}? It will be commissioned again as a new machine on next PXE boot.`)) action(async () => { await api('DELETE', `/api/v1/machines/${current}`); $('#back').click(); return 'deleted'; }, 'delete'); };
|
||||
|
||||
function schedule() { clearInterval(timer); timer = setInterval(() => { if (!current && document.visibilityState === 'visible') loadList(); }, 10000); }
|
||||
function schedule() {
|
||||
clearInterval(timer);
|
||||
timer = setInterval(() => {
|
||||
if (document.visibilityState !== 'visible') return;
|
||||
if (!current) return loadList();
|
||||
const m = machines.find(x => x.id === current);
|
||||
if (m && (m.job || m.state === 'deploying' || m.state === 'releasing')) { loadDetail(current); loadList(); }
|
||||
}, 4000);
|
||||
}
|
||||
schedule();
|
||||
loadList();
|
||||
@@ -33,7 +33,7 @@
|
||||
</div>
|
||||
<table id="machines">
|
||||
<thead><tr>
|
||||
<th>ID</th><th>State</th><th>Node</th><th>Profile</th><th>Model</th><th>Serial</th><th class="num">CPU</th><th class="num">RAM</th>
|
||||
<th>ID</th><th>State</th><th>Node</th><th>Profile / OS</th><th>Model</th><th>Serial</th><th class="num">CPU</th><th class="num">RAM</th>
|
||||
<th class="num">Disks</th><th class="num">Storage</th><th class="num">NIC</th><th class="num">IB</th><th class="num">GPU</th>
|
||||
<th>BMC</th><th>Last seen</th>
|
||||
</tr></thead>
|
||||
@@ -42,10 +42,11 @@
|
||||
<p id="empty" class="muted" hidden>No machines yet. PXE-boot an unknown server: it commissions itself and appears here.</p>
|
||||
|
||||
<details id="profiles-box" style="margin-top:18px">
|
||||
<summary>Profiles & images (stateless provisioning)</summary>
|
||||
<summary>Provisioning: OS images (deployment), profiles & root images (stateless)</summary>
|
||||
<div class="grid" style="margin-top:10px">
|
||||
<div class="card wide"><h3>OS images for deployment <span class="muted" style="text-transform:none">· refreshed nightly by <code>rollcall os sync</code></span></h3><div id="osimages"></div></div>
|
||||
<div class="card"><h3>Profiles</h3><div id="profiles"></div></div>
|
||||
<div class="card"><h3>Images</h3><div id="images"></div></div>
|
||||
<div class="card"><h3>Stateless root images</h3><div id="images"></div></div>
|
||||
<div class="card wide"><h3>Add / update profile</h3>
|
||||
<form id="profile-form" class="kv" style="grid-template-columns:max-content 1fr;align-items:center">
|
||||
<label>Name</label><input name="name" required placeholder="compute">
|
||||
@@ -87,6 +88,15 @@
|
||||
<button id="assign-boot" class="btn" title="assign, bootdev pxe, power-cycle">Assign + boot</button>
|
||||
<button id="unassign" class="btn warn">Unassign</button>
|
||||
<span class="sep"></span>
|
||||
<span class="muted">Deploy:</span>
|
||||
<select id="deploy-os"><option value="">OS image…</option></select>
|
||||
<select id="deploy-disk"><option value="">disk: auto (smallest that fits)</option></select>
|
||||
<input id="deploy-host" type="text" placeholder="hostname (auto)" style="width:140px">
|
||||
<label><input id="deploy-upgrade" type="checkbox"> upgrade on first boot</label>
|
||||
<button id="deploy" class="btn primary" title="write the OS image to the disk, seed cloud-init, bootdev pxe, power-cycle">Deploy + boot</button>
|
||||
<button id="release" class="btn warn" title="wipe the disk signatures and return the machine to ready">Release</button>
|
||||
<button id="job-cancel" class="btn" hidden>Cancel job</button>
|
||||
<span class="sep"></span>
|
||||
<button id="recommission" class="btn warn">Re-commission</button>
|
||||
<button id="recommission-now" class="btn warn" title="remove per-MAC PXE config, set bootdev pxe and power-cycle via BMC">Re-commission now</button>
|
||||
<button id="delete" class="btn danger">Delete</button>
|
||||
|
||||
@@ -33,7 +33,12 @@ th { font-weight: 600; color: var(--muted); font-size: 12px; text-transform: upp
|
||||
th.num, td.num { text-align: right; font-variant-numeric: tabular-nums; }
|
||||
tbody tr { cursor: pointer; } tbody tr:hover { background: var(--hover); }
|
||||
.pill { display: inline-block; padding: 2px 8px; border-radius: 999px; background: var(--pill); font-size: 12px; }
|
||||
.pill.ready, .pill.booted { color: var(--ok); } .pill.recommission, .pill.new, .pill.stateless { color: var(--warn); }
|
||||
.pill.ready, .pill.booted, .pill.deployed { color: var(--ok); } .pill.recommission, .pill.new, .pill.stateless, .pill.deploying, .pill.releasing { color: var(--warn); }
|
||||
.pill.deploy-failed, .pill.release-failed { color: var(--danger); }
|
||||
.bar { position: relative; height: 10px; background: var(--pill); border-radius: 999px; overflow: hidden; margin: 6px 0; }
|
||||
.bar > span { position: absolute; inset: 0 auto 0 0; background: var(--accent); transition: width .5s; }
|
||||
.actions select, .actions input[type=text] { padding: 5px 8px; border: 1px solid var(--line); border-radius: 6px; background: var(--panel); color: var(--fg); font: inherit; }
|
||||
.actions label { display: inline-flex; gap: 4px; align-items: center; color: var(--muted); font-size: 13px; }
|
||||
.card form input { padding: 5px 8px; border: 1px solid var(--line); border-radius: 6px; background: var(--panel); color: var(--fg); font: inherit; min-width: 0; width: 100%; }
|
||||
#assign-profile { padding: 5px 8px; border: 1px solid var(--line); border-radius: 6px; background: var(--panel); color: var(--fg); font: inherit; }
|
||||
.ok { color: var(--ok); } .bad { color: var(--danger); }
|
||||
|
||||
@@ -12,6 +12,8 @@
|
||||
"images_dir": "/srv/rollcall/images",
|
||||
"overlays_dir": "/etc/rollcall/overlays",
|
||||
"dnsmasq_dir": "/var/lib/rollcall/dnsmasq",
|
||||
"os_dir": "/srv/rollcall/os",
|
||||
"os_sources": "/etc/rollcall/os-sources.json",
|
||||
"node_pools": {"mgmt": "10.213.77.50-10.213.77.99"},
|
||||
"cluster_vars": {
|
||||
"domain": "pxe.local",
|
||||
|
||||
@@ -27,6 +27,11 @@ install -d /srv/rollcall/images
|
||||
install -d -m 0750 -o root -g rollcall /etc/rollcall/overlays
|
||||
cp -a overlays/. /etc/rollcall/overlays/
|
||||
chgrp -R rollcall /etc/rollcall/overlays
|
||||
# OS deployment images: <os_dir>/<name>/<version>/disk.raw.zst, written by `rollcall os sync` (user rollcall)
|
||||
install -d -m 0755 -o rollcall -g rollcall /srv/rollcall/os
|
||||
[ -e /etc/rollcall/os-sources.json ] || install -m 0644 os-sources.json /etc/rollcall/os-sources.json
|
||||
install -m 0644 rollcall-os-sync.service rollcall-os-sync.timer /etc/systemd/system/
|
||||
command -v qemu-img >/dev/null || echo "WARNING: qemu-img missing (apt install qemu-utils) - rollcall os sync/import will fail" >&2
|
||||
# dnsmasq host export + reload trigger
|
||||
install -d -m 0755 /var/lib/rollcall
|
||||
install -d -m 0755 -o rollcall -g rollcall /var/lib/rollcall/dnsmasq
|
||||
@@ -36,6 +41,7 @@ install -m 0644 rollcall-controller-sync.service rollcall-controller-sync.timer
|
||||
install -m 0755 rollcall-controller-sync.sh /usr/local/bin/rollcall-controller-sync.sh
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now rollcall-dnsmasq.path >/dev/null 2>&1 || true
|
||||
systemctl enable --now rollcall-os-sync.timer >/dev/null 2>&1 || true
|
||||
if systemctl list-unit-files slurmctld.service >/dev/null 2>&1 && [ -d /etc/slurm ]; then
|
||||
systemctl enable --now rollcall-controller-sync.timer >/dev/null 2>&1 || true
|
||||
fi
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
[
|
||||
{
|
||||
"name": "debian-13",
|
||||
"description": "Debian 13 (trixie) generic cloud image",
|
||||
"family": "debian",
|
||||
"url": "https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2",
|
||||
"checksum_url": "https://cloud.debian.org/images/cloud/trixie/latest/SHA512SUMS",
|
||||
"checksum": "sha512",
|
||||
"track": "latest",
|
||||
"keep": 2
|
||||
},
|
||||
{
|
||||
"name": "ubuntu-24.04",
|
||||
"description": "Ubuntu 24.04 LTS (noble) server cloud image",
|
||||
"family": "debian",
|
||||
"url": "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img",
|
||||
"checksum_url": "https://cloud-images.ubuntu.com/noble/current/SHA256SUMS",
|
||||
"checksum": "sha256",
|
||||
"track": "latest",
|
||||
"keep": 2
|
||||
},
|
||||
{
|
||||
"name": "rocky-9",
|
||||
"description": "Rocky Linux 9 GenericCloud (Base)",
|
||||
"family": "rhel",
|
||||
"url": "https://dl.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2",
|
||||
"checksum_url": "https://dl.rockylinux.org/pub/rocky/9/images/x86_64/CHECKSUM",
|
||||
"checksum": "sha256",
|
||||
"track": "latest",
|
||||
"keep": 2
|
||||
},
|
||||
{
|
||||
"name": "almalinux-9",
|
||||
"description": "AlmaLinux 9 GenericCloud",
|
||||
"family": "rhel",
|
||||
"url": "https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-latest.x86_64.qcow2",
|
||||
"checksum_url": "https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/CHECKSUM",
|
||||
"checksum": "sha256",
|
||||
"track": "latest",
|
||||
"keep": 2
|
||||
},
|
||||
{
|
||||
"name": "centos-stream-9",
|
||||
"description": "CentOS Stream 9 GenericCloud",
|
||||
"family": "rhel",
|
||||
"url": "https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-x86_64-9-latest.x86_64.qcow2",
|
||||
"checksum_url": "https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-x86_64-9-latest.x86_64.qcow2.SHA256SUM",
|
||||
"checksum": "sha256",
|
||||
"track": "latest",
|
||||
"keep": 2
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,13 @@
|
||||
[Unit]
|
||||
Description=rollcall: refresh OS deployment images from upstream (downloads only when the vendor checksum changed)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=rollcall
|
||||
Group=rollcall
|
||||
ExecStart=/usr/local/bin/rollcall os sync
|
||||
Nice=10
|
||||
IOSchedulingClass=idle
|
||||
TimeoutStartSec=6h
|
||||
@@ -0,0 +1,10 @@
|
||||
[Unit]
|
||||
Description=rollcall: nightly OS image refresh
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 03:30:00
|
||||
RandomizedDelaySec=30m
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
Reference in new issue
Block a user