From d9ae89b295d82d500d2289ede6313fd70118ecc2 Mon Sep 17 00:00:00 2001 From: ilker Date: Tue, 27 Oct 2026 01:29:53 +0100 Subject: [PATCH] Rollcall: bare-metal commissioning server, agent, image build, lab and CI Go server (API, embedded web UI, CLI) backed by PostgreSQL with AES-GCM sealed BMC secrets; Debian live commissioning agent and live-build recipe; Proxmox lab with pve-vbmc virtual BMCs; Gitea workflows for CI, releases and deployment; project website. Co-Authored-By: Claude Fable 5.1 --- .gitea/workflows/ci.yml | 79 +++ .gitea/workflows/deploy.yml | 101 ++++ .gitea/workflows/release.yml | 43 ++ .gitignore | 6 + README.md | 140 +++++ .../rollcall-agent.cpython-314.pyc | Bin 0 -> 35732 bytes agent/build-image.sh | 121 ++++ agent/rollcall-agent.py | 508 +++++++++++++++++ agent/rollcall-agent.service | 15 + cmd/rollcall/client.go | 300 ++++++++++ cmd/rollcall/config.go | 48 ++ cmd/rollcall/crypto.go | 74 +++ cmd/rollcall/go.mod | 14 + cmd/rollcall/go.sum | 28 + cmd/rollcall/ipmi.go | 69 +++ cmd/rollcall/main.go | 68 +++ cmd/rollcall/server.go | 533 ++++++++++++++++++ cmd/rollcall/store.go | 203 +++++++ cmd/rollcall/web/app.js | 158 ++++++ cmd/rollcall/web/index.html | 75 +++ cmd/rollcall/web/style.css | 53 ++ deploy/config.example.json | 15 + deploy/install.sh | 28 + deploy/pxelinux-default | 27 + deploy/rollcall.service | 22 + lab/cloud-init/pxe-server-user.yaml | 86 +++ lab/pve-vbmc/config.example.json | 12 + lab/pve-vbmc/install.sh | 16 + lab/pve-vbmc/pve-vbmc-addrs.service | 13 + lab/pve-vbmc/pve-vbmc-addrs.sh | 13 + lab/pve-vbmc/pve-vbmc@.service | 21 + lab/pve-vbmc/pve_vbmc.py | 160 ++++++ site/index.html | 199 +++++++ 33 files changed, 3248 insertions(+) create mode 100644 .gitea/workflows/ci.yml create mode 100644 .gitea/workflows/deploy.yml create mode 100644 .gitea/workflows/release.yml create mode 100644 .gitignore create mode 100644 README.md create mode 100644 agent/__pycache__/rollcall-agent.cpython-314.pyc create mode 100644 agent/build-image.sh create mode 100644 agent/rollcall-agent.py create mode 100644 agent/rollcall-agent.service create mode 100644 cmd/rollcall/client.go create mode 100644 cmd/rollcall/config.go create mode 100644 cmd/rollcall/crypto.go create mode 100644 cmd/rollcall/go.mod create mode 100644 cmd/rollcall/go.sum create mode 100644 cmd/rollcall/ipmi.go create mode 100644 cmd/rollcall/main.go create mode 100644 cmd/rollcall/server.go create mode 100644 cmd/rollcall/store.go create mode 100644 cmd/rollcall/web/app.js create mode 100644 cmd/rollcall/web/index.html create mode 100644 cmd/rollcall/web/style.css create mode 100644 deploy/config.example.json create mode 100644 deploy/install.sh create mode 100644 deploy/pxelinux-default create mode 100644 deploy/rollcall.service create mode 100644 lab/cloud-init/pxe-server-user.yaml create mode 100644 lab/pve-vbmc/config.example.json create mode 100644 lab/pve-vbmc/install.sh create mode 100644 lab/pve-vbmc/pve-vbmc-addrs.service create mode 100644 lab/pve-vbmc/pve-vbmc-addrs.sh create mode 100644 lab/pve-vbmc/pve-vbmc@.service create mode 100644 lab/pve-vbmc/pve_vbmc.py create mode 100644 site/index.html diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml new file mode 100644 index 0000000..0fbe19a --- /dev/null +++ b/.gitea/workflows/ci.yml @@ -0,0 +1,79 @@ +# Runs on the containerized act_runner (label: ubuntu-latest). +name: CI + +on: + push: + branches: [main] + pull_request: + workflow_dispatch: + +jobs: + go: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-go@v5 + with: + go-version-file: cmd/rollcall/go.mod + cache-dependency-path: cmd/rollcall/go.sum + cache: true + + - name: gofmt + working-directory: cmd/rollcall + run: | + out=$(gofmt -l .) + if [ -n "$out" ]; then echo "gofmt needed on:"; echo "$out"; exit 1; fi + + - name: go vet + working-directory: cmd/rollcall + run: go vet ./... + + - name: go test + working-directory: cmd/rollcall + run: go test -race -timeout 5m ./... + + - name: Build (static, linux/amd64 and linux/arm64) + working-directory: cmd/rollcall + run: | + for arch in amd64 arm64; do + CGO_ENABLED=0 GOOS=linux GOARCH=$arch go build -trimpath -ldflags="-s -w" -o rollcall-linux-$arch . + done + ls -la rollcall-linux-* + + - uses: actions/upload-artifact@v3 + with: + name: rollcall-linux-amd64 + path: cmd/rollcall/rollcall-linux-amd64 + + agent: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Python syntax + shell syntax + run: | + python3 -m py_compile agent/rollcall-agent.py lab/pve-vbmc/pve_vbmc.py + for f in agent/build-image.sh deploy/install.sh lab/pve-vbmc/install.sh lab/pve-vbmc/pve-vbmc-addrs.sh; do + bash -n "$f" || sh -n "$f" + done + - name: Config examples are valid JSON + run: | + python3 -c 'import json,sys; [json.load(open(f)) for f in sys.argv[1:]]' deploy/config.example.json lab/pve-vbmc/config.example.json + - name: No secrets committed + run: | + # the agent token must stay a placeholder in the tracked PXE menu + grep -q '@AGENT_TOKEN@' deploy/pxelinux-default + ! grep -rEn 'rollcall\.token=[0-9a-f]{16,}' --exclude-dir=.git . || { echo "real agent token committed"; exit 1; } + + site: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: HTML sanity + run: | + test -f site/index.html + python3 - <<'EOF' + from html.parser import HTMLParser + class P(HTMLParser): + def __init__(self): super().__init__(); self.depth=0 + p=P(); p.feed(open('site/index.html').read()); print("ok") + EOF diff --git a/.gitea/workflows/deploy.yml b/.gitea/workflows/deploy.yml new file mode 100644 index 0000000..d9cde7b --- /dev/null +++ b/.gitea/workflows/deploy.yml @@ -0,0 +1,101 @@ +# Continuous deployment to the lab PXE server after CI passes on main. +# +# Secrets: PXE_SSH_KEY private key for the deploy user on the PXE server +# Variables: PXE_HOST e.g. 10.0.0.234 PXE_USER e.g. ilker (needs passwordless sudo) +# SITE_HOST / SITE_USER / SITE_PATH optional: rsync target for the website +# +# The image rebuild (agent/**) runs live-build on the PXE server itself (~8 min) +# and only when agent files changed, or when dispatched by hand. +name: deploy + +on: + workflow_run: + workflows: [CI] + types: [completed] + branches: [main] + workflow_dispatch: + inputs: + rebuild_image: + description: 'Rebuild the commissioning image too' + type: boolean + default: false + +concurrency: + group: deploy + cancel-in-progress: false + +jobs: + server: + if: github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-go@v5 + with: + go-version-file: cmd/rollcall/go.mod + cache-dependency-path: cmd/rollcall/go.sum + + - name: Build + working-directory: cmd/rollcall + run: CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags="-s -w" -o ../../deploy/rollcall . + + - name: SSH setup + env: + KEY: ${{ secrets.PXE_SSH_KEY }} + run: | + mkdir -p ~/.ssh && chmod 700 ~/.ssh + printf '%s\n' "$KEY" > ~/.ssh/id_ed25519 && chmod 600 ~/.ssh/id_ed25519 + printf 'Host *\n StrictHostKeyChecking accept-new\n' > ~/.ssh/config + + - name: Ship binary + deploy files and restart + env: + HOST: ${{ vars.PXE_HOST }} + USER: ${{ vars.PXE_USER }} + run: | + set -eu + ssh "$USER@$HOST" 'sudo mkdir -p /opt/rollcall/deploy /opt/rollcall/agent && sudo chown -R "$USER" /opt/rollcall' + scp -q deploy/rollcall deploy/install.sh deploy/rollcall.service deploy/pxelinux-default deploy/config.example.json "$USER@$HOST:/opt/rollcall/deploy/" + scp -q agent/rollcall-agent.py agent/rollcall-agent.service agent/build-image.sh "$USER@$HOST:/opt/rollcall/agent/" + ssh "$USER@$HOST" 'cd /opt/rollcall/deploy && chmod +x install.sh rollcall ../agent/*.sh ../agent/*.py && sudo ./install.sh && sudo rollcall list >/dev/null && echo DEPLOY_OK' + + image: + needs: server + if: > + (github.event_name == 'workflow_dispatch' && inputs.rebuild_image) || + (github.event_name == 'workflow_run' && contains(join(github.event.workflow_run.head_commit.message, ' '), '[image]')) + runs-on: ubuntu-latest + steps: + - name: SSH setup + env: + KEY: ${{ secrets.PXE_SSH_KEY }} + run: | + mkdir -p ~/.ssh && chmod 700 ~/.ssh + printf '%s\n' "$KEY" > ~/.ssh/id_ed25519 && chmod 600 ~/.ssh/id_ed25519 + printf 'Host *\n StrictHostKeyChecking accept-new\n ServerAliveInterval 30\n' > ~/.ssh/config + - name: Rebuild the live image on the PXE server + env: + HOST: ${{ vars.PXE_HOST }} + USER: ${{ vars.PXE_USER }} + run: ssh "$USER@$HOST" 'sudo /opt/rollcall/agent/build-image.sh 2>&1 | tail -20' + + site: + needs: server + if: vars.SITE_HOST != '' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: SSH setup + env: + KEY: ${{ secrets.SITE_SSH_KEY || secrets.PXE_SSH_KEY }} + run: | + mkdir -p ~/.ssh && chmod 700 ~/.ssh + printf '%s\n' "$KEY" > ~/.ssh/id_ed25519 && chmod 600 ~/.ssh/id_ed25519 + printf 'Host *\n StrictHostKeyChecking accept-new\n' > ~/.ssh/config + - name: Publish website + env: + HOST: ${{ vars.SITE_HOST }} + USER: ${{ vars.SITE_USER }} + DEST: ${{ vars.SITE_PATH }} + run: | + set -eu + tar -C site -czf - . | ssh "$USER@$HOST" "mkdir -p '$DEST' && tar -C '$DEST' -xzf - && echo SITE_OK" diff --git a/.gitea/workflows/release.yml b/.gitea/workflows/release.yml new file mode 100644 index 0000000..30409ba --- /dev/null +++ b/.gitea/workflows/release.yml @@ -0,0 +1,43 @@ +# Tag v* -> build static binaries and publish them as a Gitea release. +name: release + +on: + push: + tags: ['v*'] + +jobs: + release: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-go@v5 + with: + go-version-file: cmd/rollcall/go.mod + cache-dependency-path: cmd/rollcall/go.sum + + - name: Build + working-directory: cmd/rollcall + run: | + set -eu + for arch in amd64 arm64; do + CGO_ENABLED=0 GOOS=linux GOARCH=$arch go build -trimpath \ + -ldflags="-s -w -X main.version=${GITHUB_REF_NAME}" -o rollcall-linux-$arch . + done + sha256sum rollcall-linux-* > SHA256SUMS + + - name: Create release and upload assets + env: + API: ${{ github.server_url }}/api/v1/repos/${{ github.repository }} + TOKEN: ${{ secrets.GITHUB_TOKEN }} + TAG: ${{ github.ref_name }} + working-directory: cmd/rollcall + run: | + set -eu + body=$(printf '{"tag_name":"%s","name":"%s","draft":false,"prerelease":false}' "$TAG" "$TAG") + id=$(curl -sfS -X POST -H "Authorization: token $TOKEN" -H "Content-Type: application/json" \ + -d "$body" "$API/releases" | python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])') + for f in rollcall-linux-amd64 rollcall-linux-arm64 SHA256SUMS; do + curl -sfS -X POST -H "Authorization: token $TOKEN" \ + -F "attachment=@$f" "$API/releases/$id/assets?name=$f" >/dev/null + echo "uploaded $f" + done diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..93e3748 --- /dev/null +++ b/.gitignore @@ -0,0 +1,6 @@ +/cmd/rollcall/rollcall +/deploy/rollcall +/deploy/config.json +*.key +/site/dist/ +.DS_Store diff --git a/README.md b/README.md new file mode 100644 index 0000000..bc9cb56 --- /dev/null +++ b/README.md @@ -0,0 +1,140 @@ +# Rollcall + +Rollcall is a small, self-contained commissioning service for bare-metal servers, +in the spirit of MAAS enlistment and commissioning, without the rest of MAAS. + +A server that PXE-boots for the first time gets an ephemeral Debian live image. +The image takes a full hardware inventory (DMI, DIMMs, CPUs, disks, NVMe, NICs, +InfiniBand HCAs, GPUs, PCI, LLDP neighbours, TPM), creates an administrator +account on the server's BMC through in-band IPMI, reports everything to the +Rollcall server and powers the machine off. From then on the server is known: +it shows up in the web UI and the CLI, its BMC can be driven through the API, +and it will not boot the commissioning image again unless you ask for it. + +Everything server-side is **one static Go binary**: HTTP API, embedded web UI, +CLI, PostgreSQL schema management and the PXE bookkeeping. + +## How it works + +``` +unknown server ──PXE──▶ dnsmasq (DHCP+TFTP) ──▶ pxelinux default menu + │ 8 s timeout + ▼ + Debian live image over HTTP (kernel+initrd via TFTP, + squashfs via HTTP, runs entirely from RAM) + │ + rollcall-agent: inventory + BMC user + POST /api/v1/commission + │ + rollcall server: PostgreSQL ◀─┘──▶ writes pxelinux.cfg/01- + (next boot: "commissioned → power off") +``` + +* **Enlist**: a machine is identified by its DMI system UUID (fallback: serial, + then boot MAC). First contact creates it in state `new`. +* **Commission**: the inventory upload moves it to `ready`. The BMC password + the agent generated is sealed with AES-256-GCM (key in + `/etc/rollcall/master.key`) before it touches the database. It never leaves + the server again: power and boot-device operations run `ipmitool` server-side. +* **Re-commission**: removes the per-MAC PXE override; optionally sets + bootdev=pxe and power-cycles the machine through its BMC. + +## Repository layout + +| Path | What | +|---|---| +| `cmd/rollcall/` | Go server + CLI + embedded web UI (`web/`) | +| `agent/` | `rollcall-agent.py` (runs inside the live image), its systemd unit, `build-image.sh` (live-build recipe) | +| `deploy/` | `install.sh`, `rollcall.service`, `config.example.json`, `pxelinux-default` | +| `lab/` | Proxmox test lab: `pve-vbmc` (virtual IPMI BMC for VMs backed by the Proxmox API), cloud-init for the PXE server VM | +| `site/` | Project website (static) | +| `.gitea/workflows/` | CI (fmt/vet/build/tests), release on tags, deploy on push to `main` | + +## Server + +```sh +# build +cd cmd/rollcall && CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o rollcall . + +# PostgreSQL role + database +sudo -u postgres psql -c "CREATE ROLE rollcall LOGIN PASSWORD '...';" -c "CREATE DATABASE rollcall OWNER rollcall;" + +# config (see deploy/config.example.json), tokens from `rollcall keygen` +sudo install -d -m 0750 /etc/rollcall && sudo install -m 0640 config.json /etc/rollcall/config.json + +# install binary, systemd unit, pxelinux menu, tftp symlinks +cd deploy && sudo ./install.sh +``` + +`config.json` keys: `listen`, `image_dir` (served under `/rollcall/`), `tftp_root`, +`database_url`, `master_key_file`, `admin_token` (web UI + management API), +`agent_token` (what the live image presents), `commission_append` (kernel +command line baked into generated PXE files), `bmc_hints` (out-of-band BMC +mapping for machines without in-band IPMI, e.g. VMs). + +### CLI (same binary) + +```sh +rollcall list +rollcall show m-2ef1dab828c8 [-json] +rollcall events m-2ef1dab828c8 +rollcall power m-2ef1dab828c8 status|on|off|soft|cycle|reset +rollcall bootdev m-2ef1dab828c8 pxe|disk|cdrom|bios|none +rollcall recommission m-2ef1dab828c8 [-now] +rollcall delete m-2ef1dab828c8 +``` + +Reads `ROLLCALL_URL` / `ROLLCALL_TOKEN`, or the config file when readable. + +### API + +| Method | Path | Auth | Purpose | +|---|---|---|---| +| POST | `/api/v1/commission` | agent token | inventory upload from the live image | +| GET | `/api/v1/bmc-hint?macs=` | agent token | out-of-band BMC hint for VMs | +| GET | `/api/v1/machines` | admin | list (summaries) | +| GET | `/api/v1/machines/{id}` | admin | full record: inventory, BMC (no secret), events | +| POST | `/api/v1/machines/{id}/power` | admin | `{"action":"on"}` etc., runs ipmitool server-side | +| POST | `/api/v1/machines/{id}/bootdev` | admin | `{"device":"pxe"}` | +| POST | `/api/v1/machines/{id}/recommission[?now=1]` | admin | boot the image again on next PXE (`now=1`: via BMC immediately) | +| DELETE | `/api/v1/machines/{id}` | admin | forget the machine | +| GET | `/rollcall/*` | none | image files (kernel, initrd, squashfs) | +| GET | `/` | token in browser | web UI | + +Admin auth is `Authorization: Bearer `; the agent sends +`X-Rollcall-Token`. + +## Commissioning image + +`agent/build-image.sh` builds a Debian 13 live image with live-build (about +8 minutes, ~780 MB squashfs). It contains the full Debian kernel with non-free +firmware, `ipmitool`, `freeipmi`, `rdma-core`, `infiniband-diags`, `mstflint`, +`nvme-cli`, `smartmontools`, `lldpd`, `lshw` and an SSH server (root, key only) +for debugging. Output goes to `image_dir`. + +Kernel parameters understood by the agent: `rollcall.server=URL` (required), +`rollcall.token=…`, `rollcall.bmcuser=NAME` (default `commissioner`), +`rollcall.nopoweroff` (stay up after reporting). + +### BMC account + +In-band (`/dev/ipmi0`): find the LAN channel, pick the existing `commissioner` +slot or the first empty user ID ≥ 3, set name/password, ADMINISTRATOR privilege, +channel access, enable, SOL payload. The password is random (16 chars) and is +reported once to the server. If no in-band IPMI exists the agent asks the server +for a hint (`bmc_hints`) and verifies it out-of-band. + +Vendor quirks (iDRAC/iLO password policies, missing empty slots) are reported in +`bmc.errors` and visible in the UI. + +## Lab + +`lab/` documents a Proxmox SDN test setup: an isolated vnet, a PXE server VM +running dnsmasq + Rollcall, two diskless VMs, and `pve-vbmc`, a tiny +IPMI-over-LAN BMC emulator that drives VMs through the Proxmox API so that +`ipmitool power on` and `chassis bootdev pxe` work against virtual machines. + +## Status + +Working end to end in the lab (virtual machines, out-of-band BMC hints). The +in-band BMC path is implemented but has not yet been exercised on real +hardware. diff --git a/agent/__pycache__/rollcall-agent.cpython-314.pyc b/agent/__pycache__/rollcall-agent.cpython-314.pyc new file mode 100644 index 0000000000000000000000000000000000000000..6a8e66d7c63fed472f45cd2bf0eb18ec4ab06831 GIT binary patch literal 35732 zcmcJ&d2k!oohROn`#yM+;sM^ID3YQsiaJg45Oq;hlL94~5)Bd{35z742B-r*LT6GH z%63A^Rz~nx^D{hl%Jew?nXY8Y@+P~XGnuinGrMUDw80?SvZp40thau(B}@KoCdvM4 zf1mdn-2f?2_RQ9{#D~}2uiyQ?_xrv&i`l^7`hWiWXHWk7;|%l9+%}23h*62rB5WGN`1#s-TMgs)K6!s|jl8uQsTqzq+7~{_2Bz`fCUp=&vzo zqQB;#>8!G+P%1w8+f$N^OEFHd4YeXywfMuhO$<}WoVArQoPtxHV>wl`QhasHcvjz2 zmPnTJ6U^Y$xlAyVb8wnmW}FRXaiyG=#w#u;cVmop-D9%n+B&zTVxa2A9c zI4iUH>75*PxJEsb!Cel1YBJ#PO%8h;-pPR1?QsOh zJPwb4%roKPUE_{%??t?sK;j02q0Kiw?imgS94^P0i+5jg@#L$E$j8TD+2k1ZO^$d+ zr+5!ZaNt;*!!l=jv?3Zc}LKPF9II^ zqKB{d^S+Cyhuh(s3fB8Z#IMG@$Y#XHBP%{Qg?df6hVjf3sNH1n`vRzv2j5-^;@K$4 zjYmy~k-vi!>Gxgo@B!Me#AgoQ$OzhV#KTW|#vQ{G?s4Sh@cD!2o4_uEL;k)&tkmAI zV9>v-v9W1$!6@qsHwSni?HN^=nf<9hw-PLKXH7%R}ekr5sXwc+xe2k@Go*d59@eTOKJ#qF@EgRPl zO^u9rcvN^eu}10iQzIlR^1(C$0#BTbiP@s?vjcJ}Cp&=2C|(@foQRh&$)^J<>3xp` zpQH!)J&`k*Z`Fu*+|!4MdSrYmFxFkGimUx- zMlh~Cdi-Eq8MqRNYXU(xW?!u$uAB&r##Ll?1V}xr9PwP*X^qkZtEaaO{ws0ixNnrV zAW<>?0$UMGGb`qT@YZmpV028k-&N^nDz5i_qRPLkHC!*hvGLl*sJ8H~&OGA|SHyHB zUn&`0(F1(O43iF&E*&DcS3OWs0KI`3klrvoayQD4m{N5302u^&Aue~|wnS=BaaGWBIT+Uly%Qcf$X|R5WyhIX#TQQ_!sj5U)x`~gsUd)(VNW0s zR{}`J4c-7CWY7griCGfl;|8AAV%X>Q@T70z_D)Q#ZeOqyK+oO5^FBUq=@U!oxa_BZ zLKRm6+VEMVpiJ@&>kp90UR71);LYS|z7$j<5|}_R&D>LI)wy>~IkVl*cZavW`Q$?8 zPllsayKZk=aa4SzWX;Vp?B`b7?9*4DelIJ0;Nq2rVn<$zf+9=%UxsP zR}8DpU9sjoZ@FvCju|VKj1|d8_mp_~$3O*2{C2ChHJACwU?O3zxplwtqpA!NwkS!s zk3~3~I{8Qw=&J9g^-ZqKepZ@xmyn(>cY>Or*vy`oRH;)P-;zthV+q`)_<}qXhebiV z2Ll`Az5ZGiPiO|(nQ;X_9A|xrz6AUxeHm9`eEBNGn(!AO9Xie2HCgABQB%=#9n%L^ z@(RMPo7$PC+2*UwFSH1%>IjE`w<2Zw_Ps>&NY_5UVgr-1XmTlplKZHK^w63MOI44ntJ$_^w%al+JV2oJ_P75opGius>`2ly{AyBvp#c_z8IVhJs%2} zzVM8otBV8?U~XAs#*!st37Jk=acd@-Oj&;nkSYGXqI%{Xww`%6yH%t7NX;UogSo1M zwMa-O(0`(4H zQ~%h2$bT@lTJ<04Rfv9MRgy5n+*+pmsFX!GjO6k~`jk}h!3u(DrjhZdj23z`F>;R@ z7aP-gm|^^^lT9~594qA^;h<5JSI8Uz+W_K$%oTbi({T!x>CsEilfRr&%IOkv;LH6g zr%KAxP4nzi9Rm~5uOLw3$WeyT1ARL&f`oKvB=Em6K^=zl63TDrjm%JEDZzw-FJ0gxGrX_S~ga#zezu|Tnk!N<)BeK2FSU%V;hv5$*`|+(unr~3 z7wJYiy#E=IX{LeUw8vRLI>pAE#Pn`trWBok-WTl2CCVu|m6L6dJ9|iR1S_9(r5}`^ zQ^6@VfqRjz-K;bOqm-KiQkTj&I4}^*v z@v?Dv+!Y8k;!UI1O(u(bYB)GZl7}4YzZVOswettix{y?##MkwNkPlfBC4-(J+#278 zlzcl$ZkK9$ekxJTkU8;a#5Lg^zY@wJ9LSIl?AfX|raoU)R&Fx=s^*8XhR4u4Z(wyc zCh2MLir*8;mhM)6YL^}(15dC%=o$?fhrGT(Jy=Dzj}Ms=H?(yjJH6vQ;~vz!-VOex zRzZ;&U0}7!xaz#;N`T*s_QzQ_g^pM^;;J#A^9RKa;Jc8Z2Y-Pn1k=oYh0638JKg@J zf!VSns@w5?)_m)WZF3vK?ntX(tBRVc@93-F&zf!(bUQxJ%?lTOl3N|i<%C@BN85#_ z6VW6?fe&T3}IYtD>$j?p+ZJy}f5qzjATv8RS7){e%f4Hw_^v!fl zU(IbTQ?25O2*AP&S3Ph5p-iU&OO{h4=e|=0oynEtXR-d2{Fd~9QbJ__r=J0w0=V_a zz!7C8E>0z_3TfTBF`)_>PL1^GfTp~P&M~PgFI`%x2Ip3-(MA?*g`&qO)gbxnlwm{( za|E1>Q#pd30camWJbmr8YiplC22U{zr*o=Wka^bPudmgKt84?(1Ah@TMj2gxCm!(4pEc`A!0b`_glcf)21|~`~H0~Qd z56m_^#q)&MqpOrBgbKeOL4a_URTRsuTCw71d>`JPCCUB)!8G$bgL&quaQmC(5ziYN z1xtO@&@kP$l94s%o@raLW!|t{vwUiE%&1n3*4h16_lH|z#>zX!%9YH*u=}+qUwLvq z^Htxx>yzTmvEm&|#XF+KJ8#)PDc&t)?7{b@jM-yXkA=%WH5IRzZ7&W+vfe6ry<{OP zR=Q)kbjPi&ixvN2_wC@XzxAtc34@;4@sZ`@BP%%tH=exq@`BDIiwH|D4ZWMBf#lQ}V5DZ!0}FtfbS2A(nbWM3 zLSCqxj?*`$F7$CMv;UK&88{=z1k*b8GE2S7S=LEml~dT(Ng7QeJ)~=Wr^y(oHEdnkyS13?;pi$dW4` z$d;aR6}gm9mB6=@8&~OM8i_!#>I+hOT-EChc_5EWqn;P@37at}Qmi2(5C<>? zC0`-Lq||K4+)IC@$0{C;iK{pg8X-}i7$jsch%#d*ZXBKBjU5%}^95bwy9NSvH3ROt zT8SN+@Jx6oM|@}}c|ACECFluw85KbxZtujzU|`$_R>w->fkC;eR?$NQyllna1H$H% zfw7)u`j}2gOfJ~DRTPH=uoCzkgMW^_%uK66J3vfECM(f!Y*#`ksNf$*X5DWyZ!6-8 zhRysz+$BV|BBiL`1AT|*vY+1<&qo{3yA6BDc#jA6HOQ}?C0_xSGGCxhLRshAo_YJN z4Ie!F{<_br{imWW__83}`dY^;9g&7;Vbj9i z#nO)}KdQWKjvjEXn6lp13v3OSKI~nUcz0`MFI5QgU z8@p%JX!OP}jf|!6ULIq!Pj}tb8D{m*>xGP}AF1bk(VBfx-Tvv;&$PO`nK@|A zrE8bM7hVp%ynjZwV#*KijJW@D|NPOPmEZFGbmNMt=w@Jv%zg1p;*k*ukov$nzVF|hSN%Y z`fxirtzSDJvqm8A)37pYag0;L=}5n(U`9Q#(IdaL$&yS7Qy~`v9UYAH41^t!RGw$Y(p~$!n%2321c2akayG8>PwE?`7_#oB6rU$@8&MO08 z;Aj9~#p=FxN&pvla}~f<#+4T*E_c`J`R|e%C<7NK;!4-W%W)NgEpaWv?M+)o00IM^ zZG&6z&O;PyW3ItTZ(Qe^xI7qywuGO=#{k}b{O|-Pq{3|e0?9cC%}xWVzfTflm?@~n zf-IV3-!?~@R*J^IRI3iuf`NY(H@>r{55~3k;r*~2r zhvQIt$N;(m>SZd@F(7}3D58n0>GQa*H|X*Ep+F58$;%1XWyi6uP~yfrNpJK=o7|3J z=(;^2-O(*_Y^xZHTb!ag+#7NcNhALq^hd3hitza$={yZjRV8sHFfm0s3BE4i<3YfH zn`57ZMikUcoP}O@IIg3{csQ`1uY?(cI_2gW0_`s*;5@4z7n)#d`UYSwNT>9Rbn7Me# zTpSKW&E-qFiWPgtjqGdLbD_wVsJ$v?-z3;K&6nM=Z<=XcsoON~e79w$eXjWj1u=Ws zlD#alHEOSpnra>>nT+}`)r`e)U#BoVN#s5nrlKZl*#7#quxml}cFT+VpywI8IyQIV zh4atvdwttrrT!B zORAi^IfY-+i8il(&-k|SN9G6g4q@&ji!_Gkj(={nOdtNkeIsMs5g`2VFC4n|Qq?c& zvk?yCUcLw!N>N1?0ClQN2w_Z<224n7K=<^qX#sjQ&<0dG zR6d;rP_{LF$w>(m&;)e71gJqeoXW03Dny{`2sjP7N zQfyKl>1v#|Ub6g=v#(QX#=knxRCYUkJ!N+L8`m>Oss)$3PCaD?{Tt_*C*}E#zmQq{ zMr}XGsZwVQAq>E%15eWVDHA9XDS{QCol-Oq4Gq}t3zxWp)UUY>tL_Sk9zHRvz)d-| z-HIbcV$Ob@U&|FaS+Mys;p3EH5?7K!;y6I!VAe}DkXp+r!RV)(qos0+GN+m=cj_87 zGNGgNO1c^?NS<0k@(hiikDI8p5C%wI7-R56UKm$RUYr2KO)x7Zb$BcbP(gtS@V~>s zaZpKK{B_~)$+ub2K)^IXkTIBDq9p+#;N>TX{aV~6#Re1FmbjS*XG_rOAP+8|_mZr& zV;dNiM>I-XR`b^n&&hML4kE#4my0tZh0i8zYIiNmU&1f%L+~)kRuF(d z0ftd&$hLRM2W1E-hh2Q&!;XauHv^HTi1UXnZ|#fhTVP`)TZED=KkZ16ZuxcWO=}$+ z*S3oj0Ads}%5fv)Cqbf;g^mnN^k0$MOd|*>>W|}Z2xISt0hfw&8viC<5H{)gmS%6+TpB;iN!Ag&RWRY>l+2ni3wCQ#sGX2g}3E+IOGc}9(d zMD0@`1nJReeZVsu()!1)kO(9IC@`TGh;3OQi2C~YT2jXp z+AT$tVDt}rAyN_-0HM$MD|5&B)se}lWBWpAvFYROA8i*7JsxcvSh1Bv>~Cei zo*nsCw6ytF(QW&$vwxL+`p|E*@ zi*4Amv|-P!Q_&5bD>g@@>83tCuUB(VxI00Owg2M&YcV& z{ZaY6=N$+^@^5ZkBFi*dd?32v0G0CG=(^SgDNt7ReC|Re`ape8tuP&jR79aZ&i>(k zDH)bO+%IO#?Q9?elj;{$1&0~K?dqJvYSph48id1mDqlcVA}P#>q_F(LqouNA`5rcd zGD;zukfcO~ki?A)UkdpN0fcKV2|01f26j`*I}pf#qtpTtC9S=ruVexY%qG#XNcV0u z4fkw1wP@0?i^_HsN5E*qf>K7N$+D6%8LCX|@Pp(_N8U=c>92UyTS6clfwqN^2VfGk z34#=s9DZHW7PH5aEMJO2rmuGD(#hsI4Kzpu8R3$C{rZ1%ZrU`t8UD?=>C)t8bn4fP zk-jHWtbb~l)H%QcNbfgs&?0dr&dgajD`(^ETn3lfs7}`-P7{~4ij>Nh`^AiRIh3%1 z1_iUj#N|?RN6cF&-1GeAK5e@B!sV~i_XRS!v>~OBvQpb2#xZ_{AiBPO1QRQWE9x_G z#eHTFbA(T&SI@sT-_)^`@^$=gbw`$1pkplH6lF|*NbEp zXkOS@Q4)tnVSOzUR&i}_qCpxrkIg7N8c2Wne<0Z?InTvfW85&<({^aEt+lPILo{mW z#!G5y$rEJ^wO!zeT@Zhf1VIv9MnH5hL$wzE1c|Fqet_>K@5q{1W6Qw*U*y?Al2AP{ zI6MyF*aUb6h+HQKZ3Y{N@c{qd@j5`jt4wIo2vtGxpOOqn@HZq?9;LK?r?ubF7TvYx z3%dNfuz$D|HVBsTsG$Ou54McC12g3b%ZDfkg#(doFhQHmitQXtNAlZBJ_IU z&y3_U9pS2%k42h) zbl|hF@(EPCEawSQvS7 z#gv<%RzF_~0CZRC%F97=ZT&_e^7uP*MyZMOqU^!C({@||BN60zmYgg=PiC5xC=mB$Rs!j6@W;9|6h2pLf+IM0Q-za*e4d!5B zj3!eH;}j+q7TXYhCQanhi5Zfe(s_UgAkBp6_Ck#VrGmVrZF*jR-+qCJ-)K<|q zu-W0Xq%L1ulFs?RUy}YCm1J0_Bv^BRa{8J!amJlp3N)~V z|8=u4!>Lc(gCZyNH6xbjrA$%_Cksj^JFfKv$H-n#{w+KcHJNNZwH^#vsObx=Pd3&J z^w!mO^G_g2$l5;TANO3AAA^dZ4?;Ody)e;%)eY2n9(VG+@(Pq#WdGjzP)YBU-%r(E z*boX+eUgY!Q|AWGp4|vS3fln5<4|of9r?~dL_&dGH4Pj0r@ZmIuq`v56_BzVBuNKp zQV5=ULXISrw&j_c{kwL_)XOMX-bUQAk8NO)9Jq;Rp2{_NShUGnRBNjs>raAwJ~a| zi`jN8*>-$t+X*!OFiB!G3AT<;jU6jBweM-)*1n^kX}^(wEkC?9qKw)rqNYlSF6=E| z=@`&P-f*U1E{f`lrw_<>9pw=&s;hxXM_D-tfrHl$hFh;6p3j%uxET^@mSfKQmu}RC&*J%c<(-k=JIV+Eg#)_446J<(;hRg>6Wh^W0H6>9z%A z*n*@R>yk-hSrtoJ70X#w3sU(8(`@bYwL;coi)FtY`q=%EJ8C#G-S(N*uu{G8tt+oz zS+KwHbga5%sk$Xvy?aI_PzC5+y$Mat3I}i-)mNl6cKZS!)$O9CLaKD|m4lJin}=y( zB}=M&n8l{s&p~!xP|9JCYBl8mtycb{v|6i;RFS&i(Wya!Nc%&8#9&IW$|l^&Lf;0; zn209Gt`Ov|bi$kj2Cu&aiiX+8>hwUyYED5}X;_8O zH>2%S6WjN6qz@sznl8BoH@n7lhoRh?7N3`3`D%L zCcXV@(r2ur%T+hZBB``}k(sIuC$p2%$Aly#ThGXjO3_dys3_uNu})5<%ne}ARLbtD!nnfgk86NMpm&Qa#(i-;MSE1Qp$1;?@F8o; z#wQ3H8vm$m6uEk;Wx;-W*D!2@+^|(5`>H2`Q*k}^M#4%3sy0}u414320Ppo(9He%T z7e@l2oJ0nVtDkqr^{{${H4)Hr$S$Rk9|6z#FK&zLNl_G4#We}l9}u7iq6%vJ8D|@@ zQIGeGcrRnm9*}Sr&=~(V`icKr5~y*DO|$Y-7&jq5a7KA}fgR!ScQNTDZY^p}DMmL6 zUm&6L@mz*>ClJ?>Jyx{SbTBR&CmTMgNhpu;gd}9jPc}XA1ea22MXesy13eLlvoP}_ zC4z>T^bX^GoHvt_RgjIi`C<}Pk{1uFcKClH^(Vpqi@(5sL(+s=r!H#R6fc$UE)?mGgK)n7VDhu zd*|sc#JP$WcZW+~t9+$0VvH7S61E*#E;vHu4|Yc+Golu(RWqu)R{QLwtCwbfk?^j?HamKCG-fGYvJ~IsU%T|mrC3ShQc2^&wuP!&T4D2J(UR7v z<-m;cb8!1@;p~KV#v65PhW>$ClHQv7Y2o5iDJG$wXx~VZ4}S_qJo6_PpxY?C>4^j* zBf_QwLPcA&xII>UR46`rTZObYvm+Ie9HDxrP|_03-xbS0Amkso<%Wo1@#M#+KRSIo zAoMw-9sRM6$Co-D7Y0TJ@A>79ahU&PZ(ZsbUu0ME@?TTGq7IL}YMtM)l-Kl?lF8XK zcVh0+56(o&Kgq8Ak#fFu{`|Y$(VE?{?A=24ZhQrUqU^2rj9RNvm6s9lg> zfIy@7GrF==f6wG~<=6auQ60izJe03I#0dO*U;`#wyCEfI7kWbQFMo#!*il(aEI}BG z$N!GN)vv*GWTvjUl9mO2z-Meez+4rtud9cX1+HO}xFcDc&Ur#p^rct1D3bOI|AyMi z)+|zKoH7NQsZb+TvPPVG&ASw;g43>fr%U-xzvi7G<(+ZOJ5$QLR7_0jl7fjj>zeOu z$?wu&X?tS|b~y6aB5vqo($O|tMxTPqOv#;$OI9*>E_=Z6$awm4c`Ss$}%drWx)3!Afw^=-~Ldn4I{SQ*Mq!Q)9Z*SImHYou&) z<3>0x0KW`#41&5tnnn?Y8({nppyn~2kg+j&1KSCbOx;L+;s#1LjJvLYVlXx)Z?JV9 zo2ONH;)YGk1h{HfD9)KjcM67<0ofwJh#)e&kgq#WK8>J|)w-39ip_RZxDGK6)6bZ=y)F)z*Bu-H> zpAP`WYY-4C5f?Nw-XYRCJk}6{%_3p^#8so?z9D`iN!f&ZeiMmNRRy)HA)+O|jy$U; z0g;)&$fbkC50Y0uBmpTx6vz_ku!CHpqM??&Ab}NsfiDn9thqV!di#w-FC7w!o1QcST@2i=SUeNm-?L)NyIBbfq4G$7#2hVXSTO$V zYp{0~NLw!)QC##Zdb)SC&e6nCc7y*ahO&IcAY z3iW%V)%%3UdX~9<;qd|C%vs^=xNvGB%1si{zbAYlJRHdlAB*N}oIm_C^`d=Ix!7{c z7v0${^qg59a0#9f!96OB`h}qj(E(m097;jz92QC%qxqZBJO8Nowf0xqUt4}<5zag= zoE{JchJ}82tjlw!%X59pjpl33vCQ(N%ZCz z?ZSB)S6z9J7|ux=dzOiy@dlkFPt3xT=XxcbtVcNnf<#>d_M;c;&_ z1sITC;x?|AZ3k;L{9%kFY{o{Y%@~-0G2bNsB#(D=Y{&;*0ls9&?n7c0mycp$yapr0 z6#hgV825PKK8aZS!*z+yjSUZg!LU=LdGN2`BhizR3{j4tM4kkd_zV0w5`de~8fFg8 zofeEGQLQ6l|3q6Jxv;FQ`HeO6E|im3p1<Zf*Bb9QR=fAL1qOGOEDz`d{UUDyyS-+QNg?@Cto+{h0Oh0lMIwQ&Zv zX!g&Iw&|lFu(8Aq&l~2NzHeS3PadBB8q}2%x}?%B7OFG+idVYnnDQ2c@!BhOa{E{= z?Ve^QyOrfwxNy{^>>!c}=7&vXG4u+m;a_FXNebVJzX!yl#U56+=mf_I8e6Vcx+at| zy|UygVZ~XLpa-DmBfBN}8e{sqk7bR*6_yv81#asQZW2(L-^ zRXL_b+;V}uE5TmML1Oy~V*+;?E6*KdmY`7PU#OX9#=iGYEwI;{Z|r+%U!*3MStn%H z{rJ=$=p6gA@+!D=<3AwTnq7fmuXk|V69hYqTDiTWUepeLsk~6L!vb>@u0i7LB!8OZ zPrypmW8&^kQ)-3yC-9n3NC8;lGyj5_YR!&W%a^R>Z;re*@%qFk)~%S6Mlr;rS@kb?|C$?&J@UOA=#s>swy0qKwI(U1&mTqpUv6tO=` zS0u4Zb@AAy1(=d(N0rzPNG&G3yc=~V(#!xfv}B<{`)!zcI_&<$S~h=h*|<$mZF_j4 zlNm!NdYpcg*iMLC)?O04h51__W)qxXIa4=%nFu^eD;tw1aZ+9SD0911IGvE3LR-<_ zL|T@(KY+5-kIJK;s*8>|6Ot(weaccr5vFQ48YUWclp_YPc!E5j3S4P9aQS>!;ITW;=?*h^ZdT086+~GPf6F zVxucu9ychvQ?NAf1#l#_{+DNHdf#&AWEcSdN~G=dO6Te^Ek`GRE#W_L#5l+X$=)4K z1A|L~+hw^m+2xX?CN80N{5B_i22;5u#$&Y;fE4}(o?;e~eM3l>Me<7- z&ulIyWjr;~IIJ1ZJ&-uTXtO7O?YJcUP;$AcS%6h3UQPzY zSScaKoaQ9yl{6am+nlx$U|~Qf`5Uebdz#iy{ms`-ximiF+8L0hfV^AEZ|Ab%AG{!C zp33X3muXPbx0UJMI_>mrNxFAjRmxMjmyq7hRi}HC@=PwBtKn+7ja*%W4t3a+E=7Z~ zLG91r>iwC{46F%=F8o=^)#sG;Xem$=9A#qmW3_P&`0|@Tphj+UPpveb$zNv{*Oa<0 zrM`xwZi`b1&96KwF+V^VP@E}a7;Y=KZB@JEeoEU`dFGZdeTtsCWG$o^m)*;7+xu*A zG?OU}v&cR7*$GbKb~rN|Q))!uT59d(Z+24HzvgWM_HU%~G}(q`ZYS5WHYaWuYO>pD zr})?azIhc6^(+0^6oX+s4&wOI(b|6OQ1_wUKCZRzI0vWl^!uZ&-46HzoP?+KhPVzU zGH^dQtF9d}|){%Abb+9rL(Z$J@z z0VzjCTEu^X{C5$t8PnYz>N-b_kl+G`xWXZ3K;$>bPkEk(2m65M8{7``Hf*e|fcDfa z$}2U*H9c4w>?)h`P$i?tCk#?=WFi2@DQ-V65X=#TgPBpK^vE-cvb4uv=^W z=k$v96Zir)5uh6hcal2m65Jv14OtyT&O?3aC`Iw&6gr?=bkHF3g>e?zT6xIRu!Ti% zSC3&LLzsps^dUdG^-p99t*x(Thg;`6->qG^vb6EB>)JVXuHuIFCG8v8;?}=Ev(~UT z>@{rq)(6pgnJWg(wYKf&lskVW!<#g1msgzFJHU_VomDooiQWMZxu$nW$IvcH^iINk z#nYsHgesR@c@tKde-=+KkT+Pp;-<&UW3Qi#unVepM{n8R9>0EYv1!ryamz<7*ALFM z&be4#^1ejeZA$w^*^*2hf% zCti3gaS!@@Ll8Y#Nl9eBi;fBTxQRX>wgQvEkvOgKV89Ex7fA;{q$A+?CXo|e{Bd@3 ztu|y&u6su>1t7d1DMH5HsUhh29mibQ`pJ_g8SSEXWye84uS+;_;ZRS=vUBs6hNcS+ zCxH_1){))j#iL9Vb zfC}>|k2-~C*d^c!!V?sg?nA_xFxpeWB~_5hEBLeI^9BSc@)`WVI>P4`HywajWcYFN zp7cOmMSvJjmJ@%IydjiJTmeRQFO5u%$JG>ih*o}2lZQ&;!&c$HMAS7NKLD%mJRc+9 z5>XEoGo(42LX!T)4as%QzlLJF@E6EYQQ68rTMy4DiIea%u+B3TeX1*l^xbxZ{hdnp z-NWBLJageY$L@-v7V55e6E=O`F;+uJxMIwlI~Be>pCK4F(ai^Q+rl|>4N-IHbDeO_ zURgcg^k)C{Q{kcO0}I#X3av`vSs^o2a%?%VxE1o`g&ft{Cd%R5NXuU z9sV493*0m1cg-1dO>;xvA0S&AvgSPD{x^r`OWzm~vg@O^hO6os1q48I_U|{(Jsr95 zX>QG3dw#g|KuxM`hH@57Ryu#P? zVSR*+Y<;UG(lTH6QaScfSPG!O*O|WkG`1$pbbZep(-ki13bDmtS+{M)nlWQo$<8N@ z_)@`cJde$-Y{ob?w~~0Ib*@)5N__duOZDP2gm|B5Qg$AFJrK>_D0ak(J?lp9wcKzK zoS%qkeg|LRJHk8O+!_hK(IRBjN6ihi8;Nf)Xmf6uubE%AMienyxnL`YL59gP+x2{x zko|-(G!iw9CbqsxzQHQqR7P5fZeqUmjl*wP7jj@_k^5opqWz}@v6e$iEr+5lM{b`I z`cK9B&n@+1%jpx*ekexHFZG{aDR96CSoQ1G=rXT?1Ov%xWASyJUf zfV@nQM%3E6xGiev!VVd&;oDDPQY;aW8qGX*I~cX}!6=11UNZ`tmvv2a6z=M?Uw{GW z=X&d0c36XUw!NX7uN5#x^oaqQ73CXU^FspC<4MlV%y93kd6BI!-e`YEFX*D{Y>Rtez*%0yW$P+Pcj#Jf1D@icP*=Suc)-&J^b9^ISnRh%lz=d zu8+A7`P+R$?;y4(3s(2C%7Z?dY5DH6-+4A%9`^oa!~CJ;g6*Vk?(m^VaK3jT|J`#a zv*orMB6{HzHqp3+Y>(i(ux#T=-Jt|5!M@~IF3l)sx|eht=&qGN+&|8eSxZH2f48Ba zXTNF~;6=U&HX(lxK;#l>7xYi?6r3WEmvORzQo8boPS7TlQYDVrl_3)xB?e(QUEfDE zwKn*u9rwa_*D=V;-9pqp@cb=^uM1+2Dwj_PV|y=120Vpk#VF4`@nM>r|yFgGL@9& zbnD~<*E^CJhBK^_CdD9jofNVdi!(v+39oYEKH-OIVH+R>0KL}9DWyIV%gWi;No#mrt~;j*~w21TlRB2|q_8S~41IJPZCL@SG)`eXqp*jE+I zjdREXq^ZF=;y!oHeV%*|62}b&A7iWLOsQj(PMF2X?ry+ZtIHV&^QiMTI@qJ7^7$Ky zcETkUMCNO+9+3QNuatQUke~;XeN>K`0r2hyr%ukTu`y_ZCK`5W(jPue6qwe;);fW6CcufXy=2@*B#-%)1 z%^|Wos!geO7|p6JuW&9juw zP13%E*oE0D@i_}ZKdvHuLOiEjh_gKZSEQ)& zI2FT;9UWvxX)X41+{vm)kgCq`>3Md(_FdD$$Sv2;Js)}&J8!o|_w}q8?Q=~xwqM&G z&UtxnWaGT+Jog~QQ?{fl|Gw}701Q_JbL&*6KdHrxlr_URw)S;+cH;8xSmw|}^O zG3Rz>boa4nGi*!XceOv}e0<3X)6yZqJrZ?}LT^4-dZY4MW!Mlo5Y4D2-bZVv+X<~# z5w3`6!o{CtR4-(pE$foZD;tU;_E(xgBoY#8C4WQs!YjGBGcBpISG4+>6W_VChEm$O ztZSj@{P(5k5OkGb-u9>8!F@WM#>PygOQzB{x5dhvm&%)emMa)r*PQr6X2`c&t2*nL zk7{x|bJZX1*CGD*xyH^K{oj|VNW4Z#!n$n-%atFOu?U9=u9PmIei3)}VU>KHwjQ`f zc1x8Gi%J*~Mx`35fN!b-6EW}>g;SXnYfAYM?<37BF-N~DMcpF{+#wPL@8?$o_dmL1 zq6A!XIaM&ylaI4YgMl?ESUnA#L$Ao+I!_Tj zGW<+7X8<@}ytXO;Q^B7WGVrV_<4E9fE$AFgpBe{9NJ;UFXWBjms2jN*P+}HJ33`g9 zSn?M}H7P<`9FL{)Yxir9{1Sp@8uMZ+L*o26T_{6>;mNZBODACcf)5>Vhu2&$y80Y^0a-{jQ{g18z!!r%byIg(<5JYc>2 z`{eDbctST*lT+l#m9QlsHRgYc6eRn==SW3t39Pfbp5HY$8Z{Q<_=F{6Wz<-W1BMK? z*@mkPF@s~t;D~fab_#WSh0=YCeapR1M0L1)32I3&ULdn2A8 zrC{Irr{CFfZkVo_Vz$a9TjiUV1Z(40aOzY*PHL-^An`56Hl^_w zdsT@4tx`!sO=Wwv`fux++RN0xEK?&sj4$QOh!`c{yI{SWYVjcf0CFDEqirO}4kMX( z#i*v*v?UL&Dq+q+>WaNcTDfdF!pXn{7C)$mc&$^eOmZhT@k*qlREvbeqV)BazbE8g zK(kIM@)CPk!^~;GY{=|DdKW`-?H(*0jHakjW{n&lPtmN~{2MRS9YQz(y&@9yumGp?fgo zgC3xX*Clqgp-!-AkRfSLL-GqL*3%@#lE2A)VAz?4mJoS2yh31C8dpf`OY{x&l1313 ziq^EGIN64EcC*PoJ0G4s#Ig8EeWDep!G%}St}gkg329Cz^oc+yPCZwW%50z|s32*H z-{98+4R4p)kaW)FH#&_Yxog&@BiW9DX6c*$0%`$hVVp0#IKTIdhQ%dU37VCh{rvC(|O}kTUO|afh|0h^hfC!*tru3Wr zvKQFZs|xI6Dw&lij+6bprmLUd!d3QJ(tcMZ^%7UTPKp{iMQxt}y5zh@71-rE(sS$7 zDYsLHcl@RT_zJ3eEbIn+VaG zxL}X7P2G~!b2(vY0Clo{SBeUtpf3|IBpa31^i=EN)m9M2>FlA<5%9ZEL? zcSyC7M-I3S!UMT~Mya4ErT)#*7Z4f9sYGr2IS{tzSXf}IrMhuDlc`fi6he%(*J``$ z0m9JJi)-O_B`w>4u~CMD86ZPTFA%nvGla0^@=jCuj*(wwyOqv1M>ou85o;hFcLQk#J%W)4X&~O9hez@e^cFpZg?d&K zx7yYsfw=wWf8lKiYEihRCC*43-66#;M@ZY*dgy3JyCamDva>4apJ>G2Mx3VRpP=^j zAxrW@V)#$Z5<=Uc`%T(QY{D_^aHKYxXd6r$cnEwPIJ~#sH&Rcu?+z+J@scVL{vxhK zH-*#)b~#|;09_zGO_KV#k-!?kjn480ukfYhg~2_AZCk`6av+p1 zcBq4Bnf=7q3Jxy7ISM$0jwgEYkQN7WL&>_A-%0K&arnfzD3XHrW#YaKwj^+x2G4O< z$a(c*W}z&cr|t_j%C8!T9G@p}6e2Ykg^=3B4U<0VR@pZ)5~@QXS4bC3`5iUM=MMBl zqMR#~SHsZ`IG}-kMpENbsL1Ic68F72DL)tw)6SxXAXG9(Bx|CKjf{M475^CNrXUFj zk<0%%35W@xsPTQBBquszV#-1eh=?0;8jv)p#LeCW^gbEDs{v06V(+N-(DrF7G7&Uy#SY!(Sj%B{HWOFSNjAuh~9(>guVv zv8cIZy7M!m9qMJlbZ}YM^&6{g1@gSJSI>sG(wD!hZd$1LpyvIWTg}leZNKb|RUf)j zeF%0WHP>pwFzzqE%}*a8-tlzTbm9D{tvqJiF4(q5o}E4fz2U6)dG8D7rw`s$X$8ZE zaA%k(?Uq%QpBoFpZIQzHOScTa?2NTL@3cFk?WdryzJ4;iKbl=Pe`}40- z48hUO{K=oR%^$tB^OsIx*U{Tome06^?xB_J;y1NIR?YmLpA`rj+ZU()@x0J^MmRgX z?C}Z%Pd-qxS>r5x^C|DCH2Rv~O6Z*}Uz& zD}Q}OXz5;VJRY_8EHxgVFTGxNqxxEPETe2Gqb$<&X+|ZUF4=oTrSRR1oR_N*)jW5E zG*Oqgtjhnqq>Svak6ekCG{s7qg%bE9Mi4a~S}JJ{+kf8iVaHh~?zKlT;yPmf}${;}8lA}1DG?=j3__PF9p;@$qZ;%0le z^)K4-@;*)VfMM(>6i8y@NT}~s;A?fC;&Y3gxDCK8v6RFtm4c-*vSs? z^3?p^Xi3Yhos0b+KmO6!r0{!kth!AKy{H%|JDAv)0j{(bERNL zK6lKODbB5D_rADyWniCg6=_$JCPzUh9=`xo9<&(9zH zq&DNu z#_wK!?(&=|rY(Xq$+!CGdTCkK?fNV$Z{~uaD+#+0kO8qQLyw+fnSnd zvtah&R)+c!Ko$Wzcv zj8jGUObmVaegYQ-5_~{ziLiku(i;&ym_&U{168l^Zy{OSl6aKR(1`ni>qrI!X7hx% z;B_RRh=){&C$y)yp^vbO{9`1b=%tUuo*_Ym1ivA{?@1t|7tzhgKappYWTBXZY?z|E zjzmi)idbGJ(bpcoN}5O_^GjZjL1gK-u7n*^zaAk~2<6gYcew}j&q_woM~sqvo) z=n-OkS@uhnkySm&V_4H~8RKsmJ^gR_En}wt?Z0IV^m*oQne5LuZ}~v`zV-vl`<7eJ z#x{4~+1&k{duCsF>wM`anoUAqf2{A^Qs230<$aBcb$n&fv$HX7w);0+%7XHfj!0YC;;l9R!C!MI>G4{S88=q{> zMX)WCJ;Z*s$;jHjDps)jzcR4w_WL>=GRtXYS?j$n1!J{O8%gQr`<6`B`c;DgpB3z5 k&G!x{7?o)zw5-f0mKa^%?z-Cdg7Z^V7A&Jw1+=UFf7Ut&u>b%7 literal 0 HcmV?d00001 diff --git a/agent/build-image.sh b/agent/build-image.sh new file mode 100644 index 0000000..ebf86c5 --- /dev/null +++ b/agent/build-image.sh @@ -0,0 +1,121 @@ +#!/bin/bash +# Build the Debian 13 commissioning live image with live-build. +# Run as root on pxe-server. Output: $OUT/{vmlinuz,initrd.img,filesystem.squashfs} +set -euo pipefail +SRC=$(cd "$(dirname "$0")" && pwd) +BUILD=${BUILD:-/srv/rollcall/build} +OUT=${OUT:-/srv/rollcall/image} +MIRROR=${MIRROR:-http://deb.debian.org/debian} +AUTHKEYS=${AUTHKEYS:-/home/ilker/.ssh/authorized_keys} + +mkdir -p "$BUILD" "$OUT" +cd "$BUILD" +if [ -d config ]; then lb clean --purge >/dev/null 2>&1 || true; rm -rf config; fi + +lb config \ + --distribution trixie --architectures amd64 \ + --archive-areas "main contrib non-free non-free-firmware" \ + --mirror-bootstrap "$MIRROR" --mirror-binary "$MIRROR" \ + --binary-images iso-hybrid --bootloaders syslinux \ + --debian-installer none --memtest none \ + --firmware-chroot true --firmware-binary false \ + --apt-recommends false \ + --bootappend-live "boot=live components quiet" \ + --linux-flavours amd64 \ + --iso-volume "rollcall" --iso-application "rollcall commissioning image" \ + --cache-packages true --cache-indices false + +# Package set: hardware discovery + BMC + IB + firmware + debugging +cat > config/package-lists/rollcall.list.chroot <<'EOF' +linux-image-amd64 +live-boot +live-config +live-config-systemd +systemd-sysv +systemd-timesyncd +busybox +kmod +pciutils +usbutils +dmidecode +util-linux +lshw +ethtool +iproute2 +isc-dhcp-client +nvme-cli +smartmontools +hdparm +sg3-utils +lsscsi +ipmitool +freeipmi-tools +rdma-core +ibverbs-utils +infiniband-diags +mstflint +lldpd +python3 +curl +wget +ca-certificates +openssh-server +openssh-client +less +vim-tiny +procps +pv +jq +mdadm +lvm2 +nvme-cli +efibootmgr +firmware-linux +firmware-linux-nonfree +firmware-misc-nonfree +firmware-bnx2 +firmware-bnx2x +firmware-qlogic +firmware-realtek +firmware-cavium +firmware-netronome +amd64-microcode +intel-microcode +EOF + +# Files baked into the rootfs +mkdir -p config/includes.chroot/usr/local/bin config/includes.chroot/etc/systemd/system \ + config/includes.chroot/root/.ssh config/includes.chroot/etc/modules-load.d \ + config/hooks/live +install -m 0755 "$SRC/rollcall-agent.py" config/includes.chroot/usr/local/bin/rollcall-agent.py +install -m 0644 "$SRC/rollcall-agent.service" config/includes.chroot/etc/systemd/system/rollcall-agent.service +printf 'ipmi_msghandler\nipmi_devintf\nipmi_si\nipmi_ssif\n' > config/includes.chroot/etc/modules-load.d/ipmi.conf +if [ -r "$AUTHKEYS" ]; then + install -m 0600 "$AUTHKEYS" config/includes.chroot/root/.ssh/authorized_keys + chmod 0700 config/includes.chroot/root/.ssh +fi + +cat > config/hooks/live/0100-rollcall.hook.chroot <<'EOF' +#!/bin/sh +set -e +systemctl enable rollcall-agent.service +systemctl enable lldpd.service +systemctl enable ssh.service +# allow root login with key only +sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config +# lldpd: speak LLDP + CDP so switch ports are discovered quickly +printf 'configure lldp tx-interval 5\n' > /etc/lldpd.d/rollcall.conf +echo rollcall > /etc/hostname +EOF +chmod +x config/hooks/live/0100-rollcall.hook.chroot + +echo "=== lb build started $(date)" +lb build 2>&1 | tee "$BUILD/build.log" | grep -E "^P:|^E:|error" || true + +for f in vmlinuz initrd.img filesystem.squashfs; do + src=$(ls binary/live/$f* 2>/dev/null | head -1) + [ -n "$src" ] || { echo "missing $f in binary/live" >&2; exit 1; } + install -m 0644 "$src" "$OUT/$f" +done +ls -la "$OUT" +echo "=== done $(date)" diff --git a/agent/rollcall-agent.py b/agent/rollcall-agent.py new file mode 100644 index 0000000..fe55573 --- /dev/null +++ b/agent/rollcall-agent.py @@ -0,0 +1,508 @@ +#!/usr/bin/env python3 +"""Commissioning agent: runs once inside the ephemeral live image. + +Collects a hardware inventory, configures a BMC account via in-band ipmitool +(falls back to a server-provided out-of-band hint for virtual machines), +posts everything to rollcall and powers the machine off. + +Kernel cmdline options: + rollcall.server=http://10.213.77.1 (required) rollcall base URL + rollcall.nopoweroff stay up (debugging) + rollcall.bmcuser=commissioner BMC account name to create + rollcall.token=... agent token (sent as X-Rollcall-Token) +""" +import glob +import json +import os +import re +import secrets +import socket +import string +import subprocess +import sys +import time +import urllib.error +import urllib.request + +LOG = open("/dev/console", "w", buffering=1) if os.path.exists("/dev/console") else sys.stdout + + +def log(msg): + line = f"[commission] {msg}" + print(line, file=LOG) + if LOG is not sys.stdout: + print(line, flush=True) + + +def run(cmd, timeout=120, text=True): + """Return (rc, stdout). Never raises.""" + try: + p = subprocess.run(cmd, shell=isinstance(cmd, str), capture_output=True, + text=text, timeout=timeout) + return p.returncode, p.stdout if text else p.stdout + except FileNotFoundError: + return 127, "" + except subprocess.TimeoutExpired: + return 124, "" + + +def out(cmd, timeout=120): + rc, o = run(cmd, timeout) + return o.strip() if rc == 0 else "" + + +def jout(cmd, timeout=120): + o = out(cmd, timeout) + try: + return json.loads(o) if o else None + except json.JSONDecodeError: + return None + + +def readf(path, default=""): + try: + with open(path) as f: + return f.read().strip() + except OSError: + return default + + +def cmdline(): + opts = {} + for tok in readf("/proc/cmdline").split(): + if tok.startswith("rollcall."): + k, _, v = tok[len("rollcall."):].partition("=") + opts[k] = v if v else True + return opts + + +# --------------------------------------------------------------------------- DMI +def dmi_system(): + keys = ["system-manufacturer", "system-product-name", "system-version", + "system-serial-number", "system-uuid", "system-sku-number", "system-family", + "baseboard-manufacturer", "baseboard-product-name", "baseboard-serial-number", + "chassis-manufacturer", "chassis-type", "chassis-serial-number", "chassis-asset-tag", + "bios-vendor", "bios-version", "bios-release-date"] + d = {k: out(["dmidecode", "-s", k]) for k in keys} + # fallbacks via sysfs + d["system-uuid"] = d["system-uuid"] or readf("/sys/class/dmi/id/product_uuid") + d["system-serial-number"] = d["system-serial-number"] or readf("/sys/class/dmi/id/product_serial") + return d + + +def dmi_blocks(dtype): + """Parse `dmidecode -t N` into a list of dicts (one per handle).""" + txt = out(["dmidecode", "-t", str(dtype)]) + blocks, cur = [], None + for line in txt.splitlines(): + if line.startswith("Handle "): + cur = {"_handle": line.split(",")[0].split()[1]} + blocks.append(cur) + elif cur is not None and line.startswith("\t") and ":" in line and not line.startswith("\t\t"): + k, _, v = line.strip().partition(":") + cur[k.strip()] = v.strip() + return blocks + + +def memory(): + dimms = [] + for b in dmi_blocks(17): + size = b.get("Size", "") + if not size or size.startswith("No Module"): + dimms.append({"locator": b.get("Locator"), "bank": b.get("Bank Locator"), "populated": False}) + continue + dimms.append({ + "locator": b.get("Locator"), "bank": b.get("Bank Locator"), "populated": True, + "size": size, "type": b.get("Type"), "type_detail": b.get("Type Detail"), + "speed": b.get("Speed"), "configured_speed": b.get("Configured Memory Speed"), + "manufacturer": b.get("Manufacturer"), "part_number": b.get("Part Number"), + "serial": b.get("Serial Number"), "rank": b.get("Rank"), "form_factor": b.get("Form Factor"), + }) + total_kb = 0 + m = re.search(r"MemTotal:\s+(\d+)", readf("/proc/meminfo")) + if m: + total_kb = int(m.group(1)) + return {"total_bytes": total_kb * 1024, "dimm_slots": len(dimms), + "dimms_populated": sum(1 for d in dimms if d["populated"]), "dimms": dimms} + + +def cpu(): + j = jout(["lscpu", "-J"]) or {} + flat = {} + for e in j.get("lscpu", []): + flat[e["field"].rstrip(":")] = e["data"] + for c in e.get("children", []) or []: + flat[c["field"].rstrip(":")] = c["data"] + want = ["Architecture", "CPU(s)", "Thread(s) per core", "Core(s) per socket", "Socket(s)", + "NUMA node(s)", "Vendor ID", "Model name", "CPU family", "Model", "Stepping", + "CPU max MHz", "CPU min MHz", "L1d cache", "L2 cache", "L3 cache", "Virtualization", "Flags"] + res = {k: flat.get(k) for k in want if k in flat} + flags = res.pop("Flags", "") or "" + res["flags_count"] = len(flags.split()) + res["features"] = sorted(f for f in flags.split() if f in + ("vmx", "svm", "avx", "avx2", "avx512f", "sse4_2", "aes", "sha_ni", "amx_tile")) + return res + + +# --------------------------------------------------------------------------- storage +def storage(): + lsblk = jout(["lsblk", "-J", "-O", "-b", "-d"]) or {} + disks = [] + for d in lsblk.get("blockdevices", []): + if d.get("type") not in ("disk", "nvme") and not d.get("name", "").startswith("nvme"): + continue + if d.get("name", "").startswith(("loop", "ram", "sr", "zram")): + continue + disks.append({k: d.get(k) for k in ("name", "size", "model", "serial", "vendor", "rev", "tran", + "rota", "wwn", "hctl", "subsystems", "log-sec", "phy-sec", + "rm", "ro", "path", "pttype", "fstype")}) + nvme = jout(["nvme", "list", "-o", "json"]) or {} + smart = {} + for d in disks: + s = jout(["smartctl", "-i", "-H", "-j", d["path"]], timeout=30) or {} + if s: + smart[d["name"]] = { + "model_family": s.get("model_family"), "firmware": s.get("firmware_version"), + "rotation_rate": s.get("rotation_rate"), "form_factor": (s.get("form_factor") or {}).get("name"), + "healthy": (s.get("smart_status") or {}).get("passed"), + "power_on_hours": (s.get("power_on_time") or {}).get("hours"), + "interface_speed": ((s.get("interface_speed") or {}).get("current") or {}).get("string"), + } + ctrls = [p for p in pci_devices() if p["class_id"][:2] == "01"] # mass storage controllers + return {"disks": disks, "disk_count": len(disks), + "total_bytes": sum(int(d["size"] or 0) for d in disks), + "nvme": nvme.get("Devices"), "smart": smart, "controllers": ctrls} + + +# --------------------------------------------------------------------------- PCI +_PCI_CACHE = None + + +def pci_devices(): + global _PCI_CACHE + if _PCI_CACHE is not None: + return _PCI_CACHE + txt = out(["lspci", "-Dvmmnnk"]) + devs, cur = [], {} + for line in txt.splitlines() + [""]: + if not line.strip(): + if cur: + devs.append(cur) + cur = {} + continue + k, _, v = line.partition(":") + cur[k.strip()] = v.strip() + res = [] + for d in devs: + m_cls = re.search(r"\[([0-9a-f]{4})\]$", d.get("Class", "")) + m_ven = re.search(r"\[([0-9a-f]{4})\]$", d.get("Vendor", "")) + m_dev = re.search(r"\[([0-9a-f]{4})\]$", d.get("Device", "")) + cls = m_cls.group(1) if m_cls else "0000" + kind = {"01": "storage", "02": "network", "03": "gpu", "0c": "serial-bus", "06": "bridge"}.get(cls[:2], "other") + if cls == "0302" or (cls == "1200"): + kind = "accelerator" + res.append({ + "slot": d.get("Slot"), "class": re.sub(r"\s*\[[0-9a-f]{4}\]$", "", d.get("Class", "")), + "class_id": cls, "kind": kind, + "vendor": re.sub(r"\s*\[[0-9a-f]{4}\]$", "", d.get("Vendor", "")), + "device": re.sub(r"\s*\[[0-9a-f]{4}\]$", "", d.get("Device", "")), + "vendor_id": m_ven.group(1) if m_ven else None, "device_id": m_dev.group(1) if m_dev else None, + "subsystem": d.get("SDevice"), "driver": d.get("Driver"), "rev": d.get("Rev"), + }) + _PCI_CACHE = res + return res + + +def gpus(): + return [p for p in pci_devices() if p["kind"] in ("gpu", "accelerator")] + + +# --------------------------------------------------------------------------- network +def ethtool_info(ifn): + info = {} + for line in out(["ethtool", "-i", ifn]).splitlines(): + k, _, v = line.partition(":") + if k in ("driver", "version", "firmware-version", "bus-info"): + info[k.replace("-", "_")] = v.strip() + et = out(["ethtool", ifn]) + m = re.search(r"Speed:\s*(\S+)", et) + info["speed"] = m.group(1) if m else None + m = re.search(r"Duplex:\s*(\S+)", et) + info["duplex"] = m.group(1) if m else None + m = re.search(r"Link detected:\s*(\S+)", et) + info["link"] = (m.group(1) == "yes") if m else None + m = re.search(r"Supported ports:\s*\[([^\]]*)\]", et) + info["ports"] = m.group(1).split() if m else None + sup = re.search(r"Supported link modes:(.*?)Supported pause", et, re.S) + if sup: + info["supported_link_modes"] = sup.group(1).split() + pm = out(["ethtool", "-P", ifn]) + m = re.search(r"([0-9a-f]{2}(?::[0-9a-f]{2}){5})", pm) + info["permanent_mac"] = m.group(1) if m else None + return info + + +def network(): + links = jout(["ip", "-j", "-d", "link"]) or [] + addrs = {a["ifname"]: a for a in (jout(["ip", "-j", "addr"]) or [])} + nics = [] + for l in links: + ifn = l["ifname"] + if ifn == "lo" or l.get("link_type") not in ("ether", "infiniband"): + continue + sysd = f"/sys/class/net/{ifn}" + if not os.path.exists(f"{sysd}/device"): + continue # virtual (bridge, bond, vlan, veth ...) + pci = os.path.basename(os.path.realpath(f"{sysd}/device")) + nic = { + "name": ifn, "mac": l.get("address"), "mtu": l.get("mtu"), "operstate": l.get("operstate"), + "link_type": l.get("link_type"), "pci": pci, + "sriov_totalvfs": readf(f"{sysd}/device/sriov_totalvfs") or None, + "numa_node": readf(f"{sysd}/device/numa_node") or None, + "ipv4": [a["local"] + "/" + str(a["prefixlen"]) for a in addrs.get(ifn, {}).get("addr_info", []) + if a.get("family") == "inet"], + } + if l.get("link_type") == "ether": + nic.update(ethtool_info(ifn)) + nics.append(nic) + return nics + + +def infiniband(): + hcas = [] + for d in sorted(glob.glob("/sys/class/infiniband/*")): + name = os.path.basename(d) + ports = [] + for p in sorted(glob.glob(f"{d}/ports/*")): + ports.append({ + "port": int(os.path.basename(p)), "state": readf(f"{p}/state"), + "phys_state": readf(f"{p}/phys_state"), "rate": readf(f"{p}/rate"), + "link_layer": readf(f"{p}/link_layer"), "lid": readf(f"{p}/lid"), + "sm_lid": readf(f"{p}/sm_lid"), "port_guid": readf(f"{p}/gids/0") or None, + }) + hcas.append({ + "name": name, "node_guid": readf(f"{d}/node_guid"), "sys_image_guid": readf(f"{d}/sys_image_guid"), + "fw_ver": readf(f"{d}/fw_ver"), "hca_type": readf(f"{d}/hca_type"), "board_id": readf(f"{d}/board_id"), + "node_desc": readf(f"{d}/node_desc"), + "pci": os.path.basename(os.path.realpath(f"{d}/device")) if os.path.exists(f"{d}/device") else None, + "ports": ports, + }) + return {"hcas": hcas, "ibstat": out(["ibstat"], timeout=30) or None} + + +def lldp(wait=25): + if run(["which", "lldpcli"])[0] != 0: + return None + deadline = time.time() + wait + neigh = None + while time.time() < deadline: + j = jout(["lldpcli", "-f", "json", "show", "neighbors", "details"], timeout=20) + if j and j.get("lldp") and j["lldp"].get("interface"): + neigh = j["lldp"]["interface"] + break + time.sleep(3) + return neigh + + +# --------------------------------------------------------------------------- BMC +def gen_password(n=16): + alphabet = string.ascii_letters + string.digits + return "".join(secrets.choice(alphabet) for _ in range(n)) + + +def ipmi(args, timeout=30): + return run(["ipmitool"] + args, timeout=timeout) + + +def parse_kv(txt): + d = {} + for line in txt.splitlines(): + if ":" in line: + k, _, v = line.partition(":") + k = k.strip() + if k and k not in d: + d[k] = v.strip() + return d + + +def bmc_inband(user_name): + """Discover the BMC in-band, create/refresh user_name with ADMINISTRATOR on the LAN channel.""" + for mod in ("ipmi_msghandler", "ipmi_devintf", "ipmi_si", "ipmi_ssif"): + run(["modprobe", mod]) + time.sleep(2) + if not os.path.exists("/dev/ipmi0"): + return None + res = {"source": "inband", "errors": []} + rc, mc = ipmi(["mc", "info"]) + res["mc_info"] = parse_kv(mc) if rc == 0 else None + rc, fru = ipmi(["fru", "print", "0"], timeout=60) + res["fru"] = parse_kv(fru) if rc == 0 else None + # find LAN channel + chan, lan = None, None + for c in range(1, 15): + rc, o = ipmi(["lan", "print", str(c)]) + if rc == 0 and "IP Address" in o: + kv = parse_kv(o) + if kv.get("IP Address") and kv["IP Address"] != "0.0.0.0": + chan, lan = c, kv + break + chan, lan = chan or c, lan or kv + if chan is None: + res["errors"].append("no LAN channel found") + return res + res["channel"] = chan + res["lan"] = {k: lan.get(k) for k in ("IP Address Source", "IP Address", "Subnet Mask", "MAC Address", + "Default Gateway IP", "802.1q VLAN ID", "Cipher Suite Priv Max")} + res["address"] = lan.get("IP Address") + res["mac"] = lan.get("MAC Address") + # user table + rc, ul = ipmi(["user", "list", str(chan)]) + users = [] + for line in ul.splitlines()[1:]: + m = re.match(r"^\s*(\d+)\s+(.*?)\s+(true|false)\s+(true|false)\s+(true|false)\s+(\S.*)$", line) + if m: + users.append({"id": int(m.group(1)), "name": m.group(2).strip(), "callin": m.group(3) == "true", + "link": m.group(4) == "true", "ipmi_msg": m.group(5) == "true", "priv": m.group(6).strip()}) + res["users_before"] = users + uid = next((u["id"] for u in users if u["name"] == user_name), None) + if uid is None: + uid = next((u["id"] for u in users if u["id"] >= 3 and not u["name"]), None) + if uid is None: + res["errors"].append("no free user slot") + return res + pw = gen_password() + steps = [ + ["user", "set", "name", str(uid), user_name], + ["user", "set", "password", str(uid), pw], + ["user", "priv", str(uid), "4", str(chan)], + ["channel", "setaccess", str(chan), str(uid), "callin=on", "ipmi=on", "link=on", "privilege=4"], + ["user", "enable", str(uid)], + ] + ok = True + for s in steps: + rc, o = ipmi(s) + if rc != 0: + ok = False + res["errors"].append(f"ipmitool {' '.join(s[:3])} rc={rc}") + # some BMCs (e.g. iDRAC) also want SOL payload access enabled + ipmi(["sol", "payload", "enable", str(chan), str(uid)]) + res.update({"user": user_name, "user_id": uid, "password": pw if ok else None, "configured": ok}) + if res["address"] and res["address"] != "0.0.0.0": + rc, _ = ipmi(["-I", "lanplus", "-H", res["address"], "-U", user_name, "-P", pw, "mc", "info"], timeout=20) + res["verified_oob"] = (rc == 0) + return res + + +AGENT_TOKEN = "" + + +def agent_headers(): + h = {"Content-Type": "application/json"} + if AGENT_TOKEN: + h["X-Rollcall-Token"] = AGENT_TOKEN + return h + + +def bmc_hint(server, macs): + try: + req = urllib.request.Request(f"{server}/api/v1/bmc-hint?macs={','.join(macs)}", headers=agent_headers()) + with urllib.request.urlopen(req, timeout=10) as r: + h = json.load(r) + except (urllib.error.URLError, json.JSONDecodeError, OSError): + return None + if not h or not h.get("address"): + return None + res = {"source": "hint", "address": h["address"], "port": h.get("port", 623), + "user": h.get("user"), "password": h.get("password"), "errors": []} + base = ["-I", "lanplus", "-H", h["address"], "-p", str(h.get("port", 623)), "-U", h.get("user", ""), + "-P", h.get("password", "")] + rc, mc = ipmi(base + ["mc", "info"], timeout=20) + res["mc_info"] = parse_kv(mc) if rc == 0 else None + res["verified_oob"] = (rc == 0) + res["configured"] = (rc == 0) + if rc != 0: + res["errors"].append("hinted BMC did not answer") + return res + + +# --------------------------------------------------------------------------- main +def post(server, payload, tries=30): + data = json.dumps(payload).encode() + req = urllib.request.Request(f"{server}/api/v1/commission", data=data, headers=agent_headers(), method="POST") + for i in range(tries): + try: + with urllib.request.urlopen(req, timeout=30) as r: + return json.load(r) + except (urllib.error.URLError, OSError) as e: + log(f"post failed ({e}), retry {i + 1}/{tries}") + time.sleep(5) + return None + + +def main(): + opts = cmdline() + server = opts.get("server") + if not server: + log("rollcall.server= missing on kernel cmdline; nothing to do") + return 2 + server = server.rstrip("/") + global AGENT_TOKEN + AGENT_TOKEN = opts.get("token") if isinstance(opts.get("token"), str) else "" + user_name = opts.get("bmcuser") or "commissioner" + started = time.time() + log(f"starting, server={server}") + + inv = {"agent_version": "1", "started_at": started, "hostname": socket.gethostname(), + "kernel": readf("/proc/version"), "errors": []} + for key, fn in [("system", dmi_system), ("cpu", cpu), ("memory", memory), ("storage", storage), + ("network", network), ("infiniband", infiniband), ("pci", pci_devices), ("gpus", gpus)]: + try: + inv[key] = fn() + log(f"collected {key}") + except Exception as e: # noqa: BLE001 + inv[key] = None + inv["errors"].append(f"{key}: {e!r}") + log(f"FAILED {key}: {e!r}") + inv["tpm"] = [os.path.basename(p) for p in glob.glob("/sys/class/tpm/tpm*")] + inv["lldp"] = lldp() + log("collected lldp") + + macs = [n["mac"] for n in (inv.get("network") or []) if n.get("mac")] + try: + bmc = bmc_inband(user_name) + if bmc is None: + log("no /dev/ipmi0, asking server for an out-of-band hint") + bmc = bmc_hint(server, macs) + except Exception as e: # noqa: BLE001 + bmc = {"source": "error", "errors": [repr(e)]} + inv["bmc"] = bmc + log(f"bmc: {json.dumps({k: v for k, v in (bmc or {}).items() if k in ('source', 'address', 'user', 'configured', 'verified_oob', 'errors')})}") + + sysinfo = inv.get("system") or {} + inv["identity"] = {"uuid": sysinfo.get("system-uuid"), "serial": sysinfo.get("system-serial-number"), + "macs": macs, "boot_mac": None} + # the interface that got DHCP is the one live-boot used + for n in inv.get("network") or []: + if n.get("ipv4"): + inv["identity"]["boot_mac"] = n["mac"] + break + inv["duration_s"] = round(time.time() - started, 1) + + resp = post(server, inv) + if resp: + log(f"server accepted: machine {resp.get('id')} state={resp.get('state')}") + else: + log("could not deliver inventory to server") + with open("/root/inventory.json", "w") as f: + json.dump(inv, f, indent=1) + if opts.get("nopoweroff") or not resp: + log("staying up (nopoweroff or delivery failure)") + return 0 if resp else 1 + log("powering off") + time.sleep(3) + run(["systemctl", "poweroff"]) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/agent/rollcall-agent.service b/agent/rollcall-agent.service new file mode 100644 index 0000000..71f9780 --- /dev/null +++ b/agent/rollcall-agent.service @@ -0,0 +1,15 @@ +[Unit] +Description=Hardware commissioning agent (ephemeral image) +After=network-online.target lldpd.service +Wants=network-online.target lldpd.service +ConditionKernelCommandLine=|rollcall.server + +[Service] +Type=oneshot +ExecStart=/usr/local/bin/rollcall-agent.py +StandardOutput=journal+console +StandardError=journal+console +TimeoutStartSec=30min + +[Install] +WantedBy=multi-user.target diff --git a/cmd/rollcall/client.go b/cmd/rollcall/client.go new file mode 100644 index 0000000..8719bf3 --- /dev/null +++ b/cmd/rollcall/client.go @@ -0,0 +1,300 @@ +package main + +import ( + "bytes" + "encoding/json" + "flag" + "fmt" + "io" + "net/http" + "os" + "strings" + "time" +) + +type client struct { + base, token string + http *http.Client +} + +func newClient() (*client, error) { + base, token := os.Getenv("ROLLCALL_URL"), os.Getenv("ROLLCALL_TOKEN") + if base == "" || token == "" { + if cfg, err := loadConfig(envOr("ROLLCALL_CONFIG", "/etc/rollcall/config.json")); err == nil { + if base == "" { + base = "http://" + strings.Replace(cfg.Listen, "0.0.0.0", "127.0.0.1", 1) + } + if token == "" { + token = cfg.AdminToken + } + } + } + if base == "" || token == "" { + return nil, fmt.Errorf("set ROLLCALL_URL and ROLLCALL_TOKEN (or make the config file readable)") + } + return &client{base: strings.TrimRight(base, "/"), token: token, http: &http.Client{Timeout: 90 * time.Second}}, nil +} + +func (c *client) do(method, path string, body any, out any) error { + var rd io.Reader + if body != nil { + b, _ := json.Marshal(body) + rd = bytes.NewReader(b) + } + req, err := http.NewRequest(method, c.base+path, rd) + if err != nil { + return err + } + req.Header.Set("Authorization", "Bearer "+c.token) + req.Header.Set("Content-Type", "application/json") + resp, err := c.http.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + data, _ := io.ReadAll(resp.Body) + if resp.StatusCode >= 400 { + var e apiError + _ = json.Unmarshal(data, &e) + if e.Error == "" { + e.Error = string(data) + } + return fmt.Errorf("%s: %s", resp.Status, e.Error) + } + if out != nil { + return json.Unmarshal(data, out) + } + return nil +} + +func human(n float64) string { + units := []string{"B", "K", "M", "G", "T", "P"} + i := 0 + for n >= 1024 && i < len(units)-1 { + n /= 1024 + i++ + } + if i <= 2 { + return fmt.Sprintf("%.0f%s", n, units[i]) + } + return fmt.Sprintf("%.1f%s", n, units[i]) +} + +func runClient(cmd string, args []string) error { + c, err := newClient() + if err != nil { + return err + } + need := func(n int) error { + if len(args) < n { + usage() + } + return nil + } + switch cmd { + case "list": + var ms []Summary + if err := c.do("GET", "/api/v1/machines", nil, &ms); err != nil { + return err + } + f := "%-16s %-12s %-28s %-16s %4s %7s %3s %7s %3s %2s %3s %s\n" + fmt.Printf(f, "ID", "STATE", "MODEL", "SERIAL", "CPU", "RAM", "DSK", "STORAGE", "NIC", "IB", "GPU", "BMC") + for _, m := range ms { + bmc := "-" + if m.BMC != nil { + b := *m.BMC + ok := "FAIL" + if v, _ := b["configured"].(bool); v { + ok = "ok" + } + bmc = fmt.Sprintf("%s (%s,%s)", str(b["address"]), str(b["source"]), ok) + } + model := strings.TrimSpace(deref(m.Manufacturer) + " " + deref(m.Product)) + fmt.Printf(f, m.ID, m.State, trunc(model, 28), trunc(deref(m.Serial), 16), m.CPUs, human(m.MemoryBytes), + fmt.Sprint(int(m.Disks)), human(m.StorageBytes), fmt.Sprint(m.NICs), fmt.Sprint(m.IBHCAs), fmt.Sprint(m.GPUs), bmc) + } + case "show": + _ = need(1) + fs := flag.NewFlagSet("show", flag.ExitOnError) + asJSON := fs.Bool("json", false, "raw json") + _ = fs.Parse(args[1:]) + var d obj + if err := c.do("GET", "/api/v1/machines/"+args[0], nil, &d); err != nil { + return err + } + if *asJSON { + b, _ := json.MarshalIndent(d, "", " ") + fmt.Println(string(b)) + return nil + } + printMachine(d) + case "events": + _ = need(1) + var d obj + if err := c.do("GET", "/api/v1/machines/"+args[0], nil, &d); err != nil { + return err + } + for _, e := range toList(d["events"]) { + fmt.Printf("%s %-24s %s\n", str(dig(e, "ts")), str(dig(e, "event")), str(dig(e, "detail"))) + } + case "power": + _ = need(2) + var d obj + if err := c.do("POST", "/api/v1/machines/"+args[0]+"/power", obj{"action": args[1]}, &d); err != nil { + return err + } + fmt.Println(str(d["output"])) + if ok, _ := d["ok"].(bool); !ok { + return fmt.Errorf("ipmitool failed") + } + case "bootdev": + _ = need(2) + var d obj + if err := c.do("POST", "/api/v1/machines/"+args[0]+"/bootdev", obj{"device": args[1]}, &d); err != nil { + return err + } + fmt.Println(str(d["output"])) + case "recommission": + _ = need(1) + q := "" + if len(args) > 1 && args[1] == "-now" { + q = "?now=1" + } + var d obj + if err := c.do("POST", "/api/v1/machines/"+args[0]+"/recommission"+q, nil, &d); err != nil { + return err + } + fmt.Printf("%s -> %s\n", str(d["id"]), str(d["state"])) + if d["ipmi"] != nil { + fmt.Println(str(d["ipmi"])) + } + case "delete": + _ = need(1) + var d obj + if err := c.do("DELETE", "/api/v1/machines/"+args[0], nil, &d); err != nil { + return err + } + fmt.Println("deleted", str(d["deleted"])) + } + return nil +} + +func trunc(s string, n int) string { + if len(s) > n { + return s[:n] + } + return s +} + +func toList(v any) []any { + l, _ := v.([]any) + return l +} + +func printMachine(d obj) { + s, _ := d["summary"].(obj) + inv, _ := d["inventory"].(obj) + sys, _ := inv["system"].(obj) + cpu, _ := inv["cpu"].(obj) + mem, _ := inv["memory"].(obj) + st, _ := inv["storage"].(obj) + fmt.Printf("%s state=%s hostname=%s\n", str(s["id"]), str(s["state"]), str(s["hostname"])) + fmt.Printf(" %s %s S/N %s UUID %s\n", str(sys["system-manufacturer"]), str(sys["system-product-name"]), + str(sys["system-serial-number"]), str(sys["system-uuid"])) + fmt.Printf(" BIOS %s %s (%s) board %s\n", str(sys["bios-vendor"]), str(sys["bios-version"]), + str(sys["bios-release-date"]), str(sys["baseboard-product-name"])) + fmt.Printf(" CPU: %s sockets=%s cores/socket=%s threads=%s numa=%s\n", str(cpu["Model name"]), str(cpu["Socket(s)"]), + str(cpu["Core(s) per socket"]), str(cpu["CPU(s)"]), str(cpu["NUMA node(s)"])) + fmt.Printf(" RAM: %s DIMMs %s/%s\n", human(num(mem["total_bytes"])), str(mem["dimms_populated"]), str(mem["dimm_slots"])) + for _, dv := range toList(mem["dimms"]) { + if d, _ := dv.(obj); d["populated"] == true { + speed := str(d["configured_speed"]) + if speed == "" { + speed = str(d["speed"]) + } + fmt.Printf(" %-14s %-8s %s %s %s %s\n", str(d["locator"]), str(d["size"]), str(d["type"]), speed, + str(d["manufacturer"]), str(d["part_number"])) + } + } + fmt.Printf(" Storage: %s disks, %s\n", str(st["disk_count"]), human(num(st["total_bytes"]))) + for _, dv := range toList(st["disks"]) { + d, _ := dv.(obj) + kind := "hdd" + if d["rota"] == false { + kind = "ssd" + } + fmt.Printf(" %-10s %8s %-5s %-4s %s S/N %s\n", str(d["name"]), human(num(d["size"])), str(d["tran"]), kind, + str(d["model"]), str(d["serial"])) + } + for _, cv := range toList(st["controllers"]) { + c, _ := cv.(obj) + fmt.Printf(" ctrl %s %s %s [%s]\n", str(c["slot"]), str(c["vendor"]), str(c["device"]), str(c["driver"])) + } + fmt.Println(" NICs:") + for _, nv := range toList(inv["network"]) { + n, _ := nv.(obj) + var ips []string + for _, ip := range toList(n["ipv4"]) { + ips = append(ips, str(ip)) + } + fmt.Printf(" %-10s %s %-10s %-10s link=%v fw=%s pci=%s %s\n", str(n["name"]), str(n["mac"]), str(n["driver"]), + str(n["speed"]), n["link"], str(n["firmware_version"]), str(n["pci"]), strings.Join(ips, ",")) + } + if hcas := toList(dig(inv, "infiniband", "hcas")); len(hcas) > 0 { + fmt.Println(" InfiniBand:") + for _, hv := range hcas { + h, _ := hv.(obj) + for _, pv := range toList(h["ports"]) { + p, _ := pv.(obj) + fmt.Printf(" %s port%s %s %s %s guid=%s fw=%s %s\n", str(h["name"]), str(p["port"]), str(p["link_layer"]), + str(p["state"]), str(p["rate"]), str(h["node_guid"]), str(h["fw_ver"]), str(h["board_id"])) + } + } + } + if g := toList(inv["gpus"]); len(g) > 0 { + fmt.Println(" GPUs/accelerators:") + for _, gv := range g { + x, _ := gv.(obj) + fmt.Printf(" %s %s %s [%s]\n", str(x["slot"]), str(x["vendor"]), str(x["device"]), str(x["driver"])) + } + } + if lldp, ok := inv["lldp"].(obj); ok && len(lldp) > 0 { + fmt.Println(" LLDP neighbors:") + for ifn, ents := range lldp { + for _, e := range asList(ents) { + eo, _ := e.(obj) + ch, _ := eo["chassis"].(obj) + name := "?" + for k := range ch { + name = k + } + fmt.Printf(" %s: %s port %s\n", ifn, name, str(dig(eo, "port", "id", "value"))) + } + } + } + b, _ := d["bmc"].(obj) + fmt.Printf(" BMC: source=%s address=%s user=%s configured=%v verified_oob=%v secret_stored=%v\n", str(b["source"]), + str(b["address"]), str(b["user"]), b["configured"], b["verified_oob"], d["has_bmc_secret"]) + if b["lan"] != nil { + lb, _ := json.Marshal(b["lan"]) + fmt.Printf(" lan: %s\n", lb) + } + if errs := toList(b["errors"]); len(errs) > 0 { + fmt.Printf(" errors: %v\n", errs) + } + if errs := toList(inv["errors"]); len(errs) > 0 { + fmt.Printf(" agent errors: %v\n", errs) + } + var evs []string + for _, e := range toList(d["events"]) { + evs = append(evs, str(dig(e, "event"))) + } + fmt.Printf(" events: %s\n", strings.Join(evs, "; ")) +} + +func asList(v any) []any { + if l, ok := v.([]any); ok { + return l + } + return []any{v} +} diff --git a/cmd/rollcall/config.go b/cmd/rollcall/config.go new file mode 100644 index 0000000..63fdade --- /dev/null +++ b/cmd/rollcall/config.go @@ -0,0 +1,48 @@ +package main + +import ( + "encoding/json" + "fmt" + "os" +) + +// Config is /etc/rollcall/config.json. +type Config struct { + Listen string `json:"listen"` // "10.213.77.1:80" + ImageDir string `json:"image_dir"` // live image files (vmlinuz, initrd.img, filesystem.squashfs), served under /rollcall/ + TFTPRoot string `json:"tftp_root"` // pxelinux.cfg/01- files are written here + DatabaseURL string `json:"database_url"` // postgres://user:pw@host/db + MasterKeyFile string `json:"master_key_file"` // 32-byte hex key for AES-GCM at-rest encryption + AdminToken string `json:"admin_token"` // bearer token for the management API / web UI + AgentToken string `json:"agent_token"` // X-Rollcall-Token for the agent endpoints ("" = open) + CommissionAppend string `json:"commission_append"` // kernel cmdline used in generated pxelinux files + IPMITool string `json:"ipmitool"` // path, default "ipmitool" + BMCHints []BMCHint `json:"bmc_hints"` // out-of-band BMC mapping for machines without in-band IPMI +} + +// BMCHint maps a machine MAC to a known BMC (virtual lab nodes). +type BMCHint struct { + MAC string `json:"mac"` + Address string `json:"address"` + Port int `json:"port"` + User string `json:"user"` + Password string `json:"password"` + Comment string `json:"comment,omitempty"` +} + +func loadConfig(path string) (*Config, error) { + b, err := os.ReadFile(path) + if err != nil { + return nil, err + } + c := &Config{Listen: "0.0.0.0:80", IPMITool: "ipmitool", MasterKeyFile: "/etc/rollcall/master.key"} + if err := json.Unmarshal(b, c); err != nil { + return nil, fmt.Errorf("%s: %w", path, err) + } + for _, k := range []struct{ v, n string }{{c.ImageDir, "image_dir"}, {c.TFTPRoot, "tftp_root"}, {c.DatabaseURL, "database_url"}, {c.AdminToken, "admin_token"}} { + if k.v == "" { + return nil, fmt.Errorf("%s: %s is required", path, k.n) + } + } + return c, nil +} diff --git a/cmd/rollcall/crypto.go b/cmd/rollcall/crypto.go new file mode 100644 index 0000000..dbd7128 --- /dev/null +++ b/cmd/rollcall/crypto.go @@ -0,0 +1,74 @@ +package main + +import ( + "crypto/aes" + "crypto/cipher" + "crypto/rand" + "encoding/hex" + "errors" + "fmt" + "os" + "strings" +) + +// Sealer encrypts BMC secrets at rest with AES-256-GCM. +type Sealer struct{ aead cipher.AEAD } + +// loadOrCreateKey reads a 32-byte hex key from path, creating it (0600) when absent. +func loadOrCreateKey(path string) (*Sealer, error) { + b, err := os.ReadFile(path) + if errors.Is(err, os.ErrNotExist) { + raw := make([]byte, 32) + if _, err := rand.Read(raw); err != nil { + return nil, err + } + if err := os.WriteFile(path, []byte(hex.EncodeToString(raw)+"\n"), 0o600); err != nil { + return nil, fmt.Errorf("create master key: %w", err) + } + b = []byte(hex.EncodeToString(raw)) + } else if err != nil { + return nil, err + } + key, err := hex.DecodeString(strings.TrimSpace(string(b))) + if err != nil || len(key) != 32 { + return nil, fmt.Errorf("%s: expected 32-byte hex key", path) + } + block, err := aes.NewCipher(key) + if err != nil { + return nil, err + } + aead, err := cipher.NewGCM(block) + if err != nil { + return nil, err + } + return &Sealer{aead: aead}, nil +} + +// Seal returns nonce||ciphertext; aad binds the secret to a machine id. +func (s *Sealer) Seal(plaintext, aad string) ([]byte, error) { + nonce := make([]byte, s.aead.NonceSize()) + if _, err := rand.Read(nonce); err != nil { + return nil, err + } + return append(nonce, s.aead.Seal(nil, nonce, []byte(plaintext), []byte(aad))...), nil +} + +func (s *Sealer) Open(blob []byte, aad string) (string, error) { + n := s.aead.NonceSize() + if len(blob) < n { + return "", errors.New("ciphertext too short") + } + pt, err := s.aead.Open(nil, blob[:n], blob[n:], []byte(aad)) + if err != nil { + return "", err + } + return string(pt), nil +} + +func randomToken(nbytes int) string { + b := make([]byte, nbytes) + if _, err := rand.Read(b); err != nil { + panic(err) + } + return hex.EncodeToString(b) +} diff --git a/cmd/rollcall/go.mod b/cmd/rollcall/go.mod new file mode 100644 index 0000000..7d7ee22 --- /dev/null +++ b/cmd/rollcall/go.mod @@ -0,0 +1,14 @@ +module rollcall + +go 1.24 + +require github.com/jackc/pgx/v5 v5.7.5 + +require ( + github.com/jackc/pgpassfile v1.0.0 // indirect + github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect + github.com/jackc/puddle/v2 v2.2.2 // indirect + golang.org/x/crypto v0.37.0 // indirect + golang.org/x/sync v0.13.0 // indirect + golang.org/x/text v0.24.0 // indirect +) diff --git a/cmd/rollcall/go.sum b/cmd/rollcall/go.sum new file mode 100644 index 0000000..85a678b --- /dev/null +++ b/cmd/rollcall/go.sum @@ -0,0 +1,28 @@ +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM= +github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg= +github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo= +github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM= +github.com/jackc/pgx/v5 v5.7.5 h1:JHGfMnQY+IEtGM63d+NGMjoRpysB2JBwDr5fsngwmJs= +github.com/jackc/pgx/v5 v5.7.5/go.mod h1:aruU7o91Tc2q2cFp5h4uP3f6ztExVpyVv88Xl/8Vl8M= +github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo= +github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= +github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= +golang.org/x/crypto v0.37.0 h1:kJNSjF/Xp7kU0iB2Z+9viTPMW4EqqsrywMXLJOOsXSE= +golang.org/x/crypto v0.37.0/go.mod h1:vg+k43peMZ0pUMhYmVAWysMK35e6ioLh3wB8ZCAfbVc= +golang.org/x/sync v0.13.0 h1:AauUjRAJ9OSnvULf/ARrrVywoJDy0YS2AwQ98I37610= +golang.org/x/sync v0.13.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA= +golang.org/x/text v0.24.0 h1:dd5Bzh4yt5KYA8f9CJHCP4FB4D51c2c6JvN37xJJkJ0= +golang.org/x/text v0.24.0/go.mod h1:L8rBsPeo2pSS+xqN0d5u2ikmjtmoJbDBT1b7nHvFCdU= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/cmd/rollcall/ipmi.go b/cmd/rollcall/ipmi.go new file mode 100644 index 0000000..53192a9 --- /dev/null +++ b/cmd/rollcall/ipmi.go @@ -0,0 +1,69 @@ +package main + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "os/exec" + "strconv" + "strings" + "time" +) + +// bmcCreds is what the server needs to talk to a machine's BMC. +type bmcCreds struct { + Address string + Port int + User string + Password string +} + +var powerActions = map[string][]string{ + "status": {"chassis", "power", "status"}, + "on": {"chassis", "power", "on"}, + "off": {"chassis", "power", "off"}, + "soft": {"chassis", "power", "soft"}, + "cycle": {"chassis", "power", "cycle"}, + "reset": {"chassis", "power", "reset"}, +} + +var bootDevices = map[string]bool{"pxe": true, "disk": true, "cdrom": true, "bios": true, "none": true, "safe": true} + +func (a *App) credsFor(m *Machine) (*bmcCreds, error) { + if m.BMC == nil || m.bmcSecret == nil { + return nil, fmt.Errorf("no BMC credentials stored") + } + var b struct { + Address string `json:"address"` + Port int `json:"port"` + User string `json:"user"` + } + if err := json.Unmarshal(m.BMC, &b); err != nil { + return nil, err + } + if b.Address == "" || b.User == "" { + return nil, fmt.Errorf("BMC record has no address/user") + } + pw, err := a.sealer.Open(m.bmcSecret, m.ID) + if err != nil { + return nil, fmt.Errorf("decrypt BMC secret: %w", err) + } + if b.Port == 0 { + b.Port = 623 + } + return &bmcCreds{Address: b.Address, Port: b.Port, User: b.User, Password: pw}, nil +} + +// ipmi runs ipmitool server-side; the password never leaves this process. +func (a *App) ipmi(ctx context.Context, c *bmcCreds, args ...string) (string, error) { + ctx, cancel := context.WithTimeout(ctx, 60*time.Second) + defer cancel() + full := append([]string{"-I", "lanplus", "-H", c.Address, "-p", strconv.Itoa(c.Port), "-U", c.User, "-E"}, args...) + cmd := exec.CommandContext(ctx, a.cfg.IPMITool, full...) + cmd.Env = []string{"IPMI_PASSWORD=" + c.Password, "PATH=/usr/sbin:/usr/bin:/sbin:/bin"} + var out bytes.Buffer + cmd.Stdout, cmd.Stderr = &out, &out + err := cmd.Run() + return strings.TrimSpace(out.String()), err +} diff --git a/cmd/rollcall/main.go b/cmd/rollcall/main.go new file mode 100644 index 0000000..a6b6044 --- /dev/null +++ b/cmd/rollcall/main.go @@ -0,0 +1,68 @@ +// rollcall: MAAS-like commissioning server, web UI and CLI in one binary. +// +// rollcall serve [-config /etc/rollcall/config.json] +// rollcall keygen print a random token (admin_token / agent_token) +// rollcall list | show ID [-json] | events ID +// rollcall power ID status|on|off|soft|cycle|reset +// rollcall bootdev ID pxe|disk|cdrom|bios|none +// rollcall recommission ID [-now] (-now: bootdev pxe + power cycle via BMC) +// rollcall delete ID +// +// Client commands read ROLLCALL_URL / ROLLCALL_TOKEN, falling back to the +// config file when it is readable (so root on the server needs no setup). +package main + +import ( + "flag" + "fmt" + "log" + "os" +) + +func usage() { + fmt.Fprintln(os.Stderr, `usage: + rollcall serve [-config PATH] + rollcall keygen + rollcall list + rollcall show ID [-json] + rollcall events ID + rollcall power ID status|on|off|soft|cycle|reset + rollcall bootdev ID pxe|disk|cdrom|bios|none + rollcall recommission ID [-now] + rollcall delete ID +env: ROLLCALL_URL (default from config listen), ROLLCALL_TOKEN, ROLLCALL_CONFIG`) + os.Exit(2) +} + +func main() { + log.SetFlags(log.Ldate | log.Ltime | log.Lmicroseconds) + if len(os.Args) < 2 { + usage() + } + cmd, args := os.Args[1], os.Args[2:] + switch cmd { + case "serve": + fs := flag.NewFlagSet("serve", flag.ExitOnError) + cfg := fs.String("config", envOr("ROLLCALL_CONFIG", "/etc/rollcall/config.json"), "config file") + _ = fs.Parse(args) + if err := serve(*cfg); err != nil { + log.Fatal(err) + } + case "keygen": + fmt.Println(randomToken(24)) + case "list", "show", "events", "power", "bootdev", "recommission", "delete": + if err := runClient(cmd, args); err != nil { + fmt.Fprintln(os.Stderr, "error:", err) + os.Exit(1) + } + default: + usage() + } +} + +func envOr(k, d string) string { + if v := os.Getenv(k); v != "" { + return v + } + return d +} diff --git a/cmd/rollcall/server.go b/cmd/rollcall/server.go new file mode 100644 index 0000000..b819dc5 --- /dev/null +++ b/cmd/rollcall/server.go @@ -0,0 +1,533 @@ +package main + +import ( + "context" + "crypto/subtle" + "embed" + "encoding/json" + "errors" + "fmt" + "io" + "io/fs" + "log" + "net/http" + "os" + "path/filepath" + "regexp" + "strings" + "time" +) + +//go:embed web +var webFS embed.FS + +type App struct { + cfg *Config + store *Store + sealer *Sealer +} + +type apiError struct { + Error string `json:"error"` +} + +func writeJSON(w http.ResponseWriter, code int, v any) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(code) + enc := json.NewEncoder(w) + enc.SetIndent("", " ") + _ = enc.Encode(v) +} + +func fail(w http.ResponseWriter, code int, err error) { + if errors.Is(err, errNotFound) { + code = http.StatusNotFound + } + writeJSON(w, code, apiError{err.Error()}) +} + +func tokenEqual(a, b string) bool { + return b != "" && subtle.ConstantTimeCompare([]byte(a), []byte(b)) == 1 +} + +func (a *App) isAdmin(r *http.Request) bool { + h := r.Header.Get("Authorization") + return strings.HasPrefix(h, "Bearer ") && tokenEqual(strings.TrimPrefix(h, "Bearer "), a.cfg.AdminToken) +} + +func (a *App) adminAuth(next http.HandlerFunc) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + if !a.isAdmin(r) { + w.Header().Set("WWW-Authenticate", `Bearer realm="rollcall"`) + fail(w, http.StatusUnauthorized, errors.New("admin token required")) + return + } + next(w, r) + } +} + +func (a *App) agentAuth(next http.HandlerFunc) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + if a.cfg.AgentToken != "" && !tokenEqual(r.Header.Get("X-Rollcall-Token"), a.cfg.AgentToken) && !a.isAdmin(r) { + fail(w, http.StatusUnauthorized, errors.New("agent token required")) + return + } + next(w, r) + } +} + +func logging(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + start := time.Now() + rw := &statusWriter{ResponseWriter: w, code: 200} + next.ServeHTTP(rw, r) + if !strings.HasPrefix(r.URL.Path, "/rollcall/") || rw.code >= 400 { + log.Printf("%s %s %s %d %s", r.RemoteAddr, r.Method, r.URL.RequestURI(), rw.code, time.Since(start).Round(time.Millisecond)) + } + }) +} + +type statusWriter struct { + http.ResponseWriter + code int +} + +func (s *statusWriter) WriteHeader(c int) { s.code = c; s.ResponseWriter.WriteHeader(c) } + +func (a *App) routes() http.Handler { + mux := http.NewServeMux() + // agent endpoints + mux.HandleFunc("POST /api/v1/commission", a.agentAuth(a.handleCommission)) + mux.HandleFunc("GET /api/v1/bmc-hint", a.agentAuth(a.handleHint)) + // management endpoints + mux.HandleFunc("GET /api/v1/machines", a.adminAuth(a.handleList)) + mux.HandleFunc("GET /api/v1/machines/{id}", a.adminAuth(a.handleGet)) + mux.HandleFunc("DELETE /api/v1/machines/{id}", a.adminAuth(a.handleDelete)) + mux.HandleFunc("POST /api/v1/machines/{id}/recommission", a.adminAuth(a.handleRecommission)) + mux.HandleFunc("GET /api/v1/machines/{id}/power", a.adminAuth(a.handlePower)) + mux.HandleFunc("POST /api/v1/machines/{id}/power", a.adminAuth(a.handlePower)) + mux.HandleFunc("POST /api/v1/machines/{id}/bootdev", a.adminAuth(a.handleBootdev)) + mux.HandleFunc("GET /api/v1/whoami", a.adminAuth(func(w http.ResponseWriter, r *http.Request) { + writeJSON(w, 200, map[string]any{"ok": true}) + })) + // image files for the live boot (no auth: the kernel/initrd/squashfs must be fetchable) + imgDir := a.cfg.ImageDir + mux.Handle("GET /rollcall/", http.StripPrefix("/rollcall/", http.FileServer(http.Dir(imgDir)))) + // embedded web UI + sub, _ := fs.Sub(webFS, "web") + mux.Handle("GET /", http.FileServer(http.FS(sub))) + return logging(mux) +} + +// ---------------------------------------------------------------- helpers over inventory JSON + +type obj = map[string]any + +func dig(m any, path ...string) any { + cur := m + for _, p := range path { + mm, ok := cur.(obj) + if !ok { + return nil + } + cur = mm[p] + } + return cur +} + +func str(v any) string { + switch t := v.(type) { + case nil: + return "" + case string: + return t + case float64: + return fmt.Sprintf("%v", t) + default: + return fmt.Sprintf("%v", t) + } +} + +func strPtr(v any) *string { + s := str(v) + if s == "" { + return nil + } + return &s +} + +func listLen(v any) int { + if l, ok := v.([]any); ok { + return len(l) + } + return 0 +} + +func normMAC(s string) string { return strings.ToLower(strings.ReplaceAll(s, "-", ":")) } + +var nonAlnum = regexp.MustCompile(`[^a-z0-9]`) + +func machineID(uuid, serial string, macs []string) string { + u := strings.ToLower(uuid) + if u != "" && u != "00000000-0000-0000-0000-000000000000" && !strings.Contains(u, "not ") { + u = strings.ReplaceAll(u, "-", "") + if len(u) > 12 { + u = u[:12] + } + return "m-" + u + } + if s := nonAlnum.ReplaceAllString(strings.ToLower(serial), ""); s != "" && !strings.Contains(strings.ToLower(serial), "not ") { + if len(s) > 16 { + s = s[:16] + } + return "m-" + s + } + if len(macs) > 0 { + return "m-" + strings.ReplaceAll(macs[0], ":", "") + } + return "m-" + randomToken(6) +} + +// Summary is the list-view projection of a machine (no secrets, no full inventory). +type Summary struct { + ID string `json:"id"` + State string `json:"state"` + Hostname *string `json:"hostname"` + Serial *string `json:"serial"` + UUID *string `json:"uuid"` + Manufacturer *string `json:"manufacturer"` + Product *string `json:"product"` + FirstSeen time.Time `json:"first_seen"` + LastSeen *time.Time `json:"last_seen"` + MACs []string `json:"macs"` + CPU string `json:"cpu"` + CPUs string `json:"cpus"` + MemoryBytes float64 `json:"memory_bytes"` + Disks float64 `json:"disks"` + StorageBytes float64 `json:"storage_bytes"` + NICs int `json:"nics"` + IBHCAs int `json:"ib_hcas"` + GPUs int `json:"gpus"` + BMC *obj `json:"bmc"` +} + +func num(v any) float64 { + f, _ := v.(float64) + return f +} + +func summarize(m *Machine) Summary { + var inv obj + _ = json.Unmarshal(m.Inventory, &inv) + s := Summary{ID: m.ID, State: m.State, Hostname: m.Hostname, Serial: m.Serial, UUID: m.UUID, + Manufacturer: m.Manufacturer, Product: m.Product, FirstSeen: m.FirstSeen, LastSeen: m.LastSeen, MACs: m.MACs, + CPU: str(dig(inv, "cpu", "Model name")), CPUs: str(dig(inv, "cpu", "CPU(s)")), + MemoryBytes: num(dig(inv, "memory", "total_bytes")), Disks: num(dig(inv, "storage", "disk_count")), + StorageBytes: num(dig(inv, "storage", "total_bytes")), NICs: listLen(inv["network"]), + IBHCAs: listLen(dig(inv, "infiniband", "hcas")), GPUs: listLen(inv["gpus"])} + if m.BMC != nil { + var b obj + if json.Unmarshal(m.BMC, &b) == nil { + s.BMC = &obj{"source": b["source"], "address": b["address"], "user": b["user"], + "configured": b["configured"], "verified_oob": b["verified_oob"]} + } + } + return s +} + +// ---------------------------------------------------------------- pxelinux per-MAC files + +const commissionedTmpl = `# generated by rollcall %s +# machine %s %s %s serial=%s +DEFAULT menu.c32 +PROMPT 0 +TIMEOUT 150 +MENU TITLE %s - commissioned (%s %s, S/N %s) + +LABEL poweroff + MENU LABEL Commissioned: power off (default in 15s) + MENU DEFAULT + KERNEL poweroff.c32 + +LABEL recommission + MENU LABEL Re-run commissioning + KERNEL rollcall/vmlinuz + INITRD rollcall/initrd.img + APPEND %s + +LABEL local + MENU LABEL Boot from local disk + LOCALBOOT 0 +` + +func (a *App) pxePath(mac string) string { + return filepath.Join(a.cfg.TFTPRoot, "pxelinux.cfg", "01-"+strings.ReplaceAll(normMAC(mac), ":", "-")) +} + +func deref(p *string) string { + if p == nil { + return "" + } + return *p +} + +func (a *App) writePXE(m *Machine) error { + man, prod, ser := deref(m.Manufacturer), deref(m.Product), deref(m.Serial) + body := fmt.Sprintf(commissionedTmpl, time.Now().Format(time.RFC3339), m.ID, man, prod, ser, + m.ID, man, prod, ser, a.cfg.CommissionAppend) + for _, mac := range m.MACs { + p := a.pxePath(mac) + if err := os.WriteFile(p+".tmp", []byte(body), 0o644); err != nil { + return err + } + if err := os.Rename(p+".tmp", p); err != nil { + return err + } + log.Printf("wrote %s", p) + } + return nil +} + +func (a *App) removePXE(m *Machine) { + for _, mac := range m.MACs { + if err := os.Remove(a.pxePath(mac)); err == nil { + log.Printf("removed %s", a.pxePath(mac)) + } + } +} + +// ---------------------------------------------------------------- handlers + +func (a *App) handleCommission(w http.ResponseWriter, r *http.Request) { + body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 32<<20)) + if err != nil { + fail(w, 400, err) + return + } + var inv obj + if err := json.Unmarshal(body, &inv); err != nil { + fail(w, 400, fmt.Errorf("bad json: %w", err)) + return + } + ident, _ := inv["identity"].(obj) + var macs []string + if l, ok := ident["macs"].([]any); ok { + for _, v := range l { + if s := normMAC(str(v)); s != "" { + macs = append(macs, s) + } + } + } + if boot := normMAC(str(ident["boot_mac"])); boot != "" { + for i, mac := range macs { + if mac == boot && i != 0 { + macs = append([]string{boot}, append(macs[:i], macs[i+1:]...)...) + break + } + } + } + m := &Machine{ + ID: machineID(str(ident["uuid"]), str(ident["serial"]), macs), + UUID: strPtr(ident["uuid"]), + Serial: strPtr(ident["serial"]), + Hostname: strPtr(inv["hostname"]), + Manufacturer: strPtr(dig(inv, "system", "system-manufacturer")), + Product: strPtr(dig(inv, "system", "system-product-name")), + MACs: macs, + } + // Pull the BMC password out of the payload before anything is persisted. + var secret []byte + detail := "bmc=none" + if bmc, ok := inv["bmc"].(obj); ok { + if pw := str(bmc["password"]); pw != "" { + secret, err = a.sealer.Seal(pw, m.ID) + if err != nil { + fail(w, 500, err) + return + } + bmc["password"] = "" + } + m.BMC, _ = json.Marshal(bmc) + detail = fmt.Sprintf("bmc=%s:%s configured=%v", str(bmc["source"]), str(bmc["address"]), bmc["configured"]) + } + m.Inventory, _ = json.Marshal(inv) + + isNew, err := a.store.UpsertCommissioned(r.Context(), m, secret) + if err != nil { + fail(w, 500, err) + return + } + _ = a.store.AddEvent(r.Context(), m.ID, "commissioned", detail) + if err := a.writePXE(m); err != nil { + log.Printf("pxe write failed for %s: %v", m.ID, err) + _ = a.store.AddEvent(r.Context(), m.ID, "pxe-write-failed", err.Error()) + } + log.Printf("commissioned %s new=%v (%s %s, serial %s, %d macs) %s", m.ID, isNew, deref(m.Manufacturer), + deref(m.Product), deref(m.Serial), len(macs), detail) + writeJSON(w, 200, map[string]any{"id": m.ID, "state": "ready", "new": isNew}) +} + +func (a *App) handleHint(w http.ResponseWriter, r *http.Request) { + want := map[string]bool{} + for _, m := range strings.Split(r.URL.Query().Get("macs"), ",") { + want[normMAC(m)] = true + } + for _, h := range a.cfg.BMCHints { + if want[normMAC(h.MAC)] { + port := h.Port + if port == 0 { + port = 623 + } + writeJSON(w, 200, obj{"address": h.Address, "port": port, "user": h.User, "password": h.Password}) + return + } + } + writeJSON(w, 200, obj{}) +} + +func (a *App) handleList(w http.ResponseWriter, r *http.Request) { + ms, err := a.store.List(r.Context()) + if err != nil { + fail(w, 500, err) + return + } + out := make([]Summary, 0, len(ms)) + for _, m := range ms { + out = append(out, summarize(m)) + } + writeJSON(w, 200, out) +} + +func (a *App) handleGet(w http.ResponseWriter, r *http.Request) { + m, err := a.store.Get(r.Context(), r.PathValue("id")) + if err != nil { + fail(w, 500, err) + return + } + evs, _ := a.store.Events(r.Context(), m.ID) + var bmc, inv any + _ = json.Unmarshal(m.BMC, &bmc) + _ = json.Unmarshal(m.Inventory, &inv) + writeJSON(w, 200, obj{"summary": summarize(m), "bmc": bmc, "has_bmc_secret": m.bmcSecret != nil, + "inventory": inv, "events": evs}) +} + +func (a *App) handleDelete(w http.ResponseWriter, r *http.Request) { + m, err := a.store.Get(r.Context(), r.PathValue("id")) + if err != nil { + fail(w, 500, err) + return + } + a.removePXE(m) + if err := a.store.Delete(r.Context(), m.ID); err != nil { + fail(w, 500, err) + return + } + log.Printf("deleted %s", m.ID) + writeJSON(w, 200, obj{"deleted": m.ID}) +} + +func (a *App) handleRecommission(w http.ResponseWriter, r *http.Request) { + m, err := a.store.Get(r.Context(), r.PathValue("id")) + if err != nil { + fail(w, 500, err) + return + } + a.removePXE(m) + _ = a.store.SetState(r.Context(), m.ID, "recommission") + _ = a.store.AddEvent(r.Context(), m.ID, "recommission requested", "") + // optional: power-cycle it right away when the caller asks (?now=1) + out := obj{"id": m.ID, "state": "recommission"} + if r.URL.Query().Get("now") == "1" { + if c, err := a.credsFor(m); err == nil { + o, ierr := a.ipmi(r.Context(), c, "chassis", "bootdev", "pxe") + o2, ierr2 := a.ipmi(r.Context(), c, "chassis", "power", "cycle") + if ierr2 != nil && strings.Contains(o2, "off") { // cycle fails when off; power on instead + o2, ierr2 = a.ipmi(r.Context(), c, "chassis", "power", "on") + } + out["ipmi"] = strings.TrimSpace(o + "\n" + o2) + out["ipmi_ok"] = ierr == nil && ierr2 == nil + } else { + out["ipmi"] = err.Error() + } + } + writeJSON(w, 200, out) +} + +func (a *App) handlePower(w http.ResponseWriter, r *http.Request) { + m, err := a.store.Get(r.Context(), r.PathValue("id")) + if err != nil { + fail(w, 500, err) + return + } + action := "status" + if r.Method == http.MethodPost { + var req struct { + Action string `json:"action"` + } + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + fail(w, 400, err) + return + } + action = req.Action + } + args, ok := powerActions[action] + if !ok { + fail(w, 400, fmt.Errorf("unknown power action %q", action)) + return + } + c, err := a.credsFor(m) + if err != nil { + fail(w, 409, err) + return + } + out, err := a.ipmi(r.Context(), c, args...) + if action != "status" { + _ = a.store.AddEvent(r.Context(), m.ID, "power "+action, out) + } + writeJSON(w, 200, obj{"id": m.ID, "action": action, "ok": err == nil, "output": out}) +} + +func (a *App) handleBootdev(w http.ResponseWriter, r *http.Request) { + m, err := a.store.Get(r.Context(), r.PathValue("id")) + if err != nil { + fail(w, 500, err) + return + } + var req struct { + Device string `json:"device"` + } + if err := json.NewDecoder(r.Body).Decode(&req); err != nil || !bootDevices[req.Device] { + fail(w, 400, fmt.Errorf("device must be one of pxe, disk, cdrom, bios, none, safe")) + return + } + c, err := a.credsFor(m) + if err != nil { + fail(w, 409, err) + return + } + out, err := a.ipmi(r.Context(), c, "chassis", "bootdev", req.Device) + _ = a.store.AddEvent(r.Context(), m.ID, "bootdev "+req.Device, out) + writeJSON(w, 200, obj{"id": m.ID, "device": req.Device, "ok": err == nil, "output": out}) +} + +func serve(cfgPath string) error { + cfg, err := loadConfig(cfgPath) + if err != nil { + return err + } + sealer, err := loadOrCreateKey(cfg.MasterKeyFile) + if err != nil { + return err + } + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + store, err := openStore(ctx, cfg.DatabaseURL) + cancel() + if err != nil { + return fmt.Errorf("database: %w", err) + } + app := &App{cfg: cfg, store: store, sealer: sealer} + srv := &http.Server{Addr: cfg.Listen, Handler: app.routes(), ReadHeaderTimeout: 10 * time.Second} + log.Printf("rollcall listening on %s (image=%s tftp=%s agent_token=%v)", cfg.Listen, cfg.ImageDir, cfg.TFTPRoot, cfg.AgentToken != "") + return srv.ListenAndServe() +} diff --git a/cmd/rollcall/store.go b/cmd/rollcall/store.go new file mode 100644 index 0000000..05b9956 --- /dev/null +++ b/cmd/rollcall/store.go @@ -0,0 +1,203 @@ +package main + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "time" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgxpool" +) + +const schema = ` +CREATE TABLE IF NOT EXISTS machines ( + id text PRIMARY KEY, + uuid text, + serial text, + hostname text, + manufacturer text, + product text, + state text NOT NULL DEFAULT 'new', + first_seen timestamptz NOT NULL DEFAULT now(), + last_seen timestamptz, + commissioned_at timestamptz, + macs jsonb NOT NULL DEFAULT '[]', + bmc jsonb, + bmc_secret bytea, + inventory jsonb +); +CREATE INDEX IF NOT EXISTS machines_uuid ON machines(uuid); +CREATE INDEX IF NOT EXISTS machines_macs ON machines USING gin(macs); +CREATE TABLE IF NOT EXISTS events ( + id bigserial PRIMARY KEY, + ts timestamptz NOT NULL DEFAULT now(), + machine_id text NOT NULL, + event text NOT NULL, + detail text +); +CREATE INDEX IF NOT EXISTS events_machine ON events(machine_id, ts); +` + +var errNotFound = errors.New("not found") + +type Store struct{ pool *pgxpool.Pool } + +type Machine struct { + ID string `json:"id"` + UUID *string `json:"uuid"` + Serial *string `json:"serial"` + Hostname *string `json:"hostname"` + Manufacturer *string `json:"manufacturer"` + Product *string `json:"product"` + State string `json:"state"` + FirstSeen time.Time `json:"first_seen"` + LastSeen *time.Time `json:"last_seen"` + CommissionedAt *time.Time `json:"commissioned_at"` + MACs []string `json:"macs"` + BMC json.RawMessage `json:"bmc,omitempty"` + Inventory json.RawMessage `json:"inventory,omitempty"` + bmcSecret []byte +} + +type Event struct { + TS time.Time `json:"ts"` + Event string `json:"event"` + Detail string `json:"detail"` +} + +func openStore(ctx context.Context, url string) (*Store, error) { + pool, err := pgxpool.New(ctx, url) + if err != nil { + return nil, err + } + if _, err := pool.Exec(ctx, schema); err != nil { + return nil, fmt.Errorf("schema: %w", err) + } + return &Store{pool: pool}, nil +} + +const machineCols = `id, uuid, serial, hostname, manufacturer, product, state, first_seen, last_seen, + commissioned_at, macs, bmc, bmc_secret, inventory` + +func scanMachine(row pgx.Row, full bool) (*Machine, error) { + m := &Machine{} + var macs []byte + err := row.Scan(&m.ID, &m.UUID, &m.Serial, &m.Hostname, &m.Manufacturer, &m.Product, &m.State, + &m.FirstSeen, &m.LastSeen, &m.CommissionedAt, &macs, &m.BMC, &m.bmcSecret, &m.Inventory) + if errors.Is(err, pgx.ErrNoRows) { + return nil, errNotFound + } + if err != nil { + return nil, err + } + _ = json.Unmarshal(macs, &m.MACs) + if !full { + m.Inventory = nil + } + return m, nil +} + +func (s *Store) List(ctx context.Context) ([]*Machine, error) { + rows, err := s.pool.Query(ctx, `SELECT `+machineCols+` FROM machines ORDER BY first_seen`) + if err != nil { + return nil, err + } + defer rows.Close() + var out []*Machine + for rows.Next() { + m, err := scanMachine(rows, true) + if err != nil { + return nil, err + } + out = append(out, m) + } + if out == nil { + out = []*Machine{} + } + return out, rows.Err() +} + +func (s *Store) Get(ctx context.Context, id string) (*Machine, error) { + return scanMachine(s.pool.QueryRow(ctx, `SELECT `+machineCols+` FROM machines WHERE id=$1`, id), true) +} + +func (s *Store) Events(ctx context.Context, id string) ([]Event, error) { + rows, err := s.pool.Query(ctx, `SELECT ts, event, coalesce(detail,'') FROM events WHERE machine_id=$1 ORDER BY ts`, id) + if err != nil { + return nil, err + } + defer rows.Close() + evs := []Event{} + for rows.Next() { + var e Event + if err := rows.Scan(&e.TS, &e.Event, &e.Detail); err != nil { + return nil, err + } + evs = append(evs, e) + } + return evs, rows.Err() +} + +func (s *Store) AddEvent(ctx context.Context, id, event, detail string) error { + _, err := s.pool.Exec(ctx, `INSERT INTO events(machine_id, event, detail) VALUES ($1,$2,$3)`, id, event, detail) + return err +} + +// UpsertCommissioned stores a fresh inventory; returns true when the machine was new. +func (s *Store) UpsertCommissioned(ctx context.Context, m *Machine, secret []byte) (bool, error) { + tx, err := s.pool.Begin(ctx) + if err != nil { + return false, err + } + defer tx.Rollback(ctx) + macs, _ := json.Marshal(m.MACs) + tag, err := tx.Exec(ctx, `INSERT INTO machines(id, state) VALUES ($1,'new') ON CONFLICT (id) DO NOTHING`, m.ID) + if err != nil { + return false, err + } + isNew := tag.RowsAffected() == 1 + if isNew { + if _, err := tx.Exec(ctx, `INSERT INTO events(machine_id, event) VALUES ($1,'enlisted')`, m.ID); err != nil { + return false, err + } + } + _, err = tx.Exec(ctx, `UPDATE machines SET uuid=$2, serial=$3, hostname=$4, manufacturer=$5, product=$6, + state='ready', last_seen=now(), commissioned_at=now(), macs=$7, bmc=$8, bmc_secret=$9, inventory=$10 WHERE id=$1`, + m.ID, m.UUID, m.Serial, m.Hostname, m.Manufacturer, m.Product, macs, m.BMC, secret, m.Inventory) + if err != nil { + return false, err + } + return isNew, tx.Commit(ctx) +} + +func (s *Store) SetState(ctx context.Context, id, state string) error { + tag, err := s.pool.Exec(ctx, `UPDATE machines SET state=$2 WHERE id=$1`, id, state) + if err != nil { + return err + } + if tag.RowsAffected() == 0 { + return errNotFound + } + return nil +} + +func (s *Store) Delete(ctx context.Context, id string) error { + tx, err := s.pool.Begin(ctx) + if err != nil { + return err + } + defer tx.Rollback(ctx) + if _, err := tx.Exec(ctx, `DELETE FROM events WHERE machine_id=$1`, id); err != nil { + return err + } + tag, err := tx.Exec(ctx, `DELETE FROM machines WHERE id=$1`, id) + if err != nil { + return err + } + if tag.RowsAffected() == 0 { + return errNotFound + } + return tx.Commit(ctx) +} diff --git a/cmd/rollcall/web/app.js b/cmd/rollcall/web/app.js new file mode 100644 index 0000000..1043fa1 --- /dev/null +++ b/cmd/rollcall/web/app.js @@ -0,0 +1,158 @@ +'use strict'; +const $ = (s, el = document) => el.querySelector(s); +const esc = s => String(s ?? '').replace(/[&<>"']/g, c => ({'&':'&','<':'<','>':'>','"':'"',"'":'''}[c])); +const human = n => { n = Number(n || 0); const u = ['B','K','M','G','T','P']; let i = 0; while (n >= 1024 && i < u.length - 1) { n /= 1024; i++; } return (i <= 2 ? n.toFixed(0) : n.toFixed(1)) + u[i]; }; +const ago = ts => { if (!ts) return '-'; const s = (Date.now() - new Date(ts)) / 1000; if (s < 90) return `${s | 0}s ago`; if (s < 5400) return `${(s / 60) | 0}m ago`; if (s < 172800) return `${(s / 3600) | 0}h ago`; return `${(s / 86400) | 0}d ago`; }; + +let token = ''; +try { token = localStorage.getItem('rollcall.token') || ''; } catch (e) { /* storage unavailable */ } +let machines = []; +let current = null; +let timer = null; + +async function api(method, path, body) { + const r = await fetch(path, { method, headers: { 'Authorization': 'Bearer ' + token, 'Content-Type': 'application/json' }, body: body ? JSON.stringify(body) : undefined }); + if (r.status === 401) { askToken(); throw new Error('unauthorized'); } + const data = await r.json().catch(() => ({})); + if (!r.ok) throw new Error(data.error || r.statusText); + return data; +} + +function askToken() { + const dlg = $('#login'); + if (!dlg.open) dlg.showModal(); +} +$('#login form').addEventListener('submit', () => { + token = $('#token').value.trim(); + try { localStorage.setItem('rollcall.token', token); } catch (e) { /* ignore */ } + $('#token').value = ''; + loadList(); +}); +$('#logout').onclick = () => { token = ''; try { localStorage.removeItem('rollcall.token'); } catch (e) { /* ignore */ } askToken(); }; +$('#refresh').onclick = () => current ? loadDetail(current) : loadList(); +$('#filter').oninput = renderList; +$('#back').onclick = () => { current = null; $('#detail').hidden = true; $('#list').hidden = false; loadList(); }; + +function setStatus(msg, bad) { const el = $('#status'); el.textContent = msg; el.className = bad ? 'bad' : 'muted'; } + +async function loadList() { + if (!token) return askToken(); + try { + machines = await api('GET', '/api/v1/machines'); + setStatus(`updated ${new Date().toLocaleTimeString()}`); + renderList(); + } catch (e) { if (e.message !== 'unauthorized') setStatus(e.message, true); } +} + +function renderList() { + const q = $('#filter').value.toLowerCase(); + const rows = machines.filter(m => !q || JSON.stringify([m.id, m.state, m.manufacturer, m.product, m.serial, m.macs, m.hostname]).toLowerCase().includes(q)); + $('#count').textContent = `${rows.length} / ${machines.length}`; + $('#empty').hidden = machines.length > 0; + $('#machines tbody').innerHTML = rows.map(m => { + const b = m.bmc; + const bmc = b ? `${esc(b.address || '-')} ${esc(b.source)}` : '-'; + return ` + ${esc(m.id)}${esc(m.state)} + ${esc((m.manufacturer || '') + ' ' + (m.product || ''))}${esc(m.serial || '')} + ${esc(m.cpus)}${human(m.memory_bytes)}${m.disks | 0} + ${human(m.storage_bytes)}${m.nics}${m.ib_hcas}${m.gpus} + ${bmc}${ago(m.last_seen)}`; + }).join(''); + for (const tr of $('#machines tbody').children) tr.onclick = () => loadDetail(tr.dataset.id); +} + +function card(title, inner, wide) { return `

${esc(title)}

${inner}
`; } +function kv(pairs) { return '
' + pairs.filter(([, v]) => v !== undefined && v !== null && v !== '').map(([k, v]) => `
${esc(k)}
${v}
`).join('') + '
'; } +function table(cols, rows) { return rows.length ? `${cols.map(c => ``).join('')}${rows.map(r => `${r.map(c => ``).join('')}`).join('')}
${esc(c)}
${c ?? ''}
` : 'none'; } + +async function loadDetail(id) { + try { + const d = await api('GET', '/api/v1/machines/' + encodeURIComponent(id)); + current = id; + renderDetail(d); + $('#list').hidden = true; $('#detail').hidden = false; + } catch (e) { if (e.message !== 'unauthorized') setStatus(e.message, true); } +} + +function renderDetail(d) { + const s = d.summary, inv = d.inventory || {}, sys = inv.system || {}, cpu = inv.cpu || {}, mem = inv.memory || {}, st = inv.storage || {}, b = d.bmc || {}; + $('#d-title').textContent = `${s.id} ${sys['system-manufacturer'] || ''} ${sys['system-product-name'] || ''}`; + $('#d-state').textContent = s.state; $('#d-state').className = 'pill ' + s.state; + $('#action-out').textContent = ''; + const cards = []; + cards.push(card('System', kv([ + ['Manufacturer', esc(sys['system-manufacturer'])], ['Product', esc(sys['system-product-name'])], ['Serial', esc(sys['system-serial-number'])], + ['UUID', `${esc(sys['system-uuid'])}`], ['SKU', esc(sys['system-sku-number'])], + ['Board', esc([sys['baseboard-manufacturer'], sys['baseboard-product-name'], sys['baseboard-serial-number']].filter(Boolean).join(' / '))], + ['Chassis', esc([sys['chassis-type'], sys['chassis-serial-number'], sys['chassis-asset-tag']].filter(Boolean).join(' / '))], + ['BIOS', esc([sys['bios-vendor'], sys['bios-version'], sys['bios-release-date']].filter(Boolean).join(' '))], + ['Kernel', esc((inv.kernel || '').split(' (')[0])], ['TPM', esc((inv.tpm || []).join(', ') || 'none')], + ['Hostname', esc(s.hostname)], ['First seen', esc(new Date(s.first_seen).toLocaleString())], ['Last seen', esc(s.last_seen ? new Date(s.last_seen).toLocaleString() : '')], + ['Agent run', esc(inv.duration_s ? inv.duration_s + ' s' : '')], + ]))); + cards.push(card('CPU', kv([ + ['Model', esc(cpu['Model name'])], ['Sockets', esc(cpu['Socket(s)'])], ['Cores / socket', esc(cpu['Core(s) per socket'])], + ['Threads / core', esc(cpu['Thread(s) per core'])], ['Logical CPUs', esc(cpu['CPU(s)'])], ['NUMA nodes', esc(cpu['NUMA node(s)'])], + ['Max MHz', esc(cpu['CPU max MHz'])], ['L3', esc(cpu['L3 cache'])], ['Virtualization', esc(cpu['Virtualization'])], + ['Features', esc((cpu.features || []).join(' '))], + ]))); + cards.push(card(`Memory ${human(mem.total_bytes)} · ${mem.dimms_populated ?? '?'}/${mem.dimm_slots ?? '?'} DIMMs`, + table(['Slot', 'Size', 'Type', 'Speed', 'Vendor', 'Part', 'Serial'], + (mem.dimms || []).filter(x => x.populated).map(x => [esc(x.locator), esc(x.size), esc(x.type), esc(x.configured_speed || x.speed), esc(x.manufacturer), `${esc(x.part_number)}`, `${esc(x.serial)}`])))); + cards.push(card(`Storage · ${st.disk_count ?? 0} disks · ${human(st.total_bytes)}`, + table(['Dev', 'Size', 'Bus', 'Kind', 'Model', 'Serial', 'FW', 'Health'], + (st.disks || []).map(x => { const sm = (st.smart || {})[x.name] || {}; return [esc(x.name), human(x.size), esc(x.tran), x.rota === false ? 'ssd' : 'hdd', esc(x.model), `${esc(x.serial)}`, esc(sm.firmware), sm.healthy === true ? 'ok' : sm.healthy === false ? 'FAIL' : '']; })) + + (st.controllers && st.controllers.length ? '

Controllers

' + table(['PCI', 'Vendor', 'Device', 'Driver'], st.controllers.map(c => [`${esc(c.slot)}`, esc(c.vendor), esc(c.device), esc(c.driver)])) : ''), true)); + cards.push(card(`Network · ${(inv.network || []).length} NICs`, + table(['If', 'MAC', 'Driver', 'FW', 'Speed', 'Link', 'PCI', 'NUMA', 'VFs', 'IPv4'], + (inv.network || []).map(n => [esc(n.name), `${esc(n.mac)}`, esc(n.driver), esc(n.firmware_version), esc(n.speed), n.link === true ? 'up' : n.link === false ? 'down' : '', `${esc(n.pci)}`, esc(n.numa_node), esc(n.sriov_totalvfs), esc((n.ipv4 || []).join(' '))])), true)); + const hcas = (inv.infiniband || {}).hcas || []; + if (hcas.length) cards.push(card(`InfiniBand · ${hcas.length} HCAs`, + table(['HCA', 'Port', 'Layer', 'State', 'Rate', 'LID', 'Node GUID', 'FW', 'Board', 'PCI'], + hcas.flatMap(h => (h.ports || []).map(p => [esc(h.name), esc(p.port), esc(p.link_layer), esc(p.state), esc(p.rate), esc(p.lid), `${esc(h.node_guid)}`, esc(h.fw_ver), esc(h.board_id), `${esc(h.pci)}`]))), true)); + if ((inv.gpus || []).length) cards.push(card('GPUs / accelerators', table(['PCI', 'Vendor', 'Device', 'Driver'], inv.gpus.map(g => [`${esc(g.slot)}`, esc(g.vendor), esc(g.device), esc(g.driver)])))); + if (inv.lldp && Object.keys(inv.lldp).length) { + const rows = []; + for (const [ifn, ents] of Object.entries(inv.lldp)) for (const e of (Array.isArray(ents) ? ents : [ents])) { + const ch = e.chassis || {}; const name = Object.keys(ch)[0] || '?'; const c = ch[name] || {}; + rows.push([esc(ifn), esc(name), esc((c.descr || '').slice(0, 60)), esc((e.port || {}).id && (e.port.id.value || e.port.id)), esc((e.port || {}).descr), esc((e.vlan || {})['vlan-id'] || (e.vlan || {}).value)]); + } + cards.push(card('LLDP neighbors (switch ports)', table(['If', 'Switch', 'Descr', 'Port', 'Port descr', 'VLAN'], rows), true)); + } + cards.push(card('BMC', kv([ + ['Source', esc(b.source)], ['Address', `${esc(b.address)}${b.port && b.port !== 623 ? ':' + esc(b.port) : ''}`], ['MAC', `${esc(b.mac)}`], + ['User', esc(b.user)], ['Password', d.has_bmc_secret ? 'stored encrypted on server' : 'none'], + ['Configured', b.configured ? 'yes' : 'no'], ['Verified OOB', b.verified_oob === true ? 'yes' : b.verified_oob === false ? 'no' : ''], + ['Channel', esc(b.channel)], ['LAN', b.lan ? `${esc(JSON.stringify(b.lan))}` : ''], + ['Firmware', esc((b.mc_info || {})['Firmware Revision'])], ['Manufacturer', esc((b.mc_info || {})['Manufacturer Name'])], + ['Errors', (b.errors || []).length ? `${esc(b.errors.join('; '))}` : ''], + ]) + ((b.users_before || []).length ? '

User table before commissioning

' + table(['ID', 'Name', 'Priv', 'IPMI'], b.users_before.map(u => [u.id, esc(u.name), esc(u.priv), u.ipmi_msg ? 'yes' : 'no'])) : ''))); + cards.push(card('Events', table(['Time', 'Event', 'Detail'], (d.events || []).slice().reverse().map(e => [esc(new Date(e.ts).toLocaleString()), esc(e.event), `${esc(e.detail)}`])))); + if ((inv.pci || []).length) cards.push(card(`PCI devices · ${inv.pci.length}`, table(['Slot', 'Class', 'Vendor', 'Device', 'Driver'], inv.pci.map(p => [`${esc(p.slot)}`, esc(p.class), esc(p.vendor), esc(p.device), esc(p.driver)])), true)); + if ((inv.errors || []).length) cards.push(card('Agent errors', `${esc(inv.errors.join('\n'))}`, true)); + $('#d-body').innerHTML = cards.join(''); + $('#d-json').textContent = JSON.stringify(d, null, 1); +} + +async function action(fn, label) { + const btns = document.querySelectorAll('.actions button'); + btns.forEach(b => b.disabled = true); + $('#action-out').textContent = label + ' …'; + try { const r = await fn(); $('#action-out').textContent = r; } + catch (e) { $('#action-out').textContent = 'error: ' + e.message; } + finally { btns.forEach(b => b.disabled = false); } +} +document.querySelectorAll('[data-power]').forEach(b => b.onclick = () => action(async () => { + const r = await api('POST', `/api/v1/machines/${current}/power`, { action: b.dataset.power }); return r.output || (r.ok ? 'ok' : 'failed'); +}, 'power ' + b.dataset.power)); +document.querySelectorAll('[data-bootdev]').forEach(b => b.onclick = () => action(async () => { + const r = await api('POST', `/api/v1/machines/${current}/bootdev`, { device: b.dataset.bootdev }); return r.output || (r.ok ? 'ok' : 'failed'); +}, 'bootdev ' + b.dataset.bootdev)); +$('#recommission').onclick = () => action(async () => { const r = await api('POST', `/api/v1/machines/${current}/recommission`); await loadDetail(current); return `state ${r.state}: next PXE boot runs commissioning`; }, 'recommission'); +$('#recommission-now').onclick = () => { if (confirm('Remove PXE override, set bootdev=pxe and power-cycle this machine via its BMC?')) action(async () => { const r = await api('POST', `/api/v1/machines/${current}/recommission?now=1`); await loadDetail(current); return r.ipmi || r.state; }, 'recommission now'); }; +$('#delete').onclick = () => { if (confirm(`Delete ${current}? It will be commissioned again as a new machine on next PXE boot.`)) action(async () => { await api('DELETE', `/api/v1/machines/${current}`); $('#back').click(); return 'deleted'; }, 'delete'); }; + +function schedule() { clearInterval(timer); timer = setInterval(() => { if (!current && document.visibilityState === 'visible') loadList(); }, 10000); } +schedule(); +loadList(); diff --git a/cmd/rollcall/web/index.html b/cmd/rollcall/web/index.html new file mode 100644 index 0000000..5a074ad --- /dev/null +++ b/cmd/rollcall/web/index.html @@ -0,0 +1,75 @@ + + + + + +rollcall + + + +
+

rollcall hardware commissioning

+
+ + + +
+
+ + +
+

Admin token

+

The admin_token from /etc/rollcall/config.json. Stored only in this browser.

+ + +
+ + +
+
+
+ + +
+ + + + + + + +
IDStateModelSerialCPURAMDisksStorageNICIBGPUBMCLast seen
+ +
+ + +
+ + + diff --git a/cmd/rollcall/web/style.css b/cmd/rollcall/web/style.css new file mode 100644 index 0000000..d9c2854 --- /dev/null +++ b/cmd/rollcall/web/style.css @@ -0,0 +1,53 @@ +:root { + --bg: #f6f7f9; --panel: #ffffff; --fg: #1d2129; --muted: #6b7280; --line: #e3e6ea; + --accent: #2563eb; --accent-fg: #ffffff; --warn: #b45309; --danger: #b91c1c; --ok: #15803d; + --pill: #e8eefc; --hover: #f0f4ff; --mono: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; +} +@media (prefers-color-scheme: dark) { + :root:not([data-theme="light"]) { + --bg: #0f1216; --panel: #171b21; --fg: #e6e8eb; --muted: #9aa3ad; --line: #2a3139; + --accent: #3b82f6; --pill: #1e2a44; --hover: #1b2230; --warn: #f59e0b; --danger: #ef4444; --ok: #22c55e; + } +} +:root[data-theme="dark"] { + --bg: #0f1216; --panel: #171b21; --fg: #e6e8eb; --muted: #9aa3ad; --line: #2a3139; + --accent: #3b82f6; --pill: #1e2a44; --hover: #1b2230; --warn: #f59e0b; --danger: #ef4444; --ok: #22c55e; +} +* { box-sizing: border-box; } +body { margin: 0; background: var(--bg); color: var(--fg); font: 14px/1.45 system-ui, -apple-system, Segoe UI, Roboto, sans-serif; } +header { display: flex; align-items: center; justify-content: space-between; padding: 12px 16px; border-bottom: 1px solid var(--line); background: var(--panel); position: sticky; top: 0; z-index: 2; gap: 12px; flex-wrap: wrap; } +h1 { font-size: 18px; margin: 0; } h1 .sub { font-weight: 400; color: var(--muted); font-size: 13px; margin-left: 8px; } +h2 { font-size: 16px; margin: 0; } +main { padding: 16px; max-width: 1400px; margin: 0 auto; } +.muted { color: var(--muted); } .mono { font-family: var(--mono); font-size: 12px; white-space: pre-wrap; } +.btn { border: 1px solid var(--line); background: var(--panel); color: var(--fg); padding: 6px 10px; border-radius: 6px; cursor: pointer; font: inherit; font-size: 13px; } +.btn:hover { background: var(--hover); } .btn.primary { background: var(--accent); color: var(--accent-fg); border-color: var(--accent); } +.btn.ghost { border-color: transparent; } .btn.warn { color: var(--warn); } .btn.danger { color: var(--danger); } +.btn:disabled { opacity: .5; cursor: progress; } +.hdr-right { display: flex; gap: 8px; align-items: center; } +.toolbar { display: flex; gap: 12px; align-items: center; margin-bottom: 10px; } +input[type=search], input[type=password] { padding: 7px 10px; border: 1px solid var(--line); border-radius: 6px; background: var(--panel); color: var(--fg); font: inherit; min-width: 280px; } +table { width: 100%; border-collapse: collapse; background: var(--panel); border: 1px solid var(--line); border-radius: 8px; overflow: hidden; } +th, td { padding: 8px 10px; text-align: left; border-bottom: 1px solid var(--line); white-space: nowrap; } +th { font-weight: 600; color: var(--muted); font-size: 12px; text-transform: uppercase; letter-spacing: .03em; } +th.num, td.num { text-align: right; font-variant-numeric: tabular-nums; } +tbody tr { cursor: pointer; } tbody tr:hover { background: var(--hover); } +.pill { display: inline-block; padding: 2px 8px; border-radius: 999px; background: var(--pill); font-size: 12px; } +.pill.ready { color: var(--ok); } .pill.recommission, .pill.new { color: var(--warn); } +.ok { color: var(--ok); } .bad { color: var(--danger); } +.detail-head { display: flex; gap: 12px; align-items: center; margin-bottom: 10px; } +.actions { display: flex; gap: 6px; align-items: center; flex-wrap: wrap; padding: 10px; background: var(--panel); border: 1px solid var(--line); border-radius: 8px; margin-bottom: 14px; } +.sep { width: 1px; height: 20px; background: var(--line); margin: 0 6px; } +#action-out { margin-left: auto; color: var(--muted); max-width: 50%; } +.grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(420px, 1fr)); gap: 12px; } +.card { background: var(--panel); border: 1px solid var(--line); border-radius: 8px; padding: 12px; overflow: auto; } +.card.wide { grid-column: 1 / -1; } +.card h3 { margin: 0 0 8px; font-size: 13px; text-transform: uppercase; letter-spacing: .03em; color: var(--muted); } +.kv { display: grid; grid-template-columns: max-content 1fr; gap: 3px 14px; } +.kv dt { color: var(--muted); } .kv dd { margin: 0; word-break: break-all; } +.card table { border: none; } .card th, .card td { padding: 4px 8px; font-size: 13px; } +details { margin-top: 14px; } summary { cursor: pointer; color: var(--muted); } +dialog { border: 1px solid var(--line); border-radius: 10px; background: var(--panel); color: var(--fg); padding: 20px; min-width: 360px; } +dialog::backdrop { background: rgba(0,0,0,.45); } dialog menu { padding: 0; margin: 12px 0 0; display: flex; justify-content: flex-end; } +dialog input { width: 100%; margin-top: 8px; } +@media (max-width: 720px) { main { padding: 16px 16px; } .grid { grid-template-columns: 1fr; } input[type=search] { min-width: 0; width: 100%; } table { display: block; overflow-x: auto; } } diff --git a/deploy/config.example.json b/deploy/config.example.json new file mode 100644 index 0000000..28f9e42 --- /dev/null +++ b/deploy/config.example.json @@ -0,0 +1,15 @@ +{ + "listen": "10.213.77.1:80", + "image_dir": "/srv/rollcall/image", + "tftp_root": "/srv/tftp", + "database_url": "postgres://rollcall:@127.0.0.1:5432/rollcall?sslmode=disable", + "master_key_file": "/etc/rollcall/master.key", + "admin_token": "", + "agent_token": "", + "commission_append": "boot=live components fetch=http://10.213.77.1/rollcall/filesystem.squashfs rollcall.server=http://10.213.77.1 rollcall.token= console=tty0 console=ttyS0,115200n8", + "ipmitool": "ipmitool", + "bmc_hints": [ + {"mac": "bc:24:11:8e:1e:6c", "address": "10.213.77.11", "port": 623, "user": "admin", "password": "password", "comment": "pxe-node01 (VM 211, pve-vbmc)"}, + {"mac": "bc:24:11:36:90:3f", "address": "10.213.77.12", "port": 623, "user": "admin", "password": "password", "comment": "pxe-node02 (VM 212, pve-vbmc)"} + ] +} diff --git a/deploy/install.sh b/deploy/install.sh new file mode 100644 index 0000000..30a770e --- /dev/null +++ b/deploy/install.sh @@ -0,0 +1,28 @@ +#!/bin/sh +# Install the rollcall binary on pxe-server. Run as root from this directory. +# Prereqs: PostgreSQL running locally with role+db "rollcall"; +# /etc/rollcall/config.json present (see config.example.json). +set -eu +cd "$(dirname "$0")" +getent passwd rollcall >/dev/null || useradd --system --no-create-home --shell /usr/sbin/nologin --user-group rollcall +install -d -m 0750 -o root -g rollcall /etc/rollcall +install -d /srv/rollcall/image +[ -x ./rollcall ] && install -m 0755 rollcall /usr/local/bin/rollcall +install -m 0644 rollcall.service /etc/systemd/system/ +[ -e /etc/rollcall/config.json ] || { echo "write /etc/rollcall/config.json first (see config.example.json)" >&2; exit 1; } +chown root:rollcall /etc/rollcall/config.json && chmod 0640 /etc/rollcall/config.json +# master key is created by the service on first start; the dir must be writable for that +chown rollcall:rollcall /etc/rollcall +# rollcall writes per-MAC pxelinux configs +chgrp rollcall /srv/tftp/pxelinux.cfg && chmod 2775 /srv/tftp/pxelinux.cfg +install -m 0644 /usr/lib/syslinux/modules/bios/poweroff.c32 /srv/tftp/poweroff.c32 +sed "s/@AGENT_TOKEN@/$(jq -r .agent_token /etc/rollcall/config.json)/g" pxelinux-default > /srv/tftp/pxelinux.cfg/default && chmod 0644 /srv/tftp/pxelinux.cfg/default +[ -e /srv/tftp/rollcall ] || ln -s /srv/rollcall/image /srv/tftp/rollcall +[ -d /srv/tftp/efi64 ] && ln -sfn ../rollcall /srv/tftp/efi64/rollcall +# old python prototype +rm -f /usr/local/bin/rollcall /usr/local/bin/commctl +systemctl daemon-reload +systemctl enable --now rollcall.service +systemctl restart rollcall.service +sleep 1 +systemctl --no-pager status rollcall.service | head -5 diff --git a/deploy/pxelinux-default b/deploy/pxelinux-default new file mode 100644 index 0000000..1983d63 --- /dev/null +++ b/deploy/pxelinux-default @@ -0,0 +1,27 @@ +# /srv/tftp/pxelinux.cfg/default - unknown machines land here. +# Commissioned machines get a per-MAC file (01-xx-xx-...) written by rollcall. +DEFAULT menu.c32 +PROMPT 0 +TIMEOUT 80 +MENU TITLE pxe-server (10.213.77.1) - unknown machine: commissioning in 8s + +LABEL commission + MENU LABEL Commission this machine (inventory + BMC account) + MENU DEFAULT + KERNEL rollcall/vmlinuz + INITRD rollcall/initrd.img + APPEND boot=live components fetch=http://10.213.77.1/rollcall/filesystem.squashfs rollcall.server=http://10.213.77.1 rollcall.token=@AGENT_TOKEN@ console=tty0 console=ttyS0,115200n8 + +LABEL commission-debug + MENU LABEL Commission, but stay up afterwards (ssh root@) + KERNEL rollcall/vmlinuz + INITRD rollcall/initrd.img + APPEND boot=live components fetch=http://10.213.77.1/rollcall/filesystem.squashfs rollcall.server=http://10.213.77.1 rollcall.token=@AGENT_TOKEN@ rollcall.nopoweroff console=tty0 console=ttyS0,115200n8 + +LABEL netbootxyz + MENU LABEL netboot.xyz (needs internet via NAT) + KERNEL netboot.xyz.lkrn + +LABEL local + MENU LABEL Boot from local disk + LOCALBOOT 0 diff --git a/deploy/rollcall.service b/deploy/rollcall.service new file mode 100644 index 0000000..dbe9a78 --- /dev/null +++ b/deploy/rollcall.service @@ -0,0 +1,22 @@ +[Unit] +Description=rollcall - MAAS-like commissioning server (API, web UI) +After=network-online.target postgresql.service +Wants=network-online.target +Requires=postgresql.service + +[Service] +Type=simple +ExecStart=/usr/local/bin/rollcall serve -config /etc/rollcall/config.json +Restart=always +RestartSec=3 +User=rollcall +Group=rollcall +AmbientCapabilities=CAP_NET_BIND_SERVICE +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=yes +ReadWritePaths=/etc/rollcall /srv/tftp/pxelinux.cfg + +[Install] +WantedBy=multi-user.target diff --git a/lab/cloud-init/pxe-server-user.yaml b/lab/cloud-init/pxe-server-user.yaml new file mode 100644 index 0000000..51edcf2 --- /dev/null +++ b/lab/cloud-init/pxe-server-user.yaml @@ -0,0 +1,86 @@ +#cloud-config +hostname: pxe-server +fqdn: pxe-server.mynodes.xyz +manage_etc_hosts: true +users: + - name: ilker + sudo: ALL=(ALL) NOPASSWD:ALL + shell: /bin/bash + lock_passwd: true + ssh_authorized_keys: + - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDSAJ4owXlJreHP3sa4HstasxA5pk1ZAPi6vnzOpA5R1 ilker@t440s +package_update: true +packages: + - qemu-guest-agent + - dnsmasq + - pxelinux + - syslinux-common + - syslinux-efi + - nftables + - curl +write_files: + - path: /etc/dnsmasq.d/pxe.conf + content: | + # PXE/DHCP for SDN vnet pxenet (10.213.77.0/24). Only bound to the pxenet leg. + interface=__PXEIF__ + bind-interfaces + domain-needed + bogus-priv + domain=pxe.local + local=/pxe.local/ + expand-hosts + dhcp-range=10.213.77.100,10.213.77.200,255.255.255.0,12h + dhcp-option=option:router,10.213.77.1 + dhcp-option=option:dns-server,10.213.77.1 + dhcp-authoritative + enable-tftp + tftp-root=/srv/tftp + dhcp-match=set:efi64,option:client-arch,7 + dhcp-match=set:efi64,option:client-arch,9 + dhcp-match=set:bios,option:client-arch,0 + dhcp-boot=tag:bios,pxelinux.0 + dhcp-boot=tag:efi64,efi64/syslinux.efi + log-dhcp + - path: /srv/tftp/pxelinux.cfg/default + content: | + DEFAULT menu.c32 + PROMPT 0 + TIMEOUT 300 + MENU TITLE pxe-server (10.213.77.1) - PXE menu + + LABEL netbootxyz + MENU LABEL netboot.xyz (internet gerekir, NAT uzerinden) + KERNEL netboot.xyz.lkrn + + LABEL local + MENU LABEL Yerel diskten boot + LOCALBOOT 0 + - path: /etc/nftables.conf + permissions: '0755' + content: | + #!/usr/sbin/nft -f + flush ruleset + table ip nat { + chain postrouting { + type nat hook postrouting priority 100; + oifname "__EXTIF__" ip saddr 10.213.77.0/24 masquerade + } + } + - path: /etc/sysctl.d/99-pxe-forward.conf + content: | + net.ipv4.ip_forward=1 +runcmd: + - sysctl --system + - PXEIF=$(ip -o -4 addr show | awk '/10\.213\.77\.1\//{print $2}'); sed -i "s/__PXEIF__/$PXEIF/" /etc/dnsmasq.d/pxe.conf + - EXTIF=$(ip -o -4 route show default | awk '{print $5; exit}'); sed -i "s/__EXTIF__/$EXTIF/" /etc/nftables.conf + - mkdir -p /srv/tftp/efi64 + - cp /usr/lib/PXELINUX/pxelinux.0 /srv/tftp/ + - cp /usr/lib/syslinux/modules/bios/ldlinux.c32 /usr/lib/syslinux/modules/bios/menu.c32 /usr/lib/syslinux/modules/bios/libutil.c32 /usr/lib/syslinux/modules/bios/libcom32.c32 /srv/tftp/ + - cp /usr/lib/SYSLINUX.EFI/efi64/syslinux.efi /srv/tftp/efi64/ + - cp /usr/lib/syslinux/modules/efi64/ldlinux.e64 /usr/lib/syslinux/modules/efi64/menu.c32 /usr/lib/syslinux/modules/efi64/libutil.c32 /usr/lib/syslinux/modules/efi64/libcom32.c32 /srv/tftp/efi64/ + - ln -sfn ../pxelinux.cfg /srv/tftp/efi64/pxelinux.cfg + - curl -fsSL --retry 3 -o /srv/tftp/netboot.xyz.lkrn https://boot.netboot.xyz/ipxe/netboot.xyz.lkrn || true + - systemctl enable --now nftables + - systemctl enable --now qemu-guest-agent + - systemctl restart dnsmasq + - systemctl enable dnsmasq diff --git a/lab/pve-vbmc/config.example.json b/lab/pve-vbmc/config.example.json new file mode 100644 index 0000000..bc46f71 --- /dev/null +++ b/lab/pve-vbmc/config.example.json @@ -0,0 +1,12 @@ +{ + "api": "https://10.0.0.10:8006", + "node": "pve1", + "token": "root@pam!vbmc=", + "verify_tls": false, + "user": "admin", + "password": "password", + "bmcs": [ + {"vmid": 211, "address": "10.213.77.11", "port": 623}, + {"vmid": 212, "address": "10.213.77.12", "port": 623} + ] +} diff --git a/lab/pve-vbmc/install.sh b/lab/pve-vbmc/install.sh new file mode 100644 index 0000000..ac632b2 --- /dev/null +++ b/lab/pve-vbmc/install.sh @@ -0,0 +1,16 @@ +#!/bin/sh +# Run on pxe-server as root from the directory containing these files. +# Expects /etc/pve-vbmc/config.json to exist already (see config.example.json). +set -eu +cd "$(dirname "$0")" +getent passwd pve-vbmc >/dev/null || useradd --system --no-create-home --shell /usr/sbin/nologin --user-group pve-vbmc +install -m 0755 pve_vbmc.py /usr/local/bin/pve_vbmc.py +install -m 0755 pve-vbmc-addrs.sh /usr/local/bin/pve-vbmc-addrs.sh +install -m 0644 pve-vbmc@.service pve-vbmc-addrs.service /etc/systemd/system/ +chgrp pve-vbmc /etc/pve-vbmc/config.json && chmod 0640 /etc/pve-vbmc/config.json +systemctl daemon-reload +systemctl enable --now pve-vbmc-addrs.service +for v in $(python3 -c 'import json;print(" ".join(str(b["vmid"]) for b in json.load(open("/etc/pve-vbmc/config.json"))["bmcs"]))'); do + systemctl enable --now "pve-vbmc@$v.service" +done +systemctl --no-pager --type=service | grep pve-vbmc diff --git a/lab/pve-vbmc/pve-vbmc-addrs.service b/lab/pve-vbmc/pve-vbmc-addrs.service new file mode 100644 index 0000000..23f4a55 --- /dev/null +++ b/lab/pve-vbmc/pve-vbmc-addrs.service @@ -0,0 +1,13 @@ +[Unit] +Description=pve-vbmc: add one BMC IP alias per VM on the pxenet interface +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/usr/local/bin/pve-vbmc-addrs.sh add +ExecStop=/usr/local/bin/pve-vbmc-addrs.sh del + +[Install] +WantedBy=multi-user.target diff --git a/lab/pve-vbmc/pve-vbmc-addrs.sh b/lab/pve-vbmc/pve-vbmc-addrs.sh new file mode 100644 index 0000000..b3272b4 --- /dev/null +++ b/lab/pve-vbmc/pve-vbmc-addrs.sh @@ -0,0 +1,13 @@ +#!/bin/sh +# Adds/removes the BMC listen addresses from /etc/pve-vbmc/config.json +# as secondary IPs on the interface that carries the PXE subnet. +set -eu +CONF=/etc/pve-vbmc/config.json +PXE_IF=$(ip -o -4 addr show | awk '/10\.213\.77\.1\//{print $2; exit}') +[ -n "$PXE_IF" ] || { echo "pxenet interface not found" >&2; exit 1; } +for a in $(python3 -c 'import json;print(" ".join(b["address"] for b in json.load(open("'"$CONF"'"))["bmcs"]))'); do + case "$1" in + add) ip -4 addr replace "$a/24" dev "$PXE_IF" label "$PXE_IF:bmc" ;; + del) ip -4 addr del "$a/24" dev "$PXE_IF" 2>/dev/null || true ;; + esac +done diff --git a/lab/pve-vbmc/pve-vbmc@.service b/lab/pve-vbmc/pve-vbmc@.service new file mode 100644 index 0000000..13ac500 --- /dev/null +++ b/lab/pve-vbmc/pve-vbmc@.service @@ -0,0 +1,21 @@ +[Unit] +Description=pve-vbmc virtual BMC for Proxmox VM %i +After=network-online.target pve-vbmc-addrs.service +Wants=network-online.target +Requires=pve-vbmc-addrs.service + +[Service] +Type=simple +ExecStart=/usr/local/bin/pve_vbmc.py --config /etc/pve-vbmc/config.json --vmid %i +Restart=always +RestartSec=3 +AmbientCapabilities=CAP_NET_BIND_SERVICE +User=pve-vbmc +Group=pve-vbmc +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=yes + +[Install] +WantedBy=multi-user.target diff --git a/lab/pve-vbmc/pve_vbmc.py b/lab/pve-vbmc/pve_vbmc.py new file mode 100644 index 0000000..11baadc --- /dev/null +++ b/lab/pve-vbmc/pve_vbmc.py @@ -0,0 +1,160 @@ +#!/usr/bin/env python3 +"""pve-vbmc: minimal virtual BMC (IPMI-over-LAN) for Proxmox VE VMs. + +One process serves one VM. It speaks IPMI 2.0 LAN via pyghmi and translates +chassis power / boot-device commands into Proxmox API calls. + +Supported ipmitool commands: + power status|on|off|soft|cycle|reset + chassis bootdev pxe|disk|cdrom|none + chassis status, mc info, mc guid +""" +import argparse +import ipaddress +import json +import logging +import sys +import time +import uuid + +import requests +import urllib3 +from pyghmi.ipmi.bmc import Bmc + +log = logging.getLogger("pve-vbmc") + + +class PveBmc(Bmc): + def __init__(self, cfg, entry): + self.api = cfg["api"].rstrip("/") + self.node = cfg["node"] + self.vmid = int(entry["vmid"]) + self.http = requests.Session() + self.http.headers["Authorization"] = "PVEAPIToken=" + cfg["token"] + self.http.verify = cfg.get("verify_tls", False) + if not self.http.verify: + urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + self.bootdev = "default" + self.guid = uuid.uuid5(uuid.NAMESPACE_DNS, f"pve-vbmc-{self.node}-{self.vmid}") + + addr = ipaddress.ip_address(entry["address"]) + listen = f"::ffff:{addr}" if addr.version == 4 else str(addr) + authdata = {entry.get("user", cfg.get("user", "admin")): + entry.get("password", cfg.get("password", "password"))} + super().__init__(authdata, port=int(entry.get("port", 623)), address=listen) + log.info("BMC for VM %s listening on %s:%s", self.vmid, addr, entry.get("port", 623)) + + # --- Proxmox API helpers ------------------------------------------------- + def _url(self, path): + return f"{self.api}/api2/json/nodes/{self.node}/qemu/{self.vmid}/{path}" + + def _get(self, path): + r = self.http.get(self._url(path), timeout=10) + r.raise_for_status() + return r.json()["data"] + + def _post(self, path, **data): + r = self.http.post(self._url(path), data=data, timeout=30) + r.raise_for_status() + return r.json()["data"] + + def _put(self, path, **data): + r = self.http.put(self._url(path), data=data, timeout=30) + r.raise_for_status() + return r.json()["data"] + + def _wait_task(self, upid, timeout=60): + url = f"{self.api}/api2/json/nodes/{self.node}/tasks/{upid}/status" + for _ in range(timeout * 2): + r = self.http.get(url, timeout=10) + r.raise_for_status() + if r.json()["data"]["status"] == "stopped": + return r.json()["data"].get("exitstatus") + time.sleep(0.5) + return "timeout" + + # --- pyghmi Bmc interface ------------------------------------------------- + def get_system_guid(self): + return self.guid + + def get_power_state(self): + return "on" if self._get("status/current")["status"] == "running" else "off" + + def is_active(self): + return self.get_power_state() == "on" + + def power_on(self): + if self.get_power_state() == "off": + log.info("VM %s: power on", self.vmid) + self._post("status/start") + + def power_off(self): + if self.get_power_state() == "on": + log.info("VM %s: power off (hard)", self.vmid) + self._post("status/stop") + + def power_shutdown(self): + log.info("VM %s: soft shutdown", self.vmid) + self._post("status/shutdown", forceStop=1, timeout=60) + + def power_reset(self): + if self.get_power_state() == "on": + log.info("VM %s: reset", self.vmid) + self._post("status/reset") + else: + self.power_on() + + def power_cycle(self): + log.info("VM %s: power cycle", self.vmid) + if self.get_power_state() == "on": + self._wait_task(self._post("status/stop")) + self._post("status/start") + + def cold_reset(self): + return 0 # BMC reset is a no-op + + def pulse_diag(self): + raise NotImplementedError + + def get_boot_device(self): + return self.bootdev + + def set_boot_device(self, bootdevice): + cfg = self._get("config") + disks = [k for k in ("scsi0", "virtio0", "sata0", "ide0") if k in cfg] + cds = sorted(k for k, v in cfg.items() if "media=cdrom" in str(v)) + nets = sorted(k for k in cfg if k.startswith("net") and k[3:].isdigit()) + if bootdevice == "network": + order = nets + disks + cds + elif bootdevice == "hd": + order = disks + nets + cds + elif bootdevice == "optical": + order = cds + disks + nets + else: # default / none / safe / setup ... + order = disks + cds + nets + if order: + log.info("VM %s: bootdev %s -> order=%s", self.vmid, bootdevice, ";".join(order)) + self._put("config", boot="order=" + ";".join(order)) + self.bootdev = bootdevice + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--config", default="/etc/pve-vbmc/config.json") + ap.add_argument("--vmid", type=int, required=True) + ap.add_argument("--debug", action="store_true") + args = ap.parse_args() + logging.basicConfig(level=logging.DEBUG if args.debug else logging.INFO, + format="%(asctime)s %(name)s %(levelname)s %(message)s") + with open(args.config) as f: + cfg = json.load(f) + entry = next((b for b in cfg["bmcs"] if int(b["vmid"]) == args.vmid), None) + if entry is None: + log.error("vmid %s not in config", args.vmid) + sys.exit(2) + bmc = PveBmc(cfg, entry) + bmc.listen() + + +if __name__ == "__main__": + main() diff --git a/site/index.html b/site/index.html new file mode 100644 index 0000000..85761cd --- /dev/null +++ b/site/index.html @@ -0,0 +1,199 @@ + + + + + +Rollcall + + + + +
+ + +
+

Bare-metal commissioning,
one static binary.

+

Rollcall does what MAAS does when a new server first shows up: PXE-boot an ephemeral image, take a full hardware inventory, create an administrator account on the BMC, and remember the machine. No cloud controller, no agents on your OS, no fleet of services.

+
+ Quick start + Repository +
+
+ Go · single binaryPostgreSQLDebian live image + IPMI / ipmitoolInfiniBand awareWeb UI + CLI + API +
+
+ +
+

How it works

+

A machine goes from unknown to ready in one boot, without anyone typing anything.

+
+

PXE boot

dnsmasq hands out DHCP and TFTP. Unknown MACs land on the default pxelinux menu, which starts commissioning after 8 seconds.

+

Ephemeral image

A Debian 13 live image (full kernel, non-free firmware, IB and NVMe tooling) is fetched over HTTP and runs from RAM. Disks are never touched.

+

Inventory

DMI, DIMMs, CPUs, disks with SMART, NVMe, NICs with firmware and link, InfiniBand HCAs, GPUs, every PCI device, LLDP switch ports, TPM.

+

BMC account

In-band ipmitool finds the LAN channel and creates commissioner with ADMINISTRATOR rights and a random password.

+

Report & power off

Everything is posted to the server. The password is sealed with AES-256-GCM before it reaches PostgreSQL. The machine powers off.

+

Known from now on

A per-MAC PXE file stops future boots of the image. Power and boot device are controlled through the API, UI or CLI, with ipmitool running on the server.

+
+
+ +
+

Features

+

Small surface, enterprise hardware in mind.

+
+

One binary, everything inside

HTTP API, embedded web UI, CLI subcommands, schema migrations and PXE bookkeeping ship as a single static Go executable. Deploy with scp.

+

Secrets stay on the server

BMC passwords are encrypted at rest with a local master key, never returned by the API, and only ever used by ipmitool inside the server process.

+

Real hardware first

The image carries firmware for Broadcom, QLogic, Intel and Mellanox NICs, rdma-core, infiniband-diags, mstflint, nvme-cli, smartmontools, lldpd.

+

Virtual machines too

No in-band IPMI? Rollcall can map MACs to known BMCs. The repo ships pve-vbmc, an IPMI BMC emulator for Proxmox VMs, so the whole flow runs in a lab.

+

Two tokens, two audiences

The live image authenticates with an agent token baked into the kernel command line; people and automation use a separate admin bearer token.

+

Re-commission on demand

Drop the PXE override, set bootdev pxe and power-cycle through the BMC with one click or rollcall recommission ID -now.

+
+
+ +
+

Quick start

+

On the host that already runs your DHCP/TFTP for the provisioning network.

+
+
+ 
# 1. database
+sudo -u postgres psql \
+  -c "CREATE ROLE rollcall LOGIN PASSWORD 'secret';" \
+  -c "CREATE DATABASE rollcall OWNER rollcall;"
+
+# 2. config
+rollcall keygen   # admin_token
+rollcall keygen   # agent_token
+sudo install -d -m 0750 /etc/rollcall
+sudo cp config.json /etc/rollcall/config.json
+
+# 3. install binary, unit, pxelinux menu
+cd deploy && sudo ./install.sh
+
+# 4. build the live image (~8 min)
+sudo ../agent/build-image.sh
+
+
+ 
$ rollcall list
+ID               STATE   MODEL                   RAM   NIC IB  BMC
+m-2ef1dab828c8   ready   Dell PowerEdge R760     1.0T  4   2   10.4.0.21 (inband,ok)
+m-9c01a3bd7e12   ready   Supermicro SYS-421GE    2.0T  6   4   10.4.0.22 (inband,ok)
+
+$ rollcall power m-2ef1dab828c8 status
+Chassis Power is on
+
+$ rollcall show m-2ef1dab828c8 | head -3
+m-2ef1dab828c8  state=ready
+  Dell Inc. PowerEdge R760  S/N 7XK2Q94
+  CPU: Intel Xeon Platinum 8468  sockets=2 ...
+

Then point the UI at http://<server>/ and paste the admin token. Machines appear as they boot.

+
+
+
+ +
+

API

+

JSON over HTTP. Admin calls carry Authorization: Bearer <admin_token>; the agent sends X-Rollcall-Token.

+ + + + + + + + + + + + +
MethodPathWhoPurpose
POST/api/v1/commissionagentInventory upload from the live image
GET/api/v1/bmc-hint?macs=agentOut-of-band BMC hint for machines without in-band IPMI
GET/api/v1/machinesadminList with summaries (CPU, RAM, disks, NICs, IB, GPUs, BMC state)
GET/api/v1/machines/{id}adminFull record: inventory, BMC (without secret), events
POST/api/v1/machines/{id}/poweradmin{"action":"on|off|soft|cycle|reset|status"}
POST/api/v1/machines/{id}/bootdevadmin{"device":"pxe|disk|cdrom|bios|none"}
POST/api/v1/machines/{id}/recommission[?now=1]adminBoot the image again on next PXE, or right now via the BMC
DELETE/api/v1/machines/{id}adminForget the machine
+
+ +
+

What the inventory contains

+

Collected by rollcall-agent.py inside the live image and stored as JSON per machine.

+
+

System

Manufacturer, product, serial, UUID, SKU, baseboard, chassis, asset tag, BIOS vendor/version/date, TPM.

+

Compute

CPU model, sockets, cores, threads, NUMA nodes, caches, virtualization flags; memory total plus every DIMM slot with size, type, speed, vendor, part and serial.

+

Storage

All block devices with model, serial, bus, rotational flag, sector sizes, SMART health and firmware, NVMe list, storage controllers from PCI.

+

Network

NICs with driver, firmware, speed, link state, permanent MAC, PCI address, NUMA node, SR-IOV capacity; LLDP neighbours to map switch ports.

+

InfiniBand

HCAs with node and port GUIDs, firmware, board id, per-port state, physical state, rate, LID and link layer.

+

BMC

LAN channel, IP source, MAC, gateway, VLAN, firmware, FRU, user table before commissioning, created account and verification result.

+
+
+ + +
+ +