From fad4e1c8bd76ac6c49feab5085aeb7678dcf7ae6 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Tue, 27 Oct 2026 10:10:39 +0100 Subject: [PATCH] OS deployment: image catalog with upstream sync, deploy/release jobs, agent deploy mode, UI and CLI - rollcall os list|sync|import|promote|rm: ///{disk.raw.zst,meta.json}, current symlink, vendor checksum verification (GNU/BSD formats, sha256/sha512), qemu-img -> raw, zstd in-process, keep-N pruning, nightly rollcall-os-sync.timer, deploy/os-sources.json (Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9, CentOS Stream 9) - jobs table + API: deploy (disk pick, hostname/IP allocation, job token, per-MAC deploy PXE entry), release (wipe), progress, cancel, cloud-init phone_home - agent/rollcall-deploy.py: streamed write with sha256 + pv progress, NoCloud seed, SELinux autorelabel, efibootmgr on UEFI; commissioning image gains zstd/parted/xfsprogs/... - web UI: deploy/release controls, deployment card with progress, OS catalog table - README: deployment section, image freshness policy Co-Authored-By: Claude Fable 5.1 --- README.md | 77 +- .../rollcall-agent.cpython-314.pyc | Bin 35732 -> 36048 bytes .../rollcall-deploy.cpython-314.pyc | Bin 0 -> 28126 bytes agent/build-image.sh | 10 +- agent/rollcall-agent.py | 4 + agent/rollcall-deploy.py | 385 ++++++++++ cmd/rollcall/client.go | 146 +++- cmd/rollcall/config.go | 10 + cmd/rollcall/deploy.go | 643 ++++++++++++++++ cmd/rollcall/go.mod | 1 + cmd/rollcall/go.sum | 2 + cmd/rollcall/main.go | 14 +- cmd/rollcall/osimages.go | 694 ++++++++++++++++++ cmd/rollcall/server.go | 33 +- cmd/rollcall/stateless.go | 9 +- cmd/rollcall/store.go | 31 +- cmd/rollcall/web/app.js | 55 +- cmd/rollcall/web/index.html | 16 +- cmd/rollcall/web/style.css | 7 +- deploy/config.example.json | 2 + deploy/install.sh | 6 + deploy/os-sources.json | 52 ++ deploy/rollcall-os-sync.service | 13 + deploy/rollcall-os-sync.timer | 10 + 24 files changed, 2194 insertions(+), 26 deletions(-) create mode 100644 agent/__pycache__/rollcall-deploy.cpython-314.pyc create mode 100644 agent/rollcall-deploy.py create mode 100644 cmd/rollcall/deploy.go create mode 100644 cmd/rollcall/osimages.go create mode 100644 deploy/os-sources.json create mode 100644 deploy/rollcall-os-sync.service create mode 100644 deploy/rollcall-os-sync.timer diff --git a/README.md b/README.md index 937e63c..3752c2d 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,8 @@ unknown server ──PXE──▶ dnsmasq (DHCP+TFTP) ──▶ pxelinux default |---|---| | `cmd/rollcall/` | Go server + CLI + embedded web UI (`web/`) | | `agent/` | `rollcall-agent.py` (runs inside the live image), its systemd unit, `build-image.sh` (live-build recipe) | -| `deploy/` | `install.sh`, `rollcall.service`, `config.example.json`, `pxelinux-default` | +| `deploy/` | `install.sh`, `rollcall.service`, `config.example.json`, `pxelinux-default`, `os-sources.json`, sync/dnsmasq/controller units, `overlays/` | +| `images/` | stateless root image recipes (`build-rootfs.sh`, `compute-debian13/`) | | `lab/` | Proxmox test lab: `pve-vbmc` (virtual IPMI BMC for VMs backed by the Proxmox API), cloud-init for the PXE server VM | | `site/` | Project website (static) | | `.gitea/workflows/` | CI (fmt/vet/build/tests), release on tags, deploy on push to `main` | @@ -97,6 +98,15 @@ Reads `ROLLCALL_URL` / `ROLLCALL_TOKEN`, or the config file when readable. | POST | `/api/v1/machines/{id}/bootdev` | admin | `{"device":"pxe"}` | | POST | `/api/v1/machines/{id}/recommission[?now=1]` | admin | boot the image again on next PXE (`now=1`: via BMC immediately) | | DELETE | `/api/v1/machines/{id}` | admin | forget the machine | +| GET/POST/DELETE | `/api/v1/profiles[/{name}]` | admin | stateless profiles | +| POST | `/api/v1/machines/{id}/assign` `/unassign` | admin | stateless node assignment | +| GET | `/api/v1/os` | admin | OS image catalog | +| POST | `/api/v1/machines/{id}/deploy` | admin | `{"os":"rocky-9","disk":"/dev/sda","hostname":"","package_upgrade":false,"boot":true}` | +| POST | `/api/v1/machines/{id}/release` | admin | wipe job | +| GET / POST | `/api/v1/machines/{id}/jobs[/cancel]` | admin | job history / cancel | +| GET | `/api/v1/jobs/self`, POST `…/progress` | job token (`X-Rollcall-Job`) | deploy agent | +| POST | `/api/v1/phone-home/{token}` | job token in path | cloud-init phone_home | +| GET | `/os/*` | none | OS disk images | | GET | `/rollcall/*` | none | image files (kernel, initrd, squashfs) | | GET | `/` | token in browser | web UI | @@ -157,6 +167,64 @@ dress it with per-node files, instead of installing an OS on disk. States: `ready → stateless → booted`; `rollcall unassign` releases name and IPs. +## OS deployment (install to disk) + +For nodes that should run an operating system from their own disk, Rollcall +writes a vendor cloud image onto the disk and lets cloud-init do first-boot +configuration. No installer, no kickstart/preseed: a deployment is a streamed +disk write and takes about as long as the image download. + +``` +rollcall deploy rocky-9 -boot -watch + │ server: picks the disk (smallest that fits, or -disk), allocates hostname + IP, + │ creates a job with a one-time token, writes pxelinux.cfg/01- + │ ("deploy" entry = commissioning image with rollcall.mode=deploy), power-cycles via BMC + ▼ +commissioning image, deploy mode (agent/rollcall-deploy.py) + GET /api/v1/jobs/self → image URL, sha256, disk, rendered cloud-init seed + curl | tee sha256 | pv | zstd -d | dd of=/dev/sda (progress every 3 s) + mount root fs → /var/lib/cloud/seed/nocloud/{meta-data,user-data}, datasource_list: [NoCloud, None] + (SELinux images: /.autorelabel; UEFI hosts: efibootmgr entry for the ESP loader) + POST progress state=done → server: state deployed, PXE entry = local disk, reboot + ▼ +first boot: cloud-init sets hostname/fqdn, SSH keys (root + default user), optional +package_upgrade, then phone_home → POST /api/v1/phone-home/ (shown as "phoned home") +``` + +### Image catalog and freshness + +``` +os_dir///disk.raw.zst raw disk image, zstd +os_dir///meta.json sizes, sha256, upstream checksum, import time +os_dir//current -> what new deployments use +``` + +`os_sources` (`deploy/os-sources.json`) lists upstream "latest" URLs with the +vendor checksum file for each: Debian 13, Ubuntu 24.04, Rocky 9, AlmaLinux 9, +CentOS Stream 9. `rollcall os sync` fetches the checksum file, and only when the +vendor checksum differs from every imported version does it download the image, +verify it, convert to raw (`qemu-img`), compress, write a new version directory +and flip `current` atomically. `rollcall-os-sync.timer` runs it nightly; +`keep` versions are retained per source, nothing younger than a day is pruned, +and a job always pins the exact version it started with. Sources with +`"track": "pinned"` still download new versions but keep `current` until +`rollcall os promote `. Local files go in with +`rollcall os import `. Image freshness is monthly at best +upstream; the deploy option `-upgrade` (`package_upgrade`) pulls the rest on +first boot. + +```sh +rollcall os list | sync [NAME...] [-force] | import NAME FILE [-family debian|rhel] | promote NAME VERSION | rm NAME VERSION +rollcall deploy ID OS [-version V] [-disk /dev/nvme0n1] [-hostname H] [-upgrade] [-packages a,b] [-user-data FILE] [-boot] [-watch] +rollcall release ID [-wipe quick|full] [-boot] [-watch] # wipe the disk, back to ready +rollcall jobs ID [-watch] | job cancel ID +``` + +States: `ready → deploying → deployed` (`deploy-failed` on error; the agent stays +up for ssh). A deployed machine's PXE entry defaults to local disk, so boot order +can stay "network first" on BIOS hosts; on UEFI hosts the agent adds a boot entry +for the installed loader. `rollcall recommission` or `release` take it back. + ## Lab `lab/` documents a Proxmox SDN test setup: an isolated vnet, a PXE server VM @@ -166,6 +234,7 @@ IPMI-over-LAN BMC emulator that drives VMs through the Proxmox API so that ## Status -Working end to end in the lab (virtual machines, out-of-band BMC hints). The -in-band BMC path is implemented but has not yet been exercised on real -hardware. +Commissioning, stateless nodes and disk deployment work end to end in the lab +(virtual machines with 30 GB disks, SeaBIOS, out-of-band BMC hints). The +in-band BMC path and the UEFI boot-entry step are implemented but have not yet +been exercised on real hardware. diff --git a/agent/__pycache__/rollcall-agent.cpython-314.pyc b/agent/__pycache__/rollcall-agent.cpython-314.pyc index 6a8e66d7c63fed472f45cd2bf0eb18ec4ab06831..3027ddc58c4261d2aa3cf0c0f3a231eff9438dea 100644 GIT binary patch delta 1914 zcmYjSeQZ-z6u;;7wXfYEzTi3PS=)Txm_QgKf2MY{Pn3TIRM)G8Atb}3dp6xdF z!PW^x!cZU{32F?GXiWUaWIm!qW8e=zfIoDM1$7}t&_BdTf-wP&{^4!QTl?B~-#O?0 z?m546-nsX^d4XGao6A^}VN4_J+HvsbL%;dvGoAvxtLV7(+=r9G*xT4-F5rdOXhXXlTV&mB4u93FlBKG7{(?be8Stb|$Qc1ECS8*BK531O1VJ zlPy=;`Dy@%^0Ia_2PMi?VH-e=!W+B*4N6_wd4P81miQ20i*m)-sDYizBh$PF_9~aM zU(-PpD{UV_@V&b>Eu2-pcb)_|rt}qg^ze~#rs5rd^GbQ;kO4kXuzrbyAC-fRe{10< zrN8Z>08c*N{I38e<-e|-JPZ8-b#u^wBmQGhk5B#Apjr8JR|ud@DcL`0c)kgyR#b#;)C9b_+8OnLyFy*YTzV#ts|GODon z(i~B*f~5?6e5BsIqIUr`MDz(1q@k}QUq?lmlPvfsRDc(KX%(_YN(N2ufpk*Csoorf z8fga2?9ot@l#P8O@g-4emhw~r)S`lEmWOz?GEcK5mTlsy3|Kj7cDZ8FqML5~)I+*O9>vw#kVsQ2UUU_2V;BcsRS$HN)y5dv8VfIOg z1Jut$^3+%6WXKzJlJRIsuwYXjqf{bRV}_!!M;#g~T`s{IS@T}%2NW&T7a zAN?xR>TBY@=E|tRf8$<7)A<{mF*aJ<)J$%AMA~e+*`{OTx7GRGM7z)=QeJzTuiM$K zz1^;5@}gO!4VuLQ6RpuK)@Yf0hlt%K!?M1w?{NQ6sIRY_dpOSW@RvjQ7eE$z15rO?;52(7dY5vse_rN%h61&Laqum!=T zuWbzv(;_0LD)NKGs6R|J6mEFY4fNrgNUu!w8-(d>i(>(dd=n6l)5e02pH* zh9X9Im;Dw#2QbNAsO&Ytd+c8A*8<#RpEvv|!7RJB@w@^4V7uD?Hh_)wJeUwU_a#~) zKpoypJ_j}U*!C+BV}+gj0PbVskL;9TKmM}wFX(4KJ@J(QTiE+g{wTtW?A)H?61>8K z{cRHGeCluvU<*_mziU;+0I_<(}nj;=?NcxnZSg~)dAGaNm*W_{=Iw6m&6LJjHNKMq-siU%D z$Haawo;xCE@)YNy07|9LIy>2JJS0TOA)%dTN*R?(92oEl z@oK#Kumdx#CcHC{U*Rp~36%9}mYOA}Q>4CyCpkVph6jV-z^Uc{?imb008b72xP6D) zTKsI#=gMJ=R1pE9)Z;&cE?@EgvaMDcRnGW^ta5B0a=|9tAs1kJ$OBDyW~c*f_`^^D zO3nYH(G}AD3N5!N_D=xS-LG4^%FH212q( zZA`;ibpg^Ln24V!EvmBR1!Kg}NTH!>p@z-aI#OcE)f!3Ku;1mhMO8(4gsxqjgKC@_ z@f7E<>r@QyOPk6<*V7uDAMwe#j zh>Z-kem4AlK-TXvea z&8lr%ynG3&xg{lDz9(sD6!@;vX%s9qp^|T)g8zWT9LUEDOc^#A+H_m%IL?i4j+V4& z1@XXPU}wiJ3v%g!X8zKFXSl)MqlY1k z-yeMmu4CV^&8uelPv$be4mw0aM#(LC#YAkhmCYZsiJ6azTye8-%~6(ciyzmy6F%{} z!;-bdC5aIEWRpx~`=>TNw_gecG+&xF%L%76?W{@Ir5U@#{h5$Vjk=jCJJsoK=p^pn z$d_r*^qsN0yKhJDzV7aFVJ^*eCClN!Z7ze~vVWCwnDY=E7_WtCoErZYX7R*CmFH&; h;3qj~;`Xi%0Dcu^5Px^$7ZbOnjbOQZKVCQ){1+i8Kn(x@ diff --git a/agent/__pycache__/rollcall-deploy.cpython-314.pyc b/agent/__pycache__/rollcall-deploy.cpython-314.pyc new file mode 100644 index 0000000000000000000000000000000000000000..da5dca180c88a8082bde3f366bd2977028b49975 GIT binary patch literal 28126 zcmbuo33MCRnJ!oW3Oh(}-5<%w*0xGqR*< zC-$7~`R*-L0i(<@suK)el%k7yN76$Iv_&+}VqkRnXPvk{$hD6}rUuqa; zn&Fu+GsN(kqnfaWKDA*jed@wG`eef_ed@z{`ZRC+T8(Wg0Vp-*er@}#c6 zNcmjy*BVl(!>y8J`8eoX>_|!RNrQM2=jVgX8n?Ud89bujUKj*YJh#YxyGhTb``r zi#yITjJJgL$0v>lD&@>VB`VM&M$C8mx2S*O<4+qctCOexfmPG1aP7Dh^ z_XM{;D0VjQzvvUlcRQQCe(`*h;J(xp5yMuBIapU=yA0>LS7qkqyL=K6w%$qP5?51^TTrMgRkKO9DvgkUhtjfxG)y4+qb z=jKj#_Z;O$NFv5bxFYwTyrcGuemT8?6MUhd5GGHO)MQDF?q2SaKO_(6hGt)w@_Wvs z+y1dhcX&!bd$yxpf*5WNxY5`4HtwYVQS>2EZL&?F;3c2H1xH6Qdfn*kmGC&3A2MTf zBDqQQTIA~GxjN0CXy82KKF@h^YNCM-Sz=u1AUMb@1Z77*y(HBxzpRyIO1*_Y22~Hy~DGo)ra4RC$V?w;?=_^ zJN4c)JOWo-jE!m0hBSj}E|+F7vC9~&?cIrRC6kODWKo#9eg#e`5qKWcc(ce(O0AJV ziDXSYi(P}CLmx4plNaif9QwwRp>raj9}P^2<9!Y`t`DKB;kfR^sUvZnctwmG#IQFw z6?SOjx(RVC&XNrxk~Y?H@gh1;&B``ul&fwEU5V=g!7;&(TpWI4E4*oDEu$bRMtzdC zY`SZmHP6((JaCIGUN>4^uDQPL+P0XnY~7SGk^;;2a4v*2M2# zKSpnw8KjZK>sLomjRFjFF|8V&86CeQ zQuIj4OqNnY{n<(=`Bb1OZ{m#_vI_Oe7sfR%vO3Ihy*Ta*1mc+a1lMgtmi=AA431~um6gI7IA7MBm_M^QH<*Vf8!Rv2nZZ@PkMY< zM`0ljFoG&4J%FTf8$UHk>gpDRpb%%D4*DnQI`atvnh+NJA%{M$12h)$(fqicZu_|Q z{H3@iB$C;ztE&S*A^jGL6^{(8_!_)vX5ExCoB4cZR9rSqzPW8-;O8~5>UPOAc_R!@ z;$6glYK-{jCy-_O(5FUQH2ZqlwX*r_mF(*E%#t^sSk(P|V0Fu$kGJf3)w3kNfB6@e zZ|Z*eEvfwE&C}9=b9G>Nd0<%LpOT((OCz4uk*AkOo?aaZEsum^BZ4GelEll>$Q8*Q zkxHM5Wj-sJo?Wxrrh5hKRK}CKSdECF0OEJF>7}XLSX~19G{c%H?UZg0v##tPQUV%ds;>J@0bkZHVI2&?@$Kyt? zZ`3^%z?vErNn`5jhk2!< zQ~$RX1peN7$Zr0?%p&-KT}S+!j6>DB52`fqJ;<&)6_l?bYMN?pW?qRc% z#q=Cwn2^S$NjE>dMyW#q5@Q&WGUj+0!{Cr0D}}mbK3=O~`prsw^2zIznl7ORp#}hr zRqFK8I-TrEAdzRZbtrWF1jCq_3V=&`L}WCRo!p#2h9YJKyf^j5H3ysyJ)INm(@?-4 z#>NLS5Ed`_F_*dkCfgjsyFU!12YVRv9|k}Gk`&kA|8QJ$k?voL;?VsYHzvL#>_JP~ z@e|$fa9UV1a7@_qqQf!!mR0-qW&8Hq_Qo0Anl*E_^J?eZ_El@$inVU7p!l_#x%07- zZBog0si0xCpnbWZeNl`R?7Mno=1{ci)6%kQ?)lo8t{0AcZe+5W?wA-`*7R|CBeM9E z;(%ex8!>dS0^htM`^s6U`(j_hHaP zAW9mYS3mOBLC;&Brw()_d-oQ!0iZR8WgSmUwa2su8y?yx&x9D;*Hvjj;3qz(9dA%{iS zPx5c02_r%8mAD?5l1~)C%3y$W`7QuC2gMcF_~RNOuJMT$Bmh&YlL<;0-0DCU31pu> zf}eO4BLq-e-|%sL$BS*Vd!OGM^~S8FtJdmeYxRn?cF7AEY{{B!y4tjA;g&7jn*|HC zKPg{yu5Ig(_MVXXhE|^PNsiGwI>uUckI`7iHPeUx(P~GjjGXAM?+wmq?r2Ts{Ppa@ z>lN23Rr%>zT}w`vr`( zaISW)U{08?oiC6I>thy%snMJ5#f>N9)(?6X&&UA1pnwr{!5=*;Q_{fT(y7?~C#9j0 zTl+oIKCe{cixrGYS);%!Uh4X4J*(il?V4>hYs+%hmNzd+_GTay_zX=1Mq$qX@-vfd z*7Ce1x^p%|#9C%P*y%W2t^HL=!C|}cS34{S|JrUnTxtDv2}{D2I^tJX_LS&9)Ewfv z>-4{=tHaAbWN3(=sqM+v|3khWVGk-(oq{ml1E68tn6f9CFisTkqz1ye0wxH$5u6nk zn7xj2VDvnTcLT47Z{z{XXe{vlu-V0gExZ;7y(Mny7eJ)}Wlu-3n(md+#lcfP*p$x2 zOsdU67-_OYSZUoioGD3u4$7$LX$n+Q&pM@U1~i;D{2uN>M3xTnD@+uc0)Dh6D zw?VoSmW4=Wp-$e|rr}}28X6vsXDid%q(nK9O(dHhd)?8x_}G$GGS$tnFIu8qFZa%Q zUmIV~E5F|N{k~XUEn>~&?bP>A#qze$w}Q&mg1V0j>K42|d;0CCZ{)2M^dR24hLvx< zYF#xIFPn;Qn@TB0)#p(s7M~*%24u=~?>7B51{%-J?mwfz*JMPm3|9Of8TQLhU>p$vDvap*g05`>E|H6*=*~h;W37Yvz z<)I?|2SwIH_2vzmpU@I?#t5PYB>3MvV}!AljtRqSQrRKt*jGaul@+5!9YCu#yJlTT z0~QVbCFxZluqwLDv%LPS2262-f&gQK>b`LXt5q~n1w24b!Y?^J{2iYBEXmg z=XBH!?wl7`i7C0dfMEouCcVi;M$huNx%;xm7b55qWx6?Z@{&7^S_Qg<$nF3hKtY8d z;w7G`E;G7HM-eAZ!Ao6ed8KpC*PnRxiG}>1>leN6j=wX0LlfKa=#B21ZU3d?qYf$8 zInzZb?!r5)&fK(avque5QQM;TM%AjV>kA`YZ(rRvF zDCPA|PB4c-UfSOwv1H%U;=*@Hh%9%}0FNNO3LnWZieph6kK5JtE=LF?Gud{S1)OB& ztLE8^s~Ioa*Rl%d^5=~U`HT7`-;Im!1>X^+fitTE=avV~t!3p$#knK%rx%VcUV8Ui zi{H9YjrE(=qnTkp1M1l>mF`~3i`hFRQzxA`G?tzVw4GjjOCA8SXauro9c*7;Uk{cM z8FK3NIjnJ<8v?5F$0up(4AR zt~=qsLnxj&xzEn)StZ3k>?EyIER&@fRjmfAB5?kB$K?wPD|q8N$k8 z?)?iqx6aK@gTu$`PHDgwQ#p`cAaFRVHH6W>*(F)y5R#CZsB@{@Gs+t!L%PW&*!6VP zHsOSCVuCuC&c(KB3Hlr`1=#_%>Kl<*@Lg5ZG5G=IbV$d_=e13Ya>gn7DwzSC9z@_0 z5Jv*%9~dA2B<9|q!P~DT$1wOdy9ncOK$~_z)UK0if&XgKfxRS)^1&^P1||j?U~ad} z4wOM%+~gVe2fTt0xGd0HQ0Bf#Z(IklC271r?3)nd8blDe003d&yhujAj+>sASI%|3 z#dB#VoAv|GbqB=GCX(;pAwk>;?~l{WC)ouvht_O%g04g_ym<2Wwk+`D^RMMcBeCr2 z)$DpHyMBJ;w#{)*&*T)+brrXb5Ud&UIzOZiK(c00RYco(>`QIq2!6rHslG!eyBqos zIeQzNkN0;s3>b-K^l zf9hzT(_s|8k8j8lhcXg63=b0s8ebF$HX{509uZxjRYDbss3qRNCmtCG8QD-MVu`6hK=Q(EN|Jy?23 zU9Un5WU1-_sT^`y%xQKC+iptRv!7uubfs+_tE_pRA%|ds*$xEI1%xt{yVh@1TAFA`$gkIZh3m!p79j~)FR>}W9Nc|+>ATpHRItR5nD z+}IB6Yl5I}a=Z}&fdmbn(hqE{6e;Ela(!vXLFLM-JqahiZrac?EtPhTs_pMW`wcF` z!|QZuQ`%Gtj<$-nQ{Ju_ecbRBa7HaIAl$swW!N|}h_kh*Z3U5NAROZquCXdb{f3$| zS`5nE5y@w=hLrKb>|y-LECCvlS^EFg%S_rM`{ub$<=Ub@DJ>aP`@(0@-ejlpze&5) z^$qmtYu0y;(o1z-vEv|4!Yq7Ee{z))d>)zW#ExjitR?p<%jc)A3Z;Zg*YTcO;zVK( zfZiv9rH~OLXjED;2A&P%D#;!exOAA?!VNt%_NxR~a=8=zA--7QfMdjh=LhYLSSN#W=FS6L z#K4VQ0a~{Ki2x{PC=$_uz;ueF(gjeSI%w(yg0@MGZzW7Zt+0UN!jIvB>7;{d2srGo z;4Ov7iJSHZ+!G^ScW1m5xV3kf>N=@sy^{Esq*OkHyt z-a$_#@`>}lD@1Tixn4TKjjl(CuN4K*!Q^}ueo8!gbQ0`AI7mFQSSUXo!vp0dhY3&2 z55Y*F>GUR2F}F}BqHPCzA8SCQlrfYG+>>5WBy5KSqg`SVEpC>In-}=SymyPVkgtIi1t{{s`T{>f-9nB+y}uNh;uORpNEJKx&%#;%2eA0JrkS+3dp;$t^<-f;b9 z&+6eb%ZJZMXQx&VUz~kx=CN0e)7_Ho(x*o2bXdlj>{-!*m#)lpt=PHwL-V6QIf{mq zy;HU{c0-8mJa)VG*r&P0bMDuDulnYDUl0C8aItE!_wTmfC|GGZ@=xW}bDFi%ve$RL zx@*4ZhX)r%ZAKk~kuD5L!@gTjj!J`L(pXU9 zLMvq#)@o{B>xEpkq6TDs>DGnXg@T1=mkuqB{}0EP4&1a#&L^aSbJDpEke&H8+j4kJr{F11K2=RAa+cj78gM;&{DwN zv_R(0E#xg27DlDgU9n6kkhHI{=I4%m``B`B$MmrkcF&p>XzEKH)2IG)w~@)XtohU3 zS|(#$BW}Z5`Jk@wXeIkAZ7K1q4e)S6NH}J+L70XLa0ZEsADupgd(N0rwH}qu3Lwcf~^}Kl#4w3vmB%%-PM~i#{rL>ro zU{d>_jw;DLRpHyY^uOG9t=Qjk*F~eFayoVOYTqu zHvzC87V-2paDj^xO>qm22|@QM95+nS7mzKZ^66tCg`gVAHM--bi({T~B3F#)8r_ou zfulruMnEy5kvS@Y>On4r9QxAI*eLjjfFT4y*}s(*{Zc-#acm_gGF;~^V#$5KPi^7w%@ij5Gh~reFLL6&Dd7h{B_niQxZKb znMzmKGQ50j+Oopd5jo<9`U+k+FxPY2TK(_}i=u6Fd6KDOg{_q99h`5IO!X_QgS-|; zdnHr(ZMH(Dg;IAB!A{Bhy+^Rdra^gcJS_3P;gC#S)oE#Nb#FYBhH-`ndz%$5o*8S^ zVplwZU155-)@M3OAgN!mY42kqQ))}YzDU*0!0vxE!2!@y%nUd_@H3-J58vc6!Z%Z# z)Y4)NnN#pME6r^qX)P*k1(=YD@uaRFnBQgxP4f&s(`9bbrQ`$iXR~{FDsShr6y|>j z8uuZq%hFGfjZIFW4@duk;-CdeaynfQ=qjkQSII!&0K8dtE~5gA9~54NOfFNus=uO? zNYU$fh|WjI#%Gg%3hT6zIcrm9(hFp6gO0C_=Tf1!(xQ;nsqNpA{H_w>a|V^F7^ii} zqfm@aE9NuTspqR2$R4ykjg)SfZ`UP@i3%keOcoQ<>wG`b2y z8D~}~O-z}UgUOy3xiZkpVi%jb6VRqsrA>TEQ|j56j4$QOT$&B1Q!=d_DJd&5rM3#b zazo1hhjqu7 z5H)b16h$Pv8Ct@s=6Dly`XgBzqe2%W`*3J+jX^F%jyXsN_HiCm#5InJ11C6g$`K=m zT}r4Cdi#@<>H<}^HBzvZdzK6Pd|YS3vI5`WI=M)`1M#7YT;n7v;u`&23)ct=Ovkl! zMv9RH0}U_N==E@o&K-D#;Y4usz}CpVt=x!spzS2*9}T$24vbC(0`$a1DtY(%1(1U< zLYTaGU{t&^>46;x%pt@BlaO;nOoRX=B;uDW*VxI$*%7xmj?>=b#IZ|`9DzUuArD?( z+?dE5(QiTJTqKJaP*4(**mi)Lj1(lyR|r*^5EBw@Yyn_f79&LR<+0{q_cGxQd&c*2 zW1uKOC=z-I-zO~;eoeA}2v1hpg!FO5C;Wy){sX)}{43la7_gp&LcBUkWWwtQB)>K& zQse*{QkVTAR6Gp=v>jktLFup}l11MyYe4lM?e7M<0_81X0P%v8cw{3|%_jl)A0sO} zSzPYGl&lg?1sSk6@J|Yq&q@`g=m>Endn55Eijpb(ClY=g9=-u_DoA7@AiPT=3E_&4 zPECwL_2S~FK;TW>M3xc}n$YYXDpvvrm)JCbryUH76uUCsz;f68=BN$My`&gr^U$Wz|Sq zcF)wTW#+s9Bx&86F}wTfZmQ5)wQ4O}v6ijdGG~uIe>Ccb*q$IrcMME!pXR3L=IJ|( zWR#g1D;Z?&AO|NIZni(sq zkLrmoSLy#|&s#4jp4&37{gLAblTjTyW6qd0T{T6EUpqXno9BOcVqsvRdoh18e8YHa zcdyibR66>&^u&<#_^`C?soN&^n$h~)pMU$$=c-nWrJ%^9^47(!cTc@@N^0%8@#u~4 zZ=d<#nbjj_mXDl?9eI59$nf%!;n)#qUMrF8wL~PI z9k*o>33&FGBH0KZ8gq}Cm=8-TkGAPQ++#=hH|8ApAGKMJ>CGP9;DXMOV6OhNSvT>j%%U`Lr@KI zm?_mJe1QJ-ZJ0C`F}e8NBUsf&a(^S?p`dpT$St8SLV%m_SH$~kcqz;{;cxIBFQdFG zLL-Vvp_{^p`>&(~fpgQ$U)RVCxSuw?T}f#n;b)|{VN|5-P!p+vgk;t!dvVkqhD>`F zc_=FmM>?2oKB(XFr%pMCdj^Ow3`Tu%6ATT9kDNZ*h1>)U&c;PMze}bP#x(UOcqH?IKS!1??uUf(1nR;Vt!S&;BEg6=#c2eezl`wA( zu4U&-Y(CLJ&5d?1S9Yvs>`Bt2MK2tDZEVH5Wuf-38o+p(=~}a8%ywVxj&6(D%2#bQ zlC5SwXVtoG*}5%eZTPwAwzXsFvG-5^;`EJq%0WoYVmDtih-L*3rk7z`)$i7#2Sj+yE0p12HHsDI~PJNMfY7oBc(kW08X)aXS zQtSdDPS&JIDk$a!ZjQD|ds5THkYi49XMiuJzK3)vyH$r;)x8RZDLodJ!3h?uiJs}h zu-VX4u7vPI)Q2@k5I&;wvVv#aGy%Q`kRDM!h*{xBBvucLTJRpoPNVWqSr)A5VhI0C z;)(aa;wSEchbd(Z&mI5H@u+^Dy~S=>v*ye=-U`pLbKYyuF6P{_v@PitpS@){JbjqJ zU0t+ol`WE~b{xH`BZEz0-+yIEozO|wZZxFYNVU7P3iU(jI+T%JI`y=4kc3KebS)|_ zuFw@@z`H@Ck(N-mnOgvwfK8Ty>`4O{`M(23>Qbn40>?MGbe17!YBu&Mt>S?SW0n5G z1|`{Bl`cg8wYazPHl>9_YFm{U^hKrU!6DRb9C^@q>Pi5W z{C^}kjs#z&4%`hQ&$LrNi~89uDhB~|gBq0`P0LBv1{)r->8R_kRYJ)pcqh58)I4>` zG$o{rlCnm<4!TB@b3@ve*P>0%P(N~PcFNhgVI=aEL+OD&6r}XQ_>FxibfxBbNFTrx zOO9$$Dh2H_sr{q0ICwBBHJ^TM2=mN~&xQnY6!#S}4{Jp>LD? zKIbM*AQa@Z%2-o6SbL&Lp@YAXzILnhbxMnz8(O5)AKBC+5> zd<$+zfYtpZDC$TqX)7_;_fJk;CWiQ<{;^2q&_MTzqkX5(9;o+Cjs^t}DCuW`l+a5Z zF8UmijOHfyR5%E}w0p!Ch;$-5DUVwY+#^t-g;Z9)@BzGdm=Y6(I^2ekEC7+LAYWNP zFGq;jX(NIn7!34YiAd2`ApkbmBbfEfR{kfs)!O17LIRuFe9f6p_-@|hh$VVKd zSIIFcPmA6UH9J87>I~E>K<%4_FUX4#(NyA07B3D@LeCRxXN-N4*p_K(0>5O#OReA| z+jM246oW%RH1E)==$#0lpdtJu2mmyaU>Kwwg^ec0Hf{-u!->Ad zH4`GyDN(G162ecI7eH8=h28SskSKnJ0O(vx#q?pQ`s%k+L;Yy`OM$t=D<^m4VfZMn8>aUfQ^Czi8!<}jg#k3WC>HSO!hSBXZ8ng?pdxjSj#f16lwoa@tjr4s=aO7@_7a%llQY2vu$Q9>VY((gwlf2ZeAL2N&Fe< z%w@@TWyKU(&&;Kl8I(3jdCjrR9Wy#Y%kO!9kCeZADfhjR_r1UHu38U&Y(*EcqKBiq z<}#%0I<(#f&F7Y7Tg!^Ajp##X^{?0-ThA|<^UR%|ZhD*ZfIxf=swbGh;OzYEwO%-XGZ)-<@Tcux(#@)4V6)>>`~yTAa3v~yL5PN zI_ue_&Y23`DYom<$kZXd4MrQ%0g@Y)x`|42D%OD0noYvxQb283$5sIEItjeXz*iy! zjjfRJ8(K%0beo)$XssBWL>^*v5(`xo+frJa(4T36_MD04Gt+!*NDoU*qh~f5PwM*BkP<^IrD%IB$&)u}JN%IR)@1&MWMwIK#}6rMOJ6o*awbpXgQVrW z`5Q);unjjkC+<&5okyom!N$_vZ2=uPvyz_LkB~lPgn_%35jl>E`|ue~UFxaNXVyG4 zJ*A{QiG8HA9)ny=#YEG~miR1ZY98=cHalh2VueCGF=TOC(#c1XGOCBkF8Wm*Gx-&# zitlV%gL@z?OpcKq5~-Z8`L>Er@wrXv>IV+|L;?p+ttF4o2hLP5SfjMSsYxrRDpasA z<;fQTw<>n(n=DEUWL(z%GE^uw1riOjR? zn_7xISTK@(GMa# z7}pE}rzEl4n#FPd#O3z3CRDA)jR{jDN?bn8_bAvs&BQy4tSCsapGXOCO}ubKAsV+V zQAIc0b;WfL^v5lzkX$D&QozX7ldN1^<8Wvr*3^Dk zQ##}@665O;KeX!%QzKB(m?9=tp7U4YW_Mu1P4UpU!2>$KBAo81;FBg4^E# z35g@0GDPT@3;&8Qr=Wnh@ET{w0zqmz9v~N<$m(hoo1=0c7MNAsL^xi+{RF{)SHVO) zj#Nctm;ws~rP*E-Lxn6}(8Wa-Pnkee}llGifR^-2DdQO$a8{`KA0 zc9Sbx<_G4x7xEV_EO-{P7uy!=r0PAZxqDY~_pVje&0m=JEaWV7EwoGJEzzFe=M@vJ zuOZDBFeodydhz#J`M}olTVd*$Rj{60czw_J_Ylleszv_YvD}XL+TP#upZDC<-I$V2 z3`yO?v3*b7%6)3ZGq&mpeC!EGAyEp3fi6#Jj%o3j)Ew8|1OHibLi>L+Ox_9YOxLIA zS#kNB+P92v7~itLVP9kyANxDwQf`bpu*!8WbKN&QH``)d|E=QwsNqvP48xmMJnXiq z><;c5DS(2Df?=(g%09DX%se|Y39-YfrFPj;3$|8-EYIf{ik~u!&27% z+qMHlaCB57e|Zo}H+l2n1?SIO7sYoky>n^lQf&8;SZnXiTB-lEbmpA&q(`!OVNamYF+qCfWt_d=sptbU!n`ZCo_Q z>N}*ZdzP{#>pqFy_s9FY6L@I-T6PW86I&(w)?22n#OP07|JD6kb!33FekmTtdi+`5 z(H7A9x7&|Z z=r-K0MA%9c1--$NCH$Z|Pg=EZ^;mmQw9v!|ElI{eIz%>UvQ8BL)1tLnTnP!Z6ql)> zRm6d@kVIhBV;MGegfyehrQfIwuO5YllssvTgLswI^k8q2W)v<%$IDp!IGn-2RFPeL zb+dB5nJNQ8MV~jOqGvb)t*L7DDQZEsG__=aJ+3}f5zfhl@Ui+eod5Bf`*NmnH zO7ai16fCWcBdj8soAWB)#4M=n%`;mz_8it_WF5hMZ9B4uJbcckZ>op-*M0L6Y85Vr z%r4`o%$QTho(IB|jx9F#>Y3gd@ z%U#B_ZLCmw&R3?e1R)-PNYS*h4zK`odCqC&tDF|TdQe5c`^hQ0$)!UY`5NV0YVR>3 zKs;+v^A^4?$=3b1!V4GMKm-{Hm$Zv-Hiv938^kkPWpxKLW(O)8xXQ*&?xp-P6FeO# zpAh{LqHzitju1vAvXJl%5+Vp;M9bAj%4ACc2M4u~KmbTu7@DPt+feI)Gf_JNN5q&g zM}S3xZ1QkGc#~9N;E3_TfiI0vJSCrt8wsBXp-7>k;X*lr@-3w#tuN99tWds?ROLD* zP($}9P#&t-Lh7Qb6??ge7T0{HE_7ID2oM z^GZ+nB;TksHoiI*SRM;V6H~FVi_++&TW2o=At71+LHkDLx|WUf zRueEBX$KZaJcf#)?H2wogyIIQc1WUyRy3Fzdy5%}R+*MWdYnLC)yWX{A#XE&Vi!eU z*K+csqpxhA6JNjd>ZSQhvC8Jv%3V_BuEmy_E$f*%v;OD(FPxt~vIeb<-f#ELoSUnX z${QEkrh8Y|j&*BZ)cL|eh?K3FGvR2@OV7sWEv81xOxtY7)sE<~n6+}%S}R#==Xb`8 zTNc2SL$qS<@&fxa)7z%SqFDW2Y3shF_8YY~3M3O+ax|rRbC-bD_59@7`AMmK*Ai{$ zBkOEdG!NY@qxvWIC_C5wN?^VVde2J1$LF+CnVP+@YM%0DV};GjY~FO&%oG%EzVj^6 zf+;|ScUVY@jh zxt>}HOs)pR<$$;vxUw9$g7x$a^jE-kpU|ArLal&2mPf`Be_DG^yJjzXZOfbNTjsws z|Jb@Pc8lA!cp+A@=Z;=$-#2{(YD>1<>AnZ$&V+;_$KX98K6wzmgduu*ctq|LC%4Y3 zH}(PbAoWlO=`>a?anOCyZL|R0Aiv*WH>#8ooL}=z;}DK|s*$?+%SqHzmE=}3LytiE zsG^_H_y!LHmYGBo+?wQcqTT89*pf)(5HxG};#BQomm&3_OClAhSw&WXW@*#Pv@?{R z@|i>*$JbcAcA$f83=pbKfF-3fjX&TpUx>1WC~WC5D6ai)^rl1C-4 zMaP#UzqVma@}cXD)+lWT0#pcTcoC$~1b}UFe36n5Qs!^yeKWZ}pi~B;^#m$9;5mhD zBuI{XS^tq(h~RJ)wj(-{3td+K=m9Eii}-{fQEZK55Vh8wC!x6t=Es3Z;n%6K;^_DJ z&L4n99W*F$b2OxNaSLuP0LI~Q3tr)UeAAzWK^Dkq7s(_UIQkXHq5=_)ey088Ms?W~ z9a-Vmq|qM|kDdZ=l28nufBG>E1KYn%r_fE3PWUL0a=9-j`6I~LLB=3?#L()JLjtwZ z5&1kfp|o;@$ft(|r$<6CCLBgBLB3vHF3Ue!SIv) zvC17YwMvPz(F?ddq-(xiD%rNkl7nRzh1V-=0kz}DAs|5?(F?`Q1Bz)Z{O!w8L(Et>*S=z` zQn>|E*0#m26I2)^NUozZae{1@{bh9A#NWWx#Oky9C zAt#1gbp(H)kTXY?4PlIB*OqvOK&+?L8&vfZA0%h_uOSeXT_x(0(CTYyS9fYsOHhGN zDw;2A7^H8VS{_7zMl>XK8`aX-_aKqT$w8eGKZdP?crq^3L28v46To8dqf=7!$yzrO zSCDO}1T4|TQCf~05|Z^=aq-WlyF(!d(eRn61QsE`2lJG+q`N&8@+Oeosb%`fZrtRQ z*@V5i4i@XWW@ttBWHHPooR>lyE)=TTgI*@&WEg?R+eWm_0db$}H1K)56DM`Z z=rpEXp~bixg3tnIA6~6Yp6__d39|7QsJ(6Ghagc1CoBjN-0UG?5`T-91>maZbf2oB0Q4(A|fIrG1 z_`H#vQK-{Uqg2Gxzm37gb@=NT5E^a%^B9pFk9(5dPe6)r$PZfFhI}rt}ru+Ew1GpH|eI=41cbgO4S41RP zCWG|vh`?4h;z+iYs`NlDiQLhETk-J6Rm5@1g?MDYKh)`nn`QMUPZ(xN^4~duX$VnW zMm@OZ$b{^}xQRfFo2T%-VWO$z$fH-3Q%nPeWHp|nWL^mCNS4wZBO$Wk>9u^qGbD7F zcwNMMnRr9QBcv%sFz6j!DTXlwgyN=O0@P$Q<{SyN(^ukAtcDCExl~^a#kIbcxNh=t zShOIHP{Fd3u^D#44GWsH_!d!HOBk!*R{R(C0E+>|JAQRMdO4QKt!CCqnRWAzu4XhX zXEZH{v5Z!bZGez)7_t1vtMX`J^zRK zAmJq2o)y#H-`le&u2S}zYYry1#DJIF*q^db6?my-E~K6*4M6W zYg`!po9#XJcd{6( zeYXB;eYAb9J!Ywv^^qIrbkQl85`)@SFfQAwvHW9W`F!qz;Z0b~U|@Eym^#*Qr9u14 zCuc6e2zj1(s&R;HP*Msax#-R2YvYR%$@J(7d-(TO+aK>YGPeBtj27dEdNMf}PwVFQ zOST;=rk($bJ?kq_r~2&w=bnLd`j4>oA(;*OT0&5dM=xCSNw%^TQ#mHUT)PJO*uHD~ z<}Uwr?!sd~E%>>(wDWH-M)$>XJ74PhOU&aIl4#xQ8SD9kM`(EOs5Xv-)Z#7 zPwv4Z3m7-k;XV)mfVQ~q@~9ZskAww0NKC6k>pzQ=wg*6jMUx!bzCYyQDPPK3A2$vU zdxIX_j#o+d8%bFq9uc|<#7a#d+eYvaZvbB02J11XH8uh41_hyqL}ezT5?6oAwAC$A zf}GB>@g5-v1quo%0Qf)2>whC21qD>t+elvb5O0up6bj6e&@YKciJ8wx2unC2LCzwL zBqG=pXhHU}@Eo#`?)$@UGqTYJ5GR1z{qkLboq`v+@ShR?2_9r^8qMddRm0vZVl=i- z80#mDnf}lIgt2_WWYL)1Pni6F+|lx`@g3v4neSvS{n_e{zK?hGecL literal 0 HcmV?d00001 diff --git a/agent/build-image.sh b/agent/build-image.sh index ebf86c5..fe3478b 100644 --- a/agent/build-image.sh +++ b/agent/build-image.sh @@ -68,8 +68,15 @@ pv jq mdadm lvm2 -nvme-cli efibootmgr +zstd +gdisk +parted +xfsprogs +e2fsprogs +btrfs-progs +dosfstools +bash firmware-linux firmware-linux-nonfree firmware-misc-nonfree @@ -88,6 +95,7 @@ mkdir -p config/includes.chroot/usr/local/bin config/includes.chroot/etc/systemd config/includes.chroot/root/.ssh config/includes.chroot/etc/modules-load.d \ config/hooks/live install -m 0755 "$SRC/rollcall-agent.py" config/includes.chroot/usr/local/bin/rollcall-agent.py +install -m 0755 "$SRC/rollcall-deploy.py" config/includes.chroot/usr/local/bin/rollcall-deploy.py install -m 0644 "$SRC/rollcall-agent.service" config/includes.chroot/etc/systemd/system/rollcall-agent.service printf 'ipmi_msghandler\nipmi_devintf\nipmi_si\nipmi_ssif\n' > config/includes.chroot/etc/modules-load.d/ipmi.conf if [ -r "$AUTHKEYS" ]; then diff --git a/agent/rollcall-agent.py b/agent/rollcall-agent.py index fe55573..1a4bd1e 100644 --- a/agent/rollcall-agent.py +++ b/agent/rollcall-agent.py @@ -10,6 +10,7 @@ Kernel cmdline options: rollcall.nopoweroff stay up (debugging) rollcall.bmcuser=commissioner BMC account name to create rollcall.token=... agent token (sent as X-Rollcall-Token) + rollcall.mode=deploy rollcall.job=.. run rollcall-deploy.py instead (OS deployment / release job) """ import glob import json @@ -441,6 +442,9 @@ def post(server, payload, tries=30): def main(): opts = cmdline() + if opts.get("mode") == "deploy": + # same image, different job: hand over to the deploy/release agent + os.execv("/usr/bin/python3", ["python3", "/usr/local/bin/rollcall-deploy.py"]) server = opts.get("server") if not server: log("rollcall.server= missing on kernel cmdline; nothing to do") diff --git a/agent/rollcall-deploy.py b/agent/rollcall-deploy.py new file mode 100644 index 0000000..4bebc01 --- /dev/null +++ b/agent/rollcall-deploy.py @@ -0,0 +1,385 @@ +#!/usr/bin/env python3 +"""Deploy/release agent: runs inside the commissioning live image when the +kernel cmdline carries rollcall.mode=deploy rollcall.job=. + +deploy : stream //disk.raw.zst onto the target disk (sha256 + verified), seed cloud-init NoCloud files into the written root fs, + add a UEFI boot entry when booted via UEFI, report, reboot. +release: wipe the target disk (quick: signatures + first/last 64 MiB), report, + power off. + +Everything the agent needs (image URL, checksum, disk, seed files) comes from +GET /api/v1/jobs/self; it never decides policy itself. +""" +import glob +import json +import os +import re +import subprocess +import sys +import threading +import time +import urllib.error +import urllib.request + +LOG = open("/dev/console", "w", buffering=1) if os.path.exists("/dev/console") else sys.stdout +SERVER = "" +TOKEN = "" +EFI_GUID = "c12a7328-f81e-11d2-ba4b-00a0c93ec93b" +ROOT_FS = ("ext4", "xfs", "btrfs", "ext3") + + +def log(msg): + line = f"[deploy] {msg}" + print(line, file=LOG) + if LOG is not sys.stdout: + print(line, flush=True) + + +def run(cmd, timeout=600, check=False, **kw): + p = subprocess.run(cmd, shell=isinstance(cmd, str), capture_output=True, text=True, timeout=timeout, **kw) + if check and p.returncode != 0: + raise RuntimeError(f"{cmd if isinstance(cmd, str) else ' '.join(cmd)}: rc={p.returncode} {p.stderr.strip()[-400:]}") + return p + + +def readf(path, default=""): + try: + with open(path) as f: + return f.read().strip() + except OSError: + return default + + +def cmdline(): + opts = {} + for tok in readf("/proc/cmdline").split(): + if tok.startswith("rollcall."): + k, _, v = tok[len("rollcall."):].partition("=") + opts[k] = v if v else True + return opts + + +# --------------------------------------------------------------------------- server +def api(method, path, body=None, tries=20): + data = json.dumps(body).encode() if body is not None else None + req = urllib.request.Request(SERVER + path, data=data, method=method, + headers={"Content-Type": "application/json", "X-Rollcall-Job": TOKEN}) + last = None + for i in range(tries): + try: + with urllib.request.urlopen(req, timeout=30) as r: + return json.load(r) + except urllib.error.HTTPError as e: + if 400 <= e.code < 500: + raise RuntimeError(f"{path}: HTTP {e.code} {e.read().decode(errors='replace')[:200]}") + last = e + except (urllib.error.URLError, OSError) as e: + last = e + time.sleep(3) + raise RuntimeError(f"{path}: {last}") + + +class Progress: + def __init__(self): + self.phase, self.pct, self.msg = "starting", 0, "" + self.lock = threading.Lock() + self.stop = threading.Event() + self.thread = threading.Thread(target=self._loop, daemon=True) + self.thread.start() + + def set(self, phase=None, pct=None, msg=None, now=False): + with self.lock: + if phase is not None: + self.phase = phase + if pct is not None: + self.pct = pct + if msg is not None: + self.msg = msg + if now: + self._post() + + def _post(self): + with self.lock: + body = {"phase": self.phase, "pct": round(self.pct, 1), "msg": self.msg} + try: + api("POST", "/api/v1/jobs/self/progress", body, tries=1) + except Exception as e: # noqa: BLE001 + log(f"progress post failed: {e}") + + def _loop(self): + while not self.stop.wait(3): + self._post() + + def finish(self, state, msg="", error="", details=None): + self.stop.set() + body = {"state": state, "phase": state, "pct": 100 if state == "done" else self.pct, + "msg": msg, "error": error, "details": details or {}} + return api("POST", "/api/v1/jobs/self/progress", body) + + +# --------------------------------------------------------------------------- disks +def lsblk(dev=None): + cmd = ["lsblk", "-J", "-b", "-o", "NAME,PATH,TYPE,SIZE,SERIAL,RM,ROTA,TRAN,FSTYPE,PARTTYPE,PARTN,PKNAME,MOUNTPOINT"] + if dev: + cmd.append(dev) + p = run(cmd) + if p.returncode != 0: + return [] + out = [] + + def walk(items): + for d in items: + out.append(d) + walk(d.get("children") or []) + walk(json.loads(p.stdout).get("blockdevices", [])) + return out + + +def find_disk(spec): + want, serial, need = spec.get("disk") or "", spec.get("disk_serial") or "", int(spec.get("raw_size") or 0) + disks = [d for d in lsblk() if d.get("type") == "disk" and not d["name"].startswith(("loop", "ram", "zram", "sr"))] + if want: + for d in disks: + if d["path"] == want or d["name"] == want.replace("/dev/", ""): + if serial and d.get("serial") and d["serial"] != serial: + # device names move around; trust the serial over the name + for e in disks: + if e.get("serial") == serial: + log(f"{want} has serial {d['serial']}, expected {serial}; using {e['path']} instead") + return e + raise RuntimeError(f"{want} has serial {d['serial']}, expected {serial}; refusing to write") + return d + if serial: + for d in disks: + if d.get("serial") == serial: + log(f"{want} not found; serial {serial} is {d['path']}") + return d + raise RuntimeError(f"target disk {want} not present") + cands = [d for d in disks if not d.get("rm") and d.get("tran") != "usb" and int(d["size"]) >= need] + if not cands: + raise RuntimeError(f"no disk >= {need} bytes") + cands.sort(key=lambda d: (int(d["size"]), d["name"])) + return cands[0] + + +def release_disk(path): + """Make sure nothing holds the disk: swap, mounts, md, lvm.""" + for d in lsblk(path): + if d.get("mountpoint"): + run(["umount", "-f", d["path"]]) + run("swapoff -a") + run(["vgchange", "-an"]) + run(["mdadm", "--stop", "--scan"]) + for d in lsblk(path): + if d.get("type") == "part": + run(["wipefs", "-a", d["path"]]) + run(["wipefs", "-a", path]) + run(["blockdev", "--rereadpt", path]) + run(["udevadm", "settle"]) + + +# --------------------------------------------------------------------------- deploy +def write_image(spec, disk, prog): + url, want, size = spec["image_url"], spec["sha256"], int(spec.get("size") or 0) + sumf, pvf = "/run/rollcall.sum", "/run/rollcall.pv" + for f in (sumf, pvf): + if os.path.exists(f): + os.remove(f) + pipeline = (f"set -o pipefail; curl -fsSL --retry 5 --retry-connrefused '{url}' " + f"| tee >(sha256sum > {sumf}) " + f"| pv -n -f -i 2 -s {size} 2>{pvf} " + f"| zstd -dc -T0 " + f"| dd of='{disk}' bs=4M iflag=fullblock oflag=direct conv=fsync status=none") + log(f"writing {url} -> {disk}") + proc = subprocess.Popen(["bash", "-c", pipeline], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True) + pct = 0 + while proc.poll() is None: + time.sleep(2) + try: + with open(pvf) as f: + lines = f.read().split() + if lines: + pct = int(lines[-1]) + except (OSError, ValueError): + pass + prog.set("writing", min(pct, 99), f"{pct}% of {size // (1 << 20)} MiB written to {disk}") + if proc.returncode != 0: + raise RuntimeError(f"image write failed rc={proc.returncode}: {proc.stderr.read().strip()[-400:]}") + got = readf(sumf).split()[0] if readf(sumf) else "" + if got != want: + raise RuntimeError(f"sha256 mismatch: got {got} want {want}") + run(["sync"]) + run(["blockdev", "--rereadpt", disk]) + run(["partprobe", disk]) + run(["udevadm", "settle"]) + time.sleep(2) + + +def find_parts(disk): + parts = [d for d in lsblk(disk) if d.get("type") == "part"] + root = esp = None + for p in sorted(parts, key=lambda d: -int(d["size"])): + if p.get("fstype") in ROOT_FS and root is None: + root = p + if (p.get("parttype") or "").lower() == EFI_GUID and p.get("fstype") == "vfat" and esp is None: + esp = p + return root, esp, parts + + +def mount(dev, where, opts=None): + os.makedirs(where, exist_ok=True) + cmd = ["mount"] + (["-o", opts] if opts else []) + [dev, where] + run(cmd, check=True) + + +def seed(root, files): + mnt = "/mnt/rollcall-root" + # xfs root written from an image can carry a duplicate UUID with nothing else; nouuid is harmless + try: + mount(root["path"], mnt) + except RuntimeError: + mount(root["path"], mnt, "nouuid") + try: + if not os.path.isdir(f"{mnt}/etc"): + raise RuntimeError(f"{root['path']} has no /etc; not a root filesystem") + for f in files: + dst = os.path.join(mnt, f["path"].lstrip("/")) + os.makedirs(os.path.dirname(dst), exist_ok=True) + with open(dst, "w") as fh: + fh.write(f["content"]) + os.chmod(dst, int(f.get("mode", "0644"), 8)) + log(f"wrote {f['path']}") + # cloud-init's stale state would stop it from re-running on a reused image + run(f"rm -rf {mnt}/var/lib/cloud/instance {mnt}/var/lib/cloud/instances/* {mnt}/var/lib/cloud/data/* 2>/dev/null") + sel = readf(f"{mnt}/etc/selinux/config") + if re.search(r"^SELINUX=(enforcing|permissive)", sel, re.M): + open(f"{mnt}/.autorelabel", "w").close() + log("SELinux enabled in image: scheduled autorelabel for first boot") + os_release = readf(f"{mnt}/etc/os-release") + m = re.search(r'^PRETTY_NAME="?([^"\n]*)', os_release, re.M) + run(["sync"]) + return m.group(1) if m else "" + finally: + run(["umount", mnt]) + + +def efi_entry(disk, esp, label): + if not os.path.isdir("/sys/firmware/efi") or esp is None: + return None + mnt = "/mnt/rollcall-esp" + try: + mount(esp["path"], mnt, "ro") + except RuntimeError as e: + return f"esp mount failed: {e}" + try: + loader = None + for name in ("shimx64.efi", "grubx64.efi"): + for d in ("debian", "ubuntu", "rocky", "almalinux", "centos", "redhat", "fedora"): + if os.path.exists(f"{mnt}/EFI/{d}/{name}"): + loader = f"\\EFI\\{d}\\{name}" + break + if loader: + break + if not loader: + hits = glob.glob(f"{mnt}/EFI/*/shimx64.efi") + glob.glob(f"{mnt}/EFI/*/grubx64.efi") + glob.glob(f"{mnt}/EFI/BOOT/BOOTX64.EFI") + if hits: + loader = "\\" + os.path.relpath(hits[0], mnt).replace("/", "\\") + if not loader: + return "no EFI loader found on ESP" + finally: + run(["umount", mnt]) + # drop stale entries with our label, then create a new one first in BootOrder + cur = run(["efibootmgr"]).stdout + for m in re.finditer(r"^Boot([0-9A-Fa-f]{4})\*?\s+(.*)$", cur, re.M): + if m.group(2).strip().startswith(label): + run(["efibootmgr", "-q", "-b", m.group(1), "-B"]) + p = run(["efibootmgr", "-c", "-d", disk, "-p", str(esp.get("partn")), "-L", label, "-l", loader]) + if p.returncode != 0: + return f"efibootmgr failed: {p.stderr.strip()[-200:]}" + return f"{label} -> {loader} (part {esp.get('partn')})" + + +def do_deploy(job, prog): + spec, files = job["spec"], job.get("files") or [] + started = time.time() + prog.set("preparing", 0, "selecting disk", now=True) + disk = find_disk(spec) + size = int(disk["size"]) + if int(spec.get("raw_size") or 0) > size: + raise RuntimeError(f"{disk['path']} ({size} bytes) smaller than image ({spec['raw_size']} bytes)") + log(f"target {disk['path']} serial={disk.get('serial')} size={size}") + prog.set("preparing", 1, f"wiping {disk['path']}", now=True) + release_disk(disk["path"]) + write_image(spec, disk["path"], prog) + prog.set("seeding", 99, "writing cloud-init seed", now=True) + root, esp, parts = find_parts(disk["path"]) + if root is None: + raise RuntimeError("no root filesystem found after writing image: " + ", ".join(f"{p['name']}:{p.get('fstype')}" for p in parts)) + pretty = seed(root, files) + efi = efi_entry(disk["path"], esp, f"rollcall {spec.get('os', '')}") + if efi: + log(f"efi: {efi}") + details = {"disk": disk["path"], "serial": disk.get("serial"), "root": root["path"], "root_fs": root.get("fstype"), + "esp": esp["path"] if esp else None, "efi": efi, "os_release": pretty, "firmware": "uefi" if os.path.isdir("/sys/firmware/efi") else "bios", + "seconds": round(time.time() - started, 1)} + return details + + +def do_release(job, prog): + spec = job["spec"] + prog.set("preparing", 0, "selecting disk", now=True) + disk = find_disk(spec) + path, size = disk["path"], int(disk["size"]) + prog.set("wiping", 5, f"wiping {path}", now=True) + release_disk(path) + mib = 64 + run(f"dd if=/dev/zero of={path} bs=1M count={mib} conv=fsync status=none") + run(f"dd if=/dev/zero of={path} bs=1M count={mib} seek={max(0, size // (1 << 20) - mib)} conv=fsync status=none") + if spec.get("wipe") == "full": + p = run(["blkdiscard", "-f", path], timeout=3600) + if p.returncode != 0: + prog.set("wiping", 10, "full zero fill", now=True) + run(f"dd if=/dev/zero of={path} bs=16M oflag=direct status=none", timeout=24 * 3600) + run(["blockdev", "--rereadpt", path]) + return {"disk": path, "serial": disk.get("serial"), "wipe": spec.get("wipe")} + + +def main(): + global SERVER, TOKEN + opts = cmdline() + SERVER = (opts.get("server") or "").rstrip("/") + TOKEN = opts.get("job") if isinstance(opts.get("job"), str) else "" + if not SERVER or not TOKEN: + log("rollcall.server= and rollcall.job= required") + return 2 + log(f"fetching job from {SERVER}") + try: + job = api("GET", "/api/v1/jobs/self") + except Exception as e: # noqa: BLE001 + log(f"cannot fetch job: {e}; staying up") + return 1 + log(f"job {job['id']} kind={job['kind']} spec={json.dumps({k: v for k, v in job['spec'].items() if k not in ('ssh_keys', 'user_data')})}") + prog = Progress() + try: + details = do_deploy(job, prog) if job["kind"] == "deploy" else do_release(job, prog) + resp = prog.finish("done", "finished", details=details) + except Exception as e: # noqa: BLE001 + log(f"FAILED: {e!r}") + try: + resp = prog.finish("failed", str(e), error=str(e)) + except Exception as e2: # noqa: BLE001 + log(f"could not report failure: {e2}") + resp = {"next": "stay"} + nxt = resp.get("next", "stay") + log(f"server says: {nxt}") + if opts.get("nopoweroff") or nxt == "stay": + log("staying up for inspection (ssh root@)") + return 0 if nxt != "stay" else 1 + time.sleep(3) + run(["systemctl", "reboot" if nxt == "reboot" else "poweroff"]) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/cmd/rollcall/client.go b/cmd/rollcall/client.go index c024564..b3a3e58 100644 --- a/cmd/rollcall/client.go +++ b/cmd/rollcall/client.go @@ -99,9 +99,19 @@ func runClient(cmd string, args []string) error { if err := c.do("GET", "/api/v1/machines", nil, &ms); err != nil { return err } - f := "%-16s %-12s %-10s %-10s %-14s %-24s %4s %7s %3s %7s %3s %2s %3s %s\n" - fmt.Printf(f, "ID", "STATE", "NODE", "PROFILE", "MGMT-IP", "MODEL", "CPU", "RAM", "DSK", "STORAGE", "NIC", "IB", "GPU", "BMC") + f := "%-16s %-14s %-10s %-16s %-14s %-24s %4s %7s %3s %7s %3s %2s %3s %s\n" + fmt.Printf(f, "ID", "STATE", "NODE", "PROFILE/OS", "MGMT-IP", "MODEL", "CPU", "RAM", "DSK", "STORAGE", "NIC", "IB", "GPU", "BMC") for _, m := range ms { + state := m.State + if m.Job != nil { + var p obj + _ = json.Unmarshal(m.Job.Progress, &p) + state = fmt.Sprintf("%s %.0f%%", m.Job.Kind, num(p["pct"])) + } + what := deref(m.Profile) + if what == "" && m.OS != nil { + what = *m.OS + } bmc := "-" if m.BMC != nil { b := *m.BMC @@ -112,7 +122,7 @@ func runClient(cmd string, args []string) error { bmc = fmt.Sprintf("%s (%s,%s)", str(b["address"]), str(b["source"]), ok) } model := strings.TrimSpace(deref(m.Manufacturer) + " " + deref(m.Product)) - fmt.Printf(f, m.ID, m.State, deref(m.NodeName), deref(m.Profile), deref(m.MgmtIP), trunc(model, 24), m.CPUs, human(m.MemoryBytes), + fmt.Printf(f, m.ID, state, deref(m.NodeName), trunc(what, 16), deref(m.MgmtIP), trunc(model, 24), m.CPUs, human(m.MemoryBytes), fmt.Sprint(int(m.Disks)), human(m.StorageBytes), fmt.Sprint(m.NICs), fmt.Sprint(m.IBHCAs), fmt.Sprint(m.GPUs), bmc) } case "show": @@ -177,12 +187,142 @@ func runClient(cmd string, args []string) error { return err } fmt.Println("deleted", str(d["deleted"])) + case "deploy", "release", "jobs", "job": + return c.deployCmd(cmd, args) default: return c.statelessCmd(cmd, args) } return nil } +func (c *client) deployCmd(cmd string, args []string) error { + switch cmd { + case "deploy": + if len(args) < 2 { + usage() + } + fs := newFlags("deploy") + ver := fs.String("version", "", "image version (default: current)") + disk := fs.String("disk", "", "target disk, e.g. /dev/sda (default: smallest disk that fits)") + host := fs.String("hostname", "", "hostname (default: existing node name or next free)") + upgrade := fs.Bool("upgrade", false, "cloud-init package_upgrade on first boot") + pkgs := fs.String("packages", "", "comma-separated packages to install on first boot") + ud := fs.String("user-data", "", "file with a complete #cloud-config to use instead of the generated one") + boot := fs.Bool("boot", false, "set bootdev pxe and power-cycle via BMC") + watch := fs.Bool("watch", false, "follow job progress") + _ = fs.Parse(args[2:]) + req := obj{"os": args[1], "version": *ver, "disk": *disk, "hostname": *host, "package_upgrade": *upgrade, "boot": *boot} + if *pkgs != "" { + req["packages"] = strings.Split(*pkgs, ",") + } + if *ud != "" { + b, err := os.ReadFile(*ud) + if err != nil { + return err + } + req["user_data"] = string(b) + } + var d obj + if err := c.do("POST", "/api/v1/machines/"+args[0]+"/deploy", req, &d); err != nil { + return err + } + fmt.Printf("%s: job %s %s/%s -> %s hostname=%s mgmt=%s\n", str(d["id"]), str(d["job"]), str(d["os"]), str(d["version"]), str(d["disk"]), str(d["hostname"]), str(d["mgmt_ip"])) + if d["ipmi"] != nil { + fmt.Println(str(d["ipmi"])) + } + if *watch { + return c.watchJob(args[0]) + } + case "release": + if len(args) < 1 { + usage() + } + fs := newFlags("release") + boot := fs.Bool("boot", false, "set bootdev pxe and power-cycle via BMC") + wipe := fs.String("wipe", "quick", "quick (signatures + first/last 64 MiB) or full") + watch := fs.Bool("watch", false, "follow job progress") + _ = fs.Parse(args[1:]) + var d obj + if err := c.do("POST", "/api/v1/machines/"+args[0]+"/release", obj{"boot": *boot, "wipe": *wipe}, &d); err != nil { + return err + } + fmt.Printf("%s: job %s wipe %s\n", str(d["id"]), str(d["job"]), str(d["disk"])) + if d["ipmi"] != nil { + fmt.Println(str(d["ipmi"])) + } + if *watch { + return c.watchJob(args[0]) + } + case "jobs": + if len(args) < 1 { + usage() + } + if len(args) > 1 && args[1] == "-watch" { + return c.watchJob(args[0]) + } + var jobs []Job + if err := c.do("GET", "/api/v1/machines/"+args[0]+"/jobs", nil, &jobs); err != nil { + return err + } + fmt.Printf("%-10s %-8s %-10s %-20s %-20s %s\n", "JOB", "KIND", "STATE", "CREATED", "FINISHED", "OS / PROGRESS / ERROR") + for _, j := range jobs { + sp := j.spec() + var p obj + _ = json.Unmarshal(j.Progress, &p) + fin := "" + if j.Finished != nil { + fin = j.Finished.Local().Format("2006-01-02 15:04:05") + } + extra := sp.OS + "/" + sp.Version + " " + sp.Disk + if j.State == "running" { + extra += fmt.Sprintf(" %s %.0f%% %s", str(p["phase"]), num(p["pct"]), str(p["msg"])) + } + if j.Error != nil { + extra += " " + *j.Error + } + fmt.Printf("%-10s %-8s %-10s %-20s %-20s %s\n", j.ID, j.Kind, j.State, j.Created.Local().Format("2006-01-02 15:04:05"), fin, extra) + } + case "job": + if len(args) < 2 || args[0] != "cancel" { + usage() + } + var d obj + if err := c.do("POST", "/api/v1/machines/"+args[1]+"/jobs/cancel", nil, &d); err != nil { + return err + } + fmt.Printf("job %s cancelled\n", str(d["job"])) + } + return nil +} + +func (c *client) watchJob(id string) error { + last := "" + for { + var jobs []Job + if err := c.do("GET", "/api/v1/machines/"+id+"/jobs", nil, &jobs); err != nil { + return err + } + if len(jobs) == 0 { + return fmt.Errorf("no jobs") + } + j := jobs[0] + var p obj + _ = json.Unmarshal(j.Progress, &p) + line := fmt.Sprintf("%s %s %-9s %-12s %3.0f%% %s", time.Now().Format("15:04:05"), j.ID, j.State, str(p["phase"]), num(p["pct"]), str(p["msg"])) + if line[9:] != last { + fmt.Println(line) + last = line[9:] + } + if j.State != "pending" && j.State != "running" { + if j.Error != nil { + return fmt.Errorf("%s", *j.Error) + } + return nil + } + time.Sleep(3 * time.Second) + } +} + func (c *client) statelessCmd(cmd string, args []string) error { switch cmd { case "images": diff --git a/cmd/rollcall/config.go b/cmd/rollcall/config.go index 4341199..e3a3db3 100644 --- a/cmd/rollcall/config.go +++ b/cmd/rollcall/config.go @@ -28,6 +28,10 @@ type Config struct { DnsmasqDir string `json:"dnsmasq_dir"` // hosts.dhcp + hosts written here for dnsmasq (dhcp-hostsfile / addn-hosts); "" disables NodePools map[string]string `json:"node_pools"` // {"mgmt": "10.0.0.50-10.0.0.99", "ib": "10.1.0.10-10.1.0.250"} ClusterVars map[string]any `json:"cluster_vars"` // available to overlay templates as .Vars (profile vars override) + + // disk-based deployment + OSDir string `json:"os_dir"` // ///{disk.raw.zst,meta.json}, served under /os/ + OSSources string `json:"os_sources"` // JSON list of upstream cloud images for `rollcall os sync` } // BMCHint maps a machine MAC to a known BMC (virtual lab nodes). @@ -68,5 +72,11 @@ func loadConfig(path string) (*Config, error) { if c.OverlaysDir == "" { c.OverlaysDir = "/etc/rollcall/overlays" } + if c.OSDir == "" { + c.OSDir = filepath.Join(filepath.Dir(c.ImageDir), "os") + } + if c.OSSources == "" { + c.OSSources = "/etc/rollcall/os-sources.json" + } return c, nil } diff --git a/cmd/rollcall/deploy.go b/cmd/rollcall/deploy.go new file mode 100644 index 0000000..4415c10 --- /dev/null +++ b/cmd/rollcall/deploy.go @@ -0,0 +1,643 @@ +package main + +// Disk-based OS deployment: a job boots the commissioning image in deploy +// mode; the agent streams //disk.raw.zst onto the chosen disk, +// seeds cloud-init (NoCloud) in the written root filesystem and reports +// progress. On success the machine boots from disk and cloud-init phones home. + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "log" + "net/http" + "sort" + "strings" + "time" +) + +type Job struct { + ID string `json:"id"` + MachineID string `json:"machine_id"` + Kind string `json:"kind"` // deploy | release + State string `json:"state"` + Spec json.RawMessage `json:"spec"` + Progress json.RawMessage `json:"progress"` + Created time.Time `json:"created"` + Started *time.Time `json:"started"` + Finished *time.Time `json:"finished"` + Error *string `json:"error"` +} + +// DeploySpec is what the agent receives (plus the rendered seed files). +type DeploySpec struct { + OS string `json:"os"` + Version string `json:"version"` + ImageURL string `json:"image_url"` + SHA256 string `json:"sha256"` + Size int64 `json:"size"` + RawSize int64 `json:"raw_size"` + Disk string `json:"disk"` + DiskSerial string `json:"disk_serial,omitempty"` + Hostname string `json:"hostname"` + Domain string `json:"domain,omitempty"` + SSHKeys []string `json:"ssh_keys"` + PackageUpgrade bool `json:"package_upgrade"` + Packages []string `json:"packages,omitempty"` + UserData string `json:"user_data,omitempty"` // full #cloud-config override + Wipe string `json:"wipe,omitempty"` // release: quick | full +} + +type seedFile struct { + Path string `json:"path"` + Mode string `json:"mode"` + Content string `json:"content"` +} + +func (j *Job) spec() DeploySpec { + var s DeploySpec + _ = json.Unmarshal(j.Spec, &s) + return s +} + +// ---------------------------------------------------------------- store + +const jobCols = `id, machine_id, kind, state, spec, progress, created, started, finished, error` + +func scanJob(row interface{ Scan(...any) error }) (*Job, error) { + j := &Job{} + err := row.Scan(&j.ID, &j.MachineID, &j.Kind, &j.State, &j.Spec, &j.Progress, &j.Created, &j.Started, &j.Finished, &j.Error) + if err != nil { + if strings.Contains(err.Error(), "no rows") { + return nil, errNotFound + } + return nil, err + } + return j, nil +} + +func (s *Store) CreateJob(ctx context.Context, j *Job, tokenHash string, sealed []byte) error { + _, err := s.pool.Exec(ctx, `INSERT INTO jobs(id, machine_id, kind, state, spec, token_hash, token) VALUES ($1,$2,$3,'pending',$4,$5,$6)`, + j.ID, j.MachineID, j.Kind, j.Spec, tokenHash, sealed) + return err +} + +func (s *Store) JobByToken(ctx context.Context, hash string) (*Job, error) { + return scanJob(s.pool.QueryRow(ctx, `SELECT `+jobCols+` FROM jobs WHERE token_hash=$1`, hash)) +} + +func (s *Store) Job(ctx context.Context, id string) (*Job, error) { + return scanJob(s.pool.QueryRow(ctx, `SELECT `+jobCols+` FROM jobs WHERE id=$1`, id)) +} + +func (s *Store) ActiveJob(ctx context.Context, machineID string) (*Job, error) { + return scanJob(s.pool.QueryRow(ctx, `SELECT `+jobCols+` FROM jobs WHERE machine_id=$1 AND state IN ('pending','running') ORDER BY created DESC LIMIT 1`, machineID)) +} + +func (s *Store) ActiveJobs(ctx context.Context) (map[string]*Job, error) { + rows, err := s.pool.Query(ctx, `SELECT `+jobCols+` FROM jobs WHERE state IN ('pending','running') ORDER BY created`) + if err != nil { + return nil, err + } + defer rows.Close() + out := map[string]*Job{} + for rows.Next() { + j, err := scanJob(rows) + if err != nil { + return nil, err + } + out[j.MachineID] = j + } + return out, rows.Err() +} + +func (s *Store) Jobs(ctx context.Context, machineID string, limit int) ([]*Job, error) { + rows, err := s.pool.Query(ctx, `SELECT `+jobCols+` FROM jobs WHERE machine_id=$1 ORDER BY created DESC LIMIT $2`, machineID, limit) + if err != nil { + return nil, err + } + defer rows.Close() + out := []*Job{} + for rows.Next() { + j, err := scanJob(rows) + if err != nil { + return nil, err + } + out = append(out, j) + } + return out, rows.Err() +} + +func (s *Store) JobProgress(ctx context.Context, id string, progress []byte) error { + _, err := s.pool.Exec(ctx, `UPDATE jobs SET state='running', started=coalesce(started, now()), progress=$2 WHERE id=$1 AND state IN ('pending','running')`, id, progress) + return err +} + +func (s *Store) FinishJob(ctx context.Context, id, state, errMsg string) error { + _, err := s.pool.Exec(ctx, `UPDATE jobs SET state=$2, finished=now(), started=coalesce(started, now()), error=nullif($3,'') WHERE id=$1`, id, state, errMsg) + return err +} + +func (s *Store) SetDeployed(ctx context.Context, id, os, version string) error { + _, err := s.pool.Exec(ctx, `UPDATE machines SET state='deployed', os=$2, os_version=$3, deployed_at=now(), booted_at=NULL, last_seen=now() WHERE id=$1`, id, os, version) + return err +} + +func (s *Store) SetReleased(ctx context.Context, id string) error { + _, err := s.pool.Exec(ctx, `UPDATE machines SET state='ready', os=NULL, os_version=NULL, deployed_at=NULL, booted_at=NULL, last_seen=now() WHERE id=$1`, id) + return err +} + +func (s *Store) SetNodeAddr(ctx context.Context, id, name, mgmt, ib string) error { + _, err := s.pool.Exec(ctx, `UPDATE machines SET node_name=$2, mgmt_ip=$3, ib_ip=$4 WHERE id=$1`, id, name, nilIfEmpty(mgmt), nilIfEmpty(ib)) + return err +} + +func (s *Store) MarkPhoneHome(ctx context.Context, id string) error { + _, err := s.pool.Exec(ctx, `UPDATE machines SET booted_at=now(), last_seen=now() WHERE id=$1`, id) + return err +} + +// ---------------------------------------------------------------- disk selection + +type invDisk struct { + Name, Path, Serial, Tran, Model string + Size int64 + Rota, Removable bool +} + +func inventoryDisks(m *Machine) []invDisk { + var inv obj + _ = json.Unmarshal(m.Inventory, &inv) + var out []invDisk + for _, dv := range toList(dig(inv, "storage", "disks")) { + d, _ := dv.(obj) + rm, _ := d["rm"].(bool) + rota, _ := d["rota"].(bool) + id := invDisk{Name: str(d["name"]), Path: str(d["path"]), Serial: str(d["serial"]), Tran: str(d["tran"]), + Model: str(d["model"]), Size: int64(num(d["size"])), Rota: rota, Removable: rm} + if id.Path == "" { + id.Path = "/dev/" + id.Name + } + out = append(out, id) + } + return out +} + +// pickDisk: the smallest non-removable, non-USB disk that fits the image (boot/OS disks are +// usually the small ones on servers; data disks stay untouched). Explicit choice wins. +func pickDisk(disks []invDisk, want string, need int64) (*invDisk, error) { + if want != "" { + for i := range disks { + if disks[i].Path == want || disks[i].Name == want || "/dev/"+disks[i].Name == want { + if disks[i].Size > 0 && disks[i].Size < need { + return nil, fmt.Errorf("disk %s (%s) is smaller than the image (%s)", want, human(float64(disks[i].Size)), human(float64(need))) + } + return &disks[i], nil + } + } + if len(disks) == 0 { + return &invDisk{Path: want, Name: strings.TrimPrefix(want, "/dev/")}, nil + } + return nil, fmt.Errorf("disk %s not in inventory", want) + } + var cands []invDisk + for _, d := range disks { + if d.Removable || d.Tran == "usb" || d.Size < need { + continue + } + cands = append(cands, d) + } + if len(cands) == 0 { + return nil, fmt.Errorf("no disk >= %s in inventory (pass disk explicitly)", human(float64(need))) + } + sort.Slice(cands, func(i, j int) bool { + if cands[i].Size != cands[j].Size { + return cands[i].Size < cands[j].Size + } + return cands[i].Name < cands[j].Name + }) + return &cands[0], nil +} + +// ---------------------------------------------------------------- cloud-init seed + +func yamlStr(s string) string { b, _ := json.Marshal(s); return string(b) } +func yamlList(l []string) string { + if l == nil { + l = []string{} + } + b, _ := json.Marshal(l) + return string(b) +} + +func (a *App) renderSeed(j *Job, sp DeploySpec, token string) []seedFile { + meta := fmt.Sprintf("instance-id: %s\nlocal-hostname: %s\n", j.ID, sp.Hostname) + ud := sp.UserData + if ud == "" { + var b strings.Builder + b.WriteString("#cloud-config\n# generated by rollcall for " + j.MachineID + " (" + j.ID + ")\n") + fmt.Fprintf(&b, "hostname: %s\n", yamlStr(sp.Hostname)) + if sp.Domain != "" { + fmt.Fprintf(&b, "fqdn: %s\n", yamlStr(sp.Hostname+"."+sp.Domain)) + } + b.WriteString("manage_etc_hosts: true\npreserve_hostname: false\ndisable_root: false\nssh_pwauth: false\n") + fmt.Fprintf(&b, "ssh_authorized_keys: %s\n", yamlList(sp.SSHKeys)) + fmt.Fprintf(&b, "users:\n - default\n - name: root\n ssh_authorized_keys: %s\n", yamlList(sp.SSHKeys)) + if sp.PackageUpgrade || len(sp.Packages) > 0 { + b.WriteString("package_update: true\n") + } + if sp.PackageUpgrade { + b.WriteString("package_upgrade: true\npackage_reboot_if_required: true\n") + } + if len(sp.Packages) > 0 { + fmt.Fprintf(&b, "packages: %s\n", yamlList(sp.Packages)) + } + fmt.Fprintf(&b, "phone_home:\n url: %s/api/v1/phone-home/%s\n post: [instance_id, hostname, fqdn, pub_key_ed25519]\n tries: 10\n", a.cfg.ServerURL, token) + b.WriteString("final_message: \"rollcall: deployed $hostname up after $UPTIME s\"\n") + ud = b.String() + } + return []seedFile{ + {Path: "var/lib/cloud/seed/nocloud/meta-data", Mode: "0600", Content: meta}, + {Path: "var/lib/cloud/seed/nocloud/user-data", Mode: "0600", Content: ud}, + {Path: "etc/cloud/cloud.cfg.d/99-rollcall.cfg", Mode: "0644", Content: "# written by rollcall at deploy time\ndatasource_list: [ NoCloud, None ]\n"}, + } +} + +// ---------------------------------------------------------------- PXE + +const deployTmpl = `# generated by rollcall %s +# machine %s job %s %s %s/%s -> %s +DEFAULT menu.c32 +PROMPT 0 +TIMEOUT 30 +MENU TITLE %s - %s %s (job %s) + +LABEL deploy + MENU LABEL Run deployment job (default in 3s) + MENU DEFAULT + KERNEL rollcall/vmlinuz + INITRD rollcall/initrd.img + APPEND %s + +LABEL recommission + MENU LABEL Re-run commissioning + KERNEL rollcall/vmlinuz + INITRD rollcall/initrd.img + APPEND %s + +LABEL local + MENU LABEL Boot from local disk + LOCALBOOT 0 +` + +const deployedTmpl = `# generated by rollcall %s +# machine %s deployed %s/%s +DEFAULT menu.c32 +PROMPT 0 +TIMEOUT 30 +MENU TITLE %s - deployed %s (%s) + +LABEL local + MENU LABEL Boot %s from local disk (default in 3s) + MENU DEFAULT + LOCALBOOT 0 + +LABEL recommission + MENU LABEL Re-run commissioning + KERNEL rollcall/vmlinuz + INITRD rollcall/initrd.img + APPEND %s +` + +// deployAppend derives the deploy-mode cmdline from commission_append (same image, no agent token). +func (a *App) deployAppend(token string) string { + var parts []string + for _, p := range strings.Fields(a.cfg.CommissionAppend) { + if strings.HasPrefix(p, "rollcall.token=") || strings.HasPrefix(p, "rollcall.nopoweroff") { + continue + } + parts = append(parts, p) + } + parts = append(parts, "rollcall.mode=deploy", "rollcall.job="+token) + return strings.Join(parts, " ") +} + +func (a *App) writePXEDeploy(m *Machine, j *Job, sp DeploySpec, token string) error { + body := fmt.Sprintf(deployTmpl, time.Now().Format(time.RFC3339), m.ID, j.ID, j.Kind, sp.OS, sp.Version, sp.Disk, + m.ID, j.Kind, sp.OS, j.ID, a.deployAppend(token), a.cfg.CommissionAppend) + return a.writePXEBody(m, body) +} + +func (a *App) writePXEDeployed(m *Machine) error { + body := fmt.Sprintf(deployedTmpl, time.Now().Format(time.RFC3339), m.ID, deref(m.OS), deref(m.OSVersion), + deref(m.NodeName), deref(m.OS), m.ID, deref(m.OS), a.cfg.CommissionAppend) + return a.writePXEBody(m, body) +} + +// ---------------------------------------------------------------- job creation + +type deployRequest struct { + OS string `json:"os"` + Version string `json:"version"` + Disk string `json:"disk"` + Hostname string `json:"hostname"` + SSHKeys []string `json:"ssh_keys"` + PackageUpgrade bool `json:"package_upgrade"` + Packages []string `json:"packages"` + UserData string `json:"user_data"` + Boot bool `json:"boot"` + Wipe string `json:"wipe"` +} + +func (a *App) configSSHKeys() []string { + var keys []string + for _, k := range toList(a.cfg.ClusterVars["ssh_keys"]) { + if s := str(k); s != "" { + keys = append(keys, s) + } + } + return keys +} + +// startJob creates the job row, PXE entry and (optionally) power-cycles the machine. +func (a *App) startJob(ctx context.Context, m *Machine, kind string, sp DeploySpec, boot bool) (obj, error) { + if m.Profile != nil { + return nil, fmt.Errorf("machine is a stateless node (profile %s); unassign it first", *m.Profile) + } + if j, err := a.store.ActiveJob(ctx, m.ID); err == nil { + return nil, fmt.Errorf("job %s (%s, %s) is still active; cancel it first", j.ID, j.Kind, j.State) + } + token := randomToken(16) + sum := sha256.Sum256([]byte(token)) + sealed, err := a.sealer.Seal(token, m.ID) + if err != nil { + return nil, err + } + specB, _ := json.Marshal(sp) + j := &Job{ID: "j-" + randomToken(5), MachineID: m.ID, Kind: kind, State: "pending", Spec: specB} + if err := a.store.CreateJob(ctx, j, hex.EncodeToString(sum[:]), sealed); err != nil { + return nil, err + } + state := "deploying" + if kind == "release" { + state = "releasing" + } + _ = a.store.SetState(ctx, m.ID, state) + if err := a.writePXEDeploy(m, j, sp, token); err != nil { + return nil, err + } + _ = a.store.AddEvent(ctx, m.ID, kind+" requested", fmt.Sprintf("job=%s os=%s/%s disk=%s hostname=%s", j.ID, sp.OS, sp.Version, sp.Disk, sp.Hostname)) + out := obj{"id": m.ID, "job": j.ID, "state": state, "os": sp.OS, "version": sp.Version, "disk": sp.Disk, "hostname": sp.Hostname} + if boot { + out["ipmi"], out["ipmi_ok"] = a.pxeBoot(ctx, m) + } + log.Printf("%s %s: job %s os=%s/%s disk=%s", kind, m.ID, j.ID, sp.OS, sp.Version, sp.Disk) + return out, nil +} + +func (a *App) handleDeploy(w http.ResponseWriter, r *http.Request) { + m, err := a.store.Get(r.Context(), r.PathValue("id")) + if err != nil { + fail(w, 500, err) + return + } + var req deployRequest + if err := json.NewDecoder(r.Body).Decode(&req); err != nil || req.OS == "" { + fail(w, 400, fmt.Errorf("os is required")) + return + } + meta, err := osResolve(a.cfg.OSDir, req.OS, req.Version) + if err != nil { + fail(w, 404, err) + return + } + disk, err := pickDisk(inventoryDisks(m), req.Disk, meta.RawSize) + if err != nil { + fail(w, 409, err) + return + } + // node name + management IP: keep an existing one, otherwise allocate like a stateless node + name, mgmt, ib := deref(m.NodeName), deref(m.MgmtIP), deref(m.IBIP) + if req.Hostname != "" && req.Hostname != name { + name = "" + } + if name == "" { + if name, mgmt, ib, err = a.allocate(r.Context(), str(a.cfg.ClusterVars["name_format"]), req.Hostname); err != nil { + fail(w, 409, err) + return + } + if err := a.store.SetNodeAddr(r.Context(), m.ID, name, mgmt, ib); err != nil { + fail(w, 500, err) + return + } + m, _ = a.store.Get(r.Context(), m.ID) + if err := a.exportDnsmasq(r.Context()); err != nil { + log.Printf("dnsmasq export: %v", err) + } + } + keys := req.SSHKeys + if keys == nil { + keys = a.configSSHKeys() + } + sp := DeploySpec{OS: meta.Name, Version: meta.Version, + ImageURL: a.cfg.ServerURL + "/os/" + meta.Name + "/" + meta.Version + "/disk.raw.zst", + SHA256: meta.SHA256, Size: meta.Size, RawSize: meta.RawSize, Disk: disk.Path, DiskSerial: disk.Serial, + Hostname: name, Domain: str(a.cfg.ClusterVars["domain"]), SSHKeys: keys, PackageUpgrade: req.PackageUpgrade, + Packages: req.Packages, UserData: req.UserData} + out, err := a.startJob(r.Context(), m, "deploy", sp, req.Boot) + if err != nil { + fail(w, 409, err) + return + } + out["mgmt_ip"] = mgmt + writeJSON(w, 200, out) +} + +func (a *App) handleRelease(w http.ResponseWriter, r *http.Request) { + m, err := a.store.Get(r.Context(), r.PathValue("id")) + if err != nil { + fail(w, 500, err) + return + } + var req deployRequest + _ = json.NewDecoder(r.Body).Decode(&req) + if req.Wipe == "" { + req.Wipe = "quick" + } + disk, err := pickDisk(inventoryDisks(m), req.Disk, 0) + if err != nil { + fail(w, 409, err) + return + } + sp := DeploySpec{OS: deref(m.OS), Version: deref(m.OSVersion), Disk: disk.Path, DiskSerial: disk.Serial, Wipe: req.Wipe, Hostname: deref(m.NodeName)} + out, err := a.startJob(r.Context(), m, "release", sp, req.Boot) + if err != nil { + fail(w, 409, err) + return + } + writeJSON(w, 200, out) +} + +func (a *App) handleJobs(w http.ResponseWriter, r *http.Request) { + m, err := a.store.Get(r.Context(), r.PathValue("id")) + if err != nil { + fail(w, 500, err) + return + } + jobs, err := a.store.Jobs(r.Context(), m.ID, 20) + if err != nil { + fail(w, 500, err) + return + } + writeJSON(w, 200, jobs) +} + +func (a *App) handleJobCancel(w http.ResponseWriter, r *http.Request) { + m, err := a.store.Get(r.Context(), r.PathValue("id")) + if err != nil { + fail(w, 500, err) + return + } + j, err := a.store.ActiveJob(r.Context(), m.ID) + if err != nil { + fail(w, 404, fmt.Errorf("no active job")) + return + } + _ = a.store.FinishJob(r.Context(), j.ID, "cancelled", "cancelled by operator") + a.afterJob(r.Context(), m, j, "cancelled", "cancelled by operator") + writeJSON(w, 200, obj{"id": m.ID, "job": j.ID, "state": "cancelled"}) +} + +// afterJob settles machine state and the PXE entry once a job stopped (done/failed/cancelled). +func (a *App) afterJob(ctx context.Context, m *Machine, j *Job, state, errMsg string) { + sp := j.spec() + switch { + case state == "done" && j.Kind == "deploy": + _ = a.store.SetDeployed(ctx, m.ID, sp.OS, sp.Version) + m, _ = a.store.Get(ctx, m.ID) + if err := a.writePXEDeployed(m); err != nil { + log.Printf("pxe write %s: %v", m.ID, err) + } + _ = a.store.AddEvent(ctx, m.ID, "deployed", fmt.Sprintf("job=%s os=%s/%s disk=%s", j.ID, sp.OS, sp.Version, sp.Disk)) + case state == "done" && j.Kind == "release": + _ = a.store.SetReleased(ctx, m.ID) + m, _ = a.store.Get(ctx, m.ID) + _ = a.writePXE(m) + _ = a.store.AddEvent(ctx, m.ID, "released", fmt.Sprintf("job=%s disk=%s wipe=%s", j.ID, sp.Disk, sp.Wipe)) + default: + _ = a.store.SetState(ctx, m.ID, j.Kind+"-failed") + m, _ = a.store.Get(ctx, m.ID) + _ = a.writePXE(m) // next PXE boot: commissioned menu (power off), no retry loop + _ = a.store.AddEvent(ctx, m.ID, j.Kind+" "+state, "job="+j.ID+" "+errMsg) + } + log.Printf("job %s (%s %s): %s %s", j.ID, j.Kind, m.ID, state, errMsg) +} + +// ---------------------------------------------------------------- agent side (job token) + +func (a *App) jobFromRequest(w http.ResponseWriter, r *http.Request) (*Job, string) { + tok := r.Header.Get("X-Rollcall-Job") + if tok == "" { + fail(w, 401, errors.New("X-Rollcall-Job required")) + return nil, "" + } + sum := sha256.Sum256([]byte(tok)) + j, err := a.store.JobByToken(r.Context(), hex.EncodeToString(sum[:])) + if err != nil { + fail(w, 401, errors.New("unknown job token")) + return nil, "" + } + return j, tok +} + +func (a *App) handleJobSelf(w http.ResponseWriter, r *http.Request) { + j, tok := a.jobFromRequest(w, r) + if j == nil { + return + } + if j.State != "pending" && j.State != "running" { + fail(w, 409, fmt.Errorf("job %s is %s", j.ID, j.State)) + return + } + sp := j.spec() + out := obj{"id": j.ID, "machine_id": j.MachineID, "kind": j.Kind, "spec": sp} + if j.Kind == "deploy" { + out["files"] = a.renderSeed(j, sp, tok) + } + if j.State == "pending" { + _ = a.store.JobProgress(r.Context(), j.ID, []byte(`{"phase":"started","pct":0}`)) + _ = a.store.AddEvent(r.Context(), j.MachineID, j.Kind+" started", "job="+j.ID+" from "+r.RemoteAddr) + } + writeJSON(w, 200, out) +} + +func (a *App) handleJobProgress(w http.ResponseWriter, r *http.Request) { + j, _ := a.jobFromRequest(w, r) + if j == nil { + return + } + var p struct { + State string `json:"state"` // "" (progress), done, failed + Phase string `json:"phase"` + Pct float64 `json:"pct"` + Msg string `json:"msg"` + Error string `json:"error"` + Details map[string]any `json:"details"` + } + if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&p); err != nil { + fail(w, 400, err) + return + } + if j.State != "pending" && j.State != "running" { + writeJSON(w, 200, obj{"job": j.ID, "state": j.State, "next": "poweroff"}) + return + } + prog, _ := json.Marshal(obj{"phase": p.Phase, "pct": p.Pct, "msg": p.Msg, "details": p.Details, "updated": time.Now()}) + _ = a.store.JobProgress(r.Context(), j.ID, prog) + m, err := a.store.Get(r.Context(), j.MachineID) + if err != nil { + fail(w, 500, err) + return + } + next := "continue" + switch p.State { + case "done": + _ = a.store.FinishJob(r.Context(), j.ID, "done", "") + a.afterJob(r.Context(), m, j, "done", "") + next = "reboot" + if j.Kind == "release" { + next = "poweroff" + } + case "failed": + if p.Error == "" { + p.Error = p.Msg + } + _ = a.store.FinishJob(r.Context(), j.ID, "failed", p.Error) + a.afterJob(r.Context(), m, j, "failed", p.Error) + next = "stay" + } + writeJSON(w, 200, obj{"job": j.ID, "state": p.State, "next": next}) +} + +// handlePhoneHome is cloud-init's phone_home from the freshly deployed OS (form-encoded POST). +func (a *App) handlePhoneHome(w http.ResponseWriter, r *http.Request) { + sum := sha256.Sum256([]byte(r.PathValue("token"))) + j, err := a.store.JobByToken(r.Context(), hex.EncodeToString(sum[:])) + if err != nil || j.Kind != "deploy" { + fail(w, 404, errNotFound) + return + } + _ = r.ParseForm() + _ = a.store.MarkPhoneHome(r.Context(), j.MachineID) + detail := fmt.Sprintf("job=%s from %s hostname=%s fqdn=%s", j.ID, r.RemoteAddr, r.Form.Get("hostname"), r.Form.Get("fqdn")) + if k := strings.TrimSpace(r.Form.Get("pub_key_ed25519")); k != "" { + detail += " hostkey=" + k + } + _ = a.store.AddEvent(r.Context(), j.MachineID, "phone-home", detail) + log.Printf("phone-home %s: %s", j.MachineID, detail) + writeJSON(w, 200, obj{"ok": true}) +} diff --git a/cmd/rollcall/go.mod b/cmd/rollcall/go.mod index 7ef15e6..f8534e4 100644 --- a/cmd/rollcall/go.mod +++ b/cmd/rollcall/go.mod @@ -11,6 +11,7 @@ require ( github.com/jackc/pgpassfile v1.0.0 // indirect github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect github.com/jackc/puddle/v2 v2.2.2 // indirect + github.com/klauspost/compress v1.18.0 // indirect golang.org/x/sync v0.13.0 // indirect golang.org/x/sys v0.32.0 // indirect golang.org/x/text v0.24.0 // indirect diff --git a/cmd/rollcall/go.sum b/cmd/rollcall/go.sum index f600d5d..abe7afa 100644 --- a/cmd/rollcall/go.sum +++ b/cmd/rollcall/go.sum @@ -9,6 +9,8 @@ github.com/jackc/pgx/v5 v5.7.5 h1:JHGfMnQY+IEtGM63d+NGMjoRpysB2JBwDr5fsngwmJs= github.com/jackc/pgx/v5 v5.7.5/go.mod h1:aruU7o91Tc2q2cFp5h4uP3f6ztExVpyVv88Xl/8Vl8M= github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo= github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= +github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo= +github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= diff --git a/cmd/rollcall/main.go b/cmd/rollcall/main.go index 308acea..c7eff41 100644 --- a/cmd/rollcall/main.go +++ b/cmd/rollcall/main.go @@ -36,10 +36,16 @@ func usage() { rollcall unassign ID rollcall overlay ID [-stage system|runtime] [-extract DIR] rollcall render PROFILE PATH [-stage system|runtime] + rollcall os list | sync [NAME...] [-force] | import NAME FILE [-family F] [-version V] | promote NAME VERSION | rm NAME VERSION + rollcall deploy ID OS [-version V] [-disk /dev/sdX] [-hostname H] [-upgrade] [-user-data FILE] [-boot] [-watch] + rollcall release ID [-boot] [-watch] + rollcall jobs ID [-watch] | job cancel ID env: ROLLCALL_URL (default from config listen), ROLLCALL_TOKEN, ROLLCALL_CONFIG`) os.Exit(2) } +func newFlags(name string) *flag.FlagSet { return flag.NewFlagSet(name, flag.ExitOnError) } + func main() { log.SetFlags(log.Ldate | log.Ltime | log.Lmicroseconds) if len(os.Args) < 2 { @@ -57,11 +63,17 @@ func main() { case "keygen": fmt.Println(randomToken(24)) case "list", "show", "events", "power", "bootdev", "recommission", "delete", - "images", "profile", "assign", "unassign", "overlay", "render": + "images", "profile", "assign", "unassign", "overlay", "render", + "deploy", "release", "jobs", "job": if err := runClient(cmd, args); err != nil { fmt.Fprintln(os.Stderr, "error:", err) os.Exit(1) } + case "os": + if err := osCmd(args); err != nil { + fmt.Fprintln(os.Stderr, "error:", err) + os.Exit(1) + } default: usage() } diff --git a/cmd/rollcall/osimages.go b/cmd/rollcall/osimages.go new file mode 100644 index 0000000..590ec98 --- /dev/null +++ b/cmd/rollcall/osimages.go @@ -0,0 +1,694 @@ +package main + +// OS image catalog for disk-based deployment. +// +// ///disk.raw.zst zstd-compressed raw disk image +// ///meta.json OSMeta +// //current -> symlink flipped atomically by sync/promote +// +// Sources (upstream "latest" URLs + vendor checksum files) live in +// os_sources (default /etc/rollcall/os-sources.json). `rollcall os sync` +// downloads only when the vendor checksum changed, verifies it, converts +// to raw with qemu-img, compresses with zstd and prunes old versions. + +import ( + "bufio" + "context" + "crypto/sha256" + "crypto/sha512" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "os" + "os/exec" + "path" + "path/filepath" + "regexp" + "runtime" + "sort" + "strconv" + "strings" + "time" + + "github.com/klauspost/compress/zstd" +) + +type OSSource struct { + Name string `json:"name"` + URL string `json:"url"` + ChecksumURL string `json:"checksum_url"` + Checksum string `json:"checksum"` // sha256 | sha512 (type of the vendor checksum file) + Family string `json:"family"` // debian | rhel (informational) + Description string `json:"description"` + Track string `json:"track"` // latest (default): new versions become current; pinned: only on promote + Keep int `json:"keep"` // versions to keep (default 2) +} + +type OSMeta struct { + Name string `json:"name"` + Version string `json:"version"` + Family string `json:"family"` + Description string `json:"description"` + SourceURL string `json:"source_url"` + SourceSHA256 string `json:"source_sha256"` + SourceSHA512 string `json:"source_sha512"` + SourceModified string `json:"source_modified"` + SourceFormat string `json:"source_format"` + SHA256 string `json:"sha256"` // of disk.raw.zst + Size int64 `json:"size"` // compressed bytes + RawSize int64 `json:"raw_size"` // virtual disk size the image needs + Imported time.Time `json:"imported"` + Current bool `json:"current,omitempty"` +} + +type OSImage struct { + Name string `json:"name"` + Current *OSMeta `json:"current"` + Versions []OSMeta `json:"versions"` + Source *OSSource `json:"source"` + Warning string `json:"warning,omitempty"` +} + +func loadSources(p string) ([]OSSource, error) { + b, err := os.ReadFile(p) + if err != nil { + if errors.Is(err, os.ErrNotExist) { + return nil, nil + } + return nil, err + } + var src []OSSource + if err := json.Unmarshal(b, &src); err != nil { + return nil, fmt.Errorf("%s: %w", p, err) + } + for i := range src { + if src[i].Track == "" { + src[i].Track = "latest" + } + if src[i].Keep <= 0 { + src[i].Keep = 2 + } + if src[i].Checksum == "" { + src[i].Checksum = "sha256" + } + if strings.ContainsAny(src[i].Name, "/ \t") || src[i].Name == "" { + return nil, fmt.Errorf("%s: bad source name %q", p, src[i].Name) + } + } + return src, nil +} + +var versionRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*$`) + +func osVersions(dir, name string) ([]OSMeta, string) { + ents, err := os.ReadDir(filepath.Join(dir, name)) + if err != nil { + return nil, "" + } + cur, _ := os.Readlink(filepath.Join(dir, name, "current")) + cur = path.Base(cur) + var out []OSMeta + for _, e := range ents { + if !e.IsDir() || !versionRe.MatchString(e.Name()) || strings.HasSuffix(e.Name(), ".tmp") { + continue + } + b, err := os.ReadFile(filepath.Join(dir, name, e.Name(), "meta.json")) + if err != nil { + continue + } + var m OSMeta + if json.Unmarshal(b, &m) != nil { + continue + } + m.Name, m.Version = name, e.Name() + m.Current = e.Name() == cur + out = append(out, m) + } + sort.Slice(out, func(i, j int) bool { return out[i].Imported.After(out[j].Imported) }) + return out, cur +} + +func osCatalog(dir, sourcesFile string) ([]OSImage, error) { + src, err := loadSources(sourcesFile) + if err != nil { + return nil, err + } + byName := map[string]*OSSource{} + for i := range src { + byName[src[i].Name] = &src[i] + } + names := map[string]bool{} + for n := range byName { + names[n] = true + } + if ents, err := os.ReadDir(dir); err == nil { + for _, e := range ents { + if e.IsDir() { + names[e.Name()] = true + } + } + } + out := []OSImage{} + for n := range names { + img := OSImage{Name: n, Source: byName[n]} + vs, cur := osVersions(dir, n) + img.Versions = vs + for i := range vs { + if vs[i].Version == cur { + img.Current = &vs[i] + } + } + switch { + case img.Current == nil && len(vs) == 0: + img.Warning = "not imported" + case img.Current == nil: + img.Warning = "no current version (promote one)" + case time.Since(img.Current.Imported) > 45*24*time.Hour: + img.Warning = fmt.Sprintf("current version is %d days old", int(time.Since(img.Current.Imported).Hours()/24)) + } + out = append(out, img) + } + sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name }) + return out, nil +} + +// osResolve returns the metadata of name/version ("" or "current" = current). +func osResolve(dir, name, version string) (*OSMeta, error) { + if strings.ContainsAny(name, "/ \t") || name == "" { + return nil, fmt.Errorf("bad os name %q", name) + } + vs, cur := osVersions(dir, name) + if version == "" || version == "current" { + version = cur + if version == "" { + return nil, fmt.Errorf("os %s has no current version", name) + } + } + for i := range vs { + if vs[i].Version == version { + if _, err := os.Stat(filepath.Join(dir, name, version, "disk.raw.zst")); err != nil { + return nil, err + } + return &vs[i], nil + } + } + return nil, fmt.Errorf("os %s version %s: %w", name, version, errNotFound) +} + +func osSetCurrent(dir, name, version string) error { + if _, err := os.Stat(filepath.Join(dir, name, version, "disk.raw.zst")); err != nil { + return err + } + link := filepath.Join(dir, name, "current") + tmp := link + ".tmp" + _ = os.Remove(tmp) + if err := os.Symlink(version, tmp); err != nil { + return err + } + return os.Rename(tmp, link) +} + +func osRemove(dir, name, version string) error { + if name == "" || version == "" || !versionRe.MatchString(version) { + return fmt.Errorf("name and version required") + } + _, cur := osVersions(dir, name) + if cur == version { + return fmt.Errorf("%s/%s is the current version; promote another one first", name, version) + } + return os.RemoveAll(filepath.Join(dir, name, version)) +} + +// ---------------------------------------------------------------- import pipeline + +type importOpts struct { + Dir string + Name string + Version string // "" = derive + Family string + Description string + SourceURL string + Modified string + SHA256 string // of the source file, if already known + SHA512 string + Log io.Writer +} + +func logf(w io.Writer, f string, a ...any) { + if w != nil { + fmt.Fprintf(w, f+"\n", a...) + } +} + +// osImportFile converts src (qcow2/raw/vmdk...) into /// and returns the meta. +func osImportFile(ctx context.Context, src string, o importOpts) (*OSMeta, error) { + if !versionRe.MatchString(o.Name) { + return nil, fmt.Errorf("bad name %q", o.Name) + } + info, err := qemuInfo(ctx, src) + if err != nil { + return nil, err + } + if o.SHA256 == "" || o.SHA512 == "" { + logf(o.Log, "hashing %s", src) + if o.SHA256, o.SHA512, err = hashFile(src); err != nil { + return nil, err + } + } + if o.Version == "" { + d := o.Modified + if d == "" { + st, _ := os.Stat(src) + d = st.ModTime().UTC().Format("20060102") + } + o.Version = d + "-" + o.SHA256[:6] + } + if !versionRe.MatchString(o.Version) { + return nil, fmt.Errorf("bad version %q", o.Version) + } + final := filepath.Join(o.Dir, o.Name, o.Version) + if _, err := os.Stat(filepath.Join(final, "meta.json")); err == nil { + return nil, fmt.Errorf("%s/%s already exists", o.Name, o.Version) + } + work := final + ".tmp" + _ = os.RemoveAll(work) + if err := os.MkdirAll(work, 0o755); err != nil { + return nil, err + } + defer os.RemoveAll(work) + + raw := src + if info.Format != "raw" { + raw = filepath.Join(work, "disk.raw") + logf(o.Log, "converting %s (%s, %s virtual) to raw", filepath.Base(src), info.Format, human(float64(info.VirtualSize))) + cmd := exec.CommandContext(ctx, "qemu-img", "convert", "-O", "raw", "-S", "4k", src, raw) + if out, err := cmd.CombinedOutput(); err != nil { + return nil, fmt.Errorf("qemu-img convert: %v: %s", err, strings.TrimSpace(string(out))) + } + } + logf(o.Log, "compressing to disk.raw.zst") + sum, size, err := compressZstd(raw, filepath.Join(work, "disk.raw.zst")) + if err != nil { + return nil, err + } + if raw != src { + _ = os.Remove(raw) + } + m := &OSMeta{Name: o.Name, Version: o.Version, Family: o.Family, Description: o.Description, SourceURL: o.SourceURL, + SourceSHA256: o.SHA256, SourceSHA512: o.SHA512, SourceModified: o.Modified, SourceFormat: info.Format, + SHA256: sum, Size: size, RawSize: info.VirtualSize, Imported: time.Now().UTC()} + mb, _ := json.MarshalIndent(m, "", " ") + if err := os.WriteFile(filepath.Join(work, "meta.json"), mb, 0o644); err != nil { + return nil, err + } + if err := os.Rename(work, final); err != nil { + return nil, err + } + logf(o.Log, "imported %s/%s (%s compressed, %s raw)", o.Name, o.Version, human(float64(size)), human(float64(info.VirtualSize))) + return m, nil +} + +type qemuImgInfo struct { + Format string `json:"format"` + VirtualSize int64 `json:"virtual-size"` +} + +func qemuInfo(ctx context.Context, p string) (*qemuImgInfo, error) { + out, err := exec.CommandContext(ctx, "qemu-img", "info", "--output=json", p).Output() + if err != nil { + return nil, fmt.Errorf("qemu-img info %s: %w", p, err) + } + var i qemuImgInfo + if err := json.Unmarshal(out, &i); err != nil { + return nil, err + } + return &i, nil +} + +func hashFile(p string) (s256, s512 string, err error) { + f, err := os.Open(p) + if err != nil { + return "", "", err + } + defer f.Close() + h1, h2 := sha256.New(), sha512.New() + if _, err := io.Copy(io.MultiWriter(h1, h2), bufio.NewReaderSize(f, 4<<20)); err != nil { + return "", "", err + } + return hex.EncodeToString(h1.Sum(nil)), hex.EncodeToString(h2.Sum(nil)), nil +} + +func compressZstd(src, dst string) (sum string, size int64, err error) { + in, err := os.Open(src) + if err != nil { + return "", 0, err + } + defer in.Close() + out, err := os.Create(dst) + if err != nil { + return "", 0, err + } + defer out.Close() + h := sha256.New() + cw := &countWriter{w: io.MultiWriter(out, h)} + enc, err := zstd.NewWriter(cw, zstd.WithEncoderLevel(zstd.SpeedDefault), zstd.WithEncoderConcurrency(runtime.NumCPU())) + if err != nil { + return "", 0, err + } + if _, err := io.Copy(enc, bufio.NewReaderSize(in, 4<<20)); err != nil { + enc.Close() + return "", 0, err + } + if err := enc.Close(); err != nil { + return "", 0, err + } + if err := out.Sync(); err != nil { + return "", 0, err + } + return hex.EncodeToString(h.Sum(nil)), cw.n, nil +} + +type countWriter struct { + w io.Writer + n int64 +} + +func (c *countWriter) Write(p []byte) (int, error) { + n, err := c.w.Write(p) + c.n += int64(n) + return n, err +} + +// ---------------------------------------------------------------- sync + +// parseChecksums understands "HASH file", "HASH *file" and "ALGO (file) = HASH". +func parseChecksums(txt string) map[string]string { + out := map[string]string{} + bsd := regexp.MustCompile(`^\s*[A-Za-z0-9]+\s*\((.+)\)\s*=\s*([0-9a-fA-F]+)\s*$`) + gnu := regexp.MustCompile(`^\s*([0-9a-fA-F]{32,128})\s+\*?(.+?)\s*$`) + for _, line := range strings.Split(txt, "\n") { + if m := bsd.FindStringSubmatch(line); m != nil { + out[path.Base(m[1])] = strings.ToLower(m[2]) + } else if m := gnu.FindStringSubmatch(line); m != nil { + out[path.Base(m[2])] = strings.ToLower(m[1]) + } + } + return out +} + +func httpGetText(ctx context.Context, url string) (string, error) { + req, _ := http.NewRequestWithContext(ctx, "GET", url, nil) + resp, err := http.DefaultClient.Do(req) + if err != nil { + return "", err + } + defer resp.Body.Close() + if resp.StatusCode != 200 { + return "", fmt.Errorf("%s: %s", url, resp.Status) + } + b, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) + return string(b), err +} + +// upstreamChecksum fetches the vendor checksum for the source's file. +func upstreamChecksum(ctx context.Context, s OSSource) (string, error) { + txt, err := httpGetText(ctx, s.ChecksumURL) + if err != nil { + return "", err + } + sums := parseChecksums(txt) + if v, ok := sums[path.Base(s.URL)]; ok { + return v, nil + } + if len(sums) == 1 { + for _, v := range sums { + return v, nil + } + } + return "", fmt.Errorf("%s: no entry for %s", s.ChecksumURL, path.Base(s.URL)) +} + +func metaChecksum(m *OSMeta, typ string) string { + if typ == "sha512" { + return m.SourceSHA512 + } + return m.SourceSHA256 +} + +// download streams url to dst while hashing; verifies against want (hex, type typ). +func download(ctx context.Context, url, dst, typ, want string, log io.Writer) (s256, s512, modified string, err error) { + req, _ := http.NewRequestWithContext(ctx, "GET", url, nil) + resp, err := http.DefaultClient.Do(req) + if err != nil { + return "", "", "", err + } + defer resp.Body.Close() + if resp.StatusCode != 200 { + return "", "", "", fmt.Errorf("%s: %s", url, resp.Status) + } + if t, err := http.ParseTime(resp.Header.Get("Last-Modified")); err == nil { + modified = t.UTC().Format("20060102") + } + f, err := os.Create(dst) + if err != nil { + return "", "", "", err + } + defer f.Close() + h1, h2 := sha256.New(), sha512.New() + pr := &progressReader{r: resp.Body, total: resp.ContentLength, log: log, last: time.Now()} + if _, err := io.Copy(io.MultiWriter(f, h1, h2), pr); err != nil { + return "", "", "", err + } + s256, s512 = hex.EncodeToString(h1.Sum(nil)), hex.EncodeToString(h2.Sum(nil)) + got := s256 + if typ == "sha512" { + got = s512 + } + if want != "" && got != want { + return "", "", "", fmt.Errorf("%s: %s mismatch: vendor %s, downloaded %s", url, typ, want, got) + } + return s256, s512, modified, nil +} + +type progressReader struct { + r io.Reader + n int64 + total int64 + log io.Writer + last time.Time +} + +func (p *progressReader) Read(b []byte) (int, error) { + n, err := p.r.Read(b) + p.n += int64(n) + if p.log != nil && time.Since(p.last) > 10*time.Second { + p.last = time.Now() + if p.total > 0 { + logf(p.log, " %s / %s (%d%%)", human(float64(p.n)), human(float64(p.total)), p.n*100/p.total) + } else { + logf(p.log, " %s", human(float64(p.n))) + } + } + return n, err +} + +// osSync refreshes the named sources (all when names is empty). +func osSync(ctx context.Context, cfg *Config, names []string, force bool, log io.Writer) error { + src, err := loadSources(cfg.OSSources) + if err != nil { + return err + } + if len(src) == 0 { + return fmt.Errorf("no sources in %s", cfg.OSSources) + } + want := map[string]bool{} + for _, n := range names { + want[n] = true + } + var failed []string + for _, s := range src { + if len(want) > 0 && !want[s.Name] { + continue + } + if err := syncOne(ctx, cfg, s, force, log); err != nil { + logf(log, "%s: FAILED: %v", s.Name, err) + failed = append(failed, s.Name) + } + } + if len(failed) > 0 { + return fmt.Errorf("failed: %s", strings.Join(failed, ", ")) + } + return nil +} + +func syncOne(ctx context.Context, cfg *Config, s OSSource, force bool, log io.Writer) error { + vs, cur := osVersions(cfg.OSDir, s.Name) + up, err := upstreamChecksum(ctx, s) + if err != nil { + return err + } + for i := range vs { + if metaChecksum(&vs[i], s.Checksum) == up { + if !force { + logf(log, "%s: up to date (%s, upstream %s unchanged)", s.Name, vs[i].Version, s.Checksum) + if cur == "" && s.Track == "latest" { + return osSetCurrent(cfg.OSDir, s.Name, vs[i].Version) + } + return osPrune(cfg.OSDir, s, log) + } + break + } + } + tmpDir := filepath.Join(cfg.OSDir, ".tmp") + if err := os.MkdirAll(tmpDir, 0o755); err != nil { + return err + } + dl := filepath.Join(tmpDir, s.Name+"-"+path.Base(s.URL)) + defer os.Remove(dl) + logf(log, "%s: downloading %s", s.Name, s.URL) + s256, s512, mod, err := download(ctx, s.URL, dl, s.Checksum, up, log) + if err != nil { + return err + } + logf(log, "%s: vendor %s verified", s.Name, s.Checksum) + m, err := osImportFile(ctx, dl, importOpts{Dir: cfg.OSDir, Name: s.Name, Family: s.Family, Description: s.Description, + SourceURL: s.URL, Modified: mod, SHA256: s256, SHA512: s512, Log: log}) + if err != nil { + return err + } + if s.Track == "latest" || cur == "" { + if err := osSetCurrent(cfg.OSDir, s.Name, m.Version); err != nil { + return err + } + logf(log, "%s: current -> %s", s.Name, m.Version) + } else { + logf(log, "%s: %s imported but pinned to %s (rollcall os promote %s %s)", s.Name, m.Version, cur, s.Name, m.Version) + } + return osPrune(cfg.OSDir, s, log) +} + +// osPrune keeps the newest `keep` versions plus current; nothing younger than a day is removed +// so a deploy job that pinned a version keeps its file. +func osPrune(dir string, s OSSource, log io.Writer) error { + vs, cur := osVersions(dir, s.Name) + kept := 0 + for _, v := range vs { + if v.Version == cur || kept < s.Keep { + kept++ + continue + } + if time.Since(v.Imported) < 24*time.Hour { + continue + } + logf(log, "%s: pruning %s", s.Name, v.Version) + if err := os.RemoveAll(filepath.Join(dir, s.Name, v.Version)); err != nil { + return err + } + } + return nil +} + +// ---------------------------------------------------------------- CLI (local, filesystem side) + +func osCmd(args []string) error { + if len(args) < 1 { + usage() + } + cfg, err := loadConfig(envOr("ROLLCALL_CONFIG", "/etc/rollcall/config.json")) + if err != nil && args[0] != "list" { + return fmt.Errorf("os %s needs the server config: %w", args[0], err) + } + ctx := context.Background() + switch args[0] { + case "list": + if c, cerr := newClient(); cerr == nil { + var imgs []OSImage + if err := c.do("GET", "/api/v1/os", nil, &imgs); err != nil { + return err + } + printOSList(imgs) + return nil + } + imgs, err := osCatalog(cfg.OSDir, cfg.OSSources) + if err != nil { + return err + } + printOSList(imgs) + case "sync": + fs := newFlags("os sync") + force := fs.Bool("force", false, "re-import even when the upstream checksum is unchanged") + _ = fs.Parse(args[1:]) + return osSync(ctx, cfg, fs.Args(), *force, os.Stdout) + case "import": + if len(args) < 3 { + usage() + } + fs := newFlags("os import") + fam := fs.String("family", "", "debian | rhel") + desc := fs.String("description", "", "description") + ver := fs.String("version", "", "version label (default -)") + cur := fs.Bool("current", true, "make it the current version") + _ = fs.Parse(args[3:]) + m, err := osImportFile(ctx, args[2], importOpts{Dir: cfg.OSDir, Name: args[1], Version: *ver, Family: *fam, Description: *desc, + SourceURL: "file://" + args[2], Log: os.Stdout}) + if err != nil { + return err + } + if *cur { + return osSetCurrent(cfg.OSDir, m.Name, m.Version) + } + case "promote": + if len(args) < 3 { + usage() + } + return osSetCurrent(cfg.OSDir, args[1], args[2]) + case "rm": + if len(args) < 3 { + usage() + } + return osRemove(cfg.OSDir, args[1], args[2]) + default: + usage() + } + return nil +} + +func printOSList(imgs []OSImage) { + fmt.Printf("%-18s %-20s %-7s %8s %8s %-6s %-19s %s\n", "OS", "CURRENT", "FAMILY", "ZST", "RAW", "TRACK", "IMPORTED", "VERSIONS / WARNING") + for _, i := range imgs { + cur, fam, zst, raw, imp, track := "-", "", "", "", "", "" + if i.Source != nil { + track, fam = i.Source.Track, i.Source.Family + } + if i.Current != nil { + cur, zst, raw = i.Current.Version, human(float64(i.Current.Size)), human(float64(i.Current.RawSize)) + imp = i.Current.Imported.Local().Format("2006-01-02 15:04") + if fam == "" { + fam = i.Current.Family + } + } + var vs []string + for _, v := range i.Versions { + vs = append(vs, v.Version) + } + extra := strconv.Itoa(len(i.Versions)) + " (" + strings.Join(vs, ", ") + ")" + if i.Warning != "" { + extra += " ! " + i.Warning + } + fmt.Printf("%-18s %-20s %-7s %8s %8s %-6s %-19s %s\n", i.Name, cur, fam, zst, raw, track, imp, extra) + } +} + +// ---------------------------------------------------------------- HTTP + +func (a *App) handleOS(w http.ResponseWriter, r *http.Request) { + imgs, err := osCatalog(a.cfg.OSDir, a.cfg.OSSources) + if err != nil { + fail(w, 500, err) + return + } + writeJSON(w, 200, imgs) +} diff --git a/cmd/rollcall/server.go b/cmd/rollcall/server.go index 6e61ec7..47edda6 100644 --- a/cmd/rollcall/server.go +++ b/cmd/rollcall/server.go @@ -81,7 +81,7 @@ func logging(next http.Handler) http.Handler { start := time.Now() rw := &statusWriter{ResponseWriter: w, code: 200} next.ServeHTTP(rw, r) - if (!strings.HasPrefix(r.URL.Path, "/rollcall/") && !strings.HasPrefix(r.URL.Path, "/images/")) || rw.code >= 400 { + if (!strings.HasPrefix(r.URL.Path, "/rollcall/") && !strings.HasPrefix(r.URL.Path, "/images/") && !strings.HasPrefix(r.URL.Path, "/os/")) || rw.code >= 400 { log.Printf("%s %s %s %d %s", r.RemoteAddr, r.Method, r.URL.RequestURI(), rw.code, time.Since(start).Round(time.Millisecond)) } }) @@ -120,10 +120,20 @@ func (a *App) routes() http.Handler { mux.HandleFunc("GET /api/v1/machines/{id}/overlay", a.adminAuth(a.handleAdminOverlay)) mux.HandleFunc("GET /api/v1/render/{profile}/{path...}", a.adminAuth(a.handleRender)) mux.HandleFunc("GET /api/v1/nodes/self/overlay", a.handleNodeOverlay) // node token auth inside + // disk-based deployment + mux.HandleFunc("GET /api/v1/os", a.adminAuth(a.handleOS)) + mux.HandleFunc("POST /api/v1/machines/{id}/deploy", a.adminAuth(a.handleDeploy)) + mux.HandleFunc("POST /api/v1/machines/{id}/release", a.adminAuth(a.handleRelease)) + mux.HandleFunc("GET /api/v1/machines/{id}/jobs", a.adminAuth(a.handleJobs)) + mux.HandleFunc("POST /api/v1/machines/{id}/jobs/cancel", a.adminAuth(a.handleJobCancel)) + mux.HandleFunc("GET /api/v1/jobs/self", a.handleJobSelf) // job token auth inside + mux.HandleFunc("POST /api/v1/jobs/self/progress", a.handleJobProgress) // job token auth inside + mux.HandleFunc("POST /api/v1/phone-home/{token}", a.handlePhoneHome) // image files for the live boot (no auth: the kernel/initrd/squashfs must be fetchable) imgDir := a.cfg.ImageDir mux.Handle("GET /rollcall/", http.StripPrefix("/rollcall/", http.FileServer(http.Dir(imgDir)))) mux.Handle("GET /images/", http.StripPrefix("/images/", http.FileServer(http.Dir(a.cfg.ImagesDir)))) + mux.Handle("GET /os/", http.StripPrefix("/os/", http.FileServer(http.Dir(a.cfg.OSDir)))) // embedded web UI sub, _ := fs.Sub(webFS, "web") mux.Handle("GET /", http.FileServer(http.FS(sub))) @@ -225,6 +235,10 @@ type Summary struct { MgmtIP *string `json:"mgmt_ip"` IBIP *string `json:"ib_ip"` BootedAt *time.Time `json:"booted_at"` + OS *string `json:"os"` + OSVersion *string `json:"os_version"` + DeployedAt *time.Time `json:"deployed_at"` + Job *Job `json:"job,omitempty"` // active deployment job, if any } func num(v any) float64 { @@ -241,7 +255,8 @@ func summarize(m *Machine) Summary { MemoryBytes: num(dig(inv, "memory", "total_bytes")), Disks: num(dig(inv, "storage", "disk_count")), StorageBytes: num(dig(inv, "storage", "total_bytes")), NICs: listLen(inv["network"]), IBHCAs: listLen(dig(inv, "infiniband", "hcas")), GPUs: listLen(inv["gpus"]), - Profile: m.Profile, NodeName: m.NodeName, MgmtIP: m.MgmtIP, IBIP: m.IBIP, BootedAt: m.BootedAt} + Profile: m.Profile, NodeName: m.NodeName, MgmtIP: m.MgmtIP, IBIP: m.IBIP, BootedAt: m.BootedAt, + OS: m.OS, OSVersion: m.OSVersion, DeployedAt: m.DeployedAt} if m.BMC != nil { var b obj if json.Unmarshal(m.BMC, &b) == nil { @@ -415,9 +430,12 @@ func (a *App) handleList(w http.ResponseWriter, r *http.Request) { fail(w, 500, err) return } + active, _ := a.store.ActiveJobs(r.Context()) out := make([]Summary, 0, len(ms)) for _, m := range ms { - out = append(out, summarize(m)) + s := summarize(m) + s.Job = active[m.ID] + out = append(out, s) } writeJSON(w, 200, out) } @@ -429,11 +447,16 @@ func (a *App) handleGet(w http.ResponseWriter, r *http.Request) { return } evs, _ := a.store.Events(r.Context(), m.ID) + jobs, _ := a.store.Jobs(r.Context(), m.ID, 10) + s := summarize(m) + if j, err := a.store.ActiveJob(r.Context(), m.ID); err == nil { + s.Job = j + } var bmc, inv any _ = json.Unmarshal(m.BMC, &bmc) _ = json.Unmarshal(m.Inventory, &inv) - writeJSON(w, 200, obj{"summary": summarize(m), "bmc": bmc, "has_bmc_secret": m.bmcSecret != nil, - "inventory": inv, "events": evs}) + writeJSON(w, 200, obj{"summary": s, "bmc": bmc, "has_bmc_secret": m.bmcSecret != nil, + "inventory": inv, "events": evs, "jobs": jobs}) } func (a *App) handleDelete(w http.ResponseWriter, r *http.Request) { diff --git a/cmd/rollcall/stateless.go b/cmd/rollcall/stateless.go index 6548b72..c21bab2 100644 --- a/cmd/rollcall/stateless.go +++ b/cmd/rollcall/stateless.go @@ -185,7 +185,7 @@ func parsePool(spec string) ([]string, error) { return out, nil } -func (a *App) allocate(ctx context.Context, p *Profile, wantName string) (name, mgmt, ib string, err error) { +func (a *App) allocate(ctx context.Context, nameFormat, wantName string) (name, mgmt, ib string, err error) { ms, err := a.store.List(ctx) if err != nil { return @@ -204,7 +204,7 @@ func (a *App) allocate(ctx context.Context, p *Profile, wantName string) (name, } name = wantName if name == "" { - f := p.NameFormat + f := nameFormat if f == "" { f = "node%03d" } @@ -601,6 +601,9 @@ func (a *App) writePXEAny(ctx context.Context, m *Machine) error { } return a.writePXEStateless(m, p, token) } + if m.State == "deployed" { + return a.writePXEDeployed(m) + } return a.writePXE(m) } @@ -722,7 +725,7 @@ func (a *App) handleAssign(w http.ResponseWriter, r *http.Request) { // keep the existing name/IPs when re-assigning an already named node name, mgmt, ib := deref(m.NodeName), deref(m.MgmtIP), deref(m.IBIP) if name == "" || (req.Name != "" && req.Name != name) { - if name, mgmt, ib, err = a.allocate(r.Context(), p, req.Name); err != nil { + if name, mgmt, ib, err = a.allocate(r.Context(), p.NameFormat, req.Name); err != nil { fail(w, 409, err) return } diff --git a/cmd/rollcall/store.go b/cmd/rollcall/store.go index f8b0790..aa92361 100644 --- a/cmd/rollcall/store.go +++ b/cmd/rollcall/store.go @@ -56,6 +56,25 @@ ALTER TABLE machines ADD COLUMN IF NOT EXISTS node_token bytea; ALTER TABLE machines ADD COLUMN IF NOT EXISTS host_key bytea; ALTER TABLE machines ADD COLUMN IF NOT EXISTS booted_at timestamptz; CREATE INDEX IF NOT EXISTS machines_node_token ON machines(node_token_hash); +ALTER TABLE machines ADD COLUMN IF NOT EXISTS os text; +ALTER TABLE machines ADD COLUMN IF NOT EXISTS os_version text; +ALTER TABLE machines ADD COLUMN IF NOT EXISTS deployed_at timestamptz; +CREATE TABLE IF NOT EXISTS jobs ( + id text PRIMARY KEY, + machine_id text NOT NULL, + kind text NOT NULL, + state text NOT NULL DEFAULT 'pending', + spec jsonb NOT NULL, + progress jsonb, + token_hash text, + token bytea, + created timestamptz NOT NULL DEFAULT now(), + started timestamptz, + finished timestamptz, + error text +); +CREATE INDEX IF NOT EXISTS jobs_machine ON jobs(machine_id, created); +CREATE INDEX IF NOT EXISTS jobs_token ON jobs(token_hash); ` var errNotFound = errors.New("not found") @@ -81,6 +100,9 @@ type Machine struct { MgmtIP *string `json:"mgmt_ip"` IBIP *string `json:"ib_ip"` BootedAt *time.Time `json:"booted_at"` + OS *string `json:"os"` + OSVersion *string `json:"os_version"` + DeployedAt *time.Time `json:"deployed_at"` bmcSecret []byte nodeToken []byte hostKey []byte @@ -104,14 +126,16 @@ func openStore(ctx context.Context, url string) (*Store, error) { } const machineCols = `id, uuid, serial, hostname, manufacturer, product, state, first_seen, last_seen, - commissioned_at, macs, bmc, bmc_secret, inventory, profile, node_name, mgmt_ip, ib_ip, node_token, host_key, booted_at` + commissioned_at, macs, bmc, bmc_secret, inventory, profile, node_name, mgmt_ip, ib_ip, node_token, host_key, booted_at, + os, os_version, deployed_at` func scanMachine(row pgx.Row, full bool) (*Machine, error) { m := &Machine{} var macs []byte err := row.Scan(&m.ID, &m.UUID, &m.Serial, &m.Hostname, &m.Manufacturer, &m.Product, &m.State, &m.FirstSeen, &m.LastSeen, &m.CommissionedAt, &macs, &m.BMC, &m.bmcSecret, &m.Inventory, - &m.Profile, &m.NodeName, &m.MgmtIP, &m.IBIP, &m.nodeToken, &m.hostKey, &m.BootedAt) + &m.Profile, &m.NodeName, &m.MgmtIP, &m.IBIP, &m.nodeToken, &m.hostKey, &m.BootedAt, + &m.OS, &m.OSVersion, &m.DeployedAt) if errors.Is(err, pgx.ErrNoRows) { return nil, errNotFound } @@ -220,6 +244,9 @@ func (s *Store) Delete(ctx context.Context, id string) error { if _, err := tx.Exec(ctx, `DELETE FROM events WHERE machine_id=$1`, id); err != nil { return err } + if _, err := tx.Exec(ctx, `DELETE FROM jobs WHERE machine_id=$1`, id); err != nil { + return err + } tag, err := tx.Exec(ctx, `DELETE FROM machines WHERE id=$1`, id) if err != nil { return err diff --git a/cmd/rollcall/web/app.js b/cmd/rollcall/web/app.js index 50f3aa5..4452040 100644 --- a/cmd/rollcall/web/app.js +++ b/cmd/rollcall/web/app.js @@ -36,6 +36,7 @@ $('#back').onclick = () => { current = null; $('#detail').hidden = true; $('#lis function setStatus(msg, bad) { const el = $('#status'); el.textContent = msg; el.className = bad ? 'bad' : 'muted'; } let profiles = []; +let osImages = []; async function loadList() { if (!token) return askToken(); try { @@ -56,6 +57,12 @@ async function loadProfiles() { const sel = $('#assign-profile'); const cur = sel.value; sel.innerHTML = '' + profiles.map(p => ``).join(''); sel.value = cur; + osImages = await api('GET', '/api/v1/os'); + $('#osimages').innerHTML = table(['OS', 'Current version', 'Family', 'Compressed', 'Disk needed', 'Imported', 'Track', 'Versions', 'Status'], + osImages.map(i => { const c = i.current || {}; const s = i.source || {}; return [esc(i.name), `${esc(c.version || '-')}`, esc(s.family || c.family), c.size ? human(c.size) : '', c.raw_size ? human(c.raw_size) : '', c.imported ? esc(new Date(c.imported).toLocaleString()) : '', esc(s.track || ''), (i.versions || []).length, i.warning ? `${esc(i.warning)}` : 'ok']; })); + const os = $('#deploy-os'); const curOS = os.value; + os.innerHTML = '' + osImages.filter(i => i.current).map(i => ``).join(''); + os.value = curOS; } catch (e) { /* non-fatal */ } } $('#profile-form').onsubmit = async ev => { @@ -75,9 +82,12 @@ function renderList() { $('#machines tbody').innerHTML = rows.map(m => { const b = m.bmc; const bmc = b ? `${esc(b.address || '-')} ${esc(b.source)}` : '-'; + const prog = m.job && m.job.progress ? m.job.progress : null; + const state = m.job ? `${esc(m.state)} ${prog ? Math.round(prog.pct || 0) + '%' : ''}` : esc(m.state); + const what = m.profile ? esc(m.profile) : m.os ? `${esc(m.os)}
${esc(m.os_version || '')}` : ''; return ` - ${esc(m.id)}${esc(m.state)} - ${esc(m.node_name || '')}${m.mgmt_ip ? `
${esc(m.mgmt_ip)}` : ''}${esc(m.profile || '')} + ${esc(m.id)}${state} + ${esc(m.node_name || '')}${m.mgmt_ip ? `
${esc(m.mgmt_ip)}` : ''}${what} ${esc((m.manufacturer || '') + ' ' + (m.product || ''))}${esc(m.serial || '')} ${esc(m.cpus)}${human(m.memory_bytes)}${m.disks | 0} ${human(m.storage_bytes)}${m.nics}${m.ib_hcas}${m.gpus} @@ -106,7 +116,27 @@ function renderDetail(d) { $('#action-out').textContent = ''; if (s.profile) $('#assign-profile').value = s.profile; $('#assign-name').value = s.node_name || ''; + const diskSel = $('#deploy-disk'); const curDisk = diskSel.value; + diskSel.innerHTML = '' + ((st.disks || []).map(x => ``).join('')); + diskSel.value = curDisk; + $('#job-cancel').hidden = !s.job; const cards = []; + const job = s.job || (d.jobs || [])[0]; + if (s.os || job) { + const p = (job && job.progress) || {}; const sp = (job && job.spec) || {}; + const active = job && (job.state === 'pending' || job.state === 'running'); + cards.push(card('Deployment', kv([ + ['OS', s.os ? `${esc(s.os)} ${esc(s.os_version || '')}` : ''], + ['Deployed', esc(s.deployed_at ? new Date(s.deployed_at).toLocaleString() : '')], + ['Phoned home', s.deployed_at ? (s.booted_at ? `${esc(new Date(s.booted_at).toLocaleString())}` : 'not yet') : ''], + ['Hostname', esc(s.node_name)], ['Management IP', s.mgmt_ip ? `${esc(s.mgmt_ip)}` : ''], + ['Job', job ? `${esc(job.id)} ${esc(job.kind)} ${esc(job.state)} ${esc(ago(job.created))}` : ''], + ['Target', sp.disk ? `${esc(sp.disk)} ← ${esc(sp.os || '')}/${esc(sp.version || '')}` : ''], + ['Progress', active ? `
${esc(p.phase || '')} ${Math.round(p.pct || 0)}% ${esc(p.msg || '')}` : ''], + ['Result', p.details && Object.keys(p.details).length ? `${esc(JSON.stringify(p.details))}` : ''], + ['Error', job && job.error ? `${esc(job.error)}` : ''], + ]) + ((d.jobs || []).length > 1 ? '

History

' + table(['Job', 'Kind', 'State', 'Created', 'OS', 'Error'], d.jobs.map(j => [`${esc(j.id)}`, esc(j.kind), `${esc(j.state)}`, esc(new Date(j.created).toLocaleString()), esc(((j.spec || {}).os || '') + '/' + ((j.spec || {}).version || '')), esc(j.error || '')])) : ''), true)); + } if (s.profile) cards.push(card('Stateless node', kv([ ['Node name', esc(s.node_name)], ['Profile', esc(s.profile)], ['Management IP', `${esc(s.mgmt_ip)}`], ['InfiniBand IP', s.ib_ip ? `${esc(s.ib_ip)}` : ''], ['Booted', esc(s.booted_at ? new Date(s.booted_at).toLocaleString() : 'not yet')], @@ -188,10 +218,29 @@ const doAssign = boot => action(async () => { $('#assign').onclick = () => doAssign(false); $('#assign-boot').onclick = () => { if (confirm('Assign and power-cycle this machine into the stateless image?')) doAssign(true); }; $('#unassign').onclick = () => { if (confirm('Remove the profile? The node name and IP are released.')) action(async () => { const r = await api('POST', `/api/v1/machines/${current}/unassign`); await loadDetail(current); return 'state ' + r.state; }, 'unassign'); }; +$('#deploy').onclick = () => { + const os = $('#deploy-os').value; if (!os) { $('#action-out').textContent = 'pick an OS image'; return; } + const disk = $('#deploy-disk').value; + if (!confirm(`Deploy ${os} to ${disk || 'the auto-selected disk'} of ${current}?\nThe disk is wiped and the machine is power-cycled.`)) return; + action(async () => { + const r = await api('POST', `/api/v1/machines/${current}/deploy`, { os, disk, hostname: $('#deploy-host').value.trim(), package_upgrade: $('#deploy-upgrade').checked, boot: true }); + await loadDetail(current); return `job ${r.job}: ${r.os}/${r.version} → ${r.disk} as ${r.hostname}${r.ipmi ? ' · ' + r.ipmi : ''}`; + }, 'deploy'); +}; +$('#release').onclick = () => { if (confirm(`Release ${current}? The OS disk is wiped (signatures + first/last 64 MiB) and the machine returns to ready.`)) action(async () => { const r = await api('POST', `/api/v1/machines/${current}/release`, { boot: true }); await loadDetail(current); return `job ${r.job} (wipe ${r.disk})${r.ipmi ? ' · ' + r.ipmi : ''}`; }, 'release'); }; +$('#job-cancel').onclick = () => { if (confirm('Cancel the active job? A running write is not interrupted on the machine; it just stops being tracked.')) action(async () => { const r = await api('POST', `/api/v1/machines/${current}/jobs/cancel`); await loadDetail(current); return `job ${r.job} cancelled`; }, 'cancel job'); }; $('#recommission').onclick = () => action(async () => { const r = await api('POST', `/api/v1/machines/${current}/recommission`); await loadDetail(current); return `state ${r.state}: next PXE boot runs commissioning`; }, 'recommission'); $('#recommission-now').onclick = () => { if (confirm('Remove PXE override, set bootdev=pxe and power-cycle this machine via its BMC?')) action(async () => { const r = await api('POST', `/api/v1/machines/${current}/recommission?now=1`); await loadDetail(current); return r.ipmi || r.state; }, 'recommission now'); }; $('#delete').onclick = () => { if (confirm(`Delete ${current}? It will be commissioned again as a new machine on next PXE boot.`)) action(async () => { await api('DELETE', `/api/v1/machines/${current}`); $('#back').click(); return 'deleted'; }, 'delete'); }; -function schedule() { clearInterval(timer); timer = setInterval(() => { if (!current && document.visibilityState === 'visible') loadList(); }, 10000); } +function schedule() { + clearInterval(timer); + timer = setInterval(() => { + if (document.visibilityState !== 'visible') return; + if (!current) return loadList(); + const m = machines.find(x => x.id === current); + if (m && (m.job || m.state === 'deploying' || m.state === 'releasing')) { loadDetail(current); loadList(); } + }, 4000); +} schedule(); loadList(); diff --git a/cmd/rollcall/web/index.html b/cmd/rollcall/web/index.html index b54ba33..ddbc225 100644 --- a/cmd/rollcall/web/index.html +++ b/cmd/rollcall/web/index.html @@ -33,7 +33,7 @@ - + @@ -42,10 +42,11 @@
- Profiles & images (stateless provisioning) + Provisioning: OS images (deployment), profiles & root images (stateless)
+

OS images for deployment · refreshed nightly by rollcall os sync

Profiles

-

Images

+

Stateless root images

Add / update profile

@@ -87,6 +88,15 @@ + Deploy: + + + + + + + + diff --git a/cmd/rollcall/web/style.css b/cmd/rollcall/web/style.css index 46afba9..18ab061 100644 --- a/cmd/rollcall/web/style.css +++ b/cmd/rollcall/web/style.css @@ -33,7 +33,12 @@ th { font-weight: 600; color: var(--muted); font-size: 12px; text-transform: upp th.num, td.num { text-align: right; font-variant-numeric: tabular-nums; } tbody tr { cursor: pointer; } tbody tr:hover { background: var(--hover); } .pill { display: inline-block; padding: 2px 8px; border-radius: 999px; background: var(--pill); font-size: 12px; } -.pill.ready, .pill.booted { color: var(--ok); } .pill.recommission, .pill.new, .pill.stateless { color: var(--warn); } +.pill.ready, .pill.booted, .pill.deployed { color: var(--ok); } .pill.recommission, .pill.new, .pill.stateless, .pill.deploying, .pill.releasing { color: var(--warn); } +.pill.deploy-failed, .pill.release-failed { color: var(--danger); } +.bar { position: relative; height: 10px; background: var(--pill); border-radius: 999px; overflow: hidden; margin: 6px 0; } +.bar > span { position: absolute; inset: 0 auto 0 0; background: var(--accent); transition: width .5s; } +.actions select, .actions input[type=text] { padding: 5px 8px; border: 1px solid var(--line); border-radius: 6px; background: var(--panel); color: var(--fg); font: inherit; } +.actions label { display: inline-flex; gap: 4px; align-items: center; color: var(--muted); font-size: 13px; } .card form input { padding: 5px 8px; border: 1px solid var(--line); border-radius: 6px; background: var(--panel); color: var(--fg); font: inherit; min-width: 0; width: 100%; } #assign-profile { padding: 5px 8px; border: 1px solid var(--line); border-radius: 6px; background: var(--panel); color: var(--fg); font: inherit; } .ok { color: var(--ok); } .bad { color: var(--danger); } diff --git a/deploy/config.example.json b/deploy/config.example.json index ffebb50..6a61119 100644 --- a/deploy/config.example.json +++ b/deploy/config.example.json @@ -12,6 +12,8 @@ "images_dir": "/srv/rollcall/images", "overlays_dir": "/etc/rollcall/overlays", "dnsmasq_dir": "/var/lib/rollcall/dnsmasq", + "os_dir": "/srv/rollcall/os", + "os_sources": "/etc/rollcall/os-sources.json", "node_pools": {"mgmt": "10.213.77.50-10.213.77.99"}, "cluster_vars": { "domain": "pxe.local", diff --git a/deploy/install.sh b/deploy/install.sh index c780266..39bff80 100644 --- a/deploy/install.sh +++ b/deploy/install.sh @@ -27,6 +27,11 @@ install -d /srv/rollcall/images install -d -m 0750 -o root -g rollcall /etc/rollcall/overlays cp -a overlays/. /etc/rollcall/overlays/ chgrp -R rollcall /etc/rollcall/overlays +# OS deployment images: ///disk.raw.zst, written by `rollcall os sync` (user rollcall) +install -d -m 0755 -o rollcall -g rollcall /srv/rollcall/os +[ -e /etc/rollcall/os-sources.json ] || install -m 0644 os-sources.json /etc/rollcall/os-sources.json +install -m 0644 rollcall-os-sync.service rollcall-os-sync.timer /etc/systemd/system/ +command -v qemu-img >/dev/null || echo "WARNING: qemu-img missing (apt install qemu-utils) - rollcall os sync/import will fail" >&2 # dnsmasq host export + reload trigger install -d -m 0755 /var/lib/rollcall install -d -m 0755 -o rollcall -g rollcall /var/lib/rollcall/dnsmasq @@ -36,6 +41,7 @@ install -m 0644 rollcall-controller-sync.service rollcall-controller-sync.timer install -m 0755 rollcall-controller-sync.sh /usr/local/bin/rollcall-controller-sync.sh systemctl daemon-reload systemctl enable --now rollcall-dnsmasq.path >/dev/null 2>&1 || true +systemctl enable --now rollcall-os-sync.timer >/dev/null 2>&1 || true if systemctl list-unit-files slurmctld.service >/dev/null 2>&1 && [ -d /etc/slurm ]; then systemctl enable --now rollcall-controller-sync.timer >/dev/null 2>&1 || true fi diff --git a/deploy/os-sources.json b/deploy/os-sources.json new file mode 100644 index 0000000..eac9cc6 --- /dev/null +++ b/deploy/os-sources.json @@ -0,0 +1,52 @@ +[ + { + "name": "debian-13", + "description": "Debian 13 (trixie) generic cloud image", + "family": "debian", + "url": "https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2", + "checksum_url": "https://cloud.debian.org/images/cloud/trixie/latest/SHA512SUMS", + "checksum": "sha512", + "track": "latest", + "keep": 2 + }, + { + "name": "ubuntu-24.04", + "description": "Ubuntu 24.04 LTS (noble) server cloud image", + "family": "debian", + "url": "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img", + "checksum_url": "https://cloud-images.ubuntu.com/noble/current/SHA256SUMS", + "checksum": "sha256", + "track": "latest", + "keep": 2 + }, + { + "name": "rocky-9", + "description": "Rocky Linux 9 GenericCloud (Base)", + "family": "rhel", + "url": "https://dl.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2", + "checksum_url": "https://dl.rockylinux.org/pub/rocky/9/images/x86_64/CHECKSUM", + "checksum": "sha256", + "track": "latest", + "keep": 2 + }, + { + "name": "almalinux-9", + "description": "AlmaLinux 9 GenericCloud", + "family": "rhel", + "url": "https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-latest.x86_64.qcow2", + "checksum_url": "https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/CHECKSUM", + "checksum": "sha256", + "track": "latest", + "keep": 2 + }, + { + "name": "centos-stream-9", + "description": "CentOS Stream 9 GenericCloud", + "family": "rhel", + "url": "https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-x86_64-9-latest.x86_64.qcow2", + "checksum_url": "https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-x86_64-9-latest.x86_64.qcow2.SHA256SUM", + "checksum": "sha256", + "track": "latest", + "keep": 2 + } +] diff --git a/deploy/rollcall-os-sync.service b/deploy/rollcall-os-sync.service new file mode 100644 index 0000000..e5ce8f7 --- /dev/null +++ b/deploy/rollcall-os-sync.service @@ -0,0 +1,13 @@ +[Unit] +Description=rollcall: refresh OS deployment images from upstream (downloads only when the vendor checksum changed) +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +User=rollcall +Group=rollcall +ExecStart=/usr/local/bin/rollcall os sync +Nice=10 +IOSchedulingClass=idle +TimeoutStartSec=6h diff --git a/deploy/rollcall-os-sync.timer b/deploy/rollcall-os-sync.timer new file mode 100644 index 0000000..03f362d --- /dev/null +++ b/deploy/rollcall-os-sync.timer @@ -0,0 +1,10 @@ +[Unit] +Description=rollcall: nightly OS image refresh + +[Timer] +OnCalendar=*-*-* 03:30:00 +RandomizedDelaySec=30m +Persistent=true + +[Install] +WantedBy=timers.target
IDStateNodeProfileModelSerialCPURAMIDStateNodeProfile / OSModelSerialCPURAM DisksStorageNICIBGPU BMCLast seen