commit 3d4841f31c8b1fa382c6ccc9af10edff17afa07a Author: ilkermanap Date: Fri Sep 4 21:50:17 2026 +0200 Initial commit: MAAS-deployable Proxmox VE images with cluster automation Builds a Proxmox VE image that MAAS can deploy to bare metal, plus first-boot automation that configures the node and joins it to a Proxmox cluster with no manual steps. The image starts from the official Debian cloud image and installs proxmox-ve on top of it, rather than capturing a raw disk from the Proxmox ISO. That keeps MAAS in control of partitioning, networking, SSH keys and cloud-init, and makes moving between Proxmox releases a variable change instead of a rewrite. Contents: * Makefile driving the whole flow: build, verify, preseed, upload * customize-proxmox.sh, run inside the Packer build VM, which layers Proxmox onto the Debian cloud image and resets the pmxcfs node identity so one image can produce many nodes * pve-maas-init, a first-boot state machine covering /etc/hosts, node-unique identifiers, the root password, vmbr0 conversion, cluster create/join and the local-lvm thin pool; each stage is resumable across reboots * curtin-hooks, which stops curtin installing a kernel over APT and pins interface names by MAC so they match what MAAS recorded at commissioning * a MAAS curtin preseed template and cloud-init examples * deploy-cluster.sh, which builds a whole cluster through the MAAS API * verify-image.sh, 22 static checks on the produced tarball Cluster identity lives entirely in deploy-time cloud-init user-data, so a single image and preseed can build any number of independent clusters. Verified end to end against MAAS 3.7.2: proxmox-ve 9.2.0 / pve-manager 9.2.11 / kernel 7.0.14-15-pve, deployed to two machines that formed a quorate cluster with local-lvm on both, with no manual intervention. The README documents four failure modes found along the way that all fail silently: curtin rejecting "kernel: null", pvenetcommit overwriting the network configuration at boot, interface renaming leaving the link down, and a systemd ordering cycle that made systemd delete the service's start job. Co-Authored-By: Claude Opus 5 (1M context) diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..4b6e2e0 --- /dev/null +++ b/.gitignore @@ -0,0 +1,6 @@ +/build/ +*.tar.gz +*.dd.gz +OVMF_CODE.fd +OVMF_VARS.fd +.secrets/ diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..9b9f7a9 --- /dev/null +++ b/Makefile @@ -0,0 +1,251 @@ +# maas-proxmox - Proxmox VE icin MAAS'a yuklenebilir Packer imaji +# +# Hizli baslangic (build host uzerinde, root olarak): +# sudo ./scripts/install-deps.sh +# sudo make image +# make preseed +# make upload MAAS_PROFILE=admin +# +SHELL := /bin/bash +.DEFAULT_GOAL := help + +# ---------------------------------------------------------------- surum ayarlari +# Proxmox VE ana surumu. Yeni bir PVE surumu ciktiginda genelde sadece +# PVE_VERSION + DEBIAN_SERIES/DEBIAN_VERSION degistirmek yeterlidir. +PVE_VERSION ?= 9 +DEBIAN_SERIES ?= trixie +DEBIAN_VERSION ?= 13 + +# Proxmox APT deposu: pve-no-subscription | pve-enterprise | pve-test +PVE_REPO ?= pve-no-subscription +PVE_REPO_URI ?= http://download.proxmox.com/debian/pve +PVE_KEYRING_URL ?= https://enterprise.proxmox.com/debian/proxmox-archive-keyring-$(DEBIAN_SERIES).gpg + +# proxmox-ve disinda imaja girecek ek paketler +# NOT: 'vlan' ve 'vzdump' pve-manager ile catisir (Conflicts) - eklemeyin. +# VLAN destegini ifupdown2 kendisi saglar. +PVE_EXTRA_PACKAGES ?= ifupdown2 open-iscsi chrony postfix lvm2 thin-provisioning-tools \ + ethtool bridge-utils ipmitool nvme-cli lsscsi sudo + +# ---------------------------------------------------------------- imaj ayarlari +ARCH ?= amd64 +SUBARCH ?= generic +BOOT ?= uefi +TIMEOUT ?= 3h +PACKER_LOG ?= 0 + +# Build VM kaynaklari. Upstream sablon 4G/2CPU/2GB ile gelir; Debian cloud image +# (~3G) + Proxmox VE (~3G) 4G'ye sigmaz, bu yuzden sablonu yamiyoruz. +# Sonuc tgz yalnizca kullanilan dosyalari icerdiginden buyuk disk imaji sismez. +DISK_SIZE ?= 16G +BUILD_CPUS ?= 4 +BUILD_MEM ?= 4096 + +# --- derleme hizi ------------------------------------------------------------- +# stable | daily +# stable: cloud.debian.org/.../trixie/latest (sabit URL -> packer onbellegi +# calisir, tekrarlanan derlemelerde ~350MB indirme yok, tekrarlanabilir) +# daily : upstream sablonun varsayilani (her gun degisir, onbellek isabet etmez) +DEBIAN_IMAGE_CHANNEL ?= stable + +# Tarball sikistirma seviyesi. Upstream --best (9) kullaniyor; 6 belirgin daha +# hizli ve imaj yalnizca ~%2-3 buyuyor. pigz varsa cok cekirdekli calisir. +GZIP_LEVEL ?= 6 + +# Yerel APT onbellegi (or. apt-cacher-ng): http://10.0.2.2:3142 +# Packer'in user-mode aginda build VM host'u 10.0.2.2 olarak gorur. +# Bos birakilirsa proxy kullanilmaz. Bkz: make deps-cache +APT_PROXY ?= + +IMAGE_NAME ?= proxmox-ve-$(PVE_VERSION) +IMAGE_TITLE ?= Proxmox VE $(PVE_VERSION) (Debian $(DEBIAN_VERSION)) + +# ---------------------------------------------------------------- yollar +WORKDIR ?= $(CURDIR)/build +PM_REPO ?= https://github.com/canonical/packer-maas.git +# Test edilmis upstream commit'e sabitlenmistir. 'main' birakmak, upstream'de +# yapilan bir degisikligin derlemeyi haber vermeden bozmasi anlamina gelir. +# Ileri tasimak icin: PM_REF=main ile derleyip test edin, sonra yeni SHA'yi buraya yazin. +PM_REF ?= c23d5dd985f52b2eaccc893e886213a424a243a5 +PM := $(WORKDIR)/packer-maas +TPL := $(PM)/debian + +OVERLAY_TGZ := $(WORKDIR)/pve-maas-overlay.tar.gz +CUSTOMIZE := $(WORKDIR)/customize-proxmox.sh +OUTPUT ?= $(WORKDIR)/$(IMAGE_NAME).tar.gz +PRESEED := $(WORKDIR)/curtin_userdata_custom_$(ARCH)_$(SUBARCH)_$(IMAGE_NAME) + +# ---------------------------------------------------------------- MAAS ayarlari +MAAS_PROFILE ?= admin +MAAS_ARCH ?= $(ARCH)/$(SUBARCH) +MAAS_IMAGE_NAME ?= custom/$(IMAGE_NAME) +# MAAS snap kurulumu icin: /var/snap/maas/current/preseeds +MAAS_PRESEED_DIR ?= /var/snap/maas/current/preseeds + +OVMF_DIR ?= /usr/share/OVMF +OVMF_SFX ?= $(shell test -f $(OVMF_DIR)/OVMF_CODE.fd && echo "" || echo "_4M") + +# ---------------------------------------------------------------- hedefler +.PHONY: help deps deps-cache check-upstream checkout overlay customize image verify preseed install-preseed upload clean distclean lint + +help: + @echo "maas-proxmox - Proxmox VE $(PVE_VERSION) MAAS imaji" + @echo + @echo " make deps Build host bagimliliklarini kur (root gerekir)" + @echo " make deps-cache Yerel APT onbellegi kur (derlemeyi hizlandirir)" + @echo " make image Imaji derle -> $(OUTPUT) (root gerekir)" + @echo " make verify Uretilen imajin icerigini dogrula" + @echo " make check-upstream Depodaki surumleri elimizdeki imajla karsilastir" + @echo " make preseed MAAS curtin preseed dosyasini uret" + @echo " make install-preseed Preseed'i $(MAAS_PRESEED_DIR) altina kopyala (root)" + @echo " make upload Imaji MAAS'a yukle (MAAS_PROFILE=$(MAAS_PROFILE))" + @echo " make clean Ara dosyalari sil" + @echo " make distclean build/ dizinini tamamen sil" + @echo + @echo "Onemli degiskenler:" + @echo " PVE_VERSION=$(PVE_VERSION) DEBIAN_SERIES=$(DEBIAN_SERIES) PVE_REPO=$(PVE_REPO)" + @echo " IMAGE_NAME=$(IMAGE_NAME) ARCH=$(ARCH) BOOT=$(BOOT)" + @echo " OUTPUT=$(OUTPUT)" + +deps: + ./scripts/install-deps.sh + +# --- packer-maas deposunu getir ----------------------------------------------- +$(PM)/.git: + @mkdir -p $(WORKDIR) + git clone $(PM_REPO) $(PM) + +# PM_REF bir dal, etiket ya da commit SHA olabilir. +checkout: $(PM)/.git + @cd $(PM) && git fetch -q --all --tags && git checkout -q -f $(PM_REF) \ + && (git symbolic-ref -q HEAD >/dev/null && git pull -q --ff-only || true) + @echo "packer-maas: $$(cd $(PM) && git rev-parse --short HEAD) ($(PM_REF))" + +# --- imaja gomulecek overlay --------------------------------------------------- +overlay: $(OVERLAY_TGZ) + +$(OVERLAY_TGZ): $(shell find overlay -type f 2>/dev/null) + @mkdir -p $(WORKDIR) + COPYFILE_DISABLE=1 tar czf $@ -C overlay --no-xattrs --exclude='.keep' . + @echo "overlay: $@ ($$(du -h $@ | cut -f1))" + +# --- packer'in VM icinde calistiracagi customize script ------------------------- +customize: $(CUSTOMIZE) + +$(CUSTOMIZE): scripts/customize-proxmox.sh.in $(OVERLAY_TGZ) + @mkdir -p $(WORKDIR) + @sed -e 's|@@PVE_SUITE@@|$(DEBIAN_SERIES)|g' \ + -e 's|@@PVE_REPO@@|$(PVE_REPO)|g' \ + -e 's|@@PVE_REPO_URI@@|$(PVE_REPO_URI)|g' \ + -e 's|@@PVE_KEYRING_URL@@|$(PVE_KEYRING_URL)|g' \ + -e 's|@@PVE_VERSION@@|$(PVE_VERSION)|g' \ + -e 's|@@PVE_EXTRA_PACKAGES@@|$(PVE_EXTRA_PACKAGES)|g' \ + -e 's|@@PM_REF@@|$(PM_REF)|g' \ + $< > $@ + @echo '__PVE_MAAS_OVERLAY__' >> $@ + @base64 < $(OVERLAY_TGZ) >> $@ + @chmod +x $@ + @echo "customize script: $@" + +# --- imaj derleme --------------------------------------------------------------- +image: checkout $(CUSTOMIZE) + @if [ "$$(id -u)" -ne 0 ]; then echo "HATA: 'make image' root gerektirir (sudo make image)"; exit 1; fi + @command -v packer >/dev/null || { echo "HATA: packer yok, once 'make deps'"; exit 1; } + @echo "==> Sablon yamalaniyor: disk=$(DISK_SIZE) cpus=$(BUILD_CPUS) mem=$(BUILD_MEM)" + sed -i -E 's|^([[:space:]]*disk_size[[:space:]]*=[[:space:]]*).*|\1"$(DISK_SIZE)"|' $(TPL)/debian-cloudimg.pkr.hcl + sed -i -E 's|^([[:space:]]*cpus[[:space:]]*=[[:space:]]*).*|\1$(BUILD_CPUS)|' $(TPL)/debian-cloudimg.pkr.hcl + sed -i -E 's|^([[:space:]]*memory[[:space:]]*=[[:space:]]*).*|\1$(BUILD_MEM)|' $(TPL)/debian-cloudimg.pkr.hcl +ifeq ($(strip $(DEBIAN_IMAGE_CHANNEL)),stable) + @echo "==> Kararli Debian cloud image kullanilacak (packer onbellegi isabet eder)" + sed -i -E 's|/daily/latest/|/latest/|g; s|-daily\.qcow2|.qcow2|g' $(TPL)/debian-cloudimg.pkr.hcl + @grep -nE 'iso_url|iso_checksum' $(TPL)/debian-cloudimg.pkr.hcl +endif + sed -i -E 's|--best --force|-$(GZIP_LEVEL) --force|' $(PM)/scripts/fuse-tar-root + @grep -nE 'disk_size|^ cpus|^ memory' $(TPL)/debian-cloudimg.pkr.hcl + cp -v $(OVMF_DIR)/OVMF_CODE$(OVMF_SFX).fd $(TPL)/OVMF_CODE.fd + cp -v $(OVMF_DIR)/OVMF_VARS$(OVMF_SFX).fd $(TPL)/OVMF_VARS.fd + rm -rf $(TPL)/output-cloudimg $(TPL)/seeds-cloudimg.iso + cd $(TPL) && PACKER_LOG=$(PACKER_LOG) packer init . + cd $(TPL) && PACKER_LOG=$(PACKER_LOG) packer build \ + -var debian_series=$(DEBIAN_SERIES) \ + -var debian_version=$(DEBIAN_VERSION) \ + -var architecture=$(ARCH) \ + -var boot_mode=$(BOOT) \ + -var ovmf_suffix=$(OVMF_SFX) \ + -var host_is_arm=false \ + -var timeout=$(TIMEOUT) \ + -var customize_script=$(CUSTOMIZE) \ + -var filename=$(OUTPUT) \ + -var http_proxy=$(APT_PROXY) \ + . + @ls -lh $(OUTPUT) + +# --- MAAS preseed --------------------------------------------------------------- +preseed: $(PRESEED) + +$(PRESEED): maas/curtin_userdata_custom.in + @mkdir -p $(WORKDIR) + @sed -e 's|@@IMAGE_NAME@@|$(IMAGE_NAME)|g' \ + -e 's|@@ARCH@@|$(ARCH)|g' \ + $< > $@ + @echo "preseed: $@" + @echo " -> MAAS region controller uzerinde $(MAAS_PRESEED_DIR)/ altina kopyalayin" + +install-preseed: $(PRESEED) + @if [ "$$(id -u)" -ne 0 ]; then echo "HATA: root gerekir"; exit 1; fi + install -D -m 0644 $(PRESEED) $(MAAS_PRESEED_DIR)/$(notdir $(PRESEED)) + @echo "kuruldu: $(MAAS_PRESEED_DIR)/$(notdir $(PRESEED))" + +# --- MAAS'a yukleme ------------------------------------------------------------- +upload: + @test -f $(OUTPUT) || { echo "HATA: $(OUTPUT) yok, once 'make image'"; exit 1; } + maas $(MAAS_PROFILE) boot-resources create \ + name='$(MAAS_IMAGE_NAME)' \ + title='$(IMAGE_TITLE)' \ + architecture='$(MAAS_ARCH)' \ + filetype='tgz' \ + content@=$(OUTPUT) + +verify: + @test -f $(OUTPUT) || { echo "HATA: $(OUTPUT) yok, once 'make image'"; exit 1; } + ./scripts/verify-image.sh $(OUTPUT) + +# Depoda hangi surumler var, elimizdeki imaj hangi surumde? +check-upstream: + @echo "==> Depoda mevcut ($(PVE_REPO_URI) $(DEBIAN_SERIES) $(PVE_REPO)):" + @curl -fsS $(PVE_REPO_URI)/dists/$(DEBIAN_SERIES)/$(PVE_REPO)/binary-$(ARCH)/Packages.gz 2>/dev/null \ + | gunzip \ + | awk '/^Package: (proxmox-ve|pve-manager|proxmox-default-kernel)$$/{p=$$2; next} \ + /^Version: /{if(p!=""){print p" "$$2; p=""}}' \ + | sort -V | awk '{v[$$1]=$$2} END{for(k in v) printf " %-24s %s\n", k, v[k]}' \ + || echo " (depoya erisilemedi)" + @echo "==> Yerel imajda ($(OUTPUT)):" + @if [ -f $(OUTPUT) ]; then \ + tar xzf $(OUTPUT) -O ./etc/pve-maas/image-info 2>/dev/null | sed 's/^/ /' \ + || echo " (image-info okunamadi - imaj bu ozellikten once mi derlendi?)"; \ + else echo " (imaj yok - once 'make image')"; fi + +# Yerel APT onbellegi - tekrarlanan derlemelerde ~700MB indirmeyi ortadan kaldirir. +# Kurduktan sonra: sudo make image APT_PROXY=http://10.0.2.2:3142 +deps-cache: + @if [ "$$(id -u)" -ne 0 ]; then echo "HATA: root gerekir (sudo make deps-cache)"; exit 1; fi + DEBIAN_FRONTEND=noninteractive apt-get install -y apt-cacher-ng + systemctl enable --now apt-cacher-ng + @echo + @echo "Hazir. Derlemede kullanmak icin:" + @echo " sudo make image APT_PROXY=http://10.0.2.2:3142" + @echo "(10.0.2.2 = packer user-mode aginda build host'un adresi)" + +lint: + @bash -n scripts/customize-proxmox.sh.in && echo "customize-proxmox.sh.in: OK" + @bash -n overlay/usr/local/sbin/pve-maas-init && echo "pve-maas-init: OK" + @command -v shellcheck >/dev/null && shellcheck -S warning \ + overlay/usr/local/sbin/pve-maas-init scripts/install-deps.sh || true + +clean: + rm -f $(OVERLAY_TGZ) $(CUSTOMIZE) $(PRESEED) + rm -rf $(TPL)/output-cloudimg $(TPL)/seeds-cloudimg.iso \ + $(TPL)/OVMF_CODE.fd $(TPL)/OVMF_VARS.fd + +distclean: + rm -rf $(WORKDIR) diff --git a/README.md b/README.md new file mode 100644 index 0000000..8ec47da --- /dev/null +++ b/README.md @@ -0,0 +1,824 @@ +# maas-proxmox + +Build **Proxmox VE** images that [MAAS](https://maas.io) can deploy to bare metal, with +first-boot automation that configures the node and joins it to a Proxmox cluster — +without anyone logging in. + +Target: **Proxmox VE 9.x** on Debian 13 "Trixie". Older and newer releases are a +variable change away (see [Moving to a new Proxmox release](#moving-to-a-new-proxmox-release)). + +*Türkçe dokümantasyon: [README.tr.md](README.tr.md)* + +--- + +## Table of contents + +- [What this does](#what-this-does) +- [Why Debian + the Proxmox repository, and not the Proxmox ISO](#why-debian--the-proxmox-repository-and-not-the-proxmox-iso) +- [How it works](#how-it-works) +- [Requirements](#requirements) +- [Quick start](#quick-start) +- [Build configuration](#build-configuration) +- [First-boot behaviour](#first-boot-behaviour) +- [Configuration reference](#configuration-reference) +- [Deploying a cluster](#deploying-a-cluster) +- [Networking](#networking) +- [Storage](#storage) +- [Four traps this image works around](#four-traps-this-image-works-around) +- [Moving to a new Proxmox release](#moving-to-a-new-proxmox-release) +- [Build performance](#build-performance) +- [Repository layout](#repository-layout) +- [Troubleshooting](#troubleshooting) +- [Verified status](#verified-status) +- [Credits and licensing](#credits-and-licensing) + +--- + +## What this does + +MAAS provisions bare-metal machines from images. It ships images for Ubuntu and a few +other distributions, but not for Proxmox VE. This repository produces one. + +`make image` builds a tarball that you upload to MAAS as a custom image. After that, +deploying a machine in MAAS gives you a fully configured Proxmox VE node: + +| Step | Who does it | +|---|---| +| Partition the disk, write the image, configure networking, inject SSH keys | MAAS / curtin | +| Set hostname, regenerate node-unique identifiers | cloud-init + this image | +| Convert the management interface into a `vmbr0` bridge | this image | +| Set the `root@pam` password | this image | +| Create an LVM-thin pool and register it as `local-lvm` | this image | +| Create a Proxmox cluster, or join an existing one | this image | + +Everything after the MAAS handoff is driven by a single systemd service, +`pve-maas-init`, configured through cloud-init user-data supplied at deploy time. +The image itself carries no cluster identity, no hostname and no credentials, so the +same image can build any number of independent clusters. + +--- + +## Why Debian + the Proxmox repository, and not the Proxmox ISO + +There are two plausible ways to get Proxmox VE into MAAS, and the choice shapes +everything else. + +**Option A — install the Proxmox ISO under automation, capture the raw disk (`.dd.gz`).** +You get Proxmox's native layout: ZFS-on-root if you want it, `local-lvm` preconfigured. +But MAAS writes a `dd` image to the disk verbatim. Its storage layouts, its partitioning +UI and much of the curtin flow stop applying. Proxmox does not ship cloud-init on the +host, so MAAS's metadata — hostname, network configuration, SSH keys, user-data — +has to be bolted on afterwards. Growing the root filesystem becomes your problem. + +**Option B — start from the Debian cloud image and install `proxmox-ve` on top of it.** +This is a +[configuration Proxmox supports and documents](https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_13_Trixie). +The result is an ordinary Debian root filesystem, so MAAS's normal `tgz` custom-image +path applies unchanged: MAAS partitions the disk according to whatever storage layout +you configured, curtin installs the bootloader, and cloud-init consumes MAAS metadata +the way it does for Ubuntu. + +**This repository takes option B.** The cost is that `local-lvm` is not preconfigured — +the image creates the thin pool on first boot from whatever free space MAAS left in the +volume group — and ZFS-on-root is not available. In exchange, MAAS stays in control of +the parts MAAS is good at, and moving between Proxmox releases is a variable change +rather than a rewrite. + +--- + +## How it works + +### Build pipeline + +``` +Debian 13 cloud image (qcow2, official) + │ + ├─ canonical/packer-maas, "debian" template (QEMU + KVM) + │ ├─ cloud-init / netplan / curtin compatibility [upstream] + │ ├─ disk, cpu and memory patch (4G -> 16G) [this repo] + │ └─ customize-proxmox.sh [this repo] + │ ├─ add the Proxmox APT repository and keyring + │ ├─ install proxmox-default-kernel and proxmox-ve + │ ├─ remove the Debian kernel and os-prober + │ ├─ wipe the pmxcfs node identity + │ ├─ disable networking.service, clear interfaces{,.new} + │ └─ install the overlay: + │ /usr/local/sbin/pve-maas-init + │ /etc/pve-maas/pve-maas.conf + │ /etc/systemd/system/pve-maas-init.service + │ /curtin/curtin-hooks + │ + └─ proxmox-ve-9.tar.gz ──► maas boot-resources create +``` + +The upstream [`canonical/packer-maas`](https://github.com/canonical/packer-maas) +repository is cloned at build time and pinned to a tested commit. It is *not* vendored; +this repository applies a small, explicit patch to its Packer template and supplies the +customization script through the template's `customize_script` variable. + +### Deployment flow + +``` +MAAS: Deploy (osystem=custom, distro_series=proxmox-ve-9, user_data=…) + │ + ├─ PXE boot into the ephemeral environment + ├─ curtin: partition, extract the image + ├─ curtin: /curtin/curtin-hooks ── skip kernel install + │ └─ pin interface names by MAC + ├─ reboot into Proxmox VE + │ + └─ first boot + ├─ cloud-init: hostname, network (netplan), SSH keys, + │ write /etc/pve-maas/conf.d/*.conf from user-data + └─ pve-maas-init.service + hosts → identity → rootpw → network → cluster → storage +``` + +`pve-maas-init` is a state machine. Each stage records completion in +`/var/lib/pve-maas/.done`, so a stage that reboots the node (the network stage +does, by default) resumes at the next stage on the following boot, and a stage that +fails is retried on the next boot rather than leaving the node half-configured. + +--- + +## Requirements + +**Build host** — where `make image` runs: + +- Ubuntu 22.04 or newer (24.04 LTS is what this was developed and tested on), x86_64 +- **Access to `/dev/kvm`.** If the build host is itself a virtual machine, nested + virtualization must be enabled and the CPU type must pass the virtualization flags + through (on Proxmox: `--cpu host`). Without KVM the build falls back to full + emulation and is not practical. +- 4+ vCPU, 8+ GB RAM, 25+ GB free disk +- `sudo` (the build runs as root — it uses `qemu-nbd`, FUSE mounts and `tar --xattrs`) + +**Deployment side:** + +- MAAS 3.2 or newer (custom image support); tested on 3.7.2 +- curtin 21.0 or newer +- The curtin preseed from this repository installed on the MAAS region controller + +--- + +## Quick start + +```bash +# 1. Install build dependencies (packer, qemu, ovmf, nbdkit, fuse2fs, …) +sudo ./scripts/install-deps.sh + +# 2. Build the image (~11 minutes) +sudo make image +# -> build/proxmox-ve-9.tar.gz + +# 3. Check the result +make verify + +# 4. Generate the MAAS curtin preseed +make preseed +# -> build/curtin_userdata_custom_amd64_generic_proxmox-ve-9 + +# 5. Install the preseed on the MAAS region controller +sudo make install-preseed +# default: /var/snap/maas/current/preseeds/ +# for a deb-packaged MAAS: MAAS_PRESEED_DIR=/etc/maas/preseeds + +# 6. Upload the image to MAAS +make upload MAAS_PROFILE=admin +``` + +Steps 5 and 6 run against the MAAS region controller, so either run them there or copy +the two artifacts across. + +**The preseed is not optional.** Without it, curtin tries to install a kernel over APT +during deployment and the deployment fails. See +[Four traps this image works around](#four-traps-this-image-works-around). + +Then deploy a machine: + +```bash +maas $PROFILE machine deploy $SYSTEM_ID \ + osystem=custom distro_series=proxmox-ve-9 \ + user_data="$(base64 -w0 maas/examples/01-first-node.yaml)" +``` + +…or pick **Custom → Proxmox VE 9** in the MAAS web UI. For a whole cluster, use +[`scripts/deploy-cluster.sh`](scripts/deploy-cluster.sh). + +--- + +## Build configuration + +All of these are `make` variables — `sudo make image DISK_SIZE=24G`, and so on. + +### Version selection + +| Variable | Default | Meaning | +|---|---|---| +| `PVE_VERSION` | `9` | Proxmox VE major version; used in the image name | +| `DEBIAN_SERIES` | `trixie` | Debian codename that Proxmox release is built on | +| `DEBIAN_VERSION` | `13` | Debian major version number | +| `PVE_REPO` | `pve-no-subscription` | `pve-no-subscription`, `pve-enterprise` or `pve-test` | +| `PVE_REPO_URI` | `http://download.proxmox.com/debian/pve` | APT repository URI | +| `PVE_KEYRING_URL` | derived from `DEBIAN_SERIES` | Proxmox archive keyring | +| `PVE_EXTRA_PACKAGES` | `ifupdown2 open-iscsi chrony …` | Extra packages to bake in | + +> `vlan` and `vzdump` **conflict** with `pve-manager` — do not add them. +> ifupdown2 provides VLAN support natively. + +### Image and build + +| Variable | Default | Meaning | +|---|---|---| +| `IMAGE_NAME` | `proxmox-ve-9` | MAAS name (`custom/`) and preseed filename | +| `ARCH` / `SUBARCH` | `amd64` / `generic` | Target architecture | +| `BOOT` | `uefi` | Boot mode baked into the image | +| `DISK_SIZE` | `16G` | Build VM disk. Upstream's 4G cannot fit Debian + Proxmox | +| `BUILD_CPUS` / `BUILD_MEM` | `4` / `4096` | Build VM resources | +| `TIMEOUT` | `3h` | Packer build timeout | +| `PM_REF` | pinned SHA | `canonical/packer-maas` revision | + +### Speed + +| Variable | Default | Meaning | +|---|---|---| +| `DEBIAN_IMAGE_CHANNEL` | `stable` | `stable` uses a fixed URL so Packer's cache works; `daily` is upstream's default and changes every day | +| `GZIP_LEVEL` | `6` | Tarball compression. Upstream uses 9 | +| `APT_PROXY` | *(empty)* | Local APT cache, e.g. `http://10.0.2.2:3142` — see `make deps-cache` | + +### MAAS + +| Variable | Default | Meaning | +|---|---|---| +| `MAAS_PROFILE` | `admin` | `maas` CLI profile name | +| `MAAS_PRESEED_DIR` | `/var/snap/maas/current/preseeds` | Preseed directory | + +### Targets + +``` +make deps install build dependencies (root) +make deps-cache install a local APT cache to speed up rebuilds (root) +make image build the image (root) +make verify check the built image's contents +make check-upstream compare repository versions against the built image +make preseed generate the MAAS curtin preseed +make install-preseed install the preseed on this host (root) +make upload upload the image to MAAS +make lint syntax-check the scripts +make clean remove intermediates +make distclean remove build/ entirely +``` + +--- + +## First-boot behaviour + +`pve-maas-init.service` is enabled in the image and runs the following stages in order, +each exactly once: + +| Stage | What it does | +|---|---| +| `hosts` | Writes ` ` into `/etc/hosts` — `pvecm` requires the node name to resolve to a real address. Disables cloud-init's `manage_etc_hosts` so it is not reverted on later boots, fixes postfix's `myhostname`, and restarts the Proxmox services so they pick up the correct identity. | +| `identity` | Regenerates node-unique identifiers that must not be shared between nodes cloned from one image — currently the iSCSI initiator name. | +| `rootpw` | Sets the `root@pam` password. Without it the Proxmox web UI cannot be used and the node cannot be a join target for other nodes. | +| `network` | Converts the interface MAAS configured into a `vmbr0` bridge, disables the netplan/systemd-networkd configuration, and enables ifupdown2. | +| `cluster` | Creates a cluster or joins an existing one. | +| `storage` | Creates an LVM-thin pool in the free space of the volume group and registers it as `local-lvm`. | + +Progress and failures: + +```bash +systemctl status pve-maas-init +journalctl -u pve-maas-init -b +ls -l /var/lib/pve-maas/ # .done files; "complete" when finished +cat /etc/pve-maas/image-info # which image this node was built from +``` + +If a stage fails, the service exits non-zero and the remaining stages are retried on the +next boot. To force a stage to run again, delete its `.done` file along with `complete` +and restart the service. + +--- + +## Configuration reference + +Defaults live in `/etc/pve-maas/pve-maas.conf` (do not edit it). Per-node settings go +into `/etc/pve-maas/conf.d/*.conf`, which cloud-init writes from the user-data you pass +at deploy time. Files in `conf.d` override the defaults. + +```yaml +#cloud-config +write_files: + - path: /etc/pve-maas/conf.d/50-pve.conf + permissions: "0600" + owner: root:root + content: | + PVE_ROOT_PASSWORD_HASH='$6$...' + PVE_CLUSTER_MODE=join + PVE_CLUSTER_PEER=192.0.2.11 + PVE_CLUSTER_PEER_PASSWORD='...' +``` + +### General + +| Option | Default | Meaning | +|---|---|---| +| `PVE_ENABLED` | `true` | Set to `false` to disable all first-boot automation | +| `PVE_FQDN` | *(empty)* | Override the detected FQDN | + +### Credentials + +| Option | Default | Meaning | +|---|---|---| +| `PVE_ROOT_PASSWORD_HASH` | *(empty)* | `root@pam` password hash — generate with `openssl passwd -6` | +| `PVE_ROOT_PASSWORD` | *(empty)* | Plaintext alternative; prefer the hash | + +### Networking + +| Option | Default | Meaning | +|---|---|---| +| `PVE_NET_MANAGE` | `true` | Set to `false` to configure `/etc/network/interfaces` yourself | +| `PVE_NET_BRIDGE` | `vmbr0` | Bridge name | +| `PVE_NET_UPLINK` | *(auto)* | Bridge port; defaults to the interface holding the default route | +| `PVE_NET_MODE` | `auto` | `auto`, `static` or `dhcp` | +| `PVE_NET_APPLY` | `reboot` | `reboot`, `reload` (`ifreload -a`) or `none` | +| `PVE_NET_VLAN_AWARE` | `false` | Make the bridge VLAN-aware (`bridge-vids 2-4094`) | +| `PVE_NET_EXTRA` | *(empty)* | Raw text appended to `/etc/network/interfaces` | + +### Cluster + +| Option | Default | Meaning | +|---|---|---| +| `PVE_CLUSTER_MODE` | `none` | `none`, `create` or `join` | +| `PVE_CLUSTER_NAME` | *(empty)* | Cluster name, for `create` | +| `PVE_CLUSTER_PEER` | *(empty)* | Address of an existing member, for `join` | +| `PVE_CLUSTER_PEER_PASSWORD` | *(empty)* | That node's `root@pam` password | +| `PVE_CLUSTER_PEER_PASSWORD_FILE` | *(empty)* | Read the password from a file instead | +| `PVE_CLUSTER_FINGERPRINT` | *(empty)* | The peer's certificate SHA-256 fingerprint | +| `PVE_CLUSTER_FINGERPRINT_DISCOVER` | `true` | Read the fingerprint from the peer if not supplied (trust on first use) | +| `PVE_CLUSTER_LINK0` / `LINK1` | *(empty)* | This node's corosync link addresses | +| `PVE_CLUSTER_NODEID` / `VOTES` | *(empty)* | Passed through to Proxmox | +| `PVE_CLUSTER_WAIT` | `900` | Seconds to wait for the peer's API to answer | +| `PVE_CLUSTER_RETRIES` | `5` | Join attempts, 30 s apart | +| `PVE_CLUSTER_WIPE_SECRETS` | `true` | Scrub passwords from `conf.d` after joining | + +### Storage + +| Option | Default | Meaning | +|---|---|---| +| `PVE_THINPOOL` | `auto` | `auto` (the VG with the most free space), `off`, or a VG name | +| `PVE_THINPOOL_NAME` | `data` | Thin pool logical volume name | +| `PVE_THINPOOL_STORAGE` | `local-lvm` | Proxmox storage id | +| `PVE_THINPOOL_MIN_GB` | `16` | Skip the stage below this much free space | +| `PVE_THINPOOL_DISK` | *(empty)* | Build a new VG from this whole disk instead | +| `PVE_THINPOOL_VG` | `pve` | VG name used with `PVE_THINPOOL_DISK` | + +Worked examples: [`maas/examples/`](maas/examples/). + +--- + +## Deploying a cluster + +`pvecm add` prompts for a password interactively, which makes it unusable from a script. +This image uses the API endpoint that does the same job non-interactively: + +```bash +pvesh create /cluster/config/join --hostname --password --fingerprint +``` + +### By hand + +Deploy the first node with `PVE_CLUSTER_MODE=create` +([example](maas/examples/01-first-node.yaml)), then read its certificate fingerprint: + +```bash +openssl s_client -connect :8006 /dev/null \ + | openssl x509 -noout -fingerprint -sha256 | cut -d= -f2 +``` + +and deploy the remaining nodes with `PVE_CLUSTER_MODE=join` plus that fingerprint +([example](maas/examples/02-join-node.yaml)). + +A joining node waits for the peer's port 8006 to answer (`PVE_CLUSTER_WAIT`, 15 minutes +by default) and retries a failed join five times, so you can start several deployments at +once without ordering them carefully. + +### With the helper script + +[`scripts/deploy-cluster.sh`](scripts/deploy-cluster.sh) automates the whole flow: it +deploys the first node, waits for it, reads the fingerprint, and deploys the rest. + +Run it **on the MAAS region controller** — it needs the `maas` CLI profile and network +access to the nodes' port 8006. + +```bash +# Show what it would deploy, without deploying anything +./deploy-cluster.sh --name pve-prod --nodes pve1,pve2,pve3 --dry-run + +# Deploy +./deploy-cluster.sh --name pve-prod --nodes pve1,pve2,pve3 +``` + +The first host in `--nodes` creates the cluster; the rest join it. + +### Several independent clusters + +The image carries no cluster identity — it comes entirely from deploy-time user-data. +The same image and the same preseed can build any number of unrelated clusters, with no +rebuild: + +```bash +./deploy-cluster.sh --name pve-prod --nodes pve1,pve2,pve3 +./deploy-cluster.sh --name pve-dr --nodes dr1,dr2,dr3 +``` + +Each cluster gets its own root password (generated and printed if you do not supply one). +MAAS tags are a convenient way to keep the groups apart: + +```bash +maas $PROFILE tag create name=pve-prod +maas $PROFILE tag update-nodes pve-prod add=$SYSTEM_ID +maas $PROFILE machines read tags=pve-prod | jq -r '.[].hostname' +``` + +### Security notes + +- The peer's `root@pam` password **must be plaintext** — the join API does not accept a + hash. It is stored in MAAS user-data, where anyone with MAAS access can read it. Use a + short-lived password and change it after the cluster is up. +- Without `PVE_CLUSTER_FINGERPRINT` the fingerprint is read from the peer on first + contact, which is trust-on-first-use and open to interception. Supply it explicitly in + production; `deploy-cluster.sh` always does. +- Prefer `PVE_ROOT_PASSWORD_HASH` over `PVE_ROOT_PASSWORD` for the node's own password. +- With no `PVE_ROOT_PASSWORD*` at all, root stays locked: no `root@pam` web login, and + the node cannot be a join target. +- A two-node cluster loses quorum when either node goes down. Use three nodes or a + QDevice in production — that is a Proxmox property, not a limitation of this image. + +--- + +## Networking + +### Who owns the network, and when + +The image ships with `networking.service` (ifupdown2) **disabled** and an +`/etc/network/interfaces` containing nothing but loopback. On the first boot MAAS owns +the network entirely, through the netplan configuration curtin wrote and +systemd-networkd. Only once `pve-maas-init` reaches its network stage does it write the +`vmbr0` configuration, disable netplan and networkd, and enable ifupdown2. + +That order matters. If ifupdown2 were enabled in the image it would start with the stale +interface definition baked in at build time and take the real interface down before the +automation ever ran. + +### The bridge + +The network stage reads the live configuration — the interface holding the default +route, its address, gateway and DNS — and writes a Proxmox-style bridge: + +``` +auto lo +iface lo inet loopback + +iface enp6s18 inet manual + +auto vmbr0 +iface vmbr0 inet static + address 192.0.2.20/24 + gateway 192.0.2.1 + bridge-ports enp6s18 + bridge-stp off + bridge-fd 0 + +source /etc/network/interfaces.d/* +``` + +`PVE_NET_APPLY=reboot` (the default) reboots the node once to apply this, after +cloud-init has already told MAAS the deployment succeeded. `reload` uses ifupdown2's +`ifreload -a` and avoids the reboot at the cost of a brief interruption. + +### Complex topologies + +For bonds, VLANs or multiple bridges, set `PVE_NET_MANAGE=false` and write +`/etc/network/interfaces` yourself through cloud-init — see +[`maas/examples/04-advanced-network.yaml`](maas/examples/04-advanced-network.yaml). + +--- + +## Storage + +`local` (directory storage on `/var/lib/vz`) works out of the box. `local-lvm` is +created on first boot from free space in the volume group, which means **MAAS has to +leave some**. + +Choose the LVM storage layout in MAAS and make the root logical volume smaller than the +disk: + +```bash +maas $PROFILE machine set-storage-layout $SYSTEM_ID \ + storage_layout=lvm lv_size=12884901888 # 12 GiB root +``` + +The rest of the volume group is then turned into a thin pool named `data` and registered +as `local-lvm`. With less than `PVE_THINPOOL_MIN_GB` free the stage is skipped silently +and the node runs with directory storage only. + +To use a whole separate disk instead: + +``` +PVE_THINPOOL_DISK=/dev/sdb +PVE_THINPOOL_VG=pve +``` + +`storage.cfg` is cluster-wide, so a node joining an existing cluster does not overwrite a +`local-lvm` that is already defined — it adds itself to that storage's node list instead. + +--- + +## Four traps this image works around + +Each of these fails **silently**: no error, no failed unit, just a node that does not +work. They are documented here because anyone building a similar image will hit them. + +### 1. `kernel: null` in the curtin preseed + +The image already contains the Proxmox kernel, so curtin should not install one. Recent +curtin supports `kernel: null` for exactly this. The curtin that ships with MAAS may not: + +``` +install_kernel -> config.merge_config(mapping, kernel_cfg.get('mapping', {})) +AttributeError: 'NoneType' object has no attribute 'get' +``` + +Instead, the image ships `/curtin/curtin-hooks`. When that file exists, curtin runs it in +place of its built-in hooks; the hook neutralises the kernel-installation step and calls +the built-in hooks itself. This works regardless of curtin version and removes the +deployment's dependency on reaching `download.proxmox.com`. + +### 2. `pvenetcommit.service` + +Proxmox stages network changes in `/etc/network/interfaces.new`, and `pvenetcommit` +**moves that file over `interfaces` on every boot**, before `sysinit.target`: + +``` +ExecStart=sh -c 'if [ -f ${FN}.new ]; then mv ${FN}.new ${FN}; fi' +``` + +A `.new` file created during the build and left in the image silently overwrites the +node's configuration on its first boot. It is removed at build time, and again after the +network stage writes the real configuration. + +### 3. Interface names differ between commissioning and deployment + +MAAS commissions machines in an Ubuntu ephemeral environment and records the interface +name it sees there — for example `enp6s18`. The deployed Debian 13 system may use a +different udev naming scheme and call the same card `ens18`. cloud-init then tries to +rename it, fails, and **leaves the interface down**: + +``` +Failed to rename devices: [busy] Error renaming mac=… from ens18 to enp6s18 +``` + +The node loses its network moments after cloud-init fetches its metadata — late enough +that MAAS still reports a successful deployment. + +`curtin-hooks` writes the MAC-to-name mapping from MAAS's own network configuration into +`/etc/systemd/network/10-maas-.link`, so udev names the card correctly from the +start and no rename is attempted. + +### 4. systemd ordering — both a cycle and a deadlock + +Ordering `pve-maas-init.service` `After=cloud-final.service` while it is +`WantedBy=multi-user.target` creates a dependency cycle. systemd resolves it by +**deleting our job**: + +``` +multi-user.target: Found ordering cycle on pve-maas-init.service/start +multi-user.target: Found dependency on cloud-final.service/start +multi-user.target: Found dependency on multi-user.target/start +Job pve-maas-init.service/start deleted to break ordering cycle +``` + +The service never runs, and nothing reports an error. + +Waiting for cloud-init inside the script instead removes the cycle but introduces a +runtime deadlock if the unit is `Type=oneshot`, because such a unit blocks +`multi-user.target`, which `cloud-final.service` is ordered after: + +``` +cloud-final.service waiting +pve-maas-init.service running +multi-user.target waiting +``` + +The unit is therefore `Type=simple`, with no cloud-init ordering at all, and the script +calls `cloud-init status --wait` itself. Stage tracking uses `/var/lib/pve-maas/*.done` +rather than systemd state, so nothing is lost by dropping `RemainAfterExit`. + +--- + +## Moving to a new Proxmox release + +Check what is available without building anything: + +```bash +make check-upstream +``` + +It prints the current `proxmox-ve` / `pve-manager` versions in the configured repository +next to the metadata of the image you have (`/etc/pve-maas/image-info`). + +### Within the same Debian base (9.2 → 9.3 → …) + +Nothing to change. No package versions are pinned; every build takes the current +`proxmox-ve`: + +```bash +sudo make image && make verify && make upload +``` + +The image name and preseed filename stay the same, so the MAAS record is updated in +place. + +### A major release that changes the Debian base (e.g. PVE 10 on Debian 14) + +**This is not automatic**, and changing the variables in this repository is not enough. +The upstream packer-maas template contains hard-coded conditions on the Debian version: + +``` +debian/scripts/networking.sh: if [ ${DEBIAN_VERSION} == '12' ] || [ ${DEBIAN_VERSION} == '13' ] +debian/scripts/setup-boot.sh: if [ ${DEBIAN_VERSION} == '13' ] +``` + +An unknown version falls through to the `else` branch, which installs a cloud-init +package from 2020 — the image breaks silently. A major jump therefore waits on upstream +support. In order: + +1. Does `canonical/packer-maas` handle the new Debian? (look at the `DEBIAN_VERSION` + conditions in `debian/scripts/`) +2. Has Proxmox published the keyring? + `https://enterprise.proxmox.com/debian/proxmox-archive-keyring-.gpg` +3. Are the packages there? + `curl -s http://download.proxmox.com/debian/pve/dists//Release | grep Components` +4. Then: + +```bash +sudo make image PVE_VERSION=10 DEBIAN_SERIES=forky DEBIAN_VERSION=14 PM_REF=main +make preseed PVE_VERSION=10 +``` + +The preseed filename changes too (`…_proxmox-ve-10`) — install it on the region +controller. + +### Upstream pinning + +`PM_REF` points at a tested `canonical/packer-maas` commit. Leaving it at `main` means an +upstream change can break your build without warning. To move it forward: build with +`PM_REF=main`, **deploy the result and verify it**, then record the new SHA in the +Makefile. + +--- + +## Build performance + +A build takes roughly 11 minutes on a 4-vCPU builder, most of it installing packages +inside the build VM. These optimisations are on by default: + +- **`eatmydata`** — drops dpkg's per-package `fsync` calls. Safe here: the build VM's + disk is thrown away. +- **Deferred initramfs** — `update-initramfs` is diverted during installation and run + exactly once at the end, instead of being triggered by the kernel, firmware and dkms + packages in turn. +- **Stable cloud image** — a fixed URL means Packer's cache actually hits; upstream's + `daily` image changes every day, costing a ~350 MB download per build and making + builds non-reproducible. +- **`GZIP_LEVEL=6`** — upstream uses `--best` (9). With `pigz` this is noticeably faster + for a few percent more size. + +For repeated builds, a local APT cache removes about 700 MB of downloads: + +```bash +sudo make deps-cache # installs apt-cacher-ng +sudo make image APT_PROXY=http://10.0.2.2:3142 +``` + +`10.0.2.2` is the build host as seen from Packer's user-mode network. When a proxy is +configured, Debian repositories are rewritten from `https` to `http` so the cache can +serve them; package signatures are still verified. + +--- + +## Repository layout + +``` +Makefile build / preseed / upload targets +scripts/install-deps.sh build host dependencies +scripts/customize-proxmox.sh.in template for the script that runs inside the build VM +scripts/deploy-cluster.sh deploy a whole cluster through MAAS +scripts/verify-image.sh check a built image's contents +overlay/ files baked into the image + usr/local/sbin/pve-maas-init first-boot state machine + etc/pve-maas/pve-maas.conf defaults, with every option documented + etc/systemd/system/… pve-maas-init.service + curtin/curtin-hooks skips kernel install, pins interface names +maas/curtin_userdata_custom.in MAAS curtin preseed template +maas/examples/*.yaml cloud-init user-data examples +build/ generated artifacts (git-ignored) +``` + +The overlay is embedded into the generated customization script as a base64 payload +rather than served over Packer's HTTP server, which keeps the build self-contained. + +--- + +## Troubleshooting + +**Deployment ends in "Failed deployment".** +Read the installation log: + +```bash +maas $PROFILE node-script-result download $SYSTEM_ID current-installation \ + filetype=txt filters=/tmp/install.log output=all | tail -60 +``` + +The most common cause is a missing or misnamed preseed. The filename must match the +uploaded image exactly: `curtin_userdata_custom___`. Use what +`make preseed` produces. + +**MAAS says "Deployed" but the node is unreachable.** +Look at the console (`qm terminal ` on Proxmox, or your BMC). A login prompt means +the system booted and the problem is networking. Inspect the disk from rescue mode: + +```bash +maas $PROFILE machine rescue-mode $SYSTEM_ID +# then, over SSH to the ephemeral environment: +mount /dev/vgroot/lvroot /mnt/t +cat /mnt/t/etc/network/interfaces # was vmbr0 written? +ls -l /mnt/t/var/lib/pve-maas/ # which stages completed? +cat /mnt/t/var/log/cloud-init-output.log # did user-data arrive? +journalctl -D /mnt/t/var/log/journal -u pve-maas-init +``` + +An empty `/var/lib/pve-maas/` means the service never ran — check for the systemd +ordering cycle described above. + +**Deployed but the web UI rejects the login.** +Was a root password supplied? `journalctl -u pve-maas-init -b | grep rootpw`. Check the +bridge with `ip -br addr`. + +**A node did not join the cluster.** +`journalctl -u pve-maas-init -b`. Common causes: the peer is unreachable on 8006, the +root password is wrong, the fingerprint does not match, or the peer's own `/etc/hosts` +is wrong so `pvecm` fails there. To retry without redeploying: + +```bash +rm -f /var/lib/pve-maas/cluster.done /var/lib/pve-maas/complete +systemctl start pve-maas-init +``` + +**The node reports the build hostname.** +It should not — the pmxcfs database is removed from the image and regenerated from the +current hostname on first boot. If it happens, `/var/lib/pve-cluster/config.db` survived +the build; check the "resetting node identity" step in the build log. + +**The build reports that `proxmox-ve` cannot be found.** +`DEBIAN_SERIES` and `PVE_REPO` disagree. Verify: +`curl -s http://download.proxmox.com/debian/pve/dists//Release | grep Components` + +**Do not delete or replace a boot resource while a machine is using it.** +MAAS loses the machine's boot files and it can end up stuck in "Failed to exit rescue +mode", with GRUB reporting `invalid magic number`. + +--- + +## Verified status + +This repository was tested end to end against live hardware-backed infrastructure: + +| | | +|---|---| +| Image | proxmox-ve 9.2.0 / pve-manager 9.2.11 / kernel 7.0.14-15-pve | +| MAAS | 3.7.2 (snap) on Ubuntu 24.04, region + rack | +| Deployment | `custom/proxmox-ve-9`, amd64/generic, LVM layout with a 12 GiB root | +| Result | Two-node cluster, `Quorate: Yes`, `local-lvm` thin pool on both nodes | +| Automation | Bridge conversion, root password, node identity, cluster create and join — all without logging in | + +Not yet exercised: `PVE_CLUSTER_LINK0` (a separate corosync network), the +`pve-enterprise` repository, arm64, and `deploy-cluster.sh` beyond `--dry-run`. + +--- + +## Credits and licensing + +Built on [`canonical/packer-maas`](https://github.com/canonical/packer-maas) (AGPL-3.0), +which is cloned at build time rather than vendored here. + +References: + +- [MAAS — Build custom images](https://canonical.com/maas/docs/how-to-build-custom-images) +- [Proxmox VE — Install on Debian 13](https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_13_Trixie) +- [Proxmox VE — Cluster Manager](https://pve.proxmox.com/wiki/Cluster_Manager) +- [`pvecm(1)`](https://pve.proxmox.com/pve-docs/pvecm.1.html) + +No license has been chosen for this repository yet; add one before expecting others to +reuse it. diff --git a/README.tr.md b/README.tr.md new file mode 100644 index 0000000..6761f22 --- /dev/null +++ b/README.tr.md @@ -0,0 +1,521 @@ +> **Not:** Kanonik dokumantasyon [README.md](README.md) (Ingilizce). Bu Turkce +> surum geride kalmis olabilir. + +# maas-proxmox + +MAAS ile bare-metal sunuculara deploy edilebilen **Proxmox VE** imajlari uretir. +Deploy edilen dugum ilk acilista kendini yapilandirir: `vmbr0` koprusunu kurar, +`local-lvm` thin havuzunu olusturur ve istenirse bir Proxmox kumesine otomatik katilir. + +Varsayilan hedef: **Proxmox VE 9.x** (Debian 13 "trixie" tabanli). + +--- + +## Nasil calisiyor? + +``` +Debian 13 cloud image (qcow2) + │ + ├─ packer-maas / debian sablonu (QEMU + KVM) + │ ├─ cloud-init, netplan, curtin uyumlulugu [upstream] + │ ├─ disk/cpu/ram yamasi (4G -> 16G) [bu proje] + │ └─ customize-proxmox.sh [bu proje] + │ ├─ Proxmox APT deposu + anahtarlik + │ ├─ proxmox-default-kernel + proxmox-ve + │ ├─ Debian cekirdegi / os-prober kaldirilir + │ ├─ pmxcfs dugum kimligi sifirlanir + │ └─ overlay: pve-maas-init + systemd unit + curtin-hooks + │ + └─ proxmox-ve-9.tar.gz ──► maas boot-resources create + │ + ▼ + MAAS deploy (curtin) + │ + ▼ + pve-maas-init.service (ilk acilis) + hosts → identity → rootpw → + network → cluster → storage +``` + +Neden Proxmox ISO'su degil de Debian uzerine kurulum? MAAS imaji disk bolumlemesini, +ag yapilandirmasini, kullanici/SSH anahtarlarini ve cloud-init'i **curtin** ile kendisi +yonetir. Debian tabanli imaj bu akisa dogal olarak uyar; ISO'dan yakalanan ham disk +imaji (`dd.gz`) ise MAAS'in disk duzenini tamamen devre disi birakir. + +--- + +## Dogrulanmis durum + +Bu depo canli bir ortamda ucdan uca test edildi: + +| | | +|---|---| +| Imaj | proxmox-ve 9.2.0 / pve-manager 9.2.11 / kernel 7.0.14-15-pve | +| MAAS | 3.7.2 (snap), Ubuntu 24.04 region+rack | +| Deploy | `custom/proxmox-ve-9`, amd64/generic, LVM duzeni (kok 12GiB) | +| Sonuc | 2 dugumlu kume, `Quorate: Yes`, her iki dugumde `local-lvm` thin havuzu | +| Otomasyon | vmbr0 donusumu, root parolasi, dugum adi, kume olusturma + katilma - hepsi elle mudahalesiz | + +## Gereksinimler + +**Build host** (imajin derlendigi makine): + +* Ubuntu 22.04+ (24.04 LTS onerilir), x86_64 +* **Nested virtualization / KVM erisimi** — VM ise CPU tipi `host` olmali +* 4+ vCPU, 8+ GB RAM, 25+ GB bos disk +* `sudo` yetkisi + +**Deploy tarafi:** + +* MAAS 3.2+ (custom image destegi) +* Curtin 21.0+ +* Region controller uzerinde bu depodaki curtin preseed dosyasi (asagida) + +--- + +## Hizli baslangic + +```bash +# 1) Build host bagimliliklari (packer, qemu, nbdkit, ovmf, ...) +sudo ./scripts/install-deps.sh + +# 2) Imaji derle (~20-40 dk, ag hizina bagli) +sudo make image +# -> build/proxmox-ve-9.tar.gz + +# 3) MAAS curtin preseed dosyasini uret +make preseed +# -> build/curtin_userdata_custom_amd64_generic_proxmox-ve-9 + +# 4) Preseed'i MAAS region controller'a kopyala +sudo make install-preseed +# (varsayilan: /var/snap/maas/current/preseeds/ ; deb kurulumunda +# MAAS_PRESEED_DIR=/etc/maas/preseeds kullanin) + +# 5) Imaji MAAS'a yukle +make upload MAAS_PROFILE=admin +``` + +Ardindan MAAS arayuzunde makineyi deploy ederken OS olarak +**Custom → Proxmox VE 9** secin, ya da CLI ile: + +```bash +maas $PROFILE machine deploy $SYSTEM_ID \ + osystem=custom distro_series=proxmox-ve-9 \ + user_data="$(base64 -w0 maas/examples/01-first-node.yaml)" +``` + +--- + +## Makefile degiskenleri + +| Degisken | Varsayilan | Aciklama | +|---|---|---| +| `PVE_VERSION` | `9` | Proxmox VE ana surumu (imaj adinda kullanilir) | +| `DEBIAN_SERIES` | `trixie` | PVE'nin dayandigi Debian kod adi | +| `DEBIAN_VERSION` | `13` | Debian ana surum numarasi | +| `PVE_REPO` | `pve-no-subscription` | `pve-no-subscription` / `pve-enterprise` / `pve-test` | +| `PVE_EXTRA_PACKAGES` | `ifupdown2 open-iscsi chrony ...` | Imaja eklenecek ek paketler | +| `IMAGE_NAME` | `proxmox-ve-9` | MAAS'taki `custom/` ve preseed dosya adi | +| `ARCH` / `BOOT` | `amd64` / `uefi` | Mimari ve onyukleme modu | +| `DISK_SIZE` | `16G` | Build VM disk boyutu (upstream sablondaki 4G yetersiz) | +| `BUILD_CPUS` / `BUILD_MEM` | `4` / `4096` | Build VM kaynaklari | +| `DEBIAN_IMAGE_CHANNEL` | `stable` | `stable` sabit URL kullanir (packer onbellegi isabet eder); `daily` upstream varsayilani | +| `GZIP_LEVEL` | `6` | Tarball sikistirma seviyesi (upstream 9; 6 belirgin hizli, imaj ~%2-3 buyuk) | +| `APT_PROXY` | *(bos)* | Yerel APT onbellegi, or. `http://10.0.2.2:3142` - bkz. `make deps-cache` | +| `PM_REF` | `main` | Kullanilacak `canonical/packer-maas` surumu (dal, etiket ya da commit SHA) | +| `MAAS_PROFILE` | `admin` | `maas` CLI profil adi | +| `MAAS_PRESEED_DIR` | `/var/snap/maas/current/preseeds` | Preseed dizini | +| `TIMEOUT` | `3h` | Packer build zaman asimi | + +### Derlemeyi hizlandirma + +Sure agirlikli olarak VM icindeki paket kurulumunda geciyor. Uygulanan +optimizasyonlar (varsayilan olarak acik): + +* **eatmydata** - dpkg'nin her paket icin yaptigi `fsync` cagrilari devre disi. + Imaj derlemede guvenli, cunku build VM'inin diski zaten atilabilir. +* **initramfs erteleme** - `update-initramfs` kurulum boyunca diverte edilir ve + en sonda yalnizca bir kez calistirilir (cekirdek + firmware + dkms + tetikleyicileri normalde defalarca calistiriyor). +* **Kararli cloud image** - sabit URL sayesinde packer'in onbellegi isabet eder; + `daily` her gun degistigi icin her derlemede ~350MB yeniden inerdi. +* **`GZIP_LEVEL=6`** - upstream `--best` (9) kullaniyor; 6 ile `pigz` cok + cekirdekli calisip belirgin hizlaniyor. + +Tekrarlanan derlemeler icin en buyuk ek kazanc yerel APT onbellegi: + +```bash +sudo make deps-cache # apt-cacher-ng kurar +sudo make image APT_PROXY=http://10.0.2.2:3142 # ~700MB indirme onbellekten +``` + +`10.0.2.2`, packer'in user-mode aginda build host'un adresidir. Proxy verildiginde +Debian depolari `https` yerine `http` kullanacak sekilde yeniden yazilir (onbellek +CONNECT tunelini saklayamaz); paket imzalari yine dogrulandigi icin guvenlik +kaybi yoktur. + +### Yeni bir Proxmox surumune gecmek + +Once ne oldugunu gorun - derleme yapmadan, saniyeler icinde: + +```bash +make check-upstream +``` + +Depodaki guncel `proxmox-ve` / `pve-manager` surumlerini, elinizdeki imajin +kunyesiyle (`/etc/pve-maas/image-info`) yan yana gosterir. + +**Durum 1 - ayni Debian tabani icinde surum yukseltmesi (9.2 -> 9.3 -> ...)** + +Hicbir sey degistirmeniz gerekmez. Imajda surum sabitlemesi yok; her derleme +depodaki en guncel `proxmox-ve` paketini alir: + +```bash +sudo make image && make verify && make upload +``` + +Imaj adi (`proxmox-ve-9`) ve preseed dosyasi ayni kaldigi icin MAAS'taki kayit +yerinde guncellenir, preseed'e dokunmaniza gerek kalmaz. + +**Durum 2 - Debian tabanini degistiren ana surum (or. PVE 10 / Debian 14)** + +Bu **otomatik degildir**; bu depodaki degiskenleri degistirmek tek basina yetmez. +Sebep: temel aldigimiz `canonical/packer-maas` sablonunun icinde Debian surumune +ozel sabit kosullar var - + +``` +debian/scripts/networking.sh: if [ ${DEBIAN_VERSION} == '12' ] || [ ${DEBIAN_VERSION} == '13' ] +debian/scripts/setup-boot.sh: if [ ${DEBIAN_VERSION} == '13' ] +``` + +Bilinmeyen bir surumde bu kosullar `else` dalina duser ve 2020 tarihli bir +cloud-init paketi kurulur; imaj sessizce bozulur. Bu yuzden ana surum gecisi +**upstream'in yeni Debian'i desteklemesini bekler**. Sirasi: + +1. `canonical/packer-maas` yeni Debian'i destekliyor mu? + (`debian/scripts/` icindeki `DEBIAN_VERSION` kosullarina bakin) +2. Proxmox anahtarligi yayinlanmis mi? + `https://enterprise.proxmox.com/debian/proxmox-archive-keyring-.gpg` +3. Depoda paketler var mi? + `curl -s http://download.proxmox.com/debian/pve/dists//Release | grep Components` +4. Sonra: + +```bash +sudo make image PVE_VERSION=10 DEBIAN_SERIES=forky DEBIAN_VERSION=14 PM_REF=main +make preseed PVE_VERSION=10 +``` + +Yeni preseed dosyasinin adi da degisir (`..._proxmox-ve-10`) - MAAS'a kurmayi +unutmayin. + +### Upstream surum sabitlemesi + +`PM_REF`, test edilmis bir `canonical/packer-maas` commit'ine sabitlenmistir. +`main` birakmak, upstream'de yapilan bir degisikligin derlemenizi haber vermeden +bozmasi anlamina gelir. Ileri tasirken `PM_REF=main` ile derleyip **deploy ederek** +test edin, sonra yeni SHA'yi Makefile'a yazin. + +--- + +## Dugumun ilk acilis davranisi + +Imajda `pve-maas-init.service` etkin gelir. Servis su asamalari sirayla, her birini +yalnizca bir kez calistirir (`/var/lib/pve-maas/.done`): + +| Asama | Ne yapar | +|---|---| +| `hosts` | `/etc/hosts` icinde ` ` satirini kurar (pvecm bunu sart kosar), postfix `myhostname` degerini duzeltir | +| `identity` | Dugume ozel kimlikleri yeniden uretir (iSCSI IQN) | +| `rootpw` | `root@pam` parolasini ayarlar | +| `network` | MAAS'in yapilandirdigi arayuzu `vmbr0` koprusune donusturur, cloud-init/netplan ag yapilandirmasini devre disi birakir | +| `cluster` | Kume olusturur (`create`) veya mevcut kumeye katilir (`join`) | +| `storage` | VG'deki bos alanda LVM-thin havuzu acar ve `local-lvm` olarak tanimlar | + +Bir asama basarisiz olursa servis hata verir ve **sonraki acilista kaldigi yerden +devam eder**. Durum: + +```bash +systemctl status pve-maas-init +journalctl -u pve-maas-init -b +ls -l /var/lib/pve-maas/ +``` + +Yapilandirma `/etc/pve-maas/pve-maas.conf` (varsayilanlar) ve +`/etc/pve-maas/conf.d/*.conf` (cloud-init ile yazilan, ezici degerler) dosyalarindan +okunur. Tum secenekler icin [`overlay/etc/pve-maas/pve-maas.conf`](overlay/etc/pve-maas/pve-maas.conf). + +### Proxmox + MAAS birlesiminin tuzaklari + +Bu imaj asagidaki uc sorunu bilerek cozer. Kendi turevinizi yazacaksaniz +bunlari bozmayin: + +**1. systemd siralama dongusu.** `pve-maas-init.service` icin +`After=cloud-final.service` + `WantedBy=multi-user.target` yazmak bu imajda +donguye yol aciyor ve systemd dongoyu kirmak icin *bizim* servisimizi siliyor: + +``` +multi-user.target: Found ordering cycle on pve-maas-init.service/start +Job pve-maas-init.service/start deleted to break ordering cycle +``` + +Servis sessizce hic calismaz. Cozum: unit'te cloud-init siralamasi yok; +script kendi icinde `cloud-init status --wait` ile bekliyor. + +**2. `pvenetcommit.service`.** Proxmox ag degisikliklerini once +`/etc/network/interfaces.new` dosyasina yazar; `pvenetcommit` her acilista bu +dosyayi `interfaces` uzerine **tasir**. Build sirasinda olusan bir `.new` +dosyasi imajda kalirsa dugumun ilk acilisinda yapilandirmayi ezer. Hem imajda +hem de ag donusumunden sonra siliniyor. + +**3. Arayuz adi uyusmazligi.** MAAS commissioning'i Ubuntu ephemeral ortaminda +yapar ve karti orada gordugu adla kaydeder (or. `enp6s18`). Deploy edilen +Debian 13 / Proxmox ise udev'in farkli isimlendirme semasi yuzunden ayni karta +`ens18` diyebilir. Bu durumda cloud-init acilista yeniden adlandirmayi dener, +`[busy]` hatasi alir ve **arayuzu kapali birakir** - dugum agini tamamen +kaybeder: + +``` +Failed to rename devices: [busy] Error renaming mac=... from ens18 to enp6s18 +``` + +`curtin-hooks`, MAAS'in ag yapilandirmasindaki MAC -> ad eslemesini +`/etc/systemd/network/10-maas-.link` olarak yazar; udev karti en bastan +dogru adla olusturur ve yeniden adlandirma gerekmez. + +### Ilk acilista ag sahipligi + +Imaj `networking.service`'i (ifupdown2) **devre disi** ve `/etc/network/interfaces` +dosyasini yalnizca `lo` icerecek sekilde gonderir. Ilk acilista agi tamamen MAAS +yonetir (curtin'in yazdigi netplan + systemd-networkd). `pve-maas-init` ag asamasi +`vmbr0` yapilandirmasini yazdiktan sonra netplan/networkd'yi devre disi birakip +`networking.service`'i kendisi etkinlestirir. + +Bu sirala onemli: ifupdown2 etkin gelirse, imajdan gelen eski arayuz tanimiyla +acilista gercek arayuzu kapatir ve dugum daha otomasyon calismadan agini kaybeder. + +### Ag donusumu + +`PVE_NET_APPLY=reboot` (varsayilan) ile dugum, cloud-init MAAS'a "deploy tamamlandi" +sinyalini gonderdikten **sonra** bir kez yeniden baslar ve `vmbr0` ile acilir. +Yeniden baslatma istemiyorsaniz `PVE_NET_APPLY=reload` ifupdown2'nin `ifreload -a` +komutunu kullanir. + +Bond/VLAN gibi karmasik topolojilerde `PVE_NET_MANAGE=false` yapip +`/etc/network/interfaces` dosyasini cloud-init ile kendiniz yazin — +bkz. [`maas/examples/04-advanced-network.yaml`](maas/examples/04-advanced-network.yaml). + +### Depolama + +MAAS'in disk duzeni bir LVM VG birakiyorsa (MAAS'ta **LVM storage layout** secip +kok LV'yi kucultun), artan alanda `data` adinda bir thin havuz acilir ve `local-lvm` +olarak tanimlanir. Ayri bir disk kullanmak icin: + +``` +PVE_THINPOOL_DISK=/dev/sdb +PVE_THINPOOL_VG=pve +``` + +Bos alan `PVE_THINPOOL_MIN_GB` (varsayilan 16 GiB) altindaysa asama sessizce atlanir; +dugum yalnizca dizin tabanli `local` deposuyla calisir. + +--- + +## Kume otomasyonu + +`pvecm add` parolayi etkilesimli sorar, bu yuzden ayni isi yapan API ucu kullanilir: + +``` +pvesh create /cluster/config/join --hostname --password --fingerprint +``` + +**Ilk dugum** ([`01-first-node.yaml`](maas/examples/01-first-node.yaml)): + +```yaml +PVE_CLUSTER_MODE=create +PVE_CLUSTER_NAME=pve-cluster-01 +``` + +**Katilan dugumler** ([`02-join-node.yaml`](maas/examples/02-join-node.yaml)): + +```yaml +PVE_CLUSTER_MODE=join +PVE_CLUSTER_PEER=192.0.2.11 +PVE_CLUSTER_PEER_PASSWORD='...' +PVE_CLUSTER_FINGERPRINT='AA:BB:...' # onerilir +``` + +Katilan dugum, hedef dugumun 8006 portu acilana kadar bekler +(`PVE_CLUSTER_WAIT`, varsayilan 900 s) ve basarisiz denemeleri tekrarlar +(`PVE_CLUSTER_RETRIES`). Katilim bittikten sonra `conf.d` icindeki parolalar +temizlenir (`PVE_CLUSTER_WIPE_SECRETS=true`). + +### Birden fazla bagimsiz kume + +Imaj tamamen jeneriktir - kume kimligi **yalnizca** deploy anindaki cloud-init +user-data'sindan gelir. Ayni imaj ve ayni preseed ile istediginiz kadar ayri +kume kurabilirsiniz; **yeniden derleme gerekmez**. + +Elle yapmak zahmetli oldugu icin ([`scripts/deploy-cluster.sh`](scripts/deploy-cluster.sh)) +tum akisi otomatiklestirir: ilk dugumu `create` ile deploy eder, ayaga kalkmasini +bekler, sertifika parmak izini okur ve kalan dugumleri o parmak iziyle `join` +modunda deploy eder. + +**MAAS region controller uzerinde** calistirin (`maas` CLI profili ve dugumlerin +8006 portuna erisim gerekir): + +```bash +# Once ne uretecegini gorun - hicbir sey deploy etmez +./deploy-cluster.sh --name pve-prod --nodes pve1,pve2,pve3 --dry-run + +# Gercek kurulum +./deploy-cluster.sh --name pve-prod --nodes pve1,pve2,pve3 + +# Ikinci, tamamen bagimsiz kume - ayni imaj, ayni preseed +./deploy-cluster.sh --name pve-dr --nodes dr1,dr2,dr3 +``` + +Listedeki **ilk dugum kumeyi olusturur**, digerleri ona katilir. Her kume kendi +root parolasini alir (vermezseniz uretilir ve ekrana yazilir). + +Ayri bir corosync agi kullaniyorsaniz: + +```bash +./deploy-cluster.sh --name pve-prod --nodes pve1,pve2,pve3 --link0-prefix 10.10.20. +``` + +Her dugumun o onekle baslayan adresi MAAS'tan bulunup `PVE_CLUSTER_LINK0` olarak +yazilir. (Bu yol henuz canli test edilmedi.) + +Cok dugumlu kumelerde `--serial` ile katilanlar teker teker deploy edilir. +Varsayilan paralel moddur; es zamanli katilim denemeleri olursa `pve-maas-init` +zaten 5 kez, 30 saniye arayla tekrar dener. + +**Kumeleri MAAS'ta ayirmak icin** etiket kullanmak ise yarar: + +```bash +maas $PROFILE tag create name=pve-prod +maas $PROFILE tag update-nodes pve-prod add=$SYSTEM_ID +maas $PROFILE machines read tags=pve-prod | jq -r '.[].hostname' +``` + +### Guvenlik notlari + +* Hedef dugumun **root@pam parolasi duz metin olmak zorundadir** — API dogrulamasi + hash kabul etmez. Bu deger cloud-init user-data icinde MAAS'ta saklanir; MAAS + user-data'ya erisimi olan herkes gorebilir. Kisa omurlu bir parola kullanip + kurulumdan sonra degistirmeyi dusunun. +* `PVE_CLUSTER_FINGERPRINT` vermezseniz parmak izi hedef dugumden okunur (TOFU) — + ilk baglantida MITM'e aciktir. Uretimde parmak izini onceden verin. +* Dugumun kendi `root` parolasi icin `PVE_ROOT_PASSWORD_HASH` kullanin + (`openssl passwd -6`), duz metin `PVE_ROOT_PASSWORD` yerine. +* `PVE_ROOT_PASSWORD*` hic verilmezse root parolasi kilitli kalir: web arayuzune + `root@pam` ile girilemez ve bu dugum baska bir dugumun katilim hedefi olamaz. + +--- + +## Dizin yapisi + +``` +Makefile derleme / preseed / yukleme hedefleri +scripts/install-deps.sh build host bagimliliklari +scripts/customize-proxmox.sh.in build VM icinde calisan Proxmox kurulum sablonu +scripts/deploy-cluster.sh MAAS uzerinden komple bir kume kurar (cok kumeli senaryolar) +scripts/verify-image.sh uretilen imajin icerik dogrulamasi +overlay/ imaja kopyalanan dosyalar + usr/local/sbin/pve-maas-init ilk acilis durum makinesi + etc/pve-maas/pve-maas.conf varsayilan yapilandirma (tum secenekler) + etc/systemd/system/... pve-maas-init.service + curtin/curtin-hooks curtin'in apt ile cekirdek kurmasini engeller +maas/curtin_userdata_custom.in MAAS curtin preseed sablonu +maas/examples/*.yaml cloud-init user-data ornekleri +build/ uretilen dosyalar (git'e girmez) +``` + +--- + +### packer-maas uzerindeki degisiklikler + +`make image`, `build/packer-maas` altina klonlanan upstream sablonda tek bir yama +yapar: `debian-cloudimg.pkr.hcl` icindeki `disk_size` (4G -> `DISK_SIZE`), `cpus` ve +`memory` degerleri. Upstream 4G ile gelir; Debian cloud image (~3G) uzerine Proxmox VE +kurulumu bu alana sigmaz. Uretilen `.tar.gz` yalnizca kullanilan dosyalari icerdigi +icin buyuk sanal disk son imaji buyutmez. + +Yama her `make image` calistiginda `git checkout -f` sonrasi yeniden uygulanir; +upstream'i `PM_REF` ile bir commit SHA'ya sabitleyerek surprizleri onleyebilirsiniz. + +--- + +## Sorun giderme + +**Deploy "Failed deployment" ile bitiyor** +Preseed dosyasi dogru adla region controller'da mi? Ad, yuklenen imajin adiyla +birebir eslesmelidir: `curtin_userdata_custom___`. +`make preseed` ciktisindaki dosya adini kullanin. + +**Deploy tamam ama web arayuzune girilemiyor** +`root@pam` parolasi ayarlandi mi? `journalctl -u pve-maas-init -b | grep rootpw`. +Ayrica `vmbr0` ayaga kalkmis mi: `ip -br addr`. + +**Dugum kumeye katilmadi** +`journalctl -u pve-maas-init -b` ciktisina bakin. Sik nedenler: hedef dugum 8006'da +erisilemiyor, root parolasi yanlis, parmak izi eslesmiyor, ya da hedef dugumun +`/etc/hosts` kaydi eksik oldugu icin `pvecm` orada hata veriyor. +Servisi elle tekrar calistirmak icin: + +```bash +rm -f /var/lib/pve-maas/cluster.done /var/lib/pve-maas/complete +systemctl start pve-maas-init +``` + +**Deploy "Deployed" diyor ama dugume hic erisilemiyor** +Dugumun konsoluna bakin (Proxmox'ta `qm terminal `). Login istemi geliyorsa +sistem acilmis, sorun agda demektir. Kurtarma modunda diski inceleyin: + +```bash +maas $PROFILE machine rescue-mode $SYSTEM_ID +# ephemeral ortama SSH ile baglanip: +mount /dev/vgroot/lvroot /mnt/t +cat /mnt/t/etc/network/interfaces # vmbr0 yazilmis mi? +ls -l /mnt/t/var/lib/pve-maas/ # hangi asamalar tamamlandi? +cat /mnt/t/var/log/cloud-init-output.log # cloud-init user-data'yi aldi mi? +journalctl -D /mnt/t/var/log/journal -u pve-maas-init +``` + +`/var/lib/pve-maas/` bosken ag da kapaliysa, `networking.service` ile MAAS'in +netplan yapilandirmasi cakisiyor demektir; imajda bu servis devre disi olmali. + +**Dugum adi imajdaki hostname olarak gorunuyor** +Olmamasi gerekir — pmxcfs veritabani imajdan silinir ve ilk acilista guncel +hostname ile yeniden uretilir. Boyle bir durumda `/var/lib/pve-cluster/config.db` +imajda kalmis demektir; `make image` ciktisinda "Proxmox dugum kimligi sifirlaniyor" +adimini kontrol edin. + +**Deploy "Installing OS" sonrasi "Configuring OS" asamasinda patliyor** +Kurulum kaydina bakin: + +```bash +maas $PROFILE node-script-result download $SYSTEM_ID current-installation \ + filetype=txt filters=/tmp/install.log output=all | tail -60 +``` + +`install_kernel -> AttributeError: 'NoneType' object has no attribute 'get'` +goruyorsaniz preseed'de `kernel: null` kullanilmis demektir. MAAS ile gelen +curtin surumlerinin bir kismi bunu desteklemez. Bu depodaki preseed bunu +kullanmaz; cekirdek kurulumu imajdaki `/curtin/curtin-hooks` ile devre disi +birakilir. Preseed'i elle duzenlediyseniz geri alin. + +**Build "proxmox-ve paketi bulunamadi" diyor** +`PVE_SUITE`/`DEBIAN_SERIES` ile `PVE_REPO` uyumsuz. Depoyu dogrulayin: +`curl -s http://download.proxmox.com/debian/pve/dists//Release | grep Components` + +--- + +## Kaynaklar + +* [canonical/packer-maas](https://github.com/canonical/packer-maas) — temel Packer sablonlari (AGPL-3.0) +* [MAAS: Build custom images](https://canonical.com/maas/docs/how-to-build-custom-images) +* [Proxmox VE: Install on Debian](https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_13_Trixie) +* [Proxmox VE: Cluster Manager](https://pve.proxmox.com/wiki/Cluster_Manager) +* [pvecm(1)](https://pve.proxmox.com/pve-docs/pvecm.1.html) diff --git a/maas/curtin_userdata_custom.in b/maas/curtin_userdata_custom.in new file mode 100644 index 0000000..b48ae64 --- /dev/null +++ b/maas/curtin_userdata_custom.in @@ -0,0 +1,57 @@ +#cloud-config +# +# MAAS curtin preseed - Proxmox VE ozel imaji icin +# +# Dosya adi: curtin_userdata_custom_@@ARCH@@_generic_@@IMAGE_NAME@@ +# Kopyalanacak yer (MAAS snap): /var/snap/maas/current/preseeds/ +# (MAAS deb paketi): /etc/maas/preseeds/ +# +# Bu ad yalnizca 'custom/@@IMAGE_NAME@@' adiyla, '@@ARCH@@/generic' mimarisiyle +# yuklenen imaja uygulanir; diger ozel imajlariniz etkilenmez. + +# Cekirdek kurulumu imajdaki /curtin/curtin-hooks tarafindan devre disi +# birakilir (Proxmox cekirdegi imajda hazir gelir). Asagidaki blok yalnizca +# kanca herhangi bir nedenle calismazsa devreye giren yedek yoldur. +# +# DIKKAT: burada 'kernel: null' KULLANMAYIN. MAAS ile gelen curtin +# surumlerinin bir kismi bunu desteklemiyor ve kurulum su hatayla basarisiz +# oluyor: install_kernel -> AttributeError: 'NoneType' object has no +# attribute 'get' +kernel: + package: proxmox-default-kernel + fallback-package: proxmox-default-kernel + mapping: {} + +apt: + preserve_sources_list: true + +debconf_selections: + maas: | + {{for line in str(curtin_preseed).splitlines()}} + {{line}} + {{endfor}} + +late_commands: + maas: [wget, '--no-proxy', '{{node_disable_pxe_url}}', '--post-data', '{{node_disable_pxe_data}}', '-O', '/dev/null'] + + # Hedef sistemi /mnt altina bagla + late_01: mount --bind $TARGET_MOUNT_POINT /mnt + + # MAAS datasource tanimini hedef sisteme tasi + late_02: grep -A2 datasource /etc/cloud/cloud.cfg.d/91_kernel_cmdline_url.cfg | sed 's/curtin//' | tee /mnt/etc/cloud/cloud.cfg.d/91_maas_datasource.cfg + + # Debian cloud-init sablonundaki Ubuntu referanslarini duzelt + late_03: sed -i 's@ubuntu.com/ubuntu@debian.org/debian@g;s@archive@deb@g;s@ubuntu@debian@g;s@Ubuntu@Debian@g;s@security.debian.org/debian@security.debian.org@g' /mnt/etc/cloud/cloud.cfg + + # zz-update-grub kancasindaki 'set -e' tuzagini kaldir + late_04: sed -i '/^set -e/{n;N;d}' /mnt/etc/kernel/postinst.d/zz-update-grub + + # packer-maas'in netplan.io kontrolunu atlatmak icin koydugu sarmalayicilari sil + late_05: rm -f /usr/local/bin/dpkg-query /usr/local/bin/netplan + late_06: rm -f /mnt/usr/local/bin/dpkg-query /mnt/usr/local/bin/netplan + + # Guvenlik agi: ilk acilis servisi kesin etkin olsun + late_07: chroot /mnt systemctl enable pve-maas-init.service || true + + # Onceki calistirmadan kalan durum dosyalari varsa temizle + late_08: rm -f /mnt/var/lib/pve-maas/*.done /mnt/var/lib/pve-maas/complete diff --git a/maas/examples/01-first-node.yaml b/maas/examples/01-first-node.yaml new file mode 100644 index 0000000..eaafc9d --- /dev/null +++ b/maas/examples/01-first-node.yaml @@ -0,0 +1,25 @@ +#cloud-config +# Kumenin ILK dugumu: yeni bir kume olusturur. +# +# MAAS ile deploy: +# maas $PROFILE machine deploy $SYSTEM_ID \ +# osystem=custom distro_series=proxmox-ve-9 \ +# user_data="$(base64 -w0 01-first-node.yaml)" + +write_files: + - path: /etc/pve-maas/conf.d/50-pve.conf + permissions: "0600" + owner: root:root + content: | + # root@pam parolasi - web arayuzu ve diger dugumlerin katilimi icin sart. + # Hash uretmek icin: openssl passwd -6 + PVE_ROOT_PASSWORD_HASH='$6$DEGISTIRIN$REPLACE.WITH.YOUR.OWN.HASH' + + PVE_CLUSTER_MODE=create + PVE_CLUSTER_NAME=pve-cluster-01 + + # Corosync icin ayri bir ag kullaniyorsaniz: + # PVE_CLUSTER_LINK0=198.51.100.11 + + PVE_NET_APPLY=reboot + PVE_THINPOOL=auto diff --git a/maas/examples/02-join-node.yaml b/maas/examples/02-join-node.yaml new file mode 100644 index 0000000..268a8e0 --- /dev/null +++ b/maas/examples/02-join-node.yaml @@ -0,0 +1,31 @@ +#cloud-config +# Mevcut bir kumeye katilan dugum. + +write_files: + - path: /etc/pve-maas/conf.d/50-pve.conf + permissions: "0600" + owner: root:root + content: | + PVE_ROOT_PASSWORD_HASH='$6$DEGISTIRIN$REPLACE.WITH.YOUR.OWN.HASH' + + PVE_CLUSTER_MODE=join + + # Kumede zaten bulunan bir dugumun IP'si + PVE_CLUSTER_PEER=192.0.2.11 + + # O dugumun root@pam parolasi (duz metin olmak zorunda - API dogrulamasi icin) + PVE_CLUSTER_PEER_PASSWORD='degistirin' + + # Onerilen: parmak izini onceden verin (TOFU riskini kaldirir) + # openssl s_client -connect 192.0.2.11:8006 /dev/null | \ + # openssl x509 -noout -fingerprint -sha256 | cut -d= -f2 + # PVE_CLUSTER_FINGERPRINT='AA:BB:CC:...' + PVE_CLUSTER_FINGERPRINT_DISCOVER=true + + # PVE_CLUSTER_LINK0=198.51.100.12 + + # Katilimdan sonra bu dosyadaki parolalar silinir + PVE_CLUSTER_WIPE_SECRETS=true + + PVE_NET_APPLY=reboot + PVE_THINPOOL=auto diff --git a/maas/examples/03-standalone.yaml b/maas/examples/03-standalone.yaml new file mode 100644 index 0000000..fabf5fd --- /dev/null +++ b/maas/examples/03-standalone.yaml @@ -0,0 +1,17 @@ +#cloud-config +# Tek basina (kumesiz) Proxmox VE dugumu. + +write_files: + - path: /etc/pve-maas/conf.d/50-pve.conf + permissions: "0600" + owner: root:root + content: | + PVE_ROOT_PASSWORD_HASH='$6$DEGISTIRIN$REPLACE.WITH.YOUR.OWN.HASH' + PVE_CLUSTER_MODE=none + + # VLAN farkindalikli kopru isteniyorsa: + # PVE_NET_VLAN_AWARE=true + + # Ikinci diski VM depolamasi olarak kullan: + # PVE_THINPOOL_DISK=/dev/sdb + # PVE_THINPOOL_VG=pve diff --git a/maas/examples/04-advanced-network.yaml b/maas/examples/04-advanced-network.yaml new file mode 100644 index 0000000..25062af --- /dev/null +++ b/maas/examples/04-advanced-network.yaml @@ -0,0 +1,49 @@ +#cloud-config +# Karmasik ag: bond + VLAN. Otomatik vmbr0 donusumunu kapatip +# /etc/network/interfaces dosyasini kendimiz yaziyoruz. + +write_files: + - path: /etc/pve-maas/conf.d/50-pve.conf + permissions: "0600" + owner: root:root + content: | + PVE_ROOT_PASSWORD_HASH='$6$DEGISTIRIN$REPLACE.WITH.YOUR.OWN.HASH' + PVE_NET_MANAGE=false + PVE_CLUSTER_MODE=none + + - path: /etc/network/interfaces + permissions: "0644" + owner: root:root + content: | + auto lo + iface lo inet loopback + + iface eno1 inet manual + iface eno2 inet manual + + auto bond0 + iface bond0 inet manual + bond-slaves eno1 eno2 + bond-mode 802.3ad + bond-xmit-hash-policy layer3+4 + bond-miimon 100 + + auto vmbr0 + iface vmbr0 inet static + address 192.0.2.20/24 + gateway 192.0.2.1 + bridge-ports bond0 + bridge-stp off + bridge-fd 0 + bridge-vlan-aware yes + bridge-vids 2-4094 + + source /etc/network/interfaces.d/* + + - path: /etc/cloud/cloud.cfg.d/99-pve-maas-disable-network.cfg + permissions: "0644" + content: | + network: {config: disabled} + +runcmd: + - [ systemctl, enable, networking.service ] diff --git a/overlay/curtin/curtin-hooks b/overlay/curtin/curtin-hooks new file mode 100755 index 0000000..5b744f7 --- /dev/null +++ b/overlay/curtin/curtin-hooks @@ -0,0 +1,114 @@ +#!/usr/bin/env python3 +# +# curtin-hooks - MAAS/curtin kurulum kancasi (Proxmox VE imaji icin) +# +# Curtin, hedef sistemde /curtin/curtin-hooks varsa dahili curthooks yerine +# bu script'i calistirir (curtin.util.run_hook_if_exists). +# +# Amac: curtin'in APT uzerinden cekirdek kurmaya calismasini engellemek. +# Proxmox cekirdegi (proxmox-default-kernel) imajin icinde hazir gelir; +# curtin'in onu yeniden kurmasi gereksizdir ve dugumun deploy sirasinda +# download.proxmox.com'a erisebilmesini zorunlu kilar (izole aglarda calismaz). +# +# Not: Bunu preseed'de "kernel: null" ile yapmak mumkun degil - MAAS ile +# gelen curtin surumlerinin bir kismi bu degeri desteklemiyor ve +# install_kernel icinde AttributeError ile patliyor. Kancayi kullanmak +# curtin surumunden bagimsiz olarak calisir. +# +import shutil +import os + +from curtin.commands import curthooks +from curtin.config import load_command_config +from curtin.util import load_command_environment + + +def disable_kernel_install(): + """curtin'in cekirdek kurulum adimini etkisiz hale getir. + + Fonksiyon adi curtin surumune gore degisiyor (install_kernel -> + curthook_install_kernel), ikisini de yakaliyoruz. + """ + disabled = [] + for name in ("install_kernel", "curthook_install_kernel"): + if hasattr(curthooks, name): + setattr(curthooks, name, lambda *args, **kwargs: None) + disabled.append(name) + print("curtin-hooks: cekirdek kurulumu devre disi: %s" % (disabled or "hicbiri bulunamadi")) + + +def pin_interface_names(config, target): + """MAAS'in bildigi arayuz adlarini hedef sistemde MAC uzerinden sabitle. + + MAAS commissioning'i Ubuntu ephemeral ortaminda yapiyor ve arayuzu orada + gordugu adla (or. enp6s18) kaydediyor. Deploy edilen Debian 13 / Proxmox + ise udev'in farkli isimlendirme semasi yuzunden ayni karti ens18 diye + adlandirabiliyor. Bu durumda cloud-init acilista arayuzu yeniden + adlandirmaya calisiyor, su hatayi aliyor: + + Failed to rename devices: [busy] Error renaming + mac=... from ens18 to enp6s18 + + ...ve arayuzu KAPALI birakiyor; dugum agini tamamen kaybediyor. + + Cozum: MAC -> ad eslemesini systemd .link dosyasi olarak yaziyoruz, boylece + udev karti en bastan MAAS'in bekledigi adla olusturuyor ve yeniden + adlandirmaya hic gerek kalmiyor. + """ + net = config.get("network") or {} + pairs = [] + + # netplan / v2 bicimi + for name, cfg in (net.get("ethernets") or {}).items(): + cfg = cfg or {} + mac = (cfg.get("match") or {}).get("macaddress") + if mac: + pairs.append((cfg.get("set-name") or name, mac)) + + # curtin v1 bicimi + for entry in net.get("config") or []: + if not isinstance(entry, dict) or entry.get("type") != "physical": + continue + if entry.get("name") and entry.get("mac_address"): + pairs.append((entry["name"], entry["mac_address"])) + + if not pairs: + print("curtin-hooks: arayuz adi sabitlenmedi (ag yapilandirmasi bos)") + return + + link_dir = os.path.join(target, "etc/systemd/network") + os.makedirs(link_dir, exist_ok=True) + for name, mac in pairs: + path = os.path.join(link_dir, "10-maas-%s.link" % name) + with open(path, "w") as fh: + fh.write( + "# MAAS tarafindan beklenen arayuz adi - pve-maas imaji\n" + "# curtin-hooks tarafindan olusturuldu.\n" + "[Match]\n" + "MACAddress=%s\n" + "\n" + "[Link]\n" + "Name=%s\n" % (mac.lower(), name) + ) + print("curtin-hooks: arayuz adi sabitlendi %s -> %s" % (mac, name)) + + +def cleanup(): + """Kancayi hedef sistemden kaldir - iz birakma.""" + curtin_dir = os.path.dirname(os.path.abspath(__file__)) + shutil.rmtree(curtin_dir, ignore_errors=True) + + +def main(): + state = load_command_environment() + config = load_command_config(None, state) + target = state["target"] + + disable_kernel_install() + curthooks.builtin_curthooks(config, target, state) + pin_interface_names(config, target) + cleanup() + + +if __name__ == "__main__": + main() diff --git a/overlay/etc/pve-maas/conf.d/.keep b/overlay/etc/pve-maas/conf.d/.keep new file mode 100644 index 0000000..e69de29 diff --git a/overlay/etc/pve-maas/pve-maas.conf b/overlay/etc/pve-maas/pve-maas.conf new file mode 100644 index 0000000..e39e465 --- /dev/null +++ b/overlay/etc/pve-maas/pve-maas.conf @@ -0,0 +1,98 @@ +# /etc/pve-maas/pve-maas.conf +# +# pve-maas-init varsayilan yapilandirmasi. BU DOSYAYI DUZENLEMEYIN. +# Dugume ozel ayarlari MAAS deploy sirasinda cloud-init ile +# /etc/pve-maas/conf.d/*.conf altina yazin; oradaki degerler bunlari ezer. +# +# Ornek (cloud-init user-data): +# write_files: +# - path: /etc/pve-maas/conf.d/50-cluster.conf +# permissions: "0600" +# content: | +# PVE_ROOT_PASSWORD='...' +# PVE_CLUSTER_MODE=join +# PVE_CLUSTER_PEER=192.0.2.11 +# PVE_CLUSTER_PEER_PASSWORD='...' + +# ---------------------------------------------------------------- genel +# Tum ilk acilis otomasyonunu kapatmak icin false yapin. +PVE_ENABLED=true + +# Bos birakilirsa hostname -f / resolv.conf'tan turetilir. +PVE_FQDN= + +# ---------------------------------------------------------------- ag +# MAAS'in yapilandirdigi arayuz vmbr0 koprusune donusturulsun mu? +# Bond/VLAN gibi karmasik topolojilerde false yapip /etc/network/interfaces +# dosyasini cloud-init ile kendiniz yazin. +PVE_NET_MANAGE=true + +PVE_NET_BRIDGE=vmbr0 + +# Bos ise varsayilan rotayi tasiyan arayuz kullanilir. +PVE_NET_UPLINK= + +# auto | static | dhcp (auto: MAAS'in yazdigi yapilandirmadan tespit edilir) +PVE_NET_MODE=auto + +# reboot | reload | none +# reboot : yapilandirmayi yazip bir kez yeniden baslatir (en guvenilir) +# reload : ifreload -a ile canli uygular (yeniden baslatma yok) +# none : sadece yazar, uygulamaz +PVE_NET_APPLY=reboot + +# Koprude VLAN farkindaligi (bridge-vlan-aware yes, vids 2-4094) +PVE_NET_VLAN_AWARE=false + +# /etc/network/interfaces sonuna eklenecek ham metin (ek kopruler vb.) +PVE_NET_EXTRA= + +# ---------------------------------------------------------------- root parolasi +# root@pam web arayuzu girisi ve baska dugumlerin bu dugume katilmasi icin gerekli. +# Duz metin yerine hash tercih edin: openssl passwd -6 +PVE_ROOT_PASSWORD= +PVE_ROOT_PASSWORD_HASH= + +# ---------------------------------------------------------------- kume +# none | create | join +PVE_CLUSTER_MODE=none + +# create icin +PVE_CLUSTER_NAME= + +# join icin: mevcut bir kume dugumunun IP/hostname'i ve root@pam parolasi +PVE_CLUSTER_PEER= +PVE_CLUSTER_PEER_PASSWORD= +PVE_CLUSTER_PEER_PASSWORD_FILE= + +# Sertifika parmak izi. Bos ve DISCOVER=true ise dugumden okunur (TOFU). +# Guvenlik icin degeri onceden vermeniz onerilir: +# openssl s_client -connect NODE:8006 /dev/null | \ +# openssl x509 -noout -fingerprint -sha256 | cut -d= -f2 +PVE_CLUSTER_FINGERPRINT= +PVE_CLUSTER_FINGERPRINT_DISCOVER=true + +# Corosync baglantilari (ayri bir kume agi kullaniyorsaniz) +PVE_CLUSTER_LINK0= +PVE_CLUSTER_LINK1= +PVE_CLUSTER_NODEID= +PVE_CLUSTER_VOTES= + +# Kume dugumunun erisilebilir olmasi icin beklenecek sure (saniye) ve deneme sayisi +PVE_CLUSTER_WAIT=900 +PVE_CLUSTER_RETRIES=5 + +# Katilimdan sonra conf.d icindeki parolalari sil +PVE_CLUSTER_WIPE_SECRETS=true + +# ---------------------------------------------------------------- depolama +# auto | off | +# auto: en cok bos alani olan VG secilir +PVE_THINPOOL=auto +PVE_THINPOOL_NAME=data +PVE_THINPOOL_STORAGE=local-lvm +PVE_THINPOOL_MIN_GB=16 + +# Ayri bir disk verilirse o diskten yeni bir VG olusturulur. +PVE_THINPOOL_DISK= +PVE_THINPOOL_VG=pve diff --git a/overlay/etc/systemd/system/pve-maas-init.service b/overlay/etc/systemd/system/pve-maas-init.service new file mode 100644 index 0000000..9786600 --- /dev/null +++ b/overlay/etc/systemd/system/pve-maas-init.service @@ -0,0 +1,44 @@ +[Unit] +Description=Proxmox VE post-deployment configuration for MAAS +Documentation=file:/etc/pve-maas/pve-maas.conf +After=network-online.target +Wants=network-online.target +ConditionPathExists=!/var/lib/pve-maas/complete + +# DIKKAT - burada cloud-init servislerine "After=" YAZMAYIN. +# +# After=cloud-final.service + WantedBy=multi-user.target birlesimi bu imajda +# siralama dongusu olusturuyor ve systemd dongoyu kirmak icin BU servisin +# baslatma isini siliyor: +# +# multi-user.target: Found ordering cycle on pve-maas-init.service/start +# multi-user.target: Found dependency on cloud-final.service/start +# multi-user.target: Found dependency on multi-user.target/start +# Job pve-maas-init.service/start deleted to break ordering cycle +# +# Bunun yerine script kendi icinde "cloud-init status --wait" ile bekliyor; +# boylece hem conf.d dosyalari yazilmis oluyor hem de MAAS deploy sinyalini +# almis oluyor, systemd bagimlilik grafigine dokunmadan. + +[Service] +# Type=simple (oneshot DEGIL) - bu onemli. +# +# Type=oneshot ile servis bitene kadar multi-user.target ulasilmis sayilmiyor. +# cloud-final.service ise "After=multi-user.target" ile siralanmis. Script +# icinde cloud-init'i bekledigimiz icin bu bir kilitlenme uretiyordu: +# +# cloud-final.service waiting (multi-user.target bekliyor) +# pve-maas-init.service running (cloud-init bekliyor) +# multi-user.target waiting (pve-maas-init bekliyor) +# +# Type=simple ile servis hemen "basladi" sayilir, multi-user.target acilir, +# cloud-final calisir ve beklememiz doner. Asama takibi zaten +# /var/lib/pve-maas/*.done dosyalariyla yapiliyor. +Type=simple +ExecStart=/usr/local/sbin/pve-maas-init +TimeoutStartSec=0 +StandardOutput=journal+console +StandardError=journal+console + +[Install] +WantedBy=multi-user.target diff --git a/overlay/usr/local/sbin/pve-maas-init b/overlay/usr/local/sbin/pve-maas-init new file mode 100755 index 0000000..8b56ceb --- /dev/null +++ b/overlay/usr/local/sbin/pve-maas-init @@ -0,0 +1,636 @@ +#!/bin/bash +# +# pve-maas-init - MAAS ile deploy edilen Proxmox VE dugumunu ilk acilista yapilandirir. +# +# Asamalar (her biri /var/lib/pve-maas/.done ile bir kez calisir): +# hosts - hostname/FQDN'i yonetim IP'sine bagla (pvecm bunu ister) +# identity - dugume ozel kimlikleri yeniden uret (iSCSI IQN vb.) +# rootpw - root@pam parolasini ayarla (web arayuzu ve kume katilimi icin) +# network - MAAS'in verdigi arayuzu vmbr0 koprusune donustur +# cluster - kume olustur veya mevcut kumeye katil +# storage - bos alanda LVM-thin havuzu olustur (local-lvm) +# +# Yapilandirma: /etc/pve-maas/pve-maas.conf ve /etc/pve-maas/conf.d/*.conf +# (conf.d icerigi MAAS deploy sirasinda cloud-init write_files ile yazilir) +# +set -uo pipefail + +CONF_DIR=/etc/pve-maas +STATE_DIR=/var/lib/pve-maas +LOG_TAG=pve-maas-init + +mkdir -p "$STATE_DIR" "$CONF_DIR/conf.d" + +log() { echo "[$LOG_TAG] $*"; logger -t "$LOG_TAG" -- "$*" 2>/dev/null || true; } +warn() { echo "[$LOG_TAG] UYARI: $*" >&2; logger -t "$LOG_TAG" -p user.warning -- "UYARI: $*" 2>/dev/null || true; } +die() { echo "[$LOG_TAG] HATA: $*" >&2; logger -t "$LOG_TAG" -p user.err -- "HATA: $*" 2>/dev/null || true; exit 1; } + +done_flag() { echo "$STATE_DIR/$1.done"; } +is_done() { [ -e "$(done_flag "$1")" ]; } +mark_done() { date -Is > "$(done_flag "$1")"; } + +# --------------------------------------------------------------------------- +# Yapilandirmayi yukle +# --------------------------------------------------------------------------- +load_config() { + # shellcheck disable=SC1091 + [ -r "$CONF_DIR/pve-maas.conf" ] && . "$CONF_DIR/pve-maas.conf" + local f + for f in "$CONF_DIR"/conf.d/*.conf; do + [ -r "$f" ] || continue + log "yapilandirma yukleniyor: $f" + # shellcheck disable=SC1090 + . "$f" + done + + # Varsayilanlar + PVE_ENABLED="${PVE_ENABLED:-true}" + PVE_FQDN="${PVE_FQDN:-}" + + PVE_NET_MANAGE="${PVE_NET_MANAGE:-true}" + PVE_NET_BRIDGE="${PVE_NET_BRIDGE:-vmbr0}" + PVE_NET_UPLINK="${PVE_NET_UPLINK:-}" + PVE_NET_MODE="${PVE_NET_MODE:-auto}" # auto | static | dhcp + PVE_NET_APPLY="${PVE_NET_APPLY:-reboot}" # reboot | reload | none + PVE_NET_VLAN_AWARE="${PVE_NET_VLAN_AWARE:-false}" + PVE_NET_EXTRA="${PVE_NET_EXTRA:-}" + + PVE_ROOT_PASSWORD="${PVE_ROOT_PASSWORD:-}" + PVE_ROOT_PASSWORD_HASH="${PVE_ROOT_PASSWORD_HASH:-}" + + PVE_CLUSTER_MODE="${PVE_CLUSTER_MODE:-none}" # none | create | join + PVE_CLUSTER_NAME="${PVE_CLUSTER_NAME:-}" + PVE_CLUSTER_PEER="${PVE_CLUSTER_PEER:-}" + PVE_CLUSTER_PEER_PASSWORD="${PVE_CLUSTER_PEER_PASSWORD:-}" + PVE_CLUSTER_PEER_PASSWORD_FILE="${PVE_CLUSTER_PEER_PASSWORD_FILE:-}" + PVE_CLUSTER_FINGERPRINT="${PVE_CLUSTER_FINGERPRINT:-}" + PVE_CLUSTER_FINGERPRINT_DISCOVER="${PVE_CLUSTER_FINGERPRINT_DISCOVER:-true}" + PVE_CLUSTER_LINK0="${PVE_CLUSTER_LINK0:-}" + PVE_CLUSTER_LINK1="${PVE_CLUSTER_LINK1:-}" + PVE_CLUSTER_NODEID="${PVE_CLUSTER_NODEID:-}" + PVE_CLUSTER_VOTES="${PVE_CLUSTER_VOTES:-}" + PVE_CLUSTER_WAIT="${PVE_CLUSTER_WAIT:-900}" + PVE_CLUSTER_RETRIES="${PVE_CLUSTER_RETRIES:-5}" + PVE_CLUSTER_WIPE_SECRETS="${PVE_CLUSTER_WIPE_SECRETS:-true}" + + PVE_THINPOOL="${PVE_THINPOOL:-auto}" # auto | off | + PVE_THINPOOL_NAME="${PVE_THINPOOL_NAME:-data}" + PVE_THINPOOL_STORAGE="${PVE_THINPOOL_STORAGE:-local-lvm}" + PVE_THINPOOL_MIN_GB="${PVE_THINPOOL_MIN_GB:-16}" + PVE_THINPOOL_DISK="${PVE_THINPOOL_DISK:-}" + PVE_THINPOOL_VG="${PVE_THINPOOL_VG:-pve}" +} + +# --------------------------------------------------------------------------- +# Yardimcilar +# --------------------------------------------------------------------------- +primary_iface() { + if [ -n "$PVE_NET_UPLINK" ]; then echo "$PVE_NET_UPLINK"; return; fi + ip -4 -o route show default 2>/dev/null | awk '{for(i=1;i<=NF;i++) if($i=="dev") {print $(i+1); exit}}' +} + +primary_ip() { + ip -4 -o route get 1.1.1.1 2>/dev/null \ + | awk '{for(i=1;i<=NF;i++) if($i=="src") {print $(i+1); exit}}' +} + +node_fqdn() { + local host fqdn dom + host="$(hostname -s)" + if [ -n "$PVE_FQDN" ]; then echo "$PVE_FQDN"; return; fi + fqdn="$(hostname -f 2>/dev/null || true)" + if [ -n "$fqdn" ] && [ "$fqdn" != "$host" ]; then echo "$fqdn"; return; fi + dom="$(awk '/^(search|domain)[[:space:]]/{print $2; exit}' /etc/resolv.conf 2>/dev/null || true)" + if [ -n "$dom" ]; then echo "${host}.${dom}"; else echo "$host"; fi +} + +wait_for_cloud_init() { + command -v cloud-init >/dev/null 2>&1 || return 0 + log "cloud-init'in bitmesi bekleniyor" + # Unit'te After=cloud-final.service kullanamiyoruz (siralama dongusu), + # bu yuzden beklemeyi burada yapiyoruz. Hata donmesi normal olabilir. + timeout 900 cloud-init status --wait >/dev/null 2>&1 + log "cloud-init durumu: $(cloud-init status 2>/dev/null | head -1)" + return 0 +} + +wait_for_network() { + local i ip + for ((i = 0; i < 120; i += 5)); do + ip="$(primary_ip)" + [ -n "$ip" ] && { log "yonetim IP'si: ${ip}"; return 0; } + sleep 5 + done + return 1 +} + +wait_for_pmxcfs() { + local i + for i in $(seq 1 60); do + [ -e /etc/pve/local ] && return 0 + sleep 2 + done + return 1 +} + +# --------------------------------------------------------------------------- +# Asama: hosts +# --------------------------------------------------------------------------- +stage_hosts() { + is_done hosts && return 0 + local host fqdn ip + host="$(hostname -s)" + fqdn="$(node_fqdn)" + ip="$(primary_ip)" + [ -n "$ip" ] || { warn "yonetim IP'si bulunamadi, /etc/hosts atlaniyor"; return 0; } + + log "/etc/hosts guncelleniyor: ${ip} ${fqdn} ${host}" + cp -a /etc/hosts "$STATE_DIR/hosts.orig" 2>/dev/null || true + + local keep + keep="$(grep -vE "^[[:space:]]*(127\.0\.0\.1|127\.0\.1\.1|::1|ff02::[12])[[:space:]]" /etc/hosts 2>/dev/null \ + | grep -vE "[[:space:]]${host}([[:space:]]|\$)" || true)" + + { + echo "127.0.0.1 localhost.localdomain localhost" + echo "${ip} ${fqdn} ${host}" + [ -n "$keep" ] && echo "$keep" + echo + echo "# IPv6" + echo "::1 localhost ip6-localhost ip6-loopback" + echo "ff02::1 ip6-allnodes" + echo "ff02::2 ip6-allrouters" + } > /etc/hosts + + if command -v postconf >/dev/null 2>&1 && [ -f /etc/postfix/main.cf ]; then + postconf -e "myhostname = ${fqdn}" >/dev/null 2>&1 || true + systemctl try-restart postfix >/dev/null 2>&1 || true + fi + + # cloud-init manage_etc_hosts=true ise her acilista /etc/hosts'u + # 127.0.1.1 satiriyla geri yazar ve duzeltmemizi bozar. + cat > /etc/cloud/cloud.cfg.d/99-pve-maas-hosts.cfg <<'EOF' +# /etc/hosts artik pve-maas-init tarafindan yonetiliyor. +manage_etc_hosts: false +EOF + + hostname -f >/dev/null 2>&1 || warn "hostname -f hala cozulmuyor" + + # pmxcfs dugum kimligini ve sertifikalari dogru hostname/IP ile yenile. + if systemctl list-unit-files pve-cluster.service >/dev/null 2>&1; then + log "pve servisleri yeniden baslatiliyor (dogru hostname ile)" + systemctl restart pve-cluster >/dev/null 2>&1 || true + systemctl restart pvedaemon pveproxy pvestatd >/dev/null 2>&1 || true + fi + + mark_done hosts +} + +# --------------------------------------------------------------------------- +# Asama: identity - dugume ozel kimlikler +# --------------------------------------------------------------------------- +stage_identity() { + is_done identity && return 0 + if [ ! -s /etc/iscsi/initiatorname.iscsi ] && command -v iscsi-iname >/dev/null 2>&1; then + log "iSCSI initiator adi uretiliyor" + mkdir -p /etc/iscsi + echo "InitiatorName=$(iscsi-iname)" > /etc/iscsi/initiatorname.iscsi + chmod 0600 /etc/iscsi/initiatorname.iscsi + systemctl try-restart iscsid open-iscsi >/dev/null 2>&1 || true + fi + mark_done identity +} + +# --------------------------------------------------------------------------- +# Asama: rootpw +# --------------------------------------------------------------------------- +stage_rootpw() { + is_done rootpw && return 0 + if [ -n "$PVE_ROOT_PASSWORD_HASH" ]; then + log "root parolasi (hash) ayarlaniyor" + usermod -p "$PVE_ROOT_PASSWORD_HASH" root + elif [ -n "$PVE_ROOT_PASSWORD" ]; then + log "root parolasi ayarlaniyor" + echo "root:${PVE_ROOT_PASSWORD}" | chpasswd + else + warn "PVE_ROOT_PASSWORD tanimli degil; root@pam ile web arayuzune girilemez" + warn "ve bu dugum baska bir dugumun kumeye katilmasi icin hedef olamaz." + mark_done rootpw + return 0 + fi + mark_done rootpw +} + +# --------------------------------------------------------------------------- +# Asama: network - MAAS arayuzunu vmbr0'a cevir +# --------------------------------------------------------------------------- +detect_net_mode() { + [ "$PVE_NET_MODE" != "auto" ] && { echo "$PVE_NET_MODE"; return; } + if grep -rqsE '^\s*(dhcp4|dhcp6)\s*:\s*(true|yes)' /etc/netplan/ 2>/dev/null; then + echo dhcp; return + fi + if grep -rqsE '^\s*iface\s+\S+\s+inet6?\s+dhcp' /etc/network/interfaces.d/ 2>/dev/null; then + echo dhcp; return + fi + echo static +} + +neutralize_foreign_netconf() { + log "MAAS/cloud-init ag yapilandirmasi devre disi birakiliyor" + printf 'network: {config: disabled}\n' > /etc/cloud/cloud.cfg.d/99-pve-maas-disable-network.cfg + + local d + d=/etc/network/interfaces.d/disabled-by-pve-maas + mkdir -p "$d" + find /etc/network/interfaces.d -maxdepth 1 -type f -print0 2>/dev/null \ + | xargs -0 -r -I{} mv {} "$d/" + + if compgen -G "/etc/netplan/*.yaml" >/dev/null || compgen -G "/etc/netplan/*.yml" >/dev/null; then + mkdir -p /etc/netplan/disabled-by-pve-maas + mv /etc/netplan/*.y*ml /etc/netplan/disabled-by-pve-maas/ 2>/dev/null || true + fi + + if systemctl is-enabled systemd-networkd >/dev/null 2>&1; then + systemctl disable --now systemd-networkd systemd-networkd.socket >/dev/null 2>&1 || true + fi + + # Proxmox statik /etc/resolv.conf bekler. + if [ -L /etc/resolv.conf ]; then + local ns search + ns="$(resolvectl dns 2>/dev/null | tr ' ' '\n' | grep -E '^[0-9a-fA-F:.]+$' | sort -u || true)" + [ -z "$ns" ] && ns="$(awk '/^nameserver/{print $2}' /etc/resolv.conf | sort -u || true)" + search="$(awk '/^(search|domain)[[:space:]]/{$1=""; print substr($0,2); exit}' /etc/resolv.conf || true)" + if [ -n "$ns" ]; then + rm -f /etc/resolv.conf + { + [ -n "$search" ] && echo "search ${search}" + echo "$ns" | while read -r s; do [ -n "$s" ] && echo "nameserver $s"; done + } > /etc/resolv.conf + systemctl disable --now systemd-resolved >/dev/null 2>&1 || true + else + warn "nameserver bulunamadi, /etc/resolv.conf oldugu gibi birakiliyor" + fi + fi + + systemctl enable networking.service >/dev/null 2>&1 || true +} + +stage_network() { + is_done network && return 0 + [ "$PVE_NET_MANAGE" = "true" ] || { log "ag yonetimi kapali (PVE_NET_MANAGE=false)"; mark_done network; return 0; } + + local br uplink + br="$PVE_NET_BRIDGE" + uplink="$(primary_iface)" + + if ip link show "$br" >/dev/null 2>&1 && [ "$uplink" = "$br" ]; then + log "$br zaten mevcut ve varsayilan rotayi tasiyor, ag donusumu atlaniyor" + mark_done network + return 0 + fi + [ -n "$uplink" ] || { warn "uplink arayuzu bulunamadi, ag donusumu atlaniyor"; mark_done network; return 0; } + + local mode cidr gw cidr6 gw6 + mode="$(detect_net_mode)" + cidr="$(ip -4 -o addr show dev "$uplink" scope global 2>/dev/null | awk '{print $4; exit}')" + gw="$(ip -4 -o route show default dev "$uplink" 2>/dev/null | awk '{for(i=1;i<=NF;i++) if($i=="via") {print $(i+1); exit}}')" + cidr6="$(ip -6 -o addr show dev "$uplink" scope global -deprecated 2>/dev/null | awk '{print $4; exit}')" + gw6="$(ip -6 -o route show default dev "$uplink" 2>/dev/null | awk '{for(i=1;i<=NF;i++) if($i=="via") {print $(i+1); exit}}')" + + if [ "$mode" = "static" ] && [ -z "$cidr" ]; then + warn "$uplink uzerinde IPv4 adresi yok, DHCP'ye dusuluyor" + mode=dhcp + fi + + log "ag donusumu: ${uplink} -> ${br} (mod=${mode} adres=${cidr:-dhcp} gw=${gw:-yok})" + cp -a /etc/network/interfaces "$STATE_DIR/interfaces.orig" 2>/dev/null || true + neutralize_foreign_netconf + + { + echo "# /etc/network/interfaces - pve-maas-init tarafindan olusturuldu" + echo "# Kaynak yedegi: $STATE_DIR/interfaces.orig" + echo + echo "auto lo" + echo "iface lo inet loopback" + echo + echo "iface ${uplink} inet manual" + echo + echo "auto ${br}" + if [ "$mode" = "dhcp" ]; then + echo "iface ${br} inet dhcp" + else + echo "iface ${br} inet static" + echo " address ${cidr}" + [ -n "$gw" ] && echo " gateway ${gw}" + fi + echo " bridge-ports ${uplink}" + echo " bridge-stp off" + echo " bridge-fd 0" + [ "$PVE_NET_VLAN_AWARE" = "true" ] && { + echo " bridge-vlan-aware yes" + echo " bridge-vids 2-4094" + } + if [ -n "$cidr6" ]; then + echo + echo "iface ${br} inet6 static" + echo " address ${cidr6}" + [ -n "$gw6" ] && echo " gateway ${gw6}" + fi + if [ -n "$PVE_NET_EXTRA" ]; then + echo + echo "# PVE_NET_EXTRA" + printf '%s\n' "$PVE_NET_EXTRA" + fi + echo + echo "source /etc/network/interfaces.d/*" + } > /etc/network/interfaces + + # Proxmox'un pvenetcommit.service'i acilista /etc/network/interfaces.new + # dosyasini interfaces uzerine tasiyor. Kalmis bir .new dosyasi yeni + # yazdigimiz yapilandirmayi yeniden baslatmada ezer. + rm -f /etc/network/interfaces.new + + mark_done network + + case "$PVE_NET_APPLY" in + reload) + log "ifreload -a ile uygulaniyor" + ip -4 addr flush dev "$uplink" 2>/dev/null || true + ip -6 addr flush dev "$uplink" scope global 2>/dev/null || true + if ! ifreload -a; then + warn "ifreload basarisiz, yedek rota ekleniyor" + [ -n "$gw" ] && ip route replace default via "$gw" dev "$br" 2>/dev/null || true + fi + ;; + reboot) + log "ag yapilandirmasi icin yeniden baslatiliyor; kalan asamalar sonraki acilista surecek" + # Bu servis hala calisirken 'systemctl reboot' cagirmak systemd islem + # kuyrugunda "transaction is destructive" hatasina yol acabilir; bu + # yuzden reboot'u ayri bir gecici unit uzerinden tetikliyoruz. + systemd-run --no-block --unit=pve-maas-reboot --on-active=5 \ + /bin/systemctl reboot \ + || shutdown -r +1 "pve-maas-init: ag yapilandirmasi uygulanacak" + exit 0 + ;; + none) + log "PVE_NET_APPLY=none: yapilandirma yazildi, uygulanmadi" + ;; + esac +} + +# --------------------------------------------------------------------------- +# Asama: cluster +# --------------------------------------------------------------------------- +peer_fingerprint() { + local peer="$1" + openssl s_client -connect "${peer}:8006" -servername "${peer}" /dev/null \ + | openssl x509 -noout -fingerprint -sha256 2>/dev/null \ + | cut -d= -f2 +} + +wait_for_peer() { + local peer="$1" timeout="$2" i + log "kume dugumu bekleniyor: ${peer}:8006 (en fazla ${timeout}s)" + for ((i = 0; i < timeout; i += 5)); do + if timeout 4 bash -c "exec 3<>/dev/tcp/${peer}/8006" 2>/dev/null; then + log "${peer}:8006 erisilebilir" + return 0 + fi + sleep 5 + done + return 1 +} + +cluster_password() { + if [ -n "$PVE_CLUSTER_PEER_PASSWORD_FILE" ] && [ -r "$PVE_CLUSTER_PEER_PASSWORD_FILE" ]; then + head -1 "$PVE_CLUSTER_PEER_PASSWORD_FILE" + else + printf '%s' "$PVE_CLUSTER_PEER_PASSWORD" + fi +} + +wipe_cluster_secrets() { + [ "$PVE_CLUSTER_WIPE_SECRETS" = "true" ] || return 0 + local f + for f in "$CONF_DIR"/conf.d/*.conf; do + [ -r "$f" ] || continue + grep -q 'PVE_CLUSTER_PEER_PASSWORD\|PVE_ROOT_PASSWORD' "$f" || continue + log "kimlik bilgileri temizleniyor: $f" + sed -i -E "s/^([[:space:]]*(PVE_CLUSTER_PEER_PASSWORD|PVE_ROOT_PASSWORD)(_HASH)?=).*/\1''/" "$f" + done + if [ -n "$PVE_CLUSTER_PEER_PASSWORD_FILE" ] && [ -f "$PVE_CLUSTER_PEER_PASSWORD_FILE" ]; then + shred -u "$PVE_CLUSTER_PEER_PASSWORD_FILE" 2>/dev/null \ + || rm -f "$PVE_CLUSTER_PEER_PASSWORD_FILE" + fi +} + +wait_for_quorum() { + local i + for ((i = 0; i < 180; i += 5)); do + if pvecm status 2>/dev/null | grep -qE '^Quorate:[[:space:]]+Yes'; then + return 0 + fi + sleep 5 + done + return 1 +} + +stage_cluster() { + is_done cluster && return 0 + [ "$PVE_CLUSTER_MODE" = "none" ] && { log "kume islemi yok (PVE_CLUSTER_MODE=none)"; mark_done cluster; return 0; } + + wait_for_pmxcfs || { warn "/etc/pve baglanmadi, kume asamasi erteleniyor"; return 1; } + + if [ -f /etc/pve/corosync.conf ]; then + log "dugum zaten bir kumenin parcasi, kume asamasi atlaniyor" + mark_done cluster + return 0 + fi + + local -a opts=() + [ -n "$PVE_CLUSTER_LINK0" ] && opts+=(--link0 "$PVE_CLUSTER_LINK0") + [ -n "$PVE_CLUSTER_LINK1" ] && opts+=(--link1 "$PVE_CLUSTER_LINK1") + [ -n "$PVE_CLUSTER_NODEID" ] && opts+=(--nodeid "$PVE_CLUSTER_NODEID") + [ -n "$PVE_CLUSTER_VOTES" ] && opts+=(--votes "$PVE_CLUSTER_VOTES") + + case "$PVE_CLUSTER_MODE" in + create) + [ -n "$PVE_CLUSTER_NAME" ] || { warn "PVE_CLUSTER_NAME bos, kume olusturulamiyor"; return 1; } + log "kume olusturuluyor: ${PVE_CLUSTER_NAME}" + if pvecm create "$PVE_CLUSTER_NAME" "${opts[@]}"; then + wait_for_quorum && log "kume olusturuldu ve quorate" + mark_done cluster + else + warn "pvecm create basarisiz" + return 1 + fi + ;; + join) + [ -n "$PVE_CLUSTER_PEER" ] || { warn "PVE_CLUSTER_PEER bos, katilim yapilamiyor"; return 1; } + local pw fp + pw="$(cluster_password)" + [ -n "$pw" ] || { warn "kume dugumunun root parolasi verilmedi (PVE_CLUSTER_PEER_PASSWORD)"; return 1; } + + wait_for_peer "$PVE_CLUSTER_PEER" "$PVE_CLUSTER_WAIT" \ + || { warn "kume dugumu ${PVE_CLUSTER_PEER} zaman asimina ugradi"; return 1; } + + fp="$PVE_CLUSTER_FINGERPRINT" + if [ -z "$fp" ] && [ "$PVE_CLUSTER_FINGERPRINT_DISCOVER" = "true" ]; then + fp="$(peer_fingerprint "$PVE_CLUSTER_PEER")" + warn "parmak izi otomatik alindi (TOFU, dogrulanmadi): ${fp}" + fi + [ -n "$fp" ] || { warn "kume dugumunun sertifika parmak izi alinamadi"; return 1; } + + local i rc=1 + for ((i = 1; i <= PVE_CLUSTER_RETRIES; i++)); do + log "kumeye katiliniyor (deneme ${i}/${PVE_CLUSTER_RETRIES}): ${PVE_CLUSTER_PEER}" + # pvecm add etkilesimli parola sorar; API ucu ile etkilesimsiz calisiyoruz. + if pvesh create /cluster/config/join \ + --hostname "$PVE_CLUSTER_PEER" \ + --password "$pw" \ + --fingerprint "$fp" \ + "${opts[@]}"; then + rc=0 + break + fi + warn "katilim basarisiz, 30s sonra tekrar denenecek" + sleep 30 + done + + if [ "$rc" -ne 0 ]; then + warn "kumeye katilim ${PVE_CLUSTER_RETRIES} denemede basarisiz" + return 1 + fi + + systemctl restart pve-cluster pvedaemon pveproxy pvestatd >/dev/null 2>&1 || true + if wait_for_quorum; then + log "kumeye katilim tamamlandi ve quorate" + else + warn "katilim yapildi ama quorum saglanamadi, kume durumunu kontrol edin" + fi + mark_done cluster + wipe_cluster_secrets + ;; + *) + warn "gecersiz PVE_CLUSTER_MODE=${PVE_CLUSTER_MODE}" + mark_done cluster + ;; + esac +} + +# --------------------------------------------------------------------------- +# Asama: storage - LVM-thin havuzu +# --------------------------------------------------------------------------- +pick_vg() { + if [ "$PVE_THINPOOL" != "auto" ] && [ "$PVE_THINPOOL" != "off" ]; then + echo "$PVE_THINPOOL"; return + fi + vgs --noheadings --nosuffix --units b -o vg_name,vg_free 2>/dev/null \ + | awk '{gsub(/^ +/,""); print $2, $1}' | sort -rn | head -1 | awk '{print $2}' +} + +stage_storage() { + is_done storage && return 0 + [ "$PVE_THINPOOL" = "off" ] && { log "thin havuz kapali"; mark_done storage; return 0; } + + wait_for_pmxcfs || { warn "/etc/pve baglanmadi, storage asamasi erteleniyor"; return 1; } + command -v lvs >/dev/null 2>&1 || { warn "lvm2 yok"; mark_done storage; return 0; } + + local vg="$PVE_THINPOOL_VG" + + if [ -n "$PVE_THINPOOL_DISK" ]; then + if [ ! -b "$PVE_THINPOOL_DISK" ]; then + warn "PVE_THINPOOL_DISK=${PVE_THINPOOL_DISK} blok aygiti degil" + mark_done storage; return 0 + fi + if ! pvs "$PVE_THINPOOL_DISK" >/dev/null 2>&1; then + log "PV/VG olusturuluyor: ${PVE_THINPOOL_DISK} -> ${vg}" + pvcreate -ff -y "$PVE_THINPOOL_DISK" || { warn "pvcreate basarisiz"; mark_done storage; return 0; } + vgcreate "$vg" "$PVE_THINPOOL_DISK" || { warn "vgcreate basarisiz"; mark_done storage; return 0; } + fi + else + vg="$(pick_vg)" + fi + + [ -n "$vg" ] || { log "uygun VG bulunamadi, thin havuz atlaniyor"; mark_done storage; return 0; } + + local pool="$PVE_THINPOOL_NAME" + if lvs "${vg}/${pool}" >/dev/null 2>&1; then + log "${vg}/${pool} zaten var" + else + local free_b free_g + free_b="$(vgs --noheadings --nosuffix --units b -o vg_free "$vg" 2>/dev/null | tr -d ' ')" + free_g=$(( ${free_b:-0} / 1024 / 1024 / 1024 )) + if [ "$free_g" -lt "$PVE_THINPOOL_MIN_GB" ]; then + log "${vg} icinde yalnizca ${free_g}GiB bos alan var (gereken ${PVE_THINPOOL_MIN_GB}GiB), thin havuz atlaniyor" + mark_done storage + return 0 + fi + log "thin havuz olusturuluyor: ${vg}/${pool} (~${free_g}GiB)" + if ! lvcreate --type thin-pool -l 95%FREE -n "$pool" "$vg"; then + warn "lvcreate basarisiz" + mark_done storage + return 0 + fi + fi + + local storage="$PVE_THINPOOL_STORAGE" node + node="$(hostname -s)" + if pvesm status --storage "$storage" >/dev/null 2>&1; then + log "storage '${storage}' zaten tanimli" + local nodes + nodes="$(awk -v s="$storage" '$1=="lvmthin:" && $2==s {f=1; next} /^[a-z]+:/{f=0} f && $1=="nodes"{print $2}' /etc/pve/storage.cfg 2>/dev/null)" + if [ -n "$nodes" ] && ! echo ",$nodes," | grep -q ",${node},"; then + log "'${storage}' dugum listesine ${node} ekleniyor" + pvesm set "$storage" --nodes "${nodes},${node}" || warn "pvesm set basarisiz" + fi + else + log "storage tanimlaniyor: ${storage} (lvmthin ${vg}/${pool})" + local -a sopts=(--vgname "$vg" --thinpool "$pool" --content images,rootdir) + [ -f /etc/pve/corosync.conf ] && sopts+=(--nodes "$node") + pvesm add lvmthin "$storage" "${sopts[@]}" || warn "pvesm add basarisiz" + fi + + mark_done storage +} + +# --------------------------------------------------------------------------- +main() { + load_config + + if [ "$PVE_ENABLED" != "true" ]; then + log "PVE_ENABLED=false, hicbir sey yapilmiyor" + exit 0 + fi + + log "baslatiliyor (hostname=$(hostname -s))" + + wait_for_cloud_init + load_config # cloud-init conf.d dosyalarini yeni yazmis olabilir + + if ! wait_for_network; then + warn "yonetim IP'si bulunamadi; sonraki acilista tekrar denenecek" + exit 1 + fi + + local failed=0 + stage_hosts || failed=1 + stage_identity || failed=1 + stage_rootpw || failed=1 + stage_network || failed=1 + stage_cluster || failed=1 + stage_storage || failed=1 + + if [ "$failed" -eq 0 ] \ + && is_done hosts && is_done identity && is_done rootpw \ + && is_done network && is_done cluster && is_done storage; then + date -Is > "$STATE_DIR/complete" + log "tum asamalar tamamlandi" + else + warn "bazi asamalar tamamlanmadi; servis sonraki acilista tekrar denenecek" + warn "ayrintilar: journalctl -u pve-maas-init" + exit 1 + fi +} + +main "$@" diff --git a/scripts/customize-proxmox.sh.in b/scripts/customize-proxmox.sh.in new file mode 100644 index 0000000..a341103 --- /dev/null +++ b/scripts/customize-proxmox.sh.in @@ -0,0 +1,285 @@ +#!/bin/bash +# +# customize-proxmox.sh - packer-maas'in debian sablonu icinde, build VM'inde calisir. +# +# Debian cloud image uzerine Proxmox VE kurar ve MAAS ile deploy edilebilecek +# hale getirir. Bu dosya bir sablondur; Makefile @@...@@ yer tutucularini +# doldurur ve sonuna base64 kodlu overlay arsivini ekler. +# +# NOT: Bu script packer tarafindan "expect_disconnect = true" ile calistirilir. +# +set -euo pipefail + +PVE_SUITE="@@PVE_SUITE@@" +PVE_REPO="@@PVE_REPO@@" +PVE_REPO_URI="@@PVE_REPO_URI@@" +PVE_KEYRING_URL="@@PVE_KEYRING_URL@@" +PVE_VERSION="@@PVE_VERSION@@" +PVE_EXTRA_PACKAGES="@@PVE_EXTRA_PACKAGES@@" +PACKER_MAAS_REF="@@PM_REF@@" + +export DEBIAN_FRONTEND=noninteractive +APT="apt-get -y -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef" + +# eatmydata kurulduktan sonra $APT bunun uzerinden calisir; dpkg'nin her paket +# icin yaptigi fsync'ler devre disi kalir. Imaj derlemede guvenli (VM diski +# zaten atilabilir) ve paket kurulumunu belirgin sekilde hizlandirir. +use_eatmydata() { + command -v eatmydata >/dev/null 2>&1 || return 0 + APT="eatmydata ${APT}" + log "eatmydata etkin (dpkg fsync'leri devre disi)" +} + +log() { echo "==> [pve-image] $*"; } + +# --------------------------------------------------------------------------- +# 1. Gomulu overlay arsivini ac +# --------------------------------------------------------------------------- +log "Overlay dosyalari aciliyor" +sed -n '/^__PVE_MAAS_OVERLAY__$/,$p' "$0" | tail -n +2 | base64 -d \ + | tar xzf - -C / --no-same-owner --no-same-permissions +chown -R root:root /usr/local/sbin/pve-maas-init /etc/pve-maas \ + /etc/systemd/system/pve-maas-init.service /curtin +chmod 0755 /usr/local/sbin/pve-maas-init +chmod 0755 /curtin /curtin/curtin-hooks +chmod 0644 /etc/systemd/system/pve-maas-init.service +chmod 0644 /etc/pve-maas/pve-maas.conf +mkdir -p /etc/pve-maas/conf.d /var/lib/pve-maas + +# --------------------------------------------------------------------------- +# 2. Build sirasinda hostname cozulebilir olmali (pve-cluster bunu ister) +# --------------------------------------------------------------------------- +BUILD_HOST="$(hostname -s)" +BUILD_IP="$(ip -4 -o route get 1.1.1.1 2>/dev/null | awk '{for(i=1;i<=NF;i++) if($i=="src") print $(i+1)}' | head -1)" +: "${BUILD_IP:=127.0.1.1}" +log "Build hostname=${BUILD_HOST} ip=${BUILD_IP}" +cp -a /etc/hosts /etc/hosts.pve-image-backup +sed -i "/[[:space:]]${BUILD_HOST}\([[:space:]]\|$\)/d" /etc/hosts +echo "${BUILD_IP} ${BUILD_HOST}.local ${BUILD_HOST}" >> /etc/hosts +hostname -f || echo "UYARI: hostname -f cozulemedi" >&2 + +# --------------------------------------------------------------------------- +# 3. packer-maas'in kurdugu MAAS uyumlu cloud-init'i koru +# +# packer-maas, Debian'in cloud-init'i MAAS datasource'unu tam desteklemedigi +# icin Ubuntu build'ini kuruyor. full-upgrade bunu geri almasin. +# --------------------------------------------------------------------------- +log "cloud-init paketi hold'a aliniyor" +apt-mark hold cloud-init || true + +# apt-cacher-ng gibi bir HTTP onbellegi verilmisse Debian depolarini https'ten +# http'ye cevir - aksi halde onbellek isabet etmez (CONNECT tunelini cache'leyemez). +# Paket imzalari yine dogrulandigi icin guvenlik kaybi yok. +if [ -n "${http_proxy:-}" ]; then + log "APT onbellegi kullaniliyor (${http_proxy}); Debian depolari http'ye cevriliyor" + sed -i 's|https://deb.debian.org|http://deb.debian.org|g; s|https://security.debian.org|http://security.debian.org|g' \ + /etc/apt/sources.list /etc/apt/sources.list.d/*.sources /etc/apt/sources.list.d/*.list 2>/dev/null || true +fi + +log "eatmydata kuruluyor" +$APT install eatmydata || true +use_eatmydata + +# --------------------------------------------------------------------------- +# 4. Proxmox APT deposu +# --------------------------------------------------------------------------- +log "Proxmox anahtarligi indiriliyor: ${PVE_KEYRING_URL}" +curl -fsSL --retry 5 --retry-delay 3 "${PVE_KEYRING_URL}" \ + -o /usr/share/keyrings/proxmox-archive-keyring.gpg +chmod 0644 /usr/share/keyrings/proxmox-archive-keyring.gpg + +log "APT deposu ekleniyor: ${PVE_REPO_URI} ${PVE_SUITE} ${PVE_REPO}" +cat > /etc/apt/sources.list.d/proxmox.sources </dev/null 2>&1; then + echo "HATA: proxmox-ve paketi bulunamadi. PVE_SUITE=${PVE_SUITE} PVE_REPO=${PVE_REPO} dogru mu?" >&2 + exit 1 +fi + +# --------------------------------------------------------------------------- +# 5. Sistemi guncelle +# --------------------------------------------------------------------------- +log "full-upgrade" +$APT full-upgrade + +# --------------------------------------------------------------------------- +# 6. Etkilesimsiz kurulum icin debconf yanitlari +# --------------------------------------------------------------------------- +debconf-set-selections </dev/null +ln -sf /bin/true /usr/sbin/update-initramfs + +log "proxmox-default-kernel kuruluyor" +$APT install proxmox-default-kernel + +log "proxmox-ve kuruluyor" +$APT install proxmox-ve + +if [ -n "${PVE_EXTRA_PACKAGES}" ]; then + log "Ek paketler kuruluyor: ${PVE_EXTRA_PACKAGES}" + # shellcheck disable=SC2086 + $APT install ${PVE_EXTRA_PACKAGES} +fi + +log "Kurulan surumler:" +dpkg-query -W -f='${Package} ${Version}\n' proxmox-ve pve-manager proxmox-default-kernel ifupdown2 || true + +# --------------------------------------------------------------------------- +# 8. Debian cekirdegini ve os-prober'i kaldir +# --------------------------------------------------------------------------- +log "Debian cekirdegi ve os-prober kaldiriliyor" +$APT purge os-prober || true +# Yalnizca Debian'in kendi cekirdeklerini hedefle; Proxmox paketlerine dokunma. +DEB_KERNELS="$(dpkg-query -W -f='${Package}\n' 'linux-image*' 2>/dev/null \ + | grep -E '^linux-image' | grep -vE 'pve|proxmox' || true)" +if [ -n "${DEB_KERNELS}" ]; then + log "Kaldirilacak: ${DEB_KERNELS}" + # shellcheck disable=SC2086 + $APT purge ${DEB_KERNELS} || true +fi +$APT autoremove --purge || true + +log "update-initramfs geri aliniyor ve bir kez calistiriliyor" +rm -f /usr/sbin/update-initramfs +dpkg-divert --local --rename --remove /usr/sbin/update-initramfs >/dev/null +update-initramfs -u -k all + +update-grub + +# --------------------------------------------------------------------------- +# 9. Abonelik gerektiren depolari devre disi birak (varsa) +# --------------------------------------------------------------------------- +for f in /etc/apt/sources.list.d/*enterprise*; do + [ -e "$f" ] || continue + log "Devre disi birakiliyor: $f" + mv "$f" "$f.disabled" +done + +# --------------------------------------------------------------------------- +# 10. Dugum kimligini sifirla +# +# pmxcfs (/etc/pve) dugum adini config.db icinde saklar. Imaj build hostname'i +# ile olusur; MAAS baska bir hostname ile deploy edecegi icin veritabanini +# tamamen siliyoruz. pve-cluster ilk acilista guncel hostname ile yeni bir +# config.db uretir ve ExecStartPost'taki 'pvecm updatecerts' sertifikalari +# yeniden olusturur. +# --------------------------------------------------------------------------- +log "Proxmox dugum kimligi sifirlaniyor" +systemctl stop pve-guests pve-ha-lrm pve-ha-crm pvescheduler pvestatd pveproxy \ + pvedaemon pve-firewall pvefw-logger spiceproxy corosync pve-cluster 2>/dev/null || true +sleep 2 +umount /etc/pve 2>/dev/null || true +rm -f /var/lib/pve-cluster/config.db /var/lib/pve-cluster/config.db-wal \ + /var/lib/pve-cluster/config.db-shm /var/lib/pve-cluster/.pmxcfs.lockfile +rm -rf /etc/corosync/* /var/lib/corosync/* +rm -rf /var/lib/rrdcached/db/* +rm -f /etc/pve/* 2>/dev/null || true +# iSCSI initiator adi dugume ozeldir; ilk acilista yeniden uretilir. +rm -f /etc/iscsi/initiatorname.iscsi +# Kume icin uretilen SSH bilgileri imaja sizmasin. +rm -f /root/.ssh/known_hosts /etc/ssh/ssh_known_hosts +find /root/.ssh -type l -delete 2>/dev/null || true + +# --------------------------------------------------------------------------- +# 11. Ilk acilis servisini etkinlestir +# --------------------------------------------------------------------------- +log "pve-maas-init.service etkinlestiriliyor" +systemctl daemon-reload +systemctl enable pve-maas-init.service + +# --------------------------------------------------------------------------- +# 12. Temizlik +# --------------------------------------------------------------------------- +# Kalici journald: dugum ilk acilista bir kez yeniden baslayabildigi icin +# (PVE_NET_APPLY=reboot) ucucu journal ile pve-maas-init kayitlari kayboluyor. +# --------------------------------------------------------------------------- +# Ilk acilista agi MAAS'a birak +# +# ifupdown2 kurulumu /etc/network/interfaces dosyasina build VM'inin arayuz +# adini yaziyor (or. "iface ens4 inet manual") ve networking.service etkin +# geliyor. Deploy edilen dugumde MAAS agi netplan/systemd-networkd ile +# yapilandirir; ayni anda ifupdown2 devreye girince gercek arayuzu kapatiyor +# ve dugum daha pve-maas-init calismadan agini kaybediyor. +# +# Cozum: imajda temiz/bos bir interfaces dosyasi birak ve networking.service'i +# devre disi birak. pve-maas-init ag asamasinda vmbr0'i yazip bu servisi +# kendisi etkinlestirir. +# --------------------------------------------------------------------------- +log "networking.service devre disi birakiliyor (ilk acilista ag MAAS'ta)" +cat > /etc/network/interfaces <<'EOF' +# Bu dosya pve-maas-init tarafindan, dugum ilk acilista vmbr0 koprusune +# gecirilirken yeniden yazilir. O ana kadar ag MAAS/cloud-init (netplan) +# tarafindan yonetilir; bu yuzden burada yalnizca loopback tanimlidir. +auto lo +iface lo inet loopback + +source /etc/network/interfaces.d/* +EOF +rm -f /etc/network/interfaces.d/* 2>/dev/null || true +# Proxmox, ag degisikliklerini once /etc/network/interfaces.new dosyasina +# yaziyor; pvenetcommit.service acilista bunu interfaces uzerine TASIYOR. +# Build sirasinda olusan bir .new dosyasi imajda kalirsa dugumun ilk +# acilisinda temiz yapilandirmamizi ezer. +rm -f /etc/network/interfaces.new +systemctl disable networking.service 2>/dev/null || true + +# Imajin kunyesi - dagitilmis bir dugumde "bu hangi imajdan geldi?" sorusunu +# cevaplar; 'make check-upstream' de bunu okur. +log "Imaj kunyesi yaziliyor: /etc/pve-maas/image-info" +{ + echo "build_date=$(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo "pve_version=@@PVE_VERSION@@" + echo "debian_suite=${PVE_SUITE}" + echo "pve_repo=${PVE_REPO}" + echo "packer_maas_ref=${PACKER_MAAS_REF}" + dpkg-query -W -f='${Package}=${Version}\n' proxmox-ve pve-manager proxmox-default-kernel 2>/dev/null + echo "kernel=$(ls -1 /boot/vmlinuz-*-pve 2>/dev/null | sed 's|.*/vmlinuz-||' | head -1)" +} > /etc/pve-maas/image-info +cat /etc/pve-maas/image-info + +log "Kalici journald etkinlestiriliyor" +mkdir -p /var/log/journal +systemd-tmpfiles --create --prefix /var/log/journal 2>/dev/null || true + +log "Temizlik" +mv /etc/hosts.pve-image-backup /etc/hosts +rm -f /etc/postfix/main.cf.proto 2>/dev/null || true +[ -f /etc/postfix/main.cf ] && postconf -e "myhostname = localhost" >/dev/null 2>&1 || true +$APT clean +rm -rf /var/lib/apt/lists/* +find /var/log/journal -mindepth 1 -delete 2>/dev/null || true +rm -rf /var/log/*.gz /var/log/*.1 +: > /var/log/wtmp || true +: > /var/log/btmp || true +cloud-init clean --logs || true + +log "Imaj hazir: Proxmox VE ${PVE_VERSION} / Debian ${PVE_SUITE}" +exit 0 + +# Bu satirdan sonrasi Makefile tarafindan eklenen base64 overlay arsividir. diff --git a/scripts/deploy-cluster.sh b/scripts/deploy-cluster.sh new file mode 100755 index 0000000..b898b36 --- /dev/null +++ b/scripts/deploy-cluster.sh @@ -0,0 +1,283 @@ +#!/bin/bash +# +# deploy-cluster.sh - MAAS uzerinden komple bir Proxmox VE kumesi kurar. +# +# Ilk dugumu 'create' modunda deploy eder, ayaga kalkmasini bekler, sertifika +# parmak izini okur ve kalan dugumleri o parmak iziyle 'join' modunda deploy eder. +# +# NEREDE CALISTIRILIR: MAAS region controller uzerinde (ya da 'maas' CLI profili +# tanimli ve dugumlerin 8006 portuna erisebilen bir makinede). +# +# Ornek: +# ./deploy-cluster.sh --name pve-prod --nodes pve1,pve2,pve3 +# ./deploy-cluster.sh --name pve-dr --nodes dr1,dr2 --profile admin --dry-run +# +set -euo pipefail + +PROFILE=admin +CLUSTER_NAME="" +NODES="" +DISTRO_SERIES=proxmox-ve-9 +ROOT_PASSWORD="" +ROOT_PASSWORD_HASH="" +THINPOOL=auto +THINPOOL_MIN_GB=16 +NET_APPLY=reboot +LINK0_PREFIX="" +DEPLOY_TIMEOUT=2400 +PEER_TIMEOUT=1200 +SERIAL=false +DRY_RUN=false + +usage() { + # Bastaki yorum blogunu, ilk kod satirina kadar bas. + awk 'NR>1 && /^#/{sub(/^# ?/,""); print; next} NR>1 && !/^#/{exit}' "$0" + cat < Kume adi (zorunlu) + --nodes MAAS hostname listesi; ILKI kumeyi olusturur (zorunlu) + --profile maas CLI profili (varsayilan: ${PROFILE}) + --distro-series MAAS ozel imaj adi (varsayilan: ${DISTRO_SERIES}) + --root-password

root@pam parolasi. Verilmezse uretilir ve ekrana yazilir. + --thinpool auto | off | (varsayilan: ${THINPOOL}) + --thinpool-min-gb Thin havuz icin gereken en az bos alan (varsayilan: ${THINPOOL_MIN_GB}) + --net-apply reboot | reload | none (varsayilan: ${NET_APPLY}) + --link0-prefix Corosync link0 icin ayri ag oneki, or. 10.10.20. + Dugumun o agdaki adresi otomatik bulunur. + --serial Katilan dugumleri teker teker deploy et (buyuk kumelerde) + --dry-run Hicbir sey deploy etme, uretilecek user-data'yi goster + -h, --help Bu yardim + +Notlar: + * Kume dugumunun root parolasi API dogrulamasi icin DUZ METIN olarak + user-data'ya girer ve MAAS'ta saklanir. Kisa omurlu bir parola kullanip + kurulumdan sonra degistirin. + * Ayni imaj ve preseed ile birden fazla bagimsiz kume kurabilirsiniz; + kume kimligi yalnizca bu user-data'dan gelir. +EOF +} + +while [ $# -gt 0 ]; do + case "$1" in + --name) CLUSTER_NAME="$2"; shift 2 ;; + --nodes) NODES="$2"; shift 2 ;; + --profile) PROFILE="$2"; shift 2 ;; + --distro-series) DISTRO_SERIES="$2"; shift 2 ;; + --root-password) ROOT_PASSWORD="$2"; shift 2 ;; + --thinpool) THINPOOL="$2"; shift 2 ;; + --thinpool-min-gb) THINPOOL_MIN_GB="$2"; shift 2 ;; + --net-apply) NET_APPLY="$2"; shift 2 ;; + --link0-prefix) LINK0_PREFIX="$2"; shift 2 ;; + --serial) SERIAL=true; shift ;; + --dry-run) DRY_RUN=true; shift ;; + -h|--help) usage; exit 0 ;; + *) echo "Bilinmeyen secenek: $1" >&2; usage; exit 2 ;; + esac +done + +[ -n "$CLUSTER_NAME" ] || { echo "HATA: --name zorunlu" >&2; exit 2; } +[ -n "$NODES" ] || { echo "HATA: --nodes zorunlu" >&2; exit 2; } + +for c in maas jq openssl base64; do + command -v "$c" >/dev/null || { echo "HATA: '$c' bulunamadi" >&2; exit 1; } +done + +IFS=',' read -r -a NODE_LIST <<< "$NODES" +FIRST="${NODE_LIST[0]}" +JOINERS=("${NODE_LIST[@]:1}") + +log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; } +info() { printf ' %s\n' "$*"; } +die() { printf '\n\033[1;31mHATA: %s\033[0m\n' "$*" >&2; exit 1; } + +# ---------------------------------------------------------------- kimlik bilgileri +if [ -z "$ROOT_PASSWORD" ]; then + ROOT_PASSWORD="$(openssl rand -base64 15 | tr -d '/+=' | head -c 16)" + GENERATED=true +else + GENERATED=false +fi +ROOT_PASSWORD_HASH="$(openssl passwd -6 "$ROOT_PASSWORD")" + +# ---------------------------------------------------------------- MAAS yardimcilari +maas_get() { maas "$PROFILE" machines read hostname="$1" 2>/dev/null | jq -r ".[0].$2 // empty"; } + +require_ready() { + local h st + for h in "${NODE_LIST[@]}"; do + st="$(maas_get "$h" status_name)" + [ -n "$st" ] || die "MAAS'ta '$h' adinda makine yok" + [ "$st" = "Ready" ] || die "'$h' durumu '$st' - deploy icin 'Ready' olmali (once release edin)" + info "$h: Ready" + done +} + +wait_status() { + local h="$1" want="$2" timeout="$3" waited=0 st + while [ "$waited" -lt "$timeout" ]; do + st="$(maas_get "$h" status_name)" + case "$st" in + "$want") return 0 ;; + Failed*|Broken*) die "'$h' durumu '$st' - MAAS olaylarina bakin" ;; + esac + sleep 20; waited=$((waited + 20)) + done + die "'$h' ${timeout}s icinde '$want' durumuna gelmedi (son durum: ${st:-bilinmiyor})" +} + +node_ip() { maas "$PROFILE" machines read hostname="$1" 2>/dev/null | jq -r '.[0].ip_addresses[0] // empty'; } + +wait_port() { + local ip="$1" port="$2" timeout="$3" waited=0 + while [ "$waited" -lt "$timeout" ]; do + if timeout 4 bash -c "exec 3<>/dev/tcp/${ip}/${port}" 2>/dev/null; then return 0; fi + sleep 10; waited=$((waited + 10)) + done + die "${ip}:${port} ${timeout}s icinde acilmadi" +} + +fingerprint_of() { + openssl s_client -connect "$1:8006" -servername "$1" /dev/null \ + | openssl x509 -noout -fingerprint -sha256 2>/dev/null | cut -d= -f2 +} + +# Dugumun link0 oneki ile eslesen adresini MAAS'tan bul. +link0_of() { + [ -n "$LINK0_PREFIX" ] || return 0 + maas "$PROFILE" machines read hostname="$1" 2>/dev/null \ + | jq -r --arg p "$LINK0_PREFIX" '.[0].ip_addresses[]? | select(startswith($p))' | head -1 +} + +# ---------------------------------------------------------------- user-data uretimi +render_userdata() { + local mode="$1" host="$2" peer="${3:-}" fp="${4:-}" link0 + link0="$(link0_of "$host")" + + cat </dev/null +} + +# ---------------------------------------------------------------- akis +log "Kume: ${CLUSTER_NAME}" +info "ilk dugum (create) : ${FIRST}" +info "katilanlar (join) : ${JOINERS[*]:-yok}" +info "imaj : custom/${DISTRO_SERIES}" +[ -n "$LINK0_PREFIX" ] && info "corosync link0 oneki: ${LINK0_PREFIX}" + +if [ "$DRY_RUN" = true ]; then + log "DRY RUN - hicbir sey deploy edilmiyor" + echo "--- ${FIRST} (create) ---" + render_userdata create "$FIRST" + for h in "${JOINERS[@]:-}"; do + [ -n "$h" ] || continue + echo "--- ${h} (join) ---" + render_userdata join "$h" "" "" + done + echo + echo "root@pam parolasi: ${ROOT_PASSWORD}" + exit 0 +fi + +log "Makineler kontrol ediliyor" +require_ready + +log "${FIRST} deploy ediliyor (kume olusturuluyor)" +deploy_node "$FIRST" "$(render_userdata create "$FIRST")" +wait_status "$FIRST" Deployed "$DEPLOY_TIMEOUT" +info "MAAS deploy tamamlandi" + +FIRST_IP="$(node_ip "$FIRST")" +[ -n "$FIRST_IP" ] || die "'$FIRST' icin IP bulunamadi" +info "IP: ${FIRST_IP}" + +info "Proxmox arayuzunun acilmasi bekleniyor (ag donusumu icin bir kez yeniden baslar)" +wait_port "$FIRST_IP" 8006 "$PEER_TIMEOUT" + +FP="$(fingerprint_of "$FIRST_IP")" +[ -n "$FP" ] || die "'$FIRST' sertifika parmak izi okunamadi" +info "parmak izi: ${FP}" + +if [ "${#JOINERS[@]}" -eq 0 ] || [ -z "${JOINERS[0]:-}" ]; then + log "Katilacak baska dugum yok" +else + log "Katilan dugumler deploy ediliyor" + for h in "${JOINERS[@]}"; do + info "deploy: $h" + deploy_node "$h" "$(render_userdata join "$h" "$FIRST_IP" "$FP")" + if [ "$SERIAL" = true ]; then + wait_status "$h" Deployed "$DEPLOY_TIMEOUT" + info "$h: MAAS deploy tamamlandi, kumeye katilmasi bekleniyor" + sleep 60 + fi + done + [ "$SERIAL" = false ] && { + for h in "${JOINERS[@]}"; do wait_status "$h" Deployed "$DEPLOY_TIMEOUT"; info "$h: deploy tamamlandi"; done + } +fi + +log "Kume durumu bekleniyor" +EXPECTED="${#NODE_LIST[@]}" +# Tum dugumlerin 8006'da yanit vermesini bekle - kume uyeligini asagidaki +# talimatla dugum uzerinden dogrulayin (bu script dugumlere SSH yapmaz). +for h in "${NODE_LIST[@]}"; do + ip="$(node_ip "$h")" + if [ -n "$ip" ] && wait_port "$ip" 8006 300 2>/dev/null; then + info "$h ($ip): Proxmox arayuzu acik" + else + info "$h: 8006 acilmadi - 'journalctl -u pve-maas-init -b' ile bakin" + fi +done + +cat <&2 + exit 1 +fi + +. /etc/os-release +if [ "${ID:-}" != "ubuntu" ] && [ "${ID_LIKE:-}" != "debian" ]; then + echo "UYARI: Bu script Ubuntu/Debian icin yazildi (bulunan: ${PRETTY_NAME:-bilinmiyor})." >&2 +fi + +export DEBIAN_FRONTEND=noninteractive + +echo "==> Temel paketler kuruluyor" +apt-get update +apt-get install -y --no-install-recommends \ + ca-certificates curl gpg git make parted pigz jq \ + qemu-system-x86 qemu-utils ovmf cloud-image-utils \ + libnbd-bin nbdkit fuse2fs cpu-checker + +echo "==> HashiCorp APT deposu ekleniyor (packer)" +install -d -m 0755 /etc/apt/keyrings +curl -fsSL https://apt.releases.hashicorp.com/gpg \ + | gpg --dearmor --yes -o /etc/apt/keyrings/hashicorp-archive-keyring.gpg +chmod 0644 /etc/apt/keyrings/hashicorp-archive-keyring.gpg +cat > /etc/apt/sources.list.d/hashicorp.list < KVM kontrolu" +if ! kvm-ok; then + echo "HATA: KVM kullanilamiyor. VM'de nested virtualization acik mi? (Proxmox: cpu=host)" >&2 + exit 1 +fi + +# Build root olarak kosuyor ama kullaniciyi da kvm grubuna alalim. +TARGET_USER="${SUDO_USER:-}" +if [ -n "$TARGET_USER" ] && [ "$TARGET_USER" != "root" ]; then + adduser "$TARGET_USER" kvm >/dev/null 2>&1 || true +fi + +echo +echo "==> Hazir. Surumler:" +packer version +qemu-system-x86_64 --version | head -1 diff --git a/scripts/verify-image.sh b/scripts/verify-image.sh new file mode 100755 index 0000000..73da50e --- /dev/null +++ b/scripts/verify-image.sh @@ -0,0 +1,99 @@ +#!/bin/bash +# +# verify-image.sh - Uretilen MAAS tgz imajinin beklenen icerige sahip oldugunu dogrular. +# +# Kullanim: ./scripts/verify-image.sh build/proxmox-ve-9.tar.gz +# +set -uo pipefail + +IMG="${1:-}" +[ -n "$IMG" ] && [ -f "$IMG" ] || { echo "Kullanim: $0 " >&2; exit 2; } + +TMP="$(mktemp -d)" +trap 'rm -rf "$TMP"' EXIT + +echo "==> Imaj: $IMG ($(du -h "$IMG" | cut -f1))" +echo "==> Icerik listesi cikariliyor..." +tar tzf "$IMG" > "$TMP/list" || { echo "HATA: arsiv okunamadi"; exit 1; } +echo " $(wc -l < "$TMP/list") giris" + +pass=0; fail=0 +have() { grep -qx "\./$1" "$TMP/list" || grep -q "^\./$1$" "$TMP/list"; } +present() { grep -q "^\./$1" "$TMP/list"; } + +check() { + local desc="$1" cond="$2" + if eval "$cond"; then + printf ' [ OK ] %s\n' "$desc"; pass=$((pass+1)) + else + printf ' [FAIL] %s\n' "$desc"; fail=$((fail+1)) + fi +} + +echo +echo "==> Dosya varligi kontrolleri" +check "pve-maas-init mevcut" 'present "usr/local/sbin/pve-maas-init"' +check "pve-maas.conf mevcut" 'present "etc/pve-maas/pve-maas.conf"' +check "systemd unit mevcut" 'present "etc/systemd/system/pve-maas-init.service"' +check "unit multi-user.target icin etkin" 'present "etc/systemd/system/multi-user.target.wants/pve-maas-init.service"' +check "Proxmox APT deposu mevcut" 'present "etc/apt/sources.list.d/proxmox.sources"' +check "Proxmox anahtarligi mevcut" 'present "usr/share/keyrings/proxmox-archive-keyring.gpg"' +check "pveproxy ikilisi mevcut" 'present "usr/bin/pveproxy"' +check "pvecm ikilisi mevcut" 'present "usr/bin/pvecm"' +check "pvesh ikilisi mevcut" 'present "usr/bin/pvesh"' +check "ifupdown2 mevcut" 'present "usr/share/ifupdown2"' +check "cloud-init mevcut" 'present "usr/bin/cloud-init"' +check "curtin-hooks mevcut" 'present "curtin/curtin-hooks"' + +echo +echo "==> Olmamasi gerekenler" +check "pmxcfs config.db yok (dugum kimligi temiz)" '! present "var/lib/pve-cluster/config.db"' +check "corosync yapilandirmasi yok" '! present "etc/corosync/corosync.conf"' +check "iSCSI initiator adi yok (dugumde uretilir)" '! present "etc/iscsi/initiatorname.iscsi"' +check "SSH host anahtarlari yok" '! grep -qE "^\./etc/ssh/ssh_host_.*_key$" "$TMP/list"' +check "networking.service etkin DEGIL" '! present "etc/systemd/system/multi-user.target.wants/networking.service"' +check "interfaces.new yok (pvenetcommit ezmesin)" '! present "etc/network/interfaces.new"' +check "Debian cekirdegi yok" '! grep -qE "^\./boot/vmlinuz-.*[^e]-(cloud-)?amd64$" "$TMP/list"' + +echo +echo "==> Proxmox cekirdegi" +if grep -qE '^\./boot/vmlinuz-.*-pve$' "$TMP/list"; then + printf ' [ OK ] PVE cekirdegi: %s\n' "$(grep -oE 'vmlinuz-[^ ]*-pve' "$TMP/list" | head -1)" + pass=$((pass+1)) +else + printf ' [FAIL] /boot altinda *-pve cekirdegi bulunamadi\n'; fail=$((fail+1)) +fi + +echo +echo "==> Ayiklanan dosya icerikleri" +tar xzf "$IMG" -C "$TMP" \ + ./etc/apt/sources.list.d/proxmox.sources \ + ./usr/local/sbin/pve-maas-init \ + ./etc/pve-maas/pve-maas.conf 2>/dev/null + +tar xzf "$IMG" -C "$TMP" ./etc/network/interfaces 2>/dev/null +if [ -f "$TMP/etc/network/interfaces" ]; then + echo "--- /etc/network/interfaces ---" + sed 's/^/ /' "$TMP/etc/network/interfaces" + if grep -qE '^\s*(auto|iface)\s+(?!lo)' "$TMP/etc/network/interfaces" 2>/dev/null \ + || grep -qE '^[[:space:]]*iface[[:space:]]+[^l ]' "$TMP/etc/network/interfaces"; then + printf ' [FAIL] interfaces dosyasinda build VM artigi arayuz var\n'; fail=$((fail+1)) + else + printf ' [ OK ] interfaces yalnizca loopback iceriyor\n'; pass=$((pass+1)) + fi +fi + +if [ -f "$TMP/etc/apt/sources.list.d/proxmox.sources" ]; then + echo "--- proxmox.sources ---" + sed 's/^/ /' "$TMP/etc/apt/sources.list.d/proxmox.sources" +fi + +if [ -x "$TMP/usr/local/sbin/pve-maas-init" ]; then + printf ' [ OK ] pve-maas-init calistirilabilir\n'; pass=$((pass+1)) +else + printf ' [FAIL] pve-maas-init calistirilabilir degil\n'; fail=$((fail+1)) +fi + +echo +echo "==> Sonuc: ${pass} basarili, ${fail} basarisiz" +[ "$fail" -eq 0 ]