Stop rebuilding the image every night when nothing has changed

The daily workflow already had a gate, but it could never say no: both of its version comparisons were guaranteed to differ. The image installs samba from trixie-security (2:4.22.10+dfsg-0+deb13u2) while decide-build.sh read only trixie/main (...u1), so the versions never matched. And image-info recorded the kernel as the ABI string from /boot/vmlinuz-* (6.12.107+deb13-amd64), which is a package-name suffix, while the script compared it against the archive's package version (6.12.107-1) — two formats that cannot match by construction. Either one alone forced a build; together they produced a fresh multi-hundred-megabyte release every morning for nothing. Merge trixie, trixie-updates and trixie-security before picking the newest version, and record kernel_version in image-info so there is something comparable on both sides. Compare with dpkg instead of sort -V, which mishandles epochs and tildes, and with "gt" instead of "!=" so an image that is somehow ahead of the archive cannot spin in a rebuild loop. Verified against the live archive and a stubbed release API: archive == image now yields build=no, and the samba, kernel, age, first-build and image-ahead-of-archive branches all decide correctly. Releases published before this change carry no kernel_version, so the kernel check is skipped for them and says so on stderr rather than passing silently. The next build closes that gap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Force IPv4 in the build VM, where apt was losing 31s per large fetch
2026-09-07 13:32:05 +02:00 · 2026-09-05 23:59:39 +02:00
4 changed files with 139 additions and 14 deletions
--- a/10
+++ b/10
@@ -162,6 +162,16 @@ image: checkout $(CUSTOMIZE)
 	sed -i -E 's|^([[:space:]]*disk_size[[:space:]]*=[[:space:]]*).*|\1"$(DISK_SIZE)"|' $(TPL)/debian-cloudimg.pkr.hcl
 	sed -i -E 's|^([[:space:]]*cpus[[:space:]]*=[[:space:]]*).*|\1$(BUILD_CPUS)|' $(TPL)/debian-cloudimg.pkr.hcl
 	sed -i -E 's|^([[:space:]]*memory[[:space:]]*=[[:space:]]*).*|\1$(BUILD_MEM)|' $(TPL)/debian-cloudimg.pkr.hcl
+	@echo "==> Build VM apt ayari yamalaniyor (IPv4 zorlama + timeout)"
+	# Every apt fetch over roughly 15 MB inside the build VM stalls for exactly
+	# 31 seconds regardless of its size, which is a connection timeout rather
+	# than a bandwidth limit: QEMU's user-mode network offers IPv6 that does not
+	# actually work, and apt's parallel connections black-hole on it before
+	# falling back to IPv4. bootcmd runs in cloud-init's init-local stage,
+	# before SSH is up, so the setting is in place for upstream's apt calls too.
+	sed -i "/^bootcmd:/a\\  - echo 'Acquire::ForceIPv4 \"true\"; Acquire::http::Timeout \"20\"; Acquire::Retries \"3\";' > /etc/apt/apt.conf.d/99-build-speed" $(TPL)/user-data-cloudimg
+	@sed -n '/^bootcmd:/,+1p' $(TPL)/user-data-cloudimg
+
 ifeq ($(strip $(DEBIAN_IMAGE_CHANNEL)),stable)
 	@echo "==> Kararli Debian cloud image kullanilacak (packer onbellegi isabet eder)"
 	sed -i -E 's|/daily/latest/|/latest/|g; s|-daily\.qcow2|.qcow2|g' $(TPL)/debian-cloudimg.pkr.hcl
--- a/README.md
+++ b/README.md
@@ -341,11 +341,60 @@ their DNS points at a DC.
 | `ARCH` / `BOOT` | `amd64` / `uefi` | Architecture and boot mode |
 | `DISK_SIZE` | `16G` | Build VM disk; upstream's 4G is too small |
 | `PM_REF` | pinned SHA | `canonical/packer-maas` revision |
-| `APT_PROXY` | *(empty)* | Local APT cache — see `make deps-cache` |
+| `APT_PROXY` | *(empty)* | Local APT cache, e.g. `http://10.0.2.2:3142` — see `make deps-cache` |

 `make check-upstream` compares the `samba` version in Debian against the image you have,
 without building anything.

+### Build performance
+
+A full build takes about **4m40s** on a 4 vCPU / 4 GB build VM. Where that time goes was
+measured rather than guessed, and the result is not what it looked like.
+
+The obvious suspect was slow repository access: inside the build VM apt reported
+600-900 kB/s, while the build host itself pulled from `deb.debian.org` at 48 MB/s. But
+the pattern gave it away — every fetch over roughly 15 MB took *exactly* 31 seconds no
+matter how big it was, while a 14.1 MB fetch took 1 second at 23 MB/s. That is a
+connection timeout, not a bandwidth limit. QEMU's user-mode network offers IPv6 that does
+not actually work, so apt's parallel connections black-holed on it and only fell back to
+IPv4 after 30 seconds.
+
+The Makefile now patches the build VM's cloud-init seed to write
+`Acquire::ForceIPv4 "true"` from `bootcmd`, which runs before SSH is up and therefore
+covers upstream's own apt calls as well as ours. The same 28.5 MB fetch, across three
+builds:
+
+| Build | Setup | Time | Rate |
+|---|---|---|---|
+| 1 | no cache, no patch | 31 s | 914 kB/s |
+| 2 | apt-cacher-ng, no patch | 31 s | 916 kB/s |
+| 3 | apt-cacher-ng + `ForceIPv4` | **3 s** | **9152 kB/s** |
+
+That fetch goes over `https`, which a cache passes through a `CONNECT` tunnel without
+storing, so build 2 isolates the cache from the patch: the cache changed nothing, the
+one-line apt setting was worth 28 seconds.
+
+A local APT cache is still supported and does help on repeat builds, just far less than
+you would expect:
+
+```bash
+sudo make deps-cache                              # installs apt-cacher-ng
+sudo make image APT_PROXY=http://10.0.2.2:3142
+```
+
+With a fully warm cache the 26.2 MB Samba fetch went from 2s to 0s (59 MB/s) and the
+17.9 MB fetch from 1s to 0s — **about three seconds off a 4m40s build**. The rest of the
+time is qemu, dpkg and image compression, none of which a faster mirror touches. A full
+Debian trixie amd64 mirror costs about 138 GB; the cache that produced these numbers is
+44 MB. Mirror the archive if you want it for other reasons, but not to speed these builds
+up.
+
+`10.0.2.2` is the build host as seen from Packer's user-mode network. When a proxy is
+configured, repositories are rewritten from `https` to `http` so the cache can serve
+them; package signatures are still verified. Debian 13 keeps the real mirror URLs in
+`/etc/apt/mirrors/*.list` behind the `mirror+file:` method, so rewriting `sources.list`
+alone is not enough.
+
 ---

 ## Traps this image works around
@@ -411,6 +460,7 @@ Tested end to end on real infrastructure. Be sceptical of anything not listed un
 | Area | Evidence |
 |---|---|
 | Build | 28/28 checks in `make verify`; 412 MB image |
+| Build speed | Four builds measured; `ForceIPv4` took the 28.5 MB fetch from 31 s to 3 s, a warm APT cache saved a further ~3 s of 4m40s |
 | Release pipeline | Published to a release, downloaded anonymously, SHA-256 matched, uploaded to MAAS |
 | Deployment | Both DCs reach `Deployed` from `custom/samba-ad-dc` |
 | **Provisioning** | First DC created the domain in 21 s, unattended, first attempt |
--- a/scripts/ci/decide-build.sh
+++ b/scripts/ci/decide-build.sh
@@ -9,12 +9,27 @@
 #
 # A rebuild happens when any of these is true:
 #
-#   1. samba in the Debian suite differs from the published image
-#   2. the kernel package differs — kernel security fixes do not bump samba
+#   1. samba in the Debian archive is newer than the published image
+#   2. the kernel package is newer — kernel security fixes do not bump samba
 #   3. the newest release is older than max-age-days — most Debian security
 #      updates touch neither, so without a floor an image could sit unchanged
 #      for months while its openssl and glibc went stale
 #
+# Two things this gets wrong if you are not careful, both of which made it
+# rebuild every single night:
+#
+#   * The image installs from trixie, trixie-updates AND trixie-security.
+#     Reading only trixie/main reports samba 2:4.22.10+dfsg-0+deb13u1 while the
+#     image contains ...u2 from security, so the versions never match.
+#   * /boot/vmlinuz-* yields an ABI string such as 6.12.107+deb13-amd64, which
+#     is a package-name suffix, not a version. Comparing it against the
+#     archive's 6.12.107-1 never matches either. Images now also record
+#     kernel_version, which is directly comparable.
+#
+# Versions are compared with dpkg rather than sort -V, which mishandles epochs
+# and tildes, and with "gt" rather than "!=" so an image that is somehow ahead
+# of the archive does not trigger an endless rebuild loop.
+#
 # Requires GITEA_API and GITEA_TOKEN in the environment.
 set -euo pipefail
 cd "$(dirname "$0")/../.."
@@ -26,13 +41,39 @@ MAX_AGE_DAYS="${1:-30}"
 SUITE=$(make -s print-var VAR=DEBIAN_SERIES)
 ARCH=$(make -s print-var VAR=ARCH)

-# Newest version of each package we care about, in one pass over the index.
-PKGS=$(curl -fsS "http://deb.debian.org/debian/dists/${SUITE}/main/binary-${ARCH}/Packages.gz" | gunzip)
-newest() {
-    printf '%s\n' "$PKGS" \
-      | awk -v want="$1" '$1=="Package:" && $2==want {p=1; next} p && $1=="Version:" {print $2; p=0}' \
-      | sort -V | tail -1
+TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT
+INDEX="$TMP/packages"
+
+# $1 url, $2 "required"|"optional". Appends the decompressed index to $INDEX.
+fetch_index() {
+    local url=$1 need=$2
+    if curl -fsS "$url" 2>/dev/null | xz -dc >> "$INDEX" 2>/dev/null; then
+        return 0
+    fi
+    if [ "$need" = required ]; then
+        echo "cannot read package index: $url" >&2
+        exit 1
+    fi
+    echo "note: no index at $url (skipping)" >&2
 }
+
+: > "$INDEX"
+fetch_index "http://deb.debian.org/debian/dists/${SUITE}/main/binary-${ARCH}/Packages.xz"                   required
+fetch_index "http://deb.debian.org/debian-security/dists/${SUITE}-security/main/binary-${ARCH}/Packages.xz" required
+fetch_index "http://deb.debian.org/debian/dists/${SUITE}-updates/main/binary-${ARCH}/Packages.xz"           optional
+
+# Highest version of a package across every suite we merged.
+newest() {
+    local best="" v
+    while read -r v; do
+        [ -n "$v" ] || continue
+        if [ -z "$best" ] || dpkg --compare-versions "$v" gt "$best"; then best=$v; fi
+    done < <(awk -v want="$1" '
+        $1=="Package:" && $2==want {p=1; next}
+        p && $1=="Version:" {print $2; p=0}' "$INDEX")
+    printf '%s\n' "$best"
+}
+
 UP_SAMBA=$(newest samba)
 UP_KERNEL=$(newest "linux-image-${ARCH}")
 [ -n "$UP_SAMBA" ]  || { echo "samba not found in Debian ${SUITE}" >&2; exit 1; }
@@ -60,9 +101,10 @@ PREV_SAMBA="" PREV_KERNEL=""
 if [ -n "$INFO_URL" ]; then
    INFO=$(curl -fsSL "$INFO_URL" || true)
    PREV_SAMBA=$(printf '%s' "$INFO"  | awk -F= '$1=="samba"{print $2}')
-    PREV_KERNEL=$(printf '%s' "$INFO" | awk -F= '$1=="kernel"{print $2}')
+    PREV_KERNEL=$(printf '%s' "$INFO" | awk -F= '$1=="kernel_version"{print $2}')
+    [ "$PREV_KERNEL" = unknown ] && PREV_KERNEL=""
 fi
-echo "published: samba=${PREV_SAMBA:-<none>} kernel=${PREV_KERNEL:-<none>}" >&2
+echo "published: samba=${PREV_SAMBA:-<none>} kernel_version=${PREV_KERNEL:-<none>}" >&2

 AGE_DAYS=99999
 if [ -n "$CREATED" ]; then
@@ -77,13 +119,18 @@ BUILD=no
 REASON="up to date"
 if [ -z "$PREV_SAMBA" ]; then
    BUILD=yes; REASON="no published image yet"
-elif [ "$UP_SAMBA" != "$PREV_SAMBA" ]; then
+elif dpkg --compare-versions "$UP_SAMBA" gt "$PREV_SAMBA"; then
    BUILD=yes; REASON="samba ${PREV_SAMBA} -> ${UP_SAMBA}"
-elif [ "$UP_KERNEL" != "$PREV_KERNEL" ]; then
+elif [ -n "$PREV_KERNEL" ] && dpkg --compare-versions "$UP_KERNEL" gt "$PREV_KERNEL"; then
    BUILD=yes; REASON="kernel ${PREV_KERNEL} -> ${UP_KERNEL}"
 elif [ "$AGE_DAYS" -ge "$MAX_AGE_DAYS" ]; then
    BUILD=yes; REASON="image is ${AGE_DAYS} days old (limit ${MAX_AGE_DAYS}) — picking up Debian updates"
 fi
+# Releases published before image-info carried kernel_version cannot be checked
+# for kernel changes; say so rather than letting the gap pass unnoticed.
+if [ -n "$PREV_SAMBA" ] && [ -z "$PREV_KERNEL" ]; then
+    echo "note: published image records no kernel_version; kernel check skipped" >&2
+fi
 echo "decision: ${BUILD} (${REASON})" >&2

 printf 'build=%s\n'   "$BUILD"
--- a/scripts/customize-samba-ad.sh.in
+++ b/scripts/customize-samba-ad.sh.in
@@ -63,6 +63,19 @@ hostname -f || echo "WARNING: hostname -f does not resolve" >&2
 log "holding the cloud-init package"
 apt-mark hold cloud-init || true

+# If a caching proxy is configured, rewrite the repositories from https to
+# plain http: a cache cannot see inside a CONNECT tunnel, so https requests are
+# passed through uncached. Package signatures are still verified, so this costs
+# nothing in integrity. Debian 13 keeps the real mirror URLs in
+# /etc/apt/mirrors/*.list behind the "mirror+file:" method, so rewriting
+# sources.list alone would miss them.
+if [ -n "${http_proxy:-}" ]; then
+    log "APT cache in use (${http_proxy}); switching repositories to http"
+    sed -i 's|https://|http://|g' \
+        /etc/apt/sources.list /etc/apt/sources.list.d/*.sources \
+        /etc/apt/sources.list.d/*.list /etc/apt/mirrors/*.list 2>/dev/null || true
+fi
+
 apt-get update
 $APT install eatmydata || true
 use_eatmydata
@@ -224,7 +237,12 @@ log "writing /etc/adc-maas/image-info"
    echo "debian_suite=${DEBIAN_SUITE}"
    echo "packer_maas_ref=${PACKER_MAAS_REF}"
    dpkg-query -W -f='${Package}=${Version}\n' samba samba-ad-dc winbind krb5-user 2>/dev/null
-    echo "kernel=$(ls -1 /boot/vmlinuz-* 2>/dev/null | sed 's|.*/vmlinuz-||' | head -1)"
+    kabi=$(ls -1 /boot/vmlinuz-* 2>/dev/null | sed 's|.*/vmlinuz-||' | head -1)
+    echo "kernel=${kabi}"
+    # The line above is the ABI string, which is a package-name suffix and only
+    # changes on an ABI bump. CI needs something it can actually compare against
+    # the archive, so record the kernel package version as well.
+    echo "kernel_version=$(dpkg-query -W -f='${Version}' "linux-image-${kabi}" 2>/dev/null || echo unknown)"
 } > /etc/adc-maas/image-info
 cat /etc/adc-maas/image-info