Force IPv4 in the build VM, where apt was losing 31s per large fetch

Every apt download over roughly 15 MB inside the build VM took exactly 31
seconds regardless of its size, while a 14.1 MB one took 1 second at 23 MB/s.
A fixed cost that ignores size is a timeout, not a bandwidth limit: QEMU's
user-mode network advertises IPv6 that does not work, so apt's parallel
connections black-holed on it and only fell back to IPv4 when the 30-second
timeout expired.

Patch the build VM's cloud-init seed from bootcmd, which runs in the
init-local stage before SSH is up, so the setting covers upstream's apt calls
as well as ours. The same 28.5 MB fetch went from 31s (914 kB/s) to 3s
(9152 kB/s).

Also fix APT_PROXY, which never worked as documented. A cache cannot see
inside a CONNECT tunnel, so the repositories have to be rewritten to plain
http first, and Debian 13 keeps the real mirror URLs in /etc/apt/mirrors/*.list
behind the mirror+file: method, which rewriting sources.list alone misses.

The README now carries the measurements rather than estimates. The headline is
that the cache barely matters: with it fully warm it saves about three seconds
of a 4m40s build, so a local Debian mirror (138 GB for trixie amd64) would buy
nothing here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Makefile

@@ -162,6 +162,16 @@ image: checkout $(CUSTOMIZE)
 	sed -i -E 's|^([[:space:]]*disk_size[[:space:]]*=[[:space:]]*).*|\1"$(DISK_SIZE)"|' $(TPL)/debian-cloudimg.pkr.hcl
 	sed -i -E 's|^([[:space:]]*cpus[[:space:]]*=[[:space:]]*).*|\1$(BUILD_CPUS)|' $(TPL)/debian-cloudimg.pkr.hcl
 	sed -i -E 's|^([[:space:]]*memory[[:space:]]*=[[:space:]]*).*|\1$(BUILD_MEM)|' $(TPL)/debian-cloudimg.pkr.hcl
+	@echo "==> Build VM apt ayari yamalaniyor (IPv4 zorlama + timeout)"
+	# Every apt fetch over roughly 15 MB inside the build VM stalls for exactly
+	# 31 seconds regardless of its size, which is a connection timeout rather
+	# than a bandwidth limit: QEMU's user-mode network offers IPv6 that does not
+	# actually work, and apt's parallel connections black-hole on it before
+	# falling back to IPv4. bootcmd runs in cloud-init's init-local stage,
+	# before SSH is up, so the setting is in place for upstream's apt calls too.
+	sed -i "/^bootcmd:/a\\  - echo 'Acquire::ForceIPv4 \"true\"; Acquire::http::Timeout \"20\"; Acquire::Retries \"3\";' > /etc/apt/apt.conf.d/99-build-speed" $(TPL)/user-data-cloudimg
+	@sed -n '/^bootcmd:/,+1p' $(TPL)/user-data-cloudimg
+
 ifeq ($(strip $(DEBIAN_IMAGE_CHANNEL)),stable)
 	@echo "==> Kararli Debian cloud image kullanilacak (packer onbellegi isabet eder)"
 	sed -i -E 's|/daily/latest/|/latest/|g; s|-daily\.qcow2|.qcow2|g' $(TPL)/debian-cloudimg.pkr.hcl

README.md

@@ -341,11 +341,60 @@ their DNS points at a DC.
 | `ARCH` / `BOOT` | `amd64` / `uefi` | Architecture and boot mode |
 | `DISK_SIZE` | `16G` | Build VM disk; upstream's 4G is too small |
 | `PM_REF` | pinned SHA | `canonical/packer-maas` revision |
-| `APT_PROXY` | *(empty)* | Local APT cache — see `make deps-cache` |
+| `APT_PROXY` | *(empty)* | Local APT cache, e.g. `http://10.0.2.2:3142` — see `make deps-cache` |
 
 `make check-upstream` compares the `samba` version in Debian against the image you have,
 without building anything.
 
+### Build performance
+
+A full build takes about **4m40s** on a 4 vCPU / 4 GB build VM. Where that time goes was
+measured rather than guessed, and the result is not what it looked like.
+
+The obvious suspect was slow repository access: inside the build VM apt reported
+600-900 kB/s, while the build host itself pulled from `deb.debian.org` at 48 MB/s. But
+the pattern gave it away — every fetch over roughly 15 MB took *exactly* 31 seconds no
+matter how big it was, while a 14.1 MB fetch took 1 second at 23 MB/s. That is a
+connection timeout, not a bandwidth limit. QEMU's user-mode network offers IPv6 that does
+not actually work, so apt's parallel connections black-holed on it and only fell back to
+IPv4 after 30 seconds.
+
+The Makefile now patches the build VM's cloud-init seed to write
+`Acquire::ForceIPv4 "true"` from `bootcmd`, which runs before SSH is up and therefore
+covers upstream's own apt calls as well as ours. The same 28.5 MB fetch, across three
+builds:
+
+| Build | Setup | Time | Rate |
+|---|---|---|---|
+| 1 | no cache, no patch | 31 s | 914 kB/s |
+| 2 | apt-cacher-ng, no patch | 31 s | 916 kB/s |
+| 3 | apt-cacher-ng + `ForceIPv4` | **3 s** | **9152 kB/s** |
+
+That fetch goes over `https`, which a cache passes through a `CONNECT` tunnel without
+storing, so build 2 isolates the cache from the patch: the cache changed nothing, the
+one-line apt setting was worth 28 seconds.
+
+A local APT cache is still supported and does help on repeat builds, just far less than
+you would expect:
+
+```bash
+sudo make deps-cache                              # installs apt-cacher-ng
+sudo make image APT_PROXY=http://10.0.2.2:3142
+```
+
+With a fully warm cache the 26.2 MB Samba fetch went from 2s to 0s (59 MB/s) and the
+17.9 MB fetch from 1s to 0s — **about three seconds off a 4m40s build**. The rest of the
+time is qemu, dpkg and image compression, none of which a faster mirror touches. A full
+Debian trixie amd64 mirror costs about 138 GB; the cache that produced these numbers is
+44 MB. Mirror the archive if you want it for other reasons, but not to speed these builds
+up.
+
+`10.0.2.2` is the build host as seen from Packer's user-mode network. When a proxy is
+configured, repositories are rewritten from `https` to `http` so the cache can serve
+them; package signatures are still verified. Debian 13 keeps the real mirror URLs in
+`/etc/apt/mirrors/*.list` behind the `mirror+file:` method, so rewriting `sources.list`
+alone is not enough.
+
 ---
 
 ## Traps this image works around
@@ -411,6 +460,7 @@ Tested end to end on real infrastructure. Be sceptical of anything not listed un
 | Area | Evidence |
 |---|---|
 | Build | 28/28 checks in `make verify`; 412 MB image |
+| Build speed | Four builds measured; `ForceIPv4` took the 28.5 MB fetch from 31 s to 3 s, a warm APT cache saved a further ~3 s of 4m40s |
 | Release pipeline | Published to a release, downloaded anonymously, SHA-256 matched, uploaded to MAAS |
 | Deployment | Both DCs reach `Deployed` from `custom/samba-ad-dc` |
 | **Provisioning** | First DC created the domain in 21 s, unattended, first attempt |

scripts/customize-samba-ad.sh.in

@@ -63,6 +63,19 @@ hostname -f || echo "WARNING: hostname -f does not resolve" >&2
 log "holding the cloud-init package"
 apt-mark hold cloud-init || true
 
+# If a caching proxy is configured, rewrite the repositories from https to
+# plain http: a cache cannot see inside a CONNECT tunnel, so https requests are
+# passed through uncached. Package signatures are still verified, so this costs
+# nothing in integrity. Debian 13 keeps the real mirror URLs in
+# /etc/apt/mirrors/*.list behind the "mirror+file:" method, so rewriting
+# sources.list alone would miss them.
+if [ -n "${http_proxy:-}" ]; then
+    log "APT cache in use (${http_proxy}); switching repositories to http"
+    sed -i 's|https://|http://|g' \
+        /etc/apt/sources.list /etc/apt/sources.list.d/*.sources \
+        /etc/apt/sources.list.d/*.list /etc/apt/mirrors/*.list 2>/dev/null || true
+fi
+
 apt-get update
 $APT install eatmydata || true
 use_eatmydata