From f6b241b3f762384ed0218dcd4cd5129b607b812c Mon Sep 17 00:00:00 2001 From: ilkermanap Date: Sat, 5 Sep 2026 23:59:39 +0200 Subject: [PATCH] Force IPv4 in the build VM, where apt was losing 31s per large fetch Every apt download over roughly 15 MB inside the build VM took exactly 31 seconds regardless of its size, while a 14.1 MB one took 1 second at 23 MB/s. A fixed cost that ignores size is a timeout, not a bandwidth limit: QEMU's user-mode network advertises IPv6 that does not work, so apt's parallel connections black-holed on it and only fell back to IPv4 when the 30-second timeout expired. Patch the build VM's cloud-init seed from bootcmd, which runs in the init-local stage before SSH is up, so the setting covers upstream's apt calls as well as ours. The same 28.5 MB fetch went from 31s (914 kB/s) to 3s (9152 kB/s). Also fix APT_PROXY, which never worked as documented. A cache cannot see inside a CONNECT tunnel, so the repositories have to be rewritten to plain http first, and Debian 13 keeps the real mirror URLs in /etc/apt/mirrors/*.list behind the mirror+file: method, which rewriting sources.list alone misses. The README now carries the measurements rather than estimates. The headline is that the cache barely matters: with it fully warm it saves about three seconds of a 4m40s build, so a local Debian mirror (138 GB for trixie amd64) would buy nothing here. Co-Authored-By: Claude Opus 5 --- Makefile | 10 ++++++ README.md | 52 +++++++++++++++++++++++++++++++- scripts/customize-samba-ad.sh.in | 13 ++++++++ 3 files changed, 74 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 097c7d3..8da7357 100644 --- a/Makefile +++ b/Makefile @@ -162,6 +162,16 @@ image: checkout $(CUSTOMIZE) sed -i -E 's|^([[:space:]]*disk_size[[:space:]]*=[[:space:]]*).*|\1"$(DISK_SIZE)"|' $(TPL)/debian-cloudimg.pkr.hcl sed -i -E 's|^([[:space:]]*cpus[[:space:]]*=[[:space:]]*).*|\1$(BUILD_CPUS)|' $(TPL)/debian-cloudimg.pkr.hcl sed -i -E 's|^([[:space:]]*memory[[:space:]]*=[[:space:]]*).*|\1$(BUILD_MEM)|' $(TPL)/debian-cloudimg.pkr.hcl + @echo "==> Build VM apt ayari yamalaniyor (IPv4 zorlama + timeout)" + # Every apt fetch over roughly 15 MB inside the build VM stalls for exactly + # 31 seconds regardless of its size, which is a connection timeout rather + # than a bandwidth limit: QEMU's user-mode network offers IPv6 that does not + # actually work, and apt's parallel connections black-hole on it before + # falling back to IPv4. bootcmd runs in cloud-init's init-local stage, + # before SSH is up, so the setting is in place for upstream's apt calls too. + sed -i "/^bootcmd:/a\\ - echo 'Acquire::ForceIPv4 \"true\"; Acquire::http::Timeout \"20\"; Acquire::Retries \"3\";' > /etc/apt/apt.conf.d/99-build-speed" $(TPL)/user-data-cloudimg + @sed -n '/^bootcmd:/,+1p' $(TPL)/user-data-cloudimg + ifeq ($(strip $(DEBIAN_IMAGE_CHANNEL)),stable) @echo "==> Kararli Debian cloud image kullanilacak (packer onbellegi isabet eder)" sed -i -E 's|/daily/latest/|/latest/|g; s|-daily\.qcow2|.qcow2|g' $(TPL)/debian-cloudimg.pkr.hcl diff --git a/README.md b/README.md index 7b8da19..1cae7ce 100644 --- a/README.md +++ b/README.md @@ -341,11 +341,60 @@ their DNS points at a DC. | `ARCH` / `BOOT` | `amd64` / `uefi` | Architecture and boot mode | | `DISK_SIZE` | `16G` | Build VM disk; upstream's 4G is too small | | `PM_REF` | pinned SHA | `canonical/packer-maas` revision | -| `APT_PROXY` | *(empty)* | Local APT cache — see `make deps-cache` | +| `APT_PROXY` | *(empty)* | Local APT cache, e.g. `http://10.0.2.2:3142` — see `make deps-cache` | `make check-upstream` compares the `samba` version in Debian against the image you have, without building anything. +### Build performance + +A full build takes about **4m40s** on a 4 vCPU / 4 GB build VM. Where that time goes was +measured rather than guessed, and the result is not what it looked like. + +The obvious suspect was slow repository access: inside the build VM apt reported +600-900 kB/s, while the build host itself pulled from `deb.debian.org` at 48 MB/s. But +the pattern gave it away — every fetch over roughly 15 MB took *exactly* 31 seconds no +matter how big it was, while a 14.1 MB fetch took 1 second at 23 MB/s. That is a +connection timeout, not a bandwidth limit. QEMU's user-mode network offers IPv6 that does +not actually work, so apt's parallel connections black-holed on it and only fell back to +IPv4 after 30 seconds. + +The Makefile now patches the build VM's cloud-init seed to write +`Acquire::ForceIPv4 "true"` from `bootcmd`, which runs before SSH is up and therefore +covers upstream's own apt calls as well as ours. The same 28.5 MB fetch, across three +builds: + +| Build | Setup | Time | Rate | +|---|---|---|---| +| 1 | no cache, no patch | 31 s | 914 kB/s | +| 2 | apt-cacher-ng, no patch | 31 s | 916 kB/s | +| 3 | apt-cacher-ng + `ForceIPv4` | **3 s** | **9152 kB/s** | + +That fetch goes over `https`, which a cache passes through a `CONNECT` tunnel without +storing, so build 2 isolates the cache from the patch: the cache changed nothing, the +one-line apt setting was worth 28 seconds. + +A local APT cache is still supported and does help on repeat builds, just far less than +you would expect: + +```bash +sudo make deps-cache # installs apt-cacher-ng +sudo make image APT_PROXY=http://10.0.2.2:3142 +``` + +With a fully warm cache the 26.2 MB Samba fetch went from 2s to 0s (59 MB/s) and the +17.9 MB fetch from 1s to 0s — **about three seconds off a 4m40s build**. The rest of the +time is qemu, dpkg and image compression, none of which a faster mirror touches. A full +Debian trixie amd64 mirror costs about 138 GB; the cache that produced these numbers is +44 MB. Mirror the archive if you want it for other reasons, but not to speed these builds +up. + +`10.0.2.2` is the build host as seen from Packer's user-mode network. When a proxy is +configured, repositories are rewritten from `https` to `http` so the cache can serve +them; package signatures are still verified. Debian 13 keeps the real mirror URLs in +`/etc/apt/mirrors/*.list` behind the `mirror+file:` method, so rewriting `sources.list` +alone is not enough. + --- ## Traps this image works around @@ -411,6 +460,7 @@ Tested end to end on real infrastructure. Be sceptical of anything not listed un | Area | Evidence | |---|---| | Build | 28/28 checks in `make verify`; 412 MB image | +| Build speed | Four builds measured; `ForceIPv4` took the 28.5 MB fetch from 31 s to 3 s, a warm APT cache saved a further ~3 s of 4m40s | | Release pipeline | Published to a release, downloaded anonymously, SHA-256 matched, uploaded to MAAS | | Deployment | Both DCs reach `Deployed` from `custom/samba-ad-dc` | | **Provisioning** | First DC created the domain in 21 s, unattended, first attempt | diff --git a/scripts/customize-samba-ad.sh.in b/scripts/customize-samba-ad.sh.in index 41db776..09254fe 100644 --- a/scripts/customize-samba-ad.sh.in +++ b/scripts/customize-samba-ad.sh.in @@ -63,6 +63,19 @@ hostname -f || echo "WARNING: hostname -f does not resolve" >&2 log "holding the cloud-init package" apt-mark hold cloud-init || true +# If a caching proxy is configured, rewrite the repositories from https to +# plain http: a cache cannot see inside a CONNECT tunnel, so https requests are +# passed through uncached. Package signatures are still verified, so this costs +# nothing in integrity. Debian 13 keeps the real mirror URLs in +# /etc/apt/mirrors/*.list behind the "mirror+file:" method, so rewriting +# sources.list alone would miss them. +if [ -n "${http_proxy:-}" ]; then + log "APT cache in use (${http_proxy}); switching repositories to http" + sed -i 's|https://|http://|g' \ + /etc/apt/sources.list /etc/apt/sources.list.d/*.sources \ + /etc/apt/sources.list.d/*.list /etc/apt/mirrors/*.list 2>/dev/null || true +fi + apt-get update $APT install eatmydata || true use_eatmydata