From e1d2c00a976246f38865a9ad93545eec9db5a1d6 Mon Sep 17 00:00:00 2001 From: ilkermanap Date: Sat, 5 Sep 2026 23:59:48 +0200 Subject: [PATCH] Force IPv4 in the build VM and fix the APT cache path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The build logs here show the same stall measured on maas-samba-ad: a 28.5 MB apt fetch taking 31s and a 12.0 MB one taking 30s, while a 14.1 MB fetch completed in under a second. A fixed cost that ignores size is a timeout, not a bandwidth limit — QEMU's user-mode network advertises IPv6 that does not work, so apt's parallel connections black-hole on it and fall back to IPv4 only when the 30-second timeout expires. Patch the build VM's cloud-init seed from bootcmd, which runs before SSH is up and therefore covers upstream's apt calls too. On maas-samba-ad this took the same 28.5 MB fetch from 31s to 3s; no build has been run here since, and the README says so. APT_PROXY also never worked as documented: a cache cannot see inside a CONNECT tunnel, so repositories must be rewritten to plain http, and Debian 13 keeps the real mirror URLs in /etc/apt/mirrors/*.list behind the mirror+file: method, which the old sed missed. Drop the invented "roughly 700 MB" saving from the README. On maas-samba-ad a fully warm cache was worth about three seconds of a 4m40s build. This image pulls far more from the Proxmox repository, so the cache may matter more here, but that is unmeasured and is now listed as such. Co-Authored-By: Claude Opus 5 --- Makefile | 11 ++++++++++ README.md | 36 +++++++++++++++++++++++++-------- scripts/customize-proxmox.sh.in | 7 +++++-- 3 files changed, 44 insertions(+), 10 deletions(-) diff --git a/Makefile b/Makefile index da9b04a..ccbc5a0 100644 --- a/Makefile +++ b/Makefile @@ -174,6 +174,17 @@ image: checkout $(CUSTOMIZE) sed -i -E 's|^([[:space:]]*disk_size[[:space:]]*=[[:space:]]*).*|\1"$(DISK_SIZE)"|' $(TPL)/debian-cloudimg.pkr.hcl sed -i -E 's|^([[:space:]]*cpus[[:space:]]*=[[:space:]]*).*|\1$(BUILD_CPUS)|' $(TPL)/debian-cloudimg.pkr.hcl sed -i -E 's|^([[:space:]]*memory[[:space:]]*=[[:space:]]*).*|\1$(BUILD_MEM)|' $(TPL)/debian-cloudimg.pkr.hcl + @echo "==> Build VM apt ayari yamalaniyor (IPv4 zorlama + timeout)" + # Every apt fetch over roughly 15 MB inside the build VM used to stall for + # exactly 31 seconds regardless of its size, which is a connection timeout + # rather than a bandwidth limit: QEMU's user-mode network offers IPv6 that + # does not actually work, so apt's parallel connections black-hole on it and + # only fall back to IPv4 when the timeout expires. Forcing IPv4 took the + # 28.5 MB upstream fetch from 31s to 3s. bootcmd runs in cloud-init's + # init-local stage, before SSH is up, so this is in place for the upstream + # provisioners too, not just ours. + sed -i "/^bootcmd:/a\\ - echo 'Acquire::ForceIPv4 \"true\"; Acquire::http::Timeout \"20\"; Acquire::Retries \"3\";' > /etc/apt/apt.conf.d/99-build-speed" $(TPL)/user-data-cloudimg + @sed -n '/^bootcmd:/,+1p' $(TPL)/user-data-cloudimg ifeq ($(strip $(DEBIAN_IMAGE_CHANNEL)),stable) @echo "==> Kararli Debian cloud image kullanilacak (packer onbellegi isabet eder)" sed -i -E 's|/daily/latest/|/latest/|g; s|-daily\.qcow2|.qcow2|g' $(TPL)/debian-cloudimg.pkr.hcl diff --git a/README.md b/README.md index 12ce127..0f68591 100644 --- a/README.md +++ b/README.md @@ -276,7 +276,7 @@ All of these are `make` variables — `sudo make image DISK_SIZE=24G`, and so on |---|---|---| | `DEBIAN_IMAGE_CHANNEL` | `stable` | `stable` uses a fixed URL so Packer's cache works; `daily` is upstream's default and changes every day. *Only `stable` was tested.* | | `GZIP_LEVEL` | `6` | Tarball compression. Upstream uses 9 | -| `APT_PROXY` | *(empty)* | Local APT cache, e.g. `http://10.0.2.2:3142` — see `make deps-cache`. *Untested.* | +| `APT_PROXY` | *(empty)* | Local APT cache, e.g. `http://10.0.2.2:3142` — see `make deps-cache`. *Untested on this image; measured on maas-samba-ad.* | ### MAAS @@ -737,9 +737,17 @@ All of them are on by default: - **`GZIP_LEVEL=6`** — upstream uses `--best` (9). With `pigz` this is noticeably faster for a few percent more size. -For repeated builds, a local APT cache should remove roughly 700 MB of downloads. -*This path is untested and the figure is an estimate from package sizes, not a -measurement:* +- **Forcing IPv4** — the Makefile patches the build VM's cloud-init seed to write + `Acquire::ForceIPv4 "true"` before SSH comes up. Without it, every apt download over + roughly 15 MB stalled for *exactly* 31 seconds regardless of its size: QEMU's + user-mode network offers IPv6 that does not actually work, so apt's parallel + connections black-holed on it and only fell back to IPv4 once the timeout expired. + Measured on the sibling [maas-samba-ad](https://github.com/ilkermanap/maas-samba-ad) + build, the same 28.5 MB fetch went from 31s (914 kB/s) to 3s (9.2 MB/s). The 31s + stall is visible in this project's build logs too, but the fix has not yet been + measured here. + +For repeated builds you can also put a local APT cache in front: ```bash sudo make deps-cache # installs apt-cacher-ng @@ -748,7 +756,16 @@ sudo make image APT_PROXY=http://10.0.2.2:3142 `10.0.2.2` is the build host as seen from Packer's user-mode network. When a proxy is configured, Debian repositories are rewritten from `https` to `http` so the cache can -serve them; package signatures are still verified. +serve them — a cache cannot see inside a `CONNECT` tunnel — and package signatures are +still verified. Note that Debian 13 keeps the real mirror URLs in +`/etc/apt/mirrors/*.list` behind the `mirror+file:` method, so rewriting `sources.list` +alone is not enough. + +*Do not expect much from the cache on its own.* On the samba-ad build a fully warm cache +turned a 26.2 MB fetch from 2s into 0s and a 17.9 MB fetch from 1s into 0s — about three +seconds off a 4m40s build. Almost all of the time this project's builds spend on the +network is the ~800 MB pulled from the Proxmox repository, which the cache *would* serve +on a repeat build; that has not been measured. --- @@ -973,9 +990,12 @@ plaintext `PVE_ROOT_PASSWORD`, `PVE_ENABLED=false`, `PVE_FQDN`, and every none deployed. See [arm64](#arm64). - BIOS boot (`BOOT=bios`) - `DEBIAN_IMAGE_CHANNEL=daily` -- `APT_PROXY` and `make deps-cache`. The ~700 MB figure quoted under - [Build performance](#build-performance) is an estimate from download sizes, not a - measurement. +- `APT_PROXY` and `make deps-cache` *for this image*. The cache was measured end to end + on the sibling maas-samba-ad build, where it saved about three seconds; this project + pulls far more from the Proxmox repository, and that case has not been measured. +- The `Acquire::ForceIPv4` seed patch *for this image*. It was measured on maas-samba-ad + (31s to 3s on the same 28.5 MB fetch); the same 31s stall appears in this project's + logs, but no build has been run here since the patch was added. - A non-default `IMAGE_NAME` **Make targets** — `make upload` (the image was uploaded with the equivalent `maas` diff --git a/scripts/customize-proxmox.sh.in b/scripts/customize-proxmox.sh.in index 5b1b1f7..85e85d5 100644 --- a/scripts/customize-proxmox.sh.in +++ b/scripts/customize-proxmox.sh.in @@ -74,8 +74,11 @@ apt-mark hold cloud-init || true # Paket imzalari yine dogrulandigi icin guvenlik kaybi yok. if [ -n "${http_proxy:-}" ]; then log "APT onbellegi kullaniliyor (${http_proxy}); Debian depolari http'ye cevriliyor" - sed -i 's|https://deb.debian.org|http://deb.debian.org|g; s|https://security.debian.org|http://security.debian.org|g' \ - /etc/apt/sources.list /etc/apt/sources.list.d/*.sources /etc/apt/sources.list.d/*.list 2>/dev/null || true + # Debian 13 keeps the actual mirror URLs in /etc/apt/mirrors/*.list, reached + # through the "mirror+file:" method, so rewriting sources.list alone misses them. + sed -i 's|https://|http://|g' \ + /etc/apt/sources.list /etc/apt/sources.list.d/*.sources \ + /etc/apt/sources.list.d/*.list /etc/apt/mirrors/*.list 2>/dev/null || true fi log "eatmydata kuruluyor"