Force IPv4 in the build VM, where apt was losing 31s per large fetch
All checks were successful
build-image / build (push) Successful in 5m41s

Every apt download over roughly 15 MB inside the build VM took exactly 31
seconds regardless of its size, while a 14.1 MB one took 1 second at 23 MB/s.
A fixed cost that ignores size is a timeout, not a bandwidth limit: QEMU's
user-mode network advertises IPv6 that does not work, so apt's parallel
connections black-holed on it and only fell back to IPv4 when the 30-second
timeout expired.

Patch the build VM's cloud-init seed from bootcmd, which runs in the
init-local stage before SSH is up, so the setting covers upstream's apt calls
as well as ours. The same 28.5 MB fetch went from 31s (914 kB/s) to 3s
(9152 kB/s).

Also fix APT_PROXY, which never worked as documented. A cache cannot see
inside a CONNECT tunnel, so the repositories have to be rewritten to plain
http first, and Debian 13 keeps the real mirror URLs in /etc/apt/mirrors/*.list
behind the mirror+file: method, which rewriting sources.list alone misses.

The README now carries the measurements rather than estimates. The headline is
that the cache barely matters: with it fully warm it saves about three seconds
of a 4m40s build, so a local Debian mirror (138 GB for trixie amd64) would buy
nothing here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-05 23:59:39 +02:00
parent fcc54136ce
commit f6b241b3f7
3 changed files with 74 additions and 1 deletions

View File

@@ -341,11 +341,60 @@ their DNS points at a DC.
| `ARCH` / `BOOT` | `amd64` / `uefi` | Architecture and boot mode |
| `DISK_SIZE` | `16G` | Build VM disk; upstream's 4G is too small |
| `PM_REF` | pinned SHA | `canonical/packer-maas` revision |
| `APT_PROXY` | *(empty)* | Local APT cache — see `make deps-cache` |
| `APT_PROXY` | *(empty)* | Local APT cache, e.g. `http://10.0.2.2:3142` — see `make deps-cache` |
`make check-upstream` compares the `samba` version in Debian against the image you have,
without building anything.
### Build performance
A full build takes about **4m40s** on a 4 vCPU / 4 GB build VM. Where that time goes was
measured rather than guessed, and the result is not what it looked like.
The obvious suspect was slow repository access: inside the build VM apt reported
600-900 kB/s, while the build host itself pulled from `deb.debian.org` at 48 MB/s. But
the pattern gave it away — every fetch over roughly 15 MB took *exactly* 31 seconds no
matter how big it was, while a 14.1 MB fetch took 1 second at 23 MB/s. That is a
connection timeout, not a bandwidth limit. QEMU's user-mode network offers IPv6 that does
not actually work, so apt's parallel connections black-holed on it and only fell back to
IPv4 after 30 seconds.
The Makefile now patches the build VM's cloud-init seed to write
`Acquire::ForceIPv4 "true"` from `bootcmd`, which runs before SSH is up and therefore
covers upstream's own apt calls as well as ours. The same 28.5 MB fetch, across three
builds:
| Build | Setup | Time | Rate |
|---|---|---|---|
| 1 | no cache, no patch | 31 s | 914 kB/s |
| 2 | apt-cacher-ng, no patch | 31 s | 916 kB/s |
| 3 | apt-cacher-ng + `ForceIPv4` | **3 s** | **9152 kB/s** |
That fetch goes over `https`, which a cache passes through a `CONNECT` tunnel without
storing, so build 2 isolates the cache from the patch: the cache changed nothing, the
one-line apt setting was worth 28 seconds.
A local APT cache is still supported and does help on repeat builds, just far less than
you would expect:
```bash
sudo make deps-cache # installs apt-cacher-ng
sudo make image APT_PROXY=http://10.0.2.2:3142
```
With a fully warm cache the 26.2 MB Samba fetch went from 2s to 0s (59 MB/s) and the
17.9 MB fetch from 1s to 0s — **about three seconds off a 4m40s build**. The rest of the
time is qemu, dpkg and image compression, none of which a faster mirror touches. A full
Debian trixie amd64 mirror costs about 138 GB; the cache that produced these numbers is
44 MB. Mirror the archive if you want it for other reasons, but not to speed these builds
up.
`10.0.2.2` is the build host as seen from Packer's user-mode network. When a proxy is
configured, repositories are rewritten from `https` to `http` so the cache can serve
them; package signatures are still verified. Debian 13 keeps the real mirror URLs in
`/etc/apt/mirrors/*.list` behind the `mirror+file:` method, so rewriting `sources.list`
alone is not enough.
---
## Traps this image works around
@@ -411,6 +460,7 @@ Tested end to end on real infrastructure. Be sceptical of anything not listed un
| Area | Evidence |
|---|---|
| Build | 28/28 checks in `make verify`; 412 MB image |
| Build speed | Four builds measured; `ForceIPv4` took the 28.5 MB fetch from 31 s to 3 s, a warm APT cache saved a further ~3 s of 4m40s |
| Release pipeline | Published to a release, downloaded anonymously, SHA-256 matched, uploaded to MAAS |
| Deployment | Both DCs reach `Deployed` from `custom/samba-ad-dc` |
| **Provisioning** | First DC created the domain in 21 s, unattended, first attempt |