Wait for a new machine's own apt, and retry a failed install sooner

Found by building a node from nothing. Its agent tried to install libgomp1
while cloud-init and unattended-upgrades were running apt, and failed with
"Unable to lock directory /var/lib/apt/lists/": DPkg::Lock::Timeout covers
dpkg's lock, not the one `apt-get update` takes on its lists. The node was
correctly kept out of placement and said why, and then sat idle for the ten
minutes until the next pass.

A package command that fails only on a held lock is now retried for up to
ten minutes, reported as "waiting for another apt-get to finish" rather than
as a failure. A pass that does fail is retried after a minute, backing off to
ten.

The weights directory is created when the agent starts rather than by the
first download, so a new node reports its disk from its first heartbeat.

Onboarding takes several control plane addresses, comma separated, so a new
node gets the same failover list as the rest.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-28 12:14:22 +02:00
1 parent 8097a5616d
commit a40fc53e09
4 files changed
+107 -7

No files matched your search

+10 -2
View File
@@ -24,7 +24,8 @@ usage: $0 --node ID --addr HOST --server URL [options]
--node ID node id, hostname and certificate common name
--addr HOST how to reach the machine over ssh
--server URL the control plane's agent endpoint, e.g. https://10.0.0.140:7443
--server URL the control plane's agent endpoint, e.g. https://10.0.0.140:7443;
several, comma separated, for a node that fails over
--fabric NAME InfiniBand fabric this node is attached to; repeatable as
--fabric NAME:hca1,hca2 . Omit for a node with no fabric,
which can host single-node replicas and will never be given
@@ -169,7 +170,14 @@ NODE_IS_HOSTNAME=no
# below is written only when it was given, as an override.
{
echo "# $NODE, written by onboard-node.sh."
echo "server: \"$SERVER\""
# Several addresses, comma separated, are every control plane the node may
# fail over to; a standby refuses it and it moves on to the next.
if [ "${SERVER#*,}" != "$SERVER" ]; then
echo "servers:"
printf '%s\n' "$SERVER" | tr ',' '\n' | sed 's/^ *//; s/^/ - "/; s/$/"/'
else
echo "server: \"$SERVER\""
fi
echo
echo "tls:"
echo " ca: /etc/llmgw/ca.pem"