Wait for a new machine's own apt, and retry a failed install sooner
Found by building a node from nothing. Its agent tried to install libgomp1 while cloud-init and unattended-upgrades were running apt, and failed with "Unable to lock directory /var/lib/apt/lists/": DPkg::Lock::Timeout covers dpkg's lock, not the one `apt-get update` takes on its lists. The node was correctly kept out of placement and said why, and then sat idle for the ten minutes until the next pass. A package command that fails only on a held lock is now retried for up to ten minutes, reported as "waiting for another apt-get to finish" rather than as a failure. A pass that does fail is retried after a minute, backing off to ten. The weights directory is created when the agent starts rather than by the first download, so a new node reports its disk from its first heartbeat. Onboarding takes several control plane addresses, comma separated, so a new node gets the same failover list as the rest. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
8097a5616d
commit
a40fc53e09
4 files changed
+107
-7
No files matched your search
@@ -24,7 +24,8 @@ usage: $0 --node ID --addr HOST --server URL [options]
|
||||
|
||||
--node ID node id, hostname and certificate common name
|
||||
--addr HOST how to reach the machine over ssh
|
||||
--server URL the control plane's agent endpoint, e.g. https://10.0.0.140:7443
|
||||
--server URL the control plane's agent endpoint, e.g. https://10.0.0.140:7443;
|
||||
several, comma separated, for a node that fails over
|
||||
--fabric NAME InfiniBand fabric this node is attached to; repeatable as
|
||||
--fabric NAME:hca1,hca2 . Omit for a node with no fabric,
|
||||
which can host single-node replicas and will never be given
|
||||
@@ -169,7 +170,14 @@ NODE_IS_HOSTNAME=no
|
||||
# below is written only when it was given, as an override.
|
||||
{
|
||||
echo "# $NODE, written by onboard-node.sh."
|
||||
echo "server: \"$SERVER\""
|
||||
# Several addresses, comma separated, are every control plane the node may
|
||||
# fail over to; a standby refuses it and it moves on to the next.
|
||||
if [ "${SERVER#*,}" != "$SERVER" ]; then
|
||||
echo "servers:"
|
||||
printf '%s\n' "$SERVER" | tr ',' '\n' | sed 's/^ *//; s/^/ - "/; s/$/"/'
|
||||
else
|
||||
echo "server: \"$SERVER\""
|
||||
fi
|
||||
echo
|
||||
echo "tls:"
|
||||
echo " ca: /etc/llmgw/ca.pem"
|
||||
|
||||
Reference in new issue
Block a user