A restart is not refused for room another restart is giving back
CI / test (push) Successful in 8m51s

On a 4-core single-machine install, `ctl model resources` was run on the
judge (to two cores) and on the starter (to none), then both restarted.
The judge's restart was refused: its replacement fit on no node even with
its old replica stopped, because the starter still held its two cores. A
restart now waits for other restarts under way (queued, retiring or
draining) when their going would make room, up to DrainGrace.

A drained replica was also kept until its model had enough live replicas,
and a replica being restarted that still answers did not count. So with
replicas restarted one after another, each drained one waited out the
five-minute grace, holding its room.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-16 21:33:20 +02:00
1 parent 8164260e3b
commit b8274d6278
5 files changed
+116 -18

No files matched your search

+15 -17
View File
@@ -19,8 +19,8 @@ Conventions this codebase keeps, which every item here must keep too:
What is left, most needed first. The detail is in the item each names.
1. **J1** — `ctl restart --model` waits for ever on a node whose cores are
all reserved. A bug, met while testing v0.14.7. (S)
1. ~~**J1**~~ — done 2026-10-16: a restart no longer refused for room
another restart was giving back.
2. **J5** — an upgrade leaves an earlier install's judge as it was. A
v0.14.6 install on fewer than eight cores refuses every request until
fixed by hand. (S)
@@ -1556,21 +1556,19 @@ answering 429 makes the whole fallback answer 429.
Three VMs installed as HA from the setup screens, a fourth as a judge node,
a fifth as a single 4-core, 8 GB machine; machines powered off under load.
### J1. A restart that has no room waits for ever (S)
`ctl restart --model M` starts each new replica beside the old one before
stopping it. On a node whose cores (or memory) are all reserved, as a
single machine's are once init divides them, the new one fits nowhere and
the restart waits with the old still running. Met on the 4-core VM: the
judge and the starter kept their old figures until their engines were
killed by hand.
- When the new replica fits nowhere beside the old, replace in place: stop
one old replica, then start its successor, saying so (the model has a
replica fewer meanwhile; with one replica it is down for the load time).
- `ctl restart` says which it is doing before it does it.
- Files: the reconciler's rolling restart (`internal/control/reconciler`),
`internal/cli/ctl_restart.go`.
- Done when: on a node with every core reserved, `ctl restart --model`
brings every replica to its new figures without anything killed by hand.
### J1. A restart refused for room another restart was giving back (S) — done 2026-10-16
Not a wait for ever, as first written: on the 4-core VM the judge's restart
was *refused* ("fits on no node even with this replica stopped") because the
starter, restarted at the same time to reserve fewer cores, still held its
old ones. Once refused, nothing tried again, so the judges kept their old
figures. Two fixes in `internal/control/reconciler`:
- a restart whose replacement fits once other restarts under way have gone
(queued, retiring or draining) waits for them, up to DrainGrace, instead
of being refused (`fitsAfterRestarts`, restart.go);
- a drained replica counts a retiring one that still answers as serving, so
one restart after another no longer leaves the first's drained replica,
and the room it holds, waiting out its grace (reconciler.go).
`TestARestartWaitsForRoomAnotherRestartWillGiveBack` is the 4-core case.
### J2. A judge the leader adds is sized as init's is (S)
When a policy names a judge the catalog lacks, the leader serves it itself