A restart is not refused for room another restart is giving back
CI / test (push) Successful in 8m51s
CI / test (push) Successful in 8m51s
On a 4-core single-machine install, `ctl model resources` was run on the judge (to two cores) and on the starter (to none), then both restarted. The judge's restart was refused: its replacement fit on no node even with its old replica stopped, because the starter still held its two cores. A restart now waits for other restarts under way (queued, retiring or draining) when their going would make room, up to DrainGrace. A drained replica was also kept until its model had enough live replicas, and a replica being restarted that still answers did not count. So with replicas restarted one after another, each drained one waited out the five-minute grace, holding its room. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
8164260e3b
commit
b8274d6278
5 files changed
+116
-18
No files matched your search
+15
-17
@@ -19,8 +19,8 @@ Conventions this codebase keeps, which every item here must keep too:
|
||||
|
||||
What is left, most needed first. The detail is in the item each names.
|
||||
|
||||
1. **J1** — `ctl restart --model` waits for ever on a node whose cores are
|
||||
all reserved. A bug, met while testing v0.14.7. (S)
|
||||
1. ~~**J1**~~ — done 2026-10-16: a restart no longer refused for room
|
||||
another restart was giving back.
|
||||
2. **J5** — an upgrade leaves an earlier install's judge as it was. A
|
||||
v0.14.6 install on fewer than eight cores refuses every request until
|
||||
fixed by hand. (S)
|
||||
@@ -1556,21 +1556,19 @@ answering 429 makes the whole fallback answer 429.
|
||||
Three VMs installed as HA from the setup screens, a fourth as a judge node,
|
||||
a fifth as a single 4-core, 8 GB machine; machines powered off under load.
|
||||
|
||||
### J1. A restart that has no room waits for ever (S)
|
||||
`ctl restart --model M` starts each new replica beside the old one before
|
||||
stopping it. On a node whose cores (or memory) are all reserved, as a
|
||||
single machine's are once init divides them, the new one fits nowhere and
|
||||
the restart waits with the old still running. Met on the 4-core VM: the
|
||||
judge and the starter kept their old figures until their engines were
|
||||
killed by hand.
|
||||
- When the new replica fits nowhere beside the old, replace in place: stop
|
||||
one old replica, then start its successor, saying so (the model has a
|
||||
replica fewer meanwhile; with one replica it is down for the load time).
|
||||
- `ctl restart` says which it is doing before it does it.
|
||||
- Files: the reconciler's rolling restart (`internal/control/reconciler`),
|
||||
`internal/cli/ctl_restart.go`.
|
||||
- Done when: on a node with every core reserved, `ctl restart --model`
|
||||
brings every replica to its new figures without anything killed by hand.
|
||||
### J1. A restart refused for room another restart was giving back (S) — done 2026-10-16
|
||||
Not a wait for ever, as first written: on the 4-core VM the judge's restart
|
||||
was *refused* ("fits on no node even with this replica stopped") because the
|
||||
starter, restarted at the same time to reserve fewer cores, still held its
|
||||
old ones. Once refused, nothing tried again, so the judges kept their old
|
||||
figures. Two fixes in `internal/control/reconciler`:
|
||||
- a restart whose replacement fits once other restarts under way have gone
|
||||
(queued, retiring or draining) waits for them, up to DrainGrace, instead
|
||||
of being refused (`fitsAfterRestarts`, restart.go);
|
||||
- a drained replica counts a retiring one that still answers as serving, so
|
||||
one restart after another no longer leaves the first's drained replica,
|
||||
and the room it holds, waiting out its grace (reconciler.go).
|
||||
`TestARestartWaitsForRoomAnotherRestartWillGiveBack` is the 4-core case.
|
||||
|
||||
### J2. A judge the leader adds is sized as init's is (S)
|
||||
When a policy names a judge the catalog lacks, the leader serves it itself
|
||||
|
||||
Reference in new issue
Block a user