`ctl bench` has always been able to work out what one replica can serve — it
ramps concurrency until the SLO breaks and reports the last level that held.
It printed that and nothing kept it, so the one number saying what a replica is
worth lived in whoever ran it's scrollback. Everything downstream then had to
guess: admission applies a single global default for healthy queue depth, and
sizing a model against real demand had nothing to size against.
`ctl bench --save` records it; `ctl capacity` reads it back.
Keyed by model and profile. Capacity belongs to the hardware and the engine
arguments as much as to the weights, so the same model on CPU and on an H100
are two numbers, and one key would overwrite the other on every run.
The profile is looked up from the replica rather than asked for, because the
control plane already knows which profile answers on which endpoint and a
measurement filed under the wrong one is worse than none. A target that is not
a replica — the gateway, usually — is refused with that said: benchmarking the
whole path is worth doing and says nothing about the hardware.
The conclusion is stored, not the ramp. Every level and its latencies are worth
reading once while judging whether the run was sound; keeping them would grow
without bound as a model is re-benchmarked.
Re-measuring replaces. A second run is a better answer to the same question,
not another data point, and keeping both would leave a reader choosing between
numbers with no basis for the choice.
The SLO and the date are stored with it. Capacity means nothing without the
target it was measured against, and hardware and engine builds change — a
measurement is an observation with a date on it, not a fact about the model.
A failed save is reported and not fatal: the run has already cost the GPU time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The choice is unchanged and now better argued: a deployment does not announce
which language its notes are in, so any stemmer picked here is the wrong one
somewhere — and applying the wrong one does not fail loudly, it quietly stops
finding things.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first thing HA needs that a shared store does not give. Two control planes
against one store both reconcile: both see a model wanting two replicas and
none running, and both place. The model ends up with twice what it asked for,
each replica holding GPUs the other's inventory believes are free — and nothing
crashes to say so.
A lease in a table rather than a lock held by a connection. PostgreSQL has
advisory locks and SQLite has nothing like them, and this store keeps one set
of queries for both; a row with an expiry works everywhere, and it survives a
leader being powered off rather than asked to stop, which is the case that
matters. Taking and renewing are the same statement, so the two cannot disagree
about what a held lease looks like, and it is one statement, so two processes
racing cannot both believe they won.
The owner id is random rather than the hostname. A hostname would let a
restarted process match its own stale lease and believe it had been leading all
along — including through the gap that made it restart, during which somebody
else may have taken over and started placing.
A store that cannot be reached is not evidence that somebody else is leading,
so an unreachable store keeps the claim. Two leaders is worse than one, but no
leader means nothing is placed and nothing is evicted, and a brief database
blip should not stop a cluster healing itself.
Unset means leader. Every deployment that existed before this, every
single-control-plane install and every test acts unconditionally, with nothing
to configure.
The same seven assertions run against both backends, which is how the test
harness was caught inheriting a lease between subtests — it passed on SQLite,
where each subtest gets a fresh file, and failed only on PostgreSQL.
What this does not yet do is move agents. A standby holding an agent's stream
is one the leader cannot command, so failover is degraded rather than clean
until agents know more than one address.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A path is still SQLite and a postgres:// URL is PostgreSQL, so every existing
configuration keeps working untouched and moving is one line changed rather
than a new setting to find. An address of the form @/path/to/file reads the
real one out of that file, which is how a DSN carrying a password stays out of
a configuration that gets copied, diffed and pasted into issues.
The driver is pgx, which is pure Go. That is not a preference: the whole binary
is built with CGO_ENABLED=0 so it runs on a node whose libc is older than the
builder's, and a cgo driver would take that away from every deployment,
including the ones still on SQLite.
Tested against a real PostgreSQL, which found two things nothing else could.
The schema_migrations table is created in Go rather than in a migration file,
so it escaped the port and still said INTEGER — 64 bits in SQLite, 32 in
PostgreSQL, and a millisecond timestamp does not fit, so the very first
migration failed to record that it had run. And the memory migration uses an
FTS5 virtual table with three triggers keeping it in step, which has no
counterpart: PostgreSQL derives a tsvector from the row, so there is nothing
to keep in step and no trigger to forget.
That leaves exactly one query that is not the same sentence in both dialects —
the note search, MATCH against @@ — and it is written out twice rather than
hidden behind a shared spelling that would mean something subtly different in
each. Everything else is one set of queries.
The text search uses the 'simple' configuration rather than English. These are
an agent's own notes, in whatever language it chose, and stemming English
rules over Turkish would quietly stop finding things.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
