Ilker ManapandClaude Opus 5
3aa250428a
`ctl bench` has always been able to work out what one replica can serve — it ramps concurrency until the SLO breaks and reports the last level that held. It printed that and nothing kept it, so the one number saying what a replica is worth lived in whoever ran it's scrollback. Everything downstream then had to guess: admission applies a single global default for healthy queue depth, and sizing a model against real demand had nothing to size against. `ctl bench --save` records it; `ctl capacity` reads it back. Keyed by model and profile. Capacity belongs to the hardware and the engine arguments as much as to the weights, so the same model on CPU and on an H100 are two numbers, and one key would overwrite the other on every run. The profile is looked up from the replica rather than asked for, because the control plane already knows which profile answers on which endpoint and a measurement filed under the wrong one is worse than none. A target that is not a replica — the gateway, usually — is refused with that said: benchmarking the whole path is worth doing and says nothing about the hardware. The conclusion is stored, not the ramp. Every level and its latencies are worth reading once while judging whether the run was sound; keeping them would grow without bound as a model is re-benchmarked. Re-measuring replaces. A second run is a better answer to the same question, not another data point, and keeping both would leave a reader choosing between numbers with no basis for the choice. The SLO and the date are stored with it. Capacity means nothing without the target it was measured against, and hardware and engine builds change — a measurement is an observation with a date on it, not a fact about the model. A failed save is reported and not fatal: the run has already cost the GPU time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
33 lines
1.6 KiB
SQL
33 lines
1.6 KiB
SQL
-- What one replica of a model can actually serve, measured rather than guessed.
|
|
--
|
|
-- `llmgw ctl bench` has always been able to work this out — it ramps
|
|
-- concurrency until the SLO breaks and reports the last level that held. Until
|
|
-- now it printed that to a terminal and nothing kept it, so the one number that
|
|
-- says what a replica is worth lived in somebody's scrollback. Every decision
|
|
-- downstream then had to guess: admission uses a single global default for
|
|
-- healthy queue depth, and anything that wanted to size a model against real
|
|
-- demand had nothing to size against.
|
|
--
|
|
-- Keyed by model and profile, not by model alone. Capacity is a property of the
|
|
-- hardware and the engine arguments as much as of the weights: the same model
|
|
-- on a CPU profile and on an H100 profile are two different numbers, and
|
|
-- storing one would overwrite the other every time somebody benchmarked the
|
|
-- other kind of node.
|
|
--
|
|
-- The measurement is JSON for the same reason a model definition is: it is a
|
|
-- nested shape that is still growing, and as columns every field added to it
|
|
-- would be a migration.
|
|
--
|
|
-- Nothing here is authoritative about what a model may do — it is an
|
|
-- observation, with the time and the target it was taken from, so a reader can
|
|
-- decide whether it is still true. Hardware changes and engine builds change;
|
|
-- a capacity profile from six months ago is a hypothesis, not a fact.
|
|
CREATE TABLE capacity (
|
|
model TEXT NOT NULL,
|
|
profile TEXT NOT NULL,
|
|
measurement TEXT NOT NULL,
|
|
measured_at TIMESTAMP NOT NULL,
|
|
measured_by TEXT NOT NULL DEFAULT '',
|
|
PRIMARY KEY (model, profile)
|
|
);
|