Ilker ManapandClaude Opus 5
3aa250428a
`ctl bench` has always been able to work out what one replica can serve — it ramps concurrency until the SLO breaks and reports the last level that held. It printed that and nothing kept it, so the one number saying what a replica is worth lived in whoever ran it's scrollback. Everything downstream then had to guess: admission applies a single global default for healthy queue depth, and sizing a model against real demand had nothing to size against. `ctl bench --save` records it; `ctl capacity` reads it back. Keyed by model and profile. Capacity belongs to the hardware and the engine arguments as much as to the weights, so the same model on CPU and on an H100 are two numbers, and one key would overwrite the other on every run. The profile is looked up from the replica rather than asked for, because the control plane already knows which profile answers on which endpoint and a measurement filed under the wrong one is worse than none. A target that is not a replica — the gateway, usually — is refused with that said: benchmarking the whole path is worth doing and says nothing about the hardware. The conclusion is stored, not the ramp. Every level and its latencies are worth reading once while judging whether the run was sound; keeping them would grow without bound as a model is re-benchmarked. Re-measuring replaces. A second run is a better answer to the same question, not another data point, and keeping both would leave a reader choosing between numbers with no basis for the choice. The SLO and the date are stored with it. Capacity means nothing without the target it was measured against, and hardware and engine builds change — a measurement is an observation with a date on it, not a fact about the model. A failed save is reported and not fatal: the run has already cost the GPU time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
78 lines
3.6 KiB
Go
78 lines
3.6 KiB
Go
package bench
|
|
|
|
import "time"
|
|
|
|
// Capacity is what one replica was measured to serve, kept so the measurement
|
|
// outlives the terminal it was printed in.
|
|
//
|
|
// The conclusion of a run, not the run. The ramp itself — every concurrency
|
|
// level, its latencies, its errors — is worth reading once, while deciding
|
|
// whether the run was sound; it is not worth carrying around afterwards, and
|
|
// storing it would grow without bound as a model is re-benchmarked. What a
|
|
// later reader needs is the number and enough provenance to judge whether it is
|
|
// still true.
|
|
type Capacity struct {
|
|
Model string `json:"model"`
|
|
Profile string `json:"profile"`
|
|
|
|
// MaxConcurrencyAtSLO is the capacity: the highest concurrency that still
|
|
// met the target. Zero means even one request did not, which is a real
|
|
// result and not a missing one — it says this hardware cannot serve this
|
|
// model at this SLO at all.
|
|
MaxConcurrencyAtSLO int `json:"max_concurrency_at_slo"`
|
|
// ThroughputTokPerSec and the latencies are taken at that level, not at the
|
|
// peak. Peak throughput arrives long after latency has become unusable, so
|
|
// reporting it would overstate what the hardware can serve.
|
|
ThroughputTokPerSec float64 `json:"throughput_tok_per_sec"`
|
|
TTFTp99 time.Duration `json:"ttft_p99"`
|
|
TPOTp99 time.Duration `json:"tpot_p99"`
|
|
|
|
// SuggestedTargetWaiting and SuggestedKVCeiling are the queue depth and
|
|
// cache pressure seen while the SLO was still being met, which is what
|
|
// admission should treat as healthy for this model. Admission currently
|
|
// applies one global default to every model; these are the per-model
|
|
// numbers it should be using instead.
|
|
SuggestedTargetWaiting int `json:"suggested_target_waiting"`
|
|
SuggestedKVCeiling float64 `json:"suggested_kv_ceiling"`
|
|
|
|
// SLO is recorded because capacity is meaningless without it. The same
|
|
// replica has a different capacity against a 200ms first-token target than
|
|
// against a 2s one, and a number stored without the target it was measured
|
|
// against cannot be compared with anything.
|
|
SLO SLO `json:"slo"`
|
|
|
|
// Target is the endpoint the load actually went to, kept so a profile
|
|
// measured against the gateway is never mistaken for one measured against
|
|
// a replica. Only the second says anything about the hardware.
|
|
Target string `json:"target"`
|
|
MeasuredAt time.Time `json:"measured_at"`
|
|
// Stale is not a field. Whether a measurement still holds depends on
|
|
// whether the hardware and the engine build have changed since, which this
|
|
// package cannot know; MeasuredAt is given so the reader can decide.
|
|
}
|
|
|
|
// Capacity summarizes a completed run for storage.
|
|
func (r Result) Capacity(profile string) Capacity {
|
|
return Capacity{
|
|
Model: r.Model,
|
|
Profile: profile,
|
|
MaxConcurrencyAtSLO: r.MaxConcurrencyAtSLO,
|
|
ThroughputTokPerSec: r.ThroughputTokPerSec,
|
|
TTFTp99: r.TTFTp99,
|
|
TPOTp99: r.TPOTp99,
|
|
SuggestedTargetWaiting: r.SuggestedTargetWaiting,
|
|
SuggestedKVCeiling: r.SuggestedKVCeiling,
|
|
SLO: r.SLO,
|
|
Target: r.Target,
|
|
MeasuredAt: r.RanAt,
|
|
}
|
|
}
|
|
|
|
// Usable reports whether this measurement says anything.
|
|
//
|
|
// A run that never met the SLO, or one stopped before it finished a level,
|
|
// concludes at zero. That is worth recording — it is the answer to "can this
|
|
// run here at all" — but a caller sizing a model against it would compute a
|
|
// replica count of infinity, so it has to be asked about rather than used.
|
|
func (c Capacity) Usable() bool { return c.MaxConcurrencyAtSLO > 0 }
|