Files
llmgw/internal/store/capacity.go
T
Ilker ManapandClaude Opus 5 3aa250428a Keep what a benchmark found out, instead of leaving it in a terminal
`ctl bench` has always been able to work out what one replica can serve — it
ramps concurrency until the SLO breaks and reports the last level that held.
It printed that and nothing kept it, so the one number saying what a replica is
worth lived in whoever ran it's scrollback. Everything downstream then had to
guess: admission applies a single global default for healthy queue depth, and
sizing a model against real demand had nothing to size against.

`ctl bench --save` records it; `ctl capacity` reads it back.

Keyed by model and profile. Capacity belongs to the hardware and the engine
arguments as much as to the weights, so the same model on CPU and on an H100
are two numbers, and one key would overwrite the other on every run.

The profile is looked up from the replica rather than asked for, because the
control plane already knows which profile answers on which endpoint and a
measurement filed under the wrong one is worse than none. A target that is not
a replica — the gateway, usually — is refused with that said: benchmarking the
whole path is worth doing and says nothing about the hardware.

The conclusion is stored, not the ramp. Every level and its latencies are worth
reading once while judging whether the run was sound; keeping them would grow
without bound as a model is re-benchmarked.

Re-measuring replaces. A second run is a better answer to the same question,
not another data point, and keeping both would leave a reader choosing between
numbers with no basis for the choice.

The SLO and the date are stored with it. Capacity means nothing without the
target it was measured against, and hardware and engine builds change — a
measurement is an observation with a date on it, not a fact about the model.

A failed save is reported and not fatal: the run has already cost the GPU time.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 11:51:58 +02:00

89 lines
3.0 KiB
Go

package store
import (
"context"
"encoding/json"
"fmt"
"time"
)
// StoredCapacity is a benchmark's conclusion, kept for whoever has to size the
// model later.
type StoredCapacity struct {
Model string
Profile string
// Measurement is bench.Capacity as JSON. Stored whole for the reason a
// model definition is: a measurement is a nested shape that is still
// growing, and as columns every field added would be a migration.
Measurement []byte
MeasuredAt time.Time
MeasuredBy string
}
// PutCapacity records what a replica of a model was measured to serve.
//
// Replacing rather than accumulating. A second measurement of the same model on
// the same profile is a better answer to the same question, not another data
// point: hardware and engine builds change, and keeping every run would leave a
// reader choosing between numbers with no basis for the choice. The run itself
// is printed when it happens, for anyone who wants the ramp.
func (s *SQLite) PutCapacity(ctx context.Context, c StoredCapacity) error {
if c.Model == "" || c.Profile == "" {
return fmt.Errorf("a capacity measurement needs a model and a profile")
}
if !json.Valid(c.Measurement) {
return fmt.Errorf("capacity for %s/%s: the measurement is not valid JSON", c.Model, c.Profile)
}
if c.MeasuredAt.IsZero() {
c.MeasuredAt = time.Now()
}
_, err := s.db.ExecContext(ctx,
`INSERT INTO capacity (model, profile, measurement, measured_at, measured_by)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(model, profile) DO UPDATE SET
measurement = excluded.measurement,
measured_at = excluded.measured_at,
measured_by = excluded.measured_by`,
c.Model, c.Profile, string(c.Measurement), c.MeasuredAt, c.MeasuredBy)
return err
}
// Capacities lists every measurement, newest first.
//
// Newest first because the question a reader usually has is "what do we know
// lately", and an old measurement is the one most likely to be describing
// hardware that has since changed.
func (s *SQLite) Capacities(ctx context.Context) ([]StoredCapacity, error) {
rows, err := s.db.QueryContext(ctx,
`SELECT model, profile, measurement, measured_at, measured_by
FROM capacity ORDER BY measured_at DESC, model, profile`)
if err != nil {
return nil, err
}
defer rows.Close()
var out []StoredCapacity
for rows.Next() {
var c StoredCapacity
var m string
if err := rows.Scan(&c.Model, &c.Profile, &m, &c.MeasuredAt, &c.MeasuredBy); err != nil {
return nil, err
}
c.Measurement = []byte(m)
out = append(out, c)
}
return out, rows.Err()
}
// DeleteCapacity forgets a measurement.
//
// Wanted when a model is removed, and wanted on its own: a profile whose
// hardware has been replaced is worse than no measurement, because it looks
// like knowledge. Deleting one that is not there is not an error — the caller
// wanted it gone, and it is.
func (s *SQLite) DeleteCapacity(ctx context.Context, model, profile string) error {
_, err := s.db.ExecContext(ctx,
`DELETE FROM capacity WHERE model = ? AND profile = ?`, model, profile)
return err
}