Change a model's replica count without editing a file on the control plane

The count is scaling.min, and until now the only way to change it was to edit
server.yaml on the host and restart — or, for a model added at runtime, to
remove it and add it back, which means having its whole definition to hand.

`llmgw ctl scale <model> --min N` instead. Only the scaling block moves:
everything else is read back from what was stored and written out untouched, so
changing a number cannot quietly become a redefinition.

The catalog is rebuilt and validated before anything is written, as in Add. A
max below a min is refused with the stored definition untouched — persisting a
rejected change would come back on the next restart as a catalog that does not
build.

The answer reports the scaling in force rather than the request. The catalog
fills in a max that was left out, and a caller echoing its own request would
print a number the cluster is not using.

A model declared in configuration is refused, for the reason Remove already
refuses one: the YAML is read on every start, so the new count would hold until
the next restart and then silently revert. A cluster disagreeing with its own
config file is worse than an error saying where to edit.

The help says what --max does, because it is not what it looks like: a model
with a minimum runs exactly that many and nothing scales it up under load, and
an on-demand model loads one replica however high its max.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-27 11:25:05 +02:00
1 parent 10e323132c
commit 707d774c76
6 files changed
+388

No files matched your search

+67
View File
@@ -164,6 +164,73 @@ func (s *Service) Remove(ctx context.Context, name string) error {
return nil
}
// Scale changes how many replicas a model runs, without a restart.
//
// Only the scaling block moves. Everything else about the model — its source,
// its profiles, the engine arguments somebody tuned — is read back from what
// was stored and written out again untouched, so changing a number cannot
// quietly become a redefinition.
//
// A model declared in configuration is refused, for the same reason Remove
// refuses one: the YAML is read on every start, so the new number would hold
// until the next restart and then silently revert. A cluster that disagrees
// with its own config file is worse than an error message saying where to edit.
func (s *Service) Scale(ctx context.Context, name string, sc catalog.Scaling, by string) (catalog.Model, error) {
s.mu.Lock()
defer s.mu.Unlock()
if s.Declared(name) {
return catalog.Model{}, fmt.Errorf(
"model %q is declared in configuration; change its scaling there, or it reverts on the next restart", name)
}
stored, err := s.storedModels(ctx)
if err != nil {
return catalog.Model{}, err
}
next := append([]catalog.Model{}, s.declared...)
found := false
for _, e := range stored {
if e.Name == name {
e.Scaling = sc
found = true
}
next = append(next, e)
}
if !found {
return catalog.Model{}, fmt.Errorf("model %q was not added at runtime", name)
}
// Built before anything is written, like Add: catalog.New is what applies
// the defaults (max follows min when it is left out) and what refuses a
// max below a min, so the model that comes back is the one that will run
// rather than the one that was asked for.
built, err := catalog.New(next)
if err != nil {
return catalog.Model{}, fmt.Errorf("scaling %q that way would make the catalog invalid: %w", name, err)
}
final, ok := built.Resolve(name)
if !ok {
return catalog.Model{}, fmt.Errorf("model %q vanished from the catalog it was just built into", name)
}
def, err := json.Marshal(final)
if err != nil {
return catalog.Model{}, err
}
if err := s.store.PutModel(ctx, store.StoredModel{
Name: name, Definition: def, CreatedAt: time.Now(), CreatedBy: by,
}); err != nil {
return catalog.Model{}, err
}
s.apply.ReplaceCatalog(ctx, built)
s.log.Info("model scaled", "model", name,
"min", final.Scaling.Min, "max", final.Scaling.Max, "by", by)
return final, nil
}
// Load builds the catalog at startup: configuration plus what was stored.
//
// A stored model that no longer validates — its engine left the build, a