Let the catalog change while the cluster is running
Adding a model was an operator procedure with downtime: edit a file on the control plane, restart it, wait out the settling window, watch replicas churn while adoption catches up. I did that eight times in a day, and one of the restarts cost two minutes of outage while stalled replicas timed out. None of it was necessary. The reconciler already compares desired against actual on every pass, so a new catalog is simply a different desired. A Catalog stays immutable — New is the only way to build one and it refuses an invalid one — and what changes is which immutable catalog is current. Readers take the pointer and finish their operation against it, so a swap mid-request is invisible rather than a race. The router is told rather than asked, the same shape as the endpoint list and for the same reason: a push happens once per change, a pull would happen once per request. The router still seeds from configuration as well as being pushed to. A router with a static endpoint list and no control plane is a supported mode and has nothing to push it a catalog; without seeding it would resolve no model at all. Which turned up a bug that was latent while catalogs were fixed. reconcileModel only ever visits models that are in the catalog, so a replica whose model had been removed was visited by nothing: it kept its GPUs, kept counting, and kept running an engine for a name no client could ask for. The endpoint publisher skipped such a replica with the comment "the reconciler will tear it down", which was not true. It is now, and the comment says what actually happens instead of what was assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
9d8cd5e284
commit
1c57a4e378
10 files changed
+261
-34
No files matched your search
@@ -25,6 +25,17 @@ type EndpointSink interface {
|
||||
SetEndpoints(ctx context.Context, infos []endpoint.Info)
|
||||
}
|
||||
|
||||
// CatalogSink receives catalog changes. The router implements it.
|
||||
//
|
||||
// The catalog used to be fixed for the life of the process, so the router could
|
||||
// derive its view once and keep it. Now that a model can be added while the
|
||||
// cluster runs, the router has to be told — and it is told rather than asked,
|
||||
// because a push happens once per change and a pull would happen once per
|
||||
// request.
|
||||
type CatalogSink interface {
|
||||
SetCatalog(c *catalog.Catalog)
|
||||
}
|
||||
|
||||
// UsageSource reports when each endpoint was last sent a request.
|
||||
//
|
||||
// The return path of EndpointSink, and an interface for the same reason: the
|
||||
@@ -62,7 +73,7 @@ type Controller struct {
|
||||
cfg Config
|
||||
log *slog.Logger
|
||||
inv *inventory.Store
|
||||
cat *catalog.Catalog
|
||||
cat *catalog.Live
|
||||
rec *reconciler.Reconciler
|
||||
sink EndpointSink
|
||||
usage UsageSource
|
||||
@@ -76,7 +87,7 @@ type Controller struct {
|
||||
}
|
||||
|
||||
// New builds a controller.
|
||||
func New(cfg Config, cat *catalog.Catalog, disp reconciler.Dispatcher, sink EndpointSink, log *slog.Logger) *Controller {
|
||||
func New(cfg Config, cat *catalog.Live, disp reconciler.Dispatcher, sink EndpointSink, log *slog.Logger) *Controller {
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
@@ -118,11 +129,40 @@ func (c *Controller) collectUsage() {
|
||||
c.rec.RecordUsage(c.usage.LastUsed())
|
||||
}
|
||||
|
||||
// ReplaceCatalog puts a new catalog in force, without a restart.
|
||||
//
|
||||
// The whole point of the change: adding a model was an operator procedure with
|
||||
// downtime — edit a file on the control plane, restart, wait out the settling
|
||||
// window, watch adoption churn. Nothing about that was necessary. The
|
||||
// reconciler already compares desired against actual every pass, so a new
|
||||
// catalog is just a different desired, and it is picked up on the next tick
|
||||
// without anything being torn down.
|
||||
//
|
||||
// The catalog is validated by its own constructor, so an invalid one cannot
|
||||
// reach here; what can reach here is a valid catalog that removes a model
|
||||
// something is still serving, and that is handled the way it always has been —
|
||||
// the reconciler stops replicas whose model is gone, and Publish drops
|
||||
// endpoints for them first, so traffic stops being routed before anything is
|
||||
// killed.
|
||||
func (c *Controller) ReplaceCatalog(ctx context.Context, next *catalog.Catalog) {
|
||||
c.cat.Store(next)
|
||||
if cs, ok := c.sink.(CatalogSink); ok {
|
||||
cs.SetCatalog(next)
|
||||
}
|
||||
// Endpoints first, then wake the loop: a model that just left the catalog
|
||||
// should stop receiving traffic before its replicas are stopped, and
|
||||
// Publish is what takes it out of the router's list.
|
||||
c.Publish(ctx)
|
||||
c.rec.Notify()
|
||||
|
||||
c.log.Info("catalog replaced", "models", len(next.Models()))
|
||||
}
|
||||
|
||||
// Reconciler exposes the reconciliation loop, for status reports and ctl.
|
||||
func (c *Controller) Reconciler() *reconciler.Reconciler { return c.rec }
|
||||
|
||||
// Catalog exposes the model catalog.
|
||||
func (c *Controller) Catalog() *catalog.Catalog { return c.cat }
|
||||
func (c *Controller) Catalog() *catalog.Catalog { return c.cat.Load() }
|
||||
|
||||
// Start runs the reconciler and begins publishing endpoints.
|
||||
func (c *Controller) Start(ctx context.Context) {
|
||||
@@ -187,10 +227,11 @@ func (c *Controller) endpoints() []endpoint.Info {
|
||||
out := make([]endpoint.Info, 0, len(ready))
|
||||
|
||||
for _, rep := range ready {
|
||||
m, ok := c.cat.Resolve(rep.Model)
|
||||
m, ok := c.cat.Load().Resolve(rep.Model)
|
||||
if !ok {
|
||||
// The model left the catalog while a replica was still running.
|
||||
// The reconciler will tear it down; until then, do not route to it.
|
||||
// Withdraw it here first, so traffic stops before the reconciler
|
||||
// stops the engine on its next pass.
|
||||
continue
|
||||
}
|
||||
// Clients may address a model by any of its names, and the router
|
||||
|
||||
Reference in new issue
Block a user