Ilker ManapandClaude Opus 5.5
ad6d2d348d
There was no way to restart a running replica. A running engine picks up an engine key, an edited profile or a new engine version only when it starts, so after v0.10.3 the nine replicas already running stayed without a key — reachable from any node — with nothing to do about it short of draining nodes, which on a small CPU fleet has nowhere to move a model to. `llmgw ctl restart REPLICA... | --model M | --all` queues replicas to be replaced. A replica being replaced keeps serving but stops counting toward its model, so the reconciler places a replacement beside it; once that is ready the old one is drained and stopped as a node drain would. Replicas of one model go one at a time, so a model never needs the memory of every replica twice. Where no second copy fits anywhere, the old replica is stopped after the drain grace period and the new one starts in its place, and the log says so, since that is the one case with an outage. `ctl replicas` shows "(restart queued)" and "(restart replacing)" while it happens. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
49 lines
1.8 KiB
Go
49 lines
1.8 KiB
Go
package cli
|
|
|
|
import (
|
|
"encoding/json"
|
|
"errors"
|
|
"fmt"
|
|
|
|
"github.com/spf13/cobra"
|
|
|
|
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
|
)
|
|
|
|
func newRestartCmd(client func() *ctlClient) *cobra.Command {
|
|
var (
|
|
models []string
|
|
all bool
|
|
)
|
|
c := &cobra.Command{
|
|
Use: "restart [REPLICA...]",
|
|
Short: "Replace replicas with new ones, starting each replacement before stopping the old",
|
|
Long: "How a running engine picks up what only applies at start: an engine key,\n" +
|
|
"an edited profile, a new engine version.\n\n" +
|
|
"Each replacement is placed beside the replica it replaces, and the old one\n" +
|
|
"keeps serving until the new one is ready. Replicas of one model are\n" +
|
|
"replaced one at a time. Where nothing else has room for a second copy —\n" +
|
|
"the usual case on a small CPU fleet — the old one is stopped after the\n" +
|
|
"drain grace period (5 minutes by default) and the new one starts in its\n" +
|
|
"place; a model with only that replica is unserved while it loads.\n\n" +
|
|
" llmgw ctl restart qwen2.5-7b-3f2a91c0\n" +
|
|
" llmgw ctl restart --model qwen2.5-7b\n" +
|
|
" llmgw ctl restart --all",
|
|
RunE: func(cmd *cobra.Command, args []string) error {
|
|
if len(args) == 0 && len(models) == 0 && !all {
|
|
return errors.New("name replicas, --model, or --all")
|
|
}
|
|
body, _ := json.Marshal(adminapi.RestartRequest{Replicas: args, Models: models, All: all})
|
|
var res adminapi.RestartResult
|
|
if err := client().post(cmd.Context(), "/api/v1/restart", body, &res); err != nil {
|
|
return err
|
|
}
|
|
fmt.Printf("%d replica(s) queued for restart; watch with: llmgw ctl replicas\n", len(res.Queued))
|
|
return nil
|
|
},
|
|
}
|
|
c.Flags().StringArrayVar(&models, "model", nil, "every replica of this model; repeatable")
|
|
c.Flags().BoolVar(&all, "all", false, "every ready replica in the cluster")
|
|
return c
|
|
}
|