Phase 3: ctl, and make draining a rolling move
ctl talks to a management API on the admin listener, not the client-facing one. The two have different audiences: one serves inference to whoever holds an API key, the other lets a caller cordon nodes and read the cluster's shape. - nodes surfaces unhealthy GPUs and why they are excluded. That is usually the answer when a node looks idle but nothing will schedule onto it, and it was invisible before. - explain-placement prints every candidate considered and why it was rejected. - replicas shows the shards a replica is made of, with per-shard errors. - cordon, uncordon and drain are one endpoint with different targets, because the difference between cordon and drain is exactly what an operator needs to be sure of before running either. Draining had to be fixed first to make ctl drain worth having. A drained replica was stopped on the next pass, which cuts in-flight requests — the thing draining exists to avoid. Now a node marked Draining has its replicas drained, the model falls below its desired count, a replacement is placed elsewhere, and the original is stopped only once the replacement is serving. A grace period bounds it, because draining the only node that can host a model would otherwise never finish and the node could never be taken out of service. Verified against a two-node cluster of real binaries: draining moved a replica to the surviving node and traffic kept returning 200 throughout. ctl logs is deliberately absent. It needs a request/response pair the agent protocol does not have, and the failure path already ships an engine log tail with every status report — so the case that matters most is covered, and reading a healthy shard's output on demand is not worth a protocol change yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
1d1a6aebb0
commit
cd225c2bf0
12 files changed
+1143
-42
No files matched your search
+35
-1
@@ -16,7 +16,41 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased — phases 2 and 3 in progress
|
||||
## v0.5.0 — 2026-09-16
|
||||
|
||||
Phases 2 and 3. The cluster runs itself: agents report their hardware, the
|
||||
control plane places replicas, and `ctl` makes all of it visible.
|
||||
|
||||
This release covers both phases because their exit criteria depended on the same
|
||||
piece — the agent stream — and neither was demonstrable without it.
|
||||
|
||||
### ctl
|
||||
|
||||
```
|
||||
llmgw ctl nodes | models | replicas | explain-placement | cordon | uncordon | drain
|
||||
```
|
||||
|
||||
- `nodes` surfaces unhealthy GPUs and why they are excluded, which is usually
|
||||
the answer when a node looks idle but nothing schedules onto it.
|
||||
- `explain-placement` prints every candidate considered and the reason it was
|
||||
rejected. "Unschedulable" on its own says nothing about which of six
|
||||
constraints bit.
|
||||
- `drain` is a rolling move, not a stop: the replica is drained, a replacement
|
||||
is placed elsewhere, and the original is stopped only once the replacement is
|
||||
serving. Bounded by a grace period, because draining the only node that can
|
||||
host a model would otherwise never finish and leave it undrainable.
|
||||
- `cordon` is the weaker operation: no new placement, nothing disturbed.
|
||||
|
||||
The management API is on the admin listener, not the client-facing one. The two
|
||||
have different audiences: one serves inference to whoever holds an API key, the
|
||||
other lets a caller cordon nodes.
|
||||
|
||||
`ctl logs` is not here. It needs a request/response pair the agent protocol does
|
||||
not have. The failure path already ships an engine log tail with every status
|
||||
report, so the case that matters most is covered; reading a healthy shard's
|
||||
output on demand is not.
|
||||
|
||||
## Unreleased — phase 4 in progress
|
||||
|
||||
Agent groundwork. Everything here works without GPUs and is covered by tests;
|
||||
what is missing is the control plane connection, so there is no release yet.
|
||||
|
||||
@@ -15,8 +15,8 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
||||
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
||||
| v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | released |
|
||||
| v0.3.0 | 1.5 | Admission control (priority bands) | released |
|
||||
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | planned |
|
||||
| v0.5.0 | 3 | Control plane: catalog, scheduler, reconciler | planned |
|
||||
| v0.4.0 | 2 | Agent: GPU/IB discovery, engine supervision | released |
|
||||
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler, ctl | released |
|
||||
| v0.6.0 | 4 | Multi-node placement and fabric awareness | planned |
|
||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||
|
||||
@@ -33,6 +33,24 @@ make test # unit tests
|
||||
make race # with the race detector
|
||||
```
|
||||
|
||||
## Operating a cluster
|
||||
|
||||
```sh
|
||||
llmgw ctl nodes # capacity, health, what is running where
|
||||
llmgw ctl models # catalog and replica counts
|
||||
llmgw ctl replicas --model llama # replicas and their shards
|
||||
llmgw ctl explain-placement llama # why a model has not been placed
|
||||
llmgw ctl drain gpu-a-01 # move work off a node, one replica at a time
|
||||
llmgw ctl cordon gpu-a-01 # stop new placement, leave what runs
|
||||
```
|
||||
|
||||
`ctl` talks to the admin port (`:9090` by default), not the client-facing one.
|
||||
Add `--json` to any command for machine-readable output.
|
||||
|
||||
`explain-placement` is the one worth knowing about: "unschedulable" on its own
|
||||
says nothing about which of six constraints bit, so it prints every candidate
|
||||
considered and the reason it was rejected.
|
||||
|
||||
## Running the router
|
||||
|
||||
```sh
|
||||
|
||||
@@ -0,0 +1,336 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
"text/tabwriter"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
// ctlClient talks to a server's admin API.
|
||||
type ctlClient struct {
|
||||
base string
|
||||
http *http.Client
|
||||
}
|
||||
|
||||
func newCtlClient(addr string) *ctlClient {
|
||||
if !strings.HasPrefix(addr, "http://") && !strings.HasPrefix(addr, "https://") {
|
||||
addr = "http://" + addr
|
||||
}
|
||||
return &ctlClient{
|
||||
base: strings.TrimRight(addr, "/"),
|
||||
http: &http.Client{Timeout: 15 * time.Second},
|
||||
}
|
||||
}
|
||||
|
||||
func (c *ctlClient) get(ctx context.Context, path string, out any) error {
|
||||
return c.do(ctx, http.MethodGet, path, nil, out)
|
||||
}
|
||||
|
||||
func (c *ctlClient) do(ctx context.Context, method, path string, body, out any) error {
|
||||
var rdr io.Reader
|
||||
if body != nil {
|
||||
b, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
rdr = bytes.NewReader(b)
|
||||
}
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, method, c.base+path, rdr)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if body != nil {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
|
||||
resp, err := c.http.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("reaching the admin API at %s: %w\n"+
|
||||
"(is the server running, and is --server pointing at its admin_listen port?)", c.base, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
var e struct {
|
||||
Error string `json:"error"`
|
||||
}
|
||||
b, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<12))
|
||||
if json.Unmarshal(b, &e) == nil && e.Error != "" {
|
||||
return fmt.Errorf("%s: %s", resp.Status, e.Error)
|
||||
}
|
||||
return fmt.Errorf("%s: %s", resp.Status, strings.TrimSpace(string(b)))
|
||||
}
|
||||
if out == nil {
|
||||
return nil
|
||||
}
|
||||
return json.NewDecoder(resp.Body).Decode(out)
|
||||
}
|
||||
|
||||
func newCtlCmd() *cobra.Command {
|
||||
var (
|
||||
server string
|
||||
asJSON bool
|
||||
)
|
||||
|
||||
cmd := &cobra.Command{
|
||||
Use: "ctl",
|
||||
Short: "Inspect and operate a running cluster",
|
||||
Long: "Talks to a server's admin API, which listens on admin_listen (:9090 by\n" +
|
||||
"default) rather than the client-facing port.",
|
||||
}
|
||||
cmd.PersistentFlags().StringVar(&server, "server", "127.0.0.1:9090", "admin API address")
|
||||
cmd.PersistentFlags().BoolVar(&asJSON, "json", false, "emit JSON")
|
||||
|
||||
client := func() *ctlClient { return newCtlClient(server) }
|
||||
|
||||
cmd.AddCommand(
|
||||
newNodesCmd(client, &asJSON),
|
||||
newModelsCmd(client, &asJSON),
|
||||
newReplicasCmd(client, &asJSON),
|
||||
newExplainCmd(client, &asJSON),
|
||||
newNodeStateCmd(client, "cordon", "cordoned",
|
||||
"Stop placing new work on a node; what runs there keeps running"),
|
||||
newNodeStateCmd(client, "uncordon", "ready",
|
||||
"Allow placement on a node again"),
|
||||
newNodeStateCmd(client, "drain", "draining",
|
||||
"Move work off a node, one replica at a time"),
|
||||
)
|
||||
return cmd
|
||||
}
|
||||
|
||||
func newNodesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "nodes",
|
||||
Short: "List nodes and their capacity",
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var nodes []adminapi.NodeView
|
||||
if err := client().get(cmd.Context(), "/api/v1/nodes", &nodes); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return emitJSON(nodes)
|
||||
}
|
||||
if len(nodes) == 0 {
|
||||
fmt.Println("No nodes have registered.")
|
||||
return nil
|
||||
}
|
||||
|
||||
w := tab()
|
||||
fmt.Fprintln(w, "NODE\tSTATE\tGPUS\tCLASSES\tFABRICS\tREPLICAS\tLAST SEEN")
|
||||
for _, n := range nodes {
|
||||
fmt.Fprintf(w, "%s\t%s\t%d/%d free\t%s\t%s\t%d\t%s\n",
|
||||
n.ID, n.State, n.GPUFree, n.GPUTotal,
|
||||
join(n.Classes), join(n.Fabrics), n.Replicas, n.LastSeenAgo)
|
||||
}
|
||||
_ = w.Flush()
|
||||
|
||||
// Unhealthy GPUs are usually the answer when a node looks idle but
|
||||
// nothing schedules onto it.
|
||||
for _, n := range nodes {
|
||||
for _, p := range n.Unhealthy {
|
||||
fmt.Printf("\n%s: %s (kept out of placement)", n.ID, p)
|
||||
}
|
||||
}
|
||||
if hasUnhealthy(nodes) {
|
||||
fmt.Println()
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newModelsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "models",
|
||||
Short: "List the catalog and how many replicas each model has",
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var models []adminapi.ModelView
|
||||
if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return emitJSON(models)
|
||||
}
|
||||
if len(models) == 0 {
|
||||
fmt.Println("The catalog is empty.")
|
||||
return nil
|
||||
}
|
||||
|
||||
w := tab()
|
||||
fmt.Fprintln(w, "MODEL\tREPLICAS\tALIASES\tPROFILES\tFALLBACKS")
|
||||
for _, m := range models {
|
||||
replicas := fmt.Sprintf("%d/%d ready", m.Ready, m.Min)
|
||||
if m.Total > m.Ready {
|
||||
replicas += fmt.Sprintf(" (%d total)", m.Total)
|
||||
}
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n",
|
||||
m.Name, replicas, join(m.Aliases), join(m.Profiles), join(m.Fallbacks))
|
||||
}
|
||||
return w.Flush()
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
var model string
|
||||
c := &cobra.Command{
|
||||
Use: "replicas",
|
||||
Short: "List replicas and the shards they are made of",
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
path := "/api/v1/replicas"
|
||||
if model != "" {
|
||||
path += "?model=" + model
|
||||
}
|
||||
var reps []adminapi.ReplicaView
|
||||
if err := client().get(cmd.Context(), path, &reps); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return emitJSON(reps)
|
||||
}
|
||||
if len(reps) == 0 {
|
||||
fmt.Println("No replicas are running.")
|
||||
return nil
|
||||
}
|
||||
|
||||
w := tab()
|
||||
fmt.Fprintln(w, "REPLICA\tMODEL\tSTATE\tPOOL\tNODES\tAGE\tENDPOINT")
|
||||
for _, r := range reps {
|
||||
var nodes []string
|
||||
for _, s := range r.Shards {
|
||||
nodes = append(nodes, fmt.Sprintf("%s:%v", s.Node, s.GPUs))
|
||||
}
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
|
||||
r.ID, r.Model, r.State, r.Pool, join(nodes),
|
||||
(time.Duration(r.AgeSec) * time.Second).String(), r.Endpoint)
|
||||
}
|
||||
_ = w.Flush()
|
||||
|
||||
for _, r := range reps {
|
||||
if r.Err != "" {
|
||||
fmt.Printf("\n%s: %s\n", r.ID, r.Err)
|
||||
}
|
||||
for _, s := range r.Shards {
|
||||
if s.Err != "" {
|
||||
fmt.Printf(" %s on %s: %s\n", s.ID, s.Node, s.Err)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
c.Flags().StringVar(&model, "model", "", "only this model's replicas")
|
||||
return c
|
||||
}
|
||||
|
||||
func newExplainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "explain-placement <model>",
|
||||
Short: "Explain why a model has not been placed",
|
||||
Long: "Prints every candidate placement considered and the reason it was\n" +
|
||||
"rejected. \"Unschedulable\" on its own says nothing about which of six\n" +
|
||||
"constraints bit.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var v adminapi.ExplainView
|
||||
if err := client().get(cmd.Context(),
|
||||
"/api/v1/explain-placement?model="+args[0], &v); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return emitJSON(v)
|
||||
}
|
||||
|
||||
if v.Placed && len(v.Rejections) == 0 {
|
||||
fmt.Printf("%s is placed; nothing was rejected.\n", v.Model)
|
||||
return nil
|
||||
}
|
||||
if len(v.Rejections) == 0 {
|
||||
fmt.Printf("No placement has been attempted for %s yet.\n", v.Model)
|
||||
return nil
|
||||
}
|
||||
|
||||
if v.Placed {
|
||||
fmt.Printf("%s is placed. The candidates rejected along the way:\n\n", v.Model)
|
||||
} else {
|
||||
fmt.Printf("%s could not be placed. Every candidate and why:\n\n", v.Model)
|
||||
}
|
||||
w := tab()
|
||||
fmt.Fprintln(w, "PROFILE\tPOOL\tNODE\tREASON")
|
||||
for _, r := range v.Rejections {
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", r.Profile, dash(r.Pool), dash(r.Node), r.Reason)
|
||||
}
|
||||
return w.Flush()
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newNodeStateCmd(client func() *ctlClient, use, state, short string) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: use + " <node>",
|
||||
Short: short,
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var out map[string]string
|
||||
err := client().do(cmd.Context(), http.MethodPost,
|
||||
"/api/v1/nodes/"+args[0]+"/state",
|
||||
map[string]string{"state": state}, &out)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("%s is now %s\n", out["id"], out["state"])
|
||||
if state == "draining" {
|
||||
fmt.Println("Replicas will be moved off one at a time, each replacement " +
|
||||
"brought up before the original is stopped.")
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// --- output helpers ---
|
||||
|
||||
func tab() *tabwriter.Writer {
|
||||
return tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', 0)
|
||||
}
|
||||
|
||||
func emitJSON(v any) error {
|
||||
enc := json.NewEncoder(os.Stdout)
|
||||
enc.SetIndent("", " ")
|
||||
return enc.Encode(v)
|
||||
}
|
||||
|
||||
func join(v []string) string {
|
||||
if len(v) == 0 {
|
||||
return "-"
|
||||
}
|
||||
return strings.Join(v, ",")
|
||||
}
|
||||
|
||||
func dash(s string) string {
|
||||
if s == "" {
|
||||
return "-"
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func hasUnhealthy(nodes []adminapi.NodeView) bool {
|
||||
for _, n := range nodes {
|
||||
if len(n.Unhealthy) > 0 {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -20,6 +20,7 @@ import (
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
@@ -85,6 +86,7 @@ func newServerCmd() *cobra.Command {
|
||||
// published to the router (plan §6.4).
|
||||
var agentHandler http.Handler
|
||||
var agentPath string
|
||||
var admin *control.Controller
|
||||
if controlMode {
|
||||
cat, err := cfg.Catalog()
|
||||
if err != nil {
|
||||
@@ -96,16 +98,20 @@ func newServerCmd() *cobra.Command {
|
||||
return err
|
||||
}
|
||||
defer ctl.Close()
|
||||
admin = ctl
|
||||
agentPath, agentHandler = svc.Handler()
|
||||
|
||||
log.Info("control plane running",
|
||||
"models", len(cfg.Models), "agent_listen", cfg.AgentListen)
|
||||
}
|
||||
|
||||
adminHandler, err := rt.AdminHandler()
|
||||
adminMux, err := rt.AdminHandler()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if admin != nil {
|
||||
adminapi.New(admin).Register(adminMux)
|
||||
}
|
||||
|
||||
apiSrv := &http.Server{
|
||||
Handler: rt.Handler(),
|
||||
@@ -114,7 +120,7 @@ func newServerCmd() *cobra.Command {
|
||||
// enforces its own per-phase deadlines.
|
||||
}
|
||||
adminSrv := &http.Server{
|
||||
Handler: adminHandler,
|
||||
Handler: adminMux,
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
}
|
||||
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"errors"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
// errNotImplemented marks commands belonging to phases not yet built.
|
||||
var errNotImplemented = errors.New("not implemented yet (see llmgw-plan.md §11)")
|
||||
|
||||
func newCtlCmd() *cobra.Command {
|
||||
cmd := &cobra.Command{
|
||||
Use: "ctl",
|
||||
Short: "Administrative CLI (phase 3)",
|
||||
}
|
||||
for _, sub := range []struct{ use, short string }{
|
||||
{"nodes", "list nodes"},
|
||||
{"models", "show the model catalog"},
|
||||
{"replicas", "list replicas"},
|
||||
{"explain-placement", "explain why a model could not be placed"},
|
||||
{"bench", "measure profile capacity (§10)"},
|
||||
} {
|
||||
cmd.AddCommand(&cobra.Command{
|
||||
Use: sub.use,
|
||||
Short: sub.short,
|
||||
RunE: func(_ *cobra.Command, _ []string) error { return errNotImplemented },
|
||||
})
|
||||
}
|
||||
return cmd
|
||||
}
|
||||
@@ -0,0 +1,283 @@
|
||||
// Package adminapi is the control plane's management surface, which `ctl`
|
||||
// consumes (plan §11, phase 3).
|
||||
//
|
||||
// It is mounted on the admin listener rather than the client-facing one. The
|
||||
// two have different audiences: one serves inference to whoever holds an API
|
||||
// key, the other lets a caller cordon nodes and read the cluster's shape.
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
)
|
||||
|
||||
// API serves the management endpoints.
|
||||
type API struct {
|
||||
ctl *control.Controller
|
||||
}
|
||||
|
||||
// New builds the API over a controller.
|
||||
func New(ctl *control.Controller) *API { return &API{ctl: ctl} }
|
||||
|
||||
// Register mounts the endpoints on a mux under /api/v1.
|
||||
func (a *API) Register(mux *http.ServeMux) {
|
||||
mux.HandleFunc("GET /api/v1/nodes", a.listNodes)
|
||||
mux.HandleFunc("GET /api/v1/models", a.listModels)
|
||||
mux.HandleFunc("GET /api/v1/replicas", a.listReplicas)
|
||||
mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
|
||||
mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
|
||||
}
|
||||
|
||||
// --- responses ---
|
||||
|
||||
// NodeView is a node as `ctl nodes` shows it.
|
||||
type NodeView struct {
|
||||
ID string `json:"id"`
|
||||
Hostname string `json:"hostname,omitempty"`
|
||||
MgmtAddr string `json:"mgmt_addr,omitempty"`
|
||||
State string `json:"state"`
|
||||
Labels map[string]string `json:"labels,omitempty"`
|
||||
|
||||
GPUTotal int `json:"gpu_total"`
|
||||
GPUFree int `json:"gpu_free"`
|
||||
Classes []string `json:"gpu_classes,omitempty"`
|
||||
Fabrics []string `json:"fabrics,omitempty"`
|
||||
|
||||
// Unhealthy lists GPUs kept out of placement and why, which is usually the
|
||||
// answer when a node looks idle but nothing will schedule onto it.
|
||||
Unhealthy []string `json:"unhealthy_gpus,omitempty"`
|
||||
|
||||
LastSeenAgo string `json:"last_seen_ago"`
|
||||
AgentVersion string `json:"agent_version,omitempty"`
|
||||
Replicas int `json:"replicas"`
|
||||
}
|
||||
|
||||
// ModelView is a model as `ctl models` shows it.
|
||||
type ModelView struct {
|
||||
Name string `json:"name"`
|
||||
Aliases []string `json:"aliases,omitempty"`
|
||||
Source string `json:"source"`
|
||||
Min int `json:"min_replicas"`
|
||||
Max int `json:"max_replicas"`
|
||||
Ready int `json:"ready_replicas"`
|
||||
Total int `json:"total_replicas"`
|
||||
Fallbacks []string `json:"fallbacks,omitempty"`
|
||||
Profiles []string `json:"profiles"`
|
||||
}
|
||||
|
||||
// ReplicaView is a replica as `ctl replicas` shows it.
|
||||
type ReplicaView struct {
|
||||
ID string `json:"id"`
|
||||
Model string `json:"model"`
|
||||
Profile string `json:"profile"`
|
||||
State string `json:"state"`
|
||||
Pool string `json:"pool,omitempty"`
|
||||
Fabric string `json:"fabric,omitempty"`
|
||||
Endpoint string `json:"endpoint,omitempty"`
|
||||
AgeSec int64 `json:"age_seconds"`
|
||||
Err string `json:"error,omitempty"`
|
||||
Shards []ShardView `json:"shards"`
|
||||
}
|
||||
|
||||
// ShardView is one shard of a replica.
|
||||
type ShardView struct {
|
||||
ID string `json:"id"`
|
||||
Node string `json:"node"`
|
||||
Rank int `json:"rank"`
|
||||
GPUs []int `json:"gpus"`
|
||||
State string `json:"state"`
|
||||
Err string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// ExplainView is why a model could not be placed.
|
||||
type ExplainView struct {
|
||||
Model string `json:"model"`
|
||||
// Placed is true when the model has replicas, in which case there is
|
||||
// nothing to explain.
|
||||
Placed bool `json:"placed"`
|
||||
// Rejections is empty when placement was never attempted, which is itself
|
||||
// the answer: the model is satisfied, or the reconciler has not run yet.
|
||||
Rejections []RejectionView `json:"rejections,omitempty"`
|
||||
}
|
||||
|
||||
// RejectionView is one rejected candidate.
|
||||
type RejectionView struct {
|
||||
Profile string `json:"profile"`
|
||||
Pool string `json:"pool,omitempty"`
|
||||
Node string `json:"node,omitempty"`
|
||||
Reason string `json:"reason"`
|
||||
}
|
||||
|
||||
// --- handlers ---
|
||||
|
||||
func (a *API) listNodes(w http.ResponseWriter, _ *http.Request) {
|
||||
replicasOn := map[string]int{}
|
||||
for _, rep := range a.ctl.Reconciler().Replicas() {
|
||||
for _, n := range rep.NodeIDs() {
|
||||
replicasOn[n]++
|
||||
}
|
||||
}
|
||||
|
||||
inv := a.ctl.Inventory()
|
||||
out := make([]NodeView, 0)
|
||||
for _, n := range inv.Nodes() {
|
||||
v := NodeView{
|
||||
ID: n.ID, Hostname: n.Hostname, MgmtAddr: n.MgmtAddr,
|
||||
State: n.State.String(), Labels: n.Labels,
|
||||
GPUTotal: len(n.GPUs), GPUFree: len(inv.FreeGPUs(n.ID)),
|
||||
Fabrics: n.FabricNames(), AgentVersion: n.AgentVersion,
|
||||
Replicas: replicasOn[n.ID],
|
||||
}
|
||||
seen := map[string]struct{}{}
|
||||
for _, g := range n.GPUs {
|
||||
if _, dup := seen[g.Class()]; !dup {
|
||||
seen[g.Class()] = struct{}{}
|
||||
v.Classes = append(v.Classes, g.Class())
|
||||
}
|
||||
if !g.Health.OK() {
|
||||
v.Unhealthy = append(v.Unhealthy,
|
||||
fmt.Sprintf("gpu %d: %s", g.Index, g.Health.Problems[0]))
|
||||
}
|
||||
}
|
||||
if !n.LastSeen.IsZero() {
|
||||
v.LastSeenAgo = time.Since(n.LastSeen).Round(time.Second).String()
|
||||
}
|
||||
out = append(out, v)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
func (a *API) listModels(w http.ResponseWriter, _ *http.Request) {
|
||||
ready, total := map[string]int{}, map[string]int{}
|
||||
for _, rep := range a.ctl.Reconciler().Replicas() {
|
||||
total[rep.Model]++
|
||||
if rep.State == reconciler.Ready {
|
||||
ready[rep.Model]++
|
||||
}
|
||||
}
|
||||
|
||||
out := make([]ModelView, 0)
|
||||
for _, m := range a.ctl.Catalog().Models() {
|
||||
v := ModelView{
|
||||
Name: m.Name, Aliases: m.Aliases, Source: m.Source,
|
||||
Min: m.Scaling.Min, Max: m.Scaling.Max,
|
||||
Ready: ready[m.Name], Total: total[m.Name],
|
||||
Fallbacks: m.Fallbacks,
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
v.Profiles = append(v.Profiles, p.Name)
|
||||
}
|
||||
out = append(out, v)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
func (a *API) listReplicas(w http.ResponseWriter, r *http.Request) {
|
||||
model := r.URL.Query().Get("model")
|
||||
|
||||
out := make([]ReplicaView, 0)
|
||||
for _, rep := range a.ctl.Reconciler().Replicas() {
|
||||
if model != "" && rep.Model != model {
|
||||
continue
|
||||
}
|
||||
v := ReplicaView{
|
||||
ID: rep.ID, Model: rep.Model, Profile: rep.Profile,
|
||||
State: rep.State.String(), Pool: rep.Pool, Fabric: rep.Fabric,
|
||||
Endpoint: rep.Endpoint, Err: rep.Err,
|
||||
AgeSec: int64(time.Since(rep.CreatedAt).Seconds()),
|
||||
}
|
||||
for _, s := range rep.Shards {
|
||||
v.Shards = append(v.Shards, ShardView{
|
||||
ID: s.ID, Node: s.NodeID, Rank: s.Rank,
|
||||
GPUs: s.GPUIndices, State: s.State.String(), Err: s.Err,
|
||||
})
|
||||
}
|
||||
out = append(out, v)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
func (a *API) explainPlacement(w http.ResponseWriter, r *http.Request) {
|
||||
model := r.URL.Query().Get("model")
|
||||
if model == "" {
|
||||
writeError(w, http.StatusBadRequest, "model is required")
|
||||
return
|
||||
}
|
||||
if _, ok := a.ctl.Catalog().Resolve(model); !ok {
|
||||
writeError(w, http.StatusNotFound, fmt.Sprintf("model %q is not in the catalog", model))
|
||||
return
|
||||
}
|
||||
|
||||
v := ExplainView{Model: model}
|
||||
for _, rep := range a.ctl.Reconciler().Replicas() {
|
||||
if rep.Model == model && rep.State.Counts() {
|
||||
v.Placed = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if res, ok := a.ctl.Reconciler().Explain(model); ok {
|
||||
for _, rej := range res.Rejections {
|
||||
v.Rejections = append(v.Rejections, RejectionView{
|
||||
Profile: rej.Profile, Pool: rej.Pool, Node: rej.Node, Reason: rej.Reason,
|
||||
})
|
||||
}
|
||||
}
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
}
|
||||
|
||||
// setNodeState implements cordon, uncordon and drain.
|
||||
//
|
||||
// One endpoint rather than three verbs, because they are the same operation
|
||||
// with different targets and the difference matters: cordon stops new
|
||||
// placement, drain also moves existing work off.
|
||||
func (a *API) setNodeState(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("id")
|
||||
|
||||
var body struct {
|
||||
State string `json:"state"`
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
var st inventory.NodeState
|
||||
switch body.State {
|
||||
case "ready", "uncordon":
|
||||
st = inventory.Ready
|
||||
case "cordoned", "cordon":
|
||||
st = inventory.Cordoned
|
||||
case "draining", "drain":
|
||||
st = inventory.Draining
|
||||
default:
|
||||
writeError(w, http.StatusBadRequest,
|
||||
fmt.Sprintf("unknown state %q (want ready, cordoned or draining)", body.State))
|
||||
return
|
||||
}
|
||||
|
||||
if !a.ctl.Inventory().SetState(id, st) {
|
||||
writeError(w, http.StatusNotFound, fmt.Sprintf("no node %q", id))
|
||||
return
|
||||
}
|
||||
// Act on it now rather than at the next tick: an operator draining a node
|
||||
// is usually about to do something to it.
|
||||
a.ctl.Reconciler().Notify()
|
||||
|
||||
writeJSON(w, http.StatusOK, map[string]string{"id": id, "state": st.String()})
|
||||
}
|
||||
|
||||
func writeJSON(w http.ResponseWriter, code int, v any) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(code)
|
||||
_ = json.NewEncoder(w).Encode(v)
|
||||
}
|
||||
|
||||
func writeError(w http.ResponseWriter, code int, msg string) {
|
||||
writeJSON(w, code, map[string]string{"error": msg})
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
)
|
||||
|
||||
// nullSink discards published endpoints; these tests are about the API.
|
||||
type nullSink struct{}
|
||||
|
||||
func (nullSink) SetEndpoints(context.Context, []endpoint.Info) {}
|
||||
|
||||
// nullDispatcher accepts every command.
|
||||
type nullDispatcher struct{}
|
||||
|
||||
func (nullDispatcher) StartShard(context.Context, reconciler.ShardCommand) error { return nil }
|
||||
func (nullDispatcher) StopShard(context.Context, string, string) error { return nil }
|
||||
func (nullDispatcher) DrainShard(context.Context, string, string) error { return nil }
|
||||
|
||||
func setup(t *testing.T, models []catalog.Model, nodes ...inventory.Node) (*API, *control.Controller) {
|
||||
t.Helper()
|
||||
cat, err := catalog.New(models)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctl := control.New(control.DefaultConfig(), cat, nullDispatcher{}, nullSink{}, slog.New(slog.DiscardHandler))
|
||||
for _, n := range nodes {
|
||||
ctl.Inventory().Upsert(n)
|
||||
}
|
||||
return New(ctl), ctl
|
||||
}
|
||||
|
||||
func gpuNode(id string, n int, fabrics ...string) inventory.Node {
|
||||
nd := inventory.Node{ID: id, State: inventory.Ready, AgentVersion: "v1"}
|
||||
for i := range n {
|
||||
nd.GPUs = append(nd.GPUs, discovery.GPU{
|
||||
Index: i, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559,
|
||||
})
|
||||
}
|
||||
for _, f := range fabrics {
|
||||
nd.Fabrics = append(nd.Fabrics, inventory.FabricAttachment{Fabric: f, HCAs: []string{"mlx5_0"}})
|
||||
}
|
||||
return nd
|
||||
}
|
||||
|
||||
func testModels() []catalog.Model {
|
||||
return []catalog.Model{{
|
||||
Name: "llama", Aliases: []string{"l"}, Source: "/m",
|
||||
Scaling: catalog.Scaling{Min: 2, Max: 4},
|
||||
Profiles: []catalog.Profile{{Name: "tp8", GPUClass: "h100-80g", TP: 8}},
|
||||
}}
|
||||
}
|
||||
|
||||
func call(t *testing.T, a *API, method, path string, body string, out any) int {
|
||||
t.Helper()
|
||||
mux := http.NewServeMux()
|
||||
a.Register(mux)
|
||||
|
||||
var r *http.Request
|
||||
if body != "" {
|
||||
r = httptest.NewRequest(method, path, strings.NewReader(body))
|
||||
r.Header.Set("Content-Type", "application/json")
|
||||
} else {
|
||||
r = httptest.NewRequest(method, path, nil)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, r)
|
||||
|
||||
if out != nil && rec.Code == http.StatusOK {
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), out); err != nil {
|
||||
t.Fatalf("decoding %s: %v (%s)", path, err, rec.Body.String())
|
||||
}
|
||||
}
|
||||
return rec.Code
|
||||
}
|
||||
|
||||
func TestListNodes(t *testing.T) {
|
||||
a, _ := setup(t, testModels(), gpuNode("n1", 8, "ib-a"))
|
||||
|
||||
var nodes []NodeView
|
||||
if code := call(t, a, "GET", "/api/v1/nodes", "", &nodes); code != 200 {
|
||||
t.Fatalf("status %d", code)
|
||||
}
|
||||
if len(nodes) != 1 {
|
||||
t.Fatalf("got %d nodes", len(nodes))
|
||||
}
|
||||
n := nodes[0]
|
||||
if n.ID != "n1" || n.State != "ready" {
|
||||
t.Errorf("%+v", n)
|
||||
}
|
||||
if n.GPUTotal != 8 || n.GPUFree != 8 {
|
||||
t.Errorf("gpus %d/%d", n.GPUFree, n.GPUTotal)
|
||||
}
|
||||
if len(n.Classes) != 1 || n.Classes[0] != "h100-80g" {
|
||||
t.Errorf("classes=%v", n.Classes)
|
||||
}
|
||||
if len(n.Fabrics) != 1 || n.Fabrics[0] != "ib-a" {
|
||||
t.Errorf("fabrics=%v", n.Fabrics)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnhealthyGPUsAreSurfaced(t *testing.T) {
|
||||
// Usually the answer when a node looks idle but nothing schedules onto it.
|
||||
n := gpuNode("n1", 4)
|
||||
n.GPUs[1].Health.Problems = []string{"3 uncorrected ECC errors"}
|
||||
a, _ := setup(t, testModels(), n)
|
||||
|
||||
var nodes []NodeView
|
||||
call(t, a, "GET", "/api/v1/nodes", "", &nodes)
|
||||
|
||||
if len(nodes[0].Unhealthy) != 1 {
|
||||
t.Fatalf("unhealthy=%v", nodes[0].Unhealthy)
|
||||
}
|
||||
if !strings.Contains(nodes[0].Unhealthy[0], "ECC") {
|
||||
t.Errorf("the reason was lost: %q", nodes[0].Unhealthy[0])
|
||||
}
|
||||
// And such a GPU is not counted as free.
|
||||
if nodes[0].GPUFree != 3 {
|
||||
t.Errorf("free=%d, an unhealthy GPU was counted", nodes[0].GPUFree)
|
||||
}
|
||||
}
|
||||
|
||||
func TestListModels(t *testing.T) {
|
||||
a, _ := setup(t, testModels(), gpuNode("n1", 8))
|
||||
|
||||
var models []ModelView
|
||||
call(t, a, "GET", "/api/v1/models", "", &models)
|
||||
|
||||
if len(models) != 1 {
|
||||
t.Fatalf("got %d models", len(models))
|
||||
}
|
||||
m := models[0]
|
||||
if m.Name != "llama" || m.Min != 2 || m.Max != 4 {
|
||||
t.Errorf("%+v", m)
|
||||
}
|
||||
if len(m.Aliases) != 1 || m.Aliases[0] != "l" {
|
||||
t.Errorf("aliases=%v", m.Aliases)
|
||||
}
|
||||
if len(m.Profiles) != 1 || m.Profiles[0] != "tp8" {
|
||||
t.Errorf("profiles=%v", m.Profiles)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExplainPlacementForUnplaceableModel(t *testing.T) {
|
||||
// Four GPUs, a profile that needs eight.
|
||||
a, ctl := setup(t, testModels(), gpuNode("n1", 4))
|
||||
ctl.Reconciler().Reconcile(t.Context())
|
||||
|
||||
var v ExplainView
|
||||
if code := call(t, a, "GET", "/api/v1/explain-placement?model=llama", "", &v); code != 200 {
|
||||
t.Fatalf("status %d", code)
|
||||
}
|
||||
if v.Placed {
|
||||
t.Error("an unplaceable model was reported as placed")
|
||||
}
|
||||
if len(v.Rejections) == 0 {
|
||||
t.Fatal("no rejections were recorded; the operator learns nothing")
|
||||
}
|
||||
found := false
|
||||
for _, r := range v.Rejections {
|
||||
if strings.Contains(r.Reason, "needs 8") {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("the reason does not say what was missing: %+v", v.Rejections)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExplainPlacementRequiresAKnownModel(t *testing.T) {
|
||||
a, _ := setup(t, testModels(), gpuNode("n1", 8))
|
||||
|
||||
if code := call(t, a, "GET", "/api/v1/explain-placement", "", nil); code != 400 {
|
||||
t.Errorf("missing model gave %d, expected 400", code)
|
||||
}
|
||||
if code := call(t, a, "GET", "/api/v1/explain-placement?model=nope", "", nil); code != 404 {
|
||||
t.Errorf("unknown model gave %d, expected 404", code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSetNodeState(t *testing.T) {
|
||||
a, ctl := setup(t, testModels(), gpuNode("n1", 8))
|
||||
|
||||
for _, c := range []struct {
|
||||
body string
|
||||
want inventory.NodeState
|
||||
}{
|
||||
{`{"state":"cordon"}`, inventory.Cordoned},
|
||||
{`{"state":"drain"}`, inventory.Draining},
|
||||
{`{"state":"uncordon"}`, inventory.Ready},
|
||||
{`{"state":"cordoned"}`, inventory.Cordoned},
|
||||
} {
|
||||
if code := call(t, a, "POST", "/api/v1/nodes/n1/state", c.body, nil); code != 200 {
|
||||
t.Fatalf("%s gave %d", c.body, code)
|
||||
}
|
||||
n, _ := ctl.Inventory().Get("n1")
|
||||
if n.State != c.want {
|
||||
t.Errorf("%s left the node %s, expected %s", c.body, n.State, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSetNodeStateRejectsNonsense(t *testing.T) {
|
||||
a, _ := setup(t, testModels(), gpuNode("n1", 8))
|
||||
|
||||
if code := call(t, a, "POST", "/api/v1/nodes/n1/state", `{"state":"banana"}`, nil); code != 400 {
|
||||
t.Errorf("an unknown state gave %d, expected 400", code)
|
||||
}
|
||||
if code := call(t, a, "POST", "/api/v1/nodes/nope/state", `{"state":"cordon"}`, nil); code != 404 {
|
||||
t.Errorf("an unknown node gave %d, expected 404", code)
|
||||
}
|
||||
if code := call(t, a, "POST", "/api/v1/nodes/n1/state", `not json`, nil); code != 400 {
|
||||
t.Errorf("malformed body gave %d, expected 400", code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEmptyClusterReturnsArraysNotNull(t *testing.T) {
|
||||
// A JSON null would make every consumer special-case it.
|
||||
a, _ := setup(t, testModels())
|
||||
|
||||
for _, path := range []string{"/api/v1/nodes", "/api/v1/replicas"} {
|
||||
mux := http.NewServeMux()
|
||||
a.Register(mux)
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest("GET", path, nil))
|
||||
|
||||
if got := strings.TrimSpace(rec.Body.String()); got != "[]" {
|
||||
t.Errorf("%s returned %q, expected []", path, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -57,6 +57,10 @@ type Config struct {
|
||||
// Longer than UnreachableAfter on purpose: a brief network blip must not
|
||||
// tear down a replica that is still serving perfectly well.
|
||||
LostAfter time.Duration
|
||||
// DrainGrace bounds how long a draining replica waits for its replacement.
|
||||
// Without a bound, draining a node whose model cannot be placed anywhere
|
||||
// else would never finish.
|
||||
DrainGrace time.Duration
|
||||
// PortRange is the per-node port range for engine HTTP and dist-init.
|
||||
PortRange [2]int
|
||||
// MgmtInterface is the ethernet interface used for NCCL bootstrap.
|
||||
@@ -71,6 +75,7 @@ func DefaultConfig() Config {
|
||||
MaxBackoff: 10 * time.Minute,
|
||||
UnreachableAfter: 15 * time.Second,
|
||||
LostAfter: 60 * time.Second,
|
||||
DrainGrace: 5 * time.Minute,
|
||||
PortRange: [2]int{20000, 20999},
|
||||
MgmtInterface: "eth0",
|
||||
}
|
||||
@@ -93,6 +98,9 @@ func (c *Config) normalize() {
|
||||
if c.LostAfter < c.UnreachableAfter {
|
||||
c.LostAfter = max(c.UnreachableAfter*4, d.LostAfter)
|
||||
}
|
||||
if c.DrainGrace <= 0 {
|
||||
c.DrainGrace = d.DrainGrace
|
||||
}
|
||||
if c.PortRange[1] <= c.PortRange[0] {
|
||||
c.PortRange = d.PortRange
|
||||
}
|
||||
@@ -195,6 +203,7 @@ func (r *Reconciler) Notify() {
|
||||
// instead of racing a ticker.
|
||||
func (r *Reconciler) Reconcile(ctx context.Context) {
|
||||
r.expireNodes()
|
||||
r.drainCordonedNodes(ctx)
|
||||
r.reapFailed(ctx)
|
||||
|
||||
for _, m := range r.cat.Models() {
|
||||
@@ -225,6 +234,48 @@ func (r *Reconciler) expireNodes() {
|
||||
}
|
||||
}
|
||||
|
||||
// drainCordonedNodes moves work off nodes an operator is taking out of service.
|
||||
//
|
||||
// Draining a node is not the same as losing one: the engines are healthy and
|
||||
// still serving, so their replicas are marked Draining rather than Failed. That
|
||||
// makes the model look short of replicas, a replacement is placed elsewhere,
|
||||
// and the original is stopped only once the replacement is serving — which is
|
||||
// the difference between a rolling move and an outage.
|
||||
func (r *Reconciler) drainCordonedNodes(ctx context.Context) {
|
||||
var toDrain []*Replica
|
||||
|
||||
r.mu.Lock()
|
||||
for _, rep := range r.replicas {
|
||||
if rep.State != Ready && rep.State != Starting && rep.State != Placing {
|
||||
continue
|
||||
}
|
||||
for _, s := range rep.Shards {
|
||||
n, ok := r.inv.Get(s.NodeID)
|
||||
if ok && n.State == inventory.Draining {
|
||||
rep.State = Draining
|
||||
rep.DrainingAt = r.now()
|
||||
toDrain = append(toDrain, rep)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
shards := map[string][]Shard{}
|
||||
for _, rep := range toDrain {
|
||||
shards[rep.ID] = slices.Clone(rep.Shards)
|
||||
}
|
||||
r.mu.Unlock()
|
||||
|
||||
for _, rep := range toDrain {
|
||||
r.log.Info("draining replica off a node being taken out of service",
|
||||
"replica", rep.ID, "nodes", rep.NodeIDs())
|
||||
for _, s := range shards[rep.ID] {
|
||||
if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil {
|
||||
r.log.Warn("draining a shard failed", "shard", s.ID, "err", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Reconciler) failReplicasOn(nodeID, reason string) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
@@ -329,9 +380,30 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
|
||||
r.scaleDown(ctx, liveIDs[desired:])
|
||||
}
|
||||
|
||||
// Drained replicas can finally be stopped.
|
||||
// A drained replica is stopped once its replacement is serving, or once the
|
||||
// grace period expires. Stopping it the moment it was drained would cut
|
||||
// in-flight requests, which is the thing draining exists to avoid.
|
||||
replaced := len(liveIDs) >= desired
|
||||
for _, id := range drainingIDs {
|
||||
r.stopReplica(ctx, id)
|
||||
r.mu.Lock()
|
||||
rep, ok := r.replicas[id]
|
||||
var expired bool
|
||||
if ok {
|
||||
expired = !rep.DrainingAt.IsZero() && r.now().Sub(rep.DrainingAt) > r.cfg.DrainGrace
|
||||
}
|
||||
r.mu.Unlock()
|
||||
|
||||
switch {
|
||||
case !ok:
|
||||
case replaced:
|
||||
r.stopReplica(ctx, id)
|
||||
case expired:
|
||||
// The replacement never arrived — often because nowhere else can
|
||||
// host it. Draining forever would leave the node undrainable.
|
||||
r.log.Warn("stopping a drained replica without a replacement",
|
||||
"replica", id, "after", r.cfg.DrainGrace)
|
||||
r.stopReplica(ctx, id)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -496,6 +568,7 @@ func (r *Reconciler) scaleDown(ctx context.Context, excess []string) {
|
||||
continue
|
||||
}
|
||||
rep.State = Draining
|
||||
rep.DrainingAt = r.now()
|
||||
shards := slices.Clone(rep.Shards)
|
||||
r.mu.Unlock()
|
||||
|
||||
|
||||
@@ -771,3 +771,130 @@ func TestAdoptIsIdempotent(t *testing.T) {
|
||||
t.Errorf("node has %d free GPUs; the allocation was applied twice", free)
|
||||
}
|
||||
}
|
||||
|
||||
// --- draining a node (ctl drain) ---
|
||||
|
||||
func TestDrainingNodeMovesWorkOff(t *testing.T) {
|
||||
// The operator's view: a node is about to be worked on, its replicas must
|
||||
// end up elsewhere, and nothing may be cut off mid-request on the way.
|
||||
h := newHarness(t, testModel(1), gpuNode("1", 8), gpuNode("2", 8))
|
||||
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
h.r.Reconcile(t.Context())
|
||||
|
||||
reps := h.r.Replicas()
|
||||
if len(reps) != 1 {
|
||||
t.Fatalf("setup: %d replicas", len(reps))
|
||||
}
|
||||
victim := reps[0].NodeIDs()[0]
|
||||
survivor := "2"
|
||||
if victim == "2" {
|
||||
survivor = "1"
|
||||
}
|
||||
|
||||
h.inv.SetState(victim, inventory.Draining)
|
||||
h.disp.reset()
|
||||
h.r.Reconcile(t.Context())
|
||||
|
||||
// It is drained, not stopped: the engine is healthy and still serving.
|
||||
h.disp.mu.Lock()
|
||||
drained, stopped := len(h.disp.drained), len(h.disp.stopped)
|
||||
h.disp.mu.Unlock()
|
||||
if drained == 0 {
|
||||
t.Error("the replica on the draining node was not drained")
|
||||
}
|
||||
if stopped != 0 {
|
||||
t.Error("the replica was stopped before a replacement existed; in-flight requests would be cut")
|
||||
}
|
||||
|
||||
// A draining replica does not count, so a replacement is placed elsewhere.
|
||||
if got := len(h.disp.startedShards()); got == 0 {
|
||||
t.Fatal("no replacement was placed")
|
||||
}
|
||||
for _, cmd := range h.disp.startedShards() {
|
||||
if cmd.NodeID == victim {
|
||||
t.Errorf("the replacement was placed back on the draining node %s", victim)
|
||||
}
|
||||
}
|
||||
|
||||
h.markReady()
|
||||
h.r.Reconcile(t.Context())
|
||||
|
||||
// Now that the replacement serves, the original can go.
|
||||
h.disp.mu.Lock()
|
||||
stopped = len(h.disp.stopped)
|
||||
h.disp.mu.Unlock()
|
||||
if stopped == 0 {
|
||||
t.Error("the drained replica was never stopped once its replacement was ready")
|
||||
}
|
||||
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.OnNode(victim) {
|
||||
t.Errorf("replica %s is still on the drained node", rep.ID)
|
||||
}
|
||||
if !slices.Contains(rep.NodeIDs(), survivor) {
|
||||
t.Errorf("replica %s did not move to the surviving node", rep.ID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDrainGraceStopsWithoutReplacement(t *testing.T) {
|
||||
// Draining the only node that can host a model would otherwise never
|
||||
// finish, leaving the node undrainable.
|
||||
h := newHarness(t, testModel(1), gpuNode("only", 8))
|
||||
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
h.r.Reconcile(t.Context())
|
||||
|
||||
h.inv.SetState("only", inventory.Draining)
|
||||
h.r.Reconcile(t.Context()) // drains it; nowhere to place a replacement
|
||||
|
||||
if got := h.r.Replicas(); len(got) != 1 || got[0].State != Draining {
|
||||
t.Fatalf("expected a draining replica, got %+v", got)
|
||||
}
|
||||
|
||||
h.disp.reset()
|
||||
h.r.Reconcile(t.Context())
|
||||
h.disp.mu.Lock()
|
||||
early := len(h.disp.stopped)
|
||||
h.disp.mu.Unlock()
|
||||
if early != 0 {
|
||||
t.Error("the replica was stopped before the grace period expired")
|
||||
}
|
||||
|
||||
h.advance(DefaultConfig().DrainGrace + time.Second)
|
||||
h.inv.SetState("only", inventory.Draining) // advance() resets node state
|
||||
h.r.Reconcile(t.Context())
|
||||
|
||||
h.disp.mu.Lock()
|
||||
stopped := len(h.disp.stopped)
|
||||
h.disp.mu.Unlock()
|
||||
if stopped == 0 {
|
||||
t.Error("the drained replica was never stopped; the node cannot be drained")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCordonedNodeKeepsItsWork(t *testing.T) {
|
||||
// Cordon is not drain: it stops new placement and leaves what is running.
|
||||
h := newHarness(t, testModel(1), gpuNode("1", 8))
|
||||
|
||||
h.r.Reconcile(t.Context())
|
||||
h.markReady()
|
||||
h.r.Reconcile(t.Context())
|
||||
|
||||
h.inv.SetState("1", inventory.Cordoned)
|
||||
h.disp.reset()
|
||||
h.r.Reconcile(t.Context())
|
||||
|
||||
h.disp.mu.Lock()
|
||||
drained, stopped := len(h.disp.drained), len(h.disp.stopped)
|
||||
h.disp.mu.Unlock()
|
||||
if drained != 0 || stopped != 0 {
|
||||
t.Errorf("cordoning disturbed a running replica (drained=%d stopped=%d)", drained, stopped)
|
||||
}
|
||||
if got := h.r.Replicas(); len(got) != 1 || got[0].State != Ready {
|
||||
t.Errorf("the replica did not survive a cordon: %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -87,6 +87,9 @@ type Replica struct {
|
||||
|
||||
CreatedAt time.Time
|
||||
ReadyAt time.Time
|
||||
// DrainingAt is when the replica stopped taking new work, so the wait for a
|
||||
// replacement can be bounded.
|
||||
DrainingAt time.Time
|
||||
// Attempt counts how many times this model has failed to come up in a row,
|
||||
// carried across rebuilds so the backoff does not reset on every attempt.
|
||||
Attempt int
|
||||
|
||||
@@ -12,8 +12,9 @@ import (
|
||||
// AdminHandler returns the diagnostics surface: metrics, health and pprof.
|
||||
//
|
||||
// It is served on a separate listener so the client-facing port can be exposed
|
||||
// without also exposing pprof (plan §9.1).
|
||||
func (r *Router) AdminHandler() (http.Handler, error) {
|
||||
// without also exposing pprof (plan §9.1). The control plane's management API
|
||||
// is mounted here too, by the caller.
|
||||
func (r *Router) AdminHandler() (*http.ServeMux, error) {
|
||||
reg := prometheus.NewRegistry()
|
||||
reg.MustRegister(
|
||||
collectors.NewGoCollector(),
|
||||
|
||||
+10
-2
@@ -1269,9 +1269,17 @@ work_dir: /var/lib/llmgw-agent
|
||||
- [x] Catalog: models, profiles, alias and fallback resolution, validation
|
||||
- [x] Reconciler: desired vs actual, replica lifecycle, node expiry, backoff
|
||||
- [x] `EndpointSource` wired from the reconciler to the router
|
||||
- [ ] `ctl`: nodes, models, replicas, cordon, drain, explain-placement, logs
|
||||
- [x] `ctl`: nodes, models, replicas, cordon, uncordon, drain, explain-placement
|
||||
- [ ] `ctl logs`: needs a request/response pair on the agent stream, which the
|
||||
protocol does not have. The failure path already carries an engine log
|
||||
tail with every status report, so the common case is covered; what is
|
||||
missing is reading a healthy shard's output on demand.
|
||||
- [ ] The router's `EndpointSource` becomes dynamic
|
||||
- [ ] `ctl`: nodes, models, replicas, cordon, drain, explain-placement, logs
|
||||
- [x] `ctl`: nodes, models, replicas, cordon, uncordon, drain, explain-placement
|
||||
- [ ] `ctl logs`: needs a request/response pair on the agent stream, which the
|
||||
protocol does not have. The failure path already carries an engine log
|
||||
tail with every status report, so the common case is covered; what is
|
||||
missing is reading a healthy shard's output on demand.
|
||||
- **Exit criterion (met against a simulated agent fleet):** shutting down a node
|
||||
moves its replicas elsewhere while the router keeps serving. Measured in
|
||||
`test/e2e`: a node hosting a live replica is killed mid-traffic; the replica is
|
||||
|
||||
Reference in new issue
Block a user