diff --git a/CHANGELOG.md b/CHANGELOG.md index e7d5830..38e93cb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,7 +16,41 @@ is only released once its **exit criterion** is met, not when its code lands. Until v1.0.0 the minor version is the breaking one: configuration formats and Go APIs change between minors without a deprecation period. -## Unreleased — phases 2 and 3 in progress +## v0.5.0 — 2026-09-16 + +Phases 2 and 3. The cluster runs itself: agents report their hardware, the +control plane places replicas, and `ctl` makes all of it visible. + +This release covers both phases because their exit criteria depended on the same +piece — the agent stream — and neither was demonstrable without it. + +### ctl + +``` +llmgw ctl nodes | models | replicas | explain-placement | cordon | uncordon | drain +``` + +- `nodes` surfaces unhealthy GPUs and why they are excluded, which is usually + the answer when a node looks idle but nothing schedules onto it. +- `explain-placement` prints every candidate considered and the reason it was + rejected. "Unschedulable" on its own says nothing about which of six + constraints bit. +- `drain` is a rolling move, not a stop: the replica is drained, a replacement + is placed elsewhere, and the original is stopped only once the replacement is + serving. Bounded by a grace period, because draining the only node that can + host a model would otherwise never finish and leave it undrainable. +- `cordon` is the weaker operation: no new placement, nothing disturbed. + +The management API is on the admin listener, not the client-facing one. The two +have different audiences: one serves inference to whoever holds an API key, the +other lets a caller cordon nodes. + +`ctl logs` is not here. It needs a request/response pair the agent protocol does +not have. The failure path already ships an engine log tail with every status +report, so the case that matters most is covered; reading a healthy shard's +output on demand is not. + +## Unreleased — phase 4 in progress Agent groundwork. Everything here works without GPUs and is covered by tests; what is missing is the control plane connection, so there is no release yet. diff --git a/README.md b/README.md index 7b39862..f4a6b61 100644 --- a/README.md +++ b/README.md @@ -15,8 +15,8 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md). | v0.1.0 | 0 | Skeleton and sim-engine | released | | v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | released | | v0.3.0 | 1.5 | Admission control (priority bands) | released | -| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | planned | -| v0.5.0 | 3 | Control plane: catalog, scheduler, reconciler | planned | +| v0.4.0 | 2 | Agent: GPU/IB discovery, engine supervision | released | +| v0.5.0 | 3 | Control plane: catalog, placement, reconciler, ctl | released | | v0.6.0 | 4 | Multi-node placement and fabric awareness | planned | | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned | @@ -33,6 +33,24 @@ make test # unit tests make race # with the race detector ``` +## Operating a cluster + +```sh +llmgw ctl nodes # capacity, health, what is running where +llmgw ctl models # catalog and replica counts +llmgw ctl replicas --model llama # replicas and their shards +llmgw ctl explain-placement llama # why a model has not been placed +llmgw ctl drain gpu-a-01 # move work off a node, one replica at a time +llmgw ctl cordon gpu-a-01 # stop new placement, leave what runs +``` + +`ctl` talks to the admin port (`:9090` by default), not the client-facing one. +Add `--json` to any command for machine-readable output. + +`explain-placement` is the one worth knowing about: "unschedulable" on its own +says nothing about which of six constraints bit, so it prints every candidate +considered and the reason it was rejected. + ## Running the router ```sh diff --git a/internal/cli/ctl.go b/internal/cli/ctl.go new file mode 100644 index 0000000..55d3b0b --- /dev/null +++ b/internal/cli/ctl.go @@ -0,0 +1,336 @@ +package cli + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "os" + "strings" + "text/tabwriter" + "time" + + "github.com/spf13/cobra" + + "gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi" +) + +// ctlClient talks to a server's admin API. +type ctlClient struct { + base string + http *http.Client +} + +func newCtlClient(addr string) *ctlClient { + if !strings.HasPrefix(addr, "http://") && !strings.HasPrefix(addr, "https://") { + addr = "http://" + addr + } + return &ctlClient{ + base: strings.TrimRight(addr, "/"), + http: &http.Client{Timeout: 15 * time.Second}, + } +} + +func (c *ctlClient) get(ctx context.Context, path string, out any) error { + return c.do(ctx, http.MethodGet, path, nil, out) +} + +func (c *ctlClient) do(ctx context.Context, method, path string, body, out any) error { + var rdr io.Reader + if body != nil { + b, err := json.Marshal(body) + if err != nil { + return err + } + rdr = bytes.NewReader(b) + } + + req, err := http.NewRequestWithContext(ctx, method, c.base+path, rdr) + if err != nil { + return err + } + if body != nil { + req.Header.Set("Content-Type", "application/json") + } + + resp, err := c.http.Do(req) + if err != nil { + return fmt.Errorf("reaching the admin API at %s: %w\n"+ + "(is the server running, and is --server pointing at its admin_listen port?)", c.base, err) + } + defer resp.Body.Close() + + if resp.StatusCode != http.StatusOK { + var e struct { + Error string `json:"error"` + } + b, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<12)) + if json.Unmarshal(b, &e) == nil && e.Error != "" { + return fmt.Errorf("%s: %s", resp.Status, e.Error) + } + return fmt.Errorf("%s: %s", resp.Status, strings.TrimSpace(string(b))) + } + if out == nil { + return nil + } + return json.NewDecoder(resp.Body).Decode(out) +} + +func newCtlCmd() *cobra.Command { + var ( + server string + asJSON bool + ) + + cmd := &cobra.Command{ + Use: "ctl", + Short: "Inspect and operate a running cluster", + Long: "Talks to a server's admin API, which listens on admin_listen (:9090 by\n" + + "default) rather than the client-facing port.", + } + cmd.PersistentFlags().StringVar(&server, "server", "127.0.0.1:9090", "admin API address") + cmd.PersistentFlags().BoolVar(&asJSON, "json", false, "emit JSON") + + client := func() *ctlClient { return newCtlClient(server) } + + cmd.AddCommand( + newNodesCmd(client, &asJSON), + newModelsCmd(client, &asJSON), + newReplicasCmd(client, &asJSON), + newExplainCmd(client, &asJSON), + newNodeStateCmd(client, "cordon", "cordoned", + "Stop placing new work on a node; what runs there keeps running"), + newNodeStateCmd(client, "uncordon", "ready", + "Allow placement on a node again"), + newNodeStateCmd(client, "drain", "draining", + "Move work off a node, one replica at a time"), + ) + return cmd +} + +func newNodesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command { + return &cobra.Command{ + Use: "nodes", + Short: "List nodes and their capacity", + RunE: func(cmd *cobra.Command, _ []string) error { + var nodes []adminapi.NodeView + if err := client().get(cmd.Context(), "/api/v1/nodes", &nodes); err != nil { + return err + } + if *asJSON { + return emitJSON(nodes) + } + if len(nodes) == 0 { + fmt.Println("No nodes have registered.") + return nil + } + + w := tab() + fmt.Fprintln(w, "NODE\tSTATE\tGPUS\tCLASSES\tFABRICS\tREPLICAS\tLAST SEEN") + for _, n := range nodes { + fmt.Fprintf(w, "%s\t%s\t%d/%d free\t%s\t%s\t%d\t%s\n", + n.ID, n.State, n.GPUFree, n.GPUTotal, + join(n.Classes), join(n.Fabrics), n.Replicas, n.LastSeenAgo) + } + _ = w.Flush() + + // Unhealthy GPUs are usually the answer when a node looks idle but + // nothing schedules onto it. + for _, n := range nodes { + for _, p := range n.Unhealthy { + fmt.Printf("\n%s: %s (kept out of placement)", n.ID, p) + } + } + if hasUnhealthy(nodes) { + fmt.Println() + } + return nil + }, + } +} + +func newModelsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command { + return &cobra.Command{ + Use: "models", + Short: "List the catalog and how many replicas each model has", + RunE: func(cmd *cobra.Command, _ []string) error { + var models []adminapi.ModelView + if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil { + return err + } + if *asJSON { + return emitJSON(models) + } + if len(models) == 0 { + fmt.Println("The catalog is empty.") + return nil + } + + w := tab() + fmt.Fprintln(w, "MODEL\tREPLICAS\tALIASES\tPROFILES\tFALLBACKS") + for _, m := range models { + replicas := fmt.Sprintf("%d/%d ready", m.Ready, m.Min) + if m.Total > m.Ready { + replicas += fmt.Sprintf(" (%d total)", m.Total) + } + fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", + m.Name, replicas, join(m.Aliases), join(m.Profiles), join(m.Fallbacks)) + } + return w.Flush() + }, + } +} + +func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command { + var model string + c := &cobra.Command{ + Use: "replicas", + Short: "List replicas and the shards they are made of", + RunE: func(cmd *cobra.Command, _ []string) error { + path := "/api/v1/replicas" + if model != "" { + path += "?model=" + model + } + var reps []adminapi.ReplicaView + if err := client().get(cmd.Context(), path, &reps); err != nil { + return err + } + if *asJSON { + return emitJSON(reps) + } + if len(reps) == 0 { + fmt.Println("No replicas are running.") + return nil + } + + w := tab() + fmt.Fprintln(w, "REPLICA\tMODEL\tSTATE\tPOOL\tNODES\tAGE\tENDPOINT") + for _, r := range reps { + var nodes []string + for _, s := range r.Shards { + nodes = append(nodes, fmt.Sprintf("%s:%v", s.Node, s.GPUs)) + } + fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n", + r.ID, r.Model, r.State, r.Pool, join(nodes), + (time.Duration(r.AgeSec) * time.Second).String(), r.Endpoint) + } + _ = w.Flush() + + for _, r := range reps { + if r.Err != "" { + fmt.Printf("\n%s: %s\n", r.ID, r.Err) + } + for _, s := range r.Shards { + if s.Err != "" { + fmt.Printf(" %s on %s: %s\n", s.ID, s.Node, s.Err) + } + } + } + return nil + }, + } + c.Flags().StringVar(&model, "model", "", "only this model's replicas") + return c +} + +func newExplainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command { + return &cobra.Command{ + Use: "explain-placement ", + Short: "Explain why a model has not been placed", + Long: "Prints every candidate placement considered and the reason it was\n" + + "rejected. \"Unschedulable\" on its own says nothing about which of six\n" + + "constraints bit.", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + var v adminapi.ExplainView + if err := client().get(cmd.Context(), + "/api/v1/explain-placement?model="+args[0], &v); err != nil { + return err + } + if *asJSON { + return emitJSON(v) + } + + if v.Placed && len(v.Rejections) == 0 { + fmt.Printf("%s is placed; nothing was rejected.\n", v.Model) + return nil + } + if len(v.Rejections) == 0 { + fmt.Printf("No placement has been attempted for %s yet.\n", v.Model) + return nil + } + + if v.Placed { + fmt.Printf("%s is placed. The candidates rejected along the way:\n\n", v.Model) + } else { + fmt.Printf("%s could not be placed. Every candidate and why:\n\n", v.Model) + } + w := tab() + fmt.Fprintln(w, "PROFILE\tPOOL\tNODE\tREASON") + for _, r := range v.Rejections { + fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", r.Profile, dash(r.Pool), dash(r.Node), r.Reason) + } + return w.Flush() + }, + } +} + +func newNodeStateCmd(client func() *ctlClient, use, state, short string) *cobra.Command { + return &cobra.Command{ + Use: use + " ", + Short: short, + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + var out map[string]string + err := client().do(cmd.Context(), http.MethodPost, + "/api/v1/nodes/"+args[0]+"/state", + map[string]string{"state": state}, &out) + if err != nil { + return err + } + fmt.Printf("%s is now %s\n", out["id"], out["state"]) + if state == "draining" { + fmt.Println("Replicas will be moved off one at a time, each replacement " + + "brought up before the original is stopped.") + } + return nil + }, + } +} + +// --- output helpers --- + +func tab() *tabwriter.Writer { + return tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', 0) +} + +func emitJSON(v any) error { + enc := json.NewEncoder(os.Stdout) + enc.SetIndent("", " ") + return enc.Encode(v) +} + +func join(v []string) string { + if len(v) == 0 { + return "-" + } + return strings.Join(v, ",") +} + +func dash(s string) string { + if s == "" { + return "-" + } + return s +} + +func hasUnhealthy(nodes []adminapi.NodeView) bool { + for _, n := range nodes { + if len(n.Unhealthy) > 0 { + return true + } + } + return false +} diff --git a/internal/cli/server.go b/internal/cli/server.go index abdf55f..b40a41f 100644 --- a/internal/cli/server.go +++ b/internal/cli/server.go @@ -20,6 +20,7 @@ import ( "gitea.mynodes.xyz/ilker/llmgw/internal/config" "gitea.mynodes.xyz/ilker/llmgw/internal/control" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi" "gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc" "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" "gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler" @@ -85,6 +86,7 @@ func newServerCmd() *cobra.Command { // published to the router (plan §6.4). var agentHandler http.Handler var agentPath string + var admin *control.Controller if controlMode { cat, err := cfg.Catalog() if err != nil { @@ -96,16 +98,20 @@ func newServerCmd() *cobra.Command { return err } defer ctl.Close() + admin = ctl agentPath, agentHandler = svc.Handler() log.Info("control plane running", "models", len(cfg.Models), "agent_listen", cfg.AgentListen) } - adminHandler, err := rt.AdminHandler() + adminMux, err := rt.AdminHandler() if err != nil { return err } + if admin != nil { + adminapi.New(admin).Register(adminMux) + } apiSrv := &http.Server{ Handler: rt.Handler(), @@ -114,7 +120,7 @@ func newServerCmd() *cobra.Command { // enforces its own per-phase deadlines. } adminSrv := &http.Server{ - Handler: adminHandler, + Handler: adminMux, ReadHeaderTimeout: 10 * time.Second, } diff --git a/internal/cli/stubs.go b/internal/cli/stubs.go deleted file mode 100644 index 733d932..0000000 --- a/internal/cli/stubs.go +++ /dev/null @@ -1,31 +0,0 @@ -package cli - -import ( - "errors" - - "github.com/spf13/cobra" -) - -// errNotImplemented marks commands belonging to phases not yet built. -var errNotImplemented = errors.New("not implemented yet (see llmgw-plan.md §11)") - -func newCtlCmd() *cobra.Command { - cmd := &cobra.Command{ - Use: "ctl", - Short: "Administrative CLI (phase 3)", - } - for _, sub := range []struct{ use, short string }{ - {"nodes", "list nodes"}, - {"models", "show the model catalog"}, - {"replicas", "list replicas"}, - {"explain-placement", "explain why a model could not be placed"}, - {"bench", "measure profile capacity (§10)"}, - } { - cmd.AddCommand(&cobra.Command{ - Use: sub.use, - Short: sub.short, - RunE: func(_ *cobra.Command, _ []string) error { return errNotImplemented }, - }) - } - return cmd -} diff --git a/internal/control/adminapi/api.go b/internal/control/adminapi/api.go new file mode 100644 index 0000000..1ed5e77 --- /dev/null +++ b/internal/control/adminapi/api.go @@ -0,0 +1,283 @@ +// Package adminapi is the control plane's management surface, which `ctl` +// consumes (plan §11, phase 3). +// +// It is mounted on the admin listener rather than the client-facing one. The +// two have different audiences: one serves inference to whoever holds an API +// key, the other lets a caller cordon nodes and read the cluster's shape. +package adminapi + +import ( + "encoding/json" + "fmt" + "net/http" + "time" + + "gitea.mynodes.xyz/ilker/llmgw/internal/control" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler" +) + +// API serves the management endpoints. +type API struct { + ctl *control.Controller +} + +// New builds the API over a controller. +func New(ctl *control.Controller) *API { return &API{ctl: ctl} } + +// Register mounts the endpoints on a mux under /api/v1. +func (a *API) Register(mux *http.ServeMux) { + mux.HandleFunc("GET /api/v1/nodes", a.listNodes) + mux.HandleFunc("GET /api/v1/models", a.listModels) + mux.HandleFunc("GET /api/v1/replicas", a.listReplicas) + mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement) + mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState) +} + +// --- responses --- + +// NodeView is a node as `ctl nodes` shows it. +type NodeView struct { + ID string `json:"id"` + Hostname string `json:"hostname,omitempty"` + MgmtAddr string `json:"mgmt_addr,omitempty"` + State string `json:"state"` + Labels map[string]string `json:"labels,omitempty"` + + GPUTotal int `json:"gpu_total"` + GPUFree int `json:"gpu_free"` + Classes []string `json:"gpu_classes,omitempty"` + Fabrics []string `json:"fabrics,omitempty"` + + // Unhealthy lists GPUs kept out of placement and why, which is usually the + // answer when a node looks idle but nothing will schedule onto it. + Unhealthy []string `json:"unhealthy_gpus,omitempty"` + + LastSeenAgo string `json:"last_seen_ago"` + AgentVersion string `json:"agent_version,omitempty"` + Replicas int `json:"replicas"` +} + +// ModelView is a model as `ctl models` shows it. +type ModelView struct { + Name string `json:"name"` + Aliases []string `json:"aliases,omitempty"` + Source string `json:"source"` + Min int `json:"min_replicas"` + Max int `json:"max_replicas"` + Ready int `json:"ready_replicas"` + Total int `json:"total_replicas"` + Fallbacks []string `json:"fallbacks,omitempty"` + Profiles []string `json:"profiles"` +} + +// ReplicaView is a replica as `ctl replicas` shows it. +type ReplicaView struct { + ID string `json:"id"` + Model string `json:"model"` + Profile string `json:"profile"` + State string `json:"state"` + Pool string `json:"pool,omitempty"` + Fabric string `json:"fabric,omitempty"` + Endpoint string `json:"endpoint,omitempty"` + AgeSec int64 `json:"age_seconds"` + Err string `json:"error,omitempty"` + Shards []ShardView `json:"shards"` +} + +// ShardView is one shard of a replica. +type ShardView struct { + ID string `json:"id"` + Node string `json:"node"` + Rank int `json:"rank"` + GPUs []int `json:"gpus"` + State string `json:"state"` + Err string `json:"error,omitempty"` +} + +// ExplainView is why a model could not be placed. +type ExplainView struct { + Model string `json:"model"` + // Placed is true when the model has replicas, in which case there is + // nothing to explain. + Placed bool `json:"placed"` + // Rejections is empty when placement was never attempted, which is itself + // the answer: the model is satisfied, or the reconciler has not run yet. + Rejections []RejectionView `json:"rejections,omitempty"` +} + +// RejectionView is one rejected candidate. +type RejectionView struct { + Profile string `json:"profile"` + Pool string `json:"pool,omitempty"` + Node string `json:"node,omitempty"` + Reason string `json:"reason"` +} + +// --- handlers --- + +func (a *API) listNodes(w http.ResponseWriter, _ *http.Request) { + replicasOn := map[string]int{} + for _, rep := range a.ctl.Reconciler().Replicas() { + for _, n := range rep.NodeIDs() { + replicasOn[n]++ + } + } + + inv := a.ctl.Inventory() + out := make([]NodeView, 0) + for _, n := range inv.Nodes() { + v := NodeView{ + ID: n.ID, Hostname: n.Hostname, MgmtAddr: n.MgmtAddr, + State: n.State.String(), Labels: n.Labels, + GPUTotal: len(n.GPUs), GPUFree: len(inv.FreeGPUs(n.ID)), + Fabrics: n.FabricNames(), AgentVersion: n.AgentVersion, + Replicas: replicasOn[n.ID], + } + seen := map[string]struct{}{} + for _, g := range n.GPUs { + if _, dup := seen[g.Class()]; !dup { + seen[g.Class()] = struct{}{} + v.Classes = append(v.Classes, g.Class()) + } + if !g.Health.OK() { + v.Unhealthy = append(v.Unhealthy, + fmt.Sprintf("gpu %d: %s", g.Index, g.Health.Problems[0])) + } + } + if !n.LastSeen.IsZero() { + v.LastSeenAgo = time.Since(n.LastSeen).Round(time.Second).String() + } + out = append(out, v) + } + writeJSON(w, http.StatusOK, out) +} + +func (a *API) listModels(w http.ResponseWriter, _ *http.Request) { + ready, total := map[string]int{}, map[string]int{} + for _, rep := range a.ctl.Reconciler().Replicas() { + total[rep.Model]++ + if rep.State == reconciler.Ready { + ready[rep.Model]++ + } + } + + out := make([]ModelView, 0) + for _, m := range a.ctl.Catalog().Models() { + v := ModelView{ + Name: m.Name, Aliases: m.Aliases, Source: m.Source, + Min: m.Scaling.Min, Max: m.Scaling.Max, + Ready: ready[m.Name], Total: total[m.Name], + Fallbacks: m.Fallbacks, + } + for _, p := range m.Profiles { + v.Profiles = append(v.Profiles, p.Name) + } + out = append(out, v) + } + writeJSON(w, http.StatusOK, out) +} + +func (a *API) listReplicas(w http.ResponseWriter, r *http.Request) { + model := r.URL.Query().Get("model") + + out := make([]ReplicaView, 0) + for _, rep := range a.ctl.Reconciler().Replicas() { + if model != "" && rep.Model != model { + continue + } + v := ReplicaView{ + ID: rep.ID, Model: rep.Model, Profile: rep.Profile, + State: rep.State.String(), Pool: rep.Pool, Fabric: rep.Fabric, + Endpoint: rep.Endpoint, Err: rep.Err, + AgeSec: int64(time.Since(rep.CreatedAt).Seconds()), + } + for _, s := range rep.Shards { + v.Shards = append(v.Shards, ShardView{ + ID: s.ID, Node: s.NodeID, Rank: s.Rank, + GPUs: s.GPUIndices, State: s.State.String(), Err: s.Err, + }) + } + out = append(out, v) + } + writeJSON(w, http.StatusOK, out) +} + +func (a *API) explainPlacement(w http.ResponseWriter, r *http.Request) { + model := r.URL.Query().Get("model") + if model == "" { + writeError(w, http.StatusBadRequest, "model is required") + return + } + if _, ok := a.ctl.Catalog().Resolve(model); !ok { + writeError(w, http.StatusNotFound, fmt.Sprintf("model %q is not in the catalog", model)) + return + } + + v := ExplainView{Model: model} + for _, rep := range a.ctl.Reconciler().Replicas() { + if rep.Model == model && rep.State.Counts() { + v.Placed = true + break + } + } + if res, ok := a.ctl.Reconciler().Explain(model); ok { + for _, rej := range res.Rejections { + v.Rejections = append(v.Rejections, RejectionView{ + Profile: rej.Profile, Pool: rej.Pool, Node: rej.Node, Reason: rej.Reason, + }) + } + } + writeJSON(w, http.StatusOK, v) +} + +// setNodeState implements cordon, uncordon and drain. +// +// One endpoint rather than three verbs, because they are the same operation +// with different targets and the difference matters: cordon stops new +// placement, drain also moves existing work off. +func (a *API) setNodeState(w http.ResponseWriter, r *http.Request) { + id := r.PathValue("id") + + var body struct { + State string `json:"state"` + } + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + writeError(w, http.StatusBadRequest, err.Error()) + return + } + + var st inventory.NodeState + switch body.State { + case "ready", "uncordon": + st = inventory.Ready + case "cordoned", "cordon": + st = inventory.Cordoned + case "draining", "drain": + st = inventory.Draining + default: + writeError(w, http.StatusBadRequest, + fmt.Sprintf("unknown state %q (want ready, cordoned or draining)", body.State)) + return + } + + if !a.ctl.Inventory().SetState(id, st) { + writeError(w, http.StatusNotFound, fmt.Sprintf("no node %q", id)) + return + } + // Act on it now rather than at the next tick: an operator draining a node + // is usually about to do something to it. + a.ctl.Reconciler().Notify() + + writeJSON(w, http.StatusOK, map[string]string{"id": id, "state": st.String()}) +} + +func writeJSON(w http.ResponseWriter, code int, v any) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(code) + _ = json.NewEncoder(w).Encode(v) +} + +func writeError(w http.ResponseWriter, code int, msg string) { + writeJSON(w, code, map[string]string{"error": msg}) +} diff --git a/internal/control/adminapi/api_test.go b/internal/control/adminapi/api_test.go new file mode 100644 index 0000000..09ab907 --- /dev/null +++ b/internal/control/adminapi/api_test.go @@ -0,0 +1,243 @@ +package adminapi + +import ( + "context" + "encoding/json" + "log/slog" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery" + "gitea.mynodes.xyz/ilker/llmgw/internal/control" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler" + "gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint" +) + +// nullSink discards published endpoints; these tests are about the API. +type nullSink struct{} + +func (nullSink) SetEndpoints(context.Context, []endpoint.Info) {} + +// nullDispatcher accepts every command. +type nullDispatcher struct{} + +func (nullDispatcher) StartShard(context.Context, reconciler.ShardCommand) error { return nil } +func (nullDispatcher) StopShard(context.Context, string, string) error { return nil } +func (nullDispatcher) DrainShard(context.Context, string, string) error { return nil } + +func setup(t *testing.T, models []catalog.Model, nodes ...inventory.Node) (*API, *control.Controller) { + t.Helper() + cat, err := catalog.New(models) + if err != nil { + t.Fatal(err) + } + ctl := control.New(control.DefaultConfig(), cat, nullDispatcher{}, nullSink{}, slog.New(slog.DiscardHandler)) + for _, n := range nodes { + ctl.Inventory().Upsert(n) + } + return New(ctl), ctl +} + +func gpuNode(id string, n int, fabrics ...string) inventory.Node { + nd := inventory.Node{ID: id, State: inventory.Ready, AgentVersion: "v1"} + for i := range n { + nd.GPUs = append(nd.GPUs, discovery.GPU{ + Index: i, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559, + }) + } + for _, f := range fabrics { + nd.Fabrics = append(nd.Fabrics, inventory.FabricAttachment{Fabric: f, HCAs: []string{"mlx5_0"}}) + } + return nd +} + +func testModels() []catalog.Model { + return []catalog.Model{{ + Name: "llama", Aliases: []string{"l"}, Source: "/m", + Scaling: catalog.Scaling{Min: 2, Max: 4}, + Profiles: []catalog.Profile{{Name: "tp8", GPUClass: "h100-80g", TP: 8}}, + }} +} + +func call(t *testing.T, a *API, method, path string, body string, out any) int { + t.Helper() + mux := http.NewServeMux() + a.Register(mux) + + var r *http.Request + if body != "" { + r = httptest.NewRequest(method, path, strings.NewReader(body)) + r.Header.Set("Content-Type", "application/json") + } else { + r = httptest.NewRequest(method, path, nil) + } + rec := httptest.NewRecorder() + mux.ServeHTTP(rec, r) + + if out != nil && rec.Code == http.StatusOK { + if err := json.Unmarshal(rec.Body.Bytes(), out); err != nil { + t.Fatalf("decoding %s: %v (%s)", path, err, rec.Body.String()) + } + } + return rec.Code +} + +func TestListNodes(t *testing.T) { + a, _ := setup(t, testModels(), gpuNode("n1", 8, "ib-a")) + + var nodes []NodeView + if code := call(t, a, "GET", "/api/v1/nodes", "", &nodes); code != 200 { + t.Fatalf("status %d", code) + } + if len(nodes) != 1 { + t.Fatalf("got %d nodes", len(nodes)) + } + n := nodes[0] + if n.ID != "n1" || n.State != "ready" { + t.Errorf("%+v", n) + } + if n.GPUTotal != 8 || n.GPUFree != 8 { + t.Errorf("gpus %d/%d", n.GPUFree, n.GPUTotal) + } + if len(n.Classes) != 1 || n.Classes[0] != "h100-80g" { + t.Errorf("classes=%v", n.Classes) + } + if len(n.Fabrics) != 1 || n.Fabrics[0] != "ib-a" { + t.Errorf("fabrics=%v", n.Fabrics) + } +} + +func TestUnhealthyGPUsAreSurfaced(t *testing.T) { + // Usually the answer when a node looks idle but nothing schedules onto it. + n := gpuNode("n1", 4) + n.GPUs[1].Health.Problems = []string{"3 uncorrected ECC errors"} + a, _ := setup(t, testModels(), n) + + var nodes []NodeView + call(t, a, "GET", "/api/v1/nodes", "", &nodes) + + if len(nodes[0].Unhealthy) != 1 { + t.Fatalf("unhealthy=%v", nodes[0].Unhealthy) + } + if !strings.Contains(nodes[0].Unhealthy[0], "ECC") { + t.Errorf("the reason was lost: %q", nodes[0].Unhealthy[0]) + } + // And such a GPU is not counted as free. + if nodes[0].GPUFree != 3 { + t.Errorf("free=%d, an unhealthy GPU was counted", nodes[0].GPUFree) + } +} + +func TestListModels(t *testing.T) { + a, _ := setup(t, testModels(), gpuNode("n1", 8)) + + var models []ModelView + call(t, a, "GET", "/api/v1/models", "", &models) + + if len(models) != 1 { + t.Fatalf("got %d models", len(models)) + } + m := models[0] + if m.Name != "llama" || m.Min != 2 || m.Max != 4 { + t.Errorf("%+v", m) + } + if len(m.Aliases) != 1 || m.Aliases[0] != "l" { + t.Errorf("aliases=%v", m.Aliases) + } + if len(m.Profiles) != 1 || m.Profiles[0] != "tp8" { + t.Errorf("profiles=%v", m.Profiles) + } +} + +func TestExplainPlacementForUnplaceableModel(t *testing.T) { + // Four GPUs, a profile that needs eight. + a, ctl := setup(t, testModels(), gpuNode("n1", 4)) + ctl.Reconciler().Reconcile(t.Context()) + + var v ExplainView + if code := call(t, a, "GET", "/api/v1/explain-placement?model=llama", "", &v); code != 200 { + t.Fatalf("status %d", code) + } + if v.Placed { + t.Error("an unplaceable model was reported as placed") + } + if len(v.Rejections) == 0 { + t.Fatal("no rejections were recorded; the operator learns nothing") + } + found := false + for _, r := range v.Rejections { + if strings.Contains(r.Reason, "needs 8") { + found = true + } + } + if !found { + t.Errorf("the reason does not say what was missing: %+v", v.Rejections) + } +} + +func TestExplainPlacementRequiresAKnownModel(t *testing.T) { + a, _ := setup(t, testModels(), gpuNode("n1", 8)) + + if code := call(t, a, "GET", "/api/v1/explain-placement", "", nil); code != 400 { + t.Errorf("missing model gave %d, expected 400", code) + } + if code := call(t, a, "GET", "/api/v1/explain-placement?model=nope", "", nil); code != 404 { + t.Errorf("unknown model gave %d, expected 404", code) + } +} + +func TestSetNodeState(t *testing.T) { + a, ctl := setup(t, testModels(), gpuNode("n1", 8)) + + for _, c := range []struct { + body string + want inventory.NodeState + }{ + {`{"state":"cordon"}`, inventory.Cordoned}, + {`{"state":"drain"}`, inventory.Draining}, + {`{"state":"uncordon"}`, inventory.Ready}, + {`{"state":"cordoned"}`, inventory.Cordoned}, + } { + if code := call(t, a, "POST", "/api/v1/nodes/n1/state", c.body, nil); code != 200 { + t.Fatalf("%s gave %d", c.body, code) + } + n, _ := ctl.Inventory().Get("n1") + if n.State != c.want { + t.Errorf("%s left the node %s, expected %s", c.body, n.State, c.want) + } + } +} + +func TestSetNodeStateRejectsNonsense(t *testing.T) { + a, _ := setup(t, testModels(), gpuNode("n1", 8)) + + if code := call(t, a, "POST", "/api/v1/nodes/n1/state", `{"state":"banana"}`, nil); code != 400 { + t.Errorf("an unknown state gave %d, expected 400", code) + } + if code := call(t, a, "POST", "/api/v1/nodes/nope/state", `{"state":"cordon"}`, nil); code != 404 { + t.Errorf("an unknown node gave %d, expected 404", code) + } + if code := call(t, a, "POST", "/api/v1/nodes/n1/state", `not json`, nil); code != 400 { + t.Errorf("malformed body gave %d, expected 400", code) + } +} + +func TestEmptyClusterReturnsArraysNotNull(t *testing.T) { + // A JSON null would make every consumer special-case it. + a, _ := setup(t, testModels()) + + for _, path := range []string{"/api/v1/nodes", "/api/v1/replicas"} { + mux := http.NewServeMux() + a.Register(mux) + rec := httptest.NewRecorder() + mux.ServeHTTP(rec, httptest.NewRequest("GET", path, nil)) + + if got := strings.TrimSpace(rec.Body.String()); got != "[]" { + t.Errorf("%s returned %q, expected []", path, got) + } + } +} diff --git a/internal/control/reconciler/reconciler.go b/internal/control/reconciler/reconciler.go index 68629d2..b49b2b5 100644 --- a/internal/control/reconciler/reconciler.go +++ b/internal/control/reconciler/reconciler.go @@ -57,6 +57,10 @@ type Config struct { // Longer than UnreachableAfter on purpose: a brief network blip must not // tear down a replica that is still serving perfectly well. LostAfter time.Duration + // DrainGrace bounds how long a draining replica waits for its replacement. + // Without a bound, draining a node whose model cannot be placed anywhere + // else would never finish. + DrainGrace time.Duration // PortRange is the per-node port range for engine HTTP and dist-init. PortRange [2]int // MgmtInterface is the ethernet interface used for NCCL bootstrap. @@ -71,6 +75,7 @@ func DefaultConfig() Config { MaxBackoff: 10 * time.Minute, UnreachableAfter: 15 * time.Second, LostAfter: 60 * time.Second, + DrainGrace: 5 * time.Minute, PortRange: [2]int{20000, 20999}, MgmtInterface: "eth0", } @@ -93,6 +98,9 @@ func (c *Config) normalize() { if c.LostAfter < c.UnreachableAfter { c.LostAfter = max(c.UnreachableAfter*4, d.LostAfter) } + if c.DrainGrace <= 0 { + c.DrainGrace = d.DrainGrace + } if c.PortRange[1] <= c.PortRange[0] { c.PortRange = d.PortRange } @@ -195,6 +203,7 @@ func (r *Reconciler) Notify() { // instead of racing a ticker. func (r *Reconciler) Reconcile(ctx context.Context) { r.expireNodes() + r.drainCordonedNodes(ctx) r.reapFailed(ctx) for _, m := range r.cat.Models() { @@ -225,6 +234,48 @@ func (r *Reconciler) expireNodes() { } } +// drainCordonedNodes moves work off nodes an operator is taking out of service. +// +// Draining a node is not the same as losing one: the engines are healthy and +// still serving, so their replicas are marked Draining rather than Failed. That +// makes the model look short of replicas, a replacement is placed elsewhere, +// and the original is stopped only once the replacement is serving — which is +// the difference between a rolling move and an outage. +func (r *Reconciler) drainCordonedNodes(ctx context.Context) { + var toDrain []*Replica + + r.mu.Lock() + for _, rep := range r.replicas { + if rep.State != Ready && rep.State != Starting && rep.State != Placing { + continue + } + for _, s := range rep.Shards { + n, ok := r.inv.Get(s.NodeID) + if ok && n.State == inventory.Draining { + rep.State = Draining + rep.DrainingAt = r.now() + toDrain = append(toDrain, rep) + break + } + } + } + shards := map[string][]Shard{} + for _, rep := range toDrain { + shards[rep.ID] = slices.Clone(rep.Shards) + } + r.mu.Unlock() + + for _, rep := range toDrain { + r.log.Info("draining replica off a node being taken out of service", + "replica", rep.ID, "nodes", rep.NodeIDs()) + for _, s := range shards[rep.ID] { + if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil { + r.log.Warn("draining a shard failed", "shard", s.ID, "err", err) + } + } + } +} + func (r *Reconciler) failReplicasOn(nodeID, reason string) { r.mu.Lock() defer r.mu.Unlock() @@ -329,9 +380,30 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) { r.scaleDown(ctx, liveIDs[desired:]) } - // Drained replicas can finally be stopped. + // A drained replica is stopped once its replacement is serving, or once the + // grace period expires. Stopping it the moment it was drained would cut + // in-flight requests, which is the thing draining exists to avoid. + replaced := len(liveIDs) >= desired for _, id := range drainingIDs { - r.stopReplica(ctx, id) + r.mu.Lock() + rep, ok := r.replicas[id] + var expired bool + if ok { + expired = !rep.DrainingAt.IsZero() && r.now().Sub(rep.DrainingAt) > r.cfg.DrainGrace + } + r.mu.Unlock() + + switch { + case !ok: + case replaced: + r.stopReplica(ctx, id) + case expired: + // The replacement never arrived — often because nowhere else can + // host it. Draining forever would leave the node undrainable. + r.log.Warn("stopping a drained replica without a replacement", + "replica", id, "after", r.cfg.DrainGrace) + r.stopReplica(ctx, id) + } } } @@ -496,6 +568,7 @@ func (r *Reconciler) scaleDown(ctx context.Context, excess []string) { continue } rep.State = Draining + rep.DrainingAt = r.now() shards := slices.Clone(rep.Shards) r.mu.Unlock() diff --git a/internal/control/reconciler/reconciler_test.go b/internal/control/reconciler/reconciler_test.go index dab20b9..2bf3bd0 100644 --- a/internal/control/reconciler/reconciler_test.go +++ b/internal/control/reconciler/reconciler_test.go @@ -771,3 +771,130 @@ func TestAdoptIsIdempotent(t *testing.T) { t.Errorf("node has %d free GPUs; the allocation was applied twice", free) } } + +// --- draining a node (ctl drain) --- + +func TestDrainingNodeMovesWorkOff(t *testing.T) { + // The operator's view: a node is about to be worked on, its replicas must + // end up elsewhere, and nothing may be cut off mid-request on the way. + h := newHarness(t, testModel(1), gpuNode("1", 8), gpuNode("2", 8)) + + h.r.Reconcile(t.Context()) + h.markReady() + h.r.Reconcile(t.Context()) + + reps := h.r.Replicas() + if len(reps) != 1 { + t.Fatalf("setup: %d replicas", len(reps)) + } + victim := reps[0].NodeIDs()[0] + survivor := "2" + if victim == "2" { + survivor = "1" + } + + h.inv.SetState(victim, inventory.Draining) + h.disp.reset() + h.r.Reconcile(t.Context()) + + // It is drained, not stopped: the engine is healthy and still serving. + h.disp.mu.Lock() + drained, stopped := len(h.disp.drained), len(h.disp.stopped) + h.disp.mu.Unlock() + if drained == 0 { + t.Error("the replica on the draining node was not drained") + } + if stopped != 0 { + t.Error("the replica was stopped before a replacement existed; in-flight requests would be cut") + } + + // A draining replica does not count, so a replacement is placed elsewhere. + if got := len(h.disp.startedShards()); got == 0 { + t.Fatal("no replacement was placed") + } + for _, cmd := range h.disp.startedShards() { + if cmd.NodeID == victim { + t.Errorf("the replacement was placed back on the draining node %s", victim) + } + } + + h.markReady() + h.r.Reconcile(t.Context()) + + // Now that the replacement serves, the original can go. + h.disp.mu.Lock() + stopped = len(h.disp.stopped) + h.disp.mu.Unlock() + if stopped == 0 { + t.Error("the drained replica was never stopped once its replacement was ready") + } + + for _, rep := range h.r.Replicas() { + if rep.OnNode(victim) { + t.Errorf("replica %s is still on the drained node", rep.ID) + } + if !slices.Contains(rep.NodeIDs(), survivor) { + t.Errorf("replica %s did not move to the surviving node", rep.ID) + } + } +} + +func TestDrainGraceStopsWithoutReplacement(t *testing.T) { + // Draining the only node that can host a model would otherwise never + // finish, leaving the node undrainable. + h := newHarness(t, testModel(1), gpuNode("only", 8)) + + h.r.Reconcile(t.Context()) + h.markReady() + h.r.Reconcile(t.Context()) + + h.inv.SetState("only", inventory.Draining) + h.r.Reconcile(t.Context()) // drains it; nowhere to place a replacement + + if got := h.r.Replicas(); len(got) != 1 || got[0].State != Draining { + t.Fatalf("expected a draining replica, got %+v", got) + } + + h.disp.reset() + h.r.Reconcile(t.Context()) + h.disp.mu.Lock() + early := len(h.disp.stopped) + h.disp.mu.Unlock() + if early != 0 { + t.Error("the replica was stopped before the grace period expired") + } + + h.advance(DefaultConfig().DrainGrace + time.Second) + h.inv.SetState("only", inventory.Draining) // advance() resets node state + h.r.Reconcile(t.Context()) + + h.disp.mu.Lock() + stopped := len(h.disp.stopped) + h.disp.mu.Unlock() + if stopped == 0 { + t.Error("the drained replica was never stopped; the node cannot be drained") + } +} + +func TestCordonedNodeKeepsItsWork(t *testing.T) { + // Cordon is not drain: it stops new placement and leaves what is running. + h := newHarness(t, testModel(1), gpuNode("1", 8)) + + h.r.Reconcile(t.Context()) + h.markReady() + h.r.Reconcile(t.Context()) + + h.inv.SetState("1", inventory.Cordoned) + h.disp.reset() + h.r.Reconcile(t.Context()) + + h.disp.mu.Lock() + drained, stopped := len(h.disp.drained), len(h.disp.stopped) + h.disp.mu.Unlock() + if drained != 0 || stopped != 0 { + t.Errorf("cordoning disturbed a running replica (drained=%d stopped=%d)", drained, stopped) + } + if got := h.r.Replicas(); len(got) != 1 || got[0].State != Ready { + t.Errorf("the replica did not survive a cordon: %+v", got) + } +} diff --git a/internal/control/reconciler/replica.go b/internal/control/reconciler/replica.go index 1649992..b2562f6 100644 --- a/internal/control/reconciler/replica.go +++ b/internal/control/reconciler/replica.go @@ -87,6 +87,9 @@ type Replica struct { CreatedAt time.Time ReadyAt time.Time + // DrainingAt is when the replica stopped taking new work, so the wait for a + // replacement can be bounded. + DrainingAt time.Time // Attempt counts how many times this model has failed to come up in a row, // carried across rebuilds so the backoff does not reset on every attempt. Attempt int diff --git a/internal/router/admin.go b/internal/router/admin.go index 88793d7..fe7650b 100644 --- a/internal/router/admin.go +++ b/internal/router/admin.go @@ -12,8 +12,9 @@ import ( // AdminHandler returns the diagnostics surface: metrics, health and pprof. // // It is served on a separate listener so the client-facing port can be exposed -// without also exposing pprof (plan §9.1). -func (r *Router) AdminHandler() (http.Handler, error) { +// without also exposing pprof (plan §9.1). The control plane's management API +// is mounted here too, by the caller. +func (r *Router) AdminHandler() (*http.ServeMux, error) { reg := prometheus.NewRegistry() reg.MustRegister( collectors.NewGoCollector(), diff --git a/llmgw-plan.md b/llmgw-plan.md index 60c9bdc..22b9669 100644 --- a/llmgw-plan.md +++ b/llmgw-plan.md @@ -1269,9 +1269,17 @@ work_dir: /var/lib/llmgw-agent - [x] Catalog: models, profiles, alias and fallback resolution, validation - [x] Reconciler: desired vs actual, replica lifecycle, node expiry, backoff - [x] `EndpointSource` wired from the reconciler to the router -- [ ] `ctl`: nodes, models, replicas, cordon, drain, explain-placement, logs +- [x] `ctl`: nodes, models, replicas, cordon, uncordon, drain, explain-placement +- [ ] `ctl logs`: needs a request/response pair on the agent stream, which the + protocol does not have. The failure path already carries an engine log + tail with every status report, so the common case is covered; what is + missing is reading a healthy shard's output on demand. - [ ] The router's `EndpointSource` becomes dynamic -- [ ] `ctl`: nodes, models, replicas, cordon, drain, explain-placement, logs +- [x] `ctl`: nodes, models, replicas, cordon, uncordon, drain, explain-placement +- [ ] `ctl logs`: needs a request/response pair on the agent stream, which the + protocol does not have. The failure path already carries an engine log + tail with every status report, so the common case is covered; what is + missing is reading a healthy shard's output on demand. - **Exit criterion (met against a simulated agent fleet):** shutting down a node moves its replicas elsewhere while the router keeps serving. Measured in `test/e2e`: a node hosting a live replica is killed mid-traffic; the replica is