Phase 3: ctl, and make draining a rolling move

ctl talks to a management API on the admin listener, not the client-facing one.
The two have different audiences: one serves inference to whoever holds an API
key, the other lets a caller cordon nodes and read the cluster's shape.

- nodes surfaces unhealthy GPUs and why they are excluded. That is usually the
  answer when a node looks idle but nothing will schedule onto it, and it was
  invisible before.
- explain-placement prints every candidate considered and why it was rejected.
- replicas shows the shards a replica is made of, with per-shard errors.
- cordon, uncordon and drain are one endpoint with different targets, because
  the difference between cordon and drain is exactly what an operator needs to
  be sure of before running either.

Draining had to be fixed first to make ctl drain worth having. A drained replica
was stopped on the next pass, which cuts in-flight requests — the thing draining
exists to avoid. Now a node marked Draining has its replicas drained, the model
falls below its desired count, a replacement is placed elsewhere, and the
original is stopped only once the replacement is serving. A grace period bounds
it, because draining the only node that can host a model would otherwise never
finish and the node could never be taken out of service.

Verified against a two-node cluster of real binaries: draining moved a replica
to the surviving node and traffic kept returning 200 throughout.

ctl logs is deliberately absent. It needs a request/response pair the agent
protocol does not have, and the failure path already ships an engine log tail
with every status report — so the case that matters most is covered, and
reading a healthy shard's output on demand is not worth a protocol change yet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
ilkerandClaude Opus 5 committed 2026-09-16 06:27:21 +02:00
1 parent 1d1a6aebb0
commit cd225c2bf0
12 files changed
+1143 -42

No files matched your search

+35 -1
View File
@@ -16,7 +16,41 @@ is only released once its **exit criterion** is met, not when its code lands.
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased — phases 2 and 3 in progress
## v0.5.0 — 2026-09-16
Phases 2 and 3. The cluster runs itself: agents report their hardware, the
control plane places replicas, and `ctl` makes all of it visible.
This release covers both phases because their exit criteria depended on the same
piece — the agent stream — and neither was demonstrable without it.
### ctl
```
llmgw ctl nodes | models | replicas | explain-placement | cordon | uncordon | drain
```
- `nodes` surfaces unhealthy GPUs and why they are excluded, which is usually
the answer when a node looks idle but nothing schedules onto it.
- `explain-placement` prints every candidate considered and the reason it was
rejected. "Unschedulable" on its own says nothing about which of six
constraints bit.
- `drain` is a rolling move, not a stop: the replica is drained, a replacement
is placed elsewhere, and the original is stopped only once the replacement is
serving. Bounded by a grace period, because draining the only node that can
host a model would otherwise never finish and leave it undrainable.
- `cordon` is the weaker operation: no new placement, nothing disturbed.
The management API is on the admin listener, not the client-facing one. The two
have different audiences: one serves inference to whoever holds an API key, the
other lets a caller cordon nodes.
`ctl logs` is not here. It needs a request/response pair the agent protocol does
not have. The failure path already ships an engine log tail with every status
report, so the case that matters most is covered; reading a healthy shard's
output on demand is not.
## Unreleased — phase 4 in progress
Agent groundwork. Everything here works without GPUs and is covered by tests;
what is missing is the control plane connection, so there is no release yet.
+20 -2
View File
@@ -15,8 +15,8 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
| v0.1.0 | 0 | Skeleton and sim-engine | released |
| v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | released |
| v0.3.0 | 1.5 | Admission control (priority bands) | released |
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | planned |
| v0.5.0 | 3 | Control plane: catalog, scheduler, reconciler | planned |
| v0.4.0 | 2 | Agent: GPU/IB discovery, engine supervision | released |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler, ctl | released |
| v0.6.0 | 4 | Multi-node placement and fabric awareness | planned |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
@@ -33,6 +33,24 @@ make test # unit tests
make race # with the race detector
```
## Operating a cluster
```sh
llmgw ctl nodes # capacity, health, what is running where
llmgw ctl models # catalog and replica counts
llmgw ctl replicas --model llama # replicas and their shards
llmgw ctl explain-placement llama # why a model has not been placed
llmgw ctl drain gpu-a-01 # move work off a node, one replica at a time
llmgw ctl cordon gpu-a-01 # stop new placement, leave what runs
```
`ctl` talks to the admin port (`:9090` by default), not the client-facing one.
Add `--json` to any command for machine-readable output.
`explain-placement` is the one worth knowing about: "unschedulable" on its own
says nothing about which of six constraints bit, so it prints every candidate
considered and the reason it was rejected.
## Running the router
```sh
+336
View File
@@ -0,0 +1,336 @@
package cli
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strings"
"text/tabwriter"
"time"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
)
// ctlClient talks to a server's admin API.
type ctlClient struct {
base string
http *http.Client
}
func newCtlClient(addr string) *ctlClient {
if !strings.HasPrefix(addr, "http://") && !strings.HasPrefix(addr, "https://") {
addr = "http://" + addr
}
return &ctlClient{
base: strings.TrimRight(addr, "/"),
http: &http.Client{Timeout: 15 * time.Second},
}
}
func (c *ctlClient) get(ctx context.Context, path string, out any) error {
return c.do(ctx, http.MethodGet, path, nil, out)
}
func (c *ctlClient) do(ctx context.Context, method, path string, body, out any) error {
var rdr io.Reader
if body != nil {
b, err := json.Marshal(body)
if err != nil {
return err
}
rdr = bytes.NewReader(b)
}
req, err := http.NewRequestWithContext(ctx, method, c.base+path, rdr)
if err != nil {
return err
}
if body != nil {
req.Header.Set("Content-Type", "application/json")
}
resp, err := c.http.Do(req)
if err != nil {
return fmt.Errorf("reaching the admin API at %s: %w\n"+
"(is the server running, and is --server pointing at its admin_listen port?)", c.base, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
var e struct {
Error string `json:"error"`
}
b, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<12))
if json.Unmarshal(b, &e) == nil && e.Error != "" {
return fmt.Errorf("%s: %s", resp.Status, e.Error)
}
return fmt.Errorf("%s: %s", resp.Status, strings.TrimSpace(string(b)))
}
if out == nil {
return nil
}
return json.NewDecoder(resp.Body).Decode(out)
}
func newCtlCmd() *cobra.Command {
var (
server string
asJSON bool
)
cmd := &cobra.Command{
Use: "ctl",
Short: "Inspect and operate a running cluster",
Long: "Talks to a server's admin API, which listens on admin_listen (:9090 by\n" +
"default) rather than the client-facing port.",
}
cmd.PersistentFlags().StringVar(&server, "server", "127.0.0.1:9090", "admin API address")
cmd.PersistentFlags().BoolVar(&asJSON, "json", false, "emit JSON")
client := func() *ctlClient { return newCtlClient(server) }
cmd.AddCommand(
newNodesCmd(client, &asJSON),
newModelsCmd(client, &asJSON),
newReplicasCmd(client, &asJSON),
newExplainCmd(client, &asJSON),
newNodeStateCmd(client, "cordon", "cordoned",
"Stop placing new work on a node; what runs there keeps running"),
newNodeStateCmd(client, "uncordon", "ready",
"Allow placement on a node again"),
newNodeStateCmd(client, "drain", "draining",
"Move work off a node, one replica at a time"),
)
return cmd
}
func newNodesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
return &cobra.Command{
Use: "nodes",
Short: "List nodes and their capacity",
RunE: func(cmd *cobra.Command, _ []string) error {
var nodes []adminapi.NodeView
if err := client().get(cmd.Context(), "/api/v1/nodes", &nodes); err != nil {
return err
}
if *asJSON {
return emitJSON(nodes)
}
if len(nodes) == 0 {
fmt.Println("No nodes have registered.")
return nil
}
w := tab()
fmt.Fprintln(w, "NODE\tSTATE\tGPUS\tCLASSES\tFABRICS\tREPLICAS\tLAST SEEN")
for _, n := range nodes {
fmt.Fprintf(w, "%s\t%s\t%d/%d free\t%s\t%s\t%d\t%s\n",
n.ID, n.State, n.GPUFree, n.GPUTotal,
join(n.Classes), join(n.Fabrics), n.Replicas, n.LastSeenAgo)
}
_ = w.Flush()
// Unhealthy GPUs are usually the answer when a node looks idle but
// nothing schedules onto it.
for _, n := range nodes {
for _, p := range n.Unhealthy {
fmt.Printf("\n%s: %s (kept out of placement)", n.ID, p)
}
}
if hasUnhealthy(nodes) {
fmt.Println()
}
return nil
},
}
}
func newModelsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
return &cobra.Command{
Use: "models",
Short: "List the catalog and how many replicas each model has",
RunE: func(cmd *cobra.Command, _ []string) error {
var models []adminapi.ModelView
if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
return err
}
if *asJSON {
return emitJSON(models)
}
if len(models) == 0 {
fmt.Println("The catalog is empty.")
return nil
}
w := tab()
fmt.Fprintln(w, "MODEL\tREPLICAS\tALIASES\tPROFILES\tFALLBACKS")
for _, m := range models {
replicas := fmt.Sprintf("%d/%d ready", m.Ready, m.Min)
if m.Total > m.Ready {
replicas += fmt.Sprintf(" (%d total)", m.Total)
}
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n",
m.Name, replicas, join(m.Aliases), join(m.Profiles), join(m.Fallbacks))
}
return w.Flush()
},
}
}
func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
var model string
c := &cobra.Command{
Use: "replicas",
Short: "List replicas and the shards they are made of",
RunE: func(cmd *cobra.Command, _ []string) error {
path := "/api/v1/replicas"
if model != "" {
path += "?model=" + model
}
var reps []adminapi.ReplicaView
if err := client().get(cmd.Context(), path, &reps); err != nil {
return err
}
if *asJSON {
return emitJSON(reps)
}
if len(reps) == 0 {
fmt.Println("No replicas are running.")
return nil
}
w := tab()
fmt.Fprintln(w, "REPLICA\tMODEL\tSTATE\tPOOL\tNODES\tAGE\tENDPOINT")
for _, r := range reps {
var nodes []string
for _, s := range r.Shards {
nodes = append(nodes, fmt.Sprintf("%s:%v", s.Node, s.GPUs))
}
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
r.ID, r.Model, r.State, r.Pool, join(nodes),
(time.Duration(r.AgeSec) * time.Second).String(), r.Endpoint)
}
_ = w.Flush()
for _, r := range reps {
if r.Err != "" {
fmt.Printf("\n%s: %s\n", r.ID, r.Err)
}
for _, s := range r.Shards {
if s.Err != "" {
fmt.Printf(" %s on %s: %s\n", s.ID, s.Node, s.Err)
}
}
}
return nil
},
}
c.Flags().StringVar(&model, "model", "", "only this model's replicas")
return c
}
func newExplainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
return &cobra.Command{
Use: "explain-placement <model>",
Short: "Explain why a model has not been placed",
Long: "Prints every candidate placement considered and the reason it was\n" +
"rejected. \"Unschedulable\" on its own says nothing about which of six\n" +
"constraints bit.",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
var v adminapi.ExplainView
if err := client().get(cmd.Context(),
"/api/v1/explain-placement?model="+args[0], &v); err != nil {
return err
}
if *asJSON {
return emitJSON(v)
}
if v.Placed && len(v.Rejections) == 0 {
fmt.Printf("%s is placed; nothing was rejected.\n", v.Model)
return nil
}
if len(v.Rejections) == 0 {
fmt.Printf("No placement has been attempted for %s yet.\n", v.Model)
return nil
}
if v.Placed {
fmt.Printf("%s is placed. The candidates rejected along the way:\n\n", v.Model)
} else {
fmt.Printf("%s could not be placed. Every candidate and why:\n\n", v.Model)
}
w := tab()
fmt.Fprintln(w, "PROFILE\tPOOL\tNODE\tREASON")
for _, r := range v.Rejections {
fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", r.Profile, dash(r.Pool), dash(r.Node), r.Reason)
}
return w.Flush()
},
}
}
func newNodeStateCmd(client func() *ctlClient, use, state, short string) *cobra.Command {
return &cobra.Command{
Use: use + " <node>",
Short: short,
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
var out map[string]string
err := client().do(cmd.Context(), http.MethodPost,
"/api/v1/nodes/"+args[0]+"/state",
map[string]string{"state": state}, &out)
if err != nil {
return err
}
fmt.Printf("%s is now %s\n", out["id"], out["state"])
if state == "draining" {
fmt.Println("Replicas will be moved off one at a time, each replacement " +
"brought up before the original is stopped.")
}
return nil
},
}
}
// --- output helpers ---
func tab() *tabwriter.Writer {
return tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', 0)
}
func emitJSON(v any) error {
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
return enc.Encode(v)
}
func join(v []string) string {
if len(v) == 0 {
return "-"
}
return strings.Join(v, ",")
}
func dash(s string) string {
if s == "" {
return "-"
}
return s
}
func hasUnhealthy(nodes []adminapi.NodeView) bool {
for _, n := range nodes {
if len(n.Unhealthy) > 0 {
return true
}
}
return false
}
+8 -2
View File
@@ -20,6 +20,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
@@ -85,6 +86,7 @@ func newServerCmd() *cobra.Command {
// published to the router (plan §6.4).
var agentHandler http.Handler
var agentPath string
var admin *control.Controller
if controlMode {
cat, err := cfg.Catalog()
if err != nil {
@@ -96,16 +98,20 @@ func newServerCmd() *cobra.Command {
return err
}
defer ctl.Close()
admin = ctl
agentPath, agentHandler = svc.Handler()
log.Info("control plane running",
"models", len(cfg.Models), "agent_listen", cfg.AgentListen)
}
adminHandler, err := rt.AdminHandler()
adminMux, err := rt.AdminHandler()
if err != nil {
return err
}
if admin != nil {
adminapi.New(admin).Register(adminMux)
}
apiSrv := &http.Server{
Handler: rt.Handler(),
@@ -114,7 +120,7 @@ func newServerCmd() *cobra.Command {
// enforces its own per-phase deadlines.
}
adminSrv := &http.Server{
Handler: adminHandler,
Handler: adminMux,
ReadHeaderTimeout: 10 * time.Second,
}
-31
View File
@@ -1,31 +0,0 @@
package cli
import (
"errors"
"github.com/spf13/cobra"
)
// errNotImplemented marks commands belonging to phases not yet built.
var errNotImplemented = errors.New("not implemented yet (see llmgw-plan.md §11)")
func newCtlCmd() *cobra.Command {
cmd := &cobra.Command{
Use: "ctl",
Short: "Administrative CLI (phase 3)",
}
for _, sub := range []struct{ use, short string }{
{"nodes", "list nodes"},
{"models", "show the model catalog"},
{"replicas", "list replicas"},
{"explain-placement", "explain why a model could not be placed"},
{"bench", "measure profile capacity (§10)"},
} {
cmd.AddCommand(&cobra.Command{
Use: sub.use,
Short: sub.short,
RunE: func(_ *cobra.Command, _ []string) error { return errNotImplemented },
})
}
return cmd
}
+283
View File
@@ -0,0 +1,283 @@
// Package adminapi is the control plane's management surface, which `ctl`
// consumes (plan §11, phase 3).
//
// It is mounted on the admin listener rather than the client-facing one. The
// two have different audiences: one serves inference to whoever holds an API
// key, the other lets a caller cordon nodes and read the cluster's shape.
package adminapi
import (
"encoding/json"
"fmt"
"net/http"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
)
// API serves the management endpoints.
type API struct {
ctl *control.Controller
}
// New builds the API over a controller.
func New(ctl *control.Controller) *API { return &API{ctl: ctl} }
// Register mounts the endpoints on a mux under /api/v1.
func (a *API) Register(mux *http.ServeMux) {
mux.HandleFunc("GET /api/v1/nodes", a.listNodes)
mux.HandleFunc("GET /api/v1/models", a.listModels)
mux.HandleFunc("GET /api/v1/replicas", a.listReplicas)
mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
}
// --- responses ---
// NodeView is a node as `ctl nodes` shows it.
type NodeView struct {
ID string `json:"id"`
Hostname string `json:"hostname,omitempty"`
MgmtAddr string `json:"mgmt_addr,omitempty"`
State string `json:"state"`
Labels map[string]string `json:"labels,omitempty"`
GPUTotal int `json:"gpu_total"`
GPUFree int `json:"gpu_free"`
Classes []string `json:"gpu_classes,omitempty"`
Fabrics []string `json:"fabrics,omitempty"`
// Unhealthy lists GPUs kept out of placement and why, which is usually the
// answer when a node looks idle but nothing will schedule onto it.
Unhealthy []string `json:"unhealthy_gpus,omitempty"`
LastSeenAgo string `json:"last_seen_ago"`
AgentVersion string `json:"agent_version,omitempty"`
Replicas int `json:"replicas"`
}
// ModelView is a model as `ctl models` shows it.
type ModelView struct {
Name string `json:"name"`
Aliases []string `json:"aliases,omitempty"`
Source string `json:"source"`
Min int `json:"min_replicas"`
Max int `json:"max_replicas"`
Ready int `json:"ready_replicas"`
Total int `json:"total_replicas"`
Fallbacks []string `json:"fallbacks,omitempty"`
Profiles []string `json:"profiles"`
}
// ReplicaView is a replica as `ctl replicas` shows it.
type ReplicaView struct {
ID string `json:"id"`
Model string `json:"model"`
Profile string `json:"profile"`
State string `json:"state"`
Pool string `json:"pool,omitempty"`
Fabric string `json:"fabric,omitempty"`
Endpoint string `json:"endpoint,omitempty"`
AgeSec int64 `json:"age_seconds"`
Err string `json:"error,omitempty"`
Shards []ShardView `json:"shards"`
}
// ShardView is one shard of a replica.
type ShardView struct {
ID string `json:"id"`
Node string `json:"node"`
Rank int `json:"rank"`
GPUs []int `json:"gpus"`
State string `json:"state"`
Err string `json:"error,omitempty"`
}
// ExplainView is why a model could not be placed.
type ExplainView struct {
Model string `json:"model"`
// Placed is true when the model has replicas, in which case there is
// nothing to explain.
Placed bool `json:"placed"`
// Rejections is empty when placement was never attempted, which is itself
// the answer: the model is satisfied, or the reconciler has not run yet.
Rejections []RejectionView `json:"rejections,omitempty"`
}
// RejectionView is one rejected candidate.
type RejectionView struct {
Profile string `json:"profile"`
Pool string `json:"pool,omitempty"`
Node string `json:"node,omitempty"`
Reason string `json:"reason"`
}
// --- handlers ---
func (a *API) listNodes(w http.ResponseWriter, _ *http.Request) {
replicasOn := map[string]int{}
for _, rep := range a.ctl.Reconciler().Replicas() {
for _, n := range rep.NodeIDs() {
replicasOn[n]++
}
}
inv := a.ctl.Inventory()
out := make([]NodeView, 0)
for _, n := range inv.Nodes() {
v := NodeView{
ID: n.ID, Hostname: n.Hostname, MgmtAddr: n.MgmtAddr,
State: n.State.String(), Labels: n.Labels,
GPUTotal: len(n.GPUs), GPUFree: len(inv.FreeGPUs(n.ID)),
Fabrics: n.FabricNames(), AgentVersion: n.AgentVersion,
Replicas: replicasOn[n.ID],
}
seen := map[string]struct{}{}
for _, g := range n.GPUs {
if _, dup := seen[g.Class()]; !dup {
seen[g.Class()] = struct{}{}
v.Classes = append(v.Classes, g.Class())
}
if !g.Health.OK() {
v.Unhealthy = append(v.Unhealthy,
fmt.Sprintf("gpu %d: %s", g.Index, g.Health.Problems[0]))
}
}
if !n.LastSeen.IsZero() {
v.LastSeenAgo = time.Since(n.LastSeen).Round(time.Second).String()
}
out = append(out, v)
}
writeJSON(w, http.StatusOK, out)
}
func (a *API) listModels(w http.ResponseWriter, _ *http.Request) {
ready, total := map[string]int{}, map[string]int{}
for _, rep := range a.ctl.Reconciler().Replicas() {
total[rep.Model]++
if rep.State == reconciler.Ready {
ready[rep.Model]++
}
}
out := make([]ModelView, 0)
for _, m := range a.ctl.Catalog().Models() {
v := ModelView{
Name: m.Name, Aliases: m.Aliases, Source: m.Source,
Min: m.Scaling.Min, Max: m.Scaling.Max,
Ready: ready[m.Name], Total: total[m.Name],
Fallbacks: m.Fallbacks,
}
for _, p := range m.Profiles {
v.Profiles = append(v.Profiles, p.Name)
}
out = append(out, v)
}
writeJSON(w, http.StatusOK, out)
}
func (a *API) listReplicas(w http.ResponseWriter, r *http.Request) {
model := r.URL.Query().Get("model")
out := make([]ReplicaView, 0)
for _, rep := range a.ctl.Reconciler().Replicas() {
if model != "" && rep.Model != model {
continue
}
v := ReplicaView{
ID: rep.ID, Model: rep.Model, Profile: rep.Profile,
State: rep.State.String(), Pool: rep.Pool, Fabric: rep.Fabric,
Endpoint: rep.Endpoint, Err: rep.Err,
AgeSec: int64(time.Since(rep.CreatedAt).Seconds()),
}
for _, s := range rep.Shards {
v.Shards = append(v.Shards, ShardView{
ID: s.ID, Node: s.NodeID, Rank: s.Rank,
GPUs: s.GPUIndices, State: s.State.String(), Err: s.Err,
})
}
out = append(out, v)
}
writeJSON(w, http.StatusOK, out)
}
func (a *API) explainPlacement(w http.ResponseWriter, r *http.Request) {
model := r.URL.Query().Get("model")
if model == "" {
writeError(w, http.StatusBadRequest, "model is required")
return
}
if _, ok := a.ctl.Catalog().Resolve(model); !ok {
writeError(w, http.StatusNotFound, fmt.Sprintf("model %q is not in the catalog", model))
return
}
v := ExplainView{Model: model}
for _, rep := range a.ctl.Reconciler().Replicas() {
if rep.Model == model && rep.State.Counts() {
v.Placed = true
break
}
}
if res, ok := a.ctl.Reconciler().Explain(model); ok {
for _, rej := range res.Rejections {
v.Rejections = append(v.Rejections, RejectionView{
Profile: rej.Profile, Pool: rej.Pool, Node: rej.Node, Reason: rej.Reason,
})
}
}
writeJSON(w, http.StatusOK, v)
}
// setNodeState implements cordon, uncordon and drain.
//
// One endpoint rather than three verbs, because they are the same operation
// with different targets and the difference matters: cordon stops new
// placement, drain also moves existing work off.
func (a *API) setNodeState(w http.ResponseWriter, r *http.Request) {
id := r.PathValue("id")
var body struct {
State string `json:"state"`
}
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
var st inventory.NodeState
switch body.State {
case "ready", "uncordon":
st = inventory.Ready
case "cordoned", "cordon":
st = inventory.Cordoned
case "draining", "drain":
st = inventory.Draining
default:
writeError(w, http.StatusBadRequest,
fmt.Sprintf("unknown state %q (want ready, cordoned or draining)", body.State))
return
}
if !a.ctl.Inventory().SetState(id, st) {
writeError(w, http.StatusNotFound, fmt.Sprintf("no node %q", id))
return
}
// Act on it now rather than at the next tick: an operator draining a node
// is usually about to do something to it.
a.ctl.Reconciler().Notify()
writeJSON(w, http.StatusOK, map[string]string{"id": id, "state": st.String()})
}
func writeJSON(w http.ResponseWriter, code int, v any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(code)
_ = json.NewEncoder(w).Encode(v)
}
func writeError(w http.ResponseWriter, code int, msg string) {
writeJSON(w, code, map[string]string{"error": msg})
}
+243
View File
@@ -0,0 +1,243 @@
package adminapi
import (
"context"
"encoding/json"
"log/slog"
"net/http"
"net/http/httptest"
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
)
// nullSink discards published endpoints; these tests are about the API.
type nullSink struct{}
func (nullSink) SetEndpoints(context.Context, []endpoint.Info) {}
// nullDispatcher accepts every command.
type nullDispatcher struct{}
func (nullDispatcher) StartShard(context.Context, reconciler.ShardCommand) error { return nil }
func (nullDispatcher) StopShard(context.Context, string, string) error { return nil }
func (nullDispatcher) DrainShard(context.Context, string, string) error { return nil }
func setup(t *testing.T, models []catalog.Model, nodes ...inventory.Node) (*API, *control.Controller) {
t.Helper()
cat, err := catalog.New(models)
if err != nil {
t.Fatal(err)
}
ctl := control.New(control.DefaultConfig(), cat, nullDispatcher{}, nullSink{}, slog.New(slog.DiscardHandler))
for _, n := range nodes {
ctl.Inventory().Upsert(n)
}
return New(ctl), ctl
}
func gpuNode(id string, n int, fabrics ...string) inventory.Node {
nd := inventory.Node{ID: id, State: inventory.Ready, AgentVersion: "v1"}
for i := range n {
nd.GPUs = append(nd.GPUs, discovery.GPU{
Index: i, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559,
})
}
for _, f := range fabrics {
nd.Fabrics = append(nd.Fabrics, inventory.FabricAttachment{Fabric: f, HCAs: []string{"mlx5_0"}})
}
return nd
}
func testModels() []catalog.Model {
return []catalog.Model{{
Name: "llama", Aliases: []string{"l"}, Source: "/m",
Scaling: catalog.Scaling{Min: 2, Max: 4},
Profiles: []catalog.Profile{{Name: "tp8", GPUClass: "h100-80g", TP: 8}},
}}
}
func call(t *testing.T, a *API, method, path string, body string, out any) int {
t.Helper()
mux := http.NewServeMux()
a.Register(mux)
var r *http.Request
if body != "" {
r = httptest.NewRequest(method, path, strings.NewReader(body))
r.Header.Set("Content-Type", "application/json")
} else {
r = httptest.NewRequest(method, path, nil)
}
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, r)
if out != nil && rec.Code == http.StatusOK {
if err := json.Unmarshal(rec.Body.Bytes(), out); err != nil {
t.Fatalf("decoding %s: %v (%s)", path, err, rec.Body.String())
}
}
return rec.Code
}
func TestListNodes(t *testing.T) {
a, _ := setup(t, testModels(), gpuNode("n1", 8, "ib-a"))
var nodes []NodeView
if code := call(t, a, "GET", "/api/v1/nodes", "", &nodes); code != 200 {
t.Fatalf("status %d", code)
}
if len(nodes) != 1 {
t.Fatalf("got %d nodes", len(nodes))
}
n := nodes[0]
if n.ID != "n1" || n.State != "ready" {
t.Errorf("%+v", n)
}
if n.GPUTotal != 8 || n.GPUFree != 8 {
t.Errorf("gpus %d/%d", n.GPUFree, n.GPUTotal)
}
if len(n.Classes) != 1 || n.Classes[0] != "h100-80g" {
t.Errorf("classes=%v", n.Classes)
}
if len(n.Fabrics) != 1 || n.Fabrics[0] != "ib-a" {
t.Errorf("fabrics=%v", n.Fabrics)
}
}
func TestUnhealthyGPUsAreSurfaced(t *testing.T) {
// Usually the answer when a node looks idle but nothing schedules onto it.
n := gpuNode("n1", 4)
n.GPUs[1].Health.Problems = []string{"3 uncorrected ECC errors"}
a, _ := setup(t, testModels(), n)
var nodes []NodeView
call(t, a, "GET", "/api/v1/nodes", "", &nodes)
if len(nodes[0].Unhealthy) != 1 {
t.Fatalf("unhealthy=%v", nodes[0].Unhealthy)
}
if !strings.Contains(nodes[0].Unhealthy[0], "ECC") {
t.Errorf("the reason was lost: %q", nodes[0].Unhealthy[0])
}
// And such a GPU is not counted as free.
if nodes[0].GPUFree != 3 {
t.Errorf("free=%d, an unhealthy GPU was counted", nodes[0].GPUFree)
}
}
func TestListModels(t *testing.T) {
a, _ := setup(t, testModels(), gpuNode("n1", 8))
var models []ModelView
call(t, a, "GET", "/api/v1/models", "", &models)
if len(models) != 1 {
t.Fatalf("got %d models", len(models))
}
m := models[0]
if m.Name != "llama" || m.Min != 2 || m.Max != 4 {
t.Errorf("%+v", m)
}
if len(m.Aliases) != 1 || m.Aliases[0] != "l" {
t.Errorf("aliases=%v", m.Aliases)
}
if len(m.Profiles) != 1 || m.Profiles[0] != "tp8" {
t.Errorf("profiles=%v", m.Profiles)
}
}
func TestExplainPlacementForUnplaceableModel(t *testing.T) {
// Four GPUs, a profile that needs eight.
a, ctl := setup(t, testModels(), gpuNode("n1", 4))
ctl.Reconciler().Reconcile(t.Context())
var v ExplainView
if code := call(t, a, "GET", "/api/v1/explain-placement?model=llama", "", &v); code != 200 {
t.Fatalf("status %d", code)
}
if v.Placed {
t.Error("an unplaceable model was reported as placed")
}
if len(v.Rejections) == 0 {
t.Fatal("no rejections were recorded; the operator learns nothing")
}
found := false
for _, r := range v.Rejections {
if strings.Contains(r.Reason, "needs 8") {
found = true
}
}
if !found {
t.Errorf("the reason does not say what was missing: %+v", v.Rejections)
}
}
func TestExplainPlacementRequiresAKnownModel(t *testing.T) {
a, _ := setup(t, testModels(), gpuNode("n1", 8))
if code := call(t, a, "GET", "/api/v1/explain-placement", "", nil); code != 400 {
t.Errorf("missing model gave %d, expected 400", code)
}
if code := call(t, a, "GET", "/api/v1/explain-placement?model=nope", "", nil); code != 404 {
t.Errorf("unknown model gave %d, expected 404", code)
}
}
func TestSetNodeState(t *testing.T) {
a, ctl := setup(t, testModels(), gpuNode("n1", 8))
for _, c := range []struct {
body string
want inventory.NodeState
}{
{`{"state":"cordon"}`, inventory.Cordoned},
{`{"state":"drain"}`, inventory.Draining},
{`{"state":"uncordon"}`, inventory.Ready},
{`{"state":"cordoned"}`, inventory.Cordoned},
} {
if code := call(t, a, "POST", "/api/v1/nodes/n1/state", c.body, nil); code != 200 {
t.Fatalf("%s gave %d", c.body, code)
}
n, _ := ctl.Inventory().Get("n1")
if n.State != c.want {
t.Errorf("%s left the node %s, expected %s", c.body, n.State, c.want)
}
}
}
func TestSetNodeStateRejectsNonsense(t *testing.T) {
a, _ := setup(t, testModels(), gpuNode("n1", 8))
if code := call(t, a, "POST", "/api/v1/nodes/n1/state", `{"state":"banana"}`, nil); code != 400 {
t.Errorf("an unknown state gave %d, expected 400", code)
}
if code := call(t, a, "POST", "/api/v1/nodes/nope/state", `{"state":"cordon"}`, nil); code != 404 {
t.Errorf("an unknown node gave %d, expected 404", code)
}
if code := call(t, a, "POST", "/api/v1/nodes/n1/state", `not json`, nil); code != 400 {
t.Errorf("malformed body gave %d, expected 400", code)
}
}
func TestEmptyClusterReturnsArraysNotNull(t *testing.T) {
// A JSON null would make every consumer special-case it.
a, _ := setup(t, testModels())
for _, path := range []string{"/api/v1/nodes", "/api/v1/replicas"} {
mux := http.NewServeMux()
a.Register(mux)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, httptest.NewRequest("GET", path, nil))
if got := strings.TrimSpace(rec.Body.String()); got != "[]" {
t.Errorf("%s returned %q, expected []", path, got)
}
}
}
+75 -2
View File
@@ -57,6 +57,10 @@ type Config struct {
// Longer than UnreachableAfter on purpose: a brief network blip must not
// tear down a replica that is still serving perfectly well.
LostAfter time.Duration
// DrainGrace bounds how long a draining replica waits for its replacement.
// Without a bound, draining a node whose model cannot be placed anywhere
// else would never finish.
DrainGrace time.Duration
// PortRange is the per-node port range for engine HTTP and dist-init.
PortRange [2]int
// MgmtInterface is the ethernet interface used for NCCL bootstrap.
@@ -71,6 +75,7 @@ func DefaultConfig() Config {
MaxBackoff: 10 * time.Minute,
UnreachableAfter: 15 * time.Second,
LostAfter: 60 * time.Second,
DrainGrace: 5 * time.Minute,
PortRange: [2]int{20000, 20999},
MgmtInterface: "eth0",
}
@@ -93,6 +98,9 @@ func (c *Config) normalize() {
if c.LostAfter < c.UnreachableAfter {
c.LostAfter = max(c.UnreachableAfter*4, d.LostAfter)
}
if c.DrainGrace <= 0 {
c.DrainGrace = d.DrainGrace
}
if c.PortRange[1] <= c.PortRange[0] {
c.PortRange = d.PortRange
}
@@ -195,6 +203,7 @@ func (r *Reconciler) Notify() {
// instead of racing a ticker.
func (r *Reconciler) Reconcile(ctx context.Context) {
r.expireNodes()
r.drainCordonedNodes(ctx)
r.reapFailed(ctx)
for _, m := range r.cat.Models() {
@@ -225,6 +234,48 @@ func (r *Reconciler) expireNodes() {
}
}
// drainCordonedNodes moves work off nodes an operator is taking out of service.
//
// Draining a node is not the same as losing one: the engines are healthy and
// still serving, so their replicas are marked Draining rather than Failed. That
// makes the model look short of replicas, a replacement is placed elsewhere,
// and the original is stopped only once the replacement is serving — which is
// the difference between a rolling move and an outage.
func (r *Reconciler) drainCordonedNodes(ctx context.Context) {
var toDrain []*Replica
r.mu.Lock()
for _, rep := range r.replicas {
if rep.State != Ready && rep.State != Starting && rep.State != Placing {
continue
}
for _, s := range rep.Shards {
n, ok := r.inv.Get(s.NodeID)
if ok && n.State == inventory.Draining {
rep.State = Draining
rep.DrainingAt = r.now()
toDrain = append(toDrain, rep)
break
}
}
}
shards := map[string][]Shard{}
for _, rep := range toDrain {
shards[rep.ID] = slices.Clone(rep.Shards)
}
r.mu.Unlock()
for _, rep := range toDrain {
r.log.Info("draining replica off a node being taken out of service",
"replica", rep.ID, "nodes", rep.NodeIDs())
for _, s := range shards[rep.ID] {
if err := r.disp.DrainShard(ctx, s.NodeID, s.ID); err != nil {
r.log.Warn("draining a shard failed", "shard", s.ID, "err", err)
}
}
}
}
func (r *Reconciler) failReplicasOn(nodeID, reason string) {
r.mu.Lock()
defer r.mu.Unlock()
@@ -329,9 +380,30 @@ func (r *Reconciler) reconcileModel(ctx context.Context, m catalog.Model) {
r.scaleDown(ctx, liveIDs[desired:])
}
// Drained replicas can finally be stopped.
// A drained replica is stopped once its replacement is serving, or once the
// grace period expires. Stopping it the moment it was drained would cut
// in-flight requests, which is the thing draining exists to avoid.
replaced := len(liveIDs) >= desired
for _, id := range drainingIDs {
r.stopReplica(ctx, id)
r.mu.Lock()
rep, ok := r.replicas[id]
var expired bool
if ok {
expired = !rep.DrainingAt.IsZero() && r.now().Sub(rep.DrainingAt) > r.cfg.DrainGrace
}
r.mu.Unlock()
switch {
case !ok:
case replaced:
r.stopReplica(ctx, id)
case expired:
// The replacement never arrived — often because nowhere else can
// host it. Draining forever would leave the node undrainable.
r.log.Warn("stopping a drained replica without a replacement",
"replica", id, "after", r.cfg.DrainGrace)
r.stopReplica(ctx, id)
}
}
}
@@ -496,6 +568,7 @@ func (r *Reconciler) scaleDown(ctx context.Context, excess []string) {
continue
}
rep.State = Draining
rep.DrainingAt = r.now()
shards := slices.Clone(rep.Shards)
r.mu.Unlock()
@@ -771,3 +771,130 @@ func TestAdoptIsIdempotent(t *testing.T) {
t.Errorf("node has %d free GPUs; the allocation was applied twice", free)
}
}
// --- draining a node (ctl drain) ---
func TestDrainingNodeMovesWorkOff(t *testing.T) {
// The operator's view: a node is about to be worked on, its replicas must
// end up elsewhere, and nothing may be cut off mid-request on the way.
h := newHarness(t, testModel(1), gpuNode("1", 8), gpuNode("2", 8))
h.r.Reconcile(t.Context())
h.markReady()
h.r.Reconcile(t.Context())
reps := h.r.Replicas()
if len(reps) != 1 {
t.Fatalf("setup: %d replicas", len(reps))
}
victim := reps[0].NodeIDs()[0]
survivor := "2"
if victim == "2" {
survivor = "1"
}
h.inv.SetState(victim, inventory.Draining)
h.disp.reset()
h.r.Reconcile(t.Context())
// It is drained, not stopped: the engine is healthy and still serving.
h.disp.mu.Lock()
drained, stopped := len(h.disp.drained), len(h.disp.stopped)
h.disp.mu.Unlock()
if drained == 0 {
t.Error("the replica on the draining node was not drained")
}
if stopped != 0 {
t.Error("the replica was stopped before a replacement existed; in-flight requests would be cut")
}
// A draining replica does not count, so a replacement is placed elsewhere.
if got := len(h.disp.startedShards()); got == 0 {
t.Fatal("no replacement was placed")
}
for _, cmd := range h.disp.startedShards() {
if cmd.NodeID == victim {
t.Errorf("the replacement was placed back on the draining node %s", victim)
}
}
h.markReady()
h.r.Reconcile(t.Context())
// Now that the replacement serves, the original can go.
h.disp.mu.Lock()
stopped = len(h.disp.stopped)
h.disp.mu.Unlock()
if stopped == 0 {
t.Error("the drained replica was never stopped once its replacement was ready")
}
for _, rep := range h.r.Replicas() {
if rep.OnNode(victim) {
t.Errorf("replica %s is still on the drained node", rep.ID)
}
if !slices.Contains(rep.NodeIDs(), survivor) {
t.Errorf("replica %s did not move to the surviving node", rep.ID)
}
}
}
func TestDrainGraceStopsWithoutReplacement(t *testing.T) {
// Draining the only node that can host a model would otherwise never
// finish, leaving the node undrainable.
h := newHarness(t, testModel(1), gpuNode("only", 8))
h.r.Reconcile(t.Context())
h.markReady()
h.r.Reconcile(t.Context())
h.inv.SetState("only", inventory.Draining)
h.r.Reconcile(t.Context()) // drains it; nowhere to place a replacement
if got := h.r.Replicas(); len(got) != 1 || got[0].State != Draining {
t.Fatalf("expected a draining replica, got %+v", got)
}
h.disp.reset()
h.r.Reconcile(t.Context())
h.disp.mu.Lock()
early := len(h.disp.stopped)
h.disp.mu.Unlock()
if early != 0 {
t.Error("the replica was stopped before the grace period expired")
}
h.advance(DefaultConfig().DrainGrace + time.Second)
h.inv.SetState("only", inventory.Draining) // advance() resets node state
h.r.Reconcile(t.Context())
h.disp.mu.Lock()
stopped := len(h.disp.stopped)
h.disp.mu.Unlock()
if stopped == 0 {
t.Error("the drained replica was never stopped; the node cannot be drained")
}
}
func TestCordonedNodeKeepsItsWork(t *testing.T) {
// Cordon is not drain: it stops new placement and leaves what is running.
h := newHarness(t, testModel(1), gpuNode("1", 8))
h.r.Reconcile(t.Context())
h.markReady()
h.r.Reconcile(t.Context())
h.inv.SetState("1", inventory.Cordoned)
h.disp.reset()
h.r.Reconcile(t.Context())
h.disp.mu.Lock()
drained, stopped := len(h.disp.drained), len(h.disp.stopped)
h.disp.mu.Unlock()
if drained != 0 || stopped != 0 {
t.Errorf("cordoning disturbed a running replica (drained=%d stopped=%d)", drained, stopped)
}
if got := h.r.Replicas(); len(got) != 1 || got[0].State != Ready {
t.Errorf("the replica did not survive a cordon: %+v", got)
}
}
+3
View File
@@ -87,6 +87,9 @@ type Replica struct {
CreatedAt time.Time
ReadyAt time.Time
// DrainingAt is when the replica stopped taking new work, so the wait for a
// replacement can be bounded.
DrainingAt time.Time
// Attempt counts how many times this model has failed to come up in a row,
// carried across rebuilds so the backoff does not reset on every attempt.
Attempt int
+3 -2
View File
@@ -12,8 +12,9 @@ import (
// AdminHandler returns the diagnostics surface: metrics, health and pprof.
//
// It is served on a separate listener so the client-facing port can be exposed
// without also exposing pprof (plan §9.1).
func (r *Router) AdminHandler() (http.Handler, error) {
// without also exposing pprof (plan §9.1). The control plane's management API
// is mounted here too, by the caller.
func (r *Router) AdminHandler() (*http.ServeMux, error) {
reg := prometheus.NewRegistry()
reg.MustRegister(
collectors.NewGoCollector(),
+10 -2
View File
@@ -1269,9 +1269,17 @@ work_dir: /var/lib/llmgw-agent
- [x] Catalog: models, profiles, alias and fallback resolution, validation
- [x] Reconciler: desired vs actual, replica lifecycle, node expiry, backoff
- [x] `EndpointSource` wired from the reconciler to the router
- [ ] `ctl`: nodes, models, replicas, cordon, drain, explain-placement, logs
- [x] `ctl`: nodes, models, replicas, cordon, uncordon, drain, explain-placement
- [ ] `ctl logs`: needs a request/response pair on the agent stream, which the
protocol does not have. The failure path already carries an engine log
tail with every status report, so the common case is covered; what is
missing is reading a healthy shard's output on demand.
- [ ] The router's `EndpointSource` becomes dynamic
- [ ] `ctl`: nodes, models, replicas, cordon, drain, explain-placement, logs
- [x] `ctl`: nodes, models, replicas, cordon, uncordon, drain, explain-placement
- [ ] `ctl logs`: needs a request/response pair on the agent stream, which the
protocol does not have. The failure path already carries an engine log
tail with every status report, so the common case is covered; what is
missing is reading a healthy shard's output on demand.
- **Exit criterion (met against a simulated agent fleet):** shutting down a node
moves its replicas elsewhere while the router keeps serving. Measured in
`test/e2e`: a node hosting a live replica is killed mid-traffic; the replica is