Agent stream: connect the fleet to the control plane
CI / test (push) Failing after 1m29s

One bidirectional connect-go stream per node, carrying registration, heartbeats
and shard status up, and start, stop and drain down. The agent dials out: GPU
nodes sit behind firewalls far more often than a control plane does, and an
outbound connection needs no inbound rule on every node in the fleet.

- registration carries the whole hardware inventory, so there is no separate
  discovery call and a reconnecting agent refreshes what the scheduler knows.
  The fabric labels travel with it; they cannot be inferred from the wire, so
  losing them in transit would silently disable multi-node placement.
- agentsvc is both the stream handler and the reconciler's dispatcher, because
  it holds the open streams and is therefore the only thing that can deliver a
  command to a node.
- the agent validates a profile before launching anything, since an engine that
  refuses a profile does so after minutes of loading, then supervises the
  process and reports every transition with the engine's output tail on failure.
- reconnects forever with jittered backoff. A control plane restart must not
  require touching every GPU node, and a fleet must not reconnect in lockstep
  and knock it over again. Engines keep serving throughout, because the router
  talks to them directly.

llmgw agent is real now, and llmgw server runs a control plane whenever any
model declares a profile. The static endpoint list stays for router-only runs.

Verified with real binaries as separate processes: the agent registers, a
replica is placed, an engine launches through the vLLM adapter's actual argument
construction, streaming traffic is served, and killing the agent moves the node
through unreachable to lost with placement explaining why nothing can run.

config.Model and catalog.Model were the same thing described twice, so the
router now reads catalog entries directly and the validation rules live in one
place.

Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted
network until that lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
ilkerandClaude Opus 5 committed 2026-09-16 02:48:21 +02:00
1 parent c1b9ea1bc2
commit 2373726efd
24 files changed
+3645 -87

No files matched your search

+69 -8
View File
@@ -1,31 +1,92 @@
package cli
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"os/signal"
"syscall"
"text/tabwriter"
"time"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
)
func newAgentCmd() *cobra.Command {
var configPath string
var (
server string
nodeID string
mgmtAddr string
)
cmd := &cobra.Command{
Use: "agent",
Short: "Node agent: discovery and engine supervision (phase 2)",
Long: "Runs on every GPU node. It discovers the hardware, starts and supervises\n" +
"inference engines, and reports to the control plane.\n\n" +
"The control plane connection is not built yet, so for now the useful\n" +
"subcommand is `agent inventory`.",
RunE: func(_ *cobra.Command, _ []string) error {
return errNotImplemented
Short: "Node agent: discovery and engine supervision",
Long: "Runs on every GPU node. It discovers the hardware, connects to the control\n" +
"plane, and starts, supervises and stops inference engines on demand.\n\n" +
"The agent dials out and keeps one stream open. Engines keep serving if the\n" +
"agent restarts or the control plane goes away: the router talks to them\n" +
"directly, and orphan adoption picks them up again.",
RunE: func(cmd *cobra.Command, _ []string) error {
log := logger()
cfg, err := config.LoadAgent(configPath)
if err != nil {
return err
}
if server != "" {
cfg.Server = server
}
if nodeID != "" {
cfg.NodeID = nodeID
}
if mgmtAddr != "" {
cfg.MgmtAddr = mgmtAddr
}
if cfg.NodeID == "" {
host, err := os.Hostname()
if err != nil {
return fmt.Errorf("node_id is unset and the hostname is unavailable: %w", err)
}
cfg.NodeID = host
}
if cfg.Hostname == "" {
cfg.Hostname, _ = os.Hostname()
}
cfg.Version = version.Version
a, err := agent.New(cfg, agent.HTTP2Client(30*time.Second), log)
if err != nil {
return err
}
ctx, stop := signal.NotifyContext(cmd.Context(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
log.Info("agent starting",
"node", cfg.NodeID, "server", cfg.Server,
"fabrics", len(cfg.Fabrics), "nvml", discovery.NVMLSupported())
err = a.Run(ctx)
if errors.Is(err, context.Canceled) {
return nil
}
return err
},
}
cmd.Flags().StringVar(&configPath, "config", "agent.yaml", "configuration file")
cmd.Flags().StringVar(&configPath, "config", "", "configuration file")
cmd.Flags().StringVar(&server, "server", "", "control plane address (overrides the config)")
cmd.Flags().StringVar(&nodeID, "node-id", "", "node id (defaults to the hostname)")
cmd.Flags().StringVar(&mgmtAddr, "mgmt-addr", "", "address engines are reachable on")
cmd.AddCommand(newInventoryCmd())
return cmd
}