Agent stream: connect the fleet to the control plane
CI / test (push) Failing after 1m29s

One bidirectional connect-go stream per node, carrying registration, heartbeats
and shard status up, and start, stop and drain down. The agent dials out: GPU
nodes sit behind firewalls far more often than a control plane does, and an
outbound connection needs no inbound rule on every node in the fleet.

- registration carries the whole hardware inventory, so there is no separate
  discovery call and a reconnecting agent refreshes what the scheduler knows.
  The fabric labels travel with it; they cannot be inferred from the wire, so
  losing them in transit would silently disable multi-node placement.
- agentsvc is both the stream handler and the reconciler's dispatcher, because
  it holds the open streams and is therefore the only thing that can deliver a
  command to a node.
- the agent validates a profile before launching anything, since an engine that
  refuses a profile does so after minutes of loading, then supervises the
  process and reports every transition with the engine's output tail on failure.
- reconnects forever with jittered backoff. A control plane restart must not
  require touching every GPU node, and a fleet must not reconnect in lockstep
  and knock it over again. Engines keep serving throughout, because the router
  talks to them directly.

llmgw agent is real now, and llmgw server runs a control plane whenever any
model declares a profile. The static endpoint list stays for router-only runs.

Verified with real binaries as separate processes: the agent registers, a
replica is placed, an engine launches through the vLLM adapter's actual argument
construction, streaming traffic is served, and killing the agent moves the node
through unreachable to lost with placement explaining why nothing can run.

config.Model and catalog.Model were the same thing described twice, so the
router now reads catalog entries directly and the validation rules live in one
place.

Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted
network until that lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
ilkerandClaude Opus 5 committed 2026-09-16 02:48:21 +02:00
1 parent c1b9ea1bc2
commit 2373726efd
24 files changed
+3645 -87

No files matched your search

File diff suppressed because it is too large. Load diff
@@ -0,0 +1,119 @@
// Code generated by protoc-gen-connect-go. DO NOT EDIT.
//
// Source: llmgw/v1/agent.proto
// Package llmgw.v1 is the protocol between a node agent and the control plane
// (plan §5.6).
//
// The agent dials out and holds a bidirectional stream open. That direction is
// deliberate: GPU nodes sit behind firewalls and NAT far more often than a
// control plane does, and an outbound connection needs no inbound rule on every
// node in the fleet.
package llmgwv1connect
import (
connect "connectrpc.com/connect"
context "context"
errors "errors"
v1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
http "net/http"
strings "strings"
)
// This is a compile-time assertion to ensure that this generated file and the connect package are
// compatible. If you get a compiler error that this constant is not defined, this code was
// generated with a version of connect newer than the one compiled into your binary. You can fix the
// problem by either regenerating this code with an older version of connect or updating the connect
// version compiled into your binary.
const _ = connect.IsAtLeastVersion1_13_0
const (
// AgentServiceName is the fully-qualified name of the AgentService service.
AgentServiceName = "llmgw.v1.AgentService"
)
// These constants are the fully-qualified names of the RPCs defined in this package. They're
// exposed at runtime as Spec.Procedure and as the final two segments of the HTTP route.
//
// Note that these are different from the fully-qualified method names used by
// google.golang.org/protobuf/reflect/protoreflect. To convert from these constants to
// reflection-formatted method names, remove the leading slash and convert the remaining slash to a
// period.
const (
// AgentServiceConnectProcedure is the fully-qualified name of the AgentService's Connect RPC.
AgentServiceConnectProcedure = "/llmgw.v1.AgentService/Connect"
)
// AgentServiceClient is a client for the llmgw.v1.AgentService service.
type AgentServiceClient interface {
// Connect opens the agent's stream. It stays open for the agent's lifetime;
// losing it is how each side learns the other is gone.
Connect(context.Context) *connect.BidiStreamForClient[v1.AgentMessage, v1.ControlMessage]
}
// NewAgentServiceClient constructs a client for the llmgw.v1.AgentService service. By default, it
// uses the Connect protocol with the binary Protobuf Codec, asks for gzipped responses, and sends
// uncompressed requests. To use the gRPC or gRPC-Web protocols, supply the connect.WithGRPC() or
// connect.WithGRPCWeb() options.
//
// The URL supplied here should be the base URL for the Connect or gRPC server (for example,
// http://api.acme.com or https://acme.com/grpc).
func NewAgentServiceClient(httpClient connect.HTTPClient, baseURL string, opts ...connect.ClientOption) AgentServiceClient {
baseURL = strings.TrimRight(baseURL, "/")
agentServiceMethods := v1.File_llmgw_v1_agent_proto.Services().ByName("AgentService").Methods()
return &agentServiceClient{
connect: connect.NewClient[v1.AgentMessage, v1.ControlMessage](
httpClient,
baseURL+AgentServiceConnectProcedure,
connect.WithSchema(agentServiceMethods.ByName("Connect")),
connect.WithClientOptions(opts...),
),
}
}
// agentServiceClient implements AgentServiceClient.
type agentServiceClient struct {
connect *connect.Client[v1.AgentMessage, v1.ControlMessage]
}
// Connect calls llmgw.v1.AgentService.Connect.
func (c *agentServiceClient) Connect(ctx context.Context) *connect.BidiStreamForClient[v1.AgentMessage, v1.ControlMessage] {
return c.connect.CallBidiStream(ctx)
}
// AgentServiceHandler is an implementation of the llmgw.v1.AgentService service.
type AgentServiceHandler interface {
// Connect opens the agent's stream. It stays open for the agent's lifetime;
// losing it is how each side learns the other is gone.
Connect(context.Context, *connect.BidiStream[v1.AgentMessage, v1.ControlMessage]) error
}
// NewAgentServiceHandler builds an HTTP handler from the service implementation. It returns the
// path on which to mount the handler and the handler itself.
//
// By default, handlers support the Connect, gRPC, and gRPC-Web protocols with the binary Protobuf
// and JSON codecs. They also support gzip compression.
func NewAgentServiceHandler(svc AgentServiceHandler, opts ...connect.HandlerOption) (string, http.Handler) {
agentServiceMethods := v1.File_llmgw_v1_agent_proto.Services().ByName("AgentService").Methods()
agentServiceConnectHandler := connect.NewBidiStreamHandler(
AgentServiceConnectProcedure,
svc.Connect,
connect.WithSchema(agentServiceMethods.ByName("Connect")),
connect.WithHandlerOptions(opts...),
)
return "/llmgw.v1.AgentService/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case AgentServiceConnectProcedure:
agentServiceConnectHandler.ServeHTTP(w, r)
default:
http.NotFound(w, r)
}
})
}
// UnimplementedAgentServiceHandler returns CodeUnimplemented from all methods.
type UnimplementedAgentServiceHandler struct{}
func (UnimplementedAgentServiceHandler) Connect(context.Context, *connect.BidiStream[v1.AgentMessage, v1.ControlMessage]) error {
return connect.NewError(connect.CodeUnimplemented, errors.New("llmgw.v1.AgentService.Connect is not implemented"))
}
+176
View File
@@ -0,0 +1,176 @@
syntax = "proto3";
// Package llmgw.v1 is the protocol between a node agent and the control plane
// (plan §5.6).
//
// The agent dials out and holds a bidirectional stream open. That direction is
// deliberate: GPU nodes sit behind firewalls and NAT far more often than a
// control plane does, and an outbound connection needs no inbound rule on every
// node in the fleet.
package llmgw.v1;
option go_package = "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1;llmgwv1";
service AgentService {
// Connect opens the agent's stream. It stays open for the agent's lifetime;
// losing it is how each side learns the other is gone.
rpc Connect(stream AgentMessage) returns (stream ControlMessage);
}
// AgentMessage is anything a node tells the control plane.
message AgentMessage {
oneof msg {
Register register = 1;
Heartbeat heartbeat = 2;
ShardStatus shard = 3;
}
}
// ControlMessage is anything the control plane tells a node.
message ControlMessage {
oneof msg {
StartShard start = 1;
StopShard stop = 2;
DrainShard drain = 3;
Ack ack = 4;
}
}
// Register is the first message on every stream. It carries the node's full
// hardware inventory, so the control plane needs no separate discovery call and
// a reconnecting agent automatically refreshes what the scheduler knows.
message Register {
string node_id = 1;
string hostname = 2;
// mgmt_addr is the address engines are reachable on, and the address NCCL
// bootstraps over. Never a fabric address: NCCL rendezvouses over TCP before
// it moves to RDMA.
string mgmt_addr = 3;
map<string, string> labels = 4;
repeated GPU gpus = 5;
// fabrics is empty on a node without InfiniBand. Such a node is its own
// single-node island and can never host a multi-node replica.
repeated FabricAttachment fabrics = 6;
string agent_version = 7;
// engine_versions decides which CLI flags and metric names apply; both drift
// between engine releases (plan §14).
map<string, string> engine_versions = 8;
}
message GPU {
int32 index = 1;
string uuid = 2;
string name = 3;
int32 memory_total_mb = 4;
int32 memory_used_mb = 5;
string compute_capability = 6;
int32 numa_node = 7;
string pci_bus_id = 8;
string driver_version = 9;
string cuda_version = 10;
// nvlink_peers is only populated by agents built with the nvml tag.
repeated int32 nvlink_peers = 11;
// problems keeps a GPU out of placement: ECC faults, or memory already held
// by another process.
repeated string problems = 12;
int32 temperature_c = 13;
int32 ecc_errors = 14;
}
message FabricAttachment {
// fabric is the operator-assigned name. It is not derived from the GID
// prefix: OpenSM defaults to fe80:: on every fabric, so a matching prefix
// proves nothing (plan §5.2).
string fabric = 1;
repeated string hcas = 2;
int32 rate_gbps = 3;
}
// Heartbeat says the node is alive and reports anything that changed cheaply.
message Heartbeat {
string node_id = 1;
// gpu_memory_used_mb, indexed by GPU, catches a foreign process appearing on
// a card after registration.
repeated int32 gpu_memory_used_mb = 2;
repeated string new_problems = 3;
}
// ShardStatus reports a shard's lifecycle transition.
message ShardStatus {
string node_id = 1;
string replica_id = 2;
string shard_id = 3;
State state = 4;
string error = 5;
int32 exit_code = 6;
// last_output carries the tail of the engine log on a failure. When a model
// fails to load this is the only diagnosis available.
repeated string last_output = 7;
enum State {
STATE_UNSPECIFIED = 0;
STATE_PENDING = 1;
STATE_STARTING = 2;
STATE_LOADING = 3;
STATE_READY = 4;
STATE_DRAINING = 5;
STATE_STOPPED = 6;
STATE_FAILED = 7;
}
}
// StartShard launches one shard. It is idempotent by shard_id: the control
// plane retries, and a repeat must not produce a second process.
message StartShard {
string replica_id = 1;
string shard_id = 2;
int32 rank = 3;
int32 nnodes = 4;
string leader_addr = 5;
int32 leader_port = 6;
repeated int32 gpu_indices = 7;
// hcas are the adapters of the chosen fabric and nothing else. Handing a
// shard adapters from a second fabric lets NCCL pick one whose peer is
// unreachable, and the job hangs instead of failing (plan §5.3).
repeated string hcas = 8;
string mgmt_interface = 9;
int32 http_port = 10;
EngineProfile profile = 11;
int64 startup_timeout_ms = 12;
}
message EngineProfile {
string name = 1;
string engine = 2;
string gpu_class = 3;
int32 tp = 4;
int32 pp = 5;
int32 gpus_per_node = 6;
int32 nodes = 7;
repeated string args = 8;
map<string, string> env = 9;
string model_path = 10;
string served_model_name = 11;
}
// StopShard terminates a shard and frees its GPUs.
message StopShard {
string shard_id = 1;
}
// DrainShard stops routing to a shard while letting in-flight work finish.
message DrainShard {
string shard_id = 1;
}
// Ack confirms a registration or reports that a command could not be accepted.
message Ack {
string message_id = 1;
bool ok = 2;
string error = 3;
}