Agent stream: connect the fleet to the control plane
CI / test (push) Failing after 1m29s

One bidirectional connect-go stream per node, carrying registration, heartbeats
and shard status up, and start, stop and drain down. The agent dials out: GPU
nodes sit behind firewalls far more often than a control plane does, and an
outbound connection needs no inbound rule on every node in the fleet.

- registration carries the whole hardware inventory, so there is no separate
  discovery call and a reconnecting agent refreshes what the scheduler knows.
  The fabric labels travel with it; they cannot be inferred from the wire, so
  losing them in transit would silently disable multi-node placement.
- agentsvc is both the stream handler and the reconciler's dispatcher, because
  it holds the open streams and is therefore the only thing that can deliver a
  command to a node.
- the agent validates a profile before launching anything, since an engine that
  refuses a profile does so after minutes of loading, then supervises the
  process and reports every transition with the engine's output tail on failure.
- reconnects forever with jittered backoff. A control plane restart must not
  require touching every GPU node, and a fleet must not reconnect in lockstep
  and knock it over again. Engines keep serving throughout, because the router
  talks to them directly.

llmgw agent is real now, and llmgw server runs a control plane whenever any
model declares a profile. The static endpoint list stays for router-only runs.

Verified with real binaries as separate processes: the agent registers, a
replica is placed, an engine launches through the vLLM adapter's actual argument
construction, streaming traffic is served, and killing the agent moves the node
through unreachable to lost with placement explaining why nothing can run.

config.Model and catalog.Model were the same thing described twice, so the
router now reads catalog entries directly and the validation rules live in one
place.

Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted
network until that lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
ilkerandClaude Opus 5 committed 2026-09-16 02:48:21 +02:00
1 parent c1b9ea1bc2
commit 2373726efd
24 files changed
+3645 -87

No files matched your search

File diff suppressed because it is too large. Load diff
@@ -0,0 +1,119 @@
// Code generated by protoc-gen-connect-go. DO NOT EDIT.
//
// Source: llmgw/v1/agent.proto
// Package llmgw.v1 is the protocol between a node agent and the control plane
// (plan §5.6).
//
// The agent dials out and holds a bidirectional stream open. That direction is
// deliberate: GPU nodes sit behind firewalls and NAT far more often than a
// control plane does, and an outbound connection needs no inbound rule on every
// node in the fleet.
package llmgwv1connect
import (
connect "connectrpc.com/connect"
context "context"
errors "errors"
v1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
http "net/http"
strings "strings"
)
// This is a compile-time assertion to ensure that this generated file and the connect package are
// compatible. If you get a compiler error that this constant is not defined, this code was
// generated with a version of connect newer than the one compiled into your binary. You can fix the
// problem by either regenerating this code with an older version of connect or updating the connect
// version compiled into your binary.
const _ = connect.IsAtLeastVersion1_13_0
const (
// AgentServiceName is the fully-qualified name of the AgentService service.
AgentServiceName = "llmgw.v1.AgentService"
)
// These constants are the fully-qualified names of the RPCs defined in this package. They're
// exposed at runtime as Spec.Procedure and as the final two segments of the HTTP route.
//
// Note that these are different from the fully-qualified method names used by
// google.golang.org/protobuf/reflect/protoreflect. To convert from these constants to
// reflection-formatted method names, remove the leading slash and convert the remaining slash to a
// period.
const (
// AgentServiceConnectProcedure is the fully-qualified name of the AgentService's Connect RPC.
AgentServiceConnectProcedure = "/llmgw.v1.AgentService/Connect"
)
// AgentServiceClient is a client for the llmgw.v1.AgentService service.
type AgentServiceClient interface {
// Connect opens the agent's stream. It stays open for the agent's lifetime;
// losing it is how each side learns the other is gone.
Connect(context.Context) *connect.BidiStreamForClient[v1.AgentMessage, v1.ControlMessage]
}
// NewAgentServiceClient constructs a client for the llmgw.v1.AgentService service. By default, it
// uses the Connect protocol with the binary Protobuf Codec, asks for gzipped responses, and sends
// uncompressed requests. To use the gRPC or gRPC-Web protocols, supply the connect.WithGRPC() or
// connect.WithGRPCWeb() options.
//
// The URL supplied here should be the base URL for the Connect or gRPC server (for example,
// http://api.acme.com or https://acme.com/grpc).
func NewAgentServiceClient(httpClient connect.HTTPClient, baseURL string, opts ...connect.ClientOption) AgentServiceClient {
baseURL = strings.TrimRight(baseURL, "/")
agentServiceMethods := v1.File_llmgw_v1_agent_proto.Services().ByName("AgentService").Methods()
return &agentServiceClient{
connect: connect.NewClient[v1.AgentMessage, v1.ControlMessage](
httpClient,
baseURL+AgentServiceConnectProcedure,
connect.WithSchema(agentServiceMethods.ByName("Connect")),
connect.WithClientOptions(opts...),
),
}
}
// agentServiceClient implements AgentServiceClient.
type agentServiceClient struct {
connect *connect.Client[v1.AgentMessage, v1.ControlMessage]
}
// Connect calls llmgw.v1.AgentService.Connect.
func (c *agentServiceClient) Connect(ctx context.Context) *connect.BidiStreamForClient[v1.AgentMessage, v1.ControlMessage] {
return c.connect.CallBidiStream(ctx)
}
// AgentServiceHandler is an implementation of the llmgw.v1.AgentService service.
type AgentServiceHandler interface {
// Connect opens the agent's stream. It stays open for the agent's lifetime;
// losing it is how each side learns the other is gone.
Connect(context.Context, *connect.BidiStream[v1.AgentMessage, v1.ControlMessage]) error
}
// NewAgentServiceHandler builds an HTTP handler from the service implementation. It returns the
// path on which to mount the handler and the handler itself.
//
// By default, handlers support the Connect, gRPC, and gRPC-Web protocols with the binary Protobuf
// and JSON codecs. They also support gzip compression.
func NewAgentServiceHandler(svc AgentServiceHandler, opts ...connect.HandlerOption) (string, http.Handler) {
agentServiceMethods := v1.File_llmgw_v1_agent_proto.Services().ByName("AgentService").Methods()
agentServiceConnectHandler := connect.NewBidiStreamHandler(
AgentServiceConnectProcedure,
svc.Connect,
connect.WithSchema(agentServiceMethods.ByName("Connect")),
connect.WithHandlerOptions(opts...),
)
return "/llmgw.v1.AgentService/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case AgentServiceConnectProcedure:
agentServiceConnectHandler.ServeHTTP(w, r)
default:
http.NotFound(w, r)
}
})
}
// UnimplementedAgentServiceHandler returns CodeUnimplemented from all methods.
type UnimplementedAgentServiceHandler struct{}
func (UnimplementedAgentServiceHandler) Connect(context.Context, *connect.BidiStream[v1.AgentMessage, v1.ControlMessage]) error {
return connect.NewError(connect.CodeUnimplemented, errors.New("llmgw.v1.AgentService.Connect is not implemented"))
}