Let an agent find the leader instead of being told one address

The second half of leadership. A standby that refuses to place but still holds
an agent's stream is worse than useless: commands travel down the stream the
agent opened, so the leader's placements fail with "no agent connected" for a
node that is plainly connected.

So a standby refuses the stream, before reading the opening message, and says
why. And an agent takes a list rather than an address, trying each in turn — a
standby refusing, a stopped control plane refusing the connection and a crashed
one never answering are the same thing from a node, and the answer to all three
is somewhere else.

`server` and `servers` are both accepted, with `server` first when both are
given: every agent configuration written before there could be a second control
plane names one address, and an operator adding a standby expects the one
already working to stay preferred. With a single address the behaviour is
exactly what it was, including the backoff — which is skipped only while there
is another address that has not just been tried, because the delay exists to
stop hammering one control plane and the next one has not been hammered.

Removed a check that asked whether one field was empty, which is how the first
run of these tests failed: the list was populated and `server` was not, and an
agent configured entirely correctly was refused by its own validation.

The tests run a real agent against a real service over h2c. The first attempt
used a plain HTTP server, and the refusal arrived as an HTTP/2 framing error
that said nothing about why — which is exactly what an operator would have seen.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-25 19:30:06 +02:00
1 parent ade689fc8e
commit 4ba84fa90e
6 files changed
+312 -26

No files matched your search

+26
View File
@@ -31,6 +31,9 @@ type StatusSink interface {
// time the reconciler's Dispatcher: it holds the open streams, so it is the
// only thing that can deliver a command to a node.
type Service struct {
// leader gates whether this process accepts agent streams at all.
leader Leader
inv *inventory.Store
status StatusSink
log *slog.Logger
@@ -86,6 +89,19 @@ func (s *Service) Connect(
ctx context.Context,
stream *connect.BidiStream[llmgwv1.AgentMessage, llmgwv1.ControlMessage],
) error {
// A standby must not hold an agent's stream. Commands travel down the
// stream the agent opened, so whichever control plane holds it is the only
// one that can start or stop anything on that node — and if that is not
// the leader, the leader's placements fail with "no agent connected" for a
// node that is plainly connected.
//
// Refused before the opening message is read, so the agent spends nothing
// on a connection that cannot serve it.
if s.leader != nil && !s.leader.IsLeader() {
return connect.NewError(connect.CodeUnavailable,
errors.New("this control plane is a standby; connect to the leader"))
}
first, err := stream.Receive()
if err != nil {
return connect.NewError(connect.CodeInvalidArgument, fmt.Errorf("no opening message: %w", err))
@@ -345,6 +361,16 @@ func (s *Service) ConnectedNodes() []string {
// --- Dispatcher ---
// Leader reports whether this process is the one that should be holding agent
// streams.
type Leader interface{ IsLeader() bool }
// SetLeadership makes this service refuse agents while it is a standby.
//
// Nil accepts them unconditionally, which is what a single control plane wants
// and what every deployment before leadership had.
func (s *Service) SetLeadership(l Leader) { s.leader = l }
// ErrNotConnected means no agent stream is open for that node.
var ErrNotConnected = errors.New("no agent connected")