The second half of leadership. A standby that refuses to place but still holds an agent's stream is worse than useless: commands travel down the stream the agent opened, so the leader's placements fail with "no agent connected" for a node that is plainly connected. So a standby refuses the stream, before reading the opening message, and says why. And an agent takes a list rather than an address, trying each in turn — a standby refusing, a stopped control plane refusing the connection and a crashed one never answering are the same thing from a node, and the answer to all three is somewhere else. `server` and `servers` are both accepted, with `server` first when both are given: every agent configuration written before there could be a second control plane names one address, and an operator adding a standby expects the one already working to stay preferred. With a single address the behaviour is exactly what it was, including the backoff — which is skipped only while there is another address that has not just been tried, because the delay exists to stop hammering one control plane and the next one has not been hammered. Removed a check that asked whether one field was empty, which is how the first run of these tests failed: the list was populated and `server` was not, and an agent configured entirely correctly was refused by its own validation. The tests run a real agent against a real service over h2c. The first attempt used a plain HTTP server, and the refusal arrived as an HTTP/2 framing error that said nothing about why — which is exactly what an operator would have seen. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
ade689fc8e
commit
4ba84fa90e
6 files changed
+312
-26
No files matched your search
@@ -31,6 +31,9 @@ type StatusSink interface {
|
||||
// time the reconciler's Dispatcher: it holds the open streams, so it is the
|
||||
// only thing that can deliver a command to a node.
|
||||
type Service struct {
|
||||
// leader gates whether this process accepts agent streams at all.
|
||||
leader Leader
|
||||
|
||||
inv *inventory.Store
|
||||
status StatusSink
|
||||
log *slog.Logger
|
||||
@@ -86,6 +89,19 @@ func (s *Service) Connect(
|
||||
ctx context.Context,
|
||||
stream *connect.BidiStream[llmgwv1.AgentMessage, llmgwv1.ControlMessage],
|
||||
) error {
|
||||
// A standby must not hold an agent's stream. Commands travel down the
|
||||
// stream the agent opened, so whichever control plane holds it is the only
|
||||
// one that can start or stop anything on that node — and if that is not
|
||||
// the leader, the leader's placements fail with "no agent connected" for a
|
||||
// node that is plainly connected.
|
||||
//
|
||||
// Refused before the opening message is read, so the agent spends nothing
|
||||
// on a connection that cannot serve it.
|
||||
if s.leader != nil && !s.leader.IsLeader() {
|
||||
return connect.NewError(connect.CodeUnavailable,
|
||||
errors.New("this control plane is a standby; connect to the leader"))
|
||||
}
|
||||
|
||||
first, err := stream.Receive()
|
||||
if err != nil {
|
||||
return connect.NewError(connect.CodeInvalidArgument, fmt.Errorf("no opening message: %w", err))
|
||||
@@ -345,6 +361,16 @@ func (s *Service) ConnectedNodes() []string {
|
||||
|
||||
// --- Dispatcher ---
|
||||
|
||||
// Leader reports whether this process is the one that should be holding agent
|
||||
// streams.
|
||||
type Leader interface{ IsLeader() bool }
|
||||
|
||||
// SetLeadership makes this service refuse agents while it is a standby.
|
||||
//
|
||||
// Nil accepts them unconditionally, which is what a single control plane wants
|
||||
// and what every deployment before leadership had.
|
||||
func (s *Service) SetLeadership(l Leader) { s.leader = l }
|
||||
|
||||
// ErrNotConnected means no agent stream is open for that node.
|
||||
var ErrNotConnected = errors.New("no agent connected")
|
||||
|
||||
|
||||
Reference in new issue
Block a user