The nodes' agent settings kept with the cluster's, agent.yaml down to what an agent needs to start

agents.default and agents.nodes.<id> in the cluster's settings hold
everything agent.yaml can say but the agent's id, control planes, address and
certificate, which stay in the file and are refused there. The control plane
sends each node its settings on register and, a second apart, when they
change; an agent whose settings changed checks they load beside its file,
keeps them and starts again on them, engines serving throughout. An edited
agent.yaml that loads restarts the agent too. History, audit, ctl config, the
console and the portal's settings page cover them as the rest of the
settings.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

BACKLOG.md

@@ -807,11 +807,26 @@ Patroni configuration. The firewall's allowed set is the cluster's membership
 removed changes every machine's rules. Retires `firewall.Admit` and
 `firewall_networks` in server.yaml (imported into the store).
 
-### E3. agent.yaml down to bootstrap (M)
+### E3. agent.yaml down to bootstrap (M) — done 2026-10-08 (branch `settings-e`)
 agent.yaml keeps the control planes' addresses, its certificate and its id.
 Labels, fabrics, engines, heartbeat and supervisor settings move to the
 store per node and reach the agent over its stream; a change to the file is
 noticed and the agent restarts itself.
+Built as a section of the cluster's settings rather than a store of its own,
+so history, audit, `ctl config`, the console and the portal page cover it:
+`agents.default` and `agents.nodes.<id>` (config.AgentSettings; bootstrap
+keys node_id/server(s)/hostname/mgmt_addr/tls refused; each node's merge
+loaded as an agent would, so a bad fabric is refused on save). A live key:
+the control plane sends ControlMessage.AgentSettings on register and, a
+second apart per node, on change; the agent compares with what it started on,
+checks it loads beside its agent.yaml (else logs and keeps running), keeps it
+in /var/lib/llmgw-agent/agent-settings.yaml and restarts (engines adopted).
+At start: agent.yaml < kept settings < LLMGW_ env; kept settings that no
+longer load are set aside, never a reason not to start. agent.yaml is
+watched every 10 s and restarted on when it changes and loads. Left: file
+keys are not imported into the settings (they keep working under them);
+`node add` still writes a full agent.yaml; labels have two homes now
+(`ctl label`, which wins, and agents.*.labels).
 
 ### E4. Web settings page (S) — done 2026-10-08 (branch `settings-e`)
 The same document, edited on /admin/settings, for whoever has no terminal.

CHANGELOG.md

@@ -33,6 +33,14 @@ APIs change between minors without a deprecation period.
   checked as a control plane would load it, refused when made from a
   version since replaced, with the history to view and roll back. Changes
   are audited by size, never by contents.
+- **The nodes' agent settings in the cluster's settings** (BACKLOG E3):
+  `agents.default` and `agents.nodes.<id>` hold everything agent.yaml can
+  say but the agent's id, control planes, address and certificate. They are
+  sent to the agents as they change; an agent starts again on new settings,
+  or on an edited agent.yaml, with its engines serving throughout. Settings
+  that would not load are refused on save, and set aside on a node rather
+  than keep its agent from starting. Nodes need this release to follow
+  them.
 
 ## v0.14.0 — 2026-10-08
 

api/proto/llmgw/v1/agent.proto

@@ -73,6 +73,7 @@ message ControlMessage {
     ControlPlanes control_planes = 11;
     Firewall firewall = 13;
     RestartAgent restart_agent = 14;
+    AgentSettings agent_settings = 15;
   }
   // trace is the W3C trace context (traceparent, and tracestate when there
   // is one) of the control plane's span that sent this command, so what the
@@ -103,6 +104,17 @@ message Firewall {
   int32 engine_ports_to = 4;
 }
 
+// AgentSettings is what the cluster's settings say of this node's agent
+// (agents.default, with agents.nodes.<id> over it), sent when it registers
+// and whenever they change. The agent keeps it beside agent.yaml, which
+// holds only what it needs to start; over that file, it is what the agent
+// runs on. A change from what it runs on has the agent start again on it,
+// its engines serving throughout.
+message AgentSettings {
+  // yaml is agent.yaml's keys, as YAML; empty for none.
+  string yaml = 1;
+}
+
 // ControlPlanes is every control plane an agent may dial, sent by the leader
 // when an agent registers and whenever the list changes. The agent keeps it
 // beside its configuration, so a second control plane added after the node

docs/install.md

@@ -275,6 +275,33 @@ cluster keys — with a `baseline` guard policy, if the file had none — and
 from then on those keys in any server.yaml are not read (the server says so
 when it starts).
 
+**The nodes' agents** are set the same way, under `agents`. `agents.default`
+is for every node, and `agents.nodes.<node-id>` is for one node, laid over
+the default:
+
+```
+llmgw ctl config set agents.default.heartbeat_interval 5s
+llmgw ctl config set agents.nodes.cuda01.labels.gpu rtx6000
+llmgw ctl config edit                    # fabrics, engines, supervisor, discovery…
+```
+
+Each node's `agent.yaml` then needs only what its agent must know before it
+can be told anything: `node_id`, `server` or `servers`, `hostname`,
+`mgmt_addr` and `tls`. Those keys are refused under `agents`.
+
+- **Precedence.** Anything else in the file is still read, with the
+  cluster's settings over it.
+- **Applying a change.** The control plane sends each node its settings, a
+  second apart. An agent whose settings changed keeps them beside its
+  `agent.yaml` (in `/var/lib/llmgw-agent/agent-settings.yaml`) and starts
+  again on them, its engines serving throughout.
+- **Bad settings.** Settings that would not load are refused when they are
+  saved. If they somehow do not load beside one node's own file, that agent
+  keeps running on what it has and logs why.
+- **Editing `agent.yaml` itself.** The agent notices the change within ten
+  seconds and starts again on it. If the edited file does not load, the
+  agent reports it and keeps running on what it had.
+
 **This machine's own keys** stay in `/etc/llmgw/server.yaml`: `listen`,
 `admin_listen`, `metrics_listen`, `agent_listen`, `agent_tls`, `auth.store`,
 `secrets`, `tracing`, `monitoring`, `mirror.api` and `mirror.tls`, the backup

docs/reference.md

@@ -294,6 +294,7 @@ rather than repeated.
 | `mail` | `mail.Config` | [Server](#server) |
 | `serving` | `ServingSettings` | [Server](#server) |
 | `firewall` | `FirewallSettings` | [Server](#server) |
+| `agents` | `AgentSettings` | [Server](#server) |
 | `firewall_networks` | `[]string` | [Server](#server) |
 | `backup_recipients` | `[]string` | [Server](#server) |
 | `backup_copy_to` | `[]string` | [Server](#server) |
@@ -2520,6 +2521,16 @@ Firewall turns llmgw's firewall on every machine on or off.
 
 Takes a [FirewallSettings](#firewallsettings) block.
 
+### `agents`
+
+`AgentSettings`
+
+Agents are the nodes' agent settings: agent.yaml's keys but those an
+agent needs to start, for every node (default) and for one
+(nodes.<id>). Sent to the agents as they change; an agent whose
+settings changed starts again on them, its engines serving
+throughout.
+
 ### `firewall_networks`
 
 `[]string`

internal/agent/agent.go

@@ -136,6 +136,10 @@ type Agent struct {
 	// by mu.
 	kv     *kvevents.Mirror
 	kvSubs map[string]*kvevents.Subscriber
+
+	// settingsCheck says whether the cluster's settings for this agent
+	// would load beside its agent.yaml; see SetSettingsCheck.
+	settingsCheck func([]byte) error
 }
 
 // connectHTTPClient is the HTTP client connect uses; an interface so tests can
@@ -880,6 +884,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 		a.setControlPlanes(m.ControlPlanes)
 	case *llmgwv1.ControlMessage_Firewall:
 		go a.applyFirewall(m.Firewall)
+	case *llmgwv1.ControlMessage_AgentSettings:
+		go a.applySettings(m.AgentSettings)
 	case *llmgwv1.ControlMessage_RestartAgent:
 		go func() {
 			a.log.Info("restarting, as the control plane asks; the engines keep serving",

internal/agent/config.go

@@ -103,6 +103,11 @@ type Config struct {
 	Tracing obs.Tracing `koanf:"tracing"`
 
 	Version string
+
+	// Settings is the cluster's settings for this agent it was started on,
+	// as the control plane sent them: what a newly sent version is
+	// compared with.
+	Settings string `koanf:"-"`
 }
 
 // DefaultConfig returns sensible defaults.

internal/agent/settings.go

@@ -0,0 +1,109 @@
+package agent
+
+import (
+	"context"
+	"crypto/sha256"
+	"os"
+	"path/filepath"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// The cluster's settings for this agent (BACKLOG E3): what agent.yaml can
+// say but the agent's id, control planes, address and certificate, kept by
+// the control planes and sent over the stream. Kept here beside agent.yaml,
+// which then holds only what the agent needs to start; at start the two are
+// read together, these over the file.
+
+// SettingsFile keeps the settings the control plane last sent.
+var SettingsFile = "/var/lib/llmgw-agent/agent-settings.yaml"
+
+// ReadSettings is what SettingsFile holds, nil when nothing.
+func ReadSettings() []byte {
+	b, err := os.ReadFile(SettingsFile)
+	if err != nil {
+		return nil
+	}
+	return b
+}
+
+// SetSettingsCheck says how to tell whether settings would load beside this
+// agent's own file; settings that would not are not taken.
+func (a *Agent) SetSettingsCheck(f func([]byte) error) { a.settingsCheck = f }
+
+// applySettings takes the settings the control plane sent: nothing when they
+// are what this agent runs on, and otherwise kept and started again on.
+func (a *Agent) applySettings(m *llmgwv1.AgentSettings) {
+	next := m.GetYaml()
+	if next == a.cfg.Settings {
+		return
+	}
+	if a.settingsCheck != nil {
+		if err := a.settingsCheck([]byte(next)); err != nil {
+			// Kept running on what it has: settings that will not load
+			// here must not take the node out of service.
+			a.log.Error("the cluster's settings for this agent do not load beside its agent.yaml; running on what it has",
+				"err", err)
+			return
+		}
+	}
+	if err := os.MkdirAll(filepath.Dir(SettingsFile), 0o755); err != nil {
+		a.log.Error("could not keep the cluster's settings for this agent", "err", err)
+		return
+	}
+	tmp := SettingsFile + ".new"
+	if err := os.WriteFile(tmp, []byte(next), 0o600); err != nil {
+		a.log.Error("could not keep the cluster's settings for this agent", "err", err)
+		return
+	}
+	if err := os.Rename(tmp, SettingsFile); err != nil {
+		a.log.Error("could not keep the cluster's settings for this agent", "err", err)
+		return
+	}
+	a.log.Info("the cluster's settings for this agent changed; starting again on them, the engines keep serving")
+	time.Sleep(300 * time.Millisecond)
+	exitForUpgrade()
+}
+
+// WatchFile starts the agent again when its configuration file changes and
+// still loads: agent.yaml is read only at start, and a change to it that
+// waited for somebody to restart the agent was a change nobody saw take
+// effect.
+func (a *Agent) WatchFile(ctx context.Context, path string, loads func() error) {
+	if path == "" {
+		return
+	}
+	sum := func() [32]byte {
+		b, _ := os.ReadFile(path) //nolint:gosec // the agent's own configuration
+		return sha256.Sum256(b)
+	}
+	was := sum()
+	t := time.NewTicker(watchEvery)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+		}
+		now := sum()
+		if now == was {
+			continue
+		}
+		if err := loads(); err != nil {
+			// Once per change: an editor saving halfway must not stop the
+			// agent, and a broken file is reported, not restarted into.
+			a.log.Error("agent.yaml changed and does not load; running on what it had", "path", path, "err", err)
+			was = now
+			continue
+		}
+		a.log.Info("agent.yaml changed; starting again on it, the engines keep serving", "path", path)
+		time.Sleep(300 * time.Millisecond)
+		exitForUpgrade()
+		return
+	}
+}
+
+// watchEvery is how often agent.yaml is looked at.
+var watchEvery = 10 * time.Second

internal/agent/settings_test.go

@@ -0,0 +1,88 @@
+package agent
+
+import (
+	"errors"
+	"os"
+	"path/filepath"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+func settingsAgent(t *testing.T, running string) (*Agent, chan struct{}) {
+	t.Helper()
+	oldFile, oldExit := SettingsFile, exitForUpgrade
+	SettingsFile = filepath.Join(t.TempDir(), "agent-settings.yaml")
+	exited := make(chan struct{}, 1)
+	exitForUpgrade = func() { exited <- struct{}{} }
+	t.Cleanup(func() { SettingsFile, exitForUpgrade = oldFile, oldExit })
+	cfg := Config{Server: "127.0.0.1:7443", NodeID: "n1", WeightsCache: t.TempDir(), EnginesDir: t.TempDir(), Settings: running}
+	cfg.Superv.WorkDir = t.TempDir()
+	a, err := New(cfg, nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return a, exited
+}
+
+// exitedSoon is whether the agent exited within a second: it waits a
+// moment for its log line to leave first.
+func exitedSoon(ch chan struct{}) bool {
+	select {
+	case <-ch:
+		return true
+	case <-time.After(time.Second):
+		return false
+	}
+}
+
+func TestAnAgentOnItsSettingsStaysAndOneOffThemStartsAgain(t *testing.T) {
+	a, exited := settingsAgent(t, "heartbeat_interval: 5s\n")
+	a.applySettings(&llmgwv1.AgentSettings{Yaml: "heartbeat_interval: 5s\n"})
+	if exitedSoon(exited) || ReadSettings() != nil {
+		t.Error("settings it already runs on restarted it")
+	}
+	a.applySettings(&llmgwv1.AgentSettings{Yaml: "heartbeat_interval: 2s\n"})
+	if !exitedSoon(exited) || string(ReadSettings()) != "heartbeat_interval: 2s\n" {
+		t.Errorf("new settings: kept %q", ReadSettings())
+	}
+}
+
+func TestSettingsThatWouldNotLoadAreNotTaken(t *testing.T) {
+	a, exited := settingsAgent(t, "")
+	a.SetSettingsCheck(func([]byte) error { return errors.New("fabric ib-a lists no HCAs") })
+	a.applySettings(&llmgwv1.AgentSettings{Yaml: "fabrics: [{name: ib-a}]\n"})
+	if exitedSoon(exited) || ReadSettings() != nil {
+		t.Error("settings that do not load were taken")
+	}
+}
+
+func TestAChangedAgentYamlThatLoadsStartsTheAgentAgain(t *testing.T) {
+	a, exited := settingsAgent(t, "")
+	old := watchEvery
+	watchEvery = 10 * time.Millisecond
+	t.Cleanup(func() { watchEvery = old })
+	p := filepath.Join(t.TempDir(), "agent.yaml")
+	_ = os.WriteFile(p, []byte("node_id: n1\n"), 0o600)
+	var loads atomic.Bool
+	check := func() error {
+		if loads.Load() {
+			return nil
+		}
+		return errors.New("broken")
+	}
+	go a.WatchFile(t.Context(), p, check)
+	time.Sleep(30 * time.Millisecond)
+	// Saved broken: reported, not restarted into.
+	_ = os.WriteFile(p, []byte("node_id: [\n"), 0o600)
+	if exitedSoon(exited) {
+		t.Fatal("a broken agent.yaml restarted the agent")
+	}
+	loads.Store(true)
+	_ = os.WriteFile(p, []byte("node_id: n1\nlabels: {a: b}\n"), 0o600)
+	if !exitedSoon(exited) {
+		t.Error("a changed agent.yaml did not restart the agent")
+	}
+}

internal/cli/agent.go

@@ -40,10 +40,23 @@ func newAgentCmd() *cobra.Command {
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			log := logger()
 
-			cfg, err := config.LoadAgent(configPath)
+			// The cluster's settings for this agent over its file, as the
+			// control plane last sent them. Settings that no longer load
+			// beside the file are set aside rather than keeping the agent
+			// from starting: the control plane sends them again, and says
+			// why they are refused.
+			settings := agent.ReadSettings()
+			cfg, err := config.LoadAgentWith(configPath, settings)
+			if err != nil && settings != nil {
+				log.Warn("the cluster's settings kept for this agent do not load beside its agent.yaml; starting without them",
+					"err", err)
+				settings = nil
+				cfg, err = config.LoadAgent(configPath)
+			}
 			if err != nil {
 				return err
 			}
+			cfg.Settings = string(settings)
 			if server != "" {
 				cfg.Server = server
 			}
@@ -89,9 +102,17 @@ func newAgentCmd() *cobra.Command {
 			if err != nil {
 				return err
 			}
+			a.SetSettingsCheck(func(b []byte) error {
+				_, err := config.LoadAgentWith(configPath, b)
+				return err
+			})
 
 			ctx, stop := signal.NotifyContext(cmd.Context(), syscall.SIGINT, syscall.SIGTERM)
 			defer stop()
+			go a.WatchFile(ctx, configPath, func() error {
+				_, err := config.LoadAgentWith(configPath, agent.ReadSettings())
+				return err
+			})
 
 			log.Info("agent starting",
 				"node", cfg.NodeID, "server", cfg.Server,

internal/cli/clustersettings.go

@@ -241,6 +241,8 @@ var liveSettings = []string{
 	"serving",
 	// Sent to the nodes and this machine's firewall unit as they change.
 	"firewall", "firewall_networks",
+	// Sent to the nodes' agents as they change.
+	"agents",
 	// Read by server init, node add and backups, not by the server.
 	"backup_recipients", "backup_copy_to",
 }

internal/cli/server.go

@@ -263,6 +263,15 @@ func newServerCmd() *cobra.Command {
 					return &llmgwv1.Firewall{Enabled: c.Firewall.Enabled, Networks: c.FirewallNetworks,
 						EnginePortsFrom: int32(from), EnginePortsTo: int32(to)} //nolint:gosec // ports
 				})
+				// What each node's agent runs on, as the settings say now.
+				svc.SetAgentSettings(func(node string) string {
+					b, err := config.Marshal(current.Load().Agents.For(node))
+					if err != nil {
+						log.Warn("could not write a node's agent settings", "node", node, "err", err)
+						return ""
+					}
+					return string(b)
+				})
 				ctl.Reconciler().SetNotifier(notifier)
 				admin = ctl
 				agentPath, agentHandler = svc.Handler()
@@ -719,6 +728,10 @@ func newServerCmd() *cobra.Command {
 							agentSvc.BroadcastFirewall()
 						}
 					}
+					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "agents") }) && agentSvc != nil {
+						// A moment apart per node, so off this goroutine.
+						go agentSvc.BroadcastAgentSettings()
+					}
 					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "guard") }) {
 						if gate == nil {
 							// Built at start, and only when there are policies.

internal/config/agent.go

@@ -18,7 +18,12 @@ import (
 //
 // An empty path loads the defaults, which with --server and --node-id on the
 // command line is enough for a smoke test on a node with no IB.
-func LoadAgent(path string) (agent.Config, error) {
+func LoadAgent(path string) (agent.Config, error) { return LoadAgentWith(path, nil) }
+
+// LoadAgentWith is LoadAgent with the cluster's settings for this node
+// (agents.default and agents.nodes.<id>, as YAML) laid over the file: the
+// file is what the agent needs to start, the settings what it runs on.
+func LoadAgentWith(path string, settings []byte) (agent.Config, error) {
 	cfg := agent.DefaultConfig()
 	k := koanf.New(".")
 
@@ -30,6 +35,11 @@ func LoadAgent(path string) (agent.Config, error) {
 			return cfg, fmt.Errorf("parsing %s: %w", path, err)
 		}
 	}
+	if len(settings) > 0 {
+		if err := k.Load(bytesProvider(settings), yaml.Parser()); err != nil {
+			return cfg, fmt.Errorf("parsing the cluster's settings for this agent: %w", err)
+		}
+	}
 
 	err := k.Load(env.Provider(".", env.Opt{
 		Prefix: "LLMGW_",

internal/config/agents.go

@@ -0,0 +1,110 @@
+package config
+
+import (
+	"fmt"
+	"slices"
+	"sort"
+)
+
+// AgentSettings are the nodes' agent settings, kept with the cluster's
+// (BACKLOG E3): everything agent.yaml can say except what an agent needs to
+// start — its id, the control planes to dial, its address and its
+// certificate. Default applies to every node, and Nodes, by node id, over
+// it; both over the node's own agent.yaml.
+type AgentSettings struct {
+	// Default is agent.yaml's keys for every node.
+	Default map[string]any `koanf:"default"`
+	// Nodes are agent.yaml's keys for one node each, by node id, over
+	// Default.
+	Nodes map[string]map[string]any `koanf:"nodes"`
+}
+
+// agentBootstrapKeys are what an agent needs before it can be told anything:
+// they stay in agent.yaml.
+var agentBootstrapKeys = []string{"node_id", "server", "servers", "hostname", "mgmt_addr", "tls"}
+
+// For is the settings for one node: Default with the node's own over it.
+// Empty when there are none.
+func (a AgentSettings) For(node string) map[string]any {
+	out := clone(stringKeys(a.Default))
+	if n, ok := a.Nodes[node]; ok {
+		merge(out, stringKeys(n))
+	}
+	return out
+}
+
+// Check refuses settings that set an agent's bootstrap keys, or that would
+// not load as an agent's configuration for some node.
+func (a AgentSettings) Check() error {
+	check := func(where string, m map[string]any) error {
+		for _, k := range agentBootstrapKeys {
+			if _, ok := m[k]; ok {
+				return fmt.Errorf("%s.%s stays in each node's agent.yaml: an agent needs it before it can be told anything", where, k)
+			}
+		}
+		return nil
+	}
+	if err := check("agents.default", stringKeys(a.Default)); err != nil {
+		return err
+	}
+	nodes := make([]string, 0, len(a.Nodes))
+	for n := range a.Nodes {
+		nodes = append(nodes, n)
+	}
+	sort.Strings(nodes)
+	for _, n := range nodes {
+		if err := check("agents.nodes."+n, stringKeys(a.Nodes[n])); err != nil {
+			return err
+		}
+	}
+	// Each node's settings as its agent would load them, over nothing: what
+	// its own file adds cannot be known here, and what is wrong in the
+	// settings alone is wrong on every node.
+	for _, n := range append([]string{""}, nodes...) {
+		b, err := Marshal(a.For(n))
+		if err != nil {
+			return err
+		}
+		if _, err := LoadAgentWith("", b); err != nil {
+			where := "agents.default"
+			if n != "" {
+				where = "agents.nodes." + n
+			}
+			return fmt.Errorf("%s: %w", where, err)
+		}
+	}
+	return nil
+}
+
+// AgentBootstrapKeys are the keys agent.yaml keeps, for saying so.
+func AgentBootstrapKeys() []string { return slices.Clone(agentBootstrapKeys) }
+
+// stringKeys turns YAML's map[any]any, wherever it hides, into the
+// map[string]any the rest of this package merges.
+func stringKeys(m map[string]any) map[string]any {
+	out := make(map[string]any, len(m))
+	for k, v := range m {
+		out[k] = normalize(v)
+	}
+	return out
+}
+
+func normalize(v any) any {
+	switch t := v.(type) {
+	case map[string]any:
+		return stringKeys(t)
+	case map[any]any:
+		m := make(map[string]any, len(t))
+		for k, vv := range t {
+			m[fmt.Sprint(k)] = normalize(vv)
+		}
+		return m
+	case []any:
+		out := make([]any, len(t))
+		for i, vv := range t {
+			out[i] = normalize(vv)
+		}
+		return out
+	}
+	return v
+}

internal/config/agents_test.go

@@ -0,0 +1,79 @@
+package config
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestANodesAgentSettingsAreTheDefaultWithItsOwnOver(t *testing.T) {
+	a := AgentSettings{
+		Default: map[string]any{"heartbeat_interval": "5s", "supervisor": map[string]any{"work_dir": "/srv/w", "x": 1}},
+		Nodes: map[string]map[string]any{
+			"cuda01": {"supervisor": map[string]any{"work_dir": "/fast/w"}, "labels": map[string]any{"gpu": "rtx6000"}},
+		},
+	}
+	got := a.For("cuda01")
+	sup := got["supervisor"].(map[string]any)
+	if sup["work_dir"] != "/fast/w" || sup["x"] != 1 || got["heartbeat_interval"] != "5s" || got["labels"] == nil {
+		t.Errorf("cuda01: %v", got)
+	}
+	if got := a.For("node01"); got["labels"] != nil || got["supervisor"].(map[string]any)["work_dir"] != "/srv/w" {
+		t.Errorf("node01: %v", got)
+	}
+	if got := (AgentSettings{}).For("x"); len(got) != 0 {
+		t.Errorf("none: %v", got)
+	}
+}
+
+func TestAgentSettingsKeepBootstrapKeysInTheFileAndMustLoad(t *testing.T) {
+	for _, a := range []AgentSettings{
+		{Default: map[string]any{"servers": []any{"x:7443"}}},
+		{Nodes: map[string]map[string]any{"n1": {"node_id": "other"}}},
+		{Nodes: map[string]map[string]any{"n1": {"tls": map[string]any{"ca": "/x"}}}},
+	} {
+		if err := a.Check(); err == nil || !strings.Contains(err.Error(), "stays in each node's agent.yaml") {
+			t.Errorf("%+v: %v", a, err)
+		}
+	}
+	// A fabric with no adapters would not load on the node it names.
+	bad := AgentSettings{Nodes: map[string]map[string]any{"n1": {"fabrics": []any{map[string]any{"name": "ib-a"}}}}}
+	if err := bad.Check(); err == nil || !strings.Contains(err.Error(), "agents.nodes.n1") || !strings.Contains(err.Error(), "no HCAs") {
+		t.Errorf("a fabric without HCAs: %v", err)
+	}
+	if err := (AgentSettings{Default: map[string]any{"heartbeat_interval": "3s"}}).Check(); err != nil {
+		t.Errorf("a plain setting: %v", err)
+	}
+}
+
+func TestTheClusterSettingsCarryAgentsAndRefuseBadOnes(t *testing.T) {
+	good, err := Parse([]byte("agents:\n  default:\n    heartbeat_interval: 3s\n  nodes:\n    cuda01:\n      labels: {gpu: rtx6000}\n"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	cfg, err := FromMap(good)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.Agents.For("cuda01")["labels"] == nil {
+		t.Errorf("agents not read: %+v", cfg.Agents)
+	}
+	bad, _ := Parse([]byte("agents:\n  default:\n    mgmt_addr: 10.0.0.1\n"))
+	if _, err := FromMap(bad); err == nil {
+		t.Error("a bootstrap key in the cluster's settings was taken")
+	}
+}
+
+func TestTheSettingsAreLaidOverTheAgentsFile(t *testing.T) {
+	p := filepath.Join(t.TempDir(), "agent.yaml")
+	_ = os.WriteFile(p, []byte("node_id: n1\nserver: cp:7443\nheartbeat_interval: 9s\nlabels: {rack: r1}\n"), 0o600)
+	cfg, err := LoadAgentWith(p, []byte("heartbeat_interval: 2s\nlabels: {gpu: a100}\n"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cfg.NodeID != "n1" || cfg.HeartbeatInterval != 2*time.Second || cfg.Labels["rack"] != "r1" || cfg.Labels["gpu"] != "a100" {
+		t.Errorf("loaded %+v", cfg)
+	}
+}

internal/config/config.go

@@ -158,6 +158,13 @@ type Server struct {
 	// Firewall turns llmgw's firewall on every machine on or off.
 	Firewall FirewallSettings `koanf:"firewall"`
 
+	// Agents are the nodes' agent settings: agent.yaml's keys but those an
+	// agent needs to start, for every node (default) and for one
+	// (nodes.<id>). Sent to the agents as they change; an agent whose
+	// settings changed starts again on them, its engines serving
+	// throughout.
+	Agents AgentSettings `koanf:"agents"`
+
 	// FirewallNetworks are networks, besides those each machine reaches the
 	// cluster on, that llmgw's internal ports answer: a monitoring host on
 	// another subnet, nodes behind a router. Written into every machine's
@@ -887,6 +894,9 @@ func load(b []byte) (Server, error) {
 // Validate rejects configurations that would otherwise fail confusingly at
 // request time.
 func (s Server) Validate() error {
+	if err := s.Agents.Check(); err != nil {
+		return err
+	}
 	for i, o := range s.EngineOverrides {
 		if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 {
 			return fmt.Errorf("engine_overrides[%d] picks no nodes: give it a gpu_class, nodes or labels", i)

internal/control/agentsvc/agentsettings.go

@@ -0,0 +1,50 @@
+package agentsvc
+
+import (
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// SetAgentSettings gives the service each node's agent settings from the
+// cluster's (BACKLOG E3), as YAML, read at each use so a change is what is
+// sent next.
+func (s *Service) SetAgentSettings(f func(node string) string) {
+	s.mu.Lock()
+	s.agentSettings = f
+	s.mu.Unlock()
+}
+
+func (s *Service) agentSettingsMsg(node string) *llmgwv1.ControlMessage {
+	s.mu.RLock()
+	f := s.agentSettings
+	s.mu.RUnlock()
+	if f == nil {
+		return nil
+	}
+	return &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_AgentSettings{
+		AgentSettings: &llmgwv1.AgentSettings{Yaml: f(node)}}}
+}
+
+// agentSettingsGap is the pause between telling one node and the next: an
+// agent whose settings changed starts again, and a whole cluster's agents
+// doing so in the same second would all re-register at once.
+var agentSettingsGap = time.Second
+
+// BroadcastAgentSettings tells every connected agent what the cluster's
+// settings now say it runs on, a moment apart. A node not connected is told
+// when it registers.
+func (s *Service) BroadcastAgentSettings() {
+	for _, node := range s.ConnectedNodes() {
+		msg := s.agentSettingsMsg(node)
+		if msg == nil {
+			return
+		}
+		if err := s.send(node, msg); err != nil {
+			s.log.Warn("could not tell a node its agent settings; it is told when it next registers",
+				"node", node, "err", err)
+			continue
+		}
+		time.Sleep(agentSettingsGap)
+	}
+}

internal/control/agentsvc/agentsettings_test.go

@@ -0,0 +1,28 @@
+package agentsvc
+
+import (
+	"log/slog"
+	"testing"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
+)
+
+func TestEachNodeIsToldItsOwnAgentSettings(t *testing.T) {
+	s := New(inventory.NewStore(), nil, false, slog.New(slog.DiscardHandler))
+	if s.agentSettingsMsg("n1") != nil {
+		t.Error("told settings with nothing to read them from")
+	}
+	s.SetAgentSettings(func(node string) string {
+		if node == "cuda01" {
+			return "labels:\n  gpu: rtx6000\n"
+		}
+		return ""
+	})
+	if got := s.agentSettingsMsg("cuda01").GetAgentSettings().GetYaml(); got != "labels:\n  gpu: rtx6000\n" {
+		t.Errorf("cuda01 told %q", got)
+	}
+	// None is said too, so an agent whose settings were removed drops them.
+	if m := s.agentSettingsMsg("node01"); m == nil || m.GetAgentSettings().GetYaml() != "" {
+		t.Errorf("node01 told %v", m)
+	}
+}

internal/control/agentsvc/service.go

@@ -79,6 +79,9 @@ type Service struct {
 	controlPlanes func(context.Context) ([]string, error)
 	// firewall is what nodes are told of llmgw's firewall; see SetFirewall.
 	firewall func() *llmgwv1.Firewall
+	// agentSettings is what each node's agent is told to run on; see
+	// SetAgentSettings.
+	agentSettings func(node string) string
 
 	// logWaits are the log requests awaiting a reply, by request id.
 	logMu    sync.Mutex
@@ -296,6 +299,13 @@ func (s *Service) Connect(
 			return err
 		}
 	}
+	// What the cluster's settings say this agent runs on. An agent already
+	// on them ignores it; one that is not starts again on them.
+	if msg := s.agentSettingsMsg(nodeID); msg != nil {
+		if err := stream.Send(msg); err != nil {
+			return err
+		}
+	}
 
 	// Recover replica state from what this node is already running, before any
 	// reconciliation pass can conclude the cluster is empty and place

internal/docgen/reference_gen.go

@@ -372,6 +372,7 @@ var Reference = []Block{
 		{Key: "mail", Type: "mail.Config", Doc: "Mail is where the gateway's few emails go out from: a link to set a\nforgotten password. Unset, none is sent and people ask an operator.", Nested: "mail.Config"},
 		{Key: "serving", Type: "ServingSettings", Doc: "Serving stops and starts the whole cluster's service (llmgw stop, llmgw\nstart).", Nested: "config.ServingSettings"},
 		{Key: "firewall", Type: "FirewallSettings", Doc: "Firewall turns llmgw's firewall on every machine on or off.", Nested: "config.FirewallSettings"},
+		{Key: "agents", Type: "AgentSettings", Doc: "Agents are the nodes' agent settings: agent.yaml's keys but those an\nagent needs to start, for every node (default) and for one\n(nodes.<id>). Sent to the agents as they change; an agent whose\nsettings changed starts again on them, its engines serving\nthroughout.", Nested: "config.AgentSettings"},
 		{Key: "firewall_networks", Type: "[]string", Doc: "FirewallNetworks are networks, besides those each machine reaches the\ncluster on, that llmgw's internal ports answer: a monitoring host on\nanother subnet, nodes behind a router. Written into every machine's\nfirewall by `server init`, `node add` and `server add`.", Nested: ""},
 		{Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""},
 		{Key: "backup_copy_to", Type: "[]string", Doc: "BackupCopyTo is where each archive is copied once written, so it\nsurvives this machine: a directory (a mounted share, another disk) or\nssh://user@host[:port]/path, reached with BackupCopyKey and checked\nagainst /etc/llmgw/backup_known_hosts, which learns a host the first\ntime. The same number of archives is kept there as here. Refused\nwithout backup_recipients: an archive leaves the machine encrypted.", Nested: ""},