The firewall turns off and on with the cluster's settings; llmgw wipe

firewall.enabled (default true) is a cluster setting, applied live: the
leader sends a Firewall message to every agent when it registers and when
the setting or firewall_networks changes, and the agent (root) takes llmgw's
nft table out or installs it. A control plane writes the state to
/var/lib/llmgw/firewall.json, which llmgw-firewall-sync.path applies as root
with `llmgw firewall sync`; server init and server add install the units,
`llmgw firewall sync --install` on one set up before. node add and server
add put no firewall on a machine while the setting is off.

`llmgw wipe` removes llmgw from a machine after listing what goes and asking
for its name: units, engines' processes, the nft table, /etc/llmgw,
/var/lib/llmgw, the agent's state, the sudo grant, the CA from the trust
store, keepalived's configuration when llmgw wrote it, and the users.
--models, --store (Patroni member, etcd, or the local database) and --all
take more. The console has it under Do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

BACKLOG.md

@@ -793,7 +793,11 @@ leader 30s later with /health answering throughout; rollback did the same.
   facility state from the store (it kept what it read at start).
 - Retires `appendGuardIfMissing` (server init): the import does it.
 
-### E2. Generated files from settings (L)
+### E2. Generated files from settings (L) — firewall on/off done 2026-10-06
+`firewall.enabled` in the cluster's settings reaches nodes over the agent
+stream and control planes through llmgw-firewall-sync.path. Left: the mirror
+machine and store members, the allowed set from membership, the other files.
+
 Every machine runs the agent (root) with roles — node, control plane, mirror,
 store member — and the agent renders and applies what was written once by
 init: the llmgw nft table, the mirror's unit flags, keepalived, etcd and

CHANGELOG.md

@@ -41,6 +41,19 @@ APIs change between minors without a deprecation period.
   guardrails; after that, those keys in a server.yaml are ignored, with a
   warning.
 
+### Added
+
+- **llmgw's firewall can be turned off and on for the whole cluster**:
+  `llmgw ctl config set firewall.enabled false`. Every node's agent takes its
+  table out within seconds, and each control plane's new
+  `llmgw-firewall-sync.path` does the same there; `true` puts the rules back
+  as they were. `firewall_networks` changes reach the nodes the same way.
+- **`llmgw wipe`** removes llmgw from a machine — services, firewall table,
+  configuration, the authority, secrets, state, users — after listing it and
+  asking for the machine's name; `--models`, `--store` and `--all` take
+  weights, the store's database here, backups and the binary too. Also
+  **Do → Wipe this machine** in the console.
+
 ### Fixed
 
 - **A standby that becomes the leader reads the store again**: models added

api/proto/llmgw/v1/agent.proto

@@ -71,6 +71,7 @@ message ControlMessage {
     FabricTestRequest fabric_test = 9;
     UpgradeAgent upgrade = 10;
     ControlPlanes control_planes = 11;
+    Firewall firewall = 13;
   }
   // trace is the W3C trace context (traceparent, and tracestate when there
   // is one) of the control plane's span that sent this command, so what the
@@ -80,6 +81,21 @@ message ControlMessage {
   map<string, string> trace = 12;
 }
 
+// Firewall is whether llmgw's firewall is kept on the node — its nftables
+// table, which lets the cluster's networks alone reach the engines' ports and
+// the node exporter — sent when an agent registers and whenever the
+// cluster's settings change it (firewall.enabled).
+message Firewall {
+  bool enabled = 1;
+  // networks are let in besides those the node reaches the control planes
+  // on: firewall_networks, networks or single addresses.
+  repeated string networks = 2;
+  // engine_ports_from and engine_ports_to are the engines' ports
+  // (control.port_range).
+  int32 engine_ports_from = 3;
+  int32 engine_ports_to = 4;
+}
+
 // ControlPlanes is every control plane an agent may dial, sent by the leader
 // when an agent registers and whenever the list changes. The agent keeps it
 // beside its configuration, so a second control plane added after the node

docs/console.md

@@ -319,6 +319,9 @@ to restart it.
 - **Restart all replicas**, one at a time.
 - **Back up now** — the store and `/etc/llmgw` into one archive
   (`sudo llmgw server backup`).
+- **Wipe this machine** — removes llmgw's services, configuration, state and
+  users from this machine, after listing them and asking for its name
+  (`sudo llmgw wipe`; on a node too).
 - **Leave**.
 
 ## The node's console

docs/install.md

@@ -2142,6 +2142,22 @@ for nodes behind a router, goes in `server.yaml`:
 firewall_networks: [10.0.5.0/24, 192.168.1.20]
 ```
 
+**Turning it off and on.** Whether llmgw keeps this firewall is one of the
+cluster's settings:
+
+```
+llmgw ctl config set firewall.enabled false   # every machine's table taken out
+llmgw ctl config set firewall.enabled true    # and put back as it was
+```
+
+It is in force within seconds: each node's agent takes its table out or puts
+it back, and each control plane's `llmgw-firewall-sync.path` (installed by
+`server init` and `server add`; `sudo llmgw firewall sync --install` on one
+set up before) does the same there. The rules files stay, so turning it back
+on restores them. Nodes added while it is off get none. A separate mirror
+machine and the store's members keep theirs for now. `firewall_networks`
+changes reach the nodes the same way.
+
 Only those ports are touched. The table's policy is to accept, so whatever
 else the machine serves — and whatever firewall it already has — is left as
 it was; the client API (`:8080`) is not closed, being meant to be reached.
@@ -2278,6 +2294,31 @@ A cluster set up online becomes offline the same way: `bundle import` on the
 mirror and control plane, then the `offline:` block in `server.yaml` and a
 restart.
 
+## Removing llmgw from a machine
+
+```
+sudo llmgw wipe              # services, configuration, certificates, state, users
+sudo llmgw wipe --models     # and model weights, the mirror's files, engines
+sudo llmgw wipe --store      # and the store's database here (Patroni member, etcd, or local PostgreSQL)
+sudo llmgw wipe --all        # all of that, backups and the binary
+```
+
+It lists everything it will remove and asks for the machine's name (`--yes`
+does not ask), then stops llmgw's services and the engines, takes the
+firewall table out, and deletes `/etc/llmgw` — the authority and secrets
+with it — `/var/lib/llmgw`, the agent's state, the sudo grant, the
+authority from the system's trust, keepalived's configuration when llmgw
+wrote it, and the `llmgw` and `llmgw-engine` users. Packages it installed
+(PostgreSQL, Patroni, etcd, nftables, the node exporter) stay. Run it again
+to finish what failed. A node wiped is then forgotten on the control plane
+with `llmgw node remove <name>`. The console has it too: **Do → Wipe this
+machine**.
+
+This is how a machine with an older install is set up again from nothing:
+wipe it, then install as above. Wiping a control plane without `--store`
+leaves the store with the cluster's accounts, keys and settings in it, and a
+new control plane against that store takes them up.
+
 ## Kubernetes
 
 The control plane runs in Kubernetes from the Helm chart in

docs/reference.md

@@ -98,6 +98,7 @@ rather than repeated.
 | `other_watts` | `float64` | [FacilityBudget](#facilitybudget) |
 | `name` | `string` | [Federation](#federation) |
 | `interval` | `time.Duration` | [Federation](#federation) |
+| `enabled` | `bool` | [FirewallSettings](#firewallsettings) |
 | `format` | `string` | [GuardModel](#guardmodel) |
 | `categories` | `[]string` | [GuardModel](#guardmodel) |
 | `model` | `string` | [JudgePolicy](#judgepolicy) |
@@ -287,6 +288,7 @@ rather than repeated.
 | `guard` | `guard.Config` | [Server](#server) |
 | `secrets` | `Secrets` | [Server](#server) |
 | `notify` | `notify.Config` | [Server](#server) |
+| `firewall` | `FirewallSettings` | [Server](#server) |
 | `firewall_networks` | `[]string` | [Server](#server) |
 | `backup_recipients` | `[]string` | [Server](#server) |
 | `backup_copy_to` | `[]string` | [Server](#server) |
@@ -963,6 +965,19 @@ Interval is how often each peer is asked whether it is ready, timed,
 and what it offers. A peer that does not answer is taken out of
 routing until it does. Zero is 10s.
 
+## FirewallSettings
+
+### `enabled`
+
+`bool`
+
+Enabled keeps llmgw's internal ports — engines, node exporter, the
+agents' stream, the mirror, the store — to the cluster's networks on
+every machine. False removes llmgw's nftables table everywhere, for a
+site whose own firewall does that job; the rules are kept, and true
+puts them back. Changed in the cluster's settings, it is in force on
+every node and control plane within seconds.
+
 ## GuardModel
 
 ### `format`
@@ -2441,6 +2456,14 @@ Prometheus and Alertmanager in between. Empty sends nothing.
 
 Takes a [notify.Config](#notifyconfig) block.
 
+### `firewall`
+
+`FirewallSettings`
+
+Firewall turns llmgw's firewall on every machine on or off.
+
+Takes a [FirewallSettings](#firewallsettings) block.
+
 ### `firewall_networks`
 
 `[]string`

internal/agent/agent.go

@@ -36,6 +36,7 @@ import (
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/kvevents"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/obs"
 	"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
@@ -87,7 +88,11 @@ type Agent struct {
 	// serverList is every control plane to dial: the configured ones, and
 	// any the leader has named since. Read through servers().
 	serverList []string
-	inv        discovery.Inventory
+	// fwMu keeps one firewall change at a time; fwRunner runs its scripts
+	// (nil: this machine's shell).
+	fwMu     sync.Mutex
+	fwRunner firewall.Runner
+	inv      discovery.Inventory
 	// shardReplica remembers which replica a shard belongs to, so a status
 	// report can name both. The control plane keys on the replica.
 	shardReplica map[string]string
@@ -854,6 +859,8 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
 		go a.upgrade(ctx, m.Upgrade, out)
 	case *llmgwv1.ControlMessage_ControlPlanes:
 		a.setControlPlanes(m.ControlPlanes)
+	case *llmgwv1.ControlMessage_Firewall:
+		go a.applyFirewall(m.Firewall)
 	case *llmgwv1.ControlMessage_Logs:
 		reply := a.logsReply(m.Logs)
 		// Off the receive loop: a full send queue must not stop commands being

internal/agent/firewall.go

@@ -0,0 +1,89 @@
+package agent
+
+import (
+	"context"
+	"net"
+	"net/netip"
+	"net/url"
+	"strings"
+	"time"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
+)
+
+// applyFirewall keeps llmgw's firewall on this node as the cluster's
+// settings say: the engines' ports and the node exporter's for the networks
+// the node reaches its control planes on, and those named, or no table at
+// all. Off the receive loop, one at a time.
+func (a *Agent) applyFirewall(m *llmgwv1.Firewall) {
+	a.fwMu.Lock()
+	defer a.fwMu.Unlock()
+	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
+	defer cancel()
+	r := a.fwRunner
+	if r == nil {
+		r = firewall.Local{}
+	}
+	if !m.GetEnabled() {
+		had, err := firewall.Disable(ctx, r)
+		switch {
+		case err != nil:
+			a.log.Warn("could not take llmgw's firewall off this node", "err", err)
+		case had:
+			a.log.Info("llmgw's firewall is off on this node, as the cluster's settings say")
+		}
+		return
+	}
+	a.mu.Lock()
+	servers := append([]string(nil), a.serverList...)
+	a.mu.Unlock()
+	nets, err := firewall.Networks(ctx, r, controlPlaneIPs(ctx, servers))
+	if err != nil {
+		a.log.Warn("could not tell which networks reach the control planes; the firewall is left as it is", "err", err)
+		return
+	}
+	for _, n := range m.GetNetworks() {
+		if p, err := netip.ParsePrefix(n); err == nil {
+			nets = append(nets, p)
+		} else if ad, err := netip.ParseAddr(n); err == nil {
+			nets = append(nets, netip.PrefixFrom(ad, ad.BitLen()))
+		}
+	}
+	from, to := int(m.GetEnginePortsFrom()), int(m.GetEnginePortsTo())
+	ports := []firewall.Port{firewall.NodeExporter}
+	if from > 0 && to >= from {
+		ports = append(ports, firewall.Engines(from, to))
+	}
+	changed, err := firewall.Install(ctx, r, firewall.Spec{Ports: ports, Allow: nets})
+	switch {
+	case err != nil:
+		a.log.Warn("could not put llmgw's firewall on this node", "err", err)
+	case changed:
+		a.log.Info("llmgw's firewall is on this node", "networks", len(nets))
+	}
+}
+
+// controlPlaneIPs are the addresses of the control planes an agent dials.
+func controlPlaneIPs(ctx context.Context, servers []string) []string {
+	var out []string
+	for _, s := range servers {
+		host := s
+		if u, err := url.Parse(s); err == nil && u.Host != "" {
+			host = u.Hostname()
+		}
+		host = strings.Trim(host, "[]")
+		if a, err := netip.ParseAddr(host); err == nil {
+			out = append(out, a.String())
+			continue
+		}
+		ips, err := net.DefaultResolver.LookupNetIP(ctx, "ip", host)
+		if err != nil {
+			continue
+		}
+		for _, ip := range ips {
+			out = append(out, ip.Unmap().String())
+		}
+	}
+	return out
+}

internal/agent/firewall_test.go

@@ -0,0 +1,57 @@
+package agent
+
+import (
+	"context"
+	"errors"
+	"log/slog"
+	"strings"
+	"testing"
+
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// fakeShell answers the firewall package's scripts and records what was
+// installed or removed.
+type fakeShell struct {
+	present   bool
+	installed string
+	deleted   bool
+}
+
+func (f *fakeShell) Run(_ context.Context, script string, stdin []byte) ([]byte, error) {
+	switch {
+	case strings.HasPrefix(script, "ip -o route get 10.0.0.140"):
+		return []byte("10.0.0.140 dev eth0 src 10.0.0.141 uid 0\n2: eth0    inet 10.0.0.141/24 brd 10.0.0.255 scope global eth0\n"), nil
+	case strings.HasPrefix(script, "sudo -n /usr/sbin/nft list table"):
+		if f.present {
+			return nil, nil
+		}
+		return nil, errors.New("no table")
+	case strings.Contains(script, "nft delete table") && !strings.Contains(script, "cat > "):
+		f.deleted, f.present = true, false
+		return nil, nil
+	case strings.HasPrefix(script, "sudo cat "):
+		return nil, nil
+	default:
+		f.installed, f.present = string(stdin), true
+		return []byte("changed\n"), nil
+	}
+}
+
+func TestANodeKeepsOrDropsItsFirewallAsTheClusterSays(t *testing.T) {
+	sh := &fakeShell{}
+	a := &Agent{log: slog.New(slog.DiscardHandler), fwRunner: sh, serverList: []string{"https://10.0.0.140:7443"}}
+
+	a.applyFirewall(&llmgwv1.Firewall{Enabled: true, Networks: []string{"192.168.5.0/24", "10.9.9.9"},
+		EnginePortsFrom: 20000, EnginePortsTo: 20999})
+	for _, want := range []string{"10.0.0.0/24", "192.168.5.0/24", "10.9.9.9/32", "20000-20999", "9100"} {
+		if !strings.Contains(sh.installed, want) {
+			t.Errorf("the rules lack %s:\n%s", want, sh.installed)
+		}
+	}
+
+	a.applyFirewall(&llmgwv1.Firewall{Enabled: false})
+	if !sh.deleted {
+		t.Error("turned off, the table stayed")
+	}
+}

internal/cli/clustersettings.go

@@ -237,8 +237,10 @@ func (w *settingsWatch) check(ctx context.Context) {
 // again: what it reads at every use, or not at all.
 var liveSettings = []string{
 	"guard",
+	// Sent to the nodes and this machine's firewall unit as they change.
+	"firewall", "firewall_networks",
 	// Read by server init, node add and backups, not by the server.
-	"firewall_networks", "backup_recipients", "backup_copy_to",
+	"backup_recipients", "backup_copy_to",
 }
 
 // onlyLive is whether every changed key is one of liveSettings.

internal/cli/firewallcmd.go

@@ -0,0 +1,176 @@
+package cli
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/netip"
+	"os"
+	"time"
+
+	"github.com/spf13/cobra"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/config"
+	"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
+)
+
+// firewallStateFile is where a control plane writes what the cluster's
+// settings say of the firewall, for llmgw-firewall-sync to apply as root:
+// the server itself runs as llmgw and cannot change the ruleset.
+const firewallStateFile = "/var/lib/llmgw/firewall.json"
+
+// firewallState is what is written there.
+type firewallState struct {
+	Enabled  bool     `json:"enabled"`
+	Networks []string `json:"networks,omitempty"`
+}
+
+// writeFirewallState records the settings' firewall for this machine's sync
+// unit. Without the state directory — a server not run by systemd — there
+// is no unit to read it, and nothing is written.
+func writeFirewallState(cfg config.Server) error {
+	if _, err := os.Stat("/var/lib/llmgw"); err != nil {
+		return nil //nolint:nilerr // not a control plane set up by init
+	}
+	b, _ := json.Marshal(firewallState{Enabled: cfg.Firewall.Enabled, Networks: cfg.FirewallNetworks})
+	if old, err := os.ReadFile(firewallStateFile); err == nil && string(old) == string(b) {
+		return nil
+	}
+	tmp := firewallStateFile + ".new"
+	if err := os.WriteFile(tmp, b, 0o644); err != nil {
+		return err
+	}
+	return os.Rename(tmp, firewallStateFile)
+}
+
+// The units that apply it: a path unit watching the file, and the oneshot it
+// starts.
+const (
+	firewallSyncPathUnit = "llmgw-firewall-sync.path"
+	firewallSyncUnitFile = `[Unit]
+Description=llmgw firewall: apply the cluster's settings on this control plane
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/llmgw firewall sync
+`
+	firewallSyncPathFile = `[Unit]
+Description=llmgw firewall: follow the cluster's settings on this control plane
+
+[Path]
+PathChanged=` + firewallStateFile + `
+
+[Install]
+WantedBy=multi-user.target
+`
+)
+
+func newFirewallCmd() *cobra.Command {
+	c := &cobra.Command{
+		Use:   "firewall",
+		Short: "llmgw's firewall on this machine",
+		Long: "Whether llmgw keeps a firewall is the cluster's setting firewall.enabled\n" +
+			"(llmgw ctl config set firewall.enabled false). Nodes are told by the\n" +
+			"control plane; a control plane writes it to " + firewallStateFile + ",\n" +
+			"which " + firewallSyncPathUnit + " applies with 'llmgw firewall sync'.",
+	}
+	var install bool
+	sync := &cobra.Command{
+		Use:   "sync",
+		Short: "Apply what the cluster's settings say of the firewall, here (root)",
+		Args:  cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if os.Geteuid() != 0 {
+				return errors.New("the firewall is changed as root: run it with sudo")
+			}
+			if install {
+				if err := installFirewallSync(cmd.Context(), firewall.Local{}); err != nil {
+					return err
+				}
+				fmt.Fprintln(cmd.OutOrStdout(), firewallSyncPathUnit+" follows the cluster's settings here")
+				if _, err := os.Stat(firewallStateFile); err != nil {
+					return nil // the server writes it when it starts
+				}
+			}
+			b, err := os.ReadFile(firewallStateFile)
+			if err != nil {
+				return err
+			}
+			var st firewallState
+			if err := json.Unmarshal(b, &st); err != nil {
+				return fmt.Errorf("%s: %w", firewallStateFile, err)
+			}
+			ctx, cancel := context.WithTimeout(cmd.Context(), 2*time.Minute)
+			defer cancel()
+			msg, err := syncFirewall(ctx, firewall.Local{}, st)
+			if err != nil {
+				return err
+			}
+			fmt.Fprintln(cmd.OutOrStdout(), msg)
+			return nil
+		},
+	}
+	sync.Flags().BoolVar(&install, "install", false, "first install the units that run this whenever the settings change")
+	c.AddCommand(sync)
+	return c
+}
+
+// installFirewallSync writes the sync units on a machine and has the path
+// unit follow the state file.
+func installFirewallSync(ctx context.Context, r firewall.Runner) error {
+	body := firewallSyncUnitFile + "\x00" + firewallSyncPathFile
+	out, err := r.Run(ctx, `set -e
+t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
+cat > "$t/all"
+python3 - "$t" <<'PY'
+import sys
+d = sys.argv[1]
+a, b = open(d + "/all", "rb").read().split(b"\x00")
+open(d + "/service", "wb").write(a)
+open(d + "/path", "wb").write(b)
+PY
+sudo install -m 644 "$t/service" /etc/systemd/system/llmgw-firewall-sync.service
+sudo install -m 644 "$t/path" /etc/systemd/system/`+firewallSyncPathUnit+`
+sudo systemctl daemon-reload
+sudo systemctl enable --now `+firewallSyncPathUnit+` >/dev/null 2>&1`, []byte(body))
+	if err != nil {
+		return fmt.Errorf("installing %s: %v: %s", firewallSyncPathUnit, err, string(out))
+	}
+	return nil
+}
+
+// syncFirewall puts a firewall state in force on this machine.
+func syncFirewall(ctx context.Context, r firewall.Runner, st firewallState) (string, error) {
+	if !st.Enabled {
+		had, err := firewall.Disable(ctx, r)
+		if err != nil {
+			return "", err
+		}
+		if had {
+			return "llmgw's firewall is off here", nil
+		}
+		return "llmgw's firewall was already off here", nil
+	}
+	loaded, err := firewall.Enable(ctx, r)
+	if err != nil {
+		return "", err
+	}
+	if !loaded {
+		return "no llmgw rules on this machine to put back; 'llmgw server init' writes them", nil
+	}
+	var extra []netip.Prefix
+	for _, n := range st.Networks {
+		if p, err := netip.ParsePrefix(n); err == nil {
+			extra = append(extra, p)
+		} else if a, err := netip.ParseAddr(n); err == nil {
+			extra = append(extra, netip.PrefixFrom(a, a.BitLen()))
+		}
+	}
+	if len(extra) > 0 {
+		if _, err := firewall.Install(ctx, r, firewall.Spec{Allow: extra}); err != nil {
+			return "", err
+		}
+	}
+	return "llmgw's firewall is on here", nil
+}

internal/cli/firewallcmd_test.go

@@ -0,0 +1,51 @@
+package cli
+
+import (
+	"context"
+	"strings"
+	"testing"
+)
+
+type scriptLog struct {
+	scripts []string
+	reply   func(script string) string
+}
+
+func (s *scriptLog) Run(_ context.Context, script string, _ []byte) ([]byte, error) {
+	s.scripts = append(s.scripts, script)
+	if s.reply != nil {
+		return []byte(s.reply(script)), nil
+	}
+	return nil, nil
+}
+
+func TestAControlPlaneFollowsTheFirewallSetting(t *testing.T) {
+	off := &scriptLog{}
+	if _, err := syncFirewall(context.Background(), off, firewallState{Enabled: false}); err != nil {
+		t.Fatal(err)
+	}
+	if !strings.Contains(strings.Join(off.scripts, "\n"), "nft delete table inet llmgw") {
+		t.Errorf("off did not take the table out: %q", off.scripts)
+	}
+
+	none := &scriptLog{reply: func(s string) string {
+		if strings.Contains(s, "echo loaded") {
+			return "none\n"
+		}
+		return ""
+	}}
+	msg, err := syncFirewall(context.Background(), none, firewallState{Enabled: true})
+	if err != nil || !strings.Contains(msg, "server init") {
+		t.Errorf("on, with no rules here: %q %v", msg, err)
+	}
+
+	on := &scriptLog{reply: func(s string) string {
+		if strings.Contains(s, "echo loaded") {
+			return "loaded\n"
+		}
+		return ""
+	}}
+	if msg, err := syncFirewall(context.Background(), on, firewallState{Enabled: true}); err != nil || msg != "llmgw's firewall is on here" {
+		t.Errorf("on: %q %v", msg, err)
+	}
+}

internal/cli/node.go

@@ -174,8 +174,11 @@ func (na *nodeAdd) prepare() error {
 			}
 		}
 	}
-	// The node's ports — its engines', its metrics — for the cluster only.
-	if cfg, err := loadEffectiveConfig(na.serverCfg); err == nil && !o.NoFirewall {
+	// The node's ports — its engines', its metrics — for the cluster only,
+	// unless the cluster's settings turned llmgw's firewall off.
+	if cfg, err := loadEffectiveConfig(na.serverCfg); err == nil && !cfg.Firewall.Enabled {
+		o.NoFirewall = true
+	} else if err == nil && !o.NoFirewall {
 		from, to := cfg.EnginePorts()
 		o.FirewallPorts = []firewall.Port{firewall.Engines(from, to), firewall.NodeExporter}
 		o.FirewallAllow, _ = cfg.FirewallAllow()

internal/cli/root.go

@@ -59,6 +59,8 @@ func Root() *cobra.Command {
 		newSelfUpgradeCmd(),
 		newReleaseCmd(),
 		newJoinCmd(),
+		newFirewallCmd(),
+		newWipeCmd(),
 		newDocsCmd(),
 	)
 	return root

internal/cli/server.go

@@ -223,6 +223,10 @@ func newServerCmd() *cobra.Command {
 			// The control plane replaces the static endpoint list: agents
 			// register, the reconciler places replicas, and Ready ones are
 			// published to the router (plan §6.4).
+			// The settings in force, for what reads them at each use.
+			var current atomic.Pointer[config.Server]
+			current.Store(&cfg)
+			var agentSvc *agentsvc.Service
 			var agentHandler http.Handler
 			var agentPath string
 			var admin *control.Controller
@@ -247,6 +251,14 @@ func newServerCmd() *cobra.Command {
 					return err
 				}
 				defer ctl.Close()
+				agentSvc = svc
+				// Whether nodes keep llmgw's firewall, as the settings say now.
+				svc.SetFirewall(func() *llmgwv1.Firewall {
+					c := current.Load()
+					from, to := c.EnginePorts()
+					return &llmgwv1.Firewall{Enabled: c.Firewall.Enabled, Networks: c.FirewallNetworks,
+						EnginePortsFrom: int32(from), EnginePortsTo: int32(to)} //nolint:gosec // ports
+				})
 				ctl.Reconciler().SetNotifier(notifier)
 				admin = ctl
 				agentPath, agentHandler = svc.Handler()
@@ -653,11 +665,24 @@ func newServerCmd() *cobra.Command {
 			// Every change to the settings in force: what can change live
 			// does, the rest starts this process again, which systemd brings
 			// back on them.
+			// And this machine's own firewall, through its sync unit.
+			if err := writeFirewallState(cfg); err != nil {
+				log.Warn("could not record the firewall's settings for this machine", "err", err)
+			}
 			watch := &settingsWatch{path: configPath, st: rt.Store(), cur: boot, log: log,
 				live: func(next config.Server, changed []string) bool {
 					if !onlyLive(changed) {
 						return false
 					}
+					current.Store(&next)
+					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "firewall") }) {
+						if err := writeFirewallState(next); err != nil {
+							log.Warn("could not record the firewall's settings for this machine", "err", err)
+						}
+						if agentSvc != nil {
+							agentSvc.BroadcastFirewall()
+						}
+					}
 					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "guard") }) {
 						if gate == nil {
 							// Built at start, and only when there are policies.

internal/cli/serveradd.go

@@ -159,6 +159,14 @@ func (sa *serverAdd) firewall(ctx context.Context) error {
 	if err != nil {
 		return err
 	}
+	// What applies the cluster's firewall setting there when it changes.
+	if err := installFirewallSync(ctx, sa.remote); err != nil {
+		fmt.Fprintf(sa.out, "  !!    firewall: %v\n", err)
+	}
+	if !cfg.Firewall.Enabled {
+		sa.kept("firewall: off, as the cluster's settings say (firewall.enabled)")
+		return nil
+	}
 	self, err := sourceIPToward(sa.ip)
 	if err != nil {
 		return err

internal/cli/serverinit_firewall.go

@@ -23,6 +23,15 @@ func (in *serverInit) ensureFirewall(ctx context.Context) error {
 	if err != nil {
 		return err
 	}
+	// What applies the cluster's setting here when it changes: the server
+	// writes it, as llmgw; this applies it, as root.
+	if err := in.ensureFirewallSync(ctx); err != nil {
+		return err
+	}
+	if !cfg.Firewall.Enabled {
+		in.kept("firewall: off, as the cluster's settings say (firewall.enabled)")
+		return nil
+	}
 	ports := controlPlanePorts(cfg)
 	if !in.noMirror && in.mirrorURL == "" {
 		ports = append(ports, firewall.MirrorFiles)
@@ -122,3 +131,9 @@ func loopbackListen(listen string) bool {
 	ip := net.ParseIP(host)
 	return host == "localhost" || (ip != nil && ip.IsLoopback())
 }
+
+// ensureFirewallSync installs the units that apply the cluster's firewall
+// setting on this control plane.
+func (in *serverInit) ensureFirewallSync(ctx context.Context) error {
+	return installFirewallSync(ctx, firewall.Local{})
+}

internal/cli/tui_console.go

@@ -62,13 +62,14 @@ const (
 	cUpgrade
 	cRestartAll
 	cBackup
+	cWipe
 	cLeave
 	cCount
 )
 
 var consoleLabels = [cCount]string{"Overview", "Monitor", "Nodes", "GPUs", "Models", "Replicas", "Usage", "Quotas", "Guardrails", "Federation", "Mirror", "Jobs", "Logs", "Accounts", "API keys", "Organizations", "Audit", "Manual",
 	"server.yaml", "Add a node", "Add a control plane", "Replicate the store", "Serve a model", "Upgrade the cluster",
-	"Restart all replicas", "Back up now", "Leave"}
+	"Restart all replicas", "Back up now", "Wipe this machine", "Leave"}
 
 // consoleAction is what is done once the screen closes: each runs its own
 // screen, and the console comes back after.
@@ -150,7 +151,7 @@ type consoleModel struct {
 	saveNote     string
 
 	// A question open as a dialog.
-	dialog      string // "", "target", "edit", "addname", "saved", "confirm-leave", "error"
+	dialog      string // "", "target", "edit", "addname", "wipe", "saved", "note", "confirm-leave", "error"
 	dialogTitle string
 	dialogText  string
 	prompt      textField
@@ -407,7 +408,7 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
 
 func (m consoleModel) entryOK(i int) bool {
 	if m.node {
-		return i == cSettings || i == cManual || i == cLeave
+		return i == cSettings || i == cManual || i == cWipe || i == cLeave
 	}
 	return i != cReplicate || !m.storeReplicated()
 }
@@ -493,6 +494,8 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 				return m.ask("Replicate the store", "user@address of database machine", 2)
 			case cLeave:
 				return m.leave()
+			case cWipe:
+				return m.askWipe()
 			case cUpgrade, cRestartAll, cBackup:
 				return m, m.doEntry()
 			case cOverview:
@@ -744,6 +747,13 @@ func (m consoleModel) dialogKey(k tea.KeyMsg) (tea.Model, tea.Cmd) {
 				m.dialogText = floatingQuestion
 				m.prompt.Placeholder = "10.0.0.161/24"
 			}
+		case "wipe":
+			if host, _ := os.Hostname(); v != host {
+				m.dialogText = m.wipeQuestion() + "\n\n" + sBad.Render("that is not this machine's name; nothing is removed")
+				return m, nil
+			}
+			m.action = consoleAction{kind: "wipe"}
+			return m, tea.Quit
 		case "edit":
 			if err := m.doc.Set(m.editPath, m.editKind, m.prompt.Value()); err != nil {
 				m.dialogText = m.editQuestion() + "\n\n" + sBad.Render(err.Error())
@@ -1055,7 +1065,7 @@ func (m consoleModel) View() string {
 			nav = append(nav, navItem{Label: "", Heading: true}, navItem{Label: "Help", Heading: true})
 		case cSettings:
 			nav = append(nav, navItem{Label: "", Heading: true}, navItem{Label: "Settings", Heading: true})
-		case cAddNode, cLeave:
+		case cAddNode, cWipe:
 			if (i == cAddNode) != m.node {
 				nav = append(nav, navItem{Label: "", Heading: true}, navItem{Label: "Do", Heading: true})
 			}
@@ -1096,7 +1106,7 @@ func (m consoleModel) View() string {
 		f.DialogTitle = m.dialogTitle
 		f.Dialog = m.dialogText
 		switch m.dialog {
-		case "target", "edit", "addname":
+		case "target", "edit", "addname", "wipe":
 			q := m.dialogText
 			if m.dialog == "target" && m.editKind > 1 {
 				q = fmt.Sprintf("%s %d of %d", q, len(m.asked)+1, int(m.editKind))
@@ -1298,6 +1308,9 @@ func (m consoleModel) panel() (title, body, keys string) {
 	case cBackup:
 		item = "\n  Writes the store and /etc/llmgw to one archive in " + DefaultBackupDir + ",\n  and keeps the last few. Copy it off this machine.\n\n  " +
 			sDim.Render("On the command line:") + "\n  sudo llmgw server backup"
+	case cWipe:
+		item = "\n  Removes llmgw from this machine: its services, firewall table,\n  configuration, certificates, the authority, secrets, state and users,\n  so it can be set up again from nothing. Model weights, engines, the\n  store's database and backups stay. Asks for the machine's name first.\n\n  " +
+			sDim.Render("On the command line, with --models, --store or --all for more:") + "\n  sudo llmgw wipe"
 	case cLeave:
 		item = "\n  Closes the console. The cluster keeps running."
 	}
@@ -1621,6 +1634,16 @@ func runConsole(ctx context.Context, out io.Writer, file string) error {
 			// Already set up: only the store changes.
 			in.noMirror, in.noNode, in.noModel = true, true, true
 			err = in.runInitTUI(ctx, out)
+		case "wipe":
+			c := Root()
+			c.SetArgs([]string{"wipe", "--yes"})
+			c.SetOut(out)
+			c.SetErr(out)
+			if err := c.ExecuteContext(ctx); err != nil {
+				fmt.Fprintln(out, sBad.Render("✗ ")+err.Error())
+			}
+			// Nothing is left here for the console to show.
+			return nil
 		case "run":
 			c := Root()
 			c.SetArgs(act.targets)
@@ -1666,3 +1689,31 @@ func (m consoleModel) standbyNote() string {
 	}
 	return "\n  " + sDim.Render("Standing by. Shown, and changed, on the leader: "+leader) + "\n"
 }
+
+// askWipe asks for this machine's name before removing llmgw from it, saying
+// what would go.
+func (m consoleModel) askWipe() (tea.Model, tea.Cmd) {
+	if os.Geteuid() != 0 {
+		m.dialog, m.dialogTitle = "error", "Wipe this machine"
+		m.dialogText = "Removing services, users and files under /etc needs root:\nstart the console with sudo llmgw."
+		return m, nil
+	}
+	m.dialog, m.dialogTitle, m.dialogText = "wipe", "Wipe this machine", m.wipeQuestion()
+	m.prompt = newTextField()
+	m.prompt.Width = 40
+	m.prompt.Focus()
+	return m, nil
+}
+
+func (m consoleModel) wipeQuestion() string {
+	host, _ := os.Hostname()
+	w := &wiper{}
+	var b strings.Builder
+	b.WriteString("This removes, on " + host + ":\n\n")
+	for _, s := range w.plan() {
+		b.WriteString("  - " + s.what + "\n")
+	}
+	b.WriteString("\nModel weights, engines, the store's database and backups stay\n" +
+		"(sudo llmgw wipe --all takes them too).\n\nType the machine's name to remove it all:")
+	return b.String()
+}

internal/cli/tui_setup_test.go

@@ -300,3 +300,27 @@ func TestTheConsoleEditsTheClustersSettingsInTheStore(t *testing.T) {
 		t.Errorf("the cluster's key went into the machine's file:\n%s", b)
 	}
 }
+
+func TestTheConsoleWipesOnlyOnTheMachinesName(t *testing.T) {
+	m := newConsoleModel("127.0.0.1:1", filepath.Join(t.TempDir(), "server.yaml"))
+	m.width, m.height, m.loaded = 120, 40, true
+	m.menu = cWipe
+	m.dialog, m.dialogTitle, m.dialogText = "wipe", "Wipe this machine", m.wipeQuestion()
+	m.prompt = newTextField()
+	m.prompt.Focus()
+	if v := m.View(); !strings.Contains(v, "Type the machine's name") {
+		t.Errorf("the dialog does not ask for the name:\n%s", v)
+	}
+	m.prompt.SetValue("not-this-one")
+	mm := press(m, "enter")
+	if a := mm.(consoleModel).action; a.kind != "" {
+		t.Fatalf("a wrong name wiped: %+v", a)
+	}
+	cm := mm.(consoleModel)
+	host, _ := os.Hostname()
+	cm.prompt.SetValue(host)
+	mm = press(cm, "enter")
+	if a := mm.(consoleModel).action; a.kind != "wipe" {
+		t.Errorf("the right name did not: %+v", a)
+	}
+}

internal/cli/wipe.go

@@ -0,0 +1,277 @@
+package cli
+
+import (
+	"bufio"
+	"errors"
+	"fmt"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"runtime"
+	"strings"
+
+	"github.com/spf13/cobra"
+)
+
+// wipeUnits are the systemd units llmgw installs, stopped first so nothing
+// writes while the rest goes.
+var wipeUnits = []string{
+	"llmgw-server.service", "llmgw-agent.service", "llmgw-mirror.service",
+	firewallSyncPathUnit, "llmgw-firewall-sync.service", "llmgw-firewall.service",
+	"llmgw-backup.timer", "llmgw-backup.service",
+}
+
+// wiper removes llmgw from a machine. root prefixes every path, for tests;
+// run runs a command.
+type wiper struct {
+	root   string
+	run    func(name string, args ...string) error
+	store  bool
+	models bool
+	all    bool
+}
+
+// wipeStep is one thing to remove, said before it is done.
+type wipeStep struct {
+	what string
+	do   func() error
+}
+
+func (w *wiper) path(p string) string { return filepath.Join(w.root, p) }
+
+func (w *wiper) exists(p string) bool {
+	_, err := os.Lstat(w.path(p))
+	return err == nil
+}
+
+func (w *wiper) glob(pattern string) []string {
+	m, _ := filepath.Glob(w.path(pattern))
+	out := make([]string, len(m))
+	for i, p := range m {
+		out[i] = strings.TrimPrefix(p, w.root)
+	}
+	return out
+}
+
+// mentions is whether a file is there and says llmgw: a file of a name
+// another tool also uses (keepalived's, etcd's) is removed only when llmgw
+// wrote it.
+func (w *wiper) mentions(p string) bool {
+	b, err := os.ReadFile(w.path(p))
+	return err == nil && strings.Contains(string(b), "llmgw")
+}
+
+func (w *wiper) remove(paths ...string) func() error {
+	return func() error {
+		var errs []error
+		for _, p := range paths {
+			if err := os.RemoveAll(w.path(p)); err != nil {
+				errs = append(errs, err)
+			}
+		}
+		return errors.Join(errs...)
+	}
+}
+
+// plan is every step, in order, for what is on this machine.
+func (w *wiper) plan() []wipeStep {
+	var steps []wipeStep
+	add := func(what string, do func() error) { steps = append(steps, wipeStep{what, do}) }
+	files := func(what string, paths ...string) {
+		var there []string
+		for _, p := range paths {
+			if w.exists(p) {
+				there = append(there, p)
+			}
+		}
+		if len(there) > 0 {
+			add(what+": "+strings.Join(there, ", "), w.remove(there...))
+		}
+	}
+
+	// The services, stopped and gone.
+	var units []string
+	for _, u := range wipeUnits {
+		if w.exists("/etc/systemd/system/" + u) {
+			units = append(units, u)
+		}
+	}
+	if len(units) > 0 {
+		add("stop and remove "+strings.Join(units, ", "), func() error {
+			for _, u := range units {
+				_ = w.run("systemctl", "disable", "--now", u)
+				if err := os.Remove(w.path("/etc/systemd/system/" + u)); err != nil && !os.IsNotExist(err) {
+					return err
+				}
+			}
+			return w.run("systemctl", "daemon-reload")
+		})
+	}
+	// Engines run as their own user, kept alive across the agent's restarts:
+	// stopped by user, or they would keep their GPUs.
+	if w.userExists("llmgw-engine") {
+		add("stop the engines (every process of llmgw-engine)", func() error {
+			_ = w.run("pkill", "-KILL", "-u", "llmgw-engine")
+			return nil
+		})
+	}
+	add("take llmgw's table out of the firewall (nft inet llmgw), if loaded", func() error {
+		_ = w.run("nft", "delete", "table", "inet", "llmgw")
+		return nil
+	})
+	if w.mentions("/etc/keepalived/keepalived.conf") {
+		add("stop keepalived and remove llmgw's floating address (/etc/keepalived/keepalived.conf)", func() error {
+			_ = w.run("systemctl", "disable", "--now", "keepalived")
+			return w.remove("/etc/keepalived/keepalived.conf")()
+		})
+	}
+
+	files("configuration, certificates, the authority and secrets", "/etc/llmgw")
+	files("state: the admin token, a SQLite store, what the agent kept",
+		"/var/lib/llmgw", "/var/lib/llmgw-agent")
+	files("the sudo grant setup left", "/etc/sudoers.d/llmgw-setup")
+	if w.exists("/usr/local/share/ca-certificates/llmgw-ca.crt") {
+		add("the cluster's authority from the system's trust (llmgw-ca.crt)", func() error {
+			if err := w.remove("/usr/local/share/ca-certificates/llmgw-ca.crt")(); err != nil {
+				return err
+			}
+			_ = w.run("update-ca-certificates", "--fresh")
+			return nil
+		})
+	}
+	if w.exists("/etc/pki/ca-trust/source/anchors/llmgw-ca.pem") {
+		add("the cluster's authority from the system's trust (llmgw-ca.pem)", func() error {
+			if err := w.remove("/etc/pki/ca-trust/source/anchors/llmgw-ca.pem")(); err != nil {
+				return err
+			}
+			_ = w.run("update-ca-trust")
+			return nil
+		})
+	}
+
+	if w.store || w.all {
+		w.planStore(add, files)
+	}
+	if w.models || w.all {
+		files("model weights and the mirror", "/srv/models", "/var/lib/llmgw-weights", "/var/lib/llmgw-models")
+		files("installed engines", "/opt/llmgw")
+	}
+	if w.all {
+		files("backups", DefaultBackupDir)
+	}
+
+	for _, u := range []string{"llmgw-engine", "llmgw"} {
+		if w.userExists(u) {
+			add("the user "+u, func() error { return w.run("userdel", u) })
+		}
+	}
+	if w.all {
+		files("the llmgw binary", "/usr/local/bin/llmgw", "/usr/local/bin/llmgw.prev")
+	}
+	return steps
+}
+
+// planStore is the store, when it is on this machine: a member of the
+// replicated one (Patroni, etcd), or llmgw's database in a local PostgreSQL.
+func (w *wiper) planStore(add func(string, func() error), files func(string, ...string)) {
+	patroni := w.mentions("/etc/patroni/config.yml")
+	if patroni || w.mentions("/etc/default/etcd") {
+		add("stop Patroni and etcd", func() error {
+			_ = w.run("systemctl", "disable", "--now", "patroni")
+			_ = w.run("systemctl", "disable", "--now", "etcd")
+			return nil
+		})
+	}
+	if patroni {
+		data := append(w.glob("/var/lib/postgresql/*/llmgw"), w.glob("/var/lib/pgsql/*/llmgw")...)
+		files("the store's database on this member, and Patroni's configuration",
+			append([]string{"/etc/patroni", "/var/lib/postgresql/.pgpass_patroni", "/root/llmgw-pg"}, data...)...)
+	} else if w.exists("/usr/bin/psql") {
+		add("drop the llmgw database and role from the local PostgreSQL, if there", func() error {
+			_ = w.run("runuser", "-u", "postgres", "--", "psql", "-q",
+				"-c", "DROP DATABASE IF EXISTS llmgw", "-c", "DROP ROLE IF EXISTS llmgw")
+			return nil
+		})
+	}
+	if w.mentions("/etc/default/etcd") {
+		files("etcd's data and configuration", "/var/lib/etcd/default", "/etc/etcd/tls", "/etc/default/etcd")
+	}
+}
+
+func (w *wiper) userExists(name string) bool {
+	b, err := os.ReadFile(w.path("/etc/passwd"))
+	return err == nil && strings.Contains("\n"+string(b), "\n"+name+":")
+}
+
+func newWipeCmd() *cobra.Command {
+	var w wiper
+	var yes bool
+	c := &cobra.Command{
+		Use:   "wipe",
+		Short: "Remove llmgw from this machine: its services, configuration and state",
+		Long: "Stops and removes llmgw's services, takes its firewall table out, and\n" +
+			"deletes its configuration, certificates, the authority, secrets, state\n" +
+			"and users — what server init, node add, server add and mirror init left —\n" +
+			"so the machine can be set up again from nothing. It says everything it\n" +
+			"will remove first, and asks for the machine's name.\n\n" +
+			"Kept unless asked: model weights, the mirror and engines (--models), the\n" +
+			"store's database when it is on this machine (--store), backups and the\n" +
+			"binary (--all, which is all of these). Packages llmgw installed —\n" +
+			"PostgreSQL, Patroni, etcd, nftables, the node exporter — stay.\n\n" +
+			"A node wiped is then forgotten on the control plane with\n" +
+			"'llmgw node remove <name>'.",
+		Example: "  sudo llmgw wipe\n" +
+			"  sudo llmgw wipe --models\n" +
+			"  sudo llmgw wipe --all --yes",
+		Args: cobra.NoArgs,
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if runtime.GOOS != "linux" {
+				return fmt.Errorf("wipe removes llmgw from a Linux machine; this is %s", runtime.GOOS)
+			}
+			if os.Geteuid() != 0 {
+				return errors.New("wipe removes services, users and files under /etc: run it with sudo")
+			}
+			out := cmd.OutOrStdout()
+			w.run = func(name string, args ...string) error {
+				return exec.CommandContext(cmd.Context(), name, args...).Run() //nolint:gosec // fixed commands
+			}
+			steps := w.plan()
+			host, _ := os.Hostname()
+			fmt.Fprintf(out, "On %s, this removes:\n", host)
+			for _, s := range steps {
+				fmt.Fprintf(out, "  - %s\n", s.what)
+			}
+			if !yes {
+				if !isTerminal(out) {
+					return errors.New("nothing removed: give --yes to wipe without being asked")
+				}
+				fmt.Fprintf(out, "\nType this machine's name (%s) to remove all of it: ", host)
+				answer, _ := bufio.NewReader(cmd.InOrStdin()).ReadString('\n')
+				if strings.TrimSpace(answer) != host {
+					return errors.New("nothing removed")
+				}
+			}
+			var failed []string
+			for _, s := range steps {
+				if err := s.do(); err != nil {
+					failed = append(failed, s.what+": "+err.Error())
+					fmt.Fprintf(out, "  !!    %s: %v\n", s.what, err)
+					continue
+				}
+				fmt.Fprintf(out, "  ok    %s\n", s.what)
+			}
+			if len(failed) > 0 {
+				return fmt.Errorf("%d step(s) failed; running wipe again finishes them", len(failed))
+			}
+			fmt.Fprintln(out, "\nllmgw is gone from this machine. If it was a node, forget it on the control plane: "+
+				"llmgw node remove "+host)
+			return nil
+		},
+	}
+	f := c.Flags()
+	f.BoolVar(&yes, "yes", false, "do not ask")
+	f.BoolVar(&w.models, "models", false, "also remove model weights, the mirror's files and installed engines")
+	f.BoolVar(&w.store, "store", false, "also remove the store's database here: a Patroni member's data, etcd's, or llmgw's database in a local PostgreSQL")
+	f.BoolVar(&w.all, "all", false, "everything: --models, --store, backups and the binary")
+	return c
+}

internal/cli/wipe_test.go

@@ -0,0 +1,106 @@
+package cli
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func wipeFixture(t *testing.T) string {
+	t.Helper()
+	root := t.TempDir()
+	for p, body := range map[string]string{
+		"/etc/llmgw/server.yaml":                      "listen: :8080\n",
+		"/etc/llmgw/pki/ca-key.pem":                   "key",
+		"/var/lib/llmgw/admin-token":                  "t",
+		"/var/lib/llmgw-agent/control-planes.json":    "[]",
+		"/etc/systemd/system/llmgw-server.service":    "[Unit]",
+		"/etc/systemd/system/llmgw-agent.service":     "[Unit]",
+		"/etc/sudoers.d/llmgw-setup":                  "x",
+		"/etc/keepalived/keepalived.conf":             "# llmgw: the gateway's floating address\n",
+		"/etc/passwd":                                 "root:x:0:0::/root:/bin/sh\nllmgw:x:999:999::/var/lib/llmgw:/usr/sbin/nologin\nllmgw-engine:x:998:998::/:/usr/sbin/nologin\n",
+		"/srv/models/qwen/v1/q.gguf":                  "w",
+		"/opt/llmgw/engines/llamacpp/b1/llama-server": "e",
+		"/etc/patroni/config.yml":                     "scope: llmgw\n",
+		"/var/lib/postgresql/16/llmgw/PG_VERSION":     "16",
+		"/etc/default/etcd":                           "ETCD_INITIAL_CLUSTER_TOKEN=llmgw\n",
+		"/var/lib/etcd/default/member/snap":           "s",
+		"/usr/local/bin/llmgw":                        "bin",
+	} {
+		full := filepath.Join(root, p)
+		if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
+			t.Fatal(err)
+		}
+		if err := os.WriteFile(full, []byte(body), 0o600); err != nil {
+			t.Fatal(err)
+		}
+	}
+	return root
+}
+
+func TestWipeRemovesTheConfigurationAndKeepsWhatWasNotAsked(t *testing.T) {
+	root := wipeFixture(t)
+	var ran []string
+	w := &wiper{root: root, run: func(name string, args ...string) error {
+		ran = append(ran, name+" "+strings.Join(args, " "))
+		return nil
+	}}
+	for _, s := range w.plan() {
+		if err := s.do(); err != nil {
+			t.Fatalf("%s: %v", s.what, err)
+		}
+	}
+	for _, gone := range []string{"/etc/llmgw", "/var/lib/llmgw", "/var/lib/llmgw-agent",
+		"/etc/systemd/system/llmgw-server.service", "/etc/sudoers.d/llmgw-setup", "/etc/keepalived/keepalived.conf"} {
+		if _, err := os.Stat(filepath.Join(root, gone)); err == nil {
+			t.Errorf("%s is still there", gone)
+		}
+	}
+	for _, kept := range []string{"/srv/models/qwen/v1/q.gguf", "/opt/llmgw/engines", "/etc/patroni/config.yml",
+		"/var/lib/postgresql/16/llmgw", "/var/lib/etcd/default", "/usr/local/bin/llmgw"} {
+		if _, err := os.Stat(filepath.Join(root, kept)); err != nil {
+			t.Errorf("%s was removed without being asked", kept)
+		}
+	}
+	all := strings.Join(ran, "\n")
+	for _, want := range []string{"systemctl disable --now llmgw-server.service", "pkill -KILL -u llmgw-engine",
+		"nft delete table inet llmgw", "systemctl disable --now keepalived", "userdel llmgw-engine", "userdel llmgw"} {
+		if !strings.Contains(all, want) {
+			t.Errorf("did not run %q:\n%s", want, all)
+		}
+	}
+	if strings.Contains(all, "patroni") {
+		t.Error("the store was touched without --store")
+	}
+}
+
+func TestWipeAllTakesTheStoreAndTheModelsToo(t *testing.T) {
+	root := wipeFixture(t)
+	w := &wiper{root: root, all: true, run: func(string, ...string) error { return nil }}
+	for _, s := range w.plan() {
+		if err := s.do(); err != nil {
+			t.Fatalf("%s: %v", s.what, err)
+		}
+	}
+	for _, gone := range []string{"/srv/models", "/opt/llmgw", "/etc/patroni", "/var/lib/postgresql/16/llmgw",
+		"/var/lib/etcd/default", "/etc/default/etcd", "/usr/local/bin/llmgw"} {
+		if _, err := os.Stat(filepath.Join(root, gone)); err == nil {
+			t.Errorf("%s is still there", gone)
+		}
+	}
+}
+
+func TestWipeLeavesAKeepalivedItDidNotWrite(t *testing.T) {
+	root := t.TempDir()
+	p := filepath.Join(root, "/etc/keepalived/keepalived.conf")
+	_ = os.MkdirAll(filepath.Dir(p), 0o755)
+	_ = os.WriteFile(p, []byte("vrrp_instance site_vip {}\n"), 0o600)
+	w := &wiper{root: root, run: func(string, ...string) error { return nil }}
+	for _, s := range w.plan() {
+		_ = s.do()
+	}
+	if _, err := os.Stat(p); err != nil {
+		t.Error("someone else's keepalived configuration was removed")
+	}
+}

internal/config/config.go

@@ -146,6 +146,9 @@ type Server struct {
 	// Prometheus and Alertmanager in between. Empty sends nothing.
 	Notify notify.Config `koanf:"notify"`
 
+	// Firewall turns llmgw's firewall on every machine on or off.
+	Firewall FirewallSettings `koanf:"firewall"`
+
 	// FirewallNetworks are networks, besides those each machine reaches the
 	// cluster on, that llmgw's internal ports answer: a monitoring host on
 	// another subnet, nodes behind a router. Written into every machine's
@@ -763,6 +766,7 @@ func Default() Server {
 		AgentListen:    ":7443",
 		MaxConnections: 10_000,
 		Mirror:         Mirror{UpdateCheck: 6 * time.Hour},
+		Firewall:       FirewallSettings{Enabled: true},
 		Control: Control{
 			Interval:         5 * time.Second,
 			BaseBackoff:      10 * time.Second,

internal/config/firewall.go

@@ -32,3 +32,15 @@ func (s Server) EnginePorts() (int, int) {
 	d := Default().Control.PortRange
 	return d[0], d[1]
 }
+
+// FirewallSettings is whether llmgw keeps a firewall on the cluster's
+// machines.
+type FirewallSettings struct {
+	// Enabled keeps llmgw's internal ports — engines, node exporter, the
+	// agents' stream, the mirror, the store — to the cluster's networks on
+	// every machine. False removes llmgw's nftables table everywhere, for a
+	// site whose own firewall does that job; the rules are kept, and true
+	// puts them back. Changed in the cluster's settings, it is in force on
+	// every node and control plane within seconds.
+	Enabled bool `koanf:"enabled"`
+}

internal/control/agentsvc/firewall.go

@@ -0,0 +1,42 @@
+package agentsvc
+
+import (
+	llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
+)
+
+// SetFirewall gives the service whether nodes keep llmgw's firewall, read
+// at each use so a change to the cluster's settings is what is sent next.
+func (s *Service) SetFirewall(f func() *llmgwv1.Firewall) {
+	s.mu.Lock()
+	s.firewall = f
+	s.mu.Unlock()
+}
+
+func (s *Service) firewallMsg() *llmgwv1.ControlMessage {
+	s.mu.RLock()
+	f := s.firewall
+	s.mu.RUnlock()
+	if f == nil {
+		return nil
+	}
+	fw := f()
+	if fw == nil {
+		return nil
+	}
+	return &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_Firewall{Firewall: fw}}
+}
+
+// BroadcastFirewall tells every connected agent what the cluster's settings
+// now say of the firewall. A node not connected is told when it registers.
+func (s *Service) BroadcastFirewall() {
+	msg := s.firewallMsg()
+	if msg == nil {
+		return
+	}
+	for _, node := range s.ConnectedNodes() {
+		if err := s.send(node, msg); err != nil {
+			s.log.Warn("could not tell a node about the firewall; it is told when it next registers",
+				"node", node, "err", err)
+		}
+	}
+}

internal/control/agentsvc/service.go

@@ -75,6 +75,8 @@ type Service struct {
 
 	// controlPlanes lists what agents may dial; see SetControlPlanes.
 	controlPlanes func(context.Context) ([]string, error)
+	// firewall is what nodes are told of llmgw's firewall; see SetFirewall.
+	firewall func() *llmgwv1.Firewall
 
 	// logWaits are the log requests awaiting a reply, by request id.
 	logMu    sync.Mutex
@@ -286,6 +288,12 @@ func (s *Service) Connect(
 			return err
 		}
 	}
+	// Whether to keep llmgw's firewall, as the cluster's settings say now.
+	if msg := s.firewallMsg(); msg != nil {
+		if err := stream.Send(msg); err != nil {
+			return err
+		}
+	}
 
 	// Recover replica state from what this node is already running, before any
 	// reconciliation pass can conclude the cluster is empty and place

internal/docgen/reference_gen.go

@@ -117,6 +117,9 @@ var Reference = []Block{
 		{Key: "name", Type: "string", Doc: "Name is this cluster as a peer reads it, in the federated_from tag\nsent with a request to a peer set to receive tags (`ctl federation add\n--send-tags`). Empty is this machine's host name.", Nested: ""},
 		{Key: "interval", Type: "time.Duration", Doc: "Interval is how often each peer is asked whether it is ready, timed,\nand what it offers. A peer that does not answer is taken out of\nrouting until it does. Zero is 10s.", Nested: ""},
 	}},
+	{Package: "config", Name: "FirewallSettings", Title: "FirewallSettings", Doc: "", Fields: []Field{
+		{Key: "enabled", Type: "bool", Doc: "Enabled keeps llmgw's internal ports — engines, node exporter, the\nagents' stream, the mirror, the store — to the cluster's networks on\nevery machine. False removes llmgw's nftables table everywhere, for a\nsite whose own firewall does that job; the rules are kept, and true\nputs them back. Changed in the cluster's settings, it is in force on\nevery node and control plane within seconds.", Nested: ""},
+	}},
 	{Package: "catalog", Name: "GuardModel", Title: "GuardModel", Doc: "", Fields: []Field{
 		{Key: "format", Type: "string", Doc: "Format is llamaguard3 (Meta's Llama Guard 3: \"safe\", or \"unsafe\" and\nthe categories), shieldgemma (Google's ShieldGemma: Yes or No), or\njson ({\"safe\": false, \"categories\": [\"...\"]}).", Nested: ""},
 		{Key: "categories", Type: "[]string", Doc: "Categories the model is asked about, for shieldgemma — each a policy\nsentence — and the names reported for llamaguard3's S1, S2, ….", Nested: ""},
@@ -363,6 +366,7 @@ var Reference = []Block{
 		{Key: "guard", Type: "guard.Config", Doc: "Guard is the library of guardrail policies: what requests and answers\nare checked for, and what is done. Applied to every request through\nguard.default, and to parts of the tenancy tree by attaching them\n(llmgw ctl guard attach). Empty checks nothing.", Nested: "guard.Config"},
 		{Key: "secrets", Type: "Secrets", Doc: "Secrets is where secrets kept outside files are read from, and the key\nthe store's own secrets are sealed under.", Nested: "config.Secrets"},
 		{Key: "notify", Type: "notify.Config", Doc: "Notify sends what needs a person — a model parked, a node lost, a\nbudget spent, new weights published — to webhooks, signed, without a\nPrometheus and Alertmanager in between. Empty sends nothing.", Nested: "notify.Config"},
+		{Key: "firewall", Type: "FirewallSettings", Doc: "Firewall turns llmgw's firewall on every machine on or off.", Nested: "config.FirewallSettings"},
 		{Key: "firewall_networks", Type: "[]string", Doc: "FirewallNetworks are networks, besides those each machine reaches the\ncluster on, that llmgw's internal ports answer: a monitoring host on\nanother subnet, nodes behind a router. Written into every machine's\nfirewall by `server init`, `node add` and `server add`.", Nested: ""},
 		{Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""},
 		{Key: "backup_copy_to", Type: "[]string", Doc: "BackupCopyTo is where each archive is copied once written, so it\nsurvives this machine: a directory (a mounted share, another disk) or\nssh://user@host[:port]/path, reached with BackupCopyKey and checked\nagainst /etc/llmgw/backup_known_hosts, which learns a host the first\ntime. The same number of archives is kept there as here. Refused\nwithout backup_recipients: an archive leaves the machine encrypted.", Nested: ""},

internal/docgen/render.go

@@ -30,6 +30,7 @@ var Sources = []string{
 	"internal/guard/policy.go",
 	"internal/guard/judge.go",
 	"internal/config/oidc.go",
+	"internal/config/firewall.go",
 	"internal/notify/notify.go",
 	"internal/secrets/secrets.go",
 	"internal/obs/trace.go",

internal/firewall/firewall.go

@@ -11,10 +11,13 @@
 package firewall
 
 import (
+	"bytes"
 	"context"
 	"errors"
 	"fmt"
 	"net/netip"
+	"os"
+	"os/exec"
 	"slices"
 	"strconv"
 	"strings"
@@ -353,3 +356,51 @@ func Admit(ctx context.Context, r Runner, addrs []string) ([]netip.Prefix, error
 	}
 	return nets, nil
 }
+
+// Disable takes llmgw's table out of a machine's ruleset and stops it being
+// loaded at boot, keeping the rules file: Install, or enabling the unit,
+// puts it back as it was. Reports whether there was anything to take out.
+func Disable(ctx context.Context, r Runner) (bool, error) {
+	had := Present(ctx, r)
+	out, err := r.Run(ctx, `sudo systemctl disable --now `+Unit+` >/dev/null 2>&1 || true
+if sudo /usr/sbin/nft list table inet `+Table+` >/dev/null 2>&1; then
+  sudo /usr/sbin/nft delete table inet `+Table+`
+fi`, nil)
+	if err != nil {
+		return false, fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
+	}
+	return had, nil
+}
+
+// Enable loads the rules file a machine already has and has it loaded at
+// boot again, after Disable. Reports false when there is no rules file.
+func Enable(ctx context.Context, r Runner) (bool, error) {
+	out, err := r.Run(ctx, `[ -f `+RulesFile+` ] || { echo none; exit 0; }
+sudo systemctl enable `+Unit+` >/dev/null 2>&1
+sudo /usr/sbin/nft -f `+RulesFile+`
+sudo systemctl start `+Unit+` >/dev/null 2>&1 || true
+echo loaded`, nil)
+	if err != nil {
+		return false, fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
+	}
+	return strings.TrimSpace(string(out)) == "loaded", nil
+}
+
+// Local runs scripts on this machine. As root, sudo in them runs its command
+// directly, for a machine that has no sudo.
+type Local struct{}
+
+// Run runs script with sh.
+func (Local) Run(ctx context.Context, script string, stdin []byte) ([]byte, error) {
+	if os.Geteuid() == 0 {
+		script = "sudo() { \"$@\"; }\n" + script
+	}
+	cmd := exec.CommandContext(ctx, "sh", "-c", script)
+	if stdin != nil {
+		cmd.Stdin = bytes.NewReader(stdin)
+	}
+	var out bytes.Buffer
+	cmd.Stdout, cmd.Stderr = &out, &out
+	err := cmd.Run()
+	return out.Bytes(), err
+}