Keep llmgw's internal ports to the cluster's networks with an nftables table of its own

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 11:03:46 +02:00
1 parent 7e57863edd
commit 7b6276901a
17 files changed
+800 -10

No files matched your search

+10
View File
@@ -201,6 +201,16 @@ APIs change between minors without a deprecation period.
arrays more than 4 deep is refused rather than recursed into. Serve and
update jobs keep the last 100 finished, and the mirror forgets finished
downloads after a week, instead of remembering every one until restart.
- **A firewall for llmgw's internal ports.** `server init` (and the store
members it sets up with `--patroni`), `node add`, `server add` and
`mirror init` now keep the machine's internal ports — the agents' stream,
engines, the mirror, PostgreSQL, Patroni, etcd, node exporter — to the
cluster's own networks, in an nftables table of llmgw's own loaded by
`llmgw-firewall.service`. Only those ports are touched: the table accepts
everything else, so other services on the machine and any firewall it
already has are left as they were. Extra networks go in
`firewall_networks`; `--no-firewall` skips it; `llmgw doctor` says when a
node lacks it.
### Everything that can be changed on a running cluster, from the console
+39 -8
View File
@@ -1071,6 +1071,39 @@ thing to get right is the health check: point it at `/ready`, not `/health`.
so a load balancer using it would send half the traffic to something that
refuses every request.
## The firewall
`server init`, `node add`, `server add` and `mirror init` each keep that
machine's **internal** llmgw ports to the cluster's own networks, with an
nftables table of llmgw's own (`inet llmgw`, in `/etc/llmgw/firewall.nft`,
loaded at boot by `llmgw-firewall.service`):
| Machine | Ports, for the cluster only |
| --- | --- |
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
| node | engine ports (`control.port_range`, 20000–20999), 9100 |
| store member | 5432, 8008, 2379, 2380, 9100 |
| mirror | 8081, 8082, 9100 |
"The cluster's networks" are, on each machine, the network of the interface
it reaches the control planes (or the other store members) on — or just
their address, when the route goes through a gateway. Anything else that
must reach those ports, a Prometheus on another subnet or nodes behind a
router, goes in `server.yaml` before the machines are set up:
```yaml
firewall_networks: [10.0.5.0/24, 192.168.1.20]
```
Only those ports are touched. The table's policy is to accept, so whatever
else the machine serves — and whatever firewall it already has — is left as
it was; the client API (`:8080`) is not closed, being meant to be reached.
Running a command again adds to what is there rather than replacing it, so a
machine that is both a store member and a node keeps both sets. `llmgw
doctor` on a node says when the table is missing. `--no-firewall` on any of
the four commands leaves the firewall alone; `sudo systemctl disable --now
llmgw-firewall` removes it.
## Upgrading
On the control plane:
@@ -1347,11 +1380,9 @@ requests. On a slow model, set `timeouts.generation` higher.
has them. Single-node placement on real GPUs is the well-trodden path; if you
are the first to run a TP group across real cards, run `llmgw doctor --json`
on each node first and keep the reports.
- **Two engine setups are weaker than the rest.** An engine's key is passed
in its environment, where only root and its own user can read it, except
SGLang's, which takes it only as a flag: other users on the node can see it
in the process list. Mount `/proc` with `hidepid=2` on nodes where that
matters. vLLM's `launch_mode: ray` starts a Ray head without
authentication, so anyone who can reach its port can run code on the
node. The dashboard is off, but firewall the port to the other nodes in the
replica, or use the default `mp` launch mode, which needs no Ray at all.
- **One engine setup is weaker than the rest.** vLLM's `launch_mode: ray`
starts a Ray head without authentication, so anyone who can reach its
port can run code on the node, and Ray's ports are not among those
llmgw's firewall guards. The dashboard is off, but firewall the port to
the other nodes in the replica, or use the default `mp` launch mode, which
needs no Ray at all.
+10
View File
@@ -185,6 +185,7 @@ rather than repeated.
| `engines` | `map[string]string` | [Server](#server) |
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
| `backup_dir` | `string` | [Server](#server) |
| `firewall_networks` | `[]string` | [Server](#server) |
| `backup_recipients` | `[]string` | [Server](#server) |
| `releases` | `string` | [Server](#server) |
| `router` | `Router` | [Server](#server) |
@@ -1493,6 +1494,15 @@ BackupDir is where `llmgw server backup` writes, and where the newest
archive's age is read for llmgw_backup_last_success_timestamp_seconds.
Empty means /var/backups/llmgw-server.
### `firewall_networks`
`[]string`
FirewallNetworks are networks, besides those each machine reaches the
cluster on, that llmgw's internal ports answer: a monitoring host on
another subnet, nodes behind a router. Written into every machine's
firewall by `server init`, `node add` and `server add`.
### `backup_recipients`
`[]string`
+44 -1
View File
@@ -6,6 +6,7 @@ import (
"fmt"
"io"
"net"
"net/netip"
"os"
"runtime"
"strings"
@@ -13,7 +14,9 @@ import (
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
)
@@ -26,6 +29,7 @@ func newMirrorInitCmd(dir *string) *cobra.Command {
var (
controlPlanes []string
base string
noFirewall bool
)
cmd := &cobra.Command{
Use: "init",
@@ -56,11 +60,19 @@ Safe to run again: it changes only what differs.`,
if _, err := parseAllow(controlPlanes); err != nil {
return err
}
return mirrorInit(cmd.Context(), cmd.OutOrStdout(), *dir, base, controlPlanes)
if err := mirrorInit(cmd.Context(), cmd.OutOrStdout(), *dir, base, controlPlanes); err != nil {
return err
}
if !noFirewall {
mirrorFirewall(cmd.Context(), cmd.OutOrStdout(), controlPlanes)
}
return nil
},
}
cmd.Flags().StringArrayVar(&controlPlanes, "control-plane", nil,
"address of the control plane that manages this mirror; repeatable")
cmd.Flags().BoolVar(&noFirewall, "no-firewall", false,
"leave the firewall alone, instead of keeping the mirror's ports to the cluster's networks")
cmd.Flags().StringVar(&base, "base", "",
"the address nodes fetch from, http://host:8081 (default: this machine's first address)")
return cmd
@@ -123,3 +135,34 @@ func mirrorInit(ctx context.Context, out io.Writer, dir, base string, controlPla
fmt.Fprintf(out, "\nThe mirror is up. On the control plane:\n\n sudo llmgw server init --mirror %s\n\n", api)
return nil
}
// mirrorFirewall keeps the mirror's ports to the networks it reaches the
// control planes on: its files are for the cluster's nodes, its API for the
// control planes, which the API checks again by address.
func mirrorFirewall(ctx context.Context, out io.Writer, controlPlanes []string) {
allowed, _ := parseAllow(controlPlanes)
var toward []string
var nets []netip.Prefix
for _, p := range allowed {
if p.IsSingleIP() {
toward = append(toward, p.Addr().String())
} else {
nets = append(nets, p)
}
}
found, err := firewall.Networks(ctx, pgha.Local{}, toward)
if err == nil {
var changed bool
changed, err = firewall.Install(ctx, pgha.Local{}, firewall.Spec{
Ports: []firewall.Port{firewall.MirrorFiles, firewall.MirrorAPI, firewall.NodeExporter},
Allow: append(found, nets...),
})
if err == nil {
if changed {
fmt.Fprintln(out, " ok firewall: the mirror's ports answer the cluster's networks only")
}
return
}
}
fmt.Fprintf(out, " !! firewall: %v; the mirror's ports answer every network\n", err)
}
+8
View File
@@ -14,6 +14,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
@@ -122,6 +123,7 @@ engines are kept.`,
f.StringVar(&na.opts.Releases, "releases", "https://gitea.mynodes.xyz/ilker/llmgw/releases/download", "where release files are downloaded from")
f.BoolVar(&na.opts.IgnoreChecks, "ignore-checks", false, "add the node even when doctor finds a problem")
f.BoolVar(&na.opts.NoNodeExporter, "no-node-exporter", false, "leave Prometheus' node exporter off the machine")
f.BoolVar(&na.opts.NoFirewall, "no-firewall", false, "leave the machine's firewall alone, instead of keeping llmgw's ports to the cluster's networks")
f.StringArrayVar(&na.opts.DisableEngines, "disable-engine", nil, "an engine this node never installs, e.g. vllm on GPUs it does not run on; repeatable")
f.StringVar(&na.admin, "admin", "", "the control plane's admin API, to watch the node join (default: from server.yaml, else 127.0.0.1:9090)")
f.StringVar(&na.serverCfg, "config", "/etc/llmgw/server.yaml", "the control plane's configuration, for the admin API's address")
@@ -168,6 +170,12 @@ func (na *nodeAdd) prepare() error {
}
}
}
// The node's ports — its engines', its metrics — for the cluster only.
if cfg, err := config.Load(na.serverCfg); err == nil && !o.NoFirewall {
from, to := cfg.EnginePorts()
o.FirewallPorts = []firewall.Port{firewall.Engines(from, to), firewall.NodeExporter}
o.FirewallAllow, _ = cfg.FirewallAllow()
}
o.Version = version.Version
if o.Binary == "" {
exe, err := os.Executable()
+41
View File
@@ -20,6 +20,7 @@ import (
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
@@ -38,6 +39,8 @@ type serverAdd struct {
vip string
remote onboard.Remote
// noFirewall leaves the new control plane's firewall as it is.
noFirewall bool
// addr is the machine's address as given; ip what the others reach it at.
addr, ip, name string
out io.Writer
@@ -78,6 +81,8 @@ func newServerAddCmd() *cobra.Command {
f.StringArrayVar(&sa.hosts, "host", nil, "another name for the new control plane's certificate; repeatable")
f.StringVar(&sa.vip, "vip", "", "a floating address for whichever control plane leads, as 10.0.0.160/24")
f.StringVar(&sa.pkiDir, "pki", sa.pkiDir, "directory holding the certificate authority")
f.BoolVar(&sa.noFirewall, "no-firewall", false,
"leave its firewall alone, instead of keeping llmgw's internal ports to the cluster's networks")
return cmd
}
@@ -121,6 +126,12 @@ func (sa *serverAdd) run(ctx context.Context) error {
{"store", sa.storeAccess},
{"service", sa.service},
}
if !sa.noFirewall {
steps = append(steps, struct {
name string
fn func(context.Context) error
}{"firewall", sa.firewall})
}
if sa.vip != "" {
steps = append(steps, struct {
name string
@@ -136,6 +147,36 @@ func (sa *serverAdd) run(ctx context.Context) error {
return nil
}
// firewall keeps the new control plane's internal ports to the networks it
// reaches this one on. Not fatal: it serves without one.
func (sa *serverAdd) firewall(ctx context.Context) error {
cfg, err := config.Load(filepath.Join(sa.etcDir, "server.yaml"))
if err != nil {
return err
}
self, err := sourceIPToward(sa.ip)
if err != nil {
return err
}
nets, err := firewall.Networks(ctx, sa.remote, []string{self})
if err == nil {
extra, _ := cfg.FirewallAllow()
nets = append(nets, extra...)
var changed bool
if changed, err = firewall.Install(ctx, sa.remote,
firewall.Spec{Ports: controlPlanePorts(cfg), Allow: nets}); err == nil {
if changed {
sa.done("firewall: llmgw's internal ports answer the cluster's networks only")
} else {
sa.kept("firewall")
}
return nil
}
}
fmt.Fprintf(sa.out, " !! firewall: %v\n", err)
return nil
}
func (sa *serverAdd) connect(ctx context.Context) error {
out, err := sa.sh(ctx, "hostname; sudo -n true && echo sudo-ok", nil)
if err != nil {
+11
View File
@@ -81,6 +81,9 @@ type serverInit struct {
// make it serve something: a mirror, this machine as a node, and a small
// model on it.
noMirror, noNode, noModel bool
// noFirewall leaves this machine's firewall, and the store members',
// as they are.
noFirewall bool
// mirrorURL is a mirror on another machine to use instead of running one.
mirrorURL string
// pgSetup prepares PostgreSQL and returns the store address; a seam for
@@ -161,6 +164,8 @@ func newServerInitCmd() *cobra.Command {
"use the mirror on another machine, set up there by 'llmgw mirror init': its management API, http://host:8082")
f.BoolVar(&in.noNode, "no-node", false, "do not make this machine a node (and serve no model on it)")
f.BoolVar(&in.noModel, "no-model", false, "do not fetch and serve the starter model ("+starterName+")")
f.BoolVar(&in.noFirewall, "no-firewall", false,
"leave the firewall alone, instead of keeping llmgw's internal ports to the cluster's networks")
f.BoolVar(&in.postgres, "postgres", false,
"keep the store in PostgreSQL on this machine, installing it and moving an existing SQLite store into it")
f.StringArrayVar(&in.patroni, "patroni", nil,
@@ -221,6 +226,12 @@ func (in *serverInit) run(ctx context.Context) error {
return fmt.Errorf("starter model: %w", err)
}
}
if hasSystemd && !in.noFirewall {
if err := in.ensureFirewall(ctx); err != nil {
// The cluster works without it; said, not fatal.
fmt.Fprintf(in.out, " !! firewall: %v; llmgw's ports answer every network this machine is on\n", err)
}
}
in.nextSteps()
return nil
}
+124
View File
@@ -0,0 +1,124 @@
package cli
import (
"context"
"fmt"
"net"
"net/netip"
"os"
"strconv"
"strings"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
)
// ensureFirewall keeps this control plane's internal ports to the networks
// the cluster is on: the agents' stream, the metrics, the mirror's files,
// the engines and etcd when they are here. The client API is not among
// them; it is meant to be reached.
func (in *serverInit) ensureFirewall(ctx context.Context) error {
cfg, err := config.Load(in.configPath())
if err != nil {
return err
}
ports := controlPlanePorts(cfg)
if !in.noMirror && in.mirrorURL == "" {
ports = append(ports, firewall.MirrorFiles)
}
if !in.noNode {
from, to := cfg.EnginePorts()
ports = append(ports, firewall.Engines(from, to), firewall.NodeExporter)
}
if _, err := os.Stat("/etc/default/etcd"); err == nil {
// The store's third etcd vote.
ports = append(ports, firewall.EtcdClient, firewall.EtcdPeer)
}
self, err := ipv4Of(in.reachableAddr())
if err != nil {
return err
}
nets, err := firewall.Networks(ctx, pgha.Local{}, []string{self})
if err != nil {
return err
}
extra, _ := cfg.FirewallAllow()
nets = append(nets, extra...)
changed, err := firewall.Install(ctx, pgha.Local{}, firewall.Spec{Ports: ports, Allow: nets})
if err != nil {
return err
}
shown := make([]string, len(nets))
for i, n := range nets {
shown[i] = n.String()
}
if changed {
in.done("firewall: llmgw's internal ports answer %s only (%s)", strings.Join(shown, ", "), firewall.RulesFile)
} else {
in.kept("firewall: llmgw's internal ports answer %s only", strings.Join(shown, ", "))
}
return nil
}
// storeFirewall keeps each store member's database, Patroni and etcd ports to
// the networks it reaches this control plane and the other members on. A
// member it cannot be set on is said, and left as it was.
func (in *serverInit) storeFirewall(ctx context.Context, members []pgha.Node, self string, extra []netip.Prefix) {
ports := []firewall.Port{firewall.Postgres, firewall.PatroniAPI, firewall.EtcdClient, firewall.EtcdPeer,
firewall.NodeExporter}
for _, m := range members {
toward := []string{self}
for _, o := range members {
if o.IP != m.IP {
toward = append(toward, o.IP)
}
}
nets, err := firewall.Networks(ctx, m.Remote, toward)
if err == nil {
nets = append(nets, extra...)
var changed bool
if changed, err = firewall.Install(ctx, m.Remote, firewall.Spec{Ports: ports, Allow: nets}); err == nil {
if changed {
in.done("firewall on %s: the store's ports answer the cluster's networks only", m.Name)
} else {
in.kept("firewall on %s", m.Name)
}
continue
}
}
fmt.Fprintf(in.out, " !! firewall on %s: %v\n", m.Name, err)
}
}
// controlPlanePorts are a control plane's internal ports, from its
// configuration: the agents' stream, and the metrics when they listen
// beyond this machine.
func controlPlanePorts(cfg config.Server) []firewall.Port {
var ports []firewall.Port
if p := portOf(cfg.AgentListen); p > 0 {
ports = append(ports, firewall.Single(p, "control plane: the agents' stream"))
}
if p := portOf(cfg.MetricsListen); p > 0 && !loopbackListen(cfg.MetricsListen) {
ports = append(ports, firewall.Single(p, "control plane: metrics, for Prometheus"))
}
return ports
}
func portOf(listen string) int {
_, port, err := net.SplitHostPort(listen)
if err != nil {
return 0
}
p, _ := strconv.Atoi(port)
return p
}
func loopbackListen(listen string) bool {
host, _, err := net.SplitHostPort(listen)
if err != nil {
return false
}
ip := net.ParseIP(host)
return host == "localhost" || (ip != nil && ip.IsLoopback())
}
+4
View File
@@ -81,6 +81,10 @@ func (in *serverInit) ensurePatroni(ctx context.Context) (err error) {
if err := in.writeSecret(patroniRestFile(in.etcDir), setup.RestPassword); err != nil {
return err
}
if !in.noFirewall {
extra, _ := cfg.FirewallAllow()
in.storeFirewall(ctx, setup.DB, self, extra)
}
current := cfg.Auth.Store
if current == "" {
current = in.storeAddress()
+9
View File
@@ -121,6 +121,12 @@ type Server struct {
// Empty means /var/backups/llmgw-server.
BackupDir string `koanf:"backup_dir"`
// FirewallNetworks are networks, besides those each machine reaches the
// cluster on, that llmgw's internal ports answer: a monitoring host on
// another subnet, nodes behind a router. Written into every machine's
// firewall by `server init`, `node add` and `server add`.
FirewallNetworks []string `koanf:"firewall_networks"`
// BackupRecipients encrypt every archive `llmgw server backup` writes, to
// whoever holds one of their keys: age recipients (age1…) or SSH public
// keys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds
@@ -630,6 +636,9 @@ func (s Server) Validate() error {
if _, err := ParseBackupRecipients(s.BackupRecipients); err != nil {
return err
}
if _, err := s.FirewallAllow(); err != nil {
return err
}
if err := s.AgentTLS.Validate(); err != nil {
return err
}
+34
View File
@@ -0,0 +1,34 @@
package config
import (
"fmt"
"net/netip"
"strings"
)
// FirewallAllow reads firewall_networks: networks, or single addresses.
func (s Server) FirewallAllow() ([]netip.Prefix, error) {
var out []netip.Prefix
for i, n := range s.FirewallNetworks {
n = strings.TrimSpace(n)
if p, err := netip.ParsePrefix(n); err == nil {
out = append(out, p.Masked())
continue
}
a, err := netip.ParseAddr(n)
if err != nil {
return nil, fmt.Errorf("firewall_networks[%d] is %q: a network (10.0.1.0/24) or an address", i, n)
}
out = append(out, netip.PrefixFrom(a, a.BitLen()))
}
return out, nil
}
// EnginePorts is the range engines listen in on every node.
func (s Server) EnginePorts() (int, int) {
if len(s.Control.PortRange) == 2 {
return s.Control.PortRange[0], s.Control.PortRange[1]
}
d := Default().Control.PortRange
return d[0], d[1]
}
+1
View File
@@ -225,6 +225,7 @@ var Reference = []Block{
{Key: "engines", Type: "map[string]string", Doc: "Engines are the engines every node installs, and at which version. The\nagents install them themselves — the right build for their hardware,\nchecked against its publisher's checksum and seen to run — so a node is\nnever trusted to already have what somebody once put there.\n\nUnset keeps DefaultEngines. A version replaces the default; an empty\none leaves that engine to be installed by hand.", Nested: ""},
{Key: "engine_overrides", Type: "[]EngineOverride", Doc: "EngineOverrides give some nodes different engines from the rest: a\nvLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at\nall on nodes meant only for llama.cpp. Each picks nodes and lays its\nengines over the cluster's, in order, later ones last; an empty version\nremoves that engine from the nodes it picks.", Nested: ""},
{Key: "backup_dir", Type: "string", Doc: "BackupDir is where `llmgw server backup` writes, and where the newest\narchive's age is read for llmgw_backup_last_success_timestamp_seconds.\nEmpty means /var/backups/llmgw-server.", Nested: ""},
{Key: "firewall_networks", Type: "[]string", Doc: "FirewallNetworks are networks, besides those each machine reaches the\ncluster on, that llmgw's internal ports answer: a monitoring host on\nanother subnet, nodes behind a router. Written into every machine's\nfirewall by `server init`, `node add` and `server add`.", Nested: ""},
{Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""},
{Key: "releases", Type: "string", Doc: "Releases is where `llmgw ctl upgrade` fetches release binaries from,\nfor this machine and — through their agents — every node. Empty means\nthe project's own releases.", Nested: ""},
{Key: "router", Type: "Router", Doc: "", Nested: "config.Router"},
+29
View File
@@ -93,6 +93,10 @@ type Facts struct {
EnginesDir string `json:"engines_dir,omitempty"`
EnginesFreeGB float64 `json:"engines_free_gb,omitempty"`
VLLMInstalled bool `json:"vllm_installed"`
// Firewall is whether llmgw's nftables table is loaded: "present",
// "absent", or "" when it could not be read (not root).
Firewall string `json:"firewall,omitempty"`
}
// Modules checked, and why each matters.
@@ -224,9 +228,22 @@ func Collect(ctx context.Context, o Options) Facts {
f.Engines[name] = firstLine(string(out))
}
}
f.Firewall = firewallState(ctx)
return f
}
// firewallState reads whether llmgw's table is loaded.
func firewallState(ctx context.Context) string {
out, err := exec.CommandContext(ctx, "nft", "list", "table", "inet", "llmgw").CombinedOutput()
switch {
case err == nil:
return "present"
case strings.Contains(string(out), "No such file or directory"):
return "absent"
}
return "" // no nft, or not root: unknown
}
func (f *Facts) readConfig(o Options) {
path := o.ConfigPath
if path == "" {
@@ -573,6 +590,18 @@ func Evaluate(f Facts) []Finding {
"LimitMEMLOCK=infinity in the unit")
}
// --- firewall ---
// Only on a node already set up: before that, onboarding writes it.
if f.ConfigPath != "" {
switch f.Firewall {
case "present":
add(OK, "firewall", "llmgw's ports answer the cluster's networks only", "")
case "absent":
add(Warn, "firewall", "no llmgw firewall: the engines' ports answer every network this machine is on",
"run node add for it again, or list extra networks in firewall_networks first")
}
}
sort.SliceStable(out, func(i, j int) bool { return rank(out[i].Level) < rank(out[j].Level) })
return out
}
+11
View File
@@ -248,3 +248,14 @@ func TestAnAdapterNoFabricUsesDoesNotKeepTheNodeOut(t *testing.T) {
has(t, fs, Fail, "mlx5_2/1 is RoCE with no RoCE v2 GID entry")
has(t, fs, Fail, "ib_uverbs is not loaded")
}
func TestAMissingFirewallIsSaidOnASetUpNode(t *testing.T) {
f := Facts{ConfigPath: "/etc/llmgw/agent.yaml", Firewall: "absent"}
has(t, Evaluate(f), Warn, "no llmgw firewall")
f.ConfigPath = "" // before onboarding has written one
for _, x := range Evaluate(f) {
if x.Area == "firewall" {
t.Errorf("said before the node is set up: %+v", x)
}
}
}
+319
View File
@@ -0,0 +1,319 @@
// Package firewall writes and applies the nftables rules that keep llmgw's
// internal ports to the cluster's own networks.
//
// Only llmgw's ports, in a table of its own. The machines llmgw is put on
// are often not llmgw's alone — a GPU node shared with a desktop's ComfyUI,
// a database host with other databases — so the policy is not "deny
// everything else": it is "these ports answer the cluster's networks and
// this machine, and nobody else". Whatever else the machine serves, and
// whatever firewall it already has, is left as it was. The client API
// (:8080) is not among them: it is meant to be reached.
package firewall
import (
"context"
"errors"
"fmt"
"net/netip"
"slices"
"strconv"
"strings"
)
// Runner runs a shell script on a machine, as onboarding's SSH and the
// local machine both do.
type Runner interface {
Run(ctx context.Context, script string, stdin []byte) ([]byte, error)
}
// Port is one port, or a range, and what answers there.
type Port struct {
From, To int
What string
}
// Single is one port.
func Single(port int, what string) Port { return Port{From: port, To: port, What: what} }
func (p Port) nft() string {
if p.To > p.From {
return strconv.Itoa(p.From) + "-" + strconv.Itoa(p.To)
}
return strconv.Itoa(p.From)
}
// The ports of each part of a cluster.
var (
AgentPort = Single(7443, "control plane: the agents' stream")
NodeExporter = Single(9100, "node exporter: the machine's metrics, for Prometheus")
MirrorFiles = Single(8081, "mirror: weights, for the nodes")
MirrorAPI = Single(8082, "mirror: its management API, for the control planes")
Postgres = Single(5432, "store: PostgreSQL")
PatroniAPI = Single(8008, "store: Patroni's API")
EtcdClient = Single(2379, "store: etcd, for Patroni")
EtcdPeer = Single(2380, "store: etcd, between its members")
)
// Engines is the engines' port range on a node: their HTTP, for the
// router, and their rendezvous, for the other nodes of a replica.
func Engines(from, to int) Port {
return Port{From: from, To: to, What: "engines: HTTP and rendezvous"}
}
// Spec is what a machine's ruleset holds: its llmgw ports, and the
// networks they answer besides the machine itself.
type Spec struct {
Ports []Port
Allow []netip.Prefix
}
// Table is the nftables table llmgw's rules live in, and nothing else does.
const Table = "llmgw"
// RulesFile is where the ruleset is kept, for the unit that loads it at boot.
const RulesFile = "/etc/llmgw/firewall.nft"
// Unit is the systemd unit that loads it.
const Unit = "llmgw-firewall.service"
// Script is the ruleset as nft reads it. It replaces llmgw's table whole,
// so applying it again, or a changed one, leaves no stale rule behind.
func Script(s Spec) string {
ports := slices.Clone(s.Ports)
slices.SortFunc(ports, func(a, b Port) int { return a.From - b.From })
ports = slices.CompactFunc(ports, func(a, b Port) bool { return a.From == b.From && a.To == b.To })
var v4, v6 []string
for _, p := range s.Allow {
p = p.Masked()
if p.Addr().Is4() {
v4 = append(v4, p.String())
} else {
v6 = append(v6, p.String())
}
}
slices.Sort(v4)
v4 = slices.Compact(v4)
slices.Sort(v6)
v6 = slices.Compact(v6)
var list []string
for _, p := range ports {
list = append(list, p.nft())
}
set := "{ " + strings.Join(list, ", ") + " }"
var b strings.Builder
b.WriteString("# Written by llmgw. llmgw's internal ports answer this machine and the\n")
b.WriteString("# cluster's networks only; nothing else on the machine is touched.\n")
for _, p := range ports {
fmt.Fprintf(&b, "# %-11s %s\n", p.nft(), p.What)
}
fmt.Fprintf(&b, "table inet %s\ndelete table inet %s\n", Table, Table)
fmt.Fprintf(&b, "table inet %s {\n", Table)
b.WriteString("\tchain input {\n\t\ttype filter hook input priority filter - 1; policy accept;\n")
b.WriteString("\t\tiif \"lo\" accept\n")
if len(ports) > 0 {
if len(v4) > 0 {
fmt.Fprintf(&b, "\t\tip saddr { %s } tcp dport %s accept\n", strings.Join(v4, ", "), set)
}
if len(v6) > 0 {
fmt.Fprintf(&b, "\t\tip6 saddr { %s } tcp dport %s accept\n", strings.Join(v6, ", "), set)
}
fmt.Fprintf(&b, "\t\ttcp dport %s drop\n", set)
}
b.WriteString("\t}\n}\n")
return b.String()
}
// unitFile loads the ruleset at boot, and again whenever nftables.service
// is (re)started — whose own configuration commonly begins by flushing every
// table, ours included.
const unitFile = `[Unit]
Description=llmgw firewall: llmgw's internal ports for the cluster's networks only
After=nftables.service
PartOf=nftables.service
Before=network-online.target llmgw-server.service llmgw-agent.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/nft -f ` + RulesFile + `
ExecReload=/usr/sbin/nft -f ` + RulesFile + `
ExecStop=/usr/sbin/nft delete table inet ` + Table + `
[Install]
WantedBy=multi-user.target nftables.service
`
// Install writes the ruleset and its unit on a machine and applies them,
// installing nftables first where it is missing. It reports whether
// anything changed. The ruleset is checked by nft before it is put in
// place, and a broken one changes nothing.
//
// What the machine's ruleset already holds is kept: a machine can be more
// than one thing — a store member and a node — and whichever of them is set
// up second must not take the first one's ports out of the rules.
func Install(ctx context.Context, r Runner, s Spec) (bool, error) {
if old, err := r.Run(ctx, "sudo cat "+RulesFile+" 2>/dev/null || true", nil); err == nil {
prev := parse(string(old))
s.Ports = append(prev.Ports, s.Ports...)
s.Allow = append(prev.Allow, s.Allow...)
}
if len(s.Allow) == 0 {
// Ports for this machine alone would cut every node and control
// plane off. Not a firewall anybody asked for.
return false, errors.New("no network to let in; the ports were left open rather than closed to every other machine")
}
rules := Script(s)
body := rules + "\x00" + unitFile
out, err := r.Run(ctx, `set -e
if ! command -v nft >/dev/null 2>&1 && [ ! -x /usr/sbin/nft ]; then
if command -v apt-get >/dev/null 2>&1; then
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq nftables >/dev/null
elif command -v dnf >/dev/null 2>&1; then
sudo dnf install -y -q nftables >/dev/null
else
echo "no nft, and no apt-get or dnf to install it with" >&2; exit 1
fi
fi
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
cat > "$t/all"
python3 - "$t" <<'PY'
import sys
d = sys.argv[1]
rules, unit = open(d + "/all", "rb").read().split(b"\x00")
open(d + "/rules", "wb").write(rules)
open(d + "/unit", "wb").write(unit)
PY
sudo /usr/sbin/nft -c -f "$t/rules"
changed=no
if ! sudo cmp -s "$t/rules" `+RulesFile+`; then
sudo install -d -m 755 /etc/llmgw
sudo install -m 644 "$t/rules" `+RulesFile+`
changed=yes
fi
if ! sudo cmp -s "$t/unit" /etc/systemd/system/`+Unit+`; then
sudo install -m 644 "$t/unit" /etc/systemd/system/`+Unit+`
sudo systemctl daemon-reload
changed=yes
fi
sudo systemctl enable `+Unit+` >/dev/null 2>&1
if [ "$changed" = yes ] || ! sudo /usr/sbin/nft list table inet `+Table+` >/dev/null 2>&1; then
sudo /usr/sbin/nft -f `+RulesFile+`
sudo systemctl start `+Unit+` >/dev/null 2>&1 || true
echo changed
else
echo same
fi`, []byte(body))
if err != nil {
return false, fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
}
return strings.HasSuffix(strings.TrimSpace(string(out)), "changed"), nil
}
// Networks are the networks a machine reaches others on: for each address,
// the prefix of the interface its route leaves by — or the address alone,
// when the route goes through a gateway and the network behind it is not
// the machine's to know.
func Networks(ctx context.Context, r Runner, toward []string) ([]netip.Prefix, error) {
var out []netip.Prefix
for _, t := range toward {
a, err := netip.ParseAddr(t)
if err != nil {
continue
}
b, err := r.Run(ctx, "ip -o route get "+a.String()+" && ip -o addr show", nil)
if err != nil {
return nil, fmt.Errorf("finding the network toward %s: %v: %s", a, err, strings.TrimSpace(string(b)))
}
p, ok := networkToward(string(b), a)
if !ok {
return nil, fmt.Errorf("cannot tell the network toward %s from: %s", a, strings.TrimSpace(string(b)))
}
out = append(out, p)
}
return out, nil
}
// networkToward reads `ip -o route get A` followed by `ip -o addr show`.
func networkToward(text string, a netip.Addr) (netip.Prefix, bool) {
lines := strings.Split(strings.TrimSpace(text), "\n")
if len(lines) == 0 {
return netip.Prefix{}, false
}
route := strings.Fields(lines[0])
field := func(name string) string {
for i := 0; i+1 < len(route); i++ {
if route[i] == name {
return route[i+1]
}
}
return ""
}
if field("via") != "" {
// Routed: the address alone.
return netip.PrefixFrom(a, a.BitLen()), true
}
dev, src := field("dev"), field("src")
if route[0] == "local" {
// This machine's own address: the network of the interface it is on.
dev, src = "", a.String()
}
for _, l := range lines[1:] {
f := strings.Fields(l)
// "2: eth0 inet 10.0.0.147/24 brd ..."
if len(f) < 4 || (dev != "" && strings.TrimSuffix(f[1], ":") != dev) || (f[2] != "inet" && f[2] != "inet6") {
continue
}
p, err := netip.ParsePrefix(f[3])
if err != nil || (src != "" && p.Addr().String() != src) {
continue
}
if p.Contains(a) {
return p.Masked(), true
}
}
return netip.PrefixFrom(a, a.BitLen()), true
}
// parse reads back a ruleset Script wrote: its ports from the comments
// naming them, its networks from the rules.
func parse(text string) Spec {
var s Spec
for _, l := range strings.Split(text, "\n") {
if rest, ok := strings.CutPrefix(l, "# "); ok {
f := strings.Fields(rest)
if len(f) == 0 {
continue
}
from, to, isRange := strings.Cut(f[0], "-")
a, err1 := strconv.Atoi(from)
b := a
var err2 error
if isRange {
b, err2 = strconv.Atoi(to)
}
if err1 == nil && err2 == nil {
s.Ports = append(s.Ports, Port{From: a, To: b, What: strings.Join(f[1:], " ")})
}
continue
}
l = strings.TrimSpace(l)
if !strings.HasPrefix(l, "ip saddr {") && !strings.HasPrefix(l, "ip6 saddr {") {
continue
}
inner := l[strings.Index(l, "{")+1 : strings.Index(l, "}")]
for _, n := range strings.Split(inner, ",") {
if p, err := netip.ParsePrefix(strings.TrimSpace(n)); err == nil {
s.Allow = append(s.Allow, p)
}
}
}
return s
}
// Present reports whether a machine has llmgw's table loaded.
func Present(ctx context.Context, r Runner) bool {
_, err := r.Run(ctx, "sudo -n /usr/sbin/nft list table inet "+Table+" >/dev/null 2>&1", nil)
return err == nil
}
+60
View File
@@ -0,0 +1,60 @@
package firewall
import (
"net/netip"
"strings"
"testing"
)
func TestTheRulesetGuardsOnlyLlmgwsPorts(t *testing.T) {
s := Script(Spec{
Ports: []Port{NodeExporter, Engines(20000, 20999), NodeExporter},
Allow: []netip.Prefix{netip.MustParsePrefix("10.0.0.147/24"), netip.MustParsePrefix("fd00::1/64"),
netip.MustParsePrefix("10.0.0.0/24")},
})
for _, want := range []string{
"table inet llmgw\ndelete table inet llmgw\ntable inet llmgw {",
"policy accept;",
`iif "lo" accept`,
"ip saddr { 10.0.0.0/24 } tcp dport { 9100, 20000-20999 } accept",
"ip6 saddr { fd00::/64 } tcp dport { 9100, 20000-20999 } accept",
"tcp dport { 9100, 20000-20999 } drop",
} {
if !strings.Contains(s, want) {
t.Errorf("no %q in:\n%s", want, s)
}
}
if strings.Contains(s, "policy drop") || strings.Contains(s, "flush ruleset") {
t.Errorf("the ruleset reaches beyond llmgw's ports:\n%s", s)
}
}
func TestTheNetworkIsTheInterfacesOrTheAddressAlone(t *testing.T) {
addrs := `1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
2: eth0 inet 10.0.0.147/24 brd 10.0.0.255 scope global eth0\ valid_lft forever preferred_lft forever
3: eth1 inet 192.168.5.7/16 brd 192.168.255.255 scope global eth1\ valid_lft forever preferred_lft forever`
cases := []struct {
route, addr, want string
}{
{"10.0.0.140 dev eth0 src 10.0.0.147 uid 1000 \\ cache", "10.0.0.140", "10.0.0.0/24"},
{"172.16.3.4 via 10.0.0.1 dev eth0 src 10.0.0.147 uid 1000 \\ cache", "172.16.3.4", "172.16.3.4/32"},
{"192.168.9.9 dev eth1 src 192.168.5.7 uid 1000 \\ cache", "192.168.9.9", "192.168.0.0/16"},
// Its own address: the network it is on.
{"local 10.0.0.147 dev lo src 10.0.0.147 uid 1000 \\ cache <local>", "10.0.0.147", "10.0.0.0/24"},
}
for _, c := range cases {
got, ok := networkToward(c.route+"\n"+addrs, netip.MustParseAddr(c.addr))
if !ok || got.String() != c.want {
t.Errorf("toward %s: %v %v, expected %s", c.addr, got, ok, c.want)
}
}
}
func TestARulesetReadsBackAsItsSpec(t *testing.T) {
in := Spec{Ports: []Port{Postgres, Engines(20000, 20999)},
Allow: []netip.Prefix{netip.MustParsePrefix("10.0.0.0/24"), netip.MustParsePrefix("fd00::/64")}}
got := parse(Script(in))
if Script(got) != Script(in) {
t.Errorf("read back as\n%s\nfrom\n%s", Script(got), Script(in))
}
}
+46 -1
View File
@@ -20,6 +20,8 @@ import (
"fmt"
"net"
"net/http"
"net/netip"
"net/url"
"os"
"path/filepath"
"regexp"
@@ -31,6 +33,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/doctor"
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
)
@@ -63,6 +66,12 @@ func ParseFabric(s string) (Fabric, error) {
type Options struct {
// NoNodeExporter leaves Prometheus' node exporter off the machine.
NoNodeExporter bool
// NoFirewall leaves the machine's firewall as it is. Otherwise its llmgw
// ports — FirewallPorts — answer only the networks it reaches the
// control planes on, and FirewallAllow.
NoFirewall bool
FirewallPorts []firewall.Port
FirewallAllow []netip.Prefix
// Addr is how to reach the machine over SSH, and User who to log in as.
Addr, User string
@@ -687,7 +696,7 @@ func (r *Runner) checkEngines(ctx context.Context) []string {
// --- start ---
func (r *Runner) start(ctx context.Context, _ *Result) error {
func (r *Runner) start(ctx context.Context, res *Result) error {
// The user engines run as, so the agent drops root before starting one:
// in the groups that own the GPU devices.
if out, err := r.Remote.Run(ctx, `set -e
@@ -709,9 +718,45 @@ systemctl is-active llmgw-agent`, nil); err != nil {
if !r.Opts.NoNodeExporter {
r.nodeExporter(ctx)
}
if !r.Opts.NoFirewall && len(r.Opts.FirewallPorts) > 0 {
r.firewall(ctx, res)
}
return nil
}
// firewall keeps the node's llmgw ports — its engines, its metrics — to the
// networks it reaches the control planes on. A warning when it cannot: the
// node serves either way, and says how to see why.
func (r *Runner) firewall(ctx context.Context, res *Result) {
var toward []string
for _, s := range res.Servers {
u, err := url.Parse(s)
if err != nil {
continue
}
if _, err := netip.ParseAddr(u.Hostname()); err == nil {
toward = append(toward, u.Hostname())
} else if ips, err := net.LookupIP(u.Hostname()); err == nil && len(ips) > 0 {
toward = append(toward, ips[0].String())
}
}
nets, err := firewall.Networks(ctx, r.Remote, toward)
if err == nil {
nets = append(nets, r.Opts.FirewallAllow...)
_, err = firewall.Install(ctx, r.Remote, firewall.Spec{Ports: r.Opts.FirewallPorts, Allow: nets})
}
if err != nil {
r.emit(StepStart, Warn, "the firewall was not set: "+err.Error()+
"; the node's engine ports answer every network it is on", "")
return
}
var shown []string
for _, n := range nets {
shown = append(shown, n.String())
}
r.emit(StepStart, Info, "llmgw's ports on the node answer "+strings.Join(shown, ", ")+" only ("+firewall.RulesFile+")", "")
}
// nodeExporter installs Prometheus' node exporter from the distribution, for
// what llmgw does not measure: network and disk traffic, filesystems. Best
// effort: a node without it serves as well, so a failure is a warning.