Keep llmgw's internal ports to the cluster's networks with an nftables table of its own

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 11:03:46 +02:00
1 parent 7e57863edd
commit 7b6276901a
17 files changed
+800 -10

No files matched your search

+39 -8
View File
@@ -1071,6 +1071,39 @@ thing to get right is the health check: point it at `/ready`, not `/health`.
so a load balancer using it would send half the traffic to something that
refuses every request.
## The firewall
`server init`, `node add`, `server add` and `mirror init` each keep that
machine's **internal** llmgw ports to the cluster's own networks, with an
nftables table of llmgw's own (`inet llmgw`, in `/etc/llmgw/firewall.nft`,
loaded at boot by `llmgw-firewall.service`):
| Machine | Ports, for the cluster only |
| --- | --- |
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
| node | engine ports (`control.port_range`, 20000–20999), 9100 |
| store member | 5432, 8008, 2379, 2380, 9100 |
| mirror | 8081, 8082, 9100 |
"The cluster's networks" are, on each machine, the network of the interface
it reaches the control planes (or the other store members) on — or just
their address, when the route goes through a gateway. Anything else that
must reach those ports, a Prometheus on another subnet or nodes behind a
router, goes in `server.yaml` before the machines are set up:
```yaml
firewall_networks: [10.0.5.0/24, 192.168.1.20]
```
Only those ports are touched. The table's policy is to accept, so whatever
else the machine serves — and whatever firewall it already has — is left as
it was; the client API (`:8080`) is not closed, being meant to be reached.
Running a command again adds to what is there rather than replacing it, so a
machine that is both a store member and a node keeps both sets. `llmgw
doctor` on a node says when the table is missing. `--no-firewall` on any of
the four commands leaves the firewall alone; `sudo systemctl disable --now
llmgw-firewall` removes it.
## Upgrading
On the control plane:
@@ -1347,11 +1380,9 @@ requests. On a slow model, set `timeouts.generation` higher.
has them. Single-node placement on real GPUs is the well-trodden path; if you
are the first to run a TP group across real cards, run `llmgw doctor --json`
on each node first and keep the reports.
- **Two engine setups are weaker than the rest.** An engine's key is passed
in its environment, where only root and its own user can read it, except
SGLang's, which takes it only as a flag: other users on the node can see it
in the process list. Mount `/proc` with `hidepid=2` on nodes where that
matters. vLLM's `launch_mode: ray` starts a Ray head without
authentication, so anyone who can reach its port can run code on the
node. The dashboard is off, but firewall the port to the other nodes in the
replica, or use the default `mp` launch mode, which needs no Ray at all.
- **One engine setup is weaker than the rest.** vLLM's `launch_mode: ray`
starts a Ray head without authentication, so anyone who can reach its
port can run code on the node, and Ray's ports are not among those
llmgw's firewall guards. The dashboard is off, but firewall the port to
the other nodes in the replica, or use the default `mp` launch mode, which
needs no Ray at all.