Keep llmgw's internal ports to the cluster's networks with an nftables table of its own
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
7e57863edd
commit
7b6276901a
17 files changed
+800
-10
No files matched your search
+39
-8
@@ -1071,6 +1071,39 @@ thing to get right is the health check: point it at `/ready`, not `/health`.
|
||||
so a load balancer using it would send half the traffic to something that
|
||||
refuses every request.
|
||||
|
||||
## The firewall
|
||||
|
||||
`server init`, `node add`, `server add` and `mirror init` each keep that
|
||||
machine's **internal** llmgw ports to the cluster's own networks, with an
|
||||
nftables table of llmgw's own (`inet llmgw`, in `/etc/llmgw/firewall.nft`,
|
||||
loaded at boot by `llmgw-firewall.service`):
|
||||
|
||||
| Machine | Ports, for the cluster only |
|
||||
| --- | --- |
|
||||
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
|
||||
| node | engine ports (`control.port_range`, 20000–20999), 9100 |
|
||||
| store member | 5432, 8008, 2379, 2380, 9100 |
|
||||
| mirror | 8081, 8082, 9100 |
|
||||
|
||||
"The cluster's networks" are, on each machine, the network of the interface
|
||||
it reaches the control planes (or the other store members) on — or just
|
||||
their address, when the route goes through a gateway. Anything else that
|
||||
must reach those ports, a Prometheus on another subnet or nodes behind a
|
||||
router, goes in `server.yaml` before the machines are set up:
|
||||
|
||||
```yaml
|
||||
firewall_networks: [10.0.5.0/24, 192.168.1.20]
|
||||
```
|
||||
|
||||
Only those ports are touched. The table's policy is to accept, so whatever
|
||||
else the machine serves — and whatever firewall it already has — is left as
|
||||
it was; the client API (`:8080`) is not closed, being meant to be reached.
|
||||
Running a command again adds to what is there rather than replacing it, so a
|
||||
machine that is both a store member and a node keeps both sets. `llmgw
|
||||
doctor` on a node says when the table is missing. `--no-firewall` on any of
|
||||
the four commands leaves the firewall alone; `sudo systemctl disable --now
|
||||
llmgw-firewall` removes it.
|
||||
|
||||
## Upgrading
|
||||
|
||||
On the control plane:
|
||||
@@ -1347,11 +1380,9 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
has them. Single-node placement on real GPUs is the well-trodden path; if you
|
||||
are the first to run a TP group across real cards, run `llmgw doctor --json`
|
||||
on each node first and keep the reports.
|
||||
- **Two engine setups are weaker than the rest.** An engine's key is passed
|
||||
in its environment, where only root and its own user can read it, except
|
||||
SGLang's, which takes it only as a flag: other users on the node can see it
|
||||
in the process list. Mount `/proc` with `hidepid=2` on nodes where that
|
||||
matters. vLLM's `launch_mode: ray` starts a Ray head without
|
||||
authentication, so anyone who can reach its port can run code on the
|
||||
node. The dashboard is off, but firewall the port to the other nodes in the
|
||||
replica, or use the default `mp` launch mode, which needs no Ray at all.
|
||||
- **One engine setup is weaker than the rest.** vLLM's `launch_mode: ray`
|
||||
starts a Ray head without authentication, so anyone who can reach its
|
||||
port can run code on the node, and Ray's ports are not among those
|
||||
llmgw's firewall guards. The dashboard is off, but firewall the port to
|
||||
the other nodes in the replica, or use the default `mp` launch mode, which
|
||||
needs no Ray at all.
|
||||
@@ -185,6 +185,7 @@ rather than repeated.
|
||||
| `engines` | `map[string]string` | [Server](#server) |
|
||||
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
|
||||
| `backup_dir` | `string` | [Server](#server) |
|
||||
| `firewall_networks` | `[]string` | [Server](#server) |
|
||||
| `backup_recipients` | `[]string` | [Server](#server) |
|
||||
| `releases` | `string` | [Server](#server) |
|
||||
| `router` | `Router` | [Server](#server) |
|
||||
@@ -1493,6 +1494,15 @@ BackupDir is where `llmgw server backup` writes, and where the newest
|
||||
archive's age is read for llmgw_backup_last_success_timestamp_seconds.
|
||||
Empty means /var/backups/llmgw-server.
|
||||
|
||||
### `firewall_networks`
|
||||
|
||||
`[]string`
|
||||
|
||||
FirewallNetworks are networks, besides those each machine reaches the
|
||||
cluster on, that llmgw's internal ports answer: a monitoring host on
|
||||
another subnet, nodes behind a router. Written into every machine's
|
||||
firewall by `server init`, `node add` and `server add`.
|
||||
|
||||
### `backup_recipients`
|
||||
|
||||
`[]string`
|
||||
|
||||
Reference in new issue
Block a user