Ilker ManapandClaude Opus 5.5
eeaffd764d
Found by a three-machine install on fresh VMs. The second standby's run asked for the floating address's interface with `ip route get <vip>`. The machine already held the address, so that answered lo, and keepalived would not start. The interface now comes from the route to the address's network. Every cluster's VRRP router id was 51, so the PoC's two control planes and the new three on one LAN each logged the other's adverts as an invalid password; the id now comes from the address. The Caddyfile, deploy/ha and install.md send clients to the control plane whose /leader is 200, not /ready: /ready is 503 on a leader with no model yet and would take the portal away with it. Setup no longer hides a key file's path as if it were a password, and the starter's text no longer says it runs elsewhere when this machine is a node. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A reverse proxy in front of llmgw
The client-facing port (8080) is plain HTTP. On anything but a trusted network, put it behind a proxy that holds a publicly trusted certificate.
| File | For |
|---|---|
| Caddyfile | Caddy — obtains and renews its own certificate; the least work |
| nginx.conf | nginx, with a certificate from certbot |
Both are also attached to every release.
The /about page shows both, from copies in internal/web/proxy; a test fails
when those differ from these, so change them here and copy them over.
Four things both files get right, and any other proxy has to as well:
- Streaming is not buffered. Replies are server-sent events; a buffering proxy delivers them in one burst at the end.
- The read timeout is longer than a model's silence. nginx's default of 60 seconds turns a slow first token into a 504.
X-Forwarded-ProtoandX-Forwarded-Hostare set by the proxy, replacing whatever the client sent. llmgw uses the first to mark session cookiesSecureand the second for the base URL on its landing page.- The body limit matches llmgw's (32 MiB,
router.max_body_bytes), so an oversized request is refused before it is uploaded.
Only port 8080 is proxied. Agents reach 7443 directly — it is mutual TLS with the cluster's own authority, and a proxy terminating it would take away the node identity it carries. 9090 (admin, pprof) stays on loopback; 9091 (metrics) goes to your monitoring host only.
With a proxy on the same machine, bind llmgw to loopback so nothing reaches it around the proxy:
# /etc/llmgw/server.yaml
listen: "127.0.0.1:8080"