Ilker ManapandClaude Opus 5.5
eeaffd764d
Found by a three-machine install on fresh VMs. The second standby's run asked for the floating address's interface with `ip route get <vip>`. The machine already held the address, so that answered lo, and keepalived would not start. The interface now comes from the route to the address's network. Every cluster's VRRP router id was 51, so the PoC's two control planes and the new three on one LAN each logged the other's adverts as an invalid password; the id now comes from the address. The Caddyfile, deploy/ha and install.md send clients to the control plane whose /leader is 200, not /ready: /ready is 503 on a leader with no model yet and would take the portal away with it. Setup no longer hides a key file's path as if it were a password, and the starter's text no longer says it runs elsewhere when this machine is a node. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
76 lines
2.7 KiB
Caddyfile
76 lines
2.7 KiB
Caddyfile
# llmgw behind Caddy: the client-facing port (8080) on a public name, with a
|
|
# certificate Caddy obtains and renews by itself.
|
|
#
|
|
# Replace llm.example.com, point its DNS at this machine, open 80 and 443, and
|
|
# run: caddy run --config /etc/caddy/Caddyfile
|
|
#
|
|
# Only 8080 goes through here. 7443 (agents) is mutual TLS with the cluster's
|
|
# own authority and must reach llmgw directly; 9090 and 9091 stay private.
|
|
|
|
llm.example.com {
|
|
# The router reads at most 32 MiB of body (router.max_body_bytes). Refusing
|
|
# more here gives the client a 413 before the upload, not after it.
|
|
request_body {
|
|
max_size 32MB
|
|
}
|
|
|
|
# One control plane. With more than one, see the end of this file.
|
|
reverse_proxy 127.0.0.1:8080 {
|
|
# Tokens are written the moment they are generated. Without this a
|
|
# streamed reply can arrive in bursts, and a client measuring time to
|
|
# first token measures the proxy instead.
|
|
flush_interval -1
|
|
|
|
# Caddy sets X-Forwarded-Proto and X-Forwarded-Host itself and, with
|
|
# no trusted_proxies, discards whatever the client sent. llmgw relies
|
|
# on that: the proto decides whether session cookies are Secure, and
|
|
# the host is the base URL the landing page tells people to use.
|
|
# Behind a load balancer of your own, name it in trusted_proxies.
|
|
|
|
transport http {
|
|
# A long generation can be silent for a while before the first
|
|
# token. llmgw has its own limits (timeouts.*); this must not be
|
|
# the shorter one.
|
|
response_header_timeout 15m
|
|
}
|
|
}
|
|
|
|
# No `encode` on purpose: compressing a stream buffers it.
|
|
|
|
# Access log to stderr, which under systemd is the journal.
|
|
log
|
|
}
|
|
|
|
# More than one control plane (server init --standbys, server add): one of
|
|
# them leads, the others stand by. Name them all and send to the one that
|
|
# leads — /leader is 200 there and 503 on a standby. Not /ready: that is 503
|
|
# on a leader with no model running yet, and would take the portal away with
|
|
# it. A standby would pass a request on to the leader anyway, but not while
|
|
# there is none, and the hop is for nothing.
|
|
#
|
|
# With a floating address (--vip) the address already follows the leader:
|
|
# give reverse_proxy that one address instead, and keep the rest as above.
|
|
#
|
|
# llm.example.com {
|
|
# request_body {
|
|
# max_size 32MB
|
|
# }
|
|
# reverse_proxy 10.0.0.157:8080 10.0.0.158:8080 10.0.0.159:8080 {
|
|
# # The first that is healthy: the leader, whichever it is.
|
|
# lb_policy first
|
|
# health_uri /leader
|
|
# health_interval 2s
|
|
# health_timeout 2s
|
|
# # While leadership moves — a few seconds when a leader is lost —
|
|
# # none is healthy; a request waits that long for one rather than
|
|
# # failing at once.
|
|
# lb_try_duration 15s
|
|
# lb_try_interval 250ms
|
|
# flush_interval -1
|
|
# transport http {
|
|
# response_header_timeout 15m
|
|
# }
|
|
# }
|
|
# log
|
|
# }
|