Ship Caddy and nginx configurations for the client port
install.md said the client port belongs behind a reverse proxy and gave nothing to put there. deploy/proxy/ has both, attached to every release. What they get right is what a first attempt gets wrong: streamed replies are not buffered, the read timeout outlasts a model's silence before its first token (nginx's 60s default is a 504), the body limit matches the router's, and X-Forwarded-Proto and -Host are set by the proxy rather than passed through — llmgw marks session cookies Secure from the first and builds the landing page's base URL from the second. The Caddyfile was validated and run against a streaming upstream: events arrive as sent, and forged forwarded headers are replaced. nginx uses `listen ... http2` because `http2 on;` needs 1.25.1 and Ubuntu 22.04 and Debian 12 ship older. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
4865a96ec2
commit
069d334652
6 files changed
+153
-2
No files matched your search
@@ -0,0 +1,41 @@
|
||||
# llmgw behind Caddy: the client-facing port (8080) on a public name, with a
|
||||
# certificate Caddy obtains and renews by itself.
|
||||
#
|
||||
# Replace llm.example.com, point its DNS at this machine, open 80 and 443, and
|
||||
# run: caddy run --config /etc/caddy/Caddyfile
|
||||
#
|
||||
# Only 8080 goes through here. 7443 (agents) is mutual TLS with the cluster's
|
||||
# own authority and must reach llmgw directly; 9090 and 9091 stay private.
|
||||
|
||||
llm.example.com {
|
||||
# The router reads at most 32 MiB of body (router.max_body_bytes). Refusing
|
||||
# more here gives the client a 413 before the upload, not after it.
|
||||
request_body {
|
||||
max_size 32MB
|
||||
}
|
||||
|
||||
reverse_proxy 127.0.0.1:8080 {
|
||||
# Tokens are written the moment they are generated. Without this a
|
||||
# streamed reply can arrive in bursts, and a client measuring time to
|
||||
# first token measures the proxy instead.
|
||||
flush_interval -1
|
||||
|
||||
# Caddy sets X-Forwarded-Proto and X-Forwarded-Host itself and, with
|
||||
# no trusted_proxies, discards whatever the client sent. llmgw relies
|
||||
# on that: the proto decides whether session cookies are Secure, and
|
||||
# the host is the base URL the landing page tells people to use.
|
||||
# Behind a load balancer of your own, name it in trusted_proxies.
|
||||
|
||||
transport http {
|
||||
# A long generation can be silent for a while before the first
|
||||
# token. llmgw has its own limits (timeouts.*); this must not be
|
||||
# the shorter one.
|
||||
response_header_timeout 15m
|
||||
}
|
||||
}
|
||||
|
||||
# No `encode` on purpose: compressing a stream buffers it.
|
||||
|
||||
# Access log to stderr, which under systemd is the journal.
|
||||
log
|
||||
}
|
||||
Reference in new issue
Block a user