Ilker ManapandClaude Opus 5.5
069d334652
install.md said the client port belongs behind a reverse proxy and gave nothing to put there. deploy/proxy/ has both, attached to every release. What they get right is what a first attempt gets wrong: streamed replies are not buffered, the read timeout outlasts a model's silence before its first token (nginx's 60s default is a 504), the body limit matches the router's, and X-Forwarded-Proto and -Host are set by the proxy rather than passed through — llmgw marks session cookies Secure from the first and builds the landing page's base URL from the second. The Caddyfile was validated and run against a streaming upstream: events arrive as sent, and forged forwarded headers are replaced. nginx uses `listen ... http2` because `http2 on;` needs 1.25.1 and Ubuntu 22.04 and Debian 12 ship older. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
42 lines
1.5 KiB
Caddyfile
42 lines
1.5 KiB
Caddyfile
# llmgw behind Caddy: the client-facing port (8080) on a public name, with a
|
|
# certificate Caddy obtains and renews by itself.
|
|
#
|
|
# Replace llm.example.com, point its DNS at this machine, open 80 and 443, and
|
|
# run: caddy run --config /etc/caddy/Caddyfile
|
|
#
|
|
# Only 8080 goes through here. 7443 (agents) is mutual TLS with the cluster's
|
|
# own authority and must reach llmgw directly; 9090 and 9091 stay private.
|
|
|
|
llm.example.com {
|
|
# The router reads at most 32 MiB of body (router.max_body_bytes). Refusing
|
|
# more here gives the client a 413 before the upload, not after it.
|
|
request_body {
|
|
max_size 32MB
|
|
}
|
|
|
|
reverse_proxy 127.0.0.1:8080 {
|
|
# Tokens are written the moment they are generated. Without this a
|
|
# streamed reply can arrive in bursts, and a client measuring time to
|
|
# first token measures the proxy instead.
|
|
flush_interval -1
|
|
|
|
# Caddy sets X-Forwarded-Proto and X-Forwarded-Host itself and, with
|
|
# no trusted_proxies, discards whatever the client sent. llmgw relies
|
|
# on that: the proto decides whether session cookies are Secure, and
|
|
# the host is the base URL the landing page tells people to use.
|
|
# Behind a load balancer of your own, name it in trusted_proxies.
|
|
|
|
transport http {
|
|
# A long generation can be silent for a while before the first
|
|
# token. llmgw has its own limits (timeouts.*); this must not be
|
|
# the shorter one.
|
|
response_header_timeout 15m
|
|
}
|
|
}
|
|
|
|
# No `encode` on purpose: compressing a stream buffers it.
|
|
|
|
# Access log to stderr, which under systemd is the journal.
|
|
log
|
|
}
|