llmgw and Kong, written down after trying Kong 3.9 in front

docs/kong.md: a plain route or ai-proxy with llmgw as the upstream, the
client's llmgw key passed on so per-team budgets hold, Kong as a provider
behind llmgw, and Kong 3.9's streamed answers counted without completion
tokens. Closes BACKLOG L9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

BACKLOG.md

@@ -1763,7 +1763,9 @@ Agent-to-agent traffic through the gateway: agent cards' URLs rewritten to
 llmgw's, calls audited and traced. The standard is young; wait for a
 customer.
 
-### L9. llmgw behind Kong, written down (S)
+### L9. llmgw behind Kong, written down (S) — done 2026-10-25
+docs/kong.md, tried with kong:3.9: plain route and ai-proxy, keys passed on;
+Kong 3.9 counts no completion tokens of a stream. The original item follows.
 A customer that already runs Kong puts llmgw behind it: `ai-proxy` with
 `provider: openai` and `upstream_url` at llmgw (3.9, open source), or a plain
 route. A page in docs/ with the configuration, tried against Kong 3.9 in a

CHANGELOG.md

@@ -28,6 +28,10 @@ APIs change between minors without a deprecation period.
 
 ### Added
 
+- **llmgw and Kong, written down** ([docs/kong.md](docs/kong.md)): Kong 3.9
+  in front of llmgw by a plain route or its `ai-proxy`, each team's llmgw
+  key passed on; Kong as a provider behind llmgw; and that Kong 3.9 counts
+  no completion tokens of a streamed answer, so bill from llmgw's usage.
 - **A guardrail that is someone else's service.** A policy's `services`
   asks Azure Content Safety, Lakera, NeMo or a company's own classifier over
   HTTP: a body template, keys from files, dotted paths to where the answer

README.md

@@ -183,6 +183,8 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
   cluster that exists
 - [docs/kubernetes.md](docs/kubernetes.md) — the control plane on
   Kubernetes with the Helm chart, and why GPU nodes stay outside it
+- [docs/kong.md](docs/kong.md) — llmgw behind Kong Gateway, or Kong behind
+  llmgw: the configuration, what Kong counts, and which to choose
 - [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
   llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
   when a replica spanning nodes will not run

docs/kong.md

@@ -0,0 +1,95 @@
+# llmgw and Kong
+
+For a site that already runs Kong. The two meet over plain HTTP: Kong in
+front of llmgw, as one more API it governs, or Kong behind llmgw, as an
+OpenAI-compatible provider. Nothing of either is built into the other.
+
+Tried with Kong Gateway 3.9 (`kong:3.9`, the last open-source release; 3.10
+and later need a Kong licence) in DB-less mode, in front of llmgw.
+
+## Kong in front of llmgw
+
+Two ways, both tried.
+
+**A plain route.** Kong forwards the request as it came. Tried with chat
+completions, streamed and not; llmgw's other APIs — Anthropic's
+`/v1/messages`, `/v1/responses`, embeddings, pictures — go the same way,
+since Kong does not read them. Kong adds its own authentication, rate limits
+and logging; llmgw still checks the key, the budget and the guardrails.
+
+```yaml
+_format_version: "3.0"
+services:
+- name: llmgw
+  url: https://llm.example.com          # llmgw's client port
+  routes:
+  - name: llmgw
+    paths: [/llmgw]                      # /llmgw/v1/chat/completions → /v1/chat/completions
+```
+
+**Kong's `ai-proxy` plugin**, for Kong's own token statistics on a chat
+route. llmgw is an OpenAI provider whose `upstream_url` is llmgw:
+
+```yaml
+services:
+- name: llmgw-chat
+  url: https://llm.example.com
+  routes:
+  - name: chat
+    paths: [/chat]
+  plugins:
+  - name: ai-proxy
+    config:
+      route_type: llm/v1/chat
+      logging: {log_statistics: true, log_payloads: false}
+      model:
+        provider: openai
+        name: qwen-coder                 # a model llmgw serves
+        options:
+          upstream_url: https://llm.example.com/v1/chat/completions
+```
+
+**Keys.** Without an `auth` block, `ai-proxy` passes the client's
+`Authorization` header on: each team keeps its own llmgw key, and llmgw's
+budgets, limits, guardrails and usage stay per team. With
+`auth: {header_name: Authorization, header_value: "Bearer sk-llmgw-…"}` Kong
+sends one key of its own instead, and llmgw sees every caller as that key —
+then budgets per team are Kong's to keep.
+
+**What Kong counts.** For a whole answer Kong reads llmgw's `usage` and
+records prompt and completion tokens as llmgw did. For a streamed answer
+Kong 3.9 records an estimate of the prompt tokens and no completion tokens,
+even with `stream_options.include_usage` set — the usage chunk reaches it
+and is not read. llmgw's own accounting counts both; use its usage export
+(`llmgw ctl usage export`) or `/admin/usage` for figures to bill by.
+
+**Timeouts.** A long answer from a model still loading outlasts Kong's
+60-second defaults: set the service's `read_timeout` to the time a model may
+take to start, as for nginx in [install.md](install.md).
+
+## Kong behind llmgw
+
+Not tried here. Kong's AI routes speak OpenAI's API, so llmgw reaches models Kong fronts as
+it reaches any provider: a model whose source is `external`.
+
+```yaml
+- name: kong-gpt
+  source: external
+  external:
+    url: https://kong.example.com/openai     # Kong's route, without /v1
+    model: gpt-4o-mini
+    api_key_file: /etc/llmgw/secrets/kong-consumer.key
+```
+
+The key is the one Kong's own authentication asks of a consumer. As a
+fallback (`fallbacks: [kong-gpt]`) it is used only when the cluster has no
+room.
+
+## Which to choose
+
+Kong in front when the site's API policy says every API goes through Kong:
+llmgw is then one more upstream, and its per-team accounting still holds
+when each team keeps its own key. Kong behind when the site already reaches
+outside providers through Kong and llmgw should too. llmgw does not need
+Kong for anything it does itself — keys, budgets, guardrails, routing to
+its own replicas.