diff --git a/BACKLOG.md b/BACKLOG.md index 035e7cc..ebacd4e 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -1763,7 +1763,9 @@ Agent-to-agent traffic through the gateway: agent cards' URLs rewritten to llmgw's, calls audited and traced. The standard is young; wait for a customer. -### L9. llmgw behind Kong, written down (S) +### L9. llmgw behind Kong, written down (S) — done 2026-10-25 +docs/kong.md, tried with kong:3.9: plain route and ai-proxy, keys passed on; +Kong 3.9 counts no completion tokens of a stream. The original item follows. A customer that already runs Kong puts llmgw behind it: `ai-proxy` with `provider: openai` and `upstream_url` at llmgw (3.9, open source), or a plain route. A page in docs/ with the configuration, tried against Kong 3.9 in a diff --git a/CHANGELOG.md b/CHANGELOG.md index 744fcd5..7287bdc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,10 @@ APIs change between minors without a deprecation period. ### Added +- **llmgw and Kong, written down** ([docs/kong.md](docs/kong.md)): Kong 3.9 + in front of llmgw by a plain route or its `ai-proxy`, each team's llmgw + key passed on; Kong as a provider behind llmgw; and that Kong 3.9 counts + no completion tokens of a streamed answer, so bill from llmgw's usage. - **A guardrail that is someone else's service.** A policy's `services` asks Azure Content Safety, Lakera, NeMo or a company's own classifier over HTTP: a body template, keys from files, dotted paths to where the answer diff --git a/README.md b/README.md index 6ebe0df..872f02b 100644 --- a/README.md +++ b/README.md @@ -183,6 +183,8 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console. cluster that exists - [docs/kubernetes.md](docs/kubernetes.md) — the control plane on Kubernetes with the Helm chart, and why GPU nodes stay outside it +- [docs/kong.md](docs/kong.md) — llmgw behind Kong Gateway, or Kong behind + llmgw: the configuration, what Kong counts, and which to choose - [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what llmgw needs of them, checking a node with `llmgw doctor`, and what to look at when a replica spanning nodes will not run diff --git a/docs/kong.md b/docs/kong.md new file mode 100644 index 0000000..a9088cc --- /dev/null +++ b/docs/kong.md @@ -0,0 +1,95 @@ +# llmgw and Kong + +For a site that already runs Kong. The two meet over plain HTTP: Kong in +front of llmgw, as one more API it governs, or Kong behind llmgw, as an +OpenAI-compatible provider. Nothing of either is built into the other. + +Tried with Kong Gateway 3.9 (`kong:3.9`, the last open-source release; 3.10 +and later need a Kong licence) in DB-less mode, in front of llmgw. + +## Kong in front of llmgw + +Two ways, both tried. + +**A plain route.** Kong forwards the request as it came. Tried with chat +completions, streamed and not; llmgw's other APIs — Anthropic's +`/v1/messages`, `/v1/responses`, embeddings, pictures — go the same way, +since Kong does not read them. Kong adds its own authentication, rate limits +and logging; llmgw still checks the key, the budget and the guardrails. + +```yaml +_format_version: "3.0" +services: +- name: llmgw + url: https://llm.example.com # llmgw's client port + routes: + - name: llmgw + paths: [/llmgw] # /llmgw/v1/chat/completions → /v1/chat/completions +``` + +**Kong's `ai-proxy` plugin**, for Kong's own token statistics on a chat +route. llmgw is an OpenAI provider whose `upstream_url` is llmgw: + +```yaml +services: +- name: llmgw-chat + url: https://llm.example.com + routes: + - name: chat + paths: [/chat] + plugins: + - name: ai-proxy + config: + route_type: llm/v1/chat + logging: {log_statistics: true, log_payloads: false} + model: + provider: openai + name: qwen-coder # a model llmgw serves + options: + upstream_url: https://llm.example.com/v1/chat/completions +``` + +**Keys.** Without an `auth` block, `ai-proxy` passes the client's +`Authorization` header on: each team keeps its own llmgw key, and llmgw's +budgets, limits, guardrails and usage stay per team. With +`auth: {header_name: Authorization, header_value: "Bearer sk-llmgw-…"}` Kong +sends one key of its own instead, and llmgw sees every caller as that key — +then budgets per team are Kong's to keep. + +**What Kong counts.** For a whole answer Kong reads llmgw's `usage` and +records prompt and completion tokens as llmgw did. For a streamed answer +Kong 3.9 records an estimate of the prompt tokens and no completion tokens, +even with `stream_options.include_usage` set — the usage chunk reaches it +and is not read. llmgw's own accounting counts both; use its usage export +(`llmgw ctl usage export`) or `/admin/usage` for figures to bill by. + +**Timeouts.** A long answer from a model still loading outlasts Kong's +60-second defaults: set the service's `read_timeout` to the time a model may +take to start, as for nginx in [install.md](install.md). + +## Kong behind llmgw + +Not tried here. Kong's AI routes speak OpenAI's API, so llmgw reaches models Kong fronts as +it reaches any provider: a model whose source is `external`. + +```yaml +- name: kong-gpt + source: external + external: + url: https://kong.example.com/openai # Kong's route, without /v1 + model: gpt-4o-mini + api_key_file: /etc/llmgw/secrets/kong-consumer.key +``` + +The key is the one Kong's own authentication asks of a consumer. As a +fallback (`fallbacks: [kong-gpt]`) it is used only when the cluster has no +room. + +## Which to choose + +Kong in front when the site's API policy says every API goes through Kong: +llmgw is then one more upstream, and its per-team accounting still holds +when each team keeps its own key. Kong behind when the site already reaches +outside providers through Kong and llmgw should too. llmgw does not need +Kong for anything it does itself — keys, budgets, guardrails, routing to +its own replicas.