llmgw and Kong, written down after trying Kong 3.9 in front
docs/kong.md: a plain route or ai-proxy with llmgw as the upstream, the client's llmgw key passed on so per-team budgets hold, Kong as a provider behind llmgw, and Kong 3.9's streamed answers counted without completion tokens. Closes BACKLOG L9. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
9e7de57816
commit
0b37796067
4 files changed
+104
-1
No files matched your search
@@ -0,0 +1,95 @@
|
||||
# llmgw and Kong
|
||||
|
||||
For a site that already runs Kong. The two meet over plain HTTP: Kong in
|
||||
front of llmgw, as one more API it governs, or Kong behind llmgw, as an
|
||||
OpenAI-compatible provider. Nothing of either is built into the other.
|
||||
|
||||
Tried with Kong Gateway 3.9 (`kong:3.9`, the last open-source release; 3.10
|
||||
and later need a Kong licence) in DB-less mode, in front of llmgw.
|
||||
|
||||
## Kong in front of llmgw
|
||||
|
||||
Two ways, both tried.
|
||||
|
||||
**A plain route.** Kong forwards the request as it came. Tried with chat
|
||||
completions, streamed and not; llmgw's other APIs — Anthropic's
|
||||
`/v1/messages`, `/v1/responses`, embeddings, pictures — go the same way,
|
||||
since Kong does not read them. Kong adds its own authentication, rate limits
|
||||
and logging; llmgw still checks the key, the budget and the guardrails.
|
||||
|
||||
```yaml
|
||||
_format_version: "3.0"
|
||||
services:
|
||||
- name: llmgw
|
||||
url: https://llm.example.com # llmgw's client port
|
||||
routes:
|
||||
- name: llmgw
|
||||
paths: [/llmgw] # /llmgw/v1/chat/completions → /v1/chat/completions
|
||||
```
|
||||
|
||||
**Kong's `ai-proxy` plugin**, for Kong's own token statistics on a chat
|
||||
route. llmgw is an OpenAI provider whose `upstream_url` is llmgw:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
- name: llmgw-chat
|
||||
url: https://llm.example.com
|
||||
routes:
|
||||
- name: chat
|
||||
paths: [/chat]
|
||||
plugins:
|
||||
- name: ai-proxy
|
||||
config:
|
||||
route_type: llm/v1/chat
|
||||
logging: {log_statistics: true, log_payloads: false}
|
||||
model:
|
||||
provider: openai
|
||||
name: qwen-coder # a model llmgw serves
|
||||
options:
|
||||
upstream_url: https://llm.example.com/v1/chat/completions
|
||||
```
|
||||
|
||||
**Keys.** Without an `auth` block, `ai-proxy` passes the client's
|
||||
`Authorization` header on: each team keeps its own llmgw key, and llmgw's
|
||||
budgets, limits, guardrails and usage stay per team. With
|
||||
`auth: {header_name: Authorization, header_value: "Bearer sk-llmgw-…"}` Kong
|
||||
sends one key of its own instead, and llmgw sees every caller as that key —
|
||||
then budgets per team are Kong's to keep.
|
||||
|
||||
**What Kong counts.** For a whole answer Kong reads llmgw's `usage` and
|
||||
records prompt and completion tokens as llmgw did. For a streamed answer
|
||||
Kong 3.9 records an estimate of the prompt tokens and no completion tokens,
|
||||
even with `stream_options.include_usage` set — the usage chunk reaches it
|
||||
and is not read. llmgw's own accounting counts both; use its usage export
|
||||
(`llmgw ctl usage export`) or `/admin/usage` for figures to bill by.
|
||||
|
||||
**Timeouts.** A long answer from a model still loading outlasts Kong's
|
||||
60-second defaults: set the service's `read_timeout` to the time a model may
|
||||
take to start, as for nginx in [install.md](install.md).
|
||||
|
||||
## Kong behind llmgw
|
||||
|
||||
Not tried here. Kong's AI routes speak OpenAI's API, so llmgw reaches models Kong fronts as
|
||||
it reaches any provider: a model whose source is `external`.
|
||||
|
||||
```yaml
|
||||
- name: kong-gpt
|
||||
source: external
|
||||
external:
|
||||
url: https://kong.example.com/openai # Kong's route, without /v1
|
||||
model: gpt-4o-mini
|
||||
api_key_file: /etc/llmgw/secrets/kong-consumer.key
|
||||
```
|
||||
|
||||
The key is the one Kong's own authentication asks of a consumer. As a
|
||||
fallback (`fallbacks: [kong-gpt]`) it is used only when the cluster has no
|
||||
room.
|
||||
|
||||
## Which to choose
|
||||
|
||||
Kong in front when the site's API policy says every API goes through Kong:
|
||||
llmgw is then one more upstream, and its per-team accounting still holds
|
||||
when each team keeps its own key. Kong behind when the site already reaches
|
||||
outside providers through Kong and llmgw should too. llmgw does not need
|
||||
Kong for anything it does itself — keys, budgets, guardrails, routing to
|
||||
its own replicas.
|
||||
Reference in new issue
Block a user