Ilker ManapandClaude Opus 5.5
0b37796067
docs/kong.md: a plain route or ai-proxy with llmgw as the upstream, the client's llmgw key passed on so per-team budgets hold, Kong as a provider behind llmgw, and Kong 3.9's streamed answers counted without completion tokens. Closes BACKLOG L9. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
96 lines
3.5 KiB
Markdown
96 lines
3.5 KiB
Markdown
# llmgw and Kong
|
|
|
|
For a site that already runs Kong. The two meet over plain HTTP: Kong in
|
|
front of llmgw, as one more API it governs, or Kong behind llmgw, as an
|
|
OpenAI-compatible provider. Nothing of either is built into the other.
|
|
|
|
Tried with Kong Gateway 3.9 (`kong:3.9`, the last open-source release; 3.10
|
|
and later need a Kong licence) in DB-less mode, in front of llmgw.
|
|
|
|
## Kong in front of llmgw
|
|
|
|
Two ways, both tried.
|
|
|
|
**A plain route.** Kong forwards the request as it came. Tried with chat
|
|
completions, streamed and not; llmgw's other APIs — Anthropic's
|
|
`/v1/messages`, `/v1/responses`, embeddings, pictures — go the same way,
|
|
since Kong does not read them. Kong adds its own authentication, rate limits
|
|
and logging; llmgw still checks the key, the budget and the guardrails.
|
|
|
|
```yaml
|
|
_format_version: "3.0"
|
|
services:
|
|
- name: llmgw
|
|
url: https://llm.example.com # llmgw's client port
|
|
routes:
|
|
- name: llmgw
|
|
paths: [/llmgw] # /llmgw/v1/chat/completions → /v1/chat/completions
|
|
```
|
|
|
|
**Kong's `ai-proxy` plugin**, for Kong's own token statistics on a chat
|
|
route. llmgw is an OpenAI provider whose `upstream_url` is llmgw:
|
|
|
|
```yaml
|
|
services:
|
|
- name: llmgw-chat
|
|
url: https://llm.example.com
|
|
routes:
|
|
- name: chat
|
|
paths: [/chat]
|
|
plugins:
|
|
- name: ai-proxy
|
|
config:
|
|
route_type: llm/v1/chat
|
|
logging: {log_statistics: true, log_payloads: false}
|
|
model:
|
|
provider: openai
|
|
name: qwen-coder # a model llmgw serves
|
|
options:
|
|
upstream_url: https://llm.example.com/v1/chat/completions
|
|
```
|
|
|
|
**Keys.** Without an `auth` block, `ai-proxy` passes the client's
|
|
`Authorization` header on: each team keeps its own llmgw key, and llmgw's
|
|
budgets, limits, guardrails and usage stay per team. With
|
|
`auth: {header_name: Authorization, header_value: "Bearer sk-llmgw-…"}` Kong
|
|
sends one key of its own instead, and llmgw sees every caller as that key —
|
|
then budgets per team are Kong's to keep.
|
|
|
|
**What Kong counts.** For a whole answer Kong reads llmgw's `usage` and
|
|
records prompt and completion tokens as llmgw did. For a streamed answer
|
|
Kong 3.9 records an estimate of the prompt tokens and no completion tokens,
|
|
even with `stream_options.include_usage` set — the usage chunk reaches it
|
|
and is not read. llmgw's own accounting counts both; use its usage export
|
|
(`llmgw ctl usage export`) or `/admin/usage` for figures to bill by.
|
|
|
|
**Timeouts.** A long answer from a model still loading outlasts Kong's
|
|
60-second defaults: set the service's `read_timeout` to the time a model may
|
|
take to start, as for nginx in [install.md](install.md).
|
|
|
|
## Kong behind llmgw
|
|
|
|
Not tried here. Kong's AI routes speak OpenAI's API, so llmgw reaches models Kong fronts as
|
|
it reaches any provider: a model whose source is `external`.
|
|
|
|
```yaml
|
|
- name: kong-gpt
|
|
source: external
|
|
external:
|
|
url: https://kong.example.com/openai # Kong's route, without /v1
|
|
model: gpt-4o-mini
|
|
api_key_file: /etc/llmgw/secrets/kong-consumer.key
|
|
```
|
|
|
|
The key is the one Kong's own authentication asks of a consumer. As a
|
|
fallback (`fallbacks: [kong-gpt]`) it is used only when the cluster has no
|
|
room.
|
|
|
|
## Which to choose
|
|
|
|
Kong in front when the site's API policy says every API goes through Kong:
|
|
llmgw is then one more upstream, and its per-team accounting still holds
|
|
when each team keeps its own key. Kong behind when the site already reaches
|
|
outside providers through Kong and llmgw should too. llmgw does not need
|
|
Kong for anything it does itself — keys, budgets, guardrails, routing to
|
|
its own replicas.
|