# llmgw and Kong

For a site that already runs Kong. The two meet over plain HTTP: Kong in
front of llmgw, as one more API it governs, or Kong behind llmgw, as an
OpenAI-compatible provider. Nothing of either is built into the other.

Tried with Kong Gateway 3.9 (`kong:3.9`, the last open-source release; 3.10
and later need a Kong licence) in DB-less mode, in front of llmgw.

## Kong in front of llmgw

Two ways, both tried.

**A plain route.** Kong forwards the request as it came. Tried with chat
completions, streamed and not; llmgw's other APIs — Anthropic's
`/v1/messages`, `/v1/responses`, embeddings, pictures — go the same way,
since Kong does not read them. Kong adds its own authentication, rate limits
and logging; llmgw still checks the key, the budget and the guardrails.

```yaml
_format_version: "3.0"
services:
- name: llmgw
  url: https://llm.example.com          # llmgw's client port
  routes:
  - name: llmgw
    paths: [/llmgw]                      # /llmgw/v1/chat/completions → /v1/chat/completions
```

**Kong's `ai-proxy` plugin**, for Kong's own token statistics on a chat
route. llmgw is an OpenAI provider whose `upstream_url` is llmgw:

```yaml
services:
- name: llmgw-chat
  url: https://llm.example.com
  routes:
  - name: chat
    paths: [/chat]
  plugins:
  - name: ai-proxy
    config:
      route_type: llm/v1/chat
      logging: {log_statistics: true, log_payloads: false}
      model:
        provider: openai
        name: qwen-coder                 # a model llmgw serves
        options:
          upstream_url: https://llm.example.com/v1/chat/completions
```

**Keys.** Without an `auth` block, `ai-proxy` passes the client's
`Authorization` header on: each team keeps its own llmgw key, and llmgw's
budgets, limits, guardrails and usage stay per team. With
`auth: {header_name: Authorization, header_value: "Bearer sk-llmgw-…"}` Kong
sends one key of its own instead, and llmgw sees every caller as that key —
then budgets per team are Kong's to keep.

**What Kong counts.** For a whole answer Kong reads llmgw's `usage` and
records prompt and completion tokens as llmgw did. For a streamed answer
Kong 3.9 records an estimate of the prompt tokens and no completion tokens,
even with `stream_options.include_usage` set — the usage chunk reaches it
and is not read. llmgw's own accounting counts both; use its usage export
(`llmgw ctl usage export`) or `/admin/usage` for figures to bill by.

**Timeouts.** A long answer from a model still loading outlasts Kong's
60-second defaults: set the service's `read_timeout` to the time a model may
take to start, as for nginx in [install.md](install.md).

## Kong behind llmgw

Not tried here. Kong's AI routes speak OpenAI's API, so llmgw reaches models Kong fronts as
it reaches any provider: a model whose source is `external`.

```yaml
- name: kong-gpt
  source: external
  external:
    url: https://kong.example.com/openai     # Kong's route, without /v1
    model: gpt-4o-mini
    api_key_file: /etc/llmgw/secrets/kong-consumer.key
```

The key is the one Kong's own authentication asks of a consumer. As a
fallback (`fallbacks: [kong-gpt]`) it is used only when the cluster has no
room.

## Which to choose

Kong in front when the site's API policy says every API goes through Kong:
llmgw is then one more upstream, and its per-team accounting still holds
when each team keeps its own key. Kong behind when the site already reaches
outside providers through Kong and llmgw should too. llmgw does not need
Kong for anything it does itself — keys, budgets, guardrails, routing to
its own replicas.