llmgw/docs/kong.md

llmgw and Kong

For a site that already runs Kong. The two meet over plain HTTP: Kong in front of llmgw, as one more API it governs, or Kong behind llmgw, as an OpenAI-compatible provider. Nothing of either is built into the other.

Tried with Kong Gateway 3.9 (kong:3.9, the last open-source release; 3.10 and later need a Kong licence) in DB-less mode, in front of llmgw.

Kong in front of llmgw

Two ways, both tried.

A plain route. Kong forwards the request as it came. Tried with chat completions, streamed and not; llmgw's other APIs — Anthropic's /v1/messages, /v1/responses, embeddings, pictures — go the same way, since Kong does not read them. Kong adds its own authentication, rate limits and logging; llmgw still checks the key, the budget and the guardrails.

_format_version: "3.0"
services:
- name: llmgw
  url: https://llm.example.com          # llmgw's client port
  routes:
  - name: llmgw
    paths: [/llmgw]                      # /llmgw/v1/chat/completions → /v1/chat/completions

Kong's ai-proxy plugin, for Kong's own token statistics on a chat route. llmgw is an OpenAI provider whose upstream_url is llmgw:

services:
- name: llmgw-chat
  url: https://llm.example.com
  routes:
  - name: chat
    paths: [/chat]
  plugins:
  - name: ai-proxy
    config:
      route_type: llm/v1/chat
      logging: {log_statistics: true, log_payloads: false}
      model:
        provider: openai
        name: qwen-coder                 # a model llmgw serves
        options:
          upstream_url: https://llm.example.com/v1/chat/completions

Keys. Without an auth block, ai-proxy passes the client's Authorization header on: each team keeps its own llmgw key, and llmgw's budgets, limits, guardrails and usage stay per team. With auth: {header_name: Authorization, header_value: "Bearer sk-llmgw-…"} Kong sends one key of its own instead, and llmgw sees every caller as that key — then budgets per team are Kong's to keep.

What Kong counts. For a whole answer Kong reads llmgw's usage and records prompt and completion tokens as llmgw did. For a streamed answer Kong 3.9 records an estimate of the prompt tokens and no completion tokens, even with stream_options.include_usage set — the usage chunk reaches it and is not read. llmgw's own accounting counts both; use its usage export (llmgw ctl usage export) or /admin/usage for figures to bill by.

Timeouts. A long answer from a model still loading outlasts Kong's 60-second defaults: set the service's read_timeout to the time a model may take to start, as for nginx in install.md.

Kong behind llmgw

Not tried here. Kong's AI routes speak OpenAI's API, so llmgw reaches models Kong fronts as it reaches any provider: a model whose source is external.

- name: kong-gpt
  source: external
  external:
    url: https://kong.example.com/openai     # Kong's route, without /v1
    model: gpt-4o-mini
    api_key_file: /etc/llmgw/secrets/kong-consumer.key

The key is the one Kong's own authentication asks of a consumer. As a fallback (fallbacks: [kong-gpt]) it is used only when the cluster has no room.

Which to choose

Kong in front when the site's API policy says every API goes through Kong: llmgw is then one more upstream, and its per-team accounting still holds when each team keeps its own key. Kong behind when the site already reaches outside providers through Kong and llmgw should too. llmgw does not need Kong for anything it does itself — keys, budgets, guardrails, routing to its own replicas.