Traces say model calls in the OpenTelemetry GenAI conventions
CI / test (push) Successful in 8m55s

An inference request and each attempt at an engine carry gen_ai.*: the
operation, the requested and answering model, max_tokens, temperature and
top_p when the client set them, the session as the conversation id, the
input, output and cached tokens, and on an attempt the engine that answered.
A customer's Grafana, Jaeger or Langfuse then shows llmgw's calls as model
calls. No prompt or answer text is ever put in a span, and a test says so.

BACKLOG gains section L: what is worth taking from Kong's AI Gateway, of
which this is the first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-25 12:59:20 +01:00
1 parent 38dfe268c9
commit 06baa38858
7 files changed
+263 -1

No files matched your search

+86
View File
@@ -33,6 +33,8 @@ What is left, most needed first. The detail is in the item each names.
5. **J4**: the model mirror on more than one machine, the last single point of an HA install. (M)
6. **F5**: large imports limited by the embedding engine's slots. (M)
**From Kong** (section L): L1 done; L9, then L2 and L3, then L4/L5. L6 and L7 are llmgw-80's.
**Governance.**
7. **C1**: a tenant-admin role, and forms hidden from viewers.
8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
@@ -1679,6 +1681,90 @@ default: two cores are close to it, and a busy machine passes it.
- Done when: an install on a machine too slow for its judge says so before
its first refused request.
## L. Taken from Kong's AI Gateway (written 2026-10-25)
From a look at Kong AI Gateway 3.9–3.16 and 2.x. Kong's open source stops at
3.9, and only six of its AI plugins are open (proxy, prompt template,
decorator, regex prompt guard, request and response transformer); the rest
is Enterprise or Konnect. So nothing of Kong is embedded: the two meet over
plain OpenAI/Anthropic HTTP, and these are the ideas worth taking. Not taken:
prompt compression (Kong ships it as a private LLMLingua sidecar, its benefit
unshown), prompt templates and decorators (no one has asked), Konnect's
entity model. MCP items are llmgw-80's (I2); they are listed for it.
### L1. Traces in the OpenTelemetry GenAI conventions (S) — done 2026-10-25
`gen_ai.*` on the `inference` and `upstream` spans (`internal/router/trace.go`):
operation, request and response model, max_tokens, temperature, top_p, the
session as conversation id, input/output/cached tokens, the engine as
provider. Never prompt or answer text (`TestTracesSpeakTheGenAIConventionsAndHoldNoText`).
Prometheus already counted tokens by direction, cached apart. Left: the MCP
calls' spans as `gen_ai.tool.*`, once /mcp is traced at all.
### L2. Redacted personal data put back in the answer (S)
The guardrails rewrite personal data as `[KIND_n]` (`internal/guard/gate.go`,
`placeholders`), the same value always the same placeholder. A model that
answers "Dear [PERSON_1]" leaves it so. Kong's sanitizer restores.
- Per policy, `restore: true`: the placeholders of one request kept in memory
for its life, and swapped back in the answer, streamed too (a placeholder
split across two chunks held until it is whole).
- Only into the answer to the request that held them; never stored, never in
usage rows or traces.
- Done when: "write to Ayşe Yılmaz at ayse@example.com" through an enforce
policy reaches the model as placeholders and the client reads the real name
and address back.
### L3. A guardrail that is someone else's service (S)
Customers may be required to use one: Azure Content Safety, Lakera, NeMo
Guardrails, an in-house classifier. Kong's AI Custom Guardrail is a
templated HTTP call with an expression for "block".
- A detector kind `http`: URL, headers from secret files, a body template
with the text, a JSON path that says blocked and one that says why;
timeout and `fail` as the judge's. Checked on input, output or both.
- Done when: a policy with an `http` detector against a test service blocks
what it says to, and a service that does not answer is treated as `fail`
says.
### L4. Routing by latency and by cost, and failover by priority (M)
The scorers (`internal/router/scheduler/scorers.go`) know sessions, queues
and KV cache. Kong also balances by measured latency, by tokens or cost
used, and in priority tiers.
- A `latency` scorer: a moving average of time to first token per endpoint.
- Fallbacks in tiers: local replicas first, then an outside provider, with
the tier said in the usage row.
- Done when: with two replicas of one model, one on a slower card, `latency`
sends most of the traffic to the faster and the slower still gets some.
### L5. Prices by kind and by cache (S–M)
`catalog` prices are per million input/output tokens and per picture/mesh.
Kong prices cached reads and cache writes apart, and audio, image and video
tokens apart, and limits by cost.
- Price fields for cached input and by modality; the usage row's cost from
them; a budget limit in money as well as tokens (`keys limits --cost`).
- Done when: a request with 90% cached prompt costs what the cached rate says.
### L6. REST APIs as MCP tools (M) — for llmgw-80, beside I2
Kong turns OpenAPI operations into MCP tools. A company with an internal
REST API and no MCP server could offer it to agents through `/mcp`:
`llmgw ctl mcp add hr --openapi https://hr.internal/openapi.json`, each
operation a tool, under the same grants, audit, guardrails and rate as I2.
### L7. MCP sign-in by the MCP spec (M) — for llmgw-80, part of I2 P3
Kong's MCP OAuth2: llmgw's `/mcp` as an OAuth 2.1 resource server (protected
resource metadata, tokens checked by JWKS against the company's identity
provider), and a token exchanged for a short-lived one per upstream server
rather than one long-lived secret.
### L8. A2A (S, when asked)
Agent-to-agent traffic through the gateway: agent cards' URLs rewritten to
llmgw's, calls audited and traced. The standard is young; wait for a
customer.
### L9. llmgw behind Kong, written down (S)
A customer that already runs Kong puts llmgw behind it: `ai-proxy` with
`provider: openai` and `upstream_url` at llmgw (3.9, open source), or a plain
route. A page in docs/ with the configuration, tried against Kong 3.9 in a
container, and what Kong's token accounting sees.
## D. Facts worth knowing before starting
- Admin API: `127.0.0.1:9090`, behind the token in