An inference request and each attempt at an engine carry gen_ai.*: the operation, the requested and answering model, max_tokens, temperature and top_p when the client set them, the session as the conversation id, the input, output and cached tokens, and on an attempt the engine that answered. A customer's Grafana, Jaeger or Langfuse then shows llmgw's calls as model calls. No prompt or answer text is ever put in a span, and a test says so. BACKLOG gains section L: what is worth taking from Kong's AI Gateway, of which this is the first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
38dfe268c9
commit
06baa38858
7 files changed
+263
-1
No files matched your search
+86
@@ -33,6 +33,8 @@ What is left, most needed first. The detail is in the item each names.
|
||||
5. **J4**: the model mirror on more than one machine, the last single point of an HA install. (M)
|
||||
6. **F5**: large imports limited by the embedding engine's slots. (M)
|
||||
|
||||
**From Kong** (section L): L1 done; L9, then L2 and L3, then L4/L5. L6 and L7 are llmgw-80's.
|
||||
|
||||
**Governance.**
|
||||
7. **C1**: a tenant-admin role, and forms hidden from viewers.
|
||||
8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
|
||||
@@ -1679,6 +1681,90 @@ default: two cores are close to it, and a busy machine passes it.
|
||||
- Done when: an install on a machine too slow for its judge says so before
|
||||
its first refused request.
|
||||
|
||||
## L. Taken from Kong's AI Gateway (written 2026-10-25)
|
||||
|
||||
From a look at Kong AI Gateway 3.9–3.16 and 2.x. Kong's open source stops at
|
||||
3.9, and only six of its AI plugins are open (proxy, prompt template,
|
||||
decorator, regex prompt guard, request and response transformer); the rest
|
||||
is Enterprise or Konnect. So nothing of Kong is embedded: the two meet over
|
||||
plain OpenAI/Anthropic HTTP, and these are the ideas worth taking. Not taken:
|
||||
prompt compression (Kong ships it as a private LLMLingua sidecar, its benefit
|
||||
unshown), prompt templates and decorators (no one has asked), Konnect's
|
||||
entity model. MCP items are llmgw-80's (I2); they are listed for it.
|
||||
|
||||
### L1. Traces in the OpenTelemetry GenAI conventions (S) — done 2026-10-25
|
||||
`gen_ai.*` on the `inference` and `upstream` spans (`internal/router/trace.go`):
|
||||
operation, request and response model, max_tokens, temperature, top_p, the
|
||||
session as conversation id, input/output/cached tokens, the engine as
|
||||
provider. Never prompt or answer text (`TestTracesSpeakTheGenAIConventionsAndHoldNoText`).
|
||||
Prometheus already counted tokens by direction, cached apart. Left: the MCP
|
||||
calls' spans as `gen_ai.tool.*`, once /mcp is traced at all.
|
||||
|
||||
### L2. Redacted personal data put back in the answer (S)
|
||||
The guardrails rewrite personal data as `[KIND_n]` (`internal/guard/gate.go`,
|
||||
`placeholders`), the same value always the same placeholder. A model that
|
||||
answers "Dear [PERSON_1]" leaves it so. Kong's sanitizer restores.
|
||||
- Per policy, `restore: true`: the placeholders of one request kept in memory
|
||||
for its life, and swapped back in the answer, streamed too (a placeholder
|
||||
split across two chunks held until it is whole).
|
||||
- Only into the answer to the request that held them; never stored, never in
|
||||
usage rows or traces.
|
||||
- Done when: "write to Ayşe Yılmaz at ayse@example.com" through an enforce
|
||||
policy reaches the model as placeholders and the client reads the real name
|
||||
and address back.
|
||||
|
||||
### L3. A guardrail that is someone else's service (S)
|
||||
Customers may be required to use one: Azure Content Safety, Lakera, NeMo
|
||||
Guardrails, an in-house classifier. Kong's AI Custom Guardrail is a
|
||||
templated HTTP call with an expression for "block".
|
||||
- A detector kind `http`: URL, headers from secret files, a body template
|
||||
with the text, a JSON path that says blocked and one that says why;
|
||||
timeout and `fail` as the judge's. Checked on input, output or both.
|
||||
- Done when: a policy with an `http` detector against a test service blocks
|
||||
what it says to, and a service that does not answer is treated as `fail`
|
||||
says.
|
||||
|
||||
### L4. Routing by latency and by cost, and failover by priority (M)
|
||||
The scorers (`internal/router/scheduler/scorers.go`) know sessions, queues
|
||||
and KV cache. Kong also balances by measured latency, by tokens or cost
|
||||
used, and in priority tiers.
|
||||
- A `latency` scorer: a moving average of time to first token per endpoint.
|
||||
- Fallbacks in tiers: local replicas first, then an outside provider, with
|
||||
the tier said in the usage row.
|
||||
- Done when: with two replicas of one model, one on a slower card, `latency`
|
||||
sends most of the traffic to the faster and the slower still gets some.
|
||||
|
||||
### L5. Prices by kind and by cache (S–M)
|
||||
`catalog` prices are per million input/output tokens and per picture/mesh.
|
||||
Kong prices cached reads and cache writes apart, and audio, image and video
|
||||
tokens apart, and limits by cost.
|
||||
- Price fields for cached input and by modality; the usage row's cost from
|
||||
them; a budget limit in money as well as tokens (`keys limits --cost`).
|
||||
- Done when: a request with 90% cached prompt costs what the cached rate says.
|
||||
|
||||
### L6. REST APIs as MCP tools (M) — for llmgw-80, beside I2
|
||||
Kong turns OpenAPI operations into MCP tools. A company with an internal
|
||||
REST API and no MCP server could offer it to agents through `/mcp`:
|
||||
`llmgw ctl mcp add hr --openapi https://hr.internal/openapi.json`, each
|
||||
operation a tool, under the same grants, audit, guardrails and rate as I2.
|
||||
|
||||
### L7. MCP sign-in by the MCP spec (M) — for llmgw-80, part of I2 P3
|
||||
Kong's MCP OAuth2: llmgw's `/mcp` as an OAuth 2.1 resource server (protected
|
||||
resource metadata, tokens checked by JWKS against the company's identity
|
||||
provider), and a token exchanged for a short-lived one per upstream server
|
||||
rather than one long-lived secret.
|
||||
|
||||
### L8. A2A (S, when asked)
|
||||
Agent-to-agent traffic through the gateway: agent cards' URLs rewritten to
|
||||
llmgw's, calls audited and traced. The standard is young; wait for a
|
||||
customer.
|
||||
|
||||
### L9. llmgw behind Kong, written down (S)
|
||||
A customer that already runs Kong puts llmgw behind it: `ai-proxy` with
|
||||
`provider: openai` and `upstream_url` at llmgw (3.9, open source), or a plain
|
||||
route. A page in docs/ with the configuration, tried against Kong 3.9 in a
|
||||
container, and what Kong's token accounting sees.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, behind the token in
|
||||
|
||||
Reference in new issue
Block a user