Traces say model calls in the OpenTelemetry GenAI conventions

An inference request and each attempt at an engine carry gen_ai.*: the
operation, the requested and answering model, max_tokens, temperature and
top_p when the client set them, the session as the conversation id, the
input, output and cached tokens, and on an attempt the engine that answered.
A customer's Grafana, Jaeger or Langfuse then shows llmgw's calls as model
calls. No prompt or answer text is ever put in a span, and a test says so.

BACKLOG gains section L: what is worth taking from Kong's AI Gateway, of
which this is the first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-25 12:59:20 +01:00
1 parent 38dfe268c9
commit 06baa38858
7 files changed
+263 -1

No files matched your search

+12
View File
@@ -1957,6 +1957,18 @@ So "why did qwen take four minutes to start" is one trace: find `replica start`
with `llmgw.model` = qwen, and the node's spans beneath it say whether the
time went to the download, the launch or the engine loading.
An inference request and each `upstream` attempt also carry the
[OpenTelemetry GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/),
so a tool that reads them — Grafana, Jaeger, Langfuse, Arize Phoenix — shows
llmgw's calls as model calls without a dashboard made for llmgw:
`gen_ai.operation.name` (`chat`, `text_completion`, `embeddings`),
`gen_ai.request.model`, `gen_ai.request.max_tokens`, `temperature` and `top_p`
when the client set them, `gen_ai.conversation.id` (the session),
`gen_ai.response.model`, and `gen_ai.usage.input_tokens`, `output_tokens` and
`cache_read.input_tokens`. On an `upstream` span `gen_ai.provider.name` is the
engine that answered (`vllm`, `llamacpp`, `external`, …). What was asked and
what was answered are never in a span.
A node follows the control plane's decision: a start the control plane kept is
traced on the node whatever the agent's own `sample_ratio`, and one it sampled
out is not; the agent's ratio decides only for what arrives with no trace, as