An inference request and each attempt at an engine carry gen_ai.*: the operation, the requested and answering model, max_tokens, temperature and top_p when the client set them, the session as the conversation id, the input, output and cached tokens, and on an attempt the engine that answered. A customer's Grafana, Jaeger or Langfuse then shows llmgw's calls as model calls. No prompt or answer text is ever put in a span, and a test says so. BACKLOG gains section L: what is worth taking from Kong's AI Gateway, of which this is the first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
38dfe268c9
commit
06baa38858
7 files changed
+263
-1
No files matched your search
@@ -1957,6 +1957,18 @@ So "why did qwen take four minutes to start" is one trace: find `replica start`
|
||||
with `llmgw.model` = qwen, and the node's spans beneath it say whether the
|
||||
time went to the download, the launch or the engine loading.
|
||||
|
||||
An inference request and each `upstream` attempt also carry the
|
||||
[OpenTelemetry GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/),
|
||||
so a tool that reads them — Grafana, Jaeger, Langfuse, Arize Phoenix — shows
|
||||
llmgw's calls as model calls without a dashboard made for llmgw:
|
||||
`gen_ai.operation.name` (`chat`, `text_completion`, `embeddings`),
|
||||
`gen_ai.request.model`, `gen_ai.request.max_tokens`, `temperature` and `top_p`
|
||||
when the client set them, `gen_ai.conversation.id` (the session),
|
||||
`gen_ai.response.model`, and `gen_ai.usage.input_tokens`, `output_tokens` and
|
||||
`cache_read.input_tokens`. On an `upstream` span `gen_ai.provider.name` is the
|
||||
engine that answered (`vllm`, `llamacpp`, `external`, …). What was asked and
|
||||
what was answered are never in a span.
|
||||
|
||||
A node follows the control plane's decision: a start the control plane kept is
|
||||
traced on the node whatever the agent's own `sample_ratio`, and one it sampled
|
||||
out is not; the agent's ratio decides only for what arrives with no trace, as
|
||||
|
||||
Reference in new issue
Block a user