Traces say model calls in the OpenTelemetry GenAI conventions

An inference request and each attempt at an engine carry gen_ai.*: the
operation, the requested and answering model, max_tokens, temperature and
top_p when the client set them, the session as the conversation id, the
input, output and cached tokens, and on an attempt the engine that answered.
A customer's Grafana, Jaeger or Langfuse then shows llmgw's calls as model
calls. No prompt or answer text is ever put in a span, and a test says so.

BACKLOG gains section L: what is worth taking from Kong's AI Gateway, of
which this is the first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

BACKLOG.md

@@ -33,6 +33,8 @@ What is left, most needed first. The detail is in the item each names.
 5. **J4**: the model mirror on more than one machine, the last single point of an HA install. (M)
 6. **F5**: large imports limited by the embedding engine's slots. (M)
 
+**From Kong** (section L): L1 done; L9, then L2 and L3, then L4/L5. L6 and L7 are llmgw-80's.
+
 **Governance.**
 7. **C1**: a tenant-admin role, and forms hidden from viewers.
 8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too.
@@ -1679,6 +1681,90 @@ default: two cores are close to it, and a busy machine passes it.
 - Done when: an install on a machine too slow for its judge says so before
   its first refused request.
 
+## L. Taken from Kong's AI Gateway (written 2026-10-25)
+
+From a look at Kong AI Gateway 3.9–3.16 and 2.x. Kong's open source stops at
+3.9, and only six of its AI plugins are open (proxy, prompt template,
+decorator, regex prompt guard, request and response transformer); the rest
+is Enterprise or Konnect. So nothing of Kong is embedded: the two meet over
+plain OpenAI/Anthropic HTTP, and these are the ideas worth taking. Not taken:
+prompt compression (Kong ships it as a private LLMLingua sidecar, its benefit
+unshown), prompt templates and decorators (no one has asked), Konnect's
+entity model. MCP items are llmgw-80's (I2); they are listed for it.
+
+### L1. Traces in the OpenTelemetry GenAI conventions (S) — done 2026-10-25
+`gen_ai.*` on the `inference` and `upstream` spans (`internal/router/trace.go`):
+operation, request and response model, max_tokens, temperature, top_p, the
+session as conversation id, input/output/cached tokens, the engine as
+provider. Never prompt or answer text (`TestTracesSpeakTheGenAIConventionsAndHoldNoText`).
+Prometheus already counted tokens by direction, cached apart. Left: the MCP
+calls' spans as `gen_ai.tool.*`, once /mcp is traced at all.
+
+### L2. Redacted personal data put back in the answer (S)
+The guardrails rewrite personal data as `[KIND_n]` (`internal/guard/gate.go`,
+`placeholders`), the same value always the same placeholder. A model that
+answers "Dear [PERSON_1]" leaves it so. Kong's sanitizer restores.
+- Per policy, `restore: true`: the placeholders of one request kept in memory
+  for its life, and swapped back in the answer, streamed too (a placeholder
+  split across two chunks held until it is whole).
+- Only into the answer to the request that held them; never stored, never in
+  usage rows or traces.
+- Done when: "write to Ayşe Yılmaz at ayse@example.com" through an enforce
+  policy reaches the model as placeholders and the client reads the real name
+  and address back.
+
+### L3. A guardrail that is someone else's service (S)
+Customers may be required to use one: Azure Content Safety, Lakera, NeMo
+Guardrails, an in-house classifier. Kong's AI Custom Guardrail is a
+templated HTTP call with an expression for "block".
+- A detector kind `http`: URL, headers from secret files, a body template
+  with the text, a JSON path that says blocked and one that says why;
+  timeout and `fail` as the judge's. Checked on input, output or both.
+- Done when: a policy with an `http` detector against a test service blocks
+  what it says to, and a service that does not answer is treated as `fail`
+  says.
+
+### L4. Routing by latency and by cost, and failover by priority (M)
+The scorers (`internal/router/scheduler/scorers.go`) know sessions, queues
+and KV cache. Kong also balances by measured latency, by tokens or cost
+used, and in priority tiers.
+- A `latency` scorer: a moving average of time to first token per endpoint.
+- Fallbacks in tiers: local replicas first, then an outside provider, with
+  the tier said in the usage row.
+- Done when: with two replicas of one model, one on a slower card, `latency`
+  sends most of the traffic to the faster and the slower still gets some.
+
+### L5. Prices by kind and by cache (S–M)
+`catalog` prices are per million input/output tokens and per picture/mesh.
+Kong prices cached reads and cache writes apart, and audio, image and video
+tokens apart, and limits by cost.
+- Price fields for cached input and by modality; the usage row's cost from
+  them; a budget limit in money as well as tokens (`keys limits --cost`).
+- Done when: a request with 90% cached prompt costs what the cached rate says.
+
+### L6. REST APIs as MCP tools (M) — for llmgw-80, beside I2
+Kong turns OpenAPI operations into MCP tools. A company with an internal
+REST API and no MCP server could offer it to agents through `/mcp`:
+`llmgw ctl mcp add hr --openapi https://hr.internal/openapi.json`, each
+operation a tool, under the same grants, audit, guardrails and rate as I2.
+
+### L7. MCP sign-in by the MCP spec (M) — for llmgw-80, part of I2 P3
+Kong's MCP OAuth2: llmgw's `/mcp` as an OAuth 2.1 resource server (protected
+resource metadata, tokens checked by JWKS against the company's identity
+provider), and a token exchanged for a short-lived one per upstream server
+rather than one long-lived secret.
+
+### L8. A2A (S, when asked)
+Agent-to-agent traffic through the gateway: agent cards' URLs rewritten to
+llmgw's, calls audited and traced. The standard is young; wait for a
+customer.
+
+### L9. llmgw behind Kong, written down (S)
+A customer that already runs Kong puts llmgw behind it: `ai-proxy` with
+`provider: openai` and `upstream_url` at llmgw (3.9, open source), or a plain
+route. A page in docs/ with the configuration, tried against Kong 3.9 in a
+container, and what Kong's token accounting sees.
+
 ## D. Facts worth knowing before starting
 
 - Admin API: `127.0.0.1:9090`, behind the token in

CHANGELOG.md

@@ -28,6 +28,13 @@ APIs change between minors without a deprecation period.
 
 ### Added
 
+- **Traces in the OpenTelemetry GenAI conventions.** An inference request
+  and each attempt at an engine carry `gen_ai.operation.name`, the requested
+  and the answering model, `max_tokens`, `temperature` and `top_p` when set,
+  the session as `gen_ai.conversation.id`, and the input, output and cached
+  tokens, and an attempt the engine that answered as `gen_ai.provider.name`.
+  Tools that read the conventions show llmgw's calls as model calls with no
+  dashboard of their own. No prompt or answer text is ever in a span.
 - **llmgw for VS Code.** An extension of llmgw's own, served by every gateway
   at `/setup/vscode/llmgw.vsix` with that gateway's address in it, made from
   the binary itself: plain JavaScript, no dependencies, no build, and so no

docs/install.md

@@ -1957,6 +1957,18 @@ So "why did qwen take four minutes to start" is one trace: find `replica start`
 with `llmgw.model` = qwen, and the node's spans beneath it say whether the
 time went to the download, the launch or the engine loading.
 
+An inference request and each `upstream` attempt also carry the
+[OpenTelemetry GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/),
+so a tool that reads them — Grafana, Jaeger, Langfuse, Arize Phoenix — shows
+llmgw's calls as model calls without a dashboard made for llmgw:
+`gen_ai.operation.name` (`chat`, `text_completion`, `embeddings`),
+`gen_ai.request.model`, `gen_ai.request.max_tokens`, `temperature` and `top_p`
+when the client set them, `gen_ai.conversation.id` (the session),
+`gen_ai.response.model`, and `gen_ai.usage.input_tokens`, `output_tokens` and
+`cache_read.input_tokens`. On an `upstream` span `gen_ai.provider.name` is the
+engine that answered (`vllm`, `llamacpp`, `external`, …). What was asked and
+what was answered are never in a span.
+
 A node follows the control plane's decision: a start the control plane kept is
 traced on the node whatever the agent's own `sample_ratio`, and one it sampled
 out is not; the agent's ratio decides only for what arrives with no trace, as

internal/router/handlers.go

@@ -450,6 +450,8 @@ type parsedRequest struct {
 	// longest one is what a context must hold.
 	ContextTokens int
 	MaxTokens     int
+	// Temperature and TopP are the client's, when it set them: for the trace.
+	Temperature, TopP *float64
 
 	// ThreadID is the conversation this request continues, empty for the
 	// stateless path every request took before threads existed.
@@ -505,7 +507,10 @@ type inferenceBody struct {
 	Input               json.RawMessage `json:"input"`
 	MaxTokens           int             `json:"max_tokens"`
 	MaxCompletionTokens int             `json:"max_completion_tokens"`
-	StreamOptions       *struct {
+	// Read only to say them in the request's trace; sent on untouched.
+	Temperature   *float64 `json:"temperature"`
+	TopP          *float64 `json:"top_p"`
+	StreamOptions *struct {
 		IncludeUsage bool `json:"include_usage"`
 	} `json:"stream_options"`
 }
@@ -524,6 +529,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		parsed, err := r.parse(req, path)
 		if err == nil {
 			span.SetAttributes(attribute.String("llmgw.model", parsed.Model))
+			span.SetAttributes(genAIRequest(path, parsed)...)
 			parsed.RequestID = rid
 			if parsed.Header == nil {
 				parsed.Header = http.Header{}
@@ -688,6 +694,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
 		span.SetAttributes(attribute.Int("http.response.status_code", status),
 			attribute.String("llmgw.served_model", res.model),
 			attribute.String("llmgw.endpoint", res.endpointID))
+		span.SetAttributes(genAIUsage(path, res.model, res.outcome.Usage)...)
 		if status >= 500 {
 			span.SetStatus(codes.Error, http.StatusText(status))
 		}
@@ -891,6 +898,8 @@ func (r *Router) parse(req *http.Request, path string) (*parsedRequest, error) {
 		Band:            req.Header.Get(r.cfg.Admission.BandHeader),
 		ProfileOverride: req.Header.Get("x-llmgw-profile"),
 		MaxTokens:       max(b.MaxTokens, b.MaxCompletionTokens),
+		Temperature:     b.Temperature,
+		TopP:            b.TopP,
 		PromptPrefix:    promptPrefix(&b),
 		ThreadID:        req.Header.Get(threadHeader),
 	}

internal/router/router.go

@@ -1183,6 +1183,11 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 		uctx, uspan := tracer().Start(ctx, "upstream", trace.WithSpanKind(trace.SpanKindClient),
 			trace.WithAttributes(attribute.String("llmgw.endpoint", dec.Endpoint.ID),
 				attribute.String("url.full", dec.Endpoint.URL+req.Path)))
+		if op := genAIOperation(req.Path); op != "" {
+			uspan.SetAttributes(attribute.String("gen_ai.operation.name", op),
+				attribute.String("gen_ai.provider.name", genAIProvider(dec.Endpoint.Engine)),
+				attribute.String("gen_ai.request.model", model))
+		}
 		body := req.UpstreamBody
 		// A shared ComfyUI replica is told which of its models' checkpoint;
 		// any other endpoint its own upstream name.
@@ -1256,6 +1261,7 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
 		}
 		releaseAdapter()
 		uspan.SetAttributes(attribute.Int("http.response.status_code", out.Status))
+		uspan.SetAttributes(genAIUsage(req.Path, model, out.Usage)...)
 		endSpan(uspan, perr)
 		adapterGone := onDemand && !out.Wrote && out.Status == http.StatusNotFound
 		if adapterGone {

internal/router/trace.go

@@ -3,12 +3,15 @@ package router
 import (
 	"context"
 	"net/http"
+	"strings"
 
 	"go.opentelemetry.io/otel"
 	"go.opentelemetry.io/otel/attribute"
 	"go.opentelemetry.io/otel/codes"
 	"go.opentelemetry.io/otel/propagation"
 	"go.opentelemetry.io/otel/trace"
+
+	"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
 )
 
 // tracer is looked up per use rather than kept, so a provider installed after
@@ -49,3 +52,82 @@ func withTraceparent(ctx context.Context, h http.Header) http.Header {
 	otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(out))
 	return out
 }
+
+// The OpenTelemetry GenAI semantic conventions: the names a trace backend
+// reads an LLM call by, so llmgw's spans show in a customer's own tools —
+// Grafana, Jaeger, Langfuse, anything that speaks OpenTelemetry — as model
+// calls with their model and tokens, without a dashboard made for llmgw.
+//
+// What a request said and what the model answered are not among them, and
+// are never added: llmgw keeps no prompt text anywhere, and a trace is the
+// last place one should leak into.
+
+// genAIOperation is the conventions' operation for an inference path, or ""
+// for one they have no name for. The Anthropic Messages and Responses APIs
+// are answered as chat completions, and so are chat.
+func genAIOperation(path string) string {
+	switch {
+	case strings.HasSuffix(path, "/chat/completions"):
+		return "chat"
+	case strings.HasSuffix(path, "/completions"):
+		return "text_completion"
+	case strings.HasSuffix(path, "/embeddings"):
+		return "embeddings"
+	}
+	return ""
+}
+
+// genAIRequest is what a request asked for, as the conventions name it.
+func genAIRequest(path string, p *parsedRequest) []attribute.KeyValue {
+	op := genAIOperation(path)
+	if op == "" {
+		return nil
+	}
+	kv := []attribute.KeyValue{
+		attribute.String("gen_ai.operation.name", op),
+		attribute.String("gen_ai.request.model", p.Model),
+	}
+	if p.MaxTokens > 0 {
+		kv = append(kv, attribute.Int("gen_ai.request.max_tokens", p.MaxTokens))
+	}
+	if p.Temperature != nil {
+		kv = append(kv, attribute.Float64("gen_ai.request.temperature", *p.Temperature))
+	}
+	if p.TopP != nil {
+		kv = append(kv, attribute.Float64("gen_ai.request.top_p", *p.TopP))
+	}
+	if p.SessionID != "" {
+		kv = append(kv, attribute.String("gen_ai.conversation.id", p.SessionID))
+	}
+	return kv
+}
+
+// genAIUsage is what an answer cost, in the conventions' names: the model
+// that answered, and its tokens, cached ones apart.
+func genAIUsage(path, served string, u *proxy.Usage) []attribute.KeyValue {
+	if genAIOperation(path) == "" {
+		return nil
+	}
+	var kv []attribute.KeyValue
+	if served != "" {
+		kv = append(kv, attribute.String("gen_ai.response.model", served))
+	}
+	if u != nil {
+		kv = append(kv,
+			attribute.Int("gen_ai.usage.input_tokens", u.PromptTokens),
+			attribute.Int("gen_ai.usage.output_tokens", u.CompletionTokens))
+		if u.CachedTokens > 0 {
+			kv = append(kv, attribute.Int("gen_ai.usage.cache_read.input_tokens", u.CachedTokens))
+		}
+	}
+	return kv
+}
+
+// genAIProvider names who answered an attempt: the engine serving a replica
+// (vllm, llamacpp, …), or the outside provider's, for an external model.
+func genAIProvider(engine string) string {
+	if engine == "" {
+		return "llmgw"
+	}
+	return engine
+}

internal/router/trace_test.go

@@ -1,6 +1,7 @@
 package router
 
 import (
+	"strings"
 	"testing"
 
 	"go.opentelemetry.io/otel"
@@ -81,3 +82,62 @@ func TestWithoutTracingTheEngineGetsNoTraceparent(t *testing.T) {
 		t.Errorf("traceparent %q sent with tracing off", tp)
 	}
 }
+
+func TestTracesSpeakTheGenAIConventionsAndHoldNoText(t *testing.T) {
+	rec := tracetest.NewSpanRecorder()
+	prevTP := otel.GetTracerProvider()
+	otel.SetTracerProvider(sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(rec)))
+	t.Cleanup(func() { otel.SetTracerProvider(prevTP) })
+
+	e := newFakeEngine(t, "m")
+	rt := newTestRouter(t, baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}}))
+	res := post(t, rt.Handler(), map[string]any{"model": "m", "max_tokens": 64, "temperature": 0.2, "top_p": 0.9,
+		"messages": []any{map[string]any{"role": "user", "content": "the secret word is aubergine"}}}, nil)
+	if res.Code != 200 {
+		t.Fatalf("status %d: %s", res.Code, res.Body.String())
+	}
+	got := map[string]map[string]string{}
+	for _, s := range rec.Ended() {
+		attrs := map[string]string{}
+		for _, a := range s.Attributes() {
+			attrs[string(a.Key)] = a.Value.Emit()
+			// Neither the prompt nor the reply ("hi"), in any attribute.
+			if v := a.Value.Emit(); strings.Contains(v, "aubergine") || v == "hi" {
+				t.Errorf("span %s carries text in %s", s.Name(), a.Key)
+			}
+		}
+		got[s.Name()] = attrs
+	}
+	root := got["inference /v1/chat/completions"]
+	for k, want := range map[string]string{
+		"gen_ai.operation.name":      "chat",
+		"gen_ai.request.model":       "m",
+		"gen_ai.request.max_tokens":  "64",
+		"gen_ai.request.temperature": "0.2",
+		"gen_ai.request.top_p":       "0.9",
+		"gen_ai.response.model":      "m",
+		"gen_ai.usage.input_tokens":  "4",
+		"gen_ai.usage.output_tokens": "2",
+	} {
+		if root[k] != want {
+			t.Errorf("request span %s = %q, want %q", k, root[k], want)
+		}
+	}
+	up := got["upstream"]
+	if up["gen_ai.operation.name"] != "chat" || up["gen_ai.provider.name"] == "" || up["gen_ai.usage.output_tokens"] != "2" {
+		t.Errorf("upstream span %v", up)
+	}
+}
+
+func TestEveryInferencePathHasItsGenAIOperation(t *testing.T) {
+	for path, want := range map[string]string{
+		"/v1/chat/completions": "chat",
+		"/v1/completions":      "text_completion",
+		"/v1/embeddings":       "embeddings",
+		"/v1/rerank":           "",
+	} {
+		if got := genAIOperation(path); got != want {
+			t.Errorf("%s: %q, want %q", path, got, want)
+		}
+	}
+}