From 06baa38858e46e6db1b1fc3a16de58324267caac Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Sun, 25 Oct 2026 12:59:20 +0100 Subject: [PATCH] Traces say model calls in the OpenTelemetry GenAI conventions An inference request and each attempt at an engine carry gen_ai.*: the operation, the requested and answering model, max_tokens, temperature and top_p when the client set them, the session as the conversation id, the input, output and cached tokens, and on an attempt the engine that answered. A customer's Grafana, Jaeger or Langfuse then shows llmgw's calls as model calls. No prompt or answer text is ever put in a span, and a test says so. BACKLOG gains section L: what is worth taking from Kong's AI Gateway, of which this is the first. Co-Authored-By: Claude Opus 5.5 --- BACKLOG.md | 86 +++++++++++++++++++++++++++++++++++ CHANGELOG.md | 7 +++ docs/install.md | 12 +++++ internal/router/handlers.go | 11 ++++- internal/router/router.go | 6 +++ internal/router/trace.go | 82 +++++++++++++++++++++++++++++++++ internal/router/trace_test.go | 60 ++++++++++++++++++++++++ 7 files changed, 263 insertions(+), 1 deletion(-) diff --git a/BACKLOG.md b/BACKLOG.md index 0554f71..0832eae 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -33,6 +33,8 @@ What is left, most needed first. The detail is in the item each names. 5. **J4**: the model mirror on more than one machine, the last single point of an HA install. (M) 6. **F5**: large imports limited by the embedding engine's slots. (M) +**From Kong** (section L): L1 done; L9, then L2 and L3, then L4/L5. L6 and L7 are llmgw-80's. + **Governance.** 7. **C1**: a tenant-admin role, and forms hidden from viewers. 8. **Audit before and after**: what a change changed, and changes made with the CLI recorded too. @@ -1679,6 +1681,90 @@ default: two cores are close to it, and a busy machine passes it. - Done when: an install on a machine too slow for its judge says so before its first refused request. +## L. Taken from Kong's AI Gateway (written 2026-10-25) + +From a look at Kong AI Gateway 3.9–3.16 and 2.x. Kong's open source stops at +3.9, and only six of its AI plugins are open (proxy, prompt template, +decorator, regex prompt guard, request and response transformer); the rest +is Enterprise or Konnect. So nothing of Kong is embedded: the two meet over +plain OpenAI/Anthropic HTTP, and these are the ideas worth taking. Not taken: +prompt compression (Kong ships it as a private LLMLingua sidecar, its benefit +unshown), prompt templates and decorators (no one has asked), Konnect's +entity model. MCP items are llmgw-80's (I2); they are listed for it. + +### L1. Traces in the OpenTelemetry GenAI conventions (S) — done 2026-10-25 +`gen_ai.*` on the `inference` and `upstream` spans (`internal/router/trace.go`): +operation, request and response model, max_tokens, temperature, top_p, the +session as conversation id, input/output/cached tokens, the engine as +provider. Never prompt or answer text (`TestTracesSpeakTheGenAIConventionsAndHoldNoText`). +Prometheus already counted tokens by direction, cached apart. Left: the MCP +calls' spans as `gen_ai.tool.*`, once /mcp is traced at all. + +### L2. Redacted personal data put back in the answer (S) +The guardrails rewrite personal data as `[KIND_n]` (`internal/guard/gate.go`, +`placeholders`), the same value always the same placeholder. A model that +answers "Dear [PERSON_1]" leaves it so. Kong's sanitizer restores. +- Per policy, `restore: true`: the placeholders of one request kept in memory + for its life, and swapped back in the answer, streamed too (a placeholder + split across two chunks held until it is whole). +- Only into the answer to the request that held them; never stored, never in + usage rows or traces. +- Done when: "write to Ayşe Yılmaz at ayse@example.com" through an enforce + policy reaches the model as placeholders and the client reads the real name + and address back. + +### L3. A guardrail that is someone else's service (S) +Customers may be required to use one: Azure Content Safety, Lakera, NeMo +Guardrails, an in-house classifier. Kong's AI Custom Guardrail is a +templated HTTP call with an expression for "block". +- A detector kind `http`: URL, headers from secret files, a body template + with the text, a JSON path that says blocked and one that says why; + timeout and `fail` as the judge's. Checked on input, output or both. +- Done when: a policy with an `http` detector against a test service blocks + what it says to, and a service that does not answer is treated as `fail` + says. + +### L4. Routing by latency and by cost, and failover by priority (M) +The scorers (`internal/router/scheduler/scorers.go`) know sessions, queues +and KV cache. Kong also balances by measured latency, by tokens or cost +used, and in priority tiers. +- A `latency` scorer: a moving average of time to first token per endpoint. +- Fallbacks in tiers: local replicas first, then an outside provider, with + the tier said in the usage row. +- Done when: with two replicas of one model, one on a slower card, `latency` + sends most of the traffic to the faster and the slower still gets some. + +### L5. Prices by kind and by cache (S–M) +`catalog` prices are per million input/output tokens and per picture/mesh. +Kong prices cached reads and cache writes apart, and audio, image and video +tokens apart, and limits by cost. +- Price fields for cached input and by modality; the usage row's cost from + them; a budget limit in money as well as tokens (`keys limits --cost`). +- Done when: a request with 90% cached prompt costs what the cached rate says. + +### L6. REST APIs as MCP tools (M) — for llmgw-80, beside I2 +Kong turns OpenAPI operations into MCP tools. A company with an internal +REST API and no MCP server could offer it to agents through `/mcp`: +`llmgw ctl mcp add hr --openapi https://hr.internal/openapi.json`, each +operation a tool, under the same grants, audit, guardrails and rate as I2. + +### L7. MCP sign-in by the MCP spec (M) — for llmgw-80, part of I2 P3 +Kong's MCP OAuth2: llmgw's `/mcp` as an OAuth 2.1 resource server (protected +resource metadata, tokens checked by JWKS against the company's identity +provider), and a token exchanged for a short-lived one per upstream server +rather than one long-lived secret. + +### L8. A2A (S, when asked) +Agent-to-agent traffic through the gateway: agent cards' URLs rewritten to +llmgw's, calls audited and traced. The standard is young; wait for a +customer. + +### L9. llmgw behind Kong, written down (S) +A customer that already runs Kong puts llmgw behind it: `ai-proxy` with +`provider: openai` and `upstream_url` at llmgw (3.9, open source), or a plain +route. A page in docs/ with the configuration, tried against Kong 3.9 in a +container, and what Kong's token accounting sees. + ## D. Facts worth knowing before starting - Admin API: `127.0.0.1:9090`, behind the token in diff --git a/CHANGELOG.md b/CHANGELOG.md index 87c4259..3bbf6ab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,13 @@ APIs change between minors without a deprecation period. ### Added +- **Traces in the OpenTelemetry GenAI conventions.** An inference request + and each attempt at an engine carry `gen_ai.operation.name`, the requested + and the answering model, `max_tokens`, `temperature` and `top_p` when set, + the session as `gen_ai.conversation.id`, and the input, output and cached + tokens, and an attempt the engine that answered as `gen_ai.provider.name`. + Tools that read the conventions show llmgw's calls as model calls with no + dashboard of their own. No prompt or answer text is ever in a span. - **llmgw for VS Code.** An extension of llmgw's own, served by every gateway at `/setup/vscode/llmgw.vsix` with that gateway's address in it, made from the binary itself: plain JavaScript, no dependencies, no build, and so no diff --git a/docs/install.md b/docs/install.md index d5f9e4e..d996f55 100644 --- a/docs/install.md +++ b/docs/install.md @@ -1957,6 +1957,18 @@ So "why did qwen take four minutes to start" is one trace: find `replica start` with `llmgw.model` = qwen, and the node's spans beneath it say whether the time went to the download, the launch or the engine loading. +An inference request and each `upstream` attempt also carry the +[OpenTelemetry GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/), +so a tool that reads them — Grafana, Jaeger, Langfuse, Arize Phoenix — shows +llmgw's calls as model calls without a dashboard made for llmgw: +`gen_ai.operation.name` (`chat`, `text_completion`, `embeddings`), +`gen_ai.request.model`, `gen_ai.request.max_tokens`, `temperature` and `top_p` +when the client set them, `gen_ai.conversation.id` (the session), +`gen_ai.response.model`, and `gen_ai.usage.input_tokens`, `output_tokens` and +`cache_read.input_tokens`. On an `upstream` span `gen_ai.provider.name` is the +engine that answered (`vllm`, `llamacpp`, `external`, …). What was asked and +what was answered are never in a span. + A node follows the control plane's decision: a start the control plane kept is traced on the node whatever the agent's own `sample_ratio`, and one it sampled out is not; the agent's ratio decides only for what arrives with no trace, as diff --git a/internal/router/handlers.go b/internal/router/handlers.go index 0188ddf..b874234 100644 --- a/internal/router/handlers.go +++ b/internal/router/handlers.go @@ -450,6 +450,8 @@ type parsedRequest struct { // longest one is what a context must hold. ContextTokens int MaxTokens int + // Temperature and TopP are the client's, when it set them: for the trace. + Temperature, TopP *float64 // ThreadID is the conversation this request continues, empty for the // stateless path every request took before threads existed. @@ -505,7 +507,10 @@ type inferenceBody struct { Input json.RawMessage `json:"input"` MaxTokens int `json:"max_tokens"` MaxCompletionTokens int `json:"max_completion_tokens"` - StreamOptions *struct { + // Read only to say them in the request's trace; sent on untouched. + Temperature *float64 `json:"temperature"` + TopP *float64 `json:"top_p"` + StreamOptions *struct { IncludeUsage bool `json:"include_usage"` } `json:"stream_options"` } @@ -524,6 +529,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc { parsed, err := r.parse(req, path) if err == nil { span.SetAttributes(attribute.String("llmgw.model", parsed.Model)) + span.SetAttributes(genAIRequest(path, parsed)...) parsed.RequestID = rid if parsed.Header == nil { parsed.Header = http.Header{} @@ -688,6 +694,7 @@ func (r *Router) handleInference(path string) http.HandlerFunc { span.SetAttributes(attribute.Int("http.response.status_code", status), attribute.String("llmgw.served_model", res.model), attribute.String("llmgw.endpoint", res.endpointID)) + span.SetAttributes(genAIUsage(path, res.model, res.outcome.Usage)...) if status >= 500 { span.SetStatus(codes.Error, http.StatusText(status)) } @@ -891,6 +898,8 @@ func (r *Router) parse(req *http.Request, path string) (*parsedRequest, error) { Band: req.Header.Get(r.cfg.Admission.BandHeader), ProfileOverride: req.Header.Get("x-llmgw-profile"), MaxTokens: max(b.MaxTokens, b.MaxCompletionTokens), + Temperature: b.Temperature, + TopP: b.TopP, PromptPrefix: promptPrefix(&b), ThreadID: req.Header.Get(threadHeader), } diff --git a/internal/router/router.go b/internal/router/router.go index 71dd19a..b9a1600 100644 --- a/internal/router/router.go +++ b/internal/router/router.go @@ -1183,6 +1183,11 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa uctx, uspan := tracer().Start(ctx, "upstream", trace.WithSpanKind(trace.SpanKindClient), trace.WithAttributes(attribute.String("llmgw.endpoint", dec.Endpoint.ID), attribute.String("url.full", dec.Endpoint.URL+req.Path))) + if op := genAIOperation(req.Path); op != "" { + uspan.SetAttributes(attribute.String("gen_ai.operation.name", op), + attribute.String("gen_ai.provider.name", genAIProvider(dec.Endpoint.Engine)), + attribute.String("gen_ai.request.model", model)) + } body := req.UpstreamBody // A shared ComfyUI replica is told which of its models' checkpoint; // any other endpoint its own upstream name. @@ -1256,6 +1261,7 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa } releaseAdapter() uspan.SetAttributes(attribute.Int("http.response.status_code", out.Status)) + uspan.SetAttributes(genAIUsage(req.Path, model, out.Usage)...) endSpan(uspan, perr) adapterGone := onDemand && !out.Wrote && out.Status == http.StatusNotFound if adapterGone { diff --git a/internal/router/trace.go b/internal/router/trace.go index 0992e54..bc6b3a9 100644 --- a/internal/router/trace.go +++ b/internal/router/trace.go @@ -3,12 +3,15 @@ package router import ( "context" "net/http" + "strings" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/codes" "go.opentelemetry.io/otel/propagation" "go.opentelemetry.io/otel/trace" + + "gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy" ) // tracer is looked up per use rather than kept, so a provider installed after @@ -49,3 +52,82 @@ func withTraceparent(ctx context.Context, h http.Header) http.Header { otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(out)) return out } + +// The OpenTelemetry GenAI semantic conventions: the names a trace backend +// reads an LLM call by, so llmgw's spans show in a customer's own tools — +// Grafana, Jaeger, Langfuse, anything that speaks OpenTelemetry — as model +// calls with their model and tokens, without a dashboard made for llmgw. +// +// What a request said and what the model answered are not among them, and +// are never added: llmgw keeps no prompt text anywhere, and a trace is the +// last place one should leak into. + +// genAIOperation is the conventions' operation for an inference path, or "" +// for one they have no name for. The Anthropic Messages and Responses APIs +// are answered as chat completions, and so are chat. +func genAIOperation(path string) string { + switch { + case strings.HasSuffix(path, "/chat/completions"): + return "chat" + case strings.HasSuffix(path, "/completions"): + return "text_completion" + case strings.HasSuffix(path, "/embeddings"): + return "embeddings" + } + return "" +} + +// genAIRequest is what a request asked for, as the conventions name it. +func genAIRequest(path string, p *parsedRequest) []attribute.KeyValue { + op := genAIOperation(path) + if op == "" { + return nil + } + kv := []attribute.KeyValue{ + attribute.String("gen_ai.operation.name", op), + attribute.String("gen_ai.request.model", p.Model), + } + if p.MaxTokens > 0 { + kv = append(kv, attribute.Int("gen_ai.request.max_tokens", p.MaxTokens)) + } + if p.Temperature != nil { + kv = append(kv, attribute.Float64("gen_ai.request.temperature", *p.Temperature)) + } + if p.TopP != nil { + kv = append(kv, attribute.Float64("gen_ai.request.top_p", *p.TopP)) + } + if p.SessionID != "" { + kv = append(kv, attribute.String("gen_ai.conversation.id", p.SessionID)) + } + return kv +} + +// genAIUsage is what an answer cost, in the conventions' names: the model +// that answered, and its tokens, cached ones apart. +func genAIUsage(path, served string, u *proxy.Usage) []attribute.KeyValue { + if genAIOperation(path) == "" { + return nil + } + var kv []attribute.KeyValue + if served != "" { + kv = append(kv, attribute.String("gen_ai.response.model", served)) + } + if u != nil { + kv = append(kv, + attribute.Int("gen_ai.usage.input_tokens", u.PromptTokens), + attribute.Int("gen_ai.usage.output_tokens", u.CompletionTokens)) + if u.CachedTokens > 0 { + kv = append(kv, attribute.Int("gen_ai.usage.cache_read.input_tokens", u.CachedTokens)) + } + } + return kv +} + +// genAIProvider names who answered an attempt: the engine serving a replica +// (vllm, llamacpp, …), or the outside provider's, for an external model. +func genAIProvider(engine string) string { + if engine == "" { + return "llmgw" + } + return engine +} diff --git a/internal/router/trace_test.go b/internal/router/trace_test.go index 302f600..83d7638 100644 --- a/internal/router/trace_test.go +++ b/internal/router/trace_test.go @@ -1,6 +1,7 @@ package router import ( + "strings" "testing" "go.opentelemetry.io/otel" @@ -81,3 +82,62 @@ func TestWithoutTracingTheEngineGetsNoTraceparent(t *testing.T) { t.Errorf("traceparent %q sent with tracing off", tp) } } + +func TestTracesSpeakTheGenAIConventionsAndHoldNoText(t *testing.T) { + rec := tracetest.NewSpanRecorder() + prevTP := otel.GetTracerProvider() + otel.SetTracerProvider(sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(rec))) + t.Cleanup(func() { otel.SetTracerProvider(prevTP) }) + + e := newFakeEngine(t, "m") + rt := newTestRouter(t, baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}})) + res := post(t, rt.Handler(), map[string]any{"model": "m", "max_tokens": 64, "temperature": 0.2, "top_p": 0.9, + "messages": []any{map[string]any{"role": "user", "content": "the secret word is aubergine"}}}, nil) + if res.Code != 200 { + t.Fatalf("status %d: %s", res.Code, res.Body.String()) + } + got := map[string]map[string]string{} + for _, s := range rec.Ended() { + attrs := map[string]string{} + for _, a := range s.Attributes() { + attrs[string(a.Key)] = a.Value.Emit() + // Neither the prompt nor the reply ("hi"), in any attribute. + if v := a.Value.Emit(); strings.Contains(v, "aubergine") || v == "hi" { + t.Errorf("span %s carries text in %s", s.Name(), a.Key) + } + } + got[s.Name()] = attrs + } + root := got["inference /v1/chat/completions"] + for k, want := range map[string]string{ + "gen_ai.operation.name": "chat", + "gen_ai.request.model": "m", + "gen_ai.request.max_tokens": "64", + "gen_ai.request.temperature": "0.2", + "gen_ai.request.top_p": "0.9", + "gen_ai.response.model": "m", + "gen_ai.usage.input_tokens": "4", + "gen_ai.usage.output_tokens": "2", + } { + if root[k] != want { + t.Errorf("request span %s = %q, want %q", k, root[k], want) + } + } + up := got["upstream"] + if up["gen_ai.operation.name"] != "chat" || up["gen_ai.provider.name"] == "" || up["gen_ai.usage.output_tokens"] != "2" { + t.Errorf("upstream span %v", up) + } +} + +func TestEveryInferencePathHasItsGenAIOperation(t *testing.T) { + for path, want := range map[string]string{ + "/v1/chat/completions": "chat", + "/v1/completions": "text_completion", + "/v1/embeddings": "embeddings", + "/v1/rerank": "", + } { + if got := genAIOperation(path); got != want { + t.Errorf("%s: %q, want %q", path, got, want) + } + } +}