Merge C20 tracing beyond the router

Conflicts: config imports and docgen sources (both kept); agent.pb.go
regenerated with buf from the merged proto (MIG devices and trace context).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:05:29 +02:00
commit 801e2d51c0
33 files changed
+1608 -144

No files matched your search

+40
View File
@@ -1272,6 +1272,46 @@ same event about the same thing is sent at most once an hour, so a budget that
refuses a thousand requests is one message. `llmgw ctl notify test` sends a
test event to each webhook and says how each answered.
### Tracing
Metrics say that a model took four minutes to start; a trace says where they
went. Give the control plane and every agent the same OpenTelemetry collector,
over OTLP/HTTP — Jaeger, Tempo, Honeycomb, anything that takes OTLP:
```yaml
# /etc/llmgw/server.yaml, and the same block in each node's agent.yaml
tracing:
endpoint: http://otel.internal:4318
sample_ratio: 1 # of requests that arrive without a decision of their own
```
`OTEL_EXPORTER_OTLP_ENDPOINT` does the same when the file does not say. With
neither, nothing is traced and nothing is sent: every span is a no-op, and a
command to a node carries no trace context.
What is traced:
| Trace | Spans |
|---|---|
| An inference request (`llmgw-server`) | `inference /v1/…` with `admission`, `schedule` and an `upstream` per attempt beneath it; a caller's `traceparent` is continued and the engine is sent one. |
| A reconcile pass that did something | `reconcile`, and beneath it what it did: `scale up` with `place` (the profile, GPU class, pool, fabric and nodes chosen, or every candidate turned down and why) and `replica start`; `scale down`; `drain replica` and `stop replica`, each with its reason; `evict`, `preempt` with its victims; `reap replica` with the error and when it is tried again; `restart replica`, `restart refused`; `node unreachable`, `node lost`. A pass that found nothing to do is not traced. |
| A replica's start | `replica start` stays open from placement until the replica serves, fails or is stopped, with each shard's progress as an event on it and how long it took. A `send start` per shard beneath it is the command to the node, and the node's spans (`llmgw-agent`) continue the same trace: `start shard`, with `fetch weights` (local, cached, downloaded or fetched by another shard, and a `download` with its bytes), `launch engine` (and `wait for rendezvous` on a worker of a multi-node replica) and `wait for engine` (how often the engine was asked, and its last answer if it never came up). |
| A stop on a node | `stop shard` and `stop engine` in the trace of the control plane's `stop replica`, saying whether the engine had to be killed. |
| An adoption | `adopt`, when a node that registers reports engines already running, with how many nobody claimed. |
| A model job | `serve model` or `update model`, with a span per step — `registry`, `mirror`, `check`, `catalog`, `placement`, `smoke test`, `replicas` — and from the placement step a link to the replica's own start. |
| An engine install | `install engine` on the node, a trace of its own. |
So "why did qwen take four minutes to start" is one trace: find `replica start`
with `llmgw.model` = qwen, and the node's spans beneath it say whether the
time went to the download, the launch or the engine loading.
A node follows the control plane's decision: a start the control plane kept is
traced on the node whatever the agent's own `sample_ratio`, and one it sampled
out is not; the agent's ratio decides only for what arrives with no trace, as
from a control plane that is not tracing. The agent needs its own `tracing`
block either way — without one its spans go nowhere, and the control plane's
trace simply ends at `send start`.
## PostgreSQL, when SQLite is no longer the right shape
SQLite is the default and stays the right answer for most installations: one
+22 -6
View File
@@ -250,7 +250,7 @@ rather than repeated.
| `debug_pprof` | `bool` | [Server](#server) |
| `agent_insecure` | `bool` | [Server](#server) |
| `monitoring` | `Monitoring` | [Server](#server) |
| `tracing` | `Tracing` | [Server](#server) |
| `tracing` | `obs.Tracing` | [Server](#server) |
| `mirror` | `Mirror` | [Server](#server) |
| `engines` | `map[string]string` | [Server](#server) |
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
@@ -327,6 +327,7 @@ rather than repeated.
| `discovery` | `discovery.Options` | [agent.Config](#agentconfig) |
| `engines` | `engines.Config` | [agent.Config](#agentconfig) |
| `supervisor` | `supervisor.Config` | [agent.Config](#agentconfig) |
| `tracing` | `obs.Tracing` | [agent.Config](#agentconfig) |
| `name` | `string` | [config.Band](#configband) |
| `priority` | `int` | [config.Band](#configband) |
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
@@ -2063,10 +2064,11 @@ Takes a [Monitoring](#monitoring) block.
### `tracing`
`Tracing`
`obs.Tracing`
Tracing sends a trace of each inference request to an OpenTelemetry
collector. Off unless an endpoint is set here or in
Tracing sends traces to an OpenTelemetry collector: each inference
request, and on a control plane each placement, replica start and
model job. Off unless an endpoint is set here or in
OTEL_EXPORTER_OTLP_ENDPOINT.
Takes a [Tracing](#tracing) block.
@@ -2331,8 +2333,10 @@ Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
`float64`
SampleRatio is the share of requests traced, 0 to 1, when the caller
did not decide it with a sampled traceparent of its own. Zero takes 1.
SampleRatio is the share of traces kept, 0 to 1, of those nobody
upstream already decided. Zero takes 1. A request with a sampled
traceparent of its own is kept, and an agent keeps every start the
control plane kept, whatever their own ratio says.
## VLLMConfig
@@ -2639,6 +2643,18 @@ Takes a [engines.Config](#enginesconfig) block.
Takes a [supervisor.Config](#supervisorconfig) block.
### `tracing`
`obs.Tracing`
Tracing sends this node's part of each replica start — the weights
fetched, the engine launched, the wait for it to answer — to an
OpenTelemetry collector, inside the control plane's trace of that
start. Give it the collector server.yaml's tracing names. Off unless
an endpoint is set here or in OTEL_EXPORTER_OTLP_ENDPOINT.
Takes a [Tracing](#tracing) block.
## config.Band
### `name`