Tracing beyond the router: reconcile passes, placement and a replica's start traced down to the node

A reconcile pass that does something is a trace, with placement (where it
went, or every candidate turned down), scale-up/down, drain, stop, eviction,
preemption, reaping and restarts beneath it. A replica's start is a span open
from placement until it serves; the start command carries its context in a
new ControlMessage.trace field, and the agent continues the trace through
the weights fetch, engine launch, rendezvous and readiness wait. Stops,
drains, adoption and engine installs are traced on the node, model jobs are
a trace with a span per step linked to the replica's start, and agent.yaml
takes the same tracing block as server.yaml. Off, every span is the no-op one
and commands carry nothing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:03:17 +02:00
1 parent e59ff52932
commit e725adc2f0
33 files changed
+1607 -143

No files matched your search

+22 -6
View File
@@ -224,7 +224,7 @@ rather than repeated.
| `debug_pprof` | `bool` | [Server](#server) |
| `agent_insecure` | `bool` | [Server](#server) |
| `monitoring` | `Monitoring` | [Server](#server) |
| `tracing` | `Tracing` | [Server](#server) |
| `tracing` | `obs.Tracing` | [Server](#server) |
| `mirror` | `Mirror` | [Server](#server) |
| `engines` | `map[string]string` | [Server](#server) |
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
@@ -291,6 +291,7 @@ rather than repeated.
| `discovery` | `discovery.Options` | [agent.Config](#agentconfig) |
| `engines` | `engines.Config` | [agent.Config](#agentconfig) |
| `supervisor` | `supervisor.Config` | [agent.Config](#agentconfig) |
| `tracing` | `obs.Tracing` | [agent.Config](#agentconfig) |
| `name` | `string` | [config.Band](#configband) |
| `priority` | `int` | [config.Band](#configband) |
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
@@ -1817,10 +1818,11 @@ Takes a [Monitoring](#monitoring) block.
### `tracing`
`Tracing`
`obs.Tracing`
Tracing sends a trace of each inference request to an OpenTelemetry
collector. Off unless an endpoint is set here or in
Tracing sends traces to an OpenTelemetry collector: each inference
request, and on a control plane each placement, replica start and
model job. Off unless an endpoint is set here or in
OTEL_EXPORTER_OTLP_ENDPOINT.
Takes a [Tracing](#tracing) block.
@@ -2044,8 +2046,10 @@ Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
`float64`
SampleRatio is the share of requests traced, 0 to 1, when the caller
did not decide it with a sampled traceparent of its own. Zero takes 1.
SampleRatio is the share of traces kept, 0 to 1, of those nobody
upstream already decided. Zero takes 1. A request with a sampled
traceparent of its own is kept, and an agent keeps every start the
control plane kept, whatever their own ratio says.
## VLLMConfig
@@ -2316,6 +2320,18 @@ Takes a [engines.Config](#enginesconfig) block.
Takes a [supervisor.Config](#supervisorconfig) block.
### `tracing`
`obs.Tracing`
Tracing sends this node's part of each replica start — the weights
fetched, the engine launched, the wait for it to answer — to an
OpenTelemetry collector, inside the control plane's trace of that
start. Give it the collector server.yaml's tracing names. Off unless
an endpoint is set here or in OTEL_EXPORTER_OTLP_ENDPOINT.
Takes a [Tracing](#tracing) block.
## config.Band
### `name`