Tracing beyond the router: reconcile passes, placement and a replica's start traced down to the node

A reconcile pass that does something is a trace, with placement (where it
went, or every candidate turned down), scale-up/down, drain, stop, eviction,
preemption, reaping and restarts beneath it. A replica's start is a span open
from placement until it serves; the start command carries its context in a
new ControlMessage.trace field, and the agent continues the trace through
the weights fetch, engine launch, rendezvous and readiness wait. Stops,
drains, adoption and engine installs are traced on the node, model jobs are
a trace with a span per step linked to the replica's start, and agent.yaml
takes the same tracing block as server.yaml. Off, every span is the no-op one
and commands carry nothing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:03:17 +02:00
1 parent e59ff52932
commit e725adc2f0
33 files changed
+1607 -143

No files matched your search

+4 -3
View File
@@ -278,7 +278,7 @@ var Reference = []Block{
{Key: "debug_pprof", Type: "bool", Doc: "DebugPprof serves Go's profiler on the admin listener, behind the token.\nOff by default: a heap profile holds the API keys requests presented.", Nested: ""},
{Key: "agent_insecure", Type: "bool", Doc: "AgentInsecure lets a control plane run its agent port without\nagent_tls: plaintext, and any machine that reaches it may register as\nany node — be given replicas, and so be sent users' prompts — or, in\nthe other direction, a machine in the path may tell every agent what\nto run. Refused unless said here, and then the port listens on\nloopback unless agent_listen names an address. For one machine, a\ntest, or a network nothing untrusted reaches.", Nested: ""},
{Key: "monitoring", Type: "Monitoring", Doc: "Monitoring is where the dashboard reads its history from.", Nested: "config.Monitoring"},
{Key: "tracing", Type: "Tracing", Doc: "Tracing sends a trace of each inference request to an OpenTelemetry\ncollector. Off unless an endpoint is set here or in\nOTEL_EXPORTER_OTLP_ENDPOINT.", Nested: "config.Tracing"},
{Key: "tracing", Type: "obs.Tracing", Doc: "Tracing sends traces to an OpenTelemetry collector: each inference\nrequest, and on a control plane each placement, replica start and\nmodel job. Off unless an endpoint is set here or in\nOTEL_EXPORTER_OTLP_ENDPOINT.", Nested: "obs.Tracing"},
{Key: "mirror", Type: "Mirror", Doc: "Mirror points the control plane at a model mirror it can administer.\nEmpty means there is none, and /admin/mirror stays unmounted.", Nested: "config.Mirror"},
{Key: "engines", Type: "map[string]string", Doc: "Engines are the engines every node installs, and at which version. The\nagents install them themselves — the right build for their hardware,\nchecked against its publisher's checksum and seen to run — so a node is\nnever trusted to already have what somebody once put there.\n\nUnset keeps DefaultEngines. A version replaces the default; an empty\none leaves that engine to be installed by hand.", Nested: ""},
{Key: "engine_overrides", Type: "[]EngineOverride", Doc: "EngineOverrides give some nodes different engines from the rest: a\nvLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at\nall on nodes meant only for llama.cpp. Each picks nodes and lays its\nengines over the cluster's, in order, later ones last; an empty version\nremoves that engine from the nodes it picks.", Nested: ""},
@@ -315,9 +315,9 @@ var Reference = []Block{
{Key: "max_calls_per_turn", Type: "int", Doc: "MaxCallsPerTurn stops an answer calling more tools than this.", Nested: ""},
{Key: "validate_arguments", Type: "bool", Doc: "ValidateArguments checks a call's arguments against the parameters\nschema the request declared for the tool.", Nested: ""},
}},
{Package: "config", Name: "Tracing", Title: "Tracing", Doc: "", Fields: []Field{
{Package: "obs", Name: "Tracing", Title: "Tracing", Doc: "", Fields: []Field{
{Key: "endpoint", Type: "string", Doc: "Endpoint is the collector, for example http://otel.internal:4318.\nEmpty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.", Nested: ""},
{Key: "sample_ratio", Type: "float64", Doc: "SampleRatio is the share of requests traced, 0 to 1, when the caller\ndid not decide it with a sampled traceparent of its own. Zero takes 1.", Nested: ""},
{Key: "sample_ratio", Type: "float64", Doc: "SampleRatio is the share of traces kept, 0 to 1, of those nobody\nupstream already decided. Zero takes 1. A request with a sampled\ntraceparent of its own is kept, and an agent keeps every start the\ncontrol plane kept, whatever their own ratio says.", Nested: ""},
}},
{Package: "engines", Name: "VLLMConfig", Title: "VLLMConfig", Doc: "", Fields: []Field{
{Key: "bin", Type: "string", Doc: "Bin is the vllm executable.", Nested: ""},
@@ -367,6 +367,7 @@ var Reference = []Block{
{Key: "discovery", Type: "discovery.Options", Doc: "", Nested: "discovery.Options"},
{Key: "engines", Type: "engines.Config", Doc: "", Nested: "engines.Config"},
{Key: "supervisor", Type: "supervisor.Config", Doc: "", Nested: "supervisor.Config"},
{Key: "tracing", Type: "obs.Tracing", Doc: "Tracing sends this node's part of each replica start — the weights\nfetched, the engine launched, the wait for it to answer — to an\nOpenTelemetry collector, inside the control plane's trace of that\nstart. Give it the collector server.yaml's tracing names. Off unless\nan endpoint is set here or in OTEL_EXPORTER_OTLP_ENDPOINT.", Nested: "obs.Tracing"},
}},
{Package: "config", Name: "Band", Title: "config.Band", Doc: "", Fields: []Field{
{Key: "name", Type: "string", Doc: "", Nested: ""},
+1
View File
@@ -30,6 +30,7 @@ var Sources = []string{
"internal/guard/judge.go",
"internal/config/oidc.go",
"internal/notify/notify.go",
"internal/obs/trace.go",
}
// notReference are files with koanf tags that an operator never writes.