Tracing beyond the router: reconcile passes, placement and a replica's start traced down to the node

A reconcile pass that does something is a trace, with placement (where it
went, or every candidate turned down), scale-up/down, drain, stop, eviction,
preemption, reaping and restarts beneath it. A replica's start is a span open
from placement until it serves; the start command carries its context in a
new ControlMessage.trace field, and the agent continues the trace through
the weights fetch, engine launch, rendezvous and readiness wait. Stops,
drains, adoption and engine installs are traced on the node, model jobs are
a trace with a span per step linked to the replica's start, and agent.yaml
takes the same tracing block as server.yaml. Off, every span is the no-op one
and commands carry nothing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:03:17 +02:00
1 parent e59ff52932
commit e725adc2f0
33 files changed
+1607 -143

No files matched your search

+40
View File
@@ -1002,6 +1002,46 @@ same event about the same thing is sent at most once an hour, so a budget that
refuses a thousand requests is one message. `llmgw ctl notify test` sends a
test event to each webhook and says how each answered.
### Tracing
Metrics say that a model took four minutes to start; a trace says where they
went. Give the control plane and every agent the same OpenTelemetry collector,
over OTLP/HTTP — Jaeger, Tempo, Honeycomb, anything that takes OTLP:
```yaml
# /etc/llmgw/server.yaml, and the same block in each node's agent.yaml
tracing:
endpoint: http://otel.internal:4318
sample_ratio: 1 # of requests that arrive without a decision of their own
```
`OTEL_EXPORTER_OTLP_ENDPOINT` does the same when the file does not say. With
neither, nothing is traced and nothing is sent: every span is a no-op, and a
command to a node carries no trace context.
What is traced:
| Trace | Spans |
|---|---|
| An inference request (`llmgw-server`) | `inference /v1/…` with `admission`, `schedule` and an `upstream` per attempt beneath it; a caller's `traceparent` is continued and the engine is sent one. |
| A reconcile pass that did something | `reconcile`, and beneath it what it did: `scale up` with `place` (the profile, GPU class, pool, fabric and nodes chosen, or every candidate turned down and why) and `replica start`; `scale down`; `drain replica` and `stop replica`, each with its reason; `evict`, `preempt` with its victims; `reap replica` with the error and when it is tried again; `restart replica`, `restart refused`; `node unreachable`, `node lost`. A pass that found nothing to do is not traced. |
| A replica's start | `replica start` stays open from placement until the replica serves, fails or is stopped, with each shard's progress as an event on it and how long it took. A `send start` per shard beneath it is the command to the node, and the node's spans (`llmgw-agent`) continue the same trace: `start shard`, with `fetch weights` (local, cached, downloaded or fetched by another shard, and a `download` with its bytes), `launch engine` (and `wait for rendezvous` on a worker of a multi-node replica) and `wait for engine` (how often the engine was asked, and its last answer if it never came up). |
| A stop on a node | `stop shard` and `stop engine` in the trace of the control plane's `stop replica`, saying whether the engine had to be killed. |
| An adoption | `adopt`, when a node that registers reports engines already running, with how many nobody claimed. |
| A model job | `serve model` or `update model`, with a span per step — `registry`, `mirror`, `check`, `catalog`, `placement`, `smoke test`, `replicas` — and from the placement step a link to the replica's own start. |
| An engine install | `install engine` on the node, a trace of its own. |
So "why did qwen take four minutes to start" is one trace: find `replica start`
with `llmgw.model` = qwen, and the node's spans beneath it say whether the
time went to the download, the launch or the engine loading.
A node follows the control plane's decision: a start the control plane kept is
traced on the node whatever the agent's own `sample_ratio`, and one it sampled
out is not; the agent's ratio decides only for what arrives with no trace, as
from a control plane that is not tracing. The agent needs its own `tracing`
block either way — without one its spans go nowhere, and the control plane's
trace simply ends at `send start`.
## PostgreSQL, when SQLite is no longer the right shape
SQLite is the default and stays the right answer for most installations: one
+22 -6
View File
@@ -224,7 +224,7 @@ rather than repeated.
| `debug_pprof` | `bool` | [Server](#server) |
| `agent_insecure` | `bool` | [Server](#server) |
| `monitoring` | `Monitoring` | [Server](#server) |
| `tracing` | `Tracing` | [Server](#server) |
| `tracing` | `obs.Tracing` | [Server](#server) |
| `mirror` | `Mirror` | [Server](#server) |
| `engines` | `map[string]string` | [Server](#server) |
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
@@ -291,6 +291,7 @@ rather than repeated.
| `discovery` | `discovery.Options` | [agent.Config](#agentconfig) |
| `engines` | `engines.Config` | [agent.Config](#agentconfig) |
| `supervisor` | `supervisor.Config` | [agent.Config](#agentconfig) |
| `tracing` | `obs.Tracing` | [agent.Config](#agentconfig) |
| `name` | `string` | [config.Band](#configband) |
| `priority` | `int` | [config.Band](#configband) |
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
@@ -1817,10 +1818,11 @@ Takes a [Monitoring](#monitoring) block.
### `tracing`
`Tracing`
`obs.Tracing`
Tracing sends a trace of each inference request to an OpenTelemetry
collector. Off unless an endpoint is set here or in
Tracing sends traces to an OpenTelemetry collector: each inference
request, and on a control plane each placement, replica start and
model job. Off unless an endpoint is set here or in
OTEL_EXPORTER_OTLP_ENDPOINT.
Takes a [Tracing](#tracing) block.
@@ -2044,8 +2046,10 @@ Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
`float64`
SampleRatio is the share of requests traced, 0 to 1, when the caller
did not decide it with a sampled traceparent of its own. Zero takes 1.
SampleRatio is the share of traces kept, 0 to 1, of those nobody
upstream already decided. Zero takes 1. A request with a sampled
traceparent of its own is kept, and an agent keeps every start the
control plane kept, whatever their own ratio says.
## VLLMConfig
@@ -2316,6 +2320,18 @@ Takes a [engines.Config](#enginesconfig) block.
Takes a [supervisor.Config](#supervisorconfig) block.
### `tracing`
`obs.Tracing`
Tracing sends this node's part of each replica start — the weights
fetched, the engine launched, the wait for it to answer — to an
OpenTelemetry collector, inside the control plane's trace of that
start. Give it the collector server.yaml's tracing names. Off unless
an endpoint is set here or in OTEL_EXPORTER_OTLP_ENDPOINT.
Takes a [Tracing](#tracing) block.
## config.Band
### `name`