Merge C20 tracing beyond the router

Conflicts: config imports and docgen sources (both kept); agent.pb.go
regenerated with buf from the merged proto (MIG devices and trace context).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:05:29 +02:00
commit 801e2d51c0
33 files changed
+1608 -144

No files matched your search

+60
View File
@@ -0,0 +1,60 @@
## CHANGELOG
### Tracing beyond the router
- **The control plane traces what it does to the fleet.** A reconcile pass
that does something is a `reconcile` trace — passes with nothing to do are
not traced at all — with what it did beneath it: `scale up` with `place`
(the profile, engine, GPU class, pool, fabric and nodes chosen, or every
candidate turned down and why, as `ctl explain-placement` would print it),
`scale down`, `drain replica` and `stop replica` each with its reason,
`evict`, `preempt` with its victims, `reap replica` with the error and
when the model is tried again, `restart replica`, `restart refused`,
`node unreachable` and `node lost`.
- **A replica's start is one trace, down to the node.** `replica start`
stays open from placement until the replica serves, fails or is stopped,
with each shard's progress as an event and how long it took. The command
to each node is a `send start` span whose context travels in the command
(a new `trace` field on the agent protocol's `ControlMessage`), and the
agent continues the same trace: `start shard`, with `fetch weights`
(local, cached, downloaded, or downloaded by another shard; a `download`
with its bytes and where a resume began, the source's credentials and
query left out), `launch engine` (and `wait for rendezvous` on a worker
of a multi-node replica), and `wait for engine` with how often the engine
was asked and its last answer. "Why did qwen take four minutes to start"
is that one trace.
- **Stops, drains and adoptions on a node** are in the control plane's
trace too: `stop shard` and `stop engine`, saying whether the engine had
to be killed; `drain shard`; and `adopt` when a registering node reports
engines already running, with how many nobody claimed. An engine install
is an `install engine` trace on the node.
- **Model jobs are traces.** `ctl model serve` and `ctl model update` are
`serve model` and `update model`, with a span per step — registry,
mirror, check, catalog, placement, smoke test, replicas — and from the
placement step a link to the replica's own start.
- **The agent has a `tracing` block** in `agent.yaml`, the same as
`server.yaml`'s (`endpoint`, `sample_ratio`; `OTEL_EXPORTER_OTLP_ENDPOINT`
when unset). Point both at one collector. A node follows the control
plane's sampling for every start it is sent. Traces from a node carry its
id as `service.instance.id`, and a control plane's its host name.
- With no endpoint nothing changes: every span is the no-op one, and a
command carries no trace context.
## BACKLOG
**Done (branch `features-c2`):** spans for the reconcile pass (begun only
when a pass does something), placement with its answer or rejections,
scale-up/down, drain, stop, eviction, preemption, reaping, restarts and
lost nodes; a `replica start` span from placement to serving whose context
travels to the agent in the start command (`ControlMessage.trace`), where
weights fetch, engine launch, rendezvous and readiness wait continue the
trace; stops, drains, adoption and engine installs; modelserve jobs as a
trace with a span per step linked to the replica's start; an agent
`tracing` block; docs in install.md's "Tracing". Left: the router's span
for a request does not link to the replica it was sent to's start;
heartbeats, fabric scans, logs and agent upgrades are not traced; the
supervisor has no restart span because it never restarts an engine itself
(a restart is the reconciler's `restart replica` and reaches a node as a
stop and a start); canary and promote jobs, which landed on `features-c2`
after this was branched, are traced only through the shared step recording
(as `<kind> model`), unchecked by a test.