## CHANGELOG ### Tracing beyond the router - **The control plane traces what it does to the fleet.** A reconcile pass that does something is a `reconcile` trace — passes with nothing to do are not traced at all — with what it did beneath it: `scale up` with `place` (the profile, engine, GPU class, pool, fabric and nodes chosen, or every candidate turned down and why, as `ctl explain-placement` would print it), `scale down`, `drain replica` and `stop replica` each with its reason, `evict`, `preempt` with its victims, `reap replica` with the error and when the model is tried again, `restart replica`, `restart refused`, `node unreachable` and `node lost`. - **A replica's start is one trace, down to the node.** `replica start` stays open from placement until the replica serves, fails or is stopped, with each shard's progress as an event and how long it took. The command to each node is a `send start` span whose context travels in the command (a new `trace` field on the agent protocol's `ControlMessage`), and the agent continues the same trace: `start shard`, with `fetch weights` (local, cached, downloaded, or downloaded by another shard; a `download` with its bytes and where a resume began, the source's credentials and query left out), `launch engine` (and `wait for rendezvous` on a worker of a multi-node replica), and `wait for engine` with how often the engine was asked and its last answer. "Why did qwen take four minutes to start" is that one trace. - **Stops, drains and adoptions on a node** are in the control plane's trace too: `stop shard` and `stop engine`, saying whether the engine had to be killed; `drain shard`; and `adopt` when a registering node reports engines already running, with how many nobody claimed. An engine install is an `install engine` trace on the node. - **Model jobs are traces.** `ctl model serve` and `ctl model update` are `serve model` and `update model`, with a span per step — registry, mirror, check, catalog, placement, smoke test, replicas — and from the placement step a link to the replica's own start. - **The agent has a `tracing` block** in `agent.yaml`, the same as `server.yaml`'s (`endpoint`, `sample_ratio`; `OTEL_EXPORTER_OTLP_ENDPOINT` when unset). Point both at one collector. A node follows the control plane's sampling for every start it is sent. Traces from a node carry its id as `service.instance.id`, and a control plane's its host name. - With no endpoint nothing changes: every span is the no-op one, and a command carries no trace context. ## BACKLOG **Done (branch `features-c2`):** spans for the reconcile pass (begun only when a pass does something), placement with its answer or rejections, scale-up/down, drain, stop, eviction, preemption, reaping, restarts and lost nodes; a `replica start` span from placement to serving whose context travels to the agent in the start command (`ControlMessage.trace`), where weights fetch, engine launch, rendezvous and readiness wait continue the trace; stops, drains, adoption and engine installs; modelserve jobs as a trace with a span per step linked to the replica's start; an agent `tracing` block; docs in install.md's "Tracing". Left: the router's span for a request does not link to the replica it was sent to's start; heartbeats, fabric scans, logs and agent upgrades are not traced; the supervisor has no restart span because it never restarts an engine itself (a restart is the reconciler's `restart replica` and reaches a node as a stop and a start); canary and promote jobs, which landed on `features-c2` after this was branched, are traced only through the shared step recording (as ` model`), unchecked by a test.