llmgw/notes/c20.md

CHANGELOG

Tracing beyond the router

• The control plane traces what it does to the fleet. A reconcile pass that does something is a reconcile trace — passes with nothing to do are not traced at all — with what it did beneath it: scale up with place (the profile, engine, GPU class, pool, fabric and nodes chosen, or every candidate turned down and why, as ctl explain-placement would print it), scale down, drain replica and stop replica each with its reason, evict, preempt with its victims, reap replica with the error and when the model is tried again, restart replica, restart refused, node unreachable and node lost.
• A replica's start is one trace, down to the node. replica start stays open from placement until the replica serves, fails or is stopped, with each shard's progress as an event and how long it took. The command to each node is a send start span whose context travels in the command (a new trace field on the agent protocol's ControlMessage), and the agent continues the same trace: start shard, with fetch weights (local, cached, downloaded, or downloaded by another shard; a download with its bytes and where a resume began, the source's credentials and query left out), launch engine (and wait for rendezvous on a worker of a multi-node replica), and wait for engine with how often the engine was asked and its last answer. "Why did qwen take four minutes to start" is that one trace.
• Stops, drains and adoptions on a node are in the control plane's trace too: stop shard and stop engine, saying whether the engine had to be killed; drain shard; and adopt when a registering node reports engines already running, with how many nobody claimed. An engine install is an install engine trace on the node.
• Model jobs are traces. ctl model serve and ctl model update are serve model and update model, with a span per step — registry, mirror, check, catalog, placement, smoke test, replicas — and from the placement step a link to the replica's own start.
• The agent has a tracing block in agent.yaml, the same as server.yaml's (endpoint, sample_ratio; OTEL_EXPORTER_OTLP_ENDPOINT when unset). Point both at one collector. A node follows the control plane's sampling for every start it is sent. Traces from a node carry its id as service.instance.id, and a control plane's its host name.
• With no endpoint nothing changes: every span is the no-op one, and a command carries no trace context.

BACKLOG

Done (branch features-c2): spans for the reconcile pass (begun only when a pass does something), placement with its answer or rejections, scale-up/down, drain, stop, eviction, preemption, reaping, restarts and lost nodes; a replica start span from placement to serving whose context travels to the agent in the start command (ControlMessage.trace), where weights fetch, engine launch, rendezvous and readiness wait continue the trace; stops, drains, adoption and engine installs; modelserve jobs as a trace with a span per step linked to the replica's start; an agent tracing block; docs in install.md's "Tracing". Left: the router's span for a request does not link to the replica it was sent to's start; heartbeats, fabric scans, logs and agent upgrades are not traced; the supervisor has no restart span because it never restarts an engine itself (a restart is the reconciler's restart replica and reaches a node as a stop and a start); canary and promote jobs, which landed on features-c2 after this was branched, are traced only through the shared step recording (as <kind> model), unchecked by a test.