Files
llmgw/notes/c20.md
T
Ilker ManapandClaude Opus 5.5 e725adc2f0 Tracing beyond the router: reconcile passes, placement and a replica's start traced down to the node
A reconcile pass that does something is a trace, with placement (where it
went, or every candidate turned down), scale-up/down, drain, stop, eviction,
preemption, reaping and restarts beneath it. A replica's start is a span open
from placement until it serves; the start command carries its context in a
new ControlMessage.trace field, and the agent continues the trace through
the weights fetch, engine launch, rendezvous and readiness wait. Stops,
drains, adoption and engine installs are traced on the node, model jobs are
a trace with a span per step linked to the replica's start, and agent.yaml
takes the same tracing block as server.yaml. Off, every span is the no-op one
and commands carry nothing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:03:17 +02:00

61 lines
3.6 KiB
Markdown

## CHANGELOG
### Tracing beyond the router
- **The control plane traces what it does to the fleet.** A reconcile pass
that does something is a `reconcile` trace — passes with nothing to do are
not traced at all — with what it did beneath it: `scale up` with `place`
(the profile, engine, GPU class, pool, fabric and nodes chosen, or every
candidate turned down and why, as `ctl explain-placement` would print it),
`scale down`, `drain replica` and `stop replica` each with its reason,
`evict`, `preempt` with its victims, `reap replica` with the error and
when the model is tried again, `restart replica`, `restart refused`,
`node unreachable` and `node lost`.
- **A replica's start is one trace, down to the node.** `replica start`
stays open from placement until the replica serves, fails or is stopped,
with each shard's progress as an event and how long it took. The command
to each node is a `send start` span whose context travels in the command
(a new `trace` field on the agent protocol's `ControlMessage`), and the
agent continues the same trace: `start shard`, with `fetch weights`
(local, cached, downloaded, or downloaded by another shard; a `download`
with its bytes and where a resume began, the source's credentials and
query left out), `launch engine` (and `wait for rendezvous` on a worker
of a multi-node replica), and `wait for engine` with how often the engine
was asked and its last answer. "Why did qwen take four minutes to start"
is that one trace.
- **Stops, drains and adoptions on a node** are in the control plane's
trace too: `stop shard` and `stop engine`, saying whether the engine had
to be killed; `drain shard`; and `adopt` when a registering node reports
engines already running, with how many nobody claimed. An engine install
is an `install engine` trace on the node.
- **Model jobs are traces.** `ctl model serve` and `ctl model update` are
`serve model` and `update model`, with a span per step — registry,
mirror, check, catalog, placement, smoke test, replicas — and from the
placement step a link to the replica's own start.
- **The agent has a `tracing` block** in `agent.yaml`, the same as
`server.yaml`'s (`endpoint`, `sample_ratio`; `OTEL_EXPORTER_OTLP_ENDPOINT`
when unset). Point both at one collector. A node follows the control
plane's sampling for every start it is sent. Traces from a node carry its
id as `service.instance.id`, and a control plane's its host name.
- With no endpoint nothing changes: every span is the no-op one, and a
command carries no trace context.
## BACKLOG
**Done (branch `features-c2`):** spans for the reconcile pass (begun only
when a pass does something), placement with its answer or rejections,
scale-up/down, drain, stop, eviction, preemption, reaping, restarts and
lost nodes; a `replica start` span from placement to serving whose context
travels to the agent in the start command (`ControlMessage.trace`), where
weights fetch, engine launch, rendezvous and readiness wait continue the
trace; stops, drains, adoption and engine installs; modelserve jobs as a
trace with a span per step linked to the replica's start; an agent
`tracing` block; docs in install.md's "Tracing". Left: the router's span
for a request does not link to the replica it was sent to's start;
heartbeats, fabric scans, logs and agent upgrades are not traced; the
supervisor has no restart span because it never restarts an engine itself
(a restart is the reconciler's `restart replica` and reaches a node as a
stop and a start); canary and promote jobs, which landed on `features-c2`
after this was branched, are traced only through the shared step recording
(as `<kind> model`), unchecked by a test.