A reconcile pass that does something is a trace, with placement (where it went, or every candidate turned down), scale-up/down, drain, stop, eviction, preemption, reaping and restarts beneath it. A replica's start is a span open from placement until it serves; the start command carries its context in a new ControlMessage.trace field, and the agent continues the trace through the weights fetch, engine launch, rendezvous and readiness wait. Stops, drains, adoption and engine installs are traced on the node, model jobs are a trace with a span per step linked to the replica's start, and agent.yaml takes the same tracing block as server.yaml. Off, every span is the no-op one and commands carry nothing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
61 lines
3.6 KiB
Markdown
61 lines
3.6 KiB
Markdown
## CHANGELOG
|
|
|
|
### Tracing beyond the router
|
|
|
|
- **The control plane traces what it does to the fleet.** A reconcile pass
|
|
that does something is a `reconcile` trace — passes with nothing to do are
|
|
not traced at all — with what it did beneath it: `scale up` with `place`
|
|
(the profile, engine, GPU class, pool, fabric and nodes chosen, or every
|
|
candidate turned down and why, as `ctl explain-placement` would print it),
|
|
`scale down`, `drain replica` and `stop replica` each with its reason,
|
|
`evict`, `preempt` with its victims, `reap replica` with the error and
|
|
when the model is tried again, `restart replica`, `restart refused`,
|
|
`node unreachable` and `node lost`.
|
|
- **A replica's start is one trace, down to the node.** `replica start`
|
|
stays open from placement until the replica serves, fails or is stopped,
|
|
with each shard's progress as an event and how long it took. The command
|
|
to each node is a `send start` span whose context travels in the command
|
|
(a new `trace` field on the agent protocol's `ControlMessage`), and the
|
|
agent continues the same trace: `start shard`, with `fetch weights`
|
|
(local, cached, downloaded, or downloaded by another shard; a `download`
|
|
with its bytes and where a resume began, the source's credentials and
|
|
query left out), `launch engine` (and `wait for rendezvous` on a worker
|
|
of a multi-node replica), and `wait for engine` with how often the engine
|
|
was asked and its last answer. "Why did qwen take four minutes to start"
|
|
is that one trace.
|
|
- **Stops, drains and adoptions on a node** are in the control plane's
|
|
trace too: `stop shard` and `stop engine`, saying whether the engine had
|
|
to be killed; `drain shard`; and `adopt` when a registering node reports
|
|
engines already running, with how many nobody claimed. An engine install
|
|
is an `install engine` trace on the node.
|
|
- **Model jobs are traces.** `ctl model serve` and `ctl model update` are
|
|
`serve model` and `update model`, with a span per step — registry,
|
|
mirror, check, catalog, placement, smoke test, replicas — and from the
|
|
placement step a link to the replica's own start.
|
|
- **The agent has a `tracing` block** in `agent.yaml`, the same as
|
|
`server.yaml`'s (`endpoint`, `sample_ratio`; `OTEL_EXPORTER_OTLP_ENDPOINT`
|
|
when unset). Point both at one collector. A node follows the control
|
|
plane's sampling for every start it is sent. Traces from a node carry its
|
|
id as `service.instance.id`, and a control plane's its host name.
|
|
- With no endpoint nothing changes: every span is the no-op one, and a
|
|
command carries no trace context.
|
|
|
|
## BACKLOG
|
|
|
|
**Done (branch `features-c2`):** spans for the reconcile pass (begun only
|
|
when a pass does something), placement with its answer or rejections,
|
|
scale-up/down, drain, stop, eviction, preemption, reaping, restarts and
|
|
lost nodes; a `replica start` span from placement to serving whose context
|
|
travels to the agent in the start command (`ControlMessage.trace`), where
|
|
weights fetch, engine launch, rendezvous and readiness wait continue the
|
|
trace; stops, drains, adoption and engine installs; modelserve jobs as a
|
|
trace with a span per step linked to the replica's start; an agent
|
|
`tracing` block; docs in install.md's "Tracing". Left: the router's span
|
|
for a request does not link to the replica it was sent to's start;
|
|
heartbeats, fabric scans, logs and agent upgrades are not traced; the
|
|
supervisor has no restart span because it never restarts an engine itself
|
|
(a restart is the reconciler's `restart replica` and reaches a node as a
|
|
stop and a start); canary and promote jobs, which landed on `features-c2`
|
|
after this was branched, are traced only through the shared step recording
|
|
(as `<kind> model`), unchecked by a test.
|