Tracing beyond the router: reconcile passes, placement and a replica's start traced down to the node
A reconcile pass that does something is a trace, with placement (where it went, or every candidate turned down), scale-up/down, drain, stop, eviction, preemption, reaping and restarts beneath it. A replica's start is a span open from placement until it serves; the start command carries its context in a new ControlMessage.trace field, and the agent continues the trace through the weights fetch, engine launch, rendezvous and readiness wait. Stops, drains, adoption and engine installs are traced on the node, model jobs are a trace with a span per step linked to the replica's start, and agent.yaml takes the same tracing block as server.yaml. Off, every span is the no-op one and commands carry nothing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
e59ff52932
commit
e725adc2f0
33 files changed
+1607
-143
No files matched your search
@@ -1002,6 +1002,46 @@ same event about the same thing is sent at most once an hour, so a budget that
|
||||
refuses a thousand requests is one message. `llmgw ctl notify test` sends a
|
||||
test event to each webhook and says how each answered.
|
||||
|
||||
### Tracing
|
||||
|
||||
Metrics say that a model took four minutes to start; a trace says where they
|
||||
went. Give the control plane and every agent the same OpenTelemetry collector,
|
||||
over OTLP/HTTP — Jaeger, Tempo, Honeycomb, anything that takes OTLP:
|
||||
|
||||
```yaml
|
||||
# /etc/llmgw/server.yaml, and the same block in each node's agent.yaml
|
||||
tracing:
|
||||
endpoint: http://otel.internal:4318
|
||||
sample_ratio: 1 # of requests that arrive without a decision of their own
|
||||
```
|
||||
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT` does the same when the file does not say. With
|
||||
neither, nothing is traced and nothing is sent: every span is a no-op, and a
|
||||
command to a node carries no trace context.
|
||||
|
||||
What is traced:
|
||||
|
||||
| Trace | Spans |
|
||||
|---|---|
|
||||
| An inference request (`llmgw-server`) | `inference /v1/…` with `admission`, `schedule` and an `upstream` per attempt beneath it; a caller's `traceparent` is continued and the engine is sent one. |
|
||||
| A reconcile pass that did something | `reconcile`, and beneath it what it did: `scale up` with `place` (the profile, GPU class, pool, fabric and nodes chosen, or every candidate turned down and why) and `replica start`; `scale down`; `drain replica` and `stop replica`, each with its reason; `evict`, `preempt` with its victims; `reap replica` with the error and when it is tried again; `restart replica`, `restart refused`; `node unreachable`, `node lost`. A pass that found nothing to do is not traced. |
|
||||
| A replica's start | `replica start` stays open from placement until the replica serves, fails or is stopped, with each shard's progress as an event on it and how long it took. A `send start` per shard beneath it is the command to the node, and the node's spans (`llmgw-agent`) continue the same trace: `start shard`, with `fetch weights` (local, cached, downloaded or fetched by another shard, and a `download` with its bytes), `launch engine` (and `wait for rendezvous` on a worker of a multi-node replica) and `wait for engine` (how often the engine was asked, and its last answer if it never came up). |
|
||||
| A stop on a node | `stop shard` and `stop engine` in the trace of the control plane's `stop replica`, saying whether the engine had to be killed. |
|
||||
| An adoption | `adopt`, when a node that registers reports engines already running, with how many nobody claimed. |
|
||||
| A model job | `serve model` or `update model`, with a span per step — `registry`, `mirror`, `check`, `catalog`, `placement`, `smoke test`, `replicas` — and from the placement step a link to the replica's own start. |
|
||||
| An engine install | `install engine` on the node, a trace of its own. |
|
||||
|
||||
So "why did qwen take four minutes to start" is one trace: find `replica start`
|
||||
with `llmgw.model` = qwen, and the node's spans beneath it say whether the
|
||||
time went to the download, the launch or the engine loading.
|
||||
|
||||
A node follows the control plane's decision: a start the control plane kept is
|
||||
traced on the node whatever the agent's own `sample_ratio`, and one it sampled
|
||||
out is not; the agent's ratio decides only for what arrives with no trace, as
|
||||
from a control plane that is not tracing. The agent needs its own `tracing`
|
||||
block either way — without one its spans go nowhere, and the control plane's
|
||||
trace simply ends at `send start`.
|
||||
|
||||
## PostgreSQL, when SQLite is no longer the right shape
|
||||
|
||||
SQLite is the default and stays the right answer for most installations: one
|
||||
|
||||
Reference in new issue
Block a user