Merge C20 tracing beyond the router
Conflicts: config imports and docgen sources (both kept); agent.pb.go regenerated with buf from the merged proto (MIG devices and trace context). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
801e2d51c0
33 files changed
+1608
-144
No files matched your search
@@ -1272,6 +1272,46 @@ same event about the same thing is sent at most once an hour, so a budget that
|
||||
refuses a thousand requests is one message. `llmgw ctl notify test` sends a
|
||||
test event to each webhook and says how each answered.
|
||||
|
||||
### Tracing
|
||||
|
||||
Metrics say that a model took four minutes to start; a trace says where they
|
||||
went. Give the control plane and every agent the same OpenTelemetry collector,
|
||||
over OTLP/HTTP — Jaeger, Tempo, Honeycomb, anything that takes OTLP:
|
||||
|
||||
```yaml
|
||||
# /etc/llmgw/server.yaml, and the same block in each node's agent.yaml
|
||||
tracing:
|
||||
endpoint: http://otel.internal:4318
|
||||
sample_ratio: 1 # of requests that arrive without a decision of their own
|
||||
```
|
||||
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT` does the same when the file does not say. With
|
||||
neither, nothing is traced and nothing is sent: every span is a no-op, and a
|
||||
command to a node carries no trace context.
|
||||
|
||||
What is traced:
|
||||
|
||||
| Trace | Spans |
|
||||
|---|---|
|
||||
| An inference request (`llmgw-server`) | `inference /v1/…` with `admission`, `schedule` and an `upstream` per attempt beneath it; a caller's `traceparent` is continued and the engine is sent one. |
|
||||
| A reconcile pass that did something | `reconcile`, and beneath it what it did: `scale up` with `place` (the profile, GPU class, pool, fabric and nodes chosen, or every candidate turned down and why) and `replica start`; `scale down`; `drain replica` and `stop replica`, each with its reason; `evict`, `preempt` with its victims; `reap replica` with the error and when it is tried again; `restart replica`, `restart refused`; `node unreachable`, `node lost`. A pass that found nothing to do is not traced. |
|
||||
| A replica's start | `replica start` stays open from placement until the replica serves, fails or is stopped, with each shard's progress as an event on it and how long it took. A `send start` per shard beneath it is the command to the node, and the node's spans (`llmgw-agent`) continue the same trace: `start shard`, with `fetch weights` (local, cached, downloaded or fetched by another shard, and a `download` with its bytes), `launch engine` (and `wait for rendezvous` on a worker of a multi-node replica) and `wait for engine` (how often the engine was asked, and its last answer if it never came up). |
|
||||
| A stop on a node | `stop shard` and `stop engine` in the trace of the control plane's `stop replica`, saying whether the engine had to be killed. |
|
||||
| An adoption | `adopt`, when a node that registers reports engines already running, with how many nobody claimed. |
|
||||
| A model job | `serve model` or `update model`, with a span per step — `registry`, `mirror`, `check`, `catalog`, `placement`, `smoke test`, `replicas` — and from the placement step a link to the replica's own start. |
|
||||
| An engine install | `install engine` on the node, a trace of its own. |
|
||||
|
||||
So "why did qwen take four minutes to start" is one trace: find `replica start`
|
||||
with `llmgw.model` = qwen, and the node's spans beneath it say whether the
|
||||
time went to the download, the launch or the engine loading.
|
||||
|
||||
A node follows the control plane's decision: a start the control plane kept is
|
||||
traced on the node whatever the agent's own `sample_ratio`, and one it sampled
|
||||
out is not; the agent's ratio decides only for what arrives with no trace, as
|
||||
from a control plane that is not tracing. The agent needs its own `tracing`
|
||||
block either way — without one its spans go nowhere, and the control plane's
|
||||
trace simply ends at `send start`.
|
||||
|
||||
## PostgreSQL, when SQLite is no longer the right shape
|
||||
|
||||
SQLite is the default and stays the right answer for most installations: one
|
||||
|
||||
+22
-6
@@ -250,7 +250,7 @@ rather than repeated.
|
||||
| `debug_pprof` | `bool` | [Server](#server) |
|
||||
| `agent_insecure` | `bool` | [Server](#server) |
|
||||
| `monitoring` | `Monitoring` | [Server](#server) |
|
||||
| `tracing` | `Tracing` | [Server](#server) |
|
||||
| `tracing` | `obs.Tracing` | [Server](#server) |
|
||||
| `mirror` | `Mirror` | [Server](#server) |
|
||||
| `engines` | `map[string]string` | [Server](#server) |
|
||||
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
|
||||
@@ -327,6 +327,7 @@ rather than repeated.
|
||||
| `discovery` | `discovery.Options` | [agent.Config](#agentconfig) |
|
||||
| `engines` | `engines.Config` | [agent.Config](#agentconfig) |
|
||||
| `supervisor` | `supervisor.Config` | [agent.Config](#agentconfig) |
|
||||
| `tracing` | `obs.Tracing` | [agent.Config](#agentconfig) |
|
||||
| `name` | `string` | [config.Band](#configband) |
|
||||
| `priority` | `int` | [config.Band](#configband) |
|
||||
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
|
||||
@@ -2063,10 +2064,11 @@ Takes a [Monitoring](#monitoring) block.
|
||||
|
||||
### `tracing`
|
||||
|
||||
`Tracing`
|
||||
`obs.Tracing`
|
||||
|
||||
Tracing sends a trace of each inference request to an OpenTelemetry
|
||||
collector. Off unless an endpoint is set here or in
|
||||
Tracing sends traces to an OpenTelemetry collector: each inference
|
||||
request, and on a control plane each placement, replica start and
|
||||
model job. Off unless an endpoint is set here or in
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT.
|
||||
|
||||
Takes a [Tracing](#tracing) block.
|
||||
@@ -2331,8 +2333,10 @@ Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
|
||||
|
||||
`float64`
|
||||
|
||||
SampleRatio is the share of requests traced, 0 to 1, when the caller
|
||||
did not decide it with a sampled traceparent of its own. Zero takes 1.
|
||||
SampleRatio is the share of traces kept, 0 to 1, of those nobody
|
||||
upstream already decided. Zero takes 1. A request with a sampled
|
||||
traceparent of its own is kept, and an agent keeps every start the
|
||||
control plane kept, whatever their own ratio says.
|
||||
|
||||
## VLLMConfig
|
||||
|
||||
@@ -2639,6 +2643,18 @@ Takes a [engines.Config](#enginesconfig) block.
|
||||
|
||||
Takes a [supervisor.Config](#supervisorconfig) block.
|
||||
|
||||
### `tracing`
|
||||
|
||||
`obs.Tracing`
|
||||
|
||||
Tracing sends this node's part of each replica start — the weights
|
||||
fetched, the engine launched, the wait for it to answer — to an
|
||||
OpenTelemetry collector, inside the control plane's trace of that
|
||||
start. Give it the collector server.yaml's tracing names. Off unless
|
||||
an endpoint is set here or in OTEL_EXPORTER_OTLP_ENDPOINT.
|
||||
|
||||
Takes a [Tracing](#tracing) block.
|
||||
|
||||
## config.Band
|
||||
|
||||
### `name`
|
||||
|
||||
Reference in new issue
Block a user