Release v0.10.4: CPU threads, tracing, and the rest of the review

CPU replicas start on half the machine's CPUs, not all of them: two busy
llama.cpp servers each spinning on all 8 vCPUs fell to 0.8 tok/s apiece,
where at 4 threads they keep 5.5. llamacpp.threads sets it per node.

Inference requests can be traced over OTLP/HTTP (tracing.endpoint), with
admission, schedule and upstream spans and traceparent sent to the engine.

Security: onboarding no longer puts the cluster root in the system trust
store and removes it where it was; the mirror's mutual-TLS API can be
limited to named certificates (--api-client); init.log no longer keeps the
administrator's password; a warning when auth is off on a public listen;
the recorded actor is sanitised and sudo-aware; Ray's dashboard is off, and
the remaining engine gaps are documented.

reconciler.go and agent.go are split by topic, the status mapping of each
failure is tested, and the plan's checklist is brought up to date.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-30 04:00:35 +02:00
1 parent adb1b3cf2a
commit 62657f4c2a
41 files changed
+2485 -1609

No files matched your search

+44
View File
@@ -81,6 +81,7 @@ rather than repeated.
| `notice` | `string` | [Landing](#landing) |
| `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
| `parallel` | `int` | [LlamaCppConfig](#llamacppconfig) |
| `threads` | `int` | [LlamaCppConfig](#llamacppconfig) |
| `api` | `string` | [Mirror](#mirror) |
| `tls` | `tlsconf.Config` | [Mirror](#mirror) |
| `registry` | `string` | [Mirror](#mirror) |
@@ -167,6 +168,7 @@ rather than repeated.
| `landing` | `Landing` | [Server](#server) |
| `agent_tls` | `tlsconf.Config` | [Server](#server) |
| `monitoring` | `Monitoring` | [Server](#server) |
| `tracing` | `Tracing` | [Server](#server) |
| `mirror` | `Mirror` | [Server](#server) |
| `engines` | `map[string]string` | [Server](#server) |
| `backup_dir` | `string` | [Server](#server) |
@@ -184,6 +186,8 @@ rather than repeated.
| `first_token` | `time.Duration` | [Timeouts](#timeouts) |
| `idle` | `time.Duration` | [Timeouts](#timeouts) |
| `total` | `time.Duration` | [Timeouts](#timeouts) |
| `endpoint` | `string` | [Tracing](#tracing) |
| `sample_ratio` | `float64` | [Tracing](#tracing) |
| `bin` | `string` | [VLLMConfig](#vllmconfig) |
| `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
| `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
@@ -634,6 +638,20 @@ Parallel is how many requests a server handles at once, when a profile
does not say. llama.cpp defaults to one, which would make every second
request wait behind the first no matter what the router decided.
### `threads`
`int`
Threads is how many CPU threads a shard with no GPU computes on. Zero
takes half this machine's CPUs.
Not all of them, which is llama.cpp's own default: two busy servers
that each spin on every CPU fight for them, and on an 8-vCPU node
that made 7B generation fall from 10 tokens a second to under 1 each.
At half, one server alone is about a third slower and two together
keep 5.5 each. A node that only ever runs one CPU replica does better
with all of them; say so here.
## Mirror
### `api`
@@ -1246,6 +1264,16 @@ Monitoring is where the dashboard reads its history from.
Takes a [Monitoring](#monitoring) block.
### `tracing`
`Tracing`
Tracing sends a trace of each inference request to an OpenTelemetry
collector. Off unless an endpoint is set here or in
OTEL_EXPORTER_OTLP_ENDPOINT.
Takes a [Tracing](#tracing) block.
### `mirror`
`Mirror`
@@ -1366,6 +1394,22 @@ about a hundred tokens on a slow model.
`time.Duration`
## Tracing
### `endpoint`
`string`
Endpoint is the collector, for example http://otel.internal:4318.
Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
### `sample_ratio`
`float64`
SampleRatio is the share of requests traced, 0 to 1, when the caller
did not decide it with a sampled traceparent of its own. Zero takes 1.
## VLLMConfig
### `bin`