Release v0.10.4: CPU threads, tracing, and the rest of the review
CPU replicas start on half the machine's CPUs, not all of them: two busy llama.cpp servers each spinning on all 8 vCPUs fell to 0.8 tok/s apiece, where at 4 threads they keep 5.5. llamacpp.threads sets it per node. Inference requests can be traced over OTLP/HTTP (tracing.endpoint), with admission, schedule and upstream spans and traceparent sent to the engine. Security: onboarding no longer puts the cluster root in the system trust store and removes it where it was; the mirror's mutual-TLS API can be limited to named certificates (--api-client); init.log no longer keeps the administrator's password; a warning when auth is off on a public listen; the recorded actor is sanitised and sudo-aware; Ray's dashboard is off, and the remaining engine gaps are documented. reconciler.go and agent.go are split by topic, the status mapping of each failure is tested, and the plan's checklist is brought up to date. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
adb1b3cf2a
commit
62657f4c2a
41 files changed
+2485
-1609
No files matched your search
@@ -1124,3 +1124,11 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
has them. Single-node placement on real GPUs is the well-trodden path; if you
|
||||
are the first to run a TP group across real cards, run `llmgw doctor --json`
|
||||
on each node first and keep the reports.
|
||||
- **Two engine setups are weaker than the rest.** An engine's key is passed
|
||||
in its environment, where only root and its own user can read it, except
|
||||
SGLang's, which takes it only as a flag: other users on the node can see it
|
||||
in the process list. Mount `/proc` with `hidepid=2` on nodes where that
|
||||
matters. vLLM's `launch_mode: ray` starts a Ray head without
|
||||
authentication, so anyone who can reach its port can run code on the
|
||||
node. The dashboard is off, but firewall the port to the other nodes in the
|
||||
replica, or use the default `mp` launch mode, which needs no Ray at all.
|
||||
@@ -81,6 +81,7 @@ rather than repeated.
|
||||
| `notice` | `string` | [Landing](#landing) |
|
||||
| `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
|
||||
| `parallel` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
||||
| `threads` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
||||
| `api` | `string` | [Mirror](#mirror) |
|
||||
| `tls` | `tlsconf.Config` | [Mirror](#mirror) |
|
||||
| `registry` | `string` | [Mirror](#mirror) |
|
||||
@@ -167,6 +168,7 @@ rather than repeated.
|
||||
| `landing` | `Landing` | [Server](#server) |
|
||||
| `agent_tls` | `tlsconf.Config` | [Server](#server) |
|
||||
| `monitoring` | `Monitoring` | [Server](#server) |
|
||||
| `tracing` | `Tracing` | [Server](#server) |
|
||||
| `mirror` | `Mirror` | [Server](#server) |
|
||||
| `engines` | `map[string]string` | [Server](#server) |
|
||||
| `backup_dir` | `string` | [Server](#server) |
|
||||
@@ -184,6 +186,8 @@ rather than repeated.
|
||||
| `first_token` | `time.Duration` | [Timeouts](#timeouts) |
|
||||
| `idle` | `time.Duration` | [Timeouts](#timeouts) |
|
||||
| `total` | `time.Duration` | [Timeouts](#timeouts) |
|
||||
| `endpoint` | `string` | [Tracing](#tracing) |
|
||||
| `sample_ratio` | `float64` | [Tracing](#tracing) |
|
||||
| `bin` | `string` | [VLLMConfig](#vllmconfig) |
|
||||
| `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
|
||||
| `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
|
||||
@@ -634,6 +638,20 @@ Parallel is how many requests a server handles at once, when a profile
|
||||
does not say. llama.cpp defaults to one, which would make every second
|
||||
request wait behind the first no matter what the router decided.
|
||||
|
||||
### `threads`
|
||||
|
||||
`int`
|
||||
|
||||
Threads is how many CPU threads a shard with no GPU computes on. Zero
|
||||
takes half this machine's CPUs.
|
||||
|
||||
Not all of them, which is llama.cpp's own default: two busy servers
|
||||
that each spin on every CPU fight for them, and on an 8-vCPU node
|
||||
that made 7B generation fall from 10 tokens a second to under 1 each.
|
||||
At half, one server alone is about a third slower and two together
|
||||
keep 5.5 each. A node that only ever runs one CPU replica does better
|
||||
with all of them; say so here.
|
||||
|
||||
## Mirror
|
||||
|
||||
### `api`
|
||||
@@ -1246,6 +1264,16 @@ Monitoring is where the dashboard reads its history from.
|
||||
|
||||
Takes a [Monitoring](#monitoring) block.
|
||||
|
||||
### `tracing`
|
||||
|
||||
`Tracing`
|
||||
|
||||
Tracing sends a trace of each inference request to an OpenTelemetry
|
||||
collector. Off unless an endpoint is set here or in
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT.
|
||||
|
||||
Takes a [Tracing](#tracing) block.
|
||||
|
||||
### `mirror`
|
||||
|
||||
`Mirror`
|
||||
@@ -1366,6 +1394,22 @@ about a hundred tokens on a slow model.
|
||||
|
||||
`time.Duration`
|
||||
|
||||
## Tracing
|
||||
|
||||
### `endpoint`
|
||||
|
||||
`string`
|
||||
|
||||
Endpoint is the collector, for example http://otel.internal:4318.
|
||||
Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
|
||||
|
||||
### `sample_ratio`
|
||||
|
||||
`float64`
|
||||
|
||||
SampleRatio is the share of requests traced, 0 to 1, when the caller
|
||||
did not decide it with a sampled traceparent of its own. Zero takes 1.
|
||||
|
||||
## VLLMConfig
|
||||
|
||||
### `bin`
|
||||
|
||||
Reference in new issue
Block a user