Record the llama.cpp work in the plan and changelog
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
f19db242ae
commit
f84e3b642a
2 files changed
+56
-3
No files matched your search
+49
-1
@@ -8,7 +8,7 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
||||
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released |
|
||||
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | released |
|
||||
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | in progress |
|
||||
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | released as part of v0.5.0 |
|
||||
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | planned |
|
||||
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | planned |
|
||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||
@@ -16,6 +16,54 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
### llama.cpp, and CPU profiles
|
||||
|
||||
A third engine adapter, and the reason for it: llama.cpp serves the same OpenAI
|
||||
API without a GPU, so placement, supervision, scraping, scheduling and admission
|
||||
can all be exercised on ordinary machines.
|
||||
|
||||
A profile may now omit `gpu_class`, which is how it says it runs on the CPU. Only
|
||||
engines that can actually serve without a card may omit it — for vLLM an omitted
|
||||
class stays an error, because otherwise a typo becomes a replica placed on a
|
||||
machine with no hardware and diagnosed minutes later during model loading. CPU
|
||||
replicas are spread across nodes rather than bin packed: unlike GPUs there is no
|
||||
allocation to stop two of them sharing a machine's cores.
|
||||
|
||||
### Metric mappings are per engine
|
||||
|
||||
`Running`, `Waiting` and `KVCacheUsage` now have a vocabulary per engine rather
|
||||
than one for the cluster. A mixed fleet has two names for the same three numbers,
|
||||
and a single global mapping left half of it unmeasurable.
|
||||
|
||||
A mapping that leaves a metric unnamed now means the engine does not publish it,
|
||||
and the field stays zero. Naming one the engine does not serve is still a contract
|
||||
violation. `Running` and `Waiting` can never be omitted.
|
||||
|
||||
### Mutual TLS on the agent stream
|
||||
|
||||
A private CA (`llmgw ctl cert init-ca | issue-server | issue-client | inspect`),
|
||||
certificates on both ends, and the control plane refusing any agent whose claimed
|
||||
node id does not match its certificate's common name. Public authorities cannot
|
||||
supply these: they stopped issuing the `clientAuth` usage.
|
||||
|
||||
Plaintext HTTP/2 remains the default. `agent_tls:` on the server and `tls:` on the
|
||||
agent turn it on.
|
||||
|
||||
### Fixes
|
||||
|
||||
- **An agent told to stop did not stop.** Cancelling the context is not enough —
|
||||
HTTP/2 watches it while the request body is being written, and a bidirectional
|
||||
stream spends its life blocked on the read side. Measured on a deployed node:
|
||||
`systemctl stop` took 90 seconds and ended in SIGKILL. Now 0.01 seconds.
|
||||
- **An agent could not explain why it was refused.** The session cancelled its own
|
||||
context and then asked that context whether it was shutting down, so every
|
||||
stream error read as a clean close: no message, and the backoff reset.
|
||||
- **A broken metric contract was invisible.** It was logged per scrape at debug
|
||||
level, so a cluster could look healthy while every routing decision was made
|
||||
blind. It is now a warning, once, with a matching line when it clears.
|
||||
|
||||
## v0.5.0 — 2026-09-16
|
||||
|
||||
Phases 2 and 3. The cluster runs itself: agents report their hardware, the
|
||||
|
||||
+7
-2
@@ -1282,6 +1282,11 @@ work_dir: /var/lib/llmgw-agent
|
||||
### Phase 2 — Agent: discovery and single-node engines (2 weeks)
|
||||
- [x] GPU discovery (nvidia-smi path), IB discovery from sysfs
|
||||
- [x] Engine interface, vLLM and SGLang command and environment construction
|
||||
- [x] llama.cpp adapter, and with it CPU profiles: a profile may omit its GPU class,
|
||||
and only an engine that can serve without a card may do so. This is what
|
||||
makes the whole path testable on machines that have no GPUs — verified
|
||||
end to end on four VMs serving Qwen2.5-0.5B on CPU at ~50 tok/s, twelve
|
||||
concurrent requests balancing 4/4/4 across three replicas.
|
||||
- [x] Supervisor (process groups, log buffer, state machine, orphan adoption)
|
||||
- [x] NVML backend behind the `nvml` build tag, with NVLink topology. It compiles
|
||||
in CI but has not run against a driver: **verify on a GPU node before
|
||||
@@ -1358,8 +1363,8 @@ The launch path is now built too:
|
||||
stub processes. The Ray sequencing, and whether the deployed vLLM version
|
||||
wants `ray` or `mp`, can only be confirmed on real hardware.
|
||||
|
||||
- **Exit criterion:** a 2-node replica on each fabric; killing a node rebuilds the replica on the same fabric and never spreads it across two.
|
||||
- **Exit criterion:** a 2-node replica on each fabric; killing a node rebuilds the replica on the same fabric and never spreads it to the other one.
|
||||
- **Exit criterion:** a 2-node replica on each fabric; killing a node rebuilds the replica on the
|
||||
same fabric and never spreads it across two.
|
||||
|
||||
### Phase 5 — Production features (3 weeks)
|
||||
- [ ] **SQLite store and migrations.** It arrives here rather than in phase 3
|
||||
|
||||
Reference in new issue
Block a user