Record the llama.cpp work in the plan and changelog

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-16 08:19:55 +02:00
1 parent f19db242ae
commit f84e3b642a
2 files changed
+56 -3

No files matched your search

+49 -1
View File
@@ -8,7 +8,7 @@ is only released once its **exit criterion** is met, not when its code lands.
| v0.1.0 | 0 | Skeleton and sim-engine | released |
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released |
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | released |
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | in progress |
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | released as part of v0.5.0 |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | planned |
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | planned |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
@@ -16,6 +16,54 @@ is only released once its **exit criterion** is met, not when its code lands.
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased
### llama.cpp, and CPU profiles
A third engine adapter, and the reason for it: llama.cpp serves the same OpenAI
API without a GPU, so placement, supervision, scraping, scheduling and admission
can all be exercised on ordinary machines.
A profile may now omit `gpu_class`, which is how it says it runs on the CPU. Only
engines that can actually serve without a card may omit it — for vLLM an omitted
class stays an error, because otherwise a typo becomes a replica placed on a
machine with no hardware and diagnosed minutes later during model loading. CPU
replicas are spread across nodes rather than bin packed: unlike GPUs there is no
allocation to stop two of them sharing a machine's cores.
### Metric mappings are per engine
`Running`, `Waiting` and `KVCacheUsage` now have a vocabulary per engine rather
than one for the cluster. A mixed fleet has two names for the same three numbers,
and a single global mapping left half of it unmeasurable.
A mapping that leaves a metric unnamed now means the engine does not publish it,
and the field stays zero. Naming one the engine does not serve is still a contract
violation. `Running` and `Waiting` can never be omitted.
### Mutual TLS on the agent stream
A private CA (`llmgw ctl cert init-ca | issue-server | issue-client | inspect`),
certificates on both ends, and the control plane refusing any agent whose claimed
node id does not match its certificate's common name. Public authorities cannot
supply these: they stopped issuing the `clientAuth` usage.
Plaintext HTTP/2 remains the default. `agent_tls:` on the server and `tls:` on the
agent turn it on.
### Fixes
- **An agent told to stop did not stop.** Cancelling the context is not enough —
HTTP/2 watches it while the request body is being written, and a bidirectional
stream spends its life blocked on the read side. Measured on a deployed node:
`systemctl stop` took 90 seconds and ended in SIGKILL. Now 0.01 seconds.
- **An agent could not explain why it was refused.** The session cancelled its own
context and then asked that context whether it was shutting down, so every
stream error read as a clean close: no message, and the backoff reset.
- **A broken metric contract was invisible.** It was logged per scrape at debug
level, so a cluster could look healthy while every routing decision was made
blind. It is now a warning, once, with a matching line when it clears.
## v0.5.0 — 2026-09-16
Phases 2 and 3. The cluster runs itself: agents report their hardware, the
+7 -2
View File
@@ -1282,6 +1282,11 @@ work_dir: /var/lib/llmgw-agent
### Phase 2 — Agent: discovery and single-node engines (2 weeks)
- [x] GPU discovery (nvidia-smi path), IB discovery from sysfs
- [x] Engine interface, vLLM and SGLang command and environment construction
- [x] llama.cpp adapter, and with it CPU profiles: a profile may omit its GPU class,
and only an engine that can serve without a card may do so. This is what
makes the whole path testable on machines that have no GPUs — verified
end to end on four VMs serving Qwen2.5-0.5B on CPU at ~50 tok/s, twelve
concurrent requests balancing 4/4/4 across three replicas.
- [x] Supervisor (process groups, log buffer, state machine, orphan adoption)
- [x] NVML backend behind the `nvml` build tag, with NVLink topology. It compiles
in CI but has not run against a driver: **verify on a GPU node before
@@ -1358,8 +1363,8 @@ The launch path is now built too:
stub processes. The Ray sequencing, and whether the deployed vLLM version
wants `ray` or `mp`, can only be confirmed on real hardware.
- **Exit criterion:** a 2-node replica on each fabric; killing a node rebuilds the replica on the same fabric and never spreads it across two.
- **Exit criterion:** a 2-node replica on each fabric; killing a node rebuilds the replica on the same fabric and never spreads it to the other one.
- **Exit criterion:** a 2-node replica on each fabric; killing a node rebuilds the replica on the
same fabric and never spreads it across two.
### Phase 5 — Production features (3 weeks)
- [ ] **SQLite store and migrations.** It arrives here rather than in phase 3