diff --git a/CHANGELOG.md b/CHANGELOG.md index 38e93cb..e57f57a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ is only released once its **exit criterion** is met, not when its code lands. | v0.1.0 | 0 | Skeleton and sim-engine | released | | v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released | | v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | released | -| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | in progress | +| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | released as part of v0.5.0 | | v0.5.0 | 3 | Control plane: catalog, placement, reconciler | planned | | v0.6.0 | 4 | Multi-node replicas and fabric awareness | planned | | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned | @@ -16,6 +16,54 @@ is only released once its **exit criterion** is met, not when its code lands. Until v1.0.0 the minor version is the breaking one: configuration formats and Go APIs change between minors without a deprecation period. +## Unreleased + +### llama.cpp, and CPU profiles + +A third engine adapter, and the reason for it: llama.cpp serves the same OpenAI +API without a GPU, so placement, supervision, scraping, scheduling and admission +can all be exercised on ordinary machines. + +A profile may now omit `gpu_class`, which is how it says it runs on the CPU. Only +engines that can actually serve without a card may omit it — for vLLM an omitted +class stays an error, because otherwise a typo becomes a replica placed on a +machine with no hardware and diagnosed minutes later during model loading. CPU +replicas are spread across nodes rather than bin packed: unlike GPUs there is no +allocation to stop two of them sharing a machine's cores. + +### Metric mappings are per engine + +`Running`, `Waiting` and `KVCacheUsage` now have a vocabulary per engine rather +than one for the cluster. A mixed fleet has two names for the same three numbers, +and a single global mapping left half of it unmeasurable. + +A mapping that leaves a metric unnamed now means the engine does not publish it, +and the field stays zero. Naming one the engine does not serve is still a contract +violation. `Running` and `Waiting` can never be omitted. + +### Mutual TLS on the agent stream + +A private CA (`llmgw ctl cert init-ca | issue-server | issue-client | inspect`), +certificates on both ends, and the control plane refusing any agent whose claimed +node id does not match its certificate's common name. Public authorities cannot +supply these: they stopped issuing the `clientAuth` usage. + +Plaintext HTTP/2 remains the default. `agent_tls:` on the server and `tls:` on the +agent turn it on. + +### Fixes + +- **An agent told to stop did not stop.** Cancelling the context is not enough — + HTTP/2 watches it while the request body is being written, and a bidirectional + stream spends its life blocked on the read side. Measured on a deployed node: + `systemctl stop` took 90 seconds and ended in SIGKILL. Now 0.01 seconds. +- **An agent could not explain why it was refused.** The session cancelled its own + context and then asked that context whether it was shutting down, so every + stream error read as a clean close: no message, and the backoff reset. +- **A broken metric contract was invisible.** It was logged per scrape at debug + level, so a cluster could look healthy while every routing decision was made + blind. It is now a warning, once, with a matching line when it clears. + ## v0.5.0 — 2026-09-16 Phases 2 and 3. The cluster runs itself: agents report their hardware, the diff --git a/llmgw-plan.md b/llmgw-plan.md index 00f3ae0..6750bac 100644 --- a/llmgw-plan.md +++ b/llmgw-plan.md @@ -1282,6 +1282,11 @@ work_dir: /var/lib/llmgw-agent ### Phase 2 — Agent: discovery and single-node engines (2 weeks) - [x] GPU discovery (nvidia-smi path), IB discovery from sysfs - [x] Engine interface, vLLM and SGLang command and environment construction +- [x] llama.cpp adapter, and with it CPU profiles: a profile may omit its GPU class, + and only an engine that can serve without a card may do so. This is what + makes the whole path testable on machines that have no GPUs — verified + end to end on four VMs serving Qwen2.5-0.5B on CPU at ~50 tok/s, twelve + concurrent requests balancing 4/4/4 across three replicas. - [x] Supervisor (process groups, log buffer, state machine, orphan adoption) - [x] NVML backend behind the `nvml` build tag, with NVLink topology. It compiles in CI but has not run against a driver: **verify on a GPU node before @@ -1358,8 +1363,8 @@ The launch path is now built too: stub processes. The Ray sequencing, and whether the deployed vLLM version wants `ray` or `mp`, can only be confirmed on real hardware. -- **Exit criterion:** a 2-node replica on each fabric; killing a node rebuilds the replica on the same fabric and never spreads it across two. -- **Exit criterion:** a 2-node replica on each fabric; killing a node rebuilds the replica on the same fabric and never spreads it to the other one. +- **Exit criterion:** a 2-node replica on each fabric; killing a node rebuilds the replica on the + same fabric and never spreads it across two. ### Phase 5 — Production features (3 weeks) - [ ] **SQLite store and migrations.** It arrives here rather than in phase 3