diff --git a/llmgw-plan.md b/llmgw-plan.md index 4165b017..bf545c39 100644 --- a/llmgw-plan.md +++ b/llmgw-plan.md @@ -1286,6 +1286,9 @@ work_dir: /var/lib/llmgw-agent ## 11. Roadmap +`[x]` is built and shown to work, `[ ]` is not built, and `[~]` is built +but not yet run where it matters — a real engine, real hardware. + ### Phase 0 — Skeleton (1 week) - [x] `go mod init`, cobra commands, koanf config, slog, Makefile, golangci-lint - [x] `sim-engine`: OpenAI-compatible fake engine (tunable latency, token rate, queue, vLLM-format `/metrics`) @@ -1476,7 +1479,10 @@ The launch path is now built too: `docs/reference.md` generated from the configuration structs ### Phase 6 — v2 ideas -- [ ] Precise KV index (§7.5 tier 2) — vLLM KV events plus tokenizer integration +- [~] Precise KV index (§7.5 tier 2) — vLLM KV events plus tokenizer integration. + Built in v0.11.2 (`kv_events` on a profile, the agent forwarding block + changes, the router's index and scorer); never run against a real vLLM, + which is what is left (BACKLOG C24) - [ ] Dynamo-style block cost function, tuning the `softmax` picker - [x] Admission displacement: a total queue bound, and the newest request of the lowest band turned away (503, Retry-After) for a higher one @@ -1489,17 +1495,23 @@ The launch path is now built too: - [x] CPU cores reserved like memory, and memory required of CPU models - [x] Node sensors collected: GPU temperature, power and throttling, CPU temperature and RAPL power, board inlet air; published and shown -- [ ] Rack power, temperature and cooling as placement signals: planned in - [docs/facility-aware-placement.md](docs/facility-aware-placement.md), - deferred — it needs each site's rack, PDU and power data first +- [~] Rack power, temperature and cooling as placement signals: planned in + [docs/facility-aware-placement.md](docs/facility-aware-placement.md). + Built in v0.11.2: location labels, spreading across racks and PDUs, + `keep_within`, held domains, power caps from `facility.budgets`. Left: + temperature and cooling, which need each site's sensor data - [ ] Defragmentation for large tensor-parallel replicas across NVLink domains -- [ ] Control plane HA (Raft) +- [x] Control plane HA, by leader election on PostgreSQL (v0.8.0) and from + setup (v0.11.0) rather than by Raft: the store is replicated already, so + a second consensus layer would only have to agree with it - [x] Weight downloading and cache management: the mirror, agent fetches checked against their digest, and reclaiming what nothing uses - [ ] Prefill/decode disaggregation (§12) - [x] LoRA adapters on llama.cpp: every replica loads a model's adapters and applies one per request, so any replica serves any adapter -- [ ] LoRA affinity for engines that load adapters on demand (vLLM) +- [~] LoRA affinity for engines that load adapters on demand (vLLM): built in + v0.11.2 (`max_loaded_adapters`, the router's adapter tracker, the + affinity filter); never run against a real vLLM - [x] External provider support: a catalog model served by an OpenAI-compatible API elsewhere, usable as a fallback, under the same allowlists and budgets - [x] Read-only dashboard on the admin listener (nodes, replicas, models, @@ -1511,8 +1523,9 @@ The launch path is now built too: promoting, resetting passwords and issuing keys stay on the CLI - [x] The dashboard reachable over the web at `/admin`, behind a session and an admin flag, without exposing the listener that carries pprof -- [ ] An admin UI that can *change* things, which needs roles this deliberately - does not have +- [x] An admin UI that can *change* things: staff roles (viewer, operator, + billing, admin) and the audit log in v0.11.2, the cluster's settings on + the portal in v0.14.1 ---