The plan's phase 5 says what is built, what is built but unproven, and what is not
CI / test (push) Failing after 6m22s
CI / test (push) Failing after 6m22s
The precise KV index and vLLM's LoRA affinity were built in v0.11.2 and never run against a real vLLM; facility-aware placement in v0.11.2 less temperature and cooling: marked [~], a mark now explained at the top of the roadmap. Control-plane HA is done, by leader election on PostgreSQL rather than Raft, and the admin UI changes things since staff roles (v0.11.2) and the settings page (v0.14.1). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
cf41adbf25
commit
41601bc5ef
1 file changed
+21
-8
+21
-8
@@ -1286,6 +1286,9 @@ work_dir: /var/lib/llmgw-agent
|
|||||||
|
|
||||||
## 11. Roadmap
|
## 11. Roadmap
|
||||||
|
|
||||||
|
`[x]` is built and shown to work, `[ ]` is not built, and `[~]` is built
|
||||||
|
but not yet run where it matters — a real engine, real hardware.
|
||||||
|
|
||||||
### Phase 0 — Skeleton (1 week)
|
### Phase 0 — Skeleton (1 week)
|
||||||
- [x] `go mod init`, cobra commands, koanf config, slog, Makefile, golangci-lint
|
- [x] `go mod init`, cobra commands, koanf config, slog, Makefile, golangci-lint
|
||||||
- [x] `sim-engine`: OpenAI-compatible fake engine (tunable latency, token rate, queue, vLLM-format `/metrics`)
|
- [x] `sim-engine`: OpenAI-compatible fake engine (tunable latency, token rate, queue, vLLM-format `/metrics`)
|
||||||
@@ -1476,7 +1479,10 @@ The launch path is now built too:
|
|||||||
`docs/reference.md` generated from the configuration structs
|
`docs/reference.md` generated from the configuration structs
|
||||||
|
|
||||||
### Phase 6 — v2 ideas
|
### Phase 6 — v2 ideas
|
||||||
- [ ] Precise KV index (§7.5 tier 2) — vLLM KV events plus tokenizer integration
|
- [~] Precise KV index (§7.5 tier 2) — vLLM KV events plus tokenizer integration.
|
||||||
|
Built in v0.11.2 (`kv_events` on a profile, the agent forwarding block
|
||||||
|
changes, the router's index and scorer); never run against a real vLLM,
|
||||||
|
which is what is left (BACKLOG C24)
|
||||||
- [ ] Dynamo-style block cost function, tuning the `softmax` picker
|
- [ ] Dynamo-style block cost function, tuning the `softmax` picker
|
||||||
- [x] Admission displacement: a total queue bound, and the newest request of
|
- [x] Admission displacement: a total queue bound, and the newest request of
|
||||||
the lowest band turned away (503, Retry-After) for a higher one
|
the lowest band turned away (503, Retry-After) for a higher one
|
||||||
@@ -1489,17 +1495,23 @@ The launch path is now built too:
|
|||||||
- [x] CPU cores reserved like memory, and memory required of CPU models
|
- [x] CPU cores reserved like memory, and memory required of CPU models
|
||||||
- [x] Node sensors collected: GPU temperature, power and throttling, CPU
|
- [x] Node sensors collected: GPU temperature, power and throttling, CPU
|
||||||
temperature and RAPL power, board inlet air; published and shown
|
temperature and RAPL power, board inlet air; published and shown
|
||||||
- [ ] Rack power, temperature and cooling as placement signals: planned in
|
- [~] Rack power, temperature and cooling as placement signals: planned in
|
||||||
[docs/facility-aware-placement.md](docs/facility-aware-placement.md),
|
[docs/facility-aware-placement.md](docs/facility-aware-placement.md).
|
||||||
deferred — it needs each site's rack, PDU and power data first
|
Built in v0.11.2: location labels, spreading across racks and PDUs,
|
||||||
|
`keep_within`, held domains, power caps from `facility.budgets`. Left:
|
||||||
|
temperature and cooling, which need each site's sensor data
|
||||||
- [ ] Defragmentation for large tensor-parallel replicas across NVLink domains
|
- [ ] Defragmentation for large tensor-parallel replicas across NVLink domains
|
||||||
- [ ] Control plane HA (Raft)
|
- [x] Control plane HA, by leader election on PostgreSQL (v0.8.0) and from
|
||||||
|
setup (v0.11.0) rather than by Raft: the store is replicated already, so
|
||||||
|
a second consensus layer would only have to agree with it
|
||||||
- [x] Weight downloading and cache management: the mirror, agent fetches
|
- [x] Weight downloading and cache management: the mirror, agent fetches
|
||||||
checked against their digest, and reclaiming what nothing uses
|
checked against their digest, and reclaiming what nothing uses
|
||||||
- [ ] Prefill/decode disaggregation (§12)
|
- [ ] Prefill/decode disaggregation (§12)
|
||||||
- [x] LoRA adapters on llama.cpp: every replica loads a model's adapters and
|
- [x] LoRA adapters on llama.cpp: every replica loads a model's adapters and
|
||||||
applies one per request, so any replica serves any adapter
|
applies one per request, so any replica serves any adapter
|
||||||
- [ ] LoRA affinity for engines that load adapters on demand (vLLM)
|
- [~] LoRA affinity for engines that load adapters on demand (vLLM): built in
|
||||||
|
v0.11.2 (`max_loaded_adapters`, the router's adapter tracker, the
|
||||||
|
affinity filter); never run against a real vLLM
|
||||||
- [x] External provider support: a catalog model served by an OpenAI-compatible
|
- [x] External provider support: a catalog model served by an OpenAI-compatible
|
||||||
API elsewhere, usable as a fallback, under the same allowlists and budgets
|
API elsewhere, usable as a fallback, under the same allowlists and budgets
|
||||||
- [x] Read-only dashboard on the admin listener (nodes, replicas, models,
|
- [x] Read-only dashboard on the admin listener (nodes, replicas, models,
|
||||||
@@ -1511,8 +1523,9 @@ The launch path is now built too:
|
|||||||
promoting, resetting passwords and issuing keys stay on the CLI
|
promoting, resetting passwords and issuing keys stay on the CLI
|
||||||
- [x] The dashboard reachable over the web at `/admin`, behind a session and an
|
- [x] The dashboard reachable over the web at `/admin`, behind a session and an
|
||||||
admin flag, without exposing the listener that carries pprof
|
admin flag, without exposing the listener that carries pprof
|
||||||
- [ ] An admin UI that can *change* things, which needs roles this deliberately
|
- [x] An admin UI that can *change* things: staff roles (viewer, operator,
|
||||||
does not have
|
billing, admin) and the audit log in v0.11.2, the cluster's settings on
|
||||||
|
the portal in v0.14.1
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user