CHANGELOG and BACKLOG for the rest of C24 and the engine fixes; README
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
33334c8f33
commit
803d1d5fb6
8 files changed
+227
-206
No files matched your search
+59
-2
@@ -679,11 +679,11 @@ portal is not done — the page shows; the console and ctl change.
|
||||
**More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF
|
||||
import. `agent/engines/engine.go` interface and the metric contract in
|
||||
`router/scrape`.
|
||||
24. Lower: precise KV-index routing (plan phase 6), LoRA affinity for
|
||||
24. **Done (branch `features-c2`), all six parts.** Lower: precise KV-index routing (plan phase 6), LoRA affinity for
|
||||
vLLM, multi-cluster federation via External providers, facility-aware
|
||||
placement (`docs/facility-aware-placement.md`), portal i18n, SDK/docs
|
||||
portal.
|
||||
Done so far (branch `features-c2`):
|
||||
What was built, and what is left:
|
||||
- 24 (portal i18n): built — English/Turkish catalogue with fallback, per-
|
||||
language templates, cookie/Accept-Language/`landing.language` choice
|
||||
and a footer switcher, Turkish numbers and dates, every template and
|
||||
@@ -701,6 +701,63 @@ portal is not done — the page shows; the console and ctl change.
|
||||
console's generated "Commands" page is not in the portal (it is built
|
||||
from the cobra tree, which internal/web cannot see); no search across
|
||||
documents.
|
||||
- [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent
|
||||
(loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings)
|
||||
forwarded as net block-hash changes over the agent stream; router
|
||||
index, tokenize via the replica's `/tokenize`, sha256_cbor block-hash
|
||||
chain checked against cbor2/hashlib golden values; scorer by leading-
|
||||
block share with tier 1 fallback; opt-in `kv_events` per profile;
|
||||
metrics; docs. Left: validation against a real vLLM (event shape, hash
|
||||
agreement, `/tokenize` matching the served prompt, cost of
|
||||
tokenizing); vLLM's replay endpoint instead of resetting on a gap;
|
||||
data-parallel ranks (one publisher each, ports offset); hybrid models'
|
||||
cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt
|
||||
requests (extra keys); a router-only deployment (no agents)
|
||||
subscribing directly; standby control planes have no index until they
|
||||
lead; a per-replica fallback when only some candidates have an index;
|
||||
caching tokenizations of repeated prompts.
|
||||
- **24 LoRA affinity (branch `features-c2`):** built `max_loaded_adapters`
|
||||
(catalog), vLLM adapter support at start and on demand
|
||||
(`engines/lora.go`, `vllm.go`), the agent linking fetched adapters as
|
||||
`llmgw-lora/<name>` in the engines' working directory, the router's
|
||||
adapter tracker (`router/lora.go`: loads/unloads with the engine key,
|
||||
LRU eviction never of an adapter in use, one load per adapter per
|
||||
replica, `/v1/models` sync, 404 reload) and the `adapter-affinity`
|
||||
scheduler filter in the chat, batch and long-context profiles, with
|
||||
metrics and docs; tested against a fake of vLLM's HTTP interface.
|
||||
**Left:** never run against a real vLLM — in particular that vLLM
|
||||
resolves the relative `llmgw-lora/<name>` path against its working
|
||||
directory on every node (multi-node too) and that unloading an adapter
|
||||
mid-request is harmless; the agent still fetches every adapter at
|
||||
replica start, so a new adapter needs a restart (fetching on first
|
||||
load would need an agent command and a proto change); load timeout and
|
||||
resync interval are constants, not config; SGLang's runtime LoRA API
|
||||
is not driven; no per-replica "adapters held" view in the console or
|
||||
admin API; the debug profile has no affinity.
|
||||
- 24. Multi-cluster federation — built: peers via `ctl federation` kept in
|
||||
the store, their models as PEER/MODEL synced from /v1/models
|
||||
(withdrawn-while-referenced), /ready health with latency and routing
|
||||
withdrawal, `ctl model fallbacks` for overflow, hop header loop
|
||||
refusal, residency/no_store hold, optional attribution tags, console
|
||||
view, metrics, docs. Left: server.yaml-declared models cannot name a
|
||||
peer model as a fallback (config catalog is validated before peers are
|
||||
read); per-peer CA / mTLS for peers behind a private CA; transitive
|
||||
federation (A→B→C models) deliberately not offered; latency is /ready
|
||||
RTT only (no per-request peer TTFT used in routing or weighting); the
|
||||
peer's adapters are not imported (listed as aliases); no console form
|
||||
to add/remove peers (CLI only); key file must exist on every control
|
||||
plane.
|
||||
- 24 (facility) — built: location labels (`ctl label`, agent.yaml),
|
||||
spreading across racks/PDUs, `keep_within`, held domains with drain
|
||||
(`ctl facility hold/release`, kept in the store), power caps from
|
||||
`facility.budgets` against declared/derived/measured draw, shown in
|
||||
explain-placement, `ctl nodes`, `ctl facility`, the console and the
|
||||
API. Left: temperature conditions with hysteresis and their
|
||||
enforcement (steps 3–4), a Prometheus/DCIM source, power headroom as a
|
||||
rack-levelling score, spreading multi-node replicas across racks,
|
||||
`llmgw_domain_*` metrics and alerts (near cap, over cap, stale data),
|
||||
shedding and tokens per watt (step 5), and a drained node's state
|
||||
surviving a restart.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in new issue
Block a user