Merge C24 KV-index routing from vLLM's cache events
Conflicts with the LoRA work: vLLM's arguments, the engine profile, router metrics and the scheduler's request all keep both; agent.pb.go regenerated from the merged proto; the two test fakes of vLLM given separate names. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
9e46ee83e4
43 files changed
+3498
-265
No files matched your search
@@ -0,0 +1,42 @@
|
||||
# C24 — precise KV-index routing (plan §7.5 tier 2)
|
||||
|
||||
## CHANGELOG
|
||||
|
||||
### Routing by what each vLLM replica's cache holds
|
||||
|
||||
- A vLLM profile can set `kv_events: true`. The agent then starts the engine
|
||||
publishing its KV cache events (`--kv-events-config`) to a port on
|
||||
127.0.0.1 the agent listens on, with `--prefix-caching-hash-algo
|
||||
sha256_cbor` and `PYTHONHASHSEED=1729`, so its block hashes can be
|
||||
computed outside Python. The events never leave the node.
|
||||
- The agent mirrors each engine's set of cached blocks and sends the net
|
||||
changes up its existing stream to the control plane four times a second
|
||||
(new `KVEvents` agent message), starting every new stream with a full
|
||||
copy. Lost events (a gap in vLLM's sequence numbers), an engine restart or
|
||||
`AllBlocksCleared` start a replica's index again; blocks offloaded to CPU,
|
||||
remote ones and a hybrid model's second cache group are not indexed.
|
||||
- The control plane passes them to the router's index, only for replicas
|
||||
the sending node runs, and empties what a stream described when it closes.
|
||||
- For a model with `kv_events`, the router asks one of its replicas to
|
||||
tokenize the request (vLLM's `/tokenize`, with messages and tools, so the
|
||||
chat template is applied exactly), hashes the tokens into blocks as vLLM
|
||||
does, and scores replicas by the share of the prompt's leading blocks they
|
||||
hold. The imbalance guard still applies. With no answer — no events yet,
|
||||
an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls
|
||||
back to the approximate tree (`router.prefix`) or to load.
|
||||
- The agent checks first blocks against llmgw's hashing; an engine that
|
||||
hashes another way (a profile overriding the algorithm, a changed vLLM) is
|
||||
reported in the agent's log and `llmgw_kv_index_problem`, and routed as
|
||||
though it published nothing.
|
||||
- New settings `router.kv_index.tokenize_timeout` (250ms), `min_blocks` (4)
|
||||
and `max_blocks` (4096). New metrics `llmgw_kv_index_decisions_total`
|
||||
(`kv_hit`, `kv_miss_min_load`, `load_balance_override`,
|
||||
`fallback_no_index`, `fallback_unsupported`, `fallback_tokenize_failed`),
|
||||
`llmgw_kv_index_tokenize_seconds`, `llmgw_kv_index_blocks`.
|
||||
- Experimental: tested against vLLM's event format and hashing as its source
|
||||
defines them, and against libzmq and msgspec publishing — not yet against a
|
||||
running vLLM.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
- [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent (loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings) forwarded as net block-hash changes over the agent stream; router index, tokenize via the replica's `/tokenize`, sha256_cbor block-hash chain checked against cbor2/hashlib golden values; scorer by leading-block share with tier 1 fallback; opt-in `kv_events` per profile; metrics; docs. Left: validation against a real vLLM (event shape, hash agreement, `/tokenize` matching the served prompt, cost of tokenizing); vLLM's replay endpoint instead of resetting on a gap; data-parallel ranks (one publisher each, ports offset); hybrid models' cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt requests (extra keys); a router-only deployment (no agents) subscribing directly; standby control planes have no index until they lead; a per-replica fallback when only some candidates have an index; caching tokenizations of repeated prompts.
|
||||
Reference in new issue
Block a user