Plan §7.5 tier 2, opt-in per profile with kv_events. The agent starts vLLM publishing KV cache events to a loopback port it listens on (pure-Go zmq4, msgpack in both of vLLM's encodings), with sha256_cbor hashing and a fixed PYTHONHASHSEED; it mirrors each engine's blocks and sends net changes up the existing stream, with a full copy on every new one and a reset on gaps, restarts and clears. The control plane hands them to the router, for the sending node's own replicas only. The router tokenizes through a replica's /tokenize, hashes the tokens as vLLM does (checked against cbor2/hashlib golden values), and scores replicas by the share of leading blocks held, falling back to the approximate tree. New metrics beside llmgw_prefix_decisions_total. Untested against a running vLLM. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
43 lines
3.3 KiB
Markdown
43 lines
3.3 KiB
Markdown
# C24 — precise KV-index routing (plan §7.5 tier 2)
|
|
|
|
## CHANGELOG
|
|
|
|
### Routing by what each vLLM replica's cache holds
|
|
|
|
- A vLLM profile can set `kv_events: true`. The agent then starts the engine
|
|
publishing its KV cache events (`--kv-events-config`) to a port on
|
|
127.0.0.1 the agent listens on, with `--prefix-caching-hash-algo
|
|
sha256_cbor` and `PYTHONHASHSEED=1729`, so its block hashes can be
|
|
computed outside Python. The events never leave the node.
|
|
- The agent mirrors each engine's set of cached blocks and sends the net
|
|
changes up its existing stream to the control plane four times a second
|
|
(new `KVEvents` agent message), starting every new stream with a full
|
|
copy. Lost events (a gap in vLLM's sequence numbers), an engine restart or
|
|
`AllBlocksCleared` start a replica's index again; blocks offloaded to CPU,
|
|
remote ones and a hybrid model's second cache group are not indexed.
|
|
- The control plane passes them to the router's index, only for replicas
|
|
the sending node runs, and empties what a stream described when it closes.
|
|
- For a model with `kv_events`, the router asks one of its replicas to
|
|
tokenize the request (vLLM's `/tokenize`, with messages and tools, so the
|
|
chat template is applied exactly), hashes the tokens into blocks as vLLM
|
|
does, and scores replicas by the share of the prompt's leading blocks they
|
|
hold. The imbalance guard still applies. With no answer — no events yet,
|
|
an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls
|
|
back to the approximate tree (`router.prefix`) or to load.
|
|
- The agent checks first blocks against llmgw's hashing; an engine that
|
|
hashes another way (a profile overriding the algorithm, a changed vLLM) is
|
|
reported in the agent's log and `llmgw_kv_index_problem`, and routed as
|
|
though it published nothing.
|
|
- New settings `router.kv_index.tokenize_timeout` (250ms), `min_blocks` (4)
|
|
and `max_blocks` (4096). New metrics `llmgw_kv_index_decisions_total`
|
|
(`kv_hit`, `kv_miss_min_load`, `load_balance_override`,
|
|
`fallback_no_index`, `fallback_unsupported`, `fallback_tokenize_failed`),
|
|
`llmgw_kv_index_tokenize_seconds`, `llmgw_kv_index_blocks`.
|
|
- Experimental: tested against vLLM's event format and hashing as its source
|
|
defines them, and against libzmq and msgspec publishing — not yet against a
|
|
running vLLM.
|
|
|
|
## BACKLOG
|
|
|
|
- [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent (loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings) forwarded as net block-hash changes over the agent stream; router index, tokenize via the replica's `/tokenize`, sha256_cbor block-hash chain checked against cbor2/hashlib golden values; scorer by leading-block share with tier 1 fallback; opt-in `kv_events` per profile; metrics; docs. Left: validation against a real vLLM (event shape, hash agreement, `/tokenize` matching the served prompt, cost of tokenizing); vLLM's replay endpoint instead of resetting on a gap; data-parallel ranks (one publisher each, ports offset); hybrid models' cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt requests (extra keys); a router-only deployment (no agents) subscribing directly; standby control planes have no index until they lead; a per-replica fallback when only some candidates have an index; caching tokenizations of repeated prompts.
|