Plan §7.5 tier 2, opt-in per profile with kv_events. The agent starts vLLM publishing KV cache events to a loopback port it listens on (pure-Go zmq4, msgpack in both of vLLM's encodings), with sha256_cbor hashing and a fixed PYTHONHASHSEED; it mirrors each engine's blocks and sends net changes up the existing stream, with a full copy on every new one and a reset on gaps, restarts and clears. The control plane hands them to the router, for the sending node's own replicas only. The router tokenizes through a replica's /tokenize, hashes the tokens as vLLM does (checked against cbor2/hashlib golden values), and scores replicas by the share of leading blocks held, falling back to the approximate tree. New metrics beside llmgw_prefix_decisions_total. Untested against a running vLLM. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
3.3 KiB
3.3 KiB
C24 — precise KV-index routing (plan §7.5 tier 2)
CHANGELOG
Routing by what each vLLM replica's cache holds
- A vLLM profile can set
kv_events: true. The agent then starts the engine publishing its KV cache events (--kv-events-config) to a port on 127.0.0.1 the agent listens on, with--prefix-caching-hash-algo sha256_cborandPYTHONHASHSEED=1729, so its block hashes can be computed outside Python. The events never leave the node. - The agent mirrors each engine's set of cached blocks and sends the net
changes up its existing stream to the control plane four times a second
(new
KVEventsagent message), starting every new stream with a full copy. Lost events (a gap in vLLM's sequence numbers), an engine restart orAllBlocksClearedstart a replica's index again; blocks offloaded to CPU, remote ones and a hybrid model's second cache group are not indexed. - The control plane passes them to the router's index, only for replicas the sending node runs, and empties what a stream described when it closes.
- For a model with
kv_events, the router asks one of its replicas to tokenize the request (vLLM's/tokenize, with messages and tools, so the chat template is applied exactly), hashes the tokens into blocks as vLLM does, and scores replicas by the share of the prompt's leading blocks they hold. The imbalance guard still applies. With no answer — no events yet, an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls back to the approximate tree (router.prefix) or to load. - The agent checks first blocks against llmgw's hashing; an engine that
hashes another way (a profile overriding the algorithm, a changed vLLM) is
reported in the agent's log and
llmgw_kv_index_problem, and routed as though it published nothing. - New settings
router.kv_index.tokenize_timeout(250ms),min_blocks(4) andmax_blocks(4096). New metricsllmgw_kv_index_decisions_total(kv_hit,kv_miss_min_load,load_balance_override,fallback_no_index,fallback_unsupported,fallback_tokenize_failed),llmgw_kv_index_tokenize_seconds,llmgw_kv_index_blocks. - Experimental: tested against vLLM's event format and hashing as its source defines them, and against libzmq and msgspec publishing — not yet against a running vLLM.
BACKLOG
- [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent (loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings) forwarded as net block-hash changes over the agent stream; router index, tokenize via the replica's
/tokenize, sha256_cbor block-hash chain checked against cbor2/hashlib golden values; scorer by leading-block share with tier 1 fallback; opt-inkv_eventsper profile; metrics; docs. Left: validation against a real vLLM (event shape, hash agreement,/tokenizematching the served prompt, cost of tokenizing); vLLM's replay endpoint instead of resetting on a gap; data-parallel ranks (one publisher each, ports offset); hybrid models' cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt requests (extra keys); a router-only deployment (no agents) subscribing directly; standby control planes have no index until they lead; a per-replica fallback when only some candidates have an index; caching tokenizations of repeated prompts.