Merge C24 KV-index routing from vLLM's cache events

Conflicts with the LoRA work: vLLM's arguments, the engine profile, router
metrics and the scheduler's request all keep both; agent.pb.go regenerated
from the merged proto; the two test fakes of vLLM given separate names.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 01:24:25 +02:00
commit 9e46ee83e4
43 files changed
+3498 -265

No files matched your search

+42
View File
@@ -0,0 +1,42 @@
# C24 — precise KV-index routing (plan §7.5 tier 2)
## CHANGELOG
### Routing by what each vLLM replica's cache holds
- A vLLM profile can set `kv_events: true`. The agent then starts the engine
publishing its KV cache events (`--kv-events-config`) to a port on
127.0.0.1 the agent listens on, with `--prefix-caching-hash-algo
sha256_cbor` and `PYTHONHASHSEED=1729`, so its block hashes can be
computed outside Python. The events never leave the node.
- The agent mirrors each engine's set of cached blocks and sends the net
changes up its existing stream to the control plane four times a second
(new `KVEvents` agent message), starting every new stream with a full
copy. Lost events (a gap in vLLM's sequence numbers), an engine restart or
`AllBlocksCleared` start a replica's index again; blocks offloaded to CPU,
remote ones and a hybrid model's second cache group are not indexed.
- The control plane passes them to the router's index, only for replicas
the sending node runs, and empties what a stream described when it closes.
- For a model with `kv_events`, the router asks one of its replicas to
tokenize the request (vLLM's `/tokenize`, with messages and tools, so the
chat template is applied exactly), hashes the tokens into blocks as vLLM
does, and scores replicas by the share of the prompt's leading blocks they
hold. The imbalance guard still applies. With no answer — no events yet,
an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls
back to the approximate tree (`router.prefix`) or to load.
- The agent checks first blocks against llmgw's hashing; an engine that
hashes another way (a profile overriding the algorithm, a changed vLLM) is
reported in the agent's log and `llmgw_kv_index_problem`, and routed as
though it published nothing.
- New settings `router.kv_index.tokenize_timeout` (250ms), `min_blocks` (4)
and `max_blocks` (4096). New metrics `llmgw_kv_index_decisions_total`
(`kv_hit`, `kv_miss_min_load`, `load_balance_override`,
`fallback_no_index`, `fallback_unsupported`, `fallback_tokenize_failed`),
`llmgw_kv_index_tokenize_seconds`, `llmgw_kv_index_blocks`.
- Experimental: tested against vLLM's event format and hashing as its source
defines them, and against libzmq and msgspec publishing — not yet against a
running vLLM.
## BACKLOG
- [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent (loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings) forwarded as net block-hash changes over the agent stream; router index, tokenize via the replica's `/tokenize`, sha256_cbor block-hash chain checked against cbor2/hashlib golden values; scorer by leading-block share with tier 1 fallback; opt-in `kv_events` per profile; metrics; docs. Left: validation against a real vLLM (event shape, hash agreement, `/tokenize` matching the served prompt, cost of tokenizing); vLLM's replay endpoint instead of resetting on a gap; data-parallel ranks (one publisher each, ports offset); hybrid models' cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt requests (extra keys); a router-only deployment (no agents) subscribing directly; standby control planes have no index until they lead; a per-replica fallback when only some candidates have an index; caching tokenizations of repeated prompts.