Merge C24 KV-index routing from vLLM's cache events

Conflicts with the LoRA work: vLLM's arguments, the engine profile, router
metrics and the scheduler's request all keep both; agent.pb.go regenerated
from the merged proto; the two test fakes of vLLM given separate names.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 01:24:25 +02:00
commit 9e46ee83e4
43 files changed
+3498 -265

No files matched your search

+59
View File
@@ -1103,6 +1103,65 @@ joined a load, loaded it), `llmgw_adapter_loads_total` and
adapter at start: llama-server can change the scale of an adapter it started
with, not load a new one.
### Routing by what each replica's cache actually holds
With several replicas of a vLLM model, a request is fastest on the replica
that already holds its prompt's prefix in KV cache. The router guesses which
that is from what it has sent where (`router.prefix`). It can also be told:
vLLM publishes an event every time it stores or evicts a cache block, and with
`kv_events` on a profile llmgw routes that model by those events.
```yaml
- name: llama-3.1-8b-instruct
source: meta-llama/Llama-3.1-8B-Instruct
routing_profile: chat # chat or long-context; batch ignores prefixes
profiles:
- name: h100
gpu_class: h100-80g
kv_events: true
```
How it works, so you know what you are paying for:
- The node agent starts vLLM with `--kv-events-config` publishing to a port
on 127.0.0.1 that the agent listens on, `--prefix-caching-hash-algo
sha256_cbor`, and `PYTHONHASHSEED=1729`. The events never leave the node:
the agent keeps the set of blocks each engine holds and sends the changes,
four times a second, up the stream it already has to the control plane,
which hands them to the router. A new stream — a control plane restart, a
failover — starts with a full copy. Nothing new is opened in the firewall.
- For each request to such a model, the router asks one of its replicas to
tokenize the prompt (vLLM's `/tokenize`, with the request's messages and
tools), hashes the token ids into blocks the way vLLM does, and prefers the
replica holding the most leading blocks. Tokenizing at the engine is what
makes the hashes agree: the chat template is part of the tokens, and only
the engine runs it exactly. It costs a round trip before the request is
placed and the template run twice; `router.kv_index.tokenize_timeout`
(250ms) bounds it.
- When the index has nothing to say — no events have arrived yet, the
request has an image or a `cache_salt` or names a LoRA adapter, the
tokenize call failed — the router routes as it would without it: by the
approximate tree if `router.prefix.enabled`, else by load. The imbalance
guard of `router.prefix` applies to both.
Turning `kv_events` on changes how the engine is started, so replicas already
running keep running without it (and show as stale) until they are replaced.
Leave `--prefix-caching-hash-algo` out of the profile's `args`: an engine
hashing another way publishes blocks the router can never match. The agent
checks the first block of prompts against its own hashing, and if they never
agree it says so in its log and on `llmgw_kv_index_problem`, and the replica
is routed as though it published nothing.
Watch `llmgw_kv_index_decisions_total` beside `llmgw_prefix_decisions_total`:
`kv_hit` is the index deciding, `kv_miss_min_load` is it knowing no replica
holds enough (`router.kv_index.min_blocks`, 4 blocks), and the `fallback_*`
values say why it could not answer. `llmgw_kv_index_tokenize_seconds` is what
the tokenize call costs, and `llmgw_kv_index_blocks` what each replica holds.
This has been built and tested against vLLM's event format and its hashing as
its source defines them, with libzmq and msgspec publishing — not yet against
a running vLLM. Treat it as experimental until it has been.
### Models served outside the cluster
A model can be an OpenAI-compatible API elsewhere — a hosted provider, or