Conflicts with the LoRA work: vLLM's arguments, the engine profile, router
metrics and the scheduler's request all keep both; agent.pb.go regenerated
from the merged proto; the two test fakes of vLLM given separate names.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Plan §7.5 tier 2, opt-in per profile with kv_events. The agent starts vLLM
publishing KV cache events to a loopback port it listens on (pure-Go zmq4,
msgpack in both of vLLM's encodings), with sha256_cbor hashing and a fixed
PYTHONHASHSEED; it mirrors each engine's blocks and sends net changes up the
existing stream, with a full copy on every new one and a reset on gaps,
restarts and clears. The control plane hands them to the router, for the
sending node's own replicas only. The router tokenizes through a replica's
/tokenize, hashes the tokens as vLLM does (checked against cbor2/hashlib
golden values), and scores replicas by the share of leading blocks held,
falling back to the approximate tree. New metrics beside
llmgw_prefix_decisions_total. Untested against a running vLLM.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A vLLM model can carry PEFT adapters, registered at start, or with
max_loaded_adapters held at most N per replica: the router loads an adapter
through vLLM's runtime LoRA API when a request first wants it, sends the
adapter's requests to a replica that holds it while that one is not
saturated, loads each cold adapter once, and unloads the least recently used
idle one to make room. llama.cpp keeps loading every adapter at start.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The pool-priority filter kept the highest-priority pool however full it
was. It now takes the first pool, by priority, that is not saturated by
admission's own measure, and the preferred one when all are.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Tier 1 of plan §7.5: a radix tree of request prefixes that estimates which
replica has already prefilled one. Bytes, not tokens — needing the engine's
exact tokenizer is what makes tier 2 a separate project.
It went in inert, and the decision metric is what caught it: 144 lookups, zero
hits. The cause was a rule I had written and tested for. A node's label is the
first request that created it, so the second request for the same system
prompt diverges in its last few bytes — a new question after the same
preamble, the single most common shape there is. Requiring the whole label to
match threw away the 700 bytes they shared. What matters is the shared head,
because that is what the engine cached, and a partial label match now counts.
Two unit tests asserted the wrong rule and have been inverted.
With that fixed, measured end to end against engines whose caches are too
small for the working set: 22.5% cache hits without, 81.7% with, and the
breakdown shows the misses are exactly the first round, when the tree is
empty.
The scenario matters as much as the number. With a handful of prompts and a
large cache, scattering is free — every replica ends up holding everything —
and prefix routing can only add constraint. The first version of the test
measured that case and showed a loss, which was a true statement about a
situation the feature is not for.
Two things keep it from being the disaster prefix routing usually is. The
imbalance guard, which needs both an absolute and a relative gap before it
fires, because either alone misfires — absolute on an idle cluster, relative
on a busy one. And it is a scorer rather than a filter: affinity is a vote the
queue and inflight scorers can outvote, not a rule.
The tree records the replica the picker actually chose, not the one the scorer
proposed, and a node prefers the replica it has seen most rather than most
recently — a count is stable under a randomizing picker, where recency just
chases whatever the picker last did.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The code behind plan §7.3-7.6:
- breaker: folds cooldown and circuit breaking into one component
(FailureThreshold=1 gives LiteLLM-style cooldown), with exponential backoff
and half-open probes.
- endpoint: live state registry. Inflight counting, staleness detection, drain
rate derived from counters. A scrape failure does not trip the breaker — the
engine may be serving fine while only its metrics endpoint is down.
- scrape: parses the §7.6 metric contract. Errors when any of the mandatory
three is missing; the metric name mapping is overridable, since names drift
between versions (plan §14).
- scheduler: the filter -> score -> pick pipeline. A "strategy" is now just a
set of scorer weights; chat, batch and long-context are the same code with
different weights. Scorers: queue, kv (hard penalty above the ceiling),
inflight, load, weight, session (bounded-load consistent hashing). Pickers:
max-score, p2c, softmax, round-robin.
Decision carries the rationale — which scorer produced what, which filter
dropped how many. Tuning the weights without that is guesswork.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ilkerandClaude Opus 5