Commit Graph
11 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 9e46ee83e4 Merge C24 KV-index routing from vLLM's cache events
Conflicts with the LoRA work: vLLM's arguments, the engine profile, router
metrics and the scheduler's request all keep both; agent.pb.go regenerated
from the merged proto; the two test fakes of vLLM given separate names.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:24:25 +02:00
Ilker ManapandClaude Opus 5.5 117f417ef5 Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held
Plan §7.5 tier 2, opt-in per profile with kv_events. The agent starts vLLM
publishing KV cache events to a loopback port it listens on (pure-Go zmq4,
msgpack in both of vLLM's encodings), with sha256_cbor hashing and a fixed
PYTHONHASHSEED; it mirrors each engine's blocks and sends net changes up the
existing stream, with a full copy on every new one and a reset on gaps,
restarts and clears. The control plane hands them to the router, for the
sending node's own replicas only. The router tokenizes through a replica's
/tokenize, hashes the tokens as vLLM does (checked against cbor2/hashlib
golden values), and scores replicas by the share of leading blocks held,
falling back to the approximate tree. New metrics beside
llmgw_prefix_decisions_total. Untested against a running vLLM.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:21:11 +02:00
Ilker ManapandClaude Opus 5.5 2a1e0e6c4c LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters
A vLLM model can carry PEFT adapters, registered at start, or with
max_loaded_adapters held at most N per replica: the router loads an adapter
through vLLM's runtime LoRA API when a request first wants it, sends the
adapter's requests to a replica that holds it while that one is not
saturated, loads each cold adapter once, and unloads the least recently used
idle one to make room. llama.cpp keeps loading every adapter at start.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:14:38 +02:00
Ilker ManapandClaude Opus 5.5 44f3a9e31d Reusing answers: a response cache per project, exact match, opt-in with a TTL
llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on
the console's project menu. The same request — project, endpoint, canonical
model and the body without stream, stream_options, user, metadata, store and
service_tier, keys sorted — is answered from the gateway's memory for the
TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank,
score) unless --any; only a whole 200 from the model asked for, not a
fallback's, a canary's, or one a guardrail changed.

Looked up after the limits and the input guardrails; the output policy
judges a reused answer; no_store and stored conversations bypass it.
Streamed answers are kept put back together and streamed back as chunks.
x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or
x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost,
marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded
by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total,
llmgw_cache_bytes, llmgw_cache_entries.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 23:50:53 +02:00
Ilker ManapandClaude Opus 5.5 5b92ae0277 Token-level timing and admission by SLO: TPOT, inter-token and tokens a second by model and tenant; a model past its SLO is saturated for the lower bands
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 23:13:18 +02:00
Ilker ManapandClaude Opus 5.5 47e847423d Guardrail metrics and alerts: checks, latency, judge failures, fail-open, redactions, stopped streams, tenant blocks
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 15:11:50 +02:00
Ilker ManapandClaude Opus 5.5 ce006a6e6f Publish per-tenant request and budget series, and alert on them
Requests by outcome, tokens and first-token time per tenant, at the level
router.tenant_label names (team by default), and every budget's share spent
this period. Rules alert on budgets at 80% and 100%, tenants hitting limits,
failing, spiking past their weekly average, or waiting on first tokens.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 19:55:24 +02:00
Ilker ManapandClaude Opus 5.5 91fe24a105 Close what a second security review found still open
Release / build (push) Successful in 1m8s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m50s
A review of v0.10.4-rc.4, and a look at the live network, found:

- Alertmanager listening on every interface: from any node its API took
  silences, with no login. Bound to loopback on the monitoring host, and the
  monitoring README now installs it that way; Prometheus beside it is its only
  client, and one Alertmanager needs no cluster port.
- Login and signup unbounded in memory. Each attempt runs argon2 with 64 MiB,
  and nothing limited how many ran at once; the per-address throttle kept one
  entry per address ever tried. Hashing now takes one of a few slots and
  waits for one otherwise. The throttle expires entries whose window has
  passed when it fills, and if it is still full it refuses new addresses:
  evicting live ones instead would let a flood of made-up addresses reset the
  throttle on the address actually being guessed.
- Metric labels taken from the request. A refused request counted the model
  name it asked for, so any key holder could mint a series per request.
  Requests are counted under the catalog's name, or "unknown".
- Two in code written this week. A SQLite backup made its snapshot with the
  default mode in the listable backup directory, a full copy of the store
  readable while the backup ran; it is now made in a private directory, and
  the unit sets UMask=0077. And archive links were checked by name only, so
  "a -> ." then "a/a/b -> ../.." passed and a file written through them landed
  above the install directory, as root. Archives now unpack through os.Root,
  and a link may only point at a file beside it — which every llama.cpp
  b10991 archive, CPU and CUDA, was checked to follow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 02:25:07 +02:00
Ilker ManapandClaude Opus 5 a075ff4120 Add approximate prefix routing, and find out it did nothing
CI / test (push) Successful in 1m19s
Tier 1 of plan §7.5: a radix tree of request prefixes that estimates which
replica has already prefilled one. Bytes, not tokens — needing the engine's
exact tokenizer is what makes tier 2 a separate project.

It went in inert, and the decision metric is what caught it: 144 lookups, zero
hits. The cause was a rule I had written and tested for. A node's label is the
first request that created it, so the second request for the same system
prompt diverges in its last few bytes — a new question after the same
preamble, the single most common shape there is. Requiring the whole label to
match threw away the 700 bytes they shared. What matters is the shared head,
because that is what the engine cached, and a partial label match now counts.
Two unit tests asserted the wrong rule and have been inverted.

With that fixed, measured end to end against engines whose caches are too
small for the working set: 22.5% cache hits without, 81.7% with, and the
breakdown shows the misses are exactly the first round, when the tree is
empty.

The scenario matters as much as the number. With a handful of prompts and a
large cache, scattering is free — every replica ends up holding everything —
and prefix routing can only add constraint. The first version of the test
measured that case and showed a loss, which was a true statement about a
situation the feature is not for.

Two things keep it from being the disaster prefix routing usually is. The
imbalance guard, which needs both an absolute and a relative gap before it
fires, because either alone misfires — absolute on an idle cluster, relative
on a busy one. And it is a scorer rather than a filter: affinity is a vote the
queue and inflight scorers can outvote, not a rule.

The tree records the replica the picker actually chose, not the one the scorer
proposed, and a node prefers the replica it has seen most rather than most
recently — a count is stable under a randomizing picker, where recency just
chases whatever the picker last did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 12:13:19 +02:00
ilkerandClaude Opus 5 dc38dd6096 Phase 1.5: admission control
CI / test (push) Successful in 48s
Inserted between quota checks and scheduling: admission answers when a request
may proceed, the scheduler still answers where it goes. Keeping them apart is
the point — an engine queue is FIFO and knows nothing about tenants, so once a
batch job is in it, no endpoint selection rescues the request behind it.

- saturation detector: one number per pool, the worse of queue depth and KV
  occupancy, since either alone misleads.
- priority bands with per-band saturation ceilings, so the cluster degrades from
  the bottom up.
- tenant round-robin within a band: one caller sending a thousand requests takes
  one slot in the rotation, not a thousand.
- bounded queues with capacity (503) and TTL (504); cancelled requests leave the
  queue instead of holding capacity.
- metrics for queue depth, wait time, rejections and pool saturation.

Exit criterion, measured in test/e2e: with 500 concurrent bronze requests
running continuously, interactive p99 in the platinum band stays at 1.04-1.24x
idle against a 1.5x budget. With admission off the same load pushes it to
55-60x. Both directions are asserted, so the test fails if admission regresses
and also if the load generator stops being a real load.

Two lags had to be closed, and they needed different answers. Engine metrics are
scraped and lag an interval; router inflight does not, but it counts running
requests, and treating those as queue makes a healthy engine look overloaded
every interval. What is actually queued is inflight minus batch capacity, and
capacity is not in the metric contract, so it is learned as the high-water mark
of observed running. Separately, between admission granting a request and the
router dispatching it the request is invisible everywhere, so a simultaneous
burst passed the check en masse; the controller now counts its own outstanding
leases against dispatched inflight.

One plan correction: a saturation ceiling of 1.0 was documented as "never held
back". It is not — it means "flow up to the healthy limit". A band that must
never wait needs a ceiling above 1. Plan §7.2.2 updated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:53:38 +02:00
ilkerandClaude Opus 5 a3f71aa5ee Phase 1: the router serves traffic
CI / test (push) Successful in 1m1s
llmgw server is real now. It fronts a static set of engine replicas with an
OpenAI-compatible API, scoring endpoints through the pipeline added in the
previous commit.

- proxy: SSE relayed event by event with a flush after each, cancellation
  propagated upstream, and separate deadlines for dial, headers, first token,
  inter-token idle and total. The idle watchdog is what distinguishes a wedged
  engine from a slow one.
- router: dispatch with retry and model fallback. Retry only happens while
  nothing has reached the client; after that, splicing two responses together
  would be worse than the short stream the client already sees.
- config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides.
  Validation rejects duplicate endpoint ids and aliases that shadow a model,
  because both misroute silently at request time rather than failing loudly.
- metrics: requests, duration, TTFT, tokens, plus per-scorer values and
  per-filter drops on a separate admin listener with health and pprof.

Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across
3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV
blocks on every engine; router overhead p99 is about 0.9 ms against the plan's
2 ms budget.

The handlers live in internal/router rather than an internal/router/openai
package as the plan's tree sketched: they need the dispatch loop, and splitting
them apart would have meant an interface with exactly one implementation. Plan
§4 updated to match.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:41:13 +02:00