From 803d1d5fb6b101e87ebc97735e039d57c6fc2e08 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Mon, 5 Oct 2026 01:30:01 +0200 Subject: [PATCH] CHANGELOG and BACKLOG for the rest of C24 and the engine fixes; README Co-Authored-By: Claude Opus 5.5 (1M context) --- BACKLOG.md | 61 ++++++++++++++- CHANGELOG.md | 164 ++++++++++++++++++++++++++++++++++++++++ README.md | 6 +- notes/c24-facility.md | 40 ---------- notes/c24-federation.md | 40 ---------- notes/c24-kvindex.md | 42 ---------- notes/c24-lora.md | 52 ------------- notes/engine-fixes.md | 28 ------- 8 files changed, 227 insertions(+), 206 deletions(-) delete mode 100644 notes/c24-facility.md delete mode 100644 notes/c24-federation.md delete mode 100644 notes/c24-kvindex.md delete mode 100644 notes/c24-lora.md delete mode 100644 notes/engine-fixes.md diff --git a/BACKLOG.md b/BACKLOG.md index 1ed1927..4be277b 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -679,11 +679,11 @@ portal is not done — the page shows; the console and ctl change. **More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF import. `agent/engines/engine.go` interface and the metric contract in `router/scrape`. -24. Lower: precise KV-index routing (plan phase 6), LoRA affinity for +24. **Done (branch `features-c2`), all six parts.** Lower: precise KV-index routing (plan phase 6), LoRA affinity for vLLM, multi-cluster federation via External providers, facility-aware placement (`docs/facility-aware-placement.md`), portal i18n, SDK/docs portal. - Done so far (branch `features-c2`): + What was built, and what is left: - 24 (portal i18n): built — English/Turkish catalogue with fallback, per- language templates, cookie/Accept-Language/`landing.language` choice and a footer switcher, Turkish numbers and dates, every template and @@ -701,6 +701,63 @@ portal is not done — the page shows; the console and ctl change. console's generated "Commands" page is not in the portal (it is built from the cobra tree, which internal/web cannot see); no search across documents. + - [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent + (loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings) + forwarded as net block-hash changes over the agent stream; router + index, tokenize via the replica's `/tokenize`, sha256_cbor block-hash + chain checked against cbor2/hashlib golden values; scorer by leading- + block share with tier 1 fallback; opt-in `kv_events` per profile; + metrics; docs. Left: validation against a real vLLM (event shape, hash + agreement, `/tokenize` matching the served prompt, cost of + tokenizing); vLLM's replay endpoint instead of resetting on a gap; + data-parallel ranks (one publisher each, ports offset); hybrid models' + cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt + requests (extra keys); a router-only deployment (no agents) + subscribing directly; standby control planes have no index until they + lead; a per-replica fallback when only some candidates have an index; + caching tokenizations of repeated prompts. + - **24 LoRA affinity (branch `features-c2`):** built `max_loaded_adapters` + (catalog), vLLM adapter support at start and on demand + (`engines/lora.go`, `vllm.go`), the agent linking fetched adapters as + `llmgw-lora/` in the engines' working directory, the router's + adapter tracker (`router/lora.go`: loads/unloads with the engine key, + LRU eviction never of an adapter in use, one load per adapter per + replica, `/v1/models` sync, 404 reload) and the `adapter-affinity` + scheduler filter in the chat, batch and long-context profiles, with + metrics and docs; tested against a fake of vLLM's HTTP interface. + **Left:** never run against a real vLLM — in particular that vLLM + resolves the relative `llmgw-lora/` path against its working + directory on every node (multi-node too) and that unloading an adapter + mid-request is harmless; the agent still fetches every adapter at + replica start, so a new adapter needs a restart (fetching on first + load would need an agent command and a proto change); load timeout and + resync interval are constants, not config; SGLang's runtime LoRA API + is not driven; no per-replica "adapters held" view in the console or + admin API; the debug profile has no affinity. + - 24. Multi-cluster federation — built: peers via `ctl federation` kept in + the store, their models as PEER/MODEL synced from /v1/models + (withdrawn-while-referenced), /ready health with latency and routing + withdrawal, `ctl model fallbacks` for overflow, hop header loop + refusal, residency/no_store hold, optional attribution tags, console + view, metrics, docs. Left: server.yaml-declared models cannot name a + peer model as a fallback (config catalog is validated before peers are + read); per-peer CA / mTLS for peers behind a private CA; transitive + federation (A→B→C models) deliberately not offered; latency is /ready + RTT only (no per-request peer TTFT used in routing or weighting); the + peer's adapters are not imported (listed as aliases); no console form + to add/remove peers (CLI only); key file must exist on every control + plane. + - 24 (facility) — built: location labels (`ctl label`, agent.yaml), + spreading across racks/PDUs, `keep_within`, held domains with drain + (`ctl facility hold/release`, kept in the store), power caps from + `facility.budgets` against declared/derived/measured draw, shown in + explain-placement, `ctl nodes`, `ctl facility`, the console and the + API. Left: temperature conditions with hysteresis and their + enforcement (steps 3–4), a Prometheus/DCIM source, power headroom as a + rack-levelling score, spreading multi-node replicas across racks, + `llmgw_domain_*` metrics and alerts (near cap, over cap, stale data), + shedding and tokens per watt (step 5), and a drained node's state + surviving a restart. --- diff --git a/CHANGELOG.md b/CHANGELOG.md index 6dde2e7..919f297 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,170 @@ APIs change between minors without a deprecation period. ## Unreleased +### Routing by what each vLLM replica's cache holds + +- A vLLM profile can set `kv_events: true`. The agent then starts the engine + publishing its KV cache events (`--kv-events-config`) to a port on + 127.0.0.1 the agent listens on, with `--prefix-caching-hash-algo + sha256_cbor` and `PYTHONHASHSEED=1729`, so its block hashes can be + computed outside Python. The events never leave the node. +- The agent mirrors each engine's set of cached blocks and sends the net + changes up its existing stream to the control plane four times a second + (new `KVEvents` agent message), starting every new stream with a full + copy. Lost events (a gap in vLLM's sequence numbers), an engine restart or + `AllBlocksCleared` start a replica's index again; blocks offloaded to CPU, + remote ones and a hybrid model's second cache group are not indexed. +- The control plane passes them to the router's index, only for replicas + the sending node runs, and empties what a stream described when it closes. +- For a model with `kv_events`, the router asks one of its replicas to + tokenize the request (vLLM's `/tokenize`, with messages and tools, so the + chat template is applied exactly), hashes the tokens into blocks as vLLM + does, and scores replicas by the share of the prompt's leading blocks they + hold. The imbalance guard still applies. With no answer — no events yet, + an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls + back to the approximate tree (`router.prefix`) or to load. +- The agent checks first blocks against llmgw's hashing; an engine that + hashes another way (a profile overriding the algorithm, a changed vLLM) is + reported in the agent's log and `llmgw_kv_index_problem`, and routed as + though it published nothing. +- New settings `router.kv_index.tokenize_timeout` (250ms), `min_blocks` (4) + and `max_blocks` (4096). New metrics `llmgw_kv_index_decisions_total` + (`kv_hit`, `kv_miss_min_load`, `load_balance_override`, + `fallback_no_index`, `fallback_unsupported`, `fallback_tokenize_failed`), + `llmgw_kv_index_tokenize_seconds`, `llmgw_kv_index_blocks`. +- Experimental: tested against vLLM's event format and hashing as its source + defines them, and against libzmq and msgspec publishing — not yet against a + running vLLM. + +### LoRA adapters on vLLM, loaded on demand + +- **vLLM models can carry adapters.** A PEFT adapter directory (a mirror + version ending in `/`, a path on the nodes, or a Hugging Face repository + id) is registered at start with `--enable-lora --lora-modules`, and a + request names it as its model, as on llama.cpp. `scale` applies on + llama.cpp only. SGLang and TensorRT-LLM profiles still refuse adapters. +- **`max_loaded_adapters: N` on a model** has each replica hold at most N of + its adapters. Replicas start holding none (vLLM with + `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras N`, unless the + profile's args set their own). The router loads an adapter on a replica + through `/v1/load_lora_adapter` when a request first wants it there, and + unloads the least recently used one no request is running on to make room. + vLLM profiles only; llama.cpp keeps loading every adapter at start, since + llama-server can rescale an adapter it started with but not load another. +- **Adapter affinity:** a request for an adapter goes to a replica that has + it, or is loading it, while that replica is not saturated; otherwise to one + with room for another adapter; otherwise where load sends it — which is how + a busy adapter spreads to a second replica. Requests arriving during a load + wait for it instead of loading the adapter again. +- **Self-correcting:** each replica's `/v1/models` is read when the router + first meets it and every 30 seconds, so adapters loaded before a restart or + by another router are known; a 404 for an adapter the router thought loaded + loads it again and retries. An adapter vLLM refuses (not found, rank above + `--max-lora-rank`) fails the request with vLLM's reason, without trying it + on every replica. +- **Metrics:** `llmgw_adapter_affinity_total{model,result=hit|joined|loaded}`, + `llmgw_adapter_loads_total{model,outcome}`, + `llmgw_adapter_evictions_total{model,outcome}`. +- The admin API and `ctl model adapters` read the file as GGUF only for a + model with llama.cpp profiles. + +### Federating clusters: another llmgw cluster as a provider + +- `llmgw ctl federation add NAME --url URL --key-file FILE [--send-tags]` + makes another llmgw gateway a peer. It is checked first (its `/ready`, + timed, and its `/v1/models` with the key) and kept in the store (migration + 0029, `federation_peers`) with the models it offers. +- A peer's models join the catalog as `NAME/MODEL`, always prefixed, and + follow the peer's own list every `federation.interval` (10s). A model the + peer drops goes, unless a model here still falls back to it: then it is + kept withdrawn (in the catalog, sent nothing) until the fallback changes. + The peer's aliases and its own peers' models are not taken. +- A peer whose `/ready` fails, which refuses the key, or which does not answer + is taken out of routing until it answers again. `/ready` now counts only a + gateway's own endpoints, not its peers'. +- `llmgw ctl model fallbacks MODEL [FALLBACK...]` (and `PUT + /api/v1/models/{name}/fallbacks`) sets a runtime model's fallbacks, so + overflow can go to a peer's model. +- Requests to a peer carry `x-llmgw-hop` with the ids of the clusters they + have passed through (each cluster's id is kept in the store). A request that + comes back to a cluster is refused with 508 `federation_loop`, and a peer a + request came from is not sent it. +- The peer sees one key, this cluster's, and the request's `X-Request-Id`. + With `--send-tags`, also `x-llmgw-tags: federated_from=, + org=,team=,project=` (ids); off by default. +- In-cluster residency and `no_store` keep requests off peers, by name and as + fallbacks. Usage records the peer's model as served, and `peer:NAME/MODEL` + as the replica. +- `ctl federation list|remove|sync`, a Federation view in the console, + `llmgw_federation_peer_up`, `_latency_seconds` and `_models` metrics, admin + API `/api/v1/federation`, and a "Federating clusters" section in + docs/install.md. New `federation:` block in server.yaml (`name`, + `interval`). +- A response header the gateway already set (X-Request-Id, + x-llmgw-served-model) is no longer repeated from the upstream's. + +### Facility-aware placement: racks, PDUs, holds and power caps + +- **Where a node is**, from five labels: `llmgw.io/site`, `room`, `row`, + `rack` and `pdu` (a list, for a dual-fed rack), in `agent.yaml` or set from + the control plane with **`llmgw ctl label NODE key=value… [key-…]`**, which + is kept in the store (migration 0031, `node_labels`), wins over the agent's + labels, works before the node joins, and refuses a misspelt `llmgw.io/` key. + A node with no location places as before. +- **Replicas spread across racks and PDUs.** A model's next replica goes to a + node sharing no rack or PDU with its others when there is one: an ordering + above weights already on disk and bin packing, never a filter. +- **`keep_within: rack`** (or row, room, site) on a profile keeps a + multi-node replica's nodes in one such domain. +- **Holds.** **`llmgw ctl facility hold rack:r12-07 --reason … [--drain]`** + places nothing new in a rack, row, room, site or PDU; `--drain` also drains + its nodes. **`llmgw ctl facility release`** undoes both. Kept across + restarts (migration 0031, `facility_holds`). +- **Power caps.** `facility.budgets` in `server.yaml` gives a domain's + `cap_watts` and `other_watts`, with a `margin`; a replica's draw is the + profile's `power_watts`, or its cards at the power limit they report + (`gpu_watts` otherwise) and its cores at `cpu_core_watts`, with `node_watts` + per node. Use is the larger of reserved and measured, node by node; a shard + that would take a domain past its cap less the margin is refused, and a + multi-node replica's shards are counted together. Unknown draw never blocks. +- **Seen everywhere it matters.** `ctl explain-placement` names the held + domain or the kilowatts, and keeps the rejections of a replica that was + placed, so one in an unexpected rack says why. `ctl nodes` has LOCATION, + *ready, held* and `--domain`. **`llmgw ctl facility`** lists every domain + with nodes, use, limit, reserved, measured and holds. The console's Nodes + view has WHERE; a node's menu has **Location…** and **Hold rack…/Release**. + Admin API: `GET /api/v1/facility`, `POST /api/v1/facility/holds`, + `DELETE /api/v1/facility/holds/{domain}`, `POST /api/v1/nodes/{id}/labels`. +- Docs: `docs/facility-aware-placement.md` says what is built and where it + departs from the design; `docs/install.md` and `docs/console.md`. + +### SGLang measured, checked and benchmarked as itself + +- **Fixed:** the router read SGLang replicas with vLLM's metric names, and + SGLang was started without `--enable-metrics`, so it served no `/metrics` + at all: every scrape failed as though the replica were unreachable, quietly, + and requests were routed to it blind. SGLang is now started with + `--enable-metrics` and read in its own names — `sglang:num_running_reqs`, + `sglang:num_queue_reqs`, `sglang:token_usage` (the KV pool's fill), the + prompt and generation token counters and `sglang:num_requests_total`, as + 0.5.21 publishes them. +- **Fixed:** `llmgw ctl engine-check sglang` asked the interpreter for + `--help` and checked SGLang's flags against Python's. It now asks + `python -m sglang.launch_server --help`, in the environment SGLang was + installed into (or `--bin`). +- **Fixed:** SGLang was given `--tp`, which argparse accepts only as an + abbreviation of `--tp-size`; it is given `--tp-size`. +- **Fixed:** `llmgw ctl bench --metrics` read an engine's metrics with the + key given for `--target` — a client's key when the target is the gateway, + or none — so llama.cpp and TensorRT-LLM, which guard their metrics with the + engine's key, refused every read, and the run reported a queue depth of + zero and suggested no thresholds without saying why. The engine key is now + asked of the admin API for the replica at `--metrics`' address (or given + with the new `--metrics-key`); metrics that cannot be read — refused, or in + another engine's names than `--engine` — fail the run before any load, + saying why; and a step whose metrics went unread shows `-` and the reason + rather than zeros. + ### Reusing answers - **A response cache, per project, off until asked for.** `llmgw keys cache diff --git a/README.md b/README.md index c2570e7..d1be604 100644 --- a/README.md +++ b/README.md @@ -67,8 +67,10 @@ Since v0.11.0, not yet released (see [CHANGELOG.md](CHANGELOG.md), automatic rollback, quality suites gating updates, a response cache, kept and replayed requests, batches, Anthropic's Messages API, more of OpenAI's (structured output, images, rerank, audio, moderations), - scaling by the clock, SLO-aware admission, TensorRT-LLM, MIG, and models - imported from Ollama. + scaling by the clock, SLO-aware admission, TensorRT-LLM, MIG, models + imported from Ollama, routing by what each vLLM replica's cache holds, + LoRA adapters loaded on demand, other llmgw clusters as providers, and + placement that knows racks, PDUs and power caps. - **Operations** — `server restore` and off-machine backup copies, secrets in Vault, forgetting a person and keeping a project's requests in the cluster, offline installs from a bundle, a Helm chart, tracing from the diff --git a/notes/c24-facility.md b/notes/c24-facility.md deleted file mode 100644 index 531bd57..0000000 --- a/notes/c24-facility.md +++ /dev/null @@ -1,40 +0,0 @@ -## CHANGELOG - -### Facility-aware placement: racks, PDUs, holds and power caps - -- **Where a node is**, from five labels: `llmgw.io/site`, `room`, `row`, - `rack` and `pdu` (a list, for a dual-fed rack), in `agent.yaml` or set from - the control plane with **`llmgw ctl label NODE key=value… [key-…]`**, which - is kept in the store (migration 0031, `node_labels`), wins over the agent's - labels, works before the node joins, and refuses a misspelt `llmgw.io/` key. - A node with no location places as before. -- **Replicas spread across racks and PDUs.** A model's next replica goes to a - node sharing no rack or PDU with its others when there is one: an ordering - above weights already on disk and bin packing, never a filter. -- **`keep_within: rack`** (or row, room, site) on a profile keeps a - multi-node replica's nodes in one such domain. -- **Holds.** **`llmgw ctl facility hold rack:r12-07 --reason … [--drain]`** - places nothing new in a rack, row, room, site or PDU; `--drain` also drains - its nodes. **`llmgw ctl facility release`** undoes both. Kept across - restarts (migration 0031, `facility_holds`). -- **Power caps.** `facility.budgets` in `server.yaml` gives a domain's - `cap_watts` and `other_watts`, with a `margin`; a replica's draw is the - profile's `power_watts`, or its cards at the power limit they report - (`gpu_watts` otherwise) and its cores at `cpu_core_watts`, with `node_watts` - per node. Use is the larger of reserved and measured, node by node; a shard - that would take a domain past its cap less the margin is refused, and a - multi-node replica's shards are counted together. Unknown draw never blocks. -- **Seen everywhere it matters.** `ctl explain-placement` names the held - domain or the kilowatts, and keeps the rejections of a replica that was - placed, so one in an unexpected rack says why. `ctl nodes` has LOCATION, - *ready, held* and `--domain`. **`llmgw ctl facility`** lists every domain - with nodes, use, limit, reserved, measured and holds. The console's Nodes - view has WHERE; a node's menu has **Location…** and **Hold rack…/Release**. - Admin API: `GET /api/v1/facility`, `POST /api/v1/facility/holds`, - `DELETE /api/v1/facility/holds/{domain}`, `POST /api/v1/nodes/{id}/labels`. -- Docs: `docs/facility-aware-placement.md` says what is built and where it - departs from the design; `docs/install.md` and `docs/console.md`. - -## BACKLOG - -- [x] 24 (facility) — built: location labels (`ctl label`, agent.yaml), spreading across racks/PDUs, `keep_within`, held domains with drain (`ctl facility hold/release`, kept in the store), power caps from `facility.budgets` against declared/derived/measured draw, shown in explain-placement, `ctl nodes`, `ctl facility`, the console and the API. Left: temperature conditions with hysteresis and their enforcement (steps 3–4), a Prometheus/DCIM source, power headroom as a rack-levelling score, spreading multi-node replicas across racks, `llmgw_domain_*` metrics and alerts (near cap, over cap, stale data), shedding and tokens per watt (step 5), and a drained node's state surviving a restart. diff --git a/notes/c24-federation.md b/notes/c24-federation.md deleted file mode 100644 index deb7d27..0000000 --- a/notes/c24-federation.md +++ /dev/null @@ -1,40 +0,0 @@ -## CHANGELOG - -### Federating clusters: another llmgw cluster as a provider - -- `llmgw ctl federation add NAME --url URL --key-file FILE [--send-tags]` - makes another llmgw gateway a peer. It is checked first (its `/ready`, - timed, and its `/v1/models` with the key) and kept in the store (migration - 0029, `federation_peers`) with the models it offers. -- A peer's models join the catalog as `NAME/MODEL`, always prefixed, and - follow the peer's own list every `federation.interval` (10s). A model the - peer drops goes, unless a model here still falls back to it: then it is - kept withdrawn (in the catalog, sent nothing) until the fallback changes. - The peer's aliases and its own peers' models are not taken. -- A peer whose `/ready` fails, which refuses the key, or which does not answer - is taken out of routing until it answers again. `/ready` now counts only a - gateway's own endpoints, not its peers'. -- `llmgw ctl model fallbacks MODEL [FALLBACK...]` (and `PUT - /api/v1/models/{name}/fallbacks`) sets a runtime model's fallbacks, so - overflow can go to a peer's model. -- Requests to a peer carry `x-llmgw-hop` with the ids of the clusters they - have passed through (each cluster's id is kept in the store). A request that - comes back to a cluster is refused with 508 `federation_loop`, and a peer a - request came from is not sent it. -- The peer sees one key, this cluster's, and the request's `X-Request-Id`. - With `--send-tags`, also `x-llmgw-tags: federated_from=, - org=,team=,project=` (ids); off by default. -- In-cluster residency and `no_store` keep requests off peers, by name and as - fallbacks. Usage records the peer's model as served, and `peer:NAME/MODEL` - as the replica. -- `ctl federation list|remove|sync`, a Federation view in the console, - `llmgw_federation_peer_up`, `_latency_seconds` and `_models` metrics, admin - API `/api/v1/federation`, and a "Federating clusters" section in - docs/install.md. New `federation:` block in server.yaml (`name`, - `interval`). -- A response header the gateway already set (X-Request-Id, - x-llmgw-served-model) is no longer repeated from the upstream's. - -## BACKLOG - -- [x] 24. Multi-cluster federation — built: peers via `ctl federation` kept in the store, their models as PEER/MODEL synced from /v1/models (withdrawn-while-referenced), /ready health with latency and routing withdrawal, `ctl model fallbacks` for overflow, hop header loop refusal, residency/no_store hold, optional attribution tags, console view, metrics, docs. Left: server.yaml-declared models cannot name a peer model as a fallback (config catalog is validated before peers are read); per-peer CA / mTLS for peers behind a private CA; transitive federation (A→B→C models) deliberately not offered; latency is /ready RTT only (no per-request peer TTFT used in routing or weighting); the peer's adapters are not imported (listed as aliases); no console form to add/remove peers (CLI only); key file must exist on every control plane. diff --git a/notes/c24-kvindex.md b/notes/c24-kvindex.md deleted file mode 100644 index 48c1318..0000000 --- a/notes/c24-kvindex.md +++ /dev/null @@ -1,42 +0,0 @@ -# C24 — precise KV-index routing (plan §7.5 tier 2) - -## CHANGELOG - -### Routing by what each vLLM replica's cache holds - -- A vLLM profile can set `kv_events: true`. The agent then starts the engine - publishing its KV cache events (`--kv-events-config`) to a port on - 127.0.0.1 the agent listens on, with `--prefix-caching-hash-algo - sha256_cbor` and `PYTHONHASHSEED=1729`, so its block hashes can be - computed outside Python. The events never leave the node. -- The agent mirrors each engine's set of cached blocks and sends the net - changes up its existing stream to the control plane four times a second - (new `KVEvents` agent message), starting every new stream with a full - copy. Lost events (a gap in vLLM's sequence numbers), an engine restart or - `AllBlocksCleared` start a replica's index again; blocks offloaded to CPU, - remote ones and a hybrid model's second cache group are not indexed. -- The control plane passes them to the router's index, only for replicas - the sending node runs, and empties what a stream described when it closes. -- For a model with `kv_events`, the router asks one of its replicas to - tokenize the request (vLLM's `/tokenize`, with messages and tools, so the - chat template is applied exactly), hashes the tokens into blocks as vLLM - does, and scores replicas by the share of the prompt's leading blocks they - hold. The imbalance guard still applies. With no answer — no events yet, - an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls - back to the approximate tree (`router.prefix`) or to load. -- The agent checks first blocks against llmgw's hashing; an engine that - hashes another way (a profile overriding the algorithm, a changed vLLM) is - reported in the agent's log and `llmgw_kv_index_problem`, and routed as - though it published nothing. -- New settings `router.kv_index.tokenize_timeout` (250ms), `min_blocks` (4) - and `max_blocks` (4096). New metrics `llmgw_kv_index_decisions_total` - (`kv_hit`, `kv_miss_min_load`, `load_balance_override`, - `fallback_no_index`, `fallback_unsupported`, `fallback_tokenize_failed`), - `llmgw_kv_index_tokenize_seconds`, `llmgw_kv_index_blocks`. -- Experimental: tested against vLLM's event format and hashing as its source - defines them, and against libzmq and msgspec publishing — not yet against a - running vLLM. - -## BACKLOG - -- [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent (loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings) forwarded as net block-hash changes over the agent stream; router index, tokenize via the replica's `/tokenize`, sha256_cbor block-hash chain checked against cbor2/hashlib golden values; scorer by leading-block share with tier 1 fallback; opt-in `kv_events` per profile; metrics; docs. Left: validation against a real vLLM (event shape, hash agreement, `/tokenize` matching the served prompt, cost of tokenizing); vLLM's replay endpoint instead of resetting on a gap; data-parallel ranks (one publisher each, ports offset); hybrid models' cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt requests (extra keys); a router-only deployment (no agents) subscribing directly; standby control planes have no index until they lead; a per-replica fallback when only some candidates have an index; caching tokenizations of repeated prompts. diff --git a/notes/c24-lora.md b/notes/c24-lora.md deleted file mode 100644 index fd0be42..0000000 --- a/notes/c24-lora.md +++ /dev/null @@ -1,52 +0,0 @@ -## CHANGELOG - -### LoRA adapters on vLLM, loaded on demand - -- **vLLM models can carry adapters.** A PEFT adapter directory (a mirror - version ending in `/`, a path on the nodes, or a Hugging Face repository - id) is registered at start with `--enable-lora --lora-modules`, and a - request names it as its model, as on llama.cpp. `scale` applies on - llama.cpp only. SGLang and TensorRT-LLM profiles still refuse adapters. -- **`max_loaded_adapters: N` on a model** has each replica hold at most N of - its adapters. Replicas start holding none (vLLM with - `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras N`, unless the - profile's args set their own). The router loads an adapter on a replica - through `/v1/load_lora_adapter` when a request first wants it there, and - unloads the least recently used one no request is running on to make room. - vLLM profiles only; llama.cpp keeps loading every adapter at start, since - llama-server can rescale an adapter it started with but not load another. -- **Adapter affinity:** a request for an adapter goes to a replica that has - it, or is loading it, while that replica is not saturated; otherwise to one - with room for another adapter; otherwise where load sends it — which is how - a busy adapter spreads to a second replica. Requests arriving during a load - wait for it instead of loading the adapter again. -- **Self-correcting:** each replica's `/v1/models` is read when the router - first meets it and every 30 seconds, so adapters loaded before a restart or - by another router are known; a 404 for an adapter the router thought loaded - loads it again and retries. An adapter vLLM refuses (not found, rank above - `--max-lora-rank`) fails the request with vLLM's reason, without trying it - on every replica. -- **Metrics:** `llmgw_adapter_affinity_total{model,result=hit|joined|loaded}`, - `llmgw_adapter_loads_total{model,outcome}`, - `llmgw_adapter_evictions_total{model,outcome}`. -- The admin API and `ctl model adapters` read the file as GGUF only for a - model with llama.cpp profiles. - -## BACKLOG - -**24 LoRA affinity (branch `features-c2`):** built `max_loaded_adapters` -(catalog), vLLM adapter support at start and on demand (`engines/lora.go`, -`vllm.go`), the agent linking fetched adapters as `llmgw-lora/` in the -engines' working directory, the router's adapter tracker (`router/lora.go`: -loads/unloads with the engine key, LRU eviction never of an adapter in use, -one load per adapter per replica, `/v1/models` sync, 404 reload) and the -`adapter-affinity` scheduler filter in the chat, batch and long-context -profiles, with metrics and docs; tested against a fake of vLLM's HTTP -interface. **Left:** never run against a real vLLM — in particular that vLLM -resolves the relative `llmgw-lora/` path against its working directory -on every node (multi-node too) and that unloading an adapter mid-request is -harmless; the agent still fetches every adapter at replica start, so a new -adapter needs a restart (fetching on first load would need an agent command -and a proto change); load timeout and resync interval are constants, not -config; SGLang's runtime LoRA API is not driven; no per-replica "adapters -held" view in the console or admin API; the debug profile has no affinity. diff --git a/notes/engine-fixes.md b/notes/engine-fixes.md deleted file mode 100644 index fbd202d..0000000 --- a/notes/engine-fixes.md +++ /dev/null @@ -1,28 +0,0 @@ -## CHANGELOG - -### SGLang measured, checked and benchmarked as itself - -- **Fixed:** the router read SGLang replicas with vLLM's metric names, and - SGLang was started without `--enable-metrics`, so it served no `/metrics` - at all: every scrape failed as though the replica were unreachable, quietly, - and requests were routed to it blind. SGLang is now started with - `--enable-metrics` and read in its own names — `sglang:num_running_reqs`, - `sglang:num_queue_reqs`, `sglang:token_usage` (the KV pool's fill), the - prompt and generation token counters and `sglang:num_requests_total`, as - 0.5.21 publishes them. -- **Fixed:** `llmgw ctl engine-check sglang` asked the interpreter for - `--help` and checked SGLang's flags against Python's. It now asks - `python -m sglang.launch_server --help`, in the environment SGLang was - installed into (or `--bin`). -- **Fixed:** SGLang was given `--tp`, which argparse accepts only as an - abbreviation of `--tp-size`; it is given `--tp-size`. -- **Fixed:** `llmgw ctl bench --metrics` read an engine's metrics with the - key given for `--target` — a client's key when the target is the gateway, - or none — so llama.cpp and TensorRT-LLM, which guard their metrics with the - engine's key, refused every read, and the run reported a queue depth of - zero and suggested no thresholds without saying why. The engine key is now - asked of the admin API for the replica at `--metrics`' address (or given - with the new `--metrics-key`); metrics that cannot be read — refused, or in - another engine's names than `--engine` — fail the run before any load, - saying why; and a step whose metrics went unread shows `-` and the reason - rather than zeros.