CHANGELOG and BACKLOG for the rest of C24 and the engine fixes; README
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
33334c8f33
commit
803d1d5fb6
8 files changed
+227
-206
No files matched your search
+59
-2
@@ -679,11 +679,11 @@ portal is not done — the page shows; the console and ctl change.
|
||||
**More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF
|
||||
import. `agent/engines/engine.go` interface and the metric contract in
|
||||
`router/scrape`.
|
||||
24. Lower: precise KV-index routing (plan phase 6), LoRA affinity for
|
||||
24. **Done (branch `features-c2`), all six parts.** Lower: precise KV-index routing (plan phase 6), LoRA affinity for
|
||||
vLLM, multi-cluster federation via External providers, facility-aware
|
||||
placement (`docs/facility-aware-placement.md`), portal i18n, SDK/docs
|
||||
portal.
|
||||
Done so far (branch `features-c2`):
|
||||
What was built, and what is left:
|
||||
- 24 (portal i18n): built — English/Turkish catalogue with fallback, per-
|
||||
language templates, cookie/Accept-Language/`landing.language` choice
|
||||
and a footer switcher, Turkish numbers and dates, every template and
|
||||
@@ -701,6 +701,63 @@ portal is not done — the page shows; the console and ctl change.
|
||||
console's generated "Commands" page is not in the portal (it is built
|
||||
from the cobra tree, which internal/web cannot see); no search across
|
||||
documents.
|
||||
- [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent
|
||||
(loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings)
|
||||
forwarded as net block-hash changes over the agent stream; router
|
||||
index, tokenize via the replica's `/tokenize`, sha256_cbor block-hash
|
||||
chain checked against cbor2/hashlib golden values; scorer by leading-
|
||||
block share with tier 1 fallback; opt-in `kv_events` per profile;
|
||||
metrics; docs. Left: validation against a real vLLM (event shape, hash
|
||||
agreement, `/tokenize` matching the served prompt, cost of
|
||||
tokenizing); vLLM's replay endpoint instead of resetting on a gap;
|
||||
data-parallel ranks (one publisher each, ports offset); hybrid models'
|
||||
cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt
|
||||
requests (extra keys); a router-only deployment (no agents)
|
||||
subscribing directly; standby control planes have no index until they
|
||||
lead; a per-replica fallback when only some candidates have an index;
|
||||
caching tokenizations of repeated prompts.
|
||||
- **24 LoRA affinity (branch `features-c2`):** built `max_loaded_adapters`
|
||||
(catalog), vLLM adapter support at start and on demand
|
||||
(`engines/lora.go`, `vllm.go`), the agent linking fetched adapters as
|
||||
`llmgw-lora/<name>` in the engines' working directory, the router's
|
||||
adapter tracker (`router/lora.go`: loads/unloads with the engine key,
|
||||
LRU eviction never of an adapter in use, one load per adapter per
|
||||
replica, `/v1/models` sync, 404 reload) and the `adapter-affinity`
|
||||
scheduler filter in the chat, batch and long-context profiles, with
|
||||
metrics and docs; tested against a fake of vLLM's HTTP interface.
|
||||
**Left:** never run against a real vLLM — in particular that vLLM
|
||||
resolves the relative `llmgw-lora/<name>` path against its working
|
||||
directory on every node (multi-node too) and that unloading an adapter
|
||||
mid-request is harmless; the agent still fetches every adapter at
|
||||
replica start, so a new adapter needs a restart (fetching on first
|
||||
load would need an agent command and a proto change); load timeout and
|
||||
resync interval are constants, not config; SGLang's runtime LoRA API
|
||||
is not driven; no per-replica "adapters held" view in the console or
|
||||
admin API; the debug profile has no affinity.
|
||||
- 24. Multi-cluster federation — built: peers via `ctl federation` kept in
|
||||
the store, their models as PEER/MODEL synced from /v1/models
|
||||
(withdrawn-while-referenced), /ready health with latency and routing
|
||||
withdrawal, `ctl model fallbacks` for overflow, hop header loop
|
||||
refusal, residency/no_store hold, optional attribution tags, console
|
||||
view, metrics, docs. Left: server.yaml-declared models cannot name a
|
||||
peer model as a fallback (config catalog is validated before peers are
|
||||
read); per-peer CA / mTLS for peers behind a private CA; transitive
|
||||
federation (A→B→C models) deliberately not offered; latency is /ready
|
||||
RTT only (no per-request peer TTFT used in routing or weighting); the
|
||||
peer's adapters are not imported (listed as aliases); no console form
|
||||
to add/remove peers (CLI only); key file must exist on every control
|
||||
plane.
|
||||
- 24 (facility) — built: location labels (`ctl label`, agent.yaml),
|
||||
spreading across racks/PDUs, `keep_within`, held domains with drain
|
||||
(`ctl facility hold/release`, kept in the store), power caps from
|
||||
`facility.budgets` against declared/derived/measured draw, shown in
|
||||
explain-placement, `ctl nodes`, `ctl facility`, the console and the
|
||||
API. Left: temperature conditions with hysteresis and their
|
||||
enforcement (steps 3–4), a Prometheus/DCIM source, power headroom as a
|
||||
rack-levelling score, spreading multi-node replicas across racks,
|
||||
`llmgw_domain_*` metrics and alerts (near cap, over cap, stale data),
|
||||
shedding and tokens per watt (step 5), and a drained node's state
|
||||
surviving a restart.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+164
@@ -23,6 +23,170 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Routing by what each vLLM replica's cache holds
|
||||
|
||||
- A vLLM profile can set `kv_events: true`. The agent then starts the engine
|
||||
publishing its KV cache events (`--kv-events-config`) to a port on
|
||||
127.0.0.1 the agent listens on, with `--prefix-caching-hash-algo
|
||||
sha256_cbor` and `PYTHONHASHSEED=1729`, so its block hashes can be
|
||||
computed outside Python. The events never leave the node.
|
||||
- The agent mirrors each engine's set of cached blocks and sends the net
|
||||
changes up its existing stream to the control plane four times a second
|
||||
(new `KVEvents` agent message), starting every new stream with a full
|
||||
copy. Lost events (a gap in vLLM's sequence numbers), an engine restart or
|
||||
`AllBlocksCleared` start a replica's index again; blocks offloaded to CPU,
|
||||
remote ones and a hybrid model's second cache group are not indexed.
|
||||
- The control plane passes them to the router's index, only for replicas
|
||||
the sending node runs, and empties what a stream described when it closes.
|
||||
- For a model with `kv_events`, the router asks one of its replicas to
|
||||
tokenize the request (vLLM's `/tokenize`, with messages and tools, so the
|
||||
chat template is applied exactly), hashes the tokens into blocks as vLLM
|
||||
does, and scores replicas by the share of the prompt's leading blocks they
|
||||
hold. The imbalance guard still applies. With no answer — no events yet,
|
||||
an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls
|
||||
back to the approximate tree (`router.prefix`) or to load.
|
||||
- The agent checks first blocks against llmgw's hashing; an engine that
|
||||
hashes another way (a profile overriding the algorithm, a changed vLLM) is
|
||||
reported in the agent's log and `llmgw_kv_index_problem`, and routed as
|
||||
though it published nothing.
|
||||
- New settings `router.kv_index.tokenize_timeout` (250ms), `min_blocks` (4)
|
||||
and `max_blocks` (4096). New metrics `llmgw_kv_index_decisions_total`
|
||||
(`kv_hit`, `kv_miss_min_load`, `load_balance_override`,
|
||||
`fallback_no_index`, `fallback_unsupported`, `fallback_tokenize_failed`),
|
||||
`llmgw_kv_index_tokenize_seconds`, `llmgw_kv_index_blocks`.
|
||||
- Experimental: tested against vLLM's event format and hashing as its source
|
||||
defines them, and against libzmq and msgspec publishing — not yet against a
|
||||
running vLLM.
|
||||
|
||||
### LoRA adapters on vLLM, loaded on demand
|
||||
|
||||
- **vLLM models can carry adapters.** A PEFT adapter directory (a mirror
|
||||
version ending in `/`, a path on the nodes, or a Hugging Face repository
|
||||
id) is registered at start with `--enable-lora --lora-modules`, and a
|
||||
request names it as its model, as on llama.cpp. `scale` applies on
|
||||
llama.cpp only. SGLang and TensorRT-LLM profiles still refuse adapters.
|
||||
- **`max_loaded_adapters: N` on a model** has each replica hold at most N of
|
||||
its adapters. Replicas start holding none (vLLM with
|
||||
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras N`, unless the
|
||||
profile's args set their own). The router loads an adapter on a replica
|
||||
through `/v1/load_lora_adapter` when a request first wants it there, and
|
||||
unloads the least recently used one no request is running on to make room.
|
||||
vLLM profiles only; llama.cpp keeps loading every adapter at start, since
|
||||
llama-server can rescale an adapter it started with but not load another.
|
||||
- **Adapter affinity:** a request for an adapter goes to a replica that has
|
||||
it, or is loading it, while that replica is not saturated; otherwise to one
|
||||
with room for another adapter; otherwise where load sends it — which is how
|
||||
a busy adapter spreads to a second replica. Requests arriving during a load
|
||||
wait for it instead of loading the adapter again.
|
||||
- **Self-correcting:** each replica's `/v1/models` is read when the router
|
||||
first meets it and every 30 seconds, so adapters loaded before a restart or
|
||||
by another router are known; a 404 for an adapter the router thought loaded
|
||||
loads it again and retries. An adapter vLLM refuses (not found, rank above
|
||||
`--max-lora-rank`) fails the request with vLLM's reason, without trying it
|
||||
on every replica.
|
||||
- **Metrics:** `llmgw_adapter_affinity_total{model,result=hit|joined|loaded}`,
|
||||
`llmgw_adapter_loads_total{model,outcome}`,
|
||||
`llmgw_adapter_evictions_total{model,outcome}`.
|
||||
- The admin API and `ctl model adapters` read the file as GGUF only for a
|
||||
model with llama.cpp profiles.
|
||||
|
||||
### Federating clusters: another llmgw cluster as a provider
|
||||
|
||||
- `llmgw ctl federation add NAME --url URL --key-file FILE [--send-tags]`
|
||||
makes another llmgw gateway a peer. It is checked first (its `/ready`,
|
||||
timed, and its `/v1/models` with the key) and kept in the store (migration
|
||||
0029, `federation_peers`) with the models it offers.
|
||||
- A peer's models join the catalog as `NAME/MODEL`, always prefixed, and
|
||||
follow the peer's own list every `federation.interval` (10s). A model the
|
||||
peer drops goes, unless a model here still falls back to it: then it is
|
||||
kept withdrawn (in the catalog, sent nothing) until the fallback changes.
|
||||
The peer's aliases and its own peers' models are not taken.
|
||||
- A peer whose `/ready` fails, which refuses the key, or which does not answer
|
||||
is taken out of routing until it answers again. `/ready` now counts only a
|
||||
gateway's own endpoints, not its peers'.
|
||||
- `llmgw ctl model fallbacks MODEL [FALLBACK...]` (and `PUT
|
||||
/api/v1/models/{name}/fallbacks`) sets a runtime model's fallbacks, so
|
||||
overflow can go to a peer's model.
|
||||
- Requests to a peer carry `x-llmgw-hop` with the ids of the clusters they
|
||||
have passed through (each cluster's id is kept in the store). A request that
|
||||
comes back to a cluster is refused with 508 `federation_loop`, and a peer a
|
||||
request came from is not sent it.
|
||||
- The peer sees one key, this cluster's, and the request's `X-Request-Id`.
|
||||
With `--send-tags`, also `x-llmgw-tags: federated_from=<federation.name>,
|
||||
org=,team=,project=` (ids); off by default.
|
||||
- In-cluster residency and `no_store` keep requests off peers, by name and as
|
||||
fallbacks. Usage records the peer's model as served, and `peer:NAME/MODEL`
|
||||
as the replica.
|
||||
- `ctl federation list|remove|sync`, a Federation view in the console,
|
||||
`llmgw_federation_peer_up`, `_latency_seconds` and `_models` metrics, admin
|
||||
API `/api/v1/federation`, and a "Federating clusters" section in
|
||||
docs/install.md. New `federation:` block in server.yaml (`name`,
|
||||
`interval`).
|
||||
- A response header the gateway already set (X-Request-Id,
|
||||
x-llmgw-served-model) is no longer repeated from the upstream's.
|
||||
|
||||
### Facility-aware placement: racks, PDUs, holds and power caps
|
||||
|
||||
- **Where a node is**, from five labels: `llmgw.io/site`, `room`, `row`,
|
||||
`rack` and `pdu` (a list, for a dual-fed rack), in `agent.yaml` or set from
|
||||
the control plane with **`llmgw ctl label NODE key=value… [key-…]`**, which
|
||||
is kept in the store (migration 0031, `node_labels`), wins over the agent's
|
||||
labels, works before the node joins, and refuses a misspelt `llmgw.io/` key.
|
||||
A node with no location places as before.
|
||||
- **Replicas spread across racks and PDUs.** A model's next replica goes to a
|
||||
node sharing no rack or PDU with its others when there is one: an ordering
|
||||
above weights already on disk and bin packing, never a filter.
|
||||
- **`keep_within: rack`** (or row, room, site) on a profile keeps a
|
||||
multi-node replica's nodes in one such domain.
|
||||
- **Holds.** **`llmgw ctl facility hold rack:r12-07 --reason … [--drain]`**
|
||||
places nothing new in a rack, row, room, site or PDU; `--drain` also drains
|
||||
its nodes. **`llmgw ctl facility release`** undoes both. Kept across
|
||||
restarts (migration 0031, `facility_holds`).
|
||||
- **Power caps.** `facility.budgets` in `server.yaml` gives a domain's
|
||||
`cap_watts` and `other_watts`, with a `margin`; a replica's draw is the
|
||||
profile's `power_watts`, or its cards at the power limit they report
|
||||
(`gpu_watts` otherwise) and its cores at `cpu_core_watts`, with `node_watts`
|
||||
per node. Use is the larger of reserved and measured, node by node; a shard
|
||||
that would take a domain past its cap less the margin is refused, and a
|
||||
multi-node replica's shards are counted together. Unknown draw never blocks.
|
||||
- **Seen everywhere it matters.** `ctl explain-placement` names the held
|
||||
domain or the kilowatts, and keeps the rejections of a replica that was
|
||||
placed, so one in an unexpected rack says why. `ctl nodes` has LOCATION,
|
||||
*ready, held* and `--domain`. **`llmgw ctl facility`** lists every domain
|
||||
with nodes, use, limit, reserved, measured and holds. The console's Nodes
|
||||
view has WHERE; a node's menu has **Location…** and **Hold rack…/Release**.
|
||||
Admin API: `GET /api/v1/facility`, `POST /api/v1/facility/holds`,
|
||||
`DELETE /api/v1/facility/holds/{domain}`, `POST /api/v1/nodes/{id}/labels`.
|
||||
- Docs: `docs/facility-aware-placement.md` says what is built and where it
|
||||
departs from the design; `docs/install.md` and `docs/console.md`.
|
||||
|
||||
### SGLang measured, checked and benchmarked as itself
|
||||
|
||||
- **Fixed:** the router read SGLang replicas with vLLM's metric names, and
|
||||
SGLang was started without `--enable-metrics`, so it served no `/metrics`
|
||||
at all: every scrape failed as though the replica were unreachable, quietly,
|
||||
and requests were routed to it blind. SGLang is now started with
|
||||
`--enable-metrics` and read in its own names — `sglang:num_running_reqs`,
|
||||
`sglang:num_queue_reqs`, `sglang:token_usage` (the KV pool's fill), the
|
||||
prompt and generation token counters and `sglang:num_requests_total`, as
|
||||
0.5.21 publishes them.
|
||||
- **Fixed:** `llmgw ctl engine-check sglang` asked the interpreter for
|
||||
`--help` and checked SGLang's flags against Python's. It now asks
|
||||
`python -m sglang.launch_server --help`, in the environment SGLang was
|
||||
installed into (or `--bin`).
|
||||
- **Fixed:** SGLang was given `--tp`, which argparse accepts only as an
|
||||
abbreviation of `--tp-size`; it is given `--tp-size`.
|
||||
- **Fixed:** `llmgw ctl bench --metrics` read an engine's metrics with the
|
||||
key given for `--target` — a client's key when the target is the gateway,
|
||||
or none — so llama.cpp and TensorRT-LLM, which guard their metrics with the
|
||||
engine's key, refused every read, and the run reported a queue depth of
|
||||
zero and suggested no thresholds without saying why. The engine key is now
|
||||
asked of the admin API for the replica at `--metrics`' address (or given
|
||||
with the new `--metrics-key`); metrics that cannot be read — refused, or in
|
||||
another engine's names than `--engine` — fail the run before any load,
|
||||
saying why; and a step whose metrics went unread shows `-` and the reason
|
||||
rather than zeros.
|
||||
|
||||
### Reusing answers
|
||||
|
||||
- **A response cache, per project, off until asked for.** `llmgw keys cache
|
||||
|
||||
@@ -67,8 +67,10 @@ Since v0.11.0, not yet released (see [CHANGELOG.md](CHANGELOG.md),
|
||||
automatic rollback, quality suites gating updates, a response cache,
|
||||
kept and replayed requests, batches, Anthropic's Messages API, more of
|
||||
OpenAI's (structured output, images, rerank, audio, moderations),
|
||||
scaling by the clock, SLO-aware admission, TensorRT-LLM, MIG, and models
|
||||
imported from Ollama.
|
||||
scaling by the clock, SLO-aware admission, TensorRT-LLM, MIG, models
|
||||
imported from Ollama, routing by what each vLLM replica's cache holds,
|
||||
LoRA adapters loaded on demand, other llmgw clusters as providers, and
|
||||
placement that knows racks, PDUs and power caps.
|
||||
- **Operations** — `server restore` and off-machine backup copies, secrets
|
||||
in Vault, forgetting a person and keeping a project's requests in the
|
||||
cluster, offline installs from a bundle, a Helm chart, tracing from the
|
||||
|
||||
@@ -1,40 +0,0 @@
|
||||
## CHANGELOG
|
||||
|
||||
### Facility-aware placement: racks, PDUs, holds and power caps
|
||||
|
||||
- **Where a node is**, from five labels: `llmgw.io/site`, `room`, `row`,
|
||||
`rack` and `pdu` (a list, for a dual-fed rack), in `agent.yaml` or set from
|
||||
the control plane with **`llmgw ctl label NODE key=value… [key-…]`**, which
|
||||
is kept in the store (migration 0031, `node_labels`), wins over the agent's
|
||||
labels, works before the node joins, and refuses a misspelt `llmgw.io/` key.
|
||||
A node with no location places as before.
|
||||
- **Replicas spread across racks and PDUs.** A model's next replica goes to a
|
||||
node sharing no rack or PDU with its others when there is one: an ordering
|
||||
above weights already on disk and bin packing, never a filter.
|
||||
- **`keep_within: rack`** (or row, room, site) on a profile keeps a
|
||||
multi-node replica's nodes in one such domain.
|
||||
- **Holds.** **`llmgw ctl facility hold rack:r12-07 --reason … [--drain]`**
|
||||
places nothing new in a rack, row, room, site or PDU; `--drain` also drains
|
||||
its nodes. **`llmgw ctl facility release`** undoes both. Kept across
|
||||
restarts (migration 0031, `facility_holds`).
|
||||
- **Power caps.** `facility.budgets` in `server.yaml` gives a domain's
|
||||
`cap_watts` and `other_watts`, with a `margin`; a replica's draw is the
|
||||
profile's `power_watts`, or its cards at the power limit they report
|
||||
(`gpu_watts` otherwise) and its cores at `cpu_core_watts`, with `node_watts`
|
||||
per node. Use is the larger of reserved and measured, node by node; a shard
|
||||
that would take a domain past its cap less the margin is refused, and a
|
||||
multi-node replica's shards are counted together. Unknown draw never blocks.
|
||||
- **Seen everywhere it matters.** `ctl explain-placement` names the held
|
||||
domain or the kilowatts, and keeps the rejections of a replica that was
|
||||
placed, so one in an unexpected rack says why. `ctl nodes` has LOCATION,
|
||||
*ready, held* and `--domain`. **`llmgw ctl facility`** lists every domain
|
||||
with nodes, use, limit, reserved, measured and holds. The console's Nodes
|
||||
view has WHERE; a node's menu has **Location…** and **Hold rack…/Release**.
|
||||
Admin API: `GET /api/v1/facility`, `POST /api/v1/facility/holds`,
|
||||
`DELETE /api/v1/facility/holds/{domain}`, `POST /api/v1/nodes/{id}/labels`.
|
||||
- Docs: `docs/facility-aware-placement.md` says what is built and where it
|
||||
departs from the design; `docs/install.md` and `docs/console.md`.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
- [x] 24 (facility) — built: location labels (`ctl label`, agent.yaml), spreading across racks/PDUs, `keep_within`, held domains with drain (`ctl facility hold/release`, kept in the store), power caps from `facility.budgets` against declared/derived/measured draw, shown in explain-placement, `ctl nodes`, `ctl facility`, the console and the API. Left: temperature conditions with hysteresis and their enforcement (steps 3–4), a Prometheus/DCIM source, power headroom as a rack-levelling score, spreading multi-node replicas across racks, `llmgw_domain_*` metrics and alerts (near cap, over cap, stale data), shedding and tokens per watt (step 5), and a drained node's state surviving a restart.
|
||||
@@ -1,40 +0,0 @@
|
||||
## CHANGELOG
|
||||
|
||||
### Federating clusters: another llmgw cluster as a provider
|
||||
|
||||
- `llmgw ctl federation add NAME --url URL --key-file FILE [--send-tags]`
|
||||
makes another llmgw gateway a peer. It is checked first (its `/ready`,
|
||||
timed, and its `/v1/models` with the key) and kept in the store (migration
|
||||
0029, `federation_peers`) with the models it offers.
|
||||
- A peer's models join the catalog as `NAME/MODEL`, always prefixed, and
|
||||
follow the peer's own list every `federation.interval` (10s). A model the
|
||||
peer drops goes, unless a model here still falls back to it: then it is
|
||||
kept withdrawn (in the catalog, sent nothing) until the fallback changes.
|
||||
The peer's aliases and its own peers' models are not taken.
|
||||
- A peer whose `/ready` fails, which refuses the key, or which does not answer
|
||||
is taken out of routing until it answers again. `/ready` now counts only a
|
||||
gateway's own endpoints, not its peers'.
|
||||
- `llmgw ctl model fallbacks MODEL [FALLBACK...]` (and `PUT
|
||||
/api/v1/models/{name}/fallbacks`) sets a runtime model's fallbacks, so
|
||||
overflow can go to a peer's model.
|
||||
- Requests to a peer carry `x-llmgw-hop` with the ids of the clusters they
|
||||
have passed through (each cluster's id is kept in the store). A request that
|
||||
comes back to a cluster is refused with 508 `federation_loop`, and a peer a
|
||||
request came from is not sent it.
|
||||
- The peer sees one key, this cluster's, and the request's `X-Request-Id`.
|
||||
With `--send-tags`, also `x-llmgw-tags: federated_from=<federation.name>,
|
||||
org=,team=,project=` (ids); off by default.
|
||||
- In-cluster residency and `no_store` keep requests off peers, by name and as
|
||||
fallbacks. Usage records the peer's model as served, and `peer:NAME/MODEL`
|
||||
as the replica.
|
||||
- `ctl federation list|remove|sync`, a Federation view in the console,
|
||||
`llmgw_federation_peer_up`, `_latency_seconds` and `_models` metrics, admin
|
||||
API `/api/v1/federation`, and a "Federating clusters" section in
|
||||
docs/install.md. New `federation:` block in server.yaml (`name`,
|
||||
`interval`).
|
||||
- A response header the gateway already set (X-Request-Id,
|
||||
x-llmgw-served-model) is no longer repeated from the upstream's.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
- [x] 24. Multi-cluster federation — built: peers via `ctl federation` kept in the store, their models as PEER/MODEL synced from /v1/models (withdrawn-while-referenced), /ready health with latency and routing withdrawal, `ctl model fallbacks` for overflow, hop header loop refusal, residency/no_store hold, optional attribution tags, console view, metrics, docs. Left: server.yaml-declared models cannot name a peer model as a fallback (config catalog is validated before peers are read); per-peer CA / mTLS for peers behind a private CA; transitive federation (A→B→C models) deliberately not offered; latency is /ready RTT only (no per-request peer TTFT used in routing or weighting); the peer's adapters are not imported (listed as aliases); no console form to add/remove peers (CLI only); key file must exist on every control plane.
|
||||
@@ -1,42 +0,0 @@
|
||||
# C24 — precise KV-index routing (plan §7.5 tier 2)
|
||||
|
||||
## CHANGELOG
|
||||
|
||||
### Routing by what each vLLM replica's cache holds
|
||||
|
||||
- A vLLM profile can set `kv_events: true`. The agent then starts the engine
|
||||
publishing its KV cache events (`--kv-events-config`) to a port on
|
||||
127.0.0.1 the agent listens on, with `--prefix-caching-hash-algo
|
||||
sha256_cbor` and `PYTHONHASHSEED=1729`, so its block hashes can be
|
||||
computed outside Python. The events never leave the node.
|
||||
- The agent mirrors each engine's set of cached blocks and sends the net
|
||||
changes up its existing stream to the control plane four times a second
|
||||
(new `KVEvents` agent message), starting every new stream with a full
|
||||
copy. Lost events (a gap in vLLM's sequence numbers), an engine restart or
|
||||
`AllBlocksCleared` start a replica's index again; blocks offloaded to CPU,
|
||||
remote ones and a hybrid model's second cache group are not indexed.
|
||||
- The control plane passes them to the router's index, only for replicas
|
||||
the sending node runs, and empties what a stream described when it closes.
|
||||
- For a model with `kv_events`, the router asks one of its replicas to
|
||||
tokenize the request (vLLM's `/tokenize`, with messages and tools, so the
|
||||
chat template is applied exactly), hashes the tokens into blocks as vLLM
|
||||
does, and scores replicas by the share of the prompt's leading blocks they
|
||||
hold. The imbalance guard still applies. With no answer — no events yet,
|
||||
an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls
|
||||
back to the approximate tree (`router.prefix`) or to load.
|
||||
- The agent checks first blocks against llmgw's hashing; an engine that
|
||||
hashes another way (a profile overriding the algorithm, a changed vLLM) is
|
||||
reported in the agent's log and `llmgw_kv_index_problem`, and routed as
|
||||
though it published nothing.
|
||||
- New settings `router.kv_index.tokenize_timeout` (250ms), `min_blocks` (4)
|
||||
and `max_blocks` (4096). New metrics `llmgw_kv_index_decisions_total`
|
||||
(`kv_hit`, `kv_miss_min_load`, `load_balance_override`,
|
||||
`fallback_no_index`, `fallback_unsupported`, `fallback_tokenize_failed`),
|
||||
`llmgw_kv_index_tokenize_seconds`, `llmgw_kv_index_blocks`.
|
||||
- Experimental: tested against vLLM's event format and hashing as its source
|
||||
defines them, and against libzmq and msgspec publishing — not yet against a
|
||||
running vLLM.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
- [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent (loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings) forwarded as net block-hash changes over the agent stream; router index, tokenize via the replica's `/tokenize`, sha256_cbor block-hash chain checked against cbor2/hashlib golden values; scorer by leading-block share with tier 1 fallback; opt-in `kv_events` per profile; metrics; docs. Left: validation against a real vLLM (event shape, hash agreement, `/tokenize` matching the served prompt, cost of tokenizing); vLLM's replay endpoint instead of resetting on a gap; data-parallel ranks (one publisher each, ports offset); hybrid models' cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt requests (extra keys); a router-only deployment (no agents) subscribing directly; standby control planes have no index until they lead; a per-replica fallback when only some candidates have an index; caching tokenizations of repeated prompts.
|
||||
@@ -1,52 +0,0 @@
|
||||
## CHANGELOG
|
||||
|
||||
### LoRA adapters on vLLM, loaded on demand
|
||||
|
||||
- **vLLM models can carry adapters.** A PEFT adapter directory (a mirror
|
||||
version ending in `/`, a path on the nodes, or a Hugging Face repository
|
||||
id) is registered at start with `--enable-lora --lora-modules`, and a
|
||||
request names it as its model, as on llama.cpp. `scale` applies on
|
||||
llama.cpp only. SGLang and TensorRT-LLM profiles still refuse adapters.
|
||||
- **`max_loaded_adapters: N` on a model** has each replica hold at most N of
|
||||
its adapters. Replicas start holding none (vLLM with
|
||||
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras N`, unless the
|
||||
profile's args set their own). The router loads an adapter on a replica
|
||||
through `/v1/load_lora_adapter` when a request first wants it there, and
|
||||
unloads the least recently used one no request is running on to make room.
|
||||
vLLM profiles only; llama.cpp keeps loading every adapter at start, since
|
||||
llama-server can rescale an adapter it started with but not load another.
|
||||
- **Adapter affinity:** a request for an adapter goes to a replica that has
|
||||
it, or is loading it, while that replica is not saturated; otherwise to one
|
||||
with room for another adapter; otherwise where load sends it — which is how
|
||||
a busy adapter spreads to a second replica. Requests arriving during a load
|
||||
wait for it instead of loading the adapter again.
|
||||
- **Self-correcting:** each replica's `/v1/models` is read when the router
|
||||
first meets it and every 30 seconds, so adapters loaded before a restart or
|
||||
by another router are known; a 404 for an adapter the router thought loaded
|
||||
loads it again and retries. An adapter vLLM refuses (not found, rank above
|
||||
`--max-lora-rank`) fails the request with vLLM's reason, without trying it
|
||||
on every replica.
|
||||
- **Metrics:** `llmgw_adapter_affinity_total{model,result=hit|joined|loaded}`,
|
||||
`llmgw_adapter_loads_total{model,outcome}`,
|
||||
`llmgw_adapter_evictions_total{model,outcome}`.
|
||||
- The admin API and `ctl model adapters` read the file as GGUF only for a
|
||||
model with llama.cpp profiles.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**24 LoRA affinity (branch `features-c2`):** built `max_loaded_adapters`
|
||||
(catalog), vLLM adapter support at start and on demand (`engines/lora.go`,
|
||||
`vllm.go`), the agent linking fetched adapters as `llmgw-lora/<name>` in the
|
||||
engines' working directory, the router's adapter tracker (`router/lora.go`:
|
||||
loads/unloads with the engine key, LRU eviction never of an adapter in use,
|
||||
one load per adapter per replica, `/v1/models` sync, 404 reload) and the
|
||||
`adapter-affinity` scheduler filter in the chat, batch and long-context
|
||||
profiles, with metrics and docs; tested against a fake of vLLM's HTTP
|
||||
interface. **Left:** never run against a real vLLM — in particular that vLLM
|
||||
resolves the relative `llmgw-lora/<name>` path against its working directory
|
||||
on every node (multi-node too) and that unloading an adapter mid-request is
|
||||
harmless; the agent still fetches every adapter at replica start, so a new
|
||||
adapter needs a restart (fetching on first load would need an agent command
|
||||
and a proto change); load timeout and resync interval are constants, not
|
||||
config; SGLang's runtime LoRA API is not driven; no per-replica "adapters
|
||||
held" view in the console or admin API; the debug profile has no affinity.
|
||||
@@ -1,28 +0,0 @@
|
||||
## CHANGELOG
|
||||
|
||||
### SGLang measured, checked and benchmarked as itself
|
||||
|
||||
- **Fixed:** the router read SGLang replicas with vLLM's metric names, and
|
||||
SGLang was started without `--enable-metrics`, so it served no `/metrics`
|
||||
at all: every scrape failed as though the replica were unreachable, quietly,
|
||||
and requests were routed to it blind. SGLang is now started with
|
||||
`--enable-metrics` and read in its own names — `sglang:num_running_reqs`,
|
||||
`sglang:num_queue_reqs`, `sglang:token_usage` (the KV pool's fill), the
|
||||
prompt and generation token counters and `sglang:num_requests_total`, as
|
||||
0.5.21 publishes them.
|
||||
- **Fixed:** `llmgw ctl engine-check sglang` asked the interpreter for
|
||||
`--help` and checked SGLang's flags against Python's. It now asks
|
||||
`python -m sglang.launch_server --help`, in the environment SGLang was
|
||||
installed into (or `--bin`).
|
||||
- **Fixed:** SGLang was given `--tp`, which argparse accepts only as an
|
||||
abbreviation of `--tp-size`; it is given `--tp-size`.
|
||||
- **Fixed:** `llmgw ctl bench --metrics` read an engine's metrics with the
|
||||
key given for `--target` — a client's key when the target is the gateway,
|
||||
or none — so llama.cpp and TensorRT-LLM, which guard their metrics with the
|
||||
engine's key, refused every read, and the run reported a queue depth of
|
||||
zero and suggested no thresholds without saying why. The engine key is now
|
||||
asked of the admin API for the replica at `--metrics`' address (or given
|
||||
with the new `--metrics-key`); metrics that cannot be read — refused, or in
|
||||
another engine's names than `--engine` — fail the run before any load,
|
||||
saying why; and a step whose metrics went unread shows `-` and the reason
|
||||
rather than zeros.
|
||||
Reference in new issue
Block a user