CHANGELOG and BACKLOG for the rest of C24 and the engine fixes; README

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 01:30:01 +02:00
1 parent 33334c8f33
commit 803d1d5fb6
8 files changed
+227 -206

No files matched your search

+59 -2
View File
@@ -679,11 +679,11 @@ portal is not done — the page shows; the console and ctl change.
**More engines (M each).** TensorRT-LLM/Triton adapter; Ollama/GGUF
import. `agent/engines/engine.go` interface and the metric contract in
`router/scrape`.
24. Lower: precise KV-index routing (plan phase 6), LoRA affinity for
24. **Done (branch `features-c2`), all six parts.** Lower: precise KV-index routing (plan phase 6), LoRA affinity for
vLLM, multi-cluster federation via External providers, facility-aware
placement (`docs/facility-aware-placement.md`), portal i18n, SDK/docs
portal.
Done so far (branch `features-c2`):
What was built, and what is left:
- 24 (portal i18n): built — English/Turkish catalogue with fallback, per-
language templates, cookie/Accept-Language/`landing.language` choice
and a footer switcher, Turkish numbers and dates, every template and
@@ -701,6 +701,63 @@ portal is not done — the page shows; the console and ctl change.
console's generated "Commands" page is not in the portal (it is built
from the cobra tree, which internal/web cannot see); no search across
documents.
- [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent
(loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings)
forwarded as net block-hash changes over the agent stream; router
index, tokenize via the replica's `/tokenize`, sha256_cbor block-hash
chain checked against cbor2/hashlib golden values; scorer by leading-
block share with tier 1 fallback; opt-in `kv_events` per profile;
metrics; docs. Left: validation against a real vLLM (event shape, hash
agreement, `/tokenize` matching the served prompt, cost of
tokenizing); vLLM's replay endpoint instead of resetting on a gap;
data-parallel ranks (one publisher each, ports offset); hybrid models'
cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt
requests (extra keys); a router-only deployment (no agents)
subscribing directly; standby control planes have no index until they
lead; a per-replica fallback when only some candidates have an index;
caching tokenizations of repeated prompts.
- **24 LoRA affinity (branch `features-c2`):** built `max_loaded_adapters`
(catalog), vLLM adapter support at start and on demand
(`engines/lora.go`, `vllm.go`), the agent linking fetched adapters as
`llmgw-lora/<name>` in the engines' working directory, the router's
adapter tracker (`router/lora.go`: loads/unloads with the engine key,
LRU eviction never of an adapter in use, one load per adapter per
replica, `/v1/models` sync, 404 reload) and the `adapter-affinity`
scheduler filter in the chat, batch and long-context profiles, with
metrics and docs; tested against a fake of vLLM's HTTP interface.
**Left:** never run against a real vLLM — in particular that vLLM
resolves the relative `llmgw-lora/<name>` path against its working
directory on every node (multi-node too) and that unloading an adapter
mid-request is harmless; the agent still fetches every adapter at
replica start, so a new adapter needs a restart (fetching on first
load would need an agent command and a proto change); load timeout and
resync interval are constants, not config; SGLang's runtime LoRA API
is not driven; no per-replica "adapters held" view in the console or
admin API; the debug profile has no affinity.
- 24. Multi-cluster federation — built: peers via `ctl federation` kept in
the store, their models as PEER/MODEL synced from /v1/models
(withdrawn-while-referenced), /ready health with latency and routing
withdrawal, `ctl model fallbacks` for overflow, hop header loop
refusal, residency/no_store hold, optional attribution tags, console
view, metrics, docs. Left: server.yaml-declared models cannot name a
peer model as a fallback (config catalog is validated before peers are
read); per-peer CA / mTLS for peers behind a private CA; transitive
federation (A→B→C models) deliberately not offered; latency is /ready
RTT only (no per-request peer TTFT used in routing or weighting); the
peer's adapters are not imported (listed as aliases); no console form
to add/remove peers (CLI only); key file must exist on every control
plane.
- 24 (facility) — built: location labels (`ctl label`, agent.yaml),
spreading across racks/PDUs, `keep_within`, held domains with drain
(`ctl facility hold/release`, kept in the store), power caps from
`facility.budgets` against declared/derived/measured draw, shown in
explain-placement, `ctl nodes`, `ctl facility`, the console and the
API. Left: temperature conditions with hysteresis and their
enforcement (steps 3–4), a Prometheus/DCIM source, power headroom as a
rack-levelling score, spreading multi-node replicas across racks,
`llmgw_domain_*` metrics and alerts (near cap, over cap, stale data),
shedding and tokens per watt (step 5), and a drained node's state
surviving a restart.
---
+164
View File
@@ -23,6 +23,170 @@ APIs change between minors without a deprecation period.
## Unreleased
### Routing by what each vLLM replica's cache holds
- A vLLM profile can set `kv_events: true`. The agent then starts the engine
publishing its KV cache events (`--kv-events-config`) to a port on
127.0.0.1 the agent listens on, with `--prefix-caching-hash-algo
sha256_cbor` and `PYTHONHASHSEED=1729`, so its block hashes can be
computed outside Python. The events never leave the node.
- The agent mirrors each engine's set of cached blocks and sends the net
changes up its existing stream to the control plane four times a second
(new `KVEvents` agent message), starting every new stream with a full
copy. Lost events (a gap in vLLM's sequence numbers), an engine restart or
`AllBlocksCleared` start a replica's index again; blocks offloaded to CPU,
remote ones and a hybrid model's second cache group are not indexed.
- The control plane passes them to the router's index, only for replicas
the sending node runs, and empties what a stream described when it closes.
- For a model with `kv_events`, the router asks one of its replicas to
tokenize the request (vLLM's `/tokenize`, with messages and tools, so the
chat template is applied exactly), hashes the tokens into blocks as vLLM
does, and scores replicas by the share of the prompt's leading blocks they
hold. The imbalance guard still applies. With no answer — no events yet,
an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls
back to the approximate tree (`router.prefix`) or to load.
- The agent checks first blocks against llmgw's hashing; an engine that
hashes another way (a profile overriding the algorithm, a changed vLLM) is
reported in the agent's log and `llmgw_kv_index_problem`, and routed as
though it published nothing.
- New settings `router.kv_index.tokenize_timeout` (250ms), `min_blocks` (4)
and `max_blocks` (4096). New metrics `llmgw_kv_index_decisions_total`
(`kv_hit`, `kv_miss_min_load`, `load_balance_override`,
`fallback_no_index`, `fallback_unsupported`, `fallback_tokenize_failed`),
`llmgw_kv_index_tokenize_seconds`, `llmgw_kv_index_blocks`.
- Experimental: tested against vLLM's event format and hashing as its source
defines them, and against libzmq and msgspec publishing — not yet against a
running vLLM.
### LoRA adapters on vLLM, loaded on demand
- **vLLM models can carry adapters.** A PEFT adapter directory (a mirror
version ending in `/`, a path on the nodes, or a Hugging Face repository
id) is registered at start with `--enable-lora --lora-modules`, and a
request names it as its model, as on llama.cpp. `scale` applies on
llama.cpp only. SGLang and TensorRT-LLM profiles still refuse adapters.
- **`max_loaded_adapters: N` on a model** has each replica hold at most N of
its adapters. Replicas start holding none (vLLM with
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras N`, unless the
profile's args set their own). The router loads an adapter on a replica
through `/v1/load_lora_adapter` when a request first wants it there, and
unloads the least recently used one no request is running on to make room.
vLLM profiles only; llama.cpp keeps loading every adapter at start, since
llama-server can rescale an adapter it started with but not load another.
- **Adapter affinity:** a request for an adapter goes to a replica that has
it, or is loading it, while that replica is not saturated; otherwise to one
with room for another adapter; otherwise where load sends it — which is how
a busy adapter spreads to a second replica. Requests arriving during a load
wait for it instead of loading the adapter again.
- **Self-correcting:** each replica's `/v1/models` is read when the router
first meets it and every 30 seconds, so adapters loaded before a restart or
by another router are known; a 404 for an adapter the router thought loaded
loads it again and retries. An adapter vLLM refuses (not found, rank above
`--max-lora-rank`) fails the request with vLLM's reason, without trying it
on every replica.
- **Metrics:** `llmgw_adapter_affinity_total{model,result=hit|joined|loaded}`,
`llmgw_adapter_loads_total{model,outcome}`,
`llmgw_adapter_evictions_total{model,outcome}`.
- The admin API and `ctl model adapters` read the file as GGUF only for a
model with llama.cpp profiles.
### Federating clusters: another llmgw cluster as a provider
- `llmgw ctl federation add NAME --url URL --key-file FILE [--send-tags]`
makes another llmgw gateway a peer. It is checked first (its `/ready`,
timed, and its `/v1/models` with the key) and kept in the store (migration
0029, `federation_peers`) with the models it offers.
- A peer's models join the catalog as `NAME/MODEL`, always prefixed, and
follow the peer's own list every `federation.interval` (10s). A model the
peer drops goes, unless a model here still falls back to it: then it is
kept withdrawn (in the catalog, sent nothing) until the fallback changes.
The peer's aliases and its own peers' models are not taken.
- A peer whose `/ready` fails, which refuses the key, or which does not answer
is taken out of routing until it answers again. `/ready` now counts only a
gateway's own endpoints, not its peers'.
- `llmgw ctl model fallbacks MODEL [FALLBACK...]` (and `PUT
/api/v1/models/{name}/fallbacks`) sets a runtime model's fallbacks, so
overflow can go to a peer's model.
- Requests to a peer carry `x-llmgw-hop` with the ids of the clusters they
have passed through (each cluster's id is kept in the store). A request that
comes back to a cluster is refused with 508 `federation_loop`, and a peer a
request came from is not sent it.
- The peer sees one key, this cluster's, and the request's `X-Request-Id`.
With `--send-tags`, also `x-llmgw-tags: federated_from=<federation.name>,
org=,team=,project=` (ids); off by default.
- In-cluster residency and `no_store` keep requests off peers, by name and as
fallbacks. Usage records the peer's model as served, and `peer:NAME/MODEL`
as the replica.
- `ctl federation list|remove|sync`, a Federation view in the console,
`llmgw_federation_peer_up`, `_latency_seconds` and `_models` metrics, admin
API `/api/v1/federation`, and a "Federating clusters" section in
docs/install.md. New `federation:` block in server.yaml (`name`,
`interval`).
- A response header the gateway already set (X-Request-Id,
x-llmgw-served-model) is no longer repeated from the upstream's.
### Facility-aware placement: racks, PDUs, holds and power caps
- **Where a node is**, from five labels: `llmgw.io/site`, `room`, `row`,
`rack` and `pdu` (a list, for a dual-fed rack), in `agent.yaml` or set from
the control plane with **`llmgw ctl label NODE key=value… [key-…]`**, which
is kept in the store (migration 0031, `node_labels`), wins over the agent's
labels, works before the node joins, and refuses a misspelt `llmgw.io/` key.
A node with no location places as before.
- **Replicas spread across racks and PDUs.** A model's next replica goes to a
node sharing no rack or PDU with its others when there is one: an ordering
above weights already on disk and bin packing, never a filter.
- **`keep_within: rack`** (or row, room, site) on a profile keeps a
multi-node replica's nodes in one such domain.
- **Holds.** **`llmgw ctl facility hold rack:r12-07 --reason … [--drain]`**
places nothing new in a rack, row, room, site or PDU; `--drain` also drains
its nodes. **`llmgw ctl facility release`** undoes both. Kept across
restarts (migration 0031, `facility_holds`).
- **Power caps.** `facility.budgets` in `server.yaml` gives a domain's
`cap_watts` and `other_watts`, with a `margin`; a replica's draw is the
profile's `power_watts`, or its cards at the power limit they report
(`gpu_watts` otherwise) and its cores at `cpu_core_watts`, with `node_watts`
per node. Use is the larger of reserved and measured, node by node; a shard
that would take a domain past its cap less the margin is refused, and a
multi-node replica's shards are counted together. Unknown draw never blocks.
- **Seen everywhere it matters.** `ctl explain-placement` names the held
domain or the kilowatts, and keeps the rejections of a replica that was
placed, so one in an unexpected rack says why. `ctl nodes` has LOCATION,
*ready, held* and `--domain`. **`llmgw ctl facility`** lists every domain
with nodes, use, limit, reserved, measured and holds. The console's Nodes
view has WHERE; a node's menu has **Location…** and **Hold rack…/Release**.
Admin API: `GET /api/v1/facility`, `POST /api/v1/facility/holds`,
`DELETE /api/v1/facility/holds/{domain}`, `POST /api/v1/nodes/{id}/labels`.
- Docs: `docs/facility-aware-placement.md` says what is built and where it
departs from the design; `docs/install.md` and `docs/console.md`.
### SGLang measured, checked and benchmarked as itself
- **Fixed:** the router read SGLang replicas with vLLM's metric names, and
SGLang was started without `--enable-metrics`, so it served no `/metrics`
at all: every scrape failed as though the replica were unreachable, quietly,
and requests were routed to it blind. SGLang is now started with
`--enable-metrics` and read in its own names — `sglang:num_running_reqs`,
`sglang:num_queue_reqs`, `sglang:token_usage` (the KV pool's fill), the
prompt and generation token counters and `sglang:num_requests_total`, as
0.5.21 publishes them.
- **Fixed:** `llmgw ctl engine-check sglang` asked the interpreter for
`--help` and checked SGLang's flags against Python's. It now asks
`python -m sglang.launch_server --help`, in the environment SGLang was
installed into (or `--bin`).
- **Fixed:** SGLang was given `--tp`, which argparse accepts only as an
abbreviation of `--tp-size`; it is given `--tp-size`.
- **Fixed:** `llmgw ctl bench --metrics` read an engine's metrics with the
key given for `--target` — a client's key when the target is the gateway,
or none — so llama.cpp and TensorRT-LLM, which guard their metrics with the
engine's key, refused every read, and the run reported a queue depth of
zero and suggested no thresholds without saying why. The engine key is now
asked of the admin API for the replica at `--metrics`' address (or given
with the new `--metrics-key`); metrics that cannot be read — refused, or in
another engine's names than `--engine` — fail the run before any load,
saying why; and a step whose metrics went unread shows `-` and the reason
rather than zeros.
### Reusing answers
- **A response cache, per project, off until asked for.** `llmgw keys cache
+4 -2
View File
@@ -67,8 +67,10 @@ Since v0.11.0, not yet released (see [CHANGELOG.md](CHANGELOG.md),
automatic rollback, quality suites gating updates, a response cache,
kept and replayed requests, batches, Anthropic's Messages API, more of
OpenAI's (structured output, images, rerank, audio, moderations),
scaling by the clock, SLO-aware admission, TensorRT-LLM, MIG, and models
imported from Ollama.
scaling by the clock, SLO-aware admission, TensorRT-LLM, MIG, models
imported from Ollama, routing by what each vLLM replica's cache holds,
LoRA adapters loaded on demand, other llmgw clusters as providers, and
placement that knows racks, PDUs and power caps.
- **Operations** — `server restore` and off-machine backup copies, secrets
in Vault, forgetting a person and keeping a project's requests in the
cluster, offline installs from a bundle, a Helm chart, tracing from the
-40
View File
@@ -1,40 +0,0 @@
## CHANGELOG
### Facility-aware placement: racks, PDUs, holds and power caps
- **Where a node is**, from five labels: `llmgw.io/site`, `room`, `row`,
`rack` and `pdu` (a list, for a dual-fed rack), in `agent.yaml` or set from
the control plane with **`llmgw ctl label NODE key=value… [key-…]`**, which
is kept in the store (migration 0031, `node_labels`), wins over the agent's
labels, works before the node joins, and refuses a misspelt `llmgw.io/` key.
A node with no location places as before.
- **Replicas spread across racks and PDUs.** A model's next replica goes to a
node sharing no rack or PDU with its others when there is one: an ordering
above weights already on disk and bin packing, never a filter.
- **`keep_within: rack`** (or row, room, site) on a profile keeps a
multi-node replica's nodes in one such domain.
- **Holds.** **`llmgw ctl facility hold rack:r12-07 --reason … [--drain]`**
places nothing new in a rack, row, room, site or PDU; `--drain` also drains
its nodes. **`llmgw ctl facility release`** undoes both. Kept across
restarts (migration 0031, `facility_holds`).
- **Power caps.** `facility.budgets` in `server.yaml` gives a domain's
`cap_watts` and `other_watts`, with a `margin`; a replica's draw is the
profile's `power_watts`, or its cards at the power limit they report
(`gpu_watts` otherwise) and its cores at `cpu_core_watts`, with `node_watts`
per node. Use is the larger of reserved and measured, node by node; a shard
that would take a domain past its cap less the margin is refused, and a
multi-node replica's shards are counted together. Unknown draw never blocks.
- **Seen everywhere it matters.** `ctl explain-placement` names the held
domain or the kilowatts, and keeps the rejections of a replica that was
placed, so one in an unexpected rack says why. `ctl nodes` has LOCATION,
*ready, held* and `--domain`. **`llmgw ctl facility`** lists every domain
with nodes, use, limit, reserved, measured and holds. The console's Nodes
view has WHERE; a node's menu has **Location…** and **Hold rack…/Release**.
Admin API: `GET /api/v1/facility`, `POST /api/v1/facility/holds`,
`DELETE /api/v1/facility/holds/{domain}`, `POST /api/v1/nodes/{id}/labels`.
- Docs: `docs/facility-aware-placement.md` says what is built and where it
departs from the design; `docs/install.md` and `docs/console.md`.
## BACKLOG
- [x] 24 (facility) — built: location labels (`ctl label`, agent.yaml), spreading across racks/PDUs, `keep_within`, held domains with drain (`ctl facility hold/release`, kept in the store), power caps from `facility.budgets` against declared/derived/measured draw, shown in explain-placement, `ctl nodes`, `ctl facility`, the console and the API. Left: temperature conditions with hysteresis and their enforcement (steps 3–4), a Prometheus/DCIM source, power headroom as a rack-levelling score, spreading multi-node replicas across racks, `llmgw_domain_*` metrics and alerts (near cap, over cap, stale data), shedding and tokens per watt (step 5), and a drained node's state surviving a restart.
-40
View File
@@ -1,40 +0,0 @@
## CHANGELOG
### Federating clusters: another llmgw cluster as a provider
- `llmgw ctl federation add NAME --url URL --key-file FILE [--send-tags]`
makes another llmgw gateway a peer. It is checked first (its `/ready`,
timed, and its `/v1/models` with the key) and kept in the store (migration
0029, `federation_peers`) with the models it offers.
- A peer's models join the catalog as `NAME/MODEL`, always prefixed, and
follow the peer's own list every `federation.interval` (10s). A model the
peer drops goes, unless a model here still falls back to it: then it is
kept withdrawn (in the catalog, sent nothing) until the fallback changes.
The peer's aliases and its own peers' models are not taken.
- A peer whose `/ready` fails, which refuses the key, or which does not answer
is taken out of routing until it answers again. `/ready` now counts only a
gateway's own endpoints, not its peers'.
- `llmgw ctl model fallbacks MODEL [FALLBACK...]` (and `PUT
/api/v1/models/{name}/fallbacks`) sets a runtime model's fallbacks, so
overflow can go to a peer's model.
- Requests to a peer carry `x-llmgw-hop` with the ids of the clusters they
have passed through (each cluster's id is kept in the store). A request that
comes back to a cluster is refused with 508 `federation_loop`, and a peer a
request came from is not sent it.
- The peer sees one key, this cluster's, and the request's `X-Request-Id`.
With `--send-tags`, also `x-llmgw-tags: federated_from=<federation.name>,
org=,team=,project=` (ids); off by default.
- In-cluster residency and `no_store` keep requests off peers, by name and as
fallbacks. Usage records the peer's model as served, and `peer:NAME/MODEL`
as the replica.
- `ctl federation list|remove|sync`, a Federation view in the console,
`llmgw_federation_peer_up`, `_latency_seconds` and `_models` metrics, admin
API `/api/v1/federation`, and a "Federating clusters" section in
docs/install.md. New `federation:` block in server.yaml (`name`,
`interval`).
- A response header the gateway already set (X-Request-Id,
x-llmgw-served-model) is no longer repeated from the upstream's.
## BACKLOG
- [x] 24. Multi-cluster federation — built: peers via `ctl federation` kept in the store, their models as PEER/MODEL synced from /v1/models (withdrawn-while-referenced), /ready health with latency and routing withdrawal, `ctl model fallbacks` for overflow, hop header loop refusal, residency/no_store hold, optional attribution tags, console view, metrics, docs. Left: server.yaml-declared models cannot name a peer model as a fallback (config catalog is validated before peers are read); per-peer CA / mTLS for peers behind a private CA; transitive federation (A→B→C models) deliberately not offered; latency is /ready RTT only (no per-request peer TTFT used in routing or weighting); the peer's adapters are not imported (listed as aliases); no console form to add/remove peers (CLI only); key file must exist on every control plane.
-42
View File
@@ -1,42 +0,0 @@
# C24 — precise KV-index routing (plan §7.5 tier 2)
## CHANGELOG
### Routing by what each vLLM replica's cache holds
- A vLLM profile can set `kv_events: true`. The agent then starts the engine
publishing its KV cache events (`--kv-events-config`) to a port on
127.0.0.1 the agent listens on, with `--prefix-caching-hash-algo
sha256_cbor` and `PYTHONHASHSEED=1729`, so its block hashes can be
computed outside Python. The events never leave the node.
- The agent mirrors each engine's set of cached blocks and sends the net
changes up its existing stream to the control plane four times a second
(new `KVEvents` agent message), starting every new stream with a full
copy. Lost events (a gap in vLLM's sequence numbers), an engine restart or
`AllBlocksCleared` start a replica's index again; blocks offloaded to CPU,
remote ones and a hybrid model's second cache group are not indexed.
- The control plane passes them to the router's index, only for replicas
the sending node runs, and empties what a stream described when it closes.
- For a model with `kv_events`, the router asks one of its replicas to
tokenize the request (vLLM's `/tokenize`, with messages and tools, so the
chat template is applied exactly), hashes the tokens into blocks as vLLM
does, and scores replicas by the share of the prompt's leading blocks they
hold. The imbalance guard still applies. With no answer — no events yet,
an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls
back to the approximate tree (`router.prefix`) or to load.
- The agent checks first blocks against llmgw's hashing; an engine that
hashes another way (a profile overriding the algorithm, a changed vLLM) is
reported in the agent's log and `llmgw_kv_index_problem`, and routed as
though it published nothing.
- New settings `router.kv_index.tokenize_timeout` (250ms), `min_blocks` (4)
and `max_blocks` (4096). New metrics `llmgw_kv_index_decisions_total`
(`kv_hit`, `kv_miss_min_load`, `load_balance_override`,
`fallback_no_index`, `fallback_unsupported`, `fallback_tokenize_failed`),
`llmgw_kv_index_tokenize_seconds`, `llmgw_kv_index_blocks`.
- Experimental: tested against vLLM's event format and hashing as its source
defines them, and against libzmq and msgspec publishing — not yet against a
running vLLM.
## BACKLOG
- [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent (loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings) forwarded as net block-hash changes over the agent stream; router index, tokenize via the replica's `/tokenize`, sha256_cbor block-hash chain checked against cbor2/hashlib golden values; scorer by leading-block share with tier 1 fallback; opt-in `kv_events` per profile; metrics; docs. Left: validation against a real vLLM (event shape, hash agreement, `/tokenize` matching the served prompt, cost of tokenizing); vLLM's replay endpoint instead of resetting on a gap; data-parallel ranks (one publisher each, ports offset); hybrid models' cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt requests (extra keys); a router-only deployment (no agents) subscribing directly; standby control planes have no index until they lead; a per-replica fallback when only some candidates have an index; caching tokenizations of repeated prompts.
-52
View File
@@ -1,52 +0,0 @@
## CHANGELOG
### LoRA adapters on vLLM, loaded on demand
- **vLLM models can carry adapters.** A PEFT adapter directory (a mirror
version ending in `/`, a path on the nodes, or a Hugging Face repository
id) is registered at start with `--enable-lora --lora-modules`, and a
request names it as its model, as on llama.cpp. `scale` applies on
llama.cpp only. SGLang and TensorRT-LLM profiles still refuse adapters.
- **`max_loaded_adapters: N` on a model** has each replica hold at most N of
its adapters. Replicas start holding none (vLLM with
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras N`, unless the
profile's args set their own). The router loads an adapter on a replica
through `/v1/load_lora_adapter` when a request first wants it there, and
unloads the least recently used one no request is running on to make room.
vLLM profiles only; llama.cpp keeps loading every adapter at start, since
llama-server can rescale an adapter it started with but not load another.
- **Adapter affinity:** a request for an adapter goes to a replica that has
it, or is loading it, while that replica is not saturated; otherwise to one
with room for another adapter; otherwise where load sends it — which is how
a busy adapter spreads to a second replica. Requests arriving during a load
wait for it instead of loading the adapter again.
- **Self-correcting:** each replica's `/v1/models` is read when the router
first meets it and every 30 seconds, so adapters loaded before a restart or
by another router are known; a 404 for an adapter the router thought loaded
loads it again and retries. An adapter vLLM refuses (not found, rank above
`--max-lora-rank`) fails the request with vLLM's reason, without trying it
on every replica.
- **Metrics:** `llmgw_adapter_affinity_total{model,result=hit|joined|loaded}`,
`llmgw_adapter_loads_total{model,outcome}`,
`llmgw_adapter_evictions_total{model,outcome}`.
- The admin API and `ctl model adapters` read the file as GGUF only for a
model with llama.cpp profiles.
## BACKLOG
**24 LoRA affinity (branch `features-c2`):** built `max_loaded_adapters`
(catalog), vLLM adapter support at start and on demand (`engines/lora.go`,
`vllm.go`), the agent linking fetched adapters as `llmgw-lora/<name>` in the
engines' working directory, the router's adapter tracker (`router/lora.go`:
loads/unloads with the engine key, LRU eviction never of an adapter in use,
one load per adapter per replica, `/v1/models` sync, 404 reload) and the
`adapter-affinity` scheduler filter in the chat, batch and long-context
profiles, with metrics and docs; tested against a fake of vLLM's HTTP
interface. **Left:** never run against a real vLLM — in particular that vLLM
resolves the relative `llmgw-lora/<name>` path against its working directory
on every node (multi-node too) and that unloading an adapter mid-request is
harmless; the agent still fetches every adapter at replica start, so a new
adapter needs a restart (fetching on first load would need an agent command
and a proto change); load timeout and resync interval are constants, not
config; SGLang's runtime LoRA API is not driven; no per-replica "adapters
held" view in the console or admin API; the debug profile has no affinity.
-28
View File
@@ -1,28 +0,0 @@
## CHANGELOG
### SGLang measured, checked and benchmarked as itself
- **Fixed:** the router read SGLang replicas with vLLM's metric names, and
SGLang was started without `--enable-metrics`, so it served no `/metrics`
at all: every scrape failed as though the replica were unreachable, quietly,
and requests were routed to it blind. SGLang is now started with
`--enable-metrics` and read in its own names — `sglang:num_running_reqs`,
`sglang:num_queue_reqs`, `sglang:token_usage` (the KV pool's fill), the
prompt and generation token counters and `sglang:num_requests_total`, as
0.5.21 publishes them.
- **Fixed:** `llmgw ctl engine-check sglang` asked the interpreter for
`--help` and checked SGLang's flags against Python's. It now asks
`python -m sglang.launch_server --help`, in the environment SGLang was
installed into (or `--bin`).
- **Fixed:** SGLang was given `--tp`, which argparse accepts only as an
abbreviation of `--tp-size`; it is given `--tp-size`.
- **Fixed:** `llmgw ctl bench --metrics` read an engine's metrics with the
key given for `--target` — a client's key when the target is the gateway,
or none — so llama.cpp and TensorRT-LLM, which guard their metrics with the
engine's key, refused every read, and the run reported a queue depth of
zero and suggested no thresholds without saying why. The engine key is now
asked of the admin API for the replica at `--metrics`' address (or given
with the new `--metrics-key`); metrics that cannot be read — refused, or in
another engine's names than `--engine` — fail the run before any load,
saying why; and a step whose metrics went unread shows `-` and the reason
rather than zeros.