Merge C24 LoRA adapters on demand for vLLM, with affinity

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 01:15:12 +02:00
commit bd3eca4092
24 files changed
+1773 -35

No files matched your search

+52
View File
@@ -0,0 +1,52 @@
## CHANGELOG
### LoRA adapters on vLLM, loaded on demand
- **vLLM models can carry adapters.** A PEFT adapter directory (a mirror
version ending in `/`, a path on the nodes, or a Hugging Face repository
id) is registered at start with `--enable-lora --lora-modules`, and a
request names it as its model, as on llama.cpp. `scale` applies on
llama.cpp only. SGLang and TensorRT-LLM profiles still refuse adapters.
- **`max_loaded_adapters: N` on a model** has each replica hold at most N of
its adapters. Replicas start holding none (vLLM with
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras N`, unless the
profile's args set their own). The router loads an adapter on a replica
through `/v1/load_lora_adapter` when a request first wants it there, and
unloads the least recently used one no request is running on to make room.
vLLM profiles only; llama.cpp keeps loading every adapter at start, since
llama-server can rescale an adapter it started with but not load another.
- **Adapter affinity:** a request for an adapter goes to a replica that has
it, or is loading it, while that replica is not saturated; otherwise to one
with room for another adapter; otherwise where load sends it — which is how
a busy adapter spreads to a second replica. Requests arriving during a load
wait for it instead of loading the adapter again.
- **Self-correcting:** each replica's `/v1/models` is read when the router
first meets it and every 30 seconds, so adapters loaded before a restart or
by another router are known; a 404 for an adapter the router thought loaded
loads it again and retries. An adapter vLLM refuses (not found, rank above
`--max-lora-rank`) fails the request with vLLM's reason, without trying it
on every replica.
- **Metrics:** `llmgw_adapter_affinity_total{model,result=hit|joined|loaded}`,
`llmgw_adapter_loads_total{model,outcome}`,
`llmgw_adapter_evictions_total{model,outcome}`.
- The admin API and `ctl model adapters` read the file as GGUF only for a
model with llama.cpp profiles.
## BACKLOG
**24 LoRA affinity (branch `features-c2`):** built `max_loaded_adapters`
(catalog), vLLM adapter support at start and on demand (`engines/lora.go`,
`vllm.go`), the agent linking fetched adapters as `llmgw-lora/<name>` in the
engines' working directory, the router's adapter tracker (`router/lora.go`:
loads/unloads with the engine key, LRU eviction never of an adapter in use,
one load per adapter per replica, `/v1/models` sync, 404 reload) and the
`adapter-affinity` scheduler filter in the chat, batch and long-context
profiles, with metrics and docs; tested against a fake of vLLM's HTTP
interface. **Left:** never run against a real vLLM — in particular that vLLM
resolves the relative `llmgw-lora/<name>` path against its working directory
on every node (multi-node too) and that unloading an adapter mid-request is
harmless; the agent still fetches every adapter at replica start, so a new
adapter needs a restart (fetching on first load would need an agent command
and a proto change); load timeout and resync interval are constants, not
config; SGLang's runtime LoRA API is not driven; no per-replica "adapters
held" view in the console or admin API; the debug profile has no affinity.