LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

A vLLM model can carry PEFT adapters, registered at start, or with
max_loaded_adapters held at most N per replica: the router loads an adapter
through vLLM's runtime LoRA API when a request first wants it, sends the
adapter's requests to a replica that holds it while that one is not
saturated, loads each cold adapter once, and unloads the least recently used
idle one to make room. llama.cpp keeps loading every adapter at start.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 01:14:38 +02:00
1 parent d41104a452
commit 2a1e0e6c4c
24 files changed
+1773 -35

No files matched your search

+52
View File
@@ -0,0 +1,52 @@
## CHANGELOG
### LoRA adapters on vLLM, loaded on demand
- **vLLM models can carry adapters.** A PEFT adapter directory (a mirror
version ending in `/`, a path on the nodes, or a Hugging Face repository
id) is registered at start with `--enable-lora --lora-modules`, and a
request names it as its model, as on llama.cpp. `scale` applies on
llama.cpp only. SGLang and TensorRT-LLM profiles still refuse adapters.
- **`max_loaded_adapters: N` on a model** has each replica hold at most N of
its adapters. Replicas start holding none (vLLM with
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras N`, unless the
profile's args set their own). The router loads an adapter on a replica
through `/v1/load_lora_adapter` when a request first wants it there, and
unloads the least recently used one no request is running on to make room.
vLLM profiles only; llama.cpp keeps loading every adapter at start, since
llama-server can rescale an adapter it started with but not load another.
- **Adapter affinity:** a request for an adapter goes to a replica that has
it, or is loading it, while that replica is not saturated; otherwise to one
with room for another adapter; otherwise where load sends it — which is how
a busy adapter spreads to a second replica. Requests arriving during a load
wait for it instead of loading the adapter again.
- **Self-correcting:** each replica's `/v1/models` is read when the router
first meets it and every 30 seconds, so adapters loaded before a restart or
by another router are known; a 404 for an adapter the router thought loaded
loads it again and retries. An adapter vLLM refuses (not found, rank above
`--max-lora-rank`) fails the request with vLLM's reason, without trying it
on every replica.
- **Metrics:** `llmgw_adapter_affinity_total{model,result=hit|joined|loaded}`,
`llmgw_adapter_loads_total{model,outcome}`,
`llmgw_adapter_evictions_total{model,outcome}`.
- The admin API and `ctl model adapters` read the file as GGUF only for a
model with llama.cpp profiles.
## BACKLOG
**24 LoRA affinity (branch `features-c2`):** built `max_loaded_adapters`
(catalog), vLLM adapter support at start and on demand (`engines/lora.go`,
`vllm.go`), the agent linking fetched adapters as `llmgw-lora/<name>` in the
engines' working directory, the router's adapter tracker (`router/lora.go`:
loads/unloads with the engine key, LRU eviction never of an adapter in use,
one load per adapter per replica, `/v1/models` sync, 404 reload) and the
`adapter-affinity` scheduler filter in the chat, batch and long-context
profiles, with metrics and docs; tested against a fake of vLLM's HTTP
interface. **Left:** never run against a real vLLM — in particular that vLLM
resolves the relative `llmgw-lora/<name>` path against its working directory
on every node (multi-node too) and that unloading an adapter mid-request is
harmless; the agent still fetches every adapter at replica start, so a new
adapter needs a restart (fetching on first load would need an agent command
and a proto change); load timeout and resync interval are constants, not
config; SGLang's runtime LoRA API is not driven; no per-replica "adapters
held" view in the console or admin API; the debug profile has no affinity.