A vLLM model can carry PEFT adapters, registered at start, or with max_loaded_adapters held at most N per replica: the router loads an adapter through vLLM's runtime LoRA API when a request first wants it, sends the adapter's requests to a replica that holds it while that one is not saturated, loads each cold adapter once, and unloads the least recently used idle one to make room. llama.cpp keeps loading every adapter at start. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
3.2 KiB
CHANGELOG
LoRA adapters on vLLM, loaded on demand
- vLLM models can carry adapters. A PEFT adapter directory (a mirror
version ending in
/, a path on the nodes, or a Hugging Face repository id) is registered at start with--enable-lora --lora-modules, and a request names it as its model, as on llama.cpp.scaleapplies on llama.cpp only. SGLang and TensorRT-LLM profiles still refuse adapters. max_loaded_adapters: Non a model has each replica hold at most N of its adapters. Replicas start holding none (vLLM withVLLM_ALLOW_RUNTIME_LORA_UPDATING=Trueand--max-loras N, unless the profile's args set their own). The router loads an adapter on a replica through/v1/load_lora_adapterwhen a request first wants it there, and unloads the least recently used one no request is running on to make room. vLLM profiles only; llama.cpp keeps loading every adapter at start, since llama-server can rescale an adapter it started with but not load another.- Adapter affinity: a request for an adapter goes to a replica that has it, or is loading it, while that replica is not saturated; otherwise to one with room for another adapter; otherwise where load sends it — which is how a busy adapter spreads to a second replica. Requests arriving during a load wait for it instead of loading the adapter again.
- Self-correcting: each replica's
/v1/modelsis read when the router first meets it and every 30 seconds, so adapters loaded before a restart or by another router are known; a 404 for an adapter the router thought loaded loads it again and retries. An adapter vLLM refuses (not found, rank above--max-lora-rank) fails the request with vLLM's reason, without trying it on every replica. - Metrics:
llmgw_adapter_affinity_total{model,result=hit|joined|loaded},llmgw_adapter_loads_total{model,outcome},llmgw_adapter_evictions_total{model,outcome}. - The admin API and
ctl model adaptersread the file as GGUF only for a model with llama.cpp profiles.
BACKLOG
24 LoRA affinity (branch features-c2): built max_loaded_adapters
(catalog), vLLM adapter support at start and on demand (engines/lora.go,
vllm.go), the agent linking fetched adapters as llmgw-lora/<name> in the
engines' working directory, the router's adapter tracker (router/lora.go:
loads/unloads with the engine key, LRU eviction never of an adapter in use,
one load per adapter per replica, /v1/models sync, 404 reload) and the
adapter-affinity scheduler filter in the chat, batch and long-context
profiles, with metrics and docs; tested against a fake of vLLM's HTTP
interface. Left: never run against a real vLLM — in particular that vLLM
resolves the relative llmgw-lora/<name> path against its working directory
on every node (multi-node too) and that unloading an adapter mid-request is
harmless; the agent still fetches every adapter at replica start, so a new
adapter needs a restart (fetching on first load would need an agent command
and a proto change); load timeout and resync interval are constants, not
config; SGLang's runtime LoRA API is not driven; no per-replica "adapters
held" view in the console or admin API; the debug profile has no affinity.