LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

A vLLM model can carry PEFT adapters, registered at start, or with
max_loaded_adapters held at most N per replica: the router loads an adapter
through vLLM's runtime LoRA API when a request first wants it, sends the
adapter's requests to a replica that holds it while that one is not
saturated, loads each cold adapter once, and unloads the least recently used
idle one to make room. llama.cpp keeps loading every adapter at start.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 01:14:38 +02:00
1 parent d41104a452
commit 2a1e0e6c4c
24 files changed
+1773 -35

No files matched your search

+49
View File
@@ -1028,6 +1028,55 @@ The adapter file must be a GGUF LoRA made for this model — llama.cpp's
`convert_lora_to_gguf.py` makes one from a PEFT adapter — and not a model with
the adapter already merged in, which is what most "LoRA GGUF" repositories hold.
#### Adapters on vLLM, and loading them on demand
A vLLM model carries adapters the same way. There an adapter is a PEFT adapter
directory (`adapter_config.json` and `adapter_model.safetensors`): a mirror
version ending in `/`, a path on the nodes, or a Hugging Face repository id,
which vLLM downloads itself. A request names the adapter as its model, as on
llama.cpp; `scale` applies on llama.cpp only. Without anything more, every
replica registers every adapter at start (`--enable-lora --lora-modules`).
vLLM mixes one adapter per batch by default; raise that with `--max-loras` in
the profile's `args`, and `--max-lora-rank` (16 by default) for adapters of a
higher rank.
With many adapters, have each replica hold only some of them:
```yaml
- name: qwen2.5-7b-instruct
source: Qwen/Qwen2.5-7B-Instruct
max_loaded_adapters: 4 # per replica; vllm profiles only
adapters:
- name: support-de
source: http://mirror:8081/support-de/v3/
- name: support-fr
source: someone/support-fr-lora
profiles: [...]
```
Replicas then start holding none (vLLM runs with
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras 4`). The first
request for an adapter has the router load it on a replica through vLLM's
`/v1/load_lora_adapter`, and the requests after it go to that replica while
it is not saturated; when it is, the next request goes where load sends it and
the adapter is loaded there too. Requests that arrive while an adapter is being
loaded wait for that load rather than starting another. A replica that holds
`max_loaded_adapters` already unloads the one least recently used, never one a
request is running on. Each node still fetches every adapter when a replica
starts, and links it as `llmgw-lora/<name>` in the directory engines run in,
which is the path the router has vLLM load it from — so adding an adapter
still needs the replicas restarted, as above. An adapter vLLM refuses (not
found, a rank above `--max-lora-rank`, not for this model) fails the request
with vLLM's reason.
What each replica holds is what the router loaded there, checked against each
replica's `/v1/models` when the router first meets it and every 30 seconds.
`llmgw_adapter_affinity_total` counts where requests found their adapter (hit,
joined a load, loaded it), `llmgw_adapter_loads_total` and
`llmgw_adapter_evictions_total` what that cost. llama.cpp keeps loading every
adapter at start: llama-server can change the scale of an adapter it started
with, not load a new one.
### Models served outside the cluster
A model can be an OpenAI-compatible API elsewhere — a hosted provider, or