LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters
A vLLM model can carry PEFT adapters, registered at start, or with max_loaded_adapters held at most N per replica: the router loads an adapter through vLLM's runtime LoRA API when a request first wants it, sends the adapter's requests to a replica that holds it while that one is not saturated, loads each cold adapter once, and unloads the least recently used idle one to make room. llama.cpp keeps loading every adapter at start. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
d41104a452
commit
2a1e0e6c4c
24 files changed
+1773
-35
No files matched your search
@@ -1028,6 +1028,55 @@ The adapter file must be a GGUF LoRA made for this model — llama.cpp's
|
||||
`convert_lora_to_gguf.py` makes one from a PEFT adapter — and not a model with
|
||||
the adapter already merged in, which is what most "LoRA GGUF" repositories hold.
|
||||
|
||||
#### Adapters on vLLM, and loading them on demand
|
||||
|
||||
A vLLM model carries adapters the same way. There an adapter is a PEFT adapter
|
||||
directory (`adapter_config.json` and `adapter_model.safetensors`): a mirror
|
||||
version ending in `/`, a path on the nodes, or a Hugging Face repository id,
|
||||
which vLLM downloads itself. A request names the adapter as its model, as on
|
||||
llama.cpp; `scale` applies on llama.cpp only. Without anything more, every
|
||||
replica registers every adapter at start (`--enable-lora --lora-modules`).
|
||||
vLLM mixes one adapter per batch by default; raise that with `--max-loras` in
|
||||
the profile's `args`, and `--max-lora-rank` (16 by default) for adapters of a
|
||||
higher rank.
|
||||
|
||||
With many adapters, have each replica hold only some of them:
|
||||
|
||||
```yaml
|
||||
- name: qwen2.5-7b-instruct
|
||||
source: Qwen/Qwen2.5-7B-Instruct
|
||||
max_loaded_adapters: 4 # per replica; vllm profiles only
|
||||
adapters:
|
||||
- name: support-de
|
||||
source: http://mirror:8081/support-de/v3/
|
||||
- name: support-fr
|
||||
source: someone/support-fr-lora
|
||||
profiles: [...]
|
||||
```
|
||||
|
||||
Replicas then start holding none (vLLM runs with
|
||||
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras 4`). The first
|
||||
request for an adapter has the router load it on a replica through vLLM's
|
||||
`/v1/load_lora_adapter`, and the requests after it go to that replica while
|
||||
it is not saturated; when it is, the next request goes where load sends it and
|
||||
the adapter is loaded there too. Requests that arrive while an adapter is being
|
||||
loaded wait for that load rather than starting another. A replica that holds
|
||||
`max_loaded_adapters` already unloads the one least recently used, never one a
|
||||
request is running on. Each node still fetches every adapter when a replica
|
||||
starts, and links it as `llmgw-lora/<name>` in the directory engines run in,
|
||||
which is the path the router has vLLM load it from — so adding an adapter
|
||||
still needs the replicas restarted, as above. An adapter vLLM refuses (not
|
||||
found, a rank above `--max-lora-rank`, not for this model) fails the request
|
||||
with vLLM's reason.
|
||||
|
||||
What each replica holds is what the router loaded there, checked against each
|
||||
replica's `/v1/models` when the router first meets it and every 30 seconds.
|
||||
`llmgw_adapter_affinity_total` counts where requests found their adapter (hit,
|
||||
joined a load, loaded it), `llmgw_adapter_loads_total` and
|
||||
`llmgw_adapter_evictions_total` what that cost. llama.cpp keeps loading every
|
||||
adapter at start: llama-server can change the scale of an adapter it started
|
||||
with, not load a new one.
|
||||
|
||||
### Models served outside the cluster
|
||||
|
||||
A model can be an OpenAI-compatible API elsewhere — a hosted provider, or
|
||||
|
||||
Reference in new issue
Block a user