Merge C24 LoRA adapters on demand for vLLM, with affinity
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
bd3eca4092
24 files changed
+1773
-35
No files matched your search
@@ -1033,6 +1033,55 @@ The adapter file must be a GGUF LoRA made for this model — llama.cpp's
|
||||
`convert_lora_to_gguf.py` makes one from a PEFT adapter — and not a model with
|
||||
the adapter already merged in, which is what most "LoRA GGUF" repositories hold.
|
||||
|
||||
#### Adapters on vLLM, and loading them on demand
|
||||
|
||||
A vLLM model carries adapters the same way. There an adapter is a PEFT adapter
|
||||
directory (`adapter_config.json` and `adapter_model.safetensors`): a mirror
|
||||
version ending in `/`, a path on the nodes, or a Hugging Face repository id,
|
||||
which vLLM downloads itself. A request names the adapter as its model, as on
|
||||
llama.cpp; `scale` applies on llama.cpp only. Without anything more, every
|
||||
replica registers every adapter at start (`--enable-lora --lora-modules`).
|
||||
vLLM mixes one adapter per batch by default; raise that with `--max-loras` in
|
||||
the profile's `args`, and `--max-lora-rank` (16 by default) for adapters of a
|
||||
higher rank.
|
||||
|
||||
With many adapters, have each replica hold only some of them:
|
||||
|
||||
```yaml
|
||||
- name: qwen2.5-7b-instruct
|
||||
source: Qwen/Qwen2.5-7B-Instruct
|
||||
max_loaded_adapters: 4 # per replica; vllm profiles only
|
||||
adapters:
|
||||
- name: support-de
|
||||
source: http://mirror:8081/support-de/v3/
|
||||
- name: support-fr
|
||||
source: someone/support-fr-lora
|
||||
profiles: [...]
|
||||
```
|
||||
|
||||
Replicas then start holding none (vLLM runs with
|
||||
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras 4`). The first
|
||||
request for an adapter has the router load it on a replica through vLLM's
|
||||
`/v1/load_lora_adapter`, and the requests after it go to that replica while
|
||||
it is not saturated; when it is, the next request goes where load sends it and
|
||||
the adapter is loaded there too. Requests that arrive while an adapter is being
|
||||
loaded wait for that load rather than starting another. A replica that holds
|
||||
`max_loaded_adapters` already unloads the one least recently used, never one a
|
||||
request is running on. Each node still fetches every adapter when a replica
|
||||
starts, and links it as `llmgw-lora/<name>` in the directory engines run in,
|
||||
which is the path the router has vLLM load it from — so adding an adapter
|
||||
still needs the replicas restarted, as above. An adapter vLLM refuses (not
|
||||
found, a rank above `--max-lora-rank`, not for this model) fails the request
|
||||
with vLLM's reason.
|
||||
|
||||
What each replica holds is what the router loaded there, checked against each
|
||||
replica's `/v1/models` when the router first meets it and every 30 seconds.
|
||||
`llmgw_adapter_affinity_total` counts where requests found their adapter (hit,
|
||||
joined a load, loaded it), `llmgw_adapter_loads_total` and
|
||||
`llmgw_adapter_evictions_total` what that cost. llama.cpp keeps loading every
|
||||
adapter at start: llama-server can change the scale of an adapter it started
|
||||
with, not load a new one.
|
||||
|
||||
### Models served outside the cluster
|
||||
|
||||
A model can be an OpenAI-compatible API elsewhere — a hosted provider, or
|
||||
|
||||
Reference in new issue
Block a user