LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

A vLLM model can carry PEFT adapters, registered at start, or with
max_loaded_adapters held at most N per replica: the router loads an adapter
through vLLM's runtime LoRA API when a request first wants it, sends the
adapter's requests to a replica that holds it while that one is not
saturated, loads each cold adapter once, and unloads the least recently used
idle one to make room. llama.cpp keeps loading every adapter at start.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 01:14:38 +02:00
1 parent d41104a452
commit 2a1e0e6c4c
24 files changed
+1773 -35

No files matched your search

+6 -2
View File
@@ -4930,7 +4930,7 @@
},
"source": {
"type": "string",
"description": "Source is the adapter file (GGUF for llama.cpp), fetched by each node as the model's weights are."
"description": "Source is the adapter: a GGUF file for llama.cpp; for vLLM a PEFT adapter directory (a mirror version ending in \"/\", a path on the node, or a Hugging Face repository id). Fetched by each node as the model's weights are."
},
"scale": {
"type": "number",
@@ -5077,7 +5077,11 @@
"items": {
"$ref": "#/components/schemas/catalog.Adapter"
},
"description": "Adapters are LoRA adapters served on this model's replicas, each asked for by its own name as though it were a model. Every replica loads every adapter, applying one only to a request that names it."
"description": "Adapters are LoRA adapters served on this model's replicas, each asked for by its own name as though it were a model. Every replica loads every adapter, applying one only to a request that names it — unless MaxLoadedAdapters is set."
},
"max_loaded_adapters": {
"type": "integer",
"description": "MaxLoadedAdapters, above zero, has each replica hold at most this many adapters at once, loaded when a request first asks for one and unloaded least recently used first to make room. The router sends a request for an adapter to a replica that already holds it while that one has room to take it. vLLM profiles only: llama-server loads adapters at start and cannot load one afterwards. Zero loads every adapter on every replica at start."
},
"fallbacks": {
"type": "array",
+49
View File
@@ -1028,6 +1028,55 @@ The adapter file must be a GGUF LoRA made for this model — llama.cpp's
`convert_lora_to_gguf.py` makes one from a PEFT adapter — and not a model with
the adapter already merged in, which is what most "LoRA GGUF" repositories hold.
#### Adapters on vLLM, and loading them on demand
A vLLM model carries adapters the same way. There an adapter is a PEFT adapter
directory (`adapter_config.json` and `adapter_model.safetensors`): a mirror
version ending in `/`, a path on the nodes, or a Hugging Face repository id,
which vLLM downloads itself. A request names the adapter as its model, as on
llama.cpp; `scale` applies on llama.cpp only. Without anything more, every
replica registers every adapter at start (`--enable-lora --lora-modules`).
vLLM mixes one adapter per batch by default; raise that with `--max-loras` in
the profile's `args`, and `--max-lora-rank` (16 by default) for adapters of a
higher rank.
With many adapters, have each replica hold only some of them:
```yaml
- name: qwen2.5-7b-instruct
source: Qwen/Qwen2.5-7B-Instruct
max_loaded_adapters: 4 # per replica; vllm profiles only
adapters:
- name: support-de
source: http://mirror:8081/support-de/v3/
- name: support-fr
source: someone/support-fr-lora
profiles: [...]
```
Replicas then start holding none (vLLM runs with
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras 4`). The first
request for an adapter has the router load it on a replica through vLLM's
`/v1/load_lora_adapter`, and the requests after it go to that replica while
it is not saturated; when it is, the next request goes where load sends it and
the adapter is loaded there too. Requests that arrive while an adapter is being
loaded wait for that load rather than starting another. A replica that holds
`max_loaded_adapters` already unloads the one least recently used, never one a
request is running on. Each node still fetches every adapter when a replica
starts, and links it as `llmgw-lora/<name>` in the directory engines run in,
which is the path the router has vLLM load it from — so adding an adapter
still needs the replicas restarted, as above. An adapter vLLM refuses (not
found, a rank above `--max-lora-rank`, not for this model) fails the request
with vLLM's reason.
What each replica holds is what the router loaded there, checked against each
replica's `/v1/models` when the router first meets it and every 30 seconds.
`llmgw_adapter_affinity_total` counts where requests found their adapter (hit,
joined a load, loaded it), `llmgw_adapter_loads_total` and
`llmgw_adapter_evictions_total` what that cost. llama.cpp keeps loading every
adapter at start: llama-server can change the scale of an adapter it started
with, not load a new one.
### Models served outside the cluster
A model can be an OpenAI-compatible API elsewhere — a hosted provider, or
+19 -3
View File
@@ -114,6 +114,7 @@ rather than repeated.
| `owner` | `string` | [Model](#model) |
| `external` | `*External` | [Model](#model) |
| `adapters` | `[]Adapter` | [Model](#model) |
| `max_loaded_adapters` | `int` | [Model](#model) |
| `fallbacks` | `[]string` | [Model](#model) |
| `canary` | `*Canary` | [Model](#model) |
| `eval` | `*EvalGate` | [Model](#model) |
@@ -372,8 +373,10 @@ Name is what clients ask for, like a model name.
`string`
Source is the adapter file (GGUF for llama.cpp), fetched by each node
as the model's weights are.
Source is the adapter: a GGUF file for llama.cpp; for vLLM a PEFT
adapter directory (a mirror version ending in "/", a path on the node,
or a Hugging Face repository id). Fetched by each node as the model's
weights are.
### `scale`
@@ -1078,7 +1081,20 @@ Takes a [External](#external) block.
Adapters are LoRA adapters served on this model's replicas, each asked
for by its own name as though it were a model. Every replica loads
every adapter, applying one only to a request that names it.
every adapter, applying one only to a request that names it — unless
MaxLoadedAdapters is set.
### `max_loaded_adapters`
`int`
MaxLoadedAdapters, above zero, has each replica hold at most this many
adapters at once, loaded when a request first asks for one and unloaded
least recently used first to make room. The router sends a request for
an adapter to a replica that already holds it while that one has room
to take it. vLLM profiles only: llama-server loads adapters at start
and cannot load one afterwards. Zero loads every adapter on every
replica at start.
### `fallbacks`