LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

A vLLM model can carry PEFT adapters, registered at start, or with
max_loaded_adapters held at most N per replica: the router loads an adapter
through vLLM's runtime LoRA API when a request first wants it, sends the
adapter's requests to a replica that holds it while that one is not
saturated, loads each cold adapter once, and unloads the least recently used
idle one to make room. llama.cpp keeps loading every adapter at start.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 01:14:38 +02:00
1 parent d41104a452
commit 2a1e0e6c4c
24 files changed
+1773 -35

No files matched your search

+19 -3
View File
@@ -114,6 +114,7 @@ rather than repeated.
| `owner` | `string` | [Model](#model) |
| `external` | `*External` | [Model](#model) |
| `adapters` | `[]Adapter` | [Model](#model) |
| `max_loaded_adapters` | `int` | [Model](#model) |
| `fallbacks` | `[]string` | [Model](#model) |
| `canary` | `*Canary` | [Model](#model) |
| `eval` | `*EvalGate` | [Model](#model) |
@@ -372,8 +373,10 @@ Name is what clients ask for, like a model name.
`string`
Source is the adapter file (GGUF for llama.cpp), fetched by each node
as the model's weights are.
Source is the adapter: a GGUF file for llama.cpp; for vLLM a PEFT
adapter directory (a mirror version ending in "/", a path on the node,
or a Hugging Face repository id). Fetched by each node as the model's
weights are.
### `scale`
@@ -1078,7 +1081,20 @@ Takes a [External](#external) block.
Adapters are LoRA adapters served on this model's replicas, each asked
for by its own name as though it were a model. Every replica loads
every adapter, applying one only to a request that names it.
every adapter, applying one only to a request that names it — unless
MaxLoadedAdapters is set.
### `max_loaded_adapters`
`int`
MaxLoadedAdapters, above zero, has each replica hold at most this many
adapters at once, loaded when a request first asks for one and unloaded
least recently used first to make room. The router sends a request for
an adapter to a replica that already holds it while that one has room
to take it. vLLM profiles only: llama-server loads adapters at start
and cannot load one afterwards. Zero loads every adapter on every
replica at start.
### `fallbacks`