LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

A vLLM model can carry PEFT adapters, registered at start, or with
max_loaded_adapters held at most N per replica: the router loads an adapter
through vLLM's runtime LoRA API when a request first wants it, sends the
adapter's requests to a replica that holds it while that one is not
saturated, loads each cold adapter once, and unloads the least recently used
idle one to make room. llama.cpp keeps loading every adapter at start.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 01:14:38 +02:00
1 parent d41104a452
commit 2a1e0e6c4c
24 files changed
+1773 -35

No files matched your search

+6 -2
View File
@@ -4930,7 +4930,7 @@
},
"source": {
"type": "string",
"description": "Source is the adapter file (GGUF for llama.cpp), fetched by each node as the model's weights are."
"description": "Source is the adapter: a GGUF file for llama.cpp; for vLLM a PEFT adapter directory (a mirror version ending in \"/\", a path on the node, or a Hugging Face repository id). Fetched by each node as the model's weights are."
},
"scale": {
"type": "number",
@@ -5077,7 +5077,11 @@
"items": {
"$ref": "#/components/schemas/catalog.Adapter"
},
"description": "Adapters are LoRA adapters served on this model's replicas, each asked for by its own name as though it were a model. Every replica loads every adapter, applying one only to a request that names it."
"description": "Adapters are LoRA adapters served on this model's replicas, each asked for by its own name as though it were a model. Every replica loads every adapter, applying one only to a request that names it — unless MaxLoadedAdapters is set."
},
"max_loaded_adapters": {
"type": "integer",
"description": "MaxLoadedAdapters, above zero, has each replica hold at most this many adapters at once, loaded when a request first asks for one and unloaded least recently used first to make room. The router sends a request for an adapter to a replica that already holds it while that one has room to take it. vLLM profiles only: llama-server loads adapters at start and cannot load one afterwards. Zero loads every adapter on every replica at start."
},
"fallbacks": {
"type": "array",