LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters
A vLLM model can carry PEFT adapters, registered at start, or with max_loaded_adapters held at most N per replica: the router loads an adapter through vLLM's runtime LoRA API when a request first wants it, sends the adapter's requests to a replica that holds it while that one is not saturated, loads each cold adapter once, and unloads the least recently used idle one to make room. llama.cpp keeps loading every adapter at start. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
d41104a452
commit
2a1e0e6c4c
24 files changed
+1773
-35
No files matched your search
@@ -4930,7 +4930,7 @@
|
||||
},
|
||||
"source": {
|
||||
"type": "string",
|
||||
"description": "Source is the adapter file (GGUF for llama.cpp), fetched by each node as the model's weights are."
|
||||
"description": "Source is the adapter: a GGUF file for llama.cpp; for vLLM a PEFT adapter directory (a mirror version ending in \"/\", a path on the node, or a Hugging Face repository id). Fetched by each node as the model's weights are."
|
||||
},
|
||||
"scale": {
|
||||
"type": "number",
|
||||
@@ -5077,7 +5077,11 @@
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/catalog.Adapter"
|
||||
},
|
||||
"description": "Adapters are LoRA adapters served on this model's replicas, each asked for by its own name as though it were a model. Every replica loads every adapter, applying one only to a request that names it."
|
||||
"description": "Adapters are LoRA adapters served on this model's replicas, each asked for by its own name as though it were a model. Every replica loads every adapter, applying one only to a request that names it — unless MaxLoadedAdapters is set."
|
||||
},
|
||||
"max_loaded_adapters": {
|
||||
"type": "integer",
|
||||
"description": "MaxLoadedAdapters, above zero, has each replica hold at most this many adapters at once, loaded when a request first asks for one and unloaded least recently used first to make room. The router sends a request for an adapter to a replica that already holds it while that one has room to take it. vLLM profiles only: llama-server loads adapters at start and cannot load one afterwards. Zero loads every adapter on every replica at start."
|
||||
},
|
||||
"fallbacks": {
|
||||
"type": "array",
|
||||
|
||||
Reference in new issue
Block a user