Merge C24 LoRA adapters on demand for vLLM, with affinity
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
bd3eca4092
24 files changed
+1773
-35
No files matched your search
@@ -4930,7 +4930,7 @@
|
||||
},
|
||||
"source": {
|
||||
"type": "string",
|
||||
"description": "Source is the adapter file (GGUF for llama.cpp), fetched by each node as the model's weights are."
|
||||
"description": "Source is the adapter: a GGUF file for llama.cpp; for vLLM a PEFT adapter directory (a mirror version ending in \"/\", a path on the node, or a Hugging Face repository id). Fetched by each node as the model's weights are."
|
||||
},
|
||||
"scale": {
|
||||
"type": "number",
|
||||
@@ -5077,7 +5077,11 @@
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/catalog.Adapter"
|
||||
},
|
||||
"description": "Adapters are LoRA adapters served on this model's replicas, each asked for by its own name as though it were a model. Every replica loads every adapter, applying one only to a request that names it."
|
||||
"description": "Adapters are LoRA adapters served on this model's replicas, each asked for by its own name as though it were a model. Every replica loads every adapter, applying one only to a request that names it — unless MaxLoadedAdapters is set."
|
||||
},
|
||||
"max_loaded_adapters": {
|
||||
"type": "integer",
|
||||
"description": "MaxLoadedAdapters, above zero, has each replica hold at most this many adapters at once, loaded when a request first asks for one and unloaded least recently used first to make room. The router sends a request for an adapter to a replica that already holds it while that one has room to take it. vLLM profiles only: llama-server loads adapters at start and cannot load one afterwards. Zero loads every adapter on every replica at start."
|
||||
},
|
||||
"fallbacks": {
|
||||
"type": "array",
|
||||
|
||||
@@ -1033,6 +1033,55 @@ The adapter file must be a GGUF LoRA made for this model — llama.cpp's
|
||||
`convert_lora_to_gguf.py` makes one from a PEFT adapter — and not a model with
|
||||
the adapter already merged in, which is what most "LoRA GGUF" repositories hold.
|
||||
|
||||
#### Adapters on vLLM, and loading them on demand
|
||||
|
||||
A vLLM model carries adapters the same way. There an adapter is a PEFT adapter
|
||||
directory (`adapter_config.json` and `adapter_model.safetensors`): a mirror
|
||||
version ending in `/`, a path on the nodes, or a Hugging Face repository id,
|
||||
which vLLM downloads itself. A request names the adapter as its model, as on
|
||||
llama.cpp; `scale` applies on llama.cpp only. Without anything more, every
|
||||
replica registers every adapter at start (`--enable-lora --lora-modules`).
|
||||
vLLM mixes one adapter per batch by default; raise that with `--max-loras` in
|
||||
the profile's `args`, and `--max-lora-rank` (16 by default) for adapters of a
|
||||
higher rank.
|
||||
|
||||
With many adapters, have each replica hold only some of them:
|
||||
|
||||
```yaml
|
||||
- name: qwen2.5-7b-instruct
|
||||
source: Qwen/Qwen2.5-7B-Instruct
|
||||
max_loaded_adapters: 4 # per replica; vllm profiles only
|
||||
adapters:
|
||||
- name: support-de
|
||||
source: http://mirror:8081/support-de/v3/
|
||||
- name: support-fr
|
||||
source: someone/support-fr-lora
|
||||
profiles: [...]
|
||||
```
|
||||
|
||||
Replicas then start holding none (vLLM runs with
|
||||
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras 4`). The first
|
||||
request for an adapter has the router load it on a replica through vLLM's
|
||||
`/v1/load_lora_adapter`, and the requests after it go to that replica while
|
||||
it is not saturated; when it is, the next request goes where load sends it and
|
||||
the adapter is loaded there too. Requests that arrive while an adapter is being
|
||||
loaded wait for that load rather than starting another. A replica that holds
|
||||
`max_loaded_adapters` already unloads the one least recently used, never one a
|
||||
request is running on. Each node still fetches every adapter when a replica
|
||||
starts, and links it as `llmgw-lora/<name>` in the directory engines run in,
|
||||
which is the path the router has vLLM load it from — so adding an adapter
|
||||
still needs the replicas restarted, as above. An adapter vLLM refuses (not
|
||||
found, a rank above `--max-lora-rank`, not for this model) fails the request
|
||||
with vLLM's reason.
|
||||
|
||||
What each replica holds is what the router loaded there, checked against each
|
||||
replica's `/v1/models` when the router first meets it and every 30 seconds.
|
||||
`llmgw_adapter_affinity_total` counts where requests found their adapter (hit,
|
||||
joined a load, loaded it), `llmgw_adapter_loads_total` and
|
||||
`llmgw_adapter_evictions_total` what that cost. llama.cpp keeps loading every
|
||||
adapter at start: llama-server can change the scale of an adapter it started
|
||||
with, not load a new one.
|
||||
|
||||
### Models served outside the cluster
|
||||
|
||||
A model can be an OpenAI-compatible API elsewhere — a hosted provider, or
|
||||
|
||||
+19
-3
@@ -115,6 +115,7 @@ rather than repeated.
|
||||
| `owner` | `string` | [Model](#model) |
|
||||
| `external` | `*External` | [Model](#model) |
|
||||
| `adapters` | `[]Adapter` | [Model](#model) |
|
||||
| `max_loaded_adapters` | `int` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
| `canary` | `*Canary` | [Model](#model) |
|
||||
| `eval` | `*EvalGate` | [Model](#model) |
|
||||
@@ -376,8 +377,10 @@ Name is what clients ask for, like a model name.
|
||||
|
||||
`string`
|
||||
|
||||
Source is the adapter file (GGUF for llama.cpp), fetched by each node
|
||||
as the model's weights are.
|
||||
Source is the adapter: a GGUF file for llama.cpp; for vLLM a PEFT
|
||||
adapter directory (a mirror version ending in "/", a path on the node,
|
||||
or a Hugging Face repository id). Fetched by each node as the model's
|
||||
weights are.
|
||||
|
||||
### `scale`
|
||||
|
||||
@@ -1090,7 +1093,20 @@ Takes a [External](#external) block.
|
||||
|
||||
Adapters are LoRA adapters served on this model's replicas, each asked
|
||||
for by its own name as though it were a model. Every replica loads
|
||||
every adapter, applying one only to a request that names it.
|
||||
every adapter, applying one only to a request that names it — unless
|
||||
MaxLoadedAdapters is set.
|
||||
|
||||
### `max_loaded_adapters`
|
||||
|
||||
`int`
|
||||
|
||||
MaxLoadedAdapters, above zero, has each replica hold at most this many
|
||||
adapters at once, loaded when a request first asks for one and unloaded
|
||||
least recently used first to make room. The router sends a request for
|
||||
an adapter to a replica that already holds it while that one has room
|
||||
to take it. vLLM profiles only: llama-server loads adapters at start
|
||||
and cannot load one afterwards. Zero loads every adapter on every
|
||||
replica at start.
|
||||
|
||||
### `fallbacks`
|
||||
|
||||
|
||||
Reference in new issue
Block a user