Merge C24 LoRA adapters on demand for vLLM, with affinity
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
bd3eca4092
24 files changed
+1773
-35
No files matched your search
+19
-3
@@ -115,6 +115,7 @@ rather than repeated.
|
||||
| `owner` | `string` | [Model](#model) |
|
||||
| `external` | `*External` | [Model](#model) |
|
||||
| `adapters` | `[]Adapter` | [Model](#model) |
|
||||
| `max_loaded_adapters` | `int` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
| `canary` | `*Canary` | [Model](#model) |
|
||||
| `eval` | `*EvalGate` | [Model](#model) |
|
||||
@@ -376,8 +377,10 @@ Name is what clients ask for, like a model name.
|
||||
|
||||
`string`
|
||||
|
||||
Source is the adapter file (GGUF for llama.cpp), fetched by each node
|
||||
as the model's weights are.
|
||||
Source is the adapter: a GGUF file for llama.cpp; for vLLM a PEFT
|
||||
adapter directory (a mirror version ending in "/", a path on the node,
|
||||
or a Hugging Face repository id). Fetched by each node as the model's
|
||||
weights are.
|
||||
|
||||
### `scale`
|
||||
|
||||
@@ -1090,7 +1093,20 @@ Takes a [External](#external) block.
|
||||
|
||||
Adapters are LoRA adapters served on this model's replicas, each asked
|
||||
for by its own name as though it were a model. Every replica loads
|
||||
every adapter, applying one only to a request that names it.
|
||||
every adapter, applying one only to a request that names it — unless
|
||||
MaxLoadedAdapters is set.
|
||||
|
||||
### `max_loaded_adapters`
|
||||
|
||||
`int`
|
||||
|
||||
MaxLoadedAdapters, above zero, has each replica hold at most this many
|
||||
adapters at once, loaded when a request first asks for one and unloaded
|
||||
least recently used first to make room. The router sends a request for
|
||||
an adapter to a replica that already holds it while that one has room
|
||||
to take it. vLLM profiles only: llama-server loads adapters at start
|
||||
and cannot load one afterwards. Zero loads every adapter on every
|
||||
replica at start.
|
||||
|
||||
### `fallbacks`
|
||||
|
||||
|
||||
Reference in new issue
Block a user