Merge C24 LoRA adapters on demand for vLLM, with affinity

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 01:15:12 +02:00
commit bd3eca4092
24 files changed
+1773 -35

No files matched your search

+19 -3
View File
@@ -115,6 +115,7 @@ rather than repeated.
| `owner` | `string` | [Model](#model) |
| `external` | `*External` | [Model](#model) |
| `adapters` | `[]Adapter` | [Model](#model) |
| `max_loaded_adapters` | `int` | [Model](#model) |
| `fallbacks` | `[]string` | [Model](#model) |
| `canary` | `*Canary` | [Model](#model) |
| `eval` | `*EvalGate` | [Model](#model) |
@@ -376,8 +377,10 @@ Name is what clients ask for, like a model name.
`string`
Source is the adapter file (GGUF for llama.cpp), fetched by each node
as the model's weights are.
Source is the adapter: a GGUF file for llama.cpp; for vLLM a PEFT
adapter directory (a mirror version ending in "/", a path on the node,
or a Hugging Face repository id). Fetched by each node as the model's
weights are.
### `scale`
@@ -1090,7 +1093,20 @@ Takes a [External](#external) block.
Adapters are LoRA adapters served on this model's replicas, each asked
for by its own name as though it were a model. Every replica loads
every adapter, applying one only to a request that names it.
every adapter, applying one only to a request that names it — unless
MaxLoadedAdapters is set.
### `max_loaded_adapters`
`int`
MaxLoadedAdapters, above zero, has each replica hold at most this many
adapters at once, loaded when a request first asks for one and unloaded
least recently used first to make room. The router sends a request for
an adapter to a replica that already holds it while that one has room
to take it. vLLM profiles only: llama-server loads adapters at start
and cannot load one afterwards. Zero loads every adapter on every
replica at start.
### `fallbacks`