Serve LoRA adapters on llama.cpp models, asked for by name
A model's adapters are fetched by each node with its weights and loaded unapplied at start; a request naming an adapter reaches the model's replicas with llama.cpp's per-request lora field. Adapter names share the model namespace, are listed, and follow allowlists on their own. The router's config seed and pushed catalog now build the name map in one place. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
94eec557ad
commit
cf53904cc5
18 files changed
+542
-41
No files matched your search
@@ -15,6 +15,9 @@ rather than repeated.
|
||||
|
||||
| Key | Type | In |
|
||||
|---|---|---|
|
||||
| `name` | `string` | [Adapter](#adapter) |
|
||||
| `source` | `string` | [Adapter](#adapter) |
|
||||
| `scale` | `float64` | [Adapter](#adapter) |
|
||||
| `enabled` | `bool` | [Admission](#admission) |
|
||||
| `bands` | `[]Band` | [Admission](#admission) |
|
||||
| `default_band` | `string` | [Admission](#admission) |
|
||||
@@ -99,6 +102,7 @@ rather than repeated.
|
||||
| `class` | `string` | [Model](#model) |
|
||||
| `owner` | `string` | [Model](#model) |
|
||||
| `external` | `*External` | [Model](#model) |
|
||||
| `adapters` | `[]Adapter` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
| `routing_profile` | `string` | [Model](#model) |
|
||||
| `slo` | `SLO` | [Model](#model) |
|
||||
@@ -200,6 +204,27 @@ rather than repeated.
|
||||
| `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
|
||||
| `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
|
||||
|
||||
## Adapter
|
||||
|
||||
### `name`
|
||||
|
||||
`string`
|
||||
|
||||
Name is what clients ask for, like a model name.
|
||||
|
||||
### `source`
|
||||
|
||||
`string`
|
||||
|
||||
Source is the adapter file (GGUF for llama.cpp), fetched by each node
|
||||
as the model's weights are.
|
||||
|
||||
### `scale`
|
||||
|
||||
`float64`
|
||||
|
||||
Scale is how strongly it is applied; zero takes 1.
|
||||
|
||||
## Admission
|
||||
|
||||
### `enabled`
|
||||
@@ -805,6 +830,14 @@ allowlist.
|
||||
|
||||
Takes a [External](#external) block.
|
||||
|
||||
### `adapters`
|
||||
|
||||
`[]Adapter`
|
||||
|
||||
Adapters are LoRA adapters served on this model's replicas, each asked
|
||||
for by its own name as though it were a model. Every replica loads
|
||||
every adapter, applying one only to a request that names it.
|
||||
|
||||
### `fallbacks`
|
||||
|
||||
`[]string`
|
||||
|
||||
Reference in new issue
Block a user