Serve LoRA adapters on llama.cpp models, asked for by name
A model's adapters are fetched by each node with its weights and loaded unapplied at start; a request naming an adapter reaches the model's replicas with llama.cpp's per-request lora field. Adapter names share the model namespace, are listed, and follow allowlists on their own. The router's config seed and pushed catalog now build the name map in one place. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
94eec557ad
commit
cf53904cc5
18 files changed
+542
-41
No files matched your search
@@ -671,6 +671,34 @@ llmgw ctl explain-placement qwen2.5-coder-3b-instruct
|
||||
which names every candidate and why each was rejected, rather than leaving you
|
||||
to infer it.
|
||||
|
||||
### LoRA adapters
|
||||
|
||||
A llama.cpp model can carry LoRA adapters, each asked for by its own name as
|
||||
though it were a model:
|
||||
|
||||
```yaml
|
||||
- name: qwen2.5-coder-7b-instruct
|
||||
source: http://mirror:8081/qwen2.5-coder-7b-instruct/5d567d26/model.gguf
|
||||
adapters:
|
||||
- name: coder-sql
|
||||
source: http://mirror:8081/coder-sql/v1/adapter.gguf
|
||||
scale: 1.0 # optional; 1 when left out
|
||||
profiles: [...]
|
||||
```
|
||||
|
||||
Every replica of the model fetches every adapter and loads it at start, applied
|
||||
to nothing. A request for `coder-sql` goes to the model's replicas with the
|
||||
adapter applied to it alone; a request for the model gets the model. Adapter
|
||||
names share the namespace of models and aliases, appear in `/v1/models`, and
|
||||
follow key allowlists on their own: a key allowed `coder-sql` reaches that
|
||||
adapter and not the model under it, while a key allowed the model reaches its
|
||||
adapters too. Adding or removing an adapter is a change to the model; running
|
||||
replicas load the new set when restarted (`llmgw ctl restart --model M`).
|
||||
|
||||
The adapter file must be a GGUF LoRA made for this model — llama.cpp's
|
||||
`convert_lora_to_gguf.py` makes one from a PEFT adapter — and not a model with
|
||||
the adapter already merged in, which is what most "LoRA GGUF" repositories hold.
|
||||
|
||||
### Models served outside the cluster
|
||||
|
||||
A model can be an OpenAI-compatible API elsewhere — a hosted provider, or
|
||||
|
||||
@@ -15,6 +15,9 @@ rather than repeated.
|
||||
|
||||
| Key | Type | In |
|
||||
|---|---|---|
|
||||
| `name` | `string` | [Adapter](#adapter) |
|
||||
| `source` | `string` | [Adapter](#adapter) |
|
||||
| `scale` | `float64` | [Adapter](#adapter) |
|
||||
| `enabled` | `bool` | [Admission](#admission) |
|
||||
| `bands` | `[]Band` | [Admission](#admission) |
|
||||
| `default_band` | `string` | [Admission](#admission) |
|
||||
@@ -99,6 +102,7 @@ rather than repeated.
|
||||
| `class` | `string` | [Model](#model) |
|
||||
| `owner` | `string` | [Model](#model) |
|
||||
| `external` | `*External` | [Model](#model) |
|
||||
| `adapters` | `[]Adapter` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
| `routing_profile` | `string` | [Model](#model) |
|
||||
| `slo` | `SLO` | [Model](#model) |
|
||||
@@ -200,6 +204,27 @@ rather than repeated.
|
||||
| `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
|
||||
| `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
|
||||
|
||||
## Adapter
|
||||
|
||||
### `name`
|
||||
|
||||
`string`
|
||||
|
||||
Name is what clients ask for, like a model name.
|
||||
|
||||
### `source`
|
||||
|
||||
`string`
|
||||
|
||||
Source is the adapter file (GGUF for llama.cpp), fetched by each node
|
||||
as the model's weights are.
|
||||
|
||||
### `scale`
|
||||
|
||||
`float64`
|
||||
|
||||
Scale is how strongly it is applied; zero takes 1.
|
||||
|
||||
## Admission
|
||||
|
||||
### `enabled`
|
||||
@@ -805,6 +830,14 @@ allowlist.
|
||||
|
||||
Takes a [External](#external) block.
|
||||
|
||||
### `adapters`
|
||||
|
||||
`[]Adapter`
|
||||
|
||||
Adapters are LoRA adapters served on this model's replicas, each asked
|
||||
for by its own name as though it were a model. Every replica loads
|
||||
every adapter, applying one only to a request that names it.
|
||||
|
||||
### `fallbacks`
|
||||
|
||||
`[]string`
|
||||
|
||||
Reference in new issue
Block a user