Serve LoRA adapters on llama.cpp models, asked for by name
Release / build (push) Successful in 1m11s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m47s

A model's adapters are fetched by each node with its weights and loaded
unapplied at start; a request naming an adapter reaches the model's
replicas with llama.cpp's per-request lora field. Adapter names share the
model namespace, are listed, and follow allowlists on their own. The
router's config seed and pushed catalog now build the name map in one place.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-30 23:40:47 +02:00
1 parent 94eec557ad
commit cf53904cc5
18 files changed
+542 -41

No files matched your search

+28
View File
@@ -671,6 +671,34 @@ llmgw ctl explain-placement qwen2.5-coder-3b-instruct
which names every candidate and why each was rejected, rather than leaving you
to infer it.
### LoRA adapters
A llama.cpp model can carry LoRA adapters, each asked for by its own name as
though it were a model:
```yaml
- name: qwen2.5-coder-7b-instruct
source: http://mirror:8081/qwen2.5-coder-7b-instruct/5d567d26/model.gguf
adapters:
- name: coder-sql
source: http://mirror:8081/coder-sql/v1/adapter.gguf
scale: 1.0 # optional; 1 when left out
profiles: [...]
```
Every replica of the model fetches every adapter and loads it at start, applied
to nothing. A request for `coder-sql` goes to the model's replicas with the
adapter applied to it alone; a request for the model gets the model. Adapter
names share the namespace of models and aliases, appear in `/v1/models`, and
follow key allowlists on their own: a key allowed `coder-sql` reaches that
adapter and not the model under it, while a key allowed the model reaches its
adapters too. Adding or removing an adapter is a change to the model; running
replicas load the new set when restarted (`llmgw ctl restart --model M`).
The adapter file must be a GGUF LoRA made for this model — llama.cpp's
`convert_lora_to_gguf.py` makes one from a PEFT adapter — and not a model with
the adapter already merged in, which is what most "LoRA GGUF" repositories hold.
### Models served outside the cluster
A model can be an OpenAI-compatible API elsewhere — a hosted provider, or