A vLLM model can carry PEFT adapters, registered at start, or with max_loaded_adapters held at most N per replica: the router loads an adapter through vLLM's runtime LoRA API when a request first wants it, sends the adapter's requests to a replica that holds it while that one is not saturated, loads each cold adapter once, and unloads the least recently used idle one to make room. llama.cpp keeps loading every adapter at start. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
53 lines
3.2 KiB
Markdown
53 lines
3.2 KiB
Markdown
## CHANGELOG
|
|
|
|
### LoRA adapters on vLLM, loaded on demand
|
|
|
|
- **vLLM models can carry adapters.** A PEFT adapter directory (a mirror
|
|
version ending in `/`, a path on the nodes, or a Hugging Face repository
|
|
id) is registered at start with `--enable-lora --lora-modules`, and a
|
|
request names it as its model, as on llama.cpp. `scale` applies on
|
|
llama.cpp only. SGLang and TensorRT-LLM profiles still refuse adapters.
|
|
- **`max_loaded_adapters: N` on a model** has each replica hold at most N of
|
|
its adapters. Replicas start holding none (vLLM with
|
|
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras N`, unless the
|
|
profile's args set their own). The router loads an adapter on a replica
|
|
through `/v1/load_lora_adapter` when a request first wants it there, and
|
|
unloads the least recently used one no request is running on to make room.
|
|
vLLM profiles only; llama.cpp keeps loading every adapter at start, since
|
|
llama-server can rescale an adapter it started with but not load another.
|
|
- **Adapter affinity:** a request for an adapter goes to a replica that has
|
|
it, or is loading it, while that replica is not saturated; otherwise to one
|
|
with room for another adapter; otherwise where load sends it — which is how
|
|
a busy adapter spreads to a second replica. Requests arriving during a load
|
|
wait for it instead of loading the adapter again.
|
|
- **Self-correcting:** each replica's `/v1/models` is read when the router
|
|
first meets it and every 30 seconds, so adapters loaded before a restart or
|
|
by another router are known; a 404 for an adapter the router thought loaded
|
|
loads it again and retries. An adapter vLLM refuses (not found, rank above
|
|
`--max-lora-rank`) fails the request with vLLM's reason, without trying it
|
|
on every replica.
|
|
- **Metrics:** `llmgw_adapter_affinity_total{model,result=hit|joined|loaded}`,
|
|
`llmgw_adapter_loads_total{model,outcome}`,
|
|
`llmgw_adapter_evictions_total{model,outcome}`.
|
|
- The admin API and `ctl model adapters` read the file as GGUF only for a
|
|
model with llama.cpp profiles.
|
|
|
|
## BACKLOG
|
|
|
|
**24 LoRA affinity (branch `features-c2`):** built `max_loaded_adapters`
|
|
(catalog), vLLM adapter support at start and on demand (`engines/lora.go`,
|
|
`vllm.go`), the agent linking fetched adapters as `llmgw-lora/<name>` in the
|
|
engines' working directory, the router's adapter tracker (`router/lora.go`:
|
|
loads/unloads with the engine key, LRU eviction never of an adapter in use,
|
|
one load per adapter per replica, `/v1/models` sync, 404 reload) and the
|
|
`adapter-affinity` scheduler filter in the chat, batch and long-context
|
|
profiles, with metrics and docs; tested against a fake of vLLM's HTTP
|
|
interface. **Left:** never run against a real vLLM — in particular that vLLM
|
|
resolves the relative `llmgw-lora/<name>` path against its working directory
|
|
on every node (multi-node too) and that unloading an adapter mid-request is
|
|
harmless; the agent still fetches every adapter at replica start, so a new
|
|
adapter needs a restart (fetching on first load would need an agent command
|
|
and a proto change); load timeout and resync interval are constants, not
|
|
config; SGLang's runtime LoRA API is not driven; no per-replica "adapters
|
|
held" view in the console or admin API; the debug profile has no affinity.
|