Merge C24 LoRA adapters on demand for vLLM, with affinity
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
bd3eca4092
24 files changed
+1773
-35
No files matched your search
@@ -0,0 +1,52 @@
|
||||
## CHANGELOG
|
||||
|
||||
### LoRA adapters on vLLM, loaded on demand
|
||||
|
||||
- **vLLM models can carry adapters.** A PEFT adapter directory (a mirror
|
||||
version ending in `/`, a path on the nodes, or a Hugging Face repository
|
||||
id) is registered at start with `--enable-lora --lora-modules`, and a
|
||||
request names it as its model, as on llama.cpp. `scale` applies on
|
||||
llama.cpp only. SGLang and TensorRT-LLM profiles still refuse adapters.
|
||||
- **`max_loaded_adapters: N` on a model** has each replica hold at most N of
|
||||
its adapters. Replicas start holding none (vLLM with
|
||||
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras N`, unless the
|
||||
profile's args set their own). The router loads an adapter on a replica
|
||||
through `/v1/load_lora_adapter` when a request first wants it there, and
|
||||
unloads the least recently used one no request is running on to make room.
|
||||
vLLM profiles only; llama.cpp keeps loading every adapter at start, since
|
||||
llama-server can rescale an adapter it started with but not load another.
|
||||
- **Adapter affinity:** a request for an adapter goes to a replica that has
|
||||
it, or is loading it, while that replica is not saturated; otherwise to one
|
||||
with room for another adapter; otherwise where load sends it — which is how
|
||||
a busy adapter spreads to a second replica. Requests arriving during a load
|
||||
wait for it instead of loading the adapter again.
|
||||
- **Self-correcting:** each replica's `/v1/models` is read when the router
|
||||
first meets it and every 30 seconds, so adapters loaded before a restart or
|
||||
by another router are known; a 404 for an adapter the router thought loaded
|
||||
loads it again and retries. An adapter vLLM refuses (not found, rank above
|
||||
`--max-lora-rank`) fails the request with vLLM's reason, without trying it
|
||||
on every replica.
|
||||
- **Metrics:** `llmgw_adapter_affinity_total{model,result=hit|joined|loaded}`,
|
||||
`llmgw_adapter_loads_total{model,outcome}`,
|
||||
`llmgw_adapter_evictions_total{model,outcome}`.
|
||||
- The admin API and `ctl model adapters` read the file as GGUF only for a
|
||||
model with llama.cpp profiles.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**24 LoRA affinity (branch `features-c2`):** built `max_loaded_adapters`
|
||||
(catalog), vLLM adapter support at start and on demand (`engines/lora.go`,
|
||||
`vllm.go`), the agent linking fetched adapters as `llmgw-lora/<name>` in the
|
||||
engines' working directory, the router's adapter tracker (`router/lora.go`:
|
||||
loads/unloads with the engine key, LRU eviction never of an adapter in use,
|
||||
one load per adapter per replica, `/v1/models` sync, 404 reload) and the
|
||||
`adapter-affinity` scheduler filter in the chat, batch and long-context
|
||||
profiles, with metrics and docs; tested against a fake of vLLM's HTTP
|
||||
interface. **Left:** never run against a real vLLM — in particular that vLLM
|
||||
resolves the relative `llmgw-lora/<name>` path against its working directory
|
||||
on every node (multi-node too) and that unloading an adapter mid-request is
|
||||
harmless; the agent still fetches every adapter at replica start, so a new
|
||||
adapter needs a restart (fetching on first load would need an agent command
|
||||
and a proto change); load timeout and resync interval are constants, not
|
||||
config; SGLang's runtime LoRA API is not driven; no per-replica "adapters
|
||||
held" view in the console or admin API; the debug profile has no affinity.
|
||||
Reference in new issue
Block a user