Files
llmgw/notes/c24-lora.md
T
Ilker ManapandClaude Opus 5.5 2a1e0e6c4c LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters
A vLLM model can carry PEFT adapters, registered at start, or with
max_loaded_adapters held at most N per replica: the router loads an adapter
through vLLM's runtime LoRA API when a request first wants it, sends the
adapter's requests to a replica that holds it while that one is not
saturated, loads each cold adapter once, and unloads the least recently used
idle one to make room. llama.cpp keeps loading every adapter at start.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:14:38 +02:00

3.2 KiB

CHANGELOG

LoRA adapters on vLLM, loaded on demand

  • vLLM models can carry adapters. A PEFT adapter directory (a mirror version ending in /, a path on the nodes, or a Hugging Face repository id) is registered at start with --enable-lora --lora-modules, and a request names it as its model, as on llama.cpp. scale applies on llama.cpp only. SGLang and TensorRT-LLM profiles still refuse adapters.
  • max_loaded_adapters: N on a model has each replica hold at most N of its adapters. Replicas start holding none (vLLM with VLLM_ALLOW_RUNTIME_LORA_UPDATING=True and --max-loras N, unless the profile's args set their own). The router loads an adapter on a replica through /v1/load_lora_adapter when a request first wants it there, and unloads the least recently used one no request is running on to make room. vLLM profiles only; llama.cpp keeps loading every adapter at start, since llama-server can rescale an adapter it started with but not load another.
  • Adapter affinity: a request for an adapter goes to a replica that has it, or is loading it, while that replica is not saturated; otherwise to one with room for another adapter; otherwise where load sends it — which is how a busy adapter spreads to a second replica. Requests arriving during a load wait for it instead of loading the adapter again.
  • Self-correcting: each replica's /v1/models is read when the router first meets it and every 30 seconds, so adapters loaded before a restart or by another router are known; a 404 for an adapter the router thought loaded loads it again and retries. An adapter vLLM refuses (not found, rank above --max-lora-rank) fails the request with vLLM's reason, without trying it on every replica.
  • Metrics: llmgw_adapter_affinity_total{model,result=hit|joined|loaded}, llmgw_adapter_loads_total{model,outcome}, llmgw_adapter_evictions_total{model,outcome}.
  • The admin API and ctl model adapters read the file as GGUF only for a model with llama.cpp profiles.

BACKLOG

24 LoRA affinity (branch features-c2): built max_loaded_adapters (catalog), vLLM adapter support at start and on demand (engines/lora.go, vllm.go), the agent linking fetched adapters as llmgw-lora/<name> in the engines' working directory, the router's adapter tracker (router/lora.go: loads/unloads with the engine key, LRU eviction never of an adapter in use, one load per adapter per replica, /v1/models sync, 404 reload) and the adapter-affinity scheduler filter in the chat, batch and long-context profiles, with metrics and docs; tested against a fake of vLLM's HTTP interface. Left: never run against a real vLLM — in particular that vLLM resolves the relative llmgw-lora/<name> path against its working directory on every node (multi-node too) and that unloading an adapter mid-request is harmless; the agent still fetches every adapter at replica start, so a new adapter needs a restart (fetching on first load would need an agent command and a proto change); load timeout and resync interval are constants, not config; SGLang's runtime LoRA API is not driven; no per-replica "adapters held" view in the console or admin API; the debug profile has no affinity.