Ilker ManapandClaude Opus 5.5 2a1e0e6c4c LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters ...

A vLLM model can carry PEFT adapters, registered at start, or with
max_loaded_adapters held at most N per replica: the router loads an adapter
through vLLM's runtime LoRA API when a request first wants it, sends the
adapter's requests to a replica that holds it while that one is not
saturated, loads each cold adapter once, and unloads the least recently used
idle one to make room. llama.cpp keeps loading every adapter at start.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

2026-10-05 01:14:38 +02:00

..

catalog_test.go

A judge model on the cluster: role guard in the catalog, judge in a guard policy

2026-10-04 15:09:31 +02:00

catalog.go

LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

2026-10-05 01:14:38 +02:00

cpu_test.go

Add the llama.cpp engine, and let a profile ask for no GPU

2026-09-16 08:14:05 +02:00

live.go

Let the catalog change while the cluster is running

2026-09-17 01:08:21 +02:00

lora_test.go

LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

2026-10-05 01:14:38 +02:00

mig_test.go

MIG: a card in MIG mode is placed as its instances, each a GPU of class <card>-mig-<profile>

2026-10-05 00:01:14 +02:00

schedule_test.go

Scaling by the clock: schedule windows with their own min and max, prewarm, control.timezone

2026-10-04 23:16:56 +02:00

trtllm_test.go

TensorRT-LLM as an engine, and models from Ollama into the mirror

2026-10-05 00:02:01 +02:00