LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters
A vLLM model can carry PEFT adapters, registered at start, or with max_loaded_adapters held at most N per replica: the router loads an adapter through vLLM's runtime LoRA API when a request first wants it, sends the adapter's requests to a replica that holds it while that one is not saturated, loads each cold adapter once, and unloads the least recently used idle one to make room. llama.cpp keeps loading every adapter at start. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
d41104a452
commit
2a1e0e6c4c
24 files changed
+1773
-35
No files matched your search
@@ -168,6 +168,11 @@ type Request struct {
|
||||
// ContentType is the body's, when it is not JSON: an audio upload's
|
||||
// multipart form.
|
||||
ContentType string
|
||||
// RetryStatus is one more status, besides the overloaded and unavailable
|
||||
// ones, that is returned to the caller unwritten rather than relayed: a
|
||||
// 404 for an adapter the router believed loaded and another router had
|
||||
// unloaded, which a load and a second try fix. Zero adds none.
|
||||
RetryStatus int
|
||||
// OnEvent, when set, is told the time since the previous streamed event
|
||||
// for each event after the first: about one token's time each.
|
||||
OnEvent func(gap time.Duration)
|
||||
@@ -277,7 +282,7 @@ func (p *Proxy) Do(ctx context.Context, w http.ResponseWriter, r Request) (Outco
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if retryableStatus(resp.StatusCode) {
|
||||
if retryableStatus(resp.StatusCode) || resp.StatusCode == r.RetryStatus {
|
||||
// Drain a little so the connection can be reused, then let the caller
|
||||
// pick somewhere else. Nothing has been written to the client.
|
||||
_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 4<<10))
|
||||
|
||||
Reference in new issue
Block a user